Given that mask-wearing proved to be an important tool to slow the spread of infection during the COVID-19 pandemic, investigating the psychological and cultural factors that influence norms for mask wearing across cultures is exceptionally important. One factor that may influence mask wearing behavior is the degree to which people believe masks potentially impair emotion recognition. Based on previous research suggesting that there may be cultural differences in facial regions that people in Japan and the United States attend to when inferring a target’s emotional state, we predicted that Americans would perceive masks (which cover the mouth) as more likely to impair emotion recognition, whereas Japanese would perceive facial coverings that conceal the eye region (sunglasses) to be more likely to impair emotion recognition. The results showed that Japanese participants reported wearing masks more than Americans. Americans also reported higher expected difficulty in interpreting emotions of individuals wearing masks (vs. sunglasses), while Japanese reported the reverse effect. Importantly, expectations about the negative impact of facial masks on emotion recognition explained cultural differences in mask-wearing behavior, even accounting for existing social norms.