Research themes

How does the brain decipher voice information in spoken language?

The field of social cognitive neuroscience is developing rapidly, and there is a recent surge of interest in how humans communicate their emotions and respond to emotional stimuli. In this research program, we investigate a topic that has been somewhat neglected in this growing field—how emotions are expressed and understood from the human voice while speaking, and how related mental functions are structured in the brain.

To recognize vocal expressions of emotion, for example that convey anger or joy, listeners must process dynamic acoustic properties of speech–i.e., ongoing fluctuations in pitch, loudness, and rhythm which differentiate over time in emotionally meaningful ways. The fact that emotional expressions in the voice are uniquely represented across time raises a critical empirical issue: how quickly do we detect emotions when listening to a speaker's voice? And what neural mechanisms are involved?

Another question that we are addressing is: when listeners recognize vocal expressions of emotion, does this information guide their visual attention and/or judgements of visual stimuli (e.g., facial expressions) in systematic ways? Answering this question will tell us much about natural social interactions, where humans are typically confronted by emotional cues in more than one sensory modality and must integrate these different cues in socially meaningful and adaptive ways.

Most of our studies involve young, healthy adults and our questions are being tested from different vantage points, using behavioural approaches, eye-tracking, electrophysiology (ERPs), and neuroimaging. Our research will lead to a more sophisticated model of the neuro-cognitive mechanisms that support emotional communication through the voice, and in broad terms, it will shed light on the uniquely human capacity to communicate both linguistic and emotional meanings using complex auditory signals.

→ Learn about the Facial Affect Decision Task (FADT)

Representative publications

• Jiang, X., Sanford, R. & Pell, M.D. (2018). Neural architecture underlying person perception from in-group and out-group voices. NeuroImage, 181, 582-597.
• Pell, M.D., Rothermich, K., Liu, P., Paulmann, S., Sethi, S, & Rigoulot, S. (2015). Preferential decoding of emotion from human non-linguistic vocalizations versus speech prosody. Biological Psychology, 111, 14-25.
• Jiang, X. & Pell, M.D. (2015). On how the brain decodes vocal cues about speaker confidence. Cortex, 66, 9-34.
• Pell, M.D. & Kotz, S.A. (2011). On the time course of vocal emotion recognition. PLoS ONE, 6(11): e27256.
• Pell, M.D. (2006). Cerebral mechanisms for understanding emotional prosody in speech. Brain and Language, 96, 221-234.
• Pell, M.D. (2005). Prosody-face interactions in emotional processing as revealed by the facial affect decision task. Journal of Nonverbal Behavior, 29(4), 193-215.

Research funded by:

How does our culture and identity affect vocal communication?

It is widely believed that due to their biological importance, emotional expressions display invariant properties which can be recognized irrespective of an individual's learning or cultural background (i.e., that emotional communication involves 'universal' principles, Ekman, Sorenson, & Friesen, 1969). However, this idea is based largely on evidence from facial expressions of emotion as opposed to vocal expressions.

The goal of this research program is to determine whether vocal expressions of discrete emotion are similar in the acoustic form across languages and whether they are recognized in a similar manner in speech by listeners from different language backgrounds.

Our experiments are focusing on the communication abilities of young healthy adults using an inventory of emotional utterances produced by speakers of major world languages (e.g., English, Hindi, Arabic, Mandarin Chinese). We are using behavioural approaches (e.g., priming, auditory gating) and conducting acoustic analyses to answer the following questions: Do emotion expressions vary in their acoustic form among languages? Does emotion recognition rely on the same acoustic features and have the same time course among languages? And, do listeners recognize vocal emotions when listening to a foreign language? We hypothesize that different facets of emotional communication are subject to language-independent ('universal') as well as language-specific factors that will be systematically illuminated by our studies.

Our cross-cultural studies will highlight the extent to which all humans share a common biological endowment for communicating emotions in the voice, while pointing to conditions where socio-cultural conventions might lead to mistaken impressions or "miscommunications" in this channel. These questions are increasingly pertinent in today's "global village", particularly in Canada, where people from different linguistic and cultural backgrounds are increasingly in contact with one another.

Representative publications

• Jiang, X., Gossack-Keenan, K. & Pell, M.D. (2020). To believe or not to believe: How voice and accent information in speech alter listener impressions of trust. Quarterly Journal of Experimental Psychology, 73(1), 55-79.
• Liu, P., Rigoulot, S., & Pell, M.D. (2017). Cultural immersion alters emotion perception: Neurophysiological evidence from Chinese immigrants to Canada. Social Neuroscience, 12(6), 685-700.
• Liu, P., Rigoulot, S., & Pell, M.D. (2015). Culture modulates the brain response to human expressions of emotion: electrophysiological evidence. Neuropsychologia, 67, 1-13.
• Pell, M.D., Paulmann, S., Dara, C., Alasseri, A., & Kotz, S.A. (2009). Factors in the recognition of vocally expressed emotions: a comparison of four languages. Journal of Phonetics, 37, 417-435.
• Pell, M.D., Monetta, L., Paulmann, S. & Kotz, S.A. (2009). Recognizing emotions in a foreign language. Journal of Nonverbal Behavior, 33, 107-120.

Research funded by:

How does Parkinson's disease affect social aspects of communication?

Over the lifespan, human health and well-being hinges on the ability to communicate effectively with others and to maintain social relationships. While most take these skills for granted, the ability to function successfully in social situations is highly complex and depends on the preservation of different mental abilities, such as the ability to infer the emotions of others, to determine their perspectives and intentions, and to appropriately use this knowledge to guide behaviour and interpersonal communication.

In adults with Parkinson's disease, degeneration of the brain leads to progressive difficulties that affect movement, but also specific mental abilities that are critical for social functioning.

The goal of this research is to specify how changes in mental functioning in individuals with Parkinson's disease impact on their ability to process emotions and to make social inferences (e.g., take other people's perspectives, show empathy, etc.). At the same time, our research will document the impact of Parkinson's disease on interpersonal communication from the perspective of listeners naïve to the patient's disease status.

Our undertakings will lead to new clinical insights about how mental and communication skills are affected by Parkinson's disease, and shed light on the psychological and social barriers faced by individuals suffering from the disease. Our findings will serve as a vital resource for clinicians and caregivers who work to maintain the quality of life and independence of individuals living with Parkinson's disease.

Representative publications

• Schwartz, R. & Pell, M.D. (2017). When emotion and expression diverge: the social costs of Parkinson's disease. Journal of Clinical and Experimental Neuropsychology, 39(3), 211-230.
• Garrido-Vásquez, P., Pell, M.D., Paulmann, S., Strecker, K., Schwarz, J., & Kotz, S.A. (2013). An ERP study of vocal emotion processing in asymmetric Parkinson's disease. Social, Cognitive and Affective Neuroscience, 8(8), 918-927.
• Jaywant, A. & Pell, M.D. (2010). Listener impressions of speakers with Parkinson's disease. Journal of the International Neuropsychological Society, 16, 49-57.
• Monetta, L., Grindrod, C. & Pell, M.D. (2009). Irony comprehension and theory of mind deficits in patients with Parkinson's disease. Cortex, 45, 972-981.
• Pell, M.D. & Leonard, C.L. (2003). Processing emotional tone from speech in Parkinson's disease: a role for the basal ganglia. Cognitive, Affective, & Behavioral Neuroscience, 3, 275-288.

Research (2004-2016) was funded by:

Recording inventories available to researchers

Emotional prosody recording inventories

Our lab has made significant efforts over the years to develop acoustically and perceptually-defined speech materials that can be used in studies of emotional prosody.

Currently, we have several databases containing short utterances produced in four different languages: English, Arabic, Hindi, and Mandarin Chinese. Each database was constructed in a similar manner and is comprised of several hundred individual recordings produced to four native speakers of each language.

Each database contains:

• pseudo-utterances and utterances that are emotionally congruent with the prosody;
• produced by 2 female and 2 male speakers of the language;
• produced in 6 different emotions (happy, angry, fear, sad, disgust, surprise) and neutral affect;
• approximately 30 unique statements per emotion category (or 30 unique pseudo-utterances produced to convey all emotions).

RISC Video Database — Relational Inference in Social Cognition

RISC is an English-language database composed of short video vignettes depicting sincere, jocular, sarcastic, and white lie social exchanges between two people. Stimuli carefully manipulated the social relationship between communication partners (e.g., boss/employee, couple) and the availability of contextual cues (e.g. preceding conversations, physical objects) while controlling for major differences in the linguistic content of matched items. In total, the database entails 600 videos that have been validated concerning the identification of different speaker intentions as well as their perceived politeness, appropriateness and socio-emotional impact. Results of the validation studies are, among other information, provided in a detailed excel sheet.

We believe that RISC will prove highly constructive as a tool in future research on social cognition, inter-personal communication, and the interpretation of speaker intentions in both healthy adults and clinical populations. The open-access and peer-reviewed research article is available through the PLOS ONE website.

If you would like to know more about the project and the primary researcher behind it, Dr. Kathrin Rothermich, listen to the Academic Minute and check out the media coverage by McGill University, IFL Science, and Daily Mail. If you are interested in obtaining videos from the RISC database for research purposes, please fill out the RISC Request Form.

Research funded by: