The (neuro)cognition of visual language

How do people comprehend graphics and visual narratives? We are especially interested in how people’s brains process graphics and sequences of images, and connect them with written language.

Our primary line of research has been to investigate the processing of sequences of images. Many people assume that sequential images are universally understood and don’t involve specialized knowledge to comprehend. We have shown that these assumptions are not true.

We’ve argued that comprehending a sequence of images involves a “narrative grammar” that is structured with similar principles to the grammar of sentences. This narrative grammar helps package aspects of meaning that belong to a parallel conceptual structure.

We’ve investigated the way the people process visual sequences using many techniques, but many of our studies use EEG, which is a way to measure the electrical activity of the brain, resulting in brainwaves. Using this technique, we’ve shown that processing a sequence of images evokes similar brain responses as processing a sequence of words in sentences.

All these aspects of visual narrative comprehension are explored at length in the book Who Understands Comics? However, additional key papers from us and our colleagues are:

Visual Narrative Grammar

With our collaborators we’ve also explored the processing of visual narratives by people with neurodiversities and by children. For example, along with Emily Coderre (University of Vermont), we’ve been exploring how people with autism process visual narrative sequences. Though visual narratives have been shown to have some potential for processing by individuals with autism, we have not actually observed the widespread assumption that “images” are easier to process than text.

Together with Annika Andersson (Linnaeus University) we’ve been examining similar processing in neurotypical kids and those with Developmental Language Disorder. We’ve also looked at visual narrative development with Mirella Manfredi (University of Zurich), who has recently begun exploring how kids start understanding of the sequencing of visual narratives.

Again, the book Who Understands Comics? explores a lot of the key ideas of this research. However, some important works related to these lines of research are: