New paper: What’s your neural function, narrative conjunction?
I’m excited to announce that my new paper “What’s your neural function, narrative conjunction: Grammar, meaning, and fluency in sequential image processing” is now out in the open access journal Cognitive Research: Principles and Implications. This study was co-authored by Marta Kutas, who was my advisor while I was a postdoctoral fellow at UC San Diego.
Simple take home message: The way you process the sequences in comics depends on which ones you read.
The more detailed version… This study is maybe the coolest brain study I’ve done. Here, we examine a particular pattern in the narrative grammar used in comics: Environmental-Conjunction. This is when you have characters in different panels at the same narrative state, but you infer that they belong to the same spatial environment.
Most approaches to comprehending sequential images focus on just the comprehension of meaning (like in “panel transitions”). However, my theory says that this pattern involves both the construction of meaning and the processing of this narrative pattern. The patterning uses only a narrative grammar which is independent of meaning.
When analyzing people’s brain responses to conjunction patterns, we found support for two processes. We found one brainwave associated with an “updating” of a mental model (the P600), and another associated with grammatical processing (an anterior negativity). Crucially, this grammatical processor was insensitive to manipulations of meaning, indicating that it was only processing the conjunction pattern. So, you don’t just process meaning, but also the narrative pattern.
But, that’s only the first part…
In other analyses, we’ve shown that Japanese manga use more Environmental-Conjunction than American or European comics. So, we used a statistical analysis to analyze whether participants’ background reading habits influenced their brain processing of conjunction. And… it did!
Specifically, we found that participants who more frequently read manga “while growing up” tended to rely more on the grammar processing, while infrequent manga readers used more updating. In other words, since frequent manga readers were exposed to the conjunction pattern more in their reading habits, their brains used a more automatic, grammatical process to comprehend it. Note: this result is especially cool, because our comics stimuli were not manga, they were manipulated Peanuts strips that used a pattern frequent in manga.
This result contradicts the idea that comics are uniformly understood by all people, or even the idea that their processing uses a single cognitive process (like “closure“). Rather, comics are understood based on people’s fluency with the patterns found in specific visual languages across the world.
You can read the paper online here, download the pdf here, or check out the poster summary.
Abstract:
Visual narratives sometimes depict successive images with different characters in the same physical space; corpus analysis has revealed that this occurs more often in Japanese manga than American comics. We used event related brain potentials to determine whether comprehension of “visual narrative conjunctions” invokes not only incremental mental updating as traditionally assumed, but also, as we propose, “grammatical” combinatoric processing. We thus crossed (non)/conjunction sequences with character (in)/congruity. Conjunctions elicited a larger anterior negativity (300-500ms) than non-conjunctions, regardless of congruity, implicating “grammatical” processes. Conjunction and incongruity both elicited larger P600s (500-700ms), indexing updating. Both conjunction effects were modulated by participants’ frequency of reading manga while growing up. Greater anterior negativity in frequent manga readers suggests more reliance on combinatoric processing; larger P600 effects in infrequent manga readers suggest more resources devoted to mental updating. As in language comprehension, it seems that processing conjunctions in visual narratives is not just mental updating but also partly grammatical, conditioned by comic readers’ experience with specific visual narrative structures.
Reference:
Cohn, Neil and Marta Kutas. 2017. “What’s your neural function, visual narrative conjunction? Grammar, meaning, and fluency in sequential image processing.” Cognitive Research: Principles and Implications. 2(27): 1-13
Comments
Interesting. I'll look into your work on sequential images and see if I can work some of it into my ideograms for English at neoideograms.wordpress.com. What I do is definitely visual language. I am aiming at efficiency of communication, and I suppose that the sequentiality of which you speak might have a place.