Here is another repost of a review I did awhile ago (1/22/09).
This study examines the neurocognitive processes involved with comprehending a series of pictures, like in comics. The experimenters pulled frames from an animated movie to create static picture sequences. There were two possible endings for each sequence: one with a normal ending, and one with a non-sequitur panel that did not make sense.
Comparison of these sequences used a technique called “event-related potentials” (ERP) that examines people’s brainwaves with an EEG recording. The electrical field is measured off the top of the scalp through an electrode cap (like in hospitals), and by averaging out the noise at the critical point (the “event” — here the last panel) it can give you a nice smooth waveform that can tell you about the nature of the cognitive process. Unlike fMRI, ERPs don’t tell you much about “where” in the brain things happen, but they do tell you a lot about “when” and a little about the nature of the process.
In this case, your brain distinguishes the difference in processing at less than half a second. The result was a “negative” deflection of the waveform roughly 400 milliseconds after the final panel appeared on the screen (panels appeared one-by-one). These waveforms are from the frontal right part of the head:
The BLUE line represents the normal sequence ending, the RED line the non-sequitur ending. Note that the lines separate and there is a bump labeled “N400” that shows the processing difference (negative is up here). Because of the separation, we can tell that the brain is working harder to process the non-sequitur panel. If it was treated the same, the lines would stay together, like at the beginning of the waveforms.
This N400 also appears in language under similar conditions: where the brain is working harder to integrate semantic information into a meaning, though with language it appears in different locations on the scalp (more back of the head than front). In fact, the first paper that found an N400 for language used this same manipulation: comparing normal and incongruous words at the end of a sentence.
Unfortunately, more experiments of this sort have not really been done with sequential images. Fortunately, it’s only a matter of months until I do more. Phil Holcomb, one of the authors, is also one of my advisors. My upcoming projects will be doing these types of brainwave studies using more targeted manipulations of the visual grammar.
EDIT (8/4/11): I have now done a study examining visual narrative structure and am soon going to do several more!
West, W., & Holcomb, P. (2002). Event-related potentials during discourse-level semantic integration of complex pictures Cognitive Brain Research, 13 (3), 363-375 DOI: 10.1016/S0926-6410(01)00129-X
Comments
The N400, is a nice find, but expected. A lot of work with N400’s and line drawings has been done. What would be interesting to see is if a P600 could somehow be elicited. Patel, Gibson, Ratner, Besson, and Holcomb did an ERP study that found some pretty interesting evidence that P600s could be elicited by music that was structurally dissonant. I think their model might be adapted to the modality of sequential art. I’m thinking specifically of your work with Peanuts comics and those stars. So, I guess what I’m trying to say is, can I reference/borrow a bit of your work for an ERP study? It’s not going to be publishable and I promise to give you credit for the idea.
Almost forgot, the name of the study was Processing Syntactic Relations in Language and Music: An Event-Related Potential Study.
Right, well, they essentially imitated the classic Kutas and Hillyard study, just with images. I think that there's quite a few ways you could get a P600, for instance with the types of things done with videos in "Sitnikova, Holcomb, & Kuperberg. 2008. Neurocognitive Mechanisms of Human Comprehension" with manipulation of event structure.
I'm actually in the midst of doing a study with the Peanuts Action Stars. I'm finishing up a behavioral Self Paced Reading study that right now is trending towards significance and will follow it up with an ERP version.
However, I would be fine if you used parts of my work for an ERP study, though I'd want to hear about the results. The more work on this stuff the better!