Here is another repost of a review I did awhile ago (1/22/09).
This study examines the neurocognitive processes involved with comprehending a series of pictures, like in comics. The experimenters pulled frames from an animated movie to create static picture sequences. There were two possible endings for each sequence: one with a normal ending, and one with a non-sequitur panel that did not make sense.
Comparison of these sequences used a technique called “event-related potentials” (ERP) that examines people’s brainwaves with an EEG recording. The electrical field is measured off the top of the scalp through an electrode cap (like in hospitals), and by averaging out the noise at the critical point (the “event” — here the last panel) it can give you a nice smooth waveform that can tell you about the nature of the cognitive process. Unlike fMRI, ERPs don’t tell you much about “where” in the brain things happen, but they do tell you a lot about “when” and a little about the nature of the process.
In this case, your brain distinguishes the difference in processing at less than half a second. The result was a “negative” deflection of the waveform roughly 400 milliseconds after the final panel appeared on the screen (panels appeared one-by-one). These waveforms are from the frontal right part of the head:
The BLUE line represents the normal sequence ending, the RED line the non-sequitur ending. Note that the lines separate and there is a bump labeled “N400” that shows the processing difference (negative is up here). Because of the separation, we can tell that the brain is working harder to process the non-sequitur panel. If it was treated the same, the lines would stay together, like at the beginning of the waveforms.
This N400 also appears in language under similar conditions: where the brain is working harder to integrate semantic information into a meaning, though with language it appears in different locations on the scalp (more back of the head than front). In fact, the first paper that found an N400 for language used this same manipulation: comparing normal and incongruous words at the end of a sentence.
Unfortunately, more experiments of this sort have not really been done with sequential images. Fortunately, it’s only a matter of months until I do more. Phil Holcomb, one of the authors, is also one of my advisors. My upcoming projects will be doing these types of brainwave studies using more targeted manipulations of the visual grammar.
EDIT (8/4/11): I have now done a study examining visual narrative structure and am soon going to do several more!
West, W., & Holcomb, P. (2002). Event-related potentials during discourse-level semantic integration of complex pictures Cognitive Brain Research, 13 (3), 363-375 DOI: 10.1016/S0926-6410(01)00129-X