I’m excited to say that my collaborator Tom Foulsham and I have a new paper out, “Meaning above (and in) the head: Combinatorial visual morphology from comics and emoji” which appears in the journal Memory and Cognition.
This paper examines the (neuro)cognition of “upfixes” which is what we call the elements that float above characters’ heads to express aspects of emotion or cognition. I call them upfixes because they are visual affixes that are up above a head.
In prior work, we showed that people recognize that upfixes have a variety of constraints, such as that upfixes need to be placed above a head (not next to it), and that the face needs to match with the upfix. These constraints apply both the conventional upfixes that people are familiar with, and those that might be novel, which shows upfixes in general use an abstract pattern.
In this paper, we further explored these constraints by measuring people’s reaction times to upfixes that either matched or mismatched their faces, and then we measured people’s brainwaves to the same stimuli. For the reaction time study, it was actually the first (of now many!) studies that we ran online. So, if you took the experiment, thanks!
Overall, we found again that matches were harder to comprehend than mismatches, but that conventionality mattered. The more conventional the matching upfix, the faster people responded to say it made sense. This advantage of conventionality didn’t appear for mismatching upfixes though, showing that people don’t store them in their mental visual vocabulary.
Similar results came in the brain responses. We showed that mismatching and conventionality both impacted different brain mechanisms, particularly brainwaves associated with the processing of meaning and pattern recognition. All of this showed that upfixes use an abstract template which can inform the creation of new upfixes.
Here’s the official abstract of the paper:
“Compositionality is a primary feature of language, but graphics can also create combinatorial meaning, like with items above faces (e.g., lightbulbs to mean inspiration). We posit that these “upfixes” (i.e., upwards affixes) involve a productive schema enabling both stored and novel face–upfix dyads. In two experiments, participants viewed either conventional (e.g., lightbulb) or unconventional (e.g., clover-leaves) upfixes with faces which either matched (e.g., lightbulb/smile) or mismatched (e.g., lightbulb/frown). In Experiment 1, matching dyads sponsored higher comprehensibility ratings and faster response times, modulated by conventionality. In Experiment 2, event-related brain potentials (ERPs) revealed conventional upfixes, regardless of matching, evoked larger N250s, indicating perceptual expertise, but mismatching and unconventional dyads elicited larger semantic processing costs (N400) than conventional-matching dyads. Yet mismatches evoked a late negativity, suggesting congruent novel dyads remained construable compared with violations. These results support that combinatorial graphics involve a constrained productive schema, similar to the lexicon of language.”
Cohn, N., Foulsham, T. (2022). Meaning above (and in) the head: Combinatorial visual morphology from comics and emoji. Memory and Cognition. https://doi.org/10.3758/s13421-022-01294-2
Finally, here’s a plug: if you like upfixes, we have both conventional and unconventional upfixes on shirts for sale!
[…] in such a way where it is floating above a character head Neil Cohn describes in his paper, “Meaning above (and in) the head“, this type of placement which he has dubbed upfixes. These are essentially floating […]