Emoji research and creation

Emoji have become a fundamental way that people communicate digitally, particularly along with their written language. They also often draw on the visual conventions already established in comics and manga around the world. We have both carried out psycholinguistic research on emoji and how they function, and have used our research to contribute to the creation of new emoji.

Emoji research

We have carried out research on the psychology and linguistics of emoji, particularly in ongoing collaboration with Ben Weissman (Rensselaer Polytechnic Institute). This work has focused on how emoji convey meaning, and how they interact in multimodal interactions with written language. Our work has shown that people incorporate acquire the meanings of emoji with relative strengths. If emoji substitute for words in sentences, they are readily integrated into the grammar. However, emoji do not seem to be combined into sequences using a grammar of their own, and the way emoji combine into linear sequences is somewhat unnatural for graphic communication. Here’s some links to our research:

  • Weissman, Benjamin, Neil Cohn, and Darren Tanner. The electrophysiology of lexical prediction of emoji and text. Neuropsychologia. 198 (108881):1-11 (Journal)
  • Weissman, Ben, Jan Engelen, Elise Baas and Neil Cohn (2023), The Lexicon of Emoji? Conventionality Modulates Processing of Emoji. Cognitive Science, 47: e13275. (Journal)
  • Cohn, Neil, Jan Engelen, and Joost Schilperoord. 2019. The grammar of emoji? Constraints on communicative pictorial sequencing. Cognitive Research: Principles and Implications. 4(1):33. (Online version & PDF)
  • Cohn, Neil, Tim Roijackers, Robin Schaap, and Jan Engelen. 2018. Are emoji a poor substitute for words? Sentence processing with emoji substitutions. In T.T. Rogers, M. Rau, X. Zhu, & C. W. Kalish (Eds.), Proceedings of the 40th Annual Conference of the Cognitive Science Society. (pp. 1524-1529) Austin, TX: Cognitive Science Society. (PDFPoster version)

Additional work is connected to emoji, such as the “upfixes” that float above characters heads like hearts or lightbulbs. This work has focused on the constraints on upfix structure, or how combining elements creates non-literal meanings:

  • Cohn, Neil, Beena Murthy, and Tom Foulsham. 2016. Meaning above the head: Combinatorial constraints on the visual vocabulary of comics.  Journal of Cognitive Psychology. 28(5): 559-574. (PDF

Emoji creation

Along with Jennifer Daniel at Google, I have helped propose several emoji that have become implemented in keyboards. Because many face emoji originally used the symbols from manga, we drew from the extensive research on comics and manga to help create new emoji. These include the breath face 😮‍💨, melting face 🫠, dotted line 🫥, and holding back tears emoji 🥹. Other emoji are still under review, and we are actively working on more emoji proposals. 

Here’s some media about our emoji:

The Melting Face Emoji Has Already Won Us Over. New York Times. By Anna P. Kambhampaty, September 29, 2021. 

Smeltend gezicht en wolkje voor de mond. by Stan Schrijen. Brabants Dagblad. October 26, 2021.

Neil onderzoekt én verzint nieuwe emoji’s (video). AD news website. October 27, 2021.

Dít is de ontwerper van jouw emoji’s (video). Khalid en Sophie on NPO1 television. October 26, 2021.

Face emoji