Meet the Lab

Along with Neil Cohn (PI), The Visual Language Lab consists of an international and neurodiverse mix of scholars from all levels of scholarship, both at Tilburg University along with external graduate students and postdocs. We also have many collaborators around the world.

Affiliated faculty

Joost Schilperoord Ph.D. has been a professor at Tilburg University since 1999. He studied Communication sciences and Linguistics at Utrecht University, where he wrote a dissertation on the psycholinguistics of written text production. His interests gradually shifted from psycholinguistics to visual and multimodal language and communication. He published several papers and gives bachelor and master courses about the rhetorics and cognition of visual genres like advertising and editorial cartoons, i.e. metaphorical conceptualization in visual expressions, visual incongruities, irony and hyperboles, and (visual) optimal innovation. Over the past several years, his main preoccupation has been multimodal language. Together with Neil Cohn he has been developing a cognitive linguistic model of multimodality which addresses questions concerning the substance, grammar and meaning of multimodal expressions: what is the structure of such messages?; how do these structures interact?; and how do the separate contributions of each modality interface to produce an expression’s meaning and what kinds of implications does all this have for the (multimodal) lexicon and (multimodal) constructions?; i.e. what is the role of productivity, multimodal compositionality and storage of elements in the lexicon? 

Joost Schilperoord

Postdoctoral fellows

Ana Krajinović, Ph.D. is a linguist with an interdisciplinary mindset and passion for comics and visual storytelling. Her curiosity about language diversity took her from describing the grammar of the Malabar Indo-Portuguese creole (MA thesis and book) to fieldwork in Vanuatu. Based on her fieldwork on Nafsan, a Southern Oceanic language, Ana researched the typology of linguistic categories of perfect aspect, as in present/past perfect in English, and irrealis mood, similar to the subjunctive. Ana holds an MA from the University of Lisbon and a joint PhD degree from the Humboldt University of Berlin and the University of Melbourne. She worked as a postdoc at the Heinrich Heine University of Düsseldorf since 2020, where she has expanded her ideas about language to multimodal communication with internet memes. She studies how the meanings of viral exploitable memes evolve to become more abstract, parallel to the processes of grammaticalization in language. In her free time, Ana writes and draws (and learns how to draw) comics about neurodivergence and other topics. (Website)

Graduate students

Irmak Hacımusaoğlu M.Sc. is a Ph.D. student at Tilburg University and a doodle artist. She holds a MSc degree in Cognitive Neuropsychology (Research) from the Vrije Universiteit Amsterdam (2019). Her master’s research combined behavioral and electrophysiological (EEG) approaches. She worked with Prof. Dr. Martijn Meeter in her thesis about false memory generations in educational settings. Before that, she received her BA in Psychology (magna cum laude) with a specialization in Brain & Cognitive Sciences from Koç University, 2017. Her primary research interests include cultural memories, cognitive aspects of visual narratives (e.g., mimicry and the visual language), event cognition, and the cognition of drawing. She is fascinated by the intersect between neuroscience and humanities and has become a member of NeuroGenderings Network. (Twitter)

Bien Klomberg M.Sc. is a Ph.D. student at Tilburg University, excited to combine her lifelong love for stories and linguistics with an equally long passion for drawing and comics. During her bachelor at University College Roosevelt, Middelburg, she studied stylistics, rhetorics, literature, and cognitive linguistics, with figurative language and point of view as special interests, leading her to co-author the book Picturing Fiction through Embodied Cognition (Routledge, 2022). Her masters in Communication and Cognition at Tilburg University focused on inferencing techniques in visual narratives. Her PhD is focusing on (dis)continuity in visual narrative sequencing, particularly in mappings across domains. She enjoys mixing craft with science, still draws, and loves stories in written form as well as Netflix series. (Website)

Lenneke Lichtenberg is a Research Master student studying Linguistics and Communication Sciences at Tilburg University. She is interested in the processing of information across different modalities in neurocognition. During her bachelor Communication and Information Sciences, she specialized in Cognition and Communication. Her honors bachelor’s thesis focused on the processing of visual morphology of upfixes, focusing on their literal and symbolic/metaphoric meanings. She plans to learn more about EEG research and conduct EEG experiments in the future. After her Research Master’s, she hopes to pursue a PhD in (Neuro)Cognition and Communication sciences. (Twitter)

Researchers

Sharitha van der Gouw M.Sc. has served as the “lab manager” for the TINTIN Project, and has an interest in visual storytelling and illustration, she is excited to also incorporate her long-lasting passion for comics into research. During her bachelor’s in Japanese Studies, she focused on anthropology and sociology of law, exploring the socio-legal issues surrounding the manga industry in Japan. Furthermore, her bachelor in Communication and Multimedia Design focused on visual storytelling and interaction design, additionally exploring how we interact with technology. She did a master’s in New Media Design at Tilburg University centered around user experience design and (interactive) storytelling, with her thesis focussing on analyzing domain usage in American and Japanese comic styles. She hopes to further research characters or subjectivity in comics, for example by pursuing a Ph.D. in the field.

Sharitha van der Gouw

Affiliated graduate students and postdocs

Aditya Upadhyayula, Ph.D. (University of California, Davis) is a recently graduated PhD formerly working with Jon Flombaum on how we represent, process, and thereby group moments in time. Prior to pursuing a Ph.D. in Cognitive Psychology, he was trained as an Electrical and Computer Engineer, and a Physicist. He loves walking, cooking, and playing the guitar and the piano. He once made traditional South Indian potato fries with Italian seasoning (Although ironic, he maintains that it was delicious). He is often seeing going on long walks wondering how he has ended up pursuing cognitive psychology. Nevertheless, he is happy to be studying how the mind works. He is very interested in building and using computational models to understand human cognition. He recently began a postdoc working with Jeff Zacks at Washington University in St. Louis. (Website, Twitter)

Marianna Pagkratidou, Ph.D. (TU Dublin, Ireland) graduated with a Ph.D. in Cognitive and Experimental Psychology from the University of Cyprus. Marianna’s research focuses on spatial memory representations that are constructed through narratives, specifically through comics. She uses computerized behavioural tasks and eye tracking techniques to study how people foreground spatial situation models when reading comics under different circumstances. Currently, Marianna is a Postdoctoral Researcher at the School of Electrical and Electronic Engineering at TU Dublin. She coordinates the SellSTEM (Spatially Enhanced Learning Linked to STEM) Marie Skłodowska-Curie Actions project and examines the relationship of spatial abilities in and for STEM learning. (Website, Twitter)

Marianna Pagkratidou

Morgan Patrick (Northwestern University, USA) is a Ph.D. student in music theory and cognition. His collaboration with Neil focuses around enumerating the interface between musical form, broadly construed, and aspects of narrative grammar. This work takes both theoretical and empirical directions, involving three areas of inquiry: affective dynamics, structural parallels, and attentional processes involved the analysis and cognition of temporal structures. He is also engaged in research on film music analysis and musical theme learning, having most recently presented work on leitmotifs at Music and the Moving Image. (Website coming soon)

Maki Miyamoto (Japan Advanced Institute of Science and Technology Japan) is a Ph.D student in Knowledge Science at JAIST. She is a visiting PhD student at Visual Language Lab in Tilburg University from September 2023 to August 2024. She is interested in communication by ‘language’ and her original interest comes from ideophones or onomatopoeia in comic/manga. She has two simple questions: (1) Can people understand or feel sounds or states of things by seeing a scene of comic/manga even if they do not have knowlege of the language used in the comic/manga? (2) Do people who are sharing the same language actually understand or feel same sounds or states of things when they see the same scene of comic/manga? If they do, why they can do? Her master’s research focused on these questions, and she has investigated communicability of Japanese ideophone sounds to non-Japanese speakers from the view of sound symbolism. In her Ph.D research, she aim to understand what people can know through ideophones in comic/manga.

Lab alumni

Bruno Cardoso Ph.D. was a Postdoctoral Researcher (2020-2023) at Tilburg University and the creator of the Multimodal Annotation Software Tool (MAST) for the TINTIN project. His research interests are focused on the field of Human-Computer Interaction, specifically topics of emotion, usability, context-awareness in mobile devices and domain-specific languages. His research has been presented in venues like the UIST, CHI, IUI, UbiComp and CIKM. Bruno received his BSc and MSc in Computer Science at the University of Évora (Évora, Portugal). Following years as a consultant and freelance, he received a Ph.D. in Computer Science at the Nova University of Lisbon (Lisbon, Portugal). Close to the end of his studies, Bruno spent a sunny Winter as a research intern at Telefonica R&D (Barcelona, Spain). After obtaining his Ph.D., Bruno spent 3 years as a Postdoctoral Research Fellow at KU Leuven (Leuven, Belgium). A pretty eclectic person, Bruno loves learning new things, science, technology (emphasis on all things computer) and culture. (Website)

Bruno Cardoso

Fernando Casanova (University of Murcia, Spain) graduated with his PhD sponsored by a FPU scholarship granted by the Spanish Ministry of Education and Culture. He holds two degrees in Language Sciences (Spanish, English and French) and an MA in Theoretical and Applied Linguistics. His doctoral thesis focuses on the study of interjections and onomatopoeias, their emotions, and the co-speech body and facial gestures they incorporate. His aims are focused on observing intercultural differences from an emotional, cognitive, psychological, paralinguistic and multimodal perspective. From that multimodal level, he is analyzing the representation of human beings through drawings (comics) and physically with real people (NewsScape Television News Library, developed by the Red Hen Lab). He is fascinated by the psychological and cognitive connection of emotions and gestures to express all kinds of information and that allows us to communicate in an efficient way.

Ian Joo (HK PolyU, Hong Kong) is a lecturer the Nagoya University of Commerce & Business. His two main research interests are iconicity (How can linguistic form resemble linguistic meaning?) and areality (How do different geographical areas show different linguistic patterns?). His Master’s Thesis, now published in Linguistic Typology investigated the iconic relationship between the forms and the meanings of basic wordlists of 66 languages (such as ’to blow’ and labial sounds). His doctoral project involves building a database called Phonotacticon, consisting of basic phonotactic information of hundreds of Eurasian languages, in order to measure the phonological distance between them and thereby detect areal patterns across Eurasia. He was born in South Korea but has spent about half of his life abroad in many different places: Canada, France, Taiwan, Hong Kong, Germany, Sweden, and soon in Netherlands. (Website, Twitter)