The last podcast I did with the VizThink folks was so fun I decided to do another. This one is about the various functional roles that panels play in the visual language used in comics. Among the topics I hit are:
• focusing information within panels
• navigating page layouts
• visual “storytelling”
• text-image relationships
It’s a slightly pared down and also expanded (at the same time!) version of the talk I gave at the VizThink conference. Enjoy!
Comments
Great talk as always Neil, I have a couple of comments/critiques towards your crit of the Vizthink artwork however.
I have to say it seems a bit nitpicky as I think it’s clear (and you suss out pretty early on) that the original intent is meant to be glossing over of the possible information to be had at the conference. It’s meant to have the “where’s Waldo” effect and as agreed, is succesful (maybe a word from the artist on that would help, however…) That just how I see it’s intent, so I assume we’re only critiquing it for an an example of an intent it didn’t necessarily have.
But if it’s true, and the intent was more linearly based, I’d have to argue that it does indeed look busy, but the linear story is held mainly in the text (shown here as signs, of which there are about 14 or 15) I’d call these the “Active elements” of the metapanel here. I know you argue later that text built into an image is an “inherit text/image interface” and there may be room to continue that thought, but this kind of “idea map” present in the vizthink poster seems such a departure from a “narrative” (ie. comic or story) and more like a flow chart of concepts or bullet points, not unlike a PPT, etc…
The people and various visual activities represented don’t give much real information, and as the composition is open and evenly filled, gives the impression of being “inactive elements” and per your definition. I argue this because as you read the map, you gloss over most if not all of the visual representations, in favor of jumping from sign to sign. The visual elements are more or less clip-art of business activities, scene setting information and therefore are more like “Inactive elements” of a panel. (In this case, one meta-panel) So if the “active elements” of the narrative are the graphic signs of text, I find them to be well paced, and not too busy at all. It’s easy to gloss over and find things you have an interest in, all while being ushered about by the inactive elements towards the greater idea that this is an entertaining and informative conference.
I also have to argue that your Gasoline Alley example, while is indeed incredibly interesting, is in reality just as confusing visually. If you watch the active elements of the characters as they move through the z-shaped rhythm of panels you find them moving in completely erratic ways sometimes in the hole, sometimes out, seemingly out of time and coherence (I can’t read the text at this size, so it could be that it accounts for their behavior.) but visually a far cry from being a much better example of more linear visual storytelling.
As always, I appreciate the engaging conversations and concepts! I find the Text/image Intereface breakdowns to be especially interesting. Especially “Bundling” and “independent” and “Disjunct” interfaces as it could relate to hypercomics.
–Neal
Hi Neil,
Thanks for initiating this discussion. Visual language is a fascinating topic with a lot of open territory. I’m so glad to have a chance to hash some of these issues out with people who have passion for, and interest in, the subject.
I’d like to weigh in here, since I was one of the people involved in the design of the VizThink image you use as an example here.
I can confirm that our primary goal was to give people a sense of what their experience might be like if they chose to attend the conference. Since this was the first VizThink conference, there was no way to show photographs of past conferences, so the next best thing we could think of was to imagine and attempt to visualize the event in advance.
We felt that a main part of the value of VizThink would be the melting pot that would occur due to the mashing up of various visual thinking disciplines, and part of our intent was to try to give people a sense of the circus-like atmosphere which would result.
Those of us who attended the conference would probably confirm that the experience was much more non-linear (even chaotic!) than a structured sequence of panels might have indicated.
I would characterize the image as an information landscape, which organizes itself spatially rather than sequentially, like this image from the Japanese Edo period. It was designed to invite wandering and invoke a feeling of discovery.
As you point out, releasing information from a linear flow gives control to the audience with unpredictable results.
I agree that the ability to direct an audience’s attention has benefits, but it also carries costs. Directing the audience’s attention is not always the point.
One of the great advantages visual language has over written language is precisely its capacity to present information in a non-linear way, such as you might find in a map, for example.
While comics and graphic novels are conventionally read in a “Z” pattern, this pattern seems to be derived from patterns in verbal language and storytelling, which is linear in nature. In fact the Z pattern is precisely the pattern you use when you read a book written in English. But visual language is capable of so much more than that!
This may be a controversial point in some circles, but I don’t believe sequentiality needs to be a prerequisite in order to call something a language. Such a definition would exclude things like Venn diagrams or international symbols like the ones we see in airports, which communicate concepts visually, and very successfully, but do not require a sequential reading.
Other visual language forms, like information landscapes, maps, bar charts and tables require different reading conventions and invite exploration along multiple dimensions.
I think of the “cartoon map” of Disney World that I had on my wall as a child — I loved that map because it invited me to imagine future experiences I might have, and gave me the freedom to choose different paths or sequences that I might take.
That map could certainly have been divided into panels and my attention could have been directed through it so as not to miss anything, but only at the price of diminishing the joy of my exploration and discovery.
This is a rich topic and I am very interested to hear other opinions on this thread.
Thanks for your lengthy comments Neal and Dave!
As I perhaps didn’t make clear enough in the talk, I didn’t wish to diminish that the image succeeded in its intents. It did. However, I wanted to use it as an example for how broader phenomena like that can be deficient as communication tools.
Neal, I think your assessment of the image is pretty spot on. If you noticed in the talk that I did not try to say that the VizThink image should use a linear narrative, or that it was one (like the Gasoline Alley strip… which I found via Derik, btw). Rather, no narrative needed to be added and Divisional panels simply could have given the reader a navigational reading path. And, as I mention in my talk, that navigational path is not the z-path necessarily.
Dave, I would certainly be one of those people who says that sequentiality is a requisite component of language (and more than just sequence, but full on grammar). This was actually what the whole first part of my talk at VizThink was devoted to explaining (Functions of Panels being the second part).
My notion of “visual language” as a phenomenon does not include diagrams or open graphic displays of the type you mention. To me, those are no more language than a person randomly spewing out various related words.
I disagree with Bob Horn’s (and anybody’s) use of the term “visual language” to mean any type of graphic communication that may or may not be paired with words. These things are certainly “visual communication,” but “language” they are not.
Bob’s use of the term “language” is metaphorical — I mean it literally. I don’t limit this out of bias for what people conventionally call “comics”, but because of looking at the components that all real, natural languages have.
Neil,
What are the components that all natural languages have?
How would you define grammar? A set of rules governing usage?
Are you sure that grammar requires sequence?
Is a grocery list “language?”
Dave, glad to see you return! Boiling down my talk to a list, all natural languages have…
Structural
1. Modality
2. Concepts
3. Grammar
4. Infinite creativity on all levels of structure (“combinatoriality”)
Social
5. Communicative function (at base)
6. A community of “speakers”
7. Diverse structures based on community of speakers (no universality)
8. Create an identity for speakers
Psychological
9. Unconscious processing
10. Potential relativity (external structure contributes to internal structure)
Developmental
11. Innate capacity meshes with external structure
12. Critical learning period of development
A grammar (in the limited sense of “syntax”) is a set of cognitive rules and constraints that specify acceptable from unacceptable productions. Grammars require sequence in order to create distributional categories of “formatives” (i.e. nouns and verbs).
The semantic portion of the broader sense of grammar (as an organization of the Structural aspects above) does not require sequence. Syntax functions to give order to that unordered meaning.
A grocery list is an instantiation of language that lacks grammar. The system it comes from may have grammar, but it does not. As a “portion” of that system, YES it is language.
BUT: if all I know of Korean is names of items in a grocery store, and I do not have the grammar to string it together, in that instance NO, it is not language (for me).
Does that brief explanation make sense?
I should also add a note about text-image interactions…
Images + text do not equal language (as Bob Horn would say), they do equal multimodality. Maybe as equations:
“Verbal/written Language + Visual language/communication ≠ Language”
“Verbal/written Language + Visual language/communication = Multimodal communication of concepts”
…which is comparable to:
“Verbal Language + Gesture ≠ Language”
“Verbal Language + Gesture = Multimodal communication of concepts”
Surely the key element in any definition of language must be the transfer of meaning!
I think that visual language as I define it can meet all of your criteria except for sequence, and I do think that your requirement for sequence, while essential for verbal languages, is not a prerequisite for visual language.
All written language is, by definition, visual. Even though the written alphabet represents sounds, it employs visual signs and structures to do so.
In fact most people seem to agree that these signs originated as pictures, i.e., that they originally represented physical things and over time that meaning was transferred to sounds.
Of course, a spoken language is by nature sequential, so a grammar that will be useful for representing a spoken language will require sequence as a key element of the grammar.
But as we see in such things as grocery lists, tables of contents, indices, road signs and many other written conventions, sequence is not a requirement for written language to be intelligible and useful as a medium for communication.
I am making two arguments here:
1. That it’s possible to transfer meaning in any language without using sequence.
If you can read a menu in Korean then you may not be fluent in Korean, however you might be said to have “survival skills” or a basic familiarity with the language. You can convey meaning by pointing to your mouth and saying the name of the food you want, for example.
If you watch people order coffee at Starbucks, they don’t need sentences — the exchange often looks like this:
“Double cappucino”
“Three forty-five”
“Thanks!”
No grammar or sentence required for that exchange.
Admittedly this kind of communication is primitive but it’s still language, although that’s not my main point.
2. The second argument, and my main point, is that you can have a structured grammar that is not sequential in nature.
In your visual definition of visual language, you employ non-linear grammar to make your point. You use a matrix-like structure that compares the three types of language horizontally while constructing a logical argument vertically.
Further down you use a starlike structure to describe six categories, or aspects, without implying hierarchy or requiring a sequential reading.
This is something that would simply not be possible with spoken language and yet it is an absolutely essential element of visual language.
One of the key advantages of visual language is its potential to convey complex or non-linear relationships.
Why would you want to exclude non-sequential grammatical structures from visual language? It would seem to ignore some aspects of the visual medium that offer true distinctiveness and advantage over spoken languages.
I do agree that conveying meaning is essential for language, but it is not the essential component. Sequence (governed by grammatical rules) is necessary for language.
You’re right, visual communication may not have a grammatical sequence (and a few others I’d argue), but that is exactly what makes it not language, just like gestures are not language, but sign language is. Also, we don’t say that sequential pantomime is Language — it requires a specific type of sequence. The difference is grammar, and I know of no grammar in the linguistic sense that does not have sequence.
“Writing” is essentially the transfer of verbal structure into the graphic domain and is “non-native” in this respect. It did not necessarily follow from pictures — and for this I reference you to my thesis “¡Eye [heart] græfIk Semiosis!”
Again, your examples of ordering off menus and grocery lists are not about a system of language, but about the usage of that system. I’m identifying the system as a whole. The usage of that system can vary wildly…
For instance, my visual description of visual language is also more an example of multimodal communication of meshing the VL with text. There, I’d say that the visual grammar may not even come through at all. In that case, the text is guiding the meaning (see “Interactions and Interfaces”).
I do not make these limitations out of bias for “comics.” I do them out of knowledge about natural languages, from my understandings of studying linguistics and psychology. While my views may expand the notions of language a tad, they don’t change anything as it previous stands.
It would be easy for me not to be a “wet blanket” and join the Big Tent viewpoint. It would probably win over a lot more people to my ideas. But, I think it would be dishonest and inaccurate to the reality of the argument and the evidence.
Metaphoric reference to “language” is fine, but it’s important to know when it’s metaphor.
Well Neil, it’s your blog so you get to decide what is or is not language in this context.
It still sounds to me that your primary definition of language is that is is first verbal, then becomes visual when the concepts are expressed in writing.
I am not a linguist but I do use language every day, so I can consider myself an expert in the “user” sense.
When you separate the concept of a system from its usage you seem to be arguing against one of your primary definitions; that language is emergent. I understood that to mean that the usage comes first, and that the system is defined by observing and defining the rules as they are used. So this is a circular argument.
A technical description of language as a system that does not include such things as lists — which are clearly part of the system, and are used on a daily basis by almost everyone who can read and write — would seem to be incomplete.
I am not saying that a list is a language. But a list is one of the structures that is a clear component of the language system, when it is expressed visually at least.
A list, for example, is far more prevalent and commonly used than the comic panel.
My understanding is that the first known writing system, cuneiform, initially was developed as a system for tracking inventory; that is, a list-based system, and in fact NOT a speech-based system.
Some argue that cuneiform, which many consider the first writing system, was initially developed not to represent speech at all but to represent literal objects and keep track of how many “its” there were.
I use “list” as an example because it is one of the most basic systems for organizing and communicating information.
In a quick search for definitions of language, I don’t see broad agreement that sequence is a requirement. The first definition that comes up:
“A systematic means of communicating by the use of sounds or conventional symbols”
would seem to leave room for the possibility of a visual language with a non-linear grammar.
Writing would not seem to be a necessary component of language. For example, people can use language who can’t read and write.
However, if writing is to be considered as a component of the overall language “system,” then the aspects of writing that offer greater possibilities and opportunities must also be considered.
Hierarchical methods of organization such as lists, and location-based methods such as maps, are part of the writing system, aren’t they?
This leads to a couple of fundamental questions:
1. If part of the definition of language is that the concept MUST be expressible verbally, then why are we talking about visual language at all?
2. If the definition of language does not REQUIRE a verbal equivalent to every written expression, AND if a language must be considered as a complete system, then any additional properties that come about through the process of visualization MUST be considered — even if that might expand or change our current conception of language.
One final question:
IS there a clear, specific definition of language that is broadly agreed to by most, if not all, linguists?
And more specifically, is there a definition that would accommodate such a concept as a visual language that has a grammar and syntax that is visual in nature?
I haven’t had time to read your thesis yet but I will read it with interest for additional insights.
I also look forward to hearing your additional thoughts.
I’m afraid I’m really not sure what you mean by this last thread. I absolutely do not believe that language is primarily verbal or must pass through a verbal filter. Much of my work is devoted to showing that there is “a visual language that has a grammar and syntax that is visual in nature”.
At this point, this thread doesn’t really have much to do with the podcast, and has deteriorated into general concerns about the theory of visual language and a pretty standard set of misunderstandings about it.
So, I refer you a few places that can give a bigger picture, so that I don’t have to fight the smaller issues which are symptomatic of the bigger ones.
I recommend anything on the “Getting Started” page, especially the other podcasts/video that are posted there.
I look forward to continuing the discussion.
Sorry to hear you think the conversation is deteriorating Neil.
I don’t think I am misunderstanding visual language. I think I understand your view of it pretty well. But I do disagree with you on a fundamental point.
We agree that sequence is a requirement for verbal language construction, and that a verbal grammar requires it.
You believe this requirement extends to visual language and I disagree. There are many methods of construction that are non-sequential in nature.
You may not believe you are subject to a verbal filter, but in my view your sequence requirement is an unnecessary limitation, imposed by a concept of language construction that is verbal in nature.
I see sequence as an element of a visual grammar but only one of many elements. I disagree with your claim that sequence is a requirement for a visual grammar.
My point is that any grammar and syntax that claims to be visual in nature must account for and include non-sequential construction, or else it is incomplete.
I thought it was deteriorating from the nature of the podcast, yes, but not from substance.
I can understand how you might think a grammar can be non-sequential, but I think it is a misunderstanding of how a grammar works and functions.
Grammar isn’t just about “construction” of visual arrays. Grammars function to limit and constrain a representation by use of systematic patterns of grammatical categories. Not all sequences are grammatical — only those that follow the unconscious rules of the system.
This isn’t just the case in verbal form. Sign language requires this sequence too, but also uses spatial information as well. Sign language sometimes features concurrent signs and does encode grammatical information spatially, but the discrete sequence is still fundamentally important in it.
I know that there are grammatical rules in sequences because I can create examples that violate them in specific ways. No one invented these rules, they just happen as a result of cognitive processing.
If you can show me examples of (unconscious) explicit rules that are violated by non-linear graphics that are guided by categories in distributional patterns, I then might begin to believe that there is a non-sequential visual grammar.
Otherwise, what you’re talking about isn’t “grammar” in any actual sense. And that’s okay — but it’s also not language.
Just because the visual form may have a capacity for non-linear information does not mean that it has a “destiny” to be that way. Why should the graphic form be excepted in the way its grammar works from all other types of languages (and grammars)?
This is very helpful Neil.
It’s true the conversation is evolving — maybe this thread deserves a new post.
I think the thread is valuable though, because visual language, like any language, is an evolving thing, still in the early stages of being defined.
If there is such a thing as a visual grammar — or, perhaps, if there are multiple visual languages with multiple grammars — at some point it will be important that people come to agreement on what those things are.
An understanding of linguistics as it is traditionally defined seems both very important and very helpful as we try to grasp what visual language is and is not.
I appreciate your patience as I try to understand some of these linguistic concepts.
In answer to your final question, a visual grammar might be excepted from rules that apply to other grammars because it has the advantage of additional dimensions that a verbal language doesn’t have, such as
— Spatial two-dimensional layout
— A broader vocabulary which includes non-alphabetic symbols and other images.
The challenge you present here
“If you can show me examples of (unconscious) explicit rules that are violated by non-linear graphics that are guided by categories in distributional patterns, I then might begin to believe that there is a non-sequential visual grammar.”
is excellent because it lays out a specific test that could serve as a litmus test. I’m not sure I understand it though. Could you expand a bit on what you mean by “guided by categories in distributional patterns”?
There are also some interesting thoughts on visual grammar in this paper by Yuri Engelhardt.
Thanks for your patience Neil, I do appreciate the dialogue.
No problem Dave. Actually, Yuri emailed me right before VizThink (Yuri, are you out there reading this?). I have a sense that he and I might also disagree about grammar, but it might be a further distinction between the types of disciplines we are in.
I don’t see why the visual form should be excepted at all. It certainly is the case that the visual-graphic form gives it unique properties, but they still work within the same general guidelines as other languages/grammars.
“guided by categories in distributional patterns” means that grammars involve “parts of speech.” The categories in a verbal grammar would be “nouns” and “verbs.”
These categories are determined because they generalize certain patterns of distribution. Nouns are not defined as “people, places, things, or ideas”. Those are semantic elements that may interface often with the syntactic nouns.
Rather, nouns are defined as such because they have certain distributional patterns: (in English…) they can act as subjects and objects, they follow determiners (the, some, a), follow adjectives (big, salty), are modified within prepositional phrases, can take plurals, etc.
You can also do various tests with them like substitution and deletion to further identify their patterns beyond just overt placement.
Also, units are not inherently these categories. The word “hit” can be a noun (a hit), a verb (to hit), an adjective (hit record), all depending one where it is distributed relative to other units.
Grammars must have these sorts of categories that emerge strictly from the distribution. And, these categories are unconscious and non-overt. Nobody sat down and said they are as such, we’ve had to discover them to be the case.
In the visual form, the categories are not nouns and verbs (but are things like Initials and Peaks), though they do seem to also map to semantic prototypes like the verbal form.
I’m actually just finishing up a series of experiments right now that show the visual form does have the types of categories I’m talking about (I hope… data still being collected/analyzed!).
After thinking about it a bit, I should say also that there are limited places where sequence does not have to occur in a visual grammar. These are places within the constraints of the grammar that do not matter as much about order.
This could hypothetically occur in verbal grammar too. Preceding a noun, you can have quite a few adjectives, only some of which enforce a specific order:
big smelly drooly shaggy dog
shaggy smelly big drooly dog
Within this “unconstrained” place of the sequence, I could see an allowance for broader spatial organization. However, this is a limited portion that is a result of the way the visual grammar is constructed. It doesn’t make the grammar — it is a “fallout” of it.