Graphics and Words

When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make

sense? How to integrate images and words The Narrative thread

Consider that hieroglyphs gave way to more abstract symbols

Why turn back the clock?

x 1

i

-17K years

Imagens

Pavio’s dual coding theory

Visual System

AuditorySystem

Verbal

VisualImage Information

InformationFrom speech

Text Information

Associate structure

Logogens Verbalresponses

Non-verbalresponses

Capacity of visual working memory (Vogal, Woodman, Luck, 2001)

Task – change detection Can see 3.3 objects Each object can have several attributes

OrientationShape size, texture

1 second

Capacity of verbal working memory

Used to be though of a 7 +/- 2 It is now thought of as more a duration of

~ 2sec and a store of proto-verbal codes. Words have an enormously rich semantic

structure Sometimes we have a choice

Spatial prepositions: above, across, alongside, amidst, apart, around, back, behind, below, beneath, beside, between, beyond, by (out of approx 80: From Pinker)

Reasoning with words and graphics

1 2 3 4 5 6 7 8 9 Get a set of numbers, such that 3 of them

add to 15.

Can we have a visual language?

The nature of language Words are arbitrary Chomsky, grammar with innate deep structures. Common to computer languages Austin - Illocutionary force (speech acts) Critical period for language development But, being verbal is not essential to language

development Sign languages for the deaf are the most perfect

examples of visual language

Sign languages

ASL Danish SL Chinese SL

Arose spontaneouslyAre not related to verbal languagesHave grammarBecome more abstract over time

NSLGoldin-Meadow

What can we do with language?

Description The ability to communicate procedures and

sequences of operations – including logic – if, but, causes, do a then b then c

A vehicle for thought when internalized Communicate goals, intentions (illucutionary acts). Make things happen Nit picking (mutual grooming)

“If there is no fresh trout at Whole Foods then go to the fishmarketOn 19th and get salmon.”

There are special brain areas for language

Production

Wernicke

BrocaUnderstanding

Production

VisualCortex

AuditoryCortex

Can we have a visual language?

Yes – but …

If we don’t learn it early somesome capabilities are best handled verbally

The visual system gives us

Visual processing andvisual recognition

Rapid recognition and pattern finding

Abstraction Pattern

Jane is Jim’s boss Jim is Joe’s boss Anne works for Jane Mark works for Jim Anne is Mary’s boss Anne is Mike’s boss

Joe Mary

Jane

Mike

Jim

Mark

Anne

Org chart

Visual and verbal pseudo-code

While letters in stack Take a letter Put a stamp on it Put it in the ‘out tray’

Visual programming languages have a history of failure

get line of textfrom input file

change charactersto upper case

write line to outputfile

more input?yes

no

Data flow diagrams are defunct

Examples of visual languages

Sanscrit Petri-nets Khoros

SanscritSum integers from 1 to 3

for i = 1 to 3 dosum = sum + i;

Petri Net languagePetri nets are stochastic – timed attributed (tokens on nodes, transitions)

Khoros

Attaching images and words

Pointing, Diexis and the diectic gesture

Can be a glance or a nod An elementary speech

act Shown to disambiguate

verbal communications Why the mouse is so

powerful The basis of shared

environments

Attaching words to images- use gestalt laws

A square

helplet me out!

Some simple shapes

hexagon

ab

c

d

Integrated pictures and words moreEffective: Chandler and Sweller 1991

Narrative: Visualizations can be used to help tell a story

Can something visual carry a narrative.

In psychological terms the important thing is a leading of the thread of ideas (=attention). In data analysis the analyst has the narrative

thread. In a presentation the audience is following the

presenter’s thread. (actually a hybrid is best).

Narrative

Can be accomplished with either images or words

A leading of attention

Degrees of narrative

Strong leading Free exploration

XPlane

Recap

Words for procedure, logic and abstract meanings/ images for pattern and structure

Both words and images can sustain narratives (leading attention)

Emotional and motivational effects (Images can be more efficient) and automatic.

Gillnet Set Surface Buoy

Lobster Trap Trawl

Surface Buoy

Landry 2002

Theory: Graphics and Words

Words for procedural logic: conditionals, qualifiers, if-then else, while. + causality.

Not the formal logic of math but a kind of concrete logic. (A vowel must have an odd number on the back).

Other kinds of gestures

Beat gestures for emphasis Verb gestures showing how to do

something

McNeil Hand and mind

Issues in shared environments

Speech + Pointer + Visuals – most important components

Subtle ways of directing attention also important in meeting dynamics.

Peter