Inferential and Expressive Capacities of Graphical Representations
Tutorial
Diagrams 2004University of Cambridge March 23, 2004
Survey and Some Generalizations
School of Knowledge ScienceJapan Advanced Institute of Science & Technology
ATR Media Information Science Labs
Atsushi Shimojima
Purpose
• To understand three concepts useful to capture the inferential-expressive capacities of many graphical systems.
• In my personal terminology:– Free ride– Over-specificity– Derived meaning
Why Important?
These concepts are very often alluded to in the
literature
Their ranges of application never explicated in full
Their exact contents seldom defined
but
Different people used different terms to refer to them, sometimes missing
important connections of their ideas and findings
thus
Plan for the hour
Free ride
Over-specificity
Derived meaning
ExamplesAnalysis & definition Connections
Outstanding questions
1. Free Ride
A toy example:
• Jon, Ken, Gil, Bob, and Ron run races• of the kind with no “ties” in arrival
Suppose:
• Different ways of expressing the information that Jon defeated Bob :
Compare:
Defeated(Jon,Bob)
Jon defeated Bob.
Jon Bob
Defeated(Jon,Bob)
Jon defeated Bob.
Jon Bob
Atomic sentence of a first-order language (FOL) with:
• two-place predicate Defeated • its arguments Jon and Bob
Defeated(Jon,Bob)
Jon defeated Bob.
Jon Bob
Representation of PD system (position diagrams) where:
Horizontal relation of names indicate arrival order of people.
Defeated(Jon,Bob)
Jon defeated Bob.
Jon Bob
Sentence of English describing the arrival order of two people.
PD system (a bit more precisely)
Syntactic rules:– Two or more of the
names “Jon”, “Ken”, “Gil”, “Bob”, and “Ron” appear in a horizontal row.
– The same name appears at most once.
Semantic rules:– If the name X
appears to the left of the name Y, the bearer of X defeated the bearer of Y.
Defeated(Jon,Bob)FOL
Jon defeated Bob.English
Jon BobPD
Look Similar
But behave quite differently when more information expressed
Difference 1
Defeated(Jon,Bob) & Lost_to(Ken,Bob)FOL
Jon defeated Bob and Ken lost to Bob.English
Jon Bob KenPD
Express information:Jon defeated Bob.Ken lost to Bob.
Jon Bob KenPD
The PD system expresses an additional piece of information…
Express information:Jon defeated Bob.Ken lost to Bob.
Defeated(Jon,Bob) & Lost_to(Ken,Bob)FOL
Jon defeated Bob and Ken lost to Bob.English
…while FOL and English don’t.
Express information:Jon defeated Bob.Ken lost to Bob.
• In PD, expressing certain sets of information results in the expression of additional, consequential information.
Free rides=
Another example: Venn diagrams
Express information:All As are Bs.No Bs are Cs.
As
Bs Cs
As
Bs Cs
As
Bs Cs
Expressing certain sets of information results in the expression of additional, consequential
information
Another example: Euler diagrams
Express information:A ⊂ BC ∩ B = φ
A
B
CA
B
Expressing certain sets of information results in the expression of additional, consequential
information
Another example: Maps
Express information:B’s house is in front of F’s house across the river.
SCD1PEFLA 7MW SCD1PEFLA 7MWBExpressing certain sets of information results in the expression of additional, consequential
information
Of course we cannot always do the manipulations in our heads: we may have to draw a diagram on paper, or re-arrange parts of a scale model, in order to see the effects.…The main point is that the ability to apply such subroutines to parts of analogical configurations enables us to generate, and systematically inspect, ranges of related possibilities, and then…to make valid inferences, for instance about the consequences of such possibilities. (p. 220.)
Sloman (1971)
Diagrams are physical situations….As such, they obey their own set of constraints…By choosing a representational scheme appropriately, so that the constraints on the diagrams have a good match with the constraints on the described situation, the diagram can generate a lot of information that the user never need infer. Rather, the user can simply read off facts from the diagram as needed.
Barwise & Etchemendy (1990)
We have seen that formally producing perceptual elements does most of the work of solving the geometry problem. But we have a mechanism---the eye and the diagram---that produces exactly these “perceptual” results with little effort. We believe the right assumption is that diagrams and the human visual system provide, at essentially zero cost, all of the inferences we have called “perceptual.” As shown above, this is a huge benefit. (p. 99.)
Larkin & Simon (1987)
Other conceptions
• Non-deductive representation systems where the operation of the construction process entails the “making” of the inferences (Lindsay 1988, p. 112)
• Inference by recognition (Novak 1995)
• Inference by inspection and transformation (Olivier 2002, p. 72--74)
• Emergence effect (Kulpa 2003, p. 90)
• Emergent properties (Koedinger 1992, as cited by Olivier 2001)
• Emergent relations (Chandrasekaran, Kurup, and Banerjee 2004)
But
What, more exactly, is the “free-ride” capacity?What is the general condition---semantic mechanism---for a system to have that property?
Basic Assumption
indicates
a property
a property Represented object Y
Representation X
A representation X expresses information about the represented object Y by having a property that indicates the corresponding property of Y.
Example: PDs
indicates
[the name “Jon” appears to the left of the name “Bob”]
[Jon defeated Bob] a particular running race
a particular position diagram
Jon Bob
Example: Euler diagrams
indicates
[the circle “A” appears inside the circle “B”]
[A B]⊂a particular groupof objects
a particular Euler diagram
AB
Condition for Free Ride: PD system
The name “Ken” appears to the right of the name “Bob”
The name “Jon” appears to the left of the name “Ken”.
The name “Jon” appears to the left of the name “Bob”
Jon defeated Ken.Jon defeated Bob. Ken lost to Bob.
indicates indicates indicates
constraint
constraint
Condition for Free Ride: Euler Diagram
A circle “C” and the circle “B” has no overlap.
The circle “C” and the circle “A” has no overlap.
A circle “A” appears inside a circle “B”.
C ∩ A = φA B⊂ C ∩ B = φ
indicates indicates indicates
constraint
constraint
Condition for Free Ride: General
indicates indicates indicates
constraint
constraint………
………
Constraints on representations themselves track constraints in the represented domain.
(Shimojima 1996a, 1996b)
Thus:A system with a free-ride property supports deductive inference through physical manipulation of representations on an external display, not in the head.
A (paradigm) case of distributed cognition
Connection: AI systems
• WHISPER for the prediction of the collapsing of objects (Funt 1980)
• REDRAW I & II for the deflection shape problem (Tessler, Iwasaki, and Law 1995a, 1995b)
• KAP for the prediction of the movements of cam-follower pairs and meshing gears (Olivier, Ormsby, and Nakata 1996)
• DRS component (Chandrasekaran et al. 2004)
Some AI systems utilize the free-ride capacities of graphical systems by installing some manipulation-inspection abilities on diagrams.
Connection: graphical simulations
• Dynamic behaviors of strings, flexible rods, and rings, falling in free space, etc. (Gardin and Meltzer 1995)
• Liquid behaviors in and out of containers with complex shapes (Decuyper, Keymeullen, and Steels 1995)
Some graphical simulations can be considered free rides in an extended sense, combining computer-controlled dynamic constraints with geometrical-topological constraints on graphics.
Connection: studies of sketching in designFree-ride capacities may be an essential factor of the utility of pictorial sketches in design process.
Each move is a local experiment which contributes to the global experiment of reframing the problem. Some moves are resisted (the shape cannot be made to fit the contours), while others generate new phenomena. As Quist reflects on the unexpected consequences and implications of his moves, he listens to the situation’s back talk, forming new appreciations which guide his further moves. (p. 94.)
Schoen (1982)
Thus the drawing represents a sort of hypothesis or “what if” tool. With a plan, for example, the architect can say, what if the kitchen were here, the dining-room next to it and the living-room there? How could I then organize the entrance and the stairs? (p. 242 , colored emphasis by me.)
Lawson (1997)
• Goldschmidt (1994)– “One reads off the sketch more information than was invested in i
ts making” (p. 164)– Such reading-off of unexpected reading-off as an essential step in
“interactive imagery” in design
• Suwa, Gero, and Purcell (2000)– Relationship between unexpected discoveries in sketches and inv
ention of new design requirements
Also:
Warning: Recognition problem
• “Cheap rides” (Gurr, Lee, and Stenning 1998, Gurr 1999)• Expertise in diagram construction to facilitate the recogniti
on of useful consequences (Novak 1995)
Free rides only guarantee the expression of consequential information in the representation, not its recognition by the user.
2. Over-Specificity
Difference 2
Express information:Jon defeated Bob.Ken defeated Bob.
Defeated(Jon,Bob) & Defeated(Ken,Bob)FOL
Jon defeated Bob and Ken defeated Bob.English
Jon Ken BobPD Ken Jon Bob? ?
Jon Ken BobLD Ken Jon Bob? ?
The PD system can’t express the info without additional info…
Express information:Jon defeated Bob.Ken defeated Bob.
…while FOL and English can.
Defeated(Jon,Bob) & Defeated(Ken,Bob)FOL
Jon defeated Bob and Ken defeated Bob.English
Express information:Jon defeated Bob.Ken defeated Bob.
• In PD, certain sets of information cannot be expressed without expressing additional, non-warranted information.
Over-specificity=
Another example: Euler diagrams
Express information:A ⊂ BC ∩ B ≠ φ
A
B
A
B
C
Certain sets of information cannot be expressed without expressing additional, non-warranted information.
How do you placeThis circle?
CC
C
Another example: Maps
Express information:K’s house is between A’s house and B’s house.
SCD1PEFLA 7MWBCertain sets of information cannot be expressed without
expressing additional, non-warranted information.
K
Where do you placethis icon?
``the demand by a system of representation that information in some class be specified in any interpretable representation'' (p. 98)
Stenning and Oberlander (1995)
Specificity
=
However, there are some ways in which a picture can often carry too much information or indicate a degree of precision which may be inappropriate….It would be difficult to construct a drawing which did not suggest other features of the form of the finished product which might restrict a future designer. (p. 242.)
Lawson (1997)
[In geometrical proofs,] though we do not for the purpose of the proof make any use of the fact that the quantity in the triangle (for example, which we have drawn) is determinate, we nevertheless draw it determinate in quantity (As cited by Kulpa 2003, p. 101.)
Aristotle (350 B. C. E.)
Other conceptions
• Analog property of representation systems as opposed to digital property (Dretske 1981)
• Smaller degree of discretion (Norman 2000, p. 110)
• Particularity feature (Kulpa 2003, p. 96, p. 101)
Not in the sense of Goodman
(1982)!
Analysis of over-specificity: Euler Diagram
A circle “C” and the circle “B” has some overlap.
A circle “A” appears inside a circle “B”.
A B⊂ C ∩ B ≠ φ
indicates indicates indicates
constraint
C ∩ A ≠ φC ∩ A =φ
indicates
The circle “C” and the circle “A” has some overlap.
The circle “C” and the circle “A” has no overlap.
constraint
Analysis of over-specificity: Position Diagram
indicates indicates indicates
constraint
indicates
(1’)
(1’) The name “Jon” appears to the left of the name “Bob”.
(2’)
(2’) The name “Ken” appears to the left of the name “Bob”.
(2)
(2) Ken defeated Bob(1) Jon defeated Bob
(1)
(4’) The name “Ken” appears to the left of the name “Jon”.
(4’)
(3’) The name “Jon” appears to the left of the name “Ken”.
(3’)
constraint
(3)
(3) Jon defeated Ken
(4)
(4) Ken defeated Jon
Analysis of over-specificity: General
indicates indicates indicates
constraint
constraint
indicates
………
………
………
………
(Shimojima 1996b)
Thus:
The expressive efficacy of a system depends on “what it allows you to leave unsaid” as well as “what it allows you to say” (Levesque 1988, p.370).
A system with an over-specificity property prohibits the exclusive expression of certain (small) sets of information even when they are in the system’s expressive coverage.
“Knowledge expressed in propositional format can determine part of the state of the world while conveniently leaving other parts undetermined.” (Ioerger 1992, as cited by Kulpa 2003, p. 101.)
In contrast:
“[Descriptions] need only express their intended information; they can leave unsaid and indeterminate some aspect of what is described; they need not convey more than is required.” (Norman 2000, p. 110.)
Also:Expressing such a set of information in the system produces a representation with semantic contents beyond that set.
Accidental features
=
“Having demonstrated that the three angles of an isosceles rectangular triangle are equal to two right ones, I cannot therefore conclude this affection agrees to all other triangles which have neither a right angle nor two equal sides” (Introduction, paragraph 16).
Berkeley (1710)
More recent discussion:Problem of “unintended exclusion” (Giaquinto 1993) Problem of “over-looked divergence” (Kulpa 2003)
• Hyperproof (Barwise and Etchemendy 1994)• System for geometry proof (Luengo 1995, Winterstein et
al. 2000)
An over-specificity property of a representation system poses a challenge to some (all?) attempts to build a formal deductive system based on that system.
Thus:
3. Derived Meaning
Difference 3
Express information:
Jon defeated Bob.Gil defeated Jon.
Defeated(Jon,Bob) & Defeated(Bob,Ken) & Defeated(Gil,Jon) & Defeated(Ken,Ron)FOL
Jon defeated Bob and Bob defeated Ken and Gil defeated Jon and Ken defeated Ron.English
Gil Jon Bob Ken RonPD
Bob defeated Ken.Ken defeated Ron.
Gil Jon Bob Ken RonPD
You can count the number of names to find out the count of people satisfying a certain condition…
Express information:
Jon defeated Bob.Gil defeated Jon.
Bob defeated Ken.Ken defeated Ron.
Defeated(Jon,Bob) & Defeated(Bob,Ken) & Defeated(Gil,Jon) & Defeated(Ken,Ron)FOL
Jon defeated Bob and Bob defeated Ken and Gil defeated Jon and Ken defeated Ron.English
…while you can’t, in FOL and English.
Express information:
Jon defeated Bob.Gil defeated Jon.
Bob defeated Ken.Ken defeated Ron.
Defeated(Jon,Bob) & Defeated(Bob,Ken) & Defeated(Gil,Jon) & Defeated(Ken,Ron)FOL
Jon defeated Bob and Bob defeated Ken and Gil defeated Jon and Ken defeated Ron.English
The count of names don’t mean the count of people.
Express information:
Jon defeated Bob.Gil defeated Jon.
Bob defeated Ken.Ken defeated Ron.
• In PD, some additional meaning relation holds that does not hold in FOL and English.
• Moreover, that relation is derivative in that it is not written in basic semantic rules.
Derivative Meaning=
Another example: Tables
Gil Jon Bob Ken Ron
Count of circles in a row means the count of people to the right
Ron ÅõÅõÅõÅõGil ÅõKen ÅõÅõBob ÅõÅõÅõJon RonGilKenBobJonCount of circles in a column means the count of people to the left
Additional meaningrelation
Another example: Scatter plots
151050 20100151050 20100
I IIThe shape formed by dots means a general fact about the distribution:
• Existence of correlation, • Its strength, • Existence of an exceptional instance, etc.
Additional meaningrelation
From Tufte (1983)
Scatter plots...employ point symbols (such as dots, small triangles, or squares) as content elements. The height of each point symbol indicates an amount. These displays typically include so many points that they form a cloud; information is conveyed by the shape and the density of the cloud. (p. 46.)
Kosslyn (1994)
Another example: Data mapsFrom Tufte (1983)
Additional meaningrelation
The concentration of dots along the Broad Street band means a concentration of deaths along Broad Street.
Another example: Node-edge graphs
Additional meaningrelation
Concentration of lines on one node means that the corresponding station is a “hub” of the subway system.
Another example: Node-edge graphs
K
C
E
H
G
H
D
A
F
JAdditional meaningrelation
“H” node’s being lower in the same group as “K” node means that K is an ancestor of H.
Olivier (2001)
Analysis of derived meaning: PDs
(1) At least two people defeated Jon
indicates
constraint
indicates
(1*) At least two names appear to the left of the name “Jon”
constraint
…
…
indicates indicates
A particular way (1*) holds
A particular way (1) holds
Enlarged view
The name “Gil” is to the left of the name “Jon”
The name “Bob” is to the left of the name “Jon”
Gil defeated Jon Bob defeated Jon
indicates indicates
A particular way (1*) holds
A particular way (1) holds
(1) At least two people defeated Jon
constraint
(1*) At least two names appear to the left of the name “Jon”
constraint
Analysis of Derived Meaning: General
indicates
constraint
indicates
constraint
… … …
… … …
indicates indicates
A particular way holds
A particular way holds
(Shimojima 1999, 2002)
Thus:A representation system with a meaning derivation property allows the simultaneous presentation of local information and global information implied by the local information.
Concerning the original numbers, they can be easier and more accurately read off from a list of numbers, without the expense of producing that graph. What such graphs are really for is something different---namely, a possibility to see at a glance some general conclusion, i.e., a result of some reasoning that follows from the interaction of these numbers. (p. 111)
Kulpa (2003)
aa
'96'97'98'99-100102030mln $
Indeed, the central purpose of many scientific diagrams is to depict relationships and interactions….If students are to understand such diagrams, they need to be able to do more than just decode the symbols used. They must also be able to uncover and assimilate salient relationships between the symbols that constitute a diagram and appreciate how these relationships map onto the real-world situation being represented. (p. 28.)
Lowe (1989)
Petre (1995): Generally critical about the insensitive use of graphics in programming environment, but mentions “gestalt response” as an “informative impression of the whole that provides insight into the structure” (p. 42) and admits it as a potential benefit of graphics.
Also:
Ratwani, Trafton, and Boehm-Davis (2003): Assumes the difference of global/trend reading and local reading, and goes on to demonstrate that different mental operations are involved in them.
Connection: expert reading of graphicsCorrectly assessing and recognizing derived meaning may be a major component of expertise in reading graphical representations.
For example:
• Lowe (1989): Compared the professional meteorologists’ way of inspecting (incomplete) meteorological charts with non-meteorologists’ and find their greater appreciation of large-scale patterns of organization.
• Winn (1991): Discusses expertise of reading topographical maps as perceptual chunking of contours to form larger features, such as valleys.
Summary
Derived meaning
Expertise in graphics readingInformation graphics/
visualization
Graphics semantics
Free ride
Distributed cognition
Graphical simulation
AI systems with diagrams
Design studies
Over-specificity
Formal deduction systemsJustificatory status of
diagrammatic proofs
Design studies
Useful and central set of concepts
Free ride
Over-specificity
Derived meaning
Mental transformation of graphicsNarayanan, Suwa & Motoda (1995), Schwartz (1995), Trafton & Trickett (2001), Shimojima (2003)
Perceptual chunking of graphical elementsLowe (1989), Anderson & Koedinger (1992), etc.
Spatial indexing of informationLarkin & Simon (1989), etc.
Low-encoding diagramsCheng (2003) etc.
Auto-consistencyBarwise & Etchemendy (1995), Stenning & Inder (1995), Lemon & Pratt (1997)
Concepts not covered today
Top Related