Building the Design Studio of the Future Aaron Adler Jacob Eisenstein Michael Oltmans Lisa Guttentag...
-
Upload
george-blankenship -
Category
Documents
-
view
220 -
download
0
Transcript of Building the Design Studio of the Future Aaron Adler Jacob Eisenstein Michael Oltmans Lisa Guttentag...
Building the Design Studio of the Future
Aaron AdlerJacob EisensteinMichael OltmansLisa GuttentagRandall Davis
October 23, 2004
AARON ADLER Building the Design Studio of the Future October 23, 2004
Sketching Is Not the Whole Picture
AARON ADLER Building the Design Studio of the Future October 23, 2004
Speech and Gestures
• Express more precise spatial and structural relationships
• Relate sketches that are drawn from different perspectives
• Describe temporal events or paths of motion
• Disambiguate using multiple modalities
AARON ADLER Building the Design Studio of the Future October 23, 2004
Roadmap
Exploratory Studies
Multimodal Communication
DomainModeling
DialogueManagement
Recognition Components
WoZ Dialogue Studies
DomainReasoning
Multimodal Understanding
Multimodal Intelligent
Design Studio
Usability
Human Dialogue
- Out-of-vocabulary terms
- Illuminate unclear parts- Possible design mistakes
AARON ADLER Building the Design Studio of the Future October 23, 2004
Outline
• Sketching is not the whole picture
• Roadmap
• Trimodal Studies
• Preliminary Components– Speech and Sketching– Gesture and Speech
AARON ADLER Building the Design Studio of the Future October 23, 2004
Outline
• Sketching is not the whole picture
• Roadmap
• Trimodal Studies
• Preliminary Components– Speech and Sketching– Gesture and Speech
AARON ADLER Building the Design Studio of the Future October 23, 2004
Interesting Questions to Study
• Sketch– What information is best communicated with
sketching?– How do people sketch when they’re
explaining things?– How do they sketch when they’re taking
notes?
AARON ADLER Building the Design Studio of the Future October 23, 2004
Interesting Questions to Study
• Speech– What information is best communicated with
speech?– How do people talk when they’re drawing?
How can ASR better support speech in the context of other modalities?
– How should computers generate and understand back-channel feedback?
AARON ADLER Building the Design Studio of the Future October 23, 2004
Interesting Questions to Study
• Gesture– What information is best communicated with gesture?– Which features of gesture are idiosyncratic? Which
are universal?– What does gesture tell us about discourse structure?– Are gestures cues for disfluencies? Sentence
breaks?– How can we build embodied agents with realistic body
language?
AARON ADLER Building the Design Studio of the Future October 23, 2004
Trimodal Interaction Study
• How are speech, gesture, and sketching used together?– 15 pairs of participants– Two video cameras, a Mimio, and a Tablet PC– 4.5 hours of data
AARON ADLER Building the Design Studio of the Future October 23, 2004
Looking Ahead to Recognition
• Captured digital video
• Colored gloves used for future hand tracking
• Headset microphones
• Time-stamped sketch data
AARON ADLER Building the Design Studio of the Future October 23, 2004
Trimodal Setup
Listener
Speaker
AARON ADLER Building the Design Studio of the Future October 23, 2004
Trimodal Setup
Listener SetupSpeaker Setup
AARON ADLER Building the Design Studio of the Future October 23, 2004
Details to Observe
• Structure then function• Back channel communication• Collaboration between speaker and
listener
AARON ADLER Building the Design Studio of the Future October 23, 2004
Early Results
• We identified two phases in the explanations:– Structure phase– Function phase
• Participants used back channel communication during the explanations
• The amount of sketching the listener did varied
AARON ADLER Building the Design Studio of the Future October 23, 2004
Outline
• Sketching is not the whole picture
• Roadmap
• Trimodal Studies
• Preliminary Components– Speech and Sketching– Gesture and Speech
AARON ADLER Building the Design Studio of the Future October 23, 2004
Results: Speech and Sketching
• Filled pauses important: “ahh”, “umm”
• Key phrases: “there are”, “and”
AARON ADLER Building the Design Studio of the Future October 23, 2004
Multimodal Understanding and Integration
• Combining speech and sketching we can make it easier to perform some operations
“There are three identical pendulums.”
“There are three touching pendulums.”
AARON ADLER Building the Design Studio of the Future October 23, 2004
Speech-Gesture Integration
• Align referential pronouns and gestures to the diagram
“ this piece is made of wood”
• “it moves back and forth like this”
deictic tracing trajectory
Time
this piece goes here this…andso
AARON ADLER Building the Design Studio of the Future October 23, 2004
Results: Speech and Gesture
• Corpus-based approach– Feature set based on linguistic knowledge
• Improves performance– 84% F-measure for baseline system– 95% F-measure for optimization system