Active Capture and Folk Computing Ana Ramírez and Marc Davis ICME 2004 – Taipei, Taiwan 29 June...

Active Capture and Folk Active Capture and Folk ComputingComputing

Ana Ramírez and Marc Davis

ICME 2004 – Taipei, Taiwan29 June 2004

UC Berkeley - Garage Cinema Research - Group for User Interface Research

6/29/2004 2

Smart Multimedia Acquisition Smart Multimedia Acquisition SystemsSystems• First two papers – automatic camera

calibration Image Audio

• Third paper – understand structure of what is being captured to edit in real time

• Active capture - smart cameras that interactively guide and capture human action

6/29/2004 3

OutlineOutline

• Sample applications

• Active Capture

• Designing Active Capture algorithms

• Future work

6/29/2004 4

SampleSample ApplicationsApplicationsAutomatic Movie TrailersAutomatic Movie Trailers

Play Video

• Video of capture process

http://www.cs.berkeley.edu/~anar/presentations/icme2004/HeadTurn.mpg

6/29/2004 5

SampleSample ApplicationsApplicationsAutomatic Movie TrailersAutomatic Movie Trailers

Play Video

• Video of automatically created movie trailer

6/29/2004 6

SampleSample ApplicationsApplicationsSports InstructionSports Instruction

6/29/2004 7

SampleSample ApplicationsApplicationsTelemedicineTelemedicine

Rural Town

Large City

leishmaniasis

6/29/2004 8

SampleSample ApplicationsApplicationsAutomated Health ScreeningAutomated Health Screening

Rural Town

Large City

leishmaniasis

6/29/2004 9

Active CaptureActive Capture

Capture Interaction

Processing

Direction/Cinematography

Human-Computer Interaction

Computer Vision/

Audition

ActiveCapture

6/29/2004 10


Capture Interaction

Processing



Computer Vision/

Audition

ActiveCapture

6/29/2004 11


Capture Interaction

Processing



Computer Vision/

Audition

ActiveCapture

6/29/2004 12


Capture Interaction

Processing



Computer Vision/

Audition

ActiveCapture

6/29/2004 13


Capture Interaction

Processing



Computer Vision/

Audition

ActiveCapture

6/29/2004 14

Active CaptureActive Capture• Traditionally, signal processing

algorithms avoid interacting with the user

• Signal processing + interaction => more sophisticated recognizers

• How to design hybrid algorithms that involve capture, interaction, and processing

6/29/2004 15

Components of Active Capture Components of Active Capture AlgorithmsAlgorithms

• Simple computer vision and audition recognizers / sensors Motion Eyes Sound

• Desired action in terms of recognizers

• Interaction script

6/29/2004 16

Design ProcessDesign Process

• Input: Desired action = head turn Recognizers = motion, eyes

MotionEyes

time

6/29/2004 17



• Step 1: Express desired action in terms of

recognizers

MotionEyes

time

No Motion

No Eyes

Motion

Eyes

No Motion

6/29/2004 18



• Step 1: Express desired action in terms of

recognizers

• Step 2: Design interaction script

6/29/2004 19

Design Process – Step IIDesign Process – Step II

time

no motion

head turning

no motion

"Please face left..." "Turn to face the camera..."

1.5 - 4 sec

> 1.5 sec< 1 sec

< 4.5 sec< 3.5 sec

> 0.25 sec

"Great!"

> 0.9 sec

capture

Motion

Eyes

0.25-1.25s

0.25-2.75s

no eyes

eyes

“GoAhead”

"Please stand still..."motion

eyes "Please look away..."

0.25-2.75s"Please look at end of turn..."

0.25-2.75s"Didn’t look at camera..."

0.25-2.75s"Please turn slower..."

"Didn’t move..."0.25-2.75s

6/29/2004 20


time

no motion

head turning

no motion


1.5 - 4 sec

> 1.5 sec< 1 sec

< 4.5 sec< 3.5 sec

> 0.25 sec

"Great!"

> 0.9 sec

capture

Motion

Eyes

0.25-1.25s

0.25-2.75s

no eyes

eyes

“GoAhead”






"Didn’t move..."0.25-2.75s

6/29/2004 21


time

no motion

head turning

no motion


1.5 - 4 sec

> 1.5 sec< 1 sec

< 4.5 sec< 3.5 sec

> 0.25 sec

"Great!"

> 0.9 sec

capture

Motion

Eyes

0.25-1.25s

0.25-2.75s

no eyes

eyes

“GoAhead”






"Didn’t move..."0.25-2.75s

Play Video

6/29/2004 22


time

no motion

head turning

no motion


1.5 - 4 sec

> 1.5 sec< 1 sec

< 4.5 sec< 3.5 sec

> 0.25 sec

"Great!"

> 0.9 sec

capture

Motion

Eyes

0.25-1.25s

0.25-2.75s

no eyes

eyes

“GoAhead”






"Didn’t move..."0.25-2.75s

6/29/2004 23


time

no motion

head turning

no motion


1.5 - 4 sec

> 1.5 sec< 1 sec

< 4.5 sec< 3.5 sec

> 0.25 sec

"Great!"

> 0.9 sec

capture

Motion

Eyes

0.25-1.25s

0.25-2.75s

no eyes

eyes

“GoAhead”






"Didn’t move..."0.25-2.75s

Play Video

6/29/2004 24

Design ChallengesDesign Challenges

Step I - Description of action Approximate timing Strict and non strict ordering

Step II – Interaction script What to do if something goes wrong –

mediation

Step I – Action DescriptionStep I – Action Description

6/29/2004 26

• Observations• Commands• Capture• Time constraints• Strict ordering• Non-strict ordering

Step I - Action DescriptionStep I - Action DescriptionVisual LanguageVisual Language

6/29/2004 27

Step I - Action DescriptionStep I - Action DescriptionVisual LanguageVisual Language• Observations• Commands• Capture• Time constraints• Strict ordering• Non-strict ordering

6/29/2004 28


6/29/2004 29


6/29/2004 30


6/29/2004 31


time

waving

noise

"Say your name and wave"

> 4.5 sec

capture

Waving

Noise

<1 sec <1 sec


6/29/2004 32


time

waving

noise

"Say your name and wave"

> 4.5 sec

capture

Waving

Noise

<1 sec <1 sec


Step II – Interaction ScriptStep II – Interaction Script

6/29/2004 34

Step II – Interaction ScriptStep II – Interaction ScriptContextual InquiriesContextual Inquiries• Golf instructor• Aikido instructor• 911 emergency phone operator• Triage nurse• Children’s portrait photographer• Film and theatre directors

[Jeffrey Heer, Nathaniel S. Good, Ana Ramirez, Marc Davis, and Jennifer Mankoff. “Presiding Over Accidents: System Direction of Human Action.” In: Proceedings of the Conference on Human Factors in Computing Systems (CHI 2004) in Vienna, Austria. ACM Press, 463-470, 2004. ]

6/29/2004 35

Step II – Interaction ScriptStep II – Interaction ScriptDirection and Feedback StrategiesDirection and Feedback Strategies

• External aids

Play Video

6/29/2004 36


• Decomposition and “Show”

Play Video

6/29/2004 37


• Method shift from “Show” to “Tell”

Play Video

6/29/2004 38


6/29/2004 39

SummarySummary• Active Capture – smart cameras that

interactively guide and capture human action

• Sample applications Automated health screening Automated movie clips Sports trainer

• Design Challenges Description of action Interaction script

6/29/2004 40

Future WorkFuture Work

• Support design and implementation of Active Capture applications

• Evaluate the relative contribution of signal analysis and user interaction in these hybrid algorithms

6/29/2004 41

QuestionsQuestions

Ana Ramí[email protected]

www.cs.berkeley.edu/~anar

Garage Cinema Researchhttp://garage.sims.berkeley.edu

Group for User Interface Researchhttp://guir.berkeley.edu

Active Capture and Folk Computing Ana Ramírez and Marc Davis ICME 2004 – Taipei, Taiwan 29 June...

Documents

Transcript of Active Capture and Folk Computing Ana Ramírez and Marc Davis ICME 2004 – Taipei, Taiwan 29 June...