Automated Critique of Sketched Mechanisms

Jon Wetzel and Ken Forbus

Qualitative Reasoning Group

Northwestern University

Outline

• Motivation• Algorithm

– Input: CogSketch

– State Transition Verification• Generate requirements from sketches

• Make predictions using QM

• Verify that predictions reflect requirements

• Evaluation• Related Work• Future Work

Motivation

Engineering Design and Communication Courseat Northwestern University

http://www.segal.northwestern.edu/undergraduate/edc/

Example Sketch of EDC project

Palm-actuated one-handed fingernail clipper

Long-term Goal: Design Buddy

• Given: An explanation consisting of…

• Return:– Is the explanation logically consistent?

– Why or why not?

– Provide feedback that improves users’ communication skills

Sketches Language-like input

+

Outline

• Motivation• Algorithm

– Input: CogSketch

– State Transition Verification• Generate requirements from sketches

• Make predictions using QM

• Verify that predictions reflect requirements

• Evaluation• Related Work• Future Work

Critique of Sketched Mechanisms

Critique is a three-step process

1. Derive a set of state transition requirements using the qualitative representation of a sketch

2. Use qualitative mechanics (QM) to predict outcome of each state• Qualitative mechanics (Wetzel and Forbus, 2008)

based on Nielsen (1988) and Kim (1993)

3. Compare prediction with requirements

Input: CogSketch

• User draws three categories of things– Objects

– Annotations

– Relations

• User labels each using our ontology (based on OpenCyC KB)

• No reliance on recognition, except for arrows

Input: CogSketch

Sketches include knowledge representation

(isa PalmRest RigidObject)

(isa Spring1 Spring-Device)

(forceApplied PalmRest Down) . . .

Input: CogSketch

CogSketch computes spatial relations

(touchesDirectly PalmRest LeverArm)

(connectedTo-Directly Spring1 LeverArm)

(QualitativeVectorBetween (tail ForceArrow1) (head ForceArrow1) Down)

Input: CogSketch

• Multiple subsketches

• One subsketch per mechanical state

Up (open) position

Down (closed) position

Each sketch has a metalayer

• Contains all of the subsketches

• User draws relation arrows indicating cause-effect relationship

Input: CogSketch

Outline

• Motivation• Algorithm

– Input: CogSketch

– State Transition Verification• Generate requirements from sketches

• Make predictions using QM

• Verify that predictions reflect requirements

• Evaluation• Related Work• Future Work

State Transition Verification

• Input: CogSketch sketch with labeled state transitions

• Output: Comparison of requirements and verification1. Sketches depict the required result of previous state

2. Qualitative mechanics predicts state behaviors

3. Verify the predicted behavior makes required results

Step 1/3: Derive Requirements

Spatial relations are computed between subsketches

(relativePositionChange State1 State2 PalmRest ClipperFrame Down)

(q-2D-orientation State1 PalmRest Horizontal)

(q-2D-orientation State2 PalmRest Horizontal)

. . .

Step 1/3: Derive Requirements

Spatial relations become the state transition requirements.

(relativePositionChange State1 State2 PalmRest ClipperFrame Down)

(q-2D-orientation State1 PalmRest Horizontal)

(q-2D-orientation State2 PalmRest Horizontal)

. . .

To go from state1 from state2, PalmRest must move down.

To go from state1 from state2, PalmRest must not rotate.

. . .

RequirementsSpatial Relations

Step 2/3: QM Predicts Behavior

Each requirement becomes a query for the initial state.

To go from state1 from state2, PalmRest must move down.

To go from state1 from state2, PalmRest must not rotate.

. . .

Query: (in State1 (transMotion PalmRest ?dir))How will PalmRest translate?

Query: (in State1 (rotMotion PalmRest ?dir))How will PalmRest rotate?

. . .

Requirements Queries based on requirements

Step 2/3: QM Predicts Behavior

Qualitative Mechanics rules generate prediction• Last year:

– Representation for forces, rigid objects and their surfaces

– Rules propagate forces and motion between surface contacts

• New this year: – Non-rigid body: springs

– Representation for surface textures

– Gears

Based on work by Nielsen (1988) and Kim (1993)

Step 2/3: QM Predicts Behavior

Qualitative Mechanics rules generate prediction

QM

Query: (in State1 (transMotion PalmRest ?dir))How will PalmRest translate?

Query: (in State1 (rotMotion PalmRest ?dir))How will PalmRest rotate?

. . .

(transMotion PalmRest Down))PalmRest will move down.

(rotMotion PalmRest CW))

PalmRest will rotate clockwise.. . .

Result of verificationQueries based on requirements

Step 3/3: Compare Results

Check to see if requirements are verified

(transMotion PalmRest Down))PalmRest will move down.

(rotMotion PalmRest CW))

PalmRest will rotate clockwise.. . .

To go from state1 to state2, PalmRest must move down.

To go from state1 to state2, PalmRest must not rotate.

. . .

Requirements Result of verification

Disagree

Indicates a possible bug in the design or explanation

Agree

Outline

• Motivation• Algorithm

– Input: CogSketch

– State Transition Verification• Generate requirements from sketches

• Make predictions using QM

• Verify that predictions reflect requirements

• Evaluation• Related Work• Future Work

Evaluation

• Selected EDC projects from previous years– Mechanical systems

– Could be easily represented in two dimensions

• Final corpus:16 student projects worked with CogSketch and our QM– 6 were very similar to other projects, so 10 were

sketched

Evaluation

• Success criterion: – Illustrate motions and forces in the design

– Successful state transition verification• All requirements correctly derived from sketch

• QM successfully verifies all requirements

• Result: succeeded on examples representing 15 of the 16 projects

Examples from EDC Projects

Examples from EDC Projects

Examples from EDC Projects

Examples from EDC Projects

Related Work

• SketchIt (Stahovich et al, 1998)

• Recognition: (Alvarado and Davis, 2004; Hammond & Davis, 2005; Kurtoglu and Stahovich, 2002)

• Physics simulation: (Alvarado and Davis, 2001)

Future Work

• Extend and improve QM– Use surface texture representation to implement

friction

– Automatically suggest axes of rotation

– Predict possible outcomes from ambiguous input

• Studies with EDC students this Fall– Add interface for language-like input– Generate feedback which improves communication

skills

Acknowledgements

This research was supported by NSF SLC Grant SBE-0541957, the Spatial Intelligence and Learning Center (SILC)