Understanding Early Stage Design Processes

Maria YangRobert N. Noyce Career Development Assistant ProfessorEngineering Systems Division and Dept. of Mechanical EngineeringNovember 25, 2009

ESD.83 Doctoral Seminar in Engineering Systems

2

Early stage of design for products and systems

High-impact phase within design and development process

Good design process leads to good design outcome Challenge: Early stage of design fluid, ambiguous,

difficult to measure Goal: Understand (and measure) informal design

activities through their outputs

3

General approach

Descriptive rather than predictive Many models about what goes on in design that are

based on intuition and experience Is that what really happens? Instrument the design process (Leifer)

What are artifacts of design process? How to capture their evolution? (Yang)

4

Engineering vs. psychology approach

NSF creativity workshops (with Frey) Psychologists use controlled studies (Paulus)

Pros – Min. confounding factors, individuals Cons – Short exercises (not realistic design), need

many participants (500 psych students – ltd domain knowledge)

Engineers coarser grain (Leifer, Agogino) Pros – More complex design activities, longer

projects, groups Cons – Confounding factors such as group dynamics

5

Who to observe

Ideal: Real world projects, access to process and project data Do you think this is always possible? Often, companies do not like to open themselves to

scrutiny Confidentiality Embedded (Owens)

Students in the classroom Novices Assessment inherent part of education process

6

Design Preferences

Design consists of 2 distinguishing activities Idea generation (synthesis) Idea selection

Idea selection assumes set of preferences Formal design synthesis approaches require formal

weightings for a preferences (Antonsson) Populate design space given sets of preference weightings

Reality – Design preferences given informally (“I like this better than

that” vs. “weight1 = .2, weight2 = .3”) Design preferences often aggregate of group opinion. Not

easy to do explicitly.

7

8

Metrics for design process

Research

Early stage design process

Designers

Design problem Design outcome

Design data

Clarify Generate Select

9

Metrics for design process

Research

Early stage design process

Designers

Design problem Design outcome

Design data

Clarify Generate Select

Sketching and Concepts (Yang 09; Yang 03)

Prototyping (Yang 04; Yang 05)

Design Information Retrieval (Yang, et al 05; Yang and Cutkosky 97, 98; Wood, Yang, et al 98; Yang, et al 98)

Designers and teams (Yang & Jin 07, 08)

Sketching Skill (Yang & Cham 07; Cham & Yang 05)

Preference Information (Ji, et al 07; Yang & Ji 07)

Users & needs (Lai, et al 09)

Sketching skill

11

Sketching in Design

Sketches capture & communicate [Ullman 90; Verstijnen 98; McKim 80; Schön & Wiggins 92]

Sketching process linked with design cognition [Nagai & Noguchi 03; Suwa & Tversky 97; Goel 95]

Sketching is “dialogue” [Cross 99; Shah, et al 01; Goldschmidt 91; Tovey, et al 03]

If sketching is language of design, is sketching proficiency linked to design process or performance? [Yang & Cham 07; Cham & Yang 05]

12

Research questions

What is the nature of sketching skill in design? Is drawing a generic ability? How are different drawing skills related? Research in mental imagery [Kosslyn 84; Kosslyn 94]

1. Comprehensive, generic “trait” 2. Task-based skill3. Somewhere between 1) and 2)

Hypothesis Sketching ability similar to (3)

13

Research questions

How is sketching ability linked to fluency? Hypothesis: Those who draw better also

draw more How is skill related to design outcome?

Hypothesis: Can sketching skill serve as an indicator of outcome?

14

Related work

For conceptual design, sketching preserves ambiguity [Goel 95; Kavakli, et al 98]

Sketch classification Function [Ullman 90;

Ferguson 92; van der Lugt 05; Goel 95]

Elements [McGown 98; Rodgers 00]

Sketching and outcome Teams who sketch vs.

those who don’t [Schütze 03]

3D sketching & outcome [Song & Agogino 04]

What about sketching skill?

15

Survey to assess drawing skill(do try this at home)

1. In 3 minutes, draw a bicycle with as much detail as possible.

2. Hold out the items given to you in your non-dominant hand (left-hand for right-handed persons). In 3 minutes, make a drawing of your hand and the items [two small candy bars].

3. Visualize and draw the following in 2 minutes: A rectangular box that is open at the top. Inside the box is a rubber ball. The front of the box has a large button, and each side of the box has a large “X” painted on it.

16

Survey goals & assessment

Engineering sketches may utilize many elements1. Bike task - Mechanical recall

Recall and sketch familiar mechanical object Structure, function (“Look like a bike? Could you ride it?”)

2. Hand task - Drawing facility Realistic, well-composed drawings from a still Proportions, realism (“Does this look like a hand?”)

3. Box task - Novel visualization Visualize specific features Proportions, 3D perspective, realism

17

Drawing Facility Task

Novel Visualization Task

Mechanical Recall Task

Level 1 Level 3 Level 5

18

Design outcomes

Sketch fluency Paper design logbooks; relatively objective Perspective drawings; more skill required

Grades for class and for final project Rankings by external judges Spearman Correlations

19

Results: Types of sketching skill

Possible results1. Comprehensive skill: Strong correlations between

tasks2. Task-based skill: No correlation3. Skill lies between the two: Range of correlations

Results suggest option 3

Correlation between sketch tasks. N = 32, Rs >= 0.296 for = 0.10.

0

0.05

0.1

0.15

0.2

0.25

0.3

Bike task and Handtask

Bike task and Boxtask

Hand task and Boxtask

Co

rrel

atio

n c

oef

fici

ent,

Rs

20

Sketching ability and fluency

Total: Drawing “well” correlates positively 3D: Bike task correlates negatively Drawing skill vs. other means of visualization?

N = 32, Rs >= 0.296 for = 0.10

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

Bike task Hand task Box task

Co

rrel

atio

n C

oef

fici

ent,

Rs

Total sketches

3D sketches only

21

Sketching and Design Outcome

Sketch fluency: Positive but no sig. correlation Sketching skill: No clear trends Design process depends on many skills/factors Project type, outcome measures More studies needed

N = 33, Rs >= 0.291 for = 0.10 N = 32, Rs >= 0.296 for = 0.10

0

0.05

0.1

0.15

0.2

0.25

0.3

Project grade Class grade Avg. ranking

Co

rrel

atio

n c

oef

fici

ent,

Rs

Total sketches

3D sketches only

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

Project grade Class grade Avg. ranking

Co

rrel

atio

n c

oef

fici

ent,

Rs

Bike task

Hand task

Box task

22

Conclusions

1. Is sketching ability generic? Sketching skills not created equal Possible reason: Different cognitive skills required

(gearhead and artist)

2. Is sketching ability linked to sketch fluency? Hand and box task correlate, but not bike Sketch fluency (partly) influenced by how much a

designer can design without drawing. Possible reasons

Mechanical recall = visualization in head Common complaint: “I don’t need to keep a logbook”

23

Conclusions

3. Sketch ability linked to design performance? No relationship between sketch tasks and outcome “Good” sketchers did not necessarily do well (or vice

versa) Possible reasons

Engineering design complex, requires many skills; sketching is only one

Sketching may be behavioral rather than a necessary element of design activity (doodler)

Extraction of preferential probabilities from design team discussion

25

Overview

Making choices is one key activity in design Designers express "design preferences" by assigning

priorities to a set of possible choices Assigning preferences can be complex for a team

Elicitation of preferences from a group (surveys, voting) Aggregation of preferences among the group

[Mark 02]

26

Research Questions

How can preferential probabilities of a design team be extracted? (Ji, et al 07; Yang & Ji 07)

Obtained implicitly, not explicitly How to address aggregation?

Do preferential probabilities evolve over time? Way to describe how a team selects alternatives throughout the

design process How does extracted information compare with that obtained

explicitly? Consistent with preference information captured via surveys?

Preferential probability: Likelihood one alternative will be selected as “most preferred” over all others

27

Approach

Extract preferential probabilities from transcripts of design team discussion Design alternatives known a priori Assume preference-related information embedded

No formal aggregation of individual information Simple collection of words

Assumptions What designers think in one time interval relates to what

they thought in the previous interval Designers tend to speak positively about the design

alternative they prefer more and negatively about those they prefer less

28

Related Work

Preference Extraction Surveys: The lottery method [Hazelrigg, 99; Otto & Antonsson, 93] Pair-wise comparison: AHP [Saaty 00], fuzzy outranking [Wang 97] Multi-criteria overall aggregation function using MoI [Scott & Antonsson 98] Conjoint Analysis [Green 90] and Discrete choice analysis [Hensher &

Johnson 81; Ben-Akiva & Lerman 85] Collaborative filtering [Kohrs & Merialdo 00]

Group Preference Aggregation Cardinal utility functions for accumulating group preferences [Keeney 76] Structured pair-wise comparison chart [Dym, Wood & Scott 02] Aggregation with equal weights [Bask & Saaty 93] Aggregation with unequal weights [Jabeur, et al. 99; See & Lewis 05] Arrow’s Theorem: no guarantee of consistency in a group [Arrow 70, 86]

Design Process Evolution Surveys [Brockman 96], Coding of design journals [Jain & Sobek 06] Team cohesion analysis (“Story telling”) [Song, et al 03]

29

Models

Preference Transition Model (PTM): relationship between preferences in 2 consecutive time intervals

Utterance-Preference Model (UPM): relationship between preferences and utterances in one interval

i+1 n i m

p P( =a | =a )= 1-p

N-1

when n m

when n mp p

ìï =ïïíï ¹ïïîMost-preferred alternative in i+1 Probability designers won’t change most-

preferred alternative

0 1p£ £

Alternative uttered in time interval i

Most-preferred alternative in interval i

Probability designer will utter their most-preferred alternative

i n i m

q P( =a | =a )= 1-q n

N-1

when n m

when me p

ìï =ïïíï ¹ïïî

11q

N< £

30

Case Study 1: Large Scale Space System Design

Highly concurrent, real-world design team working on concept stage of space system architecture

17 experienced scientists and engineers; range of disciplines

Focused on group of 4 working on single subsystem Three 3-hour sessions of discussion ~28,000 words Primary team member talked nearly 85% of the

time Two component selection problems

[http://history.nasa.gov]

31

Case study 1: Results from Large Scale Space System Design

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5 6 7 8 9 10 11 12

Time Interval (in 10 minutes)

Mo

st-P

refe

rred

Pro

bab

ilit

y

Alternative a1

Alternative a2

Alternative a3

[Ji, Yang & Honda, 2007, ASME IDETC 2007]

a2 and a3 alternate with each other as most-preferred choice

Alternative a1 is least preferred

View of how probability of preference changes over time for a re-design problem

32

Case Study 2: Coffee Maker Design

Small engineering team with 3 graduate studentsName/ID Glass carafe Stainless-steel carafe Plastic carafe

Photo

Description Glass with warming plate

thermal-insulated stainless-steel

thermal-insulated plastic (inside glass)

Cost $10.00 $20.00 $15.00

Footprint size Big Small Small

Fragility Fragile Strong Fragile inside

Durability Durable Durable Less durable

Heat retention Good Satisfactory Good

Weight Light Heavy Light

Portability Not portable Portable Portable

Easy to clean Easy to clean Not easy to clean Not easy to clean

Style Moderately attractive Very attractive Not attractive

Capacity 2 - 6 cups 2 - 6 cups 2 - 6 cups

Spout Does not dribble Dribbles after pouring Dribbles after pouring

Can tell how much coffee is left

Yes No No

33

Case study 2: Preferential Probability Results From Transcript Analysis

0

0. 1

0. 2

0. 3

0. 4

0. 5

0. 6

0. 7

0. 8

0. 9

1

12: 00 22: 45 32: 30 41: 22 50: 20

Ti me (mm: ss)

Pref

eren

tial

Pro

babi

lity

gl asssteelpl ast i c

34

Case study 2: Comparison of Preferential Probabilities from Transcripts and Surveys

0

0. 2

0. 4

0. 6

0. 8

1

12: 00 22: 45 32: 30 41: 22 50: 20

Ti me (mm: ss)

Pref

eren

tial

Pro

babi

lity

PPSPPT

0

0. 2

0. 4

0. 6

0. 8

1

12: 00 22: 45 32: 30 41: 22 50: 20

Ti me (mm: ss)

Pref

eren

tial

Pro

babi

lity

PPSPPT

Stainless Steel Carafe

Plastic Carafe

0

0. 2

0. 4

0. 6

0. 8

1

12: 00 22: 45 32: 30 41: 22 50: 20

Ti me (mm: ss)

Pref

eren

tial

Pro

babi

lity

PPS

PPT

Glass Carafe

Measure Possible Ranges

Results

L1 norm [0, 16] 1.84

L2 norm [0, 4] 0.543

Cosine similarity [0, +1] 0.974

Pearson product-moment correlation coefficient

[-1, +1] 0.956p-value:

7.67E-9

Spearman’s rho [-1, +1] 0.833p-value:

6.56E-5

35

Conclusions

Approach capable of extracting preferential probabilities

Preferential probabilities extracted from transcripts changed over the course of the design process

In this work, preference-related information extracted from the transcripts was consistent over time with those from surveys

Future work

37

Integrated view of design activities

Design thinking manifests itself in different forms at different points of the design process

What are these forms? How do they collectively evolve over time? What is their relationship to outcome?

Sketches Prototypes

Text

Sketches Prototypes

Text

Time

38

System modeling

Formulate better system level models to improve system design and reliability

Consider emergent properties: nonlinear, complex interactions between subsystems Draws on existing subsystem models and empirical

system data Allows prediction of future states, balancing of design

trade-offs System model

Thermal-hydraulic

subsystem

Structuressubsystem

Controlssubsystem

Other subsystems…

Complex interactions among subsystems

39

Modeling the language of design

Understand how designers express preference in natural language

Linguistically and mathematically model preference as expressed in engineering design texts

Advance basic knowledge of the “language of design” Challenge: Model uncertainties in preference and

convert into mathematical models applied to formal design decision-making

Recommended for NSF Award

Language of RiskPreference, Choice and , Uncertainty &

Preference

Mathematical Models

Engineering Design

Validation

40

Teaching

ESD.40 Product Design & Development 2.009 Product Engineering Processes IAP 2.97 Design-A-Palooza (new, mostly ugrad)

Focus on defining problems

41

Acknowledgments

Thoughtful support of MIT Engineering Systems Division and Department of Mechanical Engineering

2006 NSF CAREER Award DMI-0547629 NASA Cooperative Agreement NNA04CL15A