Mental Models of Cost Estimation: Mental Models of Cost Estimation: A Focus on Uncertainty AssessmentA Focus on Uncertainty Assessment

24th International Forum on COCOMO and Systems/Software Cost Modeling

Dr. Ricardo Valerdi, MITDr. Linda Newnes, University of BathDr. Yee-May Goh, University of BathMelanie Kreye, University of BathDr. Jairus Hihn, NASA JPL November 3, 2009

Outline

• Functions of mental models

• Example mental model from NASA

• Mental models of risk assessment

• Methodology for capturing mental models

• Measuring change in mental models

• Implications & next steps

© Northrop Grumman Mission Systems

What is the value of π?

• Physics: 3.14159265358979323946…

• Math: c/d

• COCOMO user: ~3.14 (round up to 4.0)

• Accounting: 3.14 or whatever you want

• Sales: 3.14 (before the mail-in rebate)

• CEO: where’s my cake?

Human Memory

• Organized around schemas– Abstract and general rules

based on experience and other people/information

• When schemas are initiated, they direct behavior

• In a novel situation, individuals recall certain schemas in order to make sense of the new information

Mental ModelsPsychological representations of real, hypothetical or

imaginary situations (Craik, K. The Nature of Explanation, 1943).

• Mental models include what a person thinks is true, not necessarily what is actually true

• Mental models are similar in structure to the thing or concept they represent

• Mental models allow a person to predict the results of their actions

• Mental models are simpler than the thing or concept they represent. They include only enough information to allow accurate predictions

Functions of Mental Models

Rouse, W. B., People and organizations: explorations of human-centered design , Wiley 2007.

Example: Estimation Process at NASA(Normative Model)

Source: Hihn, J., Griesel, A., Bruno, K., Fouser, T., Tausworthe, R., Mental Models of Software Forecasting, NASA JPL.

Example: Estimation Process at NASA

(Descriptive Model)

Source: Hihn, J., Griesel, A., Bruno, K., Fouser, T., Tausworthe, R., Mental Models of Software Forecasting, NASA JPL.

Mental Models of Risk Assessment

Canadian Standards Association

Source: Fischoff, B., Realistic Risk Disclosure in Newly Normal Times.

Risk Assessment and Management

Source: Cooper, L. P., How Project Teams Conceive of and Manage Pre-Quantitative Risks, PhD Dissertation, University of Southern California, 2008.

Methodology for Capturing Mental Models

• Semi-structured interviews (normative)– What models do you use to conceive of risk?– How is risk captured as part of the cost

estimation activity?

• Ethnography (descriptive)– Document descriptive risk perception based

on spontaneous responses to certain problem

situations– Differentiate risk perceptions across roles (cost

estimator, PM, software engineer, etc.) and context (mission critical, IT development, telecom, etc.)

Temporal Properties of Mental Models

Immediate Past FutureSource: Radioactive Waste Management Committee, Report No. NEA/RWM/FSC(2003)7/REV1, The Mental Models Approach to Risk Research.

Measuring Change in Mental Models

Doyle, J. K., Radzicki, M. J., Trees, W. S., Measuring the effect of systems thinking interventions on mental models, pp. 129-132 in George P. Richardson and John D. Sterman, eds. Proceedings of the 1996 International Conference of the System Dynamics Society. Cambridge, Massachusetts. July 22-25, 1996.

1. Attain a high degree of experimental control

2. Separate measurement and improvement

3. Collect data from individuals in isolation

4. Collect detailed data from the memory of each individual

5. Measure change rather than perceived change

6. Obtain quantitative measures of characteristics of mental models

7. Employ a naturalistic task and response format

8. Obtain sufficient statistical power

Way Forward• Implications

– Documenting mental models will help understand the schemas for pre-quantitative risk assessment

– Richness of mental models will allow implicit information to influence quantitative risk assessment

– Contextual differences in risk assessment can be made explicit

• Next steps– Elicitation of mental models from a

varied sample of engineering decision makers

– Integration of uncertainty models with cost models

Thank YouThank You