Linear Judgment Models: What Do They

Suggest About Human Judgment?

Psychology 466: Judgment & Decision Making

Instructor: John Miyamoto 10/07/2015: Lecture 02-2

Note: This Powerpoint presentation may contain macros that I wrote to help me create the slides. The macros aren’t needed to view the slides. You can disable or delete the macros without any change to the presentation.

Outline

• Four linear judgment models

• Finish: Model of the judge

• SMART model

• Unit weighting model

• What do studies of linear judgment models showabout human judgment?

Psych 466, Miyamoto, Aut '15 2

Psych 466, Miyamoto, Aut '15 3

Four Linear Judgment Models

1. Using multiple regression on objective data for which the true state is known.

2. Using multiple regression on judgment data where the true state is not known.

3. SMART (Simple Multi-Attribute Rating Technique)

4. Unit Weighting Model

--------------------------------------------

o Except for Model 4 (unit weighting), the models do NOT describe everyday judgment.

o Some natural models are similar to these models.

o Using any of these models can improve human performance.

Same Slide with Expanded Comments about Each Model

Psych 466, Miyamoto, Aut '15 4

Judgment Data Used in Baron's Chapter 20

COL = college GPA. This is the criterion. This is what the judge wants to predict.

SAT = SAT score; REC = judge's rating of the recommendation; ESS = judge's rating

of the student's essay; GPA = high school GPA. These are the cues.

Comment re Qualitative Variables

Psych 466, Miyamoto, Aut '15 5

Method 2: Model of the Judge

• Researcher has available

the scores for SAT,

REC, ESS and GPA.

• The values of the

criterion (COL) are

NOT available.

• Researcher asks the judge to make intuitive, global predictions for these cases.

This produces the column labeled "JUD”

(next slide).

Same Slide Except JUD Column Added to Table

Not

Ava

ilab

le

Not

Ava

ilab

le

Psych 466, Miyamoto, Aut '15 6

Method 2: Model of the Judge

• Researcher has available

the scores for SAT,

REC, ESS and GPA.

• The values of the

criterion (COL) are

NOT available.

• Researcher asks the judge to make intuitive, global predictions for these cases.

This produces the column labeled "JUD."

Same Slide Except Column Labeled “MUD” Added to Table

Not

Ava

ilab

le

Not

Ava

ilab

le

Psych 466, Miyamoto, Aut '15 7

Method 2: Model of the Judge

• Compute a regression model

that predicts JUD

(Model of the jUDge or

MUD).

• “Policy Capturing”.

• Example:

MUD = (–5x10–18)·SAT + 0.1·REC + 0.1·ESS + 2.0·GPA 4.8

• Use the MUD model to predict college GPA (COL) for these

cases or future cases.

Discussion of Model of the Judge

Not

Ava

ilab

le

Not

Ava

ilab

le

It is an accident that in this example, JUD and MUD are identical

Psych 466, Miyamoto, Aut '15 8

Discussion of Case 2: Model of the Judge (MUD)

• Empirical findings for Case 2 are the same as for Case 1

o MUD more accurate than the intuitive judgments of the judge.

o Judges think that they make their judgments by means of a complex, interactive, nonlinear evaluation of the cues.

o Whether or not this is really true, studies show that a much simpler combination of the cues produces better predictions.

• We don't need to know the value of the criterion (COL) in order to find a statistical formula (prediction equation) that can outperform the judge. .

Why Does MUD Outperform the Judge?

Psych 466, Miyamoto, Aut '15 9

Why Does MUD Outperform the Judge?

The model of the judge (MUD) is a model of the judge’s decisions, not of the criterion (reality).

• The MUD outperforms the judge because the statistical formula is consistent – it treats each case by the same formula.

• A human judge has all sorts of random variations in his or her judgment.

These random variations simply increase the inaccuracy (error) of the

judge's predictions.

• A human judge sees various “special circumstances”

(anecdotal memories) that suggest that a specific case

should be judged differently from the standard procedure.

Four Methods – SMART Method is Next

Psych 466, Miyamoto, Aut '15 10

Four Ways to Compute a Statistical Prediction Model

Method 1: Multiple regression applied to existing data.

Called a “proper linear model”

Method 2: Multiple regression applied to a judge’s predictions

Called a “model of the judge”

Method 3: SMART Method with "importance" weights

Called the SMART method or importance weighting method

Method 4: Unit weighting modelCalled the “unit weighting model or unit weighting method”

SMART Method

Next

Psych 466, Miyamoto, Aut '15 11

Method 3: The SMART Method

Simple Multi-Attribute Rating Technique (SMART)

A.k.a. Importance Weighting Model

• This technique lets us rank order the individual cases

(high school students). • A rank order tells us who is predicted to do better or worse.

This is enough to guide our selection of students.

• The SMART method can be modified to actually predict college GPA (COL), but we won’t go into this complication because it is not needed in a choice situation.

Step by Step Explanation of SMART Method

Psych 466, Miyamoto, Aut '15 12

Step 1: Convert the predictors, SAT, REC, ESS and GPA

to z-scores.

Step 2: The judge decides what is the relative importance of these variables,

e.g., SAT is twice as important as ESS, GPA is 50% more important than SAT,

etc.

Step 3: Create a prediction equation that reflects the judge’s judgment as to the

relative importance of the predictor variables.

• Example of Prediction of MAUT Model =

Predicted Score = 2·Z.sat + 3·Z.rec + 1·z.ess + 2·Z.gpa

Tables Showing Original Data, Z-Scores, and Prediction

Method 3: The SMART Method

Psych 466, Miyamoto, Aut '15 13

• Table on the left shows the initial data.

• Table on the right shows the z-scores for the predictor

variables, and the predicted rating for each student. • Predicted Score = 2·Z.sat + 3·Z.rec + 1·z.ess + 2·Z.gpa• The “Predicted” column tells you who is predicted to do better or

worse. It does not tell you the predicted GPA.

Example: Computing the Predicted Score for One Student

Z-Scores

Method 3: The SMART Method

Psych 466, Miyamoto, Aut '15 14

• Example of Prediction of MAUT Model =

Student Score = 2·Z.sat + 3·Z.rec + 1·z.ess + 2·Z.gpa

• Example for 1st student:

2(2.39) + 3(.32) + 1(.41) + 2(1.58) = 9.31

Comment on How to Convert Prediction to a Predicted College GPA

Z-Scores

Method 3: The SMART Method

Psych 466, Miyamoto, Aut '15 15

• To convert the “Predicted” column to an actual predicted

college GPA, you need to furnish a guess as to the mean and

variance of college GPA’s (unnecessary for your assignment).• Producing a predicted college GPA by this method requires a few math

tricks that are not worth discussing in Psych 466.

Findings for the SMART Method

Z-Scores

Method 3: The SMART Method

Psych 466, Miyamoto, Aut '15 16

Findings for Method 3: SMART Method

• Findings for Method 3 are the same as Method 2 (Model of the Judge)

o SMART method is more accurate than the intuitive judgments.

o The predicted ratings that are computed by the SMART method correlate better with actual results than do the judges intuitive predictions.

More Discussion of the SMART Method

Psych 466, Miyamoto, Aut '15 17

Discussion of Method 3: SMART Method

• The SMART method does not require that we know the value of the criterion (COL) for a set of known cases.

• We don't need a complicated calculation of the optimal regression weights (Methods 1 & 2 require this calculation).

• The SMART Method is better than intuitive judgment because ….o It approximately captures the relative importance of the predictor variables, and

….o It is consistent – it is not bothered by shifts of attention, fatigue, distractions,

anecdotal memories, etc.

Four Methods – Next We Discuss Unit Weighting Model

Psych 466, Miyamoto, Aut '15 18

Four Ways to Compute a Statistical Prediction Model

Method 1: Multiple regression applied to existing data.

Called a “proper linear model”

Method 2: Multiple regression applied to a judge’s predictions

Called a “model of the judge”

Method 3: SMART Method with "importance" weights

Called the SMART method or importance weighting method

Method 4: Unit weighting modelCalled the “unit weighting model or unit weighting method”

Unit Weighting Method

Next

Psych 466, Miyamoto, Aut '15 19

Reminder: What Are the Weights?

Method 1: Proper Linear Model

PRE = 0.000175·SAT + 0.092·REC + 0.217·ESS + 1.893·GPA 5.161

Method 2: Prediction Equation for the Judge’s Predictions

MUD = (–5x10–18)·SAT + 0.1·REC + 0.1·ESS + 2.0·GPA 4.8

Method 3: Predicted Scores Based on Importance Weights:

Student Score = 2·Z.sat + 3·Z.rec + 1·z.ess + 2·Z.gpa

• The weights are the numbers that are assigned to the predictor variables.

• Unit Weighting Model: Assign +1 to all positive attributes;

Assign –1 to all negative attributes.

Student Score = 1·Z.sat + 1·Z.rec + 1·z.ess + 1·Z.gpa

Further Explanation of the Unit Weighting Model

Psych 466, Miyamoto, Aut '15 20

Method 4: Unit Weighting Model

• Table on the right shows the z-scores for the predictor

variables, and the predicted rating for each student.

• Example for Case 1:

1(2.39) + 1(.32) + 1(.41) + 1(1.58) = 4.70

Findings for the Unit Weighting Model

Z-Scores

Psych 466, Miyamoto, Aut '15 21

Findings for Method 4: Unit Weighting Model

• Unit weighting model more accurate than the intuitive judgments of the

judge.

• Judges think that they make their judgments by means of a complex,

interactive, nonlinear evaluation of the cues. Whether or not this is really

true, studies show that a much simpler combination of the cues produces

better predictions.

• We don't even need to calculate optimal regression weights.

• All we need to know is which cues are positively related to the criterion and which are negatively related to the criterion.

Discussion of the Unit Weighting Model (cont.)

Psych 466, Miyamoto, Aut '15 22

Discussion of Method 4: Unit Weighting Model (cont.)

• Unit weighting model outperforms human judge because it is consistent - it treats each case by the same formula.

o A human judge has all sorts of random variations in his or her judgment. These random variations simply increase the inaccuracy (error) of the judge's predictions.

o A human judge may be influenced by anecdotal memories.The unit weighting model is not influenced by these memories.

• Comment: Results for unit weighting model show that the critical weakness

of the human judge is inconsistency, not

the inability to produce optimal weights.

Table: Pros and Cons of Prediction Methods

Psych 466, Miyamoto, Aut '15 23Repeat this Table Without the Red Rectangles

Onl

y if

you

wan

t to

mak

e a

quan

tita

tive

pre

dict

ion

Psych 466, Miyamoto, Aut '15 24Conclusions

Psych 466, Miyamoto, Aut '15 25

What We Have Learned from the Study of Linear Judgment Models

• Human judges typically believe that they use complex judgment processes to make judgments from complex cues. (This may be true.)

• The part of the human judgment process that validly predicts the criterion is well modeled by a linear model.

• We don't need to use optimal regression models to outperform human judges.

• We don't need to know the value of the criterion in order to create a model that outperforms human judges.

o Evidence: Model of the judge, unit weighting and importance weighting models outperform the human judge.

o Note: These methods work only because the human judge has some valid knowledge of the relationship between the cues and the criterion.

Sample Size Affects the Predictive Accuracy of Multiple Regression & MUD

Sample Size Affects the Accuracy of Multiple Regression Model and Model of the Judge

• This point is emphasized by Gigerenzer:

If you are using a multiple regression model or the model of the judge, the predicted accuracy of the model is bad if the sample size is too small.

♦ If the sample size is small, then the regression weights will tend to be inaccurate. (Technically, the variance of the regression weights increases as the sample size gets smaller.)

♦ If the sample size is small, the unit weighting model can be more accurate than multiple regression model or the model of the judge. (This point is emphasized by Gigerenzer.)

Psych 466, Miyamoto, Aut '15 26 Why Do Statistical Model Outperform Humans?

Psych 466, Miyamoto, Aut '15 27

Why Do Statistical Models Outperform Human Judges?

• Human judgment is affected by internal random variation;

statistical model is not.

• Human judgment is affected by vivid individual cases (anecdotes); statistical

model is not.

• Speculation:

Human judge tries to fit information into a story;

statistical model ignores story; it just adds up the evidence.

But is the human preference for stories bad?

General Discussion of Linear Judgment Models

General Discussion of Linear Models

• Why aren’t linear judgment models used more widely inpractical decision making?

♦ College or graduate admissions

♦ NIH or NSF grant review committees

♦ Political decisions like where to locate a prison; where to locate ahomeless shelter;

♦ The Denver bullet study

• Are linear judgment methods dehumanizing, e.g., when choosing who will be admitted to a college?

Psych 466, Miyamoto, Aut '15 28END