"Let's leave that up to the computer" Chris Snijders.
-
Upload
marjorie-rogers -
Category
Documents
-
view
220 -
download
0
Transcript of "Let's leave that up to the computer" Chris Snijders.
"Let's leave that up to the computer"
Chris Snijders
www.chrissnijders.com/supercrunchers2012
Chris Snijders, Super Crunchers, 2012
Chris [email protected]
Martijn [email protected]
Kees van [email protected]
Chris Snijders, Super Crunchers, 2012
Passing the course …
• Presence and active participation, including presentation, evaluation of others, …
• Create “CaseFile” based on the SuperCrunchers book
• Write assignment about own “Super Cruncher” idea
Chris Snijders, Super Crunchers, 2012
Overview of today
• A famous example: Cook county hospital• The science behind it• Computers as decision makers
• Warning! Upcoming assignment
Chris Snijders, Super Crunchers, 2012
Case: Cook county hospital
Emergency Department
- 250.000 patients per year
- many persons without insurance
- not enough rooms, overworked staff
- 1996: Brendan Reilly director
(see Gladwell, 2005)
Chris Snijders, Super Crunchers, 2012
Problem 1: acute chest pain
Diagnose through:
blood pressure, stethoscope: fluid in the lungs, how long have you been experiencing pain, how does it feel precisely, where does it hurt, does it always hurt or only when you exercise, have you had heart problems before, how about your cholesterol, do you have diabetes, let's look at your ECG, are there any heart problems in the family, do you use drugs, how old are you, are you in shape, do you smoke, do you drink, check appearance: stressed, overweight, ....
High risk : 8
Medium risk : 12
Go home30 p/day
Chris Snijders, Super Crunchers, 2012
Reilly finds Goldman: obv 10,000 cases
Only 4 things matter
ECGBlood pressureFluid in your lungs"unstable angina"
Chris Snijders, Super Crunchers, 2012
Great! So let's do that! Or not...
Implementation: … physicians protest …
A test: 20 cases were given to several physicians
Hardly any agreement between physicians!
Chris Snijders, Super Crunchers, 2012
Reilly tests Goldman’s idea
vs
82% 95%
physician Goldman’s scheme
Chris Snijders, Super Crunchers, 2012
A literature check …
Clinical versus statisticalprediction
For instance (zie Grove et al., 2000)
– Survival probabilities in medical procedures– Probability of recidivism– Probability of success of starting firms– Choice of job candidates– Diagnosing schizofrenia– Predicting school success– …
Chris Snijders, Super Crunchers, 2012
What would a typicaltopic be, where you
would expect the humanto win?
Chris Snijders, Super Crunchers, 2012
The results …
Over 160 studies
When given the same info,the number of cases in whichthe expert wins = ??
Chris Snijders, Super Crunchers, 2012
Models beat Humans (often)How can this be?
Chris Snijders, Super Crunchers, 2012
... we have some clues ...
We emphasize the improbable’ (Stickler)
Confirmation bias (Edwards, Wason)
Hindsight bias (Fischhoff)
Cognitive dissonance (Festinger)
“Dealing with probabilities / Base rate neglect”(Bar-Hillel)
Mental sets (Redelmayer, Tversky)
Our memory fools us (Wagenaar)
Chris Snijders, Super Crunchers, 2012
And there are more of these
"Mental Floating Frankfurters"
Chris Snijders, Super Crunchers, 2012
Restriction 1: “Mental sets”
Connect the 9 dots with at most 4 straight lines, without lifting your pen from the paper.
• • • • • • • • •
Chris Snijders, Super Crunchers, 2012
Restriction 2: Memory
“Where were you, when …”
Shuttle Columbia Crew Lost Feb. 1, 2003
Chris Snijders, Super Crunchers, 2012
Restriction 3: the “availability heuristic”
Which is more likely, a plane crash or a car crash?
Chris Snijders, Super Crunchers, 2012
Restriction 4a: dealing with probabilities
Suppose: a manager has a good intuition in business:
– when a problem will arise: he gets a gut-feeling that something is wrong with probability 90%
– when no problem will arise: he gets a gut-feeling that something is wrong with probability 10%
On average, there is a problem in 5% of the transactions.
The manager starts a transaction, and he gets a gut-feeling that something might be wrong.
What is the probability that something is indeed wrong?
Chris Snijders, Super Crunchers, 2012
Restriction 4a - solution
1,000 transactions
50 are problematic950 are not problematic
45xsignal
5xNo signal
855xNo signal
95xsignal
If a signal, then the probability of a problem = 45 / (45+95) = 0.32
Chris Snijders, Super Crunchers, 2012
Restriction 4b: dealing with probabilities
A murder has been committed. The only evidence available is DNA, found at the murder scene. DNA-research shows a match with your DNA.
The probability that two persons are diagnosed as having the same DNA is about 1 in 100.000.
How likely is it that you are the murderer?
Chris Snijders, Super Crunchers, 2012
Restriction 5: overconfidence
Trivial Pursuit: estimate how many questions you will know
Estimates are generally too high ... and this gets worse with expertise!
Chris Snijders, Super Crunchers, 2012
Restriction 6:Finding non-existent patterns
Chris Snijders, Super Crunchers, 2012
Restriction 7: the noble art of finding a broken leg
Chris Snijders, Super Crunchers, 2012
Restriction 8: where is the feedback?
Chris Snijders, Super Crunchers, 2012
Decision making =Store, retrieve, combine
Chris Snijders, Super Crunchers, 2012
Where were we?
End of 1 set of reasons "why" humans (even expert humans) are often outperformed by computer models.
Chris Snijders, Super Crunchers, 2012
The competition:
• “Naturalistic decision making” (cf. work by Shanteau)
• Fast and frugal heuristics (cf. Gigerenzer)
Chris Snijders, Super Crunchers, 2012
The timelines of ideas
... and the science behind many more questions that you can ask in relation to such topics
• This is an innovation process • Can we find consistencies across topics?• Which kind(s) of crunchers are more likely to
be adopted?• etc...
idea implementation
Chris Snijders, Super Crunchers, 2012
Reilly finds Goldman: obv 10,000 cases
Only 4 things matter
ECGBlood pressureFluid in your lungs"unstable angina"
Chris Snijders, Super Crunchers, 2012
AND?are we using this today?
Chris Snijders, Super Crunchers, 2012
NO!
Chris Snijders, Super Crunchers, 2012
Chris Snijders, Super Crunchers, 2012
When and why do the models win?
Can we use the experts’ knowledge somehow?
When are the models used, and when not (and why is that)?
What can/should you do when you want to have a model-based solution?
What prevents people from using models?
Chris Snijders, Super Crunchers, 2012
The Supercrunchers book
Intro
Who's doing your thinking for you?
Creating your own data with the
flip of a coin
Government by chance
Evidence based medicine
Experts vs equations
Why now?
Are we having fun yet?
On the web site
[CaseFile]
[Example]
[Issue]
[Method]
Chris Snijders, Super Crunchers, 2012
“Super Crunching” = using data
• Supercrunching = Using (lots of) data to predict something (think Twitter, Blogs, Airmiles, …) that we normally cannot predict
• Supercrunching = (also) Using data to predict something that humans normally tend to predict– Experts vs models– Experiment– “Natural experiments”
Chris Snijders, Super Crunchers, 2012
<show website now>
Chris Snijders, Super Crunchers, 2012
Passing the course …
• Presence and active participation, including presentation, evaluation of others, …
• Create “CaseFile” based on the SuperCrunchers book
• Write assignment about own “Super Cruncher” idea (or: we take a single topic and each do parts of it)
Chris Snijders, Super Crunchers, 2012
Up next …
• Next week: no class (read! And start preparing the first Assignments)
• The week after that: we get creative…
Chris Snijders, Super Crunchers, 2012
To do for next week
• Read the book – cover to cover
• Think about the Cook-county case, try to think about general patterns
• Upcoming assignment will be to create a “casefile” for one of the topics in the book: check for topics that interest you, if possible choose one already, and let me know