SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL
description
Transcript of SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL
![Page 1: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/1.jpg)
SIMPSON’S PARADOX,ACTIONS, DECISIONS,
AND FREE WILL
Judea PearlUCLA
www.cs.ucla.edu/~judea
![Page 2: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/2.jpg)
SIMPSON’S PARADOX(Pearson et al. 1899; Yule 1903; Simpson 1951)
• Any statistical relationship between two Any statistical relationship between two variables may be variables may be reversedreversed by including by including additional factors in the analysis.additional factors in the analysis.
Application: The adjustment problemApplication: The adjustment problem
• Which factors Which factors shouldshould be included in the be included in the analysis.analysis.
![Page 3: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/3.jpg)
SIMPSON’S PARADOX(1951 – 1994)
M R RT 2 8 10T 9 21 30 11 29 40
M R RT 18 12 30T 7 3 10 25 15 40
R RT 20 20 40T 16 24 40 36 44 80
+ =
0.6 < 0.7 0.2 < 0.3 0.5 > 0.4T – Treated T – Not treatedR – Recovered R – DeadM – Males M – Females
Easy question (1950-1994)• When / why the reversal?Harder questions (1994)• Is the treatment useful? Which table to consult?• Why is Simpson’s reversal a paradox?
![Page 4: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/4.jpg)
SIMPSON’S REVERSAL
Pr(recovery | drug, male) < Pr(recovery | no-drug, male)Pr(recovery | drug, female) < Pr(recovery | no-drug, female)
Group behavior:
Overall behavior:Pr(recovery | drug) > Pr(recovery | no-drug)
![Page 5: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/5.jpg)
TO ADJUST OR NOT TO ADJUST?
Treatment
Recovery
GenderT F Mediating
factorF
TreatmentT
RRecoveryR
Solution: Adjust iff F blocks all back-door paths
![Page 6: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/6.jpg)
TWO PROOFS:1.
2. When two causal models generate the same statistical data and in one we decide to use the drug yet in the other not to use it, our decision must be driven by causal and not by statistical considerations.
Thus, there is no statistical criterion to warn us against consulting the wrong table.
Q. Can Temporal information help?A. No!, see Figure (c). F = Car Type
THE INEVITABLE CONCLUSION:THE INEVITABLE CONCLUSION:THE PARADOX STEMS FROMTHE PARADOX STEMS FROM
CAUSAL INTERPRETATIONCAUSAL INTERPRETATION
1. Surprise surfaces only when we speak about “efficacy,” not about evidence for recovery. )|())(|( TRPTdoRP
![Page 7: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/7.jpg)
WHY TEMPORAL INFORMATION DOES NOT HELP
• In (c), F may occur before or after T, and the correct answer is to consult the combined table.
• In (d), F may occur before or after T, and the correct answer is to consult the F-specific tables.
Treatment
Recovery
GenderT F
R
Treatment
Recovery
BloodPressure
Treatment
Recovery
TF
R
T F
R
T F
R
(a) (b) (c) (d)
Car
![Page 8: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/8.jpg)
WHY “CAR TYPE”SHOULD NOT MATTER
• Listening to the Bell creates association between Treatment and Recovery though Treatment does not affect Recovery
• Treatment efficacy is distorted in the Bell-specific tables.
• Treatment efficacy is properly represented in the combined table.
Treatment Recovery
Coin-1 Coin-2
Bell
![Page 9: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/9.jpg)
THE SOLUTION OF THEADJUSTMENT PROBLEM
P(y | do(x)) is estimable if there is a set Z ofvariables such that Z d-separates X from Y in Gx.
Z6
Z3
Z2
Z5
Z1
X Y
Z4
Z6
Z3
Z2
Z5
Z1
X Y
Z4
Z
Moreover, P(y | do(x)) = P(y | x,z) P(z)(“adjusting” for Z) z
Gx G
![Page 10: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/10.jpg)
WHY TEMPORAL INFORMATION DOES NOT HELP
• In (c), F may occur before or after T, and the correct answer is to consult the combined table.
• In (d), F may occur before or after T, and the correct answer is to consult the F-specific tables.
Treatment
Recovery
GenderT F
R
Treatment
Recovery
BloodPressure
Treatment
Recovery
TF
R
T F
R
T F
R
(a) (b) (c) (d)
Car
![Page 11: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/11.jpg)
WHY SIMPSON’S PARADOXEVOKES SURPRISE
1. People think causes, not proportions.
2. "Reversal" is possible in the calculus of proportions but impossible in the calculus of causes.
![Page 12: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/12.jpg)
CAUSAL CALCULUSPROHIBITS REVERSAL
Pr(recovery | drug) < Pr(recovery | no-drug)
Pr (male | do{drug}) = Pr (male | do{no-drug})
Pr(recovery | drug, male) < Pr(recovery | no-drug, male)
Pr(recovery | drug, female) < Pr(recovery | no-drug, female)
Group behavior:
Entailed overall behavior:
Assumption:
do{drug} do{no-drug}
do{drug} do{no-drug}
do{drug} do{no-drug}
![Page 13: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/13.jpg)
THE SURE THING PRINCIPLE
Theorem 6.1.1 An action A that increases the probability of an event E in each subpopulation must also increase the probability of E in the population as a whole, provided that the action does not change the distribution of the subpopulations.
![Page 14: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/14.jpg)
Female0.5
Male0.5
T T TT
R0.6
R0.2
R0.7
R0.3
R R0.7
RR
0.3 0.2 0.35 0.15 0.1 0.4 0.15 0.35
0.3 < 0.35 0.1 < 0.15
T FGender
R
(a)
DECISION TREE FOR F = Gender
F
![Page 15: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/15.jpg)
-HBP0.75
R0.6
R0.7
R0.2
E0.3
R0.4
-E0.7
R0.3
R0.8
0.45 0.2 0.05 0.2 0.175 0.075 0.225 0.525
0.5 0.4
T FBloodPressure
R
(b)T T
HBP0.25
-HBP0.25
HBP0.75
>
DECISION TREE FOR F = Blood Pressure
F F
![Page 16: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/16.jpg)
R0.5
R0.4
R0.5
T FBloodPressure
R
(b)
T T
R0.6
0.5 > 0.4
COMPRESSED TREE FOR F = Blood Pressure
![Page 17: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/17.jpg)
WHAT DECISION TREESHOULD WE BUILD
FOR F = Car ?
Treatment
Recovery
TF
R
Car
(c)
???
Should F = Car be handled likeF = GenderorF = Blood Pressure
???
![Page 18: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/18.jpg)
0.3
0.6
Cheap0.5
Expensive0.5
T T TT
R0.2
R0.7
R0.3
R R0.7
RR
0.2 0.35 0.15 0.1 0.4 0.15 0.35
0.3 < 0.35 0.1 < 0.15
TEMPORAL ORDER IS DECEPTIVE
(c)
Treatment
Recovery
TF
R
Car
F
R 0.6
Wrong
0.3
![Page 19: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/19.jpg)
R0.5
R0.4
R0.5
(c)
T T
R0.6
0.5 > 0.4
Treatment
Recovery
TF
R
Car
CORRECTDECISION TREE
)|())(|( TRPTdoRP ))(|()|( TdoRPTRP
![Page 20: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/20.jpg)
0.3
0.6
Cheap0.5
Expensive0.5
T T TT
R0.2
R0.7
R0.3
R R0.7
RR
0.2 0.35 0.15 0.1 0.4 0.15 0.35
0.6 = P(R | T,F) P(R | do(T), F)
WHAT WENT WRONG WITH F = Car ?
(c)
Treatment
Recovery
TF
R
Car
F
R 0.6
Wrong
0.3
![Page 21: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/21.jpg)
0.6
Right
0.3
Female0.5
Male0.5
T T TT
R
R0.2
R0.7
R0.3
R R0.7
RR
0.2 0.35 0.15 0.1 0.4 0.15 0.35
T FGender
R
(a)
WHAT WENT RIGHTWITH F = Gender ?
0.6 = P(R | T,F) = P(R | do(T), F)
F
![Page 22: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/22.jpg)
Treatment
Recovery
TF
R
Car
(c)
Compelled Act
Treatment
Recovery
TF
R
Car
(c)
Free Act
THE PERCEPTION OF FREE WILL
(Explained reaction) (Decision)
![Page 23: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/23.jpg)
DISAMBIGUATINGTHE TWO ROLES OF ACT
Whatever evidence an act might provide On motives that cause such acts, Should never be used to help one decide On whether to choose that same act.
(p. 109)
![Page 24: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/24.jpg)
Treatment
Recovery
TF
R
Car
(c)
Compelled Act
Treatment
Recovery
TF
R
Car
(c)
Free Act
THE PARADOXOF FREE WILL
(Explained reaction) (Decision)
God GodPhysics Physics
![Page 25: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/25.jpg)
Free will is an illusion that demands a computational model, to explain: 1. What mental activity generates the illusion of
acting freely? 2. Why/how is such illusion generated from the
activity above?3. How can we survive the distortion generated by
the illusion? Is the distortion harmless? Is the illusion dispensable?
4. What is the computational usefulness of the illusion?
COMPUTATIONAL RESOLUTIONOF THE PARADOX OF FREE WILL
![Page 26: SIMPSON’S PARADOX, ACTIONS, DECISIONS, AND FREE WILL](https://reader035.fdocuments.in/reader035/viewer/2022070504/56816844550346895dde1b99/html5/thumbnails/26.jpg)
CONCLUSIONS
1. Simpson's Paradox is resolved through the calculus of causation.
2. People think causally, not probabilistically 3. Decision trees cannot replace causal models 4. Free-will is a software illusion to be explained
in computational terms.