Post on 18-Jan-2018
description
Lucila Ohno-Machado, MD, PhDmachado@dsg.harvard.edu
Division of Health Sciences and Technology
Harvard Medical SchoolMassachusetts Institute of Technology
Introduction to HST 951Medical Decision Support
Welcome
Objectives• Provide a practical approach to medical decision support• Put a strong emphasis on computer-based applications that
utilize concepts from the fields of artificial intelligence and statistics
• Focus on principled predictive modeling in biomedicine
Audience• Background in quantitative methods is desirable• Undergraduates• Graduate students and post-doctoral fellows (MDs) in medical
informatics
Goals
Model Selection
Data Pre-Processing
ModelConstruction
SystemEvaluation
Decision Support Cycle
Types of Models
What type of support is needed?
• “Exploratory analysis”• “Confirmatory analysis” (gold-standard)
• Clustering• Classification
Inputs
Age 34
2Gender
4
.6
.5
.8
.2
.1
.3.7
.2
“Probabilityof Cancer”
0.6
.4
.2
Mitoses
Neural Networks
Inputs
Coefficients
Output
Independentvariables
Prediction
Age 34
1Gender
4
.5
.8
.40.6
“Probability
of cancer”
p = 1 1 + e -( + cte)
Mitoses
Logistic Regression
CART
Rough Sets
Models
Requirements, Strengths and Weaknesses, Application Examples
• Naïve Bayes• Bayesian Networks• Logistic Regression• Neural Networks• Classification Trees• Rough Set Models• Support Vector Machines• Clustering (Hierarchical and Partitioning)
Evaluation and Comparisons
Classification• Calibration (plots, goodness-of-fit)• Discrimination (ROC areas)• Explanation (variable selection)• Outliers, influential observations (case selection)
Clustering• Distance metrics• Homogeneity• Inter-cluster distance
nl disease
threshold
1.0 3.01.7
FN
TN
FP
TP
“D”
“nl”
nl D
40
4010
10
50 50
50
50
Sensitivity = 40/50 = .8Specificity = 40/50 = .8
ROCcurve
“D”
“nl”
nl D
50
30 0
20
50 50
70
30
“D”
“nl”
nl D
40
4010
10
50 50
50
50
“D”
“nl”
nl D
40
5010
0
50 50
40
60
Sens
itivi
ty
1 - Specificity0 1
1
Thre
shol
d 1.
4Th
r esh
old
1 .7
Thre
shol
d 2.
0
ROC Curves
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Sensitivity
1-Sp
ecifi
city
LRNNRS
Sum
of s
yste
m’s
est
imat
es
Sum of real outcomes0 1
1
overestimation
Calibration Curves
RS Model
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8
Observed
LR Model
0
0.2
0.40.6
0.8
1
0 0.2 0.4 0.6 0.8
Observed
NN Model
0
0.2
0.40.6
0.8
1
0 0.2 0.4 0.6 0.8
Observed
Important Topics
• Decision Analysis• Cost-effectiveness analysis
• Design of Experiments
• Real-World Applications
• Blocking inferences: quantifying anonymity
Examples of Projects
Students have worked in the past in different domains• Diagnosis of
– Coronary Artery Disease– Breast Cancer– Melanoma
• Prognosis in – Interventional Cardiology– Spinal Cord Injury– AIDS– Pregnancy
Data Mining and Predictive Modeling in
(Bio) Medical Databases
0.75
0.77
0.79
0.81
0.83
0.85
0.87
0.89
0.91
1 2 3 4 5 6year
Area
und
er R
OC
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
bala
nce
Logistic Neural Net
We emphasize comparison of different models
0.8 y = e-(X)
LogisticRegression
Modeling the Risk of Major In-Hospital Complications
Following Percutaneous Coronary Interventions
Frederic S. Resnic, Lucila Ohno-Machado, Gavin J. Blake, Jimmy Pavliska, Andrew Selwyn, Jeffrey J. Popma
ACC, 2000
Methods
• Consecutive BWH patients, 1/97 through 2/99 randomly divided into training (n = 1,877) and test (n = 927) sets
• Outcomes: death and combined death, CABG or MI (MACE)
• Validation using independent dataset: 3/99 - 12/99 (n = 1,460)
History Presentation Angiographic Procedural Operator/Lab
age acute MI occluded number lesions annual volumegender primary lesion type multivessel device experiencediabetes rescue (A,B1,B2,C) number stents daily volume iddm CHF class graft lesion stent types (8) lab devicehistory CABG angina class vessel treated closure device experienceBaseline creatinine
Cardiogenic shock
ostial gp 2b3a antagonists
unscheduled case
CRI failed CABG dissection postESRD rotablator
hyperlipidemia atherectomyangiojetmax pre stenosis
Data Source:
max post stenosis
Medical Record
no reflow
Clinician Derived
Dataset: Attributes
Study Population
Cases 2,804 1,460
Women 909 (32.4%) 433 (29.7%)
1/97-2/99 3/99-12/99 Development Set Validation Set
Age > 74yrs 595 (21.2%) 308 (22.5%)
Acute MI 250 (8.9%) 144 (9.9%) Primary 156 (5.6%) 95 (6.5%) Shock 62 (2.2%) 20 (1.4%)
Class 3/4 CHF 176 (6.3%) 80 (5.5%)
gp IIb/IIIa antagonist 1,005 (35.8%) 777 (53.2%)
Death 67 (2.4%) 24 (1.6%) Death, MI, CABG (MACE) 177 (6.3%) 96 (6.6%)
p=.066
p=.340
p=.311
p=.214
p=.058
p=.298
p<.001
p=.110
p=.739
Inputs
Coefficients
Output
Independentvariables
Prediction
Age 34
1Gender
4
.5
.8
.40.6
“Probability
of cancer”
p = 1 1 + e -( + cte )
Mitoses
Logistic Regression
Logistic regression
These models are based on statistics and can only discover linear relationships among the data
Probability of complication
0.6
age
IDDM
CHF class
type
number
procedure
Complications in Coronary Intervention
Logistic and Score Models for Death
OddsRatio p-value
2.51 0.022.12 0.052.06 0.138.41 0.005.93 0.030.57 0.200.53 0.127.53 0.001.70 0.172.78 0.04
Age > 74yrsB2/C LesionAcute MIClass 3/4 CHFLeft main PCIIIb/IIIa UseStent UseCardiogenic ShockUnstable AnginaTachycardicChronic Renal Insuf. 2.58 0.06
Logistic Regression Model
Logistic and Score Models for Death
OddsRatio p-value
2.51 0.022.12 0.052.06 0.138.41 0.005.93 0.030.57 0.200.53 0.127.53 0.001.70 0.172.78 0.04
Age > 74yrsB2/C LesionAcute MIClass 3/4 CHFLeft main PCIIIb/IIIa UseStent UseCardiogenic ShockUnstable AnginaTachycardicChronic Renal Insuf. 2.58 0.06
Logistic Regression Model
beta Riskcoefficient Value
0.921 20.752 10.724 12.129 41.779 3-0.554 -1-0.626 -12.019 40.531 11.022 20.948 2
Prognostic Risk Score Model
Inputs
WeightsIndependentvariables
Dependentvariable
Prediction
Age 34
2Gender
4
.6
.5
.8
.2
.1
.3.7
.2
WeightsHiddenLayer
“Probabilityof Cancer”
0.6
.4
.2
Mitoses
Neural Network
Neural networks
These are mathematical models that can discover non-linear relationships
among the data
Neural networks for predicting death and complications
disease free
death
other complications
age
IDDM
CHF class
type
number
procedure
Death ModelsValidation Set: 1460 Cases
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
0.00 0.20 0.40 0.60 0.80 1.00
1 - Specificity
Sens
itivi
ty LRScoreaNN
ROC AreaLR: 0.840Score: 0.855aNN: 0.835ROC = 0.50
53.6%
12.4%
21.5%
2.2%0
500
1000
1500
2000
2500
3000
0 to 2 3 to 4 5 to 6 7 to 8 9 to 10 >10
Risk Score Category
Num
ber o
f Cas
es
0%
10%
20%
30%
40%
50%
60%
Risk Score of Death: BWH ExperienceUnadjusted Overall Mortality Rate = 2.1%
Mortality Risk
Number of Cases
62%
26%
7.6%2.9% 1.6% 1.3%0.4% 1.4%
CART
Regression TreesThese are models that partition the data using
one variable at a time, and can model non-linear relationships among data
Diagnosis of Melanoma(Michael Binder, Greg Sharp et al., 1999)
Dermatoscopy
Dermatoscopy 0- TEST: null VALUE: null Num Cases: 700.0 Num Dsrd: 241.0 2- TEST: breath VALUE: 1 Num Cases: 75.0 Num Dsrd: 1.0 ********PRUNED!!! ********PRUNED!!! 1- TEST: breath VALUE: 0 Num Cases: 625.0 Num Dsrd: 240.0 4- TEST: CWtender VALUE: 1 Num Cases: 11.0 Num Dsrd: .0 3- TEST: CWtender VALUE: 0 Num Cases: 614.0 Num Dsrd: 240.0 8- TEST: age VALUE: >32 Num Cases: 611.0 Num Dsrd: 240.0 10- TEST: Duration VALUE: >72 Num Cases: 3.0 Num Dsrd: .0 9- TEST: Duration VALUE: <=72 Num Cases: 608.0 Num Dsrd: 240.0 12- TEST: Duration VALUE: >48 Num Cases: 2.0 Num Dsrd: 2.0 11- TEST: Duration VALUE: <=48 Num Cases: 606.0 Num Dsrd: 238.0 14 - TEST: prevang VALUE: 1 Num Cases: 340.0 Num Dsrd: 92.0 18 - TEST: Epis VALUE: 1 Num Cases: 8.0 Num Dsrd: .0 17 - TEST: Epis VALUE: 0 Num Cases: 332.0 Num Dsrd: 92.0 22- TEST: Worsening VALUE: >72 Num Cases: 6.0 Num Dsrd: .0 21- TEST: Worsening VALUE: <=72 Num Cases: 326.0 Num Dsrd: 92.0 28 - TEST: Duration VALUE: >36 Num Cases: 3.0 Num Dsrd: .0 27- TEST: Duration VALUE: <=36 Num Cases: 323.0 Num Dsrd: 92.0 36 - TEST: Worsening VALUE: >28 Num Cases: 3.0 Num Dsrd: 2.0 35 - TEST: Worsening VALUE: <=28 Num Cases: 320.0 Num Dsrd: 90.0 44 - TEST: age VALUE: >55 Num Cases: 240.0 Num Dsrd: 81.0 52 - TEST: Worsening VALUE: >0 Num Cases: 238.0 Num Dsrd: 81.0 64 - TEST: OldMI VALUE: 1 Num Cases: 49.0 Num Dsrd: 9.0 74 - TEST: Smokes VALUE: 0 Num Cases: 37.0 Num Dsrd: 9.0 86 - TEST: age VALUE: >65 Num Cases: 30.0 Num Dsrd: 5.0 ********PRUNED!!! ********PRUNED!!! 85 - TEST: age VALUE: <=65 Num Cases: 7.0 Num Dsrd: 4 .0 98 - TEST: Worsening VALUE: >2 Num Cases: 5.0 Num Dsrd: 2.0 97 - TEST: Worsening VALUE: <=2 Num Cases: 2.0 Num Dsrd: 2.0 73 - TEST: Smokes VALUE: 1 Num Cases: 12.0 Num Dsrd: .0 63- TEST: OldMI VALUE: 0 Num Cases: 189.0 Num Dsrd: 72 .0 72 - TEST: Nausea VALUE: 0 Num Cases: 165.0 Num Dsrd: 57. 0 84 - TEST: Duration VALUE: >16 Num Cases: 3.0 Num Dsrd: 2.0 83 - TEST: Duration VALUE: <=16 Num Cases: 162.0 Num Dsrd: 55.0 ********PRUNED!!! ********PRUNED!!! 71 - TEST: Nausea VALUE: 1 Num Cases: 24.0 Num Dsr d: 15.0 82 - TEST: Back VALUE: 0 Num Cases: 21.0 Num Dsrd: 15.0 94 - TEST: post VALUE: 1 Num Cases: 1.0 Num Dsrd: .0 93 - TEST: post VALUE: 0 Num Cases: 20.0 Num Dsrd: 15.0 81 - TEST: Back VALUE: 1 Num Cases: 3.0 Num Dsrd: .0 51 - TEST: Worsening VALUE: <=0 Num Cases: 2.0 Num Dsrd: .0 43 - TEST: age VALUE: <=55 Num Cases: 80.0 Num Dsrd: 9.0 50 - TEST: Worsening VALUE: >1 Num Cases: 68.0 Num Dsrd: 5.0 ********PRUNED!!! ********PRUNED!!! ********PRUNED!!! ********PRUNED!!! ********PRUNED!!! ********PRUNED!!! ********PRUN ED!!! ********PRUNED!!! 49 - TEST: Worsening VALUE: <=1 Num Cases: 12.0 Num Dsrd: 4.0 60 - TEST: age VALUE: >47 Num Cases: 10.0 Num Dsrd: 2.0 68 - TEST: OldMI VALUE: 1 Num Cases: 1.0 Num Dsrd: 1.0 67- TEST: OldMI VALUE: 0 Num Cases: 9.0 Num Dsrd: 1.0 ********PRUNED!!! ********PRUNED!!! 59 - TEST: age VALUE: <=47 Num Cases: 2.0 Num Dsrd: 2.0 13 - TEST: prevang VALUE: 0 Num Cases: 266.0 Num Dsrd: 146.0 16- TEST: Duration VALUE: >0 Num Cases: 259.0 Num Dsrd: 146.0 20- TEST: post VALUE: 1 Num Cases: 13.0 Num Dsrd: 2.0 26 - TEST: Diabetes VALUE: 1 Num Cases: 1.0 Num Dsrd: 1.0 25 - TEST: Diabetes VALUE: 0 Num Cases: 12.0 Num Dsrd: 1.0 ********PRUNED!!! ********PRUNED!!! 19 - TEST: post VALUE: 0 Num Cases: 246.0 Num Dsrd: 144.0 24 - TEST: Nausea VALUE: 0 Num Cases: 202.0 Num Dsrd: 105.0 32 - TEST: OldMI VALUE: 1 Num Cases: 13.0 Num Dsrd: 1.0 42 - TEST: BP VALUE: 1 Num Cases: 1.0 Num Dsrd: 1.0 41 - TEST: BP VALUE: 0 Num Cases: 12.0 Num Dsrd: .0 31 - TEST: OldMI VALUE: 0 Num Cases: 189.0 Num Dsrd: 104.0 40 - TEST: age VALUE: >37 Num Cases: 184.0 Num Dsrd: 103.0 48 - TEST: Epis VALUE: 1 Num Cases: 8.0 Num Dsrd: 2.0 58 - TEST: Duration VALUE: >8 Num Cases: 2.0 Num Dsrd: 2.0 57- TEST: Duration VALUE: <=8 Num Cases: 6.0 Num Dsrd: .0 47 - TEST: Epis VALUE: 0 Num Cases: 176.0 Num Dsrd: 101.0 56 - TEST: Duration VALUE: >15 Num Cases: 2.0 Num Dsrd: .0 55 - TEST: Duration VALUE: <=15 Num Cases: 174.0 Num Dsrd: 101 .0 66- TEST: Lipids VALUE: 1 Num Cases: 1.0 Num Dsrd: 1.0 65 - TEST: Lipids VALUE: 0 Num Cases: 173.0 Num Dsrd: 100 .0 76 - TEST: Sweating VALUE: 0 Num Cases: 73.0 Num Dsr d: 32.0 ********PRUNED!!! ********PRUNED!!! 75 - TEST: Sweating VALUE: 1 Num Cases: 100.0 Num Dsrd: 68.0 88 - TEST: Duration VALUE: >8 Num Cases: 7.0 Nu m Dsrd: 2.0 104 - TEST: Rarm VALUE: 0 Num Cases: 5.0 Num Dsrd: .0 103- TEST: Rarm VALUE: 1 Num Cases: 2.0 Num Dsrd: 2.0 87 - TEST: Duration VALUE: <=8 Num Cases: 93.0 Num Dsrd: 66.0 ********PRUNED!!! ********PRUNED!!! 39- TEST: age VALUE: <=37 Num Cases: 5.0 Num Dsrd: 1.0 23 - TEST: Nausea VALUE: 1 Num Cases: 44.0 Num Dsrd: 39.0 30 - TEST: age VALUE: >47 Num Cases: 41.0 Num Dsrd: 39.0 38 - TEST: Duration VALUE: >7 Num Cases: 7.0 Num Dsrd: 5.0 46 - TEST: Larm VALUE: 0 Num Cases: 1.0 Num Dsrd: .0 45 - TEST: Larm VALUE: 1 Num Cases: 6.0 Num Dsrd: 5.0 54 - TEST: Rarm VALUE: 0 Num Cases: 5.0 Num Dsrd: 5.0 53 - TEST: Rarm VALUE: 1 Num Cases: 1.0 Num Dsrd: .0 37 - TEST: Duration VALUE: <=7 Num Cases: 34.0 Num Dsrd: 34.0 29- TEST: age VALUE: <=47 Num Cases: 3.0 Num Dsrd: .0 15 - TEST: Duration VALUE: <=0 Num Cases: 7.0 Num Dsrd: .0 7- TEST: age VALUE: <=32 Num Cases: 3.0 Num Dsrd: .0
asymmetry
border
detail
“benigh”
color
“malig”
borderdetail
< 2
R
< 2
A
detail
Y
“malig”
> 10
“benign”
detail
<2
Y
Performance using ABCD rule
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1ROC CURVES ABCD RULE
1 - SPECIFICITY
SE
NS
ITIV
ITY
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1ROC CURVES OVERALL DIAGNOSIS
1 - SPECIFICITY
SE
NS
ITIV
ITY
Rough Sets
Rough Sets
These are mathematical models that derive rules for grouping cases based
on boolean logic
Multiple subsamples of a large table are created and combined for rule extraction
# Sex T3 FTI TT4 TSH Med Status
1 F 1.05 49.9 48 3.8 N OK
2 M 1.10 50.1 49 4.7 Y sick
3 F 1.3 170 51 5.8 N OK
4 M 1.4 175 200 0.4 N sick
# Sex T3 FTI TT4 TSH Med Status
1 F 1.05 49.9 48 3.8 N OK
2 M 1.10 50.1 49 4.7 Y sick
3 F 1.3 170 51 5.8 N OK
4 M 1.4 175 200 0.4 N sick
# Sex T3 FTI TT4 TSH Med Status
1 F 1.05 49.9 48 3.8 N OK
2 M 1.10 50.1 49 4.7 Y sick
3 F 1.3 170 51 5.8 N OK
4 M 1.4 175 200 0.4 N sick
# Sex T3 FTI TT4 TSH Med Status
1 F 1.05 49.9 48 3.8 N OK
2 M 1.10 50.1 49 4.7 Y sick
3 F 1.3 170 51 5.8 N OK
4 M 1.4 175 200 0.4 N sick
# Sex T3 FTI TT4 TSH Med Status
1 F 1.05 49.9 48 3.8 N OK
2 M 1.10 50.1 49 4.7 Y sick
3 F 1.3 170 51 5.8 N OK
4 M 1.4 175 200 0.4 N sick
# Sex T3 FTI TT4 TSH Med Status
1 F 1.05 49.9 48 3.8 N OK
2 M 1.10 50.1 49 4.7 Y sick
3 F 1.3 170 51 5.8 N OK
4 M 1.4 175 200 0.4 N sick
# Sex T3 FTI TT4 TSH Med Status
1 F 1.05 49.9 48 3.8 N OK
2 M 1.10 50.1 49 4.7 Y sick
3 F 1.3 170 51 5.8 N OK
4 M 1.4 175 200 0.4 N sick
# Sex T3 FTI TT4 TSH Med Status
1 F 1.05 49.9 48 3.8 N OK
2 M 1.10 50.1 49 4.7 Y sick
3 F 1.3 170 51 5.8 N OK
4 M 1.4 175 200 0.4 N sick
# Sex T3 FTI TT4 TSH Med Status
1 F 1.05 49.9 48 3.8 N OK
2 M 1.10 50.1 49 4.7 Y sick
3 F 1.3 170 51 5.8 N OK
4 M 1.4 175 200 0.4 N sick
# Sex T3 FTI TT4 TSH Med Status
1 F 1.05 49.9 48 3.8 N OK
2 M 1.10 50.1 49 4.7 Y sick
3 F 1.3 170 51 5.8 N OK
4 M 1.4 175 200 0.4 N sick
# Sex T3 FTI TT4 TSH Med Status
1 F 1.05 49.9 48 3.8 N OK
2 M 1.10 50.1 49 4.7 Y sick
3 F 1.3 170 51 5.8 N OK
4 M 1.4 175 200 0.4 N sick
# Sex T3 FTI TT4 TSH Med Status
1 F 1.05 49.9 48 3.8 N OK
2 M 1.10 50.1 49 4.7 Y sick
3 F 1.3 170 51 5.8 N OK
4 M 1.4 175 200 0.4 N sick
# Sex T3 FTI TT4 TSH Med Status
1 F 1.05 49.9 48 3.8 N OK
2 M 1.10 50.1 49 4.7 Y sick
3 F 1.3 170 51 5.8 N OK
4 M 1.4 175 200 0.4 N sick
# Sex T3 FTI TT4 TSH Med Status
1 F 1.05 49.9 48 3.8 N OK
2 M 1.10 50.1 49 4.7 Y sick
3 F 1.3 170 51 5.8 N OK
4 M 1.4 175 200 0.4 N sick
# Sex T3 FTI TT4 TSH Med Status
1 F 1.05 49.9 48 3.8 N OK
2 M 1.10 50.1 49 4.7 Y sick
3 F 1.3 170 51 5.8 N OK
4 M 1.4 175 200 0.4 N sick
# Sex T3 FTI TT4 TSH Med Status
1 F 1.05 49.9 48 3.8 N OK
2 M 1.10 50.1 49 4.7 Y sick
3 F 1.3 170 51 5.8 N OK
4 M 1.4 175 200 0.4 N sick
# Sex T3 FTI TT4 TSH Med Status
1 F 1.05 49.9 48 3.8 N OK
2 M 1.10 50.1 49 4.7 Y sick
3 F 1.3 170 51 5.8 N OK
4 M 1.4 175 200 0.4 N sick
If [(number>2) and …]
then
Complication = true
Rules
Comparison of Practical Prediction Models for Ambulation Following
Spinal Cord Injury(Rowland et al, 1998)
Study Population Spinal Cord Injury Model Systems of Care Database
• Admitted to one of 24 federally funded designated regional SCI care systems
• 17,861 patients who sustained a spinal cord injury between 1973 and 1997
• 1755 patients had data for LEMS scores, 1993 to 1997• 1138 had complete data for variables of interest
SCI Mortality NN DesignInput & Output
Admission Info (9 items)
system daysinjury daysagegenderracial/ethnic grouplevel of neurologic fxnASIA impairment indexUEMSLEMS
Ambulation (1 item)
yesno
Results: ROC Curve Area
Model ROC Curve Area Standard Error
Logistic Regression 0.925 0.016
Neural Network 0.923 0.015
Rough Set 0.914 0.016
Results: ROC Curves
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Sensitivity
1-Sp
ecifi
city
LR
NNRS
Other methods
Support Vector Machines, multiple variations of the nearest neighbor
algorithm, etc.
Heart Attack Alert Program(Wang et al., 2001)
Cox’s Models for Prediction
time (years)
Genetic Algorithms
Search mechanism
• Used for variable selection (model construction)
• Case selection (regression diagnostics)
• Multidisorder diagnosis
People
• Brigham and Women’s Hospital • Children’s Hospital• EECS MIT• School of Public Health• Partners Information Systems
Administrivia
Grading based on• 30% homeworks (almost every week)/participation• 30% midterm, open notes• 40% project (no final exam)
Lectures on the WWW for referenceHandouts with Prof. Szolovits’ assistant at NE-43 r416
Questions/Suggestions
• machado@dsg.harvard.edu• isaac_kohane@harvard.edu• psz@mit.edu