Immunosuppression & Immune Tolerance Report 3HMT - Carbungco & Enverga
Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4....
Transcript of Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4....
![Page 1: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/1.jpg)
Ensemble Learning and the Heritage Health Prize
Jonathan Stroud, Igii Enverga, Tiffany Silverstein,Brian Song, and Taylor Rogers
iCAMP 2012University of California, Irvine
Advisors: Dr. Max Welling, Dr. Alexander Ihler, Sungjin Ahn, and Qiang Liu
December 4, 2013
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 2: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/2.jpg)
Outline
I Heritage Health PrizeI DataI Evaluation
I Our ApproachI MotivationI ModelsI Blending
I Results
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 3: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/3.jpg)
The Heritage Health Prize
I Goal: Identify patients who will be admitted to a hospitalwithin the next year, using historical claims data.[1]
I 1,659 teams
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 4: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/4.jpg)
Purpose
I Reduce cost of unnecessary hospital admissions per yearI Identify at-risk patients earlier
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 5: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/5.jpg)
Evaluation
Root Mean Squared Logarithmic Error (RMSLE)
ε =
√√√√1
n
n∑i
[log(pi + 1) − log(ai + 1)]2
Threshold: ε ≤ .4
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 6: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/6.jpg)
Our Approach
I Motivation
I Individual Models
I Optimized Ensemble
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 7: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/7.jpg)
Blending
Blend several predictors to create a more accurate predictorMotivated by solution to the Netflix Prize [3]
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 8: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/8.jpg)
Prediction Models
I Preprocessing: Feature Selection
I K-Nearest Neighbors
I Random Forests
I Gradient Boosting Machines
I Logistic Regression
I Support Vector Regression
I Neural Networks
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 9: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/9.jpg)
Feature Selection
I Used Market Makers method [2]
I Reduced each patient to vector of 139 features
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 10: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/10.jpg)
Kitchen Sink
I Generate large number of featuresI Need method for eliminating useless features
I Compare distributions of feature in test and training setsI Basis pursuit
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 11: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/11.jpg)
Basis Pursuit
I Find best generated features (reduces error the most)
I Add to feature set
I Repeat until enough useful features are found
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 12: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/12.jpg)
K-Nearest Neighbors
I Weighted average of closest neighbors
I Very slow
Neighbors: k = 1000RMSLE: 0.475197
(996th place)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 13: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/13.jpg)
Decision Trees
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 14: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/14.jpg)
Random Forests
RMSLE: 0.464918(469th place)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 15: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/15.jpg)
Gradient Boosting Machines
Trees = 8000Shrinkage = 0.002
Depth = 7Minimum Observations = 100
RMSLE: 0.462998(325th place)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 16: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/16.jpg)
Logistic Regression
I Optimized with gradient descent
RMSLE: 0.466726(580th place)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 17: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/17.jpg)
Support Vector Regression
ε = .02RMSLE: 0.467152
(629th place)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 18: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/18.jpg)
Neural Networks
Number of hidden neurons = 7Number of cycles = 3000
RMSLE: 0.465705(511th place)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 19: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/19.jpg)
Individual Predictors (Summary)
I K-Nearest Neighbors 0.475197 (996th place)
I Support Vector Regression 0.467152 (629th place)
I Logistic Regression 0.466726 (580th place)
I Neural Networks 0.465705 (511th place)
I Random Forests 0.464918 (469th place)
I Gradient Boosting Machines 0.462998 (325th place)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 20: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/20.jpg)
The Blending Equation
X as a combination of predictors
X = Xw
Minimize cost function
C =1
n
N∑i=1
(Yi − Xi )2
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 21: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/21.jpg)
The Blending Equation
Optimizing predictors’ weights
wc = (Y TX )(XTX )−1
Y TX =∑i
X 2ic +
n∑i=1
(Xi − Yi )2 −
n∑i=1
Y 2i
Y TX =∑i
X 2ic + nε20 − nε2c
Y = Actual values (Unknown)X = Our predictions (Known)
ε = Feedback (Known)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 22: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/22.jpg)
Blending Results
RMSLE: 0.46143288th place on the final milestone leaderboard
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 23: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/23.jpg)
Results(Summary)
I K-Nearest Neighbors 0.475197 (996th place)
I Support Vector Regression 0.467152 (629th place)
I Logistic Regression 0.466726 (580th place)
I Neural Networks 0.465705 (511th place)
I Random Forests 0.464918 (469th place)
I Gradient Boosting Machines 0.462998 (325th place)
I Blending 0.461432 (88th place)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 24: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/24.jpg)
Observations and Problems
I Fewer repeated classifiers worked better
I Overfitting based on feedback
I Test and Training data were inconsistent
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 25: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/25.jpg)
Future Work
I Optimizing Blending Equation with Regularization Constant
wc = (Y TX )(XTX + λI )−1
I More predictors
I Adjust for changes over time
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 26: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/26.jpg)
Questions
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 27: Ensemble Learning and the Heritage Health Prizerickl/courses/ics-h197/2013-fq-h197/... · 2016. 4. 3. · RMSLE: 0.462998 (325th place) Stroud, Enverga, Silverstein, Song, and Rogers](https://reader035.fdocuments.in/reader035/viewer/2022071105/5fdfdda7b2232e3c031cb9fd/html5/thumbnails/27.jpg)
References
Heritage provider network health prize, 2012.http://www.heritagehealthprize.com/c/hhp.
David Vogel Phil Brierley and Randy Axelrod.Market makers - milestone 1 description.September 2011.
Andreas Toscher and Michael Jahrer.The bigchaos solution to the netflix grand prize.September 2009.
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning