Hyperparameter Optimization 101
-
Upload
sigopt -
Category
Technology
-
view
2.499 -
download
1
Transcript of Hyperparameter Optimization 101
![Page 1: Hyperparameter Optimization 101](https://reader031.fdocuments.in/reader031/viewer/2022022203/58727c0c1a28abc7068b55cf/html5/thumbnails/1.jpg)
Hyperparameter Optimization 101Alexandra Johnson
Software Engineer, SigOpt
![Page 2: Hyperparameter Optimization 101](https://reader031.fdocuments.in/reader031/viewer/2022022203/58727c0c1a28abc7068b55cf/html5/thumbnails/2.jpg)
What are Hyperparameters?
![Page 3: Hyperparameter Optimization 101](https://reader031.fdocuments.in/reader031/viewer/2022022203/58727c0c1a28abc7068b55cf/html5/thumbnails/3.jpg)
Hyperparameters affect model performance
![Page 4: Hyperparameter Optimization 101](https://reader031.fdocuments.in/reader031/viewer/2022022203/58727c0c1a28abc7068b55cf/html5/thumbnails/4.jpg)
How Do I Find The Best Hyperparameters?
![Page 5: Hyperparameter Optimization 101](https://reader031.fdocuments.in/reader031/viewer/2022022203/58727c0c1a28abc7068b55cf/html5/thumbnails/5.jpg)
Step 1: Pick an Objective Metric
Classification models Accuracy
Regression models Root MSE
![Page 6: Hyperparameter Optimization 101](https://reader031.fdocuments.in/reader031/viewer/2022022203/58727c0c1a28abc7068b55cf/html5/thumbnails/6.jpg)
Caveat: Cross Validate to Prevent Overfitting
![Page 7: Hyperparameter Optimization 101](https://reader031.fdocuments.in/reader031/viewer/2022022203/58727c0c1a28abc7068b55cf/html5/thumbnails/7.jpg)
Cross Validation
4 5 60 1 2 3 7 8 9
4 5 6 7 8 90 1 2 3data
train validate metric
![Page 8: Hyperparameter Optimization 101](https://reader031.fdocuments.in/reader031/viewer/2022022203/58727c0c1a28abc7068b55cf/html5/thumbnails/8.jpg)
Cross Validation
4 5 60 1 2 3 7 8 9
4 5 6 7 8 90 1 2 3data
train
6 7 91 2 4 5 0 3 8train
7 8 90 2 3 6 1 4 5train
metric
metric
metric
K ti
mes validate
validate
validate
![Page 9: Hyperparameter Optimization 101](https://reader031.fdocuments.in/reader031/viewer/2022022203/58727c0c1a28abc7068b55cf/html5/thumbnails/9.jpg)
Grid Search Random Search Bayesian Optimization
Step 2: Pick an Optimization Strategy
![Page 10: Hyperparameter Optimization 101](https://reader031.fdocuments.in/reader031/viewer/2022022203/58727c0c1a28abc7068b55cf/html5/thumbnails/10.jpg)
Step 3: Evaluate N Times
N Times
![Page 11: Hyperparameter Optimization 101](https://reader031.fdocuments.in/reader031/viewer/2022022203/58727c0c1a28abc7068b55cf/html5/thumbnails/11.jpg)
What is the Best Hyperparameter Optimization Strategy?
![Page 12: Hyperparameter Optimization 101](https://reader031.fdocuments.in/reader031/viewer/2022022203/58727c0c1a28abc7068b55cf/html5/thumbnails/12.jpg)
Primary Consideration: How Good are the “Best” Hyperparameters?
![Page 13: Hyperparameter Optimization 101](https://reader031.fdocuments.in/reader031/viewer/2022022203/58727c0c1a28abc7068b55cf/html5/thumbnails/13.jpg)
“Best Found Value” Distributionsex
perim
ents
accuracy
![Page 14: Hyperparameter Optimization 101](https://reader031.fdocuments.in/reader031/viewer/2022022203/58727c0c1a28abc7068b55cf/html5/thumbnails/14.jpg)
Secondary Consideration: How Much Time Do You Have?
![Page 15: Hyperparameter Optimization 101](https://reader031.fdocuments.in/reader031/viewer/2022022203/58727c0c1a28abc7068b55cf/html5/thumbnails/15.jpg)
Number of Evaluations Required
Grid Search Random Search Bayesian Optimization
2 parameters 100 ?? 20-40
3 parameters 1,000 ?? 30-60
4 parameters 10,000 ?? 40-80
5 parameters 100,000 ?? 50-100
![Page 16: Hyperparameter Optimization 101](https://reader031.fdocuments.in/reader031/viewer/2022022203/58727c0c1a28abc7068b55cf/html5/thumbnails/16.jpg)
SigOptEasy-to-use REST API, R, Java, Python Clients
Ensemble of Bayesian optimization techniques
Free trial, academic discount, we’re hiring!
![Page 17: Hyperparameter Optimization 101](https://reader031.fdocuments.in/reader031/viewer/2022022203/58727c0c1a28abc7068b55cf/html5/thumbnails/17.jpg)
SigOpt Tutorial VideosVersus untuned models:
+315.2% accuracy with TensorFlow CNN
+49.2% accuracy with Xgboost + unsupervised features