Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods...

29
CS639: Data Management for Data Science Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1

Transcript of Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods...

Page 1: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

CS639:DataManagementfor

DataScienceLecture17:EvaluatingMachineLearningMethods

TheodorosRekatsinas

1

Page 2: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Announcements

1. Youcanseeyourexamsduringofficehours

2. Homeworkwillbeannouncedlaterthisweek;wewillhaveonlytwomoreprojectsnotthree

2

Page 3: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Today

1. EvaluatingMLmodels

3

Page 4: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Howcanwegetanunbiasedestimateoftheaccuracyofalearnedmodel?

Page 5: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Testsets

• Howcanwegetanunbiasedestimateoftheaccuracyofalearnedmodel?

• Whenlearningamodel,youshouldpretendthatyoudon’thavethetestdatayet

• Ifthetest-setlabelsinfluencethelearnedmodelinanyway,accuracyestimateswillbebiased

5

Page 6: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Learningcurves

• Howdoestheaccuracyofalearningmethodchangeasafunctionofthetraining-setsize?

• Thiscanbeassessedbylearningcurves

Page 7: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Learningcurves

• Givenatraining/testsetpartition• Foreachsamplesizesonthelearningcurve

• (optionally)repeatntimes• Randomlyselectsinstancesfromthetrainingset• Learnthemodel• Evaluatethemodelonthetestsettodetermineaccuracya• Plot(s,a)

Page 8: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Validation(tuning)sets

• Supposewewantunbiasedestimatesofaccuracyduringthelearningprocess(e.g.tochoosethebestlevelofdecision-treepruning)?

Partition training data into separate training/validation sets

Page 9: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Limitationsofusingasingletraining/testpartition• Wemaynothaveenoughdatatomakesufficientlylargetrainingandtestsets

• Alargertestsetgivesusmorereliableestimatesofaccuracy(i.e.,alowervarianceestimate)

• But… alargertrainingsetwillbemorerepresentativeofhowmuchdataweactuallyhaveforlearningprocess

• Asingletrainingsetdoesnottellushowsensitiveaccuracyistoaparticulartrainingsample

Page 10: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Randomresampling

• Wecanaddressthesecondissuebyrepeatedlyrandomlypartitioningtheavailabledataintotrainingandsetsets.

Page 11: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Stratifiedsampling

• Whenrandomlyselectingtrainingorvalidationsets,wemaywanttoensurethatclassproportionsaremaintainedineachselectedset

Page 12: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Crossvalidation

Page 13: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Crossvalidationexample

• Supposewehave100instances,andwewanttoestimateaccuracywithcrossvalidation

Page 14: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Crossvalidationexample

• 10-foldcrossvalidationiscommon,butsmallervaluesofnareoftenusedwhenlearningtakesalotoftime

• Inleave-one-outcrossvalidation,n=#instances• Instratifiedcrossvalidation,stratifiedsamplingisusedwhenpartitioningthedata

• CVmakesefficientuseoftheavailabledatafortesting• Notethatwheneverweusemultipletrainingsets,asinCVandrandomresampling,weareevaluatingalearningmethodasopposedtoanindividuallearnedmodel

Page 15: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Internalcrossvalidation

• Insteadofasinglevalidationset,wecanusecross-validationwithinatrainingsettoselectamodel(e.g.tochoosethebestlevelofdecision-treepruning)

Page 16: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Confusionmatrices

• Howcanweunderstandwhattypesofmistakesalearnedmodelmakes?

Page 17: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Confusionmatrixfor2-classproblems

Page 18: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Isaccuracyanadequatemeasureofpredictiveperformance?

Page 19: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Otheraccuracymetrics

Page 20: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

ROCcurves

Page 21: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

ROCcurveexample

Page 22: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

ROCcurvesandmisclassificationcosts

Page 23: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

AlgorithmforcreatinganROCcurve

Page 24: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Otheraccuracymetrics

Page 25: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

Precision/recallcurves

Page 26: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

ToAvoidCross-ValidationPitfalls

Page 27: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

ToAvoidCross-ValidationPitfalls

Page 28: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

ToAvoidCross-ValidationPitfalls

Page 29: Lecture 18 Eval ML - GitHub Pages · 2019-05-08 · Lecture 17: Evaluating Machine Learning Methods Theodoros Rekatsinas 1. Announcements 1. You can see your exams during office hours

AblationStudies


PC GAMES Media Technologies Theodoros Nikitopoulos Chamilothoris.

PC GAMES Media Technologies Theodoros Nikitopoulos Chamilothoris.

Oracle Technology Overview Theodoros Demosthenous Principal Consultant theodoros.demosthenous@oracle.com.

Oracle Technology Overview Theodoros Demosthenous Principal Consultant [email protected].

Lecture 1 Intro - GitHub PagesLecture 1: Intro to Data Science and Course Overview Theodoros Rekatsinas 1 2 3 Big science is data driven. 4 Increasingly many companies see themselves

Lecture 1 Intro - GitHub PagesLecture 1: Intro to Data Science and Course Overview Theodoros Rekatsinas 1 2 3 Big science is data driven. 4 Increasingly many companies see themselves

HawkesTopic: A Joint Model for Network Inference and Topic Modeling from Text-Based Cascades Xinran He 1, Theodoros Rekatsinas 2, James Foulds 3, Lise.

HawkesTopic: A Joint Model for Network Inference and Topic Modeling from Text-Based Cascades Xinran He 1, Theodoros Rekatsinas 2, James Foulds 3, Lise.

TEISIAS AND THEODOROS: EAST BOIOTIAN POTTERS · TEISIAS AND THEODOROS: EAST BOIOTIAN POTTERS ... Dionysian Imagery in Archaic Greek Art, ... The Local Scripts of Archaic Greece, ...

TEISIAS AND THEODOROS: EAST BOIOTIAN POTTERS · TEISIAS AND THEODOROS: EAST BOIOTIAN POTTERS ... Dionysian Imagery in Archaic Greek Art, ... The Local Scripts of Archaic Greece, ...

THEODOROS KITSOS

THEODOROS KITSOS

1 Advanced Databases Data Mining Dr Theodoros Manavis tmanavis@ist.edu.gr.

1 Advanced Databases Data Mining Dr Theodoros Manavis [email protected].

Photopets - Theodoros

Photopets - Theodoros

Lecture 2 Stats - GitHub Pages · Lecture 2: Statistical Inference and Exploratory Data Analysis Theodoros Rekatsinas 1. 2 Announcements • Waiting list: you receive invitations

Lecture 2 Stats - GitHub Pages · Lecture 2: Statistical Inference and Exploratory Data Analysis Theodoros Rekatsinas 1. 2 Announcements • Waiting list: you receive invitations

Theodoros Chailis Sample Portfolio

Theodoros Chailis Sample Portfolio

Lecture 25 EDA - GitHub Pages · Lecture 25: EDA Theodoros Rekatsinas 1. Data Visualizations Today 2. Data Visualizations Today 3. 4 Standard Data Visualization Recipe: 1.Loaddataset

Lecture 25 EDA - GitHub Pages · Lecture 25: EDA Theodoros Rekatsinas 1. Data Visualizations Today 2. Data Visualizations Today 3. 4 Standard Data Visualization Recipe: 1.Loaddataset

Theodoros Chiotis is a poet and literary theorist. He ...

Theodoros Chiotis is a poet and literary theorist. He ...

Agios Theodoros Stoudiths

Agios Theodoros Stoudiths

Theodoros - Photopets

Theodoros - Photopets

Lecture 16 ML DT - GitHub Pages · Lecture 16: Intro to ML and Decision Trees Theodoros Rekatsinas (lecture by AnkurGoswamimany slides from David Sontag) 1. Today’s Lecture 1. Intro

Lecture 16 ML DT - GitHub Pages · Lecture 16: Intro to ML and Decision Trees Theodoros Rekatsinas (lecture by AnkurGoswamimany slides from David Sontag) 1. Today’s Lecture 1. Intro

COM211 Communications and Networks CDA College Theodoros Christophides Email: theo_christopher@hotmail.com

COM211 Communications and Networks CDA College Theodoros Christophides Email: [email protected]

1 Advanced Database Dr Theodoros Manavis tmanavis@ist.edu.gr.

1 Advanced Database Dr Theodoros Manavis [email protected].

Lecture 17 ML supp - GitHub Pages · Lecture 17: Linear Classifiers and Support Vector Machines Theodoros Rekatsinas (lecture by AnkurGoswamimany slides from David Sontag) 1. Today’s

Lecture 17 ML supp - GitHub Pages · Lecture 17: Linear Classifiers and Support Vector Machines Theodoros Rekatsinas (lecture by AnkurGoswamimany slides from David Sontag) 1. Today’s

Theodoros Arvanitis IoT Midlands Presentation Oct 2014

Theodoros Arvanitis IoT Midlands Presentation Oct 2014

Theodoros Axiotis- Argo

Theodoros Axiotis- Argo