Active Learning and Human-in-the-Loop

49
Lukas Biewald

Transcript of Active Learning and Human-in-the-Loop

Page 1: Active Learning and Human-in-the-Loop

Lukas Biewald

Page 2: Active Learning and Human-in-the-Loop

The Effect of Better Algorithms

Naïve Bayes Maximum Entropy SVM0%

5%

10%

15%

20%

25%

Classifier Error Rate

Active Semi-Supervised Learning for Improving Word Alignment(Vamshi ACL ’10)

Real World Data

2

Page 3: Active Learning and Human-in-the-Loop

The Effect of Better Features

Unigrams Bigrams Unigrams+Bigrams0%

5%

10%

15%

20%

25%

30%

Classifier Error Rate

3

Page 4: Active Learning and Human-in-the-Loop

The Effect of More Data

Active Semi-Supervised Learning for Improving Word Alignment(Vamshi ACL ’10)

Real World Data

N 2N 4N0%

2%

4%

6%

8%

10%

12%

14%

Classifier Error Rate

4

Page 5: Active Learning and Human-in-the-Loop

The Effect of Cleaner Data

90% Accurate Data 95% Accurate Data 100% Accurate Data0%

2%

4%

6%

8%

10%

12%

14%

Classifier Error Rate

5

Page 6: Active Learning and Human-in-the-Loop

6

Where Do Data Scientists Spend Their Time?

Source: CrowdFlower Data Science Report 2015

Page 7: Active Learning and Human-in-the-Loop

7

CrowdFlower Data Enrichment Platform

Page 8: Active Learning and Human-in-the-Loop

8

Color Data

Page 9: Active Learning and Human-in-the-Loop

9

Page 10: Active Learning and Human-in-the-Loop

10

Page 11: Active Learning and Human-in-the-Loop

11

Page 12: Active Learning and Human-in-the-Loop

12

Page 13: Active Learning and Human-in-the-Loop

13

Page 14: Active Learning and Human-in-the-Loop

14

Page 15: Active Learning and Human-in-the-Loop

Apple Watch

15

Page 16: Active Learning and Human-in-the-Loop

Apple Watch

16

Page 17: Active Learning and Human-in-the-Loop

Apple Watch

17

Page 18: Active Learning and Human-in-the-Loop

Apple Watch

18

Page 19: Active Learning and Human-in-the-Loop

Collecting the Same Data Over and Over

19

Page 20: Active Learning and Human-in-the-Loop

20

Open Data

Page 21: Active Learning and Human-in-the-Loop

21

Make Your Data Public Setting

Page 22: Active Learning and Human-in-the-Loop

22

Data for Everyone

Page 23: Active Learning and Human-in-the-Loop

23

Data For Everyone Library

Page 24: Active Learning and Human-in-the-Loop

24

Data for Everyone

Page 25: Active Learning and Human-in-the-Loop

25

Data For Everyone

Page 26: Active Learning and Human-in-the-Loop

Open Data API

26

Page 27: Active Learning and Human-in-the-Loop

URL Categorization

27

Page 28: Active Learning and Human-in-the-Loop

Categorize URLs

28

Page 29: Active Learning and Human-in-the-Loop

Record Data

29

Page 30: Active Learning and Human-in-the-Loop

Extracting Names and Titles

30

Page 31: Active Learning and Human-in-the-Loop

Summarization

31

Page 32: Active Learning and Human-in-the-Loop

Is an Image Funny?

32

Page 33: Active Learning and Human-in-the-Loop

Classifying Medical Images

33

Page 34: Active Learning and Human-in-the-Loop

Attributes of People

34

Page 35: Active Learning and Human-in-the-Loop

35

Page 36: Active Learning and Human-in-the-Loop

Kaggle accuracy

Baseline 12-May 13-May 14-May 15-May0%

10%

20%

30%

40%

50%

60%

70%

Accuracy of Best Performing Model

Accu

racy

36

Page 37: Active Learning and Human-in-the-Loop

Kaggle accuracy over time

12-M

ay

14-M

ay

16-M

ay

18-M

ay

20-M

ay

22-M

ay

24-M

ay

26-M

ay

28-M

ay

30-M

ay1-J

un3-J

un5-J

un7-J

un9-J

un

11-Ju

n

13-Ju

n

15-Ju

n

17-Ju

n

19-Ju

n

21-Ju

n

23-Ju

n

25-Ju

n

27-Ju

n

29-Ju

n1-J

ul3-J

ul5-J

ul0%

10%

20%

30%

40%

50%

60%

70%

80%

Accuracy of the Best Performing Model

Accu

racy

37

Page 38: Active Learning and Human-in-the-Loop

Kaggle Participation

12-M

ay

14-M

ay

16-M

ay

18-M

ay

20-M

ay

22-M

ay

24-M

ay

26-M

ay

28-M

ay

30-M

ay1-J

un3-J

un5-J

un7-J

un9-J

un

11-Ju

n

13-Ju

n

15-Ju

n

17-Ju

n

19-Ju

n

21-Ju

n

23-Ju

n

25-Ju

n

27-Ju

n

29-Ju

n0

200

400

600

800

1000

1200

1400

Number of Participating Teams

38

Page 39: Active Learning and Human-in-the-Loop

AIClassifier OutputConfident

Human in the Loop

39

Page 40: Active Learning and Human-in-the-Loop

Human in the Loop

Confident OutputAIClassifier Not Confident Human

Annotation

40

Page 41: Active Learning and Human-in-the-Loop

Human in the Loop

Confident OutputAIClassifier

Active Learning

Not Confident Human Annotation

41

Page 42: Active Learning and Human-in-the-Loop

Human in the Loop

Confident OutputAIClassifier

Active Learning

Not Confident Human Annotation

42

Page 43: Active Learning and Human-in-the-Loop

Active Learning

From hunch.net active learning tutorial ICML ‘0943

Page 44: Active Learning and Human-in-the-Loop

Active Learning Accuracy Improvement

44

Page 45: Active Learning and Human-in-the-Loop

Google Cars Miles Per Disengage

45

Page 46: Active Learning and Human-in-the-Loop

Adaptive Cruise Control

Image source: ExtremeTech

46

Page 47: Active Learning and Human-in-the-Loop

Advanced Chess

Image source: Computer Chess

47

Page 48: Active Learning and Human-in-the-Loop

AlphaGo

48

Page 49: Active Learning and Human-in-the-Loop

Lukas [email protected]@L2K

Thank You

mailto:[email protected]

Closed-Loop Control of Active Sensing Movements Regulates ...

Closed-Loop Control of Active Sensing Movements Regulates ...

This is Active Loop Television

This is Active Loop Television

Active Loop Antenna - Cloudinaryg_center/assets/… · Remote Operation (Optional) As per ANSI C63.4-2014, the use of an active loop antenna for compliance testing is permitted in

Active Loop Antenna - Cloudinaryg_center/assets/… · Remote Operation (Optional) As per ANSI C63.4-2014, the use of an active loop antenna for compliance testing is permitted in

Human-In-The-LoopControland TaskLearningforPneumatically ... · human-in-the-loop control and learning applications. In this study, we propose a biologically inspired multimodal human-in-the-loop

Human-In-The-LoopControland TaskLearningforPneumatically ... · human-in-the-loop control and learning applications. In this study, we propose a biologically inspired multimodal human-in-the-loop

Active Detection via Adaptive Submodularityproceedings.mlr.press/v32/chena14-supp.pdf · Learning object detectors with human in the loop Active learning has been used successfully

Active Detection via Adaptive Submodularityproceedings.mlr.press/v32/chena14-supp.pdf · Learning object detectors with human in the loop Active learning has been used successfully

Comparison of active magnetic loop antennas - DD1US

Comparison of active magnetic loop antennas - DD1US

Cyber Physical System Challenges for Human-in-the-Loop Control · -Humans in the loop, but no active feedback-Sometimes active feedback-Hierarchical control-Considers physiological

Cyber Physical System Challenges for Human-in-the-Loop Control · -Humans in the loop, but no active feedback-Sometimes active feedback-Hierarchical control-Considers physiological

HARDWARE-IN-THE-LOOP STUDY OF A HYBRID ACTIVE FORCE ...

HARDWARE-IN-THE-LOOP STUDY OF A HYBRID ACTIVE FORCE ...

Attributes - Georgia Institute of Technology · 2016-11-02 · Recap: Human Computation •Active Learning: Let the classifier tell you where more annotation is needed. •Human-in-the-loop

Attributes - Georgia Institute of Technology · 2016-11-02 · Recap: Human Computation •Active Learning: Let the classifier tell you where more annotation is needed. •Human-in-the-loop

PA0FRI's ACTIVE LOOP ANTENNA

PA0FRI's ACTIVE LOOP ANTENNA

Synthesis for Human-in-the-Loop Control Systems

Synthesis for Human-in-the-Loop Control Systems

WORK LOOP AND ASHBY CHARTS OF ACTIVE MATERIALS PI: Dr. …

WORK LOOP AND ASHBY CHARTS OF ACTIVE MATERIALS PI: Dr. …

Multi Loop Digital Control for Active Filter and DC/DC ...publications.lib.chalmers.se/records/fulltext/218116/218116.pdf · Multi Loop Digital Control for Active Filter and DC/DC

Multi Loop Digital Control for Active Filter and DC/DC ...publications.lib.chalmers.se/records/fulltext/218116/218116.pdf · Multi Loop Digital Control for Active Filter and DC/DC

RF-PRO-1B Active Magnetic Loop - DX Engineering · PDF fileRF-PRO-1B® Active Magnetic Loop ... Force his “Moebius Strip Shielded Magnetic Loop Antenna” architecture outperforms

RF-PRO-1B Active Magnetic Loop - DX Engineering · PDF fileRF-PRO-1B® Active Magnetic Loop ... Force his “Moebius Strip Shielded Magnetic Loop Antenna” architecture outperforms

Active Region Loop Studies in Oslo

Active Region Loop Studies in Oslo

Small Active Receiving Loop Antennas Wellbrook ALA1530LNP · Small Active Receiving Loop Antennas Wellbrook ALA1530LNP ... portable antenna, ... A city or urban environment will contain

Small Active Receiving Loop Antennas Wellbrook ALA1530LNP · Small Active Receiving Loop Antennas Wellbrook ALA1530LNP ... portable antenna, ... A city or urban environment will contain

Human-in-the-loop and other human centric …...SIMULTECH 2018 Porto Portugal Human-in-the-loop and other human centric computing paradigms for modeling, simulation and decision support

Human-in-the-loop and other human centric …...SIMULTECH 2018 Porto Portugal Human-in-the-loop and other human centric computing paradigms for modeling, simulation and decision support

Synthetic Human-in-the-loop Operations Research ...

Synthetic Human-in-the-loop Operations Research ...

Model 6502 Active Loop Antenna - EMCTEST · model 6502 active loop antenna © emc test systems, l.p. – september 2002 rev e – pn 399058

Model 6502 Active Loop Antenna - EMCTEST · model 6502 active loop antenna © emc test systems, l.p. – september 2002 rev e – pn 399058

Human-in-the-loop Active Covariance Learning for Improving … · 2019-03-19 · Human-in-the-loop Active Covariance Learning for Improving Prediction in Small Data Sets Homayun Afrabandpey

Human-in-the-loop Active Covariance Learning for Improving … · 2019-03-19 · Human-in-the-loop Active Covariance Learning for Improving Prediction in Small Data Sets Homayun Afrabandpey