Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School...

34
Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Transcript of Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School...

Page 1: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Toward Explainable Uncertainty

AlanFernandTomDie/erich

SchoolofElectricalEngineeringandComputerScienceOregonStateUniversity

Page 2: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Goal 1: Uncertainty Aware ML Systems

• Designmachinelearningsystemsthat“knowwhattheyknow”[Li,Li/man,WalshICML’08]•  ProvideguaranteesonpredicNons•  Allowsystemstoabstainand/orproduceambiguouspredicNons

• Achievethisin:•  ClosedWorlds=KnownUnknowns•  OpenWorlds=UnknownUnknowns

• Why?•  SafeandTrustworthyAI•  EndUserAcceptability•  ComputaNonalEfficiency–usemorecomplexmodelifsimplermodelisuncertain

Page 3: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Goal 2: Transparent Uncertainty in ML Systems

• Designmachinelearningsystemsthatcan“explaintheiruncertainty”•  Giveinsightintowhytheyabstainedorproducedambiguousanswer

• Achievethesegoalsin:•  ClosedWorlds=KnownUnknowns•  OpenWorlds=UnknownUnknowns

• Why?•  Basisforfeedbacktolearningsystems•  BasisforinvesNgaNnganomalies•  Mechanismforbuildingtrust

Page 4: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Outline

• ConformalPredicNonforUncertaintyAwareClassificaNon•  Empiricalperformanceinclosedworlds•  Empiricalperformanceinopenworlds•  NoteffecNveinopenworlds→SuggestsintegraNngwithanomalydetecNon

•  ExplanaNonsforAnomalyDetecNon• WhatisananomalyexplanaNon?•  HowtocomputeexplanaNons?•  HowtoevaluateexplanaNons?

Page 5: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Standard ClassificaCon

Predictor

Animal

Human

Human

Classes:{Human,Animal,Hybrid}

UncertaintyinthesystemisnotmadeexplicitinthepredicNons.

Page 6: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Conformal PredicCon [Vovk et al., 2005]

ConformalPredictor

{Animal,Hybrid}

{Human}

{Animal,Human,Hybrid}

Classes:{Human,Animal,Hybrid}

•  Conformalpredictorsoutputsetsoflabels.

•  Labelsetiscorrectifitcontainstruelabel.

MostbasictypeofexplanaNonofuncertainty

Page 7: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Conformal PredicCon: Accuracy

ConformalPredictor

{Animal,Hybrid}

{Human}

{Animal,Human,Hybrid}

Classes:{Human,Animal,Hybrid}

AccuracyConstraint(e.g.95%)

Requirement:mustreturnlabelsetsthatachieveaccuracyconstraint

Page 8: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Conformal PredicCon: Accuracy

ConformalPredictor

{Animal,Human,Hybrid}

{Animal,Human,Hybrid}

{Animal,Human,Hybrid}

Classes:{Human,Animal,Hybrid}

AccuracyConstraint(e.g.95%)

Butwecanget100%accuracybyalwaysreturningalllabels.

Requirement:mustreturnlabelsetsthatachieveaccuracyconstraint

Page 9: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Conformal PredicCon: Ambiguity

ConformalPredictor

{Animal,Hybrid}

{Human}

{Animal,Human,Hybrid}

Classes:{Human,Animal,Hybrid}

AccuracyConstraint(e.g.95%)

Ambiguity= # of predicted labels−1/# of labels−1 ∈[𝟎,𝟏]

Wanttominimizeambiguityofreturnedlabelsets.

ambiguity=0

ambiguity=1

ambiguity=1/2

Page 10: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Conformal PredicCon: Constrained ObjecCve

ConformalPredictor

Classes:{Human,Animal,Hybrid}

AccuracyConstraint(e.g.95%)

Minimize:expectedambiguitySubjectto:accuracyconstraint

{Animal,Hybrid}

{Human}

{Animal,Human,Hybrid}

Page 11: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Inside Conformal PredicCon

YourFavoritePredictor

ConformalPredicNon

Non-conformityScores[human,animal,hybrid]

AccuracyConstraint(e.g.95%)

ConformalpredicNonisawrapperaroundanypredictorthatproduces“non-conformityscores”overtheclassesrelaNvetotrainingdataPredictorqualityinfluencesambiguityofconformalpredicNon.

{Animal,Hybrid}

{Human}

{Animal,Human,Hybrid}

[6,0.5,0.1]

[0.1,0.5,0.15]

[0.01,10,7]

TrainingData

Page 12: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Non-conformity Scores: Nearest Neighbor

NearestNeighbor NonConformity(X,Y)= distance to closest Y/distance to closest non Y 

X

TrainingData

X

𝑌∈{human,animal,hybrid}

Page 13: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Non-conformity Scores: Random Forest

RandomForest NonConformity(X,Y)= # of trees not predicting Y/# of trees 

X

X

animal hybrid human hybrid human

NonConformity(X,animal)=4/5NonConformity(X,human)=3/5NonConformity(X,hybrid)=3/5

Page 14: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Conformal PredicCon: Neural Networks

NeuralNetworks NonConformity(X,Y)= max output for not Y /output for Y 

X

X NonConformity(X,animal)=0.8/0.15=5.3NonConformity(X,human)=0.8/0.05=16NonConformity(X,hybrid)=0.15/0.8=0.19

0.8hybrids

0.15animal

0.05human

Page 15: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Inside Conformal PredicCon

YourFavoritePredictor

ConformalPredicNon

AccuracyConstraint(e.g.95%)TrainingData

CalibraNonData

Non-ConformityScoreHistogram

Non-conformityScores[0.05,0.9,0.1]

AnimalHybridHuman

Areatleast5%ofthecalibraNonscoresweirderthanthislabelwiththisexample?

Page 16: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Inside Conformal PredicCon Non-ConformityScoreHistogram

AnimalHybridHuman

Areatleast5%ofthecalibraNonscoresweirderthanthislabelwiththisexample?

YourFavoritePredictor

ConformalPredicNon

AccuracyConstraint(e.g.95%)TrainingData

CalibraNonData

Non-conformityScores[0.05,0.9,0.1]

Page 17: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Conformal PredicCon: Empirical EvaluaCon

• VeryfewempiricalevaluaNonsofconformalpredicNon•  Rarelylookatambiguity

• MostresultsforNearestNeighbor--otenyieldslargeambiguityinourexperience

• Howdoesambiguityvarywithamountoftrainingdatainclosedworlds?

• HowdoesconformalpredicNonperforminopenworlds?

Page 18: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Closed World: Random Forest Results

• Arrhythmia:452datapoints,13labels,majorclassimbalances

• Cardiotocography:2126instances,10labels,balancedclasses

•  ImageSegmentaNon:2310instances,8labels,balancedclasses

•  Iris:150instances,3labels,balancedclasses

Page 19: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Closed World: Random Forest

Numberofexamples Numberofexamples

AccuracyConstraint=95%Accuracy

Arrhythmia

Ambiguity

Arrhythmia

OtherdatasetsarequalitaNvelysimilar.

Page 20: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Closed World: Random Forest

AccuracyConstraint

Ambiguity

Arrhythmia

AccuracyConstraint

Ambiguity

Cardiotocography

Page 21: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Closed World: Random Forest

AccuracyConstraint

Ambiguity

SegmentaNon

AccuracyConstraint

Ambiguity

Iris

Page 22: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Closed World: Random Forest

• Overall,wesee“idealbehavior”onthesedatasets.• Closeto0ambiguitywithsmallamountofdata.

Page 23: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Closed World: ConvoluConal Network: Cifar 10

Page 24: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Closed World: Deep Net

NumberofCalibraNonExamples

Ambiguity

Cifar10

AccuracyConstraint

Ambiguity

Cifar10

Page 25: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Closed World ObservaCons

• Overall,wesee“idealbehavior”onthesedatasetsforbothrandomforestandconvoluNonalnetwork.

• Closeto0ambiguitywithsmallamountofdata.

• Neuralnetworkveryrarelyabstains(negaNveambiguity)comparedtotherandomforest

Page 26: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Open World: Conformal PredicCon

Predictor

{Animal}

{Human,Animal}

???

TrainingLabels={Human,Animal,Hybrid}

ConformalPredicNon

Wedon’thavealabelfor“monster”!

Onlyreasonableoutputistoabstain(emptyset)

∅

Page 27: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Open World Experiments

•  Feednovelclassestoconformalpredictor

• RandomForest:withheldalabelfromeachtrainingset

• Convolu@onalNetwork:feeditimagesthathavenothingtodowithCifar10

Page 28: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Open World: Random Forest

Numberofexamples

Ambiguity

Arrhythmia

NumberofexamplesAm

biguity

Cardiotocography

Abstains=Knowswhatitknows

Confidentinsinglelabel

Ambiguityforjustnovelclasses

Page 29: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Open World: Random Forest

Numberofexamples

Ambiguity

SegmentaNon

NumberofexamplesAm

biguity

Cardiotocography

Ambiguityforjustnovelclasses

Confidentinsinglelabel

Confidentinsinglelabel

Page 30: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Open World: ConvoluConal Network

Nethackspritesheetimages

Page 31: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Open World: ConvoluConal Network

LearningCurvesvsCalibraNonScores

NumberofcalibraNonexamples

Ambiguity

Cifar10

AmbiguityforjustnovelNethackimages

Confidentinsinglelabel

Page 32: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Open World ObservaCons

•  InallbutonecasetherewaspracNcallynoabstenNon

•  ThetheoryofconformalpredicNondoesnotaddresstheissueofopenworlds

• AppearsthatstandardconformalpredicNononitsownisnotsufficientforopenworlds

Page 33: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Next Steps for Open Worlds

Predictor ConformalPredicNon

AccuracyConstraint

Non-conformityScores

Newalgorithmsfortrainingpredictors.Goal:yieldreliableabstenNonfornovelclasses.

Page 34: Toward Explainable Uncertainty · Toward Explainable Uncertainty Alan Fern and Tom Die/erich School of Electrical Engineering and Computer Science Oregon State University

Next Step for Open Worlds

Predictor ConformalPredicNon

AnomalyDetector

normaldata

AnomalyHandler

anomalousdata

AccuracyConstraint

Non-conformityScores

Howtoselectanomalythreshold?Canweprovideanyguaranteesinopenworlds?


EXPLAINABLE ARTIFICIAL INTELLIGENCE ... › dms_pub › itu-s › opb › journal › S-JOURNAL...EXPLAINABLE ARTIFICIAL INTELLIGENCE: UNDERSTANDING, VISUALIZING AND INTERPRETING DEEP

EXPLAINABLE ARTIFICIAL INTELLIGENCE ... › dms_pub › itu-s › opb › journal › S-JOURNAL...EXPLAINABLE ARTIFICIAL INTELLIGENCE: UNDERSTANDING, VISUALIZING AND INTERPRETING DEEP

Explainable Artificial Intelligence: Objectives ...

Explainable Artificial Intelligence: Objectives ...

Cross-domain Correspondences for Explainable …

Cross-domain Correspondences for Explainable …

Explainable AI

Explainable AI

Explainable Artificial Intelligence for Sarcasm Detection ...

Explainable Artificial Intelligence for Sarcasm Detection ...

Peter Sarlin. Toward robust early-warning models: A horse race, ensembles and model uncertainty

Peter Sarlin. Toward robust early-warning models: A horse race, ensembles and model uncertainty

Explainable AI: the new 42?

Explainable AI: the new 42?

RecSim NG: Toward Principled Uncertainty Modeling for ...

RecSim NG: Toward Principled Uncertainty Modeling for ...

EXPLAINABLE AND EFFICIENT - Stanford University

EXPLAINABLE AND EFFICIENT - Stanford University

Explainable Artificial Intelligencefsvg.math.ntua.gr/RulesLogApps/2019/NickBassiliades-XAI.pdf · 2019-12-27 · Explainable Artificial Intelligence Nick Bassiliades, Professor School

Explainable Artificial Intelligencefsvg.math.ntua.gr/RulesLogApps/2019/NickBassiliades-XAI.pdf · 2019-12-27 · Explainable Artificial Intelligence Nick Bassiliades, Professor School

EXPLAINABLE MACHINE LEARNING

EXPLAINABLE MACHINE LEARNING

Explainable Deep Reinforcement Learning for Portfolio ...

Explainable Deep Reinforcement Learning for Portfolio ...

Explainable Clinical Decision Support from Text

Explainable Clinical Decision Support from Text

Towards Explainable Requirements Engineering Tools · Towards Explainable Requirements Engineering Tools @andivogelsang andreas.vogelsang@tu-berlin.de 07.01.2019 Technische Universität

Towards Explainable Requirements Engineering Tools · Towards Explainable Requirements Engineering Tools @andivogelsang [email protected] 07.01.2019 Technische Universität

Evaluating Explainable AI: Which Algorithmic Explanations ...

Evaluating Explainable AI: Which Algorithmic Explanations ...

Explainable Patterns

Explainable Patterns

Explainable Reinforcement Learning:A Survey

Explainable Reinforcement Learning:A Survey

Explainable and Interpretable Machine Learning

Explainable and Interpretable Machine Learning

TOWARD A GENERALIZED THEORY OF UNCERTAINTY (GTU) Lotfi A. Zadeh Computer Science Division

TOWARD A GENERALIZED THEORY OF UNCERTAINTY (GTU) Lotfi A. Zadeh Computer Science Division

Explainable Neural Computation via Stack Neural Module …openaccess.thecvf.com/content_ECCV_2018/papers/Ronghang_Hu_Explainable... · Explainable Neural Computationvia Stack Neural

Explainable Neural Computation via Stack Neural Module …openaccess.thecvf.com/content_ECCV_2018/papers/Ronghang_Hu_Explainable... · Explainable Neural Computationvia Stack Neural