Metromaps as a Tool for Minimizing Human Interaction with Learning Bayesian Classifiers

.

Learning = Social (software) Robotics

M.Zhanikeev -- [email protected] -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 2/25...

2/25

.

Social Robotics in Knowledge

Rebot

(careless) Input

Human Human

{structure}

(pinpoint) Select

Browse (or use otherwise)

Some Knowledge

(folksonomies, knowledge bases, databases, indexes, ontologies, etc.)

(metromaps )

07 myself+0 "On Context Management Using Metro Maps" SOCA, Matsue, Japan (2014)


3/25

.

Metromap: The Basic Concept

07 myself+0 "On Context Management Using Metro Maps" SOCA, Matsue, Japan (2014)

14 K.Nesbitt+0 "Getting to more abstract places using the metro map metaphor" 8th IV (2004)


4/25

.

A Practical Setting

Accident Something happened at Site A Causes Part A, Part B, Part C, … Human Factors… All Parts Part Z, Part Y, …, Human Manuals, … Rating

Blackswan scenario management platform

Storage, Database

Human judgment

Auto judgement

Report on site

07bmyself+0 "Black Swan Disaster Scenarios" PRMU研 (2014)


5/25

.

Definitions, Objectives, Terminology.Different Viewpoint..

.

classifier is not for finding hidden relations, but for clear separationbetween known and new.Learning Classifier...... a classifier that improves its inference over time based on human feedback

.Metromaps..

.... are used as the graphical interface between humans and robots

• MDC: Multi-Dimensional Classification• MC: Metromap Classifier

• folksonomy: BigData with very frivolous management of metadata


6/25

.

Existing MDC Methods


7/25

.

MDC Basics: Binary Relevance (BR)

• binary: YES or NO for each Y 11

• problem: no relation between classes Y -- this is where metromaps can behelpful

Training Tuples x1 x2 Y1 Y2 Y3

1 0.7 0.4 1 1 0 2 0.6 0.2 1 1 0 3 0.1 0.9 0 0 1 4 0.3 0.1 0 0 0

h1: X → Y1 h2: X → Y2 h3: X → Y3

11 J.Read+3 "Classifier chains for multi-label classification" Machine Learning, SpringerS (2011)


8/25

.

MDC Basics: PairWise Sets (PW)

• relations can be found by creating new classes for all unique pairs in Y 11

• problem: many classes = fuzzy results = low reliability


1 0.7 0.4 1 1 0 2 0.6 0.2 1 1 0 3 0.1 0.9 0 0 1 4 0.3 0.1 0 0 0

h1: X → Z1 h2: X → Z2

Z1 Z2 1 0 0 1 0 0 0 0



9/25

.

MDC Basics: Label Combination (LC)

• basically, the extreme case of PW 11

• the same problem only worse -- there are too many classes!


1 0.7 0.4 1 1 0 2 0.6 0.2 1 1 0 3 0.1 0.9 0 0 1 4 0.3 0.1 0 0 0

h: X → Z

Z 1 0 0 0



10/25

.

MDC Basics: Classifier Chains (CC)• classes are used in sequence 11

• merit: small number of classes -- only the necessary ones are used

• demerit: what is the correct order?


1 0.7 0.4 1 1 0 2 0.6 0.2 1 1 0 3 0.1 0.9 0 0 1

0.3 0.1 0 0 0

h1: X → Y1 h2: Y1 → Y2 h3: Y2 → Y3

h2 h1 h3

4



11/25

.

MDC Basics: Graphical Methodology

• the graphicalmethodologybehind MDC 03

• all about jointprobability and howit is calculated usinggraph theory

03

D.Koller+1 "Probabilistic Graphical

Models: Principles and Techniques"

MIT Press (2009)


12/25

.

Metromap Classifier (MC)


13/25

.

Metromap Classifier (before)• problem: human load is too high!, ex: disaster scenarios 07b

Human judgment

Auto judgement

Folksonomy

07bmyself+0 "Black Swan Disaster Scenarios" PRMU研 (2014)


14/25

.

Metromap Classifier (after)

Human judgment

Auto judgement

Folksonomy


15/25

.

Metromap Classifier : Features

• human role1. build the metromap = relations between classes2. when robot fails, do the work manually3. do the human part (by design) of the work

• robot role1. classify incoming data into YES or NO for question: should human seethis?


16/25

.

Experiment


17/25

.

Experiment : Setup

• IEICE/ken is the source of data -- over 3000 presentations over 2-3 lastyears

• various combinations of title, keywords, abstract• usecase: which presentations should I look at closely?

◦ ... meaning the metromap reflects my personal research interests• Dumb Classifier (DC): one-dimensional yes or no


18/25

.

Metromap Design: The Human


19/25

.

Metromap Classifier: Logic• logic followed by the MC Robot


20/25

.

Results: Title only

0 20 40 60 80 100 120Time sequence

0102030405060708090

Goo

d c

ount

Dumb ClassifierMetromap Classifier(smart) Hits on a timelinetitle


21/25

.

Results: Title + Keywords

0 20 40 60 80 100Time sequence

0

10

20

30

40

50

60

70

80G

ood

cou

ntDumb ClassifierMetromap Classifier(smart) Hits on a timeline

title:keywords


22/25

.

Results: Title + Keywords + Abstract

0 20 40 60 80 100 120Time sequence

0102030405060708090

Goo

d c

ount

Dumb ClassifierMetromap Classifier(smart) Hits on a timelinetitle:keywords:abstract


23/25

.

Wrapup: Not Good Enough

• not perfect: about 30% of wrong decisions◦ FP: robot makes human look at bad stuff (false positive)◦ FN: robot passes on good stuff (false negative)

• future improvements: need a solid logic which avoids FP and FN cases

• note: current naive and MDCs are at most 40-60% reliable -- no help here!


24/25

.

That’s all, thank you ...


25/25

Metromaps as a Tool for Minimizing Human Interaction with Learning Bayesian Classifiers

Technology

Transcript of Metromaps as a Tool for Minimizing Human Interaction with Learning Bayesian Classifiers