Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu...

30
Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/ smartselection IUI 2009

TAGS:

Transcript of Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu...

Page 1: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Learning to Generalize for Complex Selection Tasks

Alan RitterUniversity of Washington

Sumit BasuMicrosoft Research

research.microsoft.com/~sumitb/smartselection

IUI 2009

Page 2: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Outline

1. Smart Selection2. Learning to Generalize3. User Study4. Conclusions

Page 3: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Multiple Selection

• Files• HTML list boxes• E-mail• PowerPoint objects• Spreadsheets• Etc…

• Complex selection tasks currently require programming knowledge– Unix Shell– Regular Expressions

Page 4: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Our Task: File Selection

Page 5: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Tedious Selection Tasks

• Files do not group together by sorting– Substring of file name (e.g. “copy” or “backup”)

• Users forced to click on a large number of files

Page 6: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Smart Selection

Selection Classifier

LabelClassify

Page 7: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Related Work

• Text editing– LAPIS (Miller and Myers IUI 02)

• DOM extraction– REFORM (Toomim et. al.CHI 09)– KARMA (Tuchinda et. al. IUI 08)

• Other Domains– Image regions: Crayons (Fails and Olsen IUI 03)– Image search: CueFlik (Fogarty et. al. CHI 08)

Page 8: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Selection Classifier

LabelClassify

one session

Few Labels Available

Page 9: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Selection Classifier

LabelClassify

many users, many sessions

one session

How to use historical tasks?

Page 10: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Our Contributions

1. Make use of many people’s historical data– Learning to Generalize

2. Flexible selection classifier– Works well for the File Browser Domain

Page 11: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Demo

Page 12: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Outline

1. Smart Selection2. Learning to Generalize3. User Study4. Conclusions

Page 13: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Basic Classification Framework

Selection Classifier:• Boosted Decision Trees• Limited depth (2)• Adjustable complexity

Features:• File name substrings• File extension• Creation Date• Size

Selection Classifier

LabelClassify

Limited Training Data

Available

Page 14: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

How can we improve?

• bad idea: Heuristics about user’s behavior?• better option: Learn to generalize from

Historical Data!

Page 15: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Example Behavioral Feature

Foo.py

Food.txt

Foo2.py

Bar.py

FBaz.py

BFoo.doc

FBFoo.py

BazFoo.py

Foo.py

Foo2.py

FBaz.py

FBFoo.py

Bar.py

Positive Evidence?

Learn this from Data!

Page 16: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Selection Classifier

LabelClassify

many users, many sessions

one session

How do we Learn from Behavioral Data?

Page 17: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

α

Selection Classifier

LabelClassify

many users, many sessions

Label Regressor

Behavior Features

one session

Predict Labels for Unlabeled Data

α

Page 18: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Extract FeaturesExtract FeaturesExtract Features

Labels

Training the Label Regressor

Step 1 Step 2 …Step 2 Step nStep nStep 1 Step 2 Step 3User Applies some operation on files

Page 19: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Lots of labeled data!

• m files, n steps, j tasks, k users

• Produces labeled examples

• Plenty of data available for LR!– No need to manually label– Personalization

)(mnjk

Page 20: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Training Selection Classifier

• Explicit Labels• Implicit Labels– Label Regressor produces a label

and a weight/confidence– Weight modulated by

Selection Classifier

LabelClassify

Label Regressor

α α

Page 21: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Recap of Our Method• Label Regressor– Features based on User’s

Behavior – Makes weighted predictions– Trained on historical data

• Selection Classifier– Predicts which items to select– Trained on:• User’s explicit examples• Modulated predictions from

Label Regressor

α

Page 22: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Outline

1. Smart Selection2. Learning to Generalize3. User Study4. Conclusions

Page 23: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

User Study

• 9 User pilot study– Gathered training data for LR

• Full Study: 12 Participants• 3 Conditions:– A: Standard shell (control)– B: Selection Classifier, but no Label Regressor– C: Selection Classifier + Label Regressor

Page 24: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Tasks

• 8 tasks for each condition– Isomorphic

• Widely varying– Half were easy with sorting/block select– Widely varying directory sizes

Page 25: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Number of Examples

manual selection

smart selection, explicit labels only

smart selection, explicit + soft labels

task

# ex

ampl

es

Page 26: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

How accurate are LR posteriors?

Page 27: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Selection Accuracy

smart selection, explicit labels only

smart selection, explicit + soft labels

task

accu

racy

Page 28: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Closer to goal in early rounds

smart selection, explicit labels only

smart selection, explicit + soft labels

step

accu

racy

Advantages:• Less Dramatic changes• Switch to manual & quickly complete

Page 29: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Conclusions

• Take advantage of data from other tasks!– Lots of data– Cheap

• Behavior features can reliably predict selection

THANK YOU!

Page 30: Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection.

Quotes:

• “The 2nd method (B) seemed a more "aggressive" version of method 1 (C). However the UI presentation i.e. the selection and deselection of large numbers of files strained my eyes and annoyed me.”

• “Selecting more files than desired can seem dangerous in some situations - especially when selecting files to delete or modify.”


Sumit testicular tumors

Sumit testicular tumors

Ryen White Microsoft Research ryenw@microsoft.com research.microsoft.com/~ryenw/talks/ppt/WhiteIMT542E.ppt.

Ryen White Microsoft Research [email protected] research.microsoft.com/~ryenw/talks/ppt/WhiteIMT542E.ppt.

Sumit nair portfolio

Sumit nair portfolio

Sumit verka project.docx

Sumit verka project.docx

sumit dpxs 2

sumit dpxs 2

Generalize System preference

Generalize System preference

Sumit Project

Sumit Project

Developers Sumit 2011

Developers Sumit 2011

Http://research.microsoft.com/en-us/projects/knows/ Microsoft Research.

Http://research.microsoft.com/en-us/projects/knows/ Microsoft Research.

sumit ammendmation hdfc

sumit ammendmation hdfc

Maths by Sumit Goyal By Sumit Goyal

Maths by Sumit Goyal By Sumit Goyal

Sumit grover

Sumit grover

MFS Sumit Leasing

MFS Sumit Leasing

NA Name Generalize - Weeblymsrakowski.weebly.com/uploads/9/3/0/9/9309760/when_marian_sang... · Name When Marian Sang ... Generalize • Sometimes authors generalize, or make a broad

NA Name Generalize - Weeblymsrakowski.weebly.com/uploads/9/3/0/9/9309760/when_marian_sang... · Name When Marian Sang ... Generalize • Sometimes authors generalize, or make a broad

Heamophilia sumit

Heamophilia sumit

Sumit Thesis 3

Sumit Thesis 3

SUMIT CHOUHAN RF1001B52

SUMIT CHOUHAN RF1001B52

Sumit Main Report

Sumit Main Report

Sumit Analysis

Sumit Analysis

Sumit (Page)

Sumit (Page)