Preference Elicitation for Interface Optimization · •Based on two interaction methods for...
Transcript of Preference Elicitation for Interface Optimization · •Based on two interaction methods for...
![Page 1: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/1.jpg)
Preference Elicitation forInterface Optimization
Krzysztof Gajos and Daniel S. WeldUniversity of Washington, Seattle
![Page 2: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/2.jpg)
Krzysztof Gajos
![Page 3: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/3.jpg)
Motivation:Supple Model-Based Interface Renderer
+
Hierarchy of State Vars +Methods
Screen Size,Available Widgets &Interaction Modes
Func.Interface
Spec.
DeviceModel
User Trace
CustomInterfaceRendering
+
Model of an IndividualUser’s Behavior(or that of a Group)
{<root, -, -> <LeftLight:Pow er, off, on> <Vent, 1, 3> <Projector:Input, video, com puter> … }
DecisionTheoreticOptimization
![Page 4: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/4.jpg)
[Gajos & Weld, IUI’04]
Supple Output
![Page 5: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/5.jpg)
Container factor weight: 0.0 Tab Pane factor weight: 100.0 Popup factor weight: 1.0 Spinner for integers factor weight: 5.0
Spinner (domain size) factor weight: 49.5238 Spinner for non-integers factor weight: 6.0
Slider factor weight: 45.7143 Progress bar factor weight: 0.0 Checkbox factor weight: 0.0 Radio button factor weight: 0.5
Horizontal radio button factor weight: 10.0 Radio button (>=4 values) factor weight: 0.0
Radio button (>=8 values) factor weight: 74.2857 Radio button for booleans factor weight: 14.2857
Checkbox list factor weight: 0.5 Horizontal Checkbox list factor weight: 0.5 Checkbox list (>=4 values) factor weight: 0.0 Checkbox list (>=8 values) factor weight: 1.0
Text field factor weight: 100.0 Text field for enumerable types factor weight: 14.2857
Single param action widget factor weight: 0.0 Button factor weight: 0.0
Combo box factor weight: 14.2857 Combo box (>10 values) factor weight: 45.7143
Combo box (>25 values) factor weight: 1.0 Combo box (>50 values) factor weight: 1.0
List factor weight: 71.4286 Reduced width list factor weight: 10.0
List under-size factor weight: 2.8571 List over-size factor weight: 57.1429
List for containers factor weight: 10.0 Label factor weight: 3.0 Text Area factor weight: 0.0 Text area under-width factor weight: 0.1
Text area under-height factor weight: 0.23809 Image-based factor weight: 0.0
Image based under-sized factor weight: 3.5714
[Gajos & Weld, IUI’04]
Supple Depends on Weights
![Page 6: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/6.jpg)
RIA
[Zhou +, UIST’04; IUI’05]
![Page 7: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/7.jpg)
[Zhou +, UIST’04; IUI’05]
![Page 8: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/8.jpg)
Expected Costof Interruption
Probability of aninterruptabilitystate I
i
BusyBody
Cost ofinterrupting ifuser is in state I
i
[Horvitz +, CSCW’04]
![Page 9: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/9.jpg)
BusyBody
Expected Costof Interruption
Probability of aninterruptabilitystate I
i
Cost ofinterrupting ifuser is in state I
i
Needs to be elicitedfrom the user for everyinterruptability state I
i
[Horvitz +, CSCW’04]
![Page 10: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/10.jpg)
[Agrawala +, SIGGRAPH’01]
LineDrive
![Page 11: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/11.jpg)
Arnauld: A Tool forPreference Elicitation
ArnauldOptimizing
UI Application
Weights
Raises level of abstraction:– instead of directly choosing weights…,– designers now interact with concrete outcomes
![Page 12: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/12.jpg)
Arnauld: A Tool forPreference Elicitation
ArnauldOptimizing
UI Application
Weights
Raises level of abstraction:– instead of directly choosing weights…,– designers now interact with concrete outcomes
![Page 13: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/13.jpg)
Arnauld: A Tool forPreference Elicitation
ArnauldOptimizing
UI Application
Weights
Raises level of abstraction:– instead of directly choosing weights…,– designers now interact with concrete outcomes
![Page 14: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/14.jpg)
ArnauldOptimizing
UI Application
Weights
Raises level of abstraction:– instead of directly choosing weights…,– designers now interact with concrete outcomes
Arnauld: A Tool forPreference Elicitation
![Page 15: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/15.jpg)
Benefits
• Saves Developers Time– By factor of 2-3x
• Improves Quality of Weights– Learned weights out-perform hand-tuned
• Users May Want to Override Default Params– Individual preferences– Multiple uses
![Page 16: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/16.jpg)
Our Contributions• Implemented Arnauld system for preference elicitation
– Applicable to most optimization-based HCI applications– Implemented on SUPPLE
• Based on two interaction methods for elicitingpreferences
• Developed a fast machine learning algorithm thatlearns the best set of weights from user feedback– Enables interactive elicitation
• Investigated two query generation algorithms– Keep the elicitation sessions short
![Page 17: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/17.jpg)
Outline
• Motivation• Elicitation techniques
– Example critiquing– Active elicitation
• User responses constraints• Learning from user responses• Generating queries• Results & Conclusions
![Page 18: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/18.jpg)
Example Critiquing
![Page 19: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/19.jpg)
Via Customization Facilities
Click!
![Page 20: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/20.jpg)
Provides Training Example!
before after
Result of Customization
<
![Page 21: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/21.jpg)
Example Critiquing
Exploits natural interaction Occuring during process of customizing interface
Effective when cost function is almost correct
But…
Can be tedious during early stages of parameterlearning process
Requires customization support to be provided bythe UI system (e.g. RIA, SUPPLE, etc.)
![Page 22: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/22.jpg)
Active Elicitation
![Page 23: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/23.jpg)
Active ElicitationUI in Two Parts
Structure provided by ARNAULD
![Page 24: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/24.jpg)
Active ElicitationUI in Two Parts
Content provided by the interface system forwhich we are learning weights
![Page 25: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/25.jpg)
Active Elicitation Convenient during early stages of parameter
learning process
Binary comparison queries easy for user
Doesn’t require any additional support from UIsystem, for which parameters are generated
But
Doesn’t allow designer to direct learning process
Choice of Best Question is Tricky
![Page 26: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/26.jpg)
Limitations of Isolated Feedback
Both examples so far provided feedback of the form
“All else being equal, I prefer sliders to combo boxes”
<
But what if using a better widget in one placeMakes another part of the interface crummy?!
![Page 27: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/27.jpg)
>
<In isolation, sliders are preferred
But using them may cause badness elsewhere
![Page 28: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/28.jpg)
Situated Feedbackwith Active Elicitation
![Page 29: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/29.jpg)
Situated Feedbackwith Example Critiquing
![Page 30: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/30.jpg)
Summary of Elicitation Interactions
ActiveElicitation
ExampleCritiquing
SituatedIsolated
![Page 31: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/31.jpg)
Outline
• Motivation• Elicitation techniques• User responses constraints• Learning from user responses• Generating queries• Results & Conclusions
![Page 32: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/32.jpg)
Turning User Responses IntoConstraints
!=
=K
k
kk interfacefuinterface1
)()cost(
All systems studied had linearly decomposable cost functions; these can be expressed as:
A “factor” reflectingpresence, absenceor intensity of someinterface property
A weightassociatedwith a factor
![Page 33: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/33.jpg)
From User Responses to Constraints
sliderhorizontalslidernumberforboxcomboboxcombo uuuu_____
+!+
<
) ≥ cost(cost( )
1
1
___
_
=
=
numberforboxcombo
boxcombo
f
f
1
1
_ =
=
sliderhorizontal
slider
f
f
!!==
"K
k
kk
K
k
kk interfacefuinterfacefu1
2
1
1 )()(
![Page 34: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/34.jpg)
Outline
• Motivation• Elicitation techniques• User responses constraints• Learning from user responses• Generating queries• Results & Conclusions
![Page 35: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/35.jpg)
Learning Algorithm
Given constraints of the form:
Find values of weights uk Satisfying a maximum number of constraints And by the greatest amount
!!==
"K
k
kk
K
k
kk interfacefuinterfacefu1
2
1
1 )()(
![Page 36: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/36.jpg)
Our Approach
Use a max-margin approachEssentially a linear Support Vector Machine
Reformulate constraints:
i
K
k
kk
K
k
kk slackmargininterfacefuinterfacefu +!"##== 1
2
1
1 )()(
![Page 37: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/37.jpg)
Our Approach
Use a max-margin approachEssentially a linear Support Vector Machine
Reformulate constraints:
i
K
k
kk
K
k
kk slackmargininterfacefuinterfacefu +!"##== 1
2
1
1 )()(
Per-constraint slack thataccommodates
unsatisfiable constraints
Shared margin bywhich all constraints
are satisfied
![Page 38: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/38.jpg)
Learning as Optimization
Set up an optimization problem that maximizes:
Subject to the constraints:
i
K
k
kk
K
k
kk slackmargininterfacefuinterfacefu +!"##== 1
2
1
1 )()(
!"i
islackmargin
![Page 39: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/39.jpg)
Learning as Optimization
Set up an optimization problem that maximizes:
Subject to the constraints:
i
K
k
kk
K
k
kk slackmargininterfacefuinterfacefu +!"##== 1
2
1
1 )()(
!"i
islackmarginSolved with standard
linear programming methods
in less than 250 ms.
![Page 40: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/40.jpg)
Outline
• Motivation• Elicitation techniques• User responses constraints• Learning from user responses• Generating queries• Results & Conclusions
![Page 41: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/41.jpg)
Generating Queries
• Important part of Active Elicitation– Like game of 20 questions, order is key
• Optimality is intractable• Introducing two heuristic methods
– Searching ℜn space of weights• General method: applies to all opt-based UI
– Search space of semantic differences• Faster• Requires tighter integration with the UI appl’ctn
![Page 42: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/42.jpg)
Generating Queries
• Why is it important?– Like game of 20 questions, order is key
• Optimality is intractable• Introducing two heuristic methods
– Searching ℜn space of weights• General method: applies to all opt-based UI
– Search space of semantic differences• Faster• Requires tighter integration with the UI appl’ctn
![Page 43: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/43.jpg)
Visualizing the searchthru ℜn space of weights
A binary preference question cleaves the space
![Page 44: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/44.jpg)
Preferred Region
Answering Question Creates Region
![Page 45: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/45.jpg)
Midway thru the Q/A Process…
What is the bestimmediate (greedy)question forcleaving?
![Page 46: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/46.jpg)
Good Heuristics for Cleaving
1. As close tothe centroidas possible
2. Perpendicularto the longestaxis of region
![Page 47: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/47.jpg)
Outline
• Motivation• Elicitation techniques• User responses constraints• Learning from user responses• Generating queries• Results & Conclusions
![Page 48: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/48.jpg)
Informal User Study• Four users
– Two Supple developers– Two “sophisticated users”
• I.e. programmers w/o Supple experience
• Developers asked to hand-build cost function– Hand-coding took 2-3x longer– Resulting function “wrong” 35% of the time!
• Using Arnauld to create cost function– Got robust cost function in 10-15 minutes– All said Arnauld much easier & more accurate
![Page 49: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/49.jpg)
Number of Elicitation Steps
Learning RateR
atio
of L
earn
ed F
unct
ion
to Id
eal
Of different question-generation algorithms
![Page 50: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/50.jpg)
Sensitivity to Noise10% User Errors
Number of Elicitation Steps
Rat
io o
f Lea
rned
Fun
ctio
n to
Idea
l
![Page 51: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/51.jpg)
Related Work
• Gamble Queries– Outcomex vs.pBest + (1-p)Worst
• Bayesian Learning– [Chajewska,
ICML’01]– Too slow for
interactive use
40 seconds(too slow)
1 second(large error)
![Page 52: Preference Elicitation for Interface Optimization · •Based on two interaction methods for eliciting preferences •Developed a fast machine learning algorithm that learns the best](https://reader035.fdocuments.in/reader035/viewer/2022070810/5f094c737e708231d426293f/html5/thumbnails/52.jpg)
Conclusions• Implemented Arnauld system for preference elicitation
– Applicable to most optimization-based HCI applications– Saves developers time– Creates better weights
• Based on two interaction methods– Example Critiquing– Active Elicitation– Investigated two query generation algorithms
• Novel machine learning algorithm– Learns good weights from user feedback– Fast enough for interactive elicitation