Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime...
-
date post
21-Dec-2015 -
Category
Documents
-
view
215 -
download
0
Transcript of Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime...
![Page 1: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/1.jpg)
Preference Elicitation
in Scheduling Problems Ulaş Bardak
Ph.D. Thesis ProposalCommittee
Jaime Carbonell, Eugene Fink, Stephen Smith,
Sven Koenig (University of Southern California)
![Page 2: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/2.jpg)
Outline
Introduction
Example
Preliminary results
Plan of work
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
![Page 3: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/3.jpg)
Motivation
Improve resource planning by reducing uncertainty of the available knowledge.
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
![Page 4: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/4.jpg)
Hypothesis
By asking the questions with the highest
potential to reduce uncertainty, we can improve the quality of the resource plan while minimizing the cost of elicitation.
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
![Page 5: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/5.jpg)
Missing info:• Invited talk: – Projector need• Poster session: – Room size – Projector need
Assumptions:• Invited talk: – Needs a projector• Poster session: – Small room is OK
– Needs no projector
Initial scheduleAvailable rooms:
Roomnum.
Capacity
Projector
123
500 100 80
YesNoYes
Requests:• Invited talk, 9–10am: Needs big room• Poster session, 9–11am: Needs a room
1 2
3
Initial schedule:
Talk
Posters
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
![Page 6: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/6.jpg)
Choice of questions
1
Initial schedule:
Talk
Posters
Candidate questions:• Invited talk: Needs a projector?• Poster session: Needs a larger room? Needs a projector?
Requests:• Invited talk, 9–10am: Needs a large room• Poster session, 9–11am: Needs a room
Useless info: There are no large rooms w/o a projector
×
Useless info: There are no unoccupied larger rooms×Potentially useful info
√
2
3
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
![Page 7: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/7.jpg)
Improved scheduleRequests:• Invited talk, 9–10am: Needs a large room• Poster session, 9–11am: Needs a roomInfo elicitation:System:Does the poster sessionneed a projector?
User:A projector may be useful,but not really necessary.
Posters
1
Initial schedule:
Talk
Posters2
3
1
New schedule:
Talk
2
3
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
![Page 8: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/8.jpg)
Architecture
ElicitorNatural Lang. Optimizer
Ask user and get answers
Chooseand sendquestions
Updateresource
allocation
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
![Page 9: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/9.jpg)
Inside the Elicitor
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Get list of questions
For each question i getutilities for possible answers
Return top N questions
, costutility iScore i i Get question score
Each uncertain variable is a potential question
Plug in possible answers to the utility functionto get change in utility.
![Page 10: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/10.jpg)
Optimizer
Uses hill climbing to allocate resources
Searches for an assignment of resources with the greatest expected utility
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
![Page 11: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/11.jpg)
Related Work
Example critiquing [Burke et al.]Have users tweak result set
Collaborative filtering [Resnick], [Hill et al.]Have the user rank related items
Similarity-based heuristics [Burke]Look at past similar user ratings
Focusing on targeted use [Stolze]
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
![Page 12: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/12.jpg)
Related Work
Clustering utility functions [Chajewska] Decision tree [Stolze and Ströbel] Min-max regret [Boutilier]
Choose question that reduces max regret Auctions [Smith], [Boutilier],
[Sandholm]
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
![Page 13: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/13.jpg)
What is different?
No bootstrapping Continuous variables Large number of uncertain variables Tight integration with the optimizer Integration of multiple approaches Dynamic elicitation costs
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
![Page 14: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/14.jpg)
Example Domain
Assigning rooms to conference sessions
Rooms have properties. Sessions have preferences,
constraints, and importance values. Each preference is a function from a
room property to utility.
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
![Page 15: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/15.jpg)
Rooms have properties.
Sessions have preferences, constraints, and importance values.
Each preference is a function from a room property to utility.
Example Domain
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Room 1 can accommodate 200 people.Room 1 can accommodate 200 people.Room 3 has one projector : 80% chanceRoom 3 has no projectors : 20% chanceRoom 3 has one projector : 80% chanceRoom 3 has no projectors : 20% chance
![Page 16: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/16.jpg)
Rooms have properties.
Sessions have preferences, constraints, and importance values.
Each preference is a function from a room property to utility.
Example Domain
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Room 3 has one projector : 80% chanceRoom 3 has no projectors : 20% chanceRoom 3 has one projector : 80% chanceRoom 3 has no projectors : 20% chance
Invited talk cannot be before 2 p.m.Invited talk is more important than poster session.Invited talk cannot be before 2 p.m.Invited talk is more important than poster session.Invited talk very important : 40% chanceInvited talk moderately important : 60% chanceInvited talk very important : 40% chanceInvited talk moderately important : 60% chance
![Page 17: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/17.jpg)
Rooms have properties.
Sessions have preferences, constraints, and importance values.
Each preference is a function from a room property to utility.
Example Domain
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Capacity of Room 1 is 200.Capacity of Room 1 is 200.
Capacity preference: 150 people is minimum,200 people is acceptable, 250 people is best.Capacity preference: 150 people is minimum,200 people is acceptable, 250 people is best.
Invited talk very important : 40% chanceInvited talk moderately important : 60% chanceInvited talk very important : 40% chanceInvited talk moderately important : 60% chance
Capacity preference is [150, 200, 250] : 40% chanceCapacity preference is [50, 100, 150] : 60% chanceCapacity preference is [150, 200, 250] : 40% chanceCapacity preference is [50, 100, 150] : 60% chance
Room 3 has one projector : 80% chanceRoom 3 has no projectors : 20% chanceRoom 3 has one projector : 80% chanceRoom 3 has no projectors : 20% chance
![Page 18: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/18.jpg)
Experiments
Evaluation of RADAR 15 room properties 88 rooms 84 sessions 2500 variables 700 uncertain valuesSystem asked to provide 50 top
questions.
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
![Page 19: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/19.jpg)
0.72
0.78
Utility
10 3020 40 50No. of Questions
0.58
CertainIncrementalOptimizerestimate
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Incremental elicitation
![Page 20: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/20.jpg)
Completed work
Questions based on potential reduction of uncertainty
Empirical evaluation
Integration with RADAR
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
![Page 21: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/21.jpg)
Contributions
Fast computation of expected impact for potential questions
Use of the optimizer for calculating more accurate question weights.
Use of past elicitation results to improve the elicitation process.
Unifying different elicitation strategies.
√
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
![Page 22: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/22.jpg)
Search for optimal questions
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
100-150: 40% chance151-200: 60% chance
160-200: 50% chance100-160: 50% chance
100-130: 25% chance130-160: 25% chance
h=20, max utility increase = 20
h=10, max utility increase = 30h=10
h=15, max utility increase = 100
h=15
Best-first search with the optimizer used as the heuristic function.
Example: Uncertain room size
![Page 23: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/23.jpg)
Elicitation rules
Encoding of elicitation heuristics
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
mean 10,000
std-dev 5,000
type room
size room
size room
Conditions: Auditorium
elicit size roomAction:
rule Uncertain-Auditorium-Size(room)
![Page 24: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/24.jpg)
Learning of elicitation rulesDerive rules based on past elicitations
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
mean 1
100
type room
projector room
importance event
Conditions: Auditorium
elicit projector roomAction:
rule Learned-Rule(room,event)
… EventImp.
Room Prop.
Mean Prop.
Elicit. Result
… 110 Proj. 0.5 +
… 115 Size 250 -
… 105 Proj. 0.9 +
… 90 Proj. 0.5 -
… 200 Size 100 +
… 150 Proj. 0.3 +
… SessionType
RoomType
EventImportance
Room Property
MeanValue
ElicitationResult
… Invited Talk
Auditorium
110 Proj. 0.5 +
… Posters Meeting R.
115 Size 250 -
… Best Paper
Auditorium
105 Proj. 0.9 +
… Posters Classroom 90 Proj. 0.5 -
… Talk Auditorium
200 Size 100 +
… Keynote Auditorium
150 Proj. 0.3 +
![Page 25: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/25.jpg)
Dynamic question costs
Same cost for all questions Different cost for different question
types Learning of the question costs for
each type Learning of the question costs for
each information source
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
√
√
![Page 26: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/26.jpg)
Compare different approaches: Current system Search for optimal questions Hand coded elicitation rules Learned elicitation rules Unified system Human elicitorMeasure utility gain after each answer; also
evaluate running time
Experiments
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
![Page 27: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/27.jpg)
TimelineBest-First Search
Syntax for rules
Learning of rules
Experiments
Aug 2
007
Mar
200
7
Nov 2
006
July 2
006
Mar
200
6Unified System
Dec 2
007
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Writing
Learning of costs
![Page 28: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/28.jpg)
![Page 29: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/29.jpg)
Addendum
Outline – Introduction – Example – Preliminary Results – Plan of Work – Outline – Introduction – Example – Preliminary Results – Plan of Work – QuestionsQuestions
![Page 30: Preference Elicitation in Scheduling Problems Ulaş Bardak Ph.D. Thesis Proposal Committee Jaime Carbonell, Eugene Fink, Stephen Smith, Sven Koenig (University.](https://reader030.fdocuments.in/reader030/viewer/2022032522/56649d6a5503460f94a48475/html5/thumbnails/30.jpg)
0
0.05
0.1
0.15
0 10 20 30 40 50
Question number (more important to less important)
Abs
olut
e ch
ange
in s
ched
ule
qual
ity
0
0.05
0.1
0.15
0 10 20 30 40 50
Question number (more important to less important)
Abs
olut
e ch
ange
in s
ched
ule
qual
ity