RecSys 2016 Talk: Feature Selection For Human Recommenders
-
Upload
katherine-livins -
Category
Technology
-
view
521 -
download
0
Transcript of RecSys 2016 Talk: Feature Selection For Human Recommenders
![Page 1: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/1.jpg)
Proprietary and confidential
![Page 2: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/2.jpg)
Proprietary and confidential
![Page 3: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/3.jpg)
Proprietary and confidential
![Page 4: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/4.jpg)
Proprietary and confidential
Human Computation At Stitch Fix
![Page 5: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/5.jpg)
Proprietary and confidential
Heavy and repetitive computation
Large-scale working memory
Large-scale long-term memory
Context sensitivity/nuance
Aesthetic judgements
Relationship building
Novel inferences
Unstructured data
![Page 6: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/6.jpg)
Proprietary and confidential
Processes information to make recommendations
Can specify internal mechanisms
Can specify the data being used
Recommendations improve with better features (data)
Needs to be trained and tuned
Comes with internal mechanisms
Can consider the entire world
![Page 7: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/7.jpg)
Proprietary and confidential
Processes information to make recommendations
Can specify internal mechanisms
Can specify the data being used
Recommendations improve with better features (data)
Needs to be trained and tuned
Comes with internal mechanisms
Can consider the entire world
![Page 8: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/8.jpg)
Proprietary and confidential
Determine what they’re processing
Determine what they should be
processing
Change/shape what they’re processing
![Page 9: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/9.jpg)
Proprietary and confidential
Determine what they’re processing
Determine what they should be
processing
Change/shape what they’re processing
Make more recommendations
Deliver those recommendationsReceive feedback
![Page 10: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/10.jpg)
Proprietary and confidential
1: Determining what they’re processing
![Page 11: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/11.jpg)
Proprietary and confidential
If someone isn’t attending to something, but you’re showing it anyways you might
■ Make your worker less efficient (slower)■ Fatigue them (unnecessary filtering)■ Lose opportunities for including something more useful
Figure out what your human workers are attending to while they make their recommendations
If they aren’t attending to a feature, then they’re not making recommendations off of it
![Page 12: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/12.jpg)
Proprietary and confidential
Exploration
![Page 13: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/13.jpg)
Proprietary and confidential
‘Online’ Observations
What you get
○ Ability to reduce the hypothesis space
○ Higher granularity observations ○ Time-dependent observations
(when is something considered)
![Page 14: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/14.jpg)
Proprietary and confidential
Mouse Tracking
Cheap measure of attention
Non-invasive
Easy widespread deployment
![Page 15: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/15.jpg)
Proprietary and confidential
Eye Tracking
![Page 16: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/16.jpg)
Proprietary and confidential
[Visual] search patterns lie somewhere between random and systematic…. humans will attempt a more systematic search, but will still suffer from imperfect memory.
(Nickles et al., 2003)
![Page 17: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/17.jpg)
Proprietary and confidential
Eye Tracking
Resistant to strategy
Deterministic
Higher accuracy
![Page 18: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/18.jpg)
Proprietary and confidential
AREA OF INTEREST (AOI)
Eye Tracking
Resistant to strategy
Deterministic
Higher accuracy
![Page 19: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/19.jpg)
Proprietary and confidential
![Page 20: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/20.jpg)
Proprietary and confidential
Features You Want To Select!
![Page 21: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/21.jpg)
Proprietary and confidential
2: Determining what they should be
processing
![Page 22: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/22.jpg)
Proprietary and confidential
You’re interested in overall performance and can optimize for whatever is most important to you
■ True hits, false positives, false negatives■ Processing time
Given the features that they’re using, which ones produce the best recommendations?
![Page 23: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/23.jpg)
Proprietary and confidential
The Logic:○ Workers may vary in what features they use○ Look for correlations between attention to features and positive
metrics
Allows you to learn the optimal features amongst your current candidates
![Page 24: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/24.jpg)
Proprietary and confidential
Feature Drop Out Studies
![Page 25: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/25.jpg)
Proprietary and confidential
Feature Drop Out Studies A/B
![Page 26: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/26.jpg)
Proprietary and confidential
Feature Drop Out Studies
Logic
Show a feature to one cell, and remove it for another
If a positive difference in performance is observed, then that feature promotes better outcomes
![Page 27: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/27.jpg)
Proprietary and confidential
Feature Drop Out Studies
Optimal Conditions
A highly controlled “offline” environment
○ Allows for true participant randomization
○ Allows for repeated measures○ Allows for high “internal validity”
![Page 28: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/28.jpg)
Proprietary and confidential
Task-relevantbackground information(optional)
Ability to provide a response - track accuracy, RT, confidence, etc.
Trial-specific stimuli - use historic data with known outcomes
![Page 29: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/29.jpg)
Proprietary and confidential
Correct ~ Condition + (1|participant_id)
Condition differences
Feature promotes better recommendations
Feature either isn’t considered or makes no difference to recommendations if it is
No condition differences
![Page 30: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/30.jpg)
Proprietary and confidential
Further Use Of ‘Online’
Observations
What you get
○ Ability to determine whether there are certain times at which certain features are beneficial
○ Ability to figure out how information is searched for
![Page 31: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/31.jpg)
Proprietary and confidential
-Status: loved-Department: top-Color: purple
-Status: loved-Department: dress-Color: green
-Status: hated-Department: pants-Color: orange
-Status: ...-Department: ...-Color: ...
Start with a study to determine correlations
![Page 32: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/32.jpg)
Proprietary and confidential
Multiple metrics possible
■ Overall trajectories (http://www.eyetracking-r.com/)■ Saccade patterns■ Fixation times and locations
Correlate with success
![Page 33: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/33.jpg)
Proprietary and confidential
correct ~ fixated_on_loves + fixated_on_color_matches + … + (1|participant_id) …
Factors predict success
Attention to features may promote better recommendations
Attention to features may make no difference to recommendations
Factors don’t predict success
![Page 34: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/34.jpg)
Proprietary and confidential
correct ~ condition + … + (1|participant_id)
Follow up with a full experiment to determine whether the behavior
actually causes better recommendations
Manipulation congruent with ‘positive’ behaviors
![Page 35: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/35.jpg)
Proprietary and confidential
3: Shaping What They’re
Processing
![Page 36: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/36.jpg)
Proprietary and confidential
Controlled Lab Study Full A/B Test
![Page 37: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/37.jpg)
Proprietary and confidential
Stitch Fix’s “Styling Lab”
Full A/B Test in the live styling
environment
![Page 38: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/38.jpg)
Proprietary and confidential
Behavior Shaping : Humans :: Tuning : Computers Algorithms
Can be “in the moment”
● UX Changes● Directed Attention
Can be more sustained
● Training
![Page 39: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/39.jpg)
Proprietary and confidential
Change how the information is displayed - exploit human perception (consult UX)
![Page 40: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/40.jpg)
Proprietary and confidential
Testing
● Create questions relevant to what you want to train
● Have participants complete them
● Use IRT to determine question difficulty
Training
● Order questions by difficulty
● Have those being trained complete them in that order
● Given feedback on performance along the way
● Reinforce key concepts
Experimental Approach!
![Page 41: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/41.jpg)
Proprietary and confidential
This approach is grounded in Cognitive research!
Progressive Alignment prescribes giving people tasks that they’re more likely to succeed at, then progressively making those tasks harder
.02
.08
![Page 42: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/42.jpg)
Proprietary and confidential
Processes information to make recommendations
Can specify internal mechanisms
Can specify the data being used
Recommendations improve with better features (data)
Needs to be trained and tuned
Comes with internal mechanisms
Can consider the entire world
![Page 43: RecSys 2016 Talk: Feature Selection For Human Recommenders](https://reader031.fdocuments.in/reader031/viewer/2022030305/587181381a28ab230b8b7973/html5/thumbnails/43.jpg)
Proprietary and confidential
Questions?