Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010
description
Transcript of Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010
![Page 1: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/1.jpg)
Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and PerformanceRebecca FiebrinkPerry Cook, AdvisorFPO, 12/13/2010
![Page 2: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/2.jpg)
2
![Page 3: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/3.jpg)
3
![Page 4: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/4.jpg)
4
![Page 5: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/5.jpg)
5
function [x flag hist dt] = pagerank(A,optionsu)[m n] = size(A);if (m ~= n) error('pagerank:invalidParameter', 'the matrix A must be square');end; options = struct('tol', 1e-7, 'maxiter', 500, 'v', ones(n,1)./n, … 'c', 0.85, 'verbose', 0, 'alg', 'arnoldi', … 'linsys_solver', @(f,v,tol,its) bicgstab(f,v,tol,its), … 'arnoldi_k', 8, 'approx_bp', 1e-3, 'approx_boundary', inf,… 'approx_subiter', 5);if (nargin > 1) options = merge_structs(optionsu, options);end;if (size(options.v) ~= size(A,1)) error('pagerank:invalidParameter', … 'the vector v must have the same size as A');end;if (~issparse(A)) A = sparse(A);end;% normalize the matrixP = normout(A);switch (options.alg) case 'dense’ [x flag hist dt] = pagerank_dense(P, options); case 'linsys’ [x flag hist dt] = pagerank_linsys(P, options) case 'gs’ [x flag hist dt] = pagerank_gs(P, options); case 'power’ [x flag hist dt] = pagerank_power(P, options); case 'arnoldi’ [x flag hist dt] = pagerank_arnoldi(P, options); case 'approx’ [x flag hist dt] = pagerank_approx(P, options); case 'eval’ [x flag hist dt] = pagerank_eval(P, options); otherwise
error('pagerank:invalidParameter', ...
'invalid computation mode specified.');
end;
function [x flag hist dt] = pagerank(A,optionsu)
![Page 6: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/6.jpg)
6
![Page 7: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/7.jpg)
7
![Page 8: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/8.jpg)
8
useful
usableeffective, efficient, satisfying
![Page 9: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/9.jpg)
9
Machine learning
algorithms?
![Page 10: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/10.jpg)
10
Outline
• Overviews of interactive computer music and machine learning
• The Wekinator software• Live demo• User studies• Findings and Discussion• Contributions, Future Work, and Conclusions
![Page 11: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/11.jpg)
interactive computer music
![Page 12: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/12.jpg)
12
Interactive computer music
sensed action
interpretation
response (music, visuals, etc.)
computer
human with microphone, sensors, control interface, etc.
audio synthesis or processing,
visuals, etc.
![Page 13: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/13.jpg)
13
Example 1: Gesture recognition
sensed action
identification
response
computer
Bass drum:
“Gesture 1”
![Page 14: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/14.jpg)
14
Example 1: Gesture recognition
sensed action
response
computer
Bass drum:
Hi-hat“Gesture 2”
identification
![Page 15: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/15.jpg)
15
Model of sensed action to meaning
sensed action
response
computer
model
meaning
![Page 16: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/16.jpg)
16 computer
Example 2: Continuous gesture-to-sound mappings
![Page 17: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/17.jpg)
17
sensed action
interpretation
sound generation
computer
mapping
human + control interface
Example 2: Continuous gesture-to-sound mappings
![Page 18: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/18.jpg)
18
A composed system
sensed action
mapping/model/
interpretation
response
mapping/model/
interpretation
![Page 19: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/19.jpg)
supervised learning
![Page 20: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/20.jpg)
20
algorithm
trainingdata
Training
Supervised learning
model
inputs
outputs
![Page 21: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/21.jpg)
21
algorithm
trainingdata
Training
Supervised learning
model
inputs
outputsRunning
“Gesture 1” “Gesture 2” “Gesture 3”
“Gesture 1”
![Page 22: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/22.jpg)
22
Supervised learning is useful
• Models capture complex relationships from the data and generalize to new inputs. (accurate)
• Supervised learning circumvents the need to explicitly define mapping functions or models. (efficient)
Has been demonstrated to be useful in musical applications, but no usable, general-purpose tools exist for composers to apply algorithms in their work.
![Page 23: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/23.jpg)
23
Criteria for a usable supervised learning tool for composers
1. General-purpose: many algorithms & applications
2. Runs on real-time signals
3. Compatible with other compositional tools
4. Supports interaction via a GUI
✓
Weka and similar
general-purpose tools
✓
✓✓
???
✓✓
✓(some)
Existing computer
music tools
✗✓
✓(some)
✗ ✗ ✓5. Supports appropriate end-user interactions with the supervised learning process
✗✗
![Page 24: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/24.jpg)
24
Criteria for a usable supervised learning tool for composers
1. General-purpose: many algorithms & applications
2. Runs on real-time signals
3. Compatible with other compositional tools
4. Supports interaction via a GUI
✓
Weka and similar
general-purpose tools
✓
✓✓
???
✓✓
✓(some)
Existing computer
music tools
✗✓
✓(some)
✗ ✗ ✓5. Supports appropriate end-user interactions with the supervised learning process
✗✗
Interactively…• create training data• evaluate models• modify training data
![Page 25: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/25.jpg)
25
Interactive machine learning (IML)
• Fails and Olsen (2003): training set editing for computer vision systems
• More recent work:– application to other domains
(Shilman et al. 2006; Fogarty et al. 2008; Amershi et al. 2009; Baker et al. 2009)
– other types of interactions, e.g. manipulating confusion matrices(e.g., Talbot et al. 2009, Kapoor et al. 2010)
![Page 26: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/26.jpg)
26
Research questions
•Which interactions are possible and useful?•What are the essential differences between IML and conventional ML?
•How can IML be used in real-time and creative contexts?
![Page 27: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/27.jpg)
27
Outline
• Overviews of interactive computer music and machine learning
• The Wekinator software• Live demo• User studies• Findings and Discussion• Contributions, Future Work, and Conclusions
![Page 28: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/28.jpg)
28
The Wekinator
• Built on Weka API• Downloadable at http://code.google.com/p/wekinator/
(Fiebrink, Cook, and Trueman 2009; Fiebrink, Trueman, and Cook 2009; Fiebrink et al. 2010)
1. General-purpose: many algorithms & applications
2. Runs on real-time signals
3. Compatible with other compositional tools
4. Supports interaction via a GUI
✓✓✓✓
5. Supports appropriate end-user interactions with the supervised learning process
✓
![Page 29: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/29.jpg)
29
Running models in real-time
model(s)
.01, .59, .03, ....01, .59, .03, ....01, .59, .03, ....01, .59, .03, ...
5, .01, 22.7, …5, .01, 22.7, …5, .01, 22.7, …5, .01, 22.7, …
time
time
Feature extractor(s)
Parameterizable process
![Page 30: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/30.jpg)
30
Real-time, interactive design
The Wekinator supports user interaction with all stages of the model creation process.
![Page 31: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/31.jpg)
31
3.3098 Class24
Under the hood
Model1 Model2 ModelM
Feature1 Feature2 Feature3 FeatureN…
Parameter1 Parameter2 ParameterM
…
…
joystick_x joystick_y
pitchvolume
webcam_1
![Page 32: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/32.jpg)
32
3.3098 Class24
Under the hood
Model1 Model2 ModelM
Feature1 Feature2 Feature3 FeatureN…
Parameter1 Parameter2 ParameterM
…
…
Learning algorithms:Classification:
AdaBoost.M1J48 Decision TreeSupport vector machineK-nearest neighbor
Regression:MultilayerPerceptron
![Page 33: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/33.jpg)
33
Tailored but not limited to music
The Wekinator• Built-in feature extractors for music & gesture• ChucK API for feature extractors and synthesis
classes
Other modules for sound synthesis,
animation, …?
Other feature extraction modules
Open Sound Control (UDP)
![Page 34: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/34.jpg)
34
Outline
• Overviews of interactive computer music and machine learning
• The Wekinator software• Live demo• User studies• Findings and Discussion• Contributions, Future Work, and Conclusions
![Page 35: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/35.jpg)
35
Outline
• Overviews of interactive computer music and machine learning
• The Wekinator software• Live demo• User studies• Findings and Discussion• Contributions, Future Work, and Conclusions
![Page 36: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/36.jpg)
36
Study 1: Participatory design process with composers
• Method:– 7 composers– 10 weeks, 3 hours / week– Group discussion, experimentation,
and evaluation– Iterative design– Final questionnaire
• Outcomes:– Composers focused on instrument-building, 2 ultimately used
in performance– Much-improved software and lots of feedback
(Fiebrink, Trueman, et al., 2010)
![Page 37: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/37.jpg)
37
Study 2: Teaching interactive systems building in an undergraduate course
• COS/MUS 314 Spring 2010 (PLOrk)• Focus on building interactive music
systems• Method:– One component of the midterm assignment:
students building interactive systems using supervised learning
– 1 continuous + 1 discrete system– Logging + short answer questions
• Outcomes:– Successful project completion, use in midterm and final
performances– Logs from 21 students
![Page 38: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/38.jpg)
38
Study 3: Bow gesture recognition
• Work with a composer/cellist to build gesture classifiers for a commercial sensor bow, the “K-Bow”
38
![Page 39: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/39.jpg)
39
Study 3: Bow gesture recognition
• Work with a composer/cellist to build gesture classifiers for a commercial sensor bow
• 8 classifiers for standard bow gestures:e.g., up/down bow (2 classes), articulation (7 classes)
• Method:– All classification tasks defined by cellist and addressed in order of her
choice• All 8 tasks in 1st session, 5 tasks in 2nd session
– Logging, observations, final questionnaire– Cellist assigned each iteration’s classifier a quality rating (1 to 10)
• Outcome:– Successful classifiers created for all 8 tasks (rated “9” or “10”)
39
![Page 40: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/40.jpg)
40
Study 4: Composer case studies
• Clapping Music Machine Variations (CMMV) by Dan Trueman, faculty
![Page 41: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/41.jpg)
41
Study 4: Composer case studies
• CMMV by Dan Trueman, faculty• The Gentle Senses / MARtLET by Michelle Nagai, graduate student
![Page 42: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/42.jpg)
42
Study 4: Composer case studies
• CMMV by Dan Trueman, faculty• The Gentle Senses / MARtLET by Michelle Nagai, graduate student• G by Raymond Weitekamp, undergraduate
![Page 43: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/43.jpg)
43
Outline
• Overviews of interactive computer music and machine learning
• The Wekinator software• Live demo• User studies• Findings and Discussion• Contributions, Future Work, and Conclusions
![Page 44: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/44.jpg)
Discussion of Findings
1. Users took advantage of interaction in their work with the Wekinator.
2. Users employed a variety of model evaluation criteria, and subjective evaluation did not always correlate with cross-validation accuracy.
3. Feedback from the Wekinator influenced users’ actions and goals.
4. The Wekinator was a useful and usable tool.5. Interactive supervised learning can be a tool for
supporting creativity and embodiment.
![Page 45: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/45.jpg)
45
An iterative approach to model-building
![Page 46: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/46.jpg)
46
An iterative approach to model-building
(mean per student for each task)
(mean per classification task)
PLOrk con-tinuous
PLOrk discrete
KBow 1st session
KBow 2nd session
0
2
4
6
Mean # Trainings
![Page 47: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/47.jpg)
47
Frequent modifications to training data in-between re-trainings
Add Data Edit Data Delete Data Clear All Data
Change Learner
Change Learn
Params
Change Features
0
0.5
1
1.5
2
2.5
3
KB
owpe
r-ta
sk a
vera
ge
Add Data
Edit Data
Delete
Data
Clear A
ll Data
Chang
e Learn
er
Chang
e Learn
er P
arams
Change
Fea
tures
0
2
4
6
Cont.Disc.P
LOrk
per-
stud
ent a
vera
ge
![Page 48: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/48.jpg)
48
Interaction and the training dataset
• Training data is an interface for key tasks:– defining the learning problem– clarifying the learning problem to fix errors– communicating changes in problem over time
• User incrementally makes model more complex• User changes his mind about how a model should work
– providing a “sketch” that the computer fills in
![Page 49: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/49.jpg)
49
Thoughts: Data editing makes sense
• The training data is the most effective and appropriate interface for accomplishing these tasks.
• Potentially appropriate for many problem domains where:– User has knowledge of learning problem– User is capable of creating examples
• The interfaces for creating and editing the training data are crucial to the user’s ability to effectively and efficiently accomplish these tasks.
![Page 50: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/50.jpg)
50
Playalong data recording
•Employed by– 3 of 8 composers in Studies 1 and 4– 8 of 21 PLOrk students
•These users found playalong highly useful– Allowed training data to represent more fine-grained
information– Enabled some composers to engage their musical and
physical expertise• Allowed practice and attention to “feel”
![Page 51: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/51.jpg)
51
“Conventional” model evaluation
model
Available data
Training set
Evaluation set
Train
Evaluate
Cross-validation repeats this process with different data partitions.
![Page 52: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/52.jpg)
52
“Direct” evaluation in Wekinator
model
Training set Train
Evaluate
![Page 53: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/53.jpg)
53
Direct evaluation used most frequently
• Composers in participatory design and case studies: only direct evaluation
• KBow and PLOrk:
Cross-val. Acc.
Train. Acc. Direct Eval.0
1
2
3
4
5
6
PLOrk cont.PLOrk disc.KBow 1KBow 2
Mea
n #
times
act
ion
take
n
![Page 54: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/54.jpg)
54
Roles of cross-validation and training accuracy
• K-bow: Cross-validation used to quickly and objectively compare different feature selections and learning algorithms
• PLOrk:– Treated as reliable evidence a model was performing
well– Used to validate the user’s own ability
![Page 55: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/55.jpg)
55
Roles of direct evaluation
• Used to assess behavior of the model against subjective criteria
• Used to obtain feedback that shapes the users’ future interactions with the system
![Page 56: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/56.jpg)
Discussion of Findings
1. Users took advantage of interaction in their work with the Wekinator.
2. Users employed a variety of model evaluation criteria, and subjective evaluation did not always correlate with cross-validation accuracy.
3. Feedback from the Wekinator influenced users’ actions and goals.
4. The Wekinator was a useful and usable tool.5. Interactive supervised learning can be a tool for
supporting creativity and embodiment.
![Page 57: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/57.jpg)
57
Accuracy measured by direct evaluation
• Important in building discrete gesture classifiers– Accuracy = models outputs are correct according to
learning concept definition
• Accuracy still important for open-ended instrument-building tasks– Accuracy = matching a user’s expectations, especially on
inputs like the training examples
![Page 58: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/58.jpg)
58
Other criteria
•For discrete classifiers:– Cost: consequences and locations of model errors– Decision boundary smoothness
•For continous mappings:– Complexity, difficulty – Unexpectedness and surprise– “Feel”
![Page 59: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/59.jpg)
59
Subjective evaluation criteria & CV
• K-bow:– Cross-validation sometimes correlates with subjective
quality, but sometimes it doesn’t!
Task: Horizontal Position
Vertical Position
Bow Direction
On/Off String
Speed Articulation
R: -0.59 -0.44 -0.74 -0.50 0.65 0.93
Pearson’s correlation for tasks with > 3 iterations:
![Page 60: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/60.jpg)
60
Subjective evaluation criteria & CV
• K-bow:– Cross-validation sometimes correlates with subjective
quality, but sometimes it doesn’t!
Task: Horizontal Position
Vertical Position
Bow Direction
On/Off String
Speed Articulation
R: -0.59 -0.44 -0.74 -0.50 0.65 0.93
Pearson’s correlation for tasks with > 3 iterations:
![Page 61: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/61.jpg)
61
Subjective evaluation criteria & CV
• K-bow:– Cross-validation sometimes correlates with subjective
quality, but sometimes it doesn’t!
Task: Horizontal Position
Vertical Position
Bow Direction
On/Off String
Speed Articulation
R: -0.59 -0.44 -0.74 -0.50 0.65 0.93
Pearson’s correlation for tasks with > 3 iterations:
![Page 62: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/62.jpg)
62
Thoughts: Is generalization accuracy important?
• Yes!– Human and environmental variations are inevitable
• BUT it may not be the only or most important factor• Generalization estimated from the training set (e.g.,
using cross-validation) is not always informative• Implies that models designed for human use should
be evaluated by human use.
![Page 63: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/63.jpg)
63
Thoughts: Choosing or designing better algorithms• Most algorithms’ training process aims for a model with
good generalization• BUT the user is employing the training data as an
interface– User relies on good training accuracy to influence the model’s
behavior
• Better algorithms might– Privilege training accuracy (e.g., k-nearest neighbor)– Provide parameters for interactive improvement against other
subjective criteria (e.g., using regularization parameter for boundary smoothness)
![Page 64: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/64.jpg)
Discussion of Findings
1. Users took advantage of interaction in their work with the Wekinator.
2. Users employed a variety of model evaluation criteria, and subjective evaluation did not always correlate with cross-validation accuracy.
3. Feedback from the Wekinator influenced users’ actions and goals.
4. The Wekinator was a useful and usable tool.5. Interactive supervised learning can be a tool for
supporting creativity and embodiment.
![Page 65: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/65.jpg)
65
Interaction involves control and feedback
control
feedback**Running the models (“direct evaluation”)Cross-validation and training accuracy
Machine learning
algorithms
![Page 66: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/66.jpg)
66
Running models informs future actions
• For example:– locate errors add correctly-labeled examples– detect total failure delete all the data
![Page 67: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/67.jpg)
67
Running models trains users to be more effective supervised learning practitioners
•Users especially learned to create better training datasets– Minimize noise– Balance the number of examples in each class– Vary examples along all the dimensions that might vary
in performance
• Important for novice users
![Page 68: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/68.jpg)
68
Running models informs users’ goals for machine learning• Some PLOrkers and composers relied on the Wekinator to
help formulate the problem definition– composers liked being inspired by mapping behaviors
• Users learned what was most easily accomplished
… and exploited flexibilities in the learning concept definition to create a model that most easily met their most important goals• PLOrk: Define classes based on what’s easy to classify• K-Bow: Add more classes when a model performed well (e.g.,
Bow Speed)• A key benefit of IML!
![Page 69: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/69.jpg)
69
Running models teaches users about themselves and their work
K-Bow cellist:Model’s confusion of spiccato and riccocet
realization that her spiccato was too much like riccocet
improved technique
improved classifiers
![Page 70: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/70.jpg)
70
Thoughts: Impact in other domains
• Direct evaluation on user-generated inputs could offer similar advantages in other applied machine learning domains– Appropriate when user is qualified to create inputs and
evaluate results
![Page 71: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/71.jpg)
Discussion of Findings
1. Users took advantage of interaction in their work with the Wekinator.
2. Users employed a variety of model evaluation criteria, and subjective evaluation did not always correlate with cross-validation accuracy.
3. Feedback from the Wekinator influenced users’ actions and goals.
4. The Wekinator was a useful and usable tool.5. Interactive supervised learning can be a tool for
supporting creativity and embodiment.
![Page 72: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/72.jpg)
72
Barriers to usability
• Long training times (a few users)• Algorithms’ inability to model the desired concept
[easily]– e.g., producing a linear function with a neural network
• Difficulty in debugging – No guidance on choosing a better algorithm or
algorithm parameters– Especially due to novice users
![Page 73: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/73.jpg)
73
Usability and usefulness: Study 1 composers
Statement 5-point Likert mean (std. dev.)
“The Wekinator allows me to create more expressive mappings than other techniques.”
4.5 (.8)
“The Wekinator allows me to create mappings more easily than other techniques.”
4.7 (.5)
![Page 74: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/74.jpg)
74
Usability and usefulness: PLOrk students
Statement 5-point Likert mean (std. dev.)
“I can reliably predict what sound my model will make for a given inputgesture.”
4.5 (.7)
“Wekinator eventually learned what I wanted it to.”
4.3 (.9)
“My model provides reliable gesture classifications” (discrete task)
4.9 (.2)
“My model is musically expressive” (continuous task)
4.1 (.7)
![Page 75: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/75.jpg)
75
Usability and usefulness: PLOrk students
• Model-building was fast– 27.1 minutes for continuous mapping– 16.1 minutes for discrete classifier
• Students enjoyed the Wekinator– “Learning by experimentation was a lot of fun!”– “It’s so cool, the Wekinator rocks.”
![Page 76: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/76.jpg)
76
Usability and usefulness: K-Bow
Task Rating (1 to 10) CV Accuracy (%)Direction 10 87.3On/Off String 10 83.5Grip 10 100.0Roll 10 98.2Horizontal Position 10 89.3Vertical Position 10 90.0Speed 9 87.5Articulation 9 98.8
• Models successfully created for all 8 tasks:
![Page 77: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/77.jpg)
77
Usability and usefulness: K-Bow
Statement 5-point Likert response
“The Wekinator was able to create accurate bow stroke classiers in our work so far”
4
“The Wekinator was able tocreate bow stroke classiers more easily than other approaches”
“10 (so 5)"
![Page 78: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/78.jpg)
78
Usability and usefulness: Case studies
create mappings more easily
Series1
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Trueman Nagai Weitekamp
Agreement (1-5)
create mappings that were more expressive
create a kind of music thatisn't possible or that is hard to
create using other techniques
approach the process of composition in a new way
The Wekinator allowed me to:
![Page 79: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/79.jpg)
Discussion of Findings
1. Users took advantage of interaction in their work with the Wekinator.
2. Users employed a variety of model evaluation criteria, and subjective evaluation did not always correlate with cross-validation accuracy.
3. Feedback from the Wekinator influenced users’ actions and goals.
4. The Wekinator was a useful and usable tool.5. Interactive supervised learning can be a tool for
supporting creativity and embodiment.
![Page 80: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/80.jpg)
80
“There is simply no way I would be able to manually create the mappings that the Wekinator comes up with; being able to playfully explore a space that I've roughly mapped out, but that the Wekinator has provided the detail for, is inspiring.”
![Page 81: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/81.jpg)
81
“The ability to map sound and gesture, in a very immediate and intuitive (yet unpredictable) way is really the most inspiring and useful aspect of the wekinator for me right now. I can see the possibility of building interfaces or instruments as needed, flexibly, on the fly, for different kinds of projects, and being able to quickly map them out to existing sound sets with only minor programming changes.”
![Page 82: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/82.jpg)
82
Qualities important to composers
• Speed and ease of creating and exploring mappings (especially complex mappings)– IML is useful: Demonstration can be faster and more
efficient than coding.• Access to surprise and discovery– IML is useful: Neural networks fill in the details of the
training data sketch.• Balancing surprise and complexity with predictability and
control– IML is useful: Users can reliably steer model behavior
using the training data.
![Page 83: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/83.jpg)
83
Creativity support in HCI
• Qualities of the Wekinator align with “creativity support tool” guidelines proposed by Shneiderman (2000, 2007) and Resnick et al. (2005)– Support exploration, discovery, and sketching– Support diverse users (e.g., novices and experts) and applications– Operate seamlessly with other [composition] tools
• IML integral to Wekinator’s realization of guidelines• Interactive supervised learning may be an effective tool
for supporting creativity in many other domains.
![Page 84: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/84.jpg)
84
Embodiment is also important• Composers valued the ability to take an embodied approach
to design, which engaged their physical expertise as musicians• Designing this way also produced instruments that “felt right”
![Page 85: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/85.jpg)
85
“I have never before been able to work with a musical interface … that allowed me to really ‘feel’ the music as I was playing it and developing it. The Wekinator allowed me to approach composing with electronics and the computer more in the way I might if I was writing a piece for cello, where I would actually sit down with a cello and try things out.”
![Page 86: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/86.jpg)
86
Embodiment is also important• Wekinator engaged users’ physical expertise as musicians• And produced instruments that “felt right”– Users were physically engaged in the creation of the data and the
evaluation of the models (especially with playalong)• Embodied cognition: body plays an important role in learning,
understanding, and thinking• Physical knowledge, kinesthetic memory, and bodily skill are
important considerations in the design of interactive systems (Klemmer et al. 2006)
• Interactive machine learning may be useful for supporting embodied design in other real-time, interactive domains
![Page 87: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/87.jpg)
87
Outline
• Overviews of interactive computer music and machine learning
• The Wekinator software• Live demo• User studies• Findings and Discussion• Contributions, Future Work, and Conclusions
![Page 88: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/88.jpg)
88
Contributions
1. The Wekinator software and “playalong” interaction for training data creation
![Page 89: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/89.jpg)
89
“Well, I had basically lost interest in the whole process of digital controller-based instrument building, so the Wekinator's very existence has enabled and inspired me to get back into the game... The Wekinator enables you to focus on what your primary sonic and physical concerns are, and takes away the need to address so many details, and it does so in such a way that even if you DID spend all the time on building the mappings manually, you would *never* come up with what the Wekinator comes up with. So, the process becomes more focused, more musical, more creative, more playful. I actually *want* to do it.”
![Page 90: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/90.jpg)
90
Contributions
2. A demonstration of the important roles that interaction can play in the development of supervised learning systems…
…and a greater understanding of the differences between interactive and conventional machine learning contexts.
communicating evolving problem definitions, evaluating models against subjective criteria, training users to become better at machine learning, embodied design, …
emphasis on generalization accuracy and role of the training set, training dataset size, time constraints
![Page 91: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/91.jpg)
91
Contributions
3. A better understanding of the requirements and challenges in the analysis and design of algorithms and interfaces for interactive supervised learning in real-time and creative problem domains.
enabling inspiration, supporting novice users, providing access to complexity, enabling evaluation against subjective criteria, supporting embodiment, rapid prototyping, …
useful
usable
![Page 92: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/92.jpg)
92
Contributions
4. A clearer characterization of composers’ goals and priorities for interacting with computers in music composition and instrument design and a demonstration that interactive supervised learning is useful in supporting composers in their work.
![Page 93: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/93.jpg)
93
Contributions
5. A demonstration of the usefulness of interactive supervised learning as a creativity support tool.
Supporting exploration, discovery, and sketchingSupporting diverse users (e.g., novices and experts) and applications
Operating seamlessly with other tools
![Page 94: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/94.jpg)
94
Future work
• Improve the Wekinator as a compositional tool and a research platform– Collaborate with musicians, researchers, and other
users in participatory processes
• Investigate new algorithms, interactions, and interfaces for music performance and beyond– Further explore how interactive machine learning can
be made more useful and usable by more people applying it to more problems
![Page 95: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/95.jpg)
95
Final Conclusions
• It is important to consider the human-computer interfaces and context of algorithms in practice
• Interactive supervised learning has the potential to significantly improve the usability and usefulnesss of conventional learning algorithms, and to enable application to new problems by new users
• Computer music is a fantastic area to be doing research– Musicians make great collaborators
![Page 96: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/96.jpg)
96
Thanks!• Perry Cook• Dan Trueman• Dan Morris• Ken Steiglitz• Adam Finkelstein• Szymon Rusinkiewicz• Michelle Nagai• Cameron Britt• Konrad Kaczmarek• Michael Early• MR Daniel• Anne Hege• Raymond Weitekamp• All the PLOrk students
• Meg Schedel• Andrew McPherson• Barry Threw• Keith McMillen Instruments• Ge Wang• Jeff Snyder• Xiaojuan Ma• Sonya Nikolova• Matt Hoffmann• Merrie Morris• Sumit Basu• Ichiro Fujinaga
• National Science Foundation GRFP
• Francis Lathrop Upton Fellowship
• National Science Foundation grants 0101247 and 0509447
• The Kimberly and Frank H. Moss '71 Research Innovation Fund
• The David A. Gardner '69 Magic Project
• The John D. and Catherine T. MacArthur Foundation
• Everyone else I’m forgetting
![Page 97: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/97.jpg)
97
Related publications• Fiebrink, R. 2006. An exploration of feature selection as an optimization tool for musical genre
classification. Master’s thesis, McGill University. • Fiebrink, R., P. R. Cook, and D. Trueman. 2009. “Play-along mapping of musical controllers.” Proc.
International Computer Music Conference.• Fiebrink, R., M. Schedel, and B. Threw. 2010. “Constructing a personalizable gesture-recognizer
infrastructure for the K-Bow.” International Conference on Music and Gesture (MG3).• Fiebrink, R., D. Trueman, C. Britt, M. Nagai, K. Kaczmarek, M. Early, M.R. Daniel, A. Hege, and P. R.
Cook. 2010. “Toward understanding human-computer interactions in composing the instrument.” Proc. International Computer Music Conference.
• Fiebrink, R., D. Trueman, and P. R. Cook. 2009. “A meta-instrument for interactive, on-the-fly learning.” Proc. New Interfaces for Musical Expression.
• Fiebrink, R., G. Wang, and P. R. Cook. 2007. “Don't forget the laptop: Using native input capabilities for expressive musical control.” Proc. International Conference on New Interfaces for Musical Expression.
• Fiebrink, R., G. Wang, and P. R. Cook. 2008. “Support for MIR prototyping and real-time applications in the ChucK programming language.” Proc. International Conference on Music Information Retrieval.
• Wang, G., R. Fiebrink, and P. R. Cook. 2007. “Combining analysis and synthesis in the ChucK programming language.” Proc. International Computer Music Conference.
![Page 98: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/98.jpg)
98
References
• Amershi, S., Fogarty, J., Kapoor, A., and Tan, D. 2010. “Examining Multiple Potential Models in End-User Interactive Concept Learning.” Proc CHI 2010.
• Baker, K., A. Bhandari, and R. Thotakura. 2009. “Designing an Interactive Automatic Document Classification System.” Proc. HCIR 2009, pp. 30–33.
• Fails, Jerry, and Dan Olsen. 2003. “Interactive machine learning.” Proc. IUI, pp. 39–45.• Fels, S. S. and G. E. Hinton. 1993. “Glove-Talk: A neural network interface between a data-glove and a
speech synthesizer.” IEEE Trans. on Neural Networks, vol. 4.• M. Lee, A. Freed, and D. Wessel. 1992. “Neural networks for simultaneous classification and parameter
estimation in musical instrument control.” Adaptive and Learning Systems, vol. 1706, pp. 244-55.• Raphael, Chris. 2001. “A probabilistic expert system for automatic musical accompaniment.” Journal of
Computational and Graphical Statistics, vol. 10, no. 3, pp. 487-512.• Shneiderman, B. 2000. “Creating Creativity: User interfaces for supporting innovation.” ACM Trans. CHI, vol.
7, no. 1, pp. 114–138.• Shneiderman, B. 2007. “Creativity support tools: Accelerating discovery and innovation.” Comm. ACM vol.
50, no. 12, Dec. 2007, pp. 20–32.• Witten, I., and E. Frank. 2005. Data Mining: Practical Machine Learning Tools and Techniques, 2nd ed. San
Francisco: Morgan Kaufmann.
![Page 99: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/99.jpg)
99
Training set size – why so small?
• Learning concepts were “easier” ? (i.e., lower sample complexity)
• Users learned to provide the most useful training examples for representing the problem?– like active learning, but the user is in charge
• Users defined the learning concept definition in order to negotiate the tradeoffs between what they wanted and what was possible in a given amount of time to create training data and train the algorithms?
![Page 100: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/100.jpg)
100
Running models enables users to practice employing them more effectively
• Through practice, they learn to use models more effectively• Users accepted or expected the need to adapt their behaviors
![Page 101: Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010](https://reader036.fdocuments.in/reader036/viewer/2022062410/56816620550346895dd97389/html5/thumbnails/101.jpg)
101
An HCI view on algorithms
• Algorithms afford certain possible interactions, control, and feedback – i.e., they have an innate potential to be useful
• User interfaces can hide or expose these affordances– And can expose them in more or less usable ways
• The Wekinator exploits the fact that supervised learning models can be manipulated through the training dataset
• Algorithms can be made more useful and usable– through more appropriate interfaces– through affording more appropriate interactions, control, and
feedback