KEYKEYeded User Interface User InterfaceAn HCI Theoretic Design of a Novel An HCI Theoretic Design of a Novel

Music Composition InterfaceMusic Composition Interface

byby

Farhan Mohamed

supervised by

Dr. Sidney Fels Human Communication Technologies Laboratory

The University of British ColumbiaDepartment of Electrical and Computer Engineering Vancouver, British ColumbiaCanada

MotivationMotivation

“While composing with my music studio computer, I find it stressful to use my piano keyboard, computer keyboard and mouse repetitively, especially when I feel creative and work over long hours”

- Composer/Producer Kevin Kowal

Computer Keyboard

Piano KeyboardComputer Mouse

Kevin’s Studio Computer Setup

GoalGoal

MomentaryFootswitch

Electronic Piano Keyboard Electronic Piano Keyboard for for

Performance and EditingPerformance and Editing

Single-Point TouchpadSingle-Point Touchpad

The KEYed User Interface project was created to bind music performance and editing tasks by using the piano keyboard as the sole input device for all tasks

Kevin’s Setup using KEYed User Interface

Overview of PresentationOverview of Presentation • Current music composition workstation setup

• Alternative approaches and their shortcomings

• Applying findings in HCI

• KEYed User Interface system

• Experiments and results

• Conclusions

Switching between multiple input devices• Long device acquisition time• Not comfortable to use• Lots of physical space occupied by the input devices

Music Composition Workstation Music Composition Workstation

Computer Keyboard

Piano KeyboardComputer Mouse

Composition Workstation Setup

Alternative ApproachesAlternative Approaches

Mackie’s ® Human User Interface ® Meredith Brooks’ Studio Layout

• Interface separates performance and editing

- Mackie HUI hardware

• Interface integrates performance and editing

- Logic® Audio Music Software

Applying HCI FindingsApplying HCI Findings

• We apply HCI studies to composition, integrating the two

tasks

• HCI studies have been applied to performance and editing

HCI FindingsHCI Findings

• Mode Errors and their Prevention

• Human Manual Studies

- Bimanual

• Tacit Knowledge and

Metaphors

• Space - Time Multiplexing Tradeoffs in Input Devices

HCI FindingsHCI Findings

MomentaryFootswitch

Electronic Piano Electronic Piano KeyboardKeyboard

Single-Point TouchpadSingle-Point Touchpad

Overview ofOverview of KEYKEYeded User User InterfaceInterface

HCI Principles AppliedHCI Principles Appliedtoto KEYKEYeded User Interface User Interface

MomentaryFootswitch

Electronic Piano Electronic Piano KeyboardKeyboard

Single-Point TouchpadSingle-Point Touchpad

Mode Error Prevention

Bimanual Action

Space Multiplexed

E2

Arrange Octave (C4-B4)Arrange Octave (C4-B4)

Mapping Example:Mapping Example:

Transport Octave (C3-Transport Octave (C3-B3)B3)

Mapping Example:Mapping Example:C3 Open/Close transport window

C5 Open/Close key edit window

F#6 Undo

Tracks Parts

Sequence

C2

B2

C3

B3

C4

B4

C5

B5

C6

B

6

Touch Pad Octave (C2-B2)Touch Pad Octave (C2-B2)

Mapping Example:Mapping Example:

General Edit Octave (C6-B6)General Edit Octave (C6-B6)

Mapping Example:Mapping Example:

Key Edit Octave (C5-B5)Key Edit Octave (C5-B5)

Mapping Example:Mapping Example:

F4 Move down the track

E2 Selecting volume

G3 Click On/Off (B3)

Note

Piano MappingsPiano Mappings

KEYKEYeded UI Prototype 1 UI Prototype 1

• Prototype I - Steinberg ® Cubase ® 5.0 VST 32 on Windows ® 98

• KEYed Pedal for Mode Switching• Octaves and the Notes are labeled

KEYKEYeded UI Prototype I StudiesUI Prototype I Studies

• The interaction methods compared are:

a) Computer Keyboard + Piano Keyboard

b) KEYed User Interface Prototype I

• Testing for Response Time with 6 Composers

• Composing Tasks A&B, Repeated Once

Note: Task A and Task B involve repetitive performing and editing, thereby requiring the composers to switch back and forth between modes on their own.

KEYKEYeded UI Prototype I ResultsUI Prototype I Results

0

20

40

60

80

100

120

140

seco

nds

Computer keyboard

Piano controller

Mean Time with 95% Confidence error bars

Test 1 Test 2

• Learning Time: 5 minutes• Results using paired t-test:

Test 1 - Piano controller faster, significant (p<0.05)

Test 2 - Piano controller faster, not significant (p<0.06)• No mode errors detected

KEYed UI Prototype IIKEYed UI Prototype II

• Prototype II platform− Steinberg ® Nuendo ® 1.5 on Windows ® 2000

• KEYed Pedal and Sustain Pedal• A single-point touchpad for 1D and 2D tasks

Single-Point Touchpad

KEYed Pedal

KEYKEYeded UI Prototype II StudiesUI Prototype II Studies1. The interaction methods compared are:

a) Computer Keyboard / Mouse + Piano Keyboard

b) KEYb) KEYeded UI II with Audio

c) KEYc) KEYeded UI II without Audio • Testing for Response Time with 10 Composers• Simple, Moderate and Complex editing tasks

Note: All tasks involve repetitive composing and editing, thereby requiring the composers

to switch back and forth between modes on their own.

2. Case study for a one week period with 1 Composer

KEYKEYeded UI Prototype II StudiesUI Prototype II Studies

Experiment Booth

KEYed user interface Layout

Mean Completion times

KEYKEYeded UI Prototype II ResultsUI Prototype II Results

• Learning Time: 16 minutes• KEYed Prototype II with Audio vs. Computer Keyboard / Mouse :

Response time faster with the KEYed UI with no significance • KEYed Prototype II without Audio vs. Computer Keyboard / Mouse :

Response time faster with the KEYed UI with no significance

0

10

20

30

40

50

60

70

80

90

se

co

nd

s

Computer Keyboard /Mouse

KEYed (Audio)

KEYed (No Audio)

Simple Moderate Complex

KEYKEYeded UI Prototype II ResultsUI Prototype II Results

Case study results• Positive feedback• Very natural to use• Requested more mappings• Suggested the use of musical chords• Single-point touchpad too small

• Piano keyboard can be used as an alternative input device in music composition workstations

• A momentary foot pedal like the KEYed foot pedal is easy to acquire, and is an ideal mode switch for switching between the performing and editing modes

• Though small, the single-point touchpad is found to be natural to use for bimanual tasks

ContributionsContributions

ContributionsContributions

• KEYed user interface illustrates how an appropriate mapping of the layout, feedback, and context is important in the design of user interfaces.

• Results published at CHI’2002 and ICMC’2002.

• Overlaying composition functions on other controllers

• Exploring musical structures and mappings, for example, scales and chords

• Providing user customizable mappings

Future DirectionsFuture Directions

ConclusionsConclusions

• Current approaches to music composition workstations design divide the performance and editing tasks

• In pursuing the KEYed project, we have found that findings in HCI can be applied to the problems in current music composition workstation design

• Gives rise to faster response time in devices with minimal learning

• Ultimately, greater comfort leads to greater creative flow

and better music

QuestionsQuestions

KEYed UI System DesignKEYed UI System Design

MIDI OUT MIDI IN MIDI THRU

SB PCI 128

MIDI OUT MIDI IN MIDITHRU

SB Live

MIDI OUT MIDI IN MIDITHRU

KEYed Messaging

System

Cubase 5.0 VSTOr

Nuendo 1.5

KEYed DynamicLink Library

(Window Handles)

EventWindows

Queue

KEYed Sustain Pedal Pedal

Serial Port

Single-point Touchpad

Win

do

ws A

pp

lication

s

Measured Mean Task Completion Time vs Predicted KLM Times Measured Means

(Seconds)Predicted Time A

(Seconds)Predicted Time B

(Seconds)

Task A

Computer Keyboard 116.2218333 67.4 + R1 73.4 + R1

KEYed Prototype I 101.3736667 58.6 + R1 58.6 + R1

Task B

Computer Keyboard 96.71233333 56.2 + R2 62.2 + R2

KEYed Prototype I 86.05816667 42.2 + R2 42.2 + R2

Task C

Computer Keyboard (CK) / Mouse (M)

40.74 CK: 44.4 + R3 CK: 46 + R3

M: 39.8 + R3 M: 41.4 + R3

KEYed Prototype II (with audio)

40.7399 32.2 + R3 32.2 + R3

KEYed Prototype II (without audio)

42.1412 32.2 + R3 32.2 + R3

Touchpad 52.1371 39.8 + R3 41.4 + R3

Task D

Computer Keyboard (CK) / Mouse (M)

62.6374 CK: Not measurable CK: Not measurable

M: 41.3 + R4 M: 43.3 + R4

KEYed Prototype II(with audio)

58.245 22.6 + R4 22.8 + R4

KEYed Prototype II(without audio)

50.0287 22.6 + R4 22.8 + R4

Touchpad 83.6633 41.3 + R4 43.3 + R4

Task E

Computer Keyboard (CK) / Mouse (M)

64.923 CK: Not measurable CK: Not measurable

M: 44.2 + R5 M: 46.4 + R5

KEYed Prototype II(with audio)

62.5232 28.8 + R5 29.2 + R5

KEYed Prototype II(without audio)

59.9132 28.8 + R5 29.2 + R5

Touchpad 99.1176 44.2 + R5 46.4 + R5

Measured Mean Task Completion Time vs Predicted KLM Times

0

20

40

60

80

100

120

se

co

nd

s

Measured Mean Times

Predicted Time A

Predicted Time B

0

20

40

60

80

100

120

140s

ec

on

ds

Measured Mean Times

Predicted Time A

Predicted Time B

CK KP1 CK KP1

TASK A TASK B

CK/M KP2A KP2NA TP CK/M KP2A KP2NA TP CK/M KP2A KP2NA TP TASK C (Simple) TASK D (Moderate) TASK E (Complex)

Where,

CK = Computer Keyboard

KP1 = KEYed Prototype I

KP2 = KEYed Prototype I

M = Mouse

KP2A = KEYed Prototype II with Audio

KP2NA = KEYed Prototype II without Audio

TP = Touchpad

Single point touchpad Ex. Drawing graphs with the touchpad

TASK A

1. Open the Transport window 2. On the current MIDI track, perform a recording by playing on the piano for the

pre-set number of bars with the sustain pedal pressed continuously during the recording

3. After the recording stops, stop playing and move the play cursor to the 0 position

4. Play the bars 0 to 10 of the pre-recorded piece once

5. Stop playing and move the play cursor to the 0 position 6. Close the Transport window

TASK B 1. Open the Transport window

2. On the current MIDI track, perform a recording by playing on the piano for the

pre-set number of bars with the sustain pedal pressed continuously during the recording

3. After the recording stops, stop playing and move the play cursor to the 0 position 4. Close the Transport window 5. Copy the part you just recorded 6. Forward to bar 20

7. Paste the selected track at bar 20 8. Open the VST mixer

9. Move the master fader from the position 0 to position -30.6, or from -30.6 to 0 on

the VST mixer 10. Close the VST mixer

TASK C

1. Select MIDI track 2 2. Select the MIDI part on track 2

3. Open the Key Edit window

4. Draw a note for 3 bars

5. Draw a fade-in volume graph on the volume graph area for 2 bars

6. Close the Key edit window 7. Open the Transport window

8. Perform a Recording by playing on the piano for the preset bars with the sustain

pedal pressed continuously during the recording 9. After the recording stops, stop playing and move the play cursor to the 0 position 10. Close the Transport window

11. Open the EQ window by opening the mixer

12. Turn the Hi-Mid Gain knob to +24

13. Close the EQ and Mixer windows

14. Open the EQ window by opening the mixer

15. Turn the Low-Mid frequency knob to 20000 Hz

16. Close the EQ and Mixer window

REACION TIME FOR TASKS USING DIFFERENT INPUT DEVICES

MEANS ComplexityID STAT TASK 1 TASK 2 TASK 3 ID OVLL

Mean 40.74 62.6374 64.923 56.1BL s 3.92 21.46 20.62

N 10 10 10 30

Mean 52.1371 83.6633 99.1176 78.31TP s 5.92 39.94 25.68

N 10 10 10 30

Mean 40.7399 58.245 62.5232 53.84KA s 6.46 31.63 29.05

N 10 10 10 30

Mean 42.1412 50.0287 59.9132 50.69KW s 6.46 12.45 20.6

N 10 10 10 30

Mean 43.94 63.64 71.62TASK sOVLL N 40 40 40