Musical Virtual Reality Applications
description
Transcript of Musical Virtual Reality Applications
![Page 1: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/1.jpg)
Musical Virtual Reality Applications
Michael Kriegel
![Page 2: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/2.jpg)
Contents
My 2005 final MSc project at the Centre for Virtual Environments (University of Salford)
A current Heriot Watt student project: Wii Piano by Thomas Methven
![Page 3: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/3.jpg)
Motivation
Personal interest in electronic music technology (sampling, synthesizers, MIDI)
Use VR as a new interface to electronic musical instruments
Original idea: create a Virtual Reality drum kit interface with optical tracking input
Why do this? Might provide interesting musical performance good method of testing/measuring optical Vicon tracking
system Virtual interface doesn’t take up space (many people
don’t have space for a real drum kit)
![Page 4: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/4.jpg)
Research Questions
Can the feeling of playing the drums be replicated with a VR interface?
Is optical tracking suited (fast and accurate enough) for this task ?
![Page 5: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/5.jpg)
Literature Review 1
First initial literature review strived to find similar comparable projects
Not much on the academic sector: University of Twente in Netherlands had a similar project,
but used different technology – also no papers on that project
Axel Mulder, Phd thesis, Simon Fraser University, Canada – data glove / hand gestures to create sound
Interdisciplinary Centre for Scientific Research in Music (ICSRiM) at University of Leeds have a research framework “Music Via Motion” that tries similar things, however again with different technology
![Page 6: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/6.jpg)
Literature Review 1
Related commercial and art projects: Electronic Drum Kits Theremin, Roland‘s D-Beam controller, Alesis Air
FX Eye Toy Soundbeam Myron W. Krueger‘s installations
![Page 7: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/7.jpg)
![Page 8: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/8.jpg)
Technology - MIDI
Musical Instrument Digital Interface A protocol that allows electronic musical
instruments to communicate with each other (or with a computer)
A MIDI instrument (or computer) usually has a MIDI-in port to receive messages and a MIDI-out port to send messages
MIDI protocol defines messages: e.g. Note-On, Note-Off
![Page 9: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/9.jpg)
Technology - Creating drum sounds electronically
Can be easily done via Samplers, Drum Machines, Synthesizers, etc. (both hard- and software)
For my project: Using a free sampling software and a pre-defined drum kit (collection of drum sound samples) that can communicate with the application via MIDI
![Page 10: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/10.jpg)
Technology – optical tracking
Vicon tracking / motion capture system consists of 5 infrared cameras Dedicated computer for pattern recognition Pattern recognition software
![Page 11: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/11.jpg)
Technology – optical tracking
Objects that need to be tracked are equipped with reflective markers
Objects need distinctive marker patterns By observing the marker pattern from the 5
different perspectives, the objects position and orientation can be calculated
Software has training mode to learn to recognize new objects
![Page 12: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/12.jpg)
Tracked objects
Shutter glasses
Drum sticks
Feet
![Page 13: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/13.jpg)
Technology – Display System
Barco Trace Large Screen Display Back Projection Screen with 120 Hz refresh
rate for 1280 x 1024 resolution Active Stereo with
stereo shutter glasses
![Page 14: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/14.jpg)
System Design
![Page 15: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/15.jpg)
Implementation Steps
Create 3D Models of Drum Kit with 3DS Max Import them scene graph Create interface with tracking system
(glasses, hand, feet) Create MIDI connection between application
and Software Sampler Implement collision detection between virtual
drum stick positions and drum skins (collision triggers Midi out note, velocity of impact determines the volume of the sound)
![Page 16: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/16.jpg)
Main Program Loop
Get current position of sticks Calculate if a collision has occurred (also compare with
previous position to avoid continuous sounds) If collision has occurred
Calculate speed (again by comparing with the previous positions and their timestamps)
Send a midi message (Note On) with Note number relating to the object that was hit and volume relating to the speed
Update Scene (set new position of sticks) Render Scene Could be improved by seperating the collision
detection/sound output and grpahics in different threads
![Page 17: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/17.jpg)
Screenshot
![Page 18: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/18.jpg)
Problems discovered
Missing haptic feedback seemed to be a major drawback
Some more research into haptic devices that could be used in this context, Spidar seems to be the only well suited device for drum sticks, but not available at our department
![Page 19: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/19.jpg)
Change of Research Direction
Since the haptic path was a dead end due to availability of hardware …
and I was ahead of schedule with the implementation..
The implementation of two other virtual instruments was decided: Virtual Piano – to compare both tracking performance
and haptics with drums Virtual Synth Control (Theremin like experimental
instrument) – to compare replications of traditional instruments with experimental instrument
![Page 20: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/20.jpg)
Virtual Piano
Changes: New controller: gloves Different MIDI sound
module:free software piano
Piano model generated by the application and not by modelling package
Different collision detection algorithm (need note-on and note-off events)
![Page 21: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/21.jpg)
Screenshot
![Page 22: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/22.jpg)
Virtual Synth Control (VSC)
Design objectives: Easy to use No haptics required Psychedelic look and sound
Input mode: drum sticks Sound module used: Free Software
synthesizer JX 220
![Page 23: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/23.jpg)
Screenshot
![Page 24: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/24.jpg)
Evaluation
Main Purpose was to compare the user’s reception of the 3 instruments.
12 subjects Multiple choice questions:
Which instrument did you like most? Which of the instruments, if any, could be best used for
making real music? Majority chose VSC for both questions (10/12 and
11/12) No-one chose “none” so everyone could see some
potential in the technology
![Page 25: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/25.jpg)
Evaluation
Another set of questions was aimed at evaluating all instruments individually on a scale of 1 to 7
Some findings: No depth perception problems with any of the
instruments (question 1) Lack of haptics is severe, even more so for the
piano (question 2) People found it hard to drum to a rhythm / easy
to play VSC along with backing track (question 3)
![Page 26: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/26.jpg)
3 ½ years later
![Page 27: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/27.jpg)
Wii Piano
Basic idea: use the WiiMote’s infrared camera for very cheap finger tracking
2D Compare with other input methods (e.g.
mouse, touch screen) Applications: Drawing, Piano
![Page 28: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/28.jpg)
WiiMote
Besides the usual buttons the WiiMote has 2 novel forms of input: Accelerometers Infrared Camera
![Page 29: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/29.jpg)
WiiMote Finger tracking
Normally the infrared camera is used to evaluate the WiiMote’s position relative to the Wii’s sensor bar
For WiiMote Finger tracking, the WiiMote’s camera is used other way round:
WiiMote doesn’t move Infrared Light source moves
![Page 30: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/30.jpg)
WiiMote Finger tracking
No active light source on finger is used but reflection (see passive markers in mocap)
![Page 31: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/31.jpg)
WiiMote Finger tracking
![Page 32: Musical Virtual Reality Applications](https://reader035.fdocuments.in/reader035/viewer/2022062221/56813dfd550346895da7d5b1/html5/thumbnails/32.jpg)
User Interface: Button Activation