MotionMotionSensors Sensors + Audio Visual + Audio Visual Performance:Performance: Using Kinect to Control Live Visuals and Sound

by Joey Bargsten, Ph.D. Florida Atlantic University • SCMS • MTEn

MotionMotionSensors Sensors + Audio Visual + Audio Visual Performance:Performance: Using Kinect to Control Live Visuals and Sound

by Joey Bargsten, Ph.D. Florida Atlantic University • SCMS • MTEn

Thanks!

http://film-gate.org

http://sweatrecordsmiami.com

Format/Overview:

• Presentation / Demo (noon – 12:40)

• Questions? (12:40 – 12:50)

• Break (10 min)

• Workshop on PureData (1pm – 2pm)

Background on Kinect

Background on Kinect

• Released November 2010

Background on Kinect

• Released November 2010

• Sold 8M units in 60 days

Background on Kinect

• Released November 2010

• Sold 8M units in 60 days

• SDK Drama!

Kinect (Development)Community• Gaming and Virtuality

Kinect (Development)Community• Gaming and Virtuality

• Medicine

Kinect (Development)Community

• Gaming and Virtuality

• Medicine

• Surveillance, Security, Facial Recognition

Kinect (Development)Community• Gaming and Virtuality

• Medicine

• Surveillance, Security, Facial Recognition

• Robotics

Kinect (Development)Community• Gaming and Virtuality

• Medicine

• Surveillance, Security, Facial Recognition

• Robotics

• 3D Scanning and Printing

Kinect (Development)Community

• Gaming and Virtuality

• Medicine

• Surveillance, Security, Facial Recognition

• Robotics

• 3D Scanning and Printing

• Avant-Garde Art, Music, and Performance

Suggestions,ApproachesSystems & Configurations

My Configuration

• Kinect Sensor (2011); Model 1414, EN/ES US HD

(this is the most ‘open’ one!)

• Synapse Middleware (converts skeleton to OSC)

• PureData v. 0.43.4-Extended (incl. MrPeach library)

• Quartz Composer v. 4.0 (+ old FW400 SD

camcorder)

• 2011 MBP, 2.4mHz, running OSX 7.5 “Lion”.

Approaches to Performance• One-Person Band (virtuoso!)

– One person, one controller

– Controls both audio and visual

Video on YouTubehttp://www.youtube.com/watch?v=1ge7RcIBWsc

Approaches to Performance• Division of Labor (conventional

ensembles)

Approaches to Performance• Division of Labor (conventional

ensembles)

“For every new technology that comes along,

make an ensemble out of it!”—JB, EMV (2011)

Approaches to Performance• Division of Labor (conventional

ensembles)

“For every new technology that comes along,

make an ensemble out of it!”—JB, EMV (2011)

• Applied to Kinect:–Multiple controllers –Dedicate separate software to audio and visuals–Multiple performers for audio and visuals

What Do You Want to Control?

How Do You Want To Control It?

• Discontinuous, trigger events

–model: keyboard

• Continuous, gradual events

–model: theremin

• Hybrid Control: multiple parameters

–model: sample player

Kinect Control Parameters

Kinect Kombo™:Quad Theremin + Sample Player

• Built in PureData • Free, cc-by-d-nc

Kinect Kombo™:Skeleton View + Performer Interface

Kinect Kombo™

Kinect Kombo™

Controlling Visuals

Kinect and

Quartz

Composer

TheAugmentedDance:

LiveBody—Kinect— VirtualBody

Variations• Figure

– Simple Camera Figure and Skeleton

– Primitives, 3D Objects, Video Mapped to

Skeleton

• Paint Brush/Image Hose (Effects Mapped to

Skeleton)

• Background (Texture or Stylized Video Controlled by Skeleton)

“My Profundis (2012)”

Video on YouTube http://www.youtube.com/watch?v=gT1lbhR1Qj8

Resources

• OpenKinect.org

• OpenNI.org

• PureData.info

• Synapsekinect.tumblr.co

m

• CreateDigitalMusic.com

• CreateDigitalMotion.com

• V002.info (vade &

bangnoise)

• Cycling74.com

• IdeaChannel ‘Kinect’

Episode

• OffBook ‘Creative Coding’

Episode

• www.fau.edu/scms (MFA -

MTEn)

• www.badmindtime.com

• Melancholalaland™ (my

multimedia opera)