The X-Motion System

Luiz Velho and Julio Lucio Martins

VISGRAF LaboratoryIMPA

Abstract

In this document, we describe the X-Motion system for interactive real-timegraphics with dance.

The system captures trajectories of points in the dance performer’s body andgenerates motion paths which produce dynamic graphics by procedural visualrules.

Chapter 1

Introduction

The goal for this project was to create a system that enables a performer to de-sign the behavior and control in real-time the graphics that are projected onto thebackground while they are performing.

For this, we have designed a system, called X-Motion, that uses input datacoming from a camera, keyboard or any other input device to generate real timeanimated graphics.

1.1 OverviewThe system captures the position from the markers the dancers are wearing usingan camera sensitive to infrared light (IR). It then extract parameters from the videoimages; position, speed, curvature, etc. of the paths captured for each marker.Then it binds a Finite State Machine (FSM) to each marker. Each state of theFSM is characterized by a graphical behavior that will determine how the drawingparameters are affected for the captured curve parameters. The states transitions(which determines the drawing behaviors) will be fired out by conditions in thecurve parameters (i.e. for example: Speed x > 0.7 pixels/frame). The performercan design the FSMs for the markers, as well as control the drawing state whileperforming.

An overview of the X-Motion System with all its entities is shown in Fig-ure 1.1.

1

Figure 1.1: X-Motion System Overview.

2

1.2 EntitiesThese are the objects and data structures that build our system architecture:

• Cameras and input devices:The input from different input devices is captured.

• Sensing module:Gets input devices data and analyzes acording with the directions of themotion engine to modify some global state parameters.

• User Interface:Is the entitie that let us define the initial parameters used to initialize theprograming module and FSMs.

• Programming module:Takes controll of the program flow:

1. Commands how the data captured is analized.2. Updates motion engine FSMs.3. How the render program renders the data.4. Controls the motors.

• Motion Engine:Takes information from the global state parameters and drives the outputgraphics to be projected.

• Global state:A set of classes that stores information of the parameters calculated at eachprogram update. When the FSMs are updated with these parameters, somestate transition can be produced. The FSMs are part of the global state too.

• Graphics interpretator module:When is updated, gets information from the global state parameters andtranslates into a rendered video output.

• Projectors:Show the rendered images on a screen.

3

Chapter 2

The System

The X-Motion system analyzes a video stream of a dancers performance and thencalculates the time positions of optical markers the dancer is wearing. This timeposition data is then used by the sensing module to calculate parameters to drivethe different states of a drawing finite state machine (FSM), as well as some draw-ing parameters of each state. By creating state machines and events for state tran-sitions we let the dancer design how the performance is projected on a backgroundas well as control the drawings in real time.

2.1 ArchitectureThe system architecture is based on four modules: the sensing module whichcaptures and analyzes the data coming from the input devices; the programmingmodule which manages the procedural library; the motion engine module whichcontrols the Finite State Machine and global state; and the graphics interpretermodule which renders the results.(see Figure 2.1.)

4

Figure 2.1: System Architecture

2.2 Real-Time InputThe performer movements are captured using the an infrared camera and retro-reflective markers that are attached to points on the performer’s body. This infor-mation is analyzed in real time.

2.2.1 Data Capture and TrackingThe data capture system is composed of: camera, incandescent lamps, infraredfilter and reflective spheres. The incandescent lamps emit light that bounces fromthe reflective markers and is captured by the camera. We filter all other higherenergy frequencies so the marker image is more intense than any other sourcein the video. Using libraries from the open source project OpemFrameworks(http://openframeworks.cc/) we then subtract a background image from the cap-tured frame and by thresholding the resulting image we get a black image withwhite circles corresponding to the markers (see Figure 2.2.)

5

Figure 2.2: Sensing module.

2.2.2 Data AnalysisBy using the openCV library we can find contours in the video input image and bygeting the center of mass of these contours we have the position of each marker.The m-markerFinder class have also a method that identifies markers and keeptrack of the identifiers through time. The time position of the markers defines acurve. A bicubic spline interpolation method is used to refine the captured curve.Once we have the set of positions of the marker in time, the system calculatesother parameters for each one of these vertices. We calculate scalar and vectorialspeeds, scalar and vectorial accelerations and curvature.

2.3 Motion EngineThe motion engine is responsible for the real-time interaction of the system. (seeFigure 4.1).

6

Figure 2.3: Finite state machine module.

2.3.1 Finite State MachineThe Finite State Machine (FSM) is defined by the states and the transition betweenthese states. Each state is characterized by a drawing behavior that controls what isdrawn and how the parameters from this drawing will be affected by the capturedcurve parameters (i.e. We can define a state that is drawing a circle for eachcurve vertex, and we could define for this drawing state that the circle radius willchange with the marker speed x component). We store this state graphic behaviorin a paf file which controls which shape will be drawn, which drawing parametersare linked to curve parameters, which values are constants, which are linked tomotors and what are the maximum and minimum values for each of the drawingparameters

Transitions will let the system change from one drawing state to another. Todefine a transition we need to define the initial and final state as well as the eventthat will initiate the transition. The events could be of different types: curveevents, keyboard, mouse, midi. etc... The system analyzes the possible transi-tions for a current state and when a transition event occurs the system changes thedrawing behavior.

7

2.3.2 SchedulerThe scheduler is the core of the system. It is the one that updates all the othermodules. Keeps track of the sequence of tasks of our system in real time. Wecould say that is the clock of our system

2.4 Graphics InterpretationFigure 2.4 shows the graphics interpretation module distribution.

Figure 2.4: Graphics Interpretation module.

8

2.4.1 RenderingThe system keeps track of the markers positions and calculated parameters. Ac-cording to the current state it creates the final output graphics using an openGLwindow showing the results of the interaction. The architecture of the system isdesigned to allow the overloading of several objects of the callBack class. Threetypes of base virtual callBack classes have been implemented: callBackDraw class(responsible for the drawings), callBacAnimation class (handles the 3D animationof draws, not implented jet) , and callBackArray (make 3D array repetitions ofthe drawings, is not implemeted jet). Arrays and Animation CallBack behave likecommutative operators.

9

Chapter 3

Configuration Files

We use a set of text files to define the performer-system interaction results:

3.1 Engine FileIn this file we set the system engine parameters. An example of this file can beseen in Appendix A. Here we tell the engine how many markers will be captured,the number of vertex the system will keep track of for each marker, and the statesmachine file and location for each marker.

3.2 State Machines FileThis is where parameters like the number of states and their possible transitionsare defined. For each state a set of callBack functions are defined, its types ofdrawing and the name and location of the .paf file that defines the drawing behav-ior parameters. An example of this file can be seen at the Appendix B.

3.3 Graphics Parameters FileThe interaction parameters and parameter linkage for this drawing behavior typeare defined here. There are several drawing behaviors types implemented at themoment and new drawing types can be added by a programmer by implementing anew derived class from the virtual curveDraw class. We have tried to generalizedin some way a drawing and have created one implementation for general curve

10

drawing. You can find an exaple of a parameter file for one interaction for thisclass implementation in Appendix C.

Several different types of parameters are defined in this file:

• Booleans and: they can take the values 0 and 1 (false and true) and they aresummable (more than one can be true at the same time).

• Booleans or: They are also booleans , but just one of them can be true.• Booleans And Or: For the main ones that we define like true, just one of thesecondary ones can bne true.

• Integers: Real values from 0 to a maximum value( in the file the first is theinitial value and de second the maximum).

• Floats: Float numbers that can vary between a minimum and a maximumvalue. (initial , minimum and maximum in the file).

After that , the way some of the used parameters change is defined. There arethree different ways they can vary:

• Linked to one of the curve parameters, which is defined by a list of thelinked parameters from the draw with some of the curve parameters.

• Randoms: Parameters that will change randomly each update of the darw.• Linears: That will change from the minimum to the maximum value. Thevariation speed is specified in the file too.

• Oscillers:They will oscillate harmonically between a minimum and maxi-mum value. In the file is specified the speed and frequency of oscillation.

11

Chapter 4

Results

Once the systemwas implemented, we invited a dance performer to try the system.Miss Analivia Cordeiro is well recognized for her research in Labam Parametriza-tion of the dance performance, and was the best ever candidate for our experience.She chose a music and did the choreography for testing our system. After explain-ing to her how the engine works, she chose some of the graphic behaviors shewanted to use, and we tested the system, recording the experience in video andphotos.

4.1 ExamplesHere you can see some of the images taken during the experiment.

Figure 4.1: Unsquare Dance.

12

Chapter 5

Conclusion

There is still much work to do on this system, what we have is just the start ofa much more powerful tool than the one has been developed. The User interfacecould be rewritten using some window menus instead of the text files; more draw-ing behavior classes should be implemented; animation and array for callbacksshould be implemented too. Because all the vector data operations are imple-mented in three dimensions, the capture of data could be done using more thanone camera, so we can capture 3D splines and use the extra z information to han-dle more events for state transitions and braw parametrizations.

13

Appendix A

Engine file example

numberOfMarkers= 2

Machine_for_marker= 0..\\files\\machine00\\machine00_teste.smfnumberOfVertex= 1000000

Machine_for_marker= 1..\\files\\machine00\\machine00_teste.smfnumberOfVertex= 1000000

isInterpolating= 0interpolatGrade= 0.5

14

Appendix B

States fachine file example

numberOfStates= 7state= state01numberOfFunctions= 1type= 1curveDrawType= 7parametersFile= ..\\files\\machine00\\curveGeral00_estado01.paf

state= state02...state= state07...

numberOfTransitions= 6

transition= tras01To02fromState= state01toState= state02numberOfEvents= 1type= 2key= 109......transition= tras05To06fromState= state05toState= state06

15

numberOfEvents= 1type= 1parameterName= vertexSpeedconditionOperator= <conditionOperator= !=value= 0value= 10000numberOfCallBacks= 0......

16

Appendix C

Parameters array file example

BOOLEANS_AND9exagero_curva 0balana_curva 0ondula_curva 0expande_curva 1junta_curva 0move_curva 0cheio 1repete_curve 0gradual 1

BOOLEANS_OR0

BOOLEANS_AND_OR1

brush10linha 0linhagrosa 0linhagrosa_discontinua 0circulo 1circulo_azar 0

17

ellipse 0poligono 0estrelha 0ameba 0tinta 0

INTEGERS7poligon_sides 4 6

estrelha_pontas 5 25

ameba_logitude_1 10 50ameba_logitude_2 10 50ameba_radio_centro 5 10ameba_radio_lados 2 10

discontinuidade 3 10

FLOATS30grado_exagero .1 0 1altura_exagero 5 0 50

grado_separa 2 2 6grado_junta 2 0 10

frecuena_balano . 09 0 2amplitude_balano 1.0007 1 2

frecuena_onda 0.09 0 2amplitude_onda 20 10 50

velocidade_move_x 0 -3 3velocidade_move_y 1 1 2

radio_circulos 5 50 4

18

poligon_radio 15 10 30poligono_velocidade_rotaao 0.001 0.001 0.1

estrelha_radio_1 3 2 10estrelha_radio_2 10 3 25estrelha_velocidade_rotaao .1 0 10

coor_curva_r 255 0 255coor_curva_g 0 0 255coor_curva_b 0 0 255coor_curva_a 100 0 255

num_vertices 100 0 100000

grosor_linhagrosa 2 0 20

repete_direction_x 0 -1 1repete_direction_y 1 0 1repete_distancia 1 0 50repete_numero 21 2 30

radio_ellipse_x 15 5 10radio_ellipse_y 40 10 40

radio_circulos_azar_min 2 10 20radio_circulos_azar_max 8 20 40

LINKEDS_LIST2coor_curva_r vertex_position_xcoor_curva_g vertex_position_y

RANDOMS3

LINEALS2

19

coor_background_g 0.1coor_background_b 0.1

OSCILERS0

vertex_position_xvertex_position_yvertex_direction_xvertex_direction_yvertex_speed_xvertex_speed_yvertex_normal_xvertex_acceleration_xvertex_acceleration_yvertex_scalar_speedvertex_scalar_accelerationvertex_curvaturecurve_time

20