Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

35
Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang

Transcript of Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Page 1: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Interactive Sand Art DrawUsing RGB-D Sensor

Presenter : Senhua Chang

Page 2: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Main Purpose

Present an interactive sand art drawing system which supports the common sand drawing functions, such as sand erosion, sand spilling, and sand leaking by using one RGB-D sensor, such as Kinect.

•Firstly, system adopts a vision based bare hand detection method which computes the hand position and recognizes the hand gestures.

•Secondly, the drawing position and the drawing action are sent to the sand drawing system.

•Finally, the sand drawing system performs the corresponding sand manipulation action.

Page 3: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Introduction

What is Sand Art ?

Sand drawing is a kind of drawing technique that people manipulate the sand on beach or sandy ground to create beautiful pictures. Sand art is a style of live performance art that introduces an indoor and dynamic way of sand painting. Sand art Performance is improvisational and continuous.

•How about see a good video to get experience what is Sand Art ?

Page 4: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Introduction

Critical issues that need to tackle

•First, we need to use the gestures of one hand to control the drawing actions. •Second, once the drawing actions are detected, we need to analyze the motion and gesture of another hand to draw virtual sand.

Page 5: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Introduction Critical issues that need to tackle

A vision-based hand tracking system using one RGB-D sensor is used which is integrated with the sand drawing system.

The hand tracking system computes the positions of the hands of a user and also recognizes the hand gestures.

Then, the hand tracking system drives the sand drawing system to draw sand.

Page 6: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Introduction

Why Kinect ?

Kinect is a low-cost RGB-D sensor.

What is RGB-D sensor?

RGB plus Depth ( represent as a gray pic)

Each pix represent the depth from sensor to objective.

Page 7: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Introduction

Page 8: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Related Work

HCI based on hand interaction

There are fruitful techniques that have been developed in the field of HCI, such as hand detection and gesture recognition techniques. The resulting information can be applied to interact with virtual objects. As a hand has high degrees of freedom, it is critical to compute hand features that are needed in performing HCI.

The hand detection and gesture recognition are two major tasks that should be performed properly in the applications of using hand interaction.

Page 9: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Related Work

HCI for artistic authoring

The HCI itself is an important element in designing artistic authoring system. In painting, the writing pad is an intuitive tool that people can use the stylus pen to draw directly on the drawing surface.

Haptic-based interface

Gesture- based interface

Page 10: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Related Work

Particle System and Sand Simulation Sand has liquid-like fluidity. But its shape is solid when it is static. Also, the sand drifts in the air like gas.

Since sand has so interesting motion behaviors, the methods for sand simulation are studied extensively in graphics.The methods can be categorized into particle based methods and height map based methods. Also, there are hybrid methods that combine both a particle system and a height map to simulate sand.

Page 11: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Related Work

Particle System and sand simulation There are techniques which are devoted to sand art called SandCanvas which is a multi-touch media for sand art drawing. The system support a rich set of gestures to manipulate sand. The user study indicates that the system is useful in creating interesting sand art pictures. The sand drawing system combined a particle system and a height map. For the sand in the air, particles are used. If the sand piles up on the drawing plane,

the sand is converted into a height value. Thus, the computation of the approach is

efficient.

Lets see a video about SandCanvas.

Page 12: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

System OverviewSystem allows A user can use gestures of one hand to select the action and uses the gestures of another hand to manipulate sand in the virtual environment.

system consists of three phases: image preprocessing, hand feature extraction, and motion/gesture management. The purpose of the image pre-processing phase is to eliminate the noise in the captured data and the regions that do not belong to a hand, such as human face. After the image is filtered and the portion for a hand is kept, we can compute the features of the hand, such as hand position, and then determine the hand gesture. The motion/gesture management is the bridge between the user and the sand art drawing system. The extracted hand data, such as the position of the hand and the hand gesture, are passed to the sand art drawing system. Then, the virtual sand is drawn in the virtual environment.

Page 13: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

System Flow Chart and Interface

Page 14: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

System Flow Chart and Interface

Page 15: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

System

Image Pre-processingIn the image pre-processing phase, our goal is to remove the redundant portions that do not belong to hands in the captured color and depth images. In this way, the search space for the hands can be reduced. To extract the portion of the hand in an image, we perform background subtraction, depth segmentation, and skin color detection.

Page 16: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

System

Image Pre-processing

Depth segmentation

Page 17: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

System

Hand feature extractionIn the hand feature extraction phase, the goal is to analyze the image and extract the information that is useful for the application of sand art drawing.

Page 18: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

System

Hand feature extraction

Step1: distance transform

Step2: finger extraction based on the time series curve.

Step3: Computation of the finger position

Page 19: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

System

Motion/gesture management After studying user's habits in sand art drawing so that we can properly design the key gestures for the sand art drawing system. We also note the following facts about sand drawing actions:

1. When users use their hands to erode the sand, they tend to use only one finger. The index finger is used the most.

2. In sand spilling and leaking, the users grab the sand and release a portion of the sand from their hands.

3. In area erosion, users may open their palm, and use their palm to feel about the sand and erode the sand within the area.

Page 20: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

System

Motion/gesture management

Page 21: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

System

Motion/gesture management

Page 22: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Experimental Result Environment

Page 23: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Experimental Result Environment

Page 24: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Experimental Result Environment

Page 25: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Experimental Result

Result of gesture recognition

Page 26: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Experimental Result Result of sand art drawing

Page 27: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Experimental Result User testing study

Page 28: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Critique thinking

This paper elaborate what is this sand art system and how it implemented.

Do a lot of experiment, and the result is convincing.

To much details can seem as an advantage and a shortcoming. ( A lot of paragraphs appear repeatedly)

Page 29: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

MotionDraw: A Tool for Enhancing Art and Performance Using Kinect

Presenter : Senhua Chang

Page 30: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Introduction Contemporary staged performances frequently utilize advanced lighting and projection techniques. The design and creation of these stage effects are rarely accessible to actual performers and must be designed by professional lighting designers or highly-paid programmers. With MotionDraw we want to create an affordable system that is easily controlled and manipulated by performers. With intuitive gestures, non-specialized users can control the MotionDraw visual library and interact with the captured visual record of their own movements.

Page 31: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Introduction Computer graphics, projection techniques, animations,

and professional lighting enable effective stage effects in

large-scale events that support the combination of live performance, such as dance, and digital arts. However, creating those exciting environments comes with the high cost of specialized hardware and often with dedicated software developed by professional programmers. MotionDraw, a Kinect-based tool that augments performance with projected visualizations created from tracked live movements of users, enables artists, performers, dancers and even the audience to easily design, create and control hybrid digital performances.

Page 32: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Related work Problem

(1) the choreographer can be in close

conversation with the computer operators and graphic

artists, dancers and choreographers can not easily make

changes to the graphic display on their own.

(2) Projects like Sandde allow for three-dimensional digital

drawing using gesture, but rely on hardware (a magnetic

pen) to track movement.

Page 33: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

The MotionDraw SystemMotionDraw consists of a set-up with one or two Kinects,

that record the position of joints of a person's body over

time and draws the resulting trail on a 3D canvas. One

Kinect points at the dancers and tracks their movements.

A GUI-based control panel, enables a separate user to

change the parameters of the projected drawing in

real-time, such as (a) change the color and width of the

trails, (b) move the position of the camera-view, and (c)

enable or disable the visualization of specific joints

tracked. The program also saves the Kinect stream to do for further post-production.

With a two-Kinect set-up the second Kinect

captures a "conductor" who can interact directly with the MotionDraw system and change the visualization parameters through gestures. This integration is covered in more details below

Page 34: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

Future Work

Page 35: Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang.

All is overThank you very much