Post on 06-Apr-2018
8/2/2019 Teleimmersion Final Report
http://slidepdf.com/reader/full/teleimmersion-final-report 1/10
Final Report HART Demonstrator Project
Lisa Wymore – Assistant Professor, UC Berkeley Department of Theater, Dance,
and Performance Studies
The project is entitled:
Portable Durable Tele-Immersion Technology for Teaching, Artistic Performance,
Global Networking, and Experiments in Creativity and Collaboration
The project was initially conceived in Summer 2008. At that time I was creating a large-
scale performance project in conjunction with the Merce Cunningham Dance Companyresidency at UC Berkeley for a November 2008 performance. Cal Performances, the
prestigious presenting organization at UC Berkeley, produced the event on November
14th. The perfor mance was entitled: Panorama-Multi-Media
Happening
and involved
numerous art and technology units across campus. For Panoram
a
I continued my creative
explorations with Tele-Immersion, a technology that I have been working with since
2005 with the labs here at UC Berkeley and at the Univer sity of Illinois predominately.
For this project it became clear that we needed to create a portable Tele-Immersion
system for the November 14 performance so that Tele-Immersion could be experiencedt h
by the audience members attending the event. So, working with Ruzena Bajcsy’s Tele-
Immersion lab within the Electrical Engineer ing and Computer Science Department at
UC Berkeley, we cr eated two portable systems that existed 100 feet apart from each
other. User s could engage with the system and see one another across the performance
space and also on large monitors sharing the virtual “meeting environment”.
For those unfamiliar with Tele-Immersion technology, it can best be described as adistributed collaboration tool that utilizes 3D image based rendering of users in real time.
Remote users can meet in virtual envir onments and share virtual worlds together, not as
traditional avatars which use animation graphics, but as new kinds of avatars that use real
time motion and video based images of users. Further more, any 3D data can be uploadedto the virtual environment to be investigated within collaborative virtual settings.
Examples of the kind of work we do in the UC Berkeley Tele- Immersion lab can be seenat the following website:
http://tele-immersion.citris-uc.org/video
This HART Demonstr ation Project funded a graduate student, Ram Vasudevan, to create
a more robust ver sion of the portable Tele-Immersion system to be used within a
dedicated dance studio space within my Department of Theater, Dance, and Performance
Studies. The larger goal of this project is to create what I am calling Integrated Distance
Labs ( ID-labs) within other art/dance based departments across the country and hopefullythroughout the world, that allows a new kind of choreography to emerge. We are also
working on the integration of body based sensors and sound within the system. We have
tests with UC Merced and UC Davis planned this coming spring 2010, as well as tests on
the UC campus between the dance studio “I D-lab” and the Tele-Immer sion lab in
Electrical Engineering and Computer Science. Furthermore, in spr ing 2011 an entireevening of perfor mance will be dedicated to questions around dance and technology and
8/2/2019 Teleimmersion Final Report
http://slidepdf.com/reader/full/teleimmersion-final-report 2/10
the portable Tele-I mmersion lab, that HART has funded, will be at the heart of this
perfor mance.
I mention these things because they address the major issues that we are tackling with
Tele-I mmersion at this time: 1. The robustness and transferability of Tele-I mmersion
technology to a larger public, and 2. The integration of sound and body based sensors
into Tele-Immersion technology, in real time, to enhance remote user experience andcollaboration within the system. The hope is that the expansion of Tele-Immersion labs,
using the same vision and rendering protocols will allow for a network of distributed labs
to communicate and collaborate around a variety of 3D data and information.
The results that we have produced from our Demonstration Project funding:
1. A highly functional lab has been built in Zellerbach Room 170 dance studio. This
space is now a dedicated “dance lab” and will soon become an Integrated Dance
Lab (ID- lab) with body based sensors, surround sound speakers and dedicated
Tele-I mmersion technology installed into the space permanently
2. All equipment needed for the Portable Tele-Immersion system has been
purchased and tested.
a. Projection equipment is installed
b. Special stage lights have installed
c. Internet has been installed (3 IP addresses)
d. Adequate electricity has been installed
e. Proper projection sur faces and flooring has been installed
3. The renderer necessary for bi-located Tele-Immersion has been updated in
collaboration with UC Davis’s CAVE labs.
4. Intense software updates and code updates have been completed by Ram
Vasudevan (funded by the HART Demonstration project) and the numer ous other
computer scientists and engineers working in the UC Tele-Immersion lab.5. A test between the two UC Berkeley Tele- Immersion labs (dance studio and
EECS lab) has occurred using one of the 3D cameras. (see images below) Later
this f all we will test with four 3D cameras.
6. Future tests are scheduled between UC Merced and UC Davis with regard to
Tele-I mmersion technology in general. See website listed above for videos
showing the kinds of collaboration that we are working on.7. Other Dance Programs and Departments are showing interest in developing Tele-
Immersion (ID-labs). Namely UC Santa Cruz and UC Irvine will be working in
partnership with the UC Berkeley labs over the next two years.
8/2/2019 Teleimmersion Final Report
http://slidepdf.com/reader/full/teleimmersion-final-report 3/10
8/2/2019 Teleimmersion Final Report
http://slidepdf.com/reader/full/teleimmersion-final-report 4/10
Regarding my academic efforts: I t is hard for me to repor t on this in the tr aditional sense.
I am a chor eographer working with electrical engineers and computer scientists on ahighly complex and multi-modal project within a research based institution. The only
way that I can answer this is to say that the Demonstration Project is inspiring on many
levels to me and in a sense inspiration and creativity are my resear ch. Dance involves
bodies in space and time and Tele-Immersion technology prioritizes bodies in space andtime. It also involves people collaborating together using visual feedback, which is
another means by which dance choreography is made. So in essence, by work with Tele-
Immersion in general has been extremely helpful to my research as a dancechoreographer. With regard to the HART demonstration project, bringing a portable
Tele-I mmersion lab into a dance studio is unprecedented.
It is my hope that by creating this lab it will become a beacon for other departments to do
the same so that a new kind of body based interactive mechanism for collaboration and
communication can be developed. This tactic has been working. As mentioned above,
other UC dance departments ar e getting involved and a renowned dance and technology
artist from the University of Utah, Ellen Bromberg, will be on campus in spring 2011 towork directly with the dance studio Portable Tele-Immersion lab, funded by HART. So
this pr oject has had incredible impact on my visibility as a dance ar tist who engages with
and collaborates with technology. This has had a positive effect f or the public perception
of UC Berkeley’s engagement with new technologies in relationship to the arts in
particular.
With regard to impact within the larger Project Bamboo community, it is difficult to say.Project Bamboo prioritizes easily shared programs and established digital systems rather
than development of physical labs. It is my hope that Tele- Immersion technology remains
one of the future concepts for Pr oject Bamboo to invest in. One can imagine that a
collaborative 3D environment in real time, based on real body movements and real bodyimages will one day be highly usef ul and “gr een” technology. It goes beyond traditional
webcasts or tele-conferences because information can be viewed from infinite angles, the
user can engage with the system as either first person or viewer, there is infinite amountof digital data that can be used within the system for analysis, collaboration, and
archiving purposes. In some ways it is a kind of futuristic library or public
space/commons – a place to come and find information, meet in virtual rooms or
environments, make things in collaboration, explore through multi-modal approaches
(vision, sound, body based sensors, robotic interaction with data, etc.)
8/2/2019 Teleimmersion Final Report
http://slidepdf.com/reader/full/teleimmersion-final-report 5/10
8/2/2019 Teleimmersion Final Report
http://slidepdf.com/reader/full/teleimmersion-final-report 6/10
5. How do we tackle issues of internet bandwidth for real time interactivity.
6. Is off-line data useful to other humanists? And should more of f-line engagement
with Tele-Immersion be investigated?
Regarding lessons learned from this project:
As I write this section I am realizing that I am continually learning from this project. Themajor concepts and ideas that I take away from this project are:
1. It is essential to have engaged and talented graduate students working on multi-
disciplinary projects. I cannot speak highly enough of Ram Vasudevan, who was
the graduate student funded thr ough this Demonstration Project.
2. Collaboration on a pr oject such as this requires weekly meetings and involvement
from all who work in the lab; even if other projects seem unrelated the exchangeof infor mation and ideas that comes f rom meeting is essential.
3. Technology labs are filled with extremely creative people who often don’t always
speak the same language. And being an artist entering this world it requires
patience to become an active participant. Interaction over a long period of time,
sharing of presentations and regular meetings are essential for success.4. It takes much longer to accomplish a specific goal within a collaborative setting.
Often the engineer s that I work with ar e engaged with other research projects and
it takes time and perseverance to create a successful collaboration between artists
and technologists.
From the technologist perspective Ram Vasudevan, has learned much from his work on
this Demonstration Project. He has established an entirely novel way of dealing with
images and texture within the Tele-I mmersion system. All together there are 48 cameras,which can simultaneously capture images with 15 – 20 frames per second. The clusters
are arranged around the user to capture a full-body 360-degree model in real time. Each
of the clusters performs stereo calculation on pairs of cameras by matching reliablefeatures. The features are selected using a novel approach by first performing
triangulation of the image, which divides the image into triangular segments of various
sizes depending on the texture and color consistency of the scene. The homogenous (i.e.
textureless) regions result in larger triangles while the textured regions have many small
triangles. The stereo matching is performed on the acquired triangle nodes.Using this tr iangulation has several benefits, (1) it helps find reliable features, (2) it
reduces the number of matches that need to be performed, (3) it keeps a global structure
of the 3D representation, and (4) it allows for efficient compr ession of the data. This
adaptive triangulation allows for fast (under 25 ms) and robust real-time 3D
reconstruction. The reconstruction rate is, however, limited by the image acquisition rate
of the cameras and the networ k bandwidth to the frame rate of about 20 FPS.
Once the stereo calculation is performed, the triangulation is encoded along with thecolor and dispar ity information for each node. The data is packaged and sent to the local
renderer or remote gateway. The gateway controls the network traffic by routing the
packets over long distances to minimize the delays. Once the renderer receives the data,
3D mesh is extracted and transformed into a common coordinate system. The renderer
8/2/2019 Teleimmersion Final Report
http://slidepdf.com/reader/full/teleimmersion-final-report 7/10
needs to have exact information of the camer a clusters, which is obtained by prior
geometric camera calibration. The render ing algorithm combines the information from
diff erent views by weighted contribution from different clusters, (overlapping clustersmay reconstruct the same surface) using real-time ray-tracing technique to generate
appropriate images in the visual buffer.
Future Directions:As one can see from the above information, Tele-Immersion technology is a complex
system involving endless calibration, software and hardware updates, and
synchronization of data from distributed labs. The Demonstrator Project f unding has beenessential in establishing a kind of codified protocol for futur e work in this area. Ram’s
mathematical calculations, attention to calibration and his “cleaning up” of the code have
made for a more robust and realistic experience. Gregorij Kurillo, the Head Engineer in
the lab, has also added to the system incredibly by establishing renderer protocols,
expanding the visual perspective within the system, adding new 3D data, establishing
collaboration with UC Merced and UC Davis, and much mor e than I can list here. Also,
post-docs, grad students and visiting computer scientist from all over the world have
contributed to the lab in areas r elated to visual texture, human motion detection,skeletonization, sensors, robotics, and the like. It is a VERY active lab, led by Ruzena
Bajscy.
I have ever y intention of continuing my work as collaborator in Tele-Immersion
technology – adding my expertise with regard to body movement, collaboration
protocols, creativity enhancement, live performance engagement, and improvement of
computer human interaction. Another aspect of the pr oject that I would like to seek support for relates more to project Bamboo and that is the creation of an inter active web-
site (or shared tool) where protocols, programs, lab designs, best practices, equipment
lists, data sets, results from experiments, publications, ongoing discourses, etc. are made
public. The desire is to foster this kind of open engagement with the public so that Tele-Immersion technology is adopted within other university/research settings and
infor mation is shared.
As I mentioned earlier in this report, my future work will involve the creation of an
Integrated Distance Lab (I D-lab). I am currently working with Po Yan, and engineer
within the UC Berkeley Tele-Immersion lab, to work with body based sensors and sound.
Imagine a moment when a dancer enters a 3D Tele-Immersive environment surrounded
by multiple 3D digital cameras and displays, where internal and external cues for creative
movements come not only from physical objects located within the 3D Tele-Immersive
virtual space but also from remote participants who are geographically distributed, yetvisually present in the virtual space. The dancers collaborate with one another utilizing
visual cues to solve choreographic problems or to investigate new movement potential
within this virtual dance studio. Suddenly the choreographer has an entir ely new
paradigm within which to make creative choices. Now imagine adding a somatically
derived oper ation to this model – the addition of body-based sensors whose outputs can be synchronized to sound, and this sound can be streamed, in near real time, and with
8/2/2019 Teleimmersion Final Report
http://slidepdf.com/reader/full/teleimmersion-final-report 8/10
some synchronized accuracy, to the visual data being received by the user . The 3D Tele-
Immersion system will now allow user s to not only see one another but also “hear” each
other. In a way it is like they can sense each other’s bodily responses through the digitaldivide and make movement decisions based on this information.
The sensors can be synchronized to realistic body sounds ie: the heartbeat, or they can be
linked to musical digital interfaces that allow for infinite sonic possibilities - using avisual programming language f or multi-media called Max/MSP1. The dancer s will then
be able to perceive each other through both visual an
d
audio feedback within the virtual
dance studio. Offline, and in the "real! dance studio, the main way dancers sense eachother is through listening to breath patterning and through the sensing of other dancers’
levels of exertion and effort. The ID-lab seeks to replicate, albeit through a different
mode of awareness, the feelings that dancers experience within "real” dance studios. The
ID-lab will do this by attaching small, very sensitive, microphones near the user’s mouth
and also by adding two sensitive body sensors: a physiological biosensor that captures
heart rate and br eath patterns; and an accelerometer that captures velocity of gesture and
spatial intent of gesture.
Using MaxMSP the output from these sensors will be converted to sound files locally and
streamed to the participating Integrated Distance Labs. The user at each site will hear the
streamed sound files from the participating Integrated Distance Lab via a surround sound
speaker system. Each lab will be equipped with the same sound system for calibration,
and synchronization purposes. With the sound being experienced "around! the user and
the sounds being heard constructed from the users’ bodies themselves, a r ich audio wor ld
will be created. A computer scientist and signal-processing expert, Gerald Friedland, isadvising the pr oject on the audio and sensor data streaming parts.
The Demonstrator Process: Should demonstrators be included as part of future Bambooactivity? I f so, how would you improve the demonstrator process? I have just a few
suggestions:
1. I felt that the day the UC Berkeley HART funded projects wer e able to sharedisplays with one another was really useful. It was a kind of mini-conference
where ideas were shared, people got to mingle and look at displays/tables, and
talk about their projects with on another. I wonder how that collegially experience
could continue? Per haps another mini- conference or open studio to visit each
other’s work areas?
2. I wonder if Demonstration Projects fr om other campuses could post slides or
mini-presentations on the Bamboo site? Perhaps they do already and I am missing
this? But it relates in some ways to the “story collection” or “recipe gathering”that Project Bamboo is wor king into their system.
3. I would love to be able to demonstrate the Portable Tele-Immersion system
sometime at a Project Bamboo conference, and see other practice/site/lab-based
technologies as well demonstrated live.
4. I miss seeing a kind of live, body based interaction with technology within theBamboo community, but this is my bias. I realize that this type of wor k is not
8/2/2019 Teleimmersion Final Report
http://slidepdf.com/reader/full/teleimmersion-final-report 9/10
necessary considered “shared”. That is can we interact with technologies at
Bamboo conferences more? We sometimes have web casts but what else can we
“work with” live in real time?
Thank you so much for funding my wor k as a Demonstration Project. I look f orward
to being part of the Bamboo Community for a long time to come!
Lisa Wymor e
8/2/2019 Teleimmersion Final Report
http://slidepdf.com/reader/full/teleimmersion-final-report 10/10