Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video...
Transcript of Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video...
![Page 1: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/1.jpg)
Immersive Multimedia and
Virtual Crowds
Farzad Safaei
![Page 2: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/2.jpg)
Farzad Safaei
Slide 2
Physical and Virtual
Mixed reality continuum
The boundary between physical and virtual is
being blurred
Augmented Reality Augmented Virtuality Real Virtual
![Page 3: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/3.jpg)
Farzad Safaei
Slide 3
Augmented reality
Augmentation of physical with virtual
![Page 4: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/4.jpg)
Farzad Safaei
Slide 4
Augmented virtuality
Augmentation of virtual with physical
![Page 5: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/5.jpg)
Farzad Safaei
Slide 5
Augmented virtuality challenges
• Presentation of video and audio within a virtual
space
• New methods of rendering
• New modes of consumption
• Complexity of assessing quality
![Page 6: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/6.jpg)
Farzad Safaei
Slide 6
Examples
• Scalable video collaboration
• Free viewpoint video
![Page 7: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/7.jpg)
Scalable Video Collaboration
![Page 8: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/8.jpg)
Farzad Safaei
Slide 8
iSee
• Combines virtual spaces with video conferencing
• Emulates a natural human gathering
• Can scale to very large meetings
![Page 9: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/9.jpg)
Farzad Safaei
Slide 9
Why augmented virtuality?
![Page 10: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/10.jpg)
Farzad Safaei
Slide 10
Technical scalability
The required capacity grows
as the square of number of
participants
![Page 11: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/11.jpg)
Farzad Safaei
Slide 11
Current solution
![Page 12: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/12.jpg)
Farzad Safaei
Slide 12
Current solution
![Page 13: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/13.jpg)
Farzad Safaei
Slide 13
Cognitive scalability
Utility will diminish if there
are too many video and audio
streams
![Page 14: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/14.jpg)
Farzad Safaei
Slide 14
iSee scalability
• By emulating a natural human gathering,
iSee has both technical and cognitive
scalability
• But also smaller meetings become more
usable
demo
![Page 15: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/15.jpg)
Farzad Safaei
Slide 15
iSee scalability
• Uses much less bandwidth than conventional
tools – 70% - 90% less bandwidth
– The bandwidth saving will be more for larger meetings
![Page 16: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/16.jpg)
Farzad Safaei
Slide 16
iSee scalability
• Allows easy interaction in large
gatherings
– Multiple simultaneous conversations
– Moving from one conversation to another
![Page 17: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/17.jpg)
Farzad Safaei
Slide 17
Key methods for technical scalability
Visibility Culling
• Only receive those video streams that are
within the visual field of view of this user
• Distance
• View frustum
• Back face
• Occlusions
P. Pourashraf, F. Safaei, and D. R. Franklin, "Distributed Area of Interest Management for Large-
Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE
International Conference on, 2012, pp. 139-144.
![Page 18: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/18.jpg)
Farzad Safaei
Slide 18
Visibility Culling
• More than 70% reduction with 25 participants – The saving is more significant as the number of users increases
![Page 19: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/19.jpg)
Farzad Safaei
Slide 19
Visibility Culling
• Dealing with movement
– Predicting future visibility based on motion vectors
![Page 20: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/20.jpg)
Farzad Safaei
Slide 20
Key methods for technical scalability
Quality Culling
• Only receive the video quality that is needed based on the
perspective of the user
• Virtual distance
• Virtual orientation
• Our technique for this is called Perceptual Pruning
F. Safaei, P. Pourashraf, and D. Franklin, "Large-scale immersive video
conferencing by altering video quality and distribution based on the virtual context,"
Communications Magazine, IEEE, vol. 52, pp. 66-72, 2014.
![Page 21: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/21.jpg)
Farzad Safaei
Slide 21
Quality Culling
• The required quality is dependent on orientation and
distance of the avatars
![Page 22: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/22.jpg)
Farzad Safaei
Slide 22
Quality Culling
Texel to pixel mapping • Video is rendered as texture on the front surface of the avatar
• Will result in texture minification and magnification
![Page 23: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/23.jpg)
Farzad Safaei
Slide 23
Quality Culling
Perceptual pruning
• DCT block down-sampling based on texture minification
![Page 24: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/24.jpg)
Farzad Safaei
Slide 24
Quality Culling
• Significant bandwidth saving in addition to visibility culling
![Page 25: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/25.jpg)
Farzad Safaei
Slide 25
Quality Culling
• 61% saving in worst-case scenario (lecture theatre)
![Page 26: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/26.jpg)
Farzad Safaei
Slide 26
Key methods for technical scalability
Saliency ranking
• During congestion, reduce quality of less salient videos
• Saliency measured by a number of attributes
• Virtual distance
• Virtual orientation
• Voice activity
![Page 27: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/27.jpg)
Farzad Safaei
Slide 27
Scalability tests
• Tests conducted during
2014
![Page 28: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/28.jpg)
Farzad Safaei
Slide 28
Problems solved
• Scalability, and more …
• Easy for the students
– Natural interaction and socialisation
– Peer-based learning
• Easy for the instructors
– No need to adapt to a new mode of delivery
![Page 29: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/29.jpg)
Farzad Safaei
Slide 29
Case studies
• DoE debating competition
• DoE Aurora College – https://drive.google.com/a/iseevc.com/file/d/0B9wPZZq-45gIM3FRclBsTWFEQkE/view?usp=sharing
• OTEN classes by invitation
• AIS art gallery – http://youtu.be/wfOfUWe58-4
• UOW Engineering Project Pitch – https://drive.google.com/file/d/0BzDp2xswG__IX0lKMTcySVctRTQ/view?usp=sharing
• UOW Alumni Fair
![Page 30: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/30.jpg)
Farzad Safaei
Slide 30
OTEN Health Class
![Page 31: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/31.jpg)
Farzad Safaei
Slide 31
UOW Engineering Pitch
![Page 32: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/32.jpg)
Farzad Safaei
Slide 32
Some of the feedback … UOW pitch
• 90% of students and 100% of industry participants indicated they wanted
to participate in a similar activity again – 80% of industry participants said they would not have been able to participate in the event had it
been held on-camps
Lee, Mark, et al., “Engaging Industry in Student Projects Through a Video-Augmented 3D Virtual Trade Fair”,
Under review, IEEE Transactions on Education – contact me for a copy of current manuscript.
![Page 33: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/33.jpg)
Farzad Safaei
Slide 33
Some of the feedback … UOW pitch
• Improvements in pitch presentation was significant
Lee, Mark, et al., “Engaging Industry in Student Projects Through a Video-Augmented 3D Virtual Trade Fair”,
Under review, IEEE Transactions on Education – contact me for a copy of current manuscript.
![Page 34: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/34.jpg)
Farzad Safaei
Slide 34
Some of the feedback … UOW pitch
Nikolic S, et al., “2D versus 3D Collaborative Online Spaces for Student Team Meetings: Comparing a Web
Conferencing Environment and a Video-Augmented Virtual World”, To appear, Proceedings of Australasian
Association for Engineering Education Conference, AAEE2015.
![Page 35: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/35.jpg)
Farzad Safaei
Slide 35
UOW Fair
![Page 36: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/36.jpg)
Farzad Safaei
Slide 36
Some of the feedback … Alumni Fair
• Student responses:
![Page 37: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/37.jpg)
Farzad Safaei
Slide 37
Some of the feedback … Alumni Fair
• Industry responses:
![Page 38: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/38.jpg)
Farzad Safaei
Slide 38
Summary
• Intersection between virtual and physical will
be of significant social and commercial
importance
• Potential for new industries (and leaders) to
emerge
![Page 39: Immersive Multimedia and Virtual Crowds - telsoc.org • Scalable video ... Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE ... Farzad Safaei](https://reader030.fdocuments.in/reader030/viewer/2022021501/5aab259c7f8b9a8f498b865c/html5/thumbnails/39.jpg)
ICT Research Institute University of Wollongong