Immersive actor feedback system

1 Research & Innovation

Immersive actor feedback system

• What sees the actor on set ?

( Answer! )



• Viewpoint dependent projection– Virtual reality: CAVE system

• Integration into Chroma-keying environment– Retro-reflective cloth (BBC patent)



• Retro-reflective cloth– “truematte” developed by BBC

Camera with LED ring

LED off

LED on



Projector

3D model of Actor

3D model of virtual scene

ViewpointRenderer

Head position

Mask generation


Immersive actor feedback system• View dependent projection

The Ambassadors (1533), by Hans Holbein the Younger. National Gallery London


Immersive actor feedback

• Head tracking

Fast template matching filter on voxel data


Production Visualisation Tools

• real-time, low-quality

Monitor

Actor Feedback

Director’s View

3D model of real scene

3D model of virtual scene

On-set

Planning,On-set


Bamzooki

• BBC Children Production


Free-viewpoint video for sports

• ‘Virtual Replay’ of football scenes– Allows viewer interactive

review

– Manually modelled offline

– Not realistic


Free-viewpoint video for Sport visualisation

• Piero

• Overcome limitations: use multi-camera system, iview project


Applying multi-camera techniques to sport scenarios

• Test shoot England-Wales in Old Trafford 9 Oct 2004


Where to put the cameras?

• Broadcast coverage cameras

• Pro:– Works without extra setup

• Cons:– Unreliable framing


Applying multi-camera techniques to sport scenarios

• Calibration with wand• Problems:

– Cameras knocked overnight– Cameras not rigidly mounted


Line-based calibration

• Colour-based edge detection

• Line-fitting• Calibration of camera

parameters (T,R,f) against dimensions of the pitch

• Runs at video frame rate


Processing - Segmentation

• Chroma-key against green


Processing – 3D Reconstruction

• 3D representations– Billboard (flat polygon, sprite)– Visual hull (Shape from silhouette)


Results

From 16 cameras, visual hull + graph-cut shape optimisation

(results from iview project, jointly with University of Surrey)


Results


Summary

• Computer vision techniques are important building blocks for the automated generation of visual effects

• For the integration of video and graphics several optical phenomena have to be harmonised. In descending order of importance:– Camera perspective, occlusions, shadows, ..

• Visual feedback systems integrated into the production pipeline are important to allow interaction with virtual objects

• Free-viewpoint video allows new insights– Goal: Stream 3D to the viewer at home


3D Modelling

• Graph-cut

• Details: A Bayesian Framework for Simultaneous Matting and 3D Reconstruction, J.-Y. Guillemaut, A. Hilton, J. Starck, J. Kilner, and O. Grau, 3DIM’07


Results

• Video from ORIGAMI demo production


Applications: Requirements

I

coarse rich detailed

3-D

II

III IV

2 1/2-D2-D

Applications:

a) virtual studiob) On-set Visualisationc) Pre-Visualisationd) Interactive e) Digital effects

Optical phenomenon:

1) camera perspective2) occlusions3) shadows4) reflections

1 2 3 4

I

II

III

IV

a

b

e

Shape classes:

d c


Introduction

• Conventional work flow in (feature) TV or film productions

PlanningPhase

On-setPhase

PostProduction

Animations /Virtual Set

Storyboard

FinalProgramme

expensive


Introduction

• ORIGAMI approach– Mixed-reality techniques for visualisation through the production

PlanningPhase

On-setPhase

PostProduction

Animations / Virtual Set

Storyboard

FinalProgramme

Immersive actor feedback system

Documents

Transcript of Immersive actor feedback system