Creativity SupportIn Projection-based Augmented EnvironmentsBruno Simões, Federico Prandi, and Raffaele de Amicis (et.al.)F. Graphitech

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

Framework

Problem Statement

Introduction &Context

Conclusions12

3

4

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

AgendaPath of Enlightenment

LowCost

CreativityCommunication

Culture

Content CreationCollaboration

Crowdsourcing

to develop new creative tools (based on computer vision techniques and pico-projection) that can transform the surrounding environment into an user interface

Introductionc-Space Objective

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

LowCost

CreativityCommunication

Culture

Content CreationCollaboration

Crowdsourcing

to develop new creative tools (based on computer vision techniques and pico-projection) that can transform the surrounding environment into an user interface

Introductionc-Space Objective

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

What is Creativity?A. Einstein

“seeing what everyone else has seen, andthinking what no one else has thought”

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

IntroductionWhat is Creativity?

What is Creativity?Betty Edwards

“creativity is the ability to find new solutionsto a problem or new modes of expression;thus it brings into existence something newto the individual and to the culture.”

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

IntroductionWhat is creativity?

Presentation and InteractionCrowdsourcing and gamified data collection 4D Reconstruction

Introductionc-Space State-of-Art

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

Introductionc-Space Objective

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

What About Creativity

Introductionc-Space Objective

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

What About Creativity• Knowledge transfer: How mechanisms implemented promote interaction between users• Germinality: how well does it open up to new perspectives• Does it really new ways of expression

Introductionc-Space State-of-Art Limitations

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

Limitations:- Simple user experience (AR)- 2DoF (storylines) - Ergonomics- Embarrassment factor- …

to develop new creative tools (based on computer vision techniques and pico-projection) that can transform the surrounding environment into an user interface

Culture LowCost

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

Crowdsourcing

Creativity

Content CreationCollaboration

Communication

Introductionc-Space Objective

to develop new creative tools (based on computer vision techniques and pico-projection) that can transform the surrounding environment into an user interface

Display limitation ?

Projection-Based Augmented RealityUniting physical and virtual objects

“combination of physical reality with computergenerated data projected directly within or on theuser's physical space”

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

IntroductionProjection-based AR

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

IntroductionProjection-based AR

Projection Mapping vs Traditional ProjectionUses the projection surface as a larger screenCustomizes the appearanceof an object/environment

Spatial AR

uses real physical objects to both represent and interact with computer-generated

information

Tangible User Interfaces (TUI)+

IntroductionPossible Advantages

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

• Real objects can be physically handled • Does not require see-through displays or to wear a head-mounted display• Multi-user experience • Facilitates co-located collaboration• Ergonomics, e.g. might reduce physical stress

1. Visual creation of augmented environments and automatic recomputationof projection maps on camera movementsProjective augmented reality using mobile devices raises calibration issues due to mobility that must be addressed.

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

Research ProblemProjection-based AR

2. Automatic Background Color CompensationUnfortunately, without accommodating for the different background colors, the user may be subject to viewing inaccurate colors and reduced contrast

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

Research ProblemProjection-based AR

3. Visual Pollution, Privacy & Human SafetyThe invasiveness of projected content can lead to "visual pollution" or bring annoyance to other people in the vicinity. Additionally projected content can blind people.Projective interfaces exposes the end-user to the risk of projecting sensitive information, e.g. PM popups.

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

Research ProblemProjection-based AR

4. Find effective new forms of user interactionWe want the surrounding environment itself to be transformed into an interface for the users to interact through gestures, wearables or simply by moving around.

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

Research ProblemProjection-based AR

Collaboration and Information Exchange!

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

FrameworkResearch Problem 1

Solution to visually create new augmented environments and to automatically re-compute projection maps on camera movements

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

FrameworkPPP Setup

Marker-less tracking, perspective correction, and safety monitoringcomputer generated content real worldPico-projector

processing unit + WIFICamera

projection

Setup

(Client

)

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

FrameworkPPP Setup

Server

Client-S

erver

real world

Normative policies, collaboration features

descriptors, video samples, 3D mapping setups, visible objects

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

FrameworkMobile Editor

Setup

project

ion ma

pping

Mapping handlersVisual Object

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

FrameworkPPP Procedure

Client-S

erver

Device

AROutput

QueryState

Object

UpdateApplication

Logic

RenderGraphics

Cloud/DeviceTarget DatabaseCamera CameraFrame

Pixel FormatConversion

ConvertedFrame

User-DefinedTargets

ImageTargets

FrameMarkers

Multi-ImageTargets

CylinderTargets

WordTargets

VirtualButtons

Detect New ObjectsTrack Detected ObjectsEvaluate Virtual Buttons

RenderCameraPreview

ServerServer

Projector

calibration& perspective correction

collaboration

BCC Descriptors Stack

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

FrameworkBackground Color Compensation

Render

ing

Lights Data

ColorCorrectionSystem

CameraFrameBuffer

ColorCorrectionFrameBuffer

LightDetectionSystem

Gamma function Estimation, Homer H. Chen

How can we create an easy-to-use normative policy to automatically regulate the use and the power of PPP technology?e.g. streets, where drivers or passengers could be temporarily blinded by the projection.

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

FrameworkResearch Problem 3

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

FrameworkResearch Problem 3

imagga

Automated image tagging and keywording

During the initial user trials, a small group of participants reported:1. No difficulty in learning the techniques that were proposed to setup a 3D projection mapping2. They found appealing the ability to combine information in a shared workspace.3. Found useful the feature to download content to their smartphones from projected animated QR codes4. Picking sometimes was difficult to archive

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

ConclusionsTest Case Results

1. Current pico-projectors have a luminance between 5 and 100 lumens. Hence, these projectors can only be used indoors or outdoors at night.2. The use markerless descriptors under certain conditions can be an issue:1. Poor illumination of the environment2. Surfaces with few features, e.g. white halls, etc.3. Projection without auto-focus can be disruptive

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

ConclusionsBiggest Limitations

1. Blending multiple projections into a seamless projection2. Restrict the projection in such way that it will never hit people in the face, but while keeping the projecting around the person.3. Better way to impose normative policies: executed at a real time frame rate.4. Explore interaction concept with drones, e.g. Wristwatch Drone.5. Drop the back-server requirement when the use of QR codes is possible

c-SpaceGrant agreement no: 611040

fostering new creative paradigms based on recording and sharing “casual” videos through the internet.The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

ConclusionsFuture Work

For your attentionbruno.simoes@graphitech.it

@brunogsimoes

linkedin.com/in/brunogsimoes

Grant agreement no: 611040