JonathanReeves

October9,2017

SeeWhatWeSee- SharingMixedRealityExperienceswithWebRTC

1©2017Arvizio,Inc.

Introduction

MixedRealityisagamechangingtechnologyforenterprisesincludingAEC,education,energy,engineering&healthcare

Keyrequirementsforenterpriseadoption:§ Mixedrealityplatformsmustscaletohandlecomplex3Dmodels§ Experiencesshouldbesharedbothlocallyandremotely§ IntegrationwithexistingITsystemsessentialforseamlessworkflow

Arvizio’s MRStudio platformgoal:§ Dynamicservingoflarge3DmodelswithrichLOD(levelofdetail)§ Sharingofexperiencesandcollaborationacrosslocations§ FlexibledataintegrationwithexistingITsystemsandIoTdata

2©2017Arvizio,Inc.

TheMRStudioPlatform

MRStudioImmerse

MRStudioRemoteASPENscalable3DmodelLODprocessingReal-timesession&sharingservicesOnPrem,edgeorclouddeployment

Large3DmodelvisualizationwithLODSharing&collaborationacrosslocationsSupportformultipleAR/MRdevicetypes

Viewmixedrealityscenesremotely2Wayvideo,audio&annotation

MRStudioDirector

IoTData&ITSystems

4G/5GorWiFi

3©2017Arvizio,Inc

IPNetwork

WebRTC

MRStudioDirector

Advanced3DVisualization§ Optimizeslarge3DmodelswithLOD(LevelofDetail)§ Mesh,CAD,Pointcloud,BIM,GIS&DICOM§ Workflowwizardsformanyspatialdataformats

Real-timeSharingServices§ SharedviewingformultipleHoloLensusers§ HoloLensusersorMRSDirectormayleadsessions§ WebRTCgatewayforsecureremoteparticipation

DataIntegration§ ConnectsMRStudiowithexternalITsystems§ ModularApacheNiFi dataflowbasedarchitecture§ ModulestoconnectwithIoTsensors,bigdataetc.

4©2017Arvizio,Inc.

LODDataPyramid

5©2017Arvizio,Inc..

Full 3D model

2nd Level of Detail

3rd Level of Detail

4th Level of Detail

ImmerseAppFramework

3DModelViewingwithLODLifeSizeModelWalkThroughSharingforMultipleHoloLens

RemoteVideoChatMixedRealityRTCFeed

DataintegrationwithITsystemsBigDataSystems,IoTStreams

MRStudioDirector

Annotation

MRStudioImmerse

6©2017Arvizio,Inc..

Local&RemoteCollaboration

7©2017Arvizio,Inc.

HoloSync allowsMRSDirectortointeractivelypublish3Dscenesandmodels

MRStudioRemoteActiveviewingandannotationoftheHoloLensfeedCreatetextualorgraphicalhologramsinthescene

WebRTC

LODDataServedonDemand

WebRTCAudio,Video&SpatialDataExchange

WebRTCRecap

VP8/9&H.264VideoCompression- AppleoffersonlyH.264inIOS11

STUN&TURNAllowNATRouterTraversal

ICE– InteractiveConnectivityEstablishmentmanagestheconnectivityprocess

8©2017Arvizio,Inc.

Remoteviewing&annotationoverWebRTC

9©2017Arvizio,Inc.

§ Remoteusersmayplaceholographicannotationsintothefieldofview§ Supportedonmanybrowsersacrossmultipleplatforms§ Bandwidthefficient- operatesoverbandwidthconstrainedwideareanetworks§ HoloLensfeedislimitedto720p,alsoheadmovementcanbeachallenge

MRDeviceView

RemoteAnnotationToolbar

CapturingtheSharedExperienceHoloLensSpectatorView

§ LocalviewingofmixedrealityscenesinHDbutnoremotestreaming§ Calibrationcheckerboardrequired,complexsoftwareinstall§ RequiresdedicatedHoloLens&digitalSLR,milledaluminumbracket§ Powerfulconcept,couldthisbeimprovedupon?

10©2017Arvizio,Inc.

MRStudioViewPointStreamingofMixedRealityinHD

§ Localviewing/recording&remoteWebRTCstreamingofmixedrealityscenes§ Headsetrequiredduringinitialsetuponly§ Usedepthcameraorwebcams(staticplacement)inplaceofDSLR§ Potentialtoworkacrossmultipletypesofdevice

11©2017Arvizio,Inc.

CameraCaptureUnity App

ComposerWebRTCGateway

Director Sharing Server

Depth SensingStereo Camera

MR Studio Director

WebRTCClient

LocalHD View(s)

RemoteView

PC

Audio

MRStudioViewPoint Architecture

12©2017Arvizio,Inc.

Shared Viewing

MRStudioUseCasesAEC(Architecture,Engineering&Construction)

§ Visualizationduringthedesign,engineeringandconstructionprocess§ Bringingprojectstolifeduringthesalesprocess§ Accelerateprojectapprovalprocess

IndustrialDesign&Engineering§ 3Dvisualizationofcomplexmachinery§ Assistedmaintenancetaskswithremotevideoexpertassistance

Training&Education§ Training&educationacrossindustries§ Classroom,lectures,onsitetrainingandremoteparticipants

Healthcare§ MRhospitalroomconcept,MRassistancefortelemedicine§ Presurgeryvisualization,prepareforcomplexsurgeries

13©2017Arvizio,Inc.

Demonstration