Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK
-
Upload
augmentedworldexpo -
Category
Technology
-
view
64 -
download
1
Transcript of Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK
![Page 1: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/1.jpg)
Deep-Dive into the Wikitude SDK
![Page 2: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/2.jpg)
2
Who is talking
Philipp Nagele
● CTO of Wikitude (Product & Technology)
● MSc in Computer System Engineering from Halmstad University in Sweden
● Previous companies:
- 3united/VeriSign Inc.
- T-Mobile Austria
![Page 3: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/3.jpg)
Agenda for today
●Who is Wikitude
●Architecture of the Wikitude SDK
●A deeper look at the JavaScript API and the Plugins API
![Page 4: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/4.jpg)
v v v
Wikitude AR at a glance.
First AR app in the world in
2008
10,000 apps powered by
Wikitude SDK
3D tracking
technology
v
REFERENCES
![Page 5: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/5.jpg)
5
HQ office in Salzburg, Austria
![Page 6: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/6.jpg)
6
![Page 7: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/7.jpg)
AR creator and CMSCloud Recognition
Wikitude AR Products & Tools
Connect your own CMS
![Page 8: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/8.jpg)
Mock-ups only
… understand what you are looking at
Mock-ups only
![Page 9: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/9.jpg)
… make the World your playground
![Page 10: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/10.jpg)
….augment books
Mock-ups only
![Page 11: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/11.jpg)
… augment print, ads, catalogues, packaging
![Page 12: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/12.jpg)
… virtually mount products
![Page 13: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/13.jpg)
Maintenance
![Page 14: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/14.jpg)
![Page 15: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/15.jpg)
Platform agnostic technology
15
AR/VR wearables TabletsSmartphones
multi platform technology core based on C++
![Page 16: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/16.jpg)
Tailored SDKs for Smart Glasses
Partnership since 2014, optimized SDK for BT-200, full documentation →BT-300 in pipeline
Partnership since 2016 optimized SDK for R-7
Partnership since 2014, SDK for M100, full documentation →M300 in pipeline
![Page 17: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/17.jpg)
Native API JavaScript API
Easy access to AR mobile development
Xamarin
PhoneGapCordova
titaniumunity
Android
iOS
![Page 18: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/18.jpg)
Wikitude SDK Architecture
2D On-DeviceReco &
Tracking
2D Cloud Reco
3D Reco & Tracking
Unity3D
Unity Plugin
Plugins API
Custom CV plugin
Custom camera plugins
Your App
Your App Cordova
PluginTitanium Module
Xamarin Componen
t
Your App
Cordova
Titanium
Xamarin
Your App
Your App
JavaScript APINative API
Your App
3D Rendering
Augmen-tations LBS
Plugin Manager
OpenGL ES2 Rendering
ARMv7 NEON
optimizations
ARMv8 optimization
sCamera
GPU optimization
s
OpenCL
Metal APIIMU
Hardware Abstraction Hardware Optimizations
C++ Layer
Java/Obj C
![Page 19: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/19.jpg)
Wikitude JS SDK Java API
Wikitude JS SDK has a very slim Java API
• ArchitectView• Lifecycle events (onCreate, onPause, onResume, …
methods)• Load Architect Worlds
Everything else is optional• ArchitectUrlListener• ArchitectWorldLoadedListener• CaptureScreenCallback
![Page 20: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/20.jpg)
Architect view composition
20
OpenGL ES 2 view* Renders camera stream* Renders augmentations
Web view* Loads main .html file* Transparent background
![Page 21: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/21.jpg)
JavaScript Augmented Reality Experience
21
Architect Worlds consist of:• .html file(s)• .js file(s)• .css file(s)
• Defines your augmented reality experience• Use the Wikitude JS API provided within the Architect
view context<script src=“https://wikitude.com/libs/architect.js"></script>
• Loaded by the Wikitude SDKs Architect view
![Page 22: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/22.jpg)
Architect world
22
Javascript execution triggers actions in the underlying C++ layer
![Page 23: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/23.jpg)
Types of augmentations
23
Augmentations are represented by AR.Drawable subclasses
• Images - AR.ImageResource / AR.ImageDrawable• Videos (also with alpha channel) - AR.VideoDrawable• 3D models - AR.Model• Web views - AR.HTMLDrawable• Labels - AR.Label
![Page 24: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/24.jpg)
Wikitude JavaScript API
24
Model• AR.GeoLocation• AR.RelativeLocation• AR.Tracker• AR.ImageResource
View• AR.ImageDrawable• AR.VideoDrawable• AR.Model• AR.Label
Controller• AR.GeoObject• AR.Trackable2D• AR.PropertyAnimation• AR.AnimationGroup
![Page 25: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/25.jpg)
Geo AR
25
How to define e.g. a point of interest (POI)
• Define where the augmentation is placed by creating AR.GeoLocations (latitude/longitude coordinates)
• Define the augmentation by creating AR.Drawables
• Combine location and drawables using AR.GeoObjects
= The same geo object is displayed at different locations
POI CPOI A
POI A
Poi B
![Page 26: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/26.jpg)
Pokémon Go like Geo AR
26
How to build a Pokémon Go like AR experience:
![Page 27: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/27.jpg)
Pokémon Go like Geo AR
27
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon• Relative to the current user position using AR.RelativeLocation
//a relative location being 12 meters south, 20 meters west and 13 meters lower than current position of the uservar relativeLocation = new AR.RelativeLocation(null, -12, -20, -13);
![Page 28: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/28.jpg)
Pokémon Go like Geo AR
28
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon• Relative to the current user position using AR.RelativeLocation• Absolute to the user using AR.GeoLocation
//latitude, longitude, altitudevar location = new AR.GeoLocation(47.77317, 13.069929, 320.);
![Page 29: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/29.jpg)
Pokémon Go like Geo AR
29
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon• Relative to the current user position using AR.RelativeLocation• Absolute to the user using AR.GeoLocation
• AR.Model object to render the 3D model of a Pokémon
//create a new Modelvar model = new AR.Model(“models/pokemon.wt3”);
![Page 30: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/30.jpg)
Pokémon Go like Geo AR
30
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon• Relative to the current user position using AR.RelativeLocation• Absolute to the user using AR.GeoLocation
• AR.Model object to render the 3D model of a Pokémon• Can be animated through .fbx defined animations
//We want to start an animation inside a modelvar model = new AR.Model(“models/pokemon.wt3");var modelAnim = new AR.ModelAnimation(model, "hit"); //start hit-animation defined inside model
![Page 31: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/31.jpg)
Pokémon Go like Geo AR
31
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon• …
• AR.Model object to render the 3D model of a Pokémon• …
• AR.GeoObject to actually bring the model onto the screen
// a GeoObject with drawables set on creation timevar geoObject = new AR.GeoObject(location, { drawables : { cam: [model] //the drawable representing the GeoObject in the camera view }});
![Page 32: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/32.jpg)
Pokémon Go like Geo AR
32
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger• trigger custom events when the Pokémon becomes visible in the
camera
geoObject.onEnterFieldOfVision = function() { /* your custom behaviour e.g. start a certain model animation */};geoObject.onExitFieldOfVision = function() { /* your custom behaviour */};
![Page 33: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/33.jpg)
Pokémon Go like Geo AR
33
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger• trigger custom events when the Pokémon becomes visible in the
camera• Define a direction indicator drawable for your Pokémon
• Helps users to find the Pokémon around them
var imageResource = new AR.ImageResource(“assets/pokemonIndicator.png”);var pokemonIndicator = new AR.ImageDrawable(imageResource, 0.2);
// a GeoObject with drawables set on creation timevar geoObject = new AR.GeoObject(location, { drawables : { cam: [model], //the drawable representing the GeoObject in the camera view indicator: pokemonIndicator }});
![Page 34: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/34.jpg)
Pokémon Go like Geo AR
34
![Page 35: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/35.jpg)
GeoObjects & HTML content
Menu barjQuery support
Geo-located POIs
Multiple Drawables- 3 images- 3 text labels
Web View
Augmented Reality View
![Page 36: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/36.jpg)
Pokémon Go like Geo AR
36
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger• trigger custom events when the Pokémon becomes visible in the
camera• Define a direction indicator drawable for your Pokémon
• Helps users to find the Pokémon around them• Use the AR.Radar component to permanently show the position of
a Pokémon in the AR scene
![Page 37: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/37.jpg)
Pokémon Go like Geo AR
37
![Page 38: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/38.jpg)
Pokémon Go like Geo AR
38
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger• trigger custom events when the Pokémon becomes visible in the
camera• Define a direction indicator drawable for your Pokémon
• Helps users to find the Pokémon around them• Use the AR.Radar to permanently show the position of a Pokémon in
the AR scene• Capture a screenshot after you caught a Pokémon using the
corresponding ArchitectView Java API.
![Page 39: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/39.jpg)
Image Recognition
Augment target images using JavaScript API capabilities (Drawables, Sounds, Animations,Videos, 3D Models)
var kitchenTracker = new AR.Tracker("Kitchen.wtc"); var overlay = new AR.VideoDrawable("howdens.mp4", 0.65); var trackable2DObject = new AR.Trackable2DObject(
kitchenTracker: "Kitchen", { drawables:
{ cam: overlay }});
![Page 40: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/40.jpg)
Wikitude SDK Architecture
2D On-DeviceReco &
Tracking
2D Cloud Reco
3D Reco & Tracking
Unity3D
Unity Plugin
Plugins API
Custom CV plugin
Custom camera plugins
Your App
Your App Cordova
PluginTitanium Module
Xamarin Componen
t
Your App
Cordova
Titanium
Xamarin
Your App
Your App
JavaScript APINative API
Your App
3D Rendering
Augmen-tations LBS
Plugin Manager
OpenGL ES2 Rendering
ARMv7 NEON
optimizations
ARMv8 optimization
sCamera
GPU optimization
s
OpenCL
Metal APIIMU
Hardware Abstraction Hardware Optimizations
C++ Layer
Java/Obj C
![Page 41: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/41.jpg)
41
Plugins API in detail
Custom Frame
Provider
Image Frame
Standard Frame
Provider
Wikitude CV EngineRendering
Custom CV Plugin
Plugins API
class Plugin { public: … void cameraFrameAvailable(const Frame&; cameraFrame_); void update(const vector<RecognizedTarget> recognizedTargets_); void startRender(); void endRender …};
1234567
class RecognizedTarget { public: const string& getIdentifier() const; const Mat4& getModelViewMatrix() const; const Mat4& getProjectionMatrix() const; const float getDistanceToCamera() const;};
![Page 42: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/42.jpg)
Use-Cases for Plugins API
● Run own computer-vision related code in parallel to Wikitude SDK-OCR Engine-Barcode Reader-Face Recognition● Own the camera stream and the camera rendering-Custom effects and shaders-Remote camera stream with local processing
![Page 43: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/43.jpg)
….architecture, real estate….. ….gaming, filming industry….. ….indoor navigation, retail…..
New algorithms and concepts
3D Tracking / SLAMObject Reco & Tracking
![Page 44: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK](https://reader035.fdocuments.in/reader035/viewer/2022062822/587fc4ff1a28ab3b158b5d25/html5/thumbnails/44.jpg)
See more.At Booth B6