Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

123
Social Street View: Blending Immersive Street Views with Geo-tagged Social Media Ruofei Du and Amitabh Varshney The Augmentarium| UMIACS | University of Maryland, College Park {ruofei, varshney} @ cs.umd.edu www.SocialStreetView.com

Transcript of Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Page 1: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Ruofei Du and Amitabh VarshneyThe Augmentarium| UMIACS | University of Maryland, College Park

{ruofei, varshney} @ cs.umd.eduwww.SocialStreetView.com

Page 2: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

image courtesy: plannedparenthood.org

IntroductionSocial Media

Page 3: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

image courtesy: adweek.com

IntroductionSocial Media - Interactivity

Page 4: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

image courtesy: forbes.com

IntroductionSocial Media - Versatility

Page 5: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

image courtesy: huffingtonpost.com

IntroductionSocial Media - Social Relevance

Page 6: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
Page 7: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

EXIF

image courtesy: B&T Magazine

Page 8: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

IntroductionMeta data - Time of Creation

image courtesy: Brian Clegg

Page 9: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

IntroductionMeta data - Location of Creation

image courtesy: squarespace.com

Page 10: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

IntroductionMeta data - Camera Parameters

image courtesy: naldzgraphics.net

Page 11: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

image courtesy:

instagram.com,

facebook.com,

twitter.com

Related WorkLinear narrative visualization

Page 12: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

image courtesy:

pinterest.com

Related Work2D layout

Page 13: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Related WorkNatural Immersive Virtual Reality?

image courtesy: drivenhealthy.com

Page 14: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Related WorkKarnath et al. and Loomis et al.

Page 15: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Related WorkKarnath et al. and Loomis et al.

Page 16: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Related WorkKarnath et al. and Loomis et al.

Page 17: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Related WorkVisualization of Geo-tagged Information

Page 18: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Related WorkVisualization of Geo-tagged Information

Page 19: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Related WorkVisualization of Geo-tagged Information

Page 20: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Related WorkVisualization of Geo-tagged Information

Page 21: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Related WorkVisualization of Geo-tagged Information

Page 22: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

image courtesy: panoramio.com

Related WorkPanoramio - Photos of the WorldCuenca and Manchón, 2005 - Now

Page 23: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

image courtesy: twitterstand.umiacs.umd.edu

Related WorkTwitterStand: News in TweetsSankaranarayanan et al. SIGGIS 2009

Page 24: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

image courtesy: flickr.com

Related WorkFlickr - World MapSerdyukov et al. SIGIR 2009

Page 25: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

image courtesy: photostand.umiacs.umd.edu

Related WorkPhotoStand for News PhotosSamet et al. VLDB 2013

Page 26: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Related WorkVisualization of Geo-tagged Information (cont.)

Page 27: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Related WorkVisualization of Geo-tagged Information (cont.)

Page 28: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Related WorkVisualization of Geo-tagged Information (cont.)

Page 29: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Related WorkVisualization of Geo-tagged Information (cont.)

Page 30: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Related WorkVisualization of Geo-tagged Information (cont.)

Page 31: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Related WorkVisualization of Geo-tagged Information (cont.)

Page 32: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Our Approach?

Page 33: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Social Street View

Page 34: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Social Street View

Page 35: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

DemonstrationThe Augmentarium, UMIACS6000 x 3000 pixels

Page 36: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
Page 37: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Conception, architecting & implementation

Social Street View

A mixed reality system that can depict geo-tagged social media in immersive 3D web environments

Page 38: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Blending multiple modalities of

Street View + Social Media

Depth maps, normal maps, and road orientationGPS coordinates and time creation

Page 39: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Enhancing visual augmentation

Maximal Poisson-disk sampling

Evaluated by image saliency metrics

Page 40: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Achieving cross-platform compatibility by

WebGL + Three.js

smartphones, tablets, desktop, high-resolution large-area wide field of view tiled display walls, as well as head-mounted displays.

Page 41: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Technical Challenges?

Page 42: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

System Overview

Page 43: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

ArchitectureSocial Street View System Flowchart

Page 44: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

image courtesy: Google

Page 45: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Street View Cars - Cameras, Lasers and GPSImage courtesy from Google Street View

image courtesy: Google

Page 46: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

image courtesy: Google

Page 47: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Tiles of Panoramic ImagesImage courtesy from Google Street View

image courtesy: Google

Page 48: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
Page 49: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Panorama

Page 50: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
Page 51: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Depth MapDecoded from Google Maps API v3 and GSVPanoDepth.js

Page 52: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
Page 53: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Normal MapDecoded from Google Maps API v3

Page 54: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
Page 55: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Road OrientationsDecoded from Google Maps API v3

Page 56: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

image courtesy: Instagram

Page 57: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Multi-level Strategy for Streaming Social MediaImage courtesy from Instagram users in public domains

image courtesy: Instagram

Page 58: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

SQL DatabaseIndexed by B+ Treelogarithmic insertion & query

Page 59: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Bipartite GraphG = V(V, S, E)

Page 60: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Haversine FormulaAndrew, 1805

Page 61: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Interface

Page 62: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
Page 63: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

BaselineRandom Uniform Sampling

Page 64: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Without uniform samplingAccumulation

Page 65: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

BaselineRandom Uniform Sampling

Page 66: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Not preferredOverlays high saliency regions

Page 67: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Add depth mapRemove sky and ground (most)

Page 68: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Add depth mapRemove sky and ground (most)

Page 69: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Can we ensure each image

aligns with the building geometries?

Page 70: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Add normal mapRemove all ground, align images

Page 71: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
Page 72: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Can we reduce visual clutter and occlusion?

Page 73: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Maximal Poisson-disk Sampling Gamito et al. Remove visual clutter and occlusion

Page 74: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Dart-throwing Algorithm PixelPie by Ip et al. using vertex and fragment shaders

image courtesy:

PeterPan23@Wikipedia

Page 75: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Pixel-Pie Algorithm Remove when occlusion occurs

Page 76: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Project Social Media PicturesBy Maximal Poisson-disc Sampling

Page 77: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Sampling ComparisonRemove visual clutter and occlusion

Page 78: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Scenic LandscapesUsing orientation of the road

Page 79: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Scenic LandscapesUsing orientation of the road

Page 80: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Scenic LandscapesUsing orientation of the road

Page 81: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
Page 82: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Experiment100 panoramas, 80,000+ social media

Page 83: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

ExperimentNvidia Quadro K6000

Page 84: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

ExperimentConditions of panoramic images

Immersive high-resolution screens

Common Consumer-level Displays

Page 85: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Initialization TimePanorama takes a while to load

Page 86: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

After Prefetching¾ - ⅘ time reduced

Page 87: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Rendering TimeAlmost 58~60 FPS

Page 88: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Saliency MapHou et al. TPAMI 2011

Page 89: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Social Media Coverage100 panoramas for each algorithm

Page 90: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Social Media Coverage100 panoramas for each algorithm

Page 91: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Social Media Coverage100 panoramas for each algorithm

Page 92: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Potential Applications?

Page 93: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
Page 94: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Stuck in traffic on our way to Cabo with this awesome view#roadtrip #cabo #view #mexico

Daniela on InstagramJuly 12, 2014

Page 95: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
Page 96: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
Page 97: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Business AdvertisingMuseum, restaurant, real-estate ...

Page 98: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

... dinner started off with amazing oysters paired with my favorite Ruinart blanc de blancs champagne

By frankiextah on Instagram

Page 99: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Business AdvertisingMuseum, restaurant ...

Page 100: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Learning CultureTaierzhuang, Chinese Spring Festivals

Page 101: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Crowd-sourced Tourism

Page 102: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Crowd-sourced Tourism

Page 103: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

What if …Temporal information is used for filtering and rendering?

image courtesy: mindtheproduct.com

Page 104: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Visual EnhancementSpatio-Temporal Augmented Reality

image courtesy: webneel.com

Page 105: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
Page 106: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Seasonal EffectWinter

Page 107: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Seasonal EffectWinter

Page 108: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Seasonal EffectSpring

Page 109: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Seasonal EffectAutumn in Paris

Page 110: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Failure CasesA Square in London

Page 111: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Future WorkDigital City by 3D Reconstruction, Depth Fusion and Augmented Reality

image courtesy: wallpapervortex.com

Page 112: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Future WorkIncorperate with Open3D in Web3D 2016?

image courtesy: Lu et al.

Page 113: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

AcknowledgementGeollery | Shangfu Peng and Ruofei Duwww.geollery.com

Page 114: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

AcknowledgementAugmentarium Lab | GVIL | UMIACS

Page 115: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

AcknowledgementNSF | Nvidia | MPower | UMIACS

Page 116: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Video FieldsWeb3D Sunday 9:20 - 9:40 AM

Page 117: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Social Street Viewwww.SocialStreetView.com

Thank you! Any questions or comments are welcome!

Ruofei Du and Amitabh VarshneyAugmentarium Lab | GVIL | UMIACS

Web3D 2016

Page 118: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Social Street View

Backup Slides

Page 119: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Future WorkSaliency

- Low-level image saliency for layout- Real-time or post-processed saliency

maps

Page 120: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Future WorkUser Study

- High-resolution large-area screens - Head-mounted displays

Page 121: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Future Work3D Reconstruction and AR

- Real-time geometric fusion- HoloLens

Page 122: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

Future Work3D Reconstruction and AR

- What is the (experimentally-justifiable) motivation for putting generic images into scenes?

- What is the relevance of which photos are displayed? And how this affects the worth?

- Whether it is possible to try and improve this relevance using vision techniques?

Page 123: Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

What is ignored in current social media sites?