Lost In The Clouds
-
Upload
georgejames -
Category
Technology
-
view
984 -
download
4
description
Transcript of Lost In The Clouds
Lost in the Clouds
An example of a schemaless database application in the cloud
SlipstreamUSA :: March 25, 2009
The Project
OpenStreetMap A free map of the planet
Free geospatial data Licensed under CC-BY-SA
Created from Crowd-sourced contributions 90,000 registered users 1,000+ contributors per day
Used for: Wikipedia Flikr OpenCycleMap.org OpenPisteMap.org YourNavigation.org Mobile devices
© Constantin Litvak
Coverage
CC-BY-SA 2.0 © OpenStreetMap
The OpenStreetMap Database Server
In the cloud MySQL Ways: 24,956,171
Nodes: 305,956,583
Tags: 3.6 billion highway=motorway name=Broadway foo=bar
CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
Interfaces
API For updates
(1,000 users per day) Queries of up to
0.25² degrees Weekly Planet Dump
100Gbyte XML file Changes
Feed of changes Daily / Hourly / Minutely One file per minute
(~30Kb compressed)CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
Problem Statement
ProblemWant to be able to query the data in bigger chunks and to do it more selectively
SolutionCloud based schemaless database with simple to use query interface and lots of indexes
CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
XAPI Service
GT.M or Cache 250Gbyte database
Nodes: 30Gb Node tags: 115Gb Node indexes: 70Gb Ways: 21Gb Way tags: 10Gb Way indexes: 18Gb
Apache + serverLink Licensed as AGPL
CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
XAPI Service
In the cloud Queries of up to 100² degrees Query by tags and tag values Scaleable REST style interface
http://xapi.openstreetmap.org/api/0.6/*[tourism=hotel][bbox=-75.7,40.0,-75.4,40.2]
Import planet dump Import minute diff
CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
XAPI Service
adaabcdcabaadab
Geospatial Index – quadstrings
XAPI Service
^way(27016525)="adaabcdcabaadab"^way(27016525,1)=296138118^way(27016525,2)=296138119^way(27016525,3)=296138120^way(27016525,4)=296138121^way(27016525,5)=296138118
^waytag(27016525,"addr:housenumber")=2^waytag(27016525,"building")="yes"
^wayx("building","*","adaabcdcabaadab",27016525)=""^wayx("building","*","adaabcdcabaadab",27028298)=""^wayx("building","*","adaabcdcabaadab",27028299)=""^wayx("building","*","adaabcdcabaadab",27028326)=""^wayx("building","*","adaabcdcabaadab",27028327)=""^wayx("building","*","adaabcdcabaadab",27035972)=""^wayx("building","*","adaabcdcabaadab",27035973)=""^wayx("building","*","adaabcdcabaadab",27035974)=""^wayx("building","*","adaabcdcabaadab",27035975)=""^wayx("building","*","adaabcdcabaadab",27035984)="“
<way id='27016525'><way id='27016525'> <nd ref='296138118'/><nd ref='296138118'/> <nd ref='296138119'/><nd ref='296138119'/> <nd ref='296138120'/><nd ref='296138120'/> <nd ref='296138121'/><nd ref='296138121'/> <nd ref='296138118'/><nd ref='296138118'/> <tag k='addr:housenumber' v='2'/><tag k='addr:housenumber' v='2'/> <tag k='building' v='yes'/><tag k='building' v='yes'/> </way></way>
Deployment
Currently 5 instances: London Paris San Diego Zurich Amsterdam
(Wikimedia-de) More servers needed:
Internet connectivity 300Gb disk storage
CC-BY-SA 2.0 Image © Nikolaj Data © OpenStreetMap
Wikipedia: Query to Map
{{osm-query |way=name=Arizona Canal}}
CC-BY-SA 2.0 © OpenStreetMap
Internals
Debian Apache web-server REST
URI represents a resource
Xpath flavoring serverLink
WebLink emulator GT.M or Cache
POM AGPL
XAPI Future
Load Sharing / balancing More Xpath-like queries JSON output Generic indexing service More servers
CC-BY-SA 2.0 Image © ITO! Data © OpenStreetMap
Thank you
Thank youQuestions?