Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 •...
Transcript of Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 •...
![Page 1: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/1.jpg)
Enrich your data with geocoordinates from
OpenStreetMap or ArcGIS
Andreas ‚ads‘ Scherbaum
![Page 2: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/2.jpg)
Andreas Scherbaum • Works with databases since 1997, with
PostgreSQL since 1998
• Founding member of PGEU
• Board of Directors: PGEU
• PostgreSQL Regional Contact for Germany
• Ran my own company around PostgreSQL
for 7+ years
• Joined EMC in 2011, working on PostgreSQL and Greenplum projects
![Page 3: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/3.jpg)
Get Social • Google+: http://goo.gl/e3EtB1
• Twitter: @ascherbaum
![Page 4: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/4.jpg)
ToC
![Page 5: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/5.jpg)
Table of Content • Why (enrich your data)
• How (to enrich your data)
• Measure (the results)
• What‘s next
![Page 6: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/6.jpg)
Why (enrich your data)
![Page 7: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/7.jpg)
Address Format • If you are lucky:
Street HouseNr City ZIP Country
… … … … …
… … … … …
… … … … …
![Page 8: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/8.jpg)
Address Format • If not so lucky:
Address
1 Eagle Dr, Missouri, United States
1198 Toorak Rd, Camberwell VIC 3124, Australia
Widdersdorfer Straße 427, 50933 Köln
Need to parse and resolve that …
![Page 9: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/9.jpg)
Geolocation Services • Verify and normalize addresses
• Identify customer or address location o Country, Region, City, Street
• Fraud Prevention
• Geo Marketing
• Geo Tagging
• Geo Targeting
• Value Added Tax in the European Union o Tax is paid based on customer location
• Weather Forecast
• Concurrency Calculation
• …
![Page 10: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/10.jpg)
Car Insurance Fraud • Insurance companies offer savings based on
information like state/parking/living address o Good to know how far away from the registered address (usual overnight
parking place) an accident happened or a car was stolen
o Address manipulation is a serious issue and a growing area for insurance companies: compare all known addresses for a customer
o Find „hot spots“
![Page 11: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/11.jpg)
Insurance Fraud • „Hot Spots“
Is there a car shop nearby, involved in the cases?
![Page 12: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/12.jpg)
Health Care Fraud • Discover if policy holders claim money for other
people o Look at billing addresses, doctors and hospitals addresses ect.
![Page 13: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/13.jpg)
Health Care Fraud • Germany has a good Health Care system
o Ever since 2006 Germany is rolling out a „health card“ (Euphemism)
o Everyone can visit a doctor if he possesses a health card
o A photographic image is required on the „health card“
• The photo cannot be verified
![Page 14: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/14.jpg)
Nearest shop/office • Retailers, banks and many others can figure out the
closest shop or branch office for every customer o Send personalized (news)letters
o Include nearest address and distance (even time, with routing)
• Small shops can figure out if it‘s worth sending
advertisement to a customer
![Page 15: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/15.jpg)
Verify addresses • Sometimes it‘s useful to just verify addresses
• Ever been in a shop where they ask for your zip
code? o „We want to improve our service“
o I usually answer: 78266
• Ever tried to match addresses? o „Müller Straße, 51149 Köln“
o „Mueller Straße, 51149 Koeln“
o „Müller Strasse, Köln, 51149“
o „Muller Str, Cologne, 51149“
Remember this one
![Page 16: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/16.jpg)
In my area of work … • I work for a Data Warehouse company …
o Many of our customers have millions of addresses in the EDW
![Page 17: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/17.jpg)
Let‘s Talk Business • Let‘s imagine, you run a small business
• A pizza shop
• State-of-the-art website
• Online orders
• Your claim: • Always hot pizza, or money back
![Page 18: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/18.jpg)
Berlin
© Wikipedia
![Page 19: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/19.jpg)
Germany!
© Wikipedia
![Page 20: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/20.jpg)
Do you know your customers?
• Where is: • Uthlande Street 1, 25849 Pellworm, Germany
• (address of the local government)
![Page 21: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/21.jpg)
Do you know your customers?
• Where is: • Uthlande Street 1, 25849 Pellworm, Germany
• (address of the local government)
• It‘s in the „Northern Friesland“ district
© Wikipedia
![Page 22: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/22.jpg)
Do you know your customers?
• Where is: • Uthlande Street 1, 25849 Pellworm, Germany
• (address of the local government)
• Only reachable via ferry • Good luck with your hot pizza in the evening
© Wikipedia
![Page 23: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/23.jpg)
Do you know your customers?
• Where is: • Junkerstraße 86, 78266 Büsingen am Hochrhein, Germany
• (address of the local government)
![Page 24: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/24.jpg)
Do you know your customers?
• Where is: • Junkerstraße 86, 78266 Büsingen am Hochrhein, Germany
• (address of the local government)
• It‘s entirely surrounded by the Swiss canton of
Schaffhausen (enclave) • Food delivery from Germany to Switzerland has to pass customs
© Wikipedia
Remember this one?
![Page 25: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/25.jpg)
Do you know your customers?
• Where is: • Ullasvej 23, 3700 Rønne, Denmark
• (address of the local government)
![Page 26: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/26.jpg)
Do you know your customers?
• Where is: • Ullasvej 23, 3700 Rønne, Denmark
• (address of the local government)
• It‘s on Bornholm o Danish island in the Baltic Sea
o Google says: need a ferry
o and 4.5 hours
© Wikipedia
![Page 27: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/27.jpg)
Who cares? • No one cares about corner cases, right?
• Let‘s deal with that later … • Yeah, always a good idea, especially in IT
![Page 28: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/28.jpg)
Let‘s start easy • Zip codes, that works everywhere, every street has
one
• Germany uses 5 digits for zip codes • Other countries use 4, or 6, or characters
• But note: close number‘s don‘t imply close
distance
![Page 29: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/29.jpg)
Geography of Germany • Wikipedia: „The territory of Germany covers
357,021 km2 …”
𝑥
1 𝑧𝑖𝑝 𝑐𝑜𝑑𝑒=
357021 𝑘𝑚²
99999 𝑧𝑖𝑝 𝑐𝑜𝑑𝑒 𝑥 =
357021 𝑘𝑚²
99999 𝑧𝑖𝑝 𝑐𝑜𝑑𝑒= 3.57 km²
![Page 30: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/30.jpg)
Geography of Germany • Wikipedia:
o northernmost point in mainland Germany: Aventoft, Schleswig-Holstein 54°54′N 8°49′E
o southernmost point: Haldenwanger Eck, Oberstdorf, Bavaria 47°16′12.39″N 10°10′41.95″E
𝑥
1 𝑧𝑖𝑝 𝑐𝑜𝑑𝑒=
850 𝑘𝑚
99999 𝑧𝑖𝑝 𝑐𝑜𝑑𝑒 𝑥 =
850 𝑘𝑚
99999 𝑧𝑖𝑝 𝑐𝑜𝑑𝑒= 0.0085 km
SELECT ST_Distance_Sphere( ST_GeomFromText( 'POINT(54.900176 8.816667)', 4326 ), ST_GeomFromText( 'POINT(47.270108 10.178319)', 4326 )) / 1000;
?column?
------------------
850.346457590326
(1 row)
![Page 31: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/31.jpg)
Let‘s start easy • Question: what is the distance between:
• 08523 Plauen, Germany
• 95028 Hof (Saale), Germany
𝑥
86505 𝑧𝑖𝑝 𝑐𝑜𝑑𝑒=
850 𝑘𝑚
99999 𝑧𝑖𝑝 𝑐𝑜𝑑𝑒
𝑥 =850 𝑘𝑚 ∗ 86505 𝑧𝑖𝑝 𝑐𝑜𝑑𝑒
99999 𝑧𝑖𝑝 𝑐𝑜𝑑𝑒= 735.4 km
![Page 32: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/32.jpg)
Let‘s start easy • Question: what is the distance between:
• 08523 Plauen, Germany
• 95028 Hof (Saale), Germany
• Merely 42 km
© Google Maps
![Page 33: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/33.jpg)
Ok … • That escalated quickly …
• Zip codes aren‘t easy
• Ferry doesn‘t work, and neither does customs
$ psql psql (9.3.4) Type "help" for help.
![Page 34: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/34.jpg)
Time … • to GeoCode addresses
![Page 35: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/35.jpg)
Geocoding • Look up geocoordinates
o Longitude: X
o Latitude: Y
© Wikipedia
![Page 36: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/36.jpg)
Reverse Geocoding • Look up near(est) address for geocoordinates
![Page 37: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/37.jpg)
How (to enrich your data)
![Page 38: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/38.jpg)
Data Sources • OpenStreetMap
• ArcGIS
• Google Maps
• Bing Maps
• …
• TIGER (USA)
• …
![Page 39: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/39.jpg)
OpenStreetMap • „OpenStreetMap is built by a community of
mappers that contribute and maintain data about
roads, trails, cafés, railway stations, and much more,
all over the world.” o Quote from http://www.openstreetmap.org/about
• Project „Nominatim“ provides a webservice for
Geocoding and Reverse Geocoding
• You can (and should) setup your own server
![Page 40: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/40.jpg)
ArcGIS • Commercial service, offering maps, web platform,
app framework, paid services and much more
• Also provides free Geocode and Reverse Geocode
in the web API
![Page 41: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/41.jpg)
PostgreSQL • Let‘s do this from with the database:
o pg_osm: https://github.com/andreasscherbaum/pg_osm
o pg_arcgis: https://github.com/andreasscherbaum/pg_arcgis
arcgis=# SELECT * from arcgis.find('3495 Deer Creek
Road Palo Alto, CA 94304');
NOTICE: resolving: 3495 Deer Creek Road Palo Alto, CA
94304
CONTEXT: PL/Python function "find"
x | y | wkid
----------------+---------------+------
-122.148659529 | 37.3939849106 | 4326
(1 row)
![Page 42: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/42.jpg)
PostgreSQL arcgis=# select arcgis.find_yx('3495 Deer
Creek Road Palo Alto, CA 94304');
find_yx
--------------------------------
{37.3939849106,-122.148659529}
(1 row)
![Page 43: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/43.jpg)
find_yx(lat, lon)
© Google Maps
![Page 44: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/44.jpg)
Measure (the results)
![Page 45: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/45.jpg)
First demo set • This started as a project for Pivotal Greenplum
Database (a PostgreSQL fork) - performance
• Pivotal has 17 offices worldwide
• Scrape the addresses from the website and load
them into the database
![Page 46: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/46.jpg)
DEMO?
![Page 47: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/47.jpg)
DEMO! • Lookup addresses (live/cached)
o Single address (live)
o Pivotal offices (live)
![Page 48: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/48.jpg)
Pivotal addresses • ArcGIS: worked out of the box (just removed „floor“,
„suite“ ect) for 15 out of 17 addresses o China and South Korea are listed for weak data
arcgis=# select address, location_tmp from test_arcgis.pivotal_addresses order by id;
address | location_tmp
--------------------------------------------------------------------------------------------+------------------------------
3495 Deer Creek Road Palo Alto, CA 94304 | -122.148659529 37.3939849106
38 Xiaoyun Road, Beijing, 100027 China |
City Gate, Mahon, Cork, Ireland | -8.405141449 51.8861076
Am Kronberger Hang 2A, Schwalbach Frankfurt 65824, Germany | 8.532926108 50.1649573
Bentima House, 168-172 Old Street, London EC1V 9BP, United Kingdom | -2.61271281 51.225472175
Ribera del Loira 8, Edificio Paris, Campo de las Naciones, Madrid 28042, Spain | -3.591640162 40.462199799
Via Spadolini, 5, Edificio A, Milano 20141, Italy | 9.190325913 45.443185172
2000 Bandra Kurla Complex, Bandra East, Mumbai, Maharashtra, 400051, IND | 72.859927111 19.062134031
625 Avenue of the Americas, New York, NY 10011-2020, United States | -73.9948157784 40.7401885687
80 Quai Voltaire, Bezons Cedex 95876, France | 2.2202362 48.92280968
875 Howard St, San Francisco, CA 94103, United States | -122.404315996 37.7819550187
152 Teheran-ro, Seoul, 135-984, Korea |
1 Changi Business Park Central 1, #08-101, Changi City, Singapore 486036 | 103.962589042 1.33351667
207 Pacific Highway, St Leonards, Sydney, NSW, 2065, Australia | 151.193129783 -33.823683444
2-1-1 Yoyogi Shibuya-ku, Tokyo 151-0053, Japan | 139.698956933 35.686568319
1 Toronto Street, Suite 1100, Toronto, Ontario, M5C 2V6, Canada | -79.376037693 43.649893575
Edisonbaan 14b, 3439 MN Nieuwegein, 3430 AB Nieuwegein, Netherlands PO Box 97, Netherlands | 5.098633003 52.033586606
(17 rows)
![Page 49: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/49.jpg)
Pivotal addresses • OSM: out of the box, only 4 addresses resolved
arcgis=# select address,location_tmp from test_osm.pivotal_addresses order by id;
address | location_tmp
---------------------------------------------------------------------------------------------+----------------------------
3495 Deer Creek Road Palo Alto, CA 94304 | -122.14782867439 37.394282
38 Xiaoyun Road, Beijing, 100027 China | 116.4616591 39.9602201
City Gate, Mahon, Cork, Ireland |
Am Kronberger Hang 2A, Schwalbach Frankfurt 65824, Germany |
Bentima House, 168-172 Old Street, London EC1V 9BP, United Kingdom |
Ribera del Loira 8, Edifico Paris, Campo de las Naciones, Madrid 28042, Spain |
Via Spadolini, 5, Edificio A, Milano 20141, Italy |
2000 Bandra Kurla Complex, Bandra East, Mumbai, Maharashtra, 400051, IND |
625 Avenue of the Americas, New York, NY 10011-2020, United States |
80 Quai Voltaire, Bezons Cedex 95876, France |
875 Howard St, San Francisco, CA 94103, United States | -122.4137689 37.774516
152 Teheran-ro, Seoul, 135-984, Korea |
1 Changi Business Park Central 1, #08-101, Changi City, Singapore 486036 |
207 Pacific Highway, St Leonards, Sydney, NSW, 2065, Australia | 151.1883288 -33.8245562
2-1-1 Yoyogi Shibuya-ku, Tokyo 151-0053, Japan |
1 Toronto Street, Suite 1100, Toronto, Ontario, M5C 2V6, Canada |
Edisonbaan 14b , 3439 MN Nieuwegein, 3430 AB Nieuwegein, Netherlands PO Box 97, Netherlands |
(17 rows)
![Page 50: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/50.jpg)
Pivotal addresses • OSM: with „address tuning“, max 10 addresses resolved
o But: China and South Korea resolved!
arcgis=# select address,location_tmp from test_osm.pivotal_addresses order by id;
address | location_tmp
--------------------------------------------------------------------------------+----------------------------
3495 Deer Creek Road Palo Alto, CA 94304 | -122.14782867439 37.394282
38 Xiaoyun Road, Beijing, 100027 China | 116.4616591 39.9602201
City Gate, Mahon, Cork, Ireland |
Am Kronberger Hang 2A, Schwalbach 65824, Germany | 8.5330088 50.1643734
168-172 Old Street, London EC1V 9BP, United Kingdom | -0.0905437 51.5251618
Ribera del Loira 8, Edificio Paris, Campo de las Naciones, Madrid 28042, Spain |
Via Spadolini, 5, Edificio A, Milano 20141, Italy |
2000 Bandra Kurla Complex, Bandra East, Mumbai, Maharashtra, 400051, IND |
625 Avenue of the Americas, New York, NY 10011, United States | -73.9949875 40.7402113
80 Quai Voltaire, Bezons Cedex 95876, France |
875 Howard St, San Francisco, CA 94103, United States | -122.4137689 37.774516
152 Teheran-ro, Seoul, 135, South Korea | 127.063216 37.5087407
1 Changi Business Park Central 1, #08-101, Changi City, Singapore 486036 |
207 Pacific Highway, St Leonards, Sydney, NSW, 2065, Australia | 151.1883288 -33.8245562
2-1-1 Yoyogi Shibuya-ku, Tokyo 151-0053, Japan |
1 Toronto Street, Toronto, Canada | -79.3759569 43.6500879
Edisonbaan 14b, Nieuwegein, Netherlands | 5.0963629 52.034968
(17 rows)
![Page 51: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/51.jpg)
Second demo set • Pivotal is a spin-off from EMC and VMware
o Let‘s look after EMC addresses
• Yielded 196 addresses worldwide o Excluding USA (another test set)
o Excluding some addresses which not even Google Maps could resolve
![Page 52: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/52.jpg)
EMC has funny offices
© Google Maps
![Page 53: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/53.jpg)
Second demo set • EMC addresses worldwide (non USA)
o 196 addresses in data set
o ArcGIS: 165 resolved
o OSM: 71 resolved
• OSM found 3 places which ArcGIS couldn‘t find
![Page 54: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/54.jpg)
Second demo set • EMC addresses worldwide (non USA)
o 196 addresses in data set
o ArcGIS: 165 resolved (84%)
o OSM: 71 resolved (36%)
• OSM found 3 places which ArcGIS couldn‘t find
o 68 locations resolved from both services (34%)
• min distance: 6.7 meters
• max distance: 6177.34 meters (6.1 km)
• avg distance: 539.44 meters
• stddev: 1063.97 (wide spread)
![Page 55: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/55.jpg)
DEMO! • Lookup addresses (live/cached)
o EMC (cached)
![Page 56: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/56.jpg)
3rd & 4th data set • 3rd data set: EMC offices USA
o 216 addresses in total
o ArcGIS: 216 resolved (100%)
o OSM: 133 resolved (61%)
• 4th data set: Target shops USA o 1779 addresses in total
o ArcGIS: 1779 resolved (100%)
o OSM: 1205 resolved (67%)
![Page 57: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/57.jpg)
DEMO • Find nearest shop per office
![Page 58: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/58.jpg)
Conclusion • ArcGIS:
o 100% resolve rate in USA
o 80% - 90% resolve rate outside the USA
• OpenStreetMap: o 60% - 70% resolve rate in USA
o Drops below 40% outside the USA
o Good data available where ArcGIS is weak
o In general more data available (if address found)
o Returned data is not formalized (returns whatever user entered for the data point) – makes it hard to parse
![Page 59: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/59.jpg)
Performance • Lookups happen per row
o In PostgreSQL, only one connection is serving the request -> everything is serialized
o In Greenplum, every segment can do the lookup in parallel
![Page 60: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/60.jpg)
Performance Greenplum
Standby
Segment Segment Segment Segment
Master
Master Server: Talks SQL to
the client
1 – n Segment Server: „hidden“ MPP
database
Row processing happens here, in parallel
![Page 61: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/61.jpg)
Performance Greenplum
GP DB
GPDB Standard Module
27.5TB Data (Usable Physical Capacity Per module)
110TB Useable User Data Capacity per module, 4x compression)
Four 2U servers per module, each server contains: • 2 Sandybridge sockets/16
cores • 64GB Mem • 24x 900GB SAS 10K Drives
• 8 Segmente je Server
• 32 Segmente je Module
• 128 Segmente je Rack
• = 128 parallele Requests
![Page 62: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/62.jpg)
What‘s next
![Page 63: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/63.jpg)
Python • Add proper http headers
• Better error handling
• Configuration (code already there, not used)
• Add pg_google module
![Page 64: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/64.jpg)
PostgreSQL • Packages
• More test data
• Test „Address Standardizer“
![Page 65: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/65.jpg)
ArcGIS • Integrate (some) paid services
o Route calculation
o Route plotting on map
![Page 66: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/66.jpg)
OpenStreetMap • Find a better service than „Nomination“
o MapQuest Resolver
o Photon
• Integrate routing service o Announced a few weeks ago
• Talk with the „Nominatim“ developers
![Page 67: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/67.jpg)
Why did you not … • Integrate country-specific modules (like TIGER)
o Users don‘t care, they want the address resolved
o Certainly they don‘t care about specific countries
o But: I might implement it „under the hood“
• Google/Bing Maps o Working on it …
![Page 68: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/68.jpg)
Questions • Will you upload the slides?
o http://andreas.scherbaum.la/ -> writings
![Page 69: Enrich your data with geocoordinates from OpenStreetMap or ... · PostgreSQL since 1998 • Founding member of PGEU • Board of Directors: PGEU • PostgreSQL Regional Contact for](https://reader034.fdocuments.in/reader034/viewer/2022042320/5f0a91777e708231d42c4593/html5/thumbnails/69.jpg)
Thank you