Deep Dive: Take a Tour With Geospatial...“demands that each attribute in the list have a formal...

© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Deep Dive: Take a Tour With Geospatial Jake Fowler (Senior Software Engineer) & Jennifer Tsau (Product Manager)

SLIDE: 2


Disclaimer

MarkLogic 9 is currently under development. The features and how they are implemented are subject to change.

SLIDE: 3


Before we get started, let’s do a quick poll… … How many of you are using MarkLogic’s geospatial features today?

… How many of you are not using MarkLogic’s geospatial features today,

but are familiar with the broader community of geospatial technology (GIS)?

…How many of you attended the

Designing with MarkLogic –

Geospatial Workshop?

SLIDE: 4


80% of all data has a geospatial component

SLIDE: 6


• A Paradigm Shift & Unique Solution: MarkLogic Geospatial

• Technical Deep-Dive: A Journey Into the Internals of MarkLogic Geospatial

Today’s Itinerary DEEP DIVE: TAKE A TOUR WITH GEOSPATIAL

PARADIGM SHIFT

AHEAD

SLIDE: 7


FT_FeatureID FT_Geometry FT_FeatureType FT_FeatureUse

1 Polygon Non-residential School

2 Polygon Non-residential Training Center

3 Polygon Non-residential Church

4 Polygon Residential Apartment Complex

Geospatial concepts have been boxed by a relational mindset

TODAY’S WORLD

SLIDE: 8


FT_FeatureID FT_Geometry FT_FeatureType FT_FeatureUse

1 Polygon Non-residential School

2 Polygon Non-residential Training Center

3 Polygon Non-residential Church

4 Polygon Residential Apartment Complex

Geospatial concepts have been boxed by a relational mindset

TODAY’S WORLD

OGC OpenGIS Abstract Specification Topic 5: Features Geometry Schema: “demand that rules for representing each feature type with WKTs is explicit” Attribute Schema “demands that each attribute in the list have a formal attribute name, and that each attribute take a value of a type specific for that attribute”

SLIDE: 9


Think of, interact with “location” in a different way

WELCOME TO MARKLOGIC WORLD

SLIDE: 10


Variance in geospatial representation (e.g. point, polygon)

Variance in attributes

Paradigm Shift #1 – Create features that mirror reality

1651 Union Street, San Francisco, CA

<doc>

<feature>

<address>1651 Union Street, San Francisco, CA<address>

<coordinates>POINT(-122.4283871 37.7980631)</coordinates>

<type>Non-residential</type>

<address-usage>

<use>school</use>

<geometry>MULTIPOLYGON((…))</geometry>

<name>Sherman Elementary</feature-name>

<population>1200</population>

</address-usage>

<address-usage>

<use>training center</use>


<contact-email>[email protected]</contact-email>

<contact-phone>1-415-123-4567</contact-phone>

</address-usage>

<address-usage>

<use>church</use>

<geometry>POLYGON((…))</geometry>

<notes>Meets in school auditorium</notes>


</address-usage>

</feature>

...

</doc>

FT_FeatureID FT_Geometry FT_Type FT_Use FT_Address_ID

1 Multipolygon((…)) Non-residential

School 1

2 Multipolygon((...)) Non-residential

Training Center 1

3 Polygon((…)) Non-residential

Church 1

FT_Address_ID FT_Address FT_Coordinates

1 1651 Union Street, San Francico, CA

Point(-122.42838 37.79806)

… … …

• What do I do when there is variance in my attributes?

• What if I introduce another feature that doesn’t “fit” the same schema?

4 Point(-122.42840 37.79817

Residential Apartment Complex

2

2

1600 Clay Street, San Francisco, CA

Point(-122.42840 37.79817)

<doc>

<feature>

<address>1651 Union Street, San Francisco, CA<address>


<type>Non-residential</type>

<use>school</use>


<name>Sherman Elementary</feature-name>

<population>1200</population>

<use>training center</use>


<contact-email>[email protected]</contact-email>


<use>church</use>

<geometry>POLYGON((…))</geometry>


</feature>

</doc>

FT_FeatureID FT_Geometry FT_Type FT_Use FT_Address_ID

1 Multipolygon((…)) Non-residential

School 1

2 Multipolygon((...)) Non-residential

Training Center 1

3 Polygon((…)) Non-residential

Church 1

FT_Address_ID FT_Address FT_Coordinate

1 1651 Union Street, San Francico, CA

Point(-122.42838 37.79806)

… … …

• What do I do when there is variance in my attributes?

• What if I introduce another feature that doesn’t “fit” the same schema?

4 Point(-122.42840 37.79817

Residential Apartment Complex

2

2

1600 Clay Street, San Francisco, CA

Point(-122.42840 37.79817)

Greater flexibility in defining a “feature”

Greater flexibility in building your location-aware applications

SLIDE: 13


Paradigm Shift #2 – Integrate geospatial across the org

FT_ID FT_Location FT_FacilityType FT_FeatureName

1 Polygon((..)) Hospital Oakland Clinic

2 Polygon((..)) Hospital UCSF

3 Polygon((..)) Medical Foundation

Palo Alto Medical Foundation

FT_ID FT_Address PatientID

9012 Point(…) 7432

9013 Point(…) 7433

9014 Point(…) 7434

9015 Point(…) 7435

PatientID PatientName PatientAddress PatientDOB SSN

7432 Betty Crocker 123 Cherry Lane 1973-10-04 589-78-4562

7433 Sam Kumar 523 Happy Ave 1982-04-12 789-75-9865

7434 John Chen 999 Holly Street 1965-07-23 612-89-1268

7435 Sally Smith 681 Branch Lane 1942-11-13 432-32-5932

PAT_RecID PAT_First PAT_Last PAT_Home

432-32-5932 Sally Smith 314 Franklin Dr

311-54-2391 Albus Dumbledore 97 Hogwarts Ln

431-59-4321 Hermoine Granger ….

695-27-1093 Ron Weasley ….

EncounterID PatientID ProviderID

65871 7435 5-87965

65872 … …

MED_ID LAST FIRST ADDRESS

9435-12 Jennifer Lawrence 233 Holly Street

… … … …

MeetingRec MED_ID PHYSICIAN_ID

3439204 9435-12 1234-56

65872 … …

ApptID PAT_RecID PROV_ID

YAS-41 432-32-5932 9-32156

DPD-43 311-54-2391 8-42132

SLR-52 … …

SLIDE: 14


Paradigm Shift #2 – Integrate geospatial across the org

<patient-doc> <UCSF-patient-record> <PatientID>7435</PatientId> <PatientName>Sally Smith</PatientName> <PatientAddress>681 Branch Lane</PatientAddress> <PatientDOB>1942-11-13</PatientDOB> <coordinates>POINT(-122.4283871 37.7980631)</coordinates>

... </UCSF-patient-record> <UCSF-appointment-record> <apptID EncounterID=“7435”> <ProviderID>5-87965</ProviderID> <location>POLYGON((..))</location> </apptID>

... </UCSF-appointment-record> <oakland-clinic-patient-record> <PAT_RecID>432-32-5932</PAT_RecID> <PAT_First>Sally</PAT_First> <PAT_Last>Smith</PAT_Last> <PAT_Home>314 Franklin Dr</PAT_Home> <coordinates>POINT(..)</coordinates>

... </oakland-clinic-patient-record> <oakland-clinic-appointment-record> <apptID ApptID=“YAS-41”> <PROV_ID>9-32156</PROV_ID> <location>POLYGON((..))</location> </apptID>

... </oakland-clinic-appointment-record> </patient-doc>

PatientID

PatientName PatientAddress PatientDOB

7432 Betty Crocker 123 Cherry Lane 1973-10-04

7433 Sam Kumar 523 Happy Ave 1982-04-12

7434 John Chen 999 Holly Street 1965-07-23

7435 Sally Smith 681 Branch Lane 1942-11-13






65871 7435 5-87965

65872 … …


YAS-41 432-32-5932 9-32156

DPD-43 311-54-2391 8-42132

SLR-52 … …






<patient-doc> <UCSF-patient-record> <PatientID>7435</PatientId> <PatientName>Sally Smith</PatientName> <PatientAddress>681 Branch Lane</PatientAddress> <PatientDOB>1942-11-13</PatientDOB> <coordinates>POINT(-122.4283871 37.7980631)</coordinates>

... </UCSF-patient-record> <UCSF-appointment-record> <apptID EncounterID=“7435”> <ProviderID>5-87965</ProviderID> <location>POLYGON((..))</location> </apptID>

... </UCSF-appointment-record> <oakland-clinic-patient-record> <PAT_RecID>432-32-5932</PAT_RecID> <PAT_First>Sally</PAT_First> <PAT_Last>Smith</PAT_Last> <PAT_Home>314 Franklin Dr</PAT_Home> <coordinates>POINT(..)</coordinates>

... </oakland-clinic-patient-record> <oakland-clinic-appointment-record> <apptID ApptID=“YAS-41”> <PROV_ID>9-32156</PROV_ID> <location>POLYGON((..))</location> </apptID>

... </oakland-clinic-appointment-record> </patient-doc>

PatientID

PatientName PatientAddress PatientDOB

7432 Betty Crocker 123 Cherry Lane 1973-10-04

7433 Sam Kumar 523 Happy Ave 1982-04-12

7434 John Chen 999 Holly Street 1965-07-23

7435 Sally Smith 681 Branch Lane 1942-11-13






65871 7435 5-87965

65872 … …


YAS-41 432-32-5932 9-32156

DPD-43 311-54-2391 8-42132

SLR-52 … …






Location data lives with the rest of your data

No need to manage it in a separate, special way

SLIDE: 17


Paradigm Shift #3 – Interact with location using rich, complex queries

<patient-doc>

<patient-record>

<PatientID>7435</PatientId>

<PatientName>Henry Evans</PatientName>

<PatientAddress>681 W Court street</PatientAddress>

<PatientDOB>2012-05-13</PatientDOB>


...

</patient-record>

<appointment-record>

<apptID EncounterID=“7435”>

<ProviderID>5-87965</ProviderID>

<location>POLYGON((..))</location>

<time>2016-05-03T14:30:00</time>

<Diagnosis>Food Poisoning</Diagnosis>

<Notes>

Abdominal pain, vomiting, nausea. Vomiting for

1 week, abdominal pain symptoms 2 weeks and

persisting

</Notes>

</apptID>

...

</appointment-record>

</patient-doc>

Combine full-text search with geospatial

Build alerting functionality based on location

“Find me all records for patients who live in the 5-mile radius of Flint, Michigan that mention the phrase “abdominal pain” and “nausea” in the same appointment.

“Whenever a patient under 10, who lives within a 5-mile radius of Flint, Michigan checks in, trigger an alert for the nurse to conduct a lead poisoning test.”

<patient-doc>

<patient-record>

<PatientID>7435</PatientId>

<PatientName>Henry Evans</PatientName>

<PatientAddress>681 W Court street</PatientAddress>

<PatientDOB>2012-05-13</PatientDOB>


...

</patient-record>

<appointment-record>

<apptID EncounterID=“7435”>

<ProviderID>5-87965</ProviderID>

<location>POLYGON((..))</location>

<time>2016-05-03T14:30:00</time>

<Diagnosis>Food Poisoning</Diagnosis>

<Notes>

Abdominal pain, vomiting, nausea. Vomiting for

1 week, abdominal pain symptoms 2 weeks and

persisting

</Notes>

</apptID>

...

</appointment-record>

</patient-doc>

Combine full-text search with geospatial

Build alerting functionality based on location

“Find me all records for patients who live in the 5-mile radius of Flint, Michigan that mention the phrase “abdominal pain” and “nausea” in the same appointment.

“Whenever a patient under 10, who lives within a 5-mile radius of Flint, Michigan checks in, trigger an alert for the nurse to conduct a lead poisoning test.”

Write rich, complex queries that involve location

Ask the “where” questions you couldn’t ask before

SLIDE: 20


Customers use MarkLogic Geospatial to…

SLIDE: 21


Fairfax County Unified land use search platform across

5.4 million records

Improved service to county employees and citizens

Real-time data loaded in hours, not weeks

Reduced maintenance and support costs

CUSTOMER USE CASE

SLIDE: 22


DHL Parcel Benelux Track and trace system for 150,000

packages/day

First-class, speedy service to retail clients

Flexibility to support new e-commerce companies

Faster data integration, faster ROI

CUSTOMER USE CASE

SLIDE: 23


Enterprise NoSQL Database Platform

Geospatial Support

Native JSON Store

Native XML Store

Native RDF Triple Store

POWERFUL

Flexible Indexes

Full-text Search

Real-time Alerting Bitemporal

Tiered Storage

Semantic Inference

Fully Transactional

Server-side JavaScript

REST API

Scalable and Elastic

Cloud Ready (AWS)

Hadoop and HDFS

AGILE

Multi-OS Support

SQL Support

Samplestack Schema Agnostic

XA Transactions

MarkLogic Content Pump

Index Across Data Types

Ad-hoc Queries

Monitoring and Management

Performance at scale

LDAP & Kerberos Security

Security Certifications

TRUSTED

24/7 Engineering Support

Configuration Management

Flexible Replication

ACID Transactions

Customizable Failover

Customizable Backup

Atomic Forests

Point-in-time Recovery

SLIDE: 24


MarkLogic Geospatial Features

Geospatial Point Search

WGS84 & Raw Coordinate System

indexes

Float precision indexes

A wide variety of geospatial APIs

KML GML GeoRSS Metacarta

WKB

WKT

GeoJSON

Geospatial Region Search

Double precision indexes

MARKLOGIC GEOSPATIAL Technical Deep Dive

Geospatial Data An example…

{"type":"Feature","properties":{"facility_i":"0000003185","dept":"7","school_typ":"Elementary","deptname":"SF Unified School District (SFUSD)","facility_n":"Ulloa Elementary School"},"geometry":{"type":"Point","coordinates":[-122.49938854722042,37.73733034833979]}}

<Placemark><ExtendedData><Data name="deptname"><value>SF Unified School District (SFUSD)</value></Data><Data name="facility_n"><value>Ulloa Elementary School</value></Data><Data name="facility_i"><value>0000003185</value></Data><Data name="dept"><value>7</value></Data><Data name="school_typ"><value>Elementary</value></Data></ExtendedData><Point><coordinates>-122.49938854722042,37.73733034833979</coordinates></Point></Placemark>

SLIDE: 29


Latitude and Longitude Geospatial data is stored by latitude and longitude.

These two numbers identify a point on the earth. Longitude

Latit

ude

{

"type": "Feature",

"properties": {

"facility_i": "0000003185",

"dept": "7",

"school_typ": "Elementary",

"deptname": "SF Unified School District (SFUSD)",

"facility_n": "Ulloa Elementary School"

},

"geometry": {

"type": "Point",

"coordinates": [-122.49938854722042,37.73733034833979]

}

}

SLIDE: 31


Geospatial Data in MarkLogic MarkLogic automatically understands the structure of the data.

MarkLogic does not know the meaning (semantics) of the data.

MarkLogic is schema-agnostic.

You must tell MarkLogic which part of the data is the location.

Why does MarkLogic need to know this?

To search for documents that contain locations in specific places!

Geospatial Search

SLIDE: 33


Answers location-based questions

Examples:

Have all the park benches between 5th and 7th street been painted?

Which of my customers live in or around San Francisco?

Geospatial Search

SLIDE: 34


Understanding Geospatial Search Geospatial Indexes enable fast location-based search

Search and indexing are two sides of the same coin

SLIDE: 35


What Is an index? An interesting subset of the data

References back to the source of the data

Organized to find specific items of interest

SLIDE: 36


MarkLogic Range Indexes MarkLogic’s range indexes also satisfy these three conditions.

Stores a specific subset of the data.

(Specified by XML element name, JSON property name, etc.)

Refers back to the original document URI.

Data is of a type that can be sorted.

Examples: numbers, dates, strings

URI: joe.xml

<user>

<name><first>Joe</first><last>Shmo</last></name>

<age>34</age>

<status>single</status>

</user>

<age> Document Document <age>

... ...

18 kim.xml bob.xml 19

19 bob.xml carol.xml 40

22 jeff.xml, jessie.xml jeff.xml 22

25 susan.xml jessie.xml 22

30 mary.xml, steve.xml joe.xml 34

34 joe.xml kim.xml 18

40 carol.xml mary.xml 30

52 rick.xml sam.xml 55

55 sam.xml steve.xml 30

... ...

SLIDE: 39


Geospatial Indexes Geospatial indexes are a type of range index

Points can be sorted

Sort first by latitude, then by longitude

A “two dimensional” range index

SLIDE: 40


Geospatial Indexes

SLIDE: 41


Geospatial Indexes

SLIDE: 42


SLIDE: 43


Geospatial Search Index resolution is easy for rectangles

Other regions require more steps

Let’s look at an example…

SLIDE: 44


//An example geospatial search...

var usa = cts.polygon(“POLYGON((...))”);

var qry = cts.elementGeospatialQuery(

xs.QName(“coordinates”),

usa);

cts.search(qry);

SLIDE: 46


Index Resolution Step 1: Restrict results based on bounding boxes

SLIDE: 47


SLIDE: 48



Quickly rules out many points that don’t match

Use finer bounding boxes

SLIDE: 49


SLIDE: 50




Use finer bounding boxes Fewer non-matches; minimal performance impact

See them with geo:bounding-boxes with option box-percent=100

Step 2: Check each point against the search region

SLIDE: 51


SLIDE: 52




Use finer bounding boxes Fewer non-matches; minimal performance impact

See them with geo:bounding-boxes with option box-percent=100

Step 2: Check each point against the search region

Result: Search returns accurate results

No filter step is necessary

Geometry on the Earth

SLIDE: 54



Vs.

SLIDE: 55


Map Projections

SLIDE: 56


Carl Friedrich Gauss (1777–1855)

Theorema Egregium (1827)

Any flat map will have distortion.

This is because the Earth has intrinsic curvature.

A.k.a. Gaussian curvature.


SLIDE: 57


“Non-Euclidean” geometry.

Shortest distance on Earth is not a straight line.

Shortest distance follows a Great Circle.

Also called a Geodesic.


SLIDE: 58


Great Circle Arcs

Image source: http://flylib.com/books/en/2.366.1.61/1/

SLIDE: 59



Vs.

SLIDE: 60


Geometry on the Earth Actually, Earth is not a sphere

It bulges around the equator

Due to the centrifugal force of its rotation

Earth is an ellipsoid

SLIDE: 61


Geodesic Polygons In MarkLogic, the edges of polygons are geodesics

Point-in-polygon – ray-casting algorithm

Is point P inside the polygon?

Draw a geodesic from P to exterior point (e.g. north pole)

Count number of intersections with polygon edges Odd => inside

Even => not inside

Worst case: must calculate intersections for every edge of polygon Computationally expensive for polygons with lots of vertices

SLIDE: 62


MarkLogic Geospatial Combine Geospatial Search with full-

text search and range queries to make rich applications.

Unlock your location data!

WRAP-UP

SLIDE: 64


What’s next? Wednesday, May 11

(11:00AM – 11:50AM) ESRI & MarkLogic: Do More With Your GIS (3:15PM – 4:05PM) EBCONT: Implementing a Big Data Strategy for the

Passenger Rail Agency of South Africa

SLIDE: 65


More Information & Resources MarkLogic Documentation

http://docs.marklogic.com/guide/search-dev/geospatial

MarkLogic University

Geospatial Basics (On Demand Course)

Using Node.js: The Geophoto App (On Demand Course)

Developer Community

http://developer.marklogic.com/

GitHub Repositories

Google Maps for MarkLogic

ESRI Maps for MarkLogic

http://docs.marklogic.com/guide/search-dev/geospatial

http://mlu.marklogic.com/ondemand/84f191ff

http://mlu.marklogic.com/ondemand/aa767d01

http://developer.marklogic.com/

http://grtjn.github.io/ml-google-maps-ng/

http://daveegrant.github.io/ml-esri-maps-ng/

SLIDE: 66


Q&A

Deep Dive: Take a Tour With Geospatial...“demands that each attribute in the list have a formal...

Documents

Transcript of Deep Dive: Take a Tour With Geospatial...“demands that each attribute in the list have a formal...