TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp -...

Urban Traffic Data Hackathon TfL Datasets

14/15 November 2015 mary-‐[email protected]

[email protected]

Version 1.7

Summary Slides …

15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 2

Challenge 1 / Answer 3 ….... • Data mapping to a common geography • Within TfL our most granular view of our network is defined by the Ordinance Survey Integrated TOID (Topological IdenPfier) Network (OS ITN). •  TOID = Unique ID of feature on the Ordinance Survey map.

• However, it not common that our data sets can map directly onto this. •  Really?

•  It is our vision that data fusion can enable a single view of performance which takes input from all available datasets and enable us to produce a consistent view of our network. •  You bet! Our vision too. You asked …


Tiered Topology Model Mandrel “The InformaOon Architects”

Challenge 1 In 1933 Harry Beck revoluPonized maps. In 2015 we have to leave the map again… •  Edges & Vertex

•  Unique Id •  Hierarchy

•  Topology per Mode •  Physical &

Logical

•  Linked Topologies •  Across Modes


1933

2015

Unconnected Mapping Data So Which Sensor relates To Which Bus Route & Are We On A Bus Lane?


Unconnected Mapping Data – Over OSM So Which Sensor relates To Which Bus Route & Are We On A Bus Lane?


Logical Model With Hierarchy Of Routes Per Transport Mode Links Down To Physical Model(s)

Bus 1 Stop at X,Y

(Vertex)

Bus 2 Stop at X,Y

(Vertex)

Bus 3 Stop at X,Y

(Vertex) Route Leg 2 (Edge)

Route Leg 1 (Edge)

Route (Vertex) Route Has Leg

(Edge) Route Has Leg(Edge)


Bus 3 Stop at X,Y

(Vertex)

Road Signals & Sensors (Vertex)

Road SecPons (Edge), Waypoints (Vertex), Bus Lanes

& JuncPons

We can now tell which Sensor relates To Which Bus Route & If We Are We On A Bus Lane

More Hierarchy Links Physical & Logical Views Two Physical Stops Either Side Of Road

Bus 1a Stop at X,Y

(Vertex)

Bus 2a Stop at X,Y

(Vertex)

Route

End

Bus 1a Stop at X,Y

(Vertex)

Route Leg 1a (Edge)

Bus 2a Stop at X,Y

(Vertex)

Route Leg 2a (Edge)

Marble Arch Bus “Stop” (Vertex)

Has Bus Stop (Edge)

Has Bus Stop (Edge)


Now Link Modes With Summary InformaOon

Different parts of the hierarchy are needed to map between modes

Logical model of Tube

journey ends at staPon

Physical model of staPon

has many exits

StaPon exit has walking route to

nearest bus stop of 5 min

Bus route runs from bus stop to bus stop near

home


Full Slide Deck …


Challenge 1 – An Unexpected Answer? •  Data mapping to a common geography

•  Within TfL our most granular view of our network is defined by the Ordinance Survey Integrated TOID (Topological IdenPfier) Network (OS ITN).

•  However, it not common that our data sets can map directly onto this.

•  It is our vision that data fusion can enable a single view of performance which takes input from all available datasets and enable us to produce a consistent view of our network.

•  Our answer is a vision •  In 1933 Harry Beck produced the beauPful

design for the London Underground. •  It was simple, a topology, not a map. •  For AnalyPcs 3.0 we have to take the idea

further breaking the data away from being directly dependent physical map locaPons.

•  Adopt a vision to refine and link all of the reference data in shape files to a set of linked Pered topologies covering the TfL network and beyond.


Unconnected Mapping Data So Which Sensor relates To Which Bus Route & Are We On A Bus Lane?


Unconnected Mapping Data – Over OSM So Which Sensor relates To Which Bus Route & Are We On A Bus Lane?


AnalyOcs 2.0 Early Big Data IntegraOon Pair-‐wise Linkage Of Data Sources

Find a way to smash data

together


AnalyOcs 3.0 Coherent reference data model Joins mulOple data sources

Single Reference Data Model. Tiered Linked Topologies.

Slow changing reference data:

staPons, routes, ….

High volume data. E.g. sensor and trip data.

PosiPvely Link related enPtles. Not just rely on similar map locaPons.


How Do We Know You Need This? •  We wanted to do clever things at the Hackathon but …. •  We spent too much Pme trying to merge

GIS and reference informaPon and tear apart Shape files •  Not recommended in R or Python

•  Other network based industries have gone through exactly the same pain. •  For example the mobile phone networks

are (too late) desperately trying to build unified topology models of their mulP-‐layered mulP-‐Pered mulP-‐technology networks.

•  New big data tooling is built to handle data network model. •  E.g. Spark’s GraphX


Some Detailed DirecOon

• Here is what we know interpreted from other industries ….


Everything Becomes Directed Line Graphs Stops (Vertex) & Route Leg (Edges)

Bus Route (One DirecPon)

Bus 1 Stop at X,Y

(Vertex)

Bus 2 Stop at X,Y

(Vertex)

Bus 3 Stop at X,Y


Route Leg 1 (Edge)


Introduce Hierarchy In Logical Model More VerOces and Edges Build Routes

Bus 1 Stop at X,Y

(Vertex)

Bus 2 Stop at X,Y

(Vertex)

Bus 3 Stop at X,Y


Route Leg 1 (Edge)

Route (Vertex) Route

Has Leg (Edge)

Route Has Leg (Edge) 15/11/15 © 2015 Mandrel Systems Limited, All Rights

Reserved 19

More Hierarchy Links Physical & Logical Views Two Physical Stops Either Side Of Road

Bus 1a Stop at X,Y

(Vertex)

Bus 2a Stop at X,Y

(Vertex)

Route

End

Bus 1a Stop at X,Y

(Vertex)

Route Leg 1a (Edge)

Bus 2a Stop at X,Y

(Vertex)

Route Leg 2a (Edge)

Marble Arch Bus “Stop” (Vertex)

Has Bus Stop (Edge)

Has Bus Stop (Edge)


Route Legs Link To Map Physical Topology E.g. road Segments, Waypoints, Sensors Etc. (Very Simplified Drawing)

Route Leg 1 (Edge) Bus 1

Stop at X,Y

(Vertex)

Bus 2 Stop at X,Y

(Vertex)

Road Sensor 23423 (Vertex)

Road SecPons (Edge),

Waypoints (Vertex)

& JuncPons

For each transport mode you will have

one or more linked

physical and logical models.


Now Link Modes With Summary InformaOon

Different parts of the hierarchy are needed to map between modes

Logical model of Tube

journey ends at staPon

Physical model of staPon

has many exits

StaPon exit has walking route to

nearest bus stop of 5 min

Bus route runs from bus stop to bus stop near

home


Robust and PracOcable Model Only when all the model is built can you add in mulPple data sets…

…to map mulP-‐modal journeys


Step One … Add Your Own Unique IDs Everywhere

•  Everything Has a Unique Numeric ID •  So Anything can Link to Anything In Either

DirecPon •  Bus Stop To Tube Exit •  MulPple Tube Exit to Tube StaPon •  Tube StaPon To Railway StaPon •  Railway StaPon To Plaporms •  Railway StaPon To Taxi rank

•  Enables mulP-‐mode in single topology •  Some things may have a locaPon or link to a map

TOID •  But can’t just link everything via a OS map enPty

ID (TOID)

•  Extra Nodes & Edges Build Hierarchy •  A sensor is on a road segment •  MulPple road segments make up a journey leg •  MulPple journey legs make up a route run

Bus 1 Stop at X,Y

(Vertex)

Tube StaPon Exit X,Y (Vertex)

Walk: 3 Min (Edge)


Step Two … Consistent Time and Agribute Structure

•  You have at least 7 different representaPons of Time •  To simplify linkage seqle on a single reference representaPon of Pme and Pme intervals

• All your different data sources have an underlying structure •  Use it to make searching and linking feasible and Pmely


To ConOnue …

•  TacPcally •  Please do release as much reference data as you can.

•  Be kind to Web developers and also release shape files as GeoJson with Lat/Long.

•  Strategically

•  Build the coherent model. •  Please involve us.


For More InformaOon

•  Let’s Have A Chat •  mary-‐[email protected] •  [email protected]

•  See Also: •  Processing Graphs: Spark GraphX

•  hqp://spark.apache.org/docs/latest/graphx-‐programming-‐guide.html


TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp -...

Documents

Transcript of TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp -...