Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE...
Transcript of Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE...
![Page 1: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/1.jpg)
Alekh Jindal
GraphiQLGraph Intuitive Query Language for Relational Databases
Sam Madden
Mike Stonebraker
Amol Deshpande
MITUniversity
of Maryland
IEEE BigData 2014Talking onat
Supervisors
work
work
collaborate
work sabbatical
![Page 2: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/2.jpg)
Alekh Jindal
GraphiQLGraph Intuitive Query Language for Relational Databases
Sam Madden
Mike Stonebraker
Amol Deshpande
MITUniversity
of Maryland
IEEE BigData 2014Talking onat
Supervisors
work
work
collaborate
work sabbatical
Relational Database
Expensive!
Expensive!
![Page 3: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/3.jpg)
Graph Analysis =
Graph Algorithms
StoreExtractPreprocessUpdateFailoverPostprocess
+
![Page 4: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/4.jpg)
Graph Analysis =
Graph Algorithms
StoreExtractPreprocessUpdateFailoverPostprocess
+“Counting Triangles with Vertica”
“Scalable Social Graph Analytics Using the Vertica Analytic Platform,”
“Graph Analysis: Do We Have to Reinvent the Wheel?”
“Query Optimization of Distributed Pattern Matching,”
“GraphX: A Resilient Distributed Graph System on Spark,”
“Vertexica: Your Relational Friend for Graph Analytics!”
Relational DatabaseRelational Database
![Page 5: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/5.jpg)
Problem !
![Page 6: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/6.jpg)
Alekh Jindal
GraphiQLGraph Intuitive Query Language for Relational Databases
Sam Madden
Mike Stonebraker
Amol Deshpande
MITUniversity
of Maryland
IEEE BigData 2014Talking onat
Supervisors
work
work
collaborate
work sabbatical
Alekh Jindal
GraphiQLGraph Intuitive Query Language for Relational Databases
Sam Madden
Mike Stonebraker
Amol Deshpande
MITUniversity
of Maryland
IEEE BigData 2014Talking onat
Supervisors
work
work
collaborate
work sabbatical
SQL
![Page 7: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/7.jpg)
SELECT
UPDATEFROM
GROUP BY
SUM
COUNT
WHERE
![Page 8: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/8.jpg)
Redundant EffortAlekh Jindal
GraphiQLGraph Intuitive Query Language for Relational Databases
Sam Madden
Mike Stonebraker
Amol Deshpande
MITUniversity
of Maryland
IEEE BigData 2014Talking onat
Supervisors
work
work
collaborate
work sabbatical
Alekh Jindal
GraphiQLGraph Intuitive Query Language for Relational Databases
Sam Madden
Mike Stonebraker
Amol Deshpande
MITUniversity
of Maryland
IEEE BigData 2014Talking onat
Supervisors
work
work
collaborate
work sabbatical
Alekh Jindal
GraphiQLGraph Intuitive Query Language for Relational Databases
Sam Madden
Mike Stonebraker
Amol Deshpande
MITUniversity
of Maryland
IEEE BigData 2014Talking onat
Supervisors
work
work
collaborate
work sabbatical
Alekh Jindal
GraphiQLGraph Intuitive Query Language for Relational Databases
Sam Madden
Mike Stonebraker
Amol Deshpande
MITUniversity
of Maryland
IEEE BigData 2014Talking onat
Supervisors
work
work
collaborate
work sabbatical
![Page 9: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/9.jpg)
Optimizations?
![Page 10: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/10.jpg)
GraphiQL
![Page 11: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/11.jpg)
Alekh Jindal
GraphiQLGraph Intuitive Query Language for Relational Databases
Sam Madden
Mike Stonebraker
Amol Deshpande
MITUniversity
of Maryland
IEEE BigData 2014Talking onat
Supervisors
work
work
collaborate
work sabbatical
Alekh Jindal
GraphiQLGraph Intuitive Query Language for Relational Databases
Sam Madden
Mike Stonebraker
Amol Deshpande
MITUniversity
of Maryland
IEEE BigData 2014Talking onat
Supervisors
work
work
collaborate
work sabbatical
SQL
![Page 12: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/12.jpg)
Alekh Jindal
GraphiQLGraph Intuitive Query Language for Relational Databases
Sam Madden
Mike Stonebraker
Amol Deshpande
MITUniversity
of Maryland
IEEE BigData 2014Talking onat
Supervisors
work
work
collaborate
work sabbatical
Alekh Jindal
GraphiQLGraph Intuitive Query Language for Relational Databases
Sam Madden
Mike Stonebraker
Amol Deshpande
MITUniversity
of Maryland
IEEE BigData 2014Talking onat
Supervisors
work
work
collaborate
work sabbatical GraphiQL
![Page 13: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/13.jpg)
Key Features• Graph view of relational data; the system takes
care of mapping to the relational world
• Inspired from PigLatin: right balance between declarative and procedural style language
• Key graph constructs: looping, recursion, neighborhood access
• Compiles to optimized SQL
![Page 14: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/14.jpg)
Graph Table Relational Table
Alekh Jindal
GraphiQLGraph Intuitive Query Language for Relational Databases
Sam Madden
Mike Stonebraker
Amol Deshpande
MITUniversity
of Maryland
IEEE BigData 2014Talking onat
Supervisors
work
work
collaborate
work sabbatical
GraphiQL SQL
Graph Table
![Page 15: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/15.jpg)
Graph Elementsnode1node2edge1edge2edge3node3node4node5
weight typeid
node6
node7
edge4
edge5
edge6
node8
node9
edge7
edge8
edge9
outgoingincoming
Graph Table
![Page 16: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/16.jpg)
Graph Table Definition
• Create
• Load
• Drop
CREATE GRAPHTABLE g AS NODE (p1,p2,..) EDGE (q1,q2,..)
LOAD g AS NODE FROM graph_nodes DELIMITER d EDGE FROM graph_edges DELIMITER d
DROP GRAPHTABLE g
![Page 17: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/17.jpg)
Graph Table Manipulation
• Iterate
• Filter
• Retrieve
• Update
• Aggregate
FOREACH element in g [WHILE condition]
g’ = g(k1=v1,k2=v2,…,kn=vn)
GET expr1,expr2,…,exprn [WHERE condition]
SET variable TO expr [WHERE condition]
SUM, COUNT, MIN, MAX, AVG
![Page 18: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/18.jpg)
Nested Manipulation inner outer Iterate Aggregate Retrieve Update
Iterate
Aggregate
Retrieve
Update
![Page 19: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/19.jpg)
Example 1: PageRank
FOREACH n IN g(type=N) SET n.pr TO new_pr
![Page 20: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/20.jpg)
Example 1: PageRank
FOREACH n IN g(type=N) SET n.pr TO 0.15/num_nodes + 0.85*SUM(pr_neighbors)
![Page 21: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/21.jpg)
Example 1: PageRank
FOREACH n IN g(type=N) SET n.pr TO 0.15/num_nodes + 0.85*SUM( FOREACH n’ IN n.in(type=N) GET pr_n’ )
![Page 22: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/22.jpg)
Example 1: PageRank
FOREACH n IN g(type=N) SET n.pr TO 0.15/num_nodes + 0.85*SUM( FOREACH n’ IN n.in(type=N) GET n’.pr/COUNT(n’.out(type=N)) )
![Page 23: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/23.jpg)
Example 1: PageRank
FOREACH iterations IN [1:10] FOREACH n IN g(type=N) SET n.pr TO 0.15/num_nodes + 0.85*SUM( FOREACH n’ IN n.in(type=N) GET n’.pr/COUNT(n’.out(type=N)) )
![Page 24: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/24.jpg)
Example 1: PageRank
FOREACH iterations IN [1:10] FOREACH n IN g(type=N) SET n.pr TO 0.15/num_nodes + 0.85*SUM( FOREACH n’ IN n.in(type=N) GET n’.pr/COUNT(n’.out(type=N)) )
Reason about graphNeighborhood Access
Looping
Nested Manipulations
![Page 25: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/25.jpg)
Example 2: SSSP
FOREACH n IN g(type=N) SET n.dist TO min_dist
![Page 26: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/26.jpg)
Example 2: SSSP
FOREACH n IN g(type=N) SET n.dist TO MIN(n.in(type=N).dist)+1 AS dist’ WHERE dist’ < n.dist
![Page 27: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/27.jpg)
Example 2: SSSP
WHILE updates > 0 FOREACH n IN g(type=N) updates = SET n.dist TO MIN(n.in(type=N).dist)+1 AS dist’ WHERE dist’ < n.dist
![Page 28: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/28.jpg)
Example 2: SSSP
SET g(type=N).dist TO inf SET g(type=N,id=start).dist TO 0 WHILE updates > 0 FOREACH n IN g(type=N) updates = SET n.dist TO MIN(n.in(type=N).dist)+1 AS dist’ WHERE dist’ < n.dist
![Page 29: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/29.jpg)
GraphiQL Compiler• Graph Table manipulations to relational
operators:- filter � selection predicates - iterate � driver loop - retrieve � projections - update � update in place - aggregate � group-by aggregate
• Graph Tables to relational tables:- mapping
![Page 30: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/30.jpg)
GraphiQL Compilerg(type=N) � N
g(type=E) � E
g(type=N).out(type=E) � N ⋈ E
g(type=E).out(type=E) � E ⋈ E
g(type=N).out(type=N) � N ⋈ E ⋈ N
g.out.in = g.in
g.in.out = g.out
![Page 31: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/31.jpg)
Example: SSSP
SET g(type=N).dist TO inf SET g(type=N,id=start).dist TO 0 WHILE updates > 0 FOREACH n IN g(type=N) updates = SET n.dist TO MIN(n.in(type=N).dist)+1 AS dist’ WHERE dist’ < n.dist
lupdateCount>0 ( n.dist ← σn.dist>dist’ ( !min(n’.dist)+1 ( "n.id ( N ⋈ E ⋈ N’ ) ) ) )
![Page 32: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/32.jpg)
GraphiQL Optimizations
• De-duplicating graph elements
• Selection pushdown
• Cross-product as join
• Pruning redundant joins
![Page 33: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/33.jpg)
Performance
![Page 34: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/34.jpg)
Performance
Machine:
2GHz, 24 threads, 48GB memory, 1.4TB disk
![Page 35: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/35.jpg)
Performance
Machine:
2GHz, 24 threads, 48GB memory, 1.4TB disk
Dataset:
Small: 81k/1.7m directed; 334k/925k undirected Large: 4.8m/68m directed; 4m/34m undirected
![Page 36: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/36.jpg)
Performance - small graphTi
me
(sec
onds
)
0
16
32
48
64
PageRank
Shortest Path
Triangles (global)
Triangles (local)
Strong Overlap
Weak Ties
Apache GiraphGraphiQL 12x Speedup!
![Page 37: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/37.jpg)
Tim
e (s
econ
ds)
0
400
800
1200
1600
PageRank
Shortest Path
Triangles (global)
Triangles (local)
Strong Overlap
Weak Ties
Apache GiraphGraphiQL
Performance - large graph
4.3x Speedup!
![Page 38: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/38.jpg)
Summary• Several real world graph analysis are better off in
relational databases
• We need both the graph as well as relational view of data
• GraphiQL introduces Graph Tables to allows users to think in terms of graphs
• Graph Table supports recursive association, nested manipulations, and SQL compilation
• GraphiQL allows users to easily write a variety of graph analysis
![Page 39: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/39.jpg)
Thanks!
![Page 40: Graph Intuitive Query Language for Relational …jindal-web.appspot.com/slides/IEEEBigData.pdfIEEE BigData 2014 Talking on at Supervisors work work collaborate work sabbatical Alekh](https://reader036.fdocuments.in/reader036/viewer/2022070801/5f0266d37e708231d40418ec/html5/thumbnails/40.jpg)
Other LanguagesImperative languages: e.g. Green Marl XPath: e.g. Cypher, Gremlin Datalog: e.g. Socialite SPARQL: Teradata blog Procedural language: e.g. Vertex-centric