Handling billions of edges in a graph database @GoToLondon

Handling Billions Of Edges in a Graph Database

Michael Hackstein @mchacki

‣ Schema-free Objects (Vertices)

‣ Relations between them (Edges)

‣ Edges have a direction

{ name: "alice", age: 32 }

{ name: "dancing" }

{ name: "bob", age: 35, size: 1,73m }

{ name: "reading" }

{ name: "fishing" }

‣ Schema-free Objects (Vertices)

‣ Relations between them (Edges)

‣ Edges have a direction

‣ Edges can be queried in both directions

‣ Easily query a range of edges (2 to 5)

‣ Undefined number of edges (1 to *)

‣ Shortest Path between two vertices

{ name: "alice", age: 32 }

{ name: "dancing" }

{ name: "bob", age: 35, size: 1,73m }

{ name: "reading" }

{ name: "fishing" }

‣ Give me all friends of Alice

Charly DaveAlice

Eve Frank

‣ Give me all friends of Alice

Charly DaveCharly

DaveAlice

Eve Frank

‣ Give me all friends-of-friends of Alice

Charly

‣ Give me all friends-of-friends of Alice

Charly

FrankEve Frank

‣ What is the linking path between Alice and Eve

Charly

‣ What is the linking path between Alice and Eve

Charly

FrankBobEve

‣ Which Trainstations can I reach if I am allowed to drive a distance of at most 6 stations on my ticket

You are here

‣ Which Trainstations can I reach if I am allowed to drive a distance of at most 6 stations on my ticket

You are here

Friend

‣ Give me all users that share two hobbies with Alice

Hobby1 Hobby2

FriendFriend

‣ Give me all users that share two hobbies with Alice

Hobby1 Hobby2

Product

ProductFriend

‣ Give me all products that at least one of my friends has bought together with the products I already own, ordered by how many friends have bought it and the products rating, but only 20 of them.

has_bought has_b

has_boughtis_friend

Product

ProductFriend Product

‣ Give me all products that at least one of my friends has bought together with the products I already own, ordered by how many friends have bought it and the products rating, but only 20 of them.

has_bought has_b

has_boughtis_friend

‣ Give me all users which have an age attribute between 21 and 35.

‣ Give me all users which have an age attribute between 21 and 35.‣ Give me the age distribution of all users

‣ Give me all users which have an age attribute between 21 and 35.‣ Give me the age distribution of all users‣ Group all users by their name

‣ We first pick a start vertex (S)

‣ We first pick a start vertex (S)‣ We collect all edges on S

‣ We first pick a start vertex (S)‣ We collect all edges on S‣ We apply filters on edges

‣ We first pick a start vertex (S)‣ We collect all edges on S‣ We apply filters on edges‣ We iterate down one of the new vertices (A)

‣ We first pick a start vertex (S)‣ We collect all edges on S‣ We apply filters on edges‣ We iterate down one of the new vertices (A)‣ We apply filters on edges

‣ We first pick a start vertex (S)‣ We collect all edges on S‣ We apply filters on edges‣ We iterate down one of the new vertices (A)‣ We apply filters on edges‣ The next vertex (E) is in desired depth. Return

the path S -> A -> E

the path S -> A -> E‣ Go back to the next unfinished vertex (B)

the path S -> A -> E‣ Go back to the next unfinished vertex (B)‣ We iterate down on (B)

the path S -> A -> E‣ Go back to the next unfinished vertex (B)‣ We iterate down on (B)‣ We apply filters on edges

the path S -> A -> E‣ Go back to the next unfinished vertex (B)‣ We iterate down on (B)‣ We apply filters on edges‣ The next vertex (F) is in desired depth. Return

the path S -> B -> F

‣ Once: ‣ Find the start vertex

‣ For every depth: ‣ Find all connected edges ‣ Filter non-matching edges ‣ Find connected vertices ‣ Filter non-matching vertices

Depends on indexes: Hash:

Edge-Index or Index-Free: Linear in edges: Depends on indexes: Hash: Linear in vertices: Only one pass:

n * 1 n

‣ Linear sounds evil? ‣ NOT linear in All Edges O(E) ‣ Only Linear in relevant Edges n < E

‣ Traversals solely scale with their result size ‣ They are not effected at all by total amount of data ‣ BUT: Every depth increases the exponent: O((3n)d) ‣ "7 degrees of separation": (3n)6 < E < (3n)7

‣ MULTI-MODEL database ‣ Stores Key Value, Documents, and Graphs ‣ All in one core

‣ Query language AQL ‣ Document Queries

‣ Graph Queries

‣ Joins

‣ All can be combined in the same statement

‣ ACID support including Multi Collection Transactions

FOR user IN users RETURN user

FOR user IN users FILTER user.name == "alice" RETURN user

FOR user IN users FILTER user.name == "alice" FOR product IN OUTBOUND user has_bought RETURN product

Alicehas_bought

FOR user IN users FILTER user.name == "alice" FOR recommendation, action, path IN 3 ANY user has_bought FILTER path.vertices[2].age <= user.age + 5 AND path.vertices[2].age >= user.age - 5 FILTER recommendation.price < 25 LIMIT 10 RETURN recommendation

Alicehas_bought

FOR user IN users FILTER user.name == "alice" FOR recommendation, action, path IN 3 ANY user has_bought FILTER path.vertices[2].age <= user.age + 5 AND path.vertices[2].age >= user.age - 5 FILTER recommendation.price < 25 LIMIT 10 RETURN recommendation

Alicehas_bought

TVhas_bought

playstation.price < 25

PlaystationBob

alice.age - 5 <= bob.age && bob.age <= alice.age + 5

has_bought

‣ Many graphs have "Supernodes" ‣ Vertices with many inbound and/or outbound edges

‣ Traversing over them is expensive ‣ Often you only need a subset of edges

Bob Alice

‣ Remember Complexity? O((3n)d) ‣ Filtering of non-matching edges is linear for every depth

‣ Index all edges based on their vertices and arbitrary other attributes ‣ Find initial set of edges in identical time ‣ Less / No post-filtering required ‣ This decreases the n significantly

‣ We have the rise of big data ‣ Store everything you can

‣ Dataset easily grows beyond one machine ‣ This includes graph data!

‣ Distribute graph on several machines (sharding)

‣ How to query it now? ‣ No global view of the graph possible any more ‣ What about edges between servers?

‣ In a shared environment network most of the time is the bottleneck ‣ Reduce network hops

‣ Vertex-Centric Indexes again help with super-nodes ‣ But: Only on a local machine

First let's do the cluster thingy

‣ ArangoDB is the first fully certified database including the persistence primitives for DC/OS

‣ ArangoDB's cluster resource management is based on Apache Mesos

‣ Later this year we will also support Kubernetes

‣ ArangoDB can run clusters without it ‣ Setup Requires manual effort (can be scripted)

‣ This works: ‣ Automatic Failover (Follower takes over if leader dies) ‣ Rebalancing of shards ‣ Everything inside of ArangoDB

‣ This is based on Mesos: ‣ Complete self healing ‣ Automatic restart of ArangoDBs (on new machines)

➡ We suggest you have someone on call

Demo Time DC/OS

Now distribute the graph

‣ Only parts of the graph on every machine ‣ Neighboring vertices may be on different machines ‣ Even edges could be on other machines than their vertices

‣ Queries need to be executed in a distributed way ‣ Result needs to be merged locally

Random Distribution

‣ Advantages: ‣ every server takes an equal portion of

data ‣ easy to realize ‣ no knowledge about data required ‣ always works

‣ Disadvantages: ‣ Neighbors on different machines ‣ Probably edges on other machines than

their vertices ‣ A lot of network overhead is required for

querying

Random Distribution

‣ Advantages: ‣ every server takes an equal portion of

data ‣ easy to realize ‣ no knowledge about data required ‣ always works

‣ Disadvantages: ‣ Neighbors on different machines ‣ Probably edges on other machines than

their vertices ‣ A lot of network overhead is required for

querying

Index-Free Adjacency

‣ Used by most other graph databases ‣ Every vertex maintains two lists of it's edges (IN and OUT) ‣ Do not use an index to find edges ‣ How to shard this?

‣ ArangoDB uses an hash-based EdgeIndex (O(1) - lookup) ‣ The vertex is independent of it's edges ‣ It can be stored on a different machine

Domain Based Distribution

‣ Many Graphs have a natural distribution ‣ By country/region for People ‣ By tags for Blogs ‣ By category for Products

‣ Most edges in same group ‣ Rare edges between groups

ArangoDB Enterprise Edition uses Domain Knowledge

for short-cuts

SmartGraphs - How it works

DB Server 1 DB Server 2 DB Server n

Coordinator

Sneak Preview SmartGraphs

Database document write transactions per virtual CPU/second

ArangoDB 1,730

Aerospike 2,500

Cassandra 965

Couchbase 1,375

FoundationDB 750

*) Source: https://mesosphere.com/blog/2015/11/30/arangodb-benchmark-dcos/

‣ Further questions? ‣ Follow us on twitter: @arangodb ‣ Join our slack: slack.arangodb.com

‣ Follow me on twitter/github: @mchacki ‣ Write me a mail: michael@arangodb.com

Thank You

Handling billions of edges in a graph database @GoToLondon

Technology

Transcript of Handling billions of edges in a graph database @GoToLondon

Graph Based Programming - Chair for Applied Software Engineering · Graph based programming • Main idea: Program execution is message passing through a graph along edges • Inspired

Auto Graph Encoder-Decoder for Model Compression and ...computational graph because a DNN may involve billions of calculations [8], and the resulting computation graph could be huge.

ManojKumar DTU, Delhipeople.dtu.ac.in/faculty/lectures/manojkumar/DSA_MK... · 2016. 2. 8. · DFS on directed Graph Four type of edges are produces 1. Tree edges: are edges (u,v)

Kruskal MST: initialized the graph and sorted the edges ... · Kruskal MST: initialized the graph and sorted the edges. Disjoint sets: [H] [J] [M] [F] [G] [L] [T] [K] Weight of red

Graph Algorithms Searching. Review: Graphs ● A graph G = (V, E) ■ V = set of vertices, E = set of edges ■ Dense graph: |E| |V| 2 ; Sparse graph: |E|

ETERNAL DOMINATION Chip Klostermeyer. Graph 6 vertices 7 edges Dominating Set γ=2.

Journal of Graph Algorithms and Applicationsemis.maths.adelaide.edu.au/journals/JGAA/accepted/... · A graph inCGA is a classical undirected graph, consisting of nodes and edges.

Graph Databases - Roma Tre Universitytorlone/bigdata/graph-databases.pdfWhy Graph Databases? Every time we visit a profile we have to find the common friends ~ 1.2 billions tuples

Ali Shah 2552956 - Slides - Pregel-1resources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-pregel.pdf · Introduction •Analyzing large graph is hard Billions of edges Trillions

Exploring and Making Sense of Large Graphs · algorithms to massive datasets, including a Web graph of 6.6 billion edges, a Twitter graph of 1.8 billion edges, brain graphs with up

Slim Graph: Practical Lossy Graph Compression for Approximate … · 2020-01-11 · ing one trillion edges in 2015 (the Facebook social graph [48]) and 12 trillion edges in 2018 (the

Graph Partitioning Abhiram Ranade IIT Bombay. A load balancing problem Input: – undirected graph G. Vertices = processes. Edges = interprocess communication.

Introducing Apache Giraph for Large Scale Graph Processing Scale... · graph, the so called web graph –pages are vertices connected by edges that represent hyperlinks –the web

Introduction to Networking Project 3. Routing Graph abstraction for routing algorithms: graph nodes are routers graph edges are physical links link.

GRAPH: A network of NODES VERTICES) and ARCS (or EDGES ...

Parallel Algorithms for Trillion Edges Graph Problems

TABLE 1: Graph Terminology TypeEdgesMultiple Edges Allowed? Loops Allowed? Simple graphUndirectedNo MultigraphUndirectedYesNo PseudographUndirectedYes.

2IL90: Graph Drawing Introduction Fall 2006. Graphs Vertices Edges.

Routing Graph abstraction for routing algorithms: graph nodes are routers graph edges are physical links –link cost: delay, $ cost, or congestion level.

GraphChi-DB: Simple Design for a Scalable Graph Database ... · for efﬁciently managing graphs with billions of edges on disk. The PAL structure is based on the graph storage model