Why relationships are cool but "join" sucks

(c) Luca Garulli Licensed under a Creative Commons Attribution-NoDerivs 3.0 Unported License www.orientechnologies.com

Luca Garulli – Founder and CEO@Orient Technologies LtdAuthor of OrientDB

www.twitter.com/lgarulli

Why Relationshipsare cool

but the “JOIN” sucks

(c) Luca Garulli Licensed under a Creative Commons Attribution-NoDerivs 3.0 Unported License

1979First Relational DBMS available as product

2009NoSQL movement


1979First Relational DBMS available as product

2009NoSQL movement

Hey, 30 years in the IT field is so huge!


Before 2009 teams of developersalways fought to select:

Operative SystemProgramming Language

Middleware (App-Servers)

What about the Database?


One of the main resistances ofRDBMS users to pass to a NoSQL product

are related to thecomplexity of the model:

Ok, NoSQL products are super forBigData and BigScale

but...


...what about the model?


What is the NoSQL answer about managing complex domains?

Key-Value stores ?Column-Based ?

Document database ?Graph database !

No R

ela

tion

ship

ssu

pp

ort


Why most of NoSQL

productsdon’t supportRelationship

Between entities?


To understand why,let’s see how

Relational DBMSmanaged them


Domain: the super minimal “Selling App”

CustomerCustomer AddressAddress

OrderOrder StockStock

Registry system

Order system


StockStock

Registry system

Domain: the super minimal “Selling App”

OrderOrder

Order system

CustomerCustomer AddressAddress

How doesRelational DBMS

manage relationships?


Relational World: 1-1 Relationships

JOIN Customer.Address -> Address.Id

Customer

Id Name Address

10 Luca 34

11 Jill 44

34 John 54

56 Mark 66

88 Steve 68

Address

Id Location

34 Rome

44 London

54 Moscow

66 New Mexico

68 Palo Alto

Foreign key

Primary keyPrimary key


Relational World: 1-N Relationships

Inverse JOIN Address.Customer -> Customer.Id

Customer

Id Name

10 Luca

11 Jill

34 John

56 Mark

88 Steve

Address

Id Customer Location

24 10 Rome

33 10 London

44 34 Moscow

66 56 Cologne

68 88 Palo Alto


Relational World: N-M Relationships

Additional table with 2 JOINs(1) CustomerAddress.Id -> Customer.Id and(2) CustomerAddress.Address -> Address.Id

Customer

Id Name

10 Luca

11 Jill

34 John

56 Mark

88 Steve

Address

Id Location

24 Rome

33 London

44 Moscow

66 Cologne

68 Palo Alto

CustomerAddress

Id Address

10 24

10 33

34 44


What’s wrong with theRelational Model?


These are all JOINs executedeverytime you traverse a

relationship

The JOIN is the evil!Customer

Id Name

10 Luca

11 Jill

34 John

56 Mark

88 Steve

Address

Id Location

24 Rome

33 London

44 Moscow

66 Cologne

68 Palo Alto


relationship


relationship


relationship!

CustomerAddress

Id Address

10 24

10 33

34 24


A JOIN means searching for a key inanother table

The first rule to improve performanceis indexing all the keys

Index speeds up searches, but slows downinsert, updates and deletes


So in the best case a JOIN is a lookupinto an index

This is done per single join!

If you traverse hundreds of relationshipsyou’re executing hundreds of JOINs


Index Lookupis it really that fast?


Index Lookup: how does it works?

A-Z

A-L M-Z

Think to an Address Book

where we have to find the Luca’s phone

number



A-Z

A-L M-Z

A-L

A-D E-L

M-Z

M-R S-Z

Index algorithms are all similar and based on

balanced trees



A-Z

A-L M-Z

A-L

A-D E-L

M-Z

M-R S-Z

A-D

A-B C-D

E-L

E-G H-L



A-Z

A-L M-Z

A-L

A-D E-L

M-Z

M-R S-Z

A-D

A-B C-D

E-L

E-G H-L

E-G

E-F G

H-L

H-J K-L



A-Z

A-L M-Z

A-L

A-D E-L

M-Z

M-R S-Z

A-D

A-B C-D

E-L

E-G H-L

E-G

E-F G

H-L

H-J K-L

Luca

Found! This lookup took 5 steps and grows up with the index

size!

Found! This lookup took 5 steps and grows up with the index

size!


Can you imaginehow many steps a

Lookup operation does into anIndex with Millions or Billions

of records?


And this JOIN is executedforeach involved table,

multiplied foreach scanned records

!


Querying more tables can easilyproduce millions of JOINs/Lookups!

Here the rule: more entries= more lookup steps = slower JOIN


Oh! This is whyperformance of my database

drops down whenit becomes bigger,

and bigger,and bigger!


What aboutDocument Databases

like MongoDB?


How MongoDB manages relationships:

{ “_id” : “292846512”, “type” : “Order”, “number” : 1223, “customer” : “123456789”}


MongoDB uses the same approach:

it stores the _id of the connecteddocuments. At run-time it lookups upfor the _id by using an index.


Is there a better way tomanage relationships?


“A graph database is anystorage system

that providesindex-free adjacency”

- Marko Rodriguez (author of TinkerPop Blueprints)


How does GraphDB manageindex-free relationships?


Every developer knowsthe Relational Model,but who knows the

Graph one?


Back to school:Graph Theory crash course


Basic Graph

LucaLuca NoSQLDay

NoSQLDay

Likes


Property Graph Model*

Lucaname: Luca

surname: Garullicompany: Orient Tech

Lucaname: Luca

surname: Garullicompany: Orient Tech

NoSQLDay

date: Nov 15° 2013

NoSQLDay

date: Nov 15° 2013

Likes

since: 2013

Vertices and Edges can have propertiesVertices and Edges can have propertiesVertices and Edges can have properties

Vertices are directed

* https://github.com/tinkerpop/blueprints/wiki/Property-Graph-Model


Property Graph Model

LucaLuca NoSQLDay

NoSQLDay

Likes

since: 2013

Speakstitle: «Switching...»

abstract: «This talk presents...»

An Edge connects 2 vertices: use multiple edges to represents 1-N and N-M

relationships


Property Graph Model

Likes

DanielDaniel

LucaLuca

Organizes

FriendOf

NoSQLDay

NoSQLDay

UdineUdine

located

Studies


Compliments, this is your diploma in«Graph Theory»


The Graph theoryis so simple to be so

powerful


Let’s go backto the Graph Stuff

How does OrientDBmanage relationships?


Luca(vertex)

Luca(vertex)

OrientDB: traverse a relationship

label : ‘Customer’name : ‘Luca’label : ‘Customer’name : ‘Luca’

RID = #13:35RID = #13:35 RID = #13:100RID = #13:100

label = ‘Address’name = ‘Rome’label = ‘Address’name = ‘Rome’

The Record ID (RID)is the physical position

Rome(vertex)

Rome(vertex)

The Record ID (RID)is the physical position


Lives

OrientDB: traverse a relationship

out : [#14:54]label : ‘Customer’name : ‘Luca’


out: [#13:35]in: [#13:100]Label : ‘Lives’


RID = #13:35RID = #13:35 RID = #13:100RID = #13:100

in: [#14:54]label = ‘Address’name = ‘Rome’


The Edge’s RID is saved inside both vertices, as

«out» and «in»

The Edge’s RID is saved inside both vertices, as

«out» and «in»

RID = #14:54RID = #14:54

Luca(vertex)

Luca(vertex)

Rome(vertex)

Rome(vertex)


LucaLucaLives

OrientDB: traverse -> outgoing





RID = #13:35RID = #13:35

RID = #14:54RID = #14:54RID = #13:100RID = #13:100



RomeRome


LucaLucaLives

OrientDB: traverse <- incoming





RID = #13:35RID = #13:35

RID = #14:54RID = #14:54RID = #13:100RID = #13:100



RomeRome


GraphDB handles relationships as aphysical LINK to the record

assigned when the edge is created

on the other side

RDBMS computes therelationship every time you query a database

Is not that crazy?!


This means jumping from aO(log N) algorithm to a near O(1)

traversing cost is not more affectedby database size!

This is huge in the BigData age


an Open Source (Apache licensed)document-graph NoSQL dbms


OrientDB in the Blueprints micro-benchmark,on common hw, with a hot cache,

traverses 29,6 Millionsof records in less than 5 seconds

about 6 Millions of nodes traversed per sec!

*unless you live in the Google’s server farm

Do not try this at home with a RDBMS*!


Create the graph in SQL$luca> cd bin$luca> ./console.shOrientDB console v.1.6.1 (www.orientdb.org) Type 'help' to display all the commands supported.

orientdb> create vertex Customer set name = ‘Luca’Created vertex #13:35 in 0.03 secs

orientdb> create vertex Address set name = ‘Rome’Created vertex #13:100 in 0.02 secs

orientdb> create edge Lives from #13:35 to #13:100Created edge #14:54 in 0.02 secs


Create the graph in Java

Graph graph = new OrientGraph("local:/tmp/db/graph”);

Vertex luca = graph.addVertex( “class:Customer” );luca.setProperty( “name", “Luca” );

Vertex rome = graph.addVertex ( “class:Address” );rome.setProperty( “name", “Rome” );

Edge edge = luca.addEdge( “Lives”, rome );

graph.shutdown();


Query the graph in SQL

orientdb> select in(‘Lives’) from Address where name = ‘Rome’

---+------+---------|--------------------+--------------------+--------+ #| RID |@class |label |out_Lives |in |---+------+---------+--------------------+--------------------+--------+ 0| 13:35|Customer |Luca |[#14:54] | |---+------+---------+--------------------+--------------------+--------+1 item(s) found. Query executed in 0.007 sec(s).

Incoming vertices


More on query power

orientdb> select sum( out(‘Order’).total ) from Customer where name = ‘Luca’

orientdb> traverse both(‘Friend’) from Customer while $depth <= 7

orientdb> select from ( traverse both(‘Friend’) from Customer while $depth <= 7 ) where @class=‘Customer’ and city.name = ‘Udine’


Query vs traversal

Once you’ve a well connected databasein the form of a Super Graph you cancross records instead of query them!

All you need is a few“Root Vertices”where to start traversing


Query vs traversal

CustomersCustomers

LucaLuca Mark

Mark JillJill

Order2332Order2332

Order8834Order8834

WhiteSoapWhiteSoap

StocksStocksSpecialCustomers

SpecialCustomers

This is aroot

vertex


Root Vertices can be enriched byMeta Graphs

to decorate Graphs withadditional informationand make easier/faster

the retrieval


Temporal based Meta Graph

Order2333Order2333

Order2334Order2334

CalendarCalendar

Hour9/4/2013

10:00

Hour9/4/2013

10:00

Hour9/4/2013

09:00

Hour9/4/2013

09:00

Order2332Order2332

Day9/4/2013

Day9/4/2013

MonthApril 2013

MonthApril 2013

Year2013Year2013


Location based Meta Graph

Order2333Order2333

Order2334Order2334

LocationLocation

CityRomeCity

RomeCity

FiumicinoCity

Fiumicino

Order2332Order2332

StateRM

StateRM

RegionLazio

RegionLazio

CountryItaly

CountryItaly


Mix & Merge graphs

Order2333Order2333

Order2334Order2334

LocationLocation

CityRomeCity

RomeCity

FiumicinoCity

Fiumicino

Order2332Order2332

StateRM

StateRM

RegionLazio

RegionLazio

CountryItaly

CountryItaly

CalendarCalendar

Hour9/4/2013

10:00

Hour9/4/2013

10:00

Hour9/4/2013

09:00

Hour9/4/2013

09:00

Day9/4/2013

Day9/4/2013

MonthApril 2013

MonthApril 2013

Year2013Year2013


Order2333Order2333

Order2334Order2334

LocationLocation

CityRomeCity

RomeCity

FiumicinoCity

Fiumicino

Order2332Order2332

StateRM

StateRM

RegionLazio

RegionLazio

CountryItaly

CountryItaly

CalendarCalendar

Hour9/4/2013

10:00

Hour9/4/2013

10:00

Hour9/4/2013

09:00

Hour9/4/2013

09:00

Day9/4/2013

Day9/4/2013

MonthApril 2013

MonthApril 2013

Year2013Year2013

Get all the orderssold in “Fiumicino” cityon 9/4/2013 at 10:00


Start from Calendar, look for Hour 10:00

Order2333Order2333

Order2334Order2334

LocationLocation

CityRomeCity

RomeCity

FiumicinoCity

Fiumicino

Order2332Order2332

StateRM

StateRM

RegionLazio

RegionLazio

CountryItaly

CountryItaly

CalendarCalendar

Hour9/4/2013

10:00

Hour9/4/2013

10:00

Hour9/4/2013

09:00

Hour9/4/2013

09:00

Day9/4/2013

Day9/4/2013

MonthApril 2013

MonthApril 2013

Year2013Year2013



Order2333Order2333

Order2334Order2334

LocationLocation

CityRomeCity

RomeCity

FiumicinoCity

Fiumicino

Order2332Order2332

StateRM

StateRM

RegionLazio

RegionLazio

CountryItaly

CountryItaly

CalendarCalendar

Hour9/4/2013

10:00

Hour9/4/2013

10:00

Hour9/4/2013

09:00

Hour9/4/2013

09:00

Day9/4/2013

Day9/4/2013

MonthApril 2013

MonthApril 2013

Year2013Year2013

Found 2 Orders, filter by incoming

edges<

Found 2 Orders, now filter by

incoming edges


Order2333Order2333

LocationLocation

CityFiumicino

CityFiumicino

Order2332Order2332

StateRM

StateRM

RegionLazio

RegionLazio

CountryItaly

CountryItaly

CalendarCalendar

Hour9/4/2013

10:00

Hour9/4/2013

10:00

Hour9/4/2013

09:00

Hour9/4/2013

09:00

Day9/4/2013

Day9/4/2013

MonthApril 2013

MonthApril 2013

Year2013Year2013

Order2334Order2334

Only “Order 2333” has incoming

connections with “Fiumicino”

CityRomeCity

Rome



Order2333Order2333

LocationLocation

CityFiumicino

CityFiumicino

Order2332Order2332

StateRM

StateRM

RegionLazio

RegionLazio

CountryItaly

CountryItaly

CalendarCalendar

Hour9/4/2013

10:00

Hour9/4/2013

10:00

Hour9/4/2013

09:00

Hour9/4/2013

09:00

Day9/4/2013

Day9/4/2013

MonthApril 2013

MonthApril 2013

Year2013Year2013

Order2334Order2334

CityRomeCity

Rome

Or start from Location, look for Fiumicino


Order2333Order2333

Order2332Order2332

CalendarCalendar

Hour9/4/2013

10:00

Hour9/4/2013

10:00

Hour9/4/2013

09:00

Hour9/4/2013

09:00

Day9/4/2013

Day9/4/2013

MonthApril 2013

MonthApril 2013

Year2013Year2013

Order2334Order2334

Start from Location, look for Fiumicino

LocationLocation

CityRomeCity

RomeCity

FiumicinoCity

Fiumicino

StateRM

StateRM

RegionLazio

RegionLazio

CountryItaly

CountryItaly


This is your database


Get last customer bought ‘Barolo’select last(out(‘Order’).in(‘Customer)) from Stock where name = ‘Barolo’

#34:22


Get his’s country

select out(‘City’) from #34:22Udine, Italy

#55:12


Get orders from that country

select in(‘Customer’) from #55:12


Let’s move like aSpider

on the web


www.orientechnologies.com

Thanks!

www.twitter.com/orientechno

Why relationships are cool but "join" sucks

Technology

Transcript of Why relationships are cool but "join" sucks