PostgreSQL and CockroachDB SQL

@cockroachdb

PostgreSQL meetup, November 2015CockroachDB

presented by Peter Mattis / Co-Founder

Jessica Edwards

can you give a real world example here?

Jessica Edwards

while on this slide, address how cockroach approaches online schema changes. This will let the audience pause before hopping into the code snippet.

@cockroachdb

1.Overview of CockroachDB2.SQL Data Model3.Logical Data Storage4.Online/Concurrent Schema Change

Agenda

@cockroachdb

What is CockroachDB?

■Scale out SQL■Distributed■Survivable■Consistent■Open source

@cockroachdb

CockroachDB: Architecture

■Layered abstractions■SQL is starting point■Distributes at map■Replicates at physical layer

SQLTransactional KVMonolithic Map

Raft

@cockroachdb



Transactional KVMonolithic Map

Raft

GraphSQL

@cockroachdb




Raft

SQL Graph Doc

@cockroachdb




Raft

SQL

@cockroachdb




Raft

SQL

Physical

@cockroachdb

SQL Data Model

@cockroachdb

■Tables

SQL Data Model

@cockroachdb

■Tables

SQL Data Model

Inventory

@cockroachdb

■Tables■Rows

SQL Data Model

Inventory

@cockroachdb

■Tables■Rows■Columns

SQL Data Model

InventoryID Name Price

1 Glove 1.11

2 Ball 2.22

3 Shirt 3.33

4 Shorts 4.44

5 Bat 5.55

6 Shoes 6.66

@cockroachdb

■Tables■Rows■Columns■Indexes

SQL Data Model

InventoryID Name Price

1 Glove 1.11

2 Ball 2.22

3 Shirt 3.33

4 Shorts 4.44

5 Bat 5.55

6 Shoes 6.66

Name

Ball

Bat

Glove

Shirt

Shoes

Shorts

Name_Idx

@cockroachdb

PostgreSQL: Logical Data Storage

Jessica Edwards

needs image for postgres data storage

@cockroachdb

■Rows are stored in an unordered heap■Indexes are btrees■Primary key is a unique index

PostgreSQL: Data Storage

@cockroachdb

CREATE TABLE test (

id INTEGER PRIMARY KEY,

name VARCHAR,

price FLOAT,

);

PostgreSQL: Example Table

@cockroachdb

INSERT INTO test VALUES (1, “ball”, 3.33);


@cockroachdb



Tuple ID (Page# / Item#) Row

(0, 1) (1, “ball”, 3.33)

test (heap)

@cockroachdb




(0, 1) (1, “ball”, 3.33)

Index Key Tuple ID

1 (0, 1)

test (heap)test_pkey (btree)

@cockroachdb

INSERT INTO test VALUES (1, “ball”, 3.33);INSERT INTO test VALUES (2, “glove”, 4.44);



(0, 1) (1, “ball”, 3.33)

(0, 2) (2, “glove”, 4.44)

Index Key Tuple ID

1 (0, 1)

2 (0, 2)

test (heap)test_pkey (btree)

@cockroachdb

CockroachDB: Logical Data Storage

@cockroachdb

■Keys and values are strings■Monolithic, sorted map

CockroachDB: KV

@cockroachdb

Get(key)

Put(key, value)

ConditionalPut(key, value, expValue)

Scan(startKey, endKey)

CockroachDB: KV Primitives

@cockroachdb

Get(key)

Put(key, value)

ConditionalPut(key, value, expValue)

Scan(startKey, endKey)

Del(key)

CockroachDB: KV Primitives

@cockroachdb

■All tables have a primary key■One key/value pair per column

CockroachDB: Row Storage

@cockroachdb

■All tables have a primary key■One key/value pair per column■Key anatomy:/<table>/<index>/<pkey>/<column>

CockroachDB: Row Storage

@cockroachdb

CREATE TABLE test (

id INTEGER PRIMARY KEY,

name VARCHAR,

price FLOAT,

);

CockroachDB: Example Table

@cockroachdb


CockroachDB: Key Anatomy

Key: /<table>/<index>/<key>/<column>

Value

/test/primary/1/name “ball”

/test/primary/1/price 2.22

@cockroachdb




Value



/test/primary/2/name “glove”


@cockroachdb




Value


.../price 2.22

.../2/name “glove”

.../price 3.33

@cockroachdb




Value

/1000/1/1/1 “ball”

.../2 2.22

.../2/1 “glove”

.../2 3.33

@cockroachdb

■Key encoding■NULL column values■Unique indexes■Non-unique indexes

CockroachDB: The Details

@cockroachdb

■Keys and values are strings■Columns are typed data■???

CockroachDB: Key Encoding

@cockroachdb

■NULL indicates value does not exist■NULL is weird: NULL != NULL

CockroachDB: NULL Column Values

@cockroachdb

■NULL indicates value does not exist■NULL is weird: NULL != NULL■CockroachDB: NULL values are not explicitly

stored


@cockroachdb

INSERT INTO test VALUES (1, “ball”, NULL);



Value


@cockroachdb


INSERT INTO test VALUES (2, NULL, NULL);



Value


??? ???

@cockroachdb




Key: /<table>/<index>/<key>[/<column>]

Value

/test/primary/1 Ø


/test/primary/2 Ø

@cockroachdb

CREATE UNIQUE INDEX bar ON test (name);

■Multiple table rows with equal indexed values are not allowed

CockroachDB: Unique Indexes

@cockroachdb




Key: /<table>/<index>/<key> Value

/test/bar/”ball” 1

@cockroachdb






/test/bar/”glove” 2

@cockroachdb


INSERT INTO test VALUES (1, “ball”, 2.22);INSERT INTO test VALUES (2, “glove”, 3.33);INSERT INTO test VALUES (3, “glove”, 4.44);






@cockroachdb


■NULL is weird: NULL != NULL

CockroachDB: Unique Indexes (NULL Values)

@cockroachdb





/test/bar/NULL 3

@cockroachdb


INSERT INTO test VALUES (3, NULL, NULL);INSERT INTO test VALUES (4, NULL, NULL);



/test/bar/NULL 3

/test/bar/NULL 4

@cockroachdb




Key: /<table>/<index>/<key>[/<pkey>]

Value

/test/bar/NULL/3 Ø

@cockroachdb


INSERT INTO test VALUES (3, NULL, NULL);INSERT INTO test VALUES (4, NULL, NULL);


Key: /<table>/<index>/<key>[/<pkey>]

Value

/test/bar/NULL/3 Ø

/test/bar/NULL/4 Ø

@cockroachdb

CREATE INDEX foo ON test (name);

■Multiple table rows with equal indexed values are allowed

CockroachDB: Non-Unique Indexes

@cockroachdb


■Multiple table rows with equal indexed values are allowed

■Primary key is a unique index


@cockroachdb




Key: /<table>/<index>/<key>/<pkey>

Value

/test/foo/”ball”/1 Ø

@cockroachdb





Value


/test/foo/”glove”/2 Ø

@cockroachdb


INSERT INTO test VALUES (1, “ball”, 2.22);INSERT INTO test VALUES (2, “glove”, 3.33);INSERT INTO test VALUES (3, “glove”, 4.44);



Value




@cockroachdb

■Keys and values are strings■NULL column values■Unique indexes■Non-unique indexes

CockroachDB: Logical Data Storage

@cockroachdb

Logical Data Storage

PostgreSQL CockroachDBKeys are composite structures Keys are strings

Heap storage for rows Required primary key

Per-table heap/indexes Monolithic map

@cockroachdb

Online Schema Change

@cockroachdb

Schema Change Operations

CREATE INDEX foo ON test (col1, col2, …);

ALTER TABLE test DROP col1;

ALTER TABLE test ADD col3 INTEGER;

...

@cockroachdb

Schema Change (the easy way)

1. Lock table2. Adjust table data (add column, populate index,

etc.)3. Unlock table

@cockroachdb

Schema Change (the easy way)

1. Apologize for down time2. Lock table3. Adjust table data (add column, populate index,

etc.)4. Unlock table

@cockroachdb

Schema Change (the MySQL way)

1. Create new table with altered schema2. Capture changes from source to the new table3. Copy rows from the source to the new table4. Synchronize source and new table5. Swap/rename source and new table

@cockroachdb

Schema Change (the PostgreSQL way)

1. CREATE INDEX CONCURRENTLY

@cockroachdb

CockroachDB: Schema Change

■TableDescriptor contains table schema■TableDescriptor replicated on every node■Distributed atomic updates are difficult■Distributed locking is difficult■The easy way isn’t feasible

@cockroachdb

CockroachDB: CREATE INDEX

CREATE INDEX foo ON TEST

1. Backfill index entries2. Add index to TableDescriptor

@cockroachdb


T1 T2

CREATE INDEX foo ON test… INSERT INTO test…

@cockroachdb



1. Add index to TableDescriptor as write-only2. Backfill index entries3. Mark index as read-write

@cockroachdb


T1 T2

CREATE INDEX foo ON test… INSERT INTO test… orUPDATE test… orDELETE FROM test…

@cockroachdb



1. Add index to TableDescriptor as delete-only2. Wait for descriptor propagation3. Mark index as write-only4. Wait for descriptor propagation5. Backfill index entries6. Mark index as read-write

@cockroachdb

Online Schema Change

Online schema change is difficultThe database should do the heavy lifting

@cockroachdb

The End

SQL databases are KV stores on steroids

@cockroachdb

github.com/cockroachdb/cockroach

CockroachLabs.com

@cockroachdb

Thank You

https://github.com/cockroachdb/cockroach

http://cockroachlabs.com/

http://twitter.com/cockroachdb

PostgreSQL and CockroachDB SQL

Technology

Transcript of PostgreSQL and CockroachDB SQL