© 2009 VMware Inc. All rights reserved

Managing High Performance Data with vFabric SQLFire

Carter ShanklinProduct Manager for vFabric

Agenda

What is SQLFire?

Why SQL vs. NoSQL

Why SQLFire versus other SQL databases

SQLFire features + Demo

How SQLFire Scales

• Hash partitioning

• Entity groups and collocation

• Data-aware stored procedures

Consistency model

Shared nothing persistence

What is vFabric SQLFire?

SQLFire is a memory-optimized distributed SQL database.

SQLFire attacks scalability challenges in two ways:

• Relaxes ACID semantics somewhat (in transactions and in replication)

• Horizontally scalable. Add capacity by adding nodes.

SQLFire has built-in high availability and native support for replication to multiple datacenters.

SQLFire provides a real SQL interface. Ships with JDBC and ADO.NET bindings with more to come.

SQLFire can also be used as a cache in front of other databases.

SQLFire at-a-glance

4

As data changes, subscribers are pushed

notification eventsData transparently replicated and/or partitioned;Redundant storage can be in memory and/or on

disk

Many physical machine nodes appear as one logical system

Other

Synchronous read through, write through or

Asynchronous write-behind to other data sources and sinks

JavaClient

Increase/Decrease capacity on the fly

C#Client

JDBC

JDBC or ADO.NET

Databases

File system

Shared Nothing disk persistence

Each cache instance can optionally persist to disk

The database world is changing.

Many new data models (NoSQL) are emerging

• Key-value

• Column family (inspired by Google BigTable)

• Document

• Graph

Most focus on making model less rigid than SQL

Consistency model is not ACID

Different tradeoffs for different goals

Low scale High scale Very high scale

STRICT – Full ACID (RDB)

Tunable Consistency

Eventual

SQLFire Versus NoSQL

Attribute NoSQL SQLFire

DB Interface Idiosyncratic (i.e. each is custom).

Standard SQL.

Querying Idiosyncratic or not present. SQL Queries.

Data Consistency

Tunable, most favor eventual consistency.

Tunable, favors high consistency.

Transactions Weak or not present. Linearly scalable transaction model.

Interface Design Designed for simplicity. Designed for compatibility.

Data Model Wide variety of different models.

Relational model.

Schema Flexibility

Focus on extreme flexibility, dynamism.

SQL model, requires DB migrations, etc.

SQLFire Versus Other SQL Databases

Attribute SQLFire Other SQL DBs

DB Interface Standard SQL. Standard SQL.

Data Consistency

Tunable. Mix of eventual consistency and high consistency.

High consistency.

Transactions Supported. Very strong support.

Scaling Model Scale out, commodity servers.

Scale up.

SQLFire challenges traditional DB design, not SQL

Too much I/O

Design roots don’t necessarily apply today

• Too much focus on ACID

• Disk synchronization bottlenecks8

Confidential

First write to LOG

Second write to Data files

Buffers primarily tuned

for IO

SQLFire 1.0 Notable Features

Horizontally scalable with Partitioning and Replication

Multiple Topologies

• Client/Server, Asynchronous replication over WAN

Queries

• Distributed and memory-optimized

Procedures and Functions

• Standard Java stored procedures with “data awareness”

Caching

• Loader, writers, Eviction, Overflow and Expiration

Event framework

• Listeners, triggers, Asynchronous write behind

Command line tools

Manageability, Security

Scaling SQLFire

Partitioning & Replication

How SQLFire scales a common DB schema.

FLIGHTS---------------------------------------------

FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , ORIG_AIRPORT CHAR(3), DEPART_TIME TIME,…..

PRIMARY KEY (FLIGHT_ID, SEGMENT_NUMBER)

FLIGHTAVAILABILITY---------------------------------------------

FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , FLIGHT_DATE DATE NOT NULL , ECONOMY_SEATS_TAKEN INTEGER ,…..

PRIMARY KEY ( FLIGHT_ID, SEGMENT_NUMBER, FLIGHT_DATE))

FOREIGN KEY (FLIGHT_ID, SEGMENT_NUMBER) REFERENCES FLIGHTS ( FLIGHT_ID, SEGMENT_NUMBER)

FLIGHTHISTORY---------------------------------------------

FLIGHT_ID CHAR(6), SEGMENT_NUMBER INTEGER, ORIG_AIRPORT CHAR(3), DEPART_TIME TIME, DEST_AIRPORT CHAR(3),…..

1 – M

1 – 1

SEVERAL CODE/DIMENSION TABLES---------------------------------------------

AIRLINES: AIRLINE INFORMATION (VERY STATIC)COUNTRIES : LIST OF COUNTRIES SERVED BY FLIGHTSCITIES: MAPS: PHOTOS OF REGIONS SERVED

Assume, thousands of flight rows, millions of flightavailability records

Table

CREATE TABLE AIRLINES ( AIRLINE CHAR(2) NOT NULL PRIMARY KEY, AIRLINE_FULL VARCHAR(24), BASIC_RATE DOUBLE PRECISION, DISTANCE_DISCOUNT DOUBLE PRECISION,…. );

SQLFSQLF SQLF

Creating Tables

CREATE TABLE AIRLINES ( AIRLINE CHAR(2) NOT NULL PRIMARY KEY, AIRLINE_FULL VARCHAR(24), BASIC_RATE DOUBLE PRECISION, DISTANCE_DISCOUNT DOUBLE PRECISION,…. ) REPLICATE;

Replicated TableReplicated Table Replicated Table

SQLFSQLF SQLF

Replicated Tables

CREATE TABLE FLIGHTS ( FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , ORIG_AIRPORT CHAR(3), DEST_AIRPORT CHAR(3) DEPART_TIME TIME, FLIGHT_MILES INTEGER NOT NULL) PARTITION BY COLUMN(FLIGHT_ID);

Table

Partitioned TablePartitioned TablePartitioned Table

Replicated TableReplicated Table Replicated Table

SQLFSQLF SQLF

Partitioned Tables

CREATE TABLE FLIGHTS ( FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , ORIG_AIRPORT CHAR(3), DEST_AIRPORT CHAR(3) DEPART_TIME TIME, FLIGHT_MILES INTEGER NOT NULL) PARTITION BY COLUMN (FLIGHT_ID) REDUNDANCY 1;

Table

Partitioned Table

Redundant Partition

Partitioned Table

Redundant Partition

Partitioned Table

Redundant Partition

Replicated TableReplicated Table Replicated Table

SQLFSQLF SQLF

Partition Redundancy

CREATE TABLE FLIGHTAVAILABILITY ( FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , FLIGHT_DATE DATE NOT NULL , ECONOMY_SEATS_TAKEN INTEGER DEFAULT 0, …) PARTITION BY COLUMN (FLIGHT_ID) COLOCATE WITH (FLIGHTS);

Table

Partitioned Table

Redundant Partition

Partitioned Table

Redundant Partition

Partitioned Table

Redundant Partition

Replicated TableReplicated Table Replicated Table

SQLFSQLF SQLF

Partition Colocation

Colocated PartitionColocated Partition Colocated Partition

CREATE TABLE FLIGHTAVAILABILITY ( FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , FLIGHT_DATE DATE NOT NULL , ECONOMY_SEATS_TAKEN INTEGER DEFAULT 0, …) PARTITION BY COLUMN (FLIGHT_ID) COLOCATE WITH (FLIGHTS) PERSISTENT persistentStore ASYNCHRONOUS;

Table

Partitioned Table

Redundant Partition

Partitioned Table

Redundant Partition

Partitioned Table

Redundant Partition

Replicated TableReplicated Table Replicated Table

SQLFSQLF SQLF

Persistent Tables

Colocated PartitionColocated Partition Colocated Partition

sqlf backup /export/fileServerDirectory/sqlfireBackupLocation

Data dictionary is always persisted in each server

Demo Scaling with partitioned tables.

FLIGHTS---------------------------------------------

FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , ORIG_AIRPORT CHAR(3), DEPART_TIME TIME,…..

PRIMARY KEY (FLIGHT_ID, SEGMENT_NUMBER)

FLIGHTAVAILABILITY---------------------------------------------

FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , FLIGHT_DATE DATE NOT NULL , ECONOMY_SEATS_TAKEN INTEGER ,…..

PRIMARY KEY ( FLIGHT_ID, SEGMENT_NUMBER, FLIGHT_DATE))

FOREIGN KEY (FLIGHT_ID, SEGMENT_NUMBER) REFERENCES FLIGHTS ( FLIGHT_ID, SEGMENT_NUMBER)

FLIGHTHISTORY---------------------------------------------

FLIGHT_ID CHAR(6), SEGMENT_NUMBER INTEGER, ORIG_AIRPORT CHAR(3), DEPART_TIME TIME, DEST_AIRPORT CHAR(3),…..

1 – M

1 – 1

SEVERAL CODE/DIMENSION TABLES---------------------------------------------

AIRLINES: AIRLINE INFORMATION (VERY STATIC)COUNTRIES : LIST OF COUNTRIES SERVED BY FLIGHTSCITIES: MAPS: PHOTOS OF REGIONS SERVED

Hash partitioning for linear scaling

Key Hashing provides single hop access to its partitionBut, what if the access is not based on the key … say, joins are involved

Pure hash-based partitioning will only get you so far.

Consider this query :

select * from flights, flightAvailability

where flights.id = flightAvailability.flightid

and flight.fromAirport = ‘CPH’;

If both tables are simply hash partitioned the join logic will need execution on all nodes where flightavailability data is stored.

This will not scale.

• joins across distributed nodes could involve distributed locks and potentially a lot of intermediate data transfer across nodes

To scale we need partition-aware DB design.

DB architect must think about how data maps to partitions.

The main idea is to:

• minimize excessive data distribution by keeping the most frequently accessed and joined data collocated on partitions.

Read Pat Helland’s “Life beyond Distributed Transactions” and the Google MegaStore paper.

SQLFire allows partition-aware design with the “colocated” keyword.

Entity Groups

Table FlightAvailability partitioned by FlightID colocated with Flights

FlightID is the entity group Key

Solving this scalability problem with SQLFire.

Create flightAvailability as follows:

CREATE TABLE flightAvailability

…

partitioned by flightid colocate with flights;

Re-run the query:

select * from flights, flightAvailability

where flights.id = flightAvailability.flightid

and flight.fromAirport = ‘CPH’;

The query is restricted to nodes containing flights with CPH as the fromAirport.

More about partition-aware database design.

OLTP systems tend to be partitionable.

• Typically it is the number of entities that grows over time and not the size of the entity. Customer count perpetually grows, not the size of the customer info

• Most often access is very restricted and based on select entities given a FlightID, fetch flightAvailability records

given a customerID, add/remove orders, shipment records

Identify partition key for “Entity Group”

• "entity groups": set of entities across several related tables that can all share a single identifier flightID is shared between the parent and child tables

CustomerID shared between customer, order and shipment tables

Scaling Application logic with Parallel “Data Aware procedures”

Stored Procedures in SQLFire.

SQLFire stored procedures.

• Written in pure Java rather than proprietary extensions.

• Created and defined based on SQL standards.

• Supports “data awareness” and run only on nodes where applicable data resides.

• They support a map/reduce-like execution style.

Benefits:

• Write procedures in pure Java or take advantage of existing Java libraries.

• Easily take advantage of SQLFire as a highly scalable distributed system.

Procedures

Java Stored Procedures may be created according to the SQL Standard

SQLFire also supports the JDBC type Types.JAVA_OBJECT. A parameter of type JAVA_OBJECT supports an arbitrary Serializable Java object.

In this case, the procedure will be executed on the server to which a client is connected (or locally for Peer Clients)

CREATE PROCEDURE getOverBookedFlights

(IN argument OBJECT, OUT result OBJECT)

LANGUAGE JAVA PARAMETER STYLE JAVA

READS SQL DATA DYNAMIC RESULT SETS 1

EXTERNAL NAME com.acme.OverBookedFLights;

Data Aware Procedures

Parallelize procedure and prune to nodes with required data

CALL [PROCEDURE]

procedure_name

( [ expression [, expression ]* ] )

[ WITH RESULT PROCESSOR processor_name ]

[ { ON TABLE table_name [ WHERE whereClause ] } |

{ ON {ALL | SERVER GROUPS (server_group_name [, server_group_name ]*) }}

]

Extend the procedure call with the following syntax:

Fabric Server 2Fabric Server 1

Client

Hint the data the procedure depends on

CALL getOverBookedFlights( <bind arguments>

ON TABLE FLIGHTAVAILABILITY

WHERE FLIGHTID = <SomeFLIGHTID> ;

If table is partitioned by columns in the where clause the procedure execution is pruned to nodes with the data (node with <someFLIGHTID> in this case)

Parallelize procedure then aggregate (reduce)

CALL [PROCEDURE]

procedure_name

( [ expression [, expression ]* ] )

[ WITH RESULT PROCESSOR processor_name ]

[ { ON TABLE table_name [ WHERE whereClause ] } |

{ ON {ALL | SERVER GROUPS (server_group_name [, server_group_name ]*) }}

]

Fabric Server 2Fabric Server 1

Client

Fabric Server 3

CALL SQLF.CreateResultProcessor( processor_name, processor_class_name);

register a Java Result Processor (optional in some cases):

Consistency model

Consistency Model without Transactions

Replication within cluster is always eager and synchronous

Row updates are always atomic; No need to use transactions

FIFO consistency: writes performed by a single thread are seen by all other processes in the order in which they were issued

Consistency in Partitioned tables

• a partitioned table row owned by one member at a point in time

• all updates are serialized to replicas through owner

• "Total ordering" at a row level: atomic and isolated

Membership changes and consistency – need another hour

Pessimistic concurrency support using ‘Select for update’

Support for referential integrity

SQLFire Transactions

Highly scalable without any centralized coordinator or lock manager

We make some important assumptions

• Most OLTP transactions are small in duration and size

• Write-write conflicts are very rare in practice

How does it work?

• Each data node has a sub-coordinator to track TX state

• Eagerly acquire local “write” locks on each replica Object owned by a single primary at a point in time

• Fail fast if lock cannot be obtained

Atomic and works with the cluster Failure detection system

Isolated until commit

• Only support local isolation during commit

Scaling disk access with shared nothing disk files and a

“journaling” store design

Disk persistence in SQLF

Parallel log structured storage

Each partition writes in parallel

Backups write to disk also

• Increase reliability against h/w loss

MemoryTables

Append only Operation logs

OS Buffers

LOG Compressor

Record1

Record2

Record3

Record1

Record2

Record3

MemoryTables

Append only Operation logs

OS Buffers

LOG Compressor

Record1

Record2

Record3

Record1

Record2

Record3

• Don’t seek to disk• Don’t flush all the way to disk

– Use OS scheduler to time write

• Do this on primary + secondary• Realize very high throughput

Performance benchmark

How does it perform? Scale?

Scale from 2 to 10 servers (one per host)

Scale from 200 to 1200 simulated clients (10 hosts)

Single partitioned table: int PK, 40 fields (20 ints, 20 strings)

How does it perform? Scale?

CPU% remained low per server – about 30% indicating many more clients could be handled

Is latency low with scale?

Latency decreases with server capacity

50-70% take < 1 millisecond

About 90% take less than 2 milliseconds

Q & A

SQLFire beta available now

http://vmware.com/go/sqlfire