vFabric SQLFire

Fast meets scalable in VMware's new distributed SQL database.

Speed Matters.

Users demand fast applications and fast websites.The database is the hardest thing to scale.

• SQLFire is a memory-optimized, distributed SQL database.

• SQLFire satisfies modern application needs by adopting a more scalable design than traditional RDBMS.

• SQLFire focuses on Speed, Scale and SQL.

What Is SQLFire?

Speed• Memory-optimized for

maximum speed and minimum latency.

SQLFire: Speed, Scale, SQL.Scale

• Horizontally scalable.• Add or remove nodes at

any time for more capacity or availability.

SQL• Familiar SQL interface.• SQL 92 compliant.• JDBC and ADO.NET

interfaces.

• Memory Optimization– Latency is tracked to the millisecond.– Use memory to get an advantage.

• SQLFire can use disk too.– Data can be asynchronously persisted to disk.– Append-only logfile format for best update

performance.– Disaster prevention and for larger databases.

SQLFire Speed Through Memory Optimization.

What’s Really Great About Memory?

Disk seeks are 100,000 times slower than memory reads!(10,000,000 ns = 10 ms)

• Major online travel site.• Minimizing page load time a must.– Did all the usual tricks including minimizing server

round-trips, etc.• Required all data access to complete in less than

20ms – Always!• Selected a memory-optimized database from

VMware.

Who Needs That Kind Of Latency Anyway?

The Things People Do For Speed.• Many architectures

for speeding up data have appeared lately.

• All stem from

• Many architectures have appeared to make data faster.– Memcached on top of SQL Databases.– Read-only slave nodes.– Clustering / RAC.– NoSQL.

• Clear need to overcome traditional database limitations.– Relational databases were not built to serve thousands (or

even hundreds) of users at once.– RDBMS were not built for the low latency users expect.

Don’t Mask The Problem, Solve It!

SQLFireHorizontally Scalable

SQL Database

Memcache + SQL:Multi-data model

to overcome the DB.

Application Tier

Browser Tier

Application Tier

Browser Tier

What Is Horizontal Scalability?• Software that makes

multiple computers appear as one system.

• Horizontal scale gives:– Better performance.– More capacity.– Higher availability.

• SQLFire is a horizontally-scalable SQL Database.• SQLFire is horizontally scalable from the ground up.– Data inserts.– Key lookup.– Stored procedures / functions.– Joins.– Transactions.– More.

SQLFire Horizontal Scalability.

• SQLFire syntax is based on the SQL-92 standard.• SQLFire extensions are to Data Definition

Language (DDL), e.g. CREATE TABLE.• SQLFire ships with JDBC and ADO.NET drivers.– Built-in, transparent high availability.

SQLFire and SQL.

COMPARING SQLFIRE AND OTHER DATABASES

• Traditional RDBMS: Too much focus on ACID consistency.• Traditional RDBMS: Too much contention for disk.

SQLFire Challenges Traditional DB Design, Not SQL.

• Many new data models (NoSQL) are emerging– Key-value– Column family (inspired by Google BigTable)– Document– Graph

• Most focus on making model less rigid than SQL• Consistency model is not ACID

• Different tradeoffs for different goals

The database world is changing. A lot!

Low scale

STRICT – Full ACID (RDBMS)

High scale

Tunable Consistency

Very high scale

Eventual

SQLFire At A Glance.

Attribute SQLFire Other SQL DBs

DB Interface Standard SQL. Standard SQL.

Data ConsistencyTunable. Mix of eventual consistency and high consistency.

High consistency.

Transactions Supported. Very strong support.

Scaling Model Scale out, commodity servers. Scale up.

SQLFire Versus Other SQL Databases.

Attribute NoSQL SQLFireDB Interface Idiosyncratic (i.e. each is custom). Standard SQL.

Querying Idiosyncratic or not present. SQL Queries.Data Consistency Tunable, most favor eventual

consistency.Tunable, favors high consistency.

Transactions Weak or not present. Linearly scalable transaction model.

Interface Design Designed for simplicity. Designed for compatibility.Data Model Wide variety of different models. Relational model.

Schema Flexibility Focus on extreme flexibility, dynamism.

SQL model, requires DB migrations, etc.

SQLFire Versus NoSQL.

HANDS ON WITH SQLFIREIllustrating some of SQLFire’s key features.

1

2

3

4

5

6

7

8

910

SQLFire Tables Are Replicated By Default. CREATE TABLE sales

(product_id int, store_id int,

price float);

SQLFire Node 1

SQLFire Node 2

Replica

Replica

sales

Best for small andfrequently accessed

data.

1

2

3

4

5

6

7

8

910

Partitioned Tables Are Split Among Members. CREATE TABLE sales

(product_id int, store_id int,

price float)

PARTITION BY

COLUMN (product_id);

SQLFire Node 1

SQLFire Node 2

Replica

Replica

sales Partition 1

Partition 2Best for largedata sets.

Type Purpose Example

Hash Partitioning (Default)

Built-in hashing algorithm splits data at random across available servers.

PARTITION BY COLUMN (customer_id);

ListManually divide data across servers based on discrete criteria.

PARTITION BY LIST (home_state) (VALUES (‘CA’, ‘WA’), VALUES (‘TX’, ‘OK’));

RangeManually divide data across servers based on continuous criteria.

PARTITION BY RANGE (date) (VALUES BETWEEN ‘2008-01-01’ AND ‘2008-12-31’, VALUES BETWEEN ‘2009-01-01’ AND ‘2009-12-30’);

ExpressionFully dynamic division of data based on function execution. Can use UDFs.

PARTITION BY (MONTH(date));

Types Of Partitioning In SQLFire.

1

2

3

4

5

6

7

8

910

Redundancy Increases Availability. CREATE TABLE sales

(product_id int, store_id int,

price float)

PARTITION BY

COLUMN (product_id);

REDUNDANCY 1;

SQLFire Node 1

Partition 2*

SQLFire Node 2

Partition 1*

Replica

Replica

salesPartition 1

Partition 2All data is availableif Node 1 fails.

1

2

3

4

5

6

7

8

910

Collocate Data For Fast Joins. CREATE TABLE sales

(product_id int, store_id int,

price float)

PARTITION BY

COLUMN (product_id);

COLOCATE WITH customers;

SQLFire Node 1

Customer 1 Sales

SQLFire Node 2

Customer 2 Sales

Replica

Replica

Customer 1

Customer 2SQLFire can jointables withoutnetwork hops.

C1

C2

Related data placedon the same node.

1

2

3

4

5

6

7

8

910

Why Collocation Matters. -- Biggest customers in CA.

SELECT sum(value) AS total

FROM sales, customer

WHERE sales.customerid =

customer.id AND

customer.state = “CA”

ORDER BY total DESC;

SQLFire Node 1

Customer 1 Sales

SQLFire Node 2

Customer 2 Sales

Replica

Replica

Customer 1

Customer 2

Since we collocatedcustomers and salesthis query execute

independentlyon each node.

Result:The query scales

linearly!

1

2

3

4

5

6

7

8

910

Scaling Functions and Stored Procedures. CALL maxSales

WITH RESULT PROCESSOR

maxSalesReducer

ON TABLE sales;

SQLFire uses data-aware routing to

route processing tothe data.

maxSales on local data

maxSales on local data

maxSalesReducer

Result Processorsgive map/reduce

functionality.

• Server Groups:– Granular control of data placement or restrict data to certain servers.

• Disk persistence and overflow.– For additional availability and when using SQLFire as a primary

database.• Caching / Operational Data Store.• Transactions.• Indexes.• Much more!

Some SQLFire Features Not Discussed Include…

Scan Me To Learn MoreAbout These Features!

DEMOSQLFire In Action

SQLFIRE RELEASE DETAILS

• SQLFire Professional 1.0:– Release: December 13, 2011.– Part of the vFabric Advanced suite. (Per VM)– Also available standalone. (Per CPU)– Limit of 2 connected nodes per database.

• SQLFire Enterprise 1.0:– Coming 1H 2012.– Unlimited nodes.– Optional SQLFire Enterprise WAN Upgrade for active/active global

databases.

SQLFire Releases.

• Same database in multiple datacenters or the cloud.• Fully active-active with asynchronous replication.• Coming 1H 2012 with SQLFire Enterprise.

SQLFire WAN.

• Beta available now on our community site.– http://tinyurl.com/SQLFire

• General availability December 2011.– vFabric Suite or standalone.– Free for development up to 3 nodes.

• Follow us on Twitter:– http://twitter.com/vfabricsqlfire

Download SQLFire Today!

Scan To Learn More

QUESTIONS?

DEMO DETAILS

• 2 VMs.– ubuntu– livecd

• Schema:– Table called record

• threadid: int, value int• Each value is validated upon insert.• The validate constraint check is extremely CPU intensive.

• Client: 5 threads inserting simultaneously.• Total of 2500 records to be inserted.

Demo Details

Demo State 1

validate (very slow!)

Insert Complete

Insert

Ubuntu(locator)

Ubuntu(database)

CREATE TABLE record (threadid int, value int CONSTRAINT MY_CK CHECK (validate(value) = 1)) PARTITION BY COLUMN (threadid);

Note: The validate function takes a long time to run.

Create Table Code:

Demo State 2

validate (very slow!)

Insert Complete

Insert

Ubuntu(locator)

Ubuntu(database)

validate (very slow!)

Insert Complete

livecd(database)

Note:No client change!

• Base Directory:– https://github.com/cartershanklin/SQLFireStuff

• Java UDF:– https://

github.com/cartershanklin/SQLFireStuff/blob/master/java/examples/Validate.java

• Jython Test Script– https://

github.com/cartershanklin/SQLFireStuff/blob/master/jython/slowInserts.py

• Contact: @cshanklin

Source Code