Big Data trends and the rising importance of NOSQL

Abhijit Sharma, Architect,

Innovation & Incubation Lab, BMC Software

Trends in cloud, web, and even enterprise scale apps Unprecedented growth in -

Data set sizes which need to be stored, analyzed Big Data - Cloud scale services generate TB’s > PB’s – FB, eBay, Digg,

Foursquare Connectedness and democratization of data

social networks, feeds, blogs, wiki, tags, semantic web Data API’s - mash up data - use Twitter, FB, Flickr API’s

Semi structured or unstructured data

Performance requirements of these apps Humongous R/W Scalability High Availability Trading consistency for availability – ACID not mandatory

RDBMS woes Challenge - Storing and scaling humongous amounts of data

and remaining highly available Vertical scaling mostly - upper limit & expensive Horizontal scaling – no automatic sharding, no rebalancing – no

infrastructure Distributed transactions & joins due to normalization inhibit

performance, availability Schema less data models – rigid schema – alter table, null

columns Deeply connected data – not designed for this

NOSQL is NOT

No SQL

The NOSQL Alternative

NOSQL is simply

Not only SQL

The NOSQL Alternative

NOSQL – So what else is it? “One size fits all” RDBMS is not working NOSQL alternatives are polyglot solutions that

better fit the new requirements thrown up by the trends. They can be categorized along these axes -

Data Model - simple to complex Scalability – single to horizontal Persistence

NOSQL categories Graph Databases

Based on Graph theory Data model – graph, nodes, edges, properties Scalability – single node – high performance Persistence – On disk data structures Examples – Neo4J, AllegroGraph

Document Databases Based loosely on documents/Lotus Notes Data model – collections of documents Scalability – horizontal, auto-sharding & replication Persistence – B-Tree Examples – mongoDB, CouchDB

NOSQL categories Column Stores

Based on Google’s BigTable design Data model - big table, column families Scalability – horizontal, auto-sharding & replication Persistence – Memory + File (on DFS) Examples – HBase, Cassandra

Key Value Stores Based on DHT, Amazon’s Dynamo design Data model – collection of key value pairs Scalability – horizontal, auto-sharding & replication Persistence – Memory or File Examples – Redis, Amazon Dynamo, Voldemort

Graph Databases

Graph oriented data Graphs are ubiquitous – Social networks,

wikis, the web, recommendation engines et. al.

Deep trees, complex networks Graph traversal - apt for expressing graph

related problems (shortest path, network size etc.)

LinkedIn Social Graph

Why not RDBMS for large scale graphs? Difficult to model and traverse graphs in

RDBMS recursive approaches - slow SQL queries that span

many table joins Hacks like storing paths for trees

node name

1 abhijit

2 sameer

from to

1 2

2 3

Graph Databases Designed for efficient storage & traversal of

large scale graphs Natural modeling of graph network - nodes,

relationships and their properties Neo4J is a leading graph db

Supports billions of nodes/edges, traverses depths of 1000 levels in ms, 1000x of RDBMS

Handle large graphs that don't fit in memory - persistent transactional store optimized for graphs

REST API and various language bindings Graph pattern matching, Cypher Query language,

Indexer – Lucene

Graph basics

All Paths & My Network size

Shortest path between …

Is connected to?

You may know…

Mining your network Centrality Algorithms

Closeness – who has the most followers on twitter Betweenness – who has more influential people following them Eigenvector – PageRank

Document Databases

Flexible document oriented data Document style unstructured data - schema

less – e.g. JSON documents No alter table needed like in an RDBMS, de-normalized data Useful for iterative/agile development

Humongous scale - billions of documents, R/W traffic – millions/sec, horizontal scalability, availability

mongoDB is a leading document database

Document Database – Use cases Use cases :

Archiving of historic data which has undergone many schema changes

Flexible set of performance metrics – web site page views, unique visitors etc. - change over time – no need to update existing JSON documents

Track near real time metrics - optimized increment of perf counters

Geo Loc based mobile and gaming apps (Geospatial indices can be key here)

Craigslist Archival Database Premium service to

customers allowed search over their historical postings

Archival (no purging) of 10 years of postings - billions of documents Schema changes across

versions

MySQL based archival database ALTER TABLE took a month to

complete

Foursquare Find a venue whose

name is Starbucks and mayor is Abhijit

Geo : Optimized for geo location queries – Find Starbucks near my current GPS location

mongoDB Architecture

ShardShard

ShardMongo RouterMongo

Router

Mongo Configuration Server

Mongo Configuration Server

Client

mongoDB Features JSON documents, collection oriented storage Rich, document-based queries Indexes on document attributes Fast in-place updates Scalability features

Horizontal scalability Configurable replication and high-availability Auto-sharding & rebalancing

Language specific drivers – Java, Scala, Ruby etc.

Column Stores

Column Store Reasonably rich data model –

sparse, distributed, persistent multi-dimensional sorted map Sorted row keys, columns

Use cases - Large scale data storage and analysis like - Time series data along with associated dimension data

Row keys are timestamps and thus sorted – helps time range queries Google analytics

Provides aggregate statistics, # unique visitors/day, page views/URL/day Raw click table has a row for each URL + user session time ~200 TB –

ensures contiguous URLs chronologically sorted

Data Cube - CPU

Time

OS

DC

Column Store Performance

Excellent R/W performance – large storage – PB’s High scalability - horizontal scaling, auto-sharding High Availability - transparent replication of data

HBase is a leading column store on – built on Hadoop HDFS as the underlying persistence

Column Store - HBase Table defines Column Families - groups similar attributes , vertical

partitioning (Table, Row, ColumnFamily: Column, Timestamp) tuple maps to a cell -

value  Table is split into multiple equal distributed regions each of which is a range

of sorted keys (partitioned automatically by the key) Ordered Rows by key, Ordered columns in a Column Family Rows can have different number of columns Columns have value and versions (any number) Row range & column range and key range queries

Row Key Column Family (dimensions) Column Family (metric)

112334-7782 server : host1 dc : PUNE value:20

112334-7783 server:host2 value:10

HBase Architecture

Key Value Stores

Key Value Stores Simplest possible data model

Caching a user’s personalized, rendered page – avoid DB

S3 bucket storage for blob data against a unique id

Range of KV stores Distributed, scaleable persistent key-value storage

– Dynamo, Voldemort Auto-Partitioned key space Replicated KV Highly Available

Largely in-memory KV stores – Redis, memcached Redis blazing fast for cache and other interesting

operations

Redis In memory KV store

Blazing fast – 100 K/sec R/W Async snapshot to disk

More than KV store – a data structure store – Supports lists, queues, sets and operations on

them Sorted list range operations Set operations UNION, INTERSECTION, DIFF

Redis – Use Cases Web session caching with EXPIRE set for

session expiry Live real time bit.ly URL stats like clicks etc –

fast increments of counters Auto Complete – Type first few characters –

maps to a sort list and a range query is fired Publish / Subscribe – fan out a message to

subscribers Set operations – My Twitter <Followers

INTERSECTION Followees> - tells me who all I follow but they don’t follow me back

Thanks

Email : abhijit.sharma@gmail.comTwitter : sharmaabhijitBlog : abhijitsharma.blogspot.com