KVIV / NoSQL : the new generation of database servers

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org

NoSQLde nieuwe generatie van database serversKVIV IT - 3/6/2010

http://www.flickr.com/photos/wolfgangstaudt/2215246206/

Who am I

» Steven Noels - stevenn@outerthought.org

»Outerthought : scalable content applications

»makers of Daisy and Lily open source CMS

An evolution driven by pain.

History

hierarchical databases

OODBMS

XMLDB RDBMS

1. standardization

2. simplification

History

RDBMS NOSQL

cachingdenormalisationshardingreplication ...

3. pain

4. rethinkingthe problem

Scalability

Numbers of scale

http://qos.doubleclick.net/counters/

Types of scaling

» scaling for usage» volume of users

» volume of data

availabilityreplication

» scaling types of ops» concurrent read

» concurrent write

partioningconsistency

distribution

Scaling

database app server users

Scaling

Scaling: vertical partitioning

Scaling: horizontal partitioning

databases app servers users

complexity

Scaling through architecture

data layer app layer users

Distributed systems are hard !

8 fallacies of distributed computing» The network is reliable.

» Latency is zero.

» Bandwidth is infinite.

» The network is secure.

» Topology doesn't change.

» There is one administrator.

» Transport cost is zero.

» The network is homogeneous.

Scaling relational systems

»When scaling relational systems you loose their advantages but retain their overhead

»The pain is all about locking (i.e. writes)

»Caching alleviates the read pain to the cost of complexity

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 17

The Perspective of Cost

Enter NoSQL

Cambrian Explosion

?Buzz-oriented development

Cambrian Explosion

N-O-SQL

Common themes

» SCALE SCALE SCALE

» new datamodels

» devops

»N-O-SQL

»The Cloud :technology is of no interest anymore

New Data

» sparse structures

»weak schemas

» graphs

» semi-structured

» document-oriented

»Not a movement.

»Not ANSI NoSQL-2010.

»Not one-size-fits-all.

»Not (necessarily) anti-RDBMS.

»No silver bullet.

NoSQL = pro Choice

NoSQL = toolbox

The NOSQL footprint

free-structured or sparse data

referential integrity,typed data

ly scalable an

davailab

le (com

Cassandra

CouchDB

MongoDB

NOSQL, if you need ...

» horizontal scaling (out rather than up)

» unusually common data (aka free-structured)

» speed (especially for writes)

» the bleeding edge

SQL/RDBMS, if you need ...

» SQL

»ACID

» normalisation

» a defined liability

Theory

Academic background

»Amazon Dynamo

»Google BigTable

» Eric Brewer CAP theorem

Amazon Dynamo

» coined the term ‘eventual consistency’

» consistent hashing

Consistent hashing

- node C+ node D

http://www.lexemetech.com/2007/11/consistent-hashing.html

»multi-dimensional column-oriented database

» on top of GoogleFileSystem

» object versioning

Google BigTable

CAP theorem

strong consistency

highavailability

partition-tolerance

»Strong Consistency: all clients see the same view, even in the presence of updates

»High Availability: all clients can find some replica of the data, even in the presence of failures

»Partition-tolerance: the system properties hold even when the system is partitioned

Culture Clash

»ACID» highest priority: strong

consistency for transactions

» availability less important

» pessimistic

» rigorous analysis

» complex mechanisms

» BASE» availability and scaling

highest priorities

» weak consistency

» optimistic

» best effort

» simple and fast

spectrum

Availability ≠ total async !

The Enterprise Service Bus

✘bus =

congestion

Bus systems

» objects don’t fit in a pipe

» object ➙ message

» serialization / de-serialization cost

»message size

» queuing = cost

Use a mixture of both

»async + sync

stuff which matters !

Processing large datasets :

Map/Reduce

Hadoop: HDFS + MapReduce» single filesystem + single execution-space

MapReduce example: WordCount

Hadoop ecosystem» Hadoop Common

» Subprojects

» Chukwa: A data collection system for managing large distributed systems.» HBase: A scalable, distributed database that supports structured data storage for

large tables.» HDFS: A distributed file system that provides high throughput access to application

data.» Hive: A data warehouse infrastructure that provides data summarization and ad hoc

querying.

» MapReduce: A software framework for distributed processing of large data sets on compute clusters.

» Pig: A high-level data-flow language and execution framework for parallel computation.

» ZooKeeper: A high-performance coordination service for distributed applications.» Mahout: machine learning libaries

Processing large datasets with MR

»Benefit from parallellisation

» Less modelling upfront (ad-hoc processing)

»Compartmentalized approach reduces operational risks

»AsterData et al. have SQL/MR hybrids for huge-scale BI

Market overview

Four Trends

»Trend 1 : Data Size

»Trend 2 : Connectedness

»Trend 3 : Semi-structure

»Trend 4 : Architecture

2006 2007 2008 2009 2010

ExaBytes (10!") of data stored per year

Trend 1: Data size

Data source: IDC 2007

Each year more and more digital data is created. Over two years we create more digital data than all the data created in history before that.

Trend 2: Connectedness

Text documents

Folksonomies

Tagging

User-generated

contentWikis

Hypertext

2000 2010 2020

web 1.0 web 2.0 “web 3.0”

Ontologies

Global

Graph (GGG)

Over time data has evolved to be more and more interlinked and connected.Hypertext has links,Blogs have pingback,Tagging groups all related data

Trend 3: Semi-structure

! Individualization of content

• In the salary lists of the 1970s, all elements had exactly one job

• In the salary lists of the 2000s, we need 5 job columns! Or 8?

Or 15?

!All encompassing “entire world views”

• Store more data about each entity

!Trend accelerated by the decentralization of content generation

that is the hallmark of the age of participation (“web 2.0”)

Trend 4: Architecture

Application

1980s: Mainframe applications

Application

1990s: Database as integration hub

Application Application

DBDB DB

Application

2000s: (moving towards) Decoupled serviceswith their own backend

Application Application

Products overview

We welcome the Polyglot Persistence overlords.

Categories

» key-value stores

» column stores

» document stores

» graph databases

Key-value stores

» Focus on scaling to huge amounts of data

»Designed to handle big loads

»Often: cfr. Amazon Dynamo

» ring partitioning and replication

»Data model: key/value pairs

Key-value stores

»Redis

»Voldemort

»Tokyo Cabinet

»REmote DIctionary Server

» http://code.google.com/p/redis/

» vmware

Redis Features» persisted memcache, ‘awesome’

» RAM-based + persistable

» key ➙ values: string, list, set

» higher-level ops

» i.e. push/pop and sort for lists

» fast (very)

» configurable durability

» client-managed sharding

Voldemort

» http://project-voldemort.com/

» LinkedIn

Voldemort

» persistent

» distributed

» fault-tolerant

» hash table

Voldemort

API: GET, PUT,DELETE

Voldemort

routing logic moving up the stack,smaller latency

Column stores

»BigTable clones

» Sparseness!

»Data model: columns ➙ column families ➙ ACL

»Datums keyed by: row, column, time, index

» Row-range ➙ tablet ➙ distribution

Column stores

»BigTable

»HBase

»Cassandra

BigTable

» http://labs.google.com/papers/bigtable.html

»Google

» layered on top of GFS

» http://hadoop.apache.org/hbase/

» StumbleUpon / Adobe / Cloudera

» sorted» distributed» column-oriented»multi-dimensional» highly-available» high-performance

» persisted» storage system

» adds random access reads and writes atop HDFS

HBase data model

»Distributed multi-dimensional sparse map

»Multi-dimensional keys:(table, row, family:column, timestamp) → value

»Keys are arbitrary strings

»Access to row data is atomic

Storage architecture

Cassandra

» http://cassandra.apache.org/

»Rackspace / Facebook

Cassandra

»Key-value store (with added structure)

»Reliability (identical nodes)

» Eventual consistent

»Distributed

»Tunable

» Partitioning

» Replication

Cassandra applicability

» Scalable reliability (through identical nodes)» Linear scaling»Write throughput» Large Data Sets

NO FIT

» Flexible indexing»Only PK-based

querying»Big Binary Data» 1 Row must fit in

RAM entirely

Document databases

»≈ K/V stores, but DB knows what the Value is

» Lotus Notes heritage

»Data model: collections of K/V collections

»Documents often versioned

Document stores

»CouchDB

»MongoDB

»Riak

»MarkLogic

CouchDB

» http://couchdb.apache.org/

» couch.io

CouchDB

» fault-tolerant

» schema-free

» document-oriented

» accessible via a RESTful HTTP/JSON API

CouchDB documents

{ “_id”: ”BCCD12CBB”, “_rev”: ”AB764C”, “type”: ”person”, “name”: ”Darth Vader”, “age”: 63, “headware”: [“Helmet”, “Sombrero”], “dark_side”: true }

CouchDB REST API

»HTTP

» PUT /db/docid

»GET /db/docid

» POST /db/docid

»DELETE /db/docid

CouchDB Views»MapReduce-based

» Filter, Collate, Aggregate

» Javascript

function (Key, Values) { var sum = 0; for(var i in Values) sum += Values[i]; return sum; }

function (doc) { for(var i in doc.tags) emit(doc.tags[i], 1); }

map reduce

CouchDB

» be careful on semantics

» replication ≠ partioning/sharding !

» distributed database = distributable database

» sharded / distributed deploymentrequires proxy layer

MongoDB

» http://www.mongodb.org/

» 10gen

MongoDB

» cfr. CouchDB, really

» except for:

» performance focus

» runtime queries (mapreduce still available)

» native drivers (no REST/HTTP layering)

» no MVCC: update-in-place

» auto sharding (alpha)

Graph databases

» Focus on modeling structure of data - interconnectivity

» Scale, but only to the complexity of data

»Data model: property graphs

Graph databases

»Neo4j

»AllegroGraph (RDF)

» http://neo4j.org/

»Neo Technology

Neo4j» data = nodes + relationships + key/value properties

»many language bindings, little remoting

» ‘whiteboard’ friendly

» scaling to complexity (rather than volume?)

» lots of focus on domain modelling

» SPARQL/SAIL impl for triple geeks

»mostly RAM centric (with disk swapping & persistence)

Market maturization

Rise of integrators

»Cloudera (H-stack)

»Riptano (Cassandra)

»Cloudant (hosted CouchDB)

» (Outerthought: HBase)

VC capital

»Cloudera

» couch.io

» 10gen

»many others

Experiences & (h)in(d)sights

the fireside conversations

http://www.flickr.com/photos/52641994@N00/516394238/

NOSQL applicability

»Horizontal scaling

»Multi-Master

»Data representation

» search of simplicity

» data that doesn’t fit the E-R model(graphs, trees, versions)

» Speed

Tool selection

» be careful on the marketeese:smoke and mirrors beware!

»monitor dev list, IRC, Twitter, blogs

»monitor project ‘sponsors’

»mix-and-match: polyglot persistency

»DON’T NOSQL WITHOUT INTERNAL SYS ARCHS & DEV(OP)S !

Our Context: Lily

» cloud-scalable content store and search repository

» successor (in many ways) of Daisy

aptness

complexity

NOSQL}S

Lily essentials

» (open source)

» scalable store (Apache HBase)

» and search (Apache SOLR)

» content repository

»α due mid 2010

»www.lilycms.org or @outerthought

Choosing a NoSQL store for Lily: step I

» automatic scaling to large data sets

» fault-tolerance

» flexible datamodel with sparse data

» commodity hardware

» efficient random access

» community-based open source

» Java if possible

» need for consistency

» atomic single-row updates

»M/R for index regeneration

Choosing a NoSQL store for Lily: step II

HBase» datamodel with column families and cell

versioning

» ordered tables with range scans

»HDFS for blob storage

»Apache

Choosing a NoSQL store for Lily: step III

lily simplified architectureIIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 107

Lily client

client

Lily store node

store node

distributed process coordination

and configuration (ZooKeeper)

query indexerupdate

WAL M/RMQ

documents2ary

indexes

Lily Store Server

HBase Region Server

Hadoop DFS

index replica

replica replica

} SOLR

inverted index

lily architecture

lily distributed architecture

Distribution,durability,and availabilityis everywhere

When combining store and search, make sure your (search) index doesn’t become the store.

Key lessons learned

» unlearning normalization is very difficult

» integrity checking in code = not so bad

» doing joins in code can be very liberating

» importance of keyspace design

» secondary indexing

» data de-normalization = size! (x3)

» schema vs. code flexibility?

» distribution is everywhereand you shouldn’t forget about it

Mis-use cases

» SQL (or ORM) is a prerequisite

»Deeply hierarchical datasets (unless graph)

»Data integrity is listed on DBA job description

»High-security apps (enforced in DB)

»Transactional data (banking)

»Usage is highly unpredictable, combinatorial, or likely to change suddenly

Reading material

»Amazon Dynamo, Google BigTable, CAP

» http://nosql.mypopescu.com/

» http://nosql-database.org/

» http://twitter.com/nosqlupdate

» http://highscalability.com/

Questions?

http://www.flickr.com/photos/leehaywood/4237636853/

» stevenn@outerthought.org

» @stevenn

Thanks for your attention !

KVIV / NoSQL : the new generation of database servers

Technology

Transcript of KVIV / NoSQL : the new generation of database servers

How to Use NoSQL - nosqlroadshow.comnosqlroadshow.com/dl/NoSQL-Road-Show/slides/Howt... · How to Use NoSQL in Enterprise Java Applications Patrick Baumgartner NoSQL Roadshow | Zürich

NoSQL Racket: A Testing Tool for Detecting NoSQL Injection ...thesai.org/Downloads/Volume8No11/Paper_78-NoSQL... · CouchDB and so on for other NoSQL Database Types (Cassandra, Amazon

Integrating Oracle Databases with NoSQL Databases for ... · Integrating Oracle Databases with NoSQL Databases for Linux on IBM LinuxONE and z System Servers Demo walkthrough Check

PostSQL Using PostgreSQL as a better NoSQL - NoSQL Matters

Big Data and Machine Learning Using MATLAB · Software Servers and Databases Repositories –SQL, NoSQL, etc. ... Weather Forecasting Energy Load Forecasting Stock Market Prediction

MYSQL & NOSQL: THE BEST OF BOTH WORLDSnosqlroadshow.com/dl/NoSQL-Amsterdam-2013/Slides/... · • NoSQL C++ API, flexaSynch benchmark • 30 x Intel E5-2600 Intel Servers, 2 socket,

Couchdb Nosql

NoSQL or not NoSQL?

NoSQL Oakland

Oracle NoSQL

TECHNOLOGY NEWS Will NoSQL Databases Live Up to Their … · NOSQL PROS AND CONS NoSQL databases have numerous advantages and disadvantages. Advantages NoSQL databases generally pro-cess

SQL vs NoSQL: The NoSQL way

Q y // NoSQL’Road’Show,’Zurich’nosqlroadshow.com/dl/NoSQL-Road-Show/slides/nosql... · NoSQL,’NewSQL’and’Beyond ... •’OrientDB ’ •’NuvolaBase ... •’ScaleBase

NOSQL PROBLEM LITERATURE REVIEW - … · NoSQL systems are viable solutions for applications that require scalable data repositories, which can easily scale out over multiple servers

MIRhosting Cloud · • Supports Java, PHP, Ruby, Python, Node.js, .NET and Docker • Software stack of 60+ preconfigured App servers, SQL \ NoSQL Databases, Load Balancers, …

Accelerating NoSQL

NoSQL Games

Adrian Colyer - Keynote: NoSQL matters - NoSQL matters Dublin 2015

A FIRST TOUCH ON NOSQL SERVERS: COUCHDB GENOVEVA VARGAS SOLAR, JAVIER ESPINOSA CNRS, LIG-LAFMIA, FRANCE Genoveva.Vargas@imag.frGenoveva.Vargas@imag.fr;

NoSQL DB properties and its data modelsBenefits of NoSQL (2) Economics •RDBMS rely on expensive proprietary servers to manage data •No SQL: clusters of cheap commodity servers