CFCamp 2016 - Couchbase Overview

Couchbase OverviewAaron Benton

"A good programmer is someone who looks both ways before crossing a one-way street."

- Doug LinderCredit: Couchbase

4

Agenda

What is NoSQL?

What is Couchbase?

Couchbase Architecture

CFCouchbase

Queries

Demo (if time permits)

What is NoSQL?

6

What is NoSQL?

Non-Relational

Cluster Friendly

Generally Open-Source

21st Century

Schema-Less

7

Scaling

Scale Vertically (RDBMS)Add resources to a single node

in a system

Enhance the server (more CPU, more RAM, etc)

High availability, difficult to implement

Scale Horizontally (NoSQL)Add more nodes to a system

More servers, distributing load

High Availability, easy to implement

Container Friendly

Cattle

8

Pets

Any type of data

Flexible

Application Managed

Change is easy

Known Models

Fixed Fields

Data Types

Database Managed

Change can be difficult

9

Schemas

Relational Non-Relational

10

Types of NoSQL Databases

Key-ValueRedis

Riak

Memcached

DocumentCouchbase

CouchDB

MongoDB

Column-FamilyCassandra

Base

BigTable

GraphNeo4J

Giraph

OrientDB

11

In Development…

Objects are assembled as a whole:Cart

Order

Product

Profile

Saving these Objects requires:Deconstructing

Multiple Rows

Multiple Tables

12

Impedance Mismatch

"The object-relational impedance mismatch is a set of conceptual and technical difficulties that are often encountered when a relational database management system (RDBMS) is being used by a program written in an object-oriented programming language or style, particularly when objects or class definitions are mapped in a straightforward way to database tables or relational schema." - Wikipedia

13

ORM

14

Relational Models

Database

// order (structure / dictionary / object / map /etc)

order['order_id'] = 3492843;

order['order_date'] = "2016-07-14T18:27:22.586Z";

order['products'] = [{

'product_id' = 78323,

'quantity' = 2,

'price' = 39.99,

'sub_total' = 79.98,

}];

order['user_id'] = 123;

order['billing_address_1'] = "1302 Pleasant Ridge Rd";

order['billing_address_2'] = "";

order['billing_city'] = "Greensboro";

order['billing_region_code'] = "NC"; order['billing_postal_code'] = "27409";

order['billing_country_code'] = "US";

order['shipping_address_1'] = "1302 Pleasant Ridge Rd";

order['shipping_address_2'] = "";

order['shipping_city'] = "Greensboro";

order['shipping_region_code'] = "NC";

order['shipping_postal_code'] = "27409";

order['shipping_country_code'] = "US";

order['card_number'] = "3337151609084503";

order['expiration_month'] = 11;

order['expiration_year'] = 2019;

Code

15

Non-Relational Models

// order (structure / dictionary / object / map /etc)

order['order_id'] = 3492843;

order['order_date'] = "2016-07-14T18:27:22.586Z";

order['products'] = [{

'product_id' = 78323,

'quantity' = 2,

'price' = 39.99,

'sub_total' = 79.98,

}];

order['user_id'] = 123;

order['billing_address_1'] = "1302 Pleasant Ridge Rd";

order['billing_address_2'] = "";

order['billing_city'] = "Greensboro";

order['billing_region_code'] = "NC"; order['billing_postal_code'] = "27409";

order['billing_country_code'] = "US";

order['shipping_address_1'] = "1302 Pleasant Ridge Rd";

order['shipping_address_2'] = "";

order['shipping_city'] = "Greensboro";

order['shipping_region_code'] = "NC";

order['shipping_postal_code'] = "27409";

order['shipping_country_code'] = "US";

order['card_number'] = "3337151609084503";

order['expiration_month'] = 11;

order['expiration_year'] = 2019;

Database (JSON)Code

{

"order_id": 3492843,

"order_date": "2016-07-14T18:27:22.586Z",

"products": [{

'product_id': 78323,

'quantity': 2,

'price': 39.99,

'sub_total': 79.98,

}];

"user_id": 123,

"billing_address_1": "1302 Pleasant Ridge Rd",

"billing_address_2": "",

"billing_city": "Greensboro",

"billing_region_code": "NC",

"billing_postal_code": "27409",

"billing_country_code": "US",

"shipping_address_1": "1302 Pleasant Ridge Rd",

"shipping_address_2": "",

"shipping_city": "Greensboro",

"shipping_region_code": "NC",

"shippping_postal_code": "27409",

"shippping_country_code": "US",

"card_number": "3337151609084503",

"expiration_month": 11,

"expiration_year": 2019

}

BASEACID

16

Transaction Processing

Atomicity

Consistency

Isolation

Durability

Basically Available

Soft-State

Eventual Consistency

Relational Non-Relational

17

CAP Theorem

18

Only NoSQL?

Is SQL going away?

NoNoSQL databases are just another tool

What is Couchbase?

History

21

High availability cache

Key-value store

Document database

Embedded database

Sync management

Couchbase Server Couchbase Lite CouchbaseSync Gateway

Data management for a broad range of use cases

22

Couchbase Tenants

Flexible data model

Consistent performance at scale

High availability

Easy, affordable scalability

24x365

23

Buckets

24

Couchbase Connectors

Architecture

Couchbase Node

26

Couchbase Server Node

Single-node type means easier administration and scaling

Single installation

Two major components/processes: Data manager cluster manager

Data manager:

C/C++

Layer consolidation of caching and persistence

Cluster manager:

Erlang/OTP

Administration UI’s

Out-of-band for data requests

27

Couchbase Read Operation

APPLICATION SERVER

MANAGED CACHE

DISK

DISKQUEUE

REPLICATIONQUEUE

DOC 1

GETDOC 1

DOC 1


Reads out of cache are extremely fast

No other process/system to communicate with

Data connection is a TCP-binary protocol

DOC 1

28

APPLICATION SERVER

MANAGED CACHE

DISK

DISKQUEUE

REPLICATIONQUEUE

Couchbase Write Operation

DOC 1

DOC 1DOC 1


Writes are async by default

Application gets acknowledgement when successfully in RAM and can trade-off waiting for replication or persistence per-write

Replication to 1, 2 or 3 other nodes

Replication is RAM-based so extremely fast

Off-node replication is primary level of High Availability

Disk written to as fast as possible –no waiting

29

Couchbase Cache Ejection

APPLICATION SERVER

MANAGED CACHE

DISK

DISKQUEUE

REPLICATIONQUEUE

DOC 1

DOC 2DOC 3DOC 4DOC 5

DOC 1

DOC 2 DOC 3 DOC 4 DOC 5


Layer consolidation means read through and write through cache

Couchbase automatically removes data that has already been persisted from RAM

30

APPLICATION SERVER

MANAGED CACHE

DISK

DISKQUEUE

REPLICATIONQUEUE

DOC 1

Couchbase Cache Miss



GETDOC 1

DOC 1

DOC 1


Layer consolidation means 1 single interface for App to talk to and get its data back as fast as possible

Separation of cache and disk allows for fastest access out of RAM while pulling data from disk in parallel

Cluster Overview

Scaling

Architecture

Couchbase Cluster: Node and SDK Interaction

34

Auto sharding – Bucket and vBuckets

vB

Data buckets

vB

1 ….. 1024

Virtual buckets

A bucket is a logical, unique key space

Multiple buckets can exist within a single cluster of nodes

Each bucket has active and replica data sets (1, 2 or 3 extra copies)

Each data set has 1024 Virtual Buckets (vBuckets)

Each vBucket contains 1/1024th portion of the data set

vBuckets do not have a fixed physical server location

Mapping between the vBuckets and physical servers is called the cluster map

Document IDs (keys) always get hashed to the same vbucket

Couchbase SDK’s lookup the vbucket -> server mapping

35

ACTIVE ACTIVE ACTIVE

REPLICA REPLICA REPLICA

Couchbase Server 1 Couchbase Server 2 Couchbase Server 3

Basic Operation

SHARD5

SHARD2

SHARD9

SHARD SHARD SHARD

SHARD4

SHARD7

SHARD8

SHARD SHARD SHARD

SHARD1

SHARD3

SHARD6

SHARD SHARD SHARD

SHARD4

SHARD1

SHARD8

SHARD SHARD SHARD

SHARD6

SHARD3

SHARD2

SHARD SHARD SHARD

SHARD7

SHARD9

SHARD5

SHARD SHARD SHARD

Application has single logical connection to cluster (client object)

Data is automatically sharded resulting in even document data distribution across cluster

Each vbucket replicated 1, 2 or 3 times (“peer-to-peer” replication)

Docs are automatically hashed by the client to a shard

Cluster map provides location of which server a shard is on

Every read/write/update/delete goes to same node for a given key

Strongly consistent data access (“read your own writes”)

A single Couchbase node can achieve 100k’s ops/sec so no need to scale reads

36

Cluster Map

37

Cluster Map

38

Cluster Map – 2 nodes added

39

Rebalance




ACTIVE ACTIVE

REPLICA REPLICA

Couchbase Server 4 Couchbase Server 5

SHARD5

SHARD2

SHARD SHARD

SHARD4

SHARD SHARD

SHARD1

SHARD3

SHARD SHARD

SHARD4

SHARD1

SHARD8

SHARD SHARD SHARD

SHARD6

SHARD3

SHARD2

SHARD SHARD SHARD

SHARD7

SHARD9

SHARD5

SHARD SHARD SHARD

SHARD7

SHARD

SHARD6

SHARD

SHARD8

SHARD9

SHARD

READ/WRITE/UPDATE


Multiple nodes added or removed at once

One-click operation

Incremental movement of active and replica vbuckets and data

Client library updated via cluster map

Fully online operation, no downtime or loss of performance

40

Fail Over Node




ACTIVE ACTIVE

REPLICA REPLICA

Couchbase Server 4 Couchbase Server 5

SHARD5

SHARD2

SHARD SHARD

SHARD4

SHARD SHARD

SHARD1

SHARD3

SHARD SHARD

SHARD4

SHARD1

SHARD8

SHARD SHARD

SHARDSHARD6

SHARD2

SHARD SHARD SHARD

SHARD7

SHARD9

SHARD5

SHARD SHARD

SHARD

SHARD7

SHARD

SHARD6

SHARDSHARD8

SHARD9

SHARD

SHARD3

SHARD1

SHARD3

SHARD


When node goes down, some requests will fail

Failover is either automatic or manual

Client library is automatically updated via cluster map

Replicas not recreated to preserve stability

Best practice to replace node and rebalance

41

XDCR

Cross Datacenter Replication

Replication to other clusters

Bi-Directional

Uni-directional

Filtered Replication

What about Mongo?

43

MongoDB

Replication: Master - Slave

Primaries and Secondaries

Not all writes are local

Need for 3rd Party Cache

No Mobile Solution

Complex Topology

Database + Collections + BSON

Cluster with 2 replicas / backups

Distributed Load

44

Couchbase vs MongoDB

Who's using Couchbase?

46

Who's using Couchbase…?

CFCouchbase 2.0

48

Documents are integral to the SDKs.

All SDK’s support JSON format

In addition: Serialized objects, Unquoted Strings, Binary pass-through

A Document contains:

Couchbase SDK

22

Property Description

ID The bucket-unique identifier

Content The value that is stored

Expiry An expiration time

CAS Check-and-Set identifier

CF Couchbase 2.0 Brings

Java SDK (2.2.5)

N1QL Support w/ GSI Indexes

Replica Reads

Document Locking

Prepared Statements

Design Document Management

Expanded Config

And More…

1.* 2.0

add insert

incrdecr

counter

delete remove

newQuery n1qlQueryviewQuery

set upsert

setMulti upsertMulti

setWithCAS replaceWithCAS

Method Changes

50

Other Couchbase SDKs

Official SDKs Java

.NET

Node.js

Python

For each of these there is:

Full Document support

Interoperability

Common yet idiomatic Programming Model

Others: Erlang, Perl, TCL, Clojure, Scala

Also fully REST accessible

PHP

C / C++

Go

Ruby

JDBC and ODBC

51

Connecting to Couchbase

22

public boolean function onApplicationStart(){

application['couchbase'] = new cfcouchbase.CouchbaseClient({

servers: "127.0.0.1",

bucketName: "beer-sample"

});

return true;

}

public boolean function onApplicationEnd(){

application.couchbase.shutdown( 10 );

return true;

}

52

Writing Documents

22

var document = {

"id": "512_brewing_company",

"name": "(512) Brewing Company",

"city": "Austin",

"state": "Texas",

"code": "78745",

"country": "United States",

"phone": "512.707.2337",

"website": "http://512brewing.com/",

"type": "brewery",

"updated": "2010-07-22 20:00:20",

"description": "(512) Brewing Company is a microbrewery located in the

heart of Austin that brews for the community using as many local,

domestic and organic ingredients as possible.",

"address": [

"407 Radam, F200"

],

"geo": {

"accuracy": "ROOFTOP",

"lat": 30.2234,

"lon": -97.7697

}

};

application.couchbase.upsert

id=document.id,

value=document

);

Define a Document Save the Document

53

Retrieving Documents

22

application.couchbase.get(

id="512_brewing_company"

);

Get the Document

54

Counter Operations

22

// increment the user counter

next_id = couchbase.counter(

id="user_counter",

value=1,

defaultValue=0

);

// write the user record

couchbase.insert("user_" & next_id, {

"user_id": next_id,

"name": "John Smith",

"email": "[email protected]"

});

Increment

// decrement the tickets counter

available_tix = couchbase.counter(

id="tickets_counter",

value=-1,

defaultValue=1000

);

// if there are no more tickets

// available redirect the user

if (available_tix < 1) {

cflocation(

url="sold-out.cfm",

addtoken=false

);

}

...

Decrement

Querying

56

Map / Reduce Views

In Couchbase, Map-Reduce is specifically used to create Indexes

Map functions are applied to JSON documents and their output or "emit" data is stored in an index

57

Querying

SELECT first_name, last_name, children FROM users

looks like SQL…WHERE EVERYchild IN children SATISFIES child.age > 10

END

58

N1QL Examples

INSERT INTO ecommerce ( KEY, VALUE )VALUES ("user_1021", {

"user_id": 1021,"doc_type": "user","first_name": "John","last_name": "Smith","email": "[email protected]"

})

UPDATE ecommerceUSE KEYS "user_1021"SET email = "[email protected]",

gender = "M",token = UUID()

RETURNING token

UPSERT INTO ecommerce ( KEY, VALUE )VALUES ("user_1021", {

"user_id": 1021,"doc_type": "user","first_name": "John","last_name": "Smith","email": "[email protected]"

})

DELETEFROM ecommerceUSE KEYS "user_1021"

N1QL Operators

59

Type Support

Arithmetic + - * / % -val

Collection ANY EVERY ARRAY FIRST EXISTS IN WITHIN

Comparison = == != <> > >= < <=(NOT) BETWEEN (NOT) LIKE IS (NOT) NULL IS (NOT) MISSING IS (NOT) VALUED

Conditional CASE expression WHEN value THEN expression

Construction Array [ value, value, ... ] Object { key:value, key:value, ... }

Logical AND OR NOT

String ||

N1QL Operators

60

ARRAY_AGG(EXP)

ARRAY_AGG(DISTINCT EXP)

AVG(EXP)

AVG(DISTINCT EXP)

COUNT(*)

COUNT(EXP)

COUNT(DISTINCT EXP)

MAX(EXP)

MIN(EXP)

SUM(EXP)

SUM(DISTINCT EXP)

Aggregate Functions

Object Functions

OBJECT_LENGTH(EXP)

OBJECT_NAMES(EXP)

OBJECT_PAIRS(EXP)

OBJECT_VALUES(EXP)

Conditionals - Unknowns

IFMISSING(EXP1, EXP2, …)

IFMISSINGORNULL(EXP1, EXP2, …)

IFNULL(EXP1, EXP2, …)

MISSINGIF(EXP1, EXP2)

NULLIF(EXP1, EXP2)

Conditionals - Numbers

IFINF(EXP1, EXP2, …)

IFNAN(EXP1, EXP2, …)

IFNANORINF(EXP1, EXP2, …)

NANIF(EXP1, EXP2)

NEGINFIF(EXP1, EXP2)

POSINFIF(EXP1, EXP2)

Comparison Functions

GREATEST(EXP1, EXP2)

LEAST(EXP1, EXP2)

Meta and UUID Functions

BASE64(EXP)

BASE64_ENCODE(EXP)

BASE64_DECODE(EXP)

META(EXP)

UUID()

Number Functions

ABS(EXP)

ACOS(EXP)

ASIN(EXP)

ATAN(EXP)

ATAN2(EXP1, EXP2)

CEIL(EXP)

COS(EXP)

DEGREES(EXP)

E(EXP)

EXP(EXP)

LN(EXP)

LOG(EXP)

FLOOR(EXP)

PI(EXP)

POWER(EXP1, EXP2)

RADIANS(EXP)

RANDOM([ EXP ])

ROUND(EXP [, DIGITS])

SIGN(EXP)

SIN(EXP)

SQRT(EXP)

TAN(EXP)

TRUNC(EXP [, DIGITS])

Type Checking Functions

ISARRAY(EXP)

ISATOM(EXP)

ISBOOLEAN(EXP)

ISNUMBER(EXP)

ISOBJECT(EXP1, EXP2)

ISSTRING(EXP)

TYPE(EXP)

Type Conversion Functions TOARRAY(EXP)

TOATOM(EXP)

TOBOOLEAN(EXP)

TONUMBER(EXP)

TOOBJECT(EXP)

TOSTRING(EXP)

N1QL Operators

61

ARRAY_APPEND(EXP, VAL)

ARRAY_AVG(EXP)

ARRAY_CONCAT(EXP1, EXP2)

ARRAY_CONTAINS(EXP, VAL)

ARRAY_COUNT(EXP)

ARRAY_DISTINCT(EXP)

ARRAY_IFNULL(EXP)

ARRAY_LENGTH(EXP)

ARRAY_MAX(EXP)

ARRAY_MIN(EXP)

ARRAY_POSITION(EXP, VAL)

ARRAY_PREPEND(VAL, EXP)

ARRAY_PUT(EXP, VAL)

ARRAY_RANGE(START, END [,STEP])

ARRAY_REMOVE(EXP, VAL)

ARRAY_REPEAT(VAL, N)

ARRAY_REPLACE(EXP, VAL1, VAL2 [,N])

ARRAY_REVERSE(EXP)

ARRAY_SORT(EXP)

ARRAY_SUM(EXP)

Array Functions Date Functions

CLOCK_MILLIS()

CLOCK_STR ([FMT ])

DATE_ADD_MILLIS(EXP, N, PART)

DATE_ADD_STR(EXP, N,PART)

DATE_DIFF_MILLIS(EXP1, EXP2, PART)

DATE_DIFF_STR(EXP1, EXP2, PART)

DATE_PART_MILLIS(EXP, PART)

DATE_PART_STR(EXP, PART)

DATE_TRUNC_MILLIS(EXP, PART)

DATE_TRUNC_STR(EXP, PART)

MILLIS(EXP)

STR_TO_MILLIS(EXP)

MILLIS_TO_STR(EXP [, FMT ])

MILLIS_TO_UTC(EXP [, FMT ])

MILLIS_TO_ZONE_NAME(EX, TZ[,FMT])

NOW_MILLIS()

NOW_STR([ FMT ])

STR_TO_MILLIS(EXP)

MILLIS(EXP)

STR_TO_UTC(EXP)

STR_TO_ZONE_NAME(EXP, TZ_NAME)

Number Functions

CONTAINS(EXP, SUBSTRING)

INITCAP(EXP )

TITLE(EXP)

LENGTH(EXP)

LOWER(EXP)

LTRIM(EXP [,CHARACTERS ])

POSITION(EXP, SUBSTRING)

REPEAT(EXP, N)

REPLACE(EXP, SBSTR, REPL [, N ])

RTRIM(EXP, [,CHARACTERS ])

SPLIT(EXP [, SEP ])

SUBSTR(EXP, POS[, LEN ])

TRIM(EXP [, CHARACTERS ])

UPPER(EXP)

Pattern Matching Functions

REGEXP_CONTAINS(EXP, PATTERN)

REGEXP_LIKE(EXP, PATTERN)

REGEXP_POSITION(EXP, PATTERN)

REGEXP_REPLACE(EXP, PTRN, REPL [, N ])

JSON Functions

DECODE_JSON(EXP)

ENCODE_JSON(EXP)

ENCODED_SIZE(EXP)

POLY_LENGTH(EXP)

62

Retrieving Documents

22

couchbase.n1qlQuery(

statement="

SELECT airport_id,

airport_name, airport_type,

municipality, geo, timezone,

airport_iata, airport_icao,

FROM `flight-data`

WHERE iso_country = $1

AND iso_region = $2

AND doc_type = $3

ORDER BY airport_name ASC

LIMIT 2

",

parameters=[

"US",

"US-ME",

"airport"

]

);

CF N1QL Query

Examples

Questions?

box install cfcouchbase

67

Resources

22

ortussolutions.com/products/cfcouchbase

ortussolutions.com/blog/category/cfcouchbase

developer.couchbase.com query.pub.couchbase.com

bit.ly/n1ql-41-language-reference

blog.couchbase.com

bit.ly/cfcouchbase-travel bit.ly/cfcouchbase-samples

bit.ly/node-fakeit

thepolyglotdeveloper.com

couchbase.com/nosql-resources/presentations

https://www.ortussolutions.com/products/cfcouchbase

https://www.ortussolutions.com/blog/category/cfcouchbase

http://developer.couchbase.com

http://query.pub.couchbase.com

http://bit.ly/n1ql-41-language-reference

http://blog.couchbase.com

http://bit.ly/cfcouchbase-travel

http://bit.ly/cfcouchbase-samples

http://bit.ly/node-fakeit

https://www.thepolyglotdeveloper.com

http://www.couchbase.com/nosql-resources/presentations

CFCamp 2016 - Couchbase Overview

Technology

Transcript of CFCamp 2016 - Couchbase Overview