Couchbase 102 - Developing with Couchbase
-
Upload
couchbase -
Category
Technology
-
view
3.166 -
download
1
description
Transcript of Couchbase 102 - Developing with Couchbase
Friday, October 11, 13
Technical Evangelist
twi0er: @scalabl3email: [email protected]
Jasdeep Jaitla
Couchbase 102: Development
Friday, October 11, 13
Technical Evangelist
twi0er: @scalabl3email: [email protected]
Jasdeep Jaitla
Couchbase 102: Development
Friday, October 11, 13
Friday, October 11, 13
SETUP SDK
Friday, October 11, 13
Supported SDK'swww.couchbase.com/communi/es
• Each supported SDK page has instrucHons for setup
• PHP, Ruby, NodeJS and Python clients are wrappers around libcouchbase C library, so libcouchbase must be installed first
• For other community clients, click on "All Clients" on leR nav, scroll down the page and you can see clients for Go, Erlang, Clojure, TCL, other nodejs and Perl.
Friday, October 11, 13
Installing Libcouchbase
• Mac Tips before Libcouchbase & SDK Install
• Make sure you have XCode & Command Line Tools Installed
• Install Homebrew if you don't have it already
• Do a $ brew update, $ brew upgrade and $ brew doctor to be sure you're up to date
www.couchbase.com/communi/es/c/geBng-‐started
Friday, October 11, 13
Installing Libcouchbase
• Mac Via Homebrew
• $ brew install libcouchbase
• PC-‐Windows
• Download appropriate Zip from website
• Redhat/CentOS
• wget the yum repositories
• $ sudo yum install -‐y libcouchbase2-‐libevent libcouchbase-‐devel
• Ubuntu
• wget ubuntu repositories
• $ sudo apt-‐get install libcouchbase2-‐libevent libcouchbase-‐dev
www.couchbase.com/communi/es/c/geBng-‐started
Friday, October 11, 13
Make a Connectionrequire 'rubygems'require 'couchbase' cb = Couchbase.connect(
:bucket => "default",:hostname => "localhost")
data = { jsonkey: "value", created_at: Time.now }
cb.add("mydata", data)puts cb.get("mydata")
#!/usr/bin/env pythonfrom couchbase import Couchbasefrom pprint import pprintfrom datetime import datetime
cb = Couchbase.connect(bucket='default')
data = { "jsonkey": "value", "created_at": datetime.now() }
cb.add('mydata', data)result = cb.get('mydata')pprint(result.value, indent=4)
RUBY
PYTHON
Friday, October 11, 13
Make a Connection
import com.couchbase.client.CouchbaseClient;import java.net.URI;import java.util.*;import com.google.gson.Gson;import com.google.gson.GsonBuilder;
public class HelloWorld {
public static void main(String[] args) throws Exception {
List<URI> hosts = Arrays.asList(new URI("http://127.0.0.1:8091/pools")
);
CouchbaseClient cb = new CouchbaseClient(hosts, "default", "");Gson json = new Gson();
Hashtable data = new Hashtable();data.put("jsonkey", "value");data.put("created_at", new Date());
cb.add("mydata", json.toJson(data))System.out.println(cb.get("mydata"));
cb.shutdown();
}}
JAVA
Friday, October 11, 13
Make a Connection
var Couchbase = require('couchbase');var cb = new Couchbase.Connection({bucket: "default"}, function(err) { });
var data = { jsonkey: "value", created_at: new Date().toString() };cb.add("mydata", data);
console.log(cb.get("mydata"));
NODEJS
<?php// adjust these parameters to match your installation$cb = new Couchbase("127.0.0.1:8091", "", "", "default");
$data = array("jsonkey" => "value", "created_at" => date("Y-m-d H:i:s"));
$cb->add("mydata",json_encode($data));var_dump($cb->get("mydata"));
?>
PHP
Friday, October 11, 13
Friday, October 11, 13
RDBMS VS COUCHBASE
Friday, October 11, 13
RDBMS Organization
• RDBMS organizes data as tables- Tables represent data in rows; n columns of m rows- Table rows have a specific schema, each column as a static
type- Simple Datatypes: strings, numbers, datetimes, booleans,
can be represented by columns in a single table- Complex Datatypes: dictionaries/hashes, arrays/lists
cannot be represented in a single table [Impedence Mismatch]
• All rows have identical schema, schema changes require taking database offline, migrations, royal pains
• Reading/Writing/Transactions require mutex and locking
Friday, October 11, 13
Couchbase Organization
• Couchbase operates like a Key-Value Document Store - Simple Datatypes: strings, numbers, datetime, boolean, and
binary data can be stored; they are stored as Base64 encoded strings
- Complex Datatypes: dictionaries/hashes, arrays/lists, can be stored in JSON format (simple lists can be string based with delimiter)
- JSON is a special class of string with a specific format for encoding simple and complex data structures
• Schema is unenforced and implicit, schema changes are programmatic, done online, and can vary from Document to Document
Friday, October 11, 13
Complex Datatypes
• Simple Types are easy, make them columns
• Complex Types are more challenging, require separate tables and joins, slower to store and retrieve
• ORM's reduce complexity but trade off additional speed/scale, hard to optimize
RDBMS
public class User {
private String name;private String email;private Integer age;private Boolean gender_male;private DateTime created_at;private ArrayList items_viewed;private Hashtable preferences;private ArrayList<Books> authored;
public User(...) {...
}
...}
Friday, October 11, 13
Complex Datatypes
• Can represent both simple and complex data types in JSON data structures
• Can modify schema on the fly, and Documents of a specific "type" can vary in schema
• "Type" is arbitrary, it's a programming strategy, there are no actual "types", but it's typical to embed the class name as a "doctype" json key
Couchbase
{"doctype": "User","name": "Jasdeep Jaitla","email": "[email protected]","age": 38,"gender_male": true,"created_at": "2013-09-20 23:59:59","items_viewed": [
"12345", "23456", 34567"],"preferences": {
"email_notifications": true,"sms_notifications": false
},"authored": [
{ "title": "Couchbase Models","price": 49.95 }
]}
Friday, October 11, 13
Benefits of JSON
• Can Represent Complex Objects and Data Structures• Very simple notation, lightweight, compact, readable• The most common API return type for Integrations - Facebook, Twitter, you name it, return JSON
• Native to Javascript (can be useful)• Serialization and Deserialization are very fast
Friday, October 11, 13
Friday, October 11, 13
OPERATIONS
Friday, October 11, 13
• get (key)– Retrieve a document
• set (key, value)– Store a document, overwrites if exists
• add (key, value)– Store a document, error/excepHon if exists
• replace (key, value)– Store a document, error/excepHon if doesn’t exist
• cas (key, value, cas)– Compare and swap, mutate document only if it hasn’t changed while execuHng this operaHon
Store & Retrieve OperaHons
Friday, October 11, 13
Atomic Counter OperationsThese operaHons are always executed in order atomically.
• incr (key)– Increase an atomic counter value, default by 1
• cb.incr(“my_counter”) # now it’s 2
• decr (key)– Decrease an atomic counter value, default by 1
• cb.decr(“my_counter”) # now it’s 1
Friday, October 11, 13
Non-JSON OperationsYou can use these creaHvely!
• prepend (key, value)– prepend exisHng string in couchbase with value
• cb.prepend(“mykey”, "jumped over the lazy dog") • cb.prepend("mykey", "the brown fox ")
• append (key, value)– append exisHng string in couchbase with value
• cb.append(“mykey2”, "oranges") • cb.append("mykey2", " apples bananas")
Friday, October 11, 13
Friday, October 11, 13
CONCURRENCY
Friday, October 11, 13
Optimistic Concurrency with CAS
• Every storage operation (including touch) creates a new "CAS" value, which is just a long int
• The CAS value simply represents the current state of the Document, it's like a version number
• You can use this CAS for "Optimistic Concurrency"- value, flags, cas = get("mykey", :extended => true)
- This will only succeed if the CAS matches• replace("mykey", newvalue, :cas => cas)
- If another process had changed the "mykey" document, a new CAS will have been generated, and that replace operation will fail
Friday, October 11, 13
Friday, October 11, 13
EXPIRATIONS
Friday, October 11, 13
Storage Operations with Expirations
• CMS Framework Cache can be configured to use Couchbase- In most frameworks this is simple, as they typically already
have memcached support
• Create/Update the Value and Expiration- [Ruby] set/add/replace("mykey", value, :ttl => 30)
- [Java] set/add/replace("mykey", 30, value)
• Update the expiration only- [Ruby] touch("mykey", :ttl => 30)
- [Java] touch("mykey", 30)
Friday, October 11, 13
Friday, October 11, 13
DURABILITY
Friday, October 11, 13
Using Storage with Observe
• Callback when it has been written to disk on active partition• Callback when it has been written to a replica(s)• Callback when it has been written to replica(s) disk• Observe Persisted to Disk and Replicated
• [Ruby] set/add/replace("mykey", value, :observe => {:persisted => 1, :replicated => 1})
• [Java] set/add/replace("mykey", 0, value, PersistTo.MASTER, ReplicateTo.ONE)
• Observe Replicated Only• [Ruby] set/add/replace("mykey", value,
:observe => {:replicated => 1})• [Java] set/add/replace("mykey", 0, value, ReplicateTo.ONE)
Friday, October 11, 13
Friday, October 11, 13
BASIC KEY PATTERNS
Friday, October 11, 13
Objects Serialized to JSON and Back User Objectstring uid
string firstname
string lastname
int age
array favorite_colors
string email
“uid”: 123456,“firstname”: “jasdeep”,“lastname”: “Jaitla”,“age”: 22,“favorite_colors”: [“blue”, “black”],“email”: “[email protected]”
}
User Objectstring uid
string firstname
string lastname
int age
array favorite_colors
string email
“uid”: 123456,“firstname”: “jasdeep”,“lastname”: “Jaitla”,“age”: 22,“favorite_colors”: [“blue”, “black”],“email”: “[email protected]”
}
add()
get()
Friday, October 11, 13
Basic Keying
• Use a Unique value for key (email, username, sku, isbn, etc.)- Users
• u::[email protected]• u::scalabl3
- Products• p::978-0321573513 [isbn]
• Predictable Keys can follow Key-Value patterns (Users typically can be done this way and are the most numerous items)
• Unpredictable Keys (GUID, UUID, etc.) require Views (Map-Reduce Indexes) to find their documents
Friday, October 11, 13
Counter-ID
ApplicaHon
id = incr("counter-‐key")
add("key" + id, data)
ApplicaHon
count = get("counter-‐key")
mulH-‐get(keys[])
Data Crea/on
Iterate Through Collec/on
Friday, October 11, 13
Counter-ID
• Similar to IDENTITY column in RDBMS• Creating New Document is a pair of operations, INCR and ADD- Initialize one Key as an Atomic Counter (I do at App Start)- Increment Counter and save new value
✴ id = client.incr("blog::couchbase::comment_count")
- Use the id as component of key for new document✴ client.add(""blog::couchbase::c"::" + id, self.to_json)
Friday, October 11, 13
Lookup Pattern
ApplicaHon
add("[email protected]", "u::550e8400-‐e29b-‐41d4-‐a716")
add("u::550e8400-‐e29b-‐41d4-‐a716", data)
add("scalabl3", "u::550e8400-‐e29b-‐41d4-‐a716")
ApplicaHon
key = get("[email protected]")
get(key)
Data Crea/on
Data Retrieval
Friday, October 11, 13
Lookup Pattern
• Create simple document that has referential data (Key) to primary document- Primary Document u::a2bf2-23317-2302- Lookup Document: u::[email protected]
{ u::a2bf2-23317-2302 }• Lookup Documents aren't JSON, they should just be the Key
as a string so you skip JSON parsing• Requires Two GET operations, first GET Lookup, then GET
primary Document- key = client.get("u::[email protected]")- doc = client.get(key)
Friday, October 11, 13
User Data Multiple Social Networks & Emails
u::count
1001
u::1001
{ "name": "Jasdeep Jaitla",
"facebook_id": 16172910,
"email": "[email protected]",
“password”: ab02d#Jf02K
"created_at": "5/1/2012 2:30am",
“facebook_access_token”: xox0v2dje20,
“twi0er_access_token”: 20jffieieaaixixj }
\::16172910
1001
nflx::2939202
1001
twtr::2920283830
1001
1001
1001
uname::scalabl3
1001
Friday, October 11, 13
Combine Counter-ID and Lookup
ApplicaHon
add("[email protected]", id)
add("u::" + id, data)
add("scalabl3", id)
ApplicaHon
key = get("[email protected]")
get(key)
Data Crea/on
Data Retrieval
id = incr("user::count")
Friday, October 11, 13
Combine Counter-ID and Lookup
Pro's• Binary Operations, overall faster than large volume of View
Queries• Essentially creates several ways to find a single document• Is always consistent, just like all other Binary operations
Con's• Increases Number of Documents, therefore Metadata usage in
RAM- But this is generally a non-issue for most people
Friday, October 11, 13
User Data (Sample)
CREATE TABLE Users
id, int, identity(1000) PRIMARY KEY
name, nvarchar(100)
facebook_id, bigint
email, nvarchar(255)
created_at, datetime
u::count
1
u::1001
{ "name": "Jasdeep Jaitla",
"facebook_id": 16172910,
"email": "[email protected]",
"created_at": "5/1/2012 2:30am" }
\::16172910
1001
1001
Friday, October 11, 13
INSERT INTO Users(name, facebook_id, email, created_at)VALUES ("Jasdeep Jaitla", 16172910, "[email protected]", "5/1/2012 2:30am")
Get User By FBSELECT * FROM UsersWHERE facebook_id = 16172910
Get User by EmailSELECT * FROM UsersWHERE email = “[email protected]”
user_data = { "name": "Jasdeep Jaitla", "facebook_id": 16172910, "email": "[email protected]", "created_at": "5/1/2012 2:30am" }
uid = couch.incr("u::count") + 1000couch.add ("u::" + uid, user_data)couch.set ("em::" + user_email, uid)couch.set ("fb::" + user_fb, uid)
Get User By FB uid = couch.get("fb::16172910")user_data = couch.get ("u::" + uid)
Get User By Emailuid = couch.get("em::[email protected]")user_data = couch.get ("u::" + uid)
Each Table Grows and it gets Slower for Each Request
RDBMS Couchbase
Friday, October 11, 13
Friday, October 11, 13
MULTIPLE DOCUMENTS
Friday, October 11, 13
Aligning Documents to Behaviors
user::1{name: “Jasdeep”,points: 1000,shopping_carts: [ 1000, 1001, 1002 ],products_bought: [ 2000, 2001, 2002],games_won: [ 3000, 3001, 3002, 3004],notifications: [ “Lorem”, “Ipsum”, “docet”, ...]}
user::1{ name: “Jasdeep” }user::1::points{ points: 1000 }user::1::shopping_carts{ carts: [ 1000, 1001, 1002 ],products_bought: [ 2000, 2001, 2002] }user::1::games_won{ game_ids: [ 3000, 3001, 3002, 3004] }user::1::notification_count57user::1::notifications::57{ message: “Hi Bob” }user::1::notifications::56{ message: “Happy Hour?” }
Friday, October 11, 13
Behavior Driven Design
• Reduce the number of User Actions that affect a single document
• Instead, separate that user document in a predictable key structure and make it accessible via getters and setters in your class
• Like TDD/BDD encourages smaller, simpler methods, that are easier to write and maintain
Friday, October 11, 13
Friday, October 11, 13
RDBMS TO COUCHBASE
Friday, October 11, 13
The Most Common Mental Adjustments #1
• In SQL we tend to want to avoid hitting the database as much as possible
• We know, intuitively, that it’s costly when tying up connection pools, and overloading the db servers
• Even with caching and indexing tricks, and massive improvements over the years, SQL still gets bogged down by complex joins and huge indexes
• In Couchbase, get’s and set’s are so fast they are trivial, and not bottlenecks, this is hard for many people to accept (reptilian SQL mind)
Friday, October 11, 13
Complex Joins vs Multiple Gets
select * from (((client left join brochure on client.cid = brochure.cid)
left join translation on brochure.bid = translation.bid )
left join version on translation.tid = version.tid)
left join language on language.lid = translation.lid
order by client.cid, brochure.bid, translation.tid, version.vid
shopping_cart_id =
couch.get(“u::1001::transaction::count”)
cart_items =
couch.get(“u::sc::” + shopping_cart_id”)
foreach item_id in cart_items.items
cart_details.push(
couch.get(“product::” + item_id) )
end
Friday, October 11, 13
The Most Common Mental Adjustments #2
• The key to finding data is the Key!
• Key design can give you many different ways to access data, being able to predict key values, and use them creatively
• Many newcomers see Views as a replacement for key design, because it seems more “SQL”-like
• Use Views for what you cannot do with Key Design, and there are lots of things you can't do with Key Design
Friday, October 11, 13
Friday, October 11, 13
Q & A
Friday, October 11, 13
Resources
Main Resource Portalwww.couchbase.com/communiHes
Code Samples Going through Opera/onswww.github.com/couchbaselabs/DeveloperDay
Couchbase Q & Awww.couchbase.com/communiHes/q-‐and-‐a
My Email: [email protected] Twiher: @scalabl3
Friday, October 11, 13
Couchbase 103: ViewsMonday October 14th, 10:00am PDT
Friday, October 11, 13