CouchConf Israel Couchbase in Production 24x7
-
Upload
couchbase -
Category
Technology
-
view
533 -
download
1
Transcript of CouchConf Israel Couchbase in Production 24x7
1
Couchbase Server 2.0 in Production
2
Couchbase Server 2.0: Overview
• Membase + CouchDB• Managed memory caching layer = super high performance• Clustering and online data redistribution (Rebalancing)• Indexing and Querying via JSON Map-Reduce• New SDK’s and client libraries
• Developer Preview available at:• http://www.couchbase.com/downloads
=Simple. Fast. Elastic.
3
www.facebook.com/farm_town_wars
Web Servers
Load Balancer
Couchbase Servers
Let’s build a social game…
4
COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE
Initial Setup
View Development Sizing
Client setup
Monitor Grow
Backup/Restore
Upgrade
Deploy
MaintainFailures
Test
5
Couchbase Server 2.0 in production: Initial Setup
Extremely easy to get up and running:
• RPM/deb/OSX/exe installation
• Simple Web UI and setup wizard
6
Data and Indexes
• Data goes “in and out” via the Memcached protocol
• Queries/indexes are created and accessed via an HTTP protocol
• Both are available separately, Couchbase-provided SDK’s will expose a single API to the developer and abstract the actual traffic
7
Data and Indexes
Example JSON Document:
{
"_id": "Keith4540",
"jsonType": "player",
"name": "Keith4540",
"level": 4,
…
…
}
Example view:
function (doc) {
if (doc.jsonType == "player"){
emit(["Level", doc.level], doc._id);}
}
+
=Secondary index of players, by level
(Demo)
8
Data and Indexes
• Indexes/views are based on incremental map-reduce:• Indexes are updated with incremental changes (not
batch)
• View processing is done per-vbucket:• Parallel processing on subset of data• Couchbase provides scatter-gather aggregation
9
Data and Indexes
• Views are “developed” off of a random (or specific) sampling of the overall dataset and then deployed• Faster• Less load on system
• Updated views can be applied without rescanning entire dataset
10
Couchbase Client SDKs
Java Client SDK
.Net SDK
PHP SDK
Ruby SDK
Python SDK
Spy memcachedConnection
HTTP couchDBconnection
Java client API
User Code
Couchbase Server
CouchbaseClient cb = new CouchbaseClient(listURIs,"aBucket", "letmein");// this is all the same as beforecb.set("hello", 0, "world");cb.get("hello");Map<String, Object> manyThings =cb.getBulk(Collection<String> keys);/* accessing a view View view = cb.getView("design_document", "my_view");Query query = new Query();query.getRange("abegin", "theend");
11
COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE
Sizing
Client setup
Deploy
12
Couchbase Server 2.0 in production: Sizing
Sizing Question: How many nodes do I need?
Considerations:RAMDiskNetworkData Safety
13
Sizing
Sizing Question: How many nodes do I need?
Considerations:RAM• Metadata• Working set• Buffer/overhead
Active+Replica
14
Sizing
500,000 Documents to begin with:-20 bytes in length-average Document size of 2k-metadata of about 150 bytes per key
~1Gb to store active data, an extra 1Gb to store replica data
Adding in some headroom: Give 3GB RAM to Couchbase to start and grow with
=
15
Sizing
Sizing Question: How many nodes do I need?
Considerations:Disk:• Sustained write rate• Index generation (space and IO)• Append-only B-Tree• Compaction• Rebalance capacity
16
Sizing
Sizing Question: How many nodes do I need?
Considerations:Network:• Client traffic• Replication• Rebalancing
17
Sizing
Sizing Question: How many nodes do I need?
Considerations:Data Safety
1 node = single point of failure (bad)2 nodes = 1 replica copy (better)3+ nodes = 1 replica copy AND data/load distribution
18
Sizing
Sizing Question: How many nodes do I need?
Considerations:RAMDisk NetworkData Safety
19
Farm Town Wars App Code
Ap
plic
atio
n s
erv
er
Co
uch
ba
se
Se
rve
r
Couchbase JavaClient library
Query API
Couchbase
11210
(“smart”) library
Farm Town Wars App Code
Ap
plic
atio
n s
erv
er
Co
uch
ba
se
Se
rve
r
Couchbase PHP Client Library
Moxi (Couchbase proxy)
Query API
Couchbase
11210
Client-side Moxi
OR80928092
Client-side Deployment
20
COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE
Monitor Grow
Backup/Restore
UpgradeMaintainFailures
21
Monitoring
IMMENSE amount of information available
-Real-time traffic graphs
-REST API accessible
-Per bucket, per node and aggregate statistics
-Application and inter-node traffic
-RAM <-> Disk
-Inter-system timing
22
23
Growth
Going from 1 hundred users to 2 million users…
– RAM usage is growing:• Ejecting data to and fetching data from disk
• Resident item ratios decreasing – might impact failover
• Cache miss ratio increases
– Disk write queue grows higher than usual
Need to add a few more nodes...
Now we have more RAM and more disk throughput without any downtime
(Demo)
24
General Maintenance
• Persistence is using CouchDB technology:– Append-only B-tree
– Reliability and data integrity
– Constantly growing on-disk files
• Compaction is the answer:– Automatic
– “Fragmentation” data
– Scheduled compaction
– Partial compaction via per-vbucket database (saves disk space and time)
25
• Backup/Restore:– Backup is as simple as a file-level copy (thanks CouchDB!)
– Server will automatically “warmup” from disk files upon reboot
• Upgrade:– Add nodes of new version, rebalance…
– Remove nodes of old version, rebalance…
– Done!
– No disruption
– Upgrade from existing Membase 1.7 installations to
Couchbase 2.0
General Maintenance
26
Failures
• Failures happen:
– Hardware
– Network– Bugs
• Failover to replica data for immediate access
• Remove and rebalance “malfunctioning” node
(Demo)
27
COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE
Initial Setup
View Development Sizing
Client setup
Monitor Grow
Backup/Restore
Upgrade
Deploy
MaintainFailures
Test
28
THANK YOU!
Q&A