CouchConf London Developing with Couchbase I: Getting Started
CouchConf-Bangalore-Couchbase 2.0-in-production
-
Upload
couchbase -
Category
Technology
-
view
455 -
download
0
Transcript of CouchConf-Bangalore-Couchbase 2.0-in-production
![Page 1: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/1.jpg)
1
Couchbase Server 2.0 in Production
Frank Weigel, Director Product Management, Couchbase Inc.
![Page 2: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/2.jpg)
2
Couchbase Server 2.0: Overview
• Membase + CouchDB• Managed memory caching layer = super high performance• Clustering and online data redistribution (Rebalancing)• Indexing and Querying via JSON Map-Reduce• New SDK’s and client libraries
• Developer Preview available today!• http://www.couchbase.com/downloads
=Simple. Fast. Elastic.
![Page 3: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/3.jpg)
3
www.facebook.com/farm_town_wars
Web Servers
Load Balancer
Couchbase Servers
Let’s build a social game…
![Page 4: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/4.jpg)
4
COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE
Initial Setup
View Development Sizing
Client setup
Monitor Grow
Backup/Restore
Upgrade
Deploy
MaintainFailures
Test
![Page 5: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/5.jpg)
5
Couchbase Server 2.0 in Production: Initial Setup
Extremely easy to get up and running:• RPM/deb/OSX/exe installation• Simple Web UI and setup wizard
![Page 6: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/6.jpg)
6
Couchbase Server 2.0 in Production: Data and Indexes
• Data goes “in and out” via the Memcached protocol
• Queries/indexes are created and accessed via an HTTP protocol
• Couchbase-provided SDK’s expose a single API to the developer and abstract the actual traffic
![Page 7: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/7.jpg)
7
Farm Town Wars App Code
Applic
ati
on s
erv
er
Couch
base
Serv
er
Coucbase Javaclient library
Query API
Couchbase
11210
(“smart”) library
Farm Town Wars App Code
Applic
ati
on s
erv
er
Couch
base
Serv
er
Couchbase PHP Client LIbrary
Moxi (Couchbase proxy)
Query API
Couchbase
11210
Client-side Moxi
OR59845984
Couchbase Server 2.0 in production: Client-side Deployment
![Page 8: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/8.jpg)
8
Couchbase Server 2.0 in Production: Data and Indexes
Example JSON values:
User_id:{ “farms”: [“one_farm”, “two_farm”] “animals”: [{ “pig”:”pink”, “sheep”:”black”, “chicken”:”yellow” }] “enemies”: [“you”, “U2”]}
Example view:
function (doc) { for(var i in doc.animals) if(doc.animals[i].sheep=="black") emit(doc._id)}+
=List of black sheep owners
![Page 9: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/9.jpg)
9
Couchbase Server 2.0 in Production: Data and Indexes
• Indexes/views are based on incremental map-reduce:• Indexes are updated with incremental changes (not
batch)
• View processing is done per-node:• Parallel processing on subset of data• Couchbase provides scatter-gather aggregation
![Page 10: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/10.jpg)
10
Couchbase Server 2.0 in Production: Data and Indexes
• Views are “developed” off of a random (or specific) sampling of the overall dataset and then deployed• Faster• Less load on system
• Updated views can be applied without rescanning entire dataset
![Page 11: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/11.jpg)
11
COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE
Sizing
Client setup
Deploy
![Page 12: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/12.jpg)
12
Couchbase Server 2.0 in production: Sizing
Sizing Question: How many nodes do I need?
Considerations:RAMDiskNetworkData Safety
![Page 13: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/13.jpg)
13
Couchbase Server 2.0 in production: Sizing
Sizing Question: How many nodes do I need?
Considerations:RAM• Metadata• Working set• Buffer/overhead
Active+Replica
![Page 14: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/14.jpg)
14
Couchbase Server 2.0 in production: Sizing
500,000 keys to begin with: -20 bytes in length -average value size of 2k -metadata of about 150 bytes per key
~1Gb to store active data, an extra 1Gb to store replica data
Adding in some headroom: Give 3GB RAM to Couchbase to start and grow with
=
![Page 15: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/15.jpg)
15
Couchbase Server 2.0 in production: Sizing
Sizing Question: How many nodes do I need?
Considerations:Disk:• Sustained write rate• Index generation (space and IO)• Append-only B-Tree
![Page 16: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/16.jpg)
16
Couchbase Server 2.0 in production: Sizing
Sizing Question: How many nodes do I need?
Considerations:Network:• Client traffic• Replication• Rebalancing
![Page 17: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/17.jpg)
17
Couchbase Server 2.0 in production: Sizing
Sizing Question: How many nodes do I need?
Considerations:Data Safety
1 node = single point of failure (bad)2 nodes = 1 replica copy (better)3+ nodes = 1 replica copy AND data/load distribution
![Page 18: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/18.jpg)
18
Couchbase Server 2.0 in production: Sizing
Sizing Question: How many nodes do I need?
Considerations:RAMDisk NetworkData Safety
![Page 19: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/19.jpg)
19
COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE
Monitor Grow
Backup/Restore
UpgradeMaintainFailures
![Page 20: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/20.jpg)
20
Couchbase Server 2.0 in production: Monitoring
IMMENSE amount of information available
-Real-time traffic graphs-REST API accessible-Per bucket, per node and aggregate statistics-Application and inter-node traffic-RAM <-> Disk-Inter-system timing
![Page 21: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/21.jpg)
21
![Page 22: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/22.jpg)
22
Couchbase Server 2.0 in production: Growth
• Going from 1 hundred users to 2 million users…– RAM usage is growing:
• Ejecting data to and fetching data from disk• Resident item ratios decreasing – might impact failover
– Disk write queue grows higher than usual
• Need to add a few more nodes...• Now we have more RAM and more disk throughput
without any downtime
![Page 23: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/23.jpg)
23
Couchbase Server 2.0 in production: General Maintenance
• Persistence is CouchDB:– Append-only B-tree– Reliability and data integrity– Constantly growing on-disk files
• Compaction is the answer:– Automatic– “Fragmentation” data– Scheduled compaction– Per-vbucket database (saves disk space and time)
![Page 24: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/24.jpg)
24
• Backup/Restore:– Backup is as simple as a file-level copy (thanks CouchDB!)– Server will automatically “warmup” from disk files upon reboot
• Upgrade:– Add nodes of new version, rebalance…– Remove nodes of old version, rebalance…– Done!– No disruption
Couchbase Server 2.0 in production: General Maintenance
![Page 25: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/25.jpg)
25
Couchbase Server 2.0 in production: Failures
• Failures happen:– Hardware– Network– Bugs
• Failover to replica data for immediate access
• Remove and rebalance “malfunctioning” node
(Demo)
![Page 26: CouchConf-Bangalore-Couchbase 2.0-in-production](https://reader030.fdocuments.in/reader030/viewer/2022032514/55d56019bb61eb28038b4626/html5/thumbnails/26.jpg)
26
COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE
Initial Setup
View Development Sizing
Client setup
Monitor Grow
Backup/Restore
Upgrade
Deploy
MaintainFailures
Test