Couchbase 3.0 Beta: Ultra-high Availability an Always-On World
What's New in Couchbase Server 3.0: Couchbase Connect 2014
-
Upload
couchbase -
Category
Data & Analytics
-
view
1.137 -
download
1
Transcript of What's New in Couchbase Server 3.0: Couchbase Connect 2014
Deep Dive: What’s New in
Couchbase Server 3.0
Cihan Biyikoglu
Director of Product Management | Couchbase
Part I – Couchbase Overview
Journey so far…
Part II – Deep Dive into Couchbase Server v3.0
Developer Empowerment
Mission Critical Scale
Ultra High Availability
Secure Communication
Simplified Administration
Q&A
Agenda
Overview
©2014 Couchbase, Inc. 4
General purpose
Elastic scalability Consistent high
performance
Always
available
Flexible, global
deployment
Enterprise grade
administration
Real time big
data
Data
mobility
Developer
focused
Most complete, scalable & highest performing NoSQL database
DEVELOPER
EMPOWERMENT
MISSION CRITICAL
SCALE
ULTRA HIGH
AVAILABILITY
SECURE
COMMUNICATION
SIMPLIFIED
ADMINISTRATION
• Fully Integrated JSON
Programmability
• Native Async and
Reactive
Programmability
• New Integration with
Popular Frameworks
• N1QL Preview Support
in SDKs
• Optimized Massive
Databases with Tunable
Memory
• Auto-Tuning Thread
Pool for Extreme IO
Throughput
• Faster Replication with
DCP
• Improved Durability
• Lower Latency Indexing
On Views
• Low Latency, In-Memory
XDCR
• Delta Node Recovery
• Faster Rebalance
• Faster Node Restart with
Efficient Warm-up
• New Incremental
Backup option
• Encryption for Data
Access
• Encryption for
Administrative Access
• XDCR Pause and
Resume
• Improved Resource
Governance
• Graceful Failover
• Cluster-wide Diagnostics
What’s new in Couchbase Server v3.0
Componentized libraries for simplify use cases
• Couchbase JVM Core - Reusable Foundation
• Handles core IO, topology changes. Reactive and asynchronous.
• Java Client Library
• Native JSON handling built-in
• Provides synchronous and asynchronous interface
• Preview of N1QL language integrated query
What’s New in SDK 2.0
Integrated native JSON data type programmability
• Simplified Development
• Native JSON handling built-in
• Preview of N1QL language integrated query
• Performance & Scale
• Improved connection management
• Native .Net:
• Simplified Configuration Management with Web.config & App.config
What’s New in SDK 2.0
Node.js, PHP
• Native JSON handling built-in
• Core IO enhancements through updated C libcouchbase
• Better connection management
Python, Ruby
• Core IO enhancements through updated C libcouchbase
• Better connection management
Framework Integrations
Spring, gevent, Twisted, Ottoman
What’s New in SDK 2.0
geventfor python
OttomanCouchbase ODM framework
Optimized Memory Usage with Metadata Ejection Policy Better optimization of memory for massive databases
Enable efficient management of rarely accessed data set
Cache only keys and data for the working set & eject all historic data
Tunable Memory - Optimization for Massive Databases
Tunable Memory - Optimization for Massive Databases
100sx Reduction in Metadata Memory Consumption from 2.5 to 3.0Note: The graph represents characteristics under data mutations. ~50M docs with value size ~0.5KB
v.3.0 – Large DB with Hot Working-set
3 GB Consumed for Metadata in RAM
2.5.1 or earlier – Real-time Latency
80 MB Consumed for Metadata in RAM
Efficient Auto-Tuning Engine Detect and allocate threads based on HW resources
Pool threads for best resource utilization
Improved latency across the board:
Faster Reads
Faster Rebalance
Faster Node reactivation
Faster Durability with Writes & PersistTo (more details on this on later)
…and many more
Auto Tuning Thread Pool
Faster Durability with Persistence
Improved latency on PersistTo
Shared Thread Pool Improves IO speed
Up to 3x better PersistTo latency from 2.5 to 3.0*Note: Absolute latency depend on disk IO subsystem throughput
*95th percentile PersistTo=1 latency (ms), 1 bucket x 200M x 1KB, 250 mutations/sec/node
Auto Tuning Thread Pool - Faster Durability
0
1000
2000
3000
4000
5000
Tim
e (
ms
)
95 Percentile PersistTo=1 Latency*
2.5.1 3.0
New DCP (Database Change Protocol) protocol for replication Eliminate Disk-IO from the replication access path
High-Performance, Stream-based, In-memory
Ordered based on Sequence Numbers
Snapshot-able
Restart-able
• Powers Node to Node Replication
Replication and Rebalance
Power Delta Node Recovery – more on this later!
Faster Durability with ReplicateTo – more on this later!
• Powers View – more on this later!
• Powers XDCR – more on this later!
Database Change Protocol
Faster Durability with Replication
Improved latency on ReplicateTo
DCP Improves replication speed and durability
Better protects your data!
Up to 150x Improvement on ReplicateTo latency from 2.5 to 3.0*Note: Absolute latency depend on network bandwidth and latency
*95th percentile ReplicateTo=1 latency (ms), 1 bucket x 200M x 1KB, 250 mutations/sec/node
Database Change Protocol - Faster Replication with DCP
0
200
400
600
Tim
e (
ms
)
95 Percentile ReplicateTo=1 Latency*
2.5.1 3.0
More up-to-date views Mutation show up at the indexes
faster with the new DCP protocol
No longer require persisting to disk for change detection
Improved consistency with views (stale=false behavior)
Ports to C in view engine
Database Change Protocol - Fresher Indexes
0
10000
20000
30000
40000
Tim
e (
ms
)
Indexing Under Load
2.5.1 3.0
Up to 50x Faster Indexing from 2.5 to 3.0*Note: Absolute latency depend on disk IO speed
*95th percentile Indexing latency (ms), 1 bucket x 20M x 2KB, non-DGM, 1 view, 250 mutations/sec/node, 400 queries/sec
Improved Protections for Your Data
Drastically Lower XDCR Latency
In memory change detection with DCP
No longer require persisting to disk
Up to 4x better on XDCR latency between clusters between 3.0 & 2.5.1*Note: Absolute latency depend on WAN latency and bandwidth
Lower Latency XDCR (Cross Data Center Replication)
0
100
200
300
400
500
600
Tim
e (
ms
)
90th Percentile. XDCR Latency
2.5.1 3.0
90th percentile replication lag (ms), 5 -> 5 UniDir, 2 buckets x 500M x 1KB, 10K SETs/sec, LAN
Improved Rebalance Latency
Efficient Data Movement
Better Resume-ability after blips and failures
Incremental catch-up
Up to 2x Faster Rebalance under Load between 3.0 and 2.5.1
*Note: Absolute latency depend on network and disk IO latency and bandwidth
Faster Rebalance
0
10
20
30
Tim
e (
ms)
Rebalance Latency to Under Load*
2.5.1 3.0
*Rebalance-out (min), 4 -> 3, 1 bucket x 100M x 2KB, 10K ops/sec
Delta Node Recovery
Shorter time to add nodes back into the cluster!
Bring back a server incrementally into the cluster without having to rebuild the full node.
Incremental Node Catch-up with Delta Node Recovery
100sx Reduction in Time to Re-Add Node from 2.5 to 3.0Note: The absolute performance improvement depend on data size and mutation count that needs to be caught up
Efficient warm-up
Parallelization on cache warm-up
Further warm-up time improvements under metadata ejection
Up to 60% Faster Warm-up between 3.0 and 2.5.1
Up to 8x Faster Warm-up with Full Ejection in 3.0
*Note: Absolute latency depend on disk IO latency and bandwidth
Faster Node Re-Activation
Minimize time and resources during backups
Efficient Recovery with Incremental Backup & Restore
• Back up only the data updated
since the last backup
• Differential Backups
• Cumulative Backups
Chris Bertsch
Systems Engineer | Factset Research Systems
FactSet provides financial software for research analysts, portfolio managers and investment backers.
Factset – Chris
©2014 Couchbase, Inc. 28
Couchbase at FactSet
• Total Capacity in Production so far…
1920 cores, 30TB of RAM with Bare Metal Nodes
XDCR with Bidirectional across 2 Data Centers
Exciting Couchbase 3.0 Features
• Incremental Backup & Restore
• Delta-node Recovery
• Stream-based XDCR
Factset – Chris
©2014 Couchbase, Inc. 29
Encrypted Admin & Data Access
Data Access Encryption
SSL based Client-server Communications
SSL based View Access
https://couchbase_server:18092/…
Admin Access Encryption – REST, CLI and HTTPS
https://couchbase_server:18091/…
Encrypted Client-Server Communication
SERVER 3SERVER 1 SERVER 2
Couchbase Server
Maintenance - XDCR Pause and Resume
Want to suspend replication? No longer need to drop replications
Ability to pause XDCR traffic anytime
Resume and incrementally catch-up from where you left off
Efficient Maintenance with XDCR
Priority Setting on Buckets
Ability to assign high and low priority to buckets
High priority buckets get higher IO priority and more throughput
Low priority buckets can still be processed without starvation
Improved Resource Governance with Bucket Priority
New option to gracefully failover a node.
Failover with a graceful handoff
May take longer to failover but prevents potential for any data loss
Node Maintenance
Collect cluster-wide diagnostics with single-click
Collect logs from all server or individual specific servers
Upload logs to shareable storage for Couchbase Support team to analyze
Cluster-wide Diagnostics
Cluster-wide Diagnostics Tool
Improved Crash Reporting
Better Serviceability
Improved Monitoring for Warm-up
Stats Enhancements
Improved Checkpointing with XDCR
Expended Options with Couchbase-cli Parallelized Warm-up
Database Change Protocol (DCP)
Increased Connection LimitsExtended Documentation
Enhanced Event Logging
Extended XDCR Resiliency
Better failover resiliency with DCP
Enhanced SSD Performance
Streamlined Build
Side by side support for DCP and TAP
Improved Error Reporting for Apps
Built-in OS Tuning for Linux Flavors
… and More
Improved Resume-ability with Intra Cluster Replication
View Engine rewrite in C
Web.Config & App.Config support with .NET in SDK 2.0
Unified New App Model with 2.0 SDK
Java SDK 2.0 built on top of RxJava
N1QL Preview Support in the 2.0 SDKs
Faster Warm-up time under Metadata Ejection
Access log for monitoring port 8091
CRAM & MD5 Support in .Net SDK
Client Side Log4Net Integration in .Net SDK
New Cluster object for Cluster Operations in SDK 2.0
Replica Read in SDK 2.0 with .Net and PHP
Added Support for Debian v7
Faster View indexing
Faster XDCR Synchronization
Faster ReplicateTo
Faster PersistTo
Faster Rebalance
Encrypted Data Access
Encrypted Admin Access
Encrypted View Access
Graceful Failover
Delta Node Recovery
Incremental Backup and Restore
Lower Latency XDCR
Auto tuning IO subsystem
Additional Configuration on .Net pooling interval
New repository for Yum for Ubuntu
Community Edition of 3.0 Release with Enterprise Edition
Ability to Name Clusters
DEVELOPER
EMPOWERMENT
MISSION CRITICAL
SCALE
ULTRA HIGH
AVAILABILITY
SECURE
COMMUNICATION
SIMPLIFIED
ADMINISTRATION
• Fully Integrated JSON
Programmability
• Native Async and
Reactive
Programmability
• New Integration with
Popular Frameworks
• N1QL Preview Support
in SDKs
• Optimized Massive
Databases with Tunable
Memory
• Auto-Tuning Thread
Pool for Extreme IO
Throughput
• Faster Replication with
DCP
• Improved Durability
• Lower Latency Indexing
On Views
• Low Latency, In-Memory
XDCR
• Delta Node Recovery
• Faster Rebalance
• Faster Node Restart with
Efficient Warm-up
• New Incremental
Backup option
• Encryption for Data
Access
• Encryption for
Administrative Access
• XDCR Pause and
Resume
• Improved Resource
Governance
• Graceful Failover
• Cluster-wide Diagnostics
RECAP
What’s new in Couchbase Server v3.0
Today• Deep Dive: Tunable Memory in Couchbase Server 3.0 – Chiyoung SeoOctober 6 @ 4:20 pm - 5:05 pm
• Best Practices: Securing a Couchbase Server Deployment – Don PintoOctober 6 @ 4:20 pm - 5:05 pm
• Deep Dive into DCP: A Streaming Replication Protocol – Mike Wiederhold and meOctober 6 @ 5:10 pm - 5:55 pm
Tomorrow• Ultra–High Availability and Disaster Recovery with Couchbase Server – Anil KumarOctober 7 @ 10:50 am - 11:35 am
• Deep Dive: Near Real-Time Map / Reduce with Views in Couchbase Server 3.0 - SarathLakshman
October 7 @ 11:40 am - 12:25 pm
Related Talks