Post on 11-Aug-2015
Application Renovation with Couchbase Clint Ricker
VSRM Intro
VSRM = Video Session & Resource Manager
Enables broadcast, on demand, and hosted DVR Video
Background scenario
Changing industry
Changing ecosystem
Sold as a 1U appliance
VSRM (Before Couchbase)
Scalability via segmentation (1 server per 100K subscribers)
No centralized management
VSR<
Platform Components
Applications
Application Protocol Adaptors
SDV Policy Mgr
S6VERM
SSP
Resource Manager
Resource Manager Adaptors
CMTS QAMBulk
Encryptor
Elem
ent Ma
nager (proprietary local
database)
VSRM (Before Couchbase)
MCMIS
VOD
Log Manager
Config Manager
HA Manager
VSRM
Clients (externally sharded to
~100K)
QAMQAMQAMQAMQAM
Mgmt
DB
VSRM
Clients (externally sharded to
~100K)
QAMQAMQAMQAMQAM
Mgmt
DB
VSRM
Clients (externally sharded to
~100K)
QAMQAMQAMQAMQAM
Mgmt
DB
VSRM
Clients (externally sharded to
~100K)
QAMQAMQAMQAMQAM
Mgmt
DB
VSRM
Clients (externally sharded to
~100K)
QAMQAMQAMQAMQAM
Mgmt
DB
VSRM Scalability (Before Couchbase)
Phase 1: Assessment
Strong Underlying Design
Pluggable components
Core components plus adaptors
Model based applications
Deep customer entrenchment
Large scale customer deployments
Deep customer integration and customization
Rich algorithms and business logic
Complex algorithms
Rich support for customer policy rules
Why not?
Bad Design
Easy to reimplement algorithms and business logic
Small customer entrenchment
High technical debt
Phase 2: Renovation
VSRM VSRM VSRM VSRM VSRM
DB DB DB DB DB
Concept
Distributed database / shared memory
Why Couchbase?
Strong consistency
Reliable low latency
Great scalability
VSR<
Platform Components
Applications
Application Protocol Adaptors
SDV Policy Mgr
S6VERM
SSP
Resource Manager
Resource Manager Adaptors
CMTS QAMBulk
Encryptor
Elem
ent Ma
nager (proprietary local
database)
VSRM
MCMIS
VOD
Log Manager
Config Manager
HA Manager
Couchbase Adaptor
DB Manager
State
Data Former Became
Per session data (session state)
In memory C struct and legacy local database
Couchbase keys (now text based)
Component data model (config / application state)
Legacy local database 1 couchbase key per tableSession data typically peeled offCounters are individual keys
Raw memory C struct Couchbase keys using memcpy on C/C++.Other APIs transparently handle objects.
Data Model Transformation
Data Sizing
Smaller keys have lower latency
Max rate of consistent update of a key–higher for small keys
Transactions needed for consistent updates of groups of keys
Design Rules
Real time operations use key/value operations
Views reserved for maintenance operations
All database calls go through abstraction layer
Locking / Serialization
Pessimistic locks as a replacement for mutex
Provides serialization, not order
Optimistic locks for read efficiency
Queries
Be intentional on what data is indexed
Weaker consistency
Limited resource, especially if stale=true
Data Model TransformationCouchbase = clustering Swiss Army knife
Lightweight directory services
Simple distributed queues with append
Master election via get w/lock
Danger Zones
Consider key contention. Break apart keys if necessary. Redesign to avoid.
Distributed databases require robust error handling.
Test error handling and edge cases at scale.
Phase 3: Makeover Grand Reveal
VSR<
Platform Components
Applications
Application Protocol Adaptors
SDV Policy Mgr
S6VERM
SSP
Resource Manager
Resource Manager Adaptors
CMTS QAMBulk
Encryptor
Elem
ent Ma
nager (proprietary local
database)
VSRM (Before Couchbase)
MCMIS
VOD
Log Manager
Config Manager
HA Manager
VSR<
Platform Components
Couchbase Adaptor
Applications
Application Protocol Adaptors
SDV Policy Mgr
S6VERM
SSP
Resource Manager
Resource Manager Adaptors
CMTS QAMBulk
Encryptor
Elem
ent Ma
nager (structure only)
VSRM Today
MCMIS
VOD
Log Manager
Config Manager
HA Manager
Web Svc Manager
Recorder Mgr
Linear Mgr
C3 A8RM-IS
RM-SS
RecorderVideo Mux
DB Manager
VSRM
Clients (externally sharded to
~100K)
QAMQAMQAMQAMQAM
Mgmt
DB
VSRM
Clients (externally sharded to
~100K)
QAMQAMQAMQAMQAM
Mgmt
DB
VSRM
Clients (externally sharded to
~100K)
QAMQAMQAMQAMQAM
Mgmt
DB
VSRM
Clients (externally sharded to
~100K)
QAMQAMQAMQAMQAM
Mgmt
DB
VSRM
Clients (externally sharded to
~100K)
QAMQAMQAMQAMQAM
Mgmt
DB
VSRM Scalability Before Couchbase
VSR<VSRMVM
Instance
VSRM Scalability Today (unified deployment model)
QAMQAMQAMQAMQAM
QAMQAMQAMQAMQAM
QAMQAMQAMQAMQAMQAMQAMQAMQAMQAM
QAMQAMQAMQAMQAM
QAMQAMQAMQAMQAM
QAMQAMQAMQAMQAM
QAMQAMQAMQAMQAM
QAMQAMQAMQAMQAM
Clients
VSRMVM
Instance
VSRMVM
Instance
VSRMVM
Instance
VSRMVM
Instance
Management
Load Balancer
Load Balancer
CouchbaseClusterCouchbase
ClusterCouchbaseCluster
Provisioning
Public or P
rivate C
loud
VSR<
VSRM Scalability Today (per service deployment model)
QAMQAMQAMQAMQAM
QAMQAMQAMQAMQAM
QAMQAMQAMQAMQAMQAMQAMQAMQAMQAM
QAMQAMQAMQAMQAM
QAMQAMQAMQAMQAM
QAMQAMQAMQAMQAM
QAMQAMQAMQAMQAM
QAMQAMQAMQAMQAM
Clients
App Micro Service
App Service
Load Balancer
Load Balancer
CouchbaseClusterCouchbase
ClusterCouchbaseCluster
Provisioning
Public or P
rivate C
loud
Resource Mgr
Service
Resource Mgr
Service
Resource Mgr
Service
App AdaptorsServices
Resource Mgr
Service
Resource AdaptorServices
Management
The End