SD Times - Docker v2
-
Upload
alvin-richards -
Category
Documents
-
view
73 -
download
10
Transcript of SD Times - Docker v2
![Page 1: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/1.jpg)
Databases & Containers: From Development to Deployment Powering New Opportunities at Scale
![Page 2: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/2.jpg)
2 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ Containers + Databases = Happy Developers ■ Ephemeral Containers + Databases = DevOps headaches ■ Data Redundancy
■ Need more than one copy of the Data
■ Database Self Discovery & Cluster formation ■ Manual / Scripted configuration of Database Cluster
■ Database Self Healing (as containers enter and leave) ■ Manual / Scripted re-configuration of Database Cluster
■ Application Tier discovery of Database ■ Manual / Scripted re-routing of Application Traffic
Conclusion
![Page 3: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/3.jpg)
3 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Existing Architectures Are Broken
Challenges • Complex
• Maintainability
• Durability
• Consistency
• Scalability
• Cost ($)
• Data Lag
Caching Layer
Operational Database
Real-time Consumer Facing
Pricing / Inventory / Billing
Real-time Decisionin
g
Streaming Data
Legacy Database (Mainframe)
RDBMS Database
Transactional Systems
Enterprise Environment
Legacy RDBMS
HDFS BASED
![Page 4: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/4.jpg)
4 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Existing Deployment Models Are Broken
4
Developer
Version control
1. Development 2. Test 3. Stage / Production
QA / QE Sysadmin
![Page 5: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/5.jpg)
5 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Infrastructure Cannot be Fully Utilized TRADITIONAL DATABASE
OS FILE SYSTEM
PAGE CACHE
BLOCK INTERFACE
SSD HDD
What You Have
BLOCK INTERFACE
SSD SSD
OPEN NVM
SSD
DATABASE
MEMORY SYSTEM
• Direct device access • Large Block Writes • Indexes in DRAM • Highly Parallelized
What You Want
![Page 6: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/6.jpg)
6 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ Containers ■ Encapsulation ■ Deployment ■ Isolation
■ Databases ■ Persistence ■ Scalability ■ Self-organizing / Self-healing ■ Resource Utilization
Meeting the Challenges
![Page 7: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/7.jpg)
Containers
![Page 8: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/8.jpg)
14 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Container Mission – Reduce Complexity
Build Ship Run
OpenStandards
Plumbing
Pla3ormClustering Distribution
Image spec Container run-time spec
Runtime
Trust
![Page 9: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/9.jpg)
15 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ Encapsulation of Dependencies ■ O/S packages & Patches ■ Execution environment (e.g. Python 2.7) ■ Application Code & Dependencies
■ Process Isolation ■ Isolate the process from anything else running
■ Faster, Lightweight virtualization
What do Containers give me?
![Page 10: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/10.jpg)
16 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
FROM python:2.7
ADD . /code
WORKDIR /code
RUN apt-get update
RUN apt-get -y install python-dev
RUN apt-get -y install libssl-dev
RUN pip install --no-cache-dir -r requirements.txt
EXPOSE 5000
CMD python app.py
Dockerfile - Example
![Page 11: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/11.jpg)
17 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Coalition of industry leaders join forces to eliminate fragmentation • Form a vendor-neutral, open source governance model under the
Linux Foundation • Establish common standards for container format and runtime • Docker donated its container format, runtime and associated
specifications • Appoint maintainers for the libcontainer project
Open Container Initiative (OCI) – Polyglot Vendors
http://www.opencontainers.org/
![Page 12: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/12.jpg)
20 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Docker Landscape in Pictures
Machine provisions Docker Engines
Engines are clustered by Docker Swarm
Compose orchestrates Container deployment
Containers are run by Docker Engine
Docker Machine Docker Compose
Docker Swarm
Docker Engine
Container
Containers encapsulates your code, dependencies…
![Page 13: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/13.jpg)
Aerospike The Enterprise NoSQL Database
![Page 14: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/14.jpg)
22 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
The Bottom Line
![Page 15: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/15.jpg)
23 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
• RIPE MD160 Hashing & Partitioning • Primary Index is Red Black Tree • Dynamically Add/Remove Nodes
Scale
• Auto Rebalance • Auto Heal • Schema-free
![Page 16: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/16.jpg)
24 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
0
100,000
200,000
300,000
400,000
Balanced Read-Heavy Aerospike Cassandra
MongoDB Couchbase 2.0*
0
2.5
5
7.5
10
0 50,000 100,000 150,000 200,000 Aver
age
Late
ncy,
m
s
Throughput, ops/sec
Balanced Workload Read Latency
Aerospike Cassandra MongoDB
0
4
8
12
16
0 50,000 100,000 150,000 200,000 Aver
age
Late
ncy,
m
s
Throughput, ops/sec
Balanced Workload Update Latency
Aerospike Cassandra MongoDB
• Flash / SSD Optimized • Primary Index in DRAM
Speed
![Page 17: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/17.jpg)
25 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Challenge • Overall SLA 750 ms • Loss of business due to latency • Every credit card transaction requires hundreds of DB reads/writes
Need to Scale Reliably • 10 à 100 TB • 10B à 100 B objects • 200k à I Million+ TPS
Aerospike In-Memory NoSQL • Built for Flash • Predictable low latency at high throughput • Immediate consistency, no data loss • Cross data center (XDR) support • 20 server cluster • Dell 730xd w/ 4NVMe SSDs
Credit Card Processing System
Fraud Detection & Protection App
Rules Rule 1 Rule 2 Rule 3
Historical Data
Rule 1-Passed Rule 2-Passed Rule 3-Failed
Account Behavior
Static Data
Account Statistics
Real-time Fraud Prevention
![Page 18: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/18.jpg)
Databases and Containers
![Page 19: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/19.jpg)
27 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ Data Redundancy ■ Containers are Ephemeral – Need more than one copy of the data
■ Dynamic Self Discovery & Cluster formation ■ Need to start and stop Conatiners when needed ■ Clusters needs to grow and shrink dynamcially
■ Self Healing ■ Loss of nodes must not be fatal to the cluster integrity ■ Addition of nodes must scale capacity
■ Application Tier discovery of Database ■ Automatic discovery of nodes ■ Automatic routing of requests to the correct nodes
Requirements
![Page 20: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/20.jpg)
28 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ Data Redundancy ■ Automatic Replication of Data to "n" nodes
■ Dynamic Self Discovery & Cluster Formation ■ Shared nothing architecture – all nodes equal ■ Multi-cast & Mesh Networking models
■ Automatic healing & rebalancing of the cluster ■ Automatic hashing of keys across the cluster ■ RIPEMD-160 collision free algorithm with Smart Partitions™
■ Application Tier discovery of Database ■ Automated cluster discovery with Smart Client™ ■ Java, C/C++, C#, Python, Node.js
Example: Aerospike and Containers
![Page 21: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/21.jpg)
Demo
![Page 22: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/22.jpg)
30 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ Build & Run an App in Development ■ Python + Aerospike
■ Deploy to a Swarm cluster in Production ■ Add more Web containers behind HAProxy
■ Scale Aerospike Cluster in production ■ Add more Database nodes
Demo: Development through to Production
![Page 23: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/23.jpg)
31 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Lets build an App!
web
Aerospike
Development
python / flask
![Page 24: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/24.jpg)
32 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Scale in Production
web2 web3 web4 webN web web1
Aerospike
Development
…
Production
HA Proxy
asd1 asdN asd2 …
![Page 25: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/25.jpg)
33 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Demo 1 : Build an App
![Page 26: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/26.jpg)
34 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
FROM python:2.7
ADD . /code
WORKDIR /code
RUN apt-get update
RUN apt-get -y install python-dev
RUN apt-get -y install libssl-dev
RUN pip install --no-cache-dir -r requirements.txt
EXPOSE 5000
CMD python app.py
Dockerfile
![Page 27: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/27.jpg)
35 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
web: build: . ports: - "5000:5000" links: - aerospike hostname: dev.awesome-counter.com environment: - AEROSPIKE_HOST=dev_aerospike_1 aerospike: image: aerospike/aerospike-server:latest volumes: - $PWD:/etc/aerospike
docker-compose.yml
![Page 28: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/28.jpg)
36 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Roll the App to Production behind HA Proxy
web web1
Development Production
Aerospike
HA Proxy
Aerospike
![Page 29: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/29.jpg)
37 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Scale the web tier
web2 web3 web4 webN web web1
Aerospike
Development
…
Production
Aerospike
HA Proxy
![Page 30: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/30.jpg)
38 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Docker Networking
![Page 31: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/31.jpg)
39 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Demo 2 : Scale the Web Tier
![Page 32: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/32.jpg)
40 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
discovery: image: aerospike/interlock:latest environment: - "DOCKER_HOST" volumes: - "/var/lib/boot2docker:/etc/docker" command: "... --plugin aerospike start" aerospike: image: aerospike/aerospike-server:latest volumes: - "$PWD:/etc/aerospike"
aes_base_cluster.yml
![Page 33: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/33.jpg)
41 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
haproxy: extends: file: haproxy.yml service: haproxy-server environment: - "constraint:node==swarm-0" net: bridge web: image: alvinr/demo-webapp-as:latest extends: file: haproxy.yml service: haproxy-app environment: - AEROSPIKE_HOST=prod_aerospike_1 net: prod
docker-compose.yml
aerospike:
extends:
file: aes_base_cluster.yml
service: aerospike
image: aerospike/aerospike-server:3.7.1
labels:
- "com.aerospike.cluster=awesome-counter"
environment:
- "affinity:com.aerospike.cluster!=awesome-counter"
net: prod
![Page 34: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/34.jpg)
42 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Scale the Aerospike cluster
42
web2 web3 web4 webN web web1
mongodb
Development
…
Production
HA Proxy
asd1 asdN asd2 …
![Page 35: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/35.jpg)
43 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Demo 3 : Scale the Cluster
![Page 36: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/36.jpg)
44 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ API for Docker Events ■ Start / Stop / Die etc ■ https://docs.docker.com/engine/
reference/api/docker_remote_api/
■ Interlock – Evan Hazlett ■ Framework to listen and publish events ■ Plugin Framework (e.g. HAPROXY) ■ https://github.com/ehazlett/interlock
■ Aerospike Interlock plugin ■ Add / Remove node from Cluster ■ https://github.com/aerospike/interlock
Docker Event API & Interlock
![Page 37: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/37.jpg)
45 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ https://github.com/aerospike/interlock
func (p AerospikePlugin) runAsinfoTip(args ...string) bool{
asinfo, err := exec.LookPath("asinfo")
if err != nil{
log.Errorf("error finding asinfo binary: %s", err)
return false
}
time.Sleep(time.Second*5) //sleep 5s for ASD to be ready
cmd := exec.Command(asinfo,args...)
Interlock Plugin - Aerospike
![Page 38: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/38.jpg)
46 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Considerations
![Page 39: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/39.jpg)
47 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Inside ■ Encapsulation of Concerns
Storage: Inside or outside the container?
Host
daemon
container
Host
daemon
container e.g. SSD
e.g. EBS
/data/db /mnt/xx:/data/db
/dev/xvdb
Outside ■ Separation of Concerns ■ Storage Features (e.g.
Snapshots)
![Page 40: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/40.jpg)
48 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Data Container ■ --volumes-from <container name> ■ Managed like other containers ■ Special rules for Destruction ■ TBD: Performance
Storage: Data Container?
Host
daemon
container
Host
daemon
container
![Page 41: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/41.jpg)
49 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Summary
• Define Container, their contents and how they work together once • Deploy the same images in Dev, Pre-Prod and Production across
Platforms
One solution from Dev -> Production
• Ops define the whitelisted images, security policies etc. • Dev use approved images to build upon • Eliminate the complexity (and cost) of deployment • Scale up & down in a Flexible and Simple way
Running Docker & Database in Production
![Page 42: SD Times - Docker v2](https://reader033.fdocuments.in/reader033/viewer/2022051300/587a94f61a28ab09758b497b/html5/thumbnails/42.jpg)
50 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ Code ■ http://github.com/alvinr/docker-demo/tree/master/aerospike
■ Docker Images ■ http://hub.docker.com/r/aerospike/
■ Aerospike & Docker deployment guide ■ http://www.aerospike.com/docs/deploy_guides/docker/
■ Contact me! ■ [email protected] ■ @jonnyeight
Thanks and Q&A