mesos-devoxx14
-
Upload
samir-bessalah -
Category
Technology
-
view
2.870 -
download
0
description
Transcript of mesos-devoxx14
![Page 1: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/1.jpg)
ROOM 9 #devoxx #DV14 #mesos #docker @samklr
Kernel for Distributed
Applications
Sam BESSALAH
@samklr
![Page 2: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/2.jpg)
#Devoxx #DV14 #mesos #docker @samklr
![Page 3: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/3.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Example of a statically partionned cluster
![Page 4: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/4.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Akka/Play Jenkins Spark/Hadoop RedisStorm
![Page 5: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/5.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Issues with static partitionning
- Increased Complexity : Machines/vm sprawlSoftware failures gets harder to handle as machines number grow
- Huge overheard, on development,- Planning for failure and maintenance gets harder- Not really developer friendly. Long cycles to
deployments.
![Page 6: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/6.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Resources utilisation under static partitionning
Play/Akka Redis Hadoop
![Page 7: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/7.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Resources utilisation under static partitionning
Play/Akka RedisHadoop
Put together, we would get
![Page 8: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/8.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Enters Google stage left ...
Google “Borg” .Unofficial name.Moved to a new system called OmegaBundles all the cluster’s machines into a single pooledview of resources.
Evolved to Omega.
![Page 9: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/9.jpg)
#Devoxx #DV14 #mesos #docker @samklr
![Page 10: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/10.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Mesos : Level of indirection
1. Run multiple types of distributed systems on the same cluster of machines, and dynamically share resources more eficiently.
2. Provide failure detection, task distribution, task tracking, task monitoring, low level resources management, and fine grained resources sharing. And scale to the thousands of nodes.
![Page 11: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/11.jpg)
#Devoxx #DV14 #mesos #docker @samklr
![Page 12: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/12.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Quick View.
![Page 13: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/13.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Mesos Architecture
![Page 14: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/14.jpg)
#Devoxx #DV14 #mesos #docker @samklr
http://mesos.apache.org/documentation/latest/mesos-architecture/
![Page 15: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/15.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Mesos Framework
A framework is an application that runs distributed applications on Mesos.
It has two components : The Scheduler : Interacts with Mesos, to book resources and
launch tasks within the mesos slaves.Executors : gets information from the framework like
environmental variables,configuration informationand run tasks inside the mesos slaves.
![Page 16: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/16.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Mesos Framework
A framework is an application that runs distributed applications on Mesos.
It has two components : The Scheduler : Interacts with Mesos, to book resources and
launch tasks within the mesos slaves.Executors : run tasks in mesos slaves.
Mesos provides an API to build its own frameworks. And there are SDKs in mulitple languages.
![Page 17: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/17.jpg)
#Devoxx #DV14 #mesos #docker @samklr
![Page 18: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/18.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Resources allocation process
- Mesos allocate resources via “resources offers”
- Resources are a snapshot of available resources that a scheduler will use to runtasks on mesos slaves.
![Page 19: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/19.jpg)
#Devoxx #DV14 #mesos #docker @samklr
![Page 20: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/20.jpg)
#Devoxx #DV14 #mesos #docker @samklr
![Page 21: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/21.jpg)
#Devoxx #DV14 #mesos #docker @samklr
![Page 22: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/22.jpg)
#Devoxx #DV14 #mesos #docker @samklr
1- resourceOffers()
![Page 23: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/23.jpg)
#Devoxx #DV14 #mesos #docker @samklr
1- resourceOffers() 2. launchTasks()
![Page 24: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/24.jpg)
#Devoxx #DV14 #mesos #docker @samklr
1- resourceOffers() 2. launchTasks()
3. lauchTask()
![Page 25: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/25.jpg)
#Devoxx #DV14 #mesos #docker @samklr
1- resourceOffers() 2. launchTasks()
3. lauchTask() 4. statusUpdate()
![Page 26: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/26.jpg)
#Devoxx #DV14 #mesos #docker @samklr
1- resourceOffers()
2. launchTasks()
3. lauchTask() 4. statusUpdate()
5. statusUpdate()
![Page 27: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/27.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Mesos slaves
![Page 28: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/28.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Mesos slaves
Linux Containers
![Page 29: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/29.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Mesos uses Linux Containers for resource isolation.
Mesos containers are built around cgroups , to provide isolation of
CPU, memory, disk … and namespaces for network isolation.
Started leveraging containerirsation since 2010
![Page 30: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/30.jpg)
#Devoxx #DV14 #mesos #docker @samklr
![Page 31: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/31.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Linux Containers
![Page 32: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/32.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Linux ContainersLXC
![Page 33: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/33.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Linux ContainersLXC
Cgroups
![Page 34: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/34.jpg)
#Devoxx #DV14 #mesos #docker @samklr
![Page 35: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/35.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Mesos provides a native support for DockerContainers since 0.20
![Page 36: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/36.jpg)
#Devoxx #DV14 #mesos #docker @samklr
![Page 37: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/37.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Fault recovery & HA
- Mesos Master uses Zookeeper for leader election and leader discovery.
- Mesos Master have a Registrar to log all the tasks and slaves info, implemented as a replicated log using MultiPaxos for consensus
![Page 38: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/38.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Fault recovery & HA
- Mesos Master uses Zookeeper for leader election and leader discovery.
- Mesos Master have a Registrar to log all the tasks and slaves info, implemented as a replicated log using MultiPaxos for consensus
- Mesos has a slave recovery mechanism. So user’s tasks keep running whenever a slave proces goes under.
![Page 39: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/39.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Fault recovery & HA
- Mesos Master uses Zookeeper for leader election and leader discovery.
- Mesos Master have a Registrar to log all the tasks and slaves info, implemented as a replicated log using MultiPaxos for consensus
- Mesos has a slave recovery mechanism. So user’s tasks keep running whenever a slave proces goes under. Slaves checkpoint information to their local disks (task info, status updates, etc ), they recover checkpointed data and reconnect with old tasks when they restart.
![Page 40: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/40.jpg)
#Devoxx #DV14 #mesos #docker @samklr
![Page 41: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/41.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Mesos Framework
A framework is an application that runs distributed applications on Mesos.
It has two components : The Scheduler : Interacts with Mesos, to book resources and
launch tasks within the mesos slaves.Executors : run tasks in mesos slaves.
Mesos provides an API to build its own frameworks. And there are SDKs in mulitple languages.
![Page 42: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/42.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Marathon
![Page 43: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/43.jpg)
#Devoxx #DV14 #mesos #docker @samklr
- A generic mesos framework to run long running services (web apps, etc …)
- A distributed Init.d for the cluster.- Runs any linux binary without modification (Tomcat, Play, …)- Cluster wide process supervisor
-Private PaaS - Service discovery- Provide a self service Rest API for deployment - Authentification & SSL- Placement constrainst (nodes, racks, etc. (Like with Spark) )- Service discovery and load balancing via HAProxy
What is Marathon ?
![Page 44: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/44.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Marathon workflow
![Page 45: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/45.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Marathon workflow
![Page 46: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/46.jpg)
#Devoxx #DV14 #mesos #docker @samklr
POST /v2/apps {“id” : “IPython Notebook”“container “ : {
“image “ : “docker://ipython/notebook” “options” : [ . . . ]}
}
Marathon Rest APi
![Page 47: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/47.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Wants to scale by addind more instances ..
PATCH /v2/apps {“instances” : 4
}
Marathon Rest APi
![Page 48: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/48.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Conclusion
![Page 49: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/49.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Conclusion
![Page 50: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/50.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Conclusion
Aurora
![Page 51: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/51.jpg)
#Devoxx #DV14 #mesos #docker @samklr
Conclusion
Kubernetes
![Page 52: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/52.jpg)
#Devoxx #DV14 #mesos #docker @samklr
![Page 53: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/53.jpg)
#Devoxx #DV14 #mesos #docker @samklr
![Page 54: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/54.jpg)
#Devoxx #DV14 #mesos #docker @samklr
![Page 55: mesos-devoxx14](https://reader035.fdocuments.in/reader035/viewer/2022081404/5598bf2a1a28abea208b4682/html5/thumbnails/55.jpg)
#Devoxx #DV14 #mesos #docker @samklr
speakerdeck.com/samklr