Service Provisioning and Automation in the Cloud · SixSq Services ‣Consultancy • Agile...
Transcript of Service Provisioning and Automation in the Cloud · SixSq Services ‣Consultancy • Agile...
Copyright © 2012 SixSq Sàrl. All rights reserved.
Service Provisioning and Automation in the CloudA practical usage of cloud technology to rationalise and automate software testing, certification and deployment
Marc-Elian BéginSixSq
SixSq Sàrl
‣Founded in 2007 by three partners‣Based in Geneva,
Switzerland‣Privately held, limited
liability company
‣sixsq.com
2
SixSq Services
‣ Consultancy• Agile software development processes and practices• Automation of those processes and practices• Cloud and virtualization technologies
‣ Software development• Distributed systems• Cloud-based domain specific vertical solutions
‣ Training and coaching• Agile and Scrum training• Coaching service to put theory into practice
3
SixSq Products
‣SlipStream• Automatic creation of new virtual machines• Automatic multi-machine deployment and
orchestration of services• Runs on public or private cloud infrastructure
‣StratusLab - European collaboration• Complete IaaS Cloud distribution• Open source
4
Collaborations and Partnerships
‣Building ‘Cloud Read Pack’ hardware/software private cloud solution with IBM and Darest
‣Amazon Independent Software Vendor
5
A"European"cloud"compu/ng"partnership:""big"science"teams"up"with"big"business"
Strategic)Plan)
Establish"mul/8tenant,"mul/8provider"cloud"infrastructure"
Iden/fy"and"adopt"policies"for"trust,"security"and"privacy"
Create"governance"structure"
Define"funding"schemes"
To"support"the"compu/ng"capacity"needs"for"the"ATLAS"
experiment"
SeEng"up"a"new"service"to"simplify"analysis"of"large"
genomes,"for"a"deeper"insight"into"evolu/on"
and"biodiversity"
To"create"an"Earth"Observa/on"plaGorm,"
focusing"on"earthquake"and"volcano"research"
Main Customers
‣Atos‣European Broadcast Union (Eurovision)‣ESA/ESTEC‣ESA/ESOC‣And more... I can’t talk about it yet, but it’s exciting!
7
Copyright © 2012 by SixSq Sàrl. All rights reserved.
My Cloud Definition
My Cloud Definition...
Virtualisation + Simple API (REST is best :-)
‣Yes!... this implies a IaaS foundation
‣What’s not in my definition:• Public/Private? Depends who takes care of elasticity• How you pay for the resources
9
Copyright © 2012 by SixSq Sàrl. All rights reserved.
Why we need automation?
Silo Development
11
Silo Development
11
Dev
Silo Development
11
Dev Test
Silo Development
11
Dev Test QA
Silo Development
11
Dev Test QA Ops
Silo Development
11
Dev Test QA Ops
Silo Development
11
Dev Test QA Ops
Silo Development
11
Dev Test QA Ops
Silo Development
11
Dev Test QA Ops
Silo Development
11
Dev Test QA Ops
Silo Development
11
Dev Test QA Ops
Silo Development
11
Dev Test QA Ops
Silo Development
11
Dev Test QA Ops
Silo Development
11
Dev Test QA OpsDuplication
Duplication
Duplication
Silo Development
11
Dev Test QA Ops
Silo Development
11
Dev Test QA Ops
Slow
Slow
Slow
Silo Development
11
Dev Test QA Ops
On-Demand Provisioning
12
On-Demand Provisioning
12
Dev
TestQA
Ops
On-Demand Provisioning
12
Dev
TestQA
Ops
On-Demand Provisioning
12
Dev
TestQA
Ops
On-Demand Provisioning
12
Dev
TestQA
Ops
On-Demand Provisioning
12
Dev
TestQA
Ops
On-Demand Provisioning
12
Dev
TestQA
Ops
On-Demand Provisioning
12
Dev
TestQA
Ops
On-Demand Provisioning
12
Dev
TestQA
Ops
On-Demand Provisioning
12
Dev
TestQA
Ops
My own
full system
now
On-Demand Provisioning
12
Dev
TestQA
Ops
On-Demand Provisioning
12
Dev
TestQA
Ops
Share
Share
Share
On-Demand Provisioning
12
Dev
TestQA
Ops
Continuous Integration
13
<parameterized>
Continuous Integration
13
<parameterized>
Agile Methods Footprint
14
User
Agile Methods Footprint
14
User Dev
Agile Methods Footprint
14
User Dev Ops
Agile Methods Footprint
14
User Dev Ops
Agile Methods Footprint
14
Scrum
User Dev Ops
Agile Methods Footprint
14
Scrum
eXtreme Programming
User Dev Ops
Agile Methods Footprint
14
Scrum
eXtreme Programming
DevOps
User Dev Ops
Agile Methods Footprint
14
Scrum
eXtreme Programming
DevOps
Copyright © 2012 by SixSq Sàrl. All rights reserved.
Case Study 1: Mission Control System
ESA ESOC
European Space Operations Centre, Darmstadt, Germany
16
Context and Problems
•Mission Control System deployment
•Deployment is manual•Time consuming (multiple days...)•Error prone resulting in configuration inconsistencies•Poor reproducibility
17
Relevance of this Case Study
•Complex legacy system•C++, Java, Perl, Python, Tcsh, CORBA (yes!)•Millions of lines of code, grown over several years
•Not designed for automation•Great benefit from automation, if possible!
18
Vision
21
•Automated on-demand deployments•Consistent and reliable deployments •Dramatic effort/cost reduction in deployment/testing
SLES11
Image References
22
SLES11
Store
SLES11
Image References
22
SLES11
Store
SLES11
Image References
22
SLES11
Store
COTS
SLES11
Image References
22
SLES11
Store
COTS
COTSConfiguration
SLES11
Image References
22
Store
COTSSLES11
SLES11
Image References
22
Store
COTSSLES11
SLES11
Image References
22
Store
COTSSLES11COTS
SLES11
SLES11
Image References
22
StoreCOTS
SLES11
COTSSLES11
SLES11
Image References
22
StoreCOTS
SLES11
S2K S/W
COTSSLES11
SLES11
Image References
22
StoreCOTS
SLES11
S2K S/W
S2KConfiguration
COTSSLES11
SLES11
Image References
22
Store
COTSSLES11
S2KCOTS
SLES11
SLES11
Image References
22
Store
COTSSLES11
S2KCOTS
SLES11
Deployment Scenario
23
Deployment Scenario
23
Deployment Scenario
23
Deployment Scenario
23
Orchestrator
Deployment Scenario
23
Orchestrator
Deployment Scenario
23
Orchestrator S2K Prime
Deployment Scenario
23
Orchestrator S2K Prime
Deployment Scenario
23
Orchestrator S2K Prime
S2K Backup
Deployment Scenario
23
Orchestrator S2K Prime
S2K Backup
Deployment Scenario
23
Orchestrator
S2K Client
S2K Prime
S2K Backup
Deployment Scenario
23
S2K Client
S2K Prime
S2K Backup
Deployment Scenario
23
S2K Client
S2K Prime
S2K Backup
Deployment Scenario
23
S2K Client
S2K Prime
Hostname
S2K Backup
Deployment Scenario
23
S2K Client
S2K Prime
Hostname
S2K Backup
Hostname
Deployment Scenario
23
S2K Client
S2K Prime
Hostname
S2K Backup
HostnameHostname
Deployment Scenario
23
S2K Client
S2K Prime
Hostname
S2K BackupReady
HostnameHostname
Solution Overview
24
Applications + deployment recipes
VM factoryDeployment engine
e.g. S2K
Cloud (IaaS)
Cloud Infrastructure
•Infrastructure as a Service (IaaS) provides:•Virtual machine life-cycle•VM image store•Storage
•StratusLab used for IaaS cloud:•License: Apache 2.0•European
25
S2K configuration in SlipStream
26
S2K configuration in SlipStream
26
S2K configuration in SlipStream
26
S2K configuration in SlipStream
26
S2K configuration in SlipStream
26
S2K configuration in SlipStream
26
S2K configuration in SlipStream
26
S2K configuration in SlipStream
26
SlipStream dashboard and reports
27
SlipStream dashboard and reports
27
SlipStream dashboard and reports
27
SlipStream dashboard and reports
27
SlipStream dashboard and reports
27
SlipStream dashboard and reports
27
7 minutes later...
28
Next steps?
•Execute Entire Verification Suite (e.g. ART)•Tighter integration with build process
•Integrate with CI solution (e.g. Hudson, Jenkins)•Generate binary packages to simplify installation
29
Continuous Integration
30
<s2k-build-version><s2k-config-profile>
Continuous Integration
30
<s2k-build-version><s2k-config-profile>
Continuous Integration
30
<s2k-build-version><s2k-config-profile>
Continuous Integration
30
<s2k-build-version><s2k-config-profile>
Continuous Integration
30
<s2k-build-version><s2k-config-profile>
Continuous Integration
30
<s2k-build-version><s2k-config-profile>
b3427 pMSG-ESOC-sim
Continuous Integration
30
<s2k-build-version><s2k-config-profile>
b3427 pMSG-ESOC-sim
b3434 pMSG-Eumetsat-ops
Continuous Integration
30
<s2k-build-version><s2k-config-profile>
b3427 pMSG-ESOC-sim
b3434 pMSG-Eumetsat-ops
b3452 pMy-Dev-deploy
Continuous Integration
30
<s2k-build-version><s2k-config-profile>
b3427 pMSG-ESOC-sim
b3434 pMSG-Eumetsat-ops
b3452 pMy-Dev-deploy
b3467 pMSG-auto-cert
Potential Deployments - ESOC Internal
31
ESOC
SlipStream Cloud
PackagePackagePackagePackage
Potential Deployments - ESOC Internal
31
ESOC
SlipStream Cloud
PackagePackagePackagePackage
Potential Deployments - ESOC Internal
31
ESOC
SlipStream Cloud
PackagePackagePackagePackage
Potential Deployments - ESOC Internal
31
ESOC
SlipStream Cloud
PackagePackagePackagePackage
Potential Deployments - ESOC Internal
31
ESOC
SlipStream Cloud
PackagePackagePackagePackage
Potential Deployments - Shared Dev
32
ESOC
SlipStream Cloud
Contractor A
CloudPackagePackagePackagePackage
Contractor B
CloudPackagePackagePackagePackage
StratusLabMarketplace
Potential Deployments - Shared Dev
32
ESOC
SlipStream Cloud
Contractor A
CloudPackagePackagePackagePackage
Contractor B
CloudPackagePackagePackagePackage
StratusLabMarketplace
Potential Deployments - Shared Dev
32
ESOC
SlipStream Cloud
Contractor A
CloudPackagePackagePackagePackage
Contractor B
CloudPackagePackagePackagePackage
StratusLabMarketplace
Potential Deployments - Shared Dev
32
ESOC
SlipStream Cloud
Contractor A
CloudPackagePackagePackagePackage
Contractor B
CloudPackagePackagePackagePackage
StratusLabMarketplace
Copyright © 2012 by SixSq Sàrl. All rights reserved.
Case Study 2: Multi-Media News Exchange Platform
Goal
•Create news exchange platform, between most European (and others) broadcasters (> 70 sites)
•In time, replace satellites•No single point failure
34
Production Deployment
35
Paris London
GenevaFrankfurt
Geneva 2
BroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcaster
Technologies
•RESTful web services•Clojure, Noir web framework•Stateless
•Persistence: Cassandra/Datastax
36
Challenges
•Need to certify full system before large deployment•>80 sites involved all together!!•Distributed development teams (4 sites)
37
Development Deployment
38
Client All-in-oneServer
Site Deployment
39
Client
Cassandra1
RESTfulWS
Cassandra2
ClientClientClientClientClient
Production Deployment
40
Paris London
GenevaFrankfurt
Geneva 2
BroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcaster
Solution Overview
41
Applications + deployment recipes
VM factoryDeployment engine
e.g. TtF WS, Cass.
Cloud (IaaS)
Copyright © 2012 by SixSq Sàrl. All rights reserved.
Case Study 3: Cluster Provisioning
Helix Nebula: ESA/SSEP
43
•SSEP•Processing based on gLite based cluster (1 CE + n WN)•Large satellite image data set made available•Users can self-provision their own cluster
Helix Nebula: ESA/SSEP
44
•SSEP...•Processing based on gLite based cluster (1 CE + n WN)•Large satellite image data set made available•Users can self-provision their own cluster
Automated Deployment
45
•30 WNs deployment•124 deployed virtual cores•248 GB RAM•Provisioned in 2 minutes, from a single action!!
Automated Deployment
45
•30 WNs deployment•124 deployed virtual cores•248 GB RAM•Provisioned in 2 minutes, from a single action!!
Copyright © 2012 by SixSq Sàrl. All rights reserved.
SlipStreamImage Factory and System Provisioning
Who’s deploying SlipStream?
47
• Academic production systems• CNRS - LAL - France• GRNET - Greece
• Commercial production systems• SixSq - Amazon EC2• Atos - Eindhoven - Holland (Helix Nebula PoC environment
Automated GPOD deployment• European Broadcast Union (Eurovision) - in progress
• Entire Transition to File server solution• Adding support for HP OpenStack Cloud Service• Certifying for IBM PureSystems
Main Features
48
•Virtual Machine Factory•Automatic creation of new virtual machines•Based on existing public images or custom built
images•Deployment
•Multi-tier machine topology modeling•Automatic deployment and orchestration of
services
•On-demand provisioning of systems
SlipStream
49
SlipStream
49
SlipStream
49
Main Features
50
•Virtual Machine Factory•Automatic creation of new virtual machines•Based on existing public images or custom built
images•Deployment
•Multi-machine deployment•Automatic deployment and orchestration of
services•On-demand provisioning of systems
SlipStream Cloud support
51
Copyright © 2012 by SixSq Sàrl. All rights reserved.
StratusLabSimple yet powerful IaaS cloud solution
Who’s deploying StratusLab?
53
•Academic production systems•CNRS (2 installations) - LAL and IBCP - France•GRNET - Greece
•Commercial production systems•Atos - Eindhoven - Holland (Helix Nebula PoC
environment with ~500 cores)•More (confidential)
•Many more private installations
Who’s behind StratusLab?
54
CNRS (FR) UCM (ES)
GRNET (GR) SIXSQ (CH)
TID (ES) TCD (IE)
Key features
• Rich contextualisation• Integration with shared/federated Marketplace• Supports copy on write and thin provisioning for fast
instantiation• Dynamic creation of disks (volumes)• Storage backend: iSCSI, Fiber Channel, Shared File System
(e.g. NFS)• Rich command-line client for automation integration• REST API (except for Open Nebula VM management with
XML-RPC)
55
Infrastructure as a Service basics
56
User
Cloud site
NetworkStorage Compute
API
Client
ClientClientClientVM
1: <init>
Network
2: <create>
3: <ssh>
User
Cloud site
NetworkStorage / Cache Compute
API
Client
ClientClientClientVM
1: <init>
Network
4: <create>
5: <ssh>
Image Marketplace0: <shop>
2: <discover>2': <retreive>
3: <validate>
IaaS Federation with StratusLab
57
User
Cloud site
NetworkStorage / Cache Compute
API
Client
ClientClientClientVM
1: <init>
Network
4: <create>
5: <ssh>
Image Marketplace0: <shop>
2: <discover>2': <retreive>
3: <validate>
IaaS Federation with StratusLab
57
Deployment Models
•Federation of Marketplaces for flexible image management
•Public cloud deployment (e.g. Atos and CNRS)•Private federated cloud deployment (e.g. EBU)•Caching of images across cloud installations
58
SixSq commitment
Maintain and further develop•StratusLab Marketplace•StratusLab Storage Service•Integration with OpenNebula
59
Copyright © 2012 by SixSq Sàrl. All rights reserved.
Conclusions
Conclusions
•Automation is paramount to succeed in building (distributed) systems
•Cloud is a key ingredient, not only virtualisation•Need a management layer and a share platform •Try... learn... have fun!!
61
Try it!!
62
slipstream.sixsq.com
slipstream.stratuslab.eu
Give us feedback...
Thanks :-)sixsq.com