Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack

Copyright © NTT Communications Corporation. All rights reserved.

Automated Deployment & Benchmarkingwith Chef, Cobbler and Rally for OpenStack

NTT Communications Corporation

Yuki NishiwakiMahito OguraTakeaki Matsumoto


About Us

1

Mahito Ogura [email protected]

Technology DevelopmentDevOps Engineer

R&D for IaaS, NoSQL, DB as a Service,Hadoop as a service

Contributing to Devstack and Rally

Yuki Nishiwaki [email protected]

Technology DevelopmentSoftware Engineer

R&D for IaaS

Contributing to Devstack and Rally-‐‑‒ 11 patches committed in Liberty


Agenda

2

1.  Company Overview

2.  Automated Deployment & Benchmarking–  Deployment–  Testing–  Benchmarking–  Evaluation

3.  Use Cases

4.  Conclusion


About NTT Communications

3

OpenStack in Production•  Cloudn VPC type ClosedNW•  Enterprise Cloud（coming soon）

Contribution to OpenStack Community•  OpenStack Foundation Corporate Sponsor•  Japan OpenStack user group•  Bug Report / Bug Fix / Blueprint


OpenStack in Production

4

Launched our first OpenStack product in October 2013

Copyright © NTT Communications Corporation. All rights reserved.5

OpenStack deployment is not easy thing to do•  So many components/modules have to be synced and integrated

•  Environment needs to be updated for bug-‐‑‒fix and enhancement

•  There is no magic script to make environment up and running each time we deploy

OpenStack Deployment


Our Challenge

6

How can we continuously deploy OpenStack while consistently keeping service performance?


2. Automated Deployment & Benchmarking

7


Improvement cycle

8

Functional Test Benchmark

Output Result

Change structure

Deploy OpenStack

Evaluate Result


Benchmark

Our Implementation (1/2)

9

Functional Test

Output Result

Change structure

Automate Automate

Hand work

Deploy OpenStackChef RallyTempest

Cobbler

Evaluate Result Hand workFluentd Grafana InfluxdbGithub

EnterpriseRally


Benchmark

Our Implementation (2/2)

10

Functional Test

Output Result

Change structure

Automate Automate

Hand work

Deploy OpenStack

Hand work

Chef RallyTempestCobbler

Evaluate Result Fluentd Grafana InfluxdbGithub

EnterpriseRally


Deployment

11

Functional Test

Output Result

Change structure

Automate Automate

Hand work

Deploy OpenStack

Hand workEvaluate Result

Benchmark


Automated deployment is divided into four steps•  Orchestration•  Configuration•  Bootstrapping•  Networking

Deployment Steps Deployment

12


Tools for Automation

Orchestration

Configuration

Bootstrapping

Networking

Setting for clusters

Setting for just one server

Installing OS into physical servers

Setting up network switch

13

Chef

Expect

Cobbler

Deployment

Chef


Infrastructure as a Code

Environmental information is maintained in text format•  Definition of network

•  System architecture of OpenStackü  Which role does the server have

•  Setting, Configuration of Server ü  Chef Repository

Version control is managed by Git

Deployment

14


Deployment flow

Git clone Update Chef Server

Settingswitch

OS installApply Chef(Configuration

role)

Apply Chef(Orchestration

role)

※Omit Detail process, later let us explain detail each step

Networking

Bootstrapping Configuration Orchestration

Deployment

15


Deployment flow


Settingswitch


role)


role)


Networking


Deployment

16


Deployment flow


Setting switch


role)


role)


Networking


Deployment

17


Automate Networking

•  Auto VLAN setup•  Preconditions

ü  Define server port nameü  Set switch port description with naming rule

“<server_̲name>_̲<server_̲port_̲name>”

eth0 eth1 eth2 eth3 eth4 eth5

eth6 eth7

Naming rule for physical port-‐‑‒ server_̲1: -‐‑‒ eth0: -‐‑‒ vlan10: -‐‑‒ native -‐‑‒ vlan100 -‐‑‒ eth1: -‐‑‒ vlan10: -‐‑‒ native

Config.yaml

When I got servers, I decide it

Networking Deployment

18


Rally

Tempest

influxdb

BenchmarkDashboard

Cobbler

Automate Networking flow (1/4)

19

Service

ChefServer

1

config.yaml (image)

-‐‑‒ server_̲1 -‐‑‒ eth0 -‐‑‒ vlan10 -‐‑‒ native -‐‑‒ vlan100-‐‑‒ eth1 -‐‑‒ vlan10


1. Picks up configuration including VLAN info

Github Enterprise



20

Service

ChefServer


Rally

Tempest

influxdb

BenchmarkDashboard

Cobbler

2. Gets port description from network switch

2

Github Enterprise


Rally

Tempest

influxdb

BenchmarkDashboard

Cobbler


21

Service

ChefServer

Server1 eth0(switch1 port 1/15) 　　　　　　 => VLAN100, 200Server1 eth1(switch2 port 1/15) => VLAN100, 200


3. Generates switch configuration based on network layout and target server settings

3

Github Enterprise



22

Service

ChefServer


Rally

Tempest

influxdb

BenchmarkDashboard

Cobbler

4. Executes the commands which Jenkins generated

4

Github Enterprise


Deployment flow


Setting switch


role)


role)


Networking


Deployment

23


Bootstrapping flow (1/3)

24

Service

ChefServer

Deployment Bootstrap

Rally

Tempest

influxdb

BenchmarkDashboard

Cobbler

1. Run “chef-‐‑‒client” via ssh2. Apply chef

ü Install cobbler and configureü Register machines

ü hostnameü IP address

2

1

Github Enterprise



25

Service

ChefServer


Rally

Tempest

influxdb

BenchmarkDashboard

Cobbler

3. IPMI power reset4. Start auto install procedure (PXE)

3

4

Github Enterprise



26

Service

ChefServer

Ubuntu


Rally

Tempest

influxdb

BenchmarkDashboard

Cobbler

5. Check ssh connection periodically

5

Github Enterprise


Deployment flow


Setting switch


role)


role)


Networking


Deployment

27


Configuration, Orchestration flow(1/3)

28

Internet

ChefServer

Ubuntu

Deployment

Rally

Tempest

influxdbCobbler

Configura2on

Orchestra2on

1. Git clone the definition file of the system architecture of OpenStack

{ “<environment>”: { “servers”: { “server1”: “backend”, “server2”: “controller” }, “dependencies”: { “backend”: [], “controller”: [{“backend”:1}] } }}

definition.json

1

Github Enterprise



29

Internet

ChefServer

Ubuntu

Deployment

Rally

Tempest

influxdb

BenchmarkDashboard

Cobbler

Configura2on

Orchestra2on

3. Apply chefü backend configuration roleü backend orchestration role…

3

2

2. Run “chef-‐‑‒client” via ssh in order of the definition file

Github Enterprise



30

Internet

ChefServer

OpenStack

Deployment

Rally

Tempest

influxdb

BenchmarkDashboard

Cobbler

Configura2on

Orchestra2on

4

Github Enterprise

4. Remove orchestration roleü divided role into configuration & orchestrationü doesnʼ’t have idempotence.


Deployment is ended


Setting switch


role)


role)


Networking


Deployment

31


Review the Automate deployment

32

Pros•  Anyone can create the same environment anytime

Cons•  Any single tool does not satisfy auto deployment

ü We need to use multiple tools in combinationü We need wide knowledge

Deployment


Functional Test

33


Functional Test

34

Functional Test

※ Omit detail process here

Output Result

Change structure

Automate Automate

Hand work

Deploy OpenStack


Benchmark


Functional Test

35

Tool for test•  We use Tempest

What is Tempest?•  Tempest is Official Integration Test Suite•  Tempest can test any OpenStack, all in one or 1000 nodes

Functional Test


Functional Test

36

Tempest is used for upstream developing

Community Workflow by http://docs.openstack.org/

Functional Test


What types of test we use

37

We use•  API Tests

ü  Unit tests that ensure API works properly•  Scenario Tests

ü  Series of actions like actual users do

Functional Test


How to use

38

We can execute it in only two steps 1.  Edit tempest.conf2.  Execute ./run_̲tempest

$ ./run_̲tempest………Ran 1124 tests in 1185.267sFAILED (failures=4)

Functional Test


Test flow (1/4)

39

Functional Test

Service

ChefServer

OpenStack

Rally

Tempest

influxdb

BenchmarkDashboard

Cobbler

1. Create a tempest container from the image

1

Github Enterprise


Test flow (2/4)

40

Functional Test

Service

ChefServer

OpenStack

Rally

Tempest

influxdb

BenchmarkDashboard

Cobbler

2. Run “chef-‐‑‒client” via ssh3. Apply chef recipe, To set up tempest.conf

3

2

Github Enterprise


Test flow (3/4)

41

Functional Test

Service

ChefServer

OpenStack

Rally

Tempest

influxdb

BenchmarkDashboard

Cobbler

4. Run tempest against OpenStack

4

Github Enterprise


Test flow (4/4)

42

Functional Test

Service

ChefServer

OpenStack

Rally

Tempest

influxdb

BenchmarkDashboard

Cobbler

5. Get the results and delete the container which I created right now

5

Github Enterprise


Review the Test

43

Pros•  Generate configuration file automatically based on auto deployed OpenStack

•  Execute functional tests simultaneously with different customized Tempest

Cons•  Doesnʼ’t cover system architecture

ü  We proposed 11 patches for fixing tempest•  Doesnʼ’t control test cases based on API level

Functional Test


Benchmark

44


Benchmark

45

Functional Test

※ Omit detail process here

Output Result

Change structure

Automate Automate

Hand work

Deploy OpenStack


Benchmark


Benchmark

46

Tool for benchmark•  Rally ( is one of the OpenStack projects )

What is Rally ?•  Released v0.1.1 in 6 October 2015•  Benchmarking tool

ü  Not only Benchmark but support functions of deploy and verification

Benchmark


How to use Rally

47

1.  Create deployment

rally@3eba20593250:~∼$ cat config.json

{ "admin": { "password": “p@ssw@rd", "tenant_̲name": ”admin", "username": ”admin" }, "auth_̲url": "http://lb:5000/v2.0", "type": "ExistingCloud"}

rally@3eba20593250:~∼$rally deployment create -‐‑‒-‐‑‒filename config.json -‐‑‒-‐‑‒name existing

Benchmark


How to use Rally

48

2. Check credentialrally@3eba20593250:~∼$ rally deployment checkkeystone endpoints are valid and following services are available:+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+| services | type | status |+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+| cinderv2 | volumev2 | Available || glance | image | Available || keystone | identity | Available || neutron | network | Available || nova | compute | Available |+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+

Benchmark


How to use Rally

49

3. Create task filerally@3eba20593250:~∼$ cat benchmarks/nova/boot-‐‑‒and-‐‑‒delete.json{% set flavor_̲name = flavor_̲name or "m1.tiny" %}{ "NovaServers.boot_̲and_̲delete_̲server": [ { "args": { "flavor": { "name": "{{flavor_̲name}}" }, "image": { "name": "^cirros.*$" } } } ]}

Task file is expressed in json or yaml and We can use statement of Jinja2.

Benchmark


How to use Rally

50

4. Start benchmarkrally@3eba20593250:~∼$ rally task start benchmarks/nova/boot-‐‑‒and-‐‑‒delete.json +-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+| Response Times (sec) |+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+| action | min | median| 90%ile| 95%ile | max | avg | success | count |+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+| nova.boot_̲server | 4.614 | 4.768 | 4.972 | 5.417 | 5.861 | 4.852 | 100.0% | 10 || nova.delete_̲server | 2.486 | 2.576 | 2.709 | 2.732 | 2.754 | 2.603 | 100.0% | 10 || total | 7.148 | 7.369 | 7.658 | 8.018 | 8.377 | 7.455 | 100.0% | 10 |+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+Load duration: 37.9388210773Full duration: 51.1114509106…

Benchmark


How to use Rally

51

5. Make result “Human Readable” (Create HTML Report)rally@3eba20593250:~∼$ rally task report -‐‑‒-‐‑‒tasks 5e8815a9-‐‑‒10eb-‐‑‒4d67-‐‑‒86b4-‐‑‒6027f4adc634 1f71cb68-‐‑‒9c1e-‐‑‒49e3-‐‑‒8af5-‐‑‒6d21dc34c455 -‐‑‒-‐‑‒out result.html -‐‑‒-‐‑‒html

Benchmark


How to use Rally

52

※1 There is blueprinthttps://blueprints.launchpad.net/rally/+spec/cli-‐‑‒plot2html-‐‑‒compare-‐‑‒tasks

We can see each detailed result of the benchmark. But we canʼ’t compare each result in HTML yet※1

Benchmark


How I use Rally

53

Benchmark

BenchmarkCompareto the past(Summary)

Nothingto do

CheckDetail

No

Detect unusual

Yes

Rally task report

Rally task start


How I use Rally

54

Benchmark

BenchmarkCompareto the past(Summary)

Nothingto do

CheckDetail

No

Detect unusual

Yes

Rally task report

BenchmarkDashboard

We created it

Rally task start


Benchmark Dashboard

55

Benchmark Dashboard

•  Provides two functionsü Compare summarized result with past resultü  Provide link to a html report created by “rally task report”

Benchmark


Benchmark Dashboard

56

Benchmark


Benchmark flow (1/4)

57

Benchmark

Service

ChefServer

OpenStack

Rally

Tempest

influxdb

BenchmarkDashboard

Cobbler

1.  Create a container from the image which I prepared

1

Github Enterprise



58

Benchmark

Service

ChefServer

OpenStack

Rally

Tempest

influxdb

BenchmarkDashboard

Cobbler

2. Register existing OpenStack3. Run each benchmark

2

3

Github Enterprise


Internet

Github Enterprise


59

Benchmark

ChefServer

OpenStack

Rally

Tempest

influxdb

BenchmarkDashboard

Cobbler

4. Upload report.html to Benchmark dashboard5. Post summarized results to influxdb

4

5


Service

Github Enterprise


60

ChefServer

OpenStack

Rally

Tempest

influxdb

BenchmarkDashboard

Cobbler

Developer

view result

We can view the result at the dashboard


Review the Benchmark

61

Benchmark

Pros•  We can compare the result with the past•  The process of Benchmark is fully automated

Cons•  We donʼ’t support many benchmark scenarios•  Best practice for benchmark isnʼ’t fixed yet


Evaluate Result

62


Deploy and Test and Benchmark cycle

63

Functional Test

Output Result

Change structure

Automate Automate

Hand work

Deploy OpenStack


Benchmark


When benchmark score is bad

64

•  Need to find bottlenecks or issuesü  Too many host resourcesü  Too many logsü  Too difficult to find bottlenecks or issues ...

•  Develop analysis systemü  To enable data collectingü  To enable monitoringü  To enable alert notify


Architecture to evaluate the result

65

OpenStack logsHost resource data

Storage

Host resources Log information

and more …Search log data

notify

store

Get WARNING/ERROR notifications


Improvement cycle

66

Functional Test

Output Result

Change structure

Automate Automate

Hand work

Deploy OpenStack


Benchmark


Use case

67


What scenario I use

68

Benchmark Scenario•  VMTask scenario ( which is contained in rally at default )

ü  Prepare the task files which are different in concurrency(30, 50)

nova boot ssh loginrun

command via ssh

nova delete

Use case


Why I use

69

Why I use VMTask scenario•  The scenario is the most common use case •  We need to know which components will be affected by increasing the number of users

Use case


What happened

70

What happened when I execute the benchmark•  Actually, at first success rate isnʼ’t 100%

Use case


Analyze cause of failure

71

Check the detailed result of the benchmark

it Doesnʼ’t have IP reachability.

Use case


Analyze cause of failure

72

Check the neutron log file$ less /var/log/neutron/neutron-‐‑‒server.log…TimeoutError: QueuePool limit of size 5 overflow 20 reached, connection timed out, timeout 10…

Overflow connection pool.Need to review the configuration about database.

Use case


Change configuration

73

Change [database] in neutron.conf-‐‑‒  max_̲retries-‐‑‒  retry_̲interval-‐‑‒  min_̲pool_̲size-‐‑‒  max_̲pool_̲size-‐‑‒  ide_̲timeout-‐‑‒  max_̲overflow

Use case


Check again

74

$ git checkout -‐‑‒b change_̲db_̲setting_̲in_̲neutron_̲conf$ git commit -‐‑‒am “foobar”$ git push origin change_̲db_̲setting_̲in_̲neutron_̲conf

Request Jenkins via Slack

Use case


Check again

75

This change improves the success rates

Use case


Conclusion

76


conclusion

77

•  Default configuration doesnʼ’t cover all use cases

•  We need to investigate the best configuration

•  Best configuration requires us to follow the cycleü  deployment, functional test, benchmark, evaluation

•  Any single tool cannot automate whole processü  multiple Open Source Software can automate whole process


License

78

Jenkins image by Jenkins project is licensed under a Creative Commons Attribution-‐‑‒ShareAlike 3.0 Unported License.


Thanks for your attention

Enterprise Cloud

1st SessionAutomate deployment & benchmark for your OpenStack with Chef, Cobbler and Rally

2nd SessionNFV Service Federation across Heterogeneous Sites

3rd SessionEnhancement on OpenStack Networking for Carrier Cloud Platform

4th SessionOpenStack x Enterprise : Formula for Developing OpenStack-‐‑‒Based Cloud for Enterprise

DevOps

Network

Enterprise


80

NTT Com BoothP9:Hiten Hall

Weʼ’re looking forward to meeting you at NTT Com Booth!

Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack

Technology

Transcript of Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack