Deploy OpenStack with SaltStackqiniuppt.qiniudn.com/Shi Shanshi.pdf · Backend (Spice, OpenStack)...
Transcript of Deploy OpenStack with SaltStackqiniuppt.qiniudn.com/Shi Shanshi.pdf · Backend (Spice, OpenStack)...
Ctrip Cloud Platform Engineering
Started in July, 2012
Infrastructure as a Service to Site Ops & Corp IT
Virtual Machines ( VMware, KVM)
Bare-Metal Nodes (Razor)
Virtual Desktop Infrastructure to Call Centers
Thin clients (GUI, SaltStack)
Backend (Spice, OpenStack)
Ctrip OpenStack
Fleets of windows servers: pets, cattle?
Forked VMware driver: local storage, virtual switch
Naive Neutron without L3: just give me an IP
• OpenStack中的扩展--以Nova为例| Kong's Blog
• Odd Bits — Integrating custom code with Nova using hooks
• PLY Git-Based Patch Management
Ctrip OpenStack
Ubuntu 12.04 with Icehouse kernel/qemu/libvirt/ovs
https://code.launchpad.net/~ubuntu-server-dev
man deb-version[epoch:]upstream-version[-debian-revision]2:2014.1.3.2-0ctrip1
Docker for debian packaging instead of schrootaptly, packer
apt-cacher, flask-pypi-proxy
• HP's OpenStack Helion: Debian all the way down - YouTube
Decisions..
sources.list, keystone endpoints, nova.conf, …
Puppet, Chef, Salt, Ansible
Salt: remote execution { + configuration management }
Salt worked.
• Learning to Scale OpenStack_Juno Update from the Rackspace Public Cloud
• SaltConf14 - Anita Kuno, HP - Using SaltStack for event-driven orchestration of OpenStack infra - YouTube
salt-master
1. salt ‘kevin’ cmd.run ‘apt-get update’
2. socket.recv{‘tgt’: ‘kevin’,‘fun’: ‘cmd.run’,‘arg’: ‘apt-get update’}
3. Encrypt the payload
4. Publish it via zmq
salt/master.py
salt-minion
1. socket.recv
2. Decrypt the payload
3. self.matcher: Am I the ‘tgt’?
4. self.functions: Loaded from salt/modules/*
5. self.functions[data[‘fun’]](data[‘arg’], …)
6. Return the result via zmq
salt/minion.py
Minion did not return
gru@jerry$ ps aux | grep apt
gru@jerry$ tail –f /var/log/salt/minion
When I run test.ping, why don't the Minions that aren't responding return anything? Returning False would be helpful.
/etc/salt/mastershow_timeout: True
Ubuntu 12.04: upgrade to the zmq from salt PPA
Job Management
topics/jobs/index.html
List active/historic jobs, query job result
{signal,term,kill}_job
• Overview of Puppet's Architecture
• The Architecture of Open Source Applications (Volume 2): Puppet
Salt States
SLS stands for SaLt State
SLS Formulas have historically been called "SLS files"
SLS files are therefore, in reality, just dictionaries
apt-key add
1. salt ‘dave' cp.cache_file 'salt://salt/files/salt.key'/var/cache/salt/minion/files/base/salt/files/salt.key
2.
Renders
ref/renderers/index.html
salt/template.py:compile_template
render_pipe
old: yaml_jinja
new: jinja|yaml
jinja: templateyaml: format
Salt Dafa
ref/modules/all/index.html
ref/states/all/index.html
http://jinja.pocoo.org/docs/dev/templates/
ref/states/index.html
ref/states/layers.html
ref/states/highstate.html
Loader
load_modules
os.listdir(mod_dir)
salt/{grains,pillar,modules,renderers,states,…}
Loader.gen_functions
mod.__grains__ = self.grains
mod.__pillar__ = self.pillar
mod.__salt__ = funcs
ref/states/vars.html
salt/loader.py
Develop Environment
mkvirtualenv salt
pip install –e .
~/.venvs/salt/etc/salt/{master,minion}
tmuxp load salt.yaml
Static Grains
roles
openstack-controller (keystone, nova-conductor, neutron-dhcp-agent, …)
openstack-compute (nova-compute, neutron-ovs-agent)
openstack-data (mysql, rabbitmq)
openstack-slb (keepalived, haproxy)
vlan
204
500
CMDB?
OpenStack Configuration
Configuring OpenStack is nontrivial
topics/development/conventions/formulas.html
apache-formula, rabbitmq-formula
parameterization
repository structure
pillar -> map.jinja -> conf templates
Model -> Controller -> View
All config options must be “declared” in map.jinja
Pillar data can override map.jinja default values
OpenStack Releases
Upstream: grizzly-eol, stable/icehouse
In-house: cloudmgr, orca
openstack/neutron/dhcp-agent.slsopenstack/neutron/icehouse/dhcp-agent.slsopenstack/neutron/grizzly/dhcp-agent.sls
Salt: include, extend
Jinja: import, include, extends
git branch can always be created as a fallback
1. Two Jinja contexts are serialized by the jinja renderer
2. Make sure you tell salt to override the parent one
Task Decomposition
nova.conf should be managed by which sls?
Because salt states are dictionaries, dictionaries cannot have duplicate keys.
dpkg –S /etc/nova/nova.confapt-cache rdepends nova-common
Other nova formulas all include openstack/nova/common.sls
States Relationship
Jinja and YAML output dictionaries
Salt handles the relationship of the dictionary items
salt/modules/state.pysalt/state.py
A feature (bug): you can require an entire sls file, but the included file has to contain at least one state declaration.
salt tries to find states which are declared in the required sls file
Overstate
Databases are not necessarily installed on the same machine
nova-conductor should require database connection string, but not the database itself
Configure the data nodes first, then the controller nodes
ref/states/layers.html#overstate
States Tutorial, Part 5 - Orchestration with Salt
salt/runners/state.py:orchestrate
Stacker
Opinionated devstack replacement
First-time users should still start with devstack
Replace apt-get install with git clone
Replace service restart with tmux
Include/Extend conf templates
All Kinds of Salt
salt-ssh, fabric
Install salt-minion with salt-ssh
Changing minion environment requires a restart
Oops, minions connect to a wrong master
salt-call
topics/tutorials/standalone_minion.html
salt-call state.sls stacker.icehouse
Salt Environments
Configuring pillar is still painful
An army of reference pillar examples is needed
shanghai, nantong-prod, ssshi-dev2, …
Directory Overlay
/srv/salt/pillar/ssshi-dev2
/srv/salt/pillar/ssshi-base
Pillar: Get all controller nodes in the same environment via salt-mine, and generate haproxy pillar entries automatically/dynamically
States: Give me my bashrc/gitconfig/tmux.conf/vimrc
Bigger than Bigger
0-10, 10-100, 100-1000
gitfs backend
Pillar encryption for code review
Orchestration
Upstream openstack-formula
Testing formulas with docker
• LinkedIn - SaltStack for Web Scale – YouTube
• Google - Management at Google Scale – YouTube
• HP Cloud - Automating operations and support with SaltStack - YouTube
Summary
DSL is powerful, you just have to learn it
Salt SLS is a description language
Know the structure of the documentation
Dev + Ops + QA + Support = Problem Solvers
Learn from the best
Effective Operations, Effective Cloud