20161027 hands on-gnocchicloudkitty
-
Upload
claire-gayan -
Category
Technology
-
view
33 -
download
0
Transcript of 20161027 hands on-gnocchicloudkitty
Configure your cloud to be able to chargeyour users using official OpenStack
components!
27 octobre 2016
Ref 20160424-ObjectifLibre-HandsOntoConfigureYourCloudtoBeAbletoChargeYourUsersUsingOfficialOpenStackComponents
Objectif Librehttp://www.objectif-libre.com
Table des matières
1 Let’s meet your hosts ! 2
2 Today’s plate ! 32.1 Start by getting your image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2 Grab an IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3 Your infrastructure ! 43.1 What’s to expect in that image ? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
4 All you need is good tools ! 54.1 Ceilometer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54.2 Gnocchi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64.3 CloudKitty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
5 Today’s setup 75.1 What do we use ? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75.2 Why do we need Ceilometer and Gnocchi ? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
6 Let’s cook gnocchis ! 86.1 Who are you ? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86.2 Installing components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86.3 Configuring Keystone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86.4 Creating Gnocchi database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106.5 Configuring Gnocchi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106.6 Enable keystone Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106.7 Update the database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106.8 Start the daemons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
7 Collecting metrics 117.1 Installing ceilometer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
8 CloudKitty’s turn ! 128.1 Installing components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128.2 Configuring Keystone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128.3 Creating the CloudKitty database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138.4 Configuring CloudKitty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148.5 Setting up rating rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
9 Here comes the pricing policy ! 169.1 First of all, restart apache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169.2 Let’s create your user/project ! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169.3 Tell CloudKitty to charge your tenant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169.4 Let’s use Horizon for the policy ! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169.5 Let’s fill some data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199.6 Resetting calculations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199.7 Reporting view . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199.8 Predictive Pricing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
i
10 It is time for questions ! 21
ii
Configure your cloud to be able to charge your users using official OpenStack components!
∙ Let’s meet your hosts
∙ Your infrastructure !
∙ All you need is good tools !
∙ Our setup
∙ Let’s cook gnocchis !
∙ Collecting the metrics
∙ Cloudkitty’s turn
∙ Here comes the pricing policy
∙ Reporting
∙ It is time for questions
Table des matières 1
Let’s meet your hosts !
∙ Stéphane Albert (Objectif Libre)Co-father of CloudKitty / Former CloudKitty PTL
∙ Christophe Sauthier (Objectif Libre)CEO of Objectif Libre / PTL and co-Father of CloudKitty
∙ Julien Danjou (Red Hat)PTL Telemetry / Father of Gnocchi
∙ Maxime Cottret (Objectif Libre)Cloud Consultant / CloudKitty Contributor
∙ Luka Peschke (Objectif Libre)DevOps trainee / Cloudkitty Contributor
2
Today’s plate !
Start by getting your image
Go to http ://olib.re/handson-bcn and save it !
Grab an IP
Go to https ://quotidien.framapad.org/p/osa-b and put your name next to an IP
3
Your infrastructure !
What’s to expect in that image ?
Some essential OpenStack components :
∙ A mariadb database.
∙ An AMQP (rabbitmq)
∙ memcached
∙ Keystone bootstrapped with an user (admin, password : adminpass) on a single project (admin).
∙ Glance
∙ Nova
∙ Horizon
∙ Neutron
Login : root / password : root
* PLEASE CHANGE IT * (using passwd once loggued by exemple)
4
All you need is good tools !
Ceilometer
OpenStack measurement project
Ceilometer (Telemetry) collects the usage of all the resources in an OpenStack Cloud.
Those informatios are stored in databases, and a REST API is provided to access them.
Some advanced capabilities (like alarming with aodh)
ArchitectureCeilometer is composed of various parts, the main ones are :
∙ ceilometer-api (controler) : API server∙ ceilometer-collector (controler) : reads AMQP messages (from other components like nova)∙ ceilometer-agent-central (controller) : polls some metrics directly∙ ceilometer-agent-compute (compute node) : fetch informations related to instances (like CPU
usage)
MetricsMetrics are the elements storing cloud usage, like :
∙ cpu_util : CPU usage percentage∙ network.incoming.bytes : number of octets received on a network port∙ disk.read.bytes : number of bytes read from a disk
Metrics can be of different types :∙ gauge : value at the precise moment of collection∙ cumulative : sum of all the previous values∙ delta : difference since the last value
Samplessamples are the basis of ceilometer calculations : raw data stored after each polling or message received.
Each sample is composed of a record date, a name, a type, a value and an unit.
Scalability issueA scalability issue raised due to the following reasons :
∙ Many requests done by API requires full scan of all stored samples∙ one can filter on any fields∙ metadata are not indexed (since free-form)
Some improvements have been tried, but many felt it wasn’t the good approach...
5
Configure your cloud to be able to charge your users using official OpenStack components!
Gnocchi
TDBaaS (Time Series Database as a Service) project started under the Ceilometer program um-brella.
The idea is to do aggregation of samples over a period of time to only store a reduced amount of data.
Gnocchi can be used as a storage backend for Ceilometer, and since it provides its own API some third partysoftware can uses the mecanism avoiding the scalability issue of Ceilometer.
Archive PolicyThe number of samples can be different over the time
∙ 1 every second over the last day∙ 1 every minute for the last week∙ 1 every hour for the last 6 months
The indexerGnocchi supports many indexer (Postgres, Mysql) that are simply in charge of the relationship between metrics andresources.
Every known OpenStack resources have a model, dropping the free-form metadata that were used in Ceilometer(and thus improving speed/scalability)
CloudKitty
An Open Source rating/chargeback component for OpenStack∙ Official component since Sept 2015∙ Developped in python with all OpenStack best practices∙ Integrated with Horizon∙ Interfaced with both Ceilometer and Gnocchi∙ Uber Modular
Modularity4 Levels of modularity
∙ Collection of usage∙ Pricing Policy∙ Output format∙ Output Storage
A Closer look at the Hashmap moduleAims at matching arbitrary values from a service type to resource metadata
∙ Group : Define set of rules that are processed together∙ Service : Match a CK Service (compute, network,..)∙ Field : Match a resource metadata field (flavor, image_id,...)∙ Mapping
∘ Can be attached to a service or a field∘ Map the value of a metadata to a cost and a type of cost
∙ Threshold∘ Can be attached to a service or a field∘ Define thresold levels on a metadata to a cost
4.2. Gnocchi 6
Today’s setup
What do we use ?
We are going to use Ceilometer/Gnocchi to collect metrics, CloudKitty for the rating, and Gnocchi to store them.
Why do we need Ceilometer and Gnocchi ?
Ceilometer is able to collect AND to store metrics. However, Gnocchi’s storage is much more efficient, that’s whywe use it.
7
Let’s cook gnocchis !
Who are you ?
First of all, export some environment variables needed for keystone authentication :
source /root/admin.sh
Installing components
Gnocchi daemons
yum -y install openstack-gnocchi-{api,carbonara,indexer-sqlalchemy,metricd}
Gnocchi client
yum -y install python-gnocchiclient
Configuring Keystone
Adding service and enpoint information
openstack service create --name gnocchi metric
Field Valueenabled Trueid c1daec9e8fc44abb84c4b047b10de33aname gnocchitype metric
openstack endpoint create --region RegionOne \GNOCCHI_ID internal \http://127.0.0.1:8000/
8
Configure your cloud to be able to charge your users using official OpenStack components!
Field Valueenabled Trueid 6435d3148bda4384900a4a4a3a2c339einterface internalregion RegionOneregion_id RegionOneservice_id GNOCCHI_IDservice_name gnocchiservice_type metricurl http://127.0.0.1:8000/
openstack endpoint create --region RegionOne \GNOCCHI_ID admin \http://127.0.0.1:8000/
Field Valueenabled Trueid c6abad5d91364ef2ab1784d3d0987449interface adminregion RegionOneregion_id RegionOneservice_id GNOCCHI_IDservice_name gnocchiservice_type metricurl http://127.0.0.1:8000/
openstack endpoint create --region RegionOne \GNOCCHI_ID public \http://127.0.0.1:8000/
Field Valueenabled Trueid 52fa79e7fbd74c9d994c1eaca04c530einterface publicregion RegionOneregion_id RegionOneservice_id GNOCCHI_IDservice_name gnocchiservice_type metricurl http://127.0.0.1:8000/
Creating gnocchi user
openstack user create --project service --password password gnocchi
Field Valuedefault_project_id f4a80b4777554cbcaa3074c95be7fe3edomain_id defaultenabled Trueid 7fb1ce78bb35436098cdb46cbafff67fname gnocchi
openstack role add --user gnocchi --project service admin
Creating Gnocchi database
6.3. Configuring Keystone 9
Configure your cloud to be able to charge your users using official OpenStack components!
mysql -uroot -pmysqlpass << EOFCREATE DATABASE gnocchi;GRANT ALL PRIVILEGES ON gnocchi.* TO 'gnocchi'@'localhost' \IDENTIFIED BY 'gnocchidbpassword';GRANT ALL PRIVILEGES ON gnocchi.* TO 'gnocchi'@'%' \IDENTIFIED BY 'gnocchidbpassword';EOF
The mysql root password is mysqlpass
Configuring Gnocchi
cp /root/handson_files/gnocchi.conf /etc/gnocchi/gnocchi.conf
We are going to use Gnocchi’s file storage :
[storage]driver = filefile_basepath = /var/lib/gnocchi
Enable keystone Authentication
Edit the /etc/gnocchi/api-paste.ini file by adding keytone authentication to pipeline:main
[pipeline:main]pipeline = gnocchi+auth
Update the database
By default, only the default resource type exists in gnocchi. This will create all resource types needed by ceilometer.
gnocchi-upgrade --create-legacy-resource-types
Start the daemons
The openstack-gnocchi-api service requires Ceilometer to be installed, so we are going to use the gnocchidaemons instead.
First, install screen :
yum -y install screen
Then, start gnocchi daemons :
screen -d -m /usr/bin/gnocchi-apiscreen -d -m /usr/bin/gnocchi-metricd
6.4. Creating Gnocchi database 10
Collecting metrics
Installing ceilometer
Installing componentsWhen gnocchi is installed, Ceilometer’s API and storage are automatically disabled, so we don’t need to installthem.
yum -y install openstack-ceilometer-{collector,polling,notification}
Creating the ceilometer user
openstack user create --project service --password password ceilometer
Field Valuedefault_project_id f4a80b4777554cbcaa3074c95be7fe3edomain_id defaultenabled Trueid 7fb1ce78bb35436098cdb46cbafff67fname ceilometer
openstack role add --user ceilometer --project service admin
Configuring Ceilometer
cp /root/handson_files/ceilometer.conf /etc/ceilometer/ceilometer.conf
Starting Ceilometer daemons
systemctl start openstack-ceilometer-collectorsystemctl start openstack-ceilometer-pollingsystemctl start openstack-ceilometer-notification
11
CloudKitty’s turn !
Installing components
CloudKitty daemons
yum -y install openstack-cloudkitty-{api,processor}
CloudKitty dashboard
yum -y install openstack-cloudkitty-ui
CloudKitty client
yum -y install python-cloudkittyclient
Configuring Keystone
Adding service and enpoint information
openstack service create --name cloudkitty rating
Field Valueenabled Trueid 88a1304ef6124b888463fd4cbc7c0e51name cloudkittytype rating
openstack endpoint create --region RegionOne \CLOUDKITTY_ID internal \http://127.0.0.1:8889/
Field Valueenabled Trueid 6435d3148bda4384900a4a4a3a2c339einterface internalregion RegionOneregion_id RegionOneservice_id CLOUDKITTY_IDservice_name cloudkittyservice_type ratingurl http://127.0.0.1:8889/
12
Configure your cloud to be able to charge your users using official OpenStack components!
openstack endpoint create --region RegionOne \CLOUDKITTY_ID admin \http://127.0.0.1:8889/
Field Valueenabled Trueid c6abad5d91364ef2ab1784d3d0987449interface adminregion RegionOneregion_id RegionOneservice_id CLOUDKITTY_IDservice_name cloudkittyservice_type ratingurl http://127.0.0.1:8889/
openstack endpoint create --region RegionOne \CLOUDKITTY_ID public \http://127.0.0.1:8889/
Field Valueenabled Trueid 52fa79e7fbd74c9d994c1eaca04c530einterface publicregion RegionOneregion_id RegionOneservice_id CLOUDKITTY_IDservice_name cloudkittyservice_type ratingurl http://127.0.0.1:8888/
Adding rating role
openstack role create rating
Field Valuedomain_id id name None 66af44925dfd403e8027ba89fdcd48e1 rating
Creating cloudkitty user
openstack user create --project service --password password cloudkitty
Field Valuedefault_project_id f4a80b4777554cbcaa3074c95be7fe3edomain_id defaultenabled Trueid 3a524d3f98cb49cea49d46e73433299cname cloudkitty
openstack role add --user cloudkitty --project service admin
Creating the CloudKitty database
mysql -uroot -pmysqlpass << EOFCREATE DATABASE cloudkitty;GRANT ALL PRIVILEGES ON cloudkitty.* TO 'cloudkitty'@'localhost' \IDENTIFIED BY 'ckdbpassword';
8.3. Creating the CloudKitty database 13
Configure your cloud to be able to charge your users using official OpenStack components!
GRANT ALL PRIVILEGES ON cloudkitty.* TO 'cloudkitty'@'%' \IDENTIFIED BY 'ckdbpassword';EOF
Configuring CloudKitty
cp /root/handson_files/cloudkitty.conf /etc/cloudkitty/cloudkitty.conf
By default, the collect period is 3600s, and only the compute service is enabled. To get quick results, we will set thecollect period to 90s.
[collect]collector=gnocchiperiod=90services=compute, image, volume
Updating the database
cloudkitty-dbsync upgrade| 2016-10-13 02:24:13.908 2816 DEBUG oslo_db.api [-] Loading backend 'sqlalchemy' from 'cloudkitty.db.sqlalchemy.api' _load_backend /usr/lib/python2.7/site-packages/oslo_db/api.py:234| INFO [alembic.runtime.migration] Context impl MySQLImpl.| INFO [alembic.runtime.migration] Will assume non-transactional DDL.| INFO [alembic.runtime.migration] Running upgrade -> 464e951dc3b8, Initial migration| INFO [alembic.runtime.migration] Running upgrade 464e951dc3b8 -> 2ac2217dcbd9, Added support for meta collector| INFO [alembic.runtime.migration] Running upgrade 2ac2217dcbd9 -> 385e33fef139, Added priority to modules_state.| INFO [alembic.runtime.migration] Context impl MySQLImpl.| INFO [alembic.runtime.migration] Will assume non-transactional DDL.| INFO [alembic.runtime.migration] Running upgrade -> 3dd7e13527f3, Initial migration| INFO [alembic.runtime.migration] Running upgrade 3dd7e13527f3 -> 4fa888fd7eda, Added threshold support.| INFO [alembic.runtime.migration] Running upgrade 4fa888fd7eda -> 54cc17accf2c, Fixed constraint name.| INFO [alembic.runtime.migration] Context impl MySQLImpl.| INFO [alembic.runtime.migration] Will assume non-transactional DDL.| INFO [alembic.runtime.migration] Running upgrade -> 4f9efa4601c0, Initial migration.
Setting up the storage
cloudkitty-storage-init| INFO [alembic.runtime.migration] Context impl MySQLImpl.| INFO [alembic.runtime.migration] Will assume non-transactional DDL.| INFO [alembic.runtime.migration] Running upgrade -> 4c2f20df7491, Initial migration.
Restarting the cloudkitty-api
systemctl restart cloudkitty-api
Setting up rating rules
List available rating modules :
cloudkitty module-list
8.4. Configuring CloudKitty 14
Configure your cloud to be able to charge your users using official OpenStack components!
Module Enabledhashmap Falsenoop Truepyscripts False
Enable the hashmap rating module :
cloudkitty module-enable -n hashmap
Module Enabledhashmap True
8.5. Setting up rating rules 15
Here comes the pricing policy !
First of all, restart apache
systemctl restart httpd
Let’s create your user/project !
openstack project create summitopenstack user create --password password --project summit summit_useropenstack role add --user summit_user --project summit admin
Tell CloudKitty to charge your tenant
openstack role add --user cloudkitty --project summit rating
Let’s use Horizon for the policy !
Log into horizon at the following url http://YOUR_IP/dashboard using the newly created user (sum-mit_user/password).
It is time to create the pricing policy !
Creation of the serviceGo to the configuration panel of the Hashmap module (inside Admin > Rating > Hashmap). Create a service called“compute”
Creation of the instance_uptime groupCreate a group (on the Groups tab). A group is a logical groupment of rating rules that will be defined later.
Creation of the fieldThen, we need to choose the name we want to give to the metric in our policy. It will be named flavor_id in ourcase.
Click on the name of the newly created field and select “Create new Mapping”.
∙ Value : the id of the flavor that will be charged by this rule. fooflavor in our case
∙ Type : it is a flat price for that rule
16
Configure your cloud to be able to charge your users using official OpenStack components!
9.4. Let’s use Horizon for the policy ! 17
Configure your cloud to be able to charge your users using official OpenStack components!
9.4. Let’s use Horizon for the policy ! 18
Configure your cloud to be able to charge your users using official OpenStack components!
∙ Group : select the group you just created (instance_uptime)
That’s it ! You have defined a rule to charge your users depending on the used flavor. Let’s create a second one witha different flavor id : barflavor.
Let’s fill some data
You can wait here for hours to use your “Cloud” or use the script :)
python /root/handson_files/gnocchi_script.py
Resetting calculations
systemctl stop cloudkitty-processormysql -u root -pmysqlpass -e 'delete from ghybrid_dataframes;' cloudkittysystemctl start cloudkitty-processor
Now we will have to wait for the processor to fill the database.
G**Pro-tip -> This is the trainee’s h@Xx0r command to check :**
while truedo mysql -uroot -pmysqlpass -e \'select * from ghybrid_dataframes order by id desc limit 1' cloudkitty \| tail -n 1sleep 5done
Reporting view
If you go into Project/rating/reporting, you should have something like this :
9.5. Let’s fill some data 19
Configure your cloud to be able to charge your users using official OpenStack components!
Predictive Pricing
openstack flavor list
Create a new mapping in the flavor_id field use the id of the m1.nano flavor as value.
Create an image_id field, and create a new mapping inside it with the ID of the glance image myimage thesame way you added the m1.nano flavor. Decide that it is a rate and it should be 10% more expensive than theusual price.
Look at the predictive pricing !Launch a few instances with the m1.nano flavor, and wait for a few minutes.
9.8. Predictive Pricing 20
It is time for questions !
Or later :
We’ll be happy to send you the latest release of these slides !
21