Doctor Demo, OPNFV Hackfest (Santa Clara, USA)
-
Upload
carlos-goncalves -
Category
Software
-
view
410 -
download
1
Transcript of Doctor Demo, OPNFV Hackfest (Santa Clara, USA)
![Page 1: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/1.jpg)
Doctor Demo
Carlos Goncalves, NEC
Ryota Mibu, NEC
July 30, 2015
OPNFV Hackfest, Santa Clara, USA
1
![Page 2: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/2.jpg)
Doctor demo
• 2 use cases being showcased
– UC1: infrastructure monitoring
• What will be shown? VMs evacuation on compute node failure
• How it is realized?
– Zabbix as monitoring tool; alerts are published to a messaging queue
– Platform integrated with Nova blueprint proposed to OpenStack
– UC2: virtual resources monitoring
• What will be shown? VNF ACT/SBY scenario
• How is it realized?
– Ceilometer as monitoring tool; alerts are sent to a HTTP callback URL
– Platform integrated with Ceilometer blueprint proposed to OpenStack
2 30 July 2015 Doctor demo, Santa Clara
![Page 3: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/3.jpg)
Blueprint status
3 30 July 2015 Doctor demo, Santa Clara
Project Name Lead contributor Developer Status
Nova New nova API call to mark nova-
compute down * Tomi Juvonen Roman Dobosz (server)
Carlos Goncalves (client)
BP approved & merged
Code under review
Ceilometer Add 'event' type and 'event_rule' to
alarm API * Ryota Mibu Ryota Mibu BP approved & merged
Ongoing implementation
Nova Get valid server state Tomi Juvonen BP under review
M-release
Nova Add notification for service status
change
Balazs Gibizer Balazs Gibizer BP under review
M-release
* These blueprints were accepted in OpenStack Liberty cycle and are being showcased in this demo
For more information and latest status, please refer to https://etherpad.opnfv.org/p/doctor_bps
![Page 4: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/4.jpg)
Doctor demo
• 2 use cases being showcased
– UC1: infrastructure monitoring
• What will be shown? VMs evacuation on compute node failure
• How it is realized?
– Zabbix as monitoring tool; alerts are published to a messaging queue
– Platform integrated with Nova blueprint proposed to OpenStack
–
•
•
–
–
4 30 July 2015 Doctor demo, Santa Clara
![Page 5: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/5.jpg)
Fault Management UC1: infrastructure
5 30 July 2015 Doctor demo, Santa Clara
Public network
Controller
Zabbix
Compute 1
Zabbix Agent
Compute 2
Zabbix Agent
MQ
Consumer
ZabbixMQ
Initial deployment Consumer* is subscribed to a AMQP system
* User-side Manager; consumer of the interfaces produced by the VIM; VNFM, NFVO, or Orchestrator in ETSI NFV terminology
![Page 6: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/6.jpg)
Fault Management UC1: infrastructure
6 30 July 2015 Doctor demo, Santa Clara
Public network
Controller
Zabbix
Compute 1
Zabbix Agent
Compute 2
Zabbix Agent
MQ
Consumer
ZabbixMQ
0. Failure occurs
![Page 7: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/7.jpg)
Fault Management UC1: infrastructure
7 30 July 2015 Doctor demo, Santa Clara
Public network
Controller
Zabbix
Compute 1
Zabbix Agent
Compute 2
Zabbix Agent
MQ
Consumer
1. Failure detected
1
ZabbixMQ
![Page 8: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/8.jpg)
Fault Management UC1: infrastructure
8 30 July 2015 Doctor demo, Santa Clara
Public network
Controller
Zabbix
Compute 1
Zabbix Agent
Compute 2
Zabbix Agent
MQ
Consumer
2
ZabbixMQ
2. Forward alert to ZabbixMQ ZabbixMQ developed to publish alerts to a AMQP system (e.g. RabbitMQ, Qpid)
![Page 9: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/9.jpg)
Fault Management UC1: infrastructure
9 30 July 2015 Doctor demo, Santa Clara
Public network
Controller
Zabbix
Compute 1
Zabbix Agent
Compute 2
Zabbix Agent
MQ
Consumer 3
ZabbixMQ
3. Publish alert to RabbitMQ (pub/sub)
![Page 10: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/10.jpg)
Fault Management UC1: infrastructure
10 30 July 2015 Doctor demo, Santa Clara
Public network
Controller
Zabbix
Compute 1
Zabbix Agent
Compute 2
Zabbix Agent
MQ
Consumer
4
ZabbixMQ
4. Consumer is notified of failure (pub/sub)
![Page 11: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/11.jpg)
Fault Management UC1: infrastructure
11 30 July 2015 Doctor demo, Santa Clara
Public network
Controller
Zabbix
Compute 1
Zabbix Agent
Compute 2
Zabbix Agent
ZabbixMQ
MQ
Consumer
5
5. Consumer invokes fault management action 5.1. Mark compute as down (Nova blueprint)
![Page 12: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/12.jpg)
Fault Management UC1: infrastructure
12 30 July 2015 Doctor demo, Santa Clara
Public network
Controller
Zabbix
Compute 1
Zabbix Agent
Compute 2
Zabbix Agent
MQ
Consumer
6
ZabbixMQ
6. Consumer invokes fault management action 6.1. Evacuate VMs from Compute 1 to Compute 2
![Page 13: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/13.jpg)
Fault Management UC1: infrastructure
13 30 July 2015 Doctor demo, Santa Clara
Public network
Controller
Zabbix
Compute 1
Zabbix Agent
Compute 2
Zabbix Agent
MQ
Consumer 6
ZabbixMQ
1. Failure detected 2. Forward alert to ZabbixMQ 3. Publish alert to RabbitMQ (pub/sub) 4. Consumer is notified of failure (pub/sub) 5 & 6. Consumer invokes fault management action
5
6
4
5
3
2
1
![Page 14: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/14.jpg)
Steps 1-4: failure detection and reporting to Consumer
14 30 July 2015 Doctor demo, Santa Clara
![Page 15: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/15.jpg)
Step 5: Mark compute host state as down by Consumer
15 30 July 2015 Doctor demo, Santa Clara
![Page 16: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/16.jpg)
Step 5: Mark compute host state as down by Consumer
16 30 July 2015 Doctor demo, Santa Clara
![Page 17: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/17.jpg)
Step 6: Evacuate all VMs running on affected compute
node
17 30 July 2015 Doctor demo, Santa Clara
![Page 18: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/18.jpg)
Step 6: Evacuate all VMs running on affected compute
node
18 30 July 2015 Doctor demo, Santa Clara
27 seconds
![Page 19: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/19.jpg)
Compute node 1 is back after admin maintenance
Setting state UP
19 30 July 2015 Doctor demo, Santa Clara
![Page 20: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/20.jpg)
20 30 July 2015 Doctor demo, Santa Clara
![Page 21: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/21.jpg)
21 30 July 2015 Doctor demo, Santa Clara
Steps 1-4 Zabbix took 6 seconds to
send out alert notification
![Page 22: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/22.jpg)
22 30 July 2015 Doctor demo, Santa Clara
Step 5
![Page 23: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/23.jpg)
23 30 July 2015 Doctor demo, Santa Clara
Step 6
![Page 24: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/24.jpg)
24 30 July 2015 Doctor demo, Santa Clara
Consumer took 0.36 seconds to react
![Page 25: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/25.jpg)
25 30 July 2015 Doctor demo, Santa Clara
Compute node is back online
![Page 26: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/26.jpg)
Doctor demo
• 2 use cases being showcased
–
•
•
–
–
– UC2: virtual resources monitoring
• What will be shown? VNF ACT/SBY scenario
• How is it realized?
– Ceilometer as monitoring tool; alerts are sent to a HTTP callback URL
– Platform integrated with Ceilometer blueprint proposed to OpenStack
26 30 July 2015 Doctor demo, Santa Clara
![Page 27: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/27.jpg)
Fault Management UC2: virtual resource
27 30 July 2015 Doctor demo, Santa Clara
VNF ACT
VNF SBY
LB
2 VMs running a VNF in ACT/SBY mode logically
connected to a load balancer
![Page 28: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/28.jpg)
Fault Management UC2: virtual resource
28 30 July 2015 Doctor demo, Santa Clara
Public network
Controller
Ceilometer
Compute 1
Ceilometer Agent
Compute 2
Consumer
VNF ACT
VNF SBY
LB
SBY
ACT
LB Ceilometer Agent
Initial deployment VNF Active / Standby mode
![Page 29: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/29.jpg)
Fault Management UC2: virtual resource
29 30 July 2015 Doctor demo, Santa Clara
Public network
Controller
Ceilometer
Compute 1
Ceilometer Agent
Compute 2
Consumer
VNF ACT
(ERR)
VNF SBY
LB
SBY
ACT
LB Ceilometer Agent
0. failure occurs
![Page 30: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/30.jpg)
Fault Management UC2: virtual resource
30 30 July 2015 Doctor demo, Santa Clara
Public network
Controller
Ceilometer
Compute 1
Ceilometer Agent
Compute 2
Consumer
VNF ACT
(ERR)
VNF SBY
LB
SBY
ACT
LB Ceilometer Agent
1
1. Failure detected
![Page 31: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/31.jpg)
Fault Management UC2: virtual resource
31 30 July 2015 Doctor demo, Santa Clara
Public network
Controller
Ceilometer
Compute 1
Ceilometer Agent
Compute 2
Consumer
VNF ACT
(ERR)
VNF SBY
LB
SBY
ACT
LB Ceilometer Agent
2
2. Ceilometer notifies Consumer of failure (Ceilometer blueprint) 2.1. HTTP callback
![Page 32: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/32.jpg)
Fault Management UC2: virtual resource
32 30 July 2015 Doctor demo, Santa Clara
Public network
Controller
Ceilometer
Compute 1
Ceilometer Agent
Compute 2
Consumer
VNF ERR
VNF ACT
LB
ACT
ERR
LB Ceilometer Agent
3
3. Activate SBY server
![Page 33: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/33.jpg)
Fault Management UC2: virtual resource
33 30 July 2015 Doctor demo, Santa Clara
Public network
Controller
Ceilometer
Compute 1
Ceilometer Agent
Compute 2
Consumer
VNF ERR
VNF ACT
LB
ACT
ERR
LB Ceilometer Agent
4
4. Switch to SBY configuration
![Page 34: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/34.jpg)
Fault Management UC2: virtual resource
34 30 July 2015 Doctor demo, Santa Clara
Public network
Controller
Ceilometer
Compute 1
Ceilometer Agent
Compute 2
Consumer
VNF ERR
VNF ACT
LB
ACT
ERR
LB Ceilometer Agent
3
1. Failure detected 2. Ceilometer notifies Consumer of failure 3. Activate SBY server 4. Switch to SBY configuration
1 2
4
![Page 35: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/35.jpg)
UC2 Deployment
35 30 July 2015 Doctor demo, Santa Clara
tipica@d718:~$ nova list +--------------------------------------+---------+--------+------------+-------------+------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+---------+--------+------------+-------------+------------------+ | bec98651-cbbb-4036-bba8-dedd1dc96e76 | server1 | ACTIVE | - | Running | private=10.0.0.4 | | 41e092a4-eb39-44be-846d-ecdee007f9b8 | server2 | ACTIVE | - | Running | private=10.0.0.5 | +--------------------------------------+---------+--------+------------+-------------+------------------+ tipica@d718:~$ neutron lb-member-list +--------------------------------------+----------+---------------+--------+----------------+--------+ | id | address | protocol_port | weight | admin_state_up | status | +--------------------------------------+----------+---------------+--------+----------------+--------+ | 49f1b303-dfc7-4076-9377-059da3fe1c26 | 10.0.0.4 | 80 | 1 | True | ACTIVE | | 7e1b3152-afe0-4875-8442-01d1fcb75165 | 10.0.0.5 | 80 | 1 | False | ACTIVE | +--------------------------------------+----------+---------------+--------+----------------+--------+ tipica@d718:~$ neutron lb-vip-list +--------------------------------------+----------+----------+----------+----------------+--------+ | id | name | address | protocol | admin_state_up | status | +--------------------------------------+----------+----------+----------+----------------+--------+ | 926d6bfa-2b7f-4fc7-9cb5-f05961c3cc34 | pool.vip | 10.0.0.3 | HTTP | True | ACTIVE | +--------------------------------------+----------+----------+----------+----------------+--------+ tipica@d718:~$ neutron floatingip-list +--------------------------------------+------------------+---------------------+--------------------------------------+ | id | fixed_ip_address | floating_ip_address | port_id | +--------------------------------------+------------------+---------------------+--------------------------------------+ | a037ce66-efe2-4c91-af81-6f9b4c1d0630 | 10.0.0.3 | 172.24.4.3 | 01d8d4ef-40e1-47eb-93d2-1dbc4e741297 | +--------------------------------------+------------------+---------------------+--------------------------------------+
![Page 36: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/36.jpg)
UC2 Initial State
36 30 July 2015 Doctor demo, Santa Clara
tipica@d718:~$ neutron lb-member-list +--------------------------------------+----------+---------------+--------+----------------+--------+ | id | address | protocol_port | weight | admin_state_up | status | +--------------------------------------+----------+---------------+--------+----------------+--------+ | 49f1b303-dfc7-4076-9377-059da3fe1c26 | 10.0.0.4 | 80 | 1 | True | ACTIVE | | 7e1b3152-afe0-4875-8442-01d1fcb75165 | 10.0.0.5 | 80 | 1 | False | ACTIVE | +--------------------------------------+----------+---------------+--------+----------------+--------+
tipica@d718:~$ ssh -i demokey.private [email protected] "/bin/systemctl status app.service | grep Active" Active: active (running) since Wed 2015-07-29 16:19:05 UTC; 12min ago tipica@d718:~$ ssh -i demokey.private [email protected] "/bin/systemctl status app.service | grep Active" Active: inactive (dead)
tipica@d718:~$ curl http://172.24.4.3/ Served by server1. tipica@d718:~$ curl http://172.24.4.3/ Served by server1. tipica@d718:~$ curl http://172.24.4.3/ Served by server1. tipica@d718:~$ curl http://172.24.4.3/ Served by server1. tipica@d718:~$ curl http://172.24.4.3/ Served by server1.
![Page 37: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/37.jpg)
UC2 Failure Injection and Auto Healing
37 30 July 2015 Doctor demo, Santa Clara
tipica@d718:~$ ps aux | grep instance-00000001 | grep -v grep | sed -e 's/-smbios.*$//' libvirt+ 27953 1.3 1.1 3676568 272476 ? Sl 01:46 0:07 qemu-system-x86_64 -enable-kvm -name instance-00000001 -S -machine pc-i440fx-trusty,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid bec98651-cbbb-4036-bba8-dedd1dc96e76 tipica@d718:~$ sudo kill 27953; date "+%H:%M:%S.%N" 01:55:05.044890318
![Page 38: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/38.jpg)
UC2 Recovered State
38 30 July 2015 Doctor demo, Santa Clara
tipica@d718:~$ neutron lb-member-list +--------------------------------------+----------+---------------+--------+----------------+--------+ | id | address | protocol_port | weight | admin_state_up | status | +--------------------------------------+----------+---------------+--------+----------------+--------+ | 49f1b303-dfc7-4076-9377-059da3fe1c26 | 10.0.0.4 | 80 | 1 | False | ACTIVE | | 7e1b3152-afe0-4875-8442-01d1fcb75165 | 10.0.0.5 | 80 | 1 | True | ACTIVE | +--------------------------------------+----------+---------------+--------+----------------+--------+
tipica@d718:~$ ssh -i demokey.private [email protected] "/bin/systemctl status app.service | grep Active" ^C tipica@d718:~$ ssh -i demokey.private [email protected] "/bin/systemctl status app.service | grep Active" Active: active (running) since Wed 2015-07-29 16:55:06 UTC; 5min ago
tipica@d718:~$ curl http://172.24.4.3/ Served by server2. tipica@d718:~$ curl http://172.24.4.3/ Served by server2. tipica@d718:~$ curl http://172.24.4.3/ Served by server2. tipica@d718:~$ curl http://172.24.4.3/ Served by server2. tipica@d718:~$ curl http://172.24.4.3/ Served by server2.
![Page 39: Doctor Demo, OPNFV Hackfest (Santa Clara, USA)](https://reader033.fdocuments.in/reader033/viewer/2022042908/58f387621a28ab9f4e8b45cb/html5/thumbnails/39.jpg)
Questions?
Doctor demo, Santa Clara 39 30 July 2015