ACTION POTENTIALS Chapter 11 Part 2 HONORS ANATOMY & PHYSIOLOGY.
Anatomy of an action
-
Upload
gordon-chung -
Category
Technology
-
view
46 -
download
2
Transcript of Anatomy of an action
Mining the event storm
Vladik RomanovskyEngineer
The Anatomy of an Action
EngineerGordon Chung
OpenStack is a wonderful place
when you use OpenStack you might see this
WTF???
if you’re lucky, you might find the real error![instance: e7933ceb-d1e7-42fe-9f37-d275ebd375bd] Instance failed to spawn
Traceback (most recent call last):......ProcessExecutionError: Unexpected error while running command.Command: qemu-img convert -O raw /opt/stack/data/nova/instances/_base/7434c85f2968d2cfb05b07d8c769d7d938cec5e8.part /opt/stack/data/nova/instances/_base/7434c85f2968d2cfb05b07d8c769d7d938cec5e8.convertedExit code: 1Stdout: u''Stderr: u'qemu-img: error while reading sector 0: Input/output error\n'
Debugging be Hard
• actions consists of multiple steps• asynchronous calls that can cause
timing issues• distributed nature of OpenStack
can make it difficult to debug• parsing log files are easy -- if you’re
a robot
Use Case: Creating an Instance
Creating an Instance
api conductor scheduler computemanager
buildnetwork
buildstorage
startguest
Creating an Instance
api conductor scheduler computemanager
buildnetwork
buildstorage
startguest
FAIL HERE
Creating an Instance
api conductor scheduler computemanager
buildnetwork
buildstorage
startguest
FAIL HERE
Creating an Instance
conductor scheduler computemanager
buildnetwork
buildstorage
startguestapi
FAIL HERE
Creating an Instance
api conductor scheduler computemanager
buildnetwork
buildstorage
startguest
notification bus
Creating an Instance
conductor scheduler computemanager
buildnetwork
buildstorage
startguestapi
FAIL HERE
notification bus
OpenStack Events
• most services emit notifications for some discrete events• the content of notification represent that state of the
environment, resource, etc… at the point in time• notifications are defined by a type to describe content• nova: compute.instance.create.*, scheduler.create_volume• neutron: port.create.*, network.create.*• cinder: volume.detach.*, volume.create.*• keystone: identity.user.*, identity.project.*• and a lot more...
Creating an Instance
api conductor scheduler hostmanager
buildnetwork
buildstorage
startguest
notification bus
consumer?
Ceilometer
• telemetry project in OpenStack• notification agent which consumes messages• listens to the queues of each OpenStack service• picks specific measurement values from notifications and
builds meters
but wait, there’s more!
every notification is also captured as an Event
Creating an Instance
api conductor scheduler hostmanager
buildnetwork
buildstorage
startguest
notification bus
ceilometer notification agentMeters Events
Ceilometer Events
• initially implemented in Icehouse (part of StackTach integration)
• an Event represents the state of an object in an OpenStack service at a point in time.
• built from INFO and ERROR level notifications emitted by all services
• ability to normalise messages by mapping key attributes from notification messages to a common name
Ceilometer Event Model
• message id• event type• timestamp• traits
• queryable, indexed attributes
• ie. payload.x.y.z => attr1• raw
• full notification
Ceilometer Event Processing
• all events are forced through pipelines
• events can be published to multiple targets• database• file• queue• http
Benefits of Centralised Events
• potential lost of data if logging locally• normalisation of data• event flow across services gives context
• individual events means nothing• end to end flow means something
connecting the dots…
Debugging be Easier
• we wanted a view to show all the events of a given action by a resource
• be able to see any errors• temporally aware -- order of events• show the flow and context of events
postmortem analysis using Elasticsearch
ElasticSearch
• document-oriented, schema free database• built on top of Apache Lucene
• focused on providing full-text search capabilities• distributed, highly available, real time db• kibana - gui interface to database
KIBANA!!!
KIBANA!!!
HORIZON!!!
HORIZON!!!
Extending Events
• there is a lot of data that isn’t published• the data that is published is disorganised• extending support in horizon
• drilling down into event to view full raw data• filter options - time range, events for a specific request
• ceilometer• alarm on events• build metrics from events
thank you
BACKUP
Horizon Events Prototype, by George Peristerakis
https://github.com/enovance/horizon/tree/event-prototype