Are We Done Yet ? Testing Your OpenStack Deployment

download Are We Done Yet ? Testing Your OpenStack Deployment

of 26

  • date post

    15-Jul-2015
  • Category

    Technology

  • view

    730
  • download

    0

Embed Size (px)

Transcript of Are We Done Yet ? Testing Your OpenStack Deployment

  • Accelerating the adoption of Cloud Computing

    Are We Done Yet ? Testing Your OpenStack Deployment

    OpenStack Summit, Paris, France November 4, 2014

  • Solinea, Inc. 2

    Who Am I ?

    Ken Pepple is the Chief Technology Officer of Solinea Prior to founding Solinea, he led the introduction of Internap's OpenStack-based public cloud services while serving as their Director of Cloud Development Author of O'Reilly "Deploying OpenStack book OpenStack code contributor since Bexar release

  • Solinea, Inc.

    !

    Who we are

    3

    Solinea is the leading software and services company that delivers integrated OpenStack solutions, enabling enterprise adoption of Cloud Computing

    Cloud is the only domain we focus on, with vertical industry and horizontal solutions specialization

    Purpose-built for cloud

    Track record of success architecting, building and operating production clouds private and public world-wide

    Proven Delivery Success

    We understand cloud adoption challenges of global enterprise companies

    Enterprise IT Experience

    Pre-configured management and monitoring platform for OpenStack and other Open Infrastructure components

    Differentiating IP

    Built the first OpenStack production clouds and contributors to the platform since its inception

    OpenStack Experience

  • Solinea, Inc.

    Agenda ! Problem statement ! Types of testing ! Testing tools ! Future improvements

    4

  • Solinea, Inc.

    Problem ! Many components, even more connections ! No two clouds are identical

    Hypervisor choices, network choices, storage choices ! Many rarely used code paths (backups ?) ! Operational and integration concerns ! Tools for operators ! Requires skills across all the services

    5

  • Solinea, Inc.

    In a nutshell distributed systems

    OpenStack Object Store

    OpenStack Image Service OpenStack Compute

    OpenStackDashboard

    OpenStack Identity Service

    OpenStack Compute API /

    Admin API

    keystone(service & admin APIs)

    nova-api(OS, EC2, Metadata, Admin)

    nova-consoleauth

    nova-cert/objectstore

    nova-consolenova-*proxy

    VNC/Spice/RDP

    OpenStack Object API

    http://www.solinea.comOpenStack Data Processing

    sahara-api

    sahara-engine saharadatabase

    Queue sahara-conductor

    OpenStack Data Processing API

    (8386)

    OpenStack Identity

    API

    OpenStack Data Processing API

    Queue

    nova-compute

    nova-scheduler

    novadatabase

    OpenStackCompute API

    (8774, 8775, 8776)

    OpenStack Image API

    Horizon

    OpenStack Image API

    (9292)

    identity backend

    swift-proxy

    objectcontaineraccount

    objectstore

    accountDB

    containerDB

    OpenStack Object API

    HTTP(S)

    OpenStackObject API

    OpenStack Identity API

    OpenStack Identity

    API

    OpenStack Identity

    API

    OpenStackImage API

    OpenStack Identity

    API

    OpenStack Image API

    catalog backendtoken backend

    OpenStack Identity

    API(5000)

    hypervisor

    libvirt, XenAPI, etc.

    HTTP(S)

    Amazon Web Services

    EC2 API

    Internet / Enterprise Network

    OpenStack Network Service

    glance-api

    glancedatabase

    OpenStack Block Storage

    OpenStack Block Storage API

    cinder-api

    cinder-volume

    neutron-server

    neutron plugin(s)

    OpenStack Identity

    API

    cinder-scheduler

    cinderdatabase

    OpenStack Network API

    (9696)

    networkprovider

    OpenStack Block

    StorageAPI

    OpenStack Network API

    policy backend

    Queue

    OpenStack Network API

    neutrondatabase

    neutron agent(s)

    nova-conductor

    memcached

    OpenStack Identity

    API

    OpenStack Command Line Tools (nova-client, swift-client, etc.) Cloud Management Tools (CloudForms, Enstratius, etc.) GUI tools (Web Browsers, Cyberduck, iPhone client, etc.)

    volume provider

    cinder-backup

    OpenStack Object API

    OpenStack Identity

    API

    Queue

    OpenStack Block Storage API

    OpenStack Orchestration

    heat-api

    heat-engine

    heatdatabase

    Queue

    cloudwatch-api

    OpenStack Orchestration API

    OpenStack Identity

    API

    OpenStack Compute API

    OpenStack Bock Storage API

    OpenStack Network API

    glance-registry

    OpenStack Metering Service

    ceilometer-api

    ceilometerdatabase

    ceilometer-collector

    OpenStack Database

    trove-api

    trove-taskmgr

    trovedatabase

    trove-conductor

    OpenStack Database API

    (8779)

    OpenStack Block Storage APIOpenStack Orchestration API

    OpenStack Compute API

    Database(mysql, redis, mongo,

    cassandra, etc.)

    Queue

    trove-guestagent

    OpenStack Identity

    API

    6

  • Solinea, Inc.

    What Should We Be Testing? ! Functionality

    User scenarios Not code quality unless running trunk

    ! Non-functional requirements Scalability Performance Security

    ! Operations High availability and failover

    7

  • Solinea, Inc.

    How Should We Test ? ! Automated

    Repeatability Multiple checks

    ! Continual ! Ops not specialists

    Should need coders or QA people Testing should be an easy command

    ! Integrated into operational procedures Let me send a canary in

    8

  • Solinea, Inc.

    Types of Testing ! Unit testing ! Integration ! User Acceptance Tests ! Operational

    9

  • Solinea, Inc.

    Functional Areas ! Networking ! Storage ! Compute ! User functional process ! OUTSIDE applications

    Billing/metering, user signup, audit/compliance

    10

  • Solinea, Inc.

    OpenStack Testing Options ! Tempest ! Rally ! Manual

    11

  • Solinea, Inc.

    Tempest ! OpenStack testing project ! Runs hundreds of tests

    Unit Scenario Stress API CLI

    ! Complex to configure ! Not portable to every cloud deployment configuration ! Requires significant investment in time

    12

  • Solinea, Inc.

    An OpenStack project to create a framework for validating, performance testing and benchmarking OpenStack at scale with Tempest https://wiki.openstack.org/wiki/Rally

    13

    Rally

  • Solinea, Inc.

    Rally Scenario $ rally task start summit/create-and-list-users.json================================================================================Task 57bc6d22-76c2-4450-b834-daa48c41d6e0 is started--------------------------------------------------------------------------------

    ================================================================================Task 57bc6d22-76c2-4450-b834-daa48c41d6e0 is finished.--------------------------------------------------------------------------------

    test scenario KeystoneBasic.create_and_list_usersargs position 0args values:{u'args': {u'name_length': 10}, u'runner': {u'concurrency': 10, u'times': 100, u'type': u'constant'}}+----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+| action | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count |+----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+| keystone.create_user | 1.767 | 3.478 | 5.944 | 4.565 | 4.694 | 100.0% | 100 || keystone.list_users | 0.077 | 0.464 | 1.516 | 1.01 | 1.143 | 100.0% | 100 || total | 1.982 | 3.942 | 6.218 | 4.956 | 5.214 | 100.0% | 100 |+----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+Whole scenario time without context preparation: 40.6584420204

    14

  • Solinea, Inc.

    Rally Database ! Rally stores all scenario/task runs ! Can be used to trend and benchmark results $ rally task list+--------------------------------------+----------------------------+----------+--------+-----+| uuid | created_at | status | failed | tag |+--------------------------------------+----------------------------+----------+--------+-----+| 57bc6d22-76c2-4450-b834-daa48c41d6e0 | 2014-11-05 12:10:38.235211 | finished | False | || 12d34393-0520-4830-9516-e7404935c0e9 | 2014-11-05 12:32:39.632810 | finished | False | || 3b160652-ed3b-4e2b-b09c-14b6d900e18e | 2014-11-05 12:40:21.544483 | running | False | |+--------------------------------------+----------------------------+----------+--------+-----+

    15

  • Solinea, Inc.

    Rally Scenarios { "KeystoneBasic.create_delete_user": [ { "args": { "name_length": 10 }, "runner": { "type": "constant", "times": 100, "concurrency": 10 } } ]}

    16

    Method

    Customization

    Concurrency

  • Solinea, Inc.

    Rally Scenarios ! Authenticate ! Ceilometer ! Cinder ! Designate ! Glance ! Heat ! Keystone

    ! Neutron ! Nova ! quotas ! Requests ! Sahara ! Tempest ! VM ! Zaqar

    17

  • Solinea, Inc.

    Rally Visualization

    18

  • RALLY DEMO

    19

  • Solinea, Inc.

    Operational HA Tests ! Kill process / hung process ! Lost interface ! Lost machine ! Lost rack

    20

  • Solinea, Inc.

    Continual Testing ! Testing should be part of monitoring and

    manageme