Monitoring IAAS & PAAS Solutions

Monitoring & Managing IaaS & PaaS Solutions with Hyperic HQIsaac Christoffersen – inCommon, Inc

SpringOne 2GX 2009. All rights reserved. Do not distribute without permission.

ADVOCATE

“When the people are invested with the power to control their own destiny, they take an interest.” – Michael Tiemann

“Simplicity is complexity resolved” – Constantin Broncusi

Goal – The Intelligent Data Center


Self-healing from power, network and system outages

Enabling active participation by consumers in demand response

Operating resiliently against physical and cyber attack

Providing data quality and integrity for 21st century needs

Accommodating all computation and storage options

Enabling new products, services, and markets

Optimizing assets and operating efficiently

The effective and efficient data center is responsive to the needs of the organizations that it supports and it is built upon a dynamic infrastructure that can be managed with business rules

Reality – Far from the goal


• Rip and replace technology upgrades

• Vendor driven licensing agreements are not flexible enough to meet changing operational demands

• Tough staffing decisions – Testers or more call center staff

• Unable to keep an accurate inventory of systems and software within the infrastructure

• Approaching space, power and cooling capacities despite efficiency improvements

• Failing to handle the increasing data and growing user demands

• Failing to satisfy scalability, availability and agility targets

• Long term maintenance contracts tie up budgets long after components have depreciated

Technological advances in the energy efficiency, server virtualization, networking, storage, software design and data management are not being utilized to their fullest potential

Opportunity Obstacles = Innovation

Opportunity• Virtualization• Consolidation• Standardization• Provisioning & Self-

Regeneration• Services• Cloud Computing

Obstacle• Server Sprawl• Density• One Ring to Rule Them All• Bob the Builder, Design By

Committee • Legacy• Cloud Computing

– Clarity, “Throw It Over the Fence”


U

System Provisioning Complexity


ToolsPeople

Policy

LinuxXenHyperic & DroolsCobbler & KoanPuppet

Linux AdministratorsSAN EngineersNetwork Engineers

Process

Infrastructure

Service Level AgreementsBudgetsStaffing Guidance

Procurement ProcessOrder Fulfillment ProcessStaff Recruitment & Hiring

I need 3 servers to use for the development of a proof of concept. It's

going to be the next big thing!

Commodity ServersSANsNetworking EquipmentPowerHVAC Systems

Buzz or the Way Forward?


Cloud Infrastructure

IaaS

PaaS

SaaS

Infrastructure as a Service (IaaS) Architectures

Platform as a Service (PaaS)Architectures

Software as a Service (SaaS)

Architectures


SaaS


PaaS

SaaS


IaaS

PaaS


PaaS


IaaS

Source: NIST, Information Technology Laboratoryhttp://csrc.nist.gov/groups/SNS/cloud-computing/cloud-computing-v25.ppt

Service Layers


Infrastructure

Software

Platform

DataCenter

Server, Storage, & Network provisioning

Database ProvisioningWeb Site Provisioning & Hosting

Application ProvisioningService CompositionComposable Applications

Capability ProvisioningEnterprise Service Providers

How do I get the most of my IT and personnel resources?

How do I store & retrieve my inventory data?How do I deploy my web services?

How can I collaborate across the enterprise?How can I view my corporate sales data?

How can I build my web hosting business?How can I manage my customer relationships?How can I run my company's IT department?

The Intelligent Data Center is built on top of separate service layers that make it easier to address individual business concerns across different technical domains. It marks the shift from technology

and business driven solutions to the era where technology and business both share the driver's seat .


"As we enjoy great advantages from the inventions of others, we should be glad of an opportunity to serve others by any invention of ours; and this we should do freely and generously." – Benjamin Franklin

EDUCATE

Software meets Infrastructure

• Common Software Engineering Techniques in Infrastructure– Domain Modeling– Continuous Integration– Design Patterns

• Service Capabilities– Provisioning– Monitoring <= A must for dynamic environments

Conway’s Law: “Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.”

Consistent, repeatable builds encourage fast deployment and consistently identical systems without the burden of managing images

Utilizing automated and scripted installs, we can:

– Create new systems quickly with little manual interaction.

– Have a setup for repeating the build process in an efficient and stable manner.

And these builds can:– Use the most current version of

software– Configure the desired server

functionality (i.e. Web, SMTP, DHCP server)

– Execute all relevant tests successfully.

A repeatable build improves product quality, reduces the mundane tasks and minimizes dependencies on key people.

build cycle

Softwarepackages

Using Continuous Integration

Using Domain Modeling


role

resource

service

site

node

hostzone has a

is a

has a

composed of

Using Design Patterns


Event-driven ConsumerControl Bus

Message Broker

Patterns from EAI Patterns (http://www.eaipatterns.com


Demo

Provisioning Workflow


Infrastructure as a Service (IaaS)

NetworkNetwork Storage

Compute

Resource PoolsResource Pools

Memory

Data ConsistencyData Consistency

Distributed Storage

Content Delivery

Service Hosting

Container PartitioningContainer Partitioning

Virtual Servers

Messaging

DurabilityDurability

HA Cluster

Grid Database

Platform as a Service (PaaS) Database

PlatformWeb Site Hosting

Web Application

Hosting

Messaging Hosting

Service Hosting

Web Server Provisioning Service

JMSNotify Web Site Hosting

Service

Process Request Create Disk Partition

Create Web VM

Create Web Server

Register System


Kickstart Snippet

# Puppet client installation with its prerequisites. Puppet is a# configuration management tool that, in conjuction with a puppet master# (puppet-server) can be used to affect the nature of the infrastructure# in real time.yum -y install ruby rubygems ruby-rdoc ruby-irb puppet factercat >> /etc/puppet/puppet.conf <<EOF server = sentry.incommon.usEOFchkconfig --add puppetchkconfig puppet on

# Puppet client installation with its prerequisites. Puppet is a# configuration management tool that, in conjuction with a puppet master# (puppet-server) can be used to affect the nature of the infrastructure# in real time.yum -y install ruby rubygems ruby-rdoc ruby-irb puppet factercat >> /etc/puppet/puppet.conf <<EOF server = sentry.incommon.usEOFchkconfig --add puppetchkconfig puppet on


Hyperic Puppet Module

class hyperic { class agent { package { "hyperic-hq-agent": ensure => installed }

file { "agent.properties": source => "puppet:///files/agent.properties", mode => 644, owner => "hyperic", group => "hyperic", require => Package[hyperic-hq-agent], ensure => present, path => "/opt/hyperic/hyperic-hq-agent/conf/agent.properties";

}}

class hyperic { class agent { package { "hyperic-hq-agent": ensure => installed }

file { "agent.properties": source => "puppet:///files/agent.properties", mode => 644, owner => "hyperic", group => "hyperic", require => Package[hyperic-hq-agent], ensure => present, path => "/opt/hyperic/hyperic-hq-agent/conf/agent.properties";

}}


ACCELERATE

“Eighty-five percent of companies are already using open-source software, with most of the remaining 15 percent expecting to do so within the next year” – Gartner, User Survey Analysis: Open-Source Software, Worldwide, 2008

“While some departments already use open source technologies, overall it has been estimated that the global loss due to proprietary software is 'in excess of $1 trillion a year.'” - Michael Tiemann

Case Study: Commercial Web Hosting Provider

Initial Condition 1100+ Individual Servers Rip-and-replace hardware upgrades Static and dedicated server resources Difficult to upgrade software versions

Desired Condition Reduced server footprint Improved ability to seamlessly integrate new

hardware and remove unused system components Ability to expand and contract based upon the system

workload

SolutionAn environment that delivers dynamic provisioning of server, storage and network resources. (IaaS - Infrastructure as a Service)

Deployment of new Platforms to provision Web, Mail, FTP, PostGres and MySQL resources. (PaaS - Platform as a Service)

Dramatic increases in licensing fees from a proprietary software solution. Non-standard configuration of servers. No means to monitor system health.

Key Open Source TechnologiesPlatform• CentOS Enterprise Linux 5• Xen (Virtualization)

Middleware• Jboss Application Server• High Availability Messaging

(JMS, SOAP)• Drools Business Rules

Monitoring & Inventory Management

• Hyperic HQ Open Source

Installation & Configuration Management

• Cobbler• Puppet

Core Software• Apache• Zimbra• Postgres, MySQL• ProFTP• Directory Server

Case Study: Department of Defense Client

Initial Condition New research project with constantly evolving

infrastructure requirements. This made it difficult to provision the data center resources.

Desired Condition Operational, multi-tenant environment in 30 days. Small footprint (1 rack) Shared storage solution Rapid provision of VMs based on pre-defined profiles Windows, RHEL 4/5 and Oracle guest machines

SolutionAn environment that delivers dynamic provisioning of server, storage and network resources. (IAAS - Infrastructure as a Service)

Deployment of new Platforms to provision Oracle database and VM resources. (PAAS - Platform as a Service)

Needed to support multiple PoC efforts. No easy way to truly evaluate software and overall solution capabilities. Needed capability to provision data and compute resources as well as reclaim resources from completed development efforts.

Key Open Source TechnologiesPlatform• Red Hat Enterprise Linux 5• Xen (Virtualization)• GFS (Distributed File System)• Oracle 11G


• Hyperic HQ Open Source

Installation & Configuration Management

• Cobbler• Puppet• Satellite Server

Core Software• Apache• Zimbra• Postgres, MySQL• ProFTP• Directory Server

Case Study: Federal Government Client

Initial Condition Content spread across 3 labs and 2 buildings. 55+ under-utilized servers Lack of available power and cooling resources. Difficult to know current state of system.

Desired Condition Consolidation into single lab Significant reduction in power utilization Ability to expand and contract based upon the system

workload

SolutionUsing a combination of grid, virtualization and service architectures, two separate grids were constructed – one for hosting services and one for hosting multi-terabyte databases.

New monitoring system provided ability to monitor infrastructure and report on current workloads and system configurations.

Needed a system to capture, manage, store, preserve, and deliver content and documents related to organizational processes. This system had to scale to handle exponentially increasing data volumes and rapidly growing user community.

Key Open Source TechnologiesPlatform• Red Hat Enterprise Linux 5• Xen (Virtualization)

Middleware• High Availability Messaging

(JMS, SOAP)• Open Source Enterprise Service

Bus


• Hyperic-based Open Source monitoring solution

Core Software• Apache• Tomcat-based application

servers


Finding Balance

Coase’s Law: “There must be a balance between the costs of the transactions that a company must pay and the opportunity to make everything in house.”

The Internet Inverse: “The size of a company will decrease until the cost of doing something inside the company will be lower than doing it outside“ (Wikinomics)

Build, Buy, Lease : The classic build, buy, lease decision points will determine the viability of utility computing:

Money: What are the costs associated with storing, moving and retrieving data?

Time: Is the data available on-demand? Is the latency associated with remote storage of data acceptable for the business process?

Resources: Does the organization have an IT Staff? Are the skill sets of the staff in line with the objectives of the organization?

The constant need for IT: Whether the data center is in-house or outsourced, IT expertise will still be in demand.

Corporate Computing is not going away: Information Technology and data both represent assets and competitive advantage to corporations


Q&A

Advocate – Educate – Accelerate


As community engagement practitioners, we seek to build ongoing, permanent relationships for the purpose of applying a collective vision for the benefit of the community.

inCommon, Inc. was started to advocate the use of Open Source and Open Architecture solutions in the U.S. government market.

As part of this mission, we seek to educate individuals on the value of Virtual, Grid and SOA architectures based on Open Source and Open Architecture constructs.

We work with clients to efficiently and effectively accelerate systems delivery while maximizing existing resources whenever possible.

Monitoring IAAS & PAAS Solutions

Technology

Transcript of Monitoring IAAS & PAAS Solutions