A Configuration Crawler forCloud Appliances

Michael Menzel, Markus Klems, Hoang Anh-Le, Stefan Tai

eOrganization Research Group Karlsruhe Institute of Technology (KIT)

March 27, 2013, International Conference on Cloud Engineering (IC2E)

Agenda

1. Foundations, Motivation & Existing Work

2. Method: A Configuration Crawler

3. Validation: Implementation for AWS EC2

4. Conclusion & Outlook

# 2

MOTIVATION & FOUNDATIONSCloud Appliances and Configuration Meta-Data

# 3

Cloud Appliances in Compute IaaS*

• Differently configured Virtual Machine Images

# 4

VM Image

Operating System

VM Image

Operating System

Libraries

Software Platforms

Executables & Data

Operating System only Full/Partial Software Stack

* Infrastructure as a Service (IaaS)

Appliances in Today‘s Public Clouds

# 5

• Not all Providers offer Appliances

• Engaged Users create many Appliances

Centralized Packaging Decentralized Packaging

Simple VM Images

CloudAppliances

Both • AWS EC2

• Rackspace

• GoGrid

Top 3 public AMI owners in US-East-1, April 13 2012

Meta-Data on Cloud Appliances

# 6

• There is Meta-Data, but not on Configuration

• Crawling needed to gain more information

Applications

• Interoperability: Convert Appliances toConfiguration Management Manifests

• Decision Support: Consider ConfigurationData in Virtual Machine Selection

• Statistics: Aggregate Configuration Data

# 7

Existing Work

• Meta-Data bundled with VM Image Files [1]

• Configuration Mgmt. to upgrade Appliances [2]

• Chef Ohai and Puppet Facter to collect installedlibraries in Systems

– For most Operating Systems

– For most Package Managers

# 8

[1] D. Lutterkort and M. McLoughlin, “Manageable virtual appliances,” Linux Symposium, 2007. [2] R. Filepp, L. Shwartz, C. Ward, R. Kearney, K. Cheng, C. Young, and Y. Ghosheh, “Image selection as a service for cloud computing environments,” in Service-Oriented Computing and Applications(SOCA), 2010 IEEE International Conference on, dec. 2010, pp. 1 –8.

A METHOD FOR CRAWLINGVIRTUAL APPLIANCE CONFIGURATIONS

# 9

Method for Configuration Crawling

• Procedure Model forCrawling Virtual Appliance Configurations

# 10

Parameter Input

Operation

Data Artifact

Discovering

• Retrieve Meta-Data via Compute Cloud API

• Filter out ineligible Virtual Appliances

# 11

Crawling Configuration Data

• Split Function allows parallel processing

• Instantiate & Crawl multiple Virtual Appliances in parallel

• Leverage configuration mgmt. Agents* to detect configuration

• Collect configuration meta-datafrom started Appliance Instance

# 12

Data Persistence

• Centralized storing of crawled configuration meta-data

• Persistent, centralized data store enables to reusedata in several applications

# 13

Data Model

• Centralized storing of configuration meta-dataneeds common scheme

# 14

Application: Decision Support

• Employ Config. Meta-Data in RequirementDefinitions for Appliance Selections

# 15

Application: Interoperability

• Generate Manifests from Config. Meta-Data

# 16

VALIDATION WITHPROOF-OF-CONCEPT

# 17

Implementation for AWS EC2 [3]

• Ruby Discoverer with filter & blacklist

• Ruby Crawler EC2 Instances injecting Chef Ohai [4] toinstantiated Appliances– Ohai requires Ruby– Intermediate Result Collection to AWS S3

• Crawling Appliance 21 min. avg., costs 1 EC2-h

• MongoDB to store JSON Data, and copy on Google AppEngine for WebApp

# 18

[3] Available at http://github.com/myownthemepark/ami-crawler[4] http://wiki.opscode.com/display/chef/Ohai

Find it online!

You can find the Crawler Database as a Web App on

... enhancing it permanently.

# 19

myownthemepark.com

CONCLUSION & OUTLOOK

# 20

Conclusion

• Crawling Configuration Data of CloudAppliances is feasible– Proposed a procedure and data model

– Validated the approach with a Proof-of-Concept

• Several Applications for collectedConfiguration Meta-Data of Appliances– Configuration Manifests for Interoperability

– Statistics and Decision Support

# 21

Outlook

• Extend implementation with support for moreCloud compute services

• Use Crawler Data in Decision Support Frameworks for Web Applications (e.g., CloudGenius [5])

# 22

[5] M. Menzel and R. Ranjan, “CloudGenius: Decision Support for Web Server Cloud Migration,” in Proceedings of the 21st International Conference on World Wide Web. New York, NY, USA: ACM, 2012.

THANK YOU!TIME FOR QUESTIONS AND COMMENTS

Discussion on the findings

# 23

Contact Me

For Questions, Discussions,or Initiating Research Exchange:

Michael Menzel

Karlsruhe Institute of Technology (KIT)

Englerstr. 11

76131 Karlsruhe

Email: menzel@kit.edu

BACKUPMore slides

# 25

Related Work

• Security Analysis:– T. Garfinkel and M. Rosenblum, “A virtual machine introspection based architecture for

intrusion detection,” in NDSS, 2003.

• Configuration Management:– R. Filepp, L. Shwartz, C. Ward, R. Kearney, K. Cheng, C. Young, and Y. Ghosheh, “Image

selection as a service for cloud computing environments,” in Service-Oriented Computing and Applications (SOCA), 2010 IEEE International Conference on, dec. 2010, pp. 1 –8.

– K. Magoutis, M. Devarakonda, N. Joukov, and N. G. Vogl, “Galapagos: Model-driven discovery of end-to-end application-storage relationships in distributed systems,” IBM Journal of Research and Development, vol. 52, no. 4.5, pp. 367 –377, july 2008.

– IBM, “Tivoli application dependency discovery manager,” http://www-01.ibm.com/software/tivoli/products/taddm/, accessed 25th April 2012.

– A. V. Dastjerdi, S. G. H. Tabatabaei, and R. Buyya, “An Effective Architecture for Automated Appliance Management System Applying Ontology-Based Cloud Discovery,” in Proceedings of the 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing, IEEE Computer Society. Ieee, 2010, pp. 104–112.

• Meta-Data in VM Image Files– D. Lutterkort and M. McLoughlin, “Manageable virtual appliances,” Linux Symposium, 2007.

# 26

Appliances in Today‘s Public Clouds

# 27

CloudAppliances

Simple VM Images

• Centralized Packaging

• Decentralized Packaging

• Centralized Packaging

Appliances in AWS‘ Public Cloud

• Amazon accounts for >50.000 AMIs, growingdaily

• AMIs differ in multiple attributes, including itssoftware configuration

# 28

AWS AMIs in Regions

# 29

AWS Decentralized AMI Creation

# 30

Full Procedure Model

# 31