*An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E....
-
date post
20-Dec-2015 -
Category
Documents
-
view
214 -
download
1
Transcript of *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E....
![Page 1: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/1.jpg)
*An Integrated Self-Testing Framework for Autonomic Computing Systems
Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke
School of Computing and Information Sciences
Florida International University
07/14/2007
FIU-SCIS Departmental Colloquium
* To appear in Issue 8 2007 of the Journal of Computers * Supported in part by the National Science Foundation under grants IIS-0552555 and HRD-0317692
![Page 2: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/2.jpg)
2
Introduction o Continual growth in size and complexity of
computing systems has led to a need for support tasks to be shifted from people to technology
o Movement towards self-managing systems that can dynamically configure, heal, protect, and optimize themselves – Autonomic Computing.
o There is a lack of techniques to dynamically validate such systems.
o How can we be sure that AC systems behave correctly after a change at runtime?
![Page 3: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/3.jpg)
3
Introduction (cont’d)o This work presents a methodology which
dynamically validates changes resulting from self-management in autonomic systems.
o Applies the concepts of autonomic managers, knowledge sources and manual management facilities to testing activities.
o Provides two general strategies based on system constraints and feasibility of use.
o Elaborates on a prototype that uses a generic design for autonomic systems that supports runtime testing.
![Page 4: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/4.jpg)
4
Outlineo Background
o Overview of Testing Approach
o Test Managers
o Test Support Components
o Challenges
o Prototype
o Related Work
o Conclusion & Future Work
![Page 5: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/5.jpg)
5
Autonomic Computing (1)o IBM’s solution to the problems of integrating and
managing highly complex computing systems
oHuman Body – ANS regulates vital bodily functions without conscious human involvement (homeostasis)
o Applied to Computing Systems – automation of low-level tasks, and high-level goal specification.
oMain CharacteristicsSelf-Configuration: configuring existing / new components
Self-Optimization: balancing workloads, tuning efficiency
Self-Protection: safeguarding from attacks or failures
Self-Healing: diagnosing and repairing problems
![Page 6: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/6.jpg)
6
Autonomic Computing (2)
Layered Architecture of Autonomic ComputingSource: An Architectural Blueprint for Autonomic Computing, Fourth Ed., IBM Corporation, June 2006.
o Management console to facilitate human activity
o Coordinates Touchpoint AMs (within or across)
o Manages resources directly through touchpoint layer
o Implements sensor and effector interface for MR
o H/W or S/W entities being managed
![Page 7: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/7.jpg)
7
Autonomic Computing (3)o Monitor: collects state
information from MR
o Analyze: determines if and when self-* is needed
o Plan: formulates a plan to address change request
o Execute: executes the change plan on the MR
o Knowledge: stores data shared by the MAPE functions
MAPE Structure of Autonomic ManagersSource: An Architectural Blueprint for Autonomic Computing, Fourth Ed., IBM Corporation, June 2006.
![Page 8: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/8.jpg)
8
Software Testing - Categorieso Two broad types of testing:
o Blackbox – specification-based, focuses on functionality, i.e., inputs → expected outputs.
o Whitebox – implementation-based, focuses on whether or not the program has been thoroughly (adequately) exercised.
o Regression Testing – determines whether or not modifications to software have introduced new errors into previously tested code.o Retest-all – retest the entire test suite
o Selective – only retest a strict subset
![Page 9: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/9.jpg)
9
Software Testing - Automationo Automating the testing process involves:
1) Designing test cases
2) Developing test scripts
3) Setting up a test harness for automatically: Setting up the test environment
Executing test cases
Logging the test results
Evaluating the test log
o What happens when the post-test evaluation passes or fails?
o Whitebox – implementation-based, focuses on whether or not the program has been thoroughly (adequately) exercised.
o Regression Testing – determines whether or not modifications to software have introduced new errors into previously tested code.o Retest-all – retest the entire test suite
o Selective – only retest a strict subset
![Page 10: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/10.jpg)
10
Safe AdaptationZhang et al. (WADS 2004)
Source: J. Zhang, B. H. C. Cheng, Z. Yang, and P. K. McKinley. Enabling safe dynamic component-based software adaptation. In WADS, pages 194–211, 2004.
![Page 11: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/11.jpg)
11
Testing Approach – Overviewo Idea: Develop an implicit self-test characteristic
for autonomic computing systems
o Integrate activities of a self-testing framework into the workflow autonomic managers (AMs)
o Test Managers interface with the autonomic system and coordinate the testing activity
o Framework consistent with grand vision of AC
o Based on two strategies: Safe Adaptation with Validation
Replication with Validation
![Page 12: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/12.jpg)
12
Testing Approach - Architecture
Integrated Self-Testing Framework for AC Systems Extends the dynamic test model by King et. al, “Towards Self-Testing in Autonomic Computing Systems”Based on two general approaches: Safe Adaptation with Validation, and Replication with Validation
![Page 13: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/13.jpg)
13
Test Managers (TMs)o Extend the concept of autonomic managers to
testing activities, i.e., Self-Testing via MAPE
o Responsible for:Test Coordination
Test Planning and Execution
Test Suite Management
Pre- and Post-test Setup
Post-test Evaluation
Storage of Test Artifacts
![Page 14: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/14.jpg)
14
High-Level Test Coordination1a,1b) Monitors OAM & TKS for state changes
2) Uploads new validation policy to Touchpoint TM
3) Touchpoint TM needs support tools to be setup
4) Invokes effector of ATS to configure support tool
5) Support tool configured successfully for TTM
6) TTM notified that it can begin test execution
Orchestrating TM Interactions Manages multiple components in the self-testing framework for high-level test coordination
![Page 15: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/15.jpg)
15
Low-Level Testing Tasks
Touchpoint TM Interactions Performs testing on managed resources of the autonomic system via two control loops (a) and (b)
![Page 16: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/16.jpg)
16
Test Support Componentso Auxiliary Test Services (ATS) allows TMs to
configure external or third-party testing tools, code profilers, performance analyzers, etc.
o Provides mechanisms for accessing information in the test knowledge sources. updating validation policies, test suites, test logs/histories
o Implements facilities for administrative functions related to manual test management, interactive test management, defect tracking, scenarios
o Administrative functions will be integrated with the console of the autonomic system
![Page 17: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/17.jpg)
17
Challengeso V&V techniques for offline systems not scalable;
and still heavily dependent on human tester.
o Drawbacks magnified for adaptive systems.
o Testing adaptive nature requires dynamic:Regression test case selection
Test suite analysis to:– generate new test cases when necessary.– identify test cases that are no longer applicable.
o Testing autonomic systems in the presence of unforeseen conditions
![Page 18: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/18.jpg)
18
Overcoming Challengeso Using proposed framework as the focal point for
research directions to support QA in AC systems
o Some Directions:Use of formal specifications to generate test
sequences, test oracles, and test data
Executable formal specifications also support visualization systems (grand challenge of AC)
Mechanisms for policy-based risk analysis and trust can support testing in unforeseen circumstances
Risk-based strategies for regression test case selection
![Page 19: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/19.jpg)
19
Prototype - Featureso Autonomic Container – Data structure with
autonomic capabilities and implicit self-testo Version 1 – Stack with self-configuration and self-
test. 80% full, reconfigure by increasing capacity
o Version 2 – Implemented as a remote stack and added self-protection. Users stack exceptions > 3, protect by disabling user
o Implemented Replication with Validation Validation required 100% pass rate for test cases,
75% for both branch and statement coverage Test suite: 24 test cases, using boundary,
random, and equivalence partitioning.
![Page 20: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/20.jpg)
20
Prototype – Setup Environmento Developed in Java 5.0 using Eclipse 3.3 SDK
with required test support plugins and libraries.o Design Tools:
StarUML – class, activity, package diagrams
OribeXML – creating and modifying XML policies
o Test Support Tools:JUnit – a Java unit testing tool from the xUnit family
of testing frameworks
Cobertura – a Java code coverage analysis tool that calculates the percentage of source code exercised by unit tests.
![Page 21: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/21.jpg)
21
Prototype – Top-Level Design
Top-Level Design of Autonomic Container
![Page 22: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/22.jpg)
22
Prototype – Manager Design
Generic Design of Autonomic Managers
![Page 23: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/23.jpg)
23
Prototype – Policy Design
![Page 24: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/24.jpg)
24
Prototype – Self-Test Classes
Minimal Class Diagram of Self-Test Subsystem
![Page 25: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/25.jpg)
25
Prototype – Evaluationo Used a mutation testing technique to evaluate
the fault detecting ability of the prototypeo Strategy: simulate faulty change requests to the
managed resource (stack) under SC and SP:SC – created mutant stack by modifying the
resize method to cause decreased capacitySP – created mutant stack by altering disable
account method to enable the usero Analyzed the results of executing the self-testing
framework on original stack and the mutants
![Page 26: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/26.jpg)
26
Prototype – Resultso Correct change scenarios:
o Incorrect change scenarios:Two TC failures each, coverage omitted
o Mutation analysis produced favorable results as testing would have prevented potentially harmful changes to the managed resource
Feature TC Pass Rate Branch Statement
Self-Config 100% 85% 100%
Self-Protect 100% 88% 100%
![Page 27: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/27.jpg)
27
Prototype – Lessons Learnedo Provided us with insight on the scope of the self-
testing subsystem w.r.t responsibilities.o Synchronization of AMs and intelligent control
loops are a major challenge.o Limitations:
Dynamic test planning (current: static lookup)
Need to implement Safe Adaptation with Validation Strategy
Code-based changes
![Page 28: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/28.jpg)
28
Related Worko Towards Self-Testing in Autonomic
Computing Systems, King et al., ISADS ‘07.
o A Self-Testing Autonomic Container, Stevens et al., ACMSE ‘07.
o Synthesizing assertions from observed behavior, Denaro et al., ACC ’05. Embeds assertions into the communication
infrastructure.Assertions are checked at runtime.
o Making components self-testable, Le Troan et al., TOOLS ’99.
![Page 29: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/29.jpg)
29
Conclusion and Future Worko Proposed a framework that dynamically
validate change requests in AC systems.
o Approach extends the current structure of AC systems to include self-testing.
o Supports two validation strategies.
o Developed a prototype to show their feasibility.
o Future work calls for: Extending the capabilities of the prototype Implementing safe adaptation with validation Evaluating efficiency of the two approaches
![Page 30: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/30.jpg)
30
Summer ’07 and Ongoing Worko REU Program 2007 – Alain E. Ramirez and
Barbara Morales; Autonomic Job Schedulero A Generic O-O Design to Support Self-Testable
Autonomic Systems (SAC ‘07)
o Student Paper to ACMSE 2008
o Fall 2007 – Gonzalo Argote-Garciao Embedding Formal Specifications into Autonomic
Components
o ? – Alain E. Ramirezo Applying Design Patterns to AC systems
![Page 31: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/31.jpg)
31
Acknowledgements
Djuradj Babich
Jonatan Alava
Ronald Stevens
Brittany Parsons
Mitul Patel
Dr. S.M. Sadjadi
![Page 32: *An Integrated Self-Testing Framework for Autonomic Computing Systems Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. Clarke School of Computing.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d455503460f94a224a8/html5/thumbnails/32.jpg)
32
Thank You
Questions?
This work has been supported in part by the National Science Foundation under grant IIS-0552555