The autonomic nervous system The autonomic nervous system Xiaoming Zhang.
Workflow Adaptation as an Autonomic Computing Problem
-
Upload
heather-cox -
Category
Documents
-
view
40 -
download
1
description
Transcript of Workflow Adaptation as an Autonomic Computing Problem
Combining the strengths of UMIST andThe Victoria University of Manchester
Workflow Adaptation as an Autonomic Computing Problem
Kevin Lee, Rizos Sakellariou, Norman W. Paton and Alvaro A. A. Fernandes
School of Computer Science, University of Manchester
{klee, rizos, norm, alvaro}@cs.man.ac.uk
Kevin Lee 25th June 2007
Combining the strengths of UMIST andThe Victoria University of Manchester
Talk Overview
1)Reasons for Adaptation in Workflows
2)Autonomic Computing
3)Our View of Workflows
4)Potential Workflow Adaptations
5)Categorising Workflow Adaptations
6)Future work
Combining the strengths of UMIST andThe Victoria University of Manchester
1. Reasons for Adaptation in Workflows
•Very long running•Small delays can have large effects due to dependencies•often involve highly distributed resources•Limited control over resources•Uncertain execution times•Uncertain queue waiting times
Execution Characteristics of Scientific Workflows
Combining the strengths of UMIST andThe Victoria University of Manchester
1. Reasons for Adaptation in Workflows•scheduling of a workflow is decided before it starts executing
•Using current information about the execution environment
•What happens if the environment changes?•Resources disappear•Loads change•New resources appear
•What if a execution resource becomes unavailable•Only real option is to Re-Submit to different resources•Reduced performance
•New resources are not taken advantage of
•The Execution characteristics of workflows combined with these lead to lower than potential execution performance
Adaptation would be desirable.But what would workflow adaptation look like?
Combining the strengths of UMIST andThe Victoria University of Manchester
Many systems nowadays face these issuesBuilding adaptive systems is hard always done in ad-hoc waysleads to brittle and non-reusable adaptationTo this end adaptive systems are often seen in the Autonomic
Systems Community as functionally decomposable into the components:
2. Autonomic Computing
Monitor: Events from a source:
log filesin-memory processsensors
Analyze: When an event occurs, what to do about it...
Plan: After the event is detected and analysed, the system needs to determine what to do about it.
Execute: Perform the necessary changes
But, how do we think about workflows in these terms?
Combining the strengths of UMIST andThe Victoria University of Manchester
3. Our View of WorkflowsTo look at workflows in a generic way, we’ve adopted a view of the use of workflows as follows:
An abstract workflow: describes the workflow at the level of tasks that perform transformations on data.
A concrete workflow: describes the workflow at the level of actual services•file-based inputs and outputs.
Mapping an abstract workflow•choosing appropriate services for tasks•finding data sources and output files
Scheduling a concrete workflow•assigning each of the services to execution nodes.
We have a more formal notation in the paper
Combining the strengths of UMIST andThe Victoria University of Manchester
4. Potential Workflow Adaptations
In general, adaptations can usefully be thought of as a revision of decisions made previously
Thus, based on the previous slide, workflow adaptations can be classified as either mapping or scheduling adaptations.
Adaptations can be performed for different reasons:•prospective (to improve future performance)•reactive (to react to previous results)•altruistic (to aid other areas of the system).
Adaptations can also affect the workflow at differentlevels of granularity:•single node•some nodes•all of the workflow.
Combining the strengths of UMIST andThe Victoria University of Manchester
4. Potential Workflow Adaptations: Mapping Adaptations
Mapping adaptations are adaptations where the mapping from the abstract workflow to the concrete workflow changes depending on the environment.
Examples:
•Change abstract node to concrete node mapping:•Reduce the number of concrete nodes for an abstract task•Increase the number of concrete nodes for an abstract task (task-splitting).•Remove an abstract node•Change data source/sink for a service
See paper for further detail
Combining the strengths of UMIST andThe Victoria University of Manchester
4. Potential Workflow Adaptations: Scheduling Adaptations
Adaptive scheduling involves the alteration of the scheduling policy in response to changes in the environment.
Examples:
•Increase the level of parallelism of a service. •Decrease the level of parallelism of a service. •Restart service. •Pause service. •Move service between execution nodes.
See paper for further detail
Combining the strengths of UMIST andThe Victoria University of Manchester
5. Categorising Workflow Adaptations
Monitoring Analysis Planning ExecutionProgress of a service
Completion of a service
Data consumption rate of a service
Data production rate of a service
Available execution nodes
Load on an execution node
Load on a network link
Memory usage on an execution node
Available services
Available data resources
Load Imbalance
Bottleneck
Potential Workflow QoS miss
Execution node failure
Free capacity
New service available
New data available
Underutilised execution node
Increase service parallelism
Reschedule a service
Replace a service
Use free execution nodes
Move services
Change data sources
Execute changes
This level of understanding, combined with a adaptivity infrastructure
provides a solid basis for providing adaptivity functionality
We used the MAPE functional decomposition to look at workflow adaptation
The Monitoring, Analysis, Planning and Execution functional phases can be used to investigate the adaptive opportunities
They provide a consistent, abstract viewpoint with which to expressadaptation strategies
The Various options in M,A,P,E can be arranged as follows:
Combining the strengths of UMIST andThe Victoria University of Manchester
6. Current activities and Future work
Creating an infrastructure to support the Systematic Development of Adaptive Systems based on the ideas presented today Ease the development of adaptive systems. Support the development of better adaptive systems Investigate the use of the infrastructure in a number of
different domains Use the infrastructure to improve the general
understanding of adaptive systems Applying the infrastructure to related domains
Simulated DAG Scheduling Workflow processing with the Pegasus team Concurrent business workflows Distributed Query Processing
case studies...
Combining the strengths of UMIST andThe Victoria University of Manchester
Project Organisation
Kevin Lee, Norman W. Paton, Alvaro A. A. Fernandes, Rizos SakellariouSchool of Computer Science, University of Manchester
Oxford Road, Manchester, M13 9PL, U.K.{klee, rizos, norm, alvaro}@cs.man.ac.uk
Jim Smith, Paul WatsonSchool of Computing Science, Newcastle University
Claremont Road, Newcastle upon Tyne, NE1 7RU, U.K.{Jim.Smith, Paul.Watson}@ncl.ac.uk
EPSRC e-Science project entitled:
“An Infrastructure for Adaptive Systems Development”
Combining the strengths of UMIST andThe Victoria University of Manchester
Questions?/Comments?