Christoph Langguth - Towards QoS for Scientific Workflows - SWF 2009 - July 10 2009, Los Angelesunibasel
Christoph Langguth, Paola Ranaldi, Heiko Schuldt
University of Basel, Database and Information Systems Group Bernoullistr 16, CH-4056, Basel, Switzerland
{firstname.lastname}@unibas.ch
This work has been partly supported by the Hasler Foundation within the project COSA (Compiling Optimized Service Architectures)
Towards Quality of ServiceFor Scientific Workflows by usingAdvance Resource Reservations
Christoph Langguth - Towards QoS for Scientific Workflows - SWF 2009 - July 10 2009, Los Angeles 2unibasel
A motivating example: weather forecast
• Long-running and data-intensive• And time-critical: must be finished by the time of evening news
ADAS-ARPSData Analysis
Satellite DataRemapper
L3 RadarRemapper
L2 RadarRemapper
TerrainPreprocessor
Gridded DataInterpolator
SurfacePreprocessor
ARPS to WRFConverter
WRFModel
WRF to ARPSConverter
ARPSPlotting
Preprocessing
Misc.Transformation
Analysis
Legend
(*) Simplified version of workflow from Droegemeier, Gannon, Reed, et al: Service-Oriented Environments for Dynamically Interacting with Mesoscale Weather
Christoph Langguth - Towards QoS for Scientific Workflows - SWF 2009 - July 10 2009, Los Angeles 3unibasel
QoS, and how to provide it – in a nutshell
• Individual services give guarantees about their execution time– Which are combined to QoS guarantees for the entire WF
– users could specify: this WF must be executed as [fast, cheap, energy-efficient,...] as possible
• Contracts negotiated using WS-Agreement
• Needs predictable resource utilization on provider's side– Resources needed must be announced, and reserved for execution
– Advance Reservations for every service call
• A bunch of metadata is required for planning the execution and setting up the reservations
Christoph Langguth - Towards QoS for Scientific Workflows - SWF 2009 - July 10 2009, Los Angeles 4unibasel
WF execution in DWARFS: the big picture
A
B
D E
F G H
CEng1Eng3
Eng2
A
B
D E
F G H
CX:50GB
T:40GB
V:35GB
W:23KBZ:10KB
Y:15KB
small amount of data
Data dependencies
large amount of data
DWARFS: Distributed Workflow execution engine with Advance Reservation Functionality Support
• Partition the process for distributed execution:
– Keeping large data transfers “local“, as far as possible
– Inter-partition data transfers are handled by special DWARFS storage subsystem
Start: 11:30End: 13:07CPU: 80%Storage: 10%
Start: 17:04End: 17:53CPU: 25%
• Scheduling, Partitioning, Reservations are all interrelated
• And depend on the metadata that service providers make available
Christoph Langguth - Towards QoS for Scientific Workflows - SWF 2009 - July 10 2009, Los Angeles 5unibasel
Required Metadata for Reservations...
CPU:
storage:
HW:
Provider A Provider B
I'll need to call Operation A w/INPUT: size=50GB, Class X
OUTPUT: 25GB, Class Y
timing:
50 % max.duration inv. prop.
30 % min.
10 % exactly
2:30 h
100 % max.duration inv. prop.
5 %min.
100 % exactly
3:50 h
Christoph Langguth - Towards QoS for Scientific Workflows - SWF 2009 - July 10 2009, Los Angeles 6unibasel
...Required Metadata for Reservations
• Call of operation X:– Input characterization (size, class)– Resource requirements– Timing prediction, and resource ↔ timing relationship– Output characterization
• Also need providers' current resource allocation schedules, and cost functions
• Unified notion of resource „share of capacity“• Data characteristics metadata and duration prediction
highly domain-specific– Only requirement for us: determine size of data
Christoph Langguth - Towards QoS for Scientific Workflows - SWF 2009 - July 10 2009, Los Angeles 7unibasel
Past, present and future
• Prototype of CPU share enforcement/timing prediction• Formal model of DWARFS Workflow
• WS-Agreement with renegotiation support• Planner implementation
• Storage Subsystem• Bringing it all together :-)
Christoph Langguth - Towards QoS for Scientific Workflows - SWF 2009 - July 10 2009, Los Angeles 8unibasel
Thank you for your attention!
Christoph Langguth - Towards QoS for Scientific Workflows - SWF 2009 - July 10 2009, Los Angeles 9unibasel
...CPU enforcement in action
Christoph Langguth - Towards QoS for Scientific Workflows - SWF 2009 - July 10 2009, Los Angeles 10unibasel
Required Metadata for reservations• Call of service operation X
•with input characterized so and so,
– How long will this take?
– Which resources are needed?t, and how does modifyingthe share influence the timing?
– Which output will this produce?
• Providers need to also make available theircurrent resource allocation schedules, and therespective cost functions
• Any kind of resource can be represented using the unified notion of „share of local capacity“
• Data characteristics metadata will be domain-specific.
– Only requirement for DWARFS: be able to determine the size of it
– (prediction of duration is also highly domain- and implementation-specific)
Top Related