Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven...
-
Upload
antony-farmer -
Category
Documents
-
view
215 -
download
2
Transcript of Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven...
Performance Architecture within ICENI
Dr Andrew Stephen McGoughLaurie Young, Ali Afzal, Steven Newhouse and John Darlington
London e-Science Centre
Department of Computing, Imperial College London
2
Outline
• Overview of ICENI
• Performance Framework
• Example
• Conclusion
3
ICENI: Imperial College e-Science Network Infrastructure
• Collect and provide relevant Grid Meta-Data
• Pluggable architecture• Test Architecture for Grid Research• Foundation for higher-level Services
and Autonomous Composition• Integrated Grid Middleware Solution• Interoperability between
architectures, APIs • Added value layer to other
middleware• Usability: Interactive Grid Workflows• Role and policy driven security• ICENI Open Source licence (extended
SISSL)
The Iceni, under Queen Boudicca, united the tribes of South-East England in a revolt against the occupying Roman forces in AD60.
4
Scheduling Workflows in ICENI
• Applications consist of a number of components linked together in a dataflow manner
• The abstract workflow needs to be mapped down to a set of component implementations which will run on resources
Linear EquationSource
Linear EquationSolver
Display VectorResults
General Equation generato
r
LU Factorisatio
n
Simple Vector Display
5
The Problem
• We have:– Multiple resources where components can run– Multiple implementations of components– The choice of one resource component
mapping can affect the others
• User wants predictable performance
• How to choose the “best” mapping of workflow over resources to give user predictability?
6
The Solution
• We need to take into account:– Execution Times of components on Resources
• Performance Data
– Inter-component effects of workflows• Workflow aware Schedulers
– Workload on resources, making sure they are free when we need them
• Reservation systems
7
The Architecture
Scheduler
ReservationService
PerformanceStore
Launcher
ApplicationService
ReservationEngine
Workflow
8
Performance Store- Persistent Performance storage
Performance Store- Persistent Performance storage
The Performance Repository Framework:
Performance Framework
Performance Processing- Conversion of raw event times into performance data
Data Collector-Collecting data on currently running applications (event times)
Performance Store- Persistent Performance storage
9
Collection of Performance Results
Data CollectorLinear EquationSource
Linear EquationSolver
Display VectorResults
Time Event 12:00 Linear Equation Source Start 12:04 Send out Equations 12:03 Linear Equation Solver Start 12:05 Receive Equations 12:12 ………..
Event:Start LinearEquation Source
Performance Processing
Workflow
Performance Store
10
Storing Performance Data
• Multiple stores can be used in one framework
• The stores may be data stores or analytical models
• All assumed to be persistent• Allows requests for predictions to be made• New Data can be added to the stores• Store data is aggregated together
– based upon reliability of store data• Provided by the store
Performance Store
11
Using Performance Data
• Scheduler Builds up workflow graph with timings requested from the Performance Repository
• Timings are based on component implementation, resource, co-allocation count and other properties defined by the component implementer
• As the store will not contain all possible combinations of these properties regression is used to provide estimates for the missing values– This is an area of ongoing research
12
Reservation Engine
ReservationEngine Framework
Reservations Database- Data about upcoming reservations,usedto test if a new reservation can be made
DRM Translation level-Communication with the underlying DRM system for supporting Reservations
Listen out for requests- Launcher services wishing to makereservations
13
Reservation Service
ReservationService Framework
Reservations Database- Data about upcoming reservations,usedto test if a new reservation can be made
Reservation orchestration-Attempts to reserve all resources for a given workflow. May shift when it runs
Listen out for Services- Launcher with reservation- Scheduling Services
14
Reservations
time →
time →
ReservationService
Scheduler
Workflow Reserve (workflow)
HOLD
Conflict
HOLD
HOLD
Reserve
Reserve
WS-AgreementRequest interval
Linear EquationSource
Linear EquationSolver
Display VectorResults
Reservations not possible on Users Desktop
15
Example: Linear Equation Solver
service list composition pane parameters
16
Inferring Workflow from Dataflow
17
Conclusion
• Better usage of resources.– Reservations of resources in the future– Determining if co-allocation of components will affect
performance– Late Enactment of components
• Critical Path analysis – can schedule this appropriately
• Provides a framework for experimentation with– Different scheduling algorithms– Different performance models– Different reservation policies
18
Performance Trinity
Performance Models
Grid Scheduling Reservation Fabric
19
The Architecture: Showing the Trinity
SchedulerReservation
Service
PerformanceStore
Launcher
ApplicationService
ReservationEngine
20
Acknowledgements
• Director: Professor John Darlington• Research Staff:
– Nathalie Furmento, Stephen McGough, William Lee– Jeremy Cohen, Marko Krznaric, Murtaza Gulamali– Asif Saleem, Laurie Young, Jeffrey Hau– David McBride, Ali Afzal
• Support Staff:– Oliver Jevons, Sue Brookes, Glynn Cunin, Keith Sephton
• Alumni:– Steven Newhouse, Yong Xie, Gary Kong– James Stanton, Anthony Mayer, Angela O’Brien
• Contact:– http://www.lesc.ic.ac.uk/ e-mail: [email protected]