Marian Bubak Institute of Computer Science and ACC CYFRONET AGH Cracow , Polan d
description
Transcript of Marian Bubak Institute of Computer Science and ACC CYFRONET AGH Cracow , Polan d
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 1
Marian BubakMarian BubakInstitute of Computer Science and ACC Institute of Computer Science and ACC CYFRONET CYFRONET
AGHAGH Cracow Cracow, Polan, [email protected]@agh.edu.pl
andand
CrossGrid, K-WfGrid, and (future) GridSpace CrossGrid, K-WfGrid, and (future) GridSpace CollaborationsCollaborations
From Interactive Applications From Interactive Applications and Knowledge Based Workflowsand Knowledge Based Workflows
to Transparent Semantic Grid to Transparent Semantic Grid (from X# and ~# to # )(from X# and ~# to # )
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 2
OverviewOverview
Trends in applications and computing Trends in applications and computing systemssystems
CrossGrid - Environment for application CrossGrid - Environment for application steeringsteering
Workflow applicationsWorkflow applications
K-WfGrid - knowledge-based environmentK-WfGrid - knowledge-based environment
New proposal – GridSpace: transparent New proposal – GridSpace: transparent semantic grid semantic grid
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 3
Trends in ApplicationsTrends in Applications
Large scale numerical simulationsLarge scale numerical simulations Computationally demanding data analysisComputationally demanding data analysis Distributed computing and storageDistributed computing and storage Remote access to experimental Remote access to experimental
equipmentsequipments A need for integration heterogeneous A need for integration heterogeneous
environments into one applicationenvironments into one application Collaborative problem solvingCollaborative problem solving Virtual organisationsVirtual organisations
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 4
Evolution in Distributed ComputingEvolution in Distributed Computing
Distributed systems operate in heterogenous Distributed systems operate in heterogenous environmentsenvironments
• Large scale resource sharingLarge scale resource sharing• InteroperabilityInteroperability• Communication via protocol stacksCommunication via protocol stacks
Service oriented architecturesService oriented architectures
Open standard integrationOpen standard integration
Virtualisation of resourcesVirtualisation of resources
Complexity of computing systems close to the Complexity of computing systems close to the limits of human capabilitylimits of human capability
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 5
CrossGridCrossGrid
•21 21 partnerspartners•2002-2005, EC IST F22002-2005, EC IST F2•Coordinated by CYFRONETCoordinated by CYFRONET•Research areasResearch areas
• CrossGrid ApplicationsCrossGrid Applications• Grid Tool Environment Grid Tool Environment • New Grid ServicesNew Grid Services• International Testbed International Testbed • ArchitectureArchitecture
www.eu-crossgrid.orgwww.eu-crossgrid.org
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 6
Main Main CrossGrid CrossGrid ObjectivesObjectives
New category of GridNew category of Grid--enabled applicationsenabled applications• computcompute-e- and data and data--intensiveintensive• distributeddistributed• nearnear--realreal--time response (a person in a loop)time response (a person in a loop)• layered layered
New programming toolsNew programming tools
Grid more user friendly, secure and efficientGrid more user friendly, secure and efficient
Interoperability with other GridsInteroperability with other Grids
Implementation of standardsImplementation of standards
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 7
Portal Login and Grid Proxy Creation
Bypass Placementand LB Mesh
Generation
Simulation Job Submission
Job MonitoringVirtual Node Navigation and
Grid Data Transfer
Patient in an MRI Scanner
MR Image Storage MR Image Segmentation
globus-lumc (Leiden)
mn (Virtual Operating Theatre at the UvA)
ce (Linz)ce2 (NIKHEF)
Blood Flow Visualization
Blood Flow Simulation
Blood Flow Rendering in VR
Virtual Medical SupportVirtual Medical Support
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 8
Flood SimulationFlood Simulation
Data sources
Hydrological simulation
Hydraulic simulation
Portal
Meteorological simulation
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 9
HHigh Energy Physics Aigh Energy Physics Applicationspplications
Interactive ANN trainingInteractive ANN training• MPI MPI
Parallel Sleuth algorithmParallel Sleuth algorithm
ATLAS DAQ events remoteATLAS DAQ events remote
processingprocessing• FeasFeasiibilitybility study study of using the of using the GGrid to process difficult events rid to process difficult events
in one in one of of the LHC experiments the LHC experiments ((DAQDAQ))
Distributed Distributed ddata ata aaccess prototypeccess prototype• Distributed filtering of ntuples (data files) distributed Distributed filtering of ntuples (data files) distributed onon
CrossGrid Storage ElementsCrossGrid Storage Elements• The output can be used in the ANN application.The output can be used in the ANN application.
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 10
MeteoMeteo / Pollution / Pollution AApppplicationlication
Weather Weather fforecastorecastinging
Air Air ppollution ollution fforecastorecastinging
Wave Wave modelingmodeling
Data miningData mining
Weather forecastWeather forecastinging is is thethe common link and provides common link and provides input data input data forfor all other all other activitiesactivities
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 11
CrossGrid Tools and ServicesCrossGrid Tools and Services
T e s t b e d T e s t b e d
Applications
Services
Tools
• 17 sites• 9 countries
• over 200 CPUs• 4 TB of storage
RoamingAccess Server
MigratingDesktop
OCM-G
PerformancePrediction
Data Access Globus
Toolkit
MPI Verification
MPI Library
Portal
Post-processing
InfrastructureMonitoring
Plugin
Plugin
Plugin
SOAP
SOAP
SOAP
SOAP
SOAP
Protocol
API
API
API
API
Links
API
API API
SOAP
SOAP
API
API(JMX)
(OMIS)
VisualizationKernel
Links
PerformanceAnalysis
API
ApplicationMonitoring
Benchmarks
NetworkMonitoring
MedicalSupport
ParticlePhysics
Meteo/Pollution
FloodSimulatio
n
DataGrid
Scheduler
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 12
Migrating Desktop ToolsMigrating Desktop Tools
Desktop toolsDesktop tools- Job Wizard- Job Wizard- Job Monitor- Job Monitor- Application Container and Application Plugin- Application Container and Application Plugin- GridFTP Commander- GridFTP Commander- User Profile Manager- User Profile Manager- Private Storage Management- Private Storage Management
- VNC/SSH console- VNC/SSH console
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 13
Migrating Desktop FunctionalityMigrating Desktop Functionality
Main Features:Main Features: Single sign-on / authorisationSingle sign-on / authorisation Platform independentPlatform independent Batch jobsBatch jobs MPI jobsMPI jobs Running interactive applications Running interactive applications
using java plugins or VNCusing java plugins or VNC Monitoring grid applicationsMonitoring grid applications Flexible Application frameworkFlexible Application framework User profile managementUser profile management Easy application add onEasy application add on Local and grid file management Local and grid file management
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 14
Roaming Access ServerRoaming Access Server
Well-defined set of web-services Well-defined set of web-services An interface for accessing HPC An interface for accessing HPC
systems and services (based on systems and services (based on various technologies) in a various technologies) in a common and standardised waycommon and standardised way
Interconnection between variousInterconnection between various grid middleware and applicationsgrid middleware and applications
Additional features:Additional features:• Virtual Directory supportVirtual Directory support• Plug-in for various grid Plug-in for various grid
middlewaremiddleware
JobSubmissionServices
Roaming Access Server
Interactive Session Services
File Management Services
Profile Management Services
Application Management Services
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 15
InteractivityInteractivity
1.1. Interactive workflow - uInteractive workflow - user ser interactivityinteractivity onon the Grid the GridA A user can continuously interact with user can continuously interact with a a Grid client without Grid client without waiting for twaiting for termination ermination of the jobsof the jobs submittedsubmitted
2.2. Online output control - uOnline output control - user ser oone-ne-wway ay iinteractivity nteractivity A A user can see the output of the application running user can see the output of the application running inin the Grid the Grid testbed on testbed on an an MD client synchronously with the applicationMD client synchronously with the application
3.3. Runtime steering - uRuntime steering - user ser ttwo-wo-wway ay iinteractivitynteractivityA useA use can steer the running application, either providing some can steer the running application, either providing some input data online as requested by the application input data online as requested by the application (also (also asynchronouslyasynchronously)), or suspending the process, changing some , or suspending the process, changing some input data and resuming itinput data and resuming it
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 16
Interactive Job Submission Interactive Job Submission The user submits an The user submits an
application job through the application job through the CG portal or Migrating CG portal or Migrating Desktop and the Roaming Desktop and the Roaming Access Server which Access Server which supports individual user supports individual user environments,environments,
The job is handled by the The job is handled by the Scheduler, which selects Scheduler, which selects the appropriate computing the appropriate computing resources,resources,
DataGrid software DataGrid software components are used for components are used for low-level Grid operations low-level Grid operations (submission for (submission for processing and delivery of processing and delivery of results),results),
The system bases on The system bases on Globus Toolkit v2,Globus Toolkit v2,
Other CrossGrid tools and Other CrossGrid tools and services can be used in services can be used in conjunction with running conjunction with running jobs, as requirements jobs, as requirements dictatedictate
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 17
Interactivity Interactivity via via MDMD
Job Submission Services CrossBroker
Logging&
Bookkeeping
Roaming Access Server
Computing Element
LRMS
Gatekeeper
JDLMigrating Desktop
RAS shadow port,
RAS shadow host
Process Launched
Interactive data
Control data
Submission flow
In/Out/Err job data
010011000
WorkerNode
Job Shadow
stdin – stdout -stderr
Console Agent
JobJob
Condor ByPass SystemCondor ByPass System
Job Shadow
Java Visualisation
plug-in
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 18
Scheduler (CrossBroker)Scheduler (CrossBroker)
Automatic job management for parallel Automatic job management for parallel applications:applications:
Search and selection of available resources, job Search and selection of available resources, job conditioning, job launching, job monitoring, job retry conditioning, job launching, job monitoring, job retry (in case of failures) and results retrieval.(in case of failures) and results retrieval.
• MPICH-P4 (intra-cluster)MPICH-P4 (intra-cluster)• MPICH-G2 (inter-cluster)MPICH-G2 (inter-cluster)• Computational WorkflowsComputational Workflows
Best effort approach to deal with failures/problems Best effort approach to deal with failures/problems
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 19
Performance Analysis ToolPerformance Analysis Tool
Monitors
Managers
ServiceManager
High-LevelAnalysisComponentService
OCM-GInterface(OMIS)
Main
MeasurementInterface
G-PMAnalysis ToolPerformance
User Interface& Visualization
P1
Component
Monitoring System
...
OCM-G
P2Processes
P3
...
...Application Pn
ModulesApplication Performance
MeasurementComponent
Local
...
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 20
Definition of MeasurementsDefinition of Measurements
Aggregation in time Aggregation in space
Objects Functions Partner objectsMetrics
Sites
Nodes
Processes
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 21
Definition of VisualizersDefinition of Visualizers
Visualization type:• Bar graph• Curve diagram• Histogram• Pie chart• Matrix diagram
Parameters:• Scales• Update interval
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 22
0102030405060708090
100
2 4 6 8
P1
P2
P3
Application MonitoringApplication Monitoring
RUNNING APPLICATIONRUNNING APPLICATIONRUNNING APPLICATIONRUNNING APPLICATION
OCM-G – MONITORING OCM-G – MONITORING OF APPLICATIONOF APPLICATION
OCM-G – MONITORING OCM-G – MONITORING OF APPLICATIONOF APPLICATION
TOOL - VISUALIZATIONTOOL - VISUALIZATIONTOOL - VISUALIZATIONTOOL - VISUALIZATION
Standard Interface (OMIS)
System-specific interface
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 23
OCM-G Monitoring SystemOCM-G Monitoring System
G-PMG-PM
Local Monitor
Local Monitor
Local Monitor
Local Monitor
ApplicationProcess 1
ApplicationProcess 1
User Interface
Computing Element
Worker Node
Worker Node
Site ASite A
Site CSite C
Service Manager
Service Manager
Local Monitor
Local Monitor
ApplicationProcess 4
ApplicationProcess 4
ComputingElement
Worker Node
Service Manager
Service Manager
Main Service
Manager
Main Service
Manager
Site BSite B
ApplicationProcess 3
ApplicationProcess 3
ApplicationProcess 2
ApplicationProcess 2
G-PMG-PM
Local Monitor
Local Monitor
Local Monitor
Local Monitor
ApplicationProcess 1
ApplicationProcess 1
User Interface
Computing Element
Worker Node
Worker Node
Site ASite A
Site CSite C
Service Manager
Service Manager
Local Monitor
Local Monitor
ApplicationProcess 4
ApplicationProcess 4
ComputingElement
Worker Node
Service Manager
Service Manager
Main Service
Manager
Main Service
Manager
Site BSite B
ApplicationProcess 3
ApplicationProcess 3
ApplicationProcess 2
ApplicationProcess 2
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 24
OCM-G – FeaturesOCM-G – Features On-line On-line operationoperation Support for Support for multi-site grid applicationsmulti-site grid applications Low perturbationLow perturbation
• Techniques for data rate Techniques for data rate reduction reduction • LLightweight and fast socket-based communicationightweight and fast socket-based communication
FlexibleFlexible, , sservices-drivenervices-driven designdesign• No fixed metrics but a set of flexible services to construct metrics with No fixed metrics but a set of flexible services to construct metrics with
desired semanticsdesired semantics• Enables custom metrics in G-PMEnables custom metrics in G-PM
ExtendibleExtendible• AAdditional servicesdditional services can easily be added can easily be added• LLoaded dynamically at run-timeoaded dynamically at run-time
SSecureecure• GSI-based security GSI-based security • Minimal security requirements Minimal security requirements
Autonomous and standardizedAutonomous and standardized • SStandard interface tandard interface • MMinimizeinimizedd effort of porting OMIS-based tools across platforms effort of porting OMIS-based tools across platforms • Enabled interoperability of multiple tools monitoring a single application.Enabled interoperability of multiple tools monitoring a single application.
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 25
GridBench: The SuiteGridBench: The Suite
Layered Layered approachapproach
Worker-nodeWorker-node
SiteSite
and VO leveland VO level
Micro-benchmarksMicro-benchmarks
Micro-kernel Micro-kernel BenchmarksBenchmarks
Application-kernel Application-kernel BenchmarksBenchmarks
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 26
UDAL - UDAL - Unified Data Access LayerUnified Data Access Layer
DatabaseStorage
NAS
Replica ManagerEDG Reptor
Applications Portal
HSM
Componentsare kinds of
plug-ins
Components are grouped by types but they differ from each other by specialization. The rule based system takes the decision which particulartype component is the best in a context.
Applications
Applications
Unified Data Access Layer based on Component-Expert Architecture
Component TypeTID1
Component TypeTID1 Component Type
TID1
Component TypeTID1
Component TypeTID2 Component Type
TID1
Component TypeTID3 Component Type
TID1
Component TypeTID1
Component TypeTID4
Unified Data Access Cost Estimation Unified Data Access via GridFTP Protocol
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 27
BrokerClient
BrokerService
Site A
Site B
RegistryService
SiteC
RegistryService
SiteC
UserBenchmark
Services
BenchmarkAnaliserServices
InfrastructureMonitoringServices
ApplicationMonitoringServices
MigrationService
Broker SupportServices
MigrationSupportServices
HLA SpeakingService
RTIExecService
HLA ManagementServices
N-th Gridsitesupporting HLA
ApplicationFederate
Code
Gridsitesupporting HLA
HLA SpeakingService
RTIExecService
HLA Management
Services
RTIExec
HLABus
Broker SupportServices
MigrationSupportServices
PerformanceDecisionService
ApplicationMonitoring
MainServiceManager
Grid HLAGrid HLA Management System Management System HLA management servicesHLA management services
• HLA-speaking Service for HLA-speaking Service for managing federatemanaging federatess
• RTIExec Service for RTIExec Service for managing RTIExec managing RTIExec (coordination process in (coordination process in RTI)RTI)
• Broker Broker for setting up a for setting up a federationfederation and and makmakinging migration decisionsmigration decisions
Broker decision servicesBroker decision services• Registry Registry for for storingstoring
location of location of HLA-speaking HLA-speaking serviceservicess
• Infrastructure Infrastructure Monitoring/BenchmarksMonitoring/Benchmarks for checking environment for checking environment of HLA serviceof HLA service
Migration support servicesMigration support services• Application Monitoring Application Monitoring
for monitoring for monitoring performanceperformance
• Migration SMigration Serviceervice
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 28
CrossGrid TestbedCrossGrid Testbed
• Testbed sites in 9 Testbed sites in 9 countriescountries
• 17 testbed sites17 testbed sites
• Three types of Three types of testbeds: testbeds: production, production, development, testdevelopment, test
• Communication: Communication: national research national research networks and networks and GEANTGEANT
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 29
CrossGrid: Innovation, Interactivity,CrossGrid: Innovation, Interactivity, InteroperabilityInteroperability
FeaturesFeatures Brings interactive applications to the GridBrings interactive applications to the Grid Enables easy access toEnables easy access to the Gridthe Grid viavia WWeb eb SServices ervices Extends and enhances Extends and enhances DataGrid, GridLab, and EuroGridDataGrid, GridLab, and EuroGrid DDeveloped according to Geveloped according to GGGF and software engineering standardsF and software engineering standards
Potential CustomersPotential Customers EEnd-users: nd-users: hospitalshospitals, , environmental authoritiesenvironmental authorities, physicists, physicists CCompanies developing computeompanies developing compute--intensive softwareintensive software SService and infrastructure providerservice and infrastructure providers
StatusStatus SStable version available table version available sincesince March 2004 as open source March 2004 as open source LLicensing: CrossGrid license based on EDG, GPLicensing: CrossGrid license based on EDG, GPL CrossGrid Tutorial available for potential usersCrossGrid Tutorial available for potential users
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 30
K-WfGridK-WfGrid
www.kwfgrid.netwww.kwfgrid.net
Fraunhofer FIRST, Berlin, GermanyFraunhofer FIRST, Berlin, Germany Institute of Computer Science, Institute of Computer Science,
University of Innsbruck,University of Innsbruck, Innsbruck, AustriaInnsbruck, Austria Institute of Informatics of the Slovak Institute of Informatics of the Slovak
Academy of Sciences, Bratislava, Academy of Sciences, Bratislava, SlovakiaSlovakia
ACC CYFRONET AGH, Kraków, PolandACC CYFRONET AGH, Kraków, Poland LogicDIS S.A., Athens, GreeceLogicDIS S.A., Athens, Greece Softeco Sismat SpA, Genova, ItalySofteco Sismat SpA, Genova, Italy
Bratislava
Athens
Genova
KrakówInnsbruck
Berlin
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 31
Flood Flood SimulationsSimulations - Workflow - Workflow
User Portal WorkflowService
Workflow Knowledge
StorageService
MeteorologyService
HydrologyService
HydraulicsService
MeteorologyVisualization
HydrologyVisualization
HydraulicsVisualization
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 32
Workflow Applications and KnowledgeWorkflow Applications and Knowledge
• Integrating services into Integrating services into
coherent application scenarioscoherent application scenarios
• Enabling automatic Enabling automatic construction and reuse of construction and reuse of workflows with knowledge workflows with knowledge gathered during operationgathered during operation
• Involving monitoring and Involving monitoring and knowledge acquisition knowledge acquisition services in order to provide services in order to provide added value for end usersadded value for end users
TechnologiesTechnologies: service-oriented Grid architecture,: service-oriented Grid architecture, software agents, software agents, ontologies, dynamicontologies, dynamic instrumentationinstrumentation
Execute workflow
Capture knowledge
Reuse knowledge
K-WfGrid
Monitor environment
Analyze information
Construct workflow
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 33
ArchitectureArchitecture of K-WfGrid of K-WfGrid
• Capturing and Capturing and reusing reusing knowledge knowledge about Grid about Grid environmentsenvironments
• Ontology-based Ontology-based optimization of optimization of workflowsworkflows
• FFramework for ramework for collaborative collaborative knowledge reuseknowledge reuse
Grid Middleware
Web Portal
User Assistant Agent
Automatic Application Builder
Workflow Composition Tools
Grid ServiceInvocation and Control
Grid Performance Analysis Service
Grid OrganizationMemory
Grid Application Building
Users
Knowledge Builder Agent
Grid Workflow Execution Service
<templates><workflows>
<users><components>
<resources>
Grid Resources Grid Performance Monitoring Service
Knowledge
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 34
User
Workflow Orchestration and Execution
Web Portal
Grid Resources
Scheduler
Grid Performance Monitoringand Instrumentation Service
Automatic Application Builder
Workflow Composition Tool
KnowledgeAssimilation Agent
Grid OrganizationalMemory
Ontological store of
knowledge
Low Level Grid Middleware (WS-RF)
Knowledge
Grid Workflow Execution Service
PerformanceAnalysis
User Assistant Agent
Grid Workflow User Interface
User interactionthrough the Portal
Guidancesfor the user
Workflow composition and execution visualizationUser’s decisions in crucial points of execution
Information onavailable
resources andtheir description
Execution of chosenGrid services
Informationabout workflow
execution
Analysed andextractedknowledge
Information aboutresources andenvironment
Information aboutperformance of
particular resources
Flow of ActionsFlow of Actions
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 35
Stages of Workflow ConstructionStages of Workflow Construction
Initial, abstract grid job
Abstract Workflow withService classes
Partially concretized Workflow prior to
execution
Fully concretized Workflow after
Successful execution
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 36
Workflow EnvironmentWorkflow Environment
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 37
Circulation of KnowledgeCirculation of Knowledge
Grid wf Exec System
Initial conditionsprovided by user
Abstract workflowmade by WCT
Concrete workflowmade by AAB
Running workflowScheduler
Service class functionality
Service instance properties
Service instance performance
Hints and guidelines
USER
Services and resourcesmonitoring data
On-line monitoring infrastructure
Events occurred duringwf composition and execution
Event publishing subsystem
Defined metrics
predefined forthe system
supplied by grid service providers
User Assistant Agent
Knowledge Assimilation Agent
Performance analysis
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 38
Monitoring and Performance AnalysisMonitoring and Performance Analysis
• Monitoring and instrumentation service (MIS)Monitoring and instrumentation service (MIS)• Performance analysis service (PAS)Performance analysis service (PAS)• Data representations and service interfaces Data representations and service interfaces
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 39
Monitoring and Performance AnalysisMonitoring and Performance Analysis Monitoring and Instrumentation ServiceMonitoring and Instrumentation Service
• Instrument code regions, and activitiesInstrument code regions, and activities• Monitor infrastructure, code regions, activity execution statusMonitor infrastructure, code regions, activity execution status
Performance Analysis ServicePerformance Analysis Service• Define performance metrics for workflows Define performance metrics for workflows • Analyze monitoring data and relate the data to the workflowAnalyze monitoring data and relate the data to the workflow• Define performance properties and search performance Define performance properties and search performance
bottlenecks of workflowsbottlenecks of workflows
Data Presentations and Service InterfacesData Presentations and Service Interfaces• XML schemas for describing CPU usage,TCP bandwidth, generic XML schemas for describing CPU usage,TCP bandwidth, generic
events, profiling data, workflow activity execution status, etc. events, profiling data, workflow activity execution status, etc. • WIRL, PDQS, common service operations and specific service WIRL, PDQS, common service operations and specific service
operationsoperations
WP InterdependenciesWP Interdependencies• Scheduler and Grid Workflow Execution Service (GWES) - WP2Scheduler and Grid Workflow Execution Service (GWES) - WP2• Grid Organizational Memory - WP4Grid Organizational Memory - WP4• Knowledge Assimilation Agent - WP5 Knowledge Assimilation Agent - WP5
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 40
Ontologies in GOMOntologies in GOM
Some general concepts, Some general concepts, that could be shared by that could be shared by many domain ontologiesmany domain ontologies
Domain specific concepts, Domain specific concepts, water flowwater flow
Computational resource, Computational resource, visualisation resourcevisualisation resource
Rainfall measurement Rainfall measurement resourceresource
File metadataFile metadata
File replicasFile replicas
File formatsFile formats
Meaning of data Meaning of data
Data formatData format
Other flood samples time Other flood samples time perdiodperdiod
Average monitoring Average monitoring parametersparameters
Service class (e.g. FFT Service class (e.g. FFT solver)-Service inputssolver)-Service inputs;; Service outputsService outputs; ; Service Service preconditionspreconditions, effects, effects
Workflow patternsWorkflow patterns
Workflow historyWorkflow history
Domain specific Domain specific requirements – no parallelrequirements – no parallel
Data
Serv
ice
Ap
plicati
on
Work
flow
Ontology Scope
On
tolo
gy T
yp
e
Generic Specific
Resou
rces
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 41
Levels of KnowledgeLevels of Knowledge Knowledge is separated into three levels Knowledge is separated into three levels
• Generic – definitions of concepts, taxonomyGeneric – definitions of concepts, taxonomy• Domain specific – definition of domain specific topicsDomain specific – definition of domain specific topics• Data – individuals from a concrete applicationData – individuals from a concrete application
Knowledge is gathered in different registries Knowledge is gathered in different registries
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 42
Knowledge Knowledge BBase ase AArchitecturerchitecture
GOM Client
Grid Organisational Memory Service
Service Matchmaker ... Service Annotation ...
Knowledge Usage Knowledge Provision
Organisational Memory Interface
State Handler Event Handler
Security Model
Event ModelCurrent State Model
Persistence
Notifier
Ontology
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 43
Technologies and StandardsTechnologies and Standards
JavaJava
Web Service, SOAPWeb Service, SOAP
Tomcat, Maven, JIBXTomcat, Maven, JIBX
WSRF, Globus Toolkit 4WSRF, Globus Toolkit 4
XML, OWL, RDF, RDQLXML, OWL, RDF, RDQL
OWL-DL (Description logic)OWL-DL (Description logic)
JENA (Java Semantic Web Toolkit)JENA (Java Semantic Web Toolkit)
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 44
Users of K-WfGridUsers of K-WfGridUser communityUser community
Environment:Environment: Flood decision crisis team support system Flood decision crisis team support system
Business:Business: Enterprise resource planning Enterprise resource planning
Public sector:Public sector: Coordinated traffic management Coordinated traffic management
Developer communityDeveloper community
Grid software developers:Grid software developers: Workflow and knowledge management Workflow and knowledge management toolstools
Application developers:Application developers: Complex distributed application Complex distributed application constructionconstruction
Interested Institutions:
Municipality of Genova Slovak Water Research Institute, Bratislava Slovak Hydrometeorological Institute, Bratislava Slovak Watermanagement Enterprise, Banska Stiavnica
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 45
Project Proposal for F2 Call 5Project Proposal for F2 Call 5
GridSpaceGridSpace
Transparent Semantic Transparent Semantic GridGrid
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 46
Programming grid Programming grid applicationapplication
• Various Grid middleware platforms for uniform access to Various Grid middleware platforms for uniform access to resources – difficult, complex to programresources – difficult, complex to program
• RRecent initiatives ecent initiatives onon Grid programming do not address the Grid programming do not address the dynamic nature of the Griddynamic nature of the Grid
Important features of the GridImportant features of the Grid• Grid is dynamicGrid is dynamic• Resource users do not instantiate resources on their ownResource users do not instantiate resources on their own• RResources esources are in differentare in different administrative domains administrative domains
Therefore, a Grid application has to beTherefore, a Grid application has to be• More loosely coupled (combined of autonomous elements)More loosely coupled (combined of autonomous elements)• Flexible to overcome (and benefit from) Grid dynamic natureFlexible to overcome (and benefit from) Grid dynamic nature• Adaptable to cross boundaries of various administration Adaptable to cross boundaries of various administration
policiespolicies
MotivationMotivation
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 47
FeaturesFeatures• Abstract, semantically rich layer between a user and the Abstract, semantically rich layer between a user and the
middlewaremiddleware• Set of tools for Grid application developer to make the Grid Set of tools for Grid application developer to make the Grid
programming easierprogramming easier• Strong support for developing flexible and adaptable Strong support for developing flexible and adaptable
applicationsapplications
ComponentsComponents• Grid programming languageGrid programming language• Interpreter with dynamic Interpreter with dynamic ad-hocad-hoc binding capabilities binding capabilities• Runtime environment based on tuple space ideaRuntime environment based on tuple space idea• Evolving language library to share and reuse applicationsEvolving language library to share and reuse applications
Technology:Technology: components, services, objects components, services, objects
Concept of the Concept of the GridSpaceGridSpace
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 48
Separates the developer from Separates the developer from ever-changing Grid resource ever-changing Grid resource layerlayer
Seamlessly introduces Seamlessly introduces dynamism into newly created dynamism into newly created applicationsapplications
Provides unified access to Provides unified access to resources by means of resources by means of semantically described semantically described abstractionsabstractions
Supports evolving and well Supports evolving and well organized library of organized library of applications used up-to-dateapplications used up-to-date
Allows easy reuse of already Allows easy reuse of already built built applicationsapplications
GridSpace abstraction of data
abstraction of computations abstraction of components
ReplicaMngmnt Schedulers Resource
Brokers ...
Unified Component Access Layer
Abs
trac
tion
ComponentDeveloper
ApplicationDeveloper
GridUserU
sers
Mid
dlew
are
GRAM Condor LSF ...GridFTP
GridSpaceGridSpace - - Additional Abstract LayerAdditional Abstract Layer
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 49
Operation of GridSpaceOperation of GridSpace
GridSpace
Interpreter
Inactive Data Tuple
Inactive Component Tuple
A
B
Provides- com(x)- print(x)
Provides- y fun(x)- fun2(x)
input A, Bloop 10x call com(A)endloopC = call fun(B)output C
Puts Script to execute
Provides Component
May reuse Script as a new Component
Active ExecutionTuple
Provides Data for computation
Provides Data for community
Dynamic binding links
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 50
From Resource Abundance to Programmable GridFrom Resource Abundance to Programmable Grid
Similar semantic description of all the resourcesSimilar semantic description of all the resources• Using common notions: dependency, requirement, Using common notions: dependency, requirement,
capabilitycapability• Based on growing Semantic Web/Grid achievementsBased on growing Semantic Web/Grid achievements• With natural ability to be extended by multiple usersWith natural ability to be extended by multiple users
AdvantagesAdvantages• Helps build new applicationsHelps build new applications• Common language for various tools and platformsCommon language for various tools and platforms• Does not enforce unification of underlying technologyDoes not enforce unification of underlying technology
Global Grid Environment
Computingpower
Data
Software
Sensors,devices
Networktransfer
DatastoragespaceCommon
SemanticDescription
Layer
The environment provides everything for The environment provides everything for anan applicationapplication: v: vast space, multiplication ast space, multiplication of resources, multitude of access of resources, multitude of access standards and protocolsstandards and protocols
IIssuessuess
• Plenty of resources to build Plenty of resources to build sophisticated applicationsophisticated applications froms from
• Each new application requires huge Each new application requires huge effort to overcome integration effort to overcome integration problemsproblems
Data/events- Sources
- Retainers- Tranformers- Consumers
Global Grid EnvironmentComputingpower
Data
Software
Sensors,devices
Networktransfer
Datastoragespace
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 51
Transparent Transparent Global GridGlobal Grid for Everyday Use for Everyday Use
The applications The applications should beshould be buil builtt/rebuil/rebuiltt almost at runtime almost at runtime Abstraction and transparencyAbstraction and transparency
• Hierarchy of resources enables customisable abstraction Hierarchy of resources enables customisable abstraction levellevel
• No more No more jjob submission black boxesob submission black boxes – control over running – control over running application will be enabled application will be enabled
• AA level of transparency helps users understand level of transparency helps users understand the the GridGrid The path The path to our main objective: plugto our main objective: pluggingging problems into the Grid problems into the Grid
to have them solveto have them solved!d!
Global Grid Environment
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 52
ContactContact
www.cyfronet.krakow.pl/en/www.cyfronet.krakow.pl/en/
[email protected]@agh.edu.pl