2006. 07. 19 APAN22 Grid Middleware Workshop KISTI ...€¦ · Job submission & allocation services...
Transcript of 2006. 07. 19 APAN22 Grid Middleware Workshop KISTI ...€¦ · Job submission & allocation services...
KMI-R1 - K*Grid Middleware Package -
KMI-R1 - K*Grid Middleware Package -
2006. 07. 19APAN22 Grid Middleware Workshop
KISTI Supercomputing CenterJaegyoon Hahm
Introduction of K*GridIntroduction of K*Grid
• Goal : Implementation of the national Grid infrastructure in Korea
• Fund : Government level support by MIC
• Period : 2002 - 2006 (5 Years)
• Total budget : about U$ 32M
• Leading organization : KISTI
• Partners : Various research partners selected from academia, industry, and government lab. through a public competition
Activities in K*GridActivities in K*Grid
participation MOA
Open Grid Forum
endorsement
support
GFK
TERA Grid
collaboration
ApGrid
collaboration
PRAGMA
Grid Testbed
SNU, KAISTPOSTECH,CBU, TIT,etc.
Supercomputing Alliance
NPACI
APEC APGrid collaboration
Singapore EC
KMI-R1: Grid Service PackageKMI-R1: Grid Service Package
• The first release as the K*Grid Middleware package to be utilized for constructing a Grid
• It contains the essential middleware components for the Grid environment.
• Developed for helping to build K*Grid infrastructure, but not limited for it.
• OGSI based Grid Service Package
MoreDreamMoreDream
• Objectives▶Development of a Grid middleware which makes possible
to organize a Grid environment easily on demand▶Middleware development for computational Grid▶ Implementation of Grid functionalities based on Globus
Toolkit with modifications
• Main Research Topics▶Grid resource management system▶Grid information system▶MPICH on Grid
• Following GGF Standard▶OGSI-compliant now and going to WSRF
Structure of MoreDreamStructure of MoreDream
GRASP(Grid Resource Allocation Services Package)
GRASP(Grid Resource Allocation Services Package)
• A package of Grid services which are related to resource allocation process in Grid
• A tool for users to submit jobs easily to Grid environment
• Main Features▶ Job submission & allocation services based on OGSI
▶ Supporting MPI-based HPC job and HTC job
▶ Automatic resource selection by Grid scheduling mechanism
▶ Enhancement of reliability of resource selection with resource reservation
▶ Job could automatically stage in files from SRB server and stageout the files to SRB server via SRB enabled globus-url-copy
▶ New job submission language, JRDL (Job & Resource Description Language), for resource matchmaking
Architecture of GRASPArchitecture of GRASP
Major Components of GRASPMajor Components of GRASP
• Job Submission Service▶Carrying out job submission process as an agent of
clients
▶Co-allocation
▶Cross-site MPI job submission
▶Integrated monitoring service about a running job
• Resource Manager Service▶Resource allocation manager for a job from remote site
▶Remote file staging
Major Components of GRASPMajor Components of GRASP
• Resource Reservation Service▶Supporting immediate or advanced reservation
• Grid Scheduling Service▶Providing appropriate resource candidate sets
satisfying the resource requirements
▶Carrying out the scheduling process between a job and selected resources
▶Making reservations to the selected resources for a scheduled job
• Fault Tolerant System▶Providing mechanism to resume the job which was
stopped abnormally
GAIS(Grid Advanced Information System)GAIS(Grid Advanced Information System)
• A package of Grid services which are related to provide mechanisms for discovery and monitoring, and hence for planning and adapting application behavior
• Index service data carrying state information from multiple Grid service instances
• Main Features▶ Universal VO management & Flexible VO construction ▶ Categorization of Grid services▶ Classification of service data▶ Decentralized architecture▶ Flat, dynamic network of directory servers▶ Group mechanism based on OGSI factory approach▶ Query load balancing for efficient VO query▶ Rich information providers▶ User-friendly interface implemented in portlets
Architecture of GAISArchitecture of GAIS
DatacanFactoryService
Prican
Pubcan
Prican
PubcanPubcan
Site A
Site B
RIPS GSGS
GS
RIPS
GS GS
Registration &Subscription
Prican
PubcanPubcan
Site C
Prican
Pubcan
Site D
DatacanFactoryService
DatacanFactoryService
DatacanFactoryService
IP IP IP IPIP * Pubcan: Public Datacan
* Prican: Private Datacan* GS: Grid Service* RIPS: Resource Information Provider Service* IP: Information Provider
RIPS
IP IPIP
VO RosterService
VO CrawlerFactory Service
VO RosterService
VO-A VO-B
VO Registration
VO CrawlerFactory Service
VO CrawlerService
VO Query(find-vo-data & find-vo-service)
Site Query(find-service-data)
VO snapshot VO CrawlerService
VO snapshot
Thread based query
DatacanFactoryService
Prican
Pubcan
Prican
PubcanPubcan
Site A
Site B
RIPS GSGS
GS
RIPS
GS GS
Registration &Subscription
Prican
PubcanPubcan
Site C
Prican
Pubcan
Site D
DatacanFactoryService
DatacanFactoryService
DatacanFactoryService
IP IP IP IPIP * Pubcan: Public Datacan
* Prican: Private Datacan* GS: Grid Service* RIPS: Resource Information Provider Service* IP: Information Provider
RIPS
IP IPIP
VO RosterService
VO CrawlerFactory Service
VO RosterService
VO-A VO-B
VO Registration
VO CrawlerFactory Service
VO CrawlerService
VO Query(find-vo-data & find-vo-service)
Site Query(find-service-data)
VO snapshot VO CrawlerService
VO snapshot
Thread based query
Components of GAISComponents of GAIS
• DataCan Factory Service ▶Enhanced version of index service
• VO Roster Service ▶Management of VO registration
• VO Crawler Factory Service ▶Agent for VO query
MPICH-GX(Message Passing Interface CHameleon-Grid eXtensible)
MPICH-GX(Message Passing Interface CHameleon-Grid eXtensible)
• Enables an MPI job to execute through Grid middleware such as GT3 by file based initialization
• Main Features▶File-based MPI initialization
▶Private IP Support for MPI communication between private IP nodes
▶MPICH libraries for fault tolerance
▶Initialization enabling a job to co-allocate among remote sites at GRASP
Components of MPICH-GXComponents of MPICH-GX
• MPICH-GF
▶Fault tolerant MPICH by check-point mechanism
▶ Integration with GRASP
• MPICH-GP
▶MPI job support in private IP environment
▶ Integration with GRASP
KGridCA SystemKGridCA System
• Certificate Authority for K*Grid Testbed operation
• Providing Web user interface (http://ca.gridcenter.or.kr )
• Approved as a production-level CA by ApGrid PMA
• Certificate Management (Requesting, Issuing, Storing) by Web browsers
• Email Notification (CSR (Certificate Signing Request) Upload, Certificate Issue)
AIService(Accounting Information System)
AIService(Accounting Information System)
• Management of Grid account & accounting information service
• OGSI-compliant Service which gathers accounting information from heterogeneous platforms and provides accounting information as a standard form
• Provide a portlet for Grid accounting information working on the GridSphere portal
Local Accounting Information
Database
AIT ( Acct. Info. Tracker )
Grid Accounting Information
Linux, Linux, OpenPBSOpenPBS
ait-client-put
AIX, AIX, LoadLevelerLoadLeveler
ait-client-lut
grid-mapfileor globus-gatekeeper.logor other info. sources
Bind Information
AIS( org.globus.kgrid.services.AIS )
GT3
UR-WG, GGF
Standard XML Format
Local Accounting Information
Database
AIT ( Acct. Info. Tracker )
Grid Accounting Information
Linux, Linux, OpenPBSOpenPBS
ait-client-put
Linux, Linux, OpenPBSOpenPBS
ait-client-put
AIX, AIX, LoadLevelerLoadLeveler
ait-client-lut
AIX, AIX, LoadLevelerLoadLeveler
ait-client-lut
grid-mapfileor globus-gatekeeper.logor other info. sources
Bind Informationgrid-mapfileor globus-gatekeeper.logor other info. sources
Bind Informationgrid-mapfileor globus-gatekeeper.logor other info. sources
Bind Information
AIS( org.globus.kgrid.services.AIS )
GT3
UR-WG, GGF
Standard XML FormatUR-WG, GGF
Standard XML Format
KMI-GridSphereKMI-GridSphere
• Portal framework from GridSphere with KMI-R1 Grid service portlets▶GRASP job submission
▶GAIS information
▶Accounting Information
▶Easy development using visual beans and the GridSphere User Interface tag library
KMI-R1 TestbedKMI-R1 Testbed
• Linux clusters located on 5 sites in Korea
A Pilot Application for KMI-R1A Pilot Application for KMI-R1
Workstation
Database
orData Grid
GenomeSequences
PreprocessorFragment genome
Splitter
...
Database
orData Grid
ComputationalGrid
GenomeFragments
Database
orData Grid
Aligned GenomeFragments
Database
orData Grid
Aligned Genome
Parallel jobsHomolog
sequence search
...
Parallel jobsSequence
assembling
...
ComputationalGrid
Workstation
Genome AnalysisResult Viewer
Start Stop
Work flowData flow
DB formatter
Filtering
Bioinformatics Application : Comparative Global Genome Sequence Analysis
Thank youThank you