HEP Data Grid in Japan Takashi Sasaki Computing Research Center KEK.
start [Open Grid Forum] - RNS Requirements from KEKRequirments...RNS Requirements from KEK based on...
Transcript of start [Open Grid Forum] - RNS Requirements from KEKRequirments...RNS Requirements from KEK based on...
RNS Requirements from KEKRNS Requirements from KEKbased on operational experiences based on operational experiences
using LCG/SRB/NAREGIusing LCG/SRB/NAREGI
Go Iwai, KEK/CRCGo Iwai, KEK/CRC
The 24th Open Grid ForumThe 24th Open Grid ForumThe Matrix Building, Singapore, SingaporeThe Matrix Building, Singapore, Singapore
September 15th September 15th ‐‐19th, 200819th, 2008
DayDay‐‐2: 11:00 2: 11:00 ‐‐ 12:3012:30 Grid File System WGGrid File System WG
OutlineOutline
1. Introduction1. Introduction
‐‐ MotivationMotivation
‐‐ What we want to do with RNSWhat we want to do with RNS
‐‐ ScenarioScenario
2. Requirements from experiences2. Requirements from experiences
‐‐ What we learn from practiceWhat we learn from practice
3. Summary3. Summary
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 22
Tohoku Univ.Tohoku Univ.
KEKKEKUniv. of TsukubaUniv. of Tsukuba
Nagoya Univ.Nagoya Univ.Kobe Univ.Kobe Univ.
Hiroshima ITHiroshima IT
IntroductionIntroduction•• KEK is the biggest laboratory for HEP in Japan.KEK is the biggest laboratory for HEP in Japan.•• Major HEP projects:Major HEP projects:
–– Belle, JBelle, J‐‐PARC, ATLAS (on going projects)PARC, ATLAS (on going projects)–– ILC, SuperILC, Super‐‐B (future projects)B (future projects)
•• We have a role to support university groups in these fields.We have a role to support university groups in these fields.–– including Grid deployment/operation.including Grid deployment/operation.
•• A VO A VO ““Accelerator Science in JapanAccelerator Science in Japan”” has been federated among us has been federated among us and in operation.and in operation.
–– By using gLite/EGEEBy using gLite/EGEE
•• NAREGI is the infrastructure of the eNAREGI is the infrastructure of the e‐‐Science JapanScience Japan–– Operational supports will be provided by NII (National InstituteOperational supports will be provided by NII (National Institute of of
Informatics) as the GOC in Japan.Informatics) as the GOC in Japan.
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 33
MotivationMotivation
•• We need to operate multi Grid middleware at the We need to operate multi Grid middleware at the same time.same time.–– Resource sharing among them is mandate Resource sharing among them is mandate
•• We are also contributing to GINWe are also contributing to GIN•• SAGASAGA‐‐NAREGI adapter is being developedNAREGI adapter is being developed
–– Data exchange already have been realized by gridFTPData exchange already have been realized by gridFTP•• Middleware independent file catalogue service is Middleware independent file catalogue service is desirable desirable –– Sharing the same logical file name spaceSharing the same logical file name space
•• Virtualization of Grid middleware is our wishVirtualization of Grid middleware is our wish–– The best scenario for the application developers The best scenario for the application developers
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 44
The current situation The current situation
•• Many middleware exist in the whole world.Many middleware exist in the whole world.–– GIN provides the interoperability among middleware, GIN provides the interoperability among middleware, however, the implementation is not yet finalized however, the implementation is not yet finalized
•• gLite, NAREGI, OSG, GT, SRB, iRODS, Gfarm, ...gLite, NAREGI, OSG, GT, SRB, iRODS, Gfarm, ...–– Different implementations (APIs) for applications are Different implementations (APIs) for applications are necessary to use different Grid middleware services necessary to use different Grid middleware services
•• We can exchange files among different We can exchange files among different middleware so far, but no way to know the file middleware so far, but no way to know the file names for gridFTP transfers in the other sidenames for gridFTP transfers in the other side–– We need to peep the other middleware environment We need to peep the other middleware environment somehow somehow
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 55
Our wish is RNSOur wish is RNS
•• Single logical file name space can be shared among different Single logical file name space can be shared among different middleware middleware
•• Sharing resources transparently among different middleware Sharing resources transparently among different middleware
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 66
LFCLFCLFC
RNSRNSRNS
MCATMCATMCAT
MDSMDSMDS
/rns/rns
/srb/srb
/dir/dir‐‐AA
/dir/dir‐‐XX
/dir/dir‐‐YYfilefile‐‐11
filefile‐‐22filefile‐‐33
filefile‐‐44
/lcg/lcg
/dir/dir‐‐BB
legacylegacyusersusers
newnewcomerscomers
JunctionJunction
2. Requirements on RNS from Experiences on 2. Requirements on RNS from Experiences on LCG/gLite operation LCG/gLite operation Lessons learn from practiceLessons learn from practice
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 77
Load balancing and Redundancy Load balancing and Redundancy
•• Load balancing, redundancy and fail over are Load balancing, redundancy and fail over are required to avoid critical service down required to avoid critical service down –– Central catalogue in VO should not down Central catalogue in VO should not down especiallyespecially
•• Our experience on LCG/gLiteOur experience on LCG/gLite–– The only one LFC (LCG File Catalogue) can be The only one LFC (LCG File Catalogue) can be centrally operated in each VO.centrally operated in each VO.
–– Load balancing is only realized by the round robin Load balancing is only realized by the round robin host name resolution.host name resolution.
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 88
Requirement 1.Requirement 1.
Bulk file registration and file namingBulk file registration and file naming
•• Bulk file space registration is very useful Bulk file space registration is very useful –– 3 storage system composed of LCG, SRB and NAREGI (trial 3 storage system composed of LCG, SRB and NAREGI (trial use) are operated at KEK.use) are operated at KEK.
•• Belle already has a few PBs data in total including real Belle already has a few PBs data in total including real data and simulation data.data and simulation data.
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 99
LFCLFCLFC
SRBSRBSRBSRMSRMSRM
MCATMCATMCAT MDSMDSMDS
GfarmGfarmGfarm
2PB2PB2PBLCGLCG SRBSRB NAREGINAREGI
only focused on data grid partonly focused on data grid part
Requirement 2.Requirement 2.
physicalphysicalcopycopy
•• The The ““SregisterSregister”” command in SRB is very useful command in SRB is very useful –– We could avoid the physical copy of existing data to the LCG stoWe could avoid the physical copy of existing data to the LCG storage arearage area–– SRB works as the gridFTP server by using SRBSRB works as the gridFTP server by using SRB‐‐DSI DSI
•• LFC (LCG File Catalogue) keeps LFN, GUIDs and SURLs, but no TURLLFC (LCG File Catalogue) keeps LFN, GUIDs and SURLs, but no TURLss–– Logical File Name (LFN) Logical File Name (LFN)
•• An alias created by a user to refer to some item of data, e.g. An alias created by a user to refer to some item of data, e.g. ““lfn:/atlas/20030203/run2/track1lfn:/atlas/20030203/run2/track1””–– Grid Unique ID (GUID) Grid Unique ID (GUID)
•• A nonA non‐‐humanhuman‐‐readable unique identifier for an item of data, e.g. readable unique identifier for an item of data, e.g. ““guid:f81d4faeguid:f81d4fae‐‐7dec7dec‐‐11d011d0‐‐a765a765‐‐00a0c91e6bf600a0c91e6bf6””
–– Site URL (SURL)Site URL (SURL) (or Physical File Name (PFN) or Site FN) (or Physical File Name (PFN) or Site FN) •• The location of an actual piece of data on a storage system, e.gThe location of an actual piece of data on a storage system, e.g. .
““srm://pcrd24.cern.ch/flatfiles/cms/output10_1srm://pcrd24.cern.ch/flatfiles/cms/output10_1”” (SRM)(SRM) ““sfn://lxshare0209.cern.ch/data/alice/ntuples.datsfn://lxshare0209.cern.ch/data/alice/ntuples.dat”” (Classic SE) (Classic SE) –– Transfer URL (TURL) Transfer URL (TURL)
•• Temporary locator of a replica + access protocol: understood by Temporary locator of a replica + access protocol: understood by a SE, e.g. see belowa SE, e.g. see below–– Only SRM knows TULRs in LCG/gLiteOnly SRM knows TULRs in LCG/gLite
•• Other file catalogue in gLite will be investigatedOther file catalogue in gLite will be investigated
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 1010
LFCLFCLFC
SRBSRBSRBSRMSRMSRM
MCATMCATMCAT MDSMDSMDS
GfarmGfarmGfarm
2PB2PB2PBLCGLCG SRBSRB NAREGINAREGI
SRB‐DSISRBSRB‐‐DSIDSIGridFTPGridFTP
SregisterSregister
Requirement 2.Requirement 2.
ScommandScommand
% globus% globus--urlurl--copy gsiftp//disk.kek.jp:/lcg/file195fa222copy gsiftp//disk.kek.jp:/lcg/file195fa222--bc06bc06--4a304a30--b633b633--b194ba7e0240 file://path/to/fileb194ba7e0240 file://path/to/file
GridFTPGridFTP
•• RNS should keep SURLs as well as TURLs.RNS should keep SURLs as well as TURLs.
•• BulkBulk‐‐register CLI (Command Line Interface) is register CLI (Command Line Interface) is necessary.necessary.
•• rnsrns‐‐register [register [‐‐b|b|‐‐‐‐bulk]bulk]
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 1111
<rns><guid name=“guid:3160415c-e5ed-4f09-8787...”>
<lfn name=“/grid/vo/path/to/file-A”><surl name=“srm//srm.kek.jp/file195fa222...”>
<turl name=“gsiftp//gftp-1.kek.jp:/lcg/file195fa22...” /></surl><surl name=“srm//srm.kek.jp/file195fa223...”>
<turl name=“gsiftp//gftp-2.kek.jp:/lcg/file195fa23...” /></surl>
</lfn></guid>
</rns>
<rns><rns><guid name=<guid name=““guid:3160415cguid:3160415c--e5ede5ed--4f094f09--8787...8787...””>>
<lfn name=<lfn name=““/grid/vo/path/to/file/grid/vo/path/to/file--AA””>><surl name=<surl name=““srm//srm.kek.jp/file195fa222...srm//srm.kek.jp/file195fa222...””>>
<turl name=<turl name=““gsiftp//gftpgsiftp//gftp--1.kek.jp:/lcg/file195fa22...1.kek.jp:/lcg/file195fa22...”” />/></surl></surl><surl name=<surl name=““srm//srm.kek.jp/file195fa223...srm//srm.kek.jp/file195fa223...””>>
<turl name=<turl name=““gsiftp//gftpgsiftp//gftp--2.kek.jp:/lcg/file195fa23...2.kek.jp:/lcg/file195fa23...”” />/></surl></surl>
</lfn></lfn></guid></guid>
</rns></rns>
rough image...rough image...
Requirement 2.Requirement 2.
Automatic Replica location selection Automatic Replica location selection
•• Wish in RNSWish in RNS–– We want to access data on really faster storage.We want to access data on really faster storage.
•• i.e. If replica exists somewhere in the world, data should be obi.e. If replica exists somewhere in the world, data should be obtained from the tained from the nearest/fastest storage. nearest/fastest storage.
•• Faster file transfer is always key issue for us.Faster file transfer is always key issue for us.–– Attributes for transfer metric, e.g. Attributes for transfer metric, e.g. ““Physical locationPhysical location””, , ““Typical Typical
transfer speedtransfer speed””, are helpful for users to find the best storage., are helpful for users to find the best storage.
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 1212
% lcg-lr -v lfn:/path/to/filesrm://ccsrm02.in2p3.fr/.../w11782_03.stdhepsrm://globe-door.ifh.de/.../w11782_03.stdhepsrm://rls06.cc.kek.jp/.../filecd912a14-8be1-423f-b55c-449cbd82d347srm://srm-dcache.desy.de/.../w11782_03.stdhep
:% lcg-cp --verbose lfn://path/to/file file://path/to/file
:Source URL for copy: gsiftp://zyklop35.ifh.de:2811/.../w11782_03.stdhep
:
% lcg% lcg--lr lr --v lfn:/path/to/filev lfn:/path/to/filesrm://ccsrm02.in2p3.fr/.../w11782_03.stdhepsrm://ccsrm02.in2p3.fr/.../w11782_03.stdhepsrm://globesrm://globe--door.ifh.de/.../w11782_03.stdhepdoor.ifh.de/.../w11782_03.stdhepsrm://rls06.cc.kek.jp/.../filecd912a14srm://rls06.cc.kek.jp/.../filecd912a14--8be18be1--423f423f--b55cb55c--449cbd82d347449cbd82d347srm://srmsrm://srm--dcache.desy.de/.../w11782_03.stdhepdcache.desy.de/.../w11782_03.stdhep
::% lcg% lcg--cp cp ----verbose lfn://path/to/file file://path/to/fileverbose lfn://path/to/file file://path/to/file
::Source URL for copy: Source URL for copy: gsiftp://zyklop35.ifh.de:2811/.../w11782_03.stdhepgsiftp://zyklop35.ifh.de:2811/.../w11782_03.stdhep
::
ExampleExample
list SURLslist SURLs
expected SURLexpected SURLfor user at KEKfor user at KEK
actual TURLactual TURL
Requirement 3.Requirement 3.
Metadata RequirementsMetadata Requirementsfor typical usagefor typical usage
AttributeAttribute ContentContent
ACLACL Access Control List (Grid level)Access Control List (Grid level)GSI authentication with VOMS extension is desirable to avoid accGSI authentication with VOMS extension is desirable to avoid accidental incidents or nasty activities.idental incidents or nasty activities.
TURLTURL Transfer URL (e.g. gsiftp://...)Transfer URL (e.g. gsiftp://...)Since gridftp is a defact protocol for file transferring, TURL Since gridftp is a defact protocol for file transferring, TURL attribute boosts up to migrate to RNS.attribute boosts up to migrate to RNS.
Physical LocationPhysical Location e.g. /storage/path/to/filee.g. /storage/path/to/file
Transfer metricTransfer metric e.g. RTT etc.e.g. RTT etc.
Type of entryType of entry Directory or file or junctionDirectory or file or junction
Time stampTime stamp Last accessLast accessLast modificationLast modificationLast status changeLast status change
SizeSize File sizeFile size
# of links# of links
Access counterAccess counter
SURLSURL Storage URL (e.g. srm://...)Storage URL (e.g. srm://...)
Middleware IDMiddleware ID e.g. SRB, iRODS, gLite, NAREGIe.g. SRB, iRODS, gLite, NAREGI
User metadataUser metadata ““Run #123Run #123””; ; ““Calibration RunCalibration Run””;;
Access ProtocolAccess Protocol Gsiftp, rfio, ...Gsiftp, rfio, ...
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 1313
Especially need ACL, TURL and transfer metric Especially need ACL, TURL and transfer metric
Requirement 4.Requirement 4.
Required commandsRequired commandsfor typical usagefor typical usage
NameName DescriptionDescription
rnsrns--lsls List directory or file entries in the RNSList directory or file entries in the RNS
rnsrns--queryquery Query user specific attributes, e.g. run number, comments and sQuery user specific attributes, e.g. run number, comments and so ono on
rnsrns--rmrm Remove a file/directory in the RNSRemove a file/directory in the RNS
rnsrns--chmodchmod Change access mode of a file/directory in the RNSChange access mode of a file/directory in the RNS
rnsrns--chownchown Change owner/group of a file/directory in the RNSChange owner/group of a file/directory in the RNS
rnsrns--chgrpchgrp Change group ownership of a file/directory in the RNSChange group ownership of a file/directory in the RNS
rnsrns--set/getset/get--aclacl Set/Get ACL of a file/directory in the RNSSet/Get ACL of a file/directory in the RNS
rnsrns--renamerename Rename a file/directory in the RNSRename a file/directory in the RNS
rnsrns--lnln Make a symbolic link to a file/directory in the RNSMake a symbolic link to a file/directory in the RNS
rnsrns--mkdirmkdir Make a directory in the RNSMake a directory in the RNS
rnsrns--register/unregisterregister/unregister Register/unregister file(s) or directory into/from the RNSRegister/unregister file(s) or directory into/from the RNS
rnsrns--listlist--repicarepica List the replicas for a given LFN or GUIDList the replicas for a given LFN or GUID
rnsrns--getget--guidguid Get the GUID(s) for a given LFN, SURL or TURLGet the GUID(s) for a given LFN, SURL or TURL
rnsrns--getget--surlsurl Get the SURL(s) for a given LFN , GUID or TURLGet the SURL(s) for a given LFN , GUID or TURL
rnsrns--getget--turlturl Get the TURL(s) for a given LFN , GUID or SURLGet the TURL(s) for a given LFN , GUID or SURL
rnsrns--replicatereplicate Replicate a file from SE to another SE, after then new end pointReplicate a file from SE to another SE, after then new end point is created in the RNS is created in the RNS
rnsrns--cpcp Copy a file/directory from SE/local to local/SECopy a file/directory from SE/local to local/SE
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 1414
Mostly inspired from LFC/LCG CLIMostly inspired from LFC/LCG CLI
Requirement 5.Requirement 5.
The list of Requirements for RNSThe list of Requirements for RNSInstead of SummaryInstead of Summary
RequirementRequirement PriorityPriority
CLI CLI ““rnsrns‐‐register/unregisterregister/unregister”” enabled bulk optionenabled bulk option HIGHESTHIGHEST
RNS keeps SURLs as well as TURLs.RNS keeps SURLs as well as TURLs.RNS has an attribute for TURL. RNS has an attribute for TURL.
HIGHESTHIGHEST
Load balancing, redundancy and fail overLoad balancing, redundancy and fail over HIGHERHIGHER
Attributes for transfer metricAttributes for transfer metrice.g. e.g. ““Physical locationPhysical location””, , ““Typical transfer speedTypical transfer speed””
HIGHHIGH
GSI authentication with VOMS extensionGSI authentication with VOMS extension HIGHHIGH
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 1515
sorted by priority ordersorted by priority order
Many thanks for your attentionsMany thanks for your attentions
* perhaps reflecting my bias* perhaps reflecting my bias
HIGH
HIGH
LOW
LOW
BackupsBackups‐‐ Typical usage of SRM/LFCTypical usage of SRM/LFC‐‐ Data Grid Federation over the Belle Virtual OrganizationData Grid Federation over the Belle Virtual Organization‐‐ Current usage of data grid to access inside LCG or SRBCurrent usage of data grid to access inside LCG or SRB
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 1616
Typical Usage of SRM/LFCTypical Usage of SRM/LFC•• LFC: the only file catalogue officially supported in WLCG/EGEE.LFC: the only file catalogue officially supported in WLCG/EGEE.•• The only one LFC should be centrally operated in each VO.The only one LFC should be centrally operated in each VO.•• MySQL and OracleDB are supported by LFC as the backend so far.MySQL and OracleDB are supported by LFC as the backend so far.•• The mappings between LFNs, GUIDs and SURLs are kept in the LFC.The mappings between LFNs, GUIDs and SURLs are kept in the LFC.
–– TURLs are in SRMs.TURLs are in SRMs.•• SRM is a protocol to manage storage resources, not a file accessSRM is a protocol to manage storage resources, not a file access protocolprotocol
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 1717
LFN 1LFN 1LFN 1
LFN mLFN mLFN m
GUIDGUIDGUID
SURL 1SURL 1SURL 1
SURL nSURL nSURL n
TURL 1TURL 1TURL 1
TURL nTURL nTURL n
LFN 2LFN 2LFN 2::
::::
LFCLFC
Example of LFN, GUID, SURL and TURLExample of LFN, GUID, SURL and TURL
SRMSRM
Belle Virtual OrganizationBelle Virtual Organization
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 1818
KEKKEK
Current Usage of Data GridCurrent Usage of Data GridAccess inside LCGAccess inside LCG
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 1919
LFCLFCLFC
LCGLCG
SRMSRMSRM
MCATMCATMCAT
SRBSRB
SRBSRBSRB
2PB2PB2PB
SRB‐DSISRBSRB‐‐DSIDSI
11
22
33
Current Usage of Data GridCurrent Usage of Data GridAccess inside SRBAccess inside SRB
September 15th September 15th ‐‐ 19th, 200819th, 2008 RNS Requirements from KEK RNS Requirements from KEK ‐‐‐‐ Go Iwai, KEK/CRCGo Iwai, KEK/CRC 2020
LFCLFCLFC
LCGLCG
SRMSRMSRM
MCATMCATMCAT
SRBSRB
SRBSRBSRB
2PB2PB2PB
SRB‐DSISRBSRB‐‐DSIDSI