Data Management

The European DataGrid Project Team

http://www.eu-datagrid.org

EDG DataManagement Tutorial - n° 2

Problem Statement:How to connect

User/Programs/Data?

User logged in to a Grid “User Interface” machine, or

Logged in to a “desktop” machine

Programs On desktop

On UI

On Grid machines “god knows where”

Data May need to supply (Grid or non-Grid) data to GNW programs

GNW program may generate data, need to put it somewhere safe

How do you retrieve it from somewhere safe?

EDG DataManagement Tutorial - n° 3

Common Grid Data Management Tasks

Dealing with Data Your Job Generates Getting the data back to your desktop Putting the data “on the Grid”

Getting Data to your Job Submitting data along with your job Putting your data onto the Grid (from outside) Sending your Grid job to your Grid data

Moving Data on the Grid

How to find your data if you don’t remember where you put it

Example scripts and files: ~dgttutor/dm-tests/

EDG DataManagement Tutorial - n° 4

Grid Data Management Tools

Data Transfer mostly through gsiftp Like good old FTP except uses grid auth(oriza)(entica)tion No passwords! Can also use multiple streams for faster transfer

Resource Broker can send (small amounts) of data to/from jobs

Replica Catalog keeps track of where various copies of “grid datasets” are located

Edg-replica-manager uses gsiftp & RC to manage instantiation, registration, and replication of grid datasets

Resource Broker can use RC to find your data, and send your job to it, if you tell RB about the data you need

EDG DataManagement Tutorial - n° 5

Grid Program -> Data on your desktop

You can set up your job for “data pickup” Job generates data in current working directory on WN

At job end, the data files are placed in temp storage at RB

You get them back via “dg-job-get-output”

Key items: You need to know names of files you want to get back

OutputSandbox = {“higgs.root",“graviton.HDF"};

not intended for large files (> hundred MB) – storage limitation on Resource Broker machine

Example: output-sandbox.{jdl,sh}

EDG DataManagement Tutorial - n° 6

Putting the data “on the Grid”

Here we talk about a running Grid program, the output of which you want on the Grid. Two cases:

You let the program write output on the WN, and after the program finishes you have the job script move the data to Grid storage

You arrange for the program to write directly to Grid storage

In both cases, data is not really “on the Grid” until it is registered in the “replica catalog”

EDG DataManagement Tutorial - n° 7

Grid-generated data to Grid storage I

Your program generates data to some local file

You have to know (or be able to figure out) what the local file name is

Use the edg-replica-manager commands to Put the data onto Grid storage

Register the data as a Grid dataset

A few extras are needed Some idea of where to put the data

A “logical file name” – location-independent grid file name

EDG DataManagement Tutorial - n° 8

GGDGS (I) Cont’d

How to find out where to put data? Need to know which storage elements are out there

ldapsearch -h lxshare0225.cern.ch -p 2170 -x -b \"Mds-vo-name=local,o=grid" (objectclass=storageelement) \ seid

The command which will move your data to the desired location, and register it in the replica catalog, is edg-replica-manager-copyAndRegisterFile

edg-replica-manager-copyAndRegisterFile \ -s $(hostname)/$(pwd)/$DFILE -l $LFN -d $DEST_SE

See cr-mov-reg.{sh,jdl} examples

EDG DataManagement Tutorial - n° 9

Grid-generated data to Grid storage II

Your program generates data directly to a “close SE”

Close means you can use normal file IO to write it

You have to use a brokerinfo command to find out what the close SE is (you don’t know where your job will go!) and what the dir is

You write the data

Use the edg-replica-manager commands to Register the data as a Grid dataset

An extra is needed A “logical file name” – location-independent grid file name

EDG DataManagement Tutorial - n° 10

GGDGS II (cont’d)

Restriction: the “local file name” has to be the same as the logical file name (at least the “base” name)

File on disk: /data/spool/123fred7; LFNs: 123fred7 is OK 123fred is not OK fred7 is not OK Skippy is not OK spool/123fred7 is OK

Logical file name must not already be in catalogue

You also probably want to check that the file doesn’t exist on disk before you start to write it

Example files: cr-on-se-and-reg.{jdl,sh}

Check if it was successful: edg-replica-manager-listReplicas -c /opt/edg/etc/tutor/rc.conf \ -l whomp.119

EDG DataManagement Tutorial - n° 11

Submitting Data Along With Your Job

This is fairly easy: use the Input Sandbox

Careful – not a sandbox in the javascript sense

InputSandbox = {“input-ntuple.root"};

Example files: inp-sbox.{jdl,sh}

EDG DataManagement Tutorial - n° 12

Moving Data Onto Grid from Outside

This is almost identical to GGDGS I

Use edg-replica-manager-copyAndRegisterFile

Need to specify rc.conf file (either with RC_CONFIG_FILE variable or with –c option) … defaults in /opt/edg/etc/<vo>/rc.conf

Remember restrictions: LFN and remote file name have to match source and destination files must include hostnames

edg-replica-manager-copyAndRegisterFile –c rc.conf –l whomp.145 –s $(hostname)/$(pwd)/gls –d gppse05.gridpp.rl.ac.uk

EDG DataManagement Tutorial - n° 13

Having Grid Send Job to Your Data

Need to have data “on the Grid” == listed in RC

Tell your job (JDL) about the grid data: InputData = “LF:myfile.dat”

Resource Broker puts info about data matching in “brokerinfo” file on remote execution node

In your job execution script, use the “edg-brokerinfo” command (getselectedfile) to find location of job-local copy

Example files: find-data.{jdl,sh}

EDG DataManagement Tutorial - n° 14

Moving Data Around

Edg-replica-manager-replicateFile –c rc.conf –l <lfn> -d <dest-SE-name> -s <source-SE-name>

Try the previous test (w/ dg-job-list-match) – should find a new site willing to accept your job

EDG DataManagement Tutorial - n° 15

Finding Your Data

ldapsearch –LLL –h grid-vo.nikhef.nl –p 10389 –x –b “rc=EDGtutorialReplicaCatalog,dc=eu-datagrid,dc=org” ‘(filename=jtdmtest1)’ dn

Shows “dn”s wherever the selected “filename” exist

EDG DataManagement Tutorial - n° 16

GDMP

Tool for replication of large sets of files between sites

Can do a lot with it

Easy to get commands wrong Can’t recover from certain errors

Possible to wreck the GDMP subsystem badly enough that remote sysadmins will have to make manual fixes

Recommend not to use unless you really need it!

Ex: you don’t normally use the “dd” command to copy files!

EDG DataManagement Tutorial - n° 17

Gotchas

Edg-replica-manager commands Error messages not always on target

Careful not to use commands in ways other than intended – error trapping not good, and sometimes the command will do something but not necessarily what you want

Build error checking & trapping into your job scripts

Remember restrictions on LFN/PFN correspondence

Replica catalog Leaving out pieces of the command generally neither works

nor provides helpful messages – type carefully!

EDG DataManagement Tutorial - n° 18

EDG Replica Catalog

Based upon the Globus LDAP Replica Catalog

Stores LFN/PFN mappings and additional information (e.g. filesize):

Physical File Name (PFN): host + full path & and file name Logical File Name (LFN): logical name that may be resolved to PFNs LFN : PFN = 1 : n

Only files on storage elements may be registered

Each VO has a specific storage dir on an SE

Example PFN: lxshare0222.cern.ch/flatfiles/SE1/iteam/file1.dat host storage dir

LFN must be full path of file starting from storage dirLFN of above PFN: file1.dat

EDG DataManagement Tutorial - n° 19

globus-url-copy

Low level tool for secure copying

globus-url-copy <protocol>://<source file> \ <protocol>://<destination file>

Main Protocols: gsiftp – for secure transfer, only available on SE and CE

file – for accessing files stored on the local file system on e.g. UI, WN

globus-url-copy file://`pwd`/file1.dat \ gsiftp://lxshare0222.cern.ch/ \ flatfiles/SE1/EDGTutorial/file1.dat

EDG DataManagement Tutorial - n° 20

The Replica Manager APIs

(un)registerEntry(LogicalFileName lfn,

FileName source)

Replica Catalogue operations only - no file transfer

copyFile(FileName source,

FileName destination,

String protocol)

allows for third-party transfer

transfer between: two StorageElements or ComputingElement and Storage Element Space management policies under development

all tools support parallel streams for file transfers

EDG DataManagement Tutorial - n° 21

copyAndRegisterFile(LogicalFileName lfn,

FileName source,

FileName destination,

String protocol)

third-party transfer but :

files can only be registered in Replica Catalogue if destination PFN contains a valid SE (i.e. needs to be registered in the RC)!

replicateFile(LogicalFileName lfn,

FileName source,

FileName destination,

String protocol)

deleteFile(LogicalFileName lfn,

FileName source)

The Replica Manager APIs