iRODS hands-on tutorial

Christine StaigerSURFsara

Amsterdam, 8th May 2014

8th May 2014

The Basics

① Installation

② Tools for administration

③ Adding data and metadata

Complex storage systems

④ Federations between data grids

⑤ Resources and Compound resources

Contents

The Basics

Demo Data Grids10 Virtual machines: ssh user@IP

4

IP adress User Password

145.100.58.65 irods01 irods0104

145.100.57.78 irods2 irods204

145.100.57.82 irods3 irods304

145.100.57.83 irods4 irods404

145.100.58.67 irods5 irods504

145.100.57.86 irods6 irods604

145.100.57.87 irods7 irods704

145.100.57.91 irods8 irods804

145.100.57.107 irods9 irods904

145.100.57.110 irods10 irods1004

Installation

5

• Execute ./irodssetup in the folder iRODS

• Export the icommandsexport PATH=$PATH:/home/user/iRODS/clients/icommands/bin

• ~/iRODS/config/irods.config

# Database configuration$DATABASE_TYPE = 'postgres';…$DATABASE_HOST = 'localhost';$DATABASE_PORT = '5432';$DATABASE_ADMIN_PASSWORD = 'irods2';$DATABASE_ADMIN_NAME = ’alice';

# iRODS configurationIRODS_HOME = '/home/alice/iRODS';$IRODS_PORT = '1247’;…$IRODS_ADMIN_PASSWORD = ’alice';$IRODS_ICAT_HOST = '';

iRODS i-commands

Documentation: https://wiki.irods.org/index.php/icommands

(we don’t provide a exhaustive list of i-commands)

6

iRODS Control

• Execute ./irodsctl arg in the folder iRODS

• Restarting, starting and stopping the iRODS and iCAT server witharg = restart, start, stop

• Restarting, starting and stopping only the iRODS server witharg = irestart, istart, istop

• Initialise an account and log in with iinit

.irodsEnv file # iRODS personal configuration file.## This file was automatically created during iRODS installation.# Created Mon Jun 18 13:13:27 2012## iRODS server host name:irodsHost ’surfsara-cloud'# iRODS server port number:irodsPort 1247# Default storage resource name:irodsDefResource 'demoResc'# Home directory in iRODS:irodsHome '/alicesZone/home/alice'# Current directory in iRODS:irodsCwd '/alicesZone/home/alice'# Account name:irodsUserName ’alice'# Zone:irodsZone ‘alicesZone'

8

The .irodsEnvdetermines the zone and the user for the icommands-client.You can only have onesuch file at a time.

The icommands client• icommands are automatically installed when you install the iRODS

server

• Users will have to install them seperately to connect to an iRODS server

• Categories:• Informational• UNIX and FTP like• Metadata• Functional

9

ihelp

10

• ihelp Lists all commands

• ihelp -h Usage : ihelp [-ah] [icommand] Display i-commands synopsis or a particular i-command help text Options are: -h this help -a print the help text for all the i-commands

Run with no options to display a synopsis of the i-commands

iRODS Version 3.3.1 February 2014 ihelp

ienv• Lists the iRODS environment variables.

• Equivalent to “iinit -l”

• Usage : ienv [-h]

11

NOTICE: Release Version = rods3.3.1, API Version = dNOTICE: irodsHost=surfsara-cloudNOTICE: irodsPort=1247NOTICE: irodsDefResource=demoRescNOTICE: irodsHome=/alicesZone/home/aliceNOTICE: irodsCwd=/alicesZone/home/aliceNOTICE: irodsUserName=aliceNOTICE: irodsZone=alicesZone

iuserinfo• Show information about your iRODS user account or the entered

user

• Usage: iuserinfo [-vVh] [user]

12

name: aliceid: 10007type: rodsadminzone: alicesZoneinfo: comment: create time: 01396887567: 2014-04-07.18:19:27modify time: 01396887567: 2014-04-07.18:19:27member of group: alicemember of group: public

imiscsvrinfo

• Connect to the server and retrieve some basic server information

• Can be used as a simple test for connecting to the server.

• Usage: imiscsrvinfo [-hvV]

13

RCAT_ENABLEDrelVersion=rods3.3.1apiVersion=drodsZone=alicesZoneup 0 days, 5:54

ilsresc• ilsresc lists iRODS resources and resource-groups

• Usage: ilsresc [-lvVhA] [Name]

14

alice@sara-cloud:~$ ilsrescdemoResc

alice@sara-cloud:~$ ilsresc –l demoRescresource name: demoResc resc id: 10010 zone: alicesZonetype: unix file system class: archive location: surfsara-cloud vault: /home/alice/iRODS/Vault …

Some commands• Unix-like:

ils, ipwd, icd, imkdir, irm, icp ichmod, ipasswd, irsync, ichksum, imv

• FTP-like:iinit, iput, iget, iexit

• Test: Create folder in the logical name space ipwd

imkdir NewFolderils NewFolderirm –r NewFolder

15

icommands - ACLs

16

List ACLs (Access Control Lists) in a collection:

alice@sara-cloud:~$ ils -A/alice/home/alice: ACL - alice#alicesZone:own Inheritance - Disabled

Administration

17

• Change to admin mode: iadmin• Quit admin mode: q

• Make/delete a user: mkuser, rmuser, moduser (iadmin) mkuser user1 rodsuser (iadmin) moduser user1 password *****

• Parameters and their values: iadmin lt, iadmin lt user_type

• iadmin mkresc, iadmin rmresc

• Create new resource: mkdir Data (in your home, not in the iRODS environment) iadmin mkresc NewDataResc “unix file system” cache 145.100.57.139 “/home/alice/Data”

Put and replicate data

18

• Add the file /home/login/put1.txt to resource iput –v put1.txt iput -h

• ichksum put1.txt (check sum of copy in Vault) md5sum put1.txt (check sum of file in /home/login)

• iput -R NewDataResc put2.txt

• Replicate data: irepl -R NewDataResc put1.txt

• Trim number of copies: itrim -S demoResc -N 1 put1.txt

Access Control Lists

19

• Change permission for a user: ichmod read user1 put1.txt List access: ils -A

• Change permissions for group “public”: Make a directory in the iRODS environment: imkdir temp Make directory readable for “public”: ichmod -r read public temp

• ils –A /bobsZone/home/bob/temp: ACL - bob#bobsZone:own g:public#bobsZone:read object Inheritance - Disabled

iRODS Metadata• Commands: imeta, iquest, idbo

• Add metadata: imeta add -d put1.txt “Date” “08.10.2013” imeta add -d put1.txt “AnotherDate” “04.04.2014” imeta ls -d put1.txt

• Remove metadata: imeta rm -d put1.txt “Date” “08.10.2013”

• Querying the iCAT: iquest "SELECT DATA_NAME,DATA_CHECKSUM WHERE

DATA_RESC_NAME like ’%Resc%’” DATA_NAME = put1.txt DATA_CHECKSUM = c16d7d3488677a3348b12eb82795d28c

20

Complex storage systems

Federation between data grids• List all existing zones: ils /• Zone A acknowledges Zone B and vice-cersa

iadmin mkzone B remote Host:Port• Zone A adds remote user from Zone B: iadmin mkuser user#B rodsuser

• Example: Alice and Bob:Alice acknowledges Bob’s zone:

iadmin mkzone bobsZone remote 145.100.57.74:1247 Bob acknowledges Alice’s zone:

iadmin mkzone alicesZone remote 145.100.57.66:1247 Alice creates remote user Bob:

iadmin mkuser bob#bobsZone rodsuser Now Bob can list and alter files in Alice’s zone: ils /alicesZone

22

Trouble shooting

ERROR: putUtil: put error for /alicesZone/home/bob#bobsZone/put1.txt, status = -9000 status = -9000 SYS EXCEED CONNECT CNT

Host configuration problem on the server.Solution: Edit iRODS/server/config/irodsHost

localhost you.rIP.XX.XX surfsara-cloudlocalhost 12.0.0.1

./irodsctl restart

23

Data Replication between grids• Bob copies a file from his zone to his account in Alice’s zone irsync –R demoResc i:/bobsZone/home/bob/put1.txt i:/alicesZone/home/bob#bobsZone/put1.txt -R specifies the target resource

• Bob copies a frile from his home directory to Alice’s zone irsync –R demoResc put3.txt i:/alicesZone/home/bob#bobsZone/put3.txt

• Bob copies a file from Alice’s zone to his home directory mkdir test irsync i:/alicesZone/home/bob#bobsZone/put3.txt test/put3.txt

• Bob deletes a file in Alice’s zone irm /alicesZone/home/bob#bobsZone/put3.txt

24

Remote ResourcesInstall iRODS, iCAT disabled

Build an iRODS server [yes]? Make this Server ICAT-Enabled [yes]? no

Host running iCAT-enabled iRODS server? 145.100.58.11…Resource name [demoResc2]? myResc2

Resource storage area directory [/home/irodsadmin/iRODS/Vault]? /home/irodsadmin/iRODS/alicesVault

…The irodsServer will authenticate to the other servers viaan existing iRODS administrator account.Existing iRODS admin login name [rods]? alice

Password [rods]? ********

Enter the zone as defined in your ICATiRODS zone name [tempZone]? alicesZone

Resource Groups

Exercise Resource Groups

• Create two data resources of type cache• Group them: iadmin atrg groupName resourceName• Deposit data in resource group: iput file groupName • Add some data to the resource group• Check with ils where the file is located virtually and physically

• How to automatically replicate data to resources in the same resource group?

iRODS Microservices

Micro-services are small, well-defined procedures/functions that perform a simple task (https://www.irods.org/index.php/iRODS_Micro-services)

Examples: “msiCollCreate”, “msiDataObjRepl”, “msiDataObjChksum”, …

To convert any C procedure (createCollection) into a microservice create an interface routine (msiCreateCollection)

27

iRODS Rules• Example: replicate.r

replicate{ msiWriteRodsLog("starting replication", *status); msiWriteRodsLog("source = *source", *status); msiDataObjRsync(*source, "IRODS_TO_IRODS", "null", *destination, *rsyncStatus); } INPUT *source="/bobsZone/home/bob/put3.txt", *destination="/alicesZone/home/bob#bobsZone/put3.txt" OUTPUT ruleExecOut

Execute irule -vF ~/rules/replicate.r

Check with ils /alicesZone/home/bob#bobsZone

28

Time delayed and periodic jobs• Example test.r

myTestRule{ delay("<PLUSET>60s</PLUSET>"){ msiWriteRodsLog("Hello World.", *status); } } INPUT *A="1" OUTPUT ruleExecOut

• Writes the output to iRODS/server/log/reLog*, also check rodsLog*• Use iqstat, iqdel and iqmod to list, delete and modify delayed rules. • delay(”<PLUSET>5m</PLUSET>EF>1h</EF>”) Start job in 5 minutes and restart it every 1 hour

• Predefined rules: iRODS/server/config/reConfigs/core.re

29

Compound resources

• List resource types: iadmin lt resc_type• A compound resource consists of cache and a

storage resource (s3, MSS universal driver)• Storage resource can be located on several different severs

• Example: Universal Mass StorageUser writes data to cache, system replicates data to storage and deletes it from cache

admin mkresc Cache "unix file system" cache 145.100.57.197

/home/alice/Cache iadmin mkresc mssStore 'MSS universal driver' compound 145.100.57.197

/home/alice/MSS iadmin atrg mssGroup Cache iadmin atrg mssGroup mssStore

Compound resources

• The mssStore can be of any hardware type which might require certain protocols to put and get data, e.g. gridFTP

• The resource type 'MSS universal driver’ triggers iRODS to look for a file iRODS/server/bin/cmd/univMSSInterface.sh to overwrite the put and get functions

• Template in iRODS/server/bin/cmd/examples

• Example file in TODO

The Universal Mass Storage Interface

• Logging: _log() LOGFILE=/home/irodsadmin/univMSSInterface.log Change the path

• Copy between cache and storage: syncToArch(), stageToCache()

_log 2 syncToArch "entering syncToArch()=$*" _log 2 syncToArch "executing: /bin/cp -f \"$1\" \"$2\""

/bin/cp -f "$1" "$2” error=$? if [ $error != 0 ] # copy failure then STATUS="FAILURE" fi _log 2 syncToArch "The status is $error ($STATUS)" return $error

Replicate data

replicateDiskCache { delay("<PLUSET>30s</PLUSET><EF>15m</EF>") {

…*Condition = "DATA_RESC_NAME = '*CacheRescName' AND COLL_NAME

like '*Collection%'"; msiMakeGenQuery("DATA_NAME, COLL_NAME, DATA_SIZE,

order(DATA_CREATE_TIME)",*Condition,*Query2); msiExecGenQuery(*Query2,*List);

while ( *ContInxOld > 0 ) { foreach ( *List ) { msiGetValByKey(*List,"DATA_NAME",*D); msiGetValByKey(*List,"COLL_NAME",*C);

msiDataObjRepl("*C/*D","backupRescName=*PnfsRescName++++rescName=*CacheRescName++++irodsAdmin=",*status)}

} }}

Replicate data

• Save as YourFilename.re in iRODS/server/config/reConfigs/ Edit server/config/server.config

reRuleSet core,YourFilename

• irule replicateDiskCache "*Collection=/yourZone%*PnfsRescName=yourMssStore%*CacheRescName=yourCache" ruleExecOut

• alice@surfsara-cloud:~/iRODS$ iput -R mssGroup ../put1.txt alice@surfsara-cloud:~/iRODS$ ils -l

/alicesZone/home/alice: alice 0 Cache 7 2014-04-24.09:26 & put1.txt

• Check with iqstat -l when the rule will be executed

Some time later:• alice@surfsara-cloud:~/iRODS$ ils -l

/alicesZone/home/alice: alice 0 Cache 7 2014-04-24.09:26 & put1.txt alice 1 mssStore 7 2014-04-24.09:36 & put1.txt

No space!

Purge Cache

Write a rule to purge the cache

• Purge cache when data is replicated and if not enough space left in cache

• Call by: irule purgeDiskCache "*Collection=/testZone%*PnfsRescName=mssStore%*CacheRescName=Cache%*MaxSpAlwdTBstring=2" ruleExecOut

• MaxSpAlwdTBstring: the size when to purge, string

Thank you !