AliEn Tutorial MODEL 2009 4 th May, 2009. 2 [email protected] May 2009 Installation of the AliEn...
-
Upload
verity-blair -
Category
Documents
-
view
213 -
download
0
Transcript of AliEn Tutorial MODEL 2009 4 th May, 2009. 2 [email protected] May 2009 Installation of the AliEn...
AliEn Tutorial
MODEL 2009
4th May, 2009
2 [email protected] May 2009
Installation of the AliEn software
AliEn and the GRID
Authentication
File Catalogue
Submitting jobs
Outline
3 [email protected] May 2009
Installation
Please, do:wget http://alien.cern.ch/alien-torrent-installerchmod +x alien-torrent-installer./alien-torrent-installer
Default installationFirst two commands fastLet’s leave the 3rd running...
4 [email protected] May 2009
AliEn, AliEn2
ALICE ENvironment on the GRIDGRID middleware
Developed as single entry point to the GRID for ALICE (CERN)
All the components necessary to build a GRID and interact with other GRIDs
File System with metadataAuthorization, authentication, job optimization and execution, storage managementAudit, quotas, monitoringInterfaces to various GRID implementations
Used since:2002 for centrally managed productions2006 for user analysis
Used by several Virtual Organizations:ALICE, PANDA, CBM, GPCALMA
http://alien.cern.ch
5 [email protected] May 2009Pablo Saiz 5
CERN
6 [email protected] May 2009Pablo Saiz 6
LHC
27km particle accelerator100 meters underground
About to start collecting data
7 [email protected] May 2009Pablo Saiz 7
A Large Ion Collider Experiment
8 [email protected] May 2009
Why do we need a catalogue?
Loads of data (several PB per year!!)Distributed community
Same structure from everywhere
Contains location of the fileIt does not contain the file
Possibility of several copies of filesSelect the closest one
Metadata informationWe will see this later on...
9 [email protected] May 2009
AliEn2 File Catalogue
Mapping from LFN to SE and PFNUNIX-like file systemMetadataGUIDUsed by
all other services
Tier1
|--./| |--cern.ch/| | |--user/| | | |--a/| | | | |--admin/| | | | || | | | |--aliprod/| | | || | | |--f/| | | | |--fca/| | | || | | |--p/| | | | |--psaiz/| | | | | |--as/| | | | | || | | | | |--dos/| | | | | || | | | | |--local/
|--simulation/| |--2001-01/| | |--V3.05/| | | |--Config.C| | | |--grun.C
| |--36/| | |--stderr| | |--stdin| | |--stdout| || |--37/| | |--stderr| | |--stdin| | |--stdout| || |--38/| | |--stderr| | |--stdin| | |--stdout
| | | || | | |--b/| | | | |--barbera/ALICE
USERS
ALICE DATA
ALICE REMOTE
10 [email protected] May 2009
LFN and GUID catalogues
/
/alice
/alice/user/p/psaiz
/alice/simulation/2006
…
Index
1-JAN-1970
1-JAN-2006
14-FEB-2007
23-AUG-2008
…
IndexGUID PFN
LFN Catalogue GUID Catalogue
AliEn File & Metadata Catalogue
LFN GUID
LFN: Logical File Name (what the user sees)GUID: Global Unique Identifier (another way of identifying files)PFN: Physical File Name
11 [email protected] May 2009
Who uses the catalogue?
EverybodyIF A FILE IS NOT IN THE CATALOGUE,
IT DOES NOT EXIST ON THE GRID
UsersUser data, user executables ...
Job executionExecutable, input, output
PackagesSource code, binary code, post
installation
12 [email protected] May 2009
Back to the exercises...
Did the installation finish?
Let’s get ready to access the GRIDX509 Certificate authentication
Public vs Private keyCertificate AuthorityProxy certificates
13 [email protected] May 2009
Preparing certificates
In $HOME/.globus:userkey.pemusercert.pem
Readable only by the user (chmod 400)Make alien use those files:
mkdir $HOME/.alienln –s $HOME/.globus $HOME/.alien/globus
14 [email protected] May 2009
Environment files
$ALIEN_ROOT/.Environment$HOME/.alien/Environment
Bash files sourced before starting aliencat <<EOF > $HOME/.alien/Environment
export ALIEN_ORGANISATION=PANDA export ALIEN_USER=<username> EOF
Put alien in the PATHmkdir –p $HOME/binln –s $HOME/alien/bin/alien $HOME/bin/
16 [email protected] May 2009
Globus related:Permissions on $HOME/.globus/userkey.pem are not private to the user – chmod 400 userkey.pemYour certificate authority is exotic and not known to the server.Your certificate has expired.Clock skew:
• Your local computer time is in the future with respect to the server's time.
• Your local computer time is more in the past than the certificate life time.
Authentication problems
17 [email protected] May 2009
Check the certificates in your machines:Store them under e.g: $HOME/.globus/Check that the key file is read only
Setup the environment$HOME/.alien/EnvironmentPut alien in the path
Get a valid alien tokenalien proxy-init
Try it out !!
18 [email protected] May 2009
Standard bash shell with grid comandsMain bash features are available.Not all shell helper programs are available.Some local commands (like ls, cat etc) are overwritten with the corresponding GRID commands.File/path tab completion in the virtual GRID directory.
Accessing the shell
20 [email protected] May 2009
Basic commands II
22 [email protected] May 2009
Viewing the files
23 [email protected] May 2009
Shell – Copying files (add/get)
24 [email protected] May 2009
Access the alien shell.
alien
Check your user name:
whoami
List the contents of your home directory
ls
Print the working directory
pwd
Create the following directory structure in your AliEn space:
$HOME/bin (if it is not there)
$HOME/macros and $HOME/jdl
Get the information of the file:
whereis /bin/date
Shell – Try it out I
25 [email protected] May 2009
See the file /bin/datecat /bin/date
Copy the file to your local machinecp /bin/date /tmp/my_file
Exit the shell and check the file
Shell – Try it out II
26 [email protected] May 2009
Job & Transfer models
Jobs
TaskQueue Job Broker
Job Manager
CE
JA
Central servicesSite services
SplittingExpired
PrioritiesMerging
Zombies
Job optimizers
File catalogue
LFN GUIDMetadata
FTD
PackmanMonALISA
JA
JA
CE
SEPackman
MonALISA
FTD
SE
Site A
Site B
Transfer Manager TransferQueue Transfer
Broker
Transfer optimizers
fts
bbftp
xrdcp
29 [email protected] May 2009
AliEn
AliEn user interface
VTD EDGLCG /GLITECONDORARC/
NORDUGRID
Nice! I STILL do not have to
worry about ever changing GRID environment…
Interconnecting GRIDs
30 [email protected] May 2009
Job status
31 [email protected] May 2009
Your job is submitted and a <JOBID>.is assigned to it.
Submitting jobs
32 [email protected] May 2009
Checking the job status
33 [email protected] May 2009
Getting the output
34 [email protected] May 2009
Other useful commands
jobListMatchList of CE that can execute the job
SpyOutput of the job while it is running
MasterjobGive information about
35 [email protected] May 2009
Connect to alien
alien login
Submit the jobsubmit <<EOF
Executable=“date”;
EOF
Follow the execution
top –id, ps trace, ps jdl
Get the output
cp /proc/<user>/<id>/job-output/stdout /tmp/out
Try it out!!
36 [email protected] May 2009
Executable: Compulsory field where we give the lfn of the executable that should be stored in /bin or $V0/bin or $HOME/bin.Arguments: They will be passed to the executable.Packages: Type packages in the shell to see what kind of packages are installed.InputFile: The files that will be transported to the node where the job will run.InputData: It will require that the job will be executed in a site close to the files specified here.InputDataList: The filename in which the Job Agent will write the InputData list.InputDataListFormat: The format of the InputData list.
JDL fields I
37 [email protected] May 2009
OutputFile: The files that will be registered in the catalog once the job finishes.
OutputArchive: What files will be archived in a zip file.Validationcommand: Specifies the script to be used as a validation script.Email: Receive a mail when the job finishes.TTL: The maximum run time of your job.Split: Split the jobs in several sub jobs.
JDL fields II
38 [email protected] May 2009
Packages
packmangives a list of the packages defined/installedinstalls packages
39 [email protected] May 2009
Add:Packages=“pbarprod@pandaroot::nov08“;
Try other commands:Executable=“pandaroot”;
Multiple submissionSplit=“production:1-5”;
Create your own executables
Try it out !!
40 [email protected] May 2009
Conclusion
AliEnInterface to the GRIDDeveloped by ALICEAlso used by PANDA, CBM, GPCALMAIn production since 2002
File CatalogueFile and Metadata
Job submission modelData movement and replication
41 [email protected] May 2009
Registration – Certificates:
http://alien.cern.ch/twiki/bin/view/Alice/UserRegistration
https://ca.cern.ch/ca/
AliEn:
http://alien.cern.ch
Gshell:
http://alien.cern.ch/twiki/bin/view/AliEn/GAPI
User's guide:
http://project-arda-dev.web.cern.ch/project-arda-dev/alice/apise
rvice/AA-UserGuide-0.0m.pdf
References I
42 [email protected] May 2009
aliensh Grid Command Online Reference V1.0 http://project-arda-dev.web.cern.ch/project-arda-dev/alice/apiservice/guide/guide-1.0.htm
Previous tutorials:
http://aliceinfo.cern.ch/Offline/Analysis/Tutorial/
Event Tag System:
http://pcaliweb02.cern.ch/Offline/Analysis/RunEventTagS
ystem/EventTags.html#Event%20tag%20system
https://edms.cern.ch/document/788315/1 (INTERNAL
NOTE)
References II