SPI Report for the LHCC Comprehensive Review
description
Transcript of SPI Report for the LHCC Comprehensive Review
![Page 1: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/1.jpg)
SPI Report for the
LHCC Comprehensive Review
Stefan Roiser
for the SPI project
![Page 2: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/2.jpg)
19. Nov. 2007 LHCC Comprehensive Review - SPI 2
SPI services
• Software process support– Software testing, configuration & release
• Documentation support– Code documentation, web services
• Collaboration tools– Hypernews, Savannah
• Hardware infrastructure– Build servers, special purpose machines
![Page 3: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/3.jpg)
Software
![Page 4: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/4.jpg)
19. Nov. 2007 LHCC Comprehensive Review - SPI 4
Time to user
• The time from a bug report / feature request until it is used by the physicists in the experiments– Especially important to minimize this time when
running with “live” data.
![Page 5: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/5.jpg)
19. Nov. 2007 LHCC Comprehensive Review - SPI 5
Nightly Builds
• Recommendation of the 2006 internal review• Motivation
– Early integration into experiment software– Fast feedback to LCG/AA developers of changes
• Usage– Building with own experiment nightlies on top– Developers testing their code on LCG/AA stack
• Used by– LHCb– Atlas
![Page 6: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/6.jpg)
19. Nov. 2007 LHCC Comprehensive Review - SPI 6
Nightly Builds (ctd.)
5 configurations
LCG/AA projects
13 possible platforms
Builds & Tests
test evolution
![Page 7: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/7.jpg)
19. Nov. 2007 LHCC Comprehensive Review - SPI 7
Operating System
eg. Athena, CMSSW, Gaudi
Linux (slc) 32/64 bit, Mac OSX, Windows
External Software
LCG / AA software
Experiment software
Operating System
eg. Athena, CMSSW, Gaudi
Linux (slc) 32/64 bit, Mac OSX, Windows
External Software
LCG / AA software
Experiment software
Software release
56 packages, eg. Boost, Python, Qt, etc…
COOL, CORAL, POOL, RELAX, ROOT, SEAL
LCG release
• rubberstamps all package versions• provides several products
• afs installation• web pages• src tar balls• bin tar balls• documentation
![Page 8: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/8.jpg)
19. Nov. 2007 LHCC Comprehensive Review - SPI 8
Software processes
mysql
externs/w
scramcmt install.mgr.
cron /pythontar
ballsxml /
python
python
nightlybuilds
projects/w
websites docu
xml
historicallygrown
![Page 9: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/9.jpg)
19. Nov. 2007 LHCC Comprehensive Review - SPI 9
Software processes new
externs/w
install.mgr.
tarballs
nightlybuilds
projects/w
websites docu
Python /LCGCMT
![Page 10: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/10.jpg)
19. Nov. 2007 LHCC Comprehensive Review - SPI 10
LCGCMT
• Is a CMT based project • Contains the configuration information
– How to build software packages from source• Using sh/bat scripts
– Extra information • Eg. homepage, package maintainer, description, etc.
– In addition for other CMT based projects• How to use/build against the packages• Runtime environment
![Page 11: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/11.jpg)
19. Nov. 2007 LHCC Comprehensive Review - SPI 11
Example: package build
package gsl
use LCG_BuildPolicy v* LCG_Builders
macro gsl_build_config_version "$(GSL_config_version)"set LCG_tarfilename "gsl-$(gsl_build_config_version).tar.gz"
apply_pattern buildscripts_local
#!/bin/sh
cd ${LCG_builddir}tar xvfz ${LCG_tarfilename}cd gsl-${LCG_package_config_version}./configure --prefix=${LCG_extdir}/GSL/${LCG_package_config_version} /${LCG_CMTCONFIG}
#!/bin/sh
cd ${LCG_builddir}/gsl-${LCG_package_config_version}make
#!/bin/sh
cd ${LCG_builddir}/gsl-${LCG_package_config_version}make install
lxplus:~ >setenv CMTCONFIG slc4_amd64_gcc34lxplus:~/LCGCMT/LCG_Builders/gsl/cmt >cmt pkg_getlxplus:~/LCGCMT/LCG_Builders/gsl/cmt >cmt pkg_configlxplus:~/LCGCMT/LCG_Builders/gsl/cmt >cmt pkg_makelxplus:~/LCGCMT/LCG_Builders/gsl/cmt >cmt pkg_install
![Page 12: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/12.jpg)
19. Nov. 2007 LHCC Comprehensive Review - SPI 12
Benefits of new s/w process
• Avoid duplication of information• Easier maintenance / automation• All parties use the same info (nightly builds,
release, users rebuilding the software)• Speedup time to user
Current configuration
LCG 53f has been
built with these tools
![Page 13: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/13.jpg)
Communication
![Page 14: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/14.jpg)
19. Nov. 2007 LHCC Comprehensive Review - SPI 14
Hypernews
• Bulletin board system used by LHC experiments
• Moved to new IT provided hardware– 1 production machine, 1 hot standby
• Helping with instance maintenance – Managing user DB, setup of for a
• Atlas and CMS heavy users– Also used in Alice and Totem– LHCb is rethinking their usage
![Page 15: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/15.jpg)
19. Nov. 2007 LHCC Comprehensive Review - SPI 15
Savannah
• Savannah is the main bug reporting system in LCG/AA
• Moved to new IT provided hardware• Used by many other parties (IT, Grid, etc.)• Reasonable system / user maintenance
0
50
100
150
200
250
aug'02feb'03aug'03feb'04aug'04feb'05aug'05feb'06aug'06feb'07aug'07
Active Projects
0
500
1000
1500
2000
2500
3000
aug'02feb'03aug'03feb'04aug'04feb'05aug'05feb'06aug'06feb'07aug'07
Registered Users
0
50
100
150
200
(days)
aug'02feb'03aug'03feb'04aug'04feb'05aug'05feb'06aug'06feb'07aug'07
Bug Average Lifetime
05000
1000015000200002500030000
aug'02feb'03aug'03feb'04aug'04feb'05aug'05feb'06aug'06feb'07aug'07
Submitted Bugs
0
500
1000
1500
2000
2500
aug'02feb'03aug'03feb'04aug'04feb'05aug'05feb'06aug'06feb'07aug'07
Submitted Support requests
![Page 16: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/16.jpg)
Hardware
![Page 17: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/17.jpg)
19. Nov. 2007 LHCC Comprehensive Review - SPI 17
Hardware
• LCG / AA build servers– 4 linux machines (managed by IT/FIO)– 2 mac osx machines (one ppc)– 2 windows machines
• Special purpose machines– Web / database servers
• Recycling old Hypernews machines (located in CC)
– New machines for• Savannah, Hypernews
![Page 18: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/18.jpg)
Staff & Program of Work
![Page 19: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/19.jpg)
19. Nov. 2007 LHCC Comprehensive Review - SPI 19
Staff
• Change of project leader– Andreas Pfeiffer left to CMS– Stefan Roiser took over
• 1 staff left, 1 fellow joined– Manuel Gallas left for Atlas– Piotr Kolet joined
• 1 summer student– Luana Micallef
• Currently 3.2 FTE
![Page 20: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/20.jpg)
19. Nov. 2007 LHCC Comprehensive Review - SPI 20
Program of work
• Completion of the integrated software configuration / testing / release – Web pages derived from LCGCMT– Integrate remaining release steps
• Consistency checking• Production of distribution kits• Documentation generation
– Completion of integration with nightly builds– Integration of installation manager
![Page 21: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/21.jpg)
19. Nov. 2007 LHCC Comprehensive Review - SPI 21
Program of work (ctd.)
• Nightly build system– Integrate Geant4 builds
• Using IT infrastructure for– Setup of common LCG/AA web services– Setup of database services– Virtual windows machines for s/w build
![Page 22: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/22.jpg)
19. Nov. 2007 LHCC Comprehensive Review - SPI 22
Program of work (ctd.)
• Hypernews– Investigate usage of IT provided SSO– Find solution for longterm support
![Page 23: SPI Report for the LHCC Comprehensive Review](https://reader034.fdocuments.in/reader034/viewer/2022051517/56815065550346895dbe6373/html5/thumbnails/23.jpg)
19. Nov. 2007 LHCC Comprehensive Review - SPI 23
Conclusion
• Current focus on LCG/AA s/w stack– Optimize release process– Facilitate and unify processes for
• Nightly builds• Release procedure• User rebuilds
• Outsourcing of maintenance for other services, but still a lot of SPI manpower involved– Savannah– Hypernews– Hardware