Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers
-
Upload
delphia-thornton -
Category
Documents
-
view
216 -
download
0
Transcript of Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers
![Page 1: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/1.jpg)
Computing at CERN - II
Summer Student Lectures 2002Jamie Shiers
http://cern.ch/jamie/
![Page 2: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/2.jpg)
Lecture II
• Computing at CERN Today
Software at CERN Today
• The future & LHC Computing
![Page 3: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/3.jpg)
Introduction• For a long time it puzzled me how
something so expensive, so leading edge, could be so useless…
• and then it occurred to me that a computer is a stupid machine with the ability to do incredibly smart things, …
• while computer programmers are smart people with the ability to do incredibly stupid things.
• They are, in short, a perfect match.Bill Bryson: "Notes from a Big Country".
![Page 4: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/4.jpg)
Homework
Review of homework from lecture I
![Page 5: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/5.jpg)
Exercise I
• Implement a Unix utility (grep, cron, …) according to man specification
• You don’t actually need to do the exercise – just pretend you have!
![Page 6: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/6.jpg)
Software
Producing high-quality software is:
• Far from easy
• Far from cheap
• Still not a solved problem
![Page 7: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/7.jpg)
Anyone can program?
• “Everyone can be taught to sculpt: Michelangelo would have had to be taught not to. So it is with great programmers.”
![Page 8: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/8.jpg)
Overview
• Software Engineering
• Software Process
• Real examples from CERN
![Page 9: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/9.jpg)
Disclaimer
• CERN and its collaborators have produced a vast quantity of high-quality, well documented software
• Well disciplined approaches are in use in many areas of CERN
• Many people have devoted significant effort to improve the overall software process at CERN
![Page 10: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/10.jpg)
Some Large Producers…
• Microsoft
• Oracle
![Page 11: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/11.jpg)
Empower people through great software
![Page 12: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/12.jpg)
"I sense much NT in you! NT leads to Blue Screen. Blue Screen leads to downtime, downtime leads to suffering... NT is the path to the darkside!"
![Page 13: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/13.jpg)
![Page 14: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/14.jpg)
Why software quality?
• Airbus / BMW
• LHC data acquisition & processing
![Page 15: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/15.jpg)
level 1 - special hardware
100 MHz (1000 TB/sec)level 2 - embedded processorslevel 3 - PCs
75 KHz (75 GB/sec)5 KHz (5 GB/sec)100 Hz
(100 MB/sec)DB
![Page 16: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/16.jpg)
Software Engineering
• When discussing salary, its a profession;• When discussing , Bugs, Errors and Liability,
its a job; • When discussing theory, its science;• When discussing methods and practice, its
engineering; • When discussing the work and the work of
others, its a craft; • When managing it, its an art.
![Page 17: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/17.jpg)
![Page 18: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/18.jpg)
Software as a Craft
• Software as a University
• Software as Hollywood
• Software as Construction
![Page 19: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/19.jpg)
Software Engineering in HEP – The Reality
• Jürgen Knobloch, Computing in High Energy Physics, Tsukuba 1991
• “In spite of all efforts, the most valuable tool is still a good Symbolic Debugger…”
![Page 20: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/20.jpg)
Software Complexity
• Program complexity grows until it exceeds the capability of the programmer to maintain it.
• There are two ways of constructing a software design: one way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.
![Page 21: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/21.jpg)
Data and Computation for Physics Analysis
batchphysicsanalysis
batchphysicsanalysis
detector
event summary data
rawdata
eventreconstruction
eventreconstruction
eventsimulation
eventsimulation
interactivephysicsanalysis
analysis objects(extracted by physics topic)
event filter(selection &
reconstruction)
event filter(selection &
reconstruction)
processeddata
![Page 22: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/22.jpg)
04/21/23
RAW
ESD
AOD
TAG
randomseq.
1PB/yr (1PB/s prior to reduction!)
100TB/yr
10TB/yr
1TB/yr
Data
Users
Tier0
Tier1
![Page 23: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/23.jpg)
Size of CERN Software
![Page 24: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/24.jpg)
CMS Offline Software
R .W ilk in s o nD e p u ty
O b je c t P e r s is ta n c y
L .S ilv e s tr isD e p u ty
T e s tb e a m s
V .I n n o c e n teA r c h ite c t
V .I n n o c e n teF r a m e w o r k
V .I n n o c e n teU tilit ie s
T .C o xM u o n e n d c a p
A .V ite lliM u o n b a r r e l
F .B e h n e rC a lo r im e tr y
L .T a y lo rV is u a liz a t io n
E .M e s c h iT r ig g e r
L .S ilv e s tr isT e s tb e a m
C .S e e zE le c tr o n P G
U .G a s p a r in iM u o n P G
T .T o d o r o vT r a c k e r
S .W y n h o ffG e n e r a to r s
A .C a n e rb /ta u P G
S .E n oJ e t P G
S .W y n h o ffE x a m p le s
T .T o d o r o vC o m m o n D e t
S u b -S y s te m C o o r d in a to r s
C .W illia m sL ib r a r ia n
J -P .W e llis c hS W P r o c e s s M a n a g e r
D .S t ic k la n dP r o je c t M a n a g e r
![Page 25: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/25.jpg)
Software Cop-Outs• That's a feature, not a bug.• If there are no questions, everyone must be
happy.• If there are no bug reports then noone is using it.• We've lost the source code.• We're too busy to document that.• It must be a hardware problem.• That could never fail -- don't bother testing for it.• It's fixed, but is waiting for the next release cycle.
![Page 26: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/26.jpg)
Software Cop-Outs• That's a feature, not a bug.• If there are no questions, everyone must be
happy.• If there are no bug reports then noone is using it.• We've lost the source code.• We're too busy to document that.• It must be a hardware problem.• That could never fail -- don't bother testing for it.• It's fixed, but is waiting for the next release cycle.
![Page 27: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/27.jpg)
![Page 28: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/28.jpg)
The Software Process
• “The software process is the set of tools, methods and practices that are used to produce a software product.”
Watts S. Humphrey, Managing the Software Process
![Page 29: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/29.jpg)
Capability Maturity Model
![Page 30: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/30.jpg)
• The mortal struggle of great beasts in the tar pits…
Software DevelopmentThe Mythical Man-Month
Frederic Brooks
![Page 31: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/31.jpg)
Software Scheduling
• 1/3 planning• 1/6 coding• 1/4 component testing• 1/4 full system testingFrom “The Mythical Man Month”
The real cost is in maintenance & support!
![Page 32: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/32.jpg)
The World’s First Programmer
![Page 33: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/33.jpg)
Grace Hopper
• Invented world’s first compiler – A0
• Discovered world’s first computer bug
![Page 34: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/34.jpg)
Example I
The CERN Program Library:CERNLIB
![Page 35: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/35.jpg)
CERNLIB
• Arguably CERN’s most famous “product” prior to the Web– And it included CERN in the name…
• Written over nearly 40 years by at least as many authors– Try calculating the cost! €100M or more!
• Mainly Fortran, but some assembler, Pascal, C, …• Used by virtually all HEP experiments world-wide,
including those at the LHC!• No defined software process
– But steps in that direction…
![Page 36: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/36.jpg)
CERNLIB: What is it?
• Libraries (initially) and packages aimed at scientific computing– Histogramming, fitting, mathematical
routines, graphics, analysis, detector simulation, event generators …
• “Tool kit” for physics software applications
![Page 37: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/37.jpg)
CERNLIB cont.• CERN Program Librarian – many incarnations• Source code management: now CVS + cpp;
previously home-grown cpp-equivalent• Code conventions: must compile• Build procedures: moved to make in 1990s• Release procedures: old, pro & new areas
– User testing of new area for weeks prior to release
![Page 38: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/38.jpg)
Hall of fame: Make
• Introduced to the world with Unix– Along with SCCS – “forerunner” of
CVS
• Significant impact on software build process
![Page 39: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/39.jpg)
Example II
AIS Applications
![Page 40: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/40.jpg)
AIS Applications
![Page 41: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/41.jpg)
Paper Purchase Order
![Page 42: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/42.jpg)
Web Purchase Order
![Page 43: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/43.jpg)
Web Leave RequestUser’s viewManager’s View
![Page 44: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/44.jpg)
Standards and Inspections
• All Code must conform to coding standards– Informal Code Inspections
• With follow-up
http://edh.cern.ch/CodingStandards
For more information see:
![Page 45: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/45.jpg)
Example III
LCG ApplicationsAnaphe; Geant-4; ROOT; CMS,
…http://wenaus.home.cern.ch/wenaus/peb-app/
![Page 46: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/46.jpg)
LCG Applications
• Anaphe: a C++ replacement for CERNLIB
• GEANT4: a C++ detector simulation programme
![Page 47: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/47.jpg)
Anaphe releases• Do not use CERNLIB-style old / pro / new
• Limited flexibility (esp. with shared libs)• Bad “recognisability” : “pro” in outside institutes may
differ from “pro” at CERN (and even within institutes) use version numbers
• Version numbers for each package• Component based architecture allows for (semi-)
independent development• Version numbers in library names
• lib<pkg>.<vers>.so (link: lib<pkg>.so -> lib<pkg>.<vers>.so)
• Coherent set of versioned packages as “release”
![Page 48: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/48.jpg)
Geant4 releases• Major releases
– include major changes and updates, including public interface changes. May require porting of users’ code
– represented by major revision number XX in XX.YY
• Minor releases– include updates, bug-fixes and new features NOT
affecting public interfaces in the code– represented by minor revision number YY in XX.YY
• Public patches– include exclusively bug-fixes to a public release
• Development releases– include “state-of-art” development and fixes not yet
submitted to acceptance as public supported release
![Page 49: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/49.jpg)
Software development: the traditional approach
• Correcting software errors very expensive: errors should not be in the code in the first place– get it right the first time
• Sound traditional engineering techniques must be applied
• This leads to Big-Design-Up-Front– Waterfall, SEI CMM, and other techniques were
developed for this purpose
• They are High ceremony processes
![Page 50: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/50.jpg)
An analogy: building a skyscraper
• Detailed architectural and structuraldesigns are needed
• Specialized architects and engineerscreate the design
• The building is made by technicians and workers, following the design
• The skyscraper is made right the first time!
![Page 51: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/51.jpg)
![Page 52: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/52.jpg)
What should we take from XP?
![Page 53: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/53.jpg)
Pair Programming
![Page 54: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/54.jpg)
What should we take from RUP?
• We should follow all the suggested "best practices" – Develop iteratively. – Manage requirements. – Use component architectures. – Model visually. – Verify quality. – Control changes.
![Page 55: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/55.jpg)
Summary
![Page 56: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/56.jpg)
Summary
Producing high-quality software is:
• Far from easy
• Far from cheap
• Still not a solved problem
![Page 57: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/57.jpg)
Lecture III
• Computing at CERN Today
• Software at CERN Today
The future & LHC Computing
![Page 58: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/58.jpg)
Homework
![Page 59: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/59.jpg)
Exercise II
• What will the CERN Computing environment look like in 10 years?
• Hint: some of the key elements exist today, albeit possibly in a different flavour.
![Page 60: Computing at CERN - II Summer Student Lectures 2002 Jamie Shiers](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfbe1a28abf838ca2cb1/html5/thumbnails/60.jpg)
End Lecture II