ATLAS Operations 2008 Getting ready / First beam / combined running & Shutdown planning

ATLAS Operations 2008Getting ready / First beam / combined running

& Shutdown planning

T. Wengler

DESY seminar

19 Jan 08

The ATLAS collaboration

2

37 countries169 Institutes2500 Authors

The ATLAS Detector

3

Solenoid field: 2TToroid field: 4T

Inner Detector

4

Barrel

The LHC complex

5

Need full LHC for any beam through ATLAS

Milestone commissioning weeks

6

Commissioning of the integrated system

2007

BarrelCalorimeters

BarrelCalorimetersBarrel Muon

M2 + EC CalorimetersEC MuonBarrel SCT, TRT

2008

M1 M2 M3 M4 M6M5

M3 + (-)SCT r/oL1 Mu/Calo

M4 + Pixel r/o(-) TRT

M5 + SCT detectors

M6 highlights 1/3

7

SCT Barrel (~1/2 SCT), stable cooling throughout M6 TRT 25% Barrel A. 10% Barrel C, top + bottom sections RPC Sect 7/8, TGC 3 Sect/side, MDT Sect 1-12 Barrel, Big Wheels, 650 chambers

Reconstructed cosmic SCT-TRT-MuonReconstructed cosmic SCT-TRT-Muon

Phi difference SCT+TRT track segment RPC

M6 highlights 2/3

8

EMB pulse

shape

L1Calo trigger seen in LArL1Calo trigger seen in LAr

Extensive testing of Calorimeters and L1Calo during M6 and after, including trigger timing studies

L1Calo trigger seen in TileL1Calo trigger seen in Tile

M6 highlights 3/3

9

Analysis of one M6 cosmics commissioning run , March 2008– The trigger is requiring TRT tracks reconstructed online within the pixel

volume (equivalent to d0 25 cm)

– Triggered events (red) end up in one stream file, non-triggered events (blue) into another one: proves trigger and streaming are working

All TRT reco tracks

Triggered TRT tracksHigh LevelTrigger

Month Date System Requirements, remarks Parallel Shifts

Week 1721/4 - 27/4

TDAQ/HLT week ID ROSs in useL1Calo ROS requested 25/4

ID standalone tests w/o CTP; BCM integration

April Week 1828/4 - 30/4

L1Calo+Calo run No TDAQ 24 hour period

May Week 1830/4 - 4/5

May 1st Muon-CSC test

Sub-systems: Transition to tdaq-01-09

Week 195/5 – 11/5

5/5 – 6/5 Muon-CSC test 7/5 – 9/5 TRT test8/5 11:00 – 9/5 11:00 TDAQ 24h

SCT likely not fully available yet (cooling) to start on Thursday

Start of SCT calibration [Round 1]Start of magnet test ~HLT algos available

May Week 20

12/5-18/5 12-13/5 for SCT+CTPCalo+L1calo+HLT start on 14th

-Timing, calo DQ, debugging, high rate, algo tests- Finish with a stable week end run

Week days: morning expert work; evening calo + central desks

WE: 16/7 calos + central desks

Week 21

19/5-25/5 19/5 CaloMuon+L1Mu+HLT starts on 20/5

-Same as above

- Finish with a stable week end run with calos

Week days: morning expert work; evening muon + central desks[SCT calibration round 2]WE: 16/7 muon + calos + central desks

Week 22

26/5-1/6 ID+DAQ+HLT -Same as above-Dedicated DAQ test after detector testing and before WE run with muons + calos

Week days: morning expert work; evening ID (Muon/calo?) + central desksWE: 16/7 ID (muon/calos) + central desks

June Week 232/6 – 8/6

Keep system in one piece, perform HLT tests 2-3/6, end 9:00 on 4/6First week of semi continuous running

No Tier-0 ! Magnet testFDR-2 10

Schedule: May-June

Milestone commissioning weeks

11

Commissioning of the integrated system

2008

Full system(-) CSC

2009

M8M7

Full system

first beam M9 ?

cosmics comb.

Magnet system

12

The magnets and their cryogenics, powering, dump etc systems have been fully commissioned to nominal field strength

Another importantmile stone

Closure of the LHC beam pipe

13

The Final Piece: Closure of the LHC beam pipe ring on 16thJune 2008 ATLAS was ready for data taking in August 2008

ACR for 2008 running

14

Trigger•Configuration•Performance

Trigger•Configuration•Performance

Level-1•L1Calo,L1Muon, CTP h/w & r/o•Overall Timing

Level-1•L1Calo,L1Muon, CTP h/w & r/o•Overall Timing

DAQ/HLT•DAQ/Dataflow•HLT/DAQ farms

DAQ/HLT•DAQ/Dataflow•HLT/DAQ farms

Muon detectors•TGC, RPC, MDT, CSC

Muon detectors•TGC, RPC, MDT, CSC

SLIMOS•CIC, safety, access•CCC/TC communication

SLIMOS•CIC, safety, access•CCC/TC communication

Run Control•Main RunControl

Run Control•Main RunControl

Shift Leader•Shift Leader tasks•LHC communications•DCS central operation

Shift Leader•Shift Leader tasks•LHC communications•DCS central operation

Data Quality•Central DQ•Online/Offline/Tier-0

Data Quality•Central DQ•Online/Offline/Tier-0

Tile Tile

Exp. Supervisor•Chair + Laptop

Exp. Supervisor•Chair + Laptop

Fwd Detectors•Lucid, ZDC (ALFA …)•Luminosity

Fwd Detectors•Lucid, ZDC (ALFA …)•Luminosity

TRT TRT

SCT SCT

Pixel Pixel

ID Gen•Group two screens intoID Gen access station(evap. cool etc.)•No separate shifter

ID Gen•Group two screens intoID Gen access station(evap. cool etc.)•No separate shifter

LAr•Two shifters with separate access

LAr•Two shifters with separate access

Going into beam op. Sep 10th

• Muon system (MDT, RPC, TGC) on at reduced HV

• LAr (-FCAL HV), Tile on

• TRT on, SCT reduced HV, Pixel off

• BCM, LUCID, MinBias Scint. (MBTS), Beam pickups (BPTX)

• L1 trigger processor, DAQ up and running, HLT available (but used for streaming only)

ATLAS was ready for first beam:

tertiarycollimators

140 m

BPTX175 m

LHC start-up scenario:• Go step-by-step, stopping beam on

collimators, re-align with centre

• Open collimator, keep going

• Last collimator before beam through ATLAS – tertiary collimators (protection of triplets)

• Splash event from these collimators for each beam shot with collimator closed

LHC beamloss monitor

Strategy for first beam / first signals

• Splash events are first– only 10 shots expected – not

enough time to set up BPTX

– Rely on small radius triggers with well defined cosmics timing (L1Calo + MBTS)

– Catch the first events!

• Start timing in Beam Pickups (our timing reference) rapidly to trigger on through-going beam– First signal seen in ATLAS

– Multi-turns!

BPTX

Trigger

Worked out nicely, with very stable data taking

RF captured beam for > 20 min on day 2!

16

The first events – did we get them?

17

Zoom into the first beam ‘splash’ activity with beam-1 closed collimator, recorded in run 87764.

Of 11 beam ‘splashes’ (solid-blue), 9 have been triggered (ATLAS events are dashed-black, the event numbers are indicated).

“First beam event seen in ATLAS” (not quite !)

Timing of the trigger inputs

18

1 BC = 25 ns

Intense programme of timing adjustments of trigger inputs

Only about 30 hours of (any kind of) beam in 2008 …

The first has been delivered to the CCC

20092009 20092009

Delivered!

LHC incident on Sept. 19th

20

See slides R. Aymar attached below for full details

Powering tests in Sect 3-4 for

5.5 TeV operation

Powering tests in Sect 3-4 for

5.5 TeV operation

Resistive zone in bus between C24 and Q24

Resistive zone in bus between C24 and Q24

In < 1 sec power converter tripped and dump started

In < 1 sec power converter tripped and dump started

Systems reacted as expected until here

Electrical arc in bus, punctured

helium enclosure

Electrical arc in bus, punctured

helium enclosure

Helium release into isolation vacuum of

cryostat

Helium release into isolation vacuum of

cryostat

Safety valves start venting helium to the

tunnel

Safety valves start venting helium to the

tunnel

Pressure rise too fast for valves,

pressure wave in isolation vacuum

Pressure rise too fast for valves,

pressure wave in isolation vacuum

Damage to super-insulation

for some magnets

Damage to super-insulation

for some magnets

Displacements where wave hits vacuum

barriers

Displacements where wave hits vacuum

barriers

Both beam pipes cut open

and contaminated

Both beam pipes cut open

and contaminated

About 60 magnets to be removed for repairs etc

About 60 magnets to be removed for repairs etc

No more beam in

2008

No more beam in

2008

Q27R3

21

R. Aymar, PECFA, 28 Nov 08

QQBI.27R3

22


Magnet movements Sector 3-4

23

Displacements status in sector 3-4 (From Q17R3 to Q31L4)Based on measurements by TS-SU, TS-MME and AT-MCS

P3Q17 A18 B18 C18 Q18 A19 B19 C19 Q19 A20 B20 C20 Q20 A21 B21 C21 Q21

Cryostat <2 <2 <2 <2 <2 <2 <2 <2 <2 <2 <2 <2 <2 <2 <2 <2 <2CM Longi ? ? ? ? ? ? ? ? <2 <2 <2 <2 <2 <2 <2 <2 <2CM Vert ? ? ? ? ? ? ? ? <2 <2 <2 <2 <2 <2 <2 <2 <2CM Rad ? ? ? ? ? ? ? ? <2 <2 <2 <2 <2 <2 <2 <2 <2

Q21 A22 B22 C22 Q22 A23 B23 C23 Q23 A24 B24 C24 Q24 A25 B25 C25 Q25

Cryostat <2 <2 <2 <2 -7 <2 <2 <2 -187 <2 <2 <2 <2 <2 <2 <2 <2CM Longi <2 <2 <2 <2 -20 -65 -104 -141 <2 -186 -127 -70 <2 <2 <2 <2 <2CM Vert <2 <2 <2 <2 <2 -6 -5 -4 <2 -4 -5 -5 <2 <2 <2 <2 2CM Rad <2 <2 <2 <2 <2 0/10 11/8 7/3 <2 15/3 8/13 11/3 <2 <2 <2 <2 <2


Cryostat <2 <2 <2 <2 <2 <2 <2 <2 474 -4 <2 <2 11 <2 <2 <2 <2CM Longi <2 <2 <2 <2 <2 57 108 168 -38 232 188 145 95 70 35 3 <2CM Vert 2 <2 <2 <2 <2 -5 -5 -4 -26 58/-7 -7/-5 -8/33 12 -5 <2 <2 <2CM Rad <2 <2 <2 <2 <2 2/<2 8/9 3/15 22 20/<2 <2/12 16/6 <2 <2 <2 <2 <2


Cryostat <2 <2 <2 <2 <2 <2 <2 <2 188 <2 <2 <2 5 <2 <2 <2 <2CM Longi <2 <2 <2 <2 <2 19 81 146 <2 141 102 63 10 <2 <2 <2 <2CM Vert <2 <2 <2 <2 <2 <2 -5 -4 <2 -11/-5 -6/-5 -5 3 <2 <2 <2 <2CM Rad <2 <2 <2 <2 <2 <3 3/6 10/17 <2 -3/6 6 6/<2 <2 <2 <2 <2 <2

Q33 A34 B34 C34 Q34 C34 B34 A34 Q33 C33 B33 A33 Q32 C32 B32 A32 Q31

SSS with vacuum barrier Open interconnection Disconnected P4>0 To P4, up, center Electrical interruptions Removed[mm] Values are in mm Dipole circuit (diode) XYZ Reinstalled? Not measured yet Electrically damaged IC

Cold mass displacement Buffer zones Electrical cantonsCryostat displacement Date 25/11/2008 JPh Tock


LHC incident on Sept. 19th

• Prevention of similar incidents– Search for indicators of faulty connections in commissioning

data– Retests of all suspicious locations– Improvements on Quench Protection Systems

• Mitigation of consequences in case of similar incidents– Increase number and discharge capacity of relief valves– Reinforce external anchoring of cryostats at the locations of the

vacuum barriers

24

See slides R. Aymar attached below for full details

Shutdown intervention schedule

25

ATLAS reacted with a revised running and shutdown plan:

Running 2008

26

Oct Nov Dec41 42 44 45 46 47 48 49 50 5143

41 42 44 45 46 47 48 49 50 5143

So +To Sol. only

Sub-system (group) commissioning24/7 combined run

Stop TGC n-pentane

Start to open

Full shiftcoverage as agreed for this period

9:30 RunMeeting27. Oct

Weekly run meetings only

• SLIMOS + SL(RC)• TDAQ/Sys-admin on-call• Effort from sub-systems as

needed for ongoing activity• RPC S12-14 available for

combined runs

Run Priorities until Oct 27th

1. Collect enough data with small wheels to check efficiencies of chambers potentially damaged by overpressure before n-pentane stops. Implications:– Recording: Record all TGC triggered events

– Trigger: enable full TGC trigger coverage for some time (partial blocking of trigger with good timing for ID end-caps)

2. Collect large samples of good ID tracks with Solenoid on/off. Implications:– High recording rate of L1 trigger = small event size (HLT not fully

efficient for these conditions)

3. Collect large samples of good Muon spectrometer tracks with field off/on/off. Implications:– Same as 2.

4. Run combined system for cross-detector studies and stability– All (available) systems in readout 27

✔

✔

✔

✔

Data overview

28

High rate running

• Re-visited at the end of combined period– Stability (i.e. people not touching essential parts of their DAQ

and FE) had started to erode somewhat …– Still managed to do several high rate runs, with random triggers

at L1, filtered at HLT

• Today’s evidence suggests we can run combined with about 40 kHz L1A– No indication for bottlenecks at nominal rates downstream

• Also did test on stop-less recovery (automatic disabling of blocked read-out links) – switched on without problems, but – no systematic testing (provoked failures) during combined

running 29

After the end of the combined run

• Moved immediately into system commissioning– ID running 24/7 shifts for flat-out commissioning work into

December– Frequent L1Calo/Calo runs with and without CTP, including 2

weeks of overnight/WE runs from Nov 3rd – RPC trigger test and threshold scans– …

• Major events still in 2008– ID combined run (RPC trigger, TRT fast OR) 26th Nov – 1st Dec– New major TDAQ release beginning of December

(testing by sub-detectors starts before end of 2008)

• Test Programme– Bare-bone run transition timing (all monitoring etc turned off)– Check situation on amount of messages in MRS

30

Collection of cosmic ray data• Full cosmic rate (dominated by RPC) at L1 is ~500 Hz

– Nominal ATLAS recording rate is 200 Hz @ 1.5 MB/event– For cosmics with full system 3-13 MB/event, dominated by number

of LAr samples read out → We need the HLT for cosmics!

• Example: Rate of Pixel tracks:

31

Full L1 raterecorded (no LAr)

HLT cosmics filtering learning curve

New TRT fast-OR at L1

Pixel alignment

32

An example of using cosmic-ray data for detector alignment

The TRT as …

33

Bubble chamber

A cosmic event with

field on

Occupancy ~ 1%

A beamsplash event

Fixed target detector

Occupancies up to 30%

Transition radiation in the TRT

34

Transition radiation (TR) photons generated by radiator foils (boundary of 2 materials with different dielectric constants)

Effect starts at γ = (E/m) ≈ 103

➞ mostly for electron ID

In the TRT, photons are absorbed in chamber gas

➞ large pulse ➞ passes high threshold

γ

Turn-on of TR from cosmicsat about γ=103 as expected

Muon spectrometer

Optical alignment system• Dead channels < 1%• End-cap: 40 μm o.k.➞• Barrel: 250 μm in progress ➞

Very good correlation between RPC (trigger chambers) andMDT (precision chambers) hits

X-ray of the ATLAS cavern with cosmic muons

Elevators Access shafts

Calorimetry

36

1 MeV

Pedestal stability in LAr EM layer over a 5 months period

Tile cell uniformity measured with horizontal beam

L1Calo trigger tower (x) vs. calorimeter full readout (y),Spread will decrease with full calibration

How much of ATLAS is operational

• Inner Detector– Critical item was evaporative

cooling plant (for Pixel/SCT) after failure on May 1st 2008 (repair finished end of July 2008)

– About 2.5 % dead channels due to cooling/module problems, similar amount of dead channels in TRT

• Calorimeters– For LAr ~0.01% dead channels in

EM, and ~0.1% in HEC, plus LV power supply affecting ¼ of one endcap (under repair)

– No dead channels in FCAL

– About 1 % dead channels in Tile cal

• Muon spectrometer– Low noise and low number of dead

channels

– MDT: ~1.5%, CSC: 0.5%, TGC: 0.03%, RPC: 6% (commissioning still ongoing)

• Magnets– Toroids and Solenoid fully

operational

• Trigger and DAQ– All L1 trigger inputs and central L1

trigger fully operational

– HLT farm and algorithms, and DAQ system fully operational (will be scaled up according to need)

37

In all cases expect to recover a significant fraction during shutdown from an already small number of missing channels

Conclusions

• We had a very successful start-up– Caught the first beam events

– Managed to go through part of the single beam commissioning programme on day 1+2, with beam

– We start to be able to operate and steer ATLAS as one system

– On-the-fly L1-pre-scale changes and r/o recovery are extremely useful

• Only the CSC now missing from regular combined data taking– CSC chambers up and running, r/o

ROD issues being addressed, first tracks seen in combined runs

– Pixel off on day one for safety reasons, but has seen many cosmic tracks in combined running since

• Timing needs to be completed– As much as possible with test-

pulses & cosmics over the next months, but need beam to finalise it

– Will be in good shape for collisions

• The operation model is working– Control room is working

– Operational structure is sound

• Main work for operations– Bring down overheads (and up

efficiency)

• run start/stops

• recovery for power cuts

• DQ tools / turn-around

• …38

Additional Slides

Inner Detector

40

Calorimeter system

41

Muon system

42

Beam RF capture

43

Correct phasingCorrect reference

‘Mountain range’plot

‘Each line = one bunch

orbit

LHC RF group

Beam splash events in LAr and Tile

44

Energy in LAr EM layer 2

Average cellenergy in Tile

LHC incident on 19th Sep 2008

45

Incident during powering

The magnet circuits in the seven other sectors of the LHC had been fully commissioned to their nominal currents (corresponding to beam energy of 5.5 TeV) before the first beam injection on 10 September 2008. For the main dipole circuit, this meant a powering in stages up to a current of 9.3 kA. The dipole circuit of sector 3-4, the last one to be commissioned, had only been powered to 7 kA prior to 10 September 2008. After the successful injection and circulation of the first beams at 0.45 TeV, commissioning of this sector up to the 5.5 TeV beam energy level was resumed as planned and according to established procedures.

On 19 September 2008 morning, the current was being ramped up to 9.3 kA in the main dipole circuit at the nominal rate of 10 A/s, when at a value of 8.7 kA, a resistive zone developed in the electrical bus in the region between dipole C24 and quadrupole Q24. No resistive voltage appeared on the dipoles of the circuit, so that the quench of any magnet can be excluded as initial event. In less than 1s, when the resistive voltage had grown to 1 V and the power converter, unable to maintain the current ramp, tripped off, the energy discharge switch opened, inserting dump resistors in the circuit to produce a fast power abort. In this sequence of events, the quench detection, power converter and energy discharge systems behaved as expected.


Summary Report on the analysis of the 19th September 2008 incident at the LHC

46

Sequence of events and consequences

Within the first second, an electrical arc developed and punctured the helium enclosure, leading to release of helium into the insulation vacuum of the cryostat.

The spring-loaded relief discs on the vacuum enclosure opened when the pressure exceeded atmospheric, thus relieving the helium to the tunnel. They were however unable to contain the pressure rise below the nominal 0.15 MPa absolute in the vacuum enclosures of subsector 23-25, thus resulting in large pressure forces acting on the vacuum barriers separating neighboring subsectors, which most probably damaged them. These forces displaced dipoles in the subsectors affected from their cold internal supports, and knocked the Short Straight Section cryostats housing the quadrupoles and vacuum barriers from their external support jacks at positions Q23, Q27 and Q31, in some locations breaking their anchors in the concrete floor of the tunnel. The displacement of the Short Straight Section cryostats also damaged the “jumper” connections to the cryogenic distribution line, but without rupture of the transverse vacuum barriers equipping these jumper connections, so that the insulation vacuum in the cryogenic line did not degrade.



47

Inspection and diagnosticsThe number of magnets to be repaired is at maximum of 5 quadrupoles (in Short Straight Sections) and 24 dipoles, but more (42 dipoles and 15 quadrupoles) will have to be removed from the tunnel for cleaning and exchange of multilayer insulation.

Spare magnets and spare components are available in adequate types and sufficient quantities for allowing replacement of the damaged ones.

The extent of contamination to the beam vacuum pipes is not yet fully mapped, but known to be limited; in situ cleaning is being considered to keep to a minimum the number of magnets to be removed.

The plan for removing/reinstallation, transport and repair of magnets in sector 3-4 is being established and integrated with the maintenance and consolidation work to be performed during the winter shutdown.

It should be available for the next Council meeting in December.

The corresponding manpower resources have been secured.



48

Follow-up actions (preliminary)

Two different goals, namely to prevent any other occurrence of this type of initial event, and to mitigate its consequences should it however reproduce accidentally. Precursors of the incident in sector 3-4 are being scrutinized in the electrical and calorimetric data recorded on all sectors, which remain cold, in order to spot any other problem of the same nature in the machine.

• An improvement of the quench detection system is currently tested, before being implemented.

• The relief devices on the cryostat vacuum vessels will be increased in discharge capacity and in number.

• The external anchoring of the cryostats at the locations of the vacuum barriers will be reinforced to guarantee mechanical stability.

Until now, no other interconnection resistance has been identified as above specification, but two (?) connections inside the cold masses (which have been tested successfully to 9T) have been measured higher than specified.


ATLAS Operations 2008 Getting ready / First beam / combined running & Shutdown planning

Documents

Transcript of ATLAS Operations 2008 Getting ready / First beam / combined running & Shutdown planning