April 26, 2006Bill Wisniewski1 Run 5 progress Operations and Safety Systems Status SVT Intervention...

April 26, 2006 Bill Wisniewski 1

Run 5 progress Operations and Safety Systems Status

SVT Intervention Decision

DCH Front-end electronics upgrade

DIRC EMC IFR

Barrel upgrade progress and plans Avalanche mode

Trigger Z tracking upgrade

MDI Identification of Needs Summary

Detector Operations


Run 5 Progress

Run 4 ended soon after last review: 112 fb-1 (9/17/03-7/31/04).

Run 5: started 4/16/05 Expected to end 7/31/05

not the full story…

ROD

MD


BaBar Weekly Operations

During running cycle, daily meetings at 7:45 and 3:45:

Work is authorized at both meetings, though the format and constituencies of the meetings differ.

Work Authorizers: Technical Coordinator, Operations Manager, Run Coordinators, Chief Engineer (who is IR2 Area Manager), with participation of Directorate Safety Officers.



7:45 meeting: Attended typically by: TC, Ops Mgr, at least one RC, at

least one of the DSOs, Building Manager, chaired by CE. Other attendees: External groups working in IR2 from/for

Accelerator Dept and CEF Work to be performed in IR2 (and IR12) reviewed.

work by Hall Crew under supervision of Chief Engineer work done by others for BaBar or Accelerator Department

Safety issues related to this work are discussed, for example:

new 480v feeder lines to be connected to Motor Control Center work plan, worker qualifications and training (JHAM, line supervisor signoff)

cable installation work plan, JHAM, RWCF (rad work) if protection devices disturbed

Permit Required Confined Space….



7:45 meeting (continued): updates of ongoing work, reports discussion of general lab safety issues as they pertain

to BaBar work: changes in regulations/training for work on elevated surfaces (fall restraint), use of scissors lifts, hoisting and rigging.

discussion of safety issues that may affect the upcoming major installation.

record of the meeting: BaBar Experiment Logbook reported in header of day shift log, lab daily report binder of copies of Work Authorization forms (extended

projects)



3:45 meeting: attended typically by: Spokesperson, TC, Ops Mgr, the Run

Coordinators, operations managers for all the systems including computing tasks, safety officer. Chaired by Run Coordinator on duty. This is primarily a meeting of physicists.

other attendees (rare): second shift crew representative, typically from Accelerator Department

review by system activities of preceding day (much prompt repair) as well as plans for the next day. Understanding of data taking efficiency losses. Safety issues discussion.

form of authorization: minutes of the meeting, which are retained in Hypernews posting.

monthly system reports on Wednesdays (rotate thru systems)



Prompt Repair: running systems break down and need immediate

repair work authorization can not wait for the next operations

meeting person performing the repair reports to the Shift

Leader (pilot). Run Coordinator on duty is involved if loss of beam time greater than 15 minutes (systems work); approves work by other agencies.



Interactions with Accelerator: Run Coordinators report on preceding day’s

performance/activities by BaBar at 8:00 MCC Operations Meeting

PEPII-BaBar daily coordination meeting (8:15) includes typcially Spokesperson, Tech Coordinator, Run Coordinators with PEPII opposite numbers to plan for the day.

PEPII Accelerator Meeting (M,Th; 3:00): SP,TC,RCs, MDI manager as active observers

PEPII-BaBar Weekly Meeting: SP, TC, OpsMgr, RCs, system ops mgrs and concerned individuals. Updates of weeks activities and performance, plans for upcoming weeks (machine development, repairs).



PEPII-BaBar Meeting account for

downtime/lost efficiency

review backgrounds experience of the week

4/17

4/24

10

The Run 5 Wall of Shame

SVT 34.9 hours

DRC 21.0 hours

IFR/LST 10.0 hours

DCH 5.2 hours

EMC 6.1 hours

IFR/RPC 3.0 hours

SVTRAD 5.5 hours

ODC 15.4 hours

ODF 6.1 hours

DAQ 3.6 hours

TRG 3.3 hours

ORC 2.0 hours

OEP 0.4 hours

COMP 0.7 hours

Total BaBar down time (with PEP up) in Run 5 = 142 hours (~3%) (~110 hrs in Run 5a, 4.4%; balance Run 5b)

+ Solenoid (12 hrs), Gopher (8 hrs), power glitches(2 hrs), Gas shack (1.5 hr), VESDA (1 hr)


BaBar Training

Training for BaBar members authorized to work: Shifters:

gatekeepers: Run Coordinators tool for RCs: SLAC Training Assessment adapted for BaBar must have Safety Intro, Electrical, GERT (rad training)

Training 1st two are required of all SLAC employees

review IR2 Area Hazards Analysis go over Shifter JHAM with Run Coordinators who are the line

supervisors for this activity shifter training session by RCs:

review ESH issues for IR2; walk the site specific training for pilot (lead) and navigator (data

quality) shadow shifts

level of training helps with shifter responsiveness to detector problems; denser blocks/student training


BaBar Training

Training for BaBar members authorized to work: System Workers:

gatekeepers: system managers (or their system ops manager if delegated)

systems specific manuals for repairs and maintenance (web available): OJT

job specific hazards: DCH: PRCS (permit required confined space training) DCH: LOTO (lock-out tag-out) SVT: backward cabling: PRCS IFR & EMC: fall restraint training


BaBar Training

SLAC Training Assessment (for BaBar)


SVT

Issue confronting BaBar at the end of 2004:

Does the SVT have to be removed from BaBar in the next long down for repairs?

Limiting factor to the lifetime of the SVT is radiation damage.

Damage to the sensors: instantaneous (p-stop short: efficiency) & integrated (increase in leakage current, decrease in charge collection efficiency)

Damage to the electronics: increase in noise & decrease in gain decrease S/N; inefficiency from digital failures

Sensors tested ok to 9 Mrad: non-mid-plane modules will reach ~1-1.5 Mrad in 2009 (not a problem), mid-plane modules have problems earlier.


SVT Radiation Dose

scrubbing


SVT

Limiting factor to the usefulness of the SVT is occupancy.

Extrapolate background studies to future running conditions:

YEAR 2004

YEAR 2007


SVT Occupancy

Hit Efficiency Effects

Resolution


BL1M04 (Backward West)

5Mrads

5Mrads

5Mrads 20%20%

20%

DOSE

DOSE

DOSE BEAM OCCUPANCY

20%5Mrads

DOSE

BEAM OCCUPANCY

BEAM OCCUPANCY

BEAM OCCUPANCY


FL1M01 (Forward East)

DOSE

DOSE

DOSE

DOSE

5Mrads5Mrads

5Mrads 5Mrads

20%20%

20% 20%

BEAM OCCUPANCY

BEAM OCCUPANCY

BEAM OCCUPANCY

BEAM OCCUPANCY


Summary of chips affected by radiation/occupancy

Dose only Occupancy only Both

2005

2006

2007

2008

0

0

4

8

11

15

15

14

0

0

2

4


Physics Consequences

Studies on the impact of physics results have been performed for a number of scenarios where we lose the functionality of a different number of chips in the mid-plane:

35.3%

34.5%

B J/s

Example among the most sensitive modes: soft from B D*X

56%51%

Set E = 2 midplanechips off inL1& 2 (32 ICs)

Scenarios with mid-plane L1-2modules off:(UNREALISTIC)


SVT Repair Conclusion

The SVT group concluded in July 2004 that there was insufficient benefit to warrant replacing SVT modules in the next long shutdown: Occupancy in the dying regions would make the replacements useless.

The physics cost of failure to replace the modules is modest and acceptable, given the complexity and risk in replacing the modules. Conclusion endorsed by the physics team.


25

Pedestal shift vs dose (krad)

BL1M4z

RED = old thrBLACK = new thr

Inefficiency vs position after the threshold change

SVT Vicissitudes: Pedestal Shift

Bullet dodged: Can compensate for this dose related shift


SVT Vicissitudes: Bias Current

Layer 4 Problem. Observed initially as an increase in occupancy. Tracked to bias current increase. Swift growth of current would find many L4 modules in trouble in a few months!

10 uA

300 uA 200 uA

25 uA

FL4M11 FL4M16

Bia

s cu

rren

t (uA

)

Bia

s cu

rren

t (uA

)



Intense effort to understand the source of this problem

Direct IV measurements confirm it Not a radiation damage issue; geography is funny: L5 ~no

effect Beams off, bias on give current decrease; converse also

yields current decrease: limit damage source. Changing relative voltage between L4 & L5 has effect. (&

humidity)



Changing relative voltage between L4 & L5 has effect. By properly adjusting the voltages, can fix the problem

Interpretation is static charge accumulation. Radiation ionizes the air between L4 & L5. The voltage difference drifts ions and electrons to the silicon surface.

Simulate layer of charge accumulating on oxide & effect on junction: increase of field at the edge of the junction, which is known to cause localized junction breakdown.

++++++ ++++++ ++++++++


SVT Humidity Tests

Bias voltage problems continue in some modules after center tap change: introduce humidity to reduce charge build-up.

First attempt coincides with damage to two modules, BL3M5, BL3M6. Conjecture is water condensed onto boards. Recovered 32 of 40 chips on these modules.

Second attempt under better controlled circumstances yields success. Set up more robust humidifying system in October 2005.


SVT Reliability Upgrades

Modify power supply boards to allow flexibility on changing the reference voltages

Add two new chillers to provide redundancy DAQ firmware upgraded to fix configuration issues


DCH Run 5 Experience

Test chamber aging studies (two studies)

indicate that DCH should be fine well

beyond the exposure received during the balance of the life of

the experiment.


Motivation: Reduce deadtime due to

serialization and shipping of data from DIOM to ROM

Upgrade in two steps:

Phase 1 (2004): Ship only half the waveform information (3216 bytes) firmware change

Phase 2 (2005): Modify electronics: FPGA to do feature extraction before transmission hardware change

DCH Electronics Upgrade


DCH Electronics Upgrade

Phase 1: waveform decimation

used in Run 5a

Phase 2: new readout board

uses FPGA downloads allow code

updates


Phase 1 in Run 5a

Ship 16 of 32 bytes Keeps timing stamp Reduces dE/dx sampling: no significant performance loss Worked fine except(!):

Feature extraction bug in ROM meant that 4.2% of DCH hits were labelled bad waveforms. Hits were lost. Losses were uniform. Cause: a typo in the assembler code in the ROMs. Problem fixed August.

Tracking fix implemented that compensates for the effects of lost hits.


Phase 2 Installation Three front end electronics modules, one of each

of three types, installed during Run 5a to gain experience with the new design. Waveform decimation implemented.

Installation of balance of modules planned for October down.

Installation went less smoothly than expected: after installation of the first dozen front end electronics

boxes, subtle thermal sensitivity problems were noticed with the outer boxes; the simple fix for this problem, the addition of a capacitor, was quickly implemented.

another system issue was discovered: greater sensitivity to uniformity of DIOM timing (trigger related); this problem was also quickly diagnosed and fixed.

Good planning for the installation included substantial float; installation was completed 2 November.


Phase 2 Performance

Phase 2 electronics appear to be performing well. Downloaded code ran in waveform decimation mode at start

of 5b. In order to get the full benefit of this phase of the

electronics upgrade, move front end feature extraction from the ROMs into the FPGA on the ROB.

FEX code was developed and tested. Deployment mid-February

Recalibration needed & completed for analysis

Fly-in-the-ointment: Single Event Upsets Experience ~1/day in December (recovery difficulties for

shifters) Scheme for quick recovery functioning implemented Long term: aim for redundant implementation/configuration

checks


DIRC

Robust system that now performs well. Front end electronics upgrade during 2004 down:

DCC modifications for rate handling. Got ahead of aging curve in 2005 down: replaced

front end electronics fan trays that were reaching the end of their service life.

PMT watch: effect of water on face of tubes monitored; no accelerated aging; ok for life of experiment.

PMT internal failures (“Christmas Trees”) continue at a low, acceptable rate.


EMC

radiation damage to CsI(Tl) crystals is via formation of color centers

monitor effects of radiation damage via source calibration

perform radiation tests, using an intense Cs source, of array of 16 crystals that simulate the barrel. Exposure comparable to expected dose over life of the experiment

check overall response: OK check effect on uniformity (more of a worry

since can not easily monitor this in situ): uniformity does change with dose, however it is small enough to be ignored, though a correction factor by crystal grower could be applied.

continuing offline activities: improvements to calibration

Monitor Light Yield with Source Calibration

monitor dose with radfets, leakage current


IFR (RPC)

Efficiency (July ’05)

Barrel RPC efficiency dropping ~1%/month,

but endcap RPCs doing well


RPC Forward End-cap Preservation Society


Humid IFR Gas

Apr. 24 Jun 16Layer 13 Layer 13

Layer 16 Layer 16

Humid gas since Feb. 18

Humid gas since May 5

Humid gas since June 8


Avalanche Mode RPCs

Forward endcap RPCs, currently run in streamer mode, see high rates. Rates highest at edge of middle chambers that is closed to beam line.

Lifetime of chambers limited by integrated charge. Limit charge by moving from streamer to avalanche mode on middle chambers. Challenge is smaller signal.

Strategy: install preamps on chambers where backgrounds most localized; test for noise, measure integrated charge reduction. Understand gas mix, voltages, efficiencies in situ. Determine if avalanche mode is worth extending in 2006.


RPC Forward End-cap Preservation Society


Layer 1 Layer 4 Layer 8

Layer 10 Layer 12

Feb. 2003

Oct. 2003

Dec. 2003

Jan. 2004

Jun. 2004

Jun. 2005

Efficiency Loss at Small Radii


Running the RPCs in avalanche mode instead of in streamer mode will reduce the charge/unit area and extend the life of the RPCs. This may be especially crucial for layers 15, 16 and portions of the inner layers. Work has been done at Princeton and Ferrara to understand if it is possible to install preamps in critical regions so that chambers can be run in avalanche mode: should work. Test on endcap inner layers.

RPCs & Avalanche Mode

Forward end-cap services gap


During the October shutdown we have inserted the preamplifier boards, and switched the gas mix for front endcap east door, layer #1, #3, #5 middle section RPCs. Total 6 RPCs are switched.

Disconnect the connector;

Add extension cables and insert the preamplifier board.

For the present test total 480 preamp channels are inserted.

Avalanche Mode RPCs

Dark current reduced by factor of 6


Avalanche Mode RPCs

Streamer mode average

Increase operating voltage 200v

efficiencyFlexibility from gas mixture:Red is current mix: R134A/Ar/Isob/SF6 (75.5/19.4/4.5/0.6)Purple is (75.5/19.4/4.5/0.6)

+FEC threshold is ~2x (–FEC) threshold


New layer

New RPC belt

10cm steel

5 layers of 2.5 cm brass

Barrel6 Brass layers RPC LST2 sectors in 20044 more in 2006

Barrel IFR Upgrade


Summer 2004 Safety Plan Review


Summer 2004 Safety


Forward side

Bottom sextant before and after installation


Backward side

Bottom sextant before and after installation


LST Installation

New east platform for electronics

Special tooling for backward corner block

Peter Kim & Charlie Young


LST Top Sextant Installation

Brass Strips

Modules


HV supply

High Voltage system provides 4 wires for every tube

Enable and control box for HV


Readout electronics

Single ended signals are sent to Front End Cards (outside the detector) and there amplified and discriminated.

A completely new electronics has been developed to readout the signals from:

- strips (z coordinate – beam line direction) positive signal

- wires (phi coordinate – azimuthal angle)

negative signal

Daughter board

Mother board

Ferrara


Successful Milestones

Plan Actual

Start bottom sextant 8/16/04 8/15/04

Finish bottom sextant 9/8/04 9/2/04

Start top sextant 9/18/04 9/16/04

Finish top sextant 10/5/04 9/29/04


Summer 2004 Safety Experience

The plan: Everyone working on the detector during the down

period receives IR2-specific training During April Safety Stand-down, July collaboration

meeting & whenever there was a need during the down time (e.g. weekly sessions, special training sessions for newly arrived techs).

Each shift has Safety Oversight Dedicated team led by Sandy Pierson, including Frank

O’Neill, Joe Kenny, Karen Holtemann, Michael Scharfenstein, John Shepardson, Rick Challman (aside from Frank and Sandy, borrowed from other divisions).

Procedures prepped by Jim Krebs and Bill Sands for disassembly of magnet steel, installation of LSTs and reassembly of the magnet. These procedures include Job Hazard Analyses.

Tailgate safety meetings held for each shift to discuss the work to be done and safety.


Summer 2004 Safety Experience

Post-mortem on experience (LST engineering) Overall the program was successful Engineers and technicians

Initial fears: safety oversight might be too intrusive and cause inefficiencies

Actual events: the team moved into a comfortable working relationship

Instances of unsafe practice typically quickly identified and stopped.

But… Block dropped when hall crew members ignore procedure for

moving a corner block; nylon sling broke and block dropped ~1 ft to floor

Lesson learned: procedures for complicated lifts must be followed, no shortcuts.


LST Modules at Collider Hall

2004

2006


LST 2005/6 Progress

All modules are at SLAC All HV cables, long and short haul, at SLAC All brass at SLAC, machining and layer selection done Two ‘day long’ engineering workshops held including safety

planning Fab of fixtures, platforms, mock-ups almost complete

IFR slow controls integral of RPC & LST Simulations have been validated Geometry has been validated, new sextants aligned to DCH Muon ID: cut based selector checked & tuned; NN selector

tuned on simulation, in process for data


Preparations for 2006 Down

LST installation in the diagonal sextants of the barrel requires substantial disruption of detector services.

For SVT, cables will need to be pulled back to the outer edge of the detector. This was allowed for in the original system design to accommodate pulling back the EMC endcap.

For EMC, the disruptions are greater: load transfer of the forward end of the calorimeter; very extensive uncabling of the calorimeter, and disruption of cooling services and source calibration system.


EMC Load Transfer Review

Held 22-23 September at SLAC with presentations by engineering team of Krebs, Boyce and Dittert. Presentations included design and analysis of the EMC load transfer fixture, installation of the EMC load transfer hardware, and written load transfer procedures.

Reviewers from SLAC: Metcalfe, Skarpaas VII, Doyle, DeBarger

Outcome of review: The design looks good, but some details needed to be improved.



EMC load transfer fixture Pre-fit support beam pads Check clearances from cables and

services at pins

EMC services survey Coolant lines Cable slack (no endcap un-cable?)

Replace aft cableways & rearrange EMC cables for simpler LST install



LST’s will be installed in 4 sextants during the scheduled 2006 down.

Only 4 months available to do more than twice the work done in 2004: very tight schedule with 10 or 11 hall crew shifts a week, owl shift for testing+.

Perform substantial prep work in October October work:

LST HV cable runs under EH platforms and on walls redone so that there is space for next year’s installation, as well as re-routing cables to supplies in the electronics racks on top of the EH

LST on-detector racks installed under the detector west mezzanine


Conduit(Backward East)

LST October Cable Work


Draft Schedule for ’06 Installation

pg 1 of 16


Below-the-hook Fixtures for ’06


Readiness Task List for ’06

169 item list. Items not completed typically load tests, mock up tests, final procedures. These should be complete by ’06 down, though it will be tight.


Installation Preps


Hoisting and Rigging for ’06

There are more than 400 crane related tasks on the project schedule that involve on the order of 2000 crane lifts.

Detailed lifting procedures (~20) for ~100 critical lifts. Remaining lifts are ordinary lifts, which would require a

couple of hundred lift plans. Project schedule threatened by H&R requirements

uncertainty, possible requirement to generate a large number of lift plans for ordinary lifts.


Safety Plan for ’06

Plan based on ’04 safety plan Plan takes into account current training

requirements


Level 1 DCT selects tracks with high Pt (PTD) DCZ allows selection on track Z0

reduces L1 rate due to beam related background by cutting on the Z0 of the track.

Essential for running at luminosities ~1034 to keep event rate low enough to avoid deadtime from bottlenecks (see later talks).

Physics events Beam background events

Trigger: Level 1 Upgrade


DCZ does its job just as well as we hoped Finding tracks and measuring z0 and pT

How is this information used? ZPD counts tracks with cuts on |z0| and |1/pT|

Have tried |z0| < 15(12)cm, |pT| > 200(800) MeV

GLT can combine “Z tracks” with other trigger objects

It’s a delicate balancing act Physics efficiency (which physics?) and robustness Maximum reduction of L1 trigger rate

L1 Trigger Rate


Trigger

Ironed out new DCZ trigger during Run 5a: commissioned in parallel with DCT TSF firmware problem diagnosed and ‘repaired’ TSF ZPD transmission errors fixed DCH-DCT calibration problem fixed DCZ power cycle sequencing understood DCZ configured out of the database nagging power sequencing problem almost ironed

out

EMT has had fast monitoring updates IFT timing aligned better than Run 1-4

(problem between LST and RPC resolved) Great progress in trigger simulation


MDI

How BaBar helps: Background diagnostics ( doses <doses as in past>, aborts

<1/day vs 3/day>, neutron background sources <region .5 to 2m forward; DIRC & RPC susceptibility>)

BaBar based accelerator diagnostics ( trickle performance, background rate along train, real-time luminous region monitoring, IP parameter measurements )

New diagnostic instrumentation (LER x-ray beam size monitor <6% precision>)

Simulations: Bhabha generator incorporated GEANT4 model to Q2 septum LER Turtle + G4 simulations for subsystems NEED: HER Turtle + G4 rays; vacuum model; neutron

validation; more details in G4 model (Q4…etc) Much accomplished, but luminosity will increase by >2x.

Need to increase effort in MDI.


Identification of Needs

In December the Technical Board reviewed mid and long term plans for the detector systems and online.

hardware improvements expected? Fallout: online upgrade; return to dataflow bottlenecks. changes/ improvements to reconstruction envisioned?

What are the systems long term manpower needs? minimum number of FTEs needed in years after 2006

challenge: can you continue to turn out high quality data with much less manpower?

Fallout: round of discussions between Spokesperson-elect, Technical Coordinator and System Managers to explore service task coverage and to identify key positions to be filled. Drop seen in TC’s annual service task audit (FTE per capita) for some institutions adds impetus to this identification of open positions.

Result: develop system task tables to carry through to the end of BaBar.


Identification of Needs: DCH


Identification of Needs: DIRC


Identification of Needs: Trigger


Summary (I)

BaBar has: demonstrated ability to efficiently take data with low

overall down time: even at 1x1034 we have been able to take data with low deadtime.

identified obstacles in the path to efficient data-taking: trigger upgrade that substantially reduces

backgrounds while retaining interesting physics Drift Chamber electronics upgrade to remove a data

flow bottleneck promptly dealt with in a sequenced approach that stayed ahead of luminosity increases

upgrade(d) online compute farms to deal with higher data rates, and to avoid hardware end-of-life problems

see Weaver & MacFarlane talks later: backgrounds and future risks for data flow


Summary (II)

BaBar has: adopted a phased approach to IFR upgrades

that make use of experience gained with detector performance and installation experience to complete the task safely and effectively and to schedule

RPC upgrade for endcap extension of RPC endcap life via avalanche

mode two campaign LST upgrade for barrel to

take advantage of lessons learned developed tools for understanding and

applying manpower resources and safety going into the final years of the experiment

April 26, 2006Bill Wisniewski1 Run 5 progress Operations and Safety Systems Status SVT Intervention...

Documents

Transcript of April 26, 2006Bill Wisniewski1 Run 5 progress Operations and Safety Systems Status SVT Intervention...