CERN [email protected] ITER – Machine Protection [email protected] MPP Workshop –...

CERN

[email protected] ITER – Machine [email protected] MPP Workshop – Annecy 2013 1

WELCOME to the MPP Workshop 2013 in Annecy!

CERN

[email protected] ITER – Machine [email protected] MPP Workshop – Annecy 2013

Past reviews

●MP related reviews that were done in the past• External Machine Protection review (June 05 - overall system)• BIC audit summary (Sept 06)• BIC audit sequel summary (June 09)• LBDS audit summary (Feb 07)• LBDS audit sequel summary (June 09)• BLM audit summary (July 08)• BLM audit sequel summary (June 09)• MPS internal review (June 2010)• Collimation system (several reviews)• External LHC Machine Protection Review (September 2010 - are we ready

to go beyond 30 MJ?)• Machine Protection Workshop for linear accelerators (June 2012)• Machine Protection Workshop (March 2013)

CERN


Mandate

●Discuss mid-and longer-term improvements of the MP systems for the LHC + injector complex:

• review of the current operational experience with MP systems during the first running period (2009-2013).

• understanding the planned changes of MP equipment during LS1 and the consequences/potential limitations for operation after LS1.

• identify areas where improvements are required.

• ensuring coherence between the different MP systems.

• identify misses.

CERN


Program

● Indico Page●6 sessions, 2 ½ days, ~ 85 participants

• MPS experience (2008-2012) and outlook (D.Wollmann)

• Injection and LHC beam dumping system (J.Uythoven)

• Beam Diagnostics (B.Dehning)

• Collimation and movable devices (S.Redaelli)

• Electrical circuit related protection (M.Zerlauth)

• Operation after LS1 (J.Wenninger)

• Summary and discussion (R.Schmidt)

• Executive summary @ CERN in special LMC (date tbc)

https://indico.cern.ch/conferenceDisplay.py?ovw=True&confId=227895



CERN


A bit of general info first….

●No (more) coffee break on MON morning!

●Free WiFi in the whole hotel, consume with moderation

●Lunch scheduled for 12.30 in the restaurant (seated)

●Rooms will be ready around 16.00

● In case of any problems, please contact Mariane

(+41 76 487 87 53)

5

LHC Machine Protection WorkshopM. Zerlauth March 2013

Thanks to : B.Dehning, K.Fuchsberger, B.Goddard, S.Redaelli, J.Uythoven, R.Schmidt, J.Wenninger, D.Wollmann, MPP and rMPP members et al

1v0

MPS issues and MP approach concerning operation and MDs

• 3 years of MPS in short • Organization and issues• MP protection during commissioning and production• Non-standard operation• Conclusions

CERN


3 years of LHC Machine Protection in short ● Some 3499 clean beam dumps performed since Oct 2009:

1582 beam dumps above 450GeV.In 2012 a majority of dumps with beam energies > 100MJ (reaching max 146MJ).

● No beam induced magnet quenches @3.5/4.0TeV.● No serious equipment damaged due to beam (excl. heating, corrector coils of

RQX.3L2, ALICE detector).● Reasons and MPS response reasonably well documented, analyzed and validated

for all beam dumps > 450GeV. More details see talk from B. Todd

7

2012

2011

2010

Courtesy B. ToddCourtsey of B.Todd – Evian 2012

CERN


In the end there was reason to be content…

>23 fb-1

CERN


Driving along the edge throughout…● LBDS 12V● Quench detection issues on IPQ, 600A EE● TL collimators● Injecting H9● False collimator settings (2 x TCTV IR2, 2x IR3)● Roman Pot Controls● BLM High Voltage Cable ● OFSU reference problems● BSRT Mirror● MDs and other non-standard machine

operation● MKI flashovers ● QFB not usable in squeeze due to poor signal● Instrumentation problem in triplet L8 after TS2● Loss of redundant protection (60A power

permits, LHCb dipole , CMS solenoid,…)● ….

9

CERN


Mandate and Organization of MP Work● Assure (potentially) dangerous situations are recognized, documented and

distributed and actions followed up throughout commissioning and production

● MPP (in present form since Sept 2008) for scientific ground work and preparing decisions – ties and certain overlap with COLL WG, LIBD, LBOC, AWG,..• Will need to increasingly look into HL-LHC and injectors as well

● rMPP added during production run (Sep 2010) to follow up and decide on ‘urgent’ issues and non-standard operation• Very good for ad-hoc discussions (outside large bodies), sometimes had to decide in

absence of few key members -> Still need a common line from rMPP

• Need for (more) regular meetings? Update of rMPP membership?

• Follow-up & responsibility of recommendations/ procedures less obvious (MD documents, checklists,….)

● Interplay of MPP and MP3 for performance of magnet powering system to be defined/improved for post LS1?

10

CERN


Examples of issues and how we dealt with them

● Safety critical issues in MPS systems● Inversion of HTS instrumentation cable on

RB.A45 found following CL quench on 7th April 2011 due to cooling problem on DFBAI.L5 ● Stopped powering for verification of

all >6kA circuits for 1.5 days

● Common 12V dependency in LBDS system: Found 8th of June 2012 during Lab test (follow up of LBDS review on redundant powering) ● LHC coordinator and rMPP informed,

ongoing fill dumped + operation stopped for some 5 hours for implementation of watch-dog

11

CERN



● Wrong TL-TCDI settings for SPS Q20 optics found on 19th of November 2012 (up to 1.3 σ)● CCC informed and operation stopped for 12 hours to recalculate + revalidate

● Missing BLM HV noticed by sanity check on 12th of April 2011● System reconfigured and validated (using supply and spare cable from LSS),

later SIS interlock added

12

Cell 12-34

First indication in Pre-op Check

CERN



● Issues in MPS systems resulting in known (and protected) failure cases• E.g. non redundant LBDS powering, MKI flash-over, injection of H9,… • Operation in general continued, further minimizing risk through longer term

mitigations● Issues with non-MPS systems with certain potential to become unsafe

• E.g. OFSU without reference, BSRA, RP controls• Operation in general continued straight away, short-term mitigations put in

place to further minimize risk, e.g. postponed next intensity step, new SIS interlocks and additional sequencer and operator checks

13

Missing Orbit reference

CERN


Would a MP ‘responsible’ be beneficial?

● Major issues were mostly ‘dormant’ failures and appropriate action was taken immediately• Lessons should be learned and commissioning procedures tightened up

● Some issues persisted for some time & reaction might vary as function of individuals knowledgeable about it

● Double role of EIC/Machine Coordinator to optimize performance and assure protection not always easy to combine

● To increase and facilitate support to operation crews: Role of an ‘MP responsible / week’ for post LS1?• Contact (representing rMPP) for operation crews in case of doubts and

non-standard operation• Assure necessary documentation, follow-up of operational issues, e.g.

analysis of dumps,...• Follow-up of MPS commissioning, operational changes + necessary

revalidations

14

CERN


Documentation of MPS tests and issues - Commissioning● Very rigorous and thorough approach to preparation, execution & documentation● Required time was in general allocated /available ● Commissioning procedures need to be updated and revised during LS1

• Profit to organize execute as well ‘difficult’ tests (redundant powering,…)● MPS site worked fine – driven by few individuals in parallel to coordination, progress &

status of advancement not obvious● Use ACCTEST for more dependable + automatic sequencing, dependency tracking + follow-

up

15

See talk of K.Fuchsberger

CERN


From commissioning towards production● Commissioning mindset should be maintained throughout full running period, suffered

towards end increasingly due to scheduling pressure, routine (we have done this before without problem), relatively smooth running, fatigue….

● Initial (commissioned) safety level certainly degrades under such circumstances● Possibilities to counter-act?

16

• Tracking of (relevant) system changes• More coherent approach for follow-up of magnet &

beam related MPS issues• Define & enforce minimum (non-negotiable)

validation costs of changes, e.g. new setting validation with loss-maps & asynch dumps, critical interventions,… -> Takes ‘responsibility’ off the piquet teams, operation crews, coordinators,…

• Automatic tools/dependency models could help to enforce this for well known cases

• MP responsible for remaining cases

CERN

[email protected] MPP Workshop – Annecy 2013

Change Management

17

● MPS HW and SW changes tracked during past 2 years in MPS commissioning SharePoint Site, however…

● We have to clearly define what is mandatory to be tracked

● Requires corresponding culture and level of responsibility of experts (many changes not even known/advertised!)

● Even more so for SW deliverables -> Culture of best practice (Good design, Unit Testing, Integration Testing , Functional Testing)

● Could a commissioning tool like ACCTEST (in conjunction with a dependency management) be re-used to enforce re/validation process after changes?● Concentrate on few most critical &

reoccurring items first, like powering system, settings validations, …

CERN


During production running● Post Mortem DB with operator and MPS expert comments (works well for dumps above

injection energy, how to improve for dumps @ injection?!)• Can and need to do more automated validation checks (pre and post-op, IQC,

XPOC…), difficulty is vast failure space and too frequent false positives● Regular distribution of MPS check lists during intensity ramp-ups and operation to MP

system experts with all dumps above injection energy for comments

18

• Considered very useful by MP experts, should this be distributed to anyone else?

• Regular reflection on past issues, comparison with own fault tracking and reminder for follow-up

• Problem of ‘self-validation’ could be improved by true approval circuit

• Should be enforced on more regular basis (e.g. did none during Ion running)

CERN


Machine Developments 1/2

19

● Machine Developments per definition explore new machine and machine protection territory and hence risks!

● MP documents did allow to address and mitigate risks – did preparation phase increase as well the efficiency of the MDs?

MD 1 MD 2 MD 3 MD 4 QT FTMD 1 FTMD 20

1

2

3

4

5

6

7

8

# MDs with MP Doc

Unwanted Dumps

MD 1 MD 2 MD 3 MD 4 QT FTMD 1 FTMD 20

1

2

3

4

5

6

7

8

9

# MDs without MP Doc

Unwanted Dumps No Docs

● Un-intentional losses of beams as (one possible) indicator for ‘efficiency’, e.g. hidden interlocks during change of SBF, (not counting machine unavailability, equipment failures,…)

● With detailed MP MD doc ~ 0.38 unintentional dumps/MD

● Without doc ~ 0.67 unintentional dumps/MD

Courtesy of D.Wollmann

CERN


Machine Developments 2/2

20

● Additional factors influencing ’efficiency’● Increasingly flexible plannings towards the end, many documents done ‘last minute’,

little time for comments and iterations● Operations not sufficiently aware of MD details and requested machine preparations

Proposal for post LS1:MD – 4 weeks:● For all MDs up-to-date MD request

document (parameter space, interlocks, program..) precondition for being scheduled

MD -2 weeks:● Draft procedure for MD Class C ● True EDMS approval circuit, all rMPP

members to signMD:● EDMS approval precondition to start● Simple additional SIS interlocks to

enforce defined parameter space ?!● OP shifters responsible to assure correct

execution of MD and eventual checklists

CERN


Summary

21

LHC Machine Protection Systems have been working well during run 1

No evidence of major loopholes or uncovered risks in MPS architecture, however a few near-misses have revealed short-comings in designs, commissioning

procedures and operation of MPS systems

Organization and response of coordinators, OP, (r)MPP, equipment experts to these problems was adequate

As a result of many different factors some of the initial rigor appeared to be lost at all levels towards to end of the run

Could an MP responsible be an additional asset for operations and help counteracting some of the issues observed?

It might become increasingly challenging to maintain systems at their needed level of dependability (additional operational challenges post LS1, allocated resources

and time,..)

CERN


Fin

CERN [email protected] ITER – Machine Protection [email protected] MPP Workshop –...

Documents

Transcript of CERN [email protected] ITER – Machine Protection [email protected] MPP Workshop –...