6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties 2011 6. Fill...

No Slide Title

6x. Shift Leader Refresher

Like driving a car Maneuvers may feel like routine, but never take the eyes off the road, the instruments, and the mirrors(=DQ)!!

16. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonEnergy 4 TeV

Intensity ramp up reduced to 7 steps in 20123 bunches for Machine Protection validation (below Safe Beam Limit)2-3 fills and 4-6 hours with 48b, 84b, 264b and 624b (cycle validation)LHCb recommissioning with beam3 fills and 20 hours with 840b, 1092b, 1380b (lumi related problems)At Ib~1.6E11, e~2.5mm, b*=3m, 840(816) bunches: L~ 3.7E32 /m~2.4 @ 2s separation

Ultimate 2012 luminosity and rate at (almost) Day 1Requires full attention and INITIAL stable operation/good data quality5 min lost in April is equal to 5min lost in October1.5 fb-1 is a worthy challenge

2

3.5E32 4E32 1 MHzRunning Conditions 20126. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonShift Leader DutiesAssist Data ManagerYour first task on shift is to get to know the DM and to estimate his experience/qualityCmp racing map reader!Do the first round of DQ checking together on EVERY FILL!In general, watch his displays and help judging data quality based on your knowledge about the running

Attend Run Meeting (send DM if you are busy)Follow running during your period of shifts (Logbook, Run News)Every Start of Shift:Talk to previous shifter and read his Shifter Summary before he leavesRead Daily Shifter InstructionsRead Common Problems and RemediesRead logbook of last 24h Make a tour of all screens to make a cerebral snap shot

Operations Twiki Webpages: https://lbtwiki.cern.ch/bin/view/Operation/WebHome


R. JacobssonShift Leader Duties contdCalling piquet/expertsMake clear who you areGeneral situation in a handful of words (We are in physics/We just dumped)Summarize main point/problem/incident in one or two sentencesIn most circumstances, experts will already have a clear list of questions to narrow down the originLet yourself be guided by experts questions and fill in with surrounding relevant information at secondary level

After interventions, invite the experts to put a brief entry with the problem/action/recipe in the central logbook, and also link to local piquet logbook entry if existing

As Shift Leader you will mainly use the following toolsLHCb Big BrotherRun ControlFarm MonitoringTrigger Rate MonitoringAlarm and Log screensRun DatabaseLHC Operational Monitoring Detector Safety SystemAccess Control SystemElog


R. JacobssonReminder: Golden RulesAlways keep acquisition together runningTake data even without collisions (Destination Local)

In case a detector is excluded to fix a problem, run the system with the rest to detect other problemsEVEN IF YOU THINK DATA WILL NOT BE USEFUL (ex. M1, DQ nuancing)

Before stopping the run while data taking to fix a problem, think twiceIs it necessary?Can another pending problem be fixed in parallel?Is the plan clear, who does what and in which order?

If you discover a problem or an anomaly in data, restart run (fast run change)

Recovery 3 attempts/10 minutes before calling expert

Dont panic!

Click calmly doubts about whether you did the action or not is a potential source of disaster..!Mouse at the centre of objectFirm single clickWait until next actionObserve all that is happening on the panel Message Box

5


R. Jacobsson

Speaker Test!Every start of shift66. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonLHCb Control TreeSystem control tree based on a Finite State Machine (FSM Tree)Control from top, commands passed down and status upSub-node status according to rules (device unit/control unit)Ownership concept: included/excludedGraphics User Interface associated with each sub-node7


R. Jacobsson

Excluded FSM ObjectsMake sure everything is included in Run Control FSM Tree and LHCb State Control TreeAt Start of Shift and Start of Fill

If something is excludedInvestigate and get confirmation from expert

8

Incomplete

Some CU excluded

Some devices disabled

Dead FSM in the tree below


R. JacobssonOwnership PadlockExcluded & Locked Out

Unlocked

Locked Included

By You

By Someone else

By Nobody9

Incomplete

Some CU excluded

Some devices disabled

Dead FSM in the tree below

Shared

By You

By Someone else

Devices

Enabled

Disabled

Golden Rule n+1: At beginning of shift, go through system to check what is excluded/disabled

Golden Rule n+2 :Handle padlocks cleanly, lock in global/unlock/lock in local/ask what is going on if padlock is left open, its likely the detector has fallen into oblivion


R. JacobssonNovelties and repeats 2012Deferred HLT trigger

End-Of-Fill calibrations

Data taking until DUMP, VELO IN until last proton leaves the machine

Tilted crossing plane

Luminosity Leveling revised

Source of trigger dead-time and how it is counted

Automatic recoveries (HV, and DAQ) Autopilot ON

High Voltage status and automatic recovery of channel trips

Event Filter Farm (HLT Farm)Majority mechanism to Start RunOn the fly removal/inclusion of entire subfarm or individual nodesHLT task automatic recovery

Ignore / Include feature on the fly of Monitoring, Reconstruction, Calibration


R. JacobssonRepeats 2012Farm Log ViewerLess messages, hopefullyMessage level and filtering

Run ConfigurationDCS must be READY!Partition / Activity / Trigger Configuration / HLT+L0 TCK / Internal ODIN TriggersData Type and Data DestinationFast Run Change

Level 3 Alarm

Power Cut

Access Handling and Access Application documented carefully on Shifter Instructions

Homework: Try and explore all these features in interfill time


R. JacobssonDeferred HLT TriggerIdea:Farm is essentially idling during interfill timeDuring fill: Overrun farm processing capacity and store rate overflow on local diskDuring interfill: Process overflow and close runs

Each farm nodes has 200 Gbyte of disk space allocated to L0 EventsSpace for >5E9 eventsExample: 10% over-commitment 15 20h, i.e 1.5 2h processing in interfill

Instead of throttling the L0 rate via the Event Request scheme, write overflow L0 events temporarily on local disk (OverflowWriter)Special Deferred HLT Trigger partition processes the events during interfillThus, Event Requests as long as disk space is availableIf disks get full, system will fall-back to throttle L0 rate by less Event Requests as 2011Two alternatives to deal with deadtime Luminosity or alternative trigger configurationIf not all processing done by next fill, simply start with less buffer and process in next interfill

Interfill processing performed on all subfarms but one One subfarm sufficient for End-of-Fill calibration, and global and local tests during interfill time


R. JacobssonReminder: Readout System13

Deferred HLT Overflow Storage. . .Rate regulated system at many levels: trigger throttling by TFC6. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonIn LHCb (before starting the run)Tick the Defer HLT boxWhen deferring Runs/Files will start increasing

Deferred HLT: Data taking14


R. JacobssonDeferred HLT: Interfill processingAt End-of-fillChoose Start Deferred HLT Stops and Deallocates LHCb Allocates and Starts LHCb_Deferred

15


R. JacobssonDeferred HLT: Processing Processing the Deferred HLTLHCb_Deferred should be RUNNINGRuns/Files should decreaseProcessing bar will increaseLHCb can be used on parallel (but only 10 A farms available)

16


R. JacobssonDeferred HLT: End of processingWhen Finished (Runs/Files = 0)Choose Stop Deferred HLTStops and Deallocates LHCb_DeferredAt the moment also Stops and Deallocates LHCb Can then Re-Allocate LHCb to take all farms back

17


R. JacobssonDeferred HLT: Notes LHCb_Deferred is like any other partitionIf in ERROR, Recover, etc.

18


R. JacobssonEnd-Of-Fill CalibrationVELO and ST would like to take some special runs after each fill in global

At End-of-fill (after starting the Deferred HLT processing)Apply the End-of-fill Activity (EOF_CALIB)Switch_On the LHCb autopilot

When both the End-of-fill and the Deferred HLT Processing are done:Stop the Deferred HLTSelect the Physics ActivitySwitch_On LHCb Autopilot

Deferred HLT procedure and EOF calibration procedure to be automated after some experienceYour feedback is very welcome


R. JacobssonPreparing LHCb for Data Taking206. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. Jacobsson

LHCb Big Brother21LHC StateLHCb StateHandshakeHV/LV StateVELO ClosureBCMMagnetGlobal clockHV Configuration6. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonLHC/LHCb Operational ProcedureFill sequence for physics fill in LHCb language:

NO_BEAM(Injection Permit = FALSE, Any state of LHCb, Internal clock)

INJECTION(Injection Permit = TRUE, VELO out)

RAMP(Injection Permit = FALSE)

PHYS_ADJUST

PHYSICS (VELO in)

(ADJUST) (VELO out)

DUMP

EOF (Internal Clock, Calibrations)

All state changes require confirmation on LHCb State Control from the shifter!

22Handshake for Injection OR TI8_SetupBeam Dump Warning (Only when directly from PHYSICS)Handshake for Adjust(LHC: FLAT TOP Reset of entire DAQ (inclusion of ignored FE and switch off data generators))6. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonMode Handshake (Ex. Injection)23PROBLEMWARNINGREADYPREPARESTANDBYVETOREADYOKLHC:NO_BEAMINJECTIONLHCb:STANDBYConfirmVETOECSGET READYREADY 5 minutes timeout ConfirmRAMPVoice assistant warns repeatedly about the timeout after 3min6. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonLHC-LHCb HandshakesTI8_SETUP handshake:Allow LHC to work with transfer line for beam 2 (TI8)Only requires injection permit for beam 2 ONLY from usTED beam stopper is IN (prevents beam from entering LHC)LHCb State goes to INJECTION

INJECTION and ADJUST handshake no changes

5 min automatic timeout in place counting from LHC WARNINGIf LHCb is not yet ready, we publish PROBLEM automaticallyVoice warns beforeReact swiftly to handshake!If in PROBLEMSlow prepare Once the subsystem is ready PROBLEM switches to READY automaticallySubdetector in error As soon as error is solved or excluded (careful!!) PROBLEM switches to READY automatically You might be contacted by LHC for explanation and time estimate

BEAM_DUMP handshake (Fully automatic in LHCb)LHC issues WARNING 5min before dumping beam unconditionallyLHCb continues data taking until the beam dump (after first two weeks) At beam dump, automatically stops run, moves VELO out, and ramps down everything which should NOT participate to End-Of-Fill calibration.

A lot of mechanism and systems now depend on the LHC internal state Always make sure you confirm from the big box! (Dont use cancel!)


R. JacobssonLHCb/LHC Operational Overview25

Status of all LHCb-LHC communication (on all LHC/LHCb monitoring displays)6. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonExample ADJUST handshake 126








R. JacobssonExample ADJUST handshake 5 - PROBLEMTimeout after 5 min! Get the system ready and PROBLEM moves to READY automatically30

Timer will be added6. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonHandshake Complete ManuallyFollowing e.g. CANCEL of request to change LHCb state or ERROR31


R. JacobssonRAMP Confirmation No handshake32

Injection is automatically VETOed when Beam Mode leaves INJECTION*If not yet terminated6. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonRAMP Confirmation No handshake33

Ramp progress6. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonBEAMS LOST Confirmation34


R. JacobssonLHCb ClocksThe LHCb clock is switched automatically to the machine clocks when LHC Beam Mode changes to INJECTION.As a brutal change in the clock has consequences for electronics, it is then advisable NOT TO RUN at this time.Currently there is no automatic switch to internal clockManually SET_INTERNAL/SET_EXTERNAL may be done on LHC State Control PanelStop the run before

35


R. JacobssonLHC Clocks LostThe LHC clock may have jumps due to resynchronization between SPS and LHC.LHC Clocks only guaranteed to be stable INJECTION DUMP36


R. JacobssonHV Status and Trip RecoveryHV Status = % Operational Channels indicated on LHCb Big Brother According to preconfigured thresholds, the HV status drives the Detector Status bits in the ODIN data bank

MUON has automatic recovery mechanism of tripping channels

37


R. JacobssonTilted Crossing planeSame crossing angle at IP8 with both polaritiesRemove external horizontal crossing angle, apply vertical angleCrossing plane tilted!

38


R. JacobssonLuminosity LevelingOptimum luminosity is a function of several parameters monitored real-timeFull event readout rate (max 1 MHz)Average number of interactions per crossing (max 2.5)Max readout network through-put (max 70 GB/s)Physics trigger overall dead-time ( 3%

Automatic emergency trigger implemented including button in LHCb & CCCFaster reaction in case of excessive luminosity After run-in of new procedure, more specific SL instructions


R. Jacobsson

Luminosity Monitor41Mu per bunchSpecific luminosity per bunchRates of L0 and luminosity countersLuminosity and pileup trendsVertical separation trend6. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonLuminosity Ramp42


R. JacobssonRadiation and BackgroundHigh radiation levels should be avoided. However, some radiation is normal, and is just due to proton-proton collisions, beam gas and halo particles.

High radiation levels for a long enough time can trigger a beam dump to avoid serious damages to the detector.Background is the normal life around an accelerator but it can be unacceptably high. This affects the quality of the data. If high, it can decrease the expected life of the detectors

Guidelines for BCM:Injection: Beam losses may reach 30% of dump threshold on BCMMay happen a few times but should not be systematically above inform LHCCirculating beam: Beam losses up to 10% is already an anomalyUp to 2% in 2010 purely due to collisionsSingle spike OK, repeatedly or constant, inform LHC

In case of doubt call Run Chief or Run Coordinator

BLS and RMS guidelines to come


R. JacobssonBackground Monitor44

BCM DumpedBCM Injection Mode6. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonBCM Dump45

Post Mortem Trigger6. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. Jacobsson

Protective DumpA protective dump is always followed by transmission of Post Mortem TriggerRequests all systems to readout history buffers and send to LHC for post mortem analysis

Beam Condition MonitorRearms automatically if dump was not by itselfFSM state changes ABORT_PMT PM_READY READYDump by BCM: Spike exceeding one of the dump thresholdsNot necessarily over 1000 per mille on display depending on which logic triggeredBCM trend on Background Monitor Display blinks BCM Dumped in redFSM stays in PM_READY waiting for manual Rearm commandCall CCC to tell them we have probably dumped and that we are investigatingCall Run Chief or Run Coordinator before rearming

46


R. Jacobsson

Timing @ PHYS_ADJUSTCheck that filling scheme is OK and no ghost bunchesPhase of Beam 1 must stay within +/- 0.5nsDeltaT (difference in arrival time B1/B2 @ IP) within 100ps47Acceleration6. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonVELO ClosureSlide material from Karol Hennessy and Malcolm John486. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonVELO ClosureClosing and opening the VELO is fully automatic

Requires ONLY confirmations and your attentivenessVELO is only fully in the shadow of machine elements in garage position

You will be requested to confirm the closure action at 29mmVELO will pause on all steps and reconstruct a new beam positionStops and measures at 20, 10, 5, 1mm

The beam monitoring requires 400 vertices to reconstruct the luminous region Make sure the trigger is working and select physics events

Call VELO piquet in case of doubt or problems


R. JacobssonVELO Closure ControlMake sure you are the owner of the Closure FSMOnly one user can operate the VELO at time

Read carefully the information that is displayed in the Status Box50


R. JacobssonVELO Closing Manager51

Open/Close Position of Motion system in absolute frameAbsolute Position of beam as calculated by VELO beam monitoringAuxiliary support panelsFSM takeCriteria ChecksLog Messages6. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonVELO Closing CriteriaSeveral criteria must be fulfilled for the VELO to be allowed to leave garage position

External via hardware:STABLE BEAM and MOVABLE DEVICES ALLOWED declared by the machine

Internal via software:LHCb DAQ Required to be running in order to monitor luminous regionVELO HV High Voltage Currents should be below thresholdBCM Beam Condition Monitor background should be below thresholdBPM Beam Position Monitors should show stable orbitVTX Luminous region stable and well centered Actual list of criteria and thresholds detailed in Criteria Panel

Criteria reflected by the panel LEDsBlue = not checkedOrange = check failedGreen = OK All LEDs must be green to allow movement If a criteria fails during closure, VELO requests opening. Auto-aborts after 2 minutes

In addition you should check general beam stability:Trigger rates, luminosity and specific luminosity trendsBackground trends

52


R. JacobssonVELO Closing ConfirmationAutomatically prompted to confirm closure/next step when all criteria fulfilled

53

All OK6. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonVELO OpeningBeam Dump or beams lost Fully automatic

Criteria failed during closure or while closedRequests openingThere is no reason for stress or panic If shifter does not confirm in 2min, opens automaticallyIf criteria is fulfilled again, clears request or if already open, requests closing Thus, in most cases investigate reason for failed criteria, it may be temporary

Exception to the 2min rule: If DAQ is stopped, there are no new vertices, position monitoring uses BPM for a 15min grace period. After 20 min VELO will ask to open

Should be no reason but closing and opening is available from FSM Object54

NOT OKCountdown(2mins)6. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonVELO EmergencyIf the closing panel is frozen or the FSM is in error (VELO_DCS -> VELO_MOTION) call the VELO piquet

If the VELO will not close or conditions (LEDs) fail more than once, call the VELO piquet

Emergency Out button55


R. JacobssonData Taking566. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonConfiguring SystemAlways send the Configure command from Top!Reset command may be sent from anywhere top or local subsystem57


R. JacobssonRun ConfigurationSetting up LHCb for a run consist of choosing and configuring:Partition composition (subdetectors and common resources)ActivityFarm sizeTrigger Configuration L0+HLT TCK + ODIN TriggersData TypeData DestinationTechnical Triggers

in addition to HV/LV obviously


R. JacobssonPartition Composition59Include subdetectors and make sure all common resources are included

NOTE: Trigger components and associated subdetectors must be excluded/included together!Make sure nothing is excluded/disabled below top FSM objects!


R. JacobssonRun Configuration: ActivityActivity defines the recipe which will be applied by all subsystems on CONFIGURE

If no special recipe is required for a particular activity, subsystem applies recipe DEFAULT

Activity contains the global run settings (#subfarms, TAE, max events, steps Trigger Configuration by default, Data type by default)60


R. JacobssonRunInfoYou may check e.g. the farm and other resources you have allocated in RunInfo61


R. Jacobsson

Run Configuration: FarmIf allocation fails because insufficient farm nodes available or if you want to change size of farm

Number of farm nodes to be allocated is configured in RunInfo:Edit


R. JacobssonRun Configuration: FarmCheck the available number of subfarms in ServiceDomains:FARM63


R. JacobssonRun Configuration: DCSMake sure Detector Control System is READY !64


R. JacobssonRun Configuration: Trigger ConfigurationTrigger Configurations define L0 and an HLT setup (Trigger Configuration Key - TCK) togetherODIN triggers for luminosity and beam-gas

Trigger Configurations may be changed on the fly Fast Run Change65


R. JacobssonRun Configuration: Trigger Configuration66


R. JacobssonRun Configuration: Data TypeData Type used OFFLINE to identify data67


R. JacobssonRun Configuration: Data DestinationData Destination is normally handled automaticallySwitches to OFFLINE as soon as physics conditions established68


R. JacobssonRun Configuration: Run/File InfoOpen by clicking on RunDBUsually good praxis to check that files are opened at the beginning of fill69

Allows changing destination manually6. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonRun Configuration: Technical TriggersTechnical triggers may be added for test purposes from TFC Control panel

TFC settings may be modified after Stop Trigger in state ACTIVE70


R. JacobssonRun Configuration: Configuring SystemAlways send the Configure command from Top!Reset command may be sent from anywhere top or local subsystem71


R. JacobssonFarm Node Status72


R. JacobssonHLT Farm AvailabilityThe HLT farm now consist of 50 Subfarms, each with 27 nodes, each with either 8 or 20 Moore tasks = 24600 HLT tasks!

A lot of work has been put into improving configuration/startup speedNew forking mechanism of tasks (being commissioned)Majority Ready mechanism to Start Run as soon as 70% of farm is READYAutomatic recovery of HLT tasksHLT FSM object gets grey and message in Message Box in Run Control panelRemoving on the fly entire subfarms / individual nodes which do not figure

73

Opened by clicking on HLT in Run Control Advice: keep open at crucial moments!

Majority mechanism uses in practice removal mechanism as soon as 70% READY1 minute after Start Run, the HLT Automatic Recovery will try to include them

Manual removal Removed farms are excluded until included Mail is sent automatically to Online Team Try to get them running while run is running if all is stable by including them againHLT Automatic Recovery will try to get them to state Running


R. JacobssonAutomatic HLT Re-inclusion/Recovery74


R. JacobssonManual Removing/Including Farm Node75


R. JacobssonMonitoring, Reconstruction, Calibration FarmONLY DCS, DAQ, TFC, HLT, STORAGE needed to take dataMonitoring, Reconstruction and Calibration Farms needed to take GOOD dataMechanism to ignore temporarily these FSM objects on the fly

76


R. JacobssonRun Control - Deadtime77

Click to getbreakdown6. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonTrigger Deadtime Many sources may generate DAQ deadtimeTechnically meaning that trigger is throttled to some level

Normal sources (ODIN)FE derandomizer occupancyStandard emulatorOTIS Emulator (OT)Beetle Emulator (VELO, ST)Trigger gap due to RICHSynchronous Front-End reset+ other non-conformities

TechnicalTELL1s problem in receiving data from FE or sending data to FARMDesynchronized or incomplete events detected in Event Filter FarmData congestion in Event Filter Farm or Storage


R. JacobssonDeadtime Display79

Click to find out which subdetectoris throttling 6. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. JacobssonTELL1 trigger throttlingStatus of throttle signals sampled every 3 seconds80


R. JacobssonErrorMost problems are recovered by a (well-chosen) sequence ofRECOVER from top nodeRESET often on only system which is in error (sometimes more than one)CONFIGURE from top (sometimes more than one)

FARM (single subfarms, single nodes), MONITORING, RECONSTRUCTION, CALIBRATION FARM may be manipulated while run is taking data perfectly well!

Recovering a subdetector from error rarely requires STOP_RUN Only STOP_TRIGGER and then reset subdetector/configure from top/start_run again

Again, failing FARM nodes/SUBFARMs may be excluded(included) on the fly


R. JacobssonFSM ProblemsSurely most difficult problem is not responding FSMsMost difficult to identify May be deep down the treeNo magic, restart FSM, often starting from lowest level in the treeNOTE: After restart, the FSM object returns to UNLOCKED Take itMay also be done from the very top of Run Control if many things should be taken

If FSM is DEAD, command restart is available from FSMIf not, right-click on FSM Object:

82


R. JacobssonAutomatic RecoveriesMany automatic recoveriesSome even with AutoPilot OFF

Do not interfere while these are in progress as seen in Message box at the bottom of the Run Control

83


R. JacobssonTELL1 ProblemsTo diagnose problems with errors from TELL1 open overview panel from Run ControlBest to check the counters when the trigger is stopped84


R. JacobssonAccess HandlingAccess procedure and application. Instructions on Shifter Twiki

85


R. JacobssonAccess HandlingThe Run Chief/Run Coordinator authorizes access requestsInforms you when the access may be done, and informs the requestor and the RPE in advance

If the LHC starts an ACCESS you should inform the Run Chief

Procedure for starting accessWait for the RPE piquet to come to the Control Room The RPE piquet will go into the cavern Wait until the RPE piquet lifts the RP-VETO and tells you to call the CCC Call the CCC and ask them to put UX85 in Restricted Mode and to delegate access control to you (Access Delegation) Give access with key to RPE piquetThe RPE piquet will inform you when the RP survey is completed and access to UX85B is allowed

See https://edms.cern.ch/document/926635 for instructions on how to get delegation and to give access


R. JacobssonAccess HandlingWhen access is cleared by RPGrant access to the people listed for the intervention(s) in the authorized access requestsee https://lbtwiki.cern.ch/bin/view/RP/AccessList2010

Make sure people leave the cavern before the end of the access period

When accesses are completed:Call the CCC and asks to take back the Delegation and close UX85Inform the RPE piquet

87


R. JacobssonLevel 3 Alarm or Fire Brigade InterventionThe Fire Brigade or CCC will contact you

Make sure the CSAM alarm screen is in Vue Alarms not HistoriqueCheck the alarm and call the GLIMOS (ph. 162029, 163051)Procedures for Level 3 Alarms are in the folder by the PC.

Make sure PZ85 is OPEN

If the alarm is on the UX85A side the Fire Brigade can go down immediately

If alarm is on detector sideInform the Fire Brigade if the magnet is on and ask if they want it offThe Fire Brigade can intervene when the RP VETO in ONIf they ask to lift the VETO call the RPE piquet (ph. 71919, 162112)In case you cannot reach it or the CERN RP service (ph. 75252 during working hours/ph. 74848 outside working hours) Make sure the RAMSES panel is workingInform Run Chief and Run Coordinator


R. JacobssonPower CutDont panick Game Over

As Shift Leader, you should stay at the helm!

Find out extent of power cut Site wide, LHCb only, LHC, surface/cavern?

If it is only local and LHC has beam, find out position of VELOMove out if needed in consultation with VELO piquet

BCM and LHC interfaces will normally be on power unless LHC is cut

Try to find out state of cavern ventilation and coolingPhone CCC-Technical Infrastructure (72201) to see about alarms

Contact DSS, all sub-systems and Run Chief and Run Coordinator

Follow-up on reestablishing power 896. Shift LeaderDutiesGolden RulesCold StartFSM ConceptsNovelties 2011Fill Preparation 2011LHCb State ControlOperational ProcedureMode HandshakesInjection StateHV StatusLuminosity LevelingBackground MonitoringProtective Beam DumpTiming MonitoringVELO ClosureFarm HandlingRun ConfigurationError and ProblemsDead TimeAccess HandlingLevel 3 AlarmPower Cut

R. Jacobsson

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties 2011 6. Fill...

Documents

Transcript of 6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties 2011 6. Fill...