1 MICE Running L. Coney – UCR MPB – 31 Oct 2012. 2Coney - MPB - 31 Oct 2012Outline Current...

Coney - MPB - 31 Oct 2012 1

MICE RunningMICE Running

L. Coney – UCRL. Coney – UCR

MPB – 31 Oct 2012MPB – 31 Oct 2012

2Coney - MPB - 31 Oct 2012

OutlineOutline Current RunningCurrent Running

Shift personnel trainingShift personnel training Organization & PreparationOrganization & Preparation

Moving toward Step IV Moving toward Step IV OperationsOperations OrganizationOrganization Operational planOperational plan Online systemsOnline systems

ConclusionsConclusions


Leading Into Step IVLeading Into Step IV Current running experience feeds into Step IV/Step VI Current running experience feeds into Step IV/Step VI

preparationpreparation Characterize muon beam & Understand detector Characterize muon beam & Understand detector

operationoperation Gain experience operating beam line and Hall equipmentGain experience operating beam line and Hall equipment

Current run personnelCurrent run personnel MOM (MICE Operations Manager)MOM (MICE Operations Manager)

Rolling monthly appointmentRolling monthly appointment Responsible for meeting scientific goals of MICE Responsible for meeting scientific goals of MICE Safety responsibility delegated from Project ManagerSafety responsibility delegated from Project Manager

BLOC – BeamLine On Call expertBLOC – BeamLine On Call expert Trained member of MICE collaborationTrained member of MICE collaboration

2 shifters for ~9 hour shift2 shifters for ~9 hour shift Trained member of MICE collaborationTrained member of MICE collaboration

SOC – Software On Call expert – often remoteSOC – Software On Call expert – often remote

ProceduresProcedures Run during ISIS User RunsRun during ISIS User Runs Normal working hours (plus weekends)Normal working hours (plus weekends)

PreparationPreparation


Shifter TrainingShifter Training Implemented training regime (May)Implemented training regime (May)

Systematic, consistent, good coverage for all shifter Systematic, consistent, good coverage for all shifter responsibilitiesresponsibilities

Record of training kept by MOMRecord of training kept by MOM MOMs lead/arrange trainingMOMs lead/arrange training

Overview & introduction – given by MOMOverview & introduction – given by MOM Introduces responsibilities and role of ShiftersIntroduces responsibilities and role of Shifters Notification of safety training requirementsNotification of safety training requirements Description of shifter dutiesDescription of shifter duties

Practical training – ends with practical testPractical training – ends with practical test Work with the equipmentWork with the equipment List of tasks must learn how to completeList of tasks must learn how to complete Specific individuals are certified to train shifters on Specific individuals are certified to train shifters on

systemssystems PPS, Target, Beamline magnets, C&M, DAQ, Online PPS, Target, Beamline magnets, C&M, DAQ, Online

Reconstruction and Monitoring, Datamover, Reconstruction and Monitoring, Datamover, Computing, DocumentationComputing, Documentation


Organization & Organization & PreparationPreparation Redmine project Redmine project

management tool management tool MicemineMicemine

Provides useful & Provides useful & necessary necessary Operations infoOperations info

Organize runningOrganize running Schedule shifters, Schedule shifters,

SOC, BLOCSOC, BLOC Develop Run PlanDevelop Run Plan

Daily to-do Daily to-do Beamline Beamline

configurationsconfigurations http://http://micewww.pp.rl.ac.micewww.pp.rl.ac.ukuk/projects/operation/projects/operations/wikis/wiki


Regular RunningRegular Running Implement changesImplement changes Operate equipment long idleOperate equipment long idle Practice Practice learn learn Train shift personnelTrain shift personnel

NEED coordination of vital personnelNEED coordination of vital personnel MAUS software expert prepare control room MAUS software expert prepare control room

branch for runningbranch for running ‘‘experimenters’ and associated personnel availableexperimenters’ and associated personnel available Shake down of systems in advance Shake down of systems in advance

Software/on call experts/remote availabilitySoftware/on call experts/remote availability If responsible for a system – must be contact-ableIf responsible for a system – must be contact-able


Moving Forward..Moving Forward.. Shift focus from basic functionality to Shift focus from basic functionality to reliability reliability

and efficiencyand efficiency

Continue:Continue: Pre-run preparationPre-run preparation Further simplify runningFurther simplify running

Improve:Improve: Automate proceduresAutomate procedures Use ConfigDB to full capabilityUse ConfigDB to full capability Increase Online Reconstruction capabilitiesIncrease Online Reconstruction capabilities Improve stability in Online SystemsImprove stability in Online Systems

MAUS – already using ‘release’ system of changesMAUS – already using ‘release’ system of changes C&M – Implement more formalized release for C&M codeC&M – Implement more formalized release for C&M code DAQ – staged upgrades – no major changes plannedDAQ – staged upgrades – no major changes planned

Prepare for long term running in Step IVPrepare for long term running in Step IV


MICE Step IVMICE Step IV Change operational modeChange operational mode Define for rest of experiment (Step IV, Define for rest of experiment (Step IV,

Step VI)Step VI) Equipment:Equipment:

Both Spectrometer SolenoidsBoth Spectrometer Solenoids Two trackers installed in the SS magnetsTwo trackers installed in the SS magnets One AFC (Focus Coil magnet & LH2 system)One AFC (Focus Coil magnet & LH2 system)


Step IV: New Step IV: New Equipment…Equipment… We will have much more in the We will have much more in the

Hall!Hall!


Step IV: OperationsStep IV: Operations Moving toward Step IV runningMoving toward Step IV running

Many configurations to test/runMany configurations to test/run Long term running/every opportunity takenLong term running/every opportunity taken All ISIS cycles – every day All ISIS cycles – every day Develop idea of not-so-remote control roomDevelop idea of not-so-remote control room

Develop robust system of OperationsDevelop robust system of Operations Includes all facets of Online Group Includes all facets of Online Group

DAQ, C&M, Online Reconstruction/Analysis, DAQ, C&M, Online Reconstruction/Analysis, InfrastructureInfrastructure

Develop plan for cooling channel Develop plan for cooling channel commissioningcommissioning

Develop operational support plan with Develop operational support plan with new personnelnew personnel


Step IV Operations Step IV Operations OrganizationOrganization Overall Operations ManagerOverall Operations Manager

Understands STFC safety and operational environmentUnderstands STFC safety and operational environment Provide oversight, link between MOMS, safety Provide oversight, link between MOMS, safety

responsibility from project managerresponsibility from project manager Local – full time – not necessarily STFC employeeLocal – full time – not necessarily STFC employee Position does not currently existPosition does not currently exist

Monthly MOMsMonthly MOMs Continues current role – rolling appointmentContinues current role – rolling appointment Responsible for daily experimental planning & running Responsible for daily experimental planning & running

dutiesduties On call On call for 1 month for 1 month

MICE Shift personnelMICE Shift personnel Trained MICE collaboratorsTrained MICE collaborators Two shifters for each ~8 hour shiftTwo shifters for each ~8 hour shift

Current on-call experts continueCurrent on-call experts continue BLOC (BeamLine), SOC (Software), TROC (Tracker)BLOC (BeamLine), SOC (Software), TROC (Tracker) Covers conventional magnets, detectors, and PPSCovers conventional magnets, detectors, and PPS


Step IV Operations Step IV Operations OrganizationOrganization STFC organizational and safety requirement STFC organizational and safety requirement

that qualified experts are on premises when that qualified experts are on premises when hazardous operations are carried outhazardous operations are carried out

New system-specific expertsNew system-specific experts Minimum of three neededMinimum of three needed Not necessarily current MICENot necessarily current MICE Positions do not currently existPositions do not currently exist

Cryogenic supportCryogenic support Magnet system maintenance, management of Magnet system maintenance, management of

cryogens, vacuum pumping rigs, instrumentationcryogens, vacuum pumping rigs, instrumentation Superconducting magnet diagnosticsSuperconducting magnet diagnostics

Cooling & powering expert, quench behavior expertCooling & powering expert, quench behavior expert LH2 expertLH2 expert

Control engineering expertise, safety proceduresControl engineering expertise, safety procedures During LH2 system commissioning – 3 experts During LH2 system commissioning – 3 experts

rotated shiftsrotated shifts


Step IV OperationsStep IV Operations Experimental program preliminary – 1 full year Experimental program preliminary – 1 full year

data-taking for Step IVdata-taking for Step IV Includes time for commissioning SS1/FC/SS2 Includes time for commissioning SS1/FC/SS2

togethertogether Not need 24-hour coverage Not need 24-hour coverage need on-call magnet expert need on-call magnet expert

Run with empty FC Run with empty FC solid absorbers solid absorbers liquid hydrogen liquid hydrogen absorberabsorber

For LH2 – For LH2 – need 24-hour supportneed 24-hour support

Given current MICE schedule for Step IV Given current MICE schedule for Step IV equipment and installation..We must be ready equipment and installation..We must be ready for 24/7 running in order to get Step IV data for 24/7 running in order to get Step IV data before ISIS shutdown in August 2014before ISIS shutdown in August 2014

Prepare:Prepare: Train shift personnelTrain shift personnel

Develop plan to staff multitude of shiftsDevelop plan to staff multitude of shifts Can our systems handle this?Can our systems handle this?

Target ok? Magnets ok?Target ok? Magnets ok? DAQ ok? Controls ok? Datamover ok?DAQ ok? Controls ok? Datamover ok?


Step IV & VI – OperationsStep IV & VI – Operations Safety Safety

Understand implications under STFC rules of new Understand implications under STFC rules of new equipmentequipment

Operations review of LH2 system under wayOperations review of LH2 system under way Understand from experts what each system Understand from experts what each system

entails/requires for commissioning & operationentails/requires for commissioning & operation PPSPPS

Understand implementation of system with SC Understand implementation of system with SC magnetsmagnets

Step VI – understand implementation with RFStep VI – understand implementation with RF

Hall Access – limitedHall Access – limited Magnet commissioning period differs from runningMagnet commissioning period differs from running

Step VI – includes everything from Step IV + RFStep VI – includes everything from Step IV + RF Need full time RF engineerNeed full time RF engineer Daresbury interviewing for this positionDaresbury interviewing for this position


Step IV: Operations & Step IV: Operations & OnlineOnline Next: focus Next: focus Functional, Safe/Secure, Reliable, Easy to use Functional, Safe/Secure, Reliable, Easy to use

FunctionalFunctional DAQ – incorporate all detectors – improve rate – DAQ upgradesDAQ – incorporate all detectors – improve rate – DAQ upgrades All C&M forAll C&M for each element each element Commissioned LH2 controlsCommissioned LH2 controls & monitoring& monitoring Fully commissioned SS controls systemFully commissioned SS controls system Fully commissioned FC controls system – incorporate with LH2 Fully commissioned FC controls system – incorporate with LH2

systemsystem Alarm Handler valuesAlarm Handler values set appropriately set appropriately Add capability toAdd capability to Online Reco/Analysis Online Reco/Analysis Automate run infrastructure – RunAutomate run infrastructure – Run Control Control Automate read/write to Configuration DatabaseAutomate read/write to Configuration Database Automate write to EPICS archiver Automate write to EPICS archiver

Safety/SecuritySafety/Security Incorporate fully commissioned PPSIncorporate fully commissioned PPS Implement formal shifter trainingImplement formal shifter training Update Operations/Online documentationUpdate Operations/Online documentation & instructions & instructions Develop comprehensive listDevelop comprehensive list of safety-critical maintenance of safety-critical maintenance Update safety paperworkUpdate safety paperwork – develop overall system for MICE – develop overall system for MICE

operationsoperations New target controller with all interlocks & Beam Protection New target controller with all interlocks & Beam Protection

System (BPS)System (BPS) Online system security reviewed – improvements madeOnline system security reviewed – improvements made Access limitedAccess limited to micenet to micenet


Step IV: Operations & Step IV: Operations & OnlineOnline Easy to useEasy to use

Simplified Ops procedures – single key exchange ISISSimplified Ops procedures – single key exchange ISIS MICE control access to HallMICE control access to Hall Improving Online ReconstructionImproving Online Reconstruction/Data Quality plots/Data Quality plots Add Online Reco for KL/EMR/accelerator analysisAdd Online Reco for KL/EMR/accelerator analysis Run Run ControlControl Mature AlarmMature Alarm Handler Handler

ReliableReliable New OS implemented – automatic updatesNew OS implemented – automatic updates Automatic/systematic backups in placeAutomatic/systematic backups in place New DAQ and C&M computers installedNew DAQ and C&M computers installed UPS coverage for critical systemsUPS coverage for critical systems Develop “stockroom” of spares – crates/computers/etcDevelop “stockroom” of spares – crates/computers/etc Automatic write to EPICS ArchiverAutomatic write to EPICS Archiver Extensive remote monitoring of Extensive remote monitoring of

equipment/computers/hardware/environment in EPICSequipment/computers/hardware/environment in EPICS Archive of all software – Launchpad repositoryArchive of all software – Launchpad repository Implement MAUS releases forImplement MAUS releases for Online Reconstruction Online Reconstruction Implement formal releases of C&M codeImplement formal releases of C&M code


ConclusionsConclusions Step I running has been good Step I running has been good

preparation/practice for futurepreparation/practice for future

Many changes in store for Step IVMany changes in store for Step IV New equipment, new procedures, long New equipment, new procedures, long

term runningterm running

Many challenges remain before ready Many challenges remain before ready for Step IV runningfor Step IV running Implementing personnel requirementsImplementing personnel requirements Staffing shiftsStaffing shifts 24/7 running – equipment & personnel24/7 running – equipment & personnel


Shifter TrainingShifter Training Trained collaboration members to be Trained collaboration members to be

qualified shifters (intro role) and Lead qualified shifters (intro role) and Lead Shifters (more advanced)Shifters (more advanced)

Recent running has made the need for more Recent running has made the need for more qualified/trained individuals very clearqualified/trained individuals very clear Even more true for Step IV runningEven more true for Step IV running

Difficult to do while running – difficult to Difficult to do while running – difficult to properly train w/o runningproperly train w/o running

Develop & implement training scheduleDevelop & implement training schedule Corollary – schedule training updates when new Corollary – schedule training updates when new

equipment arriveequipment arrive

1 MICE Running L. Coney – UCR MPB – 31 Oct 2012. 2Coney - MPB - 31 Oct 2012Outline Current...

Documents

Transcript of 1 MICE Running L. Coney – UCR MPB – 31 Oct 2012. 2Coney - MPB - 31 Oct 2012Outline Current...