Coney - MPB - 31 Oct 2012 1
MICE RunningMICE Running
L. Coney – UCRL. Coney – UCR
MPB – 31 Oct 2012MPB – 31 Oct 2012
2Coney - MPB - 31 Oct 2012
OutlineOutline Current RunningCurrent Running
Shift personnel trainingShift personnel training Organization & PreparationOrganization & Preparation
Moving toward Step IV Moving toward Step IV OperationsOperations OrganizationOrganization Operational planOperational plan Online systemsOnline systems
ConclusionsConclusions
3Coney - MPB - 31 Oct 2012
Leading Into Step IVLeading Into Step IV Current running experience feeds into Step IV/Step VI Current running experience feeds into Step IV/Step VI
preparationpreparation Characterize muon beam & Understand detector Characterize muon beam & Understand detector
operationoperation Gain experience operating beam line and Hall equipmentGain experience operating beam line and Hall equipment
Current run personnelCurrent run personnel MOM (MICE Operations Manager)MOM (MICE Operations Manager)
Rolling monthly appointmentRolling monthly appointment Responsible for meeting scientific goals of MICE Responsible for meeting scientific goals of MICE Safety responsibility delegated from Project ManagerSafety responsibility delegated from Project Manager
BLOC – BeamLine On Call expertBLOC – BeamLine On Call expert Trained member of MICE collaborationTrained member of MICE collaboration
2 shifters for ~9 hour shift2 shifters for ~9 hour shift Trained member of MICE collaborationTrained member of MICE collaboration
SOC – Software On Call expert – often remoteSOC – Software On Call expert – often remote
ProceduresProcedures Run during ISIS User RunsRun during ISIS User Runs Normal working hours (plus weekends)Normal working hours (plus weekends)
PreparationPreparation
4Coney - MPB - 31 Oct 2012
Shifter TrainingShifter Training Implemented training regime (May)Implemented training regime (May)
Systematic, consistent, good coverage for all shifter Systematic, consistent, good coverage for all shifter responsibilitiesresponsibilities
Record of training kept by MOMRecord of training kept by MOM MOMs lead/arrange trainingMOMs lead/arrange training
Overview & introduction – given by MOMOverview & introduction – given by MOM Introduces responsibilities and role of ShiftersIntroduces responsibilities and role of Shifters Notification of safety training requirementsNotification of safety training requirements Description of shifter dutiesDescription of shifter duties
Practical training – ends with practical testPractical training – ends with practical test Work with the equipmentWork with the equipment List of tasks must learn how to completeList of tasks must learn how to complete Specific individuals are certified to train shifters on Specific individuals are certified to train shifters on
systemssystems PPS, Target, Beamline magnets, C&M, DAQ, Online PPS, Target, Beamline magnets, C&M, DAQ, Online
Reconstruction and Monitoring, Datamover, Reconstruction and Monitoring, Datamover, Computing, DocumentationComputing, Documentation
5Coney - MPB - 31 Oct 2012
Organization & Organization & PreparationPreparation Redmine project Redmine project
management tool management tool MicemineMicemine
Provides useful & Provides useful & necessary necessary Operations infoOperations info
Organize runningOrganize running Schedule shifters, Schedule shifters,
SOC, BLOCSOC, BLOC Develop Run PlanDevelop Run Plan
Daily to-do Daily to-do Beamline Beamline
configurationsconfigurations http://http://micewww.pp.rl.ac.micewww.pp.rl.ac.ukuk/projects/operation/projects/operations/wikis/wiki
6Coney - MPB - 31 Oct 2012
Regular RunningRegular Running Implement changesImplement changes Operate equipment long idleOperate equipment long idle Practice Practice learn learn Train shift personnelTrain shift personnel
NEED coordination of vital personnelNEED coordination of vital personnel MAUS software expert prepare control room MAUS software expert prepare control room
branch for runningbranch for running ‘‘experimenters’ and associated personnel availableexperimenters’ and associated personnel available Shake down of systems in advance Shake down of systems in advance
Software/on call experts/remote availabilitySoftware/on call experts/remote availability If responsible for a system – must be contact-ableIf responsible for a system – must be contact-able
7Coney - MPB - 31 Oct 2012
Moving Forward..Moving Forward.. Shift focus from basic functionality to Shift focus from basic functionality to reliability reliability
and efficiencyand efficiency
Continue:Continue: Pre-run preparationPre-run preparation Further simplify runningFurther simplify running
Improve:Improve: Automate proceduresAutomate procedures Use ConfigDB to full capabilityUse ConfigDB to full capability Increase Online Reconstruction capabilitiesIncrease Online Reconstruction capabilities Improve stability in Online SystemsImprove stability in Online Systems
MAUS – already using ‘release’ system of changesMAUS – already using ‘release’ system of changes C&M – Implement more formalized release for C&M codeC&M – Implement more formalized release for C&M code DAQ – staged upgrades – no major changes plannedDAQ – staged upgrades – no major changes planned
Prepare for long term running in Step IVPrepare for long term running in Step IV
8Coney - MPB - 31 Oct 2012
MICE Step IVMICE Step IV Change operational modeChange operational mode Define for rest of experiment (Step IV, Define for rest of experiment (Step IV,
Step VI)Step VI) Equipment:Equipment:
Both Spectrometer SolenoidsBoth Spectrometer Solenoids Two trackers installed in the SS magnetsTwo trackers installed in the SS magnets One AFC (Focus Coil magnet & LH2 system)One AFC (Focus Coil magnet & LH2 system)
9Coney - MPB - 31 Oct 2012
Step IV: New Step IV: New Equipment…Equipment… We will have much more in the We will have much more in the
Hall!Hall!
10Coney - MPB - 31 Oct 2012
Step IV: OperationsStep IV: Operations Moving toward Step IV runningMoving toward Step IV running
Many configurations to test/runMany configurations to test/run Long term running/every opportunity takenLong term running/every opportunity taken All ISIS cycles – every day All ISIS cycles – every day Develop idea of not-so-remote control roomDevelop idea of not-so-remote control room
Develop robust system of OperationsDevelop robust system of Operations Includes all facets of Online Group Includes all facets of Online Group
DAQ, C&M, Online Reconstruction/Analysis, DAQ, C&M, Online Reconstruction/Analysis, InfrastructureInfrastructure
Develop plan for cooling channel Develop plan for cooling channel commissioningcommissioning
Develop operational support plan with Develop operational support plan with new personnelnew personnel
11Coney - MPB - 31 Oct 2012
Step IV Operations Step IV Operations OrganizationOrganization Overall Operations ManagerOverall Operations Manager
Understands STFC safety and operational environmentUnderstands STFC safety and operational environment Provide oversight, link between MOMS, safety Provide oversight, link between MOMS, safety
responsibility from project managerresponsibility from project manager Local – full time – not necessarily STFC employeeLocal – full time – not necessarily STFC employee Position does not currently existPosition does not currently exist
Monthly MOMsMonthly MOMs Continues current role – rolling appointmentContinues current role – rolling appointment Responsible for daily experimental planning & running Responsible for daily experimental planning & running
dutiesduties On call On call for 1 month for 1 month
MICE Shift personnelMICE Shift personnel Trained MICE collaboratorsTrained MICE collaborators Two shifters for each ~8 hour shiftTwo shifters for each ~8 hour shift
Current on-call experts continueCurrent on-call experts continue BLOC (BeamLine), SOC (Software), TROC (Tracker)BLOC (BeamLine), SOC (Software), TROC (Tracker) Covers conventional magnets, detectors, and PPSCovers conventional magnets, detectors, and PPS
12Coney - MPB - 31 Oct 2012
Step IV Operations Step IV Operations OrganizationOrganization STFC organizational and safety requirement STFC organizational and safety requirement
that qualified experts are on premises when that qualified experts are on premises when hazardous operations are carried outhazardous operations are carried out
New system-specific expertsNew system-specific experts Minimum of three neededMinimum of three needed Not necessarily current MICENot necessarily current MICE Positions do not currently existPositions do not currently exist
Cryogenic supportCryogenic support Magnet system maintenance, management of Magnet system maintenance, management of
cryogens, vacuum pumping rigs, instrumentationcryogens, vacuum pumping rigs, instrumentation Superconducting magnet diagnosticsSuperconducting magnet diagnostics
Cooling & powering expert, quench behavior expertCooling & powering expert, quench behavior expert LH2 expertLH2 expert
Control engineering expertise, safety proceduresControl engineering expertise, safety procedures During LH2 system commissioning – 3 experts During LH2 system commissioning – 3 experts
rotated shiftsrotated shifts
13Coney - MPB - 31 Oct 2012
Step IV OperationsStep IV Operations Experimental program preliminary – 1 full year Experimental program preliminary – 1 full year
data-taking for Step IVdata-taking for Step IV Includes time for commissioning SS1/FC/SS2 Includes time for commissioning SS1/FC/SS2
togethertogether Not need 24-hour coverage Not need 24-hour coverage need on-call magnet expert need on-call magnet expert
Run with empty FC Run with empty FC solid absorbers solid absorbers liquid hydrogen liquid hydrogen absorberabsorber
For LH2 – For LH2 – need 24-hour supportneed 24-hour support
Given current MICE schedule for Step IV Given current MICE schedule for Step IV equipment and installation..We must be ready equipment and installation..We must be ready for 24/7 running in order to get Step IV data for 24/7 running in order to get Step IV data before ISIS shutdown in August 2014before ISIS shutdown in August 2014
Prepare:Prepare: Train shift personnelTrain shift personnel
Develop plan to staff multitude of shiftsDevelop plan to staff multitude of shifts Can our systems handle this?Can our systems handle this?
Target ok? Magnets ok?Target ok? Magnets ok? DAQ ok? Controls ok? Datamover ok?DAQ ok? Controls ok? Datamover ok?
14Coney - MPB - 31 Oct 2012
Step IV & VI – OperationsStep IV & VI – Operations Safety Safety
Understand implications under STFC rules of new Understand implications under STFC rules of new equipmentequipment
Operations review of LH2 system under wayOperations review of LH2 system under way Understand from experts what each system Understand from experts what each system
entails/requires for commissioning & operationentails/requires for commissioning & operation PPSPPS
Understand implementation of system with SC Understand implementation of system with SC magnetsmagnets
Step VI – understand implementation with RFStep VI – understand implementation with RF
Hall Access – limitedHall Access – limited Magnet commissioning period differs from runningMagnet commissioning period differs from running
Step VI – includes everything from Step IV + RFStep VI – includes everything from Step IV + RF Need full time RF engineerNeed full time RF engineer Daresbury interviewing for this positionDaresbury interviewing for this position
15Coney - MPB - 31 Oct 2012
Step IV: Operations & Step IV: Operations & OnlineOnline Next: focus Next: focus Functional, Safe/Secure, Reliable, Easy to use Functional, Safe/Secure, Reliable, Easy to use
FunctionalFunctional DAQ – incorporate all detectors – improve rate – DAQ upgradesDAQ – incorporate all detectors – improve rate – DAQ upgrades All C&M forAll C&M for each element each element Commissioned LH2 controlsCommissioned LH2 controls & monitoring& monitoring Fully commissioned SS controls systemFully commissioned SS controls system Fully commissioned FC controls system – incorporate with LH2 Fully commissioned FC controls system – incorporate with LH2
systemsystem Alarm Handler valuesAlarm Handler values set appropriately set appropriately Add capability toAdd capability to Online Reco/Analysis Online Reco/Analysis Automate run infrastructure – RunAutomate run infrastructure – Run Control Control Automate read/write to Configuration DatabaseAutomate read/write to Configuration Database Automate write to EPICS archiver Automate write to EPICS archiver
Safety/SecuritySafety/Security Incorporate fully commissioned PPSIncorporate fully commissioned PPS Implement formal shifter trainingImplement formal shifter training Update Operations/Online documentationUpdate Operations/Online documentation & instructions & instructions Develop comprehensive listDevelop comprehensive list of safety-critical maintenance of safety-critical maintenance Update safety paperworkUpdate safety paperwork – develop overall system for MICE – develop overall system for MICE
operationsoperations New target controller with all interlocks & Beam Protection New target controller with all interlocks & Beam Protection
System (BPS)System (BPS) Online system security reviewed – improvements madeOnline system security reviewed – improvements made Access limitedAccess limited to micenet to micenet
16Coney - MPB - 31 Oct 2012
Step IV: Operations & Step IV: Operations & OnlineOnline Easy to useEasy to use
Simplified Ops procedures – single key exchange ISISSimplified Ops procedures – single key exchange ISIS MICE control access to HallMICE control access to Hall Improving Online ReconstructionImproving Online Reconstruction/Data Quality plots/Data Quality plots Add Online Reco for KL/EMR/accelerator analysisAdd Online Reco for KL/EMR/accelerator analysis Run Run ControlControl Mature AlarmMature Alarm Handler Handler
ReliableReliable New OS implemented – automatic updatesNew OS implemented – automatic updates Automatic/systematic backups in placeAutomatic/systematic backups in place New DAQ and C&M computers installedNew DAQ and C&M computers installed UPS coverage for critical systemsUPS coverage for critical systems Develop “stockroom” of spares – crates/computers/etcDevelop “stockroom” of spares – crates/computers/etc Automatic write to EPICS ArchiverAutomatic write to EPICS Archiver Extensive remote monitoring of Extensive remote monitoring of
equipment/computers/hardware/environment in EPICSequipment/computers/hardware/environment in EPICS Archive of all software – Launchpad repositoryArchive of all software – Launchpad repository Implement MAUS releases forImplement MAUS releases for Online Reconstruction Online Reconstruction Implement formal releases of C&M codeImplement formal releases of C&M code
17Coney - MPB - 31 Oct 2012
ConclusionsConclusions Step I running has been good Step I running has been good
preparation/practice for futurepreparation/practice for future
Many changes in store for Step IVMany changes in store for Step IV New equipment, new procedures, long New equipment, new procedures, long
term runningterm running
Many challenges remain before ready Many challenges remain before ready for Step IV runningfor Step IV running Implementing personnel requirementsImplementing personnel requirements Staffing shiftsStaffing shifts 24/7 running – equipment & personnel24/7 running – equipment & personnel
18Coney - MPB - 31 Oct 2012
19Coney - MPB - 31 Oct 2012
Shifter TrainingShifter Training Trained collaboration members to be Trained collaboration members to be
qualified shifters (intro role) and Lead qualified shifters (intro role) and Lead Shifters (more advanced)Shifters (more advanced)
Recent running has made the need for more Recent running has made the need for more qualified/trained individuals very clearqualified/trained individuals very clear Even more true for Step IV runningEven more true for Step IV running
Difficult to do while running – difficult to Difficult to do while running – difficult to properly train w/o runningproperly train w/o running
Develop & implement training scheduleDevelop & implement training schedule Corollary – schedule training updates when new Corollary – schedule training updates when new
equipment arriveequipment arrive
Top Related