Accelerator Operations and Efficiency Presentation by Roger Erickson SLAC Operations Review SLAC,...

Accelerator Operations and Accelerator Operations and EfficiencyEfficiency

Presentation by Roger EricksonPresentation by Roger Erickson

SLAC Operations ReviewSLAC Operations ReviewSLAC, June 15-16, 2004SLAC, June 15-16, 2004

06/16/04 Accelerator Operations Review

OutlineOutline

Accelerator Operations Section Accelerator Operations Section

and Matrix Organizationand Matrix Organization Communications with Stakeholders Communications with Stakeholders MCC Control Room StaffingMCC Control Room Staffing Operator Profile and TrainingOperator Profile and Training Conduct of Operations: Conduct of Operations:

Formality of Safety ProceduresFormality of Safety Procedures Maintenance ManagementMaintenance Management ROD and Down Time PlanningROD and Down Time Planning Hardware AvailabilityHardware Availability


Accelerator Operations SectionAccelerator Operations Section

Operations Section

R. Erickson, HeadM. Stanek, Deputy

G. Stewart, Admin. Assoc.

AcceleratorSystems

Operators

P. Schuh

A. AlarconJ. AmannC. BlanchetteM. BoyesL. ChristensenM. CyterskiK. DonnellyK. GoldsmithC. HollosiZ. KevitkyN. LipkowitzL. OttsS. SchuhT. SommerJ. Warren

OperationsEngineers

M. Stanek

H. BholatS. HooblerK. KublerG. MilanovichH. SmithZ. Van HooverS. Weathersby

Maintenance

C. W. Allen

Documentation

C. Hultquist

MCC AreaManager

W. Colocho


Matrix ManagementMatrix Management

Accelerator programs are managed through a matrix organization Accelerator programs are managed through a matrix organization structure with links to:structure with links to:

Program Coordinators for PEP-II and Linac Pulsed Programs.Program Coordinators for PEP-II and Linac Pulsed Programs. Liaison persons representing each experiment.Liaison persons representing each experiment. System Physicists for each major section of the accelerator facility.System Physicists for each major section of the accelerator facility. Area Managers – one for each major section, who report to the Area Managers – one for each major section, who report to the

System Physicists.System Physicists. Accelerator physicists and engineers responsible for subsystems or Accelerator physicists and engineers responsible for subsystems or

commissioning tasks.commissioning tasks. Technical support departments with engineering and technician Technical support departments with engineering and technician

support for on-call maintenance.support for on-call maintenance. Accelerator Department Safety Office.Accelerator Department Safety Office. ES&H Departments, including Radiation Physics and OHP.ES&H Departments, including Radiation Physics and OHP.


Communications with StakeholdersCommunications with Stakeholders

8:00 meeting every day, seven days a week, in MCC conference room.8:00 meeting every day, seven days a week, in MCC conference room.• Operations report by EOIC, including run time statistics, progress in carrying out

scheduled program, descriptions of problems, etc.• Reports by representatives of major experimental user groups.• Attended by Tech Div, Accel Dept, and Operations management reps, Program

Deputy, System Physicists, Area Managers, representatives of major technical support departments, and interested users; typical attendance: 45 persons.

8:15 Maintenance meeting, five days each week.8:15 Maintenance meeting, five days each week.• Run by Operations Section maintenance managers.• Attended by supervisors of all maintenance groups, Area Managers, and interested

physicists and engineers.• Attendance: typically 45 persons.

Accelerator Plans Meeting, weekly.Accelerator Plans Meeting, weekly.• Review of operational progress, problems, and plans.• Attended by Tech Div AD and Dept Heads, Program Coordinators, Accelerator

Section Heads and ADSO.


Control Room StaffingControl Room Staffing

Program DeputyProgram Deputy• Duty rotates among Accelerator Department physicists.• Oversees program for 3 or 4 day period.• Not always present in control room.

Operations EngineerOperations Engineer• 1 EOIC per shift (from group of 7), on rotating 7-week cycle. • Responsible for safely carrying out the scheduled program.• Exempt position; directs the Accelerator Systems Operators.• Must be present if accelerator is capable of producing beam.• All have experience as accelerator systems operators.

Accelerator Systems OperatorsAccelerator Systems OperatorsTypically 3 operators per shift on rotating 5-week cycle: • Linac and sources • PEP-II rings• Secondary program, PPS control, and backup.


Typical Operator ProfileTypical Operator Profile

Typical applicant is a recent graduate with a Typical applicant is a recent graduate with a bachelor’s degree in physics .bachelor’s degree in physics .

• 15 have bachelor’s degrees• 4 have master’s degrees• 4 have a technical background and extensive SLAC

experience.

• Accelerator Systems Operator 1• 1-2 years of experience

• Accelerator Systems Operator 2• 2-5 years of experience

• Accelerator Systems Operator 3• 5+ years of experience

• Operations Engineer• 2+ years of experience


Operator Training ProgramOperator Training Program

Workbook and evaluation for each classification:Workbook and evaluation for each classification:• Accelerator Systems Operator 1• Accelerator Systems Operator 2• Accelerator Systems Operator 3• Operations Engineer

ES&H Training (including RWT-II and LRCA)ES&H Training (including RWT-II and LRCA)

Accelerator Operations Safety Procedures TrainingAccelerator Operations Safety Procedures Training

(including PPS operation, searching, interlock testing for each zone.)(including PPS operation, searching, interlock testing for each zone.)

US Particle Accelerator SchoolUS Particle Accelerator School Downtime seminars and special projectsDowntime seminars and special projects


Main Control CenterMain Control Center


Documentation HierarchyDocumentation Hierarchy

A dm in is tra tive D ocum en ts example: A d m in istra tive C h e ck lis ts

total num ber of documents: (6 )

P rog ram C on tro l Docum en tsB e a m T im e R e q ue s ts

(17)

A cce le ra to r O pe ra tions D ocum en tsO p era tio n s L ina c T u rn -on C h e ck list

(71)

S a fe ty D ocum en tsS e arch P ro ce d ures

(41)

M ain tenance D ocum en tsC a ll- in L is ts

(4 )

T ra in ing D ocum en tsA S O -2 Q ua lifica tio n W orkbo ok

(19)

D ocum en ta tion D ocum en tsA D D o cu m e n ta tion A rch ive P lan

(5 )

A cce le ra to r D epa rtm en t O pe rations D irec tives


Safety ProceduresSafety Procedures

Operators are trained to carry out certain well-defined tasks, Operators are trained to carry out certain well-defined tasks, following formally controlled procedures.following formally controlled procedures.

Examples:Examples:• Interlock Checklists• Search Procedures• Entry and Exit Procedures• Safety Inspection Checklists

Safety procedures:Safety procedures:• undergo a formal review and approval process.• are updated as conditions change.• are available on the AD Documentation web site, although the signed

originals are preserved in paper form.

Completed checklists and log books are archived.Completed checklists and log books are archived.


AcceleratorAccelerator Department Safety Department Safety DocumentsDocuments

Accelerator Operations BAS Key and Lock Procedures Hot SheetAccelerator Operations BAS Instructions Incident Response ProceduresAlarm Response Procedures Laser Safety ProceduresArcs PPS Log Linac etc Building Emergency PlanBeam Containment System Procedures Linac PPS LogBSOIC Certification Checklists Logging Requirement for Work…BSY PPS Log MCC Building Manager Checklists CID PPS Log MCC Emergency PlanDRIP/DR PPS Log MCC Key Safe CheckoutElectrical Hazard Test Procedures PEP PPS LogEmergency Notification Form PEP PPS OverviewEntry and Exit Procedures PLS Keys Procedures and LogEquipment Checkout Log Positron PPS LogESA PPS Log PPS Interlock ChecklistsFFTB PPS Log PPS Zone MapsFinal Focus PPS Log Safety Configuration ControlFire Alarm System Reference Manual Safety Inspection ChecklistsGuidelines for Work in SLAC Accelerator Housings Search ProceduresGun Test Lab Safety Procedure VVS Remote Turn-on ProcedureHigh Voltage Test Facility Safety Procedures Warning Response Procedures


Maintenance ManagementMaintenance Management

Problems that interrupt the scheduled program trigger immediate repairs Problems that interrupt the scheduled program trigger immediate repairs to restore operations.to restore operations.

24-hour on-site staff to repair:24-hour on-site staff to repair:• Mechanical utilities• Power supplies• Modulators• Cryogenics

On-call staff for:On-call staff for:• Electrical power• Magnets and other mechanical systems• RF systems• Vacuum systems• OHP

Extensive call-in lists identify first responders, alternates, and experts for Extensive call-in lists identify first responders, alternates, and experts for all systems.all systems.


Opportunistic MaintenanceOpportunistic Maintenance

Opportunistic ApproachOpportunistic Approach• Whenever possible, identified maintenance, repair, and installation jobs are pre-

planned, but deferred until an opportunity arises.

CoordinationCoordination• Daily 8:15am meeting.• Coordinated by Operations Maintenance Office.• Area Managers are advocates for their areas.

Standby Maintenance List (SML)Standby Maintenance List (SML)• Jobs that are planned and can be launched on short notice.• Provides options for EOIC to take advantage of unscheduled interruptions.

Repair Opportunity Day (ROD)Repair Opportunity Day (ROD)• Launched only when required by a driving event.• Carefully planned, and typically involve 100 or more jobs.

ARTEMIS Electronic DatabaseARTEMIS Electronic Database


ARTEMISARTEMIS

ARTEMIS (Accelerator Remedy Trouble Entry and Maintenance ARTEMIS (Accelerator Remedy Trouble Entry and Maintenance Information System) is an Oracle database with a Remedy interface, Information System) is an Oracle database with a Remedy interface, accessible over the internet by everyone involved with SLAC accessible over the internet by everyone involved with SLAC accelerator maintenance.accelerator maintenance.

Operators report problems as they occur by making entries in Operators report problems as they occur by making entries in ARTEMIS.ARTEMIS.

Repair work is launched immediately if the problem is inhibiting the Repair work is launched immediately if the problem is inhibiting the scheduled accelerator program.scheduled accelerator program.

New ARTEMIS entries are reviewed at the daily maintenance meeting. New ARTEMIS entries are reviewed at the daily maintenance meeting. Open questions are addressed and jobs are assigned.Open questions are addressed and jobs are assigned.

Jobs may be assigned to the Standby Maintenance List (SML), or to the Jobs may be assigned to the Standby Maintenance List (SML), or to the Repair Opportunity Day (ROD) list, or marked for the next long down Repair Opportunity Day (ROD) list, or marked for the next long down time.time.

Jobs are tracked using ARTEMIS until they are finally closed.Jobs are tracked using ARTEMIS until they are finally closed.


Daily Maintenance Meeting Daily Maintenance Meeting


Review of New ProblemsReview of New ProblemsSample from the June 10 maintenance meeting.


Standby Maintenance ListStandby Maintenance List


Repair Opportunity DaysRepair Opportunity Days

The accelerator systems are The accelerator systems are notnot turned off for maintenance turned off for maintenance on a routine, scheduled basis.on a routine, scheduled basis.

A ROD list of pending repair jobs is derived from the A ROD list of pending repair jobs is derived from the ARTEMIS database and maintained by the Operations ARTEMIS database and maintained by the Operations Section. Installation and PM jobs are added as needed.Section. Installation and PM jobs are added as needed.

All ROD jobs are reviewed in the daily maintenance All ROD jobs are reviewed in the daily maintenance meetings.meetings.

The ROD list can be viewed over the internet.The ROD list can be viewed over the internet. When a “driving event” occurs, a detailed plan can be When a “driving event” occurs, a detailed plan can be

assembled and launched quickly, using the ROD list.assembled and launched quickly, using the ROD list. A typical ROD day includes about 100 distinct jobs.A typical ROD day includes about 100 distinct jobs. Three ROD days so far in 2004.Three ROD days so far in 2004.


Long Down Time PlanningLong Down Time Planning

Long Down Times usually occur once each year (although the linac Long Down Times usually occur once each year (although the linac hasn’t seen a long down time for two years) and typically last two to hasn’t seen a long down time for two years) and typically last two to three months.three months.

Area Managers collect job requests from system physicists, project Area Managers collect job requests from system physicists, project engineers, maintenance groups, and the ARTEMIS database.engineers, maintenance groups, and the ARTEMIS database.

A comprehensive schedule is prepared, updated, and reviewed at A comprehensive schedule is prepared, updated, and reviewed at weekly meetings, starting about three months before the down time.weekly meetings, starting about three months before the down time.

This schedule lists each job with an assigned shop, and lists the Project This schedule lists each job with an assigned shop, and lists the Project or Group Manager, Responsible Task Person, and Area Manager. or Group Manager, Responsible Task Person, and Area Manager.

Schedule specifies time and duration of each job (with 1 day time Schedule specifies time and duration of each job (with 1 day time resolution), and number of persons required.resolution), and number of persons required.

Detailed (shift by shift) schedule of pre-run testing, safety certification, Detailed (shift by shift) schedule of pre-run testing, safety certification, and turn-on activities is prepared by the Operations Section and revised and turn-on activities is prepared by the Operations Section and revised as needed on a real-time basis.as needed on a real-time basis.


Maintenance StatisticsMaintenance Statistics

ARTEMIS entries since Jan 1989: 66,883 ARTEMIS entries since Jan 1989: 66,883 (as of June 14, 2004)(as of June 14, 2004)

Total identified problems not yet resolved: 620Total identified problems not yet resolved: 620Many of these involve non-critical problems that were observed, but went away without Many of these involve non-critical problems that were observed, but went away without clear explanation; e.g., a flow switch trip or a power supply found to be out of tolerance.clear explanation; e.g., a flow switch trip or a power supply found to be out of tolerance.

ARTEMIS entries since September 1, 2003:ARTEMIS entries since September 1, 2003:• 2306 new problems (8.0 per day).• 339 problems resulted in lost PEP-II run time.• 2222 problems closed (7.7 per day).

Active jobs:Active jobs:• SML: 83• Next ROD: 89• Aug-Sept down time: 109


Accelerator Performance StatisticsAccelerator Performance Statistics

Revealed failures: malfunctions resulting in lost beam time. Also called “events”.Revealed failures: malfunctions resulting in lost beam time. Also called “events”.

Unscheduled down time:Unscheduled down time: hours lost from scheduled program due to malfunctions.hours lost from scheduled program due to malfunctions.

Mean Time to Fail:Mean Time to Fail:

MTTF = MTTF = Scheduled beam timeScheduled beam timeEventsEvents

Mean Time to Repair:Mean Time to Repair:

MTTR = MTTR = Unscheduled down timeUnscheduled down timeEventsEvents

Availability = 1 - Availability = 1 - Unscheduled down timeUnscheduled down timeScheduled beam timeScheduled beam time

NOTE:NOTE: PEP-II aborts are not counted as downtime, unless the event is reported; PEP-II aborts are not counted as downtime, unless the event is reported; i.e., unless we stop to fix something and make a database entry. i.e., unless we stop to fix something and make a database entry.


Unscheduled Down StatisticsUnscheduled Down Statistics

Jan 10, 2000 – June 29, 2003Jan 10, 2000 – June 29, 2003


Down Time Analysis DetailsDown Time Analysis DetailsExample: PEP-II systems, Run 3Example: PEP-II systems, Run 3


Availability, MTTF, and MTTRAvailability, MTTF, and MTTR


PEP-II Time Accounting by QuarterPEP-II Time Accounting by Quarter

Start of Trickle Charging

Accelerator Operations and Efficiency Presentation by Roger Erickson SLAC Operations Review SLAC,...

Documents

Transcript of Accelerator Operations and Efficiency Presentation by Roger Erickson SLAC Operations Review SLAC,...