Designing a HEP Experiment Control System,
Lessons to be Learned From 10 Years Evolution and Operation of
the DELPHI Experiment.
André Augustinus8 February 2000
CHEP 2000 8 February 2000 2
Overview
The DELPHI experiment control system
The evolution of the DELPHI ECS
Observations and lessons learned from over 10 years of operation
Conclusions
CHEP 2000 8 February 2000 3
What is an ECS ?
What is an Experiment Control System ? All systems controlling or monitoring
(parts of) an experiment ‘Classical’ controls
Power supplies, temperatures Data-flow controls
Start/stop of a run Interface with external systems
Accelerator, safety system
CHEP 2000 8 February 2000 4
The DELPHI ECS
What is the DELPHI Experiment Control System ?
Two main components: Slow Controls system (SC) Data Acquisition System (DAS)
Other components: Trigger system Communication with LEP accelerator Data quality monitoring
CHEP 2000 8 February 2000 5
The DELPHI ECS (SC)
Slow Controls system Several 1000 channels Power supplies, temperatures, pressures Independent per sub-detector
G64/G96 crates (~100), PLCM6809, M68340 processorsOS9, RPC/OSI, TCP/IP
CHEP 2000 8 February 2000 6
The DELPHI ECS (SC)
The ECS is used to Prepare sub-detectors for data taking Report (and correct) anomalies Integrate ancillary systems
Gas, magnet, safety Store detector status on database(s) Control & monitor the hardware
CHEP 2000 8 February 2000 7
The DELPHI ECS (DAS)
Data Acquisition System: Over 250 000 channels 20 partitions + central partition
Can run independentlyFastbus (~180), OS9, TCP/IP
The ECS is used to Configure the readout Initialise the partitions Control and monitor the data flow
CHEP 2000 8 February 2000 8
The DELPHI ECS (Other)
Trigger system Decision and timing Hierarchical: local and central
ECS is used to Configure and initialise the trigger
system Download look-up tables Monitoring of counters
CHEP 2000 8 February 2000 9
The DELPHI ECS (Other)
LEP communications Bi-directional exchange of experiment
and LEP machine parameters Luminosity, background, machine settings
Data quality monitoring Several 100 histograms Event processing farm
CHEP 2000 8 February 2000 10
The DELPHI ECS (Software)
Two main packages:SMI++
Models the behaviour of a system as a FiniteState Machine in a dedicated language (SML) Objects
Associated: represent concrete entitiesAbstract: behaviour defined in SMLObjects are in a definite stateObjects can receive action requestsLogically related objects are grouped in domains
CHEP 2000 8 February 2000 11
The DELPHI ECS (Software)
User interface State of objects are presented to the
operator Commands are given by sending action
requests
DIM Publish/Subscribe paradigm Universal data exchange package
Refer to previous talk by Clara Gaspar
CHEP 2000 8 February 2000 12
The Evolution of the ECS
Started taking data in 1989 No ‘real’ ECS yet Only a rudimentary version of SMI Line mode interfaces ‘Human’ synchronisation
CHEP 2000 8 February 2000 13
The Evolution of the ECS
Taking advantage of SMI (1990-1991) Completion of SC and DAS domains Centralised control of the experiment Abstraction
Variety of hardware can be represented by one type of object
Logically related objects can be summarised in one single object
Uniformity across sub-detectors
CHEP 2000 8 February 2000 14
The Evolution of the ECS
Introduction of DIM (1993 onward) Make use of user interfaces more flexible Solve communication problems inside SMI Evolved to a universal data exchange
package in the experiment DIM really started an integrated ECS :
Trigger, LEP communications became integrated part of ECS
CHEP 2000 8 February 2000 15
The Evolution of the ECS
Reengineering of SMI (1996) Improve maintainability and portability Using OO techniques Smooth transition because of well defined
interfaces (already in design phase) Hardware and Software upgrades
New sub-detectors New technologies New versions of operating systems
CHEP 2000 8 February 2000 16
Automation in the ECS
Automation is unavoidable and imperative for efficient running of a complex experiment Too complex for a non-expert operator Gain in time, efficiency Ensure consistency
Relatively easy to implement because of the use of SMI in all domains
CHEP 2000 8 February 2000 17
Automation in the ECS
Examples of automation Trip recovery
Automatic reaction to trips DAS auto-pilot
Automatic reaction to a variety of anomalies SMI analyser
Analyse combination of SMI states ‘Big Brother’
Interconnection of various domains ‘Hands-Free’ running of the experiment
CHEP 2000 8 February 2000 18
Future ECS
LEP experiments were probably the first generation that needed an ECS
One should take advantage of the expertise gained in running these big experiments
CHEP 2000 8 February 2000 19
Future ECS
Partitioning Well thought out to allow stand-alone
running (debugging, calibration) Unhindered operation when part of the
experiment is off Central control
Small crew of operators Well structured commands
CHEP 2000 8 February 2000 20
Future ECS
Uniformity across ECS subsystems Will ease integration and automation Will reduce maintenance efforts
Uniformity across sub-detectors Common hardware and software
reduced costs and maintenance effort reduced development efforts easier operation
Use of commercial solutions
CHEP 2000 8 February 2000 21
Future ECS
Central support team Strong central support is a great benefit Provide guidelines and frameworks
enforce uniformity Provide common solutions for common
problems Will ease maintenance over lifetime
CHEP 2000 8 February 2000 22
Future ECS
Flexibility ECS is never ‘finished’ Many changes will happen over the lifetime
of an experiment, the ECS should cope with Modification or addition of a sub-detector Upgrades with new technology Change of working points or operational
procedures Easy to modify or reconfigure
Good documentation
CHEP 2000 8 February 2000 23
Future ECS
Efficient day-to-day operation Abstraction
Non-expert operators can run the experiment Hide detailed information Uniform representation and commands
Automation Automatic error recovery Automate ‘standard operations’
Proper training and adequate documentation
CHEP 2000 8 February 2000 24
Summary
Designing an ECS Strong central support Partitioning Uniformity Flexibility Abstraction Automation
Top Related