Alarm Training Making the transition from alh to CSS Alarm Tools
-
Upload
hammett-livingston -
Category
Documents
-
view
28 -
download
1
description
Transcript of Alarm Training Making the transition from alh to CSS Alarm Tools
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27 2010
Alarm Training
Making the transition from alh to CSS Alarm Tools
EPICS Collaboration Meeting Aix-en-ProvenceJune 1st , 2010
Matthias Clausen - DESY
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 2
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Alarm Training
• Record Processing• Monitor Types• Setting Alarms• Alarm Handling
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 3
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
EPICS Record Processing
P T
EPICS (Experimental Physics and Industrial Control System)
I/O System(Field bus)
I/O Controller(control loops, Sequence programs)
Client(Configuration- and Operator Interfaces)
Channel AccessIP based Communication Protocol
I/O Controller is independent from clients
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 4
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
EPICS Information exchange
P T
Client initiating basic requstes: get/ set/ monitor
Whenever Displa/ Archive/ Alarm-Deadband/ Status changes -> check whether clients have been registered.If true: -> send updated data
ClientRegister for Changes•Display-Deadband•Archive-Deadband•Alarm-Status Sending request via
Channel Access
Register Client for chanegs in specified record for Alarm changes
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 5
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
EPICS default: focus on ‘requesting’ data-> not ‘expecting’ data
P T
Client initiating basic requstes: get/ set/ monitor
get/
set
register
for
monitors
New/ Changed
Alarm Limits
Alarm/Status
How to stay ‘in synch’ with alarm deadbands/ status in IOC?
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 6
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Introducing a Message based Protocol: JMS(keeping Channel Access for ‘basic’ monitors)
P T
Basic: get/ set/ monitor
get/
set
register
for
monitors
New/ Changed
Alarm Limits
Alarm/Status
Accepting any alarms from IOC
InterConnectionServer
JMS
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 7
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
JMS: For Alarms and all Log-based Messages
P T
Basic: get/ set/ monitor
get/
set
register
for
monitors
New/ Changed
Alarm Limits
Alarm/Status
InterConnectionServer
JMS
SysLog-Messages
PutLog-Messages
SNL-Log-Messages
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 8
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Existing Log Server Implementations
P T
Basic: get/ set/ monitor
get/
set
register
for
monitors
PutLog-MessagesSysLog-Messages
Sys-LogMessage
Server
Put-LogMessage
Server
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 9
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
JMS does not add more levels of complexity(just new ones …)
P T
Basic: get/ set/ monitor
get/
set
register
for
monitors
New/ Changed
Alarm Limits
Alarm/Status
InterConnectionServer
JMS
SysLog-Messages
PutLog-Messages
SNL-Log-Messages
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 10
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
How to implement the new Logging Mechanism
• Using the existing Alarm Hook in Record Processing
• Using the existing Hooks in System Logging• Using the existing Hooks in (ca)Put Logging• Using the new Function in SNL Logging
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 11
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Alarm Logging
Changes in iocCore for Alarm Logging• very small hook in recGbl recGblResetAlarms/* Hook Routines */RECGBL_ALARM_HOOK_ROUTINE recGblAlarmHook = NULL;.......
• loading and starting the alarm logging software redefines recGblAlarmHook....... if(sevr!=nsev || stat!=nsta) { /* HOOK for alarm logging */ if (recGblAlarmHook) (*recGblAlarmHook)(pdbc, stat, sevr);.......
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 12
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Alarm Logging
Hook Routine
Control task
SCAN context
Message tasks
Port B Port A
The special queue has sequential input ....
and parallel access if space is rare and the record is stored in the overflow area.
Beacon task
add
replace
implenented as ring buffer
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 13
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
asTrapWriteRegisterListener(caPutLog2Array)
caPutLog2Array (...) (caller context)
caPutLog2Q (...) (interrupt context)
caPutLogTask ()
EPICS
copy into Queue
register Hook Routine for caPut operation
write caPut-Message into Array
timeout
Channel-Access-Put-Logging
index queue
send Message to Network
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 14
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
SNL Log
‘Just’ a new supported function in the snl executive.
Code example:
when(TV001zu){
snlMsgLog( SEVR_INFO, "<Start> TV001 has been closed");
} state Auf50
Sent to the log table in CSS.
Different severy types can be set -> used to color-code the messages
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 15
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
EPICS
IOC-Logging Overview
Beacon
Alarms caPut SysLog SNL-Log
AlarmAlarm caPutLog sysMsg snlMsg
Command Reply
ioc log control
ICSICSICS
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 16
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Alarm Table
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 17
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Looking back into the Archive (24h)
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 18
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Selecting the Time Span
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 19
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Any Message in the System
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 20
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Log Messages (System, Put, SNL)
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 21
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Properties View (default Eclipse view)
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 22
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Alarm System - Alarm Tree view
• Shows the current status of the alarm state from the IOC• Delete and create records and subcomponents by context menu• Configuration-Changes are stored in the LDAP server (or in XML file)• Alarm status is propagated to root component• Property view to display and edit tree items
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 23
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Demo Alarm Views
• Changing / setting alarm Limits• Severity Settings• Different kind of Displays
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 24
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
How to get there without JMS?
The internal Alarm Model in CSS (DESY) has been modified to accept Alarm-Messages from two different sources:
- JMS- DAL (Channel Access)(currently only available in the
development environmentforeseen for the next release)
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 25
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
How to display an Alarm Tree without LDAP?
• Use the new alternative to configure your LDAP Tree from an XML file– Create XML file by Hand– Create XML file from DCT (next release)– Create XML file from ‘your’ database creation tool– Use LDAP Tree to manually configure your tree and
save your configuration from tree ‘save as’
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 26
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
How to integrate IOC’s without iocLogClient-Support?
A new headless application will handle this:• Reading an XML config file• Use channel access to register for Alarm
Changes• Send alarm changes to JMS
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 27
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
The transition Plan:
• Start using the ‘DAL’ option for Alarm Tables and the Alarm Tree– Get your configuration file e.g. from DCT– Get the hierarchy visualized in the Alarm Tree– Get all configured alarms in the Alarm Table
• Install JMS Servers– Immediately get your CSS log messages into you CSS Alarm views– Use dal2jms to send Alarm State changes to JMS– Install iocLogClient-Support on your IOCs
• Install interConnection-Server(s)• Get ALL your messages (alarm and log) from your IOCs
– Install jms2Ora (rdb) to store your messages
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 28
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
CSS Instance
Acknowledge
Alarm message
Alarm System - Acknowledgement
Ack.
Message
(JMS)
Update
Persistant Store (LDAP)
Persistant Store (LDAP)
JMS Server
Ack
Ack
Ack
Ack
CSS InstanceCSS InstanceCSS InstanceCSS Instance
Ack
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 29
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
CSS Alarm
Configu-
rator
Alarm System - Alarm Management System (AMS)
DBDB
Filter
ManagerFilter
SMS
JMS
Read
configuration
Action
Alarm
Message
(JMS)
Write
Configuration
SMS Connector Voice Mail Connector Mail Connector
JMS
JMS
Voice
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 30
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Alarm Management System
Slides from Markus Möller
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 31
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
AMS - Overview
ActiveMQJMS-Server
DepartmentDecision(FilterManager)
MessageMinder
Distributor
Derby Db
Filter-Konfiguration
VoicemailConnector
EMailConnector
SmsConnector
JmsConnector
ALARM
T
T_AMS_MESSAGEMINDER
T
T_AMS_DISTRIBUTE
T
T_AMS_CON_SMS
T
T_AMS_CON_MAIL
T
T_AMS_CON_JMS
T
T_AMS_CON_VOICEMAIL
T
Derby Db
Filter-Konfiguration
ActiveMQJMS-Server
MessageSources
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 32
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
AMS - Applications
• DepartmentDecision (FilterManager)
Compares the incoming message with the filter configuration
• MessageMinder
Reduces message storms
• Distributor
Creates the alarm messages and sends them to the responsible Connector
• Connector
Sends the message to the user via
SMS
telephone call
JMS message producer to a user defined JMS topic
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 33
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
User – Configuration (GUI)
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 34
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
User – Configuration
• Name
- The name of the user
• Rubrik (rubric, category)
- Name of the configuration rubric
(just for display puposes)
- E-Mail address of the user
• SMS number
- Number of the user‘s cell phone
• Voicemail number
- Number that is used for the telephone call
• Prefered alarming type
• Status code
- To deactivate a user or a group
• Confirm code
- To confirm a alarm chain
• User is active
- Only active users receive a alarm message
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 35
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Group – Configuration (GUI)
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 36
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Group – Configuration
• Name
- The name of the group
• Rubrik (rubric, category)
- Name of the configuration rubric
(just for display puposes)
• Minimale Anzahl aktiver Mitglieder
- Minimum number of active users in this group
• Wartezeit bis Rückmeldung
- Number of seconds to wait for an alarm chain
answer
• Alarmgruppe aktiv
- Only the member of an active group receive
messages
• List of group members
- Every member of a group can be set active
or not active
- Only active member of a group receive
messages
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 37
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Condition – Configuration (GUI)
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 38
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Condition – Configuration
• Name
- The name of the condition
• Rubrik (rubric, category)
- Name of the configuration rubric
(just for display puposes)
• Description
- Description for this condition
• Filtertype
- Type of this filter
(String, String Array, PV, Time based)
• Compare Key
- Name of the message property that is
compared by this condition
• Operator
- Type of operator that is used to compare the
value
• Compare Value
- The value that is used for comparison
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 39
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Filter – Configuration (GUI)
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 40
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Filter – Configuration
• Name
- The name of the condition
• Rubrik (rubric, category)
- Name of the configuration rubric
(just for display puposes)
• Templates
- Message template (not used yet)
• Default Message
- Alarm message that will be send to the users
• List of receivers
- Name of the receiver
- Type of action (SMS, Mail, Voicemail, JMS)
• Operator
- Type of operator that is used to compare the
value
• Compare Value
- The value that is used for comparison
• Filterconditions
- The configuration / combination of the conditions
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 41
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Questions?
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 42
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Active ICS Connection OverviewICS: InterConnection Server (between IOC and JMS)
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 43
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Disabling Messages (for Testing/ Reboot …)
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 44
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 45
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Alarm / Log message
Sources
Alarm system - Structure
EPICS IOC D3 PCM CSS Instance
JMS ServerPersistent
Store (LDAP)Persistent
Store (LDAP)Archive DBArchive DB
CSS Alarm
Tools
(Views, Con-
figuration, …)
Message
Table
Message
Archive
Alarm Management
System
AMS
ConfigurationAlarm Tree
SM
S
Mai
l
Updated from IC
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 46
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
JMS Communication
Message sources
Special Format
Other Sources
• EPICS IOC and D3 PCM send alarm messages in special format
• Interconnection Server (EPICS) and D3 Alarms (D3) translates alarm messages in JMS format
• CSS uses log4j and sends log messages in JMS format
• Generic message system for alarm messages
• Easy to add other sources
EPICS IOC D3 PCM
CSS Instance
InterconnectionServer
D3 AlarmReader
Special Format
JMS Server(Active MQ)
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 47
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Alarm System - AMS Filter
Filter:• Checks if the filter matches• Creates a new message with the
relevant information of the alarm message
• Forwards the message to an actionFilter condition:• A Filter is a combination of filter
conditions• Filter conditions can be connected
with AND and OR• Available condition types are:
Compare strings, Check current PV, Time based condition, …
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 48
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Alarm System - AMS operators and groups
Operators:• Receive alarm messages via mail, sms, …• Status active or inactive can be set• PIN Code to acknowledge alarm messages
Groups:• Operators responsible for specific facilities• Defines priority who should be informed
first, second, …• Maximum delay for acknowledgment of
alarm messages
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 49
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Logging : 2X Log Server Verbindung
CSS mit direct Channel Access (DAL) connectionAs a transition to JMSDemo DALDemo JMSICS und JMS FolienAMS FolienConfig in CSS Log Table Topics
Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 50
XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser
Application Development
Integrating new I/O devices Configuration of I/O HardwareAnd Process Controls
Programmer Process Engineer
Get driverfrom
EPICS site
Prepare newIOC
Load Image
ConfigureDatabase
DatabaseCreation
Tool
Write text fileto load
directory
Load IOC with-Operating System-EPICS core-Database
Hardware Engineer
ProfibusConfiguration
Tool
ConfigureProfibus