JÉRÔME ALLARD [CEA-IRFU-DIS-LDISC] · 2019. 6. 12. · IRFU-DIS 3rd International Workshop of...

DATA COLLECTION FOR ANALOGICAL MAGNET SECURITY SYSTEMS

AT CEA-IRFUJÉRÔME ALLARD [CEA-IRFU-DIS-LDISC]

spare HDD

external monitoring

#1: FEPC

#4: -

#2: NAS

#3: Control-PC

ISEULT-controller

ISEULT-NAS

ISEULT-FEPC

ISEULT-userpc

2nd level

CABINET

test & maintenance [Arret] [Arret] {} [Enter]

Gentoo-MRnexcom nise 3500

NAS-1U-option+CentOS

CentOS 7

IPC4U Atemation-v2bMSE 7e-SP1 [CW]

CentOS

CentOS

#right

#2

NAS Link

wire 227

USB

Network Attached Storage

ISEULT-SCADA

+ VIEW & ANALYSISTHIRD PARTY TOOLs

spreadsheet

numerical analysis software

calc databrowser

ASNet User Toolkit

exportview

VIEW & ANALYSIS (on/off-line)[ASNet-View]

[FEtool]

[Muscade]

CONFIGURATION

SUPERVISION IHM

ASNet vPC

virtual user PC

off-line useremote user

Permanent Redundant Controller

SMTP servermx.extra.cea.fr

12:00:00.000

112 ana# - 96 dig#

MSSanalogical/digital signals

trigger signal

ISEULT-LAN

FRONTEND-PC

user PC

ASNet KVM

+ network access point

+ custom job launch

+ FEPC connectivity

archive storage

10 years = 1+9[compressed] years

last 50 [with associated kHz data]

daily stream

500MB (250MB)

20GB (6GB)continuous kHzreduced stream [10Hz]

event triggered acquisition 10kHz

local storage

last 20

60 days

20 days 1 year = 1.5+10.5[compressed] months

500G

B3.

5TB

512G

B20

GB

+ documentation hub

+ hand archive management

extra

ISEULT-bubble

DSIfirewall/router

FILE BASED STORAGE

LAB

SERV

ICE

ELOG

NTP reference clock


BACKUP ON TAPE2 years persistence

network disks

IRFU

SER

VICE

S

MA

D

1

MIV

A

3

MIV

A

4

MIV

A

5

MA

D

0

MA

D

2

MIV

A

8

MIV

A

6

MIV

A

7CPU

MIV

A

9 10

PS (2)

PN

CONTROL-PC

MHW-clock

SSD2SSD1

3rd International Workshop of Superconducting Magnets Test Stands 11-12/06/2019 2IRFU-DIS

Magnet Control System

LAN

Cryogenic Facility

Supervision

TRIG

GER

fieldbus

OPTIONAL FIELDBUS INTERFACE

Data analysisMSS monitoring

external power-supply for boards

export converted data

unsynchronized - correlation only

import process data

TRIG

GER

analog/digital instrumentation signals

analog/digital instrumentation signalsconditioned

Analogical Magnet Safety System

Magnet LTMS

data acquisition

AS-Net

{ redundancy with voting logic }

INTRODUCTION

ASNet stands for Acquisition System over Network. It is designed as an external acquisition module for CEA-IRFU analogical Magnet Safety System. It has been deployed for over 10 years on test stands or around big custom magnet.

A LAB STORY FROM 2003


AN ACQUISITION PRIMARY NEED

A superconducting magnet is a tricky machine that may experience a quench.Magnet Safety System (MSS) will then operate and turn the system off. A posteriori analysis will then follows. But in any case data recording turns out to be essential.

MSS needs an acquisition moduleReliable quench event recording as a black boxOscilloscope feature: on-line & offline as monitoring purposeAbility to handle tens/hundreds of channels – 50-200 analog or digitalCompatible with magnet physics - 1-100ms celerity & 2min disturbance time windows around triggerSlow trends expected to be observable – hours, daysSynchronous acquisitionWork with physical values & symbols

temperature sensor around magnet [YQ]

bundle of signals to record [YQ]

like an airplane black box as feature n°1

a quench record [FM]


Implementation concepts for Acquisition System on NetworkIndustrial PC & embedded general purpose OS - a ++ technology basis for developmentReal-time tasks handled in electronicsModularity & configurability first - channels, sampling, time windows for quench, conversionsInteractive use for magnet tests or background mode for operationShare data with process via local fieldbusOffer a long term view for slow trendsAdditional storage system to keep long term dataAS-Net status visible by a local supervision

→ Reliable but non critical system – trade off→ Juggle with multiple frequencies – source/distribution/storage/extraction

R&D, development and finally first deployment2003 - First test with a COTS board - DAQ22052004 - R&D developing a custom board: MIVA2008 - deployed as 2nd acquisition system for Station Essai Huit Testla

SEHT mechanical view [IRFU]

FROM PROTOTYPE TO OPERATION


ASNet FAMILY OVERVIEW

alpha beta

v3

v1

v2

ASNet main releases

IRM

test facility

[B198]

enhanced magnet test

[B192]

[B192]

[B198][lab]

development

particles separator

test facility

[2+ years][lab] [B126]

[10 years]

[GSI-1: cave C]

operation

[lab]

upgrade

development

development

operation

[Grenoble][lab]

[Neurospin]

validation

validation

validation

2011 2012 2013 2014 20152009 2010

2011 2012 2013 2014 20152009 2010 20182017 2019

2020 2021

2020 2021

FEPC - WorldFIP

SEHTFEPC - Profinet

Q4

STAARQ

2FEPC - Profinet

serie of coils validation

R1R0

R3B-1 R3B-2

FEPCv1 - WorldFIP, Profinet

Katrin

2008

2008

2016 201920182017

2016

R3B-3

JT60-SA CTF

ISEULT

LNCMI

operation

discontinuous runs

serie of coils validation

2FEPC - Profinet - 2*220V - NAS

FEPC - Profinet

FEPC - 48VDC - 2xNAS & backup

2FEPC - Profinet - NAS & backup

JA - 15.04.2019

+FEPCv2 - Profinet - NAS

IRFU

imag

e

ENS

imag

e

IRFU

imag

e

IRFU image

A LAYERED AND MODULAR ARCHITECTURE

Control & instrumentation school - CEA Saclay - 06/14/2013

| PAGE 7


ACQUISITION LAYER - OVERVIEW

IPC4U Atemation-v2b

MSE 7e-SP1 [CW]

IPC4U Atemation-v2b

MSE 7e-SP1 [CW]

PCI link

network link

RAM

DATA PATH

SRAM & SDRAM

fieldbus

vprocess 10Hz

FEPC variables 20Hz

FRONTEND-PC [MSS]

fieldbus

vprocess 10Hz

FEPC variables 20Hz

FRONTEND-PC [auxiliary]

0 1

1 0

0 1

1 0

MIVA/MAD external power supply

PC4U 220VAC-48VDC

experience LAN

convertedrawraw

HzkHzquench process

converted

event-triggered 1-50 kHz acquisition [PRE+POST]continuous kHz acquisition with bufferization

synchronization & datationconfiguration & diagnostics tools

{ process acquisition via fieldbus }{ data export to fieldbus }

ACQUISITION

MSS analogical/digital signals auxiliary analogical/digital signalstrigger signal

MA

D

10

MIV

A

1

MIV

A

2

MIV

A

3

MIV

A

4

MIV

A

5

MIV

A

6

MIV

A

7

MIV

A

8 9

MA

D

PN

CPU

PS(2)

CPU

PS

PN

10

MIV

A

1

MIV

A

2

MIV

A

3 4

MA

D

5 6 7 8 9

MA

D

MA

D

MHW-clock HW-clock

M


A custom analog card: MIVA16 analog input channels with 16bits ADC, voltage/current 1-50 kHz sampling rateSynchronization signal with other boards [master/slave]External power supplyQuench buffer [256 MB SDRAM – 8 M samples] – black box feature, triggeredReal-time buffer [6 kB SRAM – 500 samples] – continuous acquisition feature

PCI board for [PLX bridge]Processing via FPGA [VHDL]Simulation functions

ACQUISITION LAYER – MIVA & MAD INSIDE [1/2]

MIVA sister for digital inputs: MADDesign same as MIVA but for digital inputs32 digital input channels

MAD

MIVA

Inside the rack


CONTROLLER & ARCHIVING LAYERS

spare HDD

CentOS


headless

mirrored HDD

CentOS

spare virtual controllerheadless experimental

lambda laptop

data Gb link

BACKUP ON TAPEnetwork disks

storage - long term data

offline jobs platformdata backup agent

self backup area

hidden & compressed data

capacity: 30/60 rolling days for kHz/rms

standalone - minimalist footprint

Redundant Software Controller

produce reduced stream

storage - current data

archive manager

SCADA interface

health agent

[ legacy user toolkit ]

DATA HUB

CONTROL & MONITOR

DATA ARCHIVING

DATA DISTRIBUTION Disk Attached Storage


USB

ASNet base Controller

e-SATA | USB lab resources

external resources

[2nd level]Network Attached Storage

REMOTE BACKUP

partial backup

OS + blackbox

stream data area

MEDIA CAPACITY AVAILABILITY

partial | fullsystematic | on-demand

CONTROL-PC

RAID HDD

SSD

RAMdisk | SSD

months -> years

day -> months

hour

4TB-40TB

64GB-1TB

2GB

archive storage

local storage

in-process current hour - online

backup storage RAID HDD | tape -months -> years offline, from d+1

offline, from h+1

offline, from h+1

strorage hierrachy layout

spare diskUSB [1TB]

SSD2SSD1


DEPLOYMENT OVERVIEW - ISEULT EXAMPLE

spare HDD

external monitoring

#1: FEPC

#4: -

#2: NAS

#3: Control-PC

ISEULT-controller

ISEULT-NAS

ISEULT-FEPC

ISEULT-userpc

2nd level

test & maintenance [Arret] [Arret] {} [Enter]

CABINETDC mode


NAS-1U-option+CentOS

CentOS 7

IPC4U Atemation-v2bMSE 7e-SP1 [CW]

CentOS

CentOS

#right

#2

NAS Link

wire 227

USB


ISEULT-SCADA

+ VIEW & ANALYSISTHIRD PARTY TOOLs

spreadsheet

numerical analysis software

calc databrowser

ASNet User Toolkit

exportview

VIEW & ANALYSIS (on/off-line)[ASNet-View]

[FEtool]

[Muscade]

CONFIGURATION

SUPERVISION IHM

ASNet vPC

virtual user PC

off-line useremote user

Permanent Redundant Controller

SMTP servermx.extra.cea.fr

12:00:00.000

3xMSS with voting logic

10 years interuptionless operation

112 ana# - 96 dig#

MSS

trigger signal

ISEULT-LAN

FRONTEND-PC

user PC

ASNet KVM

+ network access point

+ custom job launch

+ FEPC connectivity

archive storage

10 years = 1+9[compressed] years

last 50 [with associated kHz data]

daily stream

500MB (250MB)

continuous kHzreduced stream [10Hz]

event triggered acquisition 10kHz

local storage

last 20

60 days

1 year = 1.5+10.5[compressed] months

500G

B3.

5TB

512G

B20

GB

+ documentation hub

+ hand archive management

extra

ISEULT-bubble

DSIfirewall/router

FILE BASED STORAGE

LAB

SERV

ICE

ELOG

NTP reference clock


BACKUP ON TAPE2 years persistence

network disks

IRFU

SER

VICE

S

20GB (6GB) 20 days

kHz: 240kB/s

analogical/digital signals

MA

D

1

MIV

A

3

MIV

A

4

MIV

A

5

MA

D

0

MA

D

2

MIV

A

8

MIV

A

6

MIV

A

7CPU

MIV

A

9 10

PS (2)

CONTROL-PC

MHW-clock

SSD2SSD1

ASNET THROUGH USE CASES


| PAGE 12


Before use a configuration has first to define through FEtoolsracks, boardschannels: names, unit, range, conversion…quench parameters: sampling rate, time window, masterfieldbus interface: variables to export

Each channel must be calibrated with a reference – wizard for linear correction

CONFIGURATION & CALIBRATION STEP

V physical ADC correction V correctedV rawV signal'sensor V signal conversion V measured

calibrationreferences [bounds -/+]

wire & conditioning


SYSTEM MONITORING & DRIVING

System is user monitored and driven through a Muscade interface. This may be integrated to the experiment SCADA, which allows triggering alarms for example to ask for maintenance on NAS disks, to warn about a wrong timing or to notify immediately a quench event.


WORKING WITH EVENT TRIGGERED DATA

Once a trigger occurs a snapshot around event is made with a resolution up to 20us (blackbox feature). Analysis can then begin...


WORKING WITH CONTINUOUS DATA STREAMS ½

Data acquisition is continuous with a synchronous kHz stream and an optional 10Hz fieldbus process stream. We can then extract offline, from controller or archive storage, working at hour or ms scale, superimposing source for correlation if necessary...


WORKING WITH CONTINUOUS DATA STREAMS 2/2

zoom anywhere use for long trends

online oscilloscope feature

access data inside archive

mix source as wanted

On dedicated user PC an online view is available to monitor each channel (kHz or process). It is particularly useful for testbench or full-scale test configurations.

Thanks to reduced stream ASNet is usable with long time windows, typically weeks, for example to follow magnet cooling or summarize a coil testing cycle…


ANALYZE MORE IN DEPTH

A few mathematical functions are available in viewer. So are some calculated variables. But for more in depth analysis just export a data subset and work elsewhere with a spreadsheet software or other numerical analysis tool.


SPECIAL USES

[JT60] Due to data proximity and computational resource NAS may be used for offline jobs. In JT60 we use it to produce a user defined data compilation for each tested coil.

[ISEULT - experimental] we are currently trying to analyze miscellaneous noise for instance in correlation with power supply mode. Idea is that in continuous magnet mode, a noise change may indicate something significant.

[STAARQ test stand facility (future)] Both analog and digital MSS will be installed. Our legacy analog MSS acquisition system will make its traditional job. It will also record digital MSS input signals and be used as reference for a new acquisition implementation attempt.

ASNET INTERNALS


| PAGE 20


Sum additional project requests & contexts – manage a modular and homogeneous versionHave an operational & validated solution for operation – moderate upgrade, use versioningTake into account expected long lifetime – choose robust hardware, buy spareAllow system evolution – work on new deployment window, test on redundant chain, use idle facility, cross project feedbackOffer maximum availability – implement redundancy, add monitoring & reporting, fault-tolerance. Virtualization may also helpReduce maintenance constraints – be tolerant to degraded mode, include remote access

HANDLING BOTH OPERATION AND DEVELOPMENT

HARDENING

redundant power supply

industrial network device

Control-PC: PC > shoebox

CONSOLIDATION

background mode

monitorability

watchdog set

data selective backup on tape

network access [subnet/VLAN/bubble/VPN]

separate user PC

2nd level storage in lab

long development timeUPGRADE

miva: serie 1 > serie 2

FEPC driver: PLX, Profinet

FEPC OS: XPe -> 7e

Control-PC OS

FEPC rack: IPCv1 -> IPCv2 (a,b)

XP -> 7e -> gentoo -> centOS

ENHANCEMENTR3B / JT60 / ISEULT / LNCMI / Q4

digital channels

data export

calculated variables

multi-frequencies management

multi-racks feature

process input integration

reduced mergeable stream

data long storage

reporting

RAMdisk option

virtualized user PC

soft redundant controller


TOWARD A RESILIENT SYSTEM: LONG IS THE ROAD...

Feedback on typical unexpected incidents● Out of order system disk on NAS● Unexplained NAS ECC memory error with a temporary loss of a 4GB stick ● Emergency stop punch on cabinet before electrical maintenance we were not aware...● Cement dust environment in an experience hall…● Flooding in remote backup room after severe weather and building issue

SSD: sensitive use & a bit specific compared to HDDgood bandwidth, constant time access, parallelism prone but caution with write intensive application → †

● Weared symptoms : file system errors, read-only file system● Caution with write amplification effect: you may write much more than expected● On linux compulsory use of ext4 with noatime & discard options● Caution with swap: no swap or minimize it via swapiness variable● Increasing disk size gives more lifetime (linear)● Accurate Monitoring via SMART (nowadays)● Intermediate RAMdisk minimizes SSD wear


NEXT STEPS

Main limitations:→ Custom hardware is painful to produce & test. Nowadays COTS boards exist→ System is too much person dependent→ Bad initial options we live with: legacy file format, time stamping, FEPC windows platform

Make it work better (developpement & test in-progress...)● enhanced anaconda python console for immediate expert analysis● more integrated math functions: derivative, integration, FFT● export to standard format: hdf,…● out-of-band management option● rewrite fieldbus interface● data transparency rms / kHz / trigger● on-line mode with long time windows● on-demand archiving● manage acquisition configuration● manage a experiment database for users

Make it work further● Use standard framework (EPICS?)● Use COTS hardware for acquisition layer (CompactRIO,PXI,uTCA?)

RE-BASEMENT NEEDED TO BE COMPATIBLE WITH DIGITAL MSS


| PAGE 24


A PROTOTYPE IN OPERATION WITH A BEST EFFORT POLICYLONG LIFETIME EXPECTED DEVELOPMENT STILL IN PROGRESS…RE-BASEMENT NEEDED FOR FUTURE

A TEAM WORKElectronics works project management F.Molinié

MIVA/MAD design S.Sube, P.DeAntoni

MIVA/MAD embedded programming S.Sube

Frontend custom code C.Walter, J.Allard

Fieldbus interface WordFip/Profinet G.Durand, J-L.Fallou

Controller core code C.Walter, J.Allard

Data extraction C.Walter, J.Allard

Archive, distribution, backup A.Gomes, P-F.Honoré, J.F.Lecointe, J.Allard

Configuration tool C.Walter

Visualization tool J.Allard

Network C.Walter, A.Gomes, J.Allard

OS C.Walter, P.Mattei ,J.F.Le cointe, J.Allard

Experience support C.Walter, J.F.Lecointe, J.Allard

Electrical support A.Lotodé

THANKS FOR YOUR ATTENTION

Slide 1fast acquisition and its surroundingSlide 3Acquisition neededfrom prototype to operationSlide 6A custom acquisition system for Magnet Safety SystemSlide 8MIVA & MAD inside – custom HARDWARE 1/2Slide 10Slide 11Slide 12configuration & calibration stepSlide 14Slide 15Slide 16Slide 17Slide 18Slide 19Slide 20Handling evolutions & experience long lifetimeSlide 22Slide 23Slide 24Slide 25

JÉRÔME ALLARD [CEA-IRFU-DIS-LDISC] · 2019. 6. 12. · IRFU-DIS 3rd International Workshop of...

Documents

Transcript of JÉRÔME ALLARD [CEA-IRFU-DIS-LDISC] · 2019. 6. 12. · IRFU-DIS 3rd International Workshop of...