Outline

1 Gunther [email protected]

1Photon Area ControlsFAC Review June 09

Photon Systems Controls

and Data Systems

Gunther Haller

Particle Physics and Astrophysics DivisionPhoton Control and Data Systems (PCDS) Group

June 9, 2009

v2



Outline

LCLS versus LUSI Responsibilities

PPS/HPS ( see previous presentation by Enzo)

XTOD X-Ray Transport Optics & Diagnostics

AMO Instrument Controls

LUSI Instruments

Diagnostics

Machine Protection System

Laser Safety System

EPICS Software

Femto-Second Laser Timing System

Data Acquisition (see Amedeo Perazzo presentation)

2-D Detector Control and Readout (XPP, CXI, XCSS)

Offline/Computing



LCLS & LUSICommon photon-area Controls and DAQ System Design for LCLS and LUSI

Includes Front-End Enclosure (FEE), Near Experimental Hall (NEH), X-Ray Tunnel (XRT) and Far Experimental Hall (FEH)Most of XTOD is located in FEE

LCLS Controls & DAQ W.B.S 1.6.2AMO experiment (NEH, hutch 1)

All controls and DAQCommon services for all hutches

Examples PPS/HPS/LSS120 Hz beam-quality data interfaceMachine protection system interfaceNetwork interfaceLocal science data online processing & cache

2-D detector Control and DAQ for detector

LUSI Control & DAQ W.B.SXPP, XCS, CXI Instruments, Diagnostics

Examples of documents released1.1.523 XES-LUSI ICD (includes Controls and Data systems)1.1-516 XES Photon Controls to Electron Beam Controls System ICD 1.1.517 XES Photon Controls to Electron Beam Controls MPS ICD



XES Near & Far Hall Hutches and Beamline Layout (not to XES Near & Far Hall Hutches and Beamline Layout (not to scale)scale)

XES Near & Far Hall Hutches and Beamline Layout (not to XES Near & Far Hall Hutches and Beamline Layout (not to scale)scale)

Electron Dump

FEE

Hutch 1 Hutch 2 Hutch 3

X-Ray TunnelHutch 4

Hutch 5

Hutch 6S0

S1

S2

SH 1

M 1S M2S

M 3/4S

X3 X4

M6

S 3

SH2

S4

S5

S 6

Horizontal Offset Mirrors

PPS Stopper Set

X- Ray Mirror

X-Ray Crystal

Primary Movable Elements

Experiment

Near Experimental Hall

Far Experimental Hall

Moveable Dump

XPCS Pos2

XPCS

Pos1CXI

HED

XPP Pos1

XPP

Pos2

AM O

SXP

SXP

230 m

SXR

AMO

MEC



SolidAttenuator

Gas Attenuator

Fast Valve+Fixed Mask+

Slit

5 mm diameter

collimators

Muon Shield

FEL Offset mirror system

Gas detector

Gas detector

TotalEnergy

Calorimeter

Low-energy beam

High-energy beam

24

mm

ver

tica

l

900

mm

ho

rizo

nta

l

10-10 Torr

Indirect Imager / K measurement /

spectral

Direct Imager

reticule

Diagnostic Package

XR Diagnostics in Front End Enclosure



XTOD in FEE

Phase 1 of XTOD in commissioning

Controls installedBeing tested paced by availability of beam-line items installed



Photon Area Controls and Data Systems Devices

Created web-pages to list supported controls and data-acquisition devices for photon area

For all experiments in xray area (AMO, XPP, XCS, SXR, CXI, etc)

SXR might have up to 8 different end-stations

http://confluence.slac.stanford.edu/pages/viewpage.action?pageId=9175609Additional devices can be added after discussions with Photon Controls and Data Systems (PCDS, G. Haller) group



AMO Instrument Control

Requirements listed in Engineering Specification Documents (ESD’s) 1.6-108 AMO Controls ESD, status: released 1.6-109 AMO DAQ ESD, status: released

Interfaces specified in Interface Control Document (ICD) 1.1-515 XES AMO Controls ICD, status: released

Held Final Design Review 11-10-08http://confluence.slac.stanford.edu/display/PCDS/AMO-FDR

Purchased all hardware

Racks with infrastructure/AC installed in hutch 1

Controllers/DAQ being tested in lab before moving to NEH early next month

See pictures later in presentation



AMO and its DAQ Instruments

High Field Physics End-Stationelectron time-of-flight spectrometersion spectrometers

time-of-flight, imaging, momentum

x-ray emission spectrometers

Diagnostic Section

magnetic bottle electron time-of-

flight spectrometer

x-ray emission spectrometers

pulse energy monitor

beam profile screens



Example: Vacuum

All gauge controllers are MKS 937A Interface

Terminal server – DIGI TS16 MEI

Automation Direct PLC

All ion pump controllers are Gama Vacuum DIGITEL MPC dual

All valves are controlled by PLC relay module

The out/not-out state of all valves go into the MPS system to prevent damage if a valve closes unexpectedly.



Example: Motion

Control System provides support for all motionsMotors

IMS MDrive Plus2 integrated controller and motor

IMS MForce Plus2 controller for control of in vacuum and other specialized motors

Newport motor controllers

Others as required

Pneumatic motionSolenoid Driver chassis, SLAC 385-001



AMO Test Setup in Lab

Racks in lab, AMO controllers being installed

Pulse picker and also DAQ test setup



AMO in Hutch 1

Racks installed, AC connected

Will move loaded crates from lab to NEH racks in July



Software: AMO Example: Moving Tables



AMO Test Setup in Lab

AMO table motion control mock-up for motion software algorithm

Smart Motors



LUSI & SXR

LUSI includes the instrumentsXPP (NEH, hutch 3)XCS (FEH, hutch 4)CXI (FEH, hutch 5)Have written and released ESD’s and ICD’s for each instrumentHeld XPP Final Design Review, CXI & XCS Preliminary Design Review

See under AMO and CXI and XCS at http://confluence.slac.stanford.edu/display/PCDS/PCDS+Home

SXRBeamline was added to the scope

Have drafts of ESD and ICD’s

Held discussions with several end-stations on how to plan to make them compatible with LCLS



Common Diagnostics Readout

• Four-diode design

R2

R121

L

Target

Quad-Detector

FEL

• On-board calibration circuits not shown

E.g. intensity, profile monitor, intensity position monitorsE.g. Canberra PIPS or IRD SXUV large area diodes (single or quad)

Amplifier/shaper/ADC for control/calibration/readout

• Board designed, fabricated, loaded, is in test



MPS Rack Layout (Reminder)

RS-232

RS-232

De

dic

ate

d S

ing

le-M

od

e O

ptic

al F

ibe

r P

air

De

dic

ate

d S

ing

le M

od

e O

ptic

al F

ibe

r P

air

10

00

Ba

se-L

X

Cat-6 Ethernet

Cat-6 Ethernet

(To Main Control Center)

MPSLink Node

MPSLink Node

Terminal Server

Ethernet Switch

Network-Controlled Power Distribution Unit

Cat-6 Ethernet

10

00

Ba

se-L

X(H

ot

Sp

are

)

Dedicated Optical fiber1 single-mode pair per Link Node for custom MPS signaling protocol

TCP/IP NetworkingRedundant single-mode pairs for 1000Base-LX Ethernet

Access to EPICS process variables on MPS Link Nodes via local Ethernet switchMPS Link Node console logins via Terminal ServerRemote power-cycling of Link Nodes via APC AP7900 Switched Rack PDU

All optical networking paths route through experiment hall patch panels and terminate at fiber patch panel in MCC



Installation Status

Near Experiment Hall Machine Protection Rack, (B950S-04), Installed.48-strand fiber trunk pulled between B950B-42, (NEH Server Room), and B950S-04. The NEH MPS optical networking path is complete.Ethernet Switch Installed in rack B950S-04, connected to fiber, and checked.



MPS Installation Status

MPS Trunk Field Rails assembledMPS trunk cables pulled between the hutches, the FEE, and rack B950S-04.

All DIN rails for rack B950S-04 assembled, tested, and installed

12 MPS Link Node Input Aggregation Rails6 MPS Trunk Head-End Rails



PPS/MPS Racks

Installed in NEH



NEH Laser Bay Laser Safety System Layout and Components

LED Hazard Displays

LED Hazard Display

Emergency-Off Buttons

LSS Controller Rack

Door Interlock Switches

LSS Control Panel

Timed Door Interlock Bypass Switches

Access Control Panel

Electronic Strike Plate



Laser Safety System

Laser Room Entrance

Start installation on first table

Laser Safety System control rack



EPICS/Python/Qt

EPICS (Experimental Physics and Industrial Control System):

Control software for RT systemsMonitor (pull scheme)AlarmArchiveWidely used at SLAC and other labsMore: http://www.aps.anl.gov/epics/

Python/Qt is a user interface between the EPICS drivers and records and the userSystem is currently used for XTOD and AMO, provided as part of the XES Photon Controls Infrastructure

Later used for SXR, CXI, XPP, XCS, MEC



Consoles: PyQt versus EDM



Status

AMO EPICS drivers almost completeMore work as scheduled over the next two months

Plus need high level control panels



Femto-Second Laser Timing System

RF Cavity System provided by SLAC

Laser Fiber driver/receiver system provided by LBL

Tested performance of LBL system prototype: better than 20 fsec drift performance (100 fsec is spec)

2-Receiver, 1-Driver system will be installed at SLAC in July 09.



Designed/Built at LBL



RF Cavity Electronics for Femto-Second Timing System in Lab

RF Cavity SLAC

Chassis

Control/DAQ crate with IOC and SLAC 119 MHZ waveform

digitizer



Example: LUSI Instrument CXI

Diagnostics &Wavefront Monitor

1 micron Sample Environment

* 0.1 micron KB & Sample Environment,Particle Injector and IToF (CD-4)

1 micron KBReference Laser

Diagnostics/Common Optics

All Early Science except *



Controls SubsystemsVacuumMotionViewingPower SuppliesRacks and CablingOther itemsSoftware: EPICS/Python/QtType of controls

Valve ControlVacuum ControlsPop-In Profile Monitor ControlsPop-In Intensity Monitor ControlsIntensity-Position Monitor ControlsSlit ControlsAttenuator ControlsPulse Picker ControlsKB Mirror ControlsX-Ray Focusing Lense ControlSample Environment ControlsParticle Injector ControlsIon ToF ControlsVision Camera ControlsDetector Stage ControlsReference Laser ControlsDAQ Controls



CXI Components to Control

X-Ray Optics KB systemMotionVendor provided, integration with LCLS

Reference LaserMotion

Sample EnvironmentSample Chamber

Motion, vacuum, visionIon ToF

HV, DC/pulser, digitizerInstrument Stand

MotionDetector Stage

Motion, vacuum, thermal Particle Injector

Motion, vacuum, digitizer, vision, integration of commercial component

Vacuum SystemValve and Vacuum Controls



CXI Components to Control con’t

Diagnostics and Common OpticsPop-In Profile Monitor

Motion, ViewingPop-In Intensity

Motion, DigitizationIntensity Position

Motion, DigitizationSlit System

MotionAttenuator

MotionPulse-Picker

Motion, ViewingX-Ray Focusing Lense

Motion

CXI specific interface and programmingRacks & CablingWorkstationsVision CamerasBeam Line ProcessorChannel Access GatewayMachine Protection SystemConfigurationData Acquisition

Use standard controllers on photon area controller list as much as possible



LUSI Data Acquisition for Pixel Detectors

Cornell and Brookhaven 2-D pixel detectors are configured & read out using the SLAC ATCA Reconfigurable Cluster Element modules

Details in following slidesXPP XAMP Detector with custom ASIC

CXI Detector with custom ASIC



Photon Control Data Systems (PCDS)XPP specific

Digitizers + Cameras

Timing L0: Control

(One)

L1: Acquisition(Many)

L2: Processing

(Many)

L3: Data Cache(Many)

DAQ system primary features

Trigger and readout

Process and veto

Monitoring

Storage

Beam Line

Data

Data System Architecture



XAMP 2D-Detector Control and DAQ Chain

XAMP (XPP) LUSI instrument custom integrated circuits from Brookhaven are already connected at SLAC to SLAC LCLS high-performance DAQ system

XPP BNL XAMP Detector 1,024 x 1,024 arrayUses 16 each 64-channel FexAmps BNL custom ASICsInstantaneous readout: 4 ch x 20 MHz x 16bit= 20 Gbit/sec into FPGAOutput FPGA: 250 Mbytes/s at 120 Hz (1024x1024x2x120)

In addition BNL has ATCA crate with SLAC modules to develop software and test with detector

ATCAAdvanced Telecommunication Computing Architecture

Based on backplane serial communication fabric, 10-G E2 SLAC custom boards (also used in other SLAC experiments)

8 x 2.5 Gbit/sec links to detector modulesDataflow and processingManaged 24-port 10-G Ethernet switchingEssentially 480 Gbit/sec switch capacityNaturally scalable

SLAC FPGA front-end board

Fiber

ATCA crate with SLAC DAQ boards, e.g. the SLAC Reconfigurable Cluster Element Module

Brookhaven XPP/XCS 2D detector-ASIC

Beamline Instrument Detectors



Example: XPP Online Processing

Electronics gain correction (in RCE)

Response of amplifying electronics is mapped during calibration

Science data images are corrected for channel gain non-uniformity + non-linearity.

Dark image correction (in RCE)

Dark images accumulated between x-ray pulses

Averaged dark image subtracted from each science data image

Flat field correction (in RCE)

Each science data image is corrected for non-uniform pixel response

Event filtering (in RCE or later)

Events are associated with beam line data (BLD) via timestamp and vetoed based

upon BLD values. Veto action is recorded.

Images may be sparsified by predefined regions of interest.

Event binning (processing stage)

Images (and normalization) belonging to the same bin (t, E, ..) are summed

together



Example: XPP Monitoring

A copy of the data is distributed (multicast) to monitoring nodes on the DAQ subnet.

The monitoring nodes will provide displays for experimenters’ viewing:

Corrected XAMPS images at ≥ 5 Hz

Histories of veto rates, beam intensity, + other BLD values.

Reduced analysis of sampled binned data (versus scan parameter)

Implemented with Qt (C++/Python open source GUI)



Example: XPP XAMPS Data Rates

XAMPSL1:

RCE

L2:

Processing

L3:

Cache4 x 2.5Gb PGP

240 MB/s(480 MB/s)

10 GbE

< 200 MB/s10 GbE

n x (200 MB – 20 GB)Binned data archived

at end of run (mins – hrs)

Photon Control Data Systems (PCDS)XPP specific

Digitizers + Cameras

Timing L0: Control

(One)

L1: Acquisition(Many)

L2: Processing

(Many)

L3: Data Cache(Many)

Beam Line

Data

Expect ~ 6 – 60 GB / day



CXI 2D-Detector Mechanical/Electrical Vacuum Assembly

Cut-outs in base plate for cold straps and cables

Positioning plate•Supports quadrant raft•Mounts to drive system

Cam follower mounted to torque ring

Cold strap

Hole size remotely adjustableVia PCDS Controls

One Quadrant Raft Removed

Pixel Detectors

SLAC PPA Engineering

Electrical/Mechanical Package



Quadrant Board and Electrical Interfaces

Cold strap

Mounting feet

Quadrant raft

Quadrant board 1

Detector

Quadrant raft provides structural support and stability for the double-detector packages

Feet mount on quadrant raft through holes in the quadrant boards This is also the thermal path

Quadrant boards provide grounding, power, and signal interface to the PAD detector package

1 flex cable per detector1 FPGA on each quadrant board

Quadrant board 2

Double-detector package (4 per quadrant)



ASIC Board

Rigid-flex ASIC board (SLAC design)

ASIC Bump-bonded to detector

ASIC/detector package bonded to carrier board



CXI 2D-Detector Control and DAQ Chain

Ground-isolation

Vacuum

Fiber

Cornell detector/ASIC with SLAC quadrant board ATCA crate with SLAC DAQ Boards

Each Cornell detector has ~36,000 pixelsControlled and read out using Cornell custom ASIC

~36,000 front-end amplifier circuits and analog-to-digital convertersInitially 16 x 32,000-pixel devices, then up to 64 x 32,000-pixel devices

4.6 Gbit/sec average with > 10 Gbit/sec peak

Carrier Board

S:AC RCE ATCA Module



CXI Online Processing

Electronics gain correction (in RCE)

Response of amplifying electronics is mapped during calibration

Science data images are corrected for channel gain non-uniformity + non-

linearity.

Dark image correction (in RCE)

Dark images accumulated between x-ray pulses

Averaged dark image subtracted from each science data image

Flat field correction (in RCE)

Each science data image is corrected for non-uniform pixel response

Event filtering (in RCE or later)

Events are associated with beam line data (BLD) via timestamp and vetoed

based upon BLD values. Veto action is recorded.

Images may be sparsified by predefined regions of interest.



CXI Online Processing con’tEvent processing (processing stage)

Examples are

Sparcification (region of interest)

Locating center

Reducing data by binning pixels

Mask errant pixels (saturated, negative intensity from dark image subtraction due to e.g.

noise, non-functioning pixels, edge pixels from moving center)

Filling in missing data with centro-symmetric equivalent points

Transforming camera geometry due solid angle coverage and dead space between tiles

Radial averaging, showing intensity versus scattering angle or momentum transfer

Compute 2D autocorrelation function (single FFT) and store. Essentially at rate of 1 Hz with

4 MB (2Mpixel x 2 bytes) frames.

Peak finding (locate and fit Gaussian intensity peaks). There may be multiple peaks in some

cases and the peak finding algorithms should be able to identify up to a few thousand

peaks.

The CXI instrument will have an Ion Time-of-Flight which will produce data at 120Hz. The

online processing of this data involves data reduction based on thresholding and vetoing

based on thresholding or the fitting of peak positions and height.



CXI Monitoring

A copy of the data is distributed (multicast) to monitoring nodes on the DAQ subnet.

The monitoring nodes will provide displays for experimenters’ viewing:

corrected detector images at ≥ 5 Hz

histories of veto rates, beam intensity, + other BLD values.

Reduced analysis of sampled binned data (versus scan parameter) or other processing tbd

Implemented with Qt (C++/Python open source GUI)



Offline

Offline Analysis ExamplesXCS Multi-tau multi-Q processingCXI Classification

AveragingAlignmentPhase Retrieval

To be addressedCXI

Data Sorting/Averaging (Classification)Phase Retrieval



Offline Data Management System Functions

Translate data into a form for long-term scientific access

Translate to HDF5 format From online, XTC, Extended tag Container format, C++

Store data attributes in a database (science metadata)Store translated data and archive to tapeProvide access for scientists to stored and archived data

Access by attributes (science metadata)Manage data ownership and access restrictionsData access for:

Processing clustersUser workstationsOffsite export (network and other means)

Manage derived data products



Offline System Architecture

Steffen Luitz



Components

Data FormatHDF5/NeXUS chosen because of support, wide availability of language bindings

File ManageriRODS (next-generation of SDSC SRB)

Tape ManagerInterface iRODS to SLAC HPSS and mass storage system (SL8500 )

High performance file systemsLustre

DatabasesMySQL

MonitoringGanglia

Data transferCurrently performing data transfer studies with CAMP group in Germany (Peter Holl) to write and retrieve few hundred Gigabyte files using a few data transfer options

Sftp (expect 2 Mbytes/s)Bbcp (expect 20 Mbyte/sec using around 15 streams)FTD (around 20 Mbytes/sec using 64 streams)

User accounts, etc via SCCS, unixIn 2010 will move to server-based solution

XROOTDiRodshttp

Steffen Luitz



Some Analysis Options

NeXus API

Can use NeXus or HDF5 Tools and Analysis Packages

HDF5File API

Open Genie LAMP

GumTree Nathan

Redas Scilab Amortool More …

Open Source

Cactus Chombo dxhsf5

H5PartRoot HL-HDF

ParaView PyTables

VisAD

Many more …

Open Source

Commercial

IDL-HDF5 Matlab

Mathematica O-Matrix ViTables More …

Steffen Luitz



Comments regarding Users

Users can configure LCLS equipment/experiments from terminals (e.g. control room or hutch, including photon area loaner laptop), plus remotely via tunneling

Several Phyton/Qt screens providedUser can add to it (support provided by PCDS (Photon Controls and Data Systems) group)Or PCDS can provide requested screens

generate and apply calibration correction constantsmonitor online performance on screens (plus remotely via tunneling)

Images, before/after correction, other parameters depending on experiment (e.g. centroid movement) including dataflow monitoring (time-stamp, event increments, missing images, etc)Quality of dataAgain users can add screens or PCDS group can provide custom screens

have access to data for post processingFile manager provided

have ability to transfer data to media or offsite

Technical support provided by group building the systemIncludes Python/QtIntegration of user data processing algorithms



Comments regarding non-LCLS End Stations

XPP, CXI, XCS detectors are integrated into LCLS ATCA RCE data-acquisition system

Configuration, readout of detectors/ASICsEvent building of images with

120-Hz beam-line data Time-stamp

Allows real-time online vetoing, sorting, or other processing which depends on e.g. energy, time-difference between laser and xray (for XPP)Processing using LCLS NEH farmStorage using LCLS storageRetrieval

pnCCD (MPI) detector with Juelich DAQ, camera, and Acqiris waveform digitizer for e.g. CAMP

Have been in contact for several months with MPI and DESY to work towards being able to integrate. Recommended changes to their system to be able to integrate.

Started same with potential LBL FastCCD currently using Argonne DAQ for SXRAbove need to be integrated into LCLS system otherwise

Results would be that end-stations would be run in “stand-alone”, potentially withNo time stampsNo 120-Hz BLDNo processingNo storage/retrieval

Not acceptable for end-stations or LCLS, especially for high data rate systems



Example CAMP

Offline: latency of e.g min to hours (HDF5 format)Depends on size of runs plus availability of offline system (not required to be 100% available)

Data from L3 storage cache (XTC format): latency 10 – 30 secData processed online, C++: seconds latency

CAMP collaborators write code interfacing to DAQ online XTC data format to monitor performance and to allow human time-scale feedback controlRuns on “Observer” L2 processing ATCA blades



NEH Server RoomHouses mainly

Machine Timing DistributionNetwork switchesEPICS serversL2 processor farmL3 local data-storage

Terminals

Existing racks with data cache and servers

Phase 2 racks still

to be installed



SummaryAll long-haul cables for FEE and NEH are installed, so are racks

XTODInstrumentation for phase 1 is installed, testing in progress

PPS, HPS, MPS, and LSS are all well on their way, most already installed

AMO Instrument Controls and Data Acquisition Installation in progress, will continue over next 3 months10 weeks of science starting October 09

LUSI Instrument Controls (XPP, CXI, XCS)Final Design Review held for XPPPreliminary Design Reviews held for CXI and XCS

SXR Beam-Line control components being orderedIn discussions with SXR Endstation leaders to ensure compatibility

Femto-Second Timing System well under wayRF cavity electronics to be installed in Undulator hall in JulyLBl Laser Fiber system to be installed in July



Summary con’t

2-D Detector Prototype signal chains in operation from detector to DAQ system

DAQ well under way (see Amedeo’s talk)

Offline infrastructure in design

Scientific Computing for XPP, XCS, AMO in progress

Scientific Computing for CXI still needs to be definedAlgorithms are being developed. Resulting computing needs to be determined

Outline

Documents

Transcript of Outline