Systems and Communications 1

CERN Openlab – 10/11 December 2013

ESRF: IT Challenges

Systems and Communications 2

ESRF Beamlines

X-ray standing waves

anomalous scattering

surface diffraction

gamma spectroscopy

dultra-fast scattering/ high pressure

materials sciencecircular polarisation

microbeamprotein crystallography

inelastic scattering

medical

nuclear spectroscopy

high energy

absorption spectroscopy

powder diffraction

spectroscopy on ultra diluted

samplessi

protein crystallography

e

Systems and Communications 3

What is a Beamline?

Systems and Communications 4

IT General architecture

Central storage

Infrastructureservers

Network

TapeBackup

ComputeClusters

Detectors

Systems and Communications 5

How much do we store each day?

Systems and Communications 6

Challenges

ESRF Phase II (higher photon flux) starts in 2015Already, two Eiger detectors arrive at ESRF in Fall 2014 4 Mpixel x 3 bytes = 12MB/image 750Hz = 750*12MB = 9GB/s Local compression => 1GB/s minimum 10% duty cycle => 9TB/day

Limiting factors: Human resources – data sorting and exploitation Data Analysis – online Data Analysis – offline Speed of the Central storage

Systems and Communications 7

Ongoing works

LBS only for Linux-based detectorsDSS time consuming but 300-500MB/s per unitHSN High-Speed Network 40Gbps

10Gbps

10GbpsCentral storage

Infrastructureservers

Network

TapeBackup

ComputeClusters

Detectors

Local Buffer Storage (LBS)

Dedicated Storage Server (DSS)

Systems and Communications 8

Requirements for one Beamline

ONLINEDATA ANALYSIS

(CPU+GPU)

EXPORTSTORAGE

BACKUP

ARCHIVE

BUFFERSTORAGE

BUFFERSTORAGE

LONG TERMSTORAGE

LONG TERMSTORAGE

OFFLINEDATA ANALYSIS

(CPU+GPU)

Etc.

Systems and Communications 9

Overall requirements

Typically 6 – 10 Beamlines simultaneously Typically 6 high-speed detectors per Beamline Typically 2 detectors operating simultaneously per

Beamline Peak bandwidth per Beamline = 1 – 10 Gbytes/sec

Systems and Communications 10

Overall architecture, with one Beamline

BUFFERSTORAGE

WindowsDETECTOR

PC

> 1 GB/

REALTIME ONLINEDATA ANALYSIS

(CPU+GPU)

VISUALISATION

LONG TERMSTORAGE

+OFFLINE

DATA ANALYSIS(CPU+GPU)

BACKUP ARCHIVE USEREXPORT

GREEN = DEDICATED

TO A BEAMLINE

Buffer Storage boxesmay be either dedicated to a detectoror shared by several detectors

LinuxDETECTOR

PC

LinuxDETECTOR

PC

BUFFERSTORAGE

BUFFERSTORAGE

>1 GB/s

BLUE = SHARED BETWEEN

BEAMLINES

Systems and Communications 11

Other challenges

Data Confidentiality

Systems and Communications 12

Questions?