A DAQ Architecture for Agata experimentagata.pd.infn.it/documents/week9152003/GaetanoMaron.pdf · A...

G. Maron, Agata Week, Legnaro, September 2003

1

A DAQ Architecture for Agata experiment

Gaetano MaronINFN – Laboratori Nazionali di Legnaro


2

Outline• On-line Computing Requirements

• Event Builder

• Technologies for the On-line System

• Run Control and Slow Control

• Agata Demonstrator 2003-2007

• Project Organization

• Off-line Infrastructure


3

Front-end electronic and pre-processing

Pulse Shape Analysis

Event Builder

Tracking

200 Gbps (1 Gbps x 200)

Max 10 Gbps (50 Mbps x 200)

10 Gbps

1 Gbps

1.5 x 106 SI95(?) (present algorithm)

Agata Global On-line Computing Requirements

5 x 103 SI95

3 x 105 SI95 (no GLT)3 x 104 SI95 (GLT @ 30 kHz)

SI95 = SpecInt 951 SI95 = 10 Cern Unit = 40 MIPS

GLT = Global Level TriggerStorage


4

Input to the DAQ System

PREAMFADC

FADC

DSPDSPDSPDSPFPGA

Detector i=1-200

Mux PC

PSA Farm

50 Mbps x 2001 Gbps x 2001.5 MSI95 (now)


5

Farming• Commodity PC farms + linux

are a well established technology

• Rack mountable cases allow to fit hundreds PCs in few racks (with blade form factor)– about 16 kSI95 per rack in

2004– about 50 kSI95 per rack in

2007– About 150 kSI95 per rack

in 2010

Computational nodes Data Servers

Gateways


6

Agata on-line system

Builder Network

B1 B2 B20

Event Builder

HPCC builder

Tracking Farm

F1 F200F3F2Front-End

PSA Farm R1 R200R3R2

200 Gbps

10 Gbps

10 Gbps

1 Gbps

ds1 ds2 ds3 ds4

Storage (1000 TB)

Data Servers1 Gbps100 Mbps> 1 Gbps


7

Event Building: simple caseR200

T01T02T03T04

T05T06T07T08

T09T10T11T12

T01T02

T05T06T07

T09

T01T02T04T05T06

T09T10T12

T07T08

T09T12

T01T03

T07

T09T12

PSA FarmsClock R1 R2 R3

T01T03Time Slot 1

Time Slot 2

Builder Network (10 Gbps)Time Slot 3

BU1 BU2 BUn Builder Units

Where n could range (now) from 10 to 15 according to the efficiency of the event builder algorithm and to the communication protocol used

In the final configuration (~ 2005) we could imageTo have a single 10 Gbps output link and a single BU


8

Ev #Ev #Ev #

Ev #Ev #Ev #MUX

Det iEv #Ev #Ev #

TTC assigns EV #(time stamp)

MUX buffers events according to a given ruleand then define the Time SlotMUX assigns a buffer # to this collection of events

Ev #

Buf #Ev #Ev #

MUX distributes the buffers to PSA farm according to their buffer #

PSA FARM

PSA farm shrink down the incoming buffers, a further buffering is then neededPSA assigns a EB (Event Builder) # to the new buffersPSA distributes the buffer to Event Builder farm according to such number.

Ev #Ev #Ev #

Ev #Ev #Ev #

Buf #

EB #Buf #Buf #

EB FARM

All t

his

is s

ynch

rono

us fo

r all

the

dete

ctor

sEv

ent M

ergi

ng in

the

EB F

arm

is th

en fe

asib

le

Slic

e 1

Slic

e 2

Slic

e 20

0

Detector Slice

Time Slot Assignment


9Agata Event Builder: some more requirements

T01T02

T05T06T07

T01T02T04T05T06

T07T08

T09T12

T01T03

T07

T09T12

T01T03

R200

T01T02T03T04

Time Slot 1

T05T06T07T08

T09T10T11T12

T09 T09T10T12

Readout FarmsClck R1 R2 R3

Time Slot 2

Builder Network (10 Gbps)Time Slot 3

BU1 BU2 BU3 Builder Units

- delayed coincidence can span more time slots.- fragments of the same events are in different BUs.


10

HPCC for Event Builder

High PerformanceComputing andCommunication

(HPCC)System

- High speed links (> 2 Gbps)- low latency switch- fast inter processor comm- low latency message passing

Builder Network


11

Technologies for Agata On-line System

• On-line Software• Networking Trends• Event Builder • CPU Trends• Building blocks for the Agata Farms • Storage Systems


12

On-line Software: a common framework• Environment for data acquisition applications

– communication over multiple network technologies concurrently• e.g. input on Myrinet, output on TCP/IP over Ethernet, Infiniband, etc.

– configuration (parametrization) and control• protocol and bookkeeping of information

– cross-platform deployment• write once, use on every supported platform (Unix, RTOS)

– high-level provision of system services• memory management, synchronized queues, tasks

– built-in efficiency enablers• zero-copy and buffer loaning schemes usable by everyone

• Aim at creating interoperable systems– PSA farm applications, event builder, tracking commonly managed

• http://xdaq.web.cern.ch/xdaq/

CMS


13

2000 2001 2002

128 Gbps

192 Gbps

Aggregate bandwidthwithin a single switch

• Local networking is not an issue. Already now Ethernet fits the future needs of the NP experiments– link speed max 1 Gbps– switch aggregate bdw O(100) Gbps– O(100) Gbit ports per switch– O(1000) FastEthernet per switch

• If HCCP is requested (e.g. Agata builder farm) options are Myrinet, Infiniband = 4 x Myrinet, 10 Gbit Eth

10 µsec latency time

Myrinet one way latency Myrinet throughput

250 MB/s

GigaEthernet

Myrinet

256 GbpsInfiniband10 GbEth

Networking Trends

64 Gbps

2003


14

Event Builder and Switch Technologies


15

CMS EVB Demonstrator 32x32

CMS


16

Myrinet EVB (with Barrel Shifter)

CMS


17

Raw GbEth EVBCMS


18

GbEth full Standard TCP/IP

CMS

CPU Load 100 %


19

TCP/IP CPU Off-loading - iSCSI• Internet SCSI (iSCSI) is a standard protocol for encapsulating SCSI

command into TCP/IP packets and enabling I/O block data transport over IP networks

• iSCSI adapters combines NIC and HBA functions. 1. take the data in block

form2. handle the segmentation

and processing with TCP/IP processing engine

3. send IP packets across the IP networkApplication

Layer

DriverLayer

Network Interface Card Storage HBAFC Storage iSCSI Adapter

IP Server FC Server IP Server

IP Packets

File Block Block

BlockBlock

IP Packets Intel GE 1000 TIP Storage Adapter

I 80200Processor

LinkLayer

IP Packets FC Packetson Ethernet on Ethernet


20

Comments on the Agata Event Builder• Agata Event Builder is not an issue (also now). CMS experiment has already

shown the ability to work with an order of magnitudine better that the Agatarequirements

• Agata could work, also in the prototype, fully on standard TCP/IP

• Agata could require an hpcc based Event Builder. Technologies already exist for that, but never applied to event builder problems. Should not be a big issue, but requires time to develop, debug, test, etc.– Myrinet (now)– 10 GEthernet on desktops (now, but expensive)– Infiniband (soon)

• All this does not mean Agata Event Builder is an easy task:– We have a 2 event builder chain (single detector level, global level) – 3 different numbers to be assigned (event number, buffer number, event

builder number). All of them have to be synchronized, etc. etc.

• All this means Agata Event Builder is a feasible task, also now; providing the proper resources (human in particular)


21

Processors and Storage trends

2007 CPU = 250 SI952010 CPU = 700 SI95

80 SI95 Now 250 GB Now Year 20071 disk = 1 TByte


22

Building Blocks for the Agata Farms

Configurations

Farms Type

1500201050020072002004

SI95 x BoxYear~ 1700 euro now

1 U CPU Box with 2 processors40 Boxes x Rack

1/21/10-

51-

1/41/20-

255

Nr. Racks

Nr. Racks

Nr. Boxes

Nr. Boxes

Nr. Racks

Nr. Boxes

1

204-Track, Farm

GLT

20040-Track. Farm

No GLT

102-Builder Farm

100020035PSA Farm

2010200 Detectors

200715 Detectors

20041 Detector


23

SW2

Blade Based Farms1 Blade Box with 2 processors14 Boxes x crate (7 U)6 Blade crates x rack = 108 BoxesPower = 16 KW x Rack

Configurations

Farms Typebackplane

SW130 Gbps backplane

~30000 euro now

~ 2150 euro now

1/5< crate-

22/5-

< crate

< crate-

102

Nr. Racks

Nr. Racks

Nr. Blades

Nr. Blades

Nr. Racks

Nr. Blades

1/3

204-Track, Farm

GLT

20040-Track. Farm

No GLT

102-Builder Farm

100020035PSA Farm

2010200

Detectors

200715 Detectors

20041 Detector

2 x 4 x 1 Gbps uplinks


24

On-line Storage• On-line Storage needs

– 1-2 weeks experiments– Max 100 TByte / experiment (no GBT)– Max 1000 TByte/year– 2010 1 disk = 4 Tbyte– Storage Agata System: 250 disks (+ 250 for mirroring)

• Archiving– O (1000) TB per year can not be handled as normal flat files

– Not only physics data stored• run conditions• Calibration

– Correlation between physics data, calibration and run condition are important for off-line analysis

– Data Base technology already plays an important role in physics data archiving (Babar, LHC experiments, etc.). Agata can exploit their experience and development


25

Storage Technologies Trends Application Servers

gateway

Infiniband

Commodity Storage Area Network share all the farms nodes. Technololgies interested for us are:- iSCSI over Giga (or 10 Giga) Ethernet- InfinibandFull integration between the SAN and the farm is realized if a Cluster File System is used.Example of Cluster File System are:- LUSTRE (www.lustre.org)- STORAGE TANK (IBM – soon off. released)

Data ServersGEth/iSCSI

SAN enabled disk array


26

Example of a iSCSI SAN available today

Application Servers

Data Servers

LSI logic iMegaRAID

SATA1

16

iSCSI ControllerRAID – SATA Controller

GEth/iSCSI

= ~ 5 Tbyte x controller2 x GEHost adapters:- Intel GE 1000 T- Adaptec ASA-7211- LSI 5201- ecc.

SATA = Serial ATA


27

Data Archiving: Advance Parallel Server

InternetIntranet

Low latency Interconnection(e.g. HPCC)

Data Servers

Storage Area Network

InputLoad balancing

switch

Shared Data Caching

Scalability

Example: Oracle Real Application Cluster


28

Run Control and Slow controlFront-end electronic and pre-processingSlow Control

Pulse Shape AnalysisRun Control and

MonitorSystem

Event Builder

Tracking

Storage


29Run Control and Slow Control Technological Trends

• Virtual Counting Room– Take a shift from distance– Tele-presence (web cam, videoconference and chatting, etc.)

• Web based technologies– SOAP– Web Services and Grid Services (Open Grid Service Architecture)– Data Base

• Demonstrators in operation at CMS test beams facilities


30

RCMS present demonstrators

SOAPCMS

Java MySQLOr

Oracle

Java TomCatContainers

OrGrid Services


31

Slow Control Trends• Ethernet every where

• Agata could be fully controlled by Ethernet connections, including the front end electronics

• This lead to have an homogeneous network avoiding the use of bridges between busses, software drivers to peform the bridging, etc.

• Embedded web server and embedded java virtual machines on the electronics

• Embedded Java should guarantee an homogeneous development environment, portability, etc.

Tini system

Xilink Virtex II Pro


32

Builder Network

B1

Event Builder

HPCC builder

Tracking Farm

Data Servers

T1 T2 3 full Blade Centers no GLT4 blades with GLT

2 Dual processor Servers + MyrinetB2

Storage Area network = iSCSI

1 16 1 16

SAN

8 + 8 TByte

Agata Demonstrator (2003-2007)

F1 F15F3F2Front-End

PSA Farm P1 P15P3P2 14 Blade Centers

15 x 2 Eth Switch

iSCSI Disk Array + SATA disks

1 Gbps100 Mbps


33

Project Break Down

B1

F1 F15F3F2

P1 P15P3P2

T1 T2

B2

1 16 1 16

SAN

Builder Network

1. Main Data Flow

3. Slow Control

2. Run Control &Monitoring


34

Human Resources• Agata has requirements very closed to the LHC experiments. We should

exploit the LHC experiments technologies in particular for:– Data Acquisition (framework, readout, event builder, etc.)– Run Control (framework for distributed system, basic GUI, etc.)– Slow Control– Data Storage (storage techniques, DBs, etc.)

• In the following table it is assumed to adopt CMS (a LHC experiment) technologies for most of the above items. The red numbers indicates the FTEs required to build the Agata applications over the CMS Frameworks.

1 FTE (LNL) + 3 FTE5 FTE (Cern)Main Data Flow (FE read out, event builders, storage, etc.

2 FTE4 FTE (LNL + Cern)Run Control & Monitoring

Slow Control

Agata’s Applications(fte x year)

Framework Development and maintenance (fte x year)

2004-2005-2006


35

Resources Distribution• We could organize 3 development centers

– Main Data Flow: Legnaro– Run Control: XX– Slow Control: YY

• Human resources should be concentrated in the related dev.centers– Legnaro: 4 FTE (1 from LNL , 3 from ext)– XX: 2– YY: ?


36

Conclusions(I)• No fundamental technological issues for the final Agata on-line system:

– The experiment requirements and the present understanding of the PSA algorithm fit with a final (2010) moderate size ( O(1000) machines) on-line system. Only a 3 times improvement in the PSA calculations lead to a system much more manageable (3 racks).

– Both network and event builder issues already fit with the today available technologies.

– Storage requirements (1000 TByte) fit with the evolution of the storage technologies.

• Demonstrator– Same architecture of the final system, only scaled down to the foreseen

number of detector.


37

Conclusions (II)

• Strong collaboration with LHC experiments and in particular with CMS is suggested, as we can benefits of:– Their developments– Support (partially)– Experience and advice


38

OFF-LINE INFRASTRUCTUREA GRID APPROACH


39

> = 10 Gbps links

T0

T1T1

T1

T1T1

T1

T1

T1

T2s

LHC Off-line InfrastructureCERN Data Production Center:- on-line system- online storage ( ~ PBytes)- central archive

Regional Computing Facilities (Country bounded):- computing power for analysis- on-line storage- local archive

Local (in the Country) Computing Facilities:- Dep. Farm Analysis

• Agata could exploit the LHC off-line infrastructure and in particular the Regional Computing (RC) centers

• Agata Data Production Center should be linked properly to such regional centers• typical experiment will take about 1 day to copy the entire data set• no tape copy; tape only for backup• New model for off-line analysis: Grid approach


40

World Wide Farming: the GRID– GRID is an emerging model of large distributed computing. Main

aims are:– Transparent access to multi-petabyte distributed data bases– Easy to plug in– Hidden complexity of the infrastructure

– GRID tools can help significantly the computing of the experiments promoting the integration between data centers and the regional computing facilities

– GRID technology fits the Agata experiment off-line requests

LHC Computing Grid(LCG)

InfrastructureCNAF/IT

RAL/UK

LYON/FR

KARLSRUHE/GE

FermiLab/USA

Brookhaven/USA

T0

T1

T1

T1

T1T1

T1


41

• EU has funded the DataGrid project. The main activities of the project have been:– Workload Management

• Services for grid scheduling and resource management

– Data Management• High speed data mover and replica

– Monitoring Services• Global states and error monitoring

– Local Farm Management• tools for new automated system

management techniques of large computing farms

• Based on Globus (Argonne) toolkit

• Evolution in EU FP6: EGEE

ResourceBroker

Wkld Mngm- find free resources

- submit jobsto grid-farms

Local jobsubmission

Data Mover

Local FarmManagement

DataGRID


42

Off-line Agata GRID: a possible scenario• Data are produced by a given experiment in the Agata Data Taken Center

• The experiment data set is temporary stored in the Center and optionally (if computing power is available) pre-processed with the aim to reduce their size. Original data set could be saved in the Center

• The (pre-processed) data set is then moved to the Regional Centers. (It could be moved directly to the final destination (lab, dep., etc.) if enough storage is available)

• Massive data processing (like sorting, matrix production, etc.) can be performed at the level of Regional Centers.

• Grid helps to provide the location of such data sets, indicating the computing power available and their location, etc. All the data sets are grid reachable and then available for processing to the whole collaboration. Outcomes are still grid reachable and then available for further analysis

• Final analysis can be performed exploiting all the Grid computing power available.


43On-line Agata Grid? Take control from a distance

UserInterface

SupportingServices

Farm

Serv

ices Instr.Instr.Farm

StorageServices

Instr.Instr.DBs

Virtual Control Room

Virtual Control Room

UserInterface

UserInterface

DAQSvc

TRGSvc

SlowSvc

TelprsSvc

Tn Farms

Raw datafor on-line monitoring, Special exp line,ProcessingFarm extension?

GRIDDomainGRID

Domain

Video Conf. & Chat Service On/Off-line DBs

e.g. CalibrationsData Repres.Service


44

Agata GRID: conclusions• GRID is becoming a serious and robust infrastructure, well funded and

supported (see EGEE EU project)

• All the LHC experiments (see LCG project: http://lcg.web.cern.ch/LCG/) will use it to integrate both the computing power and storage capability provided by the Regional Centers and by the production center (CERN) into a cooperative environment.

• Agata could join this infrastructure providing (or taken an agreement with) the needed computing power and storage to the Regional Centers.

• Agata, adopting the GRID middleware as proposed by LCG, has to take care of the impact on the development. This will be mainly on:– On/Off-line Data Bases of the experiment and the way to access them– Local farms (department level)– Production Center Farm (if any) – On-line software for what is concerning the DB access

• GRID technology could also be used to have a real time control of the experiment (control the experiment from a distance)

A DAQ Architecture for Agata experimentagata.pd.infn.it/documents/week9152003/GaetanoMaron.pdf · A...

Documents

Transcript of A DAQ Architecture for Agata experimentagata.pd.infn.it/documents/week9152003/GaetanoMaron.pdf · A...