Instruments and Sensors on the Grid Gaetano Maron [email protected] Istituto Nazionale di...

Instruments and Sensors on the Grid

Gaetano [email protected]

Istituto Nazionale di Fisica Nucleare – Laboratori di LegnaroLegnaro (PD), Italy

EU FP6 Project

The GridCC Project

G. Maron, KVI Seminar, Groningen, March 2007 2www.gridcc.org (EU FP6 Contract 511382)

GRIDCC project: goals & objectives Architecture overview Strategic targets Pilot applications Focus on the new provided capabilities

Instrument Element RMM-JMS as fast publishing system Virtual Control Room

Conclusions

Outline


GRIDCC project

Participant name Country

Istituto Nazionale di Fisica Nucleare Italy

Institute Of Accelerating Systems and Applications Greece

Brunel University UK

Consorzio Interuniversitario per Telecomunicazioni Italy

Sincrotrone Trieste S.C.P.A Italy

IBM (Haifa Research Lab) Israel

Imperial College of Science, Technology & Medicine UK

Istituto di Metodologie per l’Analisi ambientale – Consiglio Nazionale delle Ricerche

Italy

Universita degli Studi di Udine Italy

Greek Research and Technology Network S.A. Greece

Grid Enabled Remote Instrumentation with Distributed Control and Computation It is a 3-years EU FP6 project started in September 2004 Web site: www.gridcc.org


1. Remote control and monitoring of complex and distributed instrumentation

2. Tight integration between instrument grid and classical computational grid

3. Human interaction with Grids via Virtual Control Room (collaborative environment)

4. Enactment of complex workflows

GRIDCC: Goals & Objectives


Execution ServicesCollaborative Environment

GRIDCC: Architecture (I)Instruments Grid Computational Grid

IE

IE

CE

CE

SE

Instrument Element

Problem Solver

InstrumentManager Information &

Monitoring System

DATA

ES

VCR

VCR

VCR

VCR

Workflow

Service Oriented Architecture – SOA

The GRIDCC services are Web Services compliant, according to the WS-I convention.

Any type of client WS-I compliant, can access the GRIDCC services without any specific software library.

X509 based security is used When performances are

required , a kerberos based mechanism is used to access IEs.


GRIDCC Architecture

Web ServiceInterface

Execution

Service

WfMS

WMS

AS

Instrument elements

(IE)StorageElement

(SE)

Instrument elements

(IE)StorageElement

(SE)

Instrument Element

(IE)StorageElement

(SE)

Compute element

(CE)

Compute element

(CE)

Computing Element

(CE)

StorageElement

(SE)

StorageElement

(SE)

StorageElement

(SE)

Global ProblemSolver

Information and Monitoring

Services(IMS)

Information System(BDII)

Security Services

Virtual Control

Room

Virtual Control

Room

CollaborativeServices

(CS)

WMS Work Management System

WfMS Work Flow Mng System

AS Agreement Service

Broker


GridCC PermanentTest Bed

VOMSVOMS

AS, KrbServerAS, KrbServer

VCR, SE, IE, BDIIVCR, SE, IE, BDII

IE, CE/CREAM,SE, LFC IE, CE/CREAM,SE, LFC

WMProxyWMProxy

WfMS WfMS

VCR, IE VCR, IE


Strategic plan: 3 main target areas(Large-scale) scientific

experimentsHigh energy particle physics

(Radio-) Telescopes

Remote process control Accelerator control (Tele-) Biomedicine

Robotics Automotive

Electronic microscopes

Widely Sparse Instrumentation Power Grids

Monitoring of the territory Monitoring of the seaGeo-hazard prediction

Distributed laboratories Transportation monitoring

Sensor network

GRIDCCMiddleware


GRIDCC pilot applications

Power Grid

Power Grid

High Energy Physics

CMS

High Energy Physics

CMS

Particle Accelerator

Particle Accelerator

Device Farm

Device Farm

Geohazard MonitoringGeohazard Monitoring

MeteorologyMeteorology


The GridCC middleware has been deployed to control the run of the CMS (Compact Muon Solenoid), one of the four high energy experiments in LHC (Large Hadron Collider) at CERN laboratory.

CMS Magnet Test and Cosmic Challenge (MTCC), a milestone in the CMS construction, positively carried out.

Pilot applications: CMS - I

Top IE

CSCIE

TrackerIE

HCALIE

DAQIE

RPC IE

TriggerIE

DTIE

ECAL IE

DAQIM

FB RB FF

xdaq

xdaqxdaq

DAQRS

DAQIMS

UserInterface

CMS Control Structure

CMS Detector


Pilot applications: Power grid

Instrument Manager

Instrument Element

...

Virtual Control Room


Gas

Solar Panel

Power Grid V.O.

GRIDCC deployed to monitor:• A 50kW generator• A 1 kw Photo-Voltaic array

Power Grid. GRIDCC deployed to monitor a 50kW generator and a 1 kW Photo-Voltaic array


Pilot applications: Remote Operation of an Accelerator

Elettra Synchrotron


Meteorology (Ensemble Limited Area Forecasting) Weather forecasting system to detect hazardous weather

Device Farm for the Support of Cooperative Distributed Measurements in Telecommunications and Networking Laboratories The Device Farm consists of a pool of Measurement

Instruments for Telecommunication Experiments

Geo-hazards: Remote Operation of Geophysical Monitoring Network The monitoring net will be characterized by different levels of

activity: stand-by, pre-alert, alert, plus a control modality An event worth to be monitored is for example the evolution

of a “landslip”

GRIDCC: other applications


Instrument Element Complete integration of Instruments into SOA. Instruments become

“services” accessible via standard Web Service Complete integration into the computational and storage GRID with the

embedded Sorage Element (SE) High virtualization and abstraction of diverse physical devices A very scalable and platform-independent framework (Java based)

Fast Data Publishing System High-performance reliable multicast via RMM-JMS

Virtual Control Room Allows effective human interactions with the grid Provides a homogeneous collaborative environment

Focus on the new capabilities provided by GRIDCC


The IE permits SOA accessible operations to control and monitor the instruments (via VIGS), such as:

• execute a command• get / set parameters

different data outputs:• data mover to/from a grid Storage Element (via SRM) • high bandwidth channel for data publishing (via IMS).• low bandwidth channel for logs, states etc. (via IMS).

Instrument Element: Versatile I/O multichannel

VIG

SCommands

Status

Data Subscribers

Grid Interaction

SR

M

VCR

StorageElement

(SE)

StorageElement

(SE)

StorageElement

(SE)

Logs, Errors,

States, Monitors

Instruments

IMS

ParametersES

VIGS Virtual Instrument Grid ServiceIMS Information and Monitor ServiceSRM Storage Resource Management

IE


The Instrument Element (IE) is one of the novel elements of the GRIDCC architecture.

It offers a standard web service interface to integrate scientific and general purpose instruments and sensors within the grid.

An IE can control a set of instruments with the possibility to decide on which sub-set to operate.

The IE is the key decoupling element between real instruments and the grid

Instrument Element: Instruments as Services

Web Services

Instrument Element

Any Protocol or physical connection

Sensor Network

Instrument

Instrument

Service Oriented Access

Instrumentation


Instrument Element: Instrument Virtualization

Instrument

Parameters

Attributes

XMLBased

Language

ControlMode

The IE grid instruments representation is basically based on: providing the XML description of the

instruments managed by the IE providing the Finite State Machine that defines

the dynamic behaviour of the control performed on the set of instruments managed by the IE

simple grid accessible operations to control and monitor the instruments like execute a command, get or set a parameter or a set of parameters of the instrument (VIGS).

providing different data acquisition output paths ranging from the accessing of a Grid Storage Element (SRM/SE) to a very efficient communication channel when an high bandwidth output stream is required (via IMS).

Instruments

VIG

S

SEIE

IMS


VIG

S

IMSProxy

ControlManager

DataCollector

Real Instruments

Data Flow

Control Flow

State FlowError FlowMonitor Flow

The term Instrument Element describes a set of services that provide the needed interface and implementation that enables the remote control and monitoring of physical instruments.

InputManager

EventProcessor

FSMEngine

ResourceProxy

Control Manager

IMSInstrument Element Architecture

ResourceService

IMS

ProblemSolver

InstrumentManager

Instrument Element

Data Publisher

Acc

ess

Con

trol

Man

ager

SRM/SEGridFTP

SE

IMS

Instrument Protocols


Instrument Element Framework Characteristics

The IE framework is modular, fully customizable and can be adapted to different environments and different type of instruments to be controlled.

Customizable plug-in modules are used to interface the framework with the communication protocol (both hardware and software) used by the instruments.

the dynamic behaviour of the control that has to be performed is programmable via the built in Finite State Machine (FSM).

relatively fast control feedback loops can be provided using the customizable built in Event Processor (EP).

Error repairing actions due to malfunctioning of the instruments can even be handled by EP or by the Local Problem Solver (LPS).

IE

ResourceService

IMS

LocalProblem

Solver

InstrumentManagerA

cces

s C

ontr

ol M

anag

er

DataMover

InstrumentManager

VIG

S

InstrumentInstrumentInstrumentInstrument

IMSProxy

ControlManager

DataCollector

InputManager

EventProcessor

FSMEngine

ResourceProxy

Control Manager


VIG

S

IMS

Instrument Manager

ResourceService

IMS

ProblemSolver

InstrumentManager

Instrument Element

Data Publisher

Acc

ess

Con

trol

Man

ager

SRM/SEGridFTP

SE

IMS

InstrumentManager

IMSProxy

ControlManager

DataCollector


21

Instrument Manager

IM is composed by 3 main components:- Control Manager:

- Input Manager. It handles all the input events of the IM. These includes commands from GUIs or other IMs,errors/state/log/monitor messages. - Event Processor. It handles all the incoming message and decide where to send them. It has processing capability

- FSM. A finite state machine is implemented - Resource Proxy. It handles all the outgoing connections with the resources.

- Data Collector. It get data from the controlled instruments and make them available to the data mover. A local storage of the data is even foreseen.- IMS Proxy. It receives error/state/log/monitor information from the controlled resources and forward them to IMS

IMSProxy

DataCollector

Instrument Manager

InputManager

EventProcessor

FSMEngine

ResourceProxy

Control Manager

Instruments

Data Flow

State Flow

Error Flow

Monitor Flow

Control Flow

Customizable Plug-in modules to interface to the instruments


Status• data collection • data publishing on RMM-JMS • data on file in progress• data on DB to be started• sync commands (enable/disable done trough the “set param” of the IE façade (VIGS)

Data Collector

ResourceService

IMS

LocalProblem

Solver

InstrumentManager

Instrument Element

IMSProxy

ControlManager

DataCollector

Acc

ess

Con

trol

Man

ager

InputManager

EventProcessor

FSMEngine

ResourceProxy

Control Manager

DataMover

VIG

SIE File

System

RMM-JMSDB persistency

enable/disabledata stream

InstrumentInstrumentInstrument

Instrument

data stream

Data Subscribers

(Monitor, d

ata analysis, etc.)


VIG

S

IMSProxy

ControlManager

DataCollector

IMS

Resource Service

ResourceService

IMS

ProblemSolver

InstrumentManager

Instrument Element

Data Publisher

Acc

ess

Con

trol

Man

ager

SRM/SEGridFTP

SE

IMS

ResourceService


Resource Service Architecture

The Resource Service (RS) handles all the resources of an IE and manages their partition (if any). A resource can be any hardware or software component involved in the IE (instruments, Instrument

Managers, IMS components) RS stores the configuration data of the resources and download them to resource target when

necessary Resources can be discovered, allocated and queried. It is the responsibility of the RS to check resource availability and contention with other active

partitions when a resource is allocated for use. A periodic scan of the registered resources keeps the configuration database up to date.

DiscoveryManager

SubscribeManager

Partition&LockManager

ConfigurationManager

Available Resources

PartitionDefinitions

ConfigurationDefinitions

RS

Dat

a B

ases

Partition/Configurationretrieve methods

Partition and Locksetting methods

Configurationsetting methods

Discoverymethods


Technology used http://www.jxta.org/

Resources Discovery


VIG

S

ControlManager

DataCollector

IMS

Information and Monitor Service

ResourceService

IMS

ProblemSolver

InstrumentManager

Instrument Element

Data Publisher

Acc

ess

Con

trol

Man

ager

SRM/SEGridFTP

SE

IMS

IMS

IMSProxy


27

Information and Monitor System (IMS)

The Information and Monitor Service (IMS) collects messages and monitor data coming from the IMS Proxy of the IMs.

The messages are catalogued according to their type, severity level and timestamp. Data can be provided in numeric formats, histograms, tables and other forms.

The IMS collects and organizes the incoming information either in a database or a pub/sub system or both. It publishes the data to subscribers. These subscribers can register for specific messages categorized by a number of selection criteria, such as timestamp, information source and severity level.

IMSProxy

IMSProxy

IMSProxy

PE

RS

IST

EN

CY

Pub

lish

/ S

ubsc

rib

SU

BS

CR

IBE

RS

SU

BS

CR

IBE

RS


IMS Fast Data Publishing: RMM-JMS

IE

IE D

ata

Pro

duc

er

Dat

a S

ubsc

ribe

rs Same data are sent to several subscribers.Multicast protocols can have a benefic impact on performances

• JMS provides a standard set of APIs for the communication• Many commercial and academic JMS implementations both in C/C++ and Java (NaradaBrokering, Sun, IBM)• GRIDCC (IBM Haifa lab) has implemented a Reliable Multicast protocol (RMM) JMS compliant• RMM-JMS works within a LAN but an efficient bridge technology has been developed to allow inter-LAN multicast communication


VIG

S

IMSProxy

ControlManager

DataCollector

IMS

Problem Solver

ResourceService

IMS

ProblemSolver

InstrumentManager

Instrument Element

Data Publisher

Acc

ess

Con

trol

Man

ager

SRM/SEGridFTP

SE

IMS

ProblemSolver


Problem Solver

IMSProxy

ControlManager

Instrument Manager

Pub/Sub

IMSProxy

ControlManager

Instrument Manager

IMSProxy

ControlManager

Instrument Manager

IMSProxy

ControlManager

Instrument Manager

DBData Mining Tools

Algorithms evaluations :Rule Induction, Tree, Functions, Lazy, Clusters and Associative

State FlowError FlowMonitor Flow

On Line Analisys

Problem Solver

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

iris

glas

s

brea

st c

ance

r

bupa

votin

g-re

cord

s

hous

ing

bala

nce-

scal

e

Bre

ast

Can

cer

Wis

cons

in

Pim

a-In

dian

s-D

iabe

tes

tic-t

ac-t

oe

Seg

men

t

Seg

men

tatio

n

Sic

k-eu

thyr

oid

Pag

e-B

lock

s

mus

hroo

m

Shu

ttle

(2)

Lett

erR

ecog

nitio

n

krko

pt

Shu

ttle

(1)

conn

ect-

4

dataset

accu

racy

Average Rule Accuracy

Average Tree Accuracy

Average Function Accuracy

Average Instance Accuracy

Average Cluster Accuracy

Step 1 The control manager can perform an autonomous recovery action where

the cost for the determination it is not so heavy .

Step 2 Persistent information can be analyzed in order to extract knowledge

Step 3 On-line information can be analyzed in order to detect possible

malfunctions


Status: installed and running• SRM 2.2 • STORM implemetation (INFN)• Bestman implementation (Berkeley))• XFS IE file system

Embedded SRM compliant Grid Storage Element (SE)

ResourceService

IMS

LocalProblem

Solver

InstrumentManager

Instrument Element

Acc

ess

Con

trol

Man

ager

SEBackend

VIG

SIE File

System

SR M GRID

accessibile data


Instrument Element: Scalable on embedded systems

FPGAPPC405

Xilink Virtex IV

Custom Board

Cus

tom

Ele

ctro

nics

1 Gbps Ethernet

Web

Ser

vice

Grid

CC

IM

Java

VM

Linu

x

USER INTERFACE

Custom Logic

Montavista www.montavista.com

JamVM http://jamvm.sourceforge.net

Standalone Axis

Instrument Manager on a chip …… towards a grid on a chip

IEInstrument Manager


IE technologies Web Service compliant (WS-I)

Tomcat + Axis (and Java) and Axis standalone are the main technologies of the IE

All the services are deployed on a single or multiple instances of Tomcat, according to the needs of the application

Message oriented middleware (Pub/Sub) is based on the Java Messaging System (JMS). The following implementations are used in the project

Sun Narada Brokering RMM - JMS (GridCC IBM)

MySQL and Oracle are used as Data Base for the RS

SRM interface version 2.2 used


The VCR is a collaborative web portal Enable the user to run, aggregate and display plug-ins that act as “mini applications”



Conclusion - I The GridCC project is integrating instruments into the “classic”

computational/storage Grids.

Novel concepts introduced by GridCC are: The Instrument Element, allowing a virtualisation of the instruments to

be controlled and their insertion in a Grid The Virtual Control Room, providing an highly interactive environment

with IEs/CEs/SEs. VCR even provides cooperative tools to allow the cooperation (logbook, chat, videoconf., etc.) between remote users

hard (reservation of IE) and soft (statistical prediction) guarantees of the IE’s methods execution times

Fast Data Publication via a Message Oriented middleware (RMM-JMS) to distribute data and information from an IE to the world wide Grid.

The IE is highly customizable and can be adapted in different environments.


Conclusion - II The IE can be shrunk down into a chip allowing grid enabled embedded control

of the instrumentation

Several heterogeneous pilot applications are deploying and running the IE middleware

We support and encourage the adoption of our middleware in other projects/experiments.


Any Questions?

Thank you for your attention


Spare

Tecnologie usate http://www.jxta.org/

http://gladgw:2002/InstrumentElementKeeper/ da dove vedere la lista degli strumenti

http://gladgw.lnl.infn.it:2002/SeedsFinder/ da dove scaricare l’IE da web-start


GridCC Scalability DemoImperial

Brunel

Elettra

IBM

CNIT

IASA

IMAA

GRNet

UniUD

INFN

VCR Monitor GUI

Network Information

Provider

InstrumentElement

Network CoreMachine


Pilot applications: CMS - II

CMS Detector

2 107 electronics channels 40 MHz

100 Hz

O(104 ) distributed Objects to be controlled configured monitored

On-line diagnostics and problem solving capability

Highly interactive system (human reaction time - fractions of second)

World Wide distributed monitor and control


Instrument Element – Interconnections II

IE

IE

IE

VCRControlPanel

Computing Element

(CE)

Fast DataDisplay

Fast Data Publishing

Information and Monitor SystemLog Display

Con

trol

and

Sta

tus

Exi

stin

g G

rid E

lem

ents

LogPersistency

DataConsumer

WorkFlowMng.

System

Storage Element

(SE)


- WS-I, http://www.ws-i.org - Java Message Service (JMS), http://java.sun.com/products/jms - RMM-JSM,

http://www.haifa.ibm.com/projects/software/rmsdk/gridcc.html - gLite, http://glite.web.cern.ch/glite/ - SRM, http://sdm.lbl.gov/srm-wg - StoRM, http://grid-it.cnaf.infn.it/storm - Montavista, http://www.montavista.com - JamVM, http://jamvm.sourceforge.net

References


The INFN team working in Legnaro, in collaboration with

CERN at Geneva:

E. Frizziero M. Gulmini F. Lelli G. Maron A. Petrucci S. Traldi

Acknowledgments


Instrument Element: Versatile I/O multichannel

VIG

SCommands

Status

Data Subscribers

Grid InteractionSE

1) A Virtual Instrument Grid Service (VIGS) interface has been defined. The VIGS provides a Web Service acccess to the instrumentation

2) The Data Publishing (DP) channel disseminates the acquired information to the data subscribers

3) A SRM/SE interface is provided to allow file movement with other Grid Element, like SE, CE and other IEs.

4) Information messages (logs, errors, etc.) are published from the IE (via IMS)


Execution

Service

WfMS

WMS

AS

Compute element

(CE)

Compute element

(CE)

Computing Element

(CE)

StorageElement

(SE)

StorageElement

(SE)

StorageElement

(SE)

IE

IE

Logs, Errors,

States, Monitors

IMS

DP

VIGS Virtual Instrument Grid ServiceIMS Information and Monitor ServiceSE Storage ElementDP Data Publishing


Fast Data Publishing: Some results

Msg rate- msg size 100Bytes

1000

10000

100000

1000000

1 2 5 10 15 20 25 30

Number of Publishers

ms

g/s

ec RMM

MQ3.6

Manta


1000

10000

100000

1 2 5 10 15 20 25 30


ms

g/s

ec RMM

MQ3.6

Manta


1000

10000

1 2 5 10 15 20 25 30


ms

g/s

ec RMM

MQ3.6

Manta

• 32 Dual Xeon 2.4GHz 1.5GB RAM machines, 1 GB Ethernet switch • At most 1 publisher, subscriber, or broker- (Sun MQ3.6) per machine • No message lost• RMM throughput: 75-90 Mbytes/sec. (for 5 and more publishers)

Message Rate: Case many-to-one


Architecture: New “instruments grid”

Component Name Description

Instrument Element (IE)This is a unique concept to GRIDCC. It consists of a coherent collection of services which

provide all the functionalities to configure, partition and control the physical instruments

Information & Monitor Service (IMS)

It gathers from GRIDCC resources information and monitor data to be disseminated

through a publish / subscribe systems or to be stored in persistent repositories.

Problem Solver (PS)

It offers automated problem solving in a Grid environment at two levels. A local PS,

within a given Instrument Element, allows to solve local problems related to functionalities

of a given instrument. A global PS, allows to solve system-wide problems.

Virtual Control Room (VCR)

It provides a common set of collaboration tools and allows users to build complex

workflows, which are then submitted to the Execution Services, and to directly monitor

and control remote instruments in real-time.

Execution Services (ES)

They control the execution of the workflows defined by the user in the VCR, maintaining

the status of the tasks that make up the workflow.

They also support the advance reservation of resources.

Security Services (SS)

GRIDCC uses a split security system. When interacting with components of other Grids

the GSI security will be used and the users identified by their X.509 proxy certificate.

When interacting with the IE the user will be identified by a Kerberos ticket.

Instruments and Sensors on the Grid Gaetano Maron [email protected] Istituto Nazionale di...

Documents

Transcript of Instruments and Sensors on the Grid Gaetano Maron [email protected] Istituto Nazionale di...