Derived Metrics Prototyping

Dennis.Waldron@cern.ch

IT-FIO-FS

Dennis Waldron: Derived Metrics Prototyping – n° 2

Objective

Initially: How to calculate derived metrics in a global context?.

Expanded: Evaluation of Heidelberg Fault Tolerance package in the context of a

global framework of recovery.

Note: Derived metrics also referred to as: Global correlations

Combined metrics

Dennis Waldron: Derived Metrics Prototyping – n° 3

Motivation

Produce a: Simple

Extensible

Highly Configurable

Powerful

A prototype sensor exists ‘CMsensor’ Coded in PERL

Adheres to WP4 architecture

Can be used in both local and global context Limitations apply to global context

Dennis Waldron: Derived Metrics Prototyping – n° 4

Architecture

Measurement Repository

MR Server

(OraMonServer)

Database

MR API

User

System Boundary

CMsensor

XML Interpreter

MRs sensorAPI

SensorCache

Collector Agent

(MSA)

Sensor

Sensor

Node

1…*

XMLXML

XML

Configuration File(s)Subscription

Dennis Waldron: Derived Metrics Prototyping – n° 5

CMsensor

Responsible for: Configuration Management

Autoload of configuration changes on the fly, instantiating, removal, alteration

Subscription Management Manages active subscriptions, re-subscription requests, shutdowns,

reconnects, etc…

Metric Triggers Calculation of derived metrics can be triggered by:

MSA scheduling (GET request) Subscription callback

Data Caching

Rule Evaluation and Error Handling

Dennis Waldron: Derived Metrics Prototyping – n° 6

CMsensor cont.

1 XML file exists for each derived metric - defines: Metric name and description

Subscription requirements

Metric processing code (rule)

Example 1:

<sensorMetric name=“example1” description=“this is an example metric”>

<metric>10002</metric>

<rule>

# PERL code here

$value = &getMetric($host, $metric) * 50;

# return value to MR

&storeSample(03, $mid, 0, $host, $value);

</rule>

</sensorMetric>

Note: $host, $metric, $mid (re-injection id) locally scoped at execution

Dennis Waldron: Derived Metrics Prototyping – n° 7

CMsensor cont.Example 2:

<sensorMetric name=“daemonUp” description=“Does at least 1 instance of a daemon exist across a defined list of nodes”>

<rule>

# PERL code here

my $retval = 0;

my $node;

my @node_list = split(/ /, $params);

foreach $node (sort @node_list) {

if (&getMetric($node, 12345) == 1) {

$retval = 1;

last;

}

}

&storeSample(01, $mid, 0, $retval);

</rule>

</sensorMetric>

Dennis Waldron: Derived Metrics Prototyping – n° 8

CMsensor cont.

A sensorAPI PERL module implementing the latest ASCII MSA – MS protocol (v1.3) now exists.

Sensor utilises the new PERL simplified API for MR access.

No hard coded metrics!

All fatal messages are trapped, logged and appropriate action triggered.

System allows for access to alternative sources of information other the central measurement repository e.g. LSF.

Dennis Waldron: Derived Metrics Prototyping – n° 9

Limitations Large volume re-injection causes performance issues with the MSA.

The impact of large volume insertions with the MR is unknown.

Derived metrics requiring a lot of processing can cause a backlog of metrics to be processed, hence some schedule executions may be skipped.

The extraction of large volumes of data takes a long time. Can retrieve approx 266 metrics/second through the simplified API in comparison to 4668 through direct SQL

calls.

Current MR queries limited to 12,000 values EDG Bugzilla 2380, 2381

Segmentation Faults in subscription mechanism via PERL MRs EDG Bugzilla 2320, 2366

CMsensor lacks some functionality

Dennis Waldron: Derived Metrics Prototyping – n° 10

Questions?