CN8861Network Management

Lecture-615 June 2014

Govi Ravindran

Dept. of Electrical & Computer Engineering

Ryerson University

Section 1Distributed Management

RFC References – Distributed Management

RFC 3165, “Definitions of Managed Objects for the Delegation of Management Scripts”

RFC 3231, “Definitions of Managed Objects for Scheduling Management Operations”

RFC 2982, “Distributed Management Expression MIB”

Overview

Distributed Management System– Definition, Characteristics, and Technologies

Distributed Management MIBs– Script MIB– Schedule MIB– Expression MIB

Distributed Management Systems

Complexity of Management Tasks– In a centralized system, the central manager carries out the

entire management task.– In a weakly distributed system, complex tasks are executed at

the top level manager whereas simple management tasks are delegated to mid-level managers.

– Strongly distributed systems decentralize management processing to each and every network element; management tasks are no longer confined to managers.

Distributed Management Systems

Computational load on the top-level manager is reduced by delegating functions elsewhere.• Resource requirements at the top-level manager is reduced.

The communication load on the network is reduced.• By locating mid-level managers as close to the devices as

possible, only processed data moves between managers.• With dynamic delegation, management tasks can be assigned

and changed dynamically.

Distributed Management MIBs

IETF Distributed Management (DISMAN) Working Group had proposed standard MIBs to realize – Distributed Managers and/or – Self-Managing devices

The DISMAN MIBs that are of interest include:– Schedule MIB– Script MIB– Expression MIB

IETF Distributed Management MIBs

mib-2

Schedule MIB(schedMIB)

Script MIB(scriptMIB)

{mib-2 63} {mib-2 64}

Event MIB(dismanEventMIB)

{mib-2 88}

Expression MIB(dismanExpressionMIB)

{mib-2 90}

{mgmt 1}

mgmt{internet 1}

internet{.iso.org.dod 1}

Script MIB

Script MIB defines managed objects for delegation of management functions.

Management functionality is delegated as scripts which are transferred to the location where they are executed.

The Scripts can be started remotely, arguments can be passed to them, and results can be returned back to the initiator.

Script MIB - Capabilities

Script MIB provides following capabilities: – Transfer management scripts to a distributed manager.– Transfer arguments for management scripts. – Initiate, suspend, resume or terminate management scripts. – Monitor and control running management scripts.

– Transfer the results produced by the management scripts.

Script MIB – Script Transfer

There are two different ways to transfer management scripts to a distributed manager: – Push Model: The high-level manager pushes the script to the

distributed manager. – Pull Model: The manager tells the distributed manager the

location of the script and the distributed manager retrieves the script itself.

The Script MIB also supports management scripts that are part of the Script MIB implementation.

Scripts are stored in an non-volatile storage. This allows a distributed manager to restart scripts.

Every script is identified by an administratively assigned name.

Script MIB – Framework

Script MIB – Script Transfer

The ‘push model’ is realized by a table which allows a manager to write scripts by sending a sequence of SNMP set requests.

The ‘pull model’ is realized by the use of URLs that point to the script source.– The manager writes the URL by an SNMP set request. – The distributed manager is then responsible for retrieving the

script using the protocol specified in the URL (such as FTP or HTTP).

Script MIB – Script Execution

The Script MIB allows execution of management scripts– Arguments can be passed to a script.– Scripts can return a single result value.– Scripts can also export complex results. For instance, a URL can

be returned that points to a file containing script output.

When runtime errors terminate active scripts, an exit code and an error message is left in the MIB.

Running scripts have associated status object which allows script execution to be suspended, resumed, or aborted.

A history of finished scripts is kept in the MIB.

Distributed Management Script MIB

smTraps

scriptMIB{.iso.org.dod.internet.mgmt.mib-2 64}

smObjects smConformance{scriptMIB 2}

{scriptMIB 3}

smExtsnTable

smScriptException

smNotifications

{scriptMIB 1}

smLangTable

smScriptObjectssmRunObjects

smScriptAbort

smScriptResult

smScriptTable smCodeTable

smLaunchTable smRunTable

Script MIB – Language Group

Language Group – The smLanguageGroup is used to provide information about

the languages and the language extensions supported by a Script MIB implementation. This group includes two tables. The smLangTable lists all languages supported by a Script MIB implementation and the smExtsnTable lists the extensions that are available for a given language.

Script MIB – Script Group

Script Group – The ‘smScriptGroup’ consists of a single table called

‘smScriptTable’ which lists all scripts known to a Script MIB implementation.

– The source of a script is defined by the ‘smScriptSource’ object. – ‘smScriptSource’ may contain a URL pointing to a remote

location. – If the ‘smScriptSource’ is null, the script source is read from the

smCodeTable or from local storage. – The ‘smScriptStorageType’ object is used to distinguish between

scripts read from local storage and scripts read from the smCodeTable.

Script MIB – Script Group

Scripts are automatically loaded once the ‘smScriptAdminStatus’ object is set to ‘enabled’. – Loading a script includes retrieving the script, compiling (if

required), and making the code available to the runtime system. The ‘smScriptOperStatus’ object is used to indicate the

status of the loading process. – ‘smScriptOperStatus’ object will start in the state ‘retrieving’,

switch to the state ‘compiling’ and finally reach the state ‘enabled’.

– Errors during the retrieval or compilation phase will result in an error state such as `compilationFailed'.

Script MIB – Script GroupsmScriptObjects

{scriptMIB.smObjects 3}

smScriptTable{smScriptObjects 1}

smCodeTable

smScriptOwner

smScriptName

smScriptDescr

smScriptLanguage

smScriptSource

smScriptAdminStatus

smCodeIndex

smCodeText

smCodeRowStatus

{smScriptObjects 2}

smScriptOperStatus

smScriptStorageType

smScriptRowStatus

smScriptError

smScriptLastChange

Script MIB – Code Group

The smCodeGroup consists of a single table, the ‘smCodeTable’ – ‘smCodeTable’ provides the ability to transfer and modify scripts

via SNMP set requests. – A script can be fragmented over multiple rows of the

smCodeTable in order to handle SNMP message size limitations.

Script MIB – Launch Group

The smLaunchGroup contains a single table, the smLaunchTable.

The smLaunchTable allows the following operations: – associate a script with an owner used during script execution– provide arguments and parameters for script invocation– invoke scripts with a single set operation

Script MIB – Launch GroupsmRunObjects

{scriptMIB.smObjects 4}

smLaunchTable{smRunObjects 1}

smLaunchOwner

smLaunchName

smLaunchScriptOwner

smLaunchScriptName

smLaunchArgument

smLaunchMaxRunning

smLaunchMaxCompleted

smLaunchLifeTime

smLaunchExpireTime

smLaunchStart

smLaunchControl

smLaunchAdminStatus

smLaunchOperStatus

smLaunchRunIndexNext

smLaunchStorageType

smLaunchRowStatus

smLaunchError

smLaunchLastChange

smLaunchRowExpireTime

Script MIB – Run Group

Every row in the smRunTable contains the argument passed during script invocation, the result, the script exit code, run state, and start and end time-stamps.

There are three writable objects in the smRunTable: – The smRunLifeTime object defines the maximum time a running

script may run before it is terminated by the Script MIB implementation.

– The smRunExpireTime object defines the time that a completed script can stay in the smRunTable before it is aged out.

– The smRunControl object allows running scripts to be suspended, resumed, or aborted.

Script MIBsmRunObjects

{scriptMIB.smObjects 4}

smRunTable{smRunObjects 2}

smRunIndexsmRunArgument

smRunStartTime

smRunEndTime

smRunLifeTime

smRunExpireTimesmRunExitCodesmRunResultsmRunControl

smRunStatesmRunError

smRunResultTime

smRunError

IETF Script MIB

IETF Script MIB

Script MIB – Application

Software update:– Updating and configuring software on a managed node– Following scripts can be used to accomplish the above:

a script checking the version currently installed, a script to stop and uninstall the old software release, an install script for the new release, and a a configure script for the new release

Schedule MIB

Schedule MIB supports scheduling of actions periodically or at specified dates and times.

The actions can be used to trigger management functions in a distributed manager.

Schedules can be enabled or disabled by modifying a control object.

Schedule MIB – Schedules

Periodic schedules are based on fixed time periods between the initiation of scheduled actions.

Calendar schedules trigger scheduled action at specified days of the week and/or days of the month.

One-shot Schedules are similar to calendar schedules. The difference is a one-shot schedule will automatically disable itself once it has been invoked.

Schedule MIB – Actions

Scheduled actions are invoked by SNMP set operations on local MIB variables.

Scheduled actions in the Schedule MIB are further restricted to objects of type INTEGER.– Simple actions such as setting a status MIB object (e.g.

ifAdminStatus). – Complex actions such as triggering a management script.

Distributed Management Schedule MIB

schedTraps

schedMIB{.iso.org.dod.internet.mgmt.mib-2 63}

schedObjects schedConformance{schedMIB 2}

{schedMIB 3}

schedTableschedActionFailure

schedOwner

schedName

schedDescr

schedInterval

schedWeekday

schedMonth

schedDay

schedHour

schedMinute

schedValue

schedType

schedAdminStatus

schedOperStatus

schedFailures

schedLastFailure

schedLastFailed

schedNotifications

{schedMIB 1}

schedLocalTime

schedVariable

schedContextName

schedStorageType

schedRowStatus

schedTriggers

Expression MIB

The Expression MIB is a powerful way to monitor large, complex systems.

Expression MIB is a way to create new, customized MIB objects for monitoring. – Expression MIB supports externally defined expressions of

existing MIB objects. – In the Expression MIB the results of an evaluated expression are

MIB objects that are used like any other MIB objects. – Without these capabilities monitoring would be limited to the

objects in predefined MIBs. The MIB is usually appropriate in a relatively powerful,

resource-rich managed system and not necessarily in a severely limited environment.

Expression MIB - Sampling

The MIB supports three types of object sampling for the MIB objects that make up the expression: absolute, delta, and changed. – Absolute samples are simply the value of the MIB object at the

time it is sampled. – Delta sampling requires the implementation to maintain the

value of the last sample.– Changed sampling is a boolean indicating whether or not the

object changed value since the last sample.

Expression MIB - Evaluation

If there are no delta or change values in an expression, the evaluation occurs on demand, i.e. when a requester attempts to read the value of the expression. In this case the requester gets a freshly calculated value.

For expressions with delta or change values, evaluation goes on continuously, every sample period. In this case requesters get the value as of the last sample period.

Simple Expression MIB

No wildcards– Such an implementation would allow expressions made up of

individual MIB objects but would not be suitable for expressions applied across large tables as each instance in the table would require a separate expression definition.

– Furthermore it is suitable for tables with arbitrary, dynamic instances, as expressions definitions could not predict what instance values to use.

– Significantly reduces retrieving and processing complexity No deltas

– Leaving out delta processing significantly reduces state that must be kept and processing burden.

– Unfortunately it makes expressions on counters unusable, as counters have meaning only as deltas.

– An implementation without deltas might be useful for a resource limited, self-managing system that has no need for expressions or events on counters.

Expression MIB – Structure

Resource Section Has objects to manage resource usage by wildcarded delta

expressions, a potential major consumer of CPU and memory. Definition Section

Contains the tables that define expressions. The expression table contains parameters that apply to the

entire expression, such as the data type of the result, and the sampling interval if it contains delta or change values.

The object table contains the parameters that apply to the individual objects that go into the expression, including the object identifier, sample type, discontinuity indicator, and such.

Value Section Contains the values of evaluated expressions. For a given expression only one of the columns is instantiated,

depending on the result data type for the expression.

Expression MIB - Structure

dismanExpressionMIB{.iso.org.dod.internet.mgmt.mib-2 90}

dismanExpressionMIBObjects dismanExpressionMIBConformance{dismanExpressionMIB 2}

expValue

{dismanExpressionMIB 1}

expResourceexpDefine

expValueTable

expExpressionTable expObjectTableexpErrorTable

Section 2Service Level Monitoring

Defining Delay, Jitter, & Packet loss

Multi-media applications are sensitive to the Delay, Jitter, and Packet Loss characteristics of data networks.

Delay is the time taken for packets to travel across a network. • One-way delay calculations require clock synchronization across

nodes, whereas measuring round-trip delay is easier.

Jitter is the variation in delay. • A jitter buffer temporarily stores arriving packets in order to

minimize delay variations.

Packet loss describes the % of packets that did not reach their intended destination. • Reasons for packet loss include link failure, network congestion,

and Random Early Discard (RED) among others.

Cisco IP SLA

Cisco IP SLA is an IOS feature that enables measurement of jitter, latency, or packet loss between network end-points.

IP Services can be simulated by specifying various packet sizes, ports, class of service, packet spacing, and frequencies.

Jitter/Latency/Packet-loss measurements per class of service are made to validate service differentiation for data, voice, and video.

Cisco IP SLA also helps to establish an network performance baseline and allow the user to understand trends and anomalies from the baseline.

SAA and RTTMON

Cisco IP SLA is supported by the Service Assurance Agent (SAA) and the Round-Trip Time Monitoring (RTTMON) MIB. – The SAA and the RTTMON MIB are Cisco IOS software features

available in versions 12.0 (5)T and higher.

The features enable you to test and collect delay, jitter, and packet loss statistics on the data network.– Cisco IOS routers are used as probes to simulate end points.– The probes can be configured to monitor the network for delay and

jitter and alert network management stations when thresholds are exceeded.

SLAs for IP Networks

Measure Either PE–CE or PE-PE

Links

Enterprise Site 2

Enterprise Site 1

Measure Either CE–PE

or CE–CE Links

P Router

SP Converged IP Network

UDP Jitter Operation

Source

IP CoreResponder

Send train of packets with constant Interval

Receive train of packets atInterval impacted by Network

Add a receive time stamp, and calculate delta, the processing time.

Per-direction inter-packet delay (Jitter)

Per-direction packet loss

Average Round Trip Delay

Responder

Source Router

Responder

Target Router

T1

T4

T3

T2

D = T3 - T2

Responder factors out destination processing time making results highly accurate

Responder allows for one-way measurements for latency, jitter, and packet loss.

UDP Jitter Operation

Time

Frequency

= IP SLA UDP Jitter test packet – Operation 1 destination1= IP SLA UDP Jitter test packet – Operation 2 destination2

IntervalNumber of

Packets

IP SLA Example

IP Class of Service

One-way Latency

Packet Loss Jitter

Priority Voice Traffic

< 80 ms < 5% < 35 ms

Real-Time Traffic - Video

< 80 ms < 3%

Priority Data Traffic

< 100 ms < 2%

Best Effort Traffic

No target No target No target

Polling the RTTMON MIB

The source SLA probe collects and places data in the RTTMON SNMP MIB tables.• rttMonStatsTable• rttMonLatestJitterOper

‘show rtr collection−stats’

Polling the RTTMON MIB

saarouter1# show rtr collection−statsCollected StatisticsEntry Number: 100Target Address: 172.18.179.10, Port Number: 14384Start Time: 13:06:04.000 09:25:00 Tue Mar 21 2000RTT Values:NumOfRTT: 600 RTTSum: 873 RTTSum2: 1431Packet Loss Values:PacketLossSD: 0 PacketLossDS: 0PacketOutOfSequence: 0 PacketMIA: 0 PacketLateArrival: 0InternalError: 0 Busies: 0Jitter Values:MinOfPositivesSD: 1 MaxOfPositivesSD: 1NumOfPositivesSD: 23 SumOfPositivesSD: 23 Sum2PositivesSD: 23MinOfNegativesSD: 1 MaxOfNegativesSD: 1NumOfNegativesSD: 1 SumOfNegativesSD: 1 Sum2NegativesSD: 1MinOfPositivesDS: 1 MaxOfPositivesDS: 1NumOfPositivesDS: 7 SumOfPositivesDS: 7 Sum2PositivesDS: 7MinOfNegativesDS: 1 MaxOfNegativesDS: 1NumOfNegativesDS: 18 SumOfNegativesDS: 18 Sum2NegativesDS: 18

Simulating a voice call

Simulating a G.711 voice call:– RTP/UDP port 14384– 64 Kb/s– 172 byte packets (160 byte payload + 12 byte RTP header)

The jitter probe operation includes:– Send the request to RTP/UDP port number 14384.– Send 172 byte packets– Send 3000 packets for each frequency cycle.– Send every packet 20 milliseconds apart for a duration of 60

seconds and sleep 10 seconds before starting the next frequency cycle.

– ((3000 datagrams * 160 bytes per datagram)/ 60sec) * 8 = 64 kb/s

Section 3Agent EXtentisibilty (AgentX)

Why AgentX?

There is a real need to dynamically extend the set of managed objects within a node.

The SNMP framework does not describe how a set of managed objects supported by an agent may be changed dynamically

Agent EXtentisibilty (AgentX)

Facilitates extension of SNMP Agents with dynamic addition of new MIB module implementations.

Separates SNMP protocol engines from MIB Instrumentation.

Maintains transparency to Management Stations. Proposed IETF Standard

– RFC 2741 (AgentX Protocol)– RFC 2742 (AgentX MIB)

AgentX Framework

In the AgentX framework, the SNMP agent is defined to consist of:– a master agent, which sends and receives SNMP protocol

messages but has little or no direct access to management information.

– one or more subagents, which are shielded from the SNMP protocol messages, but have access to management information.

– a protocol (agentX) that operates between the master agent and subagents, permitting subagents to connect to the master agent

AgentX Design Features

Master agents are concerned with SNMP protocol operations and the translations to and from AgentX protocol operations

Subagents are concerned with management instrumentation

Subagents are not aware of any other existing subagents. A single subagent is "authoritative" for a particular

region of the MIB– "region" may extend from an entire MIB down to a single

object- instance

AgentX - Basic Structure

Master Agent

Protocol Operations

MIB MIB MIB

AgentX

Sub-agent Sub-agent Sub-agent

SNMP

Master/Sub Agent Functions

Master agent– sends and accepts SNMP protocol messages.– sends and receives AgentX protocol messages to access

management information from sub-agents. – provides instrumentation for certain MIB objects.– forwards notifications on behalf of subagents.

A sub-agent– initiates AgentX session with the master agent– registers MIB regions with the master agent– provides instrumentation for managed objects – performs management operations on variables– initiates notifications

AgentX – Usage Scenarios

Subagents implement separate MIB modules– for example, subagent `A' implements "mib-2", subagent `B'

implements "host-resources-mib". This will be the most common subagent configuration.

Subagents implement rows in a "simple table". – A simple table is one in which row creation is not specified, and for

which the MIB does not define an object that counts entries in the table. This is the most commonly defined type of MIB table

Subagents share MIBs along non-row partitions.– Subagents register "chunks" of the MIB that represent multiple

rows, due to the nature of the MIB's index structure. Examples include registering tcpConnEntry.a.b.c.d, where a.b.c.d represents an IP address on a subagent’s interface

AgentX – OID Registration

MIB Registration– Example: REGISTER (mib-2)

RANGE Registration– Example: REGISTER (mib-2.interfaces, mib-2.tcp)

udptcpinterfacessystem

TOP REGISTRATION RANGE REGISTRATION

mib-2

... udptcpinterfacessystem

mib-2

...

AgentX – VarBind Splitting

MIB

MIB

MIB

A B C

A

B C

Sub-agent

Sub-agent

Sub-agent

Ma

ste

r Ag

en

t

SNMP MANAGER

Section 4HP Openview NNM:

Scalability and Distribution

NNM Scalability and Distribution

Scalability is referred to – the configuration of NNM to perform well for a wide range of

network sizes and degrees of complexity.

Distribution of NNM is – Refers to the distribution of network management functions to

multiple, distributed systems.

Objective is to optimize:– System resources at the management station.– Reduce the amount of management traffic over the network.

NNM Scalability/Distribution Features

Distributed Discovery and Monitoring Management Consoles Support for viewing large map (the ‘panner’) On-demand Sub-maps Filters Distributed Threshold Monitoring Event Forwarding

Distributed Discovery and Monitoring

Distributed discovery and monitoring is the ability to move much of the network discovery, topology monitoring, and status polling from the central management station to one or more distributed stations.

Distributed Discovery & Monitoring

Consists of Management and Collection stations Management station makes available network

management functionality to users, either directly or via one or more management consoles.

Collection stations function as data collection points. A collection station typically performs:– network discovery,– topology checks,– IP status monitoring, – data collection, – event forwarding

Distributed Discovery & Monitoring

ManagementStation A

CollectionStation A

CollectionStation B

CollectionStation n

ManagementConsole

ManagementConsole

Changes in status and topology are relayed from the collectionstations to the management station.

Discovery and status polling occur at the local level.

Managed Devices

Managed Devices

Managed Devices

Managed Devices

NNM Filters

A filter is essentially a way to remove unneeded data from further processing.

Filters result in less data to process resulting in improved NNM performance and usability for the administrators.– for instance, to an administrator whose only concern is the

routers and hubs, a filter can be applied to the map, so that only routers and hubs appear.

Filters can be applied at different points in the system:– Discovery filtering at Management or Collection station.– Topology filtering at the Collection station.– Map filtering at the Management station.– Failover filtering at the Management station.

NNM Filters

A data-stream filter acts as a screen through which objects flow.– contains criteria that permit some objects to pass the screen

while others are removed at the point of the filter. – once an object has passed a data-stream filter, later changes to

the filter have no effect on it. A set-defining filter is applied to a pool of candidate

objects to determine which objects belong in the final set based on some criteria contained in the filter.– If a set-defining filter is changed, all candidate objects

are re-evaluated to determine which ones belong in the new set.

NNM Filters

Filter Type Purpose

Discovery Limit scope of objects added to database.

Topology Limit information forwarded from collection station to management station.

Map Show only items of interest to operator on map.

Persistence Disable on-demand sub-maps for third party applications.

Failover Limit nodes polled by management station when collection station fails.

Distributed Discovery & Monitoring

NNM Adv Edition A

NNM Station C

Dual-role station

ManagementConsole

Domain A Devices

NNMAdv Edition B

ManagementConsole

Web Client

ManagementConsole

Domain B Devices

ManagementConsole

LAN data flowWAN data flowFilter in data path

NNM Station D

Collection station

NNM Station E

Collection station

NNM Station F

Dual-role station

NFS

NFS

Objects discovered & monitored by Station C

Objects discovered & monitored by Station D Objects discovered &

monitored by Station E

Objects discovered & monitored by Station F

Section 5RMON

RMON

SNMP Remote Network Monitoring (RMON) was created to enable efficient management of remote networks using dedicated devices.

RMON defines a MIB module with object definitions that permit dedicated network management capabilities. – RFC 1271, "Remote Network Monitoring Management

Information Base," published in 1991. – RFC 2819, published in May 2000, updates RMON to use the

SMIv2 specification.

RMON

Remote network monitoring devices, often called remote probes or monitors are stand-alone devices and devote significant internal resources for the sole purpose of managing a network. – An organization may employ many of these devices, one per

network segment, to manage its internet. RMON MIB objects are not intended for processing by a

management application and for direct manipulation by humans.

Most RMON objects are suitable for the management of any type of network– Objects specific to managing Ethernet networks are defined in

the ‘etherStatsTable’ and ‘etherHistoryTable’.– The RMON-2 MIB objects enable analysis up to the application

layer.

RMON System

RMON Benefit

Offline Operation:– There are times when a management station is not in contact

with its remote monitoring devices. • by design or • during network failures

– RMON MIB allows a probe to be configured to collect statistics continuously or run diagnostics, even when communication with the management station may not be possible or efficient. • The probe may notify the management station when an exceptional

condition occurs. • Fault, performance, and configuration information may be

continuously accumulated and forwarded to the management station efficiently.

RMON MIB Structure

An RMON MIB consists of a several functional groups. – Functional groups have control and data tables.– The control data specify how and when to collect the statistical

data.

RMON MIB Structure

All groups in RMON MIB are optional. Implementations of this MIB must also implement the system group of MIB-II and the IF-MIB. The objects are arranged in following groups:– ethernet statistics– ethernet history– alarm– host – hostTopN– matrix– filter– packet capture– event

RMON MIB Structure

RMON MIB Structure

The Ethernet Statistics Group – The ethernet statistics group contains statistics for each

monitored Ethernet interface on this device.

The Alarm Group– The alarm group periodically compares statistical samples to

configured thresholds. If the monitored variable crosses a threshold, an event is generated.

– This group requires the implementation of the event group.

The Host Group– The host group contains statistics associated with each host

discovered on the network. This group discovers hosts on the network by keeping a list of source and destination MAC addresses seen in packets received from the network.