InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.
-
Upload
hortense-davis -
Category
Documents
-
view
216 -
download
0
Transcript of InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.
![Page 1: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/1.jpg)
InfiniBand SNMP Agent
PSL Lab Spring 2005 – Winter 2005 2006
![Page 2: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/2.jpg)
2
People
• Student– Ali Ayoub
• Supervisors– Liran Liss– Eitan Zahavi– Amit Krig
![Page 3: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/3.jpg)
3
Abstract
The rapid growth in InfiniBand usage requires more standard mange tools, most popular monitoring applications for networks use Simple Network Management Protocol (SNMP); the goal of this project is to enable monitoring the InfiniBand network using this protocol.
![Page 4: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/4.jpg)
4
Introduction
InfiniBand networks, as other networks, need to be monitored and managed for administrator and development needs, the usage of this technology become more widespread; it’s involved in bigger clusters and high-performance networking, therefore, the need for such a tool becomes more essential.Besides, a lot of information about InfiniBand traffic is stored in the hardware, this is derived from its architecture which implements most of the layers in hardware, and this information is usually not exposed to the application layer, which makes the task to retrieve this information more difficult without a tool designed for this purpose.
![Page 5: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/5.jpg)
5
Introduction
To achieve this goal, we used in this project a standard network management protocol called SNMP, because this protocol is one of the most popular solutions for network monitoring, there are many monitors which relies on this protocol to offer to the user a friendly interface such as IBM Tivoli and HP OpenView, for the same reason, collecting information from InfiniBand devices and other devices (CPU, memory, NIC…) , can be done using the same tool. So far, there was no such a tool for InfiniBand network management, although it was invented and used since 1996.The tool we present in this project is the only one of its kind in this field.
![Page 6: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/6.jpg)
6
Background
• SNMPSimple Network Management Protocol (SNMP) is an application layer protocol that facilitates the exchange of management information between network devices. It is part of the Transmission Control Protocol/Internet Protocol (TCP/IP) protocol suite. SNMP enables network administrators to manage network performance, find and solve network problems, and plan for network growth.
![Page 7: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/7.jpg)
7
Background
• MIBA Management Information Base (MIB) is a collection of information that is organized hierarchically. MIBs are accessed using a network-management protocol such as SNMP. They are comprised of managed objects and are identified by object identifiers.
![Page 8: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/8.jpg)
8
Background
• OIDThe Object ID (OID), each SNMP element manages specific objects with each object having specific characteristics. Each object / characteristic has a unique object identifier (OID) consisting of numbers separated by decimal points (i.e., 1.3.6.1.4.1.2682.1).
![Page 9: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/9.jpg)
9
![Page 10: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/10.jpg)
10
Overview of the solution
• Components– InfiniBand SNMP Agent– NET-SNMP
Advantages vs. disadvantages– Management Network System.
Examples– InfiniBand Management Information Bases (IB MIBs)
![Page 11: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/11.jpg)
11
Overview of the solution
• Flow descriptionThe MNS have the MIBs definitions; according to them the MNS sends a request to a machine with InfiniBand SNMP Agent asking it for the information, the InfiniBand SNMP Agent includes the MIBs module which are used to retrieve the requested data and send it back to the MNS, then the MNS display the information to the user by its own interface, polling InfiniBand information in high frequency can serve to draw graphs for InfiniBand traffic for example.
![Page 12: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/12.jpg)
12
IB network
InfiniBand SNMP agent and NET-SNMP run on the machines in the network.
SNMP Monitor
IB MIBs
IP network
![Page 13: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/13.jpg)
13
Implementation details
• InfiniBand MIBs– IPoIB workgroup– InfiniBandMIB tree– Implemented IB MIBs
• IB SNMP Agent– High-level layers– Agent subagnet architecture
AgentX– Data caching
• Design– Low-level layers
• Tools• Testing• Issues uncovered
![Page 14: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/14.jpg)
14
InfiniBand MIBs
• The MIBs we use and implement are written by the IP over IB workgroup as published in IETF (Internet Engineering Task Force) internet drafts, they are part of set of MIBs for IB fabric management, these MIBs are located under InfiniBandMIB (OID: 1.3.6.1.3.117) sub-tree, and will have a unique OIDs once they are registered by IANA (Internet Assigned Numbers Authority).
![Page 15: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/15.jpg)
15
O root-node | | iso[1] | | +---+------+ | | | | | |[1] [2] identified-organization[3] |
| +-------+--+--+ | | | | | |[1] ... [5] dod[6] |
| internet[1] |
| +-------------+-----+------------------+----------------+ | | | | | | | |directory[1] management[2] experimental[3] private[4] |
| InfiniBandMIB[117]InfiniBandMIB[117] |
| +----------+-----------+----------+-+---------+-----------+----------+ | | | | | | | | | | | | | |ibTcMIB[1] ibIfMIB[2] ibSmaMIB[3] ibCaMIB[4] ibBmaMIB[5] ibPmaMIB[6] ibSmMIB[7]
![Page 16: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/16.jpg)
16
InfiniBand MIBs
• IPoIB MIBs– ibTcMIB: Textual Conventions MIB
– ibIfMIB: InfiniBand Interfaces MIB
– ibSmaMIB: Subnet Management Agent MIB
– ibCaMIB: Channel Adapters MIB
– ibBmaMIB: Baseboard Management MIB
– ibPmaMIB: Performance Management Agent MIB
– ibSmMIB: Subnet Manager MIB
![Page 17: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/17.jpg)
17
InfiniBand MIBs
1. Channel adapter MIB (ibCaMib)– This MIB describes the objects for managing the InfiniBand Channel
Adapters.– OID: 1.3.6.1.3.117.4– The purpose of this MIB is to define the objects necessary to
manage Channel Adapters (HCAs and TCAs).– This includes general CA information, CA attributes, CA port
attributes, and the associated CA port GID info.– Current I-D: draft-ietf-ipoib-channel-adapter-mib-07.txt– Expiration date: March 2006
![Page 18: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/18.jpg)
18
InfiniBand MIBs
2. InfiniBand Interfaces MIB (ibIfMib) – Describes the managed objects for managing IBA-defined InfiniBand interfaces.– OID: 1.3.6.1.3.117.2– The purpose of this MIB is to augment the standard SNMP Interface MIB (RFC
2863). – Defines standard InfiniBand port ifType as InfiniBand(199).– It contains an InfiniBand Port statistics table, a Virtual Lane Traffic Table for each
port’s VLs, and a table for the IPOIB Link layer addresses (20-octet value that uniquely identifies each IPOIB client end-point, i.e., QPN + GID).
– Current I-D: draft-ietf-ipoib-ibif-mib-08.txt– Expiration date: March 2006
![Page 19: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/19.jpg)
19
InfiniBand MIBs
• MIBs Modules Integration– Extending master agent– Separated shared libraries to be loaded into the master agent– Implementing MIBs modules into a SNMP subagent
• Which way did we use? Why?– We’ll discuss this way later
![Page 20: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/20.jpg)
20
InfiniBand SNMP Agent
• Slave agent
• MIBs– Definitions, where they are used?– MIBs Modules = implementation
• Generic
• AgentX
![Page 21: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/21.jpg)
21
InfiniBand SNMP Agent
• AgentX, – why?
The two most commonly used protocols for subagents are AgentX and SMUX.SNMP Master/Subagent protocols overcome a very real need to dynamically extend the managed objects in a node. The protocols allow you having a single SNMP Agent and several Subagents that can connect and register several managed objects without having to interrupt the management service. We decided to use AgentX protocol, because it is the first IETF standard-track specification for extensible SNMP agents and is expected to gradually replace all the other open and proprietary agent extensibility solutions (such as SMUX, DPI etc.).
![Page 22: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/22.jpg)
22
InfiniBand SNMP Agent
• AgentX, – How?
The subagent (InfiniBand SNMP Agent) registers itself in the master agent, when this process is completed the master agent knows that there is a subagent which implements some OIDs
when the monitor sends a SNMP request to retrieve data from a machine with InfiniBand SNMP Agent, first the master agent receives the request, according to the OIDs requested, it passes the request to the subagent (using AgentX protocol), where the suitable function in InfiniBand SNMP Agent will be called to supply the needed information requested, and update the data structures involved
Then the master agent will return to the requester the requested data, which could be a single attribute or sub-tree of attributes. Besides, this protocol is supported by NET-SNMP, that’s why we used AgentX protocol in implementing our InfiniBand SNMP Agent.
![Page 23: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/23.jpg)
23
![Page 24: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/24.jpg)
24
![Page 25: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/25.jpg)
25
InfiniBand SNMP Agent
• Data caching– Why?
SNMP monitors usually poll data from the SNMP agent in high frequency (for drawing graphs for example), this could create stress on the InfiniBand device to answer all the agent’s queries.
– How? Data structure Refresh rate Configuration
– IB-CA-MIB vs. IB-IF-MIB
![Page 26: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/26.jpg)
26
Design
• InfiniBand SNMP Agent
• IBSNMP Agent driver API
• SNMP agent gen2 layer
• SNMP agent gen1 layer
• Agent IPoIB layer
• MADs structures
![Page 27: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/27.jpg)
27
Design
![Page 28: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/28.jpg)
28
Design
• Advantages– Isolation
SNMP vs. IB MADs and IPoIB libraries
– Reusability For future MIBs For future drivers
– Genericity Easy way for linkage with different stacks
![Page 29: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/29.jpg)
29
How to retrieve InfiniBand MIBs information
• MADs– Management datagrams for IB-IF-MIB
• Driver verbs– Static data, for IB-CA-MIB
• Operating system commands and files– Like /proc/ files and IP interfaces commands.
![Page 30: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/30.jpg)
30
Tools
• Languages– C, ASN.1
• Protocols– SNMP v2/v3
• AgentX
• Net-SNMP Master agent
• Mib2c
![Page 31: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/31.jpg)
31
Testing
• Different Operating systems
• Different IB Stacks
• Direct tests
• Real events
• Monitors
![Page 32: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/32.jpg)
32
Issues revealed
• Bad InfiniBand cables
• Unplugged cables
• Unloaded drivers
• Also, for retiring information for the HCA remotely without logging on the machine– Number of ports– MAC– GUID
![Page 33: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/33.jpg)
33
SNMP Monitors
• MRTG (Multi Router Traffic grapher)– Open source SNMP monitor for displaying traffic on graphs on real time.– Could be installed on web-server for real time monitoring from
everywhere.– Good for dynamic data (IB-IF-MIB)
• IReasoning– Commercial – Good for viewing tables (IB-CA-MIB)
![Page 34: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/34.jpg)
34
IBMRTG Monitor
![Page 35: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/35.jpg)
35
IReasoning
![Page 36: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/36.jpg)
36
Future Work
• Extending the IB SNMP Agent– More MIBs
Some are available in IPoIB workgroup– Support future IB stacks
![Page 37: InfiniBand SNMP Agent PSL Lab Spring 2005 – Winter 2005 2006.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e605503460f94b5b6a7/html5/thumbnails/37.jpg)
37
Thank you!