ClusterValidation v0.13 - Interdisciplinary | Innovativekorobkin/tmp/SC10/tutorials/...pbs-serv2...

SC20101

Validation of an HPC Cluster: A Sometimes Neglected Aspect of System Administrationwalk through of methods and procedures

tut118

Michael HebenstreitINTEL® corp.CRT Datacenter, Senior Cluster Architect

1

SC20102

Agenda• CRT-DC – the Customer Response Data

Center• The problem• Tier 1: The hardware• Tier 2: the installed image• Tier 3: performance tests• A look at MS Windows*• Commercial solutions

2

SC20103



3

SC20104

CRT Datacenter Challenges• Support for variety

– Multitude of different hardware architectures– Early access often leads to alpha and beta systems used

in cluster configuration• Support for different customers

– OEMs, End users, ISVs– Some want their own configuration– Manage access while preserving security of data for

each user– Protect the internal network and Intel IP from external

disclosure• Support for scaling

– Often requires exclusive period due to custom configurations

– Remove compute nodes out of circulation for the duration of the project

SC20105

CRT-DC cluster configurationPanasas*

/home

compile

pbs-serv2

pbs-serv1admin2admin1

login

QDR IB

LFS5(SSD)

LFS4(HDD)

DDN*Lustre

long-termstorage

Force10*network

360Urbannacomputenodes

24 GB RAM400 GB SAS HD

64Supermicro*

computeNodes

24 GB RAM500 GB SATA HD

1GbE networkQDR InfiniBand network

login 2

Panasas*

/home

compile

pbs-serv2


login

QDR IB

LFS5(SSD)

LFS4(HDD)

DDN*Lustre

long-termstorage

Force10*network

360Urbannacomputenodes

24 GB RAM400 GB SAS HD

64Supermicro*

computeNodes

24 GB RAM500 GB SATA HD


login 2

SC20106

Exemplary Configurations• Nodes

– 360 Intel SR1600UR: Xeon® X5670 (WSM),2.93 GHz,12cores/node,24 GB– 64 Supermicro 6026T-NTR+: 34 Xeon® X5560 (NHM,2.8GHz,8 cores/node),

40 Xeon® X5677 (WSM,3.47GHz,8 cores/node), all 24 GB• Cluster File System

– Panasas *(70 TB storage) – DDN* Lustre (28 TB storage) – HDD Lustre (23 TB storage) – SSD Lustre (3 TB storage)

• Distributed GigE:– Force10* Networks C-300 backbone, Force10 Networks S50N top-of-rack

• Distributed InfiniBand*: – Mellanox* MTS3600Q, 18 spine, 28 leaf switches, 504 ports

• Software stack:– RedHat* EL5, OFED 1.3+,Lustre 1.6.4.3+

• has been on Top 500 since June 2006 (best ranking #68, worst #153)

SC20107


Center• The problem• Tier 1: The hardware• Tier 2: the installed image• Tier 3: performance tests• A look at MS Window*• Commercial solutions

7

SC20108

Classification• Hardware and software defects: systems

dead or does not operate correctly• Inconsistencies: configuration (config

files, installed rpms…) are not identical across the cluster

• Degradation: system performs correctly but lost performance->keep log files

SC20109

The Linux Toolbox• Executing commands in parallel – pdsh*• Consolidating pdsh output – dshbak*• cat, grep, sum, sed, awk…• shell scripting• advanced programming languages like

Python* or Perl*

pdsh homepage: http://sourceforge.net/projects/pdsh

redirect – To send the output of a file or command into another file[user]$ echo "\"To err is human -" > text1[user]$ echo "and to blame it on a computer is even more so."\" > text2[user]echo "Robert Orben" > text3--------------------------------------------------------------------------------------------------------------------------------------------cat (concatenate) Displays the contents of one or more files to standard output. It is most commonly used to display a single file to a monitor.[user]$ cat text1"To err is human –[user]$ cat text2and to blame it on a computer is even more so."[user]$ cat text3Robert Orben[user]$ cat text1 text2 text3"To err is human -and to blame it on a computer is even more so."Robert Orben[user]$ cat text1 text2 text3 > text4[user]$ cat text4"To err is human -and to blame it on a computer is even more so."Robert Orben---------------------------------------------------------------------------------------------------------------------grep – Used to find a text pattern within a file and return the line(s) containing the pattern. Most commonly used to find a word, but can find a character, phrase, sentence or any regular expression.[user]$ grep computer text4and to blame it on a computer is even more so."grep –i Because grep is case sensitive, -i is used to ignore case[user]$ grep to text4and to blame it on a computer is even more so."[user]$ grep –i to text4"To err is human -and to blame it on a computer is even more so."grep –c To count the number of lines which contain the expression being grep’d.[user]grep -c is text42

redirect – To send the output of a file or command into another file [smartuser@server1~]$ echo "\"To err is human -" > text1 [smartuser@server1~]$ echo "and to blame it on a computer is even more so."\" > text2 [smartuser@server1~]echo "Robert Orben" > text3

-------------------------------------------------------------------------------------------------------------------------------------------- cat (concatenate) Displays the contents of one or more files to standard output. It is most commonly used to display a single file to a monitor.

[smartuser@server1~]$ cat text1 "To err is human – [smartuser@server1~]$ cat text2 and to blame it on a computer is even more so." [smartuser@server1~]$ cat text3 Robert Orben [smartuser@server1~]$ cat text1 text2 text3 "To err is human - and to blame it on a computer is even more so." Robert Orben [smartuser@server1~]$ cat text1 text2 text3 > text4 [smartuser@server1~]$ cat text4 "To err is human - and to blame it on a computer is even more so." Robert Orben

--------------------------------------------------------------------------------------------------------------------- grep – Used to find a text pattern within a file and return the line(s) containing the pattern. Most commonly used to find a word, but can find a character, phrase, sentence or any regular expression.

[smartuser@server1~]$ grep computer text4 and to blame it on a computer is even more so."

grep –i Because grep is case sensitive, -i is used to ignore case [smartuser@server1~]$ grep to text4 and to blame it on a computer is even more so." [smartuser@server1~]$ grep –i to text4 "To err is human - and to blame it on a computer is even more so."

grep –c To count the number of lines which contain the expression being grep’d. [smartuser@server1~]grep -c is text4 2

grep –v To search for lines which do not contain the expression

[smartuser@server1~]grep -v is text4 Robert Orben

grep –q Searches and quietly exits if the expression is found. When the grep is finished, the exit code is set to the variable $?. If we echo $?, we will see if the expression is present or not. Succcess = 0, Failure = 1. This is useful in “if” statements to avoid confusing output to a user.

[smartuser@server1~]grep -q man text4; echo $? 0 [smartuser@server1~]grep -q woman text4; echo $? 1

--------------------------------------------------------------------------------------------------------------------- sum – Computes a 16-bit checksum for each given file and counts the blocks each file occupies. This is calculated after a file transfer and compared to the checksum of the original file to ensure file integrity.

[smartuser@server1~]$ sum text4 05333 1 [smartuser@server1~]$ sum text1 text2 text3 24872 1 text1 63331 1 text2 20594 1 text3

--------------------------------------------------------------------------------------------------------------------- awk (printing a specific column) – awk generally is used to search output or a file for a pattern and then manipulate it. When awk finds a specified pattern in a line, it assigns each part of that line to unique variables, e.g. $1 $2 $3 $4 $NF. The smart user can then manipulate the values by using the variables.

[smartuser@server1~]$ cat text4 "To err is human - and to blame it on a computer is even more so." Robert Orben [smartuser@server1~]$ err to Orben

To limit the output we can use an option telling awk to only consider the line that begins with “and” [smartuser@server1~]$ awk /^and/'{print $3" "$6" "$7}' text4 blame a computer

piping with "|" – The pipe lets us direct output from one command directly into another. So here is another way to get to the same output.

[smartuser@server1~]$ grep blame text4 | awk '{print$3" "$6" "$7}' blame a computer

-------------------------------------------------------------------------------------------------------------------------------------------- sed (changing text) – sed is most useful for making text transformations on an input stream, whether from a file or a pipeline. The single quotes contain the logic sed is to follow, s = substitute, computer is the expression to find and dog is the expression to put in it’s place, g means global and tells sed not to stop at the first occurrence, but to make the change anywhere in the file where the expression “computer” occurs.

[smartuser@server1~]$ grep blame text4 | awk '{print$3" "$6" "$7}' | sed 's/computer/dog/g' blame a dog

Or

awk /ând/'{print $3" "$6" "$7}' text4 | sed 's/computer/cat/g' [smartuser@server1~]$ awk /ând/'{print $3" "$6" "$7}' text4 | sed 's/computer/cat/g' blame a cat

For fun [smartuser@server1~]OTHERS="horse pig mouse goat" [smartuser@server1~]echo $OTHERS horse pig mouse goat [smartuser@server1~]for i in $OTHERS; do awk /ând/'{print"You should "$3" "$6" "$7}' text4 | sed 's/computer/'$i'/g';done You should blame a horse You should blame a pig You should blame a mouse You should blame a goat

sort is used to sort either alphabetically or numerically. If you | standard output to sort, you will see sorted results on your monitor. You can > the output of sort into a file or | it to another command.

[smartuser@server1~]for i in $OTHERS; do awk /ând/'{print"You should "$3" "$6" "$7}' text4 | sed 's/computer/'$i'/g';done | sort You should blame a goat You should blame a horse You should blame a mouse You should blame a pig

uniq is used to manage successive identical lines. Most commonly it is used to omit duplicate lines from standard output. Piping to sort so uniq will find the identical lines adjacent to each other.

[smartuser@server1~]cat other horse pig mouse goat horse pig mouse [smartuser@server1~]for i in `cat other`; do awk /ând/'{print"You should "$3" "$6" "$7}' text4 | sed 's/computer/'$i'/g'; done | sort | uniq You should blame a goat You should blame a horse You should blame a mouse You should blame a pig

SC201010

Conventions• Examples are in Courier, output will appear in black, commands entered are set in bold blue

• Commands usable without special privileges are prefixed by [user]$, those requiring administrative privileges by [root]#

[user]$ ls -l /etc/shadow-r-------- 1 root root 29218 Aug 10 16:42 /etc/shadow

[root]# sum /etc/shadow47530 29

SC201011

Executing commands in parallel• Requirements:

– grouping of nodes– ssh based– options for timeout and fanout– leading optional nodename is helpful

• pdsh is one (but not only) solution

[user]$ pdsh -w et01,et03,et[10-15] -f 2 -u 5 'date; sleep 1'et01: Fri Aug 6 17:36:48 PDT 2010et03: Fri Aug 6 17:36:48 PDT 2010et10: Fri Aug 6 17:36:49 PDT 2010et11: Fri Aug 6 17:36:49 PDT 2010et13: Fri Aug 6 17:36:50 PDT 2010et14: Fri Aug 6 17:36:52 PDT 2010et15: Fri Aug 6 17:36:53 PDT 2010pdsh@eln1: et12: command timeoutsending SIGTERM to ssh et12 pid 27323pdsh@eln1: et12: ssh exited with exit code 0

SC201012

Consolidating pdsh output: dshbak -c

• Requirements:– consolidate into groups of equal output– lists of nodes compatible with pdsh

[user]$ pdsh -w et[01,03,10-11,13-15] pwd | dshbak -c----------------et[01,03,10-11,13-15]----------------/home/user1

[user]$ pdsh -w et[01,03,10-11,13-15] 'ps aux | wc -l' | dshbak -c----------------et[11,13,15]----------------306

----------------et[01,03,10,14]----------------308

SC201013

Connecting to the Linux cluster• Using Putty*:

– enter IP address into the “Hostname” field

– press open– enter username– enter password

Putty* homepage: http://www.chiark.greenend.org.uk/~sgtatham/putty/

SC201014

How to protect against disconnects

• Use screen*:– to start type “screen”– to detach press “ctrl-a d”– to create a new window press “ctrl-a c”– to switch to window 0 press “ctrl-a 0”– to switch to window 1 press “ctrl-a 1”– to reconnect after a connection loss type “screen -x” (lower case x)

SC201015



15

SC201016

Components of cluster

/home

compile

pbs-serv2


login

IB

HPFS

long-termstorage

network

computenodes


SC201017

Components of a node

Motherboard

BIOS

CPU

IB CARD

Networkadapter

BMC

DISK

RAM Power supply

Fans

RaidController

SC201018

CPU - /proc/cpuinfo• contains information on all processors in

the system• particularly interesting: “model name”,

“cpu MHZ”, “cache size”• Note: Hyperthreading can not be directly

discerned[user]$ pdsh -w et[60-78] -u 3 'grep MHz /proc/cpuinfo | sort | uniq' | dshbak -c----------------et[60-64,69-73,77]----------------cpu MHz : 3458.000----------------et[65-68]----------------cpu MHz : 2926.000----------------et[74-76,78]----------------cpu MHz : 2793.000

[user]$ cat /proc/cpuinfoprocessor : 0vendor_id : GenuineIntelcpu family : 6model : 23model name : Intel(R) Xeon(R) CPU E5462 @ 2.80GHzstepping : 6cpu MHz : 2400.000cache size : 6144 KBphysical id : 0siblings : 4core id : 0cpu cores : 4apicid : 0fpu : yesfpu_exception : yescpuid level : 10wp : yesflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmovpat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tscpni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 lahf_lmbogomips : 5585.99clflush size : 64cache_alignment : 64address sizes : 38 bits physical, 48 bits virtualpower management:…….

SC201019

Methodology of Cluster Validation

• divide cluster into groups of identical nodes

• run series of tests watching for differences – or simply incorrect values

• if a new problem appears, include a test to check for it

• re-validate once per week, after changes and after rebootset[65-68]----------------cpu MHz : 2926.000----------------et[74-76,78]----------------cpu MHz : 2793.000

SC201020

More elaborate Example /proc/cpuinfo

shows different #cores – because HT is off[user]$ cat parse_cpuinfo.shgrep "model name" /proc/cpuinfo | sort | uniqFREQS=`grep "cpu MHz[[:space:]]*:" /proc/cpuinfo | sort | uniq | awk '{print $NF}'`for I in $FREQSdoCOUNT=`grep -c "cpu MHz[[:space:]]*: $I" /proc/cpuinfo`echo " $COUNT cores at $I MHz"

done

[user]$ pdsh -w et[60-64,69-73] -u 3 'sh parse_cpuinfo.sh' | dshbak -c----------------et[73]----------------model name : Intel(R) Xeon(R) CPU X5677 @ 3.47GHz8 cores at 3458.000 MHz

----------------et[60-64,69-72]----------------model name : Intel(R) Xeon(R) CPU X5677 @ 3.47GHz16 cores at 3458.000 MHz

SC201021

Sleep and Turbo states of the CPU • modern CPUs can optimize power

consumption according to load• idle cores might go into sleep, busy cores

could switch into Turbo mode• information is available on a “per core”

basis• require that corresponding kernel

modules are loaded

Redhat* Enterprise Linux: look for kernel module “acpi_cpufreq”

SC201022

Information on Sleep/Turbo states

• C-states:cat /proc/acpi/processor/CPU#/power

• Intel® SpeedStep:cd /sys/devices/system/cpu/cpu#/cpufreq/– available frequencies:

cat scaling_available_frequencies2927000 2926000 2793000 … 1729000 1596000

– possible control options:cat scaling_available_governorsondemand userspace performance

– currently used frequency: cat scaling_cur_freq2400000

– currently used controller: cat scaling_governorondemand

Examples for C-states:[user]$ cat /proc/acpi/processor/CPU0/poweractive state: C1max_cstate: C8bus master activity: 00000000states:

*C1: type[C1] promotion[C2] demotion[--] latency[000] usage[03603131] duration[00000000000000000000]

C2: type[C2] promotion[--] demotion[C1] latency[040] usage[1215642043] duration[00000004126523448417][user]$ ssh en001 cat /proc/acpi/processor/CPU0/poweractive state: C1max_cstate: C1bus master activity: 00000000states:

*C1: type[C1] promotion[--] demotion[--] latency[000] usage[00000000] duration[00000000000000000000]

Note: in the second example C-states were limited to C1 for performance reasons

SC201023

Controlling Frequency

• Governor must be “userspace”• for all CPUs: write the same value into

“scaling_setspeed”[root]# cd /sys/devices/system/cpu/[user]$ cat cpu*/cpufreq/scaling_governor | sort | uniqondemand[root]# for I in cpu*/cpufreq/scaling_governor; do echo userspace > $I; done[user]$ cat cpu*/cpufreq/scaling_governor | sort | uniquserspace[user]$ cat cpu*/cpufreq/scaling_cur_freq | sort | uniq2400000[root]# for I in cpu*/cpufreq/scaling_setspeed ; do echo 2800000 > $I; done[user]$ cat cpu*/cpufreq/scaling_cur_freq | sort | uniq2800000

SC201024

Memory and swap• /proc/meminfo: information on memory

subsystem; important fields:– MemTotal: recognized total memory– MemFree, Buffers, Cached: after boot

• /proc/swaps: lists mounted swap space[user]$ pdsh -w et[60-64,69-73] -u 3 'grep SwapTotal /proc/meminfo' | dshbak -c----------------et[60-64,69-73]----------------SwapTotal: 8393952 kB

[user]$pdsh -w et[60-64,69-73] -u 3 ‘grep MemTotal /proc/meminfo' | dshbak –c----------------et[60-64,69-72]----------------MemTotal: 24673976 kB

----------------et73----------------MemTotal: 24673984 kB

[root@eln1 ~]# cat /proc/swapsFilename Type Size Used Priority/dev/sda2 partition 8393952 58672 -1

[root@eln1 ~]# cat /proc/meminfoMemTotal: 65984928 kBMemFree: 44880144 kBBuffers: 56252 kBCached: 19738156 kBSwapCached: 5724 kBActive: 774872 kBInactive: 19527424 kBHighTotal: 0 kBHighFree: 0 kBLowTotal: 65984928 kBLowFree: 44880144 kBSwapTotal: 8393952 kBSwapFree: 8335284 kBDirty: 2964 kBWriteback: 0 kBAnonPages: 505000 kBMapped: 13928 kBSlab: 696548 kBPageTables: 35260 kBNFS_Unstable: 0 kBBounce: 0 kBCommitLimit: 41386416 kBCommitted_AS: 982696 kBVmallocTotal: 34359738367 kBVmallocUsed: 278100 kBVmallocChunk: 34359458135 kBHugePages_Total: 0HugePages_Free: 0HugePages_Rsvd: 0Hugepagesize: 2048 kB

SC201026

Motherboard and BIOS tools• usually works without special

requirements– lspci– dmidecode

• depends on hardware and vendor– ipmitool– syscfg

SC201027

lspci• generates a list of PCI devices• lspci can also read configuration space• Note: setpci can MODIFY configuration

space[user]$ pdsh -w et[01,03,10-15] '/sbin/lspci | grep InfiniBand' | dshbak -c----------------et[01,03,10-15]----------------04:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 …

[root]# pdsh -w et[01,03,10-15] '/sbin/lspci -xxx -s 04:00.0 ' | dshbak -c----------------et[01,03,10-15]----------------04:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 …00: b3 15 3c 67 06 04 10 00 a0 00 06 0c 10 00 00 0010: 04 00 e0 fb 00 00 00 00 0c 00 00 f8 00 00 00 00…f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

A typical output might contain:

[root]# lspci00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 22)00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 22)00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 22)00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 22)00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22)00:09.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 22)00:13.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller (rev 22)00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Registers (rev 22)00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 22)00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 22)00:14.3 PIC: Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers (rev 22)00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #400:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #600:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #200:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 100:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 500:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #100:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #200:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #300:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #100:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)04:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev a0)06:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge A (rev 09)06:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge B (rev 09)0a:00.0 IDE interface: JMicron Technology Corp. JMB368 IDE controller0b:01.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02)

The output of configuration space, as example the InfiniBand card. For instance you could

A typical output might contain: [root]# lspci 00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 22) 00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 22) 00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 22) 00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 22) 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22) 00:09.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 22) 00:13.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller (rev 22) 00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Registers (rev 22) 00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 22) 00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 22) 00:14.3 PIC: Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers (rev 22) 00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4 00:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6 00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2 00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 1 00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 5 00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1 00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2 00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3 00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90) 00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller 00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller 00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 04:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev a0) 06:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge A (rev 09) 06:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge B (rev 09) 0a:00.0 IDE interface: JMicron Technology Corp. JMB368 IDE controller 0b:01.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02) The output of configuration space, as example the InfiniBand card. For instance you could check (and correct) if the card is correctly initialized as PCI Gen2 card.

[root]# lspci -xxx -s 04:00.0 04:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev a0) 00: b3 15 3c 67 06 04 10 00 a0 00 06 0c 10 00 00 00 10: 04 00 e0 fb 00 00 00 00 0c 00 00 f8 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 b3 15 08 00 30: 00 00 00 00 40 00 00 00 00 00 00 00 0b 01 00 00 40: 01 48 03 00 00 00 00 00 03 9c ff 7f 11 11 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 10 00 02 00 01 8e e8 07 20 10 00 00 82 f4 03 08 70: 00 00 82 00 00 00 00 00 00 00 00 00 00 00 00 00 80: 00 00 00 00 1f 00 00 00 00 00 00 00 00 00 00 00 90: 02 00 00 00 00 00 00 00 00 00 00 00 11 60 ff 80 a0: 00 c0 07 00 00 d0 07 00 05 00 8a 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SC201025

Memory refined• common problem: slight differences in

recognized memory[user]$ pdsh -w et[60-64,69-73] -u 3 'awk '\''/MemTotal/{print $1,int($2/1024),"MB"}'\'' /proc/meminfo' | dshbak -c----------------et[60-64,69-73]----------------MemTotal: 24095 MB

[user]$ cat memtotal.sh#!/bin/sh

awk '/MemTotal/{print $1,int($2/1024),"MB"}' /proc/meminfo

[user]$ pdsh -w et[60-64,69-73] -u 3 sh memtotal.sh | dshbak -c----------------et[60-64,69-73]----------------MemTotal: 24095 MB

[root@eln1 ~]# cat /proc/swapsFilename Type Size Used Priority/dev/sda2 partition 8393952 58672 -1

[root@eln1 ~]# cat /proc/meminfoMemTotal: 65984928 kBMemFree: 44880144 kBBuffers: 56252 kBCached: 19738156 kBSwapCached: 5724 kBActive: 774872 kBInactive: 19527424 kBHighTotal: 0 kBHighFree: 0 kBLowTotal: 65984928 kBLowFree: 44880144 kBSwapTotal: 8393952 kBSwapFree: 8335284 kBDirty: 2964 kBWriteback: 0 kBAnonPages: 505000 kBMapped: 13928 kBSlab: 696548 kBPageTables: 35260 kBNFS_Unstable: 0 kBBounce: 0 kBCommitLimit: 41386416 kBCommitted_AS: 982696 kBVmallocTotal: 34359738367 kBVmallocUsed: 278100 kBVmallocChunk: 34359458135 kBHugePages_Total: 0HugePages_Free: 0HugePages_Rsvd: 0Hugepagesize: 2048 kB

SC201028

accessing BIOS with dmidecode

• lists hardware and vendor information• entire table is large• needs root privileges• highly useful:

– BIOS version/date and supported features– Board type– CPU type and features– Memory DIMM size, type, manufacturer…

[root]# pdsh -w et[01,03,10-11,13-15] 'dmidecode | grep Date' | dshbak -c----------------et[01,03,10-11,13-15]----------------

Release Date: 04/19/2010

dmidecode dumps the Desktop Management Interface (DMI) information (also known as System Management BIOS aka SMBIOS). It contains a list of hardware components as well as system vendor information like mother board type and BIOS version. The information is possibly unreliable, but in most cases highly useful.

# dmidecode 2.10SMBIOS 2.5 present.80 structures occupying 3493 bytes.Table at 0x7FA32000.

Handle 0x0001, DMI type 38, 18 bytesIPMI Device Information

Interface Type: KCS (Keyboard Control Style)Specification Version: 2.0I2C Slave Address: 0x10NV Storage Device: Not PresentBase Address: 0x0000000000000CA2 (I/O)Register Spacing: Successive Byte Boundaries

Handle 0x0002, DMI type 1, 27 bytesSystem Information

Manufacturer: IntelProduct Name: S5400SFVersion: Not SpecifiedSerial Number: .........UUID: 76A516B5-84B5-11DC-BA6C-001517470362Wake-up Type: LAN RemoteSKU Number: ..........Family: Not Specified

Handle 0x0003, DMI type 2, 16 bytesBase Board Information

Manufacturer: IntelProduct Name: S5400SFVersion: FRU Ver 0.03Serial Number: BZSR74300552Asset Tag: Not SpecifiedFeatures:

dmidecode dumps the Desktop Management Interface (DMI) information (also known as System Management BIOS aka SMBIOS). It contains a list of hardware components as well as system vendor information like mother board type and BIOS version. The information is possibly unreliable, but in most cases highly useful. # dmidecode 2.10 SMBIOS 2.5 present. 80 structures occupying 3493 bytes. Table at 0x7FA32000. Handle 0x0001, DMI type 38, 18 bytes IPMI Device Information Interface Type: KCS (Keyboard Control Style) Specification Version: 2.0 I2C Slave Address: 0x10 NV Storage Device: Not Present Base Address: 0x0000000000000CA2 (I/O) Register Spacing: Successive Byte Boundaries Handle 0x0002, DMI type 1, 27 bytes System Information Manufacturer: Intel Product Name: S5400SF Version: Not Specified Serial Number: ......... UUID: 76A516B5-84B5-11DC-BA6C-001517470362 Wake-up Type: LAN Remote SKU Number: .......... Family: Not Specified Handle 0x0003, DMI type 2, 16 bytes Base Board Information Manufacturer: Intel Product Name: S5400SF Version: FRU Ver 0.03 Serial Number: BZSR74300552 Asset Tag: Not Specified Features: Board is a hosting board Board is replaceable Location In Chassis: Not Specified Chassis Handle: 0x0000 Type: Motherboard Contained Object Handles: 0 Handle 0x0004, DMI type 3, 22 bytes Chassis Information Manufacturer: ..... Type: Rack Mount Chassis Lock: Not Present Version: Not Specified Serial Number: Not Specified Asset Tag: Not Specified Boot-up State: Safe Power Supply State: Safe Thermal State: Safe Security Status: Unknown

OEM Information: 0x81581CF8 Height: 1 U Number Of Power Cords: 1 Contained Elements: 0 Handle 0x0005, DMI type 0, 24 bytes BIOS Information Vendor: Intel Corporation Version: S5400.86B.06.00.0030.112620081512 Release Date: 11/26/2008 Address: 0xE8000 Runtime Size: 96 kB ROM Size: 4096 kB Characteristics: PCI is supported PNP is supported BIOS is upgradeable BIOS shadowing is allowed Boot from CD is supported Selectable boot is supported EDD is supported Print screen service is supported (int 5h) 8042 keyboard services are supported (int 9h) Serial services are supported (int 14h) CGA/mono video services are supported (int 10h) ACPI is supported USB legacy is supported LS-120 boot is supported ATAPI Zip drive boot is supported BIOS boot specification is supported Function key-initiated network boot is supported Targeted content distribution is supported BIOS Revision: 6.0 Firmware Revision: 0.0 Handle 0x0006, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J9A1 Internal Connector Type: None External Reference Designator: Keyboard External Connector Type: PS/2 Port Type: Keyboard Port Handle 0x0007, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J9A1 Internal Connector Type: None External Reference Designator: Mouse External Connector Type: PS/2 Port Type: Mouse Port Handle 0x0008, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J1B1 Internal Connector Type: DB-9 male External Reference Designator: COM 1 External Connector Type: DB-9 male

Port Type: Serial Port 16550A Compatible Handle 0x0009, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J9A2 Internal Connector Type: None External Reference Designator: COM 2 External Connector Type: RJ-45 Port Type: Serial Port 16550A Compatible Handle 0x000A, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J1J1 Internal Connector Type: None External Reference Designator: USB0 HEADER External Connector Type: Access Bus (USB) Port Type: USB Handle 0x000B, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J1J1 Internal Connector Type: None External Reference Designator: USB1 HEADER External Connector Type: Access Bus (USB) Port Type: USB Handle 0x000C, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J4G1 Internal Connector Type: None External Reference Designator: USB2 BRIDGE BOARD External Connector Type: Access Bus (USB) Port Type: USB Handle 0x000D, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J4G1 Internal Connector Type: None External Reference Designator: USB3 BRIDGE BOARD External Connector Type: Access Bus (USB) Port Type: USB Handle 0x000E, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J4G1 Internal Connector Type: None External Reference Designator: USB4 BRIDGE BOARD External Connector Type: Access Bus (USB) Port Type: USB Handle 0x000F, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J5A1 Internal Connector Type: None External Reference Designator: USB5 PORT External Connector Type: Access Bus (USB) Port Type: USB

Handle 0x0010, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J5A1 Internal Connector Type: None External Reference Designator: USB6 PORT External Connector Type: Access Bus (USB) Port Type: USB Handle 0x0011, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J1C2 Internal Connector Type: None External Reference Designator: USB7 RMM CONNECTOR External Connector Type: Access Bus (USB) Port Type: USB Handle 0x0012, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J9K1 Internal Connector Type: Other External Reference Designator: CPU1 FAN External Connector Type: None Port Type: Other Handle 0x0013, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J4K1 Internal Connector Type: Other External Reference Designator: CPU2 FAN External Connector Type: None Port Type: Other Handle 0x0014, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J3K2 Internal Connector Type: Other External Reference Designator: FRNT FAN1 External Connector Type: None Port Type: Other Handle 0x0015, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J3K3 Internal Connector Type: Other External Reference Designator: FRNT FAN2 External Connector Type: None Port Type: Other Handle 0x0016, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J1B3 Internal Connector Type: Other External Reference Designator: FRNT FAN3 External Connector Type: None Port Type: Other

Handle 0x0017, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J1C1 Internal Connector Type: Other External Reference Designator: FRNT FAN4 External Connector Type: None Port Type: Other Handle 0x0018, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J3G2 Internal Connector Type: On Board IDE External Reference Designator: OnBoard Primary IDE External Connector Type: None Port Type: Other Handle 0x0019, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: JA8A2 Internal Connector Type: None External Reference Designator: LAN 1 External Connector Type: RJ-45 Port Type: Network Port Handle 0x001A, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: JA8A1 Internal Connector Type: None External Reference Designator: LAN 2 External Connector Type: RJ-45 Port Type: Network Port Handle 0x001B, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J6A1 Internal Connector Type: None External Reference Designator: Onboard Video External Connector Type: DB-15 female Port Type: Video Port Handle 0x001C, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J1H1 Internal Connector Type: Other External Reference Designator: SATA_0 External Connector Type: None Port Type: SATA Handle 0x001D, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J1G2 Internal Connector Type: Other External Reference Designator: SATA_1 External Connector Type: None Port Type: SATA Handle 0x001E, DMI type 8, 9 bytes

Port Connector Information Internal Reference Designator: J1G1 Internal Connector Type: Other External Reference Designator: SATA_2 External Connector Type: None Port Type: SATA Handle 0x001F, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J1F2 Internal Connector Type: Other External Reference Designator: SATA_3 External Connector Type: None Port Type: SATA Handle 0x0020, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J1F1 Internal Connector Type: Other External Reference Designator: SATA_4 External Connector Type: None Port Type: SATA Handle 0x0021, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J1E2 Internal Connector Type: Other External Reference Designator: SATA_5 External Connector Type: None Port Type: SATA Handle 0x0022, DMI type 10, 6 bytes On Board Device Information Type: Video Status: Enabled Description: ATI Rage XL Handle 0x0023, DMI type 10, 6 bytes On Board Device Information Type: Ethernet Status: Enabled Description: Intel 82563EB Ethernet 1 Handle 0x0024, DMI type 10, 6 bytes On Board Device Information Type: Ethernet Status: Enabled Description: Intel 82563EB Ethernet 2 Handle 0x0025, DMI type 10, 6 bytes On Board Device Information Type: Other Status: Enabled Description: ESB2 Integrated PATA Controller Handle 0x0026, DMI type 10, 6 bytes On Board Device Information

Type: Other Status: Enabled Description: ESB2 Integrated SATA Controller Handle 0x0027, DMI type 10, 6 bytes On Board Device Information Type: Other Status: Enabled Description: NS PC87427 SIO3 Handle 0x0028, DMI type 13, 22 bytes BIOS Language Information Installable Languages: 1 en|US|iso8859-1 Currently Installed Language: en|US|iso8859-1 Handle 0x0029, DMI type 32, 20 bytes System Boot Information Status: No errors detected Handle 0x002A, DMI type 11, 5 bytes OEM Strings String 1: String 2: String 3: String 4: String 5: Handle 0x002B, DMI type 12, 5 bytes System Configuration Options Option 1: J1D2 2-3: Close to clear Password Option 2: J1D3 2-3: Close to clear CMOS Option 3: J3H1 1-2: Close to run BIOS Low Bank Option 4: J1D1 2-3: Close to Force Update Mode Option 5: J1D5: SATA RAID Key Handle 0x002C, DMI type 129, 8 bytes OEM-specific Type Header and Data: 81 08 2C 00 01 01 02 00 Strings: Intel ASF Intel ASF_001 Handle 0x002D, DMI type 4, 40 bytes Processor Information Socket Designation: CPU_1 Type: Central Processor Family: Xeon Manufacturer: Intel(R) Corporation ID: 76 06 01 00 FF FB EB BF Signature: Type 0, Family 6, Model 23, Stepping 6 Flags: FPU (Floating-point unit on-chip) VME (Virtual mode extension) DE (Debugging extension) PSE (Page size extension)

TSC (Time stamp counter) MSR (Model specific registers) PAE (Physical address extension) MCE (Machine check exception) CX8 (CMPXCHG8 instruction supported) APIC (On-chip APIC hardware supported) SEP (Fast system call) MTRR (Memory type range registers) PGE (Page global enable) MCA (Machine check architecture) CMOV (Conditional move instruction supported) PAT (Page attribute table) PSE-36 (36-bit page size extension) CLFSH (CLFLUSH instruction supported) DS (Debug store) ACPI (ACPI supported) MMX (MMX technology supported) FXSR (Fast floating-point save and restore) SSE (Streaming SIMD extensions) SSE2 (Streaming SIMD extensions 2) SS (Self-snoop) HTT (Hyper-threading technology) TM (Thermal monitor supported) PBE (Pending break enabled) Version: Intel(R) Xeon(R) CPU E5462 @ 2.80GHz Voltage: 1.1 V External Clock: 1600 MHz Max Speed: 3400 MHz Current Speed: 2800 MHz Status: Populated, Enabled Upgrade: ZIF Socket L1 Cache Handle: 0x002F L2 Cache Handle: 0x002E L3 Cache Handle: Not Provided Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Core Count: 4 Core Enabled: 4 Thread Count: 4 Characteristics: 64-bit capable Handle 0x002E, DMI type 7, 19 bytes Cache Information Socket Designation: L2-Cache Configuration: Enabled, Not Socketed, Level 2 Operational Mode: Write Back Location: Internal Installed Size: 12288 kB Maximum Size: 12288 kB Supported SRAM Types: Asynchronous Installed SRAM Type: Asynchronous Speed: Unknown Error Correction Type: Single-bit ECC System Type: Unified

Associativity: 24-way Set-associative Handle 0x002F, DMI type 7, 19 bytes Cache Information Socket Designation: L1-Cache Configuration: Enabled, Not Socketed, Level 1 Operational Mode: Write Back Location: Internal Installed Size: 128 kB Maximum Size: 128 kB Supported SRAM Types: Asynchronous Installed SRAM Type: Asynchronous Speed: Unknown Error Correction Type: Single-bit ECC System Type: Data Associativity: 8-way Set-associative Handle 0x0030, DMI type 4, 40 bytes Processor Information Socket Designation: CPU_2 Type: Central Processor Family: Xeon Manufacturer: Intel(R) Corporation ID: 76 06 01 00 FF FB EB BF Signature: Type 0, Family 6, Model 23, Stepping 6 Flags: FPU (Floating-point unit on-chip) VME (Virtual mode extension) DE (Debugging extension) PSE (Page size extension) TSC (Time stamp counter) MSR (Model specific registers) PAE (Physical address extension) MCE (Machine check exception) CX8 (CMPXCHG8 instruction supported) APIC (On-chip APIC hardware supported) SEP (Fast system call) MTRR (Memory type range registers) PGE (Page global enable) MCA (Machine check architecture) CMOV (Conditional move instruction supported) PAT (Page attribute table) PSE-36 (36-bit page size extension) CLFSH (CLFLUSH instruction supported) DS (Debug store) ACPI (ACPI supported) MMX (MMX technology supported) FXSR (Fast floating-point save and restore) SSE (Streaming SIMD extensions) SSE2 (Streaming SIMD extensions 2) SS (Self-snoop) HTT (Hyper-threading technology) TM (Thermal monitor supported) PBE (Pending break enabled) Version: Intel(R) Xeon(R) CPU E5462 @ 2.80GHz Voltage: 1.1 V

External Clock: 1600 MHz Max Speed: 3400 MHz Current Speed: 2800 MHz Status: Populated, Enabled Upgrade: ZIF Socket L1 Cache Handle: 0x0032 L2 Cache Handle: 0x0031 L3 Cache Handle: Not Provided Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Core Count: 4 Core Enabled: 4 Thread Count: 4 Characteristics: 64-bit capable Handle 0x0031, DMI type 7, 19 bytes Cache Information Socket Designation: L2-Cache Configuration: Enabled, Not Socketed, Level 2 Operational Mode: Write Back Location: Internal Installed Size: 12288 kB Maximum Size: 12288 kB Supported SRAM Types: Asynchronous Installed SRAM Type: Asynchronous Speed: Unknown Error Correction Type: Single-bit ECC System Type: Unified Associativity: 24-way Set-associative Handle 0x0032, DMI type 7, 19 bytes Cache Information Socket Designation: L1-Cache Configuration: Enabled, Not Socketed, Level 1 Operational Mode: Write Back Location: Internal Installed Size: 128 kB Maximum Size: 128 kB Supported SRAM Types: Asynchronous Installed SRAM Type: Asynchronous Speed: Unknown Error Correction Type: Single-bit ECC System Type: Data Associativity: 8-way Set-associative Handle 0x0033, DMI type 16, 15 bytes Physical Memory Array Location: System Board Or Motherboard Use: System Memory Error Correction Type: Multi-bit ECC Maximum Capacity: 128 GB Error Information Handle: Not Provided Number Of Devices: 16

Handle 0x0034, DMI type 19, 15 bytes Memory Array Mapped Address Starting Address: 0x00000000000 Ending Address: 0x003FFFFFFFF Range Size: 16 GB Physical Array Handle: 0x0033 Partition Width: 0 Handle 0x0035, DMI type 17, 27 bytes Memory Device Array Handle: 0x0033 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 2048 MB Form Factor: FB-DIMM Set: 1 Locator: ONBOARD DIMM_A1 Bank Locator: Channel A Type: DDR2 FB-DIMM Type Detail: Synchronous Speed: 667 MHz Manufacturer: 80CE Serial Number: 5107EA63 Asset Tag: Not Specified Part Number: M395T5750EZ4-CE65 Handle 0x0036, DMI type 20, 19 bytes Memory Device Mapped Address Starting Address: 0x00000000000 Ending Address: 0x0007FFFFFFF Range Size: 2 GB Physical Device Handle: 0x0035 Memory Array Mapped Address Handle: 0x0034 Partition Row Position: 1 Interleave Position: 1 Interleaved Data Depth: 1 Handle 0x0037, DMI type 17, 27 bytes Memory Device Array Handle: 0x0033 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 2048 MB Form Factor: FB-DIMM Set: 2 Locator: ONBOARD DIMM_A2 Bank Locator: Channel A Type: DDR2 FB-DIMM Type Detail: Synchronous Speed: 667 MHz Manufacturer: 80CE Serial Number: 5107EA88 Asset Tag: Not Specified Part Number: M395T5750EZ4-CE65

Handle 0x0038, DMI type 20, 19 bytes Memory Device Mapped Address Starting Address: 0x00080000000 Ending Address: 0x000FFFFFFFF Range Size: 2 GB Physical Device Handle: 0x0037 Memory Array Mapped Address Handle: 0x0034 Partition Row Position: 1 Interleave Position: 1 Interleaved Data Depth: 1 Handle 0x0039, DMI type 17, 27 bytes Memory Device Array Handle: 0x0033 Error Information Handle: Not Provided Total Width: Unknown Data Width: Unknown Size: No Module Installed Form Factor: FB-DIMM Set: 3 Locator: ONBOARD DIMM_A3 Bank Locator: Channel A Type: DDR2 FB-DIMM Type Detail: Synchronous Speed: Unknown Manufacturer: MemUndefined Serial Number: MemUndefined Asset Tag: Not Specified Part Number: MemUndefined Handle 0x003A, DMI type 17, 27 bytes Memory Device Array Handle: 0x0033 Error Information Handle: Not Provided Total Width: Unknown Data Width: Unknown Size: No Module Installed Form Factor: FB-DIMM Set: 4 Locator: ONBOARD DIMM_A4 Bank Locator: Channel A Type: DDR2 FB-DIMM Type Detail: Synchronous Speed: Unknown Manufacturer: MemUndefined Serial Number: MemUndefined Asset Tag: Not Specified Part Number: MemUndefined Handle 0x003B, DMI type 17, 27 bytes Memory Device Array Handle: 0x0033 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 2048 MB

Form Factor: FB-DIMM Set: 1 Locator: ONBOARD DIMM_B1 Bank Locator: Channel B Type: DDR2 FB-DIMM Type Detail: Synchronous Speed: 667 MHz Manufacturer: 80CE Serial Number: 5107EA2F Asset Tag: Not Specified Part Number: M395T5750EZ4-CE65 Handle 0x003C, DMI type 20, 19 bytes Memory Device Mapped Address Starting Address: 0x00100000000 Ending Address: 0x0017FFFFFFF Range Size: 2 GB Physical Device Handle: 0x003B Memory Array Mapped Address Handle: 0x0034 Partition Row Position: 2 Interleave Position: 2 Interleaved Data Depth: 1 Handle 0x003D, DMI type 17, 27 bytes Memory Device Array Handle: 0x0033 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 2048 MB Form Factor: FB-DIMM Set: 2 Locator: ONBOARD DIMM_B2 Bank Locator: Channel B Type: DDR2 FB-DIMM Type Detail: Synchronous Speed: 667 MHz Manufacturer: 80CE Serial Number: 5107EAB5 Asset Tag: Not Specified Part Number: M395T5750EZ4-CE65 Handle 0x003E, DMI type 20, 19 bytes Memory Device Mapped Address Starting Address: 0x00180000000 Ending Address: 0x001FFFFFFFF Range Size: 2 GB Physical Device Handle: 0x003D Memory Array Mapped Address Handle: 0x0034 Partition Row Position: 2 Interleave Position: 2 Interleaved Data Depth: 1 Handle 0x003F, DMI type 17, 27 bytes Memory Device Array Handle: 0x0033 Error Information Handle: Not Provided

Total Width: Unknown Data Width: Unknown Size: No Module Installed Form Factor: FB-DIMM Set: 3 Locator: ONBOARD DIMM_B3 Bank Locator: Channel B Type: DDR2 FB-DIMM Type Detail: Synchronous Speed: Unknown Manufacturer: MemUndefined Serial Number: MemUndefined Asset Tag: Not Specified Part Number: MemUndefined Handle 0x0040, DMI type 17, 27 bytes Memory Device Array Handle: 0x0033 Error Information Handle: Not Provided Total Width: Unknown Data Width: Unknown Size: No Module Installed Form Factor: FB-DIMM Set: 4 Locator: ONBOARD DIMM_B4 Bank Locator: Channel B Type: DDR2 FB-DIMM Type Detail: Synchronous Speed: Unknown Manufacturer: MemUndefined Serial Number: MemUndefined Asset Tag: Not Specified Part Number: MemUndefined Handle 0x0041, DMI type 17, 27 bytes Memory Device Array Handle: 0x0033 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 2048 MB Form Factor: FB-DIMM Set: 5 Locator: ONBOARD DIMM_C1 Bank Locator: Channel C Type: DDR2 FB-DIMM Type Detail: Synchronous Speed: 667 MHz Manufacturer: 80CE Serial Number: 5107EA75 Asset Tag: Not Specified Part Number: M395T5750EZ4-CE65 Handle 0x0042, DMI type 20, 19 bytes Memory Device Mapped Address Starting Address: 0x00200000000 Ending Address: 0x0027FFFFFFF

Range Size: 2 GB Physical Device Handle: 0x0041 Memory Array Mapped Address Handle: 0x0034 Partition Row Position: 1 Interleave Position: 1 Interleaved Data Depth: 1 Handle 0x0043, DMI type 17, 27 bytes Memory Device Array Handle: 0x0033 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 2048 MB Form Factor: FB-DIMM Set: 6 Locator: ONBOARD DIMM_C2 Bank Locator: Channel C Type: DDR2 FB-DIMM Type Detail: Synchronous Speed: 667 MHz Manufacturer: 80CE Serial Number: 5107EA71 Asset Tag: Not Specified Part Number: M395T5750EZ4-CE65 Handle 0x0044, DMI type 20, 19 bytes Memory Device Mapped Address Starting Address: 0x00280000000 Ending Address: 0x002FFFFFFFF Range Size: 2 GB Physical Device Handle: 0x0043 Memory Array Mapped Address Handle: 0x0034 Partition Row Position: 1 Interleave Position: 1 Interleaved Data Depth: 1 Handle 0x0045, DMI type 17, 27 bytes Memory Device Array Handle: 0x0033 Error Information Handle: Not Provided Total Width: Unknown Data Width: Unknown Size: No Module Installed Form Factor: FB-DIMM Set: 7 Locator: ONBOARD DIMM_C3 Bank Locator: Channel C Type: DDR2 FB-DIMM Type Detail: Synchronous Speed: Unknown Manufacturer: MemUndefined Serial Number: MemUndefined Asset Tag: Not Specified Part Number: MemUndefined Handle 0x0046, DMI type 17, 27 bytes

Memory Device Array Handle: 0x0033 Error Information Handle: Not Provided Total Width: Unknown Data Width: Unknown Size: No Module Installed Form Factor: FB-DIMM Set: 8 Locator: ONBOARD DIMM_C4 Bank Locator: Channel C Type: DDR2 FB-DIMM Type Detail: Synchronous Speed: Unknown Manufacturer: MemUndefined Serial Number: MemUndefined Asset Tag: Not Specified Part Number: MemUndefined Handle 0x0047, DMI type 17, 27 bytes Memory Device Array Handle: 0x0033 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 2048 MB Form Factor: FB-DIMM Set: 5 Locator: ONBOARD DIMM_D1 Bank Locator: Channel D Type: DDR2 FB-DIMM Type Detail: Synchronous Speed: 667 MHz Manufacturer: 80CE Serial Number: 5107EA70 Asset Tag: Not Specified Part Number: M395T5750EZ4-CE65 Handle 0x0048, DMI type 20, 19 bytes Memory Device Mapped Address Starting Address: 0x00300000000 Ending Address: 0x0037FFFFFFF Range Size: 2 GB Physical Device Handle: 0x0047 Memory Array Mapped Address Handle: 0x0034 Partition Row Position: 2 Interleave Position: 2 Interleaved Data Depth: 1 Handle 0x0049, DMI type 17, 27 bytes Memory Device Array Handle: 0x0033 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 2048 MB Form Factor: FB-DIMM Set: 6

Locator: ONBOARD DIMM_D2 Bank Locator: Channel D Type: DDR2 FB-DIMM Type Detail: Synchronous Speed: 667 MHz Manufacturer: 80CE Serial Number: 5107EA6D Asset Tag: Not Specified Part Number: M395T5750EZ4-CE65 Handle 0x004A, DMI type 20, 19 bytes Memory Device Mapped Address Starting Address: 0x00380000000 Ending Address: 0x003FFFFFFFF Range Size: 2 GB Physical Device Handle: 0x0049 Memory Array Mapped Address Handle: 0x0034 Partition Row Position: 2 Interleave Position: 2 Interleaved Data Depth: 1 Handle 0x004B, DMI type 17, 27 bytes Memory Device Array Handle: 0x0033 Error Information Handle: Not Provided Total Width: Unknown Data Width: Unknown Size: No Module Installed Form Factor: FB-DIMM Set: 7 Locator: ONBOARD DIMM_D3 Bank Locator: Channel D Type: DDR2 FB-DIMM Type Detail: Synchronous Speed: Unknown Manufacturer: MemUndefined Serial Number: MemUndefined Asset Tag: Not Specified Part Number: MemUndefined Handle 0x004C, DMI type 17, 27 bytes Memory Device Array Handle: 0x0033 Error Information Handle: Not Provided Total Width: Unknown Data Width: Unknown Size: No Module Installed Form Factor: FB-DIMM Set: 8 Locator: ONBOARD DIMM_D4 Bank Locator: Channel D Type: DDR2 FB-DIMM Type Detail: Synchronous Speed: Unknown Manufacturer: MemUndefined Serial Number: MemUndefined Asset Tag: Not Specified

Part Number: MemUndefined Handle 0x004D, DMI type 9, 13 bytes System Slot Information Designation: Slot 1, PCI EXP x16 Type: x16 PCI Express Current Usage: In Use Length: Long ID: 1 Characteristics: 3.3 V is provided PME signal is supported SMBus signal is supported Handle 0x004E, DMI type 9, 13 bytes System Slot Information Designation: I/O Module Type: x4 PCI Express Current Usage: Available Length: Short ID: 10 Characteristics: 3.3 V is provided PME signal is supported SMBus signal is supported Handle 0x004F, DMI type 24, 5 bytes Hardware Security Power-On Password Status: Not Implemented Keyboard Password Status: Not Implemented Administrator Password Status: Disabled Front Panel Reset Status: Disabled Handle 0xFEFF, DMI type 127, 4 bytes End Of Table

SC201029

IPMI*Intelligent Platform Management Interface

IPMI is an industry standard to control a system on hardware level

– depends on the presence of a BMC (baseboard management controller)

– defines local and remote interfaces

– sometimes implemented together with a remote management module providing a web interface

picture from: http://download.intel.com/design/servers/ipmi/IPMI_and_CIM_Spring2005_IDF.pdf

IPMI homepage: http://developer.intel.com/design/servers/ipmi/index.htm

SC201030

IPMI* Features For Validation

• SEL: system event log• Sensors: current information on system

status• accessible via: ipmitool

– ipmitool sensorlists current sensor reading

– ipmitool sel listlists system event log

– ipmitool sel infoinformation on system event log (free space)

– ipmitool power resethardware reset

The OS driver has to be loaded. Under Redhat* Enterprise Linux one can start the driver via:/etc/init.d/ipmi start

Exemplary output:[root]# ipmitool sensor listBB 12V AUX | 11.904 | Volts | ok | na | 10.416 | 10.726 | 13.144 | 13.578 | naBB 1.1V Vtt | 1.103 | Volts | ok | na | 1.002 | 1.033 | 1.184 | 1.216 | naBB 1.5V AUX | 1.474 | Volts | ok | na | 1.334 | 1.373 | 1.622 | 1.669 | naBB 1.5V ESB | 1.513 | Volts | ok | na | 1.357 | 1.404 | 1.591 | 1.638 | naProc 1 Vcc | 1.159 | Volts | ok | na | na | na | na | na | naProc 2 Vcc | 1.135 | Volts | ok | na | na | na | na | na | naBB 3.3V | 3.354 | Volts | ok | na | 2.941 | 3.027 | 3.578 | 3.681 | naBB 5V | 5.044 | Volts | ok | na | 4.446 | 4.576 | 5.408 | 5.564 | naBB 1.25V_FXD | 1.235 | Volts | ok | na | 1.092 | 1.131 | 1.365 | 1.417 | naBB 1.8V | 1.795 | Volts | ok | na | 1.622 | 1.673 | 1.907 | 1.969 | naBB 1.5V FBD | 1.532 | Volts | ok | na | 1.316 | 1.354 | 1.626 | 1.673 | naBB 0.9V | 0.898 | Volts | ok | na | 0.811 | 0.835 | 0.955 | 0.979 | naBB 3.3V STB | 3.354 | Volts | ok | na | 2.958 | 3.044 | 3.578 | 3.681 | naBB Temp | 37.000 | degrees C | ok | na | 5.000 | 10.000 | 61.000 | 66.000 | naFront Panel Temp | 29.000 | degrees C | ok | na | 0.000 | 5.000 | 44.000 | 48.000 | naMCH Therm Margin | -38.000 | degrees C | ok | na | na | na | 3.000 | 20.000 | naMem Therm Margin | na | degrees C | na | na | na | na | 6.000 | 10.000 | naFan 1A | 12412.000 | RPM | ok | na | 1044.000 | 2030.000 | na | na | naFan 2A | 12354.000 | RPM | ok | na | 1044.000 | 2030.000 | na | na | naFan 3A | 12702.000 | RPM | ok | na | 1044.000 | 2030.000 | na | na | naFan 4A | 12470.000 | RPM | ok | na | 1044.000 | 2030.000 | na | na | naFan 5A | 12296.000 | RPM | ok | na | 1044.000 | 2030.000 | na | na | naFan 1B | 15984.000 | RPM | ok | na | 2442.000 | 3404.000 | na | na | naFan 2B | 15910.000 | RPM | ok | na | 2442.000 | 3404.000 | na | na | naFan 3B | 15910.000 | RPM | ok | na | 2442.000 | 3404.000 | na | na | naFan 4B | 16280.000 | RPM | ok | na | 2442.000 | 3404.000 | na | na | naFan 5B | 16132.000 | RPM | ok | na | 2442.000 | 3404.000 | na | na | naP1 Therm Margin | -62.000 | degrees C | ok | na | na | na | na | na | naP2 Therm Margin | -69.000 | degrees C | ok | na | na | na | na | na | naP1 Therm Ctrl % | 0.000 | unspecified | ok | na | na | na | na | 49.530 | naP2 Therm Ctrl % | 0.000 | unspecified | ok | na | na | na | na | 49.530 | naHSBP Temp | 29.000 | degrees C | ok | na | 0.000 | 5.000 | 50.000 | 54.000 | naPower Unit Stat | 0x0 | discrete | 0x0000| na | na | na | na | na | naWatchdog | 0x0 | discrete | 0x0000| na | na | na | na | na | naPlatform Secu V | 0x0 | discrete | 0x0000| na | na | na | na | na | naPhysical Scrty | 0x0 | discrete | 0x0000| na | na | na | na | na | naFP Interrupt | 0x0 | discrete | 0x0000| na | na | na | na | na | naSystem Event Log | 0x0 | discrete | 0x0000| na | na | na | na | na | naSession Audit | 0x0 | discrete | 0x0000| na | na | na | na | na | naSystem Event | 0x0 | discrete | 0x0000| na | na | na | na | na | naBB Vbat | 0x0 | discrete | 0x0000| na | na | na | na | na | naACPI State | 0x0 | discrete | 0x0100| na | na | na | na | na | naButton | 0x0 | discrete | 0x0000| na | na | na | na | na | naSMI Timeout | 0x0 | discrete | 0x0000| na | na | na | na | na | naNMI Signal State | 0x0 | discrete | 0x0000| na | na | na | na | na | na

The OS driver has to be loaded. Under Redhat* Enterprise Linux one can start the driver via: /etc/init.d/ipmi start Exemplary output: [root]# ipmitool sensor list BB 12V AUX | 11.904 | Volts | ok | na | 10.416 | 10.726 | 13.144 | 13.578 | na BB 1.1V Vtt | 1.103 | Volts | ok | na | 1.002 | 1.033 | 1.184 | 1.216 | na BB 1.5V AUX | 1.474 | Volts | ok | na | 1.334 | 1.373 | 1.622 | 1.669 | na BB 1.5V ESB | 1.513 | Volts | ok | na | 1.357 | 1.404 | 1.591 | 1.638 | na Proc 1 Vcc | 1.159 | Volts | ok | na | na | na | na | na | na Proc 2 Vcc | 1.135 | Volts | ok | na | na | na | na | na | na BB 3.3V | 3.354 | Volts | ok | na | 2.941 | 3.027 | 3.578 | 3.681 | na BB 5V | 5.044 | Volts | ok | na | 4.446 | 4.576 | 5.408 | 5.564 | na BB 1.25V_FXD | 1.235 | Volts | ok | na | 1.092 | 1.131 | 1.365 | 1.417 | na BB 1.8V | 1.795 | Volts | ok | na | 1.622 | 1.673 | 1.907 | 1.969 | na BB 1.5V FBD | 1.532 | Volts | ok | na | 1.316 | 1.354 | 1.626 | 1.673 | na BB 0.9V | 0.898 | Volts | ok | na | 0.811 | 0.835 | 0.955 | 0.979 | na BB 3.3V STB | 3.354 | Volts | ok | na | 2.958 | 3.044 | 3.578 | 3.681 | na BB Temp | 37.000 | degrees C | ok | na | 5.000 | 10.000 | 61.000 | 66.000 | na Front Panel Temp | 29.000 | degrees C | ok | na | 0.000 | 5.000 | 44.000 | 48.000 | na MCH Therm Margin | -38.000 | degrees C | ok | na | na | na | 3.000 | 20.000 | na Mem Therm Margin | na | degrees C | na | na | na | na | 6.000 | 10.000 | na Fan 1A | 12412.000 | RPM | ok | na | 1044.000 | 2030.000 | na | na | na Fan 2A | 12354.000 | RPM | ok | na | 1044.000 | 2030.000 | na | na | na Fan 3A | 12702.000 | RPM | ok | na | 1044.000 | 2030.000 | na | na | na Fan 4A | 12470.000 | RPM | ok | na | 1044.000 | 2030.000 | na | na | na Fan 5A | 12296.000 | RPM | ok | na | 1044.000 | 2030.000 | na | na | na Fan 1B | 15984.000 | RPM | ok | na | 2442.000 | 3404.000 | na | na | na Fan 2B | 15910.000 | RPM | ok | na | 2442.000 | 3404.000 | na | na | na Fan 3B | 15910.000 | RPM | ok | na | 2442.000 | 3404.000 | na | na | na

Fan 4B | 16280.000 | RPM | ok | na | 2442.000 | 3404.000 | na | na | na Fan 5B | 16132.000 | RPM | ok | na | 2442.000 | 3404.000 | na | na | na P1 Therm Margin | -62.000 | degrees C | ok | na | na | na | na | na | na P2 Therm Margin | -69.000 | degrees C | ok | na | na | na | na | na | na P1 Therm Ctrl % | 0.000 | unspecified | ok | na | na | na | na | 49.530 | na P2 Therm Ctrl % | 0.000 | unspecified | ok | na | na | na | na | 49.530 | na HSBP Temp | 29.000 | degrees C | ok | na | 0.000 | 5.000 | 50.000 | 54.000 | na Power Unit Stat | 0x0 | discrete | 0x0000| na | na | na | na | na | na Watchdog | 0x0 | discrete | 0x0000| na | na | na | na | na | na Platform Secu V | 0x0 | discrete | 0x0000| na | na | na | na | na | na Physical Scrty | 0x0 | discrete | 0x0000| na | na | na | na | na | na FP Interrupt | 0x0 | discrete | 0x0000| na | na | na | na | na | na System Event Log | 0x0 | discrete | 0x0000| na | na | na | na | na | na Session Audit | 0x0 | discrete | 0x0000| na | na | na | na | na | na System Event | 0x0 | discrete | 0x0000| na | na | na | na | na | na BB Vbat | 0x0 | discrete | 0x0000| na | na | na | na | na | na ACPI State | 0x0 | discrete | 0x0100| na | na | na | na | na | na Button | 0x0 | discrete | 0x0000| na | na | na | na | na | na SMI Timeout | 0x0 | discrete | 0x0000| na | na | na | na | na | na NMI Signal State | 0x0 | discrete | 0x0000| na | na | na | na | na | na SMI Signal State | na | discrete | na | na | na | na | na | na | na Proc 1 Status | 0x0 | discrete | 0x8000| na | na | na | na | na | na Proc 2 Status | 0x0 | discrete | 0x8000| na | na | na | na | na | na PCIe Link0 | 0x0 | discrete | 0x0000| na | na | na | na | na | na PCIe Link1 | 0x0 | discrete | 0x0000| na | na | na | na | na | na PCIe Link2 | 0x0 | discrete | 0x0000| na | na | na | na | na | na PCIe Link3 | 0x0 | discrete | 0x0000| na | na | na | na | na | na PCIe Link4 | 0x0 | discrete | 0x0000| na | na | na | na | na | na

PCIe Link5 | 0x0 | discrete | 0x0000| na | na | na | na | na | na PCIe Link6 | 0x0 | discrete | 0x0000| na | na | na | na | na | na PCIe Link7 | 0x0 | discrete | 0x0000| na | na | na | na | na | na PCIe Link8 | 0x0 | discrete | 0x0000| na | na | na | na | na | na PCIe Link9 | 0x0 | discrete | 0x0000| na | na | na | na | na | na PCIe Link10 | 0x0 | discrete | 0x0000| na | na | na | na | na | na PCIe Link11 | 0x0 | discrete | 0x0000| na | na | na | na | na | na PCIe Link12 | 0x0 | discrete | 0x0000| na | na | na | na | na | na PCIe Link13 | 0x0 | discrete | 0x0000| na | na | na | na | na | na Proc 1 VRD Hot | 0x0 | discrete | 0x0000| na | na | na | na | na | na Proc 2 VRD Hot | 0x0 | discrete | 0x0000| na | na | na | na | na | na Proc 1 Vcc OOR | 0x0 | discrete | 0x0000| na | na | na | na | na | na Proc 2 Vcc OOR | 0x0 | discrete | 0x0000| na | na | na | na | na | na CPU Popul Error | 0x0 | discrete | 0x0000| na | na | na | na | na | na DIMM A1 | 0x0 | discrete | 0x0400| na | na | na | na | na | na DIMM A2 | 0x0 | discrete | 0x0400| na | na | na | na | na | na DIMM A3 | 0x0 | discrete | 0x0000| na | na | na | na | na | na DIMM A4 | 0x0 | discrete | 0x0000| na | na | na | na | na | na DIMM B1 | 0x0 | discrete | 0x0400| na | na | na | na | na | na DIMM B2 | 0x0 | discrete | 0x0400| na | na | na | na | na | na DIMM B3 | 0x0 | discrete | 0x0000| na | na | na | na | na | na DIMM B4 | 0x0 | discrete | 0x0000| na | na | na | na | na | na Memory Error A | 0x0 | discrete | 0x0000| na | na | na | na | na | na Memory Error B | 0x0 | discrete | 0x0000| na | na | na | na | na | na Memory Error C | 0x0 | discrete | 0x0000| na | na | na | na | na | na Memory Error D | 0x0 | discrete | 0x0000| na | na | na | na | na | na B0 DIMM Spar En | na | discrete | na | na | na | na | na | na | na B0 DIMM Spar Red | na | discrete | na | na | na | na | na | na | na

B1 DIMM Spar En | na | discrete | na | na | na | na | na | na | na B1 DIMM Spar Red | na | discrete | na | na | na | na | na | na | na DIMM C1 | 0x0 | discrete | 0x0400| na | na | na | na | na | na DIMM C2 | 0x0 | discrete | 0x0400| na | na | na | na | na | na DIMM C3 | 0x0 | discrete | 0x0000| na | na | na | na | na | na DIMM C4 | 0x0 | discrete | 0x0000| na | na | na | na | na | na DIMM D1 | 0x0 | discrete | 0x0400| na | na | na | na | na | na DIMM D2 | 0x0 | discrete | 0x0400| na | na | na | na | na | na DIMM D3 | 0x0 | discrete | 0x0000| na | na | na | na | na | na DIMM D4 | na | discrete | na | na | na | na | na | na | na Drv 1 Stat | 0x0 | discrete | 0x008e| na | na | na | na | na | na Drv 2 Stat | 0x0 | discrete | 0x008e| na | na | na | na | na | na Drv 3 Stat | 0x0 | discrete | 0x008e| na | na | na | na | na | na Drv 1 Pres | 0x0 | discrete | 0x0280| na | na | na | na | na | na Drv 2 Pres | 0x0 | discrete | 0x0280| na | na | na | na | na | na Drv 3 Pres | 0x0 | discrete | 0x0280| na | na | na | na | na | na [root]# ipmitool sel info SEL Information Version : 1.5 (v1.5, v2 compliant) Entries : 1921 Free Space : 27112 bytes Percent Used : 53% Last Add Time : 07/27/2010 21:28:43 Last Del Time : 12/03/2007 14:23:13 Overflow : false Supported Cmds : 'Delete' 'Partial Add' 'Reserve' 'Get Alloc Info' # of Alloc Units : 3276 Alloc Unit Size : 20 # Free Units : 1355 Largest Free Blk : 1355 Max Record Size : 5 [root]# ipmitool sel list 2>&1 | head -20 4 | 12/03/2007 | 14:23:13 | Event Logging Disabled #0x09 | Log area reset/cleared | Asserted 18 | 12/03/2007 | 14:23:13 | Processor #0x90 | Presence detected | Asserted 2c | 12/03/2007 | 14:23:14 | Processor #0x91 | Presence detected | Asserted 40 | 12/03/2007 | 14:23:34 | System Event #0x83 | Timestamp Clock Sync | Asserted 54 | 12/03/2007 | 14:23:34 | System Event #0x83 | Timestamp Clock Sync | Asserted 68 | 12/03/2007 | 14:24:20 | System Event #0x01 | OEM System boot event | Asserted

7c | 12/03/2007 | 14:25:48 | System ACPI Power State #0x82 | S0/G0: working | Asserted 90 | 12/03/2007 | 14:29:55 | Button #0x84 | Power Button pressed | Asserted a4 | 12/03/2007 | 14:29:56 | System Event #0x83 | Timestamp Clock Sync | Asserted b8 | 12/03/2007 | 14:29:56 | System Event #0x83 | Timestamp Clock Sync | Asserted cc | 12/03/2007 | 14:29:56 | Power Unit #0x01 | Power off/down | Asserted e0 | 12/03/2007 | 14:30:00 | Power Unit #0x01 | Power off/down | Deasserted f4 | 12/03/2007 | 14:30:01 | Button #0x84 | Power Button pressed | Asserted 108 | 12/03/2007 | 14:30:16 | Drive Slot #0x09 | Device Present 11c | 12/03/2007 | 14:30:18 | System Event #0x83 | Timestamp Clock Sync | Asserted 130 | 12/03/2007 | 14:30:18 | System Event #0x83 | Timestamp Clock Sync | Asserted 144 | 12/03/2007 | 14:31:14 | System Event #0x01 | OEM System boot event | Asserted 158 | 12/03/2007 | 14:32:52 | System ACPI Power State #0x82 | S0/G0: working | Asserted 16c | 12/03/2007 | 14:58:35 | System Event #0x83 | Timestamp Clock Sync | Asserted 180 | 12/03/2007 | 14:58:35 | System Event #0x83 | Timestamp Clock Sync | Asserted Note: not all information is standardized and a Vendor specific tool might be necessary to retrieve the complete information both for SEL and sensors.

SC201031

ipmitool Examples[root]# ipmitool sensorBB 12V AUX | 11.904 | Volts | ok | na | 10.416 | 10.726 | 13.144 | 13.578 | naFront Panel Temp | 29.000 | degrees C | ok | na | 0.000 | 5.000 | 44.000 | 48.000 | naMCH Therm Margin | -38.000 | degrees C | ok | na | na | na | 3.000 | 20.000 | naMem Therm Margin | na | degrees C | na | na | na | na | 6.000 | 10.000 | naFan 1A | 12412.000 | RPM | ok | na | 1044.000 | 2030.000 | na | na | na

[root]# ipmitool sel infoSEL InformationVersion : 1.5 (v1.5, v2 compliant)Entries : 1921Free Space : 27112 bytesPercent Used : 53%Last Add Time : 07/27/2010 21:28:43Last Del Time : 12/03/2007 14:23:13Overflow : falseSupported Cmds : 'Delete' 'Partial Add' 'Reserve' 'Get Alloc Info'# of Alloc Units : 3276Alloc Unit Size : 20# Free Units : 1355Largest Free Blk : 1355Max Record Size : 5

[root]# ipmitool sel list 2>&1 | head -204 | 12/03/2007 | 14:23:13 | Event Logging Disabled #0x09 | Log area reset/cleared | Asserted18 | 12/03/2007 | 14:23:13 | Processor #0x90 | Presence detected | Asserted

SC201032

Vendor Tools to access BIOS: syscfg

• Many vendors provide tools to access and modify BIOS via command line

• for Intel servers: syscfg

Example to dump complete BIOS settings:

[root]# cd /usr/local/syscfg[root]# ./syscfg /s ini /b[root]# less syscfg.INI

[root]# cat /usr/local/syscfg/syscfg.INI; Warning!!! Warning!!! Warning!!!; ---------------------------------; This file has been generated in a system with the BIOS/Firmware; specifications as mentioned under [SYSTEM] section. Please do not; modify or edit any information in this section. Attempt to restore; these information in incompatible systems could cause serious; problems to the sytems and could lead the system non-functional.; Note: The file is best seen using wordpad.

[SYSTEM]BIOSVersion=S5500.86B.01.00.0050.050620101605 ; This field should not be editedFWBootVersion=16 ; This field should not be editedFWOpcodeVersion=53 ; This field should not be editedPIAVersion=53 ; This field should not be edited

[BIOS]

[BIOS::ADVANCED]

[BIOS::ADVANCED::MEMORY CONFIGURATION]

[BIOS::ADVANCED::MEMORY CONFIGURATION::MEMORY RAS AND PERFORMANCE CONFIGURATION]Select Memory RAS Configuration=0 ;Options: 3=Sparing: 0=Maximum PerformanceNUMA Optimized=1 ;Options: 1=Enabled: 0=Disabled

[BIOS::ADVANCED::MASS STORAGE CONTROLLER CONFIGURATION]Intel(R) SAS RAID Module=1 ;Options: 1=Enabled: 0=DisabledConfigure Intel(R) SAS RAID Module=0 ;Options: 1=Intel(R) ESRTII: 0=IT/IR RAIDOnboard SATA Controller=1 ;Options: 1=Enabled: 0=DisabledSATA Mode=1 ;Options: 2=SW RAID: 1=AHCI: 3=COMPATIBILITY: 0=ENHANCEDAHCI Option ROM=1 ;Options: 1=Enabled: 0=Disabled

[BIOS::ADVANCED::SERIAL PORT CONFIGURATION]

[BIOS::ADVANCED::SERIAL PORT CONFIGURATION::SERIAL A ENABLE]Serial A Enable=1 ;Options: 1=Enabled: 0=DisabledAddress=1016 ;Options: 744=2E8: 1000=3E8: 760=2F8: 1016=3F8IRQ=4 ;Options: 4=4: 3=3

[BIOS::ADVANCED::SERIAL PORT CONFIGURATION::SERIAL B ENABLE]Serial B Enable=1 ;Options: 1=Enabled: 0=DisabledAddress=760 ;Options: 744=2E8: 1000=3E8: 760=2F8: 1016=3F8IRQ=3 ;Options: 4=4: 3=3

[BIOS::ADVANCED::USB CONFIGURATION]USB Controller=1 ;Options: 1=Enabled: 0=DisabledLegacy USB Support=0 ;Options: 2=Auto: 1=Disabled: 0=Enabled

[root]# cat /usr/local/syscfg/syscfg.INI ; Warning!!! Warning!!! Warning!!! ; --------------------------------- ; This file has been generated in a system with the BIOS/Firmware ; specifications as mentioned under [SYSTEM] section. Please do not ; modify or edit any information in this section. Attempt to restore ; these information in incompatible systems could cause serious ; problems to the sytems and could lead the system non-functional. ; Note: The file is best seen using wordpad. [SYSTEM] BIOSVersion=S5500.86B.01.00.0050.050620101605 ; This field should not be edited FWBootVersion=16 ; This field should not be edited FWOpcodeVersion=53 ; This field should not be edited PIAVersion=53 ; This field should not be edited [BIOS] [BIOS::ADVANCED] [BIOS::ADVANCED::MEMORY CONFIGURATION] [BIOS::ADVANCED::MEMORY CONFIGURATION::MEMORY RAS AND PERFORMANCE CONFIGURATION] Select Memory RAS Configuration=0 ;Options: 3=Sparing: 0=Maximum Performance NUMA Optimized=1 ;Options: 1=Enabled: 0=Disabled [BIOS::ADVANCED::MASS STORAGE CONTROLLER CONFIGURATION] Intel(R) SAS RAID Module=1 ;Options: 1=Enabled: 0=Disabled Configure Intel(R) SAS RAID Module=0 ;Options: 1=Intel(R) ESRTII: 0=IT/IR RAID Onboard SATA Controller=1 ;Options: 1=Enabled: 0=Disabled SATA Mode=1 ;Options: 2=SW RAID: 1=AHCI: 3=COMPATIBILITY: 0=ENHANCED AHCI Option ROM=1 ;Options: 1=Enabled: 0=Disabled [BIOS::ADVANCED::SERIAL PORT CONFIGURATION] [BIOS::ADVANCED::SERIAL PORT CONFIGURATION::SERIAL A ENABLE] Serial A Enable=1 ;Options: 1=Enabled: 0=Disabled Address=1016 ;Options: 744=2E8: 1000=3E8: 760=2F8: 1016=3F8 IRQ=4 ;Options: 4=4: 3=3 [BIOS::ADVANCED::SERIAL PORT CONFIGURATION::SERIAL B ENABLE]

Serial B Enable=1 ;Options: 1=Enabled: 0=Disabled Address=760 ;Options: 744=2E8: 1000=3E8: 760=2F8: 1016=3F8 IRQ=3 ;Options: 4=4: 3=3 [BIOS::ADVANCED::USB CONFIGURATION] USB Controller=1 ;Options: 1=Enabled:

ClusterValidation v0.13 - Interdisciplinary | Innovativekorobkin/tmp/SC10/tutorials/...pbs-serv2...

Documents

Transcript of ClusterValidation v0.13 - Interdisciplinary | Innovativekorobkin/tmp/SC10/tutorials/...pbs-serv2...