Application Note: Troubleshooting Guide v1.0 for the Avaya ... · Application Note: Troubleshooting...

_____________________________________________________________________________________________ Shenoy; Miller; © 2004 Avaya Proprietary. 1 Feb 2004 All rights reserved.

Application Note

February 2004

Application Note: Troubleshooting Guide v1.0 for the Avaya S8300, S8500, and S8700 Media Servers

ABSTRACT

This guide provides basic information about some of the commands, directories, and files that can help you to troubleshoot the Linux-based servers that run Avaya Communication Manager.

_____________________________________________________________________________________________ Shenoy; Miller; © 2004 Avaya Proprietary. 2 Feb 2004 All Rights Reserved.

Troubleshooting Guide v1.0 for the Avaya S8300, S8500, and S8700 Media Servers

Table of Contents

1. Introduction ...........................................................................................................................5

2. Intended audience ..................................................................................................................5

3. References ..............................................................................................................................5

4. Versions of this document.......................................................................................................5

5. Acronyms ...............................................................................................................................5

6. Introduction to Linux .............................................................................................................5

6.1. Version numbers.................................................................................................................. 5

6.2. Shells and shell scripts ......................................................................................................... 6 6.2.1. Types of shells ................................................................................................ 6 6.2.2. Example of a shell command within Communication Manager ..... 6

7. Commonly used text editors within Linux ..............................................................................6

7.1. The vi editor......................................................................................................................... 6 7.1.1. Movement commands.................................. 7 7.1.2. Text entry commands .................................. 7 7.1.3. Cut, copy, paste, and undo commands ...................... 7 7.1.4. Search and replace commands............................ 8 7.1.5. Save and quit commands................................ 8

7.2. The emacs editor.................................................................................................................. 8 7.2.1. General commands.................................... 9 7.2.2. Movement commands.................................. 9 7.2.3. Cutting and pasting commands ........................... 9

8. Commonly used Linux and Communication Manager commands ........................................10

8.1. Commands that are related to Communication Manager ................................................. 17

9. Commonly accessed directories and files for troubleshooting................................................24

9.1. The /etc/opt/ecs directory ...................................................................................................24

9.2. The /etc/hosts file ................................................................................................................24

9.3. The /etc/opt/defty directory ................................................................................................24

9.4. The/var/log/ecs directory ....................................................................................................25

9.5. The /var/log/messages file ...................................................................................................25

9.6. The /var/crash directory .....................................................................................................25

9.7. The /var/log/defty/dumps directory....................................................................................25

10. Core dumps and mini core dumps ........................................................................................26


11. Introduction to commonly encountered entities within Communication Manager................26

12. Appendix A: Screenshots and additional information on Communication Manager commands 29

12.1. The /sbin/ifconfig command ...............................................................................................29 12.1.1. Avaya S8300 server ................................. 29 12.1.2. Avaya S8700 servers (Multi-Connect and IP-Connect)......... 30

12.2. The almclear command ......................................................................................................31

12.3. The almdisplay command...................................................................................................31

12.4. The fasttop command .........................................................................................................32

12.5. The modserv command ......................................................................................................33 12.5.1. Modem not connected ............................... 33 12.5.2. Modem administered by disabled ....................... 33 12.5.3. Modem administered and ready for access ................. 33

12.6. pingall .................................................................................................................................33

12.7. The restartcause command ................................................................................................34

12.8. The server command ..........................................................................................................35 12.8.1. Avaya S8700 server ................................. 35 12.8.2. Avaya S8300 and S8500 servers......................... 35

12.9. The statapp or systat command .........................................................................................36 12.9.1. Avaya S8700 and S8500 servers......................... 36 12.9.2. Avaya S8300 server (with IA770 on-board messaging installed) .. 36

12.10. The statuslicense command ............................................................................................37 12.10.1. Avaya S8700 and S8500 servers......................... 37 12.10.2. Avaya S8300 server ................................. 37

12.11. The swversion command ................................................................................................38

13. Appendix B: Server behavior on interchange .......................................................................40

14. Appendix C: Connecting to an S8300, S8500, or S8300 server from a Windows-based personal computer.........................................................................................................................................43

14.1. To configure the remote PC for PPP modem connection ..................................................43

14.2. Use Windows for PPP modem connection .........................................................................43

14.3. Using telnet .........................................................................................................................44

14.4. Using ftp..............................................................................................................................45 14.4.1. Example FTP session using the commands listed above : ....... 45

15. Appendix D: A case study in which a segmentation fault causes switch reset........................48

15.1. Log in and connect to the system........................................................................................48

15.2. Examine the logs .................................................................................................................48

15.3. Analyze the problem...........................................................................................................48


1. Introduction This guide provides basic information about some of the commands, directories, and files that can help you to troubleshoot the Linux-based servers that run Avaya Communication Manager.

2. Intended audience This application note is for TSO, ITAC, and Business Partner technicians, and others who must troubleshoot the S8300, the S8500, and the S8700 Media Servers.

3. References The Linux Documentation Project: http://www.tldp.org/

The Avaya Support Web site: http://support.avaya.com/

4. Versions of this document Version # Change

1.0 Original version

5. Acronyms Acronym Meaning CLI command line interface FTP File Transfer Protocol IPSI IP server interface LSP local spare processor MAP maintenance action process MO drive magneto-optical drive SAT system access terminal SNMP Simple Network Management Protocol

6. Introduction to Linux

6.1. Version numbers Linux is a freely distributed version of the Unix operating system. Each particular CD-ROM distribution of Linux has a version number, such as “Red Hat Linux 8.0” or “SuSE Linux Version 6.0.” These version numbers relate only to a particular distribution. The Linux kernel, and the other components of Linux, generally have their own version numbers. The media servers that run Communication Manager 2.0 use Version 2.4.20-AV9, which is packaged with Linux 8.0.


6.2. Shells and shell scripts The shell is a Linux utility that reads and runs the commands that you type. The shell also schedules jobs, redirects input and output, and so on. A shell script is a file that contains a series of shell commands that you run at the shell command prompt.

6.2.1. Types of shells Many types of shells are available for Linux. Some basic aspects of these shells are the same, such as the prompt, most of the commands, and so on. The primary difference is the command language that the scripts within the shells use. Linux shells include the C shell, the Bourne shell, and Bourne again shell. The Bourne again or bash shell is a variant of the Bourne shell. The bash shell includes many advanced features such as job control, command history, a vi-like or an emacs-like interface for command-line editing, and other extensions to the Bourne shell. The bash shell is the default with Communication Manager.

6.2.2. Example of a shell command within Communication Manager The following example shows the ls command with the –l option. You run this command at the bash command prompt. The system displays this command prompt when you log in to Communication Manager. The command prompt is in the format login@servername, which in this example is init@saturn2.

init@saturn2> ls –l total 32 -rw-r--r-- 1 inads susers 27916 Oct 6 11:22 ip2ext.jar -rw-r--r-- 1 inads susers 244 Sep 29 16:59 mibs.env

When you run commands at the shell, you can invoke older commands or edit previous commands that you ran at the shell. To invoke earlier commands, type the following command at the Linux command line interface (CLI), and substitute the name of your text editor:

set –o <name of text editor>

If you use vi as your text editor, type set –o vi. If you use emacs, type set –o emacs.

7. Commonly used text editors within Linux Knowing how to work with text editors is especially useful when you troubleshoot the S8300, S8500, and S8700 servers The most popular Linux text editors are vi and emacs.

7.1. The vi editor The vi editor is a Unix text editing program. To start vi, type vi <file name> at the Unix prompt. If the file name does not exist, vi creates the file for you.


The three basic modes of operation in vi are the edit mode, the command mode, and the visual mode. In this guide, we discuss the edit mode and the command mode. Use the edit mode to enter data. Use the command mode to move the cursor and modify text. The vi editor begins in the command mode. To change to the edit mode, use one of the text entry commands. To change back to the command mode, press ESC. Press Return or Enter after commands that begin with a colon ( : ), a slash (/), or a question mark (?).

7.1.1. Movement commands

Command Action h Move the cursor left. j Move the cursor down. k Move the cursor up. l Move the cursor right. w Move the cursor forward one word. B Move the cursor backward one word. H Move the cursor to the top line on the screen. L Move the cursor to the bottom line on the screen. 1G Move the cursor to the beginning of the file. G Move the cursor to the end of the file. Crtl-f Move forward by one screen. Ctrl-b Move backward by one screen. :<line#> Move to that line number. For example, : 8

means move to line 8. Note: In most instances, you can use the cursor keys to move through a document as you edit the text.

7.1.2. Text entry commands

Command Action a Append text after the cursor. A Append text at the end of the line. i Insert text before the cursor. I Insert text at the beginning of the line. o Open the next line for text insertion. O Open the previous line for text insertion. R Overwrite the rest of line from the cursor. r Replace one character. s Replace one character and enter insert mode.

ESC Return to command mode.

7.1.3. Cut, copy, paste, and undo commands


Command Action cw Replace a word. c$ Replace to the end of the line. dw Delete (cut) the word at the cursor. dW Delete (cut) until the next white space. dd Delete (cut) the current line. d$ Delete to the end of the line. p Paste text after the cursor. P Paste text before the cursor. yw Yank (copy) the word at the cursor. yy Yank (copy) the current line. u Undo the last change. x Delete (cut) the character at the cursor.

7.1.4. Search and replace commands

Command Action /<string> Search forward for

<string>. ?<string> Search backward for <

string >. n Repeat the last search. N Reverse the direction of

the last search. :%s/<pat1>/<pat2>/g Replace all instances of

<pat1> with < pat2>.

7.1.5. Save and quit commands

Command Action :w Save the file without quitting. :q! Quit without saving the file. :e! Return to the last saved version of the file. :wq Save the file and quit. :x Save the file and quit. ZZ Save the file and quit.

7.2. The emacs editor Within the emacs editor, C-x denotes a control character, such as CTRL-X. To type a control character, you press and hold down CTRL, and then press the character key. M-x


denotes a meta character. To type a meta character, you press ESC, and then press the character key. You can also press and hold down the meta key, and then press the character key. On Sun keyboards, the Escape key is ESC (or CTRL-[). The meta key is next to the spacebar, and is labeled with a small diamond.

7.2.1. General commands

7.2.2. Movement commands

Command Action C-b Move backward one character. C-f Move forward one character. C-n Move down one line. C-p Move up one line. C-v Move down one screen. M-v Move up one screen.

7.2.3. Cutting and pasting commands

Command Action DEL Delete the character before the cursor. C-@ or C-SPACE

Place a mark in text.

C-d Delete the character under the cursor. C-k Kill to the end of the line. C-w Delete from the mark to the current position in

the text. C-x DEL Kill to the beginning of the sentence. C-y Yank the previous kill from the kill ring.

Command Action C-g Stop a partially typed or an executing command. C-h t View the emacs tutorial. C-l Redraw the screen. C-r Search backward. C-s Search forward. C-x i Insert a file. C-x s Save a file. C-x u Undo. C-x C-c Exit the emacs editor. C-x C-f Change to a new file. C-x C-w Write to a specified file. M-x info View emacs info.


Command Action M-DEL Kill to the beginning of the previous word. M-d Kill to the end of the next word. M-k Kill to the end of the sentence. M-z <char> Kill to the next occurrence of <char>. 8. Commonly used Linux and Communication Manager

commands Command Summary of use

cd Use the cd command to change the current directory. Type: cd <name of directory> You are always in a current directory, and can access any directories that are hierarchically above or below the current directory, as long as you have the right permissions. Example: cd games If the directory games is not located below the current directory, you must provide the complete path name. Example : cd /usr/games To move up one directory, type: cd ..

exit or logout

Use exit or logout to end a user session. Both commands have the same effect, which is to disconnect from your Linux–based computer, or end a user session. Note that leaving your account logged on can be a security concern. Avaya recommends that you always promptly log out when you finish using your account, especially if you are logged in as root.

find Use the find command to search for files on your file system.

For example, use find . / -name <file name> -print to find the file called <file name> on your file system, starting with the / (root) directory. The value for < file name > can contain wildcards, which you introduce with the special characters asterisk (*) or question mark (?). Use the -print option to print the results to


standard output.

The find command has many powerful search options. For example, you can search by date, size, permissions, or owner. Use the man find command or the info find command to get more information on the find command.

The following commands are more complex examples of how to use the find command to accomplish some useful tasks:

find $HOME -name core -exec rm -f {} This command starts at your home directory, and finds all files with names that contain the string “core.” On each file, this command then performs the action rm –f, which deletes the file without confirmation. The braces symbol {} stands for the file that the command finds.

Caution: The rm –f command permanently deletes all files that match the criterion.

Caution: If you run this command at the highest root directory (/) level in your system, this command permanently deletes all files with names that contain “core.”

• find /dev -user "peter" |more This command prints the file name for all devices that the user "peter" owns. The |more option displays one page or screen at a time, and waits for user input to display the next page. Since the default action of find is to print the file name, you do not have to specify the print option within the command.

grep Certain patterns in the log messages in /var/log/ecs can provide clues on some obvious problems that can help you with basic troubleshooting. Use the grep command to search the ecs log files.

In all of the following examples, we are in the directory /var/log/ecs, and running the following command:

grep –i string *.log | more Note that the –i option makes the search case insensitive. Replace string above with one of the following sample patterns:

• interchange. Displays records in *.log files that include the word interchange (interchange of the active and standby servers).

• CheckSlot. Helps you debug potential problems with IPSIs,


especially when fatal errors are reported. • ALLOC_BUFF. Tells you when Communication Manager, as a

whole, runs out of buffers. This condition causes an interchange on the duplicated S8700 servers that is equivalent to a segmentation fault. ALLOC_BUFF failures, and analysis of the aftermath, can provide much information on what takes place on a particular system.

• EIP. Tells you when a segmentation fault occurred in a process. A segmentation fault causes a restart of some type to occur in the system, depending on the restart level of the process that experienced the segmentation fault. Analysis of the segmentation fault can uncover bugs in the code.

• FATAL. Indicates that an IPSI board entered FATAL FAULT. This state means that all LAPD links, and therefore all calls, that are associated with that board fail. Note that in a port network with duplicated IPSIs, the presence of a FATAL in the log might not indicate call failures on that port network.

• OVERLOAD (1 or 9). Indicates that the switch is running at a very high occupancy, and is no longer handling off-hooks. Calls are being lost if you see this message in the ecs logs.

Many other messages in the ecs log files provide similar help, but are too numerous to discuss here.

kill Use kill to kill a process that is stopped. Type:

kill <pid>

where pid is the exact number of the process that you want to kill.

See the ps command below for how to find the PID of a process that you want to kill.

Caution: Be careful when you use the kill command on a Communication Manager platform. Killing certain processes can have serious negative effects.

locate Use the locate command to find data files, programs, directories, and objects that match your search. For example, type:

locate license.txt

The locate command shows you every related file or directory that matches your search criterion. The result of the locate command can be a long list. To narrow the search, type:

locate <search term> | more


The locate command works quickly, if the locate database is up to date. The system automatically updates the locate database each night. So, any new files or commands that were created in the system on the day that you run locate might not appear in the output. If you have root privileges, you can log in as root, and run the updatedb command to update the locate database manually.

logv logc logw

The logv command puts the selected output into a temporary file. The logv command then puts you in the editor on that file, while the logc command outputs the selected output to standard output. The logw command watches the logs for a certain pattern until you enter ^C. The following examples show the use of these commands.

• logv today pcd This command gives you all the process manager messages from the debug trace log. The debug trace log is the default for the logv command.

• logv yesterday lm wd lxsys This command merges and sorts the debug trace log (lm, for logmanager), watchdog (wd), and /var/log/messages (lxsys, for linux syslog) logs from yesterday.

• logc 20031001 > <filename> This command puts the contents of the debug trace log from Oct 1, 2003 into the file <filename>.

• logw pro= This command watches the debug trace log for software errors. This command then prints the errors out as the errors occur until you enter ^ C.

The following description shows the options that are available for the logv, the logc, and the logw commands from the Linux CLI of a server that is running Communication Manager:

logv [OPTIONS]... [LOGS...] [FILTERS]... logc [OPTIONS]... [LOGS...] [FILTERS]... logw [OPTIONS]... [LOGS...] [FILTERS]...

Options to modify command operation: -? produce this message -c cat the log instead

(set if called with name logc) -ls list the names and the sizes of log files -w watch the log instead


(set if called with name logw) -v show the version of the command -b remove blank lines -l only the latest file in a log searched

(the default is all) -s strip off the timestamp

(or other header) on each line -clear clear the log files

Logs that can be displayed: all all possible lm logmanager debug trace (default) lxboot Linux boot messages lxcron Linux cron (scheduling) daemon lxsys Linux syslog lxsec Linux access security log lxwtmp Linux login/logout/reboot log lxx Linux file transfer log cmds platform command history log wd watchdog logs Filters to display events that match a particular criterion. Multiple applications and times can be given:

App filter events for a particular application or process(for example, VFM,audit,NW, ...)

time filter for a particular date or time yyyy[mm[dd:[HH[MM[SS[mmm]]]]]]

today yesterday

ls Use ls <directory-name> to view a list of the files and subdirectories in the directory named <directory-name>. If you omit < directory-name >, the listing is for the current directory. For example, if you are in the current directory named “games,” and you type ls, the system lists all files and subdirectories that are in the games directory. Examples: Type ls -ltr to see a list of all files and all directories that are located in the current directory, sorted by the time of file creation. This command is especially useful if you are looking for the latest logs in /var/log/ecs. To view those logs, type ls /var/log/ecs –ltr. Use the -a option with ls to list hidden files, such as .rc files. To list hidden files, type ls –a, or ls –altr for a time-sorted list.

man Use the man command to find help for most commands. The man


command displays the manual pages, which tell you about all the other commands. The man command has two forms: man [section-number] page_name man -k search_expression The first form displays the page for a specific command. For example, man ls displays the manual page for the ls command. Use the section option to narrow the search to a particular part of the documentation. In addition to the man command, Linux provides two other ways to get help with a specific command: <command_name> -? <command_name> --help

more Use the more command to view the contents of a text file. Type more <filename> Note that if you use more with a nontext file type, the system displays a list of unrecognizable characters.

ps Use the ps command to displays all current processes. This command is also directly linked to issues with stopped processes, or "stopped jobs." Type ps to list all of current processes that are running or stopped. For example: PID TTY STAT TIME COMMAND 23036 pl S 0:00 -csh 23070 pl R 0:00 vi The number under PID is the process identification number. Type ps –ef <process name> to look up the PID of the process <process name>. See Section 11 of this document for information on Communication Manager processes.

pwd Use the pwd command to view the full path name of the directory that you are currently in. This command is especially useful when you forget which directory that you changed to.

uname Use the uname command to print system information. To use the -a option to include all information, such as kernel, nodename, and so on, type: uname -a

whatis Use the whatis command to determine what a specific command does. At the command line, type: whatis <command>


The result is usually a one-line summary of the command. which Use the which command to show the path name to a command. Type:

which <command>


8.1. Commands that are related to Communication Manager

Command Brief description Caveats and frequently used options

almcall Set the telephone numbers to call the INADS alarm receivers. Enable or disable alarm abbreviation on those numbers, and the interval between retries.

almdisplay Display server alarms. almclear Clear server alarms. Typically used with the –a

option. Clears server alarms only. Communication Manager alarms must first be cleared within Communication Manager before this command actually clears the alarms at the Linux shell.

almenable Enable and disable dial out and SNMP alarm origination.

almnotif Set "restart notification,” “clear alarm notification,” and "suspension threshold."

almsnmpconf Administer SNMP alarming to INADS.

almsuppress Suppress or unsuppress alarm origination, or check the state of alarm suppression.

Use the -s option with the ((y)es) parameter to suppress alarm origination for 30 minutes.

corevector Request or clear core dump requests. The core dump files are generated prior to the execution of the restart requested. Use corevector -? for detailed information.

dhelp List popular Linux commands for the servers that are running Communication Manager

dsat Run a system access



terminal (SAT) session for Communication Manager.

ftpserv Configure FTP service and access.

Use –f on/off to enable or disable FTP service. Use –a on/off to enable or disable anonymous FTP. Note: This command works either when you are logged in as root, or when you use sudo to run the command (for example, sudo ftpserv –a on).

ipsiversion Get the version of the IPSIs that are in the system.

Use –a for all IPSIs. Use–c for a particular IPSI.

loginreport Report on Linux logins. Use –a for all login reports. Use –sl for successful Linux logins. Use –ul for unsuccessful Linux logins. Use –sd for successful CM logins. Use –ud for unsuccessful CM logins. Use –lo for login lockouts.

loadlicense Command line version of the utility to load the license file.

loadpwd Command line version of the utility to load the Avaya authentication file. This file is also called the password file.

locktrans Lock translations so that no save translations can be performed during the period of the lockout.

See also unlocktrans.

mo Perform maintenance that is related to the magneto-optical (MO) drive on the S8700 platforms.

Use –c to check the MO drive. Use –l to list the contents of the MO drive. Use –g to get a file from the drive.

patch_apply Apply an “installed” patch to Communication Manager (see patch_install).

Communication Manager 1.1 and early 1.2 systems require the sudo command before the patch_apply command, for example, sudo patch_apply. In servers that run the later 1.2


Command Brief description Caveats and frequently used options version and all 1.3 versions of Communication Manager, the sudo command is not required. Also, with Communication Manager 2.0, the patch_install command no longer exists. See update_activate for more information.

patch_info Provides information about a patch that is already installed.

With Communication Manager 2.0, the patch_info command no longer exists. See update_info for more information.

patch_install Installs the patch into the system (see patch_apply).

Communication Manager 1.1 and early 1.2 systems require the sudo command before the patch_install command, for example, sudo patch_install. In servers that run the later 1.2 version and all 1.3 versions of Communication Manager, the sudo command is not required. Also, with Communication Manager 2.0, the patch_install command no longer exists. See update_unpack for more information.

patch_remove Script removes a previously activated patch on the system. For a list of patches that are installed in the system, use patch_show.

Communication Manager 1.1 and early 1.2 systems require the sudo command before the patch_remove command, for example, sudo patch_remove. In servers that run the later 1.2 version and all 1.3 versions of Communication Manager, the sudo command is not required. Also, with Communication Manager 2.0, the patch_remove command no longer exists. See update_remove for more information.



patch_show Lists the patches that are installed on the system.

Communication Manager 2.0, the patch_show command no longer exists. See update_show for more information.

patch_uninstall Script uninstalls a patch from the target machine. For a list of patches that are installed in the system, use patch_show.

Communication Manager 1.1 and early 1.2 systems require the sudo command before the patch_uninstall command, for example, sudo patch_uninstall. In servers that run the later 1.2 version and all 1.3 versions of Communication Manager, the sudo command is not required. Also, with Communication Manager 2.0, the patch_uninstall command no longer exists. See update_deactivate for more information.

pingall With various options, provides the ability to ping certain interfaces.

Mostly applies to the S8700 and the S8500 platforms.

productid Query or set the product ID for Communication Manager or Messaging products.

Use -p to set the Communication Manager product ID. Use -m to set the Messaging product ID.

resetipsi Reset one or more IPSI boards.

Use -a to reset all IPSIs. Use -c to reset a particular IPSI. This command applies only to the S8700 and the S8500 platforms.

restartcause Displays the list of the known restarts that occurred on Definity© or Communication Manager.

sat See dsat. save_trans Equivalent of the

save_trans command that you run from the SAT.

serialnumber Responds with the serial number in the license file, or detected in the network by the server.

Use -l to report the serial number from the license file. Use -q to force a query of IPSIs or G700s for the serial numbers.



server Display server duplication

status if applicable, and the simultaneous display of status for each server, along with the overview of server health.

Use -b busyout this server. Use -c for continuous looping of status updates. Use -i to interchange active and standby servers. Use -r to release this server if previously busied out. Use -f (with –i) to force an interchange if necessary.

setnic Primarily used on servers that are running Communication Manager to set the speed and duplex settings on the NIC of the server.

Use -q <interface> to query the current settings for <interface >. Caution: You can use setnic to query the interface Speed/Duplex settings. However, if you use setnic to set the Speed/Duplex of an interface, that particular interface will be reset. Be careful about doing this on a customer system, especially on an active server.

sudo Use to run some commands as root temporarily. This command is usually used by customers or service logins that are not root.

start Usually used to start Communication Manager processes.

Use -a to start all processes. Use –c for continuous update of output.

statapp Verify what Communication Manager applications are currently running on the servers.

See systat.

statuslicense Check the current license status.

Use -v (for verbose) to obtain more information.

statusserver This command is similar to the server command

See server.

stop Stop all Communication Manager processes.

Use -a to stop all processes. Use –c for continuous update of output. Use –f to force a process to shut down if necessary.

swversion Check the current Use -r to check the standby



software version on the active or the standby partitions. Note: You must be sroot to run portions of this command successfully.

partition. Use -v to check the Communication Manager version string. Use -l to check the Communication Manager load string. Use -a to check all fields.

systat Get an update of the processes that are currently running for Communication Manager.

Use –c for continuous update of output. Use -d to list all applications that are not functioning. Use -D to exit continuous update when no processes are functioning. Use -U to exit continuous update when all processes are functioning. Use –l to list the status of the applications given. Use –s to show the status of only the named applications (see -l .) See also statapp.

telnetenable Configure the telnet service for the IPSIs.

Use -d to disable telnet service. Use –c [cab#carrier] to target an IPSI board with cabinet number and carrier ID. Use -p <ip_address> to target an IPSI board using its IP address.

testdupboard Test the duplication memory card on the S8700 server.

Use –s to run the short test. This option is the default.

testinads Test the functionality of alarm processing between the system and INADS.

testled Test all the LEDs in the server.

Use –a: to test LEDs on both the server and the duplication memory card LEDs Use –d: to test the Trans LED on the duplication memory card. Use –s: to test the LEDs on the server. This command applies only to the S8700 and the S8500 platforms.

testmodem Tests the modem that is connected to the server that is running Communication Manager.

Use –s (short) to run the handshake and the offhook tests. Use –t to specify a certain component to be tested.



uname Print system information. Use –a to print all information, such as the kernel, the node name, and so on.

unlocktrans Unlock the “locked” translations. Translation saves will now be successful.

See also locktrans.

update_info Display information about the specified update. This command replaces the patch_info command that was used prior to Communication Manager 2.0.

Usage: update_info <update-id> Note: In Communication Manager 2.0, the term patch is changed to software update.

update_show Display information about the specified update. This command replaces the patch_show command that was used prior to Communication Manager 2.0.

Use –a: to display activated updates only. Use –u: to display unpacked updates only. Use –h: to display help for this command.

update_unpack Unpack the software update. This command is the step prior to “activating” a software update. This command replaces the patch_install command that was used prior to Communication Manager 2.0.

update_activate Activates a previously unpacked update on the server. This command replaces the patch_apply command that was used prior to Communication Manager 2.0.

update_deactivate Deactivates a previously activated update on the server. This command replaces the



patch_apply command that was used prior to Communication Manager 2.0.

upate_remove Permanently removes a “deactivated” update from the server. This command replaces the patch_uninstall command that was used prior to Communication Manager 2.0.

update_remove <update-id> to remove the update. Use <update-id> update_remove –a to remove all updates.

vilog Run vi and open current Communication Manager log files.

9. Commonly accessed directories and files for troubleshooting

This section describes some useful files, and some common things to look for in log files that can be quick indicators of problems.

9.1. The /etc/opt/ecs directory

This directory contains several important files:

• ecs.conf This file is the configuration file for the switch. This file is essential for Communication Manager Applications to run correctly. Flags that are set incorrectly in this file can cause numerous problems in the switch.

• servers.conf This file contains information on the IP addresses of the servers and the control networks. This information is useful for troubleshooting possible network problems.

9.2. The /etc/hosts file This file contains the IP addresses of all IPSIs, Cajun-family devices, and servers in the system. This information is useful for troubleshooting possible network problems.

9.3. The /etc/opt/defty directory

This directory contains the translation files, which are named xln1 and xln2. The xln1 and xln2 files are identical in all S8X00 servers, and are shadowed in the S8700 server. The


translation files store all the information that the switch uses with regard to stations, trunks, and so on. Beginning with Communication Manager 2.0, the xln2 file is no longer used.

This directory also contains an lspList. This file is usually 0 bytes long, unless one or more local spare processors (LSPs) are registered to this server. If LSPs are registered, this file contains the IP address of the LSPs to which Communication Manager has tried to send the translation files.

9.4. The/var/log/ecs directory This directory that contains most of the log files that the switch generates. This directory contains three very useful types of files.

• ecs log files

These log files are marked by the date on which the log files occur. These files provide information from Communication Manager processes.

• Commandhistory

This file contains the history of commands that are issued on the server. This file shows such things as when server interchanges were done, when patches were applied, and when servers were started and stopped. Note that this file does not record every command that is run at the Linux CLI.

• wdlog

This file is the watchdog log, the process in Communication Manager Applications that watches over all other processes to ensure proper behavior. This log outputs occupancy profiles on a per-process basis if the system is running at high occupancy.

9.5. The /var/log/messages file This file contains more information about system behavior, including information on modems, security, and traps.

9.6. The /var/crash directory If the core-vector is set on a server that is running Communication Manager, a core dump is generated. This directory contains core dumps on Linux-based servers. See Section 10, “Core dumps and mini core dumps,” for some basic information about core dumps.

9.7. The /var/log/defty/dumps directory


If the core vector is not set on a server that is running Communication Manager, a mini core dump might be generated. As the name suggests, the mini core dump is a smaller version of the core dump. This directory contains core dumps on Linux-based servers. See Section 10, “Core dumps and mini core dumps,” for some basic information about mini core dumps.

10. Core dumps and mini core dumps

A core dump is a file that contains a snapshot of the memory image of the server at the time that the core dump is generated. A core dump is required to debug system failures in depth. System failures can vary from a single process restart to a reload of the server. To generate a core dump, you set a flag in the low-level maintenance monitor (LMM). This flag can be enabled or disabled. When enabled, this flag can generate core dumps under various conditions. On Linux-based servers, the /var/crash directory contains core dumps. A mini core dump is usually generated without setting any flags. However, a mini core dump generates less useful information than a core dump. On Linux-based servers, the /var/log/defty/dump contains mini core dumps.

11. Introduction to commonly encountered entities within Communication Manager

This section describes some of the many processes that comprise Communication Manager. All messages in the /var/log/ecs log files come from one of these entities. The following list is not a complete list of all the processes, and these descriptions are very basic. However, even a general idea of what these processes are, and how these processes communicate with one another, can help you troubleshoot problems on the system.

• capro. The call-processing process for Communication Manager Applications. This process is made up of many subprocesses, including the user manager, the service dispatcher, and the connection manager. Each of these subprocesses performs different functions.

• hmm. The high-level maintenance manager that handles all maintenance for Communication Manager Applications. This process prints all process errors (proc-errors), and is ultimately responsible for the initialization of the system. This process also dispatches the maintenance action processes or maps that handle much of the maintenance work, including the initmap. Note that the high-level maintenance manager process is responsible for only printing proc-errors (process errors).

• lmm. The low-level maintenance monitor. This bootstrap loader loads Oryx-Pecos into memory and then passes control to it.

• prc_mgr. The process manager. This process is partially responsible for creating and destroying all processes. The process manager ensures that the system comes up correctly, and uses the high-level maintenance manager process and the initmap process to manage what recovery actions to take when a process has a problem.

• gmm. The general maintenance manager. This process receives alarms directly from platform processes, and ensures that alarms are created and reported


appropriately. For MultiVantage processes, the maintenance data manager spawns an rma_mgr if an alarm is necessary.

• rma_mgr. The process that the MDM spawns to call out an alarm to the general maintenance manager process. The rma_mgr process is created and immediately destroyed by the prc_mgr after the process performs the call.

• arbiter. The process that makes decisions on server interchanges. This process takes input from the packet control driver, the process manager, the general maintenance manager, and the watchdog processes to make decisions on which server is healthier. This process then uses that information to determine which server should be active.

• watchd. The watchdog process that monitors all processes in the system, and maintains heartbeats with both Communication Manager and the platform processes. This process is responsible for stopping and starting processes when necessary. This process monitors the entire system, while the initmap essentially monitors only the processes that are related to Communication Manager.

• pcd. The packet control driver process provides the interface between the Communication Manager application software and the PKTINT. All packet bus communications that come up from the PKINT travel through the pcd (over IP for Linux-based servers) to reach capro.

• lip. The link initiator process manages the pktint resource, and is the layer 2 LAPD manager. This process deals only with managing the links, and is not concerned with the data that travels through the links.

• gip. The gaz interface process creates all sockets for the user_mgr to communicate with IP endpoints. One of the primary purposes of this process is to serve as the resource manager for the CLAN board.

• fastmap. A maintenance action process (map) that the high-level maintenance manager process spawns when high priority actions are required. The fastmap process is usually allocated by the high-level maintenance manager process to handle IPSI interchanges or various other SPECON recovery actions. This process has a higher priority than the call processing process.

• border. The process that provides communications between platform processes and Communication Manager processes. For example, the arbiter process requires the border process to relay messages to the pcd.

• initmap. This process is spawned by the high-level maintenance manager process, and is responsible for creating all permanent processes and maintaining sanity with those processes. This process also notifies the system when the system reaches normal_op (normal operation).

• ndm. The duplication manager process maintains shadowing between an active server and a standby server. This process performs tests to ensure that the duplication strategy of shadowing and filesync works as designed, and verifies that the standby side is refreshed.

• pam. The permissions and access manager process prints all commands that are entered in a SAT into the log file. This information is useful when troubleshooting to help you determine what commands were issued on a system.


12. Appendix A: Screenshots and additional information on Communication Manager commands

12.1. The /sbin/ifconfig command Use the /sbin/ifconfig command to determine the IP addresses and the status of all the Ethernet interfaces on the servers that run Communication Manager.

12.1.1. Avaya S8300 server

Interface name Function eth0 S8300 services interface eth1.0000 S8300 NIC or PROCR interface ppp0 IP address for the modem connection Interface eth1.0000 indicates the default VLAN, and eth1.4095 indicates the internal VLAN. L0 is the loopback interface. init@definity1> /sbin/ifconfig eth0 Link encap:Ethernet HWaddr 00:04:0D:05:D9:F8 inet addr:192.11.13.6 Bcast:192.11.13.7 Mask:255.255.255.252 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2120 errors:0 dropped:0 overruns:0 frame:0 TX packets:1657 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 Interrupt:5 Base address:0x1080 Memory:d0f5e000-d0f5ec40 eth1 Link encap:Ethernet HWaddr 00:04:0D:05:D9:F2 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:30073925 errors:0 dropped:0 overruns:0 frame:0 TX packets:23022685 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 Interrupt:5 Base address:0x10c0 Memory:d0f60000-d0f60c40 eth1.0000 Link encap:Ethernet HWaddr 00:04:0D:05:D9:F2 inet addr:172.16.151.226 Bcast:172.16.151.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:25179279 errors:0 dropped:0 overruns:0 frame:0 TX packets:21176579 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 eth1.4095 Link encap:Ethernet HWaddr 00:04:0D:05:D9:F2 inet addr:127.1.1.31 Bcast:0.0.0.0 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:4894861 errors:0 dropped:0 overruns:0 frame:0 TX packets:1846100 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.255.0.0 UP LOOPBACK RUNNING MTU:3924 Metric:1 RX packets:9343463 errors:0 dropped:0 overruns:0 frame:0


TX packets:9343463 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 ppp0 Link encap:Point-to-Point Protocol inet addr:10.1.253.1 P-t-P:10.1.253.2 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1 RX packets:82 errors:0 dropped:0 overruns:0 frame:0 TX packets:73 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:10

12.1.2. Avaya S8700 servers (Multi-Connect and IP-Connect)

Interface name

Multi-Connect IP-Connect

eth0 Control-Network A Control-Network A Customer LAN eth1 S8700 services interface S8700 services interface eth2 S8700 duplicated interface (or

“dup” interface) for this server S8700 duplicated interface (or “dup” interface) for this server

eth3 Control-Network B Unused eth4 Customer LAN Unused ppp0 IP address for the modem

connection IP address for the modem connection

To save space, the following example represents only a partial configuration from an S8700 Multi-Connect system. init@8700a1> /sbin/ifconfig eth0 Link encap:Ethernet HWaddr 00:02:B3:D0:2B:66 inet addr:198.152.254.201 Bcast:198.152.254.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1884883092 errors:7150295 dropped:0 overruns:0 frame:7150295 TX packets:2147483647 errors:0 dropped:0 overruns:2 carrier:0 collisions:0 txqueuelen:100 Interrupt:10 Base address:0xef00 Memory:e0a71000-e0a71c40 eth0:0 Link encap:Ethernet HWaddr 00:02:B3:D0:2B:66 inet addr:198.152.254.200 Bcast:198.152.254.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:10 Base address:0xef00 Memory:e0a71000-e0a71c40 eth1 Link encap:Ethernet HWaddr 00:02:B3:D0:2B:67 inet addr:192.11.13.6 Bcast:192.11.13.7 Mask:255.255.255.252 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:16137 errors:0 dropped:0 overruns:0 frame:0 TX packets:19158 errors:28 dropped:0 overruns:0 carrier:28 collisions:0 txqueuelen:100 Interrupt:10 Base address:0xee80 Memory:e0a73000-e0a73c40 eth2 Link encap:Ethernet HWaddr 00:00:50:07:01:57 inet addr:192.11.13.13 Bcast:192.11.13.15 Mask:255.255.255.252 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:25942051 errors:53189 dropped:0 overruns:1 frame:53189 TX packets:26826200 errors:0 dropped:0 overruns:58 carrier:0 collisions:0 txqueuelen:100 Interrupt:7 Base address:0xdf00 Memory:e0a75000-e0a75c40


eth3 Link encap:Ethernet HWaddr 00:00:50:07:01:58 inet addr:198.152.255.201 Bcast:198.152.255.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:193025650 errors:0 dropped:0 overruns:0 frame:0 TX packets:191898744 errors:0 dropped:0 overruns:4 carrier:0 collisions:0 txqueuelen:100 Interrupt:11 Base address:0xde80 Memory:e0a77000-e0a77c40 eth3:0 Link encap:Ethernet HWaddr 00:00:50:07:01:58 inet addr:198.152.255.200 Bcast:198.152.255.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:11 Base address:0xde80 Memory:e0a77000-e0a77c40 eth4 Link encap:Ethernet HWaddr 00:00:50:07:01:59 inet addr:128.8.33.45 Bcast:128.8.33.63 Mask:255.255.255.192 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:11455953 errors:0 dropped:0 overruns:0 frame:0 TX packets:8468314 errors:2 dropped:0 overruns:28 carrier:2 collisions:9824 txqueuelen:100 Interrupt:5 Base address:0xdd80 Memory:e0a79000-e0a79c40 eth4:0 Link encap:Ethernet HWaddr 00:00:50:07:01:59 inet addr:128.8.33.44 Bcast:128.8.33.63 Mask:255.255.255.192 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:5 Base address:0xdd80 Memory:e0a79000-e0a79c40 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:3924 Metric:1 RX packets:120368503 errors:0 dropped:0 overruns:0 frame:0 TX packets:120368503 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 ppp0 Link encap:Point-to-Point Protocol inet addr:10.1.7.237 P-t-P:10.1.7.238 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1 RX packets:64 errors:0 dropped:0 overruns:0 frame:0 TX packets:65 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:10

12.2. The almclear command Use the almclear command to clear all outstanding server alarms. This command clears only server alarms. If Communication Manager alarms exist that are unresolved within Communication Manager, those alarms might be cleared temporarily when you run the almclear command. These alarms persist, however, and reappear until you resolve these alarms within Communication Manager. See Section 12.3, “The almdisplay command,” for more information. The following example shows the command to clear all server alarms. init@grays-icc> almclear -a All Server alarms cleared.

12.3. The almdisplay command


Use the almdisplay command to display messaging alarms, Communication Manager alarms, and server alarms. A server alarms is generated at the Linux level, and is the only type of alarm that you can clear with the almclear command. See Section 12.2, “The almclear command,” for more information.

The following example shows the results of the almdisplay command.

init@chicc> almdisplay MESSAGING ALARMS ================ No MESSAGING Alarms CommunicaMgr ALARMS =================== No CommunicaMgr Alarms SERVER ALARMS ============= No Server Alarms

12.4. The fasttop command Use the fasttop command to view the occupancy numbers for the various Communication Manager processes. The fasttop command is a customized version of the standard top command that is available within Linux. The following example shows the results of the fasttop command. Sun Jan 18 12:27:31.917 processor at 851943 khz 5.00 seconds 34.19 % used 29.12 % prio Summary 108.29 seconds 14.00 % used 10.32 % prio PID PPID %occ name pol prio memory 12822 12789 20.73 hmm FIFO 21 7036928 12827 12789 3.22 capro FIFO 17 133120000 29344 12789 2.28 map FIFO 4 3936256 12804 12789 1.75 pcd FIFO 22 7221248 12959 12789 1.51 mdm FIFO 6 3317760 12830 12789 1.49 dap FIFO 18 3756032 12828 12789 0.48 bdm FIFO 18 3665920 12808 12807 0.42 pcd FIFO 22 7221248 12988 12789 0.37 gip FIFO 20 7254016 2581 12789 0.25 map FIFO 4 3936256


3511 12789 0.22 tmr_mgr FIFO 23 3424256 3509 12789 0.16 tim FIFO 24 2895872

12.5. The modserv command Use the modserv command to turn on or turn off modem access for Avaya Services. Note that for releases prior to Communication Manager 2.0, you might have to use the sudo version of this command, and thus type sudo modserv to turn modem access on and off.

12.5.1. Modem not connected The following example shows the current modem status when the modem is not connected. craft@saturn2> modserv -v Access through the modem is administered to disabled. A modem has not been detected.

12.5.2. Modem administered but disabled The following example shows the current modem status when the modem is disabled. craft@saturn2> modserv -v Access through the modem is administered to disabled. Access through the modem is currently disabled.

12.5.3. Modem administered and ready for access The following example shows the current modem status when the modem enabled. init@saturn2> modserv -v Modem access is administered for incoming calls.

12.6. pingall Use the pingall command to ping all available interfaces in the S8X00 server view. The

pingall command is typically used on the S8500 server and the S8700 server to run a quick ping of all available interfaces. The following example shows the results of the pingall command for all network entities on the remote side. init@saturn2> pingall -a Testing.


HostName IPAddress Status saturn1-cnb 135.9.9.98 Pass saturn1-cna 135.9.10.98 Pass ipsi-A09b 135.9.9.9 Pass ipsi-A09a 135.9.10.9 Pass ipsi-A05b 135.9.9.5 Pass ipsi-A05a 135.9.10.5 Pass ipsi-A04b 135.9.9.4 Pass ipsi-A04a 135.9.10.4 Pass ipsi-A03b 135.9.9.3 Pass ipsi-A03a 135.9.10.3 Pass ipsi-A02b 135.9.9.2 Pass ipsi-A02a 135.9.10.2 Pass saturn1-dup 192.11.13.13 Pass saturn1 135.9.49.98 Pass

12.7. The restartcause command Use the restartcause command to view the time of the reset, and other important details that were captured by the “cause” of the reset. The restartcause command from the Linux CLI is similar in function to the display initcauses command that is available from the Communication Manager system access terminal (SAT). The following example shows the results of the restartcause command. init@saturn2> restartcause RESTART CAUSES Cause Action Escalated Time Internal Request 1 (WARM) no 11/14 6:02 Internal Request 2 (COLD_2) no 11/14 6:28 Initialized 4 (RELOAD) no 11/19 16:42 Internal Request 1 (WARM) no 11/19 16:43 Craft Request 4 (RELOAD) no 11/19 16:46 Internal Request 1 (WARM) no 11/19 16:47 Internal Request 4 (RELOAD) no 11/19 17:28 Internal Request 2 (COLD_2) no 11/20 22:38 Internal Request 1 (WARM) no 11/20 22:38 Initialized 4 (RELOAD) no 12/01 16:12 Internal Request 1 (WARM) no 12/01 16:13 Internal Request 4 (RELOAD) no 12/01 17:31 Software Request 4 (RELOAD) no 12/08 14:16 Internal Request 1 (WARM) no 12/08 14:17 Internal Request 1 (WARM) no 12/08 14:17 Internal Request 1 (WARM) no 01/08 14:41

Cause Description Internal Request The restart request was from an internal process as a result of an

interchange. Software Request Communication Manager initiated the restart of the server.


Cause Description Craft Request The server reset was initiated on demand by a user who logged in

from the SAT. Initialized The server went through a power cycle or was intentionally shut

down.

Restart level Description WARM Restarts all Communication Manager processes without reinitializing

data that is related to active calls. All stable calls are preserved. COLD_2 Restarts all Communication Manager processes and preserves

translation data, but reinitializes the data space. All calls are dropped. RELOAD Reloads Communication Manager, and reads in translations. All calls

are dropped.

12.8. The server command Use the server command to display the status of the S8X00 servers.

12.8.1. Avaya S8700 server The following output represents a set of ideal duplicated Avaya S8700 servers. Note that Standby Refreshed ? is “yes,” Duplication link is “up,” no Major or Minor alarms exist, and the control network is healthy. The entries in bold type are the entries of most interest for troubleshooting. init@saturn2> server SERVER STATUS Duplicated? yes Standby Busied? no Standby Refreshed? yes Standby Shadowing: on Duplication Link: up Elapsed Time since Init/Interchange: 9d 21:36:05 saturn2 saturn1 Mode: Active Mode: Standby Major Alarms: no Major Alarms: no Minor Alarms: no Minor Alarms: no Control Network: 5 / 5 Control Network: 5 / 5 Server Hardware: okay Server Hardware: okay Processes: okay Processes: okay

12.8.2. Avaya S8300 and S8500 servers The following output represents an S8300 simplex server or an S8500 simplex server. Note that with simplex servers, “Duplicated” is set to “no,” and one of the servers displays the status of “Not Ready.” The “Not Ready” status is normal and acceptable.


init@grays-icc> server SERVER STATUS Duplicated? no Standby Busied? no Standby Refreshed? no Standby Shadowing: off Duplication Link: init Elapsed Time since Init/Interchange: 4d 02:56:34 grays-icc Remote Mode: Active Mode: Not Ready Major Alarms: no Major Alarms: no Minor Alarms: no Minor Alarms: no Control Network: 0 / 0 Control Network: Server Hardware: okay Server Hardware: Processes: okay Processes:

12.9. The statapp or systat command

12.9.1. Avaya S8700 and S8500 servers The following example shows the results of the statapp command for the S8500 and S8700 media servers.

init@saturn2> statapp Watchdog 19/19 UP TraceLogger 4/ 4 UP ENV 0/ 1 OFF LicenseServer 3/ 3 UP INADSAlarmAgen 1/ 1 UP G3AlarmAgent 1/ 1 UP GMM 5/ 5 UP SNMPManager 1/ 1 UP arbiter 3/ 3 UP filesyncd 9/ 9 UP dupmgr 1/ 1 UP MasterAgent 3/ 3 UP MIB2Agent 1/ 1 UP MVSubAgent 1/ 1 UP SME 9/ 9 UP CommunicaMgr 90/90 UP

12.9.2. Avaya S8300 server (with IA770 on-board messaging installed)


The following example shows the results of the statapp command for the S8300 media server with IA770 installed. init@chicc> statapp Watchdog 19/19 UP TraceLogger 4/ 4 UP slotmon 1/ 1 UP ENV 0/ 1 OFF LicenseServer 3/ 3 UP INADSAlarmAgen 1/ 1 UP G3AlarmAgent 1/ 1 UP GMM 6/ 6 UP SNMPManager 1/ 1 UP arbiter 0/ 3 OFF filesyncd 9/ 9 UP dupmgr 0/ 1 OFF MasterAgent 3/ 3 UP MIB2Agent 1/ 1 UP MVSubAgent 1/ 1 UP SME 8/ 8 UP CommunicaMgr 67/67 UP Messaging 1/ 1 UP

12.10. The statuslicense command

12.10.1. Avaya S8700 and S8500 servers The following example shows the results of the statuslicense command. init@saturn2> statuslicense CommunicaMgr License Mode: Normal Network used for License: Carrier 02a License Serial Number is 01DR08125609 on carrier 02a Note that that the license serial number in the example is the serial number of the IPSI to which this S8700 server can talk. In a system with multiple IPSIs, this IPSI is also known as the “golden” IPSI.

12.10.2. Avaya S8300 server init@chicc> statuslicense CommunicaMgr License Mode: Normal Network used for License: Carrier MGP License Serial Number is 03J210708758 on carrier MGP


Note that that the license serial number in the example is the serial number of the gateway in which this S8300 resides. The example shows the serial number of a G700 Media Gateway.

12.11. The swversion command The following example shows the results of the swversion command. init@saturn2> swversion Operating system: Linux 2.4.20-AV9 i686 i686 Built: Nov 17 13:36 2003 Contains: 00.0.219.0 Reports as: R012x.00.0.219.0 Release String: S8700-012-00.0.219.0 UPDATES: 00.0.219.0-6227 activated The call preserving super patch for 219 Translation Saved: Jan 16 12:00 License Installed: Jan 2 13:18

The swversion command produces the following output:

• Contains: 00.0.219.0 This output is the value of the Release string in the file /opt/ws/ENV.dat. This value is the release value of the software itself, which is built in when the binaries are compiled, and loaded during integration. Integrators refer to this value as the "Definity load string."

• Reports as: R012x.00.0.219.0 This output is the value of the Version string in the file /etc/opt/ecs/ecs.conf, the so-called "registry" of the server. This value is the official generic release name for the software. Integrators refer to this value as the "Definity version string."

• Release String: S8500-012-00.0.219.0

This output is a concatenation of the following three information elements, with hyphens as separators:

− ServerType, which is another entry in the file /etc/opt/ecs/ecs.conf. The following table shows how the string that appears in that file maps into the displayed string:


ServerType Displayed string

isp2100 S8700

icc S8300

ibmx305 S8500

− Product string, which is also an entry in the file /opt/ws/ENV.dat. In this case, 012 indicates Communication Manager 2.0.

− Release string, which is the Definity load string that appears on the "Contains:" line.

The information that is presented on the three lines is redundant. However, the information does tell you whether the version of Communication Manager that is running on the server is consistent with the /etc/opt/ecs/ecs.conf file that is created or modified during configurations and upgrades. If "Contains:" and "Reports as:" show different Definity load string components, the software is configured incorrectly. In that case, you might want to examine the /opt/ws/ENV.dat and the /etc/opt/ecs/ecs.conf files, to determine what underlies the inconsistency.


13. Appendix B: Server behavior on interchange The following output is a portion of the logs from the file /var/log/ecs on the server when a server interchange takes place. The standby server is being told to go active. The prc_mgr pauses the necessary processes, and a warm start takes place on the server. STANDBY TO ACTIVE TRANSITION: Standby -> Told to go active -> WARM Restart -> Goes Active 20030113:151838676:2254:Arbiter(22656):MED:[Being told to Go Active!!!] 20030113:151838676:2255:Arbiter(22656):MED:[remot: gmm 0500, pcd 03/03, dup 70, wd 01, actv 005] 20030113:151838676:2256:Arbiter(22656):MED:[State of other side is Handoff] 20030113:151838676:2257:Arbiter(22656):MED:[IntchgMtce: NOREQ->COMEH] 20030113:151838676:2258:Arbiter(22656):MED:[STANDBY->ACTIVE :interchange request] 20030113:151838676:2259:Arbiter(22656):MED:[State Transition: STANDBY to ACTIVE ] 20030113:151838676:2260:Arbiter(22656):HIGH:[dupmem_rxoff called] 20030113:151838691:2261:Arbiter(22656):HIGH:[Interchange-Driven Warm restart] 20030113:151838691:2262:Arbiter(22656):HIGH:[Setting 0x00000800, new SOH = 0x00000b70] 20030113:151838691:2263:Arbiter(24336):MED:[Requesting a WARM restart via border process] 20030113:151838692:2264:Arbiter(22656):HIGH:[dupmem_rxoff called] 20030113:151838707:2265:border(23741):HIGH:[platreq.c: pi is 4 ] 20030113:151838707:2266:prc_mgr(23032):HIGH:[restart_req: level=0481, internal-requested] 20030113:151838707:2267:prc_mgr(23032):HIGH:[Escalation Levs: SharedMem=0, DefRestartCnt=0] 20030113:151838708:2268:prc_mgr(23032):HIGH:[prc_mgr: restart_req: internal-requested WARM: doing WARM] 20030113:151838708:2269:prc_mgr(23032):HIGH:[prc_mgr: restart_req: Upon exit, DefRestartCnt=0] 20030113:151838708:2270:prc_mgr(23032):MED:[setRstLvl: sending SIGDEFPAUSE to pid 23088 net_st, attributes=a0] 20030113:151838708:2271:prc_mgr(23032):MED:[setRstLvl: sending SIGDEFPAUSE to pid 23086 mcp, attributes=a0]

Continues… 20030113:151838710:2302:prc_mgr(23032):HIGH:[rest_def: all processes rstLevel set] 20030113:151838710:2303:tim(23734):HIGH:[sighandler: SIGDEFTERM received] 20030113:151838710:2304:tim(23734):HIGH:[now calling exit] 20030113:151838710:2305:net_mgr(23733):HIGH:[sighandler: SIGDEFTERM received] 20030113:151838710:2306:net_mgr(23733):HIGH:[now calling exit] 20030113:151838711:2307:phantom(23732):HIGH:[sighandler: SIGDEFTERM received]


20030113:151838711:2308:phantom(23732):HIGH:[now calling exit]

The output continues on as all processes receive their pauses. The following log is from the other server. This output shows what occurs when the active server is told to go to standby. ACTIVE TO STANDBY TRANSITION: Active -> Told to go Standby -> Handoff -> COLD-2 Restart -> Warm Restart -> Warm Restart -> Goes to Standby 20030113:151838015:3133:Arbiter(28117):MED:[local: gmm 0500, pcd 03/03, dup 70, wd 01, actv 005] 20030113:151838015:3134:Arbiter(28117):MED:[IntchgMtce: NOREQ->GOWAY] 20030113:151838016:3135:Arbiter(28117):MED:[Preparing for Interchange ...] 20030113:151838016:3136:filesyncd(28160):MED:[fileset.c: performing filesync for DUP] 20030113:151838237:3137:filesyncd(28160):MED:[fileset.c: performing filesync for DUP] 20030113:151838367:3138:filesyncd(28160):MED:[fileset.c: performing filesync for DUP] 20030113:151838497:3139:filesyncd(28160):MED:[fileset.c: performing filesync for DUP] 20030113:151838597:3140:filesyncd(28160):MED:[fileset.c: performing filesync for LSP] 20030113:151838674:3141:filesyncd(28160):MED:[fileset.c: this is a primary.] 20030113:151838674:3142:border(29003):HIGH:[platreq.c: pathname = ADM_UM pi is 8 ] 20030113:151838675:3143:border(29003):HIGH:[platreq.c: return 0 reached ] 20030113:151838675:3144:Arbiter(31998):MED:[fsync_request complete, result=1] 20030113:151838676:3145:Arbiter(28117):MED:[local: gmm 0500, pcd 03/03, dup 70, wd 01, actv 005] 20030113:151838676:3146:Arbiter(28117):MED:[ACTIVE ->Handoff:interchange prepartion complete] 20030113:151838676:3147:Arbiter(28117):MED:[local: gmm 0500, pcd 03/03, dup 70, wd 01, actv 005] 20030113:151838676:3148:Arbiter(28117):MED:[remot: gmm 0100, pcd 03/03, dup 370, wd 01, actv 017] 20030113:151838676:3149:Arbiter(28117):MED:[planned_interchange: rsync took 0.660478 seconds] 20030113:151838676:3150:Arbiter(28117):MED:[State Transition: ACTIVE to Handoff] 20030113:151838692:3151:Arbiter(28117):MED:[TIMESYNC: ostime=80ee8, tod=3e233b3e] 20030113:151838692:3152:Arbiter(28117):MED:[State of other side is ACTIVE ] 20030113:151838692:3153:Arbiter(28117):MED:[Handoff->STANDBY:interchange requested by CRAFT]


20030113:151838692:3154:Arbiter(28117):MED:[State Transition: Handoff to STANDBY] 20030113:151838692:3155:Arbiter(28117):HIGH:[Cold-2 restart for New Standby] 20030113:151838692:3156:Arbiter(28117):HIGH:[PDmShd = 0x2d000000] 20030113:151838692:3157:Arbiter(28117):HIGH:[Setting 0x00000800, new SOH = 0x00000870] 20030113:151838693:3158:Arbiter(32011):MED:[Requesting a COLD-2 restart via border process] 20030113:151838694:3159:border(29003):HIGH:[platreq.c: pi is 4 ] 20030113:151838694:3160:prc_mgr(28536):HIGH:[restart_req: level=0482, internal-requested] 20030113:151838694:3161:prc_mgr(28536):HIGH:[Escalation Levs: SharedMem=0, DefRestartCnt=0] 20030113:151838694:3162:Arbiter(32013):MED:[exec-ing STANDBY state script] 20030113:151838695:3163:prc_mgr(28536):HIGH:[prc_mgr: restart_req: internal-requested COLD_2: doing COLD_2] When a server goes to standby, the normal restart scenario is Cold-2, Warm, Warm. The following output represents one of the Warm requests: 20030113:151844883:3488:Arbiter(32088):MED:[Requesting a WARM restart via border process]


14. Appendix C: Connecting to an S8300, an S8500, or an S8300 server from a Windows-based personal computer

This appendix contains excerpts from the Maintenance Procedures document, 555-245-103. The Maintenance Procedures document is available on the Avaya Support Web site: http://support.avaya.com/

14.1. Configuring a remote computer that is running Windows 2000 or Windows XP for PPP modem connection

To configure a Windows-based remote computer for PPP modem connection:

1. On the desktop of your computer, right-click My Network Places, and then click Properties. The system displays the Network and Dial-up Connections window.

2. Double-click the connection name that you made earlier. Note: Depending on your system, the Connect window may appear. If the system displays the Connect window, click Properties.

3. Click the Security tab. 4. Click Advanced (custom settings). 5. Select Show terminal window. 6. Click the Networking tab. 7. In the Components box, verify that Internet Protocol (TCP/IP) and Client for

Microsoft Networks are both selected. 8. Select Internet Protocol (TCP/IP), and click Properties. 9. Click Advanced. 10. Clear Use default gateway on remote network. 11. Click OK three times to exit and save the changes.

14.2. Using Windows 2000 or Windows XP for PPP modem connection Note: Access to the system through a PPP modem connection may require RAS access and ASG Mobile access. To use Windows 2000 or Windows XP for PPP modem connection:

1. Return to the Network and Dial-up Connections window. Right-click the connection that you just created.

2. Select Connect. 3. Leave the User Name, Password, and Domain fields blank. If the Dial field is blank,

enter the appropriate telephone number. 4. Click Dial. When the media server modem answers, the system displays the After

Dial Terminal window. 5. Log on to the LAN.

a. Enter your remote access login name and password.


b. When the Start PPP Now! message appears, click Done. The system displays a small double-computer icon in the lower right portion of the screen.

6. Double-click the double-computer icon. The system displays the Dialup Status box for the connection.

7. Click the Details tab. 8. Note the Server IP address. 9. To open a telnet session to the Avaya S8X00 server, type telnet <ip address>,

where <ip address> is the IP address of the S8300 that you noted in the Dialup Status box in Step 8.

10. Access SAT or use the CLI commands as needed.

14.3. Using telnet telnet - user interface to the TELNET protocol Syntax telnet [host[port]] Description Use the telnet interface to communicate with another host by way of the TELNET protocol. If telnet is invoked without arguments, telnet enters command mode, which is indicated by the prompt, telnet>. In this mode, telnet accepts and runs the commands below. If telnet is invoked with arguments, telnet performs an open command (see below) with those arguments. To issue telnet commands when in input mode, precede the commands with the telnet escape sequence. Initially, the telnet escape sequence is the control character followed by a right bracket, ^+] . When in command mode, use the normal terminal editing conventions.

Example

The following command connects your terminal to the host with IP address 10.3.4.10: $ telnet 10.3.4.10

Options with the telnet command telnet> ? Commands may be abbreviated. Supported options are: close close current connection display display operating parameters mode try to enter line-by-line or character-at-a-time mode open connect to a site quit exit telnet send transmit special characters (send ? for more) set set operating parameters (set ? for more) unset unset operating parameters (unset ? for more)


status print status information toggle toggle operating parameters (toggle ? for more) slc change state of special characters (slc ? for more) z suspend telnet ! invoke a subshell environ change environment variables (environ ? for more) ? print help information

14.4. Using ftp

Basic commands The following table includes some commonly used commands to establish and work within

an FTP session.Command Action

open Initiate a session between the client computer and the remote server.

nlist, dir, ls List the hierarchical organization of files on the remote server. lls List the hierarchical organization of files on your local

computer. cd Change directories on the remote computer either up or down. lcd Change directories on your local computer either up or down. pwd View the current directory and path on the remote server. lpwd View the current directory and path on your local client

computer. get Copy a file from the server to your local client computer. put Copy a file from your client computer to the remote server. mget Copy multiple files from the server to your local client

computer. mput Copy multiple files from your client computer to the remote

server. prompt Set interactive prompting. "On" is a safety feature that prompts

you for verification of each step of a multiple command. "Off" allows the command to act unimpeded

ascii/binary Specify the type of file to transfer. quit End the connection and end the session. To view the full set of FTP commands, type help or ? at the FTP> prompt.

14.4.1. Example of an FTP session that uses the commands that are listed above This example shows the how to use FTP to copy files from one computer to another.

Assumptions

Computer A is your personal computer, usually the laptop computer of a technician. Computer B might be another computer, or the Communication Manager server that

can be reached from computer A over a LAN, a WAN, or the Internet.


You want to copy files Afile_1.exe, Afile_2.exe, and Afile_3.txt from computer A to computer B. Here, Afile_1.exe might represent a software update file for Communication Manager such as 00.0.219.0-6276.tar.gz.

You want to copy the file Bfile.exe from computer B, which might be a Communication Manager server, to computer A, which might be the laptop computer of a technician.

Computer B has the host name or the IP address <B_IP>. The files on computer A are located in a directory called <A_ directory>. The files on computer B are located in a directory called < B_directory>.

Command Description

Assume that you are already logged on to Computer A. C:\> cd /<path>/<A_directory> On computer A, go to the

<A_ directory > directory. C:\> ftp <B_IP> Set up an FTP session to computer B.User: <your User-ID on computer B> Log on to the FTP session on

computer B. Enter your User Id at the User prompt. If you are using Anonymous FTP, enter anonymous.

Password: <your password> Enter your password at the Password prompt. If you are using Anonymous FTP, enter your e-mail address.

ftp> cd <B_directory> Go to the <B- directory> directory on computer B. If computer B is a media server that is running Communication Manager release 2.0 or later, use cd to change to the /var/home/ftp/pub directory. The /var/home/ftp directory is not writable in 2.0 and later releases.

ftp> ascii

Specify an ASCII file type (used to transfer text files only).

ftp> put Afile_3.txt Copy Afile_3.txt from computer A to <B_ directory> on computer B.

ftp> binary Specify a binary file type. Note: To transfer binary files such as Communication Manager software updates , or “patches,” you must be in binary mode.

ftp> mput Afile*.exe Copy Afile_1.exe and Afile_2 to the <B_ directory> on computer B.

mput Afile_1.exe? y Confirm the copy of Afile_1.exe. mput Afile_2.exe? y Confirm the copy of Afile_2.exe. ftp> get Bfile.exe Copy Bfile.exe from computer B to

<A_ directory> on computer A. You


can also use the command mget Bfile.* if you want to copy multiple files from Computer B to Computer A.

ftp> bye Close the FTP session.


Appendix D: A case study in which a segmentation fault causes switch reset

This appendix includes an example of how to use the information in this document to determine the cause of a restart on an Avaya 8700 server.

15.1. Log in and connect to the system To begin, log in to the system by way of a modem connection, over the LAN, or any other means that is available. See Appendix C , “Connecting to an S8300, S8500, or S8300 server from a Windows-based personal computer,” for how to set up a connection to a Avaya server from a Windows-based personal computer. After you log in, run the restartcause command from the bash. The system displays output that looks similar to the output in Section 12.7 of this document. Match the time stamp of the customer outage with the output on the screen to determine exactly what time the outage occurred.

15.2. Examine the logs Change directories to /var/log/ecs. Run the ls –ltr command from the Linux shell or the CLI to see a list of all the log files in that directory. The files are named with the time stamp of when the logs began. Use this information to determine which log contains information on the reset.

15.3. Analyze the problem Use vi or emacs to open the file. See Section 7.1, “The vi editor,” for how to use vi. Use the grep command to search for the keywords, as described in Section 8.4, “The grep command.” In this example, the restart is the result of a software segmentation fault. Thus, the EIP search string takes you to the location in the log file where the segmentation fault occurred. You now know that the switch reset was the result of a software problem, what process in the software had the problem, and exactly what time the outage occurred. Based on this information, the next step is to escalate this ticket to the next Tier for resolution. However, before you send the ticket to Tier 3 or Tier 4, you must determine if a core file was generated. Perform one of the following actions to determine if a core file was generated:

• Type corevector to see if the flag is turned on for traps.

• Change directories to /var/crash, as described in Section 10, “Core dumps and mini core dumps.” Run the ls –ltr command in this directory to see when the latest core files were generated. Determine if a core file exists with the same time stamp as when the outage occurred.

Next, copy both the log file in question and the previous log file to the directory /var/home/ftp. (For Communication Manager releases 2.0 and later, the directory is /var/home/ftp/pub.) Finally, if a core file exists, copy the core file to /var/home/ftp. Include this information, along with the messages log and the wdlog, with the ticket, and send the ticket to Tier 3 or Tier 4.


© 2004 Avaya Inc. All rights reserved. All trademarks identified by the ® or TM are registered trademarks or trademarks, respectively, of Avaya Inc. All other trademarks are the property of their respective owners.

Application Note: Troubleshooting Guide v1.0 for the Avaya ... · Application Note: Troubleshooting...

Documents

Transcript of Application Note: Troubleshooting Guide v1.0 for the Avaya ... · Application Note: Troubleshooting...