Implementing System Logging

Implementing System Logging

This module describes the tasks you need to implement logging services on the router.

The Cisco IOS XR Software provides basic logging services. Logging services provide a means to gatherlogging information for monitoring and troubleshooting, to select the type of logging information captured,and to specify the destinations of captured system logging (syslog) messages.

Feature History for Implementing System Logging

ModificationRelease

Platform Automated Monitoring (PAM) tool was introduced for all Cisco IOS XR 64-bitplatforms.

Release6.1.2

• Implementing System Logging , on page 1

Implementing System LoggingSystem Logging (Syslog) is the standard application used for sending system log messages. Log messagesindicates the health of the device and point to any encountered problems or simplify notification messagesaccording to the severity level. The IOS XR router sends its syslog messages to a syslog process. By default,syslog messages will be sent to the console terminal. But, syslog messages can be send to destinations otherthan the console such as the logging buffer, syslog servers, and terminal lines.

Syslog Message Format

By default, the general format of syslog messages generated by the syslog process on the Cisco IOS XRsoftware is as follows:node-id : timestamp : process-name [pid] : % message category -group -severity -message-code : message-text

The following table describes the general format of syslog messages on Cisco IOS XR software.

Table 1: Format of Syslog Messages

DescriptionField

Node from which the syslog message originated.node-id

Implementing System Logging1

DescriptionField

Time stamp in the month day HH:MM:SS format,indicating when the message was generated.

The time-stamp format can be modifiedusing the service timestamps command.

Note

timestamp

Process that generated the syslog message.process-name

Process ID (pid) of the process that generated thesyslog message.

size

Message category, group name, severity, and messagecode associated with the syslog message.

[ pid ]

Text string describing the syslog message.message-text

Syslog Message Severity Levels

In the case of logging destinations such as console terminal, syslog servers and terminal lines, you can limitthe number of messages sent to a logging destination by specifying the severity level of syslog messages.However, for the logging buffer destination, syslog messages of all severity will be sent to it irrespective ofthe specified severity level. In this case, the severity level only limits the syslog messages displayed in theoutput of the command show logging, at or below specified value. The following table lists the severity levelkeywords that can be supplied for the severity argument and the corresponding UNIX syslog definitions inorder from the most severe level to the least severe level.

Table 2: Syslog Message Severity Levels

DescriptionLevelSeverity Keyword

System unusable0emergencies

Immediate action needed1alert

Critical conditions2critical

Error conditions3errors

Warning conditions4warnings

Normal but significantcondition

5notifications

Informational messages only6informational

Debugging messages7debugging

Prerequisites for Configuring System LoggingThese prerequisites are required to configure the logging of systemmessages in your network operating center(NOC):


Implementing System LoggingPrerequisites for Configuring System Logging

• You must be in a user group associated with a task group that includes the proper task IDs. The commandreference guides include the task IDs required for each command. If you suspect user group assignmentis preventing you from using a command, contact your AAA administrator for assistance.

• You must have connectivity with syslog servers to configure syslog server hosts as the recipients forsyslog messages.

Configuring System LoggingPerform the tasks in this section for configuring system logging as required.

Configuring Logging to the Logging BufferSyslog messages can be sent to multiple destinations including an internal circular buffer known as loggingbuffer. You can send syslog messages to the logging buffer using the logging buffered command.

Configuration Example

This example shows the configuration for sending syslog messages to the logging buffer. The size of thelogging buffer is configured as 3000000 bytes. The default value for the size of the logging buffer is 2097152bytes.RP/0/RP0/CPU0:Router# configureRP/0/RP0/CPU0:Router(config)# logging buffered 3000000RP/0/RP0/CPU0:Router(config)# commit

Configuring Logging to a Remote ServerSyslog messages can be sent to destinations other than the console, such as logging buffer, syslog servers,snmp servers and terminal lines. You can send syslog messages to an external syslog server by specifying theip address or hostname of the syslog server using the logging command. Also you can configure the syslogfacility in which syslog messages are send by using the logging facility command.

The following table list the features supported by Cisco IOS XR Software to help managing syslog messagessent to syslog servers.

Table 3: Features for Managing Syslog Messages

DescriptionFeatures

Facility is the identifier used by UNIX to describe theapplication or process that submitted the log message.You can configure the syslog facility in which syslogmessages are sent by using the logging facilitycommand.

UNIX system log facility


Implementing System LoggingConfiguring System Logging

DescriptionFeatures

Cisco IOS XR Software supports hostname prefixlogging. When enabled, hostname prefix loggingappends a hostname prefix to syslog messages beingsent from the router to syslog servers. You can usehostname prefixes to sort the messages being sent toa given syslog server from different networkingdevices. Use the logging hostname command toappend a hostname prefix to syslog messages sent tosyslog servers

Hostname prefix logging

By default, a syslog message sent to a syslog servercontains the IP address of the interface it uses to leavethe router. Use the logging source-interfacecommand to set all syslog messages to contain thesame IP address, regardless of which interface thesyslog message uses to exit the router.

Syslog source address logging

Configuration Example for Logging to Syslog Server

This example shows the configuration for sending syslog messages to an external syslog server. The ip address209.165.201.1 is configured as the syslog server.Router# configureRouter(config)# logging 209.165.201.1 vrf defaultRouter(config)# logging facility kern (optional)Router(config)# logging hostnameprefix 203.0.113.1 (optional)Router(config)# logging source-interface HundredGigE 0/0/1/0 (optional)Router(config)# commit

Amongst other options, all and none are provided under the logging severity command form. If you enableall or none, all or no severity logs are logged to the syslog server, respectively. This configuration persistseven when you enable a specific operator type.

Configuration Example for Logging to SNMP Server

This example shows the configuration for sending syslog messages to an SNMP server. The logging trapcommand is used to limit the logging of messages sent to the snmp servers based on severity.Router# configureRouter(config)# snmp-server traps syslogRouter(config)# logging trap warningsRouter(config)# commit

For more information on SNMP server configurations, see the Configuring Simple Network ManagementProtocol chapter in the System Management Configuration Guide for Cisco NCS 540 Series Routers

Related Topics

• Configuring Logging to the Logging Buffer, on page 3

• Configuring Logging to Terminal Lines, on page 5


Implementing System LoggingConfiguring Logging to a Remote Server

Configuring Logging to Terminal LinesBy default syslog messages will be sent to the console terminal. But, syslog messages can also be send toterminal lines other than the console. You can send syslog messages to the logging buffer using the loggingmonitor command.


This example shows the configuration for sending syslog messages to terminal lines other than console. Inthis example, severity level is configured as critical. The terminal monitor command is configured to displaysyslog messages during a terminal session. The default severity level is debugging.RP/0/RP0/CPU0:Router# configureRP/0/RP0/CPU0:Router(config)# logging monitor criticalRP/0/RP0/CPU0:Router(config)# commitRP/0/RP0/CPU0:Router# terminal monitor

Modifying Logging to Console TerminalBy default syslog messages will be sent to the console terminal. You canmodify the logging of syslogmessagesto the console terminal


This example shows how to modify the logging of syslog messages to the console terminal.RP/0/RP0/CPU0:Router# configureRP/0/RP0/CPU0:Router(config)# logging console alertsRP/0/RP0/CPU0:Router(config)# commit

Modifying Time Stamp FormatBy default, time stamps are enabled for syslogmessages. Time stamp is generated in themonth day HH:MM:SSformat indicating when the message was generated.


This example shows how to modify the time-stamp for syslog and debugging messages.RP/0/RP0/CPU0:Router# configureRP/0/RP0/CPU0:Router(config)# service timestamps log datetime localtime msec or servicetimestamps log uptimeRP/0/RP0/CPU0:Router(config)# service timestamps debug datetime msec show-timezone or servicetimestamps debug uptimeRP/0/RP0/CPU0:Router(config)# commit

Suppressing Duplicate Syslog MessagesSuppressing duplicate messages, especially in a large network, can reduce message clutter and simplify thetask of interpreting the log. The duplicate message suppression feature substantially reduces the number ofduplicate event messages in both the logging history and the syslog file.


This example shows how to suppress the consecutive logging of duplicate syslog messages.


Implementing System LoggingConfiguring Logging to Terminal Lines

RP/0/RP0/CPU0:Router# configureRP/0/RP0/CPU0:Router(config)# logging suppress duplicatesRP/0/RP0/CPU0:Router(config)# commit

Archiving System Logging Messages to a Local Storage DeviceSyslog messages can also be saved to an archive on a local storage device, such as the hard disk or a flashdisk. Messages can be saved based on severity level, and you can specify attributes such as the size of thearchive, how often messages are added (daily or weekly), and how many total weeks of messages the archivewill hold. You can create a logging archive and specify how the logging messages will be collected and storedby using the logging archive command.

The following table lists the commands used to specify the archive attributes once you are in the loggingarchive submode.

Table 4: Commands Used to Set Syslog Archive Attributes

DescriptionFeatures

Specifies the maximum number of weeks that thearchive logs are maintained in the archive. Any logsolder than this number are automatically removedfrom the archive.

archive-length weeks

Specifies the maximum total size of the syslogarchives on a storage device. If the size is exceededthen the oldest file in the archive is deleted to makespace for new logs.

archive-size size

Specifies the local storage device where syslogs arearchived. By default, the logs are created under thedirectory device/var/log. If the device is notconfigured, then all other logging archiveconfigurations are rejected. We recommend thatsyslogs be archived to the harddisk because it hasmore capacity than flash disks.

device {disk0 | disk1 | harddisk}

Specifies the maximum file size (in megabytes) thata single log file in the archive can grow to. Once thislimit is reached, a new file is automatically createdwith an increasing serial number.

file-size size

Specifies if logs are collected on a daily or weeklybasis.

frequency {daily | weekly}

Specifies the minimum severity of log messages toarchive. All syslog messages greater than or equal tothis configured level are archived while those lesserthan this are filtered out.

severity severity


This example shows how to save syslog messages to an archive on a local storage device.


Implementing System LoggingArchiving System Logging Messages to a Local Storage Device

RP/0/RP0/CPU0:Router# configureRP/0/RP0/CPU0:Router(config)# logging archiveRP/0/RP0/CPU0:Router(config-logging-arch)# device disk1RP/0/RP0/CPU0:Router(config-logging-arch)# frequency weeklyRP/0/RP0/CPU0:Router(config-logging-arch)# severity warningsRP/0/RP0/CPU0:Router(config-logging-arch)# archive-length 6RP/0/RP0/CPU0:Router(config-logging-arch)# archive-size 50RP/0/RP0/CPU0:Router(config-logging-arch)# file-size 10RP/0/RP0/CPU0:Router(config)# commit

Platform Automated MonitoringPlatform Automated Monitoring (PAM) is a system monitoring tool integrated into Cisco IOS XR softwareimage to monitor issues such as process crash, memory leak, CPU hog, tracebacks, syslog and disk usage.PAM is enabled by default on all Cisco IOS XR 64 bit platforms. When the PAM tool detects any of thesesystem issues, it collects the required data to troubleshoot the issue, and generates a syslog message statingthe issue. The auto-collected troubleshooting information is then stored as a separate file inharddisk:/cisco_support/ or in /misc/disk1/cisco_support/ directory.

PAM EventsWhen PAM detects a process crash, traceback, potential memory leak, CPU hog or a full file system, itautomatically collects logs and saves these logs (along with the core file in applicable cases) as a .tgz file inharddisk:/cisco_support/ or in /misc/disk1/cisco_support/ directory. PAM also generates a syslog messagewith severity level as warning, mentioning the respective issue.

The format of the .tgz file is: PAM-<platform>-<PAM event>-<node-name>-<PAMprocess>-<YYYYMMDD>-<checksum>.tgz.For example,PAM--crash-xr_0_RP0_CPU0-ipv4_rib-2016Aug16-210405.tgz is the file collected when PAM detects aprocess crash.

Because PAM assumes that core files are saved to the default archive folder (harddisk:/ or /misc/disk1/), youmust not modify the location of core archive (by configuring exception filepath) or remove the core filesgenerated after PAM detects an event. Else, PAM does not detect the process crash. Also, once reported, thePAM does not report the same issue for the same process in the same node again.

For the list of commands used while collecting logs, refer Files Collected by PAM Tool, on page 11.

The sections below describe the main PAM events:

Crash Monitoring

The PAMmonitors process crash for all nodes, in real time. This is a sample syslog generated when the PAMdetects a process crash:

RP/0/RP0/CPU0:Aug 16 21:04:06.442 : logger[69324]: %OS-SYSLOG-4-LOG_WARNING : PAM detectedcrash for ipv4_rib on 0_RP0_CPU0.All necessary files for debug have been collected and saved at0/RP0/CPU0 : harddisk:/cisco_support/PAM--crash-xr_0_RP0_CPU0-ipv4_rib-2016Aug16-210405.tgz

Please copy tgz file out of the router and send to Cisco support. This tgz file will beremoved after 14 days.)


Implementing System LoggingPlatform Automated Monitoring

Traceback Monitoring

The PAM monitors tracebacks for all nodes, in real time. This is a sample syslog generated when the PAMdetects a traceback:

RP/0/RP0/CPU0:Aug 16 21:42:42.320 : logger[66139]: %OS-SYSLOG-4-LOG_WARNING : PAM detectedtraceback for ipv4_rib on 0_RP0_CPU0.All necessary files for debug have been collected and saved at0/RP0/CPU0 :harddisk:/cisco_support/PAM--traceback-xr_0_RP0_CPU0-ipv4_rib-2016Aug16-214242.tgzPlease copy tgz file out of the router and send to Cisco support. This tgz file will beremoved after 14 days.)

Memory Usage Monitoring

The PAM monitors the process memory usage for all nodes. The PAM detects potential memory leaks bymonitoring the memory usage trend and by applying a proprietary algorithm to the collected data. By default,it collects top output on all nodes periodically at an interval of 30 minutes.

This is a sample syslog generated when the PAM detects a potential memory leak:

RP/0/RP0/CPU0:Aug 17 05:13:32.684 : logger[67772]: %OS-SYSLOG-4-LOG_WARNING : PAM detectedsignificant memory increase(from 13.00MB at 2016/Aug/16/20:42:41 to 28.00MB at 2016/Aug/17/04:12:55) forpam_memory_leaker on 0_RP0_CPU0.All necessary files for debug have been collected and saved at0/RP0/CPU0 :harddisk:/cisco_support/PAM--memory_leak-xr_0_RP0_CPU0-pam_memory_leaker-2016Aug17-051332.tgz

(Please copy tgz file out of the router and send to Cisco support. This tgz file will beremoved after 14 days.)

CPU Monitoring

The PAM monitors CPU usage on all nodes periodically at an interval of 30 minutes. The PAM reports aCPU hog in either of these scenarios:

• When a process constantly consumes high CPU (that is, more than the threshold of 90 percentage)

• When high CPU usage lasts for more than 60 minutes

This is a sample syslog generated when the PAM detects a CPU hog:

RP/0/RP0/CPU0:Aug 16 00:56:00.819 : logger[68245]: %OS-SYSLOG-4-LOG_WARNING : PAM detectedCPU hog for cpu_hogger on 0_RP0_CPU0.All necessary files for debug have been collected and saved at 0/RP0/CPU0 :harddisk:/cisco_support/PAM--cpu_hog-xr_0_RP0_CPU0-cpu_hogger-2016Aug16-005600.tgz(Please copy tgz file out of the router and send to Cisco support. This tgz file will beremoved after 14 days.)RP/0/RP0/CPU0:Jun 21 15:33:54.517 : logger[69042]: %OS-SYSLOG-1-LOG_ALERT : PAM detectedifmgr is hogging CPU on 0_RP0_CPU0!

File System Monitoring

The PAM monitors disk usage on all nodes periodically at an interval of 30 minutes. This is a sample sysloggenerated when the PAM detects that a file system is full:


Implementing System LoggingPAM Events

RP/0/RP0/CPU0:Jun 20 13:59:04.986 : logger[66125]: %OS-SYSLOG-4-LOG_WARNING : PAM detected/misc/config is full on 0_1_CPU0(please clean up to avoid any fault caused by this). All necessary files for debug havebeen collected and saved at0/RP0/CPU0 : harddisk:/cisco_support/PAM--disk_usage-xr_0_1_CPU0-2016Jun20-135904.tgz(Please copy tgz file out of the router and send to Cisco support. This tgz file will beremoved after 14 days.)

Disable and Re-enable PAMThe PAM tool consists of three monitoring processes—monitor_cpu.pl, monitor_crash.pl, andmonitor_show_show_logging.pl.

Before disabling or re-enabling the PAM, use these options to check if the PAM is installed in the router:

• From Cisco IOS XR Command Line Interface:

Router# show processes pam_manager location allTue Jun 14 17:58:42.791 UTCnode: node0_RP0_CPU0

Job Id: 317PID: 14070

Executable path: /opt/cisco/XR/packages/iosxr-infra.rp-6.1.1.17I/bin/pam_manager

Instance #: 1Version ID: 00.00.0000

Respawn: ONRespawn count: 4Last started: Mon Jun 13 23:08:43 2016Process state: RunPackage state: Normal

core: MAINMEMMax. core: 0

Level: 999Placement: None

startup_path:/opt/cisco/XR/packages/iosxr-infra.rp-6.1.1.17I/startup/pam_manager.startup

Ready: 0.166sProcess cpu time: 0.200 user, 0.310 kernel, 0.510 total

JID TID Stack pri state NAME rt_pri317 14070 0K 20 Sleeping pam_manager 0317 14071 0K 20 Sleeping lwm_debug_threa 0317 14076 0K 20 Sleeping pam_manager 0317 14077 0K 20 Sleeping lwm_service_thr 0317 14078 0K 20 Sleeping qsm_service_thr 0317 14080 0K 20 Sleeping pam_manager 0

• From router shell prompt:

Router# run ps auxw|egrep perlTue Jun 14 18:00:25.514 UTCroot 14324 0.0 0.2 84676 34556 ? S Jun13 0:40 /usr/bin/perl/pkg/opt/cisco/pam//monitor_cpu.plroot 14414 0.0 0.1 65404 14620 ? S Jun13 0:00 /usr/bin/perl/pkg/opt/cisco/pam//monitor_crash.pl


Implementing System LoggingDisable and Re-enable PAM

Disable PAM

To shutdown PAM agents, execute these commands from the XR EXEC mode:

For local RP:Router# process shutdown pam_manager

For all RPs:Router# process shutdown pam_manager location all

The Cisco NCS 540 Series Routers are fixed routers with only 1 RP. The process shutdown pam_managerlocation all command is not supported on the Cisco NCS 540 Series Routers.

Note

Re-enable PAM

Because pam_manager is not a mandatory process, it does not restart automatically if it was manually disabled(unless in the case of a system reload).To restart PAM agents, execute the following commands from XREXEC mode:

For local RP:Router# process start pam_manager

For all RPs:Router# process start pam_manager location all

The Cisco NCS 540 Series Routers are fixed routers with only 1 RP. The process start pam_managerlocation all command is not supported on the Cisco NCS 540 Series Routers.

Note

To start PAM on all locations, the pam_manager process should be restarted on all nodes by using the locationall option in the process start pam_manager command.

Note

Data Archiving in PAMAt any given point of time, PAM does not occupy more than 200 MB of harddisk: space. If more than 200MB is needed, then PAM archives old files and rotates the logs automatically.

The PAM collects CPU or memory usage (using top -b -n1 command) periodically at an interval of 30minutes.The files are saved under harddisk:/cisco_support/ directory with the filename as <node name>.log (forexample, harddisk:/cisco_support/xr-0_RP0_CPU0.log). When the file size exceeds the limit of 15MB, thefile is archived (compressed) into .tgz file, and then rotated for a maximum of two counts (that is, it retainsonly two .tgz files). The maximum rotation count of .tgz files is three. Also, the old file (ASCII data) isarchived and rotated if a node is reloaded. For example, xr-0_RP0_CPU0.log is archived if RP0 is reloaded.

You must not manually delete the core file generated by the PAM. The core file is named as <processname>_pid.by_user.<yyyymmdd>-<hhmmss>.<node>.<checksum>.core.gz.


Implementing System LoggingData Archiving in PAM

Files Collected by PAM ToolThe table below lists the various PAM events and the respective commands and files collected by the PAMfor each event.

You can attach the respective.tgz file when you raise a service request (SR) with Cisco Technical Support.

Commands and Files Collected by PAMEvent Name

• show install active

• show platform

• show version

• core (gz) file

• core.txt file

Process crash

• show dll pid


• show logging

• show platform

• show version

Process traceback


• show platform

• show version

• core (gz) file

• dumpcore running

• continuous memory usage snapshots

Memory leak


• show logging

• show platform

• show version

• core (gz) file

• core.txt file

Show logging event


Implementing System LoggingFiles Collected by PAM Tool

Commands and Files Collected by PAMEvent Name

• follow process

• pstack

• show dll pid


• show platform

• show version

• top -H

• core (gz) file

• CPU usage snapshots

CPU hog


• show platform

• show version

• console log

• core (gz) file

• Disk usage snapshots

Disk usage


Implementing System LoggingFiles Collected by PAM Tool

Implementing System Logging

Documents

Transcript of Implementing System Logging