57843993 Nx Troubleshooting Guide

CONFIGURATION................... 5

COMMANDS............................ 6

LOG FILES ............................ 12

DATABASE ........................... 16

PROCESSES......................... 18

DATA COLLECTION............. 18

TOOLS................................... 21

ISSUES.................................. 27

APPENDIX............................. 40

Document Version: 7.3 Date: 25-JUN-07

NetXplorer and NetEnforcer x7.x.x

System and Troubleshooting Guide

NetXplorer NX7.x.x and NetEnforcer S/E7.x.x

This document describes the system and troubleshooting techniques for the following products:

NetXplorer Software Version NX7.x.x

NetEnforcer Software Version S7.x.x

NetEnforcer Software Version E7.x.x

Customer Support Only Confidentiality Notice This document contains Proprietary Trade Secrets of Allot Communications LTD and its receipt or possession does not convey any right to reproduce, disclose its contents or to manufacture, use or sell anything that it may describe. Reproduction, disclosure or use without specific authorization from Allot Communications is forbidden. Allot reserves the right to make changes, add, remove or change the schedule of any element of the plan.

NetXplorer and NetEnforcer Troubleshooting Guide

Table of Contents

CONFIGURATION ..................................................................................................................................... 5 PORTS.......................................................................................................................................................... 5

NetXplorer Client and Server ................................................................................................................. 5 NetXplorer Server to NetEnforcer .......................................................................................................... 5 Additional ............................................................................................................................................... 5

ACCESSING SYBASE .................................................................................................................................... 5 Problems Uninstalling Sybase................................................................................................................ 6

COMMANDS................................................................................................................................................ 6 NETENFORCER ............................................................................................................................................ 6 OTHER NETENFORCER TOOLS..................................................................................................................... 7 ACSTAT ....................................................................................................................................................... 7 NICSTAT ...................................................................................................................................................... 8 ACTHRUPUT................................................................................................................................................. 8 ACMODE ...................................................................................................................................................... 9 ACMON ........................................................................................................................................................ 9 HWADMIN................................................................................................................................................. 10 LINKADMIN .............................................................................................................................................. 10 GO CONFIG NIC .......................................................................................................................................... 11

LOG FILES................................................................................................................................................. 12 NETXPLORER SERVER............................................................................................................................... 12

C:\Allot\bin........................................................................................................................................... 12 C:\Allot\log........................................................................................................................................... 12 C:\Allot\conf ......................................................................................................................................... 13 C:\Allot\netxplorer\jboss-3.2.6\server\allot\log ................................................................................... 13 C:\Allot\netxplorer\jboss-3.2.6\server\allot\deploy.............................................................................. 14 C:\Allot\netxplorer\jboss-3.2.6\server\allot\conf.................................................................................. 14

NETXPLORER CLIENT ............................................................................................................................... 14 C:\Documents and Settings\<user name>............................................................................................ 14

NETENFORCER .......................................................................................................................................... 14 $SWGL.................................................................................................................................................. 14 /tmp/...................................................................................................................................................... 15 /var/log/apache..................................................................................................................................... 15 $SWGC ................................................................................................................................................. 16

DATABASE ................................................................................................................................................ 16 NETENFORCER .......................................................................................................................................... 16

$SWGD................................................................................................................................................. 16 $SWGD/data......................................................................................................................................... 17

NETXPLORER ............................................................................................................................................ 17 C:\Allot\data\db.................................................................................................................................... 17 Performing a Backup............................................................................................................................ 17

PROCESSES............................................................................................................................................... 18 NETENFORCER .......................................................................................................................................... 18 NETXPLORER ............................................................................................................................................ 18

DATA COLLECTION............................................................................................................................... 18 NETENFORCER .......................................................................................................................................... 18

$SWGE/httpd/htdocs/bucket ................................................................................................................. 18

www.allot.com 2


$SWGE/httpd/htdocs/bucket/30 (same content for 300) ....................................................................... 18 Understanding the Manifest ................................................................................................................. 19

NETXPLORER ............................................................................................................................................ 19 C:\Allot\data\bucket\stc\<device ID> .................................................................................................. 19 C:\Allot\data\bucket\ltc_export\ ........................................................................................................... 20 C:\Allot\data\bucket\ltc_export\<device ID>....................................................................................... 20 Allot/data/bucket/ltc/device_ID ............................................................................................................ 20

TOOLS ........................................................................................................................................................ 21 Upgrading NX Server Version.............................................................................................................. 21 Enabling Compression ......................................................................................................................... 21

CHANGE ADMIN PASSWORD ..................................................................................................................... 22 MANAGING REPORTING DATABASES ........................................................................................................ 22

Recreating Default (ST and LT) Databases.......................................................................................... 22 Improving Database Performance ....................................................................................................... 22

CHANGING REPORTING DATABASE PROFILES ........................................................................................... 23 Changing LT Reduction Profile............................................................................................................ 23 Changing ST Profile Options ............................................................................................................... 23

CHANGING REPORTING DATABASE PARAMETERS..................................................................................... 24 Disabling External Hosts Reporting..................................................................................................... 24

INCREASING THE NUMBER OF BUCKETS SENT PER TIME SLICE ................................................................... 24 Changing number of buckets in the NetEnforcer..................................... Error! Bookmark not defined. Changing number of buckets in the NetXplorer ................................................................................... 25

ENABLING TAP MODE .............................................................................................................................. 25 PORT MIRROR ........................................................................................................................................... 26 STEP 1 ...................................................................................................................................................... 26 STEP 2 ...................................................................................................................................................... 26

ISSUES ........................................................................................................................................................ 27 NTP/TIME ISSUES...................................................................................................................................... 27

Synchronization issues between Client and Server............................................................................... 27 Synchronization issues between Server and NetEnforcer..................................................................... 27 Problem: GUI does not start ................................................................................................................ 29

CREATING A SNAPSHOT............................................................................................................................. 29 NetXplorer ............................................................................................................................................ 29 NetEnforcer .......................................................................................................................................... 29

TAKING A SNAPSHOT ................................................................................................................................ 29 The Manual Snapshot ........................................................................................................................... 29 The Automatic Snapshot ....................................................................................................................... 30 Sending the Snapshot............................................................................................................................ 30

HTTP SNAPSHOT ...................................................................................................................................... 30 ADD DEVICE ............................................................................................................................................. 32 CHANGE IP................................................................................................................................................ 34

Defined Behavior.................................................................................................................................. 34 Current Behavior.................................................................................................................................. 35 In-Band/Out of Band Definitions.......................................................................................................... 35

PROVISIONING CHANGES .......................................................................................................................... 36 Add Host ............................................................................................................................................... 36

CONFIGURATION CHANGES....................................................................................................................... 36 Process ................................................................................................................................................. 36 Troubleshooting.................................................................................................................................... 36

DATABASES NOT SYNCHRONIZED............................................................................................................. 37 Symptoms.............................................................................................................................................. 37 Explanation .......................................................................................................................................... 37 Troubleshooting.................................................................................................................................... 37 To Generate a Full Export.................................................................................................................... 37

www.allot.com 3


RMA/BOX REPLACEMENT ........................................................................................................................ 38 COLLECTION PROBLEMS ........................................................................................................................... 38

STC Problems Related to Software....................................................................................................... 38 Data Collection Stops Due to NTP Issues ............................................................................................ 39

DEMO INSTALLATION ISSUES .................................................................................................................... 39 Installing NetEnforcer version 7.1.0 on a NetEnforcer AC-202/302.................................................... 39 Skipping installation hardware requirements....................................................................................... 39

APPENDIX ................................................................................................................................................. 40 APPENDIX I ............................................................................................................................................... 40

Host output from $SWGL/nedbg.DataSrv.log ...................................................................................... 40 APPENDIX II .............................................................................................................................................. 42

Host output from $SWGL/nedbg.AllSnmpAgent.log............................................................................. 42

www.allot.com 4


Configuration

Ports

NetXplorer Client and Server

Port Number Description

TCP:80 HTTP for initial access to Server. Once applet is downloaded, this is not required

TCP:1099 RMI (Java J2EE protocol)

TCP:4444 RMI (Java J2EE protocol)

TCP:1098 JNP (Java J2EE protocol)

NetXplorer Server to NetEnforcer


TCP:80 Data sampling

UDP:161 SNMP Configuration updates

UDP:161 SNMP Traps (Events)

UDP:123 NTP

TCP:123 NTP

Additional


TCP:50000 For troubleshooting access to configuration database on NetXplorer Server

TCP:50001 For troubleshooting access to short term database on NetXplorer Server

TCP:50002 For troubleshooting access to long term database on NetXplorer Server

Accessing Sybase Database access on the Server may be required in order to troubleshoot certain issues, regarding configuration, data accuracy, data collection (and many more). To access the database, Sybase Central must be installed on the local PC. This can be downloaded from ftp://support:[email protected]/Sybase.

To access the database open Sybase Central and perform the following: 1. Right click on ASA9 2. Select new connection

www.allot.com 5


3. Enter user details under the Identification tab a. ID nms b. Password allot

4. Enter database (location and database) under the Database tab a. Localhost:db_port - if database resides on local server b. IP:db_port – if database resides on different server (need to ensure access to specific

server, i.e. firewall issues etc.) It is possible to open all databases simultaneously.

Problems Uninstalling Sybase At times, the uninstall procedure does not completely uninstall the Sybase application. Deleting the Allot directory and registry entries still does not complete the uninstall process. If this is the case, go to the Environmental Variables and delete the reference to the Allot folder. This will complete the uninstall process. The environmental variables can be accessed as follows: Right click on My Computer and select Properties. Click on the Advanced Tab and then click on the Environment Variables button. Under System Variables at the bottom are various entries which will show the Allot folder as the value. For additional information on uninstalling Sybase, please see KB item #6976.

Commands

NetEnforcer • acstat

• acthruput

• clientTest clientTest is an application used to get statistical data on the box (client) sent to the server. Usage: clientTest -s <statistic type 5-lines;6-pipes;0-vcs;1-conv;7-ne;> -t <time interval> (30/300 seconds) -v <specific vcs id separated with space (Max 10)> -p <specific pipes id separated with space (Max 10)> -l <specific lines id separated with space (Max 10)> Example (for VC statistics every 30 seconds): clienttest –s 0 –t 30 Output (for VC statistics): [Output can be found in the nedbg.clienttest.log file] 03-14 15:27:02(201) <DL_USER1>: StatisticClient::handleNewSample, dataLen:126, sampleObject:Collection id:270195024 StartTime:1142342814 EndTime:1142342822 Number of slices:24 Number of rows:1 Schema: SM_LINE_ID(1),SM_PIPE_ID(2),SM_PIPE_INST(3),SM_VC_ID(4),SM_VC_INST(5),SM_DIVIDED_BYTES_IN(19),SM_DIVIDED_BYTES_OUT(20),SM_PACKETS_IN(15),SM_PACKETS_OUT(16),SM_LIVE_CONNECTIONS(13),SM_NEW_CONNECTIONS(12),SM_DROPPED_CONN(14),

• swgadmin –l Output: lcd 175 DataSrv 176 SessionDispatcher 9286 coll 180 StatisticMgr 181 AllSnmpAgent 182

• go config view (see CLI document for full list of CLI commands)

www.allot.com 6


Other NetEnforcer Tools

acstat acstat shows information about the current connections running through the NetEnforcer. Usage: acstat [ -l {session/vc/pipe/h} ] [ -t / -u /-a / -n / -c / -r / -i ] [ -s ] [ -f ] [ -F ] [-x ] [ -m <max_sessions> ] [ -N ] [ -B ] [ -R <file> ] [ -I <pipe_id>,<vc_id>] [ -A <src_ip_addr>,<dst_ip_addr>] -l <LIST_TYPE> : List session/vc/pipe/hierarchy [session] -t : display TCP connections -u : display UDP connections -a : display any IP connections (other than TCP and UDP) -n : display non IP connections -c : display ICMP connections -r : display ARP connections -i : display all connections -s : display connection allocation summary (single option, default) -f : display extended view -F : display extended view - advanced -x : display internal/external (instead of client/server) -m <NUMBER> : display up to NUMBER of sessions (max 500k) -N : don't resolve names -B : dump binary data to file -R <FILE> : read binary data from FILE (single option) -I <PIPE>,<VC> : display hierarchy all connections of pipe and vc (zero means all) -A <SRC IP ADDR>,<DST IP ADDR> : display connections of specific src ip address and dst ip address (zero means all)

acstat with no flags shows connection allocation summary

Sessions are represented in the following format: Protocol Client Server State VC Client IF TTL VLAN Tag Tos St

Protocol Name of the protocol. If the name is unknown, the hexadecimal number of the protocol is shown. Raw TCP is shown as TCP-r.

Client IP of the host which initiated the session (for TCP and UDP sessions - also the port).

Server IP of the host to which the client send its request (for TCP and UDP sessions - also the port).

State Prisma Session State. Can be one of the following: OPENED, CONNECTED, WIRED, TO BE CLOSED, CLOSED, REJECTED, DROP or NONALLOCATED (the last one should never appear; if it does, there is probably a bug). If the client-server and the server-client sides of a session are in different states, both states are shown, e.g. WI-2b for WIRED - TO BE CLOSED.

VC (Virtual Channel)

VC to which the session belongs. If the client-server and the server-client sides of a session belong to different VCs, both VCs are shown.

Client IF NetEnforcer interface that the client is connected to. TTL (Time to live)

Time left (in seconds) until the session expires if no traffic arrives.

VLAN Tag Indicates if the connection is VLAN tagged and to which VLAN. ToS ToS marked value. If the number displayed is 0, then there are no ToS

markings on the packets. St (Session Status)

Possible options are Raw, Half, Dbl, Chng, Loop or NA. Raw indicates if the session is raw i.e. the connection was classified after it had been established. Dbl indicates a double session.

www.allot.com 7


nicstat Displays the mode and speed of network interfaces. It is not the speed and duplex defined in the GUI “configuration”, but the actual values. The command is used for troubleshooting access links related problems and for verification that the links are compatible with the adjacent router or switch. Certain networking related problems are coming from NICs definitions that are mis-configured. Checking the nicstat and comparing it to the router/switch definition is a useful tool in troubleshooting problems like packet loss, synchronization and network slowness issues. Command Output

nicstat +-----------+------+-------+--------+ | Interface | Link | Speed | Duplex | +-----------+------+-------+--------+ | eth0 | up | 10 | half | | eth1 | down | n/a | n/a | +-----------+------+-------+--------+ | eth2 | up | 10 | half |

acthruput Prints the amount of bits that have passed through each Interface, active pipe and active VC during one time slice (one second). The output of the command shows the bandwidth consumption of each of the active pipes/vcs and for the entire interface. It can be used also to determine if there’s a need to alter the bandwidth definition of the pipe/vc and to troubleshoot bandwidth and traffic related problems. Usage: acthruput [ -b ] [ -B ] [ -c ] [ -v ] [ -d DIR ] -b : display throughput in bits (default) -B : display throughput in bytes -c : display throughput per connection -t : display total link throughput including IgnoreQoS -d DIR : analyze data in DIR instead of / e.g. acthruput -d $W/stat/last - to analyze the last snapshot

Command Output

acthruput --------------------------------------------------------- Entity Name Bits/sec --------------------------------------------------------- INTERFACE Internal 0 --------------------------------------------------------- INTERFACE External 2896 PIPE 1 1024 VC 8 512 VC 1 512

Note: The actrhuput command should only be used for AC-x0x devices. For AC-1000 devices, please use the acmon command (see next page).

www.allot.com 8


acmode Switches between various NetEnforcer software modes. Shows, saves and restores modes and makes the NetEnforcer enter/exit software or hardware bypass. Examples: enable/disable QoS, TCP, UDP, etc. acmode [ +/-endvcs ] [ +/-srcmac ] [ +/-ignoremom ] [ +/-verbose ] [ +/-mtu ] [ +/-noweight ] [ +/-novc ] [ +/-wnyfast ] [ save ] [ restore ] [ default ] [ show ] [ hwbp ] +endvcs - enable ended vcs -endvcs - disable ended vcs +srcmac - enable source mac handling -srcmac - disable source mac handling +ignoremom - enable ignore monitoring only mode on dkm -ignoremom - disable ignore monitoring only mode on dkm +verbose - enable dkm verbose -verbose - disable dkm verbose +mtu - enable Check and Fragment IP packet according to MTU size -mtu - disable Check and Fragment IP packet according to MTU size +noweight - enable counting traffic with Ignore QoS Policy for monitoring/accounting purposes -noweight - disable counting traffic with Ignore QoS Policy for monitoring/accounting purposes +novc - enable counting traffic that passes through NE prior to policy assignment -novc - disable counting traffic that passes through NE prior to policy assignment +wnyfast - enable winny fast identify method -wnyfast - disable winny fast identify method (default) save - save current settings restore - restore saved settings default - restore default settings show - show current settings hwbp - go into hardware bypass Note: you can run acmode with a number of arguments, e.g. acmode +qos -tcp. The arguments are processed one by one in the order of appearance, with two exceptions: - hwbp (go into hardware bypass) is processed last.

acmon Used to get statistics (ONLY for AC-1000 units). Usage: acmon { -p <pipe id> / -v <vc id> / -s <service id> / -d / -r / -l <count>} [ -t <seconds> ] -p <PIPE> : monitor specific pipe rate -v <VC> : monitor specific vc rate -s <SERVICE> : monitor specific service rate -d : monitor dmu packet distribution -l : run acmon limited count number -r : monitor octet rx -t <SECONDS> : time to wait between samples in seconds [1 seconds] Example: [i ] 10:10:02 >> 0 conn ps [0] rate inbound: 0.000 bps outbound: 0.000 bps [1] rate inbound: 202.772 Kbps outbound: 0.000 bps

www.allot.com 9


HwAdmin Controls the bypass mechanism. This command can be used to send the box to hardware bypass.

Usage: HwAdmin -s : displays system status

-H : displays hardware (AC, MACH, FULL, OEM) version information.

Command Output

HwAdmin –s

Status register = 0x3 Local machine is STAND_ALONE and in ACTIVE mode Local bypass is CONNECTED Remote machine not detected

HwAdmin -H Hardware version - 402 Firmware version - 2 OEM version – 0

LinkAdmin Changing the NIC configuration on the AC-X02 and AC-1000 series: LinkAdmin will give you various options: LinkAdmin -[dsuc] <interface name> -c <interface number> [autoneg on|off] [speed 10|100|1000] [duplex half|full] -d - link down -u - link up -s - show link status -f - show supported link speed and duplex optional interface name eth1 eth0 nic1 nic0 etc. If we want to set the internal interface to full 100, you can use either of the commands: LinkAdmin 0 autoneg off speed 100 duplex full LinkAdmin -c 0 autoneg off speed 100 duplex full LinkAdmin -c eth0 autoneg off speed 100 duplex full The command needs to be followed by a reboot. Please note that these commands are for the AC-X02 and AC-1000 only.

www.allot.com 10


go config nic The NIC settings on the NetEnforcer AC-404, AC-804, and AC-808 can be configured using the go config nic CLI command. AC:~# go config nic Command: go config nic Usage: go config nic {<Label:Mode:Speed[:FailureAction]>,...} Acceptable Labels are: INTERNAL1, EXTERNAL1, MGMNT, INTERNAL2, and EXTERNAL2 Acceptable values of Mode are: half, full, and auto Acceptable values of Speed are: 10, 100, 1000, and auto (according to box type) Acceptable values of Failure Action are: none, fail_pair, fail_all, and bypass Example: go config nic INTERNAL1:full:100:fail_pair Important Note: The AC-404 does not support 1000Mbps speed, although it is possible to run the go config nic command with 1000Mbps as a speed value. Labels: For the AC-808, the acceptable labels are: INTERNAL1, EXTERNAL1, INTERNAL2, EXTERNAL2, MGMNT, INTERNAL3, EXTERNAL3, INTERNAL4, and EXTERNAL4 Speed: Acceptable value of Speed: 1000 - the interfaces are capable of working with 1 Gbps physically (be connected to 1Gbps interfaces). All of the AC-808 interfaces support 1000 Gbps physical speed. Values: Acceptable values of Failure Action:

fail_pair: if one interface within a pair (INTERNAL x - EXTERNAL x) is down, the system will disable its peer. fail_all: if one interface is down, the system will disable all other interfaces. bypass: : if one interface is down, the system will move to bypass.

Management port As of version 7.1.0 build 24, only the management port can be configured via the admin menu. The AC-80x (the new AC-802 platform, AC-804, and AC-808) management port supports 10/100/1000 (physical speed).

www.allot.com 11


Log Files

NetXplorer Server All logs are stored under Allot\. This is usually located under C:\.

C:\Allot\bin All batch and executable files are located here, including all processes (e.g. poller, keeper). File Name Explanation

Create_snapshot_logs.bat Snapshot generator

Start_<db name>.bat Batch file initializing specified database

Stop_<db_name>.bat Batch file stopping specified database

reduction_profile_upd.bat Batch file that copies selected reduction cfg file from \allot\conf\Reduction to \allot\conf

check_<db name>_db.bat Checks if specific database (CFG, STC, LTC) alive mechanism used check_db.bat file

check_db.bat Check database alive mechanism

conf_assist.exe Prepare database password for \allot\conf stc_collect.cfg and \allot\conf ltc_collect.cfg files (Not in use for users)

db_install.exe Used for Sybase install ,database create and recreate

C:\Allot\log

File Name Explanation

poller.log Poller log

converter.log Converter log

loader.log Loader log

ltc_poller.log Long Term Poller (lt_poller) log

ltc_loader.log Long Term Loader (lt_loader) log

keeper.log Keeper Server log file

allot_<db name>.txt Database work process log file

allot_<db name>_stop.txt Database stop process log file

www.allot.com 12


C:\Allot\conf


nedbg.conf Configuration file for keeperServer.exe and LTreducer.exe

reduction.cfg Configuration file for reduction process used by LTreducer.exe

stc_collect.cfg Configuration file for stc collector processes (poller, converter, loader, manifest_manager)

ltc_collect.cfg Configuration file for ltc collector processes (ltc_poller, ltc_loader)

hosts.cfg Hosts list used by LTreducer.exe

Reduction directory Optional reduction configurations

MIB directory MIB files for MIB modules supported by the agent

XML directory XML schemas for interfacing with the agent

db directory Data files for static loading of certain tables

swkeeper.ini file Process and database initialization file including log level configuration (similar to swgrun.ini on the NetEnforcer)

static.ini file Database parameters and ports

C:\Allot\netxplorer\jboss-3.2.6\server\allot\log


NMS.log Application Server log. Example messages: [EAR Deployment] Init J2EE application:…. Implication: application loading Subsequent messages: loading of each module [NamingService] Started jndi bootstat…1099… Implication: connecting to server Note: this port must be open otherwise system will not load [RARMetaData] Loading Jboss Resource Adapter… Implication: loading connection to database (will appear after above message) Subsequent messages: loading of each module, look out for [Deploy] messages. Stacked traces indicate problems

NMS.log.n Older versions of nms.log (can be up to 40 before original one is overwritten)

boot.log Jboss log

jsr77.log Jboss log

server.log Jboss log including some application server exceptions

www.allot.com 13


C:\Allot\netxplorer\jboss-3.2.6\server\allot\deploy


NMS.ear This is the NetXplorer software application. A software upgrade can theoretically be performed by replacing this file.

sybase-ds.xml Contains configuration (allot_cfg) database and password

C:\Allot\netxplorer\jboss-3.2.6\server\allot\conf


log4j.xml Contains configuration parameters for NMS.log including debug level and number of instances of log file. o maxfilesize - log size o maxbackupindex - max number of logs

NetXplorer Client

C:\Documents and Settings\<user name>


NMS.log Application client log. The contents of this file are not the same as NMS.log located on the Server.

NetEnforcer

$SWGL


ac_reboot.log Log of ac_reboot command

badCCBs Not in use.

bt Directory that contains all backtrace files.

coll_dump Various counters from collector process that can be printed upon user request.

counters.swg nedbg.keeper.log takes information from this file.

dbchanges.swg Policy changes accepted by DKM.

dkmdump Various counters from DKM process that can be printed upon user request.

errorlog.swg DKM log

hwu.HwAdmin.log HwAdmin utility log

hwu.lcd.log LCD log

kpc.SessionDispatch.log Log created by every process that uses the KPC library (IPC between user and kernel)

log.SWG Obsolete - not used.

www.allot.com 14


nedbg.acstat.log Log of acstat process

nedbg.AllSnmpAgent.log Log of SNMP agent/process (communication between Server and NetEnforcer)

nedbg.AllSnmpAgent.log.old Old SNMP log

nedbg.Collector.log Log of Collector process

nedbg.DataSrv.log

Log of DataSrv process. Issues with applying database changes and changes applied logged. In debug mode, this shows complete database update including XML command received from server, changed performed, counter ID updated and ok sent to Server.

nedbg.default.log Obsolete – not used.

nedbg.go.log CLI log

nedbg.keeper.log Log of Keeper (hardware keeper)process

nedbg.lcd.log Log of lcd process

nedbg.StatisticMgr.log Log of StatisticMgr (Statistics Manager) process. Problems with buckets will be logged.

nedbg.swKeeper.log Log of swKeeper (software keeper) process

nedbg.swKeeper.log.old Old log of nedbg.swKeeper.log

ne-instl.<date>.log Log of last installation process

notice.SWG DoS attack reported by DKM

ntp.log Log of ntp process. Can identify problems with NTP synchronization.

StatisticMgr_dump Various counters from Stat Mgr process that can be printed upon user request.

/tmp/


nedbg.ProvisionCli.log check whether content was received from the Apache Server View full XML content

/var/log/apache


access_log check whether Apache received change Look for POST to ProvisionCli.exe

www.allot.com 15


$SWGC

File Name Type Explanation

reduction.conf File Short Term reduction configuration parameters

SNMP Directory

actype File NetEnforcer version and type

addnsParameters File DNS refreshment parameters

dataCli.conf File Internal config file

dkm.conf File dkm and prisma configuration parameters

hosts.conf File List of hosts referred to during the reduction of statistic data.

keeper.ini File HWKeeper ini file managing initialization parameters of all modules controlled by the HW Keeper

lcd_version File Displays lcd version

memwatch.conf File Memory consumption levels indicated memory issues

nedbg.conf File Debug level of all nedbg log files

provisioncli.conf File Internal config file.

reduction.conf Link to file

Link to selected reduction configuration file

Reduction.* File All optional reduction configuration files

statisticmgr_boot_counter File Counter of restarts of statistic manager process.

swKeeper.ini File SWKeeper ini file managing initialization parameters of all processes controlled by Keeper

Database

NetEnforcer

$SWGD

Name Type Explanation

backup directory Location of most recent successful policy update (schema and data directories and their content)

data directory Location of policy and configuration database

schema directory Location of policy and configuration database schema

lastSnmpUpdate file Maintains timestamp of last policy update received by SNMP. Used to report on synchronization status of device against the server.

www.allot.com 16


$SWGD/data

Name Explanation

allotConfig.xml Database of NetEnforcer configuration parameters. Including: device capabilities (modes), registration parameters, device limits (e.g. Lines, VCs, Pipes, bandwidth), data collection and reduction parameters. Network parameters are not included in this file.

allotProvision.xml Policy and Catalog database. This is one file including all of the Catalog definitions and the Policy configuration.

lastPolicyFullExport Maintains timestamp of the last full policy export to the device. Used to report on synchronization status of device against the server.

lastPolicyUpdate Maintains timestamp of last policy update distributed by data server to internal clients. Used to report on synchronization status of device against the server.

NetXplorer

C:\Allot\data\db


cfg directory Location of configuration database, allot_cfg.db

ltc directory Location of long term data database, allot_ltc.db

stc directory Location of short term data database, allot_stc.db

Performing a Backup

Please note that there are two kinds of database backups for the NX server.

Cold backup – done when services can be stopped. Hot backup – done when services are running.

Cold backup 1. Stop NetXplorer Service by going to Windows Services and stopping NetXplorer Server. 2. The following lines should appear in the allot_ltc.txt and allot_stc.txt files:

“Disable all events” “End of current events”

3. Backup the database by copying the following folder: c:\Allot\data\db to a different location, preferably a different disk.

4. Start the NetXplorer Service. Hot backup In order to perform a hot backup, please see KB item 6269: "NetXplorer Backup and Restore Database". Please note that this should only be given to customers in exceptional cases.

www.allot.com 17


Processes

NetEnforcer There are several processes that should always be running on the NetEnforcer. These processes can be identified using several different commands, as follows: • swgadmin -l

lcd DataSrv SessionDispatcher coll StatisticMgr AllSnmpAgent

• ps –awx|grep ntp or ntpq –p (or use ps-ax) ntp client

• HTTP

NetXplorer There are several processes that should be running on the NetXplorer Server. These processes can be identified using several different tools:

• Windows Services (Start>Control Panel>Administrative Tools>Services) o NetXplorer Server

• Windows Task Manager (CTRL+ALT+DEL and select Task Manager) o Poller.exe o Converter.exe o Loader.exe o ltc_poller.exe o ltc_Loader.exe o ltreducer (runs periodically – therefore may not be seen) o manifest_manager.exe (runs periodically – therefore may not be

seen) o KeeperService.exe o Dbsrv9.exe (3 instances) o ntpd.exe

Data Collection

NetEnforcer

$SWGE/httpd/htdocs/bucket


30 directory Location of 30 seconds buckets data

300 directory Location of 300 (5 minutes) second buckets data

$SWGE/httpd/htdocs/bucket/30 (same content for 300)


conv_stat directory Location of conversation buckets (binary format)

www.allot.com 18


vc_stat directory Location of rules buckets (binary format)

line_burst directory Not in use

pipe_burst directory Not in use

vc_burst directory Not in use

manifest Link Link to current manifest

manifest<n> file The manifest file containing a list of buckets that need to be collected by the Poller on the NetXplorer

Understanding the Manifest The manifest can be accessed through the web, by browsing to: http://<IP of NetXplorer>/bucket/<bucket type (30 or 300)>/manifest Example: http://192.123.234.56/bucket/30/manifest Format

Boot number, bucket index, bucket type (0=vc_stat, 1=conv_stat), statistic type, start time, end time, bucket duration, actual bucket duration, compression (0=no, 1-yes). Bucket duration is not always exactly 30/300 seconds. There may be a fluctuation of 1 or 2 seconds either way (for example, 299 or 301 seconds).

NetXplorer

C:\Allot\data\bucket\stc\<device ID>


conv_stat directory Contains conversations buckets in binary and then ascii format before import to short term database

vc_stat directory Contains rules buckets in binary and then ascii format before import to short term database




www.allot.com 19


C:\Allot\data\bucket\ltc_export\


<Device ID> directory Multiple folders representing each device managed by the NetXplorer Server

manifest file Manifest file containing list of buckets that need to be imported into the long term database

C:\Allot\data\bucket\ltc_export\<device ID>


conv_stat directory Contains conversations buckets in ascii format exported from the short term database

vc_stat directory Contains rules buckets in ascii format exported from the short term database




Allot/data/bucket/ltc/device_ID


conv_stat directory Contains conversations buckets in ascii format before import to long term database

vc_stat directory Contains rules buckets in ascii format before import to long term database




For details about the data collection procedure, refer to the SE training presentation.

www.allot.com 20


Tools Upgrading NX Server Version • Stop NetXplorer Service by going to Windows Services and stopping NetXplorer Server. • Open the Windows Task Manager by pressing <CTRL + ALT + DEL> and clicking the Task

Manager button. Select the Processes tab and confirm that DbSrv9.exe does not appear in the list.

• Download the software version desired from the Allot ftp site by completing the following steps: 1. Log into the ftp site with your personal support login account (download\username) and

password. Access will only be allowed if a valid license for NetXplorer has been purchased.

2. Type cd NetXplorer/NetXplorer_Server/Current_Versions/NetXplorer_NX7xx.zip 3. Please note that the NetXplorer files are approximately 460MB and will take some time to

download. They are compressed and must be opened with WinZip or another utility. For complete instructions and full installation procedures, see the NetXplorer Quick Install Guide and NetXplorer Operation Guide from http://www.allot.com. • There is no need to remove a previous installation. It will be detected automatically by the

Installation Wizard. • The NetXplorer Service will be stopped automatically when the upgrades starts. It will resume

operation after the server is rebooted following the upgrade. • At the end of the upgrade procedure you will be asked to reboot the NetXplorer Server. Please note that if the NetXplorer Server will be down for more than 25 minutes, Real Time (Short Term) data after this period will be lost and data collection will be continued only after the server is up again. Therefore it is recommended to perform the upgrade during low traffic hours.

Enabling Compression Toggling bucket compression on/off By default, compression is turned off (i.e. regular buckets). To toggle bucket compression: 1. Edit $SWGD/data/allotConfig.xml 2. The parameter data_collection/bucket_type should be set to 1 for compression or 0 for no compression. 3. Reboot the NetEnforcer. Note: Compression is not recommended as a default configuration, but only in situations where it is absolutely necessary. Enabling compression places additional heavy load on the NetEnforcer.

www.allot.com 21


Change Admin Password If the admin password has been lost, it is possible to replace it with the original password allot. In the SYSTEM_USERS table of the allot_cfg database, replace the admin password with: 53xXk0LYvZI=

Managing Reporting Databases

Recreating Default (ST and LT) Databases It is possible to recreate empty (default) collector databases (STC and LTC). Data for the Device table will be loaded from Application Server (CFG database) as soon as the NetXplorer Server service is initialized after running the procedure. This utility replaces the current database files with clean databases (according to the configuration files c:\Allot\conf\static.ini and c:\Allot\conf\dynamic.ini created during installation process).

Procedure

1. Stop the NetXplorer Server 2. Open MSDOS command window (Start>Run> type cmd). 3. c:\Allot\bin\recreate_default_db.bat <STC| LTC>.

a. STC – recreate STC database; b. LTC – recreate LTC database.

The following message appears in the command window - Recreate database <STC|LTC> successful or failed.

4. If the process has been successful restart the NetXplorer Server service. Note: There is more chance that the ST DB will get stuck, as it is in use approximately every 10 seconds, while the LT DB is only updated every hour. For problems with LT DB, please contact Escalation for additional assistance.

Improving Database Performance To ensure better performance for complex NetEnforcer deployments managed by a single NetXplorer server, the following post-install changes for STC and LTC databases may be considered:

• Change temporary file location • Change transaction log location • Change dbspaces location (rename DBspace) • Allocate additional disk space for DBspaces

Deployment: 4 (four) files located in the directory \allot\bin:

• run_post_install_stc.bat; post_install_stc.vbs; - for STC database; • run_post_install_ltc.bat; post_install_ltc.vbs – for LTC database;

Usage: First - NetXplorer Server service should be stopped. Before running, the VBscript files - post_install_stc.vbs, post_install_stc.vbs should be manually edited. Carefully read all remarks,

www.allot.com 22


comment unnecessary commands, set real paths for database files and necessary sizes for dbspaces. Recommendations for all post-install steps are available in the mentioned VBscript files. In case dbspaces file locations (paths) are changed, it is necessary to change (manually edit) the dbspaces locations in \allot\conf\dynamic.ini file. Open a command window (cmd.exe). From the command-line, run: \allot\bin\ run_post_install_stc.bat or run_post_install_ltc.bat. The following message will appear after the command has completed successfully: See post installation log in -\allot\tmp\install\post_install_stc.log

Changing Reporting Database Profiles

Changing LT Reduction Profile Change the reduction.cfg file for the LTreducer application. The installation copies enterprise normal profile file into directory \allot\conf. The mentioned profile then becomes active (file name is reduction.cfg). All reduction profile files are located in the \allot\conf\Reduction directory. This utility will copy the active reduction profile file in \allot\conf from the \allot\conf\Reduction directory. The possible reduction profile types are: ent_normal; ent_accuracy; ent_history; isp_normal; isp_accuracy; and isp_history. Please note that ent = enterprise and isp = Internet Service Provider.

Usage: Open command window (cmd.exe). From the command-line, run: \allot\bin\ reduction_profile_upd.bat <profile type>. Profile types are: ent_normal; ent_accuracy; ent_history; isp_normal; isp_accuracy; and isp_history. Example: \allot\bin\ reduction_profile_upd.bat isp_accuracy For more information on profiles, see the Excel chart on profiles in the knowledge base (http://support.allot.com) (item #6423), the SE Internal Training (item #6059), and item #6836.

Changing ST Profile Options Purpose: Change data aging parameters in STC database PARAM table for second, minute and hour statistical data.

Server Usage: Open command window (cmd.exe). From the command-line, run: \allot\bin\ stc_profile_upd.bat <profile type>. Profile types are: ent_normal; ent_accuracy; ent_history; isp_normal; isp_accuracy; and isp_history. NetXplorer Server service (or STC database) should be restarted. Example: \allot\bin\ stc_profile_upd.bat isp_accuracy

NetEnforcer Change collection profile: go config data_collect <environment:profile> Acceptable values of Reduction Environment are: ent and isp Acceptable values of Reduction Profile are: normal, accuracy and history

www.allot.com 23

http://support.allot.com/


Changing Reporting Database Parameters

Disabling External Hosts Reporting To disable external host collection, use the following CLI command: go config data_collect -no_ext_host enable The NetEnforcer will reboot after 5 seconds. Please note that by default, the AC-1000 does not include external hosts as part of the collection key and the AC-400/AC-800 does.

Increasing the number of buckets sent per time slice Changing number of buckets in the NetEnforcer Note: This should only be used in situation where the need for increasing the buckets is critical. The default number of buckets sent is 5. There is an option to increase this number, to a maximum of 48 buckets (on x0x devices). In the AC-10x0/AC-25x0 devices there is no HDD and it is not recommended to increase this number at all. Increasing the number of buckets should be followed by enabling compression on the device (see page 21 on how to enable compression). This is done as follows: 1. CD to $SWGD/data 2. Vi to allotConfig.xml 3. Modify the line marked in bold below from 5 to the new number: <data_collection> <sample_interval>30</sample_interval> <bucket_type>1</bucket_type> <max_emb_rec>0</max_emb_rec> <max_st_bkts>5</max_st_bkts> <max_lt_bkts>5</max_lt_bkts> <bkt_mgmt_enable>0</bkt_mgmt_enable> <service_statistics>4. Restart the StatisticMgr module in order to include the modification. Note: Increasing this parameter would increase the number of buckets for 30 second as well as 300 second. 48 buckets is equal to 4 hours of 5 minute resolution, and 24 minutes of 30 second resolution.

www.allot.com 24


Changing number of buckets in the NetXplorer Every bucket has a time stamp. When the server receives a bucket, it checks the timestamp. If the timestamp is older than UTC time minus delta, it discards the buckets. In order to increase this delta, it is necessary to do the following: 1. Enter Sybase Central.

2. Enter the STC database.

3. Go to the PARAM(nms) table in the Table folder.

4. Choose the Data tab.

5. Go to line 66 The max time for a 30 seconds bucket time to be before of the current UTC.

6. Change the INT_VAL value from 180 to a value larger than 30sec x selected number of

buckets.

7. Do the same on line 67 The max time for a 300 seconds bucket time to be before of the current UTC.

8. Change the INT_VAL value from 1800 to a value larger than 300sec x selected number of buckets.

Enabling TAP Mode To enable TAP mode, right-click on a NetEnforcer and select configuration. On the Networking tab, check TAP Mode and save. TAP mode will now be enabled. Note: TAP Mode is not supported on the NetEnforcer AC-1040.

www.allot.com 25


Port Mirror Many customers do not wish to install a NetEnforcer inline between the LAN switch and the WAN router, even in monitoring-only mode, since they need to disconnect the line when installing the NetEnforcer. Therefore they wish to install the NetEnforcer on the switch mirror port, or span port, instead and monitor the traffic in that way. The switch mirror port mirrors the traffic received and transmitted on the port to the WAN router. The NetEnforcer is used as a simple monitoring probe and the Internal or External port is connected to the switch mirror port. Therefore only one port is connected. The NetEnforcer can still monitor traffic in this case, however there are two modifications needed for the NetEnforcer to operate properly.

Procedure

Step 1 Bridge learning must be disabled in order to prevent the NetEnforcer from learning and maintaining a bridge forwarding table for the port connected to the switch mirror port. 1. Connect to the NetEnforcer console via the Console port or a Telnet/SSH session. Login as

user ‘root’ with password ‘bagabu’ (unless changed). 2. Open the file /usr/local/SWG/bin/init_modules for editing using the vi editor by entering the

following command: vi /usr/local/SWG/bin/init_modules

3. Change the line prisma_args="stree=${STREE_MODE} to prisma_args="nolearn=1 stree=${STREE_MODE}

4. Save the changes by entering the following command :wq

5. Reboot the NetEnforcer for the change to take effect.

Step 2 When the NetEnforcer has rebooted and has become active again, the handling of “double sessions” must be changed as follows: 1. Connect to the NetEnforcer console again via the serial port or a Telnet/SSH session. Login

as user ‘root’ with password ‘bagabu’ (unless changed). 2. Type the following command:

acmode +dbs 3. Type the following command

acmode –qos 4. The QoS software will restart automatically, no need to reboot.

Conclusion Traffic between the LAN switch and the WAN router may now be monitored from the switch mirror port. All the different monitoring graphs should work with the exception of the ‘Connections’ graphs. NetAccountant and the Long Term Monitoring may also be used.

www.allot.com 26


Issues

NTP/Time issues

Synchronization issues between Client and Server The NetXplorer Client and NetXplorer Server have a tolerance of 10 minutes time difference. The devices may be on different time zones. For example, if the Server is set to 10:03, and the device is set to 10:05, then this is acceptable. The same goes if the time zone difference is +2:00 (12:05). Note: Daylight savings time may cause an issue with the time zones.

Symptoms If the clocks are out of sync, the graphs/logs times are inconsistent.

Troubleshooting After login to the client, there is always a log of the time (UTC time dump). Check c:\Documents and Settings\<User name>\NMS.log to view this time dump.

Synchronization issues between Server and NetEnforcer

Symptoms If the clocks become out of sync, then there can be many issues including data collection. When statistics are gathered by the NetEnforcer, a bucket is created with a timestamp based on the NetEnforcer clock. Periodically, these buckets are collected by the poller process (on the NetXplorer). The NetXplorer compares the time of the bucket with its internal clock. If the NetEnforcer and NetXplorer Server have a time difference larger than 180 seconds for 30 second buckets and 1800 seconds for 300 second buckets, it will discard the bucket. When the user tries to generate real time monitoring graphs, no real-time data will be displayed, and an error message will appear in red displaying: “No data for the time selected”. The following alarm/event is received if data collection is stopped (can be found in the poller.log file located in C:\Allot\log): 'invalid bucket time on device NetEnforcer404' (id 208 - Current bucket time is older that current UTC minus delta)

Cause How does the synchronization functionality work? The ntpdate command is initiated once at startup. It connects to the NTP server(s) and sets system time according to the time value received from the first server that responds. The ntpd process is initiated once the time is set by ntpdate. It is the daemon that keeps the unit time properly synchronized. If ntpdate fails to synchronize, ntpd will not be started. ntpd does not update the time at regular intervals. The update intervals are based on certain calculations to determine when synchronization is required. Typically, this is once every 30 to 60 minutes.

www.allot.com 27


ntpdate may not initiate at startup for the following reasons: • The NetXplorer Server is rebooted at the same time the NetEnforcer is booting up. • The NetEnforcer does not manage to synchronize with the NTP Server because:

o The server is down. o There are communication issues.

Troubleshooting It is important to check the NetXplorer server first, then continue to the NetEnforcer if the problem has not been solved. NTP/NetXplorer Server

• Verify that the NTP service is running. By default, this runs on the NetXplorer server. If this is the case, run the following command: C:\Allot\ntp-server\ntpq -p

ntpq:read:Connection refused This error indicates that the NTP service is not running on the NX Server. To initiate the NTP service on the NetXplorer server, do the following: 1. Go to Services in Administrative Tools on the PC, and start the Network Time

Protocol Service. 2. To verify that the service is running, run Task Manager and search for the

process ntpd.exe. If this process is found, run the ntpq -p command, as described above.

3. Reboot the NetEnforcer to see if the synchronization will take place after reboot.

NetEnforcer

• Ensure that the NTP service is running on the NetXplorer server before continuing. • Verify that the NTP process is running on the NetEnforcer:

ps –awx|grep ntp 89 ? SL 0:00 /usr/sbin/ntpd -l /usr/local/SWG/logs/ntp.log The above line shows that the NTP process is running. If the process is not found, initiate the NTP Daemon by rebooting the NetEnforcer.

• Verify that synchronization is against the NTP server IP (NX or ext. NTP server): AC-202:~# ntpq -p remote refid st t when poll reach delay offset jitter ============================================================================== *10.4.70.1 LOCAL(1) 11 u 4 64 377 0.624 -2.455 0.291 LOCAL(0) LOCAL(0) 14 l 59 64 377 0.000 0.000 0.008

Status 16 indicates failure to sync against NTP server. Verify that synchronization is against the NTP server, and not the internal

(local) clock of the NetEnforcer. This is marked by an asterisk (*) at the beginning of the line with the NTP server.

• Verify that the Windows firewall is not enabled on the server (this is enabled by

default) which could block the NTP requests.

www.allot.com 28


For more information, the NTP manuals may be found at http://ntp.isc.org/bin/view/Main/DocumentationIndex. A document describing NTP and NTP on the NetEnforcer in general (for version 5.x) can be found at KB item 4723.

Problem: GUI does not start To solve this issue, go to control panel on the machine that cannot access the NetXplorer and choose Java. 1. On the General tab, under Temporary Internet Files, click on delete and then OK. 2. Open browser with NX server IP address (http://NXServer-IP) and launch the application. Note: If this does not solve the problem, run javaws.exe from the Java 1.5 environment. This may typically be located at a location similar to: C:\Program Files\Java\jre1.5.0_06\bin. Delete anything shown on this screen (this will clear the cache).

Creating a Snapshot

NetXplorer o \allot\bin contains a batch file called create_snapshot_logs.bat. This file takes all the relevant

logs and prepares a snapshot file that can be sent via e-mail. Please note that this file can be large at times (approx. 9MB).

o The snapshot will be created under \allot\tmp\snapshot_<date>.tar.gz

NetEnforcer The snapshot procedure is the same as in previous NetEnforcer versions. To generate a snapshot run snapshot.

Taking a Snapshot The Snapshot File is a file used to help Allot Customer Support in the troubleshooting process. The file itself is a zip file that contains files which provide Allot Customer Support with a precise picture of what was happening inside the NetEnforcer when a particular event occurred. These files include log files, policy definitions, system settings, etc. The Snapshot is an essential support tool that is vital in solving any support issues. There are two ways of taking the Snapshot: Manually and Automatically.

The Manual Snapshot The Snapshot can be run manually. If an Allot Customer Support Engineer requests you take a Snapshot of the box, it is best to run the Snapshot process manually. To run the Snapshot manually, simply login into the NetEnforcer as root, and from the command prompt, run the command snapshot. This will create a Snapshot file in the directory, /usr/local/SWG/snapshots/. The Snapshot file is created with the name snapshot.date_time.tgz.

Core Snapshot While taking a regular snapshot, core files (all files under /usr/local/SWG/logs/core) will also be included in it. In some cases the core files might be very big. In cases where the size of the

www.allot.com 29

http://ntp.isc.org/bin/view/Main/DocumentationIndex

http://nxserver-ip/


snapshot is more than 15M, the NetEnforcer will create an additional snapshot with core files only. Example:

core.snapshot.07.05.02_09.27.00.tgz

The Automatic Snapshot There may be some specific cases where Customer Support requests that you run an Automatic Snapshot. This process configures the Snapshot to run automatically every four hours. The snapshot files are deposited in the /usr/local/SWG/snapshots/ directory. To start the Automatic Snapshot, type snap_on_cron To stop the Automatic Snapshot, type snap_off_cron

Prisma Snapshot An automated snapshot that is generated after DKM or Collector restarts. The Prisma Snapshot is a “short” version of the regular snapshot and contains only /proc/prisma directory and /usr/local/SWG/logs directory.

Sending the Snapshot Note: This script does exist in the box, but there is a bug. Do not use this script for now. Normally, this takes a snapshot but currently cannot send it.

Current Snapshot A utility is included on the NetEnforcer for sending the Snapshot files directly to Allot Customer Support. The utility is called send_snapshot and the syntax is send_snapshot. This utility will automatically take a snapshot of the unit’s current state and log into the Allot Customer Support FTP Server. It will then open a directory (named with box number of the NetEnforcer) and send the snapshot. The file/s are copied into the opened directory.

Saved Snapshot A snapshot which has been taken previously and saved may be sent using the syntax Send_snapshot_file(s). For example, if you have a saved Snapshot file, snapshot.01.03.00_09.54.39.tgz and you would like to send it to the Customer Support for analysis; you would type the following command from the command prompt:

send_snapshot snapshot.01.03.00_09.54.39.tgz This will contact the Allot Customer Support FTP server, log in, create a numerical directory and copy in the snapshot file selected.

HTTP Snapshot Some NetEnforcer and NetXplorer units do not have access to FTP. Therefore, it is not possible to send a snapshot directly from the box. If the unit does not have a public address or Internet access, use this workaround:

1. Create the snapshot by typing: snapshot The snapshot file is saved to the following directory: /usr/local/SWG/snapshots/

2. Copy the snapshot file to the /usr/local/SWG/etc/httpd/htdocs directory: cp /usr/local/SWG/snapshots/snapshot.15.03.06_16.08.33.tgz

www.allot.com 30


/usr/local/SWG/etc/httpd/htdocs (in this example, the file is named snapshot.15.03.06_16.08.33.tgz).

3. Point the browser to the NetEnforcer URL: http://<NetEnforcer IP>/snapshot name For example: http://192.1.1.2/snapshot.15.03.06_16.08.33.tgz.

4. This will start an HTTP download of the snapshot file to the PC. It is now possible to email this snapshot, or place it on an FTP server for access to Allot personnel.

Note: If an FTP Server is available, it is also possible to connect to the NetEnforcer using the FTP, browse to where the snapshot is located, and use the mget command to get the snapshot (using bin mode).

www.allot.com 31

http://192.1.1.2/snapshot.15.03.01_16.08.33.tgz


Add Device When adding a device to the NetXplorer NX730, there are 10 stages that need to be completed. Therefore, when adding a device and getting a "failed to create topology device" error, it is important to know on which stage it failed. Stage 1: configuration : create device topology

Stage 2: event : create device event counter entry

Stage 3: configuration : check device software version

Stage 4: import configuration : set configuration from Device to DB

Stage 5: catalog : export (deviceTopology)

Stage 6: policy : export default policy (deviceTopology)

Stage 7: register to snmp trap : register AS To Snmp Tables Listeners

Stage 8: collector : assign device to collector

Stage 9: configuration : set admin and oper to 1 - ON

Stage 10: get the latest topology object

To do this, go to the NMS.log, located under Allot_Home:\Allot\netxplorer\jboss-3.2.6\server\allot\log and search for the word "CREATE": 2006-04-01 01:44:13 [RMI TCP Connection(57)-122.122.4.101] INFO topology.dto.TopologyDTOManager - CREATE(1/9) [admin/122.122.4.32] create device topology to DB - started 2006-04-01 01:44:13 [RMI TCP Connection(57)-122.122.4.101] INFO topology.dto.TopologyDTOManager - CREATE(1/9) [admin / 122.122.4.32 #2] create device topology to DB - finished 2006-04-01 01:44:13 [RMI TCP Connection(57)-122.122.4.101] INFO topology.dto.TopologyDTOManager - CREATE(2/9) [admin / 122.122.4.32 #2] create device event counter entry - started 2006-04-01 01:44:13 [RMI TCP Connection(57)-122.122.4.101] INFO topology.dto.TopologyDTOManager - CREATE(2/9) [admin / 122.122.4.32 #2] create device event counter entry - finished The first two stages almost always complete successfully. Keep track of the CREATE (by searching) until the failed stage is found. Fail on stage 4 - set configuration from device to database In this stage, the server reads IP configuration from rc.conf. The following indication will probably be found: 2006-04-01 02:07:22 [RMI TCP Connection(171)-122.122.4.101] ERROR management.ejb.ConfigurationFacadeEJB - failed to setConfigurationFromDeviceToDB null; CausedByException is: Device 122.122.4.101/161 is unreachable when trying to send pdu This indicates that the probe could not send the configuration updates to the server on port 161. In this case, check the following: • Run netstat -an on the NetEnforcer or Server and check whether a connection on port 161 is

established.

www.allot.com 32


• Check that nothing is blocking SNMP traffic along the way. • Check that the database is up and available. Fail on stage 5 - exporting catalogs from the Server to the NetEnforcer In this stage, the Application Server connects to the Apache Server (using CGI on port 80) on the NetEnforcer using the following link: http://122.122.4.32:80/cgi-bin/ProvisionCli.exe. In the NMS.log the following will be seen: 2006-04-01 02:06:48 [RMI TCP Connection(169)-122.122.4.101] INFO topology.dto.TopologyDTOManager - CREATE(5/9) [admin / 122.122.4.32 #6] export Catalogs - started 2006-04-01 02:06:52 [RMI TCP Connection(169)-122.122.4.101] DEBUG

catalog.synch.SynchUtils - send to device= http://122.122.4.32:80/cgi-bin/ProvisionCli.exe is name=<message id="5" type="req"> Potential problems: • Authentication failure may also result from incorrect password. Another indication to that

would appear in $SWGL/nedbg.DataSrv.log on the NetEnforcer. • Make sure the correct admin password was entered. • Try to reset the admin password.

• Communication exception:

Indication: 2006-04-04 11:52:16 [RMI TCP Connection(52)-10.254.48.100] DEBUG catalogs.ejb.CatalogFacadeEJB - EXCEPTION = com.allot.nms.common.net.CommunicationException • Check for access lists (on the NetEnforcer, routers, firewalls, etc). • Check with netstat -an that a connection from the NetEnforcer to the Server on port 80

was established. • Try to connect the NetEnforcer to a different switch (this has worked in the past).

• According to the Troubleshooting Guide. Please note that these problems have never been

encountered: • Check that DataSrv and ProvisionCli.exe are running. • Check in $SWGL/nedbg.DataSrv.log whether DataSrv received the changes (check for

full export). • /tmp/nedbg.ProvisionCli.log - check whether content was received from the Apache

Server (view full XML content). • /var/log/apache/access_log - check whether Apache received change (look for POST to

ProvisionCli.exe). Fail on stage 6 - exporting default policy from the Server to the NetEnforcer Failing on stage 6 may be a result due to large catalogs on the server that need to be added to the NetEnforcer. The NetXplorer server has a timeout of 1 minute to complete the add process. If the process takes longer, it may reach step 6 before stopping. There is no workaround to solve this on site. R&D involvement is needed in order to reduce the processing time on the NetEnforcer to less than the 1 minute limitation. Fail on stage 7 - Register AS to SNMP Tables

www.allot.com 33

http://122.122.4.32/cgi-bin/ProvisionCli.exe

http://122.122.4.32/cgi-bin/ProvisionCli.exe


Failing on stage 7 is most likely to happen when adding a device while management traffic goes through the box. The NetEnforcer reboots and the addition fails. The workaround is to switch the NetEnforcer to bypass, and then add the device. Stage 8 (assign device to collector), 98 (set admin and oper to 1 ON), and 10 (return topology object) may fail if the Application Server cannot connect to the database. The only workaround for this is to stop and start the service and ensure that the 3 databases: CFG, STC and LTC are up and running. If one of the databases are stuck, it must be recreated before the device can be added again. Indications that databases are up and running: • In allot_cfg.log, look for the following:

02/26 11:59:14. Running on Windows XP Build 2600 Service Pack 2 I. 02/26 11:59:14. Database server started at Sun Feb 26 2006 11:59 I. 02/26 11:59:14. Trying to start SharedMemory link... I. 02/26 11:59:14. SharedMemory link started successfully I. 02/26 11:59:14. Trying to start TCPIP link... I. 02/26 11:59:14. Starting on port 50000 I. 02/26 11:59:19. TCPIP link started successfully I. 02/26 11:59:19. Now accepting requests

• In allot_stc.log and allot_ltc.log, look for Enable all events:

I. 04/03 09:15:33. Running on Windows XP Build 2600 Service Pack 2 I. 04/03 09:15:37. Database server started at Mon Apr 03 2006 09:15 I. 04/03 09:15:37. Trying to start SharedMemory link... I. 04/03 09:15:37. SharedMemory link started successfully I. 04/03 09:15:37. Trying to start TCPIP link... I. 04/03 09:15:37. Starting on port 50001 I. 04/03 09:15:42. TCPIP link started successfully I. 04/03 09:15:42. Now accepting requests I. 04/03 09:16:08. Enable all events

Change IP

Defined Behavior There are three locations where the IP of the NetEnforcer can be changed:

• The NetEnforcer itself, using the LCD, CLI or Admin menu • The IP Properties tab within the Configuration Menu of the NetXplorer Server

for a specific NetEnforcer • The properties window of a specific NetEnforcer within the NetXplorer Server

GUI Note: If the IP address cannot be changed for any reason, manually edit the IP address in the rc.conf file, located in the /etc/rc.d directory.

The NetEnforcer Changing the IP address via the NetEnforcer does not impact the NetXplorer Server. The purpose of this is to enable a user to change the IP address of the NetEnforcer and move it to another Server, without affecting the configuration properties of the NetEnforcer within the Server. This will therefore allow another NetEnforcer to be installed in place of this NetEnforcer (using the same model and version) while maintaining the original policy configuration.

www.allot.com 34


An event will be sent to the NetXplorer Server indicating an IP change on the NetEnforcer. An alarm may be assigned to this event within the Event Types Configuration window. To complete an IP address change, the address will also need to be configured within the device properties within the NetXplorer Server.

IP Properties tab within the Configuration Menu Changing the IP address of the NetEnforcer within the IP Properties of the Configuration Menu will change the address of the device itself and the Properties of the device within the Network tree.

Properties window of a specific NetEnforcer Changing the IP address of the NetEnforcer within the Device Properties menu (accessed by right clicking on the device within the Network tree and selecting Properties) does not change any IP definitions on the NetEnforcer. This change will point the NetXplorer Server to connect to the specified IP address. To effect a change on the actual IP address of the device, the address must be defined within either the Configuration Menu or on the device itself.

Current Behavior Please note the differences below: Changing the device IP address via the configuration menu will update the device properties (topology tree). This process will take effect approximately 30 seconds after entered.

In-Band/Out of Band Definitions The NetXplorer and NetEnforcer do not support In Band IP configuration. Currently, the GUI displays both in and out of band. The in-band option is be grayed out. The option will remain since it is a feature that will be available in future versions. On the NetEnforcer itself, currently the CLI enables definition of an in-band address. The LCD does not have this option.

www.allot.com 35


Provisioning Changes

Add Host

Process 1. Server sends XML command to NetEnforcer. 2. NetEnforcer performs changes and updates counters. 3. NetEnforcer sends trap to Server.

Troubleshooting • Server: C:\Allot\netxplorer\jboss-3.2.6\server\allot\log\NMS.log - check whether

changes have been sent. a. …send to device = location… b. XML changes. c. …result from device = location…

o <status>err <error_msg> explanation (development not complete). o <status>ok.

Note: Asynchronous messages may not be displayed together. • NetEnforcer: $SWGL/nedbg.DataSrv.log - check whether DataSrv received

changes. • Identify receipt, change applied and confirmation. • Example successful output, see Appendix I.

• $SWGD/data/allotProvision.xml – check counter ID and new catalog entry • $SWGL/nedbg.AllSnmpAgent.log – check for trap sent.

• Example successful output, see Appendix II.

Configuration Changes

Process 1. SNMP config changes sent. 2. SNMP config changes applied.

Troubleshooting 1. Check NMS.log on Server. 2. $SWGL/nedbg.AllSnmpAgent.log – check for SET command. 3. $SWGL/nedbg.swKeeper.log (system changes) – check for set_conf.

www.allot.com 36


4. $SWGL/nedbg.DataSrv.log (application changes) – look for XML.

Databases Not Synchronized

Symptoms Full export of database from Server.

Explanation This can occur due to

• manual XML changes • CLI changes made when SNMP agent down • NetEnforcer in rescue • And others

Troubleshooting 1. $SWGL/nedbg.AllSnmpAgent.log – Check for PolModifyTag=3 (bad database)

• 1 = good 2. $SWGL/nedbg.DataSrv.log – Check for Full Export and complete XML

To Generate a Full Export Touch $SWGD/data/allotProvision.xml.

www.allot.com 37


RMA/Box Replacement Important note: If there is no unit to replace, do not delete the unit from the server until you have another unit to replace it. Unit A is connected to the server.

Unit B should replace unit A.

1. Connect unit B and add it to the server with a different IP address than unit A. 2. After unit B is reachable, disconnect both units (A and B) through the management port. 3. Set unit B with the original IP address that was defined in unit A. 4. Reconnect management port to unit B. 5. Delete the IP address that was used to define unit B. 6. Perform touch to allotProvision.xml.

Collection Problems

STC Problems Related to Software There may be problems with the STC database due to software running on the NetXplorer PC which may be interrupting the database processes.

Symptom The short term collector is stuck.

• no monitoring reports • NX server reports event/alarm on STC_DEF

Troubleshooting The following message can be found in allot_stc.txt: E. 10/28 01:19:12. *** ERROR *** Assertion failed: 100909 (9.0.2.3137) E. 10/28 01:19:12. Error deleting transaction log file I. 10/28 01:19:12. *** ERROR *** Assertion failed: 100909 (9.0.2.3137) I. 10/28 01:19:12. Error deleting transaction log file I. 10/28 01:19:12. I. 10/28 01:19:12. Attempting to save dump file at 'C:\WINDOWS\TEMP\sa_dump.dmp' I. 10/28 01:19:12. Dump file saved

Explanation The first error, assertion failed error 100909: Error deleting transaction log file is usually caused the transaction log is locked. This indicates that there is another software application currently using the transaction log, preventing the NetXplorer databases from accessing it. Since the NetXplorer databases cannot access the log, the database is shut down. Potential software applications that may lock up the transaction log are:

• system backup software • anti-virus software • defragmentation tools • or others.

www.allot.com 38


Workaround The identified application must be configured not to access specific Sybase files (.db and .log files). Go to http://www.sybase.com/detail?id=1025501 for information on ASA, anti-virus and backup software. It is highly recommended NOT to run such programs on folders where the databases reside. After disabling such programs, it may be necessary to recreate the database. For details on this procedure, see the Recreating Default (ST and LT) Databases section on page 22.

Data Collection Stops Due to NTP Issues

Symptoms The following event/alarm is received on the NetXplorer Server:

invalid bucket time on device <device name> (id 208 - Current bucket time is older that current UTC minus delta).

Troubleshooting See section on Troubleshooting NTP Issues on page 27.

Demo Installation Issues When installing the NetXplorer for demo or training purposes there are a couple of tricks to avoid the installation requirements. Note: These tools are internal and should only be used in exceptionally specific internal situations. This information should NEVER be distributed to anyone outside of Allot CS.

Installing NetEnforcer version 7.1.0 on a NetEnforcer AC-202/302 This should only be done for training purposes. The NetXplorer only supports NetEnforcer models AC-40x (AC-80x, AC-10x0 and AC-25x0 in the future). To disable the check for the NetEnforcer model, create the file /tmp/nocheck.

Skipping installation hardware requirements From NetXplorer Version 25.27 (the current version is 23.25) it is possible to avoid hardware requirements such as memory and available ports. To disable the check, create the file c:\nocheck.

www.allot.com 39

http://www.sybase.com/detail?id=1025501


Appendix

Appendix I

Host output from $SWGL/nedbg.DataSrv.log 09-12 06:36:15(163) <DL_TRACE>: Message received from AS: <message id="4" type="req"> <check>53xXk0LYvZI=</check> <owner>168427883</owner> <change_id>4</change_id> <ops> <op id="1"> <opcode>create</opcode> <location>//catalogs/*/host/parent::*</location> <ID/> <data> <host a_right="1" id="2" name="10.1.1.1" scope="0" type="0"> <entries> <host_entry id="2" type="2"> <ip>167837953</ip> </host_entry> </entries> <queries/> </host> </data> </op> </ops> </message> 09-12 06:36:15(163) <DL_NOTIFY>: Create element, location: //catalogs/*/host/parent::* 09-12 06:36:15(163) <DL_TRACE>: Created element: <host a_right="1" id="2" name="10.1.1.1" scope="0" type="0"> <entries> <host_entry id="2" type="2"> <ip>167837953</ip> </host_entry> </entries> <queries/> </host> 09-12 06:36:15(163) <DL_TRACE>: PmChangeValidator::buildValidNewHostEntry. Validating. 09-12 06:36:15(163) <DL_TRACE>: Returned ID: 2 09-12 06:36:15(163) <DL_TRACE>: Set catalogs update counter to 4 09-12 06:36:15(163) <DL_TRACE>: Set update owner to 168427883 09-12 06:36:15(163) <DL_NOTIFY>: touch file data/lastPolicyUpdate : 1126506975

www.allot.com 40


09-12 06:36:15(163) <DL_TRACE>: Sending notification to clients. 09-12 06:36:15(163) <DL_TRACE>: Update counter 4, number of changed catalogs 1 09-12 06:36:15(163) <DL_TRACE>: Changed catalog: type host_cat, name Host 09-12 06:36:15(163) <DL_TRACE>: Deleted entries. 09-12 06:36:15(163) <DL_TRACE>: Number of entries: 0 09-12 06:36:15(163) <DL_TRACE>: New entries. 09-12 06:36:15(163) <DL_TRACE>: Number of entries: 1 09-12 06:36:15(163) <DL_TRACE>: QuadID: 2. 09-12 06:36:15(163) <DL_TRACE>: Entry: <host a_right="1" id="2" level="trace" name="10.1.1.1" scope="0" type="0"> <entries> <host_entry id="2" type="2"> <ip>167837953</ip> </host_entry> </entries> <queries/> </host> 09-12 06:36:15(163) <DL_TRACE>: Modified entries. 09-12 06:36:15(163) <DL_TRACE>: Number of entries: 0 09-12 06:36:15(163) <DL_TRACE>: Tracked entries. 09-12 06:36:15(163) <DL_TRACE>: Number of entries: 0 09-12 06:36:15(163) <DL_NOTIFY>: CatSvr notify (0x83d89a0) [0] 09-12 06:36:15(163) <DL_TRACE>: Sending notification to clients. 09-12 06:36:15(163) <DL_TRACE>: Update counter 4, number of changed catalogs 1 09-12 06:36:15(163) <DL_TRACE>: Changed catalog: type host_cat, name Host 09-12 06:36:15(163) <DL_TRACE>: Deleted entries. 09-12 06:36:15(163) <DL_TRACE>: Number of entries: 0 09-12 06:36:15(163) <DL_TRACE>: New entries. 09-12 06:36:15(163) <DL_TRACE>: Number of entries: 1 09-12 06:36:15(163) <DL_TRACE>: QuadID: 2. 09-12 06:36:15(163) <DL_TRACE>: Entry: <host a_right="1" id="2" level="trace" name="10.1.1.1" scope="0" type="0"> <entries> <host_entry id="2" type="2"> <ip>167837953</ip> </host_entry> </entries> <queries/> </host> 09-12 06:36:15(163) <DL_TRACE>: Modified entries. 09-12 06:36:15(163) <DL_TRACE>: Number of entries: 0 09-12 06:36:15(163) <DL_TRACE>: Tracked entries. 09-12 06:36:15(163) <DL_TRACE>: Number of entries: 0 09-12 06:36:15(163) <DL_NOTIFY>: CatSvr notify (0x84b40d8) [0] 09-12 06:36:15(163) <DL_TRACE>: Message returned to AS:

www.allot.com 41


<message id="4" type="res"> <check>53xXk0LYvZI=</check> <owner>168427883</owner> <change_id>4</change_id> <ops> <op id="1"> <status>ok</status> <ID>2</ID> </op> </ops> </message> 09-12 06:36:15(163) <DL_NOTIFY>: Detach session (0x83c3cc8), client

Appendix II

Host output from $SWGL/nedbg.AllSnmpAgent.log 09-12 06:36:14(169) <DL_NOTIFY>: PolModifyTag= 1. snmp_t= 1126506686, pol_xml_t= 1126506686, pol_upd_t= 1126506686, rescue_t= 0, full_t= 1126506686 09-12 06:36:15(169) <DL_NOTIFY>: PmRegisterUser updates [260] 09-12 06:36:15(169) <DL_NOTIFY>: handleDataUpdates(): id= 168427883, count=4 09-12 06:36:15(169) <DL_NOTIFY>: Catalog instance 2 is changed :[ <host a_right="1" id="2" level="trace" name="10.1.1.1" scope="0" type="0"> <entries> <host_entry id="2" type="2"> <ip>167837953</ip> </host_entry> </entries> <queries/> </host>] 09-12 06:36:15(169) <DL_TRACE>: Trap counter = 3, index in NotifyLog= [7.99.97.116.95.108.111.103.3] 09-12 06:36:15(169) <DL_TRACE>: Trap counter = 3, index in NotifyLog= [1.3.6.1.2.1.92.1.3.1.1.9.7.99.97.116.95.108.111.103.3] 09-12 06:36:15(169) <DL_TRACE>: New trap counter = 4, index in NotifyLog= [1.3.6.1.2.1.92.1.3.1.1.9.7.99.97.116.95.108.111.103.3] 09-12 06:36:15(169) <DL_NOTIFY>: Trap sent, oid= [1.3.6.1.4.1.2603.0.2], prev_count= 3, cur_count= 4 09-12 06:36:15(169) <DL_NOTIFY>: Set [1.3.6.1.4.1.2603.5.2.203.0] counter to 4

www.allot.com 42

57843993 Nx Troubleshooting Guide

Documents

Transcript of 57843993 Nx Troubleshooting Guide