Ace Troubleshooting

182
Welcome to Cisco DocWiki. We encourage registered Cisco.com users to contribute to this wiki to collaborate on Cisco product documentation. You do not need to log in to read the text. However, you must log in to edit the text. Select the "edit" tab to edit an article or select the "discussion" tab to submit questions or comments about documentation content. See Terms of Use and About DocWiki for more information about Cisco DocWiki. Click here to go to the Cisco ACE Module documentation on www.cisco.com. Click here to go to the Cisco ACE Appliance documentation on www.cisco.com. Contents 1 Audience 2 Organization 3 Creating a PDF of the ACE Troubleshooting Wiki 4 Related Documentation 4.1 ACE Module Documentation 4.2 ACE Appliance Documentation This article provides a systematic approach to identifying and remedying problems that may arise as you use your ACE over a period of time. This guide is not intended to replace configuration best practices or to be an all-inclusive guide for every application. Rather, it is an attempt to provide you with the knowledge and skills necessary to correct the most common issues that you may encounter. Audience This article is intended for all trained network administrators who have experience with the configuration and maintenance of the ACE. Organization This article consists of the following major sections: Overview of ACE Troubleshooting Understanding the ACE Module Architecture and Traffic Flow Preliminary ACE Troubleshooting Troubleshooting ACE Boot Issues Troubleshooting with ACE Logging Cisco Application Control Engine (ACE) Troubleshooting Guide 07/26/11 1

Transcript of Ace Troubleshooting

Page 1: Ace Troubleshooting

Welcome to Cisco DocWiki. We encourage registered Cisco.com users to contribute to this wiki to collaborateon Cisco product documentation. You do not need to log in to read the text. However, you must log in to edit thetext. Select the "edit" tab to edit an article or select the "discussion" tab to submit questions or comments aboutdocumentation content.

See Terms of Use and About DocWiki for more information about Cisco DocWiki.

Click here to go to the Cisco ACE Module documentation on www.cisco.com.

Click here to go to the Cisco ACE Appliance documentation on www.cisco.com.

Contents

1 Audience• 2 Organization• 3 Creating a PDF of the ACETroubleshooting Wiki

4 Related Documentation4.1 ACE Module Documentation♦ 4.2 ACE Appliance Documentation♦

This article provides a systematic approach to identifying and remedying problems that may arise as you use yourACE over a period of time. This guide is not intended to replace configuration best practices or to be anall-inclusive guide for every application. Rather, it is an attempt to provide you with the knowledge and skillsnecessary to correct the most common issues that you may encounter.

Audience

This article is intended for all trained network administrators who have experience with the configuration andmaintenance of the ACE.

Organization

This article consists of the following major sections:

Overview of ACE Troubleshooting

Understanding the ACE Module Architecture and Traffic Flow

Preliminary ACE Troubleshooting

Troubleshooting ACE Boot Issues

Troubleshooting with ACE Logging

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 1

Page 2: Ace Troubleshooting

Troubleshooting Connectivity

Troubleshooting Ethernet Ports (ACE appliance)

Troubleshooting Remote Access

Troubleshooting Access Control Lists

Troubleshooting Network Address Translation

Troubleshooting ACE Health Monitoring

Troubleshooting Layer 4 Load Balancing

Troubleshooting Layer 7 Load Balancing

Troubleshooting Redundancy

Troubleshooting SSL

Troubleshooting Compression

Troubleshooting Performance Issues

ACE Resource Limits

Managing Resources

Show Counter Reference

Creating a PDF of the ACE Troubleshooting Wiki

You can create a PDF of one or more articles in this wiki, including the entire ACE Troubleshooting Wiki. Fordetails, see the Creating a PDF article.

Related Documentation

ACE Module Documentation

Customer Documentation for the Cisco Application Control Engine (ACE) Module• Cisco Application Control Engine (ACE) Configuration Examples on DocWiki•

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 2

Page 3: Ace Troubleshooting

ACE Appliance Documentation

Hardware Installation Guides for the Cisco ACE 4710 Appliance•

Release Notes for the Cisco 4700 Series Application Control Engine Appliance•

Command Reference for the Cisco 4700 Series Application Control Engine Appliance•

Configuration Guides for the Cisco 4700 Series Application Control Engine Appliance•

Cisco CSS-to-ACE Conversion Tool User Guide•

Cisco Application Control Engine (ACE) Configuration Examples on DocWiki•

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 3

Page 4: Ace Troubleshooting

This article introduces the basic concepts, methodology, and general troubleshooting guidelines for problems thatmay occur when you configure and use your ACE.

Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference

Contents

1 Overview of the ACE Troubleshooting Process• 2 Verifying the ACE Image• 3 Enabling ACE Logging• 4 Gathering ACE Troubleshooting Information

4.1 Rebooting the ACE♦ 4.2 Using show Commands♦ 4.3 Capturing Packets in Real Time♦ 4.4 Copying Core Dumps♦ 4.5 After Gathering Troubleshooting Information♦

5 Verifying the Physical Connectivity Between the ACE and the EndHosts

6 Verifying the ACE Layer 2 Connectivity• 7 Verifying the ACE Layer 3 Connectivity• 8 Contacting Cisco Technical Support•

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 4

Page 5: Ace Troubleshooting

Overview of the ACE Troubleshooting Process

To troubleshoot your ACE, follow these general guidelines:

Maintain a consistent and recommended software version across all your ACEs. See the "Verifying theACE Image" section.

1.

See the ACE module release notes for your software version for the latest features, operatingconsiderations, caveats, and CLI command changes.

2.

Before you introduce configuration changes, use the ACE checkpoint feature to bookmark a known goodconfiguration and save your configuration. If you run into problems with the new configuration, you canroll back the new configuration to the known good configuration. See the Cisco Application ControlEngine Module Administration Guide. Troubleshoot new configuration changes immediately after addingthem.

3.

Verify that your configuration is correct for your network application. Make any required changes to therunning-config file, and then test the configuration. If it is satisfactory, save it to the startup-config fileusing the copy running-config startup-config command for a particular virtual context or the writememory command from the Admin context to copy all running-config files in every virtual context totheir respective startup-config files.

4.

Enable system message logging. See the "Enabling ACE Logging" section.5. Gather information that defines the specific symptoms. See the "Gathering ACE TroubleshootingInformation" section.

6.

Verify the physical connectivity between your device and end devices. See the "Verifying the PhysicalConnectivity Between the ACE and the End Hosts" section.

7.

Verify the ACE Layer 2 connectivity. See the "Verifying the ACE Layer 2 Connectivity" section.8. Verify the ACE end-to-end connectivity and the routing configuration. See the "Verifying the ACE Layer3 Connectivity" section.

9.

After you have determined that your troubleshooting attempts have not resolved the problem, contact theCisco Technical Assistance Center (TAC) or your technical support representative. See the "ContactingCisco Technical Support" section.

10.

Verifying the ACE Image

To display the version of the software image and the image filename that is currently running in your ACE, enterthe following command:

ACE_module5/Admin# show versionCisco Application Control Software (ACSW)TAC support: http://www.cisco.com/tacCopyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved.The copyrights to certain works contained herein are owned byother third parties and are used and distributed under license.Some parts of this software are covered under the GNU PublicLicense. A copy of the license is available athttp://www.gnu.org/licenses/gpl.html.

Software

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 5

Page 6: Ace Troubleshooting

loader: Version 12.2[121]system: Version A2(2.0) [build 3.0(0)A2(2.0)] <--------system image file: [LCP] disk0:c6ace-t1k9-mzg.A2_2_0.bin <--------

installed license: no feature license is installed

Hardware Cisco ACE (slot: 5) cpu info: number of cpu(s): 2 cpu type: SiByte cpu: 0, model: SiByte SB1 V0.2, speed: 700 MHz cpu: 1, model: SiByte SB1 V0.2, speed: 700 MHz memory info: total: 955396 kB, free: 289704 kB shared: 0 kB, buffers: 2336 kB, cached 0 kB cf info: filesystem: /dev/cf total: 1000000 kB, used: 494912 kB, available: 505088 kB

last boot reason: NP 1 Failed : NP ME Hung

configuration register: 0x1ACE_module5 kernel uptime is 4 days 22 hours 42 minute(s) 41 second(s)

This command provides other useful information, for example:

Slot in which the ACE resides in the Catalyst 6500 series switch (in this case, slot 5)• Available control plane memory• Last boot reason• Configuration register (confreg) value (0x0 boot to rommon, 0x1 boot using boot string)• ACE uptime•

Enabling ACE Logging

To enable logging on the ACE module and to send system logging (syslog) messages to the monitor, enter thefollowing commands:

ACE_module5/Admin(config)# logging enableACE_module5/Admin(config)# logging monitor 7ACE_module5/Admin(config)# exitACE_module5/Admin# terminal monitor

Note: Use the terminal no monitor command to stop viewing log messages in your remote session.

For more information about logging, see the "Troubleshooting with ACE Logging" section.

Gathering ACE Troubleshooting Information

The following sections recommend ways to gather information that is relevant to the problem that is occurring.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 6

Page 7: Ace Troubleshooting

Rebooting the ACE

Do not reboot the ACE unless it is absolutely necessary. Some information that is important to troubleshootingyour problem may not survive a reboot. Try to gather as much information as possible before rebooting.

Using show Commands

You can use a number of show commands in Exec mode to gather information specific to the symptoms you areobserving in your ACE. In most cases, you can gather the information you need to troubleshoot the ACE byentering the show tech-support command. This command runs many show commands that are useful fortroubleshooting the ACE. You can redirect the output of the show tech-support command to one the followingdestinations:

ACE_module5/Admin# show tech-support > ?<File> Name of file to redirect stdout.disk0: Enter the URI to redirect the output.ftp: Enter the URI to redirect the output.sftp: Enter the URI to redirect the output.tftp: Enter the URI to redirect the output.volatile: Enter the URI to redirect the output.

Capturing Packets in Real Time

Capturing packets (sometimes referred to as a "TCP dump") is a useful aid in troubleshooting connectivityproblems with the ACE or for monitoring suspicious activity. The ACE can track packet information for networktraffic that passes through the ACE. The attributes of the packet are defined by an ACL. The ACE buffers thecaptured packets, and you can copy the buffered contents to a file in Flash memory on the ACE or to a remoteserver. You can also display the captured packet information on your console or terminal.

The ACE captures packets subject to the following guidelines:

One capture session is used per context• Capture is triggered at flow setup• Capture is configured on the client interface where the flow is received•

Note: Probe traffic will not hit a security ACL, so ACLs cannot control the capture of those packets.Therefore, probe traffic cannot be captured by the packet capture utility.

If possible, you should capture packets using the ACE packet capturing utility before and after symptoms appear.Save the packet captures to a file.

To capture packets in real time, follow these steps:

1. Create an ACL for packet capturing or use an existing ACL if it meets the packet capture requirements byentering the following command:

ACE_module5/Admin(config)# access-list FILTER line 10 extended permit tcp any any eq wwwACE_module5/Admin# exit

2. Enter the capture command, for example:

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 7

Page 8: Ace Troubleshooting

ACE_module5/Admin# capture CAPTURE1 interface vlan 200 access-list FILTER

Note: Ensure that the ACL you specify in the capture command is for an input interface. If you configure thepacket capture on the output interface, the ACE will fail to match any packets.

3. Display the capture status to determine the capture status and the buffer size by entering the followingcommand:

ACE_module5/Admin# show capture CAPTURE1 status

Capture session : TESTBuffer size : 64 KCircular : noBuffer usage : 0.00%Status : stopped

Notice that the capture has not started yet. The default buffer size is 64 KB. You can specify a maximum of buffersize of 5000 KB and you can specify a circular buffer.

4. Start the packet capture on the ACE by entering the following command:

ACE_module5/Admin# capture CAPTURE1 start11:56:15.354930 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 62: 209.165.201.10.4144 > 172.16.1.100.80: S [bad tcp cksum 2aee!] 805889668:805889668(0) win 64240 <mss 1460,nop,nop,nop,nop> (DF) (ttl 127, id 2355, len 48)11:56:15.355257 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 62: 209.165.201.10.4144 > 192.168.1.11.80: S [bad tcp cksum c6d3!] 1247081510:1247081510(0) win 64240 <mss 1460,nop,nop,nop,nop> (DF) (ttl 127, id 2355, len 48, bad cksum aa70!)11:56:15.355669 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 58: 192.168.1.11.80 > 209.165.201.10.4144: S [tcp sum ok] 1187651879:1187651879(0) ack 1247081511 win 5840 <mss 1460> (DF) (ttl 64, id 0, len 44)11:56:15.355979 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 58: 172.16.1.100.80 > 209.165.201.10.4144: S [bad tcp cksum 641b!] 746460037:746460037(0) ack 805889669 win 5840 <mss 1460> (DF) (ttl 64, id 0, len 44, bad cksum de68!)11:56:15.356442 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 56: 209.165.201.10.4144 > 172.16.1.100.80: . [tcp sum ok] ack 1 win 64240 (DF) (ttl 127, id 2356, len 40)11:56:15.356839 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 56: 209.165.201.10.4144 > 192.168.1.11.80: . [bad tcp cksum 9be4!] ack 1 win 64240 (DF) (ttl 127, id 2356, len 40, bad cksum aa77!)11:56:15.357203 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 494: 209.165.201.10.4144 > 172.16.1.100.80: P [tcp sum ok] 1:441(440) ack 1 win 64240 (DF) (ttl 127, id 2357, len 480)11:56:15.357918 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 494: 209.165.201.10.4144 > 192.168.1.11.80: P [bad tcp cksum 9be4!] 1:441(440) ack 1 win 64240 (DF) (ttl 127, id 2357, len 480, bad cksum a8be!)11:56:15.358436 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 56: 192.168.1.11.80 > 209.165.201.10.4144: . [tcp sum ok] ack 441 win 6432 (DF) (ttl 64, id 59820, len 40)11:56:15.358582 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 56: 172.16.1.100.80 > 209.165.201.10.4144: . [bad tcp cksum 641b!] ack 441 win 6432 (DF) (ttl 64, id 59820, len 40, bad cksum f4bf!)11:56:15.358822 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 272: 192.168.1.11.80 > 209.165.201.10.4144: P [tcp sum ok] 1:219(218) ack 441 win 6432 (DF) (ttl 64, id 59822, len 258)11:56:15.359106 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 272: 172.16.1.100.80 > 209.165.201.10.4144: P [bad tcp cksum 641b!] 1:219(218) ack 441 win 6432 (DF) (ttl 64, id 59822, len 258, bad cksum f3e3!)11:56:15.359391 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 407: 192.168.1.11.80 > 209.165.201.10.4144: P [tcp sum ok] 219:572(353) ack 441 win 6432 (DF) (ttl 64, id 59824, len 393)11:56:15.359751 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 407: 172.16.1.100.80 > 209.165.201.10.4144: P [bad tcp cksum 641b!] 219:572(353) ack 441 win 6432 (DF) (ttl 64, id 59824, len 393, bad cksum f35a!)11:56:15.360101 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 56: 192.168.1.11.80 > 209.165.201.10.4144: F [tcp sum ok] 572:572(0) ack 441 win 6432 (DF) (ttl 64, id 59826, len 40)11:56:15.360238 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 56: 172.16.1.100.80 > 209.165.201.10.4144: F [bad tcp cksum 641b!] 572:572(0) ack 441 win 6432 (DF) (ttl 64, id 59826, len 40, bad cksum f4b9!)11:56:15.360378 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 56: 209.165.201.10.4144 > 172.16.1.100.80: . [tcp sum ok] ack 572 win 63669 (DF) (ttl 127, id 2358, len 40)11:56:15.360523 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 56: 209.165.201.10.4144 > 192.168.1.11.80: . [bad tcp cksum 9be4!] ack 572 win 63669 (DF) (ttl 127, id 2358, len 40, bad cksum aa75!)11:56:15.360686 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 56: 209.165.201.10.4144 > 172.16.1.100.80: . [tcp sum ok] ack 573 win 63669 (DF) (ttl 127, id 2359, len 40)11:56:15.360831 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 56: 209.165.201.10.4144 > 192.168.1.11.80: . [bad tcp cksum 9be4!] ack 573 win 63669 (DF) (ttl 127, id 2359, len 40, bad cksum aa74!)11:56:15.360973 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 56: 209.165.201.10.4144 > 172.16.1.100.80: F [tcp sum ok] 441:441(0) ack 573 win 63669 (DF) (ttl 127, id 2362, len 40)11:56:15.361130 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 56: 209.165.201.10.4144 > 192.168.1.11.80: F [bad tcp cksum 9be4!] 441:441(0) ack 573 win 63669 (DF) (ttl 127, id 2362, len 40, bad cksum aa71!)11:56:15.361290 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 56: 192.168.1.11.80 > 209.165.201.10.4144: . [tcp sum ok] ack 442 win 6432 (DF) (ttl 64, id 59828, len 40)11:56:15.361436 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 56: 172.16.1.100.80 > 209.165.201.10.4144: . [bad tcp cksum 641b!] ack 442 win 6432 (DF) (ttl 64, id 59828, len 40, bad cksum f4b7!)

ACE_module5/Admin# capture CAPTURE1 stop

5. Copy the packet capture to disk0: by entering the following command:

ACE_module5/Admin# copy capture CAPTURE1 disk0:CAPTURE1

You can also copy the packet capture to an FTP, SFTP, or TFTP server.

6. Display the messages and connections within a packet capture by entering the following command:

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 8

Page 9: Ace Troubleshooting

ACE_module5/Admin# show capture CAPTURE10001: msg_type: ACE_HIT ace_id: 637 action_flag: 0x30002: msg_type: CON_SETUP con_id: 1308623156 out_con_id: 1677724630003: msg_type: PKT_RCV con_id: 1308623156 other_con_id: 00004: msg_type: PKT_XMT con_id: 167772463 other_con_id: 00005: msg_type: PKT_RCV con_id: 167772463 other_con_id: 00006: msg_type: PKT_XMT con_id: 1308623156 other_con_id: 0<snip>0025: msg_type: PKT_RCV con_id: 167772463 other_con_id: 00026: msg_type: PKT_XMT con_id: 1308623156 other_con_id: 00027: msg_type: CON_CLOSE con_id: 167772463 reason: 00028: msg_type: CON_CLOSE con_id: 1308623156 reason: 0

7. Display the details of each packet within a capture by entering the following command:

ACE_module5/Admin# show capture CAPTURE1 detail0001: msg_type: ACE_HITace_id: 637 action_flag: 0x3src_addr: 209.165.201.10 src_port: 4144dst_addr: 172.16.1.100 dst_port: 80l3_protocol: 0 l4_protocol: 6message_hex_dump:0x0000: 0006 0104 0000 027d 0000 0000 d1a5 c90a .......}........0x0010: ac10 0164 0609 0013 1030 0050 0000 0000 ...d.....0.P....0x0020: 0052 0000 05b4 0000 0000 027d 0300 0000 .R.........}....0x0030: 0000 0040 0000 0000 0000 0000 0000 0000 [email protected]: 0000 0000 0000 0001 ........

0002: msg_type: CON_SETUPcon_id: 1308623156 out_con_id: 167772463src_addr: 209.165.201.10 src_port: 4144dst_addr: 172.16.1.100 dst_port: 80l3_protocol: 0 l4_protocol: 6message_hex_dump:0x0000: 0006 0101 4e00 0134 0a00 012f 0000 0000 ....N..4.../....0x0010: d1a5 c90a ac10 0164 06e9 0013 1030 0050 .......d.....0.P0x0020: e5b3 f25e 0012 0000 05b4 0100 0a00 012f ...^.........../0x0030: 0000 0000 0018 0480 2445 0000 0000 0001 ........$E......0x0040: 0000 0030 faf0 0010 05b4 0000 3008 e685 ...0........0...0x0050: 0000 0000 e1ad 6b69 0000 0000 0000 027d ......ki.......}0x0060: 0000 0000 e1ad 6b69 0000 0000 0000 0000 ......ki........0x0070: 0000 0000 0000 0000 0000 0000 0000 0000 ................0x0080: c0a8 010b d1a5 c90a 06a1 0018 0050 1030 .............P.00x0090: 1a4c 0da2 0055 0000 05b4 0100 4e00 0134 .L...U......N..40x00a0: 0000 0000 0018 0480 2445 0022 0000 0000 ........$E."....0x00b0: 0000 0000 0000 0000 05b4 0000 0000 0000 ................0x00c0: 4a54 f427 e1ad 6b6b 0000 0000 0000 027d JT.'..kk.......}0x00d0: 0000 0000 e1ad 6b6b 0000 0000 0000 0000 ......kk........0x00e0: 0000 0000 0000 0000 0000 0000 ............

0003: msg_type: PKT_RCVcon_id: 1308623156 other_con_id: 0message_hex_dump:0x0000: 0500 0050 0050 8034 0008 0014 0010 1488 ...P.P.4........0x0010: 0020 000b fcfe 1b01 001c f909 1800 0800 ................0x0020: 4500 0030 0933 4000 7f06 aa70 d1a5 c90a [email protected]: ac10 0164 1030 0050 3008 e684 e5b3 f25e ...d.0.P0......^0x0040: 7002 faf0 18fd 0000 0204 05b4 0101 0101 p...............

0004: msg_type: PKT_XMT

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 9

Page 10: Ace Troubleshooting

con_id: 167772463 other_con_id: 0message_hex_dump:0x0000: 4010 0050 0050 8034 0000 0028 0000 0088 @..P.P.4...(....0x0010: 0004 000c 29f3 cde6 000b fcfe 1b01 0800 ....)...........0x0020: 4500 0030 0933 4000 7f06 aa70 d1a5 c90a [email protected]: c0a8 010b 1030 0050 4a54 f426 0000 0000 .....0.PJT.&....0x0040: 7002 faf0 18fd 0000 0204 05b4 0101 0101 p...............

0005: msg_type: PKT_RCVcon_id: 167772463 other_con_id: 0message_hex_dump:0x0000: 0500 004c 0050 8034 0008 0028 0010 2888 ...L.P.4...(..(.0x0010: 0020 0018 b9a6 890d 000c 29f3 cde6 0800 ..........).....0x0020: 4500 002c 0000 4000 4006 de68 c0a8 010b E..,..@[email protected]: d1a5 c90a 0050 1030 46ca 2127 4a54 f427 .....P.0F.!'JT.'0x0040: 6012 16d0 6df5 0000 0204 05b4 `...m.......

0006: msg_type: PKT_XMTcon_id: 1308623156 other_con_id: 0message_hex_dump:0x0000: 4010 004c 0050 8034 0000 0014 0000 0088 @..L.P.4........0x0010: 0004 001c f909 1800 000b fcfe 1b01 0800 ................0x0020: 4500 002c 0000 4000 4006 de68 ac10 0164 E..,..@[email protected]: d1a5 c90a 0050 1030 2c7e 1385 3008 e685 .....P.0,~..0...0x0040: 6012 16d0 6df5 0000 0204 05b4 `...m.......

0007: msg_type: PKT_RCVcon_id: 1308623156 other_con_id: 0message_hex_dump:0x0000: 0500 004a 0050 8034 0008 0014 0010 1488 ...J.P.4........0x0010: 0020 000b fcfe 1b01 001c f909 1800 0800 ................0x0020: 4500 0028 0934 4000 7f06 aa77 d1a5 c90a E..([email protected]: ac10 0164 1030 0050 3008 e685 2c7e 1386 ...d.0.P0...,~..0x0040: 5010 faf0 05ad 0000 0000 P.........

0008: msg_type: PKT_XMTcon_id: 167772463 other_con_id: 0message_hex_dump:0x0000: 4010 004a 0050 8034 0000 0028 0000 0088 @..J.P.4...(....0x0010: 0004 000c 29f3 cde6 000b fcfe 1b01 0800 ....)...........0x0020: 4500 0028 0934 4000 7f06 aa77 d1a5 c90a E..([email protected]: c0a8 010b 1030 0050 4a54 f427 46ca 2128 .....0.PJT.'F.!(0x0040: 5010 faf0 05ad 0000 0000 P.........

0009: msg_type: PKT_RCVcon_id: 1308623156 other_con_id: 0message_hex_dump:0x0000: 0500 0200 0050 8034 0008 0014 0010 1488 .....P.4........0x0010: 0020 000b fcfe 1b01 001c f909 1800 0800 ................0x0020: 4500 01e0 0935 4000 7f06 a8be d1a5 c90a [email protected]: ac10 0164 1030 0050 3008 e685 2c7e 1386 ...d.0.P0...,~..0x0040: 5018 faf0 a0bb 0000 4745 5420 2f73 6d61 P.......GET./sma0x0050: 6c6c 2e68 746d 6c20 4854 5450 2f31 2e31 ll.html.HTTP/1.10x0060: 0d0a 486f 7374 3a20 3137 322e 3136 2e31 ..Host:.172.16.10x0070: 2e31 3030 0d0a 5573 6572 2d41 6765 6e74 .100..User-Agent

0010: msg_type: PKT_XMTcon_id: 167772463 other_con_id: 0message_hex_dump:0x0000: 4010 0200 0050 8034 0000 0028 0000 0088 @....P.4...(....0x0010: 0004 000c 29f3 cde6 000b fcfe 1b01 0800 ....)...........0x0020: 4500 01e0 0935 4000 7f06 a8be d1a5 c90a E....5@.........

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 10

Page 11: Ace Troubleshooting

0x0030: c0a8 010b 1030 0050 4a54 f427 46ca 2128 .....0.PJT.'F.!(0x0040: 5018 faf0 a0bb 0000 4745 5420 2f73 6d61 P.......GET./sma0x0050: 6c6c 2e68 746d 6c20 4854 5450 2f31 2e31 ll.html.HTTP/1.10x0060: 0d0a 486f 7374 3a20 3137 322e 3136 2e31 ..Host:.172.16.10x0070: 2e31 3030 0d0a 5573 6572 2d41 6765 6e74 .100..User-Agent <snip>

Note: If you view the ACE capture file in a third-party sniffer (for example, Wireshark), you will notice onlythe messages or type PKT_RCV and PKT_XMT are displayed. This situation is expected because thesniffer is not aware of the ACE's internal messaging.

Copying Core Dumps

If the ACE fails with a core dump, the core dump files may contain useful information. The core dump files residein the core: directory. To view the contents of the core: directory, enter the following command:

ACE_module5/Admin# dir core:

123589 Feb 22 00:34:20 2009 qnx_1_mecore_log.999.tar.gz 30361 Feb 22 00:34:22 2009 ixp1_crash.txt

Usage for core: filesystem 153950 bytes total used 202943138 bytes free 203097088 total bytes

You can copy the contents of the core: directory to several locations by using the copy core: command. Thesyntax of this command is as follows:

copy {core:filename | disk0:[path/]filename | running-config | startup-config}{ftp://server/path[/filename] | sftp://[username@]server/path[/filename] |tftp://server[:port]/path[/filename]}

The ACE provides core dumps for both the control plane and the data plane. Each core dump file contains thefollowing information:

Version• Time of failure• Number of CPUs• Current CPU• BKL status• IRQ lock status• Buffers•

After Gathering Troubleshooting Information

After you have gathered all the above information, be prepared to send the information to your customer servicerepresentative or TAC. You can send the information in the following ways:

FTP• SFTP• TFTP•

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 11

Page 12: Ace Troubleshooting

Verifying the Physical Connectivity Between the ACE and theEnd Hosts

To verify the physical connectivity of the ACE, follow these steps:

Check all cable connections on the Catalyst 6500 series switch or Cisco 7600 series router that mayimpact the ACE.

1.

Use the extended ping command to send an ICMP Echo request to the end devices.2.

ACE_module5/Admin# pingTarget IP address: 10.1.1.2Repeat count [5]: 4Datagram size [100]: 200Timeout in seconds [2]: 10Extended commands [n]: 4Pinging 10.1.1.2 with timeout = 10, count = 4, size = 200 ....

Response from 10.1.1.2 : seq 1 time 0.494 msResponse from 10.1.1.2 : seq 2 time 0.367 msResponse from 10.1.1.2 : seq 3 time 0.264 msResponse from 10.1.1.2 : seq 4 time 0.237 ms4 packet sent, 4 responses received, 0% packet loss

If a host is one hop away and you are unable to reach the host, then ping the intermediary gateway. If the gatewayis not reachable, enter the show ip route command and check to make sure that the correct route is displayed. Forexample, enter:

ACE_module5/Admin# show ip route

Routing Table for Context Admin (RouteId 0)

Codes: H - host, I - interface S - static, N - nat A - need arp resolve, E - ecmp

Destination Gateway Interface Flags------------------------------------------------------------------------0.0.0.0 10.2.2.1 vlan130 S [0xc]10.2.2.0/24 0.0.0.0 vlan130 IA [0x30]172.27.15.0/24 0.0.0.0 vlan100 IA [0x30]172.27.16.0/24 0.0.0.0 vlan200 IA [0x30]172.19.110.0/26 0.0.0.0 vlan55 IA [0x30]172.27.16.16/29 0.0.0.0 vlan200 N [0x280]172.27.16.33/32 0.0.0.0 vlan100 N [0x280]

Total route entries = 7

If necessary, enter a static route for the gateway.

Verifying the ACE Layer 2 Connectivity

To verify the Layer 2 connectivity of the ACE, follow these steps:

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 12

Page 13: Ace Troubleshooting

1. Verify that the ARP table is populated with the IP addresses and corresponding MAC addresses of the ACE,the gateway, the local interface, and other IPs that the ACE has learned.

switch/Admin# show arp

Context Admin================================================================================IP ADDRESS MAC-ADDRESS Interface Type Encap NextArp(s) Status================================================================================10.86.215.208 00.02.7e.39.51.9c vlan130 LEARNED 5 2350 sec up10.86.215.228 00.e0.81.22.78.ff vlan130 LEARNED 6 6379 sec up10.86.215.234 00.1a.a1.48.f3.44 vlan130 LEARNED 4 1114 sec up10.86.215.1 00.00.0c.07.ac.00 vlan130 GATEWAY 2 153 sec up10.86.215.2 00.11.5d.e1.2f.fc vlan130 LEARNED 3 12054 sec up10.86.215.134 00.18.b9.a6.91.15 vlan130 INTERFACE LOCAL _ up================================================================================Total arp entries 6

2. Verify that the ACE is connected to the switch fabric of the Catalyst 6500 series switch or the Cisco 7600series router. The ACE uses a 10-Gigabit Ethernet switch fabric interface (SFI) to connect to the chassisbackplane as opposed to the CSM, which uses a port channel. The ACE uses the following format for thisinterface:

Te<slot>/1

For example, if the ACE is in slot 5, you can see the status of the backplane connection by entering the followingcommand on the Catalyst 6500 series switch or the Cisco 7600 series router:

cat6k# show interface te5/1 status

Port Name Status Vlan Duplex Speed TypeTe5/1 connected trunk full 10G MultiService Module

If there is no output from this command, then either the ACE is not installed properly or the ACE is powereddown.

3. Verify the association of the ACE MAC entries with the allocated VLAN interfaces. Enter the followingcommand at the Supervisor CLI:

cat6k# show mac-address-table dynamicLegend: * - primary entry age - seconds since last seen n/a - not available

vlan mac address type learn age ports------+----------------+--------+-----+----------+--------------------------...* 130 0018.b9a6.9115 dynamic Yes 40 Te5/1 <------- MAC address should be in the range displayed by the show module command...

cat6k# show module 5Mod Ports Card Type Model Serial No.--- ----- -------------------------------------- ------------------ -----------

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 13

Page 14: Ace Troubleshooting

5 1 Application Control Engine Module ACE10-6500-K9 SAD1031044S

Mod MAC addresses Hw Fw Sw Status--- ---------------------------------- ------ ------------ ------------ ------- 5 0018.b9a6.9114 to 0018.b9a6.911b 1.1 8.7(0.22)ACE A2(2.0) Ok <------- MAC address range assigned to the ACE

Mod Online Diag Status---- ------------------- 5 Pass

4. Check the status of the Te5/1 port to ensure that it is in the forwarding state by entering the followingcommand:

cat6k# show spanning-tree vlan 130

MST0 Spanning tree enabled protocol mstp Root ID Priority 32768 Address 0001.632f.2c17 Cost 200019 Port 642 (GigabitEthernet6/2) Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32768 (priority 32768 sys-id-ext 0) Address 0011.bc06.f800 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type---------------- ---- --- --------- -------- --------------------------------Gi2/14 Desg FWD 20000 128.142 P2pGi2/37 Desg FWD 200000 128.165 P2pTe3/1 Desg FWD 2000 128.257 Edge P2pTe5/1 Desg FWD 2000 128.513 Edge P2pGi6/2 Root FWD 200000 128.642 Shr Bound(STP)Te8/1 Desg FWD 2000 128.897 Edge P2p

Verifying the ACE Layer 3 Connectivity

Use the traceroute command to check the route between the ACE and the end devices.

ACE_module5/Admin# traceroute 10.20.12.153traceroute to 10.20.12.153 (10.20.12.153), 30 hops max, 40 byte packets 1 10.20.215.2 (10.20.215.2) 0.532 ms 0.436 ms 0.362 ms 2 10.20.239.161 (10.20.239.161) 0.421 ms 0.488 ms 0.404 ms 3 10.20.238.93 (10.20.238.93) 0.471 ms 0.422 ms 0.413 ms 4 172.27.16.177 (172.27.16.177) 0.488 ms 0.435 ms 0.430 ms 5 172.27.16.226 (172.27.16.226) 0.474 ms 0.363 ms 0.368 ms 6 192.168.0.134 (192.168.0.134) 0.624 ms 0.510 ms 0.494 ms 7 10.20.12.153 (10.20.12.153) 23.982 ms 24.702 ms 25.976 ms

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 14

Page 15: Ace Troubleshooting

Contacting Cisco Technical Support

If you are unable to resolve a problem after using the troubleshooting suggestions in the articles in this wiki,contact the Cisco Technical Assistance Center (TAC) for assistance and further instructions. Before you call, havethe following information ready to help your TAC engineer assist you as quickly as possible:

Date that you received the ACE• Chassis serial number (located on a label on the right side of the rear panel of the chassis)• Type of software and release number (if possible, enter the show version command)• Maintenance agreement or warranty information• Brief description of the problem• Brief explanation of the steps that you have already taken to isolate and resolve the problem•

For information on steps to take before calling Technical Support, see the "Gathering ACE TroubleshootingInformation" section.

You can reach TAC in several ways as follows:

Create a service request online•

Call the TAC at the telephone numbers on this page.•

Contact the Cisco Small Business Support Center•

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 15

Page 16: Ace Troubleshooting

This article describes the ACE architecture and how data flows into, gets processed, and flows out of the ACE. Itprovides a basic understanding of these concepts to assist you in troubleshooting the ACE.

Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference

Contents

1 Understanding the ACE Architecture1.1 Overview of the ACEHardware Architecture

1.2 Control Plane♦ 1.3 Data Plane

1.3.1 Classification andDistribution Engine

1.3.2 Network Processors◊ 1.3.3 SSL Crypto Module◊

2 Understanding the ACE Traffic Flow2.1 To-the-ACE Traffic♦ 2.2 Through-the-ACE Traffic♦

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 16

Page 17: Ace Troubleshooting

Understanding the ACE Architecture

Having a basic understanding of the ACE architecture and data flow can help to make troubleshooting the ACEeasier. This section describes the major functional areas of the ACE and how they work together.

Overview of the ACE Hardware Architecture

The ACE hardware architecture is divided into a series of functional areas or subsystems that are defined byprocessors or groups of processors and interfaces as shown in Figure 1.

Figure 1. ACE Module Architecture

A console connection allows direct access to the ACE control plane (CP) for initial configuration, management,and troubleshooting. The supervisor engine connection allows you to determine the status of the ACE, to loadimages into the ACE, to reboot the ACE, and to provide remote access to the ACE from the Catalyst 6500 series

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 17

Page 18: Ace Troubleshooting

switch or Cisco 7600 series router when you use the session command. Because the ACE has no external ports,packets enter the ACE through the Switch Fabric Interface (SFI) connected to the Catalyst 6500 series switch orCisco 7600 series router back plane. The two major functional areas of the ACE are as follows:

Control plane• Data plane•

Control Plane

The control plane (CP) is used to configure the ACE and for management traffic, syslogs, ARP, DHCP, and soon. You can access the CP directly by using the console port. For remote management, you must configure amanagement interface and enable remote access using a management policy to permit Telnet or SSH access, forexample. The CP is responsible for the following ACE functions:

Device management and control• Configuration management (CLI or XML interface)• Server health monitoring• syslogs• SNMP• Address Resolution Protocol (ARP)• DHCP relay• Redundancy (also known as high availability or fault tolerance)• Access control list (ACL) compilation•

Data Plane

The data plane (DP) is responsible for distributing and processing packets and connections that do not match amanagement policy.

In the ACE, the CP and the DP are separated and run on different processors for maximum performance. SeeFigure 2.

Figure 2. ACE Data Plane

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 18

Page 19: Ace Troubleshooting

The DP is responsible for the following ACE functions:

Access control lists (ACLs)• Connection management• TCP termination• Network address translation (NAT)• SSL processing (termination, initiation, encryption, and decryption)• Regular expression matching• Load balancing and forwarding• Application protocol inspection•

The DP consists of the following functional areas:

Classification distribution engine (CDE)• Network processors (NPs)• SSL Crypto Module• Daughter card interfaces (for future feature expansion)•

Classification and Distribution Engine

The Classification and Distribution Engine (CDE) is the traffic controller for the ACE. Its main purpose is toforward packets that it receives from the SFI to the two network processors (NPs). It also acts as the central pointof contact among all the major subsystems within the ACE. The CDE computes, and if necessary, adjusts the IP,TCP, and UDP checksums of every packet that it receives.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 19

Page 20: Ace Troubleshooting

The CDE appends a special header known as the IMPH header to each packet before sending it to the fast path.The IMPH header is 18 bytes long and contains information from the DBUS header (the header sent to the ACEModule by the Catalyst 6500 series switch or Cisco 7600 series router) as well as special messaging directlyunderstood by the fast path. Fields in the IMPH header can include notification of a checksum error, Layer 3 orLayer 4 offsets, source and destination ports of the CDE, the VLAN for determining the interface that the fastpath will use, and so on.

Network Processors

The ACE has two network processors (NP1 and NP2) that perform most of the packet processing in the ACE. Alltraffic entering the ACE must traverse one or both NPs after being forwarded by the CDE. Each NP contains aCPU (XScale) and several components called microengines (MEs). See Figure 3.

Figure 3. Network Processor Micorengines

Each microengine can handle eight simultaneous threads or processes and performs a specific function for the NPas follows:

Receive - One ME for receiving incoming packets• Fast Path - Four MEs for the hardware accelerated data path that is used for MAC rewrite, NAT, TCPnormalization, and so on (essentially all operations performed on a per-packet basis)

ICM - One ME for the inbound connection manager• OCM - One ME for the outbound connection manager• CCM - One ME for the connection close manager• TCP - Two MEs for TCP termination with a full TCP stack• HTTP - Two MEs for HTTP parsing• Unused (future expansion) - One ME• SSL Record Layer - One ME for the SSL record layer•

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 20

Page 21: Ace Troubleshooting

IP fragmentation timers - One ME for IP fragmentation reassembly and timer management• DNS and ICMP Inspection - One ME for DNS and ICMP packet inspection•

The XScale microprocessor is programmed to handle the following features:

Load-balancing algorithms• SSL handshake• FTP and Real-Time Streaming Protocol (RTSP) inspection• HTTP inspection (although a considerable part is performed by the microengines)• High-availability heartbeat generation• Returned statistics for most connection-related commands•

Each network processor has RDRAM memory to store ACL entries, routing table entries, ARP entries, andinspection policies. Additional SRAM memory provides faster access times and is used to store regularexpressions and statistics on a per?virtual system basis, among other things.

SSL Crypto Module

The SSL Crypto Module is responsible for SSL record layer processing. This processing includes encrypting anddecrypting data for SSL flows.

Understanding the ACE Traffic Flow

Because the ACE has no native ports, it relies on the switch fabric in the Catalyst 6500 series switch or the Cisco7600 series router back plane to send and receive packets to and from the network. Packets that are marked with adestination VLAN and Layer 2 information enter the ACE through the SFI on the 10 Gbps Ethernet link. Packetsentering or leaving the ACE traverse this link using VLAN tagging. The switch fabric interface (SFI) forwards tothe CDE all packets that are destined to the ACE. See Figure 1.

The CDE classifies the packets based on the configured traffic policies, fills out the IMPH header information,and forwards the traffic to one of the NPs. To determine which NP to forward the traffic to, the CDE hashesincoming packets based on the traffic type as follows:

TCP/UDP - Hash of source/destination port• Non-TCP/UDP IP - Hash of source/destination IP address• Non-IP - Hash of source/destination Layer 2 MAC address•

The NPs process the traffic and forward it back through the CDE to either the control plane or the SSL CryptoModule for further processing.

To-the-ACE Traffic

To-the-ACE traffic is traffic that is destined to an interface VLAN IP address on the ACE. This traffic must matcha class map of type management, which is associated with a policy map and applied as a service policy on aninterface VLAN. The management class map supports the following protocols:

Hypertext Transfer Protocol (HTTP)•

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 21

Page 22: Ace Troubleshooting

Hypertext Transfer Protocol Secure (HTTPS)• Internet Control Message Protocol (ICMP)• Keepalive Application Protocol (KAL-AP)• User Datagram Protocol (UDP)• Simple Network Management Protocol (SNMP)• Secure Shell (SSH) Protocol• Telnet•

This management traffic is called control plane traffic because it is destined to the CP. Because of the separationof the CP traffic from the data plane traffic on different processors, the control plane traffic will never interferewith data plane traffic, even if the control plane is oversubscribed.

Through-the-ACE Traffic

The CDE sends traffic that requires load balancing, forwarding, routing, or other processing by the ACE to one ofthe NPs.

The NPs comprise two parallel forwarding paths that maintain their own connection state information and forwardtraffic independently.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 22

Page 23: Ace Troubleshooting

This article describes some basic troubleshooting steps that you can perform to rule out some of the simpler issuesbefore delving deeper into the troubleshooting process.

Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference

Contents

1 Preliminary ACE Troubleshooting Steps• 2 Checking the ACE Status from the Supervisor Engine• 3 Verifying the MSFC VLAN Configuration• 4 Establishing a Session with the ACE from the SupervisorEngine

5 Verifying the ACE is Receiving VLAN Allocations from theMSFC

6 Verifying the ACE Image• 7 Verifying Your ACE Licenses• 8 Configuring an ACL to Permit Input Traffic to the ACE• 9 Verifying that the ACE is Sending and Receiving Traffic• 10 Verifying to-the-ACE Traffic•

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 23

Page 24: Ace Troubleshooting

Preliminary ACE Troubleshooting Steps

Check the status of the ACE module from the Catalyst 6500 series switch or Cisco 7600 series router. Seethe "Checking the ACE Status from the Supervisor Engine" section.

1.

Verify that you have allocated the correct VLANs to the ACE in the Multilayer Switch Feature Card(MSFC) VLAN configuration. See the "Verifying the MSFC VLAN Configuration" section.

2.

Verify that you can establish a session to the ACE from the supervisor engine. See the "Establishing aSession with the ACE from the Supervisor Engine" section.

3.

Verify that the ACE is receiving VLAN allocations from the MSFC. See the Verifying the ACE isReceiving VLAN Allocations from the MSFC" section.

4.

Verify your ACE bandwidth, SSL, and virtualization licenses. See the "Verifying Your ACE Licenses"section.

5.

Verify that you have configured an access control list (ACL) to permit traffic on the interfaces on whichyou wish the ACE to receive traffic. If you do not configure an ACL to permit traffic on an interface, alltraffic destined to that interface will be blocked by the ACE. See the "Configuring an ACL to PermitInput Traffic to the ACE" section.

6.

Verify that the ACE is sending and receiving traffic. See the "Verifying that the ACE is Sending andReceiving Traffic" section.

7.

Verify the management traffic to the control plane. See the "Verifying to-the-ACE Traffic"section.8.

Checking the ACE Status from the Supervisor Engine

Before you begin to troubleshoot your ACE, Telnet to the Catalyst 6500 series switch or Cisco 7600 series routersupervisor engine, log in, and check the status of the ACE.

telnet 10.1.1.2

User Access Verification

Password:cat6k> enablePassword:cat6k# show moduleMod Ports Card Type Model Serial No.--- ----- -------------------------------------- ------------------ ----------- . . 5 1 Application Control Engine Module ACE20-6500-K9 SAD1031044S . .

Mod MAC addresses Hw Fw Sw Status--- ---------------------------------- ------ ------------ ------------ -------

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 24

Page 25: Ace Troubleshooting

. . 5 0018.b9a6.9114 to 0018.b9a6.911b 1.1 8.7(0.22)ACE A2(2.0) Ok <------- ACE status

. .

Mod Sub-Module Model Serial Hw Status--- --------------------------- ------------------ ----------- ------- ------- 6 Policy Feature Card 3 WS-F6K-PFC3A SAL09094NUB 2.5 Ok 6 MSFC3 Daughterboard WS-SUP720 SAL09094N33 2.5 Ok

Mod Online Diag Status--- ------------------- . . 5 Pass

. .

Verifying the MSFC VLAN Configuration

To verify that the VLANs that you intend to use in your ACE have been configured and allocated to the ACE inthe MSFC, follow these steps:

1. Check the VLANs configured and allocated to the ACE by entering the following command from thesupervisor engine:

cat6k# show run | include svclc

svclc module 5 vlan-group 123,130,133

2. Ensure that the VLAN groups that you intend to use for your ACE are allocated properly in the MSFCconfiguration by entering the following commands:

cat6k# show svclc module 5 vlan-groupModule Vlan-groups------ ----------- 05 123,130,133

cat6k# show svclc vlan-groupDisplay vlan-groups created by both ACE module and FWSM commands

Group Created by vlans----- ---------- -----123 ACE 103,105,107,111-112,119,134,160,171,200,203,205,207,211-212,226,253,260130 ACE 130133 ACE 100,194,221,256-257

3. Verify that the VLANs you intend to use in your ACE are configured in the MSFC by entering thefollowing command:

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 25

Page 26: Ace Troubleshooting

cat6k# show interface te5/1 trunk

Port Mode Encapsulation Status Native vlanTe5/1 on 802.1q trunking 1

Port Vlans allowed on trunkTe5/1 100,103,105,107,111-112,119,130,134,160,171,194,200,203,205,207,211-212,221,226,253,256-257,260

Port Vlans allowed and active in management domainTe5/1 100,103,105,107,111-112,119,130,134,160,171,194,200,203,205,207,211-212,221,226,253,256-257

Port Vlans in spanning tree forwarding state and not prunedTe5/1 100,103,105,107,111-112,119,130,134,160,171,194,200,203,205,207,211-212,221,226,253,256-257

4. Ensure that traffic is routed to two ACEs in the same chassis when both client- and server-side VLANsare configured as switched virtual interfaces (SVIs) on the MSFC in routed mode by entering thefollowing command:

cat6k# show svclc multiple-vlan-interfacesMultiple ACE vlan interfaces feature is enabled

Establishing a Session with the ACE from the SupervisorEngine

To verify that you can establish a session with the ACE from the supervisor engine in the Catalyst 65000 seriesswitch or the Cisco 7600 series router, enter the following command:

cat6k# session slot 5 processor 0The default escape character is Ctrl-^, then x.You can also type 'exit' at the remote prompt to end the sessionTrying 127.0.0.50 ... Open

ACE_module5 login:

Verifying the ACE is Receiving VLAN Allocations from theMSFC

Ensure that the VLAN that you intend to use on ACE are allocated properly in the MSFC configuration byentering the following command:

ACE-1/Admin# show vlansVlans configured on SUP for this module vlan123 vlan130 vlan133

If interface VLANs are already assigned on the ACE you can use the show interface vlan <num> command toverify the interface is properly assigned on the MSFC and up on the MSFC:

ACE-1/Admin# show interface vlan 123

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 26

Page 27: Ace Troubleshooting

vlan10 is up Hardware type is VLAN MAC address is 00:18:b9:a6:89:0d Mode : routed IP address is 10.10.10.1 netmask is 255.255.255.0 FT status is non-redundant Description:not set MTU: 1500 bytes Last cleared: never Alias IP address not set Peer IP address not set

Assigned from the Supervisor, up on Supervisor 7101679 unicast packets input, 878043707 bytes 0 multicast, 0 broadcast 0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops 6387914 unicast packets output, 1541924399 bytes 0 multicast, 22826 broadcast 0 output errors, 0 ignored

Verifying the ACE Image

To display the version of the software image and the image filename that is currently running in your ACE, enterthe following command:

ACE_module5/Admin# show versionCisco Application Control Software (ACSW)TAC support: http://www.cisco.com/tacCopyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved.The copyrights to certain works contained herein are owned byother third parties and are used and distributed under license.Some parts of this software are covered under the GNU PublicLicense. A copy of the license is available athttp://www.gnu.org/licenses/gpl.html.

Software loader: Version 12.2[121]

system: Version A2(2.0) [build 3.0(0)A2(2.0)] <--------system image file: [LCP] disk0:c6ace-t1k9-mzg.A2_2_0.bin <--------

installed license: no feature license is installed

Hardware Cisco ACE (slot: 5) cpu info: number of cpu(s): 2 cpu type: SiByte cpu: 0, model: SiByte SB1 V0.2, speed: 700 MHz cpu: 1, model: SiByte SB1 V0.2, speed: 700 MHz memory info: total: 955396 kB, free: 289704 kB shared: 0 kB, buffers: 2336 kB, cached 0 kB cf info: filesystem: /dev/cf total: 1000000 kB, used: 494912 kB, available: 505088 kB

last boot reason: NP 1 Failed : NP ME Hung

configuration register: 0x1ACE_module5 kernel uptime is 4 days 22 hours 42 minute(s) 41 second(s)

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 27

Page 28: Ace Troubleshooting

This command provides other useful information, for example:

Slot in which the ACE resides in the Catalyst 6500 series switch (in this case, slot 5)• Available memory• Last boot reason• Configuration register (confreg) value• ACE uptime•

Verifying Your ACE Licenses

Log in to your ACE and enter the following command to display the SSL, virtualization, and bandwidth licensesthat are currently installed and in use in your ACE:

ACE_module5/Admin# show license usageLicense Ins Lic Status Expiry Date Comments Count--------------------------------------------------------------------------------ACE-08G-LIC No - Unused -ACE-16G-LIC No - Unused -ACE-UPG1-LIC No - Unused -ACE-UPG2-LIC No - Unused -ACE-VIRT-020 No - Unused -ACE-VIRT-050 No - Unused -ACE-VIRT-100 No - Unused -ACE-VIRT-250 Yes 1 In use never -ACE-VIRT-UP1 No - Unused -ACE-VIRT-UP2 No - Unused -ACE-VIRT-UP3 No - Unused -ACE10-16G-LIC No - Unused -ACE-SEC-LIC-K9 No - Unused -ACE-SSL-05K-K9 No - Unused -ACE-SSL-10K-K9 No - Unused -ACE-SSL-15K-K9 No - Unused -ACE-SSL-20K-K9 No - Unused -ACE-SSL-UP1-K9 No - Unused -ACE-SSL-UP2-K9 No - Unused -ACE-SSL-UP3-K9 No - Unused -

ACE_module5/Admin# show license statusLicensed Feature Count------------------------------ -----SSL transactions per second 1000Virtualized contexts 250Module bandwidth in Gbps 4

You can also see the licenses that reside on the Flash disk by entering the following command:

ACE_module5/Admin# dir disk0:236 Oct 17 09:18:26 2006 ACE-SSL-05K-K9.lic <--------235 Oct 17 09:16:58 2006 ACE-VIRT-250.lic <--------

1024 Sep 28 19:11:11 2006 cv/1654606 Oct 26 12:56:16 2006 dplug

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 28

Page 29: Ace Troubleshooting

Usage for disk0: filesystem 2759552 bytes total used 8405120 bytes free 11164672 total bytes

In the above example, there is an SSL 5K TPS license on the Flash disk that has not yet been installed in the ACE.

To install the license, enter the following command:

ACE_module5/Admin# license install disk0:ACE-SSL-05K-K9.licInstalling license... doneACE_module5/Admin#

Configuring an ACL to Permit Input Traffic to the ACE

You must configure an ACL to allow the ACE to receive traffic. All traffic to the ACE is blocked until you do so.For example, to configure an ACL that permits all IP trafffic except from the 10.1.1.0 network, enter thefollowing commands:

ACE_module5/Admin(config)# access-list ACL1 extended deny ip 10.1.1.0 255.255.255.0 anyACE_module5/Admin(config)# access-list ACL1 extended permit ip any anyACE_module5/Admin(config)# interface vlan 100ACE_module5/Admin(config-if)# access-group input ACL1

Verifying that the ACE is Sending and Receiving Traffic

You can tell if traffic is reaching the ACE by using the show svclc module number traffic command on theCatalyst 6500 series switch or Cisco 7600 series router. This command displays counters (packets input andpackets output) that increase when the switch or router sends packets to or receives packets from the ACE.

cat6k# show svclc module 5 trafficACE module 5:

Specified interface is up line protocol is up (connected) Hardware is C6k 10000Mb 802.3, address is 0018.b9a6.9114 (bia 0018.b9a6.9114) MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 10Gb/s input flow-control is on, output flow-control is unsupported Last input never, output never, output hang never Last clearing of "show interface" counters 4d02h Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 1000 bits/sec, 2 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec

528888 packets input, 41329093 bytes, 0 no buffer <------- Received 469945 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 input packets with dribble condition detected

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 29

Page 30: Ace Troubleshooting

7776 packets output, 746361 bytes, 0 underruns <------- 0 output errors, 0 collisions, 0 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out

A trace of the Te?/1 (10-Gbps switch fabric interface, where ? = the module number) interface will show youwhether packets are arriving at the switch fabric interface (SFI).

Another useful command is the show cde health command on the ACE. This command shows the current state ofthe Classification Distribution Engine (CDE). The network processors (NP1 and NP2) are represented by IXP0and IXP1, respectively. You should not observe any drops, errors, or flow control issues in the output of thiscommand. If the Packets Received or the Packets Transmitted counters of the CDE Hyperion Interface are notincreasing, then packets are not coming into or going out of the ACE.

ACE_module5/Admin# show cde health

CDE BRCM INTERFACE======================Packets received 4933Packets transmitted 2437922Broadcom interface CRC error count 0BRCM VOQ status [empty] [not full]BRCM pull status [pulling]

CDE HYPERION INTERFACE======================Packets received 29913371 <-------Packets transmitted 8034 <-------Short packets drop count 0Fifo Full drop count 0Protocol error drop count 0FCS error drop count 0CRC error drop count 0Num times flow control triggered on hyp interface 0Num self generated multicast packets filtered 1880HYP IXP0 VOQ status [empty] [not full]HYP IXP1 VOQ status [empty] [not full]HYP SLOW VOQ status [empty] [not full]HYP tx pull status [pulling]

CDE IXP0 INTERFACE======================Packets received 784985Packets transmitted 27827116Num bad pkts recvd on fast spi channel0 0Num bad pkts recvd on slow spi channel8 0Num bad pkts recvd on fast spi channel2 0Num bad pkts recvd on slow spi channel4 0IXP0 Fast VOQ status [empty] [not full]IXP0 BRCM VOQ status [empty] [not full]IXP0 pull status [pulling]IXP0 spi src status [healthy]IXP0 spi snk status [healthy]

CDE1 SWITCH1 INTERFACE======================Packets received (hyp, ixp0) 4415Packets received (bcm) 1656608

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 30

Page 31: Ace Troubleshooting

Packets received (daughter card 0) 0Packets received (daughter card 1) 0Packets Errors received (hyp, ixp0) 0Packets Errors received (bcm) 0Packets Errors received (daughter card 0) 0Packets Errors received (daughter card 1) 0Packets transmitted (ixp1) 2089360Packets transmitted (nitrox) 0Packets Errors transmitted (ixp1) 0Packets Errors transmitted (nitrox) 0

CDE2 SWITCH2 INTERFACE======================Packets received (ixp1) 2089360Packets received (nitrox) 0Packets Errors received (ixp1) 0Packets Errors received (nitrox) 0Packets transmitted (hyp, ixp0) 4415Packets transmitted (broadcom) 1656608Packets transmitted (daughter card 0) 0Packets transmitted (daughter card 1) 0Packets Errors transmitted (ixp1) 0Packets Errors transmitted (nitrox) 0Packets Errors transmitted (daughter card 0) 0Packets Errors transmitted (daughter card 1) 0

CDE IXP1 INTERFACE======================Packets received 1661023Packets transmitted 2089360Num bad pkts recvd on fast spi channel0 0Num bad pkts recvd on slow spi channel8 0Num bad pkts recvd on fast spi channel2 0Num bad pkts recvd on slow spi channel4 0IXP1 Fast VOQ status [empty] [not full]IXP1 BRCM VOQ status [empty] [not full]IXP1 pull status [pulling]IXP1 spi src status [healthy]IXP1 spi snk status [healthy]

CDE NITROX INTERFACE======================Packets received 0Packets transmitted 0Num bad pkts recvd on fast spi channel0 0Num bad pkts recvd on slow spi channel8 0Num bad pkts recvd on fast spi channel2 0Num bad pkts recvd on slow spi channel4 0NTX Fast VOQ status [empty] [not full]NTX BRCM VOQ status [empty] [not full]NTX pull status [pulling]NTX spi src status [healthy]NTX spi snk status [healthy]== Backplane ==ITASCA_SYS_CNTL1 0x300 data 0x61f0000ITASCA_SYS_CNTL2 0x304 data 0x80c30000

You can also use the show interface command on the ACE to display traffic that is sent and received on theinterface for each VLAN that is configured on the ACE.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 31

Page 32: Ace Troubleshooting

ACE_module5/Admin# show interface

bvi2 is administratively down Hardware type is BVI MAC address is 00:18:b9:a6:91:15 Mode : unknown FT status is non-redundant Description:not set MTU: 1500 bytes Last cleared: never Alias IP address not set Peer IP address not set 0 unicast packets input, 0 bytes 0 multicast, 0 broadcast 0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops 0 unicast packets output, 0 bytes 0 multicast, 0 broadcast 0 output errors, 0 ignored

vlan100 is administratively down Hardware type is VLAN MAC address is 00:18:b9:a6:91:15 Mode : unknown FT status is non-redundant Description:not set MTU: 1500 bytes Last cleared: never Alias IP address not set Peer IP address not set Assigned from the Supervisor, up on Supervisor 0 unicast packets input, 0 bytes 0 multicast, 0 broadcast 0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops 0 unicast packets output, 0 bytes 0 multicast, 0 broadcast 0 output errors, 0 ignored

vlan130 is up Hardware type is VLAN MAC address is 00:18:b9:a6:91:15 Mode : routed IP address is 10.86.215.134 netmask is 255.255.255.0 FT status is non-redundant Description:not set MTU: 1500 bytes Last cleared: never Alias IP address not set Peer IP address not set Assigned from the Supervisor, up on Supervisor

59858 unicast packets input, 41711169 bytes <-------193118 multicast, 280789 broadcast <-------0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops <-------6260 unicast packets output, 785167 bytes <-------0 multicast, 1892 broadcast <-------0 output errors, 0 ignored <-------

Verifying to-the-ACE Traffic

Traffic that is destined to the ACE itself arrives at the control plane in one of the following ways:

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 32

Page 33: Ace Troubleshooting

Directly from the console connection• Directly from the supervisor engine connection• Traffic from the SFI that is forwarded by the CDE in the data plane•

Use the following commands to verify that traffic is going to and coming from the control plane.

ACE_module5/Admin# show netio stats

High Priority (Control) Normal Priority (Data)----------------------- ----------------------Net Rx Packets : 224 Net Rx Packets : 2521794Net Rx Bytes : 17528 Net Rx Bytes : 196704169Net Rx Unsupported L2 : 0 Net Rx Unsupported L2 : 0Net Rx Lock Errors : 0 Net Rx Lock Errors : 0Net Rx Interface Miss : 0 Net Rx Interface Miss : 2326290Net Rx No Arp Client : 0 Net Rx No Arp Client : 0Net Rx Alias Drops : 0 Net Rx Alias Drops : 0Net Rx Repl. Errors : 0 Net Rx Repl. Errors : 0Net Rx Repl. If Err : 0 Net Rx Repl. If Errs : 0Net Rx Internal Errs : 0 Net Rx Internal Errs : 0

Net Tx Packets : 0 Net Tx Packets : 5213Net Tx Bytes : 0 Net Tx Bytes : 414073Net Tx Lock Errors : 0 Net Tx Lock Errors : 0Net Tx Bad Context ID : 0 Net Tx Bad Context ID : 0Net Tx No Route Found : 0 Net Tx No Route Found : 0Net Tx No Adjacency : 0 Net Tx No Adjacency : 1Net Tx Invalid If ID : 0 Net Tx Invalid If ID : 0Net Tx If Down : 0 Net Tx If Down : 0Net Tx No Src Addr : 0 Net Tx No Src Addr : 0Net Tx No Encap : 0 Net Tx No Encap : 0Net Tx FIFO Errors : 0 Net Tx Fifo Errors : 0Net Tx No VMAC Errors : 0 Net Tx No VMAC Errors : 0

IPC Tx Packets : 76 IPC Tx Packets : 0IPC Tx Bytes : 17638 IPC Tx Bytes : 0IPC Tx Fifo Errors : 0 IPC Tx Fifo Errors : 0

Client Rx Queue Full : 0 Client Rx Queue Full : 0

Pseudo Rx Queue Full : 0 Pseudo Rx Queue Full : 0

ACE_module5/Admin# show fifo stats

High Priority (Control) Normal Priority (Data)----------------------- ----------------------Rx Packets : 224 Rx Packets : 2524886Rx Bytes : 17528 Rx Bytes : 196952927Rx DMA Errors : 0 Rx DMA Errors : 0Rx Drop Events : 0 Rx Drop Events : 0Rx Descr Errors : 0 Rx Descr Errors : 0Rx Bad Descrs : 0 Rx Bad Descrs : 0Rx Length Errors : 0 Rx Length Errors : 0

Tx Packets : 76 Tx Packets : 5241Tx Bytes : 17682 Tx Bytes : 464991Tx Drops : 0 Tx Drops : 0Tx DMA Errors : 0 Tx DMA Errors : 0

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 33

Page 34: Ace Troubleshooting

Tx SOP Errors : 0 Tx SOP Errors : 0

Global Errors-------------Rx Underflows : 0Rx Overflows : 0Tx Underflows : 0Tx Overflows : 0Resets : 0Zbuff alloc fail : 0

Interrupt Stats---------------Total Interrupt count : 2529603Rx Interrupt count : 2524302 Tx interrupt count : 5310

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 34

Page 35: Ace Troubleshooting

This article describes how to troubleshoot basic ACE boot issues.

Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference

Contents

1 Understanding ROMMON Mode and the ACE BootConfiguration

1.1 Setting the Boot Method from the ConfigurationRegister

1.2 Booting the ACE from the ROMMON Prompt♦ 1.3 Setting the BOOT Environment Variable♦ 1.4 Displaying the ACE Boot Configuration♦

2 Restarting the ACE2.1 Restarting the ACE from the ACE CLI♦ 2.2 Restarting the ACE from the Supervisor Engine♦

3 Establishing a Console Connection to the ACE• 4 Troubleshooting ACE Boot Problems•

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 35

Page 36: Ace Troubleshooting

Understanding ROMMON Mode and the ACE BootConfiguration

You can control how the ACE performs its boot process through either the ACE configuration mode or ROMMonitor (ROMMON) mode. ROMMON is the ROM-resident code that starts executing at power up, reset, orwhen a fatal exception occurs.

Two user-configurable parameters determine how the ACE boots:

Boot field in the configuration register (confreg)• BOOT environment variable•

The ACE enters ROMMON mode if it does not find a valid system image, if the Flash memory configuration iscorrupted, or if the configuration register is set to enter ROMMON mode.

Note: You can manually enter ROMMON mode by restarting the ACE and then pressing the Break key duringthe first 60 seconds of startup. If you are connected to the ACE through a terminal server, you canescape to the Telnet prompt and then enter the send break command to enter the ROMMON mode.

Setting the Boot Method from the Configuration Register

To change the configuration register settings and how the ACE boots from the CLI, use the followingconfiguration mode command:

config-register value

The value argument-supported entries are as follows:

0?ACE boots to the ROMMON prompt. The ACE remains in ROMMON mode at startup.•

1?ACE boots from the system image identified in the BOOT environment variable. If the ACEencounters an error or if the image is not valid, it will try the second image (if one is specified). If thesecond image also fails to boot, the ACE returns to ROMMON mode.

For example, to set configuration register to boot the system image identified in the BOOT environment variable,enter the following command:

ACE_module5/Admin(config)# config-register 1

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 36

Page 37: Ace Troubleshooting

Booting the ACE from the ROMMON Prompt

If you specify a value of 0 for the config-register command, this configuration register setting forces the ACE toenter the ROMMON mode upon a reload or power cycle of the ACE. The ACE remains in ROMMON mode untilyou identify the location of an image file to boot.

The ACE supports two methods of booting the module from the ROMMON prompt:

To manually change the configuration register setting in ROMMON mode, use the confreg commandfollowed by a value of 0 or 1.

To change the boot characteristics using onscreen prompts, use the confreg command without a value.•

To instruct the ACE to manually boot from a particular system image, use the confreg command and specify aconfiguration register value of 1. Identify the name of the system image file that the ACE uses to boot.

For example, to use the confreg command at the ROMMON prompt to instruct the ACE to boot from thec6ace-t1k9-mzg.3.0.0_A2_2_0.bin system image, enter the following command:

rommon 1 > confreg 1rommon 2 > BOOT=disk0:c6ace-t1k9-mzg.3.0.0_A2_2_0.binrommon 3 > sync

To instruct the ACE to automatically boot from the image specified in the BOOT variable, use the confregcommand without specifying a configuration register value to launch the Configuration Summary menu-basedutility. You can then instruct the ACE to boot from the system image identified in the BOOT environmentvariable. See the "Setting the BOOT Environment Variable" section.

For example, to use the confreg command to display the onscreen prompts for changing the boot characteristicsof the ACE and change the configuration register to boot from an image on disk0:, enter the following command:

rommon 4 > confreg

Configuration Summary (Virtual Configuration Register: 0x2000)enabled are:ignore system config infoconsole baud: 9600boot: the ROM monitor

do you wish to change the configuration? y/n [n]: ydisable "ignore system config info"? y/n [n]:change the boot characteristics? y/n [n]: yenter to boot: 0 = ROM Monitor 1 = boot file specified in BOOT variable[0]: 1

Configuration Summary (Virtual Configuration Register: 0x2001)enabled are:ignore system config infoconsole baud: 9600boot: the file specified in BOOT variable

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 37

Page 38: Ace Troubleshooting

do you wish to change the configuration? y/n [n]:You must reset/power cycle for new config to take effectrommon 7 > dir disk0:Directory of disk0:

23951 31071143 -rw- c6ace-t1k9-mzg.A2_2_0.bin2 74448896 -rw- TN-CONFIG4546 32505856 -rw- TN-CERTKEY-STORAGE6530 11534336 -rw- TN-LOGFILE7234 11534336 -rw- TN-HOME7938 209715200 -rw- TN-COREFILE20738 1048576 -rw- lkcddump22689 250 -rw- scripted_hm.txt24584 30337516 -rw- c6ace-t1k9-mz.A2_1_1.bin29540 1048640 -rw- ACE_FUR_BOOT_ROM.img.rel.2008Apr01_ver12129605 1048640 -rw- ACE_BOOT_ROM.img.rel.2008Apr01_ver121rommon 8 > BOOT=disk0:c6ace-t1k9-mzg.A2_2_0.binvariable name contains illegal (non-printable) charactersrommon 9 > sync

Setting the BOOT Environment Variable

The BOOT environment variable specifies a list of image files from which the ACE can boot at startup. To set theBOOT environment variable, use the boot system image: command. The syntax of this command is as follows:

boot system image:image_name

The image_name argument specifies the name of the system image file. If the file does not exist (for example, ifyou entered the wrong filename), then the filename is appended to the bootstring, and the "Warning: File notfound but still added in the bootstring" message appears. If the file does exist, but is not a valid image, the file isnot added to the bootstring, and the "Warning: file found but it is not a valid boot image" message appears.

For example, to set the BOOT environment variable, enter the following command:

ACE_module5/Admin(config)# boot system image:c6ace-t1k9-mzg.3.0.0_A2_2.0.bin

Displaying the ACE Boot Configuration

To display the current BOOT environment variable and configuration register setting, use the show bootvarcommand in Exec mode. For example, to display the BOOT environment variable settings, enter the followingcommand:

ACE_module5/Admin# show bootvar

BOOT variable = "disk0:c6ace-t1k9-mzg.3.0.0_A2_2_0.bin"Configuration register is 0x1

Restarting the ACE

You can reload the ACE directly from its CLI or reboot it by using the supervisor engine CLI. You may need to

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 38

Page 39: Ace Troubleshooting

reboot the ACE from the supervisor engine if you cannot reach the ACE through an external Telnet session or aconsole connection (for example, the ACE is remote).

Restarting the ACE from the ACE CLI

To reboot the ACE directly from its CLI and reload the configuration, use the reload command in Exec mode.The reload command reboots the ACE and performs a full power cycle of both the hardware and software. Thereset process can take several minutes. Any open connections with the ACE are dropped after you enter thereload command.

Caution: Configuration changes that are not written to the Flash partition are lost after a reload. Before rebooting,enter the copy running-config startup-config command in Exec mode to store the current configuration in Flashmemory. If you fail to save your configuration changes, the ACE reverts to its previous settings upon restarting.

When you enter the reload command, the ACE prompts you for confirmation and performs a cold restart of theACE:

ACE_module5/Admin# reload

This command will reboot the systemSave configurations for all the contexts. Save? [yes/no]: [yes]Generating configuration....running config of context Admin savedPerform system reload. [yes/no]: [yes]

Restarting the ACE from the Supervisor Engine

To restart the ACE from the supervisor engine CLI, use the hw-module command. The syntax of this command isas follows:

hw-module module mod_num reset

For example, to use the supervisor engine CLI to reset the ACE located in slot 5 of the chassis, enter the followingcommand:

cat6k# hw-module module 5 resetProceed with reload of module?[confirm]% reset issued for module 5

Establishing a Console Connection to the ACE

In case the ACE becomes unresponsive or you cannot boot the ACE using the reload command from the Admincontext, you can establish a direct serial connection between your terminal (laptop) and the ACE by making aserial connection to the console port on the front of the ACE. The console port is an asynchronous RS-232 serialport with an RJ-45 connector. Any device connected to this port must be capable of asynchronous transmission.Connection requires a terminal configured as 9600 baud, 8 data bits, 1 stop bit, no parity.

Note: Only the Admin context is accessible through the console port; all other contexts can be reached throughTelnet or SSH sessions.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 39

Page 40: Ace Troubleshooting

After you connect the terminal to the console port, use any terminal communications application to access theACE CLI. The following procedure uses HyperTerminal for Windows.

To access the ACE by using a direct serial connection, follow these steps:

1. Launch HyperTerminal. The Connection Description window appears.

2. Enter a name for your session in the Name field.

3. Click OK. The Connect To window appears.

4. From the drop-down list, select the COM port to which the device is connected.

5. Click OK. The Port Properties window appears.

6. Set the following port properties:

Baud Rate = 9600• Data Bits = 8• Flow Control = none• Parity = none• Stop Bits = 1•

7. Click OK to connect.

8. Press Enter to access the ACE login prompt.

switch login:

9. If the ACE does not find a valid software image on disk0: or if the ACE is configured to enter ROMMONmode upon booting up, the ROMMON prompt appears.

rommon 1>

Troubleshooting ACE Boot Problems

The ACE module receives power from the chassis back plane and boots up automatically when you insert themodule into the chassis. If your ACE does not boot up when you insert it into the chassis or when you enter thereload Exec mode command from the Admin context, you cannot Telnet to the ACE or establish a session fromthe supervisor engine. In these cases, use the following steps to troubleshoot the issue and boot the ACE:

1. Log in to the Catalyst 6500 series switch or the Cisco 7600 series router and check the status of the ACE byentering the following command:

User Access Verification

Password:cat6k>enablePassword:cat6k# show module 5Mod Ports Card Type Model Serial No.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 40

Page 41: Ace Troubleshooting

--- ----- -------------------------------------- ------------------ ----------- 5 1 Application Control Engine Module ACE10-6500-K9 SAD1031044S <------- Module is receiving power

Mod MAC addresses Hw Fw Sw Status--- ---------------------------------- ------ ------------ ------------ ------- 5 0018.b9a6.9114 to 0018.b9a6.911b 1.1 Unknown Unknown Other <------- Firmware and software image status is Unknown

Mod Online Diag Status---- ------------------- 5 Unknown <------- Diagnostics status is Unknown

The first row of information is populated, so you know that the ACE is powered up. The firmware and softwareversions are Unknown and the Status is Other. At this point, you cannot session into the ACE from the supervisorengine.

2. Power cycle the ACE from the supervisor engine to attempt to boot the ACE by entering the followingcommands:

cat6k# config tEnter configuration commands, one per line. End with CNTL/Z.cat6k(config)# no power enable module 5cat6k(config)# power enable module 5

Wait long enough for the ACE to boot up. Try to Telnet or session to the ACE. If you still cannot Telnet orsession to the ACE, continue with Step 3.

3. Establish a console connection to the ACE. For details about establishing a console connection to the ACE, seethe "Establishing a Console Connection to the ACE" section.

rommon 1>

4. Check the ACE configuration register (confreg) by entering the following command:

rommon 2> confregConfiguration Summary (Virtual Configuration Register: 0x1) enabled are: console baud: 9600 boot: the file specified in BOOT variable

A value of 0x1 instructs the ACE to boot from the image in disk0:. A value 0x0 instructs the ACE to boot to theROMMON prompt. If the image specified in the BOOT variable is not in disk0:, then the ACE boots to theROMMON prompt as shown in this example issue.

5. Check the BOOT variable by entering the following command:

rommon 3> setPS1=rommon ! >RELOAD_REASON=reload command by adminBOOT=disk0:c6ace-t1k9-mz.3.0.0_A2_2_0.binARGV0=quiet?=0

6. Ensure that the software image specified in the BOOT variable is present in disk0: by entering the followingcommand:

rommon 4> dir disk0:

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 41

Page 42: Ace Troubleshooting

31071143 Dec 1 17:01:06 2008 c6ace-t1k9-mzg.A2_2_0.bin 250 Feb 8 20:04:44 2008 scripted_hm.txt 30337516 Jul 31 05:47:42 2008 c6ace-t1k9-mz.A2_1_3.bin 1048640 Aug 8 11:45:06 2008 ACE_FUR_BOOT_ROM.img.rel.2008Apr01_ver121 1048640 Aug 8 13:27:32 2008 ACE_BOOT_ROM.img.rel.2008Apr01_ver121

Usage for image: filesystem 506789888 bytes total used 517210112 bytes free 1024000000 total bytes

7. If the specified image is not in disk0:, then you can boot from another image in disk0: by entering the followingcommand:

rommon 5> boot system disk0:image_name

8. If there is no image on the ACE disk0: to boot from, you can still boot from the supervisor engine. Copy theimage to the supervisor engine's disk0: or disk1:, and then from the supervisor CLI, enter the following command:

cat6k(config)# boot device module slot_number disk[0 | 1]:image_name

The ACE boots and stops at the ROMMON prompt.

9. At the ROMMON prompt on the ACE console, enter the following command to boot the ACE from theEthernet Out-of-Band Channel (EOBC) between the ACE and the Catalyst 6500 series switch or the Cisco 7600series router:

rommon 6> boot eobc:

10. If the ACE is not local or you cannot establish a console connection for any other reason, use the followingprocedure to finish booting the ACE from the supervisor engine with the boot eobc: command:

cat6k# remote login switchTrying Switch ...Entering CONSOLE for Switch

cat6k-sp# svclc console 5Entering svclc ROMMON of slot 5 ...Type "end" to end the session.

rommon 7> boot eobc:

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 42

Page 43: Ace Troubleshooting

This article describes the ACE system logging facility, how to enable logging, and how to use system messages astroubleshooting tools.

Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference

Contents

1 Overview of ACE SystemLogging

2 Enabling ACE Logging• 3 Logging Severity Levels• 4 Adding Information tosyslogs

5 Troubleshooting ACELogging

5.1 Displaying LoggingStatistics

5.2 Displaying theLogging History

5.3 Displaying LoggingMessages

5.4 Displaying LoggingPersistence

5.5 Displaying theLogging Rate Limit

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 43

Page 44: Ace Troubleshooting

Overview of ACE System Logging

The ACE provides a system logging (syslog) facility that collects and saves system messages and outputs them todestinations that you specify as follows:

Buffer• Console• Flash memory• Host (remote syslog server)• Monitor• Standby• Supervisor•

Each virtual context generates logs independently from the other virtual contexts. The admin virtual context doesnot log on behalf of the other contexts in the ACE.

The ACE logs the following connection setup and teardown messages in the CP at the connection speed: 106023,302022, 302023, 302024, and 302025. These setup and teardown syslogs are directly forwarded to a remoteserver. Because of the potentially large number of these messages or if you require high-rate system logging ofconnection setup and teardown messages, use the logging fastpath command. This command disables CP syslogsand enables logging of these messages through the DP using a slightly different format and the following syslogIDs: 106028, 302028, 302029, 302030, and 302031, respectively. You can log these messages through the fastpath only to an external syslog server. All other enabled logging destinations are disabled by the logging fastpathcommand.

You can limit the rate at which the ACE generates syslog messages by using the logging rate-limit command.This command allows you to rate limit syslogs based on one of the following criteria:

Time interval• Logging level• Message ID•

Besides logging system messages, the ACE logs access control list (ACL) deny entries.

Note:

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 44

Page 45: Ace Troubleshooting

Remember to enable logging on the standby ACE in a redundant configuration by entering the loggingstandby command. To log failover information on the standby ACE, you need to set the logging levelto 4.

For details about ACE system message logging, see the Cisco Application Control Engine Module SystemMessage Guide (Software Version A2(1.0)).

Enabling ACE Logging

To enable logging on the ACE module and send syslogs to the monitor, enter the following commands:

ACE_module5/Admin(config)# logging enableACE_module5/Admin(config)# logging monitor 7ACE_module5/Admin(config)# logging trap 7ACE_module5/Admin(config)# no logging message 111008ACE_module5/Admin(config)# no logging message 111009ACE_module5/Admin(config)# logging timestampACE_module5/Admin(config)# do terminal monitor

Use the logging monitor severity_level command only when you are troubleshooting problems on the ACE orwhen there is minimal load on the network. Using this command at other times when the ACE is active maydegrade performance.

Note: logging trap defines the severity sent to the syslog server.

Note: If you do not see syslog messages on the console after enabling logging with the logging enable andlogging monitor 7 commands, log out of the ACE and then log in again.

To enable logging to a syslog server, use the following command syntax:

logging host ip_address [tcp | udp [/port#]] | [default-udp] | [format emblem]

Note: If you specify the default-udp option and TCP logging fails, the ACE sends logging messages overUDP.

You can verify that the ACE defaults to UDP by entering the following command:

ACE_module5/Admin# show logging

Syslog logging: enabledFacility: 20History logging: disabledTrap logging: enabled (level - debugging)Timestamp logging: disabledFastpath logging: disabledPersist logging: disabledStandby logging: disabledRate-limit logging: disabled (min - 0 max 100000 msgs/sec)Console logging: disabledMonitor logging: disabled Logging to 5.1.0.40 tcp/514 default-udp

(sending on UDP)Device ID: disabled

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 45

Page 46: Ace Troubleshooting

Message logging: noneBuffered logging: enabled (level - debugging) maximum size 681984Buffer info: current size - 681984 global pool - 1048576 used pool - 1048576 min - 0 max - 681984 cur ptr = 42894 wrapped - yes

Use the logging supervisor command to allow the aggregation of critical syslogs from multiple virtual devices tothe Catalyst 6500 series switch or to the Cisco 7600 series router syslog. For example, enter the followingcommand:

ACE_module5/Context(config)# logging supervisor ? <0-7> 0-emerg;1-alert;2-crit;3-err;4-warn; 5-notif;6-inform;7-debug

cat6k# show logging...cat6k#17w3d: %TRINITY-7-TRINITY_SYSLOG_DEBUG: %ACE-7-111009: User 'admin' executed cmd: show running.

Logging Severity Levels

The severity_level argument specifies the maximum level for system log messages sent to the console. Theseverity level that you specify indicates that you want syslog messages at that level and messages less than thelevel. For example, if the specified level is 3, the syslog displays level 3, 2, 1, and 0 messages. We recommendthat you use a lower severity level, such as 3, since logging at a high rate may impact the performance of theACE.

Allowable entries are as follows:

0?emergencies (System unusable messages)• 1?alerts (Take immediate action)• 2?critical (Critical condition)• 3?errors (Error message)• 4?warnings (Warning message)• 5?notifications (Normal but significant condition)• 6?informational (Information message)• 7?debugging (Debug messages)•

Adding Information to syslogs

After you have enabled system message logging and have specified a destination for the system messages, youcan add more information to the system messages that may be helpful in troubleshooting issues with your ACEmodule. For example, you can do the following:

Add a timestamp• Identify the messages sent to a syslog server• Identify the ACE device ID in messages that are sent to a syslog server•

To add a timestamp to syslog messages, enter the following command:

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 46

Page 47: Ace Troubleshooting

ACE_module5/Admin(config)# logging timestamp

To identify messages that are sent to a syslog server by severity level, enter the following command:

ACE_module5/Admin(config)# logging trap severity_level

For example, to identify the ACE device ID in messages that are sent to a syslog server, use the followingcommand syntax:

ACE_module5/Admin(config)# logging device-id {context-name | hostname | ipaddress interface_name | string text}

Troubleshooting ACE Logging

The commands in the following sections are useful for troubleshooting the system message logging facility.

Displaying Logging Statistics

ACE_module5/Admin# show logging statistics

Syslog statistics: sent 349 discarded 64

Messages sent: console 0 buffer 348 persistent 0 supervisor 1 history 0 monitor 0 host 0 misc 0

Messages discarded: cfg rate-limit 0 hard rate-limit 0 server down 5 queue full 59 errors 0

SNMP-related counters: notifications sent 0 history table flushed 0 messages ignored 0

NP-related counters: to-CP dropped 0 fastpath sent 0 fastpath dropped 0

ACE_module5/Admin# show logging queue

Logging Queue length limit : 80 msg(s), 59 msg(s) discarded. Current 0 msg on queue, 80 msgs most on queue

CP messages received: 426 , 59 msg(s) discarded.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 47

Page 48: Ace Troubleshooting

IXP messages received: 82 Xscale messages received: 0

System Max Queue size: 20080 System Free Queue size for allocation: 19920

In the above example, the ACE has discarded 59 control plane (CP) messages. By default, the syslog messagequeue can hold 80 messages. You can increase the size of the syslog message queue by using the logging queuecommand in configuration mode. Set the queue size before you start collecting syslog messages. When traffic isheavy, messages may be discarded if the queue size is too small. The maximum number of messages that thequeue can hold is 8192.

Displaying the Logging History

To display the ACE logging history, enter the following command from the console:

ACE_module5/Admin# show logging historysyslog_trinity_show_history for context 0: 1(Mar 24 2009 16:39:36): from "KERN" ACE-5-111008:User 'admin' executed the 'logging history 7' command. 2(Mar 24 2009 16:39:48): from "KERN" ACE-5-111008:User 'admin' executed the 'logging console 7' command. 3(Mar 24 2009 16:39:56): from "KERN" ACE-7-111009:User 'admin' executed cmd: do sho logging history 4(Mar 24 2009 16:49:50): from "KERN" ACE-7-111009:User 'admin' executed cmd: do show logging message 5(Mar 24 2009 16:51:35): from "KERN" ACE-4-405001:Received ARP REQUEST collision from 10.1.1.240 00.0c.29.74.51.fa on interface vlan100 6(Mar 24 2009 16:51:36): from "KERN" ACE-4-405001:Received ARP RESPONSE collision from 10.1.1.240 00.0b.fc.fe.1b.03 on interface vlan100 7(Mar 24 2009 16:51:40): from "KERN" ACE-4-405001:Received ARP REQUEST collision from 10.1.1.240 00.0c.29.74.51.fa on interface vlan100 8(Mar 24 2009 16:51:41): from "KERN" ACE-4-405001:Received ARP RESPONSE collision from 10.1.1.240 00.0b.fc.fe.1b.03 on interface vlan100 9(Mar 24 2009 16:54:46): from "KERN" ACE-7-111009:User 'admin' executed cmd: telnet 10.1.1.130

Displaying Logging Messages

If a particular system message does not appear in the syslog history, check that the message is enabled and thatthe logging level is correct for that message. To display the default logging level and the current status of a systemmessage, all system messages, or disabled system messages, enter the following command:

ACE_module5/Admin# show logging message message_id | all | disabled

For example, to display all disabled system messages in the ACE, enter the following command:

ACE_module5/Admin# show logging message disabledMessage logging: message 111008: default-level 5 (disabled) message 111009: default-level 7 (disabled)

Displaying Logging Persistence

The logging persistence command enables logging to disk0: in Flash memory. The messages are stored in asubdirectory of disk0: called /messages.

To display logging persistence, enter the following command:

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 48

Page 49: Ace Troubleshooting

ACE_module5/Admin# show logging persistentPersist info: current size - 6626 global pool - 1048576 used pool - 6626 min - 0 max - 189582 cur ptr = 6638 wrapped - no

Mar 24 2009 09:51:31 Admin: %ACE-4-405001: Received ARP REQUEST collision from 10.1.1.240 00.0c.29.74.51.fa on interface vlan100Mar 24 2009 09:51:32 Admin: %ACE-4-405001: Received ARP RESPONSE collision from 10.1.1.240 00.0b.fc.fe.1b.03 on interface vlan100Mar 24 2009 09:51:36 Admin: %ACE-4-405001: Received ARP REQUEST collision from 10.1.1.240 00.0c.29.74.51.fa on interface vlan100Mar 24 2009 09:51:37 Admin: %ACE-4-405001: Received ARP RESPONSE collision from 10.1.1.240 00.0b.fc.fe.1b.03 on interface vlan100

Displaying the Logging Rate Limit

To display the logging rate limit, enter the following command:

ACE_module5/Admin# show logging rate-limitRate-limit logging: (min - 0 max 100000 msgs/sec) 100000 messages 1 seconds level 7

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 49

Page 50: Ace Troubleshooting

This article describes how the ACE establishes connections and how to troubleshoot connectivity issues with yourACE.

Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference

Contents

1 Overview of ACE Connection Handling• 2 Internal Mapping of ACE TCP and UDPFlows

3 ACE Connection Table Entries• 4 Tracking Connections Through the ACE• 5 Troubleshooting Connections•

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 50

Page 51: Ace Troubleshooting

Overview of ACE Connection Handling

This article describes how the ACE handles connections at Layer 4 (L4) and Layer 7 (L7). For L4 connections,the ACE receives a TCP packet from a client and load balances the connection to a server on the first packet (seeFigure 1). The SYN-ACK from the server matches an existing flow and the rest of the connection is handled inthe fast path (hardware accelerated path in the network processors), which is represented here as "shortcut." TheACE completes the TCP handshake . This process applies to the following functions:

Basic load balancing• Source IP sticky• TCP/IP normalization•

Figure 1. Layer 4 Flow Setup

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 51

Page 52: Ace Troubleshooting

For L7 flows (for example, L7 load balancing, URL parsing, and generic TCP payload parsing), the ACE acts as aproxy (spoofs the server), intercepts the client's VIP request that matches an L7 rule, and terminates the TCPconnection. See Figure 2. The ACE sends a SYN-ACK to the client in response to the client's TCP SYN. Theclient responds with an ACK to complete the TCP handshake and an L7 request method (for example, HTTP GETor POST).

Figure 2. Layer 7 Flow Setup -- Client Connection

After the ACE receives the L7 information (for example, HTTP GET), it sets up the back-end connection to thereal server based on the load-balancing method and other criteria. See Figure 3.

Figure 3. Layer 7 Flow Setup -- Server Connection

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 52

Page 53: Ace Troubleshooting

Finally, the ACE unproxies the connection with the client and splices it together with the back-end connection tothe server. For the life of the HTTP flow, the client communicates directly with the server through the fast path(hardware-accelerated path in the network processors), which is depicted in the figures as "Shortcut." See Figures4.

Figure 4. Layer 7 Flow Setup -- Splicing the Flows Together

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 53

Page 54: Ace Troubleshooting

Figure 5 shows how the ACE adjusts the sequence numbers and ACK numbers when it splices the two flowstogether.

Figure 5. Layer 7 Flow Setup -- Adjusting the Sequence and ACK Numbers

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 54

Page 55: Ace Troubleshooting

With the persistence rebalance (connection keepalive) command configured, the ACE reproxies and parsessubsequent HTTP 1.1 requests over the same TCP connection. In this case, the ACE again spoofs the server andACKs the HTTP GET as shown in Figure 6. The sequence shown in Figure 2 through Figure 5 repeats for eachnew HTTP 1.1 request over the same TCP connection.

Figure 6. Layer 7 Flow Setup -- Reproxy

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 55

Page 56: Ace Troubleshooting

For SSL termination, TCP server reuse, application protocol inspection, and HTTP 1.1 pipelining, the ACE fullyterminates the client TCP connection. This connection remains fully proxied because the ACE is acting on behalfof the real server. For SSL termination, the ACE completes an SSL handshake after it establishes the TCPconnection with the server. See Figure 7.

Figure 7. SSL Handshake

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 56

Page 57: Ace Troubleshooting

For SSL termination, TCP server reuse, application protocol inspection, and HTTP 1.1 pipelining, the client andserver connections are completely independent and flows are handled in the software, not in the fast path. SeeFigure 8.

Figure 8. Layer 7 Flow Setup -- Full Proxy

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 57

Page 58: Ace Troubleshooting

Internal Mapping of ACE TCP and UDP Flows

The ACE maps TCP and UDP flows as two halves of the same flow: one input flow and one output flow. You candisplay the current connections in the ACE by entering the show connections command. See Figure 8.

Figure 8. Internal Flow Mapping

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 58

Page 59: Ace Troubleshooting

ACE Connection Table Entries

Understanding ACE?s Conn Table Entries During:

L4 TCP Connection Setup (3 Way Handshake)Normalisation Enabled♦ Normalisation Disabled♦

L7 TCP Connection Setup (3 Way Handshake)• TCP Connection Teardown

3 Way Handshake♦ 4 Way Handshake♦ Reset♦

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 59

Page 60: Ace Troubleshooting

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 60

Page 61: Ace Troubleshooting

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 61

Page 62: Ace Troubleshooting

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 62

Page 63: Ace Troubleshooting

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 63

Page 64: Ace Troubleshooting

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 64

Page 65: Ace Troubleshooting

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 65

Page 66: Ace Troubleshooting

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 66

Page 67: Ace Troubleshooting

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 67

Page 68: Ace Troubleshooting

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 68

Page 69: Ace Troubleshooting

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 69

Page 70: Ace Troubleshooting

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 70

Page 71: Ace Troubleshooting

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 71

Page 72: Ace Troubleshooting

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 72

Page 73: Ace Troubleshooting

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 73

Page 74: Ace Troubleshooting

Tracking Connections Through the ACE

You can display the IDs for the request and response connections in the ACE by entering the following command:

ACE_module5/Admin# show np 1 me-stats "-c 9"Connection ID:seq: 9[0x9].6 <------- Request and response connection ID Other ConnID : 3[0x3].4 Proxy ConnID : 0[0x0].0 Next Q : 1124073484[0x4300000c]

192.168.12.15:1985 -> 10.1.1.2:1985 [RX-NextHop: Drop] [TX-NextHop: TX] Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No L3 Protocol : IPv4 L4 Protocol : 17 Inbound Flag : 0 Interface Match : Yes Interface MatchID: 0 EncapsID:ver : 0:0 TCP ACK delta : 0x0 MSS : 0 TOS Stamp : 0 Repeat mode : No ARP Lookup : No TOS Stamp : No TCP Window Check: No ACE ID : 152 NAT Policy ID : 0 Post NAT hop : 0 Packet Count : 0 Byte Count : 0 TCP Information: (State = 0)

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 74

Page 75: Ace Troubleshooting

Window size : 0 Window scale : 0 FIN seen : No FIN/ACK seen : No FIN/ACK exp : No Close initiator : No FIN/ACK expval: 0 Last seq : 0 timestamp_delta: 0 Last ack : 1ce862 No Trigger : 0 Trigger Status : 0 Timestamp : 26ebc96d TCP options negotiated: Sack:Allow TS:Allow Windowscale: Allow Reserved: Allow Exceed MSS: Allow Window var: Allow

You can display both the front-end and the back-end connection statistics by entering the "-v" (verbose) option ofthe show np command as follows:

ACE_module5/Admin# show np 1 me-stats "-c 9 -v"Connection ID:seq: 9[0x9].2 Other ConnID : 7[0x7].14 Proxy ConnID : 0[0x0].0 Next Q : 0[0x0]

10.1.1.5:23 -> 172.27.16.143:4837 [RX-NextHop: TX] [TX-NextHop: CP] Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No L3 Protocol : IPv4 L4 Protocol : 6 Inbound Flag : 0 Interface Match : Yes Interface MatchID: 7 EncapsID:ver : 3:0 TCP ACK delta : 0x0 MSS : 1260 TOS Stamp : 0 Repeat mode : No ARP Lookup : No TOS Stamp : No TCP Window Check: No ACE ID : 148 NAT Policy ID : 0 Post NAT hop : 4 Packet Count : 347 Byte Count : 24476 TCP Information: (State = 3) Window size : 5840 Window scale : 0 FIN seen : No FIN/ACK seen : No FIN/ACK exp : No Close initiator : No FIN/ACK expval: 5b40000 Last seq : 53768a51 timestamp_delta: 0 Last ack : 658c1f72 No Trigger : 0 Trigger Status : 0 Timestamp : 459f781e TCP options negotiated: Sack:Clear TS:Clear Windowscale: Clear Reserved: Allow Exceed MSS: Deny Window var: Allow Flags: debug: 0 TCP Normalize: Yes Syslog: No Reproxy Request: No Policying Reqd: No Inbound IPSec: No Replicated: No Data Channel: No L7: No Fin Detect: Yes FP Timeout: No Standby: No ConnState: 2 ACA Method: 0 ReqTS: 00000000 RspTS: 00000000

Raw Connection Entry0000 0x00000000 0x0a56d786 0xa12c438f 0x062100070010 0x001712e5 0x00000000 0x00030000 0x04ec10040020 0x4e000007 0x00000000 0x00080480 0x244500000030 0x0000015b 0x00005f9c 0x16d00030 0x05b400000040 0x53768a51 0x658c1f72 0x459f781e 0x000000000050 0x00000094 0x00000000 0x45729985 0x000000000060 0x00000000 0x00000000 0x00000000 0x00000000

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 75

Page 76: Ace Troubleshooting

Doing verbose output for proxy id: 0

No valid proxy entry.No valid TCB proxy entry.No valid HTTP proxy entry.No valid SSL proxy entry.No valid AI proxy entry.Connection ID:seq: 7[0x7].14 Other ConnID : 9[0x9].2 Proxy ConnID : 0[0x0].0 Next Q : 0[0x0]

172.27.16.143:4837 -> 10.1.1.5:23 [RX-NextHop: CP] [TX-NextHop: TX] Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No L3 Protocol : IPv4 L4 Protocol : 6 Inbound Flag : 1 Interface Match : Yes Interface MatchID: 7 EncapsID:ver : 3:0 TCP ACK delta : 0x0 MSS : 1460 TOS Stamp : 0 Repeat mode : No ARP Lookup : No TOS Stamp : No TCP Window Check: No ACE ID : 148 NAT Policy ID : 0 Post NAT hop : 0 Packet Count : 486 Byte Count : 19810 TCP Information: (State = 3) Window size : 65371 Window scale : 0 FIN seen : No FIN/ACK seen : No FIN/ACK exp : No Close initiator : No FIN/ACK expval: 5b40000 Last seq : 658c1f72 timestamp_delta: 0 Last ack : 53768a51 No Trigger : 0 Trigger Status : 0 Timestamp : 459f781e TCP options negotiated: Sack:Clear TS:Clear Windowscale: Clear Reserved: Allow Exceed MSS: Deny Window var: Allow Flags: debug: 0 TCP Normalize: Yes Syslog: No Reproxy Request: No Policying Reqd: No Inbound IPSec: No Replicated: No Data Channel: No L7: No Fin Detect: Yes FP Timeout: No Standby: No ConnState: 2 ACA Method: 0 ReqTS: 00000000 RspTS: 00000000

Raw Connection Entry0000 0x00000000 0xa12c438f 0x0a56d786 0x06e900070010 0x12e50017 0x00000000 0x00030000 0x05b410000020 0x02000009 0x00000000 0x00080481 0x244500000030 0x000001e6 0x00004d62 0xff5b0030 0x05b400000040 0x658c1f72 0x53768a51 0x459f781e 0x000000000050 0x00000094 0x00000000 0x45729985 0x000000000060 0x00000000 0x00000000 0x00000000 0x00000000

Doing verbose output for proxy id: 0

No valid proxy entry.No valid TCB proxy entry.No valid HTTP proxy entry.No valid SSL proxy entry.No valid AI proxy entry.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 76

Page 77: Ace Troubleshooting

Troubleshooting Connections

To troubleshoot suspected connectivity issues, follow these steps:

1. Check the ACL hit count by entering the show access-list acl_name command. If the hit count is increasing, goto Step 2. Otherwise, verify that the access list is configured properly to permit traffic.

ACE_module5/Admin# show access-list anyone detailaccess-list:anyone, elements: 1, status: ACTIVE remark :access-list anyone line 8 extended permit ip any any (hitcount=3438) [0x44c2baf1] <------- Hit count

2. Check the service policy hit count by entering the show service-policy detail command. If the hit count is 0,verify that the service policy is active (show service-policy command) and the server farm is up (showserver-farm detail command). If the service policy is large, use the show service-policy policy_name summarycommand for more information as follows:

ACE_module5/Admin# show service-policy VIP summary

service-policy: VIPClass VIP Prot Port VLAN State Curr Conns Hit Count Conns DropVIP 192.168.12.192 tcp eq 443 100 IN-SRVC 0 0 0 192.168.12.192 tcp eq 80 100 IN-SRVC 0 0 0 192.168.12.193 tcp eq 80 100 IN-SRVC 0 0 0 192.168.12.193 tcp eq 443 100 IN-SRVC 0 0 0VIP2 192.168.12.194 tcp eq 80 100 IN-SRVC 0 0 0 192.168.12.194 tcp eq 443 100 IN-SRVC 0 0 0

3. Check the load-balancing statistics by entering the show stats loadbalance command. If the Layer 4 or Layer 7rejections or the Layer 4 or Layer 7 policy misses are increasing, check the configured class maps for anymisconfiguration.

ACE_module5/Admin# show stats loadbalance

+------------------------------------------+ +------- Loadbalance statistics -----------+ +------------------------------------------+ Total version mismatch : 0 Total Layer4 decisions : 0

Total Layer4 rejections : 3 <-------| Total Layer7 decisions : 0 |------- Failed connections due to traffic not matching the configured class maps

Total Layer7 rejections : 7 <-------| Total Layer4 LB policy misses : 0 Total Layer7 LB policy misses : 0

Total times rserver was unavailable : 10 <------- Failed connections due to no real server' Total ACL denied : 0 Total IDMap Lookup Failures : 0

To clear the load-balancing statistical information stored in the ACE buffer, enter the clear stats loadbalancecommand.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 77

Page 78: Ace Troubleshooting

4. If none of the error statistics is increasing, check the connection record by entering the show conn detailcommand and checking the connections for the affected VIP.

ACE_module5/Admin# show conn detail

total current connections : 6

conn-id np dir proto vlan source destination state----------+--+---+-----+----+---------------------+---------------------+------+7 1 in TCP 130 10.1.1.2:1171 10.1.1.134:23 ESTAB [ idle time : 00:00:00, byte count : 60055 ] [ elapsed time: 04:15:29, packet count: 1473 ]9 1 out TCP 130 10.1.1.134:23 10.1.2.74:1171 ESTAB [ conn in reuse pool : FALSE] [ idle time : 00:00:00, byte count : 64880 ] [ elapsed time: 04:15:29, packet count: 1086 ]

5. Display existing ACE connection statistics by entering the following command:

ACE_module5/Admin# show stats connection

+------------------------------------------++------- Connection statistics ------------++------------------------------------------+ Total Connections Created : 628950 Total Connections Current : 7 Total Connections Destroyed: 389 Total Connections Timed-out: 3958Total Connections Failed : 624596 <------- Server did not reply to a SYN within the pending timeout period or it replied with a RST

The Total Connection Failed counter increases when the ACE cannot set up the back-end connection with theserver. To clear the statistical information stored in the ACE buffer, enter the clear stats connection command.

6. Display service policy statistics by entering the following command:

ACE/Context# show service-policy client-vips detailStatus : ACTIVEDescription: ------------------------------------------Interface: vlan 211 service-policy: client-vips class: VIP-HTTPS VIP Address: Protocol: Port:

172.16.11.190 tcp eq 443 <------- Shows the VIP address, port, and protocol loadbalance: L7 loadbalance policy: HTTPS-POLICY VIP Route Metric : 77 VIP Route Advertise : DISABLED VIP ICMP Reply : ENABLED-WHEN-ACTIVE

VIP State: INSERVICE <------- Service is INSERVICE curr conns : 22 , hit count : 22 dropped conns : 0 client pkt count : 0 , client byte count: 0 server pkt count : 0 , server byte count: 0 max-conn-limit : 0 , drop-count : 0 conn-rate-limit : 0 , drop-count : 0 bandwidth-rate-limit : 0 , drop-count : 0 L7 Loadbalance policy : HTTPS-POLICY

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 78

Page 79: Ace Troubleshooting

class/match : class-default LB action : primary serverfarm: backend-ssl backup serverfarm : -

hit count : 22 <------- Shows the hit count dropped conns : 0

7. Display server farm connection statistics by entering the following command:

ACE/Context# show serverfarm HTTPS-FARM detailserverfarm : HTTPS-FARM, type: HOST total rservers : 4 active rservers: 4 description : - state : ACTIVEpredictor : ROUNDROBIN <------- Shows the load-balancing predictor that was used

failaction : - back-inservice : 0 partial-threshold : 0 num times failover : 0 num times back inservice : 0 total conn-dropcount : 0 --------------------------------- ----------connections----------- real weight state current total failures ---+---------------------+--------+---------------------+-----------+----------+--------- rserver: linux-1

192.168.1.11:0 8 OPERATIONAL 0 0 0 <------- Shows connection statistics foreach real server

max-conns : - , out-of-rotation count : - min-conns : - conn-rate-limit : - , out-of-rotation count : - bandwidth-rate-limit : - , out-of-rotation count : - retcode out-of-rotation count : -

The Connections Failures counter for a real server in a server farm may increment for one of the followingreasons:

SYN timeout (the three-way handshake fails to complete)• RST received (a client sends an RST to the server)• Internal exception (internal software issue)•

8. Display the statistics for a connection parameter map by entering the following command:

ACE_module5/Admin# show parameter-map CONN_PARAMMAP

Number of parameter-maps : 1

Parameter-map : CONN_PARAMMAP Type : connection nagle : disabled slow start : disabled buffer-share size : 32768 inactivity timeout (seconds) : TCP: 3600, UDP: 120, ICMP: 2 embryonic timeout (seconds) : 5 ack-delay (milliseconds) : 200 WAN Optimization RTT (milliseconds): 65535

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 79

Page 80: Ace Troubleshooting

half-closed timeout (seconds) : 3600 TOS rewrite : disabled syn retry count : 4 TCP MSS min : 0 TCP MSS max : 1460 tcp-options drop range : 0-0 tcp-options allow range : 0-0 tcp-options clear range : 1-255 selective-ack : clear timestamp : clear window-scale : clear window-scale factor : 0 reserved-bits : allow random-seq-num : enabled SYN data : drop exceed-mss : drop urgent-flag : allow conn-rate-limit : disabled bandwidth-rate-limit : disabled

9. Reset the ACE connection statistics by entering the following commands:

clear conn [all | flow {icmp | tcp | udp} | rserver server_name]• clear stats conn• clear tcp statistics• clear udp statistics•

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 80

Page 81: Ace Troubleshooting

This article describes how to troubleshoot issues involving ACE remote access.

Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference

Contents

1 Overview of ACE Remote Access• 2 Configuring a Management Policy for RemoteAccess

3 Troubleshooting Remote Access3.1 Troubleshooting Telnet♦ 3.2 Troubleshooting SSH♦ 3.3 Troubleshooting KAL-AP♦

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 81

Page 82: Ace Troubleshooting

Overview of ACE Remote Access

You can access the ACE remotely using several different protocols as follows:

HTTP• HTTPS• ICMP• KALAP-UDP• SSH• SNMP• Telnet•

These protocols require that you configure a management traffic policy on the ACE and associate that policy withthe interface that you intend to use for management traffic. A management policy allows the classification anddistribution engine (CDE) to classify (match) incoming management traffic to the management policy and toforward that traffic to the control plane (CP). For complete details about remote access, see the Cisco ApplicationControl Engine Module Administration Guide (Software Version A2(1.0)).

Configuring a Management Policy for Remote Access

To configure a management policy that allows remote access to the ACE using ICMP, SSH, or Telnet, enter thefollowing commands:

ACE_module5/Admin(config)# access-list ACL1 extended permit ip any anyACE_module5/Admin(config)# class-map type management match-any MGMT_CLASSACE_module5/Admin(config-cmap-mgmt)# 2 match protocol icmp anyACE_module5/Admin(config-cmap-mgmt)# 3 match protocol ssh anyACE_module5/Admin(config-cmap-mgmt)# 4 match protocol telnet anyACE_module5/Admin(config-cmap-mgmt)# exitACE_module5/Admin(config)# policy-map type management first-match MGMT_POLICYACE_module5/Admin(config-pmap-mgmt)# class MGMT_CLASSACE_module5/Admin(config-pmap-mgmt-c)# permitACE_module5/Admin(config-pmap-mgmt-c)# exitACE_module5/Admin(config-pmap-mgmt)# exitACE_module5/Admin(config)# interface vlan 100ACE_module5/Admin(config-if)# ip address 192.168.12.15 255.255.255.0ACE_module5/Admin(config-if)# service-policy input MGMT_POLICYACE_module5/Admin(config-if)# access-group input ACL1

Troubleshooting Remote Access

If you cannot access the ACE module remotely, follow these steps:

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 82

Page 83: Ace Troubleshooting

1. Beginning with ACE software release A2(1.1), by default, the ACE CLI is only locally accessible either usingthe ACE console port or through the supervisor by entering the session command. Remote access to the ACE (forexample, Telnet, SSH, and so on) is disabled until you change the admin user account password from the default.Access to the XML API is also disabled until you change the www user account password from the default. TheACE will display these warnings each time you access the CLI using the the console port or the supervisor untilyou change these passwords.

cat6k#session slot 5 processor 0The default escape character is Ctrl-^, then x.You can also type 'exit' at the remote prompt to end the sessionTrying 127.0.0.20 ... Open

ACE_module5 login: adminPassword:Cisco Application Control Software (ACSW)TAC support: http://www.cisco.com/tacCopyright (c) 2002-2009, Cisco Systems, Inc. All rights reserved.The copyrights to certain works contained herein are owned byother third parties and are used and distributed under license.Some parts of this software are covered under the GNU PublicLicense. A copy of the license is available athttp://www.gnu.org/licenses/gpl.html.Please change the password for admin user.Admin user is allowed to login only from supervisor until the password is changed.User 'www' is disabled. Please change the password to enable the user.

Use the following commands to change the passwords of the admin and www user accounts:

ACE_module5/Admin# configEnter configuration commands, one per line. End with CNTL/Z.ACE_module5/Admin(config)# username admin password 0 cisco123 role Admin domain default-domainACE_module5/Admin(config)# username www password 0 cisco123 role Admin domain default-domainACE_module5/Admin(config)# exit

Note that, although the passwords were entered in clear text above, they will be stored in the ACE configurationin an encrypted format:

ACE_module5/Admin# show run | i usernameGenerating configuration....username admin password 5 $1$M7gtcvBC$9ca78Q.ZH5jZpqDVuLnkN0 role Admin domain default-domainusername www password 5 $1$ulc7KHL5$2HlgNTEez03.ElmbiWKyY/ role Admin domain default-domain

2. Ensure that the remote access method protocol (for example, Telnet or SSH) that you are trying to use isconfigured in the management class map and that the management class has been permitted in the managementpolicy. If necessary, correct your ACE configuration. To display your management policy configuration elements,enter the following Exec mode commands:

ACE_module5/Admin# show running-config class-mapGenerating configuration....

class-map type management match-any MGMT_CLASS 2 match protocol icmp any 3 match protocol ssh any 4 match protocol telnet any

ACE_module5/Admin# show running-config policy-map MGMT_POLICYGenerating configuration....

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 83

Page 84: Ace Troubleshooting

policy-map type management first-match MGMT_POLICY class MGMT_CLASS permit class class-default

3. Ensure that the management policy is applied to the correct interface and that you are using the correct IPaddress for that interface. If necessary, correct your configuration. Enter the following command:

ACE_module5/Admin# show running-config interface

interface vlan 100 ip address 192.168.12.15 255.255.255.0 access-group input ACL1 access-group output ACL1 service-policy input MGMT_POLICY no shutdown

4. Check the status of the management interface by entering the following command:

ACE_module5/Admin# show interface vlan 100

vlan100 is up Hardware type is VLAN MAC address is 00:18:b9:a6:91:15 Mode : routed IP address is 192.168.12.15 netmask is 255.255.255.0 FT status is non-redundant Description:not set MTU: 1500 bytes Last cleared: never Alias IP address not set Peer IP address not set Assigned from the Supervisor, up on Supervisor 115303 unicast packets input, 74570169 bytes 273637 multicast, 521226 broadcast 0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops 12591 unicast packets output, 2120271 bytes 0 multicast, 4604 broadcast 0 output errors, 0 ignored

5. If the interface is down, ensure that the no shutdown command is configured on the interface to enable it. Ifnecessary, correct your configuration. Enter the following command:

ACE_module5/Admin# show running-config interface

6. Ensure that you have not exceeded the allocated resources for management connections or maximummanagement bandwidth by entering the following commands:

ACE_module5/Admin# show resource usage resource mgmt-connections Allocation Resource Current Peak Min Max Denied-------------------------------------------------------------------------------Context: Admin mgmt-connections 2 10 0 100000 0Context: C1 mgmt-connections 0 0 0 100000 0

ACE_module5/Admin# show resource usage resource rate mgmt-traffic Allocation

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 84

Page 85: Ace Troubleshooting

Resource Current Peak Min Max Denied-------------------------------------------------------------------------------Context: Admin mgmt-traffic rate 78 3588 0 125000000 0Context: C1 mgmt-traffic rate 0 0 0 125000000 0

7. If necessary, allocate more resources to management connections by entering the following command:

ACE_module5/Admin(config-resource)# limit-resource mgmt-connections minimum number maximum number

8. Ensure that traffic is reaching the CP by entering the following command:

ACE_module5/Admin# show netio stats

High Priority (Control) Normal Priority (Data)----------------------- ----------------------Net Rx Packets : 680 Net Rx Packets : 10799640 <-------Net Rx Bytes : 52902 Net Rx Bytes : 842376636 <-------Net Rx Unsupported L2 : 0 Net Rx Unsupported L2 : 0Net Rx Lock Errors : 0 Net Rx Lock Errors : 0Net Rx Interface Miss : 0 Net Rx Interface Miss : 10470926Net Rx No Arp Client : 0 Net Rx No Arp Client : 0Net Rx Alias Drops : 0 Net Rx Alias Drops : 0Net Rx Repl. Errors : 0 Net Rx Repl. Errors : 0Net Rx Repl. If Err : 0 Net Rx Repl. If Errs : 0Net Rx Internal Errs : 0 Net Rx Internal Errs : 0

Net Tx Packets : 0 Net Tx Packets : 10469 <-------Net Tx Bytes : 0 Net Tx Bytes : 1194879 <-------Net Tx Lock Errors : 0 Net Tx Lock Errors : 0Net Tx Bad Context ID : 0 Net Tx Bad Context ID : 0Net Tx No Route Found : 0 Net Tx No Route Found : 0Net Tx No Adjacency : 0 Net Tx No Adjacency : 0Net Tx Invalid If ID : 0 Net Tx Invalid If ID : 0Net Tx If Down : 0 Net Tx If Down : 0Net Tx No Src Addr : 0 Net Tx No Src Addr : 0Net Tx No Encap : 0 Net Tx No Encap : 0Net Tx FIFO Errors : 0 Net Tx Fifo Errors : 0Net Tx No VMAC Errors : 0 Net Tx No VMAC Errors : 0

IPC Tx Packets : 78 IPC Tx Packets : 0IPC Tx Bytes : 17766 IPC Tx Bytes : 0IPC Tx Fifo Errors : 0 IPC Tx Fifo Errors : 0

Client Rx Queue Full : 0 Client Rx Queue Full : 0

Pseudo Rx Queue Full : 0 Pseudo Rx Queue Full : 0

Management traffic is considered to-the-ACE traffic or CP traffic. If traffic is reaching the CP, the NormalPriority (Data) Net Rx Packets, Net Rx Bytes, Net TX packets, and Net TX bytes counters should be increasing. Ifnot, contact TAC.

9. If traffic is not arriving at the CP, ensure that traffic is reaching the classification and distribution engine (CDE)from the SFI by entering the following command:

ACE_module5/Admin# show cde health

CDE BRCM INTERFACE

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 85

Page 86: Ace Troubleshooting

======================Packets received 10580Packets transmitted 10802845Broadcom interface CRC error count 0BRCM VOQ status [empty] [not full]BRCM pull status [pulling]

CDE HYPERION INTERFACE======================Packets received 12312951 <-------Packets transmitted 17361 <-------Short packets drop count 0Fifo Full drop count 0Protocol error drop count 0FCS error drop count 0CRC error drop count 0Num times flow control triggered on hyp interface 0Num self generated multicast packets filtered 4618HYP IXP0 VOQ status [empty] [not full]HYP IXP1 VOQ status [empty] [not full]HYP SLOW VOQ status [empty] [not full]HYP tx pull status [pulling]

<snip>

If traffic is reaching the CDE, the Packets received and the CDE Hyperion Interface Packets transmitted countersshould be increasing. If not, contact TAC.

10. If packets are not reaching the CDE, ensure that the MSFC in the Catalyst 6500 series switch or the Cisco7600 series router is sending packets to the switch fabric interface (SFI) by entering the following command onthe supervisor engine:

cat6k# show svclc module 5 trafficACE module 5:

Specified interface is up line protocol is up (connected) Hardware is C6k 10000Mb 802.3, address is 0018.b9a6.9114 (bia 0018.b9a6.9114) MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 10Gb/s input flow-control is on, output flow-control is unsupported Last input never, output never, output hang never Last clearing of "show interface" counters 1w2d Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 1000 bits/sec, 2 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 912150 packets input, 74727962 bytes, 0 no buffer Received 796374 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 input packets with dribble condition detected 17390 packets output, 2145844 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 86

Page 87: Ace Troubleshooting

If the MSFC is sending traffic to the SFI, the packets input and the packets output counters should be increasing.If not, contact TAC.

Troubleshooting Telnet

If you cannot Telnet to the ACE, ensure that you have not reached the maximum connection limit for Telnet byentering the following commands:

ACE_module5/Admin# show telnet

Session ID Remote Host Active Time4254 127.0.0.51 :41985 0: 8:13

The show telnet command output shows only one Telnet session. A maximum of 15 more users can potentiallyTelnet to the Admin context.

To display the maximum number of users allowed to Telnet to a particular context, enter the following command:

ACE_module5/Admin# show telnet maxsessions

Maximum Sessions Allowed is 16

Troubleshooting SSH

If you attempt to connect to the ACE using SSH and receive the following error, follow these steps:

[linux]$ ssh [email protected]_exchange_identification: Connection closed by remote host [linux]$

1. Ensure that SSH is enabled in the management policy by entering the following command:

ACE_module5/Admin# show running-config class-map

class-map type management match-any MGMT_CLASS 2 match protocol http any 3 match protocol https any 4 match protocol icmp any

6 match protocol ssh any <------- SSH is enabled 7 match protocol telnet any 8 match protocol snmp any

switch/Admin# show running-config policy-map MGMT_POLICYGenerating configuration....

policy-map type management first-match MGMT_POLICY class MGMT_CLASS

permit <------- All protocols in the MGMT_CLASS class-map are permitted including SSH

2. Ensure that the SSH key has been generated by entering the following command:

switch/Admin# show ssh key**************************************could not retrieve rsa1 key information

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 87

Page 88: Ace Troubleshooting

**************************************could not retrieve rsa key information**************************************could not retrieve dsa key information**************************************no ssh keys present. you will have to generate them**************************************

The show ssh key command output shows that no SSH key has been generated.

3. Generate an SSH key based upon your security requirements by entering the following commands:

ACE_module5/Admin# configEnter configuration commands, one per line. End with CNTL/Z.ACE_module5/Admin(config)# ssh key rsa 4096generating rsa key(4096 bits)..............................generated rsa keyACE_module5/Admin(config)# exitswitch/Admin# show ssh keydsa rsa rsa1ACE_module5/Admin# show ssh key**************************************could not retrieve rsa1 key information**************************************rsa Keys generated:Tue Apr 7 15:55:20 2009

ssh-rsaAAAAB3NzaC1yc2EAAAABIwAAAgEAr3z0MO0knoS6YwntSxUkWDWjHZfFE7Y6nLd8qQVcPMu0XpvkabDswLwoEdC9nOWM4v4g4PDpUz+tk+2WHJ4MMgRVeomVbK/2+Zx0Eds1p2XlhiV9KPcVpflpNNt63Mr01oLHoHpjxJ8ubfJJ+gPhMoBmQyGKedQOk5tVlbOpyxS2f7yWGqzF26AzXTFFdS0xYEcN4GrtziduBlh4TYRxk99JR13ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+9C0iZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+uowKZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+gHaaZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+ryooZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+iKshZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+/cutZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+7OvNZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+w6igZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+msEFZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+XvRoZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+w34sZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+mbRkZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+e9p2ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+PEAQZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+D+TyZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+GsLzZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+EaKaZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+MNQmZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+VVkVZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+4d21ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+x6VHZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+fEGTZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+64zMZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+Hf/aZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+e9TuZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+W4nDZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+7SUPZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+leL+ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+TmwtZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+8ymk

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 88

Page 89: Ace Troubleshooting

ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+20L0ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+ym2AZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+mfe5ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+A+PyZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+k/DxZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+NeCFZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+mSK1ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+V747ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+M0xGZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+QiEPZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+CCTjZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+8lkbZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+GXOoZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+C094ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+eDPMZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+F+xpZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+eNjzsFzc1023QU8xwlrhgvL0xXKxUmITV4y2VaSmEc/0RH/c4XAinNy955X6w9ejqXG9aYFjljLBtCHKXgyAZbQ48E4UPtPwPTCC3R1pFapfNAJMQ55kfGqEjQQda/6x/6XVFPBw4okLcz6tVDLqn7dGiTEjzYgfQBwKXiIPrGg9EmBgwmQKWuJpOde+jbMD9kU1WmUoUWvTvPtXQ0=

bitcount:4096fingerprint:13:96:fe:f0:f7:d7:5f:9c:d7:2a:da:72:8a:93:53:a6**************************************could not retrieve dsa key information**************************************

Now SSH should work.

4. Try connecting to the ACE via SSH again by entering the following command:

[linux]$ ssh [email protected]: Permanently added '192.168.0.210' (RSA) to the list of known hosts.

Password:Cisco Application Control Software (ACSW) TAC support: http://www.cisco.com/tac Copyright (c) 1985-2009 by Cisco Systems, Inc. All rights reserved.The copyrights to certain works contained herein are owned by other third parties and are used and distributed under license.Some parts of this software are covered under the GNU Public License. A copy of the license is available at http://www.gnu.org/licenses/gpl.html.ACE_module5/Admin#

5. Confirm the SSH session from the ACE CLI by entering the following command:

ACE_module5/Admin# show ssh session-info

Session ID Remote Host Active Time6986 10.76.248.6 :42116 0: 0:46

Troubleshooting KAL-AP

To troubleshoot KAL-AP related issues, follow these steps:

1. Make sure that KAL-AP is enabled under the management policy by entering the following commands:

ACE_module5/Admin# show running-config class-mapGenerating configuration....

class-map type management match-any MGMT_CLASS 2 match protocol http any

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 89

Page 90: Ace Troubleshooting

3 match protocol https any 4 match protocol icmp any

5 match protocol kalap-udp any <------- KAL-AP is enabled 6 match protocol ssh any 7 match protocol telnet any 8 match protocol snmp any

ACE_module5/Admin# show running-config policy-map MGMT_POLICYGenerating configuration....

policy-map type management first-match MGMT_POLICY class MGMT_CLASS

permit <------- All protocols in the MGMT_CLASS class-map are permitted including SSH

2. Verify that traffic from the Cisco Global Site Selector (GSS) is reaching the ACE module. KAL-AP statisticsshould get incremented.

ACE_module5/Admin# sh stats kalap

+-----------------------------------------------------++---------------- KAL-AP(UDP) statistics -------------++-----------------------------------------------------+

Total bytes received : 243956Total bytes sent : 184884Total requests received : 5100Total responses sent : 5100Total requests successfully received : 5100Total responses successfully sent : 5100Total secure requests received : 0Total secure responses sent : 0Total requests with errors : 0Total requests with parse errors : 0Total response transfer errors : 0-----------------------------------------------------

3. Allow secure KAL-AP requests, and add the GSS IP address and the shared secret to the ACE by entering thefollowing commands:

ACE_module5/Admin# configACE_module5/Admin(config)# kalap udpACE_module5/Admin(config-kalap-udp)# ip address 192.168.10.52 encryption md5 cisco (GSS IP)

4. Display information about the load VIP by entering the following command:

ACE_module5/Admin# show kalap udp load vip 10.1.1.1

Error: Vip object not found!ACE_module5/Admin#

If the VIP object is not found while displaying the load value as shown above, check whether the VIP gotdownloaded in the configuration manager internal table by entering the following command:

ACE_module5/Admin# show cfgmgr internal table vip

VIP-Id VIP-Addr Ctx-Id Flags---------------------------------------------------------------------------1 10.1.1.1 1 DATA_VALID,

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 90

Page 91: Ace Troubleshooting

L3Rule_list :-->: 41: 42Load Value: 255 Load Time stamp: Wed Apr 8 05:10:20 2009

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 91

Page 92: Ace Troubleshooting

This article describes security access control lists (ACLs) in the ACE, how to configure them, and troubleshootingsteps to follow if you encounter problems with ACLs.

Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference

Contents

1 Overview of Security AccessControl Lists

1.1 ACL Types and Uses♦ 1.2 ACL ConfigurationGuidelines

1.2.1 ACL EntryOrder

1.2.2 ACL ImplicitDeny

1.2.3 MaximumNumber of ACLEntries

2 Configuring ACLs• 3 ACL-Related syslogs• 4 Troubleshooting ACLs•

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 92

Page 93: Ace Troubleshooting

Overview of Security Access Control Lists

An ACL consists of a series of statements called ACL entries that define the network traffic profile. Each entrypermits or denies network traffic (inbound and outbound) from and to the parts of your network specified in theentry. Each entry also contains a filter element that is based on criteria such as the source address, the destinationaddress, the protocol, and protocol-specific parameters such as ports and so on.

An implicit deny-all entry exists at the end of each ACL, so you must configure an ACL on each interface thatyou want to permit connections. Otherwise, the ACE denies all traffic on the interface.

ACLs allow you to control network connection setups rather than processing each packet. Such ACLs arecommonly referred to as security ACLs. You can configure ACLs as parts of other features (for example,security, Network Address Translation (NAT), server load balancing (SLB), and so on). The ACE merges theseindividual ACLs into one large ACL called a merged ACL. The ACL compiler then parses the merged ACL andgenerates the ACL lookup mechanisms. A match on this merged ACL can result in multiple actions.

Note: You can apply only one extended ACL to each direction (inbound or outbound) of an interface. You canalso apply the same ACL on multiple interfaces. You can apply EtherType ACLs only in the inbounddirection and only on Layer 2 interfaces.

ACL Types and Uses

You can configure the following two types of ACLs in the ACE:

Extended?Control network access for IP traffic (Layer 3 and Layer 4)• EtherType?Control network access for non-IP traffic on Layer 2 interfaces•

The ACE does not explicitly support standard ACLs. To configure a standard ACL, specify the destinationaddress as any and do not specify ports in an extended ACL. For details about configuring an extended ACL, seethe ?Configuring an Extended ACL? section.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 93

Page 94: Ace Troubleshooting

ACL Configuration Guidelines

This section describes the guidelines to observe when you configure and use ACLs in your network. It containsthe following topics:

ACL Entry Order• ACL Implicit Deny• Maximum Number of ACL Entries•

ACL Entry Order

An ACL consists of one or more entries. Depending on the ACL type, you can specify the source and destinationaddresses, the protocol, the ports (for TCP or UDP), the ICMP type, the ICMP code, or the EtherType as thematch criteria. By default, the ACE appends each ACL entry at the end of the ACL. You can also indicate thelocation of each entry within an ACL by specifying a line number.

The order of the entries is important. When the ACE decides whether to accept or refuse a connection, the ACEtests the packet against each ACL entry in the order in which the entries are listed. After it finds a match, the ACEdoes not check any more entries. For example, if you create an entry at the beginning of an ACL that explicitlypermits all traffic, the ACE does not check any other statements in the ACL.

Note: If there is a deny statement for packets coming to the Control Plane (CP), then the ACE skips to the nextACL entry.

ACL Implicit Deny

All ACLs have an implicit deny entry at the end of the ACL, so, unless you explicitly permit it, traffic cannotpass. For example, if you want to allow all users to access a network through the ACE except for those users withparticular IP addresses, then you must deny the particular IP addresses in one entry and permit all other IPaddresses in another entry.

Maximum Number of ACL Entries

The ACE supports a maximum of 256,000 ACL entries. Some ACLs use more memory than others, such as anACL that uses large port number ranges or overlapping networks (for example, one entry specifies 10.0.0.0/8 andanother entry specifies 10.1.1.0/24). Depending on the type of ACL, the actual limit that the ACE can supportmay be less than 256,000 entries.

If you use object groups in ACL entries, you enter fewer actual ACL entries, but the same number of expandedACL entries as you did when you entered entries without object groups. Expanded ACL entries count toward thesystem limit. To view the number of expanded ACL entries in an ACL, use the show access-list name command.

If you exceed the memory limitations of the ACE, it generates a syslog message and increments the DownloadFailures counter in the output of the show interface vlan number command. The configuration remains in therunning-config file and the interface stays enabled. The ACL entries stay the same as they were before the failingconfiguration was attempted.

For example, if you add a new ACL with ten entries, but the addition of the sixth entry fails because the ACE runs

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 94

Page 95: Ace Troubleshooting

out of memory, the ACE removes the five entries that you successfully entered.

Note: You must allocate sufficient ACL memory resources for each virtual context in the ACE. The ACE doesnot generate a syslog if you exceed the maximum number of ACL entries.

Configuring ACLs

You can configure ACLs in one of two ways:

Using the access-list command in configuration mode• Using the match access-list command in a Layer 3 and Layer 4 class map•

You can permit or deny network connections based on the IP protocol, source and destination IP addresses, andTCP or UDP ports. To configure a non-ICMP extended ACL, enter the following command:

access-list name [line number] extended {deny | permit} {protocol {any | host src_ip_address | src_ip_addressnetmask | object-group net_obj_grp_name} [operator port1 [port2]] {any | host dest_ip_address |dest_ip_address netmask | object-group net_obj_grp_name} [operator port3 [port4]]} | {object-groupservice_obj_grp_name} {any | host src_ip_address | src_ip_address netmask | object-group net_obj_grp_name}{any | host dest_ip_address | dest_ip_address netmask | object-group net_obj_grp_name}

You can also permit or deny network connections based on the ICMP type (for example, echo, echo-reply,unreachable, and so on). To configure an ICMP extended ACL, enter the following command:

access-list name [line number] extended {deny | permit} {icmp {any | host src_ip_address | src_ip_addressnetmask | object-group net_obj_grp_name} {any | host dest_ip_address | dest_ip_address netmask |object-group net_obj_grp_name} [icmp-type code [operator code1 [code2]]]} | {object-groupservice_obj_grp_name} {any | host src_ip_address | src_ip_address netmask | object-group net_obj_grp_name}{any | host dest_ip_address | dest_ip_address netmask | object-group net_obj_grp_name}

You can configure an ACL that controls traffic based on its EtherType. An EtherType is a subprotocol identifier.EtherType ACLs support Ethernet V2 frames; they do not do not support 802.3-formatted frames. To configurean Ethertype ACL, enter the following command:

access-list name ethertype {deny | permit} {any | bpdu | ipv6 | mpls}

Note: You can configure an EtherType ACL on a Layer 2 interface in the inbound direction only. If you areoperating the ACE in bridge mode, be sure to configure an ACL on all interfaces that permit BPDUs.Otherwise, a bridge loop may result.

For example, to configure an extended ACL to permit all IP traffic from any source IP address and that is destinedto any IP address on interface VLAN 200, enter the following commands:

ACE_module5/Admin(config)# access-list ACL1 extended permit ip any anyACE_module5/Admin(config)# interface vlan 200ACE_module5/Admin(config-if)# ip address 192.168.1.1 255.255.255.0ACE_module5/Admin(config)# access-group input ACL1

You can apply an ACL to all interfaces in a context at once, subject to the following conditions:

No interface in the context has an ACL applied to it.•

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 95

Page 96: Ace Troubleshooting

You can globally apply one Layer 2 and one Layer 3 ACL in the inbound direction only.• On Layer 2 bridged-group virtual interfaces (BVIs), you can apply both Layer 3 and Layer 2 ACLs.• On Layer 3 virtual LAN (VLAN) interfaces, you can apply only Layer 3 ACLs.• In a redundant configuration, the ACE does not apply a global ACL to the FT VLAN.•

For example, to apply ACL1 to all interfaces in the Admin context, enter the following command in configurationmode:

ACE_module5/Admin(config)# access-group input ACL1

The syntax of the match access-list command is as follows:

match access-list acl_name

To configure an ACL match statement in a class map, enter the following commands:

ACE_module5/Admin(config)# class-map match-any L4_CLASSACE_module5/Admin(config-cmap)# match access-list ACL1ACE_module5/Admin(config-cmap)# exitACE_module5/Admin(config)# policy-map multi-match L4_POLICYACE_module5/Admin(config-pmap)# class L4_CLASSACE_module5/Admin(config-pmap-c)#

For more details about ACLs and how to configure them, see the Cisco Application Control Engine ModuleSecurity Configuration Guide.

ACL-Related syslogs

When a packet matches an ACL entry, a syslog message is generated based on the following rules:

All ACL deny entries generate a syslog message unless logging is explicitly disabled using the nologging enable command in configuration mode.

An ACL permit entry generates a syslog message only if logging is enabled using the logging enablecommand in configuration mode.

All implicit deny entries generate the default deny syslog (%ACE-4-106023).•

To minimize syslog message generation, the ACE uses the flow cache as follows:

For the first packet hit on an ACL entry, the ACE generates a syslog and caches the flow (5-tuple) in theconnection table.

1.

For subsequent hits on the same ACL entry, the ACE checks the cache. If it finds the flow in the cache,the ACE increments a hit counter for this entry in the cache and does not generate a syslog.

2.

After some time (the default is 300 seconds, which is configurable in the ACL entry definition in the CLIas the interval_secs option), the ACE generates a syslog and sets the hit count to 0.

3.

However, if at the expiry of the above time, the hit count is 0, the ACE deletes the cache entry silently. Soby default, a cache entry is aged out 600 seconds after the last hit.

4.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 96

Page 97: Ace Troubleshooting

Troubleshooting ACLs

Many ACL issues manifest themselves by all traffic or only certain traffic being denied or permitted access to theACE or out of the ACE. Remember that, initially, all traffic to the ACE is denied until you permit traffic using anACL. Every ACL contains an implicit deny at the end of it, so only traffic that you explicitly permit will haveaccess to the ACE. To troubleshoot ACLs, follow these steps:

1. Verify that your ACL configuration is correct for your network application. Make any required changes to therunning-config file, and then test the configuration. If it is satisfactory, save it to the startup-config file using thecopy runnning-config startup-config command.

For example, to display the ACLs that you have configured in your ACE, enter the following command:

ACE_module5/Admin# show running-config access-listGenerating configuration....

access-list ACL1 remark This ACL permits any IP traffic from any source going to any destination except for ICMP traffic originating from 192.168.12.15 255.255.255.192.access-list ACL1 line 8 extended permit ip any anyaccess-list ACL1 line 10 extended deny icmp 192.168.12.15 255.255.255.192 any echo code range 1 1 (hitcount=0) [0x65af0edd]access-list ANYONE line 8 extended permit ip any any

To verify that the configured ACLs are applied to the correct interfaces and in the right directions (input oroutput), enter the following command:

ACE_module5/Admin# show running-config interfaceGenerating configuration....

interface vlan 100 ip address 10.2.1.1 255.255.255.0 access-group input ANYONE access-group output ANYONE no shutdowninterface vlan 200 ip address 192.168.1.1 255.255.255.0 access-group input ACL1 service-policy input MGMT_POLICY no shutdown

2. Verify that you have allocated sufficient resources for ACLs. To display the allocated resources in your ACE,enter the following command:

ACE_module5/Admin# show resource usage Allocation Resource Current Peak Min Max Denied-------------------------------------------------------------------------------Context: Admin conc-connections 10 18 0 8000000 0 mgmt-connections 2 10 0 100000 0 proxy-connections 584 590 0 1048574 0 xlates 0 0 0 1048574 0 bandwidth 880 16194 0 625000000 0 throughput 880 12606 0 500000000 0 mgmt-traffic rate 0 3588 0 125000000 0

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 97

Page 98: Ace Troubleshooting

connection rate 1 21 0 1000000 0 ssl-connections rate 0 0 0 5000 0 mac-miss rate 0 16 0 2000 0 inspect-conn rate 0 0 0 6000 0

acl-memory 33448 33448 7858944 70749384 0 <------- ACL memory resource allocation statistics sticky 0 0 0 0 0 regexp 0 0 0 1048576 0 syslog buffer 188416 188416 0 4194304 0 syslog rate 0 9 0 100000 0Context: C1 conc-connections 0 0 0 8000000 0 mgmt-connections 0 0 0 100000 0 proxy-connections 0 0 0 1048574 0 xlates 0 0 0 1048574 0 bandwidth 0 0 0 625000000 0 throughput 0 0 0 500000000 0 mgmt-traffic rate 0 0 0 125000000 0 connection rate 0 0 0 1000000 0 ssl-connections rate 0 0 0 5000 0 mac-miss rate 0 0 0 2000 0 inspect-conn rate 0 0 0 6000 0 acl-memory 0 0 7858944 70749384 0 sticky 0 0 0 0 0 regexp 0 0 0 1048576 0 syslog buffer 0 0 0 4194304 0 syslog rate 0 0 0 100000 0

For example, to allocate a 10 percent minimum and a maximum of unlimited resources for ACL memory in theAdmin virtual context, enter the following commands:

ACE_module5/Admin(config)# resource myclassACE_module5/Admin(config-resource)# limit-resource acl-memory minimum 10 maximum unlimitedACE_module5/Admin(config-resource)# exitACE_module5/Admin(config)# context AdminACE_module5/Admin(config-context)# member myclass

3. Display the details of an individual ACL by using the show access-list acl_name detail command. Thiscommand displays every entry in the specified ACL, the hit counts for each entry, and a 32-bit hexadecimalMD5-hash value that the ACE computes from the access-list command immediately when you configure an ACL.The ACE includes this hash value in deny syslog messages (106023) to help you identify the ACL entry thatcaused the deny syslog. For example to display the details of the ACL1 access control list, enter the followingcommand:

ACE_module5/Admin# show access-list ACL1 detail

access-list:ACL1, elements: 2, status: ACTIVE remark : This ACL permits any IP traffic from any source going to any destination except for ICMP traffic originating from 192.168.12.15 255.255.255.1.access-list ACL1 line 8 extended permit ip any any (hitcount=9) [0x894c1008] <------- 32-bit hexadecimal MD5-hash valueaccess-list ACL1 line 10 extended deny icmp 192.168.12.15 255.255.255.192 any echo code range 1 1 (hitcount=15) [0x65af0edd]

The format of the deny syslog message is as follows:

%ACE-4-106023: Deny protocol number | name src incoming-interface:src-ip dst outgoing-interface:dst-ip by access-group "acl-name"An IP packet was denied by the ACL.

Explanation: This message displays even if you do not have the log option enabled for an ACL. If apacket hits an input ACL, the outgoing interface will not be known. In this case, the ACE prints the

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 98

Page 99: Ace Troubleshooting

outgoing interface as undetermined. The source IP and destination IP addresses are the unmapped andmapped addresses for the input and output ACLs, respectively, when used with NAT.

Recommended Action: If messages persist from the same source address, messages may indicate afoot-printing or port-scanning attempt. Contact the remote host administrators.

An ACL merged list is a large ACL that the CP compiles from multiple security ACL entries and policies. Whenthe ACE executes an ACL merged list, it performs multiple actions on a flow that matches the merged list.

4. Display the actions that the ACE will perform on a flow by entering the show acl-merge merged-listcommand. For example, to display the merged list for VLAN 100, enter the following command:

ACE_module5/Admin# show acl-merge merged-list vlan 100 in non-redundant

All ACEs in merged list 2 Total:18 Non-redundant:12

Priority:1000, Lineno:0, ACE-id:211 Action:PERMIT, Path-id:0x81/0x0/0x0:6/0[6/0][6/0]Pmap:0x5, Log:FALSE/FALSE[FALSE][FALSE], Interval:0/0[0][0]Hash1:0x0 Hash2:0x0Generated:TRUE, need-to-add-in-comp:NO_ACT_NEEDED, redundant:FALSEParent:: feature:SECURITY ace-lineno:5 ACL priority:0[G:0,P:0,C:0,ACL:0]Parent:: feature:TO CP ace-lineno:2 ACL priority:16779265[G:0,P:1,C:8,ACL:1]Feature:SECURITY Policy:1[1][1] sec-level:0x0 Intratype:SKIPFeature:TO CP Policy:1[1][1] sec-level:0x0 Intratype:TERMINATEIntertype:TERMINATEIP address SRC:161.44.0.0/255.255.0.0 DST:10.86.215.134/255.255.255.255Ports SRC:RANGE 0 65535 DST:RANGE 22 22Protocol:6Hit Count:0 Active:TRUE Timerange:0...Feature:SECURITY Policy:0[0] sec-level:0x0 Intratype:TERMINATE...Feature:SLB Policy:14[14] sec-level:0x0 Intratype:TERMINATE...Feature:SRC NAT Policy:2[2] sec-level:0x0 Intratype:TERMINATE...

5. If the acl-memory Denied counter in the output of the show resource usage command is incrementing and thePeak (ACL) memory counter has not exceeded the Max Allocated ACL memory counter, the problem may liewith one of the nodes in the ACL merge tree. The ACL merge tree contains several different kinds of nodes (seethe example output below), each of a different size and each with a maximum limit. If you allocate a minimum of10 percent of the ACE resources to ACL memory, the ACE will guarantee 10% of the maximum number of eachnode. If your configuration causes the ACE to exceed the maximum value of one of these nodes, the ACLresource allocation will fail and the acl-memory Denied counter will increment.

To monitor the ACL merge tree node usage in the ACE, enter the following command:

ACE_module5/Admin# show np 1 access-list resource

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 99

Page 100: Ace Troubleshooting

ACL Tree Statistics for Context ID: Admin=========================================ACL memory max-limit: NoneACL memory guarantee: 10.00 %MTrie nodes(used/guaranteed/max-limit): 43 / 26214 / 262143 (compressed) <-------| 3 / 2199 / 21999 (uncompressed) <--------|Leaf Head nodes (used/guaranteed/max-limit): | 39 / 26214 / 262143 <--------------------|---- Maximum number of available nodes in the ACELeaf Parameter nodes (used/guaranteed/max-limit):| 594 / 52428 / 524288 <-------------------|Policy action nodes used: 153memory consumed: 23776 bytes resource-limited 4896 bytes other 28672 bytes total.min-guarantee: 7861043 bytes total, 0 % consumed.max-limit: 78610432 bytes total, 0 % consumed.

ACL Tree Statistics for the linecard====================================MTrie nodes(used): 43 (compressed) 3 (uncompressed) <--------------| (shared): 235929 (compressed) 19800 (uncompressed) |Leaf Head nodes (used/shared): 39 / 235929 <-----------------------|---- Number of used nodes in the ACELeaf Parameter nodes (used/shared): 594 / 471860 <-----------------|Policy action nodes (used/shared): 153 / 261990 <------------------|

You can calculate the percentage of use for each node type by dividing the used nodes value by the maximumnumber of nodes and multiplying the result by 100. If any of these percentages exceeds the maximum value ofallocated ACL memory for the context, increase the max value of allocated ACL memory using thelimit-resource acl-memory command in resource class configuration mode so that that value is greater than orequal to the highest used nodes percentage that you calculated. Alternatively, if you are approaching the limits ofACL resource capacity, you may consider consolidating your ACL configuration.

If the ACL nodes are depleted while the ACE is downloading ACL configurations for an interface, the completeACL merged list for that interface is deleted and no traffic flows through that interface. The ACE increments thedownload failure counter in the output of the show interface command and the ACE logs a system message fromthe configuration manager.

6. To trace a packet through a specific ACL, enter the following command:

ACE_module5/Admin# show np 1 access-list trace vlan 130 in protocol 1 source 172.27.16.23 2000 destination 192.168.12.15 3000

Root 0x2c01b00Src Mtrie (0) offset 1 curr 0x2c01b00 child 0x0 leaf 0x10a840 Dst Mtrie (0) offset 2 curr 0x10a840 child 0x0 leaf 0x3c01330 proto ICMP head node 0x4004880 proto node 0x4004880 src op range port 0/65535 dst op range port 0/65535 lineno 112000 inner match line#:112000inner match line#: 112000

packet matched priority 112000

action node 0x4c02460Action Leaf-nodeversion+aceid 0x99 (version 0 ace_id 153 dirty no)action_flag 0x10 (permit no log no punt_to_cp no capture no bridge yes)path ID 0x0

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 100

Page 101: Ace Troubleshooting

src nat 0x0 dst nat 0x0 vserver 0x0 fixup 0x0TCP conn 0x0 AAA 0x0 Websense 0x0 QOS Policer 0x0Syslog Info 0Hitcount 130426Syslog info: idx:[153:0] name_idx:[0:0] hash1:0x0 hash2:0x0 name_len:0 invalid

Number of DRAM access: 6 (2 mtrie 4 non-mtrie)

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 101

Page 102: Ace Troubleshooting

This article describes ACE network address translation (NAT), how to configure it, and how to troubleshootissues with NAT that you may encounter.

Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference

Contents

1 Overview of ACE Network AddressTranslation

2 NAT Configuration Guidelines andRestrictions

3 Configuring Dynamic NAT and PAT• 4 Configuring Server-Farm Based DynamicNAT

5 Configuring Static NAT and Port Redirection• 6 Configuring SNAT with Cookie and LoadBalancing

7 Troubleshooting ACE NAT and PAT•

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 102

Page 103: Ace Troubleshooting

Overview of ACE Network Address Translation

You can configure the ACE to translate a client source IP address to a routable address in the server's network.This process is called source NAT (SNAT). If you want to preserve the client source IP address, do not configureSNAT.

You can also configure the ACE to translate the private address of a server to a global IP address that is accessibleto clients. This process is called destination NAT (DNAT) and protects the server by hiding its real IP addressfrom the Internet.

Besides translating IP addresses, you can configure the ACE to translate TCP and UDP ports. This process iscalled port address translation (PAT).

The ACE provides the following types of NAT and PAT:

Interface-based dynamic NAT• Interface-based dynamic PAT• Server farm-based dynamic NAT• Static NAT• Static port redirection•

NAT Configuration Guidelines and Restrictions

When you configure NAT and PAT on your ACE, keep in mind the following NAT and PAT guidelines andrestrictions:

If a packet egresses an interface that you have not configured for NAT, the ACE transmits the packetuntranslated.

You can configure dynamic NAT or static NAT as an input service policy only; you cannot configure itas an output service policy.

When you remove a traffic policy from the last VLAN interface on which you applied the service policy,the ACE automatically resets the associated service-policy statistics. The ACE performs this action toprovide a new starting point for the service-policy statistics the next time that you attach a traffic policy toa specific VLAN interface.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 103

Page 104: Ace Troubleshooting

Configuring Dynamic NAT and PAT

Dynamic NAT is typically used for SNAT. When you configure dynamic NAT and PAT, be sure to configure aninterface for the client-side VLAN and an interface for the server-side VLAN.

The following SNAT configuration example shows the commands that you use to configure dynamic NAT andPAT on your ACE. In this SNAT example, packets that ingress the ACE from the 192.168.12.0 network aretranslated to one of the IP addresses in the NAT pool defined on VLAN 200 by the nat-pool command. The patkeyword indicates that ports higher than 1024 are also translated.

Note: If you are operating the ACE in one-arm mode, omit the client-side interface VLAN 100 andconfigure the service policy on interface VLAN 200.

access-list NAT_ACCESS line 10 extended permit tcp 192.168.12.0 255.255.255.0 172.27.16.0 255.255.255.0 eq http

class-map match-any NAT_CLASS match access-list NAT_ACCESS

policy-map multi-match NAT_POLICY class NAT_CLASS nat dynamic 1 vlan 200

interface vlan 100 mtu 1500 ip address 192.168.1.100 255.255.255.0 service-policy input NAT_POLICY no shutdown

interface vlan 200 mtu 1500 ip address 172.27.16.2 255.255.255.0 nat-pool 1 172.27.16.15 172.27.16.24 netmask 255.255.255.0 pat no shutdown

Configuring Server-Farm Based Dynamic NAT

The following SNAT configuration example shows the commands that you use to configure server farm-baseddynamic NAT on your ACE. In this SNAT example, real server addresses on the 172.27.16.0 network aretranslated to one of the IP addresses in the NAT pool defined on VLAN 200 by the nat-pool command.

Note: If you are operating the ACE in one-arm mode, omit interface VLAN 100 and configure the servicepolicy on interface VLAN 200.

access-list NAT_ACCESS line 10 extended permit tcp 192.168.12.0 255.255.255.0 172.27.16.0 255.255.255.0 eq http

rserver SERVER1 ip address 172.27.16.3 inservicerserver SERVER2 ip address 172.27.16.4 inservice

serverfarm SFARM1 rserver SERVER1 inservice rserver SERVER2 inservice

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 104

Page 105: Ace Troubleshooting

class-map type http loadbalance match-any L7_CLASS match http content .*cisco.comclass-map match-any NAT_CLASS match access-list NAT_ACCESS

policy-map type loadbalance http first-match L7_POLICY class L7_CLASS serverfarm SFARM1 nat dynamic 1 vlan 200 serverfarm primarypolicy-map multi-match NAT_POLICY class NAT_CLASS loadbalance policy L7_POLICY loadbalance vip inservice

interface vlan 100 mtu 1500 ip address 192.168.1.100 255.255.255.0 service-policy input NAT_POLICY no shutdown

interface vlan 200 mtu 1500 ip address 172.27.16.2 255.255.255.0 nat-pool 1 172.27.16.15 172.27.16.24 netmask 255.255.255.0 no shutdown

Configuring Static NAT and Port Redirection

The following DNAT configuration example shows those sections of the running configuration that are related tothe commands necessary to configure static NAT and port redirection on your ACE. Typically, this configurationis used for DNAT, where HTTP packets that are destined to 192.0.0.0/8 and ingress the ACE on VLAN 101 aretranslated to 10.0.0.0/8 and port 8080. In this example, the servers are hosting HTTP on custom port 8080.

access-list acl1 line 10 extended permit tcp 10.0.0.0 255.0.0.0 eq 8080 any

class-map match-any NAT_CLASS match access-list acl1

policy-map multi-match NAT_POLICY class NAT_CLASS nat static 192.0.0.0 255.0.0.0 80 vlan 101

interface vlan 100 mtu 1500 ip address 192.168.1.100 255.255.255.0 service-policy input NAT_POLICY no shutdown

interface vlan 101 mtu 1500 ip address 172.27.16.100 255.255.255.0 no shutdown

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 105

Page 106: Ace Troubleshooting

Configuring SNAT with Cookie and Load Balancing

The following configuration example shows those commands necessary to configure SNAT (dynamic NAT) withcookie load balancing. Any source host that sends traffic to the VIP 20.11.0.100 is translated to one of the freeaddresses in the NAT pool in the range 30.11.100.1 to 30.11.200.1, inclusive. If you want to use PAT instead ofNAT, replace nat dynamic 1 vlan 2021 with nat dynamic 2 vlan 2021 in the L7SLBCookie policy map.

server host http ip address 30.11.0.10 inserviceserverfarm host httpsf rserver http inservice

class-map match-any vip4 2 match virtual-address 20.11.0.100 tcp eq wwwclass-map type http loadbalance match-any L7SLB_Cookie 3 match http cookie JG cookie-value ?.*?

policy-map type loadbalance first-match L7SLB_Cookie class L7SLB_Cookie serverfarm httpsfpolicy-map multi-match L7SLBCookie class vip4 loadbalance vip inservice loadbalance L7SLB_Cookie nat dynamic 1 vlan 2021

interface vlan 2020 ip address 20.11.0.2 255.255.0.0 alias 20.11.0.1 255.255.0.0 peer ip address 20.11.0.3 255.255.0.0 service-policy input L7SLBCookie no shutdowninterface vlan 2021 ip address 30.11.0.2 255.255.0.0 alias 30.11.0.1 255.255.0.0 peer ip address 30.11.0.3 255.255.0.0 fragment min-mtu 68 nat-pool 2 30.11.201.1 30.11.201.1 netmask 255.255.255.255 pat nat-pool 3 30.11.202.1 30.11.202.3 netmask 255.255.255.255 nat-pool 1 30.11.100.1 30.11.200.1 netmask 255.255.255.255 no shutdown

Troubleshooting ACE NAT and PAT

To verify your NAT and PAT configurations and make any necessary corrections, follow these steps:

1. Display your NAT and PAT configurations by entering the following commands:

ACE_module5/Admin# show running-config class-map

class-map match-any L4_CLASS 2 match access-list ACL1

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 106

Page 107: Ace Troubleshooting

ACE_module5/Admin# show running-config policy-map

policy-map multi-match NAT_POLICY class NAT_CLASS nat dynamic 1 vlan 200

ACE_module5/Admin# show service-policy NAT_POLICY

Status : ACTIVE-----------------------------------------Interface: vlan 100 service-policy: NAT_POLICY class: NAT_CLASS nat: nat dynamic 1 vlan 200 curr conns : 0 , hit count : 0 dropped conns : 0 client pkt count : 0 , client byte count: 0 server pkt count : 0 , server byte count: 0 conn-rate-limit : 0 , drop-count : 0 bandwidth-rate-limit : 0 , drop-count : 0

ACE_module5/Admin# show running-config interface

interface vlan 100 ip Address 192.168.12.2 mtu 1500 service-policy input NAT_POLICY no shutdowninterface vlan 200 ip address 172.27.16.2 255.255.255.0 mtu 1500 access-group input acl1 nat-pool 1 172.27.16.15 172.27.16.24 netmask 255.255.255.0 pat no shutdown

2. Use the show xlate command to verify that dynamic NAT and PAT, and static NAT and port redirection, aretaking place properly.

Dynamic NAT Example

The following example output of the show xlate command shows dynamic NAT (SNAT in this example).When you use Telnet from IP address 172.27.16.5 in VLAN 2020, the ACE translates it to IP address192.168.100.1 in VLAN 2021.

host1/Admin# show xlate global 192.168.100.1 192.168.100.10

NAT from vlan2020:172.27.16.5 to vlan2021:192.168.100.1 count:1

Dynamic PAT Example

The following example shows dynamic PAT. When you use Telnet from IP address 172.27.16.5 port38097 in VLAN 2020, the ACE translates it to IP address 192.168.201.1 port 1025 in VLAN 2021.

host1/Admin# show xlate

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 107

Page 108: Ace Troubleshooting

TCP PAT from vlan2020:172.27.16.5/38097 to vlan2021:192.168.201.1/1025

Static NAT Example

The following example shows static NAT. The ACE maps real IP address 172.27.16.5 to IP address192.168.210.1.

host1/Admin# show xlate

NAT from vlan2020:172.27.16.5 to vlan2021:192.168.210.1 count:1

host1/Admin# show conn

total current connections : 2

conn-id dir prot vlan source destination state----------+---+----+----+----------------+----------------+----------+7 in TCP 2020 172.27.16.5 192.168.100.1 ESTAB6 out TCP 2021 192.168.100.1 192.168.210.1 ESTAB

Static Port Redirection (Static PAT) Example

The following example shows static port redirection (DNAT in this example). A host at IP address192.168.0.10:37766 uses Telnet to connect to IP address 192.168.211.1:3030 on VLAN 2021 on theACE. The ACE maps IP address 172.27.0.5:23 on VLAN 2020 to IP address 192.168.211.1:3030 onVLAN 2021.

host1/Admin# show xlate

TCP PAT from vlan2020:172.27.0.5/23 to vlan2021:192.168.211.1/3030Mar 24 2006 20:05:41 : %ACE-7-111009: User 'admin' executed cmd: show xlate

host1/Admin# show conn

total current connections : 2

conn-id dir prot vlan source destination state----------+---+----+----+------------------+------------------+------+6 in TCP 2021 192.168.0.10:37766 192.168.211.1:3030 ESTAB7 out TCP 2020 172.27.0.5:23 192.168.0.10:1025 ESTAB

3. To display the NAT policy and pool information for the current context, enter the show nat-fabric command.The syntax of this command is as follows:

show nat-fabric {policies | src-nat policy_id mapped_if | dst-nat static_xlate_id | nat-pools |implicit-pat| global-static}

policies -- Displays the NAT policies.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 108

Page 109: Ace Troubleshooting

src-nat policy_id mapped_if -- Displays the specified source NAT policy information. To obtain thevalues for the policy_id and mapped_if arguments, view the policy_id and mapped_if fields displayed bythe show nat-fabric policies command.

dst-nat static_xlate_id -- Displays the static address translation for the specified static XLATE ID. Toobtain the value for the static_xlate_id argument, view the static_xlate_id field displayed by the shownat-fabric policies command.

nat-pools -- Displays NAT pool information for a dynamic NAT policy.

implicit-pat -- Displays the implicit PAT policies.

global-static -- Displays global static NAT information when the static command in global configurationmode is configured.

ACE_module5/Admin# show nat-fabric policies

Nat objects:

NAT object Hash Bucket: 9 NAT object ID:2 mapped_if:8 policy_id:1 type:DYNAMIC nat_pool_id:4 Pool ID:4 PAT:1 pool_id:1 mapped_if:8 Ref_count:1 ixp_binding:in all IXPs lower:172.27.16.15 upper:172.27.16.24 Bitmap-ID:40 List of NAT object IDs: 2

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 109

Page 110: Ace Troubleshooting

This article describes ACE health monitoring (probes), how to configure it, and how to troubleshoot issues withprobes that you may encounter.

Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference

Contents

1 Overview of ACE HealthMonitoring

1.1 Configuring Probes♦ 1.2 Example of a ProbeConfiguration

2 Troubleshooting ACE HealthMonitoring

2.1 Troubleshooting anHTTP Probe Error

2.2 Troubleshooting anHTTPS Probe Error

2.3 Troubleshooting anSNMP Probe Issue

2.4 Using the Last StatusCode Field

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 110

Page 111: Ace Troubleshooting

Overview of ACE Health Monitoring

This section describes health monitoring on the ACE. The ACE uses probes that you configure to track the stateof a server. By default, no probes are configured in the ACE. Also referred to as out-of-band (OOB) healthmonitoring, the ACE verifies the server response to a probe or checks for any network problems that can preventa client from reaching a server. Based on the server response, the ACE can place the server in or out of serviceand can make reliable load-balancing decisions.

You can also use health monitoring to detect failures for a gateway or a host in high-availability (redundant)configurations. For more information, see the Cisco Application Control Engine Module Administration Guide.

The ACE evaluates the health of a server by marking the probes as follows:

Passed?The server returns a valid response.•

Failed?The server fails to provide a valid response to the ACE and the ACE is unable to reach a server fora specified number of retries.

By configuring the ACE for health monitoring, the ACE sends active probes periodically to determine the serverstate. The ACE supports 4096 unique probe configurations, which includes ICMP, TCP, HTTP, and otherpredefined health probes. The ACE can execute only up to 200 concurrent scripted probes at a time. The ACEalso allows the opening of 2048 sockets simultaneously.

You can associate the same probe with multiple real servers or server farms. Each time that you use the sameprobe again, the ACE counts it as another probe instance. You can allocate a maximum of 16,000 probe instances.

Configuring Probes

You can configure health probes on the ACE to actively make connections and explicitly send traffic to servers.The probes determine whether the health status of a server passes or fails by the server's response.

Configuring active probes is a three-step process:

1. Configure the health probe with a name, type, and attributes.

2. Associate the probe with one of the following:

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 111

Page 112: Ace Troubleshooting

A real server.◊

A real server and then associate the real server with a server farm. You can associate a singleprobe or multiple probes with real servers within a server farm.

A server farm. All servers in the server farm receive probes of the associated probe types.◊

3. Place the real server or server farm in service.

Example of a Probe Configuration

The following example shows a running configuration that load balances DNS traffic across multiple real serversand transmits and receives UDP data that spans multiple packets. The configuration uses a UDP health probe.

access-list ACL1 line 10 extended permit ip any any

probe udp UDP interval 5 passdetect interval 10 description THIS PROBE IS INTENDED FOR LOAD BALANCING DNS TRAFFIC port 53 send-data UDP_TEST

rserver host SERVER1 ip address 192.168.10.45 inservicerserver host SERVER2 ip address 192.168.10.46 inservicerserver host SERVER3 ip address 192.168.10.47 inservice

serverfarm host SFARM1 probe UDP rserver SERVER1 inservice rserver SERVER2 inservice rserver SERVER3 inservice

class-map match-all L4UDP-VIP_114:UDP_CLASS 2 match virtual-address 192.168.120.114 udp eq 53

policy-map type loadbalance first-match L7PLBSF_UDP_POLICY class class-default serverfarm SFARM1policy-map multi-match L4SH-Gold-VIPs_POLICY class L4UDP-VIP_114:UDP_CLASS loadbalance vip inservice loadbalance policy L7PLBSF_UDP_POLICY loadbalance vip icmp-reply nat dynamic 1 vlan 120 connection advanced-options 1SECOND-IDLE

interface vlan 120 description Upstream VLAN_120 - Clients and VIPs

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 112

Page 113: Ace Troubleshooting

ip address 192.168.120.1 255.255.255.0 fragment chain 20 fragment min-mtu 68 access-group input ACL1 nat-pool 1 192.168.120.70 192.168.120.70 netmask 255.255.255.0 pat service-policy input L4SH-Gold-VIPs_POLICY no shutdown

ip route 10.1.0.0 255.255.255.0 192.168.120.254

Troubleshooting ACE Health Monitoring

This section describes how to troubleshoot two common probe configuration issues.

Troubleshooting an HTTP Probe Error

In this first scenario, you have configured an HTTP probe, but the real server's health status is displayed asFAILED and the Last disconnect err field indicates that an invalid status code was received as displayed in theshow probe detail command output. You have checked your server and it is up and running. A packet capture onthe server also shows that everything is fine. Where is the issue?

1. Display the probe status details by entering the following command:

ACE_module5/Admin# show probe detail

probe : HTTP_PROBE type : HTTP state : ACTIVE description :---------------------------------------------- port : 80 address : 0.0.0.0 addr type : - interval : 10 pass intvl : 10 pass count : 3 fail count: 3 recv timeout: 10 http method : GET http url : / conn termination : GRACEFUL expect offset : 0 , open timeout : 1 expect regex : - send data : - ------------------ probe results ------------------ associations ip-address port porttype probes failed passed health ------------ ---------------+-----+--------+--------+--------+--------+------ rserver : SERVER1 192.168.10.45 80 -- 2 2 0 FAILED

Socket state : CLOSED No. Passed states : 0 No. Failed states : 1

No. Probes skipped : 0 Last status code : 200 <------- Last status code from server No. Out of Sockets : 0 No. Internal error: 0

Last disconnect err : Received invalid status code <------- Last probe time : Tue Apr 7 16:17:26 2009 Last fail time : Tue Apr 7 16:17:16 2009 Last active time : Never

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 113

Page 114: Ace Troubleshooting

The Last disconnect err field indicates that the ACE received an invalid status code. This error means that youhave has not configured the expect status command for the probe.

2. Confirm this finding by entering the following command:

ACE_module5/Admin# show running-config probeGenerating configuration....

probe http HTTP_PROBE interval 10 passdetect interval 10 open 1

3. Correct the problem by entering the following commands:

ACE_module5/Admin# configEnter configuration commands, one per line. End with CNTL/Z.ACE_module5/Admin(config)# probe http HTTP_PROBEACE_module5/Admin(config-probe-http)# expect status 200 200 <------- 200 indicates the 200 OK message from the serverACE_module5/Admin(config-probe-http)# end

4. Confirm the configuration by entering the following command:

ACE_module5/Admin# show running-config probeGenerating configuration....

probe http HTTP_PROBE interval 10 passdetect interval 10 expect status 200 200 open 1

5. Display the probe status details again and observe that the server health status value is SUCCESS by enteringthe following command:

ACE_module5/Admin# show probe HTTP_PROBE detail

probe : HTTP_PROBE type : HTTP state : ACTIVE description :---------------------------------------------- port : 80 address : 0.0.0.0 addr type : - interval : 10 pass intvl : 10 pass count : 3 fail count: 3 recv timeout: 10 http method : GET http url : / conn termination : GRACEFUL expect offset : 0 , open timeout : 1 expect regex : - send data : - ------------------ probe results ------------------ associations ip-address port porttype probes failed passed health ------------ ---------------+-----+--------+--------+--------+--------+------ rserver : SERVER1 192.168.10.45 80 -- 24 15 9 SUCCESS

Socket state : CLOSED

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 114

Page 115: Ace Troubleshooting

No. Passed states : 1 No. Failed states : 1 No. Probes skipped : 0 Last status code : 200 No. Out of Sockets : 0 No. Internal error: 0 Last disconnect err : - <------- No error indicated now. The probe is successful. Last probe time : Tue Apr 7 16:21:05 2009 Last fail time : Tue Apr 7 16:17:16 2009 Last active time : Tue Apr 7 16:20:05 2009

Troubleshooting an HTTPS Probe Error

In addition to the methods for Troubleshooting an HTTP Probe Error, use SSL statistics to troubleshoot HTTPSprobe failures. HTTPS probe traffic runs in the Admin virtual context so view the output of the show stats cryptoclient command in that context.

Troubleshooting an SNMP Probe Issue

In this scenario, you have configured an SNMP probe, but the Last disconnect err field indicates that the sum ofthe weights does not add up to the maximum weight value as displayed in the output of the show probe detailcommand.

1. Display the probe status details by entering the following command:

ACE_module5/test# show probe detail

probe : SNMP_PROBE type : SNMP state : ACTIVE description : snmp probe---------------------------------------------- port : 161 address : 0.0.0.0 addr type : - interval : 15 pass intvl : 10 pass count : 3 fail count: 3 recv timeout: 10 version : 2c community : test_comm oid string #1 : .1.3.6.1.2.1.4.3.0 type : ABSOLUTE max value : 1000000000 weight : 10000 threshold : 1000000000 ------------------ probe results ------------------ associations ip-address port porttype probes failed passed health ------------ ---------------+-----+--------+--------+--------+--------+------ serverfarm : least-loaded, predictor least-loaded real : SERVER1[0] 192.168.10.45 161 -- 0 0 0 INIT

Socket state : CLOSED No. Passed states : 0 No. Failed states : 0 No. Probes skipped : 0 Last status code : 0 No. Out of Sockets : 0 No. Internal error: 30

Last disconnect err : Sum of weights don't add up to max weight value <------- Error condition Last probe time : Never Last fail time : Never Last active time : Never Server load : 16000 <------- Note the server load value

The reason for this error is that the weight command needs to be configured when you have multiple OIDsconfigured for a single probe and from those OIDs if you want to give priority to a specific OID.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 115

Page 116: Ace Troubleshooting

The sum of the weights should equal 16000 (see the Server Load field). For a single OID, the weight commanddoes not have any significance.

2. Display the probe configuration by entering the following command:

ACE_module5/Admin# show running-config probe

probe snmp SNMP_PROBE description snmp probe port 161 interval 15 passdetect interval 10 version 2c community TEST_COMM oid .1.3.6.1.2.1.4.3.0 type absolute max 1000000000 weight 10000 <-------

In the above configuration, the weight is configured as 10000 for a single OID. The ACE is expecting anotherOID to be configured in the probe and the sum of both weights should equal 16000.

The configuration is not complete and the ACE is expecting additional parameters in the probe configuration.Because there is not another OID in the configuration, the ACE is not able to calculate the load and that is whythe "Sum of weights don't add up to max weight value" error message appears.

3. Resolve the issue by modifying the probe configuration as follows:

probe snmp SNMP_PROBE description test port 161 interval 15 passdetect interval 60 version 2c community test_comm oid .1.3.6.1.2.1.4.3.0 type absolute max 1000000000 weight 10000 oid .1.3.6.1.2.1.4.10.0 type absolute max 1000000000

weight 6000 <------- 10000 + 6000 = 16000

4. Display the probe status details again by entering the following command:

ACE_module5/test# show probe SNMP_PROBE detail

probe : snmp1 type : SNMP state : ACTIVE description : snmp probe---------------------------------------------- port : 161 address : 0.0.0.0 addr type : - interval : 15 pass intvl : 10 pass count : 3 fail count: 3 recv timeout: 10 version : 2c community : test_comm

oid string #1 : .1.3.6.1.2.1.4.3.0 type : ABSOLUTE max value : 1000000000 weight : 10000 threshold : 1000000000

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 116

Page 117: Ace Troubleshooting

oid string #2 : .1.3.6.1.2.1.4.10.0 type : ABSOLUTE max value : 1000000000 weight : 6000 threshold : 1000000000 ------------------ probe results ------------------ associations ip-address port porttype probes failed passed health ------------ ---------------+-----+--------+--------+--------+--------+------ serverfarm : least-loaded, predictor least-loaded real : SERVER1[0] 192.168.10.45 161 -- 4143 0 4143 SUCCESS

Socket state : CLOSED No. Passed states : 1 No. Failed states : 0 No. Probes skipped : 0 Last status code : 0 No. Out of Sockets : 0 No. Internal error: 0 Last disconnect err : - <------- No error indicated now. The probe is successful.

Last probe time : Mon Apr 6 09:12:54 2009 Last fail time : Never Last active time : Sun Apr 5 15:57:28 2009 Server load : 0

Using the Last Status Code Field

Details regarding the last status code field can be provided for nontrivial probes. For example, in the case ofscripted probe PROBENOTICE_PROBE, the status code 30001 means that the probe is successful and the value30002 indicates an error in probe arguments. The last disconnect error for status code 30002 displays "Did notreceive correct response from the server," but the actual issue is related to arguments in the probe configuration,which can be checked by looking at the script for the probe.

ACE_module5/Admin# show probe TEST detail

probe : TEST type : SCRIPTED state : ACTIVE description :---------------------------------------------- port : 0 address : 0.0.0.0 addr type : - interval : 15 pass intvl : 20 pass count : 3 fail count: 3 recv timeout: 10 script filename : PROBENOTICE_PROBE --------------------- probe results -------------------- probe association probed-address probes failed passed health ------------------- ---------------+----------+----------+----------+------- serverfarm : sf1 real : rs2[0] 23.0.0.5 4082 54 4028 SUCCESS

Socket state : RESET No. Passed states : 6 No. Failed states : 5 No. Probes skipped : 8 Last status code : 30001 <------- Indicates success No. Out of Sockets : 0 No. Internal error: 0 Last disconnect err : - Last probe time : Wed Apr 8 04:44:41 2009 Last fail time : Tue Apr 7 12:02:10 2009 Last active time : Tue Apr 7 12:03:45 2009

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 117

Page 118: Ace Troubleshooting

This article describes how to troubleshoot Layer 4 (L4) load balancing on the ACE.

Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference

Contents

1 Overview of ACE L4 Load Balancing1.1 Classifying L4 Traffic for ServerLoad Balancing

1.2 Example of a Layer 4 Load-BalancingConfiguration

2 Troubleshooting L4 Load Balancing on theACE

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 118

Page 119: Ace Troubleshooting

Overview of ACE L4 Load Balancing

Load balancing at L4 involves selecting a server in a server farm to service a client request based on the VIPaddress and protocol in the request. You configure a class map to classify (match) interesting traffic arriving at theACE and associate the class map with a policy map to perform an action on the traffic based on the classification.With L4 load balancing, the ACE selects a server based on the first packet it receives in a particular flow. See the"Overview of ACE Connection Handling" section in the Troubleshooting Connectivity article.

For detailed information about ACE load balancing, see the Cisco Application Control Engine Module ServerLoad Balancing Configuration Guide.

Classifying L4 Traffic for Server Load Balancing

You classify inbound network traffic destined to or passing through the ACE based on a series of flow matchcriteria specified by a class map. Each class map defines a traffic classification, which is network traffic that is ofinterest to you. A policy map defines a series of actions (functions) that you want applied to a set of classifiedinbound or outbound traffic.

ACE L3 and L4 traffic policies support the following server load-balancing (SLB) traffic attributes:

Source or destination IP address• Source or destination port• Virtual IP (VIP) address• IP protocol•

The three major steps in the traffic classification process are as follows:

Create a class map using the class-map command and the associated match commands, which comprisea set of match criteria related to Layer 3 and Layer 4 traffic classifications or Layer 7 protocolclassifications.

1.

Create a policy map using the policy-map command, which refers to the class maps and identifies aseries of actions to perform based on the traffic match criteria.

2.

Activate the policy map by associating it with a specific VLAN interface or globally with all VLANinterfaces using the service-policy command to filter the traffic received by the ACE.

3.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 119

Page 120: Ace Troubleshooting

Figure 1 provides a basic overview of the process required to build and apply the Layer 3, Layer 4, and Layer 7policies that the ACE uses for SLB. The figure also shows how you associate the various components of the SLBpolicy configuration with each other.

Figure 1. SLB Flow Diagram

Example of a Layer 4 Load-Balancing Configuration

The following example shows a L4 load-balancing configuration:

access-list ACL1 line 10 extended permit ip any any

rserver host SERVER1 ip address 192.168.252.245 inservice

rserver host SERVER2

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 120

Page 121: Ace Troubleshooting

ip address 192.168.252.246 inservice

rserver host SERVER3 ip address 192.168.252.247 inservice

rserver host SERVER4 ip address 192.168.252.248 inservice

rserver host SERVER5 ip address 192.168.252.249 inservice

rserver host SERVER6 ip address 192.168.252.250 inservice

serverfarm host SFARM1 probe TCP_PROBE predictor roundrobin rserver SERVER1 weight 10 inservice rserver SERVER2 weight 20 inservice rserver SERVER3 weight 30 inservice

serverfarm host SFARM2 probe TCP_PROBE predictor roundrobin rserver SERVER4 weight 10 inservice rserver SERVER5 weight 20 inservice rserver SERVER6 weight 30 inservice

class-map match-all L4WEB_CLASS 2 match virtual-address 192.168.120.112 tcp eq www

policy-map type loadbalance first-match LB_WEB_POLICY class class-default serverfarm SFARM1 backup SFARM2

policy-map multi-match L4WEB_POLICY class L4WEB_CLASS loadbalance vip inservice loadbalance policy LB_WEB_POLICY loadbalance vip icmp-reply active nat dynamic 1 VLAN 120

interface vlan 100 description Upstream VLAN_100 - Clients and VIPs ip address 192.168.120.1 255.255.255.0

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 121

Page 122: Ace Troubleshooting

fragment chain 20 fragment min-mtu 68 access-group input ACL1 nat-pool 1 192.168.120.70 192.168.120.70 netmask 255.255.255.0 pat service-policy input L4WEB_POLICY no shutdown

ip route 10.1.0.0 255.255.255.0 192.168.120.254

Troubleshooting L4 Load Balancing on the ACE

To troubleshoot L4 load-balancing issues, follow these steps:

1. Ensure that your load-balancing configuration is correct and that the following conditions exist:

Real servers have valid IP addresses and are in service.◊ Servers are associated with server farms of the same type.◊ A load-balancing policy exists with an associated server farm and is associated with a L4multimatch policy.

An L4 class map contains a valid match virtual-address command and is associated with the L4multimatch policy map.

The L4 policy is applied to the appropriate active interface using a service policy.◊ A static route is configured for the server network.◊

Use the following show commands:

show running-config rserver◊ show running-config serverfarm◊ show running-config policy-map◊ show running-config class-map◊ show running-config interface◊ show ip route◊

2. Check the ACE connectivity. See the "Troubleshooting Connectivity" section.

3. Verify that the L4 VIP class map is referenced in a L4 policy by entering the following command. Also, checkthe following fields:

VIP address and port◊ VIP state◊ Hit count◊ Dropped connections◊

ACE_module5/Admin# show service-policy L4WEB_POLICY detail

Status : ACTIVEDescription: ------------------------------------------Interface: vlan 100

service-policy: L4WEB_POLICY <------- L4 multimatch policy mapclass: L4WEB_CLASS <------- L4 VIP class map

VIP Address: Protocol: Port: 192.168.120.112 tcp eq 80 <------- VIP address, protocol, and port

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 122

Page 123: Ace Troubleshooting

loadbalance: L7 loadbalance policy: LB_WEB_POLICY VIP Route Metric : 77 VIP Route Advertise : DISABLED VIP ICMP Reply : ENABLED

VIP State: INSERVICE <------- VIP state should be INSERVICE curr conns : 0 , hit count : 56

dropped conns : 14 <------- Number of attempted connections to this VIP that the ACE discarded client pkt count : 6297 , client byte count: 1047583 server pkt count : 1238 , server byte count: 1325495 L7 Loadbalance policy : LB_WEB_POLICY class/match : class-default LB action : serverfarm: SFARM1

hit count : 0 <-------|-- Check these counters to see if they are increasingdropped conns : 0 <-------|

The dropped conns counter under a VIP in the output of the show service policy detail command is incrementedwhenever the ACE discards a connection request destined to that VIP. There are several reasons why the ACEdiscards such connection requests. For example:

If all the real servers in the server farm associated with the VIP go down, then the VIP will go down. So,all the incoming connections to that VIP are discarded.

If the URL in a connection request to the VIP is unknown, then the connection request is discarded.• If the server to which the ACE load balances the connection does not respond to the request, then, afterthe maximum number of retries, the ACE discards the connection.

The dropped conns counter is cumulative and the value may comprise entries from any of the following showcommand counters:

show stats loadbalance•

- Total Layer4 rejections- Total Layer7 rejections- Total Layer4 LB policy misses- Total Layer7 LB policy misses- Total times rserver was unavailable

show stats connection•

- Total Connections Timed-out- Total Connections Failed

The failures counter of the show serverfarm serverfarm_name command•

The Total drop decisions counter of the show stats inspect command•

4. Verify that the L4 policy is applied as a service policy to an active interface by entering the followingcommand:

ACE_module5/Admin# show running-config interfaceGenerating configuration....

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 123

Page 124: Ace Troubleshooting

interface vlan 100 ip address 192.168.120.1 255.255.255.0 access-group input ACL1 access-group output anyone

service-policy input L4WEB_POLICY no shutdown...

5. Check the total conn-dropcount field for the primary server farm in the output of the following command. Also,check the IP address, state, and the connection statistics for each real server that is configured in the server farm.

ACE_module5/Admin# show serverfarm SFARM1 detail

serverfarm : SFARM1, type: HOSTtotal rservers : 3active rservers: 3description : -state : ACTIVE <------- Current state of the server farmpredictor : ROUNDROBIN <------- Load-balancing method weight : - autoadjust : MAXLOADfailaction : -back-inservice : 40partial-threshold : 40num times failover : 0num times back inservice : 0total conn-dropcount : 0 <------- Total number of connection attempts to this server farm that the ACE discarded--------------------------------- ----------connections----------- real weight state current total failures ---+---------------------+------+------------+----------+----------+--------- rserver: SERVER1

192.168.252.245:0 10 INSERVICE 0 0 0 <------- Real server IP address, state, and connection statistics max-conns : 4000000 , out-of-rotation count : 0 min-conns : 4000000 conn-rate-limit : - , out-of-rotation count : - bandwidth-rate-limit : - , out-of-rotation count : - retcode out-of-rotation count : - load value : 0

rserver: SERVER2 192.168.252.246:0 20 INSERVICE 0 0 0 max-conns : 4000000 , out-of-rotation count : 0 min-conns : 4000000 conn-rate-limit : - , out-of-rotation count : - bandwidth-rate-limit : - , out-of-rotation count : - retcode out-of-rotation count : - load value : 0

rserver: SERVER3 192.168.252.247:0 30 INSERVICE 0 0 0 max-conns : 4000000 , out-of-rotation count : 0 min-conns : 4000000 conn-rate-limit : - , out-of-rotation count : - bandwidth-rate-limit : - , out-of-rotation count : - retcode out-of-rotation count : - load value : 0

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 124

Page 125: Ace Troubleshooting

6. Check the L4 load-balance statistics by entering the following command:

ACE_module5/Admin# show stats loadbalance

+------------------------------------------++------- Loadbalance statistics -----------++------------------------------------------+ Total version mismatch : 0 Total Layer4 decisions : 0 Total Layer4 rejections : 0 Total Layer7 decisions : 0 Total Layer7 rejections : 0 Total Layer4 LB policy misses : 0 Total Layer7 LB policy misses : 0 Total times rserver was unavailable : 0 Total ACL denied : 0 Total IDMap Lookup Failures : 0

Note: The ID Map is used to map real servers and server farms between the local and the remote peers in aredundant configuration. The Total IDMap Lookup Failures field increments if the local ACE fails tofind the local ACE to peer ACE ID mapping. A failure can occur if the peer ACE did not send a properremote ID for the local ACE to look up and so the local ACE could not perform a mapping or if the IDMap table was not created.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 125

Page 126: Ace Troubleshooting

This article describes how to diagnose and troubleshoot ACE L7 load-balancing issues.

Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference

Contents

1 Overview of ACE Layer 7 Load Balancing1.1 Load-Balancing Predictors♦ 1.2 Classifying L7 Traffic for ServerLoad Balancing

2 Example of a L7 Load-BalancingConfiguration

3 Troubleshooting Layer 7 Load Balancing•

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 126

Page 127: Ace Troubleshooting

Overview of ACE Layer 7 Load Balancing

Layer 7 server load balancing (SLB) is the process that the load-balancing device uses to decide which servershould fulfill a client request for an L7 service. For example, a client request may consist of a HyperTextTransport Protocol (HTTP) GET for a web page or a File Transfer Protocol (FTP) GET to download a file. Thejob of the load balancer is to select the server that can successfully fulfill the client request and do so in theshortest amount of time without overloading either the server or the server farm as a whole.

The ACE supports the load balancing of the following protocols:

Generic protocols• HTTP• Remote Authentication Dial-In User Service (RADIUS)• Reliable Datagram Protocol (RDP)• Real-Time Streaming Protocol (RTSP)• Session Initiation Protocol (SIP)•

Depending on the load-balancing algorithm?or predictor?that you configure, the ACE performs a series of checksand calculations to determine which server can best service each client request. The ACE bases server selectionon several factors including the source or destination address, cookies, URLs, HTTP headers, or the server withthe fewest connections with respect to load.

For detailed information about ACE load balancing, see the Cisco Application Control Engine Module ServerLoad Balancing Configuration Guide.

Load-Balancing Predictors

The ACE uses the following predictors to select the best server to fulfill a client request:

Application response?Selects the server with the lowest average response time for the specifiedresponse-time measurement based on the current connection count and server weight (if configured).

Hash address?Selects the server using a hash value based on either the source or destination IP address orboth. Use these predictors for firewall load balancing (FWLB). For more information about FWLB, seeConfiguring Firewall Load Balancing in the Cisco Application Control Engine Module ServerLoad-Balancing Configuration Guide (Software Version A2(1.0)).

Hash content?Selects the server using a hash value based on a content string in the Trusted Third Parties(TTP) packet body.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 127

Page 128: Ace Troubleshooting

Hash cookie?Selects the server using a hash value based on a cookie name.•

Hash header?Selects the server using a hash value based on the HTTP header name.•

Hash URL?Selects the server using a hash value based on the requested URL. You can specify abeginning pattern and an ending pattern to match in the URL. Use this predictor method to load balancecache servers.

Least bandwidth?Selects the server that processed the least amount of network traffic based on theaverage bandwidth that the server used over a specified number of samples.

Least connections?Selects the server with the fewest number of active connections based on the serverweight. For the least-connections predictor, you can configure a slow-start mechanism to avoid sending ahigh rate of new connections to servers that you have just put into service.

Least loaded?Selects the server with the lowest load based on information obtained from Simple NetworkManagement Protocol (SNMP) probes. To use this predictor, you must associate an SNMP probe with it.

Round-robin?Selects the next server in the list of real servers based on the server weight (weightedround-robin). Servers with a higher weight value receive a higher percentage of the connections. This isthe default predictor.

Note: The hash predictor methods do not recognize the weight value that you configure for real servers. TheACE uses the weight that you assign to real servers only in the least-connections, application-response,and round-robin predictor methods.

Classifying L7 Traffic for Server Load Balancing

You classify inbound network traffic destined to or passing through the ACE based on a series of flow matchcriteria specified by a class map. Each class map defines a traffic classification, which is network traffic that is ofinterest to you. A policy map defines a series of actions (functions) that you want applied to a set of classifiedinbound or outbound traffic.

ACE traffic policies support the following server load-balancing (SLB) traffic attributes:

Layer 3 and Layer 4 connection information?Source or destination IP address, source or destination port,virtual IP address, and IP protocol

Layer 7 protocol information?Hypertext Transfer Protocol (HTTP) cookie, HTTP URL, HTTP header,Remote Authentication Dial-In User Service (RADIUS), Remote Desktop Protocol (RDP), Real-TimeStreaming Protocol (RTSP), Session Initiation Protocol (SIP), and Secure Sockets Layer (SSL)

The three major steps in the traffic classification process are as follows:

Create a class map using the class-map command and the associated match commands, which comprisea set of match criteria related to Layer 3 and Layer 4 traffic classifications or Layer 7 protocolclassifications.

1.

Create a policy map using the policy-map command, which refers to the class maps and identifies aseries of actions to perform based on the traffic match criteria.

2.

Activate the policy map by associating it with a specific VLAN interface or globally with all VLANinterfaces using the service-policy command to filter the traffic received by the ACE.

3.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 128

Page 129: Ace Troubleshooting

Figure 1 provides a basic overview of the process required to build and apply the Layer 3, Layer 4, and Layer 7policies that the ACE uses for SLB. The figure also shows how to associate the various components of the SLBpolicy configuration with each other.

Figure 1. SLB Flow Diagram

Example of a L7 Load-Balancing Configuration

The following example shows a running configuration that includes multiple class maps and policy maps thatdefine a traffic policy for SLB. In this configuration, when a server farm is chosen for a connection, theconnection is sent to a real server based on one of several load-balancing predictors. The leastconns predictormethod load balances connections to the server that has the lowest number of open connections.

access-list ACL1 line 10 extended permit ip any any

probe tcp TCP interval 5 faildetect 2

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 129

Page 130: Ace Troubleshooting

passdetect interval 10 open 3

parameter-map type http PERSIST-REBALANCE persistence-rebalance <---------------------------- Enabled by default in the ACE appliance. Enabled by default in the ACE module in software version A4(1.0) and later.parameter-map type connection PRED-CONNS-UDP_CONN set timeout inactivity 300

rserver SERVER1 ip address 10.1.0.2 inservicerserver SERVER2 ip address 10.1.0.3 inservicerserver SERVER3 ip address 10.1.0.4 inservicerserver SERVER4 ip address 10.1.0.5 inservicerserver SERVER5 ip address 10.1.0.6 inservicerserver SERVER6 ip address 10.1.0.7 inservicerserver SERVER7 ip address 10.1.0.8 inservicerserver SERVER8 ip address 10.1.0.9 inservice

serverfarm host PRED-CONNS predictor leastconns rserver SERVER1 inservice rserver SERVER2 inservice rserver SERVER3 inservice rserver SERVER4 inservice rserver SERVER5 inservice rserver SERVER6 inservice rserver SERVER7 inservice rserver SERVER8 inservice

serverfarm host PRED-CONNS-UDP failaction purge predictor leastconns rserver SERVER1 inservice rserver SERVER2 inservice rserver SERVER3 probe ICMP inservice

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 130

Page 131: Ace Troubleshooting

rserver SERVER5 inservice rserver SERVER6 inservice rserver SERVER7 inservice

serverfarm host PREDICTOR probe TCP rserver SERVER1 inservice rserver SERVER2 inservice rserver SERVER6 inservice rserver SERVER7 inservice

sticky http-cookie COOKIE_TEST STKY-GRP-43 cookie offset 1 length 999 timeout 30 replicate sticky serverfarm PREDICTOR

class-map match-all L4PRED-CONNS-UDP-VIP_128:2222_CLASS 2 match virtual-address 192.168.120.128 udp eq 0 class-map match-all L4PRED-CONNS-VIP_128:80_CLASS 2 match virtual-address 192.168.120.128 tcp eq www class-map match-all L4PREDICTOR_117:80_CLASS 2 match virtual-address 192.168.120.117 tcp eq www

policy-map type loadbalance first-match L7PLBSF_PRED-CONNS_POLICY class class-default serverfarm PRED-CONNSpolicy-map type loadbalance first-match L7PLBSF_PRED-CONNS-UDP_POLICY class class-default serverfarm PRED-CONNS-UDPpolicy-map type loadbalance first-match L7PLBSF_PREDICTOR_POLICY class class-default sticky-serverfarm STKY-GRP-43policy-map multi-match L4SH-Gold-VIPs_POLICY class L4PREDICTOR_117:80_CLASS loadbalance vip inservice loadbalance policy L7PLBSF_PREDICTOR_POLICY loadbalance vip icmp-reply active nat dynamic 1 vlan 120 appl-parameter http advanced-options PERSIST-REBALANCE class L4PRED-CONNS-VIP_128:80_CLASS loadbalance vip inservice loadbalance policy L7PLBSF_PRED-CONNS_POLICY loadbalance vip icmp-reply active nat dynamic 1 vlan 120 appl-parameter http advanced-options PERSIST-REBALANCE class L4PRED-CONNS-UDP-VIP_128:2222_CLASS loadbalance vip inservice loadbalance policy L7PLBSF_PRED-CONNS-UDP_POLICY loadbalance vip icmp-reply active nat dynamic 1 vlan 120 appl-parameter http advanced-options PERSIST-REBALANCE connection advanced-options PRED-CONNS-UDP_CONN

interface vlan 120

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 131

Page 132: Ace Troubleshooting

description Upstream VLAN_120 - Clients and VIPs ip address 192.168.120.1 255.255.255.0 fragment chain 20 fragment min-mtu 68 access-group input ACL1 nat-pool 1 192.168.120.70 192.168.120.70 netmask 255.255.255.0 pat service-policy input L4SH-Gold-VIPs_POLICY no shutdown

ip route 10.1.0.0 255.255.255.0 192.168.120.254

Troubleshooting Layer 7 Load Balancing

To troubleshoot L7 load-balancing issues, use the following steps:

1. Ensure that your load-balancing configuration is correct and that the following conditions exist:

Real servers have valid IP addresses and are in service◊ Servers are associated with server farms of the same type◊ L7 load-balancing policy exists with an associated server farm and that the L7 load-balancingpolicy is associated with a L4 multimatch policy

An L4 class map contains a valid match virtual-address command and is associated with the L4multimatch policy map

The L4 policy is applied to the appropriate active interface using a service policy◊ A static route is configured for the server network◊

Use the following show commands to verify your load-balancing configuration:

show running-config rserver◊ show running-config serverfarm◊ show running-config policy-map◊ show running-config class-map◊ show running-config interface◊ show ip route◊

2. Check the ACE connectivity. See the Troubleshooting Connectivity section.

3. Verify that the L7 load-balancing policy is referenced in the L4 policy by entering the following command.Also, check the following fields:

VIP address, protocol, and port◊ VIP state◊ Hit count◊ Dropped connections◊

ACE_module5/Admin# show service-policy L4WEB_POLICY detail

Status : ACTIVEDescription: ------------------------------------------Interface: vlan 100 service-policy: L4WEB_POLICY class: L4WEB_CLASS

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 132

Page 133: Ace Troubleshooting

VIP Address: Protocol: Port: 192.168.120.112 tcp eq 80 <------- VIP address, protocol, and port

loadbalance:L7 loadbalance policy: LB_WEB_POLICY <-------L7 load-balancing policy referenced in the L4 multimatch policy

VIP Route Metric : 77 VIP Route Advertise : DISABLED VIP ICMP Reply : ENABLED

VIP State: INSERVICE <------- VIP state should be INSERVICE curr conns : 0 , hit count : 56

dropped conns : 14 <------- Number of attempted connections to this VIP that the ACE discarded client pkt count : 6297 , client byte count: 1047583 server pkt count : 1238 , server byte count: 1325495

L7 Loadbalance policy : LB_WEB_POLICY <------- L7 policy statistics class/match : class-default LB action : serverfarm: SFARM1

hit count : 0 <-------|-- Check these counters to see if they are increasingdropped conns : 0 <-------|

4. Verify that the L4 policy is applied as a service policy to an active interface by entering the followingcommand:

ACE_module5/Admin# show running-config interfaceGenerating configuration....

interface vlan 100 ip address 192.168.120.1 255.255.255.0 access-group input ACL1 access-group output anyone

service-policy input L4WEB_POLICY no shutdown...

5. Check the total conn-dropcount field for the primary server farm in the output of the following command. Also,check the IP address, state, and the connection statistics for each real server that is configured in the server farm.

ACE_module5/Admin# show serverfarm SFARM1 detail

serverfarm : SFARM1, type: HOSTtotal rservers : 3active rservers: 3description : -state : ACTIVE <------- Current state of the server farmpredictor : ROUNDROBIN <------- Load-balancing method weight : - autoadjust : MAXLOADfailaction : -back-inservice : 40partial-threshold : 40num times failover : 0num times back inservice : 0total conn-dropcount : 0 <------- Total number of connection attempts to this server farm that the ACE discarded--------------------------------- ----------connections----------- real weight state current total failures ---+---------------------+------+------------+----------+----------+--------- rserver: SERVER1

192.168.252.245:0 10 INSERVICE 0 0 0 <------- Real server IP address, state, and connection statistics

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 133

Page 134: Ace Troubleshooting

max-conns : 4000000 , out-of-rotation count : 0 min-conns : 4000000 conn-rate-limit : - , out-of-rotation count : - bandwidth-rate-limit : - , out-of-rotation count : - retcode out-of-rotation count : - load value : 0

rserver: SERVER2 192.168.252.246:0 20 INSERVICE 0 0 0 max-conns : 4000000 , out-of-rotation count : 0 min-conns : 4000000 conn-rate-limit : - , out-of-rotation count : - bandwidth-rate-limit : - , out-of-rotation count : - retcode out-of-rotation count : - load value : 0

rserver: SERVER3 192.168.252.247:0 30 INSERVICE 0 0 0 max-conns : 4000000 , out-of-rotation count : 0 min-conns : 4000000 conn-rate-limit : - , out-of-rotation count : - bandwidth-rate-limit : - , out-of-rotation count : - retcode out-of-rotation count : - load value : 0

Note: The connection failures counter increments only if the ACE attempts to load balance a connection andthe ACE does not receive a SYN-ACK from the real server in response to a SYN or if the real serverresponds to the SYN with a RST.

6. Check the L7 load-balance statistics by entering the following command:

ACE_module5/Admin# show stats loadbalance

+------------------------------------------++------- Loadbalance statistics -----------++------------------------------------------+ Total version mismatch : 0 Total Layer4 decisions : 0 Total Layer4 rejections : 0Total Layer7 decisions : 0Total Layer7 rejections : 0

Total Layer4 LB policy misses : 0Total Layer7 LB policy misses : 0

Total times rserver was unavailable : 0 Total ACL denied : 0 Total IDMap Lookup Failures : 0

Note: The ID Map is used to map real servers and server farms between the local and the remote peers in aredundant configuration. The Total IDMap Lookup Failures field increments if the local ACE fails tofind the local ACE to peer ACE ID mapping. A failure can occur if the peer ACE did not send a properremote ID for the local ACE to look up and so the local ACE could not perform a mapping or if the IDMap table was not created.

7. If you are having problems with HTTP, check the HTTP statistics and error counters by entering the followingcommand:

ACE_module5/Admin# show stats http

+------------------------------------------++-------------- HTTP statistics -----------++------------------------------------------+

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 134

Page 135: Ace Troubleshooting

LB parse result msgs sent : 0 , TCP data msgs sent : 0 Inspect parse result msgs : 0 , SSL data msgs sent : 0 sent TCP fin/rst msgs sent : 0 , Bounced fin/rst msgs sent: 0 SSL fin/rst msgs sent : 0 , Unproxy msgs sent : 0 Drain msgs sent : 0 , Particles read : 0 Reuse msgs sent : 0 , HTTP requests : 0 Reproxied requests : 0 , Headers removed : 0 Headers inserted : 0 , HTTP redirects : 0 HTTP chunks : 0 , Pipelined requests : 0 HTTP unproxy conns : 0 , Pipeline flushes : 0 Whitespace appends : 0 , Second pass parsing : 0 Response entries recycled : 0 , Analysis errors : 0 Header insert errors : 0 , Max parselen errors : 0 Static parse errors : 0 , Resource errors : 0 Invalid path errors : 0 , Bad HTTP version errors : 0 Headers rewritten : 0 , Header rewrite errors : 0

8. If you suspect a probe issue, for example, a TCP probe, check the probe statistics and error counters by enteringthe following command:

ACE_module5/Admin# show stats probe type tcp

+------------------------------------------++----------- Probe statistics -------------++------------------------------------------+ ----- tcp probe ---- Total probes sent : 0 Total send failures : 0 Total probes passed : 0 Total probes failed : 0 Total connect errors : 0 Total conns refused : 0 Total RST received : 0 Total open timeouts : 0 Total receive timeout : 0 Total active sockets : 0

9. Check the parameter map statistics for an HTTP parameter map by entering the following command:

ACE_module5/Admin# show parameter-map HTTP_PMAP

Number of parameter-maps : 1

Parameter-map : HTTP_PMAP Type : http server-side connection reuse : disabled case-insensitive parsing : disabled persistence-rebalance : disabled <---------------------------- Enabled by default in the ACE appliance. Enabled by default in the ACE module in software version A4(1.0) and later. header modify per-request : disabled header-maxparse-length : 4096 content-maxparse-length : 4096 parse length-exceed action : drop urlcookie-delimiters : /&#+

10. Clear the L7 load-balancing statistics by entering the following commands:

clear stats loadbalance [radius | rdp]◊ clear service-policy policy_name◊ clear stats http◊ clear rserver server_name◊ clear serverfarm serverfarm_name◊

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 135

Page 136: Ace Troubleshooting

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 136

Page 137: Ace Troubleshooting

This article describes the procedures for troubleshooting redundancy issues with your ACE.

Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference

Contents

1 Overview of ACE Redundancy1.1 Redundancy Protocol♦ 1.2 FT VLAN♦ 1.3 Configuration Requirements andRestrictions

1.4 Example of a Redundancy Configuration♦

2 Troubleshooting ACE Redundancy• 3 FT Peer and Group Status Details

3.1 FT Group Status Conditions3.1.1 STANDBY_COLD◊ 3.1.2 STANDBY_CONFIG◊

3.2 FT Peer Status Conditions3.2.1 PEER_DOWN◊ 3.2.2 TL_ERROR◊ 3.2.3 FT_VLAN_DOWN◊ 3.2.4 FSM_PEER_STATE_ERROR◊

3.3 About WARM_COMPATIBLE andSTANDBY_WARM

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 137

Page 138: Ace Troubleshooting

Overview of ACE Redundancy

Redundancy (or fault tolerance) allows your network to remain operational even if one of the ACEs becomesunresponsive. Redundancy ensures that your network services and applications are always available.

Redundancy provides seamless switchover of flows if an ACE becomes unresponsive or a critical host, interface,or HSRP group (ACE module only) fails. Redundancy supports the following network applications that requirefault tolerance:

Mission-critical enterprise applications• Banking and financial services• E-commerce• Long-lived flows such as FTP and HTTP file transfers•

Redundancy Protocol

You can configure a maximum of two ACE modules (peers) in the same Catalyst 6500 series switch or indifferent chassis for redundancy. You can also configure a maximum of two ACE 4710 appliances forredundancy. Each peer ACE can contain one or more fault-tolerant (FT) groups. Each FT group consists of twomembers: one active context and one standby context. For more information about contexts, see the CiscoApplication Control Engine Module Virtualization Configuration Guide or the Cisco 4700 Series ApplicationControl Engine Appliance Administration Guide (Software Version A3(2.4)). An FT group has a unique group IDthat you assign.

Both ACEs can be active at the same time, processing traffic for distinct virtual devices and backing up each other(stateful redundancy). An Active-Active configuration requires two FT groups and two virtual contexts on eachACE. See Figure 1.

Figure 1. Example of an Active-Active Configuration

The ACE uses the redundancy protocol to communicate between the redundant peers. The election of the activemember within each FT group is based on a priority scheme. The member configured with the higher priority iselected as the active member. If a member with a higher priority is found after the other member becomes active,the new member becomes active because it has a higher priority. This behavior is known as preemption and is

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 138

Page 139: Ace Troubleshooting

enabled by default.

One virtual MAC address (VMAC) is associated with each FT group. The format of the VMAC is:00-0b-fc-fe-1b-groupID. Because a VMAC does not change upon a switchover, the client and server ARP tablesdoes not require updating. The ACE selects a VMAC from a pool of virtual MACs available to it. You canspecify the pool of MAC addresses that the local ACE and the peer ACE use by configuring theshared-vlan-hostid command and the peer shared-vlan-hostid command, respectively. To avoid MAC addressconflicts, be sure that the two pools are different on the two ACEs. For more information about VMACs andMAC address pools, see the Cisco Application Control Engine Module Routing and Bridging ConfigurationGuide.

Each FT group acts as an independent redundancy instance. When a switchover occurs, the active member in theFT group becomes the standby member and the original standby member becomes the active member. Aswitchover can occur for the following reasons:

The active member becomes unresponsive.• A tracked host, interface, or HSRP group fails.• You enter the ft switchover command to force a switchover.•

FT VLAN

Redundancy uses a dedicated FT VLAN between redundant ACEs to transmit flow-state information and theredundancy heartbeat. You must configure this same VLAN on both peer ACEs. You also must configure adifferent IP address within the same subnet on each ACE for the FT VLAN. Cisco recommends twoport-channeled 1-Gigabit Ethernet links for the FT VLAN. For the appliance, when you configure theft-port-vlan command, the ACE modifies the associated Ethernet port or port-channel interface to a trunk port.

Note: Do not use the FT VLAN for any other network traffic, including HSRP traffic and data.The two redundant ACEs constantly communicate over the FT VLAN to determine the operating status of eachACE. The standby member uses the heartbeat packet to monitor the health of the active member. The activemember uses the heartbeat packet to monitor the health of the standby member. Communications over theswitchover link include the following data:

Redundancy protocol packets• State information replication data• Configuration synchronization information• Heartbeat packets•

For multiple contexts, the FT VLAN resides in the system configuration file. Each FT VLAN on the ACE has oneunique MAC address associated with it. The ACE uses these device MAC addresses as the source or destinationMACs for sending or receiving redundancy protocol state and configuration replication packets.

Note: The IP address and the MAC address of the FT VLAN do not change at switchover.

Configuration Requirements and Restrictions

Follow these requirements and restrictions when configuring the redundancy feature:

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 139

Page 140: Ace Troubleshooting

Redundancy is not supported between an ACE module and an ACE appliance operating as peers.Redundancy must be of the same ACE device type and software release.

In bridged mode (Layer 2), two contexts cannot share the same VLAN.• To achieve active-active redundancy, a minimum of two contexts and two FT groups are required on eachACE.

When you configure redundancy, the ACE keeps all interfaces that do not have an IP address in the Downstate. The IP address and the peer IP address that you assign to a VLAN interface should be in the samesubnet but should be different IP addresses. For more information about configuring VLAN interfaces,see the Cisco Application Control Engine Module Routing and Bridging Configuration Guide or theCisco ACE 4700 Series Appliance Routing and Bridging Configuration Guide

FT Interfaces are put into an automatic trunk status and, for the module, the Catalyst 6500 series switchneeds to be set to trunk the specific VLAN you are using for the FT interface.

Example of a Redundancy Configuration

The following example shows a running-configuration file that defines fault tolerance (FT) for a single ACEoperating in a redundancy configuration. You must configure a maximum of two ACEs (peers) for redundancy tofail over from the active ACE to the standby ACE.

Note: All FT parameters are configured in the Admin context.This configuration addresses the following redundancy components:

A dedicated FT VLAN for communication between the members of an FT group. You must configurethis same VLAN on both peers.

An FT peer definition.• An FT group that is associated with the Admin context.• A critical tracking and failure detection process for an interface.•

access-list ACL1 line 10 extended permit ip any any

class-map type management match-any L4_REMOTE-MGT_CLASS 2 match protocol telnet any 3 match protocol ssh any 4 match protocol icmp any 5 match protocol http any 7 match protocol snmp any 8 match protocol https any

policy-map type management first-match L4_REMOTE-MGT_POLICY class L4_REMOTE-MGT_CLASS permit

interface vlan 100 ip address 192.168.83.219 255.255.255.0 peer ip address 192.168.83.230 255.255.255.0 alias 192.168.83.200 255.255.255.0 access-group input ACL1 service-policy input L4_REMOTE-MGT_POLICY no shutdown

ft interface vlan 200 ip address 192.168.1.1 255.255.255.0 peer ip address 192.168.1.2 255.255.255.0 no shutdown

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 140

Page 141: Ace Troubleshooting

ft peer 1 ft-interface vlan 200 heartbeat interval 300 heartbeat count 10

ft group 1 peer 1 priority 200 associate-context Admin inservice

ft track interface TRACK_VLAN100 track-interface vlan 100 peer track-interface vlan 200 priority 50 peer priority 5

ip route 0.0.0.0 0.0.0.0 192.168.83.1

Troubleshooting ACE Redundancy

This section describes the methods and CLI commands that you can use to troubleshoot redundancy issues in yourACE.

1. Ensure that the software versions and licenses installed in the two ACEs are identical. A software or licensemismatch may generate the following syslog message:

%ACE-1-727006: HA: Peer is incompatible due to error str. Cannot be Redundant.

To verify the software (SRG) and license compatibility of the FT peer, enter the following command:

ACE_5/Admin# show ft peer status

Peer Id : 1State : FSM_PEER_STATE_MY_IPADDRMaintenance mode : MAINT_MODE_OFFSRG Compatibility : COMPATIBLELicense Compatibility : COMPATIBLEFT Groups : 1

If the software or license is incompatible, install the appropriate software image or license in the peer to correctthe problem.

2. Ensure that any SSL certificates (certs) and keys that exist in the active ACE are also configured in the standbyACE. SSL certs and keys are not synchronized automatically from the active to the standby. Use the cryptoexport and crypto import commands to accomplish this task. This requirement also applies to scripts andscripted probes. Failure to keep the active and standby configurations identical will cause configurationsynchronization to fail and may cause the standby ACE to enter the STANDBY_COLD state.

3. The ACE sends heartbeat packets via UDP over the FT VLAN between peers. When heartbeats are notreceived during the specified interval (the interval and count are configurable), the ACE notifies the HA processoron the CP by sending a Peer_Down interprocess communication protocol (IPCP) message. If a peer is down orunreachable, you may receive one of the following syslog messages:

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 141

Page 142: Ace Troubleshooting

%ACE-1-727001: HA: Peer IP address is not reachable. Error: error str

%ACE-1-727002: HA: FT interface interface name to reach peer IP address is down. Error: error str

4. Verify connectivity between the peers over the FT VLAN. If a peer device is physically up but connectivity isthe problem, you may end up with two active devices. If connectivity is lost due to the peer going down, rebootthe peer to restore redundancy between the two devices.

5. Display heartbeat statistics, including missed heartbeats, by entering the following command:

ACE_5/Admin# show ft statsHA Heartbeat Statistics------------------------

Number of Heartbeats Sent : 0Number of Heartbeats Received : 0Number of Heartbeats Missed : 0Number of Unidirectional HB's Received : 0Number of HB Timeout Mismatches : 0Num of Peer Up Events Sent : 0Num of Peer Down Events Sent : 0Successive HB's miss Intervals counter : 0Successive Uni HB's recv counter : 0

6. Provide an alternate path for the ACE to check the peer's status in case of missed heartbeats and configure aquery interface using the followng commands:

ACE_5/Admin# configEnter configuration commands, one per line. End with CNTL/Z.ACE_5/Admin(config)# ft peer 1ACE_5/Admin(config-ft-peer)# query-interface vlan 100

If the query interface is configured, upon receiving a PEER_DOWN message from the heartbeat process, theACE data plane attempts to ping the peer using the Query VLAN. If the ping fails, the standby transitions to theACTIVE state. If the ping is successful, the standby transitions to the STANDBY_COLD state. To recover fromthe STANDBY_COLD state, reboot the standby.

7. Each peer uses a VMAC that is dependent on the FT group number. If you are using multiple ACE modules inthe same chassis, be careful when you configure the same FT groups in more than one module.

Display the VMAC for an FT group by entering the following command:

ACE_5/Admin# show interface internal iftable vlan100vlan100--------ifid: 6Context: 0ifIndex: 16777316physid: 100rmode: 0 (unknown)iftype: 0 (vlan)bvi_bgid: 0MTU: 1500MAC: 00:18:b9:a6:91:15VMAC: 00:00:00:00:00:00 <------- Virtual MAC AddressFlags: 0x8a000800 (valid, down, admin-down, Non-redundant, tracked)ACL In: 0

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 142

Page 143: Ace Troubleshooting

ACL Out: 0Route ID: 0FTgroupID: 0Zone ID: 6Sec Level: 0L2 ACL: bpdu DENY, ipv6 DENY, mpls DENY, all DENY

LastChange: 0 (Thu Jan 1 00:00:00 1970)iflookup index: 100vlan-vmac index:0Next Shared IF: 0Lock: Unlocked, seq 5Lock errors: 0Unlock errors: 0No. of times locked: 5No. of times unlocked: 5Current/last owner: 0x40a7fc

8. If the members of an FT group are unable to reach the ACTIVE or the STANDBY_HOT state, there may be acontext name mismatch for the same FT group. You may receive the following syslog message:

%ACE-1-727003: HA: Mismatch in context names detected for FT group FTgroupID. Cannot be redundant.

Be sure that the context names within the same FT group are identical on both ACEs.

9. Check the FT group configuration on both devices. Make sure that both devices are associated with the samecontext. Enter the following command:

ACE_5/Admin# show running-config ft

10. Verify the FT peer status and configuration by entering the following command:

ACE_5/Admin# show ft peer detail

Peer Id : 1State : FSM_PEER_STATE_COMPATIBLEMaintenance mode : MAINT_MODE_OFFFT Vlan : 100FT Vlan IF State : DOWNMy IP Addr : 10.1.1.1Peer IP Addr : 10.1.1.2Query Vlan : 110Query Vlan IF State : DOWNPeer Query IP Addr : 172.25.91.202Heartbeat Interval : 300Heartbeat Count : 20Tx Packets : 318573Tx Bytes : 66301061Rx Packets : 318540Rx Bytes : 66272840Rx Error Bytes : 0Tx Keepalive Packets : 318480Rx Keepalive Packets : 318480TL_CLOSE count : 0FT_VLAN_DOWN count : 0PEER_DOWN count : 0SRG Compatibility : COMPATIBLELicense Compatibility : COMPATIBLEFT Groups : 3

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 143

Page 144: Ace Troubleshooting

11. Verify the FT group status and configuration by entering the following command:

ACE_5/Admin# show ft group detail

FT Group : 1No. of Contexts : 1Configured Status : in-serviceMaintenance mode : MAINT_MODE_OFFMy State : FSM_FT_STATE_ACTIVEMy Config Priority : 110My Net Priority : 110My Preempt : EnabledPeer State : FSM_FT_STATE_STANDBYPeer Config Priority : 100Peer Net Priority : 100Peer Preempt : EnabledPeer Id : 1Last State Change time : Thu Apr 2 00:00:00 2009Running cfg sync enabled : EnabledRunning cfg sync status : Running configuration sync has completedStartup cfg sync enabled : EnabledStartup cfg sync status : Running configuration sync has completedBulk sync done for ARP: 0Bulk sync done for LB: 0Bulk sync done for ICM: 0

For information on troubleshooting the FT group status, see the "FT Group Status Conditions"

FT Peer and Group Status Details

This section describes how to diagnose unexpected status conditions for the FT group and FT peer. Thisinformation may enable you to troubleshoot an issue directly or help you to provide additional information to yourCisco support representative.

FT Group Status Conditions

This section describes how to diagnose and troubleshoot unexpected status conditions applicable to the FT groupstatus.

STANDBY_COLD

An FT group status of STANDBY_COLD may appear when:

Config sync fails (including, incr-sync and bulk-sync), or• FT VLAN is down while the query interface is up•

Config Sync Failure

In configuration synchronization fails, the peers are not correctly exchanging configuration information. Thisfailure can be identified as follows:

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 144

Page 145: Ace Troubleshooting

Output of the show ft peer detail command shows that the peer state is "Compatible".1. Entering show ft group detail shows that the FT group is in "Standby Cold" mode and entering cfg syncstatus shows the reason for the failure. For incr-sync failure, the output shows exactly which commandresulted in an execution error on the standby. For a bulk-sync failure, the reason is "Error on Standbydevice when applying configuration file replicated from active".

2.

To further investigate a bulk-sync failure, perform these steps on the standby device:3.

For software version A2(2.0) and earlier and version A2(1.3) and earlier releases, from theAdmin context, enter show ft history cfg_cntlr and grep for "error:" to find any CLI commandsthat caused execution errors.

For later releases, enter show ft config-error ctx_name to view the failed CLI commands.◊

To work around a bulk sync failure, perform these steps to remove the CLI commands that triggered the error (asidentified from the preceding analysis) and then retrigger the bulk sync operation, as follows:

Retrigger bulk sync by disabling config sync with the no ft auto-sync running command.1. Re-enable config sync with ft auto-sync running.2.

If the problem persists, repeat the above sequence until you eliminate the CLI command that triggered theproblem.

FT VLAN Down with Query Interface Up

This condition can be identified by:

Entering show ft peer detail, which shows a peer state of FT_VLAN_DOWN.1. Entering show ft stats, which shows that heartbeats are being missed.2.

In this case, check the physical connectivity of the device. It might be a physical port or cable issue.

STANDBY_CONFIG

If a device appears to be stuck in the STANDBY_CONFIG state:

Run show ft history cfg_cntlr to determine whether the peer devices successfully exchangednotifications regarding configuration synchronization.

1.

Grep for the keywords MTS_OPC_REQ_CFG_DNLD_STATUS andMTS_OPC_CFG_DNLD_STATUS.

2.

If one or both of the messages are missing, an error occurred in the synchronization exchange process.

Note that once it is stuck in the STANDBY_CONFIG state, configuration mode will be disabled on both theactive and standby devices. It can be stuck in this state for up to 4 hours, after which a timeout period expires.

FT Peer Status Conditions

This section describes how to diagnose and troubleshoot unexpected status conditions applicable to the FT peer.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 145

Page 146: Ace Troubleshooting

PEER_DOWN

If the peer status shows PEER_DOWN:

Check whether IP addresses for the local and peer device are configured correctly on both.1. Verify that pinging or telneting to the peer IP address works. If ping fails, check whether the interface isup (show interface). If so, the interface VLAN is probably not allocated to the ACE module on thesupervisor (which suggests a configuration issue on the supervisor).

2.

Enter show arp to see if the FT peer IP address is resolved. (If ARP is not resolved and ping/telnet alsofailed, it might be an encapsulation issue requiring support).

3.

Enter show conn on both sides to see if HA connections have been set up. If connections have not beenset up, check the HA DP manager log (show ft history ha_dp_mgr). Setup may have failed for variousreasons. If this is the case, contact Cisco technical support.

4.

Enter show ft stats on both devices to see if heartbeats are being sent or received. If the Number ofHeartbeats Missed counter is incrementing, the heartbeat packets could be getting dropped. Enter shownp 1 me-stats -sfp to see if heartbeat packets are being received and forwarded to X-Scale, as indicatedby the counter Packets forward to XScale. If this counter is not incrementing, provide the information toCisco technical support.

5.

TL_ERROR

This state may occur when the telnet connection used to exchange configuration information between the peerscannot be established but heartbeat packets are exchanged successfully. To identify this issue:

Verify that heartbeats are flowing by checking the statistics, show ft stats.1. Attempt to connect by telnet or to ping the FT peer. The telnet connection attempt will likely fail.2. Run show arp to see if the FT peer IP address can be resolved.3.

If show arp indicates that the address is not resolvable and the ping or telnet connect attempts fail, it is likely anencapsulation issue on the ACE.

FT_VLAN_DOWN

This state typically occurs when the FT VLAN goes down while the query interface is up. If the heartbeatexchange fails and the query interface is determined to be up based on an ICMP message check, the status isFT_VLAN_DOWN.

To verify, attempt to connect to the FT VLAN Peer IP address by ping or telnet.

If running show ft stats indicates that heartbeats are being missed, it is likely a physical connectivity issue, suchas the physical port or cable failure.

FSM_PEER_STATE_ERROR

This indicates a Software Relationship Graph (SRG) version inconsistency between the peers. See the relationshipgraph table in the following section.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 146

Page 147: Ace Troubleshooting

About WARM_COMPATIBLE and STANDBY_WARM

While peers in a redundant configuration are designed to operate with identical versions of the software, whenyou are upgrading or downgrading the software in the ACEs, it is possible for the peers to temporarily employdifferent software versions. The WARM_COMPATIBLE and STANDBY_WARM redundancy states helpminimize the operational impact of CLI compatibility issues between the peers, and allow failovers to occur on abest-effort basis during such transitions.

When you upgrade or downgrade the ACE software in a redundant configuration with different software versions,the STANDBY_WARM and WARM_COMPATIBLE states allow the configuration and state synchronizationprocess between the peers to continue on a best-effort basis. This basis allows the active ACE to synchronizeconfiguration and state information with the standby even though the standby may not recognize or understandthe CLI commands or state information.

In the STANDBY_WARM state, as with the STANDBY_HOT state, configuration mode is disabled on thestandby ACE and configuration and state synchronization continues. A failover from the active to the standbybased on priorities and preempt can still occur while the standby is in the STANDBY_WARM state. However,while stateful failover is possible for a WARM standby, it is not guaranteed. In general, modules should beallowed to remain in this state only for a short period of time.

When redundancy peers run different software versions, the SRG compatibility field shown by the show ft peerstatus command output displays WARM_COMPATIBLE instead of COMPATIBLE. When the peer is in theWARM_COMPATIBLE state, the FT groups in the standby transition to the STANDBY_WARM state instead ofthe STANDBY_HOT state.

The following software version combinations tables indicate whether the SRG compatibility field will displayWARM_COMPATIBLE (WC) or COMPATIBLE (C):

ACE Module: C = COMPATIBLE / WC = WARM_COMPATIBLEActive(Column)/Standby(Row) A2(1.5) A2(1.6) A2(2.0) A2(2.1) A2(2.2) A2(3.0) A2(2.3)A2(1.5) C WC C C WC WC WCA2(1.6) WC C C WC WC WC WCA2(2.0) C C C C C C CA2(2.1) C WC C C WC WC WCA2(2.2) WC WC C WC C WC WCA2(3.0) WC WC C WC WC C WCA2(2.3) WC WC C WC WC WC CACE Appliance: C = COMPATIBLE / WC = WARM_COMPATIBLEActive(Column)/Standby(Row) A3(1.0) A3(2.0) A3(2.1) A3(2.2) A3(2.3) A3(2.4)A3(1.0) C C C C WC WCA3(2.0) C C C C WC WCA3(2.1) C C C C WC WCA3(2.2) C C C C WC WCA3(2.3) WC WC WC WC C WCA3(2.4) WC WC WC WC WC C

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 147

Page 148: Ace Troubleshooting

This article describes the process and CLI commands for troubleshooting SSL in the ACE.

Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference

Contents

1 Overview of ACE SSL Troubleshooting1.1 Example of an SSLTermination Configuration

1.2 Example of an SSL InitiationConfiguration

2 Troubleshooting ACE SSL•

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 148

Page 149: Ace Troubleshooting

Overview of ACE SSL Troubleshooting

Secure Sockets Layer (SSL) runs over TCP. After the TCP three-way handshake completes and the ACE hasproxied the connection, the SSL handshake takes place. For information about proxied connections, see theTroubleshooting Connectivity article. See Figure 1 for an illustration of the SSL handshake.

Figure 1. SSL Handshake

The ACE supports the following SSL configurations (see Figure 2):

SSL termination (ACE acts as an SSL server)• SSL initiation (ACE acts as a client)• End-to-end SSL (SSL termination plus SSL initiation)•

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 149

Page 150: Ace Troubleshooting

Figure 2. SSL Configurations

Before you begin to troubleshoot potential SSL issues, be sure that the following conditions exist:

You have configured basic SLB and SSL on your ACE. For details about configuring SLB, see the CiscoApplication Control Engine Module Server Load-Balancing Configuration Guide or the Cisco ACE 4700Series Appliance Server Load-Balancing Configuration Guide. For details about configuring SSL, see theCisco Application Control Engine Module SSL Configuration Guide or the Cisco ACE 4700 SeriesAppliance SSL Configuration Guide.

If you are running multiple ACEs in a redundant configuration, be sure that you have copied the SSLcertificates (certs) and keys to the standby ACE. Certs and keys are not replicated in a redundantconfiguration from the active ACE to the standby ACE. Also, ensure that the configurations on the activeand the standby are identical, including the same licenses and software versions.

Be sure that the certs and keys are no larger than 4096 bits and that they are of an RSA type supported bythe ACE. For details about configuring SSL, see the Cisco Application Control Engine Module SSLConfiguration Guide or the Cisco ACE 4700 Series Appliance SSL Configuration Guide. The ACEsupports the following RSA key pair sizes:

512 (least security)◊

768 (normal security)◊

1024 (high security, level 1)◊

1536 (high security, level 2)◊

2048 (high security, level 3)◊

4096 (high security, level 4) - For software release A2(2.4) and later in the ACE module andsoftware release A3(2.6) and later in the ACE appliance, you can use 4096-bit SSL certificates inchaingroups and authgroups. You can also import public certificates and keys that are 4096 bits in

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 150

Page 151: Ace Troubleshooting

length.

Server certs are valid, installed, and have not expired•

Example of an SSL Termination Configuration

The following example shows a running-configuration file of the ACE acting as an SSL proxy server; terminatingSSL or TLS connections from a client and then establishing a TCP connection to an HTTP server. When the ACEterminates the SSL or TLS connection, it decrypts the cipher text from the client and transmits the data as cleartext to the HTTP server.

access-list ACL1 line 10 extended permit ip any any

probe http GEN-HTTP port 80 interval 50 faildetect 5 expect status 200 200

rserver SERVER1 ip address 10.1.0.11 inservicerserver SERVER2 ip address 10.1.0.12 inservicerserver SERVER3 ip address 10.1.0.13 inservicerserver SERVER4 ip address 10.1.0.14 inservicerserver SERVER5 ip address 10.1.0.15 inservicerserver SERVER6 ip address 10.1.0.16 inservicerserver SERVER7 ip address 10.1.0.17 inservicerserver SERVER8 ip address 10.1.0.18 inservice

serverfarm host SFARM1 description SERVER FARM 1 FOR SSL TERMINATION probe GEN_HTTP rserver SERVER1 80 inservice rserver SERVER2 80 inservice rserver SERVER3 80 inservice rserver SERVER4 80 inservice

serverfarm host SFARM2 description SERVER FARM 2 FOR SSL TERMINATION

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 151

Page 152: Ace Troubleshooting

probe GEN_HTTP rserver SERVER5 80 inservice rserver SERVER6 80 inservice rserver SERVER7 80 inservice rserver SERVER8 80 inservice

parameter-map type ssl PARAMMAP_SSL_TERMINATION cipher RSA_WITH_3DES_EDE_CBC_SHA cipher RSA_WITH_AES_128_CBC_SHA priority 2 cipher RSA_WITH_AES_256_CBC_SHA priority 3 version allparameter-map type connection TCP_PARAM syn-data drop exceed-mss allow

ssl-proxy service SSL_PSERVICE_SERVER ssl advanced-options PARAMMAP_SSL_TERMINATION key MYKEY.PEM cert MYCERT.PEM

class-map type http loadbalance match-all L7_SERVER_CLASS description Sticky for SSL Testing 2 match http url .*.jpg 3 match source-address 192.168.130.0 255.255.255.0class-map type http loadbalance match-all L7_SLB-HTTP_CLASS 2 match http url .* 3 match source-address 192.168.130.0 255.255.255.0class-map match-all L4_SSL-TERM_CLASS description SSL Termination VIP 2 match virtual-address 192.168.130.11 tcp eq https

policy-map type loadbalance first-match L7_SSL-TERM_POLICY class L7_SERVER_CLASS serverfarm SFARM1 insert-http I_AM header-value "SSL_TERM" insert-http SRC_Port header-value "%ps" insert-http DEST_IP header-value "%id" insert-http DEST_Port header-value "%pd" insert-http SRC_IP header-value "is" class L7_SLB-HTTP_CLASS serverfarm SFARM1 insert-http I_AM header-value "SSL_TERM" insert-http SRC_Port header-value "%ps" insert-http DEST_IP header-value "%id" insert-http DEST_Port header-value "%pd" insert-http SRC_IP header-value "is"

policy-map multi-match L4_SSL-VIP_POLICY class L4_SSL-TERM_CLASS loadbalance vip inservice loadbalance policy L7_SSL-TERM_POLICY loadbalance vip icmp-reply ssl-proxy server SSL_PSERVICE_SERVER connection advanced-options TCP_PARAM

interface vlan 120 description Upstream VLAN_120 - Clients and VIPs ip address 192.168.120.1 255.255.255.0

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 152

Page 153: Ace Troubleshooting

fragment chain 20 fragment min-mtu 68 access-group input ACL1 nat-pool 1 192.168.120.70 192.168.120.80 netmask 255.255.255.0 pat service-policy input L4_SSL-VIP_POLICY no shutdown

ip route 10.1.0.0 255.255.255.0 192.168.120.254

Example of an SSL Initiation Configuration

The following example shows a running-configuration file of the ACE acting as an SSL proxy client, initiatingand maintaining an SSL connection between itself and an SSL server. The ACE receives clear text from an HTTPclient, and then encrypts and transmits the data as cipher text to the SSL server. On the reverse side, the ACEdecrypts the cipher text that it receives from the SSL server and sends the data to the client as clear text.

access-list ACL1 line 10 extended permit ip any any

probe http GEN-HTTP port 80 interval 50 faildetect 5 expect status 200 200

rserver SERVER1 ip address 10.1.0.11 inservicerserver SERVER2 ip address 10.1.0.12 inservicerserver SERVER3 ip address 10.1.0.13 inservicerserver SERVER4 ip address 10.1.0.14 inservicerserver SERVER5 ip address 10.1.0.15 inservicerserver SERVER6 ip address 10.1.0.16 inservicerserver SERVER7 ip address 10.1.0.17 inservicerserver SERVER8 ip address 10.1.0.18 inservice

serverfarm host SFARM1 description SERVER FARM 1 FOR SSL INITIATION probe GEN_HTTP rserver SERVER1 443 inservice rserver SERVER2 443 inservice rserver SERVER3 443 inservice rserver SERVER4 443

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 153

Page 154: Ace Troubleshooting

inservice

serverfarm host SFARM2 description SERVER FARM 2 FOR SSL TERMINATION probe GEN_HTTP rserver SERVER5 443 inservice rserver SERVER6 443 inservice rserver SERVER7 443 inservice rserver SERVER8 443 inservice

parameter-map type http PARAMMAP_HTTP server-conn reuse case-insensitive persistence-rebalanceparameter-map type ssl PARAMMAP_SSL_INITIATION cipher RSA_WITH_RC4_128_MD5 cipher RSA_WITH_RC4_128_SHA cipher RSA_WITH_DES_CBC_SHA cipher RSA_WITH_3DES_EDE_CBC_SHA cipher RSA_WITH_AES_128_CBC_SHA cipher RSA_WITH_AES_256_CBC_SHA cipher RSA_EXPORT_WITH_RC4_40_MD5 cipher RSA_EXPORT1024_WITH_RC4_56_MD5 cipher RSA_EXPORT_WITH_DES40_CBC_SHA cipher RSA_EXPORT1024_WITH_DES_CBC_SHA cipher RSA_EXPORT1024_WITH_RC4_56_SHA version allparameter-map type connection TCP_PARAM syn-data drop exceed-mss allow

ssl-proxy service SSL_PSRVICE_CLIENT ssl advanced-options PARAMMAP_SSL_INITIATION

class-map type http loadbalance match-all L7_SERVER_CLASS description Sticky for SSL Testing 2 match http url .*.jpg 3 match source-address 192.168.130.0 255.255.255.0class-map type http loadbalance match-all L7_SLB-HTTP_CLASS 2 match http url .* 3 match source-address 192.168.130.0 255.255.255.0class-map match-all L4_SSL-INIT_CLASS description SSL Initiation VIP 2 match virtual-address 192.168.130.12 tcp eq www

policy-map type loadbalance first-match L7_SSL-INIT_POLICY class L7_SERVER_CLASS serverfarm SFARM1 insert-http SRC_IP header-value "%is" insert-http I_AM header-value "SSL_INIT" insert-http SRC_Port header-value "%ps" insert-http DEST_IP header-value "%id" insert-http DEST_Port header-value "%pd" ssl-proxy client SSL_PSERVICE_CLIENT class L7_SLB-HTTP_CLASS serverfarm SFARM2 insert-http SRC_IP header-value "%is" insert-http I_AM header-value "SSL_INIT"

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 154

Page 155: Ace Troubleshooting

insert-http DEST_Port header-value "%pd" insert-http DEST_IP header-value "%id" insert-http SRC_Port header-value "%ps" ssl-proxy client SSL_PSERVICE_CLIENTpolicy-map multi-match L4_SSL-VIP_POLICY class L4_SSL-INIT_CLASS loadbalance vip inservice loadbalance policy L7_SSL-INIT_POLICY loadbalance vip icmp-reply active appl-parameter http advanced-options PARAMMAP_HTTP connection advanced-options TCP_PARAM

interface vlan 120 description Upstream VLAN_120 - Clients and VIPs ip address 192.168.120.1 255.255.255.0 fragment chain 20 fragment min-mtu 68 access-group input ACL1 nat-pool 1 192.168.120.70 192.168.120.80 netmask 255.255.255.0 pat service-policy input L4_SSL-VIP_POLICY no shutdown

ip route 10.1.0.0 255.255.255.0 192.168.120.254

Troubleshooting ACE SSL

To troubleshoot SSL issues, follow these steps:

1. For the ACE module, check the health of the Nitrox-II (crypto module) and ensure that it has not becomeunresponsive. Stop all traffic, and then enter the following command:

ACE_module5/Admin# show crypto hardware

Figure 3. Example of the Show Crypto Hardware Command Output for an Unresponsive Crypto Module

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 155

Page 156: Ace Troubleshooting

STX1 is a count of the number of packets transmitted by the Nitrox-II and IMX1 is the number of packetsreceived by the Nitrox-II. On a normal system, these values should be the same once traffic has stopped. If thevalues are not the same, the Nitrox-II has become unresponsive.

The Nitrox-II uses 0x500 TX buffers to transmit packets and 0x200 RX buffers to receive packets. If the [TR]XBuffers used count ever exceeds the amount available, the Nitrox-II has become unresponsive.

The available cores field shows which of the 22 cores of the Nitrox-II are active. When no traffic is flowing, thereshould be no numbers following the Using: statement. If there are, as in the sample output above, then that core (0in this case) is hung, and the Nitrox-II has become unresponsive.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 156

Page 157: Ace Troubleshooting

For the POM count, there are two numbers, A(B). The "A" value is the number of outstanding packets to thePacket Order Manager, while the "B" value, counts the number of packets that have been processed in the lastsecond. When no traffic is flowing, both of these values should be 0. If no traffic is flowing, and the value of "A"is nonzero as shown above, then there are outstanding requests to the POM that are not being processed, becausethe Nitrox-II has become unresponsive.

2. Ensure that appropriate ports are designated for PAT in an SSL termination configuration. By default,connections to the real server from the ACE will inherit the destination port from the client to VIP connection sothat a connection to port 443 on the VIP will go to port 443 on the real server, unless otherwise specified in theserver farm configuration. This will cause problems if you are using ACE to offload SSL between the client andthe VIP and send clear-text traffic to the real servers. The following example demonstrates a port definition in aserver farm configuration:

serverfarm host sf1 probe HTTP_PROBE rserver rs1 80 inservice rserver rs2 80 inservice

3. Verify that the SSL certificate and key are correct by entering the following command:

ACE_module5/Admin# crypto verify key cert

4. Verify that a certificate revocation list (CRL) has been downloaded, enter the following command:

ACE_module5/Admin# show crypto crl test1

test1:URL: http://192.168.12.23/test.crlLast Downloaded: not downloaded yetTotal Number Of Download Attempts: 0Failed Download Attempts: 0

5. Verify the contents of an authgroup by entering the following command:

ACE_module5/Admin# show crypto authgroup authgroup_name

6. Display client SSL statistics by entering the the following command:

ACE_module5/Admin# show stats crypto client

+----------------------------------------------++---- Crypto client termination statistics ----++----------------------------------------------+SSLv3 negotiated protocol: 0TLSv1 negotiated protocol: 0SSLv3 full handshakes: 0SSLv3 resumed handshakes: 0SSLv3 rehandshakes: 0TLSv1 full handshakes: 0TLSv1 resumed handshakes: 0TLSv1 rehandshakes: 0SSLv3 handshake failures: 0SSLv3 failures during data phase: 0TLSv1 handshake failures: 0TLSv1 failures during data phase: 0

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 157

Page 158: Ace Troubleshooting

Handshake Timeouts: 0total transactions: 0SSLv3 active connections: 0SSLv3 connections in handshake phase: 0SSLv3 conns in renegotiation phase: 0SSLv3 connections in data phase: 0TLSv1 active connections: 0 TLSv1 connections in handshake phase: 0TLSv1 conns in renegotiation phase: 0TLSv1 connections in data phase: 0

+----------------------------------------------++------- Crypto client alert statistics -------++----------------------------------------------+SSL alert CLOSE_NOTIFY rcvd: 0SSL alert UNEXPECTED_MSG rcvd: 0SSL alert BAD_RECORD_MAC rcvd: 0SSL alert DECRYPTION_FAILED rcvd: 0SSL alert RECORD_OVERFLOW rcvd: 0SSL alert DECOMPRESSION_FAILED rcvd: 0SSL alert HANDSHAKE_FAILED rcvd: 0SSL alert NO_CERTIFICATE rcvd: 0SSL alert BAD_CERTIFICATE rcvd: 0SSL alert UNSUPPORTED_CERTIFICATE rcvd: 0SSL alert CERTIFICATE_REVOKED rcvd: 0SSL alert CERTIFICATE_EXPIRED rcvd: 0SSL alert CERTIFICATE_UNKNOWN rcvd: 0SSL alert ILLEGAL_PARAMETER rcvd: 0SSL alert UNKNOWN_CA rcvd: 0SSL alert ACCESS_DENIED rcvd: 0SSL alert DECODE_ERROR rcvd: 0SSL alert DECRYPT_ERROR rcvd: 0SSL alert EXPORT_RESTRICTION rcvd: 0SSL alert PROTOCOL_VERSION rcvd: 0SSL alert INSUFFICIENT_SECURITY rcvd: 0SSL alert INTERNAL_ERROR rcvd: 0SSL alert USER_CANCELED rcvd: 0SSL alert NO_RENEGOTIATION rcvd: 0SSL alert CLOSE_NOTIFY sent: 0SSL alert UNEXPECTED_MSG sent: 0SSL alert BAD_RECORD_MAC sent: 0SSL alert DECRYPTION_FAILED sent: 0SSL alert RECORD_OVERFLOW sent: 0SSL alert DECOMPRESSION_FAILED sent: 0SSL alert HANDSHAKE_FAILED sent: 0SSL alert NO_CERTIFICATE sent: 0SSL alert BAD_CERTIFICATE sent: 0SSL alert UNSUPPORTED_CERTIFICATE sent: 0SSL alert CERTIFICATE_REVOKED sent: 0SSL alert CERTIFICATE_EXPIRED sent: 0SSL alert CERTIFICATE_UNKNOWN sent: 0SSL alert ILLEGAL_PARAMETER sent: 0SSL alert UNKNOWN_CA sent: 0SSL alert ACCESS_DENIED sent: 0SSL alert DECODE_ERROR sent: 0SSL alert DECRYPT_ERROR sent: 0SSL alert EXPORT_RESTRICTION sent: 0SSL alert PROTOCOL_VERSION sent: 0SSL alert INSUFFICIENT_SECURITY sent: 0SSL alert INTERNAL_ERROR sent: 0SSL alert USER_CANCELED sent: 0

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 158

Page 159: Ace Troubleshooting

SSL alert NO_RENEGOTIATION sent: 0

+-----------------------------------------------++--- Crypto client authentication statistics ---++-----------------------------------------------+Total SSL client authentications: 0Failed SSL client authentications: 0SSL client authentication cache hits: 0SSL static CRL lookups: 0SSL best effort CRL lookups: 0SSL CRL lookup cache hits: 0SSL revoked certificates: 0Total SSL server authentications: 0Failed SSL server authentications: 0

+-----------------------------------------------++------- Crypto client cipher statistics -------++-----------------------------------------------+Cipher sslv3_rsa_rc4_128_md5: 0Cipher sslv3_rsa_rc4_128_sha: 0Cipher sslv3_rsa_des_cbc_sha: 0Cipher sslv3_rsa_3des_ede_cbc_sha: 0Cipher sslv3_rsa_exp_rc4_40_md5: 0Cipher sslv3_rsa_exp_des40_cbc_sha: 0Cipher sslv3_rsa_exp1024_rc4_56_md5: 0Cipher sslv3_rsa_exp1024_des_cbc_sha: 0Cipher sslv3_rsa_exp1024_rc4_56_sha: 0Cipher sslv3_rsa_aes_128_cbc_sha: 0Cipher sslv3_rsa_aes_256_cbc_sha: 0Cipher tlsv1_rsa_rc4_128_md5: 0Cipher tlsv1_rsa_rc4_128_sha: 0Cipher tlsv1_rsa_des_cbc_sha: 0Cipher tlsv1_rsa_3des_ede_cbc_sha: 0Cipher tlsv1_rsa_exp_rc4_40_md5: 0Cipher tlsv1_rsa_exp_des40_cbc_sha: 0Cipher tlsv1_rsa_exp1024_rc4_56_md5: 0Cipher tlsv1_rsa_exp1024_des_cbc_sha: 0Cipher tlsv1_rsa_exp1024_rc4_56_sha: 0Cipher tlsv1_rsa_aes_128_cbc_sha: 0Cipher tlsv1_rsa_aes_256_cbc_sha: 0

7. Display SSL server statistics by entering the following command:

ACE_module5/Admin# show stats crypto server

+----------------------------------------------++---- Crypto server termination statistics ----++----------------------------------------------+SSLv3 negotiated protocol: 0TLSv1 negotiated protocol: 0SSLv3 full handshakes: 0SSLv3 resumed handshakes: 0SSLv3 rehandshakes: 0TLSv1 full handshakes: 0TLSv1 resumed handshakes: 0TLSv1 rehandshakes: 0SSLv3 handshake failures: 0SSLv3 failures during data phase: 0TLSv1 handshake failures: 0TLSv1 failures during data phase: 0

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 159

Page 160: Ace Troubleshooting

Handshake Timeouts: 0total transactions: 0SSLv3 active connections: 0SSLv3 connections in handshake phase: 0SSLv3 conns in renegotiation phase: 0SSLv3 connections in data phase: 0TLSv1 active connections: 0TLSv1 connections in handshake phase: 0TLSv1 conns in renegotiation phase: 0TLSv1 connections in data phase: 0

+----------------------------------------------++------- Crypto server alert statistics -------++----------------------------------------------+SSL alert CLOSE_NOTIFY rcvd: 0SSL alert UNEXPECTED_MSG rcvd: 0SSL alert BAD_RECORD_MAC rcvd: 0SSL alert DECRYPTION_FAILED rcvd: 0SSL alert RECORD_OVERFLOW rcvd: 0SSL alert DECOMPRESSION_FAILED rcvd: 0SSL alert HANDSHAKE_FAILED rcvd: 0SSL alert NO_CERTIFICATE rcvd: 0SSL alert BAD_CERTIFICATE rcvd: 0SSL alert UNSUPPORTED_CERTIFICATE rcvd: 0SSL alert CERTIFICATE_REVOKED rcvd: 0SSL alert CERTIFICATE_EXPIRED rcvd: 0SSL alert CERTIFICATE_UNKNOWN rcvd: 0SSL alert ILLEGAL_PARAMETER rcvd: 0SSL alert UNKNOWN_CA rcvd: 0SSL alert ACCESS_DENIED rcvd: 0SSL alert DECODE_ERROR rcvd: 0SSL alert DECRYPT_ERROR rcvd: 0SSL alert EXPORT_RESTRICTION rcvd: 0SSL alert PROTOCOL_VERSION rcvd: 0SSL alert INSUFFICIENT_SECURITY rcvd: 0SSL alert INTERNAL_ERROR rcvd: 0SSL alert USER_CANCELED rcvd: 0SSL alert NO_RENEGOTIATION rcvd: 0SSL alert CLOSE_NOTIFY sent: 0SSL alert UNEXPECTED_MSG sent: 0SSL alert BAD_RECORD_MAC sent: 0SSL alert DECRYPTION_FAILED sent: 0SSL alert RECORD_OVERFLOW sent: 0SSL alert DECOMPRESSION_FAILED sent: 0SSL alert HANDSHAKE_FAILED sent: 0SSL alert NO_CERTIFICATE sent: 0SSL alert BAD_CERTIFICATE sent: 0SSL alert UNSUPPORTED_CERTIFICATE sent: 0SSL alert CERTIFICATE_REVOKED sent: 0SSL alert CERTIFICATE_EXPIRED sent: 0SSL alert CERTIFICATE_UNKNOWN sent: 0SSL alert ILLEGAL_PARAMETER sent: 0SSL alert UNKNOWN_CA sent: 0SSL alert ACCESS_DENIED sent: 0SSL alert DECODE_ERROR sent: 0SSL alert DECRYPT_ERROR sent: 0SSL alert EXPORT_RESTRICTION sent: 0SSL alert PROTOCOL_VERSION sent: 0SSL alert INSUFFICIENT_SECURITY sent: 0SSL alert INTERNAL_ERROR sent: 0SSL alert USER_CANCELED sent: 0

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 160

Page 161: Ace Troubleshooting

SSL alert NO_RENEGOTIATION sent: 0

+-----------------------------------------------++--- Crypto server authentication statistics ---++-----------------------------------------------+Total SSL client authentications: 0Failed SSL client authentications: 0SSL client authentication cache hits: 0SSL static CRL lookups: 0SSL best effort CRL lookups: 0SSL CRL lookup cache hits: 0SSL revoked certificates: 0Total SSL server authentications: 0Failed SSL server authentications: 0

+-----------------------------------------------++------- Crypto server cipher statistics -------++-----------------------------------------------+Cipher sslv3_rsa_rc4_128_md5: 0Cipher sslv3_rsa_rc4_128_sha: 0Cipher sslv3_rsa_des_cbc_sha: 0Cipher sslv3_rsa_3des_ede_cbc_sha: 0Cipher sslv3_rsa_exp_rc4_40_md5: 0Cipher sslv3_rsa_exp_des40_cbc_sha: 0Cipher sslv3_rsa_exp1024_rc4_56_md5: 0Cipher sslv3_rsa_exp1024_des_cbc_sha: 0Cipher sslv3_rsa_exp1024_rc4_56_sha: 0Cipher sslv3_rsa_aes_128_cbc_sha: 0Cipher sslv3_rsa_aes_256_cbc_sha: 0Cipher tlsv1_rsa_rc4_128_md5: 0Cipher tlsv1_rsa_rc4_128_sha: 0Cipher tlsv1_rsa_des_cbc_sha: 0Cipher tlsv1_rsa_3des_ede_cbc_sha: 0Cipher tlsv1_rsa_exp_rc4_40_md5: 0Cipher tlsv1_rsa_exp_des40_cbc_sha: 0Cipher tlsv1_rsa_exp1024_rc4_56_md5: 0Cipher tlsv1_rsa_exp1024_des_cbc_sha: 0Cipher tlsv1_rsa_exp1024_rc4_56_sha: 0Cipher tlsv1_rsa_aes_128_cbc_sha: 0Cipher tlsv1_rsa_aes_256_cbc_sha: 0

8. Display the number of SSL data messages sent and SSL FIN/RST messages sent by entering the followingcommand:

ACE_module5/Admin# show stats http

+------------------------------------------++-------------- HTTP statistics -----------++------------------------------------------+ LB parse result msgs sent : 0 , TCP data msgs sent : 0 Inspect parse result msgs : 0 , SSL data msgs sent : 0 <------- sent TCP fin/rst msgs sent : 0 , Bounced fin/rst msgs sent: 0 SSL fin/rst msgs sent : 0 , Unproxy msgs sent : 0 <------- Drain msgs sent : 0 , Particles read : 0 Reuse msgs sent : 0 , HTTP requests : 0 Reproxied requests : 0 , Headers removed : 0 Headers inserted : 0 , HTTP redirects : 0 HTTP chunks : 0 , Pipelined requests : 0

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 161

Page 162: Ace Troubleshooting

HTTP unproxy conns : 0 , Pipeline flushes : 0 Whitespace appends : 0 , Second pass parsing : 0 Response entries recycled : 0 , Analysis errors : 0 Header insert errors : 0 , Max parselen errors : 0 Static parse errors : 0 , Resource errors : 0 Invalid path errors : 0 , Bad HTTP version errors : 0 Headers rewritten : 0 , Header rewrite errors : 0

9. Display session cache statistics for the current context by entering the following command:

switch/Admin# show crypto sessionSSL Session Cache Stats for Context------------------Number of Client Sessions: 0Number of Server Sessions: 0

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 162

Page 163: Ace Troubleshooting

This article describes how to troubleshoot performance issues with your ACE.

Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference

Contents

1 Overview of TroubleshootingPerformance Issues

2 Troubleshooting Performance Issues•

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 163

Page 164: Ace Troubleshooting

Overview of Troubleshooting Performance Issues

Before you begin to troubleshoot ACE performance issues, check and record the following items:

1. Be sure that the correct licenses are installed in your ACE.

2. Record the number of flows that you are sending to the ACE.

3. Record the performance of a single flow.

4. Identify the type of traffic: unidirectional (UDP, management) or bidirectional (TCP, HTTP, SSL, and so on)

5. Identify the ACE context that is receiving the traffic.

6. Enter the following Exec mode commands and save the output to a file:

clear stats all• show clock• show tech-support• show clock•

7. Be familiar with your application setup.

Troubleshooting Performance Issues

To troubleshoot performance issues with your ACE, follow these steps:

1. Display the resources allocated to each resource class in the ACE by entering the following command:

ACE_module5/Admin# show resource allocation---------------------------------------------------------------------------Parameter Min Max Class---------------------------------------------------------------------------

acl-memory 0.00% 100.00% default 0.00% 100.00% RC1

syslog buffer 0.00% 100.00% default 0.00% 100.00% RC1

conc-connections 0.00% 100.00% default 0.00% 100.00% RC1

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 164

Page 165: Ace Troubleshooting

mgmt-connections 0.00% 100.00% default 0.00% 100.00% RC1

proxy-connections 0.00% 100.00% default 0.00% 100.00% RC1

bandwidth 0.00% 100.00% default 0.00% 100.00% RC1

connection rate 0.00% 100.00% default 0.00% 100.00% RC1

inspect-conn rate 0.00% 100.00% default 0.00% 100.00% RC1

syslog rate 0.00% 100.00% default 0.00% 100.00% RC1

regexp 0.00% 100.00% default 0.00% 100.00% RC1

sticky 0.00% 100.00% default 5.00% 5.00% RC1

xlates 0.00% 100.00% default 0.00% 100.00% RC1

ssl-connections rate 0.00% 100.00% default 0.00% 100.00% RC1

mgmt-traffic rate 0.00% 100.00% default 0.00% 100.00% RC1

mac-miss rate 0.00% 100.00% default 0.00% 100.00% RC1

throughput 0.00% 100.00% default 0.00% 100.00% RC1

2. Display the resources allocated to the context in question by entering the following command:

ACE_module5/Admin# show resource usage context C1 Allocation Resource Current Peak Min Max Denied-------------------------------------------------------------------------------Context: C1 conc-connections 0 0 0 8000000 0 mgmt-connections 0 0 0 100000 0 proxy-connections 0 0 0 1048574 0 xlates 0 0 0 1048574 0 bandwidth 0 0 0 625000000 0 throughput 0 0 0 500000000 0

mgmt-traffic rate 0 0 0 125000000 0 <------- 1 GBps bandwidth reserved for management traffic connection rate 0 0 0 1000000 0 ssl-connections rate 0 0 0 5000 0 mac-miss rate 0 0 0 2000 0 inspect-conn rate 0 0 0 6000 0 acl-memory 0 0 0 78610432 0 sticky 0 0 209714 0 0 regexp 0 0 0 1048576 0 syslog buffer 0 0 0 4194304 0

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 165

Page 166: Ace Troubleshooting

syslog rate 0 0 0 100000 0

Note: All bandwidth values are in units of bytes per second. To convert to bits per second (bps), multiply thedisplayed bandwidth value by eight. The ACE reserves 1 Gbps of bandwidth for management(to-the-ACE) traffic.

3. From the supervisor CLI, check the connectivity to the back plane by entering the following command:

cat6k# show fabric status slot channel speed module fabric status status 2 0 8G OK OK 3 0 8G OK OK 4 0 8G OK OK

5 0 8G OK OK <-------Shows 8 Gbps connectivity to the chassis back plane 6 0 20G OK OK 8 0 8G OK OK

4. Check the fabric utilization by entering the following command:

cat6k# show fabric utilization slot channel speed Ingress % Egress % 2 0 8G 3 2 3 0 8G 0 0 4 0 8G 0 0

5 0 8G 0 0 6 0 20G 0 0 8 0 8G 2 3

5. Display the load of the network processors (NPs) in terms of packets and connection processing for eachmicroengine (ME) by entering the following command:

ACE_module5/Admin# show np 1 me-stats -cpu0 proxies open.ME Utilization Statistics--------------RECEIVE: 7FASTPATH: 44SLOWTX: 0TCP_RX: 0HTTP: 0IH_RX 0SSL_ME: 0CM_CLOSE: 36X_TO_ME: 0FIXUP: 0REASSEMBLY: 0OCM: 0TCP_TX: 0ICM: 39

ACE/Admin# show np 2 me-stats -cpu0 proxies open.ME Utilization Statistics--------------RECEIVE: 9FASTPATH: 46SLOWTX: 2TCP_RX: 0

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 166

Page 167: Ace Troubleshooting

HTTP: 0IH_RX 0SSL_ME: 0CM_CLOSE: 43X_TO_ME: 0FIXUP: 0REASSEMBLY: 0OCM: 0TCP_TX: 0ICM: 46

Note: All show np commands must be entered for both NP1 and NP2 to obtain the total combined results.NPs operate safely at any percentage of utilization. As ME functions within the NPs approach 100percent, the traffic load is stressing the system close to its architectural limits. Any ME function thatreaches 100 percent utilization can cause back pressure and lead to dropped packets or droppedconnections.

6. Monitor the CDE queues and ensure that the Fifo Full drop count counter is not incrementing by entering thefollowing command:

ACE_module5/Admin# show cde health | include FifoFifo Full drop count 0

Backpressure is the mechanism that the ACE uses to slow the system down if queues start to fill up internally.Queues that can be affected and create backpressure are as follows:

FIFOs for the CDE, NPs, and the Crypto Module• Internal queues for each ME•

It is possible that some packets that are received by the ACE could be dropped internally if backpressure isapplied.

7. Monitor the Fastpath micro engine queues and ensure that the FastQ Transmit Backpressure, the SlowQTransmit Backpressure, the Drop: Transmit Backpressure, and the Drop: Next-Hop queue full counters are notincrementing by entering the following command:

ACE_module5/Admin# show np 1 me-stats "-s fp" | include BackpressureFastQ Transmit Backpressure: 0SlowQ Transmit Backpressure: 0Drop: Transmit Backpressure: 0

ACE/Admin# show np 1 me-stats "-s fp" | include queueDrop: Next-Hop queue full: 0

8. Monitor the TCP micro engine queues and ensure the Drops due to FastTX queue full, Drops due to Fastpathqueue full, Drops due to HTTP queue full, Drops due to SSL queue full, Drops due to AI queue full, and Dropsdue to Fixup queue full are not incrementing by entering the following command. If TCP receives backpressure, itcan drop packets, fail to ACK packets, and fail to properly track the next packet in the TCP connection.

ACE/Admin# show np 1 me-stats "-s tcp" | include queueDrop reproxy msg queue full: 0Drops due to FastTX queue full: 0Drops due to Fastpath queue full: 0Drops due to HTTP queue full: 0Drops due to SSL queue full: 0Drops due to AI queue full: 0Drops due to Fixup queue full: 0

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 167

Page 168: Ace Troubleshooting

The control plane (CP) processor processes all CP traffic (ARP, HSRP, ICMP to VIPs, routing, syslogs, SNMP,probes, and so on) and handles configuration management to parse the CLI for syntactical errors and enforceconfiguration dependencies and requirements before pushing the configuration to the data plane.

9. Display a three-way moving average of the CP processor utilization (updated every five seconds) by enteringthe following command:

ACE_module5/Admin# show processes cpu | inc utilCPU utilization for five seconds: 81%; one minute: 15%; five minutes: 10%

The ACE allocates data-plane memory to guarantee concurrent connection support for basic Layer 4 connections(such as TCP, UDP, IPsec), Layer 7 connections (proxied flows, typically for application aware load balancing orinspection, and SSL connection when using SSL acceleration). The ACE can support the maximum bidirectionalconcurrent connection limit regardless of the features enabled.

Table 1. Concurrent Connection Support

Connection Type ACE Module LimitLayer 4 4,000,000Layer 7 512,000

The state for both directions (client-to-VIP/ACE and server-to-ACE) of a TCP connection is maintained withdistinct connection objects.

10. Display the connection table by entering the following command:

ACE_module5/Admin# show conn

total current connections : 6

conn-id np dir proto vlan source destination state----------+--+---+-----+----+---------------------+---------------------+------+1 1 in TCP 130 161.44.67.242:2856 10.86.215.134:23 ESTAB2 1 out TCP 130 10.86.215.134:23 161.44.67.242:2856 ESTAB4 1 in TCP 130 161.44.67.242:2837 10.86.215.134:23 ESTAB3 1 out TCP 130 10.86.215.134:23 161.44.67.242:2837 ESTAB4 2 in TCP 130 161.44.67.242:2857 10.86.215.134:23 ESTAB3 2 out TCP 130 10.86.215.134:23 161.44.67.242:2857 ESTAB

Note: You can add the detail command option to provide the following additional fields: connection idle time,elapsed time of the connection, byte count, and packet count for each connection object.

The total current connections counter is also maintained in the output of the following command:

switch/Admin# show stats connection

+------------------------------------------++------- Connection statistics ------------++------------------------------------------+ Total Connections Created : 124 Total Connections Current : 6

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 168

Page 169: Ace Troubleshooting

Total Connections Destroyed: 62 Total Connections Timed-out: 58 Total Connections Failed : 0

Note: The Total Connections Current counter counts the number of used connection objects, not the number ofTCP flows. The number of TCP flows can be roughly determined as half the number of connectionobjects minus any UDP connections. The Total Connections Current counter is always up to date andthe maximum value can be 8,000,000.

Because of the Cisco ACE Module?s architecture, with distinct paths for new and established connections, thenumber of existing concurrent connections does not heavily impact the rate at which new connections can be setup. Nevertheless, a very large number of concurrent connections will eventually affect the performance of thesystem in setting up new connections.

11. Use the command "tcp wan-optimization rtt 0" for slow connections.

The ACE module architecture includes a mechanism where connections can be moved to the fastpath in order toincrease performance for a given connection. The LB decision is made in the software (proxy) and then moved tothe fastpath (unproxy). In a persistence rebalance scenario, the proxy/unproxy can occur Many times on a givenconnection. It is possible that if a packet enters the system during the transition Between the proxy and unproxystates, a packet may not be forwarded as expected and a retransmission may be relied upon. This can affectperformance. As a workaround, it is possible to configure the ACE such that fastpath forwarding is prohibitedThis can be accomplished by configuring a parameter map with the following:

"tcp wan-optimization rtt 0"

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 169

Page 170: Ace Troubleshooting

This article describes the ACE system limits and performance numbers for various resources and configurationobjects.

Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference

Contents

1 ACE Performance Numbers and ResourceLimits

1.1 ACE Appliance Data Sheet♦ 1.2 ACE Module Data Sheets♦ 1.3 SLB-Related Limits♦ 1.4 Security-Related Limits♦ 1.5 Management-Related Limits♦

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 170

Page 171: Ace Troubleshooting

ACE Performance Numbers and Resource Limits

For the most current performance numbers for the ACE products, always refer to the data sheets for the ACEappliance and the ACE module.

ACE Appliance Data Sheet

ACE appliance data sheet

ACE Module Data Sheets

ACE10/ACE20 module data sheet

ACE30 module data sheet

If you have any questions or concerns related to ACE performance, please contact your Cisco account team forguidance.

SLB-Related Limits

Scalability Numbers The scalability numbers provided here are intended to provide guidelines related toconfiguration scalability. The scalability numbers, however, are based on basic configurations. In order to obtainscalability numbers specific to your deployment, testing with your feature combination is strongly recommended.If there are any questions or concerns related to ACE performance, please contact your Cisco account team forguidance.

SLB-Related Object ACE ModuleSystem Limit

ACE ModuleContext Limit

ACE ApplianceLimit

AdditionalInformation

ARP Entries 32,768 32,768 32,768

Bridge Table Entries 32,768 32,768 32,768A few are reserved forL2 interafces,redundancy, and so on.

Bridge-Group VirtualInterfaces (BVIs) 4096 2048 512

Concurrent Conns L4(Unproxied) 4,000,000 4,000,000 1,000,000

Concurrent ConnectionsL7 (Proxied) 512,000 512,000 128,000

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 171

Page 172: Ace Troubleshooting

Domains 2,500 10 (9) 10 (9 per context) One is used for thedefault domain.

Domain Objects None None NoneAny object within thevirtual partition can beadded to a domain.

Logical Interfaces 8,192 8,192 8,192

Resource Classes 100 (99) 1 100 (99) One is used for thedefault class.

Roles 4,000 16 (8) 16 (8) per context Eight are predefined.Sticky Groups 4,096 4,096 4,096Sticky Table Entries 4,000,000 4,000,000 800,000

Virtual Contexts 251 N/A 21 (1 Admincontext)

250 user contexts + 1Admin context

VLANs 4,000 (2-4094) 4,000 (2-4094) 4,000 (2-4094)

Security-Related Limits

Scalability Numbers The scalability numbers provided here are meant to provide guidelines related toconfiguration scalability. The scalability numbers, however, are based on basic configurations. In order to obtainscalability numbers specific to a particular customer, testing with that customer?s feature combination is stronglyrecommended before any commitment on ACE performance is made to the customer. If there are any questions orconcerns related to ACE performance, please contact your Cisco account team for guidance.

Security RelatedObject

ACE ModuleSystem Limit

ACE ModuleContext Limit ACE Appliance Limit Additional

InformationStatic NAT Policies 4096 4096 4096Dynamic NATPolicies 4096 4096 4096

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 172

Page 173: Ace Troubleshooting

Maximum ofaddresses in a NATpool

64 64 32

Maximum ofaddresses in a PATpool

63k 63k 63l

PAT Entries 4,000,000 4,000,000 1,000,000Total NAT Pools 8,192 8,192 8,192Xlates 1,000,000 1,000,000 64,000Concurrent SSLConns 100,000 100,000 100,000 Subset of L7 (proxied)

connections

RSA key size up to 4096 bits up to 4096 bits up to 4096 bits

Supported: 512, 786,1536, 1024, 2048, and4096 (imported publickeys only) bits

SSL Certs/Key files

3800/3800 (A2(3.x)and earlier)

4096/4096 (A4(1.0)and later)

3800/3800 (A2(3.x)and earlier)

4096/4096 (A4(1.0)and later)

3800/3800 (A3(1.x) andearlier)

4096/4096 (A3(2.x) andlater, incl. A4(1.0))

This number is strictlyenforced in A220,A214, and A322

Management-Related Limits

Scalability Numbers The scalability numbers provided here are meant to provide guidelines related toconfiguration scalability. The scalability numbers, however, are based on basic configurations. In order to obtainscalability numbers specific to a particular customer, testing with that customer?s feature combination is stronglyrecommended before any commitment on ACE performance is made to the customer. If there are any questions orconcerns related to ACE performance, please contact your Cisco account team for guidance.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 173

Page 174: Ace Troubleshooting

Management-RelatedObject

ACE ModuleSystem Limit

ACE ModuleContext Limit ACE Appliance Additional Information

AAA LDAP Servers 6,144 8 (24 total) 8AAA RADIUS Servers 2K (256*8) 8 (24 total) 8AAA TACACS+Servers 6K (256*24) 8 (24 total) 8

Domains 2500 64 (63) 64 (63)One domain is used for thedefault-domain and cannot beremoved

Local Users 7500 30 (Admincontext: 28)

31 (includingadmin, www, anddm)

Objects within a Domain No limit No limitAny object within the virtualpartition can be added to adomain

Resource-classes 252 Not applicable 100

Roles 4000 16 (8) 16 (8)Eight are predefined and cannotbe altered, leaving eight for youto customize

SNMP Hosts No Limit 10SSH Sessions 256 4 4Syslog buffer size 4 MB 4 MB 1 MB

Syslog CP rate 5,000 perseconds

5,000 perseconds 3,000 per seconds

Syslog DP rate 350,000 persecond

350,000 persecond

100,000 persecond

Syslog history table size 256 x 500 500Syslog Hosts 256 2 2Syslog internal queuesize 10 MB 10 MB 8,192 messages

Syslog persistence size 1M 1MSyslog rate limit tablesize 256 x 100 100 10,000 messages

per secTelnet Sessions 256 4 4

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 174

Page 175: Ace Troubleshooting

This article describes how to manage and control the ACE system resources.

Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference

Contents

1 Overview of ACE Resources• 2 Managing ACE Resources

2.1 ACE Resource Planning♦ 2.2 Creating a Resource Class for ResourceManagement

2.3 Allocating Resources Within aResource Class

2.4 Changing the Resource Allocation of aResource Class

2.5 Displaying the ACE ResourceAllocation and Usage

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 175

Page 176: Ace Troubleshooting

Overview of ACE Resources

Resource classes allow you to manage context access to ACE resources, such as concurrent connections orbandwidth rate. The ACE is preconfigured with a default resource class that it applies to the Admin context andany user context upon creation. The default resource class is configured to allow a context to operate within arange that can vary from no resource access (0 percent) to complete resource access (100 percent).

When you use the default resource class with multiple contexts, you run the risk of oversubscribing ACEresources because the ACE permits all contexts to have full access to all of the resources on a first-come,first-served basis. When a resource is utilized to its maximum limit, the ACE denies additional requests made byany context for that resource.

To avoid oversubscribing resources and to help guarantee access to a resource by any context, the ACE allowsyou to create customized resource classes that you associate with one or more contexts. A context becomes amember of the resource class when you make the association. Creating a resource class allows you to set limits onthe minimum and maximum amounts of each ACE resource that a member context is entitled to use. You definethe minimum and maximum values as percentages of all resources. For example, you can create a resource classthat allows its member contexts access to no less that 25 percent of the total number of SSL connections that theACE supports.

You can limit and manage the allocation of the following ACE resources:

ACL memory• Buffers for syslog messages and TCP out-of-order (OOO) segments• Concurrent connections (through-the-ACE traffic)• Management connections (to-the-ACE traffic)• Proxy connections• Set resource limit as a rate (number per second)• Regular expression (regexp) memory• SSL connections• Sticky entries• Static or dynamic network address translations (Xlates)•

By default, when you create a context, the ACE associates the context with the default resource class. The defaultresource class provides resources of a minimum of 0 and a maximum of unlimited for all resources except stickyentries. For stickiness to work properly, you must explicitly configure a minimum resource limit for sticky entriesby using the limit-resource command.

For more information about managing ACE resources, see the Cisco Application Control Engine ModuleVirtualization Configuration Guide (Software Version A2(1.0)).

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 176

Page 177: Ace Troubleshooting

Managing ACE Resources

You can allocate system resources to multiple contexts by creating and defining one or more resource classes andthen associating the contexts with a resource class. This section contains the following topics:

ACE Resource Planning• Creating a Resource Class for Resource Management• Allocating Resources within a Resource Class• Changing the Resource Allocation of a Resource Class•

ACE Resource Planning

When you plan the initial resource allocations for the virtual contexts in your configuration, allocate only theminimum required or estimated resources. The ACE protects resources that are in use, so to decrease a context'sresources, those resources must be unused. Although it is possible to decrease the resource allocations in realtime, it may require additional management overhead to clear any used resources before reducing them.Therefore, it is considered a best practice to initially keep as many resources in reserve as possible and allocatethe unused reserved resources as needed.

To address scaling and capacity planning, we recommend that new installations do not exceed 60 to 80 percent ofthe ACE's total capacity. To accomplish this goal, create a reserved resource class with a guarantee of 20 to 40percent of all the ACE resources and configure a virtual context dedicated solely to ensuring that these resourcesare reserved. Then, you can efficiently distribute such reserved resources to contexts as capacity demands forhandling client traffic increase over time.

Creating a Resource Class for Resource Management

You can create a resource class to allocate and manage system resources by one or more contexts. The ACEsupports a maximum of 100 resource classes. After you create and configure the resource class, use the membercommand in context configuration mode to assign a resource class to the context (see the "Associating a Contextwith a Resource Class" section). To create a resource class, use the resource-class command in configurationmode. The syntax of the command is as follows:

resource-class name

For the name argument, enter an unquoted text string with no spaces and a maximum of 64 alphanumericcharacters.

For example, to create the RC1 resource class, enter the following command:

ACE_module5/Admin(config)# resource-class RC1ACE_module5/Admin(config-resource)

To remove the resource class from the configuration, enter the following command:

host1/Admin(config)# no resource-class RC1

When you remove a resource class from the ACE, any contexts that were members of that resource class

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 177

Page 178: Ace Troubleshooting

automatically become members of the default resource class. The default resource class allocates a minimum of0.00 percent to a maximum of 100.00 percent of all ACE resources to each context. You cannot modify thedefault resource class.

Allocating Resources Within a Resource Class

You can allocate all resources or individual resources to all member contexts of a resource class. For example,you can allocate only concurrent connections or sticky table memory or management traffic. To allocate systemresources to all members (contexts) of a resource class, use the limit-resource command in resource-classconfiguration mode.

The syntax of this command is as follows:

limit-resource {acl-memory | all | buffer {syslog} | conc-connections | mgmt-connections | proxy-connections| rate {bandwidth | connections | inspect-conn | mac-miss | mgmt-traffic | ssl-bandwidth | syslog} | regexp |sticky | xlates} {minimum number} {maximum {equal-to-min | unlimited}}

Note: The limit that you set for individual resources when you use the limit-resource command overrides thelimit that you set for all resources when you use the limit-resource all command.

If you lower the limits for one context (context A) in order to increase the limits of another context (context B),you may experience a delay in the configuration change because the ACE will not lower the limits of context Auntil the resources are no longer being used by the context.

For example, to allocate 20 percent of all resources (minimum and maximum) to all member contexts of theresource class, enter the following command:

(config-resource)# limit-resource all minimum 20% maximum equal-to-min

To restore resource allocation to the default values of 0 percent minimum and 100 percent maximum for allresources to all member contexts, enter the following command:

(config-resource)# no limit-resource all

Table 1 lists the managed system resources of the ACE. You can limit these resources per context or for allcontexts associated with the resource class by using the limit-resource command. See the "Allocating Resourceswithin a Resource Class" section.

Table 1. System Resource Maximum Values

Resource Maximum ValueACL Memory 78,610,432 bytesBuffer Memory (Syslog) 4,000,000 bytesConcurrent Connections (Layer 4) 4,000,000 connectionsConcurrent Connections (SSL) 200,000Management Connections 100,000 connectionsProxy Connections (Layer 7) 524,286 connectionsSSL Proxy Connections 200,000

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 178

Page 179: Ace Troubleshooting

Rate---Bandwidth 4 gigabits per second (Gbps)

You can upgrade the ACE maximum bandwidth to 8 Gbps or 16 Gbps bypurchasing a separate license from Cisco. For more information, see theCisco Application Control Engine Module Administration Guide (SoftwareVersion A2(1.0)).

---Connections (any kind) 325,000 connections per second (CPS)---MAC miss 2000 packets per second (PPS)---Management traffic 1 Gbps---SSL transactions 1000 transactions per second (TPS), upgradeable to 15000 TPS with a

separate license. For more information, see the Cisco Application ControlEngine Module Administration Guide (Software Version A2(1.0)).

---Syslog For traffic going to the ACE (control plane), 5000 messages per second

For traffic going through the ACE (data plane), 350,000 messages per secondRegular Expression Memory 1,048,576 bytesSticky Entries 4,194,304 entriesXlates (network and port addresstranslation entries)

524,286 translations

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 179

Page 180: Ace Troubleshooting

Changing the Resource Allocation of a Resource Class

If you (as the global Admin) need to change the resource allocation in a resource class of which two or more usercontexts are members, you may do so at any time by entering the appropriate CLI commands. (For details aboutallocating resources, see the "Allocating Resources Within a Resource Class" section.) However, the shift inresources between the contexts does not take place immediately unless the appropriate resources are available toaccommodate the change. In most cases, to effect a change in resource allocation, you must inform the contextadministrators involved to ensure that the new resource allocation is possible.

For example, suppose that context A is using 100 percent of the available resources of the class and you want toallocate 50 percent of the resources to context A and 50 percent of the resources to context B. Although the CLIaccepts your resource allocation commands, context B cannot allocate 50 percent of the resources until context Adeallocates 50 percent of its resources.

In this case, you must perform the following:

Inform the Context A administrator to start deallocating resources•

Inform the Context B administrator to start allocating resources after the Context A administrator releasesthe resources

Note: As resources are released from other contexts, the ACE assigns the resources to resource-starvedcontexts (contexts where the resource-class minimum allocations have not been met).

Displaying the ACE Resource Allocation and Usage

To view the current resource allocation in your ACE, enter the following command:

ACE_mdule5/Admin# show resource allocation---------------------------------------------------------------------------Parameter Min Max Class---------------------------------------------------------------------------

acl-memory 0.00% 100.00% default

syslog buffer 0.00% 100.00% default

conc-connections 0.00% 100.00% default

mgmt-connections 0.00% 100.00% default

proxy-connections 0.00% 100.00% default

bandwidth 0.00% 100.00% default

connection rate 0.00% 100.00% default

inspect-conn rate 0.00% 100.00% default

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 180

Page 181: Ace Troubleshooting

syslog rate 0.00% 100.00% default

regexp 0.00% 100.00% default

sticky 0.00% 100.00% default

xlates 0.00% 100.00% default

ssl-connections rate 0.00% 100.00% default

mgmt-traffic rate 0.00% 100.00% default

mac-miss rate 0.00% 100.00% default

throughput 0.00% 100.00% default

To view the current resource usage, enter the following command:

ACE_mdule5/Admin# show resource usage Allocation Resource Current Peak Min Max Denied-------------------------------------------------------------------------------Context: Admin conc-connections 0 0 0 8000000 0 mgmt-connections 2 8 0 100000 0 proxy-connections 0 0 0 1048574 0 xlates 0 0 0 1048574 0 bandwidth 1094 80192 0 625000000 0 throughput 938 75902 0 500000000 0 mgmt-traffic rate 156 4290 0 125000000 0 connection rate 1 28 0 1000000 0 ssl-connections rate 0 0 0 5000 0 mac-miss rate 0 0 0 2000 0 inspect-conn rate 0 0 0 6000 0 acl-memory 23776 28616 0 78610432 0 sticky 0 0 0 0 0 regexp 0 0 0 1048576 0 syslog buffer 0 0 0 4194304 0 syslog rate 0 0 0 100000 0

Note: All bandwidth values are in bytes per second. To convert to bits per second (bps), multiply the values byeight. The ACE guarantees 1 Gbps of bandwidth for management traffic. So, the total bandwidth for a4-Gbps ACE license is actually 5 Gbps. Throughput is still 4 Gbps.

To display the data plane resource allocation and usage and to cross-check the output of the above twocommands, enter the following command:

ACE_module5/Admin# show np 1 me-stats -L0Resource limts for context : 0Rate Configured CountersPolicer Name Min Max min-toks max-toks peak-toks deny bandwidth: 0 ee6b280 0 ee6b0fa d8a4 0 throughput: 0 ee6b280 0 ee6b280 d8a4 0mgmt-traffic rate: 0 3b9aca0 0 3b9aca0 a0e 0 connection rate: 0 7a120 0 7a120 11 0ssl-connections rate: 0 9c4 0 9c4 0 0 mac-miss rate: 0 3e8 0 3e8 0 0inspect-conn rate: 0 bb8 0 bb8 0 0

Resource Configured CountersPolicer Name Min Max Min Max peak deny

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 181

Page 182: Ace Troubleshooting

conc-connections: 0 3d0900 0 0 0 0mgmt-connections: 0 c350 0 0 4 0proxy-connections: 0 7ffff 0 0 0 0ip-reassemble buffer: 0 0 0 0 0 0 tcp-ooo buffer: 0 0 0 0 0 0 regexp: 0 0 0 0 0 0 xlates: 0 7ffff 0 0 0 0

The Admin context has a context ID of 0. To display the resource allocation and and usage statistics for anothercontext, change the "0" in the "-L<context_id>" parameter to the context ID of another context.

Cisco Application Control Engine (ACE) Troubleshooting Guide

07/26/11 182