BRKSEC-3020 Advanced Firewalls

Session BRKSEC-3020

Advanced Firewalls

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSEC-3020 2

Agenda

 Packet Flow  Understanding the Architecture

 Failover  Troubleshooting  Case Studies

 Online Resources  Best Practices

Packet Flow


Understanding the Packet Flow

 To effectively troubleshoot a problem, one must first understand the packet path through the network

 Attempt to isolate the problem down to a single device  Then perform a systematic walk of the packet path through

the device to determine where the problem could be  For problems relating to the Cisco ASA/FWSM, always

Determine the flow: SRC IP, DST IP, SRC port, DST port, and protocol

Determine the interfaces through which the flow passes

Note: All Firewall Issues Can Be Simplified to Two Interfaces (Ingress and Egress) and the Rules Tied to Both


Accounting

Example Flow

  Flow SRC IP: 10.1.1.9 SRC Port: 11030 Protocol: TCP DST IP: 198.133.219.25 DST Port: 80

  Interfaces Source: Inside Destination: Outside

With the Flow Defined, Examination of Configuration Issues Boils Down to Just the Two Interfaces: Inside and Outside

Eng

Client: 10.1.1.9

Server: 198.133.219.25

Servers O

utside

Packet Flow


Packet Processing: Ingress Interface

  Packet arrives on ingress interface   Input counters incremented   Software input queue is an indicator of load   No buffers indicates packet drops, typically due to bursty traffic

ASA-5540# show interface gb-ethernet1 interface gb-ethernet1 "inside" is up, line protocol is up Hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.6214 IP address 10.1.1.1, subnet mask 255.255.255.0 MTU 1500 bytes, BW 1 Gbit full duplex 5912749 packets input, 377701207 bytes, 0 no buffer Received 29519 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 286298 packets output, 18326033 bytes, 0 underruns input queue (curr/max blocks): hardware (4/25) software (0/0) output queue (curr/max blocks): hardware (0/3) software (0/0)

Ingress Interface


Packet Processing: Locate Connection

  Check first for existing connection   If connection exists, flow is matched; bypass ACL check   If no existing connection

TCP non-SYN packet, drop and log TCP SYN or UDP packet, pass to ACL checks

Established Connection:

ASA-5540# show conn TCP out 198.133.219.25:80 in 10.1.1.9:11030 idle 0:00:04 Bytes 1293 flags UIO

Syslog Because of No Connection, and Non-SYN Packet:

ASA-6-106015: Deny TCP (no connection) from 10.1.1.9/11031 to 198.133.219.25/80 flags PSH ACK on interface inside

Existing Conn


Packet Processing: ACL Check

  First packet in flow is processed through interface ACLs   ACLs are first match   First packet in flow matches ACE, incrementing hit count by one   Denied packets are dropped and logged

Packet Permitted by ACL:

ASA-5540B# show access-list inside access-list inside line 10 permit ip 10.1.1.0 255.255.255.0 any (hitcnt=1)

Syslog When Packet Is Denied by ACL:

ASA-4-106023: Deny tcp src inside:10.1.1.9/11034 dst outside:198.133.219.25/80 by access-group "inside"

ACL Permit


Translation and NAT Order of Operations

1.  nat 0 access-list (nat-exempt) 2.  Match existing xlates 3.  Match static commands (Cisco

ASA/PIX first match; FWSM best match)

4.  Match nat commands

First Match

Match xLate

For your reference

1.  Manual NAT entries 2.  Auto NAT entries 3.  After-Auto NAT entries

First Match

Pre version 8.3 Version 8.3+

Translation Matching


Packet Processing: Inspections/Sec Checks

  Inspections are applied to ensure protocol compliance   (Optional) customized AIC inspections   NAT-embedded IPs in payload   Additional security checks are applied to the packet   (Optional) packets passed to Content Security and Control

(CSC) module

Syslog from Packets Denied by Security Check:

ASA-4-406002: FTP port command different address: 10.2.252.21(192.168.1.21) to 209.165.202.130 on interface inside

ASA-4-405104: H225 message received from outside_address/outside_port to inside_address/inside_port before SETUP

Inspections Sec Checks

Question!

What command will show you if packets are being dropped by one of the Inspection engines?


Packet Processing: NAT IP Header

  Translate the IP address in the IP header

  Translate the port if performing PAT

  Update checksums

  (Optional) Following the above, pass packet to IPS (AIP) module

Nat IP Header


Packet Processing: Egress Interface

  Packet is virtually forwarded to egress interface (i.e., not forwarded to the driver yet)

  Egress interface is determined first by translation rules   If translation rules do not specify egress interface (e.g., outbound

initial packet) the results of a global route lookup are used to determine egress interface

  Example:

static (inside, outside) 192.168.0.0 172.16.0.0 netmask 255.255.0.0 static (dmz, outside) 192.168.12.0 172.16.12.0 netmask 255.255.255.0

DM

Z

Inside Outside

172.16.0.0/16 172.16.12.0/24

172.16.12.4

Inbound Packets to 192.168.12.4 Get Routed to Inside Based on Order of Statics

Egress Interface


Packet Processing: L3 Route Lookup

  Once on egress interface, an interface route lookup is performed

  Only routes pointing out the egress interface are eligible   Remember: translation rule can forward the packet to the

egress interface, even though the routing table may point to a different interface

Syslog from Packet on Egress Interface with No Route Pointing Out Interface: %ASA-6-110003: Routing failed to locate next hop for TCP from inside:192.168.103.220/59138 to dmz:172.18.124.76/23

L3 Route


Packet Processing: L2 Address Lookup

 Once a Layer 3 route has been found, and next hop identified, Layer 2 resolution is performed

  Layer 2 rewrite of MAC header

  If Layer 2 resolution fails—no syslog

 show arp will not display an entry for the L3 next hop

 debug arp will indicate if we are not receiving an ARP reply

L2 Addr


Packet Processing: Transmit Packet

  Packet is transmitted on wire   Interface counters will increment on interface

Xmit Pkt

ASA-5585# show int Gig0/0 Interface GigabitEthernet0/0 "outside", is up, line protocol is up Hardware is bcm56801 rev 01, BW 1000 Mbps, DLY 10 usec Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps MAC address 5475.d05b.0fa6, MTU 1500 IP address 14.36.103.96, subnet mask 255.255.0.0 4337255 packets input, 394043049 bytes, 0 no buffer Received 1957325 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 0 pause/resume input 0 switch ingress policy drops 282901 packets output, 28855690 bytes, 0 underruns 0 pause/resume output


Agenda





Cisco ASA — Understanding the Architecture  ASA processes all packets in software (via the central

CPU) All packets are processed first in… usually also first out

 ASA platforms have software imposed connection limits  Multi-CPU / Multi-Core systems hash packets in the

same flow to the same CPU/core.

  10 Gig interfaces hash flow to same RX ring.  Architecture optimized for multi-flow traffic patterns  ASASM packet processing is also done in software,

unlike FWSM


Maximum ACL Limits

 No hard-coded limit on the number of elements (ACEs) in an ACL. Bound only by Memory.

 Each ACE uses a minimum of 212 bytes of RAM

 However, maximum performance may decrease (typically 10-15%) as you reach or exceed the Max Recommended ACEs.

5505 5510 5520 5540 5550 5580 5585 10/20/40/60

ASA SM

Max Recommended

ACEs 25k 80k 200k 500k 700k 750k 500k / 750k

1 / 2 million 2 million

Tested ACEs 80k 300k 700k 700k 1 million+ 500k / 750k 1 / 2 million 2 million

Max Observed (from customers) 2.74 million 2.77 million

Note: Issue show access-list | include elements to see how many ACEs you have


Warning - ACE Explosion  Object-groups:

•  Sources (10 addresses) •  Destinations (21 addresses) •  Ports (33 ports) •  Result: 10x21x33 = 6,930 rules

 Nested object-groups: •  Assume you add a SRC object-group to the above,

which contains 25 additional sources •  Result: (10+25)x21x33 = 24,255 rules (ACEs)

•  New command to reduce ACL memory impact for large ACLs. Available starting in 8.3(1)

Single line ACL

explodes to

ASA-5585(config)# object-group-search access-control


Global ACLs

 Global ACLs introduced in version 8.3

 Best used for new installations, or migration from other vendors

Interface Independent Policies

access-group <access_list> global

Global access-list

Interface Specific access-list

Default (implicit) deny ip any any

Policy Ordering

ASA Only


Object-NAT (Auto-NAT) (version 8.3+)

 Object NAT is the simplest form of NAT, and is defined within an object

object network obj-WebServer host 10.3.19.50 nat (inside,outside) static 198.51.100.50

object network Servers subnet 10.0.54.0 255.255.255.0 nat (inside,outside) static 203.0.113.0

object network InternalUsers subnet 192.168.2.0 255.255.255.0 nat (inside,outside) dynamic interface

Host NAT

Network NAT

Dynamic PAT (interface overload)


Manual NAT (Twice NAT) (version 8.3+)

 Manual NAT should be used to translate the destination, or for policy NAT

object network ServerReal host 10.3.19.50 object network ServerTrans host 198.51.100.50 object network RemoteSite subnet 10.0.0.0 255.255.255.0

nat (inside,outside) source static ServerReal ServerTrans destination static RemoteSite RemoteSite

Static NAT

Static Policy NAT

nat (inside,outside) source static ServerReal ServerTrans


NAT Order of Operation version 8.3+

  The ASA configuration is built into the NAT Table (show nat)

  The NAT Table is based on First Match (top to bottom)

Manual NAT Policies (Section 1)

Auto NAT Policies (Section 2)

Manual NAT [after auto] Policies (Section 3)

Static NAT

Dynamic NAT

NAT Table

Longest Prefix

Shortest Prefix

Longest Prefix

Shortest Prefix

First Match (in config)

First Match (in config)

For your reference


Real-IP (version 8.3+)

 Finally, a reminder that with 8.3+ Real-IPs are used in ACLs

object network obj-WebServer host 10.3.19.50 nat (inside,outside) static 198.51.100.50 ! access-list allowIn permit tcp any host 10.3.19.50 eq 80 ! access-group allowIn in interface outside

Real, Un-translated address of internal Server


FWSM—Understanding the Architecture

  Packets processed in hardware have zero impact on CPU

  Similarly, if the CPU is pegged at 100%, this has zero impact on packets processed in hardware

  Note that FWSM packet processing is different from ASA

FWSM Process Most Packets in Hardware, with Some Packets Needing to be Processed in Software—via the Control Point (CP)


Fast Path Flow Identification, Security Checks and NAT in Hardware

FWSM Architectural Overview

C6K Backplane Interface

Session Manager NP 3

Control Point (CP) Central CPU

Fast Path NP 1

Software

Hardware

FWSM

Control Point ACL Compilation, Fixups, Syslog, AAA, IPv6 in Software

Session Manager Session Establishment and Teardown, AAA Cache, ACLs

Fast Path NP 2


FWSM—Hardware Limits   FWSM has several hardware limits that should be considered in

your network design   Limits are hard set, but vary based on single or multimode   Some limits include:

2.3 (Multimode) 3.1/3.2 (Multimode) 4.0/4.1 (Multimode) 3.2 / 4.0 /4.1 Configurable

ACEs 56,627 (9,704) 72,806 (11,200) 100,567 (14,801) X AAA Rules 3,942 (606) 6,451 (992) 8,744 (1,345) X Global Statements 1K (1K) 4K (4K) 4K (4K) Static NAT Statements 2K (2K) 2K (2K) 2K (2K) Policy NAT ACEs 3,942 (606) 1,843 (283) 2,498 (384) X NAT Translations 256K (256K) 256K (256K) 256K (256K) Connections 999,990 (999,990) 999,990 (999,990) 999,990 (999,990) Route Table Entries 32K (32K) 32K (32K) 32K (32K) Fixup/Inspect Rules 32 (32 per) 4147 (1,417) 5621 (1,537) X Filter Statements 3942 (606) 2764 (425) 3747 (576) X

Increase over 2.3 Increase over 3.1

*Complete List in FWSM Docs, Appendix A (Specifications)

See Appendix


FWSM—ACL Rule Limits

  ACL rules are about the only hardware limit users encounter   In multimode, ACL resources are divided in 13 equal

partitions (12 active, one backup) If you have less than 12 contexts, wasted reserved space

Tree 0 : active = 14,801 ACEs












Tree 12 : backup

177612 combined total ACEs

Tree 0 : Active 100,567 ACEs

Backup Tree: 100,567 (mirror of active tree)

Multi-Context Single Context


Classifier in Multimode

 When the firewall receives a packet, it must classify it to determine where to send the packet (which context)

 Packets are classified based on the following Unique ingress interface/VLAN

Packet’s destination IP matches a global IP

 FWSM has a single MAC address for all interfaces  ASA has single MAC for shared interfaces (physical

interfaces have unique MACs) ASA Ver 7.2 introduces mac-address auto option to change this


Classifier in Multimode

  Inbound traffic is classified to context CTX3, based on the global IP in the NAT translation

VLA

N 3

—10

.14.

3.x

Inside

10.1.2.2

Inside

10.1.1.2

Inside

10.1.3.2

Inbound Packet Outside

VLAN 4

VLAN 5

VLAN 6

FWSM

CTX1

CTX2

CTX3

MSFC

.1

.2

.3

DST IP SRC IP 10.14.3.89 192.168.5.4

static (inside, outside) 10.14.3.89 10.1.3.2

Shared Interface

Example


Multi-Context - Common Issues on FWSM

 Overlapping statics (globals) across contexts  Missing statics (globals), and unable to classify packets –

check Admin context log

  Forgetting to ‘monitor-interface’ for Failover

  Forgetting to assign unique IP for each Transparent mode context

  Transparent mode, multi-BVI, one routing table

%FWSM-6-106025: Failed to determine security context for packet: vlan3 tcp src 192.168.5.4/1025 dest 72.163.4.161/80


Agenda





Failover Basics

  Active/Standby vs. Primary/Secondary

  Serial vs. LAN failover

  Stateful failover (optional)

  A failover only occurs when either firewall determines the standby firewall is healthier than the active firewall

  Both firewalls swap MAC and IP addresses when a failover occurs

  Level 1 syslogs will give reason of failover

Secondary (Standby)

Primary (Active)

LAN/Serial

Stateful

Internet

Corp


ASA# show failover Failover On Failover unit Primary Failover LAN Interface: failover Redundant5 (up) Unit Poll frequency 200 milliseconds, holdtime 1 seconds Interface Poll frequency 500 milliseconds, holdtime 5 seconds Interface Policy 1 Monitored Interfaces 2 of 250 maximum Version: Ours 8.2(2), Mate 8.2(1) Last Failover at: 10:37:11 UTC May 14 2010 This host: Primary - Active Active time: 1366024 (sec) slot 0: ASA5580 hw/sw rev (1.0/8.1(2)) status (Up Sys) Interface outside (10.8.20.241): Normal Interface inside (10.89.8.29): Normal Other host: Secondary - Standby Ready Active time: 0 (sec) slot 0: ASA5580 hw/sw rev (1.0/8.1(2)24) status (Up Sys) Interface outside (10.8.20.242): Normal Interface inside (10.89.8.30): Normal Stateful Failover Logical Update Statistics Link : stateful Redundant6 (up) Stateful Obj xmit xerr rcv rerr General 424525 0 424688 0 sys cmd 423182 0 423182 0

Verifying Failover Configuration

Interface Monitoring


What Triggers a Failover?

 Power loss/reload (this includes crashes) on the Active firewall

 SSM interface/module failure  The Standby becoming healthier than the

Active firewall

In LAN based Failover, what happens if the LAN interface communication is severed?


What Triggers a Failover? (Con’t)

 Two consecutive hello messages missed on any monitored interface forces the interface into testing mode

 Both units first verify the link status on the interface  Next, both units execute the following tests

Network activity test ARP test Broadcast ping test

 The first test passed causes the interface on that unit to be marked healthy; only if all tests fail will the interface be marked failed


How Well do you Understand Failover? What Happens When…

 You disable failover? (By issuing no failover)  You RMA/Replace the Primary unit?

 You don’t define Standby IP addresses?


What to Do After a Failover

 Always check the syslogs to determine root cause  Example: switch port failed on inside interface of

active firewall

ASA-4-411002: Line protocol on Interface inside, changed state to down ASA-1-105007: (Primary) Link status ‘Down’ on interface 1 ASA-1-104002: (Primary) Switching to STNDBY—interface check, mate is healthier

ASA-1-104001: (Secondary) Switching to ACTIVE—mate want me Active

Syslogs from Primary (Active) Firewall

Syslogs from Secondary (Standby) Firewall

See Appendix


Failover – Zero Downtime Upgrades

Copy new image over and reboot

Wait for failover to finish syncing, and to “normalize” – approx 2 min

Verify config; conns replicated

Issue “failover active”

Copy new image over and reboot

Wait for failover to finish syncing, and to “normalize” – approx 2 min

Verify config; conns replicated

Upgrade Complete

Issue “failover active”

Secondary Primary

Stb

Act Stb

Act

State State

Act Stb Start


Agenda





Troubleshooting Tools

 Syslogs  Debug commands

 Show commands  Packet capture  Packet tracer

 TCP Ping


Buffered

Uses of Syslogs

 Primary mechanism to record traffic to and through the firewall

 The best troubleshooting tool available

SSH Client

Internet

Archival Purposes Debugging Purposes

Syslog Server

SNMP Server

Console

Trap . Syslog


ASA Syslog Level vs. Number of Messages

Log Level Description

Number of Messages (SUM)

Ver. 7.0 Ver. 7.2 Ver. 8.0 Ver. 8.1 Ver. 8.2 Ver. 8.3 Ver. 8.4

0 Emergencies 0 0 0 0 0 0 0

1 Alerts 62 (62) 77 (77) 78 (78) 87 (87) 87 (87) 95 (95) 109 (109)

2 Critical 29 (91) 35 (112) 49 (127) 50 (137) 56 (143) 57 (152) 63 (172)

3 Errors 274 (365)

334 (446)

361 (488)

363 (500)

384 (527)

408 (560)

448 (620)

4 Warnings 179 (544)

267 (713)

280 (768)

281 (781)

315 (842)

324 (884)

357 (997)

5 Notifications

161 (705)

206 (919)

216 (984)

218 (999)

237 (1079)

246 (1130)

265 (1242)

6 Informational

234 (939)

302 (1221)

335 (1319)

337 (1336)

368 (1447)

377 (1507)

395 (1637)

7 Debugging 217 (1156)

258 (1479)

266 (1585)

267 (1603)

269 (1716)

269 (1776)

276 (1913)

More messages


What Are Modifiable Syslog Levels?

 Modifiable syslog levels Allows one to move any syslog message to any level

  Problem You want to record what exec commands are being executed on the firewall; syslog ID 111009 records this information, but by default it is at level seven (debug)

%ASA-7-111009: User ‘johndoe’ executed cmd: show run

The problem is we don’t want to log all 1775 other syslogs that are generated at debug level

[no] logging message <syslog_id> level <level>

Levels 0—Emergency 1—Alert

2—Critical

3—Errors

4—Warnings

5—Notifications

6—Informational 7—Debugging


How to Create Modifiable Syslog Levels

 Lower syslog message 111009 to level 3 (error) ASA(config)# logging message 111009 level 3

  Now our syslog looks as follows %ASA-3-111009: User ‘johndoe’ executed cmd: show run

  To restore the default syslog level ASA(config)# no logging message 111009 level 3

[no] logging message <syslog_id> level <level>

Solution

Tip: Use show logging message all to see the default level for any message

If you were only interested in logging one syslog message, how could you do it?


Logging – Common Issues

  logging flash-bufferwrap – should only be used when logging to buffer at Level 1

  logging history – should only be used when you really have an SNMP server that you want to receive all syslogs

  logging console – should only be enabled while actively troubleshooting on the Console

  logging standby – should only be used if you want to receive double the syslogs

  logging permit-hostdown – should always be used with TCP syslogging


Debug Commands

1.  Debugs should not be the first choice to troubleshoot a problem

2.  Debugs can negatively impact the CPU of the box, and also the performance of it; use with caution

3.  Debugs are not conditional*

4.  Know how much traffic, of the specified type, is passing through the firewall before enabling the respective debug

* Crypto Conditional Debugging Was Added to Cisco ASA/PIX 8.0


Debug ICMP Trace

  Valuable tool used to troubleshoot connectivity issues   Provides interface and translation information to quickly

determine flow   Echo-replies must be explicitly permitted through ACL, or ICMP

inspection must be enabled

ICMP echo-request from inside:10.1.1.2 to 198.133.219.25 ID=3239 seq=4369 length=80 ICMP echo-request: translating inside:10.1.1.2 to outside:209.165.201.22

ICMP echo-reply from outside:198.133.219.25 to 209.165.201.22 ID=3239 seq=4369 length=80 ICMP echo-reply: untranslating outside:209.165.201.22 to inside:10.1.1.2

Example debug icmp trace output

http://www.cisco.com

Internet


Show Output Filters

 Use output filters to filter the output of show command to only the information you want to see

 To use them, at the end of show <Command>, use the pipe character “|” followed by

begin Start displaying the output beginning at the first match of the RegEx, and continue to display the remaining output

include Display any line that matches the RegEx exclude Display any line that does not match the RegEx grep Same as include grep –v Same as exclude

show <cmd> | begin|include|exclude|grep [-v] <regular_exp>

See Appendix


Show CPU Usage

 Under normal conditions the CPU should stay below 50% (baseline as per network); if the CPU reaches 100% the firewall will start dropping packets

 FWSM CPU is used for limited traffic processing; during ACL compilation CPU is expected to be near 100% until ACL is compiled

 The show cpu usage command displays the CPU over time as a running average

ASA# show cpu usage CPU utilization for 5 seconds = 5%; 1 minute: 4%; 5 minutes: 4%

*First Introduced in Cisco PIX OS Version 6.0(1)/FWSM 1.1(1)


Show Processes cpu-usage

 The show processes cpu-usage command displays the amount of CPU used on a per-process basis for the last 5sec, 1min, and 5min

ASA# # show processes cpu-usage PC Thread 5Sec 1Min 5Min Process 081aa124 d51ab230 0.2% 2.0% 2.0% Dispatch Unit 08070416 d51aa660 0.0% 0.0% 0.0% aaa 081a954c d51a96a0 0.0% 0.0% 0.0% dbgtrace 08c2a91d d51a7f00 0.0% 0.0% 0.0% netfs_thread_init 0924fe95 d51a7528 0.0% 0.0% 0.0% Chunk Manager 088a6e14 d51a7138 0.0% 0.0% 0.0% IP Address Assign 08a6d7f6 d51a6f40 0.0% 0.0% 0.0% QoS Support Module 08bcf736 d51a53b0 0.0% 0.0% 0.0% Logger 08685627 d51a3a18 0.0% 0.0% 0.0% netfs_mount_handler 0851ca68 d51a3820 0.0% 0.0% 0.0% arp_timer 08b9ffab d5198ae0 0.0% 0.0% 0.0% ssh/timer 08b99aec d5195d98 3.9% 0.5% 0.1% ssh ...

*First Introduced in Cisco ASA Version 7.2(4.11), 8.0(4.5), 8.1(1.100), 8.2(1). Currently not Available in FWSM


Show Processes cpu-hog   The show processes cpu-hog command displays

a list of processes, and the function stack (Traceback) which executed, and lead to a process running on the CPU longer than the minimum platform threshold

ASA# show processes cpu-hog Process: ssh_init, NUMHOG: 18, MAXHOG: 15, LASTHOG: 10 LASTHOG At: 14:18:47 EDT May 29 2009 PC: 8b9ac8c (suspend) Traceback: 8b9ac8c 8ba77ed 8ba573e 8ba58e8 8ba6971 8ba02b4 8062413

CPU hog threshold (msec): 10.240 Last cleared: None

*First introduced in Cisco ASA Version 7.0(1). Currently not Available in FWSM

May 29 2009 14:18:47: %ASA-7-711002: Task ran for 10 msec, Process = ssh_init, PC = 8b9ac8c, Traceback = 0x08B9AC8C 0x08BA77ED 0x08BA573E 0x08BA58E8 0x08BA6971 0x08BA02B4 0x08062413

  A corresponding syslog message is also generated Note: The Traceback syslog below does not signify a crash


Show Traffic

  The show traffic command displays the traffic received and transmitted out each interface of the firewall

ASA# show traffic outside: received (in 124.650 secs): 295468 packets 167218253 bytes 2370 pkts/sec 1341502 bytes/sec transmitted (in 124.650 secs): 260901 packets 120467981 bytes 2093 pkts/sec 966449 bytes/sec inside: received (in 124.650 secs): 261478 packets 120145678 bytes 2097 pkts/sec 963864 bytes/sec transmitted (in 124.650 secs): 294649 packets 167380042 bytes 2363 pkts/sec 1342800 bytes/sec !


show np blocks (FWSM Only)

  The show np blocks command is used to see if the FWSM is over subscribed

FWSM# show np blocks

MAX FREE THRESH_0 THRESH_1 THRESH_2 NP1 (ingress) 32768 32768 0 0 550 (egress) 521206 521206 0 0 0 NP2 (ingress) 32768 32768 0 0 92 (egress) 521206 521206 0 0 0 NP3 (ingress) 32768 32768 13 460417 4427509 (egress) 521206 521206 0 0 0

Warning

Data packets dropped

Data and Control packets

dropped


Show Xlate and Show Xlate Debug

  The show xlate command displays information about the translations through the firewall

  You can limit the output to just the local or global IP ASA-5585# show xlate 5014 in use, 5772 most used TCP PAT from inside:192.168.103.220/57762 to outside:10.2.1.2/43756 flags ri idle 0:00:00 timeout 0:00:30 TCP PAT from inside:192.168.103.220/57761 to outside:10.2.1.2/54464 flags ri idle 0:00:00 timeout 0:00:30

ASA-5585# show nat pool TCP PAT pool outside, address 10.2.1.2, range 1-511, allocated 1 TCP PAT pool outside, address 10.2.1.2, range 512-1023, allocated 0 TCP PAT pool outside, address 10.2.1.2, range 1024-65535, allocated 2321 ASA-5585#

Added in version 8.3


Show Nat Detail

  The show nat command displays information about the nat table of the firewall

  The detail keyword will display object definitions ASA-5585# show nat detail Manual NAT Policies (Section 1) 1 (inside) to (outside) source static science-obj science-obj destination static vpn-obj vpn-obj translate_hits = 0, untranslate_hits = 0 Source - Origin: 192.168.0.0/16, Translated: 192.168.0.0/16 Destination - Origin: 172.16.1.0/24, Translated: 172.16.1.0/24

Auto NAT Policies (Section 2) 1 (dmz) to (outside) source static webserver-obj 14.36.103.83 translate_hits = 0, untranslate_hits = 3232 Source - Origin: 192.168.22.32/32, Translated: 14.36.103.83/32 2 (inside) to (outside) source dynamic science-obj interface translate_hits = 37723, untranslate_hits = 0 Source - Origin: 192.168.0.0/16, Translated: 14.36.103.96/16 ASA-5585/admin#


ASA# show conn 2 in use, 64511 most used

TCP outside 198.133.219.25:80 dmz 10.9.9.3:4101, idle 0:00:06, Bytes 127, flags UIO UDP outside 172.18.124.1:123 dmz 10.1.1.9:123 idle 0:00:13 flags –

Show Conn and Show Conn Detail

ASA# show conn detail 2 in use, 64511 most used Flags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN, B - initial SYN from outside, C - CTIQBE media, D - DNS, d - dump, E - outside back connection, F - outside FIN, f - inside FIN, G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data, i - incomplete, J - GTP, j - GTP data, K - GTP t3-response k - Skinny media, M - SMTP data, m - SIP media, n - GUP O - outbound data, P - inside back connection, q - SQL*Net data, R - outside acknowledged FIN, R - UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN, s - awaiting outside SYN, T - SIP, t - SIP transient, U - up, W - WAAS, X - inspected by service module

TCP outside:198.133.219.25/80 dmz:10.9.9.3/4101, flags UIO, idle 8s, uptime 10s, timeout 1h, bytes 127 UDP outside:172.18.124.1/123 dmz:10.1.1.9/123, flags -, idle 15s, uptime 16s, timeout 2m, bytes 1431

detail Adds Uptime and Timeout in 7.2(4), 8.0(4)

Idle Time, Bytes Transferred

Connection Flags

Real Interface Names Added in

7.2(4), 8.0(4)


3 ACK 5 Data 1 SYN 4 Data 2 SYN+ACK

Example—Connection Build Up

  Firewall receives an initial SYN packet from the inside; the SYN is permitted by the access-list, a translation (xlate) is built up, and the connection is also created with the flags saA

  The outside device responds to the SYN packet with a SYN+ACK; the connection flags are updated to reflect this, and now show A

  The inside device responds to the SYN+ACK with an ACK and this completes the TCP three-way handshake, and the connection is now considered up (U flag)

  The outside device sends the first data packet; the connection is updated and an I is added to the flags to indicate the firewall received Inbound data on that connection

  Finally, the inside device has sent a data packet and the connection is updated to include the O flag

U saA A UIO UI Connection Flags

Client Server

Outside Inside


1 FIN 2 FIN+ACK UfFR Uf 3 ACK UfFRr

Example—Connection Teardown

  Firewall receives a FIN packet from the inside; as the FIN passes through the firewall, it updates the connection flags by adding an f to indicate that the FIN was received on the Inside interface

  The outside device immediately responds to the FIN packet with a FIN+ACK; the connection flags are updated to reflect this, and now show UfFR

  The inside device responds to the FIN+ACK with a final ACK and the firewall tears down the connection; thus, there are no more connection flags, because the connection no longer exists

Connection Flags

Client Server

Outside Inside


For your reference

Outbound Connection Inbound Connection

Connection Flags—Quick Reference


TCP Connection Termination Reasons

  If a TCP connection is built through the firewall, it will always have a teardown reason

  The TCP teardown syslog is logged at level six

  If you are having problems with connections abnormally closing, temporally increase your logging level (or move the syslog down), and check the teardown reason

ASA-6-302014: Teardown TCP connection number for intf_name:real_IP/real_port to intf_name:real_IP/real_port duration time bytes number [reason] [(user)]

What does the Reset-O Termination reason mean in the Teardown TCP connection syslog?


For your reference

TCP Connection Termination Reasons—Quick Reference

Reason Description

Conn-Timeout Connection Ended Because It Was Idle Longer Than the Configured Idle Timeout

Deny Terminate Flow Was Terminated by Application Inspection

Failover Primary Closed The Standby Unit in a Failover Pair Deleted a Connection Because of a Message Received from the Active Unit

FIN Timeout Force Termination After Ten Minutes Awaiting the Last ACK or After Half-Closed Timeout

Flow Closed by Inspection Flow Was Terminated by Inspection Feature Flow Terminated by IPS Flow Was Terminated by IPS Flow Reset by IPS Flow Was Reset by IPS Flow Terminated by TCP Intercept Flow Was Terminated by TCP Intercept

Invalid SYN SYN Packet Not Valid

Idle Timeout Connection Timed Out Because It Was Idle Longer than the Timeout Value

IPS Fail-Close Flow Was Terminated Due to IPS Card Down SYN Control Back Channel Initiation from Wrong Side


For your reference

TCP Connection Termination Reasons—Quick Reference (Cont.)

Reason Description

SYN Timeout Force Termination After Two Minutes Awaiting Three-Way Handshake Completion

TCP Bad Retransmission Connection Terminated Because of Bad TCP Retransmission

TCP Fins Normal Close Down Sequence

TCP Invalid SYN Invalid TCP SYN Packet

TCP Reset-I TCP Reset Was Sent From the Inside Host

TCP Reset-O TCP Reset Was Sent From the Outside Host

TCP Segment Partial Overlap Detected a Partially Overlapping Segment

TCP Unexpected Window Size Variation

Connection Terminated Due to a Variation in the TCP Window Size

Tunnel Has Been Torn Down Flow Terminated Because Tunnel Is Down

Unauth Deny Connection Denied by URL Filtering Server

Unknown Catch-All Error

Xlate Clear User Executed the ‘Clear Xlate’ Command


show local-host   A local-host entry is created for any IP tracked through the

firewall

  It groups the xlates, connections, and AAA information

  Very useful for seeing the connections terminating on servers

ASA# show local-host

Add ‘show local-host detail connection arguments’

ASA# show local-host detail connection tcp 50 Interface dmz: 0 active, 0 maximum active, 0 denied Interface inside: 1 active, 1 maximum active, 0 denied local host: <192.168.103.220>, TCP flow count/limit = 798/unlimited TCP embryonic count to host = 0 TCP intercept watermark = unlimited UDP flow count/limit = 0/unlimited Conn: TCP outside:172.18.124.76/80 inside:192.168.103.220/34078, flags UO, idle 0s, uptime 0s, timeout 30s, bytes 0 TCP outside:172.18.124.76/80 inside:192.168.103.220/34077, flags UO, idle 0s, uptime 0s, timeout 30s, bytes 0 (output truncated)


show service-policy   The show service-policy command is used to quickly see

what inspection policies are applied and the packets matching them ASA-5585/admin# show service-policy

Global policy: Service-policy: global_policy Class-map: inspection_default Inspect: ftp, packet 0, lock fail 0, drop 0, reset-drop 0 Inspect: h323 h225 _default_h323_map, packet 0, drop 0, reset-drop 0 tcp-proxy: bytes in buffer 0, bytes dropped 0 Inspect: h323 ras _default_h323_map, packet 0, drop 0, reset-drop 0 Inspect: rsh, packet 0, lock fail 0, drop 0, reset-drop 0 Inspect: rtsp, packet 0, lock fail 0, drop 0, reset-drop 0 tcp-proxy: bytes in buffer 0, bytes dropped 0 Inspect: sqlnet, packet 0, lock fail 0, drop 0, reset-drop 0 Inspect: http, packet 1215927, lock fail 0, drop 0, reset-drop 0 Inspect: icmp, packet 57, lock fail 0, drop 0, reset-drop 0 ASA-5585/admin# ... Interface outside: Service-policy: VoIP Class-map: voice_marked Priority: Interface outside: aggregate drop 0, aggregate transmit 349


show service-policy flow

  Use to determine what policies a given flow will match in the Modular Policy Framework (MPF)

ASA# show service-policy flow tcp host 10.1.9.6 host 10.8.9.3 eq 1521

Global policy: Service-policy: global_policy

Interface outside: Service-policy: outside Class-map: oracle-dcd Match: access-list oracle-traffic Access rule: permit tcp host 10.1.9.6 host 10.8.9.3 eq sqlnet Action: Input flow: set connection timeout dcd


show asp drop   Packets dropped in the Accelerated Security Path (ASP) will

increment a counter   FWSM – applies only to traffic sent to the control-point   Frame drop counters are per packet, flow drops are per flow   Some counters have corresponding syslogs

ASA# show asp drop

Frame drop: Invalid encapsulation (invalid-encap) 10897 Invalid tcp length (invalid-tcp-hdr-length) 9382 Invalid udp length (invalid-udp-length) 10 No valid adjacency (no-adjacency) 5594 No route to host (no-route) 1009 Reverse-path verify failed (rpf-violated) 15 Flow is denied by access rule (acl-drop) 25247101 First TCP packet not SYN (tcp-not-syn) 36888 Bad TCP flags (bad-tcp-flags) 67148 TCP option list invalid (tcp-bad-option-list) 731 TCP MSS was too large (tcp-mss-exceeded) 10942 Bad TCP Checksum (bad-tcp-cksum) 893

*Drop Counters Are Documented in the CMD Ref, Under show asp drop


Packet Capture

  Capture command first introduced in Cisco 7.0; FWSM need to use 3.1.5 or later

  ASA 7.2(3) and 8.0(3) added a real-time option   ASDM 6.0 adds a capture wizard   Capture sniffs packets on an interface that match an ACL,

or match line   Key steps

Use the ‘match’ keyword to specify what traffic to capture (implicitly bi-directional) Define the capture and bind it to an access-list and interface View the capture on the firewall, or copy it off in .pcap format

capture <capture-name> [access-list <acl-name>] [buffer <buf-size>] [ethernet-type <type>] [interface <if-name>] [packet-length <bytes>] [circular-buffer] [type raw-data|asp-drop|isakmp|webvpn user <username>] [match <prot> {host <sip> | <sip> <mask> | any} [eq | lt |gt <port>] {host <dip> | <dip> <mask> | any} [eq | lt | gt <port>]] [real-time [dump] [detail] [trace]] [trace [detail] [trace-count <1-1000>]]


Packet Capture (Cont.)

 Traffic can be captured both before and after it passes through the firewall; one capture on the inside interface, one capture on the outside interface

 Capture buffer saved in RAM (default size 512 KB)  Default is to stop capturing when buffer is full

 Default packet length is 1518 bytes  Copy captures off via TFTP or HTTPS

Outside Inside

Capture In Capture Out

See Appendix

Inside Capture Outside Capture


Where Packets Are Captured in Packet Flow

 Packets are captured at the first and last points they can be in the flow

  Ingress packets are captured before any packet processing has been done on them

 Egress packets are captured after all processing (including L2 source MAC rewrite)

Ingress Packets Captured

Egress Packets Captured


Capturing Packets Dropped by the ASP

  Capture all packets dropped by the ASP ASA# capture drops type asp-drop all

  Capture on a specific drop reason ASA# capture drops type asp-drop tcp-not-syn

  Applies to both ASA and FWSM

ASA# capture drop type asp-drop ?

acl-drop Flow is denied by configured rule all All packet drop reasons bad-crypto Bad crypto return in packet bad-ipsec-natt Bad IPSEC NATT packet bad-ipsec-prot IPSEC not AH or ESP bad-ipsec-udp Bad IPSEC UDP packet bad-tcp-cksum Bad TCP checksum bad-tcp-flags Bad TCP flags


Packet Tracer: Overview

  Introduced in ASA ver 7.2  A packet tagged with the trace option is injected into the

interface, and processed in the data-plane  Each action taken on the packet is recorded in the packet

itself

 When the packet reaches the egress interface, or is dropped, it is punted to the control-plane

 The control-plane reads and displays the actions taken on the packet, along with the associated lines in the configuration

ASA Only


Link Back to Edit Rule

Matching Config

Define Packet

Action

Final Result

Packet Tracer: ASDM (Located off Tools Menu)


Packet Tracer: Example Output ASA# packet-tracer input inside tcp 10.1.1.2 1024 198.133.219.25 80

Phase: 1 Type: FLOW-LOOKUP Subtype: Result: ALLOW Config: Additional Information: Found no matching flow, creating a new flow

Phase: 2 Type: ACCESS-LIST Subtype: log Result: ALLOW Config: access-group in in interface inside access-list in extended permit tcp any any eq www Additional Information:

Phase: 3 Type: INSPECT Subtype: np-inspect Result: ALLOW Config: class-map match-all inspection_default match default-inspection-traffic policy-map global_policy class inspection_default inspect http service-policy global_policy global Additional Information:


Packet Tracer: Example Output (Cont.)

...

Phase: 10 Type: NAT Subtype: Result: ALLOW Config: nat (inside) 1 10.1.1.0 255.255.255.0 Additional Information: Dynamic translate 10.1.1.2/4 to 209.165.201.3/516 using netmask 255.255.255.255

...

Phase: 15 Type: ROUTE-LOOKUP Subtype: output and adjacency Result: ALLOW Config: Additional Information: found next-hop 209.165.201.1 using egress ifc outside adjacency Active next-hop mac address 000a.f331.83c0 hits 0

>>>>Packet successfully forwarded to fast path<<<<


Packet Tracer: Tracing Captured Packet

  Create a capture using the trace option

  Find the packet in the capture you want traced

  Then select that packet to be traced

ASA# show capture inside trace packet-number 4 .

ASA# capture inside access-list web interface inside trace .

ASA# show capture inside 68 packets captured 1: 15:22:47.581116 10.1.1.2.31746 > 198.133.219.25.80: S 2: 15:22:47.583465 198.133.219.25.80 > 10.1.1.2.31746: S ack 3: 15:22:47.585052 10.1.1.2.31746 > 198.133.219.25.80: . ack 4: 15:22:49.223728 10.1.1.2.31746 > 198.133.219.25.80: P ack 5: 15:22:49.223758 198.133.219.25.80 > 10.1.1.2.31746: . Ack ...

Important!


TCP Ping

 New

www server (209.165.200.225)

 New troubleshooting tool added in ASA ver 8.4.1

 Why is it needed??? Consider the following…

10.1.1.7


TCP Ping

 Previously – limited reachability tools: Ping and Traceroute

 Access to client machine?

www server (209.165.200.225)

ICMP Echo Request ICMP Echo Reply

ICMP Echo Request ICMP Echo Reply ICMP Echo Reply ICMP Echo Request

Attempts to validate the path …but with ICMP

What about NAT and/or PAT?

10.1.1.7


TCP Ping  Sources TCP SYN packet with Client’s IP and

injects it into Client’s interface of the ASA

Internal hosts are PATed to 198.51.100.2

www server (209.165.200.225) 10.1.1.7

inside outside

Packet with SRC of 10.1.1.7 injected on Inside interface

Packet PATed to 198.51.100.2

on Egress

ASA Datapath Validated

(NAT, ACLs, etc)

TCP SYN sent to server

TCP SYN+ACK sent from server


TCP Ping – The Big Picture  Validates 2 of the 3 legs of the connection from

client to server

www server (209.165.200.225) 10.1.1.7

inside outside

TCP path from client side of ASA to Server through the cloud

-Validated-

2nd Leg 1st Leg 3rd Leg


TCP Ping - Example

www server (209.165.200.225) 10.1.1.7

inside outside

asa# ping tcp Interface: inside Target IP address: 209.165.200.225 Target IP port: 80 Specify source? [n]: y Source IP address: 10.1.1.7 Source IP port: [0] Repeat count: [5] Timeout in seconds: [2] Type escape sequence to abort. Sending 5 TCP SYN requests to 209.165.200.225 port 80 from 10.1.1.7 starting port 3465, timeout is 5 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

Specify Client’s source Interface

Specify Client’s real IP Address


Agenda




Case Study Leveraging Smart Call Home


Case Study: Smart Call Home

  Objective – Send the output of a command directly to your e-mail.

  This is easily accomplished with SCH. Use the command:

call-home send <“cmd”> email <email_addr>

Example: call-home send “show run” email [email protected]

  This will send a plain-text e-mail with the output of the command to the e-mail address specified, with the command in the subject line.

Example: Subject: CLI ‘show run’ output

Email CMD Output to You

ASA Only


Case Study: Smart Call Home

 Objective – Memory appears to be depleting over time on your ASA. Use SCH to collect the detailed memory output hourly, for further investigation.

 This is easily accomplished with SCH. Setting a ”snapshot” alert-group to e-mail commands at a specified interval

 Snapshot will contain the following command: show conn count show memory detail

Collecting Memory Diagnostics over Time


Case Study: Smart Call Home Example Config

service call-home call-home alert-group-config snapshot add-command “show conn count” add-command "show memory detail“ contact-email-addr [email protected] sender from [email protected] sender reply-to [email protected] mail-server smtp-server.cisco.com priority 1 profile SENDCMD active destination address email [email protected] destination preferred-msg-format long-text destination transport-method email subscribe-to-alert-group snapshot periodic hourly

Case Study Intermittent Access to Web Server


Clients

Problem

  Most external clients are not able to load company’s web page

10.1.1.50 ASA-5510

HTTP Requests to 192.168.1.50

Web Server

NATed to 10.1.1.50

Internet

Case Study: Intermittent Access to Web Server


Traffic Spike



  show perfmon indicates high number of embryonic connections

ASA-5510# show perfmon

PERFMON STATS: Current Average Xlates 0/s 0/s Connections 2059/s 299/s TCP Conns 2059/s 299/s UDP Conns 0/s 0/s URL Access 0/s 0/s URL Server Req 0/s 0/s TCP Fixup 0/s 0/s TCP Intercept Established Conns 0/s 0/s TCP Intercept Attempts 0/s 0/s TCP Embryonic Conns Timeout 1092/s 4/s HTTP Fixup 0/s 0/s FTP Fixup 0/s 0/s AAA Authen 0/s 0/s AAA Author 0/s 0/s AAA Account 0/s 0/s

VALID CONNS RATE in TCP INTERCEPT: Current Average N/A 95.00%



  Issue show conn to see ‘who’ is creating the connections

ASA-5510# show conn 54764 in use, 54764 most used TCP outside 17.24.101.118:26093 inside 10.1.1.50:80, idle 0:00:23, bytes 0, flags aB TCP outside 111.76.36.109:23598 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 24.185.110.202:32729 inside 10.1.1.50:80, idle 0:00:25, bytes 0, flags aB TCP outside 130.203.2.204:56481 inside 10.1.1.50:80, idle 0:00:29, bytes 0, flags aB TCP outside 39.142.106.205:18073 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 75.27.223.63:51503 inside 10.1.1.50:80, idle 0:00:03, bytes 0, flags aB TCP outside 121.226.213.239:18315 inside 10.1.1.50:80, idle 0:00:04, bytes 0, flags aB TCP outside 66.187.75.192:23112 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB TCP outside 13.50.2.216:3496 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 99.92.72.60:47733 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aB TCP outside 30.34.246.202:20773 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 95.108.110.131:26224 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 76.181.105.229:21247 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB TCP outside 82.210.233.230:44115 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 134.195.170.77:28138 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB TCP outside 70.133.128.41:22257 inside 10.1.1.50:80, idle 0:00:15, bytes 0, flags aB TCP outside 124.82.133.172:27391 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aB TCP outside 26.147.236.181:37784 inside 10.1.1.50:80, idle 0:00:07, bytes 0, flags aB TCP outside 98.137.7.39:20591 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 37.27.115.122:24542 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB . . .

Random Sources Embryonic Conns




Connection Count Jumps

Traffic Permitted

SYN Flood Detected


  Apply TCP Intercept to stop the SYN flood attack

access-list 140 extended permit tcp any host 192.168.1.50 eq www ! class-map protect description Protect web server from attacks match access-list 140 ! policy-map interface_policy class protect set connection embryonic-conn-max 100 ! service-policy interface_policy interface outside



TCP Intercept Applied

Few Clients Represent 50+ % of Traffic


Why did the Connection count drop after TCP Intercept was applied?


access-list 140 extended permit tcp any host 192.168.1.50 eq www ! class-map protect description Protect web server from attacks match access-list 140 ! policy-map interface_policy class protect set connection embryonic-conn-max 100 per-client-max 25 ! service-policy interface_policy interface outside

  Apply per-client-max option to limit the number of connections any single client can establish



TCP Intercept

per-client-max



Attacks Still Occurring

Attacks Being Mitigated



Agenda





Online Resources

  Support Communities - Supportforums.cisco.com

  TAC Security Show Podcast

  Online learning modules (VoD Training)

  Security RSS Feeds


Supportforums.cisco.com

  Public wiki – anyone can author articles

  Combines supportwiki and Netpro forums

  Sections for: ASA, FWSM and PIX

  Hundreds of Sample Configs

  Troubleshooting Docs

  FAQs

http://supportforums.cisco.com/


TAC Security Podcast

 Great way to obtain valuable troubleshooting insights.

 Conversational shows, which focus on providing in-depth information on a given feature.

 New episodes posted Monthly

http://www.cisco.com/go/tacsecuritypodcast/


TAC Security Podcast Episodes

Search iTunes for TAC Security Podcast


Online Learning Modules – VoD Training

  Great way to learn about new features in the ASA

  From www.cisco.com select: Products and Services  Security

 Network Security (expand)

 Cisco ASA 5500 Series

 Training resources

 Online learning modules

  Search cisco.com for ASA Online Learning Modules

  Direct link http://www.cisco.com/en/US/partner/products/ps6120/tsd_

products_support_online_learning_modules_list.html


Security Hot Issues – RSS Feeds

  Subscribe with an RSS reader

  Receive weekly updates on the Hot Issues customers are facing

  Separate feeds for: ASA, FWSM, ASDM

https://supportforums.cisco.com/docs/DOC-5727


Agenda

  Packet Flow

  Understanding the Architecture

  Failover

  Troubleshooting

  Case Studies

  Online Resources

  Best Practices


Cisco ASA/FWSM Best Practices

  Enable ip verify reverse-path on all interfaces   Set embryonic and maximum connection counts on static and

nat statements; for 7.2.1+ use per-client-max   Configure logging to syslog server   Move messages you want to see to lower levels, instead

of raising logging levels and capturing messages you don’t want to see

  Disable telnet access! Use SSH for management access   Enable authentication for management access (console/

SSH/telnet/enable); use TACACS+ or RADIUS with LOCAL as the fallback


Cisco ASA/FWSM Best Practices

  Restrict DMZ access inbound to your internal networks

  Baseline CPU load, connection counts, xlate counts, and traffic (per interface)

  Monitor stats using MRTG or other snmp graphing tools.

  Keep config archives (and show tech ouputs) (smart call home)

  Run the latest maintenance release in your train

  Upgrade major feature trains only when you need new features, or after train has matured


ASA Software Trains 7.0.1

7.0.2 7.0.4 7.0.8

7.0

7.1

7.2

8.0

8.1

8.2

8.3

7.0.7 7.0.6 7.0.5 EOL

8.3.1

8.0.5 8.0.4 8.0.3 8.0.2

8.2.2 8.2.1

8.1.2 8.1.1

7.1.1 7.1.2

7.2.1 7.2.2 7.2.3 7.2.4

EOL

EOL

7.2.5

8.3.2

8.2.3

Bug Fixes Waterfall Down

8.4 8.4.1 8.4.2

8.2.4 8.2.5

8.5 8.5.1

ASA-5580 only

ASA-SM only

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSEC-3020 110 110

Visit the Cisco Store for Related Titles

http://theciscostores.com

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSEC-3020 111 111

  Receive 25 Cisco Preferred Access points for each session evaluation you complete.

  Give us your feedback and you could win fabulous prizes. Points are calculated on a daily basis. Winners will be notified by email after July 22nd.

  Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.

  Don’t forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and on-demand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.

Complete Your Online Session Evaluation


Thank you.


Appendix

  Lucky You

  This appendix contains extra information which you may find useful, but I just didn’t have enough time to cover in the lecture – or which was covered in previous years.

  Enjoy… :-)


Appendix   ASA 8.3 Memory Requirements   SNMP OIDs to Monitor   Example: Show Output Filters   Code Base History   Case studies

Poor Voice Quality Out-of-order packet buffering TCP MSS issue Out of memory High CPU Capture Example

  FWSM Additional Architecture Slides   Failover Extras   Packet Capture Example   Online Tools   ASDM   Information to include when opening a TAC case


Redirecting Debugs to Syslog

 Problem Log only debug output to syslog

 Solution Create a logging list with only syslog ID 711001

Enable debug output to syslogs

Log on the logging list

ASA(config)# logging trap Networkers .

ASA(config)# logging list Networkers message 711001 .

ASA(config)# logging debug-trace INFO: 'logging debug-trace' is enabled. All debug messages are currently being redirected to syslog:711001 and will not appear in any monitor session


ASA 8.3 Memory Requirements

 ASA Models 5505 – 5540 Require Memory Upgrades before upgrading to ASA version 8.3

 New ASAs ship with the upgraded RAM installed

* For the 5505, only the Security Plus or Unlimited licenses require the memory upgrade

ASA Model Original Default RAM

Required RAM for version 8.3

Upgrade Kit Part Number

5505 * 256 MB 512 MB ASA5505-MEM-512=

5510 256 MB 1024 MB ASA5510-MEM-1GB=

5520 512 MB 2048 MB ASA5520-MEM-2GB=

5540 1024 MB 2048 MB ASA5540-MEM-2GB=


SNMP OIDs

  CPU usage

•  1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 (5 sec)

•  1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 (1 min)

•  1.3.6.1.4.1.9.9.109.1.1.1.1.5.1 (5 min)

  Connections

•  1.3.6.1.4.1.9.9.147.1.2.2.2.1.5.40.6 (Current total)

•  1.3.6.1.4.1.9.9.147.1.2.2.2.1.5.40.7 (Max total)

  Traffic

•  1.3.6.1.2.1.2.2.1.{10|16}.n (in/out octets)

•  Use SNMPwalk to verify the interfaces!

For your reference


Example: Show Output Filters

Examples   Display the interface stats starting with the ‘inside’ interface

show interface | begin inside

  Display the access-list entries that contain address 10.1.1.5 show access-list | grep 10.1.1.5

  Display the config, except for the access-lists show run | exclude access-list

  Display only access-list entries that have non-zero hitcounts show access-list | grep –v hitcnt=0

  Display a count of the number of connections each host has show local-host | include host|count/limit

show <cmd> | begin|include|exclude|grep [-v] <regular_exp>

Note: You must Include a Space on Either Side of the Pipe for the Command to Be Accepted; Also, Trailing Spaces Are Counted


Feature Releases

Cisco PIX/ASA/FWSM Code Base History

6.0(1) 6.1(1) 6.2(1) 6.3(1) PIX

1.1(3)

FWSM 1.1(1)

Feature Releases

1.1(2)

Maintenance Releases

2.2(1)

Port Features Bug Fixes

2.3(1)

2.3(2)

Time

7.0(1)

3.1(1)

7.1(1) 7.2(1)

3.2(1)

PIX/ASA In Sync

4.0(1)

3.1(2) 3.1(6) GD

3.1(10) SafeHarbor

3.2(2) 3.2(4) SafeHarbor

4.1(1)

8.0(2) 8.3(1)

3.2(17)

3.1(17)

4.0(2) 4.0(4) 4.0(11) SafeHarbor

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public Presentation_ID 120

Case Study Poor Voice Quality


Problem

  Poor outbound voice quality at SOHO sites

Case Study: Poor Voice Quality

100 Mbps 100 Mbps Cable Modem 2 Mbps

WAN

ASA-5505

Outbound RTP Stream



Solution: Traffic Shaping

  What is traffic shaping, and why is it needed here?

  Why won’t policing work?

  Why won’t priority queuing alone work?

100 Mbps 100 Mbps

Cable Modem 2 Mbps

WAN

ASA-5505

Shape to 2 Mbps


Case Study: Poor Voice Quality – Configuration Example (Traffic Shaping)

class-map voice-traffic ! match dscp af13 ef!!!policy-map qos_class_policy ! class voice-traffic ! priority!!!policy-map qos_outside_policy ! class class-default ! shape average 2000000 ! service-policy qos_class_policy!!!service-policy qos_outside_policy interface outside!

Solution   Prioritize voice traffic and shape all traffic down to 2 Mbps on the outside

interface.

  To view statistics on the operation of the shaper, use the command show service-policy shape



Things to Keep in Mind:

  Shaping can only be applied to the class class-default   Shaping only works in the outbound direction on

an interface

  The shaping value is in bits per second, and must be a multiple of 8000

  The shaping policy is applied to all sub-interfaces on a physical interface

  Not supported on the ASA-5580 platform

  Not supported in Transparent or Multi-context mode


Case Study Out-of-Order Packet Buffering


Case Study: Out-of-Order Packets

  Inspections require ordered packets

  Packets sent to the SSM (AIP and CSC) require ordered packets

  Cisco ASA/PIX will buffer up to three packets by default

  Buffering can be increased on ASA by using the queue-limit option under the tcp-map


Case Study: Out-of-Order Packets

  Some networks have high numbers of out-of-order packets; often caused by asymmetric traffic flows

  If the out-of-order packet buffer isn’t large enough, traffic is dropped and packets must be retransmitted

Outside Inside

Client Server

Packet 10

10.16.9.2 192.168.1.30

Packet 12 Packet 13 Packet 14 Packet 15

Buffer

Dropped by Firewall

Packet 11 Dropped on Network

Problem


Case Study: Out-of-Order Packet Buffering Example

  How to detect? ASA# show asp drop Frame drop: ... TCP packet SEQ past window 46331 TCP packet buffer full 90943 ...

access-list OOB-nets permit tcp any 10.16.9.0 255.255.255.0 ! tcp-map OOO-Buffer queue-limit 6 ! class-map tcp-options match access-list OOB-nets ! policy-map global_policy class tcp-options set connection advanced-options OOO-Buffer ! service-policy global_policy global

  How to fix?


Case Study: Out-of-Order Packet Buffering Example

  How to verify?

ASA# show service-policy

Global policy: Service-policy: global_policy Class-map: inspection_default ... Class-map: tcp-options Set connection policy: Set connection advanced-options: OOB-Buffer Retransmission drops: 0 TCP checksum drops : 0 Exceeded MSS drops : 0 SYN with data drops: 0 Out-of-order packets: 2340 No buffer drops : 0


Case Study TCP MSS (Maximum Segment Size)


Case Study: TCP MSS

  MSS is the Maximum Segment Size—or the maximum amount of data that can be sent in a single packet

  The MSS is set in the SYN packets

  The device that receives the MSS advertisement cannot send more data in a single packet to the peer than specified by the MSS


Case Study: TCP MSS

  Some servers have broken TCP stacks and ignore the MSS advertised by the Client

  The firewall will drop packets that exceed the advertised MSS

Outside Inside

Client Server

SYN MSS=1380

SYN+ACK MSS=1400

DATA=1390

10.16.9.2 192.168.1.30

Problem


Case Study: TCP MSS Example

  How to detect? ASA# show asp drop Frame drop: TCP MSS was too large 943

%ASA-4-419001: Dropping TCP packet from outside:10.16.9.2/80 to inside:192.168.1.30/1025, reason: MSS exceeded, MSS 1380, data 1390

access-list MSS-hosts permit tcp any host 10.16.9.2 ! tcp-map mss-map exceed-mss allow ! class-map mss match access-list MSS-hosts ! policy-map global_policy class mss set connection advanced-options mss-map ! service-policy global_policy global

  How to fix?


Case Study: TCP MSS Example

  How to verify?

ASA# capture mss-capture type asp-drop tcp-mss-exceeded packet-length 1518

ASA# show capture mss-capture 0 packets captured 0 packets shown

  How else could you verify?


Case Study Out of Memory


Case Study: Out of Memory

  Users are unable to access the Internet

  No new connections are working

  All old (long lived) connections continue to work

Step 1: Check the Syslogs %PIX-3-211001: Memory allocation Error %PIX-3-211001: Memory allocation Error

Step 2: Check the Amount of Free Memory Available Hardware: PIX-515E, 64 MB RAM

pixfirewall# show memory Free memory: 714696 bytes Used memory: 66394168 bytes ------------- ---------------- Total memory: 67108864 bytes

Problem



Step 3: What Eats Up Memory (RAM) on the Cisco PIX?   Cisco PIX image (run from RAM)

  Configuration

  IPSec database

  Xlates (translations)

  Connections

What Can Eat Up 64 MB on a Cisco PIX-515E?

Step 4: Let’s Check the Translations pixfirewall# show xlate 251 in use, 258 most used PAT Global 209.165.201.26(2379) Local 10.1.1.132(52716) PAT Global 209.165.201.26(2378) Local 10.1.1.227(20276) Global 209.165.201.25 Local 10.1.1.102 PAT Global 209.165.201.26(2255) Local 10.1.1.125(12783) PAT Global 209.165.201.26(2382) Local 10.1.1.175(39197) PAT Global 209.165.201.26(2254) Local 10.1.1.34(43543)

Varied Source IPs

A Small Global Pool Is Used, Overloading to a PAT Address


pixfirewall# show conn 147456 in use, 147456 most used TCP out 64.102.144.194:80 in 10.1.1.38:26749 idle 0:00:19 Bytes 312 flags OIU TCP out 64.101.22.236:80 in 10.1.1.74:32209 idle 0:00:14 Bytes 239 flags OIU TCP out 64.102.147.77:21 in 10.1.1.48:32893 idle 0:00:48 Bytes 0 flags saA TCP out 64.103.31.215:80 in 10.1.1.136:18664 idle 0:00:46 Bytes 934 flags OIU TCP out 64.101.19.69:80 in 10.1.1.235:46712 idle 0:00:17 Bytes 8394 flags OIU TCP out 64.101.205.10:135 in 10.1.1.139:62296 idle 0:00:15 Bytes 0 flags saA TCP out 64.101.200.200:80 in 10.1.1.83:51864 idle 0:00:32 Bytes 902 flags OIU TCP out 64.102.80.27:80 in 10.1.1.66:52301 idle 0:00:03 Bytes 7813 flags OIU TCP out 64.103.95.35:80 in 10.1.1.231:51532 idle 0:00:24 Bytes 3891 flags OIU TCP out 64.102.206.172:80 in 10.1.1.223:28585 idle 0:00:28 Bytes 239 flags OIU TCP out 64.102.57.106:80 in 10.1.1.135:44945 idle 0:00:48 Bytes 9717 flags OIU TCP out 64.102.21.85:80 in 10.1.1.20:19578 idle 0:00:06 Bytes 2348 flags OIU TCP out 64.101.25.203:80 in 10.1.1.170:28149 idle 0:00:47 Bytes 419 flags OIU TCP out 64.101.86.97:135 in 10.1.1.54:43703 idle 0:00:12 Bytes 0 flags saA . . .


Step 5: Check the Connections

  Q: Why is the connection count so high?


pixfirewall# show traffic outside: received (in 25.000 secs): 1475 packets 469050 bytes 59 pkts/sec 18762 bytes/sec transmitted (in 25.000 secs): 167619 packets 9654480 bytes 6704 pkts/sec 386179 bytes/sec inside: received (in 25.000 secs): 180224 packets 10410480 bytes 7208 pkts/sec 416419 bytes/sec transmitted (in 25.000 secs): 1050 packets 118650 bytes 42 pkts/sec 4746 bytes/sec


  Vast majority of traffic is coming in the inside interface and going out the outside interface

Out

side

In

side

Traf

fic F

low

Take a Look at the Traffic Load


pixfirewall# show conn count 147456 in use, 147456 most used

pixfirewall# show xlate count 251 in use, 258 most used


Step 6: Review What We Know and Take Action

Conn Count Is Very High, but xlate Count Is Low

  Many connections per xlate

  Probably one, or a few hosts, are generating the vast majority of connections

  Most likely due to a virus on the host(s)


pixfirewall# show local-host | include host|count/limit local host: <10.1.1.131>, TCP connection count/limit = 0/unlimited UDP connection count/limit = 0/unlimited local host: <10.1.1.51>, TCP connection count/limit = 2/unlimited UDP connection count/limit = 0/unlimited local host: <10.1.1.236>, TCP connection count/limit = 0/unlimited UDP connection count/limit = 0/unlimited . . . local host: <10.1.1.99>, TCP connection count/limit = 146608/unlimited UDP connection count/limit = 0/unlimited


Step 7: Find the Host(s) Generating All the Connections

Only Show Lines That Have the Word host or count/limit in Them

  Host 10.1.1.99 is eating up all the connections, and they are TCP-based connections


pixfirewall# show local-host 10.1.1.99 Interface inside: 250 active, 250 maximum active, 0 denied local host: <10.1.1.99>, TCP connection count/limit = 146608/unlimited TCP embryonic count = 146606 UDP connection count/limit = 0/unlimited Xlate(s): Global 209.165.201.21 Local 10.1.1.99 Conn(s): TCP out 64.101.32.157:135 in 10.1.1.99:34580 idle 0:01:43 Bytes 0 flags saA TCP out 64.103.108.191:135 in 10.1.1.99:8688 idle 0:01:43 Bytes 0 flags saA TCP out 64.100.205.160:135 in 10.1.1.99:7774 idle 0:01:43 Bytes 0 flags saA TCP out 64.101.182.19:135 in 10.1.1.99:39193 idle 0:01:43 Bytes 0 flags saA TCP out 64.102.218.45:135 in 10.1.1.99:16462 idle 0:01:43 Bytes 0 flags saA TCP out 64.100.21.120:135 in 10.1.1.99:30322 idle 0:01:43 Bytes 0 flags saA TCP out 64.101.25.195:135 in 10.1.1.99:41116 idle 0:01:43 Bytes 0 flags saA TCP out 64.103.17.219:135 in 10.1.1.99:59163 idle 0:01:43 Bytes 0 flags saA TCP out 64.102.201.141:135 in 10.1.1.99:2978 idle 0:01:43 Bytes 0 flags saA TCP out 64.103.176.75:135 in 10.1.1.99:41589 idle 0:01:43 Bytes 0 flags saA . . .


Step 8: Now that We Found the Host, Let’s Look at the Connections It Is Generating

Note: All Connections Are Embryonic

Connections to Random Destinations on TCP/135– MS Blaster



  Cisco PIX provides two methods to limit the number of connections per host

TCP intercept

Max connections

  TCP intercept won’t help because the source address is valid

  Limiting the maximum number of connections each internal host can have is the only option

Question: Which One Can Be Used Here?



Step 9: Limit Infected Host(s) Impact on Network

  Configure the MAX TCP connections for NATed hosts to be 50

  Note: the local-host must be cleared before the new connection limits are applied

pixfirewall(config)# nat (inside) 1 0.0.0.0 0.0.0.0 50 0

pixfirewall(config)# clear local-host 10.1.1.99

pixfirewall(config)# show local-host 10.1.1.99 Interface inside: 250 active, 250 maximum active, 0 denied local host: <10.1.1.99>, TCP connection count/limit = 50/50 TCP embryonic count = 50 TCP intercept watermark = unlimited UDP connection count/limit = 0/unlimited . . .

The Infected Host Is Limited to 50 TCP Connections



  Things look much better now

  Question: How could we configure the Cisco PIX so the connection limit was only applied to the one host (10.1.1.99) which was infected with the virus?

pixfirewall# show conn count 126 in use, 147456 most used

pixfirewall# show memory Free memory: 47716152 bytes Used memory: 19392712 bytes ------------- ---------------- Total memory: 67108864 bytes

nat (inside) 1 10.1.1.99 255.255.255.255 50 0 nat (inside) 1 0.0.0.0 0.0.0.0 0 0

Take One Last Look at the Memory and Connection Counts After Applying the TCP Connection Limit


Case Study High CPU Usage


High CPU Usage on the Cisco PIX

  A quick overview of the show processes command

pixfirewall(config)# show processes

PC SP STATE Runtime SBASE Stack Process Hsi 001eab19 008a5a74 00557910 0 008a4aec 3628/4096 arp_timer Lsi 001f00bd 00a28dbc 00557910 0 00a27e44 3832/4096 FragDBGC Lwe 00119abf 02d280dc 0055b070 0 02d27274 3688/4096 dbgtrace Lwe 003e4425 02d2a26c 00557dd8 74440 02d28324 6936/8192 Logger Crd 001e26fb 0533940c 00557d88 6070290 05338484 3684/4096 557poll Lsi 00300a29 04c0f504 00557910 0 04c0e57c 3944/4096 xlate clean

The Name of the Process

Number of msec This Process Has Been on the CPU

Problem: Cisco PIX CPU Running Very High

For more Information on the Output of the show processes Command, See http://www.cisco.com/en/US/partner/products/hw/vpndevc/ps2030/products_tech_note09186a008009456c.shtml



Step 1: Determine What Process Is Eating the CPU

  Take the difference in output of two show processes over a period of time

  The following output was a diff of the processes taken one minute apart

Process_Name Runtime (msec) Logger 25940 pix/intf3 18410 557poll 9250 i82543_timer 4180 i82542_timer 2230

In One Minute, These Processes Account for 44 Seconds of CPU Time ~ 73%

The Interface Polling Processes Always Run, and Are not Counted in the CPU Usage


pixfirewall(config)# show log Syslog logging: enabled Standby logging: disabled Console logging: disabled Monitor logging: disabled Buffer logging: level alerts, 0 messages logged Trap logging: level warnings, 5919412 messages logged Logging to lab 172.18.173.123 History logging: disabled

. . . pixfirewall(config)# show log Syslog logging: enabled Buffer logging: level alerts, 0 messages logged Trap logging: level warnings, 6172472 messages logged Logging to lab 172.18.173.123


Step 2: Focus on the Processes with High CPU Time   Logging is taking up much of the CPU; let’s review what we have

configured to log

This Is Cumulative Since the Cisco PIX Was Last Rebooted

Notice the Change Over a Few Minutes


pixfirewall(config)# show log Buffer logging: level warnings, 31527 messages logged Trap logging: level warnings, 6453127 messages logged Logging to lab 172.18.173.123

. . . 400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab 400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab 400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab 400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab 400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab 400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab


  Enable buffered logging to same level as syslog server, and examine the buffered messages

Cisco PIX’s Interface Address

Syslog Server Is Controlled by a Different Group


pixfirewall(config)# show run | grep audit ip audit name IDS info action alarm ip audit interface lab IDS


  Syslog service was down on the syslog server

  ICMP unreachable was generated by syslog server for each syslog message the Cisco PIX sent it

  Cisco PIX’s IDS configuration also logged every ICMP unreachable message, creating the exponentially increasing problem

Syslog Server

Outside

Syslog Message ICMP Unreachable

IDS Syslog Message

Lab

Examine IDS Configuration


ip audit signature 2001 disable or

no logging message 400011


  Bring back up syslog service on server

  Take server offline

  Configure Cisco PIX to not log IDS ICMP unreachable messages

pixfirewall# show run | grep signature ip audit signature 2001 disable

pixfirewall# show cpu usage CPU utilization for 5 seconds = 2%; 1 minute: 50%; 5 minutes: 99%

Solution



  Examine the DIFF of two show processes taken over a one minute interval

  Find the process taking up the highest amount of CPU (excluding the polling processes)

  Take actions to lower that process’s CPU time

  Reexamine the CPU output, and repeat as necessary

Summary


FWSM

  Additional architecture information


FWSM Syslog Level vs. Number of Messages

Log Level Description

Number of Messages (SUM)

Ver. 2.3 Ver. 3.1 Ver. 3.2 Ver. 4.0 Ver. 4.1

0 Emergencies 0 0 0 0 0

1 Alerts 58 (58) 67 (67) 67 (67) 67 (67) 67 (67)

2 Critical 21 (79) 29 (96) 29 (96) 29 (96) 29 (96)

3 Errors 94 (173) 305 (401) 306 (402) 318 (414) 318 (414)

4 Warnings 131 (304) 194 (595) 196 (598) 199 (613) 199 (613)

5 Notifications 26 (330) 167 (762) 169 (767) 178 (791) 178 (791)

6 Informational 116 (446) 245 (1007) 248 (1015) 255 (1046) 259 (1050)

7 Debugging 23 (469) 225 (1232) 225 (1240) 226 (1272) 231 (1281)


FWSM and ACLs

  ACLs on the FWSM are compiled on the control point and pushed down into hardware (NP 3)

  During compile time, CPU should stay at ~ 99% ACL compile uses all free CPU cycles

Allows compile to complete in shortest time possible

  Once compile is complete, rules are attempted to be pushed into hardware

Successful download Access Rules Download Complete: Memory Utilization: 49%

Failed download (exceeded HW memory) ERROR: Unable to add, access-list config limit reached


FWSM and ACLs (Multimode)

  Use show np 3 acl stats to see the current ACL resource utilization in that context

FWSM/admin(config)# show np 3 acl stats ---------------------------- ACL Tree Statistics ---------------------------- Rule count : 9584 Bit nodes (PSCB's): 8760 Leaf nodes : 8761 Total nodes : 17521 (max 24260) Leaf chains : 6912 Total stored rules: 15673 Max rules in leaf : 3 Node depth : 32 ----------------------------

Total Number of ACEs

This Is the Hardware Limit

Note: One ACE Does not Equal One Node



  Use show np 3 acl tree to see which ACL tree a context is mapped to

FWSM# show np 3 acl tree -------------------------------------------- ACL Tree Instance <-> Context Name (ID) Map -------------------------------------------- Tree Instance 0 Context (001) admin Tree Instance 1 Context (002) core Tree Instance 2 Context (003) Engineering Tree Instance 3 Context (004) Accounting --------------------------------------------

ACL Tree Number

Context Name


FWSM—ACL Rule Limits

  FWSM 2.3 introduced resource acl-partition—set the number of ACL partitions allocate-acl-partition—assigns a context to a specific partition

  FWSM 3.2 introduced resource-rule—allows further customization of a partition

  FWSM 4.0 introduced resource partition—customize the size of individual partitions access-list optimization enable—merges and/or deletes redundant and conflicting ACEs without affecting the policy


FWSM(config)# context Accounting FWSM(config-context)# allocate-acl-partition 0 FWSM(config-context)# show np 3 acl tree -------------------------------------------- ACL Tree Instance <-> Context Name (ID) Map -------------------------------------------- Tree Instance 0 Context (001) admin Tree Instance 1 Context (002) core Tree Instance 2 Context (003) Engineering Tree Instance 0 Context (004) Accounting --------------------------------------------


  Use the command resource acl-partition <num-of-partitions> to reduce the number of active partitions created; default is 12

  Use the command allocate-acl-partition <num> to assign a context to a specific ACL tree

Both Use Tree 0


FWSM—Resource Rule

  FWSM 3.2 introduced resource-rule—allows further customization of a partition

show resource-rule—displays information about the current rule allocation

resource rule nat 10000 acl 2200 filter 400 fixup 595 est 70 aaa 555 console 283

FWSM# show resource rule

Default Configured Absolute CLS Rule Limit Limit Max -----------+---------+----------+--------- Policy NAT 1843 1843 10000 ACL 74188 74188 74188 Filter 2764 2764 5528 Fixup 4147 4147 10000 Est Ctl 460 460 460 Est Data 460 460 460 AAA 6451 6451 10000 Console 1843 1843 3686 -----------+---------+----------+--------- Total 92156 92156

Partition Limit - Configured Limit = Available to allocate 92156 - 92156 = 0


FWSM—Resource Partition   FWSM 4.0 introduced

resource partition—allows customization of the size of individual partitions (multi-context mode)

FWSM(config)# resource partition 10 FWSM(config-partition)# size 1000 WARNING: The rule max has been reset based on partition size 1000. The <size> command leads to re-partitioning of ACL Memory. It will not take effect until you save the configuration and reboot.

FWSM# show resource rule partition 10



FWSM# show resource rule partition 10



Before After


FWSM and Control Point   The traffic that makes it to the

control point is traffic that requires Layer 7 fixup (embedded NAT, or cmd inspection)

FTP VoIP (SIP/SKINNY/H.323/RTSP) DNS XDMCP, etc.

  Traffic sourced from, or destined to, the FWSM also goes through the control point

Syslogs AAA (RADIUS/TACACS+) URL filtering (WebSense/N2H2) Management traffic (telnet/SSH/

HTTPS/SNMP) Failover communications Routing protocols (OSPF/ RIP) etc.




Fast Path NP 1

Fast Path NP 2

FWSM


FWSM and Network Processors

  The session manager—NP 3 Processes first packet in a flow ACL checks Translation creation Embryonic/established connection

counts TCP/UDP checksums Sequence number randomization TCP intercept etc.

  The fast path—NP 1 and 2 Performs per packet session lookup Maintains connection table Performs NAT/PAT TCP checks Fragmentation reassembly etc.



Fast Path NP 1

Fast Path NP 2

FWSM




Fast Path NP 1

Fast Path NP 2

FWSM



FWSM—Enabling the Completion Unit

  Due to the FWSM’s NP architecture, there exists a possibility that packets arriving with a low inter-packet gap might be re-ordered by the firewall

  This issue might be encountered when performing TCP throughput testing, or passing high speed TCP flows through the FWSM

  Examples: CIFS, FTP, AFP, backups

  FWSM version 3.1(10) and 3.2(5) introduce a new command sysopt np completion-unit to ensure the firewall maintains the packet order (by enabling a hardware knob on the NPs called the completion unit)

  In multiple mode enter this command in the admin context configuration; It will then be enabled for all contexts on the firewall

4 3 2 1 4 2 3 1

Case Study Advanced Syslog Analysis


Problem – Find Services which are permitted through the firewall, yet the servers no longer exist

  Get a fast Linux/Solaris machine with a decent amount of memory   Learn to use the following commands:

•  cat

•  grep, egrep, fgrep •  cut

•  awk (basic) •  sort

•  uniq

•  Perl (advanced manipulation)   Pipe the commands to construct the necessary outputs!

Case Study: Advanced Syslog Analysis


  Interesting syslogs appear as follows:

May 24 2010 23:19:53: %ASA-6-302014: Teardown TCP connection 1019934 for outside:203.0.113.126/6243 to inside:10.100.19.190/21 duration 0:00:30 bytes 0 SYN Timeout

Syslog ID

Reason

Destination




  grep – used to find the syslogs we want

  awk – used to print the destination column (IP/port)

  uniq – used to print only unique entries, with a count

  sort – used to display ordered list, highest count first

syslogserver-sun% grep 302014 syslog.txt | grep "SYN Timeout" | awk '{print $13}' | uniq -c | sort -r -n

673 inside:10.100.19.190/21 451 dmz:192.168.5.13/80 392 dmz:192.168.5.11/443 358 inside:10.0.0.67/1521 119 inside:10.0.1.142/80

Results:


Case Study FWSM – Slow Single-Flow TCP Throughput

Move this case-study to the appendix


Case Study: FWSM Slow TCP Throughput

Problem

 TCP based backups are taking longer than expected through the FWSM

  iPerf performance testing is only showing ~450 Mbps through FWSM


  Due to the FWSM’s NP architecture, there exists a possibility that packets arriving with a low inter-packet gap might be re-ordered by the FWSM.

  FWSM version 3.1(10) and 3.2(5) introduce a new command sysopt np completion-unit to ensure the firewall maintains the packet order

1 TCP Flow

2 3 4 1 3 2 4

Note: In multi-mode add command to admin context, and it will be applied globally

1 TCP Flow

2 3 4 1 2 3 4

Enable np completion-unit

Case Study: FWSM Slow TCP Throughput

FWSM Only


Failover

  What to Do After a Failover

  Additional Failover Commands


ASA# show failover state

State Last Failure Reason Date/Time This host - Primary Failed Ifc Failure 12:56:00 UTC May 6 2010 Inside: Failed

Other host - Secondary Active None

====Configuration State=== Sync Done ====Communication State=== Mac set


  show failover state – will provide specific details about the failure reason.

  This information is not saved across reboots


ASA# show failover history ========================================================================== From State To State Reason ========================================================================== Disabled Negotiation Set by the CI config cmd Negotiation Just Active No Active unit found Just Active Active Drain No Active unit found Active Drain Active Applying Config No Active unit found Active Applying Config Active Config Applied No Active unit found Active Config Applied Active No Active unit found Active Failed Interface check ==========================================================================


  Starting with FWSM 2.3 and Cisco ASA/PIX 7.0, the reason for failover is saved in the failover history

  This information is not saved across reboots


ASA(config)# prompt hostname priority state ASA/sec/act(config)#

Other Useful Failover Commands

  failover exec mate – allows you to execute commands on the peer and receive the response back.

  failover reload-standby – only valid on Active unit

  prompt – changes the prompt to display failover priority and state.


Failover Prompt Display Configuration

  The firewall’s prompt maybe changed to display certain keyword   Usage

prompt <keyword> [<keyword> ...]

  Syntax keywords: Hostname Configures the prompt to display the hostname Domain Configures the prompt to display the domain Context Configures the prompt to display the current context (multi-mode only) Priority Configures the prompt to display the failover lan unit setting State Configures the prompt to display the current traffic handling state Slot Configures the prompt to display the slot location (when applicable)

  Example FWSM(config)# prompt hostname domain priority state slot

FWSM/cisco.com/sec/actNoFailover/4(config)#


Capture Example


Capture Command: Example

  Problem: user on the inside with an IP of 10.1.3.2 is having a problem accessing www.cisco.com (198.133.219.25); the user is getting PATed to 192.168.2.2

Outside Inside

Capture In Capture Out

Internet

www.cisco.com

198.133.219.25 10.1.3.2 10.1.3.2 192.168.2.2

Step 1: Create ACL for Both Inside and Outside Interface

Step 2: Create Captures on Both Inside and Outside Interface

Step 3: Have Inside User Access www.cisco.com

Step 4: Copy the Captures Off to a TFTP Server

Step 5: Analyze Captures with Sniffer Program


Capture Command: Example

  Step 1: create ACL for both inside and outside interface ! Outside Capture ACL Access-list 100 permit tcp host 192.168.2.2 host 198.133.219.25 eq 80 Access-list 100 permit tcp host 198.133.219.25 eq 80 host 192.168.2.2

! Inside Capture ACL Access-list 101 permit tcp host 10.1.3.2 host 198.133.219.25 eq 80 Access-list 101 permit tcp host 198.133.219.25 eq 80 host 10.1.3.2

  Step 2: create captures on both inside and outside interface capture out access-list 100 interface outside packet-length 1518 capture in access-list 101 interface inside packet-length 1518

  Step 3: have inside user access www.cisco.com   Step 4: copy the captures off to a TFTP server

! ASA ver 7.0+ / FWSM 3.0+ copy capture copy /pcap capture:out tftp://10.1.3.5/out.pcap copy /pcap capture:in tftp://10.1.3.5/in.pcap

! PIX ver 6.x / FWSM 2.3 copy capture copy capture:out tftp://10.1.3.5/out.pcap pcap copy capture:in tftp://10.1.3.5/in.pcap pcap

Or copy using https: https://<FW_IP>/capture/out/pcap


Packet Capture: Example

  Step 5: analyze captures with sniffer program

Outside CAP

Inside CAP Outbound SYN, No SYN+ACK


Packet Capture: Limitations on FWSM

  Capture functionality is available on the FWSM starting in 2.3

However, only packets processed by the control point could be captured

  FWSM 3.1(1) added support to capture packets in hardware

Only ingress packets were captured

  FWSM 3.1(5) both ingress and egress transient packets can be captured which flow through hardware

Capture requires an ACL to be applied

Capture copies the matched packets in hardware to the control point where they are captured; be careful not to flood the control point with too much traffic




Fast Path NP 1

Fast Path NP 1

FWSM


Online Tools

  Networking professionals connection   Bug toolkit   Output Interpreter


Networking Professionals Connection

Online Open Forum to Ask Questions

Anyone Can Ask a Question, and Anyone Can Answer

Regular Ask the Expert Events on Certain Topics


Networking Professionals Connection

http://www.cisco.com/go/netpro

Online Open Forum to Ask Questions

Anyone Can Ask a Question, and Anyone Can Answer

Regular Ask the Expert Events on Certain Topics


Bug Toolkit

On the Support Tools and Resources Page


Bug Toolkit—Product Selection

Select Security, then Cisco ASA

5500 Series


Bug Toolkit—Advanced Search

Version

Search Keywords

Severity

Status


Bug Toolkit—Search Results

Select Link to View Details of Bug


Bug Toolkit—Bug Details

First Fixed-In Releases


Output Interpreter

Great Tool for Catching Configuration Errors

Linked off the Technical Support and Documentation— Tools and Resources Section on CCO

Paste in the show run Output and Hit submit


Output Interpreter: Example Output

Warning: Unapplied Crypto Map

Warning: Invalid Crypto Map

https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl

Warning: Unused Statics


ASDM


ASDM

  Run as a standalone application using the ADSM Launcher   This allows for one-stop access to multiple firewalls   ASDM 6.0 adds Upgrade Wizard to upgrade ASA and ASDM

software direct from cisco.com   ASDM 6.2 works with ASA 8.2, ASA 8.1 and 8.0 releases   ASDM 6.1F works with FWSM 4.0, 3.2, and 3.1 releases


ASDM Home Page

CPU, Memory, Conns/Sec,

Interface Traffic

Real-Time Syslogs

Device Information


Using ASDM for Monitoring Great for Monitoring

Trends

Up to Four Different Graphs

Can Be Displayed


ASDM: Editing Rules from the Log Viewer

Select Log Entry from Viewer

Right-Click on Message to View or Edit

Associated Rule


ASDM: Syslogs Explained


Opening a TAC Case

  If after using all your troubleshooting tools you still cannot resolve the problem, please open a TAC case

http://www.cisco.com/techsupport/servicerequest/

  At a minimum include: Detailed problem description

Output from show tech

  Optionally include: Syslogs captured during time of problem

Sniffer traces from both interfaces using the capture command (capturing only the relevant packets, and saved in pcap format)

BRKSEC-3020 Advanced Firewalls

Documents

Transcript of BRKSEC-3020 Advanced Firewalls