Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in...
Transcript of Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in...
BRKSEC-3020
Advanced Firewalls
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 2
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 3
Agenda
Packet Flow
Understanding the Architecture
Failover
Troubleshooting
Case Studies
Best Practices
Packet Flow
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 5
Accounting
Example Flow Flow
SRC IP: 10.1.1.9 SRC Port: 11030 Protocol: TCP
DST IP: 198.133.219.25 DST Port: 80
Interfaces
Source: Inside Destination: Outside
With the flow defined, examination of configuration boils down to just two interfaces: Inside and Outside
Client: 10.1.1.9
Server: 198.133.219.25S
erv
ers
Ou
tsid
e
Packet Flow
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 6
Packet Processing Flow Diagram
The diagram below will be referenced on the following slides; it is shown here enlarged for reference
For your
reference
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 7
Packet Processing: Ingress Interface
Packet arrives on ingress interface
Input counters incremented
Software input queue is an indicator of load
No buffers indicates packet drops, typically due to bursty traffic
ASA-5540# show interface gb-ethernet1
interface gb-ethernet1 "inside" is up, line protocol is up
Hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.6214
IP address 10.1.1.1, subnet mask 255.255.255.0
MTU 1500 bytes, BW 1 Gbit full duplex
5912749 packets input, 377701207 bytes, 0 no buffer
Received 29519 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
286298 packets output, 18326033 bytes, 0 underruns
input queue (curr/max blocks): hardware (0/25) software (0/0)
output queue (curr/max blocks): hardware (0/3) software (0/0)
Ingress
Interface
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 8
Packet Processing: Locate Connection
Check first for existing connection
If connection exists, flow is matched; bypass ACL check
If no existing connection
TCP non-SYN packet, drop and log
TCP SYN or UDP packet, pass to ACL checks
Established Connection:
ASA-5540# show conn
TCP out 198.133.219.25:80 in 10.1.1.9:11030 idle 0:00:04 Bytes 1293 flags UIO
Syslog Because of No Connection, and Non-SYN Packet:
ASA-6-106015: Deny TCP (no connection) from 10.1.1.9/11031 to 198.133.219.25/80 flags PSH
ACK on interface inside
Existing
Conn
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 9
Packet Processing: ACL Check
First packet in flow is processed through interface ACLs
ACLs are first match
First packet in flow matches ACE, incrementing hit count by one
Denied packets are dropped and logged
Packet Permitted by ACL:
ASA-5540B# show access-list inside
access-list inside line 10 permit ip 10.1.1.0 255.255.255.0 any (hitcnt=1)
Syslog When Packet Is Denied by ACL:
ASA-4-106023: Deny tcp src inside:10.1.1.9/11034 dst outside:198.133.219.25/80 by
access-group "inside"
ACL
Permit
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 10
Packet Processing: Match Translation
First packet in flow must match a translation rule*
A quick route lookup is done only to determine egress interface
Translation rule can be to NAT, or not to NAT
NAT order of operations dictates what happens with overlapping translation rules (See Appendix)
Once translation rule is matched, connection is created
Translation Exists:ASA-5540# show xlate debug
NAT from inside:10.1.1.9 to outside:172.18.124.68 flags - idle 0:00:07 timeout 3:00:00
Syslogs When No Translation Rule Found: (305005—No NAT; 305006—No Global)ASA-3-305005: No translation group found for tcp src inside:10.1.1.9/11039 dst
outside:198.133.219.25/80
ASA-3-305006: regular translation creation failed for tcp src inside:10.1.1.9/11040 dst
outside:198.133.219.25/80
Match
xLate
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 11
Packet Processing: Inspections/Sec Checks
Inspections are applied to ensure protocol compliance
(Optional) customized AIC inspections
NAT-embedded IPs in payload
Additional security checks are applied to the packet
(Optional) packets passed to Content Security and Control (CSC) module
Syslog from Packets Denied by Security Check:
ASA-4-406002: FTP port command different address: 10.2.252.21(192.168.1.21) to
209.165.202.130 on interface inside
ASA-4-405104: H225 message received from outside_address/outside_port to
inside_address/inside_port before SETUP
Inspections
Sec Checks
Question!What command will show you if packets are being dropped
by one of the Inspection engines?
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 12
Packet Processing: NAT IP Header
Translate the IP address in the IP header
Translate the port if performing PAT
Update checksums
(Optional) Following the above, pass packet to IPS (AIP) module
Nat IP
Header
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 13
Packet Processing: Egress Interface
Packet is virtually forwarded to egress interface (i.e., not forwarded to the driver yet)
Egress interface is determined first by translation rules
If translation rules do not specify egress interface (e.g., outbound initial packet) the results of a global route lookup are used to determine egress interface
Example:
static (inside, outside) 192.168.0.0 172.16.0.0 netmask 255.255.0.0
static (dmz, outside) 192.168.12.0 172.16.12.0 netmask 255.255.255.0
DM
Z
Inside Outside
172.16.0.0/16 172.16.12.0/24
172.16.12.4
Inbound packets to 192.168.12.4 get routed to Inside based on order of statics
Egress
Interface
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 14
Packet Processing: L3 Route Lookup
Once on egress interface, an interface route lookup is performed
Only routes pointing out the egress interface are eligible
Remember: translation rule can forward the packet to the egress interface, even though the routing table may point to a different interface
Syslog from Packet on Egress Interface with No Route Pointing Out Interface:
ASA-6-110001: No route to 209.165.202.130 from 10.1.1.9
L3
Route
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 15
Packet Processing: L2 Address Lookup
Once a Layer 3 route has been found, and next hop identified, Layer 2 resolution is performed
Layer 2 rewrite of MAC header
If Layer 2 resolution fails—no syslog
show arp will not display an entry for the L3 next hop
debug arp will indicate if we are not receiving an
ARP reply
L2
Addr
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 16
Packet Processing: Transmit Packet
Packet is transmitted on wire
Interface counters will increment on interface
Output hardware and software queues indicate buffering at driver level, interface is busy
ASA-5540# show interface gb-ethernet0
interface gb-ethernet0 "outside" is up, line protocol is up
Hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.626c
IP address 172.18.124.64, subnet mask 255.255.255.0
MTU 1500 bytes, BW 1 Gbit full duplex
3529518 packets input, 337798466 bytes, 0 no buffer
Received 32277 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
5585431 packets output, 359059032 bytes, 0 underruns
input queue (curr/max blocks): hardware (0/25) software (0/0)
output queue (curr/max blocks): hardware (0/2) software (0/0)
Xmit
Pkt
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 17
Agenda
Packet Flow
Understanding the Architecture
Failover
Troubleshooting
Case Studies
Best Practices
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 18
Cisco ASA —Understanding the Architecture
ASA processes all packets in software (via the central CPU)
All packets are processed first in… usually also first out
ASA platforms have software imposed connection limits
Multi-CPU / Multi-Core systems hash packets in the same flow to the same CPU/core.
10 Gig interfaces hash flow to same RX ring.
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 19
Maximum ACL Limits ASA has no hard-coded limit on the number of elements (ACEs) in an ACL. Bound
only by Memory.
Each ACE uses a minimum of 212 bytes of RAM
FWSM has a hard-coded limit based on memory of NP 3.
5505 5510 5520 5540 5550 5580 FWSM
Max Recommended
ACEs25k 80k 200k 500k 700k 750k 100k
Max Observed (from
customers)2.74 million 2.77 million 100k
Note: Issue show access-list | include elements to see how many ACEs you have
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 20
Global ACLs
Global ACLs were introduced in ASA version 8.3
Best used for new installations, or migration from other vendors
Interface Independent Policies
access-group <access_list> global
Global access-list
Interface Specific access-list
Default (implicit) deny ip any any
Policy Ordering
ASA
Only
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 21
FWSM—Understanding the Architecture
Packets processed in hardware have zero impact on CPU
Similarly, if the CPU is pegged at 100%, this has zero impact on packets processed in hardware
FWSM Process Most Packets in Hardware, with Some Packets Needing to be Processed in Software—via the Control Point (CP)
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 22
Fast Path
Flow Identification,
Security Checks and
NAT in Hardware
FWSM Architectural Overview
C6K Backplane Interface
Session Manager
NP 3
Control Point (CP)
Central CPU
Fast Path
NP 1
Software
Hardware
FW
SM
Control Point
ACL Compilation,Fixups,
Syslog, AAA, IPv6 in
Software
Session Manager
Session Establishment and
Teardown, AAA Cache,
ACLs
Fast Path
NP 2
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 23
FWSM—Hardware Limits FWSM has several hardware limits that should be considered in
your network design
Limits are hard set, but vary based on single or multimode
Some limits include:
2.3 (Multimode) 3.1/3.2 (Multimode) 4.0/4.1 (Multimode)3.2 / 4.0 /4.1 Configurable
ACEs 56,627 (9,704) 72,806 (11,200) 100,567 (14,801) X
AAA Rules 3,942 (606) 6,451 (992) 8,744 (1,345) X
Global Statements 1K (1K) 4K (4K) 4K (4K)
Static NAT Statements 2K (2K) 2K (2K) 2K (2K)
Policy NAT ACEs 3,942 (606) 1,843 (283) 2,498 (384) X
NAT Translations 256K (256K) 256K (256K) 256K (256K)
Connections 999,990 (999,990) 999,990 (999,990) 999,990 (999,990)
Route Table Entries 32K (32K) 32K (32K) 32K (32K)
Fixup/Inspect Rules 32 (32 per) 4147 (1,417) 5621 (1,537) X
Filter Statements 3942 (606) 2764 (425) 3747 (576) X
Increase over 2.3 Increase over 3.1
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 24
Agenda
Packet Flow
Understanding the Architecture
Failover
Troubleshooting
Case Studies
Best Practices
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 25
Failover Basics Active/Standby vs. Primary/Secondary
Serial vs. LAN failover
Stateful failover (optional)
A failover only occurs when either firewall determines the standby firewall is healthier than the active firewall
Both firewalls swap MAC and IP addresses when a failover occurs
Level 1 syslogs will give reason of failover
Secondary
(Standby)Primary
(Active)
LAN/Serial
Stateful
Internet
Corp
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 26
ASA# show failover
Failover On
Failover unit Primary
Failover LAN Interface: failover Redundant5 (up)
Unit Poll frequency 200 milliseconds, holdtime 1 seconds
Interface Poll frequency 500 milliseconds, holdtime 5 seconds
Interface Policy 1
Monitored Interfaces 2 of 250 maximum
Version: Ours 8.2(2), Mate 8.2(1)
Last Failover at: 10:37:11 UTC May 14 2010
This host: Primary - Active
Active time: 1366024 (sec)
slot 0: ASA5580 hw/sw rev (1.0/8.1(2)) status (Up Sys)
Interface outside (10.8.20.241): Normal
Interface inside (10.89.8.29): Normal
Other host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5580 hw/sw rev (1.0/8.1(2)24) status (Up Sys)
Interface outside (10.8.20.242): Normal
Interface inside (10.89.8.30): Normal
Stateful Failover Logical Update Statistics
Link : stateful Redundant6 (up)
Stateful Obj xmit xerr rcv rerr
General 424525 0 424688 0
sys cmd 423182 0 423182 0
Verifying Failover Configuration
Interface
Monitoring
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 27
What Triggers a Failover?
Power loss/reload (this includes crashes) on the Active firewall
SSM interface/module failure
The Standby becoming healthier than the Active firewall
In LAN based Failover, what happens if the LAN interface
communication is severed?
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 28
What Triggers a Failover? (Con’t)
Two consecutive hello messages missed on any monitored interface forces the interface into testing mode
Both units first verify the link status on the interface
Next, both units execute the following tests
Network activity test
ARP test
Broadcast ping test
The first test passed causes the interface on that unit to be marked healthy; only if all tests fail will the interface be marked failed
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 29
What to Do After a Failover
Always check the syslogs to determine root cause
Example: switch port failed on inside interface of active firewall
ASA-4-411002: Line protocol on Interface inside, changed state to down
ASA-1-105007: (Primary) Link status ‘Down’ on interface 1
ASA-1-104002: (Primary) Switching to STNDBY—interface check, mate is healthier
ASA-1-104001: (Secondary) Switching to ACTIVE—mate want me Active
Syslogs from Primary (Active) Firewall
Syslogs from Secondary (Standby) Firewall
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 30
Agenda
Packet Flow
Understanding the Architecture
Failover
Troubleshooting
Case Studies
Best Practices
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 31
Troubleshooting Tools
Syslogs
Debug commands
Show commands
Packet capture
Packet tracer
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 32
Buffered
Uses of Syslogs
Primary mechanism to record traffic to and throughthe firewall
The best troubleshooting tool available
SSH Client
Internet
Archival Purposes Debugging Purposes
Syslog Server
SNMP Server
Console
Trap .Syslog
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 33
Logging – Common Issues
logging flash-bufferwrapshould only be used when logging to buffer at Level 1
logging history should only be used when you really have an SNMP server that you want to receive all syslogs
logging consoleshould only be enabled while actively troubleshooting on the Console
logging standbyshould only be used if you want to receive double the syslogs
logging permit-hostdownshould always be used with TCP syslogging
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 34
Debug Commands
1. Debugs should not be the first choice to troubleshoot a problem
2. Debugs can negatively impact the CPU of the box, and also the performance of it; use with caution
3. Debugs are not conditional*
4. Know how much traffic, of the specified type, is passing through the firewall before enabling the respective debug
* Crypto Conditional Debugging Was Added to Cisco ASA/PIX 8.0
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 35
Debug ICMP Trace
Valuable tool used to troubleshoot connectivity issues
Provides interface and translation information to quickly determine flow
Echo-replies must be explicitly permitted through ACL, or ICMP inspection must be enabled
ICMP echo-request from inside:10.1.1.2 to 198.133.219.25 ID=3239 seq=4369 length=80
ICMP echo-request: translating inside:10.1.1.2 to outside:209.165.201.22
ICMP echo-reply from outside:198.133.219.25 to 209.165.201.22 ID=3239 seq=4369 length=80
ICMP echo-reply: untranslating outside:209.165.201.22 to inside:10.1.1.2
Example debug icmp trace output
http://www.cisco.com
Internet
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 36
Redirecting Debugs to Syslog Problem
Log only debug output to syslog
Solution
Create a logging list with only syslog ID 711001
Enable debug output to syslogs
Log on the logging list
ASA(config)# logging trap CiscoLive .
ASA(config)# logging list CiscoLive message 711001 .
ASA(config)# logging debug-trace
INFO: 'logging debug-trace' is enabled. All debug messages are currently
being redirected to syslog:711001 and will not appear in any monitor
session
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 37
Show Output Filters
Use output filters to filter the output of show command to only the information you want to see
To use them, at the end of show <Command>, use the pipe character ―|‖ followed by
begin Start displaying the output beginning at the first match of the
RegEx, and continue to display the remaining output
include Display any line that matches the RegEx
exclude Display any line that does not match the RegEx
grep Same as include
grep –v Same as exclude
show <cmd> | begin|include|exclude|grep [-v] <regular_exp>
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 38
Show CPU Usage
Under normal conditions the CPU should stay below 50% (baseline as per network); if the CPU reaches 100% the firewall will start dropping packets
FWSM CPU is used for limited traffic processing; during ACL compilation CPU is expected to be near 100% until ACL is compiled
The show cpu usage command displays the CPU over time as a running average
ASA# show cpu usage
CPU utilization for 5 seconds = 5%; 1 minute: 4%; 5 minutes: 4%
*First Introduced in Cisco PIX OS Version 6.0(1)/FWSM 1.1(1)
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 39
Show Processes cpu-usage
The show processes cpu-usage command displays the amount of CPU used on a per-process basis for the last 5sec, 1min, and 5min
ASA# # show processes cpu-usage
PC Thread 5Sec 1Min 5Min Process
081aa124 d51ab230 0.2% 2.0% 2.0% Dispatch Unit
08070416 d51aa660 0.0% 0.0% 0.0% aaa
081a954c d51a96a0 0.0% 0.0% 0.0% dbgtrace
08c2a91d d51a7f00 0.0% 0.0% 0.0% netfs_thread_init
0924fe95 d51a7528 0.0% 0.0% 0.0% Chunk Manager
088a6e14 d51a7138 0.0% 0.0% 0.0% IP Address Assign
08a6d7f6 d51a6f40 0.0% 0.0% 0.0% QoS Support Module
08bcf736 d51a53b0 0.0% 0.0% 0.0% Logger
08685627 d51a3a18 0.0% 0.0% 0.0% netfs_mount_handler
0851ca68 d51a3820 0.0% 0.0% 0.0% arp_timer
08b9ffab d5198ae0 0.0% 0.0% 0.0% ssh/timer
08b99aec d5195d98 3.9% 0.5% 0.1% ssh
...
*First Introduced in Cisco ASA Version 7.2(4.11), 8.0(4.5), 8.1(1.100), 8.2(1).
Currently not Available in FWSM
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 40
Show Processes cpu-hog The show processes cpu-hog command displays a list of processes, and the
function stack (Traceback) which executed, and lead to a process running on the CPU longer than the minimum platform threshold
ASA# show processes cpu-hog
Process: ssh_init, NUMHOG: 18, MAXHOG: 15, LASTHOG: 10
LASTHOG At: 14:18:47 EDT May 29 2009
PC: 8b9ac8c (suspend)
Traceback: 8b9ac8c 8ba77ed 8ba573e 8ba58e8 8ba6971
8ba02b4 8062413
CPU hog threshold (msec): 10.240
Last cleared: None
*First introduced in Cisco ASA Version 7.0(1). Currently not Available in FWSM
May 29 2009 14:18:47: %ASA-7-711002: Task ran for 10 msec,
Process = ssh_init, PC = 8b9ac8c, Traceback = 0x08B9AC8C 0x08BA77ED
0x08BA573E 0x08BA58E8 0x08BA6971 0x08BA02B4 0x08062413
A corresponding syslog message is also generatedNote: The Traceback does not signify a crash
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 41
Show Traffic
The show traffic command displays the traffic received
and transmitted out each interface of the firewall
ASA# show traffic
outside:
received (in 124.650 secs):
295468 packets 167218253 bytes
2370 pkts/sec 1341502 bytes/sec
transmitted (in 124.650 secs):
260901 packets 120467981 bytes
2093 pkts/sec 966449 bytes/sec
inside:
received (in 124.650 secs):
261478 packets 120145678 bytes
2097 pkts/sec 963864 bytes/sec
transmitted (in 124.650 secs):
294649 packets 167380042 bytes
2363 pkts/sec 1342800 bytes/sec
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 42
show np blocks (FWSM Only)
The show np blocks command is used to see if the FWSM is over subscribed
FWSM# show np blocks
MAX FREE THRESH_0 THRESH_1 THRESH_2
NP1 (ingress) 32768 32768 0 0 550
(egress) 521206 521206 0 0 0
NP2 (ingress) 32768 32768 0 0 92
(egress) 521206 521206 0 0 0
NP3 (ingress) 32768 32768 13 460417 4427509
(egress) 521206 521206 0 0 0
Warning
Data
packets
dropped
Data and
Control packets
dropped
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 43
Show Xlate and Show Xlate Debug The show xlate command displays information about the
translations through the firewall
You can limit the output to just the local or global IP
ASA# show xlate debug
2 in use, 2381 most used
Flags: D - DNS, d - dump, I - identity, i - inside, n - no random,
o - outside, r - portmap, s - static
NAT from inside:10.1.1.9 to outside:172.18.124.68
flags - idle 0:02:03 timeout 3:00:00
TCP PAT from inside:10.9.9.3/4101 to outside:172.18.124.65/1024
flags r idle 0:00:08 timeout 0:00:30
ASA# show xlate
2 in use, 2381 most used
Global 172.18.124.68 Local 10.1.1.9
PAT Global 172.18.124.65(1024) Local 10.9.9.3(4101)
debug Adds interface Names,
Idle, and xlate Timeouts
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 44
ASA# show conn
2 in use, 64511 most used
TCP outside 198.133.219.25:80 dmz 10.9.9.3:4101, idle 0:00:06, Bytes 127, flags UIO
UDP outside 172.18.124.1:123 dmz 10.1.1.9:123 idle 0:00:13 flags –
Show Conn
Idle Time,
Bytes TransferredConnection Flags
Real Interface
Names Added in
7.2(4), 8.0(4)
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 45
Show Conn Detail
ASA# show conn detail
2 in use, 64511 most used
Flags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN,
B - initial SYN from outside, C - CTIQBE media, D - DNS, d - dump,
E - outside back connection, F - outside FIN, f - inside FIN,
G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data,
i - incomplete, J - GTP, j - GTP data, K - GTP t3-response
k - Skinny media, M - SMTP data, m - SIP media, n - GUP
O - outbound data, P - inside back connection, q - SQL*Net data,
R - outside acknowledged FIN,
R - UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN,
s - awaiting outside SYN, T - SIP, t - SIP transient, U - up, W - WAAS,
X - inspected by service module
TCP outside:198.133.219.25/80 dmz:10.9.9.3/4101,
flags UIO, idle 8s, uptime 10s, timeout 1h, bytes 127
UDP outside:172.18.124.1/123 dmz:10.1.1.9/123,
flags -, idle 15s, uptime 16s, timeout 2m, bytes 1431
detail Adds Uptime and
Timeout in 7.2(4), 8.0(4)
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 46
show local-host A local-host entry is created for any IP tracked through the firewall
It groups the xlates, connections, and AAA information
Very useful for seeing the connections terminating on servers
ASA# show local-host
Interface inside: 1131 active, 2042 maximum active, 0 denied
local host: <10.1.1.9>,
TCP connection count/limit = 1/unlimited
TCP embryonic count = 0
TCP intercept watermark = 50
UDP connection count/limit = 0/unlimited
AAA:
user 'cisco' at 10.1.1.9, authenticated (idle for 00:00:10)
absolute timeout: 0:05:00
inactivity timeout: 0:00:00
Xlate(s):
Global 172.18.124.69 Local 10.1.1.9
Conn(s):
TCP out 198.133.219.25:80 in 10.1.1.9:11055 idle 0:00:10 Bytes 127 flags UIO
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 47
show service-policy The show service-policy command is used to quickly see
what inspection policies are applied and the packets matching them
ASA# show service-policy
Global policy:
Service-policy: global_policy
Class-map: inspection_default
Inspect: dns maximum-length 512, packet 92, drop 0, reset-drop 0
Inspect: ftp, packet 43, drop 0, reset-drop 0
Inspect: h323 h225, packet 0, drop 0, reset-drop 0
Inspect: h323 ras, packet 0, drop 0, reset-drop 0
Inspect: http, packet 562, drop 0, reset-drop 0
Inspect: netbios, packet 0, drop 0, reset-drop 0
Inspect: rsh, packet 0, drop 0, reset-drop 0
Inspect: rtsp, packet 0, drop 0, reset-drop 0
Inspect: skinny, packet 349, drop 0, reset-drop 0
Inspect: esmtp, packet 0, drop 0, reset-drop 0
...
Interface outside:
Service-policy: VoIP
Class-map: voice_marked
Priority:
Interface outside: aggregate drop 0, aggregate transmit 349
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 48
show asp drop Packets dropped in the Accelerated Security Path (ASP) will increment a counter
FWSM – applies only to traffic sent to the control-point
Frame drop counters are per packet, flow drops are per flow
Some counters have corresponding syslogs
ASA# show asp drop
Frame drop:
Invalid encapsulation (invalid-encap) 10897
Invalid tcp length (invalid-tcp-hdr-length) 9382
Invalid udp length (invalid-udp-length) 10
No valid adjacency (no-adjacency) 5594
No route to host (no-route) 1009
Reverse-path verify failed (rpf-violated) 15
Flow is denied by access rule (acl-drop) 25247101
First TCP packet not SYN (tcp-not-syn) 36888
Bad TCP flags (bad-tcp-flags) 67148
TCP option list invalid (tcp-bad-option-list) 731
TCP MSS was too large (tcp-mss-exceeded) 10942
Bad TCP Checksum (bad-tcp-cksum) 893
*Drop Counters Are Documented in the CMD Ref, Under show asp drop
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 49
Packet Capture
Capture command first introduced in Cisco 7.0; FWSM need to use 3.1.5 or later
ASA 7.2(3) and 8.0(3) added a real-time option
ASDM 6.0 adds a capture wizard
Capture sniffs packets on an interface that match an ACL, or match line
Key steps
Create an ACL that will match interesting traffic
Define the capture and bind it to an access-list and interface
View the capture on the firewall, or copy it off in .pcap format
capture <capture-name> [access-list <acl-name>] [buffer <buf-size>] [ethernet-type
<type>] [interface <if-name>] [packet-length <bytes>]
[circular-buffer] [type raw-data|asp-drop|isakmp|webvpn user <username>]
[match <prot> {host <sip> | <sip> <mask> | any}
[eq | lt |gt <port>] {host <dip> | <dip> <mask> | any}
[eq | lt | gt <port>]]
[real-time [dump] [detail] [trace]] [trace [detail] [trace-count <1-1000>]]
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 50
Packet Capture (Cont.)
Traffic can be captured both before and after it passes through the firewall
• Ingress packets are captured before any packet processing has been done on them
• Egress packets are captured after all processing (excluding L2 source MAC rewrite)
Capture buffer saved in RAM (default size 512 KB)
Copy captures off via TFTP or HTTPS
Capture all packets dropped by the ASP
ASA# capture drops type asp-drop all
Inside
Capture In Capture OutInside Capture Outside Capture
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 51
Packet Tracer: Overview
Introduced in ASA ver 7.2
A packet tagged with the trace option is injected into the interface, and processed in the data-plane
Each action taken on the packet is recorded in the packet itself
When the packet reaches the egress interface, or is dropped, it is punted to the control-plane
The control-plane reads and displays the actions taken on the packet, along with the associated lines in the configuration
ASA
Only
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 52
Link Back
to Edit RuleMatching
Config
Define
Packet
Action
Final Result
Packet Tracer: ASDM (Located off Tools Menu)
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 53
Packet Tracer: CLIASA# packet-tracer input inside tcp 10.1.1.2 1024 198.133.219.25 80
Phase: 1
Type: FLOW-LOOKUP
Subtype:
Result: ALLOW
Config:
Additional Information:
Found no matching flow, creating a new flow
...
Phase: 15
Type: ROUTE-LOOKUP
Subtype: output and adjacency
Result: ALLOW
Config:
Additional Information:
found next-hop 209.165.201.1 using egress ifc outside
adjacency Active
next-hop mac address 000a.f331.83c0 hits 0
>>>>Packet successfully forwarded to fast path<<<<
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 54
Packet Tracer: Tracing Captured Packet
Create a capture using the trace option
Find the packet in the capture you want traced
Then select that packet to be traced
ASA# show capture inside trace packet-number 4 .
ASA# capture inside access-list web interface inside trace .
ASA# show capture inside
68 packets captured
1: 15:22:47.581116 10.1.1.2.31746 > 198.133.219.25.80: S
2: 15:22:47.583465 198.133.219.25.80 > 10.1.1.2.31746: S ack
3: 15:22:47.585052 10.1.1.2.31746 > 198.133.219.25.80: . ack
4: 15:22:49.223728 10.1.1.2.31746 > 198.133.219.25.80: P ack
5: 15:22:49.223758 198.133.219.25.80 > 10.1.1.2.31746: . Ack
...
Important!
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 55
Agenda
Packet Flow
Understanding the Architecture
Failover
Troubleshooting
Case Studies
Best Practices
56
Case StudyFWSM – Slow Single-Flow TCP Throughput
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 57
Case Study: FWSM Slow TCP Throughput
Problem
TCP based backups are taking longer than expected through the FWSM
iPerf performance testing is only showing ~450 Mbps through FWSM
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 58
Due to the FWSM‘s NP architecture, there exists a possibility that packets arriving with a low inter-packet gap might be re-ordered by the FWSM.
FWSM version 3.1(10) and 3.2(5) introduce a new command sysopt np completion-unit to ensure the firewall maintains the packet order
1
TCP Flow
234 1324
Note: In multi-mode add command to admin context, and it will be applied globally
1
TCP Flow
234 1234
Enable np
completion-unit
Case Study: FWSM Slow TCP Throughput
FWSM
Only
59
Case StudyIntermittent Access to Web Server
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 60
Clients
Problem
Most external clients are not able to load company‘s web page
10.1.1.50ASA-5510
HTTP Requests to 192.168.1.50
Web Server
NATed to 10.1.1.50
Internet
Case Study: Intermittent Access to Web Server
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 61
Traffic Spike
Case Study: Intermittent Access to Web Server
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 62
show perfmon indicates high number of embryonic connectionsASA-5510# show perfmon
PERFMON STATS: Current Average
Xlates 0/s 0/s
Connections 2059/s 299/s
TCP Conns 2059/s 299/s
UDP Conns 0/s 0/s
URL Access 0/s 0/s
URL Server Req 0/s 0/s
TCP Fixup 0/s 0/s
TCP Intercept Established Conns 0/s 0/s
TCP Intercept Attempts 0/s 0/s
TCP Embryonic Conns Timeout 1092/s 4/s
HTTP Fixup 0/s 0/s
FTP Fixup 0/s 0/s
AAA Authen 0/s 0/s
AAA Author 0/s 0/s
AAA Account 0/s 0/s
VALID CONNS RATE in TCP INTERCEPT: Current Average
N/A 95.00%
Case Study: Intermittent Access to Web Server
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 63
Issue show conn to see ‗who‘ is creating the connections
ASA-5510# show conn
54764 in use, 54764 most used
TCP outside 17.24.101.118:26093 inside 10.1.1.50:80, idle 0:00:23, bytes 0, flags aB
TCP outside 111.76.36.109:23598 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB
TCP outside 24.185.110.202:32729 inside 10.1.1.50:80, idle 0:00:25, bytes 0, flags aB
TCP outside 130.203.2.204:56481 inside 10.1.1.50:80, idle 0:00:29, bytes 0, flags aB
TCP outside 39.142.106.205:18073 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB
TCP outside 75.27.223.63:51503 inside 10.1.1.50:80, idle 0:00:03, bytes 0, flags aB
TCP outside 121.226.213.239:18315 inside 10.1.1.50:80, idle 0:00:04, bytes 0, flags aB
TCP outside 66.187.75.192:23112 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB
TCP outside 13.50.2.216:3496 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB
TCP outside 99.92.72.60:47733 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aB
TCP outside 30.34.246.202:20773 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB
TCP outside 95.108.110.131:26224 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB
TCP outside 76.181.105.229:21247 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB
TCP outside 82.210.233.230:44115 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB
TCP outside 134.195.170.77:28138 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB
TCP outside 70.133.128.41:22257 inside 10.1.1.50:80, idle 0:00:15, bytes 0, flags aB
TCP outside 124.82.133.172:27391 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aB
TCP outside 26.147.236.181:37784 inside 10.1.1.50:80, idle 0:00:07, bytes 0, flags aB
TCP outside 98.137.7.39:20591 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB
TCP outside 37.27.115.122:24542 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB
. . .
Random Sources Embryonic Conns
Case Study: Intermittent Access to Web Server
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 64
Case Study: Intermittent Access to Web Server
Connection Count
Jumps
Traffic Permitted
SYN Flood
Detected
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 65
Apply TCP Intercept to stop the SYN flood attack
access-list 140 extended permit tcp any host 192.168.1.50 eq www
!
class-map protect
description Protect web server from attacks
match access-list 140
!
policy-map interface_policy
class protect
set connection embryonic-conn-max 100
!
service-policy interface_policy interface outside
Case Study: Intermittent Access to Web Server
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 66
TCP Intercept AppliedFew Clients Represent 50+
% of Traffic
Case Study: Intermittent Access to Web Server
Why did the Connection count drop after TCP Intercept
was applied?
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 67
access-list 140 extended permit tcp any host 192.168.1.50 eq www
!
class-map protect
description Protect web server from attacks
match access-list 140
!
policy-map interface_policy
class protect
set connection embryonic-conn-max 100 per-client-max 25
!
service-policy interface_policy interface outside
Apply per-client-max option to limit the number of connections any single client can establish
Case Study: Intermittent Access to Web Server
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 68
TCP Intercept
per-client-max
Case Study: Intermittent Access to Web Server
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 69
Attacks Still Occurring
Attacks Being
Mitigated
Case Study: Intermittent Access to Web Server
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 70
Agenda
Packet Flow
Understanding the Architecture
Failover
Troubleshooting
Case Studies
Best Practices
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 71
Cisco ASA/FWSM Best Practices
Enable ip verify reverse-path on all interfaces
Set embryonic and maximum connection counts on static and natstatements; for 7.2.1+ use per-client-max
Configure logging to syslog server
Move messages you want to see to lower levels, instead of raising logging levels and capturing messages you don‘t want to see
Disable telnet access! Use SSH for management access
Enable authentication for management access (console/SSH/telnet/enable); use TACACS+ or RADIUS with LOCAL as the fallback
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 72
Cisco ASA/FWSM Best Practices
Restrict DMZ access inbound to your internal networks
Baseline CPU load, connection counts, xlate counts, and traffic (per interface)
Monitor stats using MRTG or other snmp graphing tools.
Keep config archives (and show tech ouputs) (Rancid)
Run the latest maintenance release in your train
Upgrade major feature trains only when you need new features, or after train has matured
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 73
Agenda
Packet Flow
Understanding the Architecture
Failover
Troubleshooting
Case Studies
Best Practices
Online Resources
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 75
Online Learning Modules – VoD Training
Great way to learn about new features in the ASA
From www.cisco.com select: Products and Services
Security
Network Security (expand)
Cisco ASA 5500 Series
Training resources
Online learning modules
Search cisco.com for ASA Online Learning Modules
Direct linkhttp://www.cisco.com/en/US/partner/products/ps6120/tsd_products_support_online_learning_modules_list.html
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 76
Supportforums.cisco.com
Public wiki – anyone can author articles
Combines supportwiki and Netpro forums
Sections for: ASA, FWSM and PIX
Hundreds of Sample Configs
Troubleshooting Docs
FAQshttp://supportforums.cisco.com/
Q & A
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 78
Complete Your Online Session Evaluation
Complete your session evaluation:
Directly from your mobile device by visiting www.ciscoliveaustralia.com/mobile and login by entering your badge ID (located on the front of your badge)
Visit one of the Cisco Live internet stations located throughout the venue
Open a browser on your own computer to access the Cisco Live onsite portal
80
Appendix
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 81
Translation and NAT Order of Operations
1. nat 0 access-list (nat-exempt)
2. Match existing xlates
3. Match static commands (Cisco ASA/PIX first match; FWSM best match)
Static NAT with and without access-list
Static PAT with and without access-list
4. Match nat commandsnat <id> access-list (first match)
nat <id> <address> <mask> (best match)
If the ID is 0, create an identity xlate
Use global pool for dynamic NAT
Use global pool for dynamic PAT
Firs
t Ma
tch
Match
xLate
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 82
Failover – Zero Downtime Upgrades
Copy new image over and reboot
Wait for failover to finish syncing,
and to “normalise” – approx 2 min
Verify config; conns replicated
Issue “failover active”
Copy new image over and reboot
Wait for failover to finish syncing,
and to “normalise” – approx 2 min
Verify config; conns replicated
Upgrade Complete
Issue “failover active”
SecondaryPrimary
Stb
Act Stb
Act
StateState
Act StbStart
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 83
For your
reference
Outbound Connection Inbound Connection
Connection Flags—Quick Reference
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 84
For your
referenceTCP Connection Termination ReasonsQuick Reference
Reason Description
Conn-Timeout Connection Ended Because It Was Idle Longer Than the Configured Idle Timeout
Deny Terminate Flow Was Terminated by Application Inspection
Failover Primary ClosedThe Standby Unit in a Failover Pair Deleted a Connection Because of a Message Received from the Active Unit
FIN TimeoutForce Termination After Ten Minutes Awaiting the Last ACK or After Half-Closed Timeout
Flow Closed by Inspection Flow Was Terminated by Inspection Feature
Flow Terminated by IPS Flow Was Terminated by IPS
Flow Reset by IPS Flow Was Reset by IPS
Flow Terminated by TCP Intercept
Flow Was Terminated by TCP Intercept
Invalid SYN SYN Packet Not Valid
Idle Timeout Connection Timed Out Because It Was Idle Longer than the Timeout Value
IPS Fail-Close Flow Was Terminated Due to IPS Card Down
SYN Control Back Channel Initiation from Wrong Side
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 85
For your
referenceTCP Connection Termination ReasonsQuick Reference (Cont.)
Reason Description
SYN TimeoutForce Termination After Two Minutes Awaiting Three-Way Handshake Completion
TCP Bad Retransmission Connection Terminated Because of Bad TCP Retransmission
TCP Fins Normal Close Down Sequence
TCP Invalid SYN Invalid TCP SYN Packet
TCP Reset-I TCP Reset Was Sent From the Inside Host
TCP Reset-O TCP Reset Was Sent From the Outside Host
TCP Segment Partial Overlap Detected a Partially Overlapping Segment
TCP Unexpected Window Size Variation
Connection Terminated Due to a Variation in the TCP Window Size
Tunnel Has Been Torn Down Flow Terminated Because Tunnel Is Down
Unauth Deny Connection Denied by URL Filtering Server
Unknown Catch-All Error
Xlate Clear User Executed the ‘Clear Xlate’ Command