Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in...

86
BRKSEC-3020 Advanced Firewalls

Transcript of Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in...

Page 1: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

BRKSEC-3020

Advanced Firewalls

Page 2: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 2

Page 3: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 3

Agenda

Packet Flow

Understanding the Architecture

Failover

Troubleshooting

Case Studies

Best Practices

Page 4: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

Packet Flow

Page 5: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 5

Accounting

Example Flow Flow

SRC IP: 10.1.1.9 SRC Port: 11030 Protocol: TCP

DST IP: 198.133.219.25 DST Port: 80

Interfaces

Source: Inside Destination: Outside

With the flow defined, examination of configuration boils down to just two interfaces: Inside and Outside

Client: 10.1.1.9

Server: 198.133.219.25S

erv

ers

Ou

tsid

e

Packet Flow

Page 6: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 6

Packet Processing Flow Diagram

The diagram below will be referenced on the following slides; it is shown here enlarged for reference

For your

reference

Page 7: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 7

Packet Processing: Ingress Interface

Packet arrives on ingress interface

Input counters incremented

Software input queue is an indicator of load

No buffers indicates packet drops, typically due to bursty traffic

ASA-5540# show interface gb-ethernet1

interface gb-ethernet1 "inside" is up, line protocol is up

Hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.6214

IP address 10.1.1.1, subnet mask 255.255.255.0

MTU 1500 bytes, BW 1 Gbit full duplex

5912749 packets input, 377701207 bytes, 0 no buffer

Received 29519 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

286298 packets output, 18326033 bytes, 0 underruns

input queue (curr/max blocks): hardware (0/25) software (0/0)

output queue (curr/max blocks): hardware (0/3) software (0/0)

Ingress

Interface

Page 8: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 8

Packet Processing: Locate Connection

Check first for existing connection

If connection exists, flow is matched; bypass ACL check

If no existing connection

TCP non-SYN packet, drop and log

TCP SYN or UDP packet, pass to ACL checks

Established Connection:

ASA-5540# show conn

TCP out 198.133.219.25:80 in 10.1.1.9:11030 idle 0:00:04 Bytes 1293 flags UIO

Syslog Because of No Connection, and Non-SYN Packet:

ASA-6-106015: Deny TCP (no connection) from 10.1.1.9/11031 to 198.133.219.25/80 flags PSH

ACK on interface inside

Existing

Conn

Page 9: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 9

Packet Processing: ACL Check

First packet in flow is processed through interface ACLs

ACLs are first match

First packet in flow matches ACE, incrementing hit count by one

Denied packets are dropped and logged

Packet Permitted by ACL:

ASA-5540B# show access-list inside

access-list inside line 10 permit ip 10.1.1.0 255.255.255.0 any (hitcnt=1)

Syslog When Packet Is Denied by ACL:

ASA-4-106023: Deny tcp src inside:10.1.1.9/11034 dst outside:198.133.219.25/80 by

access-group "inside"

ACL

Permit

Page 10: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 10

Packet Processing: Match Translation

First packet in flow must match a translation rule*

A quick route lookup is done only to determine egress interface

Translation rule can be to NAT, or not to NAT

NAT order of operations dictates what happens with overlapping translation rules (See Appendix)

Once translation rule is matched, connection is created

Translation Exists:ASA-5540# show xlate debug

NAT from inside:10.1.1.9 to outside:172.18.124.68 flags - idle 0:00:07 timeout 3:00:00

Syslogs When No Translation Rule Found: (305005—No NAT; 305006—No Global)ASA-3-305005: No translation group found for tcp src inside:10.1.1.9/11039 dst

outside:198.133.219.25/80

ASA-3-305006: regular translation creation failed for tcp src inside:10.1.1.9/11040 dst

outside:198.133.219.25/80

Match

xLate

Page 11: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 11

Packet Processing: Inspections/Sec Checks

Inspections are applied to ensure protocol compliance

(Optional) customized AIC inspections

NAT-embedded IPs in payload

Additional security checks are applied to the packet

(Optional) packets passed to Content Security and Control (CSC) module

Syslog from Packets Denied by Security Check:

ASA-4-406002: FTP port command different address: 10.2.252.21(192.168.1.21) to

209.165.202.130 on interface inside

ASA-4-405104: H225 message received from outside_address/outside_port to

inside_address/inside_port before SETUP

Inspections

Sec Checks

Question!What command will show you if packets are being dropped

by one of the Inspection engines?

Page 12: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 12

Packet Processing: NAT IP Header

Translate the IP address in the IP header

Translate the port if performing PAT

Update checksums

(Optional) Following the above, pass packet to IPS (AIP) module

Nat IP

Header

Page 13: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 13

Packet Processing: Egress Interface

Packet is virtually forwarded to egress interface (i.e., not forwarded to the driver yet)

Egress interface is determined first by translation rules

If translation rules do not specify egress interface (e.g., outbound initial packet) the results of a global route lookup are used to determine egress interface

Example:

static (inside, outside) 192.168.0.0 172.16.0.0 netmask 255.255.0.0

static (dmz, outside) 192.168.12.0 172.16.12.0 netmask 255.255.255.0

DM

Z

Inside Outside

172.16.0.0/16 172.16.12.0/24

172.16.12.4

Inbound packets to 192.168.12.4 get routed to Inside based on order of statics

Egress

Interface

Page 14: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 14

Packet Processing: L3 Route Lookup

Once on egress interface, an interface route lookup is performed

Only routes pointing out the egress interface are eligible

Remember: translation rule can forward the packet to the egress interface, even though the routing table may point to a different interface

Syslog from Packet on Egress Interface with No Route Pointing Out Interface:

ASA-6-110001: No route to 209.165.202.130 from 10.1.1.9

L3

Route

Page 15: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 15

Packet Processing: L2 Address Lookup

Once a Layer 3 route has been found, and next hop identified, Layer 2 resolution is performed

Layer 2 rewrite of MAC header

If Layer 2 resolution fails—no syslog

show arp will not display an entry for the L3 next hop

debug arp will indicate if we are not receiving an

ARP reply

L2

Addr

Page 16: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 16

Packet Processing: Transmit Packet

Packet is transmitted on wire

Interface counters will increment on interface

Output hardware and software queues indicate buffering at driver level, interface is busy

ASA-5540# show interface gb-ethernet0

interface gb-ethernet0 "outside" is up, line protocol is up

Hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.626c

IP address 172.18.124.64, subnet mask 255.255.255.0

MTU 1500 bytes, BW 1 Gbit full duplex

3529518 packets input, 337798466 bytes, 0 no buffer

Received 32277 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

5585431 packets output, 359059032 bytes, 0 underruns

input queue (curr/max blocks): hardware (0/25) software (0/0)

output queue (curr/max blocks): hardware (0/2) software (0/0)

Xmit

Pkt

Page 17: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 17

Agenda

Packet Flow

Understanding the Architecture

Failover

Troubleshooting

Case Studies

Best Practices

Page 18: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 18

Cisco ASA —Understanding the Architecture

ASA processes all packets in software (via the central CPU)

All packets are processed first in… usually also first out

ASA platforms have software imposed connection limits

Multi-CPU / Multi-Core systems hash packets in the same flow to the same CPU/core.

10 Gig interfaces hash flow to same RX ring.

Page 19: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 19

Maximum ACL Limits ASA has no hard-coded limit on the number of elements (ACEs) in an ACL. Bound

only by Memory.

Each ACE uses a minimum of 212 bytes of RAM

FWSM has a hard-coded limit based on memory of NP 3.

5505 5510 5520 5540 5550 5580 FWSM

Max Recommended

ACEs25k 80k 200k 500k 700k 750k 100k

Max Observed (from

customers)2.74 million 2.77 million 100k

Note: Issue show access-list | include elements to see how many ACEs you have

Page 20: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 20

Global ACLs

Global ACLs were introduced in ASA version 8.3

Best used for new installations, or migration from other vendors

Interface Independent Policies

access-group <access_list> global

Global access-list

Interface Specific access-list

Default (implicit) deny ip any any

Policy Ordering

ASA

Only

Page 21: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 21

FWSM—Understanding the Architecture

Packets processed in hardware have zero impact on CPU

Similarly, if the CPU is pegged at 100%, this has zero impact on packets processed in hardware

FWSM Process Most Packets in Hardware, with Some Packets Needing to be Processed in Software—via the Control Point (CP)

Page 22: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 22

Fast Path

Flow Identification,

Security Checks and

NAT in Hardware

FWSM Architectural Overview

C6K Backplane Interface

Session Manager

NP 3

Control Point (CP)

Central CPU

Fast Path

NP 1

Software

Hardware

FW

SM

Control Point

ACL Compilation,Fixups,

Syslog, AAA, IPv6 in

Software

Session Manager

Session Establishment and

Teardown, AAA Cache,

ACLs

Fast Path

NP 2

Page 23: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 23

FWSM—Hardware Limits FWSM has several hardware limits that should be considered in

your network design

Limits are hard set, but vary based on single or multimode

Some limits include:

2.3 (Multimode) 3.1/3.2 (Multimode) 4.0/4.1 (Multimode)3.2 / 4.0 /4.1 Configurable

ACEs 56,627 (9,704) 72,806 (11,200) 100,567 (14,801) X

AAA Rules 3,942 (606) 6,451 (992) 8,744 (1,345) X

Global Statements 1K (1K) 4K (4K) 4K (4K)

Static NAT Statements 2K (2K) 2K (2K) 2K (2K)

Policy NAT ACEs 3,942 (606) 1,843 (283) 2,498 (384) X

NAT Translations 256K (256K) 256K (256K) 256K (256K)

Connections 999,990 (999,990) 999,990 (999,990) 999,990 (999,990)

Route Table Entries 32K (32K) 32K (32K) 32K (32K)

Fixup/Inspect Rules 32 (32 per) 4147 (1,417) 5621 (1,537) X

Filter Statements 3942 (606) 2764 (425) 3747 (576) X

Increase over 2.3 Increase over 3.1

Page 24: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 24

Agenda

Packet Flow

Understanding the Architecture

Failover

Troubleshooting

Case Studies

Best Practices

Page 25: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 25

Failover Basics Active/Standby vs. Primary/Secondary

Serial vs. LAN failover

Stateful failover (optional)

A failover only occurs when either firewall determines the standby firewall is healthier than the active firewall

Both firewalls swap MAC and IP addresses when a failover occurs

Level 1 syslogs will give reason of failover

Secondary

(Standby)Primary

(Active)

LAN/Serial

Stateful

Internet

Corp

Page 26: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 26

ASA# show failover

Failover On

Failover unit Primary

Failover LAN Interface: failover Redundant5 (up)

Unit Poll frequency 200 milliseconds, holdtime 1 seconds

Interface Poll frequency 500 milliseconds, holdtime 5 seconds

Interface Policy 1

Monitored Interfaces 2 of 250 maximum

Version: Ours 8.2(2), Mate 8.2(1)

Last Failover at: 10:37:11 UTC May 14 2010

This host: Primary - Active

Active time: 1366024 (sec)

slot 0: ASA5580 hw/sw rev (1.0/8.1(2)) status (Up Sys)

Interface outside (10.8.20.241): Normal

Interface inside (10.89.8.29): Normal

Other host: Secondary - Standby Ready

Active time: 0 (sec)

slot 0: ASA5580 hw/sw rev (1.0/8.1(2)24) status (Up Sys)

Interface outside (10.8.20.242): Normal

Interface inside (10.89.8.30): Normal

Stateful Failover Logical Update Statistics

Link : stateful Redundant6 (up)

Stateful Obj xmit xerr rcv rerr

General 424525 0 424688 0

sys cmd 423182 0 423182 0

Verifying Failover Configuration

Interface

Monitoring

Page 27: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 27

What Triggers a Failover?

Power loss/reload (this includes crashes) on the Active firewall

SSM interface/module failure

The Standby becoming healthier than the Active firewall

In LAN based Failover, what happens if the LAN interface

communication is severed?

Page 28: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 28

What Triggers a Failover? (Con’t)

Two consecutive hello messages missed on any monitored interface forces the interface into testing mode

Both units first verify the link status on the interface

Next, both units execute the following tests

Network activity test

ARP test

Broadcast ping test

The first test passed causes the interface on that unit to be marked healthy; only if all tests fail will the interface be marked failed

Page 29: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 29

What to Do After a Failover

Always check the syslogs to determine root cause

Example: switch port failed on inside interface of active firewall

ASA-4-411002: Line protocol on Interface inside, changed state to down

ASA-1-105007: (Primary) Link status ‘Down’ on interface 1

ASA-1-104002: (Primary) Switching to STNDBY—interface check, mate is healthier

ASA-1-104001: (Secondary) Switching to ACTIVE—mate want me Active

Syslogs from Primary (Active) Firewall

Syslogs from Secondary (Standby) Firewall

Page 30: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 30

Agenda

Packet Flow

Understanding the Architecture

Failover

Troubleshooting

Case Studies

Best Practices

Page 31: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 31

Troubleshooting Tools

Syslogs

Debug commands

Show commands

Packet capture

Packet tracer

Page 32: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 32

Buffered

Uses of Syslogs

Primary mechanism to record traffic to and throughthe firewall

The best troubleshooting tool available

SSH Client

Internet

Archival Purposes Debugging Purposes

Syslog Server

SNMP Server

Console

Trap .Syslog

Page 33: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 33

Logging – Common Issues

logging flash-bufferwrapshould only be used when logging to buffer at Level 1

logging history should only be used when you really have an SNMP server that you want to receive all syslogs

logging consoleshould only be enabled while actively troubleshooting on the Console

logging standbyshould only be used if you want to receive double the syslogs

logging permit-hostdownshould always be used with TCP syslogging

Page 34: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 34

Debug Commands

1. Debugs should not be the first choice to troubleshoot a problem

2. Debugs can negatively impact the CPU of the box, and also the performance of it; use with caution

3. Debugs are not conditional*

4. Know how much traffic, of the specified type, is passing through the firewall before enabling the respective debug

* Crypto Conditional Debugging Was Added to Cisco ASA/PIX 8.0

Page 35: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 35

Debug ICMP Trace

Valuable tool used to troubleshoot connectivity issues

Provides interface and translation information to quickly determine flow

Echo-replies must be explicitly permitted through ACL, or ICMP inspection must be enabled

ICMP echo-request from inside:10.1.1.2 to 198.133.219.25 ID=3239 seq=4369 length=80

ICMP echo-request: translating inside:10.1.1.2 to outside:209.165.201.22

ICMP echo-reply from outside:198.133.219.25 to 209.165.201.22 ID=3239 seq=4369 length=80

ICMP echo-reply: untranslating outside:209.165.201.22 to inside:10.1.1.2

Example debug icmp trace output

http://www.cisco.com

Internet

Page 36: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 36

Redirecting Debugs to Syslog Problem

Log only debug output to syslog

Solution

Create a logging list with only syslog ID 711001

Enable debug output to syslogs

Log on the logging list

ASA(config)# logging trap CiscoLive .

ASA(config)# logging list CiscoLive message 711001 .

ASA(config)# logging debug-trace

INFO: 'logging debug-trace' is enabled. All debug messages are currently

being redirected to syslog:711001 and will not appear in any monitor

session

Page 37: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 37

Show Output Filters

Use output filters to filter the output of show command to only the information you want to see

To use them, at the end of show <Command>, use the pipe character ―|‖ followed by

begin Start displaying the output beginning at the first match of the

RegEx, and continue to display the remaining output

include Display any line that matches the RegEx

exclude Display any line that does not match the RegEx

grep Same as include

grep –v Same as exclude

show <cmd> | begin|include|exclude|grep [-v] <regular_exp>

Page 38: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 38

Show CPU Usage

Under normal conditions the CPU should stay below 50% (baseline as per network); if the CPU reaches 100% the firewall will start dropping packets

FWSM CPU is used for limited traffic processing; during ACL compilation CPU is expected to be near 100% until ACL is compiled

The show cpu usage command displays the CPU over time as a running average

ASA# show cpu usage

CPU utilization for 5 seconds = 5%; 1 minute: 4%; 5 minutes: 4%

*First Introduced in Cisco PIX OS Version 6.0(1)/FWSM 1.1(1)

Page 39: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 39

Show Processes cpu-usage

The show processes cpu-usage command displays the amount of CPU used on a per-process basis for the last 5sec, 1min, and 5min

ASA# # show processes cpu-usage

PC Thread 5Sec 1Min 5Min Process

081aa124 d51ab230 0.2% 2.0% 2.0% Dispatch Unit

08070416 d51aa660 0.0% 0.0% 0.0% aaa

081a954c d51a96a0 0.0% 0.0% 0.0% dbgtrace

08c2a91d d51a7f00 0.0% 0.0% 0.0% netfs_thread_init

0924fe95 d51a7528 0.0% 0.0% 0.0% Chunk Manager

088a6e14 d51a7138 0.0% 0.0% 0.0% IP Address Assign

08a6d7f6 d51a6f40 0.0% 0.0% 0.0% QoS Support Module

08bcf736 d51a53b0 0.0% 0.0% 0.0% Logger

08685627 d51a3a18 0.0% 0.0% 0.0% netfs_mount_handler

0851ca68 d51a3820 0.0% 0.0% 0.0% arp_timer

08b9ffab d5198ae0 0.0% 0.0% 0.0% ssh/timer

08b99aec d5195d98 3.9% 0.5% 0.1% ssh

...

*First Introduced in Cisco ASA Version 7.2(4.11), 8.0(4.5), 8.1(1.100), 8.2(1).

Currently not Available in FWSM

Page 40: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 40

Show Processes cpu-hog The show processes cpu-hog command displays a list of processes, and the

function stack (Traceback) which executed, and lead to a process running on the CPU longer than the minimum platform threshold

ASA# show processes cpu-hog

Process: ssh_init, NUMHOG: 18, MAXHOG: 15, LASTHOG: 10

LASTHOG At: 14:18:47 EDT May 29 2009

PC: 8b9ac8c (suspend)

Traceback: 8b9ac8c 8ba77ed 8ba573e 8ba58e8 8ba6971

8ba02b4 8062413

CPU hog threshold (msec): 10.240

Last cleared: None

*First introduced in Cisco ASA Version 7.0(1). Currently not Available in FWSM

May 29 2009 14:18:47: %ASA-7-711002: Task ran for 10 msec,

Process = ssh_init, PC = 8b9ac8c, Traceback = 0x08B9AC8C 0x08BA77ED

0x08BA573E 0x08BA58E8 0x08BA6971 0x08BA02B4 0x08062413

A corresponding syslog message is also generatedNote: The Traceback does not signify a crash

Page 41: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 41

Show Traffic

The show traffic command displays the traffic received

and transmitted out each interface of the firewall

ASA# show traffic

outside:

received (in 124.650 secs):

295468 packets 167218253 bytes

2370 pkts/sec 1341502 bytes/sec

transmitted (in 124.650 secs):

260901 packets 120467981 bytes

2093 pkts/sec 966449 bytes/sec

inside:

received (in 124.650 secs):

261478 packets 120145678 bytes

2097 pkts/sec 963864 bytes/sec

transmitted (in 124.650 secs):

294649 packets 167380042 bytes

2363 pkts/sec 1342800 bytes/sec

Page 42: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 42

show np blocks (FWSM Only)

The show np blocks command is used to see if the FWSM is over subscribed

FWSM# show np blocks

MAX FREE THRESH_0 THRESH_1 THRESH_2

NP1 (ingress) 32768 32768 0 0 550

(egress) 521206 521206 0 0 0

NP2 (ingress) 32768 32768 0 0 92

(egress) 521206 521206 0 0 0

NP3 (ingress) 32768 32768 13 460417 4427509

(egress) 521206 521206 0 0 0

Warning

Data

packets

dropped

Data and

Control packets

dropped

Page 43: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 43

Show Xlate and Show Xlate Debug The show xlate command displays information about the

translations through the firewall

You can limit the output to just the local or global IP

ASA# show xlate debug

2 in use, 2381 most used

Flags: D - DNS, d - dump, I - identity, i - inside, n - no random,

o - outside, r - portmap, s - static

NAT from inside:10.1.1.9 to outside:172.18.124.68

flags - idle 0:02:03 timeout 3:00:00

TCP PAT from inside:10.9.9.3/4101 to outside:172.18.124.65/1024

flags r idle 0:00:08 timeout 0:00:30

ASA# show xlate

2 in use, 2381 most used

Global 172.18.124.68 Local 10.1.1.9

PAT Global 172.18.124.65(1024) Local 10.9.9.3(4101)

debug Adds interface Names,

Idle, and xlate Timeouts

Page 44: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 44

ASA# show conn

2 in use, 64511 most used

TCP outside 198.133.219.25:80 dmz 10.9.9.3:4101, idle 0:00:06, Bytes 127, flags UIO

UDP outside 172.18.124.1:123 dmz 10.1.1.9:123 idle 0:00:13 flags –

Show Conn

Idle Time,

Bytes TransferredConnection Flags

Real Interface

Names Added in

7.2(4), 8.0(4)

Page 45: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 45

Show Conn Detail

ASA# show conn detail

2 in use, 64511 most used

Flags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN,

B - initial SYN from outside, C - CTIQBE media, D - DNS, d - dump,

E - outside back connection, F - outside FIN, f - inside FIN,

G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data,

i - incomplete, J - GTP, j - GTP data, K - GTP t3-response

k - Skinny media, M - SMTP data, m - SIP media, n - GUP

O - outbound data, P - inside back connection, q - SQL*Net data,

R - outside acknowledged FIN,

R - UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN,

s - awaiting outside SYN, T - SIP, t - SIP transient, U - up, W - WAAS,

X - inspected by service module

TCP outside:198.133.219.25/80 dmz:10.9.9.3/4101,

flags UIO, idle 8s, uptime 10s, timeout 1h, bytes 127

UDP outside:172.18.124.1/123 dmz:10.1.1.9/123,

flags -, idle 15s, uptime 16s, timeout 2m, bytes 1431

detail Adds Uptime and

Timeout in 7.2(4), 8.0(4)

Page 46: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 46

show local-host A local-host entry is created for any IP tracked through the firewall

It groups the xlates, connections, and AAA information

Very useful for seeing the connections terminating on servers

ASA# show local-host

Interface inside: 1131 active, 2042 maximum active, 0 denied

local host: <10.1.1.9>,

TCP connection count/limit = 1/unlimited

TCP embryonic count = 0

TCP intercept watermark = 50

UDP connection count/limit = 0/unlimited

AAA:

user 'cisco' at 10.1.1.9, authenticated (idle for 00:00:10)

absolute timeout: 0:05:00

inactivity timeout: 0:00:00

Xlate(s):

Global 172.18.124.69 Local 10.1.1.9

Conn(s):

TCP out 198.133.219.25:80 in 10.1.1.9:11055 idle 0:00:10 Bytes 127 flags UIO

Page 47: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 47

show service-policy The show service-policy command is used to quickly see

what inspection policies are applied and the packets matching them

ASA# show service-policy

Global policy:

Service-policy: global_policy

Class-map: inspection_default

Inspect: dns maximum-length 512, packet 92, drop 0, reset-drop 0

Inspect: ftp, packet 43, drop 0, reset-drop 0

Inspect: h323 h225, packet 0, drop 0, reset-drop 0

Inspect: h323 ras, packet 0, drop 0, reset-drop 0

Inspect: http, packet 562, drop 0, reset-drop 0

Inspect: netbios, packet 0, drop 0, reset-drop 0

Inspect: rsh, packet 0, drop 0, reset-drop 0

Inspect: rtsp, packet 0, drop 0, reset-drop 0

Inspect: skinny, packet 349, drop 0, reset-drop 0

Inspect: esmtp, packet 0, drop 0, reset-drop 0

...

Interface outside:

Service-policy: VoIP

Class-map: voice_marked

Priority:

Interface outside: aggregate drop 0, aggregate transmit 349

Page 48: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 48

show asp drop Packets dropped in the Accelerated Security Path (ASP) will increment a counter

FWSM – applies only to traffic sent to the control-point

Frame drop counters are per packet, flow drops are per flow

Some counters have corresponding syslogs

ASA# show asp drop

Frame drop:

Invalid encapsulation (invalid-encap) 10897

Invalid tcp length (invalid-tcp-hdr-length) 9382

Invalid udp length (invalid-udp-length) 10

No valid adjacency (no-adjacency) 5594

No route to host (no-route) 1009

Reverse-path verify failed (rpf-violated) 15

Flow is denied by access rule (acl-drop) 25247101

First TCP packet not SYN (tcp-not-syn) 36888

Bad TCP flags (bad-tcp-flags) 67148

TCP option list invalid (tcp-bad-option-list) 731

TCP MSS was too large (tcp-mss-exceeded) 10942

Bad TCP Checksum (bad-tcp-cksum) 893

*Drop Counters Are Documented in the CMD Ref, Under show asp drop

Page 49: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 49

Packet Capture

Capture command first introduced in Cisco 7.0; FWSM need to use 3.1.5 or later

ASA 7.2(3) and 8.0(3) added a real-time option

ASDM 6.0 adds a capture wizard

Capture sniffs packets on an interface that match an ACL, or match line

Key steps

Create an ACL that will match interesting traffic

Define the capture and bind it to an access-list and interface

View the capture on the firewall, or copy it off in .pcap format

capture <capture-name> [access-list <acl-name>] [buffer <buf-size>] [ethernet-type

<type>] [interface <if-name>] [packet-length <bytes>]

[circular-buffer] [type raw-data|asp-drop|isakmp|webvpn user <username>]

[match <prot> {host <sip> | <sip> <mask> | any}

[eq | lt |gt <port>] {host <dip> | <dip> <mask> | any}

[eq | lt | gt <port>]]

[real-time [dump] [detail] [trace]] [trace [detail] [trace-count <1-1000>]]

Page 50: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 50

Packet Capture (Cont.)

Traffic can be captured both before and after it passes through the firewall

• Ingress packets are captured before any packet processing has been done on them

• Egress packets are captured after all processing (excluding L2 source MAC rewrite)

Capture buffer saved in RAM (default size 512 KB)

Copy captures off via TFTP or HTTPS

Capture all packets dropped by the ASP

ASA# capture drops type asp-drop all

Inside

Capture In Capture OutInside Capture Outside Capture

Page 51: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 51

Packet Tracer: Overview

Introduced in ASA ver 7.2

A packet tagged with the trace option is injected into the interface, and processed in the data-plane

Each action taken on the packet is recorded in the packet itself

When the packet reaches the egress interface, or is dropped, it is punted to the control-plane

The control-plane reads and displays the actions taken on the packet, along with the associated lines in the configuration

ASA

Only

Page 52: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 52

Link Back

to Edit RuleMatching

Config

Define

Packet

Action

Final Result

Packet Tracer: ASDM (Located off Tools Menu)

Page 53: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 53

Packet Tracer: CLIASA# packet-tracer input inside tcp 10.1.1.2 1024 198.133.219.25 80

Phase: 1

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

...

Phase: 15

Type: ROUTE-LOOKUP

Subtype: output and adjacency

Result: ALLOW

Config:

Additional Information:

found next-hop 209.165.201.1 using egress ifc outside

adjacency Active

next-hop mac address 000a.f331.83c0 hits 0

>>>>Packet successfully forwarded to fast path<<<<

Page 54: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 54

Packet Tracer: Tracing Captured Packet

Create a capture using the trace option

Find the packet in the capture you want traced

Then select that packet to be traced

ASA# show capture inside trace packet-number 4 .

ASA# capture inside access-list web interface inside trace .

ASA# show capture inside

68 packets captured

1: 15:22:47.581116 10.1.1.2.31746 > 198.133.219.25.80: S

2: 15:22:47.583465 198.133.219.25.80 > 10.1.1.2.31746: S ack

3: 15:22:47.585052 10.1.1.2.31746 > 198.133.219.25.80: . ack

4: 15:22:49.223728 10.1.1.2.31746 > 198.133.219.25.80: P ack

5: 15:22:49.223758 198.133.219.25.80 > 10.1.1.2.31746: . Ack

...

Important!

Page 55: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 55

Agenda

Packet Flow

Understanding the Architecture

Failover

Troubleshooting

Case Studies

Best Practices

Page 56: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

56

Case StudyFWSM – Slow Single-Flow TCP Throughput

Page 57: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 57

Case Study: FWSM Slow TCP Throughput

Problem

TCP based backups are taking longer than expected through the FWSM

iPerf performance testing is only showing ~450 Mbps through FWSM

Page 58: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 58

Due to the FWSM‘s NP architecture, there exists a possibility that packets arriving with a low inter-packet gap might be re-ordered by the FWSM.

FWSM version 3.1(10) and 3.2(5) introduce a new command sysopt np completion-unit to ensure the firewall maintains the packet order

1

TCP Flow

234 1324

Note: In multi-mode add command to admin context, and it will be applied globally

1

TCP Flow

234 1234

Enable np

completion-unit

Case Study: FWSM Slow TCP Throughput

FWSM

Only

Page 59: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

59

Case StudyIntermittent Access to Web Server

Page 60: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 60

Clients

Problem

Most external clients are not able to load company‘s web page

10.1.1.50ASA-5510

HTTP Requests to 192.168.1.50

Web Server

NATed to 10.1.1.50

Internet

Case Study: Intermittent Access to Web Server

Page 61: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 61

Traffic Spike

Case Study: Intermittent Access to Web Server

Page 62: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 62

show perfmon indicates high number of embryonic connectionsASA-5510# show perfmon

PERFMON STATS: Current Average

Xlates 0/s 0/s

Connections 2059/s 299/s

TCP Conns 2059/s 299/s

UDP Conns 0/s 0/s

URL Access 0/s 0/s

URL Server Req 0/s 0/s

TCP Fixup 0/s 0/s

TCP Intercept Established Conns 0/s 0/s

TCP Intercept Attempts 0/s 0/s

TCP Embryonic Conns Timeout 1092/s 4/s

HTTP Fixup 0/s 0/s

FTP Fixup 0/s 0/s

AAA Authen 0/s 0/s

AAA Author 0/s 0/s

AAA Account 0/s 0/s

VALID CONNS RATE in TCP INTERCEPT: Current Average

N/A 95.00%

Case Study: Intermittent Access to Web Server

Page 63: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 63

Issue show conn to see ‗who‘ is creating the connections

ASA-5510# show conn

54764 in use, 54764 most used

TCP outside 17.24.101.118:26093 inside 10.1.1.50:80, idle 0:00:23, bytes 0, flags aB

TCP outside 111.76.36.109:23598 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB

TCP outside 24.185.110.202:32729 inside 10.1.1.50:80, idle 0:00:25, bytes 0, flags aB

TCP outside 130.203.2.204:56481 inside 10.1.1.50:80, idle 0:00:29, bytes 0, flags aB

TCP outside 39.142.106.205:18073 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB

TCP outside 75.27.223.63:51503 inside 10.1.1.50:80, idle 0:00:03, bytes 0, flags aB

TCP outside 121.226.213.239:18315 inside 10.1.1.50:80, idle 0:00:04, bytes 0, flags aB

TCP outside 66.187.75.192:23112 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB

TCP outside 13.50.2.216:3496 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB

TCP outside 99.92.72.60:47733 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aB

TCP outside 30.34.246.202:20773 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB

TCP outside 95.108.110.131:26224 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB

TCP outside 76.181.105.229:21247 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB

TCP outside 82.210.233.230:44115 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB

TCP outside 134.195.170.77:28138 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB

TCP outside 70.133.128.41:22257 inside 10.1.1.50:80, idle 0:00:15, bytes 0, flags aB

TCP outside 124.82.133.172:27391 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aB

TCP outside 26.147.236.181:37784 inside 10.1.1.50:80, idle 0:00:07, bytes 0, flags aB

TCP outside 98.137.7.39:20591 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB

TCP outside 37.27.115.122:24542 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB

. . .

Random Sources Embryonic Conns

Case Study: Intermittent Access to Web Server

Page 64: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 64

Case Study: Intermittent Access to Web Server

Connection Count

Jumps

Traffic Permitted

SYN Flood

Detected

Page 65: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 65

Apply TCP Intercept to stop the SYN flood attack

access-list 140 extended permit tcp any host 192.168.1.50 eq www

!

class-map protect

description Protect web server from attacks

match access-list 140

!

policy-map interface_policy

class protect

set connection embryonic-conn-max 100

!

service-policy interface_policy interface outside

Case Study: Intermittent Access to Web Server

Page 66: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 66

TCP Intercept AppliedFew Clients Represent 50+

% of Traffic

Case Study: Intermittent Access to Web Server

Why did the Connection count drop after TCP Intercept

was applied?

Page 67: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 67

access-list 140 extended permit tcp any host 192.168.1.50 eq www

!

class-map protect

description Protect web server from attacks

match access-list 140

!

policy-map interface_policy

class protect

set connection embryonic-conn-max 100 per-client-max 25

!

service-policy interface_policy interface outside

Apply per-client-max option to limit the number of connections any single client can establish

Case Study: Intermittent Access to Web Server

Page 68: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 68

TCP Intercept

per-client-max

Case Study: Intermittent Access to Web Server

Page 69: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 69

Attacks Still Occurring

Attacks Being

Mitigated

Case Study: Intermittent Access to Web Server

Page 70: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 70

Agenda

Packet Flow

Understanding the Architecture

Failover

Troubleshooting

Case Studies

Best Practices

Page 71: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 71

Cisco ASA/FWSM Best Practices

Enable ip verify reverse-path on all interfaces

Set embryonic and maximum connection counts on static and natstatements; for 7.2.1+ use per-client-max

Configure logging to syslog server

Move messages you want to see to lower levels, instead of raising logging levels and capturing messages you don‘t want to see

Disable telnet access! Use SSH for management access

Enable authentication for management access (console/SSH/telnet/enable); use TACACS+ or RADIUS with LOCAL as the fallback

Page 72: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 72

Cisco ASA/FWSM Best Practices

Restrict DMZ access inbound to your internal networks

Baseline CPU load, connection counts, xlate counts, and traffic (per interface)

Monitor stats using MRTG or other snmp graphing tools.

Keep config archives (and show tech ouputs) (Rancid)

Run the latest maintenance release in your train

Upgrade major feature trains only when you need new features, or after train has matured

Page 73: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 73

Agenda

Packet Flow

Understanding the Architecture

Failover

Troubleshooting

Case Studies

Best Practices

Page 74: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

Online Resources

Page 75: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 75

Online Learning Modules – VoD Training

Great way to learn about new features in the ASA

From www.cisco.com select: Products and Services

Security

Network Security (expand)

Cisco ASA 5500 Series

Training resources

Online learning modules

Search cisco.com for ASA Online Learning Modules

Direct linkhttp://www.cisco.com/en/US/partner/products/ps6120/tsd_products_support_online_learning_modules_list.html

Page 76: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 76

Supportforums.cisco.com

Public wiki – anyone can author articles

Combines supportwiki and Netpro forums

Sections for: ASA, FWSM and PIX

Hundreds of Sample Configs

Troubleshooting Docs

FAQshttp://supportforums.cisco.com/

Page 77: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

Q & A

Page 78: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 78

Complete Your Online Session Evaluation

Complete your session evaluation:

Directly from your mobile device by visiting www.ciscoliveaustralia.com/mobile and login by entering your badge ID (located on the front of your badge)

Visit one of the Cisco Live internet stations located throughout the venue

Open a browser on your own computer to access the Cisco Live onsite portal

Page 79: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow
Page 80: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

80

Appendix

Page 81: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 81

Translation and NAT Order of Operations

1. nat 0 access-list (nat-exempt)

2. Match existing xlates

3. Match static commands (Cisco ASA/PIX first match; FWSM best match)

Static NAT with and without access-list

Static PAT with and without access-list

4. Match nat commandsnat <id> access-list (first match)

nat <id> <address> <mask> (best match)

If the ID is 0, create an identity xlate

Use global pool for dynamic NAT

Use global pool for dynamic PAT

Firs

t Ma

tch

Match

xLate

Page 82: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 82

Failover – Zero Downtime Upgrades

Copy new image over and reboot

Wait for failover to finish syncing,

and to “normalise” – approx 2 min

Verify config; conns replicated

Issue “failover active”

Copy new image over and reboot

Wait for failover to finish syncing,

and to “normalise” – approx 2 min

Verify config; conns replicated

Upgrade Complete

Issue “failover active”

SecondaryPrimary

Stb

Act Stb

Act

StateState

Act StbStart

Page 83: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 83

For your

reference

Outbound Connection Inbound Connection

Connection Flags—Quick Reference

Page 84: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 84

For your

referenceTCP Connection Termination ReasonsQuick Reference

Reason Description

Conn-Timeout Connection Ended Because It Was Idle Longer Than the Configured Idle Timeout

Deny Terminate Flow Was Terminated by Application Inspection

Failover Primary ClosedThe Standby Unit in a Failover Pair Deleted a Connection Because of a Message Received from the Active Unit

FIN TimeoutForce Termination After Ten Minutes Awaiting the Last ACK or After Half-Closed Timeout

Flow Closed by Inspection Flow Was Terminated by Inspection Feature

Flow Terminated by IPS Flow Was Terminated by IPS

Flow Reset by IPS Flow Was Reset by IPS

Flow Terminated by TCP Intercept

Flow Was Terminated by TCP Intercept

Invalid SYN SYN Packet Not Valid

Idle Timeout Connection Timed Out Because It Was Idle Longer than the Timeout Value

IPS Fail-Close Flow Was Terminated Due to IPS Card Down

SYN Control Back Channel Initiation from Wrong Side

Page 85: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 85

For your

referenceTCP Connection Termination ReasonsQuick Reference (Cont.)

Reason Description

SYN TimeoutForce Termination After Two Minutes Awaiting Three-Way Handshake Completion

TCP Bad Retransmission Connection Terminated Because of Bad TCP Retransmission

TCP Fins Normal Close Down Sequence

TCP Invalid SYN Invalid TCP SYN Packet

TCP Reset-I TCP Reset Was Sent From the Inside Host

TCP Reset-O TCP Reset Was Sent From the Outside Host

TCP Segment Partial Overlap Detected a Partially Overlapping Segment

TCP Unexpected Window Size Variation

Connection Terminated Due to a Variation in the TCP Window Size

Tunnel Has Been Torn Down Flow Terminated Because Tunnel Is Down

Unauth Deny Connection Denied by URL Filtering Server

Unknown Catch-All Error

Xlate Clear User Executed the ‘Clear Xlate’ Command

Page 86: Advanced Firewallsd2zmdbbm9feqrf.cloudfront.net/2011/anz/pdf/BRKSEC-3020.pdf · First packet in flow is processed through interface ACLs ACLs are first match First packet in flow