Instructions cisco

ASA Clustering Deployment and Troubleshooting Lab

LTRSEC-2740

Goran Saradzic and Per Hagen

© 2015 Cisco and/or its affiliates. All rights reserved. LTRSEC-2740 Cisco Public

Agenda

ASA Clustering Lab (3.5hrs)

Tasks divide into Router and Switch-based mechanisms

• Equal Cost Multipath (ECMP)

1. Stand-alone ASAs via OSPF

2. L3 / Individual Mode Cluster via OSPF

3. L3 / Individual Mode Cluster via IP SLA

• Ether-Channel (ECLB)

4. L2 /Spanned Mode Cluster in Routed (OSPF to Master )

5. L2 / Spanned Mode Cluster in Transparent

Overview (30min)

• Lab POD Access

• ASA clustering options

• ASA Designs in Lab

• Exercise workflow

– Review, deploy, verify

– Bring down one ASA

– Measure convergence

– Bring ASA back online

3


Achieving the Best Uptime for Your Applications

Tolerance to failure – continuing your critical client connections

Solution resiliency – know your convergence times

Elastic scale and capacity – easily address your future growth

Efficient management – low complexity and overhead

Support for redundant locations – ability to extend to multiple sites

Workload mobility with security – migrate live apps across locations

Traffic normalization for NGFW and NGIPS services

Ensuring service and application availability

4


Realizing True Values of ASA Firewall Clustering

5

Scale to 16

Nodes

Simple Mgmt High Availability State Sharing

One Config

One Master

CCL


Deployment Options Overview of ASA cluster types, firewall and context modes

6

Must configure L2 spanned mode cluster to use Transparent firewall

L3 Individual mode requires Routed firewall

Multiple context mode works in both types of clustering

Load Balancing Firewall Modes and Features

Transparent Routed Multiple Contexts

Individual Interface

L3 Method ECMP/ PBR N/A* ✔ ✔

Spanned Interface

L2 Method Ether-Channel LB ✔ ✔ ✔


Lab Portal

7

https://labops-out.cisco.com/labops/ilt

Using Class Name,

you will log in first to add your profile

information, and then log back in,

to access PODs.

Prep

7





Pick a Pod

Prep

8


Access your POD

9

Prep

9


Lab Portal Diagram Open RDP Session Only

10

Click to RDP

login:

Administrator

password:

stgscvt

ASA, Host, and CSR

sessions are auto-opened

in SuperPutty on the

JumpBox RDP

(see next slide) .30

Prep

If needed, you can

increase the RDP

resolution size

appropriate to

your display.

10


Lab Access Credentials

• Access Lab Portal with your email and lab-ID, add profile, log back in

• JumpBox RDP session (click from portal diagram)

– RDP Login: administrator/stgscvt

– Full screen makes it easier

• ASAs, CSRs, and test hosts are open via SuperPutty shortcut, using credentials:

– ASA console: enable password is cisco

– CSR SSH: auto-login: admin/cisco

– Linux host SSH: auto-login: user/cisco

Prep

11


Login to All Devices via SuperPutty Shortcut Once inside the Jumpbox RDP

12

Inside-host

Login: user/cisco

Outside-host

Login: user/cisco

CSR1

Login: admin/cisco

CSR2

Login: admin/cisco

Inside Jumpbox, double-

click on SuperPutty and you will connect to

all devices through an out-of-band management

network 172.16.1.0/24

If any session times out,

re-login to all by double-clicking on

ASA-CSR-ENDHOSTS link within Layouts

ASA1

Enable Passwd: cisco

ASA2

Enable Passwd: cisco

Prep


Auto-arranged & Auto-login terminals in SuperPutty In the Jumpbox Double Click on a Shortcut

13

Inside-host

(IP 10.10.140.30)

./client.iperf

Outside-host

(IP 172.16.2.44)

ping 10.10.140.30

ssh [email protected]

ASA1

show route

show conn

ASA2

show route

show conn

Reconnect via Layouts

Double-click on ASA-CSR-ENDHOSTS

Inside-host

(IP 10.10.140.30)

ping 172.16.2.44


Outside-host

(IP 172.16.2.44)

./server.iperf

CSR1

show ip route

terminal monitor

(to view log msgs)

CSR2

show ip route

terminal monitor

(to view log msgs)

Prep


Before You Start, Reset Your Switch… Refresh the POD switch:

Open browser on jumpbox PC to Home Page preset to: http://172.16.2.40/

Click on the link that says Reset to (initial state)

After 1min, Confirm successful reset as shown here

On this home page are links to bring down/up ASA ports

Open IE or Firefox Home Page inside RDP

Prep

14

http://172.16.2.40/


Tasks 1-5

15

Two IP Paths

1. Stand-alone ASAs as two equal OSPF paths for CSRs 2. Move to L3 cluster with CSR OSPF ECMP

3. Switch to IP SLA, by removing OSPF on ASA L3 cluster

One IP Path over Ether-Channel Port Bundle 4. Move to L2 cluster in Routed mode with OSPF on cluster Master

5. L2 cluster in Transparent mode where CSRs peer directly

Prep


Task Workflow Example

ASA2

ASA1

Preview – section shows an overview of items followed by detailed slides

Deploy CLI to change into new design

Review ASA and CSR configurations

Verify new topology with show outputs

Proceed to test the new design

Tests – section gives order of setup tasks needed to complete the testing

Open ping/ssh/UDP connections

Find which ASA owns connection

Down a path that owns test connections

Check for connection state recovery

Record measured convergence

CSR1 CSR2

Inside Outside

Prep

16


Asymmetric Traffic Flow without state sharing

17

Inspected or Stateful

Connections traversing ASAs

IP 1.1.2.2

IP 1.1.2.3

IP 1.1.1.2

IP 1.1.1.3

ASA2

ASA1

CSR1 CSR2

Inside host

Outside host

Task 1

Test

Conns Success

UDP PASS

ping FAIL

ssh FAIL

Ping and SSH will fail

now as forward and return path of traffic

must come to the same ASA

iPerf UDP connections

are stateless and will continue to work as both

ASAs will create an entry in the connection table.

Steps Details

1 Down ASA2

2 Open Conns

3 Up ASA2


ASA Clustering Modes

Layer 3 Adjacent

Layer 3 Adjacent

Individual Interface Mode

• Each ASA has unique IP address • Adjacent routers use routing (PBR,

OSPF, ECMP)

Cluster Control Link

Spanned Etherchannel Mode

Etherchannel

Etherchannel

• Cluster members form etherchannel • Cluster members share IP, allow NSF

Cluster Control Link

Task 2 and 3 Task 4 and 5

18


CCL via

switch

Layer3 ASA Cluster Design Router (IP routes) Load-balancing

19

ASA2

ASA1

CSR1 CSR2

Inside Outside Tw o IP paths Tw o paths

Routers Load-balance to ASAs

PBR or ECMP via OSPF, IP SLA

IP-B1 IP-A1

IP-B2 IP-A2

CSR1#sh ip route (snip) O 172.16.2.0/24 [110/12] via 1.1.1.3, 00:07:41, Gig1 [110/12] via 1.1.1.2, 00:18:25, Gig1 CSR2#sh ip route (snip) O 10.10.140.0 [110/12] via 1.1.2.3, 00:10:58, Gig1 [110/12] via 1.1.2.2, 00:11:08, Gig1

ASA 9.3 releases enabled

OSPF FastHellos, allowing faster

convergence on ASA failures.

ASA Indiv idual Interface Mode

Contexts run in Routed (IP hop)

Task 2 & 3

Protocol Success

UDP PASS

ping PASS

ssh PASS

Slave

Master

Cluster Control Link (CCL) used for:

Updating state info between ASAs

Rebalancing of asymmetric traffic


master/a/asa1(config)# sh run int Po1 interface Port-channel1

lacp max-bundle 8 slave/a/asa2(config)# sh run int Po1

interface Port-channel1 lacp max-bundle 8

master/a/asa1(config)# exec clu sh port-c summ Group Port-channel Protocol Span-cluste Ports

------+------------+--------+-----------+----- 1 Po1(U) LACP No Gi0/2(P)

slave/a/asa2(config)# sh port-channel summary 1 Po1(U) LACP No Gi0/2(P)

Layer 3 ASA Cluster – Routed Firewall Individual Interface Mode (ECMP)

20

ASA2

ASA1

CSR1 CSR2

Inside

Host Outside

Host

Po1

Po2

CCL

CCL

10.10.140.0/24 172.16.2.0/24

Task 2 & 3

Inside

VLAN 7

1.1.1.0/24 1.1.2.0/24

Po1.8

(.3)

Po1.7

(.3)

Outside

VLAN 8

Po1.8

.1 (.2)

Po1.7

.1 (.2)

Slave

Master

Lab-3750-x-switch#sh etherchannel summary

Group Port-channel Protocol Ports ------+-------------+-----------+----------

1 Po1(SU) LACP Gi1/0/9(P) 2 Po2(SU) LACP Gi1/0/14(P)

Each ASA unit peers independently

to neighbor routers and maintains its

ow n instance of the routing table.


Testing Resiliency – Task 2 & 3 Individual Interface Mode (Equal Cost Multi Path)

21

ASA2

ASA1

CSR1 CSR2

Inside

Host Outside

Host

Po1

Po2

CCL

CCL

Down UP

G0/2

Down UP

G0/2 ASA1 ASA2

Test 1: Dow n ASA data port on the sw itch

for unit that ow ns TCP/UDP conns Test 3: Disable ASA node via

cluster CLI or dow n CCL port

Down UP

G0/3 Down UP

G0/3 ASA1 ASA2

Test 2: Simulate

ASA crash w ith

‘crashinfo force page-fault’

or or

Workf low:

(1) Open test connections

(2) Determine the connection owner

(3) Proceed to f ail the owner ASA

(4) Measure conv ergence

(5) Recov er down ASA


Locating Owner ASA

ASA1

!master/a/admin(config)#

changeto context admin

cluster exec show conn

asa1(LOCAL):**********************************************************

7 in use, 18 most used

Cluster stub connections: 1 in use, 2 most used

UDP outside 172.16.2.44:5001 inside 10.10.140.30:38842, idle 0:00:00, bytes 883470, flags -

TCP outside 172.16.2.44:55505 inside 10.10.140.30:22, idle 0:01:01, bytes 0, flags y

asa2:*****************************************************************



TCP outside 172.16.2.44:55505 inside 10.10.140.30:22, idle 0:01:01, bytes 3910, flags UIOB

UDP outside 172.16.2.44:5001 inside 10.10.140.30:38842, idle 0:00:00, bytes 0, flags –Y

master/a/admin(config)# Y flag means stub or backup conn

If UDP and TCP conns are on different ASAs,

pick ASA with UDP conn as owner, and

proceed to test.

Active TCP connection

Active UDP connection

22


Measuring Convergence

ASA detects that owner unit

went down

Count (–nan%) UDP packets that were lost,

and record in your convergence table

Count the missed PINGs

Protocol Lost

Pkts/Secs

ping 9 (322-330)

UDP iPerf 9 (326-334)

ssh N/A

23


Layer 2 ASA Cluster Design Switch (Ether-channel) Load-balancing

24

IP-B1 IP-A1

ASA2

ASA1

CSR1 CSR2

Inside Outside

CCL

Switch(s) load-balance traf f ic to ASAs

using Ether-Channel

C3750-X switch is used in this lab

One IP path over

Ether-Channel Interface.

CSR2# sh ip route

(snip) Gateway of last resort is 172.16.2.1 to network 0.0.0.0

O 10.10.140.0 [110/12] via 1.1.2.1, 00:21:20, Gig1

Switch

ASA Spanned Cluster Mode ASA Context can run as Routed (IP hop) or

Transparent (Bridging VLANs) firewall.

* In Transparent, routers connect directly

Task 4 & 5

The latest ASA releases enabled

Non-Stop Forwarding, convergence

on ASA failures.

Only the Master ASA unit peers to

neighboring routers and sync the

routing table to all Slave ASA units.


master/a/asa1(config)# sh port-channel summary Group Port-channel Protocol Span-cluster Ports

-----+------------+--------+------------+------ 2 Po2(U) LACP Yes Gi0/0(P)

Gi0/1(P)

Layer 2 ASA Cluster – Routed Firewall Spanned Interface (Ether-channel)

25

ASA2

ASA1

CSR1 CSR2

Inside

Host Outside

Host

Po4

Po4

CCL

CCL

10.10.140.0/24 172.16.2.0/24

Task 4

Slave

1.1.1.0/24 1.1.2.0/24

Po4.8

.1

Po4.7

.1

Outside

VLAN 8

Inside

VLAN 7

Master

.200 .200

Lab-3750-x#sh etherchannel summary Group Port-channel Protocol Ports ------+-------------+-----------+----------------

1 Po4(SU) LACP Gi1/0/7(P) Gi1/0/8(P) Gi1/0/12(P) Gi1/0/13(P)


Layer 2 ASA Cluster– Transparent Firewall Spanned Interface (Ether-channel)

26

ASA2

ASA1

CSR1 CSR2

Inside

Host Outside

Host

Po4

Po4

CCL

CCL

10.10.140.0/24

1.1.1.200/16 1.1.2.200/16

172.16.2.0/24

Task 5

Slave

Po4.8

BVI1

Po4.7

BVI1

Outside

VLAN 8

Inside

VLAN 7

Master

CSR1#sh ip route ospf Gateway of last resort is 1.1.2.200 to network 0.0.0.0 O*E2 0.0.0.0/0 [110/1] via 1.1.2.200, 00:00:15, Gig1 172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks O 172.16.2.0/24 [110/2] via 1.1.2.200, 00:00:15, Gig1

master/a/asa1(config)# sh run interface bvi1 interface BVI1 ip address 1.1.1.1 255.255.0.0 master/a/asa1/admin# sh mac-address-table interface mac address type Age(min) bridge-group --------------------------------------------------------- outside 0050.56bf.dbc2 dynamic 1 1 inside 0050.56bf.34b8 dynamic 5 1


Testing Resiliency – Task 4 & 5 Spanned Interface Mode (Ether-channel)

27

ASA2

ASA1

CSR1 CSR2

Inside

Host Outside

Host

Down UP

Test 2: Simulate

ASA crash w ith


Po4

G0/0

Po4

Down UP

G0/0

CCL

CCL

ASA1 ASA2

Test 1A: Dow n 1st ASA port on the sw itch

for unit that ow ns TCP/UDP conns Test 1B: Dow n 2nd ASA port on

Sw itch (w orst-case scenario)

Test 3: Disable ASA node via


Down UP

G0/1 Down UP

G0/1 ASA1 ASA2

Down UP

G0/3

Down UP

G0/3 ASA1 ASA2

or or

or

Workf low:

(1) Open test connections


(3) Proceed to f ail the owner ASA

(4) Measure conv ergence

(5) Recov er down ASA


Task 1: Stand-alone ASAs IP 1.1.2.2

IP 1.1.2.3

IP 1.1.1.2

IP 1.1.1.3

ASA2

ASA1

Preview

Familiarize yourself with POD topology and configurations

CSR1 and CSR2 load-balancing via OSPF

Two paths provided by ASA1 and ASA2, stand-alone firewalls NOT in failover or cluster

Verify OSPF routes on CSR1 to outside

Verify OSPF routes on CSR2 to inside

Tests

Down ASA2

Attempt connections between hosts

Bring up downed ASA2

Check if connections are still active

Attempt connections with two ASA active

CSR1 CSR2

Tw o paths Tw o paths Interna

l External

Task 1

28


.1

Master

Internal

10.10.140.0/24

External .200

CSR1 CSR2

ASA2

ASA1

Inside

VLAN 7

VLAN 15 VLAN 4

gig2 gig1 gig1 gig2

.200

1.1.1.0/24 1.1.2.0/24

.200

Po1.8

(.2)

Po2.8

(.3)

Po1.7

(.2)

Po2.7

(.3)

Stand-alone ASAs Diagram

172.16.2.0/24

Inside host

Outside host

.30 .44

Task 1

Outside

VLAN 8

29


Verify CSR1 and CSR2 routes to two next-hop ASAs

!CSR1 OSPF routes

!CSR1#

sh ip route ospf

(snip)

Gateway of last resort is 1.1.1.3 to network 0.0.0.0

O*E2 0.0.0.0/0 [110/1] via 1.1.1.3, 00:00:28, GigabitEthernet1

[110/1] via 1.1.1.2, 00:10:23, GigabitEthernet1

1.0.0.0/8 is variably subnetted, 3 subnets, 2 masks

O 1.1.2.0/24 [110/11] via 1.1.1.3, 00:00:28, GigabitEthernet1







CSR1#

CSR1

!CSR2 OSPF routes

!CSR2#

sh ip route ospf

(snip)


O*E2 0.0.0.0/0 [110/1] via 172.16.2.1, 3d14h, GigabitEthernet2




10.0.0.0/24 is subnetted, 1 subnets

O 10.10.140.0 [110/12] via 1.1.2.3, 00:01:02, GigabitEthernet1



O 172.16.3.1/32 [110/2] via 172.16.2.1, 3d14h, GigabitEthernet2

CSR2#

CSR2

Task 1

30


Verify ASA1 and ASA2 routes to CSRs

!changeto context admin to show OSPF routes

!asa1#


!asa1/admin#

sh route


C 1.1.1.0 255.255.255.0 is directly connected, inside

C 1.1.2.0 255.255.255.0 is directly connected, outside

C 172.16.1.0 255.255.255.0 is directly connected, mgmt

O 172.16.3.1 255.255.255.255 [110/12] via 1.1.2.200, 1:35:11, outside

O 172.16.2.0 255.255.255.0 [110/11] via 1.1.2.200, 1:35:11, outside

O 10.10.140.0 255.255.255.0 [110/11] via 1.1.1.200, 1:35:11, inside

O*E2 0.0.0.0 0.0.0.0 [110/1] via 1.1.2.200, 1:35:11, outside

asa1/admin#

ASA1

!asa2#


!asa2/admin#

sh route





O 172.16.3.1 255.255.255.255 [110/12] via 1.1.2.200, 1:35:58,

outside

O 172.16.2.0 255.255.255.0 [110/11] via 1.1.2.200, 1:35:58, outside

O 10.10.140.0 255.255.255.0 [110/11] via 1.1.1.200, 1:35:58, inside

O*E2 0.0.0.0 0.0.0.0 [110/1] via 1.1.2.200, 1:35:58, outside

asa2/admin#

ASA2

Task 1

31


Remove ASA2 Path Remove 2nd path b/t CSRs

Open IE/Firefox inside RDP

To shutdown ASA2 ports on the switch, use browser home page on jumpbox PC,

pointing to link: http://172.16.2.40/

Task 1

Shut down ASA2 data ports on Switch

Disable ASA2 G0/2 port

Disable ASA2 G0/3 port

32


Verify CSR1 and CSR2 routes to one ASA

!CSR1 OSPF routes

!CSR1#

sh ip route ospf

(snip)








CSR1#

CSR1

!CSR2 OSPF routes

!CSR2#

sh ip route ospf

(snip)









CSR2#

CSR2

One path between CSRs

Task 1

33


Modify iPerf Run Time Allow iPerf to run UDP throughout

duration of your lab

#user@lubuntu:~$

cat client.iperf

iperf -u –t 260 -i 1 -c 172.16.2.44 -b 0.0941m

user@lubuntu:~$

#Change –t flag to 20000, to allow iPerf to send for 4 hours

#You can use your favorite UNIX editor installed, vi or pico

#This will allow you to run UDP traffic throughout duration of the lab

pico client.iperf

#Change to: -t 20000

#user@lubuntu:~$

cat client.iperf

iperf -u –t 20000 -i 1 -c 172.16.2.44 -b 0.0941m

user@lubuntu:~$

InsideHost

Task 1

Change iPerf –t flag to from 260 to 20000

iperf –help

(snip)

-t, --time n time in seconds to transmit for

(default 10 secs)

34


Setup Test Connections

iPerf UDP packets sending from Inside to Outside Host

35

Inside-host

(IP 10.10.140.30)

./client.iperf

Outside-host

(IP 172.16.2.44)


(passwd: cisco)

Inside-host

(IP 10.10.140.30)

ping 172.16.2.44

Outside-host

(IP 172.16.2.44)

./server.iperf

Task 1

Ping Inside to Outside and SSH Outside to Inside


Setup Test Conns Cont… Ping from inside to outside linux

#On top left terminal, ping to outside-lnx

#user@inside-lnx:~$

ping 172.16.2.44

PING 172.16.2.44 (172.16.2.44) 56(84) bytes of data.

64 bytes from 172.16.2.44: icmp_req=1 ttl=62 time=1.61 ms


#on bottom left terminal, start a 4min iperf UDP connection to outside -lnx

#user@inside-lnx:~$

./client.iperf

------------------------------------------------------------

Client connecting to 172.16.2.44, UDP port 5001

Sending 1470 byte datagrams

UDP buffer size: 112 KByte (default)

------------------------------------------------------------

[ 3] local 10.10.140.30 port 46611 connected with 172.16.2.44 port 5001

[ ID] Interval Transfer Bandwidth

[ 3] 0.0- 1.0 sec 12.9 KBytes 106 Kbits/sec

[ 3] 1.0- 2.0 sec 11.5 KBytes 94.1 Kbits/sec

InsideHost

#On top right terminal, Server listens and receives client UDP traffic

#user@outside-lnx:~$

./server.iperf

------------------------------------------------------------

Server listening on UDP port 5001

Receiving 1470 byte datagrams


------------------------------------------------------------


[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams

[ 3] 0.0- 1.0 sec 11.5 KBytes 94.1 Kbits/sec 0.075 ms 0/ 8 (0%)


[ 3] 0.0- 2.5 sec 28.7 KBytes 94.1 Kbits/sec 0.083 ms 0/ 20 (0% )

### When server is not receiving packets, output will show (-nan%)

### You can count the number of seconds server could not receive packets

[ 3] 21.0-22.0 sec 0.00 Bytes 0.00 bits/sec 0.067 ms 0/ 0 (-nan%)



OutsideHost

Start iPerf UDP flow

Verify if you can ping.

Verify you can receive UDP.

Task 1

36


Setup Test Conn Cont… Ssh from outside to inside linux

#On bottom right terminal, open ssh connection outside to inside

user@outside-lnx:~$

ssh -l user 10.10.140.30

[email protected]'s password:

(snip)

Last login: Tue Nov 26 14:44:35 2013 from 172.16.2.44

user@inside-lnx:~$

OutsideHost

Verify you can ssh b/t hosts

Task 1

#If this session locks up, it should drop out within 5min w/ error

user@lubuntu:~$ Write failed: Broken pipe

user@lubuntu:~$

#You can kill it by typing ‘~.’ w/ no single quotes

#Then re-open it

user@outside-lnx:~$

ssh -l user 10.10.140.30


(snip)


user@inside-lnx:~$

OutsideHost

37


Re-enable ASA2 Enable 2nd path b/t CSRs

Open IE link inside RDP

To shutdown ASA1 or ASA2 ports on the switch, use browser on jumpbox PC and

open link: http://172.16.2.40/

This will add asymmetry of traffic through ASAs

ASA2

ASA1

CSR1 CSR2

Tw o paths Tw o paths

Ping and SSH will fail

now as forward and return path of traffic

must come to the same ASA

iPerf UDP connections

are stateless and will continue to work as both

ASAs will create an entry in the connection table.

Task 1

Enable ASA2 G0/2

Enable ASA2 G0/3

38


Verify CSR1 and CSR2 routes to two ASAs

!CSR1 OSPF routes

!CSR1#

sh ip route ospf

(snip)












CSR1#

CSR1

!CSR2 OSPF routes

CSR2#

sh ip route ospf

(snip)











CSR2#

CSR2

Task 1

39


Verify Test Connections When traffic goes through two ASA not in a cluster…

40

Inside-host

(IP 10.10.140.30)

Here we just send

packets…

Outside-host

(IP 172.16.2.44)

ssh session stil l working?

Inside-host

(IP 10.10.140.30)

ping stil l working?

Outside-host

(IP 172.16.2.44)

UDP traffic sti l l being

received?

Task 1

…what traffic is not able to pass these stateful devices?

Protocol Task 1

Pass / Fail

ping

UDP iPerf

ssh


Task 2: L3 Cluster in OSPF IP 1.1.2.2

IP 1.1.2.3

IP 1.1.1.2

IP 1.1.1.3

ASA2

ASA1 Preview

Form individual interface mode or L3 cluster

Clear both ASA1 and ASA2 configurations

Copy task2-system.cfg to ASA1 and watch it become a master

Enter configuration on ASA2 slave via CLI and watch it detect and sync config from master

CSR1/CSR2 are still load-balancing via OSPF

Two paths provided by ASA1 and ASA2, now maintain state as L3/Individual cluster

Verify OSPF routes on CSR1 to outside

Verify OSPF routes on CSR2 to inside

Tests

Open connections through cluster

Down ASA that owns the connection using one of four failure scenarios

Check if any connections become responsive

Measure Convergence of connections

Bring ports back Up and enable down ASA

CSR1 CSR2

CCL

Interna

l External

Task 2

41


.1

Master

Slave

Internal

10.10.140.0/24

External .200

CSR1 CSR2

ASA2

ASA1

Inside

VLAN 7

VLAN 15 VLAN 4

gig2 gig1 gig1 gig2

.200

1.1.1.0/24 1.1.2.0/24

.200

Po1.8

.1 (.2)

Po2.8

(.3)

Po1.7

.1 (.2)

Po2.7

(.3)

2.2.2.0/24

G0/3

.1

G0/3

.2

Individual Cluster Diagram

Inside_pool

1.1.1.2-1.1.1.10

Outside_pool

1.1.2.2-1.1.2.10

172.16.2.0/24

Each ASA node has a unique IP on inside and

outside VLANs.

mgmt_pool

172.16.1.2-172.16.1.10

Master

CCL VLAN 25

Inside host

Outside host

.30 .44

Task 2

Outside

VLAN 8

42


Enable the Cluster ASA1 is master, ASA2 is slave in Individual mode cluster

!Feedback from ASA2 after enabling

asa2/a#(config)#ena

ClusterDisabled/a(cfg-cluster)#

Detected Cluster Master.

INFO: UC-IME is enabled, issuing 0 free TLS licenses for UC-IME

Beginning configuration replication from Master.

WARNING: Removing all contexts in the system

Removing context 'admin' (7)... Done

INFO: Admin context is required to get the interfaces

*** Output from config line 64, "arp timeout 14400"

INFO: Admin context is required to get the interfaces

*** Output from config line 65, "no arp permit-nonconnect..."

Creating context 'admin'... Done. (8)

*** Output from config line 68, "admin-context admin"

WARNING: Skip fetching the URL disk0:/a.cfg

Cryptochecksum (changed): 0e8178ab 18e3d553 aabeee98 f2192418

End configuration replication from Master.

Cluster unit asa2 transitioned from DISABLED to SLAVE

!Must enable and change to system context

changeto system

config terminal

!Clear configuration on ASA1

clear config all

sh cluster interface-mode

no cluster interface-mode

!Force the change to individual mode

cluster interface-mode individual force

copy /noconfirm milan/task2-admin.cfg task2-admin.cfg

copy /noconfirm milan/task2-system.cfg running-config

!If prompted, you MUST confirm Y for YES, remove these commands

1952 bytes copied in 5.220 secs (390 bytes/sec)

ClusterDisabled/a/asa1(config)#

!Now wait 1 min for ASA1 to become Master through election process

!Cluster unit asa1 transitioned from DISABLED to MASTER

!Save configuration on Master

write memory all

ASA1

!In system context clear cfg, enable cluster mode, and apply ASA2 cfg changeto system

config terminal

clear config all

no cluster interface-mode

cluster interface-mode individual force

!Bring up interface for CCL

interface GigabitEthernet0/3

no shut

!Define cluster group

cluster group fw

local-unit asa2

cluster-interface GigabitEthernet0/3 ip 2.2.2.2 255.255.255.0

priority 20

console-replicate

health-check holdtime 3

clacp system-mac auto system-priority 1

enable

! ASA2 will detect the Master, sync config, and become a Slave unit

!Detected Cluster Master.

!Cluster unit asa2 transitioned from DISABLED to SLAVE

!Save configuration on Slave

write memory all

ASA2

Task 2

43


Review and Verify ASA nodes in cluster and OSPF routes

!

!ASA1 is Master and ASA2 is Slave

!master/a/asa1(config)#

sh cluster inf

Cluster fw: On

Interface mode: individual

This is "asa1" in state MASTER

ID : 0

Version : 9.3(2)

Serial No.: FCH16107JEN

CCL IP : 2.2.2.1

CCL MAC : 5057.a8e1.48a4

(snip)

Other members in the cluster:

Unit "asa2" in state SLAVE

ID : 1

Version : 9.3(2)

Serial No.: FCH16107JG9

CCL IP : 2.2.2.2

CCL MAC : c464.1339.9b07

(snip)

!master/a/asa1(config)#

ASA1

!Verify OSPF relationships to CSRs from admin context



sh run router

!Verify configuration Output

router ospf 1

network 1.1.1.0 255.255.255.0 area 0

network 1.1.2.0 255.255.255.0 area 0

timers pacing lsa-group 10

timers throttle spf 100 200 1000

log-adj-changes

!


sh route ospf

(snip)


O*E2 0.0.0.0 0.0.0.0 [110/1] via 1.1.2.200, 00:23:06, outside

O 10.10.140.0 255.255.255.0 [110/11] via 1.1.1.200, 00:23:06, inside

O 172.16.2.0 255.255.255.0 [110/11] via 1.1.2.200, 00:23:06, outside

O 172.16.3.1 255.255.255.255 [110/12] via 1.1.2.200, 00:23:06, outside

ASA1

Task 2

44


Verify CSR1 and CSR2 routes to two ASAs

!CSR1 OSPF routes

!CSR1#

sh ip route ospf

(snip)












CSR1#

CSR1

!CSR2 OSPF routes

!CSR2#

sh ip route ospf

(snip)











CSR2#

CSR2

Task 2

45




46

Inside-host

(IP 10.10.140.30)

./client.iperf

Outside-host

(IP 172.16.2.44)


(passwd: cisco)

Inside-host

(IP 10.10.140.30)

ping 172.16.2.44

Outside-host

(IP 172.16.2.44)

./server.iperf

Task 2

Ping Inside to Outside and SSH Outside to Inside


Setup Test Conns Output Ping from inside to outside linux

#In first terminal, watch the ping to OutsideHost

#user@inside-lnx:~$

ping 172.16.2.44

PING 172.16.2.44 (172.16.2.44) 56(84) bytes of data.



#In second terminal, start iperf UDP connection to OutsideHost

user@inside-lnx:~$

./client.iperf

------------------------------------------------------------

Client connecting to 172.16.2.44, UDP port 5001

Sending 1470 byte datagrams


------------------------------------------------------------


[ ID] Interval Transfer Bandwidth

[ 3] 0.0- 1.0 sec 12.9 KBytes 106 Kbits/sec

[ 3] 1.0- 2.0 sec 11.5 KBytes 94.1 Kbits/sec

InsideHost

#user@outside-lnx:~$

./server.iperf

------------------------------------------------------------

Server listening on UDP port 5001

Receiving 1470 byte datagrams


------------------------------------------------------------


[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams



[ 3] 0.0- 2.5 sec 28.7 KBytes 94.1 Kbits/sec 0.083 ms 0/ 20 (0% )

### Again, when server is not receiving packets, output will show (-nan%)

### You can count the number of seconds server could not receive packets [ 3] 21.0-22.0 sec 0.00 Bytes 0.00 bits/sec 0.067 ms 0/ 0 (-nan%) [ 3] 22.0-23.0 sec 0.00 Bytes 0.00 bits/sec 0.067 ms 0/ 0 (-nan%) [ 3] 23.0-24.0 sec 0.00 Bytes 0.00 bits/sec 0.067 ms 0/ 0 (-nan%)

OutsideHost

Task 2

Start iPerf UDP flow

47


Setup Test Conns Ssh from outside to inside linux

!user@outside-lnx:~$

ssh -l user 10.10.140.30

[email protected]'s password: (cisco is the password)

(snip)


user@inside-lnx:~$

# This will serve to measure how long it takes for TCP connection to recover

# Enter a single character on this session during convergence to notice when session recovers

# If you enter more output on this session, TCP backoff mechanism will

OutsideHost

Task 2

48


Locate Owner ASA Locate conn owner ASA

ASA1

You will next down ASA that owns most connections

Task 2



cluster exec sh conn

asa1(LOCAL):**********************************************************





asa2:*****************************************************************








proceed to test.



49


Testing Resiliency Summary Individual Mode (ECMP) – Proceed to next slide for detailed instructions

50

ASA2

ASA1

CSR1 CSR2

Inside

Host Outside

Host

Po1

Po2

CCL

CCL

Down UP

G0/2

Down UP

G0/2 ASA1 ASA2

Test 1: Dow n 1st ASA port on the sw itch



Down UP

G0/3 Down UP

G0/3 ASA1 ASA2

Test 2: Simulate

ASA crash w ith


or or


(2) Shut down the port on owner ASA

Task 2


Verify Test Connections are up

51

Inside-host

(IP 10.10.140.30)

Stil l sending packets…

./client.iperf

Outside-host

(IP 172.16.2.44)

ssh session sti l l working?

ssh -l user 10.10.140.30

Inside-host

(IP 10.10.140.30)


Ping 172.16.2.44

Outside-host

(IP 172.16.2.44)

UDP packets arriving?

./server.iperf

Task 2

Measure connection convergence of each test: 1A, 1B, 2, and 3…

…after locating ASA unit that owns your connections.


Test 1 Remove the data port on owner ASA


To shutdown ASA2 ports on the switch, use browser home page on jumpbox PC, pointing to link: http://172.16.2.40/

Task 2

Observe and record if any packets were lost and

if there was any impact on SSH session

Protocol

Task 1

Lost

Pkts/Secs

ping

UDP iPerf

ssh

Disable ASA G0/2 port

52


Measure Count how many UDP packets you lost Task 2

Count how many ping packets were lost

Compare PING req counts to find lost pkt count



53


Recover Down ASA Up or ‘no shut’ G0/2 port on down ASA Task 2

Enable cluster config on down ASA to add it the

cluster immediately

Enable ASA G0/2 port

! Re-join approriate ASA unit

changeto system

config terminal


cluster group fw

enable

!Wait for ASA to detect master, finish sync, and become a Slave unit


Down ASA

Down ASA may retry to join after 5min on its

own, but will only transition to SLAVE

after G0/2 is enabled

54



55

Inside-host

(IP 10.10.140.30)

Restart if needed…

./client.iperf

Outside-host

(IP 172.16.2.44)


ssh -l user 10.10.140.30

Inside-host

(IP 10.10.140.30)


Ping 172.16.2.44

Outside-host

(IP 172.16.2.44)


./server.iperf

Task 2





ASA1


Task 2




asa1(LOCAL):**********************************************************





asa2:*****************************************************************








proceed to test.



56


Test 2 Simulate a crash on owner ASA Task 2



Protocol

Task 2

Lost

Pkts/Secs

ping

UDP iPerf

ssh

Simulate crash on owner ASA Crash owner

ASA w/ CLI

! Write configs and simulate ASA crash

write memory all

crashinfo force page-fault

!ASA will boot, detect master, perform sync, and become a

Slave unit


Owner ASA

57





went down



58


Crashed ASA Re-joins After reboot, unit rejoins cluster Task 2

Detects master, syncs config, and becomes a

slave unit

59



60

Inside-host

(IP 10.10.140.30)

Restart if needed

./client.iperf

Outside-host

(IP 172.16.2.44)


ssh -l user 10.10.140.30

Inside-host

(IP 10.10.140.30)


Ping 172.16.2.44

Outside-host

(IP 172.16.2.44)


./server.iperf

Task 2





ASA1


Task 2




asa1(LOCAL):**********************************************************





asa2:*****************************************************************








proceed to test.



61


Let’s try shorter dead-intervals ASA1 and ASA2 routes to CSRs

!change spf dead-interval from 30sec to 3sec

!CSR1#

interface GigabitEthernet1

ip ospf dead-interval 3

CSR1


!CSR1#


ip ospf dead-interval 3

CSR2

Task 2


!master/a/asa1/admin(config)#


interface inside

ospf dead-interval 3

!

interface outside

ospf dead-interval 3

ASA Master

!Verify OSPF routes


sh route ospf

(snip)


O*E2 0.0.0.0 0.0.0.0 [110/1] via 1.1.2.200, 00:15:19, outside

O 10.10.140.0 255.255.255.0 [110/11] via 1.1.1.200, 00:14:37, inside

O 172.16.2.0 255.255.255.0 [110/11] via 1.1.2.200, 00:15:19, outside

O 172.16.3.1 255.255.255.255 [110/12] via 1.1.2.200, 00:15:19, outside

ASA Master

62


Test 3 Shutdown the CCL port on owner ASA Task 2



Protocol

Task 3

Lost

Pkts/Secs

ping

UDP iPerf

ssh

Disable ASA CCL port

63





went down




64


Recover Down ASA Up the CCL port on down ASA Task 2

Enable cluster group to immediately add ASA to

the cluster

Enable ASA CCL port


changeto system

config terminal


cluster group fw

enable



Down ASA

65


Task 3: L3 Cluster in IP SLA IP 1.1.2.2

IP 1.1.2.3

IP 1.1.1.2

IP 1.1.1.3

ASA2

ASA1

Preview

Stay in L3 or individual interface mode and proceed to applying Task 3 CLI.

Remove OSPF config on ASA master only

Check IP SLA configs on CSRs

CSR1 and CSR2 still load-balancing but now via IP SLA tracks

Two paths still there with ASA1 and ASA2, still maintain state as L3/Individual cluster

Verify IP SLA routes on CSR1 to outside

Verify IP SLA routes on CSR2 to inside

Tests

Open test connections through cluster

Down ASA that owns the connection

Check when the connection state active

Measure convergence

CSR1 CSR2

CCL

Interna

l External

Task 3

66


.1

Master

Slave

Internal

10.10.140.0/24

External .200

CSR1 CSR2

ASA2

ASA1

Inside

VLAN 7

Outside

VLAN 8

VLAN 15 VLAN 4

gig2 gig1 gig1 gig2

.200

1.1.1.0/24 1.1.2.0/24

.200

Po1.8

.1 (.2)

Po2.8

(.3)

Po1.7

.1 (.2)

Po2.7

(.3)

2.2.2.0/24

G0/3

.1

G0/3

.2

Individual Cluster Diagram

Inside_pool

1.1.1.2-1.1.1.10

Outside_pool

1.1.2.2-1.1.2.10

172.16.2.0/24

Each ASA node has a unique IP on inside and

outside VLANs.

mgmt_pool

172.16.1.2-172.16.1.10

Master

CCL VLAN 25

Inside host

Outside host

.30 .44

Task 3

67


Verify CSR1 and CSR2 OSPF routes to two ASAs

!CSR1 OSPF routes

!CSR1#

sh ip route ospf

(snip)












CSR1#

CSR1

!CSR2 OSPF routes

!CSR2#

sh ip route ospf

(snip)











CSR2#

CSR2

Task 3

68


CLI On Master ASA remove OSPF

!

!ASA1 in this case is Master

master/a/asa1(config)# sh clu inf | i state

This is "asa1" in state MASTER


master/a/asa1(config)#

!Change to admin context



sh run router

router ospf 1

network 1.1.1.0 255.255.255.0 area 0

network 1.1.2.0 255.255.255.0 area 0

timers spf 1 1

timers lsa-group-pacing 1

log-adj-changes

master/a/admin(config)#


no router ospf 1

ASA Master

!Verify master routing relationships to host networks


show route





S 10.10.140.0 255.255.255.0 [200/0] via 1.1.1.200, inside

S* 0.0.0.0 0.0.0.0 [200/0] via 1.1.2.200, outside

master/a/asa1/admin(config)#

!ASA2 Slave

!slave/a/asa2/admin(config)#

sh route





S 10.10.140.0 255.255.255.0 [200/0] via 1.1.1.200, inside

S* 0.0.0.0 0.0.0.0 [200/0] via 1.1.2.200, outside

slave/a/asa2/admin(config)#

ASA Master

Task 3

69


Verify CSR1 and CSR2 static routes to two ASAs

!CSR1 IP SLA routes

!CSR1#

sh ip route

(snip)


S* 0.0.0.0/0 [200/0] via 1.1.1.3

[200/0] via 1.1.1.2


C 1.1.1.0/24 is directly connected, GigabitEthernet1

L 1.1.1.200/32 is directly connected, GigabitEthernet1




CSR1#

CSR1

!CSR2 IP SLA routes

!CSR2#

sh ip route

(snip)




S 1.1.1.0/24 [200/0] via 1.1.2.3

[200/0] via 1.1.2.2




S 10.10.140.0 [200/0] via 1.1.2.3

[200/0] via 1.1.2.2





CSR2#

CSR2

Task 3

70



71

Inside-host

(IP 10.10.140.30)

Restart if needed

./client.iperf

Outside-host

(IP 172.16.2.44)


ssh -l user 10.10.140.30

Inside-host

(IP 10.10.140.30)


Ping 172.16.2.44

Outside-host

(IP 172.16.2.44)


./server.iperf

Task 3







asa1(LOCAL):**********************************************************




TCP outside 172.16.2.44:58952 inside 10.10.140.30:22, idle 0:02:39, bytes 0, flags Y

asa2:*****************************************************************




UDP outside 172.16.2.44:5001 inside 10.10.140.30:60810, idle 0:00:14, bytes 0, flags -Y


ASA1

You will then do test 1 with this owner ASA

Task 3

72


Testing Resiliency of ASA Cluster Designs Individual Mode (ECMP)

73

ASA2

ASA1

CSR1 CSR2

Inside

Host Outside

Host

Po1

Po2

CCL

CCL

Down UP

G0/2

Down UP

G0/2 ASA1 ASA2

Test 1: Dow n 1st ASA port on the sw itch



Down UP

G0/3 Down UP

G0/3 ASA1 ASA2

Test 2: Simulate

ASA crash w ith


or or



Task 3


Test 1 Remove the data port on owner ASA



Task 3



Protocol

Task 1

Lost

Pkts/Secs

ping

UDP iPerf

ssh


74





went down




75


Verify CSR1 and CSR2 have one route to ASA

!CSR1 IP SLA routes

!CSR1#

sh ip route

(snip)


S* 0.0.0.0/0 [200/0] via 1.1.1.3







CSR1#

CSR1

!CSR2 IP SLA routes

!CSR2#

sh ip route

(snip)




S 1.1.1.0/24 [200/0] via 1.1.2.3




S 10.10.140.0 [200/0] via 1.1.2.3





CSR2#

CSR2

Task 3

76


Recover Down ASA Up or ‘no shut’ G0/2 port on down ASA Task 3


cluster immediately

Enable ASA G0/2 port


changeto system

config terminal


cluster group fw

enable



Down ASA

77





asa1(LOCAL):**********************************************************





asa2:*****************************************************************






ASA1


Task 3

78


Test 2 Simulate a crash on owner ASA Task 3



Protocol

Task 2

Lost

Pkts/Secs

ping

UDP iPerf

ssh

Simulate crash on owner ASA Crash owner

ASA w/ CLI


changeto system

write memory all


!Wait for ASA to boot up, detect master, finish sync, and become a Slave unit


Owner ASA

79




ASA crashes




80


Crashed ASA Re-joins After reboot, unit rejoins cluster Task 3

Detects master, syncs config, and becomes a

slave unit

81



82

Inside-host

(IP 10.10.140.30)

Restart if needed

./client.iperf

Outside-host

(IP 172.16.2.44)


ssh -l user 10.10.140.30

Inside-host

(IP 10.10.140.30)


Ping 172.16.2.44

Outside-host

(IP 172.16.2.44)


./server.iperf

Task 3







asa1(LOCAL):**********************************************************





asa2:*****************************************************************






ASA1


Task 3

83


Test 3 Shutdown the CCL port on owner ASA Task 3



Protocol

Task 3

Lost

Pkts/Secs

ping

UDP iPerf

ssh

Disable ASA CCL port

84




ASA switches to Master role




85


Recover Down ASA Up or ‘no shut’ CCL port on down ASA Task 3


cluster immediately

Enable ASA CCL port


changeto system

config terminal


cluster group fw

enable



Down ASA

86


Task 3 Bonus*: Add PAT IP 1.1.2.2

IP 1.1.2.3

IP 1.1.1.2

IP 1.1.1.3

ASA2

ASA1

Preview

This is a bonus task that involves ASA and CSR configuration changes.

Add Port Address Translation to outside interface of ASA L3 cluster with IP SLA.

Add equal cost routes for new PAT network on CSR2.

Verify IP SLA routes on CSR2 for PAT pool network

Tests

Open a ssh connection through cluster


Check when connection state is active

No need to reopen the connection

CSR1 CSR2

CCL

Interna

l External

Task 3*

(optional)

87


CLI ASA PAT config

!ASA master

master/a/asa1(config)#


config terminal

object network pat-ips

range 1.1.3.2 1.1.3.3

object network inside-network

subnet 10.10.140.0 255.255.255.0

!


nat (inside,outside) dynamic pat-pool pat-ips

ASA Master

!CSR2

config terminal

ip route 1.1.3.0 255.255.255.0 1.1.2.2 200 track 1

ip route 1.1.3.0 255.255.255.0 1.1.2.3 200 track 2

CSR2# show ip route

(snip)

S 1.1.3.0/24 [200/0] via 1.1.2.3

[200/0] via 1.1.2.2

CSR2 CSR2 routes to PAT network on ASA

!Must add routed on outside linux to new network

sudo route add -net 1.1.3.0/24 gw 172.16.2.200

[sudo] password for user: cisco

user@lubuntu:~$

OutsideHost

Task 3*

88


Setup Test Connections with Xlates


89

Inside-host

(IP 10.10.140.30)

./client.iperf

Outside-host

(IP 172.16.2.44)

Can not go to inside now

without a static NAT

Inside-host

(IP 10.10.140.30)

ping 172.16.2.44 or

Ssh [email protected]

Outside-host

(IP 172.16.2.44)

./server.iperf

Task 3*

Ping and SSH Inside to Outside


Verify translations Show conns and xlates on ASA cluster

! You can also try ‘show conn detail’ to decode the flags




asa2(LOCAL):**********************************************************


UDP outside 172.16.2.44:5001 inside 10.10.140.30:49741, idle 0:00:00,

bytes 2072700, flags -

ICMP outside 172.16.2.44:0 inside 10.10.140.30:6300, idle 0:00:00, bytes 5432, flags

asa1:*****************************************************************


TCP outside 172.16.2.44:22 inside 10.10.140.30:41221, idle 0:00:29,

bytes 5286, flags UxIO

UDP outside 172.16.2.44:5001 inside 10.10.140.30:49741, idle 0:00:00,

bytes 0, flags –Y

ICMP outside 172.16.2.44:0 inside 10.10.140.30:6300, idle 0:00:00, bytes 0, flags Y

ASA Master


cluster exec sh xlate

asa2(LOCAL):**********************************************************

TCP PAT from inside:10.10.140.30/41221 to outside:1.1.3.2/41221 flags

ri idle 0:00:11 timeout 0:00:30

UDP PAT from inside:10.10.140.30/49741 to outside:1.1.3.3/49741 flags ri idle 0:01:56 timeout 0:00:30

ICMP PAT from inside:10.10.140.30/6300 to outside:1.1.3.3/6300 flags ri idle 0:00:36 timeout 0:00:30

asa1:*****************************************************************

TCP PAT from inside:10.10.140.30/41221 to outside:1.1.3.2/41221 flags


UDP PAT from inside:10.10.140.30/49741 to outside:1.1.3.3/49741 flags


ICMP PAT from inside:10.10.140.30/6300 to outside:1.1.3.3/6300 flags



ASA Master

Task 3*

90


Remove PAT Remove PAT and route configs for now


config terminal


no nat (inside,outside) dynamic pat-pool pat-ips

write memory

ASA Master

Later in spanned, you will again add new PAT config

CSR2

config terminal

no ip route 1.1.3.0 255.255.255.0 1.1.2.2 200 track 1

no ip route 1.1.3.0 255.255.255.0 1.1.2.3 200 track 2

exit

Task 3*

91


Task 4: L2 Cluster in Routed IP 1.1.2.1 IP 1.1.1.1

ASA2

ASA1

Preview

Switch to L2 or spanned interface mode by moving ASA port-channel to ports assigned for spanned mode and applying Task 4 CLI.

Switch now load-balances under one IP path

Review CSR and ASA OSPF config

Ensure dead-intervals match (should be 3sec)

ASA1 and ASA2 in L2/Spanned cluster, continue to maintain state in Routed Firewall

Verify one IP route on CSR1 to outside

Verify one IP route on CSR2 to inside

Tests




Measure convergence

Bring Up downed ASA

CSR1 CSR2

CCL

One path One hop

Interna

l External

Task 4

92


.1

Master

Slave

Internal

10.10.140.0/24

External

.200

CSR1 CSR2

ASA2

ASA1

VLAN 15 VLAN 4

gig2 gig1 gig1 gig2

.200

1.1.1.0/24 1.1.2.0/24

.200 Po4.8

.1

Po4.7

.1

CCL

VLAN 25

2.2.2.0/24

G0/3

.1

G0/3

.2

ASA Spanned / Routed Cluster Diagram

Outside

VLAN 8 Inside

VLAN 7

mgmt_pool

172.16.1.2-172.16.1.10

172.16.2.0/24

ASA cluster nodes share the same IP for inside and outside VLANs.

IP pool needed only for management interface

Master

Inside host

Outside host

.30 .44

G0/0

G0/0

G0/1

G0/1

Task 4

93


CLI

! Disable clustring on ASA1 unit

changeto system

config terminal

cluster group fw

no enable

! Cluster disable is performing cleanup..done.

!All data interfaces have been shutdown due to clustering being disabled. To recover either enable clustering or remove cluster group configuration.

Cluster unit asa1 transitioned from MASTER to DISABLED

ClusterDisabled/a/asa1(cfg-cluster)#

ASA1

! Disable clustring on ASA2 unit

changeto system

config terminal

cluster group fw

no enable

! Cluster disable is performing cleanup..done.

!All data interfaces have been shutdown due to clustering being disabled. To recover either enable clustering or remove cluster group

configuration.

Cluster unit asa2 transitioned from SLAVE to DISABLED

ClusterDisabled/a/asa2(cfg-cluster)#

ASA2

Disable clustering feature on both units

And prep ASAs to change mode to Spanned cluster

Task 4

94


CLI

! Execute CLI to convert to L2 or Spanned interface mode

changeto system

config term

clear config all

cluster interface-mode spanned force

!WARNING: Cluster interface-mode is changed to 'spanned' without…(snip)


copy /noconfirm milan/task4-system.cfg running-config

!MUST confirm Y for YES, remove these commands and wait to finish sync

!Wait 1 min for ASA1 unit to become Master

!Cluster unit asa1 transitioned from DISABLED to MASTER

ASA1

! Clear ASA2 unit and convert it to L2 Spanned interface mode

changeto system

config terminal

clear config all

cluster interface-mode spanned force

!Bring up interface for CCL

interface GigabitEthernet0/3

no shut


cluster group fw

local-unit asa2

cluster-interface GigabitEthernet0/3 ip 2.2.2.2 255.255.255.0

priority 20

console-replicate

health-check holdtime 3

clacp system-mac auto system-priority 1

enable

!Wait for ASA2 to detect master, finish sync, and become a Slave unit


ASA2

Clear then re-apply L2 cluster configs

Review changes needed to move

Execute ASA2 CLI after ASA1 loads config and

becomes Master

Task 4

95


Verify

!master/a/asa1#

changeto system

show cluster info

Cluster fw: On Interface mode: spanned This is "asa1" in state MASTER ID : 0 Version : 9.3(2) Serial No.: FCH16097J8X CCL IP : 2.2.2.1 CCL MAC : c464.1339.1841 Last join : 18:43:37 UTC Jan 14 2015

Last leave: N/A

Other members in the cluster:


ID : 1 Version : 9.3(2) Serial No.: FCH16097J78 CCL IP : 2.2.2.2 CCL MAC : c464.1339.1481 Last join : 19:17:36 UTC Jan 14 2015 Last leave: N/A master/a/asa1(config)#

ASA1 Master

!master/a/asa1#

cluster exec show port-channel summary

asa1(LOCAL):**********************************************************

Group Port-channel Protocol Span-cluster Ports

------+-------------+---------+------------+---------------

2 Po2(U) LACP Yes Gi0/0(P) Gi0/1(P)

asa2:*****************************************************************

Group Port-channel Protocol Span-cluster Ports

------+-------------+---------+------------+---------------

2 Po2(U) LACP Yes Gi0/0(P) Gi0/1(P)

!master/a/asa1#

!Notice that Non-Stop Forwarding is enabled for ASA now


show run router

ASA1 Master

Review cluster state and port-channel Task 4

96


Verify CSR Routes

CSR1#

sh ip route













CSR1#

CSR1

Verify one IP path through cluster from CSRs

CSR2# sh ip route Gateway of last resort is 172.16.2.1 to network 0.0.0.0




C 1.1.2.0/24 is directly connected, GigabitEthernet1 L 1.1.2.200/32 is directly connected, GigabitEthernet1 10.0.0.0/24 is subnetted, 1 subnets O 10.10.140.0 [110/12] via 1.1.2.1, 00:21:20, GigabitEthernet1 172.16.0.0/16 is variably subnetted, 3 subnets, 2 masks C 172.16.2.0/24 is directly connected, GigabitEthernet2 L 172.16.2.200/32 is directly connected, GigabitEthernet2 O 172.16.3.1/32 [110/2] via 172.16.2.1, 3d00h, GigabitEthernet2 CSR2#

CSR2

Task 4

Are your routes missing? Make sure to sync up Master’s OSPF dead-interval to what you setup on CSRs in the Task 2.

97

Where are my OSPF routes? Hmmm. Do my dead-intervals match?



98

Inside-host

(IP 10.10.140.30)

Stil l sending packets…

./client.iperf

Outside-host

(IP 172.16.2.44)

ssh session stil l working?

Type one char and wait

ssh -l user 10.10.140.30

Inside-host

(IP 10.10.140.30)


Ping 172.16.2.44

Outside-host

(IP 172.16.2.44)


./server.iperf

Task 4

Measure connection convergence of each test: 1A, 1B, 2, and 3.

For each Test, observe and record packets lost for UDP and PING, and manually


Resiliency Tests: 1A, 1B, 2, and 3 Spanned Interface Mode (Ether-channel)

99

ASA2

ASA1

CSR1 CSR2

Inside

Host Outside

Host

Down UP

Test 2: Simulate

ASA crash w ith


Po4

G0/0

Po4

Down UP

G0/0

CCL

CCL

ASA1 ASA2






Down UP

G0/1 Down UP

G0/1 ASA1 ASA2

Down UP

G0/3

Down UP

G0/3 ASA1 ASA2

or or

or



Task 4


Test 1A Remove one of two data ports in ASA Port-Channel



Task 4




Protocol

Task 1A

Lost

Pkts/Secs

ping

UDP iPerf

ssh

100


Test 1B Remove the 2nd data port in ASA Port-Channel




Task 4


Protocol

Task 1B

Lost

Pkts/Secs

ping

UDP iPerf

ssh

Observe and record how many packets were lost

and how quickly on SSH session recovered

101


Recover ASA unit ‘no shut’ both ASA data ports on down ASA




Task 4

Re-enable cluster CLI to allow ASA to re-join

Up the ASA G0/0 port



changeto system

config terminal


cluster group fw

enable

!Wait for ASA2 to detect master, finish sync, and become

a Slave unit


Down ASA

102


Test 2 Crash connection owner ASA Task 4

Removing owner ASA from cluster

Crash owner ASA w/ CLI

Protocol

Task 2

Lost

Pkts/Secs

ping

UDP iPerf

ssh


write memory all



cluster group fw

enable



Owner ASA

103


Test 3 Take out owner ASA unit from the cluster Task 4


Protocol

Task 3

Lost

Pkts/Secs

ping

UDP iPerf

ssh

!You can do test 3 in two ways

!In the CLI, you can simply disable clustering

cluster group fw

no enable

!Or you can ‘down’ the CCL for owner ASA via web page

!As shown below in the home web page…

Down CCL on owner ASA

Owner ASA

104


Recover down ASA No Shut ASA CCL on Switch with IE

!Enable cluster on disabled Slave

!ClusterDisabled/a/asa1/admin(config)#

changeto context sys

!ClusterDisabled/a/asa1(config)#

cluster group fw

Enable


(snip)



Down ASA

Watch CSR consoles for route convergence logs

Enable cluster on ASA cli, to rejoin master

Task 4

Bring UP CCL on owner ASA

105


Task 4 Bonus*: Add PAT

Preview

This is a bonus task that involves adding back PAT configuration to ASA master.

Add Port Address Translation to outside interface of ASA L2 cluster with OSPF.

Add equal cost routes for new PAT network on CSR2.

Verify route on CSR2 for PAT pool network

Tests


Disable ASA that owns connections


Verify xlates for open connections

Task 4*

(optional) IP 1.1.2.1 IP 1.1.1.1

ASA2

ASA1

CSR1 CSR2

CCL

One path One Hop Aw ay

Interna

l External

106


CLI Add address translation cli

! If you skipped Task 3*, you will need pat-ips object



range 1.1.3.2 1.1.3.3


subnet 10.10.140.0 255.255.255.0



! Enable logging on master (this enables it on the slave too)

logging on

! Re-open your SSH connection to expose the translation info

! Notice NAT syslog now denying connection outside to inside

! Therefore, we need to SSH from inside to outside host

%ASA-7-609001: Built local-host outside:172.16.2.44

%ASA-7-609001: Built local-host inside:10.10.140.30

%ASA-5-305013: Asymmetric NAT rules matched for forward and reverse flows; Connection for tcp src outside:172.16.2.44/34770

dst inside:10.10.140.30/22 denied due to NAT reverse path failure

ASA1

Task 4*

Open connection inside to outside

user@inside-lnx:~$ ssh -l user 172.16.2.44


user@inside-lnx:~$

InsideHost

Due to PAT for inside subnet, inbound conns now need static NAT. You can test with ssh from

inside to outside linux.

ip route 1.1.3.0 255.255.255.0 1.1.2.1

sh ip route

(snip)

S 1.1.3.0/24 [1/0] via 1.1.2.1

CSR2

CSR2 needs a static route to ASA cluster PAT subnet to

redistribute into OSPF

NOTE: because we are

translating inside subnet, we need to test ssh from

inside to outside

107




108

Inside-host

(IP 10.10.140.30)

./client.iperf

Outside-host

(IP 172.16.2.44)


without a static NAT

Inside-host

(IP 10.10.140.30)

ping 172.16.2.44

Ssh [email protected]

Outside-host

(IP 172.16.2.44)

./server.iperf

Task 4*



Verify

Task 4*


cluster exec show xlate

asa1(LOCAL):**********************************************************

TCP PAT from inside:10.10.140.30/50511 to outside:1.1.3.3/50511 flags ri idle 0:00:27 timeout 0:00:30

asa2:*****************************************************************


master/a/asa1/admin(config)# cluster exec show conn

asa1(LOCAL):**********************************************************



TCP outside 172.16.2.44:22 inside 10.10.140.30:50511, idle 0:07:45,

bytes 0, flags y

asa2:*****************************************************************



TCP outside 172.16.2.44:22 inside 10.10.140.30:50511, idle 0:07:45, bytes 4102, flags UxIO


ASA1

Verify xlate(s) through cluster and OSPF route on CSR1

!CSR1# sh ip route Gateway of last resort is 1.1.1.1 to network 0.0.0.0 O*E2 0.0.0.0/0 [110/1] via 1.1.1.1, 00:42:27, GigabitEthernet1




O 1.1.2.0/24 [110/11] via 1.1.1.1, 00:42:31, GigabitEthernet1 O E2 1.1.3.0/24 [110/20] via 1.1.1.1, 00:22:50, GigabitEthernet1 10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks C 10.10.140.0/24 is directly connected, GigabitEthernet2 L 10.10.140.1/32 is directly connected, GigabitEthernet2 172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks O 172.16.2.0/24 [110/12] via 1.1.1.1, 00:42:27, GigabitEthernet1 O 172.16.3.1/32 [110/13] via 1.1.1.1, 00:42:27, GigabitEthernet1 CSR1#

CSR1

109


Remove PAT Remove PAT and route configs for now


config terminal


no nat (inside,outside) dynamic pat-pool pat-ips

write memory

ASA Master

Task 4*

Later in spanned, you will add PAT config

CSR2

config terminal

no ip route 1.1.3.0 255.255.255.0 1.1.2.1

110


Task 5: L2 Cluster in Transp

Preview

Change to Transparent mode in admin context, this clear ASA configuration

Rebuild context configuration by applying Task 5 CLI to ASAs and CSRs.

Change CSR IP addresses to /16 subnet, to allow peering OSPF through ASA

Change OSPF configs on CSRs

Verify OSPF route on CSR1 to outside

Verify OSPF route on CSR2 to inside

Tests


Down ASA that owns most connections


Measure convergence

IP 1.1.2.200/16 IP 1.1.1.200/16

ASA2

ASA1

CSR1 CSR2

Interna

l External

CCL

One Subnet Directly Connected

Task 5

111


.1

Master

Slave

Internal

10.10.140.0/24

External

.200

CSR1 CSR2

ASA2

ASA1

VLAN 15 VLAN 4

gig2

gig1 gig1

gig2 .1.200

1.1.0.0/16 1.1.0.0/16

.2.200 Po4.8

BVI1 Po4.7

BVI1

CCL

VLAN 25

2.2.2.0/24

0/3

.1

0/3

.2

ASA Spanned / Transparent Cluster Diagram

Outside

VLAN 8 Inside

VLAN 7

172.16.2.0/24

CSRs directly connected over 1.1.0.0/16 subnet

through L2 firewall

mgmt_pool

172.16.1.2-172.16.1.10

Inside and Outside interfaces Bridged by ASA cluster

Master

Inside host

Outside host

.30 .44

G0/0 G0/1

G0/0 G0/1

Task 5

112


CLI

!Install a transparent firewall context config for current admin context

config terminal

changeto system


context admin

config-url disk0:/task5-admin.cfg

.

Cryptochecksum (unchanged): dcf70f21 bc4b86f6 c570e03f 2093dcd6

INFO: Context admin was created with URL disk0:/task5-admin.cfg

INFO: Admin context will take some time to come up .... please wait.

master/a/asa1(config-ctx)#

ASA1

!master/a/asa1/admin(config-if)#

sh mac-address-table

interface mac address type Age(min) bridge-group

----------------------------------------------------------------------------

-------

inside 0050.56bf.34b8 dynamic 5 1

inside 0016.9cd3.b780 dynamic 4 1

outside 0050.56bf.dbc2 dynamic 4 1

master/a/asa1/admin(config-if)#

ASA1

Change context to Transparent FW mode

Verify mac-addresses of CSRs

Task 5

113


CLI

!Change CSR subnet to /16 so they can peer through ASA cluster

config terminal


ip address 1.1.1.200 255.255.0.0

router ospf 1 no network 1.1.1.0 0.0.0.255 area 0 network 1.1.0.0 0.0.255.255 area 0 ! Verify routes on CSRs, once they can ping each other and peer directly

show ip route ospf


O*E2 0.0.0.0/0 [110/1] via 1.1.2.200, 00:01:20, GigabitEthernet1 172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks O 172.16.2.0/24 [110/2] via 1.1.2.200, 00:01:20, GigabitEthernet1 O 172.16.3.1/32 [110/3] via 1.1.2.200, 00:01:20, GigabitEthernet1 CSR1#

CSR1

Change CSRs to directly connected routers

!Change CSR subnet to /16 so they can peer through ASA cluster config terminal interface GigabitEthernet1 ip address 1.1.2.200 255.255.0.0

router ospf 1

no network 1.1.2.0 0.0.0.255 area 0

network 1.1.0.0 0.0.255.255 area 0

! Verify routes on CSRs, once they can ping each other and peer directly show ip route ospf Gateway of last resort is 172.16.2.1 to network 0.0.0.0 O*E2 0.0.0.0/0 [110/1] via 172.16.2.1, 03:17:49, GigabitEthernet2 10.0.0.0/24 is subnetted, 1 subnets O 10.10.140.0 [110/2] via 1.1.1.200, 00:01:29, GigabitEthernet1



CSR2#

CSR2

Task 5

114


Verify



asa1(LOCAL):**********************************************************

0 in use, 19 most used Cluster stub connections: 0 in use, 6 most used asa2:***************************************************************** 2 in use, 8 most used Cluster stub connections: 0 in use, 117 most used OSPF outside 224.0.0.5 inside 1.1.1.200, idle 0:00:00, bytes 181176, flags OSPF outside 1.1.2.200 inside 224.0.0.5, idle 0:00:00, bytes 179984, flags


ASA1 Master

Show OSPF connections through ASA cluster Task 5

115


Setup Test Connections Again

116

Inside-host

(IP 10.10.140.30)

Restart if needed

./client.iperf

Outside-host

(IP 172.16.2.44)

Restart ssh session…

ssh -l user 10.10.140.30

Inside-host

(IP 10.10.140.30)


Ping 172.16.2.44

Outside-host

(IP 172.16.2.44)


./server.iperf

Task 5




Find Owner ASA Locate conn owner ASA



asa1(LOCAL):**********************************************************



OSPF outside 224.0.0.5 inside 1.1.1.200, idle 0:00:00, bytes 363712, flags





UDP outside 172.16.2.44:5001 inside 10.10.140.30:36188, idle 0:00:02, bytes 0, flags –y

asa2:*****************************************************************





ICMP outside 172.16.2.44:0 NP Identity Ifc 10.10.140.30:2841, idle 0:00:00, bytes 0, flags z

OSPF outside 1.1.2.200 NP Identity Ifc 224.0.0.5, idle 0:00:00, bytes 0, flags z


ICMP inside 10.10.140.30:2841 NP Identity Ifc 172.16.2.44:0, idle 0:00:00, bytes 0, flags z

OSPF inside 1.1.1.200 NP Identity Ifc 224.0.0.5, idle 0:00:00, bytes 0, flags z


ASA1

Shut down ASA data port on the Switch with IE

Task 5

117


Resiliency Tests: 1B, 2, and 3 Spanned Interface Mode (Ether-channel)

118

ASA2

ASA1

CSR1 CSR2

Inside

Host Outside

Host

Down UP

Test 2: Simulate

ASA crash w ith


Po4

G0/0

Po4

Down UP

G0/0

CCL

CCL

ASA1 ASA2






Down UP

G0/1 Down UP

G0/1 ASA1 ASA2

Down UP

G0/3

Down UP

G0/3 ASA1 ASA2

or or

or



Task 5


Test 1B Remove both data ports in ASA Port-Channel



Task 5





Protocol

Task 1B

Lost

Pkts/Secs

ping

UDP iPerf

ssh

119


Recover ASA unit ‘no shut’ both ASA data ports on down ASA Task 5

Re-enable cluster CLI to allow ASA to re-join




changeto system

config terminal


cluster group fw

enable



Down ASA

120


Test 2 Crash connection owner ASA Task 5


Crash owner ASA w/ CLI

Protocol

Task 2

Lost

Pkts/Secs

ping

UDP iPerf

ssh


changeto system

write memory all




Owner ASA

121


Test 3 Take out owner ASA unit from the cluster Task 5


Protocol

Task 3

Lost

Pkts/Secs

ping

UDP iPerf

ssh

!You can do test 3 in two ways

!In the CLI, you can simply disable clustering

changeto system

cluster group fw

no enable

!Or you can ‘down’ the CCL for owner ASA via web page

!As shown below in the home web page…

Down CCL on owner ASA

Owner ASA

122


Recover down ASA No Shut ASA CCL on Switch with IE

!Enable cluster on disabled Slave

!ClusterDisabled/a/asa1/admin(config)#

changeto context system

!ClusterDisabled/a/asa1(config)#

cluster group fw

enable


(snip)



Down ASA

Watch CSR consoles for route convergence logs

Enable cluster on ASA cli, to rejoin master

Task 5

Bring UP CCL on owner ASA

123


Task 5 Bonus*: Add PAT

Preview

This is a bonus task to add PAT configuration in transparent firewall mode on ASA master.

Add Port Address Translation to outside interface inside admin context.

Remove older route for PAT network on CSR2, it is not needed as PAT and CSR interfaces are now in same network

Tests




Verify xlates

Task 5*

(optional) IP 1.1.2.200/16 IP 1.1.1.200/16

ASA2

ASA1

CSR1 CSR2

Interna

l External

CCL

One Subnet Directly Connected

124


CLI

Task 5*

! If you skipped Task 3*, you will need pat-ips and inside-network objects


range 1.1.3.2 1.1.3.3


subnet 10.10.140.0 255.255.255.0




! You may need to clear existing conns to create an xlate

clear local

ASA1

!CSR2#

show ip route












CSR2#

CSR2

Introduce PAT

Remove route on CSR2

125




126

Inside-host

(IP 10.10.140.30)

./client.iperf

Outside-host

(IP 172.16.2.44)


without a static NAT, so

SSH from inside

to outside

Inside-host

(IP 10.10.140.30)

ping 172.16.2.44


Outside-host

(IP 172.16.2.44)

./server.iperf

Task 5*



Verify

Task 5*


asa1(LOCAL):**********************************************************



TCP outside 172.16.2.44:22 inside 10.10.140.30:50519, idle 0:00:06, bytes 4166, flags UIO

asa2:*****************************************************************



OSPF outside 224.0.0.5 inside 1.1.1.200, idle 0:00:00, bytes 158544,

flags





ASA1

cluster exec show xlate

asa1(LOCAL):**********************************************************


Flags: D - DNS, e - extended, I - identity, i - dynamic, r - portmap,

s - static, T - twice, N - net-to-net


asa2:*****************************************************************


Flags: D - DNS, e - extended, I - identity, i - dynamic, r - portmap,

s - static, T - twice, N - net-to-net



ASA1

Re-open test connections

Verify conn and xlates are created

127

128

CONGRATULATIONS….

on completing the LTRSEC-2740 lab


Call to Action

• Visit the World of Solutions for

– Cisco Campus – Visit Network and Content Security Booths

– Technical Solution Clinics

• Meet the Engineer – ASA experts from our team will be available to meet you

• Lunch time Table Topics

• DevNet zone related labs and sessions

• Recommended Reading: for reading material and further resources for this session, please visit www.pearson-books.com/CLMilan2015

129

http://www.pearson-books.com/CLMilan 2015




Complete Your Online Session Evaluation

• Please complete your online session evaluations after each session. Complete 4 session evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt.

• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations

130

Additional Slides


CCL

Po 100

AN

IMAT

ED

SLI

DE

Po 101

Po 102

Po 103

Po 200

Po 201

Po 202

Po 203

IP 1.1.1.1

IP 1.1.1.2

IP 1.1.1.3

IP 1.1.1.4

Outside Inside

IP 1.1.2.1

IP 1.1.2.2

IP 1.1.2.3

IP 1.1.2.4

IP 1.1.1.1

IP 1.1.1.2

IP 1.1.1.3

IP 1.1.1.4

Outside Inside

IP 1.1.2.1

IP 1.1.2.2

IP 1.1.2.3

IP 1.1.2.4

Single Attach Dual Attach

ASA Cluster to Routers Data Plane – Individual Mode

Interface Layer 3 mode • Dedicated IP/MAC addresses per ASA Interface • ECMP from both sides of ASA (outside and inside) • Improve convergence by tuning timers

CCL

vPC vPC

132


N7K/vPC Cat/VSS

cLACP

ASA Po 10 LACP vPC 100

Classic Switch

cLACP ASA Po 10

Po 100

ASA Cluster to Switch Data Plane – Spanned Mode

CCL CCL

Interface Layer 2 mode • One IP per Ether-channel interface shared by the cluster • A port ID on each ASA joins the a spanned port-channel • vPC extends the channel across two switches • Data Plane MUST use cLACP

133

Instructions cisco

Documents

Transcript of Instructions cisco