t-shoot GET

© 2012 Cisco and/or its affiliates. All rights reserved. BRKSEC - 3051 Cisco Public

Troubleshooting GETVPN Deployments BRKSEC-3051

1


GETVPN Solution Overview

What Is GETVPN and Where Does It Fit?

Introduction to GETVPN

Technology Overview

GETVPN Deployment

Configuration and Deployment Considerations

Troubleshooting

Troubleshooting Tools and Techniques

Common Troubleshooting Scenarios

Agenda


Other Related Sessions

BRKSEC-3050 – Troubleshooting Remote Access SSL VPN

and Secure Mobility

BRKSEC-3052 – Troubleshooting DMVPNs

BRKSEC-3053 – Deploying GET to Secure VPNs

BRKSEC-4054 – DMVPN Deployment Models

CiscoLive 2012

GETVPN Solution Overview


Cisco Group Encrypted Transport - GETVPN

Large-scale any-to-any encrypted

communication

Native routing without

tunnel overlay

Optimal for QoS and Multicast

support—improves application

performance

Transport agnostic—private

LAN/WAN, FR/ATM, IP, MPLS

Any - to - Any Connectivity

Real Time Scalable

Any - to - Any Connectivity

Cisco GET

VPN

What Is GETVPN?

Cisco GETVPN delivers a revolutionary solution for tunnel-less, any-to-any and confidential branch communication


Tunnel-Less VPN - A New Security Model

Scalability—an issue (N^2 problem)

Overlay routing

Any-to-any instant connectivity can’t be done to scale

Limited QoS

Inefficient Multicast replication

WAN

Multicast

Before: IPSec P2P Tunnels After: Tunnel-Less VPN

Scalable architecture for any-to-any connectivity and encryption

No overlays—native routing

Any-to-any instant connectivity

Enhanced QoS

Efficient Multicast replication


VPN Technology Positioning

Internet/Shared Network MPLS/Private

Network

EzVPN/FlexVPN Spoke

GETVPN GM DMVPN/FleVPN Spoke

DMVPN/FLexVPN Spoke

GM GM

KS KS

IPSec Agg.

WAN Edge Remot Access SW Clients

GETVPN GM GETVPN GM

Data Center Core

GET Encrypted

Internet Edge


VPN Technology Positioning (Cont.)

FlexVPN DMVPN GET VPN

Infrastructure Network Public Internet

Transport Public Internet

Transport Private IP Transport

Network Style Converged Site to Site and Remote Access

Hub-Spoke and Spoke-to-Spoke; (Site-to-Site)

Any-to-Any; (Site-to-Site)

Routing Dynamic Routing or

IKEv2 Route Distribution Dynamic routing on

tunnels Dynamic routing on IP

WAN

Failover Redundancy Route Distribution

Server Clustering

Route Distribution Model

Route Distribution Model + Stateful

Encryption Style Peer-to-Peer Protection Peer-to-Peer Protection Group Protection

IP Multicast Multicast replication at

hub Multicast replication at

hub Multicast replication in

IP WAN network

Introduction to GETVPN


Group Encrypted Transport

Uses three main components

‒ Secure Group Keys

‒ Header Preservation

‒ Key Service

Based on open standards with patented Cisco technology

Leverages existing IKE, IPSec, and multicast technologies

Takes advantage of the existing routing infrastructure


Group Security Functions

Group Member

Group Member

Group Member

Group Member

Key Server

Routing Members

Group Member Encryption Devices Route Between Secure/ Unsecure

Regions Multicast Participation

Key Server Validate Group Members Manage Security Policy Create Group Keys Distribute Policy/Keys

Routing Member Forwarding Replication Routing


Group Security Elements

Group Member

Group Member

Group Member

Group Member

Key Servers

Routing Members

Key Encryption Key (KEK)

Traffic Encryption Key (TEK)

Group Policy

RFC3547: Group Domain of

Interpretation (GDOI)

KS Cooperative Protocol


Basic GET VPN Architecture

Step 1: Group Members (GM) register via GDOI with

the Key Server (KS)

KS authenticates and authorizes the GM

KS pushes a set of IPSec SAs

for the GM to use

GM1

GM2

GM3 GM4

GM5

GM6

GM7 GM8

GM9 KS



Step 2: Data Plane Encryption

GM exchange encrypted traffic using the group keys

The traffic uses IPSec Tunnel Mode with Header Preservation

GM1

GM2

GM3 GM4

GM5

GM6

GM7 GM8

GM9 KS



Step 3: Periodic Rekey of Keys

KS pushes out replacement IPSec keys before current IPSec keys

expire; this is called a Rekey

KS

GM1

GM2

GM3 GM4

GM5

GM6

GM7 GM8

GM9


Header Preservation IPSec Tunnel Mode vs. GETVPN

IP Packet

IP Payload IP Header IPSec Tunnel Mode

ESP New IP Header

IP Payload IP Header

IPSec header inserted by VPN Gateway New IP Address requires overlay routing

IP Packet

IP Payload IP Header ESP Preserved Header GETVPN

IP Payload IP Header

IP header preserved by VPN Gateway Preserved IP Address uses original routing plane


GETVPN Data Path GM 1 GM2

Encrypted

Host1 Host2

Encrypted/Authenticated Using Group SA

Original IP Header

Data Original Src and Dst Addresses ESP


Rekey Methodology: Multicast Rekey

Rekey Message sent from key server to all group members

IP multicast message provides very efficient distribution

Rekeys resulting from configured KEK and TEK intervals

or KS policy change

PE 3

2 PE 1

P 1 P 2

P 3 P 4

GM 3 GM 4

GM 1 GM 2

Key Server

Single rekey packet sent to

multicast enabled core

Core replicates

the packet to all the GMs


Rekey Methodology: Unicast Rekey

Key Server maintains state of active group members

Group Member sends ACK in response to the rekey

messages

Remove Group Member if the GM does not acknowledge

three rekeys

PE 3

PE 1

P 1 P 2

P 3 P 4

GM 3 GM 4

GM 1 GM 2

Key Server


Requirement for Time-Based Anti-Replay

Sequence number based anti-replay only works with

single sender

Need method to work for all senders using same IPSec SA

‒ Key Server downloads Relative pseudotime and window size to

all the GMs

‒ GMs calculate pseudo-timestamp based on downloaded pseudotime and

sends out packet

‒ Receiving GM verifies packet within window size

‒ KS periodically refreshes GMs with pseudotime/window size - this means

clock does not need to be synchronized between GMs


Time-Based Anti-Replay

If Sender’s pseudotime falls in the below Receiver window,

packet accepted

T0 T10 T20

Packet1

Packet2

PTr - W PTr + W PTr

Anti-replay window

Initial

pseudotime

Reject Reject Accept

Packet 1 and Packet 2 have pseudotimeT0, providing loose anti-replay protection (unlike counter-based)


Cooperative Key Servers - HA

Single KS is a single point of failure

Two or more KSs known as COOP KSs manage a common set of keys and

security policies for GETVPN group members

Group members can register to any one of the available KSs

GM 1

GM 3

Subnet 1

Subnet 4

Subnet 2

Subnet 3

GM 4

GM 2

Cooperative KS3

Cooperative KS1

IP Network

Cooperative KS2

GDOI Registration


Cooperative Key Servers - Introduction (Cont.) One KS is elected as the Primary KS

Cooperative KSs periodically exchange and synchronize group’s database,

policy and keys

Primary KS is responsible to generate and distribute group keys

Cooperative KS3 (Secondary)

GM 1

GM 3

Subnet 1

Subnet 4

Subnet 2

Subnet 3

GM 4

GM 2

Cooperative KS1

IP Network

Cooperative KS2 (Secondary)

Announcement Messages

(Primary)

Rekey Messages

GETVPN Deployment Configuration


COOP Server Exportable RSA Keys

Exporting RSA Key from Key Server to Group Member:

‒ Public key generated in the RSA key pair, is sent to the GM at the

registration

‒ The re-keys are signed by the private key of the KS and GM verifies the

signature in the re-key with the public key of the KS

Exporting RSA Key between Key Servers:

‒ One of the key server in the redundancy group should generate the

exportable RSA keys and copy those keys to other

key servers

RSA Keys (to Be Generated only on KSs) Are Required for Rekey Authentication


crypto keyring gdoi1 pre-shared-key address 0.0.0.0 0.0.0.0 key cisco123 ! crypto isakmp policy 10 encr 3des authentication pre-share ! crypto ipsec transform-set 3DES-SHA esp-3des esp-sha-hmac ! crypto ipsec profile gdoi1 set security-association lifetime seconds 7200 set transform-set 3DES-SHA !

access-list 150 permit ip any host 225.1.1.1 ! access-list 160 deny eigrp any any access-list 160 deny pim any any access-list 160 deny udp any any eq isakmp access-list 160 deny udp any any eq 848 access-list 160 permit ip any any

Pre-shared Key

IPSec Profile

ISAKMP Policy

Access-list denying encryption for ISAKMP/GDOI/EIGRP packets and permitting

encryption for all IP traffic

Access-List used for defining

rekey (useful in multicast rekeys only)

IPSec Transform

KS Configuration


crypto gdoi group getvpn1 identity number 101 server local !rekey address ipv4 150 ! rekey lifetime seconds 14400 rekey retransmit 10 number 2 rekey authentication mypubkey rsa getvpn1 rekey transport unicast sa ipsec 1 profile gdoi1 match address ipv4 160 address ipv4 130.23.1.1 redundancy local priority 10 peer address ipv4 130.1.2.1 !

Encryption ACL

GDOI Group ID

RSA Key to authenticate rekeys

Rekey Address mapping to ACL 150 (only for multicast

rekeys)

Source address for rekeys

Unicast Rekey

Lifetime for Key Encryption Key

Rekey Retransmission

Coop Server Config

Coop Server address

Coop Server priority

KS Configuration (Cont.)


GM Configuration

crypto keyring gdoi pre-shared-key address 0.0.0.0 0.0.0.0 key cisco123 ! crypto isakmp policy 10 encr 3des authentication pre-share group 2 ! crypto gdoi group getvpn1 identity number 101 server address ipv4 130.23.1.1 ! crypto map getvpn10 gdoi set group getvpn1 ! interface FastEthernet0/0 crypto map getvpn

Crypto map on the interface

Pre-Shared key

GDOI Group

KS Address

ISAKMP Policy

GDOI configuration

mapped to crypto map


GETVPN Platform Support

Platform Group Member Key Server

Software Yes Not supported

870 Yes Not supported

1821 Yes Not supported

1841/1900 Yes Yes

2800 (AIM/SSL)/2900 Yes Yes

3800 (AIM-II/AIM-III)/3900 Yes Yes

7200 NPEG1, VAM2+ Yes Yes



7200 NPEG2, VSA Yes Yes

Cisco ASR 1000 Yes Yes (since XE3.6)


Scalability and Performance

GETVPN Provides complete segregation of control and data plane

Key Server is responsible to maintain the control plane (key

management) and GM is responsible to handle the data plane

(actual user traffic)

KS and GM can not be configured on same IOS device

KS should be properly sized for number of branches (scale) in the

network

GM should be properly sized for traffic throughput at each branch


Deployment Best Practices

IKE/IPSec

Use specific pre-shared keys for all the GMs and KSs instead of using default key

KS

Always use COOP KSs

Set the huge buffer to 65535 and add 10 buffers to permanent buffer list

Configure periodic DPDs between the COOP KSs

Enable GM authorization

Policy

Aggregate the permit access-list entries to reduce the entries

Enable Time-Based Anti-Replay

Avoid re-encrypting traffic which is already encrypted (SSH, HTTPS)

Registration

Distribute GM registration to multiple KSs by arranging the KS order in configuration

Rekey Timers

Set TEK lifetime to 7200 Seconds

Set KEK lifetime to 86400 Seconds

GETVPN Troubleshooting

―A problem well stated is a problem half solved.‖

Charles F. Kettering


Troubleshooting GETVPN

Ultimately all problems manifest at the data plane -―my user

application is not working over GETVPN!‖

But where really is the problem?

Control Plane

‒ Events that lead up to SAs getting installed on the GMs

Data plane

‒ Policy downloaded with SAs installed but traffic is not flowing


Troubleshooting GETVPN High Level Flow

Control Plane Data/Forwarding Plane

COOP

IKE

Registration

Policy Download

Rekey

Troubleshooting Flow

TimeBased Anti-Replay

Fragmentation MTU Issues

Transport Issues

Crypto policy/engine


GETVPN Control Plane

Common Control Plane Issues

‒ GM registration issues

‒ Policy download issues

‒ COOP issues

‒ Rekey failures

Understand the expected protocol flow and know how to

check for them


Control Plane Troubleshooting Tools

GETVPN provides enhanced set of show commands for

functionality verification

IOS also provided wide variety of syslog messages to verify proper

GETVPN operations, and early insight into potential problems

IPSec and GDOI related debugs can then be enabled for further

troubleshooting

GDOI conditional debugs – 15.1(3)T

GDOI event trace – 15.1(3)T


Show crypto gdoi (on KS)

Group Name : GET

Group Identity : 101

Group Members : 3

IPSec SA Direction : Both

Active Group Server : Local

Redundancy : Configured

Local Address : 130.23.1.1

Local Priority : 10

Local KS Status : Alive

Local KS Role : Primary

Group Rekey Lifetime : 1800 secs

Group Rekey

Remaining Lifetime : 88 secs

Rekey Retransmit Period : 10 secs

Rekey Retransmit Attempts: 3

Group Retransmit


IPSec SA Number : 1

IPSec SA Rekey Lifetime: 900 secs

Profile Name : gdoi1

Replay method : Count Based

Replay Window Size : 64

SA Rekey


ACL Configured : access-list 160

Group Server list : Local

Key server role

KEK lifetime remaining

coop configuration

TEK lifetime remaining

Registered GMs


Show crypto gdoi ks member (on KS)

Number of rekeys sent for group GET: 4

Group Member ID : 131.1.1.1

Group ID : 101

Group Name : getvpn1

Key Server ID : 130.2.1.1

Rekeys sent : 4

Rekey Acks Rcvd : 4

Sent seq num : 1 2 3 4

Rcvd seq num : 1 2 3 4

KS GM is registered with

GM’s IP address


Show crypto gdoi (on GM) GROUP INFORMATION

Group Name : GET


Rekeys received : 270


Active Group Server : 134.50.0.1

Group Server list : 134.50.0.1

GM Reregisters in : 5187 secs

Rekey Received(hh:mm:ss) : 00:02:30

Rekeys received

Cumulative : 270

After registration : 270

Rekey Acks sent : 270

ACL Downloaded From KS 134.50.0.1:

access-list deny eigrp any any

access-list deny tcp any any port = 179

access-list deny udp any port = 848

any port = 848

access-list permit ip any any

KEKPOLICY:

Rekey Transport Type : Unicast

Lifetime (secs) : 12295

Encrypt Algorithm : 3DES

Key Size : 192

Sig Hash Algorithm : HMAC_AUTH_SHA

Sig Key Length (bits) : 1024

TEK POLICY:

FastEthernet0/0:

IPSec SA:

sadirection:outbound

spi: 0x7C45C74A(2084947786)

transform: esp-aes esp-sha-hmac

Sa timing: remaining key lifetime

(sec): (5246)

Anti-Replay(Time Based) : 2 sec interval

Active KS

When was last rekey received

Remaining IPSec SA Lifetime


Show crypto gdoi GM acl (on GM)

Group Name: GET



access-list deny udp any any port = 500

access-list deny udp any any port = 848


ACL Configured Locally:

Map Name: getvpn-map

access-list 165 deny pim any any

ACL downloaded from KS

Locally configured ACL if present


Show crypto IPSec SA PHEONIX-GM# show crypto ipsec sa

interface: FastEthernet0/0

Crypto map tag: dgvpn, local addr 131.1.1.1

protected vrf: (none)

local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)

remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)

current_peer port 848

<SNIP>

inbound espsas:

spi: 0x1DA9D3E2(497669090)

transform: esp-3des esp-sha-hmac ,

<SNIP>

outbound espsas:

spi: 0x1DA9D3E2(497669090)


RALEIGH-GM# show crypto ipsec sa

interface: GigabitEthernet0/1

Crypto map tag: dgvpn, local addr 131.3.1.1

protected vrf: (none)

local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)

remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)

current_peer port 848

<SNIP>

inbound espsas:

spi: 0x1DA9D3E2(497669090)


<SNIP>

outbound espsas:

spi: 0x1DA9D3E2(497669090)


Same inbound and

outbound SPI on

all Group Members


GETVPN Control Plane Verification

Syslog Messages - GM Registration:

CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.11.2 for group

G1 using address 10.1.13.2

GDOI-5-GM_REKEY_TRANS_2_UNI: Group G1 transitioned to Unicast

Rekey

GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.11.2 complete for

group G1 using address 10.1.13.2

Rekey:

GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from

10.1.11.2 to 10.1.13.2 with seq # 3


GETVPN Control Plane Verification

Syslog Messages - KS

Rekey:

GDOI-5-KS_SEND_UNICAST_REKEY: Sending Unicast Rekey for group

G1 from address 101.1.1.1 with seq # 1

COOP:

GDOI-3-COOP_KS_UNREACH: Cooperative KS 10.0.9.1 Unreachable in

group G1

GDOI-5-COOP_KS_ELECTION: KS entering election mode in group G1

(Previous Primary = NONE)

GDOI-5-COOP_KS_TRANS_TO_PRI: KS 10.0.8.1 in group G1

transitioned to Primary (Previous Primary = NONE)


Control Plane Debugging Challenges Challenge

Networks are getting bigger and faster, traditional debugs may not scale

Solution

Use IPSec and GDOI conditional debugs to minimize the debugging impact

Use the minimal level of debugs required

Challenge

Problems can be unpredictable with no identifiable trigger

Solution

Syslogs

GDOI Event Trace


GDOI Debug Level Granularity

All feature components can be debugged at 5 levels

Start with the highest level, enable additional levels as needed

Debug Level What do you get Error Error Conditions

Terse Important messages to the user and protocol issues

Event State transitions and events such as send/receive rekeys

Detail Most detailed debug message information

Packet Dump of detailed packet information

All All of the above

GM1#debug crypto gdoi gm rekey ?

all-levels All levels

detail Detail level

error Error level

event Event level

packet Packet level

terse Terse level


GDOI Conditional Debugs All IPSec and GDOI debugs can now be triggered with

a conditional filter based on group or peer address

Use the unmatched flag to catch debugs with no context information

To enable conditional debugs

1) Set the conditional filter

2) Enable relevant debugs of interest as usual

KS1#debug crypto gdoi condition peer add ipv4 10.1.20.2 % GDOI Debug Condition added.

KS1#

KS1# show crypto gdoi debug-condition

GDOI Conditional Filters:

Peer Address 10.1.20.2

Unmatched NOT set

KS1#debug crypto gdoi ks registration all-levels

GDOI Key Server Registration Debug level: (Packet, Detail, Event, Terse, Error)

KS1

GM500 GM1

MPLS/Private IP

GM145

?

KS2


Best practices when using the debug commands

Turn off console logging

Use NTP to sync up times on all devices

Enable msectimestamping debug and log messages

‒ service timestamps debug datetime msec

‒ service timestamps log datetime msec

Send the debugs to a syslog server

If no syslog server isavailable, use the logging buffer with an increased buffer size

‒ logging buffered 1000000 debugging

terminal exec prompt timestamp when using the show commands to correlate show commands with the debug output


GDOI Event Trace

Light weight event buffer to supplement syslogs

Always-on

Flexible output and display options

Event buffer

Continuous real time output

Output to file

Merged output from different feature components

Circular or one-shot buffer

Extensive exit path/error tracing capability


GDOI Event Trace - Example

GM1#show monitor event-trace gdoi?

all Show all the traces in current buffer

back Show trace from this far back in the past

clock Show trace from a specific clock time/date

coop GDOI COOP Event Traces

from-boot Show trace from this many seconds after booting

infra GDOI INFRA Event Traces

latest Show latest trace events since last display

merged Show entries in all event traces sorted by time

registration GDOI Registration event Traces

rekey GDOI Rekey event Traces

GM1#show monitor event-trace gdoi merged all

*May 25 20:20:57.706: Registration_events: GDOI_REG_EVENT: REGISTRATION_STARTED:

GM 10.1.20.2 to KS 10.1.11.2 for group G1

*May 25 20:21:08.970: Registration_events: GDOI_REG_EVENT: REGISTRATION_DONE: GM

10.1.13.2 to KS 10.1.11.2 for group G1

*May 26 00:45:52.878: Rekey_events: GDOI_REKEY_EVENT: REKEY_RCVD: From 10.1.11.2

to 10.1.13.2 with seq no 131 for the group G1

*May 26 00:45:52.878: Rekey_events: GDOI_REKEY_EVENT: ACK_SENT: From 10.1.11.2

to 10.1.13.2 with seq no 131 for the group G1


crypto gdoi group G1

identity number 3333

server address ipv4 10.1.12.2


!

crypto map gm_map 10 gdoi

set group G1

!

interface Serial1/0

crypto map gm_map





!


set group G1

!

interface Serial1/0

crypto map gm_map



server local

rekey lifetime seconds 86400

rekey authenmypubkeyrsa get

rekey transport unicast

saipsec 1

profile gdoi-p

match address ipv4ENCPOL

replay time window-size 5

address ipv4 10.1.12.2

redundancy

local priority 2

peer address ipv4 10.1.11.2



server local


rekey authenmypubkeyrsa get

rekey transport unicast

sa ipsec 1

profile gdoi-p


replay counter window-size 64


redundancy

local priority 10

peer address ipv4 10.1.12.2

Troubleshooting Methodology

KS1 KS2

GM2 GM1

Ser 1/0: 10.1.20.2 Ser 1/0: 10.1.21.2

Eth 0/0: 192.168.20.1/24 Eth 0/0: 192.168.21.1/24

Ser 1/0: 10.1.11.2 Ser 1/0: 10.1.12.2

MPLS/Private IP


GETVPN Control Plane Setup Steps

COOP KS IKE Setup

COOP Election and Policy Creation

GM-KS IKE Setup

GM Authorization and Registration

GM Encryption Keys and Policy download

GM Data Encryption and Decryption

Periodic Key Renewal and Distribution (Rekeys)


GETVPN Common Issues – Control Plane

IKE Setup

COOP Setup and Policy Creation

Encryption Policy

Key Renewal—Rekey

Authorization and Registration

Control Plane Packet Fragmentation Issue

Control Plane Replay Check


COOP KS Setup and Election

IKE comes up as soon as COOP servers boot up

KS1#show crypto isakmp sa

IPv4 Crypto ISAKMP SA

Dst src state conn-id slot status

10.1.11.2 10.1.12.2 GDOI_IDLE 1078 0 ACTIVE

KS2#show crypto isakmp sa



10.1.11.2 10.1.12.2 GDOI_IDLE 1023 0 ACTIVE

One of the KSs transitions to Primary KS and other becomes Secondary

%GDOI-5-COOP_KS_ELECTION: KS entering election mode in group G1

(Previous Primary = NONE)

%GDOI-5-COOP_KS_TRANS_TO_PRI: KS 10.1.11.2 in group G1 transitioned to

Primary(Previous Primary = NONE)


COOP Configuration Mismatch

COOP configuration MUST be manually synchronized on all the KSs

Due to mismatched configuration, COOP KSs can exhibit unexpected

behavior

Syslog Messages indicates an error condition

Fixing the mismatched configuration parameters on both KSs will fix this issue

%GDOI-3-COOP_CONFIG_MISMATCH: WARNNING: replay method configuration

between Primary KS and Secondary KS are mismatched

KS2#

KS1:


server local

sa ipsec 1

replay counter window-size 64

KS2:


server local

sa ipsec 1

replay time window-size 5




Encryption Policy

Key Renewal—Rekey




IKE Setup


IKE Setup Between KS and GM

First step in GM registration is IKE setup

On successful negotiation of the IKE process, GM proceeds

with the GDOI group registration

IKE SA is established at the time of registration eventually

times out as its no longer needed after registration

KS1# show crypto isakmp sa


Dst src state conn-id status

10.1.11.2 10.1.20.2 GDOI_IDLE 1013 ACTIVE

10.1.12.2 10.1.11.2 GDOI_IDLE 1004 ACTIVE

10.1.21.2 10.1.11.2 GDOI_REKEY 0 ACTIVE

GM1# show crypto isakmp sa



10.1.11.2 10.1.20.2 GDOI_IDLE 1073 0 ACTIVE

10.1.20.2 10.1.11.2 GDOI_REKEY 1074 0 ACTIVE

Expires after IKE lifetime


IKE Setup – IKE Failure Symptoms

If a GM fails to register with the KS, it will continue to attempt

to register with the KS

Possible causes:

‒ Network issues between the GM and KS

‒ IKE negotiation failure

‒ KS policy issues

*May 24 06:40:15.581: %CRYPTO-5-GM_REGSTER: Start registration to KS

10.1.11.2 for group G1 using address 10.1.20.2

GM1#

*May 24 06:41:25.581: %CRYPTO-5-GM_REGSTER: Start registration to KS

10.1.11.2 for group G1 using address 10.1.20.2

KS1 KS2

GM2 GM1

MPLS/Private IP

%GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.11.2 complete for group



Pre-Shared Key Mismatch Troubleshooting

Verify routing information on KS and GM and try ping KS from the GM

After ruling out the connectivity issues, check the IKE SA on the GM

Verify the logs on the Key Server

GM1#show crypto isakmp sa



10.1.11.2 10.1.20.2 MM_KEY_EXCH 1038 ACTIVE


IKE SA not getting established; can’t get to GDOI_IDLE state

KS1#

%CRYPTO-4-IKMP_BAD_MESSAGE: IKE message from 10.1.20.2 failed its

sanity check or is malformed


Pre-Shared Key Mismatch Solution

Syslog pointing to a mismatched pre-shared key configuration

Can be verified using ―debug crypto isakmp‖

KS Config

GM Config

crypto isakmp key cicso address 10.1.20.2

crypto isakmp key cisco address 10.1.11.2

Mismatch!

Correct the pre-shared key configuration

KS1(config)#no crypto isakmp key cicso address 10.1.20.2

KS1(config)#crypto isakmp key cisco add 10.1.20.2

KS1(config)#^Z



IKE Setup


Encryption Policy

Key Renewal—Rekey





GM Authorization and Registration When the IKE session is successfully established, GM is

authorized (if configured) and GM registers with the KS GM1#show crypto gdoi

GROUP INFORMATION

Group Name : G1


Rekeys received : 221


Active Group Server : 10.1.11.2

<snip>

KS1#show crypto gdoi ks members | in Group

Group Member Information :


Group ID : 3333

Group Name : G1


Group ID : 3333

Group Name : G1

Group Members put into the GM database once successfully registered


Unauthorized GM Symptoms

IKE is established between the GM and the KS

GM1#show crypto isakmp sa



10.1.11.2 10.1.20.2 GDOI_IDLE 1054 ACTIVE

Even after IKE is established, GM fails to register with the KS

%CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.11.2 for group G1

using address 10.1.20.2

Registration is not complete; following message is not

displayed

%GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.11.2 complete for group


KS1 KS2

GM2 GM1

MPLS/Private IP


Unauthorized GM Troubleshooting Steps

Following Syslog Message appears on the KS:

%GDOI-1-UNAUTHORIZED_IPADDR: Group G1 received registration from

unauthorized ip address: 10.1.20.2

Verify the KS authorization policy to identify the authorization list:

KS1#show run | section crypto gdoi



server local

<snip>

authorization address ipv4 gm-author-list

...

KS1#sh access-lists gm-author-list

Standard IP access list gm-author-list

10 permit 10.1.21.2

Address not

in ACL


Unauthorized GM Solution

Add the GM to the authorization list

KS1(config)#ip access-list standard gm-author-list

KS1(config-std-nacl)#permit host 10.1.20.2

KS1#

Verify the GM can now register with the KS successfully

%CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.11.2 for


%GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.11.2 complete for




IKE Setup


Key Renewal—Rekey




Encryption Policy


GM Policy Download

As part of the registration process, KS pushes down the

encryption policies and keying material to the GM: GM1#show crypto gdoi

<snip>



access-list deny ip 224.0.0.0 0.0.0.255 any

access-list deny ip any 224.0.0.0 0.0.0.255

access-list deny udp any port = 848 any port = 848


KEK POLICY:

Rekey Transport Type : Unicast

Lifetime (secs) : 2954

<snip>

TEK POLICY:

Serial1/0:

IPSec SA:

sa direction:inbound

spi: 0x2113F73B(554956603)

transform: esp-3desesp-sha-hmac

sa timing:remaining key lifetime (sec): (99)

Anti-Replay(Time Based) : 5 sec interval

<snip>


KS Policy Issues Routing Control Plane Traffic Failure

In most environments, GETVPN runs on

the CE devices and PE devices do not

participate in GETVPN

Failure to deny control plane traffic (such as routing protocol) on the

PE-CE link will cause routing protocol to go down as soon as GM

successfully registers

To identify, look at the ACL downloaded at GM:

GM1#show crypto gdoi gm acl

Group Name: G1



access-list deny ip 224.0.0.0 0.0.0.255 any

access-list deny ip any 224.0.0.0 0.0.0.255

access-list deny udp any port = 848 any port = 848



KS1 KS2

GM2 GM1

MPLS/Private IP

BGP

BGP is not denied in the ACL downloaded from the KS


KS Policy Issues Control Plane Traffic—Solution

If most of the CEs are running BGP with the PE routers,

configure a global KS policy to deny BGP

If only a handful of CEs are running BGP with the PE routers, configure a local GM policy to deny BGP

KS1&2(config)# ip access-list extended ENCPOL

KS1&2(config-ext-nacl)#1 deny tcp any any eq bgp

KS1&2(config-ext-nacl)#2 deny tcp any eq bgp any

GM1#

!

access-list 150 deny tcp any any eq bgp

access-list 150 deny tcp any eq bgp any

!


set group G1

match address 150

!

GM1#show crypto gdoi gm acl

Group Name: G1


<snip>



Map Name: gm_map

access-list 150 deny tcp any any port = 179

access-list 150 deny tcp any port = 179 any

KS1 KS2

GM2 GM1

MPLS/Private IP

BGP



IKE Setup


Key Renewal—Rekey




Encryption Policy

Key Renewal - Rekey


GETVPN Rekeys

Once the GETVPN network is properly setup and is working,

KS is responsible for sending out rekey messages to all the

GMs

KS can use unicast or multicast rekeys

Following syslog messages will appear in the log:

PRIMARY KS:

%GDOI-5-KS_SEND_UNICAST_REKEY: Sending Unicast Rekey for group G1 from

address 10.1.11.2 with seq # 11

All the GMs:

%GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from 10.1.11.2 to

10.1.20.2 with seq # 11


Missing RSA Key Symptoms

When GM registers to the KS, the following

message shows up in the syslog:

%GDOI-4-GM_RE_REGISTER: The IPSec SA created for group G1 may have

expired/been cleared, or didn't go through. Re-register to KS.

%GDOI-1-KS_NO_RSA_KEYS: RSA Key - get : Not found, Required for group

G

As a result KS will not send rekey messages, and GM will re-register when the keys expire

KS1 KS2

GM2 GM1

MPLS/Private IP


Missing RSA Key on the KS Troubleshooting Steps

Check whether KS is sending out the rekeys or not:

KS needs RSA keys to sign the rekey messages;

check logs for clues and/or verify the RSA keys

KS1#show crypto gdoi ks rekey

Group G1 (Multicast)

Number of Rekeys sent : 0

Number of Rekeys retransmitted : 0

KEK rekey lifetime (sec) : 86400

Retransmit period : 10

Number of retransmissions : 2

IPSec SA 1 lifetime (sec) : 3600

Remaining lifetime (sec) : 166

Number of registrations after rekey : 22

No rekeys sent


Missing RSA Key on the KS Troubleshooting Steps (Cont.)

Verify RSA key configuration on the KS:

KS1#show running | section gdoi group



server local

rekey address ipv4 102


rekey authentication mypubkey rsa get

sa ipsec 1

profile gdoi-p


no replay


Verify the RSA key pair name on the router:

KS1#show crypto key mypubkey rsa | include name

Key name: key1

Key name: key1.server

Labeled RSA key not present


Missing RSA Key on the KS Solution

Generate the required RSA key pair

KS1(config)#crypto key generate rsa label get exportable modulus 1024

The name for the keys will be: getvpn-rsa-key

% The key modulus size is 1024 bits

% Generating 1024 bit RSA keys, keys will be exportable...[OK]

Verify rekey messages are now being sent on the KS

%GDOI-5-KS_SEND_UNICAST_REKEY: Sending Unicast Rekey for group G1

from address 10.1.11.2 with seq # 1

KS1#show crypto gdoi ks rekey

Group G1 (Unicast)

Number of Rekeys sent : 1

<SNIP>


Multicast Rekey Issues Multicast Rekeys Failing - Symptom

GM is not getting the multicast rekey messages and therefore

continue to re-register with the KS

Rekey starts to work when switched from multicast rekey to

unicast rekey

Possible Causes

‒ Packet delivery issue within the multicast routing infrastructure

‒ End-to-end multicast routing enabled?

‒ mVPN service provided by the MPLS core provider?


Multicast Rekey Failing Troubleshooting

Check KS to verify multicast rekey messages

are being sent

Make sure ICMP is excluded from the KS encryption policy and is used as a tool to test multicast

%GDOI-5-KS_SEND_MCAST_REKEY: Sending Multicast Rekey for group

G1 from address 10.1.11.2 to 226.1.1.1 with seq # 6

KS1#ping 226.1.1.1

Type escape sequence to abort.

Sending 1, 100-byte ICMP Echos to 226.1.1.1, timeout is 2

seconds:

Reply to request 0 from 10.1.21.2, 44 ms No response from GM1 (10.1.20.2)

KS1 KS2

GM2 GM1

Multicast Network

10.1.20.2 10.1.21.2


Multicast Rekey Failing Troubleshooting

Check the multicast forwarding path

Check the PIM neighbor

WAN#sh ip pim neighbor

PIM Neighbor Table

Neighbor Interface Uptime/Expires Ver DR

Address Prio/Mode

10.1.11.2 Serial0/0 01:03:54/00:01:16 v2 1 / S

10.1.21.2 Serial3/0 01:13:06/00:01:26 v2 1 / S

WAN#show ip mroute 226.1.1.1

<snip>

(10.1.11.2, 226.1.1.1), 00:13:18/00:02:56, flags: T

Incoming interface: Serial0/0, RPFnbr 0.0.0.0

Outgoing interface list:

Serial3/0, Forward/Sparse-Dense, 00:13:18/00:00:00 Verify the OIL


Multicast Rekey Failing Solution

Enable PIM on the WAN router towards the GM

WAN(config)#int s2/0

WAN(config-if)#ip pim sparse-dense-mode

WAN(config-if)#end

%PIM-5-NBRCHG: neighbor 10.1.20.2 UP on interface Serial2/0

(vrf default)

Check multicast routing path again

Re-test with multicast ping

Verify GM now receives the multicast rekey messages


Unicast Rekey Failing Transient Network Issues

Due to transient changes in the network, unicast

rekey packets might not make it to the GM(s)

If the GMs does not receive the rekey, it will have

to re-register

Symptoms:

Missing Following syslog on GM:

%GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from 10.1.11.2 to

10.1.21.2 with seq # 3

GM shows re-registration syslog:

%GDOI-4-GM_RE_REGISTER: The IPSec SA created for group G1 may have

expired/been cleared, or didn't go through. Re-register to KS.

%CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.11.2 for group G1

using address 10.1.20.2


Unicast Rekey Failing Troubleshooting and Solution

Verify whether the rekeys are not being sent, not being received or

not being processed

KS:

show crypto gdoi ks members

Group Member Information :

Number of rekeys sent for group G1 : 380


Group ID : 3333

Group Name : G1

Key Server ID : 10.1.11.2

Rekeys sent : 1

Rekeys retries : 0

Rekey Acks Rcvd : 0

Rekey Acks missed : 0

GM:

show crypto gdoi gm rekey

Group G1 (Unicast)

Number of Rekeys received (cumulative) : 0

Number of Rekeys received after registration : 0

Number of Rekey Acks sent : 0

Rekey (KEK) SA information :

dstsrcconn-id my-cookie his-cookie

New : 10.1.20.2 10.1.11.2 1098 44F7FC328302AC61

Current : 10.1.20.2 10.1.11.2 1098 44F7FC328302AC61

Previous: --- --- --- --- ---

Always configure retransmissions to overcome transient issues

rekey retransmit 30 number 3

Make sure UDP port 848 is not blocked in the data path

Unicast rekey dropped


Rekey Fails Signature Validation

Primary KS fails, GM receives rekey from

secondary KS, but receives error:

*Apr 27 18:18:19.511: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI mode

failed with peer at 10.1.12.2

Syslog is not conclusive, let’s see what we can get with some debugs

GM1# debug crypto isakmp

Crypto ISAKMP debugging is on

GM1#

GM1# debug crypto gdoi

GDOI Generic Debug level: (Error, Terse)

*Apr 27 18:18:19.251: ISAKMP (0:1014): received packet from 10.1.12.2 dport 848

sport 848 Global (R) GDOI_REKEY

*Apr 27 18:18:19.251: GDOI:INFRA:(G1:0:1014:HW:0):Received Rekey Message!

*Apr 27 18:18:19.259: GDOI:INFRA:(G1:0:1014:HW:0):Signature Invalid! status = 13

*Apr 27 18:18:19.259: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI mode failed

with peer at 10.1.12.2

*Apr 27 18:18:19.259: ISAKMP: Receive GDOI rekey: Processing Failed. IKMP error = 6

Signature validation failed!


Rekey Fails Signature Validation Solution

Problem:

‒ Secondary KS has its own RSA key pair instead of the

exported key pair from the primary

‒ To verify, compare the RSA key pairs

KS1 KS2

GM2 GM1

MPLS/Private IP KS#show crypto key mypubkey rsa

KS1(config)#crypto key generate rsa modulus 1024 exportable label key1

KS2(config)#crypto key import rsa key1 pem terminal <passphrase>

Solution:

Generate exportable RSA key pair on the primary KS

Export RSA key pair to all secondary KSs



IKE Setup


Key Renewal—Rekey



Encryption Policy



Control Plane Replay Check Detection

Control Plane messages can carry time sensitive information and therefore

require replay protection

‒ Rekey messages from KS to GM

‒ COOP Announcement messages between KSs

Sequence number check to protect against replayed messages

Pseudotime check to protect against delayed messages with TBAR

enabled

Control Plane Replay check added in IOS version 12.4(15)T10,

12.4(22)T3, 12.4(24)T2, 15.0(1)M, and later


Control Plane Replay Check Code interoperability issue

Problem: customer upgraded IOS on a GM to 15.0(1)M for a bug fix,

and started to experience KEK rekey failures

The following errors are observed in the syslog

%GDOI-3-GDOI_REKEY_SEQ_FAILURE: Failed to process rekey seq # 1 in seq

payload for group G1, last seq # 11

%GDOI-3-GDOI_REKEY_FAILURE: Processing of REKEY payloads failed on GM

10.1.13.2 in the group G1, with peer at 10.1.11.2

%CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI mode failed with peer at

10.1.11.2


Control Plane Replay Check Code interoperability issue - solution

KS does not support control plane replay detection, and resets the rekey

sequence # for KEK rekey

GM interprets that as a replayed rekey message

Solution is to upgrade the KS to an IOS version that also supports the

control plane replay detection

New behavior

*Apr 6 15:41:26.932: %GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from

10.1.11.2 to 10.1.13.2 with seq # 8

GM1#

*Apr 6 15:42:01.940: %GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from

10.1.11.2 to 10.1.13.2 with seq # 1

TEK Rekey with seq# reset

KEK Rekey


Control Plane Replay Check – IOS Upgrade

procedure

Recommended IOS releases

‒ IOS: 15.1(4)M4

‒ IOS-XE: 15.1(3)S3

IOS upgrade procedure

‒ Step 1. Upgrade a secondary KS first, wait until COOP KS election is completed

‒ Step 2. Repeat step 1 for all secondary KS

‒ Step 3. Upgrade primary KS

‒ Step 4. Upgrade Group Members



IKE Setup


Key Renewal—Rekey



Encryption Policy



Control Plane Fragmentation Issues COOP Announcement Packets

In a large network (1500+ GMs), COOP update packet becomes larger than the default maximum buffer size

Default huge buffer size is 18024 bytes

Syslog message appears on the KSs:

Tune buffers to increase huge buffers and add buffers to permanent list:

buffers huge permanent 10

buffers huge size 65535

Symptoms:

%SYS-2-GETBUF: Bad getbuffer, bytes= 18872 -Process= "Crypto IKMP", ipl= 0, pid= 183


Control Plane Fragmentation Issues (cont.) COOP Announcement Packets

Large ANN messages are fragmented in transit between KSs

Can have up to 40+ IP fragments

One dropped fragment -> entire ANN dropped

How to identify?

Frag1

Frag2

Frag3

Frag4

FragN

KS1 KS2

%GDOI-3-COOP_KS_UNREACH: Cooperative KS 10.1.11.1 Unreachable in group G1.

KS1#sh ip traffic | section Frags

Frags: 10 reassembled, 3 timeouts, 0 couldn't reassemble

0 fragmented, 0 fragments, 0 couldn't fragment

Need to look at transit path features that may drop fragments, Firewall, VFR, reassembly buffer size, etc.


Troubleshooting GETVPN Data Plane

Ultimately all problems manifest at the data plane -

―my user application is not working over GETVPN!‖

But where really is the problem?

Control Plane

‒ Events that lead up to SAs getting installed on the GMs

Data plane

‒ Policy downloaded with SAs installed but traffic is not flowing


Generic IPSec Data Plane Troubleshooting

Need to have complete understanding of the forwarding path and how to

checkpoint it

Which device is the culprit, encrypting or decrypting router?

In which direction is the problem happening, ingress or egress?

Some syslogs may help reveal data plane drops

‒ Data plane errors are typically rate limited

‒ Common errors include replay, authentication failures

Heavily dependent upon show commands and counters to trace the

packet path

Sniffer capture of limited use due to encryption, however

‒ ESP-NULL – same crypto processing except packets not encrypted

‒ DSCP coloring of packets to uniquely identify a flow


GETVPN Data Plane

IPSec tunnel mode just like IPSec classic so most IPSec

troubleshooting techniques still apply, however…

Symmetrical encryption policy requirement

Unique challenges with Header Preservation

‒ PMTUD

Time Based Anti-Replay

‒ Extra encapsulation overhead – Fragmentation boundary

condition calculation

‒ Timer Based Anti-Replay failure


Data Plane Troubleshooting Tools

Interface counters

Encryption/decryption counters

Netflow

IP Accounting

ACL

DSCP packet coloring

Embedded Packet Capture (EPC)


IPSec Data Plane Packet Flow Checkpoints

Encrypting GM

‒ 1. Ingress LAN interface

Input ACL

Ingress Netflow

Embedded Packet Capture

‒ 2. Crypto engine

show crypto ipsec sa

show crypto session detail

‒ 3. Egress WAN interface

Egress Netflow


Output IP precedence accounting

4 3

GM2 GM1

Client Server

1

2

6

5

Decrypting GM

4. Ingress WAN interface

Input ACL

Ingress Netflow


Input IP precedence accounting

5. Crypto engine

show crypto ipsec sa

show crypto session detail

6. Egress WAN interface

Egress Netflow


Traffic Direction

Private WAN


Importance of a “Controlled Test”

The case for ―ping x.x.x.x timeout 0‖

Separation from background traffic

‒ Poor man’s conditional filter

‒ Packet coloring/marking

‒ Tools to monitor based on DSCP/Precedence marking

‒ ESP-NULL

IP characteristics for seemingly application issues

‒ Ping works but TCP doesn’t?

‒ Why does IPSec care about TCP, or does it?


Encrypting GM Data Plane Flow

Verify clear traffic being received with Ingress Netflow

Verify encryption operation performed

Lack of per-flow granularity

interface Ethernet0/0

ip address 192.168.13.1 255.255.255.0

ip flow ingress

!

GM1#show ip cache flow

<snip>

SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts

Et0/0 192.168.13.2 Se1/0 192.168.14.2 06 E443 0017 11

TCP port 23 = telnet

GM1#show crypto session detail

<snip>

IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0

Active SAs: 4, origin: crypto map

Inbound: #pkts dec'ed 162 drop 0 life (KB/Sec) 0/146

Outbound: #pkts enc'ed 170 drop 0 life (KB/Sec) 0/146


Encrypting GM Data Plane Flow – Cont.

Verify encrypted traffic existing GM with egress Netflow

interface Serial/0

ip address 10.1.13.2 255.255.255.252

ip flow egress

!


<snip>


Et0/0 192.168.13.2 Se1/0* 192.168.14.2 32 EE5B 2BEF 170

GM1#show crypto ipsec sa

interface: Serial1/0

<snip>

current outbound spi: 0xEE5B2BEF(3998952431)

Protocol 50 = ESP

Active IPSec SA SPI

If per L4 flow granularity is desired, can use inbound precedence coloring and egress precedence accounting


Decrypting GM Data Plane Flow

Verify encrypted traffic arriving on GM with Netflow


<snip>


Se1/0 192.168.13.2 Et0/0 192.168.14.2 32 EE5B 2BEF 170

Inbound IPSec SA SPI

Protocol 50 = ESP

Verify traffic decryption

GM2#show crypto session detail

<snip>

IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0

Active SAs: 10, origin: crypto map

Inbound: #pkts dec'ed 170 drop 0 life (KB/Sec) 0/150

Outbound: #pkts enc'ed 162 drop 0 life (KB/Sec) 0/150

Verify clear traffic forwarding post decryption


<snip>


Se1/0 192.168.13.2 Et0/0* 192.168.14.2 06 E6CC 0017 170

TCP port 23 = telnet


GETVPN Common Issues – Data Plane

Other data plane issues common to IPSec

Fragmentation/Path MTU

Asymmetrical Encryption Policy


KS Policy Issues Data Plane Traffic Failure

Encryption policies (what needs to be encrypted)

are defined centrally at the KS

Symmetrical ACLs should be defined to either permit

or to deny traffic from getting encrypted

If the traffic is not being encrypted or being blocked, verify we have

symmetrical ACL

GM2 GM1

MPLS/Private IP

Eth 0/0: 192.168.20.0/24 Eth 0/0: 192.168.21.0/24

KS Access-list

ip access-list extended ENCPOL

permit ip 192.168.20.0 0.0.0.255 192.168.21.0 0.0.0.255

permit ip 192.168.21.0 0.0.0.255 192.168.20.0 0.0.0.255


Fragmentation Issues PMTU Discovery

Large packets with the DF bit set may get black-holed in the GETVPN

network

GM2 GM1

MTU 1500 MTU 1500

MTU 1000

1400B 1460B

ICMP 3/4

Server sends a large packet with the DF bit set in an attempt to perform network PMTUD


PMTUD and GETVPN

Encrypting GM adds IPSec overhead and forwards it

Intermediate router drops the packet and sends back icmp3/4 to

perform PMTUD, two possibilities

‒ This ICMP dropped by the encrypt GM because it’s not encrypted based on

the encryption policy

‒ This ICMP gets forwarded to the end host but gets dropped due to

unauthenticated payload

Bottom line: PMTUD does not work with the current header

preservation implementation of GETVPN


PMTUD and GETVPN

Solution

Implement ip tcp adjust-mss to reduce the TCP packet segment size

Clear the DF bit in the encapsulating header


ip address 192.168.13.1 255.255.255.0

ip policy route-map clear-df-bit

!

route-map clear-df-bit permit 10

match ip address 111

set ip df 0

!

access-list 111 permit tcp any any

Encrypting GM

DF=1 Data

DF=0 Data DF=0

User Traffic


IPSec drop due to packet corruption

IPSec integrity check makes IPSec packets a lot more sensitive to

packet corruption in the network

Packet corruption symptoms

%CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=695

local=192.168.14.2 remote=192.168.13.2 spi=7C4E759F seqno=00000001

How to prove packets are corrupted in the network?

Enable EPC to capture packets into a circular buffer on both GMs

Use EEM(Embedded Event Manager) to

Synchronize and stop the capture on both routers when the RECVD_PKT_MAC_ERR message is logged

Notify the network operator by email

Retrieve both captures to examine for packet corruption


GETVPN Troubleshooting Summary

Have a clear and concise problem description

Try to break the problem down to either control or data plane

Understand the expected protocol flow on the control plane and how to check for them

Understand where/how to checkpoint the data plane

Syslog is your friend

There is always TAC!


Complete Your Online

Session Evaluation Give us your feedback and you

could win fabulous prizes.

Winners announced daily.

Receive 20 Passport points for each

session evaluation you complete.

Complete your session evaluation

online now (open a browser through

our wireless network to access our

portal) or visit one of the Internet

stations throughout the Convention

Center.

Don’t forget to activate your

Cisco Live Virtual account for access to

all session material, communities, and

on-demand and live activities throughout

the year. Activate your account at the

Cisco booth in the World of Solutions or visit

www.ciscolive.com.

113

http://www.ciscolive.com


Final Thoughts

Get hands-on experience with the Walk-in Labs located in World of

Solutions, booth 1042

Come see demos of many key solutions and products in the main Cisco

booth 2924

Visit www.ciscoLive365.com after the event for updated PDFs, on-

demand session videos, networking, and more!

Follow Cisco Live! using social media:

‒ Facebook: https://www.facebook.com/ciscoliveus

‒ Twitter: https://twitter.com/#!/CiscoLive

‒ LinkedIn Group: http://linkd.in/CiscoLI

114

http://www.ciscolive365.com/

https://www.facebook.com/ciscoliveus

https://twitter.com/

http://linkd.in/CiscoLI

Appendix

GETVPN Scalability and Troubleshooting

Tools


Key Server Scalability

Platform Crypto Card Max Number of GM Time to register to KS

7200/7201 VAM2+ 2000 15 sec *

3845 AIM-VPN/SSL-3 1000 15 sec *

3825 AIM-VPN/SSL-3 500 15 sec




7200/PKI VAM2+ 1000 20 sec **

* GM registration was distributed over two KSs to reduce the registration time

** GM registration was distributed over four KSs to reduce the registration time


GM Performance Attributes

(No Features)

PPS Mbps Max IMIX Latency(ms) Avg 100 pps Latency (ms)

871 Anti-Replay 3150 28 <10

No Anti-Replay 3232 28 <5

1841-onboard Anti-Replay 3506 33 <20


1841-aim/ssl Anti-Replay 8420 84 <10










0.34

0.33

0.25

1.18

1.07

0.68

0.47



(No Features) PPS Mbps Max IMIX Latency(ms) Avg 100 pps Latency (ms)

3825-onboard Anti-Replay 35,505 283 <1

No Anti-Replay 35,500 283 <5

3825-aim/ssl Anti-Replay 44,170 199 <1


3845-onboard Anti-Replay 46,028 284 <5


3845-aim/ssl Anti-Replay 54,020 200 <1


7200-g1vam2+ Anti-Replay 60,592 266 <5


7200-g2vam2+ Anti-Replay 121,952 283 <5


7200-g2/vsa Anti-Replay

No Anti-Replay 160,000 980 TBD

ASR1000/FP5G Anti-Replay 440,000

No Anti-Replay 470,000 1,890 TBD

ASR1000/FP10G Anti-Replay 976,000 4,200

No Anti-Replay 1,011,000 4,220 <0.270

ASR1000/FP20G Anti-Replay 2,655,000 TBD

No Anti-Replay 2,685,000 8,530 <0.0150.001

0.64

0.66

0.19

TBD

TBD

0.17

0.76

0.81

0.69



(No Features)

Frame Size ASR 1004

(10Gig)

7200 VSA 3845 AIM-

VPN/SSL-3

ISRG2

3945 Onboard

Crypto

ISRG2

2951 Onboard

Crypto

ISRG2

1941 Onboard

Crypto

1400 Byte

4759 Mbps

925 Mbps 200 Mbps

820 Mbps

268 Mbps 154 Mbps

IMIX (90 Bytes 61%, 594

bytes 24%, 1418 15%)

2289 Mbps 780 Mbps 177 Mbps 261Mbps 160 Mbps 64Mbps


GETVPN Verification Common KS Syslog Messages

Syslog Messages Explanation

COOP_CONFIG_MISMATCH The configuration between the primary key server and secondary key server are

mismatched.

COOP_KS_ELECTION The local key server has entered the election process in a group.

COOP_KS_REACH The reachability between the configured cooperative key servers is restored.

COOP_KS_TRANS_TO_PRI The local key server transitioned to a primary role from being a secondary server in

a group.

COOP_KS_UNAUTH An authorized remote server tried to contact the local key server in a group. Could

be considered a hostile event.

COOP_KS_UNREACH The reachability between the configured cooperative key servers is lost. Could be

considered a hostile event.

KS_GM_REVOKED During rekey protocol, an unauthorized member tried to join a group. Could be

considered a hostile event.

KS_SEND_MCAST_REKEY Sending multicast rekey.

KS_SEND_UNICAST_REKEY Sending unicast rekey.

KS_UNAUTHORIZED During GDOI registration protocol, an unauthorized member tried to join a group.

Could be considered a hostile event.

UNAUTHORIZED_IPADDR The registration request was dropped because the requesting device was not

authorized to join the group.


Syslog Messages Explanation

GM_CLEAR_REGISTER The clear crypto gdoi command has been executed by the local group

member.

GM_CM_ATTACH A crypto map has been attached for the local group member.

GM_CM_DETACH A crypto map has been detached for the local group member.

GM_RE_REGISTER IPSec SA created for one group may have been expired or cleared.

Need to reregister to the key server.

GM_RECV_REKEY Rekey received.

GM_REGS_COMPL Registration complete.

GM_REKEY_TRANS_2_MULTI Group member has transitioned from using a unicast rekey mechanism

to using a multicast mechanism.

GM_REKEY_TRANS_2_UNI Group member has transitioned from using a multicast rekey

mechanism to using a unicast mechanism.

PSEUDO_TIME_LARGE A group member has received a pseudotime with a value that is largely

different from its own pseudotime.

REPLAY_FAILED A group member or key server has failed an anti-replay check.

GETVPN Verification Common GM Syslog Messages


Packet marking Techniques

IP TOS byte copied from inner header to the encapsulating

delivery header by default

How to mark

‒ PBR

‒ MQC

‒ Local ping

How to monitor

‒ IP precedence accounting

‒ ACL counters


ToS/Precedence/DSCP Reference Chart

7 5 6 4 3 2 1 0

IP Precedence Priority

DSCP

Least Significant Bit

ToS Byte

Hex - Decimal ToS

IP Precedence DSCP Binary

20

00

40

48

E0

C0

B8

A0

88

68

60

32

0

64

72

224

192

184

160

128

104

96

1 Priority

0 Routine

2 Immediate

7 Network Control

5 Critical

4 Flash Override

3 Flash

8 CS1

0 Dflt

16 CS2

18 AF21

56 CS7

48 CS6

46 EF

40 CS5

32 CS4

26 AF31

24 CS3

00100000

00000000

01000000

01001000

11100000

11000000

10111000

10100000

10000000

01101000

01100000

6 Internetwork Control

80

136 34 AF41 10001000


Packet marking - Examples

PBR

MQC


ip policy route-map mark

!

access-list 150 permit ip host 172.16.1.2 host 172.16.254.2

!

route-map mark permit 10

match ip address 150

set ip precedence flash-override

class-map match-all my_flow

match access-group 150

!

policy-map marking

class my_flow

set ip precedence 4

!


service-policy input marking

IP flow in question marked with precedence 4


Packet marking - Examples

Router Ping

Router#pingip

Target IP address: 172.16.254.2

Repeat count [5]: 100

Datagram size [100]:

Timeout in seconds [2]:

Extended commands [n]: y

Source address or interface:

Type of service [0]: 128

Set DF bit in IP header? [no]:

Validate reply data? [no]:

Data pattern [0xABCD]:

Loose, Strict, Record, Timestamp, Verbose[none]:

Sweep range of sizes [n]:

Type escape sequence to abort.

Sending 100, 100-byte ICMPEchos to 172.16.254.2, timeout is 2 seconds:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


Packet marking - Monitoring

IP Precedence Accounting

Interface ACL


ip address 192.168.1.2 255.255.255.0

ip accounting precedence input

middle_router#show interface precedence

Ethernet0/0

Input

Precedence 4: 100 packets, 17400 bytes

middle_router#sh access-list 144

Extended IP access list 144

10 permit ip any any precedence routine

20 permit ip any any precedence priority

30 permit ip any any precedence immediate

40 permit ip any any precedence flash

50 permit ip any any precedence flash-override (100 matches)

60 permit ip any any precedence critical

70 permit ip any any precedence internet (1 match)

80 permit ip any any precedence network


Using Packet Captures for Data Plane Issues

Packet captures can provide detailed packet information at the

bits/bytes level

The new packet capture infrastructure introduced in 12.4(20)T

makes this easy to do

‒ Ability to capture IPv4 and IPv6 packets in the CEF path

‒ Configurable capture buffer and capture point parameters

‒ Extensible output filtering and export capabilities

‒ Support for various WAN encapsulation types


Using IOS Embedded Packet Captures

Router#monitor capture buffer test-buffer

Router#monitor capture buffer test-buffer filter access-list 120

Filter Association succeeded

Router#

Router#monitor capture point ipcef test-capture serial 2/0 both

*Mar 26 20:33:10.896: %BUFCAP-6-CREATE: Capture Point test-capture created.

Router#monitor capture point associate test-capture test-buffer

Router#monitor capture point start test-capture

*Mar 26 20:34:03.108: %BUFCAP-6-ENABLE: Capture Point test-capture enabled.

Router#

Router#monitor capture point stop test-capture

*Mar 26 20:34:21.636: %BUFCAP-6-DISABLE: Capture Point test-capture disabled.

Key Configuration Steps

Create the capture buffer and capture point

Associate the capture point to the buffer

Start/stop the capture


Using IOS Embedded Packet Captures Now we have the packets captured, what’s next?

Router# show monitor capture buffer test-buffer dump

15:34:07.228 EST Mar 26 2009 : IPv4 LES CEF : Se2/0 None

05CECE30: 0F00080045C0002C ....E@.,

05CECE40: 6D170000FE0649DD 02010102 01010114 m...~.I]........

05CECE50: 0017A3530FB6B9523EF1499C 60121020 ..#S.69R>qI.`..

05CECE60: 917A0000 02040218 00 .z.......

Router# monitor capture buffer test-buffer export?

ftp: Location to dump buffer

http: Location to dump buffer

https: Location to dump buffer

rcp: Location to dump buffer

scp: Location to dump buffer

tftp: Location to dump buffer

Dump the packet on the router itself

Dump the packet on the router itself

Or export it out and analyze it in Wireshark


Use EEM and EPC to catch Packet Corruption

event manager applet detect_bad_packet

event syslog pattern "RECVD_PKT_MAC_ERR"

action 1.0 cli command "enable"

action 2.0 cli command "monitor capture point stop test"

action 3.0 syslog msg "Packet corruption detected and capture stopped!"

action 4.0 snmp-trap intdata1 123456 strdata ""

Peer1

Peer2

event manager applet detect_bad_packet

event snmp-notification oid 1.3.6.1.4.1.9.10.91.1.2.3.1.9. oid-val "123456"

op eq src-ip-address 20.1.1.1

action 1.0 cli command "enable"

action 2.0 cli command "monitor capture point stop test"

action 3.0 syslog msg "Packet corruption detected and capture stopped!"

t-shoot GET

Documents

Transcript of t-shoot GET