Troubleshooting OSPF -...

Troubleshooting OSPF

BRKRST-3310

Faraz Shamim Technical Leader

Patrick Viscosi Network Consulting Engineer

© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3310 Cisco Public

Agenda

LSA Overview

Troubleshooting Commands

Common Issues

OSPF for NX-OS and IOS-XR

Troubleshooting Enhancements

3

LSA Overview


LSA Type Review

Type LSA

1 Router

2 Network

3 Summary Network

4 Summary ASBR

5 External

6 Group Membership

7 NSSA

8 External Attributes

9–11 Opaque

5


R3#show ip ospf database router 3.3.3.3 Router Link States (Area 1)

LS age: 0 Always 0 at origination Options: (No TOS-capability) LS type: Router Links This is a router LSA Link State ID: 3.3.3.3 Router ID of R3 Advertising Router: 3.3.3.3 Router ID of R3 LS Seq Number: 80000001 Initial Seq Number Checksum: 0x146B Length: 48 Area Border Router bit B = 1 Number of links: 2 Link Connected to: a Transit Network (Link ID) Designated Router address): 192.1.1.4 (Link Data) Router Interface address): 192.1.1.3 Number of TOS metrics: 0 TOS 0 Metric: 1 Link Connected to: a Stub Network (Link ID) Network/subnet number: 192.1.4.0 (Link Data) Network Mask: 255.255.255.0 Number of TOS metrics = 0 TOS 0 Metric: 2

Router LSA of R3 for Area 1

192.1.1.3

Area 0

R3

DR

R4 1

192.1.1.4

192.1.4.0/24

8

18.10.0.5/30

2

6.6.6.6

R6

6


Router LSA of R3 for Area 0 (Cont.)

192.1.1.3

Area 0

R3

DR

R4 1

192.1.1.4

192.1.4.0/24

8

18.10.0.5/30 6.6.6.6

2 R6

LS age: 0 Always 0 at origination Options: (No TOS-capability) LS type: Router Links This is a router LSA Link State ID: 3.3.3.3 Router ID of R3 Advertising Router: 3.3.3.3 Router ID of R3 LS Seq Number: 80000001 Initial Seq Number Checksum: 0x146B Length: 48 Area Border Router bit B = 1 Number of links: 2 Link Connected to: another Router (point-to-point)

(Link ID) Neighboring Router ID: 6.6.6.6 (Link Data) Router Interface address: 18.10.0.5 Number of TOS metrics: 0 TOS 0 Metric: 8 Link Connected to: a Stub Network (Link ID) Network/subnet number: 18.10.0.4 (Link Data) Network Mask: 255.255.255.252 Number of TOS metrics = 0 TOS 0 Metric: 8

Router Link States (Area 0)

7


Router Link Types Summary

Type Description Link ID Link Data

1 Point-to-Point

Numbered Neighbor’s

RID Interface IP

Address

1 Point-to-Point Unnumbered

Neighbor’s RID

MIB-II Ifindex Value

2 Transit IP Address of the DR

Interface IP Address

3 Stub IP Network

Number Subnet Mask

4 Virtual Link Neighbor’s

RID Interface IP

Address

8


R3#show ip ospf database network 192.1.1.4 Network Link States (Area 1) LS age: 948 Options: (No TOS Capability) LS type: Network Links Link State ID: 192.1.1.4 Interface address of DR Advertising Router: 4.4.4.4 RID of DR LS Seq Number: 80000001 Initial Seq Number Checksum: 0x9A84 Length: 40 Network Mask: /24 Attached Router: 4.4.4.4 Attached Router: 3.3.3.3 Attached Router: 2.2.2.2 Attached Router: 1.1.1.1

Network LSA for 192.1.1.0

192.1.1.

3

Area 0

192.1.4.0/24

18.10.0.4/32

6.6.6.6

R3

192.1.1.

4

1.1.1.1

2.2.2.2

R2

/24

1

2

8

8

DR

R4 R1

R6

4.4.4.4

9


Area 0

R4#show ip ospf database summary 192.1.2.0 Summary Net Link States (Area 0) LS age: 1514 Options: (No TOS Capability) LS type: Summary Links (Networks) Link State ID: 192.1.2.0 (Summary Network Number) Advertising Router: 4.4.4.4 RID of ABR LS Seq Number: 80000001 Initial Seq Number Checksum: 0xFC68 Length: 28 Network Mask: /24 TOS 0 Metric: 4

Type 3 Details

192.1.1.

3

192.1.4.0/24

18.10.0.4

6.6.6.6

R3

192.1.1.

4

1.1.1.1

2.2.2.2

R2

/24

1

2

8

8 192.1.2.0/24

ABR 192.1.2.0/24

metric 4

O IA

DR

R4 R1

3

R6

ABR

10


R4#show ip ospf database asbr-summary 7.7.7.7 Summary ASB Link States (Area 0) LS age: 1548 Options: (No TOS Capability) LS type: Summary Links (AS Boundary Router) Link State ID: 7.7.7.7 (AS Boundary Router address) Advertising Router: 4.4.4.4 RID of ABR LS Seq Number: 80000001 Initial Seq Number Checksum: 0x93E2 Length: 28 Network Mask: /0 TOS: 0 Metric: 16

Type 4 Details

1.1.1.1

Area 0

192.1.4.0/24 R2

192.1.2.0/2

4

DR

R4 R1

R7

RID ASBR

7.7.7.7

8

R3 2 18.10.0.4

6.6.6.6 192.1.1.

3

192.1.1.

4

2.2.2.2

/24

1

8

3 140.10.0.0 External Route

Type 4 Summary

R6

11


Type 5 Details

1.1.1.1

Area 0

192.1.4.0/24 R2

192.1.2.0/2

4

DR

R4 R1

R7

RID ASBR

7.7.7.7

8

R3 2 18.10.0.4

6.6.6.6 192.1.1.

3

192.1.1.

4

2.2.2.2

/24

1

8


4

External Type 5 R6

R4#show ip ospf database external 140.10.0.0 LS age: 156 Options: (No TOS Capability) LS type: AS External Link Link State ID: 140.10.0.0 (External Network Number) Advertising Router: 7.7.7.7 Router ID of ASBR LS Seq Number: 80000001 Initial Seq Number Checksum: 0x93E2 Length: 36 Network Mask: /16 Metric Type: 2 bit E = 1 -> O E2 (Default) Metric: 20 Forwarding address: 0.0.0.0

12


Forwarding address

ASBR

OSPF

Network

router ospf 1

redistribute bgp 1 subnets

BGP Gateway

Router

OSPF is

running

on this

interface

R1

Must be known via intra or inter area route

13


R7#show ip ospf database nssa-external 140.10.0.0 LS age: 156 Options: (No TOS Capability, Type 7/5 translation) LS type: AS External Link Link State ID: 140.10.0.0 (External Network Number) Advertising Router: 7.7.7.7 Router ID of R7(ASBR) LS Seq Number: 80000001 Initial Seq Number Checksum: 0x01A6 Length: 36 Network Mask: /16 Metric Type: 2 Metric: 20 Forwarding address: 0.0.0.0

1.1.1.1

Area 0

192.1.4.0/24 R2

192.1.2.0/24

DR

R4 R1

8

R3 2 18.10.0.4

6.6.6.6 192.1.1.

3

192.1.1.

4

2.2.2.2

/24 1

8


4

NSSA

RID ASBR

7.7.7.7 R7

Type 7 R6

Type 7 Details

14

Troubleshooting Commands What to look for in the output?


R3#show ip ospf

Routing Process "ospf 1" with ID 3.3.3.3 and Domain ID 0.0.0.1

Supports only single TOS(TOS0) routes

Supports opaque LSA

It is an area border router

SPF schedule delay 5 secs, Hold time between two SPFs 10 secs

Minimum LSA interval 5 secs. Minimum LSA arrival 1 secs

Number of external LSA 1. Checksum Sum 0x3B57

Number of opaque AS LSA 0. Checksum Sum 0x0

Number of DCbitless external and opaque AS LSA 0

Number of DoNotAge external and opaque AS LSA 0

Number of areas in this router is 2. 2 normal 0 stub 0 nssa

External flood list length 0

Area BACKBONE(0)

Number of interfaces in this area is 2

Area has no authentication

SPF algorithm executed 2773 times

Area ranges are

Number of LSA 17. Checksum Sum 0x686B5

Number of opaque link LSA 0. Checksum Sum 0x0

Number of DCbitless LSA 0

Number of indication LSA 0

Number of DoNotAge LSA 9

Flood list length 0

Show IP OSPF

16


Show IP OSPF interface brief

R12#sh ip ospf int br

Interface PID Area IP Address/Mask Cost State Nbrs

Lo0 1 0 172.23.192.241/32 1 LOOP 0/0

Gi0/0 1 0 11.11.11.1/24 1 DR 1/1

Lo91 1 1 91.91.91.91/32 1 LOOP 0/0

Se2/0 1 1 172.17.17.1/30 48 P2P 1/1

17


R3#show ip ospf database

OSPF Router with ID (3.3.3.3) (Process ID 1)


Link ID ADV Router Age Seq# Checksum Link count

3.3.3.3 3.3.3.3 106 0x80000009 0xC3F1 3

. . .

Summary Net Link States (Area 0)

Link ID ADV Router Age Seq# Checksum

18.10.0.0 7.7.7.7 3 (DNA) 0x80000008 0x3DC2

18.10.0.0 8.8.8.8 1396 0x80000004 0x27D8

. . .



1.1.1.1 1.1.1.1 671 0x80000016 0xE6CD 2 . . .

Show IP OSPF Database

18


Show IP OSPF Database Database-Summary

R3#show ip ospf database database-summary


Area 0 database summary

LSA Type Count Delete Maxage

Router 6 0 0

Network 4 0 0

Summary Net 10 0 0

Summary ASBR 0 0 0

Type-7 Ext 0 0 0

Opaque Link 0 0 0

Opaque Area 0 0 0

Subtotal 20 0 0

Area 1 database summary

LSA Type Count Delete Maxage

Router 4 0 0

Network 1 0 0

Summary Net 10 0 0

Summary ASBR 4 0 0

. . . 19


R3#show ip ospf neighbor

Neighbor ID Pri State Dead Time Address

Interface

1.1.1.1 1 FULL/DROTHER 00:00:33 192.1.1.1

FastEthernet0/0

2.2.2.2 1 FULL/DROTHER 00:00:32 192.1.1.2

FastEthernet0/0

4.4.4.4 1 FULL/DR 00:00:39 192.1.1.4

FastEthernet0/0

6.6.6.6 1 FULL/ - 00:00:38 18.10.0.6

Serial0/0

R3#

Show IP OSPF Neighbor

192.1.1.

3

DR

R4

192.1.4.0/24

18.10.0.4

6.6.6.6

R3

192.1.1.

4

1.1.1.1

2.2.2.2

R1

R2

/24 1

2

8

R6

4.4.4.4

Area 0

20


R3#show ip ospf neighbor detail

Neighbor 1.1.1.1, interface address 192.1.1.1

In the area 1 via interface FastEthernet0/0

Neighbor priority is 1, State is 2WAY, 2 state changes

DR is 192.1.1.4 BDR is 192.1.1.2

Options is 0x2

Dead timer due in 00:00:39

Neighbor is up for 00:06:30

Index 0/0, retransmission queue length 0, number of retransmission 0

First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)

Last retransmission scan length is 0, maximum is 0

Last retransmission scan time is 0 msec, maximum is 0 msec


In the area 1 via interface FastEthernet0/0

Neighbor priority is 1, State is FULL, 6 state changes

DR is 192.1.1.4 BDR is 192.1.1.2

Options is 0x42



Index 2/2, retransmission queue length 0, number of retransmission 0

First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)



Show IP OSPF Neighbor Detail

21


R3#show ip ospf interface

FastEthernet0/0 is up, line protocol is up

Internet Address 192.1.1.3/24, Area 1

Process ID 1, Router ID 3.3.3.3, Network Type BROADCAST, Cost: 1

Transmit Delay is 1 sec, State DROTHER, Priority 1

Designated Router (ID) 4.4.4.4, Interface address 192.1.1.4

Backup Designated router (ID) 2.2.2.2, Interface address 192.1.1.2

Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5

Hello due in 00:00:03

Index 1/1, flood queue length 0

Next 0x0(0)/0x0(0)

Last flood scan length is 0, maximum is 5

Last flood scan time is 0 msec, maximum is 0 msec

Neighbor Count is 3, Adjacent neighbor count is 2

Adjacent with neighbor 2.2.2.2 (Backup Designated Router)

Adjacent with neighbor 4.4.4.4 (Designated Router)

Suppress hello for 0 neighbor(s)

Show IP OSPF Interface

22


R3#show ip ospf virtual-links

Virtual Link OSPF_VL0 to router 4.4.4.4 is up

Run as demand circuit

DoNotAge LSA allowed.

Transit area 1, via interface FastEthernet0/0, Cost of using 1

Transmit Delay is 1 sec, State POINT_TO_POINT,

Timer intervals configured, Hello 10, Dead 40, Wait 40,

Retransmit 5

Hello due in 00:00:09

Adjacency State FULL (Hello suppressed)

Index 1/3, retransmission queue length 0, number of

retransmission 1

First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)



R3#

Show IP OSPF Virtual-Links

23


Show IP OSPF Stat Detail

R1#show ip ospf stat detail OSPF process ID 1 ------------------------------------------ Area 0: SPF algorithm executed 1 times SPF 1 executed 1w0d ago, SPF type Full SPF calculation time (in msec): SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total 0 0 0 0 0 0 0 0 LSIDs processed R:1 N:0 Stub:1 SN:0 SA:0 X7:0 Change record R, LSIDs changed 1 Last 10 LSIDs: 4.4.4.4(R) Summary OSPF SPF statistic SPF calculation time Delta T Intra D-Intra Summ D-Summ Ext D-Ext Total Reason 1w0d 0 0 0 0 0 0 0 R, 24


R3#show ip ospf border-routers

OSPF Process 1 internal Routing Table

Codes: i - Intra-area route, I - Inter-area route

i 4.4.4.4 [1] via 192.1.1.4, FastEthernet0/0, ABR, Area 0, SPF 42

i 4.4.4.4 [1] via 192.1.1.4, FastEthernet0/0, ABR, Area 1, SPF 38

i 8.8.8.8 [10] via 18.10.0.6, Serial0/0, ABR/ASBR, Area 0, SPF 42

i 7.7.7.7 [17] via 192.1.1.4, FastEthernet0/0, ABR/ASBR, Area 0, SPF 42

Show IP OSPF Border-Routers

25


Other Show Commands

R3#show ip ospf database self-originate




3.3.3.3 3.3.3.3 1520 0x80000015 0xABFD 2



192.1.1.0 3.3.3.3 1520 0x80000006 0x4E1A

192.1.2.0 3.3.3.3 1521 0x80000006 0x6103

. . .



3.3.3.3 3.3.3.3 1536 0x80000028 0x612D 2

26


Other Show Commands (Cont.)

R3#show ip ospf database adv-router 7.7.7.7




7.7.7.7 7.7.7.7 871(DNA) 0x8000000D 0x8FE2 2



20.10.0.0 7.7.7.7 871 (DNA) 0x8000000A 0x39C4

Type-5 AS External Link States

Link ID ADV Router Age Seq# Checksum Tag

140.100.0.0 7.7.7.7 1944 0x80000004 0x3759 0

27


Look for input queue drops in show interface

R33#sh int gig0/0 GigabitEthernet0/0 is up, line protocol is up Internet address is 33.33.33.2/24 MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Keepalive set (10 sec) Full Duplex, 1Gbps, link type is auto, media type is T output flow-control is XON, input flow-control is XON ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters never Input queue: 5/75/1032/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 30 second input rate 7259000 bits/sec, 602 packets/sec 30 second output rate 7258000 bits/sec, 602 packets/sec 771310 packets input, 149868367 bytes, 0 no buffer Received 740151 broadcasts

28


Try to inspect traffic in the input queue if you see input queue drops incrementing

R33# sh buffer input-interface gig0/0 header Buffer information for Big buffer at 0x2A5BEBF4 data_area 0xC49ED84, refcount 1, next 0x402CBA18, flags 0xA80 linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1 if_input 0x401620E0 (GigabitEthernet0/0), if_output 0x0 (None) inputtime 16w2d (elapsed 00:00:00.004) outputtime 00:00:00.000 (elapsed never), oqnumber 65535 datagramstart 0xC49EDCA, datagramsize 1514, maximum size 1692 mac_start 0xC49EDCA, addr_start 0xC49EDCA, info_start 0x0 network_start 0xC49EDD8, transport_start 0xC49EDEC, caller_pc 0x211295A4 source: 33.33.33.1, destination: 33.33.33.2, id: 0x370F, ttl: 255, prot: 1

Malicious traffic destined for the interface of the router can cause legitimate OSPF traffic to be dropped

29


Use extended pings to verify link integrity


Neighbor ID Pri State Dead Time Address Interface

172.23.192.241 1 FULL/DR 00:00:38 3.3.3.1 GigabitEthernet0/1

R11#ping

Target IP address: 3.3.3.1

Repeat count [5]: 1

Extended commands [n]: y

Set DF bit in IP header? [no]: y

Validate reply data? [no]: y

Data pattern [0xABCD]:

Loose, Strict, Record, Timestamp, Verbose[V]:

Sweep range of sizes [n]: y

Sweep min size [36]:

Sweep max size [18024]: 1500

Sweep interval [1]:

Type escape sequence to abort.

Sending 1465, [36..1500]-byte ICMP Echos to 3.3.3.1, timeout is 2 seconds:

Packet sent with the DF bit set

Reply data will be validated

Reply to request 0 (4 ms) (size 36)



30

Common Issues

Common things that can go wrong


Common Issues

Adjacency is not coming up

OSPF neighbor stuck in ? State

Information is in the database but not in the routing table

SPF running constantly

NSSA ABR not translating Type 7 LSA

GRE Tunnel OSPF adjacency is flapping

32


Adjacency Is Not Coming Up

Show IP OSPF neighbor

Show IP OSPF interface

Debug IP OSPF adjacency

Useful commands to troubleshoot this problem:

33



R3#

R3#show ip ospf interface serial 2

Serial2 is down, line protocol is down


Process ID 1, Router ID 3.3.3.3, Network Type POINT_TO_POINT,

Cost: 64

Transmit Delay is 1 sec, State DOWN,

Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5


Layer 2 is down

34



OSPF not enabled on the interface


R3#


Serial2 is up, line protocol is up

OSPF not enabled on this interface

In more recent IOS, we see:


%OSPF: OSPF not enabled on serial 2 Tip: Check for the wrong network statement; re-enter the network statement 35


R3#show ip ospf neighbor R3# R3#show ip ospf interface e0 Ethernet0 is up, line protocol is up Internet Address 192.1.1.3/24, Area 1 Process ID 1, Router ID 3.3.3.3, Network Type BROADCAST, Cost: 10 Transmit Delay is 1 sec, State DR, Priority 1 Designated Router (ID) 192.1.1.4, Interface address 192.1.1.3 No backup designated router on this network Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5 No Hellos (Passive interface)


Interface is defined as passive

36



Mismatched subnet mask


R3#

R3#debug ip ospf adj

OSPF adjacency events debugging is on

R3#

OSPF: Mismatched hello parameters from 192.1.1.4

Dead R 40 C 40, Hello R 10 C 10 Mask R 255.255.255.192 C 255.255.255.0

37



Mismatched hello/dead interval

Tip: Default is 10 seconds on LAN


R3#



R3#

OSPF: Mismatched hello parameters from 192.1.1.4

Dead R 40 C 40, Hello R 15 C 10 Mask R 255.255.255.0 C

255.255.255.0

R4(config-if)#interface ethernet 0

R4(config-if)#no ip ospf hello-interval 15

38



Mismatched authentication key

Tip: Watch for the “space” at the end of the authentication key

Be sure encryption Type matches (i.e. MD5, ClearText)


R3#



R3#

OSPF: Rcv pkt from 192.1.1.4, Ethernet0 : Mismatch Authentication Key -

ClearText

39



R3#

Scenario 1 - Neighbor is in Area 2, but we are in Area 1:

R3#debug ip ospf adjacency


OSPF: Rcv pkt from 192.1.1.4, Ethernet0, area 0.0.0.1 mismatch area 0.0.0.2 in

the header

Scenario 2 - Neighbor is in Area 0, but we are not:

%OSPF-4-ERRRCV: Received invalid packet: mismatch area ID, from

backbone area must be virtual-link but not found from 192.1.1.4, Ethernet0


Mismatched area ID

40



Mismatched transit/stub/NSSA option


R7#



OSPF: Hello from 18.10.0.2 with mismatched Stub/Transit area option bit

7.7.7.7 8.8.8.8

R8 R7 Area 2

Area 2

NSSA

18.10.0.2 18.10.0.1

41


Options

Normal area: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0xBC4 opt 0x2 flag 0x3 len 492

E bit is 1, Allow externals, option: 0x2(HEX) = 00000010(Bin)

Stub area: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x1866 opt 0x0 flag 0x3 len 372

E bit is 0, no external allowed, options: 0x0 = 00000000

NSSA: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x118 opt 0x8 flag 0x3 len 372

N/P bit is on, options: 0x8 = 00001000

DC: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x1A1E opt 0x20 flag 0x3 len 392

DC bit is negotiated, options: 0x20 = 00100000

O DC EA N/P MC E * *

42


Show IP OSPF neighbor

Debug IP OSPF adjacency

OSPF Neighbor Stuck In ? State

Useful commands for troubleshooting this problem:

43


OSPF Adjacency States

3.3.3.3 6.6.6.6

R3 R6

ATTEMPT

INIT

2-WAY

EXSTART

EXCHANGE

LOADING

FULL

44


RID =6.6.6.6

N = 3.3.3.3

Hello

Stuck In ATTEMPT

3.3.3.3 6.6.6.6

RID =3.3.3.3

Hello

R3 R6

NBMA

45


Stuck In ATTEMPT

Our hellos are getting lost in NBMA cloud

Neighbor hellos are getting lost in NBMA cloud

We received neighbor’s hello but reject it for some reason

Misconfigured neighbor statement

Broken Unicast

Reasons:

46


RID =2.2.2.2

Hello

Stuck In INIT

2.2.2.2

RID =1.1.1.1

Hello

R2

RID =1.1.1.1

Hello

N =2.2.2.2

RID =2.2.2.2

Hello

1.1.1.1

R1

47


Stuck In INIT

One side is blocking the hello packet with access-list

One side is translating (NAT) OSPF hello

One side multicast capability is broken (Layer 2)

Dialer map or Frame Relay map is missing keyword ‘broadcast’

LLS capability issue when Cisco IOS® is upgraded

Reasons:

48


Verify IP Multicast control plane is working

We want to see a reply from the remote OSPF interface when we ping 224.0.0.5:



172.23.192.241 1 FULL/DR 00:00:33 31.31.31.1 GigabitEthernet0/1

R11#ping 224.0.0.5

Type escape sequence to abort.

Sending 1, 100-byte ICMP Echos to 224.0.0.5, timeout is 2 seconds:

Reply to request 0 from 31.31.31.1, 1 ms

49


Stuck In 2-WAY

1.1.1.1 2.2.2.2

RID =1.1.1.1 P=0

Hello

RID =1.1.1.1 P=0

Hello

N =2.2.2.2

R2 R1

RID =2.2.2.2 P=0

Hello

N =1.1.1.1

50


Stuck In 2-WAY

This is normal in broadcast network types

This is to reduce the amount of flooding on the wire

Problem can happen if all the routers are configured with priority equal to ‘0’

Only low-end routers should be configured with priority 0 so they don’t participate in DR election

Reasons:

51


Stuck In EXSTART/EXCHANGE

3.3.3.3 6.6.6.6

RID =3.3.3.3

Hello R6 R3

RID =6.6.6.6

Hello

N =3.3.3.3

MTU = 1500 flag = 0x7

DBD

Seq = 1E55

MTU = 1500 flag = 0x7

DBD

Seq = 22AB

52



Useful in debugging, defines I, M and MS bits

– OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0xBC4 opt 0x2 flag 0x3 len 492

– Flag 0x7 --> 111 means I(Initial) = 1, M = 1(More), MS = 1(Master)

– Flag 0x6 --> 110 not possible



– Flag 0x3 --> 011 means master has more data to send

– Flag 0x2 --> 010 means slave has more data to send

– Flag 0x1 --> 001 means master has no more data left to send

– Flag 0x0 --> 000 means slave has no more data left to send

0 0 0 0 0 I M MS

53



MTU mismatch—EXCHANGE on one router and EXSTART on the other - can configure ‘ip ospf mtu-ignore’

Neighbor RID is same as ours—EXSTART

– Note: If Cisco IOS is > 12.0.7, it displays msg: %OSPF-3-DUP_RTRID and OSPF neighbor list will be empty

Unicast is broken—EXCHANGE

1. Wrong VC/DLCi mapping in frame/ATM environment in highly redundant network

2. MTU problem, can’t ping across with more than certain length packet

3. Access-list blocking unicast; after two-way OSPF send unicast packet except p2p links

4. NAT is translating unicast packet

Reasons:

54


Stuck In LOADING

3.3.3.3 6.6.6.6

R6 R3

LS Type

LS Req

Link State ID

Advertising Router

# LSAs

LS Update

. . .

LSAs

55


Stuck In LOADING

LS request is being made and neighbor is sending bad packet or memory corruption exists

– a. Do show IP OSPF bad to see bad LSA

– b. Show log will show OSPF-4-BADLSATYPE message

LS request is being made and neighbor is ignoring the request

MTU mismatch problem (RFC 1583 and 2178 compatibility issue) CSCee23634. OSPF should detect if the neighbor MTU is smaller than ours. Fixed in more recent IOS®

Reasons:

56


Information is in the OSPF DB but not in the RT

Show IP OSPF interface <interface>

Show IP OSPF database <x>

Where ‘x’ can be router, network, summary, summary-asbr, external, NSSA

Useful Commands for This Problem:

57


Mismatched Network Types

Area 0

R3

3.3.3.3

R6

6.6.6.6

18.10.0.4/30




Process ID 1, Router ID 3.3.3.3, Network Type POINT_TO_POINT, Cost: 64




Process ID 1, Router ID 6.6.6.6, Network Type BROADCAST, Cost: 64



6.6.6.6 1 FULL/ - 00:00:30 18.0.0.6 Serial0

58


R6 18.10.0.4/30

Area 0

R3

3.3.3.3 6.6.6.6

Mismatched Network Types (Cont.)

R3#show ip ospf database router 3.3.3.3

. . .

Link ID = 6.6.6.6 Router id of the neighbor

Link Data = 18.10.0.5 IP interface address

Type = 1 This is a point-to-point link

# TOS metrics = 0

metric = 8

. . .


. . .

Link ID = 18.10.0.6 IP address of the DR

Link Data = 18.10.0.6 Interface address

Type = 2 This is a transit link

# TOS metrics = 0

metric = 8

59


Different Mask or IP Subnet on P2P Links

Area 0

R3

3.3.3.3

R6

6.6.6.6

19.10.0.5/30 18.10.0.6/30

R3#show interface serial 0


Internet address is 19.10.0.5/30






6.6.6.6 1 FULL/ - 00:00:30 18.0.0.6 Serial0

60


More recent IOS has a network/mask check

Area 0

R3

3.3.3.3

R6

6.6.6.6

19.10.0.5/30 18.10.0.6/30







R3#debug ip ospf adjacency


R3#

OSPF: Rcv pkt from 18.10.0.6, Serial0, area 0.0.0.0 :

src not on the same network

61


R6

Area 0

R3

3.3.3.3 6.6.6.6

Different Mask or IP Subnet on P2P Links (Cont.)


. . .




# TOS metrics = 0

metric = 8

. . .


. . .




# TOS metrics = 0

metric = 8

62


Address Flipped on Dual Links

R3

3.3.3.3

R6

6.6.6.6

18.10.0.20/30

Area 0

18.10.0.4/30 S0

S1



6.6.6.6 1 FULL/ - 00:00:30 18.10.0.6 Serial0

6.6.6.6 1 FULL/ - 00:00:33 18.10.0.22 Serial1




63


ABR2 ABR1 R2 R1 Area 1

Area 0 Area 0

Discontiguous Backbone

R1 and R2 are not be able to see each other

Summary LSA for inter-area routes must not be generated into the backbone

The solution is to create virtual link between ABR1 and ABR2

Summary LSA

Summary LSA

Area 0

64


SPF Running Constantly

Show IP OSPF stat detail

Show IP OSPF database

Show IP OSPF neighbor <interface> detail

Useful commands for troubleshooting this problem:

65



LSA Flaps Due To:

– Duplicate Router ID/IP address

– Constant link flapping in an area

Reasons:

66


Show IP OSPF Stat

R1#show ip ospf stat detail OSPF process ID 1 ------------------------------------------ Area 0: SPF algorithm executed 1 times SPF 1 executed 00:00:12 ago, SPF type Full SPF calculation time (in msec): SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total 0 0 0 0 0 0 0 0 LSIDs processed R:1 N:0 Stub:1 SN:0 SA:0 X7:0 Change record R, LSIDs changed 1 Last 10 LSIDs: 4.4.4.4(R) Summary OSPF SPF statistic SPF calculation time Delta T Intra D-Intra Summ D-Summ Ext D-Ext Total Reason 1w0d 0 0 0 0 0 0 0 R,

67


R3#debug ip ospf monitor

OSPF: Schedule SPF in area 1

Change in LS ID 1.1.1.1, LSA type R,

OSPF: schedule SPF: spf_time 0ms wait_interval

861421816s

OSPF: Begin SPF at 0x33585480ms, process time 752ms

spf_time 0ms, wait_interval 861421816s

OSPF: End SPF at 0x33585488ms, Total elapsed time 8ms

Intra: 4ms, Inter: 0ms, External: 0ms


68



R3#show ip ospf database




3.3.3.3 3.3.3.3 106 0x80000009 0xC3F1 3

. . .



18.10.0.0 7.7.7.7 3 (DNA) 0x80000008 0x3DC2

18.10.0.0 8.8.8.8 1396 0x80000004 0x27D8

. . .



1.1.1.1 1.1.1.1 2 0x80000016 0xE6CD 2

. . . 69


Show ip ospf neighbor <interface> detail

R3#sh ip ospf nei gig0/1 det


In the area 0 via interface GigabitEthernet0/1

Neighbor priority is 1, State is FULL, 6 state changes

DR is 3.3.3.2 BDR is 3.3.3.1

Options is 0x12 in Hello (E-bit, L-bit)

Options is 0x52 in DBD (E-bit, L-bit, O-bit)

LLS Options is 0x1 (LR)



Index 1/1, retransmission queue length 0, number of

retransmission 0

First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)



70


Area 0

router ospf 1

network 10.10.10.10 0.0.0.0 area 0

RID: 7.7.7.7

Type 7–5 Conversion

NSSA ABR Not Translating Type 7 LSA

Only the NSSA ABR with the highest RID does the conversion

No Type 7/5 Translation

Area 1

NSSA Type 7

R8 RID: 8.8.8.8

NSSA

ABR

NSSA

ASBR

71


Area 0

NSSA

ABR

Area 1

NSSA

router ospf 1

network 10.10.10.10 0.0.0.0 area 0

R8 RID: 8.8.8.8

NSSA

ASBR

Type 7

NSSA ABR Not Translating Type 7 LSA

Only NSSA ABR with the highest RID does the conversion


RID: 7.7.7.7

NSSA

ABR

72


Area 0

NSSA

ABR

Area 1

NSSA

R8 RID: 8.8.8.8

NSSA

ASBR

Type 7

New NSSA command to select NSSA translator

In 15.2T and 15.2M IOS, you can now hardcode an NSSA LSA Translator:

R7(config)#router ospf 1

R7(config-router)#area 1 nssa translate type7 always

RID: 7.7.7.7

NSSA

ABR

R9 RID:

9.9.9.9


73


GRE Tunnel OSPF Adjacency is going up and down

GRE Tunnel destination must not be learned via the Tunnel

R1#sh run int tunnel1

interface Tunnel1

ip address 7.7.7.1 255.255.255.252

ip ospf cost 1

tunnel source GigabitEthernet0/0

tunnel destination 2.2.2.2

R2 R1 Area 0 Area 0

1.1.1.2 1.1.1.1

R3 Area 0

2.2.2.2 2.2.2.1

Area 0

74


Tunnel destination learned via Tunnel tears down the Tunnel and the OSPF neighbor

R1#sh ip ospf nei


7.7.7.2 0 FULL/ - 00:00:39 7.7.7.2 Tunnel1

R1#sh ip route | incl Tunnel1

O 2.2.2.0 [110/11] via 7.7.7.2, 00:00:01, Tunnel1

*May 15 18:00:26.375: %TUN-5-RECURDOWN: Tunnel1 temporarily disabled due to

recursive routing

*May 15 18:00:26.375: %OSPF-5-ADJCHG: Process 1, Nbr 7.7.7.2 on Tunnel1 from FULL to

DOWN, Neighbor Down: Interface down or detached

*May 15 18:00:26.379: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1,

changed state to down

75

Cisco NX-OS OSPF on the Nexus Platforms


You must first enable the OSPF feature in NX-OS before OSPF commands can be entered:

Nexus(config)# feature ospf

Feature OSPF

77


Quick view of ospf config

Nexus# sh run ospf

!Command: show running-config ospf

!Time: Sun Apr 14 20:38:59 2013

version 6.1(3)

feature ospf

router ospf 100

interface Ethernet5/2

ip router ospf 1 area 0.0.0.0

78


‘show ip route’ command has different look

Nexus# sh ip route

IP Route Table for VRF "default” '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric]

1.1.1.1/32, ubest/mbest: 1/0

*via 9.9.9.2, Eth1/1, [110/41], 08:03:05, ospf-100, inter

5.5.5.5/32, ubest/mbest: 1/0

*via 9.9.9.2, Eth1/1, [110/41], 05:43:23, ospf-100, intra

79


OSPF event-history feature cont.

Nexus# sh ip ospf event-history ?

adjacency Adjacency formation logs cli Cli logs event Internal event logs flooding LSA flooding logs ha HA and GR logs lsa LSA generation and database logs msgs IPC logs redistribution Redistribution logs spf SPF calculation logs statistics Show the state and size of the buffers

80

Cisco IOS-XR OSPF on the IOS-XR Platforms


Cisco IOS-XR Comparison to IOS OSPF

IOS-XR introduces a hierarchical CLI for ease of configuration

IOS-XR uses CLI inheritance to simplify configuration

Interface specific parameters are now configured under the area portion of the OSPF configuration

Show command CLI no longer uses the “IP” prefix. Example: show IP OSPF neighbor becomes show OSPF neighbor

82


IOS-XR Configuration

RP/0/RP0/CPU0:ios#show run router ospf 1

router ospf 1

log adjacency changes detail

router-id 1.1.1.1

network point-to-point

auto-cost reference-bandwidth 100000

area 0.0.0.0

interface Loopback0

!

interface GigabitEthernet0/3/0/2

cost 10000

83


IOS-XR Show Commands

IOS-XR show commands are very similar to IOS

RP/0/RP0/CPU0:ios#show ospf database ?

adv-router Advertising Router link states

asbr-summary ASBR summary link states

database-summary Summary of database

external External link states

network Network link states

nssa-external NSSA External link states

router Router link states

self-originate Self-originated link states

summary Network summary link states

84


IOS-XR Troubleshooting – Using Trace

IOS-XR incorporates a built-in debug called trace

RP/0/RP0/CPU0:ios#show ospf trace

Trace Name Size Count Description

1. adj 8192 2963 adjacency

2. adj_cycle 8192 72484 dbd/flood events/pkts

3. config 2048 204 config events

4. errors 8192 1017 errors

5. events 4096 300 mda/rtrid/bfd/vrf

6. ha 8192 309 startup/HA/NSF

7. hello 2048 2852464 hello events/pkts

8. idb 8192 862 interface

9. pkt 2048 1453478 I/O packets

10. rib 8192 1927 rib batching

11. spf 8192 8848 spf/topology

12. spf_cycle 8192 12310 spf/topology detail

13. te 4096 3418 mpls-te

14. test 1024 0 testing info

15. mq 256 15510 message queue info

85


IOS-XR Troubleshooting – Show Tech

RP/0/RP0/CPU0:ios# show tech routing ospf

Output is extensive. Use the “file” keyword to save data on local media.

86

Troubleshooting Enhancements Debugging made easier


Interface Scoped Debugging

Enhancement in limiting the OSPF debug output to selected interfaces

Example below will generate debug output for only two interfaces specified below – debug condition interface Ethernet 0/0 – debug condition interface Ethernet 1/0 – debug ip ospf hello – debug ip ospf adjacency Available starting in: 12.4(4)T, 12.2(30)S, 12.0(32)S – debug ip ospf flood Further enhancement to allow an access-list filter, and a “detail” option Available starting in: 12.4(4)T, 12.2(30)S, 12.0(32)S – debug ip ospf packets Available starting in: 12.4(10), 12.2(33)SXH, 12.2(33)SRB

88


OSPF Event Logging Feature

R5#sh ip ospf Routing Process "ospf 1" with ID 5.5.5.5 Supports only single TOS(TOS0) routes Supports opaque LSA Supports Link-local Signaling (LLS) Supports area transit capability Event-log enabled, Maximum number of events: 1000, Mode: cyclic

Enabled by default

Available in 15.x IOS code and later

Can be tuned using ‘event-log’ keyword under OSPF process

Built in ospf event history logging

89


OSPF Event Logging Feature cont.

R5#sh ip ospf event ?

generic Generic Events

interface Interface State Change Events

lsa LSA Arrival and LSA Generation Events

neighbor Neighbor State Change Events

reverse Show Events in Reverse Order

rib RIB Update, Delete and Redistribution Events

spf SPF Scheduling and SPF Run Events

| Output modifiers

<cr>

Built in ospf event history logging

90


OSPF Event Logging Feature cont.

R1#sh ip ospf event interface


22 *May 15 19:26:26.114: Interface Tunnel1 state changes from POINT_TO_POINT to DOWN

72 *May 15 19:26:06.114: Interface Tunnel1 state changes from DOWN to POINT_TO_POINT

96 *May 15 19:25:05.998: Interface Tunnel1 state changes from POINT_TO_POINT to DOWN

91


Summary

Overview of OSPF LSAs

Different troubleshooting commands and what to look for in those commands

Common issues in OSPF networks; e.g., adjacency problems, information in the database but not in RT, SPF problems and NSSA problems and how to correct those problems

OSPF on Other Platforms

OSPF Troubleshooting Enhancements

What Have We Learned?

92


More Information

White Papers

Web and Mailers

Cisco Press

CCO OSPF Page:

http://www.cisco.com/en/US/tech/tk365/tk480/tsd_technology_support_sub-protocol_home.html

93


Maximize your Cisco Live experience with your

free Cisco Live 365 account. Download session

PDFs, view sessions on-demand and participate in

live activities throughout the year. Click the Enter

Cisco Live 365 button in your Cisco Live portal to

log in.

Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily.

Receive 20 Cisco Daily Challenge points for each session evaluation you complete.

Complete your session evaluation online now through either the mobile app or internet kiosk stations.

94

Troubleshooting OSPF -...

Documents

Transcript of Troubleshooting OSPF -...