Carrier Grade NAT44 on IOS-XR Deployment...

Carrier Grade NAT44 on IOS-XR Deployment Experience BRKSPG-3334

Nicolas Fevrier Rajendra Chayapathi Syed Hassan

© 2014 Cisco and/or its affiliates. All rights reserved. BRKSPG-3334 Cisco Public

Agenda

Introduction

– NAT Principles and Mechanisms

– Bulk-Port Allocation

– Port limit

– Static Port Forwarding

– ALG

– Logging

Hardware

Deployment feedback

Routing consideration and Best Practices

Redundancy

3

INTRODUCTION


Introduction

5

Do you think CGN is evil?

– Yes but it’s a necessary one

IPv4 address exhaustion

End-to-end IPv6 traffic, are you ready?

The same cards can be used for:

– NAT44

– But also for smooth transition to IPv6

Let’s jump directly into the deep end


LIR are allocating their last blocks

– On 14 September 2012, the RIPE NCC started allocating from the last /8 of IPv4 addresses received from IANA

IPv4 grey/black market is flourishing

6

Facts About IPv4 Shortage


Cisco’s strategy relies on three pillars

Preserve (Business Continuity)

– NAT44 / CGN

– Optimize the IPv4 resource and allow growth

Prepare (Encourage Adoption)

– Offer IPv6 to the customers

– 6rd: transport IPv6 on top of a IPv4 infrastructure

Prosper (Interworking)

– DS-Lite, MAP-T/E: transport of the remaining IPv4 traffic on top of a IPv6 backbone

– NAT64: translate to the IPv4 at the border

Among IOS-XR products, the ISM and VSM (ASR9000) and CGSE and CGSE+ (CRS) cards are the tools used to build these three pillars.

7

Cisco’s Strategy: 3 Pillars


i2o / o2i: inside to outside / outside to input

NAT/NAPT: Network Address (and Port) Translation

CGx: carrier grade … (CGN: Carrier Grade NAT)

LSN: Large Scale NAT

ALG: Application Layer Gateway

GRT: Global Routing Table

SL/SF: Stateless/Stateful

8

Vocabulary


Example: we have 16 public addresses

9

4 4 4 4

Stateless translation 1 external IP : 1 internal IP

No multiplexing no DB needed

Stateful translation 1 external IP : n internal IP

Multiplexing DB to maintain

Translation Protocols Illustrated Stateful vs Stateless


Stateful

1:n translation (port multiplexing)

Needs translation DB maintenance

Logging scalability can be an issue

Need static port forwarding or PCP to accept o2i initiated sessions

May need ALGs

In case of failover, we need to re-establish sessions on a new device

Stateless

1:1 translation

i2o or o2i initiated sessions are treated equally

Allows asymmetrical traffic

Better convergence time

Potential “inline” implementation

No logging required

10

Stateless vs Stateful

Protocols: NAT64SL, 6rd, MAP-T

Protocols: NAT44, NAT64SF, DS Lite

© 2014 Cisco and/or its affiliates. All rights reserved. BRKSPG-3334 Cisco Public 11

NAT44 Introduction

Preserve the investment: buy time to prepare migration to IPv6

Not the “solution” but meets a vast majority of user current needs

NAT vs NAPT

Defined since 2001 (RFC3022, RFC4787, RFC5382, RFC5508)

– Unicast

– TCP/UDP/ICMP

– Stateful

Permit “multiplexing”: several internal hosts will use the same external address, maximizing the IPv4 resource

ALGs


Stateful translation protocol from an IPv4 space to another IPv4 space

IPv4 space public or private – Usually, from private (RFC1918) to public but not necessary

Translation table or database (DB) maintained on the CGN card

IPv4

Internet IPv4

Backbone

IPv4 Traffic

CGN

Source Address = 10.1.1.10

Outside Address = 170.0.0.1

NAT44 Overview


IPv4

Internet IPv4

Backbone

Double step stateful translation:

At CPE level – Between home network and ISP access network

At CGN level – Between ISP network and public address network

From CGN perspective: NAT44 = NAT444

IPv4 Traffic

CGN CPE

Source Address = 10.1.1.10

Outside Address = 170.0.0.1

Translated Address = 10.8.1.111

NAT444 or Double NAT44

NAT PRINCIPLES and MECHANISMS


Web Server: 5.20.3.2

Inside VRF Outside VRF NAT Engine

Source 10.10.10.2:2493

Destination 5.20.3.2:80

Source 100.2.1.24:8442


Translation table

10.10.10.2:2493100.2.1.24:8442 Logging

Record Syslog

Netflow

Collector

Inside VRF Outside VRF NAT Engine

Source 5.20.3.2:80


Web Server: 5.20.3.2 Web Client

10.10.10.2

Source 5.20.3.2:80

Destination 10.10.10.2:2493 Web Client

10.10.10.2

NAT Mechanisms


EIM: End-point Independent Mapping

– destination address and port for i2o traffic not tracked

– If multiple destinations but source address and port are the same no other entry created

– Sometime referred as “full cone NAT”

EDM: End-point Dependent Mapping

– Opposite of EIM

– Destination info is maintained in DB

Source Y:1430

Source X:4828

Dest B:80

Dest B:80

Dest A:80

Inside Outside Destination

X:4828 Y:1430 *

NAT Principles EIM/EIF vs EDM/EDF


EIF: End-point Independent Filtering

– Once entry is present in the table

– For o2i traffic, we don’t verify source address/port

– Better scalability and larger support

EDF: End-point Dependent Filtering

– Opposite of EIF

– Check the source addresses for o2i traffic

– Required in some situation: bill shock effect

Dest Y:1430

Source A:80

Dest X:4828

Source B:4234

Source C:80

Inside Outsid

e

Destinatio

n

X:482

8

Y:1430 *

NAT Principles EIM/EIF vs EDM/EDF


We use the same external IP address mapping for all sessions associated with the same internal IP address (RFC4787)

Each inside odd port is mapped to an outside odd port number

Each inside even port is mapped to an outside even port number

Inside

Outside

Inside Outsid

e

X:2104 A:1030

2

X:2334

2

A:11238

X:4827

1

A:1098

5

Y:29301 B:1045

Y:43017 B:1491

Y:1024 B:1228

Source X:2104

Source A:10302

Source X:23342

Source A:11238

Source Y:29301

Source Y:43017

Source Y:1024

Source B:1491

Source B:1045 Source

B:1228

Source X:48271

Source A:10985

NAT Principles Paired IP Address Assignment


Source X:2104

Source Y:11003

A:10302 B:11237

Inside

Outside

Two endpoints on inside NAT can communicate to each others using external NAT IPv4 addresses and ports.

NAT Principles Hair Pinning


First flow per Inside source address

CGN picks an Outside address that has at least 1/3 of its ports free

All subsequent Flows from that Inside source will use the same Outside address.

NAT

IP1 IP2 IP3 IP4 IP5 IP6 IP7 IP8

?

No

?

Ok

Used port

Free port

NAT Principles Address Allocation


If that Outside address is completely exhausted, then a random selection is made from the remaining addresses, repeated until an address is chosen or it is determined that none are available (which results in an ICMP error message)

NAT


?

No

?

?

NAT


ICMP error

Used port

Free port

NAT Principles Address Allocation


Ports are randomly picked from the list of available (unused) ports associated with the chosen Outside IP address

Each port is allocated once, regardless of which L4 protocol (UDP, TCP) is being used in the Flow

CGN creates a Translation binding (state) between

– Inside source IP address + port

and

– Outside source IP address + port

IP1

NAT ?

? Inside Outside

IPa:2104 IP1:10302

Used port

Free port

NAT Principles Port Allocation


If the randomly chosen port is already being used, the selection increments (around a ring) until an available port is found; if none are available then an ICMP error message is sent

If the Inside source already has a number of Flows equal to the configured per-user limit, then the allocation is rejected and an ICMP message is returned

IP1

ICMP error

?

Inside Outside

IPa:Pa IP1:P1

IPa:Pb IP1:P2

IPa:Pc IP1:P3

IPa:Pd IP1:P4

IPa:Pe IP1:P5

IPa:Pf IP1:P6

IPa:Pg IP1:P7

IPa:Ph IP1:P8

IPa:Pi No

IP1

port-limit=8

NAT

NAT Principles Port Allocation


Often referred as “Deterministic NAT”, coming in future releases

Opposite approach than random allocation mechanisms described before

Allows predictable mapping of source addresses/ports between the inside and outside world

Based on an algorithm, each internal address will be allocated an external address and range

Predefined NAT is still stateful (translations are still stored in DB)

Main benefit: logging is no longer necessary (but will still be possible)

Main flaw: sub-optimal address allocation

– Addresses and port ranges are allocated regardless of the presence or usage of the internal users

– To meet requirements of certain ALGs, it will be necessary to allocate contiguous ports

SDNAT (stateless) draft has been discontinued

Algorithm-based / Predefined NAT

BULK PORT ALLOCATION


Aims at reduces data generated by logging

Bulk port allocation behavior

– A subscriber creates the first connection

– N contiguous ports are pre-allocated (ex: 2064 to 2080 if N=16)

– Bulk-allocation message (NFv9 and/or syslog) is logged for the port-range

– Additional connections (up to N) will use one of the pre-allocated ports

– New pool allocated if subscriber creates > N concurrent connections

– Bulk-delete message is logged when subscriber terminates all sessions from pre-allocated pool

Outside

IP1

NAT

Syslog

Netflow

Collector

Logging

Record

Bulk Port Allocation


When bulk size is changed, all current dynamic translations will be deleted

Ports below dynamic start range (< 1024) are not allocated to bulk

It can take one of the following values:

– 16, 32, 64, 128, 256, 512, 1024, 2048, 4096 (8 in IOS XR 4.3.1)

– port-limit / 4 ≤ bulk-port-alloc ≤ port-limit x 2

Recommendation: closest value to half the port-limit

Orthogonal with Destination Based Logging, can NOT be configured together

Port range allocation is random, in following examples we picked 1024-1039 and 1040-1055 for the sake of simplicity only



IPv4 Traffic

CGN

Source Address = 10.1.1.1 Outside Address

from pool = 99.0.0.1

10.1.1.2

IPv4

Internet

NAT

BPA Illustrated Example Bulk=16


1 packet from 10.1.1.1 to 30.1.1.1:80 1

1

1 packet from 10.1.1.1 to 30.1.1.1:25 2

2

1 packet from 10.1.1.2 to 40.1.1.1:80 3

3

NAT IPv4

Internet

10.1.1.1

10.1.1.2



NAT IPv4

Internet

10.1.1.1

10.1.1.2

1 packet from 10.1.1.1 to 50.1.1.1:80 1

1 packet from 10.1.1.1 to 60.1.1.1:80 2

2

1



1 packet from 10.1.1.1 to 30.1.1.1:80 7

7

1 2 3

4 5 6

Same rules for init and active timeout apply for bulk ports


BPA=16 can reduce the logging volume MUCH more than by 16


With NAT444, it’s very likely that at least one device is connected behind the CPE at any given time

Consequently, logging for the port allocation is generated once and the port block is never de-allocated or de-allocated many weeks or months later

It’s exactly what the protocol is supposed to do, but it creates some issues

– Potential issue with logging collector correlator

– Another issue could be the security. It makes one CPU always use the same port range and reduces the scope for attackers

Workaround: DHCP lease time reduced to re-assign a different IP to the CPE every couple of weeks.



Config parser will enforce the selection respecting:

– 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096

– port-limit / 4 ≤ bulk-port-alloc ≤ port-limit x 2

Recommendation: closest value to half the port-limit

service cgn POC-1

service-type nat44 nat44-1

inside-vrf Inside-1

bulk-port-alloc size 256

Bulk Port Allocation: Configuration

PORT LIMIT


For stateful translation protocols (NAT44, NAT64 SF, DS Lite), each user can be assigned a maximum number of ports. It prevents a single user to consume all port resources

?

Inside Outside

IPa:Pa IP1:P1

IPa:Pb IP1:P2

IPa:Pc IP1:P3

IPa:Pd IP1:P4

IPa:Pe IP1:P5

IPa:Pf IP1:P6

IPa:Pg IP1:P7

IPa:Ph IP1:P8

IPa:Pi No

IP1

port-limit=8

NAT

Per-user Port Limit


Port-limit can be defined per protocol

But also per VRF

– allows different treatment for different type of customers

Finding the proper port-limit is a very tricky exercise

No simple rule of the thumb

– Different for each type of customer (ADSL, Mobile, Cable, Enterprise…)

– Different for each theater (Asia, Europe, Russia, Americas…)

Scripts can be used to collect average and maximum port usage

Per-user Port Limit


Exceeding the port limit will trigger a syslog message:

[Portblockrunout 17 10.1.11.202 ivrf- 2005 - - ]

– Portblockrunout: event name signifying the port limit hit event

– 17: it was hit by a UDP packet requesting the translation

– 10.1.11.202: is the subscribers private IP

– ivrf: name of the inside VRF

– 2005: private port number

These messages are throttled

– For 10.1.11.202, once we report this message, we will not repeat them for the same subscriber until it goes below 70% of max limit and then goes up again and hits the port limit

Per-user Port Limit on CGSE

Can be used to quickly user consuming a lot of ports


It’s a safety net preventing one user to use all resources

For stateful translation protocols each user can be assigned a maximum number of ports

– NAT44 and NAT64SF will use keyword “portlimit”

We can use every value between 1 to 65535, default is 100

Defined per protocol or globally since 4.3.1

service cgn demo

service-location preferred-active 0/1/CPU0


portlimit 512

inside-vrf iVRF1

portlimit 256

inside-vrf iVRF2

!

!

Configuring Port-Limit

STATIC PORT FORWARDING and PCP


Inside Outside TCP

state 0

IPv4

Internet

IPv4 Traffic

CGN

Map pool = 99.0.0.0/24

30.0.0.1

10.1.1.1

1

1 No entry in the NAT DB,

o2i packets are discarded

2

With stateful translation mechanisms, a traffic initiated from the outside will be discarded Static Port Forwarding or Port Control Protocol necessary

Session Initiated From the Outside ?


Inside Outside TCP

state

11.1.1.1:80 99.0.0.1:80 static

Inside Outside TCP

state 0

IPv4

Internet

IPv4 Traffic

CGN

Map pool = 99.0.0.0/24

30.0.0.1

10.1.1.1

service cgn demo

service-type nat44 nat1

inside-vrf insidevrf1

protocol tcp

static-forward inside address 10.1.1.1 port 80

1

2

3

4

Static-port-forwarding creates an entry in the NAT DB

Static Port Forwarding

6000 entries max


RP/0/RP0/CPU0:R#sh cgn demo inside-translation protocol tcp inside-vrf Inside inside-address

10.12.0.250 port s 10000 e 10000

Inside-translation details

---------------------------

CGN instance : demo

Inside-VRF : Inside

--------------------------------------------------------------------------------------------

Outside Protocol Inside Outside Translation Inside Outside

Address Source Source Type to to

Port Port Outside Inside

Packets Packets

--------------------------------------------------------------------------------------------

100.0.0.58 tcp 10000 15819 static 0 0

RP/0/RP0/CPU0:R#

External address is picked by the system, not the user (based on hashing of inside address)

Verifying Static-Port-Forwarding


IPv4

Internet

IPv4 Traffic

CGN

Map pool = 99.0.0.0/24 30.0.0.1

10.1.1.1

Host on

public

network PCP Server

PCP client

on private network

PCP allows applications to create mappings from an external IP address+proto+port to an internal IP address+proto+port

PCP Server is a software instance via which clients request and manage explicit mappings

PCP Client issues requests to a server

A PCP Client can issue PCP requests on behalf of a third party device

A PCP request is transported on UDP(v4/v6) packet with destination port 5351

Supported on CGSE cards for NAT44, NAT64 and DS-Lite

http://tools.ietf.org/html/draft-ietf-pcp-base-29

Port Control Protocol


Inside Outside TCP

state 0

IPv4

Internet

IPv4 Traffic

CGN

Map pool = 99.0.0.0/24

10.1.1.1

MAP Request 1

99.0.0.1 TCP 80

MAP Response 2

Inside Outside TCP state

10.1.1.1:80 99.0.0.1:80 pcp_explicit

3

0: SUCCESS


10.1.1.1:80 99.0.0.1:80 pcp_explicit

5

4 FIN or RST



Inside Outside TCP

state

10.1.1.1:80 99.0.0.1:80 dynamic

0

IPv4

Internet

IPv4 Traffic

CGN

Map pool = 99.0.0.0/24

10.1.1.1

MAP Request 1

99.0.0.1 TCP 80

MAP Response 2

11:CANNOT_PROVIDE_EXTERNAL

Available external port: 84

Other result codes could be: – 1:UNSUPP_VERSION

– 2:NOT_AUTHORIZED

– 3:MALFORMED_REQUEST

– 4:UNSUPP_OPCODE

– 5:UNSUPP_OPTION

– 6:MALFORMED_OPTION

– 7:NETWORK_FAILURE

– 8:NO_RESOURCES

– 9:UNSUPP_PROTOCOL

– 10:USER_EX_QUOTA

– 11:CANNOT_PROVIDE_EXTERNAL

– 12 ADDRESS_MISMATCH

– 13:EXCESSIVE_REMOTE_PEERS

PCP Req/Resp



Inside Outside TCP

state 0

IPv4

Internet

IPv4 Traffic

CGN

Map pool = 99.0.0.0/24

10.1.1.1

PEER Request 1

99.0.0.1 TCP 80

PEER Response 2


10.1.1.1:80 99.0.0.1:80 pcp_implicit

3

0: SUCCESS

4 FIN or RST

Inside Outside TCP

state 5

DB entry removed


APPLICATION LAYER GATEWAYS


ALG are features allowing upper layer inspection to track a particular behavior (port negotiation, …) and make sure the protocol will be unaffected by the translation

Cisco’s position is to discourage the pursue of ALGs

– Applications are regularly rewritten and keeping track of each change is challenging

– NAT traversal is more generally handled at the application level

Supported ALGs in CGN cards

– Active FTP (passive FTP doesn’t need ALG)

– RTSP (used for some streaming services)

– PPTP (for legacy VPN applications)

Need for ALG


In active mode FTP

– the client connects from a random unprivileged port (N > 1023)

– to the FTP server's command port 21

– then, client starts listening to port N+1

– and sends the FTP command PORT N+1 to the FTP server

– the server will then connect back to the client's specified data port from its local data port, which is port 20

ALG converts the network Layer address information found inside an application payload

Note: Passive FTP Mode does NOT need any ALG…

Active FTP ALG


Real-Time Streaming Protocol is not a streaming protocol

It’s a remote control protocol for streamers (which use RTP/RTCP or RDT)

a text-based protocol based on “methods” (like requests) and transported on port554

RTSP “session” is not a connection per say since it’s not tied to a transport-level connection, even if transported by TCP

Our implementation considers the server is located “outside” and clients are “inside”

RTSP is used in many streamers like QuickTime or RealNetworks (less and less used with generalization of HTML5)

RTSP ALG


Point to Point Tunneling Protocol is used by legacy VPN solutions

Encapsulate PPP packets in IP GRE

Translation of PPTP packet is challenging because we don’t translate source ports but a peer caller ID field contained in the GRE header

PAC: PPTP Access Concentrator, in the public side (Outside)

PNS: PPTP Network Server, in the private side (Inside)

NAT IPv4

Internet PAC PNS

Control Connection (TCP1723)

PPTP

PPTP ALG


We currently support three ALGs types for NAT44 (none for NAT64SF and only FTP for DS Lite)

– ActiveFTP (not needed for PassiveFTP)

– RSTP (for Real Audio G2 and windows media player), default port is 554

– PPTP (for legacy VPN systems)

service cgn demo


alg ActiveFTP

alg rtsp port 10000

alg pptpAlg

!

Configuring ALGs


RP/0/RP0/CPU0:R#sh cgn demo inside-translation protocol tcp inside-vrf Inside inside-address

10.13.0.29 port s 1 e 65535


---------------------------

CGN instance : demo

Inside-VRF : Inside

--------------------------------------------------------------------------------------------




Packets Packets

--------------------------------------------------------------------------------------------

100.0.0.221 tcp 1043 41493 dynamic 51 55

100.0.0.221 tcp 55000 26236 dynamic 6 5

100.0.0.221 tcp 55001 16300 dynamic 6 5

100.0.0.221 tcp 55002 28942 alg 23 22

100.0.0.221 tcp 55003 4373 dynamic 5 5

RP/0/RP0/CPU0:R#

When a translation database entry will be allocated based on ALG, it will appear like:

Verify ALG Activity

LOGGING


Entries in NAT table are of temporary nature

Any Stateful protocol (NAT44, NAT64SF, DS-Lite) requires logging

Directive 2006/24/EC - Data Retention: EU Law

Logging preserves the mapping information between an internal and external

CGSE and ISM cards supports Netflow v9 and Syslog

NAT IPv4

Internet

Syslog

Netflow

Logging

Record

Need for Logging


Source IP address and port translation history

– to be able to reliably identify the private IP translated to public IP at one precise moment

– further inspection of RADIUS or DHCP database can be performed to provide the ‘identity’ of subscriber (e.g. MAC address of device or username)

Format of the information (as long as translation can be ‘inverted’ based on the input parameters):

– ASCII format

– Compressed text/binary files or relational database that contain translation history details

– Outcome of an algorithmic mapping of private IP address to public IP address/port

What CGN information needs to be stored by ISPs ?


No definitive and easy answer

The logging solutions

– Dynamic NAT Per-session logging (w/Syslog or

w/Netflow)

Bulk Port Allocation logging (w/Syslog or w/Netflow)

Destination Based Logging w/Syslog or w/NetFlow

– Pre-defined NAT

Each choice is optimizing subset of requirements at the expense of others

Pre-defined NAT

Dynamic or Pre-defined NAT ?


DBL permits to specifically log destination address and port

NAT

External

Logging

Record

Internal

A

X1 X2

X3

X4

Tim

e

Inside

IP/Port

Outside

IP/Port

Destination

IP/Port

T1 A:Pa IP1:P1 X1:Pd1

T2 A:Pb IP1:P2 X2:Pd2

T3 A:Pc IP1:P3 X3:Pd3

T4 A:Pd IP1:P4 X4:Pd4

Syslog

Netflow

Destination-Based Logging


Why would you like to use DBL?

Legal regulations in country

– Many web servers are not logging port information for each session (not respecting RFC6302 Logging Recommendations for Internet-Facing Servers)

– Others…

Need for data analytics solution e.g.

– Offers very detailed info on user behavior

Why should you avoid using DBL?

Privacy considerations

Country regulations

Interpretation of EU directive

Conflicts with Bulk Port Allocation and Deterministic NAT

Increased storage requirements

6 additional bytes in NFv9 to store A+P

draft-ietf-behave-lsn-requirements

– REQ-12: A CGN SHOULD NOT log destination addresses or ports unless required to do so for administrative reasons



• The CGN card will generate templates 271 for Add records and templates 272 for Delete records

service cgn POC-1


inside-vrf Inside-1

map address-pool 150.0.0.0/17

external-logging netflow version 9

server

address 172.16.255.254 port 5000

session-logging

!



Two options in CGN cards today:

– Syslog

– Netflow v9

Netflow is preferred since lighter

Some customers select syslog:

– existing collection infrastructure based on syslog

– to guarantee multi-vendor interoperability

IPFIX doesn’t bring anything to the CGN logging hence isn’t considered

Both NFv9 and Syslog can be configured simultaneously in a CGN system

Netflow v9 Syslog

Format Binary

Template based format

ASCII

RFC52432

Transport UDP UDP

Sequence number

Yes in header No

Scalability

High (tested) Need BPA

Syslog or Netflow v9 ?


Keep in mind before selecting your collector

– Traditional use of NFv9 or syslog requires much lower data rates (< 50k fps)

– NAT is still a relatively new application using NF hence there is no existing data analysis tool box available

– NAT requires the records to be stored in a Database

– Most NF collectors store only the analysis results in a DB, but not the records themselves and are therefore not suitable

Templates for

– NAT44

– NAT64SF

– DS Lite

with or without

– Bulk-Allocation

– Destination-based-logging.

Syslog or Netflow v9 ?


Message needs to comply to RFC5424 format

Field are separated by space and non-applicable field are “-”

<Priority> <Version> <Time stamp> <host name> - - <Application name (NAT44 or DSLITE)> - [Record 1][Record 2]…

• [EventName <L4> <Original Source IP> <Inside VRF Name> <Original Source IPv6> < Translated Source IP> <Original Port> <Translated First Source Port> <Translated Last Source Port>]

Example: NAT44 with Bulk-Port-Alloc 1 2011 May 31 10:30:45 192.168.2.3 - - NAT44 - [UserbasedA - 10.1.32.45 INVRFA - 100.1.1.28 -

12544 12671]

Syslog for CGN


Netflow v9 supports flexible field definition

Light weight transport via UDP

NFv9 records are in binary

Based on templates containing IPFIX entities (http://www.iana.org/assignments/ipfix/ipfix.xml)

Supported since the first days on CGN

Different behavior than Netflow on routers

– Record creation / deletion of NAT entries

– Doesn’t count packets

– Doesn’t sample packets headers

Generated by the CGN card and not the MSC in the CRS or the LC in ASR9K

Netflow v9 for CGN


A few examples

Netflow v9 templates for CGN


With default path MTU = 1500B, one netflow packet can hold around 50 creation records

Generation is handled at the CPU core level

An event (new translation or deletion of an existing one) will trigger the creation of a NF packet but it’s not sent directly

If other events happen for the same core, records are added to the NFv9 packet

Packet is sent if we reach the MTU size or if we exceed one second

Netflow Packet Generation


• NFv9 is supported for all stateful translation protocol. Only a single server can be defined for instance

• Templates are regenerated and sent by default every 500 packets or 30 minutes

service cgn ISM


inside-vrf Inside-1

external-logging netflow version 9

server

address 1.2.3.4 port 123

path-mtu 2000

! can be configured from 100 to 2000

refresh-rate 100

! Regenerate NF record with template flowset every 100 logging packets

timeout 10

! Regenerate NF record with template flowset every 10 minutes

session-logging

! Session logging Enable Flag

!

Configuring NFv9 Options

HARDWARE


Carrier Grade Service Engine (CGSE) for all CRS routers

CGSE-PLUS for CRS-3 and CRS-X routers

Integrated Service Module (ISM) for ASR9000 routers

Virtualized Service Module (VSM) for ASR9000 routers with RSP440

Same form-factor than any Line Card

No physical port / interfaces (except CGSE+ and VSM for future usage)

Multi-purpose cards, they can be used for different applications

Very similar to Intel server, they run a Linux distribution

Use virtual interfaces to communicate with the rest of the system

VSM introduces the Virtual Machines and the service chaining capability

Service Cards on IOS XR Routers


Supported with

– CRS-1 / CRS-3 / CRS-X fabric

– 4-slot / 8-slot / 16-slot single/multi chassis

– Up to 12 cards in the 16-slot chassis

Multi-purpose service card

– CGN

– Arbor TMS

Monte Vista Linux distribution but configuration via IOS-XR

20M translations

1M sessions established per second

20Gbps

Carrier Grade Service Engine (CGSE)


FabQs EgressQ

iPSE

ePSE

IngressQ M

I

D

P

L

A

N

E

F

A

B

R

I

C

MSC40/FP40

M

I

D

P

L

A

N

E

GLIK

FPGA

GLIK

FPGA

PLA

CGSE PLIM

Paired with MSC40 or FP40.

Carrier Grade Service Engine (CGSE)


64 cores are available an each CGSE card (2^6) and LB decision is performed by the egress PSE ASIC (eMetro)

– For i2o traffic, the least 6 bits of the source IP address will be used

– For o2i traffic, the least 6 bits of the destination IP address will be used.

It implies that we can not assign a map pool prefix longer than /26 to use each core of the system:

– /26: each core will handle a single IP address from the map pool range (outside)

– /24: each core will handle 4 IP addresses from the map pool range (outside)

Load-Balancing Traffic inside CGSE Cores


Supported with

– CRS-3 / CRS-X fabric

– 8-slot / 16-slot single/multi chassis

– Up to 12 cards in the 16-slot chassis


– CGN

– Arbor TMS (future)

– DPI / Analytics (future)

Monte Vista Linux distribution but configuration via IOS-XR

Current supports: NAT44 / 6rd

80M translations

1M+ sessions established per second

70+ Gbps

Carrier Grade Service Engine PLUS (CGSE+)


FabQs EgressQ

iPSE

ePSE

IngressQ M

I

D

P

L

A

N

E

F

A

B

R

I

C

MSC140/FP140

M

I

D

P

L

A

N

E

Beluga

PLA

CGSE+ PLIM

Paired with MSC140 or FP140 in a CRS-3 or CRS-X chassis

Not supported in CRS-1 chassis

Netlogic

NPU

DDR

16GB

Netlogic

NPU

DDR

16GB

Carrier Grade Service Engine PLUS (CGSE+)


Supported with

– RSP2 and RSP440

– 9006 and 9010 chassis (not in 9001 or 99xx)


– CGN

– CDS-IS/TV (discontinuated)

RedHat Linux distribution but configuration via IOS-XR

20M translations

1M sessions established per second

14Gbps

Integrated Service Module (ISM)


I/O

Hub

Bridge

Bridge

Fabric

ASIC

Intel CPU

PPC

B

A

C

K

P

L

A

N

E

24GB

24GB

Bridge

Bridge

Application Domain IOS-XR Domain

DRAM

ISM Architecture


Supported with

– RSP440 (and future RSPs)

– All 9x00 chassis except 9001


– CGN

– IPsec

– Mobile GW

Service chaining

KVM virtualized environment

Current CGN Supports: NAT44

60M translations

10M+ sessions established per second

60Gbps

Virtualized Service Module (VSM)


Fabric

ASIC 0

Ivy

Bridge

B

A

C

K

P

L

A

N

E

32GB

DDR3

48

ports

10GE

Application Processor Module (APM) Service Infra Module (SIM)

Typhoon

NPU

Fabric

ASIC 1

Typhoon

NPU

Niantic

Niantic

Niantic

Niantic

Niantic

Niantic

Niantic

Ivy

Bridge

Ivy

Bridge

Ivy

Bridge

32GB

DDR3

32GB

DDR3

32GB

DDR3

Quad

PHY

SFP+ SFP+ SFP+ SFP+

Crypto/DPI

Assist

Crypto/DPI

Assist

Crypto/DPI

Assist

Crypto/DPI

Assist

XAUI

PCIe

VSM Architecture


CGSE CGSE+ ISM VSM

Sessions 20M 80M 20M 60M

Target: 80M+

Establishment

Rate

1M/s 1M/s 1M/s Up to 13M/s

Bandwidth

(IMIX)

20Gbps 70Gbps 14Gbps 60Gbps

Physical

Interfaces

No 2x10G

(future)

No 4x10G

(Future)

Platform CGN Card

CRS 4-slot 3 x CGSE or CGSE+



CRS Multi-

Chassis

Supported

since 4.3.1

ASR9001 Not supported

ASR9006 3 x ISM or VSM

ASR9010 6 x ISM or VSM

ASR9922 VSM only

9k nV

Satellite

VSM is compatible

9k nV

Cluster

VSM support

targeted for 5.2.0

Performance / Scalability

DEPLOYMENT FEEDBACK


CGSE(+) PLIM are considered high powered PLIMs

– Their power consumption is higher

– But more important, they generate more heat than other PLIMs (heat will naturally go up)

In 16-slot chassis, their position must be thought carefully

Some PLIMs are considered Thermally sensitive and can not be positioned above “high powered PLIMs”:

– CRS-1 OC768 (C/L-band) DWDM PLIM

– CRS-1 OC768 DPSK C/L-BAND STD CHAN PLIM

So, CGSE should be positioned ideally in upper shelf

If necessary, they can be positioned in lower shelf but in that case it’s important to make sure another high-powered PLIM is inserted above it in upper shelf.

upper

shelf

lower

shelf

Deployment Tips


Most majority of the ISM and CGSE deployments are done for

– NAT44

– 6rd

Some new customers or customers with internal IPv4 shortage issues are now looking at DS-Lite (and MAP)

– MAP is interesting (stateless in the router / inline performance at 240G per card) but not much CPE yet

– DS-Lite is stateful (implies logging) but CPEs are very common

Many customers are testing NAT64 but some applications are not supported at all on IPv6 (ex: Skype)

Logging

– both syslog and netflow are used

– Some customers using both simultaneously

Mobile are usually using far less ports (true for handheld, not for dongles)

Key Deployment Takeaways


Prime Performance Manager supports CGSE/ISM NAT44/NAT64 monitoring

– Active Translation / Creating Rate

– I2O and O2I Forward Rate

– I2O Drop Port Limit Exceeded

– I2O Drop System Limit Reached

– Pool address totally free / used

Expect scripts can be used to collect counters from show commands

More scripts can be used to figure out the port user port usage (very important to figure out the proper port-limit)

– First, Get all IP outside addresses in use with a ‘sh cgn nat44 NAT statistics’

– Then, for each IP address, run a ‘sh cgn nat44 NAT outside-translation proto $Prot outside-address $IP port start 1 end 65535’ with $Prot: TCP/UDP/ICMP

Logs can be used to spot customers exceeding the limits

Monitoring Options


Script will collect info on used ports per external address

RP/0/RSP0/CPU0:R1#sh cgn nat44 nat44 pool-utilization inside-vrf IN address-r$

Public address pool utilization details

-----------------------------------------------

NAT44 instance : nat44

VRF : IN

-----------------------------------------------

Outside Number Number

Address of of

Free ports Used ports

-----------------------------------------------

1.2.3.0 65528 7

1.2.3.4 65525 10

1.2.3.8 65529 6

1.2.3.12 65533 2

Scripts

1.2.3.16 65517 18

1.2.3.20 65535 0

1.2.3.24 65534 1

1.2.3.28 65469 66

1.2.3.32 65522 13

1.2.3.36 65530 5

1.2.3.40 65529 6

1.2.3.48 65533 2

1.2.3.52 65534 1


RP/0/RP0/CPU0:R1#sh cgn nat44 NAT-1 outside-translation protocol tcp outside-address

196.219.0.3 port start 1 end 65535

--------------------------------------------------------------------------------------------

Inside Protocol Inside Outside Translation Inside Outside



Packets Packets

--------------------------------------------------------------------------------------------

10.193.114.195 tcp 1114 46599 dynamic 110 129

10.193.114.195 tcp 1525 59248 dynamic 26 26

10.193.208.195 tcp 1691 54882 dynamic 6 4

10.193.114.195 tcp 1845 46393 dynamic 6 6

10.193.169.131 tcp 1980 63344 dynamic 12 21

10.193.248.131 tcp 2581 51821 dynamic 25 29

10.193.254.67 tcp 2873 1469 dynamic 12 15

10.193.117.67 tcp 2958 50417 dynamic 12 11

10.193.24.131 tcp 3016 50279 dynamic 8 8

10.193.247.3 tcp 3248 32869 dynamic 27 32

10.193.114.195 tcp 3479 58883 dynamic 29 28

10.193.114.195 tcp 3664 49916 dynamic 6 6

Scripts


Scripts

10.193.114.195

10.193.114.195

10.193.208.195

10.193.114.195

10.193.169.131

10.193.248.131

10.193.117.67

10.193.24.131

10.193.247.3

10.193.114.195

10.193.114.195

10.193.24.131

10.193.114.195

10.193.114.195

10.193.114.195

10.193.114.195

10.193.114.195

10.193.117.67

10.193.169.131

10.193.208.195

10.193.247.3

10.193.248.131

Sort

10.193.24.131 1

10.193.114.195 5

10.193.117.67 1

10.193.169.131 1

10.193.208.195 1

10.193.247.3 1

10.193.248.131 1

Per user port usage

- Top X users

- Average

- …

46599

59248

54882

46393

63344

51821

1469

50417

50279

32869

58883

45

57

53

45

61

50

1

49

49

32

57

Divide

by BPA

and

round

down

Count

1 1

32 1

45 2

49 2

50 1

53 1

57 2

61 1

For a BPA=1024

- Number of ports used

per block ID

- Top X blocks

- Average usage

- … Port-limit

tweaking

BPA

tweaking


No rule of thumb to define port-limit, BPA, timers…

Example for a broadband ISP in LATAM (using a script)

– 18 ports average per user Can not be used to determine the best port-limit

– i2o 50kpps per card

– o2i 70kpps per card

– Avg i2o packet size: 200B

– Avg o2i packet size:1200B

percentage of users using less than X ports (starts at 99.8%)

Sizing the Port-Limit and BPA


Several customers have been testing extensively the most popular applications and successfully, for example:

– TFTP, SSH, Telnet

– IPSec VPN (Cisco Client), SSL VPN (AnyConnect Client)

– HTTP/HTTPS on popular sites (CNN, Facebook, Youtube, Google services, …)

– WebMail (Java)

– Skype, SkypeUpdate, Audio/Video/FileTransfer/Chat

– MSN

– Bit Torrent

– Netflix

– Video web sites like Crunchyroll.com, ign.com

– iTunes store browsing, upgrade, …

– Sony Media Go Store

– Steam Install and Update

– StarCraft 2, World of Warcraft, MineCraft, …

Impact of CGN on Applications?


A vast majority of users only need their internet connection for

– Web surfing

– Emails

– Skype

– Mobile Phone Apps on Wifi

– Occasionally p2p download

These customers will never realize they are NATed

“Per complaint” behavior:

– When customers are complaining about their connection (latency, applications not working mainly for hardcore gamers who need to be a node for multiplayer games), the ISP move them into a different VRF which is not NATed

Application Impacted by CGN


Geo-localization services

IP tracking services (advertisement system, not based on cookies)

Service Impacted by CGN

ROUTING CONSIDERATION AND BEST PRACTICES


Two types of routing should be differentiated

Intra-chassis routing

– Packets candidate for translation or tunnel encapsulation/decapsulation, when received on the router, should be forwarded to and from the CGN card

– Static routes and Access-List Based Forwarding will be use

Extra-chassis routing

– Packets should also be attracted by the CGN system able to handle them properly

– Dynamic routing protocols (BGP or IGP) will be used to advertise the prefix

Types of Routing


CGN

Card

ServiceApp2

inside VRF

ServiceApp1

Te0/0/0/0 outside VRF

Te0/1/0/0 IPv4

Internet

IPv4

Backbone

IGP

IGP/BGP

Static

Static

ABF Intra-Chassis

Extra-Chassis

CGN Routing


Aimed at forwarding packets candidate for translation or tunnel encapsulation/decapsulation, to and from the CGN card

For i2o traffic, two methods available

– Based on destination: static routes to the serviceApp interface in the global table to the serviceApp

in the global table to the serviceApp in a named VRF

in a named VRF table to the serviceApp

should be advertised in IGP and/or iBGP

– Based on source or destination: Access-list Based Forwarding applied in ingress on the interface, could be VRF-aware or not

For o2i traffic

– usually, we will rely on static routes to advertised a route back to the map pool range into the outside serviceApp

– should be advertised in external IGP or BGP

Intra-Chassis Routing


It’s necessary to attract traffic to the CGNAT device and determine which traffic is actually candidate to translation

Asymmetrical traffic is not possible with CGNAT routing, o2i must follow the path of the i2o traffic

That’s why it’s mandatory to advertise the map pool ranges to the external world to guarantee the symmetry

Some example:

Default Map pool

CGN

NAT Access

BNG

Core

Public IP Internet

Extra-Chassis Routing


A few other examples

Default Map pool

Full Table

Aggregate

Default Map pool

Full Table

CGN

NAT

CGN

NAT

Internet

Internet

Core

Private IP

L3VPN

VRF

Peering

Network

Extra-Chassis Routing


Create one static route in each VRF (inside and outside)

All packets arriving in vrf inside should be directed to the CGN card through the serviceApp1 interface

All packets arriving in vrf outside and targeted to addresses in the map pool range should be directed to the serviceApp2 interface

CGN

Card

ServiceApp2

inside VRF

ServiceApp1


Te0/1/0/0

RP/0/RSP0/CPU0:router(config)#

router static

vrf inside

address-family ipv4 unicast

0.0.0.0/0 ServiceApp1

!

vrf outside



!

Translate to

100.0.0.0/24

NAT44 Static Route Configuration


In many situations, physical interfaces can not be in a inside VRF but must be in the global routing table

We could simply use a static default in the global ipv4 table pointing to serviceApp in the inside VRF, but a global default route is not recommended:

– ALL traffic with no route in the RIB will be attracted

– if the router has a full BGP table, no packets will be routed to serviceApp1

CGN

Card

ServiceApp2

inside VRF

ServiceApp1


Te0/1/0/0

Translate to

100.0.0.0/24


router static


0.0.0.0/0 vrf inside ServiceApp1

!

NAT44 Static Route Configuration


Routing based on ACL enables decision based on source addresses

Public sources can avoid NAT // Private can be sent for NAT translation

CGN

Card

ServiceApp2

inside

VRF

ServiceApp1

Te0/0/0/0 outside

VRF

Te0/1/0/0

Translate to

100.0.0.0/24


ipv4 access-list ABF

10 permit ipv4 10.0.0.0 0.255.255.255 any nexthop1 vrf inside ipv4 1.1.1.2

20 permit ipv4 any any

interface ServiceApp1

vrf inside

ipv4 address 1.1.1.1/30

service cgn demo service-type nat44

!

interface TenGigE0/0/0/0


ipv4 access-group ABF ingress


vrf outside



!



10.0.0.0/8

30.0.0.0/8

NAT44 ABF Configuration


Return traffic

When you configure ABF for the i2o traffic, you don’t need to do it for the o2i traffic

o2i traffic must be routed to the correct Inside (default) VRF when it comes out of the Inside Service App

CGN

Card

ServiceApp2

inside

VRF

ServiceApp1

Te0/0/0/0 outside

VRF

Te0/1/0/0

Translate to

100.0.0.0/24

10.0.0.0/8

30.0.0.0/8


router static

vrf inside


10.0.0.0/8 vrf default 20.1.1.2

20.1.1.2


router static


100.0.0.0/24 vrf outside serviceApp2

NAT44 ABF Configuration


What if the next-hop address in GRT isn’t reachable (interface down for example)?

CGN

Card

ServiceApp2

inside

VRF

ServiceApp1

Te0/0/0/0 outside

VRF

Te0/1/0/0

Translate to

100.0.0.0/24

10.0.0.0/8

30.0.0.0/8


router static

vrf inside


10.0.0.0/8 vrf default 20.1.1.2

20.1.1.2


router static



Even if another path is available to reach 10.0.0.0/8 in the GRT, traffic is lost

NAT44 ABF Limitations


What if the next-hop router points to the CGN router to reach 10.0.0.0/8?

CGN

Card

ServiceApp2

inside

VRF

ServiceApp1

Te0/0/0/0 outside

VRF

Te0/1/0/0

Translate to

100.0.0.0/24

10.0.0.0/8

30.0.0.0/8


router static

vrf inside


10.0.0.0/8 vrf default 20.1.1.2

20.1.1.2


router static



In this case, the traffic will eventually find it’s way to 10.0.0.0/8 but via a sub-optimal path



ABF is performed before MPLS labels are stripped from packets

Consequently, packets are not matched

Example, the “CGN in PE” case

Workaround: loop fiber

CGN

Card VRF

Inside-1

SA1 Global

Translate to

151.0.0.0/24

SA2

CGN

Card VRF

Inside-2

SA3 Global

Translate to

151.0.1.0/24

SA4

251.5 250.5

51.5 52.5

0/0/CPU0

0/1/CPU0

PE

Te0/6/0/2 P

2 Labels

Transport

VRF

1 Label

VRF



Other example, the CSC case (CGN in CE)

CGN

Card VRF

Inside-1

SA1 Global

Translate to

151.0.0.0/24

SA2

CGN

Card VRF

Inside-2

SA3 Global

Translate to

151.0.1.0/24

SA4

251.5 250.5

51.5 52.5

0/0/CPU0

0/1/CPU0

CE

Te0/6/0/2 PE

3 Labels

Transport

VRF

CSC

1 Label

CSC

P

2 Labels

VRF

CSC


REDUNDANCY


On both CRS/CGSE and ASR9000/ISM, we support 1:1 warm standby redundancy (not supported on CGSE+ today)

Warm-standby

– translation state is not synchronized between active and standby, all connections will be re-established

– Pros: simple to configure, a single map pool is used

– Cons: only 1:1, one card on two will not be used 99% of the time

An alternative with ABF is available

– Pros: offers more options like n:1 redundancy, converges very quickly

– Cons: we can not re-use the same map pool range, so we need to configure a second range

CGSE/ISM Redundancy


Configuration

RP/0/RSP0/CPU0:CGN(config)#

service cgn demo

service-location preferred-active 0/1/CPU0 preferred-standby 0/3/CPU0

RP/0/RP0/CPU0:CGN#show services redundancy

Service type Name Pref. Active Pref. Standby

--------------------------------------------------------------------------------

ServiceInfra ServiceInfra1 0/1/CPU0 Active

ServiceInfra ServiceInfra2 0/3/CPU0 Active

ServiceCgn demo 0/3/CPU0 Standby 0/1/CPU0 Active

RP/0/RP0/CPU0:CGN#

1:1 Warm-Standby Redundancy


CGN

Card VRF

Inside-1

SA1 Global

Translate to

151.0.0.0/24

SA2

CGN

Card VRF

Inside-2

SA3 Global

Translate to

151.0.1.0/24

SA4

CGN

Card VRF

iBackUp

SA5 Global

Translate to

151.0.2.0/24

SA6

Te0/6/0/3

100.1.1.1/24

Te0/6/0/2

10.1.1.1/24

service cgn mets-cgn



inside-vrf Inside-1


!

service cgn mets-cgn-2



inside-vrf Inside-2


!

service cgn mets-cgn-backup


service-type nat44 nat44-backup

inside-vrf iBackUp


251.5 250.5

51.5 52.5

53.5 54.5

CGSE/ISM Redundancy



10 permit ipv4 10.2.0.0/24 any nexthop1 vrf Inside-1 ipv4 192.168.251.6 nexthop2 vrf iBackUp ipv4 192.168.53.6



!

router static


110.1.0.0/16 100.1.1.2 description Ixia-i2o-Default

151.0.0.0/24 ServiceApp2 description Ixia-o2i-ABF



CGN

Card VRF

Inside-1

SA1 VRF

Outside-1

Translate to

151.0.0.0/24

SA2

CGN

Card VRF

Inside-2

SA3 VRF

Outside-2

Translate to

151.0.1.0/24

SA4

CGN

Card VRF

iBackUp

SA5 Default

Translate to

151.0.2.0/24

SA6

Te0/6/0/3

100.1.1.1/24

Te0/6/0/2

10.1.1.1/24

10.2.0.0/24

10.2.1.0/24

110.1.0.0/16

251.5 250.5

51.5 52.5

53.5 54.5

Packets sourced

from 150.0.0.x

CGSE/ISM n:1 Redundancy






!

router static


110.1.0.0/16 100.1.1.2 description Ixia-i2o-Default




CGN

Card VRF

Inside-1

SA1 VRF

Outside-1

Translate to

151.0.0.0/24

SA2

CGN

Card VRF

Inside-2

SA3 VRF

Outside-2

Translate to

151.0.1.0/24

SA4

CGN

Card VRF

iBackUp

SA5 Default

Translate to

151.0.2.0/24

SA6

Te0/6/0/3

100.1.1.1/24

Te0/6/0/2

10.1.1.1/24

10.2.0.0/24

10.2.1.0/24

110.1.0.0/16

251.5 250.5

51.5 52.5

53.5 54.5

Packets sourced

from 150.0.2.x

CGSE/ISM n:1 Redundancy


1:1 warm standby

redundancy

N:1 ABF based

redundancy

Convergence time Up to 7s <1s

CAPEX Needs a standby card for

every active one Needs only a single backup

card per router

Impact on other resources (address map pools)

No map pool necessary for the backup card

No map pool necessary for the backup card

Preemption when the first card gets back

online

No preemption, the new active card stays active

The initial active card regains the active role and create a 2nd

impact

Static port forwarding No problem, the standby

re-populates the table with the static entry

Since the backup card uses a different map pool, a new static

entry will be created

111

CGSE/ISM n:1 Redundancy Limitations


CGN

Card VRF

Inside-1

SA1 Global

Translate to

151.0.0.0/24

SA2

CGN

Card VRF

Inside-2

SA3 Global

Translate to

151.0.1.0/24

SA4

CGN

Card VRF

iBackUp

SA5 Global

Translate to

151.0.2.0/24

SA6

Te0/6/0/3

100.1.1.1/24

Te0/6/0/2

10.1.1.1/24

251.5 250.5

51.5 52.5

53.5 54.5

Te0/0/0/0

10.10.1.1/24

Te0/0/0/1

100.1.2.1/24

0/0/CPU0

0/1/CPU0

0/0/CPU0

ipv4 access-list ABF-1

10 permit ipv4 any any nexthop1 vrf Inside-1 ipv4 192.168.251.6

nexthop2 vrf Inside-2 ipv4 192.168.51.6 nexthop3 ipv4 10.10.1.1

If routers are not directly connected, a GRE tunnel can be used to avoid routing loops

Extra-Chassis Redundancy


CGN cards are generating syslog and NFv9 on UDP

No mean to send backpressure if the server can’t cope

One single destination per type and inside-VRF

Workarounds exist at the collector level:

– Virtual IP addresses on the collector

– Port SPAN on the switch were is connected the collector to replicate the logging flow (second server needs some tweaking to accept the trafffic)

– Directed-Broadcast on the last router (ex: the last interface is 10.100.1.1/30 and we will generate the logging traffic to 10.100.1.4, the broadcast address of this network. Only 10.100.1.0/24 will be advertised in IGP)

RAID / DB redundancy is highly recommended at the server level

Logging Redundancy

CONCLUSION


CGN offers tools to buy time for your IPv6 preparation

The same line cards can also be used for IPv6 migration (NAT64, 6rd, DS-lite)

For the vast majority of usages: it just works

Deployment must be considered carefully for

– Routing

– Logging infrastructure for collection and storage

– Timers, BPA, Port-Limit, …

Conclusion


Complete your online session evaluation

Complete four session evaluations and the overall conference evaluation to receive your Cisco Live T-shirt

Complete Your Online Session Evaluation

116

BACKUP SLIDES UNDERSTANDING TIMERS


NAT44 (like NAT64SF and DS Lite) performs a stateful translation

Packet source address and port are rewritten

Details are stored in a translation database

A new packet from inside to outside will create a new entry in the table

No activity during a configurable period of time will trigger the suppression of this entry

We use different timers for different packet types and different situations

Stateful Protocols Understanding the Stateful Translation


IPv4

Internet IPv4 Traffic

CGN

Source Address = 10.1.1.1 Outside Address

from pool = 99.0.0.1 NAT

30.0.0.1

Src: 10.1.1.1:12345

Dst: 30.0.0.1:80

Src: 99.0.0.1:1025

Dst: 30.0.0.1:80

SYN/ACK 3

SYN 1

ACK 5

Inside Outside TCP

state 0

Inside Outside TCP

state

10.1.1.1:12345 99.0.0.1:1025 Inactive

2

Inside Outside TCP

state

10.1.1.1:12345 99.0.0.1:1025 Active

4

Now, as long as TCP traffic is received in any direction within the active timer, state is maintained as “Active”. This behavior can be changed by configuration, considering only the i2o traffic to refresh the timers.

NAT44: TCP Establishment


IPv4

Internet

CGN

NAT

30.0.0.1

Src: 10.1.1.1:12345

Dst: 30.0.0.1:80

Src: 99.0.0.1:1025

Dst: 30.0.0.1:80

ACK 3

Inside Outside TCP

state

10.1.1.1:12345 99.0.0.1:1025 Inactive

2

Inside Outside TCP

state

10.1.1.1:12345 99.0.0.1:1025 Active

Note: We are not checking the sequence numbers in the NAT engine.

1 FIN or RST

0

Inside Outside TCP

state

10.1.1.1:12345 99.0.0.1:1025 Inactive

4

3 Initial timer expires

DB is cleaned up Default timers:

TCP init: 120s

IPv4 Traffic Source Address = 10.1.1.1 Outside Address


NAT44: End of TCP Session


IPv4

Internet

CGN

NAT

30.0.0.1

Src: 10.1.1.1:12345

Dst: 30.0.0.1:80

Src: 99.0.0.1:1025

Dst: 30.0.0.1:80

SYN 1

Inside Outside TCP

state 0

Inside Outside TCP

state

10.1.1.1:12345 99.0.0.1:1025 Inactive

2


DB is cleaned up

Inside Outside TCP

state

10.1.1.1:12345 99.0.0.1:1025 Inactive

Default timers:

TCP init: 120s

Note: we are checking all timers every 10ms to clean up the time-outs

4



NAT44: TCP Initial Timeout


IPv4

Internet

CGN

NAT

30.0.0.1

Src: 10.1.1.1:12345

Dst: 30.0.0.1:80

Src: 99.0.0.1:1025

Dst: 30.0.0.1:80 Inside Outside TCP

state

10.1.1.1:12345 99.0.0.1:1025 Active

Note: We are not sending any FIN/RST to either side (inside nor outside), the translation entry is simply removed from the table.

0

Inside Outside TCP

state

10.1.1.1:12345 99.0.0.1:1025 Inactive

2


DB is cleaned up Default timers:

TCP active: 1800s

No traffic matching the DB entry flows through the system…



NAT44: TCP Active Timeout


IPv4

Internet

CGN

NAT

30.0.0.1

Src: 10.1.1.1:12345

Dst: 30.0.0.1:80

Src: 99.0.0.1:1025

Dst: 30.0.0.1:80 Inside Outside TCP

state 0

If we send TCP data packet before a complete TCP handshake…

TCP Data 1 2

… this packet is considered invalid and

dropped without ICMP being generated.



NAT44: Security Behavior


IPv4

Internet

CGN

NAT

30.0.0.1

Src: 10.1.1.1:12345

Dst: 30.0.0.1:80

Src: 99.0.0.1:1025

Dst: 30.0.0.1:80

SYN 1

Inside Outside TCP

state 0

Inside Outside TCP

state

10.1.1.1:12345 99.0.0.1:1025 Inactive

2

Inside Outside TCP

state

10.1.1.1:12345 99.0.0.1:1025 Inactive

4

If we receive a TCP data packet before a complete TCP handshake…

TCP Data 3

… this packet is translated back and passed to the host, but table state isn’t

changed from Inactive to Active. It stays at Inactive.



NAT44: Security Behavior


IPv4

Internet

CGN

NAT

30.0.0.1

Src: 10.1.1.1:12345

Dst: 30.0.0.1:80

Src: 99.0.0.1:1025

Dst: 30.0.0.1:80

UDP 1

Inside Outside UDP state 0

Inside Outside UDP state

10.1.1.1:12345 99.0.0.1:1025 Inactive

2


10.1.1.1:12345 99.0.0.1:1025 Active

4

UDP 3

Now, as long as UDP traffic is received in any direction within the active timer,

state is maintained as “Active”.



NAT44: UDP Packets


IPv4

Internet

CGN

NAT

30.0.0.1

Src: 10.1.1.1:12345

Dst: 30.0.0.1:80

Src: 99.0.0.1:1025

Dst: 30.0.0.1:80

UDP 0

Inside Outside UDP

state

10.1.1.1:12345 99.0.0.1:1025 Inactive


10.1.1.1:12345 99.0.0.1:1025 Inactive

0

Only I2O traffic passes through CGN, UDP state is Inactive 1 Now, no more I2O UDP traffic is received

Initial timer expires

DB is cleaned up 2

4

Default timers:

UDP init: 30s



NAT44: UDP Timeout Case 1


IPv4

Internet

CGN

NAT

30.0.0.1

Src: 10.1.1.1:12345

Dst: 30.0.0.1:80

Src: 99.0.0.1:1025

Dst: 30.0.0.1:80

UDP 0

UDP 0


10.1.1.1:12345 99.0.0.1:1025 Active

0

1 Now, both I2O and O2I UDP stop flowing through the CGN

Default timers:

UDP active: 120s

Inside Outside UDP

state

10.1.1.1:12345 99.0.0.1:1025 Active

Initial timer expires

DB is cleaned up 2

4



NAT44: UDP Timeout Case 2


IPv4

Internet

CGN

NAT

30.0.0.1

Src: 10.1.1.1

Dst: 30.0.0.1

Src: 99.0.0.1

Dst: 30.0.0.1

ICMP 1

ICMP 3

NAT

Info 0

NAT

Info

10.1.1.1 99.0.0.1 ICMP

2 No state in ICMP translation Only a DB entry.

Now, as long as ICMP traffic is received in any direction within the

timer, this entry will be maintained in the DB.



NAT44: ICMP


IPv4

Internet

CGN

NAT

30.0.0.1

Src: 10.1.1.1

Dst: 30.0.0.1

Src: 99.0.0.1

Dst: 30.0.0.1

ICMP 1

NAT

Info 0

NAT

Info

10.1.1.1 99.0.0.1 ICMP

2

Now, no more I2O and O2I ICMP flow through the CGN

NAT

Info

10.1.1.1 99.0.0.1 ICMP

4

ICMP timer expires

DB is cleaned up 3

Default timers:

ICMP: 60s



NAT44: ICMP Timeout Case


For stateful translation protocols (NAT44, NAT64 SF, DS Lite), the NAT DB maintains timers for each entry

service cgn demo


protocol udp

session initial timeout 10

session active timeout 30

protocol tcp



protocol icmp

timeout 30

service cgn demo

service-type nat64 stateful nat64-1

protocol udp

timeout 30

v4-init-timeout 10

protocol tcp



protocol icmp

timeout 30

service cgn demo

service-type ds-lite ds-lite1

protocol udp


session init timeout 10

protocol tcp


session init timeout 30

protocol icmp

timeout 30

Default Initial Active

TCP 120s 1800s

UDP 30s 120s

ICMP 60s

Fine Tuning Timers


Timers are refreshed when packets are translated in i2o or o2i direction. But an external attacker could send regularly one packet for every DB entry and eventually create a resource depletion

To change this default behavior, we can make the timer refresh to only take into consideration Inside-to-Outside (i2o) packets

This feature is not available for DS Lite

service cgn POC-1


refresh-direction Outbound

!

service-type nat64 stateful nat64-1

refresh-direction Outbound

!

Refresh Direction

BACKUP SLIDES LOAD BALANCING


M

I

D

P

L

A

N

E

FabQs EgressQ

BRIDGE

BRIDGE

PLA

iPSE

ePSE

IngressQ

M

I

D

P

L

A

N

E

F

A

B

R

I

C

FabQs EgressQ

PLA iPSE

ePSE

IngressQ

PLA

SPA

SPA

SPA

SPA

SPA

SPA

BRIDGE

BRIDGE

PLA

FabQs EgressQ

iPSE

ePSE

IngressQ

At ingress PSE level:

Two static routes for one

NH address pointing to two

serviceApps interfaces (L3

or L4 LB is used depending

on the configuration)

ABF is possible too and is a

better option.

At egress PSE level:

Hashing on source

address to loadbalance

traffic between 64 cores

Note: using static routes will break the principle of same external IP address mapping for all sessions

associated with the same internal IP address (RFC4787) we recommend ACL Based Forwarding.

Load-balancing Traffic Between CGSEs


Netlogic

NPU

DDR

16GB

Netlogic

NPU DDR

16GB

M

I

D

P

L

A

N

E

BRIDGE

BRIDGE

PLA

M

I

D

P

L

A

N

E

F

A

B

R

I

C

PLA

PLA

SPA

SPA

SPA

SPA

SPA

SPA

PLA

FabQs EgressQ

iPSE

ePSE

IngressQ

FabQs EgressQ

iPSE

ePSE

IngressQ

FabQs EgressQ

iPSE

ePSE

IngressQ

RP/0/RP0/CPU0:router(config)#

router static

vrf inside


0.0.0.0/0 ServiceApp11 192.168.11.2

0.0.0.0/0 ServiceApp21 192.168.21.2


192.168.21.3


192.168.21.4


192.168.21.5

!

vrf outside




CGSE ServiceApp12

192.168.12.1/2

4

inside VRF

ServiceApp11

192.168.11.1/2

4

outside VRF

Translate to

100.0.0.0/24

CGSE

PLUS ServiceApp22

192.168.22.1/2

4

ServiceApp21

192.168.21.1/2

4 Translate to

100.1.0.0/16


Netlogic

NPU

DDR

16GB

Netlogic

NPU DDR

16GB

M

I

D

P

L

A

N

E

BRIDGE

BRIDGE

PLA

M

I

D

P

L

A

N

E

F

A

B

R

I

C

PLA

PLA

SPA

SPA

SPA

SPA

SPA

SPA

PLA

FabQs EgressQ

iPSE

ePSE

IngressQ

FabQs EgressQ

iPSE

ePSE

IngressQ

FabQs EgressQ

iPSE

ePSE

IngressQ


+ ACL definition here

+ ABF applied on ingress interface here

!

vrf outside




CGSE ServiceApp12

192.168.12.1/2

4

inside VRF

ServiceApp11

192.168.11.1/2

4

outside VRF

Translate to

100.0.0.0/24

CGSE

PLUS ServiceApp22

192.168.22.1/2

4

ServiceApp21

192.168.21.1/2

4 Translate to

100.1.0.0/16


Based on the number of cores, we can’t allocate a range more specific than /30 (4 public addresses)

Load-balancing is different on the ISM than CGSE:

– First, it’s performed by the ingress NPU (Trident or Typhoon on in the ingress card) where lookup is performed and a VQI is assigned for the destination Each VQI is “linked” to a particular Niantic port, hence to a particular dispatcher process on a CPU. (2 CPUs, 2 dispatchers running on 2 different ports 4 options).

– Second, the dispatcher process will determine which CGv6 application process should be handle this packet: - i2o traffic: hash is performed on the source address 32 bits - o2i traffic: hash is performed on the destination address 32 bits

For DS-Lite, hash will be done on the B4 ipv6 address for i2o traffic and on the destination ipv4 address for o2i traffic.

24Gb

24Gb

Load-balancing Traffic inside ISM

BACKUP SLIDES NAT CONFIGURATION


CGN

Card

Interconnecting CGSE/ISM card to the rest of the system

Configuration is only needed on the router/XR side, addresses on the CGN/Linux side will be automatically created

To direct traffic into the CGN card, we’ll need one or several of these options:

– static routes

– redistribution

– ACL based forwarding rules

ServiceInfra interface

– For CGN card management

– One per card mandatory

ServiceApp interfaces

– To interconnect GRT address-family or VRF inside and outside to the CGN card

ServiceApp2

ServiceInfra1

VRF

or

address-family

ServiceApp1

Physical

Interface

VLAN

VRF

or

address-family

Physical

Interface

VLAN

Virtual Service Interfaces


To avoid routing loops, VRF are mandatory with NAT44

Inside VRF must be non-default

Outside VRF is optional, we can use the Default or a named VRF

RP/0/RSP0/CPU0:Router(config)#

vrf inside


!

vrf outside


!

interface te0/0/0/0

vrf inside

ipv4 add 10.1.1.1/24

!

interface te0/1/0/0

vrf outside

ipv4 add 100.1.1.1/24

!


vrf inside

ipv4 address 1.1.1.1 255.255.255.252


!


vrf outside

ipv4 address 2.1.1.1 255.255.255.252


CGN

Card

ServiceApp2

inside VRF

ServiceApp1


Te0/1/0/0

NAT44 Configuration


Create a nat44 instance “nat1” and associate an outside pool (Public IPv4 addresses) to a given inside VRF

A single nat44 instance can be created per CGN card

Several mechanisms exist to push traffic in2out into ServiceApp1

A static route with the map pool range will be necessary to send out2in traffic to the CGN card via ServiceApp2

CGN

Card

ServiceApp2

Inside VRF

ServiceApp1

outside VRF

or

Default

service cgn demo


inside-vrf inside


! Mapping to the default VRF in public side

service cgn demo


inside-vrf inside

map outside-vrf outside address-pool 100.0.0.0/24

! Mapping to the VRF “outside” in public side

Translate to

100.0.0.0/24

NAT44 Configuration


In current XR release, we can not configure two map pools under one VRF inside (coming in the near future)

RP/0/RP0/CPU0:Router(config-cgn-invrf)#show

Fri Jun 15 16:54:52.430 PDT

service cgn demo


inside-vrf Inside-2


!

RP/0/RP0/CPU0:Router(config-cgn-invrf)#map address-pool 151.0.1.0/24


Fri Jun 15 16:56:23.669 PDT

service cgn demo


inside-vrf Inside-2


!

RP/0/RP0/CPU0:Router(config-cgn-invrf)#

NAT44 Configuration Tips


To overcome this limit we can configure several inside VRFs:


Fri Jun 15 16:54:52.430 PDT

service cgn demo


inside-vrf Inside-1


!

inside-vrf Inside-2


!

RP/0/RP0/CPU0:Router(config-cgn-invrf)#

Challenge will now reside in directing the traffic to both inside VRF

Total of all map pools can not be larger than 65535 addresses

It doesn’t need to be into a single /16 or contiguous ranges

NAT44 Configuration Tips


RP/0/RP0/CPU0:Router#show cgn demo stat sum

Statistics summary of NAT44 instance: ’demo'

Number of active translations: 2250000

Number of sessions: 11500028

Translations create rate: 0

Translations delete rate: 0

Inside to outside forward rate: 12600

Outside to inside forward rate: 0

Inside to outside drops port limit exceeded: 0

Inside to outside drops system limit reached: 0

Inside to outside drops resorce depletion: 0

No translation entry drops: 0

PPTP active tunnels: 0

PPTP active channels: 0

PPTP ctrl message drops: 0

Number of subscribers: 0

Drops due to session db limit exceeded: 0

Pool address totally free: 25268

Pool address used: 7500

-------------------------------------------------

External Address Ports Used

-------------------------------------------------

160.0.0.8 300

160.0.0.36 300

160.0.0.52 300

…

Translation entries allocated in DB

Additional flows inside

these translations

Rate in sessions per second

Rate in packets per second

Packets dropped because of

port-limit for inside user is reached

Packets discarded because we

reached the limit of 20M sessions

or 1M internal users

Packets dropped because no public

L4 Port could be allocated

NAT44 Show Commands


RP/0/RP0/CPU0:Router#show cgn demo stat sum

Statistics summary of NAT44 instance: ’demo'

Number of active translations: 2250000

Number of sessions: 11500028

Translations create rate: 0

Translations delete rate: 0

Inside to outside forward rate: 12600

Outside to inside forward rate: 0

Inside to outside drops port limit exceeded: 0

Inside to outside drops system limit reached: 0

Inside to outside drops resorce depletion: 0

No translation entry drops: 0

PPTP active tunnels: 0

PPTP active channels: 0

PPTP ctrl message drops: 0

Number of subscribers: 0

Drops due to session db limit exceeded: 0

Pool address totally free: 25268

Pool address used: 7500

-------------------------------------------------

External Address Ports Used

-------------------------------------------------

160.0.0.8 300

160.0.0.36 300

160.0.0.52 300

…

External addresses

and ports allocated

Addresses used in the pool

Addresses available in the pool

PPTP/GRE sessions/tunnels info

out2in drops because of

no entry in the translation DB

Packets dropped after

exceeding the 20M sessions

Private addresses having at

least one active translation

NAT44 Show Commands


RP/0/RP0/CPU0:Router#show cgn demo pool-utilization inside-vrf Inside address-range 100.0.0.90 100.0.0.95

Public address pool utilization details

-------------------------------------------------------

CGN instance : demo

VRF : Inside

-------------------------------------------------------

Outside Number Number

Address of of

Free ports Used ports

-------------------------------------------------------

100.0.0.90 64512 0

100.0.0.91 64512 0

100.0.0.92 63139 1373

100.0.0.93 63138 1374

100.0.0.94 64512 0

100.0.0.95 64512 0

...

Pool utilization statistics

NAT44 Show Commands


RP/0/RP0/CPU0:router#sh cgn demo inside-translation protocol tcp inside-vrf Inside inside-address 10.12.0.29

port start 1 end 65535


---------------------------

CGN instance : demo

Inside-VRF : Inside

--------------------------------------------------------------------------------------------




Packets Packets

--------------------------------------------------------------------------------------------

100.0.0.93 tcp 1405 58529 dynamic 7 4

100.0.0.93 tcp 1406 34188 dynamic 7 4

100.0.0.93 tcp 1407 41851 dynamic 7 4

100.0.0.93 tcp 2156 38317 dynamic 7 4

100.0.0.93 tcp 2157 30504 dynamic 7 4

100.0.0.93 tcp 2158 40039 dynamic 7 4

100.0.0.93 tcp 2907 42745 dynamic 7 4

...

Translation statistics from an inside address perspective

NAT44 Show Commands


RP/0/RP0/CPU0:router#sh cgn demo outside-translation protocol tcp outside-vrf Outside outside-address

100.0.0.93 port start 1024 end 65535

Outside-translation details

---------------------------

CGN instance : demo

Outside-VRF : Outside

--------------------------------------------------------------------------------------------

Inside Protocol Outside Inside Translation Inside Outside

Address Destination Destination Type to to


Packets Packets

--------------------------------------------------------------------------------------------

10.12.0.221 tcp 1032 56742 dynamic 7 4

10.12.0.157 tcp 1033 43804 dynamic 7 4

10.12.0.157 tcp 1055 54299 dynamic 7 4

10.12.0.157 tcp 1206 41550 dynamic 7 4

10.12.0.157 tcp 1274 64801 dynamic 7 4

10.12.0.221 tcp 1306 10243 dynamic 7 4

10.12.0.221 tcp 1359 8738 dynamic 7 4

...

Translation statistics from an outside address perspective

NAT44 Show Commands

BACKUP SLIDES CONFIGURATION AND TROUBLESHOOTING TIPS



ipv4 access-list ServiceInfraFilter

100 permit ipv4 host 1.1.1.1 any

101 permit ipv4 host 1.1.1.2 any

!

interface ServiceInfra1

ipv4 address 1.1.1.1 255.255.255.0 service-location 0/0/CPU0

ipv4 access-group ServiceInfraFilter egress

!

ServiceInfra interfaces are virtual “tunnels” between the router and the CGN card and are mandatory to boot and manage it

Even if the prefix used for this card isn’t supposed to be advertised outside of the router, it’s recommended to configure a filter to protect it from potential DoS attack

Protecting ServiceInfra Interface w/ an ACL


ServiceInfra interfaces are part of the global routing table and they are the source interfaces for syslog or netflow messages. If the collector is located in the Inside VRF, it’s not possible to send it any reports by default

We need to use ABF to overcome this limitation

interface GigabitEthernet0/3/1/0

vrf Inside

ipv4 address 10.1.0.1 255.255.255.0

!

service cgn cgn1

service-location preferred-active 0/0/CPU0 preferred-standby 0/2/CPU0

service-type nat44 NAT44

inside-vrf Inside


external-logging syslog

server

address 10.1.0.3 port 3000

session-logging

Sending Logging Reports in a VRF


We define and apply an ABF on the serviceInfra interface

ipv4 access-list acl1

10 permit udp 101.100.11.0/24 host 10.1.0.3 nexthop1 vrf Inside


!

interface ServiceInfra2

ipv4 address 101.100.11.1 255.255.255.0

service-location 0/2/CPU0

ipv4 access-group acl1 ingress

!

!

router static

vrf Inside



10.1.0.3/32 GigabitEthernet0/3/1/0

!

!

Sending Logging Reports in a VRF


• For stateful translation protocols, the dynamic translations start from 1024. We can change this starting value from 1 to 65535

service cgn POC-1


dynamic-port-range start 2000

!

Dynamic Port Range


We can define an ICMP rate-limiter for CGN card (ISM, CGSE)

For CRS/CGSE: should be a multiple of 64, less than 65472

For ASR9K/ISM: should be a multiple of 8, less than 8184

It can be 0 (zero)

service cgn ISM

protocol icmp

rate-limit 8184

!

!

ICMP Rate-Limiting


A customer requested to limit the number of internal users allowed to used each external addresses of their map pool. Only for NAT44 (no dynamic-range config in DS-Lite)

Step 1: define port-limit and bulk-port-range to the same value.

– Ex: 4096 ports: rounddown[(65535-1024)/4096]=15 potential inside addresses for each external address

– Ex: 2048 ports: rounddown[(65535-1024)/2048]=31

– BPA=1024 63

– BPA=512 126, …

Step 2: if we need to reduce the number of users to something smaller than 15, let define the dynamic-port-range to an higher value

– Ex: BPA/port-limit=4096, dynamic-range start=24575 rounddown[(65535-24575)/4096]=10

Using these Features Creatively How to reduce the number of users per external address?


Not possible to change the DSCP marking of syslog or netflow packets generated by ISM or CGSE card. But a remarking can be done at the egress interface level with the proper QoS policy

RP/0/RP1/CPU0:Yanks#show policy-map interface gig 0/6/3/0.2

GigabitEthernet0/6/3/0.2 direction input: Service Policy not installed

GigabitEthernet0/6/3/0.2 output: NF

Class NF

Classification statistics (packets/bytes) (rate - kbps)

Matched : 37991/53199036 838

Transmitted : 37991/53199036 838

Total Dropped : 0/0 0

Queueing statistics

Queue ID : 23

Taildropped(packets/bytes) : 0/0

Class class-default

Classification statistics (packets/bytes) (rate - kbps)

Matched : 0/0 0

Transmitted : 0/0 0

Total Dropped : 0/0 0

Queueing statistics

Queue ID : 23

High watermark (bytes)/(ms) : 0/0

Inst-queue-len (bytes)/(ms) : 0/0

Avg-queue-len (bytes)/(ms) : 0/0

Taildropped(packets/bytes) : 0/0

RP/0/RP1/CPU0:Yanks#sh run policy-map

Wed Sep 5 03:46:20.324 PDT

policy-map NF

class NF

set dscp cs5

!

class class-default

!

end-policy-map

!

Changing Logging DSCP Marketing


NetFlow v9 / CS5

Syslog / CS5

Changing Logging DSCP Marketing


Makes sure the traffic is indeed pushed to and from the CGN cards

Show interface serviceApp * … is always expressed from the router perspective, so

– Pkts out: going into the CGN cards

– Pkts in: coming from the CGN cards into the router

RP/0/RSP0/CPU0:Nets#sh int serviceapp * accounting

ServiceApp1

Protocol Pkts In Chars In Pkts Out Chars Out

IPV4_UNICAST 2810763 348534612 37102220124 37515911766210

ServiceApp2

Protocol Pkts In Chars In Pkts Out Chars Out

IPV4_UNICAST 36742436201 37162233422198 0 0

RP/0/RSP0/CPU0:Nets#

Troubleshooting Tips


We can use “show interface serviceApp * accounting rates” to get some trends on the traffics going through the system



When using ABF: configure hardware count in ABF in order to see ABF match statistics

You should see Hits increase as ingress traffic is directed to ServiceApp NH


vrf LOOPBACK

ipv4 address 12.1.7.10 255.255.255.0

load-interval 30

ipv4 access-group ABF ingress hardware-count

!

RP/0/RP0/CPU0:router#show access-lists ABF hardware

ingress detail location 0/0/CPU0

ACL name: ABF

Sequence Number: 10

Grant: permit

Logging: OFF

Per ace icmp: ON

Next Hop Enable: ON

VRF Table Id: 4096

Next-hop: 1.1.1.2

Default Next Hop: OFF

Hits: 4063640803

Statistics pointer: 0x7ff5f

Number of TCAM entries: 1



RP/0/RSP0/CPU0:BNG#run attach 0/5/cpu0

Sat Dec 22 06:33:02.403 UTC

attach: Starting session 1 to node 0/5/cpu0

#

#

# show_nat44_stats

CORE-ID #SESSIONS(%UTIL) #USERS(%UTIL)

------------------------------------------------------------------------

0 563100(19.6%) 1877(1.43%)

1 561000(19.5%) 1870(1.43%)

2 563400(19.6%) 1878(1.43%)

3 562500(19.6%) 1875(1.43%)

4 0(0.0%) 0(0.00%)

5 0(0.0%) 0(0.00%)

6 0(0.0%) 0(0.00%)

7 0(0.0%) 0(0.00%)

------------------------------------------------------------------------

Total Sessions: 2250000 Total users: 7500

Main DB size is 2875008 and User DB size is 131072

#exit

RP/0/RSP0/CPU0:BNG#

Be extra careful with the unix level commands, one is very useful though:

Troubleshooting Tips on ISM


# show_nat44_stats

CORE ID #SESSIONS(UTIL) #USERS(UTIL)

-----------------------------------------------------------------

0 40194(11.2%) 5109(31.18%)

1 40541(11.3%) 5085(31.04%)

2 44626(12.4%) 5143(31.39%)

3 42984(12.0%) 5121(31.26%)

4 44286(12.3%) 5171(31.56%)

5 43361(12.1%) 5154(31.46%)

6 43394(12.1%) 5048(30.81%)

7 39203(10.9%) 5124(31.27%)

8 43285(12.0%) 5122(31.26%)

9 44728(12.4%) 5091(31.07%)

10 41258(11.5%) 5128(31.30%)

11 43362(12.1%) 5108(31.18%)

12 44791(12.5%) 5218(31.85%)

13 44026(12.2%) 5147(31.41%)

14 41399(11.5%) 5146(31.41%)

15 45238(12.6%) 5148(31.42%)

16 45989(12.8%) 5087(31.05%)

17 42037(11.7%) 5068(30.93%)

18 40363(11.2%) 5125(31.28%)

19 39819(11.1%) 5136(31.35%)

20 44321(12.3%) 5133(31.33%)

21 40380(11.2%) 5159(31.49%)

22 44183(12.3%) 5137(31.35%)

23 43153(12.0%) 5164(31.52%)

24 44762(12.5%) 5098(31.12%)

25 44317(12.3%) 5092(31.08%)

26 45482(12.7%) 5153(31.45%)

27 38451(10.7%) 5127(31.29%)

28 40848(11.4%) 5149(31.43%)

29 44388(12.3%) 5116(31.23%)

30 42729(11.9%) 5120(31.25%)

31 41428(11.5%) 5081(31.01%)

32 43292(12.0%) 5028(30.69%)

33 40294(11.2%) 5077(30.99%)

34 40734(11.3%) 5066(30.92%)

35 43167(12.0%) 5083(31.02%)

36 43519(12.1%) 5110(31.19%)

37 42372(11.8%) 5116(31.23%)

38 44425(12.4%) 5035(30.73%)

39 42546(11.8%) 5063(30.90%)

40 40284(11.2%) 5072(30.96%)

41 42166(11.7%) 5068(30.93%)

42 40136(11.2%) 5110(31.19%)

43 44040(12.3%) 5084(31.03%)

44 38744(10.8%) 5115(31.22%)

45 37815(10.5%) 5078(30.99%)

46 42205(11.7%) 5075(30.98%)

47 42783(11.9%) 5068(30.93%)

48 40146(11.2%) 5105(31.16%)

49 40471(11.3%) 5080(31.01%)

50 40798(11.4%) 5107(31.17%)

51 44311(12.3%) 5110(31.19%)

52 40794(11.3%) 5119(31.24%)

53 40354(11.2%) 5136(31.35%)

54 41776(11.6%) 5016(30.62%)

55 42932(11.9%) 5115(31.22%)

56 43001(12.0%) 5022(30.65%)

57 40488(11.3%) 5026(30.68%)

58 41422(11.5%) 5072(30.96%)

59 39293(10.9%) 5064(30.91%)

60 43408(12.1%) 5044(30.79%)

61 44388(12.3%) 5083(31.02%)

62 40447(11.3%) 5100(31.13%)

63 42022(11.7%) 5073(30.96%)

Troubleshooting Tips on CGSE


Optionally, configure Diagnostics on the CGSE card

If we use redundant cards, active being in 0/0/CPU0

RP/0/RP0/CPU0:CRS(config)#

service-plim-ha location 0/0/CPU0 datapath-test

service-plim-ha location 0/0/CPU0 core-to-core-test

service-plim-ha location 0/0/CPU0 pci-test

service-plim-ha location 0/0/CPU0 coredump-extraction

service-plim-ha location 0/0/CPU0 linux-timeout 500

service-plim-ha location 0/0/CPU0 msc-timeout 500

!

An error detected will trigger the reload of the PLIM.

If the card is in stand-alone (no redundancy), we add the configuration:

RP/0/RP0/CPU0:CRS(admin-config)#

hw-module reset auto disable location 0/0/CPU0

!

Online Diagnostics


Optionally, configure Diagnostics on the ISM card

RP/0/RP0/CPU0:ASR9000(config)#

service-cgv6-ha location 0/2/CPU0 puntpath-test

service-cgv6-ha location 0/2/CPU0 datapath-test

!

Online Diagnostics


Per Blade Limits CGSE CGSE+ ISM VSM

NAT44 instances supported 1 per card 1 per card 1 per card 1 (at FCS)

DS Lite instances

supported

64 per chassis N/A 64 per chassis Future

6rd instances supported 64 per chassis 64 per chassis ? Future

NAT64 instances supported 64 per chassis N/A ? Future

Number of service infra 1 1 1 1

Number of service app 890 (2000 per

system)

? 244 (per system) 4096

IP pool supported /16 to /26 (max 65535 addresses)

/16 to /26 (max 65535

addresses)

Future: longer prefix

/16 to /30 (max 65535

addresses)

/16 to /30 (max 65535

addresses)

Max Static Port forwarding 2K tested 6K 6K 6K

Max number of NAT users 1M 1M (2M) 1M 4M

Performance / Scalability


Parameter CGSE CGSE+ ISM VSM

Configuration CLIs Same Same Same Same

Uses SVI Yes Yes Yes Yes

Network Processor Yes (Metro) Yes (Pogo) No, handled by a

dedicated process

Yes (Typhoon)

Packet distribution One level:

NAT44 load-balancing

on egress Metro

One level:

NAT44 load-balancing

on egress Pogo

Two levels

a) by ingress LC using VQI

b) NAT44 load-balancing

within Dispatcher process

?

Egress FIB Lookup On iMetro On iPogo Within CGv6 App On

ServiceApp placement Anywhere Anywhere Associated with Niantic

port/VQI

Associated with

NP ports / Niantic ports

# of CGv6 instances 64 (4 octeons) 8 (2 Westmeres) 48 (in 2 logical groups)

Stateless protocols (in CGN

card)

6rd, NAT64SL 6rd, (NAT64SL future) 6rd, MAP-T/E Future: 6rd, MAP-T/E

Inline support No No Yes for SL protocols Future

Comparing the CGN Platforms

BACKUP SLIDES PPTP ALG DETAILS


NAT IPv4

Internet PAC PNS


PPTP

Outgoing Call Request Inside Call-ID

Outgoing Call Reply Outside Call-ID Inside Call-ID

Outbound

Call

Translation

DataBase

Two tuples are mapped and an entry is created in the translation DB

PPTP ALG


NAT IPv4

Internet PAC PNS


PPTP

Translation

DataBase

Two tuples are mapped and an entry is created in the translation DB

Incoming Call Reply Inside Call-ID

Incoming Call Request Outside Call-ID

Outside Call-

ID

Inbound

Call

PPTP ALG


NAT IPv4

Internet PAC PNS


PPTP

Translation

DataBase

Depending on the side initiating the disconnection, the Inside-Call-ID or Outside-Call-ID tuple will be marked for deletion from the translation DB

Call Clear Request Inside Call-ID

Outside Call-ID

Disconnect

Call Disconnect Notify

PPTP ALG

Carrier Grade NAT44 on IOS-XR Deployment...

Documents

Transcript of Carrier Grade NAT44 on IOS-XR Deployment...