Advanced NetFlow Accounting - Net130.Com · Advanced NetFlow Accounting Session NMS-4031. ... •...

Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr

111© 2003, Cisco Systems, Inc. All rights reserved.NMS-40317949_05_2003_c2


Advanced NetFlow AccountingSession NMS-4031



Table of Content

• NetFlow Basics

• NetFlow VersionsNetFlow on the Router (Version 5)NetFlow on the Router (Version 8) NetFlow on the Switches (Version 7…Version 8)NetFlow Version 9

• Advanced Concepts

• New Features

• Roadmap and Future Directions

• Appendix A: NetFlow Compared to Other Methods


This Tutorial Is Not about…

• A level 1 type of presentation• An (long) introduction about NetFlow• Marketing slides• NetFlow Collector details • Ecosystem partners applications and

mediations• Prerequisite:

NSC-1031, “Introduction to Collecting Traffic Accounting Information”Or previous NetFlow knowledge



NetFlow Basics

5


NetFlow Infrastructure

Router:• Cache creation• Data export• Aggregation

Router:• Cache creation• Data export• Aggregation

CiscoCisco

Collector:• Collection• Filtering• Aggregation• Storage

Collector:• Collection• Filtering• Aggregation• Storage

Cisco and PartnersCisco and Partners

RMON/NAMRMON/NAM

Applications:

AccountingBilling

Network Planning

• Data processing• Data presentation

Partners

RMON Application



NetFlow Partners

CollectionCollection

Traffic AnalysisTraffic AnalysisBillingBilling

Denial of ServiceDenial of Service


NetFlow Possible Applications

NetFlowNetFlow

Network PlanningNetwork Planning

Application MonitoringApplication Monitoring

Security AnalysisSecurity Analysis

User MonitoringUser Monitoring

Peering AgreementPeering Agreement

Traffic EngineeringTraffic Engineering

Network MonitoringNetwork Monitoring

XX

Usage-Based BillingUsage-Based Billing

XX

Destination Sensitive BillingDestination Sensitive Billing

XX

XX

XX

XX

XX

XX

XX



Exported Data

What Is a NetFlow Flow?

7 Keys Define a Flow• Source Address• Destination Address• Source Port• Destination Port• Layer 3 Protocol Type• TOS byte (DSCP)• Input Logical Interface

(ifIndex)

A Flow Is Unidirectional


How Does NetFlow Work?

7 Identifiers Other DataFlow IdentifiersFlow Identifiers Flow Data Flow Data

Flow IdentifiersFlow Identifiers Flow DataFlow Data

UpdateUpdate

Flow IdentifiersFlow Identifiers Flow DataFlow Data

7 Identifiers Other Data

Exported Data via UDP (*)

(*) for Speed and Simplicity



• Answers questions regarding your traffic: who, what, where, when, and how

• NetFlow became the de facto IP accounting standard throughout the industry

• Support on all interface types

• Supported on fast switching, Cisco Express Forwarding (CEF) and Distributed CEF

NetFlow Principles


NetFlow Principles

• Not a switching path

• 7 flow identifiers

• Unidirectional traffic

• For ingress traffic only (*)

• IP unicast only (*)

• Export via UDP (*)

(*) See Roadmap



NetFlow on the RouterVersion 5

13


Version 5

• Version 5 adds BGP autonomous system

• Supported on router starting from 11.1 CA and 12.0

• The most deployed version

• The most complete version in terms of exported data types

• No reason to use NetFlow version 1 unless supporting a legacy collection system



Routing andPeering

• Next Hop Address• Source AS Number• Dest. AS Number• Source Prefix Mask• Dest. Prefix Mask


Timeof Day

• Start SysUpTime• End SysUpTime• Start SysUpTime• End SysUpTime

Also Available via RMON Available via NetFlow Only




Version 5 Flow Format

From/To• Source IP Address• Destination IP Address• Source IP Address• Destination IP Address

Application

• Source TCP/UDP Port• Destination TCP/UDP Port• Source TCP/UDP Port• Destination TCP/UDP Port

PortUtilization

• Input IfIndex• Output IfIndex• Input IfIndex• Output IfIndex

QoS• Type of Service• TCP Flags• Protocol

• Type of Service• TCP Flags• Protocol

Usage• Packet Count• Byte Count• Packet Count• Byte Count


Version 5 Export

Flow 1

NetFlow Cache

Flow Entries

• Flow expired• Cache full• Timer expired

Flow 2

Flow 3

To Collector

UDPExport V5 Record

The Default Inactive Timeout: 15 Sec.

The Default Active Timeout: 30 Min.



Version 5 Configuration

router (config-if)#ip route-cache flow

router (config)#ip flow-export destination 172.17.246.225 9996

router (config)#ip flow-export version 5 <peer-as | origin-as>

Optional configuration

router (config)#ip flow-export source loopback 0

router (config)#ip flow-cache entries <1024-524288>

router (config)#ip flow-cache timeout …


Version 5 Show Commands

martel#sh ip cache verbose flowIP packet size distribution (94452 total packets):

1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480.000 .199 .342 .300 .094 .028 .012 .005 .013 .000 .001 .000 .000 .000 .000

512 544 576 1024 1536 2048 2560 3072 3584 4096 4608.000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 4456704 bytes1 active, 65535 inactive, 25322 added525430 ager polls, 0 flow alloc failureslast clearing of statistics never

Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec)-------- Flows /Sec /Flow /Pkt /Sec /Flow /FlowTCP-BGP 7 0.0 2 41 0.0 1.6 7.5UDP-TFTP 1 0.0 1 67 0.0 0.0 15.1UDP-other 19884 0.0 3 111 0.1 5.6 15.4ICMP 5429 0.0 3 41 0.0 0.9 15.5Total: 25321 0.0 3 97 0.2 4.6 15.4

SrcIf SrcIPaddress DstIf DstIPaddress Pr TOS Flgs PktsPort Msk AS Port Msk AS NextHop B/Pk ActiveSe0/1 193.1.1.3 Se0/0 172.17.246.228 11 00 10 5 00A1 /24 193 C628 /0 0 0.0.0.0 84 39.7



BGP Autonomous System

AS 101 AS 103 AS 104

AS 105

AS 106

AS 102

Note: The AS Fields Will Remain Empty unless You Configure It Explicitly with peer -as or origin-as

NetFlow Enabled

Router(config)#ip flow-export version 5 peer-as

Configuring Peer-AS• Source AS = AS 103• Destination AS = AS 105


BGP Autonomous System

AS 101 AS 103 AS 104

AS 105

AS 106

AS 102

NetFlow Enabled

Configuring Origin-AS• Source AS = AS 101• Destination AS = AS 106

Router(config)#ip flow-export version 5 origin-as

Note: The AS Fields Will Remain Empty unless You Configure It Explicitly with peer -as or origin-as



NetFlow on the RouterVersion 8

21


Introduction

• Router-based aggregation, i.e. version 8

• Enables router to summarize NetFlow data

• Reduces NetFlow export data volume

• Decreases NetFlow export bandwidth requirements

• Making collection easier



Introduction

• Supported from 12.0(3)T, 12.0(3)S and 12.1 On-board aggregation, the router maintains extra NetFlow cache(s), for aggregation(s)

• Still needs the main cache (for export with version 5)

• When flows expire from the main cache, they are added to each enabled aggregation cache

• Several aggregations can be enabled at the same time


Version 8 Export

Flow 1

NetFlow Main Cache

Flow Entries


To Collector

Flow 2

Flow 3

UDP


Export V5 Record

• Cache full• Timers expired

Export V8 Record

To Collector

UDP

Export v5

Not NecessaryExport v

5

Not Necessary

Aggreg. Cache

AS-Matrix

Prefix-Matrix

......




Source Prefix MaskSource Prefix Mask

Destination App PortDestination App Port

ASAS

Source App PortSource App Port XXXX

IP ProtocolIP Protocol XX

First TimestampFirst Timestamp XX XX XX XX XXLast TimestampLast Timestamp XX XX XX XX XXNumber of FlowsNumber of Flows XX XX XX XX XXNumber of PacketsNumber of Packets XX XX XX XX XXNumber of BytesNumber of Bytes XX XX XX XX XX

XXSource PrefixSource Prefix XX XX

XXXX XX

Input InterfaceInput Interface XX XX XXOutput InterfaceOutput Interface XX XX XX

Source ASSource AS XX XX XXDestination ASDestination AS XX XX XX

PrefixPrefixDestination-PrefixDestination-PrefixSource -PrefixSource -PrefixProtocol-PortProtocol-Port

XX XXDestination Prefix MaskDestination Prefix Mask

Destination PrefixDestination Prefix



Source Prefix MaskSource Prefix Mask

Destination App PortDestination App Port

AS-TOSAS-TOS

Source App PortSource App Port XXXX

IP ProtocolIP Protocol XX

First TimestampFirst Timestamp XX XX XX XX XXLast TimestampLast Timestamp XX XX XX XX XXNumber of FlowsNumber of Flows XX XX XX XX XXNumber of PacketsNumber of Packets XX XX XX XX XXNumber of BytesNumber of Bytes XX XX XX XX XX

XXSource PrefixSource Prefix XX XX

XXXX XX

Input InterfaceInput Interface XX XX XXOutput InterfaceOutput Interface XX XX XX

Source ASSource AS XX XXDestination ASDestination AS XX XX

Prefix-TOS

Prefix-TOS

Destination-Prefix-TOS

Destination-Prefix-TOS

Source-Prefix-TOS

Source-Prefix-TOS

Protocol-Port-TOS

Protocol-Port-TOS

XX XXDestination Prefix MaskDestination Prefix Mask

TOS (Actually DSCP)TOS (Actually DSCP) XX XX XX XX XX

XXXX

XX

XXXX

Destination PrefixDestination Prefix



Version 8 Configuration

router (config)# ip flow-aggregation cache as

router (config-flow-cache)# export destination 172.17.246.225 9996

router (config-flow-cache)# enabled

router#sh ip cache flow aggregation as

IP Flow Switching Cache, 278528 bytes 2 active, 4094 inactive, 13 added 216 ager polls, 0 flow alloc failures

SrcIf SrcAS DstIf DstAS Flows Pkts B/Pk Active

Se0/0 0 Se0/2.1 0 1 1 104 0.0

Se0/0 0 Null 0 1 1 59 0.0


NetFlow on the SwitchesVersion 7…Version 8

1. MLS Specific

2. CEF Specific

3. Generic Info

28



NetFlow Version 7

• NetFlow version 7 is only export from the switches

• Support for Catalyst® switches with a Layer 3 board:

Catalyst 5000 with a RSM (Route Switch Module)

Catalyst 6500/7600 with a MSFC (Multilayer Switching Feature Card)

• A Catalyst 6500/7600 uses: Multilayer Switching (MLS) with a SUP1

Cisco Express Forwarding (CEF) with SUP2



1. MLS Specific (SUP1)

30



MLS Example

Supervisor 1

MSFC

Vlan1

Vlan14

Candidate Packet

Layer 3 Switched, after the Shortcut Creation

Enable Packet


MLS ExampleAccounting Point of View

Supervisor 1

MSFC

Vlan1

Vlan14

Ping #1

Ping #2

Ping #3

Ping #4

Ping #5




PortUtilization

QoS

Usage

Timeof Day

• Packet Count• Byte Count• Packet Count• Byte Count






Application




Routing andPeering

• Next Hop Address• Source AS Number• Dest. AS Number• Source Prefix Mask• Dest. Prefix Mask• RouterSc

(Router Shortcut)

• Next Hop Address• Source AS Number• Dest. AS Number• Source Prefix Mask• Dest. Prefix Mask• RouterSc

(Router Shortcut)Added from Version 5Added from Version 5

Note that Some of Fields Are Not Populated; See Slide 53/54


Only Export the First Packet of the Flow Unless You Don’t Use MLS…

Only Export the First Packet of the Flow Unless You Don’t Use MLS…

Bad Design

Vlan1

Vlan14

NFC

Supervisor 1

MSFC

Export

MLS (Not) Enabled and Export v5 from the MSFC



Approximate Design

Export

Vlan1

Vlan14

NFC

Supervisor 1

MSFC

Miss the Accounting of the First Packet of the Flow

Miss the Accounting of the First Packet of the Flow

MLS Enabled and Export v7 from the SUP1


Better Design

Export

Vlan1

Vlan14

NFC

Supervisor 1

MSFC

Export

MLS Enabled and Export v7 from the SUP1Export v5 from the MSFC



Best Design

Export

Vlan1

Vlan14

NFC

Supervisor 1

MSFC

Export

Otherwise, You Will Also Count the Export Traffic

Otherwise, You Will Also Count the Export Traffic

MLS Enabled and Export v7 from the SUP1Export v5 from the MSFC

And Export in the sc0 vlan (sc0 in vlan1)


# In case of V7, set USE_SHORT_CUT_ADDRESS_AS_SOURCE_IP# to "yes" so that FlowCollector will use the address # of the router being short-cut as the source of the # corresponding flow. Default is set to No

USE_SHORT_CUT_ADDRESS_AS_SOURCE_IP No

Better/Best Design ProblemExport from 2 Different “Devices”

• No supervisor/MSFC flow records correlation

• Change the nf.resources configuration file




2. CEF Specific(SUP2, MSFC2)

40


DCEF Example

Supervisor 2

MSFC2

Vlan1

Vlan14

No Entry in the SUP2 FIB

Entry Created in the MSFC FIB

All Entries Go through the SUP2 FIB

FIB Synchronisation



MLS Best DesignDoes It Still Make Sense for CEF?

Export

Vlan1

Vlan14

NFC

Supervisor 2

MSFC2

Export

MLS Enabled and Export v7 from the SUP2Export v5 from the MSFC2

And Export in the sc0 vlan


MLS Best DesignDoes it Still Make Sense for CEF?

• (Yes) the MSFC2 will count the first packet of a destination, the one which will complete the glean adjacency; needed for precise accounting

• (No) the MSFC2 will ONLY count the first packet of a destination, the one which will complete the glean adjacency

With MLS, the MSFC will count the first packet of every single flow

• (No) the FIB entries remain the time of the ARP entries; not updated so often as the MLS entries!

With MLS, the SUP shortcut disappears when the flow expires



MLS Best DesignDoes it Still Make Sense for CEF?

• (No) most NetFlow entries on the MSFC will haveDstIf = Null (even if the packet is switched by the MSFC)

Dstif = Local (destination = MSFC)

With MLS, the DstIf correctly populated

• (Yes) some features will always go through the MSFC: NAT, IP access-list with log, etc…

• Conclusion:The MSFC is needed for accounting accuracybut less important as for MLS, as it will report less flow records


Catalyst 6500 NetFlow Version 5 Support

• Native mode: SUP2/PFC2 supports NetFlow version 5 from 12.1(13)E

http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/12_1e/swconfig/nde.htm

• Hybrid mode: SUP2/PFC2 supports NetFlow version 5 from 7.5(1)

• As a consequence…we don’t have the better/best design issue that we had with MLS: i.e. the correlation from two different sources IP addresses

New



• Next Hop Address• Source AS Number• Dest. AS Number• Source Prefix Mask• Dest. Prefix Mask• RouterSc (Router

Shortcut)

• Next Hop Address• Source AS Number• Dest. AS Number• Source Prefix Mask• Dest. Prefix Mask• RouterSc (Router

Shortcut)

Added from Version 5Added from Version 5

Version 7 Flow Format with CEFDon’t Need the RouterSc Field = Version 5

PortUtilization

QoS

Usage

Timeof Day

• Packet Count• Byte Count• Packet Count• Byte Count






Application


Routing andPeering

Note that Some of Fields Are Not Populated


Catalyst 6500, Native Mode

mls flow ip full -> flow maskmls nde src_address 10.200.8.127 version 7

-> version 7 export source ORmls nde sender -> NDE enable + NDE from the PFC uses the

source configured from the MSFC!!!!!interface vlan 1ip address 10.200.8.127 255.255.255.0ip route-cache flow

interface FastEthernet 3/2ip address 10.300.8.2 255.255.255.0ip route-cache flow

ip flow-export source vlan1 -> version 5 export sourceip flow-export version 5ip flow-export destination 172.17.246.244 9996

-> both for version 5 and 7 export




3. Generic Info

51


ContentContent V5V5 V7(*)V7(*)

Source IP AddressSource IP Address

Destination IP AddressDestination IP Address

Source TCP/UDP PortSource TCP/UDP Port

Next Hop Router IP AddressNext Hop Router IP Address

Input Physical Interface IndexInput Physical Interface Index

Output Physical Interface IndexOutput Physical Interface Index

Packet Count for This FlowPacket Count for This Flow

DestinationTCP/UDP PortDestinationTCP/UDP Port

Start of Flow TimestampsStart of Flow Timestamps

End of Flow TimestampsEnd of Flow Timestamps

XX

XX

XX

XX

XX

XX

XX

XX

XX

XX

Zero in Case of Destination-OnlyZero in Case of Destination-Only

XX

Zero in Case of Destination-Only or Source-Destination




XX

XX

XX

XX

XX

XX

New

New

New

Format Comparison

12.1(13)E12.1(13)E

12.1(13)E12.1(13)E

12.1(13)E12.1(13)E

(*) Applies Also to the New Version 5 Specific to the Switches



Format Comparison

ContentContent V5V5 V7(*)V7(*)

Type of Service ByteType of Service Byte

TCP FlagsTCP Flags

Destination AS NumberDestination AS Number

Source Subnet MaskSource Subnet Mask

Destination Subnet MaskDestination Subnet Mask

Flags (Indicate Invalid Field within the Flow)

Flags (Indicate Invalid Field within the Flow)

Source AS NumberSource AS Number

Shortcut Router IP AddressShortcut Router IP Address

XX

XX

XX

XX

XX

XX

XX



PFC1: Set to the First Packet TOS; PFC2: Not Populated

PFC1: Set to the First Packet TOS; PFC2: Not Populated

XX

XX

XX

New

New

IP Protocol (TCP=6, UDP=17) IP Protocol (TCP=6, UDP=17)

Always ZeroAlways Zero



XX

12.1(13)E12.1(13)E

12.1(13)E12.1(13)E

(*) Applies Also to the New Version 5 Specific to the Switches


Source IP AddressSource IP Address

Destination IP AddressDestination IP Address

Source App PortSource App PortDestination App PortDestination App PortIP ProtocolIP ProtocolFirst TimestampFirst TimestampLast TimestampLast Timestamp

# of Flows# of Flows# of Packets# of Packets

# of Bytes# of Bytes

RouterDstOnlyRouterDstOnly RouterSrcDstRouterSrcDst Router Full FlowRouter Full Flow

XX

XX

XXXXXXXXXX

XXXX

XX

XX

XX

XXXX

XXXX

XX

XXXX

XX

XX

XX

Cat6500 Aggregations—Version 8

• Since CatOS version 5.5(2); not yet on native• For both SUP1 and SUP2

XX



NetFlow Version 9

55

New


NetFlow Version 9 Why a New Version?

• Fixed formats (versions 5, 7, and 8) are not flexible and extensible:

Cisco needed to build a new version each time a customer wanted to export new fields

Both on the devices and the NetFlow Collector

• When new versions are created, partners need to reengineer to support the new export format

Solution: Build a Flexible and Extensible Export Format!



NetFlow Version 9 Scenario #1

Templates Definition Export

Flow Records Export

Template Definition Stored

Decode andInterpretation

Flow Records Stored


NetFlow Version 9 Scenario #2

Flow Records Export

Template Definition Stored

Decode andInterpretationTemplates Definition Export

• The NetFlow collector should store the flow record and decode it after the template definition is received

Flow Records Stored



NetFlow Version 9 Principles

• Version 9 is an export protocolNo changes to the metering processCan be used in conjunction with the main cache,

For example, MPLS aware NetFlowCan be used in conjunction with an aggregation cache,

For example, BGP Next Hop TOS aggregation

• Version 9 based on templates and separate flow records

Templates composed of type and lengthFlow records composed of template ID and value

• Available in 12.0(24)S


NetFlow Version 9 Principles

• Still a push model

• Sent the template regularly (configurable)Because we still use UDP as transport protocol

• Independent of the underlying protocol, it is ready for any reliable protocol (i.e. TCP, SCTP)SCTP: Stream Control Transport Protocol

• Advantage: we can add new technologies/data types very quickly

Example: MPLS, multicast, BGP next HOP

Just update the information model, composed initially of the NetFlow version 5, 7 and 8 data types

• http://www.cisco.com/warp/public/cc/pd/iosw/prodlit/tflow_wp.htm



Extensibility and FlexibilityPhases Approach

• Phase 1: NetFlow version 9, completedAdvantages: extensibility

Integrate new technologies/data types quicker

Integrate new aggregations quicker

Note: for now, the template definitions are fixed!

• Phase 2: flexible flow keys, under investigationAdvantages: cache content flexibility

Selection of a subset of the 7 flow keys

New flow keys will be defined and available

• Phase 3: user defined templates, radarAdvantage: export content flexibility

Selection of the data types to export


Version 9Example for Template Definition

2

L4_PROTOCOL

2

DST_AS_NUMBER2

SRC_AS_NUMBER

3(# of Fields)

1001(Template ID)

Length of TemplateStructure

Flow Set ID (0 for Template)

Template A

PACKET_COUNT

2

SRC_AS_NUMBER4

SRC_IP_PREFIX

4(# of Fields)

1002(Template ID)

Length of TemplateStructure

Flow Set ID (0 for Template)

Template B

2

2BYTE_COUNT



Tem

pla

te A

Data for Template B

Same as Template ID for Template B; Refer to

Previous Slide

Record 1 Record 2

Data for Template A

Example for Export Packet

Tem

pla

te B

Packet Header

10022(# of Records)

1.1.1.1 2.2.1.1

20 64

365 20

92894 1000

10011

35

23

700

As Defined in the Previous Slide


NetFlow Version 9 Configuration

router(config)# ip flow-export version ?

1

5

9

router(config)# ip flow-export version 9 .

Configuring Version 9 Export for the Main Cache

Configuring Version 9 Export for an Aggregation Scheme

Export Versions Available for NetFlow FlowsExport Versions Available for NetFlow Flows

router(config)# ip flow-aggregation cache as

router(config-flow-cache)# enabled

router(config-flow-cache)# export ?

destination Specify the Destination IP address

version configure aggregation cache export version

router(config-flow-cache)# export version ?

8 Version 8 export format

9 Version 9 export format

router(config-flow-cache)# export version 9

Export Versions Available for Aggregated NetFlow FlowsExport Versions Available for Aggregated NetFlow Flows



NetFlow Version 9 IETF Considerations

65


IETF: IP Flow Information Export WG (IPFIX)

• IPFIX is an effort to:

Define the notion of a "standard IP flow"

Devise data encoding for IP flows

Consider the notion of IP flow information export based upon packet sampling

Identify and address any security privacy concerns affecting flow data

Specify the transport mapping for carrying IP flow information (IETF approved congestion-aware transport protocol)



IETF: IP Flow Information Export WG (IPFIX)

• IPFIX web site for the charter, email archive, drafts, etc.

http://ipfix.doit.wisc.edu/

• Requirements draft: http://www.ietf.org/internet-drafts/draft-ietf-ipfix-reqs-09.txt

• NetFlow version 9 has recently been selected as a basis for the IPFIX protocol

Out of 5 existing protocols: CRANE from Xacct, LFAP from Riverstone, Diameter (RADIUS extension), IPDR

Based on the requirements draft

New


NetFlow Version 9 as the Basis for the IPFIX Protocol

• Requested minor improvements to the NetFlow version 9

• The initial IPFIX protocol will run on the top of TCP, as an interim solution, while waiting for standardization of Stream Control Transport Protocol Partial Reliability (SCTP-PR) or Datagram Congestion Control Protocol (DCCP)

“We believe that the IPFIX protocol, based on NetFlow v9, can be implemented in the most network elements because it makes the least demands of the exporter.”The IPFIX Evaluation Team



IETF: Packet SAMPling WG (PSAMP)

• PSAMP is an effort to:Specify a set of selection operations by which packets are sampled

Specify the information that is to be made available for reporting on sampled packets

Describe protocols by which information on sampled packets is reported to applications

Describe protocols by which packet selection and reporting configured


IETF: Packet Sampling WG (PSAMP)

• PSAMP web site for the charter, email archive, drafts, etc.

http://psamp.ccrle.nec.de/

• Agreed to use IPFIX for export protocol if suitable for PSAMP

To be improved: the variable length data type

• Note: NetFlow is already using some sampling mechanisms

New



Advanced Concepts

71


VIPRP

Main Cache(s) with VIP and Line Card

VIP2

FIB NetFlow

FIB FIBNetFlow NetFlow



VIPRP

VIP2

Aggregation Cache(s) with VIP and Line Card

.

...

.

.

Agg.MainFIB FIB

FIB

Main

Main

Agg.

Agg.


VIP/LC Caches

• Nothing to configure on the VIP/LC (use DCEF)

• VIP: if-con <slot-number>sh ip cache flow

• LC: attach <slot-number>sh ip cache flow

Execute-on <slot-number> show…

• Own independent sequence numbering per VIP/LC



NetFlow on the 12000 Router

• Engine 0—software support, both “full” and sampled NetFlow

• Engine 1—software support, both “full” and sampled NetFlow

• Engine 2—supported in ASICs, sampled NetFlow only

• Engine 3—version 5 support in software, version 8 support in ASICs, sampled NetFlow only

• Engine 4—not supported• Engine 4+—supported in ASICs, sampled

NetFlow v5/v8 only


Timing IssuesWhen Is a Flow Expired?

• Transport is completed (TCP FIN or RST)

• After 15 sec of traffic inactivity (the only way for UDP); the inactive timer

• After 30 min of traffic activity; the active timer

Note that 15sec/30min are the router default timers

• The cache is becoming full

• Note: Flow expiration from an aggregation cache will go through 2 sets of timer

Firstly the main cache timer

Secondly the aggregation cache timer



Timing IssuesVarious Time in NetFlow

UTC Time in Header

Time

Flow Ends

Flow StartsRouter Boots

Flow Exported1970

Flow End sysUpTime

Flow Start sysUpTime

Router sysUpTime in Header

DeducedDeduced


Timing IssuesVarious Time in NetFlow

• The UTC depends on the clock• Synchronization of the VIP clock, the line card

clock (in sync. since 12.0) and the RSM/MSFC clock

• Attention to the time zone on the collector• Conclusion: the device clocks must be

synchronized• NTP is a solution, NTP MIB in 12.1(4)• Which synchronization time?

Only important if you want to correlate flow records from different devicesNote that NetFlow time granularity is msec



N

Create an NetFlowEntry with

Output I/f Null

Create an NetFlowEntry with

Output I/f Null

Discard the Packet

Discard the Packet

Y

Create anNetFlow Entry

Create anNetFlow Entry

Forward the Packet with CEF


NetFlow Bypasses the Access-ListNetFlow Acceleration

Pass theACL?

Pass theACL?

Y

N



Update the NetFlow Entry Stats

Update the NetFlow Entry Stats

Y

Go Through the ACLMaybe Deny PacketGo Through the ACLMaybe Deny Packet

Update theNetFlow Entry Stats

Update theNetFlow Entry Stats

ACL AccelerationACL Acceleration

Output i/fIs Null?

Output i/fIs Null?

N

Lookup Entry in NetFlow CacheLookup Entry in NetFlow Cache

First Packetin Flow?

First Packetin Flow?


NetFlow Performance

• Enabling NetFlow version 5 and exporting increases the CPU utilization by around 15% (with a max of 20% depending on the platform)

• Enabling NetFlow version 8 increases the CPU utilization by 2 to 5%, depending on the number of aggregations enabled with a multiple of 6% for multiple aggregations

• NetFlow is done in hardware on the Cat6500 supervisor; only the export takes CPU cycles

• http://www.cisco.com/warp/public/cc/pd/iosw/prodlit/ntfo_wp.htm

• NetFlow version 9: similar results as version 5



NetFlow Performance Results at a Glance

• CPU impact:

10,000 active flows: < 4% of additional CPU utilization

45,000 active flows: <12% of additional CPU utilization

65,000 active flows: <16% of additional CPU utilization

• NetFlow data export (single/dual): no real impact

• NetFlow feature acceleration: >200 lines of ACLs

• NetFlow sampled NetFlow on the Cisco 12000: 23% vs. 3% (65,000 flows, 1:100)


How to Reduce the CPU Utilization?

• RouterGo for sampled NetFlow (packet sampling)

Use the distributed feature card enable line card modules (VIP, LC)

Use 12000 engine 3 and 4+ (hardware)

• Catalyst 6500Go for sampled NetFlow (flow sampling)

Use the distributed feature card to enable line card modules

Reduce the flow mask



TroubleshootingMissing Flows?

1. Router Problem

3. Transfer Problem(Only Remaining Explanation)

2. NetFlow Collector Problemshow tech-supportNetstat -s

Cache (show ip cache flow)

Export (show ip flow export)

Export


New Features

88



• Inserted into 12.2(2)T, 12.0(19)S and 12.0(19)ST, 2 redundant export destinations are allowed for version 5

If you try to configure more, you will get:

“Exceeded maximum export destinations”

• Only for the routers (including GSR), not the Catalysts

Dual Flow Export

router(config)#ip flow-export destination 1.1.1.1 9996router(config)#ip flow-export destination 2.2.2.2 9997

New


NetFlow on Subinterface

• Introduced in 12.2(14)S, 12.2(15)T

• For the 7200, 7400 and 7500

• http://www.cisco.com/univercd/cc/td/doc/product/software/ios122/122newft/122t/122t15/ft_nfsub.htm

• Note: NetFlow reports the dot1Q subinterface ifIndex

Introduced in 12.2(7), 12.2(7)S, 12.2(7)T

Router(config-if)#ip flow ingress

New



Egress Sampled NetFlow

• Egress sampled NetFlow on 12000 engine 3, available in 12.0(24)S

• For both IP->IP and MPLS->IP traffic

• Egress sampled NetFlow on 12000 engine 3, available in 12.0(24)

router (config-if)# ip route-cache flow sampled [input|output]

New


NetFlow BGP Next Hop TOS Aggregation

• Key fields (uniquely identifies the flow)

Origin AS

Destination AS

Inbound interface

DSCP

Next BGP hop

Output interface

• Additional export fields

Flows

Packets

Bytes

First sysUptime

Last sysUptime

• New NetFlow aggregation on the router• Configure on ingress interface• Available in 12.0(26)S for the 7500

New



NetFlow BGP Next Hop TOS AggregationThe Core Traffic Matrix

PE

PE

PoP

AS1 AS2 AS3 AS4 AS5

Server Farm 1 Server Farm 2

Cu

sto

mer

s PE

PE

PoP

Cu

sto

mer

s

• “PoP to PoP”, the PoP being the CPE or CE

CPE CPE


NetFlow in a MPLS Environment

MPLS

P

Traffic Flow

IP

Traditional NetFlow

(IP -> MPLS)

MPLS Aware NetFlow

(MPLS -> MPLS)(IP -> IP)

New

IP

MPLS Egress NetFlow(MPLS -> IP)

New

PEPE PEPE



MPLS Egress NetFlowDescription

• Introduced in 12.0(10)ST, 12.1(5)T, 12.0(22)S

• For MPLS/VPN traffic only, i.e. the traffic coming from the core

• Caches traffic on the egress interface, not the ingress interface

• Valid for version 5 and version 8

• Can be enabled on sub-interfaces

• All other NetFlow commands still apply

router(config-if)#tag-switching ip flow egress

New


MPLS Aware NetFlowDescription

• Provides flow statistics per MPLS and IP packets

MPLS packets: Labels informationAnd the v5 fields of the underlying IP packet

IP packets:Regular IP NetFlow records

• Configure on ingress interface• Supported on 12.0(24)S on the 12000, then

will be in 12.0(26)S on the 7200/7500

New



MPLS Aware NetFlow Flow Keys

• Key fields (uniquely identifies the flow)

Source IP addressDestination IP addressIP protocolInput ifIndexSource application portDestination application portDSCPUp to 3 incoming MPLS labels of interest with experimental bits and end-of-stack bitPositions of the above labels in the packet label stack

• Additional export fieldsFlowsPacketsBytesFirst sysUptimeLast sysUptimeOutput interface NetFlow version 5 fields of the underlying IP packet (TCP flags, etc…)Type of the top label:LDP, BGP, VPN, ATOM, TE Tunnel MID-PT, unknownThe forwarding equivalent class mapping to the top label


P

P

PoP

Cu

sto

mer

s

PE

PE

PoP

Cu

sto

mer

s

Server Farm 1 Server Farm 2

AS1 AS2 AS3 AS4 AS5

MPLS Aware NetFlow The Core Traffic Matrix

WR

MPLSMPLS

PE

P

P

PE

PE

PE

• “PoP to PoP”, the PoP being the CPE or CE

CPE CPE



MPLS Aware NetFlowTop Label Aggregation (12.0(25)S)

• Key Fields (uniquely identifies the flow)

Input ifIndex

The top incoming MPLS labels with experimental bits and end-of-stack bit

• Additional export fieldsFlowsPacketsBytesFirst sysUptimeLast sysUptimeOutput interface NetFlow version 5 fields of the underlying IP packet (TCP flags, etc…)Type of the top label:LDP, BGP, VPN, ATOM, TE tunnel MID-PT, unknownThe forwarding equivalent class mapping to the top label

New


Multicast—Traditional NetFlow

• There is only one flow per NetFlow configured input interface

• The 7 key fields that define a unique flow are marked in red

• Destination interface is marked as “null”

• Bytes and packets are the incoming values

Interface Ethernet 0ip route-cache flow

ip flow-export version 9 ip flow-export destination x.x.x.x <port>

Eth 0Eth 3Eth 1

Eth 2

10.0.0.2

(S, G)—(10.0.0.2, 224.10.10.100)

SrclfSrclf

Eth 0

SrclPaddSrclPadd

10.0.0.2

Dstlf

NullNull

DstlPaddDstlPadd

224.10.10.100

ProtocolProtocol

11

TOSTOS

80

Flgs

10

SrcPortSrcPort

00A2

SrcMsk

/24

DstPortDstPortDstMskNextHopBytes

00A2 /24 2310023100

Packets

2121

Active

1745

Idle

4



Multicast NetFlow Ingress(Early Field Test)

• There is only one flow per NetFlow configured input interface

• The 7 key fields that define a unique flow are marked in red

• Destination interface is marked as “null”

• Bytes and packets are the outgoing values

Interface Ethernet 0ip multicast netflow ingress


SrclfSrclf

Eth 0

SrclPaddSrclPadd

10.0.0.2

Dstlf

NullNull

DstlPaddDstlPadd

224.10.10.100

ProtocolProtocol

11

TOSTOS

80

Flgs

10

SrcPortSrcPort

00A2

SrcMsk

/24


00A2 /24 6930069300

Packets

6363

Active

1745

Idle

4

Eth 0Eth 3Eth 1

Eth 2

10.0.0.2

(S, G)—(10.0.0.2, 224.10.10.100)

New


Multicast NetFlow Egress(Early Field Test)

Interface Ethernet 0

Interface Ethernet 1ip multicast netflow egress




Eth 0Eth 3Eth 1

Eth 2

10.0.0.2

(S, G)—(10.0.0.2, 224.10.10.100)

SrclfSrclf

Eth 0

SrclPaddSrclPadd

10.0.0.2

Dstlf

Null 1Null 1

DstlPaddDstlPadd

224.10.10.100

ProtocolProtocol

11

TOSTOS

80

Flgs

10

SrcPortSrcPort

00A2

SrcMsk

/24


00A2 /24 2310023100

Packets

2121

Active

1745

Idle

4

Eth 0 10.0.0.2 Null 2Null 2 224.10.10.100 11 80 10 00A2 /24 00A2 /24 2310023100 2121 1745 4

Eth 0 10.0.0.2 Null 3Null 3 224.10.10.100 11 80 10 00A2 /24 00A2 /24 2310023100 2121 1745 4

• There is one flow per multicast NetFlow egress configured output interface• One of the 7 key fields that define a unique flow has changed from source interface to

destination interface • Bytes and packets are the outgoing values



NetFlow Input Filters: Overview

• Support pre-filtering for traffic for NetFlow processing

• Modular QoS Command Line (MQC) will provide the filtering mechanism for NetFlow

Classification by IP source and destination addresses, layer 4 protocol and port numbers, incoming interface, MAC address, DSCPLayer 2 information such as Frame Relay DE bits, Ethernet 802.1p bitsNetwork Based Application Recognition (NBAR)

• Ability to sample filtered data at different rates, depending on how interesting the traffic is

• Currently early field test

New


NetFlow Input Filters: Example

NetFlow Cache

NetFlow Cache

VOIPVOIP

VPNVPN

Best EffortBest Effort

1:1 Sampling

1:1 Sampling

1:1000 Sampling1:1000

Sampling

1:100 Sampling

1:100 Sampling

Tight Filter for Traffic of High

Importance

Tight Filter for Traffic of High

Importance

Moderately-Tight for Traffic of

Medium Importance

Moderately-Tight for Traffic of

Medium Importance

Default Wide Open Filter for Traffic of Low Importance

Default Wide Open Filter for Traffic of Low Importance

PacketsPackets



NetFlow and IPv6

• Currently in EFT for 3600, 7200, 7500

• Based on NetFlow version 9

• For both ingress and egress traffic

• Non sampled

• No data export over IPv6 (still IPv4)

New


Catalyst 6500 New Fields Population

• The following CLI commands will be available in the release 7.3(1)

• Destination and source IfIndex support is enabled by default

set mls nde {destination-index|source-index} {enable|disable}

New



Catalyst 6500 New Fields Population and Version 5

• SUP2/PFC2 (EARL6) supports from 12.1(13)E:

Source and destination BGP AS

Input and output if indexes

Next hop

Note: 12.1(13)E1 if any WAN cards

• Native mode: SUP2/PFC2 supports NetFlow version 5 from 12.1(13)E

• Hybrid mode: SUP2/PFC2 supports NetFlow version 5 from 7.5(1)

New


set mls bridged-flow-statistics enable/disable <vlan>

• The L2 switched traffic (from vlan x to vlan x) is now counted with NetFlow

• Hybrid mode: introduced in CatOS version 7.(2)

• Native mode: not yet available

• Doesn’t require a MSFC

Catalyst 6500 Switched Traffic New



Catalyst 6500 NetFlow Sampling

• 12.1(13)E support both time and packet-based sampling

• Sampling rate is configurable only for the whole box

• Accuracy of NetFlow on the platform comes to tuning the aging timers correctly

• Note: A way of minimizing packet loss, is suggesting use of DFC cards, spreading the incoming packet load evenly onto different vlans(on diff cards)

DFC: Distributed Forwarding Card

New


Cisco Catalyst 4000 NetFlow Services Card

• Version 5 in 12.1(13)EW

• Supervisor IV is required

• Feature card is also required

New



Roadmap and Future Directions

111


Targeting 12.3M

• NetFlow v9 • BGP Next hop• NetFlow Multicast• Statistical

SamplingTargeting 12.3(1)T

• Statistical Sampling

Scalability andFlexibility

TechnologyCoverage

Jul2004

May2004

Jun2004

Apr2004

Mar2004

Feb2004

Jan2004

Dec2003

Nov2003

Oct2003

Sep2003

Aug2003

Jul2003

Jun2003

May2003

Apr2003

Mar2003

Feb2003

Aug2004

Standardization

Targeting 12.3(2nd)T

• NetFlow MPLS• BGP Nexthop• NetFlow Multicast

Roadmap for NetFlow Software Platforms

Targeting 12.0(24)S

• NetFlow v9 Targeting 12.2S

• NetFlow v9 • BGP Nexthop• NetFlow Multicast• Statistical Sampling • NetFlow IPv6

Targeting 12.0(26)S• Statistical Sampling • BGP Nexthop• NetFlow MPLS

Aware

Targeting 12.0(27)S• NetFlow Input Filter• NetFlow MPLS Top Label

Targeting 12.2S

• NetFlow Input Filter

Radar

• NetFlow MIB

• Congestion Aware Export (SCTP)

• Egress

• Flexible Input and Export

• NetFlow IPSec

NB. Confirm Target Releases with Cisco IOS® NetFlow PM—Tom Zingale

Optimizing Data forFlow Processing



Roadmap for NetFlow Software 12000



Jul2004

May2004

Jun2004

Apr2004

Mar2004

Feb2004

Jan2004

Dec2003

Nov2003

Oct2003

Sep2003

Aug2003

Jul2003

Jun2003

May2003

Apr2003

Mar2003

Feb2003

Aug2004

Standardization

Targeting 12.0(24)S

• NetFlow v9• MPLS Aware• Output E3• AS Origin

and Peer• MPLS Egress

E3

TechnologyCoverage

Targeting 12.0(26)S

• V8 TOS Agg

• BGP Nexthop

Targeting 12.0(27)S• Sampled on ATM Line Card

• NetFlow MPLS Top Label

Targeting 12.0(28)S

• Statistical Sampling

• Input Filters

• Packet Header

Radar

• IPV6

• Congestion Aware Export

• Flexible Keys

• User Defined Export

• Multicast

NB. Confirm Target Releases with Cisco IOS NetFlow PM—Tom Zingale


Roadmap for NetFlow Catalyst 6500/7600



Jul2004

May2004

Jun2004

Apr2004

Mar2004

Feb2004

Jan2004

Dec2003

Nov2003

Oct2003

Sep2003

Aug2003

Jul2003

Jun2003

May2003

Apr2003

Mar2003

Feb2003

Aug2004

StandardizationTechnologyCoverage


Targeting 12.1(13)E• Version 5• Sampling• Source Dest I/F

Fields• Source Dest AS

Fields• V8 TOS Agg. PFC2Cat 6.6(6) and 7.3(1)• Source Dest I/F

Fields

Targeting 12.2(14)SX

• Sup 720 V8 Agg

Targeting

• Native V8 Aggregation

Targeting 12.2S(RIs3)

• Sup 720 Version 9

• Sup 720 IPV6

Radar

• Sup 3b NetFlow Multicast



Roadmap for NetFlow Catalyst 4000



Jul2004

May2004

Jun2004

Apr2004

Mar2004

Feb2004

Jan2004

Dec2003

Nov2003

Oct2003

Sep2003

Aug2003

Jul2003

Jun2003

May2003

Apr2003

Mar2003

Feb2003

Aug2004

StandardizationTechnologyCoverage

12.1(13)EW• Version 5 Sup 4

Targeting

• Source Dest I/F Fields

• Source Dest AS Fields

• Version 8

• BGP Next Hop



Conclusion/Summary

• NetFlow became the de facto IP accounting method

• The new NetFlow version 9 is extensible and flexible

• NetFlow version 9 has been adopted by the IETF

• A lot of new features recently added

• A lot of new features to come



Questions?


Other Network Management Sessions• Network Management

NSC-1001 Introduction to Network ManagementNSC-2001 Network Troubleshooting Tools and Techniques

• FaultNSC-1011 Principles of Fault Management

• ConfigurationNSC-2021 Configuration of Large-Scale Networks with CiscoWorks NSC-4021 Advanced Configuration Methods

• AccountingNSC-1031 Introduction to Collecting Traffic Accounting InformationNSC-4031 Advanced NetFlow Accounting

• PerformanceNSC-1041 Introduction to Performance ManagementNSC-2041 Performance Measurement with Cisco IOS SoftwareNSC-4041 Advanced Performance Management with Cisco Service Assurance Agent

• SecurityNSC-2051 Securely Managing Your Network

• ServicesNSC-1101 Understanding DNS and DHCPNSC-2102 Deploying and Troubleshooting NAT

• High AvailabilityNSC-1201 Improving Network AvailabilityNSC-2201 Deploying Highly Available Enterprise Networks



Advanced NetFlow AccountingSession NMS-4031


Please Complete Your Evaluation Form

Session NMS-4031

Advanced NetFlow Accounting - Net130.Com · Advanced NetFlow Accounting Session NMS-4031. ... •...

Documents

Transcript of Advanced NetFlow Accounting - Net130.Com · Advanced NetFlow Accounting Session NMS-4031. ... •...