Geneva, 24 March 2011
Cisco experiences of IP traffic flow measurement and billing with NetFlow
Benoit Claise,Distinguished Engineer, Cisco
ITU-T Workshop onIP Traffic Flow Measurement
(Geneva, Switzerland, 24 March 2011)
What is NetFlow?
Cache
CollectorNetFlow Records export
Over UDP or SCTP
Traffic
What is NetFlow?
NetFlow is used for traffic monitoring, security analysis, capacity planning and billing
Billing is just a few % of our customers, mainly for charge back within enterprise network (not between service providers)
NetFlow = a exporting protocol: NetFlow v5, 7, 8, 9 (RFC3954), and IPFIX (RFC5101/RFC5102)
NetFlow v9 and IPFIX work with a template based mechanismAdvantage: extensibility, just need to add new Information Element
NetFlow = a metering process: Flexible NetFlowAdvantages: cache and export content flexibility
User selection of flow keysUser definition of the records
Flexible NetFlow: Potential Key FieldsIPv4IP (Source or Destination)
Payload Size
Prefix (Source or Destination)
Packet Section (Header)
Mask (Source or Destination)
Packet Section (Payload)
Minimum-Mask (Source or Destination)
TTL
ProtocolOptions bitmap
Fragmentation Flags
Version
Fragmentation Offset
Precedence
Identification DSCP
Header Length TOS
Total Length
Interface Input
Output
Flow Sampler ID
Direction
Source MAC address
Destination MAC address
Dot1q VLAN
Source VLAN
Layer 2
IPv6
IP (Source or Destination)
Payload Size
Prefix (Source or Destination)
Packet Section (Header)
Mask (Source or Destination)
Packet Section (Payload)
Minimum-Mask (Source or Destination)
DSCP
Protocol Extension Headers
Traffic Class Hop-Limit
Flow Label Length
Option Header Next-header
Header Length Version
Payload Length
Dest VLAN
Dot1q priority
MulticastReplication Factor*
RPF Check Drop*
Is-Multicast
Flexible NetFlow: Potential Key Fields
Input VRF Name
BGP Next Hop
IGP Next Hop
src or dest AS
Peer AS
Traffic Index
Forwarding Status
Routing TransportDestination Port TCP Flag: ACK
Source Port TCP Flag: CWR
ICMP Code TCP Flag: ECE
ICMP Type TCP Flag: FIN
IGMP Type* TCP Flag: PSH
TCP ACK Number TCP Flag: RST
TCP Header Length TCP Flag: SYN
TCP Sequence Number TCP Flag: URG
TCP Window-Size UDP Message Length
TCP Source Port UDP Source Port
TCP Destination Port UDP Destination Port
TCP Urgent Pointer
Application
Application ID*
*: IPv4 Flow only
Flexible NetFlow: Potential Non-Key Fields
Plus any of the potential “key” fields: will be the value from the first packet in the flow
Counters
Bytes
Bytes Long
Bytes Square Sum
Bytes Square Sum Long
Packets
Packets Long
Timestamp
sysUpTime First Packet
sysUpTime First Packet
IPv4
Total Length Minimum (*)
Total Length Maximum (*)
TTL Minimum
TTL Maximum
(*) IPV4_TOTAL_LEN_MIN, IPV4_TOTAL_LEN_MAX (**)IP_LENGTH_TOTAL_MIN, IP_LENGTH_TOTAL_MAX
IPv4 and IPv6
Total Length Minimum (**)
Total Length Maximum (**)
Performance
Limited Resources in RouterDon’t enable all flow keysThe routers still have to route packets
NetFlow for Billing: Experience
Packet Size Standard Deviation σ f
Mean Packet Size µf
#P
ackets
Nf
Estimation Accuracy (PLT_NZIX1, S24D00, Cisco, f=5%
Issue: Can we use Sampled NetFlow for billing?
Huge amount of data, must sometimes deal with sampled NetFlow, i.e. 1 out of N packets, depending on the platformPacket Sampling for Flow Accounting: Challenges and Limitations, Tanja Zseby, Thomas Hirsch, Benoit Claise, PAM 2008
Issue: Can we use Sampled NetFlow for billing?
Square sum of bytes available in Flexible NetFlowNot used in practice, not even by the collectors!Customers afraid of legal issues with sampling along with a billing service
AS=196 E-BGP
ISP 1$5.00 per 100 MB
traffic index = 1traffic index = 1PrefixPrefix Traffic-indexTraffic-index
Forwarding Information Base
prefix twoprefix two traffic index = 2 traffic index = 2 prefix oneprefix one traffic index = 1 traffic index = 1
Destination Sensitive Billing Proposal(many years ago)
AS=193
Customer
E-BGP
AS 192
ISP 2$7.00 per 100 MB
1. BGP routing updates
2. Go through a table-map statement
3. table-map calls a route-map
4. route-map’s criteria: if criteria 1 -> traffic-index = 1
if criteria 2 -> traffic-index = 2
prefix oneprefix one traffic index = 1 traffic index = 1 Accounting
I-BGP
BGP Policy Accounting Principles
Allows to classify packets based onIP access lists, BGP community list
to characterize the exit points, where each exit point would set an specific community
BGP AS paths
The ISP
The Customer
Issue: What about the Returning Packets?
ISP 1$5.00 per 100 MB
ISP 2$7.00 per 100 MB
FTP Request
100 MB back100 MB back
Who should pay for the 100 MB back?
Destination Sensitive Billing requires also source lookup (Source Sensitive Billing)
Who should pay for the 100 MB back?
Destination Sensitive Billing requires also source lookup (Source Sensitive Billing)
The ISP
The Customer
Issue: What about the Returning Packets?
ISP 1$5.00 per 100 MB
ISP 2$7.00 per 100 MB
FTP Request
100 MB back100 MB back
Lookup:• On the outgoing packets
(on the packets coming back)
• On the source • Same selection criteria
Lookup:• On the outgoing packets
(on the packets coming back)
• On the source • Same selection criteria
The ISP
The Customerin Europe
Issue: BGP Asymmetry Problem
ISP 1 in Asia ISP 2 in US
FTP Request
100 MB back
Will charge the 10 Meg as if they were directly coming from the US!!!
Issue: BGP Asymmetry Problem
The source lookup is based on the route the router would take to reach the source!
Too Many Issues
Destination Sensitive Billing requires Source Sensitive BillingBGP asymmetry problemOnly the traffic following the BGP routes will be accounted
What if local policies outside of BGP?
Limited amount of buckets in the Destination Sensitive Billing
Doesn’t scale: too many entries
Performance issuesEntire NMS solution to be put in place
Destination Sensitive Billing
Conclusion/feedback from customers: too many issuesnot realistically deployable -> back to some sort of flat rate
Benoit’s concern:If we bill per AS-PATH and each AS get a piece of the pie, people will create new AS and try to attract trafficBad for the internet performance
Geneva, 24 March 2011
Cisco experiences of IP traffic flow measurement and billing with NetFlow
Benoit Claise,Distinguished Engineer, Cisco
ITU-T Workshop onIP Traffic Flow Measurement
(Geneva, Switzerland, 24 March 2011)
Top Related