Trex Realistic Traffic Generator - download.safeplus.pl Live Berlin 2017/DEVNET... · • Supports...
Transcript of Trex Realistic Traffic Generator - download.safeplus.pl Live Berlin 2017/DEVNET... · • Supports...
TrexRealistic Traffic Generator
Hanoch Haim – Principal Engineer
DEVNET-1120
• Introduction to Stateful /Stateless
• Current situation and problem
• Model
• High level features
• Examples
• Demo
Agenda
Stateful
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Networks include complex L4-7 features, such as• Load Balancer, DPI/AVC, Firewall, NAT
• Requires testing with stateful and realistic traffic mix
Firewall, NATDPI/AVCLB
DEVNET-1120 5
Current Situation
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Traffic generators for realistic traffic are
• Expensive ~$100-500K
• Not scalable for high rates
• Not flexible
• Implication • Limited and late testing
• Different benchmarks and test methodologies
• Real life bottlenecks and design issues
DEVNET-1120 6
What Problem is Being Solved?
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Stateful traffic generator – smart repaly
• Generates, manipulates and amplifies based on templates of real, captured flows – no TCP/IP stack
• High performance: up to 200 Gb/sec
• Low cost: C220M UCS-1RU, Cisco internal • Standard server hardware
• Flexible and Open Software - DPDK, ZMQ, Python libs
• Virtualization
• Easy installation and deployment
DEVNET-1120 7
What is TRex?
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Stateful Traffic Generation Model
DEVNET-1120 8
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
High level software architecture
DEVNET-1120 9
CP
U/S
ha
re m
em
ory
Sh
are
m
em
ory
Inte
l N
IC
DP -thread 1
CP – thread 0
DP -thread 2
Rx – thread 5
IF0Client
IF1Server
IF2Client
IF3Server
DP -thread 3
DP -thread 4
• DPDK/Multi-Threaded• Scales linearly
• ~20 Gb/sec per core
• Supports 1/10/40 Gb Intel NICs
• Flow-based• Fast event scheduler
• Generates flow templates
• Can support 1K templates
• Scales up to 100K clients,1M servers
• Flexible • Client/server generation models• Measures jitter/latency/flow order
• NAT translation/IPv6/Tunnels
• User Interface• Python API
• Benchmark automation
• GUI
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Flow Generation
DEVNET-1120 10
Total PPS =
𝑘=0
𝑛
𝐶𝑃𝑆𝑘 × 𝑓𝑙𝑜𝑤_𝑝𝑘𝑡𝑠𝑘
Total CPS =
𝑘=0
𝑛
𝐶𝑃𝑆𝑘
Concurrent flows
=
𝑘=0
𝑛
𝐶𝑃𝑆𝑘 × 𝑓𝑙𝑜𝑤_𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛𝑘
• Example of one flow with four
packets
FIF
FIF
FIF
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DNS simple profile example
DEVNET-1120 11
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DNS output
DEVNET-1120 12
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
HTTP & DNS
DEVNET-1120 13
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
HTTP & DNS
DEVNET-1120 14
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Enterprise traffic profile
DEVNET-1120 15
• Includes protocols with
Control/Data dependency • SIP
• RTSP
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
EMIX - YAML
DEVNET-1120 16
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
EMIX
DEVNET-1120 17
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
NAT/FW learning translation/randomization
DEVNET-1120 18
1-Tx-SYN
1-Rx-SYN (learn translation per flow)
2-Tx-SYN-ACK
2-Rx-SYN-ACK (learn)
3-Tx-ACK
3-Rx-ACK (learn seq number randomization)
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Client Clustering
DEVNET-1120 19
• To simulate a big network
• Scale of the number of
clients
• For Controller testing
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Client Clustering #2
DEVNET-1120 20
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Sampling
• Meta data is added to the flow for order check
DEVNET-1120 21
Flow Order Test
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Performance
DEVNET-1120 22
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Performance average packet size 600B
DEVNET-1120 23
Stateless
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Stateless High level functionality
• High scale –~10M-22MPPS/core
• Support 1/10/25/40/100 Gb/sec interfaces
• Support for multiple traffic profiles per interface
• Profile can support multiple streams, scalable to 10K parallel streams
• Supported for each stream
• Packet template
• Field engine program (src_ip = 10.0.0.1-10.0.0.255)
• Send Mode : Continues/Burst/Multi burst support
DEVNET-1120 25
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Stateless High level functionality #2
• Interactive support – GUI/TUI
• Statistic per port
• Statistic per stream (by Hardware)
• Latency Jitter per stream
• Fast Python automation support
• Python 2.7/3.0 Client API
• Python HLTAPI Client API
• Multi-user support
DEVNET-1120 26
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Traffic Profile Example
DEVNET-1120 27
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Control plane High level
DEVNET-1120 28
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi User
DEVNET-1120 29
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
TRex Objects relations
DEVNET-1120 30
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
One stream with two directions
DEVNET-1120 31
Stateless – traffic profile
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Simple Interleaving streams
DEVNET-1120 33
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Simple Interleaving streams -profile
DEVNET-1120
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi burst
DEVNET-1120 35
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi burst profile
DEVNET-1120 36
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Field Engine
DEVNET-1120 37
• Flexible engine to change any field inside the packet
• Examples
– Change TOS 1-20
– Range of client IP 10.0.0.1-10.0.0.254
– Random packet size 64-9k
– Random dest_ip range
– Support any tunnel even not valid packet like
QinQ/GRE/MPLS/Ipv6/UDP/Ipv4/HTTP
• Plan to add even more flexible engine - JITLUA
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Instructions
DEVNET-1120 38
https://trex-tgn.cisco.com/trex/doc/cp_stl_docs/api/field_engine.html
Instruction type Description API
STLVmFlowVar Define a variableSTLVmFlowVar(name, init_value=None, min_value=0
, max_value=255, size=4, step=1, op='inc')
STLVmFlowVarRepetableRandom Repeatable random varSTLVmFlowVarRepetableRandom(name, size=4, limit
=100, seed=None, min_value=0, max_value=None)
STLVmTupleGen Tuple generator structSTLVmTupleGen(name, ip_min='0.0.0.1', ip_max='0
.0.0.10', port_min=1025, port_max=65535, limit_
flows=100000, flags=0)
STLVmTrimPktSize Trim the packet size by var name STLVmTrimPktSize(fv_name)
STLVmFixIpv4 Fix IPv4 header checkssum STLVmFixIpv4(offset)
STLVmFixChecksumHw Fix TCP/UDP checksum STLVmFixChecksumHw(l3_offset, l4_offset, l4_typ
e)
STLVmWrMaskFlowVar Write a variable to bitsSTLVmWrMaskFlowVar(fv_name, pkt_offset, pkt_
cast_size=1, mask=255, shift=0, add_value=
0, offset_fixup=0, is_big=True)
STLVmWrFlowVar Write a variableSTLVmWrFlowVar(fv_name, pkt_offset, offset_f
ixup=0, add_val=0, is_big=True)
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Field Engine, Syn attack
DEVNET-1120 39
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Convert pcap Packet File to One Stream
DEVNET-1120 40
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco PublicDEVNET-1120 41
Pcap File Conversion to Streams
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
• In this mode pcap in converted to streams and push to TRex server
• It won’t work on a big pcap file
• There is an API version that push server side pcap file
• This version is limited only by server disk size. 1TB pcap file is something that we are using
DEVNET-1120 42
Pcap File Conversion to Profile of Streams #2
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Implemented using hardware assist with Intel
X710/XL710 NIC flow director rules
• With other NICs (Intel I350, 82599), implemented in
software.
DEVNET-1120 43
Per Stream Statistics
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco PublicDEVNET-1120 44
Per Stream Statistics - TUI
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco PublicDEVNET-1120 45
Per Stream Statistics – Python API
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Base on per stream stats hardware assist
• Forward specific type of packets
• Filter is based on IPV4.ID and IPv6.flow_id
• Software measures latency and jitter resolution is
~usec (not nsec)
DEVNET-1120 46
Per Stream Latency/Jitter
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Per Stream Statistics - TUI
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco PublicDEVNET-1120 48
Service Mode
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Ping / ARP
• IPv6 ND /Multicast setup
• Scan6 support
• Traffic Capturing
• Functional test
• New protocols support for setup
DEVNET-1120 49
Service Mode
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Capture Monitoring – Wireshark Pipe
DEVNET-1120 50
Stateless – Performance
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
link
Performance Setup
DEVNET-1120 52
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
link
Performance XL710 MPPS/Core
DEVNET-1120 53
Stateless – Automation
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Simple example
DEVNET-1120 55
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Complete Your Online Session Evaluation
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online
• Please complete your Online Session Evaluations after each session
• Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt
• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations
DEVNET-1120 56
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue your Education
DEVNET-1120 57
• Stateless manual
• TRex documents Index
• GitHub
• DevNet zone
Q & A
Thank You
Backup
SR-IOV support
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
SR-IOV vs vSwitch
63DEVNET-1120
VMM
vNIC vNIC
vSwitch
NIC PF
VMM
vNIC vNIC
HW SwitchNIC PF
VFVF
SR-IOVvSwitch
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
SR-IOV XL710/X710
64DEVNET-1120
VMM
vNIC vNIC
HW SwitchNIC PF
VFVF
Pool of TRex
TRex TRex
vNIC vNIC
HW Switch
NIC PF
VFVF
Pool of CSR
CSR CSR
Catalist 36xx
25/50/100GbE support
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
• 100/50/25/40/1 GbE speeds
• PCIe16 v3
• Small foot print
• Low cost - 800$
• SR-IOV support
• 10-60K TCAM rules – not perfect rule for counters
DEVNET-1120 66
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
ConnectX-4 DPDK performance
DEVNET-1120 67
• UDP 64B
• TX ~95 MPPS
• Rx 50 MPPS
• IMIX
• 90% line-rate
• Cycles/Pkt high +50% relative to Intel
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
FPGA Base – 400Gbe
68DEVNET-1120
• FPGA base
• Accurate
IPG/latency
• Scale to 400GbE
without DP SW
• ~30K$ –
4x100GbE
• Better Statistic in
Rx side
• Less CPU
resource
• Manufactured by
Cisco VIC support
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco VIC 13xx support
70DEVNET-1120
• Contribution from DPDK VIC team
• Supported on both UCS C-series and B-series (blade
server)
• PCIe16 - 2x-40GbE ports
• Can be shared by VM using Cisco vNIC (using CIMC) - it
is not SR-IOV
• Only the 13xx series, Cisco adapter is supported
Demo Statful
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Demo routing
DEVNET-1120 72
Cisco ASR 1013 ESP100
100Gb/sec 13RU - 4KW
UCS-220M2 32GB 2x8 cores 2Ghz
2x82559 NIC (4x10Gb/sec)
0.4KW 1RU , 2K$
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
-Per port stats table
ports | 0 | 1 | 2 | 3
-----------------------------------------------------------------------------------------
opackets | 71602327 | 82914082 | 71601627 | 82913259
obytes | 23990692045 | 67783397687 | 23990683213 | 67783032819
ipackets | 82914006 | 71602322 | 82913203 | 71601615
ibytes | 67783068258 | 23986183633 | 67782693539 | 23986134949
Tx Bw | 3.68 Gbps | 8.84 Gbps | 3.68 Gbps | 8.84 Gbps
Cpu Utilization : 41.6 % 15.0 Gb/core
Platform_factor : 1.0
Total-Tx : 25.04 Gbps Nat_time_out : 0
Total-Rx : 25.03 Gbps Nat_no_fid : 0
Total-PPS : 5.21 Mpps Total_nat_active: 33
Total-CPS : 103.16 Kcps Total_nat_open : 6697955
Active-flows : 90502 Clients : 248 Socket-util : 0.5795 %
Open-flows : 6713475 Servers : 5368 Socket : 90502 Socket/Clients : 364.9
-Latency stats enabled
Cpu Utilization : 7.2 %
if| tx_ok , rx_ok , rx ,error, average , max , Jitter , max window
| , , check, , latency(usec),latency (usec) ,(usec) ,
--------------------------------------------------------------------------------------------------------------
--
0 | 65649, 65648, 647320, 0, 61 , 144, 13 | 111 130 138 107 127 110
1 | 65648, 65649, 552181, 0, 37 , 62, 3 | 42 42 43 39 38 39 38
2 | 65649, 65648, 660627, 0, 57 , 144, 11 | 103 139 130 97 126
3 | 65648, 65649, 562860, 0, 37 , 55, 3 | 39 41 39 39 46 39 39
-Rx Check stats enabled
------------------------------------------------------------------------------------------------------------
rx check: avg/max/jitter latency, 63 , 176, 18 | 130 158 159 138 155 175 136
active flows: 647, fif: 52193, drop: 0, errors: 0
------------------------------------------------------------------------------------------------------------
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco PublicDEVNET-1120 74
Configure NAT & Firewall
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
-Global stats enabled
Cpu Utilization : 34.9 % 15.9 Gb/core
Platform_factor : 1.0
Total-Tx : 22.24 Gbps Nat_time_out : 3910462
Total-Rx : 22.24 Gbps Nat_no_fid : 735
Total-PPS : 4.87 Mpps Total_nat_active: 46
Total-CPS : 102.79 Kcps Total_nat_open : 11287417
Active-flows : 79813 Clients : 248 Socket-util : 0.5110 %
Open-flows : 11313545 Servers : 5368 Socket : 79813 Socket/Clients : 321.8
-Latency stats enabled
Cpu Utilization : 6.7 %
if| tx_ok , rx_ok , rx ,error, average , max , Jitter , max window
| , , check, , latency(usec),latency (usec) ,(usec) ,
--------------------------------------------------------------------------------------------------------------
--
0 | 110281, 110279, 714052, 0, 204 , 14313, 16 | 807 1276 3439 935 850 5175
1 | 110280, 110280, 644514, 0, 91 , 10907, 5 | 802 653 1344 717 701 3436
2 | 110281, 110280, 720630, 0, 201 , 14355, 14 | 808 1194 3357 972 822 5177
3 | 110280, 110281, 645768, 0, 90 , 10491, 3 | 800 642 1349 706 699 3279
-Rx Check stats enabled
------------------------------------------------------------------------------------------------------------
rx check: avg/max/jitter latency, 180 , 14785, 407 | 2321 2546 4229 1627 1912 5275
active flows: 690, fif: 57593, drop: 14179, errors: 14552
------------------------------------------------------------------------------------------------------------
DEVNET-1120 76
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
TRex on your laptop (virtualbox)
DEVNET-1120 78
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
TRex DP queues/cores #1 DP
DEVNET-1120 79
Flow
Director
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
TRex DP queues/cores #2 DP
DEVNET-1120
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
TRex CP-DP messaging
DEVNET-1120 81
• One to many
• No locks
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
TRex RX-DP messaging
DEVNET-1120 82
• One to many
• No locks
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Stateful vs Stateless
DEVNET-1120 83
Feature Stateless Stateful
Flow base No Yes
NAT No Yes
Tunnel Yes Some are supported
L7 App emulation No Yes
Any type of packet Yes No
Latency/Jitter Per Stream Per port/Per flow sample
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Interactive Console
DEVNET-1120 84
#load the trex as a server for interactive mode
$sudo ./t-rex-64 –i
#connect to the server from any server ( Python 2/3.4)
$./trex-console
#start traffic on all port
>start -a -m 1 -f stl/imix_1pkt.py
#pause traffic on all port
>pause -a
#resume traffic on all port
>resume -a
#stop traffic on all port
>stop -a
#show dynamic statistic
>tui
#show port statistic
>stats –p
#clear statistic
>clear
#show stream statistic
>streams
Shell
Console
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Interactive TUI
DEVNET-1120 85
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multiple Clients Example
DEVNET-1120 86
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multiple Clients Profile
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Teredo Tunnel (Ipv6 Over IPv4
DEVNET-1120 88
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
link
• XL710 support line rate
from 128B
Performance XL710 Line%
DEVNET-1120 89
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
link
Performance XL710 gbps/core
DEVNET-1120 90
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Demo
DEVNET-1120 92
Cisco ASR 1013 ESP100
100Gb/sec 13RU - 4KW
UCS-220M2 32GB 2x8 cores 2Ghz
2x82559 NIC (4x10Gb/sec)
0.4KW 1RU , 2K$finalized the GUI