High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j...
Transcript of High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j...
C U R A T E D B Y
IP SHOWCASE THEATER AT NAB – APRIL 8-11, 2019
High-Precision Capturing and Measuring of 2110 Streams
Using Commodity IT Equipment Willem Vermost, Ievgen Kostiukevych
IP Media Technology Architects
European Broadcasting Union
Essent ial measurements and compliance
evaluat ion in SMPTE ST 2110 based faci l i t ies using
commodity of f- the-shelf IT hardware
• Willem Vermost, EBU
• Pedro Ferreira, Bisect
• Ievgen Kostiukevych, EBU
EBU – Live IP Software Toolkit offline tool - open source project
Waveform Monitor Packet Analyser
Broadcast Engineer IT Engineer
SKILLSET – BRIDGING THE GAP
PCAP STORAGE
INFLUX DB
MONGO DB
STREAM ANALYZER
STREAM PREPROCESSOR
MIDDLEWARE GUI
HTTP SERVER
PCAP files are stored
time-series data: information about every
packet on the stream
The analyses of the data are totally based on the LIST
Core libraries, which are developed in C++
The middleware coordinates the workflows
(based on node.js)
ARCHITECTURE – OFFLINE non-time related data: user data static stream
information, …
The workflow can be accelerated if the PCAP
capture process is included in LIST
PCAP FILE
The UI is developed in JavaScript using the React
framework
Critical success factor for the analysis:
• Accuracy of IP Packet Timestamp
• Clock of capture device synchronized with PTP
PREREQUISITE FOR VALID MEASUREMENT / CAPTURE
SMPTE ST 2110-21 MEASURING THE BEHAVIOR OF A SENDER
In order to measure the behavior of a sender,
Join the multicast stream as close to the sender
Port mirror
Use a network TAP, if not available use a switch port mirror
Record the packet stream with a high precision capturing device
Make sure PTP is available
NIC
OS Kernel
Application
NIC
OS Kernel
Application
IP switch
• Accurate timestamps
• Everything locked to PTP
LAB EXPERIMENT
LAB EXPERIMENT: OUR TEST SETUP
NIC PIPELINE: STREAM JOIN, PTP INBAND
Application
ens5f0
ens5f1
/dev/ptp6
/dev/ptp7
CPU OS Kernel
PCIe lanes
Driver
SYS clock
tcpdump Stream, PTP
-j host • - j host
NIC PIPELINE: STREAM JOIN, PTP INBAND
Application
ens5f0
ens5f1
/dev/ptp6
/dev/ptp7
CPU OS Kernel
PCIe lanes
Driver
SYS clock
tcpdump Stream, PTP
-j host • CINST analysis
• - j host
NIC PIPELINE: PORT MIRRORING, PTP OUTBAND
Application
ens5f0
ens5f1
/dev/ptp6
/dev/ptp7
CPU OS Kernel
PCIe lanes
Driver
SYS clock
tcpdump Sender port mirror
PTP
-j adapter_unsynced • - j adapter_unsynced
NIC PIPELINE: PORT MIRRORING, PTP OUTBAND
Application
ens5f0
ens5f1
/dev/ptp6
/dev/ptp7
CPU OS Kernel
PCIe lanes
Driver
SYS clock
tcpdump Sender port mirror
PTP
-j adapter_unsynced • CINST analysis
• - j adapter_unsynced
PTP: CLOCK CORRECTION OVER TIME (ns)
-250
-200
-150
-100
-50
0
50
100
150
200
250
Co
rrec
tio
n (
ns)
Clocks correction over time (s)
PHC2SYS
PTP4L
• PTP4L drives PHC2SYS
LOOKING AT THE FIRST RESULTS
• Something is wrong!
• Don’t forget to lock your SDI source to
the reference
NIC PIPELINE: PORT MIRRORING, PTP OUTBAND + KERNEL OFFLOAD
Application
ens5f0
ens5f1
/dev/ptp6
/dev/ptp7
CPU OS Kernel
PCIe lanes
Driver
SYS clock
tcpdump Sender port mirror
PTP
VMA
• PTP4L drives PHC2SYS
WE STARTED WITH…
• Our first implementation
• DL 380 G10
WE MINIATURIZED …
• Mini PC
• Thunderbolt-PCIe Enclosure
+
SMPTE ST 2110-21 Traffic Shaping and Delivery Timing for Video
• Describes a virtual test for the network (CMAX) • Constrains the packet delay variation of a sender
• Narrow senders
• Wide senders
• Describes a virtual test for the receive buffer (VRXFULL) • Narrow receivers (not capable of receiving wide senders)
• Wide receivers (universal)
• Describes Packet Read Schedules
SMPTE ST 2110-21 – TRAFFIC SHAPING AND DELIVERY TIMING FOR VIDEO
SMPTE ST 2110-21 LEAKY BUCKET I: THE NETWORK COMPATIBILITY MODEL
• Tested at the output of the sender
• Measures PDV introduced by a sender
• CMAX is the maximum amount of packets allowed in the
“leaky bucket”
• If CINST > CMAX -> packets might get dropped in the
network
ST 2110-20 Sender
CMAX
CINST
Drain Rate = 1/TDRAIN (packets / second)
We just said, don’t drop any packet??
ST 2110-20 Sender
VRXFULL
VRXINST
Packet J drains at Time TPRJ
SMPTE ST 2110-21 LEAKY BUCKET II: THE VIRTUAL RECEIVE BUFFER
• Tested at the output of the sender
• The VRX buffer is drained as a function of the packet
read schedule
• VRXFULL is the maximum amount of packets allowed in
the “leaky bucket”
• If VRXINST > VRXFULL -> packet might get dropped at
receiving end
We just said, don’t drop any packet??
SMPTE ST 2110-21 PACKET READ SCHEDULES
Gapped
Packet Read Schedule
Linear
Packet Read Schedule
t
t
N ∗ TFRAME
Since Epoch
(N + 1) ∗ TFRAME
Since Epoch
TRS(gapped)
TRS(linear)
TVD
TPR0 TPR0
TPR0 TPRj
TROFFSET
TROFFSET
(𝑁𝑃𝐴𝐶𝐾𝐸𝑇𝑆 − 1) ∗ TRS(gapped)
(𝑁𝑃𝐴𝐶𝐾𝐸𝑇𝑆 − 1) ∗ TRS(linear)
SMPTE ST 2110-21 GAPPED PACKET ARRIVAL
Gapped Packet Read Schedule
t
𝐍 ∗ 𝐓𝐅𝐑𝐀𝐌𝐄 Since Epoch
(N + 1) ∗ TFRAME Since Epoch
TRS(gapped)
TVD
𝐓𝐏𝐑𝟎 TPRj
TROFFSET
TROFFSET
𝑁𝑃𝐴𝐶𝐾𝐸𝑇𝑆 ∗ TRS(gapped)
Packet Arrival
t
Tarrival gap
𝐓𝐏𝑨𝟎 TP𝐴j
First Packet Offset
TP𝐴n
Marker Bit Set
RTPTimestamp = x
𝐓𝐏𝑨𝟎
TVD
RTPTimestamp = y
First Packet Offset
• TRS (gap) = 4,4 us (1080p50)
• TPA0 = Arrival time first packet
TEST RESULTS Practical Test Results with EBU - LIST
TIMESTAMP COMPARISON OF TWO DIFFERENT CAPTURE DEVICES
-1400
-1200
-1000
-800
-600
-400
-200
0
283869560 283869760 283869960 283870160 283870360 283870560
Tim
esta
mp
dif
fere
nce
(n
s)
RTP Sequence number
Timestamp comparison of two different devices
Diff.
Linear (Diff.)
• Avoid any bottlenecks in the capturing pipeline: • Use a NIC capable of handling the bandwidth
• Use kernel bypass to avoid packet drops by the OS
• Check storage performance - enough to store the capture in real time
• Frequency and Phase lock • Use a NIC with hardware clocking / timestamp support
• Synchronize the NIC to PTP
• It is possible to perform accurate captures with COTS equipment
CONCLUSION
Want to know more tech.ebu.ch/list
Github github.com/ebu/pi-list
Docker hub.docker.com/r/ebutech/pi-list
Dematerialized list.ebu.io
JT-NM Tested Program jt-nm.org/jt-nm_tested/
Thank you!
Any Questions?