UDP Flow Charts for linux-2.6.36.2 and different CPUs...

UDP Flow Charts for linux-2.6.36.2 and different CPUs

Table of Contents

UDP PROCESS TIMING TEST INFORMATION ........................................................................................................... 3

SYSTEM INFORMATION FOR TESTS ..................................................................................................................................... 3

Linux Information .................................................................................................................................................. 3

Network Interface Information ............................................................................................................................. 3

Test for: Westmere ............................................................................................................................................... 3 netperf Side ........................................................................................................................................................................ 3 netserver Side: GC-NH5 test system .................................................................................................................................. 3

Test for: Jaketown B0 to B0 ................................................................................... Error! Bookmark not defined. netperf Side .......................................................................................................................... Error! Bookmark not defined. netserver Side ...................................................................................................................... Error! Bookmark not defined.

Test for: DELL Westmere ....................................................................................................................................... 4 netperf Side ........................................................................................................................................................................ 4 netserver Side: GC-NH5 test system .................................................................................................................................. 4

Test for: Single Core DELL Westmere .................................................................................................................... 4 netperf Side ........................................................................................................................................................................ 4 netserver Side: GC-NH5 test system .................................................................................................................................. 4

Test for: 2S Nehalem ............................................................................................................................................. 4 netperf Side ........................................................................................................................................................................ 5 netserver Side: GC-NH5 test system .................................................................................................................................. 5

UDP PROCESS TIMING RANGE ACROSS MULTIPLE PROCESSORS ............................................................................ 6

UDP PROCESS TIMING RANGE FOR WESTMERE PROCESSOR ............................................................................... 11

UDP PROCESS TIMING RANGE FOR JAKETOWN B0 TO JAKETOWN B0 ................ ERROR! BOOKMARK NOT DEFINED.

UDP PROCESS TIMING FOR NONE-SMP SINGLE CORE WESTMERE CPU ................................................................ 16

UDP PROCESS TIMING RANGE FOR DELL PRODUCTION WESTMERE .................................................................... 22

UDP PROCESS TIMING RANGE FOR NEHALEM ..................................................................................................... 28

UDP PROCESS TIMING RANGE FOR NEHALEM WITH 1 SOCKETED CPU................................................................. 33

Table of Figures

FIGURE 1: MULTIPLE SYSTEMS SMP SOCKET & UDP TX PROCESSING ABOVE IP STACK ................................................................... 6

FIGURE 2: MULTIPLE SYSTEMS SMP UDP TX PROCESSING BELOW IP STACK ................................................................................ 7

FIGURE 3: MULTIPLE SYSTEMS SMP SOCKET & UDP RX PROCESSING ABOVE IP STACK .................................................................. 8

FIGURE 4: MULTIPLE SYSTEMS SMP IXGBE INTERRUPT DRIVEN UDP RX PROCESSING BELOW IP STACK ........................................... 9

FIGURE 5: MULTIPLE SYSTEMS SMP RX PROTOCOL STACK PROCESSING OF UDP ABOVE IP STACK .................................................. 10

FIGURE 6: MULTIPLE SYSTEMS SMP IXGBE_LOW_LATENCY_RECV() DRIVER RX FLUSH PROCESSING .... ERROR! BOOKMARK NOT DEFINED.

FIGURE 7: WESTMERE SMP SOCKET & UDP TX PROCESSING ABOVE IP STACK............................................................................ 11

FIGURE 8: WESTMERE SMP UDP TX PROCESSING BELOW IP STACK ......................................................................................... 12

FIGURE 9: WESTMERE SMP SOCKET & UDP RX PROCESSING ABOVE IP STACK ........................................................................... 13

FIGURE 10: WESTMERE SMP IXGBE INTERRUPT DRIVEN UDP RX PROCESSING BELOW IP STACK .................................................. 14

FIGURE 11: WESTMERE SMP RX PROTOCOL STACK PROCESSING OF UDP ABOVE IP STACK ........................................................... 15

FIGURE 12: WESTMERE SMP IXGBE_LOW_LATENCY_RECV() DRIVER RX FLUSH PROCESSING ............. ERROR! BOOKMARK NOT DEFINED.


FIGURE 13: JAKETOWN B0 TO B0 SMP SOCKET & UDP TX PROCESSING ABOVE IP STACK ................ ERROR! BOOKMARK NOT DEFINED.

FIGURE 14: JAKETOWN B0 TO B0 SMP UDP TX PROCESSING BELOW IP STACK .............................. ERROR! BOOKMARK NOT DEFINED.

FIGURE 15: JAKETOWN B0 TO B0 SMP SOCKET & UDP RX PROCESSING ABOVE IP STACK ................ ERROR! BOOKMARK NOT DEFINED.

FIGURE 16: JAKETOWN B0 TO B0 SMP IXGBE INTERRUPT DRIVEN UDP RX PROCESSING BELOW IP STACK...... ERROR! BOOKMARK NOT

DEFINED.

FIGURE 17: JAKETOWN B0 TO B0 SMP RX PROTOCOL STACK PROCESSING OF UDP ABOVE IP STACK .. ERROR! BOOKMARK NOT DEFINED.

FIGURE 18: JAKETOWN B0 TO B0 SMP IXGBE_LOW_LATENCY_RECV() DRIVER RX FLUSH PROCESSING ERROR! BOOKMARK NOT DEFINED.

FIGURE 19: SINGLE CORE DELL WESTMERE SOCKET & UDP TX PROCESSING ABOVE IP STACK ........................................................ 16

FIGURE 20: SINGLE CORE DELL WESTMERE UDP TX PROCESSING BELOW IP STACK ..................................................................... 17

FIGURE 21: SINGLE CORE DELL WESTMERE SOCKET & UDP RX PROCESSING ABOVE IP STACK ....................................................... 18

FIGURE 22: SINGLE CORE DELL WESTMERE IXGBE INTERRUPT DRIVEN UDP RX PROCESSING BELOW IP STACK ................................ 19

FIGURE 23: SINGLE CORE DELL WESTMERE RX PROTOCOL STACK PROCESSING OF UDP ABOVE IP STACK ......................................... 20

FIGURE 24: SINGLE CORE DELL WESTMERE IXGBE_LOW_LATENCY_RECV() DRIVER RX FLUSH PROCESSING ....................................... 21

FIGURE 25: DELL WESTMERE SMP SOCKET & UDP TX PROCESSING ABOVE IP STACK .................................................................. 22

FIGURE 26: DELL WESTMERE SMP UDP TX PROCESSING BELOW IP STACK ................................................................................ 23

FIGURE 27: DELL WESTMERE SMP SOCKET & UDP RX PROCESSING ABOVE IP STACK .................................................................. 24

FIGURE 28: DELL WESTMERE SMP IXGBE INTERRUPT DRIVEN UDP RX PROCESSING BELOW IP STACK........................................... 25

FIGURE 29: DELL WESTMERE SMP RX PROTOCOL STACK PROCESSING OF UDP ABOVE IP STACK.................................................... 26

FIGURE 30: DELL WESTMERE SMP IXGBE_LOW_LATENCY_RECV() DRIVER RX FLUSH PROCESSING.................................................. 27

FIGURE 31: 2S NEHALEM SMP SOCKET & UDP TX PROCESSING ABOVE IP STACK ....................................................................... 28

FIGURE 32: 2S NEHALEM SMP UDP TX PROCESSING BELOW IP STACK ..................................................................................... 29

FIGURE 33: 2S NEHALEM SMP SOCKET & UDP RX PROCESSING ABOVE IP STACK ....................................................................... 30

FIGURE 34: 2S NEHALEM SMP IXGBE INTERRUPT DRIVEN UDP RX PROCESSING BELOW IP STACK ............................................... 31

FIGURE 35: 2S NEHALEM SMP RX PROTOCOL STACK PROCESSING OF UDP ABOVE IP STACK ........................................................ 32

FIGURE 36: 2S NEHALEM SMP IXGBE_LOW_LATENCY_RECV() DRIVER RX FLUSH PROCESSING .......... ERROR! BOOKMARK NOT DEFINED.

FIGURE 37: NEHALEM, 1 CPU, SMP SOCKET & UDP TX PROCESSING ABOVE IP STACK................................................................ 33

FIGURE 38: NEHALEM, 1 CPU, SMP UDP TX PROCESSING BELOW IP STACK ............................................................................. 34

FIGURE 39: NEHALEM, 1 CPU, SMP SOCKET & UDP RX PROCESSING ABOVE IP STACK ............................................................... 35

FIGURE 40: NEHALEM, 1 CPU, SMP IXGBE INTERRUPT DRIVEN UDP RX PROCESSING BELOW IP STACK ........................................ 36

FIGURE 41: NEHALEM, 1 CPU, SMP RX PROTOCOL STACK PROCESSING OF UDP ABOVE IP STACK ................................................. 37

FIGURE 42: NEHALEM, 1 CPU, SMP IXGBE_LOW_LATENCY_RECV() DRIVER RX FLUSH PROCESSING ... ERROR! BOOKMARK NOT DEFINED.


UDP Process Timing Test Information The tests used netperf to netserver UDP_RR data transfer to provide dataflow for collecting network

process timing data. Timing test data was collected on the side running netperf.

The timing data was collected by logging the current processor clock cycles of a moderate number of

locations using Linux kernel get_cycles() function then calculating time and averaging over 4096 packets,

then printing results and writing results to the kernel log. After the test the relevant kernel log data is

saved and put into a spread sheet to calculate the average over the whole test. Many tests with

measurements at different kernel locations with a complete kernel build and reboot were used to

collect the full timing data set.

System Information for Tests

Linux Information

The Linux release kernel 2.6.36.2 was used as the base kernel with and flush kernel change patch. An

IXGBE device driver patch to change the IXGBE device driver to the Intel low latency IXGBE test driver

version LL84P (Low Latency ixgbe-2.0.84.9P+LL-tx+tx-Fdir) for the base test kernel. Each test used the

base kernel with a test kernel patch to create the test kernel for each test.

For the test case of low latency polling disabled which is the base interrupt based test, the Low Latency

Receive Flush, Low Latency TX Flow Change and Low Latency TCP Receive Flush are disabled in the

Linux kernel configuration.

Network Interface Information

NIANTIC interface with interrupts affinitized on both sides.

Test for: Westmere

netperf Side

3.3 GHz Westmere 6 core, CPU Family 6, Model 44-X5680, Step 2 (B1 206C2) with 2 CPUs socketed,

12288KB cache, QPI 6.4 GT/s rate, Hyperthreading enabled.

1333 MHz memory bus, 12 GB memory (6 GB using 3 X 2GB memory modules per socket)

CPU Socket 0 core was used for test.

netserver Side: GC-NH5 test system

2.932 GHz Nehalem 4 core, CPU Family 6, Model 26 Nehalem B0/B1, Step 2 (106A2), with only 1 CPU

socketed. 8192 KB cache, QPI 5.866 GT/s rate, Hyperthreading enabled.

1067 MHz memory bus, 6 GB memory (6 GB using 3 X 2GB memory modules for socket 0).


Test for: DELL Westmere

System is a DELL PowerEdge 710 production server.

netperf Side

2.93 GHz Westmere 6 core, CPU Family 6, Model 44-X5670, Step 2 with 2 CPUs socketed, 12288KB

cache, Hyperthreading enabled.

12 GB memory installed.

No CPU Socket core was set for test.





Test for: Single Core DELL Westmere

System is a DELL PowerEdge 710 production server.

System is same as DELL Westmere, but with SMP disabled in kernel build for single core, non-SMP timing

data. This test provides for comparing non-SMP network data timing with SMP data timing.

netperf Side

2.93 GHz Westmere 6 core, CPU Family 6, Model 44-X5670, Step 2 with 2 CPUs socketed, 12288KB

cache, single core non-SMP kernel build.

12 GB memory installed.

No CPU Socket core was set for test.





Test for: 2S Nehalem

2S is for two CPU socketed in system. Data, data for same system with one CPU pulled out was collected

(but not included in the diagrams as of this time).


netperf Side

3.07 GHz Nehalem 4 core, CPU Family 6, Model 26, Step 2 with 2 CPUs socketed, 8192 KB cache,

Hyperthreading enabled.

Netperf was not affinitized to any CPU Socket.






UDP Process Timing Range across Multiple Processors

Figure 1: Multiple Systems SMP Socket & UDP TX Processing above IP Stack

Application{}

sock_alloc_send_skb()

ip_local_out()

(Delivers to Protocol Stack)

udp_push_pending_frames ()

126-238

241-339

56-179

66-123

255-444 (size 1)

262-440 (size 1400)

25-52

373-519

44-193

sendto()

recvfrom()

ip_append_data()

ip_push_pending_frames ()

161-263 (size 1)

168-267 (size 1400)

123-269 (size 1)

382-830 (size 1400)

2648-4071 (size 1)

3028-4762 (size 1400)

24-62

58-84

1582-2271 (size 1)

1694-2517 (size 1400)

sock_sendmsg()

sock->ops->sendmsg()

udp_sendmsg()

__sock_sendmsg()

45-329


Figure 2: Multiple Systems SMP UDP TX Processing Below IP Stack

__dev_xmit_skb()

ixgbe_tx_queue()

ip_local_out()


227-626

656-950

51-100

978-1312

54-103

33

dev_hard_start_xmit()

+dev()

dev_queue_xmit()

25-35

46-64

1582-2271 (size 1)

1694-2517 (size 1400)

sch_direct_xmit()

op->ndo_start_xmit()

ixgbe_xmit_frame()

33-72

33

152-247

33

24-31

49-430

24-346

34-46

45-329


Figure 3: Multiple Systems SMP Socket & UDP RX Processing above IP Stack

Application{}

sock->ops->recvmsg()

udp_recvmsg()

wait_for_packet()

44-138

24-29

24-89

255-444 (size 1)

262-440 (size 1400)

94-186

326-554

sendto()

recvfrom()

__skb_recv_datagram()

161-263 (size 1)

168-267 (size 1400)

145-239 (size 1)

403-788 (size 1400)

sock_recvmsg()

__sock_recvmsg()

__sock_recvmsg_nosec()

skb_free_datagram_locked()

De-Q Loop


Figure 4: Multiple Systems SMP IXGBE Interrupt Driven UDP RX Processing Below IP Stack

ixgbe_msix_clean_many()

ixgbe_clean_tx_irq()

net_rx_action()

(napi[] by n->poll())

napi_gro_receive()

__netif_receive_skb()


Interrupt Based IXGBE 84 Network Packet Processing

159-251

76-111

98-142

174-210

3063-4204

(Async Wake)

1658-1911

(Async-Wake)

1312-1484 to

Dequeue

Note: RPS not shown

396-890

HW ISR SW ISR __do_softirq()

ixgbe_clean_rxtx_many()

ixgbe_clean_rx_irq()

ixgbe_receive_skb()

__napi_gro_receive()

dev_gro_receive()

napi_skb_finish()

netif_receive_skb()

441-849

42-78

132-158

24-27

32-46

27-51

24-27

45-52

41-55

575-687

68-101

24-38

RX Packet DeQed

122-161

337-739

1928-2285

(Async Wake) 24-36

ixgbe_alloc_rx_buff

ers ()


Figure 5: Multiple Systems SMP RX Protocol Stack Processing of UDP above IP Stack


pt_prev->func()

Protocol Stack

Processing

__udp4_lib_rcv()

Queue SKB

In Socket Queue

393-576

24-39

60-134

134-221

56-78 (No-Wake)

714-895 (Async Wake)

24-56

101-130

24-27

udp_queue_rcv_skb()

ip_queue_rcv_skb()

sock_queue_rcv_skb()

sk->sk_data_ready()

(sock_def_readable())

__udp_queue_rcv_skb() skb_add_backlog()

or

wake_up_interruptible()

(Wake for Poll wait)

sk_wake_async()

(Wake for Asynchronous wait)

24-32

24-54

752-884 (No-Wake)


(for netif_receive_skb())


UDP Process Timing Range for Westmere Processor

Figure 6: Westmere SMP Socket & UDP TX Processing above IP Stack

Application{}


ip_local_out()



151-174

256-336

57-179

100-108

291-347 (size 1)

289-343 (size 1400)

26-47

394-413

44-193

sendto()

recvfrom()

ip_append_data()


172-251 (size 1)

178-251 (size 1400)

160-241 (size 1)

752-830 (size 1400)

3393-3723 (size 1)

4163-4461 (size 1400)

29-44

62-78

1996-2271 (size 1)

2370-2517 (size 1400)

sock_sendmsg()


udp_sendmsg()

__sock_sendmsg()

70-88


Figure 7: Westmere SMP UDP TX Processing Below IP Stack

__dev_xmit_skb()

ixgbe_tx_queue()

ip_local_out()


297-527

880-950

56-68

1224-1278

65-86

33


+dev()

dev_queue_xmit()

25-35

52-64

1996-2271 (size 1)

2370-2517 (size 1400)

sch_direct_xmit()


ixgbe_xmit_frame()

54-68

33

160-211

33

24-31

422-430

24-26

37-45

70-88


Figure 8: Westmere SMP Socket & UDP RX Processing above IP Stack

Application{}


udp_recvmsg()

wait_for_packet()

50-105

25-85

291-347 (size 1)

289-343 (size 1400)

95-165

465-554

sendto()

recvfrom()


172-251 (size 1)

178-251 (size 1400)

181-206 (size 1)

774-788 (size 1400)

sock_recvmsg()

__sock_recvmsg()



De-Q Loop

24


Figure 9: Westmere SMP IXGBE Interrupt Driven UDP RX Processing Below IP Stack



net_rx_action()


napi_gro_receive()




199-200

111

141-142

190-201

4131-4204

(Async Wake)

1818-1828

(Async-Wake)

1443-1484 to

Dequeue

Note: RPS not shown

885-890




ixgbe_receive_skb()


dev_gro_receive()

napi_skb_finish()

netif_receive_skb()

667-717

42-53

132-134

24

39

29

24

50-51

55

622-670

78-80

24

RX Packet DeQed

156-161

530-541

2200-2285

(Async Wake) 24

ixgbe_alloc_rx_buff

ers ()


Figure 10: Westmere SMP RX Protocol Stack Processing of UDP above IP Stack


pt_prev->func()

Protocol Stack

Processing

__udp4_lib_rcv()

Queue SKB

In Socket Queue

464-576

25-34

65-134

170-197

57 (No-Wake)


26-32

109-113

24

udp_queue_rcv_skb()

ip_queue_rcv_skb()


sk->sk_data_ready()



or



sk_wake_async()


24-31

49

809-819 (No-Wake)




UDP Process Timing for Non-SMP Single Core Westmere CPU

Figure 11: Single Core Dell Westmere Socket & UDP TX Processing above IP Stack

Application{}


ip_local_out()



114-120

228-363

54-71

97-101

239-301 (size 1)

240-292 (size 1400)

119-236

326-357

49-261

sendto()

recvfrom()

ip_append_data()


163-251 (size 1)

159-251 (size 1400)

161-177 (size 1)

653-671 (size 1400)

2057-2379 (size 1)

2771-3087 (size 1400)

37-230

69-85

1020-1315 (size 1)

1081-1206 (size 1400)

sock_sendmsg()


udp_sendmsg()

__sock_sendmsg()

58-68

Note: Swap in Delay

Across Data Sets


Figure 12: Single Core Dell Westmere UDP TX Processing Below IP Stack

__dev_xmit_skb()

ixgbe_tx_queue()

ip_local_out()


249-354

240-335

94-139

514-632

36-57

33


+dev()

dev_queue_xmit()

24-27

37-45

1020-1315 (size 1)

1081-1206 (size 1400)

sch_direct_xmit()


ixgbe_xmit_frame()

35-41

33

142-187

33

24-27

36-39

24-26

37-42

58-68


Figure 13: Single Core Dell Westmere Socket & UDP RX Processing above IP Stack

Application{}


udp_recvmsg()

wait_for_packet()

57-105

24

37-79

239-301 (size 1)

240-292 (size 1400)

93-170

266-291

sendto()

recvfrom()


163-251 (size 1)

159-251 (size 1400)

210-829 (size 1)

729-1405 (size 1400)

76-81

sock_recvmsg()

__sock_recvmsg()



De-Q Loop

88


Note: Non-SMP Interrupt Processing different interrupt and NAPI subroutines then SMP

processing.

Figure 14: Single Core Dell Westmere IXGBE Interrupt Driven UDP RX Processing Below IP

Stack

ixgbe_intr()


ixgbe_alloc_rx_buff

ers ()

net_rx_action()


napi_gro_receive()




406

2078

2610

(Async Wake)

1189-1216

(Async-Wake)

1481-1720 to

Dequeue

Note: RPS not shown

471-472


Ixgbe_poll()


ixgbe_receive_skb()


dev_gro_receive()

napi_skb_finish()

netif_receive_skb()

469-470

33-35

137-148

46-48

45

26-29

24

39

48-49

732-733

86-87

24

RX Packet DeQed

86-87

391-393

1668-1669

(Async Wake) 24


Figure 15: Single Core Dell Westmere RX Protocol Stack Processing of UDP above IP Stack


pt_prev->func()

Protocol Stack

Processing

__udp4_lib_rcv()

Queue SKB

In Socket Queue

433-463

25

51-68

165-187

24 (No-Wake)


24-34

62-68

24

udp_queue_rcv_skb()

ip_queue_rcv_skb()


sk->sk_data_ready()



or



sk_wake_async()


24-27

24-25

753-769 (No-Wake)




Figure 16: Single Core Dell Westmere ixgbe_low_latency_recv() Driver RX Flush Processing

netif_rx_ni()

netif_ni() __do_softirq()

net_rx_action()

(backlog in napi[] list,

by n->poll() call)

process_backlog()



49-50

94-97

75-82

71

30-35

846-853

(No-Wake)

753-769

(No-Wake)

75-80

48-55

40

Ixgbe_low_latency_recv()

Packet Detected

ixgbe_alloc_rx_buffer()

Ownership Released

62-123

25

300-304

24 402-410


UDP Process Timing Range for Dell Production Westmere

Figure 17: Dell Westmere SMP Socket & UDP TX Processing above IP Stack

Application{}


ip_local_out()



126-172

285-328

62-138

82-100

255-337 (size 1)

262-333 (size 1400)

27-52

396-519

103-180

sendto()

recvfrom()

ip_append_data()


161-240 (size 1)

168-241 (size 1400)

175-238 (size 1)

693-714 (size 1400)

3187-4071 (size 1)

3895-4762 (size 1400)

28-39

58-75

1986-2194 (size 1)

2130-2261 (size 1400)

sock_sendmsg()


udp_sendmsg()

__sock_sendmsg()

61-121


Figure 18: Dell Westmere SMP UDP TX Processing Below IP Stack

__dev_xmit_skb()

ixgbe_tx_queue()

ip_local_out()


269-626

812-950

53-60

1145-1268

54-103

33


+dev()

dev_queue_xmit()

25-28

56-63

1986-2194 (size 1)

2130-2261 (size 1400)

sch_direct_xmit()


ixgbe_xmit_frame()

50-56

33

152-242

33

24

393-424

24

37-46

61-121


Figure 19: Dell Westmere SMP Socket & UDP RX Processing above IP Stack

Application{}


udp_recvmsg()

wait_for_packet()

75-88

24

42-75

255-337 (size 1)

262-333 (size 1400)

100-186

364-497

sendto()

recvfrom()


161-240 (size 1)

168-241 (size 1400)

158-221 (size 1)

685-741 (size 1400)

801-827

sock_recvmsg()

__sock_recvmsg()



De-Q Loop

108


Figure 20: Dell Westmere SMP IXGBE Interrupt Driven UDP RX Processing Below IP Stack



net_rx_action()


napi_gro_receive()




251

101-106

101-107

177-196

3439-3544

(Async Wake)

1844-1911

(Async-Wake)

1325-1331 to

Dequeue

Note: RPS not shown

533-543




ixgbe_receive_skb()


dev_gro_receive()

napi_skb_finish()

netif_receive_skb()

610-651

55-56

153-158

24-25

44

46-51

26-27

45-52

41-42

592-635

79-81

33-38

RX Packet DeQed

122-123

720-739

2195-2270

(Async Wake) 33-36

ixgbe_alloc_rx_buff

ers ()


Figure 21: Dell Westmere SMP RX Protocol Stack Processing of UDP above IP Stack


pt_prev->func()

Protocol Stack

Processing

__udp4_lib_rcv()

Queue SKB

In Socket Queue

397-560

24-25

60-110

156-208

57 (No-Wake)


874-884 (No-Wake)

24-39

112-115

24-26

udp_queue_rcv_skb()

ip_queue_rcv_skb()


sk->sk_data_ready()



or



sk_wake_async()


24-26

48-50




Figure 22: Dell Westmere SMP ixgbe_low_latency_recv() Driver RX Flush Processing

netif_rx_ni()

netif_ni() __do_softirq()

net_rx_action()

(backlog in napi[] list,

by n->poll() call)

process_backlog()



39-47

157-171

91-96

91-92

18-22

958-863

(No-Wake)

874-884

(No-Wake)

70-75

65-70

40

Ixgbe_low_latency_recv()

Packet Detected

ixgbe_alloc_rx_buffer()

Ownership Released

29-32

415-417

293-384

24 402-428


UDP Process Timing Range for Nehalem

Figure 23: 2S Nehalem SMP Socket & UDP TX Processing above IP Stack

Application{}


ip_local_out()



153-238

246-339

57-143

97-123

295-329 (size 1)

295-329 (size 1400)

25-48

373-414

102-115

sendto()

recvfrom()

ip_append_data()


208-222 (size 1)

206-221 (size 1400)

176-269 (size 1)

701-815 (size 1400)

3210-3500 (size 1)

4028-4249 (size 1400)

24-62

74-84

2010-2097 (size 1)

2115-2287 (size 1400)

sock_sendmsg()


udp_sendmsg()

__sock_sendmsg()

259-329


Figure 24: 2S Nehalem SMP UDP TX Processing Below IP Stack

__dev_xmit_skb()

ixgbe_tx_queue()

ip_local_out()


248-498

878-914

57-69

1125-1294

63-89

33


+dev()

dev_queue_xmit()

25-27

51-53

2010-2097 (size 1)

2115-2287 (size 1400)

sch_direct_xmit()


ixgbe_xmit_frame()

33-68

33

158-198

33

24

51-93

295-346

34-46

259-329


Figure 25: 2S Nehalem SMP Socket & UDP RX Processing above IP Stack

Application{}


udp_recvmsg()

wait_for_packet()

53-126

24

26-80

295-329 (size 1)

295-329 (size 1400)

98-159

415-464

sendto()

recvfrom()


208-222 (size 1)

206-221 (size 1400)

193-205 (size 1)

460-744 (size 1400)

794-810

sock_recvmsg()

__sock_recvmsg()



De-Q Loop

106-107


Figure 26: 2S Nehalem SMP IXGBE Interrupt Driven UDP RX Processing Below IP Stack



net_rx_action()


napi_gro_receive()




199-201

95-96

132

174-177

3492-3623

(Async Wake)

1392-1400 to

Dequeue

Note: RPS not shown

808-810




ixgbe_receive_skb()


dev_gro_receive()

napi_skb_finish()

netif_receive_skb()

762-849

43-50

143-144

24

41

27-28

24

47-48

49-50

676-679

68-69

24

RX Packet DeQed

139

536-551

2200-2278

(Async Wake) 24

1748-1780

(Async-Wake)

ixgbe_alloc_rx_buff

ers ()


Figure 27: 2S Nehalem SMP RX Protocol Stack Processing of UDP above IP Stack


pt_prev->func()

Protocol Stack

Processing

__udp4_lib_rcv()

Queue SKB

In Socket Queue

472-561

25-33

72-121

162-201

56-57 (No-Wake)


36-56

102-108

24

udp_queue_rcv_skb()

ip_queue_rcv_skb()


sk->sk_data_ready()



or



sk_wake_async()


24-32

24-25

849-863 (No-Wake)




UDP Process Timing Range for Nehalem with 1 Socketed CPU

Figure 28: Nehalem, 1 CPU, SMP Socket & UDP TX Processing above IP Stack

Application{}


ip_local_out()



141-152

276-339

56-114

101-117

289-345 (size 1)

284-350 (size 1400)

28-45

381-392

111-120

sendto()

recvfrom()

ip_append_data()


176-249 (size 1)

176-243 (size 1400)

179-257 (size 1)

543-627 (size 1400)

3150-3498 (size 1)

3752-4137 (size 1400)

24-51

71-78

1970-2067 (size 1)

1994-2102 (size 1400)

sock_sendmsg()


udp_sendmsg()

__sock_sendmsg()

273-303


Figure 29: Nehalem, 1 CPU, SMP UDP TX Processing Below IP Stack

__dev_xmit_skb()

ixgbe_tx_queue()

ip_local_out()


288-419

884-926

60-76

1143-1312

62-81

33


+dev()

dev_queue_xmit()

25-27

51-63

1970-2067 (size 1)

1994-2102 (size 1400)

sch_direct_xmit()


ixgbe_xmit_frame()

33-65

33

160-191

33

24-25

49-84

296-343

35-42

273-303


Figure 30: Nehalem, 1 CPU, SMP Socket & UDP RX Processing above IP Stack

Application{}


udp_recvmsg()

wait_for_packet()

47-138

24

24-78

289-345 (size 1)

284-350 (size 1400)

94-162

374-383

sendto()

recvfrom()


176-249 (size 1)

176-243 (size 1400)

197-239 (size 1)

463-765 (size 1400)

816

sock_recvmsg()

__sock_recvmsg()



De-Q Loop

106


Figure 31: Nehalem, 1 CPU, SMP IXGBE Interrupt Driven UDP RX Processing Below IP Stack



net_rx_action()


napi_gro_receive()




202-203

98

136-137

178-181

3465-3471

(Async Wake)

1336-1367 to

Dequeue

Note: RPS not shown

841-853




ixgbe_receive_skb()


dev_gro_receive()

napi_skb_finish()

netif_receive_skb()

732-765

69-78

136-137

24

45-46

27

24-25

47-50

49-52

648-687

72-75

24

RX Packet DeQed

145-147

587-604

2235-2254

(Async Wake) 24

1737-1759

(Async-Wake)

ixgbe_alloc_rx_buff

ers ()


Figure 32: Nehalem, 1 CPU, SMP RX Protocol Stack Processing of UDP above IP Stack


pt_prev->func()

Protocol Stack

Processing

__udp4_lib_rcv()

Queue SKB

In Socket Queue

393-563

25-33

70-134

134-193

56-57 (No-Wake)


37-52

101-103

24-26

udp_queue_rcv_skb()

ip_queue_rcv_skb()


sk->sk_data_ready()



or



sk_wake_async()


24-30

24-25

752-766 (No-Wake)



UDP Flow Charts for linux-2.6.36.2 and different CPUs...

Documents

Transcript of UDP Flow Charts for linux-2.6.36.2 and different CPUs...