Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code...
Transcript of Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code...
![Page 1: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/1.jpg)
1
Playing BBR with a userspacenetwork stack
Hajime TazakiIIJ
April, 2017, Linux netdev 2.1, Montreal, Canada
![Page 2: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/2.jpg)
2 . 1
Linux Kernel LibraryA library of Linux kernel code
to be reusable on various platforms
On userspace applications (can be FUSE, NUSE, BUSE)As a core of UnikernelWith network simulation (under development)
Use cases
Operating system personalityTiny guest operating system (single process)Testing/Debugging
![Page 3: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/3.jpg)
2 . 2
MotivationMegaPipe [OSDI '12]
outperforms baseline Linux .. 582% (for short connections).New API for applications (no existing applications benefit)
mTCP [NSDI '14]improve... by a factor of 25 compared to the latest Linux TCPimplement with very limited TCP extensions
SandStorm [SIGCOMM '14]our approach ..., demonstrating 2-10x improvementsspecialized (no existing applications benefit)
Arrakis [OSDI '14]improvements of 2-5x in latency and 9x in throughput .. to Linuxutilize simplified TCP/IP stack (lwip) (loose feature-rich extensions)
IX [OSDI '14]improves throughput ... by 3.6x and reduces latency by 2xutilize simplified TCP/IP stack (lwip) (loose feature-rich extensions)
![Page 4: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/4.jpg)
2 . 3
Sigh
![Page 5: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/5.jpg)
2 . 4
Motivation (cont'd)1. Reuse feature-rich network stack, not re-implement or port
re-implement: give up (matured) decades' effortport: hard to track the latest version
2. Reuse preserves various semantics
syntax level (command line)API leveloperation level (utility scripts)
3. Reasonable speed with generalized userspace network stack
x1 speed of the original
![Page 6: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/6.jpg)
2 . 5
LKL outlooksh/w independent (arch/lkl)
various platforms
Linux userspaceWindows userspaceFreeBSD user spaceqemu/kvm (x86, arm) (unikernel)uEFI (EFIDroid)
existing applications support
musl libc bindcross build toolchain
EFIDroid: http://efidroid.org/
![Page 7: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/7.jpg)
2 . 6
Demo
![Page 8: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/8.jpg)
2 . 7
userspace network stack ?Concerns about timing accuracy
how LKL behaves with BBR (requires higher timing accuracy) ?Having network stack in userspace may complicate variousoptimization
LKL at netdev1.2https://youtu.be/xP9crHI0aAU?t=34m18s
![Page 9: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/9.jpg)
3 . 1
Playing BBR with LKL
![Page 10: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/10.jpg)
3 . 2
TCP BBRBottleneck Bandwidth and Round-trip propagation timeControl Tx rate
congestion not based on the packet lossestimate MinRTT and MaxBW (on each ACK)
http://queue.acm.org/detail.cfm?id=3022184
![Page 11: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/11.jpg)
3 . 3
TCP BBR (cont'd)On Google's B4 WAN (across North America, EU, Asia)Migrated from cubic to bbr in 2016x2 - x25 improvements
![Page 12: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/12.jpg)
4 . 1
1st Benchmark (Oct. 2016)netperf (TCP_STREAM, -K bbr/cubic)2-node 10Gbps b2b link
tap+bridge (LKL)direct ixgbe (native)
No loss, no bottleneck, close link
netperf(client) netserver +------+ +--------+ | | | | |sender+--------------+|receiver| | |==============>| | | | | | +------+ +--------+ Linux-4.9-rc4 Linux-4.6 (host,LKL) bbr/cubic cubic,fq_codel (default)
![Page 13: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/13.jpg)
4 . 2
1st Benchmark netperf(client) netserver +------+ +--------+ | | | | |sender+--------------+|receiver| | |==============>| | | | | | +------+ +--------+ Linux-4.9-rc4 Linux-4.6 (host,LKL) bbr/cubic cubic,fq_codel (default)
cc tput (Linux) tput (LKL)
bbr 9414.40 Mbps 456.43 Mbps
cubic 9411.46 Mbps 9385.28 Mbps
![Page 14: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/14.jpg)
4 . 3
What ??only BBR + LKL shows badInvestigation
ack timestamp used by RTT measurement needed a precise timeevent (clock)providing high resolution timestamp improve the BBR performance
![Page 15: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/15.jpg)
4 . 4
Change HZ (tick interval)cc tput
(Linux,hz1000)tput
(LKL,hz100)tput
(LKL,hz1000)
bbr 9414.40 Mbps 456.43 Mbps 6965.05 Mbps
cubic 9411.46 Mbps 9385.28 Mbps 9393.35 Mbps
![Page 16: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/16.jpg)
4 . 5
Timestamp on (ack) receiptFrom
unsigned long long __weak sched_clock(void) { return (unsigned long long)(jiffies - INITIAL_JIFFIES) * (NSEC_PER_SEC / HZ); }
To
unsigned long long sched_clock(void) { return lkl_ops->time(); // i.e., clock_gettime() }
cc tput (Linux) tput(LKL,hz100)
tput (LKLsched_clock,hz100)
bbr 9414.40Mbps
456.43 Mbps 9409.98 Mbps
![Page 17: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/17.jpg)
4 . 6
What happens if no sched_clock() ?low throughput due to longer RTT measurementA patch (by Neal Cardwell) to torelate lower in jiffies resolution
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 56fe736..b0f1426 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -3196,6 +3196,8 @@ static int tcp_clean_rtx_queue(struct sock *sk, int prior_fackets, ca_rtt_us = skb_mstamp_us_delta(now, &sack->last_sackt); } sack->rate->rtt_us = ca_rtt_us; /* RTT of last (S)ACKed packet, or -1 */+ if (sack->rate->rtt_us == 0) + sack->rate->rtt_us = jiffies_to_usecs(1); rtt_update = tcp_ack_update_rtt(sk, flag, seq_rtt_us, sack_rtt_us, ca_rtt_us);
diff --git a/net/ipv4/tcp_rate.c b/net/ipv4/tcp_rate.c index 9be1581..981c48e 100644 --- a/net/ipv4/tcp_rate.c +++ b/net/ipv4/tcp_rate.c
cc tput (Linux) tput (LKL,hz100) tput (LKL patched,hz100)
bbr 9414.40 Mbps 456.43 Mbps 9413.51 Mbpshttps://groups.google.com/forum/#!topic/bbr-dev/sNwlUuIzzOk
![Page 18: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/18.jpg)
5 . 1
2nd Benchmarkdelayed, lossy network on 10Gbps
netem (middlebox)
netperf(client) netserver +------+ +---------+ +--------+ | | | | | | |sender+--------+middlebox+------+|receiver| | |======= |======== |======>| | | | | | | | +------+ +---------+ +--------+ cc: BBR 1% pkt loss fq-enabled 100ms delay tcp_wmem=100M
cc tput (Linux) tput (LKL)
bbr 8602.40 Mbps 145.32 Mbps
cubic 632.63 Mbps 118.71 Mbps
![Page 19: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/19.jpg)
5 . 2
Memory w/ TCPconfigurable parameter for socket, TCP
sysctl -w net.ipv4.tcp_wmem="4096 16384 100000000"delay and loss w/ TCP requires increased buffer
"LKL_SYSCTL=net.ipv4.tcp_wmem=4096 16384 100000000"
![Page 20: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/20.jpg)
5 . 3
Memory w/ TCP (cont'd)default memory size (of LKL): 64MiB
the size affects the sndbuf size
static bool tcp_should_expand_sndbuf(const struct sock *sk) { (snip) /* If we are under global TCP memory pressure, do not expand. */ if (tcp_under_memory_pressure(sk)) return false; (snip) }
![Page 21: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/21.jpg)
5 . 4
Timer relatesCONFIG_HIGH_RES_TIMERS enabled
fq scheduler usesproperly transmit packets with probed BW
fq configuration
instead of tc qdisc add fq
![Page 22: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/22.jpg)
5 . 5
fq schedulerEvery fq_flow entry scheduled schedule a timer event
with high-resolution timer (in nsec)
static struct sk_buff *fq_dequeue() => void qdisc_watchdog_schedule_ns() => hrtimer_start()
![Page 23: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/23.jpg)
5 . 6
How slow high-resolution timer ?
Delay = (nsec of expiration) - (nsec of scheduled)
![Page 24: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/24.jpg)
5 . 7
Scheduler improvementLKL's scheduler
outsourced based on thread impls (green/native)minimum delay of timer interrupt (of LKL emulated)
>60 usec (green thread)>20 usec (native thread)
![Page 25: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/25.jpg)
5 . 8
Scheduler improvement1. avoid system call (clock_nanosleep) when block
busy poll (watch clock instead) if sleep is < 10usec
60 usec => 20 usec
2. reuse green thread stack (avoid mmap per a timer irq)20 usec => 3 usec
int sleep(u64 nsec) { /* fast path */ while (1) { if (nsec < 10*1000) { clock_gettime(CLOCK_MONOTONIC, &now); if (now - start > nsec) return; } } /* slow path */ return syscall(SYS_clock_nanosleep) }
![Page 26: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/26.jpg)
5 . 9
Timer delay improved ?Before (top), After (bottom)
![Page 27: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/27.jpg)
5 . 10
Results (TCP_STREAM, bbr/cubic)
netperf(client) netserver +------+ +---------+ +--------+ | | | | | | |sender+--------+middlebox+------+|receiver| | |======= |======== |======>| | | | | | | | +------+ +---------+ +--------+ cc: BBR 1% pkt loss fq-enabled 100ms delay tcp_wmem=100M
![Page 28: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/28.jpg)
6 . 1
Patched LKL1. add sched_clock()2. add sysctl configuration i/f (net.ipv4.tcp_wmem)3. make system memory configurable (net.ipv4.tcp_mem)4. enable CONFIG_HIGH_RES_TIMERS5. add sch-fq configuration6. scheduler hacked (uspace specific)
avoid syscall for short sleepavoid memory allocation for each (thread) stack
7. (TSO, csum offload, by Jerry/Yuan from Google, netdev1.2)
![Page 29: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/29.jpg)
6 . 2
Next possible stepsdo profile while (lower LKL performance)
e.g., context switch of uspace threadsVarious short-cuts
busy polling I/Os (packet, clock, etc)replacing packet I/O (packet_mmap)
short packet performance (i.e., 64B)practical workload (e.g., HTTP)> 10Gbps link
![Page 30: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/30.jpg)
6 . 3
on qemu/kvm ?Based on rumprun unikernelPerformance under investigationNo scheduler issue (notdepending on syscall)
- http://www.linux.com/news/enterprise/cloud-computing/751156-are-cloud-operating-systems-the-next-
big-thing-
![Page 31: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/31.jpg)
SummaryTiming accuracy concern was rightperformance obstacle in userspace execution
scheduler relatedalleviated somehow
Timing severe features degraded from Linuxother options (unikernel)
The benefit of reusable code
![Page 32: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/32.jpg)
6 . 46 . 5
ReferencesLKL
Other related reposhttps://github.com/lkl/linux
https://github.com/libos-nuse/lkl-linuxhttps://github.com/libos-nuse/frankenlibchttps://github.com/libos-nuse/rumprun
![Page 33: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/33.jpg)
6 . 6
Backup
![Page 34: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/34.jpg)
6 . 7
AlternativesFull Virtualization
KVMPara Virtualization
XenUML
Lightweight VirtualizationContainer/namespaces
![Page 35: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/35.jpg)
6 . 8
What is not LKL ?not specific to a userspace network stackIs a reusable library that we can use everywhere (in theory)
![Page 36: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/36.jpg)
6 . 9
How others think about userspace ?DPDK is not Linux (@ netdev 1.2)
The model of DPDK isn't compatible with Linuxbreak security model (protection never works)
XDP is Linux
![Page 37: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/37.jpg)
6 . 10
userspace network stack (checklist)PerformanceSafetyTypos take the entire system downDeveloper pervasivenessKernel reboot is disruptiveTraffic loss
ref: XDP Inside and Out ( )https://github.com/iovisor/bpf-docs/blob/master/XDP_Inside_and_Out.pdf
![Page 38: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/38.jpg)
6 . 11
TCP BBR (cont'd)BBR requires
packet pacingprecise RTT measurement
function onAck(packet) rtt = now - packet.sendtime update_min_filter(RTpropFilter, rtt) delivered += packet.size delivered_time = now deliveryRate = (delivered - packet.delivered) / (delivered_time - packet.delivered_time) if (deliveryRate > BtlBwFilter.currentMax || ! packet.app_limited) update_max_filter(BtlBwFilter, deliveryRate) if (app_limited_until > 0) app_limited_until = app_limited_until - packet.size
http://queue.acm.org/detail.cfm?id=3022184
![Page 39: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/39.jpg)
6 . 12
How timer works ?1. schedule an event2. add (hr)timer list queue (hrtimer_start())3. (check expired timers in timer irq) (hrtimer_interrupt())4. invoke callbacks (__run_hrtimer())
![Page 40: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/40.jpg)
Timer delay improved ?Before (top), After (bottom)
![Page 41: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/41.jpg)
6 . 136 . 14
IPv6 ready
![Page 42: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/42.jpg)
6 . 15
how timer interrupt works ?native thread ver.
1. timer_settime(2)instantiate a pthread
2. wakeup the thread3. trigger a timer interrupt (of LKL)
update jiffies, invoke handlers
green thread ver.
1. instantiate a green threadmalloc/mmap, add to sched queue
2. schedule an eventclock_nanosleep(2) until next event)or do something (goto above)
3. trigger a timer interrupt
![Page 43: Playing BBR with a userspace network stack · Linux Kernel Library A library of Linux kernel code to be reusable on various platforms On userspace applications (can be FUSE, NUSE,](https://reader034.fdocuments.in/reader034/viewer/2022042711/5f84e9abd1f24944085a807f/html5/thumbnails/43.jpg)
6 . 16
TSO/Checksum offloadvirtio basedguest-side: use Linux driver
TCP_STREAM (cubic, no delay)