Breaking Open Linux Switching Drivers · significantly higher-performance and more generally...
Transcript of Breaking Open Linux Switching Drivers · significantly higher-performance and more generally...
![Page 1: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/1.jpg)
Breaking Open Linux Switching Drivers
Andy Gospodarek
Cumulus Networks16 Feb 2015
netdev01
![Page 2: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/2.jpg)
cumulusnetworks.com 2
Agenda
• Why• History• Design proposal• Future possibilities
![Page 3: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/3.jpg)
cumulusnetworks.com 3
Why am I here?
![Page 4: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/4.jpg)
cumulusnetworks.com 4
Linux kernel should enable others to
create the next generation of forwarding
devices
![Page 5: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/5.jpg)
cumulusnetworks.com 5
Integrate support for offload hardware
directly into the the Linux kernel
![Page 6: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/6.jpg)
cumulusnetworks.com 6
Hardware Platform History
![Page 7: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/7.jpg)
cumulusnetworks.com 7
Market dominated by switch and
router vendors providing expensive
proprietary solutions
![Page 8: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/8.jpg)
cumulusnetworks.com 8
Proprietary software running on switches
and routers was not open for developers
and users to enhance
![Page 9: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/9.jpg)
cumulusnetworks.com 9
Today’s hardware platforms are
significantly higher-performance and more
generally available
![Page 10: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/10.jpg)
cumulusnetworks.com 10
Spare CPU cycles are available for
applications to run directly on the switch
![Page 11: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/11.jpg)
cumulusnetworks.com 11
Bare-metal platforms are now appealing
and available to commercial Linux
vendors, developers, and users
![Page 12: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/12.jpg)
cumulusnetworks.com 12
Software History
![Page 13: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/13.jpg)
cumulusnetworks.com 13
15+ years with Linux as a viable OS for host
processor on switches/routers
![Page 14: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/14.jpg)
cumulusnetworks.com 14
10+ years Linux “support” by ASIC vendors
![Page 15: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/15.jpg)
cumulusnetworks.com 15
Basic in-kernel switching/offload layer support in v3.19
![Page 16: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/16.jpg)
cumulusnetworks.com 16
Software architecture to control ASICs
has not fundamentally
changed in the last decade
![Page 17: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/17.jpg)
cumulusnetworks.com 17
What exactly does that look like?
![Page 18: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/18.jpg)
Typical packet path
cumulusnetworks.com 18
Switching ASIC
vendordriver
Userspace Applicationand
Vendor SDKSocket applications
Virtual devices (tun/tap)
NIC tools and consumers
NIC Counters
![Page 19: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/19.jpg)
Control Plane Programming
cumulusnetworks.com 19
iproute2 and friends
vendor driver
Routing suites, bridge controllers, etc
Init
Virtual devices (tun/tap)
Linux kernel structures
FDB/FIB
Switching ASIC
Userspace Applicationand
Vendor SDK
![Page 20: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/20.jpg)
cumulusnetworks.com 20
Painful for those developing switches as
management applications need to talk to
kernel/netlink and SDKs
![Page 21: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/21.jpg)
Netlink Control Path
cumulusnetworks.com 21
vendor driver
iproute2 and friends
Routing suites, bridge controllers, etc
Init
Virtual devices (tun/tap)
Linux kernel structures
FDB/FIB
Switching ASIC
Userspace Applicationand
Vendor SDK
![Page 22: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/22.jpg)
cumulusnetworks.com 22
Much better design, but each SDK
supported needs a new translation
between netlink and SDK
![Page 23: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/23.jpg)
cumulusnetworks.com 23
Kernel hackers and distribution vendors
see a simple solution
![Page 24: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/24.jpg)
cumulusnetworks.com 24
Get rid of all closed-source SDKs
![Page 25: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/25.jpg)
cumulusnetworks.com 25
Great idea!
![Page 26: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/26.jpg)
cumulusnetworks.com 26
Vendors do not want to open
source their SDKs
![Page 27: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/27.jpg)
cumulusnetworks.com 27
Can we use a userspace SDK and a
kernel driver at the same time?
![Page 28: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/28.jpg)
cumulusnetworks.com 28
![Page 29: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/29.jpg)
cumulusnetworks.com 29
Not if you want it upstream!
![Page 30: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/30.jpg)
cumulusnetworks.com 30
OK...how do I get started?
![Page 31: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/31.jpg)
cumulusnetworks.com 31
Phased Approach
• Participate!• Pick a hardware platform• Write and post a switchdev-compatible
network driver• Enhance that driver to add
ndo/offload_ops to driver
![Page 32: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/32.jpg)
cumulusnetworks.com 32
Attend conferences, participate on
mailing-lists, and post patches
![Page 33: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/33.jpg)
cumulusnetworks.com 33
Write and post a switchdev-compatible
network driver
![Page 34: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/34.jpg)
cumulusnetworks.com 34
Advantages
• Provide network access via front panel ports
• Phased approach to working upstream• Applications can developed without need
for hardware offload
![Page 35: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/35.jpg)
cumulusnetworks.com 35
What might that look like?
![Page 36: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/36.jpg)
Base Switchdev Driver
cumulusnetworks.com 36
Socket applications
Ethtool
NIC tools and consumers
switchdev compatible driver
Switching ASIC
CountersLink MgmtPort Init RxTx
![Page 37: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/37.jpg)
cumulusnetworks.com 37
Great, we are upstream!
Are we done?
![Page 38: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/38.jpg)
cumulusnetworks.com 38
No!
![Page 39: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/39.jpg)
cumulusnetworks.com 39
Add offload support to driver as
upstream infrastructure is developed
![Page 40: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/40.jpg)
cumulusnetworks.com 40
What might that look like?
![Page 41: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/41.jpg)
Switchdev Offload Driver
cumulusnetworks.com 41
Socket applications
Ethtool
NIC tools, Routing suites, bridge controllers, etc
switchdev compatible driver
Switching ASIC
CountersLink MgmtPort Init Offload Ops
iproute2 and friends
RxTx
![Page 42: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/42.jpg)
cumulusnetworks.com 42
“If you are the first you will be so cool.”
-DaveM
![Page 43: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/43.jpg)
cumulusnetworks.com 43
Get coding
![Page 44: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/44.jpg)
cumulusnetworks.com 44
“..and we’ll help you maintain it”
-DaveM
![Page 45: Breaking Open Linux Switching Drivers · significantly higher-performance and more generally available. cumulusnetworks.com 10 ... Vendor SDK. cumulusnetworks.com 22 Much better design,](https://reader035.fdocuments.in/reader035/viewer/2022071023/5fd78a2b8a2a0561556fd2e8/html5/thumbnails/45.jpg)
© 2013 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus Networks, Inc. or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis.
▪Thank You!
cumulusnetworks.com 45