Post on 30-May-2020
NetFPGA SummerCourse
Presented by: Andrew W Moore, Noa Zilberman, Gianni AntichiStephen Ibanez, Marcin Wojcik, Jong Hun Han,
Salvator Galea, Murali Ramanujam, Jingyun Zhang, Yuta Tokusashi
University of CambridgeJuly 24 – July 28, 2017
http://NetFPGA.org
ThepowerofOpenSource:buildyourownproofofconcept!
GianniAntichi
ComputerLaboratoryUniversityofCambridge
gianni.antichi@cl.cam.ac.ukhttps://www.cl.cam.ac.uk/~ga288
FPGA
Memory
PCI-Express
CPU Memory
NetFPGA Driver
Networking Software
10GbE
10GbE
10GbE
10GbE
10GbE
10GbE
10GbE
10GbE
EDA Tools(Xilinx,
Mentor, etc.)
VerilogmodulesinterconnectedbyFIFOinterfaces
NetFPGA:NetworkedFPGA
My Design
(10GE MAC is soft/replaceable)
BlueSwitch:AMultiTableOpenFlowSwitch
Yourdesigncanlookcompletelydifferent!
Blueswitch:Enablingprovablyconsistentconfigurationofnetworkswitches,HanJ.H.,Mundkur P.,Rotsos C.,Antichi G.,DaveN.,MooreA.W.,NeumannP.G.,ACM/IEEEANCS,Oakland,CA,USA,7-8May,2015
• Opensource,RISCbasedSoC architectures• RISC-V– RISC-VISAsoftprocessor,LinuxOS• CHERI– 64bitMIPSsoftprocessor,BSDOS
NetSoc:NetFPGA +OpenSourceProcessors
PrototypingRISCBased,ReconfigurableNetworkingApplicationsinOpenSource,HanJ.H.,Zilberman N.,Zeeb B.A,Fiessler A.andMoore A.W.,TechnicalReport,CoRR abs/1612.05547,2016
pciebench:anopensourcetoolforbenchmarkingPCIExpress
thereisalimitedunderstandingofPCIe functionality,northetrade-offsthatmustbemadetogetbest-performancefromPCIe systems
• pciebench toolopensourceavailable• ItbuildsonNetFPGA andNetronome boards
400
500
600
700 800 900
1000 1100 1200 1300 1400 1500 1600 1700
8 16 32 64 128 256 512 1024 2048
Late
ncy
(ns)
Transfer Size (Bytes)
LAT_RD (NFP6000-HSW)LAT_RD (NetFPGA-HSW)
LAT_WRRD (NFP6000-HSW)LAT_WRRD (NetFPGA-HSW)
PowerEfficientMAC• APlatformfor100Gb/spower-savingMACdesign(e.g.lights-outMAC)
• PortingMACdesigntoSUMEpermits:– Powermeasurements– Testingprotocol’sresponse– Reconsiderationofpower-savingmechanisms– Evaluatingsuitabilityforcomplexarchitecturesandsystems
• Acceleratingnetworkingservices• Compiling.Net programs
– Tox86– Tosimulationenvironment– TomultipleFPGAtargets
Emu:RapidFPGAPrototypingofNetworkingServicesinC#
Emu:RapidPrototypingofNetworkingServices,SultanaN.,etal.,Usenix AnnualTechnicalConference(ATC),July2017
OpenSourceNetworkTester
• NetFPGAplatformenabledthefirstprototypeofOSNT.
• TheopennatureofNetFPGAecosystemrepresentsthebeststartingpointforopenHW/SWcommunity-orientedprojects.
• OSNTaimstobuildacommunityasNetFPGAdid.OSNT:OpenSourceNetworkTester,Antichi G.etal.,IEEENetworkMagazine,SpecialissueonOpenSourceforNetworking:ToolsandApplications,2014
Opensourcehardwareandsoftwareplatformfornetworkmonitoringandtesting.
https://osnt.orgLowcost,flexibletoupdate,scale-out,noCPUusage,nanosecondresolutionmeasurements
OSNT:OpenSourceNetworkTester
OSNT:OpenSourceNetworkTester,Antichi G.etal.,IEEENetworkMagazine,SpecialissueonOpenSourceforNetworking:ToolsandApplications,2014
NetworkTesterComparison
Cost Flexibility Precision LineRate
DPDK,Moongen
OpenSourceNetworkTester§ OSNTisanopensourceHW/SWplatformfornetworktesting
OSNTHardware
OSKernelNetFPGAKernelDriver
OSNTApplications
nf0 nf1 nf2 nf3
Generator Monitor
x8LanesPCIEGen3
Port0 Port1 Port2 Port3
§ WritteninC,Python§ OpenAPIandregisters
§ WritteninVerilogHDLusingstandardXilinxprotocols
§ Userscanaddandmodifythemodules
OpenSourceNetworkTester§ OSNTisanopensourceHW/SWplatformfornetworktesting
OSNTHardware
OSKernelNetFPGAKernelDriver
OSNTApplications
nf0 nf1 nf2 nf3
Generator Monitor
x8LanesPCIEGen3
Port0 Port1 Port2 Port3
§ WritteninC,Python§ OpenAPIandregisters
§ WritteninVerilogHDLusingstandardXilinxprotocols
§ Userscanaddandmodifythemodules
YOURApplications
YOURLogic
OpenSourceNetworkTesterOSNTcurrently is:
§ 4x10Gbpstrafficgenerator.§ Capturecardwithhighresolutiontimestamp(6.4nsec).§ GPS-readysynchronizedmeasurementkit.
OpenSourceNetworkTesterOSNTcurrently is:
§ 4x10Gbpstrafficgenerator.§ Capturecardwithhighresolutiontimestamp(6.4nsec).§ GPS-readysynchronizedmeasurementkit.
astartingpoint
OSNTArchitecture
InputArbiter Monitor Generator Output
Queues
HOST
PCIe
InputArbiter Monitor Generator Output
Queues
HOST
PCIe
OSNTmonitor
OSNTArchitecture
InputArbiter Monitor Generator Output
Queues
HOST
PCIe
OSNTmonitor OSNTgenerator
OSNTArchitecture
InputArbiter Monitor Generator Output
Queues
HOST
PCIe
OSNTmonitor OSNTgenerator
OSNTArchitecture
OSNTRXinterface
MACPCS/PMASFPPacketin
§ TimestamptakenbeforeRXqueuestoreduceFIFO-inducedjitter
OSNTRXinterface
MACPCS/PMASFPPacketin
§ TimestamptakenbeforeRXqueuestoreduceFIFO-inducedjitter§ Timestampoverwritespacketdataataconfigurableoffset
InputArbiter Monitor Generator Output
Queues
HOST
PCIe
OSNTmonitor OSNTgenerator
OSNTArchitecture
OSNTTXinterface
MACPCS/PMA
SFP Packetout
§ TimestamptakenafterTXqueuestoreduceFIFO-inducedjitter
OSNTTXinterface
MACPCS/PMA
SFP Packetout
§ TimestamptakenafterTXqueuestoreduceFIFO-inducedjitter§ Timestampoverwritespacketdataataconfigurableoffset
OSNTTXinterface
MACPCS/PMA
SFP Packetout
§ TimestamptakenafterTXqueuestoreduceFIFO-inducedjitter§ Timestampoverwritespacketdataataconfigurableoffset§ Ifenabled,itwilloverwrite128bitdata:
Dst MAC ... signature pkt count tx timestamp ...
32 bit 32 bit 64 bit
OSNTTimestamp
§ Freerunningcounter?
§ Wecouldusea64-bitcounterdrivenbythe156.25MHzsystemclock(naïvesolution)
OSNTTimestamp
§ Freerunningcounter?
§ Wecouldusea64-bitcounterdrivenbythe156.25MHzsystemclock(naïvesolution)
– providesnomeansbywhichtocorrectoscillatorfrequencydrift
OSNTTimestamp
§ Freerunningcounter?
§ Wecouldusea64-bitcounterdrivenbythe156.25MHzsystemclock(naïvesolution)
– providesnomeansbywhichtocorrectoscillatorfrequencydrift– producestimestampsexpressedinunitof6.4ns
OSNTTimestamp
§ Freerunningcounter?
§ Wecouldusea64-bitcounterdrivenbythe156.25MHzsystemclock(naïvesolution)
– providesnomeansbywhichtocorrectoscillatorfrequencydrift– producestimestampsexpressedinunitof6.4ns– fixed-pointrepresentationoftimeinsecondsmoreusefultohost
OSNTTimestamp
§ DirectDigitalSynthesis(DDS)isthesolution!!
§ DDSisatechniquebywhicharbitraryvariablefrequenciescanbegenerated
OSNTTimestamp
§ DirectDigitalSynthesis(DDS)isthesolution!!
§ DDSisatechniquebywhicharbitraryvariablefrequenciescanbegenerated
– needatimereferencetocorrectDDSrate(theGPSprovideslong-termstability)
OSNTTimestamp
§ DirectDigitalSynthesis(DDS)isthesolution!!
§ DDSisatechniquebywhicharbitraryvariablefrequenciescanbegenerated
– needatimereferencetocorrectDDSrate(theGPSprovideslong-termstability)
– allow64bitvalueinfixed-pointrepresentation
OSNTTimestamp
§ DirectDigitalSynthesis(DDS)isthesolution!!
§ DDSisatechniquebywhicharbitraryvariablefrequenciescanbegenerated
– needatimereferencetocorrectDDSrate(theGPSprovideslong-termstability)
– allow64bitvalueinfixed-pointrepresentation– how Endace DAGcardworks!
InputArbiter Monitor Generator Output
Queues
HOST
PCIe
OSNTmonitor OSNTgenerator
OSNTArchitecture
OSNTGenerator
RL
DM
TS
10GTx
RL
DM
TS
10GTx
RL
DM
TS
10GTx
RL
DM
TS
10GTx
PCAPReplayEngine
QDRController
QDRController
DDR3Controller
DDR3Controller
PCIeDMA
To/FromQDR
Memory
To/FromQDR
Memory
To/FromDDR3
Memory
To/FromDDR3
Memory
DM:DelayModule
RL:RateLimiter
TS:Timestamp
InputArbiter
To/FromHostPC
OSNTGenerator
RL
DM
TS
10GTx
RL
DM
TS
10GTx
RL
DM
TS
10GTx
RL
DM
TS
10GTx
PCAPReplayEngine
QDRController
QDRController
DDR3Controller
DDR3Controller
PCIeDMA
To/FromQDR
Memory
To/FromQDR
Memory
To/FromDDR3
Memory
To/FromDDR3
Memory
DM:DelayModule
RL:RateLimiter
TS:Timestamp
InputArbiter
To/FromHostPC
§ 4x10GPCAPreplayengine
§ SRAM:27MB§ DRAM:8GB
OSNTGenerator
§ 4x10GPCAPreplayengine
§ SRAM:27MB§ DRAM:8GB
§ DelaymoduleRL
DM
TS
10GTx
RL
DM
TS
10GTx
RL
DM
TS
10GTx
RL
DM
TS
10GTx
PCAPReplayEngine
QDRController
QDRController
DDR3Controller
DDR3Controller
PCIeDMA
To/FromQDR
Memory
To/FromQDR
Memory
To/FromDDR3
Memory
To/FromDDR3
Memory
DM:DelayModule
RL:RateLimiter
TS:Timestamp
InputArbiter
To/FromHostPC
§ 4x10GPCAPreplayengine
§ SRAM:27MB§ DRAM:8GB
§ Delaymodule§ Ratelimiter
RL
DM
TS
10GTx
RL
DM
TS
10GTx
RL
DM
TS
10GTx
RL
DM
TS
10GTx
PCAPReplayEngine
QDRController
QDRController
DDR3Controller
DDR3Controller
PCIeDMA
To/FromQDR
Memory
To/FromQDR
Memory
To/FromDDR3
Memory
To/FromDDR3
Memory
DM:DelayModule
RL:RateLimiter
TS:Timestamp
InputArbiter
To/FromHostPC
OSNTGenerator
§ 4x10GPCAPreplayengine
§ SRAM:27MB§ DRAM:8GB
§ Delaymodule§ Ratelimiter§ TXtimestamping
RL
DM
TS
10GTx
RL
DM
TS
10GTx
RL
DM
TS
10GTx
RL
DM
TS
10GTx
PCAPReplayEngine
QDRController
QDRController
DDR3Controller
DDR3Controller
PCIeDMA
To/FromQDR
Memory
To/FromQDR
Memory
To/FromDDR3
Memory
To/FromDDR3
Memory
DM:DelayModule
RL:RateLimiter
TS:Timestamp
InputArbiter
To/FromHostPC
OSNTGenerator
InputArbiter Monitor Generator Output
Queues
HOST
PCIe
OSNTmonitor OSNTgenerator
OSNTArchiteture
10GRx
10GRx
10GRx
10GRx
RxQ RxQ RxQ RxQ
StatisticsCollector
HeaderExtractor
TCAM
DecisionModule
PacketsFIFO
Cut/Hash
TimeStamp
InputArbiter
Aggregatestatisticstohostsoftware
TCAMrulemanager
Cut/Hashsettingfromhost
Hostanalysissoftware
CoreMonitoring
FilteringStage
PCIe DMA
OSNTMonitor
10GRx
10GRx
10GRx
10GRx
RxQ RxQ RxQ RxQ
StatisticsCollector
HeaderExtractor
TCAM
DecisionModule
PacketsFIFO
Cut/Hash
TimeStamp
InputArbiter
Aggregatestatisticstohostsoftware
TCAMrulemanager
Cut/Hashsettingfromhost
Hostanalysissoftware
CoreMonitoring
FilteringStage
PCIe DMA
packetenteringtheboard
OSNTMonitor
§ RXtimestamp
10GRx
10GRx
10GRx
10GRx
RxQ RxQ RxQ RxQ
StatisticsCollector
HeaderExtractor
TCAM
DecisionModule
PacketsFIFO
Cut/Hash
TimeStamp
InputArbiter
Aggregatestatisticstohostsoftware
TCAMrulemanager
Cut/Hashsettingfromhost
Hostanalysissoftware
CoreMonitoring
FilteringStage
PCIe DMA
OSNTMonitor
§ RXtimestamp
§ Statscollector
10GRx
10GRx
10GRx
10GRx
RxQ RxQ RxQ RxQ
StatisticsCollector
HeaderExtractor
TCAM
DecisionModule
PacketsFIFO
Cut/Hash
TimeStamp
InputArbiter
Aggregatestatisticstohostsoftware
TCAMrulemanager
Cut/Hashsettingfromhost
Hostanalysissoftware
CoreMonitoring
FilteringStage
PCIe DMA
OSNTMonitor
§ RXtimestamp
§ Statscollector
§ TCAM-basedPacketFilter(5-tuple)
10GRx
10GRx
10GRx
10GRx
RxQ RxQ RxQ RxQ
StatisticsCollector
HeaderExtractor
TCAM
DecisionModule
PacketsFIFO
Cut/Hash
TimeStamp
InputArbiter
Aggregatestatisticstohostsoftware
TCAMrulemanager
Cut/Hashsettingfromhost
Hostanalysissoftware
CoreMonitoring
FilteringStage
PCIe DMA
OSNTMonitor
§ RXtimestamp
§ Statscollector
§ TCAM-basedPacketFilter(5-tuple)
§ Cut-Hashfunction
10GRx
10GRx
10GRx
10GRx
RxQ RxQ RxQ RxQ
StatisticsCollector
HeaderExtractor
TCAM
DecisionModule
PacketsFIFO
Cut/Hash
TimeStamp
InputArbiter
Aggregatestatisticstohostsoftware
TCAMrulemanager
Cut/Hashsettingfromhost
Hostanalysissoftware
CoreMonitoring
FilteringStage
PCIe DMA
OSNTMonitor
§ RXtimestamp
§ Statscollector
§ TCAM-basedPacketFilter(5-tuple)
§ Cut-Hashfunction
10GRx
10GRx
10GRx
10GRx
RxQ RxQ RxQ RxQ
StatisticsCollector
HeaderExtractor
TCAM
DecisionModule
PacketsFIFO
Cut/Hash
TimeStamp
InputArbiter
Aggregatestatisticstohostsoftware
TCAMrulemanager
Cut/Hashsettingfromhost
Hostanalysissoftware
CoreMonitoring
FilteringStage
PCIe DMA
OSNTMonitor
OSNTGUI§ OSNTGUI– ExtensibleGeneratorandMonitorGUIinPython.§ Command-Line-Interfaceisalsoavailable.
GeneratorGUI
MonitorGUI
49
OSNTcommandline§ Command-Line-Interfaceisavailabletocreateascriptautomatingthetestprocess.
OSNTinaction
Enablingnetworkinnovationwithaccuratenetworkingsystemscharacterization
Forwardinglatencymeasurement§ Unloadedswitchesbaselinelatencynocrosstraffic
TS-Tx
OSNT
OSKernel
Application
TS-Rx
deviceundertest
Forwardinglatencymeasurement
Blueswitch:Enablingprovablyconsistentconfigurationofnetworkswitches,HanJ.H.,Mundkur P.,Rotsos C.,Antichi G.,DaveN.,MooreA.W.,NeumannP.G.,ACM/IEEEANCS,Oakland,CA,USA,7-8May,2015
SDNTestingSuite
• SDNenablesunprecedentedflexibleandextensiblenetworkcontrol
• OpenFlow specificationslackperformancesemantics– Whatdoesabarrierreplysignifies?
• OpenFlow performanceaspectsareyettobeexplored– HowdoyoucomparetwoOpenFlow switches?
• OpenFlow flexibilityisnotalwaysportableonswitchASIC
OpenFlow toolstack X-Ray
ControlApplicationComplexity
OpenFlow toolstack X-Ray
ControlApplicationComplexity
ControlChannelCapacity
OpenFlow toolstack X-Ray
ControlApplicationComplexity
ControlChannelCapacity
Scarceco-processorresourcesSwitchOSschedulingisnontrivial
OpenFlow toolstack X-Ray
ControlApplicationComplexity
ControlChannelCapacity
Scarceco-processorresourcesSwitchOSschedulingisnontrivial
ASICdriver->policyconfiguration
OpenFlow toolstack X-Ray
ControlApplicationComplexity
ControlChannelCapacity
Scarceco-processorresourcesSwitchOSschedulingisnontrivial
ASICdriver->policyconfiguration
OpenFlow toolstack X-Ray
SDNController
SDNnetworksperformances
THEPROACTIVECASE
01.Barrierrequest02.Setofnewrules
SDNController
THEPROACTIVECASE
Barrierreply01.Barrierrequest02.Setofnewrules
SDNnetworksperformances
SDNController
THEPROACTIVECASE
Barrierreply01.Barrierrequest02.Setofnewrules
HWdatapathSWcontroller
SDNnetworksperformances
SDNController
THEPROACTIVECASE
Barrierreply01.Barrierrequest02.Setofnewrules
HWdatapath
SWcontroller?
SDNnetworksperformances
Control/DataplaneconsistencyConsistentpolicyupdateaffectssecurityinSDN.
SW0Untrusted
Port1
Untrusted
SW1
SW2
Trusted
Port2
U->SW1T->SW2
SwitchController
Targetstateneededtoupdate
T->SW1U->SW2
OSNT
OSKernel
Application
NIC
0 1 2 3
ControlChannel
Control/Dataplaneconsistency
OSNT
OSKernel
Application
NIC
0 1 2 3
ControlChannel
1.Initialrule:0® 1
2.Ruleupdate :0® 2
(asasetofdifferentIPrules)
Control/Dataplaneconsistency
OSNT
OSKernel
Application
NIC
0 1 2 3
ControlChannel
1.Initialrule:0® 1
2.Ruleupdate :0® 2
(asasetofdifferentIPrules)
Wegenerateanaggregate2Gbpswith150Bpackets
Control/Dataplaneconsistency
Blueswitch:Enablingprovablyconsistentconfigurationofnetworkswitches,HanJ.H.,Mundkur P.,Rotsos C.,Antichi G.,DaveN.,MooreA.W.,NeumannP.G.,ACM/IEEEANCS,Oakland,CA,USA,7-8May,2015
Control/Dataplaneconsistency
Blueswitch:Enablingprovablyconsistentconfigurationofnetworkswitches,HanJ.H.,Mundkur P.,Rotsos C.,Antichi G.,DaveN.,MooreA.W.,NeumannP.G.,ACM/IEEEANCS,Oakland,CA,USA,7-8May,2015
TCAM+RAM
Control/Dataplaneconsistency
Blueswitch:Enablingprovablyconsistentconfigurationofnetworkswitches,HanJ.H.,Mundkur P.,Rotsos C.,Antichi G.,DaveN.,MooreA.W.,NeumannP.G.,ACM/IEEEANCS,Oakland,CA,USA,7-8May,2015
RAM
Control/Dataplaneconsistency
Blueswitch:Enablingprovablyconsistentconfigurationofnetworkswitches,HanJ.H.,Mundkur P.,Rotsos C.,Antichi G.,DaveN.,MooreA.W.,NeumannP.G.,ACM/IEEEANCS,Oakland,CA,USA,7-8May,2015
Control/Dataplaneconsistency
Blueswitch:Enablingprovablyconsistentconfigurationofnetworkswitches,HanJ.H.,Mundkur P.,Rotsos C.,Antichi G.,DaveN.,MooreA.W.,NeumannP.G.,ACM/IEEEANCS,Oakland,CA,USA,7-8May,2015
Control/Dataplaneconsistency
Aflowmodificationrequiresonlytochangethe“action”intheRAM.TheflowisalreadypresentintheTCAM.
Blueswitch:Enablingprovablyconsistentconfigurationofnetworkswitches,HanJ.H.,Mundkur P.,Rotsos C.,Antichi G.,DaveN.,MooreA.W.,NeumannP.G.,ACM/IEEEANCS,Oakland,CA,USA,7-8May,2015
Control/Dataplaneconsistency
AflowinsertionrequiresalsotowritethematchingfieldsintheTCAM.
Dataplane performances
0
5
10
15
20
0 200 400 600 800 1000
inse
rtio
n d
ela
y (s
ec)
number of flows
Pica8 OVSPica8 L2/L3
Force10
Flowadditiondelay
OFLOPS-Turbo:TestingtheNext-GenerationOpenFlow Switch,Rostos C.,Antichi G.,Bruyere M.,OwezarskiP.,MooreA.W.,IEEEICC,London,UK,8-12June,2015
Dataplane performances
0
5
10
15
20
0 200 400 600 800 1000
inse
rtio
n d
ela
y (s
ec)
number of flows
Pica8 OVSPica8 L2/L3
Force10
Flowadditiondelay
OFLOPS-Turbo:TestingtheNext-GenerationOpenFlow Switch,Rostos C.,Antichi G.,Bruyere M.,OwezarskiP.,MooreA.W.,IEEEICC,London,UK,8-12June,2015
0
0.5
1
1.5
2
2.5
3
0 200 400 600 800 1000
inse
rtio
n d
ela
y (s
ec)
number of flows
Pica8 OVSPica8 L2/L3
Force10
Dataplane performancesFlowmodificationdelay
OFLOPS-Turbo:TestingtheNext-GenerationOpenFlow Switch,Rostos C.,Antichi G.,Bruyere M.,OwezarskiP.,MooreA.W.,IEEEICC,London,UK,8-12June,2015
0
0.5
1
1.5
2
2.5
3
0 200 400 600 800 1000
inse
rtio
n d
ela
y (s
ec)
number of flows
Pica8 OVSPica8 L2/L3
Force10
Dataplane performancesFlowmodificationdelay
OFLOPS-Turbo:TestingtheNext-GenerationOpenFlow Switch,Rostos C.,Antichi G.,Bruyere M.,OwezarskiP.,MooreA.W.,IEEEICC,London,UK,8-12June,2015
SDNnetworksperformances
THEREACTIVECASE
SDNController
packet
THEREACTIVECASE
SDNController
packet
SDNnetworksperformances
THEREACTIVECASE
SDNController
packet
SDNnetworksperformances
THEREACTIVECASE
SDNController
packet modifiedpacket
SDNnetworksperformances
ControllerperformancesinSDNnetworks
Controlchannel
DataTrafficLink
OSNT
OSKernel
OFController OFSwitch
OFControllers
ControllerperformancesinSDNnetworks
OSNT-SUME-liveGithub§ OSNT-SUME-liveispubliclyavailable.
Nick McKeown, Glen Gibb, Jad Naous, David Erickson, G. Adam Covington, John W. Lockwood, Jianying Luo, Brandon Heller, Paul Hartke, Neda Beheshti, Sara Bolouki, James Zeng,
Jonathan Ellithorpe, Sachidanandan Sambandan, Eric Lo, Stephen Gabriel Ibanez
Acknowledgments(I)
NetFPGA Team at Stanford University (Past and Present):
NetFPGA Team at University of Cambridge (Past and Present):Andrew Moore, David Miller, Muhammad Shahbaz, Martin Zadnik, Matthew Grosvenor, Yury
Audzevich, Neelakandan Manihatty-Bojan, Georgina Kalogeridou, Jong Hun Han, Noa Zilberman, Gianni Antichi, Charalampos Rotsos, Hwanju Kim, Marco Forconesi, Jinyun Zhang,
Bjoern Zeeb, Robert Watson, Salvator Galea, Marcin Wojcik, Diana Andreea Popescu, Murali Ramanujam
All Community members (including but not limited to):Paul Rodman, Kumar Sanghvi, Wojciech A. Koszek, Yahsar Ganjali, Martin Labrecque, Jeff
Shafer, Eric Keller, Tatsuya Yabe, Bilal Anwer, Yashar Ganjali, Martin Labrecque, Lisa Donatini, Sergio Lopez-Buedo , Andreas Fiessler, Robert Soule, Pietro Bressana, Yuta Tokusashi
Patrick Lysaght, Kees Vissers, Michaela Blott, Shep Siegel, Cathal McCabe
Steve Wang, Erik Cengar, Michael Alexander, Sam Bobrowicz, Garrett Aufdemberg,Patrick Kane, Tom Weldon
Acknowledgements (II)