A Fast and Flexible Hardware-based Virtualization ...
Transcript of A Fast and Flexible Hardware-based Virtualization ...
A Fast and Flexible Hardware-based Virtualization
Mechanism for Computational Storage Devices
Dongup Kwon, Dongryeong Kim, Junehyuk Boo, Wonsik Lee, and Jangwoo Kim
Department of Electrical and Computer Engineering, Seoul National University
Background: Computational Storage
• SSD-FPGA integration for near-storage processing
− Fast data transfers between the storage and computation units
− Programmable operators and on-chip interconnects in an FPGA
NVMeSSD
FPGA
PCIeSwitch
DRAM DRAM Controller
SSD Controller
On-chipSwitch
Operators
StorageDevice
Computational storage = SSD + FPGA + near-storage processing
2/21
Background: I/O Virtualization
• SW-based virtualization: Paravirtualization (VirtIO)
• HW-assisted virtualization: Passthrough, SR-IOV, FVM*
*FVM: FPGA-assisted Virtual Device Emulation for Fast, Scalable, and Flexible Storage Virtualization, OSDI 2020
I/O virtualization enables resource sharing between VMs.
VMVirtio Front-end Driver
HostSW
Virtio Back-end Driver
HW
Linux Kernel
KVM/QEMU
SSD FPGA
HW SSD FPGA
SR-IOV SR-IOV
IOMMU
Paravirtualization
VMSSD/FPGA Driver
Linux Kernel
SR-IOV
3/21
Outline
• Background
• Motivation
− SW-based virtualization for computational storage
• FlexCSV: HW-assisted Virtualization Stack
• Evaluation
• Conclusion4/21
SW-based Virtualization Approach
• SW emulation of SSD-FPGA integrated devices
• Host SW-level device resource allocation and scheduling
SW-based virtualization provides flexible virtual device construction mechanisms.
SSD FPGA
PCIeSwitch
VM
Device EmulationHostSW
HW
User Application
SSD FPGA
Resource Allocation
VM’s view of the virtual devices
5/21
Limitation #1:CPU-centric Device Emulation
• CPU-centric device orchestration & data transfers
• Cannot achieve full potential of near-storage processing
The bottleneck shifts to the SW components in a virtualized environment.
Percentage of Execution Time
SW HW(SSD+FPGA)
Hash
Filter
Grep
100%50%0%
VM
Device EmulationHostSW
HW
User Application
SSD FPGA
Resource Allocation
6/21
Limitation #2:Static Resource Allocation
• Static VM-to-HW resource allocation & scheduling
• Cannot achieve cost-effectiveness due to inefficientresource sharing
Static resource allocation incurs extra costs for the additional HW resources to meet QoS requirements.
.
FPGA
SSD
Operator
Operator
OperatorVMs
StaticVM-to-HW mapping
StorageDevice
7/21
Limitation #3:Coupled HW Architecture
• SSD-FPGA coupled designs & fixed provisioning
• Cannot provide flexible device/resource configurations
FPGA
SSD
StorageDevice
FPGA
SSD
StorageDevice
SSD-FPGA coupled architectures suffer from limited device scalability.
Operator
Operator
Operator
Operator
FPGA BW > SSD BW FPGA capacity < SSD capacity
Operator
Operator
Operator
Operator
8/21
Design Goals
Device Sharing
Device Scalability
High Performance
SW-basedVirtualization
Low Cost
Trap-and-emulate
Tightly-coupled architecture
CPU-centric orchestration
Static resource allocation
Design Goals
9/21
Outline
• Background
• Motivation
• FlexCSV: HW-assisted Virtualization Stack
• Evaluation
• Conclusion
10/21
FlexCSV: SW/HW Architecture
• HW virtualization for computational storage
VM
HW
User Application
SSD
FlexCSV Engine
IOMMU
SSD FPGA SSD/DRAM Controllers
Switch
FlexCSV Engine (FPGA)
SR-IOV
Device Emulation
Resource Allocation
Operators
FlexCSV offloads a virtualization stack for computational storage devices.
11/21
FlexCSV: Key Ideas
FlexCSV
HW-assisted virtualization(including SR-IOV)
SSD-FPGA decoupled architecture
HW-level resource orchestration
Dynamic resource allocation
Device Sharing
Device Scalability
High Performance
Low Cost
Design Goals Key Ideas
12/21
Key Idea #1:HW-assisted Virtualization
• SR-IOV implementation in FlexCSV Engine
• SSD/FPGA sharing between VMs with direct HW access
FlexCSV Engine virtualizes itself through SR-IOV and offers device sharing.
FlexCSV Engine
⋯
Host OS VM 0 VM 1 VM n⋯
PF VF 0 VF 1 VF n
Virtualization Layer
FPGA FPGASSD SSDSSD
13/21
Key Idea #2:HW-level Orchestration
• NVMe extension for data processing requests
• Guest/host OS bypassing and direct data communications
FlexCSV Engine orchestrates SSD and FPGA operations without SW arbitration.
struct nvme {
u8 rw;
u64 rsvd;
u64 prp1;
u64 prp2;
u64 slba;
}
struct proc {
u64 op;
u64 src;
u64 dst;
u32 size;
u64 param;
} DRAM
SSD
Switch
Operators
FPGA
FlexCSVEngine
1
23
456
SSD ctrl
FPGActrl
SQ CQ
14/21
Key Idea #3:Dynamic HW Allocation
• Renaming of user-requested HW resources
• Efficient use of HW resources – High HW utilization
FlexCSV Engine implements HW renaming logic for dynamic resource allocation.
FPGA
Operator
Operator
Operator
VMs
FlexCSV Engine
ResourceAllocation
OperatorRenaming
SSD
15/21
FPGA
Key Idea #4: SSD-FPGA Decoupled Architecture
• Decoupled HW through board-level PCIe switches
• Scalable virtual devices with many PCIe-attached cards
FlexCSV Engine provides scalable and flexible device/resource configurations.
FlexCSVEngine
VM’s view HW implementation
PCIeSwitch
FPGASSD FPGASSDSSDFPGAFlexCSV
EngineFPGASSDSSD
SwitchNetwork
16/21
FlexCSV Prototype
• HW Prototype
−Supermicro Server 4029GP-TRT2
− Intel Optane 900P SSDs
− Xilinx U250 FPGA (FlexCSV Engine)
• SW Frameworks
− Ubuntu / Linux kernel v5.3
− KVM / QEMU v3.0
FlexCSV prototype is built on off-the-shelf HW devices and open-source SW.
Intel Xeon Gold 5118
XilinxAlveo U250
Intel Optane900P SSDs
17/21
Outline
• Background
• Motivation
• FlexCSV: HW-assisted Virtualization Stack
• Evaluation
• Conclusion
18/21
Near-storage Processing Performance
• 8 FPGA benchmarks with direct SSD read & write
• Guest/host OS bypassing + fast data copy ➔ 2.4x speedup
XXX
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Encrypt Decrypt Hash Aggregate Filter Grep KNN Bitmap
Read-and-Write
Speedup
Full SW FlexCSV (SW)Full SW SW + P2P FlexCSV Higher is better
FlexCSV achieves high performance through its HW-assisted virtualization.
19/21
QoS Evaluation with Oversubscription
• 2 operators + 4 VMs with different request rates
• Dynamic allocation + partial reconfiguration ➔ 2.4x better
0.0
0.2
0.4
0.6
0.8
1.0
0.54
14
24
44
84
QoS V
iola
tion R
atio
Static (worst) Static (best) Dynamic Dynamic + PR
LowHigh FPGA Resource Contention
FlexCSV achieves lower QoS violations through its efficient HW resource use.
Lower is better
20/21
Thank You!
A Fast and Flexible Hardware-based Virtualization Mechanism for Computational Storage Devices, ATC 2021
Dongup Kwon, [email protected], https://hpcs.snu.ac.kr/~dongup/
Intel Xeon Gold 5118
XilinxAlveo U250
Intel Optane900p SSDs SSD/DRAM Controllers
Switch
FlexCSV Engine (FPGA)
SR-IOV
Device Emulation
Resource Allocation
Operators
21/21