Ally: OS-Transparent Packet Inspection Using Sequestered Cores
-
Upload
mara-randolph -
Category
Documents
-
view
26 -
download
0
description
Transcript of Ally: OS-Transparent Packet Inspection Using Sequestered Cores
1
Ally: OS-Transparent Packet Inspection Using Sequestered Cores
Jen-Cheng Huang 1, Matteo Monchiero2, Yoshio Turner3, Hsien-Hsin
Lee1
1Georgia Tech 2Intel Labs 3HP Labs
2
Deep Packet Inspection (DPI)
Data Center
MiddleBoxes
Intrusion Detection
ContentInsertion
TrafficClassification
Internet
Deployment of Packet Processing Services
3
Problem
Internet
Data Center
MiddleBoxes
Local Traffic is growing in importance…
But The traffic within the data center is not inspected!
4
Approach
“Co-locate” DPI with the server
DPI appliance
Server
Leverage abundant CPU resources
Leverage existing management interfaces on servers, e.g. HP iLO
Compatible with heterogeneous architecture, e.g. on-chip accelerators
5
Requirements
• Transparency– Independent to the server’s software stack
• Efficiency– Low overhead packet interception
• Isolation– Resistant to attacks
6
ETTM: a scalable fault tolerant network manager. C. Dixon et al. NSDI ‘11
Related Work
Transparency
Hypervisor Overhead
Hypervisor Vulnerability
Virtualization Support for DPI deployment
Hypervisor
DPI VM
Guest VM
HW
SW
Virtualized Platform
Processors
7
NIC
Unprivileged Partition
Multi-core processor
core
Privileged Partition
Ally Architecture
Software Stack (OS + Applications)
Software Stack (DPI Application)
core core core core core
NIC Traffic
8
Outline
• Introduction & Motivation• Architecture
– Overview– Multicore Partitioning– Packet interception
• Evaluation• Conclusions
9
Northbridge
MMUMMU
MMUMMU
ServiceProcessor
NIC
Memory Controlle
r
Interconnect
IOMMUInterrupt
Unit
BIOS
ExternalNetwork
Main Memory
Core
Inte
rrup
tC
ontr
olle
r
MMU
Core
Inte
rrup
tC
ontr
olle
r
MMU
LastLevelCach
e
Management Network
Baseline Architecture
10
Northbridge
MMUMMU
MMUMMU
ServiceProcessor
NIC
Memory Controlle
r
Interconnect
IOMMUInterrupt
Unit
BIOS
ExternalNetwork
Main Memory
Core
Inte
rrup
tC
ontr
olle
r
MMU
Core
Inte
rrup
tC
ontr
olle
r
MMU
LastLevelCach
e
Management Network
Ally ArchitectureUnprivileged
partitionPrivileged partition
11
Outline
• Introduction & Motivation• Architecture
– Overview– Multicore Partitioning– Packet interception
• Evaluation• Conclusions
12
Multicore Partitioning
NIC
Unprivileged Partition
Multi-core processor
core
Privileged Partition
Software Stack (OS + Applications)
Software Stack (DPI Application)
core core core core core
Invisible
13
Core SequestrationModify the BIOS to hide privileged core information from the OS
BSP core - the first core that boots AP cores - the other cores IPI - Inter-processor interrupts
OS retrieves cores information
BSPAP
AP
AP
Core InfoTable
Wakeup IPI Update
Ally Booting Procedure:
AP DPIEngine DPI core waits for
IN/OUT packetsInitialize
…...
14
Memory Protection
TLB
TLB Miss Handler
CR3BoundaryRegister
Page Table
Range Checking
Main Memory
Privileged partition
Unprivileged partition
MMUUnprivileged Core
Partition the memory into two physically contiguous regions
TLB Miss
TLB Fill
15
Outline
• Introduction & Motivation• Architecture
– Overview– Multicore Partitioning– Packet interception
• Evaluation• Conclusions
16
NIC
Unprivileged Partition
Multi-core processor
core
Privileged Partition
Packet Interception
Software Stack (OS + Applications)
Software Stack (DPI Application)
core core core core core
NIC Traffic
17
Packet InterceptionVirtualization of the Descriptor Queues
NIC
OS memory
Descriptor queues replicated
DPI memory
Only one copy of the packet buffers
Descriptor queues
18
Packet Interception
• Virtualization of the Descriptor Queues– Device independent, software independent– No copying on packet buffers
• Processor and NIC communication– Queue manipulation uses Memory Mapped IO (MMIO)
accesses– NIC event notification uses Interrupt
19
MMIO redirection
MMU
MMU detects specific MMIO addresses
MMU redirects RW to a reserved region in DPI memory
MMU sends IPI to DPI core
DPI memory
DPI core
OS core
IPI
R/W redirection
Load/store
20
Ally Hardware Properties
• Simple extensions to existing hardware components
• No impact expected on critical timing paths
• Compatible with virtualization support (Intel VT-x/EPT, AMD SVM/NPT)
21
Outline
• Introduction & Motivation• Architecture
– Overview– Multicore Partitioning– Packet interception
• Evaluation• Conclusions
22
Evaluation
Full system emulationQEMU
Core sequestration
HW changes
Real machine prototype Hardware– Intel Core 2 duo 2.66 GHz with 1 Gbit Intel NICBenchmarks– Netperf– SPECwebSystems– Ally, Linux and Xen
23
System Configurations
Queue Virtualization
NIC Driver
Kernel
Netperf/Specweb
Snort
DPI core OS coreHW
SW NIC Driver
Kernel
Netperf/Specweb
Snort
DPI core OS coreHW
SW
IP queue
Ally Linux
24
System Configurations
Hypervisor
Dom0 Kernel
Netperf/Specweb
Snort
DPI core OS coreHW
SW
Xen
DomUKernel
25
Netperf CPU Usage
26
SPECweb CPU Usagecy
cles
/req
ues
t *
10
6
27
Outline
• Introduction & Motivation• Architecture
– Overview– Multicore Partitioning– Packet interception
• Evaluation• Conclusions
28
Conclusions
Ally: a framework for transparent deployment of packet inspection appliances
Ally uses a set of simple HW/FW extensions enable reliable multicore partitioning and efficient packet inspection
Ally is fully compatible with new virtualization technology as well as heterogeneous architecture
29
Thanks
30
Throughput
31
DPI using Network Processor
32
NIC
Unprivileged Partition
Multi-core processor
core
Conventional Architecture
Software Stack (OS + Applications)
core core cores cores cores
33
NIC
Unprivileged Partition
Multi-core processor
core
Privileged Partition
Transmission Path
Software Stack (OS + Applications)
Software Stack (DPI Application)
core core core core core
34
NIC
Unprivileged Partition
Multi-core processor
core
Privileged Partition
Receive Path
Software Stack (OS + Applications)
Software Stack (DPI Application)
core core core core core
35
Integrated Northbridge
DPI core
Loca
l A
PIC
MMU
Interface
DPI core
Loca
l A
PIC
MMU
Interface
OS core
Loca
l A
PIC
MMU
Interface
OS core
Loca
l A
PIC
MMU
Interface
Platform Controller Hub
NIC
Memory Controll
er
On chip interconnect
Processor
IOMMUPCIe ctrlInterrupt
Unit
BIOSNetwork
Main Memory
DMI Ctrl
OS core
Loca
l A
PIC
MMU
Interface
Unprivileged partition Privileged partition
DPI core
Loca
l A
PIC
MMU
Interface
Last Level Cache
IOAPIC
Management NIC
Service Processo
r
Management Network
Privileged partition
Unprivileged partition
36
Integrated Northbridge
DPI core
Loca
l A
PIC
MMU
Interface
DPI core
Loca
l A
PIC
MMU
Interface
OS core
Loca
l A
PIC
MMU
Interface
OS core
Loca
l A
PIC
MMU
Interface
Platform Controller Hub
NIC
Memory Controll
er
On chip interconnect
Processor
IOMMUPCIe ctrlInterrupt
Unit
BIOSNetwork
Main Memory
DMI Ctrl
OS core
Loca
l A
PIC
MMU
Interface
Unprivileged partition Privileged partition
DPI core
Loca
l A
PIC
MMU
Interface
Last Level Cache
IOAPIC
Management NIC
Service Processo
r
Management Network
Privileged partition
Unprivileged partition
37
MMU Modification – Memory Protection
TLB
TLB Miss Handler
CR3Special_re
g
Page Table DPI core boundary register
phys_addr >
special_reg ?
Main Memory
Privileged partition
Unprivileged partition
38
Memory Protection Procedure
TLB
TLB Miss HandlerTLB miss
Virtual Address
CR3Special_re
g
Page Table DPI core boundary register
phys_addr >
special_reg ?
Main Memory
Privileged partition
Unprivileged partition
39
Memory Protection Procedure
TLB
TLB Miss HandlerTLB miss
Virtual Address
TLB fill
CR3Special_re
g
Page Table DPI core boundary register
phys_addr >
special_reg ?
Main Memory
Privileged partition
Unprivileged partition
40
NIC
Unprivileged Partition
Multi-core processor
core
Privileged Partition
Memory Protection
Software Stack (OS + Applications)
Software Stack (DPI Application)
core core core core core
Invisible
41
Integrated Northbridge
DPI core
Loca
l A
PIC
MMU
Interface
DPI core
Loca
l A
PIC
MMU
Interface
OS core
Loca
l A
PIC
MMU
Interface
OS core
Loca
l A
PIC
MMU
Interface
Platform Controller Hub
NIC
Memory Controll
er
On chip interconnect
Processor
IOMMUPCIe ctrlInterrupt
Unit
BIOSNetwork
Main Memory
DMI Ctrl
OS core
Loca
l A
PIC
MMU
Interface
Unprivileged partition Privileged partition
DPI core
Loca
l A
PIC
MMU
Interface
Last Level Cache
IOAPIC
Management NIC
Service Processo
r
Management Network
Privileged partition
Unprivileged partition
42
Integrated Northbridge
DPI core
Loca
l A
PIC
MMU
Interface
DPI core
Loca
l A
PIC
MMU
Interface
OS core
Loca
l A
PIC
MMU
Interface
OS core
Loca
l A
PIC
MMU
Interface
Platform Controller Hub
NIC
Memory Controll
er
On chip interconnect
Processor
IOMMUPCIe ctrlInterrupt
Unit
BIOSNetwork
Main Memory
DMI Ctrl
OS core
Loca
l A
PIC
MMU
Interface
Unprivileged partition Privileged partition
DPI core
Loca
l A
PIC
MMU
Interface
Last Level Cache
IOAPIC
Management NIC
Service Processo
r
Management Network
Privileged partition
Unprivileged partition
43
MMU Modification – MMIO Redirection
TLB
Redirection BitPhysical Page
TLB Miss HandlerCheck
uncacheable address map
Redirection
Table
Physical Address
Remapped Address
44
MMIO Redirection – TLB Miss
• On a TLB miss, the TLB miss handler does the page table walk
TLB
Redirection BitPhysical Page
Virtual Address
TLB missTLB Miss Handler Page Table Lookup
45
MMIO Redirection – TLB Miss
• The TMH checks if the resulting physical address falls in an uncacheable page and hence potentially a MMIO page
TLB
Redirection BitPhysical Page
TLB Miss Handler Physical AddressCheck
uncacheable address map
46
MMIO Redirection – TLB Miss
• If the page is uncacheable, the TMH looks up the redirection table to check if any address in this page needs to be redirected
TLB
Redirection BitPhysical Page
TLB Miss HandlerCheck
uncacheable address map
Redirection
Table
Physical Address
Remapped Address
Physical Address
47
MMIO Redirection – TLB Miss
• If any address in the page needs to be redirected, the TMH sets the redirection bit in addition to fill the TLB
TLB
Redirection BitPhysical Page
TLB Miss HandlerCheck
uncacheable address map
TLB fill
Redirection
Table
Physical Address
Remapped Address
48
MMIO Redirection – TLB Hit
• On a TLB hit, if the redirection bit is set, the MMU looks up the Last Level Cache (LLC) used to cache translations in Redirection Table
TLB
Redirection Bit
Physical Page
Offset
Physical Address
Virtual Address
LLC
Physical Address
Remapped Address
49
MMIO Redirection – TLB Hit
• If a translation is found, the MMU returns the translated address and sends IPI to privileged cores.
TLB
Redirection Bit
Physical Page
LLC
Physical Address
Remapped Address
Translated Address
Generate IPI
Physical Address
Hit
50
MMIO Redirection – TLB Hit
• If the LLC misses, then Redirection Table Lookup is performed
TLB
Redirection Bit
Physical Page
LLC
Physical Address
Remapped Address
Redirection Table LookupPhysical
AddressMiss
51
Interrupt Unit Modification
DPI core
OS core
Interrupt Unit
NIC
If Source == NIC, Redirect Interrupt
52
• When NIC raises an interrupt, The interrupt Unit redirects the interrupt to DPI core
Interrupt Redirection
DPI core
OS core
Interrupt Unit
NIC
If Source == NIC, Redirect Interrupt
Interrupt
53
• After the NIC interrupt is handled, DPI core sends an IPI to OS core mimicking NIC interrupt
Interrupt Redirection
DPI core
OS core
Interrupt Unit
NIC
If Source == NIC, Redirect Interrupt
IPI
54
Summary of Hardware Modifications
Unit Description Purpose
OS-core MMU
Prevent memory accesses to DPI memory from OS-core
Protection
Redirect MMIO accesses to DPI memory from OS-core and interrupt DPI core
Packet Interception
IOMMU Prevent non authorized DMA to DPI Memory Protection
IOAPIC Redirect NIC interrupts to DPI-core Packet Interception
All Units Protected configuration registers Protection
55
Functional Evaluation
Full system emulation• QEMU• Validate Hardware and Firmware Changes
56
DPI core Usage
57
SPECweb Cache Misses
58
NIC
Unprivileged Partition
Multi-core processor
core
Privileged Partition
Memory Protection
Software Stack (OS + Applications)
Software Stack (DPI Application)
core core core core core
Invisible
How?Modified MMU
59
Challenges
- Make privileged partition protected and invisible from the unprivileged partition- Core Sequestration- Memory Protection
- Intercept packets efficiently- Packet Interception
60
Ally System
NIC
Linux
kernel
NIC Traffic
Queue Virtualization
NIC Driver
Other Apps
Snort
DPICore
Core
61
Linux System
NIC
Linux
kernel
NIC Traffic
IP queue
NIC Driver
Other Apps
Snort
Core Core
62
Xen System
NIC
Linux
VM #0
NIC Traffic
IP queue
Hypervisor
Other Apps
Snort
Core Core
VM #1