Towards Software Defined Persistent Memory

Towards Software-defined Persistent Memory:

Rethinking Software Support for Heterogeneous Memory Architectures

Swaminathan Sundararaman*

Nisha Talagala*Dhananjoy Das Amar Mudrankit*

Dulcardo Arteaga*

*Work done at Fusion-io/SanDisk

CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL 2

Memory-Storage Convergence (Trend 1)

L1, L2, L3 CPU Caches DRAM Hard Drives

MicrosecondsNanoseconds

CPU WAIT CYCLES

Tiered Memory Solutions

Main Memory System Storage Systems

Milliseconds

ACCESS DELAY

2 cycles 1,000,000s100s 1,000s 10,000s

chasm

2

FlashMemory

Persistent Memories

PM blurs the line between storage and memory


Challenges with Current Persistent Memory SolutionsAccess Granularity Byte (Memory) Block (I/O) Hybrid

Memory Technology PCM ReRAM/ Memristor

SRAM (backed by Cap.) NVDIMM

Capacity 1 - 100s GB 1 - 100s GB 32K – 2GB 4 – 32GB

Local Attach Point PCIe NVMe SAS DDR

Access Mechanism File System Object Store KV Store …

Memory Location Local Remote Replicated

Network Connect Infiniband Ethernet PCIe …

3

Many possible combinations!


• Rewrite applications for different deployments

▪ Not practical given the number of scenarios

• What about existing applications / deployments?

• User data is constantly growing and needs not fit in persistent memory

What Should Application Developers Do?

4


Moving Towards a Software-defined World… (Trend 2)

Software-Defined Networking (SDN)Enables administrators to manage network services

by abstracting higher level functionality

Abstraction of logical storage services and capabilities from the underlying physical storage systems

Software-Defined Storage (SDS) Software-Defined Flash (SDF)Abstract or expose flash specific details to enable software to

realize the raw bandwidth and storage capacity of Flash

Software-Defined Data Center (SDDC)All elements of the infrastructure such as networking, storage, CPU and security are virtualized and delivered as a service

5

Abstraction of logical storage services and capabilities from the underlying physical persistent memory hardware and interconnect

Software-Defined Persistent Memory (SDPM)


Our SDPM Solution• The first instance of a software defined approach to PM that can bring

the benefits of PM to a gamut of practical deployments.

• Abstract the heterogeneity in PM hardware from applications

• Provide file system API & programming libraries to access PM

• Use currently available PM hardware to show the feasibility of an SDPM

▪ PCIe & DDR4 attached PM (both local & remote attach)

▪ Using Infiniband & 10G Ethernet for remote access

• The prototype architecture provides good performance and near optimal acceleration for a range of local and remote PM configurations

6


• Introduction

• Design

• Evaluation

• Conclusion

Outline

7


• Support a variety of local and remote attach points with differing performance but identical functionality and semantic guarantees.

• Enable tiering of data between PM and flash, with caching in DRAM, to enable different cost/performance configurations

• Support hybrid (i.e., both memory and I/O) access, traditional storage management, and persistence guarantees to combine the best of memory and storage worlds.

• Enable a single application programming model to work across a variety of hardware

SDPM Design Goals

8


Software Defined Persistent-Memory Architecture

Applications

Programming Libraries

File System

Persistent MemoryManager

Block DeviceDriver

SDPM

PCIe/DDR

SAS/SATA/NVMe

PM

Infiniband/Ethernet/PCIe/…

Memory/Block (I/O)

• Persistent Memory Manager ▪ Abstracts hardware and interconnect

details from the file system / applications ▪ Exports APIs to guarantee persistence

• File System ▪ Unified persistent namespace to PM &

Flash ▪ Hybrid access to PM ▪ Transparent & non-transparent

acceleration

• Programming Libraries ▪ Unified access APIs to applications ▪ OS bypass for remote access


• A flash-optimized POSIX compliant Linux file system

▪ Extended NVMFS to support PM in addition to Flash

• Provides a unified and persistent namespace to both PM and flash

▪ Hybrid (memory & I/O) access to PM and applications can switch back and forth

▪ Transparent application acceleration (by tiering data between PM & flash)

▪ Supports “direct” mmap mode to directly map and use PM without caching in DRAM

▪ Supports a single programming model via a combination of application specific libraries over direct mmap and transparent access via POSIX APIs

Non-Volatile Memory File System (NVMFS)

10


Persistent Memory Manager: Hardware Abstractions• Memory Mapping Types

▪ PM can be mapped in multiple ways depending on the hardware. We need to ensure that each memory type is default mapped to the optimal model possible for its physical attach (by default map it to write combining)

▪ Enable FS operations that allow the app. to control the per file memory mapping.

• Guaranteeing Persistence

▪ We need mechanism(s) to guarantee all acknowledged in-flight data (such as in CPU caches, registers, etc.) have reached the PM device independent of its attach point

▪ PMM provides a barrier() operation (to NVMFS and user space libraries) that guarantees that all data is moved to the persistence domain as needed for the attach point

11


• Introduction

• Design

• Evaluation

• Conclusion

Outline

12


Evaluation - ConfigurationConfigurations Attach Method Local or Remote

Config-1 DDR NVDIMM LocalConfig-2 PCIe MMIO LocalConfig-3 DDR NVDIMM Remote EthernetConfig-4 PCIe MMIO Remote EthernetConfig-5 DDR NVDIMM Remote InfinibandConfig-6 PCIe MMIO Remote Infiniband

Config-1, Config-2(Local)

Config- 3, 4, 5, 6 (Remote)

System Configuration HP DL380, 96GB DDR DRAM, x86_64 Linux 3.14 kernel

MySQL Version Percona 5.5

Flash Device (PCIe) Fusion-io Gen 2 ioMemory 1.2TB

Persistent Memory PCI-e: ACM (512KB)DDR4: Viking NVDIMM (8GB)

Network Interconnect N/A Infiniband: ConnectX-3 56 Gbit IB Ethernet: Intel 82599ES 10-Gigabit

13


Evaluation - Setup

Host A(Source)MySQL

Programming Libraires

NVMFS PMM

ioMemory VSL

Replication Src

PM

SDPM

Local

Con

fig-2

Con

fig-1

Host B(Sink)

Programming Libraires

NVMFSPMM

ioMemory VSL

Replication Sink(daemon)

PM

SDPM

Con

fig-4

Con

fig-3

10Gbit Ethernet56Gbit Infiniband

Remote

14C

onfig

-5

Con

fig-6


Local PCIe MMIO Vs DDR4 (config-1 Vs Config-2) Ba

ndw

idth

(MB/

sec)

0

1000

2000

3000

4000

Size (bytes)64 128 256 512 1024 2048 4096

DDR-BW MMIO-BW

Late

ncy

(nse

c)

0

2000

4000

6000

8000

Size (bytes)64 128 256 512 1024 2048 4096

DDR-LatencyMMIO-Latency

15

Smaller data sizes -> barrier operation Larger data sizes -> transport media


Barrier Overheads: Local (config-1) Vs Remote (config-3 & config-5)La

tenc

y (u

s)

0

12500

25000

37500

50000

Synchronization Frequency (#Ops)1 10 100 1000

Ethernet FLUSH InfiniBand RDMA Local ACMPM

16

20x

4x

# of operations between synchronizations increases, the performance becomes closer

CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL

Tran

s Pe

r Sec

ond

(x10

00)

0

15,000

30,000

45,000

60,000

REMOTE LOCAL VANILLA NO_LOG

Bin & Tr Log Tr log ONLY

MySQL: Insert Heavy Workload

17

Config-3 Config-1 on Flash Infinitely fast logging


Ope

ratio

ns/s

ec (i

n x1

000)

0

7500

15000

22500

30000

MySQL: LinkBench 10x Workload

18

No Logs Vanilla Local(config-1)

Remote(config-5)

31%17%

Facebook’s social graph workload

10x: 100 million nodes

Inserts, delete, update, and lookup

30% writes (insert/update/delete)

70% reads (lookup)

Infiniband performance for small updates is sufficient

3%


• Introduction

• Design

• Evaluation

• Conclusion

Outline

19


Conclusions

• PM is going to change the storage-memory landscape • Many different forms / capacities / attach points / performance

• SDPM: a software-defined approach to using persistent memory

• Abstracts heterogeneity in memory hardware

• Applications can transparently run on local & remote persistent memory

• Selectively abstracts PM characteristics to provide optimal performance

• Transparently tier data between PM & Flash to exploit both existing & new applications written to run on PM.

• Our evaluation shows near optimal performance for local & remote attach PM

20


Thank You

Iif you are interested in trying out SDPM: [email protected]

Towards Software Defined Persistent Memory

Technology

Transcript of Towards Software Defined Persistent Memory