SAY-Go: Towards Transparent and Seamless Storage-As-You-Go ... · 1. File system with dynamic...

SAY-Go:TowardsTransparentandSeamlessStorage-As-You-GowithPersistentMemory

Hyeonho Song,SamH.NohUNIST

HotStorage2018

§ PersistentMemory

§ Motivation

§ SAY-Go• Design• Implementation

§ Evaluation

§ Summary

§ Futurework

Contents

1

Persistentmemory

PRAM

3DXPoint STT-MRAM

MemoryStorage

• Non-volatility• Byte-levelrandomaccess• Fastaccesstime(nanoseconds)

PersistentMemoryFeatures

2

Persistentmemory

Intel®Optane SSDDCP4800X

Intel®OptaneMemory

DIMMBasedOptane memory

1H’2017 Comingsoon2H’20181H’2017

Blockdevice Blockdevice Characterdevice

3

Persistentmemory


Intel®OptaneMemory



Blockdevice Blockdevice Characterdevice

4

Ourtarget

Persistentmemory

CPU

DRAM

PM

CPU

DRAM

traditionalHDD,SSD

PM(cache)


Intel®OptaneMemory



5

CPU

DRAM

traditionalHDD,SSD

PM

Motivation

SFS

VFS

Devicedrivermemorymanager

Virtualmemorymanager

DRAM storage

6

Applicationsdemandmoreandmorememory

Motivation

SFS

VFS



fullyusedused

free

7

DRAMcapacityneverenough!

storageisrelativelyroomyduetohighdensity

Traditionalsolution

SFS

VFS



fullyusedused

free

8

DRAMcapacityisnotenough!

Swapmechanismwithstorage

§ WhenDRAMrunsout

Swapmechanism

Mainmemory

DRAM

Storagefullyused swap

area freeblocks

Userapplication

9

traditionalHDD

pageallocationisnotpossible

datasave(copy)toswaparea

§ SecurefreespaceinDRAM

Swapmechanism: swap-out

Mainmemory

DRAM

Storagefullyused swap

area freeblocks

Userapplication

10

traditionalHDD

Mainmemory

§ SecurefreespaceinDRAM

Swapmechanism: swap-out

DRAM

Storagenotfull swap

area freeblocks

Userapplication

11

traditionalHDD

pageallocationispossible

§ Whenapplicationwantstoaccessdatainswap area

Swapmechanism: swap-in

Mainmemory

DRAM

Storagenotfull swap

area freeblocks

Userapplication

12

traditionalHDD

§ Whenapplicationwantstoaccessdatainswaparea

Swapmechanism:swap-in

Mainmemory

DRAM

Storagenotfull swap

area freeblocks

Userapplication

13

traditionalHDD

datarestore(copy)tofreespaceinDRAM

blockdeviceisnotdirectlyaccessiblebytheCPU

§ Whenapplicationwantstoaccessdatainswaparea

Swapmechanism:swap-in

Mainmemory

DRAM

Storagenotfull swap

area freeblocks

Userapplication

14

traditionalHDD

accesstodata

§ Largeoverheadwithslowstorage

• Datacopyisessential

• Selectingpagestobeswappedout

isacomplexprocess

§ Swapareacannotbeusedasstoragespace

• Staticpartitionoffilesystem

Weaknessoftraditionalswapmechanism

15

Mainmemory

DRAM

Storagenotfull swap

area freeblocks

Userapplication

traditionalHDD

swap-out

swap-in

Simplesolution:swapwithPMstorage

SFS

VFS



fullyused

16

DIMMBased PM

§ PMperformanceisbetterthantraditionalstorage• Reduceddatacopytime• Overheadoftheswapmechanismisalsoreduced

§ Doesthissuffice?

Thereshouldbeabetterway?

SFS

VFS



fullyused

17

DIMMBased PM

§ PMstoragecanbeaccesseddirectlyfromtheCPU

§ PMasbothmainmemoryandstorage• DynamicallyusePM• Increasecapacityoftheworkingmemorywhenneeded

§ Dynamicadjustmentofthememoryandstorageboundary

OurGoal

User

DRAM traditionalHDD,SSD

18

SFS

VFS



User

DRAM

SFS

VFS

Devicedriver


PM

memorymanager

SAY-Gosystem(Storage-As-You-Go)

dynamicusageofPM

§ Dynamicadjustmentofthememoryandstorageboundary

OurGoal

User

DRAM traditionalHDD,SSD

19

SFS

VFS



User

DRAM

SFS

VFS

Devicedriver


PM

memorymanager

SAY-Gosystem(Storage-As-You-Go)

dynamicusageofPM

similarattemptswereproposed

§ DynamicusageofPM

Relatedwork

20

User

DRAM

SFS

VFS

Devicedriver


PM

memorymanager 1) Jung,J.Y.andCho,S.“Memorage:Emergingpersistentrambasedmalleablemain

memoryandstoragearchitecture.” InProceedingsoftheInternationalACMConference

onInternationalConferenceonSupercomputing(ICS)(2013).

2) Kannan,S.,Ada,G.,andKarsten,S.“pVM:persistentvirtualmemoryforefficientcapacity

scalingandobjectstorage.”InProceedingsoftheEuropeanConferenceonComputer

Systems(EuroSys)(2016).

Memorage1) – ICS2013: Workingmemoryexpansion

pVM2) – EuroSys 2016:DirectallocationofPMpagetoworkingmemory

§ DynamicusageofPM

Relatedwork

21

User

DRAM

SFS

VFS

Devicedriver


PM

memorymanager



1) Jung,J.Y.andCho,S.“Memorage:Emergingpersistentrambasedmalleablemain






§ WhenDRAMrunsout

Memorage

22

Mainmemory

DRAM

fullyused

Userapplicationpageallocationisnotpossible

Storage

freeblocks

PM

§ Workingmemoryexpansion

Memorage

23

Mainmemory

DRAM

Storagefullyused freeblocks

Userapplication

PM

1.selectfreeblockstobeusedinfilesystem


Memorage

24

Mainmemory

DRAM


Userapplication

PM

2.blocksaretransformedintoallocatablefreepagesbyreorganizingthemasdatastructuresusedintheOSmemorymanagementlayer


expand


Memorage

25

Mainmemory

DRAM


Userapplication

PM

3. setthatregiontoadditionalmemoryusinghot-plugfeatureofOS

2.blocksaretransformedintoallocatablefreepagesbyreorganizingthemasdatastructuresusedintheOSmemorymanagementlayer


expand


Memorage

26

Mainmemory

DRAM


Userapplication

PM

pageallocationbecomespossible

§ Largeruntimeoverhead• Processofchangingfreeblocks

intopagesproceedsatruntime

§ Memoryhot-plugfeatureisrequired

§ Overlookstheissueofconsistency• Consistencyofpageallocatorisnotconsidered• Memoryleakcanoccurwithfaults

WeaknessofMemorage

27

blocks->pages

§ DynamicusageofPM

Relatedwork

28

User

DRAM

SFS

VFS

Devicedriver


PM

memorymanager



1) Jung,J.Y.andCho,S.“Memorage:Emergingpersistentrambasedmalleablemain






§ WhenDRAMrunsout

pVM

Mainmemory

DRAM

Persistentstorefullyused freePMpages

Userapplication

29

pageallocationisnotpossible

PM

§ WhenDRAMrunsout

pVM

Mainmemory

DRAM

Persistentstorefullyused freePMpages

Userapplication

30

directallocation- npmalloc()- nvmmap(PERSISTorNOPERSIST)

PM

§ Requiresapplicationmodification

• Explicitlibrary functioncallsneedtobemade

− npmalloc(),nvmmap()

§ Similarapproachtopreviouswork• NV-Heap

• Mnemosyne

• NVMDuet

WeaknessofpVM

31

directallocation

Comparisonofrelatedworks

32

Swap Memorage pVM

Goal freeupworkingmemoryspace

memoryexpansion

persistentstore

Memory-Storagedivision fixed fixed fixed

Consistency notconsidered notconsidered yes

Transparency yes yes no

Runtimeoverhead yes yes no

Direct accesstodata no yes yes

strengthweakness

SAY-Godesigngoals

33

PM

User

SFS

VFS



DRAM

Dynamicpartition

• Seamlessintegration- PMroleisdynamicallyadjustedasneedbe

• Transparency- Automaticscalingofworkingmemory- Withoutmodificationoftheapplication

• Consistency- EnsureconsistencyofthePMallocator- Preventmemoryleaks

forworkingmemory forstorage

§ Consistsoftwoparts

SAY-Gocomponents

34

Filesystemwithdynamicpartitioning


2. Integratedmemorymanager- Pageallocatorforbothworkingmemoryandstorage- Ensureconsistencyofmemoryallocator

1. Filesystemwithdynamicpartitionsupport- Supportdynamiclayout- Regionmanagedinpage unit

DRAM PM

Integratedmemorymanagerforworkingmemory forstorage


Layout

35



DRAM PM


DynamicPartitionFileSystem(DPFS), inprogress



PersistentMemoryBuddy(PMB), HotStorage ‘18


Layout

36


DRAM PM



PersistentMemoryBuddy(PMB), HotStorage ‘18

ImplementedinLinux

DynamicPartitionFileSystem(DPFS), inprogress



PMBimplementationfeatures

37

PersistentMemoryBuddy(PMB)Filesystemwithdynamicpartitioning


DRAM PM

forworkingmemoryPMB

forstorage

• Seamlessintegration- PMroleisdynamicallyadjustedasneedbe

• Transparency- Automaticscalingofworkingmemory- Withoutmodificationoftheapplication

• Consistency- EnsureconsistencyofthePMallocator- Preventmemoryleaks

PMBdesignedwithSAY-Gogoalinmind

PMBLayoutinLinux

38

• DRAMandPMaremanagedinZONEunits- ZONE_NORMAL- plus2newZONEs*ZONE_MIGRATEand ZONE_STRG

• EachZONEhasfreepages- buddymemoryallocator

VMmanager

ProcessesDPFS

Devicedriver

VFS

ZONEMIGRATE

ZONESTRG

ZONENORMAL

PM pagesDRAMpages

PMB

forstorageforworkingmemory

§ CharacteristicsinLinux

• Unsorted

• Circular linkedlist

• Pushandpopoperation

occursonlyattheendofthelist

• Pushandpopoperations

areimplementedbyatomicoperations

Buddymemoryallocator

ZONE

ZONE

ZONE

...

order(10)Free_list

order(1)order(0)

order(2)

...

page

page

page4KB

4MB

...

page

page

4MB

...

page

page

4MB

page4KB

39

PMBbasedonbuddymemoryallocator

PMBLayoutinLinux

40

VMmanager

ProcessesDPFS

Devicedriver

VFS

ZONEMIGRATE

ZONESTRG

ZONENORMAL

PM pagesDRAMpages

PMB


• DRAMandPMaremanagedinZONEunits- ZONE_NORMAL- plus2newZONEs*ZONE_MIGRATEand ZONE_STRG

• EachZONEhasfreepages- buddymemoryallocator

• EachZONEhasdifferentrole

RoleofeachZONE

41

• ZONE_NORMAL- DRAMiscoveredbythisZONE

• ZONE_STRG- Minimumstoragearea- Persistentmetadatasavedtothisarea- Persistentpagedescriptor- Bitmapthatmaintainstheallocationstate- LogtableforPMBconsistency

VMmanager

ProcessesDPFS

Devicedriver

VFS

ZONEMIGRATE

ZONESTRG

ZONENORMAL

PM pagesDRAMpages

PMB


RoleofeachZONE

42

• ZONE_MIGRATE- KeycomponentinPMB- Eitherallocateddependingonitsuse

*aspersistent(storage)page*asnon-persistent(memory)page

VMmanager

ProcessesDPFS

Devicedriver

VFS

ZONEMIGRATE

ZONESTRG

ZONENORMAL

PM pagesDRAMpages

PMB


Pageallocation

43

• ZONE_NORMAL- Allocateworkingmemory

• StorageallocatedfromZONE_STRG- Allocatestorage

• ZONE_MIGRATE- AllocatePMpagestobothzones

- Allocationdoneusingtheexistinginterface

- Allocationdonebykernelthreadinthebackground

VMmanager

ProcessesDPFS

Devicedriver

VFS

PM pagesDRAMpages

PMB

ZONEMIGRATE

ZONESTRG

ZONENORMAL


ZONEcapacity

44

• ZONE_NORMAL- Initially,onlyDRAMpages

- Willgrowwithdemand

• ZONE_STRG- Initially,essentialstoragearea


VMmanager

ProcessesDPFS

Devicedriver

VFS

ZONEMIGRATE

ZONESTRG

ZONENORMAL

PM pagesDRAMpages


PMB

ZONENORMAL

§ ZONENORMALexpansion

Pagemovement

45

VMmanager

ProcessesDPFS

Devicedriver

VFS

ZONEMIGRATE

ZONESTRG

ZONENORMAL

PM pagesDRAMpages


PMB

PMpages



- PMpagemigration

§ ZONENORMALexpansion

Pagemovement

46

VMmanager

ProcessesDPFS

Devicedriver

VFS

ZONEMIGRATE

ZONESTRG

ZONENORMAL

PM pagesDRAMpages


PMB



- PMpagemigration

- IncreasedZONE_NORMALsize

§ ZONENORMALretraction

Pagemovement

47

VMmanager

ProcessesDPFS

Devicedriver

VFS

ZONENORMAL

ZONEMIGRATE

ZONESTRG

PM pagesDRAMpages


PMB

ZONENORMAL



- PMpageretrieval

§ ZONENORMALretraction

Pagemovement

48

VMmanager

ProcessesDPFS

Devicedriver

VFS

ZONEMIGRATE

ZONESTRG

ZONENORMAL

PM pagesDRAMpages


PMB



- PMpageretrieval

- ReducedZONE_NORMALsize

§ ZONE_MIGRATE toZONE_NORMAL• BothzonesmanagedwithBuddymemoryallocator• Whennumberoffreepagesfallsbelowwatermark• Transfers4MBcontiguouspages

− simplepointermanipulation

Pagemigration

ZONENORMAL

ZONESTRG

ZONEMIGRATE

...

order(10)Free_list

order(1)order(0)

order(2)

...

page

page

...

order(10)Free_list

order(1)order(0)

order(2) 4MB

2. linkpointer

1. un-linkpointer

49

pagepage

pagepage

2GB

4MB

ZONENORMAL

ZONEMIGRATE

ZONESTRG

ZONENORMAL

PM pagesDRAMpages


PMB

...

page

pagepagepage

2GB

§ ZONE_NORMAL to ZONE_MIGRATE• Nowatermark• Retrievedreleasedbyapplication

− simplepointermanipulation

Pageretrieval

ZONENORMAL

ZONESTRG

ZONEMIGRATE

...

order(10)Free_list

order(1)order(0)

order(2)

...

order(10)Free_list

order(1)order(0)

order(2)

2. linkpointer

1. un-linkpointer

...

page

page

...

page

page

...page

page

50

4MB

4MB

ZONENORMAL

ZONEMIGRATE

ZONESTRG

PM pagesDRAMpages


PMB

ZONENORMAL

Managementunit

ZONENORMAL

ZONESTRG

ZONEMIGRATE

...

order(10)Free_list

order(1)order(0)

order(2)

4KB

...

order(10)Free_list

order(1)order(0)

order(2)

2MB

4MB

2GB

4MB • PMBtakesthe2MBhugepagesize

• Contiguousmemoryspace(20 ~210 pages)- ZONE_NORMAL:4KB~4MB- ZONE_MIGRATE,ZONE_STRG:2MB~2GB

• Pagemovementunitis4MBcontiguouspage- PerformedbetweenZONE_NORMAL(order10)andZONE_MIGRATE(order1)

51

§ Memoryleakmayoccurinpagemovingprocess

Consistency

uponcrash

52

...

page

page 4MB

2. linkpointer

pagepage

pagepage

2GB

4MB

1. un-linkpointer

ZONENORMAL

ZONESTRG

ZONEMIGRATE

...

order(10)Free_list

order(1)order(0)

order(2)

...

order(10)Free_list

order(1)order(0)

order(2)

§ Memoryleakmayoccurinpagemovingprocess

Consistency

53

...

page

page 4MB

pagepage

pagepage

2GB

4MB

1. un-linkpointer

thesepagesareleaked

ZONENORMAL

ZONESTRG

ZONEMIGRATE

...

order(10)Free_list

order(1)order(0)

order(2)

...

order(10)Free_list

order(1)order(0)

order(2)

§ Useloggingtopreventmemoryleak

Consistency

54

...

page

page 4MB

3. linkpointer

pagepage

pagepage

2GB

4MB

1. Writelogaboutpagemovementorallocation

2. un-linkpointer

4. commitlog

ZONENORMAL

ZONESTRG

ZONEMIGRATE

...

order(10)Free_list

order(1)order(0)

order(2)

...

order(10)Free_list

order(1)order(0)

order(2)

§ Useloggingtopreventmemoryleak

Consistency

55

...

page

page 4MB

3. linkpointer

pagepage

pagepage

2GB

4MB

4. commitlog• Logentryfields- Pagestructureaddress- Operationtype- Destinationofpage- Commitbit

• Logentrysize<64bit

• Usemfence andclflush instructionsforlogging

1. Writelogaboutpagemovementorallocation

2. un-linkpointer

ZONENORMAL

ZONESTRG

ZONEMIGRATE

...

order(10)Free_list

order(1)order(0)

order(2)

...

order(10)Free_list

order(1)order(0)

order(2)

PreviousworkandSAY-Go

56

Swap Memorage pVM SAY-Go

Goal freeupworkingmemoryspace

memoryexpansion

persistentstore efficient useofresources

Memory-Storagedivision fixed fixed fixed dynamic

Consistency notconsidered notconsidered yes yes

Transparency yes yes no yes

Runtimeoverhead yes yes no no

Direct accesstodata no yes yes yes

strengthweakness

§ Testplatform• Machinespecification

− IntelXeonE5-2620v32.4GHz(24cores)− 256DRAM

− DividetoDRAMandPseudo-PM

− Linuxv4.11.1

§ Workloadscharacteristics

Evaluation

57

FFT RedisScale Memory intensiveapplication In-memorydatabase

Domain Signal processing Key-valuestore

Benchmarksuite Splash2xinParsec3.0 YCSB

Input Native(largest) 1:1 (read:write)

Max memoryusage 12GB 20GB

§ Comparisonwithswap• Memoryconfiguration• ZONE_NORMALcapacityvariedbasedonworkload

− FFT:16GBforDRAM− Redis:32GBforDRAM

Evaluation

58

DRAM Pseudo-PM

ZONENORMAL

ZONESTRG

ZONEMIGRATE

16or32GB 16GBremainder

SwapPMB

DRAM Pseudo-PM

ZONENORMAL

EXT4withDAX

swaparea

16or32GB remainder(setby/dev/pmem)

tobeuseddynamically

§ Comparisonwithswap

Evaluation

0

20

40

60

80

100

120

140

16 9 6 3 0

PMBSwap

Runtim

e(sec)

DRAM(GB)usedbyapplication(ZONE_NORMALsize:32GB)

RedisFFT

59


0

50

100

150

200

250

300

350

32 15 10 5 0

§ Measurementmethod:FFTexample

Evaluation

0

20

40

60

80

100

120

140

16 9 6 3 0

PMBSwap

Runtim

e(sec)


FFT

60

• FFTmaximumworkingsetsize:12GB

• LimitamountofmainmemoryusedbyFFT- Usestresstool toconsumemainmemory

• x-axisreferstothemaximumDRAMusedbyFFT

FFTstress6GB 6GB10GB

Pseudo-PMDRAM

ZONENORMAL16GB

ZONESTRG16GB

ZONEMIGRATE224GB

§ Measurementmethod:FFTexample

Evaluation

0

20

40

60

80

100

120

140

16 9 6 3 0

PMBSwap

Runtim

e(sec)


FFT

61

FFTstress3 9GB13GB

ZONENORMAL16GB

ZONESTRG16GB

ZONEMIGRATE224GB

Pseudo-PMDRAM

• FFTmaximumworkingsetsize:12GB

• LimitamountofmainmemoryusedbyFFT- Usestresstool toconsumemainmemory

• x-axisreferstothemaximumDRAMusedbyFFT


Evaluation

0

20

40

60

80

100

120

140

16 9 6 3 0

PMBSwap

Runtim

e(sec)


RedisFFT

62


0

50

100

150

200

250

300

350

32 15 10 5 0

0

50

100

150

200

250

300

350

32 15 10 5 0


EvaluationRu

ntim

e(sec)


Redis• Swapperformance

degradesconsiderably

• PMBperformanceremainsstable

FFT

0

20

40

60

80

100

120

140

16 9 6 3 0

PMBSwap

63


Tworeasonsbehindresults

0

50

100

150

200

250

300

350

32 15 10 5 0


EvaluationRu

ntim

e(sec)


RedisFFT

0

20

40

60

80

100

120

140

16 9 6 3 0

PMBSwap

64


• Mechanismoverhead- PMBmigrationandretrieval:~400ns

- Swap-in/out:~6/41us

0

50

100

150

200

250

300

350

32 15 10 5 0


EvaluationRu

ntim

e(sec)


FFT Redis• PMB“unaffected”by

DRAMsize- swappagecopiesincreasewithreducedDRAMsize

0

20

40

60

80

100

120

140

16 9 6 3 0

PMBSwap

65


§ Numberofcopiedpages

Evaluation

0

10

20

30

40

50

60

70

16 9 6 3 0

Swap-outSwap-in

0

10

20

30

40

50

60

70

32 15 10 5 0DRAM(GB)usedbyapplication(ZONE_NORMALsize:16GB)


Num

bero

fswappe

dpages

(unit:million)

0.0M

8.3M

21.8M

60.5M63.8M

0.0M 0.2M 0.4M

7.8M

26.4M

FFT Redis• Reportedby

/proc/vmstat

66

§ PMpagemigrationandretrievalbehavior• Measured per 3s interval

Evaluation

0

200

400

600

800

1000

1200

3 6 9 1215182124273033363942454851540

1000

2000

3000

4000

5000

6000

3 27 51 75 99 123

147

171

195

219

243

267

291

315

339

#ofpages

executiontime(s)executiontime(s)

migrationretrievalpagesinNORMALmovedfromMIGRATE

FFT Redis

67

§ PMpagemigrationandretrievalbehavior• Measured per 3s interval

Evaluation

0

200

400

600

800

1000

1200

3 6 9 1215182124273033363942454851540

1000

2000

3000

4000

5000

6000

3 27 51 75 99 123

147

171

195

219

243

267

291

315

339

#ofpages

executiontime(s)executiontime(s)


FFT Redis

68

§ PMpagemigrationandretrievalbehavior• FFT and Redis executed simultaneously

Evaluation

#ofpages

executiontime(s)

0

2000

4000

6000

8000

10000

120003 21 39 57 75 93 111

129

147

165

183

201

219

237

255

273

291

309

327

• Redis runsonce

• FFTrunningtime:60seconds- Restarteduponterminationaftera20secondlapse


69

§ PMpagemigrationandretrievalbehavior• FFT and Redis executed simultaneously

Evaluation

#ofpages

executiontime(s)

0

2000

4000

6000

8000

10000

120003 21 39 57 75 93 111

129

147

165

183

201

219

237

255

273

291

309

327

• Redis runsonce

• FFTrunningtime:60seconds- Restarteduponterminationaftera20secondlapse

- Pagesarebeingfreedandretrievedinthevicinityof60,140,220,300seconds

60s 60s 60s 60s20s 20s 20s 20s

70

§ ProposedasystemcalledStorage-As-You-Go(SAY-Go)• TransparentlyadjuststheuseofPM• PMcanbeusedasmemoryaswellasstorageasneedbe• Breaktheboundarybetweenmemoryandstorage

§ PersistentMemoryBuddy(PMB)• Memoryallocationservicethatcanfreelygrowandshrinkworkingmemory• ExperimentalresultsshowPMBperformsbetterthantraditionalswap

Summary

71

§ DPFS• Requiresdynamicpartitionresizingsupport• Inthedesignandearlyimplementationphase

§ PMB• Moredetailedpoliciesformanagingpage• Variousoptimizations

Futurework


DRAM PM

DPFS

PMB

72

Q& A

73

SAY-Go: Towards Transparent and Seamless Storage-As-You-Go ... · 1. File system with dynamic...

Documents

Transcript of SAY-Go: Towards Transparent and Seamless Storage-As-You-Go ... · 1. File system with dynamic...