PARAID: A Gear-Shifting Power-Aware RAID

PARAID:PARAID:A A Gear-ShiftingGear-Shifting

Power-Aware RAIDPower-Aware RAID

Charles Weddle, Mathew Oldham, Jin Qian, An-I Andy Wang – Florida St. UniversityCharles Weddle, Mathew Oldham, Jin Qian, An-I Andy Wang – Florida St. University

Peter Reiher – University of California, Los AngelesPeter Reiher – University of California, Los Angeles

Geoff Kuenning – Harvey Mudd CollegeGeoff Kuenning – Harvey Mudd College

MotivationMotivation

Energy costs are risingEnergy costs are rising An increasing concern for serversAn increasing concern for servers

No longer limited to laptopsNo longer limited to laptops

Energy consumption of disk drivesEnergy consumption of disk drives 24% of the power usage in web servers24% of the power usage in web servers 27% of electricity cost for data centers27% of electricity cost for data centers Root to other issues, e.g. server room coolingRoot to other issues, e.g. server room cooling

Is it possible to reduce energy consumption in Is it possible to reduce energy consumption in RAID devices without degrading performance RAID devices without degrading performance while maintaining reliability?while maintaining reliability?

PARAID: A Gear-Shifting Power-Aware RAID

ChallengesChallenges

EnergyEnergy Not enough opportunities to spin down RAIDsNot enough opportunities to spin down RAIDs

PerformancePerformance Essential for peak loadsEssential for peak loads

ReliabilityReliability Server-class drives are not designed for Server-class drives are not designed for

frequent power switchingfrequent power switching


Existing WorkExisting Work

Most trade performance for energy Most trade performance for energy savings directly. savings directly. For example, vary the speed of disksFor example, vary the speed of disks

Most are simulated resultsMost are simulated results


ObservationsObservations

Over provisioning of resourcesOver provisioning of resources RAID is configured for peak performanceRAID is configured for peak performance RAID keeps all drives spinning for light loadsRAID keeps all drives spinning for light loads

Unused storage capacityUnused storage capacity Over-provision of storage capacityOver-provision of storage capacity Unused storage can be traded for energy savingsUnused storage can be traded for energy savings

Fluctuating loadFluctuating load Cyclic fluctuation of loadsCyclic fluctuation of loads Infrequent on-off power transitions can be effectiveInfrequent on-off power transitions can be effective


Power-Aware RAIDPower-Aware RAID

Skewed striping for energy savingsSkewed striping for energy savings

Preserving peak performancePreserving peak performance

Maintaining reliabilityMaintaining reliability

EvaluationEvaluation

ConclusionConclusion

QuestionsQuestions


Skewed StripingSkewed Striping for Energy Saving for Energy Saving

Use over-provisioned spare storageUse over-provisioned spare storage Can use fewer disks for light loadsCan use fewer disks for light loads


RAID

SoftState

Gears123

1 2 3 4 5


Operate in gear 1Operate in gear 1 Disks 4 and 5 are powered offDisks 4 and 5 are powered off


RAID

SoftState

Gears123

1 2 3 4 5


Approximate the workloadApproximate the workload Gear shift into most appropriate gearGear shift into most appropriate gear

Minimize the opportunity lost to save powerMinimize the opportunity lost to save power

Energy( PoweredOn Disks )

Workload( Disk Parallelism )

Conventional RAID PARAID

workload



Adapt to cyclic fluctuating workloadAdapt to cyclic fluctuating workload Gear shift when gear utilization threshold is metGear shift when gear utilization threshold is met

time

load

utilization threshold

gear shift


Preserving Peak PerformancePreserving Peak Performance

Operate in the highest gear Operate in the highest gear When the system demands peak performanceWhen the system demands peak performance Uses the same disk layoutUses the same disk layout

Maximize parallelism within each gearMaximize parallelism within each gear Load is balanced on each gearLoad is balanced on each gear Uniform striping pattern within each gearUniform striping pattern within each gear

Delay block replication until gear shiftsDelay block replication until gear shifts Track block writes as an optimizationTrack block writes as an optimization


Maintaining ReliabilityMaintaining Reliability

Reuse existing RAID levels (RAID-5)Reuse existing RAID levels (RAID-5) Also used in soft stateAlso used in soft state

Drives have a limited number of power Drives have a limited number of power cyclescycles Ration number of power cyclesRation number of power cycles


File System

RAID

PARAID Block HandlerPARAID Block Handler

Disk Device Driver

User Space

Linux kernel

Soft RAIDSoft RAID

Reliability ManagerReliability Manager

MonitorMonitor

Disk ManagerDisk Manager

Admin ToolAdmin Tool

Logical Component DesignLogical Component Design


Data LayoutData Layout


Disk 1 Disk 2 Disk 3 Disk 4 Disk 5

Soft State(RAID-5)

(1-4) 8 12 ((1-4),8,12)

16 20 (16,20,_) _

RAID-5

1 2 3 4 (1-4)

5 6 7 (5-8) 8

9 10 (9-12) 11 12

13 (13-16) 14 15 16

(17-20) 17 18 19 20

Parity for 5 disks does not work for 4 disksParity for 5 disks does not work for 4 disks For example, replicated block 12 on disk 3For example, replicated block 12 on disk 3

Data LayoutData Layout



Soft State(RAID-5)

(1-4) 8 12 ((1-4),8,12)

16 20 (16,20,_) _

RAID-5

1 2 3 4 (1-4)

5 6 7 (5-8) 8

9 10 (9-12) 11 12

13 (13-16) 14 15 16

(17-20) 17 18 19 20

Cascading parity updatesCascading parity updates Must also update parity in soft stateMust also update parity in soft state

Update PropagationUpdate Propagation

Up-shift propagationUp-shift propagation Full synchronizationFull synchronization On-demand synchronizationOn-demand synchronization For example, shifting from 3 to 5 disksFor example, shifting from 3 to 5 disks

Downshift propagationDownshift propagation Full synchronizationFull synchronization


Asymmetric Gear-Shifting PoliciesAsymmetric Gear-Shifting Policies

Up-shift (aggressive)Up-shift (aggressive) Moving average + moving standard deviation > Moving average + moving standard deviation >

thresholdthreshold

Downshift (conservative)Downshift (conservative) Modified moving average + moving standard Modified moving average + moving standard

deviation < thresholddeviation < threshold Moving average modified to account for extra parity Moving average modified to account for extra parity

updatesupdates


ImplementationImplementation

Prototyped in Linux 2.6.5Prototyped in Linux 2.6.5 Open source, software RAIDOpen source, software RAID

Implemented block I/O handler, monitor, Implemented block I/O handler, monitor, disk managerdisk manager

Implemented user admin tool to configure Implemented user admin tool to configure devicedevice

Updated Raid Tools to recognize PARAID Updated Raid Tools to recognize PARAID levellevel



ChallengesChallenges Prototyping PARAIDPrototyping PARAID Commercial machinesCommercial machines Benchmarks are designed to measure peak Benchmarks are designed to measure peak

performanceperformance Trace replayTrace replay Time consumingTime consuming



multimeter

USB cableclient

server

powersupply

12v & 5vpower lines

powermeasurement

probes

SCSIcable

crossover cable

Xeon 2.8 Ghz, 512 MB RAM36.7 GB 10k RPM SCSI

P4 2.8 Ghz, 1 GB RAM160 GB 7200 RPM SATA

RAID

RAID

RAID

RAID

RAID

BOOT


Measurement frameworkMeasurement framework


Measurement frameworkMeasurement framework


Server Client

Processor Intel Xeon 2.8 Ghz Intel Pentium 4 2.8 Ghz

Memory 512 Mbytes 1 Gbytes

Network Gigabit Ethernet Gigabit Ethernet

Disks36.7Gbytes 15K RPM

SCSI Ultra 320160 Gbytes 7200 RPM

SATA


Three different workloads using two different Three different workloads using two different RAID settingsRAID settings Web trace - RAID level 0 – 2 gears (2,5)Web trace - RAID level 0 – 2 gears (2,5)

Mostly read activityMostly read activity Cello99 - RAID level 5 – 2 gears (3,5)Cello99 - RAID level 5 – 2 gears (3,5)

I/O intensive workload with writesI/O intensive workload with writes Postmark - RAID level 5 – 2 gears (3,5)Postmark - RAID level 5 – 2 gears (3,5)

Measure peak performance and gear shifting overheadMeasure peak performance and gear shifting overhead

Speed up trace playbackSpeed up trace playback To match hardwareTo match hardware Explore range of speed up factors and power savingsExplore range of speed up factors and power savings


Web TraceWeb Trace

UCLA CS Dept Web Servers (8/11/2006 – 8/14/2006)UCLA CS Dept Web Servers (8/11/2006 – 8/14/2006) File system: ~32 GB (~500k files)File system: ~32 GB (~500k files) Trace replay: ~95k requests with ~4 GB data (~260 MB unique)Trace replay: ~95k requests with ~4 GB data (~260 MB unique)


0

0.1

0.2

0.3

0.4

0.5

0.6

1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96

hours

GB/hour

Web Trace Energy SavingsWeb Trace Energy Savings

0

10

20

30

40

50

60

0 5 10 15 20 25 30

hours

wattsRAID-0

PARAID-0

0

10

20

30

40

50

60

0 5 10 15 20 25 30

hours

wattsRAID-0

PARAID-0


64x – 60 requests/sec

128x – 120 requests/sec 256x – 240 requests/sec

0

10

20

30

40

50

60

0 5 10 15 20 25 30

hours

wattsRAID-0

PARAID-0

64x - 34%

128x - 28%

256x - 10%

Energy Savings

Web Trace LatencyWeb Trace Latency

0

0.2

0.4

0.6

0.8

1

1 10 100 1000 10000 100000

msec

RAID-0

PARAID-0

0

0.2

0.4

0.6

0.8

1

1 10 100 1000 10000 100000

msec

RAID-0

PARAID-0


256x

128x 64x

0

0.2

0.4

0.6

0.8

1

1 10 100 1000 10000 100000

msec

RAID-0

PARAID-0256x - within 2.7%

64x - 240% 80ms vs. 33ms

Overhead

Web Trace BandwidthWeb Trace Bandwidth

-20

30

80

130

180

0 5 10 15 20 25 30

hours

MB/secRAID-0

PARAID-0

-20

30

80

130

180

0 5 10 15 20 25 30

hours

MB/secRAID-0

PARAID-0


256x

128x 64x

0

20

40

60

80

100

120

140

160

180

0 5 10 15 20 25 30

hours

MB/secRAID-0

PARAID-0

256x - within 1.3% in high gear

Overhead

Cello99 TraceCello99 Trace

Cello99 WorkloadCello99 Workload HP Storage Research LabsHP Storage Research Labs 50 hours beginning on 9/12/199950 hours beginning on 9/12/1999 I/O intensive with 42% writesI/O intensive with 42% writes


Cello99 Energy SavingsCello99 Energy Savings

0

10

20

30

40

50

0 10 20 30 40 50

hours

wattsRAID-5

PARAID-5

05

101520253035404550

0 10 20 30 40 50

hours

wattsRAID-5

PARAID-5





0

10

20

30

40

50

0 10 20 30 40 50

hours

wattsRAID-5

PARAID-532x - 13%

64x - 8.2%

128x - 3.5%

Energy Savings

Cello99 Completion TimeCello99 Completion Time

0.9

0.92

0.94

0.96

0.98

1

1 10 100 1000 10000 100000

msec

RAID-5

PARAID-5

0.9

0.92

0.94

0.96

0.98

1

1 10 100 1000 10000 100000

msec

RAID-5

PARAID-5


128x

64x 32x

0.9

0.92

0.94

0.96

0.98

1

1 10 100 1000 10000 100000

msec

RAID-5

PARAID-5

32x - 1.8ms,26% slowerdue to time spent in lowgear

Overhead

Cello99 BandwidthCello99 Bandwidth

1

10

100

1000

0 500000 1000000 1500000

request number

MB/secRAID-5

PARAID-5

1

10

100

1000

0 500000 1000000 1500000

requests

MB/secRAID-5

PARAID-5


1

10

100

1000

0 500000 1000000 1500000

request number

MB/secRAID-5

PARAID

64x 32x

128x

Overhead

< 1% degra-dation duringpeak hours

Postmark BenchmarkPostmark Benchmark

Popular synthetic benchmarkPopular synthetic benchmark Generates ISP-style workloadsGenerates ISP-style workloads Stresses peak read/write performance of storage Stresses peak read/write performance of storage

devicedevice


Postmark PerformancePostmark Performance


0

50

100

150

200

1K files, 50K trans 20K files, 50K trans 20K files, 100Ktrans

seconds

RAID-5 PARAID-5 high gear PARAID-5 low-gear

Postmark Power MeasurementsPostmark Power Measurements


0

10

20

30

40

50

60

70

80

1 11 21 31 41 51 61 71 81 91 10 111 12 13 14 151 16 171

seconds

wat

ts RAID5

PARAID

Ongoing WorkOngoing Work

Try more workloadsTry more workloads Optimize PARAID gear configurationOptimize PARAID gear configuration Explore asynchronous update propagationExplore asynchronous update propagation Speed up recoverySpeed up recovery Live testingLive testing


Lessons LearnedLessons Learned

Third version of design, early design not portableThird version of design, early design not portable Data alignment problemsData alignment problems Difficult to measure system under normal loadDifficult to measure system under normal load Hardware and operating system optimizationsHardware and operating system optimizations Matching trace environmentMatching trace environment


ConclusionConclusion

PARAID reuses standard RAID-levels without PARAID reuses standard RAID-levels without special hardware while decreasing their energy special hardware while decreasing their energy use by 34%.use by 34%. Optimized version can save even more energyOptimized version can save even more energy

Empirical evaluation importantEmpirical evaluation important


QuestionsQuestions


PARAID Gear-ShiftingPARAID Gear-Shifting

256x 128x 64x

Number of gear switches 15.2 8.0 2.0

% time spent in low gear 52% 88% 98%

% extra I/Os for updatepropagations

0.63% 0.37% 0.21%

128x 64x 32x

Number of gear switches 6.0 5.6 5.4

% time spent in low gear 47% 74% 88%

% extra I/Os for updatepropagations

8.0% 15% 21%

Web Trace Gear-Shifting Stats

Cello99 Gear-Shifting Stats

Storage consumption Si for the total RAID for the ith gear;

MiSGGGS

S

iiii

M

ijj

M

ii

..2,1

1

111

1

• PARAID uses around (D – G1)/(D – 1) of the total RAID-5 storage to store soft states.

• For RAID-5, D > 3 disks, M gears with Gi disks within the ith gear (1 < i < M, 3 < Gi < Gi+1 < GM = D)

Storage Requirement for PARAID5Storage Requirement for PARAID5

Storage Requirement for PARAID5Storage Requirement for PARAID5


Gear 1RAID-5

(1-4) 8 12 ((1-4),8,12)

16 20 (16,20,_) _

Gear 2RAID-5

1 2 3 4 (1-4)

5 6 7 (5-8) 8

9 10 (9-12) 11 12

13 (13-16) 14 15 16

(17-20) 17 18 19 20

S1

S2

G1 = 4 G2 = 5

S2(G2 – G1)

S1(G1 – 1)

Space needed for extra parity blocks

S1 + S2 = 1

S1(G1 – 1) = S2(G2 – G1)

PARAIDPARAID

Target percentage energy savings;

idleactive

idleactivedbys

DP

PGPGD

/

/1tan11

• Energy savings increase with more disks and fewer disks in the lowest gear.

• A higher active/standby ratio.

• D = the number of disks in the array• G1 = the disks in gear 1.• P = power in standby/active/idle

PARAIDPARAID

Modified moving utilization;

• Accounts for cascading parity writes in lower gear.

• Gi = the disks in gear i.• Aread/write = the read or write activity• W = the weight to account for additional parity writes (RAID5 = 1.5)

WG

GG

G

GAA

G

GUU

i

ii

i

iwriteread

i

i 11

1

'

PARAIDPARAID

max min configuration empty storagemax energy

savings formula

4 3 3-4 33.00% 25.00% 0.333333

5 3 3-5 50.00% 40.00% 0.5

5 4 4-5 25.00% 20.00% 0.25

5 3 3-4-5 50.00% 40.00% 0.5

6 3 3-6 60.00% 50.00% 0.6

6 4 4-6 40.00% 33.00% 0.4

6 5 5-6 20.00% 17.00% 0.2

6 3 3-4-6 60.00% 50.00% 0.6

6 3 3-5-6 60.00% 50.00% 0.6

6 4 4-5-6 40.00% 33.00% 0.4

6 3 3-4-5-6 60.00% 50.00% 0.6

7 3 3-7 66.67% 57.14% 0.666667

7 4 4-7 50.00% 42.86% 0.5

7 5 5-7 33.00% 28.57% 0.333333

7 6 6-7 16.67% 14.29% 0.166667

PARAID: A Gear-Shifting Power-Aware RAID

Documents

Transcript of PARAID: A Gear-Shifting Power-Aware RAID