Quantcast File System (QFS) - Alternative to HDFS

!

Presented by Silvius Rus, Director, Big Data Platforms!

December 2013!!

!

!

QuantcastPetabyte Storage at Half Price with QFS!

12-13!1!

Big Data Gurus Meetup!

© Quantcast 2012!

Quantcast File System (QFS)!

A high performance alternative to the Hadoop Distributed File System (HDFS).!

!

12-13!Quantcast File System! 2!

Manages multi-petabyte Hadoop workloads with significantly faster I/O than HDFS and uses only half the disk space.!

Offers massive cost savings to large scale Hadoop users (fewer disks = fewer machines).!

Production hardened at Quantcast under massive processing loads (multi exabyte).!

Fully Compatible with Apache Hadoop.!

100% Open Source.!

!

!

© Quantcast 2012!

Quantcast Technology Innovation Timeline!


Quantcast!Measurement!

Launched!!

Quantcast!Advertising!Launched!

!

!Launch!

QFS!

Started!using!

Hadoop!

Using and!sponsoring!

KFS!

Turned!off!

HDFS!

Receiving!1TB/day!

Receiving!10TB/day!

Receiving!20TB/day!

Receiving!40TB/day!

Processing!1PB/day!

Processing!10PB/day!

Processing!20PB/day!

2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013!

© Quantcast 2012!

Architecture!


Rack 1

Client

MetaserverRack 2

Chunk servers

Chunk servers

Chunk Server·∙ Handles IO to locally stored 64MB chunks

·∙ Monitors host file system health·∙ Replicates and recovers chunks as metaserver directsMetaserver

·∙ Maps /file/paths to chunk ids·∙ Manages chunk locations·∙ Directs clients to chunk servers

Client·∙ Implements high level file interface (read/write/delete)

·∙ On write, RS encodes chunks and distributes stripes to nine chunk servers.

·∙ On read, collects RS stripes from six chunk servers and recomposes chunk.

Locate or allocate chunks

Read/write RS encoded data from/to chunk servers

Chunk replicationand rebalancinginstructions

Copy/Recoverchunks

© Quantcast 2012!

QFS vs. HDFS!

Broadly comparable feature set, with significant storage efficiency advantages.!


Feature! QFS! HDFS!

Scalable, distributed storage designed for efficient batch processing! ü! ü!

Open source! ü! ü!

Hadoop compatible! ü! ü!

Unix style file permissions! ü! ü!

Error Recovery mechanism! Reed-Solomon encoding!

Multiple data copies!

Disk space required (as a multiple of raw data)! 1.5x! 3x!

© Quantcast 2012!

Reed-Solomon Error CorrectionLeveraging high-speed modern networks!

HDFS optimizes toward data locality for older networks.!

10Gbps networks are now common, making disk I/O a more critical bottleneck.!

QFS leverages faster networks to achieve better parallelism and encoding efficiency.!

Result: higher error tolerance, faster performance, with half the disk space.!


1.  Break original data into 64K stripes.!

2.  Reed-Solomon generates three parity stripes for every six data strips!

!3.  Write those to nine

different drives.!

4.  Up to three stripes can become unreadable...!

5.  …yet the original data can still be recovered !Every write parallelized across 9 drives, every read across 6!

Reed-Solomon Parallel Data I/O!

© Quantcast 2012!

MapReduce on 6+3 Erasure Coded Filesversus 3x Replicated Files!

Positives!

Writing is ½ off, both in terms of space and time!

Any 3 broken or slow devices will be tolerated vs. any 2 with 3-way replication!

Re-executed stragglers run faster due to reading from multiple devices (striping)!


Negatives!

There is no locality, reading will require the network!

On read failure, recovery is needed – however it’s lightning fast on modern CPUs (2 GB/s per core)!

Writes don’t achieve network line rate as original + parity data is written by a single client!

© Quantcast 2012!

Read/Write Benchmarks!


0

2

4

6

8

10

12

14

16

18

Write Read

End

-to-e

nd ti

me

(min

utes

)

HDFS 64 MB

HDFS 2.5 GB

QFS 64 MB

End-to-end 20 TB write test End-to-end 20 TB read test 8,000 workers * 2.5 GB each Tests ran as Hadoop MapReduce jobs

© Quantcast 2012!

Read/Write Benchmarks!


0

2

4

6

8

10

12

14

16

18

Write Read

End

-to-e

nd ti

me

(min

utes

)

HDFS 64 MB

HDFS 2.5 GB

QFS 64 MB

End-to-end 20 TB write test End-to-end 20 TB read test 8,000 workers * 2.5 GB each Tests ran as Hadoop MapReduce jobs

Host network behavior during tests QFS write = ½ disk I/O of HDFS write QFS write à network/disk = 8/9 HDFS write à network/disk = 6/9 QFS read à network/disk = 1 HDFS read à network/disk = very small

© Quantcast 2012!

0 50 100 150 200 250 300

ls

mkdir

rmdir

stat

Operations per second (thousands)

QFS HDFS

Metaserver Performance!


Intel E5-2670 64 GB RAM 70 million directories

© Quantcast 2012!

Production Hardening for Petascale!

Continuous I/O Balancing!

•  Full feedback loop!

•  Metaserver knows the I/O queue size of every device!

•  Activity biased towards under-loaded chunkservers!

•  Direct I/O = short loop!


Optimization!

•  Direct I/O and fixed buffer space = predictable RAM and storage device usage!

•  C++, own memory allocation and layout!

•  Vector instructions for Reed Solomon coding!

Operations!

•  Hibernation!

•  Evacuation through recovery!

•  Continuous space/integrity rebalancing!

•  Monitoring and alerts!

© Quantcast 2012!

Fast and Efficient MapReduceQuantsort: All I/O over QFS!


Concurrent append. 10,000 writers append to same file at once.

http://qc.st/QCQuantsort

Largest sort = 1 PB Daily = 1 to 2 PB, max = 3 PB

© Quantcast 2012!

How Well Does It Work!

Reliable at Scale!

Hundreds of days of metaserver uptime common!

Quantcast MapReduce sorter uses QFS as distributed virtualized store instead of local disk!

8 petabytes of compressed data!

Close to 1 billion chunks!

7,500 I/O devices!

! 12-13!Quantcast File System! 13!

© Quantcast 2012!


Reliable at Scale!





7,500 I/O devices!


Fast and Large!

Ran petabyte sort last weekend.!

Direct I/O not hurting fast scans: Sawzall query performance similar to Presto:!

! Presto/HDFS

Turbo/QFS

Seconds 16 16

Rows 920 M 970 M

Bytes 31 G 294 G

Rows/sec 57.5 M 60.6 M

Bytes/sec 2.0 G 18.4 G

© Quantcast 2012!


Reliable at Scale!





7,500 I/O devices!


Fast and Large!

Petabyte sort.!

Direct I/O not hurting fast scans: Sawzall query performance similar to Presto:!

!

Easy to Use!

1 Ops Engineer for QFS and MapReduce on 1,000+ node cluster!

Neustar set up multi petabyte instance without help from Quantcast!

Migrate from HDFS using hadoop distcp!

Hadoop MapReduce “just works” on QFS!

Presto/HDFS

Turbo/QFS

Seconds 16 16

Rows 920 M 970 M

Bytes 31 G 294 G

Rows/sec 57.5 M 60.6 M

Bytes/sec 2.0 G 18.4 G

© Quantcast 2012!

Metaserver Statistics in Production!


QFS metaserver statistics over Quantcast production file systems in July 2013. •  High Availability is nice to have but not a must-have for MapReduce. There are certainly

other use cases where High Availability is a must. •  Federation may be needed to support file systems beyond 10 PB, depending on file size

© Quantcast 2012!

Chunkserver


Other FeaturesTiered Storage!

RAM

2 SSDs

10 Disks

Chunkserver

RAM

2 SSDs

10 Disks

And 450 more just like them. Tier Range as File Attribute. Use tier across 450 machines. Used in production to accelerate MapReduce fanout

© Quantcast 2012!12-13!Quantcast File System! 18!

94.5

16.7

8.5 4.8

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

HDFS Default HDFS Small Blocks QFS on Disk QFS in RAM

Bro

adca

st T

ime

(s)

Configuration

Other FeaturesFast Broadcast through Wide Striping!

© Quantcast 2012!12-13!Quantcast File System! 19!

700

7 0

100 200 300 400 500 600 700 800

HDFS QFS

Time (msec)

Time (msec)

Refreshingly Fast Command Line Toolhadoop fs -ls / versus qfs –ls /!

© Quantcast 2012!

Who will find QFS valuable?!

Likely to benefit from QFS!

Existing Hadoop users with large-scale data clusters.!

Data heavy, tech savvy organizations for whom performance and efficient use of hardware are high priorities.!


May find HDFS a better fit!

Small or new Hadoop deployments, as HDFS has been deployed in a broader variety of production environments.!

Clusters with slow or unpredictable network connectivity.!

Environments needing specific HDFS features such as head node federation or hot standby.!

!

© Quantcast 2012!

Summary!

Key Benefits of QFS!

Delivers stable high performance alternative to HDFS in a production-hardened 1.0 release!

Offers high performance management of multi-petabyte workloads!

Faster I/O than HDFS with half the disk space.!

Fully Compatible with Apache Hadoop!

100% Open Source!

© Quantcast 2012!

Future Work!

What QFS Doesn’t Have Just Yet!

Kerberos Security – under development!

HA – No strong case at Quantcast, but nice to have!

Federation – Not a strong case either at Quantcast!

Contributions welcome!!

New York432 Park Avenue SouthNew York, NY 10016!

San Francisco201 Third StreetSan Francisco, CA 94103!

London48 Charlotte StreetLondon, W1T 2NS!

© Quantcast 2012!

Thank You. Questions? !

!Download QFS for free at:github.com/quantcast/qfs!

12-13! 23!Quantcast File System!

Quantcast File System (QFS) - Alternative to HDFS

Technology

Transcript of Quantcast File System (QFS) - Alternative to HDFS