GFS : Google File System

Ömer Faruk İnceFatih University - Computer Engineering

Cloud Computing25.03.2014

Overview• What is the GFS? • Why is the GFS designed for?• Design Overview– Assumptions– Interfaces– Architecture– Master Server– Chunk Server– Metadata• System Interactions• Master Operations– Garbage Collections– Fault Tolerance– Data Integrity• System Interactions• Conclusions

What is the GFS ? • The GFS is designed for meeting the rapidly

growing demands of Google Data processing needs and GFS shares many of the same aims as previous distributed file systems such as performance, scalability, reliability and avaliability.

Why is the GFS designed for ?Google needed a good distributed file system.Redundant storage of massive amounts of data

on cheap and unreliable computers.Why not Google use an existing file system?

Google’s problems are different from anyone else’s

Different workload and design prioritiesGFS is designed for Google apps and workloads.Google apps are designed for GFS.

Assumptions• System built from many inexpensive commodity components.• System stores modest number of large files.

– Few million, each typically 100 MB or larger in size. Multi-GB common.– Small files must be supported, but need not optimize.

• Workload is primarily:– Large streaming reads– Small random reads– Many large sequential appends.

• Must efficiently implement concurrent, atomic appends.– Producer-consumer queues.– Many-way merging.

Interface• GFS provides a familiar file system interface, though it

does not implement a standard API. Files are organized hierarchically in directories and identified by pathnames. It supports the usual operations to create, delete, open, close, read, and write files.

• Moreover, GFS has snapshot and record append operations. Snapshot creates a copy of a file or a directory tree at low cost. Record append allows multiple clients to append data to the same file concurrently while guaranteeing the atomicity of each individual client’s append. It is useful for implementing multi-way merge results and producer consumer queues that many clients can simultaneously append to without additional locking.

Arhitecture• A GFS cluster consists of;

A single master, Multiple Chunkservers, Multiple Clients as shown in Figure-1

Chunk Size• Files stored as chunks Fixed size (64MB)• Chunk size is one of the key design parameters.

Metadata• The master stores three major types of metadata:

File and chunk namespaces Mapping from files to chunks Locations of each chunk’s replicas

• All metadata is kept in the master’s memory. • The first two types(namespaces and file to chunk

mapping) are also kept persistent by logging mutations to an operation log stored on the master’s local disk and replicated on remote machines.

Master’s Responsibilities• Metadata storage• Namespace management/locking• Periodic communication with chunkservers– give instructions, collect state, track cluster health

• Chunk creation, re-replication, rebalancing– balance space utilization and access speed– spread replicas across racks to reduce correlated failures– re-replicate data if redundancy falls below threshold– rebalance data to smooth out storage and request load

Master’s Responsibilities (2)• Garbage Collection

simpler, more reliable than traditional file deletemaster logs the deletion, renames the file to a

hidden namelazily garbage collects hidden files

• Stale replica deletiondetect “stale” replicas using chunk version

numbers

System Interactions1. The client asks the master which

chunkserver holds the current lease for the chunk and the locations of the other replicas.

2. The master replies with the identity of the primary and the locations of the other (secondary) replicas. Client caches.

3. Client pre-pushes data to all replicas.

4. After all replicas acknowledge, client sends write request to primary.

5. Primary forwards write request to all replicas.

6. The secondaries all reply to the primary indicating that they have completed the operation.

7. Primary replies to client. Errors handled by retrying.

Read Algorithm1. Application originates the read request2. GFS client translates request and sends it to master3. Master responds with chunk handle and replica

locations.

Read Algorithm4. Client picks a location and sends the request5. Chunkserver sends requested data to the client6. Client forwards the data to the application

Write Algorithm1. Application originates the request2. GFS client translates request and sends it to master3. Master responds with chunk handle and replica

locations

Write Algorithm4.Client pushes write data to all locations. Data is stored

in chunkserver’s internal buffers

Write Algorithm5. Client sends write command to primary6. Primary determines serial order for data instances in

its buffer and writes the instances in that order to the chunk

7. Primary sends the serial order to the secondaries and tells them to perform the write

Atomic Record Append1. Client pushes data to all replicas.2. Sends request to primary. Primary check maximum size.

– Within maximum size• Append the data to its replica

– Exceed maximum size• Send error to client• Retried on the next chunk

3. Replicas of the same chunk may contain different data possibly including duplicates of the same record in whole or in part. These are handled by the client.

• It only guarantees that the data is written at least once as an atomic unit.

Snapshot• A snapshot is a copy of a system at a moment

in time.– To quickly create branch copies of huge data sets.– To checkpoint the current state before experimenting with changes

that can later be committed or rolled-back easily .

• When the master receives a snapshot request, it first revokes any outstanding leases on the chunks in the files it is about to snapshot.

Master Operations • Namespace Management and Locking GFS logically represents its namespace as a lookup table mapping full

pathnames to metadata.

• Need locking to prevent:– Two clients from trying to create the same file at the same time.– Changes to a directory tree during snapshotting.

• Solution:– Lock intervening directories in read mode.– Lock final file or directory in write mode.– For snapshot lock source and target in write mode.

Replica Management• Maximize data reliability and availability• Maximize bandwidth utilization– Need to spread chunk replicas across machines

and racks

Creation, Re-replication and Rebalancing• Replicas created for three reasons:

– Chunk creation– Re-replication– Load balancing

• Creation– place new replicas on chunk servers with below-average disk space

utilization.– Spread replicas across racks.

• Re-replication– re-replicates a chunk as soon as the number of available replicas falls below

a user-specified goal.

• Rebalancing– Periodically examines distribution and moves replicas for better disks pace

and load balancing.

Garbage Collection• Storage reclaimed lazily by GC.• File first renamed to a hidden name.• Hidden files removes if more than three days old.• When hidden file removed, in-memory metadata is

removed.• Regularly scans chunk namespace, identifying

orphaned chunks. These are removed.• Chunkservers periodically report chunks they have

and the master replies with the identity of all chunks that are no longer present in the master’s metadata. The chunkserver is free to delete its replicas of such chunks.

Fault Tolerance• High availability– Fast recovery– Master and Chunkservers restartable in a few seconds

• Chunk replication– Each chunk is replicated on multiple chunkservers on

different tracks. Users can specify different levels for different parts of the file namespace.

– default: 3 replicas.

• Shadow masters– Data integrity– Checksum every 64KB block in each chunk

Data Integrity• Each chunkserver uses checksumming to

detect corruption of stored data.• Checksums are kept in memory.– Separate from data.

• On read error, error is reported to master. – Master will re-replicate the chunk.– Requestor read from other replicas

Performance Testing• GFS cluster consisting of:– One master

• Two master replicas– 16 chunkservers– 16 clients

• Machines were:– Dual 1.4 GHz PIII– 2 GB of RAM– 2 80 GB 5400 RPM disks– 100 Mbps full-duplex Ethernet to switch– Servers to one switch, clients to another. Switches

connected via gigabit Ethernet.

Reads• N clients

reading 4 MB region from 320 GB file set simultan-eously.

• Read rate slightly lower as clients go up due to probability reading from same chunkserver.

80%

75%

Writes• N clients writing

to N files simultaneously.

• Low write rate is due to delay in propagating data among replicas.

• Slow write is not major problem with aggregate write bandwidth to large clients.50%

50%

Record Appends• N clients appending

to a single file simultaneously.

• Append rate slightly lower as clients go up due to network congestion by different clients.

• Chunkserver network congestion is not major issue with large n clients appending to large shared files.

Conclusions• GFS demonstrates how to support large-scaleprocessing workloads on commodity hardware– design to tolerate frequent component failures– optimize for huge files that are mostly appended

and read– feel free to relax and extend FS interface as required– go for simple solutions (e.g., single master)

• GFS has met Google’s storage needs,therefore good enough for them.

Thanks for listening…