The Google File System Presenter: Gladon Almeida Authors: Sanjay Ghemawat Howard Gobioff Shun-Tak...

The Google File System

Presenter:

Gladon Almeida

Authors:• Sanjay Ghemawat• Howard Gobioff• Shun-Tak Leung

Year: OCT’2003

Google File System

14/9/201

3

Key Design Considerations

Component failures are the norm rather than the exception.

Files are huge by traditional standards.

Most files are mutated by appending new data rather than overwriting existing data.

Co-designing the applications and the file system API.

Google File System 24/9/2013

Google File System 3

Assumptions Inexpensive commodity hardware that often fail.

Modest number of large files(typically ~100MB) will be stored. Small files supported but not optimized for.

Large streaming read / small random reads

Large sequential write that append data to files

System must efficiently implement well-defined-semantics for multiple clients that concurrently append to the same file

High sustained bandwidth is more important than low latency

4/9/2013

Architecture

Single master and multiple chunk servers

Each is a typical commodity Linux machine running a user level server process

Files are divided into fixed sized chunks each of which is identified by a globally unique 64-bit chunk handle

Typically 3 replica’s of each chuck spread over different chunk servers

Master maintains all file system metadata Namespace

Access control information

Mapping from files to chunks

Current locations of chunk

Master also control system-wide activities like: Lease management

Garbage collection of orphaned chunks

Chunk migration between chunk servers

Periodic handshake messages with chunk servers

GFS client code uses this file system API to communicate with Master and chunk servers


Architecture

Figure: GFS Architecture

4/9/2013

Design Overview

Chuck Size (64MB - Large)

Reduces client-master interactions

Reduces network overhead

Reduces size of metadata on the master.

Metadata:

Namespace and file-to-chunk mappings persistent storage as mutation logs on master (as well as remote locations)

Operation Logs:

Historical record of critical metadata changes

Defines the order of concurrent operations

Critical:

Replication on multiple remote machines

Changes are 1st made persistent on local as well as remote location and then made visible to client.

Fast recovery: (1 minute for few million files)

Replay operation logs

Checkpoints (B-tree like form)Google File System 64/9/201

3

Consistency Model Consistent region:

If all the clients will always see the same data, regardless of which replicas they read from.

Defined region:

After a file data mutation if it is consistent and clients will see what the mutation writes in its entirety.


3

Consistency Model – contd. After a sequence of successful mutations, the

mutated is guaranteed to be defined by:

1. Applying mutations to all replicas in the same order

2. Using chunk version number to detect any replica that becomes stale

What if client caches stale chunk location?

Such window limited by the cache entry’s timeout

Most files are append-only – Stale replica returns a premature end of chunk rather than outdated data


3


System Interactions: Leases and Mutation order Leases: used to maintain a consistent mutation

order across replicas.

Lease is granted by the master to one of the replicas called the primary

The primary picks a serial order for all mutations to the chunk.

Initial timeout of lease of 60 sec which can be extended

Extension requests piggybacked on heartbeat messages between master and chunk server

4/9/2013

Control and Data flow for a write


3


Decoupling of Data and Control Flow

Control flow:

Master -> Client -> Primary Chunk -> Secondary Chunks

Data flow:

Decoupled from control flow to use the network efficiently

Data pushed linearly in pipeline fashion

Each machine forwards data to the closest machine (Determined by the IP address)

Outbound b/w is fully utilized (No tree structure)

Switched networks and full-duplex links

4/9/2013


Atomic Record Appends (record append)

Concurrent serializable appends

Client specifies only data (No offset)

GFS uses append-at-least-once-atomically policy

GFS appends the data and returns the offset to the client

Heavily used:

Multiple-producer / Single-Consumer queues

Concurrent merged results from many different clients

If failure, the client retries the operation

GFS doesn’t guarantee that each chunk is byte-wise identical, it only guarantees that data is written at least once as an atomic unit.

Successful operations: Defined regions

Intervening regions: Undefined regions

4/9/2013


Snapshots Why ?

Makes copy of a file or directory almost instantaneously

Copy-on-write technique

Steps when a snapshot request is received:

1. Revoke nay outstanding leases

2. Log the operation to the disk

3. Duplicate the metadata for source file / directory tree

When write request to these chucks is received

Notices that reference count is greater than 1

Creates copy of chunk locally

Informs other replicas to do the same

Returns new chunk handle4/9/201

3

Master Operations - Namespace Management and Locking

No per-directory data structure

No support for aliases

lookup table mapping full path names to metadata

Each node in namespace tree (file/directory) has associated read/write lock

Why locks ?

Example: to lock /d1/d2/d3/leaf for write

Example: How mechanism prevents a file /home/user/foo from being created while /home/user is being snapshotted to /save/user

Snapshot:

Read locks on /home, /save

Write locks on /home/user, /save/user

Create:

Read lock on /home, /home/user

Write lock on /home/user/fooGoogle File System 144/9/201

3


Policies: Chunk replica replacement:

1. Maximize data reliability and availability

2. Maximize network bandwidth utilization

3. Spread replicas across machine as well as racks

New chunk creation:

1. New replicas created on below-average disk utilization

2. Limit the number of “recent” creations on each chunk server

3. spread replicas of a chunk across racks

Re-replication: soon as the number of available replicas falls below a user-specified

goal. Priority considering:

1. How far from replication goal?

2. Is chunk blocking client?

3. Is file live?

Occasionally rebalance replicas

4/9/2013


Garbage Collection Lazy garbage collection

Steps:

1. Log the deletion immediately

2. Rename file to hidden name (Deleted in 3 days)

3. During regular scans remove the such hidden files

Master’s regular scans of chunk namespace

Identify orphaned chunks and erase metadata

During heartbeat message exchange this info with the chunk servers

Chunk servers delete these chunks

Stale replica detection:

Using version numbers

4/9/2013


Fault Tolerance: High Availability:

Fast recovery: Servers designed to restart fast.

Chunk Replication:

Different replication levels for different parts on namespace

Default level = 3

Master Replication:

Operation log and checkpoint replicated remotely

Master process externally monitored.

On failure new process started using remotely saved checkpoint and logs

Use of canonical names

Shadow Masters

Data Integrity:

chunk broken into 64 KB blocks. Each has 32 bit checksum.

Chunk server verifies before returning – Hence, no error propagation

4/9/2013


Relating it to CSCI-572 !

GFS led to development of Hadoop Distributed File System

HDFS ideal for large workloads which can use the Map-Reduce framework for high degree of parallelism using commodity hardware.

Ideal for search engine workloads like:

Crawling,

Generating inverted index,

PageRank calculation, etc.

Other:

Large-scale machine learning problems

Clustering problems

Large scale graph computation

4/9/2013


Pros and Cons Pros:

Assumptions at the beginning of the paper are later backed up by experiment results.

Very important paper: Led to development of Hadoop Distributed File System (HDFS).

Cons:

Only talks about workloads which are sequential reads and file appends. GFS is not suitable for random read/writes.

The authors don’t provide any performance results for random read/writes

4/9/2013


Conclusion:

GFS supports large-scale data processing workloads in commodity hardware

Design decisions specific to Google's needs but many may apply to data processing tasks of a similar magnitude and cost consciousness.

Component failures are the norm rather than exception

Optimization priority:

1. Concurrent appends

2. Read

Fault tolerance by monitoring, replication and fast recovery

High aggregate throughput to many concurrent readers

4/9/2013

The Google File System Presenter: Gladon Almeida Authors: Sanjay Ghemawat Howard Gobioff Shun-Tak...

Documents

Transcript of The Google File System Presenter: Gladon Almeida Authors: Sanjay Ghemawat Howard Gobioff Shun-Tak...