Caching in Distributed File System

Ke WangCS614 – Advanced System

Apr 24, 2001

Key requirements of distributed system Scalability from small to large networks Fast and transparent access to

geographically Distributed File System(DFS)

Information protection Ease of administration Wide support from variety of vendors

Background DFS -- a distributed implementation of a

file system, where multiple users share files and storage resources.

Overall storage space managed by a DFS is composed of different, remotely located, smaller storage spaces

There is usually a correspondence between constituent storage spaces and sets of files

DFS Structure Service - a software entity providing a

particular type of function to client Server - service software running on a

single machine Client - process that can invoke a

service using a set of operations that forms its client interface

Why caching? Retaining most recently accessed disk

blocks. Repeated accesses to a block in cache

can be handled without involving the disk.

Advantages - Reduce delays

- Reduce contention for disk arm

Caching in DFS Advantages

Reduce network traffic Reduce server contention

Problems Cache-consistency

Stuff to consider Cache location (disk vs. memory) Cache Placement (client vs. server) Cache structure (block vs. file) Stateful vs. Stateless server Cache update policies Consistency Client-driven vs. Server-driven protocols

Practical Distributed System NFS: Sun’s Network File System AFS: Andrew File System (CMU) Sprite FS: File System for the

Sprite OS ( UC Berkeley)

Sun’s Network File System(NFS)

Sun’s Network File System(NFS) Originally released in 1985 Build on top of an unreliable

datagram protocol UDP (change to TCP now)

Client-server model

Andrew File System(AFS) Developed at CMU since 1983 Client-server model Key software: Vice and Venus Goal: high scalability (5,000-

10,000 nodes)

Andrew File System(AFS)

Andrew File System(AFS) VICE is a multi-threaded server process

with each thread handling a single client request

VENUS is the client process that runs on each workstation which forms the interface with VICE

User-level processes

Prototype of AFS One process for one client Client cache file Verify timestamp every open

-> a lot of interaction with server -> heavy network traffic

Improve AFS To improve prototype

Reduce cache validity check Reduce server processes Reduce network traffic

Higher scalability!

Sprite File System Designed for networked

workstation with large physical memories

(can be diskless) Expect memory of 100-500Mbytes Goal: high performance

Caches in Sprite FS

Caches in Sprite FS(cont) When a process makes a file access, it

is presented first to the cache(file traffic). If not satisfied, request is passed either to a local disk, if the file is stored locally(disk traffic), or to the server where the file is stored(server traffic). Servers also maintain caches to reduce disk traffic.

Caching in Sprite FS Two unusual aspects

Guarantee complete consistent view

Concurrent write sharing Sequential write sharing

Cache size varies dynamically

Cache LocationDisk vs. Main Memory Advantages of disk caches

More Reliable Cached data are still there during

recovery and don’t need to be fetched again

Cache LocationDisk vs. Main Memory(cont) Advantages of main-memory caches:

Permit workstations to be diskless More quick access Server caches(used to speed up disk

I/O) are always in main memory; using main-memory caches on the clients permits a single caching mechanism for servers and users

Cache PlacementClient vs. Server Client cache reduce network traffic

Read-only operations on unchanged files do not need go over the network

Server cache reduce server load Cache is amortized across all clients

( but needs to be bigger to be effective)

In practice, need BOTH!

Cache structure Block basis

Simple Sprite FS, NFS

File basis Reduce interaction with servers

AFS Cannot access files larger than cache

Compare NFS: client memory(disk), block

basis AFS: client disk, file basis Sprint FS: client memory, server

memory, block basis

Stateful vs. Stateless Server Stateful – Servers hold information

about the client

Stateless – Servers maintain no state information about clients

Stateful Servers Mechanism

Client opens a file Server fetches information about the

file from its disk, store in memory, gives client a unique connection id and open file

id is used for subsequent accesses until the session ends

Stateful Servers(cont) Advantages:

Fewer disk access Read-ahead possible RPCs are small, contains only an id File may be cached entirely on client,

invalidated by the server if there is a conflicting write

Stateful Servers(cont) Disadvantage:

Server loses all its volatile state in crash

Restore state by dialog with clients, or abort operations that underway when crash occurred

Server needs to be aware of client failures

Stateless Server Each request must be self-

contained Each request identifies the file and

position in the file No need to establish and terminate

a connection by open and close operations

Stateless Server(cont) Advantage

A file server crash does not affect clients

Simple Disadvantage

Impossible to enforce consistency RPC needs to contain all state, longer

Stateful vs. Stateless AFS and Sprite FS are stateful

Sprite FS servers keep track of which clients have which files open

AFS servers keep track of the contents of client’s caches

NFS is stateless

Cache Update Policy Write-through

Delayed-write

Write-on-close (variation of delayed-write)

Cache Update Policy(cont) Write-through – all writes be

propagated to stable storage immediately

Reliable, but poor performance

Cache Update Policy(cont) Delayed-write – modification

written to cache and then written through to server later

Write-on-close – modification written back to server when file close Reduces intermediate read and write

traffic while file is open

Cache Update Policy(cont) Pros for delayed-write/write-on-close

Lots of files have lifetimes of less than 30s Redundant writes are absorbed Lots of small writes can be batched into

larger writes Disadvantage:

Poor reliability; unwritten data may be lost when client crash

Caching in AFS Key to Andrew’s scalability Client cache entire file in disk Write-on-close

Server load and network traffic reduced Contacts server only on open and close Retain across reboots Require local disk, large enough

Cache update policy NFS and Sprite delayed-write

Delay 30 seconds AFS write-on-close

Reduce traffic to server dramatically Good scalability of AFS

Consistency Is locally cached copy of data

consistent with the master copy?

Is there danger of “stale” data?

Permit concurrent write sharing?

Sprite:Complete Consistency Concurrent Write Share

A file open on multiple clients At least one client write

Server detects Require write back to server Invalidate open cache

Sprite:Complete Consistency Sequential Write Sharing

A file modified, closed, opened by others

Out-of-date blocks Compare version number with server

Current data in other’s cache Keep track of last writer

AFS: session semantics Session semantics in AFS

Writes to an open file invisible to others

Once file closed, changes visible to new opens anywhere

Other file operations visible immediately

Only guarantee sequential consistency

Consistency Sprite guarantees complete

consistency AFS uses session semantics NFS not guarantee consistency

NFS is stateless. All operations involve contacting the server; if server is unreachable, read & write cannot work

Client-driven vs. Server-driven Client-driven approach

Client initiates validity check Server check whether the local data

are consistent with master copy Server-driven approach

Server records files client caches When server detect inconsistency, it

must react

AFS: server-driven Callback (key to scalability)

Cache valid if have callback on Server notify before modification When reboot, all suspect reduces cache validation

requests to server

Client-driven vs. Server-driven AFS is server-driven (callback)

Contributes to AFS’s scalability Whole file caching and session

semantics also help NFS and Sprite are client-driven

Increased load on network and server

AFS:Effect on scalability

Sprite:Dynamic cache size Make client cache as large as

possible Virtual memory and file system

negotiate Compare age of oldest page Two problems

Double caching Multiblock pages

Why not callback in Sprite?

Why not callback in Sprite? Estimated improvement is small Reason

Andrew is user-level process Sprite is kernel-level implementation

Comparison

Performance – running time

Performance – running time Use Andrew benchmark Sprite system is fastest

Kernel-to-kernel PRC Delayed write Kernel implementation (AFS is user-

level)

Performance – CPU utilization

Performance – CPU utilization Use Andrew benchmark Andrew system showed greatest

scalability File-based cache Server-driven Use of callback

Nomadic Caching New issues

If client become disconnected? Weakly connected(by modem)?

Violate key property: transparency!

Nomadic Caching Cache misses may impede

progress Local update invisible remotely Update conflict Update vulnerable to loss, damage

Coda file system