Self-Securing Storage:

Protecting Data in Compromised Systems

John D. Strunk, Garth R. Goodson, Michael L. Scheinholtz, Craig A.N. Soules

Gregory R. Ganger

Carnegie Mellon University

Abstract

Self-securing storage prevents intruders from unde-tectably tampering with or permanently deletingstored data. To accomplish this, self-securing stor-age devices internally audit all requests and keepold versions of data for a window of time, regard-less of the commands received from potentially com-promised host operating systems. Within the win-dow, system administrators have this valuable in-formation for intrusion diagnosis and recovery. Ourimplementation, called S4, combines log-structuringwith journal-based metadata to minimize the per-formance costs of comprehensive versioning. Exper-iments show that self-securing storage devices candeliver performance that is comparable with conven-tional storage systems. In addition, analyses indi-cate that several weeks worth of all versions can rea-sonably be kept on state-of-the-art disks, especiallywhen di�erencing and compression technologies areemployed.

1 Introduction

Despite the best e�orts of system designers and im-plementors, it has proven diÆcult to prevent com-puter security breaches. This fact is of growing im-portance as organizations �nd themselves increas-ingly dependent on wide-area networking (providingmore potential sources of intrusions) and computer-maintained information (raising the signi�cance ofpotential damage). A successful intruder can obtainthe rights and identity of a legitimate user or admin-istrator. With these rights, it is possible to disruptthe system by accessing, modifying, or destroyingcritical data.

Even after an intrusion has been detected and termi-nated, system administrators still face two diÆculttasks: determining the damage caused by the intru-sion and restoring the system to a safe state. Dam-age includes compromised secrets, creation of backdoors and Trojan horses, and tainting of stored data.Detecting each of these is made diÆcult by crafty in-truders who understand how to scrub audit logs and

disrupt automated tamper detection systems. Sys-tem restoration involves identifying a clean backup(i.e., one created prior to the intrusion), reinitializ-ing the system, and restoring information from thebackup. Such restoration often requires a signi�-cant amount of time, reduces the availability of theoriginal system, and frequently causes loss of datacreated between the safe backup and the intrusion.

Self-securing storage o�ers a partial solution to theseproblems by preventing intruders from undetectablytampering with or permanently deleting stored data.Since intruders can take the identity of real users andeven the host OS, any resource controlled by the op-erating system is vulnerable, including the raw stor-age. Rather than acting as slaves to host OSes, self-securing storage devices view them, and their users,as questionable entities for which they work. Theseself-contained, self-controlled devices internally ver-sion all data and audit all requests for a guaranteedamount of time (e.g., a week or a month), thus pro-viding system administrators time to detect intru-sions. For intrusions detected within this window,all of the version and audit information is availablefor analysis and recovery. The critical di�erence be-tween self-securing storage and host-controlled ver-sioning (e.g., Elephant [29]) is that intruders can nolonger bypass the versioning software by compromis-ing complex OSes or their poorly-protected user ac-counts. Instead, intruders must compromise single-purpose devices that export only a simple storageinterface, and in some con�gurations, they may haveto compromise both.

This paper describes self-securing storage and ourimplementation of a self-securing storage server,called S4. A number of challenges arise when stor-age devices distrust their clients. Most importantly,it may be diÆcult to keep all versions of all data foran extended period of time, and it is not acceptableto trust the client to specify what is important tokeep. Fortunately, storage densities increase fasterthan most computer characteristics (100%+ per an-num in recent years). Analysis of recent workloadstudies [29, 34] suggests that it is possible to ver-

sion all data on modern 30{100GB drives for severalweeks. Further, aggressive compression and cross-version di�erencing techniques can extend the intru-sion detection window o�ered by self-securing stor-age devices. Other challenges include eÆciently en-coding the many metadata changes, achieving secureadministrative control, and dealing with denial-of-service attacks.

The S4 system addresses these challenges with anew storage management structure. Speci�cally, S4uses a log-structured object system for data ver-sions and a novel journal-based structure for meta-data versions. In addition to reducing space utiliza-tion, journal-based metadata simpli�es backgroundcompaction and reorganization for blocks sharedacross many versions. Experiments with S4 showthat the security and data survivability bene�ts ofself-securing storage can be realized with reason-able performance. Speci�cally, the performance ofS4-enhanced NFS is comparable to FreeBSD's NFSfor both micro-benchmarks and application bench-marks. The fundamental costs associated with self-securing storage degrade performance by less than13% relative to similar systems that provide no dataprotection guarantees.

The remainder of this paper is organized as follows.Section 2 discusses intrusion survival and recoverydiÆculties in greater detail. Section 3 describes howself-securing storage addresses these issues, identi-�es some challenges inherent to self-securing storage,and discusses design solutions for addressing them.Section 4 describes the implementation of S4. Sec-tion 5 evaluates the performance and capacity over-heads of self-securing storage. Section 6 discussesa number of issues related to self-securing storage.Section 7 discusses related work. Section 8 summa-rizes this paper's contributions.

2 Intrusion Diagnosis and Recovery

Upon gaining access to a system, an intruder hasseveral avenues of mischief. Most intruders attemptto destroy evidence of their presence by erasing ormodifying system log �les. Many intruders also in-stall back doors in the system, allowing them to gainaccess at will in the future. They may also installother software, read and modify sensitive �les, oruse the system as a platform for launching addi-tional attacks. Depending on the skill with whichthe intruders hide their presence, there will be somedetection latency before the intrusion is discoveredby an automated intrusion detection system (IDS)or by a suspicious user or administrator. During this

time, the intruders can continue their malicious ac-tivities while users continue to use the system, thusentangling legitimate changes with those of the in-truders. Once an intrusion has been detected anddiscontinued, the system administrator is left withtwo diÆcult tasks: diagnosis and recovery.

Diagnosis is challenging because intruders can usu-ally compromise the \administrator" account onmost operating systems, giving them full controlover all resources. In particular, this gives themthe ability to manipulate everything stored on thesystem's disks, including audit logs, �le modi�ca-tion times, and tamper detection utilities. Recov-ery is diÆcult because diagnosis is diÆcult and be-cause user-convenience is an important issue. Thissection discusses intrusion diagnosis and recovery ingreater detail, and the next section describes howself-securing storage addresses them.

2.1 Diagnosis

Intrusion diagnosis consists of three phases: detect-ing the intrusion, discovering what weaknesses wereexploited (for future prevention), and determiningwhat the intruder did. All are diÆcult when theintruder has free reign over storage and the OS.

Without the ability to protect storage from compro-mised operating systems, intrusion detection maybe limited to alert users and system administratorsnoticing odd behavior. Examining the system logsis the most common approach to intrusion detec-tion [7], but when intruders can manipulate the log�les, such an approach is not useful. Some intrusiondetection systems also look for changes to importantsystem �les [16]. Such systems are vulnerable to in-truders that can change what the IDS thinks is a\safe" copy.

Determining how an intruder compromised the sys-tem is often impossible in conventional systems, be-cause he will scrub the system logs. In addition,any exploit tools (utilities for compromising com-puter systems) that may have been stored on thetarget machine for use in multi-stage intrusions areusually deleted. The common \solutions" are to tryto catch the intruder in the act or to hope that heforgot to delete his exploit tools.

The last step in diagnosing an intrusion is to discoverwhat was accessed and modi�ed by the intruder.This is diÆcult, because �le access and modi�ca-tion times can be changed and system log �les canbe doctored. In addition, checksum databases areof limited use, since they are e�ective only for static�les.

2.2 Recovery

Because it is usually not possible to diagnose anintruder's activities, full system recovery generallyrequires that the compromised machine be wipedclean and reinstalled from scratch. Prior to erasingthe entire state of the system, users may insist thatdata, modi�ed since the intrusion, be saved. Themore e�ort that went into creating the changes, themore motivation there is to keep this data. Unfortu-nately, as the size and complexity of the data grows,the likelihood that tampering will go unnoticed in-creases. Foolproof assessment of the modi�ed datais very diÆcult, and overlooked tampering may hidetainted information or a back door inserted by theintruder.

Upon restoring the OS and any applications on thesystem, the administrator must identify a backupthat was made prior to the intrusion; the most re-cent backup may not be usable. After restoring datafrom a pre-intrusion backup, the legitimately mod-i�ed data can be restored to the system, and usersmay resume using the system. This process oftentakes a considerable amount of time|time duringwhich users are denied service.

3 Self-Securing Storage

Self-securing storage ensures information survivaland auditing of all accesses by establishing a secu-rity perimeter around the storage device. Conven-tional storage devices are slaves to host operatingsystems, relying on them to protect users' data. Aself-securing storage device operates as an indepen-dent entity, tasked with the responsibility of not onlystoring data, but protecting it. This shift of stor-age security functionality into the storage device's�rmware allows data and audit information to besafeguarded in the presence of �le server and clientsystem intrusions. Even if the OSes of these sys-tems are compromised and an intruder is able toissue commands directly to the self-securing storagedevice, the new security perimeter remains intact.

Behind the security perimeter, the storage deviceensures data survival by keeping previous versionsof the data. This history pool of old data versions,combined with the audit log of accesses, can be usedto diagnose and recover from intrusions. This sec-tion discusses the bene�ts of self-securing storageand several core design issues that arise in realizingthis type of device.

3.1 Enabling intrusion survival

Self-securing storage assists in intrusion recovery byallowing the administrator to view audit informationand quickly restore modi�ed or deleted �les. Theaudit and version information also helps to diagnoseintrusions and detect the propagation of maliciouslymodi�ed data.

Self-securing storage simpli�es detection of an in-trusion since versioned system logs cannot be im-perceptibly altered. In addition, modi�ed systemexecutables are easily noticed. Because of this, self-securing storage makes conventional tamper detec-tion systems obsolete.

Since the administrator has the complete picture ofthe system's state, from intrusion until discovery, itis considerably easier to establish the method usedto gain entry. For instance, the system logs wouldhave normally been doctored, but by examining theversioned copies of the logs, the administrator cansee any messages that were generated during the in-trusion and later removed. In addition, any exploittools temporarily stored on the system can be recov-ered.

Previous versions of system �les, from before theintrusion, can be quickly and easily restored by res-urrecting them from the history pool. This preventsthe need for a complete re-installation of the operat-ing system, and it does not rely on having a recentbackup or up-to-date checksums (for tamper detec-tion) of system �les. After such restoration, criticaldata can be incrementally recovered from the historypool. Additionally, by utilizing the storage device'saudit log, it is possible to assess which data mighthave been directly a�ected by the intruder.

The data protection that self-securing storage pro-vides allows easy detection of modi�cations, selec-tive recovery of tampered �les, prevention of dataloss due to out-of-date backups, and speedy recov-ery since data need not be loaded from an o�-linearchive.

3.2 Device security perimeter

The device's security model is what makes the abil-ity to keep old versions more than just a user con-venience. The security perimeter consists of self-contained software that exports only a simple stor-age interface to the outside world and veri�es eachcommand's integrity before processing it. In con-trast, most �le servers and client machines run amultitude of services that are susceptible to attack.Since the self-securing storage device is a single-

function device, the task of making it secure is mucheasier; compromising its �rmware is analogous tobreaking into an IDE or SCSI disk.

The actual protocol used to communicate with thestorage device does not a�ect the data integrity thatthe new security perimeter provides. The choice ofprotocol does, however, a�ect the usefulness of theaudit log in terms of the actions it can record and itscorrectness. For instance, the NFS protocol providesno authentication or integrity guarantees, thereforethe audit log may not be able to accurately linka request with its originating client. Nonetheless,the principles of self-securing storage apply equallyto \enhanced" disk drives, network-attached storageservers, and �le servers.

For network-attached storage devices (as opposed todevices attached directly to a single host system),the new security perimeter becomes more useful ifthe device can verify each access as coming fromboth a valid user and a valid client. Such veri�cationallows the device to enforce access control decisionsand partially track propagation of tainted data. Ifclients and users are authenticated, accesses can betracked to a single client machine, and the device'saudit log can yield the scope of direct damage fromthe intrusion of a given machine or user account.

3.3 History pool management

The old versions of objects kept by the device com-prise the history pool. Every time an object ismodi�ed or deleted, the version that existed justprior to the modi�cation becomes part of the his-tory pool. Eventually an old version will age andhave its space reclaimed. Because clients cannot betrusted to demarcate versions consisting of multiplemodi�cations, a separate version should be kept forevery modi�cation. This is in contrast to versioning�le systems that generally create new versions onlywhen a �le is closed.

A self-securing storage device guarantees a lowerbound on the amount of time that a deprecated ob-ject remains in the history pool before it is reclaimed.During this window of time, the old version of theobject can be completely restored by requesting thatthe drive copy forward the old version, thus mak-ing a new version. The guaranteed window of timeduring which an object can be restored is called thedetection window. When determining the size of thiswindow, the administrator must examine the trade-o� between the detection latency provided by a largewindow and the extra disk space that is consumedby the proportionally larger history pool.

Although the capacity of disk drives is growing ata remarkable rate, it is still �nite, which poses twoproblems:

1. Providing a reasonable detection window in ex-ceptionally busy systems.

2. Dealing with malicious users that attempt to�ll the history pool. (Note that space ex-haustion attacks are not unique to self-securingstorage. However, device-managed versioningmakes conventional user quotas ine�ective forlimiting them.)

In a busy system, the amount of data written couldmake providing a reasonable detection window dif-�cult. Fortunately, the analysis in Section 5.2 sug-gests that multi-week detection windows can be pro-vided in many environments at a reasonable cost.Further, aggressive compression and di�erencing ofold versions can signi�cantly extend the detectionwindow.

Deliberate attempts to over ow the history pool can-not be prevented by simply increasing the spaceavailable. As with most denial of service attacks,there is no perfect solution. There are three awedapproaches to addressing this type of abuse. The�rst is to have the device reclaim the space held bythe oldest objects when the history pool is full. Un-fortunately, this would allow an intruder to destroyinformation by causing its previous instances to bereclaimed from the over owing history pool. Thesecond awed approach is to stop versioning objectswhen the history pool �lls; although this will allowrecovery of old data, system administrators wouldno longer be able to diagnose the actions of an in-truder or di�erentiate them from subsequent legiti-mate changes. The third awed approach is for thedrive to deny any action that would require addi-tional versions once the history pool �lls; this wouldresult in denial of service to all users (legitimate ornot).

Our hybrid approach to this problem is to try to pre-vent the history pool from being �lled by detectingprobable abuses and throttling the source machine'saccesses. This allows human intervention beforethe system is forced to choose from the above pooralternatives. Selectively increasing latency and/ordecreasing bandwidth allows well-behaved users tocontinue to use the system even while it is under at-tack. Experience will show how well this works inpractice.

Since the history pool will be used for intrusion di-agnosis and recovery, not just recovering from acci-

dental destruction of data, it is diÆcult to constructa safe algorithm that would save space in the his-tory pool by pruning versions within the detectionwindow. Almost any algorithm that selectively re-moves versions has the potential to be abused byan intruder to cover his tracks and to successfullydestroy/modify information during a break-in.

3.4 History pool access control

The history pool contains a wealth of informationabout the system's recent activity. This makes ac-cessing the history pool a sensitive operation, sinceit allows the resurrection of deleted and overwrit-ten objects. This is a standard problem posed byversioning �le systems, but is exacerbated by theinability to selectively delete versions.

There are two basic approaches that can be takentoward access control for the history pool. The �rstis to allow only a single administrative entity to havethe power to view and restore items from the historypool. This could be useful in situations where theold data is considered to be highly sensitive. Havinga single tightly-controlled key for accessing historicaldata decreases the likelihood of an intruder gainingaccess to it. Although this improves security, it pre-vents users from being able to recover from theirown mistakes, thus consuming the administrator'stime to restore users' �les. The second approachis to allow users to recover their own old objects(in addition to the administrator). This providesthe convenience of a user being able to recover theirdeleted data easily, but also allows an intruder, whoobtains valid credentials for a given user, to recoverthat user's old �le versions.

Our compromise is to allow users to selectively makethis decision. By choice, a user could thus delete anobject, version, or all versions from visibility by any-one other than the administrator, since permanentdeletion of data via any other method than agingwould be unsafe. This choice allows users to en-joy the bene�ts of versioning for presentations andsource code, while preventing access to visible ver-sions of embarrassing images or unsent e-mail drafts.

3.5 Administrative access

A method for secure administrative access is neededfor the necessary but dangerous commands that aself-securing storage device must support. Suchcommands include setting the guaranteed detectionwindow, erasing parts of the history pool, and ac-cessing data that users have marked as \unrecover-able." Such administrative access can be securely

granted in a number of ways, including physical ac-cess (e.g., ipping a switch on the device) or well-protected cryptographic keys.

Administrative access is not necessary for users at-tempting to recover their own �les from accidents.Users' accesses to the history pool should be han-dled with the same form of protection used for theirnormal accesses. This is acceptable for user activity,since all actions permitted for ordinary users can beaudited and repaired.

3.6 Version and administration tools

Since self-securing storage devices store versions ofraw data, users and administrators will need assis-tance in parsing the history pool. Tools for travers-ing the history must assist by bridging the gap be-tween standard �le interfaces and the raw versionsthat are stored by the device. By being aware ofboth the versioning system and formats of the dataobjects, utilities can present interfaces similar tothat of Elephant [29], with \time-enhanced" versionsof standard utilities such as ls and cp. This is ac-complished by extending the read interfaces of thedevice to include an optional time parameter. Whenthis parameter is speci�ed, the drive returns datafrom the version of the object that was valid at therequested time.

In addition to providing a simple view of data ob-jects in isolation, intrusion diagnosis tools can utilizethe audit log to provide an estimate of damage. Forinstance, it is possible to see all �les and directo-ries that a client modi�ed during the period of timethat it was compromised. Further estimates of thepropagation of data written by compromised clientsare also possible, though imperfect. For example,diagnosis tools may be able to establish a link be-tween objects based on the fact that one was readjust before another was written. Such a link betweena source �le and its corresponding object �le wouldbe useful if a user determines that a source �le hadbeen tampered with; in this situation, the object �leshould also be restored or removed. Exploration ofsuch tools will be an important area of future work.

4 S4 Implementation

S4 is a self-securing storage server that transpar-ently maintains an eÆcient object-versioning systemfor its clients. It aims to perform comparably withcurrent systems, while providing the bene�ts of self-securing storage and minimizing the correspondingspace explosion.

RPC Type Allows

Time-Based

Access

Description

Create no Create an object

Delete no Delete an object

Read yes Read data from an object

Write no Write data to an object

Append no Append data to the end of an object

Truncate no Truncate an object to a speci�ed length

GetAttr yes Get the attributes of an object (S4-speci�c and opaque)

SetAttr no Set the opaque attributes of an object

GetACLByUser yes Get an ACL entry for an object given a speci�c UserID

GetACLByIndex yes Get an ACL entry for an object by its index in the object's ACL

table

SetACL no Set an ACL entry for an object

PCreate no Create a partition (associate a name with an ObjectID)

PDelete no Delete a partition (remove a name/ObjectID association)

PList yes List the partitions

PMount yes Retrieve the ObjectID given its name

Sync not applicable Sync the entire cache to disk

Flush not applicable Removes all versions of all objects between two times

FlushO not applicable Removes all versions of an object between two times

SetWindow not applicable Adjusts the guaranteed detection window of the S4 device

Table 1: S4 Remote Procedure Call List { Operations that support time-based access accept a time in addition to the

normal parameters; this time is used to �nd the appropriate version in the history pool. Note that all modi�cations create new

versions without a�ecting the previous versions.

4.1 A self-securing object store

S4 is a network-attached object store with an in-terface similar to recent object-based disk propos-als [9, 24]. This interface simpli�es access controland internal performance enhancement relative to astandard block interface.

In S4, objects exist in a at namespace managedby the \drive" (i.e., the object store). When ob-jects are created, they are given a unique identi�er(ObjectID) by the drive, which is used by the clientfor all future references to that object. Each objecthas an access control structure that speci�es whichentities (users and client machines) have permissionto access the object. Objects also have metadata, �ledata, and opaque attribute space (for use by client�le systems) associated with them.

To enable persistent mount points, a S4 drive sup-ports \named objects." The object names are anassociation of an arbitrary ASCII string with a par-ticular ObjectID. The table of named objects is im-plemented as a special S4 object accessed through

dedicated partition manipulation RPC calls. Thistable is versioned in the same manner as other ob-jects on the S4 drive.

4.1.1 S4 RPC interface

Table 1 lists the RPC commands supported by theS4 drive. The read-only commands (read, getattr,getacl, plist, and pmount) accept an optional time

parameter. When the time is provided, the driveperforms the read request on the version of the ob-ject that was \most current" at the time speci�ed,provided that the user making the request has suÆ-cient privileges.

The ACLs associated with objects have the tradi-tional set of ags, with one addition|the Recovery ag. The Recovery ag determines whether or not agiven user may read (recover) an object version fromthe history pool once it is overwritten or deleted.When this ag is clear, only the device administratormay read this object version once it is pushed into

the history pool. The Recovery ag allows users todecide the sensitivity of old versions on a �le-by-�lebasis.

4.1.2 S4/NFS translation

Since one goal of self-securing storage is to providean enhanced level of security and convenience onexisting systems, the prototype minimizes changesto client systems. In keeping with this philosophy,the S4 drive is network-attached and an \S4 client"daemon serves as a user-level �le system transla-tor (Figure 1a). The S4 client translates requestsfrom a �le system on the target OS to S4-speci�crequests for objects. Because it runs as a user-levelprocess, without operating systemmodi�cations, theS4 client should port to di�erent systems easily.

The S4 client currently has the ability to trans-late NFS version 2 requests to S4 requests. TheS4 client appears to the local workstation as a NFSserver. This emulated NFS server is mounted viathe loopback interface to allow only that worksta-tion access to the S4 client. The client receives theNFS requests and translates them into S4 opera-tions. NFSv2 was chosen over version 3 because itsclient is well-supported within Linux, and its lack ofwrite caching allows the drive to maintain a detailedaccount of client actions.

Figure 1 shows two approaches to using the S4 clientto serve NFS requests with the S4 drive. The �rstplaces the S4 client on the client system, as describedpreviously, and uses the S4 drive as a network-attached storage device. The second incorporatesthe S4 client functionality into the server, as a NFS-to-S4 translator. This con�guration acts as a S4-enhanced NFS server (Figure 1b) for normal �le sys-tem activity, but recovery must still be accomplishedthrough the S4 protocol since the NFS protocol hasno notion of \time-based" access.

The implementation of the NFS �le system overlays�les and directories on top of S4 objects. Objectsused as directories contain a list of ASCII �lenamesand their associated NFS �le handles. Objects usedas �les and symlinks contain the corresponding data.The NFS attribute structure is maintained withinthe opaque attribute space of each object.

When the S4 client receives a NFS request, the NFS�le handle (previously constructed by the S4 client)can be directly hashed into the ObjectID of the di-rectory or �le. The S4 client can then make requestsdirectly to the drive for the desired data.

S4 DriveApplicationClient

Kernel

Client

S4 Client

S4 RPCNFS

Kernel

RPC

S4 RPC

Drive

(a) Baseline S4 (network-attached object store)

Client Drive

Kernel Kernel

Application S4 Drive

NFS

S4 RPCNFS S4 RPC

Client S4-NFSTranslator

(b) S4-enhanced NFS server

Figure 1: Two S4 Con�gurations { This �gure shows twoS4 con�gurations that provide self-securing storage via a NFS

interface. (a) shows S4 as a network-attached object store

with the S4 client daemon translating NFS requests to S4-

speci�c RPCs. (b) shows a self-securing NFS server created

by combining the NFS-to-S4 translation and the S4 drive.

To support NFSv2 semantics, the client sends an ad-ditional RPC to the drive to ush bu�ered writes tothe disk at the end of each NFS operation that mod-i�es the state of one or more objects. Since this RPCdoes not return until the synchronization is com-plete, NFSv2 semantics are supported even thoughthe drive normally caches writes.

Because the client overlays a �le system on top ofthe at object namespace, some �le system oper-ations require several drive operations (and henceRPC calls). These sets of operations are analogousto the operations that �le systems must performon block-based devices. To minimize the numberof RPC calls necessary, the S4 client aggressivelymaintains attribute and directory caches (for readsonly). The drive also supports batching of setattr,getattr, and sync operations with create, read,write, and append operations.

4.2 S4 drive internals

The main goals for the S4 drive implementationare to avoid performance overhead and to minimizewasted space, while keeping all versions of all objectsfor a given period of time. Achieving these goals re-

quires a combination of known and novel techniquesfor organizing on-disk data.

4.2.1 Log-structuring for eÆcient writes

Since data within the history pool cannot be over-written, the S4 drive uses a log structure similar toLFS [27]. This structure allows multiple data andmetadata updates to be clustered into fewer, largerwrites. Importantly, it also obviates any need tomove previous versions before writing.

In order to prune old versions and reclaim unusedsegments, S4 includes a background cleaner. Whilethe goal of this cleaner is similar to that of the LFScleaner, the design must be slightly di�erent. Specif-ically, deprecated objects cannot be reclaimed unlessthey have also aged out of the history pool. There-fore, the S4 cleaner searches through the object mapfor objects with an oldest time greater than the de-tection window. Once a suitable object is found,the cleaner permanently frees all data and meta-data older than the window. If this clears all ofthe resources within a segment, the segment can bemarked as free and used as a fresh segment for fore-ground activity.

4.2.2 Journal-based metadata

To eÆciently keep all versions of object metadata,S4 uses journal-based metadata, which replaces mostinstances of metadata with compact journal entries.

Because clients are not trusted to notify S4 whenobjects are closed, every update creates a new ver-sion and thus new metadata. For example, whendata pointed to by indirect blocks is modi�ed, theindirect blocks must be versioned as well. In a con-ventional versioning system, a single update to atriple-indirect block could require four new blocksas well as a new inode. Early experiments with thistype of versioning system showed that modifying alarge �le could cause up to a 4x growth in disk us-age. Conventional versioning �le systems avoid thisperformance problem by only creating new versionswhen a �le is closed.

In order to signi�cantly reduce these problems, S4encodes metadata changes in a journal that is main-tained for the duration of the detection window. Bypersistently keeping journal entries of all metadatachanges, metadata writes can be safely delayed andcoalesced, since individual inode and indirect blockversions can be recreated from the journal. To avoid

����������������

������������

������������

BlockData

BlockIndirect

BlockData

BlockIndirect

BlockData

BlockIndirect

LogInode

BlockNew

IndirectNew

InodeNew

BlockNew

SectorJournal

Inode

InodeInode

Original state

Journal-based MetadataConventional Versioning

Figure 2: EÆciency of Metadata Versioning { The

above �gure compares metadata management in a conven-

tional versioning system to S4's journal-based metadata ap-

proach. When writing to an indirect block, a conventional

versioning system allocates a new data block, a new indirect

block, and a new inode. Also, the identity of the new in-

ode must be recorded (e.g., in an Elephant-like inode log).

With journal-based metadata, a single journal entry suÆces,

pointing to both the new and old data blocks.

rebuilding an object's current state from the jour-nal during normal operation, an object's metadatais checkpointed to a log segment before being evictedfrom the cache. Unlike conventional journaling, suchcheckpointing does not prune journal space; only ag-ing may prune space. Figure 2 depicts the di�erencein disk space usage between journal-based metadataand conventional versioning when writing data to anindirect data block.

In addition to the entries needed to describe meta-data changes, a checkpoint entry is needed. Thischeckpoint entry denotes writing a consistent copyof all of an object's metadata to disk. It is necessaryto have at least one checkpoint of an object's meta-data on disk at all times, since this is the startingpoint for all time-based and crash recovery recre-ations.

Storing an object's changes within the log is done us-ing journal sectors. Each journal sector contains thepacked journal entries that refer to a single object'schanges made within that segment. The sectors areidenti�ed by segment summary information. Jour-nal sectors are chained together backward in time toallow for version reconstruction.

Journal-based metadata can also simplify cross-version di�erential compression [3]. Since the blockschanged between versions are noted within each en-try, it is easy to �nd the blocks that should be com-

pared. Once the di�erencing is complete, the oldblocks can be discarded, and the di�erence left inits place. For subsequent reads of old versions, thedata for each block must be recreated as the en-tries are traversed. Cross-version di�erencing of olddata will often be e�ective in reducing the amountof space used by old versions. Adding di�erencingtechnology into the S4 cleaner is an area of futurework.

4.2.3 Audit log

In addition to maintaining previous object versions,S4 maintains an append-only audit log of all re-quests. This log is implemented as a reserved objectwithin the drive that cannot be modi�ed except bythe drive itself. However, it can be read via RPC op-erations. The data written to the audit log includescommand arguments as well as the originating clientand user. All RPC operations (read, write, and ad-ministrative) are logged. Since the audit log mayonly be written by the drive front end, it need notbe versioned, thus increasing space eÆciency and de-creasing performance costs.

5 Evaluation of self-securing storage

This section evaluates the feasibility of self-securingstorage. Experiments with S4 indicate that compre-hensive versioning and auditing can be performedwithout a signi�cant performance impact. Also, es-timates of capacity growth, based on reported work-load characterizations, indicate that history win-dows of several weeks can easily be supported inseveral real environments.

5.1 Performance

The main performance goal for S4 is to be compara-ble to other networked �le systems while o�ering en-hanced security features. This section demonstratesthat this goal is achieved and also explores the over-heads speci�cally associated with self-securing stor-age features.

5.1.1 Experimental Setup

The four systems used in the experiments had thefollowing con�gurations: (1) a S4 drive runningon RedHat 6.1 Linux communicating with a Linuxclient over S4 RPC through the S4 client module(Figure 1a), (2) a S4-enhanced NFS server running

on RedHat 6.1 Linux communicating with a Linuxclient over NFS (Figure 1b), (3) a FreeBSD 4.0server communicating with a Linux client over NFS,and (4) a RedHat 6.1 Linux server communicatingwith a Linux client over NFS. Since Linux NFS doesnot comply with the NFSv2 semantics of commit-ting data to stable storage before operation com-pletion, the Linux server's �le system was mountedsynchronously to approximate NFS semantics. In allcases, NFS was con�gured to use 4KB read/writetransfer sizes, the only option supported by Linux.The FreeBSD NFS con�guration exports a BSD FFS�le system, while the Linux NFS con�guration ex-ports an ext2 �le system. All experiments were run�ve times and have a standard deviation of less than3% of the mean. The S4 drives were con�gured witha 128MB bu�er cache and a 32MB object cache. TheLinux and FreeBSD NFS servers' caches could growto �ll local memory (512MB).

In all experiments, the client system has a 550MHzPentium III, 128MB RAM, and a 3Com 3C905B100Mb network adapter. The servers have a 600MHzPentium III, 512MB RAM, a 9GB 10; 000RPM Ul-tra2 SCSI Seagate Cheetah drive, an Adaptec AIC-7896/7 Ultra2 SCSI controller, and an Intel Ether-Express Pro100 100Mb network adapter. The clientand server are on the same subnet and are connectedby a 100Mb network switch. All versions of Linuxuse an unmodi�ed 2.2.14 kernel, and the BSD sys-tem uses a stock FreeBSD 4.0 installation.

To evaluate performance for common workloads,results from two application benchmarks are pre-sented: the PostMark benchmark [14] and the SSH-build benchmark [36]. These benchmarks crudelyrepresent Internet server and software developmentworkloads, respectively.

PostMark was designed to measure the performanceof a �le system used for electronic mail, netnews,and web based services. It creates a large num-ber of small randomly-sized �les (between 512B and9KB) and performs a speci�ed number of transac-tions on them. Each transaction consists of two sub-transactions, with one being a create or delete andthe other being a read or append. The default con-�guration used for the experiments consists of 20,000transactions on 5,000 �les, and the biases for trans-action type are equal.

The SSH-build benchmark was constructed as areplacement for the Andrew �le system bench-mark [12]. It consists of 3 phases: The unpack phase,which unpacks the compressed tar archive of SSHv1.2.27 (approximately 1MB in size before decom-

Transaction Time Creation Time0

100

200

300

400

500

600

700

800

900

Tim

e (S

econ

ds)

S4 S4−NFS BSD−NFS Linux−NFS

Figure 3: PostMark Benchmark

pression), stresses metadata operations on �les ofvarying sizes. The con�gure phase consists of theautomatic generation of header �les and Make�les,which involves building various small programs thatcheck the existing system con�guration. The buildphase compiles, links, and removes temporary �les.This last phase is the most CPU intensive, but italso generates a large number of object �les and afew executables.

5.1.2 Comparison of the servers

To gauge the overall performance of S4, the foursystems described earlier were compared. As hoped,S4 performs comparably to the existing NFS servers.

Figure 3 shows the results of the PostMark bench-mark. The times for both the creation (time to cre-ate the initial 5000 �les) and transaction phases ofPostMark are shown for each system. The S4 sys-tems' performance is similar to both BSD and LinuxNFS performance, doing slightly better due to theirlog structured layout.

The times of SSH-build's three phases are shownin Figure 4. Performance is similar across the S4and BSD con�gurations. The superior performanceof the Linux NFS server in the con�gure stage is dueto a much lower number of write I/Os than in theBSD and S4 servers, apparently due to a aw in thesynchronous mount option under Linux.

Unpack Configure Build0

50

100

150

Tim

e (s

econ

ds)

S4 S4−NFS BSD−NFS Linux−NFS

Figure 4: SSH-build Benchmark

5.1.3 Overhead of the S4 cleaner

In addition to the more visible process of creatingnew versions, S4 must eventually garbage collectdata that has expired from the history pool. Thisgarbage collection comes at a cost. The potentialoverhead of the cleaner was measured by runningthe PostMark benchmark with 50,000 transactionson increasingly large sets of initial �les. For each setof initial �les, the benchmark was run once with thecleaner disabled and once with the cleaner compet-ing with foreground activity.

The results shown in Figure 5 represent PostMarkrunning with the initial set of �les �lling between2% and 90% of a 2GB disk. As expected, when theworking set increases, performance of the normal S4system degrades due to increasingly poor cache anddisk locality. The sharp drop in the graph from 2%to 10% is caused by the fact that the set of �lesand data expands beyond the bounds of the drive'scache.

Although the S4 cleaner is slightly di�erent, it wasexpected to behave similarly to a standard LFScleaner, which has up to an approximate 34% de-crease in performance [30]. The S4 cleaner is slightlymore intrusive, degrading performance by approxi-mately 50% in the worst case. The greater degra-dation is attributed mainly to the additional readsnecessary when cleaning objects rather than seg-ments. In addition, the S4 cleaner has not beentuned and does not include known techniques forreducing cleaner performance problems [21].

0% 20% 40% 60% 80% 100%0

5

10

15

20

25

30

35

40

45

Tra

nsac

tions

/Sec

ond

Disk capacity utilization

No CleanerCleaner

Figure 5: Overhead of foreground cleaning in S4 {

This �gure shows the transaction performance of S4 running

the PostMark benchmark with varying capacity utilizations.

The solid line shows system performance on a system without

cleaning. The dashed line shows system performance in the

presence of continuous foreground cleaner activity.

5.1.4 Overhead of the S4 audit log

In addition to versioning, self-securing storage de-vices keep an audit log of all connections and com-mands sent to the drive. Recording this audit logof events has some cost. In the worst case, all datawritten to the disk belongs to the audit log. In thiscase, one disk write is expected approximately ev-ery 750 operations. In the best case, large writes,the audit log overhead is almost non-existent, sincethe writes of the audit log blocks are hidden inthe segment writes of the requests. For the macro-benchmarks, the performance penalty ranged be-tween 1% and 3%.

For a more focused view of this overhead, a set ofmicro-benchmarks were run with audit logging en-abled and disabled. The micro-benchmarks proceedin three phases: creation of 10,000 1KB �les (splitacross 10 directories), reads of the newly created �lesin creation order, and deletion of the �les in creationorder.

Figure 6 shows the results. The create and deletephases exhibit a 2.8% and 2.9% decrease in perfor-mance, respectively, and the read phase exhibits a7.2% decrease in performance. Read performancesu�ers a larger penalty because the audit log blocksbecome interwoven with the data blocks in the createphase. This reduces the number of �les packed intoeach segment, which in turn increases the number ofsegment reads required.

Creates Reads Deletes0

50

100

150

200

250

300

350

400

10,000 1KB File Accesses

File

s/S

econ

d

Auditing No Auditing

Figure 6: Auditing Overhead in S4 { This �gure shows

the impact on small �le performance caused by auditing in-

coming client requests.

5.1.5 Fundamental performance costs

There are three fundamental performance costsof self-securing storage: versioning, auditing, andgarbage collection. Versioning can be achieved atvirtually no cost by combining journal-based meta-data with the LFS structure. Auditing creates asmall performance penalty of 1% to 3%, accordingto application benchmarks. The �nal performancecost, garbage collection, is more diÆcult to quantify.The extra overhead of S4 cleaning in comparison tostandard LFS cleaning comes mainly from the dif-ference in utilized space due to the history pool.

The worst-case performance penalty for garbage col-lection in S4 can be estimated by comparing thecleaning overhead at two space utilizations: thespace utilized by the active set of objects and thespace utilized by the active set combined with thehistory pool. For example, assume that the activeset utilizes 60% of the drive's space and the historypool another 20%. For PostMark, the cleaning over-head is the di�erence between cleaning performanceand standard performance seen at a given space uti-lization in Figure 5. For 60% utilization, the clean-ing overhead is 43%. For 80% utilization, it is 53%.Thus, in this example, the extra cleaning overheadcaused by keeping the history pool is 10%.

There are several possibilities for reducing cleaneroverhead for all space utilizations. With expecteddetection windows ranging into the hundreds ofdays, it is likely that the history pool can be ex-tended until such a time that the drive becomes idle.During idle time, the cleaner can run with no observ-

0 100 200 300 400 500

AFS 96

HPUX 99

NT 99

Time in Days

Base DifferencingCompression

Figure 7: Projected Detection Window { The expected

detection window that could be provided by utilizing 10GB

of a modern disk drive. This conservative history pool would

consume only 20% of a 50GB disk's total capacity. The base-

line number represents the projected number of days worth of

history information that can be maintained within this 10GB

of space. The gray regions show the projected increase that

cross-version di�erencing would provide. The black regions

show the further increase expected from using compression in

addition to di�erencing.

able overhead [2]. Also, recent research into tech-nologies such as freeblock scheduling o�er standardLFS cleaning at almost no cost [18]. This techniquecould be extended for cleaning in S4.

5.2 Capacity Requirements

To evaluate the size of the detection window thatcan be provided, three recent workload studies wereexamined. Figure 7 shows the results of approxi-mations based on worst-case write behavior. Spa-sojevic and Satyanarayanan's AFS trace study [32]reports approximately 143MB per day of write traf-�c per �le server. The AFS study was conducted us-ing 70 servers (consisting of 32; 000 cells) distributedacross the wide area, containing a total of 200GB ofdata. Based on this study, using just 20% of a mod-ern 50GB disk would yield over 70 days of historydata. Even if the writes consume 1GB per day perserver, as was seen by Vogels' Windows NT �le us-age study [34], 10 days worth of history data can beprovided. The NT study consisted of 45 machinessplit into personal, shared, and administrative do-mains running workloads of scienti�c processing, de-velopment, and other administrative tasks. Santry,et al. [29] report a write data rate of 110MB perday. In this case, over 90 days of data could bekept. Their environment consisted of a single �lesystem holding 15GB of data that was being used

by a dozen researchers for development.

Much work has been done in evaluating the eÆciencyof di�erencing and compression [3, 4, 5]. To brie yexplore the potential bene�ts for S4, its code basewas retrieved from the CVS repository at a singlepoint each day for a week. After compiling the code,both di�erencing and di�erencing with compressionwere applied between each tree and its direct neigh-bor in time using Xdelta [19, 20]. After applyingdi�erencing, the space eÆciency increased by 200%.Applying compression added an additional 200% fora total space eÆciency of 500%. These results are inline with previous work. Applying these estimates tothe above workloads indicates that a 10GB historypool can provide a detection window of between 50and 470 days.

6 Discussion

This section discusses several important implicationsof self-securing storage.

Selective versioning: There are data that userswould prefer not to have backed up at all. The com-mon approach to this is to store them in directoriesknown to be skipped by the backup system. Sinceone of the goals of S4 is to allow recovery of exploittools, it does not support designating objects as non-versioned. A system may be con�gured with non-S4partitions to support selective versioning. While thiswould provide a way to prevent versioning of tempo-rary �les and other non-critical data, it would alsocreate a location where an intruder could temporar-ily store exploit tools without fear that they will berecovered.

Versioning vs. snapshots: Self-securing stor-age can be implemented with frequent copy-on-writesnapshots [11, 12, 17] instead of versioning, so longas snapshots are kept for the full detection window.Although the audit log can still provide a record ofwhat blocks are changed, snapshots often will not al-low administrators to recover short-lived �les (e.g.,exploit tools) or intermediate versions (e.g., systemlog �le updates). Also, legitimate changes are onlyguaranteed to survive malicious activity if they sur-vive to the next snapshot time. Of course, the po-tential scope of such problems can be reduced byshrinking the time between snapshots. The compre-hensive versioning promoted in this paper representsthe natural end-point of such shrinking|every mod-i�cation creates a new snapshot.

Versioning �le systems vs. self-securing stor-age: Versioning �le systems excel at providing users

with a safety net for recovery from accidents. Theymaintain old �le versions long after they would bereclaimed by the S4 system, but they provide lit-tle additional system security. This is because theyrely on the host's OS for security and aggressivelyprune apparently insigni�cant versions. By combin-ing self-securing storage with long-term landmarkversioning [28], recovery from users' accidents couldbe enhanced while also maintaining the bene�ts ofintrusion survival.

Self-securing storage for databases: Mostdatabases log all changes in order to protect internalconsistency in the face of system crashes. Some in-stitutions also retain these logs for long-term audit-ing purposes. All information needed to understandand recover from malicious behavior can be kept, indatabase-speci�c form, in these logs. Self-securingstorage can increase the post-intrusion recoverabil-ity of database systems in two ways: (1) by prevent-ing undetectable tampering with stored log records,and (2) by preventing undetectable changes to datathat bypass the log. After an intrusion, self-securingstorage allows a database system to verify its log'sintegrity and con�rm that all changes are correctlyre ected in the log|the database system can thensafely use its log for subsequent recovery.

Client-side cache e�ects: In order to improve ef-�ciency, most client systems use caches to minimizestorage latencies. This is at odds with the desireto have storage devices audit users' accesses andcapture exploit tools. Client-side read caches hidedata dependency information that would otherwisebe available to the drive in the form of reads followedquickly by writes. However, this information couldbe provided by client systems as (questionable) hintsduring writes. Write caches cause a more seriousproblem when �les are created then quickly deleted,thus never being sent to the drive. This could causediÆculties with capturing exploit tools, since theymay never be written to the drive. Although clientcache e�ects may obscure some of the activity in theclient system, data that are stored on a self-securingstorage device are still completely protected.

Object-based vs. block-based storage: Imple-menting a self-securing storage device with a blockinterface adds several diÆculties. Since objects aredesigned to contain one data item (�le or directory),enforcing access control at this level is much moremanageable than attempting to assign permissionson a per-block basis. In addition, maintaining ver-sions of objects as a whole, rather than having to col-lect and correlate individual blocks, simpli�es recov-ery tools and internal reorganization mechanisms.

Multi-device coordination: Multi-device coordi-nation is necessary for operations such as stripingdata or implementing RAID across multiple self-securing disks or �le servers. In addition to the co-ordination necessary to ensure that multiple copiesof data are synchronized, recovery operations mustalso coordinate old versions. On the other hand,clusters of self-securing storage devices could main-tain a single history pool and balance the load ofversioning objects. Note that a self-securing storagedevice containing several disks (e.g., a self-securingdisk array) does not have these issues. Additionally,it has the ability to keep old versions and currentdata on separate disks.

7 Related Work

Self-securing storage and S4 build on many ideasfrom previous work. Perhaps the clearest example isversioning: many versioned �le systems have helpedtheir users to recover from mistakes [22, 10]. Santry,et al., provide a good discussion of techniques fortraversing versions and deciding what to retain [29].S4's history pool corresponds to Elephant's \keepall" policy (during its detection window), and it usesElephant's time-based access. The primary advan-tage of S4 over such systems is that it has been par-titioned from client operating systems. While thiscreates another layer of abstraction, it adds to thesurvivability of the storage.

A self-securing disk drive would be another instanceof many recent \smart disk" systems [1, 8, 15, 26,35]. All of these exploit the increasing computationpower of such devices. Some also put these deviceson networks and exploit an object-based interface.There is now an ANSI X3T10 (SCSI) working grouplooking to create a new standard for object-basedstorage devices. The S4 interface is similar to these.

The standard method of intrusion recovery is to keepa periodic backup of �les on trusted storage. Sev-eral �le systems simplify this process by allowing asnapshot to be taken of a �le system [11, 12, 17].This snapshot can then be backed-up with standard�le system tools. Spiralog [13] uses a log-structured�le system to allow for backups to be made duringsystem operation by simply recording the entire logto tertiary storage. While these systems are e�ectivein preventing the loss of long-existing critical data,the window of time in which data can be destroyedor tampered with is much larger than in S4|often24 hours or more. Also, these systems are generallyreliant upon a system administrator for operation,with a corresponding increase in cost and potential

for human error. In addition, intrusion diagnosis isextremely diÆcult in such systems. Permanent �lestorage [25] provides an unlimited set of puncture-proof backups over time. These systems are unlikelyto become the �rst-line of storage because of lengthyaccess times.

S4 borrows on-disk data structures from several sys-tems. Unlike Elephant's FFS-like layout [23], thedisk layout of S4 more closely resembles that of alog structured �le system [27]. Many �le systemsuse journaling to improve performance while main-taining disk consistency [6, 31, 33]. However, thesesystems delete the journal information once check-points ensure that the corresponding blocks are allon disk. S4's journal-based metadata persistentlystores metadata versions in a space-eÆcient man-ner.

8 Conclusions

Self-securing storage ensures data and audit log sur-vival in the presence of successful intrusions and evencompromised host operating systems. Experimentswith the S4 prototype show that self-securing stor-age devices can achieve performance that is com-parable to existing storage appliances. In addition,analysis of recent workload studies suggest that com-plete version histories can be kept for several weekson state-of-the-art disk drives.

Acknowledgments

We thank Brian Bershad, David Petrou, Garth Gib-son, Andy Klosterman, Alistair Veitch, Jay Wylie,and the anonymous reviewers for helping us re�nethis paper. We thank the members and companiesof the Parallel Data Consortium (including CLARi-iON, EMC, HP, Hitachi, In�neon, Intel, LSI Logic,MTI, Novell, PANASAS, Procom, Quantum, Sea-gate, Sun, Veritas, and 3Com) for their interest, in-sights, and support. We also thank IBM Corpora-tion for supporting our research e�orts. This workis partially funded by the National Science Founda-tion via CMU's Data Storage Systems Center andby DARPA/ISO's Intrusion Tolerant Systems pro-gram (Air Force contract number F30602-99-2-0539-AFRL). Craig Soules is supported by a USENIXscholarship.

References

[1] Anurag Acharya, Mustafa Uysal, and Joel Saltz. Activedisks: programming model, algorithms and evaluation.Architectural Support for Programming Languages and

Operating Systems (San Jose, California), pages 81{91.ACM, 3{7 October 1998.

[2] Trevor Blackwell, Je�rey Harris, and Margo Seltzer.Heuristic cleaning algorithms in log-structured �le sys-tems. Annual USENIX Technical Conference (New Or-leans), pages 277{288. Usenix Association, 16{20 Jan-uary 1995.

[3] Randal C. Burns. Di�erential compression: a general-

ized solution for binary �les. Masters thesis. Universityof California at Santa Cruz, December 1996.

[4] M. Burrows and D. J. Wheeler. A block-sorting lossless

data compression algorithm. 124. Digital EquipmentCorporation Systems Research Center, Palo Alto, CA,10 May 1994.

[5] Michael Burrows, Charles Jerian, Butler Lampson, andTimothy Mann. On-line data compression in a log-structured �le system. Architectural Support for Pro-

gramming Languages and Operating Systems (Boston,MA, 12{15 October 1992). Published as Computer Ar-chitecture News, 20(special issue):2{9, October 1992.

[6] Sailesh Chutani, Owen T. Anderson, Michael L. Kazar,Bruce W. Leverett, W. Anthony Mason, and Robert N.Sidebotham. The Episode File System. Annual USENIXTechnical Conference (San Francisco, CA), pages 43{60,Winter 1992.

[7] Dorothy Denning. An intrusion-detection model. IEEETransactions on Software Engineering, SE-13(2):222{232, February 1987.

[8] Garth A. Gibson, David F. Nagle, Khalil Amiri, Je� But-ler, Fay W. Chang, Howard Gobio�, Charles Hardin,Erik Riedel, David Rochberg, and Jim Zelenka. A cost-e�ective, high-bandwidth storage architecture. Architec-tural Support for Programming Languages and Operat-

ing Systems (San Jose, CA, 3{7 October 1998). Pub-lished as SIGPLAN Notices, 33(11):92{103, November1998.

[9] Garth A. Gibson, David F. Nagle, William CourtrightII, Nat Lanza, Paul Mazaitis, Marc Unangst, and JimZelenka. NASD scalable storage systems. USENIX.99

(Monterey, CA., June 1999), 1999.

[10] Robert Hagmann. Reimplementing the Cedar �le sys-tem using logging and group commit. ACM Symposium

on Operating System Principles (Austin, Texas, 8{11November 1987). Published as Operating Systems Re-

view, 21(5):155{162, November 1987.

[11] David Hitz, James Lau, and Michael Malcolm. Filesystem design for an NFS �le server appliance. Win-

ter USENIX Technical Conference (San Francisco, CA).Published as Proceedings of USENIX, pages 235{246.USENIX Association, 19 January 1994.

[12] John H. Howard, Michael L. Kazar, Sherri G. Menees,David A. Nichols, M. Satyanarayanan, Robert N. Side-botham, and Michael J. West. Scale and performance ina distributed �le system. ACM Transactions on Com-

puter Systems, 6(1):51{81, February 1988.

[13] James E. Johnson andWilliamA. Laing. Overview of theSpiralog �le system. Digital Technical Journal, 8(2):5{14, 1996.

[14] Je�rey Katcher. PostMark: a new �le system bench-

mark. TR3022. Network Appliance, October 1997.

[15] Kimberly Keeton, David A. Patterson, and Joseph M.Hellerstein. A case for intelligent disks (IDISKs). SIG-MOD Record, 27(3):42{52, September 1998.

[16] Gene H. Kim and Eugene H. Spa�ord. The designand implementation of Tripwire: a �le system integrity

checker. Conference on Computer and Communications

Security (Fairfax, Virginia), pages 18{29, 2{4 November1994.

[17] Edward K. Lee and Chandramohan A. Thekkath. Petal:distributed virtual disks. Architectural Support for

Programming Languages and Operating Systems (Cam-bridge, MA). Published as SIGPLAN Notices, 31(9):84{92, 1{5 October 1996.

[18] Christopher Lumb, Jiri Schindler, Gregory R. Ganger,David F. Nagle, and Erik Riedel. Towards higher diskhead utilization: Extracting \free" bandwidth from busydisk drives. Symposium on Operating Systems Design

and Implementation (San Deigo, CA, 23{25 October2000). ACM, October 2000.

[19] Josh MacDonald. File system support for delta compres-

sion. Masters thesis. Department of Electrical Engineer-ing and Computer Science, University of California atBerkeley, 2000.

[20] Josh MacDonald, Paul N. Hil�nger, and Luigi Se-menzato. PRCS: The project revision control system.European Conference on Object-Oriented Programming

(Brussels, Belgium, July, 20{21). Published as Proceed-ings of ECOOP, pages 33{45. Springer-Verlag, 1998.

[21] Jeanna Neefe Matthews, Drew Roselli, Adam M.Costello, Randolph Y. Wang, and Thomas E. Ander-son. Improving the performance of log-structured �lesystems with adaptive methods. ACM Symposium on

Operating System Principles (Saint-Malo, France, 5{8October 1997). Published as Operating Systems Review,31(5):238{252. ACM, 1997.

[22] K. McCoy. VMS �le system internals. Digital Press,1990.

[23] Marshall K. McKusick, William N. Joy, Samuel J. Lef- er, and Robert S. Fabry. A fast �le system for UNIX.ACM Transactions on Computer Systems, 2(3):181{197,August 1984.

[24] Object based storage devices: a command set proposal.Technical report. October 1999. http://www.T10.org/.

[25] Rob Pike, Dave Presotto, Ken Thompson, and HowardTrickey. Plan 9 from Bell Labs. UKUUG Summer (Lon-don), pages 1{9. United Kingdom UNIX systems UserGroup, Buntingford, Herts, 9{13 July 1990.

[26] Erik Riedel and Garth Gibson. Active disks|remote

execution for network-attached storage. TR CMU-CS-97-198. December 1997.

[27] Mendel Rosenblum and John K. Ousterhout. The de-sign and implementation of a log-structured �le system.ACM Transactions on Computer Systems, 10(1):26{52,February 1992.

[28] Douglas J. Santry, Michael J. Feeley, and Norman C.Hutchinson. Elephant: the �le system that never forgets.Hot Topics in Operating Systems (Rio Rico, AZ, 29{30March 1992). IEEE Computer Society, 1999.

[29] Douglas S. Santry, Michael J. Feeley, Norman C.Hutchinson, Ross W. Carton, Jacob O�r, and Alistair C.Veitch. Deciding when to forget in the Elephant �le sys-tem. ACM Symposium on Operating System Principles

(Kiawah Island Resort, South Carolina). Published asOperating Systems Review, 33(5):110{123. ACM, 12{15December 1999.

[30] Margo Seltzer, Keith A. Smith, Hari Balakrishnan,Jacqueline Chang, Sara McMains, and Venkata Padman-abhan. File system logging versus clustering: a perfor-mance comparison. Annual USENIX Technical Confer-

ence (New Orleans), pages 249{264. Usenix Association,

16{20 January 1995.

[31] Margo I. Seltzer, Gregory R. Ganger, M. Kirk McKusick,Keith A. Smith, Craig A. N. Soules, and Christopher A.Stein. Journaling versus Soft Updates: AsynchronousMeta-data Protection in File Systems. USENIX Annual

Technical Conference (San Diego, CA), 18{23 June 2000.

[32] M. Spasojevic and M. Satyanarayanan. An empirical-study of a wide-area distributed �le system. ACM Trans-

actions on Computer Systems, 14(2):200{222, May1996.

[33] Adam Sweeney. Scalability in the XFS �le system.USENIX. (San Diego, California), pages 1{14, 22{26January 1996.

[34] Werner Vogels. File system usage in Windows NT 4.0.ACM Symposium on Operating System Principles (Ki-awah Island Resort, Charleston, South Carolina, 12{15December 1999). Published as Operating System Review,33(5):93{109. ACM, December 1999.

[35] Randolph Y. Wang, David A. Patterson, and Thomas E.Anderson. Virtual log based �le systems for a pro-grammable disk. Symposium on Operating Systems

Design and Implementation (New Orleans, LA, 22{25February 1999), pages 29{43. ACM, Winter 1998.

[36] Tatu Ylonen. SSH | Secure login connections over theinternet. USENIX Security Symposium (San Jose, CA).USENIX Association, 22{25 July 1996.