1

The Mars Rover Spirit FLASH AnomalyGlenn ReevesTracy Neilson

Jet Propulsion Laboratory (JPL)Pasadena, CA 91109

818-393-1051Glenn.E.Reeves@jpl.nasa.gov, Tracy.A.Neilson@jpl.nasa.gov

Abstract—The Mars Exploration Rover “Spirit” suffered adebilitating anomaly that prevented communication withEarth for several anxious days. With the eyes of the worldupon us, the anomaly team used each scrap of information,our knowledge of the system, and sheer determination toanalyze and fix the problem, then return the vehicle tonormal operation. This paper will discuss the SpiritFLASH anomaly, including the drama of the investigation,the root cause and the lessons learned from theexperience.1,2

TABLE OF CONTENTS

1. INTRODUCTION ................................................ 12. SEQUENCE OF EVENTS....................................... 1

SOL 18 ............................................................ 2SOL 19 ............................................................ 2SOL 20 ............................................................ 3SOL 21 ............................................................ 4THE NEXT 11 SOLS............................................ 5

3. THE ROOT CAUSE ............................................. 6DOS FILE SYSTEM USAGE.................................. 6DOS LIBRARY DESIGN FLAW............................. 7CONFIGURATION ERRORS.................................. 7REPETITIVE RESETS........................................... 8UNFORTUNATE SIDE EFFECTS............................. 8

4. HINDSIGHT IS 20-20........................................... 8SOL 18 REVISITED............................................. 8SOL 19 REVISITED............................................. 9SOL 20 REVISITED............................................. 9SOL 21 REVISITED............................................. 9

5. CONTRIBUTING FACTORS & LESSONS LEARNED ... 9COMPRESSED SCHEDULE.................................... 9INCOMPLETE DEVELOPMENT.............................. 9UNANTICIPATED BEHAVIOR ..............................10INADEQUATE TELEMETRY.................................10TEST PROGRAM DESIGN....................................11TEST DATA REVIEW .........................................11BUILD FOR THE UNEXPECTED ............................12OTHER LESSONS ..............................................12

6. OPERATIONAL & FSW CHANGES.......................127. ACKNOWLEDGEMENTS......................................138. CONCLUSION...................................................139. REFERENCES...................................................1410. BIOGRAPHY....................................................14

1 0-7803-8870-4/05/$20.00© 2005 IEEE

2 IEEEAC paper #1426, Version 5, Updated December 2, 2004

1. INTRODUCTION

NASA’s Mars Exploration Rovers (MER) project landedtwo rovers, Spirit and Opportunity, on Mars on January 4

and January 25, 2004,respectively, where theycontinue to operate inextended mission mode as ofOctober 2004. An anomalyoccurred on the MER-Avehicle, Spirit, on Sol3 18, orJanuary 21, 2004, which

precluded normal operation of the vehicle. The anomaly isnow well understood and changes have been made to boththe operational procedures and the Flight Software (FSW)to address the root cause. However, during the anomalyevent period, the team was faced with the dauntingchallenge of doing detective work at interplanetarydistances. Individual challenges included working withvery little data and a vehicle that appeared to behavedifferently than designed.

Spirit remained in an uncontrolled condition until Sol 21when the vehicle was commanded into a degraded, butusable, configuration. Subsequent diagnostic activitiesoccurred over the next 11 sols. Spirit was finally restoredto its normal operational state on Sol 32, and nominalscience activities resumed on Sol 33.

The root cause of the anomaly was a design error in thesoftware module that provides file system services. Inaddition, two significant configuration errors detrimentallyaffected the overall behavior of the system once the rootcause error occurred.

In addition to the technical factors related to the anomaly,there are also programmatic contributors and other lessonslearned that became apparent as the anomaly investigationprogressed.

2. SEQUENCE OF EVENTS

By January 21, (Sol 18), the Spirit rover team waseuphoric. The rover had safely landed, all the complexreleases and deployments went off without a hitch, and theoperations team had driven her off the Lander and onto theMartian surface. Every science instrument checked out fineand every camera worked great. The operations team had

3 A sol is a Martian day, 24 hours 37 minutes long

2

just settled into the fast turn-around of nominal missionoperations.

A nominal sol consisted of three UHF communicationwindows4 with the overhead orbiters (either Odyssey orMars Global Surveyor), early in morning and in theafternoon. Direct-to-earth (DTE) X-band communicationwindows around 09:00 local solar time (LST) includeddata downlinks and sequence uplinks to the rover for thedaily activities. The rovers are designed to “sleep” throughthe night and wakeup autonomously for thesecommunication windows. During sleep, most of the roverelectronics, including the receiver and processor, are turnedoff, but the mission clock and Battery Charger Board(BCB) remain on.

Sol 18

Sol 18 Plan—The plan for Sol 18 was for the rover towake up for an early morning UHF communicationwindow with the Odyssey orbiter to downlink yestersol’sdata. It would then return to sleep until 8:30 LST, when itwould wake up again to warm up the actuators on the HighGain Antenna (HGA) for a DTE X-band communicationwindow. During the DTE window, a sequence wouldexecute to characterize the actuator that controls a mirror inthe Miniature Thermal Emission Spectrometer (MTES).The purpose of this activity was to gather calibration datafor operating the mechanism at the cold morningtemperatures. The rest of the sol would be spent brushinga rock with the Rock Abrasion Tool (RAT). AfternoonDTE and UHF windows were scheduled to downlinkengineering and science data.

Sol 18 Observations—The early morning UHF windowtelemetry showed no problems and the 9:00 LST DTEwindow started right on time, initially indicating that thevehicle was healthy. The operations team began the uplinkof the sequences. Eleven minutes into the window,telemetry showed that uplink errors were detected onboard.The downlink became spotty. At approximately 9:16LST, the signal dropped out completely, 14 minutes earlierthan scheduled. At this point, without any moreinformation, poor weather at the ground station (Canberra,Australia) was blamed for the signal dropout.

The operations team expected a beep5 at either 10:00 LSTto indicate the new master sequence was onboard andrunning, or at 10:10 LST to tell us the old sequence wasstill running. The team did not detect the carrier signal for

4 The MER vehicles have pre-commanded, scheduled communication

windows. The operations team specifies the start time, configuration,and duration; the vehicle will autonomously power on (if necessary),prepare the telemetry, configure the telecommunication hardware, andinitiate transmission.5 A beep is a sequenced short duration communication window that

sends a DTE carrier-only signal through the Low Gain Antenna (LGA).No modulated telemetry data are sent. Beep sequences can be activatedeither by an immediate command from the operations team or bysequence, usually to tell the team that the onboard sequence is running.

either beep. Again, the team blamed the weather atCanberra for the lack of any signal.

At 11:20 LST, the team commanded a 30-minute highpriority HGA communication window, but no signal wasreceived. Again, the Deep Space Network station was theprime suspect, but the team began to suspect antennapointing problems or telecom hardware failures.

At 12:45 LST, a beep was commanded and this one wasreceived as predicted, both in start time and duration. Thistold us that the vehicle was commandable and that none ofthe onboard system level fault protection responses, thatchange the uplink rate, had executed.

The operations team then commanded the afternoon mastersequence to start and they received the beep embedded inthe sequence per predict. The team breathed a sigh ofrelief, but it was short-lived.

Just a half hour later, the team waited for the pre-scheduled14:00 LST HGA DTE communication window, but nocarrier signal was detected. Because the beeps (whichworked previously) use the LGA (which does not have tobe pointed by the rover), the speculation was that the rovercould have an HGA pointing problem. Since the team stillhadn’t received modulated data, the problem could still bethe hardware interface board to the radio or possibly a FSWproblem.

During the next two hours, the team tested the sequenceson the highest fidelity test beds and checked out a fewtheories before the next pre-scheduled UHF window. Thiswas to be the first UHF window since the trouble started,but unfortunately the Odyssey orbiter detected no signalfrom Spirit. The testing revealed no suspects either.

Since UHF communication uses a completely differentpath than the X-band DTE communication, the idea that aHGA pointing problem was to blame was thrown outbecause there is no pointing of the UHF antenna. Panicstarted to set in for the operations team.

Sol 18 Conclusions—By the end of this sol, the teamknew the rover had not sent modulated data, but it didcommunicate via carrier signal by command. Potentialcandidates for the observed behavior at this time were alow power situation, an overheat condition, a FSWcommunication behavior problem, a problem with theinterface card to the radios or FSW-initiated resets. At thispoint, the Project formed an anomaly team and our nowgrim project manager, Pete Theisinger, reported to thepress that "a very serious anomaly" had occurred.

Sol 19

Sol 19 Plan—Sol 19 plans were replaced by anomalyresponse activities. The objectives for Sol 19 were toestablish commandability and to establish modulation inan X-band communication window.

3

Sol 19 Observations—The next pre-scheduled window wasan early morning Mars Global Surveyor orbiter UHFcommunication window at 1:45 LST. This orbiterdetected the signal from Spirit and the window started atthe correct time, but no valid data were received. Twominutes and 20 seconds of repeating Pseudo Noise (PN)code were the only data received. This told us that therover woke up for the communication window and turnedon the UHF radio, but no valid data was passed to theradio for transmission.

No signal was detected during the pre-scheduled 4:30 LSTOdyssey window, or the pre-scheduled 9:00 LST HGADTE window.

At this point, the team started commanding beeps again, aswell as listening for the sequenced beeps6. If the onboardfault protection response to a low power event had run, itwould have added a communication window at 11:00 LST,but no signal was observed at this time. The anomalyteam finally received a beep, but it was one that wascommanded at the 7.8125 bps uplink rate, which is thedata rate the onboard fault protection configures when amajor fault has been detected. Note that a single processorreset will not result in this fault uplink rate. This meantthat some new event had occurred on the vehicle thatcaused the behavior to change. The anomaly team wasmystified.

The pre-scheduled afternoon UHF window failed toproduce a signal, and a commanded DTE window on theLGA also failed. However, the commands were sent veryclose to Earthset and may not have reached the rover.

Sol 19 Conclusions—The anomaly team had establishedcommandability, but had not yet received modulatedtelemetry.

The team knew a system-level fault that changed the uplinkrate had occurred sometime between Sol 18 13:30 LST andSol 19 14:40 LST. The only autonomous responses thatwould change the uplink rate were responses to a lowpower event, a mission clock failure, an uplink loss faultor an ‘X-band fault’7. A mission clock failure wasunlikely since the vehicle had tried to communicate at thecorrect time for the first morning UHF window, and anuplink loss response was unlikely because the timer wasn’texpected to expire until later the next sol. There were noindications of a power problem from the last receivedtelemetry on the morning of Sol 18, and an ‘X-band fault’should not affect the UHF communication.

6 The main (master) sequence usually spanned two sols with the

expectation that the second sol would have its own master sequence thatwould terminate the prior sol’s master sequence. The operations teamembedded beeps in each master sequence that would indicate the newmaster sequence had taken over or, if the beep was received at a latertime, that the old master sequence was still in control.7 An X-band fault can result when the FSW detects transponder failures,

failure of a coax switch or the waveguide transfer switch, failure to turnon the amplifier, attitude estimation failures that drive the HGA beyondits allowable range, or HGA positioning problems during acommunication window.

FSW initiated resets seemed to be the only likely behaviorfor the lack of communication using the two separatecommunication paths, but the response to a reset does notautomatically change the uplink data rate.

This was a puzzling sol for the anomaly team. PeteTheisinger briefed the press that “the spacecraft thinks it'sin the fault side of the tree somehow, for some reason.That would mean that we've got positive power, someelements of the software are working, … the X-bandsystem is working, … the transponder; all that stuff isworking, so that would be more information -- goodnews.”

Sol 20

Sol 20 Observations—None of the overnight UHF orbitersreceived any signal from the rover, and two attempts atcommanded beeps during the 9:00 hour failed. However,the team was relieved to receive an autonomous 10 bpssignal at the pre-scheduled 9:55 LST communicationwindow. The carrier signal lasted only 10 minutes, but theteam did receive a single garbled telemetry packet. TheFSW team attempted to decode it but quickly determinedthe data were garbage. However, the time of this attemptedcommunication, and the data rate, convinced us that anonboard ‘X-band fault’ response had run.

The team commanded an LGA DTE communicationwindow and was overjoyed to receive the first set oftelemetry, which unfortunately did not include any healthinformation. Our hopes were quickly dashed when it wasdiscovered that the next few packets contained exactly thesame telemetry. The same information was sent over andover again. The signal then terminated suddenly, earlierthan the scheduled cut-off time.

At 11:50 LST, the anomaly team commanded anotherLGA DTE communication window. This time, the signaland data for the full window duration were received, andthe telemetry indicated many reset/reboot cycles hadoccurred. The multitude of resets explained much of theobserved behavior.

When the FSW detects a severe error, it reacts by forcing areset of the flight computer and a re-initialization of theFSW. If the FSW determines that a reset is necessaryduring the FSW initialization process, the FSW delays theactual reset until a minimum time period has passed. Thisprocess is referred to as a “delayed reset”. The intent is toallow sufficient time for ground intervention.

When the delayed reset time expires, the reset is initiated.The system keeps track of the number of repetitive resetsand modifies the delayed reset time interval. The bootlogic part of the software also keeps track of the number ofinitialization attempts and is designed to load and startalternating copies of the FSW with each reset. By design,the time period between resets is set to a fifteen minutedelay for the first severe error, fifteen minutes for thesecond, and then one hour for the third. This pattern then

4

repeats (see Figure 1). Commands can be processed andcommunication windows can occur during the delayed resetperiod.

15 60 1515 15

Repetitive error occurs during early initialization each time

Actual reset is delayed to allow ground intervention

Resets

Figure 1: Delayed reset behavior

The telemetry confirmed that the ‘X-band fault’ responsehad executed, and it had changed the uplink data rate to7.8125 bps. The team deduced this was caused by anattempt to point the high gain antenna when knowledge ofthe antenna position was unknown. The FSW does notremember the antenna position if an unexpected resetoccurs (and the ground must command a calibration actionto reacquire the antenna position knowledge). When thenext pre-scheduled HGA communication window initiated,the FSW could not point the antenna. The FSW declaredan 'X-band fault', which aborted this window and changedthe uplink rate to 7.8125 bps. The mystery of why thebehavior changed between the afternoon of Sol 18 and theafternoon of Sol 19 was solved.

But Spirit threw us another puzzle. The power and thermaltelemetry indicated higher temperatures and a lower batterystate-of-charge than predicted (given the nominal loads andnominal solar array current). This indicated that the roverelectronics had been on much longer than expected and thevehicle had possibly not shut down overnight. The vehicleshould have autonomously shut down the processor andelectronics each afternoon around 15:30 LST to save powerand then autonomously woken up at approximately 9:30LST each morning via solar wakeup (when the solar arraycurrent reaches 2.0 amps for the first time each sol).

The telemetry also indicated that the MTEScharacterization sequence on Sol 18 never completed. Thisinformation helped pinpoint the first reset event to the firstSol 18 HGA communication window.

At 14:00 LST on Sol 20, the rover began another pre-scheduled DTE window at the fault configuration (10 bpsdownlink on the LGA), as predicted.

The priority at this point was to shut the vehicle downearly in the afternoon, in an effort to charge the batteries.The team commanded a shutdown, which terminated thecurrent communication window, and the loss of signaloccurred at the predicted time. Fifty minutes later, theteam commanded a beep at 7.8125 bps to alert us if theshutdown command did not work, and much to ourdisappointment, the beep was received! This information

confirmed our fear that the onboard shutdown process wasnot working.

The anomaly team scrambled to delete the next few pre-scheduled UHF windows to conserve battery charge, butthese commands did not reach Spirit successfully. TheEarth had set below the Martian horizon.

Quite surprisingly, the Odyssey communication window at16:23 LST produced 73 Mbits of telemetry for the fullduration. This was the first UHF window that lasted thepredicted duration since the start of this anomaly. Thetelemetry provided additional indications that multipleresets were occurring and that some of the commands sentduring the afternoon were not received correctly (due to adelayed reset occurring in the middle of the transmission).However, only real-time data generated during the windowwere received. The majority of the telemetry was fill dataand no recorded data were ever received.

The operations team requested the orbiters to delete theirovernight communication windows to save rover power.The orbiters turned off their beacons so Spirit’s UHF radiowould not attempt to transmit.

Sol 20 Conclusions—By the end of Sol 20, the anomalyteam knew FSW was in a continuous delayed reset loop.The first reset occurred during the Sol 18 morning DTEwindow coincident with the MTES actuator checkout.Both commanded and autonomous shutdowns were failingand the vehicle probably had not shut down since Sol 19.

Memory corruption was unlikely because different FSWcopies were being loaded and used. The lack of recordeddata (which are stored in FLASH memory) indicated thatthe FSW was not reading files from the FLASH filesystem, so the prime suspects became the FLASHmemory, the FLASH file system and the FSW that readsthe data files and creates telemetry for downlink.

The MER project manager upgraded Spirit’s condition to“critical” at the press conference.

Sol 21

Sol 21 Plan—The plan, at Earthrise, was to send ahardware command to place Spirit in a “crippled mode”.This command tells the FSW “do not use the FLASH filesystem” 8.

Sol 21 Observations—Between 8:30 LST to 11:08 LST,the team repetitively sent the crippled command. Nosignal was detected at 9:35 LST for the pre-scheduled DTEcommunication window. This should have been an FSW-adjusted LGA DTE communication window at 10 bps.Loss of this window was not entirely unexpected given

8 The crippled mode command is one of a few commands that are

processed entirely by hardware. The command only sets individual bitsin a software accessible register that the FSW interrogates duringinitialization to ascertain if any off-nominal initialization actions arerequired. If set, the FSW avoids the use of the FLASH file system.

5

that the vehicle could not shut down overnight and thebatteries may have drained. This suspicion was confirmedat 11:00 LST when an LGA communication window thatwas instigated by the onboard ‘low power fault’ responsewas received.

Eight minutes later, the team commanded a reset and anLGA DTE communication window with a higher data rate.The telemetry indicated that the BCB hardware faultprotection9 had tripped between 4:28 and 6:15 LST. Therover woke up 8:57 LST via a solar wakeup (neither thereceiver nor the processor were on until this time).

When the low power condition was detected by the FSW,it executed the ‘low power fault’ response. The faultresponse changed the communications configuration andattempted to shut down the vehicle. Since this shutdownprocess failed too, the hardware fault protection embeddedin the BCB took the batteries off the bus and theelectronics were finally turned off. Note that the missionclock is powered directly by the batteries, so the systemdid not lose time. Upon solar wakeup, the onboard FSWfault response eliminated the pre-scheduled communicationwindows and created “fault windows” at 11:00 LST.

At 12:20 LST, the anomaly team commanded the vehicleto shut down for 24 hours. Thirty minutes later, the teamcommanded a beep and received no response. Anotherbeep was commanded but not received, which confirmedthat the shut down was successful!

Sol 21 Conclusions—The rover had browned out overnightand was now in low power mode. The team could preventcontinuous resets by issuing the crippled command and wecould shut the vehicle down.

The vehicle would still boot up into the error state in themorning because the crippled command only sets volatileregisters, which revert to the nominal state after theelectronics are powered off. The Spirit rover requiredmanual intervention each sol to restart into the “crippled”mode until access to the FLASH file system could berestored If the vehicle had to continue to operate in thismode, the mission return would be severely impacted dueto an inability to use the FLASH file system to storescience data products.

However, at this point, the anomaly team had regainedcontrol of Spirit, which was extremely important becausethe twin rover “Opportunity” was due to land on Mars inthe next 12 hours! Our Project Manager reported to themedia that he had upgraded Spirit’s condition from"critical" to just "serious".

The Next 11 Sols

For the next 11 sols, the anomaly team attempted todiagnose the precise cause of the problem, recover science

9 The BCB is responsible for protecting the batteries from completely

draining and it performed perfectly on this sol.

and engineering data still in the FLASH memory, andreturn the vehicle to a science-taking machine. During thistime, the press started to refer to us as us “space-agesurgeons”. The timeline of the anomaly investigation andrecovery is shown in Figure 2.

Commanding was limited by the fact that the vehiclewould not wakeup until 9:00 LST each sol (on solarwakeup) and it was only commandable until 15:30 LSTwhen the Earth set. The team could use the afternoonOdyssey orbiter UHF window, but morning UHF windowswould fail because the vehicle would reboot after 15minutes. Since all transmissions drain energy from thebatteries, time was limited for the rover to send data.

An additional challenge was accessing any of thediagnostic data or the FLASH file system when the systemwas in crippled mode. The team had to invent newtechniques to capture the error cause and make thesediagnostic data available in crippled mode.

So each sol, a number of activities were performed inparallel, including collecting telemetry, planning the nextsol’s commands, exploring theories in the test bed,validating commands in the test bed, communicating withthe management and the media and trying to get somesleep.

On Sol 25, the team calibrated the encoders and regainedthe use of the HGA, which increased the uplink anddownlink capability. The team also took two images ofthe science instruments still on the rock Adirondack. Allof the science instruments and cameras were checked out inthe subsequent sols.

On Sol 27, the team deleted a large number of obsoletedata products and their subdirectories left over from thelaunch FSW load (called the R7 data products), booted upin nominal mode and verified that the FSW was no longerautonomously rebooting. However, because of themultiple failed file system accesses and resets, some datain the FLASH file system was corrupted. In order toremove all possible corrupted data, the team decided tofully erase the FLASH memory and rebuild the filesystem.

On Sol 31, the anomaly team shut down the vehicle earlyto fully charge the batteries and to cool down theelectronics for the next sol, because Spirit needed to beawake for nine hours. Early on Sol 32 at 6:30 LST, theteam manually booted the rover into crippled mode andcopied the FSW image into RAM (for insurance). Theteam then ran a sequence that erased small chunks ofFLASH at a time, and checked both FSW images (toensure the commands didn’t inadvertently corrupt or deletean image) after each chunk. This conditional sequencewould have stopped if any command failed or if any FSWimage corruption was detected. The team chose thisconservative strategy because there was a small fear that theroot cause may still be in the FLASH devices or interfacehardware.

6

OpportunityLanding

OpportunityLanding

1717 1818 1919 2020 2121 2222 2323 2424 2525 2626 2727 2828 2929 3030 3131 3232 3333

first

out

-of-m

emor

y ev

ent

“X-b

and

fault

”su

cces

sful

bee

p at

fault

rate

MGS

pass

- PN

cod

e on

lybr

owno

ut/e

nter

low

powe

r mod

e

faile

d co

mm

ande

d sh

utdo

wn

first

telem

etry

of c

urre

nt st

ate

com

man

ded

cripp

led m

ode

succ

essf

ul co

mm

ande

d sh

utdo

wn

reset occurred eachcomm window

delet

ed c

ruise

dat

a pr

oduc

ts

& bo

oted

into

nor

mal

mod

e

task traceattempts

file system debugerase FLASHproceduredesign

shut

down

ear

ly to

cha

rge

batt

eries

eras

ed &

form

atte

d

FLAS

H &

boot

nor

mal

first

telem

etry

of s

uspe

nded

task

scan

FLA

SH fo

r eve

nt te

lemet

ryob

taine

d So

l 18

reco

rded

telem

etry

obta

ined

histo

ry o

f unu

sed

mem

ory

Reco

vere

d fro

m lo

w po

wer m

ode

perfo

rmed

IDD/

RAT/

APXS

act

ivitie

s

investigated potentialFSW & hardware suspects

star

ted

FLAS

H file

dum

ps

crippled modecrippled modenormalops

normalops

using FLASH file systemcollecting science data

using FLASH file systemcollecting science data

first

use

of H

GA

star

ted

task

trac

e at

tem

pts

Spirit Flash Anomaly Recovery Timeline

repeating resetsno shutdowns

normalops

normalops

Sol

Figure 2: Anomaly events and recovery timeline

After the team verified the sequence completion andchecksums, they commanded a FLASH file system formatand booted back into normal mode. The FLASH memorywas formatted as the system booted back up. The formatwas successful and the FLASH file system was ready to go.

At this point, the anomaly team declared victory and madeplans for a work-free weekend!

3. THE ROOT CAUSE

DOS File System Usage

The MER FSW uses a commercial operating system andcapitalizes upon many of the bundled functions and librariesthat are included. Among these functions is a bundled filesystem library, which supports a DOS file system structure(referred to as the “DOS Library”). The file system modelprovides a very convenient and intuitive interface for manyonboard functions. It is also straightforward to support in atest and simulation environment when real hardware is notavailable.

There are multiple file systems onboard the vehicle withpartitioning to accommodate both differences in use anddifferences in the underlying storage media. There are notraditional disk drives on the vehicle. Each file system hasRAM, EEPROM, or FLASH as the underlying storagemedia. Table 1 describes the onboard file systems.

Table 1: Onboard file systems

Name UnderlyingMemory Type

Size Purpose

Ram FileSystem

RAM 4Mbytes

Temporarystorage ofuplinked files

TemporaryFile System

RAM 2Mbytes

Temporary dataproduct store

FLASH FileSystem

FLASH 224Mbytes

Science andengineering dataproducts

PrimarySequence

EEPROM 700Kbytes

Sequencestorage

SecondarySequence

EEPROM 700Kbytes

RedundantSequencestorage

7

PrimaryDownlink

EEPROM 50Kbytes

Downlink TableStorage

SecondaryDownlink

EEPROM 50Kbytes

RedundantDownlink TableStorage

The DOS Library module uses the operating system I/Oservices to interact with the physical memory devices via adevice driver layer. Each memory type has its own devicedriver. In the case of the FLASH file system, an additionalservice layer provides management for erasure, writeleveling, reading and writing.

The DOS file system use on MER includes the use of manysubdirectories as an organizing mechanism. Every file isstored in a subdirectory (nominally, no regular files arestored in the “root” directory). The primary telemetrymechanism is through the use of “data products” (see Ref.[2]). Data products are effectively files that contain theengineering or science data of interest plus a metadata filecontaining ancillary information (time of collection, priorityof data, etc.).

DOS Library Design Flaw

A design flaw exists in the DOS Library. The error is inhow the DOS Library logic represents deleted filesinternally and how this impacts the size of the interlinkeddata structures held in the DOS Library private memoryarea.

The DOS Library module creates an interlinked collection ofdata structures10 in the flight computer RAM to representthe file system structure (directories, sub-directories, numberof files in directories, etc.). The initial memory space forthis data structure collection is allocated out of the freeRAM space at initialization time. The DOS Library logicrelies on the services of another bundled OS service (“MemLibrary”) to manage this private space.

When a file or a sub-directory is created, this internalrepresentation of the file system structure is updated (as isthe file system itself in FLASH). As the file systemstructure expands, one or more additional memoryallocations, from the private area, will be required toprovide the space for new elements of the data structurecollection.

A DOS subdirectory is nothing more than a file itself exceptthat it is specially structured as an array11 that contains theinformation about the files and the subdirectories. Theinformation contained within an array element includes thefile name, attributes, date/time, file size, and a pointer tothe starting cluster of the file (a cluster is the smallest unitof space that can be allocated to a file). When asubdirectory is first created, all of the array elements are

10

This was done to optimize performance.11

This is the simplistic description. The detailed description can be foundin Ref [1].

initialized to the unused state (all zero values). As files arecreated in the subdirectory, the array is searched in a linearfashion until the first unused (or the first deleted) entry isfound; the information for the new file is then placed in thatentry. Never-used entries and entries for deleted files aretreated differently. When a file is deleted, the only actionperformed is to change the value of the first letter in the filename to a special value (the infamous hex byte code valueE5h). This is a special tag that tells the system "this filehas been deleted".

The DOS Library logic manages both the actual file system(on the FLASH) and the private, RAM resident, internaldata structure collection in parallel. The internal datastructure collection is created when the file system is“mounted” during initialization and then updated as fileinteractions occur. This data structure collection is notretained when the vehicle is turned off but it is alwaysrecreated when the processor is turned back on. When thefile system mount action occurs during initialization, theentirety of the file system structure is traversed in order tobuild the initial representation. Each subdirectory is readfrom the “disk” and traversed in order to create its internaldata structure collection. Directory list entries that havenever been used do not require any representation in the datastructure collection (since they never represented a file) andno memory space is required for them. However, deletedfile entries are represented. The premise being that this isnecessary to maintain the symmetry between the internalrepresentation and the actual file system structure.

The unfortunate consequence of this feature is that when afile is deleted, the space in the file system (on FLASH) isfreed but the internal data structure space is still required. Infact, the space is never released.

Effectively, the “high water mark” or the maximum numberof files that ever existed in the subdirectory is the directcontributor to the amount of memory space that will beneeded to represent the file system structure.

The ramification of this is twofold: simply deleting filesdoes not reduce the amount of memory needed to representthe file system structure, and the total size of the file systemstructure (from the perspective of the total memory spacerequired to represent it) is not expressed by the number ofcurrent files but by the sum of the maximum number offiles that had ever existed in each subdirectory.

Configuration Errors

Two configuration errors conspired to place the system intoa condition where it would reset repeatedly and alsoprevented the vehicle from autonomously shutting itself offto save power. A configuration error in the DOS Librarymodule allowed the size of the private memory area toexpand by allocating additional space from the free systemspace12. A configuration error in the Mem Library module

12

The free system memory is > 4 Mbytes normally

8

silently resulted in a suspended task when the request foradditional memory could not be satisfied.

DOS Library Configuration Error

As described earlier, the DOS Library module creates aninterlinked collection of data structures in the flightcomputer memory to represent the file system structure. Theinitial amount of memory that was allocated to this privatearea is one of the DOS Library configuration parameters. Inthe MER configuration, this initial size is 256K bytes. When new elements need to be added to the internal datastructure collection, a block of free space from this area isused for the new element(s). Another DOS Libraryconfiguration parameter allows for the expansion of theprivate memory space by requesting additional space fromthe free system memory area. Unfortunately, thisconfiguration parameter was not set correctly; expansion waspermitted. In this erroneous configuration, when the initialspace was exhausted, the DOS Library logic made allocationrequests in increments of 256K bytes from the free systemmemory. This new space is added to the DOS Libraryprivate area. Additionally, a second parameter can be set tolimit the growth to a maximum amount of space. Thissecond parameter was also in error; no limit was specified.These configuration errors allowed all of the free systemmemory space to be consumed.

Mem Library Configuration Error

The second configuration error existed in the Mem Librarymodule. The Mem Library module logic provides a simplemalloc/free service13 and operates using memory spacepreviously provided to it. When an application program (orlibrary) requests a buffer of a given size, the Mem Libraryprovides a pointer to a buffer of that size. The availablespace shrinks accordingly. The actual process involves themanagement of a linked list of data structures that representboth the allocated (in use) space and the unallocated (free)space. In the MER architecture, the free system memory ismanaged using this service.

By convention, applications are allowed to request memoryallocations only during initialization. This space is neverreturned to the free system memory space. No dynamicallocation and release of system space is permitted (byconvention).

The DOS Library consumed the free system memory spaceuntil no further allocations could be satisfied. The MemLibrary module can be configured to react in various wayswhen there is insufficient free space to satisfy an allocationrequest. Ideally, the system should have reacted by failingthe file system activity in progress (such as a file open, filecreation, or file write). Unfortunately, the configurationparameter was not set correctly, and the task performing thefile system activity was silently suspended.

The suspension of a task is a severe error and is neversupposed to occur. The MER architecture includes a 13

Actually it provides a far more general memory management service.

“software health” function that continuously checks forsuspended tasks. When the health monitor functionidentifies a suspended task, it forces a reset of the avionicselectronics, including the flight computer, and the re-initialization of the FSW.

Repetitive Resets

As described earlier, each time the software initializes, theFLASH file system is mounted. This caused the re-creationof the RAM resident data structures, which, in turn,consumed all of the available memory. Subsequently, whenthe FSW attempted to create a new file in the FLASH filesystem, insufficient space existed in the private area (theprivate area has been expanded many times already). TheDOS Library attempted to allocate additional space from thefree system memory, but the request could not be satisfied.The task that attempted to add the file was suspended. TheFSW “health check” mechanism detected the suspended taskand signaled a severe error, resulting in a reset (after thedelay period).

The cycle then repeated.

Unfortunate Side Effects

An unfortunate side effect of the task suspension was thatthe file systems became inaccessible to the other FSWtasks. The DOS Library logic uses semaphores to protectcritical regions. Unfortunately, the memory allocationfailure occurred within one of these critical regions. Withthe task suspended, the access control semaphore was neverreleased and no other tasks could gain access. This sideeffect prevented recorded telemetry from being produced (thetask which reads files from the file system was blockedattempting access), prevented the shut down (the task whichcontrols the shutdown actions could not idle the filesystem), and was the cause of the repeating telemetry (thetask which pushes new telemetry data into the hardware wasalso blocked).

4. HINDSIGHT IS 20-20

Now that the root cause of the problem is understood andthe overall behavior of the system can be traced, it ispossible to correlate the observed behavior and data with theactual events that occurred on the vehicle.

Sol 18 Revisited

Three FSW resets occurred on Sol 18. The first was due tothe creation of a large number of motion history dataproducts that resulted in new files being created in theFLASH file system. This increase in the number of filesrequired additional space in the DOS Library privatememory area that led to additional memory allocations fromthe system memory pool. When one of these allocationsfailed, the Mirror Ram to FLASH (MRF) task (whichcreates data product files in the FLASH file system), was

9

suspended. The health monitoring function detected thesuspension and forced the first reset.

The next reset did not occur for another two hours. A pre-scheduled communication window contained a HGAencoder calibration to ensure the HGA is pointed correctly.Unfortunately this calibration created more motion historyreport files, leading to another out-of-memory event andreset.

The afternoon HGA window failed because the encoderswere left un-initialized following the reset. The on-boardfault protection declared an ‘X-band fault’, aborted thewindow and set the communication rates to the faultconfiguration.

During the afternoon UHF window, another out-of-memoryevent occurred when the FSW autonomously created a datasummary report file14. Earlier commanded beeps on Sol 18succeeded because no data summary report file is generatedfor beeps.

Sol 19 Revisited

When the rover woke up for the early morning UHFwindow on Sol 19, another out-of-memory event occurred.The reset was delayed 15 minutes, so the FSW continuedwith the communication window, including turning on theUHF radio. The reset took place before any data waspushed to the radio and the FSW turned the radio off in thenext initialization, so the orbiter received PN code for onlytwo minutes and 20 seconds.

This is the probable start of the continuous delayed resets(given the number of resets recorded in telemetry). Many ofthe future communication windows were interrupted becausethe reset would take place during the setup or transmissionportion of the window.

Sol 20 Revisited

The first telemetry on Sol 20 was received at the faultconfiguration rate and garbled. A FSW reset had occurred10 minutes into the window. Due to the size andmanagement of the hardware buffer space, no valid data wereradiated before the radio was shut off after the reset. On thenext commanded LGA session, 11 complete transfer frameswere transmitted, but they were all identical. The reason forthis repeating nature was the FSW task that packages frameswas blocked from generating new frames. The radio'stransmitter continued to pump out the last frame that was inits hardware buffer. A delayed reset terminated the sessionafter approximately 20 minutes.

The next commanded window occurred within the one hourdelayed reset period, so it was not interrupted by a reset asthe other windows had been. Only real-time data were 14

This engineering telemetry data product tells operators which dataproducts exist. The report is autonomously produced for eachcommunication window.

received because recorded data is stored in FLASH memoryand the FSW could not access it.

The vehicle would not shut down on Sols 19 and 20because the task that suspended during initialization heldthe semaphore that was needed by the shutdown logic togracefully idle the FLASH system. Since the shutdownlogic could not access the semaphore and there was notimeout or other logic to force the continuation of theshutdown actions, the shutdown process halted and theprocessor was never turned off.

Sol 21 Revisited

The crippled mode mechanism allowed us to regain controlof the vehicle. This debug feature forces the FSW to createa file system in RAM with the same logical device name asthe FLASH file system. The file system created in RAM isempty; there are no residual files and subdirectories. Asresult, the memory space necessary is considerably less, sothe FSW was able to complete the initialization withouterror.

The system is not as capable in this configuration since theRAM based file system is 10 times smaller and volatile.However, all functionality is restored. All of theapplication and system software use the logical file systemname so the change in configuration is completelytransparent to the rest of the software.

5. CONTRIBUTING FACTORS & LESSONS LEARNED

Although the anomaly has a precise technical cause, therewere other contributing factors in which the cause and effectrelationship is much more subjective. This sectiondescribes these other contributing factors and lessons thathave been identified during and after the anomalyinvestigation.

Compressed Schedule

Many of the contributing factors are related to thecompressed development schedule (three years from conceptto launch). The MER project was always challenged by thetight schedule and the FSW development was tailored toaddress this reality. During the development process, therewas a continuous reprioritization of activities and focus. One impact of this dynamic process was that only thehighest priority issues and problems could be addressed.Several of the contributing factors can be described assimply lower priority activities whose completion orcompleteness could not be realized.

Incomplete Development

The behavior of the DOS Library module, at least in regardto expanding the private memory area, was known by asubset of the FSW development team. This was identifiedas a potential problem area and it was recognized that the

10

behavior of the system would need to be characterized inorder to establish the flight configuration. The DOSLibrary configuration parameters were set with theexpectation that over the course of the test activities, apattern of use could be identified that would permit a finalconfiguration to be identified. The intent was to run thesystem using the initial configuration, then gather data to:

• Investigate how often requests for additional memoryoccurred (some were expected). A significant differencefrom what was expected should have triggeredadditional investigation.

• Identify the maximum amount of space ever allocated.

• Identify any combination of activities or operations thatresulted in the free system space being entirelyconsumed. This specific situation never occurred.

This investigation was never completed so the configurationremained unchanged. No errors were identified duringtesting and as a result, there was no immediate need tofinalize the configuration.

Lesson—Many lower priority development activities couldnot be completed in the development time frame. This wasan accepted risk. There should have been a formalmanagement of incomplete items that required resolution.This would have had the beneficial effect of creating achecklist of items requiring closure and thus would haveinsured that this specific investigation was completed.

Unanticipated Behavior

There was a belief among the FSW development team thatthe system would not exhibit the behavior that is the rootcause of the anomaly (even though it was known to be apossibility). This posture affected the priority of manyactivities including the investigation of related problemreports, the analysis of test results, and the closeout ofrelated investigations. This unfounded belief was based ona number of factors including:

• A recognition that the DOS Library implementationwill request and release temporary buffer space from thefree system memory during its normal operation. Thisis unrelated to the private memory area use. This is notan ideal characteristic, but the implementation iscorrect; no memory leaks occur. It had the unfortunateside effect of desensitizing the developers and testers toother similar, but incorrect behavior, as was seen in theanomaly.

• A belief that the only limitation regarding the use ofthe FLASH file system, was the total file systemcapacity. At the time there was no indication orknowledge regarding the mismanagement of deletedfiles.

• A false impression that the number of files in the filesystem would be limited to the number of data product

files that would be allowed to exist. The data productmanagement software limits the number of dataproducts to 8192 unique items.

Lesson—Understand the behavior of any Commercial Off-The-Shelf (COTS) software that is utilized. There shouldhave been an effort to review the implementation of theDOS Library module (and others) with an emphasis onreview of the basic logic and function. The vendor couldhave participated in these reviews and could have beentasked to brief the FSW development team on thesefunctions.

Lesson—Enforce the design guidelines. There was a designguideline that prohibited allocation of space from the freesystem memory space after initialization was complete. Thisguideline was enforced for the JPL-developed software andit should have been enforced for the COTS software as well.

Lesson—Verify assumptions regarding the expectedbehavior. No developer was explicitly assigned toinvestigate or review the DOS Library. This allowed themodule to be used without detailed review or scrutiny.There needed to be a responsible individual to address anydesign or test issues.

Inadequate Telemetry

The telemetry important to detecting the underlyingproblem was not a part of the normal telemetry process. TheFSW does produce the required information, but it istelemetered in a format that does not feed into the normaldata analysis tools. In fact, only a few FSW team memberswere even aware that the information existed.

Figure 3 shows Spirit’s in-flight telemetry of the freememory space from launch through recovery. Note thesegments are not connected on the figure because that wouldbe misleading regarding the rate of decrease in the systemmemory space. Insufficient data exists to extrapolate wheneach request for new space occurred. This data was notexamined until the anomaly occurred.

Lesson—The FSW should have included flight telemetryfor resources (such as the free system memory space) so thatthe actual and expected behavior of the system can becompared.

11

Free System Memory Space

0

500000

1000000

1500000

2000000

2500000

3000000

3500000

4000000

4500000

5000000

105000000 110000000 115000000 120000000 125000000 130000000

Time (SCLK)

Byte

s

Launch

Surface FSW loadinstallation

Recovery

Landing

Anomaly

Figure 3: Flight telemetry of Spirit's free memory space

Test Program Design

The MER system and FSW test activities were extensiveand comprehensive. The test program included unit,design, functional, system level, verification, andvalidation activities. These test activities were performedusing test beds with various degrees of fidelity includingmultiple test beds with hardware in-the-loopconfigurations. The test activities were performed bymultiple organizations within the MER project structure.

The project performed several realistic test exercisesintended to demonstrate the performance of the system andpersonnel in a realistic flight-like manner. There wereseveral Project Operational Readiness Tests (PORTs) wherethe system was exercised in a manner believed to berepresentative of the actual flight use. The anomaly wasnever seen during these (or any other) test activities. Thereare several factors that contributed to this result:

• The operational tests did not exercise the system fullyin a flight-like way. This was because both the surfaceoperation processes and the FSW were immature whenthe tests were done and this limited the activities thatwere performed during the test. As a result, theoperational tests did not produce the diversity of dataproducts or the volume of data products that werecreated during the surface mission.

• The number of files on the flight vehicle on Sol 1 wasgreater than the number of files on test bed during theoperational tests. Although the operations teamattempted to reproduce the flight-like conditions bywalking through launch, cruise, entry, descent, andlanding, they did not reproduce every turn, maneuverand communication window performed in flight.

Observation—The operations team exercised the system inwhat was considered a “flight-like” manner during theoperational tests. Once the rover reached the surface ofMars, the experience and training allowed the operationsteam to develop activities that exceeded the envelope of thetest activities. It is not apparent that anything other than alonger duration operational test would have exposed theanomaly, but a longer operational test (over 11 sols) wasimpractical within the overall MER schedule.

Test Data Review

A post-anomaly review of one operational test showed thatthe memory leak was evident in the data. Figure 4 showsa portion of the telemetry of free system memory space andit indicates a memory consumption trend. The data set issparse because the telemetry for this data was set to a lowpriority and was not consistently telemetered.Unfortunately, the team did not analyze this data duringthe test execution or during the post-test data review.

Observation—The analysis of this test data by the FSWteam was a lower priority activity when compared to otheractivities occurring in the same timeframe.

Lesson—A suite of tests and automated analysis toolsshould have been created early in the development process.

The FSW should have included support for reportingresource usage. The system (FSW, test scripts, groundsupport equipment) should have included (additional)support for collecting, archiving, and analyzing thisinformation.

12

PORT 7/9 Free System Memory Space

0

500000

1000000

1500000

2000000

2500000

3000000

3500000

4000000

4500000

128600000 128650000 128700000 128750000 128800000 128850000 128900000 128950000 129000000 129050000

Time (SCLK)

Byte

s

Figure 4: Project operational test 7/9 free system memory space

Build for the Unexpected

The idea for crippled mode originated on the MarsPathfinder project. Without this capability, the MERmission may well have been lost.

L e s s o n —The system design should include themechanisms to address both problems envisioned as wellas the unforeseen and “unknown-unknowns”. Contingencycommands and similar mechanisms need to be includedwhere they can be put to use to resolve both failures anddesign errors such as the Spirit anomaly scenario. Ofcourse, the contingency commands are insufficient unlessthe system is robust enough to take the necessary actionsto both maintain a safe configuration and to let theoperations team know what is happening onboard.

Observation—The MER design placed the primaryresponsibility for initiating communication on the rover.This removed the ambiguity of a situation where no data orsignal is received, which would force the operations teamto blindly attempt to initiate contact. Just knowing thatthe rover was attempting to communicate at a particulartime gave us clues as to what was happening onboard.

Lesson— Build in the ability to continue autonomouscommunication, even when onboard fault protectionresponses run. The onboard fault responses did initiatecommunication at unique times so that even if a full dataset didn’t get through, the time of the communicationattempt described what’s happening onboard.

Other Lessons

Lesson—A different file system type, or a more robustimplementation, is required for future missions. The lackof compaction for deleted files in the directory structures isa fundamental flaw for long duration missions.

6. OPERATIONAL & FSW CHANGES

Operational Changes

The problem can be avoided by better oversight of the filesystem use, at the cost of a more conservative approach togathering science data. Since the amount of memoryconsumed is directly related to the structure of the filesystem and the number of files, it is possible to monitorthe free memory space and to terminate activities if theamount of free space drops below a designated limit.

The anomaly team recommended, and the project adopted,the following changes and guidelines. These apply to bothrovers.

• Monitor the amount of free memory space in thesystem. If the remaining space drops to 800 Kbytes orless, then terminate all science and engineeringactivities.

• Aggressively command the deletion of received dataproducts after they have been received on Earth. Thedata management team has reduced the latency for thisaction from 48 to 24 hours.

13

• Remove subdirectories when no more data productfiles exist. This action forces the release of memoryused to represent the subdirectory and eliminates thespace utilized for deleted file entries. This operationalguideline became unnecessary once the FSW wasupdated in April.

• Permanently upgrade the priority of the data productsthat contain the telemetry generated duringinitialization that shows the free system memory spaceand the results of the autonomous file system check.

Onboard Post-Anomaly FSW Changes

Several FSW changes were included in the April 2004FSW load to address the problems and issues discoveredduring the anomaly investigation. These changes included:

• Additional logic to remove the directory list structureentries for deleted files. This change “compacts” thedirectory list structure and thus, removes the “highwater mark” effect. This compaction action runsduring the initialization process before the FLASH filesystem is mounted.

• Additional logic to autonomously enter crippled modewhen multiple resets have occurred. This autonomousaction allows the system to initialize correctly ifanother similar anomaly should occur.

• Modifications to the shutdown logic to use the alarmclock hardware function as a secondary watchdog. Thischange forces a power cycle of the avionics electronicsin cases where the FSW cannot perform all of theshutdown actions. If this occurs during the night, thevehicle will remain powered off until sunrise.

• The register set by the crippled command is volatileand the value is not retained across power cycles. TheFSW was changed to also examine one of the sparenon-volatile registers bits, so the operations team maypermanently force the system to initialize in crippledmode, if necessary.

• Modified the shutdown logic to wait a limited amountof time for the FLASH file system to become idle. Inthe previous FSW version, the shutdown logic wouldwait indefinitely. This change addressed thesemaphore/deadlock issue discovered during theanomaly investigation.

FSW Changes Considered but Not Included

In hindsight, the correct implementation would have beento limit the private memory area used by DOS Library to afixed size. This strategy is consistent with the way allapplication memory space on MER is allocated andmanaged. In this configuration, the free system memorywould not have been consumed and no out-of-memoryevent would have occurred.

However, this change was not incorporated into the newFSW. This was a considered decision that balanced thebenefit versus risk of performing this optimal change.These tradeoffs included:

• Although a modification to limit the size of the DOSLibrary memory area is straightforward, the changemight expose other, unexplored, behavior when theprivate memory space is exhausted. The test programto verify the overall behavior of the system in alloperational modes would be challenging, complex,and time consuming.

• The vehicle lifetime is limited. A test program of thiscomplexity would have significantly delayed theupload of the new FSW version. The new FSWversion also contained many changes unrelated to theanomaly that increased the robustness of the system,optimized the collection and processing of sciencedata, and enhanced the mobility capabilities of thevehicle

• The operational changes addressed the necessary stepsto limit the number of active files in the file system.

7. ACKNOWLEDGEMENTS

The recovery of the vehicle was truly a team effort. Manymembers of the operations team, including the missionmanagers, the flight directors and the subsystem teamsmade extremely valuable contributions to the recovery fromthis anomaly.

There is a small group of individuals the authors wouldlike to recognize for their extraordinary effort: Khaled Ali,Jeff Biesiadecki, Mike Deliman, Jim Donaldson, EdGamble Jr., David Hecox, Roger Klemm, Todd Litwin,Cindy Oda, Ed Odell, David Smyth and Joseph Snyder.

The rover design described in this paper was carried out atthe Jet Propulsion Laboratory, California Institute ofTechnology, under contract with the National Aeronauticsand Space Administration.

8. CONCLUSION

In January 2004, Spirit suffered an anomaly that preventednominal communication for several days. Given every cluethe rover presented us, the anomaly team successfullydiagnosed the problem and recovered this rover back toperfect health. These clues included when the rovercommunicated (and when it didn't), the rates it used totransmit (pre-scheduled rates vs. onboard fault responserates), the type of information it transmitted (real-time vs.recorded, repeating event reports, PN code), and thecommands to which the rover would respond (as well asthose to which it would not respond).

The root cause was a design error in the file services FSWmodule that resulted in an out-of-memory event, causing a

14

processor reset. During re-initialization, when it wouldaccess the FLASH memory, another out-of-memory eventwould occur, so the rover experienced repetitive resets.This rippled into difficulties communicating with Earth, aswell as the inability to shut down the solar-powered rover’selectronics at night to save power.

The predominant factor that led to this error was thecompressed design schedule. Incomplete development,inadequate telemetry and limited testing were a direct resultof the breakneck development pace. Fortunately, this wasoffset by a dedicated team that designed a system withbuilt-in diagnostic tools, autonomous communication, androbust fault protection. It was this foresight that led to therecovery of Spirit.

9. REFERENCES

[1] Glenn Reeves, Tracy Neilson & Todd Litwin, “MarsExploration Rover Spirit Vehicle Anomaly Report,” JetPropulsion Laboratory Document No. D-22919, July 5,2004.

[2] Joseph F. Snyder, David E. Smyth, “Data Managementfor Mars Exploration Rovers”, Jet Propulsion LaboratoryDocument No. D-30712, July 19, 2004.

10. BIOGRAPHY

Glenn E. Reeves is a Principal Engineer at the JetPropulsion Laboratory. Mostrecently he was the FlightSoftware Architect and teamleader for the Mars ExplorationRover project. He also performeda very similar role for the MarsPathfinder mission. The flightsoftware architecture he developedfor the Mars Pathfinder missionhas been adapted and used on

several subsequent spacecraft. Glenn’s professionalpassion is the development of evolutionary andrevolutionary spacecraft autonomy. His currentassignmennt is with the JPL Office of the Chief Engineer.

Tracy Neilson is in the flight systems engineering sectionat JPL. Her role on the MERproject was fault protection leadand she designed various onboardalgorithms for the cruise andsurface phases. One of Tracy’sprevious assignments includedlaunch behavior specification andon-onboard fault protectiondesign for the Deep Space 1mission, a project that flight-tested new technologies such as

deep space ion propulsion and autonomous navigation.She was the Attitude and Articulation Control Subsystemlead for the Galileo mission’s Jupiter orbit insertion and

probe relay, and for the first two years of orbitaloperations.