8/12/2019 They Write the Right Stuff _ Fast Company
1/19
07/08/12 hey Write the Right Stuff | Fast Company
Page 1ttp://www.fastcompany.com/28121/they-write-right-stuff
Skip navigation
Subscribe
Newsletter
Search
Follow @fastcompany 536K followers
Like 15k
Co.DESIGN
Co.CREATE
Co.EXIST
Co.LEAD
TECHNOLOGY
MOST CREATIVE PEOPLE
MAGAZINE
Features
They Write the RightStuff
y Charles Fishman
ecember 31, 1996
As the 120-ton space shuttle sits surrounded by almost 4 million pounds of rocket fuel, exhaling noxious fumes,
isibly impatient to defy gravity, its on-board computers take command.
The right stuff kicks in at T-minus 31 seconds.
As the 120-ton space shuttle sits surrounded by almost 4 million pounds of rocket fuel, exhaling noxious fumes,
isibly impatient to defy gravity, its on-board computers take command. Four identical machines, runningdentical software, pull information from thousands of sensors, make hundreds of milli-second decisions, vote on
very decision, check with each other 250 times a second. A fifth computer, with different software, stands by to
ake control should the other four malfunction.
At T-minus 6.6 seconds, if the pressures, pumps, and temperatures are nominal, the computers give the order to
ght the shuttle main engines -- each of the three engines firing off precisely 160 milliseconds apart, tons of
uper-cooled liquid fuel pouring into combustion chambers, the ship rocking on its launch pad, held to the ground
http://www.fastcompany.com/most-creative-people/2012http://www.fastcompany.com/technologyhttp://www.fastcompany.com/leadershiphttp://www.fastcodesign.com/http://ad.doubleclick.net/clk;254486090;45085438;o?http://itunes.apple.com/us/app/fast-company-magazine/id500813317?mt=8http://ad.doubleclick.net/clk;254486090;45085438;o?http://itunes.apple.com/us/app/fast-company-magazine/id500813317?mt=8http://ad.doubleclick.net/click%3Bh%3Dv8/3cca/2/0/%2a/d%3B257850051%3B0-0%3B0%3B43924277%3B3454-728/90%3B48567120/48565548/1%3B%3B~aopt%3D2/1/ff/0%3B~sscs%3D%3fhttp://googleads.g.doubleclick.net/aclk?sa=L&ai=BDi9XlochUKqeI5KwsQfX7oDQDqeZpbkCx4uPnS7PibiXFvD21gUQARgBINTsmhU4AFDjkIGJ______8BYOXj84SAFrIBE3d3dy5mYXN0Y29tcGFueS5jb226AQk3Mjh4OTBfYXPIAQLaATdodHRwOi8vd3d3LmZhc3Rjb21wYW55LmNvbS8yODEyMS90aGV5LXdyaXRlLXJpZ2h0LXN0dWZmqQIREOj2lVh0PsACBMgCt4-XHKgDAcgDHegDnAXoAwX1AwIEAEDIBAGgBgI&num=1&sig=AOD64_1YvM5cZ0jK2MJhuimDu8zQo0zB9Q&client=ca-pub-9569526099065935&adurl=http://www.telcel.com/portal/roaming/begin.do%3Fmid%3D1500http://ad.doubleclick.net/click%3Bh%3Dv8/3cca/2/0/%2a/d%3B257850051%3B0-0%3B0%3B43924277%3B3454-728/90%3B48567120/48565548/1%3B%3B~aopt%3D2/1/ff/0%3B~sscs%3D%3fhttp://googleads.g.doubleclick.net/aclk?sa=L&ai=BDi9XlochUKqeI5KwsQfX7oDQDqeZpbkCx4uPnS7PibiXFvD21gUQARgBINTsmhU4AFDjkIGJ______8BYOXj84SAFrIBE3d3dy5mYXN0Y29tcGFueS5jb226AQk3Mjh4OTBfYXPIAQLaATdodHRwOi8vd3d3LmZhc3Rjb21wYW55LmNvbS8yODEyMS90aGV5LXdyaXRlLXJpZ2h0LXN0dWZmqQIREOj2lVh0PsACBMgCt4-XHKgDAcgDHegDnAXoAwX1AwIEAEDIBAGgBgI&num=1&sig=AOD64_1YvM5cZ0jK2MJhuimDu8zQo0zB9Q&client=ca-pub-9569526099065935&adurl=http://www.telcel.com/portal/roaming/begin.do%3Fmid%3D1500http://ad.doubleclick.net/click%3Bh%3Dv8/3cca/2/0/%2a/d%3B257850051%3B0-0%3B0%3B43924277%3B3454-728/90%3B48567120/48565548/1%3B%3B~aopt%3D2/1/ff/0%3B~sscs%3D%3fhttp://googleads.g.doubleclick.net/aclk?sa=L&ai=BDi9XlochUKqeI5KwsQfX7oDQDqeZpbkCx4uPnS7PibiXFvD21gUQARgBINTsmhU4AFDjkIGJ______8BYOXj84SAFrIBE3d3dy5mYXN0Y29tcGFueS5jb226AQk3Mjh4OTBfYXPIAQLaATdodHRwOi8vd3d3LmZhc3Rjb21wYW55LmNvbS8yODEyMS90aGV5LXdyaXRlLXJpZ2h0LXN0dWZmqQIREOj2lVh0PsACBMgCt4-XHKgDAcgDHegDnAXoAwX1AwIEAEDIBAGgBgI&num=1&sig=AOD64_1YvM5cZ0jK2MJhuimDu8zQo0zB9Q&client=ca-pub-9569526099065935&adurl=http://www.telcel.com/portal/roaming/begin.do%3Fmid%3D1500http://www.fastcompany.com/28121/they-write-right-stuff#searchhttp://www.fastcompany.com/#contenthttp://www.fastcompany.com/user/charles-fishmanhttp://www.fastcompany.com/magazine/6http://www.fastcompany.com/magazinehttp://www.fastcompany.com/most-creative-people/2012http://www.fastcompany.com/technologyhttp://www.fastcompany.com/leadershiphttp://www.fastcoexist.com/http://www.fastcocreate.com/http://www.fastcodesign.com/http://ad.doubleclick.net/clk;254486090;45085438;o?http://itunes.apple.com/us/app/fast-company-magazine/id500813317?mt=8http://www.fastcompany.com/28121/they-write-right-stuff#searchhttp://www.fastcompany.com/newslettershttps://magazine.fastcompany.com/loc/FST/topnavhttp://www.fastcompany.com/#contenthttp://ad.doubleclick.net/click%3Bh%3Dv8/3cca/2/0/%2a/d%3B257850051%3B0-0%3B0%3B43924277%3B3454-728/90%3B48567120/48565548/1%3B%3B~aopt%3D2/1/ff/0%3B~sscs%3D%3fhttp://googleads.g.doubleclick.net/aclk?sa=L&ai=BDi9XlochUKqeI5KwsQfX7oDQDqeZpbkCx4uPnS7PibiXFvD21gUQARgBINTsmhU4AFDjkIGJ______8BYOXj84SAFrIBE3d3dy5mYXN0Y29tcGFueS5jb226AQk3Mjh4OTBfYXPIAQLaATdodHRwOi8vd3d3LmZhc3Rjb21wYW55LmNvbS8yODEyMS90aGV5LXdyaXRlLXJpZ2h0LXN0dWZmqQIREOj2lVh0PsACBMgCt4-XHKgDAcgDHegDnAXoAwX1AwIEAEDIBAGgBgI&num=1&sig=AOD64_1YvM5cZ0jK2MJhuimDu8zQo0zB9Q&client=ca-pub-9569526099065935&adurl=http://www.telcel.com/portal/roaming/begin.do%3Fmid%3D1500https://twitter.com/intent/user?original_referer=http%3A%2F%2Fwww.fastcompany.com%2F28121%2Fthey-write-right-stuff®ion=count_link&screen_name=fastcompany&source=followbutton&variant=2.0https://twitter.com/intent/follow?original_referer=http%3A%2F%2Fwww.fastcompany.com%2F28121%2Fthey-write-right-stuff®ion=follow_link&screen_name=fastcompany&source=followbutton&variant=2.08/12/2019 They Write the Right Stuff _ Fast Company
2/19
07/08/12 hey Write the Right Stuff | Fast Company
Page 2ttp://www.fastcompany.com/28121/they-write-right-stuff
nly by bolts. As the main engines come to one million pounds of thrust, their exhausts tighten into blue
iamonds of flame.
Then and only then at T-minus zero seconds, if the computers are satisfied that the engines are running true,
hey give the order to light the solid rocket boosters. In less than one second, they achieve 6.6 million pounds of
hrust. And at that exact same moment, the computers give the order for the explosive bolts to blow, and 4.5
million pounds of spacecraft lifts majestically off its launch pad.
's an awesome display of hardware prowess. But no human pushes a button to make it happen, no astronaut
ockeys a joy stick to settle the shuttle into orbit.
The right stuff is the software. The software gives the orders to gimbal the main engines, executing the dramatic
elly roll the shuttle does soon after it clears the tower. The software throttles the engines to make sure the craft
oesn't accelerate too fast. It keeps track of where the shuttle is, orders the solid rocket boosters to fall away,
makes minor course corrections, and after about 10 minutes, directs the shuttle into orbit more than 100 miles
p. When the software is satisfied with the shuttle's position in space, it orders the main engines to shut down --
weightlessness begins and everything starts to float.
But how much work the software does is not what makes it remarkable. What makes it remarkable is how well
he software works. This software never crashes. It never needs to be re-booted. This software is bug-free. It is
erfect, as perfect as human beings have achieved. Consider these stats : the last three versions of the program
- each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors.
Commercial programs of equivalent complexity would have 5,000 errors.
This software is the work of 260 women and men based in an anonymous office building across the street from
he Johnson Space Center in Clear Lake, Texas, southeast of Houston. They work for the "on-board shuttle
roup," a branch of Lockheed Martin Corps space mission systems division, and their prowess is world
enowned: the shuttle software group is one of just four outfits in the world to win the coveted Level 5 ranking of
he federal governments Software Engineering Institute (SEI) a measure of the sophistication and reliability of the
way they do their work. In fact, the SEI based it standards in part from watching the on-board shuttle group do its
work.
The group writes software this good because that's how good it has to be. Every time it fires up the shuttle, their
oftware is controlling a $4 billion piece of equipment, the lives of a half-dozen astronauts, and the dreams of the
ation. Even the smallest error in space can have enormous consequences: the orbiting space shuttle travels at
7,500 miles per hour; a bug that causes a timing problem of just two-thirds of a second puts the space shuttle
hree miles off course.
NASA knows how good the software has to be. Before every flight, Ted Keller, the senior technical manager of
he on-board shuttle group, flies to Florida where he signs a document certifying that the software will not
ndanger the shuttle. If Keller can't go, a formal line of succession dictates who can sign in his place.
Bill Pate, who's worked on the space flight software over the last 22 years, says the group understands the
8/12/2019 They Write the Right Stuff _ Fast Company
3/19
07/08/12 hey Write the Right Stuff | Fast Company
Page 3ttp://www.fastcompany.com/28121/they-write-right-stuff
takes: "If the software isn't perfect, some of the people we go to meetings with might die.
Software is everything. (It also sucks.)
n the history of human technology, nothing has become as essential as fast as software.
Virtually everything -- from the international monetary system and major power plants to blenders and microwavevens -- runs on software. In office buildings, the elevators, the lights, the water, the air conditioning are all
ontrolled by software. In cars, the transmission, the ignition timing, the air bag, even the door locks are
ontrolled by software. In most cities so are the traffic lights. Almost every written communication that's more
omplicated than a postcard depends on software; every phone conversation and every overnight package
elivery requires it.
Software is everything. It also sucks.
t's like pre-Sumerian civilization," says Brad Cox, who wrote the software for Steve Jobs NeXT computer and is
professor at George Mason University. "The way we build software is in the hunter-gatherer stage."
ohn Munson, a software engineer and professor of computer science at the University of Idaho, is not quite so
enerous. "Cave art," he says. "It's primitive. We supposedly teach computer science. There's no science here at
ll."
Software may power the post-industrial world, but the creation of software remains a pre-industrial trade.
According to SEI's studies, nearly 70% of software organizations are stuck in the first two levels of SEI's scale of
ophistication: chaos, and slightly better than chaos. The situation is so severe, a few software pioneers from
ompanies such as Microsoft have broken away to teach the art of software creation (see "Drop and Code me
Twenty!" )
Mark Paulk, a senior member of the SEI technical, says the success of software makes its weaknesses all the
more dramatic. "We've developed software products that are enormously complex and enormously powerful.
We're critically dependent on it," says Paulk. "Yet everyone complains how bad software is, with all the defects. If
ou bought a car with 5,000 defects, you'd be very upset."
n this software morass, the on-board shuttle group stands out as an exception. Ten years ago the shuttle group
was considered world-class. Since then, it has cut its own error rate by 90%.
To be this good, the on-board shuttle group has to be very different -- the antithesis of the up-all-night, pizza-and-
oller-hockey software coders who have captured the public imagination. To be this good, the on-board shuttle
roup has to be very ordinary -- indistinguishable from any focused, disciplined, and methodically managed
reative enterprise.
n fact, the group offers a set of textbook lessons that applies equally to programmers, in particular, and
roducers, in general. A look at the culture they have built and the process they have perfected shows what
http://www.fastcompany.com/magazine/06/dropcode.html8/12/2019 They Write the Right Stuff _ Fast Company
4/19
07/08/12 hey Write the Right Stuff | Fast Company
Page 4ttp://www.fastcompany.com/28121/they-write-right-stuff
oftware-writing must become if software is to realize its promise, and illustrates what almost any team-based
peration can do to boost its performance to achieve near-perfect results.
Software for Grown-Ups
Shipping hell continued today. Grind, grind, grind. We'll never make it. Have I said that already? Why do we
lways underestimate our shipping schedules? I just don't understand. In at 9:30 AM; out at 11:30 PM Dominosor dinner. And three diet Cokes."
No, it's not the on-board shuttle group. It's Douglas Coupland's "Microserf's," a true-to-life fictional account of life
n the software-fast-lane. And it's the dominant image of the software development world: Gen-Xers sporting T-
hirts and distracted looks, squeezing too much heroic code writing into too little time; rollerblades and mountain
ikes tucked in corners; pizza boxes and Starbucks cups discarded in conference rooms; dueling tunes from
Smashing Pumpkins, Alanis Morrisette and the Fugees. Its the world made famous, romantic, even inevitable by
tories out of Sun Microsystems, Microsoft, and Netscape.
's not the story of the on-board shuttle group. Their quarters are a study in white-collar pedestrian. The most
triking thing is how ordinary they look. Other than the occasional bit of shuttle memorabilia, you could be in the
ffices of any small company or government agency. Everyone has his or her own small office, and the offices
ave desks, PCs, and sparse personal artifacts. People wear moderately dressy clothes to work, neat but
othing flashy, certainly nothing grungy.
's strictly an 8-to-5 kind of place -- there are late nights, but they're the exception. The programmers are
ntense, but low-key. Many of them have put in years of work either for IBM (which owned the shuttle group until
994), or directly on the shuttle software. They're adults, with spouses and kids and lives beyond their
emarkable software program.
That's the culture: the on-board shuttle group produces grown-up software, and the way they do it is by being
rown-ups. It may not be sexy, it may not be a coding ego-trip -- but it is the future of software. When you're
eady to take the next step -- when you have to write perfect software instead of software that's just good enough
- then it's time to grow up.
Ted Keller, 48, the group's senior technical manager, looks and sounds like the headmaster of a small private
igh school. It's Keller's job to make sure the software gets delivered on time, with all its capabilities, without
egard to turf battles. He's a compact, bald man, a little officious and persnickety, qualities any astronaut would
nd reassuring. He has a gentle, geeky sense of humor, not so much with outsiders, but with his crowd.
comes out in a meeting between members of the software group and their NASA counterparts. It's held in a
mall conference room overstuffed with 22 people and an overhead projector. Several times, from the back of
he room, Keller issues a wry remark about the speed of code delivery, or the detail of some specifications, and
he room lightens with laughter.
Otherwise, the hour-long meeting is sober and revealing, a brief window on the culture. For one thing, 12 of the
8/12/2019 They Write the Right Stuff _ Fast Company
5/19
07/08/12 hey Write the Right Stuff | Fast Company
Page 5ttp://www.fastcompany.com/28121/they-write-right-stuff
2 people in the room are women, many of them senior managers or senior technical staff. The on-board shuttle
roup, with its stability and professionalism, seems particularly appealing to women programmers.
For another, it's an exercise in order, detail, and methodical reiteration. The meeting is a classic NASA
erformance -- a rehearsal for an almost identical meeting several weeks away. It consists of walking through an
normous packet of data and view -- graphs which describe the progress and status of the software line by line.
Except for Keller's occasional asides, the tone is businesslike, almost formal, the view - graphs flashing past as
uickly as they can be read, a blur of acronyms, graphs, and charts.
What's going on here is the kind of nuts-and-bolts work that defines the drive for group perfection -- a drive that is
ggressively intolerant of ego-driven hotshots. In the shuttle group's culture, there are no superstar
rogrammers. The whole approach to developing software is intentionally designed not to rely on any particular
erson.
And the culture is equally intolerant of creativity, the individual coding flourishes and styles that are the signature
f the all-night software world. "People ask, doesn't this process stifle creativity? You have to do exactly what the
manual says, and you've got someone looking over your shoulder," says Keller. "The answer is, yes, the processoes stifle creativity."
And that is precisely the point -- you can't have people freelancing their way through software code that flies a
paceship, and then, with peoples lives depending on it, try to patch it once its in orbit. "Houston, we have a
roblem," may make for a good movie; it's no way to write software. "People have to channel their creativity into
hanging the process," says Keller, "not changing the software."
The tight strictures the group practices can make the siren song of the rock n roll software world hard to resist.
Quinn Larson, 34, had worked on shuttle software for seven years when he left last January to go to work for
Micron Technology in Boise, Idaho, automating the manufacture of Microns memory chips.
At Micron, Larson was given the task to automate the saws that cut finished chip wafers to the right size. Screw
p the program, you destroy the valuable wafers.
t was up to me to decide what to do," says Larson. "There were no meetings, there was no record-keeping." He
ad freedom; it was a real kick. But to Larson's way of thinking, the culture didn't focus on, well, the right stuff.
Speed there was the biggest thing," he says. "The engineers would say, these are our top priorities, and we
eed to get'em as fast as we can." But the impression Larson got was that engineers didn't seem too concerned
bout how well the finished software actually worked. "Basically, they wanted quick software -- just put it out the
oor."
arson started back at the shuttle group in mid-August. "The people here are just of the highest caliber," he said
n his first day back in Clear Lake.
t's the Process
8/12/2019 They Write the Right Stuff _ Fast Company
6/19
07/08/12 hey Write the Right Stuff | Fast Company
Page 6ttp://www.fastcompany.com/28121/they-write-right-stuff
How do they write the right stuff?
The answer is, it's the process. The group's most important creation is not the perfect software they write -- it's
he process they invented that writes the perfect software.
's the process that allows them to live normal lives, to set deadlines they actually meet, to stay on budget, to
eliver software that does exactly what it promises. It's the process that defines what these coders in the flat
lains of southeast suburban Houston know that everyone else in the software world is still groping for. It's the
rocess that offers a template for any creative enterprise that's looking for a method to produce consistent - and
onsistently improving -- quality.
The process can be reduced to four simple propositions:
. The product is only as good as the plan for the product.
At the on-board shuttle group, about one-third of the process of writing software happens before anyone writes a
ne of code. NASA and the Lockheed Martin group agree in the most minute detail about everything the newode is supposed to do -- and they commit that understanding to paper, with the kind of specificity and precision
sually found in blueprints. Nothing in the specs is changed without agreement and understanding from both
ides. And no coder changes a single line of code without specs carefully outlining the change. Take the upgrade
f the software to permit the shuttle to navigate with Global Positioning Satellites, a change that involves just
.5% of the program, or 6,366 lines of code. The specs for that one change run 2,500 pages, a volume thicker
han a phone book. The specs for the current program fill 30 volumes and run 40,000 pages.
Our requirements are almost pseudo-code," says William R. Pruett, who manages the software project for
NASA. "They say, you must do exactly this, do it exactly this way, given this condition and this circumstance."
This careful design process alone is enough to put the shuttle organization in a class by itself, says John Munson
f the University of Idaho. Most organizations launch into even big projects without planning what the software
must do in blueprint-like detail. So after coders have already started writing a program, the customer is busily
hanging its design. The result is chaotic, costly programming where code is constantly being changed and
nfected with errors, even as it is being designed.
Most people choose to spend their money at the wrong end of the process," says Munson. "In the modern
oftware environment, 80% of the cost of the software is spent after the software is written the first time -- they
on't get it right the first time, so they spend time flogging it. In shuttle, they do it right the first time. And they
on't change the software without changing the blueprint. That's why their software is so perfect."
. The best teamwork is a healthy rivalry.
Within the software group, there are subgroups and subcultures. But what could be divisive office politics in other
rganizations is actually a critical part of the process.
8/12/2019 They Write the Right Stuff _ Fast Company
7/19
07/08/12 hey Write the Right Stuff | Fast Company
Page 7ttp://www.fastcompany.com/28121/they-write-right-stuff
The central group breaks down into two key teams: the coders - the people who sit and write code -- and the
erifiers -- the people who try to find flaws in the code. The two outfits report to separate bosses and function
nder opposing marching orders. The development group is supposed to deliver completely error-free code, so
erfect that the testers find no flaws at all. The testing group is supposed to pummel away at the code with flight
cenarios and simulations that reveal as many flaws as possible. The result is what Tom Peterson calls "a
riendly adversarial relationship."
They're in competition for who's going to find the errors," says Keller. "Sometimes they fight like cats and dogs.
The developers want to catch all their own errors. The verifiers get mad, 'Hey, give it up! You're taking away from
ur time to test the software!'"
The developers have even begun their own formal inspections of the code in carefully moderated sessions, a
gorous proof-reading they hope will confound the testers. The verifiers, in turn, argue that they deserve credit
or some errors found before they even start testing. "From the verification group's perspective," says Pat
McLellan, a senior manager, "we're aware that if there was no independent verification group, the developers
would tend to be more lax. Just the presence of our group makes them more careful."
The results of this friendly rivalry: the shuttle group now finds 85% of its errors before formal testing begins, and
9.9% before the program is delivered to NASA.
. The database is the software base.
There is the software. And then there are the databases beneath the software, two enormous databases,
ncyclopedic in their comprehensiveness.
One is the history of the code itself -- with every line annotated, showing every time it was changed, why it was
hanged, when it was changed, what the purpose of the change was, what specifications documents detail the
hange. Everything that happens to the program is recorded in its master history. The genealogy of every line of
ode -- the reason it is the way it is -- is instantly available to everyone.
The other database -- the error database -- stands as a kind of monument to the way the on-board shuttle group
oes about its work. Here is recorded every single error ever made while writing or working on the software,
oing back almost 20 years. For every one of those errors, the database records when the error was discovered;
what set of commands revealed the error; who discovered it; what activity was going on when it was discovered -
testing, training, or flight. It tracks how the error was introduced into the program; how the error managed to slip
ast the filters set up at every stage to catch errors -- why wasn't it caught during design? during development
nspections? during verification? Finally, the database records how the error was corrected, and whether similar
rrors might have slipped through the same holes.
The group has so much data accumulated about how it does its work that it has written software programs that
model the code-writing process. Like computer models predicting the weather, the coding models predict how
many errors the group should make in writing each new version of the software. True to form, if the coders and
esters find too few errors, everyone works the process until reality and the predictions match.
8/12/2019 They Write the Right Stuff _ Fast Company
8/19
07/08/12 hey Write the Right Stuff | Fast Company
Page 8ttp://www.fastcompany.com/28121/they-write-right-stuff
We never let anything go," says Patti Thornton, a senior manager. "We do just the opposite: we let everything
other us."
. Don't just fix the mistakes -- fix whatever permitted the mistake in the first place.
The process is so pervasive, it gets the blame for any error -- if there is a flaw in the software, there must be
omething wrong with the way its being written, something that can be corrected. Any error not found at the
lanning stage has slipped through at least some checks. Why? Is there something wrong with the inspection
rocess? Does a question need to be added to a checklist?
mportantly, the group avoids blaming people for errors. The process assumes blame - and it's the process that
s analyzed to discover why and how an error got through. At the same time, accountability is a team concept: no
ne person is ever solely responsible for writing or inspecting code. "You don't get punished for making errors,"
ays Marjorie Seiter, a senior member of the technical staff. "If I make a mistake, and others reviewed my work,
hen I'm not alone. I'm not being blamed for this."
Ted Keller offers an example of the payoff of the approach, involving the shuttles remote manipulator arm. "Weelivered software for crew training," says Keller, "that allows the astronauts to manipulate the arm, and handle
he payload. When the arm got to a certain point, it simply stopped moving."
The software was confused because of a programming error. As the wrist of the remote arm approached a
omplete 360-degree rotation, flawed calculations caused the software to think the arm had gone past a
omplete rotation -- which the software knew was incorrect. The problem had to do with rounding off the answer
o an ordinary math problem, but it revealed a cascade of other problems.
Even though this was not critical," says Keller, "we went back and asked what other lines of code might have
xactly the same kind of problem." They found eight such situations in the code, and in seven of them, the
ounding off function was not a problem. "One of them involved the high-gain antenna pointing routine," says
Keller. "That's the main antenna. If it had developed this problem, it could have interrupted communications with
he ground at a critical time. That's a lot more serious."
The way the process works, it not only finds errors in the software. The process finds errors in the process.
ust a Software Problem
The B-2 bomber wouldn't fly on its maiden flight -- but it was just a software problem. The new Denver airportwas months late opening and millions of dollars over budget because its baggage handling system didn't work
ght -- but it was just a software problem. This spring, the European Space Agency's new Ariane 5 rocket blew
p on its maiden launch because of a little software problem. The federal government's major agencies - from
he IRS to the National Weather Service -- are beset with projects that are years late and hundreds of millions of
ollars over budget, often because of simple software problems. Software is getting more and more common and
more and more important, but it doesn't seem to be getting more and more reliable.
8/12/2019 They Write the Right Stuff _ Fast Company
9/19
07/08/12 hey Write the Right Stuff | Fast Company
Page 9ttp://www.fastcompany.com/28121/they-write-right-stuff
As the rest of the world struggles with the basics, the on-board shuttle group edges ever closer to perfect
oftware. Admittedly they have a lot of advantages over the rest of the software world. They have a single
roduct: one program that flies one spaceship. They understand their software intimately, and they get more
amiliar with it all the time. The group has one customer, a smart one. And money is not the critical constraint: the
roups $35 million per year budget is a trivial slice of the NASA pie, but on a dollars-per-line basis, it makes the
roup among the nation's most expensive software organizations.
And that's the point: the shuttle process is so extreme, the drive for perfection is so focused, that it reveals what's
equired to achieve relentless execution. The most important things the shuttle group does -- carefully planning
he software in advance, writing no code until the design is complete, making no changes without supporting
lueprints, keeping a completely accurate record of the code -- are not expensive. The process isn't even rocket
cience. Its standard practice in almost every engineering discipline except software engineering.
Plastered on a conference room wall, an informal slogan of the on-board shuttle group captures the essence of
eeping focused on the process: "The sooner you fall behind, the more time you will have to catch up."
Charles Fishman ([email protected]) is a writer based in Raleigh, North Carolina.
A version of this article appears in the Dec 1996/Jan 1997issue of Fast Company.
You Might Also Like
[?]
Share:
Tweet
Watch This ManBuild An AmazingCardboard BicycleThat You CanActually Ride
CO.EXIST
The 4 DumbestRules That Will KillYour Company'sCulture | FastCompany
FAST COMPANY
Top 5 FranchisesWorth Looking At
THE BUSINESS
FINANCE STORE
ShareShare
45Like
FROM THE WEB
http://www.facebook.com/plugins/like.php?action=like&api_key=178479832213933&channel_url=http%3A%2F%2Fstatic.ak.facebook.com%2Fconnect%2Fxd_arbiter.php%3Fversion%3D9%23cb%3Df19dc5c88%26origin%3Dhttp%253A%252F%252Fwww.fastcompany.com%252Ff356e1e4c8%26domain%3Dwww.fastcompany.com%26relation%3Dparent.parent&extended_social_context=false&href=http%3A%2F%2Fwww.fastcompany.com%2Fnode%2F28121&layout=button_count&locale=en_US&node_type=link&sdk=joey&send=false&show_faces=false&width=90#http://www.fastcompany.com/print/28121http://www.fastcompany.com/28121/they-write-right-stuff#http://www.fastcompany.com/magazine/6https://twitter.com/intent/tweet?original_referer=http%3A%2F%2Fwww.fastcompany.com%2F28121%2Fthey-write-right-stuff&related=null&source=tweetbutton&text=They%20Write%20the%20Right%20Stuff&url=http%3A%2F%2Fwww.fastcompany.com%2Fnode%2F28121&via=FastCompanyhttp://void%280%29/8/12/2019 They Write the Right Stuff _ Fast Company
10/19
07/08/12 hey Write the Right Stuff | Fast Company
Page 10ttp://www.fastcompany.com/28121/they-write-right-stuff
Like
Sort by best rating
Newsletter
Subscribe
Browse Current Issue !
Real-time updating is enabled. (Pause)
Showing 11 comments
LoginAdd New Comment
Type your comment here.
Hi man I and hundreds of others was writing flawless code back in the dark ages, apart from raw
machine code, assembler and Fortran, there was C not "C++" or "C#" or C~!@#$%^ just plane C the
brain child of two American heros Brian Kernigan and Dennis M Ritchie and straight out of Bell Labs.
using C everything from 4-bit micro controllers to the UNIX operating system has been crafted; and here
is the rub. Something like the sequence control you describe for the Space Shuttle is fairly basic
compared to an Oil Refinery or a Nuclear Power Plants safety critical systems which if it fails would be a
lot more lethal, if less spectacular than a ST brewing up on the pad. You can also bet these are multe-
redundant systems with every system duplicated not once but a number of times. The Apollo 11 moon
lander Eagle had 7 systems - theyt all failed just above the surface of the moon so it was left to NeilArmstrong to land by the seat of his pant - proof that the best redundant system is a human, you cannot
beat the skills of trained U.S military pilots Navy, Marine or Airforce, nor the observational capacity of
the mark-one eyeballs attached to such gentalmen.
Spirals
cortney valentina
http://disqus.com/fastcompany-249515/http://www.fastcompany.com/28121/they-write-right-stuff#http://www.fastcompany.com/28121/they-write-right-stuff#http://www.fastcompany.com/28121/they-write-right-stuff#http://www.fastcompany.com/magazine/currenthttp://www.fastcompany.com/magazine/currenthttps://magazine.fastcompany.com/loc/FST/articlehttp://www.fastcompany.com/newslettershttp://www.fastcompany.com/forward/?path=node/28121http://www.fastcompany.com/28121/they-write-right-stuff#http://www.fastcompany.com/28121/they-write-right-stuff#http://www.fastcompany.com/28121/they-write-right-stuff#8/12/2019 They Write the Right Stuff _ Fast Company
11/19
07/08/12 hey Write the Right Stuff | Fast Company
Page 11ttp://www.fastcompany.com/28121/they-write-right-stuff
This is a pretty interesting article, with some strong points. Thanks for the post.
so good paper. thanks gor your share good information,plz keep up good work.
...hot videos
james lena
Outside, in the real world, this isnt possible.
Costs way too much, has external clients to please...
Done that way, a photoshop licence would cost millions...
Bob
Bleh, I think if I had to work in an environment like that I would shoot myself within a month.
Programming is as much art as science, and the best programmers I know are also the most creative.
While I admit that when programs are controlling life and death, you obviously don't want mistakes in
your code, but most programs are not about life and death.
Making a hugely complex program that has 5000+ bugs is normal because we don't want to take 20
years to write 420,000 lines of code. We want to do 1,000,000+ lines in less than a year, get things to a
state that they work in almost all cases, then move on to bigger and better things. Those 5000 bugs thearticle talks about are usually edge cases that just don't MATTER (i.e. If my character walks too close to
a specific model of a standing lamp when I have the hairstyle blonde_long_31, the ends of the hair clip
slightly into the lamp. So what. We leave those 5000 bugs in because no one gives a shit about them).
Obviously, much worse bugs DO slip through the cracks sometimes, and no one is proud of that, but
bugs like that are the relatively rare exception. Evolution is about making mistakes and all good
programmers make mistakes. But the industry as a whole has made incredible technology possible
because we've not been afraid to try big things and fall flat on our faces sometimes. The Computer
Science profs the article quotes are idiots. Our industry is not in a cave-man stage at all. We've
propelled many other industries OUT of the cave-man stage because of the HUGE leaps and bounds of
progress in software engineering.And contrary to what the article implies, the mountain-dew-fueled,
bleary-eyed programmer culture is just as responsible for the success of that endeavor as the suit-and-
tie programmers in that company. I'll bet they use IDEs made by those imperfect programmers. I'll bet
they didn't write the OS of the computers they use to write their code. I'm pretty damn sure the science
they rely on to launch that shuttle was created by lots of engineers, physicists, chemists, etc. that would
never have been able to make the discoveries, manufacturing processes, or or economies of scale that
made things possible without using software created by imperfect programmers.The mindset of anyone
that would want all programmers to be like the ones in the article boggles me. The corporate suits want
programmers to be predictable and boring. In a limited number of cases (specifically ones like the article
Arthur Lee Wilson
http://www.facebook.com/people/Arthur-Lee-Wilson/578098929http://disqus.com/facebook-578098929/http://disqus.com/guest/84ea925f6472174984dd0819c893c10c/http://disqus.com/guest/82e1733fc7fdfafd6698593824208887/http://www.porn007.net/http://disqus.com/guest/e020003025524beaae569097d06f0494/8/12/2019 They Write the Right Stuff _ Fast Company
12/19
07/08/12 hey Write the Right Stuff | Fast Company
Page 12ttp://www.fastcompany.com/28121/they-write-right-stuff
describes - where the software HAS to be right or people DIE), that kind of programming is necessary
and I applaud the programmers who can do it. If we ALL worked like that, though, the world would be a
VERY different (and far less advanced!) place!
I could probably write this perfection of space shuttle software in about 200 lines of PHP... Just sayin'.
Greg Bowler
"Four identical machines, running identical software, pull information from thousands of sensors, make
hundreds of milli-second decisions, vote on every decision, check with each other 250 times a second"
This is where the Amaizing happens..pretty impressive computations..
Albert Mutangiri
That's where the Amazing happens => Four identical machines, running identical software, pull
information from thousands of sensors, make hundreds of milli-second decisions, vote on every
decision, check with each other 250 times a second. Pretty impressive computations!!
Albert Mutangiri
Ive never been interested in all that nasa stuff, but this article made me think. im of to wikipedia to read
some more :)
andy k
that is stupid to pay 35 millions just for maintenance
Video Girl
With all due respect to the author, $35 million per year to maintain a 420,000 line program is expensive.
Tony BenBrahim
http://disqus.com/guest/5492e2550d35190f483a5d3ed9a56cbf/http://disqus.com/guest/c016ca2c042566a87133982520c8857f/http://disqus.com/guest/759093e5aeedf1d175981ca2447a795c/http://disqus.com/guest/7af215eebd2abd3ef8ee6d3f0cc7298f/http://disqus.com/guest/7af215eebd2abd3ef8ee6d3f0cc7298f/http://disqus.com/fastcompany-d4ea190043f3b44b7b91229d7a23b44c/8/12/2019 They Write the Right Stuff _ Fast Company
13/19
07/08/12 hey Write the Right Stuff | Fast Company
Page 13ttp://www.fastcompany.com/28121/they-write-right-stuff
M Subscribe by email S RSS
Reactions
Who's That Woman in the Twitter Bot Profile?
Most organizations and consumers are not willing to shoulder than kind of expense. And no, this kind of
process is not standard in every engineering discipline. There are tradeoffs for example in automotive
design, it is possible to design and build a car that never breaks down, but would anyone be able to
afford it? Yes, there are more rigourous engineering processes when the stakes are high (airplanes,
chemical plants, etc..), but they come with commensurate costs. For a consumer operating system or a
word processor, or a corporate line of business application, faster and cheaper wins out very time over
flawless and unaffordable.
http://twitter.com/rhetonik/status/232354139571630080http://ad.doubleclick.net/click%3Bh%3Dv8/3cca/3/0/%2a/h%3B253829833%3B0-0%3B0%3B43924277%3B18754-336/150%3B47738169/47753592/1%3B%3B~aopt%3D2/1/ff/0%3B~sscs%3D%3fhttp://ad.doubleclick.net/clk;256069769;76982327;hhttp://www.fastcompany.com/3000064/whos-woman-twitter-bot-profilehttp://www.fastcompany.com/3000064/whos-woman-twitter-bot-profilehttp://fastcompany.disqus.com/they_write_the_right_stuff/latest.rsshttp://www.fastcompany.com/28121/they-write-right-stuff#8/12/2019 They Write the Right Stuff _ Fast Company
14/19
07/08/12 hey Write the Right Stuff | Fast Company
Page 14ttp://www.fastcompany.com/28121/they-write-right-stuff
Inside Axe's Highly Scientific, Totally Irresistible
Marketing Machine Built On Lust
It Takes Guts To Start A Company--So How Do YouGet 'Em?
The Surprisingly Smart Strategy Behind LondonsInfamous Olympic Branding
http://www.fastcodesign.com/1670429/the-surprisingly-smart-strategy-behind-london-s-infamous-olympic-brandinghttp://www.fastcompany.com/3000132/it-takes-guts-start-company-so-how-do-you-get-emhttp://www.fastcompany.com/3000132/it-takes-guts-start-company-so-how-do-you-get-emhttp://www.fastcompany.com/3000210/inside-axes-highly-scientific-totally-irresistible-marketing-machine-built-lusthttp://www.fastcocreate.com/1681417/inside-axes-highly-scientific-totally-irresistible-marketing-machine-built-on-lusthttp://ad.doubleclick.net/click;h=v8/3cca/0/0/%2a/v;253330114;0-0;0;43924277;4307-300/250;46615438/46632081/1;;~aopt=2/1/ff/0;~sscs=%3fhttp://www.fastcodesign.com/8/12/2019 They Write the Right Stuff _ Fast Company
15/19
07/08/12 hey Write the Right Stuff | Fast Company
Page 15ttp://www.fastcompany.com/28121/they-write-right-stuff
Office 15: Microsoft's Best Bet For Beating Dropbox, iCloud, Google Drive
With one billion Office users worldwide, the newest upgrade to Microsoft's productivity suite ...
Infographic: A Map Of Every Big Idea Ever Printed
http://www.fastcodesign.com/1670395/infographic-a-map-of-every-big-idea-ever-printedhttp://ad.doubleclick.net/click;h=v8/3cca/0/0/%2a/p;257075505;0-0;0;43924277;18754-336/150;48159853/48174101/1;;~aopt=2/1/ff/0;~sscs=%3fhttp://fastcompanyconnection.com/http://www.fastcompany.com/embed/90049c4f0b4bbhttp://www.fastcompany.com/embed/6bad939b18b32http://www.fastcompany.com/embed/06a56413c5fc0http://www.fastcompany.com/1842877/office-15-microsofts-best-bet-beating-dropbox-icloud-google-drivehttp://www.fastcompany.com/3000154/surprisingly-smart-strategy-behind-london%E2%80%99s-infamous-olympic-branding8/12/2019 They Write the Right Stuff _ Fast Company
16/19
07/08/12 hey Write the Right Stuff | Fast Company
Page 16ttp://www.fastcompany.com/28121/they-write-right-stuff
5 Years In The Making, Lufthansas Ingenious NewBusiness Class Seats
The 4 Dumbest Rules That Will Kill Your Company'sCulture
Hate Small Talk? These 5 Questions Will Help YouWork Any Room
http://www.fastcompany.com/1843752/hate-small-talk-these-5-questions-will-help-you-work-any-roomhttp://www.fastcompany.com/1844194/4-dumbest-rules-will-kill-your-companys-culturehttp://www.fastcompany.com/1844194/4-dumbest-rules-will-kill-your-companys-culturehttp://www.fastcompany.com/1844346/5-years-making-lufthansa%E2%80%99s-ingenious-new-business-class-seatshttp://www.fastcodesign.com/1670423/5-years-in-the-making-lufthansa-s-ingenious-new-business-class-seatshttp://www.fastcompany.com/1844470/infographic-map-every-big-idea-ever-printed8/12/2019 They Write the Right Stuff _ Fast Company
17/19
8/12/2019 They Write the Right Stuff _ Fast Company
18/19
07/08/12 hey Write the Right Stuff | Fast Company
Page 18ttp://www.fastcompany.com/28121/they-write-right-stuff
Minnesota Courts Cyclists With A New MobilePlatform
masthead
Executive Editor:Noah Robischon
Editorial Director:Tyler Gray
News Editor:Mac Montandon
Editor, Co.Design:Cliff KuangEditor, Co.Exist:Morgan Clendaniel
Editor, Co.Create:Teressa Iezzi
Editor, Co.Lead:Erin Schulte
more
http://www.fastcompany.com/staff-mastheadhttp://www.fastcompany.com/user/erin-schultehttp://www.fastcocreate.com/users/teressa-iezzihttp://www.fastcoexist.com/users/morgan-clendanielhttp://www.fastcodesign.com/users/cliff-kuanghttp://www.fastcompany.com/user/maccabee-montandonhttp://www.fastcompany.com/user/tyler-grayhttp://www.fastcompany.com/user/noah-robischonhttp://www.fastcompany.com/1844218/minnesota-courts-cyclists-new-mobile-platformhttp://www.fastcodesign.com/1670392/minnesota-courts-cyclists-with-a-new-mobile-platformhttp://www.fastcompany.com/1844101/why-apples-next-iphone-important-first-one8/12/2019 They Write the Right Stuff _ Fast Company
19/19
07/08/12 hey Write the Right Stuff | Fast Company
Page 19ttp://www fastcompany com/28121/they-write-right-stuff
dvertise
bout Us
vents
ubscribe
Manage Subscriptions
ewsletters
eprints
rivacy
erms of Service
opyright 2012 Mansueto Ventures LLC. All rights reserved. Fast Company, 7 World Trade Center, New York, NY
0007-2195
Close
http://www.fastcompany.com/28121/they-write-right-stuff#http://www.aboutads.info/http://www.fastcompany.com/terms-and-conditionshttp://www.fastcompany.com/privacy-policyhttp://www.fastcompanyreprints.com/http://www.fastcompany.com/newslettershttp://www.fastcompany.com/renewhttps://magazine.fastcompany.com/loc/FST/Footerhttp://www.fastcompany.com/eventshttp://www.fastcompany.com/about-ushttp://mediakit.fastcompany.com/http://ad.doubleclick.net/click;h=v8/3cca/0/0/%2a/l;255400019;0-0;0;43924277;4307-300/250;46933357/46949660/1;;~aopt=2/1/ff/0;~sscs=%3fhttp://www.fastcocreate.com/Top Related