The Wrong and the Right Way to The Wrong and the Right Way to Integrate Hardware, Software, and Integrate Hardware, Software, and

PeoplePeople

Arthur PysterSenior Vice President and

Director of Systems Engineering and Integration

March 15, 2006

3/15/06 2

TopicsTopics

• SAIC Profile

• Illustrative Examples of Wrong and Right Way to Approach Building and Integrating Systems with Significant Hardware, Software, and People Requirements

– Wrong

• Standard Terminal Automation Replacement System (FAA)

• Delphi Enterprise Resource Management System (DOT)

– Right

• Common Driver Trainer (Army)

• Mobile Inshore Undersea Warfare (Navy)

• Four Strategies and Their Rationale

• Conclusions

3/15/06 3

SAIC ProfileSAIC Profile

O u r V i s i o nBe a leading systems and solutions company, solving our customers’ most important business and mission-critical problems through innovative applications of technology and domain knowledge

”From Science to Solutions™”

O u r S u c c e s s e s More than 3 decades of continuous revenue growth

$7.2 billion in annual revenues for FY05 FORTUNE 500® company - #276 15.5% revenue CAGR over last 5 years

Superb staff of qualified professionals More than 43,000 personnel worldwide 11,000 employees with advanced degrees 20,000 with security clearances

Key positions on programs of national importance Including DoD transformation, border security,

intelligence analysis, cancer research and other national priorities

More than 10,000 active contracts Leading provider of contracted R&D services $3 billion in cash to support growth

Homeland Security Co. of the Year Frost & Sullivan | May 2005

#1 IT Contractor at OSDINPUT | January 2005

#8 Largest DoD ContractorDoD | January 2005

#2 Federal IT Contractor INPUT | April 2005

#2 Top Systems Integrator Federal Computer Week | September 2005

#3 Professional Services Firm FederalTimes.com | June 2005

#3 Technology Contractor Government Executive | August 2005

Leadership – Top IT Outsourcing Vendor In North AmericaMeta Group METAspectrum | January 2004

Leadership – Desktop and Help Desk Outsourcing in North AmericaGartner Magic Quadrant | December 2004

#3 Top Systems Integrator in North America Gartner Dataquest | August 2005

America’s Most Admired Companies in Computer and Data Services FORTUNE | 2001-2005

3/15/06 4

STRATEGIES for Effective Hardware,STRATEGIES for Effective Hardware,Software, and People IntegrationSoftware, and People Integration

Generally, nothing flashy is required. Follow modern systems engineering practices and you will succeed.

Strategy 1. Keep users involved throughout.

Strategy 2. Embrace and manage the business process reengineering that is inevitable when large critical systems are replaced.

Strategy 3. Anticipate the next system you will build and incrementally create a product line architecture so that the next system (and ones beyond it) can be more rapidly constructed and fielded.

Strategy 4. Build a little, test a little, field a little. Do it often.

3/15/06 5

First, the “wrongs”

System developments thatused poor strategies…

3/15/06 6

CUSTOMER

STARS OverviewSTARS Overview

1

The primary automation to control an airplane when it is beyond an airport and lower than 18,000 feet

FACT: 2,000,000 passengers are handled by TRACONS every day

A perfect example of how not to succeed

3/15/06 7

Picture of Early ARTS, the STARS PredecessorPicture of Early ARTS, the STARS Predecessor

3/15/06 8

Context in Which STARS was CreatedContext in Which STARS was Created

• From early 1980s through early 1990s, FAA spent $4B to modernize air traffic control – Advanced Automation System

• One of the most studied examples of disastrous large-scale acquisition, systems engineering, and software engineering

• “Big bang” replacement planned for most legacy systems controlling air traffic – failed because problem was much too complex, too poorly understood, and too volatile

• In 1996, new FAA acquisition management system put in place separate from Federal Acquisition Regulations

• New strategy was set to modernize the National Airspace System (NAS) incrementally with multiple vendors, multiple acquisitions, driven by enterprise architecture

3/15/06 9

STARS Shone Dimly at FirstSTARS Shone Dimly at First

• ARTS system, designed more than 20 years earlier, was being used for low altitude air traffic control across U.S.

• In 1996, Raytheon was awarded $960M contract to replace ARTS.

• STARS was billed as a quick delivery system because it would mostly be COTS with rollout complete in 2004 or 2005.

• Raytheon was to largely reuse system they had deployed for Norway.

• First rollout was to be in Boston in 1998.

• Air traffic controllers would not accept capabilities and demanded extensive customization. FAA management accepted demands without understanding the implications.

• STARS became major development effort, needing hundreds of thousands of lines of new code. Cost escalated to $1.4B. (Now up to $1.7B)

3/15/06 10

User Interface MorassUser Interface Morass

• FAA requirements team, systems engineers, and acquisition team were from different organizations and didn’t come together into an effective integrated team.

• Controllers had no effective way to articulate their requirements.

• FAA did not have a good way to elicit, negotiate, or manage requirements.

• There was little human factors expertise on the FAA STARS team or, more broadly, within the FAA.

• Controllers made demand after demand for features without a good process for either side to understand the consequences.

• Engineers specified a controller interface that was radically different from what controllers wanted and were used to.

3/15/06 11

User Interface ResolutionUser Interface Resolution

• Unlike ARTS, new display had no “shortcut” switches and knobs.

• Everything in STARS was to be controlled through a trackball and keyboard – it felt more like a Windows PC interface.

• Drop-down menus cluttered screen, blocking controller’s view of air traffic.

• Controllers union went to Congress and the press for changes.

• After much finger-pointing between controllers union and FAA, the FAA adopted new approach led by their new Chief Scientist for Human Factors.

• New process included prototyping, impact studies, method to resolve conflicts between controllers and program, ….

• FAA Administrator stayed personally involved in regular STARS meetings for next several years.

3/15/06 12

Picture of Early STARSPicture of Early STARS

3/15/06 13

STARS Backend MorassSTARS Backend Morass

• In addition to radically revising the frontend user interface, the FAA faced numerous new requirements from controllers and technicians for new backend features around improved robustness, maintenance, and performance.

• Major rework of the backend processing system would take years to complete.

• Pressure mounted to get something in the field quickly, but the STARS lifecycle was waterfall development and incremental deployment.

3/15/06 14

Backend RecoveryBackend Recovery

• New lifecycle was defined, enabling incremental capability delivered to field.

• FAA Administrator began to talk about “build a little, test a little, deploy a little.”

• STARS Early Display Configuration (EDC) defined to have old ARTS backend with the new display front-end.

• STARS Full Scale-1 (FS-1) and Full Scale–2 (FS-2) defined incremental backend capability.

• By 2001, STARS EDC was deployed at two low-volume TRACONS El Paso and Syracuse. Recall original roll out for STARS had first deployment at Boston – one of the busiest TRACONS in the country.

3/15/06 15

Meanwhile…Meanwhile…

• Legacy system “ARTS”, that STARS was to replace, kept advancing under a contract with Lockheed Martin.

• Common ARTS began to offer many of the advantages that STARS was to offer, including large color screens.

• Critics begin to question whether the FAA should abandon or cut back on STARS and field more Common ARTS systems.

• Eventually, FAA agreed to a hybrid with some Common ARTS systems and some STARS systems.

• Today, STARS deployment is proceeding, but won’t be completed for several more years.

• Boston’s STARS system was commissioned in May 2004.

3/15/06 16

Picture of Common ARTSPicture of Common ARTS

3/15/06 17

Delphi Enterprise Resource PlanningDelphi Enterprise Resource Planning

• Legacy system “DAFIS” built in 1970s as custom application for Department of Transportation (DOT)

• Dozens of “bolt-on” applications developed over the years to compensate for its shortfalls

• Was recently replaced by new Delphi system that was based on Oracle Federal Financials.

• FAA is by far the dominant part of DOT, having nearly all the complexity because it acquires billions of dollars worth of systems annually and operates a real-time safety-critical system worth more than $25B.

• DOT managed the acquisition of Delphi without a good understanding of FAA complexities. The primary acquisition leadership largely reflected the other parts of DOT that managed grants, investigated accidents, kept statistics, etc. The gulf in perspectives was large.

3/15/06 18

Business Process Reengineering MisunderstoodBusiness Process Reengineering Misunderstood

• DOT insisted FAA reengineer their business processes in order to make effective use of Delphi.

• FAA leadership insisted that DOT didn’t understand the magnitude of the reengineering and that DOT wasn’t willing to fund it.

• FAA line organizations had operated in a fairly decentralized way using DAFIS with lots of significant variation between organizational elements in how financial and contractual functions were performed. FAA line organizations did not want to change their ways.

• DOT was under great pressure from OMB to get Delphi operational quickly.

• FAA opted to minimize business process change in the interests of speedy implementation and because reaching consensus about new business processes was very hard.

• FAA didn’t appreciate they couldn’t avoid much of the business process reengineering anyhow, leading to gross underestimation of the amount of time to convert to Delphi, including data conversion and user training. Symptoms included being behind in paying hundreds of millions of dollars in bills.

3/15/06 19

Next, the “rights”

System developments that followed smart strategies…

CDTSV_0504_051

Common Driver Trainer ArchitectureCommon Driver Trainer Architecture

FACT: 272 casualties in operations Iraqi and Enduring Freedom (through 12/3/05) were caused by private automobile accidents and vehicle crashes—the majority in non-hostile conditions.

Advanced virtual simulators built from a common architecture used to train Army personnel to drive many types of vehicles

A model of how to succeed

3/15/06 21

Combination of words and pictorials(Depiction of the project in operation – how it satisfies the shortfall)

Modular Reconfiguration for Multiple Vehicles, Scenarios, and Training Objectives

Product Line Architecture describes combined software/hardware system

Product line includes instructor operator station, training management system, automated data collection and after action review

3/15/06 22

Mobile Inshore Underwater WarfareMobile Inshore Underwater Warfare

Provide surface and subsurface surveillance in shore areas and provide supporting command and control

FACT: MIUW improve-ment and deployment has been rapid since the attack on the USS Cole in 2000.

3/15/06 23

Mobile Inshore UnderwaterMobile Inshore UnderwaterWarfare InstallationWarfare Installation

3/15/06 24

STRATEGIES for Effective Hardware,STRATEGIES for Effective Hardware,Software, and People IntegrationSoftware, and People Integration

Generally, nothing flashy is required. Follow modern systems engineering practices and you will succeed.

Strategy 1. Keep users involved throughout.

Strategy 2. Embrace and manage the business process reengineering that is inevitable when large critical systems are replaced.

Strategy 3. Anticipate the next system you will build and incrementally create a product line architecture so that the next system (and ones beyond it) can be more rapidly constructed and fielded.

Strategy 4. Build a little, test a little, field a little. Do it often.

3/15/06 25

Strategy 1. Strategy 1. Keep Users InvolvedKeep Users Involved

• STARS failed to systematically involve controllers from the beginning in a way that built up their confidence and really analyzed their needs, how they really operated their equipment. This built up animosity and created an atmosphere that took years to correct.

• STARS failed to involve human factors experts early in the lifecycle and had to recreate the user interface later at great expense and time.

• STARS failed to say no to users who wanted features but didn’t pay for them.

• CDT recognized the gap between SEs who specify a system and the actual users. SAIC specifiers are usually twice the age of military system users. Young soldiers have different mental models and handle many tasks simultaneously. Use SEs with characteristics of real users where possible.

• MIUW recognized that their users are often reservists, who don’t get the same level of training as regular forces. They needed to make it possible to both use and maintain MIUW equipment with minimal training.

• MIUW used some of their own technicians in the design; e.g., technicians suggested that each case has storage room for the cables that go with that case in order to make setup and tear down easier for the user.

• MIUW observed that the resource loaded network is a great communications device with the customer.

3/15/06 26

Strategy 2. Strategy 2. Embrace and Manage ChangeEmbrace and Manage Change

• Delphi demonstrated that putting in place a major new system will be disruptive to existing business processes.

• COTS systems bring constraints on business processes that may clash with old business processes.

• Delphi demonstrated that senior leadership must embrace the change in business process and then manage it.

• Delphi demonstrated that failure to embrace change will lead to underestimating the costs of training, the time for conversion, and the disruption to current business operations.

3/15/06 27

Strategy 3. Strategy 3. Anticipate the Next SystemAnticipate the Next System

• Product line architectures (PLAs) enable speedy creation of individual systems.

• There is little time or $$ to create fully generic PLAs outside real projects. Product line is more a mindset than a toolset – any standard architectural tool can support this.

• Begin with the order for a single product, but anticipate a second product and design an architecture for both.

• Since first product is delivered, focus more attention on ensuring PLA supports it. Support for second product is more notional. Second product proves architectural robustness.

• Keep asking how to build next product while building current one – improve product line architecture incrementally.

• Keep line between HW and SW as fluid as possible.

3/15/06 28

Strategy 4. Strategy 4. Build quickly. Field often.Build quickly. Field often.

• STARS initially tried to get all requirements “correct” and define a big bang delivery. They “backed into” a spiral design with incremental delivery. By backing into that strategy, the program looked weak. Had they embraced it from the beginning, they would have looked smart and been smart.

• CDT assumes that they won’t get things right at the beginning and fields trainers anticipating rapid cycles of improvement. Trial and error and rapid release is especially important for such highly interactive systems as trainers.

• MIUW fielded early prototypes to ensure they really understood the requirements and that the requirements were really right. They also encountered the downside of this approach because they had to keep the prototype in the field longer than expected and it wasn’t ruggedized.

• MIUW used modeling, simulation, quality assurance, and testing early and often to keep quality high. QA and testing are especially important for integration of components from others.

“Impatience is a virtue.” – Dave Pratt

3/15/06 29

In Conclusion…In Conclusion…

The big bang is dead.

The waterfall is dead.

Ignored users kill systems.

Impatience is a virtue.

Failure to anticipate leads to failure.

Keep your users close.

Embrace change.

Anticipate before you are asked.

Build quickly, get more right than wrong, fix it, repeat.

You must…

Because…