Lockheed Martin Aeronautics Company Copyright ©2007 Lockheed Martin Corporation. All Rights...
-
Upload
thomas-hampton -
Category
Documents
-
view
222 -
download
4
Transcript of Lockheed Martin Aeronautics Company Copyright ©2007 Lockheed Martin Corporation. All Rights...
Lockheed Martin Aeronautics Company
Copyright ©2007 Lockheed Martin Corporation.
All Rights Reserved.
Empirical Study of Software Quality and Reliability
Michael F. Siok, PELockheed Martin Aeronautics CompanyP.O. Box 748, MZ 8604Fort Worth, TX 76101Tel: (817) [email protected]
High Assurance Systems Engineering ConferenceHigh Assurance Systems Engineering ConferenceIEEE HASE 2007IEEE HASE 2007
14 November 2007
Jeff Tian, Ph.D., PESouthern Methodist UniversityDallas, TX USA 75275Tel: (214) 768-2861FAX: (817) [email protected]
Lockheed Martin Aeronautics Company 2© 2007 Lockheed Martin Corporation
Avionics Software Development
How much software process is enough ?
How much software is enough ?
Benchmarks ?
In-house Best Practice ?
Cost too Much
Takes too long
Metrics, metrics, metrics
Good Software = f (productivity, reliability, quality)Good Software = f (productivity, reliability, quality)
Lockheed Martin Aeronautics Company 3© 2007 Lockheed Martin Corporation
Aircraft Software at LM Aero . . .
Radar
MissionComputer
Gyros
Accelerometers
Flight Control
Rudder(s)
Electro-Optical
GlobalPositioning
Inertial
Stabilizers
Ailerons
Controls Displays
WeaponsExternalStores
Stick andThrottle
Engine(s)
CharacteristicsCharacteristics
• Large Complex Systems
• Decades Lifespan
• Frequent Software Updates
• Mix of Computation Types
• Computational
• Displays
• Logic/State Machine
• Signal Processing
• Feedback Control
• Hard & Soft Real-Time
• Severe Computing Resource Constraints
• COTS requirements
• Legacy reuse
Lockheed Martin Aeronautics Company 4© 2007 Lockheed Martin Corporation
Aircraft Software at LM Aero . . .
Radar
MissionComputer
Gyros
Accelerometers
Flight Control
Rudder(s)
Electro-Optical
GlobalPositioning
Inertial
Stabilators
Ailerons
Controls Displays
WeaponsExternalStores
Stick andThrottle
Engine(s)
Object AObject AObject AObject A
Object A Object A Object A
Object A
Object A Object A Object A
Object A Object A
Object A Object A
Object A Object A
• Embedded Code size ~2 MSLOC & climbing fast
• Hundreds of object instances
• Some COTS
• Products: F-16, C-130, U-2, F-22, JSF, support equipment, others . . . .
Lockheed Martin Aeronautics Company 5© 2007 Lockheed Martin Corporation
Background (Cont’d)
Avionics Software Managers want to know . . . How are projects performing, individually and collectively? Is OO better than SA/SD? Does programming language make a difference? Productivity, reliability, quality . . . How are we doing? Issues . . .
Measuring success Benchmarking How to improve
Use statistics to provide some answers
Lockheed Martin Aeronautics Company 6© 2007 Lockheed Martin Corporation
Topics
Background
Avionics Software Project Data
Data Analysis
Wrap up and Next Steps
Empirical Study of Embedded Software Quality and ProductivityBy Michael F. Siok and Jeff Tian
IEEE HASE 2007 Paper:
Lockheed Martin Aeronautics Company 7© 2007 Lockheed Martin Corporation
Avionics Software Project Data
39 software projects carefully chosen for study Project Application Domain (a.k.a. project complexity) Project Size Software Development Methodology Programming Language used
Metrics Framework Size Cost Schedule TP & Quality
Project Size OOA/OOD SA/SD Ada C J73
Small
Medium
Large 9 4 9 4
Small
Medium 4 1 3
Large 7 7
Small 4 3 1
Medium 7 5 2
Large 4 4
# of Projects 9 30 9 9 21Total 39 39
Methodology Language
Pro
ject
Co
mp
lexi
ty
Co
mp
ute
Inte
nsi
veD
isp
lay
Inte
nsi
veL
og
icIn
ten
sive
Lockheed Martin Aeronautics Company 8© 2007 Lockheed Martin Corporation
Avionics Software Project Data (Cont’d)
Metrics Framework provides reference to capture metric and scope
27 metrics available from all projects and normalized, where appropriate Comparable
Individually In Aggregate
Lockheed Martin Aeronautics Company 9© 2007 Lockheed Martin Corporation
Data Analysis -- Approach
Study metrics data using Descriptive Statistics Individual Metrics Metrics in aggregate
Perform Hypothesis Tests to answer management questions Domain Separation OO –vs- SA/SD Size –vs- Reliability Size –vs- Productivity Cost –vs- Reliability Cost –vs- Productivity Language –vs- Productivity
Summarize & Report Results
Lockheed Martin Aeronautics Company 10© 2007 Lockheed Martin Corporation
Data Analysis (Cont’d) -- Descriptive Statistics
Used to discover interesting characteristics about each metric in dataset
Lockheed Martin Aeronautics Company 11© 2007 Lockheed Martin Corporation
Data Analysis (Cont’d) -- Hypothesis Tests
Hypothesis Testing Since data not normally distributed, used non-parametric tests
to accept or reject hypotheses H0 – Data compared is from same population
H1 – Data compared is from different populations
Wilcoxon Rank Sum Test
Lockheed Martin Aeronautics Company 12© 2007 Lockheed Martin Corporation
Data Analysis (Cont’d) -- Reporting
Hypothesis Testing (Continued) Conducted on metrics, results collected, summarized
Majority rules policy on Accept/Reject test results Reject = difference at .05 level of significance Accept = no difference
Lockheed Martin Aeronautics Company 13© 2007 Lockheed Martin Corporation
Wrap Up and Next Steps
Avionics software organization needed methodology to Assess project performance Assess project performance relative to other similar projects Identify and act on opportunities for improvement
Software project data was difficult to acquire Data actually very easy to get Projects had to demonstrate selected process controls
Managed variability in metrics Projects had to submit data to company metrics repository and use it Projects had to validate data in company repository
Analysis method fairly simple, straight-forward Descriptive Statistics to study metrics behaviors Hypothesis Testing Summary reporting to capture analysis results for action
Lockheed Martin Aeronautics Company 14© 2007 Lockheed Martin Corporation
Wrap Up and Next Steps (Cont’d)
Lockheed Martin Aeronautics Company 15© 2007 Lockheed Martin Corporation
Wrap Up and Next Steps (Cont’d)
Statistical Testing did not uncover a clear ‘best all around project’ Want to identify well-rounded best-in-class project(s) Project demonstrated best in cost, schedule, performance, and quality
Use Data Envelopment Analysis (DEA) as benchmarking method to identify best in class software projects Non-parametric analysis method Establishes Multivariate production efficiency frontier
Statistical Analysis coupled with DEA will provide repeatable methodology to study & assess company software project data To understand software project and organizational performance To identify best performing software projects To clearly identify practical software process & product improvement
opportunities
10110001010101100010
10110001010101100010
10100010100100010010
10100010100100010010
00101001010111001010
00101001010111001010
11010101010111111001
11010101010111111001
01011111100000101010
01011111100000101010
101010100010111101010100010111
. . . to better the business practice of software development. . . to better the business practice of software development
Lockheed Martin Aeronautics Company 16© 2007 Lockheed Martin Corporation
Points of Contact
Michael F. Siok, PELockheed Martin Aeronautics CompanyP.O. Box 748, MZ 8604Fort Worth, TX 76101Tel: (817) [email protected]
Jeff Tian, Ph.D., PESouthern Methodist UniversityDallas, TX USA 76Tel: (214) 768-2861FAX: (817) [email protected]