AVIATION SAFETY AND SECURITY PROGRAM … ASMM PROJ PLAN v...2 AVIATION SYSTEM MONITORING & MODELING...
72
1 AVIATION SAFETY AND SECURITY PROGRAM (AvSSP) 2.1 Aviation System Monitoring & Modeling (ASMM) Sub-Project Plan February 2004 Version 4.0
Transcript of AVIATION SAFETY AND SECURITY PROGRAM … ASMM PROJ PLAN v...2 AVIATION SYSTEM MONITORING & MODELING...
1
AVIATION SAFETY AND SECURITY PROGRAM
(AvSSP)
2.1 Aviation System Monitoring & Modeling
(ASMM) Sub-Project Plan
February 2004 Version 4.0
2
AVIATION SYSTEM MONITORING & MODELING SUB-PROJECT PLAN SYSTEM SAFETY TECHNOLOGY PROJECT AVIATION SAFETY AND SECURITY PROGRAM
SST Project Approved
Brian E. Smith, Manager System Safety Technology Project
Date
ASMM Sub-Project Submitted
Irving C. Statler Aviation System Monitoring & Modeling Sub-Project Manager
Date
3
ASMM SUB-PROJECT PLAN
1
1.0 INTRODUCTION................................................................................................................................... 7 1.1 PROJECT SUMMARY............................................................................................................................ 7 1.2 CUSTOMER DEFINITION AND ADVOCACY..............................................................................…. 8 2.0 GOALS, OBJECTIVES & WORK BREAKDOWN STRUCTURE...............................................…… 9 2.1 GOALS.........................................................................................................................………………… 9 2.2 OBJECTIVES.........................................................................................................................………….. 10 2.3 WORK BREAKDOWN STRUCTURE .............……………………………………………………… 10 3.0 PROJECT AUTHORITY/MANAGEMENT....................................................................................…... 11 3.1 ORGANIZATION.................................................................................................................................... 11 3.2 RESPONSIBILITIES................................................................................................................................ 11 3.2.1 ASMM sub-projectManagers......................................................................................................................
11
3.2.2 ASMM Element Managers.................................................................................................................... 13 3.2.3 ASMM Boards & Committees............................................................................................................... 13 4.0 TECHNICAL APPROACH.................................................................................................................…. 14 4.1 PROJECT REQUIREMENTS…………………………………………………………………………. 14 4.1.1 Data Analysis Tools Development (ASMM WBS Element 2.1.1) 30 4.1.2 Intramural Monitoring (ASMM WBS Element 2.1.5) 32 4.1.3 Extramural Monitoring (ASMM WBS Element 2.1.2) ......................................................................... 34 4.1.4 Modelling and Simulations (ASMM WBS Element 2.1.3)................................................................... 37 4.2 PROJECT/ELEMENT CAPABILITIES AND PRODUCTS................................................................... 39 4.3 METRICS................................................................................................................................................. 44 4.3.1 Safety Goal Metrics…………………………………………………………………………………... 44 4.3.2 Project Success Metrics………………………………………………………………………………. 44 4.4 IMPLEMENTATION STRATEGY..................................................................................................…... 45 5.0 AGREEMENTS....................................................................................................................................… 47 5.1 NASA........................................................................................................................................................ 47 5.1.1 Other AvSP Projects.............................................................................................................................. 47 5.1.2 Other NASA Programs.......................................................................................................................... 47 5.2 NON-NASA.............................................................................................................................................. 47 5.3 PARTNERS..........................................................................................................................................… 48 6.0 RESOURCES........................................................................................................................................… 49 6.1 FUNDING REQUIREMENT................................................................................................................... 49 6.1.1 Resources Funding Chart by Project/Element/Center........................................................................... 49 6.1.2 Acquisition Strategy Plan....................................................................................................................... 50 6.2 WORKFORCE......................................................................................................................................... 51 6.2.1 Workforce Chart by elements/center..................................................................................................... 51 6.3 FACILITIES USAGE CHARTS.............................................................................................................. 53 6.3.1 Graphic.................................................................................................................................................. 53 6.3.2 Test and Verification of Technology...............................................................................................….. 53 7.0 TECHNOLOGY TRANSFER/COMMERCIALIZATION..........................................................……... 54 8.0 PROJECT ASSURANCE.....................................................................................................................… 54 8.1 DESCOPE METHODOLOGY................................................................................................................. 54 8.2 RISK MITIGATION................................................................................................................................. 55 9.0 REVIEWS………………………………………………………………………………………………. 63 9.1 MONTHLY……………………………………………………………………………………………... 63 9.2 INDEPENDENT INTERNAL REVIEW (IIR)…………………………………………………… 63 9.3 AVIATION SAFETY PROGRAM EXECUTIVE COUNCIL (AVSPEC)……………………………. 63 9.4 AD HOC REVIEWS……………………………………………………………………………………. 63 9.5 OTHER PROJECT MEETINGS……………………………………………………………………….. 64 10.0 TAILORING…………………………………………………………………………………………... 64 11.0 ACCOMPLISHMENTS………………………………………………………………………………. 65
4
FIGURE 1.0 ASMM FOCUS ON PRECURSORS 7
FIGURE 1.2 ASMM COLLABORATIONS/ SAFETY-RELATED CUSTOMERS 8
FIGURE 3.0 ASMM IN AvSP 12
FIGURE 4.1(A) ASMM SUB-PROJECT OVERVIEW 14 FIGURE 4.1(B) ASMM CYCLIC CONCEPT OF PROACTIVE MANAGEMENT OF RISK 15
FIGURE 4.1(C) ASMM SUB-PROJECT ROADMAP 17
FIGURE 4.1(D) AvSP PROGRAM & ASMM SUB-PROJECT& ELEMENT MILESTONES 18
FIGURE 4.1(E) MILESTONES FOR ASMM WBS 2.1.1, DATA ANALYSIS TOOLS DEVELOPMENT 19 FIGURE 4.1(F) MILESTONES FOR ASMM WBS 2.1.2, INTRAMURAL MONITORING 20 FIGURE 4.1(G) ASMM WBS 2.1.3, EXTRAMURAL MONITORING 21
FIGURE 4.1(H) MILESTONES FOR ASMM WBS 2.1.4, MODELING & SIMULATION 22 FIGURE 4.1(I) NASA TECHNOLOGY READINESS LEVELS 23
FIGURE 4.1(J) ASMM IMPLEMENTATION READINESS LEVELS 24
FIGURE 4.2(A) ASMM PRODUCTS, CAPABILITIES, SYSTEM ANALYSIS & BENEFITS 39
FIGURE 4.2 (B) RELATIONSHIPS OF ASMM PRODUCTS TO PROACTIVE MANAGEMENT PROCESS 43
FIGURE 4.4 ASMM IMPLEMENTATION STRATEGY 45
FIGURE 6.12 ASMM SUB-PROJECT RESOURCES 50
LIST OF TABLES
TABLE 4.1(A) MILESTONE CHART ASMM PROJECT 25
TABLE 4.1(B) MILESTONE CHART ASMM WBS ELEMENT 2.1.1, DATA ANALYSIS TOOLS DEVELOPMENT 26
TABLE 4.1(C) MILESTONE CHART ASMM WBS ELEMENT 2.1.5, INTRAMURAL MONITORING 27
TABLE 4.1(D) MILESTONE CHART ASMM WBS ELEMENT 2.1.2, EXTRAMURAL MONITORING 28
TABLE 4.1(E) MILESTONE CHART ASMM WBS ELEMENT 2.1.3, MODELING & SIMULATION 29
TABLE 4.2 ASMM PRODUCTS 41
TABLE 6.1 ASMM SUB-PROJECT FUNDING BREAKDOWN (728-10) 49
TABLE 6.2.1(A) ASMM 728-10 CIVIL SERVANTS WORKFORCE (DIRECT) (FTE) 51
TABLE 6.2.1(B) ASMM 7287-10 PERFORMANCE BASED CONTRACTORS (PBCs) 52
TABLE 6.3.1 ASMM FACILITY REQUIREMENTS 53
TABLE 8.2 TECHNOLOGY/IMPLEMENTATION RISKS AND MITIGATION STRATEGIES 56
LIST OF FIGURES
5
ABBREVIATIONS AND ACRONYMS AATT Advanced Air Transportation Technology Program (includes CTAS, AvSTAR, SMA, etc) AFA Association of Flight Attendants ALPA Airline Pilots Association AM Accident Mitigation AOPA Aircraft Owners & Pilots Association AOS Aviation Operations Systems APA Allied Pilots Association APMS Aircraft Performance Measurement System ARC NASA/Ames Research Center ASAP ASIST Aviation Safety Investment Strategy Team ASMM Aviation System Monitoring & Modeling ASRS Aviation Safety Reporting System ATA Air Transport Association ATC Air Traffic Control ATM Air Traffic Management ATS Air Traffic Services AvSP Aviation Safety Program AvSPEC Aviation Safety Program Executive Council BA British Airways BASIS British Airways Safety Information System BMA British Midland Airways CAA Civil Aviation Authority CAST Commercial Aviation Safety Team CICT Computing, Information, & Communications Technology Program. CICTT CAST/ICAO Common Taxonomy Team CORBA Common Object Request Broker Architecture CVSRF Crew Vehicle Systems Research Facility DFRC NASA/Dryden Flight Research Center DLR German Aerospace Center DOD Department of Defense DOE Department of Energy DOT Department of Transportation DSL Digital Subscriber Line ECS Engineering Complex Systems FAA Federal Aviation Administration FFC Future Flight Central FOQA Flight Operational Quality Assurance FSF Flight Safety Foundation FTE Full-time Equivalent FY Fiscal Year GA General Aviation GAIN Global Aviation Information Network GRC NASA/Glenn Research Center GSA Government Services Administration HAI Helicopter Association International
6
IATA International Air Transport Association ICAC In-Close Approach Change ICAO International Civil Aviation Organization ICASS International Confidential Aviation Safety Systems IDEAS International Data Exchange on Aviation Safety IIR Independent Internal Review IMA International Association of Machinists & Aerospace Workers IRL Implementation Readiness Level KLM Royal Dutch Airlines LaRC NASA/Langley Research Center MIDAS Model-Based Design, Integration, and Analysis Systems MOA Memorandum of Agreement MOU Memorandum of Understanding NBAA National Business Aircraft Association NAOMS NAS Aviation Operational Monitoring System NAR Non-Advocate Review NAS National Aviation System NASA National Aeronautics and Space Administration NATCA National Air Traffic Controllers Association NIAC NAS Information Architecture Committee NLR National Aerospace Laboratory (The Netherlands) NPSS Numerical Processing Simulation System NRA NASA Research Announcement NRL Naval Research Laboratory NTSB National Transportation Safety Board OMG Object Management Group ONERA Office National d'Etudes et de Recherches Aerospatiales (the French National Aerospace
Research Establishment) OSAT Offices of Safety & Assurance Technology PBC Performance-Based Contract PDARS Performance Data Analysis and Reporting System RTCA Radio Technical Commission on Aeronautics SAA Space Act Agreement SAAP Single Aircraft Accident Prevention SAE Society of Automotive Engineers SITA Societe Internationale de Telecommunications Aeronautiques SMA Surface Movement Advisor SST System Safety Technology Project SV Synthetic Vision SWAP System-Wide Accident Prevention TBS To Be Supplied TRACON Terminal Radar Approach Control TRL Technology Readiness Level WAP Weather Accident Prevention WBS Work Breakdown Structure XML Extensible Markup Language
7
1.0 INTRODUCTION This plan provides a description of the NASA Aviation Safety and Security Program’s (AvSSP) Aviation System Monitoring and Modeling (ASMM) Sub-Project of the System Safety Technology (SST) Project. The ASMM Sub-Project focuses on the development of technologies to enable proactive management of safety-risk in the operations of the national aviation system. The purpose of this plan is to present the objectives, technical approach, resources, commercialization, and programmatic risk management for the ASMM Sub-Project. 1.1 SUB-PROJECT OVERVIEW Background There is a need for a comprehensive, accurate, and insightful method for monitoring the operational performance of the National Aviation System (NAS). US aviation policy makers do not currently have any reliable measures of the frequencies or the trends of aviation safety incidents. Technology developers need to know whether new equipment or procedures inserted into the aviation system are producing expected improvements and/or unwanted side effects. The government and the world aviation community continue to routinely amass large quantities of data that could be sources of information relevant to aviation safety. Increasingly, the accumulation of these data outpaces the community’s ability to put them to practical use. It is difficult to combine data related to the same subject when they come from diverse, heterogeneous sources. Often safety data cannot be retrieved after they have been put into computerized storage because of the way that the data were categorized. The ability to monitor continuously, convert the collected data into reliable information, and share that information for collaborative decision making is the basis for a proactive approach to identifying and alleviating life-threatening aviation conditions and events. Similarly, fast-time simulations provide predictions of system-wide effects of proposed interventions. Purpose Aviation System Monitoring and Modeling (ASMM) is one of the sub-projects of the System Safety Technology (SST) Project under the Aviation Safety and Security Program. The other projects are aimed at developing solutions to problems that have been identified as causes of past accidents. ASMM, instead, is primarily concerned with gaining insight to the health and safety of the NAS by providing technologies to facilitate efficient, comprehensive, and accurate analyses of data collected from various sources throughout the NAS during daily normal operations. As shown in Figure 1.1, fatal accidents are only a portion of the data relating to overall aviation safety. The focus of the ASMM Sub-Project is to ensure that precursors of the next accident are reliably identified, assessed, and managed.
FIGURE 1.1 ASMM FOCUS ON PRECURSORS
ASMM Focus
Routine Operations
Incidents
Accidents
Fatal Accidents
Latent Precursors
Overt Precursors
8
Therefore, the aim of the ASMM sub-project is to exploit information technology resources to ¬ provide decision makers in the aviation community with regular, accurate, and insightful
measures of the health, performance, and safety of the NAS, thereby enabling definition of operational trends and identification of developing conditions that could compromise NAS safety, and
¬ provide decision makers with the capabilities for reliable evaluations of the operational significance and causal factors of identified incidents or trends, thereby enabling well designed interventions, and
¬ provide technology and procedure developers with reliable predictions of the system-wide effects of the changes they are introducing into the NAS, thereby,
¬ enabling an industry-wide, and eventually worldwide, proactive approach to identifying and alleviating life-threatening conditions and events,
1.2 CUSTOMER DEFINITION AND ADVOCACY The ASMM approach emphasizes identification of user needs up front. The users, including FAA, NASA, other government agencies, aviation industry, vendors, unions, universities and international organizations, have identified their needs through user-needs studies or focus groups and have been participating in evolving ASMM developmental efforts from the start. This will help with user acceptance and ensure a clear transition path to industry implementation. Project Element
Government Customers
Industry Customers
Academia Customers
International Customers
Data Analysis Tools
FAA: ATS, System Safety, System Capacity, Human Factors, Flight Standards, Tech Center
DOD, DOE, DOT, NTSB
NASA: AATT, AvSTAR, ECS
ATA, AOPA, NBAA, Air carriers, GA and Rotorcraft operators, Data-processing vendors, aircraft manufactures
Unions: ALPA, APA, AFA, IMA, NATCA
Technology developers and human factors researchers
GAIN, CAA, ICAO, IATA, NLR, ONERA, BA, KLM, BMA, ICASS, IDEAS, Euroncontrol
Intramural Monitoring
FAA: System Safety, System capacity
ATA, AOPA, NBAA, Air carriers, GA and Rotorcraft Operators
Unions: ALPA, APA, AFA, IMA, NATCA
Extramural Monitoring
FAA: System Safety ATA, AOPA, NBAA, Air carriers, GA and Rotorcraft Operators
Modeling & Simulations
FAA; Human Factors, Tech Center
DOD, DOT, DOE
NASA: AATT, AvSTAR, ECS
Air carriers, GA and Rotorcraft Operators
Technology developers and human factors researchers
NLR, ONERA, Eurocontrol
FIGURE 1.2
ASMM COLLABORATIONS/ SAFETY-RELATED CUSTOMERS
9
As indicated in Figure 1.2, the ASMM sub-project entails extensive interactions with stakeholder organizations to ensure that the capabilities under development are truly responsive to the needs of the aviation community. Each element in the ASMM sub-project has been coordinated and reviewed with the FAA and Industry. For those elements that have operational personnel sensitivities, additional coordination has taken place with the various relevant unions. Element activities have been designed based on lessons learned from interaction with the aviation community during the Aviation Performance Measuring System (APMS) project started in NASA by the FAA, the Aviation Safety Reporting System (ASRS) managed by NASA for the FAA, the Capacity Programs, and the Aviation Operations Systems (AOS) Project of the Computing, Information, & Communications Technology (CICT) Program. To insure trust, integrity, and continued advocacy, regular meetings, telecons, and national workshops are conducted with target users. Formal agreements are entered into with potential users to test and evaluate new tools and concepts in the operational environment. Agreements are frequently executed with the commercial vendor selected by the customer to assist in the evolutionary development of the capabilities and in the commercialization of those that the customer finds useful. Additionally, for new task implementations, pilot trials are conducted to evaluate and test operational concepts in order to address any user issues. 2.0 GOALS, OBJECTIVES & WORK BREAKDOWN STRUCTURE The basic ASMM concept and purpose is to support proactive management of aviation-safety risk by developing technologies that enable:
¬ efficient and effective feedback - continuously monitoring the aviation system operations - identifying potential safety risks - characterizing the frequency and severity of safety risks - gaining insightful understanding of the data
¬ reliable prediction - evaluating system-wide effects of alternative interventions
¬ easy sharing of information - integrating information derived from diverse, heterogeneous databases - collaborating on decisions and intervention strategies
2.1 GOALS The overall goal of the ASMM sub-project is to enable and support proactive management of safety and risk in the National Aviation System (NAS). ASMM will help provide decision-makers in air carriers, air traffic management, and other air-service providers with regular, accurate, and insightful measures of the health, performance, and safety of the NAS. ASMM outputs will also provide technology and procedure developers with reliable predictions of the system-wide effects of the changes they are introducing into the aviation system. This capability will enable definition of operational and safety trends and the identification of developing conditions that could compromise NAS safety. It will also allow an industry-wide, and eventually a worldwide, proactive approach to identification and alleviation of life-threatening aviation conditions and events.
10
2.2 OBJECTIVES The ASMM Objectives are to develop the technologies to:
a) identify causal factors, accident precursors, and off-nominal conditions in the aviation data,
b) provide health, performance and safety information to decision makers, and c) ensure seamless aviation information services.
Meeting these objectives will represent qualitative measurements of the ASMM system. The goals and objectives of the each ASMM activity element are discussed in Sections 4.1.1 through 4.1.4. The four-fold approach of the ASMM sub-project is to:
¬ Develop tools to extract and display reliable information from large databases with which experts can gain insight into the performance and safety of the NAS and can identify situations that may indicate changes to levels of safety,
¬ Develop methodologies and tools to enable efficient monitoring of the NAS by routinely processing large masses of both anecdotal and quantitative data pertaining to all aspects of the NAS,
¬ Assist and encourage stakeholders in the NAS in the use of these tools for their operational evaluation and continuous evolutionary development, and
¬ Develop fast-time simulations that enable reliable predictions of system-wide effects of proposed technological or procedural changes.
2.2.1 Definitions Throughout this document, we use certain terms that are frequently used in the literature but with various definitions. We specify the following definitions to ensure the understanding of these terms as we use them:
θ The term “precursor” is used to mean the symptom of a systemic problem that is conducive to human error and has the potential to result in an unwanted state or accident if left unresolved. A symptom is a measurable deviation from expectations or normal standards.
θ The term “causal factors” are latent and proximate factors that include: –Conditions necessary for the occurrence of a precursor
AND –Conditions that increase the probability of occurrence of that precursor.
The treatment of the causal factors often entails a re-design, a new procedure, and new training. θ Safety risk assessment is the probability of transitioning to an unwanted or anomalous state
from a safe state after an incident (precursor) occurs. Risk = P (incident) x P (consequence/incident) x P (severity/consequence)
Another consideration that is fundamental to the objectives of the ASMM sub-project is where to look for evidence of the precursors. This notion is conveyed in the Figure 2.1. ASMM is developing the capabilities to efficiently extract information from all of these data sources.
11
FIGURE 2.1 WHERE ARE PRECURSORS TO BE FOUND?
2.3 WORK BREAKDOWN STRUCTURE ASMM currently consists of four elements that emphasize development and application of information technology research to capitalize on aviation data:
¬ Data Analysis Tools Development (ASMM WBS Element 2.1.1)
¬ Intramural Monitoring (ASMM WBS Element 2.1.5) ¬ Extramural Monitoring (ASMM WBS Element 2.1.2) ¬ Modeling & Simulations (ASMM WBS Element 2.1.3)
For the first two years of the AvSSP Program, ASMM also had an element 2.1.4 called Information Sharing that had been designed to address the need for a reliable and secure infrastructure for sharing information both intramurally and extramurally. During the re-scoping study at the end of FY’01, this element was eliminated and the activities in support of intramural and extramural monitoring were moved to those respective elements. This Sub-Project Plan for ASMM does not address Information Sharing per se, because all of the work completed under the Information Sharing element during its two years was entirely related to the other elements of ASMM. Therefore, the previous and future activities relevant to information sharing are described within the supported elements. ASMM will merge these activity elements and their products into a system-wide framework enabling aviation safety-risk management by aviation policy makers whether they are in government or industry, while respecting the proprietary rights to some sources of data and sensitivities to potential misuse should they be released outside the owning organization.
Accident Data Operational Data Operational Surveys Anecdotal Reports Projections
Forensics (post-hoc analysis & deep treatment)
Diagnostics (epidemiology
& trending)
SUBJ
ECTI
VE
<- d
ata
cont
inuu
m ->
OBJ
ECTI
VE PA
ST PRESENT FU
TUR
E
Prognostics (predictions & expert opinion)
12
3.0 PROJECT AUTHORITY/MANAGEMENT 3.1 ORGANIZATION The Aviation Safety and Security Program (AvSSP) Level 1 Director is George Finelli. Brian E. Smith is the Manager of the System Safety Technology Project at the Ames Research Center (ARC). The Executive Board of the Aero-Space Technology Enterprise is responsible for the oversight of the project. The project authority is derived from the approval of the AvSSP Program from PMC. Any changes to that approval must go through the AvSSP Program Office to the Enterprise Executive Board for final approval. As shown in Figure 3.0, the ASMM sub-project is one of the projects under System Safety Technology of the Aviation Safety and Security Program. Lead program personnel for ASMM reside at ARC. The ASMM Sub-Project Manager is Irving C. Statler. The ASMM Element managers are as follows:
• Irving C. Statler and Michael G. Shafto (WBS 2.1.1: Data Analysis Tools) • Thomas R. Chidester (WBS 2.1.5: Intramural Monitoring) • Linda J. Connell and Mary M. Connors (WBS 2.1.2: Extramural Monitoring) • Irving C. Statler (WBS 2.1.3, Modeling & Simulations)
3.2 RESPONSIBILITIES 3.2.1 ASMM sub-project Managers The ASMM Sub-Project Manager is responsible for implementation of this AvSP project with full authority to manage the project within the defined objectives, technical scope, schedules, and resources. The ASMM Sub-Project Manager reports to the System Safety Technology Manager. Specific responsibilities include:
• Defining and implementing the technical project within the technical, cost, and schedule constraints established by the program plan
• Executing project control, with authority to reprogram element resources across Centers as necessary to address technical, schedule, and resource priorities, but not to exceed the smaller of $750K or 15% of a given year’s budget and providing that Project or Program level milestones are not adversely affected.
• Management of all resources (facilities, workforce, and funding) required to meet the milestones identified for the project
• Providing advice and recommendations for changes to the Program Plan to the AvSP Program Director, and implementing changes upon approval
• Preparing periodic element reports, annual AvSP Office reviews, and other reviews as required
• Acting as primary interface with outside customers and partners to ensure effective technical direction and implementation of the project elements
• Representing technical plans, objectives, approaches, and progress to NASA/Headquarters management, other government agencies, interagency coordinating committees, technology committees, and working and steering groups
• Maintaining cognizance of related program activities (including NASA base and focused programs, as well as FAA, industry, and international efforts) and periodically reporting on their status and relevance
13
1.2
Prog
ram
Inte
grat
ion
Mic
hael
Bas
ehor
e, F
AA
Li
aiso
n Ca
rrie
Wal
ker,
HQ
PROGRAM
2.1
Avi
atio
n Sy
stem
M
onito
ring
&
Mod
elin
g Irv
ing
Stat
ler
(AR
C)
2.2
Syst
em-W
ide
Acc
iden
t Pr
even
tion
Bet
tina
Bea
rd
(AR
C)
2.3
Sing
le A
ircra
ft A
ccid
ent
Prev
entio
n C
arrie
Wal
ker
(LaR
C)
2.4
Wea
ther
A
ccid
ent
Prev
entio
n G
us M
artz
aklis
(G
RC
)
2.5
Acc
iden
t M
itiga
tion
Rob
ert
McK
nigh
t (G
RC
)
2.6
Synt
hetic
V
isio
n D
anie
l Bai
ze
(LaR
C)
2.7
Air
craf
t Ici
ng
Mar
y W
adel
(G
RC
)
1.1
Tech
nica
l Int
egra
tion
Fran
k Jo
nes
(LaR
C)
1.3
Veh
icle
Saf
ety
Tech
nolo
gy
1.4
Wea
ther
Saf
ety
Tech
nolo
gy
1.5
Syst
ems S
afet
y Te
chno
logy
1.2
Prog
ram
Inte
grat
ion
Mic
hael
Bas
ehor
e, F
AA
Lia
ison
Ca
rrie
Wal
ker,
HQ
Avi
atio
n Sa
fety
Pro
gram
Offi
ce
Geo
rge
Fine
lli, D
irect
or
John
Whi
te, D
eput
y D
irect
or
Conn
ie S
mith
-Buf
fin, S
ecre
tary
Dou
glas
Roh
n, D
ep P
rog
Mgr
(GRC
) Br
ian
Smith
, Dep
Pro
g M
gr (A
RC)
Virg
inia
Mar
ks, S
enio
r Pr
og
Ana
lyst
2.8
Sear
ch a
nd
Res
cue
Dav
id A
ffen
s (G
SFC
)
PROJECTS
FIG
URE
3.0
A
SMM
IN A
vSP
14
3.2.2 ASMM Element Managers The ASMM Element Managers are responsible for implementing the ASMM Sub-Project within each ASMM element with full authority to manage within the defined objectives, technical scope, schedules, and resources. Element Managers report to the ASMM Sub-Project Manager. Specific responsibilities include:
• Defining and implementing technical activities within the technical, cost, and schedule constraints established by this plan and their respective element plans
• Authority to reprogram element funding resources up to 15% of guideline across sub-element activities within their element as necessary, to address technical, schedule, and resource metrics.
• Ensuring technical integration is implemented across all sub- elements • Providing advice and recommendations for changes to the respective element
plans to the ASMM Element Manager, and implementing changes upon approval • Preparing monthly project reports, technical highlights, annual ASMM Sub-
Project reviews, and other reviews as required • Representing technical plans, objectives, approaches, and progress to AvSP
management, other government agencies, interagency coordinating committees, technology committees, and working and steering groups
• Maintaining cognizance of related program activities (including NASA base and focused programs, as well as FAA, industry, and international efforts) and periodically reporting on their status and relevance.
• Ensuring that a Risk Management Process is in place for the element 3.3 CONTROLS The Sub-Project Manager exercises control through the Configuration Management process, which records changes and maintains the current status of programmatic baseline documentation. Changes to the controlled documents will follow a configuration management process to assess the impacts and approve proposed changes, notify all affected parties, and verify/update designated documents. The Sub-Project Plan will be updated as required to maintain compatibility between the plan and changes in resource availability. A monthly report by the Sub-Project Manager to the Project and Program Managers will be developed. This report is an integrated assessment of technical, cost, and schedule progress versus plan and will contain significant technical highlights. Issues and/or concerns (including potential impact and proposed action) and any major interactions with partners will be identified. Additionally, the Sub-Project Manager will support Center Program Management Council meetings as required by the Program Manager or Center management. 3.4 DATA MANAGEMENT The Sub-Project Manager is responsible for the protection of the information generated within the Sub-Project and will take appropriate actions to protect it depending on its sensitivity. 3.4 LOGISTICS This technology development Project does not provide mission, flight, or systems hardware intended for long-duration use and as such is exempt from logistics management. THIS PROJECT IS IN FULL COMPLIANCE WITH NPG 7120.5B.
15
4.0 TECHNICAL APPROACH 4.1 SUB-PROJECT REQUIREMENTS The work of ASMM exploits information technology to address the problem of monitoring the aviation system by
¬ Developing the tools and methodologies for a strategy of dual complementary capabilities of “bottom-up” and “top-down” monitoring to obtain feedback from aviation data,
¬ Developing the infrastructure capabilities for sharing information specific to the developed tools and methodologies, and
¬ Utilizing the collected textual and quantitative data to support the development and validation of system-wide models and simulations for predictions and safety risk assessments.
These goals will be realized by the work to be accomplished under the four elements of the ASMM WBS; 2.1.1: Data Analysis Tools Development, 2.1.5: Intramural Monitoring, 2.1.2: Extramural Monitoring, and 2.1.3: Modeling and Simulations as shown in Figure 4.1 (A) below.
FIGURE 4.1(A)
ASMM SUB-PROJECT OVERVIEW
Enable proactive management of Aviation Safety
Identify Causal Factors, Accident Precursors, and
Off-nominal Conditions in the
Aviation data
Ensure Seamless Aviation Information
Services
Provide Health, Performance & Safety
Information to Decision Makers
Data Sensitivity Disparate Data Sources
Real-time Monitoring Capability
Goal
Objectives
Challenges
Approach
ELEMENTS
PRODUCTS
Develop Tools to Extract Reliable Information from Large Databases
Enable Efficient Routine Monitoring
Of the NAS
Simulations for Predictive and
Assessment Capabilities
ASMM WBS
Element
2.1.1
Data Analysis
Tools
Development
ASMM WBS Element
2.1.5
Intramural
Monitoring
• Data Analysis Tools
• APMS Tools • PDARS
• NAOMS • Incident-Report Enhancements
Risks
• Fast-time Simulation of System-Wide
• Air carriers • ATC
• Incident Reporting • NAOMS
• Modeling • Simulations
• Database Mining
• Database Linkage
• Causal Analysis
Collaborate with Users to Test, Evaluate, and
Evolve tools to Monitor the NAS
ASMM WBS Element
2.1.2
Extramural Monitoring
ASMM WBS Element
2.1.3
Modeling & Simulations
ASMM WBS Element
2.1.4
Information Sharing
Element 2.1.4-Information Sharing was de-scoped in FY’01.
16
The ASMM sub-project is developing the technologies to enable the industry to move from solely reactive management to proactive management of safety risk, i.e., minimize risk by learning continuously from normal operational experience, and identifying and responding to precursor events. The cyclic concept of proactive management of safety risk, illustrated in Figure 4.1(B), entails the following four primary steps that relate directly to the concept as portrayed in Figure 4.4:
• IDENTIFYING – monitoring the system performance continuously and comparing with established standards to identify potential risks.
• EVALUATING - diagnosing the causal factors, estimating the likelihood of future occurrences, and assessing the severity and possible impacts
• FORMULATING – proposing changes, assessing the safety risk, estimating benefits and costs, and developing a strategy for implementing a change
• IMPLEMENTING – implementing on a small scale, evaluating the intervention, refining, establishing performance standards, and monitoring to assess the efficacy of the intervention and to identify unwanted side effects.
Throughout this process, decisions must be made by aviation-domain experts in the industry, who set the performance standards, gain insight from monitoring, propose the changes, and develop the intervention strategies. ASMM cannot replace this expertise with automation. However, ASMM can provide the methodologies, the computational tools, and the infrastructure to assist the experts in making the best possible decisions.
FIGURE 4.1(B)
ASMM CYCLIC CONCEPT OF PROACTIVE MANAGEMENT OF RISK
INTERVENTION STRATEGY - Design
- Operations
- Investment IMPLEMENTING
Other factors
Expert Opinion
Feedbac
k
Feedback
CONSIDER CHANGE - Technology - Training - Procedure FORMULATING
Models & Simulation Models & Simulation
Predictio
n
Prediction Safety-Risk Assessment
Safety-Risk Assessment
Estimate - Benefits - Costs
Estimate - Benefits - Costs
EVALUATING Diagnose - statistical analysis - causal analysis
Diagnose - statistical analysis - causal analysis
Characterize - Diagnose for frequency & severity
Characterize - Diagnose for frequency & severity
Heterogeneous Data Sources
IDENTIFYING
Monitor, codify, classify, & merge Monitor, codify, classify, & merge
Identify failure modes Identify failure modes
Convert to Information Convert to Information
GAIN
INSIGHT SET
PERFORMANCE
STANDARDS
Compare Compare
17
The Data Analysis Tools Development element provides for the research that will result in software to perform tasks that presently can only be performed by experts with much time and effort. We will develop capabilities to process textual and numeric aviation data, and recognize relevant information in diverse databases, including those derived from the activities under Intramural and Extramural Monitoring. We will develop tools to convert these data into displays of meaningful information to help analysts achieve the insight needed to understand the circumstances and propose mitigating actions. The Modeling and Simulations element of ASMM addresses the need to support predictions and safety risk assessments by developing and validating system-wide models and simulations. The data collected from all of the activities under Intramural and Extramural Monitoring will be used to support the development and validation of models of the National Aviation System. Our approach to monitoring entails a dual strategy of complementary monitoring capabilities for feedback. Two distinct, independent but complementary aviation-monitoring capabilities will be created. The Intramural Monitoring element is intended to provide the air-service operators with the tools needed to monitor their own performance and safety continuously, effectively, and economically within their own organizations. The Extramural Monitoring element complements Intramural Monitoring and is a comprehensive system-wide survey mechanism for monitoring the performance and safety of the overall National Aviation System and for detecting and evaluating the effects of new technologies as they are inserted into the system. While helping air services organizations build their intramural capability for safety monitoring and for establishing a potential database for the Aviation Safety Program, we will develop a comprehensive survey system for monitoring safety performance on a national scale. The concepts and capabilities of the two approaches (i.e., top-down extramural monitoring and bottom-up intramural monitoring) will evolve independently in parallel. However, information derived from each will complement the other elements of ASMM in the process of identifying precursors, monitoring the effects of changes, and developing predictive capability. The four sub-elements of ASMM are interdependent and interrelated. They have been planned and are being worked in concert as evidenced in the ASMM sub-project and Element Milestones shown in Figure 4.1 (C), in the Element Roadmaps shown in Figures 4.1 (D) – (G), and in the descriptions of the Elements that follow in Sections 4.1.1 through 4.1.4. Figure 4.1(C) shows the ASMM sub-project and Elements Milestones for each year of the project. The ASMM sub-project milestones are shown with triangles and the ASMM Element milestones are shown with diamonds for each of four sub-elements. Completed milestones are solid colors. Figures 4.1(D) through 4.1(G) show the corresponding underlying roadmaps for each of the four ASMM elements. The descriptions of the indicated milestones and their exit criteria are presented in Tables 4.1 (A) through 4.1 (E). All elements of the ASMM sub-project are worked in close collaboration with industry to address continuously the challenges of the sensitivity to potential misuse of data, integration of information from disparate data sources, and the pilots’ concerns related to real-time monitoring of their performance. These hurdles are common and controlling to a wide range of similar problems in other industries. ASMM team members are part of national and international aviation activities that are addressing these particular problems faced in data management. As these data-management hurdles are resolved, they will also minimize key hurdles in system-wide monitoring of aviation safety information. This will simplify element implementations for key activities such as APMS, PDARS, and NAOMS.
18
1 0
8
Prog
ram
2000
20
01
2002
20
03
2004
1
2 3
4 1
2 3
4 1
3
4
2 3
4 1
2 3
4 2
1
1 Prel
imin
ary
Inte
grat
ed
Prog
ram
Ass
essm
ent
2 Sa
fety
-Impr
ovem
ent
Con
cept
s D
efin
ed
3 In
teri
m In
tegr
ated
Pr
ogra
m A
sses
smen
t
Sim
ulat
ion
& F
light
Tes
t Eva
luat
ions
of S
afet
y Im
prov
emen
t Sys
tem
s w
ithin
AvS
P C
ompl
ete
5
Inte
grat
ed F
ull-M
issi
on A
pps.
Si
mul
atio
ns &
Val
idat
ion
6 In
tegr
ated
Pr
ogra
m A
sses
smen
t
2.1.
1 Da
ta
Anal
ysis
Too
ls
Deve
lopm
ent
2.1.
3 M
odel
ing
& Si
mul
atio
ns
2.1.
2 Ex
tram
ural
M
onito
ring
2.1
Avia
tion
Sys.
Mon
itorin
g &
Mod
elin
g
2005
3
4 1
2
Fast
-Tim
e Si
m
of S
yste
m-W
ide
Risk
s
Appl
y AP
MS
to A
TC
O
pera
tiona
l Tes
t of R
isk
Asse
ssm
ent A
id
Sy
s W
ide
Risk
Ass
essm
ent b
ased
on
Mer
ged
FOQ
A Da
ta
M
erge
d FO
QA
Data
w
ith P
DARS
Dat
a
- Lev
el I
Mile
ston
e - L
evel
II M
ilest
one
- Lev
el II
I Mile
ston
e
- Lev
el II
Mile
ston
e R
oll-
up
Blu
e –
Rol
l-up
to L
evel
I #2
R
ed –
Rol
l-up
to L
evel
I #4
G
reen
– R
oll-u
p to
Lev
el I
#5
#
1
2
4
7 8
7
3
11
Caus
al
Anal
ysis
o
f Inc
iden
ts
Ope
ratio
nal T
est o
f R
isk
Asse
ssm
ent
Risk
Ass
essm
ent T
ool
Usin
g M
erge
d D
ata
11
De
mo
Valu
e of
Pilo
t Su
rvey
Impl
emen
t Fl
ight
C
rew
Sur
vey
Esta
blis
h NA
OM
S W
orki
ng G
roup
Dem
onst
rate
Use
of
NAO
MS
for R
isk
Asse
ssm
ent
Dem
o Im
p. M
etho
ds fo
r Co
st-e
ffect
ive
Surv
eys
3
8 G
A Pi
lot
Surv
ey
4
4 4
4
Mod
els
Bas
ed o
n M
erge
d D
ata
Dem
onst
rate
d Sy
stem
-wid
e Si
mul
atio
n D
emon
stra
ted
Pred
ictiv
e Ca
pabi
lity
Verif
ied
Pred
ictiv
e of
Sys
tem
-wid
e Ef
fect
s of
Cha
nges
Val
idat
ed
3
3 G
A Pi
lot S
urve
y
2.1.
5 In
tram
ural
M
onito
ring
2
1 2
Dem
o M
erge
from
Te
xt &
Dig
ital D
ata
Est M
etho
dolo
gy fo
r Su
rvey
of A
TC
8
8
12
Risk
Ass
ess
of S
/W
Effe
cts
of
Chan
ges
Valid
ated
1 Ap
ply
APM
S to
AT
C O
pera
tiona
l Tes
t of R
isk
Asse
ssm
ent
Mer
ged
FOQ
A Da
ta
Iden
tify
Vend
ors
to
Com
mer
cial
ize
APM
S To
ols
7 M
erge
d FO
QA
Data
fr
om M
ul/ A
/C
8 Tr
ansf
er A
PMS
Tool
s to
Ve
ndor
s of
FO
QA
Prog
ram
s
8
PDAR
S Pr
oven
at
Ope
ratio
nal S
ite
1
2 2
7
Dem
o Li
nk o
f Sim
s to
Au
to R
isk
Asse
ssm
ent
12
10
M
erge
d FO
QA
Data
w
ith P
DARS
Dat
a
PDAR
S Pr
oven
at
Ope
ratio
nal S
ite
Risk
Ass
ess
of S
/W
Effe
cts
of C
hang
es
Valid
ated
Dem
o Va
lue
of P
ilot S
urve
y
1 1 1 2
4 8
FY
QU
AR
TER
FIG
URE
4.1
(C)
AvS
P PR
OG
RAM
& A
SMM
SU
B-PR
OJE
CT&
ELE
MEN
T M
ILES
TON
ES
10
11
12
12
8
19
FIG
URE
4.1
(D)
MIL
ESTO
NES
FO
R A
SMM
WBS
2.1
.1, D
ATA
AN
ALY
SIS
TOO
LS D
EVEL
OPM
ENT
Data
An
aly
sis
To
ols
2005
3.0
61
3.4
22
1.5
93
0.9
95
1.1
00
0.4
50
2001
2002
2003
2004
FY
1998
1999
2000
2.1
.1P
re
-AvS
PA
vS
P P
has
e 1
Extra
mu
ral
Mo
nit
orin
g
Intra
mu
ral
Mo
nit
orin
g
Taxonom
y
Develo
pm
ent
Analy
st
Advis
or
Machin
e
Com
pre
hensio
n
Data
base
Lin
kage
Data
Min
ing
Casual
Analy
ses
Ris
k
Assessm
ent
Test
&
Evalu
atio
n
by
Co
lla
bo
ra
to
rs
UA
LU
AL
AA
AA
AA
AA
AA
Ala
ska
Ala
ska
Ala
ska
Ala
ska
Ala
ska
Ala
ska
Com
merc
ial
Imple
menta
tion
Mo
deling
&
Sim
ula
tio
n
Pre
-AV
SP
mn
Pro
ject M
ilesto
ne
Ele
ment M
ilesto
ne
PD
AR
S-F
AA
ATC
12
45
7
1.2
9 1.1
1
APM
S-
Flig
ht D
ata
1.7
20
FI
GU
RE 4
.1(E
) M
ILES
TON
ES F
OR
ASM
M W
BS 2
.1.5
, IN
TRA
MU
RAL
MO
NIT
ORI
NG
2.1
.5P
re
-AvS
PA
vS
P P
has
e 1 2003
2004
FY
1998
1999
2000
2005
2.7
50
2.6
57
2.5
38
1.0
26
2001
2002
Intr
am
ural
Mo
nit
orin
gD
ata
An
aly
sis
To
ols
Ex
tra
mu
ra
l
Mo
nit
orin
g
UA
LU
AL
AA
AA
AA
AA
AA
Ala
ska
Ala
ska
Ala
ska
Ala
ska
Ala
ska
Ala
ska
Com
merc
ial
Imple
menta
tion
Te
st
& E
valu
ati
on
by C
ollab
orato
rs
in
Op
erati
on
al En
vir
on
me
nt
Mo
deling
&
Sim
ula
tio
n
Valid
atio
n
Flig
ht
Opera
tions
Engin
eering
Main
tenance
Exte
nd
APM
S t
o A
TC P
erf
orm
ance
Merg
e A
TC &
Flight
Info
Merg
e w
ith o
ther
data
Sourc
es f
or
risk a
ssesm
ent
mn
Pro
jectt M
ilesto
ne
Ele
ment M
ilesto
ne
12 3
57
17 8
AP
MS
PD
AR
S
FA
A A
TS &
NA
TCA
9 10
8
11
.2
21
FI
GU
RE 4
.1(F
) M
ILES
TON
ES F
OR
ASM
M W
BS 2
.1.2
, EX
TRA
MU
RAL
MO
NIT
ORI
NG
2.1
.2P
re
-AvS
PA
vS
P P
has
e 1
2001
2002
2003
FY
1998
1999
2004
2005
0.9
75
1.6
00
1.7
39
1.8
52
2000
1.9
97
0.9
00
Sys
tem
-
wid
e
Incid
en
t
Mo
nit
orin
g
To
ols
NA
S
Op
erati
on
al
Mo
nit
orin
g
Sta
te-o
f th
e-a
rt
Data
base
Surv
ey D
esig
n &
Focused T
rail
Ele
ctr
onic
Report
Subm
issio
n
Auto
mate
d A
naly
st
Advis
or
Imple
ment
for
Tra
nsport
Flight
Cre
ws
Im
ple
ment
fo
r G
A p
ilots
Com
merc
ial
Imple
menta
tion
of
NA
OM
S
Data
An
aly
sis
To
ols
Intr
am
ural
Mo
nit
orin
g
Pre
-AvS
P
Sce
nario
s f
or M
od
elin
g &
Sim
ula
tio
ns
mn
Pro
ject M
ilesto
ne
Ele
mentM
ilesto
ne
12
36
1.3
1.1
0
58
9
10
11
Meth
odolo
gy f
or
air
traff
ic contr
ollers
1.1
1
22
FI
GU
RE 4
.1(G
) M
ILES
TON
ES F
OR
ASM
M W
BS 2
.1.3
, MO
DEL
ING
& S
IMU
LATI
ON
Sim
ula
tio
ns
Mo
de
ls
2005
0.7
58
0.8
72
1.1
08
1.3
76
1.4
31
0.5
00
2001
2002
2003
2004
FY
1998
1999
2000
2.1
.3P
re
-AvS
PA
vS
P P
has
e 1
Data
An
aly
sis
To
ols
mn
Pro
ject M
ilesto
ne
Ele
ment M
ilesto
ne
Intr
am
ural
Mo
nit
orin
g
Extr
am
ural
Mo
nit
orin
g
Model Capabilit
ies A
ssessm
ent
Identi
fy Requirem
ents
Identi
fy &
In
tegra
te
Hum
an B
ehavio
r M
odels
Develo
p Syste
m-w
ide
Models
Valid
ate
Sim
ula
tion
Pre
dic
tions
Verify
Sim
ula
tion
Pre
dic
tions
Conduct
Sim
ula
tion S
tudie
s
Develo
p S
yste
m-w
ide
Sim
ula
tions
Merg
e w
ith o
ther
data
Sourc
es f
or
risk a
ssessm
ent
Pre
-AvSP#
1Pre
-AV
SP#
2
Validate
Ris
k A
ssessm
ent
12
34
57
41
1
98
23
The ASMM sub-project milestones have performance metrics that specify Technology Readiness Level (TRL) as part of the exit criteria. Figure 4.1(I) below shows the NASA Technology Readiness Levels. Another part of the exit criteria is a validation of project objectives and requirements. TRL and Implementation Readiness Level (IRL) vary according to type of product and the degree to which research is developed in conjunction with industry partners. Industry participation will determine the level of technology development. Desired implementation level of the product will depend on many scheduling, cost, political, and management factors.
System Test and Operations
System/Subsystem Development
Technology Demonstration
Technology Development
Research to Prove Feasibility
Basic Technology Research
Level 9
Level 8
Level 7
Level 6
Level 4
Level 5
Level 3
Level 2
Level 1
Actual system “operationally proven” at operational site
Analytical and experimental critical function, or characteristic proof-of-concept
Technology concept and/or application formulated (candidate selected) Basic principles observed and reported
Components or Integrated Components verified in a relevant environment
Actual system completed and “operationally qualified” through test and demonstration System prototype demonstrated in operational environment
System/subsystem model or field demonstrated/ validated in a relevant environment
Component or Integrated Components tested in a laboratory environment
Con
cept
Exp
lora
tion
Con
cept
Dev
elop
men
t
Pro
toty
pe D
evel
opm
ent
Full
Sca
le D
evel
opm
ent
FIGURE 4.1(I) NASA TECHNOLOGY READINESS LEVELS
Implementation of ASMM products differs significantly from typical engineering products because they are seldom defined in terms of certification criteria. The definitions shown in Figure 4.1(J) are meaningful IRL levels to assign to the ASMM milestones. These nine levels approximately correspond to the definitions of IRL established for the AvSP program.
24
FIGURE 4.1(J) ASMM IMPLEMENTATION READINESS LEVELS
Table 4.1 (A) below presents the ASMM sub-project milestones and exit criteria. These milestones roll up into AVSP Program milestones as shown in the right-most column. Tables 4.1(B)–(E) presents the ASMM Element Milestones. Program-wide technology readiness levels are shown in Tables 4.1(A) through 4.1(E). Sections 4.1.1 through 4.1.4 define the goals, objectives, approach & milestones/TRLs for each of the four ASMM WBS Elements.
9
8
7
6
5
4
3
2
1
IRL
Operation of Certified System - Recommendations, guidelines, technology concept, etc. required and evaluated by government regulation
Certification Approved - Recommendations, guidelines, technology concept required by government regulation
Certification Standard Established - Recommendations, guidelines, technology concept incorporated into industry and/or government advisory document
Draft Certification Standard Developed - Standards incorporating recommendations, guidelines or technology concept drafted into industry/or government advisory document
RTCA/SAE or Equivalent Convened - Presented to Industry group for consideration in standards industry implementation
Application for Certification - Results/products publicly presented and available for industry implementation
Commercial Product Development Initiated - Product ready for hand off for commercial or customer refinement
Industry R&D Funding Committed - Industry partner has committed funds and/or staffing resources to research
Technology Transfer Initiated - Results presented to industry and research partners
25
TABL
E 4.
1 (A
) SU
B-PR
OJE
CT
MIL
ESTO
NE
CH
AR
T
2.1
ASM
M S
UB-
PRO
JEC
T N
o.
Mile
ston
es
Exit
crite
ria
TRL/
IRL
Dat
e M
o/Y
r A
vSP
Rol
l-up
(Mo/
Yr)
PRO
DU
CT
2.1/
1 A
pply
Air
craf
t Per
form
ance
Mon
itori
ng
Syst
em (A
PMS)
to A
ir T
raffi
c Con
trol
(A
TC):
Dem
onst
rate
app
licat
ion
of A
PMS
conc
epts
and
met
hodo
logi
es to
ATC
for
perfo
rman
ce m
onito
ring.
Acc
epta
nce
by th
e A
TC c
omm
unity
an
d ex
tens
ion
to o
ther
sect
ors o
f na
tiona
l ATC
. At l
east
6 fa
cilit
ies a
re
activ
ely
parti
cipa
ting
in e
valu
atio
n, a
nd
plan
s for
nat
iona
lizat
ion
are
mad
e.
4/1
04/0
0 C
omp
Apr
00
#2 (0
9/01
)
PDA
RS
TOO
LS
2.1/
2 O
pera
tiona
l Tes
t of R
isk A
sses
smen
t A
id: D
emon
stra
te, i
n op
erat
iona
l en
viro
nmen
t, to
ols f
or m
ergi
ng
hete
roge
neou
s dat
abas
es to
aid
cau
sal
anal
ysis
and
risk
ass
essm
ent
Use
r con
curr
ence
on
the
pote
ntia
l for
re
liabl
e an
d va
luab
le a
utom
ated
as
sist
ance
for r
isk
asse
ssm
ent.
At l
east
tw
o ai
r ser
vice
pro
vide
rs a
re p
rovi
ding
da
ta fo
r dev
elop
ing
5/3
09/0
2 C
omp
Sep
02
#4 (0
3/03
)
A
PMS
TOO
LS
2.1/
3 G
A P
ilot S
urve
y: N
atio
nal A
viat
ion
Syste
m A
viat
ion
Ope
ratio
nal M
onito
ring
Syste
m (N
AO
MS)
add
s the
GA
pilo
t co
mm
unity
to th
e su
rvey
syst
em.
Fiel
d tri
als c
ompl
eted
and
GA
surv
ey
laun
ched
with
at l
east
800
inte
rvie
ws
com
plet
ed.
3/1
09/0
2 C
omp
Sep
02
#5 (0
6/04
)
NA
OM
S 2.
1/4
Fast
-Tim
e Sim
ulat
ion
of S
yste
m-W
ide
Risk
s: P
redi
ctio
ns o
f sys
tem
-wid
e ef
fect
s us
ing
the
linke
d hu
man
per
form
ance
/air
traff
ic m
odel
s are
val
idat
ed a
gain
st d
ata
colle
cted
on
in-c
lose
app
roac
h m
aneu
vers
to
LA
X.
Val
idat
ion
base
d on
airc
raft
resp
onse
to re
ques
t, ai
rcra
ft pe
rform
ance
to
thre
shol
d, a
nd fl
ight
cre
w p
roce
dure
s.
A m
easu
re o
f the
pre
dict
ive
accu
racy
(i.
e., c
orre
latio
n be
twee
n st
atis
tics
prod
uced
by
sim
ulat
ion
and
thos
e ob
tain
ed fr
om A
PMS,
PD
AR
S, a
nd
NA
OM
S) o
f 0.6
0 or
bet
ter w
ill b
e ca
lcul
ated
.
3/1
09/0
3 C
omp
Sep
03
#4 (0
4/03
)
FAST
-TIM
E SI
M. O
F
SYS-
WID
E R
ISK
S
2.1/
5 D
emon
stra
te S
urve
y of
Ful
l NA
S: T
he
NA
OM
S is
inco
rpor
atin
g in
puts
from
the
air-
carr
ier,
com
mer
cial
flig
ht c
rew
, air-
traff
ic-c
ontro
l, ca
bin-
crew
, m
echa
nics
/tech
nici
ans,
corp
orat
e, a
nd G
A
com
mun
ities
rout
inel
y.
Con
tinue
d po
sitiv
e re
spon
se fr
om a
ll of
th
e so
licite
d co
mm
uniti
es.
At l
east
60%
inte
rvie
w c
ompl
etio
n ra
te.
6/3
06/0
5 #5
(0
6/05
)
2.1/
6 Tr
ansf
er o
f Avi
atio
n Ex
traN
et to
In
dust
ry
Avi
atio
n Ex
traN
et c
ore
back
bone
link
s an
d se
rvic
es tr
ansf
erre
d to
third
par
ties.
M
iddl
ewar
e se
rvic
es v
alid
ated
and
m
aint
aine
d by
third
par
ties.
End
use
r ac
cess
to to
p 30
US
airp
orts
, rea
l-tim
e ac
cess
to F
AA
cen
ters
, ter
min
al a
nd
surf
ace
syst
ems d
ata,
and
bac
kbon
e ac
cess
to U
S ai
rline
ext
rane
ts.
6/3
03/0
4 #5
(0
6/05
)
AV
IATI
ON
DA
TA
SHA
RIN
G T
OO
LS &
M
IDD
LEW
AR
E
2.1/
7 Sy
stem
-wid
e R
isk A
sses
smen
t bas
ed o
n M
erge
d FO
QA
Dat
a: D
emon
stra
te v
alue
to
syst
em-w
ide
risk
asse
ssm
ent o
f bas
ing
anal
ysis
on
mer
ged
FOQ
A d
ata
from
a
maj
or p
ortio
n of
the
NA
S us
er c
omm
unity
.
Two
air c
arrie
rs a
re p
rovi
ding
dat
a fo
r m
ergi
ng in
to a
com
mon
dat
abas
e.
Subj
ect-m
atte
r exp
erts
acc
ept a
nd u
se
indi
catio
ns o
f sys
tem
-wid
e sa
fety
is
sues
from
ana
lyse
s of t
he d
atab
ase.
3/1
09/0
4 #5
(06/
05)
APM
S TO
OLS
26
2.1/
8 M
erge
FO
QA
Dat
a w
ith P
erfo
rman
ce
Dat
a A
naly
sis a
nd R
epor
ting
Syst
em
(PD
AR
S) D
ata:
Dem
onst
rate
the
valu
e to
sy
stem
-wid
e sa
fety
-ris
k as
sess
men
t of
mer
ging
FO
QA
dat
a w
ith P
DA
RS
data
fro
m th
e A
TC c
omm
unity
.
Two
air c
arrie
rs a
re p
rovi
ding
dat
a fo
r m
ergi
ng w
ith P
DA
RS
data
from
at
leas
t 6 A
TC si
tes.
Sub
ject
-mat
er
expe
rts a
ccep
t and
use
the
indi
catio
ns
of sy
stem
-wid
e is
sues
rela
ted
to s
afet
y fro
m a
naly
sis o
f the
dat
abas
e.
6/3
09/0
4 #5
(06/
05)
APM
S &
PD
AR
S TO
OLS
2.
1/9
Mer
ge F
OQ
A D
ata
with
Mod
els &
Si
mul
atio
ns:
Dem
o us
e of
FO
QA
dat
a to
en
hanc
e, v
erify
, and
val
idat
e sy
stem
m
odel
s and
sim
ulat
ions
.
Subj
ect-m
atte
r exp
erts
use
cap
abili
ty to
pr
edic
t sys
tem
-wid
e ef
fect
s. A
t lea
st 2
maj
or a
ir ca
rrier
s are
usin
g sim
ulat
ions
to
aid
eva
luat
ions
of p
ropo
sed
inte
rven
tions
6/3
06/0
5 #5
(0
6/05
) FA
ST-T
IME
SIM
O
F SY
S-W
IDE
RIS
KS
2.1/
10
PDA
RS
Ope
ratio
nally
Pro
ven
at
Ope
ratio
nal S
ite:
Ope
ratio
nal v
alue
of
PDA
RS
prov
en b
y its
dai
ly u
se a
t ATC
fa
cilit
ies.
Softw
are
and
netw
ork
equi
pmen
t tra
nsfe
rred
to F
AA
ATC
to c
ontin
ue
oper
atio
n w
ith a
ll de
velo
ped
PDA
RS
tool
s. T
he fa
cilit
ies i
n at
leas
t 3 o
f the
9
ATC
regi
ons a
re n
etw
orke
d on
PD
AR
S.
6/3
06/0
5 #5
(06/
05)
PD
AR
S TO
OLS
2.
1/11
D
emon
stra
te V
alue
of P
ilot S
urve
y:
Res
ults
of s
urve
ys o
f car
rier a
nd G
A p
ilots
ar
e st
atis
tical
ly a
naly
zed
to id
entif
y op
erat
iona
l iss
ues.
Exam
ples
of o
pera
tiona
lly si
gnifi
cant
in
form
atio
n de
rived
from
the
NA
OM
S su
rvey
s of t
he a
ir ca
rrie
r and
the
GA
pi
lots
hav
e be
en p
rese
nted
. Su
bjec
t m
atte
r exp
erts
acc
ept a
nd a
gree
with
at
leas
t 75%
of t
he is
sues
iden
tifie
d.
6/1
06/0
5 #5
(06/
05)
N
AO
MS
2.1/
12
Risk
Ass
essm
ent o
f Sys
tem
-wid
e Effe
cts
of C
hang
es V
alid
ated
: A
naly
tical
risk
as
sess
men
t too
ls a
re li
nked
to fa
st-ti
me
simul
atio
ns fo
r the
scen
ario
s of “
In-c
lose
A
ppro
ach
Cha
nges
”. .
Risk
ass
essm
ents
ar
e m
ade
for a
rang
e of
a/c
mix
es, w
eath
er,
and
cont
rolle
r req
uest
tim
ing.
Val
idat
ion
of
the
risk
proj
ectio
ns a
nd c
ausa
l fac
tors
are
ba
sed
on c
orre
latio
ns w
ith d
ata
colle
cted
by
APM
S, P
DA
RS,
and
NA
OM
S
The
fully
link
ed h
uman
pe
rform
ance
/air
traff
ic si
mul
atio
n an
d th
e ris
k an
alys
is sy
stem
are
val
idat
ed
for t
he In
-clo
se A
ppro
ach
Cha
nge
scen
ario
aga
inst
dat
a co
llect
ed b
y A
PMS,
PD
AR
S, a
nd N
AO
MS.
Ris
k as
sess
men
ts a
re m
ade
for a
rang
e of
a/c
m
ixes
, wea
ther
, and
con
trolle
r req
uest
tim
ing.
Sub
ject
mat
ter e
xper
ts a
gree
w
ith p
redi
cted
ass
essm
ents
of ri
sk in
at
leas
t 75%
of t
he si
tuat
ions
mod
eled
.
6/1
06/0
5 #5
(06/
05)
PR
OTO
TYPE
SY
S-W
IDE
RIS
K
ASS
ESSM
ENT
Not
e: M
ilesto
ne 2
.1/6
Tra
nsfe
r of A
viat
ion
Extra
Net
to In
dustr
y w
as d
elet
ed w
ith th
e de
-sco
pe o
f the
Info
rmat
ion
Shar
ing
Elem
ent i
n FY
’01.
Mile
stone
s 2.1
/5 D
emo
Surv
ey o
f Ful
l NA
S an
d 2.
1/9
Mer
ge F
OQ
A D
ata
with
Mod
els &
Sim
ulat
ions
wer
e de
lete
d du
e to
in
adeq
uate
fund
s.
27
TABL
E 4.
1 (B
) EL
EMEN
T M
ILES
TON
E C
HA
RT
- 2.
1.1
DA
TA-A
NA
LYSI
S TO
OLS
DEV
ELO
PMEN
T
No.
Ti
tle/D
escr
iptio
n Ex
it C
rite
ria
TRL/
IR
L D
ate
Mo/
Yr
Leve
l II R
oll-u
p (F
Y Q
uarte
r) PR
OD
UC
T 2.
1.1/
1 Pr
e-A
vSP
Ta
xono
my
Wor
ksho
p: H
ost f
irst i
nter
natio
nal
wor
ksho
p on
“br
idgi
ng”
taxo
nom
y In
tern
atio
nal p
artic
ipat
ion
and
acce
ptan
ce o
f nee
d
7/3
12/9
98
Com
p
Dec
98
2.1/
2 (0
6/02
)
2.1.
1/2
Pre-
AvS
P
Firs
t-gen
erat
ion
Cau
sal D
atab
ase:
Cre
ate
first
-gen
erat
ion
caus
al d
atab
ase
usin
g ad
vanc
ed
codi
ng ta
xono
mie
s & p
roce
sses
Dem
onst
ratio
n of
impr
oved
acc
ess f
or
hum
an fa
ctor
s res
earc
h an
d ca
usal
an
alys
is
6/3
09/9
9 C
omp
Se
p 99
2.1/
2 (0
9/02
)
ASR
S TO
OLS
2.
1.1/
3 A
PMS
for
ATC
: D
emon
stra
te a
per
form
ance
m
onito
ring
syst
em fo
r ATC
bas
ed o
n A
PMS
test
ed in
a li
mite
d po
rtion
of t
he A
TC
com
mun
ity.
Acc
epta
nce
by th
e A
TC c
omm
unity
and
ex
tens
ions
to o
ther
sect
ors o
f the
nat
iona
l A
TC.
At l
east
6 fa
cilit
ies a
re a
ctiv
ely
parti
cipa
ting
in e
valu
atio
n, a
nd p
lans
for
natio
naliz
atio
n ar
e m
ade.
4/1
2/00
C
omp
A
pr 0
0
2.1/
2 (0
9/02
)
APM
S TO
OLS
2.
1.1/
4 C
ausa
l Ana
lysis
of I
ncid
ents
: D
emon
stra
te
pote
ntia
l cap
abili
ty fo
r cau
sal a
naly
sis o
f in
cide
nts f
rom
flig
ht d
ata
Subj
ect-m
atte
r exp
erts
agr
ee to
the
pote
ntia
l val
ue o
f aut
omat
ed a
ssis
tanc
e fo
r ana
lyzi
ng in
cide
nts.
At l
east
2 a
ir ca
rrie
rs p
rovi
ding
dat
a to
dev
elop
cau
sal
anal
ysis
cap
abili
ties
3/3
09/0
0 C
omp
Se
p 00
2.1/
2 (0
9/02
)
A
PMS
TOO
LS
2.1.
1/5
Ope
ratio
nal T
est o
f Risk
Ass
essm
ent:
Dem
onst
rate
cau
sal a
naly
sis a
nd ri
sk-
asse
ssm
ent t
ools
at A
lask
a, &
TW
A L
LC
airli
nes w
ith a
utom
ated
link
age
of
hete
roge
neou
s saf
ety-
data
sour
ces.
Use
r con
curr
ence
on
the
pote
ntia
l for
re
liabl
e an
d va
luab
le a
utom
ated
as
sist
ance
for r
isk
asse
ssm
ent.
At l
east
2 ai
r ser
vice
s pro
vide
rs a
re p
rovi
ding
dat
a fo
r dev
elop
ing
risk
asse
ssm
ent
capa
bilit
ies
5/3
09/0
2 C
omp
Se
p 02
2.1/
2 (0
9/02
)
APM
S, P
DA
RS,
DA
TA A
NA
LYSI
S,
ET A
L TO
OLS
2.
1.1/
6 A
PMS
for
Cor
pora
te A
/C:
Dem
onst
rate
firs
t bu
ilds o
f APM
S to
ols f
or c
orpo
rate
airc
raft
inco
rpor
atin
g gu
ided
exp
lora
tion
of fl
ight
-re
cord
ed d
ata.
Acc
epta
nce
of th
e co
ncep
ts o
f APM
S by
th
e co
rpor
ate
com
mun
ity.
At l
east
2
corp
orat
e op
erat
ors a
re p
rovi
ding
dat
a fo
r op
erat
iona
l ana
lysi
s.
5/3
01/0
3 2.
1/7
09/0
4 A
PMS
TOO
LS
2.1.
1/7
Risk
Ass
essm
ent T
ool u
sing
Mer
ged
Dat
a:
Dem
onst
rate
an
auto
mat
ed, r
elia
ble
capa
bilit
y to
ass
ist d
ecis
ion-
mak
ers i
n as
sess
ing
risk
from
a
syst
em-w
ide
pers
pect
ive.
Acc
epta
nce
and
use
by su
bjec
t-mat
ter
expe
rts o
f the
cap
abili
ty in
the
proc
ess o
f de
cidi
ng o
n m
itiga
ting
actio
n. T
wo
maj
or
air s
ervi
ces p
rovi
ders
are
usin
g to
ols t
o ai
d th
eir s
afet
y-ris
k as
sess
men
t.
5/3
06/0
4 2.
1.7
(09/
04)
A
PMS
TOO
LS
2.1.
1/8
APM
S fo
r G
A: D
emon
stra
te th
e po
tent
ial
appl
icab
ility
of A
PMS
to G
A.
Parti
cipa
tion
by th
e G
A c
omm
unity
in
deve
lopi
ng, a
dapt
ing,
and
evo
lvin
g A
PMS-
like
tool
s. A
t lea
st, 2
maj
or G
A
Leas
ers
are
prov
idin
g da
ta fo
r ope
ratio
nal
anal
ysis
.
4/3
03/0
4 2.
1/7
09/0
4 A
PMS
TOO
LS
28
2.1.
1/9
Mer
ge F
OQ
A D
ata
with
Mod
els &
Si
mul
atio
ns:
Dem
onstr
ate
use
of F
OQ
A d
ata
to e
nhan
ce, v
erify
, and
val
idat
e sy
stem
mod
els
and
sim
ulat
ions
.
Subj
ect-m
atte
r exp
erts
acc
ept a
nd u
se
capa
bilit
y to
pre
dict
syst
em-w
ide
effe
cts.
A
t lea
st 2
maj
or a
ir ca
rrie
rs a
re u
sing
simul
atio
ns to
aid
eva
luat
ions
of
prop
osed
inte
rven
tions
.
6/3
09/0
4 2.
1/9
03/0
5 A
PMS
TOO
LS
2.1.
1/10
D
emon
stra
te M
ergi
ng o
f Inf
orm
atio
n fr
om
Text
ual a
nd D
igita
l Dat
a So
urce
s:
Dem
onst
rate
an
auto
mat
ed c
apab
ility
to m
erge
re
leva
nt in
form
atio
n fr
om fr
ee-te
xt re
ports
and
fli
ght-r
ecor
ded
data
.
Subj
ect-m
atte
r exp
erts
acc
ept a
nd u
se th
e ca
pabi
lity
to a
ssis
t in
gain
ing
insi
ght i
nto
syst
em e
vent
s. T
wo
maj
or a
ir ca
rrie
rs
have
agr
eed
to re
st an
d ev
alua
te th
e ca
pabi
lity.
5/3
06/0
5 2.
1/12
(0
6/05
)
APM
S TO
OLS
N
OTE
: M
ilesto
nes 2
.1.1
/3, 2
.1.1
/6, 2
.1.1
/8, a
nd 2
.1.1
/9 w
ere
dele
ted
from
this
activ
ity w
ith th
e re
-stru
ctur
ing
in F
Y’0
1 an
d ha
ve b
een
assig
ned
to e
lem
ent 2
.1.5
Intra
mur
al M
onito
ring
since
FY
’02.
29
TABL
E 4.
1 (C
) EL
EMEN
T M
ILES
TON
E C
HA
RT
- 2.
1.5
INTR
AM
UR
AL
MO
NIT
OR
ING
No.
Ti
tle/D
escr
iptio
n Ex
it C
rite
ria
TRL/
IR
L D
ate
(Mo/
Yr)
Leve
l II R
oll-u
p (F
Y Q
uart
er)
PR
OD
UC
T 2.
1.5/
1 A
pply
APM
S to
ATC
: D
emon
strat
e ap
plic
atio
ns
of A
PMS
conc
epts
& m
etho
dolo
gies
to A
TC fo
r pe
rform
ance
mon
itorin
g.
Acc
epta
nce
by th
e A
TC c
omm
unity
and
ex
tens
ion
to o
ther
sect
ors o
f nat
iona
l A
TC.
At l
east
6 fa
cilit
ies a
re a
ctiv
ely
parti
cipa
ting
in e
valu
atio
n, a
nd p
lans
for
natio
naliz
atio
n ar
e m
ade.
4/1
03/0
0 C
omp
Apr
00
2.1/
1
(03/
00)
PD
AR
S TO
OLS
2.
1.5/
2 O
pera
tiona
l Tes
t of R
isk A
sses
smen
t: D
emon
stra
te, i
n op
erat
iona
l env
ironm
ent,
tool
s fo
r mer
ging
het
erog
eneo
us d
atab
ases
to a
id
caus
al a
naly
sis a
nd ri
sk a
sses
smen
t.
Use
r con
curr
ence
on
the
pote
ntia
l for
re
liabl
e an
d va
luab
le a
utom
ated
as
sist
ance
for r
isk
asse
ssm
ent.
Tw
o ai
r se
rvic
es p
rovi
ders
are
pro
vidi
ng d
ata
for
deve
lopi
ng ri
sk a
sses
smen
t cap
abili
ties
5/3
09/0
2 C
omp
Se
p 02
2.1.
2 (0
6/02
)
APM
S TO
OLS
2.
1.5/
3 M
erge
d FO
QA
Dat
a: D
emon
stra
te th
e ab
ility
an
d po
tent
ial v
alue
to sy
stem
-wid
e sa
fety
-ris
k as
sess
men
t of m
ergi
ng d
e-id
entif
ied
FOQ
A d
ata.
Two
maj
or a
ir ca
rrie
rs a
re p
rovi
ding
FO
QA
dat
a fo
r mer
ging
into
a c
omm
on
data
base
. Su
bjec
t-mat
ter e
xper
ts a
gree
w
ith in
itial
indi
catio
ns o
f sys
tem
-wid
e is
sues
rela
ted
to sa
fety
.
3/1
09/0
2 C
omp
Se
p 02
2.1.
7 (0
6/04
) 2.
1.8
(06/
05
2.1.
9 (0
6/05
))
APM
S TO
OLS
2.
1.5/
4 A
PMS
for
Cor
pora
te A
/C:
Dem
onst
rate
firs
t bu
ilds o
f APM
S to
ols f
or c
orpo
rate
airc
raft
inco
rpor
atin
g gu
ided
exp
lora
tion
of fl
ight
-re
cord
ed d
ata.
Acc
epta
nce
of th
e co
ncep
ts o
f APM
S by
th
e co
rpor
ate
com
mun
ity.
At l
east
2
corp
orat
e op
erat
ors a
re p
rovi
ding
dat
a fo
r ope
ratio
nal a
naly
sis.
5/3
03/0
3 2.
1/7
09/0
4 A
PMS
TOO
LS
2.1.
5/5
Mer
ged
FOQ
A D
ata
from
Mul
tiple
Air
C
arri
ers:
Dem
onst
rate
the
abili
ty a
nd p
oten
tial
valu
e to
syst
em-w
ide
safe
ty-r
isk
asse
ssm
ent o
f m
ergi
ng d
e-id
entif
ied
FOQ
A d
ata
from
mul
tiple
ai
r car
riers
.
Two
maj
or a
ir ca
rrie
rs a
re p
rovi
ding
FO
QA
dat
a fo
r mer
ging
into
a c
omm
on
data
base
. Su
bjec
t-mat
ter e
xper
ts a
ccep
t an
d us
e in
dica
tions
of s
yste
m-w
ide
issu
es re
late
d to
safe
ty fr
om a
naly
ses o
f th
e da
taba
se.
3/1
09/0
3 C
omp
Sep
03
2.1.
7 (0
9/04
)
A
PMS
TOO
LS
2.1.
5/6
APM
S fo
r G
A: D
emon
stra
te th
e po
tent
ial
appl
icab
ility
of A
PMS
to G
A.
Parti
cipa
tion
by th
e G
A c
omm
unity
in
deve
lopi
ng, a
dapt
ing,
and
evo
lvin
g A
PMS-
like
tool
s. A
t lea
st, 2
maj
or G
A
Leas
ers
are
prov
idin
g da
ta fo
r ope
ratio
nal
anal
ysis
.
4/3
03/0
4 2.
1/7
09/0
4 A
PMS
TOO
LS
2.1.
5/7
Mer
ge F
OQ
A D
ata
with
PD
AR
S
Dat
a: D
emon
stra
te th
e ab
ility
and
val
ue to
sy
stem
-wid
e sa
fety
-ris
k as
sess
men
t of m
ergi
ng
FOQ
A d
ata
from
a p
ortio
n of
the
air c
arrie
r com
mun
ity w
ith P
DA
RS
data
from
a
porti
on o
f the
ATC
com
mun
ity.
Two
maj
or a
ir ca
rrie
rs a
re p
rovi
ding
FO
QA
dat
a fo
r mer
ging
with
PD
AR
S da
ta fr
om a
t lea
st 3
ATC
site
s. S
ubje
ct-
mat
ter e
xper
ts a
ccep
t and
use
the
indi
catio
ns o
f sys
tem
-wid
e is
sues
rela
ted
to sa
fety
from
ana
lyse
s of t
he d
atab
ase.
6/1
09/0
4 2.
1.8
(09/
04)
APM
S TO
OLS
30
2.1.
5/8
Iden
tify
Ven
dors
to C
omm
erci
aliz
e A
PMS
Tool
s: Id
entif
y an
d ex
ecut
e ag
reem
ents
w
ith v
endo
rs w
ho a
re in
tere
sted
in
com
mer
cial
izin
g A
PMS
tool
s.
At l
east
one
ven
dor h
as e
xpre
ssed
inte
rest
in
com
mer
cial
izin
g on
e or
mor
e of
the
APM
S to
ols.
6/3
03/0
4 2.
1/7
(06/
05)
APM
S TO
OLS
2.
1.5/
9 Tr
ansf
er A
PMS
Tool
s to
Ven
dors
of
Com
mer
cial
FO
QA
Sof
twar
e Pr
ogra
ms:
En
ter i
nto
agre
emen
t with
at l
east
one
ve
ndor
to tr
ansf
er A
PMS
tool
s for
co
mm
erci
aliz
atio
n
Inve
ntio
n di
sclo
sure
, pat
ent a
pplic
atio
ns,
and
licen
sing
agre
emen
ts in
pla
ce
allo
win
g in
clus
ion
of m
arke
tabl
e to
ols i
n C
OTS
softw
are.
For
mal
agr
eem
ent i
n pl
ace
with
at l
east
one
ven
dor c
omm
itted
to
com
mer
cial
izin
g on
e or
mor
e A
PMS
tool
s
6/3
06/0
5 2.
1.7
(06/
05)
A
PMS
TOO
LS
2.1.
5/10
PD
AR
S O
pera
tiona
lly P
rove
n at
an
Ope
ratio
nal S
ite: S
oftw
are
and
netw
ork
equi
pmen
t dem
onst
rate
d to
show
usa
bilit
y an
d va
lue.
Softw
are
and
netw
ork
equi
pmen
t hav
e be
en tr
ansf
erre
d to
the
FAA
to c
ontin
ue
oper
atio
n w
ith a
ll de
velo
ped
PDA
RS
tool
s. F
AA
has
form
ally
acc
epte
d re
spon
sibili
ty fo
r con
tinue
d da
y-to
-day
op
erat
ion
of P
DA
RS
and
its n
etw
ork.
6/3
06/0
5 2.
1.10
(0
6/05
)
PDA
RS
TOO
LS
Not
e: M
ilesto
nes 2
.1.5
/2, 2
.1.5
/4, a
nd 2
.1.5
/6 w
ere
prev
ious
ly a
ssig
ned
to e
lem
ent 2
.1.1
bef
ore
re-s
truct
urin
g in
FY
’01
esta
blish
ed th
e el
emen
t 2.1
.5 In
tram
ural
Mon
itorin
g. M
ilesto
nes 2
.1.5
/#4
and
2.1.
4/#6
wer
e su
bseq
uent
ly d
elet
ed d
ue to
inad
equa
te fu
nds i
n FY
’03
–‘0
4.
31
TABL
E 4.
1 (D
) EL
EMEN
T M
ILES
TON
E C
HA
RT
- 2.
1.2
EXTR
AM
UR
AL
MO
NIT
OR
ING
No.
Ti
tle/D
escr
iptio
n Ex
it C
rite
ria
TRL/
IR
L D
ate
(Mo/
Yr)
Leve
l II R
oll-u
p (F
Y Q
uart
er)
PR
OD
UC
T 2.
1.2/
1 Pr
e-A
vSP
H
old
NA
OM
S W
orks
hop:
Con
duct
wor
ksho
p to
obt
ain
indu
stry
inpu
ts to
surv
ey d
esig
n.
Com
mun
ity su
ppor
t for
test
tria
l of
NA
OM
S 2/
3 12
/98
Com
pDec
98
2.1/
3(06
/02)
&
2.1/
11 (0
6/05
) 2.
1.2/
2
Con
duct
Tri
al o
f Sur
vey:
Con
duct
focu
sed
trial
of
NA
OM
S co
ncep
t with
air
carr
ier p
ilots
. C
omm
unity
supp
ort f
or N
AO
MS
impl
emen
tatio
n.
3/3
09/9
9 C
ompJ
an 0
0 2.
1/3(
06/0
2) &
2.1
/11
(06/
05
NA
OM
S)
2.1.
2/3
Impl
emen
t Flig
ht C
rew
Sur
vey:
Impl
emen
t the
N
AO
MS
prot
otyp
e fo
r the
com
mer
cial
air-
carr
iers
and
to fl
ight
-cre
w se
ctor
s.
Act
ive
parti
cipa
tion
by si
gnifi
cant
por
tion
of th
ese
com
mun
ities
. A
t lea
st 6
0% o
f th
e su
rvey
que
stio
nnai
res a
re c
ompl
eted
.
5/3
06/0
1 C
omp
A
pr 0
1
2.1.
3(06
/02)
&
2.1/
11 (0
6/05
) N
AO
MS
2.1.
2/4
Impl
emen
t Sur
vey
of M
echa
nics
&
Tech
nici
ans:
NA
OM
S ad
ds m
echa
nics
&
tech
nici
ans c
omm
unity
to su
rvey
syst
em.
Act
ive
parti
cipa
tion
by si
gnifi
cant
por
tion
of m
echa
nics
/tech
nici
ans c
omm
unity
. A
t le
ast 6
0% in
terv
iew
com
plet
ion
rate
.
5/3
06/0
3 2.
1/3
(09/
02) &
2.
1/5
(06/
05)
NA
OM
S 2.
1.2/
5 G
A P
ilot S
urve
y: N
AO
MS
adds
the
GA
pilo
t co
mm
unity
to th
e su
rvey
syst
em.
GA
surv
ey la
unch
ed w
ith a
t lea
st 8
00
inte
rvie
ws c
ompl
eted
. 5/
3 09
/02
Com
p
Sep
02
2.1/
3(09
/02)
NA
OM
S 2.
1.2/
6 D
emon
stra
te U
se o
f NA
OM
S fo
r R
isk
Ass
essm
ent:
Dem
onst
rate
abi
lity
of N
AO
MS
to
gene
rate
val
id h
ypot
hese
s of p
recu
rsor
s for
sy
stem
-wid
e sa
fety
-ris
k as
sess
men
t.
Subj
ect-m
atte
r exp
erts
(e.g
., A
TA,
ALP
A, A
OPA
, HA
I, ai
rline
s) a
ccep
t and
ag
ree
that
at l
east
75%
of t
he h
ypot
hese
s ar
e pl
ausi
ble
issu
es.
6/3
06/0
4 2.
1/11
(0
6/05
)
NA
OM
S 2.
1.2/
7 D
emon
stra
te S
urve
y of
Ful
l NA
S: T
he N
AO
MS
is in
corp
orat
ing
inpu
ts fr
om th
e ai
r-ca
rrie
r, co
mm
erci
al fl
ight
cre
w, a
ir-tra
ffic
-con
trol,
cabi
n-cr
ew, m
echa
nics
/tech
nici
ans,
corp
orat
e,
and
GA
com
mun
ities
rout
inel
y.
Con
tinue
d po
sitiv
e re
spon
se fr
om a
ll of
th
e so
licite
d co
mm
uniti
es.
At l
east
60%
in
terv
iew
com
plet
ion
rate
.
6/3
06/0
5 2.
1/5
(06/
05)
N
AO
MS
2.1.
2/8
Dem
onst
rate
Use
of F
OQ
A D
ata
to V
alid
ate
Prob
lem
s Ide
ntifi
ed b
y N
AO
MS:
Dem
onst
rate
th
at N
AO
MS
relia
bly
iden
tifie
s sys
tem
pro
blem
s th
at a
re v
alid
ated
and
qua
ntifi
ed b
y FO
QA
dat
a.
NA
OM
S ve
rifie
d as
a so
urce
for
gene
ratin
g hy
poth
eses
of p
recu
rsor
s. A
t le
ast 5
0% o
f pro
blem
s ide
ntifi
ed re
late
d to
FO
QA
are
val
idat
ed in
FO
QA
dat
a.
3/3
06/0
5 2.
1/5
(06/
05)
N
AO
MS
2.1.
2/9
Impl
emen
t Sur
vey
of A
TC a
nd C
abin
Cre
ws:
N
AO
MS
adds
ATC
and
Cab
in C
rew
co
mm
uniti
es to
the
surv
ey sy
stem
.
Posit
ive
parti
cipa
tion
by a
sign
ifica
nt
prop
ortio
n of
eac
h of
thes
e co
mm
uniti
es.
At l
east
60%
inte
rvie
w c
ompl
etio
n ra
te.
5/3
09/0
4 2.
1/5
(06/
05)
2.1.
2/10
Es
tabl
ish N
AO
MS
Wor
king
Gro
up: E
stab
lish
a w
orki
ng g
roup
of a
viat
ion
com
mun
ity
mem
bers
(ind
ustry
and
gov
ernm
ent)
with
who
m
study
info
rmat
ion
can
be sh
ared
and
who
can
pr
ovid
e op
erat
iona
l adv
ice.
The
Wor
king
Gro
up is
in p
lace
and
firs
t m
eetin
g he
ld w
ith p
artic
ipat
ion
by a
t lea
st 70
% o
f inv
itees
.
6/1
09/0
3 D
elay
ed to
12
/03
2.1/
11
(06/
05)
N
AO
MS
32
2.1.
2/11
Es
tabl
ish M
etho
dolo
gy fo
r Sur
vey
of A
TC:
NA
OM
S pr
epar
es th
e su
rvey
pac
kage
and
co
nduc
ts th
e fie
ld tr
ials
of a
ir tra
ffic
con
trolle
rs.
Indi
catio
ns fr
om th
e fie
ld tr
ials
of
posit
ive
parti
cipa
tion
by a
sign
ifica
nt
(i.e.
, at l
east
60%
retu
rn) o
f the
re
pres
enta
tive
grou
p fro
m th
e A
TC
com
mun
ity.
5/3
09/0
4 2.
1/11
(0
6/05
)
NA
OM
S
2.1.
2/12
D
emon
stra
te V
alue
of P
ilot S
urve
y: A
naly
ze
stat
istic
ally
the
resu
lts o
f the
surv
ey to
dat
e to
un
cove
r and
des
crib
e op
erat
iona
lly si
gnifi
cant
fe
atur
es o
f the
ope
ratio
n of
the
syst
em.
Exam
ples
of o
pera
tiona
lly si
gnifi
cant
in
form
atio
n ar
e de
rived
from
the
surv
eys
of th
e ai
r car
rier a
nd th
e G
A P
ilots
. Su
bjec
t-mat
ter e
xper
ts a
ccep
t and
agr
ee
with
at l
east
75%
of t
he is
sues
iden
tifie
d.
6/3
06/0
5 2.
1/11
(0
6/05
)
N
AO
MS
2.1.
2/13
D
emon
stra
te Im
prov
ed M
etho
dolo
gies
for
Cos
t-effe
ctiv
e Sur
veys
: Dem
onst
rate
a re
liabl
e,
effic
ient
, cos
t-eff
ectiv
e su
rvey
pro
cess
that
is
read
y to
inco
rpor
ate
and
inte
grat
e in
puts
from
all
of th
e ai
r ser
vice
s com
mun
ities
for t
he fu
ll vi
ew
of th
e N
AS.
Subj
ect-m
atte
r exp
erts
agr
ee th
at th
e pr
oces
s is p
ract
ical
and
ext
enda
ble
to
rout
inel
y su
rvey
the
cons
titue
ncie
s of t
he
avia
tion
syst
em.
At l
east
75%
agr
ee th
at
the
info
rmat
iona
l ben
efits
of a
per
man
ent
serv
ice
are
wor
th th
e co
st.
6/3
06/0
5 2.
1/11
(0
6/05
)
NA
OM
S N
OTE
: Th
e M
ilesto
nes 2
.1.2
/4 Im
plem
ent S
urve
y of
Mec
hani
cs a
nd T
echn
icia
ns, 2
.1.2
/7 D
emon
strat
e Su
rvey
of F
ull N
AS,
2.1
.2/8
D
emo
Use
of F
OQ
A to
Val
idat
e Pr
oble
ms I
dent
ified
by
NA
OM
S, a
nd 2
.1.2
/9 Im
plem
ent S
urve
y of
ATC
and
Cab
in C
rew
s wer
e de
lete
d du
e to
inad
equa
te p
roje
cted
fund
s bas
ed o
n co
st ex
perie
nce
of fi
rst s
urve
y of
Flig
ht C
rew
s in
FY’0
1.
33
TABL
E 4.
1 (E
) EL
EMEN
T M
ILES
TON
E C
HA
RT
2.
1.3
MO
DEL
ING
& S
IMU
LATI
ON
No.
#
Title
/Des
crip
tion
Exit
Cri
teri
a TR
L/I
RL
Dat
e (M
o/Y
r)
Leve
l II R
oll-u
p (F
Y Q
uart
er)
PR
OD
UC
T 2.
1.3/
1 Pr
e-A
vSP
U
se o
f QU
OR
UM
Dem
onst
rate
d:
Dem
onst
rate
use
of Q
UO
RU
M a
s mod
el o
f an
ecdo
tal r
epor
ts su
ch a
s ASR
S in
cide
nt a
nd
NTS
B a
ccid
ent r
epor
ts.
Mod
els o
f ane
cdot
al re
ports
are
su
ffic
ient
ly re
liabl
e re
pres
enta
tions
of
mea
ning
to a
utom
atic
ally
link
rele
vant
di
vers
e so
urce
s.
3/3
09/9
8 C
omp
Sep
98
2.1/
4 (0
9/03
)
2.1.
3/2
Pre-
AvS
P
Initi
al M
odel
Dem
onst
rate
d: D
emon
strat
e ca
pabi
litie
s of e
xist
ing
mod
els
Mod
els o
f hum
an/sy
stem
inte
ract
ion
oper
atin
g in
spec
ific
scen
ario
s, ris
k ca
lcul
atio
ns e
valu
ated
aga
inst
his
toric
al
data
, and
mod
el d
efic
its id
entif
ied.
3/1
09/9
9 C
omp
Sep
99
2.1/
4 (0
9/03
)
2.1.
3/3
Mod
els B
ased
on
Mer
ged
Dat
a D
emon
stra
ted:
Use
mod
elin
g ca
pabi
lity
to te
st
safe
ty ri
sk a
sses
smen
t usin
g m
erge
d in
form
atio
n fro
m g
roun
d, a
ir, a
nd e
nviro
nmen
t.
A m
odel
ing
hier
arch
y is
ope
ratin
g in
a
dem
o m
ergi
ng g
roun
d, fl
ight
, com
pany
, an
d N
AS-
mon
itorin
g da
ta to
ass
ess r
isk
in a
scen
ario
with
4 h
uman
ope
rato
rs
and
2 ai
rcra
ft in
tera
ctin
g w
ith A
TC.
3/1
09/0
0 C
omp
Oct
00
2.1/
4 (0
9/03
)
FAST
-TIM
E SI
M
OF
SYS-
WID
E R
ISK
S 2.
1.3/
4 Sy
stem
-wid
e Si
mul
atio
n D
emon
stra
ted:
D
emon
stra
te c
apab
ility
to si
mul
ate
the
avia
tion
syst
em.
A fr
amew
ork
of m
odel
s is o
pera
ting
in
simul
atio
n of
NA
S-w
ide
oper
atio
n fo
r gr
ound
and
flig
ht w
ith 2
term
inal
are
as
in a
pp/d
ep o
ps a
nd 2
en-
rout
e op
s.
4/1
09/0
1 C
omp
Sep
01
2.1/
4 (0
9/03
) FA
ST-T
IME
SIM
O
F SY
S-W
IDE
RIS
KS
2.1.
3/5
Pred
ictiv
e C
apab
ility
Ver
ified
: Dem
onst
rate
pr
edic
tive
capa
bilit
y of
sim
ulat
ion
usin
g sy
stem
-wid
e m
odel
(s) a
nd v
erify
usi
ng N
AS
data
.
Mod
els o
f NA
S op
erat
iona
l sce
nario
s an
d ou
tput
inte
grity
are
form
ally
ve
rifie
d. A
ll so
ftwar
e pe
rfor
man
ce
para
met
ers v
erifi
ed to
with
in 1
stan
dard
de
viat
ion.
4/3
09/0
2 C
omp
Sep
02
2.1/
4 (0
9/03
) FA
ST-T
IME
SIM
O
F SY
S-W
IDE
RIS
KS
2.1.
3/6
Syst
em-w
ide
Risk
Ass
essm
ent V
alid
ated
: D
emon
stra
te sy
stem
-wid
e sa
fety
risk
as
sess
men
t usi
ng p
redi
ctiv
e si
mul
atio
ns a
nd
valid
ate
usin
g N
AS
data
.
Stru
ctur
al a
nd c
ausa
l pre
dict
ions
of r
isk
are
prod
uced
in si
mul
atio
ns.
Ass
essm
ents
of s
yste
mic
“ris
k fa
ctor
s &
cont
exts
” ar
e ev
alua
ted
agai
nst i
ncid
ent
data
base
s Cor
rela
tion
betw
een
mod
el a
nd
NA
S pe
rfor
man
ce is
0.6
0 or
bet
ter
5/3
03/0
3 2.
1/4
(09/
03)
FAST
-TIM
E SI
M
OF
SYS-
WID
E R
ISK
S 2.
1.3/
7 M
erge
FO
QA
Dat
a w
ith M
odel
s &
Sim
ulat
ions
: D
emo
use
of F
OQ
A d
ata
to
enha
nce,
ver
ify, a
nd v
alid
ate
syst
em m
odel
s an
d si
mul
atio
ns.
Subj
ect-m
atte
r exp
erts
use
cap
abili
ty to
pr
edic
t sys
tem
-wid
e ef
fect
s. A
t lea
st 2
maj
or a
ir ca
rrier
s are
usin
g sim
ulat
ions
to
aid
eval
uatio
ns o
f pro
pose
d in
terv
entio
ns.
6/3
06/0
5 2.
1/9
(06/
05)
FAST
-TIM
E SI
M
OF
SYS-
WID
E R
ISK
S
2.1.
3/8
Dist
ribu
ted
Sim
ulat
ion
Cap
abili
ty:
Prot
otyp
e di
strib
uted
sim
ulat
ion
capa
bilit
y to
ass
ess r
isk
by a
cces
sing
data
ove
r a se
cure
inte
rnet
.
Abi
lity
to su
ppor
t dis
tribu
ted
clie
nt-
base
d an
alys
es o
f saf
ety
risk
with
rem
ote
acce
ss to
dat
a so
urce
s is d
emon
stra
ted
and
risk
asse
ssm
ent a
ccur
acy
is
valid
ated
.
5/3
06/0
5 2.
1/9
(06/
05
FAST
-TIM
E SI
M
OF
SYS-
WID
E R
ISK
S
34
2.1.
3/9
Pred
ictio
n of
Sys
tem
-wid
e Effe
cts o
f C
hang
es V
alid
ated
: D
emo
valid
ity o
f pr
edic
ted
stat
istic
s of w
orkl
oad
and
syst
. per
f. du
e to
cha
nges
in c
onte
xtua
l fac
tors
dur
ing
spec
ific
scen
ario
.
Cor
rela
tion
betw
een
stat
istic
s pro
duce
d by
sim
ulat
ion
and
thos
e ob
tain
ed fr
om
APM
S, P
DA
RS,
and
NA
OM
S is
0.6
0 or
be
tter..
6/1
09/0
3 C
omp
Sep
03
2.1/
4 (0
6/03
) &
2.1/
11 (0
6/05
) FA
ST-T
IME
SIM
O
F SY
S-W
IDE
RIS
KS
2.1.
3/10
D
emon
stra
te L
inka
ge o
f Hum
an
Perf
orm
ance
/Air
Tra
ffic S
imul
atio
n to
Dat
a So
urce
s and
Aut
omat
ed R
isk A
sses
smen
t: Fa
st-ti
me
simul
atio
n is
link
ed to
dat
a so
urce
s (e
.g.,
fligh
t & w
eath
er) a
nd o
utpu
t is l
inke
d to
ris
k an
alys
is to
ol.
Link
ages
per
form
as e
xpec
ted
and
at
leas
t 75%
of t
he ri
sk a
sses
smen
ts a
re
deem
ed p
laus
ible
. D
emo
of p
oten
tial
risks
& c
ausa
l fac
tors
for T
BM
co
mpa
red
with
MIT
M a
re a
ccep
ted
as
valid
by
dom
ain
expe
rts.
6/1
06/0
4 2.
1/11
(0
6/05
)
FA
ST-T
IME
SIM
O
F SY
S-W
IDE
RIS
KS
2.1.
3/11
R
isk A
sses
smen
t of S
yste
m-w
ide E
ffect
s of
Cha
nges
Val
idat
ed: D
emo
valid
ity o
f saf
ety
risk
asse
ssm
ents
of t
he T
BM
vs.
MIT
M
scen
ario
s.
Fully
link
ed h
uman
per
f./ai
r tra
ffic
sim
an
d ris
k an
alys
is sy
stem
is v
alid
ated
for
TBM
vs.
MIT
M sc
enar
ios a
gain
st
APM
S, P
DA
RS,
& N
AO
MS
data
. Su
bjec
t-mat
ter e
xper
ts a
gree
with
pr
edic
ted
asse
ssm
ents
in a
t lea
st 75
% o
f sit
uatio
ns m
odel
ed.
6/1
06/0
5 2.
1/11
(0
6/05
)
FAST
-TIM
E SI
M
OF
SYS-
WID
E R
ISK
S
NO
TE:
Mile
stone
s 2.1
.3/6
Sys
tem
-wid
e Ri
sk A
sses
smen
t Val
idat
ed, 2
.1.3
/7 M
erge
FO
QA
Dat
a w
ith M
odel
s & S
imul
atio
n, a
nd
2.1.
3/8
Dist
ribut
ed S
imul
atio
n Ca
pabi
lity
wer
e de
lete
d be
caus
e of
inad
equa
te fu
ndin
g. M
ilesto
ne 2
.1.3
/6 is
repl
aced
by
Mile
stone
s 2.
1.3/
9, 2
.1.3
/10,
and
2.1
.3/1
1 th
at sp
read
the
sam
e w
ork
over
thre
e ye
ars t
o ac
com
mod
ate
redu
ced
fund
ing.
35
4.1.1 Data Analysis Tools Development (ASMM WBS Element 2.1.1) Goal(s) The contribution of Data Analysis Tools Development to the goal of ASMM is to facilitate efficient, penetrating, and insightful analyses of textual and numerical data collected by the various components and stakeholders of the NAS to identify causal factors, accident precursors, and unsuspected features related to health, performance, and safety of the NAS. Objective(s)
• Develop tools that convert data from diverse, heterogeneous new and legacy databases into information, and create visualization capabilities that aid aviation/safety experts conduct causal analyses and safety-risk assessments.
• Demonstrate the capabilities and values of these tools so as to encourage the aviation communities to adopt their use and invest in their continuing evolutionary developments.
Approach The torrent of new data to be analyzed as the US air carriers begin to acquire flight-recorded data routinely under the FOQA program and textual data under the ASAP program, and as ATC acquires radar data routinely under the PDARS program will overwhelm the capabilities of human analysts. Consequently, the opportunity to capture safety-related information and to utilize it proactively across the aviation system will be lost. Advanced tools must be developed for efficiently converting digital and textual data into reliable information that is operationally useful for assuring safety and quality performance. The research done in this element called Data Analysis Tools Development will improve our ability to manage incoming data of all kinds and extract useful safety information from them. Based on our experience with both ASRS and APMS, we see a need for several automated capabilities that, together, will assist the analyst in uncovering and understanding the circumstances and features of an incident, and in assessing the risk of its leading to an accident. Further, we must develop the capabilities to link all such information in order to analyze reliably for the causal factors (precursors) of all incidents and to support a process of safety-risk assessment that is, in fact, the primary capability needed for proactive management of quality assurance. The efforts under the Data Analysis Tools Development element will result in advanced software that will be able to perform many tasks that only experts can presently perform with much effort. As a result, data will be better classified, more easily prioritized, more readily combined with pertinent data from other sources, and data patterns will be better understood. Data processing costs will decline because the analytic capability of humans will be supplemented by knowledge-based automation. In the development of each of these capabilities, we will select state-of-the-art tools that are adaptable to meet our needs for aviation safety analysis. We will use commercial off-the-shelf (COTS) capability whenever possible and we will augment COTS tools as necessary to fill any critical gaps just as we have for APMS. We will use these tools to explore data collected under both the Intramural and Extramural Monitoring efforts, and, on the basis of that work, we will refine, enhance, and augment the tools as necessary. We will test and evaluate these advanced tools in the operational environment in collaboration with our partners at the air carriers, the FAA ATC, and the vendors of COTS software for data processing. The roadmap of activities and the key milestones for the Data Analysis Tools Development are presented in Figure 4.1(D) and Table 4.1 (B). The primary technical challenges to developing and implementing a set of data analysis tools in the operational environment are the following:
36
• Routinely process very large quantities of data effectively and economically. • Convert data into information that is immediately and reliably meaningful to
each individual user. • Automatically analyze textual databases and knowledgeably merge information
with numerical databases. • Extend the capabilities system wide.
The identified needs are listed below. While each has value in itself, they are not independent developments. In the order in which they are presented below, they are incremental steps toward the ultimate objective of providing the information from diverse data sources for reliable causal analysis and safety-risk assessment from a system-wide perspective.
• Taxonomies • Automated Analyst Advisor • Machine Comprehension of Free Text • Database Linkage • Database Mining • Visualization of Information • Causal Analysis • Safety Risk Assessment
A significant milestone in the development of advanced tools was achieved in FY’02 with the demonstration of an operational test of risk assessment aids that supported the Level III Milestone of the same title. Up to that point, we had been building these tools as independent, stand-alone capabilities. In FY’02, we demonstrated, for the first time, the potential value of using these tools in concert
• to access information extracted from diverse data sources, • to understand the causal factors, and • to assess the risk of an identified anomalous event • from a system-wide perspective.
We used each of the ASMM tools in a set of (nearly) independent studies of the same scenario; namely, in-close approach changes
• to demonstrate the kinds of information that each data source and tool can potentially contribute to gaining insight into the complete picture of causal factors and safety risks, and
• to show the methodology for utilizing each of the tools in a complementary and synergistic process of causal analysis and safety risk assessment to aid the decision makers.
By the end of the current program in FY’05, we will have developed the tools to extract the information automatically from each of these diverse sources that is relevant to any query or scenario. In FY’02 and FY’03, our focus was on processing and analyzing digital data. In FY’04, these capabilities will be in operational test and evaluation under the Intramural Monitoring element and the focus of analytical tool development will be on textual data. Also in FY’04 we will demonstrate how these capabilities can assist decision-makers in assessing safety risk from a system-wide perspective. Information will be extracted automatically from digital data sources, such as APMS and PDARS data, and utilized in an automated risk assessment capability. In FY’05, we will augment this with the capability of extracting information from
37
textual databases such as ASRS, ASAP, and NAOMS and automatically integrating it with information from digital databases for improved risk assessment.The primary contractor in Data Analysis Tools Developments is Battelle and collaborations/sub-contracts are, or have been, with the Pacific Northwest National Laboratories, the FAA Office of System Safety, Rannoch Corp., Sandia National Laboratories, the Flight Safety Foundation, Virginia Polytechnic Institute, Oregon Graduate Institute, SUNY Stoneybrook, UC Riverside, the Naval Research Laboratory, and ONERA. Milestones/TRLs Refer to Table 4.1(A) for the ASMM sub-project Level II Milestones and to Table 4.1 (B) for the Level III Milestones for the Data Analysis Tools Development element. These Tables show the Titles and Descriptions of the milestones, their exit criteria, their technology readiness levels (TRL) and implementation readiness levels (IRL), the roll-up to the next level milestone(s), and the specific product with which the milestone is associated. 4.1.2 Intramural Monitoring (ASMM WBS Element 2.1.5) Goal(s) The contribution of Intramural Monitoring to the goal of ASMM is to assist and enable air services providers to establish the capabilities within their own organizations for proactive management of safety risk. Objective(s)
• In partnership with air carriers providers, build intramural airline-safety monitoring capabilities by testing and evaluating data analysis tools in the airline operational environment and by extending and adapting the concepts and methodologies of APMS to new applications and environments.
• In partnership with the FAA and NATCA, build intramural ATC-safety monitoring capabilities by testing and evaluating data analysis tools in the ATC operational environment and by extending and adapting the concepts and methodologies of APMS to ATC applications and environments.
• Develop a contribution to the data sources needed to support system-wide research issues, enable the development, calibration, and validation of system-wide models, and establish the baseline of operational performance against which to measure changes.
Approach This element establishes and maintains collaborations with the operational sectors to obtain the perspectives, the operational data, access to other relevant databases, and the users’ evaluations in the operational environment essential to evolving a suite of data-analysis tools offering maximum benefits at minimal cost. The collaborations are conducted under Space Act Agreements (SAA) with the individual users and the vendors and the work is maintained within the walls of each collaborator to maximize the potential for developing the concept for information sharing while minimizing the concern for misuse of data. The roadmap of activities and the key milestones for the Intramural Monitoring element are presented in Figure 4.1 (E) and Table 4.1 (C).
38
We will build on the successful APMS monitoring capability of flight-recorded data on commercial air transports and extrapolate these concepts and methodologies to enable other aviation environments to monitor their own performance. Transferring the data-analysis technology to all operating carriers, to ATC and, one day, to the GA community, is the fundamental, essential step to establishing the database that will become the essence of system-wide safety risk assessment and sharing of information across the entire community. However, it must start intramurally within each organizational element of the system. The proven approach used in the APMS program of working collaboratively with the individual users will continue. On the air-carrier side, we will continue to evolve the suite of tools to convert flight-recorded data to information on flight operational performance under the SAA’s with Alaska Airlines and with American Airlines. During FY’03, we entered into an SAA with Delta Airlines. This gives us access to a significant new source of operational data and domain expertise with which to test and continue to evolve data analysis capabilities. APMS concepts need to be extended from the current focus on flight operations to monitoring routinely aircraft and subsystem health for precursors to system incidents, to support proactive strategic planning of on-condition maintenance, and to improve flight-crew training by responding quickly to identified problems of performance. The developments of applications of APMS to all of these areas must start within each individual operator as no two pursue their quality-assurance programs in engineering, maintenance, and training exactly the same way. It is important only that the output information from these analytical processes be sufficiently standardized to enable sharing of meaningful information when the operators choose to do so. We work with the air carriers in an iterative, evolutionary development of a series of progressively more sophisticated builds. This approach enables the user to become familiar with the capabilities of the data analysis tools in a gradual learning process, and to base his requests for the next level of capability on this “hands-on” experience. It also enables the user and the vendor to agree on which tools merit consideration for commercialization. These developments have been, and will continue to be, cooperative in nature, with partners providing matching in-kind resources. NASA’s job is done when the air carriers have implemented their intramural monitoring systems and the vendors have marketed the necessary capabilities. Further, we will extend these concepts and methodologies for data analysis to performance measurement within the ATC community in much the same way as they are being extended intramurally within the air carriers. The methodologies and concepts developed for flight operations of major air carriers will be extended to meet the informational needs of air traffic management. We have established collaborations with the ATC community (FAA and NATCA) to obtain the perspectives, the operational data, access to other relevant databases, and the users’ evaluations essential to evolving a suite of data analysis tools in support of the FAA’s Performance Data and Analysis Reporting System (PDARS). The system provides daily reporting of processed data by time and place showing traffic counts, deviations, display of aircraft tracks, and replay capability that can be shared by all facilities on the network. The work is contained within the ATC community to maximize the potential for developing the concept for information sharing while minimizing the concern for misuse of data. Under the Data Analysis Tools Developments task area, we will develop the algorithms that will allow APMS concepts to be extended from air carrier flight operations to air traffic control, and we will apply these technologies in the field. The ATC environment presents a particular and special technical challenge to establish the appropriate and useful performance metrics. The test and evaluation of the PDARS concept started in FY’00 with installations at and networking of the facilities of the Western-Pacific Region and the National Traffic Management Center in Herndon, VA. By the end of FY’02, the PDARS network had been extended to include the facilities of the Southwestern and Southern Regions. During FY’03,
39
we obtained approval of a National Procedure Change (NPC) for one year that enables the extension of the test and evaluation of PDARS to all 20 Centers of the continental U.S. This extension of the secure PDARS network will be completed in FY’04. Fundamental to the concept of proactive management of risk is the need to access diverse sources of data and to share information for collaborative decision making. We have found that, even within the most advanced air carriers, there does not exist an efficient infrastructure to share information quickly and reliably across intramural sectors. The same situation maintains among the facilities of the ATC community. In the “bottom-up” aspect of monitoring the system, there is a need for the infrastructure to enable the sharing of aviation safety information across each organization's internal operations. Until this need is met within the organizations, the ASMM sub-project will not be able to achieve its broader system-wide safety objectives that rely on sharing such information across organizations and nations. NASA Ames has unique and well-regarded expertise in the development and utilization of sophisticated middleware (software that smoothes the transfer, translation, and combination of data drawn from different sources); and reliable information security systems. Therefore, as we evolve the data analysis tools within each organization, we will also assist each organization develop its internal infrastructure for sharing the information created by these tools. Then we will develop a secure network infrastructure that provides access to safety-relevant information developed within the organizations, and adapt commercial off-the-shelf software for automated data-handling services that provide translation, integration, and validation services to support system-wide applications and queries. The milestone for FY’04 calls for the identification of vendors to commercialize APMS tools. Disclosures of Invention have been processed for five digital-data processing and analyses tools. In FY’05, we will transfer the APMS tools through licensing and we will transfer the PDARS network and software to FAA Air Traffic Services. In developing the intramural capabilities for monitoring these operations, as in the development of the extramural monitoring, progress will depend on engaging the active involvement of the aviation community. The primary contractor in this element of Intramural Monitoring is Battelle and collaborations with the FAA Office of System Safety, FAA Office of System Capacity Requirements, ProWorks, and the ATAC Corp. APMS and PDARS are identified as ASMM deliverable products. Milestones/TRLs Refer to Table 4.1(A) for the ASMM Sub-Project Level Milestones and to Table 4.1 (C) for the Level III Milestones for the Intramural Monitoring element. These Tables show the Titles and Descriptions of the milestones, their exit criteria, their technology readiness levels (TRL) and implementation readiness levels (IRL), the roll-up to the next level milestone(s), and the specific product with which the milestone is associated. 4.1.3 Extramural Monitoring (ASMM WBS Element 2.1.2) Goal(s) The contribution of Extramural Monitoring to the goal of ASMM is the “top-down” element of the dual strategy for monitoring. This element aims at establishing the methodologies for a permanent field implementation of a National Aviation Operational Monitoring Service (NAOMS) responsible for developing and maintaining a comprehensive and coherent survey of the safety and performance of the NAS.
40
Objective(s) • Develop comprehensive survey methods for monitoring the overall state of the
National Aviation System. • Provide decision makers in air carriers, air traffic management, and other air
services providers with regular, accurate, and insightful measures of the health, performance, and safety of the National Aviation System.
• Ensure that changes of technology or procedures introduced into the system are producing expected improvements without producing unwanted side effects.
• Develop a contribution to the data sources needed to support system-wide research issues, enable the development, calibration, and validation of system-wide models, and establish the baseline of operational performance against which to measure changes.
Approach National Aviation Operational Monitoring Service (NAOMS) It is necessary to find a source of aviation data and to create a data-collection mechanism that is expressly tailored to the objectives and needs of AvSP. After careful consideration of the alternatives at the outset of this study, it appeared that the best way to develop such data is to survey the operators of the aviation system (i.e., its pilots, controllers, mechanics, dispatchers, flight attendants, and others) on a regular basis. The 27-year history of the Aviation Safety Reporting System (ASRS) had demonstrated to us the value of user reporting in creating feedback on the day-to-day operation of the NAS. This element aims at developing and validating the methodologies for a permanent implementation of NAOMS (an ASMM deliverable product) to maintain a comprehensive and coherent survey of the safety and performance of the NAS. The roadmap of activities and the key milestones for the Extramural Monitoring element are presented in Figure 4.1 (F) and Table 4.1 (D). The purposes of this effort are to:
(1) Create a mechanism to routinely measure the safety of the NAS in a quantitatively precise way,
(2) Demonstrate the use of this mechanism to assess trends in NAS safety and to identify the factors driving those trends, and
(3) Identify safety and efficiency effects of new flight and Air Traffic Management (ATM) technologies and/or procedures as they are inserted into the operating environment.
Keys to the success of NAOMS include: 1) Plausibility and understandability of NAOMS statistics (e.g., reasonable and
reliable representation of the relative frequencies with which unwanted events occur),
2) stability and interpretability of NAOMS statistical trends, 3) sensitivity to industry concerns about data misuse, and 4) timely and appropriate disclosures of NAOMS findings.
This list is not exhaustive, but it points to several related ideas with a common root. The data NAOMS collects must yield indications of system safety that are interpretable and deemed credible by the industry. NAOMS data must be seen as providing useful information that ultimately supports actions (or in some cases, justifies inactions) that make the NAS a safer place in which to fly.
41
Accordingly, the NAOMS Team has devoted a great deal of energy to developing a methodologically sound survey process. Trade-offs have been made among precision, accuracy, and cost. The main variables that can be manipulated to accomplish these tradeoffs are sample size and the recall period. The very successful Field Trial of NAOMS in FY99/00 entailing more than 600 interviews helped us to quantify those trades. The Field Trial also showed us that a few outlier observations of severe events could have a substantial effect on the calculated average frequencies for many safety events and, therefore, on the statistical stability of calculated event frequencies. Fortunately, statistical methods are available that will allow the NAOMS team to generate stable period-to-period estimates of average event frequencies while identifying outlying values that may point to localized risk issues. NAOMS must be able to distinguish operationally meaningful changes in event frequencies from year-to-year. This is much more demanding then generating single-point estimates for just one year. Based on the mean tendencies and variability observed in the Field Trial data, the NAOMS Team concluded that 8000 observations per constituent group (i.e., air carrier pilots, GA pilots, controllers, mechanics, etc. are each one constituent group) would probably yield data of sufficiently fine resolution to detect year-to-year event rate changes of 20 percent or more with a high degree of certainty (except for the rarest event types). This is the sample size that was employed in FY01 for the survey of commercial flight crews. The first survey target group was active commercial pilots. The survey of the air carrier pilots was initiated in April 2001. By September 30, 2002, well over 10,000 interviews had been completed and the response rate exceeded the 70% goal. The second target group included GA, corporate, and helicopter pilots. This survey was initiated in August 2002. By September 30, 2002, 656 interviews had been completed and, again, the response rate exceeded 70%. With the implementation of the GA-pilots’ survey under NAOMS, the Level II and Level III Milestones of FY’02 were achieved. A crucial, practical measure of NAOMS success is its ability to support the aviation community in its assessment of safety risks and the efficacy of government/industry interventions. Accordingly, NAOMS has cultivated a close association with the aviation industry and organized labor including CAST. The JIMDAT Team of CAST sees NAOMS as a valuable tool for measuring the system-wide impacts of the interventions that their JSAT Teams have developed. NAOMS will actively support this process by incorporating core Safety Event questions that address CAST priorities and by developing Topical questions that address focused safety concerns. By the end of FY’03, NAOMS will have accumulated a significant database of pilot interviews. However, it is important to obtain the inputs from the air traffic controllers for a balanced perspective on the NAS operations. Although there are insufficient funds in the program to undertake a full survey of the ATC community, we will establish the methodology for such a survey as reflected in the Level III Milestone for FY’04. The NAOMS Team will prepare the survey package and conduct the field trials of the air traffic controllers. There has been reluctance on the part of the NAOMS Team to release information from the survey that might be viewed by some sectors of the community as sensitive, controversial, or premature. Nevertheless, it is important to demonstrate to the community the value of the survey process. The establishment of a NAOMS Working Group as a Level III Milestone in FY’03 is consistent with this continuing effort to keep NAOMS tuned to the perceived informational needs of the aviation community. The resolution of some issues of membership and objectives delayed the first meeting of this group to December 2003. This working group is composed of aviation community members (representing both industry and government) with whom study information can be shared and who can provide advice on the operation of the NAOMS and on the dissemination of possibly sensitive information. The Level III Milestone in FY’05 provides for a demonstration of the value of the NAOMS
42
concept based on the statistical analyses of the survey of the pilots to that date. The purpose will be to show examples of operationally significant information that have been derived solely from the surveys of the air carrier and the GA pilots. This Level III Milestone rolls-up into the Level II Milestone 1.10. All of these considerations enter into the quantified definitions of the exit criteria as they are stated in the Milestone Charts of Tables 4.1(A) and 4.1(D). However, these are merely current estimates based on the understanding gained from the field trials and the pilot surveys. The methodology continues to evolve, particularly as we look to minimize costs. The sample sizes and sampling rates for desired accuracy and precision may well be different for each constituent community. Therefore, these requirements will evolve as new sectors are incorporated into the survey and further experience is gained during program operation. A Level III Milestone in FY’05 is to demonstrate improved methodologies for cost-effective surveys. The objective is to demonstrate by FY’05 a reliable, efficient, cost-effective survey process that is ready to incorporate and integrate inputs from all of the air services communities for the full view of the NAS. A primary requirement for the implementation of this survey service is finding a permanent “home” and funding. NASA will have developed the scientific methodologies to maximize the useful information and minimize the cost, but the AvSP does not provide for the permanent service after the concept is developed and its value proven. System-wide Incident Reporting Enhancements One of the many diverse sources of existing information on the health and performance of the aviation system is the Aviation Safety Reporting System (ASRS) managed by NASA and funded by the FAA Office of System Safety. The ASRS is one of the world's best-known and most highly regarded repositories of safety information. It has received nearly 500,000 safety reports from throughout the aviation community over its 25-year history. While the ASRS per se is not formally an activity of the AvSP, it was the experience with the ASRS that stimulated the development of the NAS Operational Monitoring Service (NAOMS) as the “top-down” element of the dual strategy of monitoring of the ASMM Project. Furthermore, the ASRS offers a database for testing and evaluating of some of the tools for processing and analyzing anecdotal data that are being developed under the ASMM Project. However, many of the underpinning ASRS operations are legacy system, built in the early to mid-1980s. Its infrastructure must be upgraded to enhance its efficiency and the quality of its diagnostic processes. Further, ASRS was constructed as a standalone information system before the advent of the Internet and other modern networking tools. Studies have identified the requirements to update the processing of ASRS reports, but these have not been implemented due, in part, to concerns for maintaining confidentiality and, in part, to inadequate funding. This area of research will tackle those needs. ASRS extensions will be accomplished through upgrades of the supporting infrastructure, improved capabilities such as electronic report submission, and test and evaluation of an analyst decision-support system to be developed under “Data Analysis Tools Development”. The AvSP will benefit from the convenient and knowledgeable access to this unique “test bed” for capabilities that will ultimately be needed for the NAOMS and other sources of textual data. The ASRS will benefit from upgrading its legacy systems to state-of-the-art capabilities for managing, processing, and accessing textual incident reports. The aviation community will benefit from having ASRS data more accessible and better connected to other aviation safety resources as an element of the AvSP monitoring concept. We will also explore other similar data sources such as the ASAP and we are developing an intramural text-processing capability to interface with that program. The first generation of that capability is to be tested under the Intramural Monitoring element during FY’04 in collaboration with American Airlines under an SAA.
43
The primary contractor in this element is Battelle and collaborations/sub-contracts are with the Center for Public Health Research and Evaluations, Ohio State University, and Dodd Associates. Milestones/TRLs Refer to Table 4.1(A) for the ASMM Sub-Project Level Milestones and to Table 4.1 (D) for the Level III Milestones for the Extramural Monitoring element. These Tables show the Titles and Descriptions of the milestones, their exit criteria, their technology readiness levels (TRL) and implementation readiness levels (IRL), the roll-up to the next level milestone(s), and the specific product with which the milestone is associated. 4.1.4 Modeling and Simulations (ASMM WBS Element 2.1.3) Goal(s) The contribution of Modeling and Simulations to the goal of ASMM is to support reliable prediction of the impact of new technologies and procedures on operations, communications, and, in particular, the potential for human error. Models and simulations (an ASMM deliverable product) will be validated with data obtained from Intramural and Extramural Monitoring. These predictions will identify the system-wide effects of design or procedural changes introduced into, or proposed for, the National Aviation System. Objective(s):
• Evaluate extant system and human-performance models, for applicability to system-wide representations
• Identify human-performance characteristics contributory to the safety of NAS operations; sensitivities of these characteristics to changes in design, procedures, and training; and requirements for human-performance models to fill gaps in modeling technologies,
• Develop empirical evaluation methods to validate models as reliable bases for causal analyses and safety risk assessment.
• Develop simulations to support reliable prediction of the system-wide effects of new technologies or procedures.
Approach A proactive approach to ensuring the continued performance and safety of the National Aviation System (NAS) requires a capability for modeling and simulation in order to predict the effects of changes in operational procedures or new technologies before they are implemented. The objectives of the Modeling and Simulation sub-element are to establish consistent and predictable relationships among elements of the NAS with emphasis on incorporating appropriate human behavioral models. We will develop a rigorous and effective methodology for validating the models against measured data. The simulation sub-element will establish the operating conditions and the performance parameters to answer queries about a range of technology developments by exercising models of the system. The roadmap of activities and the key milestones for the Modeling & Simulations element are presented in Figure 4.1 (G) and Table 4.1 (E).
44
Models This element provides for the development of tools for modeling the NAS at a level of detail sufficient to track key safety characteristics to support prediction and decision-making. Our work is to develop models that explicitly incorporate human performance into existing NAS modeling tools appropriate for representing system-wide operations. We expect to develop a hierarchy of models appropriate to the variety of questions that will be addressed, and to structurally characterize the relationship between incidents and accidents. Modeling the elements of the NAS including the human participants in that system serves as a computational test bed for simulating and analyzing system performance, including the contributions of individual operators, individual elements of the system, technologies and large-scale system flow and control issues. Simulations The models of the system elements will need to be exercised in simulation to predict system performance. This sub-element provides for using computer simulations to test continuously the models and to identify errors or gaps as models are developed for system-wide assessment. Data and information obtained by the Intramural and Extramural Monitoring processes will be used to validate and verify the prediction of the system models. Simulation can be undertaken at time-references that are of particular interest to the analyst (fast, slow, and real time). We will use fast-time simulations (an ASMM deliverable product) to support safety risk assessment, identify performance metrics, and focus requirements for the more expensive man-in-the-loop simulations. The primary contractor for this work is Battelle and collaborations/sub-contracts have been, or are with, the ATAC Corp., San Jose State University, Georgia Tech, the Rannoch Corporation, the Sandia Laboratories, the Naval Research Laboratory, and the Flight Safety Foundation. Glenn Research Center is responsible for the work on modeling and predictions of engine problems from the perspective of system-wide effects. Milestones/TRLs Refer to Table 4.1(A) for the ASMM Sub-Project Level Milestones and to Table 4.1 (E) for the Level III Milestones for the Modeling & Simulations element. These Tables show the Titles and Descriptions of the milestones, their exit criteria, their technology readiness levels (TRL) and implementation readiness levels (IRL), the roll-up to the next level milestone(s), and the specific product with which the milestone is associated. 4.2 SUB-PROJECT/ELEMENT CAPABILITIES AND PRODUCTS The ability to monitor continuously, convert the collected data into reliable information, and share that information among the stakeholders in the aviation system for collaborative decision making is the basis for a revolutionary, proactive approach to managing the aviation system. The end products of the ASMM element are to provide the industry with the tools and methodologies to enable this proactive approach to the prevention of accidents. Figure 4.2 (A) shows the ASMM products (highlighted in italics), capabilities, system analyses and benefits. Table 4.2 shows the ASMM deliverable Products including the targeted problem.
45
FIGURE 4.2 (A)
ASMM PRODUCTS, CAPABILITIES, SYSTEM ANALYSIS & BENEFITS The flow chart of Figure 4.2 (A) indicates the key roles that the ASMM products play in the process of analyzing both quantitative (e.g., APMS and PDARS) and anecdotal (e.g., NAOMS, and ASRS) data to identify and evaluate precursors and performance trends. Formulation of an intervention strategy requires capabilities for causal analyses and safety-risk assessments (e.g., fast-time simulations). These lead to improved safety analysis and more reliable intervention strategies for the prevention of accidents. Each of these ASMM Products has stand-alone capabilities that continue to evolve as the Data Analysis Tools are adapted to meet the needs of the constituency and for which it was primarily developed. However, the true and overriding value of the ASMM Products is as an integrated suite of tools to enable the achievement of the ASMM objective of providing a system-wide perspective on proactive management of the safety risk of the NAS.
Aviation System Monitoring and Modeling
Pro
du
cts C
ap
ab
ilit
y
Sy
stem
s A
na
lysi
s B
en
efi
t
Reduced accident rate
Measure and identify NAS performance trends
Aircraft Performance Measurement System (APMS) tools
upgrades
Incident Reporting Enhancement tools
NAS monitoring system
Improve safety analysis
Improve intervention/prevention/mitigation system designs
Increase technology performance Increase technology impact Reduce technology performance and impact uncertainties
Training: improve response to performance issues
Maintenance: planning for on-condition maintenance
Engineering:
continuous flight monitoring
Identify precursors to system incidents Increase info fidelity for causal analysis & risk assessment
PDARS tools
supporting FAA
Decision support tools Infrastructure upgrades
Identify new safety strategy need
NAS Aviation Operational Monitoring System (NAOMS)
NAS operator survey tools Feedback to NAS operators
Fast-time Simulation of System Wide Risks
Flight Operations
46
TAB
LE 4
.2
ASM
M P
ROD
UCT
S
Prod
uct
Nam
e an
d D
efin
itio
n
Prod
uct
Form
Ex
it C
rite
ria
Cus
tom
er/E
nd
Use
r T
arge
ted
Perf
orm
ance
Mile
ston
e LI
I #
Sy
stem
-wid
e In
cide
nt-r
epor
ting
:
Upgr
ade
of t
he 2
7-ye
ar o
ld
tech
nolo
gy o
f th
e A
SRS
data
base
to
incl
ude:
con
vers
ion
of A
SRS
lega
cy
data
base
to
ORA
CLE;
ele
ctro
nic
subm
issi
on o
f re
port
s; a
nd t
est
and
eval
uatio
n an
ana
lyst
dec
isio
n-su
ppor
t sy
stem
Data
base
co
nver
ted
to
ORA
CLE,
har
dwar
e an
d so
ftw
are
to
perm
it el
ectr
onic
re
port
sub
mis
sion
an
d te
stin
g of
an
alys
t ad
viso
r sy
stem
.
Elec
tron
ic r
epor
t su
bmis
sion
and
an
alys
t's
wor
kben
ch a
re u
sed
in d
ay-
to-d
ay o
pera
tion
of p
roce
ssin
g A
SRS
repo
rts.
App
licat
ions
to
othe
r re
port
ing
syst
em (
e.g.
, ASA
P re
port
s an
d NA
OM
S su
rvey
) ar
e de
mon
stra
ted.
FAA
, airl
ine
oper
ator
s, fl
ight
cr
ews,
cab
in c
rew
s,
mec
hani
cs, A
TC, a
nd
rese
arch
ers
in
avia
tion
safe
ty a
nd
hum
an f
acto
rs
Impr
ove
colle
ctio
n an
d ut
iliza
tion
ad
hoc
anec
dota
l saf
ety
repo
rts
from
fro
nt-
line
NAS-
wid
e pe
rson
nel
#1.2
&
#1.1
1
Fast
-tim
e Si
mul
atio
n of
Sy
stem
-wid
e Ri
sks:
Ri
goro
usly
va
lidat
ed s
yste
m-w
ide
mod
els
and
sim
ulat
ions
of
rela
tions
hip
amon
g el
emen
ts o
f th
e NA
S to
sup
port
pr
edic
tions
and
saf
ety-
risk
asse
ssm
ents
of s
yste
m-w
ide
effe
cts
of n
ew fl
ight
and
ATC
tec
hnol
ogie
s an
d/or
pro
cedu
res
befo
re t
hey
are
inse
rted
into
the
ope
ratin
g en
viro
nmen
t. I
nclu
des
engi
neer
ing
mod
els,
ope
ratin
g co
ncep
t m
odel
s,
supp
ort/
logi
stic
s m
odel
s, h
uman
pe
rfor
man
ce m
odel
s an
d ris
k an
alys
es
A h
iera
rchy
of
mat
hem
atic
al
mod
els
Stru
ctur
al a
nd c
asua
l pre
dict
ions
of
risk
are
bein
g pr
oduc
ed.
Risk
as
sess
men
ts a
re e
valu
ated
aga
inst
in
cide
nt a
nd in
tram
ural
dat
abas
es f
or
spec
ific
"ris
k fa
ctor
s an
d co
ntex
ts".
A
mea
sure
of
pred
ictiv
e ac
cura
cy
(i.e.
, cor
rela
tion
betw
een
mod
el a
nd
NASA
per
form
ance
) of
.60
or b
ette
r is
cal
cula
ted
for
mod
el's
pe
rfor
man
ce
FAA
, NA
SA a
nd
rese
arch
ers
in
avia
tion
safe
ty a
nd
hum
an f
acto
rs
Abi
lity
to a
sses
s th
e sy
stem
-wid
e sa
fety
im
pact
of n
ew
tech
nolo
gies
or
proc
edur
es b
efor
e th
ey a
re
impl
emen
ted
#1.4
&
#1.1
1
Prot
oty
pe S
yste
m-w
ide
Risk
A
sses
smen
t C
apab
ility
: A
ca
pabi
lity
that
dem
onst
rate
s th
e fe
asib
ility
and
val
ue o
f au
tom
atic
ally
m
ergi
ng d
e-id
entif
ied
disp
arat
e da
ta
sour
ces
to a
sses
s sy
stem
-wid
e sa
fety
ris
ks
PC-b
ased
so
ftw
are,
do
cum
enta
tion,
tr
aini
ng a
nd
guid
elin
es f
or it
s ut
iliza
tion
At
leas
t 2
maj
or a
ir ca
rrie
rs a
nd t
he
ATC
are
suf
ficie
ntly
sat
isfie
d w
ith
valid
ity o
f pr
edic
tions
to
be u
sing
th
e A
SMM
too
ls f
or c
ausa
l ana
lysi
s an
d ris
k as
sess
men
t an
d fo
r fa
st-
time
sim
ulat
ions
to
aid
eval
uatio
ns o
f pr
opos
ed in
terv
entio
ns.
FAA
; All
pers
onne
l in
the
avia
tion
indu
stry
in
clud
ing
fligh
t cr
ews,
ca
bin
crew
s m
echa
nics
, te
chni
cian
s, A
TC,
airp
ort
oper
atio
ns
and
rese
arch
ers
in
avia
tion
safe
ty a
nd
Crea
tion
of c
apab
ility
to
(1)
pro
vide
de
cisi
on m
aker
s w
ith
relia
ble
info
rmat
ion
on s
afet
y of
NA
S;
(2)
iden
tify
caus
al
fact
ors,
acc
iden
t pr
ecur
sors
, and
off
-no
min
al c
ondi
tions
;
#1.8
&
#1.1
1
47
hum
an f
acto
rs
(3)
enab
le a
nd
enco
urag
e sh
arin
g in
form
atio
n
Avi
atio
n Pe
rfor
man
ce M
easu
ring
Sy
stem
(A
PMS)
Too
ls:
APM
S is
an
inte
grat
ed s
uite
of
tool
s to
fa
cilit
ate
impl
emen
tatio
n of
rou
tine
fligh
t-da
ta a
naly
ses
with
in e
ach
air-
serv
ice
prov
ider
. A
PMS
deve
lops
and
do
cum
ents
the
sof
twar
e an
d pr
oced
ures
for
dat
a m
anag
emen
t an
d an
alys
es o
f fli
ght-
reco
rded
dat
a th
at e
nabl
e us
ers
to in
terp
ret
impl
icat
ions
in s
afet
y an
d ef
ficie
ncy
of fl
ight
PC-b
ased
so
ftw
are,
do
cum
enta
tion,
tr
aini
ng a
nd
guid
elin
es f
or it
s ut
iliza
tion
At
leas
t 2
maj
or a
ir ca
rrie
rs a
re
prov
idin
g FO
QA
dat
a fo
r m
ergi
ng
into
a c
omm
on d
atab
ase.
Sub
ject
-m
atte
r ex
pert
s ac
cept
and
use
in
dica
tions
of
syst
em-w
ide
safe
ty
issu
es f
rom
ana
lyse
s of
the
da
taba
se.
Air
carr
iers
and
ge
nera
l avi
atio
n op
erat
ors
- all
func
tiona
l dis
cipl
ines
, in
clud
ing
fligh
t op
erat
ions
, tra
inin
g,
engi
neer
ing
and
mai
nten
ance
Enab
le e
ach
user
to
impl
emen
t a
polic
y of
pro
activ
e m
anag
emen
t by
m
onito
ring
and
anal
yzin
g la
rge
mas
ses
of fl
ight
-re
cord
ed d
ata
on a
co
ntin
uous
bas
is
#1.2
&
#1.
7 &
#1
.8 &
#
1.11
Perf
orm
ance
Dat
a A
naly
sis
and
Repo
rtin
g S
yste
m (
PDA
RS)
Too
ls:
Pro
vide
s th
e ca
pabi
lity
to:
colle
ct, e
xtra
ct, a
nd p
roce
ss A
TC
oper
atio
nal d
ata;
com
pute
qu
antit
ativ
e op
erat
iona
l per
form
ance
m
easu
res
on a
reg
ular
bas
is r
elat
ing
to s
afet
y, d
elay
, fle
xibi
lity,
pr
edic
tabi
lity
and
user
acc
essi
bilit
y;
cond
uct
caus
al a
naly
ses
and
oper
atio
nal p
robl
em id
entif
icat
ion
and
anal
yses
; acc
ess
desi
gn a
nd
sim
ulat
ion
tool
s fo
r "w
hat-
if"
anal
yses
and
for
iden
tific
atio
n an
d em
ulat
ion
of s
yste
m im
prov
emen
t op
tions
; ach
ieve
per
form
ance
st
atis
tics
and
basi
c op
erat
iona
l dat
a fo
r us
e in
res
earc
h de
velo
pmen
t an
d
Hard
war
e: P
Cs,
prin
ters
, tap
e dr
ives
, des
ktop
so
ftw
are;
Ne
twor
king
: Ro
uter
s, h
ub,
switc
hes,
DSL
&
T1 li
nes;
Tap
s: t
o TR
ACO
Ns a
nd
Cent
ers;
Sof
twar
e:
tap
clie
nt, s
erve
r cl
ient
mod
ules
, ce
ntra
l mer
ge
proc
ess,
site
ad
apta
tion;
Use
r m
ater
ials
: Cu
stom
ized
sa
mpl
e re
port
Acc
epta
nce
by t
he A
TC c
omm
unity
an
d ex
tens
ion
to o
ther
sec
tors
of
natio
nal A
TC.
At
leas
t 14
fac
ilitie
s ar
e ac
tivel
y pa
rtic
ipat
ing
in
eval
uatio
n, a
nd p
lans
for
na
tiona
lizat
ion
are
mad
e by
the
FA
A
FAA
Co
ntin
uous
, rou
tine
mon
itorin
g of
pe
rfor
man
ce m
etric
s to
ena
ble
the
impl
emen
tatio
n of
a
polic
y of
pro
activ
e NA
S m
anag
emen
t
#1.1
&
#1.2
&
#1.8
&
#1.9
&
#1.1
1
TAB
LE 4
.2
ASM
M P
ROD
UCT
S
(CO
NTI
NU
ED)
48
plan
ning
stu
dies
sc
ripts
, hel
p in
form
atio
n an
d m
anua
ls, t
rain
ing
Nat
iona
l A
viat
ion
Sys
tem
O
pera
tion
al M
onit
orin
g S
yste
m
(NA
OM
S):
Aim
s at
the
per
man
ent
field
impl
emen
tatio
n of
NA
OM
S re
spon
sibl
e fo
r de
velo
ping
and
m
aint
aini
ng a
com
preh
ensi
ve a
nd
cohe
rent
sur
vey
of t
he s
afet
y an
d pe
rfor
man
ce o
f th
e NA
S fr
om t
he
pers
pect
ive
of f
ront
line
per
sonn
el
NAS-
wid
e. i
t is
a p
roac
tive
com
pani
on t
o th
e ad
hoc
sub
mitt
al
proc
ess
embo
died
with
in A
SRS
Surv
ey t
ool a
nd
valid
ated
m
etho
dolo
gy
Cont
inue
d po
sitiv
e re
spon
se f
rom
all
of t
he s
olic
ited
com
mun
ities
. A
t le
ast
60%
inte
rvie
w c
ompl
etio
n ra
te
FAA
; All
pers
onne
l in
the
avia
tion
indu
stry
in
clud
ing
fligh
t cr
ews,
ca
bin
crew
s m
echa
nics
, te
chni
cian
s, A
TC,
airp
ort
oper
atio
ns
and
rese
arch
ers
in
avia
tion
safe
ty a
nd
hum
an f
acto
rs
Crea
tion
of a
m
echa
nism
for
m
easu
ring
the
over
all s
afet
y of
the
NA
S in
a
quan
titat
ive,
pre
cise
an
d re
peat
able
way
on
an
ongo
ing
basi
s
#1.
2 &
#1
.3 &
#1
.10
49
From another perspective, the matrix of Figure 4.2(B) portrays the relationships of the products of the ASMM sub-project to the four primary steps of the process of proactive management of safety risk. The key attribute of all of these products is to facilitate efficient and insightful analyses of all relevant data to identify causal factors, accident precursors, and unsuspected features in the data collected pertaining to the health, performance, and safety of operations in the NAS. Consequently, as indicated in Figure 4.2(B), NASA’s role in the process is predominantly to aid IDENTIFYING and EVALUATING. For the most part, it is up to the experts in industry and the FAA to FORMULATE and IMPLEMENT the interventions. However, it is in these latter two steps that ASMM developments in modeling and simulations will aid in predicting system-wide effects of proposed changes and developments in information sharing will aid the collaborative decision-making process.
FIGURE 4.2(B) RELATIONSHIPS OF ASMM PRODUCTS TO THE PROACTIVE MANAGEMENT PROCESS
50
4.3 METRICS 4.3.1 Safety Goal Metrics Assessment of the safety goal impact of Aviation System-wide Monitoring will be evaluated in three ways.
• The first method will track progress toward raising the TRL of the ASMM project’s products.
• The second method will be the overall degree of aviation community penetration of the ASMM products. The success levels are specified in the exit criteria for each ASMM product.
• The third method will be the degree of system-wide integration accomplished amongst the ASMM products. This integration provides both system-wide analysis capabilities as well as the feedback & validation capabilities that will be crucial for this unique capability.
- Minimum success will be the demonstration of heterogeneous integration of
ASRS, APMS, NAOMS, and PDARS to assess specific operational issues of the NAS.
- Full project success will include system-wide predictions that are validated by the monitoring and data analysis tools.
4.3.2 Project Success Metrics Assessment of the success of the Aviation System Monitoring & Modeling (ASMM) will be primarily by tracking progress toward raising the TRL of the project’s products. A secondary metric, where applicable will measure a technology’s ability (at any given TRL) to meet the element’s target performance goals. Success of the ASMM sub-project will be based on four additional criteria:
• Ability to meet Project milestones within cost, schedule and resources. • Ability for ASMM enabling technologies to achieve Technology Readiness Levels
(TRL) as shown in Tables 4.1(A) through 4.1(E). • Ability of ASMM enabling technologies to sufficiently impact AvSP goals. • Ability of ASMM enabling technologies to be implemented into the aviation
community. Project and Element milestones will be tracked by the AvSP on a monthly basis as well as cost, schedule and technical progress down to the sub-element level. Each Element milestone will be tied into a Project milestone. Project status will be evaluated as follows:
a. Major Problem Schedule: Milestone delayed more than 1 quarter Cost: More than +/- 15% of planned cost Technical: Problems meeting most of exit criteria
b. Minor Problem Schedule: Milestone delayed between 1 and 4 months Cost: Between 10-15% of planned cost Technical: Problems meeting some exit criteria
c. No Known Problem Schedule: On (<1 month) or ahead of baseline schedule
51
Cost: Within +/-10% of planned cost Technical: Meeting or exceeding exit criteria
The TRL levels will be tracked by the Project Manager on a quarterly basis to ensure ASMM product maturity is obtained based on the projected implementation schedule maintained by the AvSP Program Assessment Team. Impact assessments on accident categories will be performed by the AvSP Program Assessment Team on a periodic basis. 4.4 IMPLEMENTATION STRATEGY Figure 4.4 reflects the implementation strategy of all of the capabilities developed under ASMM leading to the overall ASMM goal to support proactive management of Aviation Safety.
FIGURE 4.4 ASMM IMPLEMENTATION STRATEGY
Proactive Risk Management is implemented in an iterative loop of:
1. Identify 2. Evaluate 3. Formulate 4. Implement (and then continuous, iterative refinements of steps 1-4).
The relationship of the activities and developments under the ASMM sub-project to this concept of the cycle of proactive management of safety risk of Figure 4.4 is conveyed in Figure 4.5.
IDENTIFY
EVALUATE
FORMULATE
IMPLEMENT
PROACTIVE
RISK
MANAGEMENT
52
FIGURE 4.5 ASMM AND THE CYCLE OF PROACTIVE MANAGEMENT OF RISK
Identification is the process of continuously monitoring and comparing to expectations to uncover potential risks and to recognize improvements. This accomplished in a parallel bottom-up and top-down approach: Intramural monitoring:
- FOQA/APMS – air carriers, ALPA, FAA, NASA, Teledyne Controls, Austin Digital, SFIM Inc., SimAuthor, etc.
- ASAP – air carriers, FAA, APA, ALPA - PDARS – FAA, NASA, NATCA
Extramural monitoring:
ASRS – FAA, NASA, the US aviation community SAPS – FAA NASDAC – FAA GAIN – FAA, the international aviation community SPAS – FAA NAOMS - NASA
Evaluation involves the diagnosis of causal conditions, the quantification of frequency, and the assessment of severity. The following data analysis tools will be used to meet the evaluation requirements:
- Automated data quality checking and filtering - Database storage & middleware technologies - Automated pattern-search tools - Automated extraction of information from textual reports - Automated linkage of relevant information from heterogeneous data sources - Automated advisors of significant factors in numeric & textual data
Developments in these areas will require significant coordination, validation, and calibration efforts among all participating research organizations, government agencies, international and industry efforts.
¬ Identify
¬ Data Analysis Tools ¬ Evaluate
¬ Modeling & Simulations ¬ Formulate
¬ Feedback to evaluate interventions
¬ Intramural Monitoring •PDARS (Performance Data Analysis and Reporting System) •APMS (Aviation Performance Measuring System)
¬ Extramural Monitoring •NAOMS (National Aviation Operations Monitoring Service)
53
Formulation is the consideration of changes, the assessment of safety risk, and the estimation of benefits and costs of intervention. This process is fundamentally the responsibility of the Industry and the FAA and entails more than just the safety-risk assessment that is the focus of ASMM. However, formulation of an intervention will rely on the following tools:
- Information visualization tools - FAA, NASA, International & Industry Community - Fast-time simulations to evaluate safety intervention strategies – NASA & FAA
Implementation is also the responsibility of the Industry and the FAA with the ASMM tools playing only a supportive role in the decision-making process. It is accomplished via prototypes, their effectiveness is evaluated, refinements are implemented, and then full-scale deployments are facilitated. The implementation of the ASMM tools per se to support the decision makers throughout this cycle of proactive management is addressed throughout the process of their development. Requirements for data analysis tools start from the results of a User-needs Study conducted with each potential user in each operational environment (i.e., air carriers or ATC). The prototypes of the required capabilities for data analyses are developed under Data Analysis Tools Development. Collaboration is established with the potential user under a Space Act Agreement (SAA) to test and evaluate the prototypes in the operational environment under Intramural Monitoring. This initiates an iterative process during which the user becomes sufficiently familiar with the capability to suggest revisions or enhancements. This evolutionary process results in the customized product to meet the individual user’s needs. In the case of an air carrier, the user may already have an on-going exceedance-based FOQA program in which case the flight-recorded data are being downloaded and processed by a commercial vendor of that capability. We enter into a separate SAA with that vendor so that, if and when the air carrier and the vendor agree that the capability should be commercialized, we work with that vendor to transfer the technology. In the case of ATC data analysis tools, the primary user is the FAA and the PDARS program is already being carried out in collaboration with the FAA Office of System Capacity Requirements and the FAA Office of System Safety. From FY’00 through the present, PDARS is being tested and evaluated in the facilities of the ATC Western-Pacific Region. During FY’02, the PDARS network was extended to include the facilities of the Southwest Region. During FY’03, PDARS was extended to the facilities of the Southern ATC Region. In FY’03, the FAA announced the schedule for extending the test and evaluation of PDARS to all 20 ATC Centers in the continental U.S. before the end of FY’04. Once the prototypes have completed these developmental phases, we may consider that the technologies have been effectively transferred and the overall plan for their future evolution and application is primarily an Industry-FAA task. ASMM is also a significant part of the “Aviation Safety Risk Analysis” research area of the FAA/NASA Aviation Safety Research Joint Working Group. Joint roadmaps of FAA and NASA activities in this research area have been submitted for approval by the Co-Chairs of the Working Group. ASMM products have been identified as supporting 15 initiatives in the FAA’s Flight Plan 2004-2008. 5.0 AGREEMENTS NASA does not have sufficient funds to address all the monitoring, data analysis and data sharing issues identified by industry customers. To complete the research picture the ASMM team coordinates with other projects so that the research is complementary and addresses the
54
highest priority issues. The Program manager reviews other’s research projects on a continuing basis. It is critical to partner with other research organizations in order to leverage our separate monitoring and data analysis efforts. Sharing of results, methods and metrics can enhance all research in these areas. Since research is often intrusive to operations, coordination of research logistics can be useful in minimizing disruption to customer operations. 5.1 NASA 5.1.1 Other AvSP Projects ASMM entails capabilities that cut across several of the other AvSP Projects. ASMM is coordinating with the Single Aircraft Accident Prevent (SAAP) project on engine health monitoring. The APMS tool for extracting atypical flights can be adapted to monitoring engine operations for on-condition maintenance. The ASMM Modeling and Simulation tools may well be used to evaluate potential issues of human factors associated with some of the products being developed under the other projects. Additional work is being defined for coordination of data handling, middleware, and standards for operational data usage and analysis for the Weather and Synthetic Vision projects. APMS has focused on major air carriers due to limitations of funding. However, there is a clear recognition of the need to adapt these capabilities to the special requirements of the GA and the Rotorcraft communities.
5.1.2 Other NASA Programs The ASMM sub-project is also coordinating closely with the Advanced Air Transportation Technology (AATT) Project and with the Engineering for Complex Systems (ECS) Program in several areas: modeling, fast-time simulation, air traffic control monitoring and analysis, and metrics and measurements of safety, as well as data management/access issues. 5.2 NON-NASA The plan for ASMM developments is compatible with, and complementary to on-going activities in the US and Europe such as the FAA-Industry FOQA program, the UK CAA OFDM program, the ICAO initiatives, GAIN, Eurocontrol Performance Review Commission, ATAQ/ALPA/APA/NATCA positions and initiatives, etc., as well as NASA’s current tools such as ASRS. The implementation approach will help improve current research activities for APMS, PDARS, NAOMS and various other data analysis tools as well as commercial and industry tools such as BASIS and OASIS. This will be accomplished through regular workshops to share technical issues, implementation results, and ongoing research and development strategies.
5.3 PARTNERS Data Analysis Tools Development, Intramural Monitoring, and Extramural Monitoring:
- Flight Safety Foundation Icarus Committee’s Working Group on Flight Operations Risk Assessment System
- ICASS (International Confidential Aviation Safety Systems) committee - Active participation of IATA and ICAO in the taxonomy workshop - Member of the Global Aviation Information Network (GAIN) Working Group B - Commercial Aviation Safety Team (CAST) - Alaska Airlines
55
- American Airlines - Delta Airlines - Teledyne Controls - SFIM, Inc. - SimAuthor Corp
Modeling & Simulation:
- FAA Tech Center - FAA Office of System Capacity Requirements - FAA Office of System Safety - Air Force (human behavioral modeling) - International Organizations: Eurocontrol, Civil Aviation Authority (CAA), Office
National d'Études et de Recherches Aérospatiales (the French National Aerospace Research Establishment) (ONERA), National Aerospace Laboratory NLR (Netherlands) and German Aerospace Center (DLR)
56
6.0 RESOURCES FUNDING REQUIREMENT 6.1.1 Resources Funding Chart by Project/Element/Center
TABLE 6.1 ASMM SUB-PROJECT FUNDING BREAKDOWN (728-10)
FY 00 FY 01 FY 02 FY 03 FY 04 FY 05 Total
2.1.1 Data Analysis and -10 3.061 3.422 1.593 0.995 1.100 0.450 10.171 Intramural Monitoring ARC 2.633 3.000 1.508 0.995 1.100 0.450 9.686
GRC 0.305 0.321 0.085 0.000 0.000 0.000 0.711 LaRC 0.123 0.101 0.000 0.000 0.000 0.000 0.224
2.1.2 Extramural Monitoring
-20 0.975 1.600 1.739 1.852 1.997 0.900 9.063
ARC 0.975 1.600 1.739 1.852 1.997 0.900 9.063
2.1.3 Modeling and Simulations
-30 0.758 0.872 1.108 1.376 1.431 0.500 6.045
ARC 0.586 0.645 0.825 0.904 0.959 0.500 4.419 GRC 0.172 0.227 0.283 0.472 0.472 0.000 1.626
2.1.4 Information Sharing -40 1.732 0.370 0.000 0.000 0.000 0.000 2.102 ARC 0.960 0.370 0.000 0.000 0.000 0.000 1.330 DFRC 0.772 0.000 0.000 0.000 0.000 0.000 0.772
2.1.5 Intramural Monitoring -50 2.750 2.657 2.538 1.026 8.971 ARC 2.750 2.657 2.538 1.026 8.971
2.1.X Research Support XX 0.000 0.000 0.000 0.000 0.000 4.000 4.000 ARC 0.000 0.000 0.000 0.000 0.000 4.000 4.000
Net Totals 6.526 6.264 7.190 6.880 7.066 6.876 40.802
ARC 5.154 5.615 6.822 6.408 6.594 6.876 37.469 DFRC 0.772 0.000 0.000 0.000 0.000 0.000 0.772 GRC 0.477 0.548 0.368 0.472 0.472 0.000 2.337 LaRC 0.123 0.101 0.000 0.000 0.000 0.000 0.224
Service Activities 2.125 2.947 2.430 4.207 4.326 4.424 20.459 ARC 1.718 1.931 2.146 4.124 4.243 4.424 18.586 DFRC 0.140 0.160 0.000 0.000 0.000 0.000 0.300 GRC 0.159 0.197 0.217 0.083 0.083 0.000 0.739 LaRC 0.108 0.659 0.067 0.000 0.000 0.000 0.834
Gross Totals 8.651 9.211 9.620 11.087 11.392 11.300 61.261 ARC 6.872 7.546 8.968 10.532 10.837 11.300 56.055 DFRC 0.912 0.160 0.000 0.000 0.000 0.000 1.072 GRC 0.636 0.745 0.585 0.555 0.555 0.000 3.076 LaRC 0.231 0.760 0.067 0.000 0.000 0.000 1.058
Note: Funding available for procurement in FY’05 had not been defined as of the date of this version of the Project Plan pending resolution of full cost accounting. The milestones and deliverables scheduled for FY’05 are based on current expectations.
57
6.1.2 Acquisition Strategy Plan
FIGURE 6.1.2 ASMM SUB-PROJECT RESOURCES
The ASMM sub-project is using a variety of procurement vehicles to address acquisition requirements. These vehicles include:
- NASA Research Agreements (NRA’s) - In-House Contractors - Memorandum of Understanding (MOU)/Memorandum of Agreement (MOA) with
other agencies and research organizations - University Grants
The vast majority of ASMM procurements are software development oriented in nature. Most hardware components are on the GSA schedule. Very few items require sole source justification and there is no significant customized hardware development. GRC and DFRC used existing contracts and procurement vehicles on related programs to simplify development and procurement of unique hardware and software during their participation in the ASMM Sub-Project.
ASMM/Resource Estimates Assumptions
Assumptions
• Strong Industry Participation in Data Access
o Providing key controlled access to sensitive data
• No new major facilities required • Resource Planning based on
existing programs, e.g.: o APMS & ASRS o AATT Programs: CTAS, SMA,
AvSTAR o DARWIN, NPSS, ECS
• FAA Collaboration o Coordinating with Tech
Center, Safety & Capacity Offices, and Human Factors Research
o ASMM intramural development approach
o minimized funding issues with the FAA
o jointly developed roadmaps
Out-of-House $12M
Program Sup $20
In-House $24M
Industry
ASMM Gross - $57M
Universities - $1M
58
6.2 WORKFORCE 6.2.1 Workforce Chart by element/center
TABLE 6.2.1(A) ASMM 728-10 CIVIL SERVANTS WORKFORCE (DIRECT) (FTE)
FY 00 FY 01 FY 02 FY 03 FY 04 FY 05 2.1.1 Data Analysis Tools Development
-10 2.6 4.7 2.0 2.0 2.0 1.0
ARC 1.6 3.7 2.0 2.0 2.0 1.0 GRC 0.5 0.5 0.0 0.0 0.0 0.0 LaRC 0.5 0.5 0.0 0.0 0.0 0.0 2.1.2 Extramural Monitoring -20
1.5 2.5 2.5 2.5 2.5 1.5
ARC 1.5 2.5 2.5 2.5 2.5 1.5 2.1.3 Modeling and Simulation -30
0.7 1.3 1.1 1.1 1.1 1.0
ARC 0.7 0.8 1.1 1.1 1.1 1.0 GRC 0.5 0.5 0.0 0.0 0.0 0.0 2.1.4 Information Sharing -40
3.1 4.6 0.0 0.0 0.0 0.0
ARC 1.5 4.6 0.0 0.0 0.0 0.0 DFRC 1.6 0.0 0.0 0.0 0.0 0.0 2.1.5 Intramural Monitoring -50
3.0 3.0 3.0 1.5
ARC 3.0 3.0 3.0 1.5 TOTAL 8.4 13.1 8.6 8.6 8.6 5.0 ARC 5.3 11.6 8.6 8.6 8.6 5.0 DFRC 1.6 0.0 0.0 0.0 0.0 0.0 GRC 1.0 1.0 0.0 0.0 0.0 0.0 LaRC 0.5 0.5 0.0 0.0 0.0 0.0
59
TABLE 6.2.1(B) ASMM 728-10 PERFORMANCE BASED CONTRACTORS (PBCs)
FY 00 FY 01 FY 02 FY 03 FY 04 FY05
-10 15.5 17.5 12 8.5 8.5 4.0 2.1.1 Data Analysis and
Intramural Monitoring ARC 7.0 10.0 5.0 5.0 5.0 2.5
Rannoch 2.0 2.0 2.0 1.5 1.5 0.0
Sandia 2.0 2.0 2.0 1.5 1.5 1.5
FSF 1.5 1.5 0.0 0.0 0.0 0.0
Naval Research Lab 0.5 0.5 1.0 0.5 0.5 0.0
VPI 1.0 0.0 0.0 0.0 0.0 0.0
Univ of Alabama 1.0 0.0 0.0 0.0 0.0 0.0
Ultimode 1.0 0.0 0.0 0.0 0.0 0.0
Oregon Graduate Institute 0.0 1.0 1.0 0.0 0.0 0.0
SUNY Stoneybrook 0.0 1.0 1.0 0.0 0.0 0.0
2.1.2 Extramural Monitoring -20 5.0 9.0 9.0 9.0 9.0 6.0
ARC 5.0 9.0 9.0 9.0 9.0 6.0
2.1.3 Modeling &Simulations -30 6.0 6.0 6.5 6.0 5.0 6.0
ARC 1.5 1.5 1.5 1.0 1.0 1.5.
SJSU 1.0 1.0 1.5 1.5 1.5 1.5
ATAC 1.5 1.5 1.5 1.5 1.5 2.0
GA TECH 1.0 1.0 1.0 1.0 1.0 1.0
NLR 1.0 1.0 1.0 1.0 0.0 0.0
2.1.4 Information Sharing -40 5.2 5.2 0.0 0.0 0.0 0.0
ARC 5.2 5.2 0.0 0.0 0.0 0.0
2.1.5 Intramural Monitoring -50 0.0 0.0 9.0 8.0 8.0 6.0
ARC 0.0 0.0 9.0 8.0 8.0 6.0
TOTAL PBCs 30.7 36.7 36.5 31.5 30.5 22.0
ARC 18.7 25.7 24.5 23.0 23.0 16.0
Rannoch 2.0 2.0 2.0 1.5 1.5 0.0
Sandia 2.0 2.0 2.0 1.5 1.5 1.5
FSF 1.5 1.5 0.0 0.0 0.0 0.0
Naval Research Lab 0.5 1.0 1.0 0.5 0.5 0.0
VPI 1.0 0.0 0.0 0.0 0.0 0.0
Univ of Alabama 1.0 0.0 0.0 0.0 0.0 0.0
Ultimode 1.0 0.0 0.0 0.0 0.0 0.0
Oregon Graduate Institute 0.0 1.0 1.0 0.0 0.0 0.0
SUNY Stoneybrook 0.0 1.0 1.0 0.0 0.0 0.0
SJSU 1.0 1.0 1.5 1.5 1.5 1.5
ATAC 1.5 1.5 1.5 1.5 1.5 2.0
GA TECH 1.0 1.0 1.0 1.0 1.0 1.0
NLR 1.0 1.0 1.0 1.0 0.0 0.0
60
6.3 FACILITIES USAGE CHARTS 6.3.1 Graphic
TABLE 6.3.1
ASMM FACILITY REQUIREMENTS (utilization duration TBD)
FY 00 FY01 FY02 FY03 FY04 FY05 FACILITY CTR SCV
ACT. 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
Future Flight Central (FFC) ARC TBD
Numeric Aerodynamic
Simulator ARC TBD
CVSRF ARC TBD
100% 41-60% 81-99% 21-40% 61-80% 0-20% TBD - data were unavailable
Note: Although their use may have been contemplated at the start of the project, ASMM now foresees no requirement for utilization of these or any other facility. 6.3.2 Test and Verification of Technology Each ASMM element may have particular needs for test and verification. In some cases, hardware, software, or integration system testing, or some combination, is an output or significant component of the ASMM element. Test and verification plans will be developed and referenced in the ASMM sub-project or element plans. Most of the products of ASMM rely on collaborations with the user communities to test and evaluate them in the operational environment. In those cases, established processes will be applied to the components under development. In the few situations in which the indicated facilities may be used, these processes will be monitored by the Office of Safety, Environment, and Mission Assurance at NASA/ARC.
61
7.0 TECHNOLOGY TRANSFER/COMMERCIALIZATION The ASMM sub-project emphasizes rapid and effective dissemination of the technology to the U.S. industry. The primary ASMM strategy for technology transfer is to involve partners as participants in the development of the new technologies. Requirements for data analysis tools start from the results of a User-needs Study conducted with each potential user in each operational environment (i.e., air carriers or ATC). Collaboration is established with the potential user under a Space Act Agreement (SAA) to test and evaluate the prototypes in the operational environment. This initiates an iterative process during which the user becomes sufficiently familiar with the capability to suggest revisions or enhancements. This evolutionary process results in the customized product to meet the individual user’s needs. In the case of an air carrier, the user may already have an on-going exceedance-based FOQA program in which case the flight-recorded data are being downloaded and processed by a commercial vendor of that capability. We enter into a SAA with that vendor so that when the air carrier and the vendor agree that the capability should be commercialized we work with that vendor to transfer the technology. In the case of ATC data, the primary user will be the FAA and the PDARS program is already being carried out in collaboration with the FAA Office of System Capacity. PDARS is being evaluated in the FAA’s air traffic control facilities. Once the prototypes have completed these developmental phases, we consider that the technologies have been effectively transferred and the plan for their future use and evolution is primarily an Industry-FAA task. NASA inventions have been and will be disclosed as such and patents or copyrights applied for as appropriate. Non-exclusive licensing agreements will be pursued with current vendors of similar software. APMS tools, for example, are likely to be of interest to the vendors of software in support of current FOQA exceedances-based programs. Milestones in FY’04-’05 are to identify such vendors and transfer the technologies under licensing agreements. Some technology exists in Department of Defense (DOD) applications; the feasibility of incorporation of similar technology into the civil fleet will be evaluated with the civil partners. In addition, transfer of technology, such as PDARS, is being coordinated with related FAA research activities and will be further facilitated by testing in FAA facilities under representative operational conditions. Finally, development of new technology and data will be distributed to industry via reports, society conferences, and workshops. 8.0 PRODUCT ASSURANCE A plan has been established by the project manager for proactive decision making to continually identify, analyze, plan, track, control, and communicate risk within the elements. This plan will follow the guidelines of NASA Programs and Project Management Processes and Requirements document, NPG 7120.5A. The plan will utilize the AvSP common process, methods and tools, in order to establish a disciplined approach. The Offices of Safety & Assurance Technology (OSAT) trained personnel from ASMM sub-project in the Continuous Risk Management process during FY2000 and FY2001.
8.1 DE-SCOPE METHODOLOGY Project Level Strategies
62
Phase I - Extend or de-scope ASMM sub-project milestones Phase II - Reduce the number of target users for given element implementation
Element Level Strategies
Phase III - Reduce the scope of the Information Sharing Element Phase IV - Reduce the scope and duration of NAOMS development and deployment Phase V - Reduce the scope and duration of APMS development and deployment Phase VI - Reduce the scope of the Modeling and Simulation Element Phase VII - Reduce the scope and duration of PDARS development and deployment Phase VIII - Discontinue the Modeling & Simulation Element Phase IX - Discontinue ASMM project
8.2 RISK MITIGATION
Risk Management is a structured, continuous process for proactive AvSP decision making to identify program risks (i.e., what could go wrong), prioritize these risks to determine which need to address, and implement strategies to deal with unacceptable risks. For the AvSP, risk management and control requirements are tailored to each Project. Each Center S&MA Office will be an active participant with the projects and provide risk management guidance.
The primary risk categories for AvSP are: (1) technical—critical enabling technologies encountering unexpected developmental difficulties, (2) physical risk to personnel and property, and (3) programmatic—resources unavailable because of competing technology development priorities. The AvSP has been planned with no program reserve, which increases the technical and programmatic risks to each of the elements. Each Element Manager will identify and prioritize risks and document them in the applicable element plans. Several potential risks and mitigation strategies have been identified for the ASMM sub-project (see Table 8.2). These risks will be incorporated into the element risk identification process.
63
Pr
oduc
t and
Haz
ard
Prob
abili
ty o
f O
ccur
renc
e C
onse
quen
ces o
f O
ccur
renc
e M
itiga
tion
Stra
tegy
an
d A
ctio
n Im
plem
enta
tion
Prob
abili
ty
AV
IATI
ON
PER
FOR
MA
NC
E M
EASU
RIN
G S
YST
EM (A
PMS)
: APM
S is
an
inte
grat
ed su
ite o
f too
ls to
faci
litat
e im
plem
enta
tion
of ro
utin
e fli
ght-d
ata
anal
yses
with
in e
ach
air-
serv
ice
prov
ider
. A
PMS
deve
lops
and
doc
umen
ts th
e so
ftwar
e an
d pr
oced
ures
for d
ata
man
agem
ent a
nd a
naly
ses o
f flig
ht-r
ecor
ded
data
that
ena
ble
user
s to
inte
rpre
t im
plic
atio
ns in
safe
ty a
nd e
ffic
ienc
y of
flig
ht
V
ery
Hig
h: S
ome
APM
S to
ols
are
alre
ady
in d
aily
us
e. P
artn
ers
have
show
n in
tere
st in
impl
emen
ting
mos
t APM
S ad
vanc
ed to
ols.
V
endo
rs c
omm
erci
aliz
ed
two
APM
S pr
oduc
ts a
nd a
re
inte
rest
ed in
oth
ers.
FAA
has
mad
e it
man
dato
ry th
at a
ir ca
rrie
rs w
ith F
OQ
A
prog
ram
s sub
mit
sum
mar
y FO
QA
dat
a to
the
FAA
. A
ir ca
rrie
rs a
re c
autio
usly
con
tinui
ng F
OQ
A p
rogr
ams w
hile
ne
gotia
ting
the
requ
irem
ent.
The
y fe
ar th
at th
ese
data
may
be
misu
sed
by F
AA
or b
y m
edia
or b
y la
wye
rs (i
f acc
esse
d un
der F
OIA
by
subp
oena
). P
ilots
, uni
ons,
and
airli
nes a
re
conc
erne
d th
e FA
A m
ay u
se th
ese
data
to in
vest
igat
e in
divi
dual
pilo
ts or
airl
ines
for v
iola
tions
of F
AR
s.
HIG
H: F
AA
has
iss
ued
its F
OQ
A
rule
that
mak
es it
m
anda
tory
for a
ir ca
rrie
rs w
ith F
AA
-ap
prov
ed F
OQ
A
prog
ram
s to
prov
ide
aggr
egat
e da
ta
It ha
s bee
n di
ffic
ult
for A
PMS
to g
ain
acce
ss to
flig
ht-
reco
rded
dat
a ev
en
thou
gh it
cur
rent
ly
oper
ates
with
de-
iden
tifie
d da
ta
behi
nd e
ach
air
carr
ier’
s fire
wal
ls.
Cont
inue
to sh
ow a
ir ca
rrie
rs
the
feas
ibili
ty a
nd v
alue
of
rout
inel
y m
onito
ring
and
anal
yzin
g de
-iden
tifie
d fli
ght d
ata
to a
id in
pro
activ
e m
anag
emen
t of s
afet
y ris
k.
FAA
’s n
ew F
OQ
A ru
le
prov
ides
pro
tect
ion
for
indi
vidu
al a
irmen
. FA
A
spon
sors
mee
tings
am
ong
FOQ
A a
ir ca
rrie
rs to
en
cour
age
exch
ange
of
info
rmat
ion.
.
NA
SA m
ay n
ot b
e ab
le to
acc
ess a
nd m
erge
the
info
rmat
ion
obta
ined
by
mul
tiple
air
carr
iers
from
ro
utin
ely
anal
yzin
g fli
ght d
ata
beca
use
of c
once
rns f
or
FOIA
subp
oena
and
pos
sibl
e m
isuse
of t
he in
form
atio
n.
MED
IUM
: Air
carr
iers
are
ex
pres
sing
con
cern
sin
ce th
e iss
uanc
e of
the
FAA
rule
on
FOQ
A d
ata.
ASM
M c
ould
not
ac
hiev
e its
prim
ary
obje
ctiv
e of
ena
blin
g pr
oact
ive
syst
em-
wid
e sa
fety
risk
m
anag
emen
t.
Dem
onstr
ate
the
valu
e of
sy
stem
-wid
e m
onito
ring.
Co
ntin
ue to
bui
ld tr
ust w
ith
airli
nes a
nd u
nion
s in
NA
SA
as a
n “o
bjec
tive
brok
er”
of
data
.
NA
SA m
ay b
e lim
ited
in th
e si
ze o
f the
dat
abas
es to
whi
ch
it is
giv
en a
cces
s eith
er b
ecau
se o
f the
lim
itatio
ns o
f fle
et
type
, siz
e an
d op
erat
ions
of c
urre
nt c
olla
bora
ting
air
carr
iers
or b
ecau
se o
f con
stra
ints
on
mer
ging
dat
a fr
om
mul
tiple
air
carr
iers
.
LOW
: Ala
ska,
A
mer
ican
, and
D
elta
Airl
ines
hav
e pr
ovid
ed a
cces
s to
adeq
uate
dat
abas
es.
We
coul
d no
t de
mon
strat
e ab
ility
to
ext
end
data
base
-m
inin
g to
ols t
o la
rge
data
base
s.
Cont
inue
to w
ork
tact
fully
w
ith th
e se
vera
l air
carr
iers
w
ho se
em so
mew
hat
rece
ptiv
e to
mer
ging
thei
r de
-iden
tifie
d da
taba
ses.
64
Prod
uct a
nd H
azar
d Pr
obab
ility
of
Occ
urre
nce
Con
sequ
ence
s of
Occ
urre
nce
Miti
gatio
n St
rate
gy
and
Act
ion
Impl
emen
tatio
n Pr
obab
ility
PE
RFO
RM
AN
CE
DA
TA A
NA
LYSI
S A
ND
R
EPO
RTI
NG
SY
STEM
(PD
AR
S): P
rovi
des t
he
capa
bilit
y to
: col
lect
and
pro
cess
ATC
ope
ratio
nal d
ata;
co
mpu
te q
uant
itativ
e op
erat
iona
l per
form
ance
mea
sure
s on
a re
gula
r bas
is re
latin
g to
est
ablis
hed
FAA
met
rics;
co
nduc
t ope
ratio
nal p
robl
em id
entif
icat
ion
and
caus
al
anal
yses
; acc
ess s
imul
atio
n to
ols
for a
naly
ses o
f sys
tem
im
prov
emen
t opt
ions
; ach
ieve
per
form
ance
sta
tistic
s an
d ba
sic
oper
atio
nal d
ata
for u
se in
rese
arch
dev
elop
men
t and
pl
anni
ng s
tudi
es
V
ery
Hig
h: P
DA
RS
is
bein
g us
ed d
aily
by
faci
litie
s in
Wes
tern
-Pac
ific,
Sou
th &
So
uthw
est R
egio
ns.
Feed
back
is u
nani
mou
sly
enth
usia
stic
. FA
A h
as
anno
unce
d a
sche
dule
for
inst
allin
g PD
AR
S in
the
rem
aini
ng A
TC C
ente
rs.
FAA
may
cho
ose
to u
se in
form
atio
n de
velo
ped
by
PDA
RS to
dis
cipl
ine
cont
rolle
rs.
If th
is oc
curs
, the
N
atio
nal A
ir Tr
affic
Con
trolle
rs A
ssoc
iatio
n (N
ATC
A)
will
take
act
ion
to p
reve
nt a
ny fu
rther
rout
ine
anal
yses
of
ATC
dat
a.
LOW
: FA
A h
as
said
that
this
w
ould
not
hap
pen.
N
ATC
A h
as
signe
d an
ag
reem
ent t
o co
oper
ate.
PDA
RS w
ould
not
be
abl
e to
acc
ess d
ata
need
ed to
per
form
m
eani
ngfu
l na
tionw
ide
anal
ysis
With
our
ass
ista
nce,
FA
A
and
NA
TCA
hav
e ex
ecut
ed
a fo
rmal
agr
eem
ent t
hat
PDA
RS is
to b
e us
ed in
a
non-
puni
tive
envi
ronm
ent
and
a cu
lture
of t
rust
es
sent
ial t
o sa
fety
.
FAA
may
cho
ose
to d
eplo
y PD
ARS
nat
iona
lly.
Curr
ent
ASM
M re
sour
ces a
re o
nly
suff
icie
nt to
impl
emen
t and
m
aint
ain
the
PDA
RS e
valu
atio
n in
the
Wes
tern
-Pac
ific
Sout
hwes
t, an
d So
uth
ATC
Reg
ions
.
HIG
H: F
AA
has
m
ade
de fa
cto
deci
sion
to e
xten
d PD
ARS
.
FAA
will
be
resp
onsib
le fo
r de
ploy
men
t afte
r FY
’05
Seek
fund
ing
from
FA
A to
le
ad th
e U
ser-
need
s Stu
dies
in
oth
er A
TC re
gion
s as
need
ed.
Som
e ra
dar t
rack
dat
a ha
ve e
rror
s or d
ropo
uts.
HIG
H
Bad
data
will
skew
re
sults
D
evel
op d
ata-
anal
ysis
algo
rithm
s to
iden
tify
&
elim
inat
e ba
d da
ta
The
data
to b
e pr
oces
sed,
man
aged
, and
stor
ed fr
om th
e en
tire
natio
nal A
TC sy
stem
may
ove
rwhe
lm c
urre
nt
PDA
RS c
apab
ilitie
s for
dai
ly re
porti
ng.
MED
IUM
PD
ARS
wou
ld n
ot
be re
ady
for
natio
naliz
atio
n.
Enco
urag
e FA
A to
pro
vide
th
e fu
ndin
g ne
eded
to d
esig
n a
PDA
RS to
cop
e w
ith
VER
Y la
rge
volu
mes
of d
ata
of a
nat
iona
l sys
tem
.
FAA
’s fo
cus f
or P
DA
RS is
prim
arily
on
capa
city
and
this
may
con
flict
with
the
focu
s of N
ASA
’s A
viat
ion
Safe
ty
Prog
ram
on
safe
ty.
LOW
: Not
ver
y pr
obab
le w
ith th
e cu
rren
t FA
A
parti
cipa
nts
FAA
and
NA
SA
may
div
erge
on
the
obje
ctiv
es o
f PD
ARS
.
Cont
inue
to w
ork
clos
ely
with
bot
h th
e O
ffic
e of
Sy
stem
Cap
acity
and
the
Off
ice
of S
yste
m S
afet
y in
th
e FA
A to
ens
ure
that
bot
h as
pect
s are
app
ropr
iate
ly
addr
esse
d in
PD
ARS
.
TABL
E 8
.2 (
CON
TIN
UED
) TE
CHN
OLO
GY
/IMPL
EMEN
TATI
ON
RIS
KS
AN
D M
ITIG
ATI
ON
STR
ATE
GIE
S
65
Prod
uct a
nd H
azar
d Pr
obab
ility
of
Occ
urre
nce
Con
sequ
ence
s of
Occ
urre
nce
Miti
gatio
n St
rate
gy
and
Act
ion
Impl
emen
tatio
n Pr
obab
ility
N
AS
OPE
RA
TIO
NS
MO
NIT
OR
ING
SE
RV
ICE
(NA
OM
S): A
ims a
t dev
elop
ing
met
hodo
logi
es fo
r a p
erm
anen
t fie
ld im
plem
enta
tion
of
NA
OM
S re
spon
sibl
e fo
r mai
ntai
ning
a c
ompr
ehen
sive
and
cohe
rent
surv
ey o
f the
safe
ty a
nd p
erfo
rman
ce o
f the
NA
S fr
om th
e pe
rspe
ctiv
e of
fron
t lin
e pe
rson
nel N
AS-
wid
e. I
t is
a pr
oact
ive
com
pani
on to
the
ad h
oc su
bmitt
al p
roce
ss
embo
died
with
in A
SRS
H
igh:
The
re is
no
othe
r vi
able
way
to a
ddre
ss th
e ta
rget
ed p
robl
em.
The
reac
tion
of th
e co
mm
unity
to
the
initi
al p
hase
of t
he
surv
ey h
as b
een
very
po
sitiv
e. T
here
is n
o re
ason
to
exp
ect t
hat t
he o
ther
co
nstit
uenc
ies w
ill b
e an
y le
ss re
spon
sive
whe
n th
ey
are
appr
oach
ed.
Both
que
stio
ners
and
inte
rvie
wee
s in
NA
OM
S su
rvey
pr
oces
s hav
e in
tern
al b
iase
s. H
IGH
Bi
ases
may
skew
su
rvey
resu
lts
Subj
ectiv
ity is
inhe
rent
in
surv
eys,
but c
an b
e ta
ken
into
acc
ount
by
deve
lope
d sc
ient
ific
proc
edur
es
Inte
rvie
wee
s in
the
NA
OM
S pr
oces
s are
self-
sele
cted
. H
IGH
M
ay sk
ew a
naly
sis
resu
lts to
war
ds th
ose
mos
t lik
ely
to re
port
prob
lem
s.
This
to c
an b
e ac
coun
ted
for
with
cor
rect
and
pro
ven
scie
ntifi
c pr
oced
ures
.
Out
side
parti
es m
ay p
ress
to g
ain
acce
ss to
NA
OM
S su
rvey
dat
a pr
emat
urel
y.
HIG
H
Resu
lts a
re
mis
inte
rpre
ted
ther
eby
dam
agin
g in
tegr
ity o
f sur
vey
proc
ess
Lim
it ac
cess
to th
e N
AO
MS
surv
ey to
the
NA
OM
S Te
am
of e
xper
ts u
ntil
the
proc
ess
has m
atur
ed.
Out
side
parti
es m
ay p
ress
to g
ain
acce
ss to
NA
OM
S su
rvey
dat
a an
d us
e th
em in
thei
r ow
n an
alys
es
MED
IUM
N
AO
MS
resu
lts a
re
used
in q
uest
iona
ble
fash
ion
to su
ppor
t pa
roch
ial i
nter
ests
of
user
s the
reby
da
mag
ing
inte
grity
of
the
proc
ess
Lim
it ac
cess
to th
e N
AO
MS
surv
ey p
roce
ss to
the
NA
OM
S Te
am o
f exp
erts
un
til th
e pr
oces
s has
m
atur
ed.
TABL
E 8
.2 (
CON
TIN
UED
) TE
CHN
OLO
GY
/IMPL
EMEN
TATI
ON
RIS
KS
AN
D M
ITIG
ATI
ON
STR
ATE
GIE
S
66
Pr
oduc
t and
Haz
ard
Prob
abili
ty o
f O
ccur
renc
e C
onse
quen
ces o
f O
ccur
renc
e M
itiga
tion
Stra
tegy
an
d A
ctio
n Im
plem
enta
tion
Prob
abili
ty
SYST
EM-W
IDE
INC
IDEN
T R
EPO
RTI
NG
EN
HA
NC
EMEN
TS: U
pgra
de o
f the
24-
year
old
te
chno
logy
of t
he A
SRS
data
base
to in
clud
e: c
onve
rsio
n of
ASR
S le
gacy
dat
abas
e to
OR
AC
LE; e
lect
roni
c su
bmiss
ion
of re
ports
; and
test
and
eva
luat
ion
an a
naly
st
deci
sion-
supp
ort s
yste
m
V
ery
Hig
h: S
ome
of th
e im
prov
emen
ts a
re a
lread
y in
da
ily u
se b
y th
e A
SRS
Off
ice
and
it is
exp
ecte
d th
at
all w
ill b
e im
plem
ente
d ev
entu
ally
. It
is v
ery
likel
y th
at m
ost,
if no
t all,
of t
hese
w
ill fi
nd a
pplic
atio
n in
si
mila
r dat
abas
es li
ke A
SAP
and
NA
OM
S In
suff
icie
nt fu
nds t
o ad
apt n
ew te
chno
logi
cal c
apab
ilitie
s de
velo
ped
by o
ther
ele
men
ts of
ASM
M to
rout
ine
proc
essi
ng o
f inc
iden
t rep
orts.
HIG
H.
Safe
ty p
erso
nnel
w
ould
not
be
able
to
utili
ze to
the
fulle
st
data
sour
ces t
hat a
re
esse
ntia
l to
caus
al
anal
yses
and
pr
oact
ive
man
agem
ent o
f risk
.
Cont
inue
to se
ek fu
ndin
g fr
om o
ther
sour
ces t
hat
coul
d eq
ually
ben
efit
from
ef
ficie
nt a
nd re
liabl
e au
tom
ated
pro
cess
ing
of
anec
dota
l rep
orts
.
Elec
troni
c su
bmis
sion
of re
ports
may
be
perc
eive
d as
ha
ving
inad
equa
te se
curit
y.
MED
IUM
So
urce
s will
be
unw
illin
g to
pro
vide
da
ta.
Cont
inue
to w
ork
with
air-
serv
ices
pro
vide
rs to
ens
ure
that
acc
epta
ble
proc
edur
es
are
built
into
syst
em.
TABL
E 8
.2 (
CON
TIN
UED
) TE
CHN
OLO
GY
/IMPL
EMEN
TATI
ON
RIS
KS
AN
D M
ITIG
ATI
ON
STR
ATE
GIE
S
67
Prod
uct a
nd H
azar
d Pr
obab
ility
of
Occ
urre
nce
Con
sequ
ence
s of
Occ
urre
nce
Miti
gatio
n St
rate
gy
and
Act
ion
Impl
emen
tatio
n Pr
obab
ility
FA
ST-T
IME
SIM
ULA
TIO
N O
F SY
STEM
-W
IDE
RIS
KS:
Rig
orou
sly v
alid
ated
mod
els a
nd
simul
atio
ns o
f rel
atio
nshi
p am
ong
elem
ents
of t
he N
AS
to
supp
ort p
redi
ctio
ns a
nd sa
fety
-ris
k as
sess
men
ts o
f sys
tem
-w
ide
effe
cts o
f new
flig
ht a
nd A
TC te
chno
logi
es a
nd/o
r pr
oced
ures
bef
ore
they
are
inse
rted
into
the
oper
atin
g en
viro
nmen
t. In
clud
es e
ngin
eerin
g m
odel
s, op
erat
ing
conc
ept m
odel
s, su
ppor
t/log
istic
s mod
els,
hum
an
perf
orm
ance
mod
els a
nd ri
sk a
naly
ses
M
ediu
m to
Low
: Thi
s is
the
only
via
ble
appr
oach
to
addr
essi
ng th
e ta
rget
ed
prob
lem
in a
n ob
ject
ive
man
ner,
but i
t will
like
ly
alw
ays r
equi
re s
peci
aliz
ed
expe
rtise
from
gov
ernm
ent
or a
cade
mia
to u
tiliz
e th
e ca
pabi
lity
corr
ectly
and
ef
fect
ivel
y A
ir ca
rrie
rs, u
nion
s, an
d re
gula
tory
age
ncie
s typ
ical
ly d
o no
t hav
e th
e re
quire
d ex
perti
se to
mak
e co
rrec
t and
ad
equa
te u
se o
f mod
els a
nd si
mul
atio
ns, p
artic
ular
ly w
hen
they
con
tain
repr
esen
tatio
ns o
f hum
an b
ehav
ior a
nd
perf
orm
ance
.
HIG
H
Airl
ines
may
use
m
odel
s and
sim
ulat
ion
tool
s in
corr
ectly
or n
ot
cons
ider
thei
r pr
edic
tions
at a
ll.
Use
NA
SA e
xper
ts a
s “c
usto
dian
s” o
f sim
ulat
ion
tool
s, in
adv
isor
y ro
le to
us
ers.
The
curr
ent t
asks
to d
evel
op m
odel
ing
and
sim
ulat
ions
in
clud
ing
hum
an b
ehav
ior u
tiliz
e a
larg
e an
d di
vers
e te
am
of c
ontra
ctor
s. N
ot a
ll m
ay a
chie
ve e
qual
stan
dard
s of
perf
orm
ance
.
MED
IUM
If
one
subc
ontra
ctor
do
es n
ot p
erfo
rm,
proj
ect o
bjec
tives
m
ay n
ot b
e m
et
Mai
ntai
n st
rong
and
kn
owle
dgea
ble
man
agem
ent
with
wel
l stru
ctur
ed a
nd
defin
ed p
rogr
am a
nd
sche
dule
of s
ub-ta
sks t
o w
hich
all
resp
onsib
le
parti
cipa
nts a
gree
.
Expe
rtise
in h
uman
beh
avio
r mod
elin
g is
lim
ited
to v
ery
few
exp
erts
wor
ldw
ide.
Eur
opea
n re
sear
cher
s, in
pa
rticu
lar,
have
mad
e no
tabl
e co
ntrib
utio
ns to
mod
elin
g hu
man
beh
avio
r. N
ASA
’s a
bilit
y to
exp
loit
this
know
ledg
e is
limite
d be
caus
e of
cur
rent
con
stra
ints
aga
inst
fu
ndin
g Eu
rope
an re
sear
ch.
HIG
H
Proj
ect w
ill n
ot
achi
eve
resu
lts th
at
coul
d ha
ve b
een
poss
ible
with
Eu
rope
an in
put
Dev
ise
som
e fu
ndin
g m
echa
nism
by
whi
ch N
ASA
ca
n in
clud
e Eu
rope
an
expe
rtise
on
mod
elin
g te
am.
The
TRL
of M
odel
ing
and
Sim
ulat
ions
will
not
ach
ieve
th
e le
vels
atta
ined
by
the
prod
ucts
of In
tram
ural
and
Ex
tram
ural
Mon
itorin
g.
HIG
H
Mod
elin
g an
d sim
ulat
ion
tech
nolo
gies
will
re
quire
con
tinue
d go
vern
men
t in
vest
men
t in
thei
r ef
fect
ive
utili
zatio
n an
d de
velo
pmen
t.
Wor
k to
est
ablis
h an
ef
fect
ive
serv
ice
to th
e in
dustr
y w
ithin
NA
SA.
TABL
E 8
.2 (
CON
TIN
UED
) TE
CHN
OLO
GY
/IMPL
EMEN
TATI
ON
RIS
KS
AN
D M
ITIG
ATI
ON
STR
ATE
GIE
S
68
Pr
oduc
t and
Haz
ard
Prob
abili
ty o
f O
ccur
renc
e C
onse
quen
ces o
f O
ccur
renc
e M
itiga
tion
Stra
tegy
an
d A
ctio
n Im
plem
enta
tion
Prob
abili
ty
PRO
TOTY
PE S
YST
EM-W
IDE
RIS
K
ASS
ESSM
ENT
CA
PABI
LITY
: A c
apab
ility
that
de
mon
strat
es th
e fe
asib
ility
and
val
ue o
f aut
omat
ical
ly
mer
ging
de-
iden
tifie
d di
spar
ate
data
sour
ces t
o as
sess
sy
stem
-wid
e sa
fety
risk
s
M
ediu
m to
Hig
h: T
he
tech
nolo
gy fo
r mer
ging
da
taba
ses
and
extra
ctin
g us
eful
info
rmat
ion
exits
, but
th
ere
rem
ains
a "
polit
ical
" is
sues
of g
ettin
g ag
reem
ents
fr
om th
e ai
r car
riers
, the
un
ions
, and
the
ATC
to
prov
ide
acce
ss to
the
data
to
dem
onst
rate
the
valu
e (c
ontin
gent
on
the
avai
labi
lity
of F
Y 0
5 fu
ndin
g)
Ther
e is
no
prov
isio
n in
the
curr
ent A
SMM
sub-
proj
ect
Plan
for m
ergi
ng a
ll of
the
hete
roge
neou
s dat
a an
d in
form
atio
n ac
quire
d by
tool
s suc
h as
APM
S fr
om fl
ight
da
ta a
nd P
DA
RS fr
om A
TC d
ata
and
NA
OM
S fr
om
surv
ey d
ata
and
Sim
ulat
ions
and
oth
er d
iver
se so
urce
s.
The
prop
osed
enh
ance
men
ts a
nd a
ugm
enta
tions
to e
nabl
e su
ch in
form
atio
n to
be
mer
ged
and
anal
yzed
rout
inel
y m
ay
not b
e fu
nded
.
HIG
H
ASM
M w
ould
fail
to
achi
eve
its p
rimar
y ob
ject
ive
of
prov
idin
g th
e in
dustr
y w
ith re
gula
r, re
liabl
e, in
sigh
tful,
and
imm
edia
tely
us
eful
info
rmat
ion
to
aid
in a
sses
sing
the
perf
orm
ance
and
sa
fety
of t
he
natio
nwid
e N
AS.
Aug
men
t the
ASM
M su
b-pr
ojec
t for
FY
’05-
FY’0
9 as
pr
opos
ed.
Key
dat
a so
urce
s (ai
rline
s and
FA
A A
TC) d
o no
t tru
st
each
oth
er a
nd m
ay n
ot a
gree
to m
erge
thei
r dat
a.
HIG
H
Inab
ility
to m
erge
ai
rline
and
FA
A
ATC
ope
ratio
nal
data
will
lim
it ca
usal
an
alys
is a
nd ri
sk
asse
ssm
ent.
Wor
k ta
ctfu
lly w
ith a
ll si
des
to d
evel
op a
dequ
ate
trust
an
d ac
cept
able
“qu
id p
ro
quo”
to p
artic
ipat
e in
a
dem
onstr
atio
n of
the
bene
fits t
o al
l of m
ergi
ng
fligh
t and
ATC
dat
a.
TABL
E 8
.2 (C
ON
TIN
UED
) TE
CHN
OLO
GY
/IMPL
EMEN
TATI
ON
RIS
KS
AN
D M
ITIG
ATI
ON
STR
ATE
GIE
S
69
Impl
emen
tatio
n of
var
ious
ASM
M sy
stem
s as s
epar
ate
entit
ies w
ill p
rovi
de to
ols t
o ha
ndle
dat
a, b
ut in
tegr
atin
g th
eir p
rodu
cts i
s the
real
obj
ectiv
e of
the
prog
ram
to
enab
le n
atio
nal a
naly
sis o
f pre
curs
ors a
nd c
ausa
l dat
a.
Not
eno
ugh
emph
asis
has b
een
put o
n in
tegr
atio
n.
HIG
H
Dat
a in
sepa
rate
sy
stem
s can
not b
e vi
ewed
for a
nat
iona
l pe
rspe
ctiv
e
Begi
n pr
oces
s of
inve
stig
atin
g ho
w A
SMM
to
ols c
ould
be
inte
grat
ed to
su
ppor
t saf
ety-
risk
asse
ssm
ent f
rom
a n
atio
nal
pers
pect
ive.
Supp
orta
bilit
y of
ASM
M sy
stem
s for
the
syst
em-w
ide
pers
pect
ive
may
bec
ome
a co
st is
sue
as th
ey g
row
in si
ze
and
if N
ASA
take
s ove
r the
role
as n
atio
nal d
ata
repo
sitor
y
HIG
H
Not
eno
ugh
reso
urce
s to
supp
ort
full
syst
em
impl
emen
tatio
n an
d in
tegr
atio
n fo
r the
na
tionw
ide
pers
pect
ive.
Turn
ove
r sys
tem
supp
ort t
o pr
ivat
e en
titie
s tha
t can
put
A
SMM
dat
a to
com
mer
cial
us
e, o
r tra
nsfe
r sup
port
func
tions
to c
ontri
butin
g ai
rline
s, or
seek
a
Cong
ress
iona
l lin
e ite
m fo
r N
ASA
’s c
usto
dian
ship
of a
na
tiona
l res
ourc
e.
TABL
E 8
.2 (C
ON
TIN
UED
) TE
CHN
OLO
GY
/IMPL
EMEN
TATI
ON
RIS
KS
AN
D M
ITIG
ATI
ON
STR
ATE
GIE
S
70
9.0 REVIEWS Various reviews have been established to communicate the aviation safety information to AvSP management and committees.
9.1 MONTHLY Monthly Report by the Project Managers to the AvSP Manager has been developed:
• This report is an integrated technical, cost, and schedule assessment of progress versus plans and will contain significant technical highlights. The monthly report will be prepared in a standard, consistent electronic format, including appropriate graphics and accompanying explanatory text.
• A narrative description will also be developed to identify any problems, issues, and concerns (along with potential impact and proposed action) and any major interactions with industry
9.2 INDEPENDENT REVIEWS The Sub-Project will participate as appropriate in reviews with significant industry and other Government agency partner participation, including the FAA. Collaboration among partners is encouraged at the reviews. Each participant will present his/her accomplishments and plans, and significant technology developments will be demonstrated. As required, the Sub-Project Manager will present technical status overviews at Agency-sponsored Independent Implementation Reviews that are intended to assess Project stability; and participate in independent Aviation Safety Program Executive Council meetings or Aviation Safety Working Group meetings that are intended to review technical progress, assess technology impact, and assess technology relevance. The Sub-Project Managers will also participate as required in technical quality reviews, such as those conducted by the National Research Council. 9.3 AD HOC REVIEWS During FY’01, two reviews were conducted of the Aviation Performance Measuring System (APMS) project. The first was for the ASRS/APMS Advisory Subcommittee on November 14, 2000. This group was asked to
• Assess the state of the art and expected near-term developments of FOQA. • Identify key technical challenges in analyzing and fully capitalizing on FOQA-like data. • Review the progress of APMS to date. • Evaluate our plans and expected products. • Recommend direction and focus of future NASA work.
At the conclusion of the presentations and discussions, the members of this groups said that they • Supported the proposed future plans, • Generally recognized and approved past contributions, • Could not address priorities, • Encouraged us to help the GA and rotorcraft communities, • Cautioned against pursuing developments beyond points of diminishing returns.
The second review was for the Aviation Safety Program’s APMS Industry Review on February 26-27, 2001. This group was asked to To assess status of the FOQA industry
71
• Identify key challenges in 3-5 year timeframe • Assess APMS contributions to current status • Assess APMS current plans • Recommend APMS changes to advance FOQA
The report of this review • Credited APMS with moving FOQA forward in the US during mid to late ‘90’s. • Commended partnering with operators and vendors for focusing on operational problems. • Noted that COTS vendors have closed gap with APMS tools for exceedance-based
FOQA. • Recognized that the private sector is focused on existing products and give little thought
to R&D for future needs. • Found remarkable consensus among vendors, government and Board members on
types of tools that should be addressed by APMS. This group gave recognition to the past accomplishments of the APMS Research Team and directed it to focus on advanced functions in the future. Ever since the initial implementation of the PDARS initial experiment in the ATC Western-Pacific Region in early FY’00, meetings of the users have been convened every 3 months to obtain feedback, provide recurrent training, and discuss future enhancements. The NAOMS Team meets irregularly with groups representing the various constituencies of the aviation community to keep them informed of progress and plans for the survey. During FY’03, a Panel of the National Research Council conducted a review of the Aviation Safety Program. The first step of this review entailed written responses to a set of questions. This was followed with formal presentations on all aspects of the ASMM sub-project to the entire Panel in February 2003. On May 7, 2003, a subset of the Panel traveled to ARC to conduct an on-site full-day review of the ASMM sub-project during which the members of the review team met and spoke with all of the key civil servant and contractor personnel. The results of this review were included in the NRC report “An Assessment of NASA’s Aeronautics Technology Programs” issued in November 2003. 9.4 OTHER PROJECT MEETINGS TBS
10.0 TAILORING Based on the descriptions, definitions, and requirements in NPG 7120.5a, the following tailoring has been applied to this plan:
• Customer Definition and Advocacy is discussed under Section 1.0, Introduction. • Project Authority, Management and Control are discussed under Section 3.0, Project
Authority/Management. • Technical Summary, Schedules, Implementation Approach and Technical
Assessments are discussed under Section 4.0, Technical Approach.
72
• Agreements and Program/Project Dependencies are discussed under section 5.0, Agreements.
• Resources and Acquisition Summary are discussed under Section 6.0, Resources. • Commercialization is discussed under Section 7.0, Technology
Transfer/Commercialization.
Performance Assurance, Environmental Impact and Safety are not considered applicable to the Aviation Safety Program. 11.0 ACCOMPLISHMENTS Monthly status reports and accomplishments can be found at the following url location: https://ace.arc.nasa.gov:443/postdoc/t/folder/main.ehtml?url_id=11237
