Propulsion Diagnostic Method ... - Glenn Research …€¦ · National Aeronautics and Space ......
Transcript of Propulsion Diagnostic Method ... - Glenn Research …€¦ · National Aeronautics and Space ......
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES):A Public Approach for Benchmarking Gas Path Diagnostic Methodspp g g
Donald L. SimonNASA Glenn Research Center
Propulsion Controls and Diagnostics Research WorkshopD b 8 10 2009
www.nasa.gov 1
December 8-10, 2009Cleveland, OH
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
Overview
– Recent technology reviews have revealed that while Engine Health Management (EHM) related research and development has g ( ) pincreased significantly in recent years, there exists a fundamental inconsistency in defining and representing EHM problems.
– Currently many of the EHM solutions published in the open literature are applied to different platforms, with different levels of complexity, addressing different problems, and using different metrics for evaluating performance As such it is difficult to performmetrics for evaluating performance. As such it is difficult to perform a one-to-one comparison of candidate solutions.
– The ProDiMES propulsion gas path diagnostic benchmark problemThe ProDiMES propulsion gas path diagnostic benchmark problem and metrics have been constructed, and is proposed to serve as a standard benchmark, or theme problem, to facilitate the development and comparison of candidate propulsion gas path di ti th d l i
www.nasa.gov
diagnostic methodologies.
2
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
Outline
• Background & Motivationg
• Gas Path Diagnostic Benchmark Process
• Software Demonstration
F t S h d l• Future Schedule
• Summary
www.nasa.gov 3
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
Background & Motivation
• NASA Glenn sponsored a survey of recent advancements in Engine Health Management (EHM) technologies (in 2004 with Scientific Monitoring Inc.) g )
• Survey results showed that EHM related R&D activities have increased significantly since late 1990’s
• Survey also found that often …– Terminologies are different– Algorithms are differentgo t s a e d e e t– Applications are different– Presentations are different
N b i f i• → No basis of comparison
• Recommendation: Define and put forth a standardized process, with realistic complexity, to compare the merits of different EHM
www.nasa.gov
with realistic complexity, to compare the merits of different EHM approaches
4
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
Approach
• Participants are provided (free of charge)…– A common toolset / environment to enable the independent development,
evaluation and benchmarking of gas path diagnostic methods This includesevaluation and benchmarking of gas path diagnostic methods. This includes all types of diagnostic methods:
• Data driven (empirical) methods• Analytical (model-based) methods• Hybrid or fused methods
– Identical blind test cases to facilitate a one-to-one comparison of techniques developed by various participants
• Participants are expected to provide …– Diagnostic assessments based on the provided blind test cases
• Participants will receive their blind test case diagnostic results, plus the anonymous results of other participants
– Participation in follow-on workshop to share results and lessons learned
• Benefits of participation …– Access to a common gas path diagnostic theme problem– Results from a broad comparative study of candidate gas path diagnostic
methods developed by multiple participants
www.nasa.gov
methods developed by multiple participants
5
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
Key Decision Points
• Decided to specifically focus on ground-based diagnostics– Processing “snap-shot” measurements collected from each engine, each flight, with
realistic operating condition variations.
• Implemented in Matlab software. Produces simulated data using a generic turbofan engine model:
– Cons:Cons: • Does not capture all of the nuances inherent in actual engine data
– Pros:• Avoids ITAR and proprietary issues
P id bi “ d t th” t t k l d• Provides unambiguous “ground-truth” state knowledge• Enables significant quantities of test and evaluation data to be generated
• Provides an initial assessment of diagnostic methodsg– It is readily acknowledged that additional development and maturation would be
required to mature any technology to the level of practical implementation
• It is not intended to formulate this as a competition
www.nasa.gov
• It is not intended to formulate this as a competition– The generally positive intent is to provide an environment that will truly facilitate the
development, evaluation and sharing of EHM capabilities and ideas6
National Aeronautics and Space Administration
ProDiMESDiagnostic Benchmarking Process
www.nasa.gov 7
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
ProDiMES Diagnostic Benchmarking ProcessProDiMES enables independent development & evaluation
and a blind test case comparison
Engine Fleet Simulator User’sDiagnostic
EvaluationMetricsDiagnosticSensed parameter
Independent D l t
ProDiMES Public Benchmarking Process
Solutions Diagnosticassessments
Sensed parameterhistories
ResultsOutputs
Fault occurs
Development and
Evaluation“Ground truth” information
1a. Engine fleet simulator: Enables user to specify the type and number of gas path fault cases. 2. Solution providers apply
their individual diagnostic solutions
3. Evaluation Metrics: Defined and applied to provide a uniform assessment of performance
4. Results: archived in common format
1b. Blind test cases: User has no a priori knowledge of fault existence or fault type
Blind Test Case Data User’sDiagnosticSolutions
EvaluationMetrics &“GroundTruth”
Blind Test Case Side-by-Side
Diagnosticassessments
Sensed parameterhistories
www.nasa.gov 8
Truth”Information ResultsOutputs
Fault occurs
Comparison
“Ground truth” “Ground truth” information
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
Engine Fleet Simulator
• Engine Fleet Simulator (EFS) Includes:– Graphical User Interface: Allows end user to select the number
of engines in the fleet flights per engine instances of each faultof engines in the fleet, flights per engine, instances of each fault type, flight of fault occurrence, and fault evolution rates.
– Case Generator: Applies random engine operating conditions, deterioration profiles, and fault magnitudesdeterioration profiles, and fault magnitudes
– Commercial Modular Aero-Propulsion System Simulation Steady-State (C-MAPSS SS)
P tC MAPSS SSC G tU I t f ParameterHistories
C-MAPSS SSSteady-StateEngine Model
Case GeneratorUser Interface
2000
2100
Engine 1Engine 2
0 500 1000 1500 2000 2500 3000 3500 4000 4500 50001500
1600
1700
1800
1900
Flight Cycle
T48 / θ
(° R
)
25 30 35 40Cruise Pressure Altitude (K ft)
0.72 0.74 0.76 0.78 0.8 0.82 0.84Cruise Mach
−100 −80 −60 −40 −20 0Cruise Tamb (°F)
5000 6000 7000 8000Cruise Net Thrust − Fn (lbs)
www.nasa.govEngine Fleet Simulator Architecture
9
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
EFS - Graphical User Interface
Number of flights each
Specifies the number of engines and
Number of flights each engine experiences
Flight of fault initiation (fixed or random)engines and
number of instances of each no-fault, and fault type
(fixed or random)
Fault evolution rate (abrupt, rapid or random)
and fault type
• 18 different fault types Sensor noise (on / off)
• Individual engines can experience no more than
Initiates EFS Run
Estimates run
www.nasa.gov 10
no more than a single fault
time, and provides run status
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
EFS - Case Generator
Case Generator – Randomly assigns the following y g ginformation for each engine in the fleet based upon defined distributions:
• Performance deterioration profile as a function of flight cycles (approximated from NASA Contractor reports)
• Variations in Takeoff and Cruise operating conditions each flightVariations in Takeoff and Cruise operating conditions each flight
• Sensor measurement non-repeatability effects
F lt if t ti hibit d i ti i• Fault manifestations exhibit random variation in …– Magnitude– Evolution rate (if selected to occur randomly)– Flight of fault occurrence (if selected to occur randomly)
www.nasa.gov 11
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
C-MAPSS Steady State Engine ModelCommercial Modular Aero-Propulsion System Simulation
Steady-State (C-MAPSS SS):• NASA developed generic non-linear, component level, engine
model representative of a large commercial turbofanp g• Coded in Matlab• A steady-state version of C-MAPSS is used within the EFS:
– No closed loop control logic, or transient operating capability– Logic is applied to ensure operating limits are not violated– Steady-state solver balances engine to specified operating point– Captures coupled fault effects (e.g. a corrected rotor speed sensor
fault will result in mis-scheduled variable geometry).• Produces 11 sensed outputs
– 3 aircraft parameters (Pamb P2 T2)3 aircraft parameters (Pamb, P2, T2)– 8 engine measurements (Nf, Nc, P24, Ps30, T24, T30, T48, Wf36)
BypassNozzle
Atmos-phere
Inlet withRam
RecoveryFan
Nozzle
LPCHPC
& Combustor HPT LPT CoreNozzleVBV
www.nasa.gov
VSV Nozzle
C-MAPSS SS Engine Model Block Diagram12
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
ProDiMES Diagnostic Benchmarking Process
Sensedparameter Diagnostic
Engine Fleet Metrics
parameterhistories
Diagnosticassessments
Results
UserProvidedDiagnostic
Simulator Metrics
“Ground truth” engine condition
Solutions
www.nasa.gov 13
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
User Provided Diagnostic Solutions
Participants have the capability to develop data driven, or analytical diagnostic solutions:
• Data-driven solutions: The Engine Fleet Simulator is designed to provide participants the flexibility to generate data of the desired type and quantity
• Analytical Solutions: Participants will be provided access to the C-MAPSS SS engine model to facilitate the development of analytical diagnostic methods
www.nasa.gov 14
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
User Provided Diagnostic Solutions
Available Information Note: Diagnosis is to be solely based on sensed information collected at and prior to the current(stored in Matlab format):
Sensed Parameter Histories:
Required Solution(stored in Matlab
format):
collected at, and prior to, the current flight. Although future flight sensed information is available, it is not to be used for diagnosis
• 11 sensed values collected from each engine, each flight, at takeoff & cruise
format):
Diagnostic assessment for each engine, each
User Provided Diagnostic Solution g
flight consisting of the “Fault ID”
T t f l l t i
Solution
Ground truth fault information including:
Target false alarm rate is once per 1000 flights
Note: Ground truth fault information is available for d l t t i i d
• Fault type• flight of fault initiation• fault evolution rate
www.nasa.gov
development, training and evaluation purposes …but is not to be used for diagnosis.
15
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
ProDiMES Diagnostic Benchmarking Process
Sensedparameter Diagnostic
Engine Fleet Metrics
parameterhistories
Diagnosticassessments
Results
UserProvidedDiagnostic
Simulator Metrics
“Ground truth” engine condition
Solutions
www.nasa.gov 16
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
Metrics
• Defined to assess:– Fault detection performance
• True positives• True positives• True negatives• False negatives (missed detections)• False positives (false alarms)
– Fault classification performance• Correct classification rate• Kappa Coefficient
– Diagnostic latency– Diagnostic latency• Number of flights required to diagnose a fault
• A provided Matlab routine automatically evaluates performance p y pagainst the defined metrics, and archives results to a Microsoft Excel spreadsheet by …– Fault type
F lt l ti t ( b t id)
www.nasa.gov
– Fault evolution rate (abrupt or rapid)– Fault magnitude (small, medium, large)
17
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
Metrics (cont.)iti
on
= True Condition
= Diagnosed ConditionTrue positive detectionIncorrect classification
e Fa
ult C
ondi
Fault presenton flights ≥ K
False positive
d t ti
No fault present on flights 1 through K-1
Engi
ne
0( f lt)
detectionFalse
negativedetections
True positives &Correct
classifications
flights 1 through K 1
(no fault)
2 Pre fault window:1 Initial window: The 3 Fault window: Diagnostic 4 Post-fault window:2. Pre-fault window: Diagnostic assessments produced on flights 11 through K-1 are assessed for false positive or true negative detections
1. Initial window: The “no fault” condition is guaranteed for first 10 flights. Diagnostic assessments produced on these flights are excluded from metrics.
3. Fault window: Diagnostic assessments on this window of flights are monitored for true positive or false negative detections, and correct or incorrect classifications
4. Post-fault window: Diagnostic assessments produced on all flights after the fault window are excluded from metric assessments
www.nasa.gov
Flight numberK
18
Diagnostic window partitioning of engine flight history
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
Metrics (cont.)
Predicted State
Detection Decision Matrix
No FaultFault
Predicted State
False Negatives
No Fault
Fault
True NegativesFalse Positives(false alarms)
(missed detections)
True Positives
rue
Stat
e
(false alarms)Tr
True PositivesTPR 100%= ⋅+
= ⋅+
TPR 100%True Positives False Negatives
False PositivesFPR 100%False Positives True Negatives
True Positive Rate
False Positive Rate
www.nasa.gov
g
19
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
Metrics (cont.)Classification Confusion Matrix
Predicted State
Fault 1
Fault n. . . Fault 2Fault 1
C1n. . .C12C11
:
Fault 2 C2n. . .C22C21
:: :rue
Stat
e
Fault n
:
Cn2
:
Cn1
:
Cnn. . .
:. . .Tr
ppp n
CPCC 100%
C= ⋅
∑Percent Correct Classification Rate
www.nasa.gov
pqq 1
C=∑
20
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
Metrics (cont.)
Kappa Coefficient – Measure of an algorithm’s ability to correctly classify a fault, which takes into account the expected
b f t l ifi ti i b h It bnumber of correct classifications occurring by chance. It can be calculated from the elements of a Confusion Matrix as follows:
Kappa N(correctly classified) N(expected correct by chance)KappaCoefficient
n
N(correctly classified) N(expected correct by chance)N(total) N(expected correct by chance)
where
−κ =
−
n
ppp 1
n n
pq
N(correctly classified) C
N(total) C
=
=
=
∑
∑∑p 1 q 1
n npq
qpq 1 q 1
CN(expected correct by chance) C
N(total)
= =
= =
⎧ ⎫= ⋅⎨
⎩∑ ∑
n
p 1=⎬⎭
∑
www.nasa.gov 21
National Aeronautics and Space Administration
Example Diagnostic Solution
www.nasa.gov 22
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
Example Diagnostic Solution““Snap-shot” parameter histories collected at discrete operating points each flight
2. Event Detection:Rapid/abrupt trend shift detection
3. Fault Isolation:Identifies & ranks most
1. Trend Monitoring:Monitors trends in engine performance
Parameter Correction: Rapid/abrupt trend shift detection
logicIdentifies & ranks most plausible root causes for fault
Monitors trends in engine performance “deltas” relative to established baseline
exponential+ ∆ ∆yema detect∆∆y single recordyessensed
parameters
Correction:Accounts for variation in ambient conditions at the engine inlet
fleetaverage
exponentialmovingaverage
baselineengine
+
-
∆y differencecalculation
∆yema detectthresholdexceeded
?
∆∆yema
record
singlefault
isolator
fault type,& flight ofdetection
y
no
y
δθ
θWfWf
NN
c
C
=
=11
1
genginemodel
parametersybaseline
no-faultfound
δθc
engineoperatingconditions
www.nasa.gov
( ) 22
1
* )(⎥⎥⎦
⎤
⎢⎢⎣
⎡⎟⎟⎠
⎞⎜⎜⎝
⎛ ∆∆−∆∆= ∑
=
m
i k
kkn
iieσ
23
National Aeronautics and Space Administration
Blind Test Case Data
www.nasa.gov 24
National Aeronautics and Space Administration
Propulsion Diagnostic Method Evaluation Strategy (ProDiMES)
Blind Test Case Data
• Includes data from approximately 10,000 engines• 50 flights of data are provided for each engine• 50 flights of data are provided for each engine• Both abrupt and rapid fault types are included in the blind
test cases• A target false positive detection rate (false alarm rate) ofA target false positive detection rate (false alarm rate) of
once per 1,000 flights is specified• Participants are to submit their blind test case diagnostic
assessments to NASA.• NASA will evaluate results against the metrics routine• Participants will receive their own results, plus the
anonymous results of other participants
www.nasa.gov 25
National Aeronautics and Space Administration
Software Demonstration
www.nasa.gov 26
National Aeronautics and Space Administration
Future Schedule
www.nasa.gov 27
National Aeronautics and Space Administration
ProDiMES Schedule• ProDiMES released through the NASA GRC Software Repository (August 2009)
(https://technology.grc.nasa.gov/software/)
• Release includes:• Release includes:– Software Usage Agreement (must be signed prior to gaining access to the software)– Matlab software for independent development and evaluation– Identical blind test cases
• Individual development & evaluation (August 2009 – Spring 2011)
• Blind test case results evaluated (Spring 2011)– Participants will receive their own results, plus the anonymous results of other participants
• Hold NASA workshop to disseminate results and lessons learned (Spring 2011)
Important! Users are not to publicly disseminate ProDiMESp p yresults prior to the workshop!
• For future ProDiMES schedule updates please visit the website:www grc nasa gov/WWW/cdtb/software/ehmbenchmark html
www.nasa.gov
www.grc.nasa.gov/WWW/cdtb/software/ehmbenchmark.html
28
National Aeronautics and Space Administration
Summary
• A public process for benchmarking gas path diagnostic methodologies is proposedmethodologies is proposed– Allows end user’s to independently develop and evaluate candidate
gas path diagnostic methods– Automatically assesses diagnostic performance against a defined
set of metricsset of metrics– Identical blind test cases to be provided to enable a one-to-one
comparison of candidate approaches
F dditi l d t il d d t• For additional details and updates:– Contact: Don Simon (1-216-433-3740); [email protected]– To request ProDiMES visit the website:
https://technology.grc.nasa.gov/software/p gy g g– For schedule updates visit the website:
www.grc.nasa.gov/WWW/cdtb/software/ehmbenchmark.html
www.nasa.gov 29