Army Materiel Systems Analysis Activity (AMSAA) …...Reliability Scorecard PSM Workshop 21 March...
Transcript of Army Materiel Systems Analysis Activity (AMSAA) …...Reliability Scorecard PSM Workshop 21 March...
Army Materiel Systems Analysis Activity (AMSAA) Resources
AMSAA
Dr. David Mortin
21 March 2018
PSM Workshop 21 March 2018 1Army Materiel Systems Analysis Activity (AMSAA)
Resources
DISTRIBUTION STATEMENT A - Approved for public release.
Distribution is unlimited
Resources
• Design Modeling
• Assessment tools• Scorecard• Reliability Growth Models
• Data• Sample Data Collection• CBM
• Metrics
• Analysis
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources2
https://www.amsaa.army.mil/CRG_Tools.html
Preparing for a Challenging Environment
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources3
Reliability Scorecard
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources4
• Provides standard method to identify reliability risk areas
• Based on IEEE, academia, Raytheon, Alion, RIAC, GEIA recommendations, best practices, etc.
• Structured approach allows refinement and improvement.
• General Scorecard has 8 Categories (40 elements)
• Software systems version has 7 Categories (57 elements)
Reliability Requirements and
PlanningTraining and Development Reliability Analysis Reliability Testing
Reliability activities that are integral
to design and testing are clearly
identified and incorporated into the
program Integrated Master
Schedule
Sufficiently-sized reliability
engineering staff is directly tied to
the design team
Conduct failure modes, effects,
and criticality analysis (FMECA)
and fault tree analysis (FTA);
crosswalk to low-level testing and a
failure mechanism analysis to
ensure programmatic coverage
Conduct low-level testing early to
identify and mitigate failure modes.
Conduct highly accelerated life
testing (HALT) and highly
accelerated stress screening
(HASS)
Supply Chain Management Failure Tracking and Reporting Validation and Verification Reliability Improvements
Plan and establish mature and well-
documented manufacturing procedures
Utilize failure reporting, analysis and
corrective action system (FRACAS)
Modify testing and procedures based
on failure occurrencesDocument lessons learned
Sample Elements
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources5
Reliability Design Analyses
This Circuit Board Fails Due
to VibrationThis Circuit Board Does Not
(just by adding two screws)
Initial Design
Electronics in an Army Hand Held Device
Design after Physics-of-
Failure Vibrations Analysis
Design modification costing almost nothing changes product from being unreliable to reliable
Thermal Analysis
Provided
Additional Insights
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources6
System-Level Modeling
Full-system Modeling provides tremendous insights that reduce testing and
eliminate failures (pre and post fielding)
• Mitigate sources of failure
• Quickly develop solutions for failures
• Positively impact
design
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources7
Root-Cause Analysis
Critical Components:
• Stresses, forces,
accelerations, etc. are
available for
components in the
model
• Models allow for quick
visualization of
multiple fixes or design
enhancements
• Identify weak spots and redesign in
virtual environment – not test-fix-
test (Too Expensive!)
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources8
Schofield Barracks
Joint Base Lewis-McChord
Ft. Irwin
Ft. Carson
Ft. HoodFt. Bliss
Ft. Rucker
Ft. Campbell
Ft. Stewart
Vilseck
Kuwait
Poland
Ft. Riley
ANAD
• 197 contractors at 11 CONUS and 3 OCONUS locations
• Over 7,044 ground systems and 508 aviation systems
• Reliability Metrics
• Maintenance Hours
• Trends
• New Maintenance Pilots
• Fixing Specific Field Issues
Sample Data Collection & Analysis
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources9
Download to Laptop
(Includes at platform Diagnostic
Trouble Codes)
Analysis
Algorithms
Digital Source Collector (DSC)
& JPRO
Server
AMSAA SDC&A
Complete & Accurate
Field Level Maintenance
Data
CBM+: Fusing Data to Provide
Actionable Information
Web-based CBM+ Tools
Motor Pool Maintenance
Management
Fleet Management
Vehicle Health Alerts
Using Data
Analytics of 200+
Billion Sensor
Readings &
Maintenance
Events to Provide
Actionable
Information at
Field and Fleet
Levels
AMSAA has 2,300+ DSCs
installed on vehicles at 8
world-wide locations
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources10
Brakes ParametersFront Axle Speed
Relative Speed, Front Axle, Left Wheel
Relative Speed, Front Axle, Right Wheel
Vehicle Speed
Brake Switch
Engine ParametersAccelerator Pedal Position
Barometric Pressure
Boost Pressure
Coolant Level
Engine Coolant Temperature
Engine Load
Engine Oil Pressure
Engine Oil Temperature
Engine Speed
Fuel Rate
Fuel Temperature
Injection Control Pressure
Intake Manifold Temperature
Engine Percent Torque
Vehicle Speed
PTO Mode
Transmission ParametersTransmission Oil Temp
Transmission Range Attained
Transmission Range Selected
Transmission Output Shaft Speed
Input Shaft Speed
Transmission Actual Gear Ratio
Transmission Current Gear
Transmission Selected Gear
Torque Converter Lockup Engaged
Diagnostic Trouble CodesEngine
Transmission
Brakes
Typical CBM+ Data Collected
CBM+ to Enable
Data Driven Fleet
Management
Improved equipment
Readiness
Cost savings
Reduced repair time
Value added
maintenance activities
Correct parts ordered
Better Options for Reliability Prediction
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources11
0 500 1000 1500 2000 2500 3000 3500 4000 4500
System 14
System 13
System 12
System 11
System 10
System 9
System 8
System 7
System 6
System 5
System 4
System 3
System 2
System 1
Mean Time Between Failure (Hours)
Demonstrated
Prediction 1) Hard to keep handbooks up-to-date with current technology
2) Represent only a small portion of the overall system failure rate
3) Does not consider critical factors that influence reliability (e.g. thermal cycling)
4) Does not provide insight on how or why the system will fail
5) Data not current (lagging years)
Economic Useful Life Analysis
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources12
0
2
4
6
8
10
12
14
16
18
20
No
n-M
iss
ion
Ca
pa
ble
Ac
tio
ns
(p
er
XX
XX
mil
es
)
Odometer Reading (miles)
Based on 6 years of Sample Data Collection field data
Vehicles range in age from 3 to 12 years old
Represents the 10th, 50th, and 90thpercentiles
for each mileage bin
For this vehicle, no statistical indications of aging over the period
Multiple ways for PSMs to analyze to ensure optimum Economic Useful Life decisions
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources13
Leveraging Data and Information to the Maximum Extent Possible
σf Subsystem Strength
Probability
of
Failure
Using Physics-Based-
Reliability methodology
to determine impacts of
weight or mission
changes for ground
systems
Field data from
existing systems
combined with
engineering modeling
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources14
14
Reliability Metric Differences Pre and Post Fielding
Environment Incident Classification Metric
Test Test incidents classified using FD/SC (i.e.
EMA, UMA, OMF, SA, EFF)
MTBSA,
MTBEFF
Field Unscheduled maintenance actions classified
using TM-10 PMCS (i.e. NMC)
MTBNMCA,
MTBUMA
How can we compare
test metrics to field
metrics?
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources15
A Case Study – Ground Vehicle
Test Field
Available Data Failure data collected via TIRs
“Detailed” description of
maintenance performed
Unscheduled maintenance and
associated part replacements
data collected in field
“Basic” fault description,
correction narrative and failure
description (if applicable or
known)
Mapped NSN’s that cause NMC
events in the field to test incident
reports
Common rule sets
in the field were
applied to test data
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources16
Ground System Reliability Metrics
Adjustments to test data allow for a more direct comparison of test and field
reliability metrics.
Statistical tests can be utilized to determine if there are any significant
differences in the test and field data sets.
Mean Miles
Between
NMC Action
(MMBNMCA)
Total Mileage
# NMC Actions=
Reliability Comparison
*Also contrasted to baseline
requirement
Another way to
compare pre and post
fielding reliability is to
score the field data as
was done in testing
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources17
80
114
175
215220
CAP1
CAP2
CAP3
CAP4
0
50
100
150
200
250
300
Idealized Curve Customer Test Initial DT LUT LUT Excursion IOT
$869 M$894 M
$1,103 M
$1,701 M
$2,457 M
Test Time (Hours)CAP – Corrective Action Period
Reliability Growth and its Impact On Support Costs
Me
an
Tim
e B
etw
ee
n F
ail
ure
Reliability Growth Planning,
Tracking, & Projection tools
along with the reliability
scorecards are available for
free to US Government
organizations and supporting
contractors
https://www.amsaa.army.mil/CRG_Tools.html
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources18
Establishing Solid Reliability Growth Strategies
Category Low Risk Medium Risk High Risk
< 70% 70 - 80% > 80%
IOT&E Producer’s Risk ≤ 20% 20+ - 30% > 30%
IOT&E Consumer’s Risk ≤ 20% 20+ - 30% > 30%
Management Strategy < 90% 90 - 96% > 96%
Fix Effectiveness Factor ≤ 70% 70+ - 80% > 80%
< 2 2 - 3 > 3
Time to Incorporate and
Validate Fixes in IOT&E
Units Prior to Test
Adequate time and
resources to have fixes
implemented & verified
with testing or strong
engineering analysis
Time and resources for
almost all fixes to be
implemented & most
verified w/ testing or
strong engineering
analysis
Many fixes not in place by
IOT&E and limited fix
verification
Initial MTBF
(DT) Goal MTBF
Potential Growth MTBF
(DT) Goal MTBF
Risk
assessment of
curves now
included in
some
reliability
growth
modeling tools
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources19
Supply Chain & Inventory Analysis Demand Planning
Analysis of Requirements,
New Logistics Concepts,
and PBL Arrangements
Army Prepositioned Stock Computations
$- $1 $2 $3 $4 $5 $6 $7 $8 $9
$10
0-2Years
2-5Years
5-10Years
10-20Years
20-50Years
50+Years
NFD
Bill
ions
Insurance Items BDROSafety Level LeadTime Rqmt24 Months Dmd Econ/LOT RetnLOT-Excess Excess
Establish business based
performance metrics to track
efficiency and value added
Assess requirements feasibility
along with mission impact
Selected Essential Stocks for Availability Methodology (SESAME)
Optimum stock levels and identification of excess inventory Analyzing accuracy of forecasts
Impact of multi-level additive
manufacturing and its potential
effects on demand planning
Optimum Stock Requirements Analysis Program
Optimized packages for Class I, II, IIIP, IV and IX
Stocks, Forecasting, and Requirements Assessment
PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)
Resources20
Materiel Management
Supply Chain
Inventory Optimization
Demand Planning
Army Prepositioned Stocks
Field Operations
Sample Data Collection & Analysis
Operational Sustainment
Condition Based Maintenance
Reliability Centered Maintenance
Acquisition Support
Logistics Footprint
Level of Repair
Life Cycle Cost
Schedule Risk
Reliability
Center for Reliability Growth
Physics of Failure
Requirements Determination
Test Support & Efficiencies
Operational Energy
Fuel Consumption
Power Sufficiency
Expeditionary Basing Energy
Operational Logistics Planning
75%
80%
85%
90%
95%
100%
$0.0 $0.5 $1.0 $1.5 $2.0
SA
$ Safety Level (Billions)
99% SA
85% SA
90% SA
95% SA
Resources and Contact Info
David Mortin, Ph.D.
410.278.6248