Development of an Industry Standardized Auditing and Surveillance Tool: Minimizing Maintenance...
-
Upload
theodora-lyons -
Category
Documents
-
view
217 -
download
0
Transcript of Development of an Industry Standardized Auditing and Surveillance Tool: Minimizing Maintenance...
Development of an Industry Standardized Auditing and Surveillance Tool: Minimizing
Maintenance Errors
Web-based Surveillance & Auditing Tool
Human Computer Systems Laboratory
Industrial Engineering
Clemson University, Clemson, SC 29634
www.ces.clemson.edu/ie/research/hcsl/websat
Clemson WebSAT Team:Dr. Anand K. Gramopadhye (Professor and IE Department Chair) Nikhil Iyengar (Manager, HCSL)
Dr. Joel S. Greenstein Kunal KapoorPallavi Dharwada
Sponsor: FAA
Industry Partner: FedEx
Web-based Surveillance & Auditing Tool
Research Objective
Identify a list of impact variables (process measures) affecting aviation safety across aircraft maintenance organizations.
Develop data collection/reduction and analysis protocol toanalyze errors for impact variables identified.
Develop and implement an application performingsurveillance/monitoring maintaining a consistent levelof oversight.
Slide 1
Web-based Surveillance & Auditing Tool
Deliverables
Slide 2
Year IIdentify list of impact variables to be used. Identify limitations and assess use of specific data sources in theauditing and surveillance tool.
Year II
Establish final design specifications.
Report on web based tool development process.
Analyze data collected using the web based tool.
Year III
Deliver prototype tool incorporating the trend analysis module.
Provide guidelines and recommended practices to use auditing/
Surveillance tool for overcoming maintenance oversight.
Deliver final auditing/surveillance tool source code and documentation.
Web-based Surveillance & Auditing Tool
WebSAT Mission Statement
Slide 3
Web-based Surveillance & Auditing ToolProduct description
Application incorporating recommended categorization, data collection, reduction and analysis schemes to facilitating preliminary data examination for surveillance and auditing of maintenance operations.
Key business goals
Achieve standardized data collection, reduction and analysis of maintenance errors across the airline industry.
Develop proactive system to capture maintenance errors.
Accomplish trend analyses in future versions of WebSAT.
Market Air Transportation Industry
Assumptions Develop WebSAT adhering to FAA standard research software design specifications (e.g. SQL server, ASP.NET, PHP)
Stakeholders FAA
Quality Assurance Department of Air Carriers (incl. QA representatives and auditors).
Carrier Information Technology Department
Web-based Surveillance & Auditing Tool
Phase I: Identifying process measures• Define process measures to incorporate response and observation-based data collected
during surveillance, audits, and airworthiness directives control processes.
• Discover, Identify and Validate using data gathering methods adopted interviews, focus groups, observation sessions, studying documentation and questionnaires.
• WebSAT modules identified are surveillance, technical audits, internal audits and airworthiness directives.
– Note: Airworthiness directives module will include display of the airworthiness data. It will not include a risk model.
Slide 5
Year I (A)
Identification of Impact Variables and
Data Sources(A1)
Validation(A2)
Deliverables to FAA(A3)
Identified List of Process Measures
WebSAT
Surveillance Auditing AD Control
Technical Audits Internal Audits
Administration
Records
Manuals
Procedures
Safety
Training
Compliance/Documentation
Inspection
Facility Control
Training & Personnel
Procedures
Data Control
Safety
Loading & Tracking
Information Verification
Documentation Surveillance
Facility Surveillance
Procedures and Manual Violation
In Process
Verification
Final Aircraft Walk around
Web-based Surveillance & Auditing Tool
Step I with FedEx
Step II with additional partnering airlines (e.g. Representatives of NWA, ATA, AA)
Slide 6
Year I (A)
Identification of Impact Variables and
Data Sources(A1)
Validation(A2)
Deliverables to FAA(A3)
Online-survey for process measure validation
Web-based Surveillance & Auditing Tool
Deliverables to FAA(A3)
Slide 7
Year I (A)
Identification of Impact Variables and
Data Sources(A1)
Validation(A2)
Year I deliverables to FAA:Identified list of impact variables to be used.
Identified limitations and assessed use of specific data sources in the auditing and surveillance tool.
Status Update
Year II & III
Work Accomplish
ed (B)
Work in Progress
(C)
Prototyping
(B1)
Concept Selection
(B2)
Software Developme
nt (B3)
Technical Audit
Module Validate with Controlled
Studies
Interface
Design to
Match the
User Persona
s(C1)
Identify Data
Reduction Techniqu
es
(C2)
Risk Model
for Data
Analysis
(C3)
Web-based Surveillance & Auditing Tool
Work accomplished:Revised product map using the feedback provided by QA audit Managers at
FedEx.The team developed three concepts for WebSAT:– Concept 1: Based on Google Search Engine with multiple search criteria– Concept 2: Based on Microsoft Outlook– Concept 3: Based on Tab metaphor Generated Prototypes.Selected Final Concept.Designed and Developed the Interface for technical audit module.
Next Slide shows Product Map
Prototyping
(B1)
Concept Selection
(B2)
Software Developme
nt (B3)
Technical Audit
Module
Work Accomplishe
d(B)
Slide 8
Web-based Surveillance & Auditing Tool
Concept Selection
(B2)
Software Developme
nt (B3)
Technical Audit
Module
Work Accomplish
ed (B)
Prototyping (B1)
Slide 9
Web-based Surveillance & Auditing Tool
Selection of final concept
Performed concept selection with HCI Experts, MIS Expert & Domain Experts
Users preferred organization structure of the tab metaphor.
Also preferred grid feature of the MS Outlook concept.
Software Developme
nt (B3)
Technical Audit
Module
Work Accomplish
ed (B)
Slide 10
.Concept Selection
(B2)
Prototyping
(B1)
Final Concept
Web-based Surveillance & Auditing Tool
Designed a database model.
Developed technical audits module using ASP.NET
with SQL Server.
Developed scenarios for a technical auditor.e.g., Thomas entered the audit data partially a week before and would
now want to enter the remaining data. He logs into the system, finds the appropriate audit into which the data needs to be entered. Enters the data and saves the changes.
Thomas also wants to search for all the audits performed in the past on the same vendor.
Concept Selection
(B2)
Work Accomplish
ed (B)
Prototyping
(B1)
Slide 11
.Technical
Audit Module
Software Developme
nt (B3)
Web-based Surveillance & Auditing Tool
Status Update
Year II & III
Work Accomplish
ed (B)
Work in Progress
(C)
Prototyping
(B1)
Concept Selection (B2)
Software Development (B3)
Technical Audit
Module Validate with
Controlled Studies
Interface
Design to
Match the
User Persona
s(C1)
Identify Data
Reduction Techniqu
es
(C2)
Risk Model
for Data Analysi
s
(C3)
Identified 3 research areas for enhancing product
Design interfaces to match the user personas.
Reduction of maintenance data to predict vendor performance and aircraft safety index.
Development of a risk model to analyze maintenance data.
Work in Progress – Research Inputs to Enhance WebSAT
(C)
Risk Model
for Data
Analysis
(C3)
Slide 12
.
Interface Design - User
Personas
(C1)
Identify Data
Reduction Techniqu
es(C2)
Web-based Surveillance & Auditing Tool
WebSAT report generation will involve:Identification of various fields for the report.
Indicate the required format of the report.
Generate report.
WebSAT risk model generation involves:Collection of historic data.
Identification of various variables/factors which may contribute to variance
in the responses.
Each work function will have a set of variables.
Hierarchical arrangement of the variables in an order such that the lowest level will represent the dependent variable. e.g., audit responses.
Generate data files using the established order of hierarchy.
Carry out hierarchical logistical analyses to determine the variables with significant effect.
Use significant variables to generate models for the auditing and surveillance work functions.
Slide 13
Interface Design - User
Personas
(C1)
Risk Model
for Data
Analysis
(C3)
Work in Progress – Research Inputs to Enhance WebSAT
(C)
Identify Data
Reduction Technique
s(C2)
Web-based Surveillance & Auditing Tool
WebSAT risk model application
Will use various risk models to answer specific questions asked by the
representative.
Some of these typical questions for the Technical Audit work function are:
a) How has the vendor performed over a period of time for an audit type?
Where β1, β2, β3 represent the regression coefficients of the various significant factors X1, X2, X3.
b) How has the vendor performed on the current audit or substantial maintenance event?
c) How has the vendor performed on the current audit or substantial maintenance for a particular process measure?
d) How has a particular audit type or type of aircraft performed?
Slide 14
Interface Design - User
Personas
(C1)
Risk Model
for Data
Analysis
(C3)
Work in Progress – Research Inputs to Enhance WebSAT
(C)
Identify Data
Reduction Technique
s(C2)
1 1 2 2 3 3 0vendorY X X X
Web-based Surveillance & Auditing Tool
Data Model Example
Rep.Variable
Name Description Min Max Range Name
X1 Audit type
This variable describes the number of different audit types
1 4 1 Fuel
2 Line
3 Ramp
4 Supplier/vendor
X2 DurationThis variable represents the time taken to complete an audit
Continuous var.
X3 ContinentThis variable describes the geographic location of the audit
1 3 1 America
2 Asia
3 Europe
X4Process measure
This variable refers to the categorical collection of the responses
1 7 1Compliance /
Documentation
2 Inspection
3 Facility Control
4Training and Personnel
5 Procedures
6 Data Control
7 Safety
Slide 15
Web-based Surveillance & Auditing Tool
Data Model Example contd.
Y = - 3.9665 (X11) - 0.9310 (X12) - 2.2.568 (X13) + 0(X14) - 0.01486 (X2) - 2.1142 (X41) - 0.9973(X42)
- 1.6173(X43) - 0.6776(X44) - 1.2207 (X45) -1.7407 (X46) + 0 - 0.00868 (X2*X31) + 0.04759(X2*X32)
+ 0(X2*X33) + 0.01608(X2 * X41) + 0.008069(X2 * X42) + 0.01098(X2 * X43) + 0.003374(X2 * X44)
+ 0.01030(X2 * X45) + 0.004742(X2 * X46) + 0(X2 * X47) + 3.4952(X31*X11) - 0.2180(X31*X12)
+ 1.0189 (X31*X13) + 0(X31*X14) + 0 (X32*X11) + 0(X32*X12) + 0(X32*X13) + 0(X32*X14)
+ 0(X33*X14) + 1.2708(X31*X41) + 0.8188(X31*X42) + 1.4562(X31*X43) - 0.1469 (X31*X44)
+ 1.2363(X31*X45) + 0.8479(X31*X46) + 0(X31*X47) + 0.6220(X32*X41) + 1.1018(X32*X42)
- 0.6775(X32*X43) - 1.0791 (X32*X44) - 0.06895(X32*X45) + 0.3316 (X32*X46) + 0(X32*X47)
+ 0(X33*X41) + 0(X33*X42) + 0(X33*X43) + 0(X33*X44) + 0(X33*X45) + 0(X33*X46) + 0(X33*X47)
+ 0.01822(X2*X31*X41) - 0.00209(X2*X31*X42) - 0.00213(X2*X31*X43) + 0.008828 (X2*X31*X44)
- 0.00121(X2*X31*X45) + 0.004484 (X2*X31*X46) + 0 (X2*X31*X47) - 0.09046(X2*X32*X41)
- 0.09971 (X2*X32*X42) - 0.09662 (X2*X32*X43) -0.02079(X2*X32*X44) - 0.07381(X2*X32*X45)
- 0.09860 (X2*X32*X46) + 5.0835
Model based on preliminary analysis
Slide 16
Web-based Surveillance & Auditing Tool
Data Model Example contd.
The GLIMMIX Procedure - Type III Tests of Fixed Effects
Effect DF DF F Value Pr > F
Audit Type 3 15844 10.12 <.0001
Duration mean 1 15844 25.26 <.0001
Continent 2 15844 1.4 0.2455
Process measure 6 15844 9.61 <0.0001
Duration mean * Continent 2 15844 3.63 0.0265
Duration mean * Process measure 6 15844 3.32 0.0029
Continent * Audit Type 3 15844 6.5 0.0002
Continent * Process measure 12 15844 3.25 0.0001
Duration * Continent * Process measure 12 15844 3.15 0.0002
Slide 17
Web-based Surveillance & Auditing Tool
Data Model Example contd.
X1 X2 X3 X4
Y (Logit)
Odds ProbabilityAudit type Duration Continent PM
X11 X12 X13 X14 X2 X31 X32 X33 X41 X42 X43 X44 X45 X46 X47
1 0 0 0 5 1 0 0 1 1 1 0 0 0 0 3.1163 22.5626 0.9575
0 1 0 0 21 0 1 0 1 0 0 0 1 0 1 1.5324 4.6290 0.8223
Example 1: Predict the probability of receiving a ‘yes’ for a Fuel Audit in Los Angeles which would last for 5 days. This audit would cover process measures of compliance documentation, inspection and facility control.
Example 2: Predict the probability of receiving a ‘yes’ for a Line Audit in Lake Charles which would last for 21 days. This audit would cover process measures of compliance documentation, procedures and safety.
Slide 18
Web-based Surveillance & Auditing Tool
Significant Milestones Ahead
Slide 19
Web-based Surveillance & Auditing Tool
Conclusion
• WebSAT will provide overview of activities associated with commercial aircraft maintenance.
• Identification of maintenance related process measures and related safety indices.
• Identification of potential problems at geographically dispersed maintenance sites through data standardization.
• Promotion of accessible display of trends in data.
Slide 20
Web-based Surveillance & Auditing Tool
Dissemination
• Iyengar, N., Kapoor, K., Dharwada, P., Greenstein, J. S., and Gramopadhye, A. K., WebSAT: Development of Process Measures for Aircraft Safety, International Journal of Applied Aviation Studies, September 2005.
• Kapoor, K., Dharwada, P., Iyengar, N., Greenstein, J. S., and Gramopadhye, A. K., Strategy to Identify Process Measures for Surveillance in Aviation, Proceedings of the Industrial Engineering Research Conference, Atlanta, May 2005.
• Kapoor, K., Dharwada, P., Iyengar, N., Greenstein, J. S., and Gramopadhye, A. K., Strategy for Evaluation of Aircraft Maintenance Operations Using Process Measures, International Symposium of Aviation Psychology, Oklahoma City, April 2005.
• Kapoor, K., Dharwada, P., Iyengar, N., Greenstein, J. S., and Gramopadhye, A. K., A Strategy for the Development of a Web-based Tool to Reduce Aviation Maintenance Errors, Proceedings of Human Factors and Ergonomics Society, New Orleans, September 2004.
• Iyengar, N., Kapoor, K., Greenstein, J. S., and Gramopadhye, A. K., Selection of Data Gathering Methodologies for the Aviation Maintenance Industry, Proceedings of the Industrial Engineering Research Conference, Houston, May 2004.
• Dharwada, P., Iyengar, N., Kapoor, K., Greenstein, J. S., and Gramopadhye, A. K., Standardized Auditing and Surveillance of the Aircraft Maintenance Operations, Proceedings of the Industrial Engineering Research Conference, Houston, May 2004.
• Dharwada, P., Iyengar, N., Kapoor, K., Gramopadhye, A.K., and Greenstein, J.S., Web-Based Surveillance and Auditing Tool (WebSAT): A Proactive System to Capture Maintenance Errors, Proceedings of Safety Across High-Consequence Industries, St. Louis, Missouri, March 2004.
Slide 21
Web-based Surveillance & Auditing Tool
Acknowledgements