audit Skills Training Handbook
-
Upload
kukuh-widodo -
Category
Documents
-
view
178 -
download
6
description
Transcript of audit Skills Training Handbook
HSE Auditing: Fundamentals, Skills and Techniques for Team Members
NPC Course Manual 2005
Arthur D. Little Limited Science Park, Milton Road Cambridge CB4 0XL Telephone 01223 392090 Fax 01223 420021 Reference 20365
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 2 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Copyright © 2005 by Arthur D. Little Limited. All rights reserved.
Table of Contents
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 3 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Chapter Page
No Overview of Environmental, Health, and Safety Auditing 5 Audit Approach 12 Basic Steps in the Typical Audit Process 23 Pre-Audit Activities 25 On-site Opening Activities 42 Understanding HSE Management Systems 55 Effective Interviewing 75 Preparing Working Papers 105 Assessing Strengths and Weaknesses 135 Gathering Audit Evidence 161 Sampling Strategies 189 Evaluating Audit Results 220 Writing Audit Findings 242 Post Audit Activities 260 Appendix A – Confirmation Letter Appendix B – Roles and Responsibilities of the Audit Team Appendix C – Guide to Acronyms
List of Exercises
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 4 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise Page
No 1 Understanding HSE Management Systems 69 2 Effective Interviewing - Difficult Interview Situations 88 3 Effective Interviewing - Conducting Interviews 93 4 Preparing Working Papers 126 5 Assessing Strengths and Weaknesses 152 6 Gathering Audit Evidence -
Developing Verification Strategies 177
7 Sampling Strategies 1 190 8 Sampling Strategies 2 200 9 Evaluating Audit Results -
Specific ‘Local’ vs ‘Report’ 230
10 Writing Audit Findings -
Critiquing Audit Findings 255
Overview of Health, Safety and Environmental Auditing
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 5 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Overview of Health, Safety and Environmental Auditing
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 6 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Recognising auditing as a powerful tool for managing and communicating HSE performance, organisations around the world are developing audit programmes to: • Provide assurance to corporate officers about the company’s
compliance status with HSE requirements and good industry practices.
• Assess potential HSE liabilities. • Demonstrate effective management of HSE obligations to
companies’ key stakeholders.
Annual Report Audit Report
Community
Investors/Shareholders
CEO and Management
Facilities
Audit Programme
An increasing number of companies are including status reports on their HSE audit programmes in their annual reports and/or annual environmental reports (e.g., Union Carbide, ARCO Chemical, NOVA, Deere & Company, WMX, DuPont, etc.).
Overview of Health, Safety and Environmental Auditing
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 7 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Moreover, there are many sound business benefits of performing HSE audits.
Improve public image
Increase awareness and understanding of HSE hazards
Reduce employee and community exposure hazards
Improve compliance
Reduce costs by operating efficiently and safely
Reduce exposure to fines
To addressstakeholder
needsWhy Audit?
Overview of Health, Safety and Environmental Auditing – What is Health, Safety and Environmental Auditing?
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 8 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Employee Health and Safety
Employee Health and SafetyCompliance ManagementCompliance Management
Hazard ManagementHazard ManagementEnvironmental ProtectionEnvironmental Protection
Management ToolManagement Tool
Overview of Health, Safety and Environmental Auditing – Definition of Health, Safety and Environmental Auditing
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 9 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Auditing has become recognised throughout the world by various organisations as a useful tool in managing HSE issues. Although a variety of definitions have been established for HSE auditing, they all share common elements and themes. For example, auditing has been defined as…
• A systematic, documented, periodic, and objective review by a regulated entity of facility operations and practices related to meeting environmental requirements. (U.S. EPA Policy Statement on Environmental Auditing, July 1986)
• Internal evaluations by companies and governmental agencies to verify their compliance with legal requirements as well as their own internal policies and standards. (Environment Canada, May 1988, Environmental Protection Act, Enforcement and Compliance Policy)
• A series of activities undertaken on the initiative of an organisation’s management to evaluate environmental performance. (International Chamber of Commerce)
• An activity directed at verifying a site’s or organisation’s environmental, health, or safety status with respect to specific, predetermined criteria. (U.S. Environmental Auditing Roundtable)
Overview of Health, Safety and Environmental Auditing – Definition of Health, Safety and Environmental Auditing
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 10 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
…and the definition of auditing has broadened over the years to include the evaluation of management systems in determining a facility’s HSE performance status.
• A systematic, documented verification process of objectively obtaining and evaluating evidence to determine whether specified environmental activities, events, conditions, management systems, or information about these matters, conform to audit criteria, and communicating the results of this process to the client. (International ISO 14000 Standard)
• A management tool comprising a systematic, documented, periodic, and objective evaluation of the performance of the organisation, management system, and processes designed to protect the environment with the aim of: facilitating management control of practices which may have impact on the environment; and assessing compliance with company environmental policies. (Eco-Management and Audit Scheme—European Union)
• A systematic evaluation to determine whether or not the environmental management system and the environmental performance it achieves conform to planned arrangements, and whether or not the system is implemented effectively, and is suitable to fulfill the organisation’s environmental policy and objectives. (ISO)
Overview of Health, Safety and Environmental Auditing – Standards Against Which to Audit
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 11 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Since auditing has emerged as a systematic process intended to verify compliance with established standards and, in some cases, to review the effectiveness of management systems, it tends to be most effective for those HSE issues that are well defined by specific audit criteria.
Audit Criteria
Best Management Practices
Facility Standard Operating
Procedures (SOPs)
Corporate/Division Policies and Procedures
ManagementSystems
Laws and Regulations
(Federal/National, State/Provincial,
and Local)
Increasingly, management system requirements are being incorporated into the scope of the audit in recognition that well-designed and well-implemented management systems are an important vehicle for maintaining compliance over time.
Audit Approach
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 12 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Audit Approach
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 13 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Most companies engage in a three-phased audit process to fulfill the objectives of the audit programme.
Post-Audit Activities
On-Site Activities
Pre-Audit Activities
Three-PhasedAudit Approach
Audit Approach
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 14 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities
The pre-audit activities are designed to ensure that the audit team members and facility personnel understand the audit process, and their roles and responsibilities within that audit process, and are prepared to implement them.
Pre-Audit Activities Outcome
1. Initial planning activities
2. Document review
3. Audit plan preparation
Identification of key site issues
Preparation of detailed plans
Audit Approach
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 15 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
On-Site Activities
Arthur D. Little has developed a well-defined and systematic five-step process to provide organisation and structure to the on-site activities. This five-step process facilitates a review of how a facility manages its HSE obligations.
Basic Step
On-Site Activities Outcome
Opening meeting Tours Initial Interviews Document review
Strong working knowledge of key systems on site
Identification of key issues to review
Review of Step 1 information
Team meetings
Develop verification strategies
Reallocate team resources, if required
Identify potential impacts and management system weaknesses
Physical inspections Focused interviews Data and records
examination Verification testing
Analyse site programs Develop evidence to
substantiate findings Confirm status of
compliance Review data collected
Review factual accuracy of findings
Analyse/integrate findings of team
Prepare draft findings Confirm accuracy Identify potential root
causes
Daily debrief meetings Close-out meetings
Early, clear, consistent communication
Understand facility concerns Prepare preliminary draft
report
Step 1: UnderstandManagement Systems
Step 2: Assess Strengths & Weaknesses ofManagement Systems
Step 3: Gather AuditEvidence
Step 4: Evaluate AuditResults
Step 5: Report AuditFindings
In addition, the five-step process has been recognised by: • Hundreds of companies • Environmental Auditing Roundtable (EAR) • Canadian Environmental Auditing Association (CEAA) • International Chamber of Commerce in its Guide to Effective
Environmental Auditing • European Community in the Eco-Management and Audit
Scheme (EMAS)
Audit Approach
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 16 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Two Components of HSE Auditing
There are two important components in completing the five-step process: • Assessment—Process to develop an opinion (judgment) on
the strengths and weaknesses of the activities under review. • Verification—Process to determine adherence to specific
standards. Although assessment- and verification-based techniques play a key role in the audit process, they each provide the auditor with different information.
Principal Activities What you look for Output
Deficiencies, problems, risks, conformance with good practice (performed during Steps 1, 2 and 4)
Professional opinion as to performance with regard to accepted practice and recommendations for improvement
Evidence of compliance with regulations, policies, and procedures (performed during Step 3)
Statement of performance against standards with identification of shortcomings/areas for improvement
Assessment
Verification
Auditors need to use a mix of assessment and verification to complete the five-step process. Some skills are better suited toward assisting an auditor in the assessment stage while others suit the verification stage. Assessment is dependent upon the auditor’s knowledge of: • Site operations • Management systems (e.g., policies, procedures, etc.) • HSE requirements • Environmental technology Verification is dependent upon the auditor’s knowledge of: • Auditing skills and techniques • HSE regulations • Internal standards
Audit Approach
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 17 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
The balance between assessment and verification will vary depending on where you are in the five-step process.
Five-Step Audit Process
Step 5: Report Audit Findings
Step 2: Assess Strengths andWeaknesses of Management Systems
Step 3: Gather Audit Evidence
Step 4: Evaluate Audit Results
Step 1: UnderstandManagement Systems
Assessment-related activities Verification-related activities
Based on the information gathered, the auditor may move forward in the process or may need to reassess the information gathered in previous steps. For example, results obtained during gathering audit evidence (Step 3) or evaluating audit results (Step 4) can lead an auditor back to reassess his/her understanding of the management systems in place and/or the strengths and weaknesses of those management systems. The resulting finding may be one that identifies gaps in the facility’s management systems or the proximate or root cause of the finding.
Audit Approach
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 18 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Audit Skills, Techniques, and Tools
In implementing the five-step audit process, there are some essential audit skills, techniques, and tools that can be utilised to increase on-site efficiency and effectiveness.
Audit Skills and Techniques
Conducting interviews
Documenting the audit (working papers)
Writing audit findings
Gathering audit data
Using sampling strategies
Communicating audit resultsProtocols
Pre-audit
questionnaires
Discussion guides
Audit Tools
Protocols
Pre-audit
questionnaires
Discussion guides
Audit Tools
Audit Approach
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 19 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities
The purpose of the post-audit activities is to ensure that: • Audit results are communicated to the facility and appropriate
levels of management. • Audit findings are addressed.
Post-Audit Activities Outcome
1. Develop report
2. Distribute report
3. Develop and implementcorrective actions
Audit report (performance status)
Corrective action plan
4. Track corrective actions
Status reports
Audit Approach - Context for Health, Safety and Environmental Auditing
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 20 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Although HSE auditing has been recognised as a prominent and useful HSE management tool, it is not the only means by which an organisation manages its HSE issues. Auditing is one of many key activities within the typical HSE management processes.
Typical HSE Management Process
Identifying/managing issuesIdentifying/understanding requirementsIdentifying/evaluating risks
Influencing issues/requirementsSetting policy directionEstablishing performance standards/guidanceObtaining needed permits/approvals
Managing complianceManaging significant risksPreventing/reducing unwanted impactsRemediating past damagesResponding to emergencesImproving value to final customers
Measuring performanceAssuring performanceCommunicating performance
Auditing
Assessing Planning Implementing Reviewing
Supporting
Training & awareness Documenting / record keeping Managing information
Sound HSE management processes should include activities that address all four HSE management processes (assessing, planning, implementing, and reviewing) and incorporate the three key supporting activities (training and awareness, documenting/recordkeeping, and managing information). These same management processes are typically present at facilities in order to meet their HSE obligations.
Audit Approach - Context for Health, Safety and Environmental Auditing
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 21 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Auditing can provide a basis for guiding, measuring, and evaluating the performance of a facility’s HSE management activities. In general, an audit looks at and evaluates the management processes to comply with HSE requirements for each functional area being reviewed (e.g., air pollution control, hazardous waste management, industrial hygiene, employee safety, etc.). The specific activities that will be audited will depend upon the programme objectives. For example:
Typical HSE Management Process
Identifying confined spacesIdentifying jobs/tasks
Developing entry proceduresEstablishing rescue procedures
Issuing entry permitsMonitoring confined spacesUsing personal protective equipment
Reviewing cancelled permitsSupervisor review of confined spaces
Assessing Planning Implementing Reviewing
Supporting
Training & awareness- Training for entrants,
attendants, & supervisor
Documenting / record keeping- Permit retention- Training records- Written programme
Managing information- Training database
Audit Approach – Conducting Effective Audits
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 22 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Conducting effective and efficient audits is dependent upon: • Utilising the team’s collective experience and knowledge of
the site operations, HSE standards, and auditing skills and techniques.
• Prioritising the topics to review in terms of importance/impact. • Basing your prioritisation on a thorough review and
assessment of management systems and controls. • Developing verification strategies to gather data that will
provide meaningful insights regarding compliance. • Communicating, communicating, communicating—both oral
and written communications that occur within the team and externally with affected parties are essential factors influencing the success of auditing efforts.
Basic Steps in the Typical Audit Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 23 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Basic Steps in the Typical Audit Process – Overview
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 24 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Step 1: Understand Management Systems
Conduct opening meetingConduct orientation tourReview audit strategyUnderstand details ofmanagement systems
Step 3: Gather Audit EvidenceEvaluate what needs to be doneDetermine depth & rigor of reviewSelect types of evidenceneeded & methods to gatherthemCompare practices againstrequirementsDocument results
Step 2: Assess strengths& Weaknesses
Consider potential impactsEvaluate management systemsSet priorities for verification
Step 4: Evaluate Audit ResultsEvaluate audit resultsWrite audit findings
Step 5: Report Audit FindingsConduct exit meeting
Prepare Draft Report
Obtain Review Comments fromCorporate HSELaw departmentFacility management
Issue Final Report toFacility managementOperationsCorporate HSELaw department
Develop Action PlanDevelop proposed action(s) toaddress each findingAssign responsibility for corrective actionDevelop timetable
Conduct Follow-upTrack status of correctiveactionsConfirm closure of findings
Pre-Audit Activities On-Site Activities Post-Audit Activities
Select & Schedule Facility Audit
Plan the Audit:Correspond with the facilityAssemble & distribute background informationAssign & communicateaudit responsibilitiesConduct pre-audit meeting
Select Team Members & Confirm their Availability
Pre-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 25 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 26 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
These are primarily the responsibility of the team leader, and have therefore been treated in detail later in this manual. It is important to remember that the team participates in the pre-audit preparation, by reviewing the background information to develop an initial understanding of the facility’s operations, modifying audit protocols as/if necessary, making their own travel arrangements, having liaised with the team leader on the time for the pre-audit team meeting, obtained and reviewed applicable regulations, and organising any materials or equipment necessary to perform the audit. Before the audit team arrives at the opening meeting, it should know enough about the site to be able to formulate some preliminary hypotheses about the major risks and HSE issues. There are several ways to accomplish this: • Auditors’ basic familiarity with company operations, policies,
and procedures • Pre-audit questionnaire • Other background material provided by the facility • Conversations between the team leader/members and plant
personnel • Review of applicable regulations In our experience, it is not uncommon for team members to begin their pre-audit preparation only on the plane that is taking them to the site. Even if this were adequate for team members, it is definitely not sufficient for team leaders.
Pre-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 27 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Step 1: Understand Management Systems
Conduct opening meetingConduct orientation tourReview audit strategyUnderstand details ofmanagement systems
Step 3: Gather Audit EvidenceEvaluate what needs to be doneDetermine depth & rigor of reviewSelect types of evidenceneeded & methods to gatherthemCompare practices againstrequirementsDocument results
Step 2: Assess strengths& Weaknesses
Consider potential impactsEvaluate management systemsSet priorities for verification
Step 4: Evaluate Audit ResultsEvaluate audit resultsWrite audit findings
Step 5: Report Audit FindingsConduct exit meeting
Prepare Draft Report
Obtain Review Comments fromCorporate HSELaw departmentFacility management
Issue Final Report toFacility managementOperationsCorporate HSELaw department
Develop Action PlanDevelop proposed action(s) toaddress each findingAssign responsibility for corrective actionDevelop timetable
Conduct Follow-upTrack status of correctiveactionsConfirm closure of findings
Pre-Audit Activities On-Site Activities Post-Audit Activities
Select & Schedule Facility Audit
Plan the Audit:Correspond with the facilityAssemble & distribute background informationAssign & communicateaudit responsibilitiesConduct pre-audit meeting
Select Team Members & Confirm their Availability
Pre-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 28 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities
Schedule auditSelect team membersContact & coordinate with thefacilityArrange for travelGather & distribute auditmaterials (reference documents, working paperpads)
Administrative
Assemble & review background information & applicable regulationsDevelop audit assignments & areas to focus onReview & discuss audit team responsibilities
Planning
GoalHit the ground running
Pre-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 29 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
The pre-audit activities provide the foundation upon which the team members build their understanding of the facility’s operations and establish an audit strategy. They also provide the facility with its first impressions of the audit programme and set the tone for audit team/facility interactions. The pre-audit activities should meet the following objectives: • Provide the audit team with sufficient plant information to
enable the team members to develop a basic understanding of the facility, the processes, and the HSE management systems employed. For example, the audit team should understand: − The type of facility being audited (e.g., chemical
manufacturing, injection molding, pulp and paper, distribution center, research laboratories, etc.).
− The employee population (e.g., 60-person plant or 1,000-
person plant, extent of contract employees and contractors used on site, business organisation).
− The general applicability of regulatory requirements to the
facility’s operations (e.g., presence of wastewater treatment plant, hazardous waste generator status, number of permitted air sources, applicability of industrial hygiene programmes [e.g., respiratory protection, hearing conservation, bloodborne pathogens, etc.], presence of a fire brigade and/or spill response team, availability of routine and emergency medical personnel, etc.).
• Inform the facility as to audit programme goals, objectives,
and procedures. Typically, the more information facility personnel receive prior to the audit, the less anxious and more comfortable they will feel about the on-site portion of the audit.
Pre-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 30 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Using the information obtained, allow the audit team to develop a basic audit strategy prior to arrival at the site. In developing an audit strategy, the audit team should consider the following: • Audit objectives to be achieved, areas to receive emphasis,
and a preliminary division of responsibility among the team members.
• Questions or issues that need to be resolved during the preliminary meetings and points to be clarified.
• A preliminary agenda and schedule to be used during the audit.
To help facilitate the conduct of an efficient and thorough audit, the following pre-audit activities are frequently undertaken by the audit team leader and team members. 1. Corresponding With the Facility
Within the designated time frame established by the audit programme, the team leader should contact the facility to confirm the exact dates of the planned audit and to address the following: • Audit process and activities • Types of documents to be reviewed:
− Pre-audit information request − Pre-audit questionnaire
• Planning/logistical details:
− Safety and security requirements − Administrative/logistical details − Initial interview schedule
Pre-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 31 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Audit Process and Activities In general, the team leader should explain the audit process and the types of activities to be undertaken while on site. The audit programme objectives, purpose, and scope, and the types of interviews and tours that will be conducted, should be explained. In addition, the team leader typically requests that the facility prepare a short presentation describing the plant operations for the opening meeting. The facility should have a general understanding of how the audit will proceed and what type of time commitment will be required for the various levels of staff affected by the audit. Types of Documents to be Reviewed It is important to determine the background information to be requested and reviewed before arriving at the facility. A protocol represents a plan to be used by the auditors in conducting an audit. Protocols are produced in advance, for each of the issues to be audited, sometimes annotated with relevant regulatory standards references.The purpose is to provide the audit team with a step-by-step guide to collecting evidence about a facility’s programmes and practices included within the scope of the audit. In addition, the protocol identifies selected topics and requirements included in the audit and provides guidance regarding how the team may audit or review against those requirements. However, it typically does not include all applicable performance requirements that the team may need to review a facility’s compliance status. Typically, audit team members review applicable performance requirements prior to the audit to determine whether any other requirements are appropriate for in-depth review and verification. Other purposes of an audit protocol include: • Tool for audit planning.
Pre-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 32 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• Record for audit procedures—planned and completed. • Outline for working papers. • Record of changes in audit scope, procedures, etc., and the
rationale for any changes. • Basis for reviewing/critiquing an individual audit. Pre-audit information request. There are some materials that will be essential to have prior to the audit while other materials will only need to be looked at while on site. For some materials (e.g., training records, inspection logs, material safety data sheets, etc.), it is in the best interest of the audit team to see this material during the on-site visit. The team leader would be better advised to ask the plant to leave this information in existing files until the audit. As a result, the audit team will be able to see actual recordkeeping conditions and practices. The tables on the following pages list the types of information that are typically requested prior to the audit and information that should be available on site for review during the audit. The information listed in these tables serves only as an example. The type of documentation that should be requested will depend upon the staffing resources utilised (e.g., full-time corporate auditors, corporate or facility personnel with other full-time responsibilities, third-party auditors, etc.) and the time these resources realistically have for pre-audit preparation. When conducting pre-audit activities, it is important to keep in mind that the background materials should be requested early enough to ensure that there is enough time for: • The facility to assemble and send the information to the team
leader. • The team members to receive and review the materials prior
to the audit.
Pre-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 33 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Audit Information to be Sent in Advance
Facility plot plan or map
Directions to the facility
Visitor safety requirements (e.g., personal protective equipment, orientation, specialised training, special clearances, etc.)
Completed pre-audit questionnaire
Description of the facility’s operations/processes
Facility organisation chart, showing HSE responsibilities
Local laws, regulations, and ordinances related to the scope of the audit
List of current environmental licenses, certificates, and authorisations
Copies of permits for wastewater discharges and example air emission permits
Recent regulatory agency inspection/enforcement correspondence
Recent internal and intra-company environmental, health, and safety audit reports
Table of contents for facility-specific environmental, health, and safety policies and procedures
Pre-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 34 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Audit Materials to be Reviewed Upon Arrival On Site
Management objectives
Copies of all current environmental licenses, certificates, registrations, authorisations, and applications for such (including air and wastewater discharges; hazardous waste treatment, storage, or disposal activities; underground storage tanks; drinking water supplies; etc.)
Facility procedures and programme manuals (e.g., spill prevention plan, hazardous waste management contingency plan, respiratory protection plan, exposure control plan, hazard communication plan, etc.)
Effluent and emission monitoring reports
Training records
Hazardous waste manifests
Material safety data sheets (MSDSs)
Inventory of chemicals, including oils, in use or stored on site
Purchase orders for chemicals
Annual PCB reports (for the five years preceding the audit)
PCB transformers inspection records
Monitoring instrument calibration records and maintenance logs
Records of safety inspections, including reports of loss prevention surveys by insurance underwriters
First aid/dispensary records
Records of exposure monitoring and results, respirator fit testing, audiograms, etc.
Pre-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 35 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-audit questionnaire. In order to obtain a quick and concise facility profile to provide the audit team members with a basic understanding of the facility’s operations, a pre-audit questionnaire is typically sent to be filled out and returned by the facility. The pre-audit questionnaire should be a tool that is easily processed by the facility and provides the auditors with basic information enabling them to begin planning for the audit. An example of a pre-audit questionnaire is provided in Appendix A of this manual. Although the pre-audit questionnaire may provide the audit team members with a basis to begin their understanding, the auditors should critically review the information and remember that the information provided may not be completely reflective of the facility’s operations. For example, the facility may indicate on the pre-audit questionnaire that it does not have underground storage tanks on site. However, the facility may be unaware that there are two abandoned underground storage tanks that were inadvertently omitted from the facility’s initial spill prevention plan and have long been forgotten. During the on-site phase, the auditor may need to change his/her original audit strategy (based on the pre-audit information) to ensure that the protocol areas assigned are properly addressed. Planning/Logistical Details During the team leader’s communications with the facility contact, any planning details should be resolved, such as: • Safety and security requirements • Administrative/logistical details • Initial interview schedule
Pre-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 36 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Safety and security requirements. The team leader should clearly understand the personal protective equipment requirements for visitors who will be touring and inspecting any and all site areas (e.g., safety glasses, hard hats, safety shoes, safety clothing, etc.). The team leader should also obtain information regarding whether: • Safety orientations and specialised training (e.g., hydrogen
sulfide, respiratory protection, underground mine safety, etc.) are required to enter the general facility or regulated areas. The team leader should inquire as to the time needed to complete the necessary orientation or specialised training (e.g., 15 minutes, two hours, eight hours, etc.). Depending upon the training necessary, the team leader may need to adjust the audit schedule.
• Security clearances/passes are required for audit team
members or vehicles to enter the facility or regulated areas. • Restrictions apply to team members who are not citisens of
the country where the facility is located. • Escorts are required for team members touring and inspecting
facility areas. Administrative/logistical details. The team leader should cover the administrative and logistical details with the facility contact as early as possible in the pre-audit process. Administrative and logistical details that may need to be addressed include: • Requesting information regarding travel to the facility, as well
as lodging in the vicinity of the site. • Arranging for badges, clearances, car passes, safety
orientation/ training, and escorts. • Arranging for the audit team to stay after normal business
hours and observe second or third shift operations, where applicable.
Pre-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 37 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• Scheduling the opening meeting and setting a tentative time for the closing meeting.
• Establishing a time during the day when the audit team can
meet daily with the key HSE staff to discuss their preliminary findings and concerns. These meetings are typically referred to as daily debriefs.
• Arranging for meals (working breakfasts, lunches, or dinners). • Requesting that the facility prepare a brief presentation on the
facility’s operations for the opening meeting. • Arranging for a meeting room to be available to the audit team
during its visit. • Informing facility personnel of any documentation/equipment
(e.g., paper copy of the facility’s presentation, telephone directory, employee rosters, telephone, overhead projector, slide projector, printer, fax machine, photocopy capabilities, etc.) that needs to be made available for the audit team.
Initial interview schedule. The team leader may want to obtain the names of facility contacts to begin scheduling interviews with key HSE staff for the first one or two days of the audit. The benefits associated with having the team leader and the facility put together an initial interview schedule with key HSE staff are two-fold: 1) the facility is given an opportunity to feel a part of the audit process and to establish interviews at convenient times, lessening the impact on their day-to-day activities; and 2) the audit team will be able to begin their understanding of how the facility manages environmental, health, and safety areas immediately after the opening meeting and orientation tour.
Pre-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 38 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Finally, after arrangements have been agreed upon, the team leader should send a letter to the facility contact confirming those arrangements along with an outline of the audit process and activities, and the audit objectives, purpose, and scope. In addition to the pre-audit questionnaire, a list of the materials that the audit team would like sent prior to the audit is typically sent as an attachment to the confirmation letter. An example confirmation letter is provided in Appendix B. 2. Assembling and Distributing Background Information
Assembling Background Information Following the receipt of materials from the facility, the team leader should begin the task of assembling the available background information. This step, in general, will enable the team to develop an effective audit strategy tailored to an individual facility. This task typically involves: Reviewing facility information and responses to the pre-audit questionnaire and/or contacting the facility if background information has not been received within the specified time period. Contacting the facility to clarify any ambiguous or incomplete information received. Contacting the legal department, as appropriate, to ascertain whether the facility has any outstanding litigation or history of compliance problems. Obtaining relevant company policies and procedures and applicable federal/national, state/provincial, and local regulations. Identifying site-specific situations and requirements that may require modification of the standard audit protocols.
Pre-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 39 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assembling the appropriate audit protocols to be used for the audit, and assigning them to individual auditors according to their skills, experience and background. Distributing Background Information The background information collected by the team leader should be distributed in a timely fashion to allow the team members enough time to review the information prior to the audit. 3. Assigning and Communicating Audit Responsibilities
As the necessary background information is gathered and reviewed, the team leader should make an initial allocation of the functional areas (e.g., air pollution control, hazardous waste management, industrial hygiene, employee safety, etc.). This task involves matching the talent and expertise of the team members with specific tasks or protocols, as well as taking into consideration the team members’ prior audit assignments. These assignments are typically made by the audit team leader with input from the individual auditors. If the audit team is not in one geographical location, a conference call or video conference can be set up to establish and communicate the audit responsibilities. In addition to the assignment of functional areas, the team leader should clearly communicate all pertinent information and audit team responsibilities during the pre-audit phase. For example, audit team members are typically responsible for: • Reviewing the background information supplied by the team
leader to begin developing an initial understanding of the facility’s status with respect to each assigned functional area(s).
• Modifying the audit protocols, as appropriate, to incorporate
state/provincial, local, or facility-specific requirements or special facility conditions, plans, procedures, etc.
Pre-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 40 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• Making travel/lodging plans. • Obtaining and reviewing the applicable federal/national, state/
provincial, and local regulations. • Bringing the necessary audit materials/equipment for the audit
as directed by the team leader (e.g., personal protective equipment/ clothing, background information, audit protocols, working paper pads, regulations, computer, etc.).
4. Conducting Pre-Audit Meeting
A pre-audit meeting may be the last step of the pre-audit planning phase. This meeting is typically held immediately before the audit (i.e., evening or early morning prior to the audit) if team members are not located in the same geographical area. The purpose of the pre-audit planning meeting is to clarify any details regarding the protocols and to develop an overall audit strategy. An audit strategy is essentially an outline of the tasks that may need to be done in order to complete the five-step on-site audit process, how each task is to be accomplished, and the time required to complete each step. Audit protocols serve as a basis for developing this strategy. During this meeting, team members also: • Discuss and evaluate background information received from
the facility and determine if there are any overlapping areas (e.g., industrial hygiene and employee safety regarding personal protective equipment).
• Identify protocol steps and questions that have been modified
to reflect special facility conditions or unique state/provincial or local regulatory requirements.
Pre-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 41 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• Confirm that they understand how much time to allocate during the on-site phase to complete the five-step audit process. Typically, each auditor should allocate his/her time on site as indicated below.
Allocation of Time On Site
Percentage of Time Activity
15 Understand the management system for assigned topics
10 Assess the apparent strengths and weaknesses of those management systems
40 Gather audit evidence 30 Re-assess strengths and weaknesses and evaluate audit results 5 Formally report the audit findings to site management
In addition, some audit programmes find it beneficial, whenever possible, to conduct a pre-audit visit. During this pre-audit visit, the audit team leader will have a one-day meeting with facility personnel to review the audit programme objectives, scope, and approach; establish a preliminary interview schedule; and/or tour the facility to better understand the operations. The results from this pre-audit visit are shared with the audit team members prior to the audit to assist in the development of an audit strategy.
On-Site Opening Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 42 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
On-Site Opening Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 43 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Step 1: Understand Management Systems
Conduct opening meetingConduct orientation tourReview audit strategyUnderstand details ofmanagement systems
Step 3: Gather Audit EvidenceEvaluate what needs to be doneDetermine depth & rigor of reviewSelect types of evidenceneeded & methods to gatherthemCompare practices againstrequirementsDocument results
Step 2: Assess strengths& Weaknesses
Consider potential impactsEvaluate management systemsSet priorities for verification
Step 4: Evaluate Audit ResultsEvaluate audit resultsWrite audit findings
Step 5: Report Audit FindingsConduct exit meeting
Prepare Draft Report
Obtain Review Comments fromCorporate HSELaw departmentFacility management
Issue Final Report toFacility managementOperationsCorporate HSELaw department
Develop Action PlanDevelop proposed action(s) toaddress each findingAssign responsibility for corrective actionDevelop timetable
Conduct Follow-upTrack status of correctiveactionsConfirm closure of findings
Pre-Audit Activities On-Site Activities Post-Audit Activities
Select & Schedule Facility Audit
Plan the Audit:Correspond with the facilityAssemble & distribute background informationAssign & communicateaudit responsibilitiesConduct pre-audit meeting
Select Team Members & Confirm their Availability
On-Site Opening Activities – A “Typical” Audit
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 44 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Daily Debrief
8:00 - 8.30
Daily Debrief
8:00 - 8.30
Daily Debrief
8:00 - 8.30
Meet with HSE Staff8:00 - 8.30
Opening Meeting
11:00 - 12.00
Understand Details of Systems
8:30 - 12.00
Gather Audit Evidence
8:30 - 12.00
Continue to Gather Audit Evidence
8:30 - 12.00
Wrap Up Loose Ends
8:30 - 11.00
Conduct Close-Out Meeting
11:00 - 12.00
Lunch Lunch Lunch Lunch Lunch
Orientation Tour
1:00 - 2.00
Review Audit Plan
2:00 - 2.30
Assess Strengths and Weaknesses
1:00 - 2.00
Continue to Gather Audit Evidence
12:30 - 2.00
Gather Audit Evidence
2:00 - 4.30
Continue to Gather Audit Evidence
12:30 - 4.00
Team Meeting
4:30 - 5.30
Team Meeting
4:00 - 5.30
Understand Details of Systems
2:30 - 5.30
Evaluate Audit Results
2:00 - 6.00??
Travel to Site
Travel Home
Team Activities
Monday Tuesday Wednesday Thursday Friday
Individual Activities
On-Site Opening Activities - Purpose of the On-Site Opening Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 45 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
The opening meeting and the orientation tour are the initial activities undertaken by the audit team to begin their understanding of the facility’s management systems.
On-Site Activities
Opening meeting
Interviews
Opening meeting
Interviews
Key plans, procedures,policies
Document Review
Key plans, procedures,policies
Document Review
GoalUnderstand, who, what, where, how
Orientation tour
Observation
Orientation tour
Observation
The on-site opening activities are intended to provide the audit team with a broad and general overview of facility operations and issues. The audit team members typically obtain this broad overview by: • Conducting an opening meeting with facility management
upon arrival at the site to discuss overall facility operations and the organisational structure used to help facilitate the implementation of compliance activities.
• Conducting an orientation tour with key facility personnel to
obtain a general orientation to the plant, including its layout and size, location of operations, and location of those activities pertaining to the audit scope.
• Reviewing the audit strategy as a team to ensure that the
audit scope includes all the applicable audit topics and that resources are allocated appropriately, based upon the information gathered thus far, for each of the audit topics under review.
On-Site Opening Activities - Conducting the Opening Meeting
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 46 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
The purpose of the opening meeting is to: • Describe to facility personnel the overall objectives of the
audit programme, and the purpose, scope, and approach of the audit.
• Gain an initial overview of the facility’s programmes and
practices established to manage environmental, health, and safety issues relevant to the scope of the audit.
The opening meeting will, to a large extent, influence the overall outcome of the audit; therefore, it is important that this meeting be conducted in a professional manner which allows a comfortable exchange of information between the audit team and facility personnel.
The following table outlines the typical activities and topics included in the opening overviews of the: • Audit process presented by the team leader. • Site operations, programmes, and procedures presented by a
facility representative.
On-Site Opening Activities - Conducting the Opening Meeting
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 47 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Key Activities in Presenting the Overview of the Audit Process
Meet with the facility manager and key facility personnel
Have the team leader open the meeting
Use an opening meeting discussion guide
Use an overhead projector with viewgraphs, as appropriate
Encourage discussion among meeting participants
Tentatively schedule the exit meeting or reconfirm the previously established schedule
Reconfirm the time for the daily debriefs
Topics Typically Addressed in the Facility’s Overview
Production operations, capacities, raw materials, and product lines
Operating hours (staff and operations personnel)
Employee profile (number of salaried and hourly staff, union status)
Overview of HSE programmes
Any major HSE issues/problems
Community issues
Government inspections and compliance history
Regulatory climate
On-Site Opening Activities - Conducting the Opening Meeting
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 48 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Before beginning the orientation tour, the team leader should also reconfirm the following logistical and administrative arrangements discussed during the pre-audit phase:
Necessary clearances Safety rules and procedures Work space for the team Names of facility contacts
On-Site Opening Activities - Conducting the Orientation Tour
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 49 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
The purpose of the orientation tour is to provide the audit team a general familiarity with the plant layout and operations. Typically, this tour takes one to two hours. During the tour, the team members should: • Stay together. • Focus on obtaining an
overview, not on making inspections.
• Carry and annotate, as
necessary, a site plot plan. • Take notes on areas the
auditor wishes to revisit and employees/ staff that he/she may need to interview.
On-Site Opening Activities - Reviewing the Audit Strategy
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 50 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
After the team has gained a general understanding of the facility’s operations and issues, they should meet to review the audit strategy and discuss the next steps. For instance, the audit team should finalise an interview schedule for the remainder of the day or for the next day. Often, the team may need to sit down with the facility HSE coordinator(s) to clarify HSE roles and responsibilities as well as to gather additional information regarding the management systems in general and the overall applicability of performance requirements, if not already addressed in the pre-audit phase.
Team Member
Team Member
Team Member
Team Member
Team Member
7:00 am
8:00 am
9:00 am
10:00 am
11:00 am
12:00 pm
1:00 pm
2:00 pm
3:00 pm
4:00 pm
Breakfast with TeamBreakfast with Team
LunchLunch
Daily Debrief with Facility StaffDaily Debrief with Facility Staff
Time
On-Site Opening Activities - Example of an Opening Meeting Discussion
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 51 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
ChemplantOpening Meeting Discussion Guide
Audit Team: NPC Contacts:Lead Auditor: Plant Manager:John Clarke Jerry OsborneAuditor: Safety and Health Supervisor:Peter Tillson Pat DawsonAuditor: Environmental Co-ordinator:Paula Brown Chris Carson
Local Address: Facility Address:Best East Inn 875 Willow StreetAny road Anytown, AnyplaceAnytown, Anyplace (222) 222-2222(222) 333-3333
On-Site Opening Activities - Example of an Opening Meeting Discussion
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 52 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Purpose
The purpose of this audit is to conduct a comprehensive review of the environmental, health, and safety activities in order to: • Verify compliance with applicable national, regional (local,
state etc.), and local environmental, health, and safety laws and standards
• Verify conformance with corporate, company, and facility
environmental, health, and safety policies and procedures • Determine whether activities are consistent with good
environmental, health, and safety management practices and whether systems are in place and functioning
Scope
This environmental, health, and safety audit will address the facility’s compliance and management systems in the following areas: • Water Pollution Control, including Spill Prevention and Control • Air Pollution Control • Solid and Hazardous Waste Management • Employee Safety • Industrial Hygiene • Loss Prevention
On-Site Opening Activities - Example of an Opening Meeting Discussion
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 53 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Approach
The audit is based on: • A physical survey of the facility • Examination of a sample of environmental, health, and safety
administrative, technical, and operating records available at the facility
• Interviews and discussions with key facility management and
staff • Verification procedures designed to examine the facility’s
application of and adherence to environmental, health, and safety laws and regulations
Period of Review
January 3, 2000 through the last day of the audit Reporting
A hierarchical reporting scheme will be used: Who What When Facility/HSE Supervision All deficiencies noted When noted
Facility Manager All deficiencies noted Periodic, exit interview, final report
On-Site Opening Activities - Example of an Opening Meeting Discussion
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 54 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Report Schedule
A draft report will be issued within three weeks of the close-out meeting. After receiving comments on the draft report, a final report will be issued.
Understanding HSE Management Systems
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 55 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Understanding HSE Management Systems - Why Understand HSE Management Systems?
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 56 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Step 1: Understand Management Systems
Conduct opening meetingConduct orientation tourReview audit strategyUnderstand details ofmanagement systems
Step 3: Gather Audit EvidenceEvaluate what needs to be doneDetermine depth & rigor of reviewSelect types of evidenceneeded & methods to gatherthemCompare practices againstrequirementsDocument results
Step 2: Assess strengths& Weaknesses
Consider potential impactsEvaluate management systemsSet priorities for verification
Step 4: Evaluate Audit ResultsEvaluate audit resultsWrite audit findings
Step 5: Report Audit FindingsConduct exit meeting
Prepare Draft Report
Obtain Review Comments fromCorporate HSELaw departmentFacility management
Issue Final Report toFacility managementOperationsCorporate HSELaw department
Develop Action PlanDevelop proposed action(s) toaddress each findingAssign responsibility for corrective actionDevelop timetable
Conduct Follow-upTrack status of correctiveactionsConfirm closure of findings
Pre-Audit Activities On-Site Activities Post-Audit Activities
Select & Schedule Facility Audit
Plan the Audit:Correspond with the facilityAssemble & distribute background informationAssign & communicateaudit responsibilitiesConduct pre-audit meeting
Select Team Members & Confirm their Availability
Understanding HSE Management Systems - Why Understand HSE Management Systems?
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 57 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Purpose
The first step of the on-site activities is to develop an understanding of how the facility manages its HSE activities so that the audit team can: • Gather information that will be used to help set priorities
among the audit topics to review. • Gain insights regarding how effectively and efficiently an EHS
topic is being managed and, thus, establish a context for evaluating the audit results.
• Identify potential underlying causes that contribute to
compliance-related deficiencies. Other Driving Forces
In addition, audit standards—such as the ICC Charter for Sustainable Development, ISO 14000, EMAS, and BS7750—require a thoughtful review and evaluation of the facility’s systems for managing HSE obligations. Our approach to Step 1: Understand Management Systems will be to: • Explain what HSE management systems are and some of the
specific activities involved. • Describe how the auditor should go about understanding HSE
management systems in Step 1 of the audit process.
Understanding HSE Management Systems - What Are Management Systems?
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 58 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
In the simplest sense, HSE management systems are the actual processes used by a facility to achieve and maintain conformance with established standards, including programmes, policies, equipment, administrative controls, etc. The auditor’s role is to find out how the facility really manages its HSE obligations.
Source: Systemation, January 15, 1959, published by Systemation, Inc.,
Colorado Springs, Colorado.
Understanding HSE Management Systems - What Are Management Systems?
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 59 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
With the onset of emerging international environmental management standards, HSE management systems can be described in terms of key processes that are aligned with an appropriate organisation and resources.
Organisation Resources
Supporting
Assessing Planning Implementing Reviewing
“How are we doing?”“What do we needto manage?”
“How shouldwe manage it?” “Let’s manage it!”
Training and awareness Documenting/recordkeeping Managing information
Typical HSE Management Processes
Foundation
Understanding HSE Management Systems - What Are Management Systems?
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 60 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
The key processes in management systems can be defined and categorised as follows:
Basic Step On-Site Activities Outcome
Assessing is used by the facility for identifying conditions / aspects & materials on facility that have HSE implications, & for determining the applicability of regulations
Identify / evaluate HSE issues Identify on-site hazards Review HSE activities and projects Review applicable regulations
Planning is used by the facility for designing and establishing programs & systems for HSE compliance
Formulate HSE strategies & policies with clear objectives & targets that reflect the importance of the HSE issues applicable on site
Develop HSE procedures for compliance & record keeping activities, as well as prevention plans with specific operating criteria
Identify & design engineered controls & equipment Develop emergency response procedures; create
procedures regarding critical HSE activities / issues or departures from established criteria
Implementing is used by the facility for ensuring effective and consistent implementation of its HSE programs
Acquire permits Disseminate policies & procedures Assign & communicate roles & responsibilities Install, calibrate & maintain engineered controls Handle situations that deviate from an established
standard Undertake activities in accordance with established
schedules Reviewing is used by the facility for measuring &
assuring HSE program effectiveness Conduct drills Conduct inspections / self-audits in accordance with
documented procedures Review compliance data / performance Track continuous improvement Review the effectiveness of management systems Undertake corrective actions in response to identified
directors from procedures or established criteria
Assessing“What do we need tomanage?”
Planning“How should we manageit?”
Implementing“Let’s manage it”
Reviewing“How are we doing?”
Understanding HSE Management Systems - What Are Management Systems?
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 61 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Supporting processes (training and awareness, documenting/ recordkeeping, and managing information) tend to cross over the four key categories of assessing, planning, implementing, and reviewing.
Training and Awareness
Training programmes, both formal (e.g., classroom) and informal (e.g., on-the-job), in-house and external, for ensuring that each HSE programme element is understood and implemented properly. Programmes and/or other means for communicating HSE procedures, and addressing critical HSE issues and potential consequences of departure from specified operating procedures. Competent staff performing HSE work.
Documenting/ Recordkeeping
Systems for legibly documenting and dating activities (e.g., routine reports, inspections, audits, training, etc.) conducted during the implementation of the HSE programmes. Systems for retaining documentation in an accessible and orderly manner for periods of time required by regulation or internal standards and for removing obsolete documents.
Managing Information
Systems for communicating relevant information across the various levels and functions of the organisation. Mechanisms to track and evaluate compliance information. Systems for responding to relevant communications from external parties.
Organisation and resources should be aligned to support effective HSE management systems.
Organisation
Clear assignments and understanding of HSE responsibilities and accountabilities among HSE and line management. Visible HSE commitment and support by management. Appropriate high-level HSE reporting.
Resources
Sufficient number of qualified HSE staff. Adequate HSE staff/responsibilities to cover all business/organisation groups within the facility. Availability of needed financial and technological resources.
Understanding HSE Management Systems - The Methodology for Understanding HSE Management Systems
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 62 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
There are three principal approaches used to understand HSE management systems within the context of an audit.
Look at Key Equipment/Facilities Review Key Programme Documents
Talk to Key People
Contains (or manufactured with, if applicable)Contains (or manufactured with, if applicable)(insert name of substance), a substance which har ms(insert name of substance), a substance which har ms
public health and environment by destroying ozonepublic health and environment by destroying ozonein the upper atmosphere in the upper atmosphere
WarningWarningContains (or manufactured with, if applicable)Contains (or manufactured with, if applicable)(insert name of substance), a substance which har ms(insert name of substance), a substance which har ms
public health and environment by destroying ozonepublic health and environment by destroying ozonein the upper atmosphere in the upper atmosphere
WarningWarningContains (or manufactured with, if applicable)Contains (or manufactured with, if applicable)(insert name of substance), a substance which har ms(insert name of substance), a substance which har ms
public health and environment by destroying ozonepublic health and environment by destroying ozonein the upper atmosphere in the upper atmosphere
WarningWarning
Contains (or manufactured with, if applicable)Contains (or manufactured with, if applicable)(insert name of substance), a substance which har ms(insert name of substance), a substance which har ms
public health and environment by destroying ozonepublic health and environment by destroying ozonein the upper atmosphere in the upper atmosphere
WarningWarningContains (or manufactured with, if applicable)Contains (or manufactured with, if applicable)(insert name of substance), a substance which har ms(insert name of substance), a substance which har ms
public health and environment by destroying ozonepublic health and environment by destroying ozonein the upper atmosphere in the upper atmosphere
WarningWarningContains (or manufactured with, if applicable)Contains (or manufactured with, if applicable)(insert name of substance), a substance which har ms(insert name of substance), a substance which har ms
public health and environment by destroying ozonepublic health and environment by destroying ozonein the upper atmosphere in the upper atmosphere
WarningWarning
0102030405060
0102030405060
Understanding HSE Management Systems - The Methodology for Understanding HSE Management Systems
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 63 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Technique Activities
Talk to Key People
Talk to several people (e.g., line management, HSE staff, operating personnel, maintenance personnel) to obtain a comprehensive understanding of the activities that are in place to manage compliance and to increase your sense of confidence in the information obtained. Summarise the information obtained from each interviewee to verify the completeness of your understanding. Probe to understand inconsistencies in the information obtained. Examples of what auditors should endeavor to understand during interviews include:
• What is meant by the scope of the facility’s programmes? For example, when the facility says that, “All employees receive hazard communication training,” does this mean all employees, all employees who work in certain areas, or something even narrower?
• How does the facility handle seasonal, situational, or “non-normal” activities? For example, how does a particular activity work on the off-shift, or when a key person is on vacation, etc.?
• How does the facility develop data for preparing compliance-related reports, for determining compliance, or for identifying HSE-related problems? To understand this, the auditor may, for example, want facility personnel to describe or demonstrate how they reconcile the monthly inventory for an aboveground storage tank
Look at Key Equipment/Facilities
Walk around the facility to understand the nature of the HSE issues that need to be managed and the types of engineered controls in place. (This is primarily accomplished during the orientation tour.) At this stage, auditors should focus on understanding the nature and rationale of the engineered controls used to manage HSE hazards.
Understanding HSE Management Systems - The Methodology for Understanding HSE Management Systems
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 64 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Technique Activities
Review Key Programme Documents
Briefly review procedures, plans, etc., that explain how the facility manages HSE obligations. For example: • Compliance-related programme documents • Operational procedures • Checklists or inspection forms • Training programme description At this stage, auditors should focus on understanding: • How the programme is supposed to work. • Tools used by the facility for ensuring that the programme
works as designed and is effective. There are some key questions auditors should try to answer when understanding each stage of the management process.
Understanding HSE Management Systems - The Methodology for Understanding HSE Management Systems
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 65 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Management Process
Things the Auditor Wants to Learn
Assessing • How are regulations tracked (i.e., identifying, tracking, interpreting, and communicating regulations)?
• How are HSE risks and effects assessed (e.g., waste stream inventory, air emissions inventory, natural resources consumed, likelihood and magnitude of unplanned events, potential exposure to hazardous substances, etc.)?
• How does the facility manage changes in procedures or facility design (e.g., HSE review and consideration for new products, processes, equipment, acquisitions and divestitures, maintenance modifications, etc.)?
Planning • What type of basic compliance programmes (e.g., permitting, monitoring, training, recordkeeping, reporting, etc.) have been or are being established and do they include critical operating parameters and schedules?
• How does the facility prepare for emergencies (e.g., developing scenarios, response capability, response plans, etc.)?
• What type of issue-specific risk reduction programmes (e.g., groundwater monitoring, pollution prevention, waste management practices, spill containment programmes, ergonomics, engineered controls, etc.) are developed?
• What engineered controls or alarms are in place to help achieve desired results?
• What measures have been taken to reduce the likelihood of nonconformance with established criteria?
• What types of policy and related goals and objectives are established (e.g., vision statements, basic policies and guiding principles, specific goals and milestones, etc.)?
• What strategies are developed for managing HSE risks and effects?
Understanding HSE Management Systems - The Methodology for Understanding HSE Management Systems
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 66 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Management Process
Things the Auditor Wants to Learn
Implementing • Has the facility fully implemented the various HSE programmes?
• How does the facility maintain operating equipment (e.g., preventive maintenance programmes, testing and monitoring, etc.)?
• How is nonconformance with established criteria handled?
Reviewing • Are facility operations being inspected (e.g., routine walk-throughs, use of checklists, etc.)?
• How are HSE effects being measured? • Does the facility analyse its performance (e.g., evaluate
“findings,” “lessons learned,” trends, etc.)? • Are programmes in place for developing, implementing,
and tracking corrective actions? Training and Awareness
• What formal training have key personnel had to assist them in performing their HSE tasks and functions?
• What training and awareness activities are conducted to provide an understanding of HSE obligations and responsibilities?
• What type of training programmes (e.g., compliance-related training, emergency drills, on-the-job training, etc.) are available?
• What types of experience or background are required to perform HSE tasks?
Documenting and Record-
keeping
• Are procedures and practices for both compliance and remediation generally written down?
• What records are routinely developed and retained in carrying out various tasks and functions?
• What exception reports are developed? • What is the general nature or character of the
documentation that is developed? • How does the relative importance of HSE activities
correspond to the nature and level of documentation that is developed?
• Where is information retained? • How long is information retained?
Understanding HSE Management Systems - The Methodology for Understanding HSE Management Systems
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 67 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Management Process
Things the Auditor Wants to Learn
Managing Information
• How is important HSE information conveyed to personnel?
• Can the facility readily access HSE information? • How does the facility ensure that reports are submitted
in a timely manner and that essential records are retained?
• Are there procedures in place for responding to external requests for information?
Organisation • How are responsibilities and accountabilities defined, established, and communicated?
• How are assignments of responsibilities reinforced? • Are any key responsibilities overlapping, shared, or
conflicting? • What potential exists for “conflict of interest” in
accomplishing key HSE tasks and functions? • How is authority granted to carry out assigned
responsibilities? Resources • How have responsibilities for implementation been
communicated to personnel who need to know? • Are there sufficient resources to carry out the various
HSE programmes? • Who has the authority to waive adherence to, or
conformance with, an established standard or requirement and are deviations recorded?
• Are there perfunctory approvals—authorisations without understanding what is involved?
Understanding HSE Management Systems - How Much is Enough?
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 68 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Auditors frequently wonder whether they have collected enough information and the right kind of information to substantiate their understanding of a facility’s key management systems/programmes and physical controls. Listed below are tips for determining how much is enough. You have probably gathered enough information if:
You understand both system design (e.g., facility policies and procedures regarding regulatory tracking) and implementation (e.g., availability of the latest regulations on site).
You have interviewed all of the key personnel involved in key
HSE functions or tasks, and you can summarise to their satisfaction the basic programmes, practices, and control systems.
You understand the probable cause(s) of any differences
between management’s and employees’ perspectives, or between environmental and operational personnel’s perspectives.
You understand the types of activities that are applicable and
the range of activities being managed by the facility.
You understand the roles and responsibilities of HSE staff, the mechanisms to share relevant information, and the processes for retaining information in an accessible manner.
Once the auditor has developed a basic understanding of the management systems associated with assigned audit topics, he/she is now ready to move on to Step 2 and assess the systems under review.
Exercise 1A Understanding HSE Management Systems - Confined Space Entry
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 69 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Objective
The purpose of this exercise is to demonstrate how the auditor obtains information to understand the management systems in place at a facility (Step 1 of the audit process). Background
Based upon the preliminary information provided to you by the facility, you know the following: • The facility has confined spaces, which are entered by facility
personnel and contractors. • The facility has a confined space entry programme along with
permits. • The facility is a chemical manufacturer that handles a variety
of toxic and flammable substances. Instructions
Describe the steps you would take to understand the management systems used by the facility to implement the confined space entry programme. Include in your description: • Whom you would talk to. • What questions you would ask. • What documents and physical facilities you would want to look
at. • Any other activities you would conduct while completing Step
1 of the audit process for this topic.
Exercise 1A Understanding HSE Management Systems - Confined Space Entry
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 70 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Whom would you talk to? What questions would you ask? What documents and physical facilities would you look at? Other activities?
Exercise 1A – Potential Answers Understanding HSE Management Systems - Confined Space Entry
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 71 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Whom would you talk to? HSE Manager Maintenance Manager Maintenance Staff What questions would you ask? What is the procedure for confined space entry? Where are they formalised (HSE manual)? To Maintenance Staff – What do they do before and while entering confined space? Training PPEs? What documents and physical facilities would you look at? Permit to work Inventory of confined spaces Other activities? Ask to go and see the confined spaces Ask to observe if any maintenance is scheduled during the audit
Exercise 1B Understanding HSE Management Systems - Spill Control
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 72 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Objective
The purpose of this exercise is to demonstrate how the auditor obtains information to understand the management systems in place at a facility (Step 1 of the audit process). Background
Based upon the preliminary information provided to you by the facility, you know the following: • The facility has aboveground storage tanks containing
petroleum and chemical products. • The facility has tank truck loading/unloading operations. • The facility has a spill control team that has been provided
training. • A Spill Prevention Control and Countermeasures plan is on
site. Instructions
Describe the steps you would take to understand the management systems used by the facility to implement the spill control programme. Include in your description: • Whom you would talk to. • What questions you would ask. • What documents and physical facilities would you want to look
at. • Any other activities you would conduct while completing Step
1 of the audit process for this topic.
Exercise 1B Understanding HSE Management Systems - Spill Control
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 73 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Whom would you talk to? What questions would you ask? What documents and physical facilities would you look at? Other activities?
Exercise 1B – Potential Answers Understanding HSE Management Systems - Spill Control
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 74 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Whom would you talk to? HSE Manager Maintenance Manager and staff who supervise loading/unloading Spill control team What questions would you ask? What is the procedure for spill control and countermeasure? Who is in charge? What training is carried out? Is external personnel (truck drivers) trained/supervised? What happens if a spill occurs on a Sunday night? What documents and physical facilities would you look at? Spill prevention control and counter measure plan (Accident/Incident/Near misses register) – Loading/unloading, secondary containment Spill control kits – Waste registers Hazard material inventory – Quantities and storage locations Other activities? Observe loading/unloading
Effective Interviewing
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 75 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Effective Interviewing
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 76 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Interviewing is one of the primary techniques used in gathering audit information. Interviews provide auditors with: • A time-efficient way of gathering both broad general
information and specific details from people who should know. • A means to confirm hypotheses about site conditions,
changes, needs, and opportunities. • A current and credible source of facts and perceptions that
complement written information and physical observations. Thus, good interviewing skills are essential to the successful completion of the audit. While in one sense interviewing is a skill that comes naturally to most people, good interviewing techniques, which emphasise interaction between interviewer and interviewee, must be developed. By remembering some basic elements of good interviewing skills, the interviewer not only will be successful in gathering the information he/she desires, but will also find the interview process much more pleasant.
Effective Interviewing – The Basic Interview Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 77 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
A significant portion of the auditor’s field time is spent asking questions of and engaging in discussions or conducting interviews with facility staff. While the setting, duration, and degree of formality of such interviews can vary, all audit interviews follow this common pattern.
Planning
Opening
Conducting
Closing
Documenting
1. Planning the Interview
Prior to conducting the interview, the auditor should identify personnel to be interviewed, outline the objectives to be accomplished, and plan how to maximise the effectiveness of the interview. Key considerations include: • Iron out logistics—set a specific time and place for the
interview. • Define the desired outcome—identify the types of information
desired and/or areas to be addressed. The types of information gathered during interviews can be characterised as “hard” or “soft.”
Effective Interviewing – The Basic Interview Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 78 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Hard • Quantitative data regarding emissions, exposure monitoring, etc.
• Records prepared for compliance purposes • Historical information about a site
Soft • Why did something (not) happen? • How does the process/system work? • How does the work really get done?
Most HSE audit interviews mix “hard” and “soft” information needs. • Organise your thoughts—develop a logical sequence of
questions. • Be prepared—HSE interviews take place under all types of
distracting conditions, so plan appropriately.
2. Opening the Discussion
The quality of information gathered during an interview is closely related to the interviewee’s sense of comfort. The level of openness that develops during an interview, along with the interviewee’s confidence in the topic being discussed, depends a great deal on the rapport and atmosphere established during the initial contact.
Effective Interviewing – The Basic Interview Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 79 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
In constructing an atmosphere where positive rapport can be established, auditors should follow these general guidelines: • Arrive on time. • Introduce yourself. • Ensure appropriateness of time. • Explain the purpose of the discussion. • Explain how the information gathered in the discussion will be
used. In addition, auditors should strive to build the desired sense of comfort and confidence by: • Cultivating a friendly, nonthreatening discussion. • Attempting to respond to the interviewee’s social style. For
example:
Social Style Orientation Example Interviewee
Questions/Responses Analytical/ Thinker
Technically-oriented
“What’s your methodology?” “How will the results be used?”
Driving/Doer Results-oriented
“What can I do for you in the next ten minutes?” “What is the outcome of all this time spent talking with auditors?”
Amiable/ Feeler
Relationship-oriented
“How are you enjoying our plant/town?” “Why are we having this interview?”
Expressive/ Intuitionist
Social recognition-oriented
“Did you notice how effectively our waste management process operates?” “We believe we’ve made the most improvements in this area of anyone in the industry.”
Effective Interviewing – The Basic Interview Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 80 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• Refraining from portraying a condescending, arrogant, know-it-all attitude.
• Acting in a supportive and nonjudgmental manner. • Ensuring that the auditor and the interviewee are “on equal
ground.” 3. Conducting the Interview
Style, Flow, and Tone Once a comfortable interview setting and rapport have been established, the auditor should focus toward obtaining specific information from the interviewee. Some examples of specific items that an auditor should address include: • Request a brief overview of the interviewee’s job. • Discuss the interviewee’s responsibilities in relation to the
topic(s) being reviewed. • Use language that the interviewee can understand. • Start with some general questions, then gather more detailed
information.
• Resolve ambiguities through constructive probing. • Do not exceed the agreed-upon time limit without first
obtaining the interviewee’s approval. • Provide feedback as appropriate.
Effective Interviewing – The Basic Interview Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 81 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Establishing an appropriate interview style, flow, and tone is essential to the interview process. For example: Style • Avoid interrogation
• Be empathetic • Avoid defensiveness • Be calm, objective, and non-partisan • Be courteous, alert, and responsive
Flow • Avoid disjointed transitions that damage rapport (e.g., cutting the interviewee off because you are trying to transition to a new topic)
• Use word association to change topic focus (e.g., if the interviewee mentions his/her training, use this opportunity to raise any training questions)
Tone • Be genuine and take an interest in the interviewee’s responses • Use a soft, friendly voice
Types of Interview Questions Appropriate questioning should be utilised to obtain the desired information. For example: Type of Question
Example Questions Typical Response Outcome
Relative Value of Response Information to Auditor
Leading “Of course you notify the state of a planned discharge.” “You do test the wells every month, don’t you?”
Often unintentionally “lead” the interviewee to the desired answer.
−
Yes/No “Do you have a spill response procedure?” “Have you conducted waste audits?”
Usually receive only a “yes” or “no” answer. +
Close-Ended
“What is your current production capacity?” “What is the capacity of your wastewater treatment plant?”
Usually receive a one-word answer. +
Effective Interviewing – The Basic Interview Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 82 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Type of Question
Example Questions Typical Response Outcome
Relative Value of Response Information to Auditor
Open-Ended
“How are new chemicals selected for use in your process?” “What analyses are performed to assess HSE risks and liabilities?”
May prompt a more detailed response.
+++
How Do You Know
“How does your facility ensure that workers receive appropriate HSE training?” “How does your facility ensure that PCB transformer inspections are performed?”
Can provide the most insight into how things are actually managed. +++
Active Listening Another important component to conducting successful interviews involves active listening. Active listening allows the interviewer to: • Summarise information accurately. • Test the interviewee’s understanding of the topic being
discussed. • Probe for confirmation. • Facilitate the interview. • Display empathy/establish greater rapport. All auditors should develop the following listening techniques: • Wait until the current question is answered before asking
another question. • Encourage the interviewee in a nonverbal manner (e.g.,
maintain eye contact, display attentiveness, etc.).
Effective Interviewing – The Basic Interview Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 83 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• Interrupt only if you sense avoidance—people often feel more secure when given an opportunity to speak without being interrupted.
• Imagine the interviewee’s situation (i.e., put yourself in the
interviewee’s job scenario). • Listen for emotions and attitudes as well as facts. In addition, the interviewer should remember to listen over 90 percent of the total interview time. Paraphrasing/Summarising Information Learned The technique of paraphrasing can aid the auditor in confirming or clarifying something said or implied by the interviewee. There are three levels of paraphrasing: Level Accomplishment Example Paraphrase
1 Confirms or clarifies expressed thoughts and feelings
“So there are three factors that determine the present situation...”
2 Confirms or clarifies implied thoughts and feelings
“You would really like to change this situation...”
3 Surfaces core thoughts and feelings
“You are concerned that your company’s engineering approach is outdated...”
In using this technique, it is important to: • Paraphrase completely. • Match the levels of emotional intensity and factual content. • Use levels 1 and 2 freely; treat level 3 with caution.
Effective Interviewing – The Basic Interview Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 84 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Nonverbal Communication A significant portion of the information exchanged in any interview is done nonverbally. Nonverbal communication is a combination of the meanings expressed by the interviewer and the interviewee through gestures, facial expressions, voice inflections, and posture. Auditors should be aware of nonverbal communication and pay attention to the following: • Shake hands. • Maintain eye contact. • Keep the right distance. • Tolerate silence. • Mirror the interviewee’s body movements. • Be sensitive to culture and customs (especially outside of your
native country). Actions to Avoid In addition to the positive actions that should be taken in preparing for and conducting the interview, there are also some actions the interviewer should avoid. These include: • Debating with the interviewee or wasting time disagreeing on
any one point. • Forcing a meeting with the interviewee if he/she is otherwise
occupied. • Rushing the interview. • Amplifying criticism given. • Using sarcasm or subtle humor. • Telegraphing your assessment of the interview. • Jumping to conclusions/not watching your assumptions. • Communicating incomplete or unsubstantiated findings or
conclusions. • Using a tape recorder.
Effective Interviewing – The Basic Interview Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 85 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
4. Closing the Interview
The closing phase of the interview should appear to reach a natural conclusion. In closing the interview, the auditor should strive to: • Pace the interview to avoid rushing and, if necessary,
schedule additional interview time when mutually convenient. • Summarise and confirm information learned. • Try to bring closure to sensitive issues/questions raised earlier
in interview. • Never exceed the allotted time without confirming with the
interviewee. • Ensure that the interviewee is psychologically ready to leave
the interview (e.g., not unduly worried, upset, etc.). • End with an “open door” for communication in either direction. • End on a positive note—thank the interviewee for his/her time
and cooperation. Because the interviewee may feel that this phase of the interview is his/her final opportunity to be heard, closing questions used by the auditor can be extremely productive. Some examples include: • Is there anything you expected me to ask about that I did not
mention? • A little earlier you said that... • We have heard that...Is that really the way it is? • Can you elaborate on that point? • Would others say the same? • Is it like that at other times/places/divisions/companies? • Playing devil’s advocate, would some people fundamentally
disagree? These questions clarify open issues and help to resolve contradictory statements.
Effective Interviewing – The Basic Interview Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 86 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
5. Documenting the Interview
Most of us have limited memories, especially when it comes to recalling what people have told us; if we have no record of an interview and have to rely entirely on our memories, we are bound to make mistakes and omit important points. Therefore, in audit interviews, it is safer to take notes and not to rely on an imperfect memory, especially when you are interviewing several people during the course of any given day, which is most often the case during audits. The only problem with taking working paper notes is how to do so without distracting or intimidating the interviewee. The break in eye contact that occurs during the writing process can be distracting for the non-writing individual. Similarly, not knowing what or why an auditor is writing may intimidate the interviewee. With these points in mind, review the following suggestions on how to take notes during interviews: • Never try to hide the fact that you are taking notes. Rather,
draw the interviewee’s attention to what you are doing by explaining the need to take notes and involving the interviewee in the process. If necessary, address the issue regarding the confidentiality of working papers.
• Offer the interviewee the opportunity to review the notes taken
during your interview to put him/her at ease with the process, if necessary.
• Keep your working paper pad within easy reach, such as
attached to a clipboard on a desk or table. Always make sure you have a pen handy—and that it works.
Effective Interviewing – The Basic Interview Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 87 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• Take notes about facts that are relevant to the audit. Avoid listing points that do not relate to the audit topics of concern. For example: − Record name, title, and job description of person with
whom you spoke. − Reference the appropriate protocol step(s) addressed in
the interview. − Note relevant interview information. − Highlight key statements/observations.
• Put the interviewee’s words in quotation marks to distinguish
them from your own comments. • Spend time immediately following the interview summarising
the key points obtained from the interview in your working papers
Exercise 2 Effective Interviewing - Difficult Interview Situations
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 88 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Objective
The purpose of this exercise is to identify different types of interview situations that could potentially occur while conducting an audit and to discuss various strategies useful in resolving difficult interview situations. Instructions
Read the following scenarios and answer these questions for each: • What is happening in this interview? • What would you do in this situation? Scenario A
You are responsible for assessing the environmental training provided to site personnel during an environmental audit of the company Mining Samples. During an interview with the lab training co-ordinator , Jo Lopes, you ask ‘Could you tell me what types of environmental training site employees receive?’ Mr. Lopes responds ‘Sure. All site employees attend an initial eight-hour environmental awareness course during their new employee orientation week. Employees also attend and must successfully pass a first aid/cardiopulmonary resuscitation course.’ You say, ‘Getting back to the environmental awareness course, can you describe the types of information that are discussed in this training?’ Mr. Lopes replies, ‘Sure. We train new site employees on environmental issues such as pollution prevention and air and water contamination. Our first aid courses instruct site employees on how to respond to health and safety emergencies that may occur here at Mining Samples. Let me get a copy of the training material for you.’
Exercise 2 Effective Interviewing - Difficult Interview Situations
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 89 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Scenario B
During an internal audit of the company Big Motors, you are in charge of evaluating the company’s health and safety management systems. The facility manager suggests you talk with Harry Thomas, the facility health and safety co-ordinator to obtain a detailed understanding Big Motor’s health and safety programmes. After leaving three messages with his secretary, Mr. Thomas returns your call. You explain to Mr. Thomas that you wish to meet with him at a mutually convenient time. He responds, ‘I don’t have any free time to meet in person. Why don’t you ask your questions now?’ You begin, ‘Could you describe in general the different health and safety programmes that are in place at this facility?’ Mr. Thomas states, ‘You can find all that information in the site health and safety manual.’ You ask, ‘ Where could I find a copy of that manual?’ Mr. Thomas replies, ‘I’m not sure. There’s probably one floating around in my office somewhere.’ You then inquire, ‘Well, could you briefly describe your duties to me?’ Mr. Thomas answers, ‘I’m in charge of all the programmes listed in the site health and safety manual.’ Scenario C
As the auditor responsible for assessing the management of hazardous wastes at Gesher Manufacturing, you are in the middle of a scheduled interview with Richard Seaton, the facility maintenance manager. Although slightly preoccupied, Mr. Seaton has been responding amicably to your questions. As you listen to Mr. Seaton explain the major product lines at the facility, you recall the environmental co-ordinator, Jessica Schaeffer, did not mention any site waste recycling activities. You ask Mr. Seaton, ‘Does the facility participate in any types of pollution prevention or recycling activities?’ Mr. Seaton’s demeanor rapidly changes as he responds, ‘No. I don’t believe in those types of new-fangled practices.’
Exercise 2 Effective Interviewing - Difficult Interview Situations
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc c 90 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Scenario D
On the last day of the audit of the Orecrush Mill water pollution control programme, you realise you have a specific question about the site’s backflow prevention devices. You decide to call Mr Ahraby, the wastewater treatment operator you spoke with earlier in the week. After calling the treatment plant, you learn that Mr. Ahraby is out sick and that Mr Karami, one of the on-duty water treatment operators, is covering for Mr. Ahraby. You ask to speak with Mr Karami; after he answers the telephone, you explain that you spoke with Mr. Ahraby at length earlier in the week and have a specific question regarding water backflow prevention devices. Mr Karami answers, ‘I’d be happy to answer any of your questions. In Mike’s absence, I’m responsible for the entire wastewater treatment plant. Do you know this plant handles 20,000 gallons of wastewater a day, seven days a week? That’s over seven million gallons annually.’ You interrupt, ‘How interesting. Can you tell me if the backflow prevention devices are routinely inspected?’ Mr Karami replies, ‘Sure. Why just last week we had a full plant inspection. The inspectors even made us clean out our personal lockers. I’ll tell you, that took some doing. I had so much stuff in my locker that I couldn’t even get it open . . . ‘
Exercise 2 Effective Interviewing - Difficult Interview Situations
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 91 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Scenario E
As part of a four-person audit team assigned to audit D&R Chemical’s facility, you are in the process of interviewing Janet Kowalski, the facility health and safety co-ordinator. In an attempt to obtain information regarding site personnel health and safety training from Ms. Kowalski, you ask, ‘Could you explain how the facility ensures that new employees receive appropriate health and safety training?’ Ms Kowalski answers, ‘How do you think the facility should monitor such training.’ You respond, ‘I’m really trying to understand how this particular facility handles personnel health and safety training. Yesterday, I spoke with site training co-ordinator, John Golder, who suggested I talk with you. Could you please describe your training responsibilities to me?’ Ms Kowalski replies, ‘What did the training co-ordinator tell you that I do?’
Exercise 2 – Potential Answers Effective Interviewing - Difficult Interview Situations
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 92 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
The following are general techniques to use in any difficult interview situation: • Establish and maintain mental and physical control of
yourself. • Vary your questioning pattern to obtain maximum information
from the interviewee • Make sure you are adequately prepared for the interview
beforehand so you portray confidence and can maintain control of the interview
• Don’t be afraid to openly acknowledge the difficulty, take a
break, or ask if you can reschedule for a later time Some additional techniques to apply to specific interview situations include: • If the interviewee becomes hostile and aggressive, remain
calm and do not worsen the problem. Limit your questions to just the facts, and convey to the interviewee that the purpose of the interview is to uncover the truth
• If the interviewee can’t stop talking, do not be afraid to politely
interrupt in order to move on in the interview • If the interviewee goes off course, sum up what the
interviewee has said and either move onto the next question or return to the point where things went wrong in the interview
Exercise 3 Effective Interviewing - Conducting Interviews
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 93 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Objective
The purpose of this exercise is to practice questioning patterns that elicit the maximum amount of relevant information from the interviewee. We will also want to pay attention to planning the interview and to opening the discussion. Instructions
In this exercise, we will role play an interview between an auditor and a plant representative. If you are No. 1 you will be the Observer for this role play. If you are No. 2, you will be the Auditor. If you are No. 3, you will be the Plant Representative. There are different sets of instructions for your assigned role, which are on the following pages: Health and Safety Specialists No. 1 Observer Page 94 No. 2 Auditor Page 95 - 97 No. 3 Plant Representative Page 98 - 99 Environmental Specialists No. 1 Observer Page 100 No. 2 Auditor Page 101 - 102 No. 3 Plant Representative Page 103 - 104 We will break into groups (Observers, Auditors, and Plant Representatives in separate rooms). You will have 10 minutes to prepare for your role, 15 minutes to conduct the interview, and 5 minutes for feedback.
Exercise 3A Effective Interviewing - Conducting Interviews: ‘Lockout/tagout’
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 94 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Observer’s Instructions
Your role in this exercise is to observe the interview being conducted by the auditor, to note the techniques being used, to document the information in your working papers, and to provide feedback to the auditor and plant representative on your overall impressions of the interview at its conclusion. The auditor’s assignment is to understand and document how the facility manages its lockout/tagout procedures. The auditor will be interviewing the facility’s safety supervisor. To assist you with noting the techniques used during the interview, an index card with an outline of key interviewing techniques will be provided to you/ You may wish to note the number of leading, yes/no, and open-ended questions asked by the auditor by putting tick marks near these items on the index card each time a question is asked by the auditor. Helpful Hint
In order to minimise the distraction created by your role in this exercise, position yourself off to the side between the other two participants and make your note-taking as unobtrusive as possible. Keep track of the time and signal the auditor to wrap up when the interview period is approaching 15 minutes.
Exercise 3A Effective Interviewing - Conducting Interviews: ‘Lockout/tagout’
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 95 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Auditor’s Instructions
Your role in this exercise is to play the part of a safety and industrial hygiene auditor. During your interview with the facility’s safety supervisor, you wish to develop and document your understanding of how the facility manages its lockout/tagout procedures (refer to the protocol steps provided on the following pages). The facility’s lockout/tagout programme is designed to protect maintenance and production personnel and outside contractors from contact with energised equipment during maintenance and overhaul. In its operations, the facility utilises steam to clean process vessels, and electric and pneumatic motors to operate mixers, and has numerous product transfers lined located throughout the plant. Personnel involved in these manufacturing operations can often be exposed to high temperature liquids and electrical hazards due to we floors and working surfaces. You may want to consider the following in preparing for your interview: • Identification of equipment requiring lockout/tagout • Written lockout/tagout procedures • Training of ‘affected’ and ‘authorised’ employees • Issuance and use of locks and tags • Inspections of the lockout/tagout procedures performed • Group lockout/tagout • Contractors Take time now to plan for the interview. Think how you will open, conduct, and close the interview.
Exercise 3A Effective Interviewing - Conducting Interviews: ‘Lockout/tagout’
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 96 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Auditor(s)/ Comments
Working Paper Reference
Control of Hazardous Energy (Lockout/Tagout)
13. Examine documentation and interview key personnel to confirm that a written hazardous energy control programme has been developed.
14. Review the written hazardous energy control programme, interview affected personnel, and review records. Confirm that:
a. Site and local management have developed and implemented a written hazardous energy control programme for all operations where the unexpected energising, startup, or release of stored energy could occur and cause injury
b. Local management has provided the necessary material (e.g. locks, chains, tags) to their employees for implementing the hazardous energy control programme
c. The facility has implemented a training programme of lockout/tagout policies and procedures, including locking isolation valves
d. The facility has established a programme to perform and document periodic inspections (at least annually) of the energy control programme
e. There are procedures for briefing contractor personnel on the emergency control procedures
f. Only authorised employees implement energy control measures
Exercise 3A Effective Interviewing - Conducting Interviews: ‘Lockout/tagout’
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 97 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Auditor(s)/ Comments
Working Paper Reference
15. Examine lockout/tagout devices dedicated to controlling hazardous energy to confirm that the element is included
a. Standardised throughout the facility
b. Able to withstand environmental conditions
c. Substantial enough to prevent accidental removal
d. Traceable to the employee applying the device
16. Interview employees who work in areas where lockout/tagout devices are used to confirm that they are notified prior to the application and removal of a lockout/tagout device
17. Confirm that there are procedures in place to ensure the continuity of lockout at shift change and during group lockout/tagout activities
Exercise 3A Effective Interviewing - Conducting Interviews: ‘Lockout/tagout’
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 98 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Plant Representative’s Instructions
Your role in this exercise is to play the part of the facility safety supervisor, who is fairly co-operative, but is concerned about how the audit findings will be presented to management. Although you answer specific questions, you tend not to readily volunteer information. Your facility does have a written lockout/tagout procedures. In this exercise, you are embarrassed by the fact that the facility does not rigorously enforce its ‘one lock/one key’ policy. Combination locks are sometimes used instead of keyed locks, and several employees may know the combination. Also, because of inadequate labelling of sources, there have been a few recent instances in which a supervisor observed that the wrong energy source was locked/tagged out. Although no actual incidents occurred, this indicated a failure to verify isolation of the equipment. You will withhold this information unless asked a question that requires you, in all honesty, to reveal it. Do not lie, but make the auditor probe. Follow these guidelines when conducting the interview: • Assume one of the following difficult interviewee stances for
part of the review: − Take control of the interview − Become hostile and aggressive − Give inadequate answers and/or become unresponsive − Continue talking throughout the entire interview − Go off course from the subject matter
• Insert a ten-second pause into the conversation. • Give key information that should be recorded and one piece of
superficial information that should not be recorded.
Exercise 3A Effective Interviewing - Conducting Interviews: ‘Lockout/tagout’
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 99 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
If you are asked difficult questions for which not enough information has been provided to you, please feel free to make up information. Perhaps the next thing to do will be to imagine a facility with which you are familiar when creating your responses. Be imaginative and enjoy it!
Exercise 3B Effective Interviewing - Conducting Interviews: Air Pollution Control
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 100 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Observer’s Instructions
Your role in this exercise is to observe the interview being conducted by the auditor, to note the techniques being used, to document the information in your working papers, and to provide feedback to the auditor and plant representative on your overall impressions of the interview at its conclusion. The auditor’s assignment is to understand and document how the facility manages its air pollution control programmes. The auditor will be interviewing the facility’s environmental co-ordinator. To assist you with noting the techniques used during the interview, an index card with an outline of key interviewing techniques will be provided to you. You may wish to note the number of leading, yes/no, and open-ended questions asked by the auditor by putting tick marks near these items on the index card each time a question is asked by the auditor. Helpful Hint
In order to minimise the distraction created by your role in this exercise, position yourself off to the side between the other two participants and make your note-taking as unobtrusive as possible. Keep track of time and signal the auditor to wrap up when the interview period is approaching 15 minutes.
Exercise 3B Effective Interviewing - Conducting Interviews: Air Pollution Control
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 101 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Auditor’s Instructions
Your role in this exercise is to play the part of an environmental auditor. During your interview with the facility’s environmental co-ordinator, you wish to develop and document your understanding of how the facility manages its air pollution control programmes (refer to the protocol step provided on the following page). The facility has over 20 sources that require air permits. In addition, the facility has programmes for monitoring the ambient air, preparing emissions inventories, and inspecting control equipment. You may want to consider the following in preparing for your interview: • Emission source identification and inventory, registration, and
permitting • Emission control equipment • Monitoring programmes • Maintenance and inspection programmes • Odour control programme • Indoor air pollution issues • Training • Reporting and recordkeeping Take time now to plan for the interview. Think how you will open, conduct, and close the interview.
Exercise 3B Effective Interviewing - Conducting Interviews: Air Pollution Control
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 102 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Auditor(s)/ Comments
Working Paper Reference
Understanding Management Systems
1. Obtain and document your understanding as to how the facility manages its air emission control programmes. Considerations may include:
a. Emission source identification, registration, and permitting
b. Emission control equipment
c. Monitoring programmes
d. Maintenance and inspection programmes
e. Training
f. Reporting and recordkeeping etc.
Exercise 3B Effective Interviewing - Conducting Interviews: Air Pollution Control
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 103 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Plant Representative’s Instructions
Your role in this exercise is to play the part of the facility environmental co-ordinator. You are fairly skeptical of the audit being conducted and its usefulness to the corporation. You are very proud of your plant and its environmental record. You have been inspected by the regulatory agency several times in the past year and have received no citations from any of these inspections. You can feel free to share your skepticism of the audit and your perspective that the plant has been and is in great shape. Your facility does have 23 air permits, 26 air sources (stacks from furnesses, process vents and drains), and 6 emission control devices (scrubbers and dust collectors). You do monitor ambient air for fugitive emissions but are not required to report the results. Your maintenance department does inspect all control equipment every six months as specified in your permit applications. You are responsible for completing all emission inventories and submitting them to the regulator. In this exercise, you are embarrassed by the fact that the facility has just recently (last month) undergone expansion that has resulted in the installation of two new ovens for which the facility did not obtain the required installation permits and does not have operating permits. The facility has also added a flare without revising the permit. Also, some of your permits are past their expiration dates and timely re-applications were not submitted. You will withhold this information unless asked a question that requires you to tell the interviewer honestly about these oversights. Do not lie, just make the auditor probe to discover this fact.
Exercise 3B Effective Interviewing - Conducting Interviews: Air Pollution Control
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 104 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Follow these guidelines when conducting the interview: • Assume one of the following difficult interviewee stances for
part of the interview: − Take control of the interview − Become hostile − Give inadequate answers and/or become unresponsive − Continue talking throughout the entire interview − Go off course from the subject matter
• Insert a ten-second pause into the conversation • Give key information that should be recorded and one piece of
superficial information that should not be recorded. If you are asked difficult questions for which not enough information has been provided to you, please feel free to make up information. Perhaps the next thing to do will be to imagine a facility with which you are familiar when creating your responses. Be imaginative and enjoy it!
Preparing Working Papers
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 105 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers Definition of Working Papers
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 106 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Working papers consist of any item that documents information gathered by an auditor (i.e., rough notes, worksheets, company records and policies, etc.). Working papers serve as the basis for the audit findings. Working papers should contain: • A description of the environmental, health, and safety
management systems in place at the facility. • A description of the specific actions taken to address each
step of the protocol (tests conducted, sources(s) of information, evidence accumulated).
• A summary of the auditor’s findings and observations.
Preparing Working Papers
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 107 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Audit Planning
Step 5:Report Audit Findings
Step 1:Understand Management Systems
Step 2:Assess Strengths &
Weaknesses of Management Systems
Step 3:Gather Audit Evidence
Step 4:Evaluate Audit Results
Audit Follow-Up
WorkingPapers
AuditReport
Document scope in working papers
Record understanding in working papers
Record assessment of soundness of system design
Note explanation & disposition of all findings
Document audit findingsat exit meeting
Document verification testing plan and results
Preparing Working Papers - Purpose of Working Papers
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 108 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
The overall purpose of preparing working papers is to aid the auditor in providing reasonable assurance that an adequate audit, consistent with programme goals and objectives, was conducted. Working papers clearly document the information gathered by each auditor during the audit, and the information included in these documents should substantiate both compliance and noncompliance areas. Audit working papers, therefore, provide the principal evidential support for the audit report. Audit working papers: • Provide an organised method for ensuring that all audit steps
have been addressed in a manner consistent with the objectives and established procedures of the audit programme.
• Supplement the protocols by providing audit planning details
such as the time budgeted to individual audit tasks and the auditor’s evaluation of the management systems that may have influenced the conduct of the audit.
• Provide a record of tests conducted and evidence
accumulated. • Provide data that support the audit report and that may be
useful in subsequent action-planning and follow-up activities. • Provide information to assist in answering questions that may
arise during subsequent action planning and follow-up. • Provide a basis for quality assurance and aid in the planning,
performance, and review of future audits.
Preparing Working Papers - Retention of Working Papers
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 109 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Typically, companies will retain working papers according to policies such as the following: • Until the final report and/or corrective actions are completed. • Until the next audit of that facility. • For a particular retention time based on corporate policy (e.g.,
five years).
Preparing Working Papers - Working Paper Standards
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 110 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
The minimum standards for working papers will vary based on the company’s audit programme objectives and working paper retention policy. The principles presented below reflect the types of working paper standards that have been adopted by various companies.
Examples of Working Paper Standards
Methods/actions to complete each protocol step or rationale for not completing the step (e.g., not applicable) are documented.
Potential ambiguities or misleading comments have been clarified.
Facility compliance and noncompliance with applicable requirements are documented.
All exceptions (deficiencies), observations, and local attention items (items not for the formal report) are identified. The rationale for local attention items is documented.
If a “sampling” approach is used, the types and sizes of the “samples” are identified.
Key requirements to be reviewed are identified.
The auditor’s assessment of the management systems is documented.
The sources by which the auditor gains information are identified.
The auditor’s daily goals and the results achieved are documented.
Protocols and working papers are cross-referenced.
The nature and scope of the audit are identified.
Any changes to the scope are documented.
All exhibits are listed on an exhibit list and referenced in the working papers, as appropriate.
All pages are numbered, dated, and initialed. Any changes made by the auditor to the working papers are initialed.
Preparing Working Papers - Confidentiality of Working Papers
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 111 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Working papers should be carefully safeguarded during and after all auditing activities. Upon completion of the working papers review at the end of the audit, all audit working papers should be stored in a central file under the control of the audit programme manager, until they are destroyed in accordance with an established records retention policy. (A specific retention period should be established at the corporate level for the audit working papers that is consistent with audit objectives and supported by the legal department.) Since working papers represent documentation of the scope and conduct of the audit, they may be needed to support findings in the audit report should questions arise at a later date. Excerpts may be copied upon approval of the audit programme manager; however, working papers should not be copied without prior approval from audit management. Because there is no absolute guarantee that an audit report will not be discovered by an outside party at some future time, careful preparation of working papers offers the best protection against potential future liabilities.
Preparing Working Papers - Techniques for Recording Information
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 112 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
A number of basic techniques to be used in developing thorough and well-organised working papers should be kept in mind. • Write while conducting the audit. Notations that serve as
reminders of key points are helpful in gaining a complete understanding of facility systems or activities. An auditor should avoid relying on his/her memory and putting off documenting items until he/she “has more time.”
• Start each new topic on a new page. Many times an auditor
will obtain additional information even after he/she feels the particular topic has been completely documented. Any additional information or notes to clarify particular items can easily be inserted in existing text, and dated if the information was obtained on a different day from when the page was prepared, if each topic is entered on a new page. Also, cross-references can be added to indicate where additional information on the topic can be found.
• Clearly label each working paper page. Initial, date, and
sequentially number each page. Labelling each page with the protocol step makes it easier for both the auditor and the audit team leader to review the work performed and helps locate specific topics in the working papers. A single notation identifying the relevant protocol step is generally sufficient (e.g., Protocol Step 8b: Off-Site Shipment of Wastes).
• Keep entries factual. Each statement should be based on
sound evidence, with unconfirmed data or information qualified, and speculation and generalities avoided. For example: − Do not say, “It appears that...”; rather, state the facts that
create the appearance. − Avoid extreme language (e.g., “terrible,” “dangerous,”
“incompetent”). − Distinguish clearly between information obtained by word-
of-mouth and information observed, verified, or concluded.
Preparing Working Papers - Techniques for Recording Information
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 113 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
− Document the source of all information.
• Keep entries legible. Although no one’s penmanship is perfect, working papers should be written in a legible hand. Avoid crowding, leave plenty of space, and write only on one side of the page. This practice will aid in the audit team leader’s review of the working papers and will help the audit programme manager confirm that audit programme goals and objectives have been achieved. At a minimum, each auditor should be able to read his/her own notes. In the event of a mixed language team, the team leader will decide what the language to use in the working papers will be.
• Write clearly in an understandable style. An auditor should
strive to write clearly, so that a person not involved in the audit can understand the steps taken and can reach the same conclusions. Avoid uncommon abbreviations.
• Include photocopies of selected documents. An auditor
should sequentially number and reference selected facility documents as “exhibits” in his/her working papers (e.g., Exhibit A1). If any notations on any exhibit are made, they should be documented in the auditor’s working papers. For example: “Exhibit G1 is a copy of the facility’s air pollution control permit. Page 3 of the permit contains notes that I used to confirm all emission sources.”
• Maintain an exhibit list. To keep track of the exhibits, an
auditor should develop and update an exhibit list as each new exhibit is identified and numbered.
Preparing Working Papers - Techniques for Recording Information
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 114 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• Highlight “to do” items and findings. Many auditors find it convenient to keep a running list of “to do” items (items that call for further investigation or additional information) during the audit. This can be done by either writing them on a separate page or in some way identifying them where they are noted. Any “to do” items should be indicated as complete and then cross-referenced when they have been finalised.
• Develop and use standard “tick marks.” To increase
efficiency in developing useful working papers, many auditors develop standard “tick marks,” or legend codes (i.e., a personal type of shorthand) for many of the more common or cumbersome working paper notations.
Tick Mark Examples
Item needing further auditor attention
Item where subsequent attention has been given and noted on page 15 in the working papers
Potential report or exit interview exception/observation
Exception/observation after reporting to team leader and/or facility
Exception confirmed by auditor (as item 3) on exit meeting discussion sheet
Potential concern later determined by auditor not to be an exception (explanation on page 17 of the working papers)
PCC-15
3
PCC-17
Preparing Working Papers – Format
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 115 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
The handwritten information and photocopied facility documentation that comprise an auditor’s working papers are normally prepared on site during the audit. The following pages illustrate examples of working paper entries and documentation, including examples of flowcharting, organisational charts, interview notes, descriptions of actions taken and tests performed, interim summary, pending and completed “to do” items, and an exhibit list.
Preparing Working Papers – Format
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 116 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
1. Applicable protocol step
2. Date
3. Auditor’s initials
4. Page number
5. Source of information
6. Tick marks
7. Referenced exhibit
8. “To Do” items
9. Initialed cross-outs
10. Working paper
references
11. Confidentiality
Preparing Working Papers – Example of Flowcharting
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 117 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
MAC p1 of 675
Auditor’s initials and page #
Preparing Working Papers - Example of Organisational Charts
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 118 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers - Example of Interview Summary Notes
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 119 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
M DG 7 of 42
Preparing Working Papers - Example of Descriptions of Actions Taken and Tests Performed
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 120 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers - Example of an Interim Summary
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 121 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers - Example of “To Do” Items (pending and completed)
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 122 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers - Example of an Exhibit List
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 123 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
MDG - 85
Preparing Working Papers - Review of Working Papers
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 124 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Working papers should undergo a two-stage review process: (1) on-site review by each individual auditor of their own working papers, and (2) post-audit review by the team leader of each auditor’s working papers. 1. On-Site Review
Throughout the audit, each auditor should review his/her working papers frequently (i.e., at least daily and preferably several times during each audit day) to ensure that all audit tasks have been completed, open items have been resolved, and adequate evidence has been gathered to support the findings. The information included in the working papers should cover all elements of the audit protocol and leave no unanswered questions or open items. Working papers should be complete, free-standing records of the actions taken by the auditor that can be used to verify and document compliance and noncompliance situations. 2. Post-Audit Review
Immediately following each audit, and before the draft audit report is issued, all working papers should be reviewed by the audit team leader. This review provides for a quality assurance check on both the individual auditor and the audit topic(s) covered. The reviewer should document his/her review by signing and dating each set of working papers reviewed. The audit report and the exit meeting discussion sheets should be included in this review to ensure that the findings are reported appropriately and are substantiated by sufficient evidence documented in the working papers.
Preparing Working Papers - Review of Working Papers
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 125 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Sample Checklist to Review Working Papers
Format
Each working paper page is clearly labeled with the applicable protocol step.
Sources of information are clearly identified.
All exhibits are referenced in the working papers.
Each page is sequentially numbered, initialed, and dated.
Cross-outs are initialed; postscripts or afterthoughts are written in a manner that provides appropriate context.
Content
Each protocol step was addressed in accordance with the instructions provided.
Any departures from the protocol are described and explained.
A description of actions taken to complete each protocol step has been documented.
An understanding of how the facility is managing the items under review has been documented.
The conclusions reached as a result of testing have been documented.
All audit findings have been clearly identified.
All findings in the working papers have been included on the audit exit meeting discussion sheet.
Exercise 4 Preparing Working Papers
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 126 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
In this exercise, we will role play another interview between an auditor and a plant representative. The principal objective of this exercise is to practice documenting relevant information obtained from the interviewee. Of course, you will also want to practice the skills from the last exercise – planning the interview, opening the discussion, varying your question patterns, etc. You will have the same role number (1, 2, or 3) as in the previous interviewing exercise. However, the roles have been switched. If your number is ‘1’, you will be the ‘Observer’ for this role play. If your number is ‘2’, you will be the ‘Plant Representative’. If your number is ‘3’, you will be the ‘Auditor’. There are different sets of instruction for each role; please read only the instruction for your assigned role, as listed on the following pages Health and Safety Specialists No. 1 Observer Page 127 No. 2 Plant Representative Page 128 - 129 No. 3 Auditor Page 130 Environmental Specialists No. 1 Observer Page 131 No. 2 Plant Representative Page 132 – 133 No. 3 Auditor Page 134 We will break into groups (Observers, Auditors, and Plant Representatives in separate rooms). You will have 10 minutes to prepare for your role, 15 minutes to conduct the interview, and 5 minutes for feedback.
Exercise 4A Preparing Working Papers – Safety Inspection Programme
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 127 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Observer’s Instructions
Your role in this exercise is to document the interview being conducted and to critique the auditor’s working papers. The auditor’s assignment is to understand and document the implementation of the safety inspection programme as conducted within Department 901’s Superintendent. During the interview, document the responses of the superintendent in your working papers. At the completion of the interview, obtain the auditor’s working papers and: (1) compare them with your own working paper notes (e.g., Identification of topic and interviewee, content and completeness, legibility, etc.); (2) compare them with working paper principles and techniques; and (3) note strengths and weaknesses of these working papers. Helpful Hint:
To minimise the distraction created by your role in this exercise, position yourself to the side of the other two participants and make your note-taking as unobtrusive as possible.
Exercise 4A Preparing Working Papers – Safety Inspection Programme
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 128 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Plant Representative’s Instructions
Your role in this exercise is to play the part of the Department 901 Superintendent. You should co-operatively and comprehensively provide information to the auditor. Some information about the equipment safety inspection programme for Department 901 is listed below: • Department 901 has a safety committee (composed of
yourself, two other management representatives) that meets twice per month to discuss safety issues of concern and to tour the department to conduct safety inspections.
• The pressure of production over the last two years have
resulted in certain periods where the committee has failed both to meet and to conduct the inspections
• The result of each inspection are quickly reviewed by the
committee; the two management representatives convey the results to specific work areas for corrective actions.
• No formal corrective action plans are developed but you
believe things are getting addressed. During the interview, you should project a friendly, accommodating demeanor. You are free to develop additional information in responding to the auditor’s questions, and should provide an abundance of information (all of which need not be relevant to the aforementioned topic). The purpose of your role in this exercise is to force the auditor and the observer to think before they write: rather than acting as stenographers who record every word, the auditor and the observer should ideally record only that information which is important and pertinent to the aforementioned topic.
Exercise 4A Preparing Working Papers – Safety Inspection Programme
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 129 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Be imaginative in developing your description of the facility’s safety inspection programme and your responses to the auditor’s questions.
Exercise 4A Preparing Working Papers – Safety Inspection Programme
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 130 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Auditor’s Instructions
Your role in this exercise is to play the part of a safety and industrial hygiene auditor. During your interview with the Department 901 Superintendent, you wish to develop and document your understanding of how the department manages its safety inspections. You may want to consider the following elements in preparing for your interview: • Responsibility for conducting inspections • Frequency of inspections • Documentation/Reporting of results • Corrective actions/follow-up • Training
Exercise 4B Preparing Working Papers – Spill Response
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 131 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Observer’s Instructions
Your role in this exercise is to document the interview being conducted and to critique the auditor’s working papers. The auditor’s assignment is to understand and document how the facility manages its spill response equipment, distribution, and training programme. The auditor will be interviewing the facility’s emergency spill response team leader. During the interview, document the responses of the facility’s emergency spill response team leader in your working papers. At the completion of the interview, obtain the auditor’s working papers and: (1) compare them with your own working paper notes (e.g., identification of topic and interviewee, content and completeness, legibility, etc.); (2) compare them with working paper principles and techniques; and (3) note strengths and weaknesses of these working papers. Helpful Hint
To minimise the distraction created by your role in this exercise, position yourself to the side of the other two participants and make your note-taking as unobtrusive as possible.
Exercise 4B Preparing Working Papers – Spill Response
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 132 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Plant Representative’s Instructions
Your role in this exercise is to play the part of the facility’s emergency spill response team leader, who is co-operative and proud of the facility’s emergency spill response team and of its training and performance. Your understanding of the quantities of materials currently stored at the facility is: Tank Farm Acrylonitrile 2 x 15,000 gallons Toluene 1 x 10,000 gallons Fuel Oil 2 x 10,000 gallons Sulfuric Acid 1 x 10,000 gallons Hexane 2 x 10,000 gallons Isobutyl Methacrylate 1 x 5,000 gallons Other Chlorine 6 x 1-ton cylinders manifolded
at wastewater plant 12 x 1-ton full spares Ammonia 1 x 20-ton tank Emulsifiers 55-gallon drums Catalysts 55-gallon drums Other additives 55-gallon drums; 50-pound
bags
Exercise 4B Preparing Working Papers – Spill Response
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 133 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Some brief information about the facility is listed below: • Absorbent pads, pillows, along with absorbent granule
supplies are maintained in a central warehouse for distribution upon request.
• The spill team should receive monthly refresher training and
conduct a mock spill event once every six months • The spill team has its own meeting room and storage area for
self-contained breathing apparatus, chemical resistant suits, etc.
• The spill team’s mock spill conducted in August revealed that
some of the newer members of the team were not familiar with procedure and did not know what action to take in the event of a facility spill. As spill team training has been patchy since August, follow-up on this shortcoming has not yet taken place.
During the interview you should project a friendly, accommodating demeanor. You are free to develop additional information in responding to the auditor’s questions, and should provide an abundance of information (all of which need not be relevant to the aforementioned topic). The purpose of your role in this exercise is to force the auditor and observer to think before they write; rather than acting as stenographers who record every word, the auditor and the observer should ideally record only that information which is important and pertinent to the aforementioned topic. Be imaginative in developing your description of the facility’s spill response programme and your responses to the auditor’s questions.
Exercise 4B Preparing Working Papers – Spill Response
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 134 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Auditor’s Instructions
Your role in this exercise is to play the part of an environmental auditor. During your interview with the facility’s spill team captain, you wish to develop and document your understanding of the activities conducted at the facility relating to spill response. You may want to consider the following elements in preparing for your interview: • Hazardous materials on-site • Spill equipment available for use • Storage and maintenance of spill equipment • Spill team training • Mock drills and follow-up
Assessing Strengths and Weaknesses
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 135 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assessing Strengths and Weaknesses
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 136 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Step 1: Understand Management Systems
Conduct opening meetingConduct orientation tourReview audit strategyUnderstand details ofmanagement systems
Step 3: Gather Audit EvidenceEvaluate what needs to be doneDetermine depth & rigor of reviewSelect types of evidenceneeded & methods to gatherthemCompare practices againstrequirementsDocument results
Step 2: Assess strengths& Weaknesses
Consider potential impactsEvaluate management systemsSet priorities for verification
Step 4: Evaluate Audit ResultsEvaluate audit resultsWrite audit findings
Step 5: Report Audit FindingsConduct exit meeting
Prepare Draft Report
Obtain Review Comments fromCorporate HSELaw departmentFacility management
Issue Final Report toFacility managementOperationsCorporate HSELaw department
Develop Action PlanDevelop proposed action(s) toaddress each findingAssign responsibility for corrective actionDevelop timetable
Conduct Follow-upTrack status of correctiveactionsConfirm closure of findings
Pre-Audit Activities On-Site Activities Post-Audit Activities
Select & Schedule Facility Audit
Plan the Audit:Correspond with the facilityAssemble & distribute background informationAssign & communicateaudit responsibilitiesConduct pre-audit meeting
Select Team Members & Confirm their Availability
Assessing Strengths and Weaknesses - Purpose of Assessing Strengths and Weaknesses
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 137 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Once the auditors have developed an understanding of how the facility manages each of the environmental, health, and safety programmes included within the scope of the audit, the next step is to evaluate the soundness of these management systems in the context of the potential environmental, health, and safety impacts. The purpose of this assessment is to determine verification priorities for Step 3 activities. Spending the time to evaluate priorities will enable the auditor to maximise the effectiveness and efficiency of gathering the evidence needed to achieve the objectives of the audit.
Assessing Strengths and Weaknesses - Process for Assessing Strengths and Weaknesses
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 138 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Determine the range of potential impacts if a particular HSE issue is not
managed appropriately
Evaluate the management systems to determine if they are designed soundly. That is, consider if the systems, coupled with the controls, are appropriate given the potential impacts
Set priorities for verification so as to provide the optimum allocation of available team resources to ensure that issues representing high risk and weak management / control systems receive sufficient attention
Consider Potential Impacts
Evaluate Management Systems
Set Priorities for Verification
Assessing Strengths and Weaknesses - General Approach for Considering Potential Impacts
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 139 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Potential impacts refer to the range of potential consequences for the facility or company arising from an event or activity. In evaluating potential impacts, the auditor should ask: “If a health, safety and environmental activity at the facility is not managed appropriately, what consequences could ensue?” Potential impacts might include: • Catastrophic events (e.g., explosion) • Loss of life (e.g., confined space entry procedures) • Injury or illness (e.g., levels of airborne pollutants in the
workplace) • Environmental damage (e.g., spills of hazardous materials to
navigable waters or drinking water supplies) • Legal or financial liability (e.g., air permit violations) • Loss of operation and production • Adverse publicity (e.g., release of visible hazardous toxic
substance to air) • Recordkeeping or reporting exceptions • Occasional spills or releases to the environment • Employee/community exposures to hazardous/toxic
substances • Loss of property (e.g., fire) The auditor should evaluate the potential impacts for each protocol topic based upon a high to low spectrum.
Assessing Strengths and Weaknesses - Assessing Health, Safety and Environmental Management Systems
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 140 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Evaluating the soundness of a facility’s health, safety and environmental management of key programmes is inherently a subjective process. While regulations tend to stipulate explicit performance or technology requirements, explicit criteria/standards as to what constitutes an adequate management system are only beginning to emerge. In some instances, however, a corporation, division, or facility may have developed its own guidelines or policies as to how a particular activity or function is to be managed. In those situations where there are established management criteria—be they regulatory or internal—the auditor can look to the criteria for assistance in assessing the soundness of management systems. In all other instances, the guidance on the following pages provides a framework for assessing the strengths and weaknesses of the health, safety and environmental management systems.
Assessing Strengths and Weaknesses - General Principles for Assessing Management Systems
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 141 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Given the absence of explicit criteria for assessing management systems, the following guidance may be helpful.
Management System Descriptors
Assessing The facility has a process in place for: 1) identifying and evaluating conditions and/or materials on site that have regulatory implications, and 2) understanding the scope of applicable regulatory requirements. • Assessments are performed to identify potential risks (e.g.,
leaking underground storage tanks or pipes, carcinogens in the workplace).
• Environmental, health, and safety evaluations are conducted for modified products, processes, and operating ventures and signed off by health, safety and environmental staff.
• Programmes are in place to keep abreast of regulatory changes, to interpret the applicability of those changes to facility operations, and to develop procedures to address those changes.
Planning The facility has developed procedures and systems for managing compliance (e.g., plans, procedures, policies). • Appropriate procedures have been established to respond
to unintended events, such as process shutdowns, as well as to notify appropriate groups within the corporation and in the community.
• Usable programme and procedure guidance exists to direct facility activities to achieve health, safety and environmental goals consistently.
• A system has been developed for recordkeeping which provides documentation of health, safety and environmental activities and compliance with governmental requirements and company policy.
Assessing Strengths and Weaknesses - General Principles for Assessing Management Systems
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 142 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Management System Descriptors
Implementing The facility has structures and/or equipment in place to manage or implement compliance (e.g., pH meters, approved vendor lists, methods to address nonconformance situations, etc.). • Controls for equipment and storage areas are maintained in
an operable manner. • Systems are in place to address nonconformance situations
(e.g., permit exceedances, internal alarms). Reviewing The facility has developed a process for periodically reviewing
and monitoring compliance programmes (e.g., self-inspections, audits, supervisory review of data to cross-verify reports). • The functionality of the controls is periodically tested. • Periodic and comprehensive inspection programmes are in
place. • Deficiencies identified during inspections are corrected in a
timely manner. Organisation Clear roles and responsibilities have been established to
manage compliance with applicable regulations. • Roles and responsibilities are clearly understood with
respect to health, safety and environmental functions. • Health, safety and environmental staff have access to
appropriate line management to discuss issues and concerns.
Resources Qualified and sufficient health, safety and environmental staff and/or line personnel are involved in compliance management. • Staffing levels are appropriate to obtain environmental,
health, and safety compliance management goals. • Clear accountability exists for health, safety and
environmental performance. Training Programmes are in place to familiarise staff with the nature
and scope of compliance programmes. • Health, safety and environmental staff have appropriate
education, training, and experience to fulfill assigned duties. • Information and education programmes are sufficient to
enable employees to carry out health, safety and environmental functions.
Assessing Strengths and Weaknesses - General Principles for Assessing Management Systems
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 143 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Management System Descriptors
Documentation/Information Management
The facility has an accessible and orderly recordkeeping system for compliance-related activities (e.g., training, monitoring, governmental correspondence) and mechanisms in place to communicate relevant information between the various levels and functions of the organisation (i.e., memoranda, weekly meetings). • Records are accessible and managed in an orderly fashion. • Sufficient information is reported to management and
outside agencies, as appropriate.
Assessing Strengths and Weaknesses - General Principles for Assessing Management Systems
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 144 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Based on the information gathered during Step 1: Understand Management Systems, the auditor should assess the strengths and weaknesses of the facility’s approach to managing each protocol topic. An example of such an assessment (to verify compliance with a wastewater permit) is illustrated below.
Management System Activity
Strengths Weaknesses
Assessing Hired an outside consultant five years ago to review operations and identify wastewater issues.
Have hard copies (2004) of regulations on site. Corporate HSE provides monthly updates on changes.
Do not have formal procedure to review capital projects for wastewater impacts. Do not keep P&IDs current. Have not reviewed latest stormwater regulations.
Planning Have formal inspection and preventive maintenance (PM) programme for Wastewater Treatment Plant equipment.
Have site-specific wastewater sampling and analysis manual based on procedures in 40 CFR 136.
Have reduced total wastewater volume by 25% in past two years.
Lab chemist prepares draft Discharge Monitoring Report. HSE coordinator reviews report and raw data before sending to plant manager for typing and signature.
Do not have formal inspection and PM programme for laboratory equipment. Do not have written pollution prevention plan. Sampling and analysis manual does not address stormwater sampling. Plant manager’s secretary types final Discharge Monitoring Report for signature, keeps file copies, and does any exceedance reporting. HSE coordinator does not review final documents. Stormwater pollution prevention plan is based on draft general permit.
Assessing Strengths and Weaknesses - General Principles for Assessing Management Systems
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 145 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Management System Activity
Strengths Weaknesses
Implementing Have continuous monitors for wastewater flow, temperature, and pH at outfall. Wastewater Treatment Plant operator is responsible for preventative measures. Inspected and calibrated weekly. Periodically have outside laboratory analyze split samples as quality control check.
Do not do sampling and analysis of process-specific inputs to Wastewater Treatment Plant. Have not identified points for stormwater sampling. Process upsets cause “slug” flow to Wastewater Treatment Plant; no advance notice given. Laboratory wastewater goes to Publicly owned treatment works, not Wastewater Treatment Plant.
Reviewing Reviewed and updated site policies and procedures in June 2000. HSE coordinator spends 50% of time out in plant; sees/hears about changes as they happen.
Do not have formal self-inspection programme.
Organisation Roles and responsibilities are defined in job descriptions for Wastewater Treatment Plant operator, lab chemist, HSE coordinator; well understood.
Operations manager clearly considers wastewater compliance to be the responsibility of others (e.g., Wastewater Treatment Plant operator); end-of-pipe mentality.
Assessing Strengths and Weaknesses - General Principles for Assessing Management Systems
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 146 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Management System Activity
Strengths Weaknesses
Resources Have sufficient number of qualified staff for Wastewater Treatment Plant and lab. Person who does sampling and analysis is trained chemist with >10 years experience. HSE coordinator has degree in environmental engineering.
Training Chemist and HSE coordinator are encouraged to attend outside meetings/courses 2-3 times per year.
Have not trained internal staff on stormwater issues. Have not provided operations personnel awareness training on wastewater issues.
Documentation Excellent laboratory recordkeeping. Issue weekly newsletter to all plant personnel that includes information on compliance with Wastewater Treatment Plant effluent limitations.
Information Management
Have automated system for tracking training requirements. HSE coordinator has automated system to alert him when Discharge Monitoring Report is due.
Do not have formal records retention policy. Do not give notice to HSE coordinator when DMR is actually sent.
Assessing Strengths and Weaknesses - Balancing Management Systems/Controls Strengths vs. Potential Impacts
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 147 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Your overall judgment reflects, to a large extent, whether the facility has strong enough systems in place to minimise the potential impacts.
Potential Impacts
Systems and
Controls
Systems
and
Controls
Potential
Impacts
Potential ImpactsSystems and Controls
“Heavyweight” Systemto Manage
Low Potential Impacts
“Lightweight” Systemto Manage
High Potential Impacts
(pretty much OK) (may reflect an imprecise understanding of
impacts)
(may be vulnerableto surprises)
Balanced Approach
Often strengths or weaknesses in the management systems are linked to the presence or absence of several of the activities. The key is to identify which are most relevant to achieve a balanced approach. For example, a redundant (“belt plus suspenders”) system may be appropriate for high-impact situations, while a more straightforward (“strong leather belt”) approach may suffice where impacts are not as significant. Poor or deficient management systems can lead to “errors or omissions,” (e.g., error—failure to perform atmospheric testing prior to entry into a confined space; omission—failure to reschedule new employees for training who missed the initial training) and there are some health, safety and environmental situations in which an error and/or omission could have a substantial impact. Strong controls also reduce the risk of either an error or omission.
Assessing Strengths and Weaknesses - Setting Priorities for Verification
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 148 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Following the completion of the previous tasks, the auditor is now in a position to establish priorities for verification, that is, an approach to gathering data that will provide the auditor with sufficient data to draw conclusions regarding compliance with established standards and the effectiveness of the management systems. Setting priorities for verification should be dependent on the auditor’s evaluation of the strengths and weaknesses of the management systems, combined with an assessment of the potential impacts associated with a particular topic. This approach is based on the premise that strong management systems can (and should) be employed to mitigate otherwise high potential impacts. As a conceptual model, the auditor can “grid” each of the protocol topics covered to assist in ranking the priorities for verification.
Assessing Strengths and Weaknesses - Setting Priorities for Verification
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 149 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
In the empty matrix presented below, indicate how you would prioritise your time for the balance of the audit. Put a “1” in the box corresponding to your highest priority, a “2” in the box corresponding to your second highest priority, etc. That is, graph the strength of the management system on the y axis against the potential impacts on the x axis.
Management Systems
Potential Impacts
LowLow High
WeakWeak
Strong
The process of “gridding” assists the auditor in setting the verification priorities. That is, the gridding helps auditors select an approach to data gathering that emphasises an in-depth review of areas where potential impacts are high and management systems are weak.
Assessing Strengths and Weaknesses - Setting Priorities for Verification
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 150 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
For example: • If there are significant potential impacts associated with
noncompliance with a particular topic, and the management systems associated with that topic are judged to be weak, the auditor should ensure that sufficient time is allocated to reviewing that topic.
• Conversely, if there are low potential impacts associated with
noncompliance with a particular topic and the management systems associated with that topic are judged to be strong, the auditor need only spend a relatively small amount of time reviewing that topic.
By developing priorities for verification based on an assessment of the strengths and weaknesses of management systems, and potential impacts associated with noncompliance, the auditor helps ensure that the highest priority issues are covered in significant depth during the audit.
Verif
icat
ion
Prio
ritie
s
Level of Effort
High/Weak
High/Strong
Low/Weak
Low/Strong
Assessing Strengths and Weaknesses - Setting Priorities for Verification
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 151 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Auditors should recognise that it is generally not useful, in terms of setting priorities for verification, to conclude that all protocol topics are of equally high (or low) priority. Furthermore, it is generally not true that potential impacts are equally high (or low) and management systems are equally strong (or weak) for all topics in a given protocol. It is important for the auditor to use the Step 2 process to set priorities for Step 3. The assessment of priorities can be discussed as a team to provide a broader perspective of potential impacts and evaluation of the management systems. Following the development of verification priorities, the team should once again confirm that resources have been appropriately allocated to complete the audit.
Exercise 5A Assessing Strengths and Weaknesses
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 152 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Objective
The purpose of this exercise is to: • Assess the potential strengths and weaknesses of an HSE
programme and the potential impacts if that particular issue is not managed appropriately.
• Prioritise the order in which the protocol elements should be
completed based on your assessment of the management systems and the potential impacts.
Instructions
You have been assigned the Hazardous Waste Management Audit Protocol. During Monday and Tuesday of the week that you are auditing the Woodmount Company, you have interviewed several key people who are involved in hazardous waste management. Attached is a summary of the information that you have compiled as you begin to understand the management systems surrounding hazardous waste management. Based on your understanding of the management systems and on your assessment of the risks involved, rank the following elements of your protocol:
Protocol Element Potential Impacts
(low/high)
Management Systems
(weak/strong) Priority
Hazardous Waste Manifests
Waste Accumulation and Storage
Waste Classification Training
Exercise 5A Assessing Strengths and Weaknesses
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 153 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Based on interviews with selected key staff, you have learned the following: • The work force is quite stable and most of the employees on
site have been there for at least five years. • The facility has a hazardous waste generator number which
was obtained about ten years ago. The number is CAG 110 070 001.
• The facility generates hazardous wastes, including used
solvents and oils generated in the process of maintaining and lubricating machinery, support equipment, and machine shop activities.
• Annually, production supervisors are required to review the
waste profiles and initial the file copy to confirm that there have been no process changes that would alter the waste composition. Every five years, the facility hires a recognised consultant to review its waste characterisation programme.
• Hazardous wastes are stored in a locked shed adjacent to the
main building. − The only light in the shed comes from the indirect light
through the window openings near the ceiling. Employees are instructed not to enter the shed after dark. Exterior spot lights are available in the event of a nighttime emergency.
− Floor drains lead to a small sump, which can be pumped out manually if necessary.
− You were told that the facility has an ample supply of sorbent materials, but have not seen it yet.
Exercise 5A Assessing Strengths and Weaknesses
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 154 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• The treatment, storage, and disposal facility (TSDF), which is also the transporter, prepares the manifests for the facility when it picks up a waste load. − The TSDF comes whenever the facility calls. − The TSDF leaves the generator copy of the manifest with
the facility. − Six different Woodmount employees have signed
manifests in the last ten months.
• The receptionist keeps the facility’s manifests. − During the previous year, 25 shipments of hazardous
waste were sent off site to the TSDF. − Whenever a signed TSDF copy of a manifest comes in the
mail, the receptionist inserts it in the file with the original generator copy. (The receptionist noted that this usually occurs within a month of the shipment.)
− The current year’s manifests are kept in the receptionist’s files; older records are boxed and sent to a storage area in the manufacturing building.
− The sample manifest you pulled from the file was done on a 1984 form.
• The maintenance supervisor prepares the annual reports by recording the volume of waste per manifest and submits them to the regulatory agency every year by March 1.
Exercise 5A Assessing Strengths and Weaknesses
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 155 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• Facility hazardous waste training programmes are conducted once per year. − They last approximately eight hours. − The topics covered include HAZCOM, spill cleanup, and
an overview of hazardous waste management rules, including waste characterisation.
− Training is provided by an outside consultant, who submits a contract every year describing the approach and scope of the services rendered. The contract includes a provision for “back-up” training of staff who are absent from the on-site session.
− An agenda is prepared to reflect the topics to be covered during the day-long session.
− Production, maintenance, and selected clerical staff and supervisors attend the yearly sessions. Attendees are required to sign in for both the morning and afternoon sessions. Copies of the attendance records are kept by the maintenance supervisor.
Exercise 5B Assessing Strengths and Weaknesses
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 156 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Objective
The purpose of this exercise is to: • Assess the potential strengths and weaknesses of an HSE
programme and the potential impacts if that particular issue is not managed appropriately.
• Prioritise the order in which the protocol elements should be completed based on your assessment of the management systems and the potential impacts.
Introduction
You have been assigned the Process Safety Management (PSM) Audit Protocol. During Monday and Tuesday of the week that you are auditing the Woodmount Company, you have interviewed several key people who are involved in process safety management. Woodmount is subject to the OSHA PSM Standard because it stores and uses more than 10,000 pounds of flammable materials, chlorine, and ammonia on site. The facility has three process areas that are covered by the standard. Two of the process areas use flammable materials only. The third process area uses chlorine and ammonia. Attached is a summary of the information that you have compiled as you begin to understand the management systems surrounding process safety management. Based on your understanding of the management systems and on your assessment of the risks involved, rank the following elements of your protocol:
Protocol Element Potential Impacts
(low/high) Management Systems
(weak/strong) Priority
Operating Procedures Mechanical Integrity Training Process Hazard Assessments/HAZOPs
Exercise 5B Assessing Strengths and Weaknesses
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 157 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Based on interviews with selected key staff, you have learned the following: • The work force is quite stable and most of the employees on
site have been there for at least five years. • The facility has an objective of keeping all operating
procedures up to date. Every three years, the operating procedures are reviewed and updated as necessary.
• The facility has a standard for how to develop and write
operating procedures. • The supervisor for each department is responsible for
maintaining operating procedures. • The operations manager approves all operating procedures. • The availability of operating procedures is verified during pre-
startup reviews. • Operating procedures are updated by production engineers
and are reviewed by a committee consisting of the safety manager and senior production operators.
• Changes to operating procedures are summarised in memos
which are kept in a separate section of the operating procedures manual.
• The facility does not maintain documentation of maintenance
performed on individual pressure relief devices. • The facility has not completed an inventory of all pressure
relief devices. • Maintenance personnel have not yet received formal training
in plant safety.
Exercise 5B Assessing Strengths and Weaknesses
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 158 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• Associated piping to and from raw material and product storage tanks is not included in the facility’s preventive maintenance (PM) programme.
• The facility’s computerised PM system has not been set up to
archive the completion of the PMs, nor does the facility maintain a hard copy of this information.
• The facility has not yet implemented a periodic visual
inspection and/or nondestructive testing programme for its critical equipment (e.g., storage tanks, piping, and process vessels).
• The facility maintains a matrix of all job titles and their
applicable training requirements. • The facility has developed training “blocks” in order to
facilitate consistency. A refresher training programme has not yet been established to provide training every three years to operators. Most operators are due to receive refresher training within the next four to six months.
• The facility’s training programme concludes with a testing of
the employee’s understanding of the material presented using a written exam and a practical.
• The facility has completed process hazard analyses (PHAs)
for all three covered processes using the hazard and operability (HAZOP) methodology.
Exercise 5B Assessing Strengths and Weaknesses
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 159 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• The HAZOP studies included representatives from the following facility organisations: − Operations − Maintenance (instrumentation and process) − Engineering − HSE department, if deemed necessary by Engineering
• HAZOPs conducted included formal feedback from operators. • Results of the HAZOPs are not consistently communicated to
personnel.
Exercise 5A – Potential Answers Assessing Strengths and Weaknesses
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 160 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Protocol Element Potential Impacts
(low/high)
Management Systems
(weak/strong) Priority
Hazardous Waste Manifests Low Weak 3
Waste Accumulation and Storage High Weak 1
Waste Classification High Strong 2
Training Low Strong 4
Exercise 5B – Potential Answers Assessing Strengths and Weaknesses
Protocol Element Potential Impacts
(low/high)
Management Systems
(weak/strong) Priority
Operating Procedures High Strong 4
Mechanical Integrity High Weak 1
Training High Strong 3
Process Hazard Assessments / HAZOPs High Weak 2
Gathering Audit Evidence
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 161 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Gathering Audit Evidence
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 162 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Step 1: Understand Management Systems
Conduct opening meetingConduct orientation tourReview audit strategyUnderstand details ofmanagement systems
Step 3: Gather Audit EvidenceEvaluate what needs to be doneDetermine depth & rigor of reviewSelect types of evidenceneeded & methods to gatherthemCompare practices againstrequirementsDocument results
Step 2: Assess strengths& Weaknesses
Consider potential impactsEvaluate management systemsSet priorities for verification
Step 4: Evaluate Audit ResultsEvaluate audit resultsWrite audit findings
Step 5: Report Audit FindingsConduct exit meeting
Prepare Draft Report
Obtain Review Comments fromCorporate HSELaw departmentFacility management
Issue Final Report toFacility managementOperationsCorporate HSELaw department
Develop Action PlanDevelop proposed action(s) toaddress each findingAssign responsibility for corrective actionDevelop timetable
Conduct Follow-upTrack status of correctiveactionsConfirm closure of findings
Pre-Audit Activities On-Site Activities Post-Audit Activities
Select & Schedule Facility Audit
Plan the Audit:Correspond with the facilityAssemble & distribute background informationAssign & communicateaudit responsibilitiesConduct pre-audit meeting
Select Team Members & Confirm their Availability
Gathering Audit Evidence - Purpose of Gathering Audit Evidence
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 163 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
The purpose of Step 3 is to gather data to verify that the facility is in compliance with applicable regulations and that HSE management systems are functioning as intended. Auditors gather data during Step 3 by developing and implementing a verification strategy. A verification strategy, in the simplest sense, is an expansion of the Step 2 assessment in that it details: • The specific focus or goal of the data gathering; • The types of data to be gathered and why; and • The range of tests to be performed to confirm the validity of
facility information. This section of the manual explains the process by which (and the factors to consider) to develop a verification strategy that complements the priorities established in Step 2.
Gathering Audit Evidence – Basic Tasks
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 164 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
There are several basic tasks the auditor should follow in formulating a sound verification strategy.
Task Description
1. Evaluate what needs to be done “What do I need to look at and how much time do I have to do it?”
2. Determine the depth and rigor of review
“How deep do I need to dig? How many kinds of evidence do I need to gather?”
3. Select the types of evidence needed and the methods to gather them
“What types of data will I gather and how can I collect them?”
4. Compare practices against requirements
“How does the facility rate against the applicable requirements?”
5. Document results “What information do I need to record to help me remember what I have learned and substantiate my conclusions?”
Each of these tasks is described on the following pages.
Gathering Audit Evidence – Basic Tasks
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 165 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
1. Evaluate What Needs to Be Done
To initiate the formulation of a sound verification strategy, the auditor should first develop a good appreciation for what he/she needs to review and how much time there is to do it all. As such, the auditor should: • Determine the specific objectives of the protocol steps for
review and then prioritise them. • Review the audit resources. Determine the Specific Objectives of the Protocol Steps for Review and Then Prioritise Them As the auditor develops a verification strategy, he/she may still need to determine the specific objectives of the protocol steps for review and then prioritise them. This task builds on what the auditor has identified as priorities based on the assessment made in Step 2. The task of understanding and appreciating the objectives of the protocol steps can be more subtle than merely identifying a topic. For instance, suppose that an auditor has identified spill response training as an item that must be verified during the audit. In this situation, there could be two different aspects to verifying the training requirement. On the one hand, the auditor may only want to verify that all applicable employees have had training during the past year. Accordingly, the auditor will likely review training records and attendance sheets. Alternatively, if the auditor wanted to verify that training was adequate or appropriate to the level of the trainees, then he/she may take a different approach toward verification, such as talking with employees who were supposed to be trained and/or evaluating the content of the training manuals and comparing them to regulatory requirements and/or industry practices.
Gathering Audit Evidence – Basic Tasks
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 166 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Prior to determining the types of data to gather, the auditor should quickly review the protocol steps to develop a full understanding of what needs to be verified. For example, a protocol step might direct the auditor to verify that a written training programme has been developed or to verify the effectiveness of the training programme. Understanding these types of differences will be important when selecting/identifying the types of data to review. Following this step, the auditor should prioritise the steps that will be completed. This task builds upon the priorities established during Step 2. That is, if an auditor determined in Step 2 that priority should be given to reviewing a facility’s respiratory protection programme, he/she now needs to decide what emphasis should be given to the various elements of the respiratory protection programme—verifying that a written programme has been established, training is conducted, inspections are performed, etc. The auditor should base his/her decisions on those elements that will provide the greatest insights regarding the functioning/implementation of the respiratory protection programme. Review the Audit Resources The auditor should review the audit resources to determine whether the audit strategy established in the pre-audit planning stage is still appropriate or if any modification or reallocation of audit resources is necessary to meet the objectives of the assignment. As a team, auditors should discuss whether: Audit team resources have been appropriately allocated to adequately cover all functional areas. Protocol topics do not overlap in such a manner that on-site activities are duplicated by two or more team members.
Gathering Audit Evidence – Basic Tasks
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 167 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
2. Determine the Depth and Rigor of Review
In formulating a reliable and defensible verification strategy, the auditor needs to determine how much data to collect and how deep to dig—in other words, the depth and rigor of review. To make this determination, the auditor should rely on the assessment of the potential impacts of the issues or activities being reviewed and the assessment of the soundness of the management systems made in Step 2. For example:
Management Systems
Potential Impacts
LowLow High
WeakWeak
Strong
3 1
4 2
Dig Deepest
Dig Deep
Dig Less Deep
Don’t Dig?
After the auditor has determined how deep to dig, he/she is now in a position to select what types of information to gather and how.
Gathering Audit Evidence – Basic Tasks
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 168 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
3. Select the Types of Evidence Needed and the Methods to Gather Them
The auditor should now determine what types of evidence are potentially available to be gathered and what methods would need to be utilised to gather them. In developing an approach for collecting evidence, the auditor should take into account what he/she has learned thus far in the audit, especially in Step 1, so as to get right to the business of verification. In this task, the following activities take place: • Consider what was learned in Step 1 • Select the type(s) of audit evidence • Decide whether and how to design a test • Develop sampling strategies Consider What Was Learned in Step 1: Understand Management Systems As auditors identify the types of evidence they will gather and the methods to be employed in gathering them, they should consider what was learned in Step 1: Understand Management Systems, because the nature of the environmental, health, and safety activities and management structures at the facility can influence the verification strategies. For example: • If the auditor is trying to confirm that a facility completely lacks
a hazard communication programme, he/she should not expect to find any physical or documentary evidence because obviously there is no written document describing the non-existent programme. Instead, the auditor will base this conclusion on the testimonial evidence.
• If the spill plan has not been updated in four years, then
perhaps the auditor will need to focus efforts on gathering physical data to determine whether and how current activities compare against the elements of the plan.
Gathering Audit Evidence – Basic Tasks
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 169 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• If a facility has different operations and/or different organisational structures at one location, then the auditor will want to ensure that any data-gathering activity is representative of each operation. For instance, if a facility has two different product lines, each of which is managed by a different division, then the auditor may need to go to two different sources to verify that employees received safety training. This factor should be taken into account when evaluating how much time is needed to complete a particular protocol step.
Select the Type(s) of Audit Evidence In general, there are four different types of evidence (also referred to as “audit data” or “audit information”) that can be gathered during an audit, and each can affect the quality and reliability of the audit results. Each type of evidence is associated with a particular method, and each has advantages and limitations. • Physical evidence • Documentary evidence • Testimonial evidence • Circumstantial evidence The auditor can strengthen the weight of the evidence by obtaining several different types and sources of evidence/information.
Gathering Audit Evidence – Basic Tasks
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 170 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Physical Evidence
Definition Something that is seen or can be touched.
Method Physical evidence is obtained through observation whereby the auditor collects information through physical examination.
Examples High level alarm, secondary containment, label on a container of chemicals, ethylene oxide monitor, etc.
Advantages Physical evidence is usually one of the most reliable and persuasive types or sources of data. In many situations, the physical presence of an object or operation can, by itself, satisfy a particular requirement of compliance and, therefore, observing that physical object or operation is essential during an audit to verify compliance.
Limitations The mere presence of an object or conduct of an operation does not ensure that it is appropriate for the situation, that it is designed and functioning properly, or that it will continue to function.
Documentary Evidence
Definition Something written down on paper or recorded electronically.
Method Documentary evidence is obtained through the collection and ensuing review of something written or recorded.
Examples Facility HSE policies, standard operating procedures, reports, inspection sheets, etc.
Advantages Documentary evidence allows the auditor to “see” the facility’s practices in a formal sense (i.e., the documentation) via a paper trail. As with physical data, the documentation itself is often a requirement for compliance. Furthermore, documentation of something makes a strong argument that an activity is indeed performed.
Limitations Documentary evidence by itself does not tell the auditor that an activity actually took place. Also, its reliability can be questioned since documents can be generated or altered while preparing for the audit.
Gathering Audit Evidence – Basic Tasks
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 171 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Testimonial Evidence
Definition Evidence received from interviews with facility representatives.
Method Testimonial evidence is obtained through inquiry and simply involves asking questions, both formally and informally.
Examples Anything told to the auditor by facility personnel.
Advantages Testimonial evidence gained through inquiry is a very efficient method of gathering data during an audit. It is especially helpful when one is acquiring basic knowledge on HSE systems at a facility, trying to clarify contradictory information, or obtaining explanations to unclear items.
Limitations Testimonial evidence obtained from inquiry must consider factors such as the competency of the questioned individual concerning the topic (e.g., the person’s training); the interest the person providing the response(s) has in the subject discussed; any biases that the individual questioned may have; and the logic and reasonableness of the question (i.e., was the question understood and appreciated).
Circumstantial Evidence
Definition Indirect evidence which conveys an overall impression.
Method Circumstantial evidence is obtained through an auditor’s developing a general impression or intuitive feeling about something at the facility.
Examples The order and neatness of records and files, the attitudes of facility personnel, the apparent relevance of facility staff’s background and experience to their HSE responsibilities.
Advantages Circumstantial evidence can be useful in directing where potential deficiencies may lie within a facility’s HSE management systems.
Limitations Circumstantial evidence is the most unreliable type of evidence and, thus, should never be used to verify compliance.
Gathering Audit Evidence – Basic Tasks
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 172 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Decide Whether and How to Design a Test Testing in an audit situation refers to a wide variety of activities1 that can be employed to verify that the facility has implemented its HSE systems, programmes, and procedures and that the implementation has been effective in achieving the intended results. Testing leads to increased confidence that what the auditor was told in interviews or saw on tours is, in fact, effective. The focus of testing is frequently on compliance with specific applicable regulatory and internal requirements. Testing is most commonly done on documentary evidence, such as: • Training records • Monitoring data and reports • Material safety data sheets (MSDSs) • Emission source inventories • Inspection and maintenance logs • Incident investigation reports In the case of documentary evidence, the testing can be done through vouching, recomputation, retracing data, and/or confirmation (see the table on the next page). Depending upon the nature of the specific audit step, any one of these four may be appropriate.
1 Testing in this context does not mean effluent or emissions sampling or chemical analysis.
Gathering Audit Evidence – Basic Tasks
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 173 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Types of Testing
Vouching This test would uncover errors in reported data and involves following the paper trail back to the raw data. For example, hazardous waste shipments recorded in the annual report might be tracked back to shipping department records and the hazardous waste manifests. This process verifies that all reported data are supported.
Recomputation This test checks for the accuracy of arithmetic calculations. This would include, for example, recalculating the results of employee monitoring to determine time-weighted averages.
Retracing Data This test would uncover omissions in reported data and involves reviewing the original data records to ensure that all results are appropriately reported. For example, pH strip charts might be reviewed to identify all excursions from permit conditions. The auditor would then verify that no excursions were omitted from the monthly Discharge Monitoring Reports.
Confirmation This test seeks written confirmation of something from independent third parties. This test may be used where an auditor cannot physically observe a condition, such as the operation of an automatic sprinkler system.
An important part of the concept of “testing” is that auditors deliberately, independently, and systematically select the specific pieces of evidence they will look at. Examples of tests are provided below. • A test of the MSDS system might involve choosing a sample
of chemicals from the facility’s master inventory, from purchasing or receiving records, or from chemicals observed during tours, and verifying that current and accurate MSDSs are on site for each chemical in the sample. (In contrast, simply noting whether there are books of MSDSs present in various locations during a tour would not constitute a “test”, but an observation).
Gathering Audit Evidence – Basic Tasks
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 174 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• An auditor could start with personnel department or payroll records and develop a sample of employees who should have received particular types of training, then review training records to see whether all of these people had been trained.
Although testing is most commonly applied to documentary evidence, it can apply to testimonial or physical (but not circumstantial) evidence as well. For example, the auditor might test the effectiveness of spill response training by gathering testimonial evidence—asking a selected sample of employees to describe proper spill response procedures. Similarly, a sample of emergency eyewash stations might be selected and inspected by the auditor (gathering physical evidence) as a test to determine if they are in good working order and unobstructed. It is important to note that an auditor does not design a formal, rigorous “test” for every protocol step or topic. Conducting testing is likely to be appropriate for the protocol steps/topics associated with “Dig Deepest” and “Dig Deep” ratings (see p.167) resulting from the Step 2 assessment of strengths and weaknesses. For lower priority protocol steps, the auditor may decide to rely on inquiry and observation, rather than testing, in formulating a conclusion. Develop Sampling Strategies Even after developing a sound verification strategy, auditors can still find themselves with more data to review than time allows. When this happens, the auditor, as part of his/her verification strategy, needs to also develop a sampling strategy. Sampling is the tool the auditor utilises to look at a portion of a whole population of items. Like verification, there are strategies for effective sampling that serve to minimise bias and ensure that what the auditor looks at is representative of actual conditions at the audited facility. Indeed, the sampling strategy itself can affect the validity of the data gathered and, consequently, the validity of the conclusions reached. Sampling strategies will be discussed in detail in the next chapter of the manual.
Gathering Audit Evidence – Basic Tasks
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 175 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
4. Compare Practices Against Requirements
Following the gathering of data, the auditor needs to compare facility practices against applicable performance requirements to determine whether the facility is in compliance with these requirements, and whether there are enough data to evaluate whether systems are being implemented as designed. If the auditor finds deficiencies in compliance and/or the implementation of management systems, he/she should remember that there still may be time to explore potential underlying causes of selected deficiencies (as discussed in the chapter on evaluating audit results). As the auditor compares practices against requirements, he/she begins to enter into Step 4: Evaluate Audit Results. 5. Document Results
Auditors should document their verification strategy as well as the data gathered. In particular, the auditor should make sure to document the following in his/her working papers:
Information What Auditor Documents
Which protocol steps were reviewed and which were not
The verification strategy (i.e., what topics were examined in depth and why).
Any changes to the verification strategy once initiated
If the auditor’s strategy called for review of certain records, but those records could not be reviewed because they were inaccessible, then this fact should be documented along with whatever alternative verification strategies were employed.
What data were gathered and the source
The testing plans that were utilised and the information actually collected/reviewed by the auditor.
Conclusions Summary(ies) of the results of the verification strategy.
Gathering Audit Evidence – Basic Tasks
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 176 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Summary: Developing a Verification Strategy
Task Question Asked How to Do it
Evaluate what needs to be done
“What do I need to look at and how much time do I have to do it?”
• Determine the specific objectives of the protocol steps for review and then prioritise them.
• Review the audit resources.
Determine the depth and rigor of review
“How deep do I need to dig? How many kinds of evidence do I need to gather?”
• Determine where you want to engage in a rigorous review of facility programmes and practices, based on your assessment of the following: − High priorities − Low priorities − Where few or no systems exist
Select the types of evidence needed and the methods to gather them
“What types of data will I gather and how can I collect them?”
• Take into account what you learned in Step 1.
• Determine the types of audit information needed.
• Determine the most appropriate methods for collecting the audit data.
• Determine the areas where you need to design a test.
• Determine the most effective sampling strategy.
Compare practices against requirements
“How does the facility rate against the applicable requirements?”
• Determine if you have enough data to evaluate whether systems are being implemented as designed.
• Explore selected deficiencies for underlying cause(s).
• Begin Step 4. Document results
“What information do I need to record to help me remember what I have learned and substantiate my conclusions?”
• Document what was looked at. • Document what was not looked at
and why. • Summarise conclusions in working
papers.
Exercise 6 Gathering Audit Evidence - Developing Verification Strategies
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 177 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Objective
The purpose of this exercise is to develop a verification strategy to most effectively conduct the Step 3 portion of the audit (Gather Audit Evidence) and to determine how best to manage the remaining time on site. Instructions
Based on the information provided in one of the following scenarios, outline your strategy for verifying that the facility is in compliance with applicable requirements highlighted in the protocol steps provided in this exercise and that management systems are being implemented and followed at the facility. You should assume that you are in fact at Step 3 in the audit process, that you have already spoken with the key HSE personnel, and that you have a general understanding of management systems.
Exercise 6A Gathering Audit Evidence - Developing Verification Strategies: Contractor’s Safety
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 178 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Background
It is now Tuesday afternoon of a one-week audit (the close-out meeting is scheduled for 10:00 a.m. Friday morning). After spending all day Monday and Tuesday morning learning how the facility manages safety, you feel that you have a good understanding of the systems in place, and you are now ready to go out and “verify.” What You Understood from Step 1 About the Facility’s System for Managing Contractor Safety
The facility, which is subject to the OSHA process safety management standard, has established a contractor/visitor orientation prior to commencing work. The HSE coordinator, human resources department, and maintenance manager have all been trained to provide the orientation. The orientation covers the facility’s safety rules, emergency/evacuation procedures, and hazard communication. In addition, the facility has various modules (i.e. confined space entry, lockout/tagout, and hazardous waste practices) which are added to the basic orientation depending on the type of work the contractor will be engaged in. Once the contractor has completed the orientation, his/her badge is given a colour-coded stamp which indicates that the person has received orientation and the topics covered (e.g., yellow—basic orientation; green—includes lockout/tagout; red—includes confined space entry; and blue—includes hazardous waste practices). Security personnel and all facility personnel are aware of the facility’s contractor orientation programme and are encouraged to check contractor badges to ensure that they have received orientation.
Exercise 6A Gathering Audit Evidence - Developing Verification Strategies: Contractor’s Safety
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 179 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Protocol Steps Relating to Contractor Safety
Contractor/Visitor Safety Programmes
1. Determine if the facility has a programme for contractor activities that have the potential for affecting process safety, including maintenance or repair, turnaround, major renovation, or specialty work on or adjacent to a process. If so, review the programme and confirm that the facility:
a. Obtains and evaluates information regarding the contractor’s safety
performance and programmes when selecting a contractor. b. Informs contractors, prior to the initiation of the contractors’ work at
the site, of the known potential fire, explosion, or toxic release hazards related to the contractor’s work and the process and obtains a signed confirmation that the contractor has received and understood the information.
c. Explains to contractors, prior to the initiation of the contractors’ work at
the site, the applicable provisions of the emergency action plan. d. Develops and implements safe work practices to control the entrance,
presence, and exit of contract employees in process areas. e. Issues appropriate permits to work (hot and cold) as necessary. f. Periodically evaluates the performance of contract employees in
fulfilling their obligations. g. Maintains a contract employee injury and illness log related to the
contractor’s work in process areas.
Exercise 6A Gathering Audit Evidence - Developing Verification Strategies: Contractor’s Safety
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 180 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Instructions
1. Evaluate What Needs to Be Done Review the protocol steps highlighted and determine the specific objectives of those steps. 2. Select the Types of Evidence Needed and the Methods to
Gather Them List the types of evidence that could be gathered to address the highlighted protocol steps. In addition, identify the items or systems you will want to test, and how you will accomplish verification testing. In developing your verification strategies, consider the following: • What evidence could you gather to address the protocol steps
highlighted? • Where or from whom could you gather the evidence? • How could you design a test of the facility’s contractor safety
programme with respect to the highlighted protocol steps? • When you have decided what you could do, decide which of
these data-gathering activities you would do to address the highlighted protocol steps.
3. Consider What Was Learned During Step 1 What factors learned in Step 1 drove your verification strategy?
Exercise 6A Gathering Audit Evidence - Developing Verification Strategies: Contractor’s Safety
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 181 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
To assist in developing your verification strategy, you should use the worksheet provided below which will take you through the basic steps as described in this section. Physical
Evidence Documentary Evidence
Testimonial Evidence
What evidence would you gather?
Where or from whom would you gather the evidence?
How would you design a test of the facility’s performance with respect to the highlighted protocol steps?
Exercise 6B Gathering Audit Evidence - Developing Verification Strategies: Hazardous Waste Manifests
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 182 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Background
It is now Tuesday afternoon of a one-week audit (the close-out meeting is scheduled for 10:00 a.m. Friday morning). After spending all day Monday and Tuesday morning learning how the facility manages its hazardous waste, you feel that you have a good understanding of the systems in place, and you are now ready to go out and “verify.” What You Understood from Step 1 About the Facility’s System for Managing Hazardous Waste Manifests
• The facility’s is a large quantity generator of special (hazardous) wastes.
• The HSE coordinator and maintenance manager have been designated to fill out the waste manifests.
• Once the accumulation storage log indicates that wastes must be shipped off site because they are within two weeks of reaching an accumulation time of 90 days, the waste transporter is called and a pick-up is scheduled.
• The log must be periodically reviewed since there are no automatic systems to notify facility personnel that a shipment must be scheduled.
• Once the waste transporter picks up the hazardous waste, the facility copy of the waste manifests is retained by the Human Resources records clerk. The clerk is responsible for filing the waste manifests with the receipt copy once the signed receipt copy from the disposal facility is received.
Exercise 6B Gathering Audit Evidence - Developing Verification Strategies: Hazardous Waste Manifests
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 183 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Protocol Steps Relating to Special Waste Manifests
Hazardous Waste Manifests
1. By reviewing a representative sample of Waste Manifests, verify that the facility has a program in place to accurately prepare these shipping documents and track waste loads from the point of generation to final disposition. In particular:
a. Note whether or not signed Waste Manifests are returned to the
producer from the disposal or treatment facility. For any that were not, document facility actions to locate the waste shipment.
b. Compare the waste streams shipped off site to your list developed previously. For those materials not covered by Waste Manifests during the review period, interview staff in the operating area(s) where the waste is usually produced and determine if the waste was generated during the review period. If so, resolve how this stream was disposed of without being covered by Waste Manifests.
c. Review the Waste Manifests and determine if each box or information entry has been filled out correctly.
Exercise 6B Gathering Audit Evidence - Developing Verification Strategies: Hazardous Waste Manifests
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 184 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Instructions
1. Evaluate What Needs to Be Done Based on the information provided above, review the protocol steps highlighted and determine the specific objectives of those steps. 2. Select the Types of Evidence Needed and the Methods to
Gather Them Identify the types of evidence that could be gathered to complete the highlighted protocol steps. In addition, identify the items or systems you will want to test, and how you will accomplish verification testing. In developing your verification strategies, consider the following: • What evidence could you gather to address the protocol steps
highlighted? • Where or from whom could you gather the evidence? • How would you design a test of the facility’s hazardous waste
consignment note programme with respect to the highlighted protocol steps?
• When you have decided what you could do, decide which of
these data-gathering activities you would do to address the highlighted protocol steps.
3. Consider What Was Learned During Step 1 What factors learned in Step 1 drove your verification strategy?
Exercise 6B Gathering Audit Evidence - Developing Verification Strategies: Hazardous Waste Manifests
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 185 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
To assist in developing your verification strategy, use the worksheet provided below which will take you through the basic steps as described in this section. Physical
Evidence Documentary Evidence
Testimonial Evidence
What evidence would you gather?
Where or from whom would you gather the evidence?
How would you design a test of the facility’s performance with respect to the highlighted protocol steps?
Exercise 6A– Potential Answers Gathering Audit Evidence - Developing Verification Strategies
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 186 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Scenario A: Contractor Safety
Physical Evidence Documentary Evidence
Testimonial Evidence
What evidence would you gather?
• Badge colours observed on site.
• Contractor activities on site against the facility’s safety rules.
• Contractor training manual.
• Contractor training records.
• Facility contractor safety rules.
• Contractor sign-in log.
• Contract language.
• Contractor understanding of the training materials and facility safety rules.
• Affected facility personnel’s understanding of the contractor safety programme and its implementation.
Where or from whom would you gather the evidence?
• Locations where contractor work is in progress.
• HSE coordinator. • Site security. • Contracting /
Purchasing personnel.
• HSE coordinator. • Site security. • Maintenance
manager. • Facility personnel
who manage on-site contractors.
• On-site contractors (if practical).
How would you design a test of the facility’s performance with respect to the highlighted protocol steps?
• Compare a sample of colour badges on contractors to the activity being conducted.
• Review contractor activities in comparison to the facility’s contractor safety rules.
• Compare a sample of contractors on the sign-in log to the training records.
• Obtain a sample of contractor names and the colour badge observed and compare the colour badge to the level of training received through a records review.
• Survey a sample of contractors to test their understanding of the training provided.
• Survey affected facility personnel’s understanding of the contractor safety programme (e.g., badges).
Exercise 6B– Potential Answers Gathering Audit Evidence - Developing Verification Strategies
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 187 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Scenario B: Hazardous Waste Manifests
Physical Evidence Documentary Evidence
Testimonial Evidence
What evidence would you gather?
• Accumulation start dates on hazardous waste drums.
• Hazardous waste streams generated by the facility.
• Hazardous waste accumulation storage logs.
• Filed Waste Manifests.
• Chemical purchasing or inventory lists.
• Annual hazardous waste reports.
• Purchase orders for waste disposal.
• HSE coordinator’s and maintenance manager’s understanding of Waste Manifests.
• Human resources records clerk’s understanding of consignment note record keeping.
• Affected facility personnel’s understanding of hazardous waste accumulation.
Where or from whom would you gather the evidence?
• Hazardous waste accumulation storage area.
• Process areas where potential hazardous waste streams are generated.
• Human resources records clerk.
• Facility personnel responsible for the accumulation area.
• Purchasing personnel.
• HSE coordinator. • Maintenance
manager. • Facility personnel
responsible for the accumulation area.
• Human resources records clerk.
Exercise 6B– Potential Answers Gathering Audit Evidence - Developing Verification Strategies
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 188 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Physical Evidence Documentary Evidence
Testimonial Evidence
How would you test the facility’s management systems to address the highlighted protocol steps?
• Compare the drums in the hazardous waste storage area to the information on the accumulation storage logs.
• Check the accumulation start dates on the hazardous waste drums to determine if drums are being stored over 90 days.
• Review facility processes to identify potential hazardous waste streams and compare to the facility’s identified waste streams.
• Check consignment note signed copy from the waste disposal company, where applicable.
• Compare Waste Manifests versus the annual hazardous waste report.
• Compare Waste Manifests versus waste profiles and/or process information to verify correctness of waste classification.
• Compare lists versus Waste Manifests.
• Review purchasing / shipping records and verify that Waste Manifests are on file for a sample of shipments.
• Ask the HSE coordinator / maintenance manager how they obtain the information regarding various wastes generated (i.e., waste characterisation, time generated etc.).
• Ask affected employees how they manage the hazardous waste accumulation area.
• Ask the records clerk how she/he manages the hazardous waste consignment note records (e.g., retention, signed copy from the waste disposal company etc.).
Sampling Strategies
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 189 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 7 Sampling Strategies 1
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 190 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Objective
The purpose of this exercise is to illustrate some key considerations when developing and implementing a sampling strategy. Instructions
Your assignment is to verify that secondary containment has been provided for all bulk storage tanks. The facility’s Spill Plan indicates that there are 30 bulk storage tanks and that all of them have secondary containment. The facility has a self-inspection checklist that lists the 30 tanks. Because time is limited, you have decided to choose a sample of six storage tanks from the facility’s list. The first tank in your sample has no secondary containment. On the way to the second sample location, you discover a tank that is not listed in the SPCC Plan and note that the drain valve in the containment area is locked “open.” The second tank in your sample has a visible crack in the containment. At this point, what do you do? 1. Stop right there. 2. Finish the sample of six. 3. Check the containment at all 30 tanks.
Sampling Strategies – Overview
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 191 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
A fundamental component of any audit situation involves the review of a facility’s HSE management systems and activities to obtain evidence needed to substantiate compliance or noncompliance. Because auditing is basically a check on the overall compliance status of a facility and is conducted over a relatively short period of time, auditors generally do not examine entire populations of records, documents, or employees. Instead, auditors sample populations in order to draw conclusions regarding compliance with performance standards. To help ensure the gathering of appropriate sampling information, the following process is frequently followed by auditors: 1. Determine objective of protocol step 2. Identify population for review 3. Select sampling method 4. Determine sample size 5. Document results Each of these steps is described on the following pages.
Sampling Strategies – Basic Sampling Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 192 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
1. Determine Objective of Protocol Step
• Specify what you are trying to confirm. • Consider the nature of the regulatory or internal standard to
accurately identify the boundaries of the population under review.
2. Identify Population for Review
• Estimate size of population through: − Review of selected documents. − Observations made during initial understanding of health,
safety and environmental management systems in place. − Interviews with facility personnel.
• Pay attention to major subsets or key segments of the
population that need to be included in the review. • Define population before starting to sample.
Sampling Strategies – Basic Sampling Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 193 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Sampling Methods
Random
Block
Interval
Stratification
Judgmental Probabilistic
3. Select Sampling Method
Judgmental Sampling Judgmental sampling is used to gather examples of deficiencies or problems to support an auditor’s assessment of a weak or improper health, safety and environmental management system. Sampling is directed toward segments of the population where problems are likely to exist. Judgmental sampling cannot be used to draw compliance conclusions about an entire population because it focuses on only a portion or subset of that population. Judgmental sampling can be used as a first step to provide the auditor with an indication of whether to use a probabilistic sampling technique such as random, block, interval, or stratification sampling.
Sampling Strategies – Basic Sampling Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 194 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Probabilistic Sampling Random. The objective is to select items by a statistically-based chance. If properly done, each item in the population should have an equal chance of being selected, and there should be no subjective determinations to bias the sample. (Note: For auditing purposes, “random” is not the same as “haphazard.” Haphazard means “characterised by lack of order or planning; aimless; independent of any reasoning process.” An example of haphazard sampling would be closing your eyes and grabbing one file out of a drawer. • Block. The objective is to analyze certain segments of
records or areas of the facility. For example, if files were arranged alphabetically, in numerical order, or chronologically, one or more blocks (e.g., all the E’s, records numbered 51 through 75, or January and June files) could be selected. While the block method is easy to use, it neglects entire segments of the population.
• Interval. The objective of interval sampling is to select
samples at specific intervals (e.g., every nth segment of the population is analyzed) with the first item selected at random. Increased confidence is achieved where several intervals with different random starts are used.
• Stratification. The objective of stratification sampling is to
arrange items by categories (e.g., high versus low effluent volumes; new versus experienced employees; regular versus weekend or off-shift transactions) based on the auditor’s judgment that the probability of finding an exception is different for different segments of the population and/or that there are categories within the population that represent higher inherent risks. Higher risk categories may, thus, receive greater review and testing. Once the population has been stratified, random, block, or interval sampling can be applied to select items within each segment.
Sampling Strategies – Basic Sampling Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 195 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
4. Determine Sample Size
• There are two ways to determine sample size: − Statistically − Auditor’s judgment
• In most HSE audit situations, it is both appropriate and
adequate to develop sample sizes based upon professional judgment.
• The auditor must be sure that the sample size is large enough
to be representative of the total population.
Sampling Strategies – Basic Sampling Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 196 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
5. Document Results
• Document rationale for selecting sample. • Document how sample was selected. • Include in working papers:
− Population under review − How and why population was selected − Type of sampling method employed − Reasons sampling method was used − Potential bias in sample − Sample size and reasons for selecting sample size
Example Guideline for Selection of Sample Size
Suggested Minimum Size of Sample Size of Population
A B C
2–10 11–25 26–50
51–100 101–250 251–500
501–1,000 Over 1,000
100% 100% 50% 25% 15% 10% 5% 3%
100% 40% 20% 15% 10% 5% 3% 2%
30% 20% 15% 10% 5% 3% 2% 1-2%
(These percentages do not imply any specific confidence level but are intended as guidelines only.)
A Suggested minimum sample size for a population(s) being reviewed which is
considered to be extremely important in terms of verifying compliance with applicable requirements and/or is of critical concern to the organisation in terms of potential or actual impacts associated with noncompliance.
B Suggested minimum sample size for a population(s) being reviewed that will provide additional information to substantiate compliance or noncompliance and/or is of considerable importance to the organisation in terms of potential or actual impacts associated with noncompliance.
C Suggested minimum sample size for a population(s) being reviewed that will provide ancillary information in terms of verifying overall compliance with a requirement.
Sampling Strategies – Basic Sampling Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 197 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Random Number Table
The random number table (on the following page) is intended for use when the auditor has chosen random sampling as the preferred sampling method. (See discussion above on “random sampling.”) To use the table, start at any location on the table; then, moving in a given direction (across, down, or up), identify the numbers in sequence. For example, in randomly selecting 25 of 90 records, designate values of 1 to 90 for each record, then moving along the table in the direction you have chosen, use the first two digits of each number to choose 25 records. Assuming you start in the upper left-hand corner of the random number table, and move down, your first number is 104 (record 10); the second number is 465 (record 46); the third number is 225 (record 22); etc. If you get a duplicate or a number outside the defined range, ignore it. Stop when you have 25 records to review.
Sampling Strategies – Basic Sampling Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 198 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Random Number Table 104
465
225
616
061
534
711
568
253
179
015
905
306
898
510
213
333
585
426
301
764
287
465
676
072
491
358
110
145
350
724
569
378
421
906
035
013
733
880
116
378
192
176
602
851
652
432
457
980
813
150
255
972
078
917
186
187
690
125
164
855
337
749
071
127
524
949
232
066
902
263
358
411
325
795
822
919
605
507
011
779
206
638
113
525
215
395
887
561
355
401
687
008
635
643
528
470
707
672
587
015
853
763
163
604
706
440
600
586
114
916
903
109
973
518
602
312
145
769
047
581
069
103
866
972
241
001
064
354
386
565
217
710
207
020
834
762
094
349
851
590
695
643
711
291
508
426
056
727
354
020
309
648
394
813
906
488
184
449
185
781
094
611
710
512
893
041
831
136
591
066
170
076
507
459
990
509
287
590
281
559
517
847
543
851
439
224
825
570
099
333
889
607
056
443
222
456
490
018
948
816
891
151
535
496
150
632
849
055
649
635
939
875
081
774
198
185
987
518
221
215
641
361
949
212
478
986
378
875
680
873
331
524
369
885
907
832
052
239
963
266
496
889
438
144
055
000
269
134
755
916
279
248
713
606
219
210
425
569
289
409
526
856
772
163
553
298
234
461
304
152
182
185
132
003
810
384
079
481
109
696
726
223
700
478
229
322
926
258
059
622
467
610
582
552
995
206
574
146
006
691
534
493
570
141
818
106
323
854
695
482
927
482
139
607
448
715
643
889
616
969
228
024
168
304
649
878
987
029
100
451
326
780
232
002
442
795
826
400
979
699
633
997
261
787
737
146
992
876
977
141
939
320
008
069
443
129
895
368
882
034
889
522
475
921
011
850
947
195
999
112
293
330
741
038
662
946
985
009
542
003
415
174
654
825
340
290
270
670
283
761
566
306
321
341
857
499
241
897
966
625
340
306
749
012
428
963
143
533
332
496
334
676
810
494
652
511
177
256
328
326
270
288
920
946
804
397
271
481
064
257
761
961
596
720
655
041
325
122
141
508
004
264
634
303
292
945
746
139
864
362
526
196
977
546
995
919
636
539
709
694
363
933
977
539
648
019
351
581
541
323
876
073
246
894
657
574
312
047
508
008
915
183
996
157
857
162
170
027
008
438
733
115
354
484
703
563
756
778
964
209
191
633
163
779
729
054
102
530
799
186
176
015
859
709
449
927
357
486
584
055
873
197
366
415
832
675
806
212
654
968
211
709
947
438
558
153
276
148
807
866
914
443
571
513
602
596
286
691
864
995
396
586
375
069
564
079
170
595
568
726
300
263
293
639
059
649
070
912
224
242
587
924
007
175
341
776
444
208
793
306
362
669
114
998
388
128
982
232
704
709
152
347
109
095
183
091
392
700
544
912
995
421
399
110
699
188
181
388
058
521
082
851
744
756
551
521
976
858
741
133
002
609
228
273
132
443
978
929
538
476
278
997
181
104
593
662
638
350
756
793
069
603
073
259
198
580
528
354
864
907
241
930
818
427
988
209
577
623
901
208
841
202
285
407
010
539
337
143
470
380
458
612
021
639
305
112
704
902
106
231
739
724
813
769
137
383
119
997
767
938
570
609
546
276
428
290
627
345
554
223
483
062
166
277
310
945
843
081
315
122
271
299
907
023
540
463
099
091
253
943
153
500
516
415
977
711
954
124
218
395
206
011
804
259
351
224
346
375
887
281
541
719
936
112
082
443
726
154
413
Sampling Strategies – Basic Sampling Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 199 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Sampling Strategies
Random Sampling
Block Sampling
Interval Sampling
Stratification Sampling
Select items in entire population by chance (e.g. using a randomNumber table)
Arrange items by certain segmentsor clusters and randomly selectsome clusters as your sample
Select samples at intervals (every nth item starting randomly)
Arrange items by important categories or subsets, then sample within the groups
Exercise 8 Sampling Strategies 2
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 200 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Objective
The purpose of this exercise is to: • Use each sampling method (random, block, interval, and
stratification) on a set of hazardous waste training records to determine if affected employees have received refresher training within the last year.
• Determine whether there are any advantages or
disadvantages in using one sampling method versus another.
Background
During your audit of the ACCO Chemical Company, you need to verify whether the site is complying with the corporate requirement that all employees who have responsibility for managing or handling hazardous waste have received annual refresher training and that the training is documented. You have learned that there are three categories of employees who are involved with hazardous waste management: • Within each production department, there are designated
operators, who have routine, day-to-day responsibility for managing a variety of wastes that are generated as a result of frequent cleaning of reactors used for batch specialty chemical production. Other production employees, who might on occasion generate hazardous waste, know that they are not to handle it themselves but are to contact one of the designated operators. If the designated operators are not properly trained, the probability of waste mismanagement is high.
Exercise 8 Sampling Strategies 2
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 201 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• Maintenance workers handle a much smaller number of relatively constant hazardous waste streams on an infrequent basis. The wastes most frequently handled are used oil and spent solvents from cleaning operations. Occasionally, maintenance workers encounter other wastes (when they service plant equipment (pumps, compressors), for example).
• Supervisors have virtually no hands-on responsibility for
waste management. However, because they are ultimately responsible for the actions of the workers they supervise, it is company policy that all supervisors (not just those in production or maintenance) must receive annual training to maintain awareness.
Based on the above job descriptions and the employee roster, you have developed the list shown in Table 1, which indicates that there are 80 employees total who require annual hazardous waste refresher training according to company policy. The plant environmental coordinator has provided you with his list of employees who attended the two most recent refresher training sessions, both held within the past 12 months (see Table 2). Instructions
Depending on the group to which you are assigned, you are to use one of the following methods to create a sample of approximately 25 percent of the 80-employee total (corresponding to the suggested minimum size of sample in column A using the table on page 9-10): • Random • Block • Interval • Stratification
Exercise 8 Sampling Strategies 2
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 202 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
For your sample, determine how many of the employees in each job category have received the required training according to the documented records (Tables 1 and 2 attached). After you have completed your sampling and analysis, we will compare the results of the various methods.
Exercise 8 Sampling Strategies 2
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 203 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Table 1: Employee List Developed by Auditor
No Job Category Employee Name
Sampling Method Chosen
Trained in Past Year
(Y/N)
R B I S
1 Maintenance Domingo, P.
2 Maintenance Bell, M.
3 Maintenance Cort, A.
4 Maintenance Greeno, L.
5 Maintenance Herald, M.
6 Maintenance Higgins, H.
7 Maintenance Getchell, M.
8 Maintenance Jones, T.
9 Maintenance Monteiro, L.
10 Maintenance Obbagy, J.
11 Maintenance Plunkett, J.
12 Maintenance Reid, R.
13 Maintenance Rotberg, F.
14 Maintenance Savoie, M.
15 Maintenance Sellers, G.
16 Maintenance Young, R.
17 Operator Allen, M.
18 Operator Arnold, E.
19 Operator Bach, J.S.
20 Operator Bateman, R.
21 Operator Butler, L.
22 Operator Catmur, J.
Exercise 8 Sampling Strategies 2
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 204 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
No Job Category Employee Name
Sampling Method Chosen
Trained in Past Year
(Y/N)
R B I S
23 Operator Cavalcanti, C.
24 Operator Coburn, T.
25 Operator Davanzo, L.
26 Operator DiBerto, M.
27 Operator Fitch, T.
28 Operator Fletcher, J.
29 Operator Harris, J.
30 Operator Hasselreis, D.
31 Operator Hryciuk, R.
32 Operator Jones, M.
33 Operator Lennon, J.
34 Operator Lewis, S.
35 Operator Loren, S.
36 Operator McGinnes, M.
37 Operator McLean, R.
38 Operator Moody, C.
39 Operator Murray, E.
40 Operator Neville, N.
41 Operator Nott, M.
23 Operator Nureyev, R.
43 Operator Patel, B.
44 Operator Picasso, P.
45 Operator Ryder, W.
46 Operator Schmidt, H.
Exercise 8 Sampling Strategies 2
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 205 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
No Job Category Employee Name
Sampling Method Chosen
Trained in Past Year
(Y/N)
R B I S
47 Operator Smith, F.
48 Operator Smith, J.
49 Operator Smith, N.
50 Operator Stone, O.
51 Operator Tallchief, M.
52 Operator Umscheid, M.
53 Operator Voeller, R.
54 Operator Wescott, W.
55 Operator Windsor, C.
56 Operator Yetskalo, V.
57 Supervisor Berstein, L.
58 Supervisor Boehm, P.
59 Supervisor Canton, C.
60 Supervisor Capogna, S.
61 Supervisor Dolittle, E.
62 Supervisor Ferguson, S.
63 Supervisor Fonteyn, N.
64 Supervisor Gedanke, M.
65 Supervisor Harris, R.
66 Supervisor Hedstrom, G.
67 Supervisor Hill, R.
68 Supervisor Hogwood, C.
69 Supervisor Lopez, D.
70 Supervisor Major, J.
Exercise 8 Sampling Strategies 2
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 206 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
No Job Category Employee Name
Sampling Method Chosen
Trained in Past Year
(Y/N)
R B I S
71 Supervisor Moore, T.
72 Supervisor Morrison, J.
73 Supervisor Nutty, P.
74 Supervisor Parigot, M.
75 Supervisor Presley, E.
76 Supervisor Robinson, J.
77 Supervisor Shrimpton, J.
78 Supervisor Starr, R.
79 Supervisor Stricoff, S.
80 Supervisor Thomas, M.
Exercise 8 Sampling Strategies 2
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 207 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Table 2: Training Records by Employee Name
Attendance at First Annual Refresher Training
Employee Name Job Category
Domingo, P. Maintenance
Arnold, E. Operator
Bach, J.S. Operator
Boehm, P. Supervisor
Butler, L. Operator
Capogna, S. Supervisor
Cavalcanti, C. Operator
Coburn, T. Operator
Cort, A. Maintenance
Ferguson, S. Supervisor
Harris, R. Supervisor
Hogwood, C. Supervisor
Hryciuk, R. Operator
Lennon, J. Operator
McGinnes, M. Operator
McLean, R. Operator
Monteiro, L. Maintenance
Morrison, J. Supervisor
Neville, M. Operator
Nott, M. Operator
Obbagy, J. Maintenance
Parigot, M. Supervisor
Schmidt, H. Operator
Exercise 8 Sampling Strategies 2
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 208 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Attendance at First Annual Refresher Training
Employee Name Job Category
Sellers, G. Maintenance
Shrimpton, J. Supervisor
Smith, J. Operator
Stone, O. Operator
Tallchief, M. Operator
Voeller, R. Operator
Young, R. Maintenance
Exercise 8 Sampling Strategies 2
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 209 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Attendance at Second Annual Refresher Training
Employee Name Job Category
Bell, M. Maintenance
Berstein, L. Supervisor
Canton, C. Supervisor
Catmur, J. Operator
Davanzo, L. Operator
DiBerto, M. Operator
Dolittle, E. Supervisor
Fletcher, J. Operator
Fonteyn, M. Supervisor
Hasselreis, D. Operator
Higgins, H. Maintenance
Hill, R. Supervisor
Lewis, S. Operator
Loren, S. Operator
Moody, C. Operator
Moore, T. Supervisor
Murray, E. Operator
Nutty, P. Supervisor
Nureyev, R. Operator
Plunkett, J. Maintenance
Robinson, J. Supervisor
Savoie, M. Maintenance
Smith, N. Operator
Starr, R. Supervisor
Stricoff, S. Supervisor
Exercise 8 Sampling Strategies 2
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 210 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Attendance at Second Annual Refresher Training
Employee Name Job Category
Thomas, M. Supervisor
Umscheid, M. Operator
Wescott, W. Operator
Yetskalo, V. Operator
Exercise 8 – Potential Answers Sampling Strategies
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 211 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Determine Objective of Protocol Step
Review facility training records to confirm that employees with hazardous waste management responsibilities have received refresher training within the past year. Determine Total Population and Important Subpopulations
There are 80 employees in three categories who require training: 16 maintenance workers, 40 operators and 24 supervisors. Select Sampling Strategy and Sample Size
1. Random The total population is 80 and you have decided that this is an important issue for the audit. The suggested minimum sample size is therefore 25%. You therefore want to pick 20 random numbers between 1 and 80. One way of generating your sample is to use a random number table. If you start at the lower right-hand corner of the table and read up, the first 20 random numbers (eliminating duplicates) between 1 and 80 are:
13 36 24 6 54 19 51 18 26 41 59 77 43 75 4 15 12 46 11 16
Exercise 8 – Potential Answers (continued) Sampling Strategies
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 212 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
The attached worksheet indicates with an “R” the sample of employees selected by this method. To summarise, the random method gives a reasonably representative sample, with all three job categories included: 7 of 16 maintenance workers 10 of 40 operators 3 of 24 supervisors 2. Block For this type of information, the most logical “block” method to use might be by the first letter of the last name, by generating random numbers between 1 and 26. Using the random number generator on a programmable pocket calculator, you might have selected the numbers : 8, 18, 1, 14, 23 and 4 which correspond, respectively, to last names beginning with H, R, A, N, W and D. A quick count tells you that the first four of these letters will be enough to give a sample of 20. The attached worksheet indicates with a “B” the sample of employees selected by this method. To summarise, the block method also gives a reasonably representative sample, with all three job categories included: • 5 of 16 maintenance workers • 9 of 40 operators • 6 of 24 supervisors
Exercise 8 – Potential Answers (continued) Sampling Strategies
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 213 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
3. Interval To get a 25% sample, you need to select every fourth name on the list. You could choose any number between 1 and 4 as the starting point. The attached worksheet indicates with an “I” the sample of employees selected by this method. To summarise, the interval method also gives a reasonably representative sample, with all three job categories included: • 4 of 16 maintenance workers • 10 of 40 operators • 6 of 24 supervisors 4. Stratified Suppose you decided that annual retraining was very important for the operators, less important for the maintenance workers, and a relatively low priority for the supervisors. The guidelines would then suggest sampling 50% of the operators, 40% of the maintenance workers, and 20% of the supervisors. Applying the interval method within each stratum would require taking every third maintenance worker, every other operator, and every fifth supervisor. The attached worksheet indicates with an “S” the sample of employees selected by this method. To summarise, the stratification method also gives a reasonably representative sample, with all three job categories included: • 6 of 16 maintenance workers • 20 of 40 operators • 5 of 24 supervisors
Exercise 8 – Potential Answers (continued) Sampling Strategies
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 214 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Analyze Results
1. Random Five of the sample of 20 had not received training, according to the records. Of these, 3 were maintenance workers, 1 was an operator, and 1 was a supervisor. 2. Block Seven of the sample of 20 had not received training, according to the records. Of these, 3 were maintenance workers, 3 were operators, and 1 was a supervisor. 3. Interval Six of the sample of 20 had not received training, according to the records. Of these, 1 was a maintenance worker, 4 were operators, and 1 was a supervisor. 4. Stratification Three of the sample of 6 maintenance workers and 7 of the sample of 20 operators had not received training, according to the records. All of the supervisors in this sample of 5 had received training. (Note: When using this method, it is not correct to combine the results and say “10 out of a sample of 31”.)
Exercise 8 – Potential Answers (continued) Sampling Strategies
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 215 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Conclusion From This exercise
No two probabilistic sampling methods give exactly the same answer, in quantitative terms. However, qualitatively all four methods give comparable results. The three methods that looked at the population as a whole indicated that between 25% and 35% of the people who should have been trained were not and that persons in all three job categories has missed training. The stratification method indicated that 50% of maintenance workers and 35% of operators had not been trained. This latter method did not pick up the fact that some supervisors also had missed training. However, all four approaches picked up on the fact that a substantial fraction of the people in the two “high risk groups” had not received the training.
Exercise 8 – Potential Answers (continued) Sampling Strategies
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 216 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Table 1
Employee Roster
No Job Category Employee Name
Sampling Method Chosen
Trained in Past Year
(Y/N)
R B I S
1 Maintenance Domingo, P. B S Y
2 Maintenance Bell, M.
3 Maintenance Cort, A. I Y
4 Maintenance Greeno, L. R S N
5 Maintenance Herald, M. B N
6 Maintenance Higgins, H. R B Y
7 Maintenance Getchell, M. I S N
8 Maintenance Jones, T.
9 Maintenance Monteiro, L.
10 Maintenance Obbagy, J. S Y
11 Maintenance Plunkett, J. R I Y
12 Maintenance Reid, R. R B N
13 Maintenance Rotberg, F. R B S N
14 Maintenance Savoie, M.
15 Maintenance Sellers, G. R I Y
16 Maintenance Young, R. R S Y
17 Operator Allen, M. B S N
18 Operator Arnold, E. R B Y
19 Operator Bach, J.S. R I S Y
20 Operator Bateman, R.
21 Operator Butler, L. S Y
Exercise 8 – Potential Answers (continued) Sampling Strategies
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 217 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
No Job Category Employee Name
Sampling Method Chosen
Trained in Past Year
(Y/N)
R B I S
22 Operator Catmur, J.
23 Operator Cavalcanti, C. I S Y
24 Operator Coburn, T. R Y
25 Operator Davanzo, L. S Y
26 Operator DiBerto, M. R Y
27 Operator Fitch, T. I S N
28 Operator Fletcher, J.
29 Operator Harris, J. B S N
30 Operator Hasselreis, D. B Y
31 Operator Hryciuk, R. B I S Y
32 Operator Jones, M.
33 Operator Lennon, J. S Y
34 Operator Lewis, S.
35 Operator Loren, S. I S Y
36 Operator McGinnes, M. R Y
37 Operator McLean, R. S Y
38 Operator Moody, C.
39 Operator Murray, E. I S Y
40 Operator Neville, N. B Y
41 Operator Nott, M. R B S Y
23 Operator Nureyev, R. B Y
43 Operator Patel, B. R I S N
44 Operator Picasso, P.
45 Operator Ryder, W. B S N
Exercise 8 – Potential Answers (continued) Sampling Strategies
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 218 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
No Job Category Employee Name
Sampling Method Chosen
Trained in Past Year
(Y/N)
R B I S
46 Operator Schmidt, H. R
47 Operator Smith, F. I S N
48 Operator Smith, J.
49 Operator Smith, N. S Y
50 Operator Stone, O.
51 Operator Tallchief, M. R I S Y
52 Operator Umscheid, M.
53 Operator Voeller, R. S Y
54 Operator Wescott, W. R Y
55 Operator Windsor, C. I S N
56 Operator Yetskalo, V.
57 Supervisor Berstein, L. S Y
58 Supervisor Boehm, P.
59 Supervisor Canton, C. R I Y
60 Supervisor Capogna, S.
61 Supervisor Dolittle, E.
62 Supervisor Ferguson, S. S Y
63 Supervisor Fonteyn, N. I Y
64 Supervisor Gedanke, M.
65 Supervisor Harris, R. B Y
66 Supervisor Hedstrom, G. B N
67 Supervisor Hill, R. B I S Y
68 Supervisor Hogwood, C. B Y
69 Supervisor Lopez, D.
Exercise 8 – Potential Answers (continued) Sampling Strategies
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 219 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
No Job Category Employee Name
Sampling Method Chosen
Trained in Past Year
(Y/N)
R B I S
70 Supervisor Major, J.
71 Supervisor Moore, T. I Y
72 Supervisor Morrison, J. S Y
73 Supervisor Nutty, P. B Y
74 Supervisor Parigot, M.
75 Supervisor Presley, E. R I N
76 Supervisor Robinson, J. B Y
77 Supervisor Shrimpton, J. R S Y
78 Supervisor Starr, R.
79 Supervisor Stricoff, S. I Y
80 Supervisor Thomas, M.
Evaluating Audit Results
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 220 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Evaluating Audit Results
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 221 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Step 1: Understand Management Systems
Conduct opening meetingConduct orientation tourReview audit strategyUnderstand details ofmanagement systems
Step 3: Gather Audit EvidenceEvaluate what needs to be doneDetermine depth & rigor of reviewSelect types of evidenceneeded & methods to gatherthemCompare practices againstrequirementsDocument results
Step 2: Assess strengths& Weaknesses
Consider potential impactsEvaluate management systemsSet priorities for verification
Step 4: Evaluate Audit ResultsEvaluate audit resultsWrite audit findings
Step 5: Report Audit FindingsConduct exit meeting
Prepare Draft Report
Obtain Review Comments fromCorporate HSELaw departmentFacility management
Issue Final Report toFacility managementOperationsCorporate HSELaw department
Develop Action PlanDevelop proposed action(s) toaddress each findingAssign responsibility for corrective actionDevelop timetable
Conduct Follow-upTrack status of correctiveactionsConfirm closure of findings
Pre-Audit Activities On-Site Activities Post-Audit Activities
Select & Schedule Facility Audit
Plan the Audit:Correspond with the facilityAssemble & distribute background informationAssign & communicateaudit responsibilitiesConduct pre-audit meeting
Select Team Members & Confirm their Availability
Evaluating Audit Results
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 222 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
The purpose of Step 4 is to evaluate and summarise the audit results so as to write clear and concise findings that are supported by sufficient audit evidence.
Finding
Exception
There is a problem
Everything is OK
An identified deficiency with respect to a regulatory requirement or a company policy
Good Management Practice
ObservationAn identified weakness with respect to general (industry) standards of good practice in health, safety and environmental management
Management Systems
ObservationAn identified weakness with respect to the processes used by a facility to achieve and maintain conformance with established standards, including programs, policies, equipment, administrative controls etc.
Local Attention Item
An isolated anomaly found in existing programs where regulatory or company standards of performance exist
A conclusion based on documentary,physical, and testimonial evidence, withrespect to health, safety andenvironmental performance
In order to be sufficient to support the audit findings, audit evidence should be: • Relevant • Objective • Persuasive The first two properties relate to the appropriateness of evidence. The last requirement, persuasiveness, refers to its strength.
Evaluating Audit Results
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 223 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Relevance
HSE audit evidence should provide a logical basis of support for the findings. As a check on the relevance of the evidence, an auditor should be explicit about the hypothesis he/she is trying to verify. Thus, information developed through a review of the facility’s file of hazardous waste manifests is relevant as verification of the hypothesis that “[a sample of] shipments of hazardous waste are appropriately documented in terms of the EPA’s manifest requirements.” However, the manifest files, by themselves, would not be relevant to verify the hypothesis that “all hazardous waste shipments made by the facility have, in fact, been manifested.” In the latter case, other documents, such as shipping logs, would also be relevant. Objectivity
Audit evidence is objective if it is free from bias. The objective quality of evidence should lead two auditors examining the same evidence to reach the same conclusion. Auditors need to ask themselves (and each other) whether the evidence collected in Step 3 presents an unbiased and, therefore, representative, picture of the true situation. Bias, and a resulting lack of objectivity, could arise from at least three sources: • The auditor’s sampling strategy could introduce bias. For
example, he/she might review all of the incident investigation reports in the Safety Coordinator’s files and conclude that they were being filled out correctly and comprehensively, without realising that there was a completely separate file in the Maintenance Supervisor’s office, which addressed all of the incidents relating to maintenance personnel.
Evaluating Audit Results
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 224 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• The auditor could be making some implicit assumptions, based on his/her prior knowledge and experience at other facilities, that are, in fact, not true for the facility being audited. This could introduce an inappropriate subjective element into the evaluation process. For example, an auditor might assume that all maintenance personnel handle hazardous waste and, therefore, require annual refresher training, when this might not be the case.
• Facility personnel could overstate their adherence to internal
procedures if they feel a need to appear more confident and efficient than they really are.
Persuasiveness
Evidence is persuasive when it forces a specific conclusion to be drawn and when another reasonable and knowledgeable person would not challenge the validity of the conclusion nor propose a conceptually different alternative. For example, the evidence that life does not exist on the planet Mercury is very persuasive, but the evidence that there is no extraterrestrial intelligence elsewhere in the universe is less so. In the HSE context, the evidence is persuasive that the facility has a fire extinguisher inspection programme when fire extinguishers are tagged with inspection stickers; the facility has records indicating that inspections are conducted; and the person in charge of the inspections knows when they were last done and when the next inspections are due. The table below indicates some examples of relevant, objective, and persuasive evidence to support particular hypotheses.
Evaluating Audit Results
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 225 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Topic Hypothesis Examples of Sufficient Evidence
Data Gathered
• Written hazardous waste training programme has been prepared.
√
• Training records for personnel involved in hazardous waste management are on file.
√
Hazardous waste training programme.
The facility has implemented a hazardous waste training programme.
• Training instructor has the necessary qualifications and expertise to teach the course.
√
• Written respiratory protection programme has been prepared.
√
• Fit testing and medical clearance have been performed.
√
• Periodic inspections of respirators are conducted. √
Respiratory protection programme.
The facility has implemented an effective respiratory protection programme.
• Personnel are trained in the use and maintenance of respirators.
√
Evaluating Audit Results
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 226 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Listed below are some basic techniques for the auditor and audit team to consider in evaluating audit results and determining if there are sufficient data to substantiate the findings. As an Individual
1. Confirm the Appropriateness of the Data Gathered Relative to the Audit Scope
As each auditor completes an assigned protocol step—or a group of protocol steps on a particular topic, such as “Spill Response Training”— he/she should take time to review the actions taken to ensure that sufficient data have been gathered. In doing this, he/she should carefully weigh the data against the criteria discussed on the previous pages: • Relevance • Objectivity • Persuasiveness Auditors should also ask themselves whether they have collected an appropriate mix of physical, testimonial, and documentary evidence, using appropriate data-gathering techniques (observation, inquiry, and testing). If a sampling strategy was used, the auditor should ask him/herself one more time whether the sample taken was, in retrospect, the right one. Finally, individual auditors should be sure they have correctly identified the regulatory and internal requirements relevant to the particular protocol topic. 2. Summarise Conclusions Having satisfied him/herself that he/she has gathered sufficient evidence for each assigned protocol topic, the auditor should next summarise his/her conclusions and outline his/her findings.
Evaluating Audit Results
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 227 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
As a Team
3. Ensure That All Findings Are Substantiated by the Evidence Collected
Having reviewed and evaluated individual audit conclusions, the next step is to meet as a team to review all protocol topics included within the scope of the audit. With the team leader presiding, each of the team members should briefly discuss the actions taken and conclusions reached for each protocol area. As the responsible team member reviews the actions he/she took to address a particular area, other team members should politely and constructively challenge the conclusions. • Critically review the conclusions. It is essential in this step
to critically analyse the rationale for those protocol topics where the team did not identify any findings. For example, if no problems were found with respect to the use of respiratory protection equipment, ask yourselves: “Did we talk to the right people? Are we sure that we accurately identified the affected population? Could there be other plant areas or job tasks where respirators are used (e.g., in the laboratory, unloading rail cars, etc.)?”
• Play “devil’s advocate.” Finally, pause for a moment to ask
yourselves what you could have missed. Given what you have learned about the facility’s programmes, ask yourselves, what is the worst that could happen, and make sure the actions you took to address the protocol area were likely to catch the “worst case,” at least for your “dig deepest” and “dig deep” priorities.
Evaluating Audit Results
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 228 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
4. Prepare Written Findings The final step in evaluating audit results is to prepare written findings to present to the facility at the exit meeting. The written findings can be presented either in a draft report or simply in a list of the findings. The purpose here is to provide an organised, complete, written summary of the exceptions noted by the team. It is important to present the finding in writing because the true nature of the issue may not be fully appreciated until the facility sees the finding in black and white. As the audit team prepares this document, they should pay careful attention to the following: • Ensure factual accuracy. Because the written findings
provide the basis for the exit meeting with facility management, it is critical that each finding listed be factually accurate. Each team member, in reviewing the wording of the finding, should make sure he/she has the facts to substantiate each exception noted.
• Review with facility HSE coordinator. Prior to the exit
meeting, the audit team should review the written findings with the facility individual who is responsible for day-to-day HSE compliance. He/she will want to know what the team is presenting to “the boss” before it is presented. Also, the coordinator may have some legitimate questions and comments on the findings that the team needs to clarify prior to the exit meeting.
Evaluating Audit Results
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 229 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
5. Agree Upon Local Attention Items Local attention items should be identified and categorisation agreed upon as a team. Typically, local attention items are those that meet one or more of the following criteria: • Isolated anomalies found in existing programmes where
regulatory or company standards of performance exist (e.g., an occasional signature omission from documents requiring signatures; a single exit sign not lighted).
• Minor items that lack specific criteria (e.g., MSDSs available but not in each of several locations; responsibilities for responding to complaints or incidents not well coordinated).
• Items that are outside the audit scope (e.g., a safety deficiency observed during an environmental audit).
An item should never be for local attention if it is: • An exception to a regulatory or company policy requirement
and is associated with a flaw in the management system. • A repeat of a finding from a previous audit. • An immediate danger to health, safety, or the environment. Example Local Attention Only Items The large alcohol tank in the tank farm does not have a drain plug in place to back up the spring-loaded valve. Two unlabelled 25-liter drums were observed on the concrete pad behind the fire water pump building. (Containers elsewhere at the site were labelled properly.) One wooden ladder being used outside the aerosol gas house was found to be unstable. Facility personnel removed and disposed of it immediately. (The team did not note any other defective ladders.)
Exercise 9 Evaluating Audit Results - Specific “Local” vs. “Report” Decisions
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 230 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Objective
The purpose of this exercise is to apply the guidelines for classifying local attention items to audit results so as to gain experience in determining whether findings are isolated anomalies or whether the information is sufficient to justify determination as an audit finding. Instructions
On the following pages you will find several sets of findings developed by audit teams. What we have attempted to do is to describe situations that represent varying degrees of “severity” of deficiency with regard to a particular requirement. You are to assume that these situations are mutually exclusive. That is, the auditor observed either the situation described in a. or the situation described in b. or the situation described in c. Also, you should assume that the situation statement describes all of the observed deficiencies with respect to the indicated requirement; there are no other closely related exceptions to the requirement (i.e., if the exercise says you saw on employee not wearing hearing protection, you may infer that all other employees observed were wearing the required protection). You challenge is to work in small groups to decide, within each of these sets of specific findings, where you would draw the line between “local attention” and “for report”. Note that it is quite possible to decide that all three of the versions are in the same category (e.g., all local attention items or all reportable). Note: For purposes of this exercise, do not concentrate too much over whether you would have one or more than one finding based on the information given.
Exercise 9 Evaluating Audit Results - Specific “Local” vs. “Report” Decisions
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 231 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Set 1: Hearing Conservation Requirement: The facility has several areas in which noise levels exceed 90 dBA (8-hour TWA); therefore, hearing protection is required. In addition, it is company policy that signs be posted in these areas and that the entrances to these areas be marked off with pink or green painted lines on the floor. a. A door opens to an air compressor equipment area where
hearing protection is required. The area is posted as “Hearing Protection Required”, but no pink/green marking lines have been marked on the floor.
b. A door opens to an air compressor equipment area where hearing protection is required. The area is posted as “Hearing Protection Required”, but no pink/green marking lines have been marked on the floor. One employee was observed working in the air compressor area with no ear protection.
c. A door opens to an air compressor equipment area where hearing protection is required. The area is posted as “Hearing Protection Required”, but no pink/green marking lines have been marked on the floor. One employee was observed working in the air compressor area with no ear protection. The team also observed one employee (out of ten) in the machine shop and one employee in the pump room (both posted areas) not wearing hearing protection.
Exercise 9 Evaluating Audit Results - Specific “Local” vs. “Report” Decisions
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 232 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Set 2: Ventilation Flow Measurement Requirement: As a matter of good management practice, the facility should conduct and document inspection and maintenance activities on ventilation systems to ensure that filters are changed regularly and that accurate air flow tests are conducted to confirm performance. a. The facility has not measured the air flow for the hood in the
quality control laboratory at any time since installation, when the flow was checked by the vendor.
b. The facility has not measured the air flow for the hood in the quality control laboratory at any time since installation, when the flow was checked by the vendor. There are no flow measurement instruments available on site for testing the performance of ventilation systems used for air contaminant exposure control in the process areas; flows in process areas are checked once yearly by a contractor.
c. The facility has not measured the air flow for the hood in the quality control laboratory at any time since installation, when the flow was checked by the vendor. There are no flow measurement instruments available on site for testing the performance of ventilation systems used for air contaminant exposure control in the process areas; flows in process areas are checked once yearly by a contractor. The site has installed three new ventilation systems since the last contractor inspection; no flow measurements have been made on these systems.
Exercise 9 Evaluating Audit Results - Specific “Local” vs. “Report” Decisions
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 233 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Set 3: Chemical Containment Requirement: The company’s spill control plan states that secondary containment must be provided in all chemical storage areas. a. In the storeroom, there was one five-gallon container (not
empty) overhanging the edge of the spill containment structure.
b. In the storeroom, there was one five-gallon container (not empty) overhanging the edge of the spill containment structure. In this area, there were also ten five-gallon containers set on the floor beside the spill containment structure. The containment structure was full of cans.
c. In the storeroom, there was one five-gallon container (not empty) overhanging the edge of the spill containment structure. In this area, there were also ten five-gallon containers set on the floor beside the spill containment structure. The containment structure was full of cans. In the maintenance area, there were five-gallon drums of solvent with no containment. In the boiler room, there was a 55-gallon drum with no spill containment.
Exercise 9 Evaluating Audit Results - Specific “Local” vs. “Report” Decisions
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 234 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Set 4: Document Control Requirement: As a matter of good management practice, there should be a method of ensuring that critical HSE documents are periodically reviewed and revised as necessary. a. The facility does not have a formal document control
programme to ensure that plans such as the emergency response plan are periodically reviewed, amended, and reissued if necessary.
b. The facility does not have a formal document control programme to ensure that plans such as the emergency response plan are periodically reviewed, amended, and reissued if necessary. The current emergency response plan dates from 1990 and does not include the correct home telephone number for the alternate response coordinator.
c. The facility does not have a formal document control programme to ensure that plans such as the emergency response plan are periodically reviewed, amended, and reissued if necessary. The current emergency response plan dates from 1990 and does not include the correct home telephone number for the alternate response coordinator. It also does not contain specific plans for responding to spills at the aqueous ammonia tank installed in 1993.
Exercise 9 Evaluating Audit Results - Specific “Local” vs. “Report” Decisions
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 235 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Set 5: Management of “Used Oil” Drums Requirement: Company policy states that no more than 55 gallons of used oil is to be stored in working areas. All full drums are to be stored in designated areas, which are to be managed as 90-day accumulation areas. a. One of the six designated “used oil” storage areas showed
evidence of past spills and partially illegible container labels.
b. One of the six designated “used oil” storage areas showed evidence of past spills and partially illegible container labels. Three apparently full drums labeled “used oil” were observed in the maintenance shop and two at the south side fuelling area.
c. One of the six designated “used oil” storage areas showed evidence of past spills and partially illegible container labels. Three apparently full drums labeled “used oil” were observed in the maintenance shop and two at the south side fuelling area. Facility personnel interviewed could not identify individual(s) specifically responsible for “used oil” drum handling.
Exercise 9 – Potential Answers Specific “Local” vs. “Report” Decisions
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 236 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Set 1: Hearing Conservation
a. local b. report c. report
Set 2: Ventilation Flow Measurement
a. local b. local c. report
Set 3: Chemical Containment
a. local b. report c. report
Set 4: Document Control
a. report b. report c. report
Set 5: Management of “Used Oil” Drums
a. report b. report c. report
You may find that this opinion may change in the light of additional background information. For example the recommended answers for set 2 would be dependent on what the ventilation hood is used for, its toxicity and the frequency of exposure to employees.
Evaluating Audit Results
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 237 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Management Systems Information
In addition to the above, there are several other methods the auditor can apply to determine if the data are sufficient to develop a management systems observation. These are: Identify trends among the compliance findings. Perform a cause and effect analysis. This process is generally conducted as a team. 1. Identify Trends Among the Compliance Findings
Once the auditors have developed the complete list of findings, they should, as a team, review the list to develop an integrated, organised summary. In this step, the team should do the following: • Identify common findings. Look for situations where two or
more of the individual findings listed may relate to one basic problem, and may represent a system deficiency.
• Look for patterns or trends. In reviewing the findings, try to
find whether any patterns emerge which suggest that several findings should be combined. Ask yourselves whether there are several findings that, when viewed as a group, may have greater significance.
• Be alert to systemic issues. Ask yourselves whether the
symptoms observed (errors, omissions, etc.) are manifestations of a more fundamental systems weakness.
For example, the audit team may have several findings all related to the same general topic, such as training, and the findings themselves all point to the underlying management systems deficiency that there is no means to identify and track HSE-related training.
Evaluating Audit Results
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 238 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Individual findings related to:
Hazardous waste trainingSpill response trainingHazard communication trainingRespirator use trainingConfirmed spaces entry training
Individual findings related to:
Hazardous waste trainingSpill response trainingHazard communication trainingRespirator use trainingConfirmed spaces entry training
The management system deficiency:
No means to identify and track HSE-related training
The management system deficiency:
No means to identify and track HSE-related training
However, management systems observations are not always so easy to recognise during an audit. Thus, the team may want to relate the findings back to the management systems processes. In Step 1, the team gained a lot of insight into how HSE compliance is supposed to be managed at the facility. As the team verified conformance with the facility’s system, they most likely uncovered some gaps and breakdowns. As a means of discerning the management systems deficiencies, the team should try to relate individual findings to one or more of the management processes utilised by the facility to manage its HSE matters. The table on the following page describes how the individual findings can be categorised within the various management processes to assist in developing management system observations.
Evaluating Audit Results
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 239 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Compliance Findings Management System Activity
Management Systems Observation
• The facility has not developed an air emissions inventory.
• The facility has not conducted a noise survey.
• The facility has not characterised 12 different waste streams.
• The facility has not conducted workplace monitoring for xyz chemicals.
Assessing The facility has not assessed its HSE issues and risks in several areas, including: • Air emissions inventory • Noise survey • Waste characterisation • Workplace montinoring for
XYZ chemicals
• The facility has not conducted 7 of 12 monthly safety inspections.
• The facility does not document its fire extinguisher inspections.
• Respirators were observed stored out on shop floor.
Reviewing or Organisation/ Resources
There is no comprehensive system for HSE-related inspections. or Responsibility and accountability for the HSE inspection function have not been clearly defined.
2. Perform a Cause and Effect Analysis
Another technique to develop a management systems observation is to identify the underlying causes associated with compliance and good management practice findings. The following examples illustrate this approach.
Evaluating Audit Results
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 240 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Finding The facility exceeded its wastewater pH limit on several occasions during the review period. Upon investigation, the team learned that the wastewater treatment system controls had not been maintained regularly and, thus, failed to properly adjust the pH of the effluent. Incident/Effect An exceedance of the pH (level of acid/base) was noted at the wastewater treatment plant. Possible Causes • Failure of pH controls resulting from lack of maintenance • Training • Inspection • Lack of redundant systems • Wrong equipment • System to ensure QA (procedures) • Labor shortage
Finding During the audit, the team observed that the employee’s hand could be pulled into the mobile pumping unit because the unit does not have a guard. Upon investigation, the team learned that the guard had been removed to make access to the equipment easier and a system is not in place to periodically verify that guards are in place where required. Incident/Effect The belt-driven pulley on mobile pumping unit 5123 does not have a guard. Possible Causes • Training • Inspection • Lack of equipment
To make this technique truly useful, the cause-effect model begins at the incident, or effect, and proceeds backward by asking “why” until the appropriate management system conclusion is reached.
Evaluating Audit Results
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 241 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Root cause
Cause 3
Cause 2Cause 1 Incident
After the cause (i.e., the main point to communicate to management) is identified, the team is in a position to write a management systems observation.
Writing Audit Findings
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 242 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Writing Audit Findings
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 243 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
As the team prepares the list of audit findings, it is important to ensure that each is written in a manner to clearly and accurately communicate the facts. It is important to keep in mind several principles when wording audit findings: 1. Do Not Overstate the Facts
State the facts as you have discovered them but avoid drawing overly broad conclusions.
Don’t say... If you mean...
The facility does not have a PCB inspection programme.
The facility’s PCB inspection programme does not include PCB-contaminated transformers.
The facility does not have a respiratory protection programme.
The facility’s respiratory protection programme does not include fit testing or routine inspection and maintenance of respirators.
2. Distinguish Between Performance and Documentation
Some regulatory or corporate requirements specify that a particular activity or programme be conducted, but do not specify that the completion of the activity or programme be documented. In other cases, regulatory or corporate requirements specify that the activity or programme be conducted, and that it be documented to verify that it was conducted.
Don’t say... If you mean...
Weekly hazardous waste inspections are not conducted.
Weekly hazardous waste inspections are not documented.
Weekly ladder inspections are not conducted.
Weekly ladder inspections are not documented.
Writing Audit Findings
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 244 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
3. Avoid Generalities
Generalities and vague reporting will confuse and mislead the reader. The specific problem should be succinctly communicated. In addition, although a finding may be worded factually, it may not contain enough information to fully communicate the nature and extent of the problem.
Too general More helpful
The facility’s contingency plan is incomplete.
The facility’s contingency plan does not include the following elements: a. Agreement with local authorities. b. Types and location of fire
protection equipment. c. Listing of emergency telephone
numbers.
Employees have not received safety training.
Four of 30 maintenance mechanics have not received lockout/tagout training.
4. Do Not Draw Legal Opinions
Legal judgments, interpretations, and conclusions should be avoided when writing audit findings. Generally speaking, legal conclusions can be characterised by words such as “in violation of…,” “not in compliance with…,” “as required by…,” etc.
Writing Audit Findings
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 245 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Legal conclusion Factual conclusion
Hazardous waste drums were not labeled as required by 40 CFR 262.34.
Drums storing hazardous waste were not labeled with the words, “Hazardous Waste,” or other words indicating their contents. (40 CFR 262.34(c)(1)(ii))
During the review period, the facility was not in compliance with 29 CFR 1910.95.
During the review period, the facility did not conduct annual hearing conservation training. (29 CFR 1910.95)
5. Give Regulatory or Company Policy References
Because the basis for a finding may not always be clear to the report recipient, particularly if the report recipient is an individual who is not involved with environmental, health, and safety issues on a daily basis, regulatory or company policy references should be included.
Poor Improved
Required annual hazardous waste training has not been conducted within the past 18 months.
Annual hazardous waste training has not been conducted within the past 18 months. (40 CFR 265.16 and XYZ Company Policy, HAZWASTE 3.2)
Required annual hearing conservation testing has not been conducted within the past 18 months.
Annual hearing conservation training has not been conducted within the past 18 months. (29 CFR 1910.95 and XYZ Company Policy, HEARCON 4.7)
Writing Audit Findings
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 246 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
6. Avoid Extreme Language
Refrain from using such deprecating words as careless, terrible, dangerous, intentional, severe, reckless, incompetent, and the like. These words can be broadly interpreted and are not helpful in communicating the exact nature of the problem.
Poor Improved
The lack of spill containment provisions on the loading dock may lead to a dangerous situation.
Spill containment provisions are not available at the loading dock area where hazardous chemicals are loaded/unloaded adjacent to a drain that discharges to surface waters.
The lack of documented confined space entry procedures for the manufacturing operations may lead to an injury/accident.
The manufacturing operations do not have written confined space entry procedures.
7. Use Familiar Terminology
Not all recipients of the report will be involved in health, safety and environmental activities on a daily basis and, thus, they may not be as familiar with the health, safety and environmental acronyms, abbreviations, and regulatory jargon as the auditors are.
Poor Improved
The facility does not have pollution prevention equipment to prevent exceedances of TSS, BOD, and oil and grease in its discharges to the POTW.
The facility does not have pollution prevention equipment to prevent exceedances of total suspended solids (TSS), biochemical oxygen demand (BOD), and oil and grease in its discharges to the publicly owned treatment works (POTW).
Four of 12 P&IDs reviewed were out of date.
Four of 12 piping and instrumentation diagrams (P&IDs) were out of date.
Writing Audit Findings
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 247 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
8. Do Not Focus Criticism on Individuals or Their Mistakes
Do not identify the individual(s) involved in a performance finding or as sources of information. The audit is a review of facility programmes and practices, not of individuals.
Poor Improved
John Doe and Jane Smith were observed . . . .
The team observed maintenance personnel . . . .
9. Avoid Contradictory Messages
Activities or programmes presented in a positive light, when the ultimate message will involve pointing out deficiencies, may confuse the reader and obscure the real message being conveyed.
Poor Improved
Although the facility has a well-written waste analysis plan, it does not include parameters for each hazardous waste analyzed or the frequency of analysis.
The facility’s waste analysis plan does not include the following: a. Parameters for each hazardous
waste analyzed. b. Frequency of analysis.
Writing Audit Findings
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 248 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Poor Improved
Although the facility has a site-specific written hazard communication programme, the hazard communication inventory is not up to date, nor does the facility have a procedure to ensure that employees receive hazard communication training upon initial assignment.
In reviewing the facility’s hazard communication practices, the team noted the following: a. The facility’s hazard
communication inventory is not up to date.
b. The facility does not have a
procedure to ensure that employees receive hazard communication training upon initial assignment.
10. Group Similar Findings
Rather than state several individual findings, look for patterns or trends and group similar findings to more fully convey the message.
Individual Grouped
Quarterly Discharge Report The analysis results for one of 13 samples of the wastewater discharged to the publicly owned treatment works (POTW) was not included in the quarterly discharge report. Reporting of Analytical Results Analytical results for samples taken at internal outfalls were not reported to the POTW.
Reporting of Sample Events to POTW In reviewing quarterly monitoring reports and analytical data, the team noted the following: a. One of 13 samples taken and
analyzed of the wastewater discharged to the publicly owned treatment works (POTW) was not included in the quarterly discharge report.
b. Analytical results of samples taken
at internal outfalls were not reported to the POTW.
Writing Audit Findings
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 249 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Individual Grouped
Labelling Confined Spaces The facility has not posted danger signs or used any other equally effective means to identify the existence and location of and the danger posed by the permit-required confined spaces. Confined Space Permit Programme The confined space permit programme does not include procedures: a. To be taken in the event that a
hazardous atmosphere is detected during entry.
b. For verifying that conditions in the
permit space are acceptable for entry throughout the duration of the authorised entry.
Permit-Required Confined Spaces In reviewing the facility’s confined space entry programme, the team noted the following: a. The facility has not posted danger
signs or used any other equally effective means to identify the existence and location of and the danger posed by the permit-required confined spaces.
b. The confined space permit
programme does not include procedures:
1) To be taken in the event that a
hazardous atmosphere is detected during entry.
2) For verifying that conditions in
the permit space are acceptable for entry throughout the duration of the authorised entry.
11. Write Management Systems Observations
To be effective, management systems observations need to be: • Crisp • To the point (one point at a time) • Clearly framed • Expressed in terms meaningful to managers
Writing Audit Findings
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 250 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
What’s Wrong With This Finding?
Clearly Defined Responsibilities A reorganisation of the plant management functions had recently occurred. In conjunction with this reorganisation, a newly created position of Manager, Health, Safety, and Environment Special Projects had been established with the main job activities listed in the personnel job description form. The Safety department organisation has also been expanded. Greater emphasis is being placed on health, safety and environmental issues as evidenced by top plant management now having HSE concerns listed in their formal job description outlines. Plans are under way to expand this emphasis by including these same issues in the job descriptions of first line supervision. In the past, environmental concerns were jointly addressed by the Engineering department and the Maintenance department. The majority of the contacts with governmental agencies were handled by these two departments. Under the recent reorganisation, the Engineering and Maintenance departments have been combined under one manager and most of the previous job activities and responsibilities of both departments have been combined. Although the job duties and responsibilities of the newly created position of Manager, HSE have been defined by management, no accountability document has been prepared to indicate specific responsibility or joint accountability with other departments. In addition, it was not clearly defined who can sign HSE documents and correspondence.
This management systems observation regarding “clearly defined responsibilities”: • Uses 208 words • Includes extraneous information on:
− The reorganisation itself − The Safety department − Greater emphasis on HSE issues − Top management and supervisors job descriptions
• Unnecessarily muddles the basic accountability issue with the
lesser question of who should sign HSE documents and correspondence.
Writing Audit Findings
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 251 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
A Sharper Picture Clearly Defined Responsibilities The recent organisation has left undefined the issue of whether the new HSE Manager position shares any joint responsibilities or accountabilities with the now combined Engineering and Maintenance department.
This revised finding is: • Twenty-nine words, or an 86 percent reduction in length. • Crisp and to the point, which is that of joint accountability. • Clearly framed as a management issue, that of clarifying
accountabilities. • Much more easily read, understood, and acted upon.
Writing Audit Findings
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 252 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Example Management Systems Observations
Tracking of Training The facility has not developed a comprehensive system for tracking the movement of employees who change job positions to ensure that regulatory required training is provided in a timely fashion. For example, environmental management staff do not know which employees (some may be security contractors) are responsible for chemical and petroleum product loading and unloading. Promotion of Safety and Health Management Management commitment to the recently reissued health and safety policy does not appear fully evident in that: • A facility inspection programme that includes senior management
participation is not in place. • Promotional activities such as safety/environmental recognition awards,
safety contests, etc., have not been developed. • There are gaps in the attendance of employee, supervisory, and
management personnel in mandatory safety training sessions. • Middle management participation in at least two safety, health, and loss
prevention meetings with their employees per year does not occur. Safety Coordinator Programme In reviewing the facility’s safety coordinator programme, the team noted several issues that impact the effectiveness of this programme. For example: • The roles and responsibilities of the safety coordinators have not been clearly
defined and approved by line management. • The safety coordinators do not regularly attend and participate in safety
coordinator meetings. • Some of the safety coordinators lack the training necessary to perform their
responsibilities in accordance with facility policies and procedures.
Writing Audit Findings
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 253 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Example Management Systems Observations
Management of Health and Safety Issues The management of health and safety issues is not fully integrated into day-to-day operational activities as evidenced by the concerns noted in the areas of contractor compliance, electrical safety, and gaps in the safety coordinator inspection programme. Based on interviews with facility staff, several barriers were noted that hinder the effective integration of health and safety with operations. For example: • The absence of management directives outlining the role of health and safety
staff in the day-to-day administration of compliance programmes throughout the facility.
• The lack of periodic meetings among line management and health and safety staff to discuss and foster integration of health and safety into operations.
In addition, the process for developing and revising safety, health, and loss prevention procedures does not include a formal review by line management to help ensure approval and effective implementation.
12. Consider Using a Template
Consider using a template to prepare a preliminary draft of your findings.
Template for Writing Findings
Template Finding
The Statistical Finding Fifteen of a sample of 30 of the 52 employees who routinely enter confined spaces did not receive training during the last year
[# of deficiencies] of a sample of [# in sample] of [# in universe] of what is wrong
The Have/Do Finding The facility does not have an air pollution permit to operate the three boilers in Powerhouse B on site Facility personnel do not conduct or document inspections of the hazardous waste accumulation area
Who or what / does not / do not have / what they do not have Who / does not do it / does not do (perform) / what they do not do
Writing Audit Findings
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 254 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Template for Writing Findings
Template Finding
The Grouped Finding In reviewing the facility’s practices with respect to flammable/combustible liquid handling, the team noted the following: a. An oxidiser (nitric acid) was
stored in the same flammable liquid storage cabinet as organic solvents
b. Self-closing valves have not been installed on combustible/flammable drums in the dispensing area
c. Devices for bonding dispensing drums to portable metal containers were not in use
[29 CFR 1910.106]
In touring/reviewing / the area toured/document reviewed / the team noted the following:
a. List the exceptions.
The Observed Finding During the facility tour, the team observed an oil sheen near the outfall to the lake
Who or what / observed / what was observed?
The Told Finding The team was told that there have been several spills of hazardous waste to the storm sewers
The team was told that what the team was told
13. Avoid These Words
Adequate Inadequate Some Few Many Not all Sometimes
Occasionally Insufficient Compliance Noncompliance Required Violation
Exercise 10 Writing Audit Findings - Critiquing Audit Findings
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 255 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Objective
The purpose of this exercise is to review a set of written audit findings and determine whether the findings clearly and appropriately communicate the audit results. Instructions
Based on the principles regarding how to properly word findings, critique the following exit meeting discussion sheets. Also offer suggestions on how to rewrite the findings.
Exercise 10 Writing Audit Findings - Critiquing Audit Findings
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 256 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exit Meeting Discussion Sheet Facility XYZ Facility Discipline HSE
Audit Team Present John Collins, Brenda Fields, Bob Taylor, Jane Smith
Facility Management Present Ralph Gold
Others Present Bill Dunn
Discussion Date MM/DD/YY Prepared by John Collins
# Exception Critique
1 We could not verify that waste manifests were received from TSDFs (Treatment, Storage and Disposal Facility) within 45 days of shipment.
2 Current storage of emergency response equipment may result in increased likelihood of failure.
3 There is minimal on-site compliance with corporate or department contractor safety policy and procedures.
4 Some of the air sources are being operated without proper permits and some are not adequately maintained.
5 The facility’s central MSDS file is very neat and accessible to those employees who should see it. Not all materials used or stored by the facility have MSDSs (Material Safety Data Sheets) in the central file. Those MSDSs reviewed appeared complete and contained the appropriate information.
6 There are no toe boards and missing hand rails.
7 A discrepancy exists among the frequency of safety inspections.
Exercise 10 Writing Audit Findings - Critiquing Audit Findings
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 257 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exit Meeting Discussion Sheet (continued)
# Exception Critique
8 Ron Kline and Seth McGee were not familiar with the company’s hazard communication programme or could identify where MSDSs were located.
9 The facility’s computer monitoring programme for permit expiration was found to lack a procedure to make sure that all permits were entered into the computer system to begin with.
10 No inspection and maintenance records were available to the audit team and no documented procedures for rail car loading / unloading.
11 There is insufficient personnel to manage all HSE matters given the requirements put forth in the operating manuals which describe the gamut of environmental, health, and safety regulations.
12 The facility often goes through changes in operations which result in additional environmental impacts only to find notifications to permit conditions and variation in employee safety conditions. A management system should be addressed to fix this problem.
Exercise 10 – Potential Answers Writing Audit Findings - Critiquing Audit Findings
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 258 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exit Meeting Discussion Sheet Facility XYZ Facility Discipline HSE
Audit Team Present John Collins, Brenda Fields, Bob Taylor, Jane Smith
Facility Management Present Ralph Gold
Others Present Bill Dunn
Discussion Date MM/DD/YY Prepared by John Collins
# Exception Critique
1 We could not verify that waste manifests were received from TSDFs within 45 days of shipment.
− Implies auditor has not done job properly
− Need to include citation
− What does TSDF stand for?
2 Current storage of emergency response equipment may result in increased likelihood of failure.
− Should not use ambiguous terms (e.g. may, increased)
− So what?
− Which equipment?
3 There is minimal on-site compliance with corporate or department contractor safety policy and procedures.
− Not specific
− Does not describe problem to help site correct it
4 Some of the air sources are being operated without proper permits and some are not adequately maintained.
− Do not use “some” or “proper” or “adequately”
− Be specific; which sources, how many?
5 The facility’s central MSDS file is very neat and accessible to those employees who should see it. Not all materials used or stored by the facility have MSDSs in the central file. Those MSDSs reviewed appeared complete and contained the appropriate information.
− What does MSDS stand for?
− Combines good and bad findings
− Do not use “not all” or “appeared”
Exercise 10 – Potential Answers Writing Audit Findings - Critiquing Audit Findings
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 259 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exit Meeting Discussion Sheet (continued)
# Exception Critique
6 There are no toe boards and missing hand rails.
− So what?
− How many?
− Where?
7 A discrepancy exists among the frequency of safety inspections.
− What is the discrepancy?
− Needs clearer description
8 Ron Kline and Seth McGee were not familiar with the company’s hazard communication programme or could identify where MSDSs were located.
− Avoid using names
9 The facility’s computer monitoring programme for permit expiration was found to lack a procedure to make sure that all permits were entered into the computer system to begin with.
− Bad English
− Irrelevant information
10 No inspection and maintenance records were available to the audit team and no documented procedures for rail car loading / unloading.
− So what?
− What about the procedures for loading / unloading?
11 There is insufficient personnel to manage all HSE matters given the requirements put forth in the operating manuals which describe the gamut of environmental, health, and safety regulations.
− Judgmental not factual
− Do not use “insufficient”
12 The facility often goes through changes in operations which result in additional environmental impacts only to find notifications to permit conditions and variation in employee safety conditions. A management system should be addressed to fix this problem.
− Bad English
− What’s the problem?
− Should the finding include a recommendation?
Post-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 260 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 261 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Step 1: Understand Management Systems
Conduct opening meetingConduct orientation tourReview audit strategyUnderstand details ofmanagement systems
Step 3: Gather Audit EvidenceEvaluate what needs to be doneDetermine depth & rigor of reviewSelect types of evidenceneeded & methods to gatherthemCompare practices againstrequirementsDocument results
Step 2: Assess strengths& Weaknesses
Consider potential impactsEvaluate management systemsSet priorities for verification
Step 4: Evaluate Audit ResultsEvaluate audit resultsWrite audit findings
Step 5: Report Audit FindingsConduct exit meeting
Prepare Draft Report
Obtain Review Comments fromCorporate HSELaw departmentFacility management
Issue Final Report toFacility managementOperationsCorporate HSELaw department
Develop Action PlanDevelop proposed action(s) toaddress each findingAssign responsibility for corrective actionDevelop timetable
Conduct Follow-upTrack status of correctiveactionsConfirm closure of findings
Pre-Audit Activities On-Site Activities Post-Audit Activities
Select & Schedule Facility Audit
Plan the Audit:Correspond with the facilityAssemble & distribute background informationAssign & communicateaudit responsibilitiesConduct pre-audit meeting
Select Team Members & Confirm their Availability
Post-Audit Activities – Introduction
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 262 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
The objectives and associated responsibilities of the post-audit activities are outlined below.
Objectives Responsibilities • To ensure that the audit results
are clearly communicated to the appropriate levels of management.
Meeting this objective is typically part of the audit team’s responsibility. This is accomplished by means of a formal, written audit report, as described in this section of the manual.
• To ensure that all audit findings are addressed by management through the implementation of a formal corrective action process.
Meeting this objective is typically the responsibility of line management, although the audit team may be asked to make recommendations, review proposed action plans, and/or track the implementation and closure of corrective action.
• To evaluate the effectiveness of the audit and provide suggestions for improving future efforts.
• To share lessons learned during
the audit, especially to similar facilities.
Meeting these last two objectives is frequently among the responsibilities of the audit programme manager and team leader(s).
In this section of the manual, we discuss: • Preparation of audit reports • Protection of audit results • Audit policy statements • Quality assurance in the audit process
Post-Audit Activities – Preparation of Audit Reports
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 263 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Purpose of Audit Reports
The overall goal of an audit report is to document the audit findings clearly and accurately. Within this overall goal, an audit report has three basic purposes: • To document the scope of the audit and the audit team’s
conclusions regarding the facility’s compliance status. • To provide appropriate levels of management with information
on the results of the audit—information sufficient to meet the needs of the report’s recipients and consistent with the overall objectives of the audit programme.
• To initiate corrective action so that once exceptions to
applicable requirements have been identified, action steps are set in motion to correct the deficiencies found.
A strong linkage exists between the report’s purpose and the overall audit programme’s objective. For example, where the primary objective of the audit programme is to provide assurance to management, the purpose of the audit report is to provide top management with information on the more significant findings. Likewise, when the primary purpose of the audit programme is to provide plant management with information on the environmental, health, and safety status of the facility, the purpose of the audit report is to help facility managers.
Post-Audit Activities – Preparation of Audit Reports
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 264 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Report Recipients
What Are “Appropriate” Levels of Management? Most audit programmes use a hierarchical reporting scheme. • Most, but not all, programmes today require dissemination of
audit reports to high levels of management. • Increasingly, top corporate managers and/or boards of
directors expect to be informed of audit results. • There may, however, be some audit findings (“local attention
items”) that do not require reporting beyond the facility manager level.
An example of a hierarchical reporting scheme is shown on the following page. Example of a Hierarchical Reporting Scheme
Who How What
Facility HSE staff Daily communication All deficiencies noted
Facility manager Exit meeting; draft and final reports
All deficiencies noted
Corporate HSE Affairs; Law Department; division or group management
Draft and final reports All deficiencies noted, except local attention items
Corporate management Periodic status summary Significant matters; overall patterns and trends; general programme status
Board of directors Summary presentation at a board meeting
Overall HSE performance; most significant matters
Post-Audit Activities – Preparation of Audit Reports
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 265 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Report Form and Content
The health, safety and environmental audit report, according to international and industry standards for audit programme performance, must be in writing in order to allow results to be widely communicated, misunderstandings reduced, and follow-up facilitated. Audit reports must be: • Objective • Clear • Concise • Timely The type of information and level of detail provided in an audit report depends upon the objectives of the audit programme, the needs of the report recipients, and the problems identified. An example of a report outline that meets the minimum requirements specified by the Environmental Auditing Roundtable is provided below. I. Introduction
Purpose, date, and scope of the audit Name, location, and description of the audited site Names of the audit team members Criteria utilised in performing the audit Deviations from the planned scope
II. Exceptions and observations discovered during the audit III. General instructions for response and follow-up
Post-Audit Activities – Preparation of Audit Reports
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 266 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Similarly, ISO 14000 (ISO 14010, Section 5.7 and ISO 14011, Section 5.4.2) describes the contents of the audit report as follows:
The audit report should be dated and signed by the lead auditor. The audit report should contain the audit findings or a summary thereof with reference to supporting evidence. Subject to agreement between the lead auditor and the client, the audit report may also include the following: a. The identification of the organisation audited and of the client. b. The agreed objectives, scope, and plan of the audit. c. The agreed criteria, including a list of reference documents against which the
audit was conducted. d. The period covered by the audit and the date(s) the audit was conducted. e. The identification of the auditee’s representatives participating in the audit. f. The identification of the audit team members. g. A statement of the confidential nature of the contents. h. The distribution list for the audit report. i. A summary of the audit process, including any obstacles encountered. j. Audit conclusions, such as:
– EMS conformance to the EMS audit criteria. – Whether the system is properly implemented and maintained. – Whether the internal management review process is able to ensure the
continuing suitability and effectiveness of the EMS. Audit results should be reported in a manner that is clear and easily understood by the recipient of the report. Audit findings need to be described in an appropriate managerial context, which takes into account the recipient’s extent of familiarity with the subject matter. They also need to be free of jargon and unfamiliar terminology. Audit reports should also be factual, unbiased, and free from distortion. Findings should be prepared without prejudice and expressed as pertinent statements of fact, which are supported by sufficient, valid, and documented evidence gathered during the audit. The focus of the audit report should be on the findings developed during the audit and, depending on the scope of the audit programme, recommendations where necessary.
Post-Audit Activities – Preparation of Audit Reports
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 267 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
The length of the audit report varies widely, depending on the scope of the audit and the depth of detail needed to meet the recipients’ needs. Potential Components of an Audit Report
A strong relationship exists between the format and content of the audit report and the needs of the report’s recipients. Although audit reports should satisfy the minimum criteria described on the previous pages, there are also a number of potential components to consider. The decision to incorporate one or more of these alternatives into the audit report is really a programme design issue and not within the discretion of an individual auditor or audit team. Nevertheless, auditors need to have a clear understanding of how their company’s audit reports are designed, so that they can contribute effectively to the reporting process. Examples of audit report components are described on the following pages.
Post-Audit Activities – Preparation of Audit Reports
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 268 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exceptions Only
An exception report is essentially designed to meet the minimum report requirements. The “exceptions and observations” section of such a report is its distinguishing characteristic in that it simply communicates departures from established governmental or internal standards and observations with respect to general (industry) standards of good practice in HSE management. This format does not necessarily provide the report recipients with a means of interpreting the overall significance of the audit findings and may not represent the most desirable option for readers who lack sufficient context to understand and assess the status of the facility’s performance. This alternative is illustrated below.
II. Exceptions and Observations A. Water Pollution Control 1. Sampling Frequency (Regulatory) The facility does not sample National Pollutant Discharge Elimination System (NPDES) parameters such as Biological Oxygen Demand (BOD) and pH on a biweekly schedule. [40 CFR 122.41] 2. Stormwater Pollution Prevention Plan (Regulatory) The facility has not completed a Stormwater Pollution Prevention Plan (SWPPP) and submitted the corresponding certification form to the Bureau of Stormwater Permitting. [NJPDES General Permit NJ0088315] 3. Etc.....
Post-Audit Activities – Preparation of Audit Reports
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 269 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Recommendations
It is becoming more common for an audit programme to assist the facilities by developing suggested recommendations for corrective action to address each of the exceptions and observations. This is particularly helpful when the facility HSE personnel may not be completely familiar with the requirements (especially regulatory requirements) or to facilitate the development of corrective actions within a specified time frame. It is important to ensure that recommendations are written in a way to convey what should be done, without dictating to facility management how it is to be done. Some examples that illustrate this what/how distinction are provided in the table below.
Don’t tell them how Do tell them what
The facility should require that the plant environmental coordinator review all proposed changes to process chemicals.
The facility should establish authorisation procedures to ensure that accountable personnel assess the impacts of process chemical changes on waste stream characteristics.
The facility should hire an industrial hygienist and two additional process safety engineers.
The facility should assess the need for additional industrial hygiene and process safety resources to perform an employee exposure assessment and process hazards analysis in units A, B, and C.
Post-Audit Activities – Preparation of Audit Reports
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 270 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Audit Opinion
An audit opinion provides an overall classification of the facility’s performance in addition to a list of exceptions and observations. An opinion report provides management with a means for interpreting the significance of the audit results and assists them in focusing their resources on areas where improvement is most needed. Arthur D. Little has developed a five-level opinion scheme, as described in the table on the following page. When an audit programme design calls for an audit opinion as part of the report design, it is important to be as explicit as possible about the criteria for assigning each of the alternative opinions. Inconsistency between audit teams in assigning opinions could, for example, result in a facility’s appearing to have “lost ground” in successive audits when the only real difference was two different audit teams with two different interpretations of what “requires improvement” means. There is also an inherent concern of “grade inflation” over time. The criteria for the Arthur D. Little five-level opinion scheme are described in the table. Formulating an audit opinion is frequently the responsibility of the team leader and/or the audit programme manager, with input from individual team members. An audit opinion would not be given in instances where the team’s review was not sufficiently rigorous to substantiate conclusions about the facility’s compliance.
Post-Audit Activities – Preparation of Audit Reports
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 271 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Arthur D. Little’s Scheme
Opinion Criteria
On the basis of its review, the audit team believes that the environmental, health, and safety programmes and practices that were reviewed meet governmental and internal requirements.
The facility is in compliance with all (or virtually all) of the applicable requirements included in the audit scope. Isolated exceptions to a few requirements are noted, but are judged to be occasional, anomalous, and inconsequential in comparison to the level of compliance achieved.
On the basis of its review, the audit team believes that the environmental, health, and safety programmes and practices that were reviewed substantially meet governmental and internal requirements.
Audit results substantiate a high degree of compliance. Only a few requirements are not satisfied. These represent isolated weaknesses in implementation of an otherwise effective compliance programme.
On the basis of its review, the audit team believes that the environmental, health, and safety programmes and practices that were reviewed generally meet governmental and internal requirements, except as noted below.
Several exceptions to applicable requirements are noted. These exceptions are more than anomalies and reflect weaknesses in the design and/or implementation of certain aspects of compliance programmes.
On the basis of its review, the audit team believes that the environmental, health, and safety programmes and practices that were reviewed require improvement to meet governmental and internal requirements.
Several exceptions to applicable requirements are noted. Some of the exceptions reflect the absence of one or two required programmes, significant departures from a few established criteria, or lapses in programme implementation.
On the basis of its review, the audit team believes that the environmental, health, and safety programmes and practices that were reviewed require significant improvement to meet governmental and internal requirements.
Many exceptions to applicable requirements are noted. They included significant departures from various established criteria, the absence of several required programmes, or prolonged inattention to the resolution of previously identified compliance or liability issues.
Post-Audit Activities – Preparation of Audit Reports
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 272 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Strengths
Oral acknowledgment of the facility’s strengths frequently occurs during the team’s daily interactions with facility personnel and in the exit meeting. Increasingly, however, companies are recognising the value of giving credit to areas of real strength in the written report as well by explicitly acknowledging those areas in which the facility is performing particularly well, through design and implementation of an effective management system, which provides some context within which to evaluate the audit exceptions and observations. This acknowledgment is most commonly done in the executive summary section of the report. For example:
In its review of the facility’s programmes and practices, the team noted the following as areas of real strength: • The hazardous waste management programme is clearly understood by
staff throughout the facility and is documented to demonstrate commitment to compliance.
• The bloodborne pathogens programme is well developed, organised, and tracked.
To avoid confusion, it is important that both the auditors and facility personnel have a common understanding of what is meant by a “strength.” A strength need not necessarily be a unique, world class, or extraordinary practice. On the other hand, the mere fact that a facility is in compliance with a regulatory requirement is not an example of a strength.
Post-Audit Activities – Preparation of Audit Reports
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 273 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Executive Summary
The purpose of the executive summary is to provide a one- or two-page section at the beginning of the report that conveys the highlights of the audit to readers. An example of an executive summary is illustrated below.
A health, safety and environmental (HSE) audit was conducted at ABC Company’s XYZ facility on Month/Day/Year. The time period under review was January 1, 199X, through the last day of the audit. The purpose of the audit was to verify compliance with applicable federal, state, and local safety and health laws and regulations. The scope of the audit included air pollution control, water pollution control, solid and hazardous waste management, industrial hygiene, employee safety, and loss prevention. Specific deficiencies in the HSE programmes and practices are described in detail in the body of this report. On the basis of its review, the audit team believes that XYZ facility will need to develop and implement more formal HSE management programmes to achieve the HSE goals recently established by facility management. In the audit team’s opinion, the HSE programmes currently in place are heavily dependent upon the capability and good intention of facility personnel and are being implemented in the absence of clearly defined written procedures and plans. In addition, the level of familiarity with regulatory obligations varied among the staff responsible for HSE compliance activities. The management of XYZ facility in Anytown, Anystate, has initiated a programme to ascertain the status of HSE programmes on site, to develop appropriate corrective actions to remediate problems, and to develop overall systems to ensure compliance with governmental rules and regulations. In addition, the facility has undergone a recent management reorganisation creating a more centralised HSE group. Thus, these activities represent an increased commitment to manage compliance. To develop more formal HSE programmes suitable to the operations of XYZ facility, we recommend the following for consideration: • Developing clearly stated and well-defined descriptions of roles and responsibilities for all
HSE activities, and communicating those to facility staff. • Establishing well-developed plans and written procedures, as appropriate, for undertaking
compliance activities which are communicated and understood by all key HSE staff. • Undertaking a facility programme to periodically review and monitor HSE compliance and
to identify problems. • Developing an active, formal training and awareness programme for key staff in all areas
of HSE requirements. Lastly, we believe the facility should continue in its efforts to identify and assess site contamination and to develop appropriate remedial plans.
Post-Audit Activities – Preparation of Audit Reports
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 274 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Explanation of Requirements
In many instances, the readers of audit reports are not all equally familiar with the regulatory or company policy requirements to which the team has identified exceptions. In these situations, it may be helpful for the audit report to include a brief description of the regulation or company policy that is being cited as the basis for the finding. For ease of reading, it is useful to identify the requirement and the exception separately. For example:
Management of Satellite Accumulation Areas
Description of Requirement
Federal regulations allow for the accumulation of up to 55 gallons of hazardous waste in containers at or near the point of generation and under the control of the operator of the process generating the waste (commonly referred to as a “satellite accumulation area”), without a time limit, permit, or interim status, contingent upon the generator complying with specific container labelling and management conditions. [40 CFR 262.34(c) and 265.171-265.173]
Exception In inspecting a sample of areas where wastes are being accumulated, the team noted the following: • In the Rastex process, solvent-contaminated filters are
routinely changed and disposed of as hazardous waste. The “satellite” accumulation area for filters taken from the RASTEX 679 reactor is located across the manufacturing building at the DACSTAB UVX production area. In the team’s opinion, the DACSTAB UVX area is not “at or near” the RASTEX reactor, nor is it “under the control of the operator” of the RASTEX process.
• In several areas, the team observed containers that were used to collect solvent-contaminated rags. The containers were not marked with the words “Hazardous Waste” or with other words to describe the container contents and were not kept closed during times when waste was neither being added nor removed.
Post-Audit Activities – Audit Policy Statement
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 275 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
One of the most common concerns with audit reports and audit-related documentation is the risk of disclosure in an enforcement or litigation situation. Therefore, many organisations will establish or utilise one or several mechanisms to protect their audit results. Some of the various means that can be used to protect audit results include: • Physical limitations on distribution • Attorney-client privilege • Attorney work product • Audit privilege Physical Limitations on Distribution This protection is not founded in law, but simply requires the audit programme to exercise due care in the release and handling of all information relating to the audit. There are no specific requirements in utilising this protection other than following common sense and whatever systems or procedures are in place in individual organisations for maintaining confidentiality. Attorney-Client Privilege Attorney-client privilege protects the confidential communications between a “person” (who may, in fact, be a company) and his/her attorney. Legal protection under this privilege requires that: • The person asserting the privilege is a client of the attorney to
whom the information is entrusted. • The communication is made to the attorney or someone
working for the attorney (e.g., an audit team). • The attorney is engaged in preparing a factual investigation or
legal opinion for the client.
• The communication is kept confidential and the privilege is not breached voluntarily or inadvertently waived, i.e., information is not shared freely or recklessly.
Post-Audit Activities – Audit Policy Statement
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 276 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
In the context of an HSE audit, the corporation is the client and the audit results (including the report, the working papers, etc.) can be designated as the confidential communications between the client (the facility/ company) and the attorney (via the audit team, who is acting as the agent of the attorney). Attorney Work Product Another potential means of protection of audit results is the work product doctrine. The work product rule protects information/material prepared by an attorney in anticipation of litigation. This privilege, however, is not absolute, and can be vacated by a showing of “undue hardship and substantial need” on the part of the party seeking to discover the privileged information. Privilege under the work product rule is seldom utilised to protect audit results in the course of routine HSE audits. For one thing, most audits are not typically conducted in anticipation of litigation, but rather are used as a management tool to assure or measure compliance. Furthermore, the privilege under the work product rule rests with the attorney, unlike the attorney-client privilege, which rests with the client. Therefore, in the context of HSE audits, this type of privilege would typically require substantial involvement of counsel in the audit process.
Post-Audit Activities - Quality Assurance in the Audit Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 277
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Many audit programmes have quality assurance mechanisms such as: • Team leader reviews • Customer/facility feedback • Periodic programme reviews Team Leader Reviews
One important aspect of team leader reviews of auditor performance is working paper review, which was discussed earlier (see checklist in Tab 6). In addition, team leaders frequently evaluate and provide feedback to individual auditors regarding their technical and interpersonal skills as auditors. If a person is less than effective in one area, he/she can receive some training or other assistance in that area prior to or during the next audit. Many companies have formalised this process with questionnaires or forms for the team leader to complete. An example of a feedback form, which is based on the ISO 14012 qualification criteria for auditors, is provided in the table on the following page.
Post-Audit Activities - Quality Assurance in the Audit Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 278
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Auditor Name
Facility Audited
Audit Dates
Team Leader/Reviewer
Audit Element Score* Comments
1 2 3 4 5
Technical Knowledge • HSE science and
technology • Facility operations • Regulatory requirements • HSE management systems • Audit procedures and
techniques − Interviewing − Using the protocol − Keeping working papers − Writing findings
Personal Attributes • Clarity in oral
communication • Foreign language capability • Diplomacy, tact, and
listening skill • Independence and
objectivity • Personal organisation and
time management • Ability to reach sound
judgments based on objective evidence
*1 = poor; 3 = average; 5 = exceptional; N/A = not applicable
Post-Audit Activities - Quality Assurance in the Audit Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 279
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Customer/Facility Feedback
Upon completion of the field activities, it is quite common for the audit team to solicit feedback from the “auditees.” This may be done informally, for example by means of a debriefing conversation between the team leader and facility management after the closing meeting, or more formally. Companies that solicit such feedback have generally found it to be a valuable mechanism for improving the overall effectiveness and acceptance of the audit programme. Managers of audit programmes have also found, however, that it is important to keep the feedback form simple, so as not to overburden the facility. The table on the following page illustrates the feedback form utilised in one audit programme.
Post-Audit Activities - Quality Assurance in the Audit Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 280
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Auditee Feedback Form
Location
Date of Audit
Responder
Strongly Agree
Neither Agree or Disagree
Strongly Disagree
No Basis
1. The audit objectives were clearly communicated to me.
2. The audit took an acceptable amount of time (from entrance to exit).
3. The disruption of daily activities was minimised as much as possible during the audit.
4. My business concerns and perspective were adequately considered during the audit.
5. Communication of audit results and status to me during the audit was timely and adequate.
6. The audit team demonstrated technical proficiency in the audit areas.
7. The audit team demonstrated courtesy, professionalism, and a constructive and positive approach.
8. The audit team’s conclusions were logical and well documented.
9. Audit results were accurately reported and appropriate perspective was provided.
10. The audit report was clearly written and logically organised.
11. Overall the audit provided “value added” to my organisation.
Comments: _____________________________________________________________________________ _____________________________________________________________________________ _____________________________________________________________________________ _____________________________________________________________________________ Source: AlliedSignal Inc.
Post-Audit Activities - Quality Assurance in the Audit Process
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 281
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Periodic Programme Reviews
Audit programmes should keep pace with changes in the business and regulatory climate and cannot be static. Periodically, it pays to step back and take a hard look at the overall programme design and implementation. In these periodic reviews, all elements of the design—objectives, scope, coverage, organisation, resources, and approach—should be critically examined to confirm whether or not they are (still) meeting the needs of the programme’s stakeholders. It is also important to verify that audit teams in the field are actually implementing the programme as designed. There are a variety of mechanisms for conducting these periodic reviews. One possibility is to assemble a team or task force to examine the entire audit process and look for improvement opportunities. The team should probably include representatives of the audited facilities and of upper management, as well as some of the members of the audit programme itself. Another approach that has been used successfully in some companies is to ask the law department or the internal audit department to review the programme. Yet another possibility is to engage an independent third party from outside the company to confirm the adequacy of programme design and implementation versus self-established standards, industry and international standards, and best practices. Regardless of the composition of the reviewing body, a “high performance business” approach may prove useful. In such an approach, the review begins by identifying the key stakeholders of the audit programme and determining/confirming their needs. It is then possible to evaluate the various elements of the audit process to see if it is structured to meet those needs. Once any necessary improvements to the process have been identified, it is possible to assess whether the underlying organisation and resources are adequate for effective implementation.
Appendix A Confirmation Letter
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 282 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Appendix A Confirmation Letter
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 283 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
*[Date] *[Name] *[Title] *[Company] *[Address] *[City, State] Re: Environmental, Health, and Safety Compliance Audit Dear __________: Confirming our telephone conversation of *[Date], we will conduct an environmental, health, and safety compliance audit of the *[Facility name] Plant the week of *[Date]. The audit will address air pollution control, water pollution control, spill control and emergency response planning, solid and hazardous waste management, underground storage tanks, soil and groundwater contamination, drinking water management, PCB management, employee safety, loss prevention, and industrial hygiene as well as company policies, guidelines, etc. The audit team will arrive on site on *[Date]. They would like to meet with you and other appropriate personnel on Monday morning to briefly describe the audit and to answer any questions. At that opening conference, it would be helpful if the team could receive a brief description of the current organisation and operations at the facility as well as an orientation tour. Those facility personnel involved with environmental, health, and safety activities will be needed on site for discussions during the audit week. At the conclusion of the audit, the audit team will again meet with you and other appropriate personnel to discuss the team’s findings. This closing conference is a critical part of the audit, and it is very important that you be present.
Appendix A Confirmation Letter
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 284 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
*[Date] Page 2 *[Name] *[Title] *[Company] The audit team will make its own lodging and transportation arrangements, including cars for use during the week. A dedicated meeting room (work area) is needed at the facility with one large conference table or several smaller tables to accommodate the team members and their working papers. Also, access to a telephone and a copier would be helpful. To assist us in our preparation, we would appreciate having the facility complete the enclosed pre-audit questionnaire (Attachment 1), and provide a copy of as many as possible of the applicable items listed in Attachment 2. The pre-audit documents should be sent to *[The Team Leader] ’s attention by *[Date]. Please do not hesitate to call me if you have any questions about this audit or the Environmental, Health, and Safety Audit Programme in general. Thank you for your assistance in these matters. Sincerely, [Team Leader] Attachments cc:
Letter Attachment 1 Example of Pre-Audit Questionnaire
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 285 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 286 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
I General Information
1. Number of Employees on-site
2. Number of Contractors (Firms) on-
site
3. Number of Contractors’ employees
on-site (daily average)
4. Number of Employees in Company
5. Number of shifts worked
(please specify times)
6. Products
7. Annual production volume
8. Time company has operated at this
site
9. Nature of any previous industrial
activities on this site 1. 2. 3.
10. Name(s) of Doctors and
Occupational Nurses on-site and % time in attendance
1. 2. 3. 4.
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 287 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
II Locality/Neighbourhood Description
1. Nature of facility location e.g., commercial, industrial, residential, agricultural, rural
2. Are there any schools, hospitals,
nursing homes, prisons, churches or other public buildings within two kilometres of the facility?
3. Are there any nature reserves,
national parks or sites of specific scientific interest within two kilometres of the facility?
4. Distance to nearest industrial
neighbour from the fence line
5. Nature of neighbouring industrial
activities
6. Describe any major hazard
installations within two kilometres of the facility and distance
7. Distance to nearest residential
property
8. Approximate size of the population
near this facility (see table below)
0-1 kilometre 0-100 100-1000 +1000 1-2 kilometres 0-100 100-5000 +5000 2-5 kilometres 0-100 100-10,000 +10,000 9. Number of complaints per year
received from the local community related to HSE issues
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 288 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
10. Nature of any local community
activities in which the site participates
11. Distance to nearest surface water more than 1 kilometre (please tick) 0.5-1 kilometre adjacent to facility
boundary
within facility 12. Is the facility affected by any natural
hazards, e.g., earthquake, hurricane, flooding?
13. List any off-site facilities
(warehouses, processing units)
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 289 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
III Policy and Organisation
A Environmental Management System 1. Does the facility have any of its own
specific policies, procedures or guidelines pertaining to:
(a) Health and Safety (b) Environment
Yes No Yes No
List site-specific written HSE policies and procedures below or attach an index if preferred
2. Please provide an organisation chart
showing HSE management structure and responsibilities
3. Who is responsible for identifying
training needs?
4. Who is responsible for developing
and implementing training programmes?
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 290 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
5. List site specific training programmes/topics below or attach an index if preferred
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 291 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
IV Functional Areas
A Occupational Environment Who is responsible for occupational
environment programmes at the facility?
Name: Title:
1. Are risk assessments carried out to
evaluate: (a) Internal air quality
(b) Temperature
(c) Lighting levels
(d) Ergonomics
(e) Manual handling
(f) Exposure to hazardous
substances
Yes No N/A
Yes No N/A
Yes No N/A
Yes No N/A
Yes No N/A
Yes No N/A
2. Is there a preventative maintenance
programme covering Heating, Ventilation and Air Conditioning (HVAC) and lighting equipment?
Yes No N/A
3. Is there a formalised safety
inspection programme? (a) Who inspects? (b) How often?
Yes No N/A
4. Does the facility have procedures to
report and respond to occupational health hazards?
Yes No N/A
5. How many reports/complaints have
been made in the past two years?
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 292 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
B Technical Equipment and Machinery Protection Who is responsible for technical
equipment and machinery protection programmes at the facility?
Name: Title:
1. Are risk assessments carried out to
evaluate: (a) Moving parts
(b) Cutting/blades
(c) Rollers and pinch points
(d) Excessive temperatures
(e) High pressure
(f) Other (please specify)
Yes No N/A
Yes No N/A
Yes No N/A
Yes No N/A
Yes No N/A
2. Is there a preventative maintenance
programme covering maintenance of equipment and machinery, and associated protection?
Yes No N/A
3. Are equipment inspections
conducted to review the safe working condition(s) of the equipment/machinery?
Yes No N/A
4. Are all lifting appliances examined
and certified? Yes No N/A
5. Are all pressure vessels tested and
certified? Yes No N/A
6. Does the facility have procedures for
employees to report faults and for corrective action?
Yes No N/A
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 293 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
C Materials, Goods and Hazardous Substances Who is responsible for materials,
goods and hazardous substances?
1. Do you maintain an inventory of
hazardous materials used and stored on site?
Yes No N/A
2. Have you conducted risk
assessments on use/storage/transport of hazardous materials, goods and substances?
Yes No N/A
3. Have you a risk reduction
programme, which includes, for example, elimination, reduction of use or personal protective equipment?
Yes No N/A
4. Are Safety Material Data Sheets
giving physical, chemical and toxic properties, and environmental data kept on site for each hazardous material?
Yes No N/A
5. What are the separation distances of
hazardous material areas from the boundary fence and other buildings?
6. Does the facility control the use of
hazardous materials by contractors? Yes No N/A
7. Has the facility undertaken any
renovation or demolition activities in the last 24 months that involved the removal of asbestos?
Yes No N/A
8. Have you completed plans for the
phasing-out and replacement of all CFC’s?
Yes No N/A
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 294 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
11. Please complete the following table or supply copy of internal list: Main Hazardous Materials, Goods and Substances used and stored on-site Type of material¹ Use² Approximate
quantity used per year³
Maximum quantity stored on site
Type of storage4 Storage Area Description5
Spill6 containment type and capacity (m3)
e.g. Solvent Plant degreasing 1000 kg 10 kg AST Covered concrete pad
15m³ with retention basin
Notes: 1 Indicate major hazardous materials used in operations or activities, and include potentially hazardous waste materials 2 Describe use of hazardous material listed, for example, boiler/furnace fuel, degreasing/metal cleaning chemical, wastewater treatment
chemical, etc. 3 Indicate use per year in kilograms or litres 4 Please describe the type of container(s) the material is stored in using one of the following symbols : AST: Aboveground non-buried storage tank D200: 200 Litre drums UST: Buried underground storage tank D25: Approx. 25 litre or smaller O: Other (please specify) e.g., IBC/containers/piles 5 Briefly describe storage area (for example, inside, outside, covered, fenced, locked, restricted access, fireproofed etc.) 6 Indicate type of spill containment provided, if any (for example, retention basin, collection sump, oil/water separator, paved/sealed area,
concrete pad, curbing, bunded area, etc,) and indicate by 'yes' or 'no' if the containment will retain firewater in the event of an emergency.
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 295 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
D New or Modified Processes Who is responsible for new or
modified processes at the facility? Name: Title:
1. Does the facility have HSE
procedures for change management?
Yes No N/A
2. Have any major new processes
been introduced/installed over the past two years? e.g. new production lines, closing of production lines
If so, what were they?
Yes No N/A
3. Have any major modifications been
made to processes over the past two years?
e.g. machinery replacements If so, what were they?
Yes No N/A
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 296 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
E Hazardous Work Who is responsible for hazardous
work programmes at the facility? Name: Title:
1. Has the facility conducted formal risk
assessments of hazardous work activities?
(a) Hot work
(b) Confined spaces
(c) Energy isolation work (d) Working at height
(e) Other (please specify)
Yes No N/A
Yes No N/A
Yes No N/A
Yes No N/A
2. Has the facility established
documented procedures for hazardous activities, e.g.
(a) Permit-to-Work/Approval procedures
(b) Pre-use/activity inspections
(c) Issue of personal locks
(d) ‘Buddy’ system ( for lone working)
(e) Engineering controls, e.g guarding
(f) Provision and use of personal protective equipment
(g) Other (please specify)
Yes No N/A
Yes No N/A
Yes No N/A
Yes No N/A
Yes No N/A
Yes No N/A
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 297 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
3. Are Work Permits/Approvals issued
for: (a) Hot work
(b) Confined spaces/Entry work
(c) Electrical work
(d) Working at heights
(e) Contractor Control
(f) Other (please specify)
Yes No N/A
Yes No N/A
Yes No N/A
Yes No N/A
Yes No N/A
4. Is specific training provided or
qualifications required for employees involved in hazardous activities?
(Please ensure training topics are included in your answer to section III, question 5.)
Yes No N/A
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 298 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
F Electrical Safety Who is responsible for electrical
safety programmes at the facility? Name: Title:
1. Has the facility conducted risk
assessments for work on electrically energised equipment and electric circuits?
Yes No N/A
2. What preventative and protective
safety measures are in place, e.g. earthing, fuses and circuit breakers?
3. Is a lock-out/tag-out programme in
place at the facility for electrical and other energy isolation?
Yes No N/A
4. Are regular inspections conducted to
review the safe working condition(s) of the electrical equipment?
Yes No N/A
5. Is specific training provided
to/qualifications required for employees involved in electrical work?
(Please ensure training topics are included in your answer to section III, question 5.)
Yes No N/A
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 299 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
G Work at Height Who is responsible for working at
height programmes at the facility? Name: Title:
1. Does the facility have personnel or
contractors working at heights? (a) Routinely, as part of their work
activities, e.g. Facilities engineer (b) Non-routinely
Yes No N/A Yes No N/A
2. Has the facility conducted risk
assessments on working at height activities?
Yes No N/A
3. What controls are in place to reduce
risks, e.g. (a) Permits-to-Work
(b) Harnesses
(c) Personal protective equipment
(d) Buddy system
(e) Others (Please specify)
Yes No N/A
Yes No N/A
Yes No N/A
Yes No N/A
4. Does the facility have an inspection
and maintenance programme for ladders and scaffolding?
Does this include pre-use
inspections, tagging and approvals?
Yes No N/A Yes No N/A
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 300 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
H Noise Control Who is responsible for noise control
programmes at the facility? Name: Title:
1. What are the main sources of noise
at the facility?
Source
Internal External
e.g. compressors Yes Can be heard outside factory building
e.g. transport movements No Yes
2. Are there any regulatory limits for
noise at the facility? (a) environmental (b) occupational
Yes No N/A Yes No N/A
3. In the last three years, how many
times None
has the facility received complaints from
Less than 5
different neighbours relating to noise?
5 - 10
(please tick) More than 10
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 301 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
4. Does the facility operate its own
freight transport? Yes No N/A
5. Are there transport movements
before 06.00 and after 22.00 hours? Yes No N/A
6. Are there any loading or unloading
operations before 06.00 and after 22.00 hours?
Yes No N/A
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 302 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
I Personal Protective Equipment (PPE) Who is responsible for PPE
programmes at the facility? Name: Title:
1. For which activities is PPE required?
2. What types of PPE are provided?
3. Does the facility have an approved
list of PPE and/or an approved list of suppliers?
Yes No N/A
4. Which PPE does the inspection and
maintenance activities cover?
5. Is there a formalised
inspection/maintenance programme?
(a) Who inspects? (b) How often?
Yes No N/A
6. Is training provided to employees
involved requiring PPE? (Please ensure training topics are
included in your answer to section III, question 5.)
Yes No N/A
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 303 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
J Vehicles and Driving Who is responsible for vehicles and
driving programmes at the facility? Name: Title:
1. Does the facility use (please indicate
approximate numbers): (a) Forklift trucks
(b) Industrial plant (e.g. diggers,
cranes)
(c) Heavy Goods Vehicles (HGV)
(d) Company cars
(e) Other (please specify)
Owned? Leased? Contractors’?
2. Are there procedures for meeting
designated vehicle specifications? Yes No N/A
3. Is there a formalised
inspection/maintenance programme?
(a) Who inspects? (b) How often?
Yes No N/A
4. Is there a pre-use inspection
procedure for vehicles (other than company cars)?
Yes No N/A
5. Is training provided to employees
driving company vehicles? (Please ensure training topics are
included in your answer to section, III, question 5.)
Yes No N/A
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 304 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
K First Aid, Medical Examinations and Care Who is responsible for first aid,
medical examinations and care programmes at the facility?
Name: Title:
1. Does the facility have an equipped
medical centre/clinic? If yes, is this staffed: (a) Doctor
(b) Nurse
(c) First Aiders
Yes No N/A 24 hrs Full-time Part-time
24 hrs Full-time Part-time
24 hrs Full-time Part-time
2. How many formally qualified First
Aid providers does the facility have?
3. Where does the facility provide first
aid equipment? (a) Medical Centre
(b) Designated locations
(c) Per First Aider
(d) Security/Gate keepers
(e) Other (please specify)
Yes No N/A
Yes No N/A
Yes No N/A
Yes No N/A
4. Who inspects and maintains first aid
equipment regularly and how often?
Please attach a summary of accidents/incidents over the past 12 months (if easily available).
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 305 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
5. Are medical examinations provided
by the facility? (a) Pre-employment
(b) Annual check-ups
(c) Critical situations, e.g.
– exposure to hazardous substances
– pregnancy – return to work after
injury/illness – persons under 18/over 60 – terminal diseases – other (please specify)
Yes No N/A
Yes No N/A
Yes No N/A Yes No N/A Yes No N/A Yes No N/A Yes No N/A
6. Which general welfare and wellness
programmes are provided for employees?
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 306 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
L Fire Protection and Control Who is responsible for fire protection
and control programmes at the facility?
Name: Title:
1. Is this facility regulated in relation to
flammable/explosive substances and/or for fire protection?
If so, what
certificates/permits/licences does the facility hold?
Yes No N/A
2. Does the facility have any areas
designated as explosion hazard zones?
Yes No N/A
3. Has the facility conducted a risk
assessment of ignition, fire and explosion sources? e.g. through insurance inspections
Yes No N/A
4. Has the facility installed fire
protection equipment: (a) Fire detection (heat or smoke)?
(b) Alarms?
(c) Sprinklers or other dousing
systems?
(d) Other (please specify)?
Yes No N/A
Yes No N/A
Yes No N/A
5. Does the facility have an emergency
and evacuation plan? Yes No N/A
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 307 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
6. Does the facility provide instruction,
training, and testing of emergency plans?
Please specify
Yes No N/A
7. When was the last drill carried out?
Date:
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 308 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
L Fire Protection and Control (continued) 8. Does the facility have a trained
emergency response team or nominated fire marshals?
Please specify
Yes No N/A
9. Has the facility experienced any
explosions or fires over the past: (a) ten years
(b) five years
(c) two years
Yes No N/A
Yes No N/A
Yes No N/A
10. Has the facility experienced any
near-misses involving explosions or fires over the past:
(a) ten years
(b) five years
(c) two years
Yes No N/A
Yes No N/A
Yes No N/A
11. Of these, how many have been
reportable to a government agency
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 309 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
M Air Pollution Control Who is responsible for air pollution
control programmes at the facility? Name: Title:
1. Has the facility identified and
documented all sources of air emissions?
Yes No N/A
2. Has the facility identified the nature
and quantities of pollutants emitted to atmosphere?
Yes No N/A
3. Which air emissions from the facility
are regulated by authorities?
4. Which air pollutants emitted from the
facility are required to be monitored by a government agency? If none, state ‘none’
5. Which types of air pollution control
equipment are installed, e.g. scrubbers, dust filters?
6. Does the plant operate planned
maintenance procedures for its air pollution control equipment?
Yes No N/A
7. State average number of public
complaints for last three years attributed to air emissions from the facility, e.g. odour, dust, VOCs, smoke
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 310 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
8. How many cooling towers or static
water tanks are located in the facility?
9. Are 6-monthly chlorinations carried
out on these to prevent growth of legionella pneumophilia?
Yes No N/A
10. Does the facility dispense fuel to
motor vehicles? Yes No N/A
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 311 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
N Water Pollution Prevention Who is responsible for water
pollution prevention programmes at the facility?
Name: Title:
1. Does the facility have any targets or
programmes for reducing water consumption?
Please specify
Yes No N/A
2. Are waste water discharges
regulated by a permit or consent? Yes No N/A
3. Does the facility conduct any effluent
monitoring? Yes No N/A
4. Does the facility make use of an on-
site wastewater treatment system prior to effluent discharge?
Yes No N/A
5. If the facility has own wastewater
treatment plant, how is the sludge disposed of
6. Is any process wastewater recycled? Yes No N/A
7. Does any portion of the facility’s
drinking water supply come from on-site wells or surface water sources?
Yes No N/A
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 312 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
O Waste Management Who is responsible for waste
management programmes at the facility?
Name: Title:
1. Does the facility generate wastes
that are defined as ‘hazardous’ or ‘special’ under government regulations? Please provide details in the Table below (question 6).
Yes No N/A
2. Does the facility require/have a
permit or license for its waste activities?
Yes No N/A
3. Does waste treatment and/or
disposal take place/ever taken place on-site?
Please specify
Yes No N/A
4. Are any waste materials separated
and sent for recycling? Please provide details in the Table below (question 6).
Yes No N/A
5. Does the facility monitor off-site
disposal activities? Yes No N/A
Letter Attachment 1 Example of Pre-Audit Questionnaire
__________________________________________________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 313 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
6. Please complete the following table. Solid and Hazardous Waste Generation, Treatment and Disposal
Description of Waste
Generation
Process1 Classification e.g. hazardous,
liquid, inert
Estimate of Annual Quantity Generated (tons or Kg/year) (if available)
Method of Treatment and final Disposal2
Waste minimisation target and date (if
any) e.g. cardboard packaging
raw materials/ supplies
inert 6 tons Re/L 20% reduction by 1999
Notes : 1 Indicate which type of equipment or operation generates this waste stream. 2 Please enter one of the following letters as appropriate. If the disposal or treatment is on-site, please circle the letter.
Re = Recycled externally S = Sold for further use (please specify) T = chemical or physical Treatment
I = Incinerated
Ri = Recycled Internally L = Landfill O = Other (please specify
Letter Attachment 1 Example of Pre-Audit Questionnaire __________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 314 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
P Soil and Groundwater Protection Who is responsible for soil and
groundwater protection programmes at the facility?
Name: Title:
1. Have you conducted a survey to
identify actual areas of soil and groundwater contamination resulting from previous industrial practices or activities?
Yes No N/A
2. Has there been any remediation of
land which is contaminated? Yes No N/A
3. Does the facility have any
underground (buried) tanks in or out of service with associated piping? (See also 7)
Yes No N/A
4. Does the site have procedures to
protect soil and groundwater (e.g. spill prevention and containment programmes/facilities) and a written spill reporting procedure?
Yes No N/A
5. Does the stormwater drainage
system include any interceptors? Yes No N/A
6. Could excessive stormwater cause
secondary containment, interceptors etc. to flood and overflow?
Yes No N/A
Letter Attachment 1 Example of Pre-Audit Questionnaire __________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 315 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
7. Please complete the following table. Bulk Storage Tanks Tank Volume
Above ground AST or Buried UST
In/Out of service
Material stored (or previously stored)
Leak detection and/or release prevention
Tank age (years)
Leak test
e.g 50m³ UST In fuel oil none 35 none
Notes: 1 Indicate volume of tank in litres (I) or cubic metres (m³) 2 Indicate whether tank is above ground - use letters AST, or below ground - use
letters UST 3 Indicate whether the tank is equipped with leak detection measures, overfill
protection, or corrosion protection 4 Indicate date of most recent leak test (including hydrostatic pressure testing,
sampling and analysis of surrounding soil, etc.). If not tested, indicate ‘None’.
Letter Attachment 1 Example of Pre-Audit Questionnaire __________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 316 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Q Product Stewardship Who is responsible for product
stewardship programmes at the facility?
Name: Title:
1. Has the facility determined the
environmental impacts of all its product and packaging materials, including transport packaging materials?
Yes No N/A
2. Are environmental impacts formally
considered during development and marketing of products?
Yes No N/A
3. Do raw material specifications
routinely include HSE requirements? Yes No N/A
4. Do manufacturing machinery
specifications routinely include HSE requirements?
Yes No N/A
Letter Attachment 1 Example of Pre-Audit Questionnaire __________________________________________________________________
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 317 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
R Energy Conservation Who is responsible for energy
conservation programmes at the facility?
Name: Title:
1. Has the facility carried out a
systematic review of all its uses of energy, to identify major energy consuming equipment or activities?
Yes No N/A
2. Have energy conservation objectives
and targets been set? Please specify
Yes No N/A
3. Has the facility defined Energy
Accountable Centres (i.e., discrete operating units within the site for which energy consumption figures are available)?
Yes No N/A
Letter Attachment 2 Pre-Audit Information Request
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 318 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• Facility plot plan or map • Directions to the facility • Visitor safety requirements (e.g., personal protective
equipment, orientation, specialised training, special clearances, etc.)
• Completed pre-audit questionnaire • Description of the facility’s operations/processes • Facility organisation chart, showing HSE responsibilities • Local laws, regulations, and ordinances related to the scope
of the audit • List of current environmental licenses, certificates, and
authorisations • Copies of permits for wastewater discharges and example air
emission permits • Recent regulatory agency inspection/enforcement
correspondence • Recent internal and intra-company environmental, health, and
safety audit reports • Table of contents for facility-specific environmental, health,
and safety policies and procedures
Appendix B Roles and Responsibilities of the Audit Team
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 319 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Appendix B Roles and Responsibilities of the Audit Team - Audit Team Leader Responsibilities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 320 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities
• Select team members and assign audit responsibilities. • Gather and distribute background information. • Identify applicable federal/national, state/provincial, and local
regulations and company policies and procedures. • Conduct advance visit to the facility (if necessary). • Review and revise audit strategy and assigned duties as
necessary. • Determine and confirm arrangements with the team members
and the facility: − Travel arrangements − Hotel/travel reservations
• Prepare items for audit (forms, supplies, protocols). • Coordinate pre-audit team meeting(s). On-Site Activities
• Lead opening meeting presentation. • Serve as liaison between team and facility personnel to
ensure that all team members are appropriately scheduled to meet with facility personnel.
• Solicit feedback from each team member on the status of
work accomplished throughout the audit. • Perform audit duties as determined by the audit strategy.
Appendix B Roles and Responsibilities of the Audit Team - Audit Team Leader Responsibilities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 321 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• Review assigned protocol steps with each auditor to ensure that all steps are covered appropriately.
• Document the rationale for changing the scope of the audit (if
necessary). • Understand the context for and meaning of each finding
reported by the team. • Provide periodic feedback to facility personnel on the status of
the audit and the findings of the team. • Prepare the exit meeting discussion sheets listing all findings
as summarised by the team. • Ensure that all exit meeting discussion sheets are reviewed by
each team member. • Review all findings with key facility personnel prior to the exit
meeting to ensure the accuracy of all findings. • Lead presentation of exit meeting discussion. • Summarise reporting schedule and format. Post-Audit Activities
• Review all working papers to ensure that all topics were covered and that all findings are corroborated by working paper notes.
• Prepare draft report. • Distribute draft report for comments. • Incorporate comments where appropriate into the final report.
Appendix B Roles and Responsibilities of the Audit Team - Audit Team Member Responsibilities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 322 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities
• Make travel arrangements (if required). • Attend pre-audit team meeting (if required). • Prepare for the audit by reviewing appropriate
federal/national, state/provincial, and local regulations, company policies and procedures, and available background information.
• Modify or annotate the protocol to reflect facility-specific
requirements, state and local regulations, and information gained during review of background information.
On-Site Activities
• Perform duties assigned by the team leader during the audit. • Serve as a resource for other audit team members during the
audit. • Report on progress to the team leader throughout the audit,
including any problems encountered. • Share observations/concerns with other team members during
the audit to ensure that each is addressed appropriately. • Keep facility personnel apprised of findings as they are noted. • Summarise all findings and report them to the team leader
before the close-out meeting. • Assist with preparing the exit meeting discussion sheets.
Appendix B Roles and Responsibilities of the Audit Team - Audit Team Member Responsibilities
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 323 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
• Ensure that all findings noted in your working papers are presented on the exit meeting discussion sheets and accurately reflect the facts as you understand them.
• Contribute during the exit meeting when questions are raised
about your findings. Post-Audit Activities
• Review draft audit reports for: − Wording changes − Suggested changes in placement of findings within the
report • Provide input as necessary when findings in the draft report
are challenged.
Appendix C Guide to Some Health, Safety and Environmental Acronyms
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 324 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Appendix C Guide to Some Health, Safety and Environmental Acronyms
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 325 Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
AAQS Ambient Air Quality Standards NSR New Source Review ACGIH American Conference of Governmental Industrial
Hygienists OCAW Oil, Chemical, and Atomic Workers
AHERA Asbestos Hazard Emergency Response Act of 1986 (Title II of TSCA)
OEL Occupational Exposure Limit
ANSI American National Standards Institute OSHA Occupational Safety and Health Administration ARAR Applicable or Relevant and Appropriate
Requirements OTA Office of Technology Assessment
AQCR Air Quality Control Region PEL Permissible Exposure Limit (for workplace) ASHRAE American Society of Heating, Refrigeration, and Air
Conditioning Engineers PCBs Polychlorinated Biphenyls
BACT Best Available Control Technology pH Measure of acidity or alkalinity BOD Biochemical Oxygen Demand PHA Process Hazard Analysis BPCT Best Practicable Control Technology P&ID Piping and Instrumentation Design BPT Best Practical Treatment PM Particulate Matter (in air) BTU British Thermal Unit PM10 PM with <10 micron diameter; respirable CAA Clean Air Act PMN Premanufacture Notification (under TSCA) CAAA Clean Air Act Amendments POTW Publicly Owned Treatment Works CAS Chemical Abstracts Service Registration Number PRPs Potentially Responsible Parties (under CERCLA CAG EPA’s Carcinogen Assessment Group PSD Prevention of Significant Deterioration CAIR Comprehensive Assessment Information Rule (under
TSCA) PSM Process Safety Management
CERCLA Comprehensive Environmental Response, Compensation, and Liability Act (The Superfund Law)
RACT Reasonably Available Control Technology
CFCs Chlorofluorocarbons RCRA Resource Conservation and Recovery Act CFR Code of Federal Regulations RQ Reportable Quantity (for spill reporting) CMA Chemical Manufacturers Association RTECS Registry of Toxic Effects of Chemical Substances CO Carbon Monoxide SARA Superfund Amendments and Reauthorisation Act of
1987 COD Chemical Oxygen Demand SDWA Safe Drinking Water Act CPSC Consumer Product Safety Commission SERC State Emergency Response Commission CWA Clean Water Act SIC Standard Industrial Classification DMR Discharge Monitoring Report SIP State Implementation Plan DOT Department of Transportation SNUR Significant New Use Rule EPA Environmental Protection Agency SPCC Plan Spill Prevention Control and Countermeasures Plan EPCRA Emergency Planning and Community Right-to-Know
Act (Title III of SARA, commonly called Right-to-Know or SARA Title III)
SQG Small Quantity Generator (of hazardous waste)
FIFRA Federal Insecticide, Fungicide, and Rodenticide Act STS Standard Threshold Shift GEP Good Engineering Practice SWDA Solid Waste Disposal Act HAZOP Hazard and Operability Study TCLP Toxicity Characteristic Leaching Procedure (under
RCRA) Hazwoper Hazardous Waste Operations and Emergency
Response Title III Emergency Planning and Community Right-to-Know
Act HCS OSHA Hazard Communication Standard (Worker
Right-to-Know) TLV Threshold Limit Values (for workplace exposure)
HMTA Hazardous Materials Transportation Act TPQ Threshold Planning Quantity (for emergency planning HSWA Hazardous and Solid Waste Amendments (1984
RCRA Amendments) TSCA Toxic Substances Control Act
IARC Internal Agency for Research on Cancer TSDF Treatment, Storage, and Disposal Facility IDLH Immediate Danger of Life and Heath TSP Total Suspended Particulate (in air) IH Industrial Hygienist TSS Total Suspended Solids (in water) LAER Lowest Achievable Emission Rate UST Underground Storage Tank LEPC Local Emergency Planning Committee VHAP Volatile Hazardous Air Pollutant LOTO Lockout/Tagout (Control of Hazardous Energy) VOC Volatile Organic Compound MACT Maximum Available Control Technology WWTP Wastewater Treatment Plant MSHA Mine Safety and Health Administration Z List OSHA list of hazardous chemicals (29 CFR 1910
Subpart Z, Worker Right-to-Know) MSDS Material Safety Data Sheet NAAQS National Ambient Air Quality Standards NEPA National Environmental Policy Act NESHAP National Emission Standards for Hazardous Air
Pollutants
NIOSH National Institute for Occupational Safety and Health NOAA National Oceanic and Atmospheric Administration NOx A mixture of nitrous oxide and nitrogen dioxide NRC National Response Center NPDES National Pollutant Discharge Elimination System
NSPS New Source Performance Standards