2014 Life Science Learning Series – Overview Life Science Learning Series – Overview . ... MHRA,...

UL and the UL logo are trademarks of UL LLC © 2013

2014 Life Science Learning Series – Overview

UL EduNeering Highlights

• Partnership renewal/expansions

• Global Expansion • India

• Japan

• China

• New release

• New Content

• Expanded Advisory Services

Relationship with FDA is Extended and Expanded Partnership began in 1999 with Cooperative Research & Development Agreement (CRADA) with FDA’s Office of Regulatory Affairs ___________________________________________________________

Our solution (ComplianceWire) powers ORA’s online university (ORA-U) ___________________________________________________________

Since 1999, ORA-U used by 36,000+ federal, state, local (and now global) investigators for training ___________________________________________________________

120 online courses developed with FDA ___________________________________________________________

CRADA renewed and expanded through 2019. Focus on global investigations

import challenges use of mobile technology

Roadmap Themes

User-Centric Platform

Advanced Reporting & Quality Metrics

Globalization

Mobility

ComplianceWire Scheduled 10/25/14 Release

• New User Interface - New Login Page - To-Do List - New User Dashboards - New Curricula View - New User Profile

5

New Branding Options for 10/25/14 Release

6

Address Global Market Access Issues Focus on Specific GMP and Regulatory Content Needs Meet Specific 3rd Party Compliance & Certification Needs

Courses to be Released in 2014: • Overview of JPAL (Japan) • Overview of CFDA (China) • Overview of MDFS (Korea) • GMP courses translated into Japanese,

Chinese and Korean

Courses Released in 2014: • Combo Products • Supply Chain • ISO 14971

Courses Released in 2014: • Patient Safety Goals (HCIR) • Third Party Distributor Compliance • EU IT Validation (Q4)

Our 2014 Standard Course Strategy

And more mobile courses!

Education Along the Product LifeCycle

Authoritative Regulatory Content ______________________________________________________________

standard library courses: • ISO 14971 Risk Management Learning

Program • Regulatory and Compliance Essentials • Lean Six Sigma Training Program • Workplace Health and Safety • 60601/61010 and 60950 Training Program • FDA Inspections & Enforcement • Quality and Manufacturing Practices • Good Clinical Practices • Health Care Compliance/Sales & Marketing • Environmental, Health & Safety • Ethics, Code of Conduct and Privacy • HR Compliance & Risk Management • Leadership & Development - CrossKnowledge

ISO 14971 Risk Management Solutions Optional Solutions: 1) The public course – ISO 14971 for Medical

Device (out in July)

2) Public Instructor Lead Course on ISO 14971

3) Custom Variations to the Standard Course: We build pre-instructor lead training use cases to be leveraged as part of an on-site risk management training session.

4) Train the Trainer Training – We offer the option to train the trainer by selecting a few individuals as “super learners”. We individualize the training, provide all the training resources. Provide a seminar to train the super users and walk them through the training material to prepare them on how to educate the rest of the organization

12

Expanded Advisory Services

• Learning Management Technology Best Practices

• Outsourced support for training matrix creation, validation, system administration, etc.

• FDA Inspection Readiness Training • Mock FDA training audits and remediation

services • Audit Services: Equipment, Validation, etc. • Validation support for ComplianceWire and other

21 CFR Part 11 systems • SOP Training & Assessment Development • Global regulatory consulting and remediation • Custom, blended learning, educational programs


2013/2014 UL EduNeering ComplianceWire Client Benchmarking Study: Quality Metrics Being Used to Drive Continuous Improvement

Agenda

About our benchmarking study

Top learning priorities

• Quality-based learning maturity model

Regulatory activity

Our recommendations

UL customer (aggregated) statistics

UL Practice Leaders

New Courses New Programs

Influence Product Strategy Thought Leadership

UL Experts

Industry Experts Regulatory

Agency Agendas

Client Feedback Practice Leaders

About Our Study

• Sixth annual study

• Life science quality/compliance trends based on: • Customer survey: 85 respondents from >50

companies

• Analysis of how our 250 customers use our solution

• Our quarterly partnerships discussions with US FDA

• Market analysis and global regulatory trends

2013 Trends

• Many companies have advanced along a quality-focused learning maturity model, in which the emphasis on “check the box” compliance is shifting to quality and performance metrics;

• The most-often cited learning priority is now “Measuring Training Effectiveness,” which provides an opportunity for training organizations to demonstrate relevance to their business partners;

• The impact of global regulators has resulted in increased numbers of inspections, the drive to improve audit readiness, and the need for consistent, repeatable and scalable training approaches

Top Learning Priorities for 2014

Priorities 2014 Priority 2013 Priority

Measuring training effectiveness 67% 56%

Collecting training data to support the organization's quality or compliance metrics

66% 59%

Responding to inspections from global regulatory agencies 51% 46%

Improving SOP and policy management 46% 52%

Adding risk-based approaches to training programs 46% 26%

Do the Priority Trends Tell Us Anything?

Learning Priorities 2012 Actual Rank

2013 Actual Rank

2014 Plan Rank

Measuring training effectiveness 7 3 1


1 1 2

Responding to inspections from global regulatory agencies 4 4 3

Improving SOP and policy management 2 3 4

Our Takeaway: Global company training programs are advancing on the learning maturity model

From Compliance to Performance

Stage 1 Goals: • Addressing Basic

Compliance Issues

• Moving from Paper to Electronic Records


Stage 2 Goals: • Align SOPs to Role-

Based Curricula

• Build Qualification Curricula for high-risk job functions


Stage 3 Goals: • Measure training impact

and identify knowledge gaps

• Add OJT assessments to high-risk operations


Stage 4 Goals: • Measure impact of training

on continuous improvement metrics

• Demonstrate value of programs on performance and behavior


Stage 5 Goal: • Leverage data to

demonstrate value of quality management to business growth

Global Regulatory Authorities are Driving that Maturity Model

13

• Since 2009, the FDA has made enforcement and compliance a top priority

• Since that time, FDA has hired 25% more field inspectors

• And increased their focus on Non-US inspections and supply chain

• FDA , MHRA, EMA etc are doing joint inspections and sharing information

• CDER’s Quality Metrics and CDRH Case for Quality programs

• Shifting emphasis from compliance (check the box) to “quality culture”

Inspections on the Rise

Audit Body 2011 2012 2013

US FDA 65% 57% 77%

EMA 9% 22%

ISO 30% 31% 37%

OSHA 17% 19% 24%

As They Inspect, FDA is Finding More…

15

Training-Related Audit Observations in 2013

“Did you conduct training to support these Critical-to-Quality (CTQ) initiatives?”

CTQ % of Respondents

CAPA Initiations 60%

Audit/Inspection Findings 58%

Adverse Events 44%

Customer Complaints 41%

How Do You Measure Training Effectiveness?

Method Used % of Respondents

Regular Review of Quality Metrics 61%

Quizzes (via Quiz Creator tool) 45%

No Formal Measurement System in Place 23%

Documenting On-site Observations (via Forms tool) 13%

ComplianceWire Activity in 2013

Completion Types Completions in 2013

Main Use

CICS (Control Documents) 12,991,734 SOP Management

CICS with Quiz Creator 3,855,062 Assessments

Offline Events (ILC) 2,997,738 In-person Training

Computer Based Training (CBT) 753,464 Regulatory Knowledge

Forms and Exams 684,584 On-the-job Training

27 Million Completions in 2013 (18% increase over 2012)

Top 10 Pharmaceutical Courses in 2013

Rank Course Title

1 Handling an FDA Inspection

2 Introduction to GMPs

3 Principles of Good Documentation

4 Change Control

5 Orientation to GMP Compliance

6 GMP Principles of SOPs

7 GxPs

8 Part 11: Electronic Records; Electronic Signatures

9 DEA Compliance

10 Operating Room Conduct

Top 10 Medical Device GMP Courses in 2013

Rank Course Title

1 Orientation to GMP Compliance

2 Understanding GMPs for Facilities and Equipment

3 Meeting GMP Training Requirements

4 Principles of Auditing

5 Principles of Good Documentation

6 Handling an FDA Inspection

7 Introduction to the Quality System Regulation (QSR)

8 Part 11: Electronic Records; Electronic Signatures

9 Design Control Regulations for Medical Device Manufacturers

10 Quality Systems Inspection Technique (QSIT)

Top 10 Corporate Compliance Courses

Rank Course Title

1 Global Anti-Bribery

2 AdvaMed Code of Ethics

3 PhRMA Code

4 FCPA: Doing the Right Thing: Anti-Bribery

5 HIPAA: General Awareness

6 Diversity in the Workplace

7 Violence in the Workplace

8 Substance Abuse

9 Code of Business Conduct

10 E-mail and Corporate Communications

Top 10 EH&S Courses

Rank Course Title

1 Hazard Communication

2 Bloodborne Pathogens -- General Industry

3 Slips, Trips, and Falls

4 Personal Protective Equipment

5 Fire Extinguishers

6 Electrical Safety

7 Hearing Conservation

8 Ergonomics: Body Mechanics and Fitness

9 Access to Employee Exposure and Medical Records

10 Lockout/Tagout -- Affected

Thank You Questions?


The Case for Quality

What is the Case for Quality

2

• The FDAs movement toward assessing performance rather than just compliance.

• This is achieved by continuous improvement

through objective quality metrics • Goal: Increase product quality, reduce inspections,

steady the drug supply.

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&docid=naisoYaXEVCnzM&tbnid=7Bo1p3hdVPX0FM:&ved=0CAcQjRw&url=http://mpl-corp.com/quality/&ei=P9kXVMruLMSLjAKgooGYBg&bvm=bv.75097201,d.cGE&psig=AFQjCNHwy-P5Qq4zVqTvWEX8aoyfn51g4w&ust=1410935455224112

Why the Case for Quality

3

Food and Drug Administration Safety and Innovation Act (FDASIA) (July 9 2012) • Authority to collect user fees from industry to

fund reviews of innovator drugs, medical devices, generic drugs and biosimilar biological products

• Promote innovation to speed patient access to safe and effective products;

• Increase stakeholder involvement in FDA processes

• Enhance the safety of the drug supply chain /reduce drug shortages.

Source: http://www.fda.gov/RegulatoryInformation/Legislation/FederalFoodDrugandCosmeticActFDCAct/SignificantAmendmentstotheFDCAct/FDASIA/

Aspects of the Strategic Plan:

4

Reduce Drug Shortages by: • Collaborate with Industry to identify objective

quality metrics to be used for continuous improvement and to increase a culture of quality.

• Reduce enforcement actions • Generate Risk Based Inspection Schedule

• Increase Foreign Inspections

Benefits of Metrics

• Metrics eliminate subjectivity

• Key to risk detection (trending)

• Provide a benchmark for improvement

• Provide visibility into continuous improvement

“Employees can’t be held accountable if they don’t have the knowledge.” QA Executive, 2013 PDA Quality Metrics Conference

Metrics Identified by Industry

Site metrics that received the most votes at PDA Quality Metrics Conference (300+ QA Executives) in December 2013: • Confirmed OOS Rate • CAPA Effectiveness • Batch Failure Rate • Critical Investigation Rate • Environmental Monitoring Grade

A+B Areas Excursion Rate

ISPE Quality Metrics Recommendations Made in December 2013: • Confirmed OOS Rate • Unconfirmed OOS rate • Batch Rejection Rate • Rework and Reprocessing rate • Confirmed “Critical” Complaints

Rate • % Annual Product Quality

Reviews Completed on Time

“Culture in its most simplified definition is how people are incentivized to behave and the way people think, talk, work, and act every day.” Source: Richard J. Zarbo, MD, DMD, American Journal of Clinical Pathology, 2012

Metrics Provided By Industry

Error rates as identified by non‐conformances or deviations. A higher error rate could be evidence of: • poor process control • poor training of operators or oversight by supervisors • poorly maintained equipment and facilities • poor product and process characterization. Trends in product disposition cycle times serve as an indicator of

the level of control under which a plant operates. • Increasing cycle times may be evidence that there are more errors

and product failures that must be identified, documented, investigated, remediated and closed before a lot or batch can be released.

7 Source: PhRMA Response to Docket No. FDA–2013–N–0124: Food and Drug Administration Drug Shortages Task Force and Strategic Plan; Request for Comments 78 Fed. Reg. 9,928 (Feb. 12, 2013)

Complaint frequency trends may indicate the robustness of: • the manufacturing process and control strategy • product and process design controls The nature (severity) of complaints can indicate serious risk to patient safety, should also be considered. Internal audit findings • major and critical observations Regulatory agency inspection outcomes • major and critical observations

8 Source: PhRMA Response to Docket No. FDA–2013–N–0124: Food and Drug Administration Drug Shortages Task Force and Strategic Plan; Request for Comments 78 Fed. Reg. 9,928 (Feb. 12, 2013)


Adequacy of investment in manufacturing facilities, including routine maintenance

Adherence to, and effectiveness of, training programs Ability to complete activities within required timeframes is one

measure of a state of control, such as on‐time closure of: • Deviation investigations • Complaints • CAPAs Ability to identify, investigate and correct issues as measured by

effectiveness of CAPAs 9

Source: PhRMA Response to Docket No. FDA–2013–N–0124: Food and Drug Administration Drug Shortages Task Force and Strategic Plan; Request for Comments 78 Fed. Reg. 9,928 (Feb. 12, 2013)


Continuous Improvements to Your Training Program

• Expand your “why” training to all employees

• Make quality and training metrics visible to all levels of the organization

• Add formalized, documented monitoring (“spot checks” activities) in employee training records (via ComplianceWire Forms)

• Justify the upward trend in continuous improvement with training data (scores, assessments, monitoring and observations, etc)

Best Practices of “Why” Training

• GMP / Operator training content that provides context to an employee who otherwise may not see the big picture:

• Risks to Patients • Product shortages • Financial risks to the company

• Training could be received positively as professional

development activities, increasing employee engagement • Content focuses on how your role fits in the value chain:

• Real-world cases from industry • Regulations • Operational activities

Make Training Metrics Visible

Documented Monitoring

Forms tool can be used in the production floor to capture OJT activities

Aligning Performance with Metrics

Quarters Average Training Scores

Personnel Deviations

Audit Observations

1st Quarter 75 28 23

2nd Quarter 85 23 21

3rd Quarter 88 27 19

4th Quarter 92 18 15

Quality Assurance/Training: develop a process improvement project to reduce product errors to 15 or less per quarter

Summary: Moving from Compliance to Quality

• Changing the culture is a management requirement

• Posting daily or weekly metrics of value for each work unit

helps strengthen quality culture • Training data is a key part of capturing quality metrics that

improves both processes and employee performance

• Using ComplianceWire, clients have advanced up the “Quality-Based Learning Maturity Model”, helping them move beyond training record management and SOP “Read and Understand” activities

• With ComplianceWire and UL courses, QA teams are demonstrating to operations, manufacturing, and executives the business value of continuous improvement

Quality Standards Last the Test of Time… Comments/Questions


Training Effectiveness Training Methods/Modalities to Create Effective Learning & Retention





66% 59%




Learning Maturity Model

Focusing on Training Methods/Modalities to Create Effective Learning & Retention

Levels of Training Effectiveness • Reaction – What learners thought and felt about the

training

• Learning – The resulting increase in knowledge or capability

• Behavior – Extent of behavior and capability improvement / implementation / application

• Results – The effects on the business of the resulting from the change in learner behavior.

. . .that you have taken as an Adult

Think about the best training . . .

. . .that you have taken as an Adult

Had clear objectives Addressed “why” it was important to you Contextualized Allowed for opportunities to use the new

knowledge and existing knowledge to solve problems

Think about the best training . . .

FDA’s View On Training Effectiveness

Effective Training Programs Include the Following (for both

21 CFR Part 211 and Part 820):

• Personnel must be trained in the particular operations that they perform.

• Personnel must be trained in the current good manufacturing practices (CGMP).

• This includes the regulations and the written procedures (SOPs) required by these regulations

• Training must be conducted on a continuing basis with sufficient frequency to assure that employees remain familiar with CGMP requirements

FDA’s View On Training Effectiveness (cont)

Effective Training Programs Include the Following (for both

21 CFR Part 211 and Part 820):

• Good documentation & reporting of the training is important (e.g. date, content of training, trainer, length of training, etc.).

• Qualified individuals must conduct training. • Documentation is needed to demonstrate that employees

are sufficiently trained to perform specific operations. • Reading an SOP is not always an effective training

technique. • Must include methods to ensure employees can

effectively demonstrate their understanding of the training (e.g. assessments, OJT, etc.)

Common Deficiencies

• Lack of CGMP knowledge • Employees not trained on their specific job function

prior to performing duties - Temporary employees - Contract employees - Employees performing “non-critical” functions

• CGMP training not tailored to the firm’s training needs

9

Alicia M. Mozzachio, RPh, MPH - Branch Chief, ICB1 Center for Drug Evaluation and Research (CDER) - 5/8/2013

Common Deficiencies

• Employees not proficient in language in SOPs • Employees are “trained” but do not follow SOPs • Frequent deviations from SOPs • Operators not familiar with SOPs • Frequent operator error as the root cause of

investigation • Trainers not qualified to train

10


Common Deficiencies

• “Read and understand” as the most common training

strategy • No effectiveness or competency/skills checks,

especially for technical duties • No formal system to identify and track training needs

for each employee • Refresher CGMP training not conducted with

sufficient frequency • Personnel not trained when procedures are revised

11


Is This Important to My Firm?

YES!

Since January 2010, FDA has issued 66 compliance actions with failure to adequately train employees as one of the issues- at finished dosage sites alone! This includes 28 Warning Letters, 3 injunctions, and 1 seizure.

12


FDA 483 Example 1

There is no training and qualification program for new analysts assigned to conduct testing of raw materials and finished products. Training is limited to self-training without an evaluation of the analyst's understanding of the analytical methods and equipment.

13


FDA 483 Example 2

There is no record of training of applicable written procedures (cleaning SOPs) for the contract personnel authorized for cleaning in the sterile injection (Class A-D) and tableting departments after regular duty hours.

Furthermore, your firm delegated the training on these SOPs to the contracted cleaning company without adequate oversight to ensure the trainings are performed and employees are capable of adequately performing their cleaning duties without compromising the sterile manufacturing areas.

14


WL Example

Your firm failed to ensure that each person engaged in the manufacturing, processing, packing, or holding of a drug product has the education, training and experience, or any combination thereof, to enable that person to perform his or her assigned functions (21 CFR 211.25(a)).

For example, an employee examining microbial plates was unable to read and accurately record microbial counts. Additionally, our investigator observed employees functioning in roles supporting your sterile filling operations that were not following the procedures that govern their activities, such as glove change frequency, the handling of dropped objects, personnel monitoring, and sample acquisition.

15


Additional WL Examples

Several other recent warning letters, to U.S. and non-U.S. firms, have discussed training deficiencies:

• Training for specific job functions • Training for general CGMP • Repeated failures of personnel to follow procedures

16


Best Practices – Training Effectiveness


Training Effectiveness Continued…

Pre-Presentation Survey

What Year was UL Founded?

2



3

Tools to Support Effective Training • Initial Evaluation

• Exam or Evaluation Form

• Core Content • SOP, CBT, or ILC

• On-the-Job (OJT) Training

• Dual e-signature Form

• Refresher (3 months) • Read & Sign (Key Learning Objectives)

• Post-Assessment (6 months)

• Exam or Evaluation Form • Annual Refresher

• CBT or Read & Sign (Key Learning Objectives)



8



9

Core Content – Instructor Led Training • Based on the Risk Level … training modality may need

to be more robust

• Allows for learners to share ideas, work in groups, and debate/discuss with their peers

• Allows discussion on difficult concepts

• Explore the “gray areas” within the topics

• Can provide team building and be a Change Management tool

Core Content – Instructor Led Training • Use the electronic Class Sign-in Sheet to reduce

manual administration eliminate processing delays

On-the-Job Evaluation – Forms Training Items • Great for documenting evaluations based on

demonstrated performance • When online assessments are difficult to construct/deliver or

inadequate based on the material

• Perfect for higher risk processes that required more focused training and evaluation

• Required instructor and learner agreement to complete training

On-the-Job Evaluation – Forms Training Items

Build Baseline and Post-Training Forms: Data is captured for analysis that demonstrate improvement Capture Signatures from Both Employee and Trainer/Supervisor Provides visibility for manager oversight Auditable Format: Record can be presented as part of Certification curricula within an employee transcript

Measuring Progress via Online Forms

On-the-Job Evaluation – Forms Training Items

Manufacturing Employees: Replaces PC’s for shop floor OJT Training Sales Representatives Great for ride along activities CW accessible from all types

of devices

Use Mobile Devices for Forms

On-the-Job Evaluation – Forms Training Items Pre-Assessment Determination of Training Requirements: Automatically assign the appropriate training based on data collected in the form Failure on the OJT Evaluation can trigger additional training: Automatically assign additional ILC training Post Assessment or Annual Recertification: Automatically assign remediation training if a failure occurs at post assessment or recertification

Smart Forms Automate Assignments

Case Study: OJT Training & Assessment

Operations/Manufacturing Case Study • Role Based Curriculum includes:

• Read & Sign SOP’s with a Quiz • On-the-Job Training (OJT) • Standard Behavioral Assessment • Refresher Retraining of SOP’s

• Program shows both a minimum level of knowledge (quiz) and a minimum level of job performance (assessment)

Case Study: OJT Training & Assessment

Opportunities to Improve: • Ensure the SOP training includes an understanding of

any pre or post steps including issues with defects (supplemental training materials)

• Include a three month re-assessment of newly

qualified operators to show no decline in performance • Annual re-assessment of performance for all operators • Align with performance measurement (production,

yield, deviations, scrap, etc.)

Case Study: Targeted Training by Risk Level

Corporate Compliance Case Study • Foreign Corrupt Practices Act (FCPA) Training:

• Smart Form that asks about how the employee interacts with Foreign Government Officials

• Based on the response it classifies them in three groups (High / Medium / Low Risk)

• System automatically assigns corresponding training curriculum

• Program ensures that the right level of training is being received and taken by the employees based on their risk to the company

• Eliminates over and under training

Opportunities to Improve: • Have the manager either approve the form submitted

by the employee or fill it out in for the employee • Have the form filled out annually to account for

changes in role

Case Study: Targeted Training by Risk Level

Case Study: Blended Learning

Risk Management Case Study • Educate the organization (over 3,200 trainees) on Risk

Management (ISO 14971) • Foundational Computer Based Training (CBT) course on

the basics of Risk Management • Full day classroom session to discuss how to implement

Risk Management in their organization • Class session includes customized/localized case

studies/examples • Conducted Train-the-Trainer sessions to ensure enough

qualified trainers were available to locally train employees • Program ensured a consistent understanding of Risk

Management requirements and more detailed training on how to apply those concepts to their organization


Questions/ Comments


UL Eduneering Leadership Skill Building

Building and Managing a Leadership Program

How to Build a Leadership Program in ComplianceWire: Step 1. Build Your Leadership Curricula – Align your company’s leadership competencies to CrossKnowledge content, and organize content into “Competencies” Curricula Step 2. Build Your Leadership User Groups Step 3. Develop Your Self-Assessments – Done via SmartForms tool Step 4. Monitor Enrollment – Based on their responses to the assessment, learners are automatically placed into the appropriate Leadership User Groups

Compliance Leadership Program

AdvaMed and UL have developed a leadership program for compliance teams.

Compliance Leadership Program

What’s in Cross Knowledge?

• Managing People and Teams

• Selling/Negotiation

• Project Management

• Personal Management Skills

• Communication

• Management Fundamentals

• Understand Financial & Management Mechanisms

• Marketing Principles

• Understanding Human Resource Issues

Training Effectiveness Cycle • Initial Evaluation









ROLES: Qualification Reporting

Knowledge Transfer & Retention


Quality and Compliance Training Metrics





66% 59%




Learning Maturity Model

Focus on providing management visibility and connecting to business performance improvements.

What Metrics Build a Quality Culture?

What is the FDA expecting you to report on?

• 21 CFR Part 211 and 820 don’t state specific reports that are required … however, the FDA is not interested in the volume of training conducted

• FDA is interested in:

• Have you identified what the employee is required to train on for job function?

• Can you tell their current qualification status?

• Are you ensuring they remain qualified and are always in a qualified state prior to performing the task?

• Are your reports/records are complete, accurate, & timely?


Training Metrics Typically Include:

• Total Month/Quarter/Year Training Completions • Total Month/Quarter/Year Incomplete Training Items • Total Employees Training each Month/Quarter/Year

• Total Training Hours for the Month/Quarter/Year

Are these measurements really giving you the complete picture of Quality & Compliance?

Compliance Reporting – Standard Reports


Training Metrics Should Focus on the Training Outcome & the Current State/Risk of the Organization

• Compliance Percentage of the Company/Site/Department

• Employee Qualification Status

• Knowledge Transfer and Retention

• Behavior/Performance Assessments

• Business Performance Improvement

These measurements show that the training program is building the Quality, Compliance & Performance of your organization

Compliance Status of Your Organization

Compliance Reporting – Reporting Tool

Employee Qualification Status

Knowledge Transfer & Retention

Behavior/Performance Assessment

Business Performance Improvement

Quarters Average

Knowledge Assessment

Behavioral Assessment Scrap Rate

1st Quarter 75% 70% 10%

2nd Quarter 80% 81% 8%

3rd Quarter 88% 79% 5%

4th Quarter 92% 90% 2%

Quality Assurance/Training: develop a process improvement project to reduce scrap rate to less than 3%

Best Practices for SaaS System Validation

UL EduNeering Webinar Series UL and the UL logo are trademarks of UL LLC © 2014

Best Practices for a SaaS System Validation

Noreen Muscat, Sr. Consultant, UL EduNeering Advisory Services

http://www.uleduneering.com/webinar-series/


UL EduNeering Webinar Series

Overview

Validation _______________________________________________________

Protection of Records _______________________________________________________

Limiting System Access _______________________________________________________

Operational System Checks _______________________________________________________

Authority Checks _______________________________________________________

Personnel Qualifications _______________________________________________________

System Documentation

FDA Requirements – 21 CFR Part 11

© 2013 QACV Consulting, LLC




Overview

GAMP 5 was produced by a task team led by Guy Wingate (GSK). This work was overseen and supported by the ISPE/GAMP COP Committees.

GAMP® 5 – A Risk-Based Approach to Compliant GxP Computerized Systems

Source: GAMP 5 – A Risk-Based Approach to Compliant GxP Computerized Systems.

• GAMP COP Council • GAMP Americas Steering Committee • GAMP Europe Steering Committee • GAMP Japan Steering Committee

• Contributing regulatory authorities

included US FDA, UK MHRA, AFSSaPS (France), and Regierungsprasidium Darmstadt (Germany).




Overview

GAMP® guidance aims to achieve computerized systems that are fit for intended use and meet current regulatory requirements, by building upon existing industry good practice in an efficient and effective manner.

GAMP® 5 – Introduction

Source: GAMP 5 – A Risk-Based Approach to Compliant GxP Computerized Systems, Section 1, Pg.12.




Overview

GAMP® 5 – Life Cycle Phases

Source: GAMP 5 – A Risk-Based Approach to Compliant GxP Computerized Systems.




Overview

Establishing documented evidence which provides a high degree of assurance that a specific computerized process or operation will consistently produce a quality result meeting its predetermined specifications.

GAMP® 5 – Validation Definition

Source: GAMP 5 – A Risk-Based Approach to Compliant GxP Computerized Systems, Pg. 335, Appendix G2.




Overview

Regulated companies should seek to maximize supplier involvement throughout the system lifecycle in order to leverage knowledge, experience, and documentation, subject to satisfactory supplier assessment.

2.1.5 Leveraging Supplier Involvement

Source: GAMP 5 – A Risk-Based Approach to Compliant GxP Computerized Systems, Pg. 21.




Overview




• Planning should determine how best to use supplier documentation, including existing test documentation, to avoid wasted effort and duplication. Justification for the use of supplier documentation should be provided by the satisfactory outcome of supplier assessments, which may include supplier audits.




Overview




• Planning should determine how best to use supplier documentation, including existing test documentation, to avoid wasted effort and duplication. Justification for the use of supplier documentation should be provided by the satisfactory outcome of supplier assessments, which may include supplier audits.

• Vendor documentation should be assessed

for suitability, accuracy and completeness. There should be flexibility regarding acceptable format, structure and documentation practices.




Overview

11

One of the appeals of the SaaS application is that a company can shift some of the validation effort to the SaaS vendor. • This enables the company’s validation team to focus initially on an

audit of the vendor’s data center, as well as the vendor’s QA and validation methodology, to ensure these activities are performed at the same standard as would be performed by the client’s own QA and validation teams.

• Typically, the time spent auditing the SaaS vendor can dramatically reduce the time spent validating the system.




Overview

12

Formal Agreements are required to include clear statements of responsibilities:

• Validation deliverables – who and what • Security • Change Management • Incident Management • Backup and restore • Business continuity / Disaster recovery • Training • Escalation

Vendor Service Level




Overview

13

• SDLC Methodology • Project Planning • Personnel Qualifications • Documentation Standards &

Procedures • Methods for review & approval • Design Standards • Programming Standards • Configuration Management • Testing Standards & Procedures • Separation of Development, Test and

Production Environments

• Move to Production Process • Clearly defined responsibilities • Involvement of: o Customer/User o Quality Assurance professionals o Technology Professionals o Change Management o Training process o Process for continuous evaluation,

incident monitoring, error correction o Processes and procedures for

physical & logical security of system and data

Vendor Quality Process:




Overview

14

Typical Auditing Activities: • Provide Audit Plan and Scope, including agenda • Audit Scope may include:

o Quality System, including Training Program, Internal Audit, Document Management, etc.

o SDLC o Customer Support o Data Center / Computing Environment, including disaster

recovery, backup & restore, security, etc. _________________________________________________________________________________________

Goal – Assess supplier processes for adequacy of controls and also ability to leverage supplier SDLC and validation documentation

Vendor Audit Best Practices:




Overview

Audit planning should involve these activities in coordination with the internal audit team and the vendor’s audit team. The vendor should have experience to assist with details such as: • Defining the scope of audit

to include areas to focus on • Deciding on the audit Team

(QA, IT, SME) • Scheduling the audit with

the vendor

15

Vendor Audit Planning:




Overview

Audit planning should involve these activities in coordination with the internal audit team and the vendor’s audit team. The vendor should have experience to assist with details such as: • Defining the scope of audit

to include areas to focus on • Deciding on the audit Team

(QA, IT, SME) • Scheduling the audit with

the vendor

16

A provisional agenda should be provided to vendor so the vendor can prepare materials in advance.

Vendor Audit Planning:

EXAMPLE: Define organization and scope of audit Provide a list to the vendor of standard

documentation to be reviewed during audit prior to the meeting

Include required availability of staff during audit (Technical staff, IT staff, QA staff)

Audit details should be confirmed in writing with vendor

Notify vendor of the intended use of a third party auditor

Confidentiality agreements should be signed prior to meeting




Overview

17

An audit of a vendor’s procedures and data center minimizes the need for clients to “retest” core functionality. The client’s validation team will typically perform several activities related to the organization’s validation effort.

Vendor Audit:




Overview

18

Vendor Audit Procedures:

Best Practice for minimum validation documentation performed to satisfy regulatory agencies, including FDA, which may ask for these documents during an audit: • Validation Plan • User Requirements

Specification • User Acceptance Test Scripts

(including testing for customizations, integrations)

• Validation Report • System Governance




Overview

• Introduction > • System Overview

• Business Objective

• Risk Management

• Team Roles and Responsibilities

• Training

• Validation Approach

• Validation Deliverables/Test Plan

• System Governance Deliverables

• Acceptance Criteria

19

A typical Validation Plan will include these sections:

Validation Plan:

Example: This Validation Plan (VP) determines the steps that must be completed in order to consider <company name’s> Learning Management System (LMS), a web-based Commercial Off the Shelf (COTS) system that is a validated system in compliance with a regulated environment.




Overview

• Introduction

• System Overview > • Business Objective

• Risk Management


• Training





20


Validation Plan:

Example: The System a web-based COTS system used to electronically assign training requirements, record training completions and provides reporting on training. The system is a hosted system maintained by <company>.




Overview

• Introduction

• System Overview

• Business Objective > • Risk Management


• Training





21


Validation Plan:

Example: The Business benefits resulting from the successful implementation of the System will include: • Enhanced training compliance and reporting on training • A comprehensive Learning Management Solution that can

scale throughout the organization. • A centralized, web-based solution to facilitate training and

training records • Reportable query capabilities to facilitate faster retrieval of

training records • Will reduce future paper processes and the resources

needed to maintain paper processes • Electronic signature for more efficient record keeping • Version control for curricula




Overview

• Introduction

• System Overview


• Risk Management > • Team Roles and

Responsibilities

• Training





22


Validation Plan:

Example: The SYSTEM being implemented by <company name> is a regulated system used for GxP and/or non-GxP training activities. The system is considered low risk to patient safety, product quality and data integrity (GAMP Computer System Validation Category 3, Low risk). The system’s intended use is to facilitate and record training; department leads are responsible for user training requirements for GxP processes. The system is considered compliant with 21 CFR Part 11 regulations.




Overview

• Introduction

• System Overview


• Risk Management

• Team Roles and Responsibilities >

• Training





23


Validation Plan:

Example:

Role Name and Department Responsibility

Business Project Owner

• Implementation and management of the system for the business user community,

• Reviews and approves all deliverables

IT Project Lead

• Implementation and management of the system for the business user community

• Ensures all technical requirements are met for system use • Reviews and approves all deliverables • Approves release of the system

Project Manager

• Control of project planning and timelines • Manages resources and cost • Ensures issues are managed and resolved • Provides reporting and status updates to Sr. Management

Quality Assurance Lead

• Assures compliance with appropriate regulatory and quality requirements

• Provides support and guidance for review and approval of all deliverables

• Approves the release of system




Overview

• Introduction

• System Overview


• Risk Management


• Training > • Validation Approach




24


Validation Plan:

Example: Validation Team training requirements – Validation plan and test protocol training, Technical System training will be through System. All training must be completed for Validation Team prior to execution of test scripts. User training – Users will be trained on how to use the system by the System Administrator and/or self training material prior to gaining access.




Overview

• Introduction

• System Overview


• Risk Management


• Training

• Validation Approach >




25


Validation Plan:

Validation is the process of establishing documented evidence, which provides a high degree of assurance that a specific process will consistently produce a product meeting its predetermined specifications and quality attributes. <company name> will validate the System User requirements to make certain the system is in place, and that it performs and functions as expected. The system validation will be prospective as the validation activities will be conducted prior to the production release.

Qualified personnel will be required to perform validation activities. All deviations (test incidents) observed during testing will be documented and resolved prior to test completion. Deviations include test results that do not meet the acceptance criteria, system errors, program crashes, or other anomalies defined in the test scripts. All deviations must be clearly defined in the Validation Summary Report (VSR) prior to system release.

This test plan for the validation effort will be designed to ensure the performance of the system representing the actual intended use. All testing associated with the validation effort will be formally documented and included in the validation package.

An overall VSR will be issued by the validation team to formally close the validation effort. The VSR will be approved by appropriate personnel. All validation activities will be completed and approved prior to system release.




Overview

• Introduction

• System Overview


• Risk Management


• Training


• Validation Deliverables/Test Plan >



26


Validation Plan:

Example: <Company name> is a regulated company which requires a high level of confidence that the computer system will meet technical, commercial and regulatory requirements. <Company name> will leverage the knowledge, experience and documentation of the vendor to reduce duplication of functional and installation requirements and reduce the redundancy of testing already-tested by vendor (reference vendor audit). The System is a web-based system which is hosted and maintained by <vendor name>. The vendor has a robust process for software development of functionality of the system (reference vendor audit) so <company name> will leverage <vendor name> of functionality (OQ) and installation (IQ). <Company name> will include planning and testing on all integrations and customizations to system.




Overview

• Introduction

• System Overview


• Risk Management


• Training


• Validation Deliverables/Test Plan >



27


Validation Plan:

Example: The following documents are the deliverables to be completed for the System Validation project. 1. Validation Plan 2. User Requirements 3. Application Configuration Specifications (Note-Vendor

document, maintained by client after go live) 4. User Acceptance Testing 5. Validation Summary Report 6. Traceability Matrix 7. Vendor Audit Report The following documentation should be referenced from vendor audit report (reference vendor audit report sections) and archived with validation package. 1. IQ 2. OQ 3. 21 CFR Part 11 Mapping




Overview

• Introduction

• System Overview


• Risk Management


• Training



• System Governance Deliverables >


28


Validation Plan:

Example: Use and Operation Procedures – including general Use and Operations for Users and User e-signature certification.

System Administration Procedures – including security roles, system admin roles and responsibilities, maintenance (including system releases), configuration changes requiring change control.

Computer System Change Control – including standard operation procedures for system configuration changes, addition of new functionality, handling system releases.




Overview

• Introduction

• System Overview


• Risk Management


• Training




• Acceptance Criteria >

29


Validation Plan:

Example: The UAT acts as a final verification of the intended business use and proper functioning of the system. If the software works as intended and without issues during normal use, one can reasonably extrapolate the same level of stability in production.

Qualified personnel will be required to perform validation activities. All deviations (test incidents) observed during qualification testing will be documented and resolved prior to test protocol completion.

All validation activity and any deviations or issues must be completed/resolved prior to release of system. User Acceptance Test scripts must be executed and passed prior to system release.




Overview Best Practices for Managing System Releases

30

Agenda




Best Practices for Managing System Releases

31

____________________________________________________________________________________________

30 Days Prior to System Release:

Platform Release Notes: • Enhancement List • System Availability • Regression Test Script Access Information • Preview Testing Details • Standard Enhancement Demos

____________________________________________________________________________________________


• Platform Release Guide • Enhancement Details • Elective Enhancement Demos

Sample Notifications to System Admin:

____________________________________________________________________________________________


• Final Release Details • Premium Enhancement Demos





• Standard > • Elective • Premium • Consultative

32

Sample System Release Enhancement Categories:

Standard Enhancements: A platform-impacting enhancement that affects The System functionality for all Clients.

• Does not result in additional cost to existing The System Clients.

• Changes to the The System platform that are considered ‘Standard’ are not configurable for individual instances and cannot be turned off.

• Documentation of changes and impacts will be provided with Release Communication.





• Standard • Elective > • Premium • Consultative

33

System Release Enhancement Categories:

Elective Enhancements: A change to the The System platform functionality that is not defaulted to be ‘enabled’ for all Clients and is available without additional cost.

• Details related to the operability, impact and activation instructions will be provided for Elective Enhancements in Release Communications.





• Standard • Elective • Premium > • Consultative

34


Premium Enhancements: A change to the The System functionality or tools that requires additional cost or subscription.

• These changes will be documented on a high level in the Release Communication.

• In many cases, the implementation of such enhancements requires further analysis of a Client’s instance of The System with additional implementation and application support.

• Clients who are interested are encouraged to contact their Account Director for more details and pricing.





• Standard • Elective • Premium • Consultative >

35


Consultative Enhancements: Functionality or tools that are being built and may have some components introduced to the infrastructure of The System, but are not yet available for purchase.

• The changes will be documented on a high level in the Release Communication.

• In many cases, the implementation of Consultative Enhancements will require further analysis of a Client’s instance of The System. It may be necessary to customize changes, reconfigure or require professional services.

• Clients who are interested are encouraged to contact their Account Director for more details and pricing.





36

• Organizational administrator reviews the system release notes with appropriate business users, system owner, QA and IT to decide if any non standard enhancements will be utilized.

• Organizational administrator reviews the regression test scripts received for completeness for testing for standard enhancements and non standard enhancements being utilized.

• Organizational administrator completes an impact assessment form and coordinates with QA, IT and System owner to evaluate the impact and identify procedure changes that may require change control (i.e. New system feature being used) and any additional testing required.

System Release Process Overview – Example





37

System Release Impact Assessment – Example





38

System Release Impact Assessment – Example, Completed





39






40





• Minimize validation – do an initial audit of vendor to determine if its validation processes are sufficient

• Audit at least every 2-3 years to ensure vendor is still in compliance with your standards

• Share validation responsibility with Vendor, QA, IT, Business owner, Administrators: o Vendor audit of functional requirements should be referenced in your

validation plan to justify why you didn’t do full validation (IQ, OQ, SDS, Backup/restore, etc.)

o QA reviews and approves documentation

o IT reviews and approves documentation, sets up user PC environment

o Business Owner – authors documentation

o Administrators – runs any test scripts

41

Best Practices

Tips:


2014 Life Science Learning Series – Overview Life Science Learning Series – Overview . ... MHRA,...

Documents

Transcript of 2014 Life Science Learning Series – Overview Life Science Learning Series – Overview . ... MHRA,...