FDA News Webinar - Inspection Intelligence

1Copyright © 2016 Qualified Data Systems Inc.

Inspection IntelligenceDeveloping a Data-Driven Approach

FDA News WebinarDecember 19th, 2016

By: Armin Torres, Principal Consultant – Qualified Data Systems


Inspection Intelligence OverviewInspection Intelligence– Knowledge acquired through the collection, aggregation, analysis, and interpretation of information sources which enable proactive compliance readiness.

The Data-Driven Digital Quality Approach:• Uses External and Internal data sources to develop a complete 360 degree view of your

Compliance Readiness status• External Data sources highlight your organization’s Quality and Compliance Risk Profiles using

post-market feedback channels• Internal Data sources are intended to prepare your organization for inspections using quality and

compliance metrics derived from key QMS systems. This is your Vital Signs Monitor.• Five key Medical Device QMS Systems for Vital Signs Monitoring include:

• Corrective & Preventive Action (CAPA), Production & Process Controls (P&PC), Management Controls (MGMT), Design Controls (DES), and Document Controls (DOC)


Intelligence Information Sources

Industry

Providers

Pharma, Device, Biologics, Cosmetics, other Industry Groups(Product Life Cycle Data)

Academic, Independent, Consulting Studies, Medical Literature, Publications

Hospitals, Clinics, Diagnostic Centers(Event Data)

EU, APACUSA, LATAMROW(Regulations)

Insurance Companies(Claims Data)

HDO’s

Sources of Potential Emerging Signals

RegulatoryIntelligence

User Experience Reporting (Sentiment Data)


Intelligence Platform

External Data Sources

Data Remix & Mining

Internal Data Sources

DATAWAREHOUSE

REGULATORYINTELLIGENCE

REPORTING

ALERTS STATISTICAL

VISUALIZATION

ETL PROCESS

EXTRACT

VALIDATE CLEAN

LOAD

QMS Supply Chain Manufacturing Customer Service

CAPAAdverse Event Deviations Risk Mgt. Development Maintenance

Public Domain Data (API)

Proprietary Data - Contextualization

Products & Services

Regulatory Science Innovation

Pre-Production, Production, & Distribution Data

Post-Market Crowdsourced Data


Intelligence Platform • Provides a holistic view of external environmental factors and internal

performance data which can be measured, baselined, reported, and prioritized for improvement.

• Can help identify areas of regulatory overlap and gaps• Assists in prioritizing Risk-based reviews based upon key performance indicators• Facilitates Product & Service Quality improvements through the integration (i.e.

“Remix”) of internal and external information sources• Establishes ownership and accountability for meeting Quality Objectives and

Compliance requirements.• Establishes a solid foundation for Governance, Risk, and Compliance

Management across the enterprise.


External Open Data?Open Data Definition – “Open means anyone can freely access, use, modify, and share for any purpose (subject, at most, to requirements that preserve provenance and openness).” In other words:“Open data and content can be freely used, modified, and shared by anyone for any purpose”Why is Open Data so important?Because licenses and dedications are readily available such as Creative Commons, Open Data Common Public Domain Dedication, Open Data Commons Open Database License (ODbL), etc.

Source: http://opendatacommons.org/

http://opendatacommons.org/


External Data Brokers & Services Use of External Data Brokers:• Typically offer services for a fee (fixed or A-la-Cart)• Services often include Searching, Filtering, aggregating, and

limited downloading or reporting facilities• Payment for these services does not necessarily translate

into higher quality data. Choose wisely!• Beware of Service Providers who repackage Open Data for a

fee without proper disclosures or aggregate copyrighted information from the Web.

• Understand that there is an entire industry dedicated to filling Freedom of Information Requests for undisclosed recipients (FOIA.gov)

• Exclusions to the FOIA include some Law Enforcement and National Security Records.

Look for trusted and validated data vendors who provide transparency and promote open standards for information sharing.


Using External Open DataBenefits:• Large historical datasets readily available for data mining. Use these as lagging indicators of

performance.• Self-Service Data discovery, Analysis, and Visualization tools for Lay-Users maturing rapidly.

Vendor product eco-system is robust.• ROI easier to demonstrate through Proof-of-Concept use cases• Can provide competitive advantage in some use cases where product performance can be

benchmarked.Challenges:• Not all structured data sources use a common standards-based framework• Often requires specialized IT skills and costly tools to prepare the data for use• Use case for real-time Advanced Safety Signal Detection is still a long way off but identifying

Emerging Signals may be possible.


Identifying Emerging Signals in Medical Devices• FDA intends to use on-market performance data to continuously assess device Benefit/Risk Profiles to

identify potential Emerging Signals and communicate them to the public.• Emerging Signals may represent a new potentially causal association or a new aspect of a known association

between a medical device and an adverse event or set of adverse events.• Information about device-related signals may come to the attention of FDA through a variety of sources

including, but not limited to, Medical Device Reports (MDRs), MedSun Network reports, data from mandated post-market studies, clinical trials or data published in the scientific literature, epidemiological research including evaluation of administrative databases, health care claims data or registries, and inquiries or investigations from global, federal or state health agencies. In other words, all intelligence data sources.

• Following identification of a signal, the signal first undergoes refinement, whereby additional information from other data sources is identified, gathered, and evaluated. The signal refinement phase includes interactions with the device manufacturer(s), unless time does not permit because of the risk of patient harm or it is not feasible, e.g., CDRH cannot reach all manufacturers.

• Be ready if the FDA calls to discuss a potential emerging signal by proactively assessing External OPEN FDA DATA!


Digital Factories and their DataProactive assessment of Product Safety Signals

• Location where Product Quality is crafted • Access to data is controlled by internal policies and procedures not external entities• Timeliness and proximity to data means higher quality, integrity, and value• Where the product knowledge lives• Promotes safety innovation versus costly post-market design changes

Internal Data Challenges• Regulated Data scattered across the corporate environment and often insulated from rest of company• Data is housed in vertical silos of vendor applications with little interoperability without additional pay walls

being traversed.• Organizations lacking clarity to enterprise Data Governance, Risk, and Compliance practices• Data Management maturity highly dependent on company size and financial resources available• Data Access many times is limited to a select few who posses licenses• Heavy IT involvement still needed in some areas where skills or tools are not readily available for data

analysis


Data Quality matters• Recent focus on Quality Metric Programs developed for the Pharma and the

Medical Device Industries in cooperation with FDA, ISPE, and Xavier Health highlights the need for a renewed emphasis on Data Integrity and Quality.

• While much of the Metric programs (PQLI-Quality Metrics and CfQ-Case for Quality) focus on metric definition and reporting schemes, many industry challenges remain to successful implementation. Examples include:

• Making Raw Data available using the least burdensome approach• Cost of manual or automated data collection, review, and aggregation• Process development for Digital data extraction, cleansing, integration, and validation prior to reporting • Training and availability of resources for Data Analysis, Interpretation, and Visualization• Implementation of Tools for Analytical processing • Additional Software Validation requirements• Timely Management Reporting• Making data actionable in order to operationalize a data-driven culture


Raw Data – Least Burdensome Approach•Today, Digitization is the road that leads to the least burdensome approach for collecting, aggregating, analyzing, and interpreting raw data generated from your Quality Management Systems.

•Digitization generally speaking is the process of converting raw data into a digital format. This process may include preparing text, images, audio, video, and numerical information for subsequent use.

•The key advantages to Digitization include:• Ability preserve, access, and share the data• Ability to aggregate data from disparate data sources• Ability to search the digital records• Ability to analyze, interpret, and communicate the information


Optimizing Costs of DigitizationThere are several key aspects of Digitization that can help your organization optimize the costs associated with data collection, aggregation, analysis, interpretation, and reporting of Intelligence Data:• Standardization – Standardization can be used to generate a documented data taxonomy which includes

governance and management of your digital assets. The output of standardization should be an SOP that clearly describes the organization’s data management processes. Additional standardization should be sought for Technology components used to support your efforts. For example, standardization on Excel, Minitab, SharePoint, Microsoft BI, SAS, Tableau, or Yellowfin tools and training for personnel should be considered.

• Consolidation – Too many meaningless metrics and data expeditions can have a severe impact on the bottom line. A good way to remember this is with the following saying “Not everything that counts can be counted, and not everything that can be counted counts“. Chose a small set of actionable metrics that are impactful to your business and get management support before even starting. A useful online reference to help you get started is provided by Xavier Health. For Medical Devices: http://xavierhealth.org/device-measures-initiative/ For Pharma: http://xavierhealth.org/pharma-quality-metrics-initiative/

• Process Efficiency – Defining an strategic organizational model to support your data assets is prudent in saving time, effort, and cost. Different skills sets may be required such as Data discovery, Integration, Aggregation, Cleansing, Visualization, Analysis, and Reporting. Roles and Responsibilities should be defined in your data governance SOP’s or Work Instructions.

http://xavierhealth.org/device-measures-initiative/

http://xavierhealth.org/pharma-quality-metrics-initiative/


ETL BasicsDeveloping a process for digital data Extraction, Transformation, and Loading (ETL) need not be a difficult or complex endeavor for your External Open FDA Data. Some key considerations include:• Defining the approach to collect the data (the Extract).

• If you desire to connect directly with OPEN FDA data sources, then you will need to understand how the FDA’s OPEN DATA initiative works and how you can connect directly using an open-source Application Programming Interface (API). For more detailed information visit: https://open.fda.gov/

• If you would like a more basic and targeted level of data collection use the FDA databases directly as follows:• For Medical Devices: http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/Databases/default.htm• For Pharma: http://www.fda.gov/Drugs/InformationOnDrugs/default.htm

• Transforming the Data may involve elements of data validation and cleansing because the OPEN FDA data is continuous evolving, being enhanced, and changing. Therefore, it may be required to verify the data schemas and records from time to time. Typically, when something in the database schemas change your records downloaded will reflect these changes and adjustments will need to be made on your side.

• The loading process typically involves the storage of the downloaded database records in some form of consolidated data warehouse where other users in your company can access the data for subsequent review, analysis, and reporting.

https://open.fda.gov/

http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/Databases/default.htm

http://www.fda.gov/Drugs/InformationOnDrugs/default.htm


Organizational NeedsSupporting a Digital Quality Metrics Program requires consideration for additional Training and Development of resources. Some of the top challenges experienced by companies implementing successful programs include:• Insufficient Resources;• Lack of knowledge; and• The ability to analyze, interpret, and communicate resultsMany times resources don’t have the necessary data visualization or statistical backgrounds required to demonstrate a return on investment to Sr. Management. This is especially acute when the organization is ready to transition from descriptive to predictive analytics.


Tools for Analytical ProcessingIndustry and Vendor solutions have matured in recent years to support the growing need for business intelligence tools. A good tool to review some of the leading solution providers at a glance is using the Gartner Magic Quadrant. For further details refer to: https://www.gartner.com/doc/2668318/magic-quadrant-business-intelligence-analyticsSome of the popular tools used in the Life Sciences include:• Microsoft Power BI• Tableau• Provalis Research• Yellowfin• Birst• SAS• Qlik• SAP• Minitab

https://www.gartner.com/doc/2668318/magic-quadrant-business-intelligence-analytics


Software Validation Requirements• Begins with robust Data Hygiene

• Good documentation practice procedures• Data Entry field level verification for automated systems• Integrity and Business Logic verification for e-Records• Defined Data Management practices • 21 CFR Part 11 compliance (Security, Audit Trails, e-Signatures)• Comprehensive Software Validation Life Cycle for electronic Quality Systems

• Needs to Extend throughout the entire Quality Management System• To ensure Confidentiality, Integrity, and Availability of Regulated Information Systems• To promote cross-platform data integration, aggregation, and visualization• To enable efficient management reporting• To flatten silos of data for better collaboration, analysis, and problem resolution


The State of Data Driven Decision-Making• Technology has made Data Analysis more accessible, but:• Today’s organizations still struggle with the basics of getting their hands around the data needed

to make better decisions.• Excel Spreadsheets and PowerPoint are still the main entry points for data collection, analysis,

and presentation of descriptive historical information.• It still takes way too long to aggregate data and contextualize for Sr. Management review

(some take weeks to compile PowerPoints with hundreds of metrics which are tracked across the business). As a result, many reviews only performed quarterly.

• Near Real-time data review is still a utopian concept.• Focus from the enterprise continues to be on structured Data Analytics. Show me the numbers!• Organizations still have little demonstrated capabilities when it comes to managing and making

use of unstructured data (e.g. Text Analytics). 80% of the world’s data is of this kind.• Data Analytics and self-service enable better data discovery but not necessarily delivers on better

business outcomes.


Making Data ActionableSelect metrics that make the most business sense (gives value) for your organization. The metrics should be accurate, timely, and contextualized for it to add value in decision making.For Quality and Compliance consider the following FDA Proposed Metrics:Drugs:• Lot Acceptance Rate (LAR) as an indicator of manufacturing process performance. LAR = the

number of accepted lots in a timeframe divided by the number of lots started by the same covered establishment in the current reporting timeframe.

• Product Quality Complaint Rate (PQCR) as an indicator of patient or customer feedback. PQCR = the number of product quality complaints received for the product divided by the total number of dosage units distributed in the current reporting timeframe.

• Invalidated Out-of-Specification (OOS) Rate (IOOSR) as an indicator of the operation of a laboratory. IOOSR = the number of OOS test results for lot release and long-term stability testing invalidated by the covered establishment due to an aberration of the measurement process divided by the total number of lot release and long-term stability OOS test results in the current reporting timeframe


Making Data ActionableDevices:• Pre-Market

• Design Robustness Indicator - Identification of design elements that eliminate, reduce, and prevent design failures throughout the product lifecycle. A quality design should require minimal changes. Changes should be enhancements or due to technological advancement.

• Metric: (Total # of Product Changes/Total # of Products with initial sales in the period)

• Production• Right First Time Indicator - Tracking and trending production-related right first time data to eliminate,

reduce, and prevent repeat failures. First Time Yield metric. • Metric: (# of units mfg. w/o non-conformances/# of units started)

• Post-Market• Post-Market Indicator - Analysis of key post market surveillance data to eliminate, reduce and prevent

on-market failures. • Metric: Service Records * (0.10) + Installation Failures * (0.20) + Complaints * (0.20) + MDRs * (0.20)

+ Recalls (units) * (0.20) + Recalls (total) * (0.10)


Top Challenges in Incorporating Data Analytics

Access to DataDifficulty in obtaining, accessing, and/or compiling the data.

Lack of KnowledgeLack of understanding about data analytics.

Lack of Management Buy-inManagement not clear on scope, value, or outcomes.

Insufficient ResourcesInsufficient Resources or the need to train personnel.

Results InterpretationInability to interpret results and communicate effectively.

Time CommitmentTime required to develop and execute analytical procedures01

0305

02

0406

Source: The 2015 Data Analytics and Leadership Survey (The IIARF and Grant Thornton)


Intelligence for Inspection ReadinessWhen FDA contacts you regarding an Inspection:• Do you have a plan to get ready?• Does your plan include Data Analysis?• Do you worry FDA is going to find something you didn’t?• Do you know your vulnerabilities?• How do you know your ready?• How do you measure your readiness state?• What Quality Management Subsystems do you focus on for Inspection

Readiness?• What data do you review first?


Intelligence Strategies for Inspection Readiness

• Routinely Analyze OPEN FDA DATA• Be aware of your FDA risk profiles Quality Profile – Recalls, MDR’s Compliance Profile – 483’s, Warning Letters, Ext. Audit Data

• Benchmark your MDR and Recall reporting: By Product Code By Competitor

• Prioritize Product Problems and Device Malfunctions for Design Change• Stay on top of regulatory compliance trends• Review Internal Data for Improvement Opportunities and Compliance exposure• If you don't know where to start, start with digitizing Internal Audit data!


Assessing Internal Compliance RisksP&PCProduction and Process controls are a major contributor to FDA compliance Gaps

TrainingFDA is focusing more and more on Training Effectiveness as a CAPA activity.


Assessing Internal Compliance RisksCAPAIts important to understand how effective the CAPA process performs over time.

Internal AuditOutstanding Internal audit gapsmay be an indication of additional compliance risk.


Assessing Internal Compliance RisksDocument ControlsMonitoring ECO’s can be used as an indicator of design robustness.


Assessing External Compliance RisksComplaint AgingComplaint Aging may be pointing to other areas of the quality system that need attention.

Complaint TrendingComplaint Trends may highlight Product Performance concerns in the field.


Assessing External Compliance RisksMDR’sLate MDR reporting may lead to undesirable 483’s.

RecallsDevice Recall analysis can help in understanding Total Product Quality Risks.


Device Intelligence Use Cases for Quality ImprovementExternal FDA Data Sources Internal Data Sources Business Value

MAUDE – Product Malfunctions Design Control - Design Inputs Analysis of Product Problems for similar devices by Product code for product design

MAUDE – Events Risk Management – DFMEA Evaluation of Probability of Occurrence against Severity of Event Types and reported malfunctions

TPLC Design Control - Design Changes Analysis of Design Changes against reported Device Problems including Recalls

NIST – National Vulnerability Database (Cyber Security)

Risk Management – Software Hazard Analysis

Analysis of Exploitability of Software Vulnerabilities for Cyber Security against Product Safety

NIST – National Vulnerability Database (Cyber Security)

Software Device Verification & Validation Comparison of NIST Vulnerabilities against Internal V&V Penetration & Fuzz Testing results

FDA 483’s Quality Audits Comparison of FDA 483’s against Quality Audit Analytics for Inspection Readiness Preparation

MAUDE – Product Malfunctions Software Design Verification & Validation Analysis of Software Defects reported (escapes) against internal V&V Defect Density


Questions?

Contact Info: Armin Torrese-mail: [email protected]: www.qualifiedsystems.com

mailto:[email protected]

http://www.qualifiedsystems.com/

FDA News Webinar - Inspection Intelligence

Data & Analytics

Transcript of FDA News Webinar - Inspection Intelligence