Quarterly update on the CDRH Case for Quality Voluntary...
Transcript of Quarterly update on the CDRH Case for Quality Voluntary...
1
February 27, 2018
MDICx webinar seriesQuarterly update on the CDRH Case for Quality Voluntary Medical Device Manufacturing and Product Quality Pilot Program
George ZackTwo Harbors ConsultingKimberly KaplanCMMI InstituteJoe FriedrichBoston ScientificFrancisco VicentyFDA Center for Devices and Radiological Health
Medical DeviceDiscovery Appraisal Program
MDICxQuarterly update:CDRH Case for Quality Pilot Program February 27, 2018
• Case for Quality & MDDAP
• Program Success Indicators and Metrics
• What is CMMI
• Program Overview and Lessons Learned
• Additional Information
Agenda
Case for QualityMedical Device Manufacturing and Product Quality Program
2011: Launch of Case for Quality - Dr. Jeff Shuren Discusses Case for Quality
2014: MDIC joins the Case for Quality - Video Overview of the Case for Quality
2015: Deloitte White Paper - CMMI Selection
2016: Proof of Concept Pilots - Outcomes
2017: Launched Pilot Program
What is this program?
Reduced defects / rework
Accelerated time to market
Increased Customer Satisfaction
A culture of quality - across the organization.
This pilot program leverages the CMMI framework as the standard maturity model by which medical device organizations may measure their capability to produce high quality devices and increase patient safety. FDA will adjust their engagement activities and submission requirements as a recognition of this independent assessment of quality maturity. The CDRH Voluntary Medical Device Manufacturing and Product Quality pilot was announced in the Federal Register on December 28, 2017.
Reduced costs
• Value to ParticipantsWhat value are stakeholders getting from program?“Program Effectiveness”
• Consistency & Scalability Is the program sustainable?“Program Adoption”
• Elevating IndustryThe long term “next steps”…
• What: Appraisal identifies opportunities for improvement, reduced regulatory burden, increased innovation, faster time to market How: Survey results, participant feedback, appraiser feedback, FDA feedback, lessons learned incorporated into program
• What: Program operations are performed consistently for growing # new participantsHow: Number of appraisals, wait time to appraisal, number of appraisers trained in the program, lessons learned incorporated into program
• What: Industry baseline for organizations to benchmark improvement journeyHow: trend participant results & quality performance metrics over time
What does success in this pilot program potentially look like?
Success Components Success Identification
• # of Enrollees13 Organizations | 17 Facilities
• # of Appraisals6 Complete | 3 Scheduled
• Time from enrollment to appraisal execution (days)Range = 52 - 144 | Mean = 98 | Target = 90
Program Adoption Metrics to Date
• Did the appraisal identify areas or processes that could improve how work is performed to increase product quality?
91% YES | 9% NO
• Did the appraisal practice areas conflict with any regulatory compliance assessment areas?
98% NO | 2% YES
• Did you find the appraisal to be of value? 100% YES | 0% NO
• Would you recommend this program?100% YES | 0% NO
Program Effectiveness Metrics to Date
• The Capability Maturity Model Integration (CMMI®) is a capability improvement model that can be adapted to solve any performance issue at any level of the organization in any industry.
• The Model provides guidelines and recommendations for helping your organization diagnose problems and improve performance.
• Used by over 10,000 organizations from more than 100 countries all over the world, CMMI helps you identify and achieve measurable business goals.
What is CMMI®?
60%DECREASE
100%MILESTONE
FIDELITY
50%DECREASE
on timecost of poor quality defects
Czech Republic
50%DECREASE
non-conformities on project deliverables
INCREASE
On Budget Performance
INCREASE
On Time Performance
Customer Satisfaction
INCREASE
39% 71% 53%
Productivity 25%INCREASE
Organization High Level Roles for Pilot
Pilot SteeringCommittee
Provides leadership, direction and pilot process input
FDA Provides regulatory modifications, verifies participants, reviews detailed findings and improvement, provides pilot process input
MDIC Coordinates working groups of enrollment, appraisal, metrics, communications and program oversight; provides pilot process input
Appraisers Execute appraisals, provide findings to participants, executes checkpoints, provides full datasets to coordinating center, provides pilot process input
ParticipatingDevice
Manufacturers
Receives appraisals, drives improvements within business, participates in checkpoints to report progress, provides pilot process input
CMMI® InstituteProgram
Management Office
Provides model, manages enrollment/de-enrollment, provides playbook for appraisers, provides appraiser training, connects appraisers to participants, adjusts appraisal scope as necessary, assures appropriate appraisal and appraiser consistency, collects appraisal data, trends data, provides deidentified data to participants / steering committee, manages appraisal issues, adjust approach based on feedback from steering committee and stakeholders
Pilot SteeringCommittee FDA MDIC Appraisers
Participating DeviceManufacturers
CMMI® InstitutePMO
These benefits reduce the burden and disruption of audits, accelerate the review and approval process for changes, and shift resources to innovation and improvement.
Pilot SteeringCommittee FDA MDIC Appraisers
Participating DeviceManufacturers
CMMI® InstitutePMO
Onboarding and Scoping
• Kick-Off Call with PMO
• Intake Call & Follow-up Scoping Calls
• Sampling method for devices
• Identifying Appraisal Participants
• Determining Appraisal Schedule
• Plan reviewed with Org and PMO
• Submission of Quality Metrics to FDA
Lessons Learned: Inspection benefit moved until after appraisal is scheduled, allows FDA time to implement & ensures org engagementProgram kick-off meetingTraining created to expand appraiser bench
Pilot SteeringCommittee FDA MDIC Appraisers
Participating DeviceManufacturers
CMMI® InstitutePMO
Appraisal Execution
• 5 days, 2+ appraisal team members
• CMMI and Medical Device Experience
• Includes tour of the facility for context
• Interview-based, no doc review
• Conversations with those who do work
• Validation sessions
• Final results belong to organization
• Org identifies weaknesses to focus on
Lessons Learned: Intake/scoping templates, standardized heatmap tool, post-appraisal survey, unique individual journey, mentorship
Pilot SteeringCommittee FDA MDIC Appraisers
Participating DeviceManufacturers
CMMI® InstitutePMO
QA, Trending of Data
• Data submitted to PMO for QA
• Data aggregated for submission to FDA
• Data de-identified for trending
---
• Submission of Quality Metrics to FDA
• Metrics dashboard for individual
improvement journey
• Identification of common metrics for
de-identified trending
Lessons Learned: collection of quality metrics shifted to FDA activity
What data is shared?To Appraised Organization and CMMI PMO
Full scorecard and specific results against model
What data is shared?To FDA
Process area results and overall score for an organization• Elevating Industry
The long term “next steps”…
• What: Industry baseline for organizations to benchmark improvement journeyHow: trend participant results & quality performance metrics over time
Not actual results
Future: How we can use these results
Pilot SteeringCommittee FDA MDIC Appraisers
Participating DeviceManufacturers
CMMI® InstitutePMO
Post-Appraisal Activities
• Quarterly, 1-2 hour digital meetings
between organization and LA
• Orgs provide progress report, LA
provides feedback on activities
• Conversations inform second appraisal
---
• Submission of Quality Metrics to FDA,
org can change after appraisal results
• Individual improvement journey,
understand why metrics have changed
Lessons Learned: standardized template for check point activity, 2nd appraisal, variable submission of quality performance metrics
• FAQs: http://cmmiinstitute.com/MedicalDeviceFAQ
• October 10th FDA Public Meeting Presentation: https://www.fda.gov/MedicalDevices/NewsEvents/WorkshopsConferences/ucm568069.htm
• November 15th MDIC Meeting Presentation: http://mdic.org/cfq/case-for-quality-public-forum-presentations/
Additional Information
24 Boston Scientific Confidential -- For Internal Use Only. Do Not Copy, Display or Distribute Externally
CMMI MDDAP AppraisalFDA Case For Quality
Boston Scientific Experience
Corporate Sponsor: Conor Dolan, VP Quality Systems, BSCSite Sponsor: Joe Friedrich, Quality Director, Arden Hills, MN
25 Boston Scientific Confidential -- For Internal Use Only. Do Not Copy, Display or Distribute Externally
Quality Policy
BSC Culture of Quality
Quality Organization MissionWe exceed expectations with customer-centric quality
solutions that transform patient lives
26 Boston Scientific Confidential -- For Internal Use Only. Do Not Copy, Display or Distribute Externally
MDDAP Overview and BSC Status
BSC Arden Hills (since Jan 10)
BSC Clonmel(as of Feb 19)
27 Boston Scientific Confidential -- For Internal Use Only. Do Not Copy, Display or Distribute Externally
Why did BSC decide to participate?
• Use a defined maturity model and practice capabilities rather than a compliance approach, to understand specific areas our site can work to improve.
• Operationalize significant regulatory benefits as it relates to PMA submissions, manufacturing site change submissions, and the 30 day regulatory submission process.
• Create a relationship with an external partner organization that is proactive and focused on continuous improvements in overall product quality and patient outcomes.
• Opportunity to be part of shift towards more patient-centered culture across med device industry and FDA.
28 Boston Scientific Confidential -- For Internal Use Only. Do Not Copy, Display or Distribute Externally
Appraisal Preparation and Execution
Appraisal Planning
•Scoping: Three 2 hour meetings
•Defined Scope –Practice Areas
•Defined Duration
•Identified Interviewees
•Multi-site Approach: Two 1 hour meetings
•Site Coordinator: Lead Appraiser –Five 1 hour calls
•Locked Appraisal Plan and Logistics
Logistics
•Two Appraisal Teams
•Two conferences rooms
•1 Board Room•Internet and
Projection•On-site Lunch
Resources
•31 Participants•1 Site
Coordinator•8 Hours Active
Engagement / Interviews Over 3 Days
•4 Hours Verification
•2 Hours Results•7:30 to 5:30
Daily
Support
•No Backroom•No SOPs•No Records•No Laptops
29 Boston Scientific Confidential -- For Internal Use Only. Do Not Copy, Display or Distribute Externally
Product Sampling
For the appraisal team• The Product Sampling is a means to show appropriate site coverage
Product Line Differences Process Differences Device Class Differences
For the appraisal site• The product sampling is used to identify a range of individuals who have
participated on given sample projects. • These projects are used as a frame of reference for participants to
provide examples that support practice area questions.
30 Boston Scientific Confidential -- For Internal Use Only. Do Not Copy, Display or Distribute Externally
CMMI Appraisal vs. FDA Inspection
CMMI Appraisal FDA Inspection• Capability maturity model appraisal of
business practices and processes.
• Focused on eleven (11) CMMI Model practice areas.
• Appraisers conduct group interviews of ‘doers’ responsible for work products.
• Appraisers engage in discussions to truly understand how the business operates relative to best practices.
• Many participants outside of QA.
• Minimal disruption to site resources and no need for Back Room / Front Room
• Weaknesses are opportunities to improve business process.
• Inspection of quality system procedures and records.
• Focused on 21 CFR Part 820 requirements.
• Inspectors interrogate Quality leaders, process experts, and record owners.
• Inspectors review procedures, records, storyboards looking for evidence of noncompliance to regulations.
• Few participants outside of QA.
• Large support team with Back Room / Front Room, streams, scribes, etc.
• 483s require escalation to CAPA and formal agency response.
31 Boston Scientific Confidential -- For Internal Use Only. Do Not Copy, Display or Distribute Externally
CMMI vs. FDA: Participant Guidance
CMMI Appraisal FDA Inspection• Be open in answering questions and it’s
ok if you’re not 100% sure.
• It’s ok to volunteer information that may be tangentially related to the question.
• Highlight improvements made over time, as well as future plans if appropriate.
• Explain other systems and tools used to do your work. For example, PLCP, Business Essentials, VIPs, CI Boards, etc.
• No need to prepare for questions. Just respond as best you know based on you experience and perspective.
• Do not bring your laptop, exhibits, or documentation to your interview. Is it just an open discussion.
• Only answer questions asked. If you don’t know the answer, don’t guess.
• Don’t volunteer information that doesn’t pertain to question asked.
• Don’t discuss improvement opportunities or future plans.
• Don’t deviate from what is documented the quality system, i.e. policies, procedures, records, etc.
• Be prepared to speak to any pertinent documentation. Facilitators will ensure the right people answer questions.
• Bring in procedures and records as appropriate. A laptop may be needed to view certain electronic files.
32 Boston Scientific Confidential -- For Internal Use Only. Do Not Copy, Display or Distribute Externally
CMMI vs. FDA: Resource Allocation
CMMI Appraisal – 5 Days
Prep Appraisal TotalSite Coordinator
80 Hrs 20 Hrs 100 Hrs
Participants 60 Hrs 180 Hrs 240 Hrs
Total 140 Hrs 200 Hrs 340 Hrs
2 Day FDA Inspection 1370 Hrs
33 Boston Scientific Confidential -- For Internal Use Only. Do Not Copy, Display or Distribute Externally
BSC Feedback on CMMI Appraisal
Positives
• Appraiser Accessibility• Appraiser Competency• Valuable Participant Briefing• Structured & Organized• Valuable use of Participants Time• Conversational, Engaging and Positive Group Interviews• High-Level Questions Drove Good Discussion• Valuable verification sessions ensured site ownership
Improvements
• Initial / immediate definition / understanding of approach would be helpful• Expand Practice Area briefing to better define what will be covered• High-Level questions seemed vague. More up-front education on the level of detail.• Terminology at times was difficult, but rephrasing helped.• Pilot metrics may be difficult to define and implement.
34 Boston Scientific Confidential -- For Internal Use Only. Do Not Copy, Display or Distribute Externally
So what have we learned?
• Our practices lined up well with the CMMI model overall• “Weaknesses” in areas of project and resource management• Cross-functional groups are engaging in healthy conversations about our business practices and
improvement opportunities• Need to gain alignment with other business functions on value proposition to help drive change
from appraisal results• Many stakeholders remain skeptical of the pilot
– How will the CDRH benefits actually work and will they truly materialize?– Will pilot investment (dollars and time) be worth it?– What risks do we bring on by participating?– Will quarterly metric reviews be value added?– What will be expected of us down the road to maintain benefits?
• This is a pilot… the plane is being built as we’re flying it!
Case for Quality
Program PilotCisco Vicenty
Program Manager, Case for Quality
Office of Compliance, CDRH
February 27, 2018
35
www.fda.gov
For Discussion
What can it show?
What are we seeing?
Lessons Learned
Next Steps
Comments to the Docket
Engagement and Collaboration
36
www.fda.gov
What can it show?
• Shows performance of the organization beyond establishing compliant processes
• Quickly identifies strengths at an organization or industry level and allows pilot to shift focus
• Allows FDA to engage collaboratively where common opportunities for improvement exist
• Allows organizations to benchmark their execution, acknowledge success, and focus improvement
37
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%EST
PLAN
MC
CM
MPM
RDMPQA
GOV
II
PI
TS
Benchmark Performance
Manufacturer A Manufacturer B Manufacturer C Manufacturer D
This data is not actual data but a representative demonstration.Results scores were randomly generated.
www.fda.gov
What are we seeing?
38
• Early trends• Strengths• Opportunities
• Performance is not necessarily dependent on product classification
• Some participants are scoring highly across scoped practice areas.
• Collaboration and sharing among participants
• FDA is asking different questions
Opportunities• EST – Estimating
• Estimation of the size, effort, duration, and cost of the work and resources needed to develop, acquire, or deliver the solution
• II – Implementation Infrastructure• Ensure that the processes
important to an organization are persistently and habitually used and improved
Strengths
• TS - Technical Solution• Cost-effective design and
solution that meets customer requirements and reduces rework
• PI – Product Integration• Assemble and deliver the
solution that addresses functionality and quality characteristics
www.fda.gov
Lessons LearnedLessons
• Small manufacturers are experiencing cost concerns
• High-maturity in the medical device space for certain practices. Review of these areas may not be value-added
• Potential for manufacturers to de-enroll. Need to adjust the start of FDA Modifications
• FDA internal inspection planning process has long lead times. Needed more active mechanism for communication
• Metrics submission and discussion was confounding the appraisal
Actions• CMMI has been developing solutions to
manage costs. Participants are also providing additional options.
• Activity in-process to modify scope of appraisal and practice areas to provide value
• Unanticipated constraints impacted some enrollees. FDA is modifying start of modifications until after the scoping is confirmed by CMMI
• FDA has implemented an active notification system to communicate enrolled sites among the planning organizations
• FDA will work individually with participant during scoping phase to discuss the metrics and submission
39
www.fda.gov
Next Steps• Continue scheduled appraisals• Evaluate current learnings and adjust
pilot if necessary• First quarterly check-ins will start in
March• First modified submissions to begin
and rolled out to other participants• Expand in-scope practice areas for
increased application in 510(k) space• Evaluate metrics submitted and
correlations to progress
40
Reward the journey, not the destination
www.fda.gov
Comments on the program
• Docket Comments• Received comments from AdvaMed and
the BRIDGE Coalition• FDA is working to incorporate insights
from those comments into the pilot and into the next steps
• Current Efforts• Transparency• Increasing application to 510(k) • Common vocabulary
41
www.fda.gov
Engagement and Collaboration
42
• Voluntary Pilot was announced on December 28, 2017 and is actively enrolling participants
• Engagement and feedback are critical• For additional information, enrollment, or feedback
• http://mdic.org/cfq/• http://mdic.org/cfq/enroll/• [email protected]
• For participants• For any issues or concerns through out the pilot contact
www.fda.gov
Questions?
43
Thank You!
Questions and Discussion• Submit your questions via the chat box
• Recording and slides will be available on MDIC’s web site
Coming this springPlease plan to join MDIC, FDA and your industry colleagues at Xavier University on May 1st for a face-to-face workshop to discuss the preliminary outcome and future direction of the pilot study.
More information and registration information at: http://www.xavierhealth.org/case-for-quality-workshop.