Validation and Quality Concepts in Open Source Clinical...
Transcript of Validation and Quality Concepts in Open Source Clinical...
![Page 1: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/1.jpg)
Validation and Quality Concepts in Open Source Clinical Software: Not an Oxymoron
Brian Shoemaker, Ph.D.Principal ConsultantShoeBar Associates
![Page 2: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/2.jpg)
2
Open Source Software Quality: Not an Oxymoron
Could open-source approach be an alternative after all?
Contradictions are actually superficialPractical open-source implementation requires
disciplineStill other quality / validation opportunities to
exploreModel is win-win because all participants have an
incentive to guarantee quality
![Page 3: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/3.jpg)
3
Could OSS be an alternative after all?
Cathedral: • Close control of all
output• Proprietary product• Restricted team,
specific assignments
Bazaar:• Ideas from anywhere• Access to source for
all• Any developer can
“scratch an itch”
My own reaction to the concept was skepticism
![Page 4: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/4.jpg)
4
Could OSS be an alternative after all?
• Often, S/W purchases only follow protracted evaluation• Just planning / executing IQ is a project in itself• How much do we learn from vendor audits?• No one likes the CSV team when we find errors;
no one appreciates us when we don’t
Long-standing issue: CSV = costly overhead
Though this topic will largely be left open, it forces us to consider whether a different approach to the SDLC could accomplish more
![Page 5: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/5.jpg)
5
Could OSS be an alternative after all?
• NOT an undisciplined hackers’ free-for-all• NOT a collection of tangled code with no rules• NOT a case of constantly “shifting sands”• NOT a nebulous, undefined end product without focus
Open Source vs. S/W Quality / Validation –first understand what it isn’t
Our focus needs to shift to what OSS is, to understand where validation fits in
![Page 6: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/6.jpg)
6
Open Source Software Quality: Not an Oxymoron
Could open-source approach be an alternative after all?Contradictions are actually superficialPractical open-source implementation requires
disciplineStill other quality / validation opportunities to
exploreModel is win-win because all participants have an
incentive to guarantee quality
![Page 7: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/7.jpg)
7
Contradictions are Superficial
• Usage defined in the Study Protocol and controlled by SOPs• Provide mechanisms to retain and preserve data• Safeguards for limited access, keeping of audit trails, and date/time
stamps of records• External safeguards to limit access• Features to minimize data entry errors• Developed, maintained, and used by users with documented
training and skills for their tasks
Concern of CSCT Guidance: functioning system, not how system was developed
Data must be Attributable, Legible, Contemporaneous, Original, and Accurate (ALCOA)
![Page 8: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/8.jpg)
8
Contradictions are Superficial
• GPSV does not recommend one specific software lifecycle model
• Rather, GPSV calls for specific activities and outputs:requirementsdesigncodeunit/integration testsystem testacceptance
For the “how” of development, we need a CDRH document: General Principles of Software Validation
![Page 9: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/9.jpg)
9
Contradictions are Superficial
• In OSS, key concepts are Responsibility augmented by Community (a new idea)
• We've always known: The waterfall leaks(even Winston Royce knew this!)
![Page 10: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/10.jpg)
10
Open Source Software Quality: Not an Oxymoron
Could open-source approach be an alternative after all?Contradictions are actually superficialPractical open-source implementation requires
disciplineStill other quality / validation opportunities to
exploreModel is win-win because all participants have an
incentive to guarantee quality
![Page 11: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/11.jpg)
11
Implementation Requires Discipline
Consider models for Open Source development (1)
Managing Organization’s Version Control System
“Contributing Community” Model
Review, Evaluate
Main Code Base
Contributors’ Area
D1
D2
D3
Dn
. . .
![Page 12: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/12.jpg)
12
Implementation Requires Discipline
Consider models for Open Source development (2)
User OrganizationManaging Organization’s Version Control System
“User Customization” Model
Replicate (separate branch)
Main Code Base
User’s Code Base(w/ custom features)
D1
D2
D3
Dn
. . .
![Page 13: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/13.jpg)
13
Implementation Requires Discipline
• Online config management: code augmented, version control maintained
• Config mgmt, bug tracking and other tools (also OSS) can be tightly integrated
• Forums / Blogs allow presenting/discussing ideas• Public bug lists keep the community informed of issues• Wikis allow the supporting information structure to grow• Readily available coding standards inform contributors
what's expected
Effective use of technology makes sharing possible
![Page 14: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/14.jpg)
14
Implementation Requires Discipline
Openness invites scrutiny
. . .Therefore we expect the managing organization to be careful and complete –with documentation as well as code
![Page 15: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/15.jpg)
15
Open Source Software Quality: Not an Oxymoron
Could open-source approach be an alternative after all?Contradictions are actually superficialPractical open-source implementation requires disciplineStill other quality / validation opportunities to
exploreModel is win-win because all participants have an
incentive to guarantee quality
![Page 16: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/16.jpg)
16
Other Open Source Opportunities?
• Requirements development –Could the blog be taken one step further?
• Code review? Invite community developers to read documents and code for a new element, and post questions.
• Test writing –What if the community were invited to post test procedures for new features?
• Guiding concept: fresh eyes see an issue in a different way
![Page 17: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/17.jpg)
17
Open Source Software Quality: Not an Oxymoron
Could open-source approach be an alternative after all?Contradictions are actually superficialPractical open-source implementation requires disciplineStill other quality / validation opportunities to exploreModel is win-win because all participants have
an incentive to guarantee quality
![Page 18: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/18.jpg)
18
Win-win model: all participants have incentive to guarantee quality
• No substitute for "making sure the software does what it's supposed to do"
• No one would seriously put out CT S/W without supporting material (per GPSV)
• Nor would any organization implement that S/W without doing validation homework
![Page 19: Validation and Quality Concepts in Open Source Clinical ...shoebarassoc.com/uploads/1/4/8/8/14885450/dia_jun... · Open Source Software Quality: Not an Oxymoron Could open-source](https://reader034.fdocuments.in/reader034/viewer/2022042713/5fab2126bc5fb561f75e98f4/html5/thumbnails/19.jpg)
19
References
FDA, Guidance for Industry: Computerized Systems Used in Clinical Investigations (May 2007),http://www.fda.gov/cber/gdlns/compclintrial.htm
FDA, General Principles of Software Validation (January 11, 2002), http://www.fda.gov/cdrh/comp/guidance/938.html
Raymond, Eric S., The Cathedral and the Bazaarhttp://www.catb.org/~esr/writings/cathedral-bazaar/cathedral-bazaar/
Royce, Winston W., “Managing the development of large software systems: Concepts and techniques,” in: Proceedings, IEEE WESCON (August 1970). (Waterfall method)