#PhUSE Standard Scripts Project 2 2014 - Proposal for Qualification of Standard Scripts.
-
Upload
lilian-mills -
Category
Documents
-
view
225 -
download
1
Transcript of #PhUSE Standard Scripts Project 2 2014 - Proposal for Qualification of Standard Scripts.
#PhUSE
Main Sections
Summary of prior proposal– Concepts, definitions & metadata– Test data considerations– Heavy vs. Light qualification
Updated proposal
#PhUSE
Proposal from 2013
• Anyone can submit a script, according to a check list• Categorize scripts by complexity– Complexity:
low, medium, high, software
– Output:
tabulated data, analysis data, table, figure, listing
• Metadata for script should indicate– Type of output:
tabulated data, analysis data, table, figure, listing
– Study design:
parallel, crossover, etc
– State of qualification:
?
#PhUSE
Proposal from 2013
• Test data– Overall project should have minimum test data (SDTM & ADaM)– Scripts can propose new test data, must pass (Data fit? Open CDISC?)– Share program to produce test data, never binary test data
• 2 levels of qualification– Light vs. Heavy: according to complexity of script & output– Common elements include
• header• adhere to good programming practices• clear scope of script (e.g., study design(s))• test data matche scope & pass "FDA Data Fit" assessment (Open CDISC?)• documentation available for: usage, inputs, outputs, dependencies
#PhUSE
Proposal from 2013
• Heavy qualification– Beta package includes:
minimal elements for contribution• Specification & Documentation (could be in pgm header)• Test data (Data Fit? or Open CDISC?)• Tests & Expected results defined• Peer Review: GPP, Specs & Docn reviewed, Tests reproduced
– Draft• Write qualification plan, Review tests for completeness/suitability (e.g.,
Branch testing – are all conditional blocks/combos tested?)– Test
• Peer Review: Write qualification report, incl. log/output from tests– Final
#PhUSE
Proposal from 2013
• Light qualification– Beta package includes
skip if >1 yr production use without ERROR– Draft
minimal elements for contribution• Specification & Documentation (could be in pgm header)• Test data (Data Fit? or Open CDISC or other, as appropriate)• Tests & Expected results defined• Peer Review: GPP, Specs & Docn reviewed, Tests reproduced• Write qualification plan, Review tests for completeness/suitability (e.g.,
Branch testing – are all conditional blocks/combos tested?)– Test
• Peer Review: Write qualification report, incl. log/output from tests– Final
#PhUSE
Proposal through CSS 2104Peer Review Checklist Heavy Light
Requirement specification X ?
Documented or perhaps only documented in header X
User Guide X X
SDTM/ADaM used in input/output X X
Open CDISC validator or Data Fit used to check input/output X X
GPP in source X X
Run according to Requirement specification X ?
Tested by qualification plan, tests & results all Peer reviewed X ?
Tested by End users X ?
Robust without red errors in contributor's production environment X X
Robust and used in FDA (other) scripts repository, ranked ****** X
#PhUSE
Main Sections
Summary of prior proposalUpdated proposal– Objectives– Definitions: Qualification, States, Roles– Metadata and Test data– State Transitions
#PhUSE
Proposal 2014 - Objectives• For End-users
– Clear overview of resources available, incl. purpose & state of each– Inspire confidence from first user experience– Ease-of-use:
• clear messaging from scripts• Reproducible results in users’ own environments• Consistency of scripts, learning first one makes remaining familiar
– Ease of converting users to contributors
• For Contributors & Standard Scripts Team– Ease of contribution: Modularize & standardize workflows & checklists– Modularize core components (e.g., FUTS for dependency checking?)– Automate testing, issue identification (e.g., concept similar to Spotfire/R compatibility)– Centralize & consolidate information & results
#PhUSE
Qualification Proposalmeaningful terms in blue
http://www.phusewiki.org/wiki/index.php?title=File:WG5_P02_Proposal_-_2014.pptx
• Qualification Instructions (see embedded template ð) – Certification applies to new scripts and functionality– Confirmation applies to updates of existing script
• States:
Contributed, Develop, Review, Released• Roles
– Contributor: Anyone with appropriate skills & interests– Developer: A volunteer in CSS Working Group 5, familiar with our objectives**– Tester: A volunteer in CSS WG 05 who is familiar with our objectives**– Environment Tester: Anyone in community who is able to set up automatic test
replication in their work environment– Reviewer: Author of white papers, designers of script targets**
** suggests a quick-start onboarding page in CSS Phusewiki
Microsoft Excel Worksheet
#PhUSE
Qualification Proposal• Metadata for scripts should indicate:
YML proposal– Whitepaper & output ID uniquely identify the target
<Script: Source, Title, Target>
– Programming language & version
<Language: name, version>
– Type of output:
<Script: Type>
– Study design:
<Script: StudyDesign>
– State of qualification <Stage>:
<Qualification: Stage>
– OS is not included, since should be indicated in OS compatibility report• Test Data requirements
– available as a program or script (text, not binary)– based on expected standards (SDTM, ADaM)– data requirements clearly & accurately specified for each script– include expected results from these data for defined tests/checks
#PhUSE
Qualification Proposal
• Transitions
"Contributed" is the original State of all scripts– to Develop, checklist includes
by Developer & Reviewer• R & D confer on suitability of contribution. Suitable starting point?
[ may require analysis details, specs, from contributor ]• D reviews components (checklist to be completed)• D works with Contributor to complete minimum components
[ including Test Data and Coverage of defined tests ]• D adds standard parameter, dependency checking• D writes Qualification of CSS scripts.xlsx (see template embedded above)
– to Review, checklist includes
by Tester• Review Qualification instructions, consider coverage of tests• Execute Qualification instructions• Work with Developer to complete execution successfully
#PhUSE
Qualification Proposal
• Transitions continued– to Released by Tester & Environment Tester & Reviewer
• T updates reference test outputs from certification/confirmation• E updates & executes local tests (posting PASS/FAIL results)• R confirms script output matches intention• R confirms qualification process covers important elements and
considerations. • R also provides user (rather than technical) feedback?• Achieve “Released" state when all tests in all test environments PASS (i.e.,
match outputs that T has certified and/or confirmed) and that R agrees that target is achieved
#PhUSE
Qualification Proposal• Efforts Required
– Top priority• Finalize Qualification states, roles, workflow, checklists, and templates• Code criteria for "acceptable" (link to GPP, PhUSE)• Test definition, Test confirmation (also from White Paper team)• Test data
– Next priorities • Design test structure in google code• Develop scripts that will allow Environment Testing• Develop general components (e.g. parameter, dependency checking)• Identify Environment Testers based on
– Host environment– SAS or R version
• Identify opportunities to automate qualification. E.g.,– Environment Testers to post results back as machine readable– Script green-light/red-light qualification matrix on Phusewiki