WG5 Subgroup2Creation and validation of scripts

Kevin Kane, PHASTARJoy Li, FDA

Aims

• To create a list of proposed actions• To start to produce a flowchart that

documents the process of creating and validating scripts

• To raise list of questions, issues and points to note for other WG5 subgroups and other WGs

Scope

• Other subgroups– What scripts are required– System to house scripts

• Leverage existing standards (CDISC)• Try not to limit to specific scripting languages• Well documented and validated• Can be for outputs or data transformation

Scope

This is only a start of a framework. Much work will need to be done in the future. We hope you will continue to work with us.

Let’s make as much progress as we can – we won’t be able to finalise all details.

Script Creation

• Once a script is loaded, the original author is stored as metadata but does not have any further rights or responsibilities

• Basic set of programming standards would be useful. If they are too detailed, may conflict with individual organisations

• We should develop standard templates for specifications and user guides etc

• Investigate “V Model” further for development process• For minor changes, this should not be a separate script –

should be added as an option• Encourage backward compatibility but not an absolute

requirements

Metadata

• Program name• Language• Program version (auto?)• Platform• Purpose• SDTM/ADaM version/NA

(dropbox)• Keywords• Original Author (auto)• Usage counts• Ratings/feedback• Validation status

• Assumptions• Inputs• Outputs• Requirements• Comments/notes• Reason for change• Bug flag (DB table?)• Current author• Language version • Validation documentation

Definition of validated script

• Script does what it says in specification• Specifications are required

– Design– Inputs– Outputs

• Test under various scenarios: these scenarios become assumptions

• Code review• Validation by experience is not enough• Website/wiki needs a disclaimer• ISSUE: What documentation is required for unvalidated scripts

Process for scripts to be validated

• Upload all validation documentation• Approval by moderator (committee?)• Meets all requirements on validation checklist

Can we learn from SAS Online help web pages?

Script governance - functions

• Approve scripts• Draft specs• Call for Scripts• Template specs• Guidelines• Validation checklist• Library management• Ratings management• Define metadata• Change management• Incentive management

Script governance – documentation required

• Guidelines for creating specs• Define metadata– Overlap between specs and metadata

• Web based database?• Template for user guide• Basic programming standards• Checklist for approval to validated state• Definition of requirements to consider a script

validated

Issues to pass to platform group

• Need to be able to review and comment on scripts. Ideally with quality rating

• Create and store multiple versions• Need scripts to be able to have different states: e.g. validated;

unvalidated; in development• Metadata e.g. program name; language; parameters; bug flag;

variables; outcomes; version number (need to decide list of metadata variables)

• Check-in check-out (not 100% defined- what happens if one person checks out for long time)

• Ability to have multi-person multi-function teams• Can we have a metadata database on a Wiki

Notes

• CDISC CTO suggested we find out about FDA Open Toolbox project. Norman Stockbridge and George Rochester from the FDA (ask Mat)

• We noted that different user groups may have different requirements e.g. FDA reviewers data exploration

Incentives:Results from brainstorm

• Maybe we don’t need any incentive• Encourage people to get a top rating leading to enhanced reputation• Platform records downloads – “most cited script”• Messages to “market”:-

– Reputation factor– This system can save organisations money– This is the same code that the FDA will use

• Could offer a PhUSE discount or award• FDA recommendation to use scripts• Airmiles/points system – bronze/silver/gold• Academic encouragement : get your methodology adopted• Confirm if we need any money. Ask PHARMA???