Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User...
Transcript of Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User...
![Page 1: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/1.jpg)
CROS NT srl
Contract Research Organisation
Clinical Data Management
Statistics
Dr. Paolo Morelli, CEO
Dr. Luca Girardello, SAS programmer
Best Practice in SAS programs validation. A Case
Study
![Page 2: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/2.jpg)
AGENDAAGENDA
Introduction
Program Verification: a Business Approach
Program Verification: some case studies
![Page 3: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/3.jpg)
FACTS FACTS aboutabout CROS NTCROS NT• Headquarters in Verona (Italy)
• Founded in 1993
• Offices in Milan and Munich
• 40 employees
• Data Management, Statistical, PhV and
hosting services
• Services to Pharma, Biotech and CROs
• Cooperation with Universities of Padua, Bologna, Milan
![Page 4: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/4.jpg)
IntroductionIntroduction
• Topic of the presentation: how to maximize the quality of programming while minimizing the time to verify program.
• In the first part of the presentation we will discuss about thebusiness part:
What is program verification?Why program verification is necessary?When is program verification done?Who performs program verification?How does the verification process work?
• In the second part of the presentation we will discuss about a case study
![Page 5: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/5.jpg)
What is program verificationWhat is program verification
• Making certain that the program does what it is supposed to do, producing a documented evidence of this
![Page 6: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/6.jpg)
Why program verification is necessaryWhy program verification is necessary
• The aim of SAS validation in pharmaceutical research area is that end-users will produce high quality programs that fit the purpose for which they are designed and provide accurate results with a style that they promote:
•Reliabity
•Efficiency
•Portability
•Flexibility
•Ease of use
![Page 7: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/7.jpg)
When is program verification doneWhen is program verification done
• Program verification should performed as soon after the development of the SAS code, before putting the “product” in production
• Development and production environment should be clearly defined;
•Audit trail of program changes should be present as soon the program is released to production
![Page 8: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/8.jpg)
Who performs program verificationWho performs program verification
The SAS programmer who create the code should perform basic testing and follow coding rules, like:
• Error log search
• Warning evaluation
• Comments on critical steps
• Comments on Macro usage
• Details of the SAS program (datetime of creation, SAS programmer name, dataset used, datetime of verification, Name of second SAS programmer, etc)
•It should be emphasized to perform then a program verification by a second SAS programmer
![Page 9: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/9.jpg)
How does the verification process workHow does the verification process work
Biostatistician creates specs then Submits request
SAS developer produces TLGsThen submits verification request
Quality Control programmer verifies results
Interactive Process
Interactive Process
![Page 10: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/10.jpg)
Different Verification ProceduresDifferent Verification Procedures•SOP should define different verification procedures.üIndependent programmingüReviewing resultsüRandom review of results üVisually verify code
•Some of them should mandatory, other optional.
•The Document Containing the programming specs (for example the SAP) should define which approach to follow, illustrating program verification techniques (for example using alternative SAS programming procedures)
•The determination of the level of validation should follow a risk-based model. The key is to determine the effect on the process if the program does not produce the desired result.
![Page 11: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/11.jpg)
Error TypesError Types
• Business strategy should identify common ‘error types’ found in:ü Statistical tablesü Listingsü Graphsü Data analysis filesü Header section of SAS programsü Bad programming specifications
• Metric report related to error type should be analyzed in order to perform preventive action correction
![Page 12: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/12.jpg)
Specific CDISC SDTM Validation specs Specific CDISC SDTM Validation specs ––Metadata LevelMetadata Level
•Verifies that all required variables are present in the dataset
•Reports as an error any variables in the dataset that are not defined in the domain
•Reports a warning for any expected domain variables which are not in the dataset
![Page 13: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/13.jpg)
Specific CDISC SDTM Validation specs Specific CDISC SDTM Validation specs --Metadata LevelMetadata Level
•Notes any permitted domain variables which are not in the dataset
•Verifies that all domain variables are of the expected data typeand proper length
•Detects any domain variables which are assigned a controlled terminology specification by the domain and do not have a format assigned to them
![Page 14: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/14.jpg)
SAS Programming Rules when SAS Programming Rules when validatingvalidating
Ø Emphasizing well commented programs.
Ø Macro in order to use programs repeatedly to verify different programs (re-usability)
Ø Using alternative SAS programming procedures when validating.
Ø Define a workflow if error are identified
![Page 15: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/15.jpg)
How to optimize the processHow to optimize the process
Good specs & Good standards & Good training
= Good programming results
![Page 16: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/16.jpg)
A Case A Case StudyStudy
![Page 17: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/17.jpg)
ExampleExample ofof DerivedDerived DatasetsDatasetsValidationValidation (1/4)(1/4)
PROC COMPARE
Compare original derived datasets
versus validation derived datasets
“Second Programmer” programsall derived datasets
“First Programmer” programsall derived datasets
![Page 18: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/18.jpg)
ExampleExample ofof DerivedDerived DatasetsDatasetsValidationValidation (2/4)(2/4)
The COMPARE Procedure Comparison of WORK.LISTING with WORK.VALIDATION (Method=EXACT) Observation Summary Observation Base Compare ID First Obs 1 1 pt=121 First Unequal 79 79 pt=201 Last Unequal 79 79 pt=201 Last Obs 89 89 pt=212 Number of Observations in Common: 89. Total Number of Observations Read from WORK.LISTING: 89. Total Number of Observations Read from WORK.VALIDATION: 89. Number of Observations with Some Compared Variables Unequal: 1. Number of Observations with All Compared Variables Equal: 88.
proc compare base=listing compare=validation listbase listcomp;
id pt; run;
![Page 19: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/19.jpg)
Values Comparison Summary Number of Variables Compared with All Observations Equal: 3. Number of Variables Compared with Some Observations Unequal: 1. Total Number of Values which Compare Unequal: 1. Maximum Difference: 1.
Variables with Unequal Values Variable Type Len Label Ndif MaxDif age NUM 8 AGE (years) 1 1.000 Value Comparison Results for Variables _________________________________________________________ || AGE (years) || Base Compare pt || age age Diff. % Diff _______ || _________ _________ _________ _________ || 201 || 41 40 -1.0000 -2.4390 _________________________________________________________
ExampleExample ofof DerivedDerived DatasetsDatasetsValidationValidation (3/4)(3/4)
![Page 20: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/20.jpg)
The COMPARE Procedure Comparison of WORK.LISTING with WORK.VALIDATION (Method=EXACT) Observation Summary Observation Base Compare ID First Obs 1 1 pt=121 Last Obs 89 89 pt=212 Number of Observations in Common: 89. Total Number of Observations Read from WORK.LISTING: 89. Total Number of Observations Read from WORK.VALIDATION: 89. Number of Observations with Some Compared Variables Unequal: 0. Number of Observations with All Compared Variables Equal: 89. NOTE: No unequal values were found. All values compared are exactly equal.
ExampleExample ofof DerivedDerived DatasetsDatasetsValidationValidation (4/4)(4/4)
![Page 21: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/21.jpg)
ExampleExample ofof TablesTables ValidationValidation (1/3)(1/3)
“First Programmer” programsall tables applying the set of
layout specifications and saves outputs in Word
“Second Programmer” programsall tables avoiding to add
additional SAS code to controloutput
Compare of outputs
![Page 22: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/22.jpg)
ExampleExample ofof TablesTables ValidationValidation (2/3)(2/3)________________________________________________________________ Tmt A Tmt B ________________________________________________________________ Age (years) n 41 48 Mean (SD) 51.44 (10.39) 52.10 (11.00) Median 55.00 55.00 Min - Max 30.00- 66.00 27.00- 71.00 Gender Female 14 (34.15%) 21 (43.75%) Male 27 (65.85%) 27 (56.25%) ________________________________________________________________
First Programmer -Output in Word
Second programmer -Output SAS
proc means data=demog n mean stddev median min max;
var age; by tmt; run;
![Page 23: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/23.jpg)
________________________________________________________________ Tmt A Tmt B ________________________________________________________________ Age (years) n 41 48 Mean (SD) 51.44 (10.39) 52.10 (11.00) Median 55.00 55.00 Min - Max 30.00- 66.00 27.00- 71.00 Gender Female 14 (34.15%) 21 (43.75%) Male 27 (65.85%) 27 (56.25%) ________________________________________________________________
First Programmer -Output in Word
Second programmer -Output SAS
proc freq data=demog; tables gender*tmt; run;
ExampleExample ofof TablesTables ValidationValidation (3/3)(3/3)
![Page 24: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/24.jpg)
ExampleExample ofof ListingsListings ValidationValidation (1/2)(1/2)“Second Programmer” prints
derived datasets in SAS“First Programmer” programsall listings applying the set of
layout specifications and saves outputs in Word
Compare listing output in Word
versus output in SAS of derived dataset
![Page 25: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/25.jpg)
ExampleExample ofof ListingsListings ValidationValidation (2/2)(2/2)
Listing 1 Demographic Characteristics Subject ID Gender Age Race _______________ _______ ____ _____ 121 M 50 3 122 M 34 3 123 F 58 3 124 M 64 3 125 M 57 3 126 F 64 3 127 M 39 3 128 M 55 2 129 M 41 3 130 M 44 3 131 M 32 3 132 M 37 3 133 M 61 3 134 F 56 3 135 M 34 3 136 M 34 3
Listing Output in Word Print of Derived Dataset
![Page 26: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/26.jpg)
ExampleExample ofof RegistrationRegistration ErrorsErrors
![Page 27: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/27.jpg)
Programming41%
Specification14%
Layout45%
MetricsMetrics on on ProgrammingProgramming ErrorsErrors
Selection of Variables
14%
Calculation of variables
20%
SAS Programming
66%
Specification not detailed
40%
Wrong interpretation
of specification
60%
Output Writing
56%
Output Structure
30%
Display Variables
14%
![Page 28: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/28.jpg)
ExamplesExamples ofof ErrorsErrors
• Layout
Writing of a note in tableIncorrect: “Percentages are calculated number of patients”
Correct: “Percentages are calculated on number of patients”
![Page 29: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/29.jpg)
ExamplesExamples ofof ErrorsErrors
data age; set demog; if age<20 then age_c=1; else if 20<age<40 then age_c=2; else if age>=40 then age_c=3; run;
• Programming
data age; set demog; if age<20 then age_c=1; else if 20<=age<40 then age_c=2; else if age>=40 then age_c=3; run;
![Page 30: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/30.jpg)
ExamplesExamples ofof ErrorsErrors
• Wrong interpretation of specification
Note of a table (in SAP):“Note 1: Only patients with all value for primary analysis are included in the table.”
In SAS Program:In the table, all patients are included
![Page 31: Best Practice in SAS programs validation. A Case Studycdiscportal.digitalinfuzion.com/CDISC User Networks...Best Practice in SAS programs validation. A Case Study AGENDA Introduction](https://reader031.fdocuments.in/reader031/viewer/2022022523/5b3506af7f8b9abc218cd161/html5/thumbnails/31.jpg)
ThankThank youyou forfor youryour attentionattention
QuestionsQuestions??