Out of Specification Results (OOS) A One Day Workshop

Out of Specification Results

(OOS)A One Day Workshop

Presented by: Karen S. GinsburyPCI Pharmaceutical Consulting Israel Ltd.

For IFFMarch 2007

PCI Pharmaceutical Consulting Israel Ltd 2/82

Purpose of Workshop

Understand Current Industry Practice asit relates to all aspects of handling:– Out of Specification– Unusual– Out of Trend Results

When is retesting legitimateWhen does testing have to stopWhen do the authorities need to be notified


Workshop Structure

09:00 – 10:30 10:30 – 11:00 11:00 – 12:30

12:30 – 13:30 13:30 – 15:00

15:00 – 15:15 15:15 – 16:30

Background and Overview: Barr and AbleCoffee BreakThe Making of the FDA GuideCase study: what NOT to doLaboratory investigation and checklistLunch BreakFDA Guide Continued:Extended investigation, retesting protocolQualitative and quantitative testsCase studyTea BreakCase Study and workshop wrap-upReporting results; reporting to regulators


PART 1 – BARR and ABLE

LitigationUSA regulatory environment around

late 1980’s through 1993Current regulatory environment in USAble - 2005


Some Definitions

What is an OOS Result?

What is a test result

Definition of reportable value

“A reportable value is the end result of the complete measurement method as documented. It is the value compared with thespecification, the values collected when the term replicates is used, the values used for official reports, and the values used for anystatistical calculation or analysis.”

Torbeck (February 1999, Pharmaceutical Technology)


Before Barr – Current Practice

Prior to 1993 and the court decision – it wasCOMMON practice to retestonceor in exceptionally good companies twiceand to release the batch if the retest result waswithin the specification

Companies had not really thought about thepractice

But then…nor had the regulators


The Barr Court Case and Judge Wolin

From the New York TimesFebruary 6, 1993, Saturday(AP); Financial Desk

COMPANY NEWS; Judge Rules On Barr LabsA generic drug manufacturer must recall batches ofsome of its medicines and stop distributingothers until the company completes studies of itsmanufacturing process, a Federal judge ruled onThursday. But United States District Judge Alfred M. Wolin refused a request by Federal pharmaceuticalregulators to order a complete shutdown


Barr and OOS

Faced with potential closure, the companytook FDA to court

The judge went into great details as to themeaning and implications of OOS results

The outcome: FDA draft guidance: 1998FDA final guidance: 2006


Barr: What happened in court

The judge heard experts onbehalf of FDA and Barr regardingthe practice of retesting

FDA wanted retesting to be bannedunder all circumstances

After a long hearing at which five industry experts, anFDA investigator, and several company employeestestified, Judge Alfred M. Wolin, U.S. District Judge forthe District of New Jersey, issued a 79-page opinion


The Barr Court Case 1993

Reported problems include– misplaced records– test data recorded on scrap paper– failure to control manufacturing steps such as

those governing products' physical properties– release of products not meeting their

specifications– inadequate investigation of failed products


Barr: “Testing into Compliance”

Barr had numerous failuresPerformed retests with

– no investigations– no regard for process and product history

Tested until results met specificationsThen irrespective of previous OOS results

for the batch, released productreporting only the passing resultsreporting only the passing results

Q: How do you report passing OOS’s on COA?


Reading the Judgment

Reading the Barr Court Judgment is likereading FDA’s draft guidance and pretty similar to thefinal guidance

Judge Wolin preferred to use the term"out-of-specification" (OOS) laboratory results ratherthan the term "product failure" which was morecommon to (preferred by?) FDA's investigators

Ruled that an OOS result identified as laboratory error by a failure investigation or an outlier test, orovercome by retesting is not a product failurenot a product failureBUT

Limited situations where laboratory error could be used


Guide to Inspection: QC Labs

Issued July 1993 (must have been working onit while the court case was ongoing)

Addresses OOS results and instructsinspectors to be alert

“Evaluate the company's system to investigate laboratory test failures. These investigationsrepresent a key issue in deciding whether aproduct may be released or rejected and formthe basis for retesting, and resampling “

(Most of the information is now in FDA’s guide)


Guide to Inspection: QC Labs

OOS results fall into three categories:– laboratory error– non-process related or operator error– process related or manufacturing process error

Evaluate the company's retesting SOP for compliance withscientifically sound and appropriate procedures

A very important ruling in one recent court decision sets forth aprocedure to govern the retesting program

This district court ruling provides an excellent guide to use inevaluating some aspects of a pharmaceutical laboratory, butshould not be considered as law, regulation or bindinglegal precedent

The court ruled that a firm should have a predetermined testingprocedure and should consider a point where testing ends andproduct is evaluated. If results are not satisfactory, product isrejected.


After Barr – Current Industry Practice 2007

ALL pharmaceutical companies in developedcountries and many in less developedcountries have SOPs for handlingOut of Specification results

There are still many investigational findingsconcerning out of specification results

There are still numerous issues, particularlywith transparency:What do you report on the COA?

Able laboratories suspended activity in 2005because of Out of Specification results


When it all goes wrong….Able 2005

The Quality Unit failed to: Review computer audit trails in the Waters Empower

Data Acquisition System Provide adequate training to analytical chemistsThese practices led to: The QU releasing batches failing in-process, failing in-process,

finished product and stability specificationsfinished product and stability specifications Submission of erroneous data in Annual Reports

and Prior Approval Supplements Ceasing manufacture, distribution and recall of all

products as of 13 May 2005 and withdrawal of atleast 5 ANDAs


Resample, Re-injection, Reprocessing

18

Time for a Break


From Able Laboratories 483

Notebooks and binders lacked data from all testing conducted in the QC Laboratory.

Records did not include all data such as OOSresults, chromatograms, sample weights, andprocessing methods.

OOS results were substituted by passing results by Analysts and Supervisors.

The substitution of data was performed by cutting and pasting of chromatograms, substituting vials, changing sample weights and changingprocessing methods


That isn’t relevant…it’s fraud

Think about the following scenariobefore saying it couldn’t happen:– You are just starting up your HPLC system in

the morning– You inject 5 replicates of standard– The first four give perfect responses– The fifth is way off / makes no sense,

obviously incorrectWhat do you do?


It couldn’t happen in my company

A split second wrong judgment can appear asfraud to the inspector

You inject one more standard injection and itgives a perfect response

Clearly you were right and the original “5th”injection was dirty glassware or mis-injected,or, or, or……


It couldn’t happen in my company

But you don’t want to show 4 good injections, one lousy and another good one

So you cut out the bad one and documentonly the good ones in the notebook

Sounds “horrendous?” It has been done, probably in some of your

laboratories! Can you be sure none of youranalysts ever did this?How can you be so sure?


GMP and OOS Results

Do you know where the term OOS appears in

– EU GMP regulations it didn’tas of June 2006 it

does

– US GMPs it doesn’t

– Q7A GMP for APIs it DOESbecause Q7A was written after 1993


Q7A on OOS


Ever heard this….

“The Lab don’t know how

to test”

“Give it to Pete…

he knows how to do

that test…

Dave always gets badresults!”


OOS Case Study #1

Complex biochemical assay: 6 replicates Inherent variability allows for wider than usual

specification of 80 – 120%Results: 45, 50, 46, 52, 65, 69%

What would you think if it happened to you?The lab technician has 20 years seniorityNo other technician is familiar with the test


OOS Case Study #1 - continued

“Must be my mistake!”WRONGAnalyst retested (not in accordance with SOP)Results: 72, 69, 81, 80, 82, 81%

NOW what would you think?



“Results: 72, 69, 81, 80, 82, 81%

72 and 69% must be “OUTLIERS”

Average 81, 80, 82 and 81 and result passesBatch can be releasedReport only this set of results



The outcome…..Product complaints from patients and doctors:

Product is sub-potentLitigationProduct recall Investigation reveals weighing error in

production


What is the purpose of testing?

To find out the value of a specific parameterTo assess if that parameter meets the

pre-determined specification for each lotSo as to make a sound scientific judgment

regarding product release or rejectionWhat happens in your company when there is

an OOS result? IDEALLY the analyst doesn’t know the

specification because of “BIAS.”


Is it possible to ensure a correct result?

Not at a 100% certainty levelJust as it is not possible to prevent an incorrect

result (at the 100% certainty level)What is a correct result?A result that is identical to the true result….

BUTWhen testing, we NEVER know the true result


Consequences of an Incorrect Test Result?

FALSE NEGATIVE

Product declared fit for use when not

Side Effects Death

FALSE POSITIVE

Product declared unfit for use when fitRejection Financial Loss


Is it possible to ensure a correct result?

It is possible to ensure a result as close aspossible to the true result

Using a quality assurance program in thelaboratory

Validate methodsQualify analystsFollow methods as writtenQualify, calibrate and maintain equipmentReport deviations / malfunctions


Types of Tests

Quantitative: assays and some limits testsQualitative: e.g. sterility test, appearanceChemicalPhysicalMicrobiological


Chemical Testing

Inherently reliablePrecision is usually considerably better than

for microbiolgical and biochemical testingOutlier testing is forbidden by the FDA guide

for chemical testing(usually have less replicates anyway)

Don’t forget case study #1 – How NOT tohandle OOS results


Physical Tests

e.g.1 Black spots in powdere.g.2 Fill volume

are particularly problematic, since results arealmost certainly correct.Is there any place for retesting?In e.g. 1 Probably notIn e.g. 2 Maybe

Is there any place for resampling?


FDA OOS Guidance, October 2006

Draft from September 1998 for comments Was confusing and open to misinterpretation

particularly by persons not familiar with theindustry

Final guidance is still confusing but a lot better thanthe draft

The draft was referenced, as was the Barr court case during inspections, so the final guidance will definitelybe used

Should be considered in preparing an OOS SOP


FDA Guide on OOS Results

Finalized October 2006Contains “nonbinding recommendations”Requires that an investigation is performed

ANYTIME that an OOS is obtainedWants to include ALL suspect resultsWants an SOP stating number of retests

UPFRONT


Table of Contents

IntroductionBackground Identifying and Assessing OOS Results –

Phase I: Laboratory Investigation– Responsibility of Analyst– Responsibilities of Laboratory Supervisor


Table of Contents (continued)

Investigating OOS Results –Phase II: Full Scale OOS Investigation– Review of Production– Additional Laboratory Testing– Reporting Testing Results

Concluding the Investigation– Interpretation of Investigation Results– Cautions– Field Alert Reports



IF you don’t know the specification youshould never have an OOS, but we don’tlive in an ideal world!

HOW can you have an OOS if:– the method is validated– analysts are qualified– equipment is calibrated and

well-maintained– notebooks have full record of testing


How to Identify OOS Results

Compare result with specificationBe sure that specification is to required degree

of accuracy and round the result to this value

E.g. spec 95 - 105 94.9 passesspec 95.0 - 105.0 94.9 fails

Message for your R&D Department:think carefully before setting specifications!!!


How to Identify OOT Results

Out of Trend or unusual results are generallyresults that:– MEET the product specification– ARE outside the control limits of the process,

where control charts are usedor

– ARE different to results usually obtained(e.g. spec: 95.0 – 110.0usual results: 98.5 – 101.0OOT result: 96.4


Initial Assessment of OOS Result

Bear in mind prior:– Product history– Process history– Test history– Reliability of equipment– Reliability of the analyst– Precision of the test (validation)



Failure of a control is NOT an OOS eg:– Standard shows impurity peak and assays at

incorrect retention time– Standard shows assay of 80%– System suitability drift– In this case all test results SHOULD BE INVALIDATED

and the test REPEATED using an additional aliquot ofthe same preparation if possible, or an additionalaliquot from the same sample (What if product degrades and this is not possible? – need to defineduplicate sampling procedure so that always havespare material)

– Data collected is retained on file ANYWAY– What happens if results are OOS? Do you need to err

on the side of caution e.g. possible homogeneity issue?


OOS – Laboratory Investigation

IF you don’t question an “in-spec” result, whydo you question an OOS?

Must have a Retesting Policy:– Laboratory investigation

• equipment -e.g. glassware, calibration, maintenance

• question large differences between replicates

• calculations

• Product history etc. etc. etc. (to be continued)

• 4M’s: man, machine, methods, materials

47

Time for Lunch


FDA Guide: Introduction

Applies to chemistry – based testing[KSG: primarily HPLC and GC]

Includes in-process testing except (footnote), where the purpose is to prevent process drift[KSG: discuss – could still have OOS!]

Principles apply to CONTRACT firmsTimely, unbiased investigation [KSG: OOS Policy is major SOP approved by

senior management]


FDA: Laboratory Investigation

The source of OOS should be identified as:– Aberration of measurement process– Aberration of manufacturing process

[KSG: sampling process?]Even if the batch is rejected, an investigation is

required:– Other batches involved?– Other products involved?

Need written investigation, conclusions, follow-up


FDA: Laboratory Investigation

Assess accuracy of lab data:– Before test preparations are discarded

(composite / homogeneous source of aliquot tested)

– Hypothesis testing using same test preparationfor laboratory error or instrument malfunction

– IF no meaningful errors were made [found?],conduct a full scale OOS investigation

– Contract lab conveys data and findings to you


Invalidate resultPerform new test on same sample

Convincing evidenceof

Laboratory Error

Correct if possibleor

Reject Batch

Production Error

ResampleDouble sample

Revise sampling procedures

Sampling Error

QA decisionregarding

batch disposition

All withinspecifications

Reject Batch

One resultOOS

Retest ???further aliquots

from original sample

InconclusiveNo evidence of production error

QA Investigation

Inconclusive

Laboratory InvestigationReport to QA (copy in batch file?)

OOS result

OOS Flow Chart


Responsibility of Analyst

To follow test procedure as writtenTo be alert to errors and STOP test BEFORE

obtaining the result if error is suspected,recording what happened e.g. spill

Analyst responsible for ensuring thatinstruments meet performance specificationsand are properly calibrated [KSG: maintained?]

Once an OOS result is obtained to review allrecords relative to the test to identify possiblelaboratory error


Supervisory Role

Supervisors / team leaders / laboratory head:– Should be experienced analysts– Frequently audit while tests ARE BEING

performedin order to be able to objectively investigateOOS results


Responsibility of Supervisor

Objective assessment withoutpreconceived assumptions as to cause of OOS

Immediate assessment may include:– Re-examination of:

• actual solutions• Test units• Glassware

used in the original measurements and preparations

This could provide more credibility for laboratory error hypotheses


Responsibility of Supervisor

To INVESTIGATE:– Review notebook / worksheet with analyst :

• Was method was followed: with a copy of themethod in your hand, have the analyst describeexactly how they performed each step:confirm that the method was understood & followed

• Review raw data: Perform calculations againincluding checking dilution schemes

• Unauthorised changes to automated calculations• Examine reagents, (reference) standards, solutions• Examine glassware• Performance of instruments


Hypothesis Testing

If you re-create an OOS conclusions are stronger Might retest recent sample to confirm / refute

equipment malfunction: retro consequences? Examine retained samples e.g.

– Re-inject solutions where a transient equipmentmalfunction is suspected [air bubbles]Hard to prove, but re-injection of samepreparation can provide strong evidence

– Release rate testing of slow release tab / cap:retest of original unit may show it was damaged duringtesting

– Additional extraction to show that problem ismethod related – then revise method


Hypothesis Testing, Case Study #2

LAL test on radioactive product (short half-life) is positive

Product history – no previous failureOther products tested in same series passed Initial laboratory investigation: no evidence

of lab errorDilution used: 1:70

specification allows up to 1:140 dilution


Hypothesis Testing, Case Study #2

– Perform retest in parallel with productioninvestigation because of short half-life

– Production investigation included samplingglassware and equipment for LAL residuesNO positive results

– Repeat test at 1:70 and 1:140 dilutionsResults were in spec i.e. no positive LAL

– New LAL reagent prepared and same sampletested with old and new reagent:

• Old: failed• New: passed



If the laboratory investigation is conclusiveconclusive– Document findings– INVALIDATE original test– Perform NEW test on same sample– Report original result with investigation as well as new

result in batch record for QA review prior to release– COA carries new result only; some companies use an

asterisk and indicate that there was an OOS If the laboratory investigation is NOT NOT conclusive

inform QA (or customer for contract lab)


Out of Trend Results

If the laboratory investigation is conclusive orinconclusive, consult with QA

In most cases, DO NOT perform any additionaltesting or sampling

Make product disposition judgment based on:– Original result– Product history (e.g. stability data – statistical analyses

of particular use here)– Batch history (e.g. review indicates that there were no

processing errors / there were errors)– Other investigational findings


And now some work for you.....

Specification is 90.0 – 110.0% for an oralsuspensionThree batches are tested simultaneously, eachsample in duplicateTwo batches show (average) 98.2% and 99.7%respectivelyThird batch gives one result of 88.2% and asecond result of 89.3%

PREPARE A CHECKLIST OF QUESTIONSTO REVIEW AS PART OF THE LABORATORY INVESTIGATION


FDA: Full-Scale OOS Investigation

Use a pre-defined procedure Production/ process review and / or additional

laboratory work Identify root cause and implement CAPA QA / QCU responsibility, includes CMOs if used Documented in the batch record Involves all aspects of manufacture, quality

control and sampling Describes corrective actions and endpoint Is performed PRIOR to ANY retesting


FDA: Full-Scale OOS Investigation

If cause of OOS is identified, batch is rejected In this case need CAPA on process / product May not identify cause and may need additional

lab testing:– Retest additional portion of original sample– Resample


FDA: Retesting and Resampling

Use another analyst? Where possibleThe maximum number of retests should be

specified in advance in an SOPMay, on rare occasions, deviate from SOP but

with documented rationale and protocolThe number may vary depending upon the

variability of the test method and NOTdepending on the results obtained

Resampling raises questions as to samplingprocedure validity


Interpretation and Results Reporting

Report all results, passing and suspect andconsider for batch release decision

Averaging– Has appropriate and inappropriate uses:

• Appropriate: optical rotation test: several discretemeasurements averaged for result; microbiology;HPLC average of 2 or 3 peak responses fromreplicate injections of the same preparation = onetest and one result as described in the test methodHave acceptance criteria for deviation between replicates

This is different to analysis of different portions ofa single batch


Interpretation and Results Reporting

Averaging– Inappropriate uses:

• May conceal variations in different portions of thebatch or samplee.g. for powder blend uniformity or dosage formuniformity of content.

• Averaging the results of the original OOS andadditional retest or resample results is inappropriate

• All individual results must be provided andconsidered by the QCU / QA for release

67

Time for a Break


Outlier Tests

Possible use of the outlier test should bedetermined in advance

Should be written in an SOP for datainterpretation

Should include the specific test to be usedShould specify the minimum number of results

required to obtain a statistically significantassessment from the test

For validated chemical tests its use is suspect


FDA: Concluding the Investigation

Evaluate results and determine batch qualityRelease decision by QCU / QAAn initial OOS does not necessarily mean

that the batch fails and must be rejected.Where the suspect result is invalidated, the

result should not be used to evaluate thequality of the batch or lot

For inconclusive results – give fullconsideration to the OOS result


FDA: Concluding the Investigation

Example given shows seven retest resultswhich gives the only indication in the guideregarding numbers of retests

The example given is also extreme:89.5% OOS99.0, 98.9, 99.0, 99.1, 98.8, 99.1, 99.0%

Consider method precision and validation datain making release / reject decision


FDA: Cautions

Where a series of assay results (to producea single reportable result) have some individual results OOS and some in spec andall within the known method variability,passing results no more likely to representthe true value than the OOS result.

In this case the company must err on the side of caution and reject the batch

A result that is low but in specification shouldalso be a cause of concern


FDA on Field Alert Reports (FARs)

For products with approved (a)/NDAs,regulations require a FAR within 3 workingdays concerning any failure of a distributedbatch to meet any of the specificationsestablished in an application

Unless the OOS result is found to be invalidwithin 3 days, an initial FAR should besubmitted

A follow-up FAR should be submitted whenthe investigation is completed


OOS Case Study #3

BN 200602 gave an assay result for 3 month stability study: 89.2% (long term / RT)

Limits: 90.0 – 110.0%Test performed at a contract laboratory using

an internal instrument controlBN 200701 tested at the same time gave a

result of 95.4% (i.e. in spec. – release test)

DO YOU INFORM THE REGULATORS?


OOS Case Study #3 – cont/

The samples have to be diluted duringpreparation and as far as records showedthere was no evidence of laboratory error

Calculations were satisfactoryNo evidence that reagents were outdated or

faulty techniqueControl sample showed a 3.5% difference at

beginning and end of run: usually around 1%;NMT 5% allowed by method



Batch number 200701; in-process result: 98.2%Batch number 200602; time zero result: 92.8%

i.e. an apparent difference of 3.5% betweenresults in both cases

Hypothesis formulated:problem is with the control sample / equipment and this can be tested using both batchesi.e. in-spec and OOS



Retesting Protocol (before start of retesting):– BN 200602: OOS on inverted stability sample

at 3 monthsPerform retest at three dilutions in duplicateon:

• Same sample = 6 results vs 2 original• Upright sample = 6 additional results

– BN 200701: In spec. but 3.5% lower than IPCPerform retest at three dilutions in duplicate

• Same sample = 6 results vs 2 original



Outcome:

– BN 200602: All retest results in specification

average result: 92.2%– BN 200701: All retest results in specification

average result: 97.9%– Control sample: difference of 1% beginning and

end of run


OOS Case Study #3 – Conclusion

Original OOS result for BN 200602 is invalidOriginal OOT result for BN 200701 is invalid ?CAPA required regarding external laboratory

as follows:– Tighten allowed limits for control sample

(NMT 5% is too high)– Revalidate method? Investigation showed

validation last done 15 years ago and a newinstrument had been introduced since then!

– Closely follow BN 200602 at additional stability stations


OOS Logs and Trending Data

Should keep a record of OOS resultsAnalyse periodically according to:

– Analyst– Instrument– Product– Test method

Take appropriate actions where repeatproblems appear


And in Conclusion....

Always report OOS or OOT result toSupervisor

Have a pre-approved SOP that provides aflow-chart for handling OOS

Conduct detailed and genuine investigationDocument investigational findings clearly and

conciselyTransparency regarding reporting on COAAt some point...notification of regulators


In conclusion

All test results are subject to some element of doubt Use controls and validated, approved test methods Use replicates where inherent variation exists Use qualified, calibrated and well maintained

equipment Perform trend analyses and report deviations NEVER continue a suspect test At some point, testing ends and a product disposition

decision must be made


Thank you for your attention

Any questions?

Find me at [email protected]

mailto:[email protected]

Out of Specification Results (OOS) A One Day Workshop

Documents

Transcript of Out of Specification Results (OOS) A One Day Workshop