Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

Criterion based Validity

Dr. Alaa El Dine HassanMB BCh, MSc (PH), MD

Assist. Professor - Community and Family

Medicine

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

The Policeman’s The Policeman’s capability of detecting capability of detecting thieves is in challengethieves is in challenge

3 thieves3 thieves

6 Good People6 Good People

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

ThievesThievesGood Good

PeoplePeople

33 66

We Have: – 3 thieves – 6 good People

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

4 In Prison4 In Prison

5 Free5 Free

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

ThievesThievesGood Good

PeoplePeople

In In PrisonPrison 44

FreeFree 55

33 66

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

4 In Prison4 In Prison

5 Free5 Free

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

ThievesThievesGood Good

PeoplePeople

In In PrisonPrison 44

FreeFree 55

33 66

22

44

How Many good People Free?

How Many Thieves in Prison?

22

11

False Classification?

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

4 In Prison4 In Prison

5 Free5 Free

The policeman Correctly Identified:– 2 out of 3 thieves (66.66%)– 4 out of 6 Good People (66.66%)

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

• A new instrument is released in the market which is claimed to identify people with lung cancers.

• The instrument is supposed to give:– “Positive reading” with diseased individuals

and – “Negative reading” with Non Diseased individuals

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

• A young motivated physician wishes to evaluate the performance of the instrument.

• He applies the instrument to 100 persons of whom – 30 are known to be diseased (Diagnosed lung

cancer Cases) – 70 are known to be not diseased

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

DiseasedDiseasedNot Not

DiseasedDiseased

3030 7070

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

• The instrument gave Positive reading with 35 subjects and Negative readings with 65 Subjects.

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

DiseasedDiseasedNot Not

DiseasedDiseased

Test Test PositivePositive 3535

Test Test NegativeNegative 6565

3030 7070

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

• Out of the 35 subjects who were labeled Positive, 25 were actually Diseased.

• Out of the 65 subjects who were labeled Negative , 60 were truly disease free.

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

DiseasedDiseasedNot Not

DiseasedDiseased

Test Test PositivePositive 3535

Test Test NegativeNegative 6565

3030 7070

2525

6060

True Positives (TP)

True Negatives (TN)

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

• By Simple subtraction , we can deduce that 5 diseased subjects were falsely labeled Negative

• And 10 Non Diseased Subjects were falsely labeled Positive

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

DiseasedDiseasedNot Not

DiseasedDiseased

Test Test PositivePositive 3535

Test Test NegativeNegative 6565

3030 7070

2525

6060

True Positives (TP)

True Negatives (TN)

1010

55

False Positives (FP)

False Negatives (FN)

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

• So the test was able to correctly identify 25 out of 30 Diseased individuals (83.33%)

• 60 out of 70 Non Diseased individuals (85.7%)

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

Sensitivity

• The sensitivity is how good the test is at picking out patients with disease.

• = the proportion of patients with disease who test positive.

• = the proportion of cases labeled Positive by the test, relative to all cases that actually have the disease:

= TP / All Diseased= TP / (TP + FN).

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

DiseasedDiseasedNot Not

DiseasedDiseased

Test Test PositivePositive

TP+FPTP+FP

Test Test NegativeNegative

TN+FNTN+FN

TP+FNTP+FN(All Diseased)(All Diseased)

TN+FPTN+FP(All Disease Free)(All Disease Free)

TPTP

TNTN

FPFP

FNFN

Sensitivity = TP / (TP + FN)

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

Specificity

• Specificity is the ability of the test to pick out patients who do NOT have the disease.

• is the proportion of patients without disease who test negative.

• = TN / All Non Diseased• = TN / (TN + FP).

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

DiseasedDiseasedNot Not

DiseasedDiseased

Test Test PositivePositive

TP+FPTP+FP

Test Test NegativeNegative

TN+FNTN+FN

TP+FNTP+FN(All Diseased))(All Diseased))

TN+FPTN+FP(All Disease Free)(All Disease Free)

TPTP

TNTN

FPFP

FNFN

Specificity = TN / (TN + FP)

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

In real Life practice• Tests have false positive and false negative results,

so……

• If a patient who has undergone an ELISA test for HIV, tested positive…….

• How confident will you be that he is really infected?

• What is the probability of being REALLY infected with HIV?

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

Predictive value of a positive test

• Is the probability that a person is Diseased when a positive test result is observed.

• Is the proportion of patients with positive tests who have disease:

= = TP / All Test PositivesTP / All Test Positives= TP / (TP + FP).

• This is the same thing as posttest probability of disease given a positive test.

• It measures how well the test rules in disease.

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

DiseasedDiseasedNot Not

DiseasedDiseased

Test Test PositivePositive

TP+FPTP+FP(All Test Positive)(All Test Positive)

Test Test NegativeNegative

TN+FNTN+FN(All Test Negative)(All Test Negative)

TP+FNTP+FN TN+FPTN+FP

TPTP

TNTN

FPFP

FNFN

PPV = TP / (TP + FP)

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

Lung Cancer Example Revisited

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

DiseasedDiseasedNot Not

DiseasedDiseased

Test Test PositivePositive 3535

Test Test NegativeNegative 6565

3030 7070

2525

6060

True Positives (TP)

True Negatives (TN)

1010

55

False Positives (FP)

False Negatives (FN)

Predictive value of Positive Test

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

Predictive value of Positive test

= TP / All test Positives

= 25 / 35

= 71.4%

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

So if a patient of unknown status undergoes the

test and gets a Positive Reading, There is a

71.4% Possibility of being truly Diseased.

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

• Predictive value of a negative test is the proportion of patients with negative tests who do not have disease:

= TN / All Test Negatives= TN / All Test Negatives= TN / (TN + FN).

• It measures how well the test rules out disease.

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

DiseasedDiseasedNot Not

DiseasedDiseased

Test Test PositivePositive

TP+FPTP+FP(All Test Positive)(All Test Positive)

Test Test NegativeNegative

TN+FNTN+FN(All Test Negative)(All Test Negative)

TP+FNTP+FN(All Diseased)(All Diseased)

TN+FPTN+FP(All Disease Free)(All Disease Free)

TPTP

TNTN

FPFP

FNFN

NPV = TN / (TN + FN)

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

Lung Cancer Example Revisited

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

DiseasedDiseasedNot Not

DiseasedDiseased

Test Test PositivePositive 3535

Test Test NegativeNegative 6565

3030 7070

2525

6060

True Positives (TP)

True Negatives (TN)

1010

55

False Positives (FP)

False Negatives (FN)

Predictive value of Negative Test

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

Predictive value of Negative test

= TN / TN + FN

= 60 / 65

= 92.3%

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

So if a patient of unknown status undergoes the

test and gets a Negative Reading, There is a

92.3% Possibility of being truly Disease free.

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

Positive Likelihood Ratios

• likelihood ratios are not yet commonly reported in peer-reviewed literature or in marketing information provided by test manufacturers.

• They can be a valuable tool for comparing the accuracy of several tests to the gold standard, and

• They are NOT dependent upon the prevalence of disease.

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

The positive LR represents :

the odds of a positive test in an infected population compared to

the odds of a positive test a non infected population.

TP/ All Diseased TP/ All Diseased ————--------—————--------—

FP/ All Disease freeFP/ All Disease free

Or it can also be expressed as :Or it can also be expressed as : sensitivity sensitivity——————————

1-specificity1-specificity

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

DiseasedDiseasedNot Not

DiseasedDiseased

Test Test PositivePositive

TP+FPTP+FP(All Test Positive)(All Test Positive)

Test Test NegativeNegative

TN+FNTN+FN(All Test Negative)(All Test Negative)

TP+FNTP+FN(All Diseased)(All Diseased)

TN+FPTN+FP(All Disease Free)(All Disease Free)

TPTP

TNTN

FPFP

FNFN

Positive Likelihood Ratio

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

• Useful tests have larger positive LR • Less useful tests will have smaller positive LR.

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

Example:

A test with a positive LR of 5.0 :– For every

• 1 non-infected subjects that test positive • 5 of the infected subjects will test positive.

– For every • 1 false positive• 5 true positives

– True positives are 5 times False Positives

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

The Negative LR represents :

the odds of a the odds of a Negative testNegative test in an in an infected populationinfected population compared to compared to

the odds of a the odds of a Negative testNegative test in a in a non infectednon infected population population.

FN/ All DiseasedFN/ All Diseased------------—————------------—————TN/ All Disease freeTN/ All Disease free

Or it can also be expressed as :Or it can also be expressed as : 1- Sensitivity 1- Sensitivity

——----------------——----------------——SpecificitySpecificity

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

DiseasedDiseasedNot Not

DiseasedDiseased

Test Test PositivePositive

TP+FPTP+FP(All Test Positive)(All Test Positive)

Test Test NegativeNegative

TN+FNTN+FN(All Test Negative)(All Test Negative)

TP+FNTP+FN(All Diseased)(All Diseased)

TN+FPTN+FP(All Disease Free)(All Disease Free)

TPTP

TNTN

FPFP

FNFN

Negative Likelihood Ratio

Dr. Alaa Hassan - MB BCh, MSc (PH), MDDr. Alaa Hassan - MB BCh, MSc (PH), MD

• Useful tests have negative LR close to 0• Less useful tests have higher negative LR.