And discriminatory accuracy EPIDEMIOLOGICAL MEASURES Philippe Wagner Statistician Unit for Social...

And discriminatory accuracy

EPIDEMIOLOGICAL MEASURES

Philippe WagnerStatisticianUnit for Social Epidemiology, Lund UniversityCentre for Clinical Research, Uppsala University, Västerås

Odds ratio (OR)Population attributable fraction (PAF)Variance explained / VPC / ICCRisk diff erence (RD) / Number needed to treat (NNT)

What are the used for? In what situations? Is there any benefit in adding information about the

discriminatory accuracy (DA)?

MEASURES

Odds ratio (OR) Population attributable fraction (PAF) Variance explained / VPC / ICC Risk diff erence (RD) / Number needed to treat (NNT)

What are the used for? In what situations? Is there any benefi t in adding information about the

discriminatory accuracy (DA)?

Note: We will only be studying dichotomous risk factors and outcomes.

Note II: Will be tanking some liberties with respect to notation and mathematical rigor in order to focus on the bigger picture.

MEASURES

Warning! This presentations may contain some algebra!

A reminder

DISCRIMINATORY ACCURACY

Measured by sensitivity and specifi city Sensitivity

Proportion of exposed cases Loosely put; How well your prediction is doing on the cases.

Specifi city

Proportion of non-exposed controls. Loosely put; How well your prediction is doing on the controls

DISCRIMINATORY ACCURACY

𝑃 (𝑅=1|𝑂=1 )=𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑎𝑠𝑒𝑠 h𝑤𝑖𝑡 𝑟𝑖𝑠𝑘 𝑓𝑎𝑐𝑡𝑜𝑟𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑎𝑠𝑒𝑠

𝑃 (𝑅=0|𝑂=0 )=𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑜𝑛𝑡𝑟𝑜𝑙𝑠 h𝑤𝑖𝑡 𝑜𝑢𝑡 𝑟𝑖𝑠𝑘 𝑓𝑎𝑐𝑡𝑜𝑟𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑜𝑛𝑡𝑟𝑜𝑙𝑠

THE CLASSICAL 2X2 TABLE

Outcome No Outcome

Exposed Sensitivity 1-Specificty

Not exposed 1-Sensitivty Specificity

If we condition on the outcome we get the table above.


Outcome No Outcome

Exposed TPF FPF

Not exposed 1-TPF 1-FPF

Often expressed in terms of true and false positive fractions where


Outcome No Outcome

Exposed TPF FPF


We know that the OR is also calculated from the 2x2 table. This will help us link the OR to the discriminatory accuracy of the factor.

And risk factor DATHE ODDS RATIO

Needs no introduction – we all use it everyday.For its connection to DA, let’s go back to the 2x2

table..

THE ODDS RATIO


Outcome No Outcome

Exposed TPF FPF


The odds ratio can, from the above table, be expressed as


Outcome No Outcome

Exposed TPF FPF


Through algebra, we can re-express this as


Outcome No Outcome

Exposed TPF FPF


And we can draw..

TPF VS FPF GIVEN THE OR

Pepe AJE 2004#1 in the course reading material

Diff erences in risk factor prevalence can cause diff erent TPF/FPF scenarios.

Intuitively, this is not diffi cult to understand. When we have diff erent numbers of outcomes and

exposed we get some false negative and some false positive predictions.

SO WHAT IS HAPPENING?!

INTUITIVELY

People with outcome

People with exposure

INTUITIVELY

People with outcome


Perfect! Exposure covers all cases, AND nothing else

INTUITIVELY

People with outcome


Perfect! Exposure covers all cases, AND nothing else

TPF very high!FPF very low!

INTUITIVELY

People with outcome


INTUITIVELY

However…

INTUITIVELY

However…

False positives!FPF high!TPF high!

INTUITIVELY

Or the other way around..

INTUITIVELY

Or..

INTUITIVELY

Or..

False negatives!TPF low!FPF low!

IN THE 2X2 TABLE

Outcome No Outcome

Exposed TP FP

Not exposed FN TN

We want TP and TN to be high, FN and FP low.

IN THE 2X2 TABLE

Outcome No Outcome

Exposed TP FP

Not exposed FN TN

We want TP and TN to be high, FN and FP low.When exposure prevalence increases, TP and FP increases in relation to FN and TN.

IN THE 2X2 TABLE

Risk factor prevalence

Ris

k

FP

TP

TN

FN

PAF

POPULATION ATTRIBUTABLE

FRACTION

Definition: The PAR is used to estimate the fraction of the total disease burden in the population that would not have occurred if a causal risk factor were absent.

Often used to gauge the eff ect of a potential intervention on the population.

Used in etiological studies to indicate how much of a disease that is ”explained” by existance of an exposure in the population. Controversial use. (See for instance, Rockhill AJE 1998)

Used in etiological studies to indicate how much of a disease that is not ”explained” by existance of known exposure. Controversial use. (Rockhill AJE 1998)

PAF

Definition: The PAR is often used to estimate the fraction of the total disease burden in the population that would not have occurred if a causal risk factor was absent.

PAF

𝑃𝐴𝐹=𝑃 −𝑃𝑁𝑜𝑡

𝑃

PAF


Ris

k

FP

TP

TN

FN

PAF


Ris

k

PAF

FP

TP

TN

FN

PAF


Ris

k

PAF

FP

TP

TN

FN

We are subtracting the cases in the exposed group not due to exposure.

PAF

E A

One possible explanation. Three component causesE, A and B. Two pathways to disease.

B A

PAF


Ris

k

PAF

FP

TP

TN

FNB A

E A

Studying exposure EE

EAB

A

B

PAF


Ris

k

PAF

FP

TP

TN

FNB A

E A

Studying exposure E

Removing Eremoves the E|A cases.

Does not remove exposed E|A|B cases.

PAF=30%

E

EAB

A

B

PAF


Ris

k

PAF

FP

TP

TN

FNB A

E A

Studying exposure E

Removing Eremoves the E|A cases.PAF=30%

Removing ARemoves all cases. PAF=100%

E

EAB

A

B

Rockhil l (AJE 1998) commented on another author, stating that: . .after computing an attributable fraction of 41% for the three risk

factors “no fi rst birth by age 20 years”, “family history of breast cancer in a fi rst-degree relative”, and “family income level in the upper two terti les of the United States”, Madigan et al. state that their estimates

"suggest that a substantial proportion of breast cancer cases in the are explained by well-established risk factors“. This use of the word "explain" is misleading. According to the data of Madigan et al. , nearly the entire population

of women in the United States has at least one of the considered risk factors. Since the vast majority of such exposed women wil l not develop breast cancer, Stating that such factors explain a large proportion of breast cancer risk is misleading and even alarmist.”

PAF IN RELATION TO DA

Rockhil l (AJE 1998) commented on another author, stating that: . .after computing an attributable fraction of 41% for the three risk

factors “no fi rst birth by age 20 years”, “family history of breast cancer in a fi rst-degree relative”, and “family income level in the upper two terti les of the United States”, Madigan et al. state that their estimates

"suggest that a substantial proportion of breast cancer cases in the are explained by well-established risk factors“. This use of the word "explain" is misleading. According to the data of Madigan et al. , nearly the entire population

of women in the United States has at least one of the considered risk factors. Since the vast majority of such exposed women wil l not develop breast cancer, Stating that such factors explain a large proportion of breast cancer risk is misleading and even alarmist.”


FALSE POSTITIVES!

Rockhill also notes that in her own study of breast cancer risk factors in the US that

the PAF increases with a more liberal definition of risk factor cut-off s.


. .estimates of population attributable fraction for establ ished breast cancer risk factors can be made "high" only by defi ning risk factors in such a way that virtual ly the entire population must be labeled "exposed,“ and therefore, "at r isk.”

. . To demonstrate this, we estimated population attributable fraction for the four establ ishedbreast cancer r isk factors “early age at menarche”, “late age at fi rst ful l -term pregnancy/nul l iparity”, “history of breast cancer in mother/sister”, and “history of benign breast biopsy” and examined the sensit ivity of the population attributable fraction and its precision to changes in exposure cutpoints. Using the broad exposure defi nit ions for early age at menarche (<14 years) and for late age at fi rst ful l -term pregnancy (>20/nul l iparous), we found a high proportion exposed among white cases and controls 98%.

The population attributable fraction estimate was reduced (from 0.25 to 0.15) when the most "restrictive" exposure defi nitions of early age at menarche (<12 years) and late age at fi rst ful l -term pregnancy (S:30 years/nul l iparous) were used.

That is: “the PAF is larger with a more l iberal defi nition of r isk factor cut-off s.”


We will see that this is mathematical fact. It is known that the PAF can be expressed as

Indicating that the PAF increases with the risk factor prevalence for a given relative risk.

From before, we know that TPF and FPF also increases with risk factor prevalence, for a given OR.

We have that


𝑃𝐴𝐹=𝑝𝑅 (𝑅𝑅−1)𝑝𝑅 (𝑅𝑅−1 )+1


RR = 5

PAF


Ris

k

PAF

FP

TP

TN

FN


RR = 5

PAF


Ris

k

PAF

FP

TP

TN

FN


RR = 5

PAF


Ris

k

PAF

FP

TP

TN

FN


RR = 5

.. To demonstrate this, we estimated population attributable fraction for the four establishedbreast cancer risk factors “early age at menarche”, “late age at fi rst full-term pregnancy/nulliparity”, “history of breast cancer in mother/sister”, and “history of benign breast biopsy” and examined the sensitivity of the population attributable fraction and its precision to changes in exposure cutpoints. Using the broad exposure defi nitions for early age at menarche (<14 years) and for late age at fi rst full-term pregnancy (>20/nulliparous), we found a high proportion exposed among white cases and controls 98%.

According to the data of Madigan et al., nearly the entire population of women in the United States has at least one of the considered risk factors. Since the vast majority of such exposed women will not develop breast cancer ,


High risk factor prevalence!

High FPF!

Empirical exampleStudying myocardial infarction (MI) in the MDC-cohort

creating a risk-index from traditional risk factors and biomarkers in order to predict disease within 15 years from baseline.

Study population.The Malmö Diet and Cancer (MDC) study is a population-

based, prospective epidemiologic cohort of 28 449 persons enrolled between 1991 and 1996. From this cohort, 6103 individuals were randomly selected to participate in the MDC cardiovascular cohort.

5054 had complete information on traditional risk factors, 4764 on biomarkers and 4489 on both traditional risk factors and biomarkers.


The fact that PAF increases with risk factor prevalens, is important to keep in mind when choosing cut-off s for risk factors in your study and interpreting the results.


Another fact that is important to keep in mind, with respect to the PAF-DA relationship, is when reading PAF reported from other studies, is that a given PAF can be associated with very diff erent combinations of TPF and FPF.


In fact, the PAF does not care about the FPF at all. This is mathematical fact.

With some algebra we can show that

PAF depends only on the risk factor prevalence and the TPF,

TPF can then be re-written as

And since we know that the FPF increases with pR as well, if we fix the PAF, we can draw


𝑃𝐴𝐹=𝑇𝑃𝐹 −𝑝𝑅

1−𝑝𝑅

𝑇𝑃𝐹=𝑃𝐴𝐹+(1−𝑃𝐴𝐹 )𝑝𝑅


We can remove 60% disease burden with moderate precision. FPF=20%

PAF


Ris

k

PAF

FP

TP

TN

FN

PAF


Ris

k

PAF

FP

TP

TN

FN

We can remove 60% disease burden with high presicion.

PAF


Ris

k

PAF

FP

TP

TN

FN

TPF=PAFAll exposed cases can be removed.

With almost no FPF. Almost no unnecessary costs and side effects.

PAF


Ris

k

PAF

FP

TP

TN

FN

PAF


Ris

k

PAF

FP

TP

TN

FN

We can still remove 60%. But with a large amount of unnecessary costs and possible side effects.

Empirical example: Genotype and bladder cancer. A genetic association study (ref)

showed strong evidence that the copy number of gene GSTM1 is signifi cantly associated with risk of bladder cancer, with an OR = 1.9 corresponding to the GSTM1 null genotype (51% prevalence). If this marker were used as a binary marker for bladder cancer detection in the general population, it would result in 66% sensitivity and 50% specifi city, a poor marker for diagnostic purposes. However, if a drug were to be developed that targeted the pathway(s) by which GSTM1 null increases risk, and if the drug were 100% eff ective in preventing bladder cancer without toxic side eff ects (and ignoring costs), then treatment of all marker carriers would reduce bladder cancer by 31% (PAR%),

Ref: Li Quantifi cation of population benefi t in evaluation of biomarkers: practical implications for disease detection and prevention, Med Inform. & Desc. Mak. 2014.



If they had only presented PAF 31%.The TPF could be between 31 and 100%.The FPF could be between 0 and 100%


31% of disease could be removed, but with low precision.

FPF was 50%.

This means that 50% who did not develope disease would still get the drug.

Empirical example: CNV and neuroblastoma. A copy number variat ion associated with

neuroblastoma was reported recently (ref) . The prevalence of the marker (1q21.1) in the general population is about 9%, and the OR of the marker (copy loss) for neuroblastoma risk is est imated to be around 3. I f th is marker were dichotomized as a binary marker for predict ing the absence or presence of the disease, i t wi l l result in a 23% sensit iv ity and 91% specifi city, with a PAR% of approximately 15%, which indicates the marker could account for about 15% of neuroblastoma risk i f the disease is truly caused by the CNV (copy-number variat ion). Assume a drug is developed that targeted this marker (1q21.1) for prevention. I f the drug is 100% eff ective in disease prevention and had no s ide eff ects and al l persons who were carriers for the marker were treated with the drug, it would reduce the total d isease cases by 15% (PAR%).

However, in the more l ikely scenario, drugs have s ignifi cant s ide eff ects and are not 100% eff ective such that more extensive r isk benefi t analyses are needed.

Ref: L i Quantifi cation of population benefi t in evaluation of biomarkers: practical implications for disease detection and prevention, Med Inform. & Desc. Mak. 2014.



If they had only presented PAF 15%.The TPF could be between 15 and 100%.The FPF could be between 0 and 100%


In this example only 15% of disease could be removed.

But it could be removed with good precision. Only 9% FPF.


In this example only 15% of disease could be removed.

But it could be removed with good precision. Only 9% FPF.

But is it high enough if there are side effects?

Summary IThe PAF has a natural tendency to grow with

increasing risk factor prevalence.When it increases, so does the FPF, and many with

the given risk factor never get the disease.This may appear odd when interpreting PAF as the

proportion of disease explained by the risk factor.This fact may be highlighted by presenting TPF and

FPF together with PAF.


Summary IIThe PAF is the proportion of exposed cases that can

be removed. The TPF is all exposed cases, and not all exposed cases can be removed.

Therefore, the PAF is the lower possible value of the TPF (when PAF is say 0.8, so is the lowest possible TPF).

When they are the same, the FPF is low and disease burden can be removed with great precision.


Summary IIIBut PAF may appear with any number of TPF and FPF.The PAF is the proportion that can be removed in an

idealized situation where intervention has no costs or side eff ects.

In reality TPF and FPF need to be considered together with PAF, in order to make a real risk-/cost-/benefit analysis.


ICC and VPC

VARIANCE EXPLAINED

In this context, a measure of how much presence/abscene of exposure explains in terms of the presence/absence of the disease.

As opposed to PAF that only cares about cases and presence of exposure.

In the continous case it corresponds to the correlation between predicted and observed outcome.

VARIANCE EXPLAINED

In dichotomous outcomes several diff erent alternatives to choose from.

For the purpose of simplicit for this presentation, we chose the McKelvey and Zavoina R 2. Similar to the continuous case, it is defi ned as

with the exception that the continous predictor y * is measured on a latent scale and manifestas as a 0/1-variable below/over a certain cut-point.

In this case, y * is assumed follow the logistic distribution.

Intuitively, it be viewed as the correlation between the predicted and observed outcomes on the latent scale.

VARIANCE EXPLAINED

When the predictor is one dichotoumous exposure variable, this becomes

Where PR is the risk factor prevalence and π2/3 is the variance of the logistic distribution.

Alot of algebra, but we observe that the variance explained depends on the prevalence of the risk factor and the odds ratio.

VARIANCE EXPLAINED

VARIANCE EXPLAINED

Just like TPF and FPF, VE is dependent on the risk factor prevalence.

As TPF and the FPF depend on risk factor prevalence we can plot them togheter to see that..

VARIANCE EXPLAINED

VARIANCE EXPLAINED

RR = 5

IN THE 2X2 TABLE

To see why this is..

IN THE 2X2 TABLE


Ris

k

FP

TP

TN

FN

In order fo VE to be large..

We want TPF and FPF to be as large as possible.

IN THE 2X2 TABLE


Ris

k

FP

TP

TN

FN

We want TPF and FPF to be as large as possible.

TP and TN are where exposure and disease are concordant.

For FP and FN they are discordant.

IN THE 2X2 TABLE

IN THE 2X2 TABLE

TPF = (TP/TP+FN) grows.

FPF = (FP/FP+TN) grows as well.


Ris

k

FP

TP

TN

FN

IN THE 2X2 TABLE

IN THE 2X2 TABLE

TPF = (TP/TP+FN) grows.

FPF = (FP/FP+TN) grows as well.


Ris

k

FP

TP

TN

FN

IN THE 2X2 TABLE

TPF = (TP/TP+FN) decreases.

FPF = (FP/FP+TN) decrease as well.


Ris

k

FP

TP

TN

FN

IN THE 2X2 TABLE

IN THE 2X2 TABLE


Ris

k

FP

TP

TN

FN

We want TPF and FPF to be as large as possible. Alot of concordant pairs.

Best balance happens at near risk factor prevalence 0.5

TPF; How well your prediction is doing with the cases.FPF; How well your prediction is doing with the

controls.Variance explained; How well you are doing with both.TPF and FPF both have to be high in order for

explained variance to be high. The concepts of VE and DA in terms of

sensitivity/specificty are closely connected.

VARIANCE EXPLAINED

However, the same explained variance can be associated with diff erent sets of TPF and FPF.

VARIANCE EXPLAINED

The diff erent sets of TPF and FPF improve with increasing VE.

VARIANCE EXPLAINED

But VE is thightly linked to DA in terms of TPF and FPF improving with increasing VE.

VARIANCE EXPLAINED

VE is thightly linked to DA in terms of TPF and FPF improving with increasing VE.

As we have seen the PAF is not.

VARIANCE EXPLAINED

VE and PAF conveys very diff erent information.

COMPARE VE AND PAF


COMPARE VE AND PAF

PAF tells you that your risk factor covers cases that could be removed.

VE tells you wheter the risk factor explains presence/absence of disease


VE AND PAF

PAF is high when VE is low, for high risk factor prevalence.

This is because PAF ignores the fact that we are not explaining absence of disease in those exposed.

VE – PAF GOING BACK..


Ris

k

PAF

FP

TP

TN

FNB A

E A

May be explained using previous example.

1. Studying exposure E

2. And we increase risk factor prevalence and disease prevalence in non-exposed..

E

EAB

A

B

VE AND PAF – GOING BACK


Ris

k

PAF

FP

TP

TN

FN B A

E A

Studying exposure E

PAF is happy with knowing about E

VE is not happy until we identify the interaction with the factor A.

E

EAB

A

B

When variance explained is reported in a study, we are informed of the combined eff ect of TPF and FPF.

If high, we know we are doing well predicting both presence AND absence of disease.

As opposed to PAF who is only concerned with presence of exposure in diseased.

VARIANCE EXPLAINED - SUMMARY

But the same value of explained variance can still correspond to diff erent combinations of TPF and FPF.

Depending on the study problem, a higher TPF or a lower FPF may be preferred.

TPF may be important if disease is serious and can be prevented.

FPF may be important if treatment is expensive and/or has side eff ects.

Therefore, it may still be sensible to present TPF and FPF in addition to VE.

If VE is low, we do not know if we are doing poorly with cases or controls, or both.

VARIANCE EXPLAINED – SUMMARY II

The ICC used, for instance in multilevel analysis, can be viewed in terms of explained variance.

Take for instance, a multilevel logistic regression analysis of 5-year survival after breast cancer diagnosis in diff erent hospitals in Sweden, with hospital as a random eff ect.

The ICC corresponding to the empty model, containing random term only, is given by

Where u is the hospital eff ect.

THE ICC

𝐼𝐶𝐶=𝑉 [𝑢 ]

𝑉 [𝑢 ]+ 𝜋2

3

A high ICC tells you that, when using treating hospital to predict patient outcomes, TPF is high and the FPF is low.

Patients from high risk hospitals die within 5 years, patients from low risk hospitals do not.

Therefore, treating hospital can be used to select high-risk patients for screening. Simply select the ones from high risk hospitals.

If causal, that is; hospital-related factors, such as treatment diff erences, are actaully causing the observed diff erences, hospital is an important level to consider when imporving patient survival.

THE ICC

If the ICC is low, even if diff erences are caused by hospitals, treating hospital is not important when improving patient survival as other factors are more infl uential.

Therefore it should not be used for screening or intervention.

This is because the TPF may be low, causing us to miss high risk patients treated in low risk hospitals.

The high FPF may be serious if intervention is medical treatment and has side eff ects, because we are treating patients unnecessarily.

What a high ICC does not tell us, is wheter a given defi nition of high risk hospital is good in terms of TPF or FPF, or both.

For this you need to calculate the TPF and FPF.

THE ICC

Very briefly

THE NUMBER NEEDED TO TREAT

The number needed to treat (NNT) is used to evaluate the eff ectiveness health-care interventions.

The NNT is the number of patients who need to be treated in order to prevent one additional adverse outcome.

It is defined as NNT=1/Risk diff erence between treated and non-treated patients.

NNT = 1 is ideal, where everyone improves in the treatment group and no one improves with control group.

The greater NNT, the less eff ective the treatment is.

NNT

NNT


Ris

k

FP

TP

TN

FN

RD

Studies with different prevalence in the control group may still have the same RD and hence NNT.

NNT


Ris

kFP

TP

TN

FN

RD

Studies with different prevalence in the control group may still have the same RD and hence NNT.

This yields different TPF and FPF for the same NNT.

NNT

DA associated with NNT usually bad.

For interpretation:Consider ”Not being treated” as exposure

Numers in graphAbove risk difference, below NNT

NNT

DA associated with NNT usually bad.

Event with NNT=2 we can still have up to 75% of improved patients not treated.

For interpretation:Consider ”Not being treated” as exposure

Numers in graphAbove risk difference, below NNT

All epidemiological measures mentioned OR, VE, ICC, PAF and NNT

may benefit from additional information on the DA in many studies in order to fully be able to evaluate risk, cost and benefit from a given intervention.

CONCLUSIONS - IN SHORT

THANK YOU FOR YOUR ATTENTION

And discriminatory accuracy EPIDEMIOLOGICAL MEASURES Philippe Wagner Statistician Unit for Social...

Documents

Transcript of And discriminatory accuracy EPIDEMIOLOGICAL MEASURES Philippe Wagner Statistician Unit for Social...