pharmacoepidemiology and drug safety 2011; 20: 653–656Published online 27 April 2011 in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/pds.2149

COMMENTARY

Benefit‐risk assessment: to quantify or not to quantify, that isthe question

Zhong Yuan*, Bennett Levitan and Jesse A. Berlin

Epidemiology, Johnson and Johnson Pharmaceutical Research and Development, Titusville, NJ, USA

Received 8 November 2010; Revised 2 March 2011; Accepted 21 March 2011

The availability of drugs and medical devices hasunquestionably benefitted many patients and funda-mentally changed the way healthcare is delivered.However, drug development is lengthy, and costs arestaggeringly high, at least in part because of the lowsuccess rate with which newly discovered compoundsachieve marketing approval. Even when a chemicalentity has moved into the late phases of development,marketing authorization may not be granted becauseof an unfavorable benefit‐risk profile.1 Drugs are alsosometimes withdrawn from the market,2 or have theiruse severely curtailed,3 because of safety concerns thatemerge when the product is used by large numbers ofpeople in clinical practice for whom neither thebenefits nor the potential harms associated with thedrug have been fully studied. The resulting uncertaintyabout benefits relative to risks may impair the public’strust in the safety of all marketed drugs and furtherreduce confidence in the integrity of both the drugdevelopment process and the regulatory approvalsystem.4

For decades, the notion that “benefits outweighrisks” has served as a fundamental principle in thedrug approval process. The logic is intuitive andsound, but to many, the decision‐making process isnot transparent, because the assessment of benefitsversus risks is carried out mainly in a qualitative andsubjective manner.4–6 Because value judgments areinherently subjective, they are often stated withlimited or no support and with no clear means tocompare them between different assessments orassessors. If the assumptions underlying these judg-ments were documented, as is now encouraged in the

*Correspondence to: Z. Yuan, Epidemiology, Johnson and Johnson Pharmaceu-tical Research and Development, Titusville, NJ, USA. E‐mail: zyuan6@its.jnj.com

Copyright © 2011 John Wiley & Sons, Ltd.

benefit‐risk section of the European MedicinesAgency’s Critical Assessment Report Template andin several industry and regulatory initiatives ongoingto consider benefit‐risk methodologies and structured“framework” approaches,7–11 it would help make aclear distinction between value judgments and dataand lend transparency, and perhaps reproducibility, tothe decision‐making process. Although qualitativeclinical judgment is critical and irreplaceable, weargue here that quantitative methods will be enor-mously helpful in adding transparency to the processfor regulators, sponsors, patients, clinicians, and otherstakeholders.4,6 In this commentary, we briefly reviewseveral of the more straightforward quantitativemethods that are currently available for benefit‐riskevaluation with respect to their strengths and limita-tions. Although this review is by no means exhaustive,it is context for the remainder of the paper. In order toadvance the debate surrounding the field, we have alsoproposed a new methodology at the end of the articlethat may help move benefit‐risk evaluation in aconstructive, quantitative direction.In clinical studies, most often, efficacy (benefit) and

safety (risk) results are compared between treatmentgroups and are examined separately, expressed eitherin terms of absolute differences (e.g., in means orproportions) or in terms of relative measures (e.g.,relative risk reductions or hazard ratios). Althoughsuch numeric presentations seem straightforward,clinical interpretation of those measures may notalways be easy for patients and clinicians. As one wayto generate clinically useful measures of the con-sequences of treatment, Laupacis12 introduced theconcept of “number needed to treat” (NNT), which isdefined as the number of patients who need to betreated to prevent one additional unfavorable event.

z. yuan ET AL.654

NNT applies only to dichotomously defined out-comes, and mathematically, it is equivalent to thereciprocal of the absolute risk reduction as follows:

NNT= 1/(PC –PT)

where PC is the rate of an unfavorable outcome event(%) in the control group, and PT is the rate (%) in thetreatment group. When safety (harm) is the primaryinterest, some researchers use the concept of numberneeded to harm (NNH) as an analog of NNT. Theintroduction of NNT and NNH represents a meaning-ful advancement in benefit‐risk evaluation of amedicinal product with a quantitative measure andcan be relatively easily interpreted by doctors.13 Forexample, based on a meta‐analysis of 28 clinical trialscomparing drug‐eluting stents (DES) with bare‐metalstents (BMS), Roukoz14 reported (Table 1) that theevent rates for target vessel revascularization (TVR)were 8.4% for DES and 20.9% for BMS (meanfollow‐up of 29.6months), representing a relative risk(RR) = 0.31 (p< 0.001). The event rates for stentthrombosis were 1.4% and 1.3% for DES and BMS,respectively, representing a RR= 1.13 (p = 0.48). Inthe context of NNT and NNH, those results suggestthat eight patients need to be treated with DES toprevent one additional TVR as compared with BMS,whereas 1000 patients need to be treated with DES toobserve one additional stent thrombosis as comparedwith BMS. Simply put, if NNT<NNH, as in this case,treatment benefits outweigh the risks, at least in asetting when equal “importance” can be placed on thebenefit and the harm. Putting statistical uncertaintyaside for the moment, in this example, NNT and NNHseem more interpretable and comprehensible than RR,a commonly used traditional measure of treatmenteffects.However, as suggested earlier, an interesting

dilemma is that NNT does not distinguish clinicalsignificance of one type of event from another (e.g., inthe previous calculation, TVR and stent thrombosiswere treated as equivalent), particularly in a situation

Table 1. Number needed to treat for target vessel revascularization and stent th

Event rate Drug‐eluting stent,%

Bare‐metal sten%

Target lesion revascularization(overall)

8.4 20.9

Stent thrombosis (overall) 1.4 1.3

†Roukoz et al.14 (partially reproduced from Table 3, except for risk difference a*p= 0.48.**p< 0.001.

Copyright © 2011 John Wiley & Sons, Ltd.

where a treatment might be associated with multipleside effects that carry different levels of severity andconsequences. Additionally, NNT and NNH compar-isons do not lend themselves to comparing more thantwo outcomes at a time. To address this issue, Guyatt15

proposed the use of a threshold NNT that involvesspecifying the relative value or clinical impact associ-ated with preventing the target outcome (i.e., diseaseof interest) versus incurring the adverse effects andinconvenience associated with treatment. Relative valuewas assessed with “utility,” which can roughly beconsidered a “weight” on a 0 to 1 scale, where 1 is theweight associatedwith perfect health, and 0 is theweightassociated with immediate death.Holden and colleagues16 carried this thinking a step

further, explicitly defining the concept of relative utilityvalue, which can be calculated as follows:

RV= (1 – utility of AE)/(1 – utility of disease of interest)

Several methods can be used to assess utility values,and other approaches to assigning relative values alsoexist and can be used instead of utilities, for example,stated choice/conjoint analysis, multi‐criteria decisionanalysis, or quality‐adjusted life years, but thespecifics and the corresponding extensive literatureare beyond the scope of this paper. The importantprinciple is that different events carry differentweights for different stakeholders, and a quantitativeframework forces explicit specification of theseweights. When multiple adverse events are likely tobe of interest, Holden et al. define the RV‐adjustedNNH as follows:

RV− adjusted NNH=1/[(q21− q11) *RV1 + (q22− q12) *RV2 +…+ (q2k− q1k) *RVk]

where q1i, i= 1, 2,…, k represents the event rate of the ith

AE in the control group and q2i, i= 1, 2,…, k representsthe event rate of ith AE in the investigational treatmentgroup. In this context, a favorable benefit‐risk profile isdemonstrated, if NNT<RV−adjusted NNH.

rombosis†

t, Relativerisk

Risk difference (BMS−DES),%

NNT/NNH

0.31** 12.5 8

1.13* −0.1 −1000

nd NNT/NNH).

Pharmacoepidemiology and Drug Safety, 2011; 20: 653–656DOI: 10.1002/pds

benefit‐risk assessment 655

The method discussed earlier does not explicitlyaddress the issue related to overall net clinical benefit(NCB), that is, NNT is determined for individualendpoints rather than being combined across multiplebenefits. Composite endpoints that include bothefficacy and safety measures have been increasinglyused in reports of clinical trials. However, theinterpretation of such data without a proper weightingstrategy can be challenging, because not all clinicalendpoints are viewed as equally important byphysicians, patients, and regulators. In a recent articlerelated to the Food and Drug Administration’s reviewof prasugrel,17 a novel platelet inhibitor, Ungeracknowledged the difficulties surrounding the assess-ment of balance between the benefits (a compositeendpoint including death from cardiovascular causes,nonfatal myocardial infarction, and nonfatal stroke)and risks (bleeding). He asserted that “Not allobservers agree that the treatment effect was impor-tant, since many of the excess myocardial infarctionsin the clopidogrel group were not manifested clinicallybut were merely ‘enzyme leaks’ that were detectedduring routine monitoring in the peri‐interventionalperiod.” Nevertheless, following the Food and DrugAdministration’s review and at the end of the advisorymeeting, the author noted, “The agency and theadvisory committee concluded, however, that suchenzyme elevations indicate tissue damage and thatsuch damage has serious long‐term consequences.”,whereas the risk of “bleeding — an important adverseeffect, but one that is transient and does not result inincreases in strokes or deaths.”17

The extensive debate generated around prasugrelmay signal a need for a more structured, quantitativeevaluation, because different observers may havedifferent views on the relative clinical importanceassociated with myocardial infarction and bleeding.Different weights, whether formally or implicitlyapplied, could potentially lead to very differentindividual judgments regarding the acceptability ofthis new product. However, an agreed‐upon frame-work for a benefit‐risk discussion could reduce thelikelihood of two observers coming to vastly differentconclusions. It is possible, for example, that the totalexcess number of events (ENE), combining efficacyand safety with a proper, and mutually acceptable,weighting approach, could be a reasonable method ina given clinical situation. The concept of ENE isparticularly relevant from a public health perspec-tive.18 ENE is a simple mathematical transformationof the absolute risk difference, representing theabsolute excess risks, for example, expressed per10 000 hypothetical patients treated with each drug.

Copyright © 2011 John Wiley & Sons, Ltd.

With a proper weighting mechanism applied to theENE, the overall NCB can be expressed as follows:

NCB ¼ ∑n

i¼1ENEi *Wti

where i represents the individual endpoint (efficacy orsafety); ENEi represents the ENE between treatmentgroups for endpoint i, and Wti represents the weightfor an individual endpoint i. Weights can be derivedfrom utility values, preference studies, multi‐criteriadecision analysis, or other sources.19,20 For example,the Cost‐Effectiveness Analysis Registry is a compre-hensive database, containing a catalog of informationon 6700 cost‐effectiveness ratios and more than 9400utility weights for different health states.21 Stated‐choice methods can also be used to measurepreference weights for the endpoints of interest (i.e.,health states).22

We believe that the various methods describedabove are complementary to each other, because asingle approach is unlikely to be comprehensiveenough to quantify all issues associated with aproduct. We wholeheartedly agree with the statementby Unger that “Ultimately, deciding which drug ispreferable will be a matter of individual clinicaljudgment,” because one patient may place high valueon avoiding an unfavorable outcome of the disease,whereas another may place high value on avoiding theside effects. Having a quantitative framework couldpotentially help with individual doctor–patient discus-sions, although considerable research is needed to testthis hypothesis. When a new drug is being consideredfor approval, some quantitative framework for eval-uating benefits and risks could also help inform thoseregulatory decisions.6–8,10,11,19,20,23 Although it maybe impossible to eliminate the subjective nature of theassessment process, moving toward greater transpar-ency of regulatory decisions seems highly desirable.7

Science evolves as challenges arise, and the difficultyin creating a more standard approach to benefit‐riskassessment should not be put forth as an argumentagainst the need for a process with consistency. “If weknew what we were doing, it wouldn’t be calledresearch, would it?”—Albert Einstein (1879–1955)

DISCLAIMER

The views expressed are those of the authorscontributing to this paper and do not necessarilyrepresent the position or opinion of Johnson &Johnson or its affiliates.

Pharmacoepidemiology and Drug Safety, 2011; 20: 653–656DOI: 10.1002/pds

z. yuan ET AL.656

CONFLICT OF INTEREST

The authors are employees of Johnson and JohnsonPharmaceutical Research and Development, LLC,which is engaged in research and developmentactivities in health care and consumer products.

ETHICAL APPROVAL

This is a Commentary article. The manuscript wasreviewed and approved by J&J internal publicationcommittee, but the authors did not request for an ethicalapproval from any institutional or national ethicalcommittee.

REFERENCES

1. AstraZeneca Receives Action Letter From FDA for Exanta (ximelagatran).http://www.drugs.com/nda/exanta_041008.html [9 August 2010].

2. Merck Announces Voluntary Worldwide Withdrawal of VIOXX®. http://www.merck.com/newsroom/vioxx/pdf/vioxx_press_release_final.pdf [9 August 2010].

3. Brower V. ESAs further restricted, but debate continues. J Natl Cancer Inst2008; 100: 1344–51.

4. Eichler HG, Abadie E, Raine JM, Salmonson T. Safe drugs and the cost of goodintentions. N Engl J Med 2009; 360: 1378–80.

5. Working CIOMS Group IV. Benefit‐Risk Balance for Marketed Drugs:Evaluating Safety Signals. Council for International Organizations of MedicalSciences: Geneva, 1998.

6. EBE Report: Balancing Benefits and Risks: Sharing Responsibility. http://www.ebe‐biopharma.org/index.php?option=com_news&task=view&id=163&Ite-mid=67 [12 August 2010].

7. Breckenridge A. Regulatory challenges, reimbursement, and risk‐benefitassessment. Clin Pharmacol Ther 2010; 88: 153–4.

8. Walker S, Liberti L, McAuslane N. Refining the benefit‐risk framework for theassessment of medicines: valuing and weighting benefit and risk parameters.Clin Pharmacol Ther 2011; 89: 179–82.

Copyright © 2011 John Wiley & Sons, Ltd.

9. European Medicines Agency Benefit‐Risk Methodology Project, EMA/213482/2010 (2010). http://www.ema.europa.eu/ema/index.jsp?curl=pages/special_topics/document_listing/document_listing_000314.jsp&mid=WC0b01ac0580223ed6&-murl=menus/special_topics/special_topics.jsp&jsenabled=true [16 February 2011].

10. Coplan PM, Noel RA, Levitan BS, Ferguson J, Mussen F. Development of aframework for enhancing the transparency, reproducibility and communicationof the benefit‐risk balance of medicines. Clin Pharmacol Ther 2011; 89: 312–5.

11. Levitan BS, Andrews EB, Gilsenan A, et al. Application of the BRATframework to case studies: observations and insights. Clin Pharmacol Ther2011; 89: 217–24.

12. Laupacis A, Sackett DL, Roberts RS. An assessment of clinically usefulmeasures of the consequences of treatment. N Engl J Med 1988; 318: 1728–33.

13. Cook RJ, Sackett DL. The number needed to treat: a clinically useful measure oftreatment effect. BMJ 1995; 310: 452–4.

14. Roukoz H, Bavry AA, Sarkees ML, et al. Comprehensive meta‐analysis ondrug‐eluting stents versus bare‐metal stents during extended follow‐up. Am JMed 2009; 122: 581.e1–581.e10.

15. Guyatt GH, Sinclair J, Cook DJ, Glasziou P. Users’ guides to the medicalliterature: XVI. How to use a treatment recommendation. Evidence‐BasedMedicine Working Group and the Cochrane Applicability Methods WorkingGroup. JAMA 1999; 281: 1836–43.

16. Holden WL, Juhaeri J, Dai W. Benefit‐risk analysis: examples using quantitativemethods. Pharmacoepidemiol Drug Saf 2003; 12: 693–7.

17. Unger EF. Weighing benefits and risks‐‐the FDA’s review of prasugrel. N Engl JMed 2009; 361: 942–5.

18. O’Neill RT. A perspective on characterizing benefits and risks derived fromclinical trials: can we do more? Drug Inf J 2008; 42: 235–245.

19. Guo JJ, Pandey S, Doyle J, Bian B, Lis Y, Raisch DW. A review of quantitativerisk‐benefit methodologies for assessing drug safety and efficacy‐report of theISPOR risk‐benefit management working group. Value Health 2010; 13: 657–66.

20. Benefit‐risk Methodology Project: Work Package 2 Report: Applicability ofCurrent Tools and Processes for Regulatory Benefit‐risk Assessment. http://www.ema.europa.eu/docs/en_GB/document_library/Report/2010/10/WC500097750.pdf [16 February 2011].

21. Cost‐Effectiveness Analysis (CEA) Registry. https://research.tufts‐nemc.org/cear4/SearchingtheCEARegistry/SearchtheCEARegistry.aspx [17 February 2011].

22. Johnson FR, Hauber AB, Poulos CM. A brief introduction to the use of stated‐choice methods to measure preferences for treatment benefits and risks. RTIPress Publication No. RR‐0009‐0909. Research Triangle Park, NC: RTIInternational; 2009. http://www.rti.org/pubs/rr‐0015‐1102‐hogan.pdf

23. Reflection Paper on Benefit‐Risk Assessment Methods in the Context of theEvaluation of Marketing Authorisation Applications of Medicinal Products forHuman Use. EMEA 2008. http://www.emea.europa.eu.

Pharmacoepidemiology and Drug Safety, 2011; 20: 653–656DOI: 10.1002/pds