Evaluating the Public Health Impactof Health Promotion Interventions:The RE-AIM FrameworkRussell E. Glasgow, PhD, Thomas M. Vogt, MD, MPH, and Shawn M. Boles, PhD

Although the field of health promotionhas made substantial progress,1-13 our advancesare limited by the evaluation methods used. Wehave the potential to assess the population-based impact of our programs. However, withfew exceptions, evaluations have restrictedtheir focus to 1 or 2 of 5 "dimensions of qual-ity" we believe to be important.

There is a great need for researchmethods that are designed to evaluate thepublic health significance of interven-tions.'4 The efficacy-based research para-digm that dominates our current notions ofscience is limiting and not always the mostappropriate standard to apply.14" 5 A reduc-tionistic scientific paradigm oversimplifiesreality'1'8 in the quest to isolate efficacioustreatments. Most clinical trials focus oneliminating potential confounding variablesand involve homogeneous, highly moti-vated individuals without any health condi-tions other than the one being studied. Thisapproach provides important informationand strong internal validity; from an exter-nal validity perspective, however, it resultsin samples of nonrepresentative partici-pants and settings.'519'20

Similarly, the emphasis on developingclinically significant outcomes often pro-duces interventions that are intensive, expen-sive, and demanding of both patients andproviders.2' These interventions tend to bestudied in the rarified, "controlled" atmos-phere of specialty treatment centers usinghighly standardized protocols. This "effi-cacy" paradigm22 does not address how wella program works in the world ofbusy, under-staffed public health clinics, large health sys-tems, or community settings.'5

Our medical culture emphasizes phar-macosurgical interventions that produceimmediate results and whose dosage can beeasily defined and controlled. There is littleresearch on interventions that address wholepopulations, are long lasting, or become"institutionalized."23-26 Indeed, many inter-ventions that prove efficacious in randomizedtrials are much less effective in the generalpopuiation.14,15,19,27

In this commentary we describe theRE-AIM evaluation model, which empha-sizes the reach and representativeness ofboth participants and settings, and discussthe model's implications for public health

research. The representativeness of partici-pants9'92 is an important issue for outcomeresearch.20'29 The representativeness of set-tings-clinics, worksites, or communities-for public health interventions is equallyimportant. Many evaluations, such as theotherwise well-designed Community Inter-vention Trial for Smoking Cessation,30explicitly restrict selection of participatingcommunities (and research centers) to thosemost motivated, organized, and prepared forchange.30 This results in expert, highly moti-vated research teams and settings, whichare, by definition, unrepresentative of thesettings to which their results are to beapplied.

Recognizing some of the foregoingissues, both the National Cancer Instituteand the National Heart, Lung, and BloodInstitute have proposed sequential "stages"of research.'4'22'3' These steps move fromhypothesis generation to testing under con-trolled conditions, evaluations in "defined pop-ulations," and, finally, dissemination research.Interventions found to be efficacious thenundergo "effectiveness" evaluations, and pro-grams that prove to be effective-especiallycost-effective22'32-are selected for dissemi-nation research.

There is often difficulty, however, inmaking the transition across phases. We thinkthis may be due to a flaw in the basic model,in that many characteristics that make anintervention efficacious (e.g., level of inten-sity of the intervention and whether it isdesigned for motivated, homogeneous popu-lations) work against its being effective inmore complex, less advantageous settingswith less motivated patients and overworkedstaff.8 33'34 Low-intensity interventions thatare less efficacious but that can be deliveredto large numbers of people may have a morepervasive iMpaCt.3537

Russell E. Glasgow is with the AMC CancerResearch Center, Denver, Colo. Thomas M. Vogt iswith the Kaiser Permanente Center for HealthResearch, Honolulu, Hawaii. Shawn M. Boles iswith the Oregon Research Institute, Eugene, OR.

Requests for reprints should be sent to RussellE. Glasgow, PhD, AMC Cancer Research Center,1600 Pierce St, Denver, CO 80214 (e-mail: glasgowr@amc.org).

This paper was accepted April 17, 1999.

September 1999, Vol. 89, No. 9

Commentaries

Abrams and colleagues38 defined theimpact of an intervention as the product ofa program's reach, or the percentage ofpopulation receiving the intervention, andits efficacy (I = R X E). We expand on this"RE" (Reach x Efficacy) concept by adding3 dimensions that apply to the settings inwhich research is conducted (Adoption,Implementation, and Maintenance: "AIM")to more completely characterize the publichealth impact of an intervention.

RE-AIMModelWe conceptualize the public health

impact of an intervention as a function of5 factors: reach, efficacy, adoption, imple-mentation, and maintenance. Each of the5 RE-AIM dimensions is represented on a 0to 1 (or 0% to 100%) scale.

This framework is compatible with sys-tems-based and social-ecological think-ing 16,39,40 as well as community-based andpublic health interventions.41'42 A centraltenet is that the ultimate impact of an inter-vention is due to its combined effects on 5evaluative dimensions. The RE-AIM modelexpands on earlier workl2'38'43 and is summa-rized in Table 1.

Reach

Reach is an individual-level measure(e.g., patient or employee) of participation.Reach refers to the percentage and risk char-acteristics of persons who receive or areaffected by a policy or program. It is mea-sured by comparing records ofprogram par-ticipants and complete sample or "census"information for a defined population, suchas all members in a given clinic, healthmaintenance organization, or worksite. Ifaccurate records are kept of both the numer-ator (participants) and the denominator (pop-ulation), calculation of participation rates isstraightforward.

Reach (as well as adoption) also con-cerns the characteristics of participants.Assessing representativeness is challeng-ing.20 43 It requires demographic informa-tion-and preferably psychosocial, medicalhistory, or case mix information-on non-participants as well as participants. Detailedinformation on nonparticipants is oftenchallenging to collect and raises ethicalissues in that nonparticipants have typicallynot consented to be studied.2845 Coopera-tive arrangements that permit investigationof the extent to which participants are rep-resentative of the larger "denominator"population should be a priority for futureresearch.

Unfortunately, participants in healthpromotion activities sometimes are thosewho need them least (e.g., the "worriedwell,"46'47 those in the more affluent seg-ments of the population, and nonsmokers).20With the increasing gap between the "haves"and "have-nots" in our country,48 and thedramatic impact of socioeconomic status onhealth status,49 understanding the degree towhich a program reaches those in need isvital. Because public health interventions areaddressed to large numbers of people, evensmall differences in risk levels between par-ticipants and nonparticipants can have a sig-nificant impact on cost-effectiveness.35

EfficacyEntire textbooks have been devoted to

evaluating the efficacy of interventions.4450 51We discuss 2 specific issues: the importanceof assessing both positive and negative con-sequences of programs and the need toinclude behavioral, quality of life, and par-ticipant satisfaction outcomes as well asphysiologic endpoints.

Positive and negative outcomes.Most population-based evaluations focuson improvement in some targeted health orrisk indicator. Interventions delivered tolarge populations can also have unantici-pated negative effects. Labeling someonewith a potential illness may have profoundsocial and psychological consequences.52'53Many effective services remain underdeliv-ered, while others are delivered that are notnecessary or effective in the groups receiv-ing them. Even services that cost only a fewdollars can have substantial negative (as wellas positive) societal effects, including mis-placed resources and large opportunity costs,when delivered to millions of people. It iscritical not only to determine benefits butalso to be certain that harm does not out-weigh benefits.

Outcomes to be measured. Clinicalresearch emphasizes biologic outcomes-in particular, disease risk factors"'54-andconcerns about limited resources have ledto an increasing emphasis on health careuse.8'55'56 Such outcomes are important, buta public health evaluation should includemore than simply biologic and use mea-sures. Two other types of outcomes meritinclusion. First, behavioral outcomesshould be assessed for participants (e.g.,smoking cessation, eating patterns, physi-cal activity), for staff who deliver an inter-vention (approaching patients, deliveringprompts and counseling, making follow-upcalls), and for the payers and purchasers whosupport the intervention (adopting an inter-vention, changing policies). Second, a partic-

ipant-centered quality-of-life perspective'857should be included to allow evaluation ofpatient functioning, mental health, and con-sumer satisfaction, since these factors pro-vide a critical check on the impact ofdeliverypractices.

Adoption

Adoption refers to the proportion andrepresentativeness of settings (such as work-sites, health departments, or communities)that adopt a given policy or program.58There are common temporal patterns in thetype and percentage of settings that willadopt an innovative change.43'59 Adoption isusually assessed by direct observation orstructured interviews or surveys. Barriers toadoption should also be examined whennonparticipating settings are assessed.

Implementation

The term effectiveness is used todescribe evaluations conducted in real-worldsettings by individuals who are not part of aresearch staff.22'3' Implementation refers tothe extent to which a program is delivered asintended. It can be thought of as interactingwith efficacy to determine effectiveness(Efficacy X Implementation = Effective-ness). There are both individual-level andprogram-level measures of implementation.

At the individual level, measures ofparticipant follow-through or "adherence"to regimens are necessary for interpretingstudy outcomes.60 6 At the setting level, theextent to which staff members deliver theintervention as intended is important.Stevens et al.62 demonstrated that differen-tial levels of protocol implementation were,in large part, the reason that a brief hospital-based smoking-cessation program was moresuccessful when implemented by researchstaff than by hospital respiratory therapystaff. Implementation research is crucial indetermining which of a set of interventionsmay be practical enough to be effective inrepresentative settings.Maintenance

A major challenge at both individualand organization-community levels is long-term maintenance of behavior change.24'63'64At the individual level, relapse followinginitial behavior change is ubiquitous.65'66Equally essential is the collection of pro-gram-level measures of institutionalization,25that is, the extent to which a health promo-tion practice or policy becomes routine andpart of the everyday culture and norms ofan organization. Recently, there have been

American Journal of Public Health 1323September 1999, Vol. 89, No. 9

Commentaries

advances in identifying factors related tothe extent to which a change is institutional-ized.23'25 At the community level, mainte-nance research is needed to document theextent to which policies are enforced overtime (e.g., laws concerning alcohol sales, no-smoking policies). Maintenance measuresthe extent to which innovations become arelatively stable, enduring part of the behav-ioral repertoire of an individual (or organiza-tion or community).

Combining Dimensions

The public health impact score, repre-sented as a multiplicative combination of thecomponent dimensions (Table 1), is probablythe best overall representation of quality. TheRE-AIM model is silent on the choice of effi-cacy measure; any outcome that is quantifi-able, reliable, valid, and important to scientific,citizen, and practitioner communities isadmissible. Examples include hypertension,mammography screening, and smoking status.

Implicit in the constructs of implemen-tation and maintenance is the length of theperiod during which data are collected: aminimum of 6 months to 1 year for imple-mentation and 2 years or longer for mainte-nance. Frequency of assessment should bebased on the particular issue, goals, and set-ting. If RE-AIM dimensions are assessedmultiple times, then a RE-AIM profile can beplotted. Repeated measurements and visualdisplays67.68 can enhance understanding ofintervention effects and be used to comparedifferent interventions (Figure 1). (Additionaltables and figures related to application oftheRE-AIM framework are available at www.ori.org/-shawn/public/reaim/reaim.long.pdf.)

DiscussionThe last several years have seen a vari-

ety of provocative articles on changing para-digms of health care. 8' 234 69-73 Unfortu-nately, there have been few discussions ofevaluation models for these new population-based paradigms. Even economic analysesand outcomes research32 do not address sev-eral of the core evaluation issues and keydimensions of these evolving approaches.Evaluation methods must match the concep-tual issues and interventions being studied.With the shift to a multiple causation andholistic or systems approach to medical sci-

12,17,3573,74ence, recognition of the complex-ity and various levels of disease determi-nants is required.38'7577

While classic randomized controlledtrials have significantly advanced ourknowledge of pharmacotherapy and medi-

cosurgical interventions,'4478 they have limi-tations when applied to behavioral issuesand, especially, community interven-tions.51 79-83 Randomized controlled trialsemphasize efficacy to the de facto exclusionof factors such as adoption, reach, and insti-tutionalization.51,79'80 RE-AIM provides aframework for determining what programsare worth sustained investment and for iden-tifying those that work in real-world environ-ments. RE-AIM can be used to evaluate ran-domized controlled studies as well as studieswith other designs, and it is compatible withevidence-based medicine; RE-AIM asserts,however, that evidence should be broadenedto include dimensions in addition to efficacy.The model can also be used to guide qualita-tive research efforts by focusing inquiry oneach of these issues. To the extent that RE-AIM dimensions are incorporated into eval-uations, decision makers will have morecomplete information on which to baseadoption or discontinuance of programs.

Data collected via the RE-AIM modelcan serve several evaluative purposes: (1)assessing an intervention's overall publichealth impact, (2) comparing the publichealth impact of an intervention acrossorganizational units or over time, (3) com-paring 2 or more interventions across RE-AIM dimensions (Figure 1), and (4) mak-ing decisions about redistributing resourcestoward more effective programs.

Future Research and Policy Issues

There are several implications of theRE-AIM model. Empirical evaluationsinvolving these implications would greatlyhelp in assessing the utility of the model andinforming policy and funding decisions.

1. From the RE-AIM perspective, weexpect that programs that are very effica-cious (under highly controlled, optimal con-

ditions) may have poor implementationresults. Such an inverse relationship betweenprogram efficacy and implementation, orbetween reach and efficacy,43 has significantimplications for the types of interventionsthat should receive high priority.

2. The RE-AIM model does not explic-itly include economic factors.32 However,cost issues are addressed in 3 ways. First,we think that cost is often a major factordetermining whether a program will beadopted, implemented consistently, ormaintained.8486 This hypothesis should betested and substantiated or refuted. Sec-ond, cost-effectiveness and cost-benefitare certainly appropriate outcomes. Theydetermine how well resources are beingused and whether or not more good couldbe accomplished through alternative uses(opportunity costs). Finally, a population-based cost-effectiveness index could becalculated by dividing the resulting publichealth impact by the total societal costs32of a program. Dividing each RE-AIMcomponent score by the costs relevant tothat dimension could help identify areas ofefficiency and waste. There is a need forfurther work on similar formulas and eval-uation of the extent to which providingdecision makers with information on RE-AIM dimensions influences decisions.

3. Systematic reviews that determinethe extent to which different research fieldshave studied-or neglected-each of theRE-AIM dimensions are needed. Wehypothesize that adoption and maintenance-institutionalization will be the most under-studied dimensions, but this needs to bedocumented for different research topics.

Limitations ofRE-AIM

The precise nature of the relationshipsamong the 5 RE-AIM dimensions, and how

1324 American Journal of Public Health

TABLE 1-RE-AIM Evaluation Dimensions

Dimensiona Level

Reach (proportion of the target population that Individualparticipated in the intervention)

Efficacy (success rate if implemented as in Individualguidelines; defined as positive outcomesminus negative outcomes)

Adoption (proportion of settings, practices, Organizationand plans that will adopt this intervention)

Implementation (extent to which the intervention is Organizationimplemented as intended in the real world)

Maintenance (extent to which a program is Individual and organizationsustained over time)

aThe product of the 5 dimensions is the public health impact score (population-basedeffect).

September 1999, Vol. 89, No. 9

Commentaries

High efficacy and high-cost intervention--- Low efficacy and low-cost intervention

0.9

0.7 -

-a 0.5c

E

Z 0.3-w

0.1IReach Efficacy Adoption Implmentation Maintenance

FIGURE 1-Display of 2 different intervention programs on various RE-AIMdimensions.

they combine to determine overall publichealth impact, is unknown. We have repre-sented these factors as interacting multiplica-tively because we believe that this is closer toreality than an additive model. A highly effi-cacious program that is adopted by few clin-ics or that reaches only a small proportion ofeligible citizens will have little population-based impact. In future research, it will benecessary to determine the precise mathe-matical functions that best characterize theinterplay ofthese dimensions.

In this initial model, we have implicitlyassumed, in the absence of data to the con-trary, that all 5 RE-AIM dimensions areequally important and therefore equallyweighted. This may not always be the case.In situations in which 1 or more of theRE-AIM dimensions are considered mostimportant, differential weights could beassigned. Similarly, it may not be necessaryto assess all RE-AIM components in everystudy.

Finally, the time intervals we havesuggested for assessing implementation(6 months-I year) and maintenance(2+ years) are arbitrary. Future researchis needed to determine whether there arenecessary or optimal intervals for evaluat-ing these dimensions.

ConclusionPublic health interventions should be

evaluated more comprehensively than has tra-ditionally been done.42'55'7' Dimensions suchas reach, adoption, and implementation are

crucial in evaluating programs intended forwide-scale dissemination. We hope that theRE-AIM framework, or similar models thatfocus on overall population-based impact, willbe used to more filly evaluate public healthinnovations. Such an evaluation frameworkhelps remind us of the key purposes ofpub-lic health, organizational change, and com-munity interventions.71,83,87,88 It is time to re-aim our evaluation efforts. D

ContributorsR. E. Glasgow originated the idea for the RE-AIMframework and served as editor. T. M. Vogt draftedsections on implications and coined the acronymRE-AIM. S. M. Boles drafted content on the varioususes of the model and contributed the figure andideas about displays. All authors contributed sub-stantially to the writing of the paper.

AcknowledgmentsPreparation of this manuscript was supported bygrants ROI DK51581, ROI DK35524, and ROIHL52538 from the National Institutes of Health.

Appreciation is expressed to Drs Gary Cutter,Mark Dignan, Ed Lichtenstein, and Deborah Toobertfor their helpfuil comments on an earlier version ofthis article.

References1. Bandura A. Social Foundations ofThought and

Action: A Social Cognitive Theory. EnglewoodCliffs, NJ: Prentice Hall; 1986.

2. Green LW, Kreuter MW. Health PromotionPlanning: An Education and EnvironmentalApproach. Mountain View, Calif: MayfieldPublishing Co; 1991.

3. Green LW, Richard L, Potvin L. Ecologicalfoundations of health promotion. Am J HealthPromotion. 1996;10:270-281.

4. Glanz K, Rimer BK. Theory at a Glance. AGuide for Health Promotion Practice. Beth-esda, Md: National Cancer Institute; 1995.

5. Glasgow RE, La Chance P, Toobert DJ, et al.Long term effects and costs of brief behavioraldietary intervention for patients with diabetesdelivered from the medical office. Patient EducCounseling. 1997;32:175-184.

6. Meenan RT, Stevens VJ, Hornbrook MC, et al.Cost-effectiveness ofa hospital-based smoking-cessation intervention. Med Care. 1998;36:670-678.

7. Wasson J, Gaudette C, Whaley F, et al. Tele-phone care as a substitute for routine clinic fol-low-up. JAMA. 1992;267:1788-1793.

8. Kaplan RM. The Hippocratic Predicament:Affordability, Access, and Accountability inAmerican Health Care. San Diego, Calif: Acad-emic Press Inc; 1993.

9. Health Plan Employer Data and InformationSet 3.0. Washington, DC: National Committeefor Quality Assurance; 1996.

10. Ferguson T. Health Online. Reading, Mass:Addison-Wesley Publishing Co; 1996.

11. Street RL Jr, Rimal RN. Health promotion andinteractive technology: a conceptual founda-tion. In: Street RL Jr, Gold WR, Manning T,eds. Health Promotion and Interactive Technol-ogy: Theoretical Applications and FutureDirections. Mahwah, NJ: Lawrence ErlbaumAssociates; 1997:1-18.

12. Glasgow RE, Wagner E, Kaplan RM, et al. Ifdiabetes is a public health problem, why nottreat it as one? A population-based approach tochronic illness. Ann Behav Med. 1999;21:1-13.

13. Shaw KM, Bulpitt CJ, Bloom A. Side effects oftherapy in diabetes evaluated by a self-adminis-tered questionnaire. J Chronic Dis. 1977;30:39-48.

14. Sorensen G, Emmons KM, Dobson AJ. Theimplications of the results of community inter-vention trials. Annu Rev Public Health. 1998;19:379-416.

15. Starfield B. Quality-of-care research: internalelegance and external relevance [commentary].JAMA. 1998;280: 1006-1008.

16. Stokols D. Establishing and maintaining healthyenvironments: toward a social ecology of healthpromotion. Am Psychol. 1992;47:6-22.

17. Capra F. The Web ofLife: A New UnderstandingofLiving Systems. New York, NY: Doubleday;1997.

18. Gould SJ. Full House: The Spread ofExcel-lence From Plato to Darwin. Pittsburgh, Pa:Three Rivers Press; 1997.

19. Glasgow RE, Eakin EG, Toobert DJ. How gen-eralizable are the results of diabetes self-man-agement research? The impact of participationand attrition. Diabetes Educ. 1996;22:573-585.

20. Conrad P. Who comes to worksite wellness pro-grams? A preliminary review. J Occup Med.1987;29:317-320.

21. DCCT Research Group. Lifetime benefits andcosts of intensive therapy as practiced in theDiabetes Control and Complications Trial.JAMAX. 1996;276:1409-1415.

22. Flay BR. Efficacy and effectiveness trials (andother phases of research) in the development of

September 1999, Vol. 89, No. 9 American Joumal of Public Health 132Z

Commentaries

health promotion programs. Prev Med. 1986;15:451-474.

23. Hofstede G. Cultures and Organization. Soft-ware ofthe Mind. New York, NY: McGraw-HillInternational Book Co; 1997.

24. Goodman RM, StecklerA. The life and death ofa health promotion program: an institutionaliza-tion perspective. Int Q Community HealthEduc. 1988;89:5-19.

25. Goodman RM, McLeroy KR, Steckler A, et al.Development of level of institutionalizationscales for health promotion programs. HealthEduc Q. 1993;20:161-178.

26. Clarke GN. Improving the transition from basicefficacy research to effectiveness studies:methodological issues and procedures. J Con-sult Clin Psychol. 1995;63:718-725.

27. Paul GL, Lentz RJ. Psychosocial Treatment ofChronic Mental Patients. Cambridge, Mass:Harvard University Press; 1977.

28. Dertouzos ML. What Will Be: How the New WorldofInformation Will Change Our Lives. New York,NY: HarperEdge Publishers Inc; 1997.

29. Glasgow RE, Fisher E, Anderson BJ, et al.Behavioral science in diabetes: contributionsand opportunities. Diabetes Care. 1999;22:832-843.

30. COMMIT Research Group. Community Inter-vention Trial for Smoking Cessation (COM-MIT): summary of design and intervention.J Natl Cancer Inst. 1991;83:1620-1628.

31. Cullen JW. Design of cancer prevention studies.Cancer Detect Prev. 1986;9: 125-138.

32. Gold MR, Siegel JE, Russell LB, et al. Cost-Effectiveness in Health and Medicine. NewYork, NY: Oxford University Press Inc; 1996.

33. Bodenheimer TS, Grumbach KE. Understand-ing Health Policy: A Clinical Approach. Stam-ford, Conn: Appleton & Lange; 1995.

34. Califano JA Jr. Radical Surgery: What's Nextfor America s Health Care. New York, NY:Random House; 1994.

35. Vogt TM, Hollis JF, Lichtenstein E, et al. Themedical care system and prevention: the need fora new paradigm. HMO Pract. 1998; 12:6-14.

36. Kristein MM, Arnold CB, Wynder EL. Healtheconomics and preventive care. Science. 1977;195:457-462.

37. Hatziandrew EJ, Sacks JJ, Brown R, et al. Thecost-effectiveness of three programs to increaseuse of bicycle helmets among children. PublicHealth Rep. 1995;1 10:251-259.

38. Abrams DB, Orleans CT, Niaura RS, et al. Integrat-ing individual and public health perspectives fortreatment of tobacco dependence under managedhealth care: a combined stepped care and matchingmodel. Ann Intern Med. 1996;18:290-304.

39. Green LW, Kreuter MW Behavioral and envi-ronmental diagnosis. In: Green LW, KreuterMW, eds. Health Promotion and Planning: AnEducational and Environmental Approach.Mountain View, Calif: Mayfield Publishing Co;1991:125-149.

40. Stokols D. Translating social ecological theoryinto guidelines for community health promotion.Am JHealth Promotion. 1996;10:282-298.

41. Bracht N. Community Organization Strategiesfor Health Promotion. Newbury Park, Calif:Sage Publications; 1990.

42. Green LW, Johnson JL. Dissemination and uti-lization of health promotion and disease pre-vention knowledge: theory, research and experi-ence. Can J Public Health. 1996;87(suppl 2):Sll-S17.

43. Glasgow RE, McCaul KD, Fisher KJ. Partici-pation in worksite health promotion: a critiqueof the literature and recommendations forfuture practice. Health Educ Q. 1993;20:391-408.

44. Meinert CL. Clinical Trials: Design, ConductandAnalysis. New York, NY: Oxford UniversityPress Inc; 1986.

45. Rind DM, Kohane IS, Szolovits P, et al. Main-taining the confidentiality of medical recordsshared over the Internet and the World WideWeb. Ann Intern Med. 1997;127:138-141.

46. Vogt TM, LaChance PA, Glass A. Screeninginterval and stage at diagnosis for cervical andbreast cancer. Paper presented at: Annual Meet-ing of the American Society of PreventiveOncology; March 1998; Bethesda, Md.

47. Emmons KM. Maximizing cancer risk reduc-tion efforts: addressing risk factors simultane-ously. Cancer Causes Control. 1997;8(suppl 1):S31-S34.

48. Athanasiou T. Divided Planet: The Ecology ofRich and Poor. Boston, Mass: Little Brown &Co Inc; 1996.

49. Marmot MG, Bobak M, Smith GD. Explana-tions for social inequities in health. In: AmickBC, Levine S, Tarlov AR, Walsh DC, eds. Soci-ety and Health. New York, NY: Oxford Univer-sity Press Inc; 1995:172-2 10.

50. Cook TD, Campbell DT. Quasi-ExperimentalDesign andAnalysis Issues for Field Settings.Chicago, Ill: Rand McNally; 1979.

51. Bradley C. Designing medical and educationalintervention studies. Diabetes Care. 1993; 16:509-518.

52. Bloom JR, Monterossa S. Hypertension label-ing and sense of well-being. Am J PublicHealth. 1981;71:1228-1232.

53. Downs WR, Robertson JF, Harrison LR. Con-trol theory, labeling theory, and the delivery ofservices for drug abuse to adolescents. Adoles-cence. 1997;32:1-24.

54. Glasgow RE, Osteen VL. Evaluating diabeteseducation: are we measuring the most importantoutcomes? Diabetes Care. 1992;15:1423-1432.

55. Nutbeam D. Evaluating health promotion-progress, problems and solutions. Health Pro-motion Int. 1998;13:27-44.

56. Vinicor F. Is diabetes a public-health disorder?Diabetes Care. 1994;17:22-27.

57. Spilker B. Quality of Life in Clinical Trials.New York, NY: Raven Press; 1990.

58. Emont SL, Choi WS, Novotny TE, et al. Cleanindoor air legislation, taxation, and smokingbehaviour in the United States: an ecologicalanalysis. Tob Control. 1992;2:13-17.

59. Rogers EM. Diffusion of Innovations. NewYork, NY: Free Press; 1983.

60. Glasgow RE. Compliance to diabetes regimens:conceptualization, complexity, and determinants.In: Cramer JA, Spilker B, eds. Patient Compli-ance in Medical Practice and Clinical Trials.New York, NY: Raven Press; 1991:209-221.

61. Mahoney MJ, Thoresen CE. Self-Control:Power to the Person. Monterey, Calif: Brooks/Cole; 1974.

62. Stevens VJ, Glasgow RE, Hollis JF, et al. Imple-mentation and effectiveness of a brief smokingcessation intervention for hospital patients. MedCare. In press.

63. Marlatt GA, Gordon JR. Determinants ofrelapse: implications for the maintenance ofbehavior change. In: Davidson P, Davidson S,eds. Behavioral Medicine: Changing HealthLifestyles. New York, NY: Brunner/Mazel;1980:410-452.

64. Steckler A, Goodman RM. How to institution-alize health promotion programs. Am J HealthPromotion. 1989;3:34 44.

65. Marlatt GA, Gordon JR. Relapse Prevention:Maintenance Strategies in the Treatment ofAddictive Behaviors. New York, NY: GuilfordPress; 1985.

66. Stunkard AJ, Stellar E. Eating and Its Disor-ders. New York, NY: Raven Press; 1984.

67. Cleveland WS. The Elements of GraphingData. Monterey, Calif: Wadsworth AdvancedBooks and Software; 1985.

68. Tufte ER. Visual Explanations: Images andQuantities, Evidence and Narrative. Cheshire,Conn: Graphics Press; 1997.

69. Greenlick MR. Educating physicians for popu-lation-based clinical practice. JAMA. 1992;267:1645-1648.

70. Von Korff M, Gruman J, Schaefer J, et al. Col-laborative management of chronic illness. AnnBehav Med. 1997; 127:1097-1102.

71. Abrams DB, Emmons K, Niaura RD, et al.Tobacco dependence: an integration of individ-ual and public health perspectives. In: NathanPE, Langenbucher JW, McCrady BS, Franken-stein W, eds. The Annual Review ofAddictionsTreatment and Research. New York, NY: Perga-mon Press; 1991:391-436.

72. Sobel D. Rethinking medicine: improvinghealth outcomes with cost-effective psychoso-cial interventions. Psychosom Med. 1995;57:234-244.

73. McKinlay JB. A tale of 3 tails. Am J PublicHealth. 1999;89:295-298.

74. Stokols D, Pelletier KR, Fielding JE. The ecol-ogy of work and health: research and policydirections for the promotion of employeehealth. Health Educ Q. 1996;23: 137-158.

75. Glasgow RE, Eakin EG. Issues in diabetes self-management. In: Shumaker SA, Schron EB,Ockene JK, McBee WL, eds. The Handbook ofHealth Behavior Change. New York, NY:Springer Publishing Co; 1998:435-461.

76. Amick BC III, Levine S, Tarlov AR, et al. Soci-ety and Health. New York, NY: Oxford Univer-sity Press Inc; 1995.

77. Gochman DS. Handbook ofHealth BehaviorResearch II. NewYork, NY: Plenum Press; 1997.

78. Kazdin AE. Research Design in Clinical Psy-chology. Boston, Mass: Allyn & Bacon; 1992.

79. Bradley C. Clinical trials-time for a paradigmshift? Diabet Med. 1988;5:107-109.

80. Pincus T. Randomized controlled clinical trialsversus consecutive patient questionnaire data-bases. Paper presented at: Annual Conference

1326 American Journal of Public Health September 1999, Vol. 89, No. 9

Commentaries

of the Society of Behavioral Medicine; March1998; Rockville, Md.

81. Biglan A. Changing Cultural Practices: A Con-textualist Frameworkfor Intervention Research.Reno, Nev: Context Press; 1995.

82. Weiss SM. Community health demonstrationprograms. In: Matarazzo JD, Weiss SM, Herd JA,Miller NE, eds. Behavioral Health: A HandbookofHealth Enhancement and Disease Prevention.New York, NY: John Wiley & Sons Inc; 1984.

83. Fisher EB Jr. The results of the COMMIT Trial[editorial].Am JPublic Health. 1995;85:159-160.

84. Smith TJ, Hillner BE, Desh CE. Efficacyand cost-effectiveness of cancer treatment:rational allocation of resources based ondecision analysis. J Natl Cancer Inst. 1993;85:1460-1473.

85. Russel LB. Some of the tough decisionsrequired by a national health plan. Science.1989;246:8923-8926.

86. Aaron H, Schwartz WB. Rationing health care:the choice before us. Science. 1990;247:418-422.

87. Lichtenstein E, Glasgow RE. Smoking cessa-tion: what have we learned over the pastdecade? J Consult Clin Psychol. 1992;60:518-527.

88. Susser M. The tribulations of trials-interven-tion in communities [editorial]. Am J PublicHealth. 1995;85:156-158.

-~~~ ~ ~ ~ ~ -

Community-Oriented Primary Care:Health Care for the 21st CenturyEdited by Robert Rhyne, MD, Richard Bogue, PhD, 's AGary Kukulka, PhD, and Hugh Fulmer, MDThis book will give insight into: l* How medicine, health systems, non-health community leaders, and social services can be supportive as

America's public health practice continues to be challenged, restructured and redefined* New models of community-oriented primary care in a wide range of settings* Methods and interventions on population-derived health needs* Health promotion and disease prevention as part of the overall reorganization of health services* Understanding how community-oriented primary care values can complement managed care and

community benefit programs for the 21st Century* This practical, user-friendly compendium of specific skills and techniques for implementing a com-

munity-oriented primary care process includes how-to instructions and topics not normally learnedin health professional education.

$27.00 for APHA members* * $39.00 for non-members1998, softcover, 228 pp., Stock No. 0875532365/CDAD98

American Public Health Association * Publications SalesPO Box 753, Waldorf, MD 20604-0753.Voice: (301) 893-1894; Fax: (301) 843-0159E-mail: TASCO1@APHA.ORG; Web: www.apha.org *APHA members may purchase up to 2 copies at this price

September 1999, Vol. 89, No. 9 American Journal of Public Health 1327