Practice-Based Evidence: 45 Years ofPsychotherapy’s Effectiveness in a

Private Practice

PAUL CLEMENT, Ph.D., ABPP*

Of 2,259 patients seen during 45 years of private practice, outcome data wasproduced for 1,599 cases. The mean (SD) number of sessions per case was18.82 (29.89). The dropout rate was 18.76%. Of all treated cases withoutcome data 4 (0.25%) were rated as Much Worse; 11 (0.69%), Worse; 497(31.08%), No Change from Intake; 546 (34.15%), Improved; and 541(33.83%), Much Improved. The mean (SD) pre-/post-treatment effect size(ES) was 1.90 (1.61), the median was 1.62, and the range was from �2.91to �15.22. Patients and parents of minors rated outcomes more positivelythan the therapist did. Outcome varied significantly across diagnostic cate-gories. There was a significant, positive relationship in length of treatmentand outcome. The therapist’s effectiveness did not improve across the years.Years with the largest patient caseloads or the greatest proportion of patientswith managed-care insurance tended to show the poorest outcomes.1

KEYWORDS: practice-based evidence; private practice; treatmenteffectiveness; treatment effect size; evidence-based practice

INTRODUCTION

About 60 years ago Eysenck (1952) reviewed the published research onadult psychotherapy and concluded, “The figures fail to support thehypothesis that psychotherapy facilitates recovery from neurotic disorder”(p. 323). Five years later Levitt (1957) reviewed the research on child andadolescent psychotherapy and reached a verdict similar to that of Eysenck:“. . . the results of the present study fail to support the view that psycho-therapy with ‘neurotic’ children is effective” (p. 195). These two articlestriggered many criticisms from throughout the world, but these two

*Mailing address: 719 Fremont Avenue, South Pasadena, CA 91030. e-mail: PaulWClement@aol.com1 This paper is an update of “Outcomes from 40 Years of Psychotherapy in a Private Practice,”

which was published in the American Journal of Psychotherapy, 62/3

AMERICAN JOURNAL OF PSYCHOTHERAPY, Vol. 67, No. 1, 2013

23

psychologists updated their literature reviews and came to the sameconclusions made in their first articles (Eysenck, 1961; Levitt, 1963).

In spite of such negative reviews of research on psychotherapy, psy-chotherapists continued to practice. On rare occasions they reported theirresults. For example, Heilbrunn (1966) evaluated her outcomes from 17years of practicing psychoanalysis and psychoanalytic therapy and pub-lished her results in the American Journal of Psychotherapy. She claimedthat 77 of 173 patients (i.e., 45%) improved; however, she excluded morethan 80 patients seen for less than 20 sessions. When I read her paper a fewmonths before becoming a licensed psychologist in California, I resolvedto do something similar. That is the purpose of the present paper.

Throughout the 1950s and 1960s researchers were responding to thechallenges posed by Eysenck and Levitt. In 1970 Meltzoff and Kornreichreviewed that research. They concluded that well-designed and controlledresearch had demonstrated very positive outcomes from psychotherapy.They also reviewed research on characteristics of patients and therapiststhat contribute to positive treatment outcomes, and on patient-therapistrelationship variables that make a difference. Although their review pro-vided encouragement to psychotherapists in all work settings, it did notreveal what kind of outcomes were obtained by therapists in privatepractice. It did not identify brief outcome measures suitable for repeatedadministrations to gauge patient change across time.

Seven years later Smith and Glass (1977) introduced a quantitativeapproach for performing literature reviews of controlled treatment-out-come studies on adults. They called it “meta-analysis.” Three years laterSmith, Glass, and Miller (1980) expanded and updated the previousreview. Although many methodologists criticized their approach, manyother psychologists (and researchers from many other disciplines) adoptedand adapted meta-analysis for reviewing research findings. Many investi-gators have performed meta-analyses of controlled treatment-outcomeresearch on psychotherapy for children and adolescents. In analyzing thefindings from 27 meta-analyses of child, adolescent, and adult psychother-apy research, the mean and standard deviation effect size (ES) was 0.76(0.24). 95% CI [0.66, 0.86] (Abbass, Kisley, & Kroenke, 2009; Anderson& Lambert, 1995; Bratton, Ray, Rhine, & Jones, 2005; Casey & Berman,1985; Driessen, Cuijpers, de Matt, Abbass, de Jonghe, & Dekker, 2010;Erion, 2006; Fossum, Handegard, Martinussen, & Morch, 2008; Kazdin,Bass, Ayers, & Rodgers, 1990; Leichsenring, Rabung, & Leiging, 2004;Lewinsohn & Clarke, 1999; Maughan, Christiansen, Jenson, Olympia, &Clark, 2005; McCleod & Weisz, 2004; Messer & Abbass, 2010; Michael &

AMERICAN JOURNAL OF PSYCHOTHERAPY

24

Crowley, 2002; Oei & Dingle, 2008; Olympia & Clark, 2005; Peleikis &Dahl, 2005; Reinecke, Ryan, & DuBois, 1998a, 1998b; Shadish et al., 1997;Shapiro & Shapiro, 1982; Smith & Glass, 1977; Smith et al., 1980; Smith,Bartz, & Richards, 2007; Stage & Quiroz, 1997; Weisz, McCarty, & Valeri,2006; Weisz, Weiss, Alicke, & Klotz, 1987; Weisz, Weiss, Han, Granger,& Morton, 1995). I have used meta-analytic techniques as one of my basicways of expressing magnitude of change within and across the patients ofmy private practice. My results have consistently surpassed the mean ESsgiven in the reviews listed above.

The most common meta-analyses are reviews of the results of randomcontrolled treatments or trials (RCTs) in which one or more groups oftreated cases are compared to one or more control or contrast groups. Incontrast, my meta-analyses have compared how a patient was functioningat intake to how much that patient has changed over time. Meta-analysesof such within-cases results have not appeared as frequently as those ofRCTs, but they do exist. In analyzing the findings from 18 meta-analysesof within-patients-outcomes (pre-therapy versus post-therapy) research,the mean (SD) ES was 1.35 (0.39), 95% CI [1.16, 1.55] (Burlingame,Fuhriman, & Mosier, 2003; Clement, 2008; de Maat, de Jonghe, Schoevers,& Dekker, 2009; Driessen, Cuijpers de Maat, Abbass, de Jonghe, &Dekker, 2010; Friedman, Cardemil, Uebelacker, Beevers, Chestnut, &Miller, 2005; Hofman, Sawyer, Witt, & Oh, 2010; Huber, Henrich, &Klug, 2005; Kazdin & Whitley, 2006; Leichsenring & Leibing, 2003;Leichsenring, Rabung, & Leibing, 2004; Maughan, Christiansen, Jenson,Olympia, & Clark, 2005; Michael & Crowley, 2002; Minami, Wampold,Serlin, Kircher, & Brown, 2007; Norton & Philipp, 2008; Oei & Dingle,2008; Stiles, Barkham, Connell, & Mellor-Clark, 2008; Stiles, Barkham,Mellor-Clark, & Connell, 2008; Stiles, Barkham, Twigg, Mellor-Clark, &Cooper, 2006).

A one-way ANOVA on the mean ES of the 28 reviews of RCT researchlisted above and the 18 reviews of within-cases studies produced thefollowing results: F (1, 43) � 39.80, p � 1.299E –007. Published meta-analyses of mean ESs from RCTs have greatly underestimated how mucha given patient improves during a course of psychotherapy.

Some of the meta-analyses carried out throughout the 1980s and early1990s identified specific psychological treatments that were effective fortreating particular disorders, for example, depression (Dobson, 1989;Steinbrueck, Maxwell, & Howard, 1983), generalized anxiety disorder(Gould, Otto, Pollack, & Yap, 1997), obsessive-compulsive disorder (Cox,Swinson, Morrison, & Lee, 1993; Christensen, Hadzi-Pavlovic, Andrews,

Practice-Based Evidence from 45 Years

25

& Mattick, 1987). In response to such findings David Barlow, president ofthe American Psychological Association’s Division of Clinical Psychology,appointed a Task Force on Promotion and Dissemination of PsychologicalProcedures in 1993 (Sanderson & Woody, 1995a, 1995b). During theensuing years many journal articles and books identified, described, andprovided treatment manuals for such empirically supported treatments.But there were many protests against these lists. For many years at theannual conventions of the American Psychological Association (APA)there were debates about the appropriateness of such lists.

Partly in response to these debates APA president Ron Levant createda Presidential Task Force on Evidence-Based Practice in 2005 to investi-gate the issues raised. That task force produced a report that ultimatelybecame a policy statement of the association (APA Presidential TaskForce, 2006). The task force concluded, “Evidence-based practice requiresthat psychologists recognize the strengths and limitations of evidenceobtained from different types of research. Research has shown that thetreatment method. . . . the individual psychologist. . . . the treatmentrelationship . . . .and the patient . . . . are all vital contributors to thesuccess of psychological practice” (p. 275). The report called for thecollection of effectiveness evidence to complement efficacy results fromrandomize controlled trials (RCTs).

Unfortunately there is very little published effectiveness data gatheredfrom the routine private practice of psychotherapy. References to “usualclinical care” have been misleading (e.g., Weisz, Jensen-Doss, & Hawley,2006) because they overwhelmingly refer to findings from institutionalsettings, such as clinics, hospitals, and residential treatment centers. Incontrast, according to the APA database on “Employment Characteristicsof APA Members,” between 58% to 78% of psychologists who areemployed full-time providing mental health services are in independentpractice. In addition there are many more psychologists who are salariedby colleges, universities, hospitals, clinics, etc. who maintain part-timeprivate practices. We know almost nothing about their treatment out-comes. The present article provides an exception.

Most of the exceptions that do exist involve samples from an individualpractice or from a group practice rather than presenting outcomes for allcases seen by an individual therapist or by a group of private practitioners.For example, Persons, Burns, and Perloff (1988) gave their results fromtreating 70 depressed adult patients in private practice using cognitivetherapy. Similarly, Wise (2003) presented treatment outcomes for anintensive outpatient program with 225 patients.

AMERICAN JOURNAL OF PSYCHOTHERAPY

26

There is another movement that complements empirically supportedtreatments and evidence-based practice. It is the call for practice-basedevidence. Mellor-Clark, Barkham, Connell, and Evans (1999) gave an earlyexample. Their article introduced the Clinical Outcomes in RoutineEvaluation (CORE) information management system as well as empha-sized the importance of collecting practice-based evidence to complementthat obtained through RCTs and other avenues. More recently Barkham,Hardy, and Mellor-Clark (2010) edited a book calling for and demonstrat-ing practice-based evidence. The present article is an example of practice-based evidence of the effectiveness of psychotherapy within a privatepractice.

METHOD

In general, I followed the methods described in several earlier articles(Clement, 1996, 1999, 2011), in which I used a five-level Global Estimateof Outcome (GEO) score as follows: 1 � Much Worse at termination thanat intake (i.e., the level of functioning was at least 50% worse than atintake), 2 � Worse at termination than at intake (i.e., the level offunctioning was 11% to 49% worse than at intake), 3 � No Change sinceintake (i.e., the level of functioning at termination was within plus or minus10% of what it had been at intake), 4 � Better than at intake (i.e., the levelof functioning at termination was 11% to 49% better than at intake) , and5 � Much Better than at intake (i.e., the level of functioning at terminationwas 50% or better than at intake)(cf., Clement, 1994). When ESs wereavailable I converted to GEO scores as follows: If ES � -1.50 or less, theGEO score was 1; if ES � -0.51 to -1.49, the GEO score was 2; if ES �-0.50 to �0.50, the GEO score was 3; if ES � �0.51 to �1.49, the GEOscore was 4; and if ES � �1.50 or greater, the GEO score was 5 (cf.,Clement, 1999, 2008, 2011).

In the fall of 1988 I began reviewing all closed cases and assigned aGEO score based on all materials within the folder. For some cases,particularly children, observational data facilitated making a quantita-tive judgment. For a majority of cases such observational data did notexist.

About 25 years ago I started using problem checklists to evaluatefunctioning at intake and at subsequent re-evaluations. Each checklistcontains over 60 items. These checklists are available in Clement (1999).The patient or the parent of a minor patient rates each problem using a10-point Scale of Functioning (SOF): 10 � Excellent Functioning, 9 �Good Functioning, 8 � Slight Problem, 7 � Some Problem, 6 � Mod-

Practice-Based Evidence from 45 Years

27

erate Problem, 5 � Serious Problem, 4 � Major Problem, 3 � Unable toFunction, 2 � In Some Danger of Hurting Self or Others, and 1 � InPersistent Danger of Hurting Self or Others. The mean (SD) number ofitems scored at intake has been 16.72 (11.13), 95% CI [15.58, 17.86], themedian has been 14, and the range has been 1–62.

I also calculate a Global Assessment of Functioning (GAF) score atintake by determining the mean of the SOF scores, subtracting thestandard deviation of the SOF scores for the patient from the mean, andmultiplying the result by 10. This approach to determining the GAF scoreat intake uses the quantitative ratings of the patient as described in thepreceding paragraph to determine level of functioning. The more commonpractice is for the therapist to make an impressionistic estimate of thepatient’s level of functioning. Using my method the mean (SD) GAF scoreat intake across all cases has been 44.42 (12.84), 95% CI [43.73, 45.11],the median has been 45, and the range has been 2 to 85.

THERAPIST

I am a Caucasian male of Western European descent. I grew up on asmall farm north of Seattle. I started my career in 1965 at the Division ofMedical Psychology, UCLA Department of Psychiatry. In 1967 I joinedthe faculty of the Graduate School of Psychology, Fuller TheologicalSeminary. Given my rural origins, I have always had a bias towardidentifying what works to solve a given kind of problem and towardobtaining empirical evidence regarding how much the phenomenon inquestion has changed. I do not identify with any of the brand-namepsychotherapies, for example, psychodynamic, client-centered, behavioral,cognitive, cognitive-behavioral, Gestalt, etc. Part of my approach has beento lean on theoretical models that have been supported by empiricalresearch. In this spirit I spent the first 23 years of my career doingcontrolled treatment outcome studies of psychotherapy with children. Inlate 1988 I left academia and shifted from part-time to full-time privatepractice. I have always been interested in measuring treatment outcomeswithin my private practice. I have affirmed the empirically orientedmovements including empirically supported treatments, evidence-basedpractice, and now practice-based evidence.

PARTICIPANTS

From July 1966 through July 2011 I had 2,258 intakes to my privatepractice. Of these 201 came only for psychological assessment, 386(18.76%) dropped out without receiving any identified intervention, 40 cases

AMERICAN JOURNAL OF PSYCHOTHERAPY

28

involved consultation without any intervention, 32 intervention cases had notyet produced outcome data, and 1,599 cases had produced outcome data.

DIAGNOSES

My career has spanned all editions of the Diagnostic and StatisticalManual (DSM) of the American Psychiatric Association; however, origi-nally I used DSM-III-R (American Psychiatric Association, 1987) to recorddiagnoses for patients seen during the first 22 years of my practice. Beforeperforming the analyses for the present article I updated all diagnosesusing DSM-IV-TR (American Psychiatric Association, 2000).

In decreasing order of frequency the most common diagnoses were asfollows: dysthymic disorder (n � 468); adjustment disorders (n � 403);attention-deficit/hyperactivity disorder, combined or predominantly hyperac-tive/impulsive (326); oppositional defiant disorder (283); generalized anxietydisorder (239); partner relational problem (220); major depression (165);social phobia (141); attention-deficit/hyperactivity disorder, predominantlyinattentive type (121); no diagnosis (101); specific phobia (87); obsessive-compulsive disorder (85); panic disorder with and without agoraphobia (68);separation anxiety disorder (54); acute stress disorder and posttraumatic stressdisorder (48); parent-child relational problem (41); sibling relational problemand relational problem NOS (40); bipolar disorders (38); academic problems(36); anxiety disorders NOS (23); body dysmorphic, hypochondriasis, orsomatoform disorder (20); sexual disorders (16); Tourette’s disorder (15);schizophrenias and schizoaffective disorder (13); and autistic disorder andpervasive developmental disorder NOS (10).

Of all patients seen I had recorded only one Axis I diagnosis for 66% ofthem, two Axis I diagnoses for 34%, three Axis I diagnoses for 13%, fourAxis I diagnoses for 4%, and one Axis II diagnoses for 2.5%. I do not knowhow these numbers compare with trends in other life-span private practices.

SESSIONS PER CASE

Of the 1,599 treatment cases with outcome data the mean (SD) numberof sessions per case was 18.82 (29.89), 95% CI [17.35, 20.29], the medianwas 10, and the range was 1-344. Three-quarters of my patients completedtreatment within 20 sessions, and approximately 82% finished within 25sessions; therefore, the majority of my practice has consisted of what theliterature identifies as “short-term therapy” (e.g., Watkins, 2012).

STORAGE AND RETRIEVAL OF DATA

When I opened a new case, I entered essential data about the patientinto an electronic database (Microsoft Access). The database includes

Practice-Based Evidence from 45 Years

29

fields for identifying information on the patient, diagnoses, level of func-tioning at intake, and treatment outcome scores as well as other facts. Allpatients ever seen by me in my private practice were included. I couldretrieve information from the database by running a query on one or morefields, such as age, sex, or diagnosis. Similarly, I could arrange the retrievedinformation by giving a sort command, for example, to list patients by agefrom the youngest to the oldest. Before analyzing any quantitative data Ipasted the query from Access into Microsoft Excel.

Statistical AnalysesI copied raw data from Excel and used the StatMost statistical software

package (1994) to perform statistical analyses.

CANCELLATIONS AND NO-SHOWS

Although I did not track cancellations and no-shows during much ofmy career, I have done so since January 1999. The mean (SD) percentageof cancelled and broken appointments per week has been 15.97 (0.07),95% CI [15.44, 16.51].

RESULTSWHO MADE REFERRALS TO ME?

In descending order of frequency the following have been my sourcesof referrals: psychologists 23.85%, patients 18.96%, managed care com-panies 18.52%, miscellaneous sources 15.36%, physicians 8.13%, schoolpersonnel 4.57%, unknown sources 2.66%, clergy members 2.58%, familymembers of the patient 1.87%, health care providers other than thosementioned elsewhere in this paragraph 1.87%, attorneys 1.02%, friends ofthe patient 0.27, social workers 0.27%, and patient him/herself 0.09%.

HOW DID OUTCOME VARY BY THERAPIST ESTIMATE AND BY PATIENT (OR

PARENT) RATING?For more than half of the 45 years in question, I did not calculate ESs;

therefore, the GEO scores from those years were based on my assessmentof all information within each patient’s record. In addition, since I starteddetermining ESs many patients have drifted away from therapy withoutcompleting a termination interview. Obtaining self-ratings from patientsafter they stopped coming to sessions has been very difficult, so that thereare many cases for which I have had to determine the GEO scores withoutthe benefit of the patient’s self-rating. Only about 35% of treated patientprovided self-ratings at termination using the Scale of Functioning and oneof my checklists. For these patients I followed the procedures for trans-

AMERICAN JOURNAL OF PSYCHOTHERAPY

30

forming an ES into a GEO score as described in the first paragraph of theMethods section above. I wondered who would evaluate my outcomesmore favorably: Would I or the patients do so?

My estimates were the basis of GEO scores for 1,041 cases. The mean(SD) of these ratings was 3.78 (0.80), 95% CI [3.73, 3.83]. The patient’sor parents’ ratings using the SOF and subsequent ESs were the basis ofGEO scores for 558 cases. The mean (SD) of these ratings was 4.43 (0.73),95% CI [4.37, 4.49]. An analysis of variance on these data produced thefollowing results: F (1, 1,597) � 260.14, p � 0.0000. Patients and parentsrated the outcomes more positively than I did.

HOW MUCH DID PATIENTS IMPROVE?Percent Improved

Out of all treated cases with outcome data 4 (0.25%) were MuchWorse, 11 (0.69%) were Worse, 497 (31.08%) showed No Change fromintake, 546 (34.15%) were Improved, and 541 (33.83%) were MuchImproved, for an overall improvement rate of about 68%.

Global Estimate of Outcome (GEO) ScoreThe mean GEO (SD) score for 1,599 cases was 4.01 (0.84), 95% CI

[3.97, 4.05], the median was 4.00, and the range was 1-5.

Effect SizeThe mean (SD) ES for 558 cases at termination was 1.90 (1.61), 95%

CI [1.77, 2.04], the median was 1.62, and the range was from –2.91 to�15.22. Of all ESs 3.14% were negative.

HOW DID OUTCOMES VARY BY DIAGNOSIS?Percent Improved

Table 1 shows outcome by diagnosis with the best outcomes listedtoward the top of the table and the worst outcomes listed toward thebottom. The table only includes diagnoses with at least 10 cases. For thislist Chi Square was 99.18 with 23 df, p � 0.0000.

Effect SizeTable 2 lists mean ES by diagnosis in descending order of effectiveness.

The table only includes diagnoses with 10 or more cases. For this list F (1,464) � 1.38, p � 0.1768.

HOW DID OUTCOMES VARY BY PATIENT AGE?Percent Improved

In decreasing order of effectiveness the results were as follows: age 0.5(6 months?)-5 years, 85.33% improved (n � 75); age 6 years-12years,

Practice-Based Evidence from 45 Years

31

71.59% improved (n � 352); age 20 years-29 years, 70.11% (n � 174); age30 years-39 years, 68.04% (n � 291); age 40 years-49 years, 65.61% (n �253); age 60 years–88years 63.64% (n � 44); age 50 years-59 years 63.49%(n � 126); and age 13 years-19 years, 63.41% (n � 276). The Pearsoncorrelation between the mid point of each age range and the percentimproved was r � -0.68, n � 8, p � 0.0609.

Effect SizeThe mean ESs in descending order of magnitude by age group were as

follows: age 60 years–88 years, 2.55 (3.54), 95% CI [0.66, 4.44], n � 16;age 0.5–5 years, 2.34 (1.78), 95% CI [1.62, 3.06], n � 26; age 50 years–59years, 2.09 (1.76), 95% CI [1.59, 2.60], n � 49; age 30 years–39 years, 2.05(1.54), 95% CI [1.74, 2.35], n � 102; age 20 years–29 years, 1.96 (1.58),

Table I. IMPROVEMENT RATES IN DESCENDING ORDER BY DIAGNOSIS

Diagnostic Group n%

Improved

Separation anxiety disorder 20 90.00Encopresis and enuresis 16 87.50Panic disorder without agoraphobia 15 86.67No diagnosis on Axis I or II 28 85.71Specific phobia 34 82.35Eating disorders 15 80.00Sexual disorders 10 80.00Social phobia 60 76.67Adjustment disorders 228 75.00Generalized anxiety disorder 98 71.43Sibling relational problem 21 71.43Dysthymic disorder 221 70.14A-D/HD, combined or hyperactive/impulsive 188 68.09Oppositional defiant disorder 109 67.89A-D/HD, predominantly inattentive type 48 64.58Obsessive compulsive disorder 57 63.16Anxiety disorder NOS 13 61.54Parent-child relational problem 18 61.11Major depressive disorder 99 60.61Intermittent explosive disorder 12 58.33Bipolar disorders 24 54.17Partner relational problem 119 50.42Panic disorder with agoraphobia 13 46.15Conduct disorder 30 43.33

AMERICAN JOURNAL OF PSYCHOTHERAPY

32

95% CI [1.56, 2.36], n � 62; age 6 years 12 years, 1.83 (1.46), 95% CI[1.55, 2.10], n � 110; age 40 years–49 years, 1.75 (1.56), 95% CI [1.45,2.05], n � 105; and age 13 years–19 years,, 1.66 (1.29), 95% CI [1.40,1.92], n � 99. The Pearson correlation between the mid point of each agerange and the mean ES was r � 0.42, n � 8, p � 0.3010.

HOW DID OUTCOMES VARY BY SEX OF THE PATIENT?Percent Improved

Of female patients 68.76% improved (n � 653). Of all male patients67.60 % improved (n � 929). Although this is a slight difference, it hasheld up across the decades.

Effect SizeFor all female patients the mean ES was 1.94 (1.73), 95% CI [1.71,

2.16], n � 225. For all male patients the mean ES was 1.85 (1.15), 95% CI[1.69, 2.01], n � 354.

WAS THERE A RELATIONSHIP BETWEEN TREATMENT LENGTH AND OUTCOME?Percent Improved

To answer this I ordered all treatment cases from the least number ofsessions to the most number of sessions. Then I identified blocks ofadjacent sessions. The first six blocks contained 49 cases each. Theremaining 26 blocks contained 50 cases each. The first block had only oneor two sessions per case with a median of two. The second block had amedian of three sessions. The final block had a median of 125 sessions.

Table II. MEAN ES IN DESCENDING MAGNITUDE BY DIAGNOSIS

Diagnostic Group n ES (SD) 95% CI

Major depressive disorder 46 2.44 (2.56) 1.68, 3.20Specific phobia 10 2.19 (1.38) 1.19, 3.17Generalized anxiety disorder 39 2.11 (1.97) 1.47, 2.75Dysthymic disorder 102 2.01 (1.52) 1.71, 2.31Adjustment disorders 83 1.93 (1.36) 1.64, 2.23Social phobia 22 1.92 (2.09) 1.00, 2.85Oppositional defiant disorder 37 1.90 (1.64) 1.35, 2.44A-D/HD, combined or predominantly

hyperactive/impulsive 59 1.81 (1.31) 1.47, 2.16Bipolar disorders 12 1.70 (1.21) 0.93, 2.47Partner relational problem 22 1.43 (1.46) 0.78, 2.07Obsessive compulsive disorder 25 1.35 (1.31) 0.81, 1.89A-D/HD, predominantly inattentive 20 1.26 (1.06) 0.77, 1.76

Practice-Based Evidence from 45 Years

33

Then I calculated the mean % improved within each block. Figure 1presents the results in graphic form. The Pearson correlation betweenmedian sessions per block and percent improved was as follows: r � 0.63,n � 32, p � 9.56E-05.

Effect SizeThere was great variability in the improvement patients achieved with

a given amount of therapy as measured by ES. The Pearson correlation ofES versus sessions-per-individual case was as follows: r � 0.00, n � 683.I did a second analysis similar to the one presented in the precedingparagraph. I created blocks of sessions from the fewest sessions to themost. The first eight blocks contained 28 entries. The remaining 17 blockscontained 27. The median number of sessions per block ranged from 3 forthe first block to 115 for the last block on the right. Then I determined themean ES within each of these 25 blocks. The Pearson correlation betweenmedian sessions and mean ES was as follows: r � 0.07, n � 25, p � 0.7339.

For some individual patients I obtained from two to five ESs acrosstime. For such cases I determined the Spearman rank order correlation bycomparing the order the ESs were obtained (i.e., first, second, etc.) withthe magnitude of each ES within a patient (i.e., 1st � smallest ES, 2nd �next larger ES, etc.). The results were as follows: rSpearman � 0.63, n � 194,p � 5.42E-23.

Figure 1.PERCENT IMPROVED FOR BLOCKS OF SESSIONS ARRANGED FROM FEWEST (ONTHE LEFT) TO MOST (ON THE RIGHT)

0.00

20.00

40.00

60.00

80.00

100.00

120.00

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31

Blocks of Sessions (median of 2 sessions in block 1, median of 3 in block 2, to median of 125 in block 32)

% Im

prov

ed p

er B

lock

AMERICAN JOURNAL OF PSYCHOTHERAPY

34

WAS THERE A RELATIONSHIP BETWEEN TREATMENT FORMAT (MODALITY) AND

OUTCOME?Percent Improved

Figure 2 shows the mean percent improved for each of five treatmentformats. A Chi Square performed on the depicted data was 18.24 with4 df, p � 0.001108.

ES. Sufficient data to analyze outcome by treatment format were onlyavailable for three formats. The mean ES for individual therapy was 2.00(1.72), 95% CI [1.82, 2.17], n � 366. The mean ES for family therapy was1.77 (1.41), 95% CI [1.56, 1.98], n � 172. The mean ES for couplestherapy was 1.37 (1.34), 95% CI [0.96, 1.79], n � 42. An analysis ofvariance on these data produced the following results: F (2, 577) � 3.44,p � 0.0326.

HAS MY THERAPEUTIC EFFECTIVENESS CHANGED OVER TIME?Percent improved

The Pearson correlation between mean percent improved and year ofmy career was as follows: r � -0.35, n � 45, p � 0.013. For the first 22years of my career I maintained a part-time practice. For the more recent23 years I shifted to full-time private practice. Given this fact I ran aPearson correlation between the number of new cases opened per yearversus the mean percent improved: r � -0.31, n � 45, p � 0.0391: the

Figure 2.MEAN OUTCOME BY TREATMENT FORMAT (MODALITY). CONSULTATION N �18, GROUP N � 8, FAMILY N � 482, NDIVIDUAL N � 934, COUPLE N � 158.

100.00

87.50

69.50 69.06

51.90

0.00

20.00

40.00

60.00

80.00

100.00

120.00

Consultation Group Family Individual Couple

Mea

n %

Impr

oved

Practice-Based Evidence from 45 Years

35

more cases seen in a given year, the poorer the outcome. I also determinedthe percentage of managed care cases seen each year and calculated aPearson correlation between percentage of managed care cases and meanpercent improved per year: r � -0.32, n � 45, p � 0.0319: the moremanaged care cases seen in a given year, the poorer the outcome. Icomputed the Pearson correlation between year of my career and percentof all cases that were children (12 years old and younger): r � -0.31, n �45, p � 0.0375: as the years went by, a smaller proportion of my annualcaseload consisted of children. Finally, I correlated percent improved eachyear versus percent of cases that were children each year: r � 0.51, n � 45,p � 0.0003: the greater the proportion of my annual caseload consisting ofchildren, the better my outcomes.

Effect sizeI have only had 10 or more ESs per year for the most recent 20 years.

I ran a Pearson correlation between the year of my practice and the meanES within each year. The result was as follows: r � -0.29, n � 20, p �0.1661.

DISCUSSIONDISTINCTIVE FEATURES OF THE PRESENT REPORT

The present article includes many unique features. It covers the lifespanwith the patients ranging in age from 6 months to 88 years at intake. Itspans all editions of the DSM through DSM-IV-TR. It includes a widerange of DSM diagnoses and shows differences in outcomes among thesediagnoses. It provides quantitative analyses of treatment outcomes. Ittraverses 45 years of one psychotherapist’s private practice. It incorporatesall cases of the practice, not just a sample. It takes into account anextraordinary number of cases. It demonstrates the importance of per-forming pre-/post-treatment analyses. It reveals outcomes from routineclinical practice.

THERAPIST VERSUS PATIENT (OR PARENT) OUTCOME RATING

According to Minami, Wampold, Serlin, Kircher, and Brown (2007)measures are considered high on reactivity if they are assessed by aclinician and low on reactivity if patients provide self-report data. Mea-sures that focus on symptoms or the targets of treatment are classified ashigh on specificity but when the measures cover global functioning they areconsidered low on specificity. These authors (2007) provided pre-/post-treatment ES benchmarks in the treatment of major depression. High-reactivity/high-specificity measures produced the largest ESs. Low-reac-

AMERICAN JOURNAL OF PSYCHOTHERAPY

36

tivity/high-specificity measures produced intermediate ESs. And low-reactivity/low-specificity measures produced the smallest ESs. When mypatients (or their parents) provided pre-treatment and post-treatmentratings on problem checklists, their ratings seemed to match Minami etal.’s definition of low-reactivity/high-specificity measures. When I pro-vided a global estimate of outcome (GEO) by reviewing all evidence withina case folder, these scores seemed to be examples of high-reactivity/low-specificity. The results provided by Minami et al. did not provide a clue asto where high-reactivity/low specificity results would fall in comparison tothe three combinations that they did examine. My results were clear: thepatients and their parents rated treatment outcomes more positively usinglow-reactivity/high-specificity measures than I did using high-reactivity/low-specificity measures.

DROPOUTS

Patients who drop out of treatment before a mutually planned termi-nation date is a problem both in routine practice and controlled research.According to a recent review of 669 studies covering 83,834 patients, thedropout rate was 19.7%, 95% CI [18.70%, 20.70%] (Swift & Greenberg,2012). In my private practice of all patients seeking treatment 386(18.76%) dropped out after one, two, or three sessions without receivingany identified treatment.

IMPROVEMENT

Overall about 68% of my patients have improved, 31% have notimproved, and 1% have gotten worse. These results are strikinglysimilar to those reported for 3,672 cases of practice-based evidence inthe United Kingdom (Barkham et al., 2008). These researchers said that67.50% had improved, 31.8% showed no reliable change, and 0.7%deteriorated.

My mean ES at termination of 1.90 is probably more representative ofoutcomes in routine private practice than mean ESs reported from RCTs.As mentioned earlier Minami et al. (2007) determined benchmarks fortreatment outcomes with adults presenting with major depression. For 846“completers” plus “early terminators” the mean pre-/post-therapy ESusing the Beck Depression Inventory (a low-reactivity/high-specificitymeasure) was 1.71. For 1,387 “completers” the mean ES was 1.86. Thislatter mean is very similar to my mean ES of 1.90 across all of my patients.My mean ES for major depression was 2.44.

There are other findings from routine clinical practice to compare withmy results. For example, Barkham, Mellor-Clark, Connell, Evans, Evans,

Practice-Based Evidence from 45 Years

37

and Margison (2010) reported pre-/post-therapy ESs for 9,337 patientstreated in various service settings within the UK. The mean of the fivemeans that they published was 1.34. The grand mean falls between themean ES of 1.24 for children with oppositional defiant disorder and of1.47 for those with conduct disorder published by Kazdin and Whitley(2006) in their research program. These latter two means were also basedon pre-/post-therapy comparisons. As indicated in the Discussion sectionof this article, such pre-/post-therapy ESs are consistently greater thanbetween-groups ESs normally reported in meta-analyses of controlledtreatment outcome research (e.g., Casey & Berman, 1985; Smith & Glass,1977; Smith et al., 1980; Weisz et al., 1995).

Hollon (1996) asserted, “[T]he available evidence appears to differbetween the child and adult literatures, but in neither does [sic] outcomesobserved in applied settings exceed those observed in controlled clinicaltrials” (p. 1028). In fact, the present mean ES clearly exceeds the mean ESsreported in published meta-analyses of RCTs on child, adolescent, andadult psychotherapies. Hollon did say, “I agree with Seligman that the bestway to determine what goes on in actual clinical practice is to study itdirectly” (p. 1025), and Kazdin and Weisz (1998) declared, “[T]he mag-nitude of therapeutic change is an issue in need of much greater attention”(p. 30). The ES results reviewed above focused on magnitude of thera-peutic change.

The typical RCT is not designed to determine magnitude of therapeuticchange. Usually its purpose is to demonstrate a causal relationship betweena particular treatment and its effect in comparison to a control or contrastcondition (cf., Seligman, 1995). The resultant ESs show differences be-tween the treated and the control/contrast group. Most investigatorsemploying RCT designs do not report pre-/post-treament ESs for theircases (Kazdin, Bass, Ayers, & Rodgers, 1990).

Asay and Lambert (1999) warned, “Client outcome is principallydetermined by client variables and extratherapeutic factors rather than bythe therapist or therapy” (p. 43); hence, the positive results presentedabove can be celebrated without claiming that I or the particular treat-ments that I used were primarily responsible for the measured gains. Thesesame authors estimated that 15% of improvement comes from expectancy(placebo effects), 15% from techniques, 30% from the therapeutic rela-tionship, and 40% from extratherapeutic change. There is evidence thatplacebo effects may even be greater than the estimate of Asay andLambert. For example, Glass and Kliegl (1983) gave an average ES of 0.56from “placebo treatment.” Dush, Hirt, and Schroeder (1989) claimed,

AMERICAN JOURNAL OF PSYCHOTHERAPY

38

“Collective results from other meta-analyses suggest an average placebo ESof about a third to a half of a standard deviation, in comparison with theES of no treatment” (p. 100).

Of all ESs obtained in my results, 3.14% were negative. In comparison,Dush et al. (1989) reported that 15% of the ESs in their meta-analysis ofself-statement modification in children were negative. Smith and Glass(1977) said that 12% of the 833 ES measures that they calculated werenegative.

OUTCOME BY DIAGNOSIS

The percent of my patients who showed improvement varied substan-tially across diagnoses. Although this finding is consistent with findingsfrom some controlled research (e.g., Lambert & Archer, 2006), researcherstend to focus on one diagnostic group at a time. And therapists in privatepractice rarely publish their treatment outcomes; therefore, the presentreport, covering 45 years of one therapist’s private practice and all patientsseen, is unique. Although there was quite a large range in my mean ESsacross diagnoses, the large variance in ESs within diagnoses blurred theapparent differences. Nevertheless, these findings are distinctive in thepublished literature on outcomes of psychotherapy.

OUTCOME BY PATIENT AGE

There was no clear or consistent relationship between age groups ofpatients and treatment outcome. Regarding percent improved there was anegative but not quite statistically significant correlation, indicating thatmy younger patients improved more than my older ones. However, formean ES, there was a positive but not statistically significant correlation.Looking at just the three youngest age groups (age 0.5 years–5 years,6 years–12 years, and 13 years–19 years) there was consistency in the trendsof the two measures of outcome: percent improved and ES. Preschoolersfaired best. Children came in second. And teenagers had the poorestoutcome. This trend has held up across my career.

OUTCOME BY SEX OF PATIENT

The slight but statistically nonsignificant advantage in outcome forfemales in my practice is consistent with previous research. For example,Casey and Berman (1985) found that the greater the proportion of malesin a study, the smaller the ESs. And in their review of psychotherapy withchildren and adolescents, Weisz, Weiss, Han, Granger, and Morton (1995)reported that females faired better than males. Similarly, Smith et al.(1980) determined a small, negative correlation between the percent of

Practice-Based Evidence from 45 Years

39

males in a study and ES. Thirty years later Parry, Castonguay, Borkovec,and Wolf (2010) reported that of 220 clients treated by 57 clinicians in thecommunity, female clients showed more improvement than male clients.

OUTCOME BY DURATION OF TREATMENT

With the exception of patients who improved with just one or twosessions, the greater the number of treatments, the greater the percentageof patients who improved. This finding is consistent with that of Messerand Boals (1981). Clearly the dose-effect phenomenon is not always found.For example, in their meta-analysis of outcomes of psychotherapy withchildren Casey and Berman (1985) concluded that length of treatment wasnegatively related to mean ES: r � -0.28 (p � 0.02); whereas, Smith andGlass (1977) and Smith et al. (1980) in their meta-analyses of adultpsychotherapy outcomes found no significant relationship between theduration of treatment and outcome. In discussing such findings Howard,Kopta, Krause, and Orlinsky (1986) warned, “. . .a between-study analysis. . . has no necessary implication for the relationship between duration andbenefit within each study” (p. 159). Their article provided a quantitativemodel estimating the percentage of patients improved for specific amountsof psychotherapy. I calculated the Pearson correlation between theirmodel and my outcome data at 4, 8, 13, 26, 52, and 104 sessions: r � 0.92,n � 6, p � 0.0093. These findings are consistent with those of theConsumer Reports study (Seligman, 1995). Although there is no correlationacross all cases between number of treatment sessions and mean ES, whenmultiple ESs are obtained on the same patient across time, there is a strongrelationship between amount of treatment and outcome. This finding isalso consistent with Howard’s dose-response model.

OUTCOME BY TREATMENT FORMAT

There is a significant difference in my outcomes based on treatmentformat (modality). Unfortunately I have not found an article by any othertherapist or researcher that compared the five modalities compared inFigure 2. I am unsure whether the trends in my results would generalize tothose of other therapists. Young (2007) retrospectively evaluated heroutcomes from 55 years of practice as a psychoanalyst and psychoanalyticpsychotherapist. During that period of time she had treated just 231people. She said that she provided “classical analysis” to 22 patients(p. 316), “modified analysis” to 19 seen once or twice a week (p. 316),“analytically oriented psychotherapy” to 26 (p. 316), “insight-orientedsupportive therapy” to 65 (p. 316), and “supportive therapy” to 47(p. 316). She reported that her success rates ranged from 40.4% improved

AMERICAN JOURNAL OF PSYCHOTHERAPY

40

with supportive non-insight oriented treatment to 94.5% improved withmodified analysis.

THERAPIST’S EFFECTIVENESS ACROSS 45 YEARS

Measured by percent improved per year, not only have I failed toimprove across the years, my outcomes have gotten worse across time. Thistrend was puzzling and troubling. Switching from part-time to full-timepractice corresponded to a drop in effectiveness. This finding was similarto that reported by Borkovec, Echemendia, Ragusea, and Ruiz (2001);increases in therapists’ caseloads corresponded to lessened treatmentoutcomes. Parry et al. (2010) reported the same trend for 57 therapists inPennsylvania. Similarly, increases in the proportion of my case load peryear that had been referred by a managed care company correspondedwith poorer outcomes. During the first 23 years of my career I was afull-time academic. My research, writing, and teaching during those yearsfocused on children. Unhappily there was a significant drop in the percentof my caseload consisting of children across the 45 years in question. Thiswas too bad, because the greater the proportion of my caseload consistingof children in a given year, the greater the % of my patients that hadimproved.

The ES data did not show a significant change across time. Smith et al.(1980) concluded, “. . . there was no relationship between the years ofexperience of the therapists in a study and the magnitude of therapeuticeffect produced in that study (r � 0.00)” (p. 117); however, the averageamount of experience of therapists in the studies reviewed was only threeand a quarter years. Dawes (2008) claimed, however, that over 500 studiesof psychotherapy outcome had shown no relationship between amount ofexperience in the psychotherapists and patient outcomes.

One of my motives in starting to evaluate my treatment outcomes 24years ago was to answer the question: How good am I as a psychothera-pist? Although how many years of experience a therapist has had does notpredict treatment outcomes, there is growing evidence that therapists varygreatly in their effectiveness. In their large-scale study in the UK Stiles andBarkham (2012) found that improvement rates for individual therapistsranged from 23.5% for the least effective therapist to 95.6% for the mosteffective therapist. Seidel (2012) reported very similar ranges of effective-ness in 268 therapists who had each treated at least 30 patients in the US.I suspect that most therapists would be interested in discovering wherethey fall in such distributions from least effective to most effective butsimultaneously afraid to find out.

Practice-Based Evidence from 45 Years

41

CLOSING COMMENT

Kazdin (2008) warned, “We do not benefit as a field from the accu-mulated practice of clinicians. . . .” (p. 157). I hope that the results I haveprovided in the present report will encourage my psychotherapy colleaguesto evaluate their outcomes and to publish their results. In closing, I wantto say, “I have shown you my practice-based evidence. Now you show meyours!”

REFERENCES

Abbass, A., Kisely, S., & Kroenke, K. (2009). Short-term psychodynamic psychotherapy for somaticdisorders: Systematic review of meta-analysis of clinical trials. Psychotherapy and Psychosomat-ics, 78, 265-274. doi:10.1159/000228247

American Psychiatric Association. (1987). Diagnostic and statistical manual of mental disorders (3rd ed.,rev.). Washington, D.C.: Author.

American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed.,text revision). Washington, D.C.: Author.

Anderson, E.M. & Lambert, M.J. (1995). Short-term dynamically oriented psychotherapy: A reviewand meta-analysis. Clinical Psychology Review, 15, 503-514. doi:10.1016/0272-7358(95)00027-M

APA Presidential Task Force. (2006). Evidence-based practice in psychology. American Psychologist,61, 271-285. doi:10.1037/0003-066X.61.4.271

Asay, T.P., & Lambert, M.J. (1999). The empirical case for the common factors in therapy:Quantitative findings. In Hubble, M.A., Duncan, B.L., & Miller, S.D. (Eds.) (1999). The heartand soul of change: What works in therapy (pp. 23-55). Washington, D.C.: American Psycho-logical Association. doi:10.1037/11132-001

Barkham, M., Hardy, G.E., & Mellor-Clark, J. (Eds.). (2010). Developing and delivering practice-basedevidence: A guide for the psychological therapies. West Sussex, UK: Wiley-Blackwell. doi:10.1002/9780470687994

Barkham, M., Mellor-Clark, J., Connell, J., Evans, C., Evans, R., & Margison, F. (2010). ClinicalOutcomes in Routine Evaluation (CORE)—the CORE measures and system: Measuring,monitoring and managing quality evaluation in the psychological therapies. In M. Barkham,G.E. Hardy, & J. Mellor-Clark (Eds.). Developing and delivering practice-based evidence: Aguide for the psychological therapies. West Sussex, UK: Wiley-Blackwell. doi:10.1002/9780470687994.ch8

Barkham, M., Stiles, W.B., Connell, J., Twigg, E., Leach, C., Lucock, M., Angus, L. (2008). Effects ofpsychological therapies in randomized trials and practice-based studies. British Journal ofClinical Psychology, 47, 397-415. doi:10.1348/01446650X311713

Borcovec, T.D., Echemedia, R.J., Ragusea, S.A., & Ruiz, M. (2001). The Pennsylvania PracticeResearch Network and future possibilities for clinically meaningful and scientifically rigorouspsychotherapy effectiveness research. Clinical Psychology: Science and Practice, 8, 155-167. doi:10.1093/clipsy/8.2.155

Bratton, S.C., Ray, D., Rhine, T., & Jones, L. (2005). The efficacy of play therapy with children: Ameta-analytic review of treatment outcomes. Professional Psychology: Research and Practice, 36,376-390. doi: 10.1037/0735-7028.36.4.376

Burlingame, G.M., Fuhriman, A., & Mosier, J. (2003). The differential effectiveness of grouppsychotherapy: A meta-analytic perspective. Group Dynamics: Theory, Research, and Practice,7, 3-12. doi: 10.1037/1089-2699.7.1.3

Casey, R.J., & Berman, J.S. (1985). The outcome of psychotherapy with children. PsychologicalBulletin, 98, 388-400. doi:10.1037/0033-2909.98.2.388

Christensen, H., Hadzi-Pavlovic, D., Andrews, G., & Mattick, R. (1987). Behavior therapy and tricyclicmedication in the treatment of obsessive-compulsive disorder: A quantitative review. Journal ofConsulting and Clinical Psychology, 55, 701-711. doi: 10.1037/0022-006X.55.5.701

AMERICAN JOURNAL OF PSYCHOTHERAPY

42

Clement, P.W. (1994). Quantitative evaluation of 26 years of private practice. Professional Psychology:Research and Practice, 36, 105-113. doi: 10.1037/0735-7028.25.2.173

Clement, P.W. (1996). Evaluation in private practice. Clinical Psychology: Science and Practice, 3,146-159. doi:10/1111/j.1468-2850.1996.tb00064.x

Clement, P.W. (1999). Outcomes and incomes: How to evaluate, improve, and market your psychother-apy practice by measuring outcomes. New York: Guilford.

Clement, P.W. (2008). Outcomes from 40 years of psychotherapy in a private practice. AmericanJournal of Psychotherapy, 62, 215-239.

Clement, P.W. (2011). Research in private practice: How to determine your effectiveness as a therapist.In J.C. Thomas & M. Hersen (Eds.). Understanding research in clinical and counselingpsychology (2nd ed.)(pp. 375-398). New York: Taylor and Francis.

Cox, B.J., Swinson, R.P., Morrison, B., & Lee, P.S. (1993). Clomipramine, fluoxetine, and behaviortherapy in the treatment of obsessive-compulsive disorder: A meta-analysis. Journal of BehaviorTherapy and Experimental Psychiatry, 24, 149-153. doi: 10.1016/0005-7916(93)90043-V

Dawes, R.M. (2008). Psychotherapy: The myth of expertise. In S.O. Lilienfeld, J. Ruscio, & S.J. Lynn(Eds.). Navigating the mindfield: A user’s guide to distinguishing science from pseudoscience inmental health (pp. 311-344). Amherst, NY: Prometheus Books.

de Maat, S., de Jonghe, F., Schoevers, R., & Dekker, J. (2009). The effectiveness of long-termpsychoanalytic therapy: A systematic review of empirical studies. Harvard Review of Psychiatry,17, 1-23. doi: 10.1080/10673220902742476

Dobson, K.S. (1989). A meta-analysis of the efficacy of cognitive therapy for depression. Journal ofConsulting and Clinical Psychology, 57, 414-419. doi: 10.1037/0022-006X.57.3.414

Driessen, E., Cuijpers, P., de Maat, S.C.M., Abbass, A.A., de Jonghe, F., & Dekker, J.J.M. (2010). Theefficacy of short-term psychodynamic psychotherapy for depression: A meta-analysis. ClinicalPsychology Review, 30, 25-36. doi: 10.1016/j.cpr.2009.08.010

Dush, D.M., Hirt, M.L., & Schroeder, H.E. (1989). Self-statement modification in the treatment ofchild behavior disorders: A meta-analysis. Psychological Bulletin, 106, 97-106. doi:10.1037/0033-2909.106.1.97

Erion, J. (2006). Parent tutoring: A meta-analysis. Education & Treatment of Children, 29, 79-106.Eysenck, H.J. (1952). The effects of psychotherapy: An evaluation. Journal of Consulting Psychology,

16, 319-324. doi:10.1037/h0063633Eysenck, H.J. (1961). The effects of psychotherapy. In H.J. Eysenck (Ed.), Handbook of abnormal

psychology (pp. 697-725). New York: Basic Books.Fossum, S., Handegard, B.H., Martinussen, M., Morch, W.T. (2008). Psychosocial interventions for

disruptive and aggressive behavior in children and adolescents: A meta-analysis. European Child& Adolescent Psychiatry, 17, 438-451. doi: 10.1007/s00787-008-0686-8

Friedman, M.A., Cardemil, E.V., Uebelacker, L.A., Beevers, C.G., Chestnut, C., & Miller, I.W. (2005).The GIFT program for major depression: Integrating group, individual, and family treatment.Journal of Psychotherapy Integration, 15, 147-168. doi: 10.1037/1053-0479.15.2.147

Glass, G.V., & Kliegl, R.M. (1983). An apology for research integration in the study of psychotherapy.Journal of Consulting and Clinical Psychology, 51, 28-41. doi:10.1037/0022-006X.51.1.28

Gould, R.A., Otto, M.W., Pollack, M.H., & Yap, L. (1997). Cognitive behavioral and pharmacologicaltreatment of generalized anxiety disorder: A preliminary meta-analysis. Behavior Therapy, 28,285-305. doi: 10.1016/S0005-7894(97)80048-2

Heilbrunn, G. (1966). Results with psychoanalytic therapy and professional commitment. AmericanJournal of Psychotherapy, 20, 89-99.

Hofmann, S.G., Sawyer, A.T., Witt, A.A., & Oh, D. (2010). The effect of mindfulness-based therapyon anxiety and depression: A meta-analytic review. Journal of Consulting and Clinical Psychol-ogy, 78, 169-183. doi: 10.1037/a0018555

Hollon, S.D. (1996). The efficacy and effectiveness of psychotherapy relative to medications. AmericanPsychologist, 51, 1025-1030. doi:10.1037/0003-066X.51.10.1025

Howard, K.I., Kopta, S.M., Krause, M.S., & Orlinsky, D.E. (1986). The dose-effect relationship inpsychotherapy. American Psychologist, 41, 159-164. doi:10.1037/0003-066X.41.2.159

Huber, D., Henrich, G., & Klug, G. (2005). The scales of psychological capacities: Measuring changein psychic structure. Psychotherapy Research, 15, 445-456. doi: 10.1080/10503300500091298

Kazdin, A.E. (2008). Evidence-based treatment and practice: New opportunities to bridge clinical

Practice-Based Evidence from 45 Years

43

research and practice, enhance the knowledge base, and improve patient care. AmericanPsychologist, 63, 146-159. doi:10.1037/0003-066X.63.3.146

Kazdin, A.E., Bass, D., Ayers, W.A., & Rodgers, A. (1990). Empirical and clinical focus of child andadolescent psychotherapy research. Journal of Consulting and Clinical Psychology, 58, 729-740.doi:10.1037/0022-006X.58.6.729

Kazdin, A.E., & Whiteley, M.K. (2006). Comorbidity, case complexity, and effects of evidence-basedtreatment for children referred for disruptive behavior. Journal of Consulting and ClinicalPsychology, 74, 455-467. doi:10.1037/0022-006X.74.3.455

Kazdin, A.E., & Wiesz, J.R. (1998). Identifying and developing empirically supported child andadolescent treatments. Journal of Consulting and Clinical Psychology, 66, 19-36. doi:10.1037/0022-006X.66.1.19

Lambert, M.J., & Archer, A. (2006). Research findings on the effects of psychotherapy and theirimplications for practice. In C.D. Goodheart, A.E. Kazdin, & R.J. Sternberg (Eds.). Evidence-based psychotherapy: Where practice and research meet (pp. 111-130). Washington, DC:American Psychological Association. doi:10.1037/11423-005

Leichsenring, F. & Leibing, E. (2003). The effectiveness of psychodynamic therapy and cognitivebehavior therapy in the treatment of personality disorders: A meta-analysis. American Journalof Psychiatry, 160, 1223-1232. doi: 10.1176/appi.ajp.160.7.1223

Leichsenring, F., Rabung, S., & Leibing, E. (2004). The efficacy of short-term psychodynamicpsychotherapy in specific psychiatric disorders: A meta-analysis. Archives of General Psychiatry,61, 1208-1261. doi: 10.1001/archpsyc.61.12.1208

Levitt, E.E. (1957). The results of psychotherapy with children: An evaluation. Journal of ConsultingPsychology, 21, 189-196. doi:10.1037/h0039957

Levitt, E.E. (1963). Psychotherapy with children: A further evaluation. Behaviour Research andTherapy, 1, 45-51. doi:10.1016/0005-7967(63)90007-X

Lewinsohn, P.M. & Clarke, G.N. (1999). Psychosocial treatments for adolescent depression. ClinicalPsychology Review, 19, 329-342. doi: 10.1016/S0272-7358(98)00055-5

Maughan, D.R., Christainsen, E., Jenson, W.R., Olympia, D., & Clark, E. (2005). Behavioral parenttraining as a treatment for externalizing behaviors and disruptive behavior disorders: Ameta-analysis. School Psychology Review, 34, 267-286.

McCleod, B.D. & Weisz, J.R. (2004). Using dissertations to examine potential bias in child andadolescent clinical trials. Journal of Consulting and Clinical Psychology, 72, 235-251. doi:10.1037/0022-0006X.72.2.235

Mellor-Clark, J., Barkham, M., Connell, J., & Evans, C. (1999). Practice-based evidence and stan-dardized evaluation: Informing the design of the CORE system. European Journal of Psycho-therapy, Counselling and Health, 2, 357-374. doi:10.1080/13642539908400818

Meltzoff, J., & Kornreich, M. (1970). Research in psychotherapy. New York: Atherton Press.Messer, S.B. & Abbass, A.A. (2010). Evidence-based psychodynamic therapy with personality disor-

ders. In Magnavita, J.J. (Ed.) Evidence-based treatment of personality dysfunction: Principles,methods, and processes (pp. 79-111). Washington, DC: American Psychological Association.doi: 10.1037/12130-004

Messer, S.B., & Boals, G.F. (1981). Psychotherapy outcome in a university-based psychology trainingclinic. Professional Psychology, 12, 785-793. doi:10.1037/0735-7028.12.6.785

Michael, KD. & Crowley, S.L. (2002). How effective are treatments for child and adolescentdepression? A meta-analytic review. Clinical Psychology Review, 22, 247-269. doi: 10.1016/S0272-7358(01)00089-7

Minami, T., Wampold, B.E., Serlin, R.C., Kircher, J.C., & Brown, G.S. (2007). Benchmarks forpsychotherapy efficacy in adult major depression. Journal of Consulting and Clinical Psychology,75, 232-243. doi:10.1037/0022-006X.75.2.232

Norton, P.J. & Philipp, L.M. (2008). Transdiagnostic approaches to the treatment of anxiety disorders:A quantitative review. Psychotherapy: Theory, Research, Practice, Training, 45, 214-226.

Oei, T.P.S. & Dingle, G. (2008). The effectiveness of group cognitive therapy for unipolar depressivedisorders. Journal of Affective Disorders, 107, 5-21. doi: 10.1016/j.jad.2007.07.018

Parry, G., Castonguay, L.G., Borkovec, T.D., & Wolf, A.W. (2010). Practice research networks andpsychological services research in the UK and USA. In M. Barkham, G.E. Hardy, & J.Mellor-Clark (Eds.). Devleoping and delivering practice-based evidence: A guide for the psycho-

AMERICAN JOURNAL OF PSYCHOTHERAPY

44

logical therapies (pp. 311-325). West Sussex, UK: Wiley-Blackwell. doi:10.1002/9780470687994.ch12

Peleikis, D.E. & Dahl, A.A. (2005). Psychotherapy Research, 15, 304-315. doi: 10.1080/10503300500091835

Persons, J.B., Burns, D.D., & Perloff, J.M. (1988). Predictors of dropout and outcome in cognitivetherapy in a private practice setting. Cognitive Therapy and Research, 12, 557-575. doi:10.1007/BF01205010

Reinecke, M.A., Ryan, N.E., & DuBois, D.L. (1998a). Cognitive-behavioral therapy of depression anddepressive symptoms during adolescence: A review and meta-analysis. Journal of the AmericanAcademy of Child & Adolescent Psychiatry, 37, 26-34. doi: 10.1097/00004583-199801000-00012

Reinecke, M.A., Ryan, N.E., & DuBois, D.L. (1998b). “Meta-analysis of CBT for depression inadolescents”: Dr. Reinecke et al reply. Journal of the American Academy of Child & AdolescentPsychiatry, 37, 1006-1007.

Sanderson, W.C., & Woody, S. (Eds.)(1995a). Manuals for empirically validated treatments. TheClinical Psychologist, 48(4), 7-11.

Sanderson, W.C., & Woody, S. (Eds.)(1995b). Manuals for empirically validated treatments: A projectof the Task Force on Psychological Interventions, Division of Clinical Psychology, AmericanPsychological Association, 28 pp. http://psycnet.apa.org/psycextra/550792009-001.pdf

Seidel, J.A. (2012, August). Feedback-informed treatment: The devil is in the details. In C.D.Goodheart (Chair), Practice-based evidence of psychotherapy’s effectiveness. Symposium con-ducted at the meeting of the American Psychological Association, Orlando, FL.

Seligman, M.E.P. (1995). The effectiveness of psychotherapy: The Consumer Reports study. AmericanPsychologist, 50, 965-974. doi:10.1037/0003-066X.50.12.965

Shadish, W.R., Matt, G.E., Navarro, A.N., Siegle, G., Crits-Christoph, P., Hazelrigg, M.D.,. . .Weiss,B. (1997). Evidence that therapy works in clinically representative conditions. Journal ofConsulting and Clinical Psychology, 65, 355-365. doi: 10.1037/0022-006X.65.3.355

Shapiro, D.A. & Shapiro, D. (1982). Meta-analysis of comparative therapy outcome studies: Areplication and refinement. Psychological Bulletin, 92, 581-607. doi: 10.1037/0033-2909.92.3.581

Smith, T.B., Bartz, J., & Richards, P.S. (2007). Outcomes of religious and spiritual adaptations topsychotherapy: A meta-analytic review. Psychotherapy Research, 17, 643-655. doi: 10.1080/10503300701250347

Smith, M.L., & Glass, G.V. (1977). Meta-analysis of psychotherapy outcome studies. AmericanPsychologist, 32, 752-760. doi:10.1037/0003-066X.32.9.752

Smith, M.L., Glass, G.V., & Miller, T.I. (1980). The benefits of psychotherapy. Baltimore: John HopkinsUniversity Press.

Stage, S.A. & Quiroz, D.R. (1997). A meta-analysis of interventions to decrease disruptive classroombehavior in education settings. School Psychology Review, 26, 333-368.

StatMost statistical analysis and graphics: User’s handbook. (1994). Salt Lake City: DataMost Corpo-ration.

Steinbrueck, S.M., Maxwell, S.E., & Howard, G.S. (1983). A meta-analysis of psychotherapy and drugtherapy in the treatment of unipolar depression with adults. Journal of Consulting and ClinicalPsychology, 51, 856-863. doi: 10.1037/0022-006X.51.6.856

Stiles, W.B., & Barkham, M. (2012, August). Practice-based evidence from the United Kingdom usingthe CORE system. In C.D. Goodheart (Chair), Practice-based evidence of psychotherapy’seffectiveness. Symposium conducted at the meeting of the American Psychological Association,Orlando, FL.

Stiles, W.B., Barkham, M., Connell, J., & Mellor-Clark, J. (2008). Responsive regulation of treatmentduration in routine practice in United Kingdom primary care settings: Replication in a largersample. Journal of Consulting and Clinical Psychology, 76, 298-305. doi: 10.1037/0022-006X.76.2.298

Stiles, W.B., Barkham, M., Mellor-Clark, J., & Connell, J. (2008). Effectiveness of cognitive-behav-ioural, person-centered, and psychodynamic therapies in UK primary-care routine practice:Replication in a larger sample. Psychological Medicine, 38, 677-688. doi: 10.1017/S0033291707001511

Stiles, W.B., Barkham, M., Twigg, E., Mellor-Clark, J., & Cooper, M. (2006). Effectiveness ofcognitive-behavioural, person-centered and psychodynamic therapies as practiced in UK

Practice-Based Evidence from 45 Years

45

National Health Service settings. Psychological Medicine, 36, 555-566. doi: 10.1017/S0033291706007136

Swift, J.K. & Greenberg, R.P. (2012). Premature discontinuation in adult psychotherapy: A meta-analysis. Journal of Consulting and Clinical Psychology, 80, 547-559. doi: 10.1037/a0028226

Watkins, C.E. Jr. (2012). The study of men in short-term and long-term psychodynamic psychother-apy: A brief research note. Psychoanalytic Psychology, 29, 429-439.doi: 10.1037/a0025183

Weisz, J.R., Jensen-Doss, A., & Hawley, K.M. (2006). Evidence-based youth psychotherapies versususual clinical care: A meta-analysis of direct comparisons. American Psychologist, 61, 671-689.doi:10.1037/0003-066X.61.7.671

Weisz, J.R., McCarty, C.A., & Valeri, S.M. (2006). Effects of psychotherapy for depression in childrenand adolescents: A meta-analysis. Psychological Bulletin, 132, 132-149. doi: 10.1037/0033-2909.132.1.132

Weisz, J.R., Weiss, B., Alicke, M.D., & Klotz, M.L. (1987). Effectiveness of psychotherapy withchildren and adolescents: A meta-analysis for clinicians. Journal of Consulting and ClinicalPsychology, 55, 542-549. doi: 10.1037/0022-006X.55.4.542

Weisz, J.R., Weiss, B., Han, S.S., Granger, D.A., & Morton, T. (1995). Effects of psychotherapy withchildren and adolescents revisited: A meta-analysis of treatment outcome studies. PsychologicalBulletin, 117, 450-468. doi:10.1037/0033-2909.117.3.450

Wise, E.A. (2003). Psychotherapy outcome and satisfaction methods applied to intensive outpatientprogramming in a private practice setting. Psychotherapy: Theory, Research, Practice, Training,40, 203-214. doi:10.1037/0033-3204.40.3.203

Young, B. (2007). The efficacy of psychoanalysis and the analytic therapies: Reflections of a psycho-analyst and her former patients. Journal of the American Academy of Psychoanalysis & DynamicPsychiatry, 35, 313-334. doi:10.1521/jaap.2007.35.2.313b

AMERICAN JOURNAL OF PSYCHOTHERAPY

46