Client Feedback Form Manual - Beacon Health Options...Client Feedback Form Manual Prepared for...

On Track Outcomes Program

Client Feedback Form Manual

Prepared for Beacon Health Options by Jeb Brown, PhD, Center for Clinical Informatics

February 2014

For more information about this manual contact Jeb Brown, Ph.D. at

[email protected], or call (801) 541-9720

For more information about the On Track Outcomes Program send email to [email protected]

mailto:[email protected]

mailto:[email protected]

Beacon Health Options 2 Client Feedback Forms

Table of Contents

Section 1: Overview of the Client Feedback Form ...............................................................................4

Administration .......................................................................................................................................... 4

Outcomes Questionnaires and Clinician Feedback................................................................................... 5

Section 2: Development, Scoring, and Interpretation .........................................................................6

Development............................................................................................................................................. 6

Absenteeism/Presenteeism ...................................................................................................................... 7

Alliance Scale ............................................................................................................................................ 7

Global Distress Scoring and Interpretation............................................................................................... 8

Client Feedback Form Score Ranges for the Three Severity Levels ...................................................... 9

Monitoring Change over Time.................................................................................................................. 9

Case Mix Adjustment and Outcomes Benchmarking ............................................................................. 10

Reliability and Validity............................................................................................................................. 11

Section 4: CFF-Adult Specific Information.........................................................................................13

Reliability and Construct Validity ............................................................................................................ 13

CFF-ADULT Factor Analysis (N=86,185) .............................................................................................. 14

IRT Analysis of Global Distress and Substance Abuse Scales .............................................................. 15

Formulas to predict final score (benchmark score) for the episode of care ...................................... 16

Section 5: CFF – Adolescent Specific Information..............................................................................17


Youth CFF Factor Analysis (Responder=Youth, N=9,647) ................................................................... 18

Youth CFF Factor Analysis (Responder= Parent or other adult, N=3747)........................................... 19


Section 6: CFF – Child Specific Information .......................................................................................22


Child CFF Factor Analysis (Responder=Youth, N=3,278)..................................................................... 23

Child CFF Factor Analysis (Responder=Parent or other adult, N=6,114) ............................................ 24


References............................................................................................................................. ..........26

Appendix A: Adult CFF Normative and Benchmarking Data ..............................................................28

Results of GLM prediction - adding variables incrementally ............................................................. 29


Appendix B: Youth Self-Report CFF Normative and Benchmarking Data ............................................30


Appendix C: Youth - Parent CFF Normative and Benchmarking Data .................................................32

Results of GLM prediction - adding variables incrementally .............................................................. 33

Appendix D: Child - Self CFF Normative and Benchmarking Data .......................................................34

Results of GLM prediction - adding variables incrementally .............................................................. 35

Appendix E: Child - Parent CFF Normative and Benchmarking Data ...................................................36



Section 1: Overview of the Client Feedback Form

The Client Feedback Forms (CFFs) used in Beacon’s On Track Outcomes Program are brief, reliable and

valid client completed questionnaires designed to measure improvement in symptoms, quality of social

relations, and functioning at work/school and other daily activities. Use of these or similar outcome

questionnaires has been shown to significantly improve treatment outcomes across large sample of

clinicians treating thousands of clients.

The questionnaires measure symptom severity and improvement across a broad range of problems, and

are not intended to be diagnostic or to substitute for clinical evaluation or other assessments that a

clinician may routinely conduct.

There are three versions of the CFF now available:

Adult version (age 18 and older)

Youth version (ages 13-17; completed by youth or adult)

Child version (under age 13; completed by youth or adult)

All versions of the questionnaires have high reliability (coefficient alpha => .87). Extensive factor

analyses demonstrate high construct validity, with items loading on the common factor found in the

most commonly used measures of treatment outcomes.

The Client Feedback Forms are unique among available outcome measures in that items asking for

feedback on the working alliance are included on every questionnaire. The use of alliance items has

been shown to reduce no shows and premature termination while contributing significantly to

improved outcomes.

Administration

The questionnaires were designed for routine use in clinical practice and can be completed by

most clients in less than two minutes, using a simple paper form.

When first introducing the questionnaire to a client, it is best practice to provide a brief

explanation of the reason for the questionnaire, and how it will be used as a routine part of

treatment. At many sites, this explanation is provided by office staff when the client checks in.

Following are a sample explanation scripts used by support staff:


“Please take a moment to fill out the questionnaire. This will help you and your therapist

talk about how treatment is going.”

“We are really excited about the questionnaires. Research shows that therapists who use

questionnaires like these get much better outcomes.”

“Please answer as honestly as you can. This is important to your treatment, because it will

help your therapist understand how to help you.”

Research asking clients to provide feedback on their experience with the questionnaires

indicated a high level of honesty and willingness to complete the alliance items, especially if

they perceived that the clinician was interested in their responses. Of this group, 95% agreed

that the questionnaires were helpful in treatment.

For this reason it is recommended that the clinician acknowledges and value the client’s

willingness to give feedback. Taking a few moments at the start of the session to review the

questionnaire provides the clinician with a wealth of clinically relevant information while

acknowledging the value of the client’s time to complete the questionnaire.

Research of the past several years has revealed that clinicians who use the questionnaires with

a high percentage of their practice display significant gains in overall outcomes.

Outcomes Questionnaires and Clinician Feedback

A large body of research supports the proposition that routine use of outcomes questionnaires

combined with feedback to the clinician results in significant improvement in treatment

outcomes, as measured by pre-post change, percentage of patients improved, and reduced

dropout rates (Lambert, 2009; Goodman et al., 2013).

Among users of On Track Client Feedback Forms and related ACORN questionnaires, observation of frequency of measurement and use of the Decision Support Toolkit provides real world confirmation of results from clinical trials. Both the number of clients measured and the frequency at which the clinician views the data are strongly correlated with year-to-year improvement in treatment outcomes at the clinician level (r> .3; p<.0001; see Brown 2013 in references).


Section 2: Development, Scoring, and Interpretation Development

The CFFs were developed for Beacon’s On Track Outcomes program as part of a suite of forms

developed cooperatively through A Cooperative Online Resource Network (ACORN). The

network consists of statisticians and researchers affiliated with the Center for Clinical

Informatics along with over 1,000 clinicians using client completed questionnaires in a wide

variety of clinical settings across the country.

The initial work on developing the ACORN questionnaire items was done in collaboration with

Warren Lambert, Ph.D., at Vanderbilt University. Dr. Lambert was instrumental in the

development of the Peabody Treatment Progress Battery (PTPB) for adolescents. Items from

the PTPB were utilized with permission, and additional items for adults, adolescents and

children were added using the same item format. Normative data on these items was collected

through the ACORN network of clinicians. Takuya Minami, Ph.D., of the University of

Massachusetts Boston, further assisted in the analyses of the psychometric properties of all of

the items.

The ACORN form development process takes advantage of the network’s ability to collect large

amounts of data and continuously test and refine items over time. Rather than focus on the

development of static forms, this process centers on the development of reliable and valid

items that can be combined flexibly. The psychometric properties of each item, as well as the

unique combination, are carefully evaluated with items for a specific questionnaire selected

based on the population to be measured and the needs of the various participating

organization. The result is a large item inventory with data from over 200,000 clients.

Evidence of the validity and reliability of the CFF was derived from data on tens of thousands of

administrations in both clinical and community settings. The development process included

item analyses to determine:

Item frequencies and distributions

Item correlations

Factor structure

Construct validity

Scale reliability

Sensitivity to change The CFFs consists of items well-suited for general use in outpatient settings. The questionnaires

are designed to be as brief as possible while retaining excellent psychometric properties. The


Adolescent and Child versions may be completed either by the youth or by an adult who knows

the youth well.

Absenteeism/Presenteeism

The two absenteeism/presenteeism items on the adult form are based on the format and

wording of items in the Health and Productivity Questionnaire (HPQ), a questionnaire in the

public domain developed by Kessler and colleagues at Harvard University. Because the CFF is

designed for use by mental health professionals in a mental health setting, the HPQ wording

has been revised to focus on absenteeism/presenteeism specifically due to mental health

problems, rather than both mental and physical health problems encompassed by the original

HPQ questions.

Alliance Scale

All ACORN questionnaires also include items asking the client to provide feedback on their

experience of the prior session. The use of these so-called Alliance items ask for feedback on

elements of the therapeutic working alliance, such as agreement on treatment goals, and the

client’s perception of the quality of the relationship. Use of alliance measure has been

demonstrated to reduce treatment dropout and improve outcomes. Among the large sample

of clinicians using the CFFs, about 80% of clients complete these items routinely. These clients

have significantly better outcomes than those who fail to complete the items.

The client’s responses on these items may be easily influenced by their perception of the

clinician’s response. Clients may be reluctant to give anything other than perfect ratings in

order to avoid hurting the clinician’s feelings or out of fear of possible consequence to the

clinician if “Satisfaction” ratings are high.

It rests on the skill of the clinician to create a therapeutic environment that encourages honest

feedback on the alliance items. Skilled clinicians are about to use the alliance items as tools to

foster a strong collaborative working alliance with the client.

Consistent ratings of near perfect alliance are NOT associated with the best outcomes. Rather,

clients who provide meaningful feedback early in therapy are very likely to rate the alliance as

improved over time. This pattern, displayed by approximately one third of clients, is associated

with significantly greater improvement in treatment.

The best outcomes are associated with improvement on the Alliance Scale over the course of

the treatment episodes. This means that the patients with the best outcomes are also willing

to give feedback that the treatment encounters early in the treatment episode are less than

perfect, otherwise there is no room for improvement.


Failing to complete the alliance items is associated with less improvement. If the client leaves

the items blank, this provides the clinician with an opportunity to initiate a discussion of how

the client is experiencing the treatment process.

Global Distress Scoring and Interpretation

The core global distress scale (GDS) is scored as the mean of all non-missing items on the form.

If more than 4 items are missing from the adult global distress scale or 6 items from the child or

youth scales, the questionnaire is not scored.

Scores can be divided into three severity ranges (Normal, Moderate, Severe) based on

normative data from clinical and community samples. The cut-off scores for each range are

included with the information on the individual questionnaires.

In the case of the adult questionnaire, the cut-off score for the Normal Range was determined

by collecting a sample of over 1,000 individuals from the community who had never sought

mental health service combined with a sample of over 75,000 individuals receiving mental

health services. The cut-off score was calculated using the method proposed by Jacobson &

Truax (1991) as represented by this formula:

C = (SD1)(mean2)+(SD2)(mean1)

SD1+SD2

A score in the normal range means that the score is in a range typical of respondents from a

community sample that have never sought mental health services. Seventy-five percent (75%)

of a community sample and 25% of a clinical sample will fall into this range. Mental health

clients with scores in this range tend to not show improvement with treatment.

A score in the moderate range of distress is characteristic of individual seeking mental health

services. About 50% of a clinical sample will fall into this range. Mental health clients with

scores in this range tend to show significant improvement within a few sessions of therapy, and

most complete treatment with a good outcome in fewer than 8 sessions.

Twenty five percent (25%) of a clinical sample will have scores in the severe range, while fewer

than 10% of a community sample will fall in this range. Clients with scores in this range are

highly likely to show rapid improvement with psychotherapy, but may need more sessions to

realize the full benefit of treatment.


In the case of child and adolescent questionnaires, cut-off scores were estimated by the 25th

and 75th percentile. Practicality prevented collection of large enough community samples.

However, the cut-off scores are comparable to those reported for other similar measures with

the OQ-45 for adults and YOQ-64 for children and youth. Items were selected for the Client

Feedback Form in order to assure comparability to the OQ-45 and YOQ-64. Analysis of archival

data for the OQ-45 and YOQ-64 confirms that the Client Feedback Forms produce results very

similar to these OQ measures.

Client Feedback Form Score Ranges for the Three Severity Levels

Form Normal Range Moderate Range Severe Range Adult CFF 0 to 1.5 1.6 to 2.5 2.6 to 4.0 Adolescent CFF

(Youth completed) 0 to 1.0 1.1 to 1.8 1.9 to 4.0

Adolescent CFF (Parent/Adult completed)

0 to 1.2 1.3 to 1.9 2.0 to 4.0

Child CFF (Youth completed)

0 to 1.0 1.1 to 1.8 1.9 to 4

Child CFF (Parent/Adult completed)

0 to 1.2 1.3 to 1.9 2.0 to 4.0

Monitoring Change over Time

The ability for the clinician to monitor client change as the treatment unfolds is one of the

features of On Track that leads to improved outcomes. Identification of clients who are “off

track” assists the clinician in preventing early drop-out in treatment.

The On Track Decision Support Toolkit provides graphs for each case. The actual client scores

are compared to a “predicted score” at each assessment point. The predicted score is

determined using a statistical prediction technique known as General Linear Modeling. The

method takes advantage of all of the normative data for other clients completing multiple

questionnaires at multiple points in treatment.

The predicted score at each assessment is computed using the initial CFF Global Distress Score,

the assessment number, and the number of weeks that have passed since the initial

assessment to determine the expected assessment at each measurement point. The actual

score can them be compared to the predicted score in order to determine the extent that the

clients current score deviates from the expected score. Clients who scores are significantly

higher than expected at classified as “off track”.


Case Mix Adjustment and Outcomes Benchmarking

Simply measuring pre-post change on an outcome questionnaire provides little information

without some basis for comparison or benchmarking. The On Track program is based on a long

history of research using real world effectiveness data as well as meta-analyses of clinical trials

to establish efficacy.

At the most basic level, change scores on the CFF questionnaires are converted to effect size,

based on dividing the pre-post global distress change score by the standard deviation of the

global distress scores at intake. A simple effect size is calculated by dividing the pre-post change

score by the standard deviation of the outcome measure at intake. An effect size of one means

that the client improved one standard deviation on the measure.

In order to make results comparable to results from clinical trials, effect size is only calculated

for cases with intake scores above the clinical cutoff score. In a general outpatient population,

75% of cases will be in the clinical range. Another reason for excluding non-clinical range cases is that these who enter treatment

reporting little distress do not show improvement on average. This would have the effect of

artificially lowering effect sizes and making comparisons to results from published studies

invalid.

The second manner in which outcomes are benchmarked is by using a large and diverse

normative sample of over 90,000 outpatient treated at hundreds sites around the country. The

sample represents a wide range of ages and ethnic groups, includes those covered by

commercial insurance, EAP programs, Medicaid/Medicare, as well as self-pay and other sources

of funding. As such, the sample is highly representative of patients seen in outpatient general

practice.

In order to establish a benchmark, the statistical procedure known as General Linear Model was

applied to establish which variables collected at intake predicted the final global distress score at

the end of the episode.

Employing this model, it us evident that the first global distress score is the strongest predictor

of subsequent scores, including the final score. The intake score alone accounts for 30% to 50%

of the variance in final scores, depending on the length of treatment.

A second predictor in naturalistic data such as On Track is the session at which the first

questionnaires were administered. If the first assessment is at intake or no later than the

second appoint, measured change is larger than if the first assessment is later. In many cases,

the session number for the first assessment is unknown. In order to take session number into

account, each episode is classified as Early Assessment (session 1 or 2), Later Assessment, or

Beacon Health Options

11 Client Feedback Forms

Unknown. These three categories are included in the General Linear Model as a class variable,

but explain less than 1% of additional variance.

A third potential predictor is diagnosis. Inclusion of diagnosis adds minimally to the prediction

of final score, typically explaining less than .005% of additional variance. In some instances,

such as the youth self-reported version of the CFF, diagnosis is a non-significant predictor.

Diagnosis is not currently included in the On Track benchmarking models.

The prediction of the final (benchmark) score for the On Track program is based on predictive

formulas. These formulas take the form of a simple regression formula: Last Score = First Score

* slope + intercept). The regression formulas are coded from each version of the questionnaire,

with regression formulas differing based on session number at first assessment. Sections 4-6

contain form-specific analyses, including the specific formulas used for each CFF form and

session number.

The appendices provide detailed normative information for each questionnaire with regards to

mean intake scores, final scores, change scores, number of assessments, and average time pre-

post (reported in weeks) for the entire sample, including a breakout by diagnostic group. A

second set of normative information is provided for only those cases with intake scores in the

clinical range.

Each patient’s actual final score is compared to the benchmark target to determine to what

extent the patient reported more or less improvement than comparable patient in the

complete normative sample. The results of the General Linear Model analyses from which the

formulas are derived are provided in the appendices.

The general methodology for benchmarking outcomes has been published in a series of peer

reviewed journal articles, though in these publications the benchmark scores are calculated

using a multivariate GLM. These publications are available upon request (Minami et al. 2007;

Minami et al. 2008a and 2008b; Minami et al. 2011).

Reliability and Validity

Reliability of the global distress scale is measured using Cronbach’s alpha, which is a measure of

internal consistency. This is consistent with classical test theory, which seeks to develop single

factor scales with internal consistency.

Reliability for the CFFs are as follows:

Adult version = 0.89

Youth version = 0.87

Child version = 0.90


Validity of the scales is estimated primarily as construct validity, addressing whether the

questionnaire measures a single construct or factor. Prior research indicates that items on most

outcomes questionnaires used in behavioral health correlate highly with one another and

measure a single construct, generally referred to as “Global Distress.”

For example, Brophy et al. (1988) found that the SCL-90 subscales all load on a common factor, and likewise correlate highly with similar scales from other measures. Miller et al. (2003) found that the Outcome Rating Scale correlates highly with the OQ-45.

Enns et al. (1998) performed factor analyses on the Beck Depression Inventory and the Beck Anxiety Inventory. To quote from this study:

" [T]he parameter estimate was very high (0.784) and a unidimensional, single-factor model of negative affectivity approached the criteria for good fit. It was concluded that the Beck Anxiety and Depression Inventories assess distinct anxiety and depression phenomena to a limited extent when used in a clinically depressed sample."

Recent research by the ACORN collaboration further investigates the relationship between

items assessing states of emotional well-being and high life satisfaction with measures of

psychiatric symptoms and lost productivity (Brown & Minami, 2013). This work demonstrates

that measures of well-being and life satisfaction likewise correlate highly with the same

common factor as symptoms and lost productivity.

The existence of a global distress factor and the fact that multiple outcome questionnaires,

including widely used measures of depression, are all found to be correlated with one another

provide strong evidence of the construct validity of patient self-report outcome measures

designed to measure and assess global subjective distress.

The estimate of reliability and validity for the Alliance Scale is complicated by the fact that the

responses are not normally distributed. Rather, approximately 50% of clients report the alliance

is virtually perfect all of the time, with little variance from week to week. This pattern is the

most common, but is not associated with better than average outcomes. Less than perfect

Alliance Scores at the start of the treatment are associated with better outcomes if the client

remains engaged in treatment. To a large extent the utility of the Alliance Scale is dependent on

the skill of the clinician to elicit and utilize frank feedback from the client. The following sections describe the psychometric properties of each version of the Client

Feedback Form, including detailed information on item analyses.



Section 4: CFF-Adult Specific Information Reliability and Construct Validity

The Adult Client Feedback Form consists of 10 items which assess symptoms of depression,

anxiety, social relationships, and functioning in work and other daily activities. Three additional

items assess problems related to substance abuse. In addition, the forms contain three items

which ask for feedback on the client’s last session experience. Use of these Alliance items is

associated with better outcomes.

The adult form also includes items asking about prior treatment and presence of chronic

illnesses. The variables may be used when performing calculations for case mix adjustment.

The following assessment of the factor structure and reliability of the Adult questionnaire is

based on a sample of 86,185 clients completing the questionnaire at the start of a treatment

episode.

Factor analyses (principal components and varimax rotation) reveal that these items related to

depression, anxiety, interpersonal problems, and impaired functioning in work, school and

other daily activities all load on a common factor labeled Global Distress. Table A displays the

results of this analysis.


CFF-ADULT Factor Analysis (N=86,185)

Factor 1 in principle components is the Global Distress factor. Note that all GDS items have positive factor loadings of .54 or greater.

Varimax rotation does little to alter factor structure. Global Distress Scale contains a single factor.

Factor Method: Principle Components

Factor Method: Varimax Rotation

Scale/Subscale Item wording Factor 1 Factor 2 Factor 1 Factor 2

GDS/Symptoms ...feel unhappy or sad? .797 -.096 .801 .055

GDS /Functioning …feel unproductive at work or other daily activities?

.752

-.136

.764

.008

GDS /Symptoms …have problems with sleep (too much or too little)?

.687

-.091

.692

.040

GDS /Symptoms …feel tense or nervous? .735 -.102 .741 .038

GDS /Symptoms …have little or no energy? .744 -.160 .761 -.018

GDS /Symptoms …feel hopeless about the future? .799 -.058 .796 .093

GDS /Social

…have a hard time getting along with family, friends, or coworkers?

.591

.002

.581

.113

GDS /Social …feel lonely .717 -.049 .713 .086

GDS /Symptoms think about harming yourself .543 -.094 .516 .194

GDS /Symptoms …have a hard time paying attention?

.720

-.104

.726

.033

Substance Abuse

…have someone express concerns about your alcohol or drug use?

.257

.839

.096

.873

Substance Abuse

…have had a problem at work, school or home because of alcohol or drug use?

.223

.805

.068

.833

Substance Abuse ...consume five or more drinks on a single occasion

.190

.761

.044

.783

Variance Explained by Each Factor

5.233

2.033

5.121

2.142

Final Commonality Estimate

7.262568

Ten of the items load heavily on the Global Distress Factor. This results in high reliability for the

Global Distress Scale. Reliability as estimated using Cronbach’s coefficient alpha was 0.89.

The CFF-Adult can be scored with subscales for symptoms, interpersonal problems and

functioning. However, these subscales cannot be identified using factor analysis.



The Substance Abuse items do form a separate factor, and are scored on a separate scale with a

Cronbach’s alpha of 0.78. Correlation between the Global Distress Scale and Substance Abuse

scale was .14 (p< .001). The question “how often did you have five or more drinks of alcohol at

one time?” is based on items widely used in assessment of binge drinking behavior, such as

those used in the National Household Survey on Drug Abuse (SAMHSA, 1998) and Harvard

School of Public Health College Alcohol Study (Henry , 1997). The addition of a question that

touches on binge drinking behavior is a valuable supplement.

The evidence of a common factor for global distress demonstrates construct validity. Similar

measures of common symptoms of depression and anxiety have likewise been shown to share

a common factor. These include the Beck Depression Inventory, the Beck Anxiety Inventory,

the OQ-45, and the PHQ9. Correlational studies between the Adult questionnaire and the both

the PHQ9 and the Beck Depression Inventory show a concurrent validity of 0.8.

IRT Analysis of Global Distress and Substance Abuse Scales

The initial development of the Adult form was informed by an IRT analysis of the 10 global

distress scale and 3 substance abuse scale items. These analyses were conducted on a smaller

dataset available at the time of initial form development.

Label

N

Mean

Vari- ance

Kurt- osis

Min

Max

Item- total

MSA

Std. Loading on One Factor

Rasch

Meas- ure

Score

Infit

Out- fit

Disc.

MH (Global Distress)

Unhappy or sad 494 2.4 0.9 -0.6 0 4 0.7 0.91 0.8 39.5 0.6 0.6 1.4

Little or no energy 496 2.1 1.0 -0.5 0 4 0.5 0.94 0.6 44.3 1.0 1.0 0.9

Getting along with family and friends

496

1.6

1.1

-0.3

0

4

0.5

0.94

0.5

52.4

1.2

1.2

0.7

Hard time paying attention

495

1.9

1.2

-0.6

0

4

0.6

0.92

0.7

48.1

0.9

0.8

1.2

Unproductive at work 488 1.7 1.3 -0.7 0 4 0.6 0.91 0.7 50.7 1.0 1.0 1.0

Problems with sleep 490 2.2 1.5 -0.9 0 4 0.6 0.94 0.7 43.1 1.1 1.1 0.9

Tense or nervous 490 2.1 1.4 -0.7 0 4 0.7 0.92 0.7 44.8 0.9 0.9 1.2

Lonely 488 2.0 1.4 -0.8 0 4 0.6 0.92 0.6 46.5 1.2 1.2 0.8

Hopeless 497 1.7 1.3 -0.6 0 4 0.7 0.91 0.8 51.8 0.8 0.8 1.3

Self-harm

490

0.3

0.5

4.7

0

4

0.3

0.90

0.3

78.9

1.7

1.4

0.7

Substance Abuse

Someone expressed SA concerns

488

0.3

0.6

7.5

0

4

0.8

0.7

0.9

46.3

0.9

0.9

1.2

SA problem at work/home

488

0.2

0.4

20.8

0

4

0.6

0.7

0.7

65.8

1.3

1.0

0.9

Five or more drinks 493 0.4 0.7 4.7 0 4 0.7 0.7 0.8 37.9 1.0 1.0 1.0


The table below summarizes the overall scale characteristics based on this sample:

Scale

Items

Cronbach's alpha

MSA

Bentler CFI 1 factor

RMSEA 1 factor

Rasch Person

Reliability

Rasch Item Separation Reliability

MH (Global Distress) 10 0.87 0.92 0.96 0.06 .85-.88 1.00

SA 3 0.84 0.71 1.00 0.00 .16 - .19 0.98

Formulas to predict final score (benchmark score) for the episode of care

First assessment at session 1 or 2 Last GDS score=First GDS Score * 0.5756915262+0.4366357297 (R-square=.30)

First assessment at session 3 or later

Last GDS score=First GDS Score * 0.6236339929 + 0.4950503979 (R-square=.37)

First assessment session unknown Last GDS score=First GDS Score * 0.6088506340 + 0.4065135658 (R-square=.40)

See Appendix A for detailed normative information and predictive models.



Section 5: CFF – Adolescent Specific Information

Reliability and Construct Validity

The Adolescent version of the CFF can be completed by either the youth or a parent. The

questionnaire contains a total of 15 items, seven of which assess symptoms of anxiety and

depression, while eight assess symptoms related to attention and behavioral problems. A

single item inquires about drug or alcohol use.

Factor analysis (principal components) likewise reveals that all items load on a common factor,

as evidenced by reliability (coefficient alpha) of 0.87 for the full scale. The sample size for this

analysis was 13,394. The factor structure was the same whether the questionnaire was

completed by the youth or an adult who knew the youth well.

Factor analysis with varimax rotation indicates the existence of two factors, one containing

symptoms of anxiety and depression (sometimes referred to as internalizing symptoms) while

the second contains symptoms related to attention and behavioral problems (externalizing

symptoms). The main difference is that the internalizing items accounted for slightly more

variance than externalizing items with the youth report version, while the opposite was the

case for the parent completed version.


Youth CFF Factor Analysis (Responder=Youth, N=9,647)


Rotated Factor 1 reflects symptoms of depression and anxiety. This accounts for more of the variance with Youth completed forms than Factor 2, which reflects attention and behavioral problems. Factor 3 is substance abuse.

Factor Method:

Principle Components

Factor Method:

Varimax Rotation

Scale

Item wording Factor

1

Factor 2

Factor 3

Factor 1

Factor 2

Factor 3

GDS/Symptoms …eat a lot more or a lot less than usual?

.530

-.115

.081

.461

.224

.196

GDS/Social

…get into trouble?

.520

.527

.088

.011

.649

.368

GDS/Social …interrupt others? .555 .479 -.265 .081 .775 .040

GDS/Social …lie to get things you wanted? .576 .408 .112 .133 .594 .375

GDS/Symptoms …have a hard time controlling your temper?

.644

.180

.040

.342

.510

.268

GDS/Symptoms …worry about a lot of things? .649 -.433 .057 .767 .092 .126

GDS/Symptoms

…feel worthless?

.702

-.403

.094

.783

.131

.183

GDS/Symptoms …have a hard time having fun? .607 -.449 -.057 .751 .099 .004

GDS/Social

…have a hard time waiting your turn?

.580

.315

-.278

.212

.684

-.008

GDS/Social …hang out with kids who get into trouble?

.447

.363

.481

.059

.340

.666

GDS/Social …feel nervous and/or shy around other people?

.530

-.365

-.152

.640

.144

-.083

GDS/Functioning

…lose things you need?

.573

.172

-.162

.302

.538

.060

GDS/Symptoms …have a hard time sleeping because you were worried?

.623

-.425

.053

.743

.082

.117

GDS/Social

…annoy other people on purpose?

.498

.429

-.333

.076

.731

-.053

GDS/Social …think that you don't have any friends?

.594

-.375

-.111

.692

.161

-.030

Substance Abuse …drink alcohol or use other substances

0.284

0.108

0.768

.108

-.045

.818


5.105

2.181

1.188

3.715

3.182

1.577


8.474497



Youth CFF Factor Analysis (Responder= Parent or other adult, N=3747)


(Note that Factors 1 and 2 in varimax rotation are reversed from Youth completed version)

Factor 1 reflects attention and behavioral problems and accounts for more variance than Factor 2, symptoms of depression and anxiety.

Factor Method: Principle Components

Factor Method:

Varimax Rotation

Scale

Item wording Factor

1

Factor 2

Factor 3

Factor 1

Factor 2

Factor 3

GDS/Symptoms …eat a lot more or a lot less than usual?

.478

.215

.074

.214

.458

.155

GDS/Social


.652

-.483

.031

.782

.019

.220

GDS/Social …interrupt others? .671 -.464 -.212 .841 .059 -.013

GDS/Social …lie to get things you wanted? .647 -.374 .162 .681 .094 .335


.707

-.209

-.068

.679

.274

.108

GDS/Symptoms …worry about a lot of things? .531 .551 -.013 .070 .761 .048

GDS/Symptoms

…feel worthless?

.632

.431

.084

.197

.722

.176

GDS/Symptoms …have a hard time having fun? .529 .482 -.000 .108 .704 .067

GDS/Social …have a hard time waiting your turn?

.668

-.345

-.222

.768

.152

-.036

GDS/Social …hang out with kids who get into trouble?

.456

-.198

.602

.326

.090

.704


.416

.471

-.139

.060

.634

-.092

GDS/Functioning


.565

-.028

-.164

.484

.335

-.033

GDS/Symptoms …have a hard time sleeping because you were worried?

.471

.564

.042

.004

.731

.088

GDS/Social …annoy other people on purpose?

.649

-.383

-.152

.761

.107

.032


.536

.409

-.135

.188

.689

-.055

Substance Abuse …drink alcohol or use other substances

.194

-.005

.821

-.040

.068

.840


5.098

2.414

1.266

3.895

3.423

1.458


8.776643


The coefficient alpha for the internalizing factor was 0.84, and for the attention/behavior

problems 0.83. The correlation between the two factors was 0.41. The correlation between the

single substance abuse item and internalizing items was 0.15, while the correlation with

externalizing items was 0.18. All correlations were statistically significant (p< .001).

As with the Adult questionnaire, the Adolescent version is scored with a Global Distress Scale

with subscales for symptoms (anxiety/depression), interpersonal problems, and

attention/functioning items, with the externalizing symptom items divided between

interpersonal and attention/functioning problems.

Despite the evidence for up to three factors as revealed by varimax rotation, the correlation

between factors is strong such that they can be treated as a single factor measure. This is

consistent with scoring instructions for the OQ-64.

As with the Adult questionnaire, the factor analysis shows evidence of construct validity. The

Youth measure will likewise show a strong correlation with other similar youth measures such

as the Youth Outcome Questionnaire, the Child Behavior Checklist, and the Connors Rating

Scale.




Youth self-completed First assessment at session 1 or 2

Last GDS score=First GDS Score * 0.5756915262+0.4366357297 (R-square=.30) First assessment at session 3 or later

Last GDS score=First GDS Score * 0.6236339929 + 0.4950503979 (R-square=.37) First assessment session unknown

Last GDS score=First GDS Score * 0.6088506340 + 0.4065135658 (R-square=.40) Youth parent-completed First assessment at session 1 or 2

Last GDS Score = First GDS Score * 0.5397387039 + 0.2866221758 (R-square=.28) First assessment at session 3 or later

Last GDS Score = First GDS Score * 0.6358378356 + 0.2928912887 (R-square=.41) First assessment session unknown

Last GDS Score = First GDS Score * 0. 6439081553 + 0.3105141663 (R-square=.41)

See Appendices B and C for detailed normative information and predictive models.


Section 6: CFF – Child Specific Information

Reliability and Construct Validity

The Child version has 16 items, with similar content to the adolescent version. Some items are

substituted to be more age appropriate. The sample size of the Child version was 8,948. As

with the adolescent version, the factor structure was the same whether completed by the child

or a parent.

Factor analysis (principal components) likewise reveals that all items load on a common factor,

as evidenced by reliability (coefficient alpha) of 0.90 for the full scale.

Factor analysis with varimax rotation indicates the existence of three factors, one containing

symptoms of anxiety and depression (sometimes referred to as internalizing symptoms), the

second with items reflecting behavior problems, and the third with symptoms related to

attention problems. The coefficient alpha for the anxiety/depression factor was 0.75, for the

behavior problems 0.89, and for the attention problems 0.81. The correlation between

behavior problems and attention problems was 0.69, while the correlation between behavior

problems and symptoms of depression/anxiety was 0.42. The correlation between attention

and depression/anxiety was 0.40. All correlations were statistically significant (p<.001).

The questionnaire can be scored as a single Global Distress Scale. Despite the evidence for up

to three factors as revealed by varimax rotation, the correlation between factors is sufficiently

strong such that the instrument can be treated as a single factor measure.

As with the Youth questionnaire, the factor analysis is evidence of construct validity. The Child

measure will likewise show a strong correlation with other similar youth measures such as the

Youth Outcome Questionnaire, the Child Behavior Checklist, and the Connors Rating Scale.



Child CFF Factor Analysis (Responder=Youth, N=3,278)

Factor 1 in principle components is the Global Distress Scale. Note that all but one GDS items have positive factor loadings of .49 or greater. A single item, shyness, is at .25

Rotated Factor 1 reflects behavioral problems. Rotated Factor 2 reflects internalizing

symptoms of depression and anxiety. Rotated Factor 3 reflects problems with attention.

Combined Rotated Factors 1 and 3 for measure of externalizing symptoms.

Factor Method:


Factor Method:

Varimax Rotation

Scale

Item wording Factor

1

Factor 2

Factor 3

Factor 1

Factor 2

Factor 3

GDS/Symptoms …cry easily? 0.499 0.479 -0.120 0.226 0.659 0.079

GDS/Symptoms …feel unhappy or sad? 0.522 0.581 -0.150 0.218 0.763 0.049

GDS/Social …get into trouble? 0.716 -0.316 -0.214 0.764 0.057 0.267

GDS/Social …interrupt others? 0.700 -0.297 0.133 0.550 0.036 0.541

GDS/Social

…lie to get things you wanted?

0.670

-0.242

-0.081

0.626

0.090

0.338


0.742

-0.058

-0.229

0.684

0.210

0.221

GDS/Symptoms …worry about a lot of things? 0.485 0.605 0.033 0.079 0.752 0.174

GDS/Symptoms

…have a hard time sitting still?

0.627

-0.103

0.482

0.221

0.145

0.752

GDS/Symptoms …have a hard time paying attention?

0.672

-0.081

0.444

0.266

0.189

0.741


0.699

-0.223

0.223

0.468

0.093

0.601

GDS/Social … get into fights with family members and/or friends?

0.674

-0.045

-0.388

0.720

0.295

0.050


0.3450

0.576

0.093

-0.044

0.659

0.154

GDS/Functioning …lose things you need? 0.555 0.046 0.456 0.122 0.248 0.664

GDS/Social …argue with adults? 0.707 -0.213 -0.330 0.192 0.608 0.158


0.668

-0.266

-0.187

0.694

0.077

0.254


0.493

0.432

-0.037

0.192

0.608

0.158


6.162

1.888

1.152

3.850

2.732

2.620


9.202401


Child CFF Factor Analysis (Responder=Parent or other adult, N=6,114)

Factor 1 in principle components is the Global Distress Scale. Note that all but one GDS items have positive factor loadings of .49 or greater. A single item, shyness, is at .25

Factor structure is same as for Child completed, but with different order in factors. Rotated Factor 1 reflects behavioral problems. Rotated Factor 2 reflects problems with

attention. Rotated Factor 3 reflects internalizing symptoms of depression and anxiety. Combined Rotated Factors 1 and 2 for measure of externalizing symptoms.

Factor Method:


Factor Method:

Varimax Rotation

Scale

Item wording Factor

1

Factor 2

Factor 3

Factor 1

Factor 2

Factor 3

GDS/Symptoms …cry easily? 0.509 0.432 -0.019 0.283 0.140 0.588

GDS/Symptoms …feel unhappy or sad? 0.521 0.615 --0.114 0.306 0.018 0.754

GDS/Social


0.765

-0.270

-0.201

0.758

0.351

0.022

GDS/Social …interrupt others? 0.771 -0.210 0.095 0.563 0.566 0.105

GDS/Social …lie to get things you wanted? 0.680 -0.126 -0.057 0.570 0.372 0.135


0.758

-0.051

-0.289

0.754

0.214

0.214

GDS/Symptoms …worry about a lot of things? 0.765 -0.270 -0.201 0.758 0.351 0.028

GDS/Symptoms

…have a hard time sitting still?

0.652

-0.249

0.471

0.248

0.802

0.055

GDS/Symptoms …have a hard time paying attention?

0.669

-0.175

0.521

0.212

0.829

0.134


0.766

-0.208

0.160

0.518

0.614

0.110

GDS/Social … get into fights with family members and/or friends?

0.724

-0.008

-0.411

0.796

0.088

0.230


0.253

0.638

0.210

-0.101

0.114

0.701

GDS/Functioning


0.581

-0.049

0.446

0.248

0.802

0.055

GDS/Social

…argue with adults?

0.739

-0.115

-0.350

0.794

0.175

0.143


0.725

-0.143

-0.256

0.732

0.249

0.120


0.488

0.416

0.076

0.211

0.207

0.575


6.576

1.929

1.278

4.219

2.981

2.583


9.782697




Child self-completed First assessment at session 1 or 2

Last GDS Score = First GDS Score * 0.5978052359 + 0.2011899101 (R-square=.31) First assessment at session 3 or later


Last GDS Score = First GDS Score * 0.6421563466 + 0.2613480855 (R-square=.40) Youth parent-completed First assessment at session 1 or 2

Last GDS score = First GDS Score * 0.5558274873 + 0.5556823422 (R-square=.31) First assessment at session 3 or later


Last GDS Score = First GDS Score * 0.6515411875 + 0.3669654397 (R-square=.40)

See Appendices D and E for detailed normative information and predictive models.


References

Brophy CJ, Norvell NK, Kiluk DJ (1988) An Examination of the factor structure and convergent

and discriminant validity for the SCL-90R in an outpatient clinic population. Journal of

Personality Assessment 52(2) 334-340.

Brown (2013) Measurement Plus Feedback Equals Improved Outcomes: An evidence based

practice.

https://psychoutcomes.org/bin/view/DecisionSupportToolkit/ToolkitUsageAndOutcomes

Brown J, Minami T (2013) Quality of Life and Well-being Questionnaires.

https://psychoutcomes.org/bin/view/Questionnaires/QualityOfLife

Enns MW , Coxa BJ, Parker B JDA, & Guertinc JE (1998) Confirmatory factor analysis of the Beck Anxiety and Depression Inventories in patients with major depression Journal of Affective Disorders 47 195-200.

Goodman JD, McKay JR, DePhilippis D (2013) Progress Monitoring in Mental Health and

Addiction Treatment: A Means of Improving Care. Professional Psychology: Research and

Practice 44(4) 231-246.

Henry W. (1997) Harvard School of Public Health College Alcohol Study, Ann Arbor, MI: Inter-

university Consortium for Political and Social Research.

http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3163

Jacobson, NS., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59, 12-19.

Lambert MJ (2009) Yes, It Is Time for Clinicians to Routinely Monitor Treatment Outcome. In

Miller, S. & Hubble, (M. Eds.), Heart and Soul of Change (2nd ed.). American Psychological

Association Press: Washington.

Miller DD, Duncan Bl, Brown J et al. (2003) The Outcome Rating Scale: A preliminary study of

the reliability, validity , and feasibility of a brief visual analog measure. Journal of Brief Therapy

2 (2), 91-100.

https://psychoutcomes.org/bin/view/DecisionSupportToolkit/ToolkitUsageAndOutcomes

https://psychoutcomes.org/bin/view/Questionnaires/QualityOfLife

http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3163



Minami T, Brown GS, McCulloch J, Bolstrom B. (2011) Benchmarking therapists: Furthering the benchmarking method in its application to clinical practice. Quality & Quantity. 46:1699-1708.

Minami T, Wampold BE, Serlin RC, Hamilton EG, Brown GS, & Kircher JC. (2008a) Benchmarking the effectiveness of psychotherapy treatment for adult depression in a managed care environment: A preliminary study. Journal of Consulting and Clinical Psychology. 76, 116-124

Minami, T, Serlin, RC, Wampold, BE, Kircher, JC, & Brown, GS (2008b) Using clinical trials to benchmark effects produced in clinical practice, Quality and Quantity 42:513-525.

Minami, T., Wampold, B. E., Serlin, R. C., Kircher, J. C., & Brown, G. S. (2007) Benchmarks for psychotherapy efficacy in adult major depression, Journal of Consulting and Clinical Psychology, 75, 232-243.

SAMHSA (1998) National Household Survey on Drug Abuse.

http://www.samhsa.gov/data/nhsda/98mf.pdf

http://www.samhsa.gov/data/nhsda/98mf.pdf


Appendix A: Adult CFF Normative and Benchmarking Data

Adult CFF - Cases with pre-post change and with first assessment at session 1 or 2.

N

First GDS Score Last GDS Score Pre-post change # CFF per episode Pre-post weeks

Mean (sd) Mean (sd) Mean (sd) Mean (sd) Mean (sd)

All

ADHD

Adjustment disorder

Anxiety

Behavior disorder

Bipolar

Depression

Eating disorder

Not specified

Other

PTSD

Personality disorder

Psychosis

Substance abuse

23,774

400

3,173

2,179

69

828

4,728

57

10,473

516

994

10

106

241

2.14 (0.77) 1.64 (0.80) 0.50 (0.73) 5.37 (6.33) 14.02 (18.25)

2.04 (0.72) 1.59 (0.68) 0.45 (0.70) 4.66 (4.17) 13.29 (14.84)

1.91 (0.70) 1.46 (0.72) 0.44 (0.70) 5.04 (5.21) 13.00 (16.34)

2.15 (0.69) 1.68 (0.75) 0.44 (0.71) 5.68 (6.40) 15.27 (18.96) 1.72 (0.72) 1.34 (0.79) 0.38 (0.70) 4.12 (2.44) 10.33 (8.70)

2.46 (0.76) 1.97 (0.82) 0.49 (0.82) 6.82 (9.64) 20.28 (25.08)

2.44 (0.69) 1.84 (0.79) 0.60 (0.75) 6.14 (7.00) 16.76 (20.44)

2.13 (0.77) 1.64 (0.81) 0.49 (0.69) 6.12 (4.98) 16.58 (17.75)

2.05 (0.79) 1.54 (0.78) 0.51 (0.71) 4.82 (5.84) 11.92 (16.39)

2.16 (0.78) 1.74 (0.82) 0.42 (0.71) 5.54 (6.10) 14.56 (17.29)

2.38 (0.73) 1.94 (0.81) 0.44 (0.72) 6.69 (7.93) 17.74 (21.33)

2.50 (0.61) 1.83 (0.66) 0.67 (0.52) 4.30 (2.50) 8.37 (3.95) 2.24 (0.84) 1.86 (0.99) 0.37 (0.83) 6.51 (6.27) 21.57 (25.21)

1.72 (0.94) 1.35 (0.82) 0.38 (0.83) 5.73 (5.34) 14.41 (15.00)

Adult CFF - Cases with intake scores in the clinical range, multiple assessments within the episode and care, and the first assessment at session 1 or 2.

N



All

ADHD

Adjustment disorder

Anxiety

Behavior disorder

Bipolar

Depression

Eating disorder

Not specified

Other

PTSD


Psychosis

Substance abuse

19,250

320

2,327

1,839

44

751

4,310

46

8,050

423

892

10

88

150

2.41 (0.58) 1.80 (0.75) 0.61 (0.71) 5.56 (6.54) 14.53 (18.79)

2.29 (0.54) 1.72 (0.64) 0.58 (0.66) 4.81 (4.45) 13.93 (15.52)

2.22 (0.52) 1.64 (0.68) 0.58 (0.69) 5.24 (5.45) 13.47 (17.02)

2.35 (0.55) 1.79 (0.73) 0.56 (0.69) 5.73 (6.60) 15.46 (19.38) 2.11 (0.54) 1.58 (0.73) 0.52 (0.66) 3.82 (2.00) 10.80 (8.73)

2.61 (0.62) 2.05 (0.80) 0.56 (0.79) 6.72 (8.38) 20.22 (24.59)

2.56 (0.58) 1.90 (0.77) 0.66 (0.73) 6.20 (7.05) 16.79 (20.37)

2.41 (0.54) 1.85 (0.71) 0.56 (0.70) 5.06 (4.24) 14.62 (17.45) 2.36 (0.58) 1.73 (0.74) 0.63 (0.70) 5.04 (6.20) 12.47 (17.15)

2.41 (0.61) 1.88 (0.78) 0.53 (0.68) 5.77 (6.32) 14.98 (16.90)

2.54 (0.60) 2.04 (0.76) 0.50 (0.70) 6.70 (8.02) 17.74 (21.45)

2.50 (0.61) 1.83 (0.66) 0.67 (0.52) 4.30 (2.50) 8.37 (3.95) 2.49 (0.68) 2.02 (0.96) 0.46 (0.82) 6.89 (6.63) 21.93 (24.82)

2.32 (0.60) 1.65 (0.70) 0.68 (0.73) 6.03 (5.98) 14.40 (17.13)



Results of GLM prediction - adding variables incrementally

Adult Model: First Global Distress Score (GDS) predicts final GDS

This model includes all members, regardless of the timing of their assessments or change status

R-Square = 0.3399

Variable Parameter estimate SE t-Value Pr > |t|

Intercept 0.4368 0.0085 51.12 <.0001

firstgds 0.5917 0.0038 154.15 <.0001 Model: Last GDS Score = 0.5917*First GDS score + 0.4368 (intercept)

F=23761.7, p<.0001

Adult Model: First Score predicts final score, Broken out by first assessment (Early vs. Late)

Early R-square = 0.2998


Intercept 0.4366 0.01515999 28.80 <.0001 First GDS 0.5757 0.0065241 88.24 <.0001

Last GDS score= 0.5757*First GDS Score + 0.4366

Model F=7786.44, p <.0001

Later

R-square = 0.3739




Model F=2687.65, p <.0001


Appendix B: Youth Self-Report CFF Normative and Benchmarking Data

Youth Self-Report CFF: All cases with pre-post change and first assessment at session 1 or 2.

N



All

ADHD

Adjustment disorder

Anxiety

Behavior disorder

Bipolar

Depression

Eating disorder

Not specified

Other

PTSD


Psychosis

Substance abuse

5,048

260

426

310

90

33

480

4

3,205

140

64

1

5

30

1.43 (0.70) 1.09 (0.68) 0.34 (0.62) 4.35 (3.92) 13.14 (14.56) 1.56 (0.74) 1.13 (0.71) 0.44 (0.63) 4.24 (3.61) 14.23 (14.25)

1.38 (0.65) 0.98 (0.59) 0.39 (0.60) 4.22 (3.93) 12.23 (13.20)

1.33 (0.59) 0.98 (0.62) 0.36 (0.59) 4.70 (3.62) 15.07 (15.15)

1.52 (0.72) 1.06 (0.71) 0.46 (0.68) 3.72 (2.75) 9.78 (9.78) 1.61 (0.84) 1.17 (0.70) 0.43 (0.75) 4.64 (3.12) 14.91 (11.95)

1.60 (0.69) 1.17 (0.69) 0.43 (0.64) 4.52 (3.87) 13.22 (12.64)

1.10 (0.68) 0.53 (0.49) 0.57 (0.73) 5.50 (0.58) 22.61 (15.59)

1.40 (0.70) 1.10 (0.70) 0.30 (0.62) 4.33 (4.02) 13.03 (15.15) 1.46 (0.74) 1.06 (0.70) 0.40 (0.64) 4.42 (4.03) 13.92 (13.95)

1.42 (0.75) 1.08 (0.61) 0.33 (0.54) 4.92 (3.76) 15.04 (14.18)

2.75 (0.0) 2.13 (0.0) 0.62 (0.0) 5.00 (0.0) 14.00 (0.0)

1.34 (0.68) 1.25 (0.78) 0.09 (0.24) 3.20 (2.68) 5.80 (5.89) 1.00 (0.68) 0.78 (0.55) 0.22 (0.44) 3.10 (2.52) 6.60 (6.86)

Youth Self Report CFF: Intake scores in the clinical range, multiple assessments within the episode and care, and the first assessment at session 1 or 2.

N



All

ADHD

Adjustment disorder

Anxiety

Behavior disorder

Bipolar

Depression

Eating disorder Not specified

Other

PTSD


Psychosis

Substance abuse

4,107

229

354

260

79

27

424

2

2,537

119

52

1

4

19

1.65 (0.57) 1.22 (0.66) 0.42 (0.63) 4.45 (3.98) 13.46 (14.91)

1.71 (0.65) 1.22 (0.69) 0.49 (0.63) 4.37 (3.76) 14.49 (14.62) 1.57 (0.52) 1.09 (0.56) 0.48 (0.60) 4.19 (3.25) 12.12 (12.02) 1.51 (0.48) 1.08 (0.61) 0.43 (0.60) 4.76 (3.67) 15.07 (15.06)

1.66 (0.64) 1.14 (0.72) 0.52 (0.68) 3.64 (2.79) 9.35 (9.72)

1.86 (0.70) 1.29 (0.68) 0.57 (0.76) 4.48 (3.17) 13.99 (11.51) 1.76 (0.57) 1.27 (0.65) 0.48 (0.66) 4.57 (3.96) 13.50 (12.89) 1.69 (0.09) 0.66 (0.75) 1.03 (0.84) 5.00 (0 .0) 15.36 (11.62)

1.64 (0.56) 1.26 (0.67) 0.39 (0.62) 4.47 (4.17) 13.50 (15.78)

1.65 (0.64) 1.17 (0.69) 0.48 (0.65) 4.54 (4.27) 14.18 (14.70) 1.64 (0.63) 1.24 (0.56) 0.40 (0.56) 4.79 (3.80) 14.92 (14.38) 2.75 (0.0) 2.13 (0.0) 0.62 (0.0) 5.00 (0.0) 14.00 (0.0)

1.54 (0.59) 1.43 (0.77) 0.10 (0.28) 2.00 (0.0) 3.18 (0.71)

1.42 (0.48) 1.01 (0.53) 0.40 (0.42) 3.10 (3.00) 6.07 (7.29)



Parameter estimate SE t-Value Pr > |t| 0.1940 0.0411 4.72 <.0001 0.5601 0.0245 22.87 <.0001

Parameter estimate SE t-Value Pr > |t| 0.1809 0.0506 3.57 0.0004 0.5997 0.0356 16.83 <.0001


Youth Self-Report Model 1: First Global Distress Score (GDS) predicts final GDS


R-Square = 0.3569



Model: Last GDS Score = 0.5887*First GDS score + 0.2489(intercept)

Model F=2800.10, p <.0001

Youth Self-Report Model 2: First Score predicts final score, Broken out by first assessment (Early vs. Late)

Early

R-square = 0.3216

Variable

Intercept First GDS

Last GDS score= 0.5601*First GDS Score + 0.1940 Model F=523.25, p <.0001

Later

R-square=0.4011

Variable

Intercept

First GDS


Model F=283.25, p<.0001


Appendix C: Youth - Parent CFF Normative and Benchmarking Data

Youth - Parent CFF: All cases with pre-post change and first assessment at session 1 or 2.

N



All

ADHD

Adjustment disorder

Anxiety

Behavior disorder

Bipolar

Depression

Eating disorder

Not specified

Other

PTSD


Psychosis

Substance abuse

5,988

314

404

302

110

16

373

3

4,205

169

50

3

2

37

1.66 (0.76) 1.34 (0.76) 0.31 (0.66) 4.02 (3.51) 13.13 (15.01) 1.96 (0.83) 1.55 (0.81) 0.41 (0.76) 4.45 (5.06) 15.72 (18.10)

1.47 (0.69) 1.13 (0.72) 0.34 (0.63) 4.00 (3.04) 12.01 (12.03)

1.48 (0.69) 1.16 (0.68) 0.32 (0.63) 3.97 (2.81) 11.96 (11.75)

1.96 (0.81) 1.39 (0.83) 0.57 (0.77) 3.99 (3.30) 12.83 (15.08) 1.64 (0.87) 1.42 (0.97) 0.22 (0.63) 4.06 (1.81) 13.71 (9.49)

1.66 (0.72) 1.26 (0.71) 0.40 (0.64) 4.43 (3.83) 12.50 (13.04)

1.56 (0.45) 1.48 (0.57) 0.08 (0.13) 4.67 (3.78) 10.05 (7.47)

1.66 (0.76) 1.37 (0.76) 0.29 (0.66) 3.92 (3.35) 13.09 (15.24) 1.75 (0.82) 1.41 (0.82) 0.33 (0.72) 4.84 (5.19) 16.17 (17.79)

1.71 (0.73) 1.40 (0.72) 0.31 (0.71) 4.22 (3.07) 12.82 (18.75)

1.79 (0.38) 1.42 (0.79) 0.38 (0.65) 9.33 (8.08) 18.05 (13.79)

2.34 (0.22) 1.38 (0.18) 0.97 (0.04) 9.50 (9.19) 37.21 (23.74) 1.06 (0.65) 0.87 (0.65) 0.18 (0.63) 3.76 (1.82) 9.73 (7.92)

Youth Parent Report CFF: Intake scores in the clinical range, multiple assessments within the episode and care, and the first assessment at session 1 or 2.

N



All

ADHD

Adjustment disorder

Anxiety

Behavior disorder

Bipolar

Depression


Other

PTSD


Psychosis

Substance abuse

4,346

245

258

201

88

12

269

2

3,091

121

39

3

2

15

2.00 (0.57) 1.55 (0.73) 0.45 (0.66) 4.16 (3.73) 13.71 (15.78)

2.29 (0.60) 1.74 (0.76) 0.54 (0.74) 4.71 (5.48) 16.90 (19.62) 1.88 (0.49) 1.38 (0.69) 0.50 (0.64) 4.03 (3.04) 12.16 (12.32)

1.87 (0.47) 1.37 (0.65) 0.49 (0.62) 4.08 (3.12) 12.22 (12.59)

2.25 (0.62) 1.59 (0.78) 0.66 (0.80) 4.09 (3.49) 13.62 (16.17)

1.98 (0.73) 1.64 (0.98) 0.33 (0.67) 4.08 (1.73) 15.36 (10.44) 2.00 (0.52) 1.44 (0.69) 0.55 (0.65) 4.68 (4.08) 13.14 (13.84)

1.75 (0.44) 1.69 (0.62) 0.06 (0.18) 5.50 (4.95) 9.57 (10.50)

1.99 (0.57) 1.57 (0.72) 0.42 (0.65) 4.03 (3.53) 13.60 (15.88)

2.14 (0.60) 1.68 (0.76) 0.46 (0.74) 5.21 (5.35) 17.71 (17.65) 1.98 (0.53) 1.54 (0.72) 0.44 (0.74) 4.33 (3.33) 13.44 (20.94)

1.79 (0.38) 1.42 (0.79) 0.38 (0.65) 9.33 (8.08) 18.05 (13.79)

2.34 (0.22) 1.38 (0.18) 0.97 (0.04) 9.50 (9.19) 37.21 (23.74)

1.69 (0.39) 1.17 (0.66) 0.52 (0.64) 3.67 (1.91) 7.66 (7.70)




Youth Parent-Report Model 1: First Global Distress Score (GDS) predicts final GDS


R-Square = 0.3864

Variable Parameter estimate SE t-Value Pr > |t| Intercept 0.3079 0.0186 16.58 <.0001 First GDS 0.6250 0.0102 61.35 <.0001 Model: Last GDS Score = 0.3079*First GDS score + 0.6250 (intercept) Model F=3763.84, p <.0001

Youth Parent-Report Model 2: First Score predicts final score, Broken out by first assessment (Early vs. Late)


Variable

Intercept

Parameter estimate 0.2866

SE 0.0526

t-Value 5.45

Pr > |t| <.0001

First GDS 0.5397 0.0279 19.33 <.0001 Last GDS score= 0.5397*First GDS Score + 0.2866 Model F=373.48, p<.0001

Later

R-square=0.4148 Variable Intercept


SE 0.0624

t-Value 4.69

Pr > |t| <.0001

First GDS 0.6358 0.0410 15.50 <.0001 Last GDS score= 0.6358*First GDS Score + 0.2929 Model F=240.27, p <.0001


Appendix D: Child - Self CFF Normative and Benchmarking Data

Child - Self CFF, All cases with pre-post change and first assessment at session 1 or 2.

N



All

ADHD

Adjustment disorder

Anxiety

Behavior disorder

Bipolar

Depression

Eating disorder

Not specified

Other

PTSD

Substance abuse

2,526

135

184

129

59

3

88

2

1,831

68

24

3

1.53 (0.74) 1.22 (0.76) 0.31 (0.66) 4.07 (3.58) 13.14 (14.15)

1.93 (0.76) 1.54 (0.83) 0.39 (0.70) 4.19 (3.62) 16.46 (20.10)

1.40 (0.69) 1.14 (0.68) 0.26 (0.60) 3.37 (1.82) \10.01 (8.49)

1.44 (0.60) 1.07 (0.67) 0.36 (0.61) 4.35(2.90) 15.06 (14.08) 1.71 (0.75) 1.35 ( 0.77) 0.35 (0.68) 3.42 (1.80) 10.78 (9.91)

1.58 (0.84) 1.98 (1.61) -0.40 (1.06) 2.00 (0.00) 8.05 (5.00)

1.94 (0.72) 1.34 (0.86) 0.59 (0.81) 3.91 (2.53) 12.21 (10.60)

0.94 (0.26) 0.50 (0.53) 0.44 (0.80) 8.00 (5.66) 27.36 (13.23)

1.48 (0.73) 1.20 (0.74) 0.29 (0.65) 4.15 (3.84) 13.19 (14.35)

1.81 (0.78) 1.46 (0.91) 0.35 (0.66) 4.06 (3.23) 14.19 (12.69)

1.76 (0.62) 1.42 (0.71) 0.34 (0.75) 3.50 (2.50) 8.42 (8.54)

1.67 (0.34) 0.83 (0.50) 0.83 (0.73) 6.33 (4.04) 27.33 (31.51)

Child Self Report CFF: Intake scores in the clinical range, multiple assessments within the episode and care, and the first assessment at session 1 or 2.

N



All

ADHD

Adjustment disorder

Anxiety

Behavior disorder

Bipolar

Depression

Eating disorder

Not specified

Other

PTSD

Substance abuse

2,077

128

150

115

56

3

84

2

1,448

64

24

3

1.75 (0.61) 1.36 (0.73) 0.39 (0.66) 4.19 (3.65) 13.63 (14.68)

2.02 (0.69) 1.61 (0.79) 0.40 (0.72) 4.28 (3.70) 16.94 (20.48)

1.61 (0.57) 1.26 (0.64) 0.35 (0.57) 3.53 (1.92) 10.56 (8.71)

1.55 (0.52) 1.13 (0.62) 0.42 (0.55) 4.42 (3.01) 15.00 (14.14)

1.78 (0.69) 1.40 (0.76) 0.39 (0.68) 3.39 (1.84) 11.10 (10.06)

1.58 (0.84) 1.98 (1.61) -0.40 (1.06) 2.00 (0.00) 8.05 (5.00)

2.01 (0.64) 1.40 (0.84) 0.61 (0.82) 3.96 (2.58) 12.46 (10.74) 0.94 (0.26) 0.50 (0.53) 0.44 (0.80) 8.00 (5.66) 27.36 (13.23)

1.74 (0.59) 1.36 (0.72) 0.38 (0.66) 4.28 (3.95) 13.73 (14.99)

1.89 (0.72) 1.52 (0.90) 0.37 (0.67) 4.06 (3.25) 14.27 (12.90)

1.76 (0.62) 1.42 (0.71) 0.34 (0.75) 3.50 (2.50) 8.42 (8.54) 1.67 (0.34) 0.83 (0.50) 0.83 (0.73) 6.33 (4.04) 27.33(31.51)




Child Self-Report Model 1: First Global Distress Score (GDS) predicts final GDS


R-Square = 0.3716

Variable Parameter estimate SE t-Value Pr > |t| Intercept 0.2641 0.0275 9.59 <.0001 First GDS 0.6260 0.0162 38.63 <.0001 Model: Last GDS Score = 0.6260*First GDS score + 0.2641 (intercept) Model F=1492.28 , p<.0001

Child Self-Report Model 2: First Score predicts final score, Broken out by first assessment (Early vs. Late)

Early

R-square = 0.3108

Variable

Intercept


SE 0.0869

t-Value 2.32

Pr > |t| 0.0211

First GDS 0.5978 0.0465 12.85 <.0001

Last GDS score= 0.5978*First GDS Score + 0.2012 Model F=165.05 , p<.0001

Later

R-square=0.3338 Variable Intercept


SE 0.1098

t-Value 2.90

Pr > |t| 0.0043

First GDS 0.5801 0.0676 8.58 <.0001 Last GDS score= 0.5801*First GDS Score + 0.3190 Model F=73.67,p<.0001


Appendix E: Child - Parent CFF Normative and Benchmarking Data

Child Parent Report CFF: All cases with pre-post change and first assessment at session 1 or 2.

N



All

ADHD

Adjustment disorder

Anxiety

Behavior disorder

Bipolar

Depression

Eating disorder

Not specified

Other

PTSD


Substance abuse

4980

313

457

198

145

4

100

1

3570

147

39

1

5

1.88 (0.77) 1.60 (0.78) 0.28 (0.67) 4.12 (3.62) 13.66 (15.08)

2.36 (0.68) 2.01 (0.67) 0.34 (0.75) 4.45 (3.90) 16.57 (18.48)

1.80 (0.70) 1.59 (0.72) 0.21 (0.68) 4.73 (4.46) 14.51 (14.32)

1.82 (0.68) 1.44 (0.68) 0.39 (0.62) 4.29 (3.66) 13.57 (14.65) 2.31 (0.72) 1.86 (0.73) 0.45 (0.63) 4.51 (3.72) 13.98 (14.08)

2.00 (0.98) 1.91 (1.04) 0.09 (0.41) 3.75 (2.36) 12.14 (7.98)

2.06 (0.73) 1.73 (0.71) 0.33 (0.67) 3.87 (2.78) 12.14 (10.75)

2.06 (0.0) 0.81 (0.00) 1.25 (0.00) 6.00 (0.00) 18.00 (0.00)

1.81 (0.77) 1.54 (0.79) 0.27 (0.66) 3.94 (3.36) 13.12 (14.92)

2.05 (0.75) 1.80 (0.71) 0.25 (0.76) 5.47 (5.51) 18.73 (16.90)

2.11 (0.77) 1.79 (0.82) 0.32 (0.62) 4.92 (3.77) 13.64 (11.89)

1.88 (0.0) 2.25 (0.00) -0.38 (0.00) 8.00 (0.00) 22.00 (0.00) 1.46 (0.78) 1.51 (0.90) -0.05 (1.28) 3.40 (1.67) 13.40 (7.36)

Child-Parent Report CFF: Intake scores in the clinical range, multiple assessments within the episode and care, and the first assessment at session 1 or 2.

N

First GDS Score Last GDS Score Pre-post change # CFF per episode Pre-post weeks Mean (sd) Mean (sd) Mean (sd) Mean (sd) Mean (sd)

All

ADHD

Adjustment disorder

Anxiety

Behavior disorder

Bipolar

Depression


Other

PTSD


Substance abuse

4,048

298

361

160

132

3

86

1

2,845

125

33

1

3

2.13 (0.60) 1.76 (0.72) 0.38 (0.65) 4.25 (3.77) 14.13(15.61)

2.43 (0.61) 2.04 (0.67) 0.39 (0.73) 4.52 (3.98) 16.88 (18.79)

2.06 (0.53) 1.74 (0.65) 0.32 (0.64) 4.93 (4.82) 15.04 (14.95)

2.05 (0.53) 1.60 (0.61) 0.45 (0.62) 4.46 (3.89) 13.91 (15.32)

2.46 (0.57) 1.96 (0.67) 0.50 (0.62) 4.58 (3.83) 14.22 (14.43)

2.33 (0.89) 2.27 (0.92) 0.06 (0.50) 2.67 (1.15) 9.43 (7.16) 2.25 (0.60) 1.83 (0.69) 0.42 (0.65) 4.09 (2.92) 12.96 (11.22)

2.06 (0.00) 0.81 (0.00) 1.25 (0.00) 6.00 (0.00) 18.00 (0.00)

2.09 (0.59) 1.72 (0.73) 0.38 (0.64) 4.05 (3.50) 13.57 (15.45)

2.26 (0.59) 1.90 (0.65) 0.36 (0.70) 5.43 (5.23) 18.79 (16.74) 2.35 (0.57) 1.95 (0.78) 0.39 (0.65) 5.18 (4.04) 13.91 (12.55)

1.88 (0.00) 2.25 (0.00) -0.38 (0.00) 8.00 (0.00) 22.00 (0.00)

1.98 (0.44) 1.56 (0.22) 0.42 (0.64) 3.00 (1.00) 17.43 (2.97)




Child Parent-Report Model 1: First Global Distress Score (GDS) predicts final GDS


R-Square = 0.3867

Variable Parameter estimate SE t-Value Pr > |t| Intercept 0.4136 0.0228 18.14 <.0001 First GDS 0.6296 0.0112 56.02 <.0001 Model: Last GDS Score = 0.6296*First GDS score + 0.4136 (intercept) Model F=3138.58 , p<.0001

Child Parent-Report Model 2: First Score predicts final score, Broken out by first assessment (Early vs. Late)


Variable

Intercept


SE 0.0595

t-Value 9.34

Pr > |t| <.0001

First GDS 0.5558 0.0271 20.50 <.0001

Last GDS score= 0.5558*First GDS Score + 0.5557 Model F=420.30, p <.0001

Later

R-square=0.3568 Variable

Intercept


SE 0.0888

t-Value 6.94

Pr > |t| <.0001

First GDS 0.5810 0.0455 12.77 <.0001 Last GDS score= 0.5810*First GDS Score + 0.6167 Model F=163.08, p<.0001

Client Feedback Form Manual - Beacon Health Options...Client Feedback Form Manual Prepared for...

Documents

Transcript of Client Feedback Form Manual - Beacon Health Options...Client Feedback Form Manual Prepared for...