Review of Chapter 4 in Dollaghen (2007)

We have discussed validity related to tests and other kinds of materials throughout your program.

For our purposes, validity is related but means a bit more. Validity as a concept that differs between

internal and external evidence. From both clinical situations and published

research studies, there is an assumption of empirically controlled measures being taken.

Measures of patient preferences are still valid.

Evidence from scientific research Internal validity

extent to which empirical evidence provides a true or accurate reflection of the patients, procedures, and settings that were observed.

External validity- Generalizability. How much do the measures taken reflect how

the patient(s) behave in other settings? How much does the findings of a study reflect

the population in general?

Confounding/nuisance variables Many! May just have to acknowledge these in

the results if you cannot control for them. Subjective bias

Impossible to completely limit this. Blinding, masking, or concealment When evaluating a study, it is important to

consider whether opinions, expectations, or beliefs of participants, or observers could have influenced the findings.

The concept of whether the researchers or participants are blind to the treatment. Can be difficult in our types of treatments, but

perhaps the researcher could be blind to characteristics about the participants who are receiving treatments.

This should be assessed and thought about when looking at treatment studies.

Does the measure(s) in the study provide a valid reflection of performance? Are there problems with norm referenced

tests? What are the issues with other types of

measures in the study? In others’ research, is what they are using to

measure outcome of effectiveness appropriate?

There are gold standard designs, such as the randomized controlled trials (RCT)

Randomizing participants into treatment groups are stronger designs than nonrandomized studies.

It is important to remember that no one study can tell us all we need to know about a treatment.

The RCT is best for drawing causal inferences about average treatment effects in groups of patients.

Maybe not the best for looking at the performance of individuals, less typical types of settings and clients, and not appropriate if looking at etiology or risk factors.

Is it appropriate to randomly assign participants to different conditions or treatments?

The studies purpose, the nature of the investigation, and the historical background should determine the type of research design that is most appropriate.

Experimental/Controlled vs. Non-Experimental/Uncontrolled Experimental studies are usually controlled

usually require a manipulation or some sort of control or comparison group.

Experimental designs can be large groups, small groups, or even single subjects.

Non-experimental studies generally are not as controlled, but do have a systematic means of gathering data and reporting results.

All things being equal, the more experimental and controlled a study, the better the evidence.

Prospective vs. retrospective Prospective- the investigator plans the study,

states a hypothesis, identifies the kinds of participants and procedures, and then gathers data. Generally, experimental studies are prospective and nonexperimental can also be prospective.

Retrospective- Looking at data after a certain time frame has passed.

Retrospective are actually ranked lower than prospective studies.

Nuisance variables: All studies should be analyzed to look for factors aside from what is being studied that could influence the results.

Statistical significance- Very important, but statistics can often be deceiving.

Subjective bias Become very familiar with the client- observer drift. Conflict of interest between the role of clinician and

being a neutral observer. Can do a check by having a colleague (blinded

observer) view a sample of the participant, or gather data from multiple sources.

Quality of measurement Same concerns as with research. Do the measurements make sense? Are they valid? Are there multiple measurements? What/who should be measured?

Research designs Single Subject Designs Multiple Baseline Designs

Nuisance variables We have discussed this a lot, and these

variables need to be observed, described and controlled to some extent in our treatment.

Can be one in the same, but certainly we need to be careful.

From the book: If a patient is being studied in the course of clinical

practice solely for the purpose of benefiting him or her, and if the investigation does not impose added burdens on the patient, and if the investigator does not impose added burdens on the patient, and if the investigator does not intend to disseminate the information (present or publish), then it is not research.

It does not mean that clinical endeavors are not research based, in that there is a search for supporting evidence, a design for gathering data, and some sort of sense of internal and external validity.

For research, we ask how applicable the results of a study apply to the population in general or a specific clinical case.

For our therapy, we ask how much the behaviors being addressed in therapy can be applicable to extra-clinical settings.

Also must consider to what extent the findings can be replicated across studies, and also how the client can replicate the behaviors in other settings.