Comprehensive Exam Review

Click the LEFT mouse key ONCE to continue

Research andProgram

EvaluationPart 2

Click the LEFT mouse key ONCE to continue

Threats to Internal and External Experimental Design Validity

The “threats” to experimental design validity are potential sources of invalidity in the results of an experimental study.

Internal threats to design validity have to do with procedures or methodology.

External threats to design validity have to do with generalizability of the results.

Internal Threats

Contemporary History is invalidation resulting from events that occur during the course of the experiment (e.g., exposure to media events, life situation changes, or random personal activities).

Maturation is invalidation resulting from changes in the subjects during the course (i.e., time period) of the experiment (e.g., develop-mental change, fatigue, or hunger).

Testing (sometimes called pre-test sensitivity) is invalidation resulting from changes in subjects’ scores on a subsequent test due to having taken the test on a previous occasion (i.e., changes due to test content familiarity or recall).

Statistical Regression is invalidation resulting from the “natural” regression to the mean among persons initially at an extreme (e.g., inclination to homeostasis).

Selection is invalidation resulting from initial differences between subjects in the various groups (e.g., selection bias).

Attrition (sometimes called Experimental Mortality) is invalidation resulting from subjects dropping out of participation in a study (e.g., differential attrition across groups).

Interactions Among Factors is invalidation resulting from either confounding or compounding combinations of the preceding seven threats.

Instrumentation is invalidation resulting from changes in the measurement instruments or procedures (e.g., calibration, accuracy, interpretation, or experience).

Compensatory Equalization of Treatments is invalidation resulting from “compensation” (e.g., special treatment) being provided to a control group presumed not to be receiving the treatment.

Resentful Demoralizing of Subjects in Less Desirable Treatment Conditions is invalidation resulting from subjects in one (e.g., alternative treatment or control) group “giving up” on performing appropriately.

Compensatory Rivalry by Subjects Receiving Less Desirable Treatments is invalidation resulting from members of the control group attempting to “outperform” the experimental group (e.g., to demonstrate equivalent competence).

Diffusion or Imitation of Treatments is invalidation resulting from intra- or inter-group communication among subjects (e.g., “helping” one another during the course of an experiment).

External Threats

Reactive Effects of Testing is a generalizability limitation resulting from the difference between the experimental and real life situations (e.g., pre-test sensitization is not present in real life situations).

Interaction of Selection and Treatment is a generalizability limitation resulting from experimental subjects being particularly susceptible to the treatment (e.g., volunteers may be particularly motivated to change).

Reactive Effects of Experimental Arrange-ments is a generalizability limitation resulting from lack of comparability between the experimental (e.g., laboratory) and real world setting.

Multiple-Treatment Interference is a gen-eralizability limitation resulting from either the sequential or concurrent application of various treatments.

Single-Subject Designs

Single-subject experimental designs are sometimes known as N = 1 or as intra-subject designs.

Single-subject experimental designs are usually used to assess effectiveness in changing a specific behavior of an individual.

Common features of single-subject designs include:

Clear specification of the treatment goal, usually presented as a behavioral goal (i.e., change in a targeted behavior).

A focus on changing only one variable at a time during the treatment phase(s).

Repeated measurement of the target variable throughout the experiment.

Consistency and stability of baseline and treatment conditions.

Diligent description of all the conditions under which the target behavior is observed (i.e., measured).

Common notations for single-subject designs:

A designates the baseline condition.

B designates the treatment condition

The A-B design is the simplest, but least interpretable, of the single-subject designs.

The premise of the A-B design is that after the baseline is established, the condition (i.e., behavior) would have continued in the same pattern had the treatment not been introduced.

A B

Baseline Treatment

X X X X X

O O O O O O O O O O

Time

The A-B design

The A-B-A design is sometimes known as the “reversal” or “withdrawal” design.

The A-B-A design allows a strong causal inference if the target behavior returns to the baseline level after the treatment is terminated.

A B A

Baseline Treatment Baseline

X X X X X

O O O O O O O O O O O O O O O

Time

The A-B-A design

The A-B-A-B design affords the strongest causal inference about the effect of the treatment.

The A-B-A-B design has the advantage of ending the experiment with the treatment in effect (which may be desirable for ethical reasons).

A B A B

Baseline Treatment Baseline Treatment

X X X X X X X X X X

O O O O O O O O O O O O O O O O O O O O

Time

The A-B-A-B design

The Randomized A-B design involves a randomized presentation of the A and B (i.e., baseline and treatment) conditions over time.

Due to the random presentation of conditions, parametric statistical analyses can be applied to the resultant data.

The Randomized A-B design also can be used to analyze “carryover” effects of the treatment.

Multiple-Baseline designs employ the A-B design logic, but allow collection of data on two or more actions, subjects, situations, or combinations thereof.

In Multiple-Baselines Across Behavior designs, baselines are established for two or more independent behaviors for one subject.

The (same) treatment is first applied to one behavior, then to the next, and so on in sequence using the same time interval for each application of the treatment.

In Multiple-Baselines Across Situations designs, the same behavior is targeted across situations (e.g., settings).

The A-B design is replicated (i.e., same inter-vention and time periods) across situations and then the results are compared.

In Multiple-Baselines Across Individuals design, the behavior and the situation are held constant.

The establishment of a baseline and sub-sequent application of the treatment pro-ceed sequentially across individuals.

Computers and Research

Computers serve three major functions in regard to research: data management, document production, and internet transversal.

The data management functions include uses for data gathering, storage, and manipulation.

The data gathering functions may include use of optical scanners and computer-based assessments.

The data storage functions include data-entry and retrieval as well as storage.

The data storage functions are facilitated through use of spreadsheet and data-base programs.

Spreadsheet programs are designed to make data entry and understanding as easy as possible by using a row-column format.

Database programs are designed to make data-entry and reporting functions as easy as possible through “free format” procedures.

The data manipulation functions include data analyses and other procedures to clarify the nature of the data.

The document production functions primarily include using word processors to develop manuscript and/or electronic forms of communications of research.

The most important word processing functions include formatting, spell-checking, and grammar-checking.

A current trend in dissemination of research documents is “publication” on the internet.

The internet serves four primary functions in regard to research: aiding collaboration among researchers, simplifying information gathering, facilitating data collection, and fostering professional discourse.

A major, current trend is increased post-publication electronic communication about research results and implications.

Legal Aspects of Research

General legal principles for research include that:

Subjects have the right to freedom from invasion of privacy.

Subjects’ participation in research is always voluntary.

Any type of research is permitted if appropriate procedures are followed.

Researchers are responsible for whatever happens to subjects during or as a result of participation in the research.

The Family Educational Rights and Privacy Act (FERPA) of 1974, also known as the Buckley Amendment, stipulates that:

written parental permission is required for access to students’ records.

person-data linkages should be avoided (i.e., data should remain anonymous) whenever possible.

A written request for data about a student must include:

exact specification of the data requested,explanation of how the data are to be used, andexplanation of to whom the data will be

disclosed and how it will be disseminated.

The data obtained can be used only for the purpose for which the request was made!

The National Research Act (NRA) of 1974 applies to all persons associated with any institution and/or agency that receives any federal funds.

This act established the requirements for “informed consent” procedures in research.

Institutional Review Boards (IRBs), also known as “human subjects committees,” were established to ensure that the provisions of the NRA are implemented effectively and legally.

There is no single informed consent form. An informed consent form is created by the researcher(s) for each research project.

The informed consent form must communicate clearly:

that participation in a research activity is being requested.

the duration (and/or frequency of occurrence) of the participation requested.

an explanation of the nature of the par-ticipation requested in language that the person receiving the letter can be reason-ably expected to understand.

a (brief) description of the research procedures.

specific identification of any procedures that are experimental.

a description of any potential risks and / or discomfort resulting from participation.

a description of any benefits of participation, specifically including whether there will or will not be monetary compensation.

a statement to the extent of confidentiality and/or anonymity to be accorded to participants.

sources of assistance should harm or discomfort occur during or as a result of participation.

the name and contact information to which to direct questions or inquiries about the research.

a statement that participation is completely voluntary.

a statement that discontinuing participation at any time will NOT result in any form of harm, recourse, or loss of benefits to the participant.

Each subject must be given a copy of the in-formed consent form (separate from the one retained by the researcher).

This concludes Part 2 of the presentation on

RESEARCH AND PROGRAM

DEVELOPMENT