Ch 9 Internal and External Validity

Validity

The quality of the instruments used in the research study Will the reader believe what they are readying It’s not be the instrument that needs to be validated but how the

instrument is to be used Questions

--- Is the data accurate?

--- Is the data meaningful?

--- Is the data creditable?

--- Is the data useful?

--- Is the data appropriate?

External validity

Can the researcher use what have been observed in the research to make global generalizations beyond the research itself?

Does the instrument measure what it is suppose to? Is the research sample representative of the population? Using evidence to support external validity Does the study have sufficient controls to insure that the

conclusions or inferences are truly warranted?

1. Face validity: looks like it measures a particular characteristic, concepts, or phenomena

External validity (Continued)

2. Content validity: measures the expected outcomes. Are the results similar to what others familiar with the study’s contents might contend?

3. Criterion validity: the relationship of scores on an instrument to other similar instruments

4. Construct validity: measures characteristics, concepts, theories, phenomenon that cannot be directly observed but inferred, such as to what is inferred in the review of literature

Threats to Internal Validity

The extent to which the research design, instrument, and the data collected allow the researcher to draw accurate conclusions about the relationship within the data.

The research has not been affected by any interference with the research environment that would give the results of the research the appearance of something else other than what the research originally intended.

Threats to internal validity

1. Subject characteristics:

too much or too little representation of a characteristic: gender, age, race, beliefs, attitudes, intelligence, etc.

Threats to Internal Validity (Continued 1)

2. Mortality:

Participants dropping out of the study

3. Location:

Comfort, distance, resources, need consistency

4. Instrumentation:

Consistent use of the instrument

a. Instrument decay: nature of the instrument causes change in interpretation

b. Data collector characteristics: may interpret things differently, need consistency

Threats to Internal Validity (Continued 2)

c. Data collector bias: have the possibility of distorting information, need standards or ignorance

5. History:Unexpected evens occur, 9/11, 921earthquake, death in a family, accidents, storms, etc.

6. Maturation:Too much time taken for the completion of the instrument,

the passage of time7. Regression:

Increased performance may occur due to practice or intervention, such as special ed assistance

Threats to Internal Validity (Continued 3)

8. Attitude:

Special treatment of participants, special favors, control vs. experimental group

9. Implementation:

Consistency among participants

10. Testing:

Difference between pre-test & post-test, practice

11. Sources of bias:

The assessment, the administration of the assessment, the interpretation of the assessment

Threats to Internal Validity (Continued 4)

a. Assessment bias: Offensive questions

Inaccurate and unfair penalties related to scores

b. Bias in administration: Examiner

Students

Setting

C. Bias in interpretation: Criterion referenced

Norm referenced

Ways to overcome treats to internal validity

Prepare alternative strategies for the unexpected Be consistent in the treatment and relationship with all

participants Use an appropriate research design that will assist the researcher

to gather, collect, and analyze the data in the most efficient manner

Examining items for bias: Panel review

Empirical analysis

What is the preferred method to examine assessment bias?

Triangulation

Compare three forms of information for similar results

1. Observation

2. Interviews

3. Data gathering

4. Focus groups

Is the test appropriate to the population?

What is the composition of the test-taking population? To what extent can the assessment be administered without

encumbrance to all members of the population? Is there a translated version, adapted version or accommodated

version of the test? Are there recommendations for alternative testing procedures? Has the planned accommodation been assessed in terms of it’s

impact on validity and reliability of test scores?

Conclusion

The more valid and reliable research instruments are the more likely one will draw the appropriate conclusions from the collected data and solve the research problem in a creditable fashion.