PhD Research Seminar Series: Valid Research Designs Dr. K. A. Korb University of Jos.

PhD Research Seminar Series:Valid Research Designs

Dr. K. A. Korb

University of Jos

Outline

Threats to valid Research Design Construct Validity Internal Validity External Validity Reliability

Dr. K. A. KorbUniversity of Jos

Reliability: Consistency of results

UnreliableReliable


Validity: Measuring what is supposed to be measured

InvalidValid


Research Example

The effect of regular leisure reading on reading achievement in primary school.

Students in a primary school will be randomly assigned to either a treatment or a control group. The treatment group will spend 10 minutes everyday

reading a book with an adult. In an attempt to keep the treatment and control groups

as similar as possible, the control group will also spend 10 minutes with an adult everyday, but will instead do math problems.


Research Example

ReadingAchievement

ReadingAchievement

Control Group

Experimental Group

Read BookEvery Daywith Adult

IV: RegularLeisureReading

DV:Reading

Achievement

ReadingAchievement

ReadingAchievement

Do MathsEvery Daywith AdultDr. K. A. Korb

University of Jos

Construct Validity

Construct Validity: Establishing valid operational measures for the concepts being studied This relates to:

Independent Variable Dependent Variable

Review: Construct Definition: General explanation of the

construct by relating it to other constructs Operational Definition: Statement of specifically

how the construct will be measured or implemented in your study


Review:Construct vs. Operational Definitions

AcademicAcademicAchievementAchievement

State ExamsState Exams

NECO ScoresNECO Scores

WAEC ScoresWAEC Scores

Final Course ExamFinal Course Exam

Self-Report of Parents’Self-Report of Parents’Yearly IncomeYearly Income

Free/ReducedFree/ReducedLunch EligibilityLunch Eligibility

EconomicEconomicStatusStatus

ConstructConstruct Operational DefinitionOperational Definition


Research ExampleConstruct validity focuses on:

ReadingAchievement

ReadingAchievement

Control Group

Experimental Group

Read a BookEvery Day

IV: RegularReading

DV:Reading

Achievement

ReadingAchievement

ReadingAchievement


Construct Validity

When you develop your research project, you start with a theory. For example, regular leisure reading will lead to increased reading

achievement scores. Then you operationalize your theory into your research study.

You translate “regular leisure reading” into how it will be implemented in your study – reading a book with an adult everyday for 10 minutes.

You translate “reading achievement scores” into how it will be measured in your study – classroom exam scores.

When you finish collecting data, you hope to be able to move back to theory. You want to be able to say that not only did reading a book for 10

minutes everyday with an adult lead to higher classroom exam scores on reading achievement, but also that regular leisure reading will lead to higher reading achievement scores in general.

However, in order to do that, you must have construct valid measures of your independent and dependent variables.


Research Example

IV: RegularReading

DV:Reading

Achievement

Read bookevery day with

an adult for10 min.

Scores onreading sectionof classroom

exams

Theory:

Research Study:

IV: RegularReading

DV:Reading

AchievementTheory:


Construct Validity

Enables you to generalize from the treatment and measures in your study to the general constructs of your treatment and measures

Threats to Construct Validity Inadequate explanation of constructs in your paper Flawed matching of operationalization to constructs

The measures do not align with the constructs they were designed to measure.


Internal Validity

Internal validity: Extent to which variables other than the treatment provide plausible explanations to the experimental results Only relevant for experimental designs

Primary consideration for interventions Extraneous variable: Any variable other than treatment variable

that if not controlled can affect the experimental outcome Hold constant or eliminate all extraneous variables that might

affect the posttest Goal of Research Design: Create set of conditions so any

observed changes in your dependent variable can be attributed to experimental treatment instead of extraneous variables

Do variables other than the treatment provide plausible explanations to the experimental results?


Research Example

ReadingAchievement

ReadingAchievement

Control Group


IV: RegularReading

DV:Reading

Achievement

ReadingAchievement

ReadingAchievement

Experimental Group

Adult Attention

ReadingAbility

Interest

Maturation


Threats to Internal Validity

History: If treatment lasts over time, other events may influence dependent variable

To prevent: Keep the control group equivalent in all aspects but treatment Maturation: Physical or psychological changes may influence the dependent

variable To prevent: Use a control group

Testing: Giving a pretest may increase performance on the post-test To prevent: Don’t use a pre-test OR Use different pre- and post-tests

Instrumentation: Nature of measuring instrument has changed between pre- and post-test

To prevent: Do not use a pre-test OR Use the same for pre- and post-test Selection Bias: Treatment and control groups are different on an important

extraneous variable To prevent: Random assignment OR Use a pre-test

NOTE: Many of these preventions contradict. You must think about which threats are most relevant to your study and then develop a strategy for overcoming the most important threats.


External Validity

External Validity: Establishing the group of people to which the research findings can be generalized


Research Example

ReadingAchievement

ReadingAchievement

Control Group


ReadingAchievement

ReadingAchievement

Experimental Group

PopulationPopulation

SampleSampleDr. K. A. KorbUniversity of Jos

Types of External Validity

Population Validity: Generalizing the results from experimental sample to a defined population To establish: Give a thorough explanation of the constitution

of your sample and how your sample was selected. Ecological Validity: Generalizing the results of a study

from the artificial conditions created by researcher to real-life conditions To establish:

Give an adequate description of treatment Give an adequate description of the measure for the dependent

variable Threats to Ecological Validity:

Novelty: The treatment was different than typical treatment Experimenter Effect: The particular person delivering the treatment

affects results


Reliability

Reliability: Demonstrating that the operations of the study can be repeated with the same results Conduct research as if someone were always

looking over your shoulder Report the procedure of your study in explicit

detail


Other Issues in Valid Research Design

Experimenter Bias: Researcher’s expectations about the outcome of experiment influence participants’ response To prevent: Use research assistants who are blind to the study (aka do

not know the purpose of the study) Treatment Fidelity: Extent to which treatment conditions are

implemented according to the researcher’s specifications To support:

Carefully train research assistants Periodically check up on research assistants unexpectedly to determine if

they are following guidelines Strong Experimental Treatments: Developing a treatment that will

make a robust effect on the dependent variable To support:

Develop a thorough understand of your dependent variable and how it can be influenced.

Spend much thoughtful time developing your treatment.


PhD Research Seminar Series: Valid Research Designs Dr. K. A. Korb University of Jos.

Documents

Transcript of PhD Research Seminar Series: Valid Research Designs Dr. K. A. Korb University of Jos.