Statistical Analysis of the Two Group Post-Only Randomized Experiment.

Statistical Analysis of the Two Group Statistical Analysis of the Two Group Post-Only Randomized ExperimentPost-Only Randomized Experiment

Analysis RequirementsAnalysis Requirements

Two groupsTwo groups A post-only measureA post-only measure Two distributions, each with an average and variationTwo distributions, each with an average and variation Want to assess treatment effectWant to assess treatment effect Treatment effect = statistical (i.e., nonchance) Treatment effect = statistical (i.e., nonchance)

difference between the groupsdifference between the groups

R X OR O

Statistical AnalysisStatistical Analysis

Controlgroupmean

Treatmentgroupmean

Controlgroupmean

Treatmentgroupmean

Is there a difference?

What Does What Does DifferenceDifference Mean? Mean?

Mediumvariability

Highvariability

Mediumvariability

Highvariability

Lowvariability

Mediumvariability

Highvariability

Lowvariability

The mean differenceis the same for all

three cases.

Mediumvariability

Highvariability

Lowvariability

Which one showsthe greatestdifference?

What Does What Does DifferenceDifference Mean? Mean? A statistical difference is a function of the A statistical difference is a function of the

difference between meansdifference between means relative to the relative to the variabilityvariability..

A small difference between means with large A small difference between means with large variability could be due to variability could be due to chancechance..

Like a Like a signal-to-noisesignal-to-noise ratio. ratio.

Lowvariability

Which one showsthe greatestdifference?

What Do We Estimate?What Do We Estimate?

Lowvariability

Signal

Lowvariability

Signal

Difference between group means=

Lowvariability

Signal

Difference between group means

Variability of groups=

Lowvariability

Signal

=XT - XC

SE(XT - XC)

Lowvariability

Signal

XT - XC

SE(XT - XC)=

= t-value

The t-test, one-way analysis of variance The t-test, one-way analysis of variance (ANOVA) and a form of regression all test the (ANOVA) and a form of regression all test the same thing and can be considered equivalent same thing and can be considered equivalent alternative analyses.alternative analyses.

The regression model is emphasized here The regression model is emphasized here because it is the most general.because it is the most general.

Lowvariability

Regression Model for t-Test or One-Way Regression Model for t-Test or One-Way ANOVAANOVA

yi = 0 + 1Zi + ei

Regression Model for t-Test or One-Way Regression Model for t-Test or One-Way ANOVAANOVA

yyii = = outcome score for the ioutcome score for the ithth unit unit

00 == coefficient for the coefficient for the interceptintercept

11 == coefficient for the coefficient for the slopeslope

ZZii == 1 if i1 if ithth unit is in the treatment group unit is in the treatment group

0 if i0 if ithth unit is in the control group unit is in the control groupeeii == residual for the iresidual for the ithth unit unit

yi = 0 + 1Zi + ei

where:

In Graph Form...In Graph Form...

0(Control)

1(Treatment) Zi

0(Control)

1(Treatment)

0(Control)

1(Treatment)

0(Control)

1(Treatment)

0 is the intercepty-value when z=0.

0(Control)

1(Treatment)

1 is the slope.

Why Is 1 the Mean Difference?

0(Control)

1(Treatment)

1 is the slope.

0(Control)

1(Treatment)

Intuitive Explanation:Because slope is the change in y for a 1-unit change in x.

Change in y

Unit change in x (i.e., z)

0(Control)

1(Treatment)

Since the 1-unit change in x is the treatment-control difference, the slope is the difference between

the posttest means of the two groups.

Change in y

Why Why 11 Is the Mean Difference Is the Mean Difference

ininyi = 0 + 1Zi + ei

First, determine effect for each group:

yi = 0 + 1Zi + ei

For control group (Zi = 0):

yi = 0 + 1Zi + ei

yC = 0 + 1(0) + 0

yi = 0 + 1Zi + ei

yC = 0 + 1(0) + 0 ei averages to 0across the group.

yi = 0 + 1Zi + ei

yC = 0 + 1(0) + 0

yC = 0

ei averages to 0across the group.

For treatment group (Zi = 1):

yi = 0 + 1Zi + ei

yC = 0 + 1(0) + 0

yC = 0

yi = 0 + 1Zi + ei

yC = 0 + 1(0) + 0

yC = 0

yT = 0 + 1(1) + 0

yi = 0 + 1Zi + ei

yC = 0 + 1(0) + 0

yC = 0

yT = 0 + 1(1) + 0

yi = 0 + 1Zi + ei

yC = 0 + 1(0) + 0

yC = 0

yT = 0 + 1(1) + 0

yT = 0 + 1

ininyi = 0 + 1Zi + ei

Then, find the difference between the two groups:

yi = 0 + 1Zi + ei

yT = 0 + 1

treatment

yi = 0 + 1Zi + ei

yC = 0yT = 0 + 1

yT - yC =

controltreatment

yi = 0 + 1Zi + ei

yC = 0yT = 0 + 1

yT - yC = (0 + 1)

controltreatment

yi = 0 + 1Zi + ei

yC = 0yT = 0 + 1

yT - yC = (0 + 1) - 0

controltreatment

yi = 0 + 1Zi + ei

yC = 0yT = 0 + 1

yT - yC = (0 + 1) - 0

controltreatment

yT - yC = 0 + 1 - 0

yi = 0 + 1Zi + ei

yC = 0yT = 0 + 1

yT - yC = (0 + 1) - 0

controltreatment

yT - yC = 0 + 1 - 0

yi = 0 + 1Zi + ei

yC = 0yT = 0 + 1

yT - yC = (0 + 1) - 0

controltreatment

yT - yC = 0 + 1 - 0

yT - yC = 1

ConclusionsConclusions

t-test, one-way ANOVA and regression t-test, one-way ANOVA and regression analysis all yield analysis all yield samesame results in this results in this case.case.

The regression analysis method utilizes The regression analysis method utilizes a a dummy variabledummy variable for treatment.for treatment.

Regression analysis is the most Regression analysis is the most generalgeneral model of the three.model of the three.

Statistical Analysis of the Two Group Post-Only Randomized Experiment.

Documents

Transcript of Statistical Analysis of the Two Group Post-Only Randomized Experiment.

Document Type: Statistical Analysis Plan A Randomized ...

Observational Learning Evidence From A Randomized Natural Field Experiment

Learning in situ: a randomized experiment in video streaming

Workshop on Econometric Methods for Program Evaluation · Statistical power of a randomized experiment Practical options for randomized experiments Workshop on Econometric Methods

Girls Who Code: A Randomized Field Experiment on Gender ...

A randomized experiment of a cognitive strategies approach ...

ANOVA TABLE Factorial Experiment Completely Randomized Design.

Statistical Experiment A statistical experiment or observation is any process by which an measurements are obtained.

Differences-in-Differences and Instrumental Variables · 2 Introduction To Difference in Differences (DD) Data Randomized experiment or natural experiment (quasi-experiment), both

Final Results from a Multisite Randomized Experiment

A randomized experiment using absenteeism information · PDF fileA randomized experiment using absenteeism information ... Project Oficer . REL 2017–252 . ... absenteeism, with the

Statistical analysis plan of a randomized controlled trial ...

STATISTICAL MODELLING Appendix B Randomized layouts and ... fileApp B–1 STATISTICAL MODELLING . Appendix B Randomized layouts and sample size computations in R . B.1. Completely

3P Study: Preliminary results of an on-going randomized experiment

Social Influence bias; a randomized experiment

Combating Auto Theft in Arizona: A Randomized Experiment with

Statistical design challenges of a Phase 2/3 randomized ...

Statistical Analysis Plan A Randomized, Double-Blind ... · Document Type: Statistical Analysis Plan Official Title: A Randomized, Double-Blind, Placebo-Controlled Phase II Study

Simplification, Assistance, and Incentives: A Randomized ... · Simplification, Assistance, and Incentives: A Randomized Experiment to Increase College Savings ... A Randomized Experiment

Objectives (BPS chapter 9) Producing data: experiments Experiments How to experiment badly Randomized comparative experiments The logic of randomized.