Week 3 ETEC 668 Quantitative Research in Educational Technology

Dr. Seungoh Paek

January 29, 2014

Tonight’s Agenda

Announcements Feedback & Discussion on 611 vs. 668 Statistical Software Laulima vs. Weebly

Continuing Week 2 Experimental Research Methodology

Introduction to Statistics Designing & Conducting Educational

Technology Research

Feedback on 668 Course

Research Methodology Course

611 provides an overview of quantitative & qualitative research methods used in Educational Technology Research.

668 focuses on quantitative research methods used in Educational Technology Research.

667 focuses on qualitative research methods used in Educational Technology Research.

611 vs. 668

There will be some overlap since all three courses are research methodology classes. – The topics that overlap are ones that will be discussed and

reinforced throughout your doctoral program. – Understanding research methodologies is a major

component of doctoral work.

668 has a statistics component along with a focus on quantitative research methodology.

The focus of ETEC research will be different.

1. We will focus more on activities and application than on theoretical overview.

Course StructureSTATISTICS ETEC Research

Introduction to Statistics Introduction to Quantitative Research

Statistical Procedures Research Problem

Descriptive Statistics Literature Review

Frequency, Mean, Variance, etc. Research Methods

Inferential Statistics Research Design

Correlations Data Collection

Linear Regression Data Analysis

t-test Report Results

Analysis of Variance (ANOVA) Discussion & Presentation

2. We will focus on experimental design.

• Read experimental research articles.• Discuss experimental research design.• Conduct small-scale experimental

research.

Research Approaches

Quantitative approaches

Qualitative approaches

Survey research

Causal-comparative

research

Correlational research

Experimental research

Narrative research Phenomenological

research Ethnographic research Case study research Grounded theory research

Research Approaches

Quantitative approaches

Qualitative approaches

Survey research

Causal-comparative

research

Correlational research

Experimental research

Narrative research Phenomenological

research Ethnographic research Case study research Grounded theory research

Questions/Comments?

Statistical software Update

Laulima vs. Weebly

Continuing Week 2

Experimental Research Methods

Experiment in the lab?

~ Thorndike (1911)

Experimental Research

Experimental research is interested in the relationships between variables.

Experimental research provides strong evidence for causal interpretations.

To do so, researchers often manipulate variables, called independent variables systematically.

The results or effects from the variation of independent variables become dependent variables.

“Experimental” Designs

R X1 O (R = randomly assigned subjects; X = treatment)

R X2 O (O = observation/outcome)

Should be:– Replicable: repeat with the same results in another setting– Generalizable, representative– Cumulative: observations from earlier experiment used as a

basis for new one– Causal: establishes cause and effect (predictive)

Types of Experimental Educational Technology Research

Random assignment of subjects (“Experimental”)

Nonrandomized & single-subject designs (Quasi-experimental)

Quasi-Experimental

X1 O (X = treatment)

X2 O (O = observation/outcome)

Randomness is approximated through pre-tests to ensure “equivalence”

Group Activity II

Quasi-Experimental Research

Mayer, R. E., & Moreno, R. (1998). A split-attention effect in multimedia learning: Evidence for dual processing systems in working memory. Journal of educational psychology, 90(2), 312-320.

Dual-Processing Theory of Working Memory

a) Working memory includes an auditory working memory and a visual working memory.

b) Each working memory store has a limited capacity.

c) Meaningful learning occurs when a learner retains relevant information in each store, organize the information in each store into a coherent representation, and makes connections between corresponding representations in each store.

d) Connections can be made only if corresponding pictorial and verbal information is in working memory at the same time.

Research Design

Two experiments were conducted to test a prediction of a dual-processing theory of working memory.

Read the “Method” section for each Experiment: Group 1 = Exp 1; Group 2 = Exp 2.

Describe the procedure of each experiment. Summarize and report the findings of the

experiments.

Presentation

Dual-Processing Theory of Working Memory

Experiment 1 & 2 Results

Questions/Comments?

Yippee! I’m in statistics.

Statistics or Sadistics?:

It’s Up to You

Why this course…

Statistics for People Who (Think They) Hate Statistics – Follows an approach that is:

• Un-intimidating• Informative• Applied• Even a little fun!

Statistics: What It Is (and Isn’t)

Statistics describes “a set of tools and techniques that is used for describing, organizing, and interpreting data.”

Statistics is a problem-solving process that seeks answers to questions through data:– Ask a Question– Collect Appropriate Data– Analyze Data – Organize & Summarize– Interpret the Results

Success in this Course A few hints for successful completion of this

course– There are no dumb questions– How do you know statistics is hard?– Don’t skip lessons!! – Form a study group– Ask questions– Work through the exercises in each chapter– Practice, Practice, Practice– Look for real-world applications– Browse– HAVE FUN!!

Some websites with data

Substance Abuse & Mental Health Data Archive (SAMHDA) http://www.icpsr.umich.edu/SAMHDA/

Data2010… the Healthy People 2010 database http://wonder.cdc.gov/data2010/ftpselec.htm

http://www.lib.umich.edu/govdocs/stats.html Global Distribution of Poverty

http://sedac.ciesin.columbia.edu/povmap/

Descriptive or Inferential?

What is Descriptive Statistics?– Used to organize and describe the characteristics of a

particular data set• Example: the average age of everyone in this class!

What is Inferential Statistics?– Used to make inferences from your “sample” to the

“population”• Example: comparing the mean age of students taking this

course to average age of all students in an introductory statistics course

Basic Measurement Scales: Why Measurement?

You need to know that the data you are collecting represents what it is you want to know about.How do you know that the instrument you are using to collect data works every time (reliability) and measures what it is supposed to (validity)?

Scales of Measurement

Measurement is the assignment of values to outcomes following a set of rules

There are four types of measurement scales– Nominal– Ordinal– Interval– Ratio

Nominal Level of Measurement

Variables with characteristics that fits one and only one category

Name the attributes comprising a variable

Mutually exclusive categories such as male or female, Caucasian or African-American, etc.

Least precise level of measurement

Ordinal Level of Measurement

Next step beyond naming the attributes Characteristics being measured are ordered,

arranged in some order: from low to high, more to less, etc.

Rankings such as #1, #2, #3; liberal, middle-of-the-road, conservative

You know that a higher rank is better, but not by how much

Interval Level of Measurement

Variables having standard intervals of measurement

Intervals along the scale are equal to one another (same)

Lack a genuine zero point E.g. IQ, oC (Celsius) Interval & ratio scales usually lumped

together (in PSPP – called Scale)

Ratio Level of Measurement

Intervals along the scale are equal to one another (same)

Characterized by the presence of absolute zero on the scale

E.g. Age, oK (Kelvin scale) Distance between 10 yrs old and 20 yrs old

= Distance between 60 yrs old and 70 yrs old

Things to Remember

Any outcome can be assigned one of four scales of measurement

Scales of measures have an order The “higher” up the scale of measurement,

the more precise the data More precise scales contain all of the

qualities of the scales below it. Whenever possible, choose highest level –

ratio or interval scale

Part IIigma Freud & Descriptive

Statistics

Chapter 2 Means to an End:

Computing and Understanding Averages

What you will learn in Chapter 2

Measures of central tendency Computing the mean for a set of scores Computing the mode using the mode and

the median for a set of Selecting a measure of central tendency

Measures of Central Tendency

The AVERAGE is a single score that best represents a set of scores

Averages are also know as “Measure of Central Tendency”

Three different ways to describe the distribution of a set of scores…– Mean – typical average score– Median – middle score– Mode – most common score

Computing the Mean

Formula for computing the mean

“X bar” is the mean value of the group of scores

“” (sigma) tells you to add together whatever follows it

X is each individual score in the group The n is the sample size

XX

n

Things to remember…

N = population n = sample Sample mean is the measure of central

tendency that best represents the population mean

Mean is VERY sensitive to extreme scores that can “skew” or distort findings

Average means the one measure that best represents a set of scores– Different types of averages– Type of average used depends on the question

Weighted Mean Example

List all values for which the mean is being calculated (list them only once)

List the frequency (number of times) that value appears

Multiply the value by the frequency Sum all Value x Frequency Divide by the total Frequency (total n size)

Computing the Median

Median = point/score at which 50% of remaining scores fall above and 50% fall above.

NO standard formula– Rank order scores from highest to lowest or

lowest to highest– Find the “middle” score

BUT…– What if there are two middle scores?– What if the two middle scores are the same?

A little about Percentiles…

Percentile points are used to define the percent of cases equal to and below a certain point on a distribution– 75th %tile – means that the score received is at

or above 75 % of all other scores in the distribution

– “Norm referenced” measure• allows you to make comparisons

Computing the Mode

Mode = most frequently occurring score NO formula

– List all values in the distribution– Tally the number of times each value occurs– The value occurring the most is the mode

Democrats = 90Republicans = 70Independents = 140 – the MODE!!

– When two values occur the same number of times -- Bimodal distribution

When to Use What…

Use the Mode – when the data are categorical

Use the Median – when you have extreme scores

Use the Mean – when you have data that do not include extreme

scores and are not categorical

Glossary Terms to Know

Average Measures of Central Tendency

– Mean• Weighted mean• Arithmetic mean

– Median• Percentile points• Outliers

– Mode

Part IIigma Freud & Descriptive

Statistics

Chapter 2 Viva la Difference:

Understanding Variability

Why Variability is Important

Variability– how different scores are from one particular

score• Spread• Dispersion

So…variability is really a measure of how each score in a group of scores differs from the mean of that set of scores.

Measures of Variability

Three types of variability that examine the amount of spread or dispersion in a group of scores…– Range – Standard Deviation– Variance

Typically report the average and the variability together to describe a distribution.

Computing the Range

Range is the most “general” estimate of variability…

Two types…– Exclusive Range

• R = h - l

– Inclusive Range• R = h – l + 1

(Note: R is the range, h is the highest score, l is the lowest score)

3

2

l h

Computing Standard Deviation

Standard Deviation (SD) is the most frequently reported measure of variability

SD = average amount of variability in a set of scores

What do these symbols represent?

Why n – 1?

The standard deviation is intended to be an estimate of the POPULATION standard deviation…– We want it to be an “unbiased estimate”– Subtracting 1 from n artificially inflates the SD…

making it larger

In other words…we want to be “conservative” in our estimate of the population

Computing Variance

Variance = standard deviation squared

Standard Deviation or Variance

While the formulas are quite similar…the two are also quite different.– Standard deviation is stated in original units– Variance is stated in units that are squared

Glossary Terms to Know

Variability– Range– Standard deviation

• Mean deviation• Unbiased estimate

– Variance

Statistics Poker game

SPSS

Using SPSS

Using SPSS

Some websites with data

Substance Abuse & Mental Health Data Archive (SAMHDA) http://www.icpsr.umich.edu/SAMHDA/

Data2010… the Healthy People 2010 database http://wonder.cdc.gov/data2010/ftpselec.htm

http://www.lib.umich.edu/govdocs/stats.html Global Distribution of Poverty

http://sedac.ciesin.columbia.edu/povmap/

Designing & Conducting Educational Technology Research

Discussion

Share Your Thoughts

What is good research? How do you know? Why is it so hard to do good research in

Educational Technology?

Questioning the Questions of Instructional Technology Research

Is I.T. research socially relevant? Can I.T. research be socially relevant? Should I.T. research be socially relevant?

~ Reeves (1995)

Ed. Tech. Research Goals

Theoretical Predictive Interpretivist Postmodern Development Action

Theoretical Goals Focus on explaining

phenomena through logical analysis and synthesis of principles and results from other studies

EXAMPLE: Gagne’s theory of the conditions of learning

Predictive Goals Focus on determining

how education works by testing hypotheses related to theories of learning, teaching, performance, etc.

EXAMPLE: cooperative learning and control studies by Hooper, Temiyakarn, and Williams

Simon Hooper

Interpretivist Goals Focus on determining how

education works by describing and interpreting phenomena related to learning, teaching, performance, etc.

EXAMPLE: Delia Neuman’s observations of disabled children using commercial software

Delia Neuman

Postmodern Goals Focus on examining the assumptions underlying

educational programs with the goal of revealing hidden agendas and empowering disenfranchised minorities

EXAMPLE: Ann DeVaney’s analysis of IT in relation to race, gender, and power

Development Goals Focus on dual objectives of

developing creative approaches to solving problems and constructing reusable design principles

EXAMPLE: Cognition and Technology Group at Vanderbilt’s (John Bransford et al.) work with Jasper Woodbury problem-solving series

Evaluation/Action Goals Focus on describing,

improving, or estimating the effectiveness and worth of a particular program

EXAMPLE: Reeves and Laffey study of a problem-based learning engineering course at US Air Force Academy

Ed. Tech. Research Methods

Classification of ETRD (1989-94) & JCBI (1988-93) articles

More Recently

228 articles from the years of– 1985-1986– 1995-1996 – 2003-2004

~Jo, I. H. (2005)

Problem of Pseudoscience

Specification error Lack of linkage to robust theory Inadequate literature review Inadequate treatment implementation Measurement flaws Inconsequential outcome measures Inadequate sample size Inappropriate statistical analysis Meaningless discussion of results

Reeves’ Suggested Solution

Focus on making education work better Developmental research situated in schools

with real problem

Good Development ResearchDesign-Based Research Prerogative

Goals of designing learning environments and theories are intertwined

Development and research occur in continuous cycles

Research on designs leads to sharable theories relevant to practitioners

Research must account for how designs function in authentic settings

Development of accounts relies on methods that connect actions to outcomes

Ed Tech Research that Makes a Difference (Roblyer, 2005)

“need a more organized and persuasive body of evidence on technology’s benefits to classroom practice”

Ed tech research problems/challenges– Weak research design– Fragmented, uncoordinated studies– Poor methods-research questions match– Badly written reports– “ubiquitous interactions”– “hard-to-do science”– Technology changes quickly

Pillars of Good Educational Research

1. The Significance Criterion

2. The Rationale Criterion

3. The Design Criterion

4. The Comprehensive Reporting Criterion

5. The Cumulativity Criterion

New Ed Tech Research Agenda

Research to establish relative advantage Research to improve implementation

strategies Research to monitor impact on societal

goals Research that monitor & report on common

uses & shape desired directions

Design Research Example “Authentic Learning in

Interactive Multimedia Environments.”

Ph.D. dissertation by Jan Herrington at Edith Cowan University in Australia.

Supervised by Professor Ron Oliver.

Winner of AECT Young Researcher of the Year in 1999.

Outcome Practitioners Desired

New teachers will use a wider variety of assessment methods in their student teaching experience and eventual practice.

Learning Environment Design

Identified the critical characteristics of a situated learning model.

Developed an interactive multimedia learning environment based on those characteristics.

Situated Learning Model Herrington 1997

Provide an authentic context reflecting the way the knowledge will be used in real-life

Provide authentic activities Provide access to expert performances and

the modeling of processes Provide multiple roles and perspectives Support collaborative construction of

knowledge

Situated Learning Model Herrington 1997

Promote reflection to enable abstractions to be formed

Promote articulation to enable tacit knowledge to be made explicit

Provide coaching and scaffolding at critical times

Provide for integrated assessment of learning within the tasks.

Research Methodology

Mixed methods Videotaped preservice teachers

using program Interviewed teachers and their

supervisors in schools during student teaching practicum

Findings Problem Solved:

– Novice teachers acquired advanced knowledge while engaging in higher order thinking

– New knowledge and skills applied in practicum

Design Principles:– Situated learning model is

a successful design model for eLearning

Group Research Project Possible Group Research Project Topic

7-part Model for Conceptualizing Quantitative Ed Tech Research

1. Select a Topic2. Identify the Research Problem3. Conduct a Literature Review4. State the Research questions and hypotheses5. Determine the Research Design6. Determine the Methods7. Identify Data Analysis Procedures

Selecting a Topic

Identify general area of interest, focus Find something that you’re passionate

about Topic that would “make a difference” Something fun that you’re curious about

(hopefully!) Groups’ topic areas – future schools; cyber

charter school; tech integration; what else?

Start with Questions

As a researcher in the field of educational technology, what do you think is your task? In other words, what do you want to do as a ETEC researcher?

Is there an exemplary research paper that you have read about the field of educational technology?

What changes do you want to make to the field? Is there a contribution that you want to make? If so, in which area of this multidisciplinary area?

Identify the Research Problem

Determine problem/concern within your topic area

How important is the problem to the field How does it expand on existing knowledge Requires knowledge of the literature &

current research activities Bounce research problem/focus off others

What to do in Week 03

1. To get you started with your final research paper, discuss and post your group's tentative research topic/problem. (Forum Week 03)

2. Do the required readings for Week 04.Note: Readings are available for download on Laulima (Modules Week 04 Readings for Week 04)

What to do in Week 03

3. Read and present a experimental research article. Find a partner. Read one of the articles listed on the next slide. Prepare a 10 minute presentation of the

research paper (pretend you’re the authors of the article).

List of Readings for Multimedia Learning Principles

Moreno, R., & Mayer, R. E. (1999). Cognitive principles of multimedia learning: The role of modality and contiguity. Journal of educational psychology, 91(2), 358-368.

Mayer, R. E., Fennell, S., Farmer, L., & Campbell, J. (2004). A Personalization Effect in Multimedia Learning: Students Learn Better When Words Are in Conversational Style Rather Than Formal Style. Journal of Educational Psychology, 96(2), 389-395.

Mayer, R. E., & Johnson, C. I. (2008). Revising the redundancy principle in multimedia learning. Journal of Educational Psychology, 100(2), 380-386.

Holsanova, J., Holmberg, N., & Holmqvist, K. (2009). Reading information graphics: The role of spatial contiguity and dual attentional guidance. Applied Cognitive Psychology, 23(9), 1215-1226.