Quantitative

1

Quantitative Quantitative Data CollectionData Collection

InstrumentsInstruments

Presented by:Presented by:Dr Vanessa Dr Vanessa SchermanScherman

[email protected]@up.ac.za

Overview of the presentation

2

Introduction Starting at the beginning Important terms Measurement instruments in quantitative research Process to be followed Validity and reliability Important aspects to consider

Introduction

3

Research is to see what everybody else has seen, and to think what nobody else has

thought

Albert Szent-Gyorgyi

Introduction

4

People respond to questionnaires and other instruments many times during their lifetime

Careful development is key if unsatisfactory results are to be avoided

Not an easy task Remember the logical golden thread…

Starting at the beginning

5

Research questions

Literature review

Conceptual framework

Specific research questions

Data level questions

Instrument development is not divorced from research questions, literature and analysis

Remember Deductive Reasoning– General to the Specific

6

Theory

Hypothesis

Observation

Confirmation

7

Taken from Cohen, Manion and Morrison, 2003

Instrument – a tool used to collect data Test – a formal, systematic procedure for

gathering information Assessment – the general process of

collecting, synthesizing, and interpreting information

Measurement – the process of quantifying or scoring a subject’s performance

Important Terms

Cognitive tests – examining subject’s thoughts and thought processes

Affective tests – examining subject’s feelings, interests, attitudes, beliefs, etc

Standardised tests – tests that are administered, scored, and interpreted in a consistent manner

Important Terms

Selected response item format– respondents select answers from a set of alternatives

Supply response item format – respondents construct answers

Individual tests – tests administered on an individual basis

Group tests – tests administered to a group of subjects at the same time

Performance assessments – assessments that focus on subject’s processes or products that have been created

Important Terms

11

Valid and reliable Instruments should be attractive, brief and

easy to respond to Carefully plan the format and content Do not include items that do not relate to your

topic Structured selection type questions are

preferable

Measurement Instruments

Types of Instruments

Cognitive – measuring intellectual processes such as thinking, memorizing, problem solving, analyzing, or reasoning

Aptitude – measuring general mental ability usually for predicting future performance

Types of Instruments

Affective – assessing individual feelings, values, attitudes, beliefs, etc.

Typical affective characteristics of interest Values – deeply held beliefs about ideas, persons, or

objects Attitudes – dispositions to favorable or unfavorable

toward things Interests – inclinations to seek out or participate in

particular activities, objects, ideas, etc. Personality – characteristics that represent a person’s

Affective (continued)

Scales used for responding to items on affective tests

• Likert• Semantic differential• Thurstone• Guttman• Rating scales

Interview Schedules

Vocal questionnaires Direct interaction with participants Non-verbal and verbal responses can be

noted Higher response rate than questionnaires BUT subjectivity and bias Higher costs and more time consuming Lack of anonymity

Observation Schedules

Structured Detached from the group and processes Captures natural behaviour Relatively unobtrusive Can be costly and time consumingObserver bias Inability to probe and clarify

Process to be Followed?

Define your objectives Understand your key terms Review literature for similar studies as

yours Develop a framework for yourselfWrite questions Be wise in choosing item formats ALWAYS PILOT YOUR INSTRUMENTS

Frameworks, Frameworks Frameworks?

18

Based on literature key concepts were identifiedKey concepts are defined from literatureQuestions are designed to obtain information regarding the concept

Assessment Framework - Blueprint

19

Purpose (type) of Assessment Draft the assessment instrument (Exam, Individual Assessment) Scoring and Analysis

Assessment Framework (Curriculum documents, e.g. NCS,

TG, LAG, etc.)

Learning Area Purpose Grade level Learning Outcome Assessment Standard Item number Item format Maximum score Time Difficulty level Cognitive skills level

A

Writing of items (LA teachers)

B

C

D

E

A

B

Compilation of test/exam (Layout, typeset, etc.)

Test application (Classroom test/ exam)

Scoring / Marking / Evaluate (Memorandum/rubric)

Analysis of results (Difficulty value, discrimination index,

mean)

Moderation of papers (Self, Colleague, Head of Dept, External

(LA, specialist), Academic head Quality indicators / criteria

Pilot application

Re-working

Assessment Framework – Blueprint

20

Item no.

Mathematics Learning Outcome

AS*

Grade level

Accessibility with regard to the Grade level.

(Grade 7 (end) and/or Grade 8 (beginning))

Accessibility with regard to

the RNCS (Curriculum).

Cognitive level appropriate for

Grade 7 (end), Grade 8 (beginning) level.

Remarks

Very easy

Easy Moderate

Difficult

Not covered in Gr. 7 and/or

previous grades, AND

Knowledge

Comprehension

Application/Problem Solving

Possible NOT Possible

1 Numbers, Op. & Rel.

8 1 X N/A X


8 1 X N/A X


9 2 X N/A X


9 2 X N/A X


8 4 X N/A X


8 1 X N/A X

Questionnaire Framework

21

Construct Description Number of items

Item Format Possible data analysis strategies

Demographic information: teacher

Background information

11 Items Dichotomous ItemsOpen ended items

Descriptive statisticsRegression analysis

Educator attitude towards achievement

The importance the educator attaches to learner achievement. Positive attitude of teacher towards achievement (Mortimore, 1998). The extent to which educators are achievement oriented, positive expectations of learner achievement (Sammons, 1999).

6 Items 4 point Likert scaleStrongly agree, agree, disagree and strongly disagree

Descriptive statisticsReliability analysis – internal consistencyFactor analysisRasch analysisRegression analysis

Easy and Simple?

22

Constructing Items

23

Only include items which relate to the topic

Collect demographic information Each question should deal with a single

concept Avoid jargon Be specific – short simple items are best Avoid leading questions Avoid sensitive or touchy questions

Constructing Items

24

Avoid double barrel questions Respondents should be competent to

answer Avoid negatively phrased items

Types of Items

25

Open or closed? Level of agreement Degree of importance Frequency Semantic differential Ranked items Checklist items

Class Activity

26

Look at the first questionnaire provided Is this a good questionnaire or not? What type of items have been included? How do you think the questionnaire can be

improved upon?

Class Activity

27

Look at the second questionnaire provided Is this a good questionnaire or not? Are there any differences when compared to

the first questionnaire? What type of items have been included?

Cover letters

28

Explain what is being asked and why The letter should be brief, neat and addressed specifically to the potential responderExplain the purpose, importance and significantState a commitment to share the results!Anonymity and confidentiality

Pre-testing

29

Piloting the cover letter and instrument

Identify weaknesses and strengths

Face and content validity

What about Validity?30

Evidence based on content Evidence based on contrasting groups Evidence based on response processes Evidence based on internal structure Evidence based on relationships to other

variables

Content31

Extent to which that items represent the larger domain

Examine the relationship between content in the domain and content

Contrasting Groups32

Whether groups that should be different are different

Examine the difference between the groups

Response Processes33

Whether thinking and response patterns are consistent with intended interpretation

Examine the respondent explanations and response patterns

Internal Structure

34

Extent to which items measuring the same trait are related

Correlate items measuring the same trait

Relationships to Variables35

Whether the measure is related to similar or predicted variables unrelated to different variables

Correlate the measures of other measures of the same trait and the measures of different traits

Why is Validity Important?36

Validity implies proper interpretation Matter of degree Trust studies -make generalisations from

other studies NB INFERENCES ARE VALID NOT

INSTRUMENTS

What about Reliability?

37

Stability Equivalence Equivalence and stability Internal consistency Agreement

Stability38

Consistency of stable characteristics over time

Administer the same test to the same individuals over time

Equivalence39

Comparability of two measure of the same trait given at the same time (at about)

Administer different forms to the same individuals at the same time (at about)

Equivalence and Stability40

Comparability of two measures of the same trait given over time

Administer different forms to the same individuals at about the same time

Internal Consistency41

Split half, KR, Cronbach alpha Comparability of halves of a measure to

assess a single trait or dimension Administer one test and correlate the

items to each other

Agreement

42

Consistency of ratings or observations Two or more people rate or observe

Sources of Measurement Error43

Changes in times limits Changes in directions Different scoring procedures Interrupting administration sessions Time the test is taken, heat light and

ventilation Reactions of participants to items Mood, fatigue, attitude, test wiseness

Issues - Cognitive, Aptitude, or Affective Instruments

Bias – distortions of a respondent’s performance or responses based on ethnicity, race, gender, language, etc

Responses to affective test items Socially acceptable responses Accuracy of responses Response sets

Problems inherent in the use of self-report measures and the use of projective tests

Issues - Selecting Instruments

Non-psychometric issuesCostAdministrative timeObjections to content by

parents or othersDuplication of testing

Designing your own Instruments

Get help from others with experience developing tests

Item writing guidelinesAvoid ambiguous and confusing

wording and sentence structureUse appropriate vocabularyWrite items that have only one

correct answerGive information about the

nature of the desired answerDo not provide clues to the

correct answer

Test Administration Guidelines

Plan aheadBe certain that there is consistency

across testing sessionsBe familiar with any and all procedures

necessary to administer a test

Statistical Considerations

Cannot be divorced from development process

Method of analysis should be established before items are written

Analysis should direct format Remember deductive reasoning

Thank you for your attention

Quantitative

Documents

Transcript of Quantitative