8Module

UNESCO International Institute for Educational Planning

Maria Teresa Siniscalcoand Nadia Auriat

Questionnaire design

Quantitative research methods in educational planning

Series editor: Kenneth N.Ross

International Institute for Educational Planning/UNESCO7-9 rue Eugne-Delacroix, 75116 Paris, FranceTel: (33 1) 45 03 77 00Fax: (33 1 ) 40 72 83 66e-mail: information@iiep.unesco.orgIIEP web site: http://www.unesco.org/iiep

The designations employed and the presentation of material throughout the publication do not imply the expression of any opinion whatsoever on the part of UNESCO concerning the legal status of any country, territory, city or area or of its authorities, or concerning its frontiers or boundaries.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means: electronic, magnetic tape, mechanical, photocopying, recording or otherwise, without permission in writing from UNESCO (International Institute for Educational Planning).

Graphic design: Sabine LebeauTypesetting: Sabine LebeauPrinted in IIEPs printshop

September 2005 UNESCO

Quantitative research methods in educational planning

These modules were prepared by IIEP staff and consultants to be used in training workshops presented for the National Research Coordinators who are responsible for the educational policy research programme conducted by the Southern and Eastern Africa Consortium for Monitoring Educational Quality (SACMEQ).

The publication is available from the following two Internet Websites:http://www.sacmeq.org and http://www.unesco.org/iiep.

mailto:information@iiep.unesco.orghttp://www.unesco.org/iiephttp://www.sacmeq.orghttp://www.unesco.org/iiep

Educational research: some basic concepts and terminology Module 1

Content

1

1. Introduction 1

2. Initial planning 3Why a new questionnaire and when? 3

Relationships between research problems, research hypotheses, and variable construction 5

Characteristics of research hypotheses 6

Specifying variables and indicators 6

Operationalization of research questions 8

The problem of the cross-national validity of educational concepts, definitions, and data collection instruments 12

1. What is formal education? 12

2. Distinguishing public and private service providers 14

3. What is a school? 16

4. What is a student? 17

5. What is a teacher? 18

3. The design of questions 22Question structure 221. Closed questions 23

2. Open-ended questions 26

3. Contingency questions 28

Guidelines for writing questions 29

UNESCO

Educational research: some basic concepts and terminologyModule 1

II

Specifying the characteristics of respondents 34

A checklist for reviewing questionnaire items 35

4. Examples of questions 38Student background 381. Gender and age 39

2. Socio-economic background: occupation, education, and possessions 41

Teacher characteristics 47

School location 49

Learning, teaching, and school activities 511. Student reading activity 51

2. Teacher activities 52

3. School head activities 53

Attitudes, opinions, and beliefs 551. Likert scaling 55

5. Moving from initial draft to f inal version of the questionnaire 63Two widely-used patterns of question sequence 63

General guidelines for item placement 65

Covering letters and introductory paragraphs 66

Drafting instructions for answering questions 68

Training of interviewers or questionnaire administrators 70

Pre-testing the questionnaire 72

Basic steps in pre-testing 75

Reliability and validity 761. Validity 76

2. Reliability 78

UNESCO

The codebook 79

6. Further reading 84

III

Content

1 UNESCO

Introduction 1

This module provides guidance for the design of standardized questionnaires that are to be administered in school systems to students, teachers, and school heads. The module is divided into four sections that cover initial planning, the design of questions, examples of question types, and moving from a draft to a final questionnaire.

After reading this module, the reader should be able to design a quality survey questionnaire that is suitable for addressing the research issues at hand. He or she will know how to:

Decide on the target population for the questionnaire.

Identify the variables and indicators that will address the research issues and hypotheses on which data are to be collected.

Develop demographic, knowledge, attitude, and practice questions.

Close open ended quantitative and qualitative questions and design skip, filter, and contingency questions, where appropriate.

Decrease response bias and maximize response rates.

Design probe questions and interviewer or respondent instructions on the questionnaire.

Questionnaire designModule 8

2 3 UNESCO

Conduct a pilot test of the questionnaire, and adjust its final design according to the results.

Prepare a codebook for data entry.

Questionnaire designModule 8

2 3 UNESCO

This section reviews the steps required to determine the need for a new questionnaire, and looks at how a general research problem needs to be translated into a number of specific research questions and hypotheses. It examines the problem of valid cross-national instruments and provides helpful hints and recommendations for using comprehensive and precise definitions of key educational concepts.

Why a new questionnaire and when?This module addresses the planning and design of standardized questionnaires. A formal standardized questionnaire is a survey instrument used to collect data from individuals about themselves, or about a social unit such as a household or a school. A questionnaire is said to be standardized when each respondent is to be exposed to the same questions and the same system of coding responses. The aim here is to try to ensure that differences in responses to questions can be interpreted as reflecting differences among respondents, rather than differences in the processes that produced the answers.

Standardized questionnaires are often used in the field of educational planning to collect information about various aspects of school systems. The main way of collecting this information is by asking people questions either through oral interviews (face to face or telephone), or by self-administered questionnaires, or by using some combination of these two methods.

Initial planning 2

4

Questionnaire designModule 8

5

Initial planning

UNESCO

Although survey research, by definition, implies the use of some form of questionnaire to be administered to a sample of respondents, the questionnaire is simply one instrument that can be employed in the study of a research problem. As such, it may or may not be the most suitable tool for the task at hand.

Hence, before deciding on the need for a new questionnaire, one should consider whether or not some of the required information may already be available from other sources, for example, from statistics compiled by governments or research agencies, or from survey research archives. One should also consider whether a suitable questionnaire already exists that could be wholly or partially used.

The planner should also consider whether other means of data collection are more appropriate. These can be (a) field experiments, where people in treatment and control groups respond to a scenario devised by the investigators, (b) content analysis of newspapers or articles, (c) direct observation (such as counting the number of schools in a district, the number of blackboards in a school or the number of students per teacher in a given area), or (d) non-directive interviews where there are no pre-specified questions and the interviewer has a great deal of freedom in probing areas and specific issues during the course of the interview.

Among the types of information that can be collected by means of a questionnaire are facts, activities, level of knowledge, opinions, expectations and aspirations, membership of various groups, and attitudes and perceptions. In the field of educational planning, the information that is collected can be classified broadly into: (a) inputs to education (such as school resources or various background characteristics of schools, teachers or students), (b) learning and teaching processes, and (c) the outcomes of education (such as pupil achievement, attitudes towards school, and measures of school efficiency such as survival rates etc.).

4

Questionnaire designModule 8

5

Initial planning

UNESCO

Relationships between research problems, research hypotheses, and variable constructionThe development of a questionnaire commences with the transformation of general educational research and policy concerns into specific research questions for which the data are intended to supply an answer. Some examples of general educational policy and research concerns are: (a) policy-makers want to assess the supply of resources in their primary schools, (b) a curriculum expert wants to determine to what extent teaching methods explain differences in reading literacy among 9-year-old students, and (c) a national evaluation agency wants to investigate student attitudes towards science at the end of secondary school.

In the case of the above three examples, it would be necessary to establish empirical evidence for decision