DTC Quantitative Methods Measurement II: Questionnaire Design and Scale Construction Friday 27 th January 2012 Reading material In addition to the online course extract from Aldridge and Levine (2001), there are useful chapters in Oppenheim (1992) and De Vaus (2001). Remember While the research instrument in face-to-face surveys is arguably more correctly referred to as the interview schedule, the literature often incorporates such instruments under the heading questionnaire design. The four stages of questionnaire design (i) Mapping the semantic domain This stage involves identifying for each of the concepts fundamental to the proposed research its components (i.e. sub-concepts), context (i.e. related and similar but different concepts), and parameters (i.e. the conditions which must be present for the concept to apply). (ii) Formulating the question ideas This stage involves identifying the range of issues on which specific questions need to be asked, i.e. it identifies that a question has to be asked relating to a particular sub-concept: We need to ask a question about.... At this stage one is moving from concepts towards indicators. (iii) Writing the questions At this stage the specific wording of the questions is determined (iv) Assembling the questionnaire At this stage the questions are assembled into a coherent, structured questionnaire. Source: Halfpenny, P., Parthemore, J., Taylor, J. and Wilson, I A knowledge based system to provide intelligent support for writing questionnaires. In Westlake, A. et al. (eds) Survey and Statistical Computing. London: North Holland. Four aspects of a questionnaires contents De Vaus refers to (i) Measures of the dependent variables (ii) Measures of the independent variables (iii) Measures of test variables (i.e. variables that intervene between the independent and dependent variables, or which are temporally prior and possibly causally related to both) (iv) Background measures (i.e. demographic variables) Preliminaries Unstructured discussions with potential respondents/key informants. The idea of this qualitative work is to give the researcher a more complete picture of the topic, and the language and perspectives of respondents. A search for existing questions on the topic, (e.g. via the Question Bank). However, inheriting measures from previous research carries a risk of reflecting the pre-conceptions of earlier researchers. Piloting the draft questionnaire, to check for problems and to allow questions to be refined. Some general points Avoid putting ideas into the respondents mind early in the interview if we need spontaneous responses later on (Oppenheim). Beware of asking questions that unwittingly reveal the researchers attitude to the topic. A question that strikes the respondent as rude or inconsiderate may affect not only his [sic] reply to that particular question but also his [sic] attitude to the next few questions and to the survey as a whole (Oppenheim) Make the questionnaire attractive to the respondent by making it interesting and of obvious relevance to its stated purpose. A framework for question sequences Oppenheim outs forward the following, chronological framework (derived from a Gallup schema): Respondents awareness of issue. Respondents general feelings about issue. Respondents views on specific aspects of issue. Reasons for respondents views. Strength of respondents views. Open or Closed I? Open questions, where the respondents verbatim answer to the question is recorded in full, are easy to ask, less easy to answer and difficult to analyse. The emphasis is on the respondents perspective, but there is still the possibility that answers will reflect what is uppermost in respondents minds. Closed questions are easier to quantify, but result in a loss of spontaneity and expressiveness, and the forced choice of answers may result in bias (and shift the balance towards the researchers perspective). A compromise is to ask an Open question and then a similar Closed question later in the questionnaire/interview. Oppenheim comments that All closed questions should start their careers as open ones, except those where certain alternatives are the only ones possible. Open or closed II? Awareness of issue Closed, General feelings on issue Open, Views on specific aspects of issue Closed, Reasons for views on issue Open or Closed, Strength of views on issue Closed. Source: De Vaus (1986), adapted from Gallup. Types of forced choice answers A Likert scale: e.g. a range of answers from Strongly agree through to Strongly disagree. A semantic differential scale: A range of positions between two extremes of a continuum (e.g. Caring through to Uncaring). A checklist: e.g. a list of leisure activities. A ranking of items: e.g. placing the most important attributes of a potential partner in order. A choice between statements: e.g. a choice of responses to the acquisition of the knowledge that ones best friends partner is being unfaithful to them. Some key issues Some crunch questions that one might ask: Does the respondent understand the question (in the same way the researcher does!), Are they willing to answer it (accurately?), and Are they able to answer it? Scale construction Researchers sometimes want to measure some latent characteristic of their respondents, e.g. whether they have a traditional or a modern viewpoint on couple relationships. This is often done by asking a number of questions which each tap that characteristic and, when aggregated, collectively do so in more reliable way. Some issues... How does one know that a measure constructed by aggregating various items to give a scale is measuring the right quantity, i.e. is a valid measure? How does one ensure that what a measure is measuring is unidimensional, i.e. that it is not a composite of measures of two or more underlying concepts? How does one assess which items need to be included to maximise the reliability of the measure? And how does one assess the overall reliability of the scale? Some answers... For a discussion of assessing various forms of validity, see Oppenheim (1992). Unidimensionality can be assessed using a technique called factor analysis, see DeVellis (2003). Reliability can be assessed using a measure called Cronbachs alpha (see De Vaus, 2001; DeVellis, 2003). Factor analysis Factor analysis generates a set of underlying factors which successively maximise the amount of (remaining) variation in the items that they can explain. If a scale is working properly unidimensionally, then the first factor will explain a high proportion of the variation, and the subsequent factors similar, small amounts. Cronbachs alpha According to DeVellis (2003: 95), Alpha is an indication of the proportion of variance in the scale scores that is attributable to the true score. Items are chosen for inclusion so as to maximise that proportion, and if they have relatively high correlations with the rest of the items within the scale (viewed collectively). De Vaus (2001) suggests that a value of at least 0.7 is preferable. Some additional issues... Should all the items in the scale be treated as of equal importance? Or should their values be added in such a way as to increase/ decrease the relative importance of some items? Are the gaps between the values that a variable can take uniform in meaning?