Chapter Five: Nonexperimental Methods II: Ex Post Facto Studies, Surveys and Questionnaires,...

Chapter Five: Nonexperimental Methods

II: Ex Post Facto Studies,

Surveys and Questionnaires,

Sampling and

Basic Research Strategies

Ex Post Facto Studies

Ex post facto – is a Latin phrase meaning “after the fact.”

When we conduct an ex post facto study, we are using an IV “after the fact” – it has already varied before we arrived on the scene.

Surveys, Questionnaires, Tests, and Inventories

Surveys, Questionnaires, Tests, and Inventories Surveys typically request our opinion on some topic or

issue that is of interest to the researcher.

Surveys, Questionnaires, Tests, and Inventories Types of Surveys

Descriptive

Descriptive – seeks to determine what percentage of the population has a certain characteristic, holds a certain opinion, or engages in a particular behavior.

Examples:

Examples: The Gallup Poll

Examples: The Gallup Poll Nielsen television ratings

Descriptive Analytic

Descriptive Analytic –seeks to determine what the relevant

variables are and how they might be related.

variables are and how they might be related. Questions need to be chosen carefully.

variables are and how they might be related. Questions need to be chosen carefully. Pilot testing is generally necessary to determine the

type of questions that will be used in the final survey instrument.

Surveys, Questionnaires, Tests, and Inventories Developing a Good Survey or Questionnaire (see table

5-1, p. 89).

Surveys, Questionnaires, Tests, and Inventories Step 1

Surveys, Questionnaires, Tests, and Inventories Step 1 – decide what type of instrument to use. How

will the information be gathered?

will the information be gathered? Will you use a mail survey, conduct a phone interview, or

conduct the survey in person? Will you use trained interviewers?

Surveys, Questionnaires, Tests, and Inventories

Step 1 – decide what type of instrument to use. How will the information be gathered?

will the information be gathered? Step 2 – identify the types of questions to use.

Yes-No Questions

Yes-No Questions The respondent answers yes or no to the items.

Forced Alternative Questions

Forced Alternative Questions The respondent must select between two alternative

responses.

responses. Multiple-Choice Questions

The respondent must select the most suitable response from among several alternatives.

Likert-Type Scales

Surveys, Questionnaires, Tests, and Inventories Step 2 – identify the types of questions to use.

Likert-Type Scales The individual answers a question by selecting a

response alternative from a designated scale. A typical scale might be the following: (5) strongly agree, (4) agree, (3) undecided, (2) disagree, or (1) strongly disagree.

Surveys, Questionnaires, Tests, and Inventories Step 2 – identify the types of questions to use.

Likert-Type Scales The individual answers a question by selecting a

response alternative from a designated scale. A typical scale might be the following: (5) strongly agree, (4) agree, (3) undecided, (2) disagree, or (1) strongly disagree.

Open-Ended Questions

Surveys, Questionnaires, Tests, and InventoriesStep 2 – identify the types of questions to use.

Multiple-Choice Questions The respondent must select the most suitable response

from among several alternatives. Likert-Type Scales

The individual answers a question by selecting a response alternative from a designated scale. A typical scale might be the following: (5) strongly agree, (4) agree, (3) undecided, (2) disagree, or (1) strongly disagree.

Open-Ended Questions A question is asked to which the respondent must

construct his or her own answer.

will the information be gathered? Step 2 – identify the types of questions to use. Step 3

will the information be gathered? Step 2 – identify the types of questions to use. Step 3 – write the items: They should be clear, short,

and specific.

Surveys, Questionnaires, Tests, and Inventories Step 3 – write the items: They should be clear, short,

and specific. Use familiar vocabulary and be at the reading level of the

individuals you intend to test.

and specific. Step 4

and specific. Step 4 – Pilot-test and seek opinions from

knowledgeable others.

Surveys, Questionnaires, Tests, and Inventories Step 4 – Pilot-test and seek opinions from

knowledgeable others. It is important to ask others, especially professionals who

have expertise in your area of research interest, to review your items. They may be able to detect biases and unintended wordings that you had not considered.

knowledgeable others. Step 5

knowledgeable others. Step 5 – determine the relevant demographic data to

be collected.

Surveys, Questionnaires, Tests, and Inventories Step 5 – determine the relevant demographic data to

be collected. Includes items such as age, sex, annual income, size of

community, academic major, and academic classification.

be collected. Step 6

be collected. Step 6 – determine administration procedures and

develop instructions.

Surveys, Questionnaires, Tests, and Inventories Step 6 – determine administration procedures and

develop instructions. Instructions must be clear, concise, easy to follow, and

thoroughly rehearsed (for in-person or phone interviews). How will informed consent be obtained?

Surveys, Questionnaires, Tests, and Inventories Survey Administration Procedures

Mail Surveys

Mail Surveys Advantages

The researcher does not have to be present while the survey is being completed.

Surveys can be sent to a much larger number of participants than a single researcher could ever hope to contact in person.

Mail Surveys Disadvantages

The researcher cannot be sure who actually completes the survey.

There is no guarantee the respondent answered the questions in the same order in which they appeared in the survey.

Low return rate.

Surveys, Questionnaires, Tests, and Inventories Personal Interviews

Advantages When a trained interviewer administers a survey in a

respondent’s home, the response rate climbs dramatically.

Advantages When a trained interviewer administers a survey in a

respondent’s home, the response rate climbs dramatically.

The trained interviewer is able to clarify ambiguous questions, making sure that all questions are answered in the proper sequence, and generally assisting with any problems that the respondents may experience.

Disadvantages

Disadvantages The potential for considerable expenditure of time and

money exists.

Disadvantages The potential for considerable expenditure of time and

money exists. People not always at home.

Surveys, Questionnaires, Tests, and Inventories Telephone Interviews

Advantages

Advantages It is easy to create random samples with random-digit

dialing.

dialing. 95% of households in the U.S. have telephones.

dialing. 95% of households in the U.S. have telephones. Computer technology allows responses to be entered

directly as they are made.

Disadvantages

Disadvantages Incoming calls may be screened and potential

respondents lost.

respondents lost. It is easy to say no to a phone interviewer.

respondents lost. It is easy to say no to a phone interviewer. Cannot use visual aids to clarify questions.

respondents lost. It is easy to say no to a phone interviewer. Cannot use visual aids to clarify questions. Cannot evaluate nonverbal cues such as facial

expressions, gestures, and posture.

respondents lost. It is easy to say no to a phone interviewer. Cannot use visual aids to clarify questions. Cannot evaluate nonverbal cues such as facial

expressions, gestures, and posture. It is more difficult to establish rapport.

Surveys, Questionnaires, Tests, and Inventories Tests and Inventories

Tests and inventories are designed to assess a specific attribute, ability, or characteristic possessed by the individual being tested.

Surveys, Questionnaires, Tests, and Inventories Characteristics of Good Tests and Inventories

Validity

Validity – a test or inventory has validity when it actually measures what it is supposed to measure.

Validity (ways to establish) Content validity

Validity (ways to establish) Content validity – indicates that the test items actually

represent the type of material they are supposed to test.

Validity (ways to establish) Content validity – indicates that the test items

actually represent the type of material they are supposed to test. A panel of expert judges is often used to assess the content validity of test items.

Validity (ways to establish) Content validity Concurrent validity

Validity (ways to establish) Content validity Concurrent validity – is established when a score on a

test or inventory agrees with another measure (e.g. aggression score on a test agrees with a clinician’s assessment).

Validity (ways to establish) Content validity Concurrent validity Criterion validity

Validity (ways to establish) Content validity Concurrent validity Criterion validity – is established when a test score

compares favorably with a predicted future outcome (e.g. SAT score and college success).

Surveys, Questionnaires, Tests, and Inventories Reliability

Refers to the extent that the test or inventory is consistent in its evaluation of the same individuals over repeated administrations.

Surveys, Questionnaires, Tests, and Inventories Reliability

Refers to the extent that the test or inventory is consistent in its evaluation of the same individuals over repeated administrations.

The greater the similarity between scores produced by the same individuals on repeated administrations, the greater the reliability of the test or inventory.

Surveys, Questionnaires, Tests, and Inventories Reliability (assessment)

Test-retest procedure

Test-retest procedure The test is simply given a second time and scores from

the two tests are compared; the greater the similarity, the higher the reliability

Test-retest procedure Split-Half technique

Involves dividing a test or inventory into two halves or subtests and then administering them to the same individuals on different occasions or by administering the entire test and then splitting it into two halves.

Surveys, Questionnaires, Tests, and Inventories Types of Tests and Inventories

Achievement tests

Achievement tests Are given when an evaluation of an individual’s level of

mastery or competence is desired.

Achievement tests Are given when an evaluation of an individual’s level of

mastery or competence is desired. The score that distinguishes passing from failing

determines the minimum level of achievement that must be attained.

Achievement tests Aptitude tests

Are used to assess an individual’s ability or skill in a particular situation or job.

Achievement tests Aptitude tests Personality tests or inventories

Measure specific aspects of an individual’s motivational state, interpersonal capability, or personality.

Sampling Considerations and Basic Research Strategies Sampling

Deals with the question of who will participate in our research project and whether they are a representative group.

Sampling Considerations and Basic Research Strategies Research Strategies

The main strategies used by researchers are the:

The main strategies used by researchers are the: Single strata approach

The main strategies used by researchers are the: Single strata approach Cross sectional approach

The main strategies used by researchers are the: Single strata approach Cross sectional approach Longitudinal approach

Sampling Considerations and Basic Research Strategies Population

The complete set of individuals or events that we want to represent.

Sampling Considerations and Basic Research Strategies Sample

The group that we select to represent the population is called the sample.

Random sample

Random sample Every member of the population has an equal likelihood

of being included in the sample.

Sampling Considerations and Basic Research Strategies Random sample

Every member of the population has an equal likelihood of being included in the sample.

Random sampling without replacement

Random sampling without replacement The member of population chosen for the sample is not

eligible to be selected again once returned to the population.

Random sampling with replacement Member of population chosen for sample is eligible to be

selected again once returned to the population.

Random sampling with replacement Member of population chosen for sample is eligible to be

selected again once returned to the population. Stratified random sampling

eligible to be selected again once returned to the population. Random sampling with replacement

Member of population chosen for sample is eligible to be selected again once returned to the population.

Stratified random sampling Involves dividing the population into subpopulations or strata

and then drawing a random sample from one or more of these strata.

Sampling Considerations and Basic Research Strategies Basic Research Strategies

Single-strata approach

Single-strata approach Seeks to acquire data from a single, specified segment

of the population.

of the population. Cross-sectional research

Involves the comparison of two or more groups of participants during the same, rather limited, time span.

Longitudinal research

Longitudinal research Involves acquiring data from a single group of

participants over a long period of time.

Chapter Five: Nonexperimental Methods II: Ex Post Facto Studies, Surveys and Questionnaires,...

Documents

Transcript of Chapter Five: Nonexperimental Methods II: Ex Post Facto Studies, Surveys and Questionnaires,...

Nonexperimental research design

Questionnaires - Petronect

Toward a New Classification of Nonexperimental Quantitative ...

Experimental and Nonexperimental Research Design

© 2011 Pearson Prentice Hall, Salkind. Nonexperimental Research: Qualitative Methods.

Climate Questionnaires

Subjective questionnaires

Questionnaires Lecture11

Questionnaires TnS

Audience Questionnaires

Chapter 12 Survey Research. Nonexperimental method using interviews or questionnaires to assess attitudes, activities, opinions, or beliefs Surveys often.

Questionnaires ANKIETA „ Questionnaires about our project ”

Mediation in Experimental and Nonexperimental Studies: New ...

Jeopardy Expost Facto

The Role and What Research Is and Isn Importance of Research · nonexperimental research. ! List the three (3) different types of nonexperimental research designs. ! List the two

Choosing Research Designs II Nonexperimental Methods.

College Questionnaires

MM Questionnaires

Additional Questionnaires

Toward a New Classification of Nonexperimental Quantitative