LEARNING OUTCOMES. -...

21
Chapter 10: Experimentation and Validity A. LEARNING OUTCOMES. After studying this chapter students should be able to: Describe four categories of scientific inference. Explain the concepts of construct validity, statistical conclusion validity, internal validity, and external validity. Provide an example to illustrate these types of validity. Define and illustrate seven basic threats to internal validity. Identify the threats that properly conducted experiments control, and explain why they are controlled. Describe how demand characteristics, experimenter expectancy effects, and placebo effects can distort research results, and how experimenters try to minimize these problems. Explain and illustrate the purpose of yoked control groups. Discuss how researchers use pilot studies, manipulation checks, and debriefing to improve the validity of an experiment. Identify and illustrate different types of replications. Explain the importance of conceptual replications and the strategy of replication and extension. Describe why factorial designs are well suited for examining external validity. B.KEYWORDS Attrition (subject loss) Manipulation check Ceiling effect Masking (blinding) Complete (full) replication Maturation Conceptual replication Partial replication Construct validity Pilot study Debriefing Demand characteristics Placebo control group Placebo effect Differential attrition Quasi-experiment Direct (exact) replication Randomized control trial Double-blind procedure Regression to the mean 111

Transcript of LEARNING OUTCOMES. -...

Page 1: LEARNING OUTCOMES. - bcs.worthpublishers.combcs.worthpublishers.com/WebPub/Psychology/.../IRM/…  · Web viewStudents in my experimental psychology class are required to ... McConnell’s

Chapter 10: Experimentation and Validity

A. LEARNING OUTCOMES. After studying this chapter students should be able to: Describe four categories of scientific inference. Explain the concepts of construct validity, statistical conclusion validity, internal validity, and

external validity. Provide an example to illustrate these types of validity. Define and illustrate seven basic threats to internal validity. Identify the threats that properly conducted experiments control, and explain why they are

controlled. Describe how demand characteristics, experimenter expectancy effects, and placebo effects can

distort research results, and how experimenters try to minimize these problems. Explain and illustrate the purpose of yoked control groups. Discuss how researchers use pilot studies, manipulation checks, and debriefing to improve the

validity of an experiment. Identify and illustrate different types of replications. Explain the importance of conceptual replications and the strategy of replication and extension. Describe why factorial designs are well suited for examining external validity.

B. KEYWORDSAttrition (subject loss) Manipulation checkCeiling effect Masking (blinding)Complete (full) replication MaturationConceptual replication Partial replicationConstruct validity Pilot studyDebriefingDemand characteristics

Placebo control groupPlacebo effect

Differential attrition Quasi-experimentDirect (exact) replication Randomized control trialDouble-blind procedure Regression to the meanEcological validity ReplicationExperimenter expectancy effects Replication and extensionExternal validity SelectionFloor effect SensitivityHistory Statistical conclusion validityInstrumentation TestingInternal validity Wait-list control group

Yoked control group

111

Page 2: LEARNING OUTCOMES. - bcs.worthpublishers.combcs.worthpublishers.com/WebPub/Psychology/.../IRM/…  · Web viewStudents in my experimental psychology class are required to ... McConnell’s

112 CHAPTER 10: Experimentation and Validity

C. BRIEF CHAPTER OUTLINE

I. Critical Thinking, Inference, and ValidityA. Categories of InferenceB. Inference in Tryon’s Selective Breeding Project

II. Types of ValidityA. Construct ValidityB. Statistical Conclusion ValidityC. Internal ValidityD. External Validity

III. Basic Threats to Internal ValidityA. Seven Sources of ThreatB. How Experiments Address These Threats

IV. Other Issues Concerning Experimental ControlA. Demand CharacteristicsB. Experimenter Expectancy EffectsC. Placebo Effects and Placebo Control GroupsD. Yoked Control GroupsE. Ceiling and Floor EffectsF. Pilot Studies, Manipulation Checks, and Debriefing

V. Strategies for Replicating ResearchA. Direct ReplicationB. Conceptual ReplicationC. Replication and ExtensionD. Factorial Designs and External Validity

D. EXTENDED CHAPTER OUTLINE*Much of this summary is taken verbatim from the text.

IntroductionThis chapter highlights the importance of critical thinking in research and how students (and other

consumers of research) must evaluate all studies with a skeptical eye. At the end of the chapter students should appreciate the value of skepticism in evaluating research.

Page 3: LEARNING OUTCOMES. - bcs.worthpublishers.combcs.worthpublishers.com/WebPub/Psychology/.../IRM/…  · Web viewStudents in my experimental psychology class are required to ... McConnell’s

CHAPTER 10: Experimentation and Validity 113

Part I: Critical Thinking, Inference, and ValidityA. Categories of inference. Scientists make several types of inference and the validity of each must be

evaluated. Inferences about constructs are those in which one discusses his or her research in the context of the factors examined. Having statistical inference means that the results obtained from a study are more probabilistically due to a true relationship between the variables rather than some chance event. Causal inferences are those that make a statement of causality between two variables. Finally, inferences about generalizability are those that suggest that the research results are not limited to the specific study alone, but to other situations as well. In research, validity applies to the inferences about the study and its findings, not to the studies or findings themselves.

B. Inferences about Tryon’s selective breeding project. In 1927 psychologist Robert Tryon started a research project that began with him dividing rats into “maze-bright” and “maze-dull” groups based on the rats’ ability to learn a maze. For many years thereafter the bright rats were bred with bright rats and the dull rats were bred with other dull rats. At the end of his 10-year study Tryon found that, on average, the dull rats made many more errors when learning a maze than did the bright rats. Tryon did not directly infer that his work proved intelligence to be hereditary, but the results were used by others to support that idea.

Part II: Types of ValidityThis section describes how various types of validity apply to Tryon’s (1940) and Elliot et al.’s (2007)

research. (Elliot et al.’s work was described in Chapter 8. Briefly, it suggests a relationship between color and academic performance.)

A. Construct validity is the issue of whether the constructs that researchers claim to be studying are, in fact, the constructs that they truly are manipulating and measuring. In research, constructs are based on operational definitions that are not perfect approximations of a concept. Construct validity has to do with the validity of the leap from the operational level to the conceptual level. a. In Tryon’s study one must examine whether selective breeding constitutes a valid operational

definition of heredity, and whether the number of errors is a good way to operationalize maze learning. Also, one can argue that any number of constructs could be responsible for maze learning performance (e.g., inquisitiveness, vision, motivation, fear). The key question is therefore what does a maze performance really measure. To conclude that it was cognitive ability one must be able to rule out the other possible explanations.

b. Similarly, in Elliot et al.’s study one must ask whether exposing students to a red, green, or black code number on a booklet is a good manipulation of color, as well as whether anagram and arithmetic performance are good measures of achievement.

B. Statistical conclusion validity concerns the proper statistical treatment of data and the soundness of researchers’ statistical conclusions. In other words, is the way in which the data were analyzed appropriate for the data collected? a. Tryon was interested in statistics and he painstakingly analyzed the data to make statistical

inferences. That said, his work is said to have high statistical conclusion validity.b. Elliott and colleagues used a pretest to examine anagram-solving ability as well as participants’

self-reported SAT scores. These prerequisites were used to provide evidence that the treatment

Page 4: LEARNING OUTCOMES. - bcs.worthpublishers.combcs.worthpublishers.com/WebPub/Psychology/.../IRM/…  · Web viewStudents in my experimental psychology class are required to ... McConnell’s

114 CHAPTER 10: Experimentation and Validity

conditions did not differ due to factors other than color. This safeguard enhances the statistical conclusion validity of the results.

C. Internal validity is the degree to which a researcher is certain that a particular variable had a causal effect on another. Those who critically analyze research must ask whether other variables could have been the causal factor, and, if so, whether they were controlled in such a way as to ensure they are not potential alternative explanations. a. In Tryon’s study one must question whether it was selective breeding, or some other factor, that

affected maze performance. Since Tryon did control for many confounding factors his results are said to have high internal validity.

b. Elliot et al. also worked to control many environmental factors (e.g., the room in which subjects were tested, the size of the room, the comfort of the chairs) in their study so that the only potential explanation for achievement was color.

D. External validity is the degree to which a study’s results can be generalized to other situations than the one in which they were obtained. Generalizations may be made across populations, across settings or across species. a. Ecological validity has to do with how well the responses obtained in a research study

generalize to behavior in the natural setting. Experiments are notoriously criticized for not having high ecological validity, but some experiments—particularly field experiments—can be very ecologically valid.

b. Establishing generalizability. A single study does not provide support for external validity. Instead, external validity is something that accrues over time as the results are replicated. Replication is the process of repeating a study (directly or conceptually) to determine whether the original results are upheld.

Part III: Basic Threats to Internal ValidityA. Seven sources of threat and,

Page 5: LEARNING OUTCOMES. - bcs.worthpublishers.combcs.worthpublishers.com/WebPub/Psychology/.../IRM/…  · Web viewStudents in my experimental psychology class are required to ... McConnell’s

CHAPTER 10: Experimentation and Validity 115

B. How experiments address these threats. Experiments enable researchers to investigate cause and effect relations between variables. In

experiments, potential confounding variables are controlled so as to enable such statements. Potential sources of threat to the internal validity of a study, as well as ways experiments control these threats are as follows:

Source of Threat How Threat Is Addressed

a. History refers to events that occur while a study is being conducted and that are not a part of the experimental manipulation.

Controlled laboratory environment Short duration of participation Careful execution Random assignment

b. Maturation is the way in which people naturally change over the course of a study, independent of their participation in it.

Random assignment

c. Testing has to do with whether the act of measuring one’s responses affects his or her behavior on subsequent measures.

Pretests are given to all participants in each treatment

d. Instrumentation refers to changes in a measuring instrument during the course of data collection.

The same instrument is used to measure all participants in each treatment

e. Regression to the mean refers to the idea that extreme high scores and extreme low scores may be a fluke and that when tested again scores that are more reflective of that person’s behavior will be exhibited.

Inclusion criteria are not based on extreme pretest scores

f. Attrition occurs when participants fail to complete a study and can occur for many reasons.

If attrition occurs the experimenter examines why it occurs to understand whether it systematically varies along with the levels of the independent variable.

g. Selection is when the participants in each treatment condition, at the outset of the study, already differ from one another on some dimension.

Create multiple conditions Use random assignment

Part IV: Other Issues Concerning Experimental ControlIn addition to controlling for the threats to internal validity as described above, good researchers

will also anticipate other potential problems that may become confounding variables.

A. Demand characteristics are cues that affect a participant’s belief about the purpose of a study such as the research setting, the experimenter’s behavior, and the nature of the experimental tasks. Demand characteristics can lead participants to behave in a way that they think is in line with the researcher’s hypothesis, leading to a potential alternative explanation if the hypothesis is supported.

Page 6: LEARNING OUTCOMES. - bcs.worthpublishers.combcs.worthpublishers.com/WebPub/Psychology/.../IRM/…  · Web viewStudents in my experimental psychology class are required to ... McConnell’s

116 CHAPTER 10: Experimentation and Validity

a. An example of demand characteristics. Orne and Scheibe (1964) conducted a study in which they manipulated demand characteristics such that they were highly salient in the experimental group but not in the control group. The results of this study were that those in the experimental group reported behaviors consistent with the demand characteristics presented to them whereas the control group did not. These results support the idea that participants do behave according to demand characteristics.

b. Demand characteristics are addressed in the following ways:i. Suspicion probes are conversational strategies conducted during the debriefing process

that enable a researcher to determine whether demand characteristics were present. This strategy requires good rapport between the researcher and the participants.

ii. Increasing the psychological realism of an experiment gets participants more involved in the situation and to behave realistically.

iii. Pilot testing!!iv. Use dependent measures that are unobtrusive.v. Avoid within-subjects designs when exposure to multiple levels of the independent

variable (IV) is likely to lead participants to guess the hypothesis. vi. Analyze the results of participants to who versus do not claim to know the hypothesis.

vii. Manipulate participants’ knowledge of the hypothesis to rule it out as a factor affecting behavior.

viii. The red herring technique is one in which participants’ attention is diverted from the real issue. Because it involves an added layer of deception, ethical considerations must be made.

B. Experimenter expectancy effects are behaviors that researchers exhibit that influence their participants to respond in a manner consistent with the study’s hypothesis. These behaviors are usually unintentional and occur when the researcher knows the hypothesis or nature of the study. To address experimenter expectancy effects researchers should be rigorously trained. Additionally, one may opt to automate participant instructions, procedures for presenting tasks, and the collecting of data. A third approach is to make the experimenter blind, or unaware of the hypothesis being tested and/or the condition in which the participant has been assigned.

C. Placebo effects and placebo control groups. The placebo effect is a person’s expectation about how a treatment will affect them, which in turn alters their behavior according to this expectation. To reduce or eliminate a placebo effect some experiments create a placebo control group. This group is led to believe they have received a treatment but, in fact, have not. For example, in drug studies the placebo control group receives a treatment that does not contain the active ingredient, in other words, a placebo (e.g., a sugar pill or a saline injection). In a double-blind procedure, neither the experimenter nor the participant knows whether the participant is receiving the treatment or the placebo. In a single-blind study either the experimenter or the participant is unaware of who is receiving which condition.

D. Yoked control groups are those in which each control group member is procedurally linked to a particular experimental group member whose behavior will determine who both of them are treated. The assignment of a yoked control to an experimental participant is usually done randomly. Yoked control groups are usually created when the procedures applied to each experimental group member is dependent upon the participant’s own responses.

Page 7: LEARNING OUTCOMES. - bcs.worthpublishers.combcs.worthpublishers.com/WebPub/Psychology/.../IRM/…  · Web viewStudents in my experimental psychology class are required to ... McConnell’s

CHAPTER 10: Experimentation and Validity 117

E. Ceiling and floor effects. In order for a dependent variable it must be able to detect differences in behavior. A ceiling effect occurs when a measure produces high scores from all participants. In contrast, a floor effect is when all participants score very low regardless of the treatment in which they’re engaged in. Ideally, measures will be sensitive, or able to detect an effect that is actually present.

F. Pilot studies, manipulation checks, and debriefing. A pilot study is a trial run of an experiment that provides information to the researcher about what things in an experiment are working and what things may need to be tweaked. As such, the data collected in a pilot study are not included in the study’s final analysis. Manipulation checks are measures that assess whether the procedures used to manipulate a variable have, in fact, done so adequately. Manipulation checks often occur during the pilot study phase as a self-report of the participant’s experience in the study. Again, these data can help the experimenter decide whether the variable needs to be adjusted. Participants are debriefed after they end their participation in the study. This concept is described more fully in Chapter 3.

Part V: Strategies for Replicating ResearchResearch may be replicated by the scientist who originally conducted it or by another scientist.

Some replications are full replications in that the replication includes all of the elements of the original study; a partial replication, on the other hand, only includes some of the original conditions. This section describes the types of replications that may be full or partial, as well as their advantages and disadvantages.

A. In a direct replication, researchers follow the procedures used in the original study as closely as possible. Most importantly, the variables are operationalized the same way as in the original study. When direct replications yield the same results as the original study they provide support for the idea that the results of the original study were not a fluke.

B. A conceptual replication examines the same question investigated in the original study but operationalizes the independent variable, dependent variable, or both, differently. Conceptual replications help enhance the external validity of results because they demonstrate that they are observed under a variety of conditions.

C. A replication and extension is a replication that adds a new design element to the original study. The replication can be full or partial, or direct or conceptual. For example, sometimes another level of the independent variable is added to the existing one. This enables one to determine nonlinear relations. Other times a new independent variable is added. This can allow a researcher to determine whether another experimental level creates different behavior than not only the control group, but also the original experimental group. In all cases, extending an experiment enhances its external validity. Replication and extension also enables one to identify interactions between variables.

D. Factorial designs and external validity. Factorial designs are particularly well suited to assess the external validity of research results.

Page 8: LEARNING OUTCOMES. - bcs.worthpublishers.combcs.worthpublishers.com/WebPub/Psychology/.../IRM/…  · Web viewStudents in my experimental psychology class are required to ... McConnell’s

118 CHAPTER 10: Experimentation and Validity

E. LECTURE AND CLASSROOM ENHANCEMENTS

PART I: Critical Thinking, Inference, and Validity

A. Lecture/Discussion Topics Can memories be transferred? In 1962, James McConnell published a paper in which he claimed

that worms that ate other worms that had already learned a task, performed significantly better at the same task when they were compared to worms that did not cannibalize their counterparts. This study was referred to in Dean Koontz’ book, Phantoms, as well as in the motion picture of the same name, as being scientific truth. A quick search of the Internet suggests that the idea that consuming “smart” animals improves memory is still being perpetuated. This begs the question, “Can memories be transferred via cannibalizing learned animals?” The fact that many scientists have failed to replicate McConnell’s work suggests that the short answer is “Probably not.” After all, if it were true wouldn’t farmers be spending big bucks to train their cattle and poultry before sending them to slaughter in order to make their product more competitive? The point to illustrate here is that all research must be evaluated skeptically. Even if McConnell’s study was carefully executed in such a way as to enable causal inference, one must ask whether its results can be used to make generalizations and whether the way in which memory is operationalized in his study is equal to the concept of memory in everyday life.

B. Classroom Exercise Identifying and distinguishing between categories of inference. Ask each student to bring a peer-

reviewed, empirical research paper to class. In class, divide the students into groups of 3 or 4 members and ask each group to select one paper to discuss. The discussion should focus on finding evidence in the article to support its ability to draw (1) inferences from the constructs it examines, (2) statistical inferences, (3) causal inferences, and (4) inferences about generalizability. Based on their assessment, what do they conclude about the validity of the article and the claims it makes?

C. Web Resources Skepticism online. http://www.snopes.com is a website devoted to evaluating claims. Snopes uses

critical thinking and analysis to determine whether popular “truths” are credible.

D. Additional References McConnell’s memory transfer study: Fact or fiction?

McConnell, J. V. (1962). Memory transfer through cannibalism in planarians. Journal of Neuropsychiatry, 3, 42–48.

Page 9: LEARNING OUTCOMES. - bcs.worthpublishers.combcs.worthpublishers.com/WebPub/Psychology/.../IRM/…  · Web viewStudents in my experimental psychology class are required to ... McConnell’s

CHAPTER 10: Experimentation and Validity 119

PART II: Types of Validity

A. Lecture/Discussion Topics Construct validity and the DSM. Psychiatric disorders are typically diagnosed using criteria

established by the Diagnostic and Statistical Manual (DSM), the latest edition of which, DSM-V, was released this year. For each disorder, the DSM outlines the criteria required for a person to be diagnosed with the disorder. Through this operationism the DSM provides a construct for each psychiatric disorder. That the DSM has been revised five times suggests that the way in which psychiatric disorders are conceptualized has changed over time.

Do qualitative studies enable statistical conclusion validity? As described in Chapter 4, research can be quantitative or qualitative in nature. In quantitative research, behaviors are described numerically thus enabling statistical analyses of the results. In qualitative research behaviors are not reduced to numerical values; instead, the results are described narratively. This begs the question “Can statistical conclusion validity be drawn from qualitative research?” According to the website socialresearchmethods.net, the answer is “Yes.” They suggest that “Conclusion validity is the degree to which conclusions we reach about relationships in our data are reasonable.”

B. Classroom Exercise Evaluating validity in research. For this exercise, have students work in groups to identify evidence

in research articles that suggest: (1) construct validity, (2) statistical conclusion validity, (3) internal validity, and (4) external validity. They may bring in their own choice of peer-reviewed empirical articles or you may distribute a few of your choosing among the class. Ask the students to provide the rationale for their assessment of each type of validity. If they find any type of validity to be low, ask them to identify what may strengthen it.

C. Web Resource Construct validity in counseling. http://horan.asu.edu/d-cv-apa.htm

D. Additional References Evaluating validity in research.

Green, L. W., & Glasgow, R. E. (2006). Evaluating the relevance, generalization, and applicability of research issues in external validation and translation methodology. Evaluation & the Health Professions, 29(1), 126–153.

Hultsch, D. F., & Hickey, T. (2009). External validity in the study of human development: Theoretical and methodological issues. Human Development, 21(2), 76–-91.

Pimontel, M. A., Reinlieb, M. E., Johnert, L. C., Garcon, E., Sneed, J. R., & Roose, S. P. (2013). The external validity of MRI defined vascular depression. ‐ International journal of Geriatric Psychiatry. 21(3), S66–S67.

Page 10: LEARNING OUTCOMES. - bcs.worthpublishers.combcs.worthpublishers.com/WebPub/Psychology/.../IRM/…  · Web viewStudents in my experimental psychology class are required to ... McConnell’s

120 CHAPTER 10: Experimentation and Validity

PART III: Basic Threats to Internal Validity

A. Lecture/Discussion Topics Instrumentation refers to the potential for an instrument to affect the accuracy, or validity, of its

measure. When I think of instrumentation I think of an analog fishing scale. To obtain its weight, the fish is placed on a hook that is attached to a spring that is attached to the scale. The accuracy of the scale depends upon the integrity of the spring. When a spring is stretched repeatedly it eventually loses some elasticity and affects the measurement of the fish. For example, an 8-pound fish may weigh in at 8 pounds with a new scale, but years later another 8-pound fish yields a different weight with the same scale. The point is that instruments, mechanical or human, can change over time. In research one must exert great care to ensure that measurements are valid and reliable.

Attrition is not necessarily a bad thing. Attrition is certainly a threat to internal validity. The results of a research study could be partly explained because participants in one treatment withdrew from the study at a higher rate than participants in another treatment. For example, let’s say a researcher wanted to examine whether 6-week and 12-week smoking cessation programs were equally effective in helping people quit smoking by measuring whether participants of each were “smoke-free” one year after the start of the study. The researcher may find that there is no difference in the number of smokers and nonsmokers in the 6-week and 12-week groups and conclude that the length of treatment is not an important factor in helping someone quit smoking. However, the researcher may have also noticed that although nearly everyone in the 6-week group completed the program, in the 12-week program many participants withdrew by the seventh week. It is possible, then, that a longer term program could increase the likelihood that one will be smoke-free at one year, but only if people continue to see the program to the end. These results could provide valuable information to researchers about the design of smoking cessation programs. That attrition can provide this type of information shows that attrition is not necessarily a bad thing.

B. Classroom Exercises In this section students learn of the threats to internal validity. The book describes two research

studies and demonstrates how the various types of internal validity were compromised or controlled for. Have students work in groups to develop ways to prevent each threat in an experiment and what they might do if a threat emerges as a true problem. This exercise will help students not only think about sound research design, but they will also understand the importance of being proactive in research. The ability to identify potential pitfalls and solutions for them before they occur is certainly a desired quality among scientists.

C. Web Resources Studies with poor internal validity. This website provides citations to several studies whose

internal validity has been deemed “poor” based on the authors’ systematic evaluation. A brief explanation of what issues might affect each study’s internal validity is provided next to the reference. https://www.ncbi.nlm.nih.gov/books/NBK44516/

Internal and external validity defined. http://www.edpsycinteractive.org/topics/intro/valdgn.html

Page 11: LEARNING OUTCOMES. - bcs.worthpublishers.combcs.worthpublishers.com/WebPub/Psychology/.../IRM/…  · Web viewStudents in my experimental psychology class are required to ... McConnell’s

CHAPTER 10: Experimentation and Validity 121

D. Additional References On various threats to internal validity.

Marcellus, L. (2004). Are we missing anything? Pursuing research on attrition. CJNR (Canadian Journal of Nursing Research), 36(3), 82–98.

Streiner D. L. 2001. Regression toward the mean: Its etiology, diagnosis, and treatment. Canadian Journal of Psychiatry, 46, 72–76.

PART IV: Other Issues Concerning Experimental Control

A. Lecture/Discussion Topics Yoked control groups. In the late 1990s a paper was published that suggested that animals that ran

on running wheels had more new neurons in their hippocampus compared to animals that swam in a pool. Furthermore, the rate of neurogenesis appeared to be comparable in swimmers compared to control animals. Our laboratory wondered if stress might explain why exercise in the form of running, but not swimming, affected the birth of new neurons. We decided to perform a study in which rats that voluntarily ran on wheels were matched to rats that were forced to run on a treadmill. Each night we would record the distance run by a voluntary runner and its yoked cohort would be forced to run the same distance. In this experimental design we were able to control for the type and amount of exercise performed so as to isolate the factor of voluntary participation on neurogenesis. The experiment was an interesting experience in yoked control groups. Rats are notorious for running great distances in a day’s time, and one rat eventually ran seven miles in a 24-hour period. Needless to say its yoked control exhibited behaviors associated with stress when it was forced to run a full seven miles in one stretch. (With a pace of 1 mph so was I!)

The importance of pilot studies. Students in my experimental psychology class are required to conduct a group experiment. This project is a major part of the course and can make or break their final grade. The goal of this project is for students to take what they’ve learned in class and apply it to a research project. Throughout the course I emphasize the important of “piloting” their study before they exert a great deal of effort to recruit participants, analyze data, etc. Most groups of students who do pilot their work earn a higher mark on their project, not because they did “extra work,” but because in piloting the study they were able to identify flaws in the design and make changes accordingly. This extra step does take time, but in the long run it really pays off. Piloting a study is not just something recommended to novice researchers; seasoned scientists also benefit greatly from performing pilot studies.

B. Classroom Exercise If students are working on research projects ask them to reflect upon their project and discuss

whether demand characteristics and/or experimenter expectancy effects could potentially affect their results. Next have them identify ways to minimize each of these.

Page 12: LEARNING OUTCOMES. - bcs.worthpublishers.combcs.worthpublishers.com/WebPub/Psychology/.../IRM/…  · Web viewStudents in my experimental psychology class are required to ... McConnell’s

122 CHAPTER 10: Experimentation and Validity

C. Web Resource Demand characteristics. These links describe demand characteristics and their effect on

psychological research.o http://plato.stanford.edu/entries/mental-imagery/demand-characteristics-problem.html

o http://psychosomething.wordpress.com/2011/10/28/what-affect-can-demand- characteristics-have-on-psychological-research/

This is a really brief and interesting video summary of placebo controls and the profound effect they may have on behavior. http://infographicmovie.blogspot.com/2011/02/strange-powers-of-placebo-effect.html

D. Additional References On demand characteristics.

McCambridge, J., de Bruin, M., & Witton, J. (2012). The effects of demand characteristics on research participant behaviours in non-laboratory settings: A systematic review. PLOS ONE, 7(6), e39116.

PART V: Strategies for Replicating Research

A. Lecture/Discussion Topics Can memories be transferred? McConnell’s memory transfer studies led to subsequent

investigations, by him as well as by others, on whether learning can be transferred from one organism to another. In 1965 Babitch and colleagues performed a conceptual replication of McConnell’s original study. They used rats, not tapeworms, and instead of transferring the memory through cannibalism, they injected their animals with brain matter from the learned animals. Their findings corroborated McConnell’s results. However, the following year a group of researchers submitted a paper to Science stating that none of them was able to replicate Babitch et al.’s work. (A more detailed description of this story can be found in: Collins, H. M., & Pinch, T. (1998). The Golem: What You Should Know About Science. Cambridge University Press.)

Replications with extensions. When a researcher performs a replication with extension, he or she adds a new design element to the original study. The text describes how this can be achieved by adding another level of an independent variable, another independent variable, or a new dependent variable. However, can a replication with extension also be one in which something about the experimental design is deleted or removed? For example, there are many surveys that began with many items, but through replications with deletions now have fewer items. Discuss with the class.

B. Classroom Exercise Developing replications. Provide groups of students with the following abstracts and ask them to

generate an idea for (1) a conceptual replication, and (2) replication with extension. Have them indicate how their replication will add significantly to the existing body of literature.

Page 13: LEARNING OUTCOMES. - bcs.worthpublishers.combcs.worthpublishers.com/WebPub/Psychology/.../IRM/…  · Web viewStudents in my experimental psychology class are required to ... McConnell’s

CHAPTER 10: Experimentation and Validity 123

o Baruch, D. E., Swain, R. A., & Helmstetter, F. J. (2004). Effects of exercise on Pavlovian fear conditioning. Behavioral Neuroscience, 118, 1123–1127. Abstract: Exercise promotes multiple changes in hippocampal morphology and should, as a result, alter behavioral function. The present experiment investigated the effect of exercise on learning using contextual and auditory Pavlovian fear conditioning. Rats remained inactive or voluntarily exercised (VX) for 30 days, after which they received auditory-cued fear conditioning. Twenty-four hours later, rats were tested for learning of the contextual and auditory conditional responses. No differences in freezing behavior to the discrete auditory cue were observed during the training or testing sessions. However, VX rats did freeze significantly more compared to controls when tested in the training context 24 hr. after exposure to shock. The enhancement of contextual fear conditioning provides further evidence that exercise alters hippocampal function and learning

o Drews, F. A., Yazdani, H., Godfrey, C. N., Cooper, J. M., & Strayer, D. L. (2009). Text messaging during simulated driving. Human Factors: The Journal of the Human Factors and Ergonomics Society, 51(5), 762–770. Abstract: Objective: This research aims to identify the impact of text messaging on simulated driving performance. Background: In the past decade, a number of on-road, epidemiological, and simulator-based studies reported the negative impact of talking on a cell phone on driving behavior. However, the impact of text messaging on simulated driving performance is still not fully understood. Method: Forty participants engaged in both a single task (driving) and a dual task (driving and text messaging) in a high-fidelity driving simulator. Results: Analysis of driving performance revealed that participants in the dual-task condition responded more slowly to the onset of braking lights and showed impairments in forward and lateral control compared with a driving-only condition. Moreover, text-messaging drivers were involved in more crashes than drivers not engaged in text messaging. Conclusion: Text messaging while driving has a negative impact on simulated driving performance. This negative impact appears to exceed the impact of conversing on a cell phone while driving. Application: The results increase our understanding of driver distraction and have potential implications for public safety and device development.

C. Web Resource Replications in the news.

o http://www.psychologicalscience.org/index.php/news/releases/initiative-on-research- replication.html

o http://www.guardian.co.uk/science/sifting-the-evidence/2013/may/15/psychology- registered-replication-reports-reliability

D. Film Suggestion Replicating Milgram’s obedience study. In the television program, The Heist, the Milgram

obedience experiment is re-created by Darren Brown. Although the exact methodology is not described, what is shown in this film suggests that it was a direct replication of the famous study.

Brown, D., O’Connor, A., Owen, A., Mills, S., Sandhu, R., & Nyman, A. (Producers). Caron, B. (Director). (2006). The Heist. London: Objective Productions.

Page 14: LEARNING OUTCOMES. - bcs.worthpublishers.combcs.worthpublishers.com/WebPub/Psychology/.../IRM/…  · Web viewStudents in my experimental psychology class are required to ... McConnell’s

124 CHAPTER 10: Experimentation and Validity

E. Additional References Replications of famous experiments.

Burger, J. (2009b). Replicating Milgram: Would people still obey today? American Psychologist, 64, 1–11. doi: 10.1037/a0010932

Reicher, S. D., & Haslam, S. A. (2006). Rethinking the psychology of tyranny: The BBC prison experiment. British Journal of Social Psychology, 45, 1–40. doi: 10.1348/014466605X48998