~ 1 ~

Second Language Research- Mackey and Gass

Research: The systematic investigation into and study of materials and sources in order to establish facts and reach new conclusions. In other words, research refers to finding answers to questions. Types of Research: Quantitative, Qualitative and Mixed method. 1) Quantitative: Research that gathers numeric data through controlled procedures and analyses to answer predetermined questions or test hypothesis.

2) Qualitative: A generic term for a variety of research approaches that generally gather non-numeric data and deals with subjective interpretation of data.

3) Mixed Method: Combining both qualitative and quantitative research in s single study in order to reach a better understanding of phenomena

Three main parameters that distinguish between research types:

a) Type of data (Quantitative/Qualitative)

b) Method of Analysis (Interpretive/Statistical)

c) Manner of data collection (Experimental/Non-experimental) Typical Format of Quantitative research paper:

~ 2 ~

Title- Abstract-Introduction-Methodology-Findings/Results – Discussion-Conclusion-References-Apendices Feasibility: is whether or not it will be possible to obtain data necessary to answer research question. Research Question: refers to the questions for which answers are being sought. Research Hypothesis: Expresses what the researcher expects to be the result of the study. Replication: refers to the fact that to what extent we can repeat our research and obtain same results. The higher replicability means that our research is more valid. Participants should be informed about the 'Privacy'. This includes information about 'anonymity' and 'confidentiality'. This means that researcher can not publish participants' names and personal information without their permission. An 'informed consent' should be signed by researcher and participants. This 'disclosure' tells them what is the research about and what should be done by either researcher or participants. Finally, participants' physical and psychological safety during research should be guaranteed and the participants can leave research whenever they want without any penalty. Two main benefits of Pilot testing 1-Reveals subtle flaws in the design and implementations of the study. 2-It may result in collection of data that are essential to main study. - Some common data collection manners. These are based upon research question. Researchers may decide to use all or some of them. -Acceptability Judgments: is a task in which learners are asked whether a particular sentence is acceptable in second language or not. -Elicited Imitations: A task in which a series of sentences with grammatical errors are presented played for participants with audio tapes and thay have to repeat the 'correct' forms of those sentences. -Magnitude Estimations: -Truth-Value Judgments

~ 3 ~

John said that bill hit himself What does 'himself' refer to and why? a-John b-Bill c-either John or Bill d-Someone else Sentence Interpretation: Researcher wants participants' opinion about a question or statement. For example: consider this sentence "Sally kissed John."Researcher asks who kissed the other first? Sally or John? -Moving Window: In this method that is a poper way to measure participants' reading speed, words are presented on a screen on at a time, with each successive word appearing after a participant indicates that he/she is ready. In other words, they are in control of reading speed. Once a word appears, the previous word disappears. After the entire sentence has appeared, the participant should say whether the sentence is grammatical or not. Eye Tracking: Picture Description: They describe a picture -Spot the Difference: Tell the differences between two pictures. -Jigsaw Task: We divide a body of information into pieces and give each participant a piece of it and they have to interact with eachother in order to solve the puzzle. -Questionnaires: Based on research queston some questionnaires are used in either of the researches Hawthorne Effect: Positive impact of research on participants, simply because they know they are different. Halo Effect: Participants can give either pleasing or deceptive answers to researcher. This depend on the nature of research where it can be boring or interesting. Rater Reliability: depends on two factors. Both of these should be fulfilled if we want to say our measurement has 'rater reliability'. First one is 'inter-rater reliability'. This means that if we give answer sheet of our examinees to two or more raters, they give same score to that paper. For example, both of them give the paper 18 out of 20. On the other hand, we have 'intra-rater reliability'. Here we just deal with one rater. We give one answer sheet to him in different ocassions and compare the scores that he/she

~ 4 ~

gives to the paper. If the scores were same, we say that our measurement yields intra-rater reliability as well. Note that in the concept of reliability we talk about more than one rater, one situation or context, because we seek reliablity which should be attained continously in different situations. -Instrument Reliability: refers to the means by which we obtain our data and evaluate them. Test-Retest, Equivalence of forms and Internal Consistency are three factors that affect this reliablity Test-Retest means that we give one test to one group in two different ocassions. If the scores of both tests are similar, then we have reliability. Equivalence of forms tells us that if you are about to test a group more than once, then your questions should not vary in terms of their difficulty. They both should have equal difficulty. Internal Consistency refers to time that we have difficulties for adminsting a test more than one time. In this situations we use some techniques that reflect reliability of our measurement: - Split half method: we divide our test in halves and give participants both of them. Then we compare the scores of both halves. If a student for example gets 6.5 in one half and 7 in another half, then we have reliability (internal consistency). - Kuder-Richardson 20&21: Used when we have lots of questions. - Cronbach's alpha: used when there is more than one possible answer to questions Purpose of Quantitative Studies is to find relationship between and within variables. This research takes two forms. Associational (Correlational) Research is a type of quantitative research in which we seek the relationship between two variables and if there is any we try to find the degree of it. Experimental Research is another type of quantitative with this difference that in experimental research, we already know that there is a relationship between X and Y. We only want to find that how much of Y will occur if X occurs? Counterbalancing: An experimental design in which the test items or tasks are different for different participants. -Repeated Measure design: We give treatment many times to the group and test them many times.

~ 5 ~

-Factorial Design: used when there are more than one variable and treatments. -Time series design: observation of participants' performance during a determined period of time. -One shot design: When there is no treatment and we just ask questions about topic and record participants' opinions. Meta Analyses: Way of stepping back and comparing what others have done on a particular topic. Subjectivity Vs. Objectivity in Research: In Quan research we should observe our sample objectively. This enables us to gather numerical data for our research. Gathering numericals will pave the way for us to measure them statistically and come to a valid conclusion. On the other hand, in Qual researhes we are less concerned with the issue of generalizablity. In qual research we deal with individuals rather than a group and we try to interpret phenomena subjectively. Features of Qualitative Research: - Rich derscription of phenomena - Natural and holistic representation - Fewer participants - Open ended processes A qualitative research should be: 1- Comprehensive: it should study the phenomena in every perspective. 2- Topic Oriented: the topic of research should be clarified at the end of study. 3- Hypothesis Oriented: research should give researcher the chance to make a hypothesis. 4- Credible: What we report should be reliable and believable. 5- Transferable: we should be able to transform our results from one context to another. 6- Dependable: Research should completely demonstrate the relations of variables. 7- Confirmable: Everythin we report should be based upon firm and valid data. Triangulation: Involves using multiple research techniques and sources of data in order to explore the issue from all feasible perspectives.

~ 6 ~

Different types of Triangulation 1-Theoretical: Using multiple perspectives when analyzing data. 2-Investigator: Using several observer or interviewer. 3-Methodological: Using several measures and methods. Most Common ways for obtaining Qualitative Data: Ethnographies - Case Studies – Observation - Interview Mixed Method Involves using quantitative and qualitative methods in a single study. Benefits: -Greater Validity -Offset -Completeness:it gives comprehensive image of phenomena -Utility: we can use data of both methods in real life. Types of Mixed methods (according to Creswell) -Explanatory – Exploratory - Sequential embedded ------------------------------------------------------------------------------------------------------------------------------------

1- Which of the following is a characteristic of quantitative research? a. obtrusive b. naturalistic c. ungeneralizable d. subjective In obtrusive data collection, the subjects are aware of the fact that they are being studied, which can influence their response or behavior. Examples of obtrusive data collection methods are questionnaires or interviews. Quantitative Research is objective in approach in the sense that it only seeks precise measurements and analysis of target concepts to answer his inquiry. Qualitative Research is primarily subjective in approach as it seeks to understand human behavior and reasons that govern such behavior. Researchers have the tendency to become subjectively immersed in the subject matter in this type of research method. Generalizability is one of the key features for both types of researches.

~ 7 ~

Your goal in conducting quantitative research study is to determine the relationship between one thing [an independent variable] and another [a dependent or outcome variable] within a population. Quantitative research designs are either descriptive [subjects usually measured once] or experimental [subjects measured before and after a treatment]. A descriptive study establishes only associations between variables; an experimental study establishes causality. Quantitative research deals in numbers, logic, and an objective stance. Quantitative research focuses on numeric and unchanging data and detailed, convergent reasoning rather than divergent reasoning [i.e., the generation of a variety of ideas about a research problem in a spontaneous, free-flowing manner]. Characteristics of Quantitative Research: • The data is usually gathered using structured research instruments. • The results are based on larger sample sizes that are representative of the population. • The research study can usually be replicated or repeated, given its high reliability. • Researcher has a clearly defined research question to which objective answers are sought. • All aspects of the study are carefully designed before data is collected. • Data are in the form of numbers and statistics, often arranged in tables, charts, figures, or other non-textual forms. • Project can be used to generalize concepts more widely, predict future results, or investigate causal relationships. • Researcher uses tools, such as questionnaires or computer software, to collect numerical data. 2- Which of the following is TRUE about qualitative research? a. Outcome oriented b. goal-oriented c. verification-oriented d. discovery oriented The research questions addressed by qualitative methods are discovery-oriented, descriptive and exploratory in nature. ... Qualitative research is, in some cases, instrumental to developing an understanding of a phenomenon as a basis for quantitative research. Below are the three key elements that define a qualitative research study and the applied forms each take in the investigation of a research problem.

~ 8 ~

The Design Naturalistic -- refers to studying real-world situations as they unfold naturally; nonmanipulative and noncontrolling; the researcher is open to whatever emerges [i.e., there is a lack of predetermined constraints on findings]. Emergent -- acceptance of adapting inquiry as understanding deepens and/or situations change; the researcher avoids rigid designs that eliminate responding to opportunities to pursue new paths of discovery as they emerge. Purposeful -- cases for study [e.g., people, organizations, communities, cultures, events, critical incidences] are selected because they are “information rich” and illuminative. That is, they offer useful manifestations of the phenomenon of interest; sampling is aimed at insight about the phenomenon, not empirical generalization derived from a sample and applied to a population. The Collection of Data Data -- observations yield a detailed, "thick description" [in-depth understanding]; interviews capture direct quotations about people’s personal perspectives and lived experiences; often derived from carefully conducted case studies and review of material culture. Personal experience and engagement -- researcher has direct contact with and gets close to the people, situation, and phenomenon under investigation; the researcher’s personal experiences and insights are an important part of the inquiry and critical to understanding the phenomenon. Empathic neutrality -- an empathic stance in working with study respondents seeks vicarious understanding without judgment [neutrality] by showing openness, sensitivity, respect, awareness, and responsiveness; in observation, it means being fully present [mindfulness]. Dynamic systems -- there is attention to process; assumes change is ongoing, whether the focus is on an individual, an organization, a community, or an entire culture, therefore, the researcher is mindful of and attentive to system and situational dynamics. The Analysis Unique case orientation -- assumes that each case is special and unique; the first level of analysis is being true to, respecting, and capturing the details of the individual cases being studied; cross-case analysis follows from and depends upon the quality of individual case studies. Inductive analysis -- immersion in the details and specifics of the data to discover

~ 9 ~

important patterns, themes, and inter- relationships; begins by exploring, then confirming findings, guided by analytical principles rather than rules. Holistic perspective -- the whole phenomenon under study is understood as a complex system that is more than the sum of its parts; the focus is on complex interdependencies and system dynamics that cannot be reduced in any meaningful way to linear, cause and effect relationships and/or a few discrete variables. Context sensitive -- places findings in a social, historical, and temporal context; researcher is careful about [even dubious of] the possibility or meaningfulness of generalizations across time and space; emphasizes careful comparative case analyses and extrapolating patterns for possible transferability and adaptation in new settings. Voice, perspective, and reflexivity -- the qualitative methodologist owns and is reflective about her or his own voice and perspective; a credible voice conveys authenticity and trustworthiness; complete objectivity being impossible and pure subjectivity undermining credibility, the researcher's focus reflects a balance between understanding and depicting the world authentically in all its complexity and of being self-analytical, politically aware, and reflexive in consciousness. 3- Where can we find the statement of topic area in a typical research paper? a. Method b. discussion c. Conclusion d. introduction Title: The title should be specific and indicate the problem the research project addresses using keywords that will be helpful in literature reviews in the future. Abstract: The abstract is used by readers to quickly review the overall content of the paper. Journals typically place strict word limits on abstracts, such as 200 words, making them a challenge to write. The abstract should provide a complete synopsis of the research paper and should introduce the topic and the specific research question, provide a statement regarding methodology and should provide a general statement about the results and the findings. Because it is really a summary of the entire research paper, it is often written last. Introduction: The introduction begins by introducing the broad overall topic and providing basic background information. It then narrows down to the specific research question relating to this topic. It provides the purpose and focus for the rest of the paper and sets up the justification for the research. Literature Review: The purpose of the literature review is to describe past important

~ 10 ~

research and it relate it specifically to the research problem. It should be a synthesis of the previous literature and the new idea being researched. The review should examine the major theories related to the topic to date and their contributors. It should include all relevant findings from credible sources, such as academic books and peer-reviewed journal articles. Methods: The methods section will describe the research design and methodology used to complete to the study. The general rule of thumb is that readers should be provided with enough detail to replicate the study. Results: In this section, the results of the analysis are presented. How the results are presented will depend upon whether the research study was quantitative or qualitative in nature. This section should focus only on results that are directly related to the research or the problem. Graphs and tables should only be used when there is too much data to efficiently include it within the text. This section should present the results, but not discuss their significance. Discussion/Conclusion: This section should be a discussion of the results and the implications on the field, as well as other fields. The hypothesis should be answered and validated by the interpretation of the results. This section should also discuss how the results relate to previous research mentioned in the literature review, any cautions about the findings, and potential for future research. References/Bibliography The research paper is not complete without the list of references. This section should be an alphabetized list of all the academic sources of information utilized in the paper. The format of the references will match the format and style used in the paper. Common formats include APA, MLA, Harvard and so forth. 4- Which section of a paper includes verbal description of data, charts, figures, tables, etc.? b. Method b. results c. discussion d. review of literature 5- Which of the following characteristics of research is concerned with the possibility to obtain the data necessary to answer the question? a. Feasibility b. credibility c. conformability d. transferability Feasibility study is used to determine the viability of an idea, such as ensuring a project is legally and technically feasible as well as economically justifiable. It tells us whether a project is worth the investment. Credibility involves establishing that the results of research are credible or believable. Conformability is the degree to which others agree or corroborate with the

~ 11 ~

research findings. Transferability refers to the degree to which the results of qualitative research can be generalized or transferred to other contexts or settings. 6- The researcher can .................. Information about the research if participants are given an opportunity to be debriefed after the study. a. Disclose b. replicate c. withold d. modify In research reports, a disclosure is a statement that reveals the nature of the relationship between the analysts and participants of the research. It also provides other warning-like statements that investors should be aware of. Scientific studies ought to be replicable, meaning that a study should produce the same results if repeated exactly. 7- Which of the following refers to a small-scale trial of the proposed procedures, materials and methods of an activity which might also include coding sheets and analytic choices? a. Treatment b. Experimentation c. Pilot Testing d. Hypothesis Testing In the design of experiments, treatments are applied to experimental units in the treatment group. Experimental research is a study that strictly adheres to a scientific research design. It includes a hypothesis, a variable that can be manipulated by the researcher, and variables that can be measured, calculated and compared. Most importantly, experimental research is completed in a controlled environment. Pilot test, or pilot experiment is a small scale preliminary study conducted in order to evaluate feasibility, time, cost, adverse events, and improve upon the study design prior to performance of a full-scale research project. Hypothesis testing is used to establish whether a research hypothesis extends beyond those individuals examined in a single study. 8- The principle labeled concern for …………. entail that all people should be treated as autonomous agents and that those with diminished autonomy are entitled to special protection. a. Confidentiality b. independence c. beneficence d. anonymity Confidentiality refers to a condition in which the researcher knows the identity of a research subject, but takes steps to protect that identity from being discovered by others. Most human subjects research requires collection of a sign consent agreement from participants, and thus researchers are aware of the identity of their subjects. In such cases,

~ 12 ~

maintaining confidentiality is a key measure to ensure the protection of private information. Anonymity is a condition in which the identity of individual subjects is not known to researchers. Because most human subjects research requires signed documentation of consent, subject anonymity is not as common in human subjects research. Federal law does allow an IRB to waive the requirement for signed consent documents in cases where the collection of that document is the only identifying information linking the subject to the project. Such documentation is most often waived for projects such as online survey that present no more than minimal research to subjects. Beneficence is a concept in research ethics which states that researchers should have the welfare of the research participant as a goal of any clinical trial or other research study. 9- In a(n) ........................ , the participants are asked to decide whether a particular sense is up to the standards of a foreign or second language or not. a. Truth-value judgment b. magnitude estimation c. acceptability judgment d. moving window We often make value judgments without realizing that we are doing so. For example, a teacher who describes a student as “the best I've ever taught — very polite and obedient” is making a value judgment about the qualities (politeness and obedience) that make a student good. magnitude estimation procedure requires subjects to estimate the magnitude of physical stimuli by assigning numerical values proportional to the stimulus magnitude they perceive. Highly reliable judgments can be achieved for a whole range of sensory modalities, such as brightness, loudness, or tactile stimulation. An acceptability judgement is a basic datum for linguistics: an introspective judgement by a (native) speaker of a language concerning the acceptability of a linguistic expression (sentence, form, etc.). A moving window is a way to isolate subsets of a long string of time-dependent measurements, simply by taking the last n time segments and using each segment as an input to a network. 10- The model known as the ............. is basically concerned with what information people use in coming to an understanding of the relationship of words in a sentence. a. Pragmatic study b. competition model c. universal grammar d. ethnographic procedure

~ 13 ~

11- The underlying assumption in ………… is that if a certain sentence is part of person's grammar, it will be relatively easy to repeat. a. Interaction research b. pragmatic analysis c. elicited imitation d. conversation analysis Interaction research, studies the relationship among three or more variables and manifests the causal effect relationship of the three. Pragmatic Analysis is part of the process of extracting information from text. Specifically, it's the portion that focuses on taking a structure set of text and figuring out what the actual meaning was. Elicited imitation is a language sampling procedure in which a child is asked to repeat an utterance that is modeled by the examiner. Conversation analysis is an approach to the study of social interaction and talk-in-interaction that, although rooted in the sociological study of everyday life, has exerted significant influence across the humanities and social sciences including linguistics. 12- Which of the following is useful when a researcher wants not only to rank items in relation to one another, but also to discover how much a certain structure is better than another one? a. Magnitude estimation b. acceptability judgment c. stimulated recall d. introspective report Stimulated recall is a research method that allows the investigation of cognitive processes through inviting participants to recall their concurrent thinking during an event when prompted by a video sequence or some other form of visual recall. Introspection is often associated with consciousness, and introspective reports may well be particularly useful in the study of consciousness. But the present topic is not consciousness or experience or qualia or any particular kind of mental state, as construed by a scientific theory or by folk psychology. 13- The procedure of verbal reporting known as ……… can provide the researcher with information about general approaches to something. a. Self-disclosure b. self-report c. self-revelation d. self-talk 14- In the experiment known as the .............. , words are presented on a screen one at a time, and each successive word appear after a participant indicates that she is ready.

~ 14 ~

a. Moving window b. reaction time c. truth-value judgment d. interactive test 15- Which of the following is FALSE as one of the techniques used in interaction-based research? a. Picture description task b. spot the difference c. jigsaw task d. sentence interpretation 16- Which of the following involves pairs or groups of participants who are asked to come to an agreement on a certain subject ؟ a. Consensus task b. consciousness-raising task c. problem-solving task d. interactive task 17- The main problem with ..............is that no matter what the prompt is, the participant might choose to provide a different form from the one expected. a. Talk aloud b. stimulated narrative c. introspective report d. retrospective report 18- The use of ............. in a questionnaire allows participants to express their thoughts and ideas in their own way. a. Closed item questions b. double barreled items c.open item questions d. loaded items Closed item questions like multiple choices has one fixed and pre-determined answers. A double barreled question (item) asks two questions in one sentence and the respondent can answer only one of the two questions, and cannot indicate which one is being answered. loaded question or complex question is a question that contains a controversial or unjustified assumption. 19- There are a number of caveats in using silent films. Which of the following is LEAST likely to be one of them? a. Film must be culturally neutral b. Learners should be told about the content of film before seeing it. "this will lead to the activation of their background knowledge and prepares them for learning" c. Films must be medium length D. Learners should tell it to someone who has not seen it. 20- The major advantage of a questionnaire is that it can provide us with ……… information from learners in a short period of time.

~ 15 ~

a. Longitudinal b. cross-sectional c. objective d. reliable This means that from a simple questionnaire we can elicit information that reflect longitudinal attitudes of participants. 21- The software called …… can be used to analyze the transcripts through searches and frequency counts. a. DCT b. VARBRUL c. CLAN d. CHILDES . 22- The ......... is used to represent the average rate of agreement for an entire set of scores, accounting for the frequency of both agreements and disagreements by category? a. MANOVA b. Multiple Regression c. Cohen's Kappa d. T-test 23- On a normal curve, we can find about ……… of scores between mean and one standard deviation from it. a. 35 percent b. 50 percent c.85 percent d. 95 percent 24- Which of the following gives us an idea of how close our sample mean is to other samples from the same population? a. Variance b. SD c. SEM d. Range 25- Which of the following is commonly used when the data is obtained in the form of frequencies? a. Chi-Square b. Mann-Whitney c. Pearson correlation d. ANOVA