Validation of the Grammatical Carefulness Scale Using a

Discourse Completion Task and a Reading and Underlining Task

November 22, 2014 84th LET Chubu

Shizuoka University

The handout is available from…

The handout is available from…

Yu TAMURAGraduate School, Nagoya Univ.

yutamura@nagoya-u.jpKunihiro KUSANAGI

Graduate School, Nagoya Univ.JSPS Research Fellow kusanagi@nagoya-u.jp

Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion

Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion

Introduction• What is grammatical carefulness(GC)?

• Why is it needed?

• How can it be used?

Introduction• What is grammatical carefulness (GC)?

• a construct referring to learners’ characteristics

• personal traits reflecting learner’s behaviour and belief in language use

• explains learners’ variance • reflects highly controlled, cautious, and

analytical language use

Introduction• Why is it needed?

• To capture the variance of L2 learners’ inconsistent performance

• To get more information about individual differences focusing on grammatical performance

• To take into account speed accuracy trade-off (Dennis & Evans, 1996; Goldhammer & Kroehne, 2014; van der Linden, 2007, 2009)

Introduction• speed accuracy trade-off

• consistent within a learner • linked to other learners’ factors (e.g., psychological,

behavioural, and meta-cognitive)

Introduction• How can it be used?

• as covariance in psycholinguistic experimental research

• as independent variables in interventional experiments

• in investigating relationship between individual differences and L2 learners’ performance

• as a diagnostic assessment tool in educational settings

Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion

Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion

Background• Grammatical Carefulness Scale (GCS)

• A questionnaire developed by Kusanagi et al. (2014) • 14 items (3 sub-scales) • 7 point Likert scale • written in Japanese • Some validation has been done (Kusanagi et al., 2014)

• factorial validity • content validity • criterion-based validity

Background• Grammatical Carefulness Scale (GCS)

• 3 sub-scales • Lexical-Syntactic Carefulness (LSC) (k = 4)

• e.g., I always notice the wrong use of the words.

• Pragmatic Carefulness (PC) (k = 5) • e.g., I’m likely to think about inconsistent expressions.

• Phonological Carefulness (PHC) (k = 5) • e.g., I’m always careful about correct pronunciation.

Background• Grammatical Carefulness Scale (GCS)

• factorial validity • both exploratory and confirmatory • high reliability (α = .90)

• content validity • judgments by linguistic experts

• criterion-based validity • correlated to analytical belief (Tanaka & Ellis, 2003) more strongly

than empirical belief • correlated to the accuracy score of GJTs • correlated to the accuracy score of C-test and its time to completion

GCS is validated!

Validation never ends.

Background• Grammatical Carefulness Scale (GCS)

• factorial validity • both exploratory and confirmatory • high reliability (α = .90)

• content validity • judgments by linguistic experts

• criterion-based validity • correlated to analytical belief (Tanaka & Ellis, 2003) more strongly

than empirical belief • correlated to the accuracy score of GJTs • correlated to the accuracy score of C-test and its time to completion

Background• Grammatical Carefulness Scale (GCS)

• factorial validity • both exploratory and confirmatory • high reliability (α = .90)

• content validity • judgments by linguistic experts

• criterion-based validity• correlated to analytical belief (Tanaka & Ellis, 2003) more strongly

than empirical belief • correlated to the accuracy score of GJTs • correlated to the accuracy score of C-test and its time to completion

Background• Grammatical Carefulness Scale (GCS)

• 3 sub-scales • Lexical-Syntactic Carefulness (LSC) (k = 4)

• e.g., I always notice the wrong use of the words.

• Pragmatic Carefulness (PC) (k = 5) • e.g., I’m likely to think about inconsistent expressions.

• Phonological Carefulness (PHC) (k = 5) • e.g., I’m always careful about correct pronunciation.

Background• Grammatical Carefulness Scale (GCS)

• 3 sub-scales • Lexical-Syntactic Carefulness (LSC) (k = 4)

• e.g., I always notice the wrong use of the words.

• Pragmatic Carefulness (PC) (k = 5) • e.g., I’m likely to think about inconsistent expressions.

• Phonological Carefulness (PHC) (k = 5) • e.g., I’m always careful about correct pronunciation.

Background• RQ

• Do the scores of the tasks focusing either on LSC or PC correlate with the LSC and PC measured by GCS?

Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion

Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion

The Present Study• Participants (N = 63)

• undergraduate students (3 university) and graduate students (1 university)

• 78% of the participants were female

undergrad grad

A B C D

n 20 27 8 8

The Present Study• Tasks

• LSC • Reading and Underlining Task (RUT)

• PC • Discourse Completion Task (DCT)

The Present Study• Reading and Underlining Task (RUT)

• often used as a measure of learner’s noticing toward formal properties of TL (e.g., Izumi & Bigelow, 2000; Izumi, Bigelow, Fujiwara, & Fearnow, 1999; Uggen, 2012)

• requiring to read a short passage and underline ungrammatical parts of the sentences.

The Present Study• Reading and Underlining Task (RUT)

• passage (153 words) • from internet sources (http://www.eslyes.com) • Flesch-Kincaid Reading Ease 94.252 • Flesch-Kincaid Grade Level 1.558 • revised and controlled vocabularies

(JACET8000 Level1-3) • Glossary was given.

The Present Study• Reading and Underlining Task (RUT)

• 9 errors • inflection errors

• was killing/ *kill her; She cried/ *cryed; She stood/ *standing up

• misspelling • walked/ *wolked; could/ *cuold; always/ *alweys

• wrong use of words • Whenever she was stressed out, the stress went straight to her/

*his stomach. • Almost an hour later/ *ago,…. • I feel much/ *more better.

The Present Study• Tasks

• LSC • Reading and Underlining Task (RUT)

• PC • Discourse Completion Task (DCT)

The Present Study• Tasks

• LSC • Reading and Underlining Task (RUT)

• PC • Discourse Completion Task (DCT)

The Present Study• Discourse Completion Task (DCT)

• used to tap into learners’ intuition about how they behave in certain speech acts (e.g., Bardovi-Harlig & Dornyei, 1998; Olshtain & Blum-Kulka, 1985)

• can be both spoken and written • can be both multiple-choice or free

production

The Present Study• Discourse Completion Task (DCT)

• based on Blum-Kulka, House, and Kasper (1989) • 2 speech acts (K = 12)

• request (k =8)

• apology (k = 8)

• various dimensions of social power and social distance

• multiple-choice • situation and instructions are given in Japanese

The Present Study• Discourse Completion Task (DCT)

• example • 見知らぬ街へ出かけたメアリー。駐車禁止場所とは知らずに車を停めてしまいました。すると警察官が寄ってきて…

Policeman: ( )

Mary: Oh, I’m sorry. I didn’t know that. I’ll move the car right now.

a. Would you please move your car from here?

b. How about moving the car from here?

c. Could you move the car from here?

d. Do you want to move the car from here?

e. Move your car.

The Present Study• Discourse Completion Task (DCT)

• example • 見知らぬ街へ出かけたメアリー。駐車禁止場所とは知らずに車を停めてしまいました。すると警察官が寄ってきて…

Policeman: ( )

Mary: Oh, I’m sorry. I didn’t know that. I’ll move the car right now.

a. Would you please move your car from here?

b. How about moving the car from here?

c. Could you move the car from here?

d. Do you want to move the car from here?

e. Move your car.

The Present Study• Reliability issue

• Reading and Underlining Task (RUT)

• Some items were excluded

N K M SD Cronbach α 95%CI

60 9 .55 .21 .50 [.28, .72]

The Present Study• Reliability issue

• Reading and Underlining Task (RUT)

• Excluded items: was killing/*kill, her/*his, later/*ago

N K M SD Cronbach α 95%CI

60 6 .49 .27 .58 [.28, .72]

The Present Study• Reliability issue

• Discourse Completion Task (DCT)

• Some items were excluded.

N K M SD Cronbach α 95%CI

61 12 .56 .18 .48 [.26, .69]

The Present Study• Reliability issue

• Discourse Completion Task (DCT)

• Excluded items: Item2, 9, 11, 5, 8

N K M SD Cronbach α 95%CI

61 7 .56 .23 .56 [.35, .77]

The Present Study• Procedure

• Two English language teachers working for a college

• Two tasks were administered as part of classroom activities

• Graduate students were recruited individually

The Present Study• Analysis

• 10 students missed one of the two tasks → excluded from the analysis • 3 students showed extremely low

products of deviation → excluded from the analysis (outlier) • In total, 50 students were included in

the final analysis.

The Present Study• Analysis

• Correlation between the score of the tasks and GCs

→ Pearson product-moment correlation coefficient → Partical Correlation Coefficient

Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion

Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion

Results

N M SD skew kurtosis SE α

RUT 50 .50 .32 .56 -1.01 -.04 .58

LSC 50 3.24 1.24 .31 -.20 .18 .87

PC 50 3.18 1.20 .18 -.05 .17 .85

DCT 50 .54 .20 -.10 -.46 .03 .56

Descriptive Statistics of the Two tasks, LSC, and PC

Partical Correlation Coefficient

RUT LSC PC DCT

RUT

LSC [.013, .526]

PC [-.083, .453] [.686, .888]

DCT [-.232, .324] [.002, .518] [-.062, .470]

Interval Estimation of Population Correlation Coefficient (95%CI)

RUT LSC PC DCT

RUT

LSC .408

PC .285 .942

DCT .088 .401 .319

Adjusting Correction for Attenuation of Correlation

Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion

Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion

Discussion• RUT and LSC

• Pearson’s r →weakly correlated • Partical Correlation Coefficient →weakly correlated • Interval Estimation of Population Correlation Coefficient →◯ • Adjusting Correction for Attenuation of Correlation →moderately correlated

Discussion• RUT and LSC

• Pearson’s r →weakly correlated • Partical Correlation Coefficient →weakly correlated • Interval Estimation of Population Correlation Coefficient →◯ • Adjusting Correction for Attenuation of Correlation →moderately correlated

• The students who showed high LSC were likely to notice lexical and grammatical errors

Discussion• DCT and PC

• Pearson’s r →weakly correlated • Partical Correlation Coefficient →△ • Interval Estimation of Population Correlation Coefficient →△ • Adjusting Correction for Attenuation of Correlation →weakly correlated

Discussion• DCT and PC

• Pearson’s r →weakly correlated • Partical Correlation Coefficient →△ • Interval Estimation of Population Correlation Coefficient →△ • Adjusting Correction for Attenuation of Correlation →weakly correlated

because LSC and PC showed high correlation

Discussion• The results seemed to confirm the validity of

GCS • difficult to investigate the correlation

between L2 learners’ performance and the subscales of GC

→ GCS showed high reliability

Partical Correlation Coefficient

Limitations• Reliability of the two tasks • Underrepresentation • What about the correlation between

phonological carefulness and learners’ performance?

Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion

Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion

Conclusion• The score of LSC predicted the score of

RUT • The score of PC partly predicted the score

of DCT • Validation never ends

ReferenceBardovi-Harlig, K., & Dörnyei, Z. (1998). Do language learners recognize pragmatic violations? pragmatic versus grammatical

awareness in instructed L2 Learning. TESOL Quarterly, 32, 233–259. doi:10.2307/3587583

Blum-Kulka, S., House, J., & Kasper, G. (1985). Investigating cross-cultural pragmatics: An Introductory Overview. In S. Blum-Kulka, J. House, & G. Kasper (Eds.), Cross-cultural pragmatics: Requests and apologies (pp. 1–33). Norwood, NJ: Ablex Publisher Corporation.

Dennis, I., & Evans, J. St. B. T. (1996). The speed-error trade-off problem in psychometric testing. British Journal of Psychology, 87, 105–129.

Goldhammer, F., & Kroehne, U. (2014). Controlling individuals’ time spent on task in speeded performance measures: Experimental time limits, posterior time limits, and response time Modelling. Applied Psychological Measurement, 38, 255–267.

Izumi, S., & Bigelow, M. (2000). Does output promote noticing and second language acquisition? TESOL Quarterly, 34, 239–278.

Izumi, S., Bigelow, M., Fujiwara, M., & Fearnow, S. (1999). Testing the output hypothesis: Effects of output on noticing and second language lcquisition. Studies in Second Language Acquisition, 21, 421–452.

Kusanagi, K., Fukuta, J., Kawaguchi, Y., Tamura, Y., Goto, A., Kurita, A., & Murota, D. (2014). Gaikokugo ni okeru bumpouteki shinchou sei syakudo no kaihatsu [Development of grammatical carefulness in English as a foreign language]. 40th Annual Conference of Japan Society of English Language Education. Tokushima University, Japan.

Olshtain, E., & Blum-Kulka, S. (1985). Degree of approximation: Nonnative reactions to native speech act behaviour. In S. M. Gass & C. G. Madden (Eds.), Input in second language acquisition (pp. 303–325). Newbury House.

Tanaka, K., & Ellis, R. (2003). Study abroad, language proficiency, and learner beliefs about language learning. JALT journal, 25, 63-85. Uggen, M. S. (2012). Reinvestigating the Noticing Function of Output. Language Learning, 62, 506–540. doi:10.1111/j.

1467-9922.2012.00693.x

van der Linden, W. J. (2007). A hierarchical framework for modelling speed and accuracy on test items. Psychometrika, 73, 287–308. van der Linden, W. J. (2009). Conceptual issues in response-time modelling. Journal of Educational Measurement, 46, 247–272.