Reinventing Reinventing Teaching Assistant Teaching Assistant TrainingTraining

Maria Teresa Velez, Ph.D.Maria Teresa Velez, Ph.D.Associate Dean, Graduate CollegeAssociate Dean, Graduate College

Factors leading to reformFactors leading to reform

Complaints from GTAs about lack of Complaints from GTAs about lack of preparation.preparation.

Lack of coordination with departmental Lack of coordination with departmental trainingtraining

Termination of the SPEAK test used to assess Termination of the SPEAK test used to assess the English fluency of International TAsthe English fluency of International TAs

New TOEFL ibt: No confidence on Speaking New TOEFL ibt: No confidence on Speaking section scores. No ITA-ready assessment.section scores. No ITA-ready assessment.

Loss of international specialist at UTCLoss of international specialist at UTC Budget cutsBudget cuts

Two ABOR MandatesTwo ABOR Mandates

That Teaching Assistants receive training That Teaching Assistants receive training prior to being placed in the classroom.prior to being placed in the classroom.

That all graduate students whose native That all graduate students whose native language is not English and who are not language is not English and who are not U.S. citizens or Permanent Residents U.S. citizens or Permanent Residents demonstrate proficiency with spoken demonstrate proficiency with spoken English before they can be hired as English before they can be hired as Graduate Assistants in Teaching (GATs). Graduate Assistants in Teaching (GATs).

Reinventing TA Training Task Reinventing TA Training Task ForceForce

Faculty/staff in charge of TA Training in main departments, Faculty/staff in charge of TA Training in main departments, e.g., English, Math, Chemistry, Psychologye.g., English, Math, Chemistry, Psychology

Representative of departments using large number of ITAsRepresentative of departments using large number of ITAs

Recognized faculty mentorsRecognized faculty mentors

Administrators: VP Academic Affairs,Administrators: VP Academic Affairs,Dean of Graduate CollegeDean of Graduate College

University Teaching CenterUniversity Teaching Center

What do TAs need to know?What do TAs need to know?

Is there a skills set needed by all or most TAs?: Is there a skills set needed by all or most TAs?: Lots of disagreementLots of disagreement

Are they receiving adequate training? Are they receiving adequate training? Duplications, Gaps?: Lots of duplication: Duplications, Gaps?: Lots of duplication: everybody teaching rules and regulations. everybody teaching rules and regulations. Little pedagogy.Little pedagogy.

Would web-based instruction be effective to Would web-based instruction be effective to teach skills that are not currently provided or teach skills that are not currently provided or needed the most: some acceptance needed the most: some acceptance

TA Needs Assessment SurveyTA Needs Assessment Survey• 102 statements describing 102 statements describing

instructional tasks and/or specific instructional tasks and/or specific skills, e.g., design of instruction, skills, e.g., design of instruction, interaction with students, and how interaction with students, and how important was each skill important was each skill

• Given to 19 faculty directors of TA Given to 19 faculty directors of TA Training in main departments, e.g., Training in main departments, e.g., English, Chemistry, Math, English, Chemistry, Math, Engineering but also AIS, History, Engineering but also AIS, History, LinguisticsLinguistics

• Focus groups with TAsFocus groups with TAs

• Focus groups with undergraduate Focus groups with undergraduate studentsstudents

TAs need to knowTAs need to know Legal and procedural requirementsLegal and procedural requirements

Quick decisionsQuick decisions: threatening behavior, disruptive behavior, policies on : threatening behavior, disruptive behavior, policies on disclosure of educational records, policies on discriminatory and disclosure of educational records, policies on discriminatory and harassment behavior, Code of Conduct (cheating and plagiarism).harassment behavior, Code of Conduct (cheating and plagiarism).

Ongoing decisionsOngoing decisions: ethical issues, conflict of interest, disability : ethical issues, conflict of interest, disability issues, syllabus implications, classroom management, issues, syllabus implications, classroom management,

Best practices:Best practices: Research findings on learning, teaching. Research findings on learning, teaching. Demonstrations, tips, techniques. Where to go for help, where to refer Demonstrations, tips, techniques. Where to go for help, where to refer studentsstudents

Specific needs:Specific needs: How to stimulate discussion in How to stimulate discussion in

discussion sections?discussion sections? How to conduct lab sections?How to conduct lab sections? How to give feedback about writing?How to give feedback about writing? How to set learning objectives?How to set learning objectives? How to grade fairly, how to evaluate How to grade fairly, how to evaluate

student learning? How to set student learning? How to set rubrics?rubrics?

ITAs need to know all of the above plus ITAs need to know all of the above plus understand about US students, cultural understand about US students, cultural issues, how to deal with accent, etc.issues, how to deal with accent, etc.

HOW DO WE MEET THESE NEEDS?HOW DO WE MEET THESE NEEDS?

Focus GATO on teaching techniques, role modeling, Focus GATO on teaching techniques, role modeling, getting ready for the first day of class. Include ITAs.getting ready for the first day of class. Include ITAs.

Additional workshop specifically for ITAs: IGATOAdditional workshop specifically for ITAs: IGATO Put rules and regulations on line: Set passing rate at high Put rules and regulations on line: Set passing rate at high

level to ensure thorough understanding.level to ensure thorough understanding.

Screen ITAs with new, reliable method that helps us Screen ITAs with new, reliable method that helps us understand adequacy of TOEFL ibt Verbal score to screen understand adequacy of TOEFL ibt Verbal score to screen ITAs.ITAs.

Build self-paced online modules according to identified Build self-paced online modules according to identified needs.needs.

Conduct formative evaluations: Iterative processConduct formative evaluations: Iterative process

HOW DO WE ASSESS EFFECTIVENESS OF TRAININGHOW DO WE ASSESS EFFECTIVENESS OF TRAINING

• QUANTITATIVE:QUANTITATIVE:• GATO and I-GATO evaluations: scoresGATO and I-GATO evaluations: scores• Scores on T-BEST: comparison with TOEFL ibt Scores on T-BEST: comparison with TOEFL ibt

scoresscores• Scores on online modulesScores on online modules

Item analysis Item analysis

QUALITATIVEQUALITATIVE• IndividualIndividual GATO and IGATO GATO and IGATO comments used comments used

to improve next year’s programto improve next year’s program• Opportunity to comment on each online module: Opportunity to comment on each online module:

TA feedback to refine each module TA feedback to refine each module

GATO and I-GATOGATO and I-GATO Contract with University Training Center Contract with University Training Center

led by Kristopher Weatherly.led by Kristopher Weatherly. Teaching demonstration: Dr. Tom Teaching demonstration: Dr. Tom

Fleming.Fleming. The First Day in the Classroom and The First Day in the Classroom and

Classroom Management Issues & Classroom Management Issues & Strategies taught by faculty, experienced Strategies taught by faculty, experienced TAs.TAs.

I-GATO: Maria Teresa Velez, Amy I-GATO: Maria Teresa Velez, Amy Almquist, Kris Weatherly, experienced ITAsAlmquist, Kris Weatherly, experienced ITAs

   F2005F2005 S2006S2006 F2006F2006 S2007S2007 F2007F2007 S2008S2008 F2008F2008

Overall GATO Evaluation Overall GATO Evaluation 4.194.19 4.404.40 4.104.10 4.244.24 4.314.31 4.574.57 4.514.51

Keynote: Why are you as a TA Keynote: Why are you as a TA important to UA important to UA    4.614.61 4.504.50 4.704.70 4.534.53 4.764.76 4.644.64

Dean of Students Dean of Students Office/Academic Integrity Office/Academic Integrity 4.604.60 3.963.96 3.93.9 4.004.00         

Disability Resource centerDisability Resource center 4.094.09 4.534.53 4.204.20 4.204.20         

Sexual HarassmentSexual Harassment 3.753.75                  

Library Instructional ResourcesLibrary Instructional Resources 4.194.19                  

Teaching Theory: Talking Teaching Theory: Talking CommunityCommunity 4.314.31                  

Equal Opportunity Affirmative Equal Opportunity Affirmative Action Action    4.494.49 3.803.80 4.204.20         

Learning Technologies CenterLearning Technologies Center          4.104.10         

First Day in the Classroom First Day in the Classroom             4.234.23 4.444.44   

Classroom Management Issues Classroom Management Issues and Strategiesand Strategies             4.184.18 4.524.52   

Creating a Positive Classroom Creating a Positive Classroom EnvironmentEnvironment                   4.464.46

Feedback & Coaching for Feedback & Coaching for Better Learning Better Learning                   4.444.44

                       

GATO EVALUATIONS

I-GATOI-GATO

Teaching in the Culture of the American Teaching in the Culture of the American ClassroomClassroom for International GATsfor International GATs

Overview of American CultureOverview of American Culture• What are American Undergrads Like? What do they expect from ITAs? What are American Undergrads Like? What do they expect from ITAs? • How to Capitalize on your Strengths as an ITA?How to Capitalize on your Strengths as an ITA?• Effective BoundariesEffective Boundaries• Classroom EtiquetteClassroom Etiquette

Presentation Skills/Persuasion SkillsPresentation Skills/Persuasion Skills• Presentation stancePresentation stance, , Shared voiceShared voice, , Stand and PracticeStand and Practice• Self-confidence and self-consciousness issuesSelf-confidence and self-consciousness issues• What if they don’t understand my explanation? Paraphrasing, using the board, otherWhat if they don’t understand my explanation? Paraphrasing, using the board, other

The ITA Experience by Experienced ITAs: Good The ITA Experience by Experienced ITAs: Good practices and what to avoidpractices and what to avoid

I-GATO Fall 05-08I-GATO Fall 05-08

iGATO (Rules and Regulations)iGATO (Rules and Regulations) 4.634.63    4.84.8 4.84.8 3.973.97      

Teaching in the Culture of the Am. Teaching in the Culture of the Am. ClassroomClassroom    4.224.22

Presentation SkillsPresentation Skills    4.584.58

The ITA Experience: PanelThe ITA Experience: Panel    4.154.15

Teaching Assistant Teaching Assistant Training OnlineTraining Online

Technology Learning CenterTechnology Learning Center Gretchen GibbsGretchen Gibbs D2L (net-ID controlled)D2L (net-ID controlled) Capacity to see how many took each Capacity to see how many took each

module, test/quiz results, qualitative module, test/quiz results, qualitative feedback.feedback.

Staying Out of TroubleStaying Out of Trouble

8 lessons: Rules and Regulations8 lessons: Rules and Regulations1.1. Conflicts of interestConflicts of interest2. Family Educational Rights & Privacy Act (FERPA)2. Family Educational Rights & Privacy Act (FERPA)3. Code of Conduct3. Code of Conduct4. Disruptive and Threatening Behavior4. Disruptive and Threatening Behavior5. Non-Discrimination and Anti-Harassment5. Non-Discrimination and Anti-Harassment6. Code of Academic Integrity6. Code of Academic Integrity7.7. Teaching Students with Disabilities Teaching Students with Disabilities8. Syllabus Policy8. Syllabus Policy

TestTest

• Of the 778 students enrolled in the online module, 650 students passed the test at 95% or above after three tries. Average score 98.07%

• 9 students did not pass; only one student earned below 90% (88.8%): All ESL

• Item analysis: items receiving high percentage of wrong responses have been modified

• 75% report spending more than 1 hour on the module: Too long?

• 141 students responded to a feedback survey (~21% of those who took the test)More than 95% thought that issues addressed were important or very important.91% felt that the activities always or regularly helped to understand how

the policies apply to themMore than 90% thought they might return to the lessons and resources for information,

ideas or guidanceMost common complaints: media files and the FERPA tutorialWhat students liked best: online and self-paced, realistic situations and practice questions, clear and concise summaries of policies, logical and easy to follow.

Staying Out of Trouble 07-08Staying Out of Trouble 07-08

Staying Out of Trouble Fall 08Staying Out of Trouble Fall 08

910 students enrolled910 students enrolled 821 completed the test821 completed the test Average score 98.07Average score 98.07 Minimum score was 84%Minimum score was 84% 75% reported spending more than one hour75% reported spending more than one hour 33.92% reported having to review the material 2-33.92% reported having to review the material 2-

3 times3 times Favorable comments: could do at home, on own Favorable comments: could do at home, on own

time, self-paced; very informative; practice time, self-paced; very informative; practice questions, examples and videos; easy to navigatequestions, examples and videos; easy to navigate

Second module: Discussion as a Way of Second module: Discussion as a Way of Teaching 07-08 (not required)Teaching 07-08 (not required)

264 TAs completed the module and took the 264 TAs completed the module and took the test. Average score was 95.08%test. Average score was 95.08%

An average of 55 minutes was spent on the An average of 55 minutes was spent on the module. 24% students reported spending module. 24% students reported spending more than 30 and less than 60 minutes. 22% more than 30 and less than 60 minutes. 22% report spending less than 30 minutes. Too report spending less than 30 minutes. Too short?short?

99% felt the activities regularly or always 99% felt the activities regularly or always helped understand how the lessons applied helped understand how the lessons applied to their work as teachers.to their work as teachers.

Feedback was VERY positive. 116 chose to Feedback was VERY positive. 116 chose to respond.respond.

I felt TATO organized all the information into one straightforward program and after going I felt TATO organized all the information into one straightforward program and after going through each module the information from TATO helped me set up a clear plan/purpose for through each module the information from TATO helped me set up a clear plan/purpose for the first discussion sections I will lead this semester.the first discussion sections I will lead this semester.

It gave me an idea of what I will be doing as a TA.It gave me an idea of what I will be doing as a TA.Information was presented in a logical, concise format. Information was presented in a logical, concise format. Rapid, concise and well-organized for future reference.Rapid, concise and well-organized for future reference.

Discussion as a Way of Teaching Discussion as a Way of Teaching 08-0908-09

332 students332 students 43.8% spent 30-60 minutes completing, 43.8% spent 30-60 minutes completing,

18.9% spent more than 60 minutes. 18.9% spent more than 60 minutes. Average score was 94.95%Average score was 94.95% +Comments: short and to the point, +Comments: short and to the point,

interactive activities, well organized; interactive activities, well organized; plenty of helpful information and sufficient plenty of helpful information and sufficient resources for first-time TAs.resources for first-time TAs.

-Obvious, redundant, too long, no human -Obvious, redundant, too long, no human interaction, need more color, animation, interaction, need more color, animation, interaction.interaction.

Quiz at the end of each topic:Quiz at the end of each topic: A discussion leader realizes that the reading that will be discussed in class A discussion leader realizes that the reading that will be discussed in class

has several terms and concepts that are probably unfamiliar to the has several terms and concepts that are probably unfamiliar to the students. What step(s) should s/he take?students. What step(s) should s/he take?

A discussion leader notices that after discussion has been going on for a A discussion leader notices that after discussion has been going on for a time, students are beginning to express their opinions about the article and time, students are beginning to express their opinions about the article and its author. What should the discussion leader do?its author. What should the discussion leader do?

A teacher would like to make the material being discussed more A teacher would like to make the material being discussed more meaningful for their students. What should the teacher do?meaningful for their students. What should the teacher do?

The following are samples of ground rules:The following are samples of ground rules:

In the discussion, the leader asks a question and no one raises their hand In the discussion, the leader asks a question and no one raises their hand or speaks up to answer. The leader should…..or speaks up to answer. The leader should…..

The discussion method gives students opportunity to practice which skills?The discussion method gives students opportunity to practice which skills?

Discussion Module stats (08-09)Discussion Module stats (08-09)

Effort to weed out confusing Effort to weed out confusing questions through item analysisquestions through item analysis

Additional needs expressed by TAsAdditional needs expressed by TAs

Discussion section is not applicable to Discussion section is not applicable to sections on conducting science lab sections on conducting science lab discussions. Need examples of successful discussions. Need examples of successful TA class activities from all subject areasTA class activities from all subject areas

more info about rubrics and gradingmore info about rubrics and grading video clips of real classesvideo clips of real classes visual representation of the discussion visual representation of the discussion

process process info about assisting with lecture courses info about assisting with lecture courses

and handling office hoursand handling office hours

Pilot study resultsPilot study resultsJun Liu, Ph.D., Head, English Department, University of ArizonaJun Liu, Ph.D., Head, English Department, University of Arizona

Mindworks, Inc., Hong-Kong, Graduate CollegeMindworks, Inc., Hong-Kong, Graduate College

Fall 2008Fall 2008

Task-Based English

Speaking Test

T-BEST MethodologyT-BEST Methodology In lieu of SPEAK. Performance-based test that In lieu of SPEAK. Performance-based test that

simulates language used in real instructional settings. simulates language used in real instructional settings. Four tasks: no right/wrong answer. PromptsFour tasks: no right/wrong answer. Prompts

• (1) Descriptive Task: “Describe your most effective (1) Descriptive Task: “Describe your most effective teacher”teacher”

• (2) Interpretive Task: “Explain ‘Actions speak louder than (2) Interpretive Task: “Explain ‘Actions speak louder than words”words”

• (3) an Argumentative Task: “Do you think that face to (3) an Argumentative Task: “Do you think that face to face communication is better than written face communication is better than written communication? Why or why not?” and communication? Why or why not?” and

• (4) Comprehensive Task: student is asked to respond to a (4) Comprehensive Task: student is asked to respond to a visual scene portraying TA-student interactionvisual scene portraying TA-student interaction

Examples pertaining only to academic and classroom Examples pertaining only to academic and classroom issues connected with teaching U.S. undergraduate issues connected with teaching U.S. undergraduate students.students.

Each test lasts 10 minutes: 1 for warm-up, 6 minutes Each test lasts 10 minutes: 1 for warm-up, 6 minutes for Tasks 1-3, and 4 minutes for Task 4. for Tasks 1-3, and 4 minutes for Task 4.

Score of 7 or above for classroom teaching and 6 for Score of 7 or above for classroom teaching and 6 for individual contact/gradingindividual contact/grading

Participants Fall 08Participants Fall 08

118 International graduate students, 118 International graduate students, applying to be Teaching Assistants (TAs)applying to be Teaching Assistants (TAs)

55 have previously taken TOEFL iBT: If 55 have previously taken TOEFL iBT: If Verbal score of 26 or higher, excused from Verbal score of 26 or higher, excused from T-BEST (ETS data)T-BEST (ETS data)

65 were from CALS, Science and 65 were from CALS, Science and Engineering: “science”Engineering: “science”

53 were from Humanities, Fine Arts, Eller 53 were from Humanities, Fine Arts, Eller and SBS: “humanities”and SBS: “humanities”

Testing ProcessTesting Process

4 test dates at the beginning of Fall 084 test dates at the beginning of Fall 08

10 minutes actual testing time10 minutes actual testing time

Fall 08: Each test scored by 2 raters to ensure Fall 08: Each test scored by 2 raters to ensure reliability. Blind, 3rd rater if dif by >1.0. Those reliability. Blind, 3rd rater if dif by >1.0. Those who scored between 6-6.9 (34 students) received who scored between 6-6.9 (34 students) received personal interview to ensure fairness and personal interview to ensure fairness and determine remediation strategy. determine remediation strategy.

Spring 08: Three experienced raters. Only one Spring 08: Three experienced raters. Only one student needed to receive personal interview. student needed to receive personal interview.

Interviews are conducted when students' final score is in the 6-6.9 range

to determine what remediation classes ITAs need before teaching and for ITAs

within .25 of the cut score to see if they may teach without remediation.

Interviews are conducted when students' final score is in the 6-6.9 range

to determine what remediation classes ITAs need before teaching and for ITAs

within .25 of the cut score to see if they may teach without remediation.

Score DistributionScore Distribution

TBEST identifies TBEST identifies more shades of more shades of difference between difference between testers in 5 testers in 5 categoriescategories

iBT scores fall into iBT scores fall into just 3 categoriesjust 3 categories

TBEST SHOWS TBEST SHOWS MORE DISTINCTION MORE DISTINCTION BETWEEN TESTERSBETWEEN TESTERS

Results (Fall 2008)Results (Fall 2008)

Max score:9.94 Min score:3.57 Mean:6.8Max score:9.94 Min score:3.57 Mean:6.8 Total 118 test-takersTotal 118 test-takers 53 (45%) passed at 7: full TA duties 53 (45%) passed at 7: full TA duties 45 (38%) scored 6, conditional pass: 45 (38%) scored 6, conditional pass:

limited student contact or approved to limited student contact or approved to teach in native languageteach in native language

20 scored under 6, no pass (17%)20 scored under 6, no pass (17%) Conclusion: 45% (53 students) who Conclusion: 45% (53 students) who

otherwise would not have been authorized otherwise would not have been authorized to teach if used only ibt score, passed.to teach if used only ibt score, passed.

Comparison to iBTComparison to iBT

TBEST is highly correlated to iBT TBEST is highly correlated to iBT • (Pearson’s r=.636)(Pearson’s r=.636)

TBEST 7.5 equals iBT 22.5TBEST 7.5 equals iBT 22.5 Good guidance but too early to Good guidance but too early to

tell tell

GeneralizabilityGeneralizability

Correlation between TBEST Correlation between TBEST scores of Science & Humanities scores of Science & Humanities ITA is high: r=.857ITA is high: r=.857

No evidence of bias based on No evidence of bias based on field of studyfield of study

Strengths of TBEST vs ibtStrengths of TBEST vs ibt

Precision. TBEST’s broader score band allows Precision. TBEST’s broader score band allows for more precise ranking of testers.for more precise ranking of testers.

Efficiency. TBEST is 10 minutes and contains Efficiency. TBEST is 10 minutes and contains 4 tasks while iBT is 20 minutes long and 6 4 tasks while iBT is 20 minutes long and 6 tasks.tasks.

Relevance. TBEST tasks are created with Relevance. TBEST tasks are created with teaching scenarios specifically in mind, while teaching scenarios specifically in mind, while iBT only assesses general speaking abilityiBT only assesses general speaking ability

Cost: Much less expensive than SPEAKCost: Much less expensive than SPEAK

Weaknesses of TBESTWeaknesses of TBEST

Online: computerized test may Online: computerized test may induce anxiety in some international induce anxiety in some international students.students.

Language piped in may not be as Language piped in may not be as clear. Noise in lab may impair clear. Noise in lab may impair hearing.hearing.

No other clues, e.g., face expression, No other clues, e.g., face expression, hands, to facilitate understanding.hands, to facilitate understanding.

One web pageOne web page

www.grad.arizona.edu/TAwww.grad.arizona.edu/TA