Evaluation at the Defense Language Institute Dr. Thomas S. Parry
description
Transcript of Evaluation at the Defense Language Institute Dr. Thomas S. Parry
Evaluation at the Defense Language InstituteDr. Thomas S. Parry
Directorate of Evaluation and StandardizationDefense Language Institute
Bureau for International Language CoordinationBudapest, Hungary
May, 2006
Evaluation can make you rethink what you have been doing!
Why Evaluation?
What is Evaluation?
The process of finding the value or amount of something—to determine it’s
significance or quality.
-- Webster’s New World Dictionary
DLI Facts
• Average daily resident attendance: 3,079• Languages taught: 22• Number of students taught annually
through all Continuing Education programs: 5,000
• Highest enrollment languages:
ArabicChineseKorean
Persian-FarsiRussianSpanish Serbian-Croatian
• 0515: Physical Training
• 0630: Breakfast
• 0730: Formation with Service Unit
• 0755: Class start
• 1530: Class end
• 1600: Military Training• Cumulative:
– 6 hours of class instruction
– 45 minute special assistance or enhancement training
– 3 hours of homework plus weekend assignments
– 2-3 hours of study hall (M-Th)
– Developing & sustaining basic warrior-linguist skills, common task training, and physical readiness
Typical DLI Student Day
Basic Course
Language FY06 Student Load Faculty
Category IV Languages (63 weeks)Category IV Languages (63 weeks) ArabicArabic 1165 1165 272 272 KoreanKorean 833 833 216 216 ChineseChinese 558 558 126 126 JapaneseJapanese 3535 10 10
2591 (71%) 6242591 (71%) 624 Category III Languages (47 weeks)Category III Languages (47 weeks) DariDari 87 87 24 24 PashtoPashto 134 134 34 34 Persian FarsiPersian Farsi 248 248 68 68 RussianRussian 144 144 58 58 Uzbek, Hindi, Uzbek, Hindi, Urdu, & 6 other langUrdu, & 6 other lang 211211 9494
824 (23%) 278824 (23%) 278Category II Languages (34 weeks)Category II Languages (34 weeks) GermanGerman 2727 88 27 (1%)27 (1%) 88Category I Languages 26 weeks)Category I Languages 26 weeks) Spanish, French,Spanish, French, Italian, PortugueseItalian, Portuguese 196196 5656
196 (5%) 196 (5%) 56 56
Totals Totals 3638 966 3638 966
DLIFLC
Student Load by DifficultyResident Courses at Presidio of Monterey
94% of students taking CAT III & IV Languages
DLI Evaluation Goals
• Ensure student and teacher accountability• Provide for continuous academic program
improvement– Faculty and staff development– Curriculum development and implementation
• Gauge our return on investment• Ensure quality of life and support services
are conducive to learning
Teachers Evaluate Students
Students are held to high levels of accountability for their learning
Teachers EvaluateStudents
• Teachers evaluate student performance in the classroom through:– Participation in classroom activities– Teacher-developed chapter/unit tests– Periodic counseling– Diagnostic assessments– Learning style assessment– Performance tests– Oral Proficiency Interviews (OPI)– Defense Language Proficiency Tests (DLPT)
• Listening• Reading
Students Evaluate
Teachers Program of Study Quality of Life
Evaluation at DLI is Bi-directional
Teachers Evaluate Students
Students Evaluate Teachers
Teacher Evaluation
• Teacher evaluation at the DLI includes:– Complete student evaluation of the teacher– Scheduled and unscheduled classroom observations by
the supervisor– Periodic reviews of performance objectives with the
supervisor
• The combination of these three teacher evaluation elements provides an accurate assessment of the quality and effectiveness of foreign language instruction
Teacher Evaluation
• Each teacher is evaluated on classroom performance by all the students he/she teaches
• A personal performance profile is created for each teacher by tracking his/her performance throughout each of their courses
Teacher evaluations by students have implications for merit pay and rank advancement
Interim Student Questionnaire
• Administered via computer to all students enrolled in 47- and 63-week language courses 60% of the way through the program
• Takes roughly 60-90 minutes to complete• Survey sections include:
– Program Evaluation (PE)– Teacher Evaluation (TE)– Quality of Life (QL)
ISQ
End-of Course Student Questionnaire
• Administered via computer at the end of language courses to all graduating students
• Takes roughly 60-90 minutes to complete
• Survey sections include:– Program Evaluation (PE)
– Teacher Evaluation (TE)
– Quality of Life (QL)
ESQ
Analysis and Reporting
• Evaluation feedback from ISQ/ESQ delivered within two working days from survey administration– Deans
– Chairs
– Team leaders
– Individual teachers
• Special monthly reports provided to service commanders and individual units via separate questionnaires
• Annual summaries for each instructor sent to Provost, Vice Chancellors and school Deans
ISQ/ESQ Response Scale
• Items are rated from 1-4 with an option for no opinion:
0 No opinion
1 Strongly disagree
2 Disagree
3 Agree
4 Strongly agree
Teacher Evaluation (TE)
• Class preparation and quality of instruction
• Ability to teach language and related concepts effectively
• Adjustment of teaching strategies
• Ability to provide a variety of instructional activities
Sample Criteria
Teacher Evaluation (TE)
• Active use of foreign language in classroom
• Ability to provide challenging instruction
• Counseling and assistance offered
• Feedback given to students
Sample Criteria
Team and Individual Instructor TE Analysis
Teacher Improvement
Program Evaluation (PE)
• Testing and grading
• Team teaching
• Homework
• Curriculum and supporting materials
• Overall evaluations
Sample Criteria
Quality of Life (QL)
• Barracks, family housing and child care
• Quality of food, mess hall and related services
• Medical and dental care
• Chaplain and community services
• Military support
• Sport and recreation opportunities
• Transportation and travel office services
Sample Criteria
Quality of Life (QL)
• Equal Opportunity– Prejudice– Discrimination– Sexual harassment
• Military– Training– Duties– General quality of life
Other Areas of Concern
PE and QL Analysis
Serious Issues
• Depression
• Suicide threats
• Sexual assaults and harassment
• Fraternization and other misconduct
Immediate turnaround; sent directly to Command Group for action
Evaluation of Continuing Education Programs
• ESQ questionnaires distributed world-wide to training sites and include:– Video tele-training sites– Mobile training team deployments– Language Teaching Detachments– Command Language Programs– Special, customized programs
• Questionnaires completed and returned to DLI via on-site program managers
• Analysis and feedback provided to instructor’s school and training site within two weeks
Conclusion
• DLI evaluation practices: – Are well established—honed through years of
practice– Are rigorous and thorough– Bring about faculty and program improvement– Establish baselines for student/teacher
accountability– Give us cause to rethink what we are doing and
consider how we might do things better
Evaluation at the Defense Language Institute
Questions?
Discussion?