Tesol08 Learning Outcomes

Creating Learning Outcomes for a TESOL MA

• The results of a year-long process to create learning outcomes and assessment measures for a TESOL master’s degree program will be shared. This information will be beneficial for TESOL teacher educators at other U.S. universities who face similar tasks.

Creating Learning Outcomes for a TESOL MA

Lynn Henrichsen and Mark TannerLinguistics & English Language Department

Brigham Young University

Provo, Utah

TESOL 2008, NYC

Introduction

• Significant paradigm shift in US higher education– Stronger focus on student learning– Greater focus on program outcomes

• Accrediting agencies are now emphasizing “learning outcomes”

Introduction: Creating learning outcomes

• Not a simple process– Identify and publish outcomes– Meet specifications of clarity, learning

focus, disciplinary focus, measurability, etc.– Gather evidence that

• Students achieve outcomes• Results lead to improvement in teaching and

learning

Introduction: Overview of our presentation today

• Explain – What learning outcomes are– Why they are important– How to create them

• Share the results of our experience

• Describe the lessons we learned

Learning Outcomes: Their Nature and Importance

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Assessment of student learning

• Long standing tradition

• A revolutionary paradigm shift

A subtle yet profound paradigm shift

“A paradigm shift is taking hold in American higher education. In its briefest form, the paradigm that has governed our colleges is this: A college is an institution that exists to provide instruction. Subtly but profoundly we are shifting to a new paradigm: A college is an institution that exists to produce learning. This shift changes everything” (Barr and Tagg,

1995, p. 1).

Our new educational mission

To say that the purpose of colleges is to provide instruction is like saying that General Motors’ business is to operate assembly lines or that the purpose of medical care is to fill hospital beds. We now see that our mission is not instruction but rather that of producing learning with every student by whatever means work best” (Barr and Tagg, 1995, p. 1)

Concern: Higher education’s returns to society

• “Informed consumers of higher education”(Middle States Commission, 2003, p. 1)

– Legislators– Employers– Parents of students– The general public

U.S. Secretary of Education,

Margaret Spellings

• There is “much that requires urgent reform” in U. S. higher education.

• One of Secretary Spellings’ top priorities has been “putting ‘student learning outcomes’ at the center of accreditation” (Spellings 2006).

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

U.S. Secretary of Education,

Margaret Spellings

“Right now, accreditation…is largely focused on inputs, more on how many books are in a college library, than whether students can actually understand them…Institutions are asked, ‘Are you measuring student learning?’ And they check yes or no. That must change. Whether students are learning is not a yes-or-no question. It’s how? How much? And to what effect?” (cited in Woods, 2007, p. 3).

Accreditation agencies have responded to this call







Sample accreditation requirements

• The Quality Enhancement Plan (QEP), submitted four to six weeks in advance of the on-site review by the Commission…(2) focuses on learning outcomes… (Southern Association of Colleges and Schools Commission on Colleges, 2008b)

• The institution uses a variety of quantitative and qualitative methods to understand the experiences and learning outcomes of its students (New England Association of Schools and Colleges Commission on Institutions of Higher Education, 2005, standard 4.50)

The process

• Overtly and rigorously evaluating… – The learning accomplished by students in

a class.– The learning outcomes of an entire degree

program.– The learning outcomes of the whole

educational institution.

The bottom line

• When U.S. institutions of higher education undergo self-study, peer-review, and re-accreditation, learning outcomes are to be clearly demonstrated—or else!

• If you work in an academic degree program at such an institution and you haven’t yet encountered learning outcomes, you will!

Guidelines for Creating Quality Learning Outcomes

• Number• Clarity• Learning focus• Disciplinary Focus• Institutional focus• Measurability

General guidelines

“Inquiry may focus on a variety of perspectives, including understanding the process of learning, being able to describe student experiences and learning outcomes in normative terms, and gaining feedback from alumni, employers, and others situated to help in the description and assessment of student learning.” (New England Association of Schools and Colleges Commission on Institutions of Higher Education, 2005, standard 4.50)

General guidelines

It is essential that the institution devote “appropriate attention to ensuring that its methods of understanding student learning are trustworthy and provide information useful in the continuing improvement of programs and services for students” (New England Association of Schools and Colleges Commission on Institutions of Higher Education, 2005, standard 4.50)

General guidelines

• https://learningoutcomes.byu.edu/

• http://www.oir.uiuc.edu/assessment/home.html

• http://ctl.sdsu.edu

Number

• How many learning outcomes for an academic program?

• Possible to create many, specific, detail-oriented outcomes.– Intimidating– Counterproductive (burden-

some in assessment phase)

Number

• Fewer, more general = better• “Ideally, learning outcomes are clear,

simple, and few.” (Tanner, 2006)

• “What is most important for students to learn in this program?”

• “What should graduates of this program be able to do?” (Guidelines for Writing Expected Learning Outcomes (Program Level))

Leading questions• For faculty

– In general, what are the most important things a student gains from your field of study?

– What qualities and capabilities do you strive to foster in your students?

– How do people in this area of study differ from those in other areas (knowledge, skills, and values)? (Middle States Commission, Figure 3)

Leading questions• For students

– What is the most important knowledge you have gained from taking courses, minoring, or majoring in this subject?

– What are the most valuable skills or abilities that you have developed as a result of taking courses, minoring, or majoring in this subject?

– How does your field of study…change the way students view themselves? (Middle States Commission, Figure 3)

Clarity & Learning Focus

• Clear (unambiguous, written in language that students themselves can readily understand)

• Explain what students will be able to do after completing the degree program (that they could not do before they started it).

• Simple formula: “Program graduates will be able to (verb + ______)”



Use “action verbs”

For example: analyze, dissect, detect, test, deconstruct, discriminate, distinguish, examine, focus, find coherence, survey, compare, contrast, classify, investigate, outline, separate, structure, categorize, solve, diagram, determine (Guidelines for Writing Expected Learning Outcomes (Program Level))




• All action verbs are not equal

• Bloom’s Taxonomy of Educational Objectives

Bloom’s taxonomy

• Simple recall (label, list, recite, name, recall).

• Application (show, apply, construct)

• Analysis (classify, compare, differentiate)

• Synthesis (compose, combine, design)

• Evaluation (decide, rate, assess).

Disciplinary Focus

“Does it seem likely that the expected learning outcomes distinguish this program as a particular field of study— different from related programs at BYU and what an equivalent program would look like at other universities?”



Disciplinary Focus

• Distinguish our TESOL graduate program from graduate programs in other academic disciplines…– English (literature).– Foreign-language

(e.g., Spanish, French) teaching.

– General (K-12) teacher education programs.



Institutional Focus

• Every university/college has a unique, institutional mission.

• Embodied in its mission statement.

Institutional Focus

• United States Military Academy at West Point– “To educate, train, and inspire the Corps of Cadets

so that each graduate is a commissioned leader of character committed to the values of Duty, Honor, Country and prepared for a career of professional excellence and service to the Nation as an officer in the United States Army” (USMA West Point, 2008).




Institutional Focus

• Goshen College– “Dedicated to the development of informed,

articulate, sensitive, responsible Christians,” – “Seeks “to integrate Christian values with

educational, social and professional life,” – “Strives “to foster personal, intellectual, spiritual

and social growth in every person.” – End goal is to produce “servant-leaders for the

church and the world” (Goshen College, 2004).


Institutional Focus

• Brigham Young University (mission)– “Assist individuals in their quest for

perfection and eternal life” – “Provide a period of intensive learning in a

stimulating setting where a commitment to excellence is expected and the full realization of human potential is pursued” (Brigham Young University, 1981).

Institutional Focus

• Aims of a BYU education:– “Spiritually strengthening,– Intellectually enlarging, and– Character building, leading to …– Lifelong learning and service”

(Brigham Young University, 2008).



Institutional Focus

• Learning outcomes developed at the program level need to…– Not conflict with the institution’s overall

mission. – Reflect, harmonize with, and build up

particular aspects of the broader institutional mission.

Measurability

• Think about…– How each outcome will be

assessed later.– What criteria will be used.– What evidence of student

learning will be most relevant for each learning outcome.

Measurability

• Simple, general format:– “Student achievement of this learning

outcome is assessed by _____ and ____.”

• Reality: More complicated.

Measurability

• Multiple measures (at least two) for each learning outcome.

• Appropriate use of direct and indirect measures (Expected Learning Outcomes Formative Peer

Review, 2007, p. 2).

Direct and indirect measures

• Direct “evidence demonstrates that actual learning has occurred relating to a specific content or skill.”

• Indirect “methods reveal characteristics associated with learning, but they only imply that learning has occurred” (Middle States Commission, 2003, p. 28).

Direct measures

• Evidence of whether a student “has command of a specific subject or content area, can perform a certain task, exhibit a particular skill, or demonstrate a certain quality in his or her work” (Middle States Commission, 2003, p. 30).

• Examples: Students’ MA theses, publications, conference presentations, pass rates on certification tests.

Direct measures

• “Provide no evidence as to why the student has learned or why he or she has not learned” (Middle States Commission, 2003, p. 31).

• “Why” is especially important when students have not learned.

Indirect measures

• Involve “data that are related to the act of learning” and predict or mediate learning or perceptions about learning

• Acquired through self-report surveys, questionnaires, interviews, and demographic statistics about students and graduates

• Example: Focus group interviews, employer or alumni surveys, registration information, job placement rates.

Measurability

• Assessment measures may be…– Standard, pre-existing (GRE score, GPA,

national ranking, time to degree, etc.)– Locally developed (capstone course,

course grade, portfolio, thesis, etc.) (DeStefano et al, 1999, p. 11).

Measurability

• Sustainable?

• Learning outcomes assessment is…– Long-term process, a permanent, ongoing

part of the program’s operation.– Not a last-minute, temporary activity.

A cyclical process

• Create and publish learning outcomes for a program

• Collect evidence through direct and indirect assessments

• Use these data in a formative way to evaluate and revise the published learning outcomes.



Our Experience—The Process

• Initial Brainstorming• Connecting with

Courses• Revising, Refining,

& Consolidating• Identifying

Measures



BYU’s TESOL Program

• Undergraduate TESOL minors

• Graduate TESOL program with two stages:– TESOL graduate certificate (1st year)– TESOL MA (2nd year)

(Existing program has been in operation for over 35 years with about 800 graduates)

Initial Brainstorming

• Preliminary discussions focused on identifying what graduating students should “know,” “do,” and “communicate.”– In the area of “knowing”:

• Students will have a core knowledge of linguistic systems, past & present theories of SLA, & current and historical methods and techniques for language teaching.



Brainstorming (cont.)

– In the area of “doing”:• Students should be able to read, understand,

and critically analyze professional research. • Students should be able to create effective

teaching materials & lesson plans, and employ a range of teaching strategies.

– In the area of “communicating”:• Students should be able to give professional

level presentations on historical practices, personal research, and teaching ideas.

Connecting with Courses

• The initial brainstorm generated 26 expected learning outcomes grouped in three areas: 1) Knowledge outcomes, 2) Performance outcomes, 3) Communication outcomes.

• The list of outcomes was connected to existing courses to see where weaknesses might exist.

Revising, Refining, & Consolidating

• Counsel from Academic VP - “Keep the learning outcomes clear, simple, & few” (Tanner, 2006).

• The former list of outcomes was revised, refined, consolidated.

• Two major learning outcomes were identified with a few enabling outcomes.

Learning Outcome #1:

“All students completing the graduate certificate will be able to teach English language skills to non-native English speakers in an effective and professional manner.”

Enabling outcomes:

1. Manifest mastery of the English Language2. Demonstrate understand major systems of human

language3. Demonstrate an ability to analyze and apply

theories of SLL and SLA4. Evaluate principles underlying effective empirical

research5. Examine proven second language teaching

strategies and procedures6. Apply accepted principles of effective lesson

planning and materials development

Learning Outcome #2

“MA students will go beyond core competencies to develop specialized knowledge and skills within one or more of the following areas: – Research– Teaching knowledge & skills– Curriculum & materials development– Assessment– Administration and supervision”

Direct Measures

• A well-written Master’s thesis or project• A professional teaching portfolio• Supervisory observation & mentoring reports

of students’ classroom teaching• A well-written teaching philosophy statement• A creative and well-organized materials file• Organized curricula and lesson plans• English language proficiency exams (OPI and

advanced writing class)

Indirect Measures

• Annual external reviews of Master’s theses and projects

• Bi-annual student progress reviews by faculty• Annual survey and focus groups of graduating

students• Surveys of near-graduates in capstone course• Individual course/instructor evaluations• Alumni survey of TESOL graduates• University-administered alumni survey

Lessons Learned

• Be willing to adjust attitudes

• Be collegial, but honest

• Think innovatively• Put things in writing• Prepare for long-

term, ongoing development



Be willing to adjust attitudes

• Initial reaction by faculty - “Do we have to do this?”

• Administrators assured faculty that the process would help strengthen programs and focus attention on learning rather than teaching.

Be collegial, but honest

• The TESOL faculty began the process in the department - perceived to have the strongest level of collegiality

• Periodically, strong feelings were expressed regarding philosophical underpinnings and specific areas of change.

• Faculty had to be honest about changes occurring in the larger profession (e.g. K-12 ESL growth)

Think innovatively

The process of creating learning outcomes gave faculty an opportunity to:

– Move beyond the status quo– “Think outside the box”– Carefully consider current and future

needs of the students– Look forward to changing directions within

the profession

Put things in writing

Learning outcome meetings were more effective when:– Minutes of discussions and assignments

from the previous meeting were written down and emailed to the committee

– Faculty were accountable for completing assignments by the next scheduled meeting

– Faculty could review and criticize written descriptions of the plan

Prepare for long-term, ongoing development

• Creating program-wide learning outcomes is the tip of the iceberg

• The ongoing process includes:– Collecting, analyzing, and

reporting on data from direct and indirect measures

– Specifying learning outcomes for individual courses

– Using assessment data for regularly reviewing program-level and course specific learning outcomes

Conclusion

Creating Learning Outcomes has caused a shift in focus:

Away From: Towards:•What is taught? How are students learning?•How it is taught? In what types of learning

activities will students be engaged?

Conclusion• Creating learning outcomes has shifted the focus of

the program and individual courses away from what is taught and how it is taught towards focusing on how students will learn the content and in what types of learning activities students will be engaged.

•What is taught•How it is taught

•How students will learn•What types of learning activities students will engage in

Conclusion

• We wish you well as you engage in similar processes in the future!

• Contact us if you have questions:

[email protected][email protected]



• PowerPoint® available at www.slideshare.net/LynnHenrichsen

mailto:[email protected]

The End

Thank you very much!

Tesol08 Learning Outcomes

Education

Transcript of Tesol08 Learning Outcomes