Strategies for Incorporating Evaluation into Teacher Scholarship Programs:

A Case Study of Math for America Los Angeles Teacher Fellows Program

Pam MasonExecutive Director, MƒA LA

Rebecca EddyPresident, Cobblestone Applied Research & Evaluation, Inc.

www.mathforamerica.org/la

A Partnership of Mathematicians, Business People and Educators

The mission of Math for America is to improve mathematics and science education in US public secondary schools by building a corps of outstanding STEM teachers and leaders. Our goal is to support those outstanding STEM teachers already in the classroom and to increase the number of mathematically talented individuals entering teaching.

In 2008, The University of Southern California (USC), Claremont Graduate University (CGU) and Harvey Mudd College (HMC) teamed together with MƒA to create Math for America Los Angeles (MƒA LA).

To improve math education in our public schools by attracting, training, and retaining highly qualified secondary school mathematics teachers.

To provide the support teachers require to answer a critical need and bring about a real, sustained change.

• Monthly Professional Development

• Curriculum and Teaching Support

• National Board Certification

• Ongoing classroom observations and instructional support by experienced advisors

• Social and network building opportunities

Clusters of Fellows at individual schools create effective learning environments for students

Professional Development Outreach

Program Designed by Pam Mason, MƒA LA Executive Director Darryl Yong, Associate Chair of Mathematics at Harvey Mudd College

Relevant, “Just in Time” Relevant, “Just in Time” Professional DevelopmentProfessional Development

MƒA LA Fellows have attended regional and national conferences for mathematics teachers and have given formal presentations.

Conferences attended during the 2012-2013 school year:

• CMC South Conference in Palm Springs

• CMC North Conference in Asilomar

• NCSM Conference in Denver

• NCTM Conference in Denver

Photo by Solares PhotographyReprinted with permission from the National Council of Teachers of Mathematics

A tightly networked community

•Regular meetings

•Experienced coaches

•Social events

The MƒA LA Effect …The MƒA LA Effect …ACCORDING TO MƒA LA COACHES

“… a feeling of camaraderie …”

Purpose of Evaluation Setting goals Identifying indicators to measure program

implementation and outcomes Keeping the program on track Drawing conclusions about the merit of the

program

Evaluation designed to answer key evaluation questions

Uses a combination of qualitative and quantitative methods

Quasi-experimental design to compare students of MfA LA Fellows vs. students on non-Fellows in the same schools

Evaluation Question

To what extent is the implementation of MfA LA program activities occurring as planned?

How well have Fellows’ students performed on mathematics achievement outcomes such as CST scores compared to other students?Do students of Fellows show any additional benefits from being in their classrooms such as course-taking patterns or attitudes about mathematics in comparison to students of non-Fellows?Are there differences between the Fellows’ experiences for those at cluster sites versus individual sites?

Is there evidence that the MfA LA program improves teacher recruitment, quality, and retention in the teaching profession?

What are the strengths of the MfA LA program?

What are areas of improvement for MfA LA?

PurposeTrack program implementation and measure outcomes

Math for America Los Angeles is committed to a comprehensive evaluation of its programs and operations

DesignMixed method: Quantitative and qualitative data sources to answer evaluation questionsMatched control group quasi-experimental design for student achievement data

Evaluation ActivitiesFocus groups, interviews, classroom observations, document analysis, assessment of student test scores & surveys

• What is a logic model?• Common elements in a logic model:

• Inputs• Activities• Outputs• Outcomes

“Logic models help us plan, implement, evaluate, and communicate more effectively.” Taylor-Powell, E., &

Henert, E. (2008)

• University of Wisconsin – Extension: templates, examples

http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

• W. K. Kellogg Foundation: Development Guidehttp://www.wkkf.org/knowledge-center/resources/2006/02/wk-kellogg-foundation-logic-model-development-guide.aspx

Situation

Get Pills Take Pills Feel Better

INPUTS OUTPUTS OUTCOMES

Inputs What is invested

in MfA LA

Inputs What is invested

in MfA LA

ActivitiesWhat MfA LA

does

ActivitiesWhat MfA LA

does

ParticipationWho MfA LA

reaches

ParticipationWho MfA LA

reaches

Short-termShort-term Med-termMed-term Long-term Long-term

SituationStudents under-achieve in math; U.S. needs STEM workforce to be globally competitive

OutputsOutputs OutcomesOutcomes

AssumptionsHighly qualified math teachers produce high achieving studentsRecruiting, training & supporting highly-qualified individuals will produce highly qualified teachers who stay in the

profession

External FactorsStudents in high poverty schools in LA less likely to receive instruction from highly qualified teachersStudents in high poverty schools in LA less likely to perform well in mathematics

Fellows receive stipends + tuition support

Fellows receive coaching from mentors

Fellows receive prof. support through PD

StaffExec. DirectorCoachesFinancial Support (Grants, Donors) TimeCred. Program 

Fellows receive training

Fellows receive training

Students receive math instruction from trained Fellow

Students receive math instruction from trained Fellow

Schools receive trained teachers/ leaders

Schools receive trained teachers/ leaders

Fellows receive moral/ emotional/ instructional support from other Fellows

Fellows teach in classrooms/ work in math depts

Districts receive trained teachers

Districts receive trained teachers

Fellows have improved pedagogical skills

Fellows maintain content knowledge in math

Fellows feel supported & respected as a teacher

Fellows are more confident to lead in classroom & in department

Fellows continue to teach in schools

Fellows continue to teach in schools

Students acquire strong math skills

Students acquire strong math skills

Fellows retained in math profession

LA schools/ districts have highly qualified math teachers

LA schools/ districts have high achieving math students

Students from LA competitive in global economy

Students maintain achievement & interest in math/ other STEM fields

Students maintain achievement & interest in math/ other STEM fields

National Board Certification & continued professional growth

National Board Certification & continued professional growth

Teachers at Fellows’ school collaborate with Fellows

Teachers at Fellows’ school collaborate with Fellows

MfA LA Teacher Fellows Program Logic Model

Need Inputs

OutcomesOutputs

Activities Participation

Product

Long-termShort-term

What needs are we addressing?

Teachers lack support and professional development experiences that prevents recruitment & retention of high-quality mathematics secondary school teachers

Need Inputs

OutcomesOutputs

Activities Participation

Product

Long-termShort-term

What resources do we need for

the program?

Director/ staff expertise, funding

Need Inputs

OutcomesOutputs

Activities Participation

Product

Long-termShort-term

What activities are planned to address the

needs?

Who will participate?

Fellows attend monthly PD sessions 10 months per year

& attend professional conferences; receive

coaching support; stipends

Monitoring Implementation

Need Inputs

OutcomesOutputs

Activities Participation

Product

Long-termShort-term

What are the short- term and long-term

changes we expect to see in participants?

Fellows acquire content knowledge, classroom

management & planning skills (short-term)

Fellows are retained in the profession & are effective secondary math teachers

(long-term)

Determining Program Merit

See the following list and determine in which category each applies For example: Bake cookie dough

Is this a need, activity, outcome, etc.?

1. Start with a need 2. Identify one activity (Outputs)

• Specify how much of that activity needs to happen to address the need (Activities)

• Establish who will be reached by the activity and what is required of their participation (Participation)

3. What resources are required? (Inputs)4. What is your goal for meeting the needs? (Outcomes)

What are some common needs that you identified?

In thinking about designing an evaluation, what are some indicators that we will track? Outputs Outcomes

How can we design an evaluation to effectively address the key evaluation questions and include the indicators for implementation and outcomes?

Evaluation Question AnswerTo what extent is the implementation of MfA LA program activities occurring as planned?

Implementation: Main activities were implemented as planned (e.g., PD, coaching); Slight adjustments to coaching and professional development to meet the needs of more experienced Fellows

How well have Fellows’ students performed on mathematics achievement outcomes such as CST scores compared to other students?

CST: Fellows’ students performed similarly to a matched group of students of non-Fellows across all subject areasCAHSEE: Fellows’ students performed similarly to students of non-Fellows; 11th grade students of Fellows were more likely to pass than students of non-Fellows

Do students of Fellows show any additional benefits from being in their classrooms such as course-taking patterns or attitudes about mathematics in comparison to students of non-Fellows?

Course-Taking Patterns: Students of Fellows (57%) were more likely to advance to another math class in the next school year than students of non-Fellows (51%) for both years (2010/11 & 2011/12)Student Attitudes: At pretest, students of Fellows and non-Fellows had similar attitudes towards mathematics; fall survey to be administered and analyzed to determine if there are differences at posttest

Are there differences between the Fellows’ experiences for those at cluster sites versus individual sites?

Fellows at “cluster sites” reported more support, lesson collaboration, and similar philosophies about teaching than Fellows at “individual sites”

Is there evidence that the MfA LA program improves teacher recruitment, quality, and retention in the teaching profession? 

Recruitment: Some Fellows reported that MfA LA provided additional motivation to enter the teaching professionRetention: 97% of Fellows have been retained; 91% of Fellows plan to stay in teaching field after MfA LA Teacher Quality: Fellows have strong academic backgrounds (i.e., Praxis, college major) and build from this basis as indicated by coaches comments & classroom observations

What are the strengths of the MfA LA program?

Program Strengths: Professional and Community Support; Quality of Program Activities (e.g., PD, workshops); Financial Support

What are areas of improvement for MfA LA?

Professional Development: Lesson planning; Small-group discussionsCoach-Fellow Mentoring Model: Critical and constructive feedback from coaches; Clarification of coaches’ roles; Training for coaches

2009 Cohort

“I am very thankful for all that MƒA LA has been done to make

me a better teacher.”

2009 Cohort

“I am grateful for such a wonderful chance to inspire

young minds ...”

2009 Cohort

“I always consider it an honor to say that I am a fellow in such a

wonderful organization

Renewing and expanding our commitment to improve student achievement and mathematics education in the greater Los Angeles community with each new cohort of fellows.