NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... ·...

21
10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation 633 17 th St., Suite 2100 Denver, CO 80202 (800) 922-3636 PRESENTERS Brandie Ward, RMC Research Corporation Arlene Mitchell, RMC Research Corporation Emma Espel, RMC Research Corporation

Transcript of NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... ·...

Page 1: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

1

NOYCE PROJECT EVALUATION

John T. Sutton, Ph.D.

RMC Research Corporation

633 17th St., Suite 2100

Denver, CO 80202

(800) 922-3636

PRESENTERS

Brandie Ward, RMC Research Corporation

Arlene Mitchell, RMC Research Corporation

Emma Espel, RMC Research Corporation

Page 2: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

2

REFLECTION QUESTIONS

What type of NOYCE project are you evaluating and what is the 

greatest challenge you encounter in conducting the evaluation?

What measures or protocols do you use and for what purposes?

What methods do you use to report your findings and how useful is 

that perceived?

REFLECTION QUESTIONS

Which of the Program Evaluation Standards do you feel has the 

greatest application in the NOYCE project evaluation? Why?

If you were giving advice to another Noyce Project that was 

thinking about how to improve or enhance their evaluation, what 

would that be?

Page 3: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

3

NOYCE PROJECT EVALUATION

Brandie WardRMC Research Corporation633 17th St., Suite 2100Denver, CO(800) [email protected]

TYPE OF NOYCE PROJECT EVALUATING?

Capacity Building Grant:

University of Denver – Access in Mathematics for All (AMA)

Page 4: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

4

GREATEST CHALLENGE ENCOUNTERED IN CONDUCTING THE EVALUATION?

Small budget limits evaluator participation

Reliant on project documents

MEASURES OR PROTOCOLS USED?

Document review to monitor their communications and processes to achieve their goals for the first year.

Review of:Tasks and timelinesMeeting minutes, action items, status updatesEmailsActivity logsProject records (participant database)

Page 5: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

5

METHOD FOR REPORTING/COMMUNICATING

No report, yet

Communicate via email

Provide recommendations to the project, sharing documents, resources, and sharing a participant tracking database.

PROGRAM EVALUATION STANDARD

Utility Standard, specifically related to the subcomponents

U5 Relevant Information U6 Meaningful Processes and Products U7 Timely and Appropriate Communicating

and Reporting

Page 6: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

6

ADVICE?

Involve your evaluator at the very beginning stages of the project, if possible

Be in regular communication

Arlene P. MitchellRMC Research Corporation

Denver, CO

Page 7: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

7

Master Teachers (MTFs) 23 original MTFs plus 7 added Instructors for professional development through

UNL during summer, school year

Teaching Fellows (TFs) 14-month program to earn Masters degree Complete an entire year of interning and student

teaching in classroom with Master teacher

2 cohorts Cohort 1 (6 TFs): Completed 2nd year of teaching Cohort 2 (7 TFs): Completed 1st year of teaching

Teacher Survey (online) Quantitatively measure changes in preparedness and use of

instructional strategies, confidence and level of support for teaching and learning mathematics, and engagement in activities related to teaching and learning mathematics.

Teacher Leader Inventory (online) Adapted from the Teacher Leader Model Standards from the

Teacher Leadership Exploratory Consortium Assess quantitatively the extent to which MTFs helped with

creating a collaborative culture, assessing and using research, promoting professional learning, using assessments and data, and improving outreach and collaboration.

Page 8: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

8

Telephone interviews with randomly selected MTFs Questions about their contributions and experiences

to school, district, and UNL

Telephone interviews with district mathematics coordinators and principals Impact of the NOYCE project in their district or

contributions to their high-need building

Cohort 1 Open-ended survey about

program Open-ended survey about

1st year of teaching Teacher Survey Teacher Leader Inventory Interviews

Preparation to teach, philosophy of teaching, and intent of their instruction

Interviews with principals

Cohort 2 Open-ended survey

about program Open-ended survey

about 1st year of teaching

Teacher Survey Interviews with

principals

Page 9: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

9

Measures of central tendencies Teacher Survey: MTFs, TFs Teacher Leadership Inventory: MTFs, TFs-Cohort 1

ANOVAs Teacher Leadership Inventory for MTFs for statistically significant

changes over time Teacher Survey for TFs-Cohort 1 for statistically significant changes

over time

Cohen’s d Effect size reported for any statistically significant findings

Qualitative data Summarized using categories and themes

Utility:Credibility, attention to stakeholders, relevant information, timely and appropriate communicating and reporting

Feasibility:Project management, practical procedures, resource use

Accuracy:Justified conclusions and decisions, valid information, reliable information, explicit program and context descriptions, information management, sound designs and analyses, communication and reporting

Page 10: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

10

Importance of principal investigators Contacting MTFs and TFs to complete evaluation

pieces Communicating to adjust evaluation pieces during

project’s life

Maintaining data records for project’s life Well-organized data base to compare results over

time for impacts

Emma Espel

Page 11: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

11

What type of NOYCE project are you evaluating and what is the greatest challenge you encounter in conducting the evaluation?

What type of NOYCE project are you evaluating and what is the greatest challenge you encounter in conducting the evaluation?

Midwest Regional Robert Noyce Connections 

Page 12: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

12

What type of NOYCE project are you evaluating and what is the greatest challenge you encounter in conducting the evaluation?

Goals

Increase the personal connections among Midwest Noyce project members;

What type of NOYCE project are you evaluating and what is the greatest challenge you encounter in conducting the evaluation?

Goals

Increase the personal connections among Midwest Noyce project members;

Utilize these new and/or strengthened connections to share expertise and knowledge across Midwest Noyce projects;

Page 13: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

13

What type of NOYCE project are you evaluating and what is the greatest challenge you encounter in conducting the evaluation?

Goals

Increase the personal connections among Midwest Noyce project members;

Utilize these new and/or strengthened connections to share expertise and knowledge across Midwest Noyce projects;

Support the professional development of Noyce Scholars; and

What type of NOYCE project are you evaluating and what is the greatest challenge you encounter in conducting the evaluation?

Goals

Increase the personal connections among Midwest Noyce project members;

Utilize these new and/or strengthened connections to share expertise and knowledge across Midwest Noyce projects;

Support the professional development of Noyce Scholars; and

Build a community of practice among Noyce Scholars.

Page 14: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

14

What type of NOYCE project are you evaluating and what is the greatest challenge you encounter in conducting the evaluation?

To what extent do university‐related Noyce personnel, K‐12 Noyce Teacher Scholars, and conference participants expand their connections as a result of participating in project sponsored activities?

What type of NOYCE project are you evaluating and what is the greatest challenge you encounter in conducting the evaluation?

To what extent do university‐related Noyce personnel, K‐12 Noyce Teacher Scholars, and conference participants expand their connections as a result of participating in project sponsored activities?

Page 15: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

15

What type of NOYCE project are you evaluating and what is the greatest challenge you encounter in conducting the evaluation?

To what extent do university‐related Noyce personnel, K‐12 Noyce Teacher Scholars, and conference participants expand their connections as a result of participating in project sponsored activities?

What measures or protocols do you use and for what purposes?

Page 16: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

16

What measures or protocols do you use and for what purposes?

Quantitative and Qualitative

Online surveys

Needs assessments

Conference evaluation

Activity satisfaction 

What measures or protocols do you use and for what purposes?

Quantitative and Qualitative

Online surveys

Needs assessments

Conference evaluation

Activity satisfaction 

Social media: LinkedIn™ Group

Track group characteristics

Page 17: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

17

What methods do you use to report your findings and how useful is that perceived?

Needs assessment report

Annual report

Image from Chatty, D., Baas, S., & Fleig, A. (2003). Participatory processes towards co‐management of natrual resources in pastoral areas of the Middle East. Food and Agriculture Organization of the United Nations.

What methods do you use to report your findings and how useful is that perceived?

Needs assessment report

Annual report

Image from Chatty, D., Baas, S., & Fleig, A. (2003). Participatory processes towards co‐management of natrual resources in pastoral areas of the Middle East. Food and Agriculture Organization of the United Nations.

Page 18: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

18

Which of the Program Evaluation Standards do you feel has the greatest application in the NOYCE project evaluation? Why?

Yarbrough, D.B., Shulha, L.M., Hopson, R.K., and Caruthers, F.A. (2011). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users, 3rd ed. Mountain View, CA: Sage.

Which of the Program Evaluation Standards do you feel has the greatest application in the NOYCE project evaluation? Why?

Utility

Yarbrough, D.B., Shulha, L.M., Hopson, R.K., and Caruthers, F.A. (2011). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users, 3rd ed. Mountain View, CA: Sage.

Page 19: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

19

Which of the Program Evaluation Standards do you feel has the greatest application in the NOYCE project evaluation? Why?

Utility

Yarbrough, D.B., Shulha, L.M., Hopson, R.K., and Caruthers, F.A. (2011). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users, 3rd ed. Mountain View, CA: Sage.

Which of the Program Evaluation Standards do you feel has the greatest application in the NOYCE project evaluation? Why?

Utility

Yarbrough, D.B., Shulha, L.M., Hopson, R.K., and Caruthers, F.A. (2011). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users, 3rd ed. Mountain View, CA: Sage.

Page 20: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

20

Which of the Program Evaluation Standards do you feel has the greatest application in the NOYCE project evaluation? Why?

Utility

Yarbrough, D.B., Shulha, L.M., Hopson, R.K., and Caruthers, F.A. (2011). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users, 3rd ed. Mountain View, CA: Sage.

Which of the Program Evaluation Standards do you feel has the greatest application in the NOYCE project evaluation? Why?

Utility

Yarbrough, D.B., Shulha, L.M., Hopson, R.K., and Caruthers, F.A. (2011). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users, 3rd ed. Mountain View, CA: Sage.

Page 21: NOYCE PROJECT EVALUATIONscimath.unl.edu/noyce/conferences/2014/_files/slides/RMC... · 2015-03-10 · 10/27/2014 1 NOYCE PROJECT EVALUATION John T. Sutton, Ph.D. RMC Research Corporation

10/27/2014

21

If you were giving advice to another Noyce Project that was thinking about how to improve or enhance their evaluation, what would that be?

If you were giving advice to another Noyce Project that was thinking about how to improve or enhance their evaluation, what would that be?

Be willing to explore other options to creatively and usefully target research questions using available resources (e.g., social media as an evaluator tool)