Measuring Change: Measuring Change: Evaluation Plan forEvaluation Plan for

Middle Ground ProjectMiddle Ground Project

Educational Outfitters (2011)Educational Outfitters (2011)

Educational OutfittersEducational Outfitters

• Shanna FutralShanna Futral– Evaluation Field DirectorEvaluation Field Director

• Todd BraegerTodd Braeger– Senior Evaluator and AnalystSenior Evaluator and Analyst

Who We Are & What We Do

Why do evaluate?

Future US DOE Funding

Results from grant-funded projects guide the development of, and allocation to,

existing and new US DOE discretionary grants (including the Presidential

Academies grant that funds the Middle Ground Project).

Government Performance and Government Performance and Results Act (GPRA) & Continuation Results Act (GPRA) & Continuation

Award RequirementsAward Requirements

In return for receiving a PA grant, US DOE In return for receiving a PA grant, US DOE requires rigorous evaluation.requires rigorous evaluation.

Therefore, your completion of the Therefore, your completion of the assessments is assessments is requiredrequired. .

Continuous Improvement CycleContinuous Improvement Cycle

YOU are Middle Ground!YOU are Middle Ground!

TeachersTeachers’’ data will aid Middle Ground staff in data will aid Middle Ground staff in improving project offerings: training topics, improving project offerings: training topics,

presenters, materials, support activities, etc. presenters, materials, support activities, etc.

All Submitted Data is All Submitted Data is ConfidentialConfidential

No one other than the Project Staff and No one other than the Project Staff and

Evaluation Team will ever see your individual data.Evaluation Team will ever see your individual data.

What are we measuring?What are we measuring?

• Attendance (required)Attendance (required)

• Expectations and Satisfaction with Project Expectations and Satisfaction with Project Trainings and MaterialsTrainings and Materials

• Knowledge of History and Civics (passing score Knowledge of History and Civics (passing score and 20% gain each year)and 20% gain each year)

• Instructional Approach (20% gain each year)Instructional Approach (20% gain each year)

Teaching History & Civics Teaching History & Civics

Experts agree that thinking skills must be Experts agree that thinking skills must be integrated into the teaching of history and integrated into the teaching of history and civics, not developed as a separate skill.civics, not developed as a separate skill.

Last 30 years (big push in last 10), Last 30 years (big push in last 10), ““historical thinking skillshistorical thinking skills”” or or ““Habits of the Habits of the MindMind”” have been popping up in curriculum have been popping up in curriculum standards and assessments.standards and assessments.

““History is about facts in much the same way History is about facts in much the same way that reading is about the alphabet: Facts (and that reading is about the alphabet: Facts (and letters) are essential building blocks; without letters) are essential building blocks; without

them you cannot do history (or read). But, just them you cannot do history (or read). But, just as reading necessitates looking at how the as reading necessitates looking at how the letters and words stand in relation to one letters and words stand in relation to one another (so that you see the difference another (so that you see the difference

between a horse chestnut and a chestnut between a horse chestnut and a chestnut horse), thinking historically requires going horse), thinking historically requires going

beyond chronology or chronicle and looking at beyond chronology or chronicle and looking at the relations that the facts bear to one the relations that the facts bear to one

another.another.””

- Spoehr & Spoehr, 1994, p. 72- Spoehr & Spoehr, 1994, p. 72

Project-Level Changes Project-Level Changes notnot

Individual ChangesIndividual Changes

For For federalfederal reports: reports: All participating All participating teachersteachers’’ data are lumped together for data are lumped together for

analysis and reporting purposes.analysis and reporting purposes.

Measuring ChangeMeasuring Change

Traditional MCQsTraditional MCQs• Pre/BaselinePre/Baseline• PostPost

Guided Self-AssessmentsGuided Self-Assessments• Pre/BaselinePre/Baseline• Post: Then/NowPost: Then/Now

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

Pre Post - Then Post - Now

ContentContent

• Primary & Secondary Sources Quiz (traditional pre/post)Primary & Secondary Sources Quiz (traditional pre/post)

• Multiple Choice Answer questions (traditional pre/post)Multiple Choice Answer questions (traditional pre/post)

• Essay (guided self-assessment)Essay (guided self-assessment)

• Core Content Statements (guided self-assessment)Core Content Statements (guided self-assessment)

• Completed by TeacherCompleted by Teacher

Essay Instructions

Understanding and Responsiveness

Analysis and Reasoning

Supporting Evidence

Sequencing (5 paragraph format)

Essay Instructions

PRE

• Type your essay in Word. Save to your hard-drive. Then copy and paste it into the online system.

• You will need your original essay on hand to complete the post-assessment.

• Do not spend too much time on this!

Essay InstructionsTHEN/NOW

Do you feel that your original essay needs improvement OR does it well reflect how you would currently answer the essay question? (Check the ONE statement below that best describes how you feel.)

(1) I AM satisfied with my original essay and changing or modifying it would NOT demonstrate an improvement. Explain why.

(2) I am NOT satisfied with my original essay and changing or modifying it WOULD demonstrate an improvement in my understanding. Please revise your essay.

Content StatementsContent Statements’’ Rating Scales Rating Scales

PRE and THEN/NOW: PRE and THEN/NOW: • To what extent do you teach this content concept? To what extent do you teach this content concept? • What constraints, if any, influence the level to which you What constraints, if any, influence the level to which you

teach this content concept? teach this content concept? • How comfortable are you with teaching this content How comfortable are you with teaching this content

concept? concept?

Only NOW:Only NOW: • Which statement below best describes what your current Which statement below best describes what your current

(NOW) lesson plans reflect or include related to this content (NOW) lesson plans reflect or include related to this content concept? concept?

• Which statements below are true about your current (NOW) Which statements below are true about your current (NOW) understanding of this content concept and, if you teach it, understanding of this content concept and, if you teach it, your integration of it into your lessons?your integration of it into your lessons?

Note about Rating ScalesNote about Rating Scales

Numbers are relative, not points. Numbers are relative, not points.

For example:For example:

To what extent do you teach this content concept? (Check ONE.)To what extent do you teach this content concept? (Check ONE.)3=I dedicated enough time to this material to give my 3=I dedicated enough time to this material to give my

students a good understanding of it.students a good understanding of it.2=I briefly discussed/covered it, but more as a side note or 2=I briefly discussed/covered it, but more as a side note or

point of interest.point of interest.1=I am planning to teach it to my students.1=I am planning to teach it to my students.0=I have not taught it to my students.0=I have not taught it to my students.

Instructional ApproachInstructional Approach

• Guided self-assessment completed by TeacherGuided self-assessment completed by Teacher

• Guided assessment completed by Instructional Guided assessment completed by Instructional Specialist/Mentor for each TeacherSpecialist/Mentor for each Teacher

Instructional StatementsInstructional Statements

• Specific, observable behavior(s) the Teacher Specific, observable behavior(s) the Teacher engages in as part of his/her regular classroom engages in as part of his/her regular classroom instruction.instruction.

• For example: Student’s Background: I consider each student’s educational, cultural, and community background knowledge, skills, experiences, and perceptions. Examples include: Each student’s background knowledge and skill-level; potential misconceptions and bias of the subject matter; and their ability to use technology.

PRE and THEN/NOW:PRE and THEN/NOW:

• The level of the teacher's skills and knowledge to The level of the teacher's skills and knowledge to effectively use this strategy.effectively use this strategy.

• The teacher's level of classroom implementation of The teacher's level of classroom implementation of this strategy.this strategy.

Instructional StatementsInstructional Statements’’ Rating ScalesRating Scales

COHORT BASELINE (Now) (Pre) COHORT BASELINE (Now) (Pre) Participant Information Participant Information Self-Assessment of Instructional Approach (Self-Assessment of Instructional Approach (““Skills AssessmentSkills Assessment””))Content and Thinking Assessment (Content and Thinking Assessment (““Content AssessmentContent Assessment”” ) )

END-OF-SUMMER (Post: Then/Now)END-OF-SUMMER (Post: Then/Now)Content and Thinking Assessment End-of-Course Follow-Up QuestionnaireContent and Thinking Assessment End-of-Course Follow-Up Questionnaire

AFTER EACH TRAINING SESSIONAFTER EACH TRAINING SESSION Teacher Expectation & Satisfaction Survey (TESS)Teacher Expectation & Satisfaction Survey (TESS)

END-OF-YEAR (January 2012) (Post: Then/Now)END-OF-YEAR (January 2012) (Post: Then/Now)Content and Thinking Assessment End-of-Course Follow-Up QuestionnaireContent and Thinking Assessment End-of-Course Follow-Up QuestionnaireSkills Assessment End-of-Course Follow-Up Questionnaire Skills Assessment End-of-Course Follow-Up Questionnaire MentorMentor’’s Log & Assessment of Teachers Log & Assessment of Teacher’’s Instructional Approachs Instructional Approach

Schedule of AssessmentsSchedule of Assessments

Instructions for Completing Online AssessmentsInstructions for Completing Online Assessments

• Summer URLs are listed on the handout. Summer URLs are listed on the handout.

• Throughout the year, some URLs will be emailed directly to you from Throughout the year, some URLs will be emailed directly to you from grantsherpa@yahoo.com..

• If emailed: URL is uniquely tied to each email address. Do not forward. If emailed: URL is uniquely tied to each email address. Do not forward.

• If you experience difficulties accessing the online form, delete the If you experience difficulties accessing the online form, delete the ““cookiescookies”” from your Internet browser.from your Internet browser.

• Must be completed in one sitting.Must be completed in one sitting.

• Do not use any resources (i.e., books, websites, notes, peers). Do not use any resources (i.e., books, websites, notes, peers).

• Push Push ““DoneDone”” at the bottom of each survey. A at the bottom of each survey. A ““Thank YouThank You”” will pop up if it will pop up if it was submitted successfully.was submitted successfully.

Assessment WindowAssessment Window

60 to 120-minutes window to complete assessments when done on-60 to 120-minutes window to complete assessments when done on-site.site.

OROR

2-week window to complete each assessment when emailed to you.2-week window to complete each assessment when emailed to you.

It is very important that you complete the assessment It is very important that you complete the assessment on timeon time in in order for your gains to be accurately and fully documented.order for your gains to be accurately and fully documented.

Need Assistance?Need Assistance?

If you have If you have technicaltechnical difficulties, email difficulties, email Shanna: Shanna: grantsherpa@yahoo.com. .

Type Type ““MGP Evaluation QuestionMGP Evaluation Question”” as the as the email subject line. email subject line.