America’s History in the Making, Volume...

America’s History in the Making, Volume II

Evaluation Report

Prepared for Multnomah Education Service District

11611 NE Ainsworth Circle Portland, OR 97220

Oregon Public Broadcasting

7140 SW Macadam Avenue Portland, OR 97219

Prepared by RMC Research Corporation

111 SW Columbia Street, Suite 1200 Portland, OR 97201

September 2008

America’s History in the Making, Volume II

Evaluation Report Prepared for Kelvin Webster Multnomah Education Service District 11611 NE Ainsworth Circle Portland, OR 97220 Catherine Stimac Oregon Public Broadcasting 7140 SW Macadam Avenue Portland, OR 97219 Prepared by Chandra K. Lewis RMC Research Corporation 111 SW Columbia Street, Suite 1200 Portland, OR 97201 September 2008

iii

Contents

Exhibits ............................................................................................................................v

Introduction ......................................................................................................................1 Methods...........................................................................................................................3

Evaluation Design.......................................................................................................6 Instrument Development ............................................................................................6

Development Team Survey...................................................................................6 Teacher Surveys...................................................................................................7 Content Assessments ...........................................................................................7 Scoring Guide .......................................................................................................8 Teacher Interview..................................................................................................9

Participants.................................................................................................................9 Data Collection .........................................................................................................13 Data Analysis ...........................................................................................................13 Limitations ................................................................................................................15 Human Subjects Protection ......................................................................................16

Results...........................................................................................................................17 Provision of Professional Development....................................................................17

Videos .................................................................................................................21 Hands-On Activities.............................................................................................22

Distribution of America’s History in the Making, Volume II .......................................24 Increased Content Knowledge .................................................................................24

Subgroup Analysis ..............................................................................................27 Interviews............................................................................................................35 Improvements to the Online Content Assessment ..............................................36

Increased Ability to Interpret Primary and Secondary Sources ................................36 Increased Digital Technology Skills ..........................................................................37 Earned Credits..........................................................................................................38 Improved Pedagogy and Increased Student Achievement.......................................38

Discussion .....................................................................................................................47 Improving America’s History in the Making, Volume II..............................................48 Improving All Teaching American History Grants .....................................................49

Appendix A: Evaluation Briefs Appendix B: Development Team Survey Version 1 & 2 Appendix C: Teacher Presurvey and Postsurvey Appendix D: Content Assessments Versions 1, 2, & 3 Appendix E: Scoring Guide Appendix F: Interview Protocol

v

Exhibits

Exhibit 1 America’s History in the Making, Volume II, Logic Model.............................4

Exhibit 2 Distribution of Assessment Items .................................................................8 Exhibit 3 Characteristics of Recruited Teachers........................................................10

Exhibit 4 Reasons for Not Participating.....................................................................11 Exhibit 5 Characteristics of Cohort 1 and Cohort 2 Teachers After Attrition..............12

Exhibit 6 Testing Dates .............................................................................................13 Exhibit 7 Evaluation Questions, Data Sources, and Outcome Indicators..................14

Exhibit 8 Characteristics of Study Participants..........................................................17 Exhibit 9 Pretest Scores by Version..........................................................................24

Exhibit 10 Pretest and Posttest Scores.......................................................................25 Exhibit 11 Pre-Post Scores by Cohort.........................................................................26

Exhibit 12 Pre-Post Scores by Gender........................................................................27 Exhibit 13 Pre-Post Scores by Site .............................................................................28

Exhibit 14 Pre-Post Scores by History Course Type...................................................29 Exhibit 15 Pre-Post Scores by Grade Level Taught....................................................30

Exhibit 16 Pre-Post Scores by Years Taught ..............................................................31 Exhibit 17 Pre-Post Scores by Years History Taught..................................................32

Exhibit 18 Pre-Post Scores by “Highly Qualified” Status.............................................33 Exhibit 19 Pre-Post Scores by Degree........................................................................34

Exhibit 20 Pre-Post Scores by College Credits ...........................................................35 Exhibit 21 Pre-Post Results: Preparedness to Teach History at Assigned Level ........39

Exhibit 22 Pre-Post Results: Integration of History With Other Subjects.....................40 Exhibit 23 Pre-Post Results: Provision of History Instruction That Meets

District, State, or National Standards .........................................................41

Exhibit 24 Posttest Results: Chronological Thinking ...................................................42

Exhibit 25 Posttest Results: Historical Comprehension...............................................43 Exhibit 26 Posttest Results: Historical Analysis and Interpretation .............................44

Exhibit 27 Posttest Results: Research Capabilities.....................................................44 Exhibit 28 Posttest Results: Issues Analysis and Decision-Making Skills ...................45

RMC Research CorporationPortland, OR 1

Introduction

A consortium of local education service agencies including Multnomah Education Service District (Portland, Oregon), Malheur Education Service District (Malheur, Oregon), Grant Wood Area Education Agency (Cedar Rapids, Iowa), Riverside County Office of Education (Riverside, California), Area Cooperative Educational Services (New Haven, Connecticut), and Bedford and Coffee Counties Department of Education (rural Tennessee), in partnership with Oregon Public Broadcasting (OPB), the National Center for History in Schools (NCHS), and the Organization of American Historians (OAH) received a Teaching American History grant from the U.S. Department of Education for the development of America’s History in the Making, Volume II, a multimedia professional development workshop series that offers teachers rich historical content from the post-Reconstruction period onward that is aligned with classroom curricula created by the National Center for History in Schools. America’s History in the Making, Volume II, is an extension of America’s History in the Making, Volume I (funded by a 2004 Teaching American History grant), which covers American history prior to the Reconstruction and is currently available nationally free of charge online and through Annenberg Media Channel.

An advisory board of teachers, professional development providers, and leading experts in history guided the development, production, and implementation of this integrated workshop series composed of videos, print materials, hands-on activities, interactive website, and a workshop facilitator’s guide. During the grant period teachers in the consortium of local education service agencies participated in this professional development workshop series. Teacher content knowledge was evaluated pre- and postworkshop. At the conclusion of the evaluation the workshop was made available nationally free of charge online and through Annenberg Media Channel. Through this broad distribution, teachers in the United States and abroad have unlimited opportunity and access to delve deeply into selected biographies and events in American history.


Methods

The logic model for America’s History in the Making, Volume II, details the inputs, outputs, outcomes, and long-term impacts of the project (see Exhibit 1). The 4 goals of America’s History in the Making, Volume II, were to develop a multimedia professional development workshop series that helps teachers:

Goal 1: Learn more about the events and people that have shaped American history.

Goal 2: Interpret primary and secondary sources, maps, charts, and other visuals more effectively.

Goal 3: Build skills for using digital technologies to research American history resources.

Goal 4: Secure credits toward achieving “highly qualified” teaching status. As a result of participating in the series teachers were expected to improve their knowledge, understanding, and appreciation of traditional American history; better instruct their students; and raise student achievement. Four objectives corresponded to the project’s goals:

Objective 1: Provide teachers with at least 30 contact hours of professional development (worth at least 2 to 4 academic credits) on American history through the Reconstruction by creating a content-rich, 10-unit multimedia workshop series composed of videos, print materials, interactive website, and hands-on activities.

Objective 2: Develop a facilitator’s guide to accompany the series. Objective 3: Implement the workshops with at least 20 teachers in each of the

participating local education service areas and evaluate their effectiveness. Objective 4: At the conclusion of the 3-year grant period, make the series

available nationally free of charge online and through Annenberg Media Channel.

4 America’s History in the Making, Volume II, Evaluation Report

Exhibit 1 America’s History in the Making, Volume II, Logic Model

Inputs (What the Project Invests)

Project Outputs (What the Project Creates)

Outcomes (Short-Term Results)

Impacts (Ultimate Impacts)

Lead Partners Multnomah County ESD Malheur ESD (Malheur, OR) Grant Wood AEA (Cedar

Rapids, IA) Riverside COE (Riverside,

CA) Area Cooperative Educational

Services (New Haven, CT) Bedford and Coffee Counties

Department of Education (rural Tennessee)

Oregon Public Broadcasting (OPB)

National Center for History in Schools (NCHS)

Organization of American Historians (OAH)

Project Leadership Steering Team Advisory Board Project Director Executive Producers

Increased Content Knowledge

Teachers learn about the events and people that have shaped American history from Reconstruction onward.

(Goal 1)

Professional Development The project creates America’s History in the Making, Volume II, a 10-unit, content rich multimedia professional development series based on historical content from the Reconstruction onward. Twenty teachers from each of the partnering local education service areas participate in America’s History in the Making, Volume II, workshop series.

(Objective 1 & 3)

Earned Credits Teachers secure credits toward achieving “highly qualified” teaching status.

(Goal 4)

Increased Digital Technology Skills

Teachers increase their skills using digital technology to research American history resources.

(Goal 3)

Facilitator’s Guide The project creates a facilitator’s guide to accompany the series. The guide is to be provided to future facilitators of America’s History in the Making, Volume II.

(Objective 2) Increased Student

Achievement Students benefit from their teachers’ improved content knowledge and instructional practices, as demonstrated by improved scores on history assessments.

Improved Pedagogy Teachers improve their instructional practices as a result of improved content knowledge, ability to interpret primary source documents, and ability to use digital technology to research American history.

Nationally Available Series All of the materials needed to lead or participate in the workshop series (in group and individual learning settings) are to be available online for 10 years free of charge. Video will be streamed on demand or can be ordered.

(Objective 4)

Increased Ability to Interpret Primary and Secondary Sources

Teachers increase their ability to interpret primary and secondary sources, maps, charts, and other visuals more effectively.

(Goal 2)


RMC Research Corporation, in cooperation with the consortium of America’s History in the Making, Volume II, partners designed the following evaluation questions related to the logic model presented in Exhibit 1:

Provision of Professional Development—How did the development team function and how could Oregon Public Broadcasting improve the process for America’s History in the Making, Volume II? How successful was the project in terms of implementing the workshop series with at least 20 teachers in each of the consortium education service areas? To what extent were the participating teachers satisfied with America’s History in the Making, Volume II? To what extent did the participating teachers think the individual components of the professional development (i.e., videos, print materials, hands-on activities) met their intended purposes? To what extent did the participating teachers think the multimedia approach is an effective way of helping teachers learn American history content?

Development of a Facilitator’s Guide—To what extent did the facilitators think the facilitator’s guide fulfilled its intended purpose? (This question was addressed in the report America’s History in the Making, Volume I: Professional Development Provider Interviews.1)

Distribution of America’s History in the Making Volume I—At the conclusion of the project, was America’s History in the Making, Volume II, made nationally available?

Increased Content Knowledge—To what extent did America’s History in the Making, Volume II, increase the American history content knowledge of the participating teachers with respect to their understanding of the people and events that that shaped American history from the post-Reconstruction period onward?

Increased Ability to Interpret Primary and Secondary Sources—To what extent did America’s History in the Making, Volume II, increase the participating teachers’ appreciation of the use of primary source materials? To what extent did the use of America’s History in the Making, Volume II, help teachers understand how to interpret primary and secondary sources, maps, charts, and other visuals in both learning and teaching history content?

Increased Digital Technology Skills—To what extent did America’s History in the Making, Volume II, increase the participating teachers’ skills using digital technology to research American history resources?

Earned Credits—To what extent did America’s History in the Making, Volume II, assist participating teachers in securing credits toward achieving “highly qualified” teaching status, a requirement of the No Child Left Behind Act?

Improved Pedagogy and Increased Student Achievement—How successful was Americas’ History in the Making, Volume II, in terms of improving teachers’ instructional strategies? To what extent did America’s History in the Making,

1Carlton, P.R. & Lewis, C.K. (2007, April). America’s History in the Making, Volume I: Professional Development Provider Interviews. Portland, OR: RMC Research Corporation.


Volume II, increase the likelihood that teachers would use the National Center for History in the Schools’ historical thinking skills in their classroom?

Evaluation Design

The program evaluation was both formative and summative in nature. The formative evaluation comprised 2 components: monitoring the progress of the project toward meeting specific program objectives and examining how the development team (i.e., project staff who contributed to the creation of the professional development series) function while creating America’s History in the Making, Volume II. Throughout the 3 years of the project RMC Research communicated the findings of the formative evaluation to project staff to improve the development and implementation processes (see the evaluation briefs in Appendix A). The summative evaluation occurred during the final year of the project and involved a quasi-experimental research study of the effectiveness of America’s History in the Making, Volume II, in improving the American history content knowledge of the participating teachers.

The quasi-experimental design included 2 groups and 3 waves of measurement. In the first phase of the design both groups (i.e., Cohort 1 and Cohort 2) were administered pretests. Cohort 1 participated in America’s History in the Making, Volume II, in fall 2007 and completed the posttest while the comparison group completed a second pretest. Cohort 2 participated in America’s History in the Making, Volume II, in winter 2007 and completed the posttest while Cohort 1 completed a second posttest. The presumption was that if the increase in content knowledge were due to the professional development, the Cohort 1 teachers would show a statistically significant increase from the pretest to the first posttest and a minimal increase in content knowledge from the first posttest to the second posttest; in contrast, the Cohort 2 teachers would show no increase in knowledge from the first pretest to the second pretest but would show a significant increase from the second pretest to the posttest. This design is identical in structure to its randomized experimental version except for the lack of random assignment to group. It is superior to the simple pre-post nonequivalent groups design, and because this design ensures that all participants eventually participate in the professional development, it is probably one of the most ethically feasible quasi-experiments.

Instrument Development

RMC Research used a combination of qualitative and quantitative evaluation instruments to address the evaluation questions. This section describes the data collection instruments.

Development Team Survey

During the first year of the project RMC Research, in collaboration with Oregon Public Broadcasting and Multnomah Education Service District, created the development team


survey, which gathered information on the functioning of the team, enabling Oregon Public Broadcasting to monitor the quality of the development process and to make improvements if needed. The survey included questions on the teams’ goals and objectives, organization and procedures, preparation and participation, leadership, cooperation and conflict, communication, and progress. Respondents rated their team in these areas using a scale from 1 (strongly disagree) to 4 (strongly agree). In fall 2006 the evaluation team added questions regarding the team members’ workloads, omitted questions about preparing for the development of the series, and reworded questions to improve clarity. Both versions of the survey appear in Appendix B.

Teacher Surveys

RMC Research collaborated with project staff to create 2 versions of the teacher survey. The presurvey asked teachers how often they had their students use historical thinking skills as defined by the National Center for History in the Schools, why they decided to participate in the project, how well prepared they felt to teach history, how often they used primary or secondary source documents, and if they used digital technology to research history. The postsurvey included the same questions as the presurvey plus questions on the value of the professional development, the quality of the materials (i.e., video, print materials, hands-on activities), the usefulness of the multimedia approach, and ways in which the professional development could be improved. The postsurvey also asked teachers if they had received other professional development since they began participating in the project and whether America’s History in the Making, Volume II,had helped them earn credits toward “highly qualified” status. The teacher surveys appear in Appendix C.

Content Assessments

RMC Research collaborated with project staff to create the content assessments, which measured the teachers’ knowledge of the content covered by the professional development. Project staff initiated the process by selecting 15 of the approximately 24 themes covered in America’s History in the Making, Volume II, to include in the content assessments. These themes reflected the most important content teachers should know after participating in the professional development. Project staff then identified the primary source documents that corresponded with the chosen themes and developed prompts that directed the teachers to write about some aspect of each theme with respect to the corresponding primary source.

As Exhibit 2 shows, RMC Research reviewed the items and distributed them among the 3 content assessment versions (Versions 1, 2, and 3) so that each version included approximately the same number of items and primary source documents (which appeared in the workshop facilitator’s guide or in the participants’ print materials), a range of items spread evenly across the 9 content units, and the same number of items at each difficulty level. Three versions of the content assessment were created to control for endogenous change, which occurs when the pretest and posttest are the same and the pretest influences scores on the posttest. For example, if the same


assessment administered as the pretest were administered as the posttest, the participants’ familiarity with the test might inflate their posttest scores. Therefore, participants were randomly assigned to take 1 of the 3 versions for the pretest and another for the posttest. The content assessments appear in Appendix D.

Exhibit 2 Distribution of Assessment Items

Primary Source Document Location of Primary Source Document Unit Difficulty

Version 1

Corn Withered by Heat Photograph and Wilber Republican Excerpt

Guide and print materials

13 2

Spinning Room Photograph Print materials 15 2

Speech by Henry Cabot Lodge Print materials 16 3

Letter to the Editor of the Pittsburgh Courier Guide 19 1

The Port Huron Statement Guide 20 3

Version 2

Immigrants Arriving Cartoon and Photograph Guide 14 3

Prohibition Bust Picture Print materials 15 1

Down and Out in the Great Depression: Letters From the Forgotten Man

Guide 18 3

March Photograph Print materials 19 2

Patrick J. Buchanan Keynote Address, 1992 Republican National Convention

Print materials 21 2

Version 3

Strike Photograph Print materials 14 2

The Real White Man’s Burden Letter Guide 16 1

Studs Terkel Interview With Gardiner C. Means Print materials 18 2

United Farmworkers Photograph and Judgment and Injunction From Mendez et al. v. Westminister School District of Orange County

Print materials 20 3

Chinese Tourist Break in Front of Starbucks Coffee Shop in Shanghai Photograph

Guide 21 3

Note. Difficulty scale: 1 = easy; 3 = difficult.

Scoring Guide

RMC Research collaborated with members of the advisory board to create the scoring guide used to rate the participants’ responses on a scale from 0 (did not answer) to 4 (exceeded expectations). For a participant to receive full credit, the response had to demonstrate how the primary source document supported the theme; draw on relevant


historical knowledge to connect the primary source document to the theme; demonstrate an understanding of the theme; provide relevant historical knowledge of the era; provide an analysis of the primary source document; provide rich and detailed historical knowledge; question, critique, or extend the theme; and use the primary source document to provide an in-depth analysis of the era. The scoring guide in Appendix E indicates the criteria for the lower scores.

To ensure the rigor of the research study, the content assessments were blindly scored (i.e., the names of the participants and the pretest or posttest status were hidden) by a team of 8 teachers recruited by Multnomah Education Service District and selected by RMC Research. Selection was based on the number of years the applicant had taught history, the type of college degree earned, the type of history taught, and prior scoring experience.

Teacher Interview

RMC Research and project staff designed an interview protocol to elicit qualitative data regarding the teachers’ experiences in America’s History in the Making, Volume II. The questions, based on the teacher survey results, addressed the effect of the project on the teachers’ pedagogy, historical content knowledge, ability to interpret primary and secondary sources, and use of digital technology. The interview protocol appears in Appendix F.

Participants

Development team members (e.g., advisory board members, web developers, text writers, Oregon Public Broadcasting staff, videographers) included anyone involved in the development of the America’s History in the Making, Volume II. All development team members were eligible to complete the development team survey. RMC Research recruited the survey respondents by e-mail. In the first administration 76% (n = 35) of the surveys were returned, in the second administration 86% (n = 30) of the surveys were returned, and in the third administration 77% (n = 24) of the surveys were returned. In the second and third survey administrations the total number of possible respondents changed because as team members completed their tasks they ceased participation, and as new tasks emerged new development team members were recruited.

All participating teachers were recruited for America’s History in the Making, Volume II, from the target population of employed middle or high school teachers who taught American history as a stand-alone course or as part of a social studies or language arts curriculum in Multnomah Education Service District, Malheur Education Service District, Grant Wood Area Education Agency, Riverside County Office of Education, or Bedford and Coffee Counties Department of Education. Prior to recruiting teachers for the project Area Cooperative Educational Services opted out of the project. The 5 participating education service agencies served approximately 102 school districts and 600,000 students in kindergarten through Grade 12.


The 5 education service agencies used a variety of approaches to recruit teachers during the 2006–2007 school year. Steering team members (representatives of the education service agencies) personally recruited teachers, contacted district liaisons to distribute recruitment materials, and sent recruitment e-mails to teachers. Advisory board members (representatives of the partner organizations responsible for developing the workshop content) referred teachers to the project and distributed recruitment materials. RMC Research sent recruitment materials to teacher participants in past projects who had requested to be informed of future professional development opportunities. The goal was to recruit 20 teachers from each education service agency RMC Research encouraged the education service agencies to recruit more than 20 teachers to account for attrition.

Project staff recruited 112 teachers: 44 from Multnomah Education Service District, 5 from Malheur Education Service District, 18 from Grant Wood Area Education Agency, 24 from Riverside County Office of Education, and 21 from Bedford and Coffee Counties Department of Education. Exhibit 3 shows the demographic characteristics of the recruited teachers. Most of the teachers were Caucasian (94%), taught history as a stand-alone course (65%), taught high school (70%), held a Master’s degree (56%), and were “highly qualified” to teach history (82%). The teachers had taught an average of 10 years (minimum 0 years, maximum 32 years; SD = 8.31) and had taught history an average of 8 years (minimum 0 years, maximum 32 years; SD = 7.79). The teachers had taken an average of 32 college credits in history (minimum 0 credits, maximum 77 credits; SD = 28.73).

Exhibit 3 Characteristics of Recruited Teachers

Characteristic M or %

Gender

Female 58%

Male 42%

Race/ethnicity

Caucasian 94%

African American 3%

Hispanic 2%

Asian 1%

Native Hawaiian or Pacific Islander 1%

Teach history as a stand-alone course

Yes 65%

No 35%

(exhibit continues)


Exhibit 3 (continued)


Grade level taughta

Elementary (Grades 1–5) 2%

Middle (Grades 6–8) 30%

High (Grades 9–12) 70%

Years taught 9.86

Years taught history 7.67

“Highly qualified” to teach history

Yes 82%

No 18%

Education

Bachelor degree 43%

Master degree 56%

College credits earned in history 32 Note. n = 112. aTeachers could indicate more than one grade.

In fall 2007, prior to the start of the professional development, 20 teachers informed RMC Research that they would not be able to participate America’s History in the Making, Volume II. Exhibit 4 outlines the reasons teachers gave for nonparticipation. Many teachers cited some type of scheduling conflict.

Exhibit 4 Reasons for Not Participating

Reason Number

No reason indicated 6

Family obligations 1

Moved out of the country 1

Started a Saturday job 1

Recruited to lead extracurricular activity 1

Other unspecified schedule conflict 10

Total 20

After attrition, 92 teachers participated in America’s History in the Making, Volume II, and the corresponding research study. The integrity of the internal validity of the study and the ability to draw causal conclusions from the research results were dependent on the similarity of the Cohort 1 and Cohort 2 participant characteristics. RMC Research


compared Cohort 1 and Cohort 2 to determine whether the remaining teachers were similar after attrition. As Exhibit 5 shows, the groups were similar (i.e., the characteristics did not significantly differ between groups). In addition, RMC Research ensured that teachers recruited from the same school were randomly assigned to a cohort as a group to avoid contamination between the groups within schools. RMC Research also asked teachers to report any professional development or training they received outside of this project while participating in this research study to allow for the examination of intervening factors.

Exhibit 5 Characteristics of Cohort 1 and Cohort 2 Teachers After Attrition

Cohort 1 Cohort 2

Characteristic M or % M or %

Gender

Female 65% 53%

Male 35% 47%

Race/ethnicity

Caucasian 95% 96%

African American 5% 0%

Hispanic 0% 7%

Asian 0% 2%

Native Hawaiian Pacific Islander 0% 0%


Yes 62% 67%

No 38% 33%

Grade level taughta

Elementary (Grades 1–5) 2% 0%

Middle (Grades 6–8) 30% 43%

High (Grades 9–12) 67% 57%

Years taught 10.16 9.84

Years taught history 8.25 7.71


Yes 86% 82%

No 14% 18%

Education

Bachelor degree 37% 46%

Master degree 63% 54%

College credits earned in history 34.33 28.98 Note. Treatment n = 43; comparison n = 49. aTeachers could indicate more than one grade. Mean differences assessed using independent t test. Categorical differences assessed using chi-square.


Data Collection

Data collection occurred throughout all 3 years of the project. RMC Research administered the development team survey via e-mail or online in April 2006, September 2006, and January 2007. Each of the data collection time points corresponded to crucial points in the development process of America’s History in the Making, Volume II. A random sample of both the Cohort 1 and Cohort 2 teachers were selected to be interviewed during the final year of the project. Semi-structured qualitative telephone interviews were conducted with 25 study participants (6 from Multnomah Education Service District, 4 from Malheur Education Service District, 5 from Grant Wood Area Education Agency, 5 from Riverside County Office of Education, and 5 from Bedford and Coffee Counties Department of Education).

All participating teachers were required to have completed at least one pretest and one posttest (each comprising a content assessment and a teacher survey) so that changes in dependent variables (e.g., content knowledge, use of primary source documents) over time could be reported. To complete the tests, the teachers accessed a website created by RMC Research. The teachers logged on with a username and password supplied by RMC Research to protect the confidentiality of the teachers’ data. Exhibit 6 shows the dates that teachers were tested by site and cohort. The test dates differed slightly due to complications related to coordinating the schedules of multiple sites and facilitators. A small number of teachers unable to participate on the designated test date were allowed to take a makeup test at the testing site (the education service agency in their area) or at home.

Exhibit 6 Testing Dates

Cohort/Test Year Multnomah Malheur Grant Wood

Riverside County

Bedford and Coffee Counties

Cohort 1 Pretest 2007 09/15 09/22 09/22 09/15 09/22

Posttest 2007 11/17 11/17 12/15 11/10 11/17

Second posttest 2008 03/08 03/08 04/12 03/08 03/08

Cohort 2

Pretest 2007 09/15 09/22 09/22 09/15 09/22

Second pretest 2008 01/12 01/12 01/05 01/26 01/19

Posttest 2008 03/08 03/08 04/12 03/08 03/08

Data Analysis

Exhibit 7 shows the evaluation questions, data sources, and outcome indicators used in the analysis of the data. Basic descriptive statistics were used to summarize the data


and statistical tests were conducted to compare the pre-post data. The independent variable was America’s History in the Making, Volume II, and the dependent variables were those variables that would hypothetically be affected by the project—including, but not limited to, teachers’ content knowledge and use of historical thinking skills. In the analysis chi-square tests were used for between-group comparisons when the dependent variable was categorical in nature (e.g., comparing the percentage of female participants recruited to the percentage that participated). Wilcoxon tests were used for within-group comparisons when the dependent variable was ordinal in nature (e.g., comparing the pre-post scores regarding the extent to which participants agreed with a statement on a scale from 0 to 4). Independent t tests were used for between-group comparisons when the dependent variable was interval in nature (e.g., comparing “highly qualified” teachers’ posttest scores to the posttest scores of teachers who were not “highly qualified”). Paired t tests were used for within-group comparisons when the dependent variable interval in nature (e.g., comparing “highly qualified” teachers’ pretest scores to their own posttest scores).

Exhibit 7 Evaluation Questions, Data Sources, and Outcome Indicators

Evaluation Question Data Source Outcome Indicator

How did the development team function and how could OPB improve the development process for America’s History in the Making, Volume II?

Development team surveys

Team members’ responses to questions regarding how the team could improve and OPB’s response to the team members’ feedback.

How successful was the project in terms of implementing the workshop series with at least 20 teachers in each of the consortium education service areas?

Teacher applications

The numbers of teacher recruits and participants in America’s History in the Making, Volume II.

To what extent were the participating teachers satisfied with America’s History in the Making, Volume II?

Teacher survey

Teachers’ responses to questions regarding their satisfaction with America’s History in the Making, Volume II.

To what extent did the participating teachers think the individual components of the professional development (i.e., videos, print materials, hands-on activities) met their intended purposes?

Teacher survey

Teachers’ responses to questions regarding the individual components of America’s History in the Making, Volume II.

To what extent did the participating teachers think the multimedia approach is an effective way of helping teachers learn American history content?

Teacher survey

Teachers’ responses to questions regarding the effectiveness of the multimedia approach.

At the conclusion of the project, was America’s History in the Making, Volume II, made nationally available?

Annenberg Media Channel

Checking the website to determine whether the series was made nationally available through the Annenberg Media Channel.

(exhibit continues)



Evaluation Question Data Source Outcome Indicator

To what extent did America’s History in the Making, Volume II, increase the American history content knowledge of the participating teachers with respect to their understanding of the people and events that that shaped American history from the post-Reconstruction period onward?

Content assessment

Teacher survey and interview

Differences in the teacher pre-post content assessment scores.

Teachers’ responses to questions regarding how their content knowledge changed as a result of participation in America’s History in the Making, Volume II.

To what extent did America’s History in the Making, Volume II, increase the participating teachers’ appreciation of the use of primary source materials?

Teacher survey

Teachers’ responses to questions regarding their appreciation for the use of primary source materials.

To what extent did the use of America’s History in the Making, Volume II, help teachers understand how to interpret primary and secondary sources, maps, charts, and other visuals in both learning and teaching history content?


Teachers’ responses to questions regarding their ability to interpret primary and secondary source documents.

To what extent did America’s History in the Making, Volume II increase the participating teachers’ skills using digital technology to research American history resources?


Teachers’ responses to questions regarding their use of digital technology.

To what extent did America’s History in the Making, Volume II, assist participating teachers in securing credits toward achieving “highly qualified” teaching status, a requirement of the No Child Left Behind Act?

Teacher survey

Teachers’ responses to questions regarding the number of credits earned as a result of participation.

How successful was America’s History in the Making, Volume II in terms of improving the participating teachers’ instructional strategies?


Teachers’ responses regarding changes to their teaching practices.

To what extent did America’s History in the Making, Volume II, increase the likelihood that teachers would use the National Center for History in the Schools’ historical thinking skills in their classroom?

Teacher survey

Teachers’ responses regarding the likelihood of using the historical thinking skills in their classroom.

Limitations

This research study had 4 limitations. First, the sample frame consisted of volunteers rather than a random sample of the entire population of teachers in the 5 educational service agencies. This fact limits the generalizability of the results to the entire population of American history teachers because teachers who volunteer for these types of programs might differ from the population of American history teachers at large. Second, due to scheduling complications across sites the assessments were not conducted on exactly the same dates (though the testing dates did occur at the same time point in the professional development); ideally, the teachers would have been tested on the same dates to control for any knowledge gained between the testing


periods. Third, the test results could have been affected by the difference in testing location for the small number of teachers who completed the assessment at home. Fourth, an experimental research design would provide stronger evidence with regard to the project’s impact verses the quasi-experimental design.

Human Subjects Protection

RMC Research carried out all necessary steps to obtain approval for the evaluation plan and instruments from the appropriate Institutional Review Board. RMC Research confirmed in fall 2006 that the U.S. Department of Education did not require an external review board for the Teaching American History grants. Thus RMC Research used its internal Human Protection Committee, which required RMC Research staff to complete a Confidentiality Review (for data collected with an identification number) and a Harm Reduction Review (for data of a sensitive nature.)


Results

Provision of Professional Development

How successful was the project in terms of implementing the workshop series with at least 20 teachers in each of the consortium education service areas?

Ninety-two teachers participated in America’s History in the Making, Volume II—short of the participation goal of 100 teachers across the 5 education service areas. Of the 92 teachers 39 were from Multnomah Education Service District, 5 were from Malheur Education Service District, 17 were from Grant Wood Area Education Agency, 16 were from Riverside County Office of Education, and 19 were from Bedford and Coffee Counties Department of Education. Exhibit 8 shows the demographic characteristics of the participating teachers. Most were Caucasian (97%), taught history as a stand-alone course (65%), taught in a high school (62%), had a Master’s degree (59%), and were “highly qualified” to teach history (84%). The teachers had taught an average of 10 years (minimum 0 years, maximum 32 years; SD = 8.71) and had taught history an average of 8 years (minimum 0 years, maximum 32 years; SD = 8.07). The teachers had taken an average of 32 college credits in history (minimum 0 credits, maximum 177 credits; SD = 28.29).

Exhibit 8 Characteristics of Study Participants


Gender

Female 59%

Male 41%

Race/ethnicity

Caucasian 97%

African American 2%

Hispanic 0%

Asian 1%

Native Hawaiian Pacific Islander 0%


Yes 65%

No 35%

(exhibit continues)



Grade level taught a

Elementary (Grades 1–5) 1%

Middle (Grades 6–8) 37%

High (Grades 9–12) 62%

Years taught 9.99

Years taught history 7.87


Yes 84%

No 16%

Education

Bachelor degree 41%

Master degree 59%

College credits earned in history 31.49 Note. n = 92. aTeachers could indicate more than one grade.

To what extent were the participating teachers satisfied with America’s History in the Making, Volume II?

The teacher survey asked participants why they had decided to participate in America’s History in the Making, Volume II. Respondents most commonly reported wanting to learn more about history (48%) and become a better teacher (47%). Others indicated that the stipend or credits they received for participation was a motivating factor (18%). Others hoped to obtain resources for their classroom such as primary source documents (14%), learn how to integrate technology into their teaching (13%), collaborate with other teachers (2%), or learn how to integrate history into their English courses (4%). Some (13%) had appreciated America’s History in the Making, Volume I, and wanted to continue with the series, whereas others simply liked studying history (2%) or wanted to participate in something different (2%). One participant wanted to share the information gained with colleagues.

Participants described America’s History in the Making, Volume II, as different than other professional development they had received. They commonly cited the focus on historical content rather than teaching practices (37%) and the opportunity to interact with colleagues, which resulted in rich discussions regarding the content and the use of the materials in the classroom (27%). These discussions were enhanced by the small-group atmosphere (2%) and the high-quality facilitator who was knowledgeable and ensured a positive environment in which everyone could share their ideas (12%). One participant explained that the single facilitator model—in contrast to programs that utilize several historians or experts—fostered a sense of continuity. Others deemed the


professional development distinct because it provided high-quality multimedia materials to use in the classroom (19%), employed a multimedia format (13%), emphasized the study of primary source documents (11%), included several sessions rather than just one (7%), provided a stipend (4%), focused on learning about lesser known individuals in history (3%), provided relevant information with regard to improving pedagogy (3%), was well organized (2%), provided time to study materials independently (2%), and focused on themes (2%).

The teacher survey respondents explained how these distinct characteristics of America’s History in the Making, Volume II, were the most valuable aspects of the professional development. According to 52% of the participants the discussion time with peers was valuable because the teachers were able to learn content and to create lesson plans, whereas others (38%) considered the most valuable aspect to be the wealth of useful materials they planned to use in the classroom. Participants also cited as valuable aspects of the program learning historical content (12%), the facilitator’s vast knowledge of the content (8%), studying the materials at home prior to each session (2%), and observing the facilitator modeling best teaching practices (2%). Two teachers’ comments summed up the benefits well:

“This was something that was very pertinent to my content area. I really feel that helped me explore and understand historical significances and historical skills. Something that was different about this format was that there were many times where there was an open format for discussion so that I could hear and learn new ideas from other teachers. Many other professional developments I've been to are a lot of one person talking and the rest listening. I'm glad that this development not only taught good teaching strategies but used them!”

“This professional development was a group effort that allowed me to express my opinion and to collaborate with others. The facilitator put an emphasis on discussion of the topic and I felt at ease offering my opinion in an informal but familiar environment. It was a small group with others doing the same or similar thing that I'm doing. I enjoyed hearing and sharing ideas with other like teachers. There's not usually time to do this type of thing. Usually, you are put in a large group (no less than 35) and spend hours doing a formal activity with no or little interaction. You also are not given the quality materials that we were given. I had fun and I learned a lot without pressure but because I wanted to. I participated because I wanted to and stayed in the group because I wanted to.”

To improve America’s History in the Making, Volume II, participants suggested that the facilitator check the video setup prior to each workshop (due to technical difficulties some participants were unable to view some of the videos; 16%). Participants also recommended dedicating additional time to developing lesson plans (12%), providing more background information for teachers with less historical knowledge (7%), providing opportunities besides group discussion in which to consider the materials (3%),


covering less content (3%), giving the instructor greater flexibility with the curriculum to meet the needs of the participants (2%), and covering more content (2%). Logistical suggestions included reducing Saturday sessions to half rather than full days (8%), offering a 5-day summer institute (4%), starting later in the morning (2%), providing lunch for all-day sessions (2%), and providing more breaks (3%). Riverside County Office of Education and Grant Wood Area Education Agency participants (10%) complained that the facilitator needed to be more organized because they did not always receive complete materials or correct reading assignments. The following suggestions were made by individual teachers: schedule the workshop series during the work week and cover the cost of substitute teachers, eliminate the obvious discussion questions and keep only the questions that require deep analysis of the content, provide an overview of the content to be addressed in subsequent sessions, improve the correspondence between the content in all of the multimedia materials, group teachers by content area, provide time to work with the materials online, provide an agenda for each day, and employ an historian rather than a teacher to lead the workshops.

To what extent did the participating teachers think the individual components of the professional development (i.e., print materials, videos, hands-on activities) met their intended purposes?

The print materials provided the participants with the core content needed to develop a conceptual understanding of the 5 eras of history addressed in the professional development. Nearly all (86%) of the survey respondents reported that the print materials were a useful part of the professional development, 76% planned to use the print materials in their classroom, and 56% planned to use the primary source documents. The materials were considered useful because they prepared the teachers for the subsequent workshops and for teaching classes (43%); provided resources to use with students (15%), offered a variety of perspectives on historical topics and some lesser known historical information (12%), included useful primary source documents (12%), and provided clarification of the themes (5%). One teacher appreciated how the introduction and follow-up sections lent cohesion to the content.

Suggestions regarding the print materials included providing the materials in various formats including online and digitally (on a CD) to increase accessibility and readability (12%) and providing more background information regarding the primary source documents (10%). One teacher remarked, “Provide background information for the reading materials presented. Again, if your reader is lacking in background knowledge the primary source loses some of its meaning.” Other suggestions included assigning fewer print materials (9%), highlighting key information in the text (5%), including tips for altering the reading material to use with students (5%), including perspectives from a broader range of historians (3%), providing content of greater depth with fewer generalizations (3%), and copyediting the materials (2%). One teacher noted, “Sometimes the dates were off. Like one reference in the early Units 1 through 3 said something happened in 150 when it had to happen about 1540.” Participants (11%) also suggested that the facilitators of future workshops compile clearly legible, complete sets


of the materials for participants. One commented, “The maps, photos, and cartoons were difficult to analyze because of the quality of the reproduction. I would like to see those more improved.” Individual teachers’ recommendations included these: ensure that the text clearly reflects the key goals of the professional development; provide reflective questions to guide readers through the materials; include key historical events on all timelines; create online lesson plans that are easier to use; and provide a table of contents, bibliography, annotations, and a list of further reading.

Videos

The videos illustrated and emphasized aspects of the content presented in the print materials. Nearly all (88%) of the participants described the videos as a useful component of the professional development, 94% reported that the videos made them want to learn more about the content, and 20% planned to use the videos in their classroom. The videos were considered useful because they helped participants remember content (27%); presented information on lesser known individuals in history and varied perspectives on historical issues (22%); depicted events realistically (10%); briefly summarized the content (7%); provided information in depth (9%); complemented the print materials, group discussions, and hands-on activities (7%); presented concepts clearly (3%); were visually interesting (2%), and provided a convenient starting point for group discussions (3%). Individual teachers reported appreciating how the videos dealt with people’s understanding of history; enjoying the Faces of History sections, and appreciating the utility of the videos with substitute teachers.

Participants’ primary concern (27%) was technical problems with the videos, which occurred at every site except Multnomah Education Service District and Malheur Education Service District. In addition, teachers who attempted to use the videos in the their classroom (20%), complained that the cumbersome online registration process for each video made it impossible to stream the videos during class time. In addition, some participants (10%) expressed dissatisfaction with a lack of conclusiveness in some of the videos. Two participants commented:

“Sometimes I was left with the feeling that I wanted to know what happened next? What happened to the lady who wanted to be a Muslim chaplain in the US armed forces? Where is she now? How does the idea of a women chaplain fit into our traditional understanding of how Muslims run their worship services?”

“I think the later [videos], 18 through 22, can be improved by not being so broad as far as trying to cover too much. They need to be as detailed as the earlier ones.“

Participants’ suggestions regarding the videos included these: provide participants with DVDs to use during the workshop and in the classroom (10%), produce a complete library of videos (7%), use a more captivating host (preferably one whose appearance is not distracting; 4%), provide less lecturing and more reenactments (2%), improve the


transitions between themes (4%), ensure the accuracy of the information depicted (one teacher remarked, “Why was an aircraft carrier shown at Pearl Harbor? That kind of factual error is bothersome.”), strengthen the overall coherence of the information presented (2%), and include subject headings in the videos (2%). Individual teachers suggested making the videos both longer and shorter, providing a study guide with the video, including broader ideas and fewer lesser known individuals, including more hosts who represent minority groups, providing a less scripted delivery, eliminating the reenactments, and reworking the introduction to the Lavender Scare eliminate stereotypes.

Hands-On Activities

The hands-on activities aimed to deepen the participants’ understanding of the content by providing dynamic learning opportunities within a workshop setting. Most participants (75%) reported that the activities were a useful component of the professional development. Thirty-six percent considered the concluding group discussions to be the most valuable aspect of the activities, deeming them a thought-provoking, useful means of learning content. Others appreciated that the activities provided them with ideas for presenting historical content to their students (22%) and afforded opportunities to work with primary source documents (3%). Some participants planned to use the activities in their classroom (8%), and a few planned to use the discussion questions (2%).

Participants offered suggestions for improving the activities. Many (46%) considered the instructions clear and easy to follow, but many others suggested a need for greater clarity: 41% described the instructions as somewhat clear and easy to follow, 10% described the instructions as not very clear or easy to follow, and 3% described the instructions as not at all clear or easy to follow. At one site participants (13%) reported they wanted to actually do the activities rather than merely discuss the activities as a large group. Other suggestions included providing time to discuss as a group how to adapt the activities for the classroom (8%), covering less information in greater detail (10%), and including more of the same types of hands-on activities during the workshops (2%). Some participants complained that the facilitator was not adequately prepared to lead the activities (2%). Individual participants provided the following suggestions: provide guidance on adapting the activities for English language learners, provide more primary sources, design an activity that shows how one event affects people in various regions differently, broaden the range of activities, administer assessments immediately following each activity to measure their efficacy, provide more activities that require creative thinking (e.g., the mural activity), print all of the handouts and overheads in color, increase the font size, highlight the main points of the discussion, assemble all activity materials prior to starting an activity, provide outlines for teachers to use to take notes, conduct more small-group discussions and fewer whole-group discussions, provide greater direction during discussions, summarize the discussion(s) at the end of each activity, and encourage everyone to participate in group discussions.


To what extent did the participating teachers think the multimedia approach is an effective way of helping teachers learn American history content?

Almost all of the survey respondents (98%) reported that the multimedia approach (i.e., using a combination of print materials, video, and hands-on activities) effectively increased their content knowledge in American history. Many (65%) asserted that the multimedia approach addressed various learning styles (e.g. visual, auditory, and kinesthetic), thereby increasing the absorption of the material, though a few suggested that there was room for improvement in this regard. Teachers appreciated that the multimedia approach provided a model for teaching content to their students and materials that could be used in the classroom (14%). Two teachers commented:

“It is an effective method to increase teachers' content knowledge. History is the subject that is complained about a lot due to its typical delivery in school . . . lecture, textbook readings, etc. The students I teach need to see the human side of times past, and they aren't interested in memorizing information—and I don't blame them, I never liked that either. When we have more teaching tools, including multimedia, it just offers a variety of avenues to explore. Any time a student can decide a way to learn about a time or a place, it's nice for them to have more than one way to approach it.”

“We're not just learning content here, but also learning how to use content with our students. Actually going through the process, how to view and critique video, how to access materials, how to read critically and analyze—these are all skills necessary in our classrooms as well.”

Others (12%) liked transitioning between media—for example, from a video to an activity—because doing so kept their attention focused, and some (3%) praised the multimedia approach’s ability to bring history “to life.” In contrast, one participant considered the approach redundant: “Not everything needs to spiral back over the same material several times,” commented the teacher, “Most people taking this course have some knowledge of these events and do not need an in-depth review.” Participants were asked whether material that appeared in one medium (i.e., video, print materials, activities) would be better covered in another medium, and the responses were diverse. A few participants (3%) suggested that speech transcripts and letters would be better in audio or video. Individual suggestions included these: provide all primary sources in PDF, a format that is easier for teachers to use in the classroom; make all activities available on the website; include the video biographies in the print materials; present artwork and other visuals in a PowerPoint presentation rather than the print materials; include more examples of period artifacts in the print materials or website such as books, comics, clothing, and music; include more visuals in Units 2 through 10; include an interactive map in the facilitator materials to explain migration;


present the content in the conservatism video in a different medium; project all of the facilitator materials in the front of the classroom; ensure that the facilitator models how to use the materials, including showing the videos during the workshop and projecting the website for the participants.

Distribution of America’s History in the Making, Volume II

At the conclusion of the project, was America’s History in the Making, Volume II, made nationally available?

At the conclusion of the evaluation the workshop was made available nationally free of charge online and through Annenberg Media Channel.

Increased Content Knowledge

To what extent did America’s History in the Making, Volume II, increase the American history content knowledge of the participating teachers with respect to their understanding of the people and events that that shaped American history from the post-Reconstruction period onward?

The hypothesis was that participants would significantly improve their content knowledge scores from pretest to posttest. Each test was scored on a scale from 0 to 20. The average score on the pretest was 9.96 (minimum 3.50, maximum 18; SD = 3.31). Before comparing the scores, RMC Research conducted tests to confirm that the 3 versions of the assessment were the same level of difficulty. Exhibit 9 shows that differences between the pretest scores on the 3 versions were evident. The average score on Version 1 was 9.29, whereas the average score on Version 3 was 1.13 points higher (10.42). Due to these differences, the scores on all of the pre- and posttests were adjusted by subtracting .3102 from the scores for Version 1, adding .6624 to the scores for Version 2, and subtracting .4721 from the scores for Version 3.

Exhibit 9 Pretest Scores by Version

Version N M Min Max SD

Version 1 30 10.26 4.50 17.50 3.10

Version 2 34 9.29 3.50 18.00 3.24

Version 3 28 10.42 3.50 17.50 3.58 Note. Scoring scale 0–20.


A paired-sample t test was conducted to evaluate the difference between the participants’ pretest score immediately prior to the professional development and posttest score immediately after the professional development. As Exhibit 10 shows, the hypothesis was supported: participants’ historical knowledge increased from pretest to posttest. Participants’ mean posttest score (M = 10.32, SD = 2.74) was significantly greater than the mean pretest score [M = 9.48, SD = 3.21; t (91) = -2.59, p = .011]. The standardized effect size (d = .27) indicates that the professional development had a small to moderate impact on the participants’ content knowledge. Because the impact was not large, further analysis were conducted to determine whether other factors might have impacted the effect size.

Exhibit 10 Pretest and Posttest Scores

Note. Scoring scale 0–20. n = 92.

First, the results were analyzed to determine whether the increase in historical content knowledge could be directly attributed to the professional development. If the increase in content knowledge was due to the professional development the Cohort 1 teachers would show a statistically significant increase from the pretest administered prior to the professional development in the fall to the posttest administered after the professional development in the winter and minimal increase in content knowledge at the second posttest administered in the spring (a few months after the professional development). If the increase in content knowledge was due to the professional development the Cohort 2 teachers would not show an increase in knowledge from the pretest administered in the fall a few months prior to the professional development to the

0

2

4

6

8

10

12

14

16

18

20

Pretest Posttest


second pretest administered immediately prior to the professional development in the winter, but they would show a statistically significant increase at the posttest administered immediately after the professional development in the spring.

As Exhibit 11 shows, the hypothesis was supported for the Cohort 2 teachers: the teachers significantly increased their content knowledge from pretest to posttest [t (48) = -2.72, p = .009] and showed minimal change [t (48) = 1.78, p = .081] in content knowledge from the first pretest to the second. The standardized effect size (d = .38) was larger than the whole group comparison, but it was less than expected.

Exhibit 11 Pre-Post Scores by Cohort

Note. Scoring scale 0–20. Cohort 1 n = 43. Cohort 2 n = 49.

In contrast, Exhibit 11 shows that the hypothesis was not supported for the Cohort 1 teachers: after completing the professional development the Cohort 1 teachers did not report a significant [t (42) = -.597, p = .55] increase in their content knowledge from the pretest to the posttest, but they did demonstrate a significant [t (42) = -3.68, p = .001] increase in their content knowledge from the pretest to the second posttest. This result might be due to difficulties the facilitators experienced implementing the complex multimedia workshop with the first cohort of teachers. In January 2008 RMC Research analyzed the Cohort 1 teacher surveys and distributed to project staff a brief (Appendix A) that described some of the challenges that occurred during the implementation of the Cohort 1 workshops. Project staff used these data to meet with the steering team members and facilitators to improve the implementation of the project for the Cohort 2 teachers.

0

2

4

6

8

10

12

14

16

18

20

Fall Winter Spring

Cohort 1

Cohort 2


In addition, it appears that the Cohort 2 teachers might have been more motivated to improve their teaching: 65% participated in other professional development the same year they participated in America’s History in the Making, Volume II, and in 29% of the cases that other professional development was history related. Although 63% of the Cohort 1 teachers participated in other professional development the same year they participated in the project, only 12% of these teachers participated in history-related professional development. These differences could also suggest that the Cohort 2 scores improved due to the combination of professional development rather than America’s History in the Making, Volume II, alone. As for the Cohort 1 teachers scoring significantly higher on the second posttest, this result might be attributable to teachers taking the second posttest in a relaxed environment at home rather than at the professional development site after a long day of professional development.

Subgroup Analysis

The results were further analyzed by subgroup. Exhibit 12 displays the scores by gender. Male participants scored higher on the pretest compared to female participants, but the difference was not significant [t (90) = 1.83, p = .07]. The female [t (53) = -2.35, p = .02] participants significantly increased their scores from pretest to posttest, whereas the male participants did not [t (37) = -1.14, p = .26]. Neither group scored significantly higher than the other on the posttest [t (90) = 1.14, p = .26]. These results suggest the professional development was more effective for the female participants than the male participants, though the standardized effect size (d = .32) was small.

Exhibit 12 Pre-Post Scores by Gender

Note. Scoring scale 0–20. Male n = 38. Female n = 54. Paired-samples t test used for within group comparisons. Independent t test used for between-group comparisons.

0

2

4

6

8

10

12

14

16

18

20

Pretest Posttest

Male

Female


Exhibit 13 displays the pre-post scores by site. Multnomah Education Service District participants scored the highest on the pretest, followed by Grant Wood Area Education Agency, Riverside County Office of Education, Malheur Education Service District, and Bedford and Coffee Counties Department of Education. Overall, Multnomah Education Service District participants had the highest average pretest and posttest scores, and their scores increased from pretest to posttest [t (38) = -.66, p = .51], but the increase was not statistically significant. Although the remainder of the sites’ sample sizes were too small to conduct statistical tests, each site did score higher on the posttest compared to the pretest. Malheur Education Service District and Bedford and Coffee Counties Department of Education, the 2 sites with the lowest pretest scores, demonstrated the greatest increase from pretest to posttest. These results suggest the professional development impacted teachers with the greatest need.

Exhibit 13 Pre-Post Scores by Site

Note. Scoring scale 0–20. Multnomah n = 39 Malheur n = 5 Riverside n = 15 Grant Wood n = 16 Bedford n = 17. Paired-samples t test used for within-group comparisons.

0

2

4

6

8

10

12

14

16

18

20

Pretest Posttest

Multnomah ESD

Malheur ESD

Riverside County

Office of Education

Grant Wood Area

Education Agency

Bedford and Coffee

Counties Dept. of Ed.


Exhibit 14 displays the pretest and posttest scores with regard to whether the participants taught history as a stand-alone course. Although participants who taught history as a stand-alone course did score higher on the pretest than those who did not teach history as a stand-alone course, the scores were not significantly higher [t (89) = 1.339, p = .184]. The participants who taught history as a stand-alone course significantly improved their scores from pretest to posttest [t (58) = -1.93, p = .058], whereas those who did not teach history as a stand-alone course did not [t (31) = -1.49, p = .145]. Neither group scored significantly higher than the other on the posttest [t (89) = 1.42, p = .16]. These results suggest the program was the most effective for teachers who taught history as a stand-alone course, though the standardized effect size (d = .25) was small.

Exhibit 14 Pre-Post Scores by History Course Type

Note. Scoring scale 0–20. Taught history as a stand-alone course n = 59. Did not teach history as a stand-alone course n = 32. Paired-samples t test used for within-group comparisons. Independent t test used for between-group comparisons.

0

2

4

6

8

10

12

14

16

18

20

Pretest Posttest

Taught history as a

stand alone course

Did not teach history

as a stand alone

course


Exhibit 15 displays the pretest and posttest scores with regard to the grade level the participants taught (Grades 6–8 or Grades 9–12). Teachers of Grades 9–12 scored higher on the pretest than did teachers of Grades 6–8, but not significantly [t (89) = 1.34, p = .184]. Both groups attained higher scores on the posttest compared to the pretest; however, only teachers of Grades 9–12 scored significantly [t (56) = -2.18, p = .033] higher on the posttest whereas teachers of Grades 6–8 did not [t (33) = -1.72, p = .095]. Neither group scored significantly higher than the other on the posttest [t (89) = -1.195, p = .235]. These results suggest the professional development was most effective among teachers who taught high school, though the standardized effect size (d = .29) was small.

Exhibit 15 Pre-Post Scores by Grade Level Taught

Note. Scoring scale 0–20. Grades 6–8 n = 34. Grades 9–12 n = 57. To protect participant confidentiality data are not reported for categories with fewer than 2 participants: Grade 5. Paired-samples t test used for within-group comparisons. Independent t test used for between-group comparisons.

0

2

4

6

8

10

12

14

16

18

20

Pretest Posttest

Grades 6-8

Grades 9-12


Exhibit 16 displays the pretest and posttest scores with regard to the number of years the participants had taught (3 years or less or 4 years or more). Teachers scored similarly [t (90) = .21, p = .83] on the pretest regardless of the number of years they had taught and both groups received higher scores on the posttest compared to the pretest. The participants who had taught 4 years or more significantly [t (66) = .-2.41, p = .019] increased their scores from pretest to posttest, whereas those who had been taught 3 years or less did not [t (24) = .-.959, p = .347]. Neither group scored significantly higher than the other on the posttest [t (90) = -.175, p = .862]. These results suggest the professional development was more effective for veteran teachers, though the standardized effect size (d = .29) was small.

Exhibit 16 Pre-Post Scores by Years Taught

Note. Scoring scale 0–20. 3 years or less n = 25. 4 years or more n = 67. Paired-samples t test used for within-group comparisons. Independent t test used for between-group comparisons.

0

2

4

6

8

10

12

14

16

18

20

Pretest Posttest

3 years or less

4 years or more


Exhibit 17 displays the pretest and posttest scores with regard to the number of years the participants had taught history (3 years or less or 4 years or more). Teachers in both groups scored similarly on the pretest [t (90) = -1.779, p = .079], and both groups increased their scores from pretest to posttest but only those who had taught history 3 years or less significantly increased their scores [t (34) = -2.16, p = .038] whereas those who had taught history 4 years or more did not [t (56) = -1.60, p = .115]. Neither group scored significantly higher than the other on the posttest [t (90) = -1.779, p = .079]. These results suggest the professional development was more effective among new history teachers, though the standardized effect size was not large (d = .36).

Exhibit 17 Pre-Post Scores by Years History Taught

Note. Scoring scale 0–20. 3 years or less n = 35. 4 years or more n = 57. Paired-samples t test used for within group comparisons. Independent t test used for between group comparisons.

0

2

4

6

8

10

12

14

16

18

20

Pretest Posttest

3 years or less

4 years or more


Exhibit 18 displays the pretest and posttest scores with regard to participants’ “highly qualified” status. Teachers who were “highly qualified” to teach history scored significantly [t (90) = 3.62, p = .00] higher on the pretest compared to those who were not. Both groups increased their scores from pretest to posttest, but among the “highly qualified” participants the increase was not significant [t (76) = -1.777, p = .08] and the sample of participants who were not “highly qualified” was too small to conduct statistical tests. Regardless, these results further support the finding that the professional development assisted those teachers with the greatest need: the teachers who were not “highly qualified” increased their scores, and their posttest scores approached those of the “highly qualified” teachers.

Exhibit 18 Pre-Post Scores by “Highly Qualified” Status

Note. Scoring scale 0–20. “Highly qualified” n = 77. Not “highly qualified” n = 15. Paired-samples t test used for within-group comparisons. Independent t test used for between-group comparisons.

0

2

4

6

8

10

12

14

16

18

20

Pretest Posttest

Highly qualified

Not highly qualified


Exhibit 19 displays the pretest and posttest scores with regard to the highest degree the participants had earned (Bachelor’s or Master’s). Participants with a Master’s degree scored significantly [t (89) = -4.31, p = .00] higher on the pretest compared to participants with a Bachelor’s degree. However, the participants with a Bachelor’s degree significantly [t (37) = -3.715, p = .001] increased their scores from pretest to posttest, whereas the scores of the participants with a Master degree remained relatively stable [t (52) = .061, p = .952]. These results further support the finding that the professional development assisted teachers with the greatest need: the teachers with a Bachelor’s degree increased their knowledge, their posttest scores approached those of the teachers with a Master’s degree, and the standardized effect size was high (d = .60).

Exhibit 19 Pre-Post Scores by Degree

Note. Scoring scale 0–20. Bachelor degree n = 38. Master degree n = 53. Paired-samples t test used for within-group comparisons. Independent t test used for between-group comparisons.

0

2

4

6

8

10

12

14

16

18

20

Pretest Posttest

Bachelor degree

Master degree


Exhibit 20 displays the pretest and posttest scores with respect to the number of college credits in history the participants had (38 or fewer or 39 or more). Participants who had more credits scored significantly [t (86) = -2.578, p = .012] higher on the pretest than those who had fewer credits. However, participants with fewer credits significantly [t (55) = -2.403, p = .020] increased their scores from pretest to posttest whereas the scores of participants with many credits remained relatively stable [t (31) = -.50, p = .62]. These results further support the finding that the professional development assisted teachers with the greatest need: the teachers who had fewer credits in history increased their scores, and their posttest scores approached those of the teachers with many credits in history; the standardized effect size was not, however, large (d = .32).

Exhibit 20 Pre-Post Scores by College Credits

Note. Scoring scale 0–20. Low n = 56. High n = 32. Paired-samples t test used for within-group comparisons. Independent t test used for between-group comparisons.

Interviews

The teachers interviewed were asked whether their content knowledge of American history had increased as a result of their participation in the project. The majority of the teachers (88%) reported having gained new knowledge of American history, and many (64%) reported having learned more about people in history rather than events in history. When asked how the professional development could be modified to place further emphasize increasing teacher content knowledge, teachers suggested

0

2

4

6

8

10

12

14

16

18

20

Pretest Posttest

38 or less

39 or more


increasing the duration (24%) and customizing the sessions to address the teachers’ knowledge and interests, as determined by a needs assessment conducted prior to the professional development (8%). Individual teachers suggested grouping the cohorts by experience level to ensure greater homogeneity among participants and discussing all of the print materials in depth.

Improvements to the Online Content Assessment

RMC Research used the results from America’s History in the Making, Volume I (a Teaching American History grant funded in 2004), to improve the online assessment for America’s History in the Making, Volume II. In the first project several participants scored lower on the posttest than they had on the pretest. The participants completed the pretest at the beginning of the professional development and completed the posttest after 8 hours of professional development on a Saturday. Participants might have performed less than optimally due to fatigue. For America’s History in the Making, Volume II, teachers had the option of taking the posttest at the end of the day or on a subsequent day when they were more refreshed. Second, some of the first project participants’ responses were very short, indicating, perhaps, that they needed greater clarity on the expectations regarding the short answer questions. For America’s History in the Making, Volume II, teachers received detailed guidelines regarding such expectations and scoring.

Increased Ability to Interpret Primary and Secondary Sources

To what extent did America’s History in the Making, Volume II, increase the participating teachers’ appreciation of the use of primary source materials? To what extent did the use of America’s History in the Making, Volume II, help teachers understand how to interpret primary and secondary sources, maps, charts, and other visuals in both learning and teaching history content?

The posttest asked participants to indicate, on a scale from 1 (not at all) to 4 (to a great extent), the extent to which the professional development increased their appreciation for the use of primary source materials. The majority reported to a great extent (65%) or to some extent (28%), although some reported not very much (10%) or not at all (2%). These latter participants might have already had a strong appreciation for the use of primary source materials.

Interviewees were asked to describe how the professional development had improved their skill at interpreting primary and secondary sources and whether their use of primary source documents had changed due to their participation in the professional development. Teachers credited the technology portion of the professional development with improving their ability to locate and download primary sources (88%), understand copyright requirements (48%), and assess the credibility of a website (28%). In addition, 80% of the teachers confirmed that their use of primary source documents had


changed. Teachers (40%) reported increased use of primary sources in the classroom, explaining that they had gained the confidence to analyze the documents in greater depth with their students and encourage their students to utilize this approach themselves. Teachers who had used primary source documents in the classroom regularly reported having acquired additional resources to draw from (24%) and learning new strategies to implement in the classroom (16%). The remainder (20%) did not change the way they used primary source documents.

Regarding how the professional development could be modified to intensify the focus on increasing teachers’ ability to interpret primary and secondary sources, the teachers were divided. Many (40%) stated that they did not see a need to alter the professional development, whereas others (32%) suggested allocating more time for group discussions or activities. One teacher suggested an activity involving the review and ranking of documents according to their capacity for conveying certain themes or concepts. Two teachers expressed adesire for more background information prior to reviewing primary source documents. Individual teachers suggested spending less time providing background information and inviting historians to be guest speakers.

Increased Digital Technology Skills

To what extent did America’s History in the Making, Volume II, increase the participating teachers’ skills regarding the use of digital technology to research American history resources?

The posttest asked participants to indicate, on a scale from 1 (not at all) to 4 (to a great extent), the extent to which the professional development increased their skills using digital technology to research American history resources. Most respondents indicated to a great extent (39%) or to some extent (37%), although some reported not very much (21%) or not at all (3%). These latter participants might have already had strong skills regarding the use of digital technology.

Interviewees were asked to provide feedback on the most and least useful aspects of the aspectsof the professional development that focused on digital technology. Many teachers cited as useful instantaneous access to primary sources (48%) and videos (24%), and the majority (60%) reported utilizing technology in the classroom to project websites and stream videos.


Earned Credits

To what extent did America’s History in the Making, Volume II, assist participating teachers in securing credits toward achieving “highly qualified” teaching status, a requirement of the No Child Left Behind Act?

Of the 92 teachers who participated in America’s History in the Making, Volume II, 77 were “highly qualified” to teach history and 15 were not. Of the “highly qualified” teachers 50% (n = 39) completed additional assignments and earned college credit for completing the professional development: 3 earned 2 credits and 36 earned 3 credits. Of the teachers who were not “highly qualified” to teach history, 33% (n = 6) secured credits toward achieving “highly qualified” teaching status: 1 earned 2 credits and 5 earned 3 credits. These results suggest that given the option of receiving credits for professional development, teachers who are “highly qualified” to teach are more likely to pursue this option than are teachers who are not. Requiring all participants to complete the additional assignments necessary to earn credits would likely result in higher achievement rates for this objective.

Improved Pedagogy and Increased Student Achievement

To what extent did America’s History in the Making, Volume II, improve the participating teachers’ instructional strategies?

The hypothesis was that after participating in the professional development participants would be significantly more prepared to teach history at their assigned level; integrate history with other subjects; and provide history instruction that met district, state, or national standards. As Exhibit 21 shows, the hypothesis regarding participants’ ability to teach history at their assigned level was supported: Cohort 1 teachers demonstrated a statistically significant increase from the pretest to the posttest (z = -3.83, p < .001) and no change between the first posttest and the second posttest (z = -.47, p = .64); Cohort 2 teachers demonstrated no change between the first pretest and the second pretest (z = -.93, p = .346) and a statistically significant increase from the second pretest to the posttest (z = -4.28, p < .001). Teachers who were interviewed described how the professional development increased their preparedness to teach: 40% attributed the change to the materials provided, 36% to the materials coupled with the instruction on implementing those materials, 28% to the energizing and confidence-building effect of acquiring new knowledge and skills, and 16% to learning and sharing information with other teachers.


Exhibit 21 Pre-Post Results: Preparedness to Teach History at Assigned Level

Note. Cohort 1 n = 43. Cohort 2 n = 49. Scale: 1 = not well prepared, 2 = somewhat prepared, 3 = well prepared, 4 = very well prepared. Wilcoxon tests were conducted to compare the scores.

A similar pattern is evident in Exhibit 22 with regard to participants integrating history with other subjects. Cohort 1 teachers demonstrated a statistically significant increase from the pretest to the posttest (z = -3.13, p < .001) and no change between the first posttest and the second posttest (z = -.57, p = .57). Cohort 2 teachers demonstrated no change between the first pretest and the second pretest (z = -.016, p = .987) and a statistically significant increase from the second pretest to the posttest (z = -4.28, p < .001). Although 52% of the teachers interviewed expressed the opinion that integrating history with other subjects was not a central focus of the professional development, the remainder credited the series with giving them ideas for integrating history with other subjects, such as utilizing mathematics to study the economic consequences of the oil crisis throughout history, using autobiographical literature to augment the study of historical events, and incorporating scientific information into investigations of the role of penicillin in World War II.

1

2

3

4

Fall Winter Spring

Cohort 2

Cohort 1


Exhibit 22 Pre-Post Results: Integration of History With Other Subjects

Note. Treatment n = 43. Comparison n = 49. Scale: 1 = not well prepared, 2 = somewhat prepared, 3 = well prepared, 4 = very well prepared. Wilcoxon tests were conducted to compare the scores.

A similar pattern was expected with regard to participants’ preparedness to provide history instruction that met district, state, or national standards; Exhibit 23 shows that the hypothesis was not, however, supported. Cohort 1 teachers demonstrated an insignificant increase from the pretest to the posttest (z = -1.38, p = .16). The Cohort 2 teachers demonstrated a significant increase between the first pretest and the second pretest (z = -3.26, p < .001) and a significant increase from the second pretest to the posttest (z = -1.96, p < .05). These results suggest that throughout the school year teachers improve their skills with regard to meeting district, state, or national standards, and these improvements are attributable to factors beyond America’s History in the Making, Volume II.

1

2

3

4

Fall Winter Spring

Cohort 2

Cohort 1


Exhibit 23 Pre-Post Results: Provision of History

Instruction That Meets District, State, or National Standards

Note. Treatment n = 43. Comparison n = 49. Scale: 1 = not well prepared, 2 = somewhat prepared, 3 = well prepared, 4 = very well prepared. Wilcoxon tests were conducted to compare the scores.

During the interviews teachers elaborated on the changes in their teaching practices and strategies for increasing student achievement. The majority of the teachers (92%) reported that due to the professional development their teaching practices had changed. Teachers primarily described how their teaching had changed due to using more primary sources in the classroom to prompt discussion, to assess student achievement, and to stimulate students’ interest (36%); using the videos, activities, and print materials in their classroom (24%); focusing on cause and effect when analyzing historical events (12%); including information regarding the lesser known individuals in history in their lesson plans; and providing more in-depth and accurate historical content to their students (8%). Individual teachers credited the workshop series with reinforcing good teaching practices; helping them incorporate technology into the classroom; and increasing their enthusiasm for the content, which directly translated into increased enthusiasm among their students. In addition, almost of the teachers (92%) reported learning strategies for increasing student achievement such as using an inquiry-based instructional approach (36%); incorporating primary sources into student assessments (12%), and using scoring guidelines and charts for assessing student work and clarifying learning goals (12%).

1

2

3

4

Fall Winter Spring

Cohort 2

Cohort 1


To what extent did America’s History in the Making, Volume II, increase the likelihood that teachers would use the National Center for History in the Schools’ historical thinking skills in their classroom?

After participating in the project the teachers were asked how likely they were to engage their students in chronological thinking skills (see Exhibit 24). Participants responded on using a scale from 1 (not at all likely) to 4 (very likely). The teachers reported being likely or very likely to engage their students in using all of the chronological thinking skills.

Exhibit 24 Posttest Results: Chronological Thinking

Note. n = 92. Median score shown. Scale: 1 = not at all likely, 2 = not likely, 3 = likely, 4 = very likely.

1 2 3 4

Determine when events occurred and in what

order

Identify the beginning, middle, and end of a

historical event

Show how the end of one historical event could

be the beginning of another historical event

Create their own secondary account of a

historical event

Measure or calculate calendar time

Interpret data presented in timelines

Establish patterns for how historical

developments unfolded and how these patterns

Identify the characteristics of a particular time

period

Determine why certain characteristics were

chosen to define a time period

Create alternative ways of defining time periods


Participants were also asked how likely they were to engage their students in historical comprehension skills (see Exhibit 25). The teachers reported being very likely to engage their students in using 7 of the 8 comprehension skills.

Exhibit 25 Posttest Results: Historical Comprehension


Participants were also asked how likely they were to engage their students in historical analysis and interpretation skills (see Exhibit 26). The teachers reported being likely or very likely to engage their students in using all of the historical analysis and interpretation skills.

1 2 3 4

Reconstruct the literal meaning of a historical

event

Identify the theme of a historical document

Examine an author’s motives or intentions

Describe the past through the perspective of the

author

Assess the audience for whom a document was

intended

Describe the historical context in which historical

events unfolded

Use primary source documents to learn about

historical developments

Use secondary source documents to learn about

historical developments


Exhibit 26 Posttest Results: Historical Analysis and Interpretation


Participants were also asked how likely they were to engage their students in research skills (see Exhibit 27). The teachers reported being likely to engage their students in using the research capability skills.

Exhibit 27 Posttest Results: Research Capabilities


1 2 3 4

Formulate historical questions

Gather historical documents

Identify gaps in the available records

1 2 3 4

Compare and contrast documents with differing

ideas about the same historical event

Differentiate between historical facts and

historical interpretations

Evaluate major debates among historians

Analyze cause and effect relationships

Draw comparisons across eras and regions in

order to define enduring themes

Identify how an individual’s different actions

could have led to different consequences


Finally, participants were asked how likely they were to engage their students in issues analysis and decision-making skills (see Exhibit 28). The teachers reported being very likely to engage their students in using 4 of the 5 skills.

Exhibit 28 Posttest Results: Issues Analysis and Decision-Making Skills

Note. n = 92. Median score is shown. Scale: 1 = not at all likely, 2 = not likely, 3 = likely, 4 = very likely.

1 2 3 4

Formulate a position on an issue presented

Support their position with historical evidence

Identify moral dilemmas in history

Discuss or write about past events that have

affected contemporary issues

Discuss or write about past events imagining what

would have occurred if alternative actions were

taken


Discussion

The implementation of America’s History in the Making, Volume II, was successful. The project exceeded its recruitment goal and nearly met its goal for participants. The primary accomplishment of the project was a statistically significant increase in the teachers’ knowledge of American history, though the effect size was lower than anticipated. Analysis determined that the small effect size could be due to (a) difficulties the facilitators experienced implementing this complex multimedia workshop with the first cohort of teachers, or (b) Cohort 2 teachers were more motivated to improve their teaching. An unexpected finding was that the project had the most impact on teachers with a Bachelor’s degree. These teachers significantly increased their scores from pretest to posttest, the standardized effect size was high (d = .60), and their posttest scores approached those of the teachers with a Master’s degree. Other subgroups that exhibited a statistically significant increase in scores from pretest to posttest included participants who were female, taught history as a stand-alone course, taught Grades 9 through 12, had taught 4 years or more, had taught history 3 years or less, and had few college history credits.

Other accomplishments for America’s History in the Making, Volume II, participants included improved pedagogy. Teachers were significantly more prepared to teach at their assigned level and integrate history with other subjects after participating in the professional development. In addition, almost all of the teachers (92%) reported that their teaching practices had improved largely due to adopting the practice of using of primary source documents in the classroom. With regard to the use of primary sources, teachers reported increased appreciation, improved skills locating and downloading them, and a better understanding of copyright regulations. Teachers were also more likely to engage their students in using the National Center for History in the School historical inquiry skills after completing the professional development and more likely to meet state and district standards (this achievement could not, however, be directly attributed to the project).

The participating teachers described America’s History in the Making, Volume II, as different than other professional development they had participated in because of its focus on historical content and opportunities to interact with colleagues. They described the multimedia approach as an effective and engaging way to present historical content and appreciated that the approach addressed various learning styles and provided a model for presenting information to their own students. The teachers reported that in addition to helping them learn the content, the print materials and videos would be useful resources for their classroom. Most of the teachers (76%) planned to use the print materials in their classroom, 20% planned to use the videos, and a few planned to use the activities (8%).

An essential component of the Teaching American History grants has been a focus on the recruitment and improvement of teachers in need, which has typically been defined as those new to the field, those without a history degree, and those who are not “highly


qualified” to teach history. Results from this evaluation and the evaluations of other Teaching American History grants may warrant the U.S. Department of Education revising the definition of teachers in need. For this project the teachers in greatest need of increasing their American history content knowledge were those not “highly qualified” to teach history and those who had a Bachelor’s degree—both groups received the lowest scores on the pretest. Examining the results from this grant and other Teaching American History grants revealed that teachers who were not “highly qualified” to teach history scored the lowest of all participants on the content assessment or scored in the lowest 25th quartile. An unexpected finding was that veteran teachers scored in the lowest 25th quartile in 2 of the grants. These results confirm that teachers who are not “highly qualified” to teach history are most in need of this professional development and assert that veteran teachers are also in need of this professional development.

Improving America’s History in the Making, Volume II

The evaluation of America’s History in the Making, Volume II, produced an abundance of useful information to inform the development of this and other multimedia professional development efforts. The developers of future multimedia projects are advised to consider the following insights and recommendations based on the evaluation findings:

Provide additional training to the facilitator to ensure the facilitator knows how to use all of the requisite technology and can model the use of the materials for the participants.

Provide facilitators with a 1–page document that offers tips for leading a successful workshop including these: provide a complete set of materials for the participants with high-quality photocopies, print all handouts and overheads in color, use a large font size for projected materials, present the main points of the discussion with bulleted formatting, provide participants with outline sheets for taking notes, encourage all participants to participate in the group discussions, summarize the discussion at the end of each activity, actually have the participants complete the activity, discuss how the activity could be used in the classroom, and project all materials in front of the participants.

Consider lengthening the duration of the professional development. This was the teachers’ primary suggestion for maximizing the impact of the project on teachers’ historical content knowledge.

Consider scheduling the professional development to accommodate the needs of the teachers. For example, hold the sessions during a weeklong institute or during the week.

Provide additional time for participants to discuss teaching practices with their colleagues and develop lesson plans based on what they learned.

Provide the print materials in various formats including on the website, using various software (e.g., PDF, Microsoft Word), and on a CD to increase accessibility and readability.


Consider visually emphasizing key information in the print materials such as tips for using the print materials with students.

Provide more background information and context regarding the primary source documents in the print materials.

Ensure that print materials are edited to eliminate factual and grammatical errors. Provide teachers with the option of purchasing the videos in a DVD format or

change the lengthy registration process on the Annenberg Media Channel to ensure that teachers can easily use the videos in their classroom.

Improve the clarity of the activity instructions.

Improving All Teaching American History Grants

This evaluation and the evaluations of several other Teaching American History grants (no. 0215X040276, no. U215X040283, no. U215X070295) produced an abundance of useful information to inform the development of future Teaching American History projects. Future projects are advised to consider the following insights and recommendations based on the evaluation findings:

Expand recruitment efforts to include teachers beyond those traditionally defined as most in need. Veteran and new teachers benefit from the Teaching American History grants.

Devise techniques to recruit teachers who are not “highly qualified” to teach history and ensure they take full advantage of the opportunities provided by the professional development. For example, require teachers to obtain college credit rather than offer this opportunity as an option.

Group teachers with similar characteristics into cohorts to ensure that meaningful interactions can occur within the groups. For example, one cohort could comprise teachers new to the field and another cohort could comprise veteran teachers. When these groups were combined the veteran teachers reported that the pace was too slow and the newer teachers reported that the pace was too fast.

Provide additional time for participants to discuss teaching practices with their colleagues and develop lesson plans based on what they learned.

Conduct process evaluations to identify and address project challenges as they occur.

Conduct quasi-experimental rather than experimental designs to address the Government Performance Reporting Act measures. Quasi-experimental designs maximize the number of teacher participants (because fewer teachers leave the program due to random assignment) and consequently produce more data on the impact of the program than experimental designs.

Conduct at least a quasi-experimental design because as this evaluation has shown teachers can demonstrate statistically significant improvements that are not directly attributed to the professional development. A quasi-experimental


design determines whether participant improvements are due to the project or other intervening factors.

Improve the evaluation by providing teachers with greater clarity on the expected responses to the short answer questions on the testing website, information regarding how their responses were to be scored, a feature on the testing website that indicates the percentage of the test that had been completed, and the option of taking the posttest at the end of the day or on a subsequent day.

America’s History in the Making, Volume...

Documents

Transcript of America’s History in the Making, Volume...