© 2007 SRI International CPATH Principal Investigators Meeting: Program Evaluation Update March 26,...

© 2007 SRI International

CPATH Principal Investigators Meeting: Program Evaluation UpdateMarch 26, 2010

By the Center for Education PolicyDr. Raymond McGhee and Dr. Nancy Adelman

© 2007 SRI International 2

Presentation Goals

• Provide background on program evaluation, leadership and team

• Describe purposes of the evaluation

• Describe each component of program evaluation and how it affects the projects and their leadership

• Share lessons learned from evaluation design and data collection so far

• Answer questions to clarify any aspect of the evaluation


Evaluation Leadership and Experience

• Contracted by NSF to serve as an outside, third party evaluation contractor

• Nancy Adelman – Project Director

• Ray McGhee – Deputy Director

• Team of twelve researchers and analysts

• Experience in large-scale program evaluations


Program Evaluation Purposes

• Describe and document program implementation

• Assess how the program is promoting change in undergraduate computing

• Support NSF in it’s reporting to the Academic Competitiveness Council.


Project Evaluation Purposes

• Provide formative feedback to facilitate project refinements

• Describe initial implementation activities, successes, and challenges

• Monitor status of project activities

• Collect evidence of project successes and outcomes


Relation between Program and Project Evaluation

• Data from projects will be incorporated into the program evaluation monitoring efforts

• Data from Special project evaluators will provide data to be used to document the program’s impact

• Data from the program evaluation will be reported in the aggregate and disseminated to all the projects


Evaluation Data Sources

1. Monitoring Survey

• Data on project implementation for all projects

• Data will be used by NSF to monitor and track project activities

• Data analyses will be made available to projects in aggregate

• Timeline of first data collection: Feb 2010 – mid April 2010

• Analysis & reporting: late May 2010



2. Site Visits• Data for a cross-section of 10-12 projects in alternating years to

document project implementation in context from the participants’ perspectives

• Purposive sample with case selection criteria: project type, cohort year, location, institution type

• Data will be analyzed across projects and reported in the aggregate to NSF and projects.

• 2010 Timeline: April – May (some in summer)



3. Analysis of Special Evaluations

• America Competes Act (HR 2272) mandates rigorous designs for evaluating STEM education programs

• SRI is supporting 24 projects conducting evaluations using quasi-experimental designs and will incorporate into analysis

• Coordination and communication between evaluator and PI/project team will be important

• Timeline: April – August 2010



4. Faculty Survey

• Sampling faculty from all projects

• Faculty attitudes and practices within departments promoting change in computing

• Survey development and piloting will occur in fall 2010

• Data will be presented to NSF and projects in the aggregate

• Timeline: February – April 2011 & 2013


Additional Resources from SRI

• CPATH Monitor Web site

• Hosting meetings of the special evaluators to support quasi-experimental designs

• Assistance in completing monitoring survey


Lessons Learned

• The evolving nature of computing education reform and the diverse reform activities being implemented

• The role of institutional and regional context

• Avoiding the premature assessment of program impact

• Growing interest among different stakeholders

• Shared interests and benefits related to evaluation (for the CPATH program and projects)


Summary of Next Steps in Evaluation

• Finalizing monitoring survey data collection and analysis

• Conducting site visits and analyzing data

• Meeting with special evaluators and collecting impact data

• Faculty survey development, pilot, clearance, and administration

• Annual reporting to NSF and ACC


Questions & Answers

Wrap-Up

© 2007 SRI International CPATH Principal Investigators Meeting: Program Evaluation Update March 26,...

Documents

Transcript of © 2007 SRI International CPATH Principal Investigators Meeting: Program Evaluation Update March 26,...