Computer Science Outreach Program Evaluation Network (CS …
Transcript of Computer Science Outreach Program Evaluation Network (CS …
Webinar Agenda
• Project Goals• Timeline• Evaluation Resources• Application Overview• Questions and Discussion
NGCP Vision
The National Girls Collaborative Project brings together organizationscommitted to informing and encouraging girls to pursue careers in science, technology, engineering, and mathematics (STEM).
CS OPEN Overview and GoalsAnnouncement at the White House Council
on Women and Girls Inclusive STEM Education Meeting on July 9, 2015
Google and the National Girls Collaborative Project (NGCP) are committed to providing equitable access to CS education opportunities. As part of our efforts, we are pleased to announce the new CS Outreach Program Evaluation Network (CS OPEN) designed to improve opportunities and empower underserved girls through computer science (CS) education.
• Google will provide expertise to improve NGCP programs by offering professional development to promote evaluation of CS education initiatives within the NGCP network
• This pilot will offer opportunities for NGCP programs to learn about exemplary evaluation practices in studying CS education and provide an opportunity to apply for a total of $100K in evaluation grants from Google over the coming year.
• The goals of these evaluations will be to produce knowledge that will improve individual programs and inform the field of CS education.
Application DetailsEligibility• Girl-serving computer science education programs• Non-profit programs, university outreach• Serving K-12 students• Must be listed in The Connectory• Small organizations and programs within larger organizations
Application Components• Organizational Information• Information about the evaluation you are planning
Multiple grants ranging from $5,000-$20,000
Evaluation expertise provided to grantees
Computer Science Educationis:a scientific and engineering discipline that encompasses the study of computers and the investigation of the principles and practices behind developing hardware and software solutions, applications for solving problems across diverse arenas and the impact of technology on society, including a wide branch of study focused on human-computer interaction – the use of software by humans.
is not:about the using of software, such as spreadsheets (like Excel), word processors (like Word) or image tools (Like Photoshop)... It is not about expertise in computer games, it is not about writing content in websites, and it is not about assembling computers or knowing which computers are best buys.”
Timeline
• October 4, 2015: Application Webinar• October 4, 2015: Application Opens• October 19, 2015: Application Closes • November 19, 2015: Grantees Announced• August 31, 2016: Evaluations End
Google Confidential and Proprietary
What is evaluation?
A systematic approach to:
Determining merit, worth, value or significance for different stakeholders
Source / More Information: American Evaluation Association
Informing decisions about organizations, programs, processes, products, systems, personnel, and policies
Source / More information: National Criminal Justice Association
Google Confidential and Proprietary
Embedding evaluation helps your projectCommunicate more effectively
Clarify the program logic Tell your story better
Collect more useful dataStudy flows with the programEasier data collection, analysis, and use,Ability to turn data around quickly adds value
Produce more useful findingsMonitor data in real-time for ongoing program
adaptation and improvementVisual displays of changing dataBuild an evidence-based case for credibility
“Evaluation is not separate from, or added to, a project, but rather is part of it from the beginning”(The 2010 User-Friendly Handbook for Project Evaluation. Washington, DC: National Science Foundation, page 3).
“Once they saw how useful the data were they asked for more” (Tiffany Decker, MIT)
(more info)
Google Confidential and Proprietary
Why is evaluation important?• Essential to your work
o Planning, design, delivery, datao Collecting evidence, stories, lessons learned
• Provides actionable knowledgeo Program improvemento Context for decisions and policies
• Raises important issueso For diverse stakeholders and points of viewo Unintended or negative consequenceso Alternative explanations or competing ideaso Program theory
See: True stories from your peers in information STEM educationCenter for advancement of information science education, Chapter 1Adapted from STARS
Google Confidential and Proprietary
Key Planning Steps
➢ C. PLAN THE EVALUATION○ What info do you need?○ How will you collect it?○ From whom?○ How will you analyze?
➢ D. CREATE MANAGEMENT PLAN○ How will you report?○ What is time frame?○ What is budget?○ Who does what?
➢ A. DESCRIBE THE PROGRAM○ Purpose of program○ Who does it serve?○ How does it work?
➢ B. DEFINE THE EVALUATION○ What do you want to know?○ Who wants to know?○ How will results be used?
Google Confidential and Proprietary
Evaluation Planning will help you
1 describe the program ...including its purpose, who it serves and how it works
2 define the evaluation ...with its particular purpose, focus, audience and context
3 plan the evaluation ...with specific questions, procedures, analyses and reporting strategies
4 manage the evaluation...by defining responsibilities, managing time frames and budget
Evaluation Resources and Expertise for Applicants and Grantees
Evaluation Worksheet• Supports developing your
evaluation plan• Make your own copy and
edit• Optional to submit
worksheet with application• Worksheet links to slide
deck with many evaluation resources
Google Confidential and Proprietary
Identify key stakeholders
See also: The 2002 User-Friendly Handbook for Project Evaluation. Washington, DC: National Science Foundation, Section 2.
Stakeholders can include participants,
would-be participants, community members,
parents, families, siblings, educators,
etc.
“Listening to [program] beneficiaries is both the right and
smart thing to do.”
Twersky, Buchanan, & Threlfall, 2013
Google Confidential and Proprietary
Evaluation purposeTo judge quality and impact • What is and is not effective?• What is being accomplished and what is
not?• What are diverse experiences and
perceptions? To allocate scarce resources• What policy or program based on the
needs and concerns of diverse community members?
To improve spending or accountability• Judge if the (time, labor, money) were
worth it • Use evidence and lessons to guide
decisions• Reward work that includes creative or
useful evaluation designs
To improve implementation• Know your learners or audience• Learn about accomplishments and
struggles• Monitor and adjust based on feedback• Learn together how to do better next
cycle
To generate knowledge (see research vs. evaluation)
• For a particular program, learn lessons about what works and doesn’t work under what conditions and key success factor
• If you generalize to other groups, this is research-- crucially important if it exists.
See: Approaching an Evaluation, Brad Rose Consulting
Is the purpose to prove (“summative”),improve (“formative”), or both?
Google Confidential and Proprietary
Determine data collection methodsSurveysInterviewsFocus groupsKnowledge tests
Use of a myriad of collection methods can enrich a study by minimizing the weaknesses of any one method. Use of both quantitative and qualitative data collection can serve to triangulate findings and substantiate outcomes.
See: STARS
ObservationsDocument reviewLongitudinal study
Google Confidential and Proprietary
Plan analysis and interpretation
Evaluation Question
Data Collection Procedure
Analysis Procedure
Evaluation Criteria
Judgment Procedure
1a,1b. Do participants’ attitudes towards CS change?
Pre- and post student surveys
Calculate % agreement for confidence, interest
% agreement post - % agreement pre
Success = Gain that is statistically significant
3c. Does the program change behavior?
Counselor interviews after program and six months practice
Qualitative analysis of statements of behavioral changes
Do counselors report using new skills learned in program?
Counselors describe 2 instances of appropriate new behavior with rationale
For each evaluation question:● What data are collected when and from whom? ● How will data be analyzed? What criteria will be used? ● How will judgments be made?
Numbers do not speak for themselvesData need to be organized, summarized, and interpreted so that patterns and relationships can be seen and demonstrated.
See: The 2002 User-Friendly Handbook for Project Evaluation. Washington, DC: National Science Foundation, Section 2
Evaluation Resources and Expertise for Applicants and Grantees
Evaluation Planning and Resource Deck
Application Process• PARTS ONE, TWO: Contact Information, Eligibility• PART THREE: Basic Program Information• PART FOUR: Program Evaluation
~~~SCREENING and INVITATION TO CONTINUE~~~
• PART FIVE: Additional Program Details• PART SIX: Program Evaluation Plans• PART SEVEN: Budget Request
Save and Return Available
Sample Application Questions
• Describe your program activities, contact hours, etc.• What goals or outcomes to you want to evaluate?• Describe your evaluation context, e.g., new program, external or
internal drivers, etc.• Describe logic, theory of change, logic model NOT required• Do you have an existing budget for this evaluation?• Your timeline, staffing, budget
Application ResourcesWebsite: http://www.ngcproject.org/cs-outreach-program-evaluation-network-cs-open
• Webinar archive• Resources mentioned in this webinar• FAQ as questions are answered• Application technical assistance questions to