BUILDING AGENCY CAPACITYCreating an Evaluation Community and a

Culture of DataStephanie R. Miller, The Pell Institute

Collaboration is KEY!

• Allows for the diversity of opinions• Creates a sense of “ownership”• Engages important stakeholders• Creates inherently interactive and

voluntary relationships• Takes time and can be difficult to achieve• Cannot be mandated, forced or faked• Establishes clear, articulated and

common goals

Importance of Evaluation

• Allows institutions to better select, oversee, improve and make sense of programs and policies

• Definition: A collaborative and systematic approach to making an assessment of a program, policy or operation using data. The goal of an evaluation is to improve the effectiveness and/or efficiency of a program, policy or operation. (Mark, Henry, & Julnes, 2000)

Building Evaluation Capacity

• Evaluation becomes a core function of the organization• Involves collaboration of all stakeholders (at all levels

of the organization)• Builds an understanding of the importance of

evaluation and how it influences processes and outcomes

• Evaluation should be a continuous, routine and sustained effort

• Operates at multiple levels of the organization• Evaluation has multiple purposes and goals• Requires continuous adjustments and refinements

Goals of Evaluation Capacity Building

• The goal of ECB is to strengthen and sustain an organization’s capacity to:

1) design, implement, and manage effective evaluation projects

2) access, build, and use evaluative knowledge and skills

3) cultivate a spirit of continuous organizational learning, improvement, and accountability

4) create awareness and support for program evaluation and self-evaluation as a performance improvement strategy

(King & Volkov, 2005; King, 2007)

Leadership Commitment

Evaluation Culture and

Capacity

Data QualityAnalytic Expertise

Elements of Evaluation Capacity Building

COLLABORATION

COLLABORATION

Leadership Commitment

• Commitment from the organization’s leadership body is key to sustain evaluation efforts

• Strategies– Provide stakeholder team with continuous

resources, support, incentives and opportunities

Collaboration ~ Cooperative Partnerships

• Shared resources and expertise• Diversity of opinions and experiences• Defined roles and responsibilities• Open, productive communication• Strategies

– Join stakeholders in pursuit of common goals

– Solicit support and involvement of stakeholders

Establishing An Evaluation Culture

• Institutionalize evaluation– Incorporated into “practice” and work responsibilities– Formalized process to plan, execute and use

information from evaluations– An established, natural part of the work environment

• Strategies– Institute regular assessment – Commit to self-examination and self-improvement

Analytic Expertise

• Engage with experts (internal and external) who are knowledgeable of:– Various research methodology (quantitative,

qualitative and mixed-methods)– Relevant subject matter– Data analysis/interpretation and data

systems• Strategies

– Build staff expertise– Provide staff with technical assistance

Establishing Data Quality

• Access to credible and reliable data is essential

• Data should be consistent across all levels of the organization

• Create a Culture of Data

Building a Data Culture

• Leadership is required at all levels of the organization

• Belief that good data are an integral part of evaluation and promoting positive policy and program outcomes

• Willingness to invest time and resources in achieving useful information

Components of a Data Culture

• Leadership and Collaboration: Data should be accessible to all stakeholders

• Coordination and Articulation: There needs to be communication between the institution and smaller programs to insure standardized data collection

• Dynamic Data System: Data tool should be able to systematize the data collection process, analyze data and report results (standard reporting)

• Accuracy: Information must be correct and complete; data entry should be standardize to ensure reliability

• Security: Protect student and staff confidentiality; data systems must be secure

• Utility: Data have to provide the right information to answer questions

• Timeliness: Data should be entered in a timely manner

Factors Affecting a Data Culture

• Polices and Regulations• Standards and Guidelines• Professional Development/Training• Technology• Data Entry Environment• Timelines/Calendars• Assessment and Reporting• Integration into Practice/Culture ~ Community

of PracticeSource: www.data4ss.org

Using Data for Evaluation and Program Improvement

• Why is Data Important?– A blueprint for success, accountability and achievement– Review/analysis of complete, unbiased and measurable

results

• Typical Use of Data– Superficial use– Required reporting (e.g. annual performance reports, grants,

program reviews, etc.)

• Misuses of Data– Discarding of unfavorable data– Heavy reliance on one source of data

Building A Culture of Data

• Data-Driven Decision Making– Systematic process of data use– Routine and collaborative use of data to

inform organizational, programmatic and instructional improvement decisions

– Builds knowledge, uses a variety of data to construct knowledge, to better understand what is actually going on

Using Data for Evaluation and Program Improvement

Collaboration &

Professional Development

Data Collection

Data AnalysisReport &

Communicate Data

Using Data

• Collaboration and Professional Development– Ongoing efforts– Include key stakeholders– Formulate key questions– Establish desired outcomes– Consider cross-program evaluation

• Data Collection– Utilize multiple data sources and databases– Use appropriate data– Organize data

Using Data

• Data Analysis– Conduct meaningful analyses– Identify appropriate analysis techniques– Incorporate multiple measures

• Reporting & Communicating Data– Ensure data results are seen by the

appropriate stakeholders– Create “easy-to-interpret” charts and graphs– Establish periodic reporting

Creating Effective Data Tools

• Data warehouse– Organized storage area for data elements

• Data analysis tool– User-friendly analytic tool– Customized and pre-formatted reports– Expedited analysis

• Decision support tool– Provides real-time recommendations or alerts– Automatic actions for stakeholders

What is YOUR organization’s experience?

• How has your organization incorporated evaluation into its culture?– Success/Pitfalls?– Areas for improvement?

• Has your organization engaged in Evaluation Capacity Building? What are the challenges?

• What is the culture of data at your organization?– How is data used?– Do you use a data system? What does it look like?

• Strengths/Weaknesses?• Areas for improvement?

Resources

Evaluation Capacity Building Building Evaluation Capacity: Designing a Cross-Project Evaluation, Clewell &

Campbell, 2008 Program Evaluation: an Evaluation Culture and Collaborative Partnerships Help

Build Agency Capacity, United States GAO, 2003 Evaluation capacity building in complex organizations, Taylor-Powell & Boyd, 2008

http://www.isce.vt.edu/files/Evaluation_Capacity_Building.pdf Evaluation ToolKit: http://toolkit.pellinstitute.org/ Evaluation: An Integrated Framework for Understanding, Guiding and Improving

Public and Nonprofit Policies and Programs, Mark, Henry & Julnes, 2000 Developing evaluation capacity through process use, King, 2007• A framework for building evaluation capacity based on the experiences of three

organizations, King & Volkov, 2005

Data Data-Driven Decision Making: A Powerful Took for School Improvement

https://www.erdc.k12.mn.us/promo/sage/images/Analytics_WhitePaper.pdf Using Data to Improve Schools: What’s Working

http://www.eric.ed.gov/PDFS/ED469227.pdf Data for Student Success: http://www.data4ss.org/