TNB Roundtable: Insights from a Community Foundation’s

Evaluation Director

Mary-Kim ArnoldDirector, Evaluation & Learning

January 2016

Agenda

• About the Rhode Island Foundation• About me and my role• Your questions• Key concepts • Framework for systems work• Questions / Contact

About the Rhode Island Foundation

• Founded in 1916 – centennial• 800 million in assets – 35 million in grants (including

donor advised – “discretionary” is closer to 9 million)• serve the state of RI – largest funder in RI• www.rifoundation.org

Key concepts

• Levels of accountability• Units of analysis• Ratios – meaningful context• Performance vs. impact

Mountain of Accountability

• Basic accountability for management• Accountability for “impact”• Accountability for learning, development, and

adaptation

Units of analysis: who or what?

• Individuals• Groups• Geographical units (neighborhood, town, state)• Social interactions (relationships, social behaviors –

e.g., getting married or divorced)

Ratios: in relation to what*?

“We served 375 students this year in our after-school program.”

* in relation to what unstated condition, assumption, goal, or expectation?

Meaningful context -- examples

• Of the 375 students who participated in our after-school program, 98% enrolled for the following year. behavior

• Of the 400 eligible students in Central Falls, we served 375 in our after-school program. reach

• Our goal for the year was 200 students, which we surpassed by enrolling 175 students above our goal. implementation or demand

Meaningful context + unit of analysis

• student: Of the 375 students who participated in our after-school program, 98% enrolled for the following year. behavior

• teacher: Teachers in our after-school program had 98% retention rate for students between years 1 and 2. performance

• organization: This past year, XYZ reached its 5-year goal of 98% retention rate for students. implementation

Performance vs. “impact”

▷ High-performing organizations vs. high impact organizations

▷ Managing for performance but measuring for impact▷ How might your role and tasks change if you job was

measured for impact?

High performance vs. high impact*

High-performing organization High-impact organization

• fiscally strong• good governance• visible, respected leadership• highly-qualified staff• clean audit• growing donor base

• transforming lives• ending poverty• closing the achievement gap• ending hep-C

* Yes! Certainly, a case can be made that only high-performing organizations can produce results, but the fact that the case can be made

does not eliminate the need to distinguish between the two. **

Systems Change – Areas of Focus

Context

Components

ConnectionsInfrastructure

Scale

Source: www.ccitoolsforfeds.org/doc/AFrameworkForEvaluatingSystemsInitiative.pdf

Defining focus areas of systems work

•Improving the political environment that surrounds a system so it produces the policy and funding changes needed to create and sustain it.

CONTEXT

•Establishing high-performing programs and services within the system that produce results for system beneficiaries.

COMPONENTS•Creating strong and effective linkages across system components

that further improve results for system beneficiaries.

CONNECTIONS

•Developing the support systems need to function effectively and efficiently.

INFRASTRUCTURE

•Ensuring a comprehensive system is available to as many people as possible to produce broad and inclusive results for system beneficiaries.

SCALE

Theory of Change menu for systems workContext Components Connections Infrastructure Scale

political will – public education campaigns, awareness-building, media advocacy, community mobilization, coalition-building

programs – quality programs, services, and interventions

network – linkages between and among programs, organizations, lifecycle, coordination

capacity – of individual entities or of the network to function effectively

access – ability to increase services and numbers served while maintaining system integrity

• increases in public awareness

• public engagement or mobilization

• reframing issue

• public will

• political will

• new or more diversified funding or investments

• policy changes

• new programs or services developed

• expanded individual programs or coverage

• improved program quality

• increased program efficiency

• outcomes that precede impact

• coordinated assessments, intake, outreach

• referrals across services

• coordinated planning across components

• shared data or administration

• MOUs between system components

• entities that oversee and coordinate subsystems

• funding streams that are less categorical

• leveraged funds

• systems-level standards

• professional development structures

• practitioner training and technical assistance

• system spread

• system depth

• system sustainability

• shifts in system ownership

Defin

ition

sAn

ticip

ated

Out

com

es

Evaluation design menu for systems work

•Has the intervention changed the political environment? •Has the intervention prompted the availability of new investments, policies, or practices that will enable changes in the system?

CONTEXT

•Did program activities produce the intended benefits?•Were interventions implemented as designed?

COMPONENTS•Did the initiative design and implement connections and linkages as intended?•Did the connections and linkages produce their intended impacts?

CONNECTIONS

•Did the initiative establish infrastructure or supports that are consistent with its objectives?•Did the infrastructure or supports achieve their objectives for effectiveness, sustainability, and quality?

INFRASTRUCTURE

•Did the initiative enable system scale up with quality and fidelity?•Did scale-up result in broad impacts for beneficiaries at a system-wide population level?

SCALE

Evaluation methodologies menu for systems work

CONTEXT

Theory of change evaluation

Case studies

Public polling

Key informant interviews or surveys

Media tracking and analysis

COMPONENTS

Program evaluation methodologies

Program monitoring

Quality assessments

Efficiency analysis

Customer surveys

CONNECTIONS

Program evaluation methodologies

System mapping

Network analysis

Customer surveys

INFRASTRUCTURE

Theory of change evaluation

Case studies

Performance audits

Management information systems

Practitioner data collection

SCALE

Population-based demographic analysis

Program evaluation methodologies

System or program monitoring

Results-based accountability

Questions? Contact?

mkarnold@rifoundation.org(401) 427-4009

http://www.rifoundation.org/TransformingRhodeIsland/FoundationImpact.aspx