Evaluation for Impact and Learning Asia Value Advisors Nov 6 2014

Victor Kuo, Ph.D.郭恩勝

November 6, 2014Hong Kong, SAR, China

EVALUATION IN FOUNDATIONS:ORGANIZATIONAL SYSTEMS FOR IMPACT AND LEARNING

VK

1. Areas of Expertise:• Philanthropy, organizational development and planning• Program evaluation• Education, arts and culture, health, conservation and environment

2. Work Highlights• VK Global Advising, Founder• Seattle Community Colleges, Director of Strategic Planning & Research• Bill & Melinda Gates Foundation, U.S. Education, Program Officers• Western High School, Anaheim, California, School Teacher

3. Education• Ph.D., Stanford University, Educational Administration & Policy• M.A., Columbia University Teachers College, Secondary Science Education• B.A., Pomona College, Biology

ABOUT THE SPEAKER: VICTOR KUO

VK GLOBAL ADVISINGVK

2

Workshop attendees will learn:

1. Approaches to Effective Philanthropy:• Concepts of effective philanthropy, evaluation designs, and evaluation utilization

• Strategies and tools for developing and evaluating projects

• Problems and criticisms about effective philanthropy, evaluating philanthropy

2. Organizing Philanthropic Foundations for Learning and Impact• Definitions and terms of organizational characteristics

• Models for arranging organizations to support learning and impact

• Strategies and tools to facilitate implementation

MAIN POINTS


3

I. Approaches to Effective Philanthropy1. Lecture 1: Effective Philanthropy, Concepts and Definitions2. Lecture 2: Evaluation, Concepts and Definitions

a. Activity 1 (individual): Building a Logic Modelb. Summary and questions

3. Lecture 3: Designing evaluations for philanthropic projectsa. Activity 2 (small groups): Developing Evaluation Questions for Different Audiencesb. Summary and questions

4. Summarya. Main points and key wordsb. Questions and criticisms

II. Organizing Philanthropic Foundations for Innovation and Impact1. Lecture 4: Characteristics of Evaluation in Philanthropic Foundations2. Lecture 5: Organizational Tools for Effective Philanthropic Foundations

a. Organization structures, charts and diagramsb. Roles and job descriptions of staff for evaluationc. Budgets and typical costs for evaluationd. Contracts and consultant agreements for evaluatione. Activity 3 (small groups): Developing a “Request for Proposal (RFP)” for an evaluation projects

III. Conclusion

OUTLINE


4

BIGGEST GIVERS


5

I. About the initiative1. Improve graduation rates in US high schools to 80%, especially for low income, minority students

2. From large high schools (1,000 students) to small (300-400 students)

3. School climate 3R’s: rigor, relevance, relationships

4. Improve attendance, progression, and achievement gains

5. More than US$1 billion from 2003-2007

6. 1,900 schools in more than 40 states

7. Chicago, New York City, California, Ohio, Texas, and Washington

8. Networks of school reform models; some research based

II. Foundation strategy 3-levels:1. State: policies2. Districts: alignment and options3. Schools: new school networks and models

III. The problem:1. U.S. high school graduation rates about 60% or less2. Only 32% ready for college or work3. American public schools perceived as broken; unequal

CASE 1: GATES SMALL HIGH SCHOOLS


6

I. About the evaluation1. Study timeframe: 2001-2004

2. External research organizations: AIR & SRI

3. Quantitative and qualitative methods; statistical comparison groups

II. Findings1. Schools supported by the foundation had higher levels of rigor, relevance, relationships

2. In math, progress no different than other schools

3. Findings were still early; updates in 2012 show some progress in New York City (70% 4-year

graduation rate)

III. Major challenges1. Teacher turnover

2. Unsupportive policies

3. Class scheduling

4. Inadequate curriculum

5. Lack of a common vision

6. Leadership changes

CASE 1: GATES SMALL HIGH SCHOOLS


7

I. About the initiative1. Some of students’ learning is through online delivery of content and instruction2. Student has some control over time, place, pacing3. Learning takes place in a supervised physical location away from home4. Students “rotate” between regular classroom instruction and online instruction at a computer

station5. Foundation invested in 5 charter school organizations; 13 schools

II. Foundation strategy1. Focus on early-stage “Innovation portfolio”

2. Contribute to the evidence base to better understand an emerging learning model

3. Did not want to report a single metric; early comparisons considered inappropriate

III. The problem1. How best to blend online and face-to-face instruction?

2. How to support development of an innovation

3. Opportunity to use technology to improve children’s lives

CASE 2: DELL BLENDED LEARNING


8

I. About the evaluation1. Study of schools in first, second, or third year of implementation

2. External research organization: SRI, FSG

3. Quantitative and qualitative methods; statistical comparison groups

II. Findings1. Early signs show students using blended learning outperform those not using it

2. Schools are using blended learning models to personalize learning

3. Teachers reported that students improve procedural skills more than higher order thinking

III. Major challenges1. Integrating online and off-line curriculum

2. Budgeting time and resources for technology infrastructure

3. Need continuous improvement and innovation approach at each school

CASE 2: DELL BLENDED LEARNING


9

I. Main points1. U.S. education reform is in an era of achievement2. Philanthropy can support innovation by taking risks and facilitating

learning

II. Challenges1. Education and schools resist change2. How do you introduce and sustain an innovation?3. Implementation

III. Future opportunities1. Disciplined innovation2. Investing in research, development, and evaluation3. Understanding why institutions stay the same and how to change them

SUMMARY


APPROACHES TO EFFECTIVE PHILANTHROPY

11

I. Evaluation in U.S. philanthropy grew rapidly in late 1990’s

1. New philanthropic foundations from business entrepreneurs

2. New thinking on strategic philanthropy

3. Media attention

II. The practice of evaluation in philanthropy has challenges

1. Unclear purpose for evaluation

2. Limited staff capacity

3. Lack of culture of transparency, learning

III. Recommendations

1. Encourage experimentation

2. Build networks

3. Share results; learn from failure

MAIN POINTS


12

• News articles about philanthropy grew in the late 1990’s

1. Very large foundations established

2. Wealthy businessmen, celebrities

3. Media attention can be used positively

MEDIA ATTENTION


MOTIVATIONS FOR GIVING

2011 UBS INSEAD Study on Family Philanthropy in Asia, p. 24

VK GLOBAL ADVISING 13

• Four types of motivations1. Affiliation

2. Sector interest

3. Impact

4. Pragmatism

• Second generation, entrepreneurs1. Motivated by impact

• Impact Investing1. Financial returns

2. Social returns

3. Environmental returns

VK

14

I. Key Concepts1. Strategy

2. Strategic philanthropy

3. Theory of change

4. Environmental or external scan

II. Historical developments1. Religious, tithing; “spray and pray”

2. 3 M’s: money, modesty, mystery

3. Social impact, value

4. Strategy and management

III. Examples1. Ford Foundation

2. Robert Wood Johnson Foundation

3. Bill & Melinda Gates Foundation

STRATEGIC PHILANTHROPY

VK GLOBAL ADVISING

IV. Tools1. Strategy frameworks

2. Theory of change diagrams

V. Challenges1. Top-down approach too simplistic

2. Adjusting to constant, rapid change

3. Sustaining change requires community

involvement

VI. Future directions1. Strategic philanthropy will continue

2. Balanced with community initiated

approaches

3. Consumer feedback platforms

VK

A STRATEGY FRAMEWORK

VK GLOBAL ADVISING 15VK

A THEORY OF CHANGE


CONSUMER REVIEW PLATFORMS

Great Nonprofits - 2007

GuideStar - 1994


Center for Effective Philanthropy

Grantee Perception Reports - 2001

VK

18

I. Key Concepts1. Organizational effectiveness

2. Organizational learning

3. Collective impact

II. Historical developments1. Growth of philanthropic advisories

2. Management and communications experts

3. Systems approaches to social change

III. Examples1. Grantmakers for Effective Organizations

2. David & Lucile Packard Foundation,

Organizational Effectiveness Program

3. Bridgespan, Arabella, RPA, FSG, TCC

4. Center for Effective Philanthropy

EFFECTIVE PHILANTHROPY

VK GLOBAL ADVISING

IV. Tools1. Scorecards, dashboards

2. Learning agenda

3. Systems maps

V. Challenges1. Resistance to “administrative expenses”

2. Difficult to connect organizational

effectiveness to program outcomes

3. Competition among NGOs for funding, few

incentives to work together

VI. Future Directions1. Data visualization for management

2. Crowd-sourcing

3. Collective impact

VK

DASHBOARD EXAMPLES


LEARNING QUESTIONS


LEARNING EVENTS

Activities Frequency Participants Goals

Foundation program meetings

4 x each year • Foundation program staff • Align strategies

• Review findings

• Consult experts

Grant portfolio review

1 x each year • Foundation program staff

• Evaluation staff

• Leadership

• Review findings

• Adjust strategy

Grantee conferences

2 x each year • External evaluators

• Grantees

• Review findings

• Adjust programs


SOCIAL NETWORK DIAGRAMS


23

I. Key Concepts1. Evaluation2. Logic model, output, outcome3. Innovation

II. Historical developments1. Social science research, academic2. Community focus, utilization focus3. Management consultants, accountants,

and financial advisors4. Diverse approaches

III. Examples1. Class size reduction studies2. Formative memo, Early College High

Schools3. Annual reports for College-Ready and Post-

secondary education programs4. Infographic

EVALUATION

VK GLOBAL ADVISING

IV. Tools1. Logic model diagrams

2. Evaluation and learning questions

3. List of priority audiences

4. Evaluation Request for Proposal (RFP)

V. Challenges1. Need staff with technical skills

2. Need to balance multiple purposes and

audiences for evaluation

3. Need resources to fund evaluations

VI. Future Directions1. Diverse approaches to evaluation

2. Growth of evaluation internationally

3. Technology, “big data”

VK

24

I. What is it?

• Visual representation, “picture”, of a project

II. Outcomes

• Changes in the condition of target entities (students, schools, district,

community)

III. Outputs

• A unit or a service delivered (students receiving books, teacher workshops

delivered)

IV. Strategies

• Activities conducted to achieve outcomes (a service, research, policy education)

LOGIC MODELS


LOGIC MODEL TEMPLATE


LOGIC MODEL EXAMPLE

26

27

LOGIC MODEL TOOLS

Mapping Change: Using a Theory of Change to Guide Planning and Evaluation.

GrantCraft, 2012, in ChineseVK GLOBAL ADVISINGVK

LEARNING QUESTIONS

I. Monitoring

1. What are indicators that show

progress?

2. What early trends are

occurring?

II. Process evaluation

1. How is the project being

implemented?

2. What is supporting or

hindering implementation?

3. What adjustments are needed?


III. Outcome evaluation1. What is the result of the project?2. How have people’s lives changed?3. What is the added value of the

project?

IV. Innovation1. What is emerging during the

project?2. How are people responding to the

project?3. What about the project should be

changed?

VK

EVALUATION AUDIENCES

I. Leadership

1. Foundation board of directors

2. Foundation executives

3. Program directors

II. Program management

1. Program directors

2. School administrators

3. School teachers


III. Policy

1. Provincial executive

2. City or town mayor

3. National policy makers

4. National funders

IV. Community

1. Parents with school children

2. Business leaders

3. Local elected officials

4. Civil society organizations

VK

30

• Technical report 1x / year

• Executive summary 1x

• Report brief 1x

• Profiles, vignettes 2 – 4x

• Formative memo 4x

REPORTING OPTIONS

VK GLOBAL ADVISING

• Board presentation 2 – 4x

• Policy brief 1x

• Conference presentations 1x

• Academic journal publication End

VK

STRATEGY AND INNOVATION CYCLES

Bill & Melinda Gates Foundation, The Strategy Lifecycle: A Guide, 2011McKinsey Learning for Social Impact White Paper, Foundation Evaluation Cycle, 2010

FSG From Insight to Action, 2007

31

32

I. Challenges

1. Punishment for negative findings

2. Motivations for giving are not about impact

3. Evaluation is not useful

II. Future directions

1. Cultivate a culture of learning and experimentation

2. Build networks of funders who focus on impact

3. Design evaluations with use in mind

SUMMARY


ORGANIZING PHILANTHROPIC FOUNDATIONS FOR INNOVATION AND IMPACT

34

I. Characteristics: U.S. Foundations1. 86,192 foundations in US (2013)

2. US$55 billion in giving: foundations

3. US$316 billion in private giving:

foundations, individuals, corporations

4. Types: independent (most), operating,

corporate, community

5. Largest: Bill & Melinda Gates with US$37

billion dollars in assets

II. Issue areas1. Education: US$5 billion

2. Arts and culture: US$3 billion

3. Health: US$6 billion

SECTOR STATISTICS – U.S.

VK GLOBAL ADVISING

III. Largest U.S. Foundations1. Bill & Melinda Gates Foundation

2. Ford Foundation

3. J. Paul Getty Trust

4. Robert Wood Johnson Foundation

5. William & Flora Hewlett Foundation

6. W.K. Kellogg Foundation

7. Lilly Endowment

8. David & Lucile Packard Foundation

9. John D. and Catherine T. MacArthur

Foundation

10. Gordon and Betty Moore Foundation

IV. Education1. U.S. all spending on education: US$1 trillion

2. Gates education spending: US$400 million

(<1%)

VK

35

I. Roles and Titles1. Vice President/Director of Evaluation2. Planning and Evaluation3. Research and Evaluation4. Strategic Assessment5. Impact Planning and Assessment6. Knowledge Management7. Organizational Learning

II. Kinds of evaluation activities:• Evaluation• Performance metrics, indicators• Knowledge management• Research other than evaluation• Aiding in development of program strategy• Grantee satisfaction

III. Evaluation staffing1. About 40 foundations with evaluation staff2. Typically 2-4 staff

CHARACTERISTICS OF EVALUATION

VK GLOBAL ADVISING

IV. Reporting to:• Chief Executive Officer

• Administrator: Chief Financial Officer, Chief

Operations Officer

• Director of Programs

V. History1. Evaluation function in philanthropy greatly

expanded in 1990’s

2. Evaluation grew with shift to “strategic

philanthropy”

3. Phases: research, planning, learning, strategy

VI. Challenges1. Accountability versus learning

2. Managing down versus community driven

3. Internal versus external use

4. “What” to evaluate in a foundation

VK

36

I. Small foundations: 7%• Less than US$50M in grants each year

• 7% of program budget for evaluation

II. Medium foundations: 2%• US$50M - $200M in grants each year

• About 2% of program budget for evaluation

III. Large foundations: 2%• More than $200M in grants each year

• About 2% of program budget for evaluation

COSTS FOR EVALUATION AMONG U.S. FOUNDATIONS

VK GLOBAL ADVISING

I. Project evaluations: 10-15%1. Develop logic model, questions

2. Collect and analyze data

3. Reporting, learning events

II. Donor Engagement: 1-4%1. Design and conduct survey



III. Scholarship, Donor Advised: 3-7%1. Develop questions



COSTS FOR EVALUATIONOF SELECT PROJECT TYPES

VK

STRATEGY ASSESSMENT OFFICER JOB DESCRIPTION #1

Responsibilities

1. Contribute to the foundation’s ongoing strategy development and planning

2. Help with the design and implementation of a foundation-wide system to measure social impact, inform learning

and drive improvement

3. Play an integral role in examining new investment opportunities and mapping best practices

4. Develop evaluation tools and metrics in partnership with program teams and grantees

5. Analyze and synthesize research findings to build the foundation’s knowledge management capacity

6. Communicate information gather from funded initiatives to policy makers and non-profits

7. Manage teams of outside vendors and partnersQualifications

8. Masters degree or experience equivalent; advanced degree preferred

9. At least four years of related work experience in strategy consulting, public policy, business, research or nonprofit

management

10. Outstanding analytical skills

11. Excellent communications skills; presentation, written and listening

12. Extremely well organized and strong project management capabilities

13. Bright, energetic, collaborative and goal-driven personality

14. Ability to work independently and take initiative

We are a $2 billion foundation and seek opportunities to build informed, engaged communities through our investments in media, news and information and

through our place-based grantmaking focused on social innovation and community engagement. We are committed to transformational change and the key

elements essential to achieve it – discovery of ideas, belief in a vision, courage to push for change, know-how to get it done and tenacity in getting results.

VK GLOBAL ADVISING37

VK

EVALUATION OFFICER JOB DESCRIPTION #2Responsibilities

1. Manage existing research, tool development, and knowledge management grants

2. Develop, advocate for, and make grants in support of strategic goals (includes soliciting and reviewing proposals,

preparing proposal summaries for director and executive review, and ongoing management of grants made)

3. Performa quantitative data analyses, which support a wide range of efforts within the education team. Contribute

to the team’s knowledge of emerging research and evaluation practices

4. Represent the division in a variety of forums by presenting to policy makers and practitioners in the education

field

5. Business travel typically 25%, but could be higher if needs dictateQualifications

6. A minimum of 5 years experience in the application of qualitative and quantitative research and analysis methods

to social science interventions

7. Masters degree in statistics, mathematics, economics or other quantitative social science; a PhD is preferred

8. Experience developing and executing research plans, selecting and managing outside vendors and solid

knowledge of a broad range of social science research applications, both primary and secondary

9. Experience with secondary schools and an understanding of urban, state and /or national education policy issues

is preferred

10. Demonstrated excellent verbal and written communication skills, with experience presenting to small and large

groups

11. Ability to travel 25% - 50%

VK GLOBAL ADVISING38

VK

39

I. What is a RFP?1. Describes the project background2. Outlines priority audiences, intended use, and preliminary priority questions3. Describes report products (deliverables), timeline and budget parameters4. May include a logic model5. May describe expected methods, data sources6. May describe submission requirements7. May describe applicant scoring and selection process8. May request work samples, client references

II. Typical sections in a proposal1. Context and background2. Evaluator’s professional capacities, qualifications3. Audiences and purposes of proposed evaluation4. Study questions5. Study design6. Data collection and analysis7. Reporting and communications plan8. Establishment of an Evaluation Working Group of client representatives to guide the evaluation9. Budget, timeline, staffing and management plan

REQUEST FOR PROPOSAL


40

I. Challenges

1. Developing organizational systems for monitoring, evaluation and learning

2. Organizational structure and culture

3. Demonstrating the value add of evaluation

4. Getting to use

II. Future directions

1. Diverse approaches to evaluation

2. Professional staff with multiple skills

3. Building a supply of philanthropy professionals

4. Establishing connections with related fields: social sciences, evaluation,

accounting, finance, strategy consulting

SUMMARY


41

KEY WORDS1. Activities: processes, tools, events, technology and actions that are

part of a program’s implementation

2. Assimilation: process by which a group’s language and culture come to resemble those of another group

3. Centralization: to concentrate authority or power in a central group

4. Charter school: a school that receives public funding but operates independently of local, regional or state school system

5. Collective impact: cross-sector collaboration among organizations with a common vision to achieve social change

6. Comparison group: a group that is exposed to all the same conditions as a treatment group, except for the intervention being tested

7. Control group: a group of untreated targets that is compared with experimental groups on outcome measures in impact evaluations

8. Dashboard: a performance management framework that identifies a few key measures, such as financial, customer satisfaction, internal business processes, and learning and development

9. Empowerment evaluation: a collaborative evaluation in which the evaluator’s role includes consultation and facilitation directed toward capacity building of stakeholders to conduct evaluation on their own and to use evaluation for advocacy, change, and personal change.

10. Evaluation: the systematic investigation of the value of a program or policy

11. Formative: for the purpose of program improvement

12. Generalizability: the extent to which an impact evaluation’s findings can be extrapolate to similar programs

13. Impact: the net effects of a program

14. Indicator: a measure of a concept or social condition

15. Innovation: a new way of doing something; incremental or radical change in thinking, products, process, or organization

16. Input: resources needed to operate a program

17. Institutional theory: a field of sociological thought that examines how and why organizations stay the same, change, and diffuse.

18. Learning organization: an organization that facilitates learning in its members and continuously transforms itself; learning organizations respond to their environment to remain competitive

19. Logic model: a diagram or illustration that shows the connection between planned work and intended results; usually composed of inputs, outputs, and outcomes

20. Social network: a social structure made up of actors, such as individuals or organizations, and the relationships among them

21. Organizational effectiveness: the extent to which an organization is suited to a task or environment; some models emphasize organizations that seek goal attainment, resilience and survival, or adaption.


42

KEY WORDS22. Outcome: changes in the conditions of target entities, usually

knowledge, skills, or behaviors

23. Output: direct products of program activities; units of services delivered

24. Philanthropy: “love of humanity”; the use of private wealth to promote social good

25. Qualitative: method of inquiry or investigation that often seeks to answer questions of “why” and “how”, using non-numerical data

26. Quantitative: method of inquiry or investigation that often seeks to answer questions of “what” using numerical data

27. Randomized controlled trial: an impact research design in which experimental and control groups are formed by random assignment

28. Research: process of steps to collect and analyze information to advance knowledge or understanding of a issue

29. School reform: planned efforts to change schools in order to correct perceived social and educational problems

30. Standards-based reform: educational change through specifying skills and levels of competency that students must exhibit, usually through formal tests

31. Strategic philanthropy: approach to philanthropy of not only providing grant support to nonprofits, but to assess social problems, develop strategies to solve them, track the results of their efforts, and use results to adjust future strategies

32. Strategy: a framework for decision-making that is focused on external context and includes a hypothesized causal connection between the use of resources and goal achievement

33. Summative: for the purpose of judgment; to determine if goals were met

34. Theory of change: a theory of how and why an initiative works; describes the pathway of an initiative and the action strategies intended to lead to the achievement of outcomes

35. Treatment group: a group that participates in a program or treatment

36. Utilization focused evaluation: an approach to evaluation where an evaluation is judged based on its usefulness to intended users. In this approach, primary users should be clearly identified, and the evaluator should ensure intended uses guide the evaluation process.


43

SELECT REFERENCESArgyris, C. & Schon, D. A. (1974) Theory in practice: Increasing professional professional effectiveness. San Francisco: Jossey-Bass. Ayres, L. (1917).

Cleveland Education Survey, The Cleveland School Survey. Cleveland, OH: The Cleveland Foundation.

Bernatek, B. (2012). Blended Learning: Introduction. Dallas, TX: Michael & Susan Dell Foundation.

Buteau, E., et al., (2009) Essentials of Foundation Strategy. Boston, MA: Center for Effective Philanthropy

Carmen, J. G. & Fredericks, K. A. (Eds.) (2008). Nonprofits and Evaluation. New Directions for Evaluation, 119.

Coffman, J., Beer, T., Patrizi, P., and Thompson, E. H. (2013). Benchmarking evaluation in foundations: Do we know what we are doing? The Foundation Review5(2).

Chelimsky, E. (2001). What evaluation could do to support foundations: A framework with nine component parts. American Journal of Evaluation 22(1), 13-28.

Chen, M. Y. (2006). Developing China's Nonprofit Sector. McKinsey & Co.

Connell, J. P. & Klem, A. M. (2000). You can get there from here: Using a theory of change approach to plan urban education reform. Journal of Educational and Psychological Consultation, 11(1), 93-120.

Fetterman, D. F., Kaftarian, S., & Wandersman, A. (1995). Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks, CA: SAGE Publications.

Field, K. & Holdaway, X. (2013, July 14). A Realm of Influence. Chronicle of Higher Education

Fleishman, J. L. (2007). The Foundation: A Great American Secret. New York: Public Affairs

Fulton, K. & Blau, A. (2005). The Future of Philanthropy: An Orientation for Twenty-First Century Philanthropists. San Francisco, CA: Monitor Group.

Godeke, S. & Bauer, D. (2008). Philanthropy’s New Passing Gear: Mission-Related Investing. New York: Rockefeller Philanthropy Advisors.

GrantCraft. (2012). Mapping Change: Using a Theory of Change to Guide Planning and Evaluation. New York: GrantCraft. (in Chinese)

Kaplan, R. S & Norton, D. P. (1992). The Balanced Scorecard - Measures That Drive Performance. Harvard Business Review (January–February): 71–79.

Kramer, M., Graves, R., Hirschorn, J. & Fiske, L. (2007). From Insight to Action: New Directions in Foundation Evaluation. Boston, MA: FSG Social Impact Advisors.

Kuo, V. (2010). Transforming American High Schools: Possibilities for the Next Phase of Reform. Peabody Journal of Education, 85: 3, 389-401.

Lavizzo-Mourey, R. (2004). Perspective. IN Braverman, M. T., Constantine, N. A. & Slater, J. K. (eds.). Foundations and Evaluation: Contexts and Practices for Effective Philanthropy. San Francisco: Jossey-Bass.

Mahmood, M., Santos, F., Ghaniah, E., & Liao, J. (2011). UBS-INSEAD Study on Family Philanthropy. Singapore: UBS Philanthropy Services and INSEAD.

McKinsey & Company. (2010). Learning for social impact: What foundations can do. London: McKinsey & Company.


44

SELECT REFERENCESPatton, M. Q. (2008). Utilization-focused Evaluation. Thousand Oaks, CA: SAGE Publications.

Plantz, M. C., Greenway, M. T., & Hendricks, M. (1997). Outcome measurement: Showing results in the nonprofit sector. New Directions for Evaluation 75, 15-30.

Porter, M. E. & Kramer, M. R. (1999). Philanthropy’s new agenda: Creating value. Harvard Business Review, 121-130.

Powell, W. W. & DiMaggio, P. (1991). The New Institutionalism in Organizational Analysis. Chicago: University of Chicago Press.

REDF Roberts Enterprise Development Fund. (2001). SROI Methodology: A Social Return on Investment, Analyzing the Value of Social Purpose Enterprise Within A Social Return on Investment Framework. San Francisco: The Roberts Foundation.

Rossi, P. H., Freeman, H. W., and Lipsey, M. W. (1999). Evaluation: A Systematic Approach, Sixth Edition. Thousand Oaks, CA: SAGE Publications.

Sanders, J. R. (1997). Cluster evaluation. In E. Chelimsky & W. R. Shadish (Eds.), Evaluation for the 21st Century. Thousand Oaks, CA: Sage.

Scott, W. R. (1992). Organizations: Rational, Natural, and Open systems, third edition. Edgewood Cliffs, New Jersey: Prentice Hall.

Senge, P. M., Kleiner, A., Roberts, C., Ross, R., & Smith, B. (1994). The fifth discipline fieldbook: Strategies and tools for building a learning organization. New York: Crown Business.

Shear, L., Means, B., Mitchell, K., House, A., Gorges, T., Joshi, A., et al. (2008). Contrasting paths to small-school reform: Results of a 5-year evaluation of the Bill & Melinda Gates Foundation’s National High Schools Initiative. Teachers College Record, 111, 1986–2039.

Snow, P. E. & Baxter, L. W. (2002). Framing the big picture: Cluster reviews. Returning Results. Philadelphia, PA: Pew Trusts.

Taylor, F. W. (1911). The Principles of Scientific Management. New York: Harper.

Tyack, D. & Cuban, L. (1995). Tinkering Toward Utopia: A Century of Public School Reform. Cambridge, MA: Harvard University Press.

United Way of America. (1996). Measuring Program Outcomes: A Practical Approach. Alexandria, VA: United Way of America.

Valente, T. W. (1995). Network Models of the Diffusion of Innovations. Creskill, NJ: Hampton Press, Inc.

W. K. Kellogg Foundation. (1998). Logic Model Development Guide. Battle Creek, MI: W. K. Kellogg Foundation.

Wasserman, S., & Faust, K. (1994). Social Network Analysis: Methods and Applications. Cambridge, U.K.: Cambridge University Press.

Wilhelm, I. (2007). Philanthropy’s New Frontier. Chronicle of Philanthropy, September 20, 2007.

Weiss, C. H. (1998). Have we learned anything new about the use of evaluation? American Journal of Evaluation, 19(1), 21-32.

Woodwell, W. H. (2005). Evaluation As A Pathway to Learning: Current Topics in Evaluation for Grantmakers. Washington, D.C.: Grantmakers for Effective Organizations.


45

Victor Kuo, Ph.D.

郭恩勝

VK Global Advising

[email protected]

Seattle, Washington, USA

CONTACT


Evaluation for Impact and Learning Asia Value Advisors Nov 6 2014

Government & Nonprofit

Transcript of Evaluation for Impact and Learning Asia Value Advisors Nov 6 2014