Democracy Surveys as a Tool in Impact Evaluations

Promise and Challenges

Dr. Margaret SarlesMargaretsarles@gmail.comUS Agency for International DevelopmentCairo, March 31, 2009

Presentation objectives:

• Review the challenges of measuring impact of programs in democratic development

• Prospects and cautions of using national democracy surveys as an impact evaluation tool

• Dialogue: How can we make them more useful?

Why is this an important topic?

• A growing consensus that democracy is a component of development, either as a core objective or to further economic development

• Foreign policy priority for many countries: human rights, ROL, free and fair elections, democratic states, democratic cultures: policy imperatives and concern with trends

• High investment levels but little scrutiny of effectiveness

• Evaluations of democracy investments provide the evidence of what works, what doesn’t, under what circumstances: USAID commitment.

Measuring Impact on democratic change is difficult

• The general concepts are contested and vague. What is success? At what level? (overall, components, programs)

• Extremely diverse programs are grouped as “democracy” – community policing, strengthening civil society, elections, parties,

justice systems, human rights, transparency, anti-corruption, decentralization

– anything that could be said to be part of the political system or that contributes to democratic change (in any program)

• “Politicized” definitions by stakeholders• *Practice outruns theory and academic analysis on

processes, sequencing and causal links of democratic change variables (relevance)*

But the time is right to begin impact

evaluations in democratic development • Volume of academic work is exploding. Case

studies, quantitative work hypotheses• Renewed donor emphasis on evaluation and

evidence (including this conference)• USAID: (a) 20+ years’ experience testable

hypotheses, and (b) National Academy of Sciences recommendations and other studies

• In addition to burgeoning hypotheses, more and better data are available, particularly democracy surveys.

Regional surveys are expanding• World Values surveys: 1981; 4 waves; 80+ countries;

broad sociocultural scope

• Afrobarometer: 1993; 4 rounds; 18 countries

• Arab Barometer: 2005 “to produce scientifically reliable data on the politically relevant attributes of ordinary citizens….to contribute to political reform….” 1 round (2007), 5 countries.

• Asian Barometer: 2001; 13 E. Asian and 5 S. Asian countries; 2 rounds

• Latinobarometro: 1995, 18 countries, annual, views on economics, trade, politics

• AmericasBarometer; 1992 (1970s) 23 countries; up to 10 rounds

• Plus many independent country-level surveys

USAID has been the dominant donor supporting AmericasBarometer first as a

policy tool (NOT evaluation)

• Began with policy: The Peace Accords in Guatemala: The DIMS (now 6 rounds): what do citizens think about and expect of democracy?– Diagnostic tool for governments (legislatures, executive) and

donors– Mobilizing tool/data for reform– Increased interest in democracy reform in society (media,

NGOs; especially comparative)– “Voice” of the people in imperfect democracies– Reports are geared towards a policy audience

• This becomes a winning scenario for increasing stakeholders and keeping support

But they did also provide excellent evaluation measures-- due to

methodology

- Time series data– Transparency and high standards (sample

frame, pre-tests, etc.)– Inclusiveness (6 languages in Guatemala)– Allows multivariate analysis, and easy use for

both qualitative and quantitative analysis– Learning from other surveys: concepts,

definitions– Training of USAID democracy officers

And are good evaluation tools because of substance

• Survey questions include (1) behavior (e.g.,voting, corruption, participation; (2) political culture (e.g.,system support); (3) knowledge; (4) attitudes (e.g.,trust); (5) personal attributes

• USAID Democracy officers participated in questionnaire development, for relevance (now true for other donors)-- “Black,” “gray,” and “white” questions (ex:DLG).

• Result: Used in Bolivia in over half the performance indicators of program success (ex.)

Improvements now make surveys useful for impact evaluations

• Introduction of over-sampling in geographically based programs: 2004

• Allowed study of counterfactual for the first time• Richer comparisons (against national and other country

data)• Therefore had before/during/after, and an approximation

of with/without

• Evidence that randomization may not decrease chances of project success (rarely implemented; now being investigated )

However, they require adaptation to be used in impact

evaluation• Biggest requirement, and least understood,

is over-sampling• Randomization is widely resisted • Questions (and indices) usually need to be

added to be relevant to a particular intervention (often difficult)

• Counterfactual methodologies need to be (1) developed and (2) costed out

• Long-term commitment should be as guaranteed as possible, understanding that

Examples of Impact Evaluations Using National Surveys

• Bolivia Democratic Development and Citizen Participation Project (1996 - 2003)– Over-sampled treated municipalities; compared to non-treated

municipalities, to national sample, and to other countries.

– Questions focused on participation and satisfaction

• Kenya Civic Education Project – Innovative use of “non-treated” population through matching

– Used surveys to determine beliefs and levels/kinds of participation

Both evaluations used multiple methods, even though dominated by survey results

Nonetheless, there are significant challenges to using survey research

in impact evaluations(1)• Conceptual: Linking questions to (1) theory;

(2) processes of democratic change • Methodological: ensuring basic standards

are met for design and implementation. – Need for transparency– Post-evaluation analysis is possible but late (Exs:

Paraguay; LAPOP) – New methods help get it right in the first place

(PDAs, GPS, institution-building). national and trans-national design and methods

What are the challenges of using survey research in impact

evaluations? (2)

• Administrative: funding over project life (Guatemala); complex planning and budgeting for multi-country surveys; developing stakeholders in each country;

• Maximizing use of them to justify costs, while recognizing surveys are usually not sufficient for measuring impact on democratic change

What should be done to improve surveys for impact evaluations?

• Learn from the health model: periodic, institutionalized, health surveys with committed funding, widespread acceptance

• Support research on democracy indicators derived from survey questions and indices (gap analysis, basic research)

• Educate developmentalists on over-sampling, and other appropriate survey techniques (randomization, matching, etc.)

• Institutionalize funding/administration of surveys, against the present dispersed, decentralized, often competing regional surveys

References• AmericasBarometer.com (All country and regional reports and data from Latin American surveys

undertaken by Mitchell Seligson, Vanderbilt University)• Bollen, Ken et. Al. “Assessing International Evaluations: An Example from USAID’s Democracy and

Governance Programs,” American Journal of Evaluation, 26: 189-203, 2005• Chemonics International Inc. “Bolivia Democratic Development and Citizen Participation Final Report, 196-

2003,” November 2003• Finkel, Steven E. “The Impact of the Kenya National Civic Education Programme on Democratic

Atttitudes, Knowledge, Values, and Behavior. “ Report prepared for US Agency for International Development, Nairobi, Kenya (AEP-I-00-00-00018, Task Order No. 806). December 30, 2003.

• _________. “Kenya National Civic Education Impact Study: Report on Pre-Test, Prepared for US Agency for International Development, “Nairobi (Management Systems International and U. of Virginia) August 13, 2002.

• National Research Council of the National Academies, Improving Democracy Assistance: Building Knowledge Through Evaluations and Research. National Academies Press, Washington, DC 2008 (available download on www.nap.edu)

• Sarles, Margaret J. “Evaluating the Impact and Effectiveness of USAID’s Democracy and Governance Programmes,” in Peter Burnell, Evaluating Democracy Support: Methods and Experiences (IDEA and SIDA), 2007.