The aid effectiveness literature:

The sad results of 40 years of researchHristos Doucouliagos & Martin Paldam

Presented by

Maria Kuecken &Alexa Pattberg

12-11-10

Development Economics (Chatelain)

M2R Economie de la Mondialisation

1. INTRODUCTION: ANALYZING THE PROCESS OF RESEARCH IN AID EFFECTIVENESS

1.1 What is meta-analysis and why is AEL ideal?  MA is a systematic review procedure which combines results of

several studies all dealing with a similar set of hypotheses and estimate the same effect (aid effectiveness)

Ability to control for between-study variation 

AEL investigates the effect of aid on growth, savings and investment.

μ = ∂g/∂h (growth/aid share = aid effectiveness)

AEL only uses very similar models to estimate the same effects (μ) and differences can be quantified

It can be used as a tool for detecting and correcting publication selection bias in empirical economics (Roberts and Stanley, 2005; Stanley, 2005)

1.1 WHAT IS META-ANALYSIS AND WHY IS AEL IDEAL?

Meta analysis tries to answer the following questions:

Q1: Do the estimates in the AEL converge to something we might term ‘truth?’

Q2: Can we identify the main innovations, which cause or prevent convergence?

Q3: Are there biases along the way in uncovering ‘truth’ about aid effectiveness?

Research is a mixture of the: (a) innovation of theory, models and estimators producing new results and (b) independent replication by other authors and data seeking to confirm earlier innovations.

  All involved hope that the research process converges to

some notion of ‘truth.’

This paper combines results from previous MA conducted in 2006, 2007, and 2008.

1.1 WHAT IS META-ANALYSIS AND WHY IS AEL IDEAL?

AID - Growth: a controversial subject

No correlation between aid and growth (See Figures 1a and 1b below) Many different results in studies Example: India and China hardly receive aid and have enormous growth rates which

causes mass poverty to fall more than ever before

1.2 THREE PERENNIAL PROBLEMS

(1) Data mining: Limited data available, which are used by many different researchers

“Fishing in the common pool” of available degrees of freedom Reminder: the number of independent pieces of information that go

into the estimate of a parameter is called the degrees of freedom

 (2) Priors & (3) Incentives: Research question determined by

certain interests (personal or institutional) as aid is an emotional issue with a multi-billion dollar industry

  Stopping rule: Researcher stops investigating when his/her major

interest is satisfied or when research question is answered

  Innovation vs. Replication: Independent replication can verify

(strengthen) new results which should in general be treated with caution

  Reluctance hypothesis: Researchers seem reluctant to publish

negative results which biases empirical outcomes

Can work at micro level (stopping rules) or macro level (publication biases)

1.3 A PREVIEW OF THE RESULTS

3 Families of Models

2. ABSOLUTE AID EFFECTIVENESS AND THE AID PARADOX

E1: The social rates of return aimed by aid agencies in their projects is approximately 10% so that an aid share of 1% (of GDP) should result in a growth rate of 0.1%.

  Project evaluations find that 50% of all projects succeed thus the growth

rate would be between 0.05 and 0.1%. Average share of total aid is 7.5% which indicates a growth of 0.4 to 0.8

percentage points. Average LDC growth is about 1.6% so aid should explain between 25%

and 50% of the growth rate of developing countries. E2: Learning-by-doing should have had an impact on aid effectiveness

during the past 40 years. Barro and Sala-i-Martin (2004) found the order of magnitude of learning-by-doing of 1 to 2% per year which would mean an increase in aid effectiveness of around 50%.

  This effect would be visible in a positive trend in empirical estimates of

μ, μ = μ(t), and μ = μ(N). 

But also negative trends are observed in the two equations which is supposed to prove the reluctance hypothesis.

2.1 THE DATA, DEFINITIONS AND THE ZERO CORRELATION RESULT

From the 1960’s aid statistics have rapidly accumulated data and now grow by about 140 observations per year (WDI).

WDI covers 156 LDCs, starting from 1960 in 2000. We should have 6,200 observations when 4-year average calculated: 1,550

observations with 35% missing still leaves the 1,008 4-year averaged observations

g, real growth rate = percent of GDP per capitah = H/GDP where h is the share of total development aid H over GDP

 

2.2 FOUR REASONS WHY THE AID INEFFECTIVENESS RESULT IS PUZZLING (WHY NOT IGNORING AID)

R1: The ‘why would they’ argument.

Why did aid continue for over 40 years?

Average aid of donors is only 0.3% of their GDP and decreased. This aid fatigue is attributed to dissatisfaction of donors due to small effects. But a huge aid industry has evolved with stakeholders continue to follow their interests/activities. Some aid is given to non-humanitarian purposes (commercial, security and human rights).

 

R2: The micro evidence

50% of aid projects succeed - small aggregate effect but large individual effect

Micro - Macro paradox

Aid is partly fungible, so donors could chose a project which would have been implemented by the recipient anyhow. The marginal project (caused by aid) is likely to be less effective than the aid project (financed by the aid).

 

R3: Standard growth theory

Increased accumulation (investment) causes growth

Aid could be seen as investment. But aid does not only work through accumulation effect but could involve other channels such as health or education.

 

R4: Standard macro theory

Aid has a positive effect on the BoP and public spending

Some effects can be crowded out: “Ricardian equivalence” states that repayments to be made can be crowded out by higher savings.

As aid does not need to be repaid (includes a gift component) the positive effect should be proportional to the gift component of aid (EDA , effective development aid) and not to the total aid (ODA , official development assistance)

R1 and R4 suggest that aid should help--but it doesn’t seem to: aid paradox

2.3 SOME ADDITIONAL OBSERVATIONS

Aid share for developing countries is on average 7.5% of their GDP (substantial amount)

Raw data (from figure 1) does not show any correlation between aid and growth

Any definite result (positive or negative) must have relied on other components

EDA (introduced by Chang, Fernandez-Arias and Serven in 1998) is measure of aid, where each grant is weighted by its gift element (different from ODA - gift element above a threshold of 25%) The two data sets have a correlation of 0.83 -> very similar

Doucouliagos and Paldam (2008) have introduced an EDA dummy and found it to be negative. Thus they rejected the Ricardian Equivalence (no crowding out by savings) policy problem

Although the average share of aid in developing countries is 7.5%, the aid actually received by individuals is smaller as aid per capita falls with the size of population. Example of India and China: they contain 40% of the LDC population but receive

less than 1/4 of aid. Weighting the aid shares by population the result of the average share declines by 2.5% of GDP (before7.5%)

3.1 THE STRUCTURE OF THE AEL: THREE FAMILIES OF MODELS

Neoclassical model: focus on savings and growth Mid 1980’s to mid 1990’s: focus on investments Beginning of 1990’s: wave of aid-growth models Since 1995: conditional growth models

3.1 THE STRUCTURE OF THE AEL: THREE FAMILIES OF MODELS

The models only differ in only three ways:

(i) the data sample on which they are estimated

(ii) the set of control variables out of a “master set” of 60 variables

(iii) the exact econometric model employed

ideal for meta-analysis

3.2 DATA-MINING AND ITS DISCONTENTS

97 papers with 182 models of the 3 families, containing 1,113 regression estimates (11 per paper); half from Family B (543)

 Here two data sets have been taken: The best-set: single empirical result preferred by the author (182 data points) The all-set: each reported regression estimate is taken as a data point (1,113 data

points)

Data mining – common pool problem leads to mining collective Existence of over 5,000 annual observations of aid share in GDP About 1,000 when averaged over 4-5 year period On subsets of these 1,000, regression estimates number 1,113, with highest number

of observations at 825 BUT sum of sample sizes = 30,516 (30 times the 1,000 data available/mining ratio

30) High mining ratio prevents a valid interpretation of the t-test Mining increased the probability of Type I errors

4. META-ANALYSIS: PRIORS, INTERESTS & BIASES

Q3: Are there biases along the way in uncovering ‘truth’ about aid effectiveness?

At the operational level: biases from misspecification, choice of data, wrong

estimator, or wrong econometric model

Genuine errors can be fixed by learning or by better understanding the data

generation process (slow and discontinuous)

Signs of coefficients tend to be ‘established’ so subsequently publishing

estimates with the ‘wrong’ signs becomes difficult

A researcher’s acceptance of his /her results determined by stopping rule,

influenced by priors

4.1 PRIORS COMMONLY DETECTED BY META-STUDIES

(1) Polishing

Researchers have to publish to flourish, and journals want clear results.

Example: Easier to publish studies with statistically significant results and definitive conclusions.

Results: Polishing causes results to be ‘too good’ – if polishing were neutral it would increase the number of significantly positive and negative results reported.

(2) Ideology

Authors may hold an ideology that is consistent with a given outcome.

Example: Those with strong Marxist or libertarian viewpoints have priors for finding that aid is harmful.

Results: Some authors express political-ideological views and find results in accordance with these views.

4.2 THE GOODNESS/INTEREST TANGLE AND THE RELUCTANCE HYPOTHESIS

(3) Goodness

Researchers want to be seen as ‘good’ and for their activity to have a ‘good’ purpose. Example: It may be seen as morally superior to report significantly positive results. Results: To find a negative effect of aid is to question this ‘do-good’ enterprise; hence the

‘reluctance’

(4) Author history

Previous writings of the author and their associates cause path dependence. Results: 50% of AEL authors participate in more than one paper. Several groups compete

for the preeminence of their model.

(5) Institutional Interest

Authors often work for an institution with an interest in the results as much of AEL is financed by aid industry. Results: Generates ‘reluctance’ to publish conflicting results

Some reasons for (4) & (5): loyalty within organizations, career pressures on employees, selection/self-selection of organizations and employees

Biases and interests could be minimized if there were enough competing biases and interests.

4.2 THE GOODNESS/INTEREST TANGLE AND THE RELUCTANCE HYPOTHESIS

At least 35% of AEL researchers work for aid industry

Impossible to fully identify the institutional interests of all 104 AEL authors: 72 give university affiliation

Little bias in over-reporting positive results (Doucouliagos and Paldam, 2008)

4.3 META-ANALYTIC METHODS Meta-significance test (MST) and precision-effect testing

(PET) are used to check for the existence of a genuine empirical effect (Stanley, 2005; 2008)

Funnel asymmetry test (FAT) identifies the existence of publication selection or ‘reluctance’ Shows the distribution of a set of estimates of the same parameter Estimate distribution of estimates of some coefficient, μ = μ(N), and its t-values

as a function of sample size, N, or precision, 1/SE

From Callot and Paldam (2010):

4.3 META-ANALYTIC METHODS

Methods detect 3 properties:

(1) Convergence: Results should converge to something that differs from zero, if there is a genuine empirical effect.

(2) Polishing: Easier the smaller the sample and precision so may be detected when the reported effect increases with its standard error (Stanley, 2005; 2008) Using FAT, Doucouliagos and Paldam (2008) confirm that reported

aid effects increase with standard errors

(3) Asymmetries: The distribution of the estimated coefficients of the same effect should be symmetric around the true value. An asymmetry means that the research process is systematically

biased. Using FAT, research process favors positive aid effects consistent

with reluctance hypothesis

Note: Advances in econometric techniques are not the cause of variation in the results.

5. DOES AID CAUSE INCREASING ACCUMULATION? Results from Family A

Family A focuses on accumulation as financed by aid Subject of the Doucouliagos and Paldam (2006) meta-analysis

5.1 The savings effect is crowded out

Aid flows decreased savings in the recipient countries by the same amount Griffin and Enos (1970) and Weisskopf (1972b)

Marginal activity generated by aid did not lead to increased accumulation (fungibility)

Many studies analyze effect of aid on the rate of savings or investments: 29 studies reports 90 aid-savings effects, and 37 studies report 122 aid-investment effects

5.1 RESULTS FROM FAMILY A

The savings-investment identity for an open economy is:

I − S =(IP − SP ) + (IG − SG ) = −XMB

Aid allows investment to rise by the amount of H, provided that S does not fall. If S falls by H, the potential rise in I is fully crowded out.

One explanation: Aid leads to an increase in public consumption only - government savings rate falls correspondingly

5.2 THE RESULTS: A LARGE BUT PROBABLY NOT A FULL CROWDING OUT

5.2 THE RESULTS: A LARGE BUT PROBABLY NOT A FULL CROWDING OUT

Aid increases accumulation by about 25% of the aid.

Remaining 75% causes an increase in public consumption and hence a fall in public savings.

Both effects are of dubious significance as total effect on growth depends on what the remaining 75% of the aid does to the economy. Unproductive if it leads to public consumption because this is

known to have a negative effect on growth (Barro and Sala-i-Martin, 2004, p.525-26)

AEL papers that include public consumption in a growth regression tend to find a negative coefficient.

6. DOES AID CAUSE INCREASING GROWTH? RESULTS FROM FAMILY B

The literature of the reduced growth models (family B) started as a study of convergence, using Barro’s equation:

With a suitable set of control variables, conditional convergence occurs.

The convergence term was then replaced by the aid effectiveness term

6. DOES AID CAUSE INCREASING GROWTH? RESULTS FROM FAMILY B

Literature on Barro growth empirics has tried ~400 control variables of which 60 have been tried in the AEL (average number of control variables used is ~5)

This gives possible models from which to experiment

Even if the true value of μ is zero, 5% (or more) of these estimates will be significant at the 5% level. Of these, half should be positive.

Crucial to test results for robustness via independent replication (still neglected in AEL)

The 543 observations have a small positive average, so it is not surprising that it proves insignificant in explicit meta-regression

6.2 THE DEVELOPMENT OVER TIME, RELUCTANCE, AND SUMMARY

6.2 THE DEVELOPMENT OVER TIME, RELUCTANCE, AND SUMMARY

6.2 THE DEVELOPMENT OVER TIME, RELUCTANCE, AND SUMMARY

The two regressions show obviously multicollinearity (multiple regression several predictor variables are correlated)

Main reason why N rises is because as time passes more observations are published

6.2 THE DEVELOPMENT OVER TIME, RELUCTANCE, AND SUMMARY

Conclude that:

(C1) The variation is falling over time and with the sample size.

(C2) The best-set is typically chosen among the more extreme points.

(C3) The average result is steadily decreasing (Figure 5b). It is now +0.02 on the figures and it seems to converge to zero. Precision-effect testing confirms this small and insignificant aid effect (Doucouliagos and Paldam, 2008, p.11).

(C4) The funnel is not symmetrical around a horizontal or a rising average line (Figure 5a). The asymmetry is confirmed by the FAT (back to reluctance hypothesis)

Note: small samples give misleading results • If the same graph was presented for the first 250 estimates only, the trend line would be μ(t) = 0.174 (4.8) – 0.00008 (0.3)t T• This would lead to the conclusion that aid is amazingly effective with no asymmetry

3 facts to consider when assessing the size of the true value of μ:

(F1) The trends of the results from Figures 5a and b have reached aid-growth effects of 0.02 to 0.04. These values are of no economic significance and are not significantly different from zero.

(F2) Estimates of the growth-aid effect have yet to converge to anything ‘final’, and the average aid effect continues to fall as more evidence is accumulated.

(F3) There is a certain amount of reluctance to publish negative results, so the reported research is biased upward.

The AEL has yet to overcome the zero correlation result.That is, there are no robust and theoretically well founded set of control variables that

turns zero absolute effectiveness into a positive aid-growth effect.

6.2 THE DEVELOPMENT OVER TIME, RELUCTANCE, AND SUMMARY

• Another way to interpret zero correlation: aid works in some cases and not in others (aid-conditionality)

• Criterion, z, is searched for and scaled to have an average of ~0, where z > 0 causes aid to work, and z < 0 causes aid to harm the economy

•The interacted variable has a significant positive coefficient when the following model is estimated:

•The AEL has identified 10 candidates for the role of z over the last decade but focus on 2:

(1) Good Policy Model covering 23 studies (and 232 estimates)

(2) Medicine Model covering 16 studies (and 123 estimates)

7. IS THE EFFECT OF AID ON GROWTH CONDITIONAL? RESULTS FROM FAMILY C

•Aid works if the recipient country pursues good polices, and is harmful in countries pursuing bad policies.

•The Good Policy Model has 2 equations. (1) gives the Good Policy Index, zit, as a weighted sum of the budget balance the inflation rate and trade openness while (2) is the aid effectiveness relation.

In the original findings (Burnside and Dollar 2000), μ is insignificant, while both δ and ω are positive and significant. •BUT doesn’t withstand independent replication: when the standard tools of meta-analysis are applied to the 23 papers and 232 conditional-aid effects, key coefficient ω to is insignificant

7.1 GOOD POLICY AS THE CRITERION FOR A DIVISION OF THE SAMPLE

BURNSIDE AND DOLLAR (1996; 2000)

Uses aid itself as the condition, so previous model reduces to:

Two important coefficients: μ and ω• When μ > 0 and ω < 0 results in inverted parabola for excess growth, which has a maximum at h = h* and a positive section between h = 0 and h = 2h*

Marginal contribution of aid to growth is 2ωh < 0•About 25% of aid term squared support found in papers of Danida group.•BUT fares less well when estimated on other data sets:•Taken together, the 15 papers and 123 stimates of the Medicine Model fail to prove decisively that the two key coefficients are statistically different from zero.

7.2 THE MEDICINE MODEL: THE EFFECT OF AID SQUARED(HADJIMICHAEL ET AL (1995) AND DANIDA GROUP)

8. A PARALLEL LITERATURE: RESOURCE RENTS & DUTCH DISEASE

8.1 The Dutch disease, the resource curse and the transfer problem

Aid can be seen as an external rent leading to a “transfer problem.” Increase in income level “paid for” by lower growth rate

Dutch Disease - aka the resource curse Resource discovered -> labor shifts -> increase in revenue from the

resource -> stronger currency -> other exports become more expensive for foreign countries to buy -> less competitive -> decrease in the growth rate

The resource rent (surpluses) of LDC’s is twice as high as the aid received and much more unequally distributed.

Back to fungibility: Distinction between aid and resource rents makes little difference since both used to finance public spending.

Should see similar models but very little exploration of the Dutch Disease in AEL

8.2 THE KEY ROLE OF THE REAL EXCHANGE RATE

Dutch Disease/rent transfer implies a real revaluation of the country’s currency due to a change in the exchange rate. Reduction in economy’s competitiveness Losses in all other sectors except the booming one

On the macro level aid therefore does not seem to be favorable as originally assumed (the size of the effect matters, not the sign).

Examples Post-reunification Germany Greenland Tanzania

Thus, exchange rate movements have potentially substantial effect on aid effectiveness. But role of aid in exchange rate movements still waiting to be discussed.

9 CONCLUSIONDevelopment assistance to reduce poverty from the 1960’s until today -

40 years of review  Together with this paper, three previous meta-analyses (Doucouliagos and

Paldam 2006, 2007, 2008) conclude that aid has failed its mission.

In meta analyses, funnel plots are used to detect asymmetries and selection biases. This method gives not only a large overview of the existing literature but investigate also whether the results converge or whether they suffer from publication biases.

The analysis of AEL discovered the existence of a strong reluctancy bias. Researchers prefer to report positive results which represents an impediment to reveal the true nature of aid.

  Even though the reviewed literature finds positive results of aid effectiveness

in 74% of the cases, it is further noticeable that this meta analysis does not find any evidence of the effectiveness of aid.

The combination between reluctancy bias and polishing the outcome can prevent a convergence of research.

If there actually is an effect of development aid it must be small. Therefore more effective ways of employing development assistance

must be found.