EVALUATION OF CREDIT GUARANTEE SCHEMES (CGS): SOME CONSIDERATIONSAllan RidingMarch 2014

1. Review some of the ways in which credit guarantee schemes (CGS) have been evaluated.

2. Provoke discussion about ways in which credit guarantee schemes (CGS) should be evaluated.

3. Solicit constructive feedback on most recent attempts to evaluate CGS.

Goals for today

• Typical rationales for CGS– According to economic theory intervention is not

warranted without an imperfection, and, when there is an imperfection, the intervention should address the imperfection.

– Information asymmetries between borrower and lender are often cited as an imperfection that provides a rationale for CGS.

– Likewise, scale may also be viewed as an imperfection that supports CGS-type interventions if “good” firms are denied financing.

• Lenders are unable to distinguish good borrowers from bad borrowers because small scale loan balances do not warrant costly due diligence.

1.0 Rationales for Credit Guarantee Schemes (CGS)

• Information asymmetries are more pronounced for…– Smaller firms? (Many)– Innovative firms? (Freel 2007)

• More informationally opaque due to unknown technologies; collateral issues

– Growth-oriented firms? More informationally opaque due to chaos, uncertainty of growth (Ennew and Binks 1996)

– Women-owned firms? (CFIB, 1994; Fay and Williams, 1992)

– Etc.

1.0 On Interventions in the Credit Market

• Development banks– Often with mandate to act as complementary

lenders to those in private sector• Mixed outcomes

– Yeyati et al IIDB, 2004– Rudolph, World Bank 2009

• Credit Guarantee Schemes– The perception that SMEs require interventions

such as CGSs is pervasive:• Green (2003) reports more than 2,250 CGSs across

almost 100 countries. • OECD (2013) the instrument of choice in insulating

SMEs from the financial crisis

2.0 Interventions

• Given the prevalence of CGS, evaluations of cost/benefit balance are necessary, especially when governments are guarantors– Many governments face high debt and are operating

at deficits so value of initiatives must be understood– Financial crisis has prompted commercial lenders to

be more conservative. • Provides incentive for increased demand for CGS as

business borrowers become increasingly likely to resort to CGS

– Basel III allows lenders to use risk rating of guarantor as replacement for risk rating of borrower

• Provides incentive for increased demand for CGS from commercial lenders

3.0 Evaluating Interventions

• Starting point– Need to define precisely what are the goals

of the intervention– Facilitate additional access to financial capital among

firms likely to generate-save for their lack of access to credit-economic welfare?

– UK: to educate lenders with respect to SME lending;– Canada: to facilitate SME growth; – Japan: to rescue firms in disaster situations.– Other?

– Essential that evaluation measures progress towards the stated goals

3.0 Evaluating Interventions

• How to conduct evaluation?– Canada and UK experience (mostly on

Canada)

• “Evaluation” has many definitions– Useful summary (source of next slides):

David Storey (1998) Evaluating the Impact of Public Policies to Support Small Businesses in Developed Economies, University of Warwick Working Paper

3.0 Evaluating Interventions

• UK: – Collected questionnaire data from

borrowers who had received loans under the terms of the (then) DTI Loan Guarantee Scheme (LGS)

– Conducted interviews with primary owners of CGS borrower firms AND with matching lender loan account managers

– Analysis of internal data on frequency of defaults among LGS borrower firms (Cowling et al.)

3.1 Evaluating CGS, UK example

• Survey data issues:– Key respondent problem

• Was the responder to the survey the individual within the firm with best knowledge of issues

– Selection bias• Sampling frame may affect findings

– Non-response bias• Low response rates can compromise results,

especially if non-response was systematic

– Recollected data• Event-related data (e.g., pertaining to date of loan

disbursement) get lost in mists of time.

3.1 Evaluating CGS: On Survey Data

3.1 Six Steps

• Six categories of evaluations frequently observed with respect to government programs

Main concern is to document expenditure, make clear that expenditure is compatible with program goals.

An accounting and legal function, but no economic role.

Example: BDC

(Canada) study

3.1 Six Steps

• Six categories of evaluations frequently observed with respect to government programs

Compares the outcomes of firms assisted by the policy with those of a control group of firms that had not been assisted

Matching based on factors thought to influence sought-for outcomes

Can be difficult to

identify basis for matching

• Canada: – Collected questionnaire data from borrowers

who had received loans under the terms of the (then) Industry Canada CSBF (Canada Small Business Financing program)

– Conducted interviews with primary owners of CGS borrower firms AND with matching lender loan account managers

– Used external data on frequency of defaults among CSBF borrower firms and “comparable” non-CSBF borrowers

3.2 Key resource: Data

• Canada Key Resource– Surveys on Financing of Small and Medium

Enterprises • Initial Survey conducted in 2000 (>10,000

responses)– Follow-up survey in 2001 (~4,000 responses)

• 2004 (>12,000 responses)• 2007 (>12,000 responses)• 2010 (>12,000 responses)• 2013 (only for firms with >10 employees)

– All are stratified and weighted as to size, region, sector and age

3.2 Key resource: Data

• Canada Key Resource– Surveys on Financing of Small and Medium

Enterprises• Initial Survey conducted in 2000 (>10,000

responses)– Follow-up survey in 2001 (~4,000 responses)

• 2004 (>12,000 responses)• 2007 (>12,000 responses)• 2010 (>12,000 responses)• 2013 (only for firms with >10 employees)

– All are stratified and weighted as to size, region, sector and age

3.2 Key resource: Data

Linked to tax data for t=-1 to t=4 using unique business

identifier

• Key task of evaluation is assessing the counter-factual– What would have happened to

businesses in the absence of policy (e.g., CGS)?

• the so-called ‘counter-factual’.

– The effect of policy is defined as the difference between what actually happened and what would have happened in the absence of policy.

3.3 Dealing with the counter-factual

Not directly

observable

• Split data into two groups: (1) firms that had applied for non-guaranteed loans; (2) firms that had applied for and received guaranteed loans.

• Used sample (1) to ‘reverse engineer’ credit scoring algorithm.

• Applied credit scoring model to sample (2) finding that approximately 75% would have been rejected by credit scoring model.

• Compares the outcomes of firms assisted by the policy with outcomes for firms which had not been assisted– inference is that differences in outcomes can

be attributed to the impact of the policy– Problem is that control group firms may not

be directly comparable• For example, if control group simply comprises

other firms in the economy, these may differ systemically from firms that sought loans (arguably less growth oriented) (Recall BDC study)

3.4 Evaluation of Benefits

• Used 2004 Survey of Financing of SMEs.

• Used statistical techniques that seek to explicitly take account of sample selection bias (based on Heckman, 1977).

• Developed statistical model of the selection process and then, taking account of the factors that drive the selection process, estimated a second equation to explain the outcomes.

• Compared job creation by CSBF-assisted SMEs with that of other benchmark samples

• Found that CSBF added significantly to job and value creation

• Essentially, that the CSBF more than pays for itself.

• Two components– Costs of program administration

• Depends on level of government involvement– e.g., in Canada government is passive: keeps

records; compensates commercial lenders to honor defaults; receives fees on loans from lenders

– e.g., in some jurisdictions, government approves loan guarantees and second tier re-insurance is the norm

• Estimated from internal data.

– Costs of honoring defaults• Timing matters!!

3.5 Evaluation of costs.

• Costs of honoring defaults• Timing matters!!

4.0 Evaluation of costs.

Figure 1: Chronology of Claim Receipts

Early claims (a) are costly and (b) may reflect lenders’ poor loan

adjudication practices.Later claims

may not matter much; also

borrowers had paid fees over life of loans

Example: Costs of honoring defaults

• Costs of honoring defaults– Difficult to attribute costs to particular cohorts

• Tendency to use logistic/probit regression models to attribute costs to firm attributes/program parameters/cohort.

• BUT…– Right-censoring of data (need to wait years before

knowing whether or not a given loan will default).– Impacts of attributes etc. may vary over time

» example: age of borrower firm at time of disbursement may be a key factor immediately following disbursement (if young firms are riskier than old firms) but arguably less of a factor ten years later

3.5 Evaluation of costs.

Logistic Regression Example

• Costs of honoring defaults• Event history methods (survival analysis)

– For example, proportional hazards model (aka Cox Regression)

– Probability of default =a) A baseline pattern that depends on time since

disbursementMODIFIED BYb) A function that models how attributes, program

parameters etc. shift the baseline pattern – Used widely in medicine (how does dosage of medicine

influence survival time?), quality control (how does attribute of production item influence useful life?).

3.5 Evaluation of costs.

Example: Proportion Hazards Model Results

• In evaluating CGS, need (at least?) three dimensions– Estimation of

incrementality/additionality– Estimation of benefits (and factors that

influence benefits)– Estimation of costs

• Administrative data• Default data

4.0 Summary

THANK YOUComments, Questions, Discussion