Prediction errors in credit loss forecasting models based on ...€¦ · Case Study: UK Credit Card...

© 2013 Experian Ltd. All rights reserved.

No part of this copyrighted work may be reproduced, modified, or distributed in any form or manner without the prior written permission of Experian.

Experian Public.

Prediction errors in credit loss forecasting models based on macroeconomic data Eric McVittie Experian Decision Analytics

Credit Scoring & Credit Control XIII

August 2013

University of Edinburgh Business School

2 © 2012 Experian Information Solutions, Inc. All rights reserved.

Experian Public.

©2013 Experian Ltd. All rights reserved.

Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.

Experian Public

Prediction errors in macroeconomic credit loss forecasting models

Abstract

Loss forecasting models including aggregated economic variables

may generate substantial and persistent forecast biases when fitted

on limited historical data. Recent evidence from the UK, for example,

suggests a bias towards over-prediction of credit losses from models

estimated on short historical data periods including the 2008-2009

recession, when such models are applied to more recent economic

conditions. This paper considers possible explanations for this

pattern, and the potential for alternative approaches that are less

prone to forecast errors of this type.


Experian Public.



Experian Public


Outline

Evidence of recent deterioration in forecast accuracy for portfolio loss / pd models including macroeconomic variables:

► Market, bureau & client data

► Variety of modelling methodologies

Here concentrate on:

► Market card delinquency data

► Unemployment measures

Look at some candidate explanations.

Illustrate some issues around use of time series methods and aggregated economic variables in loss models.

Contents:

► An evaluation framework borrowed from Hendry (1995) Dynamic Econometrics, OUP

► Some evidence – lots of pictures

► Many questions

► Very few firm answers

► Some tentative suggestions

20

06

Q1

20

06

Q3

20

07

Q1

20

07

Q3

20

08

Q1

20

08

Q3

20

09

Q1

20

09

Q3

20

10

Q1

20

10

Q3

20

11

Q1

20

11

Q3

Actual

Predicted


Experian Public.



Experian Public


Model Building & Forecasting

Robust forecasts require models that replicate ‘key’ features of the (unknown) DGP

Information is inevitably lost in moving from the DGP via observed data to empirical models

Models provide better representations of the DGP the less information they lose and the more they retain

Model design criteria should select models which minimize information loss

Behaviour Measure-

ment

Data Generation Process (DGP)

Observed

Data

Empirical

Models Predictions

/ Forecasts

Not known or

directly observed


Experian Public.



Experian Public


Model Building & Forecasting: Information Losses

Measurement System Economic

activity

①

Data Transformations & Aggregation ②

Observed Data

Parameters of

Interest Data Partition ③

Marginalization ④

Lag Truncation ⑤

Functional Form ⑥

Derived Empirical Models

Forecasts

Forecast

Assumptions

Information

losses


Experian Public.



Experian Public


Model Building & Forecasting: Design Criteria

Domain Criteria / Objectives AlternativesMeasurement Data Accuracy Volatile, imprecise or Inaccurate data

Past / in-

sample

Homoskedastic, innovation errors Residual autocorrelation;

heteroskedasticity

Present Weakly exogenous conditional

variables for parameters of interest

Invalid conditioning

Future / Out of

sample

Constant, invariant parameters of

interest

Parameter non-constancy; predictive

failure

Theory consistent / structural

relationships

Implausible coefficients

Rival models Encompasses rival models / Forecasts

dominate rival models

Relative poor fit / significant

additional variables / alternative or

additional variables valuable in

forecasting / relatively poor forecasts

Design criteria require minimum information losses at all stages of reduction

relative to potential rival models.

Generally highly demanding even when data availability is good.


Experian Public.



Experian Public


Forecast errors: Hypotheses

(1) The models were never ‘right’

► Invalid / excessive marginalization

Variable Selection:

► Wrong drivers ► Missing variables?

● Economic drivers ● Quality of stock, lender

behaviour ● Interactions

Functional misspecification

(2) The models were ‘right’ but the world has changed

► Model parameters not constant / structural

Fundamental ‘regime shift’ resulting in permanent changes to relationships between economic factors and credit losses

Continuously evolving relationships between economic factors and credit losses, resulting in gradual deterioration of calibration of models estimated on historical data.

Some variants of (2) are observationally equivalent to some variants of (1) – e.g. apparent

regime shifts (in terms of observed variables) may be due to changes in unobserved

(missing) variables.


Experian Public.


Experian Public CSCC XIII – August 2013


Thinking about economic risk: Variable Selection

Search space is large even if restricted to macroeconomic (aggregate) variables:

► Many potentially-relevant variables ► Functional forms? ► Lag structures?

What about disaggregated data (e.g. for geographical areas)?

Finding good models hampered by poor data availability:

► Limited historical time series for consistent credit loss data

► Missing variables related to credit quality, lending practices, etc.

Weak tests for valid reductions.

High risk of building models based on spurious correlations.

Grid Search / Data Mining

Theory, priors & ‘expert’ judgement

Statistical Data Reduction Methods

Macroeconomic factors proxy

more direct influences on

borrowers – proxy quality varies.

Aggregated time series approaches lead to large information losses especially given

short historical time series and force additional losses due to excess marginalization

and lag truncation

Data limitations weaken tests to enforce design criteria even if attempts are made to

apply them rigorously.


Experian Public.



Experian Public

Prediction errors in macroeconomic credit loss forecasting models Case Study: UK Credit Card delinquency & aggregate unemployment

Look at models of UK market delinquency rate measure (derived from BoE data) over period 1992-2011

Focus on unemployment-related variables :

► In naive models including only unemployment measures ► Conditioned on other candidate drivers of card delinquency ► How do models perform in most recent time periods? ► Stability of coefficient estimates over sub-periods of the full sample

Advantages of modelling ‘market’ data:

► Longer (apparently) consistent history ► Better matching of borrower population to macroeconomic aggregates? (although

even here population for macro variables may not match borrower population) ► I can show you the results!

Disadvantages:

► More missing variable issues – not possible to condition on quality of stock or allow for changes in lender behaviour

► Rules out anything other than simple time series modelling approaches

Analysis and fitted models are purely illustrative– useful case study to consider some of the issues involved linking credit outcomes to economic variables as basis for loss forecasting.


Experian Public.



Experian Public


-2

-1

0

1

2

3

1992 1994 1996 1998 2000 2002 2004 2006 2008 2010

Credit Card Delinquency Rate

ILO Unemployment Rate

Claimant Unemployment Rate

Normalized series - levels

Card

Delinquency

Rate


Covariance -0.375

Correlation -0.242

t-Statistic -2.298

Probability 0.024

Claimant Rate

Covariance -0.808

Correlation -0.400

t-Statistic -4.020

Probability 0.000

Covariance Analysis


Experian Public.



Experian Public

-1.5

-1

-0.5

0

0.5

1

1997Q1 2000Q1 2003Q1 2006Q1 2009Q1

5 year

7 year

10 year

15 year

Mo

vin

g C

orr

elat

ion

Co

effi

cien

t

End of rolling sample period

Moving correlations between ILO rate & card delinquency rate

Card

Delinquency

Rate


Covariance -0.375

Correlation -0.242

t-Statistic -2.298

Probability 0.024

Claimant Rate

Covariance -0.808

Correlation -0.400

t-Statistic -4.020

Probability 0.000

Full Sample



Experian Public.



Experian Public

Normalized series – delinquency rate levels, quarterly

changes in unemployment rates (smoothed)

Covariance Analysis

-3

-2

-1

0

1

2

3

4

1992 1994 1996 1998 2000 2002 2004 2006 2008 2010

Credit Card Delinquency Rate

Quarterly Change in ILO Rate (Smoothed)

Quarterly Change in Claimant Rate (Smoothed)

Card

Delinquency

Rate

Quarterly Change: ILO Rate

Covariance 0.136

Correlation 0.755

t-Statistic 10.545

Probability 0.000

Quarterlly Change: Claimant Rate

Covariance 0.145

Correlation 0.676

t-Statistic 8.417

Probability 0.000



Experian Public.



Experian Public


Credit Card Delinquency & Unemployment

Full Sample Card

Delinquency

Rate

Quarterly Change: ILO Rate

Covariance 0.136

Correlation 0.755

t-Statistic 10.545

Probability 0.000

Quarterlly Change: Claimant Rate

Covariance 0.145

Correlation 0.676

t-Statistic 8.417

Probability 0.000

-0.2

0

0.2

0.4

0.6

0.8

1

1997Q1 2000Q1 2003Q1 2006Q1 2009Q1

5 year

7 year

10 year

15 year

Mo

vin

g C

orr

elat

ion

Co

effi

cien

t


Moving correlations between quarterly changes in ILO rate & card delinquency rate


Experian Public.



Experian Public

Extended model of credit card delinquency rate, includes:

► Quarterly change in ILO rate

► Annual growth of household real disposable income

► Quarterly change in interest rates

What can we learn from residuals?

E.g. Tendency to under-predict delinquency rate in post-recession periods (after 1992 and 2009)?

-1.2

-0.8

-0.4

0.0

0.4

0.8

1.2

1

2

3

4

5

6

1992 1994 1996 1998 2000 2002 2004 2006 2008 2010

Residual Actual Fitted

Strong autocorrelation in model residuals – could be addressed using

alternative specification in full sample but may reflect more

fundamental specification issues (wrong or missing variables).



Experian Public.



Experian Public

2.0

2.5

3.0

3.5

4.0

4.5

92 94 96 98 00 02 04 06 08 10

Recursive C(1) Estimates

± 2 S.E.

-1

0

1

2

3

4

5

6

92 94 96 98 00 02 04 06 08 10


± 2 S.E.

-.2

-.1

.0

.1

.2

.3

92 94 96 98 00 02 04 06 08 10


± 2 S.E.

-0.4

0.0

0.4

0.8

1.2

92 94 96 98 00 02 04 06 08 10


± 2 S.E.

Constant Change in ILO Rate

Income Growth Interest rate

Recursive regression coefficients (progressively extending estimation sample to include later dates)



Experian Public.



Experian Public

Moving regression coefficients between changes in ILO rate & card delinquency rate

Coefficient 2.737

Std. Error 0.359

t-Statistic 7.621

Prob. 0.000

-1

-0.5

0

0.5

1

1.5

2

2.5

3

1997Q1 2000Q1 2003Q1 2006Q1 2009Q1

5 year

7 year

10 year

Mo

vin

g R

egre

ssio

n C

oef

fici

ent


Full Sample



Experian Public.



Experian Public

Predictive accuracy for model estimated on sub-sample: 2003Q1 – 2009Q4

-1

0

1

2

3

4

5

6

2004 2005 2006 2007 2008 2009 2010 2011

Actual Predicted Residual



Experian Public.



Experian Public

Predictive accuracy for model estimated on alternative 10 year sub-samples

-2

-1.5

-1

-0.5

0

0.5

1

all 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011

2009

2010

2011

Fore

cast

Erro

r fo

r C

CD

Del

inq

Rat

e

Model estimated on 10 year sample ending:



Experian Public.



Experian Public

Some conclusions on aggregate analysis:

► Estimated relationships between credit outcomes and unemployment (and other factors) vary greatly depending on the estimation sample

► Even full-sample models tend to over-estimate losses in recent history (whether with or without other economic conditioning variables)

► Autocorrelation issues in full sample model could be resolved using more ‘sophisticated’ estimation procedures in the full data sample, but may indicate more basic problems with model specification.

► Alternative estimation procedures are generally not viable in samples available in practice.

► Over-estimation problem worse for models fitted on more recent history ► Changes in borrower population and/or lender behaviour probably play some role

● Not possible to control for these in aggregated models estimated over longer history

Similar results in other applications using macroeconomic data:

► Loss models fitted on aggregated or account-level lender data ► PD models fitted on bureau data ► Argues against explanations based on model specification ► Problem is (partially) mitigated in models using disaggregated economic data – e.g.

unemployment by local area or by age



Experian Public.



Experian Public

Good forecasting models replicate key features of ‘data generation process’ with minimal information losses

Criteria for effective models are challenging even with abundant data

Aggregate loss forecasting modelling approaches emphasizing time series movements in economic data suffer from:

► Excess / invalid aggregation of economic (and possibly credit) data

► Excess / invalid marginalization:

● Difficult to effectively control for changes in borrower characteristics, lender & borrower behaviour

● Limited ‘degrees of freedom’ forces concentration on small set of economic metrics that dominate time series movements in credit losses

► Limited scope for testing against design criteria – weak tests for innovation errors (including stationarity), exogeneity, parameter constancy, etc.

Disaggregated approaches (for both credit and economic data) reduce information losses resulting in more stable, robust forecast models.

Prediction errors in macroeconomic credit loss forecasting models Conclusions


Experian Public.


Experian Public

Prediction errors in credit loss forecasting models based on ...€¦ · Case Study: UK Credit Card...

Documents

Transcript of Prediction errors in credit loss forecasting models based on ...€¦ · Case Study: UK Credit Card...