Using AFT and FSI Tools to Validate Components of Interest ...

ALCO Partners, LLC

Using AFT and FSI Tools to Validate Components of Interest Rate Risk Models

ALCO Partners, LLC

Agenda

Validation Tests of Prepayment Models • Models applicable to securities

• Models applicable to portfolio loans

Validation Tests of Stochastic Rate Generators using FSI

• Lattice

• Monte Carlo

ALCO Partners, LLC

Regressor and Back-testing

Regressor is applicable to AFT prepayment models:

• Securities

o MBS

o CMOs

MBS data at the CUSIP level require extraction routines from Bloomberg or Intex.

CMO data at the CUSIP level require license with Intex, Chasen or other cash flow model vendor.

Agency CMO data can be extracted from Bloomberg.

ALCO Partners, LLC

Batch “Back-test” in the Regressor Model

• Output created:

o Actual CPR

o AFT forecast of CPR using actual rates

• Output is written to text files with number of files equal to the number of collateral types.

o Example: 300 CUSIPs of 15 and 30 FNMA, FHMLC, and Whole loans might reside in six text files

• All CUSIPs can be run at once in Regressor with the text files created automatically.

• Text files can be opened in EXCEL and standard “goodness of fit” or other statistics applied.


ALCO Partners, LLC

Once a macro is written, process takes less than an hour for the entire list of CUSIPs (We did this for 300 CUSIPs.)

Procedure can be learned in about 15 minutes via phone support, if all required extract files are available.

Regressor allows the aggregation across CUSIPs drawn from same collateral. Performance on sub-portfolios can be assessed.


ALCO Partners, LLC

Why can AFT Claim Regressor Provides a Back-test”?

AFT Prepayment models are estimated using collateral data

• Securities are small samples of the collateral.

• Output in Regressor at CUSIP level is based on both overlapping historical periods and projected periods, depending on the length of history and when collateral model was estimated.

o Over the historical data model period this is an “in-sample” back-test.

o When CPRs are projected beyond historical data period used for building models, the test is an “out-of-sample” back-test.

If the connectivity in the ALM system and AFT models has been validated, then the ALM model is not needed for back-testing AFT prepayment models.

ALCO Partners, LLC

Back testing Sample Output from Regressor

0

10

20

30

40

50

60

70

80

90

Nov-0

2

Jan-0

3

Mar

-03

May

-03

Jul-

03

Sep

-03

Nov-0

3

Jan-0

4

Mar

-04

May

-04

Jul-

04

Sep

-04

CP

R (

%)

Actual Predicted

55265KTS8

R2=73% 80

One of 180 graphs created for an ALCO Partners client

ALCO Partners, LLC

Dynamic Aggregator and Back testing

Dynamic Aggregator is applicable to portfolio loans

• Similar output created as in Regressor

o AFT prepayment models provide the basis for modeling. Discrete prepayment factors can be scaled to improve model fit.

o Historical data used to fit model and model statistics can be created as part of the estimation process.

o In-sample and out-of-sample model testing can be applied with similar statistics.

o If AFT-ALM model link has been validated, then the process can substitute for back test using ALM model.

ALCO Partners, LLC

Findings from Dynamic Aggregator Projects

Dynamic Aggregator has coincidentally “solved” a classic data bottleneck in many banking institutions

• IT departments now regularly archive historical data relevant for testing prepayment models

• They are very efficient at providing raw data.

• They are inefficient at “mapping” data

o Internal resources for “research and analysis” are frequently low on a priority list

o External projects where only raw data is required appear not to face the same internal hurdles because they demand very few IT resources

ALCO Partners, LLC

Findings from Dynamic Aggregator Projects

A part of prepayment model validation projects applicable to portfolio mortgages is obtaining and mapping data. But this process is no longer a large cost.

o Mapping costs have not been large. Key variable is number of systems

o Mapping rules typically remain constant over time, so annual updates become very cost effective

Once the data has been mapped into Dynamic Aggregator, the process for testing and is very cost-effective, particularly when the process is done annually as required by banking regulations

ALCO Partners, LLC

Testing Stochastic Rate Generators

In many banks stochastic rate generators are being utilized to measure value based IRR

• Base case is calibrated to market instruments

• There are known nuances and problems associated with this

• How does a bank “validate” a stochastic rate model?

ALCO Partners, LLC


Approach I

• Obtain model documentation

• Build a separate model independently

• Perform double-blind tests and compare inputs and outputs

Issues with Approach I

• Vendors may be unwilling to provide proprietary code documentation

• Building a separate model can be very expensive, including its validation

• Specialized coding/math expertise required to “build” a stochastic rate generator

ALCO Partners, LLC


Approach II

• Obtain a qualified stochastic rate generation system

• Calibrate to the same set of market instruments

• Compare inputs and outputs for tested samples

FSI: A qualified stochastic rate generation system

ALCO Partners, LLC


FSI is a fully specified two factor lattice & Monte-Carlo system designed as a portfolio management product with validation capabilities

• Provides a basis for comfort or discomfort depending on the model results and how the model results are integrated with portfolio management decisions

• Since it contains both single factor (BK) and two-factor processes, it provides a basis for understanding the sensitivity of the risk measure to the underlying process utilized

• Helps improve calibration procedures and allows a look into the “black box” of stochastic rate generation capability

ALCO Partners, LLC


Test Procedures

• Compared stochastic output from a widely used ALM model with a stochastic generator

o Compared three portfolios in two periods

o MBS & CMO

o Portfolio FRMs

o Callable Bonds

o Greeks are computed at the instrument level

o Market values and market values under rate shocks can be computed

o Market values changes compared over time at instrument level

o Results used to refine and understand calibration process and reasonableness of risk results

ALCO Partners, LLC

Callables: MV Sensitivity

-6%

-4%

-2%

0%

2%

4%

6%

8%

-150 -100 0 100 150 200

Shock (bp)

% D

M

V

FSI1F

FSI2F

ALM

Callables: MV Sensitivity Compared

-6%

-4%

-2%

0%

2%

4%

6%

8%

-150 -100 0 100 150 200

Shock (bp)

% D

M

V

FSI1F

FSI2F

ALM

Callables: MV Sensitivity Compared

12/31 3/30

Market value sensitivity measures for

callable bond portfolio where very similar

in both time periods

ALCO Partners, LLC

Callables: Duration & Convexity

Red line represents parity

0

1

2

3

4

5

6

7

8

0 1 2 3 4 5 6 7 8

Duration (FSI 2Factor)

Du

rati

on

(A

LM

)

Callables: FSI 2F vs. ALM Model

-2

0

2

4

6

8

-2 0 2 4 6 8

Convexity (FSI 2Factor) C

on

ve

xit

y (

AL

M)

Callables: FSI 2F vs. ALM Model

Duration Convexity

Results indicated some non-random differences on

individual securities. Further investigation revealed pattern

associated with “at the money” calls. Cross checks with

market values showed sensitivity to calibration mechanics

ALCO Partners, LLC

Callables: FSI 2 Factor vs. ALM Model: 12/31 to 3/31


-1%

0%

1%

2%

3%

4%

-1% 0% 1% 2% 3%

% D MV (FSI 2Factor)

% D

M

V (

AL

M)

Callables: 3/31 vs. 12/31, FSI vs. ALM Model

How does Market Value Change over Time?

ALCO Partners, LLC

FRMs: MV Sensitivity

-10%

-8%

-6%

-4%

-2%

0%

2%

4%

-150 -100 0 100 150 200

Shock (bp)

%D

M

V

FSI1F

FSI2F

ALM


-12%

-10%

-8%

-6%

-4%

-2%

0%

2%

4%

6%

-150 -100 0 100 150 200

Shock (bp)

%D

M

V

FSI1F

FSI2F

ALM


12/31 3/31

Prepayment models were identical

ALCO Partners, LLC

FRMs: Duration & Convexity


0

1

2

3

4

5

6

0 1 2 3 4 5 6


Du

rati

on

(A

LM

)

FRMs

-5

-4

-3

-2

-1

0

-5 -4 -3 -2 -1 0

Convexity (FSI 2Factor) C

on

vexit

y (

AL

M)

FRMs

Duration Convexity

ALCO Partners, LLC

FRMs: FSI (2 Factor) vs. ALM: 12/31 to 3/31


-3%

-2%

-1%

0%

-3% -2% -1% 0%

% D MV (FSI 2Factor)

% D

MV

(A

LM

) FRMs

FRMs where there are good market value indicators

combined with the OAS calculations lead to similar results

ALCO Partners, LLC

CMOs & MBS: MV Sensitivity

12/31 3/31

-8%

-6%

-4%

-2%

0%

2%

4%

-150 -100 0 100 150 200

Shock (bp)

M

V

FSI1F

FSI2F

ALM

CMOs &MBS: MV Sensitivity

Compared (12/31/2004)

-8%

-6%

-4%

-2%

0%

2%

4%

-150 -100 0 100 150 200

Shock (bp)

M

V

FSI1F

FSI2F

ALM

CMOs &MBS: MV Sensitivity

Compared (12/31/2004)

ALCO Partners, LLC

CMOs & MBS: Duration & Convexity


Duration Convexity

Circled outliers were later identified as input errors in the

ALM model

0

2

4

6

8

0 2 4 6 8


Du

rati

on

(A

LM

)

CMOs & MBS: Duration

-6

-5

-4

-3

-2

-1

0

1

-6 -5 -4 -3 -2 -1 0 1

Convexity (FSI 2Factor)

Co

nvexi

ty (

AL

M)

CMOs & MBS: Convexity

ALCO Partners, LLC

CMOs & MBS: FSI (2 Factor) vs. ALM: 12/31 to 3/31


Four outliers are the same CUSIPs as in the prior graph

-5%

-4%

-3%

-2%

-1%

0%

1%

-5% -4% -3% -2% -1% 0% 1%

%D MV (FSI 2Factor)

%D

MV

(A

LM

)

CMOs& MBS: 3/31 vs. 12/31

ALCO Partners, LLC

Yield Curve Distributions: Single Factor vs. Two Factor Stochastic Processes

Single Factor

Process:

Yield Curve

Distribution

Mean

Min

2.510th

25th

Median

75th

90th

97.5

Max

0%

5%

10%

15%

20%

25%

30%

0 24 48 72 96 120 144 168 192 216 240

Month

1M

LIB

OR

Rate

(%

)

MC Short Rate

Percentile

Single Factor 3-31

Mean

Min

2.5

10th

25th

Median75th90th97.5 Max

-6%

-5%

-4%

-3%

-2%

-1%

0%

1%

2%

3%

0 24 48 72 96 120 144 168 192 216 240

Month

Yie

ld C

urv

e (

%)

MC Yield Curve

Percentil

e

Single Factor 3-31

Mean

Min

2.5

10th

25th

Median

75th

90th

97.5

Max

0%

5%

10%

15%

20%

25%

0 24 48 72 96 120 144 168 192 216 240

Month

5Y

r L

IBO

R R

ate

(%

)

MC Long Rate

Percentile

Single Factor 3-31

ALCO Partners, LLC


Two Factor

Process:

Yield Curve

Distribution

Mean

Min

2.510th

25th

Median

75th

90th

97.5

Max

0%

5%

10%

15%

20%

25%

30%

0 24 48 72 96 120 144 168 192 216 240

Month

1M

LIB

OR

Rate

(%

)

MC Short Rate

Percentil

e

Two Factor 3-31

Mean

Min

2.510th

25th

Median

75th

90th

97.5

Max

0%

5%

10%

15%

20%

25%

0 24 48 72 96 120 144 168 192 216 240

Month

5Y

r L

IBO

R R

ate

(%

)

MC Long Rate

Percentil

e

Two Factor 3-31-2005

Mean

Min

2.5

10th

25th


-6%

-5%

-4%

-3%

-2%

-1%

0%

1%

2%

3%

0 24 48 72 96 120 144 168 192 216 240

Month

Yie

ld C

urv

e (

%)

MC Yield Curve

Percentil

e

Two Factor 3-31

ALCO Partners, LLC


Two Factor

Process:

Yield Curve

Distribution

Mean

Min

2.5

10th

25th


-6%

-5%

-4%

-3%

-2%

-1%

0%

1%

2%

3%

0 24 48 72 96 120 144 168 192 216 240

Month

Yie

ld C

urv

e (

%)

MC Yield Curve

Percentil

e

Two Factor 3-31

Mean

Min

2.5

10th

25th


-6%

-5%

-4%

-3%

-2%

-1%

0%

1%

2%

3%

0 24 48 72 96 120 144 168 192 216 240

Month

Yie

ld C

urv

e (

%)

MC Yield Curve

Percentil

e

Single Factor 3-31

Single Factor

Process:

Yield Curve

Distribution

ALCO Partners, LLC

FRMs: MV Sensitivity: Single vs. 2 Factor

-10%

-8%

-6%

-4%

-2%

0%

2%

4%

-150 -100 0 100 150 200

Shock (bp)

%D

M

V

FSI1F

FSI2F

ALM


-12%

-10%

-8%

-6%

-4%

-2%

0%

2%

4%

6%

-150 -100 0 100 150 200

Shock (bp)

%D

M

V

FSI1F

FSI2F

ALM


12/31 3/31

Impact of process is limited in this

case even though distribution of yield

curves is noticeable

ALCO Partners, LLC

Observations from Test

Stochastic models will frequently produce different measures of risk at the instrument level for instruments with embedded options

• We believe – but it is difficult to confirm without access to source code – that this is largely due to differences in calibration procedures as well as differences in modeling methods.

• With larger portfolios that include instruments with a wide variety of option structures, the differences in results will tend to cancel each other out if the overall levels of implied volatility used in the calibrations are similar.

ALCO Partners, LLC


Validation tests of stochastic models using a qualified stochastic model, such as FSI, can:

• Identify modeling errors, because large differences in results invite further investigation

• Enhance understanding of the “black box” of ALM model calibration procedures, when calibrations aren’t “all that close”

• Provide guidance for the product characteristics that are stochastic modeling technique sensitive

Validation tests of stochastic models lead to the following warning:

Beware of relying on a single stochastic model when measuring risk at the instrument level for instruments with embedded complex options

ALCO Partners, LLC


Tests of stochastic models can identify modeling errors, because large differences invite further investigation

Word to the warning about using stochastic models for risk management involving individual transactions

Using AFT and FSI Tools to Validate Components of Interest ...

Documents

Transcript of Using AFT and FSI Tools to Validate Components of Interest ...