Model Risk Management...Membership represents over 150,000 risk management practitioners and...

Model Risk Management

Nav Vaidhyanathan

Director, Head of Model Risk Management

Wintrust Financial Corporation

Global Association of Risk Professionals

August 2014

2

The views expressed in the following material are the author’s and do not

necessarily represent the views of the Global Association of Risk

Professionals (GARP), its Membership or its Management.

These materials reflect the views and opinions of the speaker and are not

intended to reflect the views, policies or practices of Wintrust Financial or its

affiliated companies. They are for informational purposes only.

3 | © 2014 Global Association of Risk Professionals. All rights reserved.

Agenda

Model Risk Management is a broad, evolving, and an expanding topic. This presentation

will cover only select components at a high level.

Inherent risk (model tier) and residual risk of model

A framework for model monitoring

Model risk aggregation framework

Model validation for stress test (e.g., DFAST) models


Model Risk Axioms*

The following can potentially be considered as self-evident truths for Model Risk. Given

everything else equal,

All models have limitations

Model Risk is related to model limitations

Model Risk increases with more models (this might fall in the grey area)

− E.g., two Tier 1 models potentially pose higher risk than 1 Tier 1 model (everything else being the

same, i.e., similar limitations, similar validation status, similar materiality, similar findings, etc.)

Model Risk for Tier 1 model > Model Risk for Tier 2 model

Model Risk for validated model ≤ Model Risk for unvalidated model (for the same model)

More findings ⇒ more Model Risk (same category and level of findings)

Certain types of findings will have higher Model Risk than others

Models with deteriorating performance pose higher Model Risk

Model Risk is exhibited by the variance between model output and observed actual (this is just one of the ways that Model Risk exhibits itself)

* may not be axioms in the purest sense of the term


Inherent Risk and Residual Risk of Model

Residual Risk Rating could be

driven by:

• Model validation findings:

• Category

• Level

• # of Open Findings

• Model performance /

monitoring

Model owners can reduce residual

risk – by improving models,

improving controls, and/or

remediating findings

Res

idu

al

Ris

k R

atin

g

High

Moderate

Low

Low (Tier 3) Moderate High (Tier 1)

Inherent Risk Rating (Model Tier)

Inherent Risk Rating could be driven by:

• Materiality

• Type of model

• Used for regulatory / financial reporting or critical business

decisions

• Feeds into another Tier 1 model?

Model owners cannot (usually) reduce inherent risk


Residual Risk Rating Framework

Residual Risk Rating (High,

Moderate, Low) is based on

the Residual Risk Score range

E.g., RRS > 200 ⇒ High, RRS ≤

100 ⇒ Low

If the model is deemed unfit for

use, the RRR is High

The Residual Risk Score

Range to Residual Risk

Ratings mapping needs to be

monitored periodically

The ranges can be determined

by typical observations from

model validation and model

monitoring exercises

If a model is unvalidated, the

residual risk rating is HIGH


Model Monitoring

Model monitoring should evaluate whether changes (including anticipated) in

products, exposures, activities, clients, or market conditions necessitate

adjustment, redevelopment, or replacement of the model

Ongoing activities include – review of input data and assumptions, outcome

analysis, model stability checks, review of overlays and expert judgments,

benchmarking, etc.

Key considerations

− Model performance monitoring

› Pros and cons of model performance measures

› Model type

› Model use

› Type of measure

− Periodicity of monitoring

− Thresholds for model recalibration or rebuild

− From performance measure to model performance rating


A Bottom Up Framework For Model Risk Aggregation

Impact of Failure (e.g., on Capital)

Tier 1 Tier 2 Tier 3

Validated … …

Yes No

High Residual

Risk

Moderate

Residual Risk Low Residual

Risk

As part of an RCSA

process, the impact of

the model failure is

estimated by model

owners

The end nodes in the

decision tree can be

used for aggregating

model risk

For DFAST projection models, for Severely Adverse Scenario, the impact on capital can be assessed

using backtesting during Great Recession period for all models (adjust the variance by severity of the

scenario compared to the Great Recession)

By looking at the distribution, a conservative starting point for capital can be chosen for each model

Tier


Some Considerations in Validation of Stress Test Models

Are estimates conditioned on macroeconomic scenarios?

Do the models produce outcomes that show separation and order between the baseline,

adverse, and severely adverse scenarios?

Are the scenario and assumptions consistent across all models?

Are exposures modeled using appropriate levels of segmentation?

Qualitative elements of the model

Relevance of third party vendor models to company specific characteristics, especially when

using proxy portfolio

Sensitivity analysis around assumptions

Use of weak models vs. expert judgment

For models not validated, compensating controls and conservativeness

Documentation

Capital buffer for model risk

C r e a t i n g a c u l t u r e o f

r i s k a w a r e n e s s ®

Global Association of

Risk Professionals

111 Town Square Place

14th Floor

Jersey City, New Jersey 07310

U.S.A.

+ 1 201.719.7210

2nd Floor

Bengal Wing

9A Devonshire Square

London, EC2M 4YN

U.K.

+ 44 (0) 20 7397 9630

www.garp.org

About GARP | The Global Association of Risk Professionals (GARP) is a not-for-profit global membership organization dedicated to preparing professionals and organizations to make

better informed risk decisions. Membership represents over 150,000 risk management practitioners and researchers from banks, investment management firms, government agencies,

academic institutions, and corporations from more than 195 countries and territories. GARP administers the Financial Risk Manager (FRM®) and the Energy Risk Professional (ERP®)

Exams; certifications recognized by risk professionals worldwide. GARP also helps advance the role of risk management via comprehensive professional education and training for

professionals of all levels. www.garp.org.


Model Risk Management for Non Banks

Michelle McCarthy

Managing Director, Risk Management

Nuveen Investments

Global Association of Risk Professionals

August 2014

2

The views expressed in the following material are the author’s and

do not necessarily represent the views of the Global Association of

Risk Professionals (GARP), its Membership or its Management.

Disclaimers

These materials reflect the views and opinions of the speaker and are

not intended to reflect the views, policies or practices of Nuveen

Investments or its affiliated companies. They are not intended to

provide investment advice and are for informational purposes only.


Contents

Differences between models and their uses at banks vs. the buy side

What should be in scope for model risk management activity in the asset

management business?

Potential tiering of asset management models, and effect on model risk

management activities


Most literature on model risk management has concerned banks

OCC 2000-16

OCC 2011-12, FRB SR 11-7

The case of AXA Rosenberg brought the conversation to the buy side, without

much guidance on how bank practices can and should differ from asset managers’

practices


Different uses of models in asset managers vs. banks

Asset managers are not computing regulatory capital

Bank-owned asset managers may need to provide operational risk capital

calculations, but this is not in service of the asset management business

We rarely use valuation models as the sole source of valuation; we frequently

use external sources

Many asset management models are an input to a decision, but not the sole

driver of that decision


Potential scope of model management activity for asset managers

6

Includes models that are used for investing or hedging

Drive investing activity to some extent

Value key assets

Does not include calculators; model must provide estimates and involve

uncertainty to require model risk management

Good change controls always an important operational control, but calibrations and

validations should be reserved for models that estimate uncertainty

Would not include compliance systems

Would not include price-to-yield calculations

Would not include simple screens for fundamental stock characteristics

Would not include pure index replication models without meaningful uncertainty

Does not include analyses that help develop an investment thesis—only the

tools used to execute the thesis, or to value assets once purchased


Tiering for materiality

A potential tiering of asset management models:

Tier 1 Tier 2 Tier 3 Tier 4Models that:

• have high potential direct

impact on company results

• are broadly applied across the

entire company

• are understood and

maintained by a relatively

narrow range of experts at the

company

• Result in instant electronic

trading with little or no ability

to intervene, and/or affecting

a large portion of the portfolio

• are tightly coupled to trading but permit

human intervention, and/or affect a

small portion of the business

• result in trading with little other input

than direction from a model (quant

strategies)

• provide hedge ratios for using

derivatives, or to balance proportions

of fixed income instruments, in

portfolios

• value smaller, less significant balance

sheet items than those captured in Tier

1

• apply a screen to a universe of

stocks, using some measure of

uncertainty, such as a measurement

of the dispersion of some

fundamental characteristic (low price-

to-earnings ratios, for example)

• compare characteristics of securities

to assist the portfolio manager in

finding securities that he or she

deems desirable

• may support a certain

amount of decision

making, but with many

other inputs to these

decisions outside the

model.

• are outside the

company’s areas of key

business risk

• are understood by a

broad range of business

people

Such as:

• Statistical arbitrage model

• High frequency trading model

• A model that is rigorously

adhered to, is key to a broad

set of strategies and is

disclosed as a critical feature

of the investment process

• Significant valuation models

where third party sources are

unavailable

• A strategy that strongly adheres to

rebalancing if Barra or Yieldbook

measures exceed a threshold

• An economic indicator model that is

the basis of an investment strategy

• High frequency trading models

• Models used for delta hedging or asset

allocation rebalancing

• Minor valuation models where third

party sources are unavailable

• Using Barra or Yieldbook to see

where a portfolio is overweight, in

order to select trades that will balance

the portfolio better

• Economic indicator screens to

provide general investment signals

• Index replication models with tilts

• Risk oversight models

• Valuation models where third party

valuations are available

• Measures of portfolios

that are not used to

drive action

• Index replication

models that approach

100% replication


Slotting typical asset management models into tiers

Tier I Tier II Tier III Tier IV

Investment Models

Fundamental

Business Models x x

Stock Selection Screen x x

Quantitative

Stock Selection x x

Portfolio Construction x x

Index Replication with Tilts x

Index Replication x

Statistical Arbitrage x x

High Frequency Trading x x

Leverage x

Hedging x

Economic Models

Growth x x

Inflation x x

Industry x x

Credit x x

Liquidity x x x

Risk Models

Capital Loss x

Liquidity x x

Ex Ante Tracking Error x x

Ex Post Tracking Error x

Duration, Beta, Factor Exposure Limits x x

Valuation Models

Third Party Pricing Services x x

Pricing Model - Third Party Developed x x x

Pricing Model - In-House Developed x x x


What steps may make sense for model management of the

different tiers?

Inventory

For material models (Tiers 1 to 2)

Initial independent validation

─ Where independence may or may not require the different reporting lines set forth for

banks: effective challenge

Documentation (model theory, assumptions, limitations, code, user instructions)

Change validation

Periodic formal, documented calibration

Periodic re-validation given results of calibration

Model management for Tier 3 models could be lighter in terms of frequency,

intensity, formality of independence

The challenge of third party vendor models

C r e a t i n g a c u l t u r e o f

r i s k a w a r e n e s s ®

Global Association of

Risk Professionals

111 Town Square Place

14th Floor

Jersey City, New Jersey 07310

U.S.A.

+ 1 201.719.7210

2nd Floor

Bengal Wing

9A Devonshire Square

London, EC2M 4YN

U.K.

+ 44 (0) 20 7397 9630

www.garp.org

About GARP | The Global Association of Risk Professionals (GARP) is a not-for-profit global membership organization dedicated to preparing professionals and organizations to make

better informed risk decisions. Membership represents over 150,000 risk management practitioners and researchers from banks, investment management firms, government agencies,

academic institutions, and corporations from more than 195 countries and territories. GARP administers the Financial Risk Manager (FRM®) and the Energy Risk Professional (ERP®)

Exams; certifications recognized by risk professionals worldwide. GARP also helps advance the role of risk management via comprehensive professional education and training for

professionals of all levels. www.garp.org.


Model Risk Management...Membership represents over 150,000 risk management practitioners and...

Documents

Transcript of Model Risk Management...Membership represents over 150,000 risk management practitioners and...