MODEL RISK MANAGEMENT: BEST PRACTICES FOR MATLAB MODELS

Information, data and drawings embodied in this presentation are strictly confidential and are supplied on the understanding that they will be held confidentially and not disclosed to third parties without the prior written consent of QuantUniversity LLC.

Sri Krishnamurthy, CFAFounder and CEOwww.QuantUniversity.comAugust 14th 2016

AG

END

A

About QuantUniversity

Stress Testing and Model Risk: A Brief Introduction

The Model Risk Analytics Solution

1

2

3

4

Agenda

Case Study5

Challenges to implementing Model Risk Management

ANALYTICS, RISK MANAGEMENT & STRESS TESTING FOR FINANCIAL INSTIT UTIONS- ADVISORY SERVICES- PLATFORM TO STRESS TESTING, ANALYTICS AND RISK ASSESSMENT- ARCHITECTURE REVIEW, TRAINING AND AUDITS

INTR

OD

UC

TION

• Founder of QuantUniversity. • Advisory and Consultancy for Financial Analytics

and Model Risk• Prior Experience at MathWorks, Citigroup and

Endeca and 25+ financial services and energy customers.

• Charted Financial Analyst and Certified Analytics Professional

• Teaches Analytics in the Babson College MBA program and at Northeastern University, Boston

Sri KrishnamurthyFounder and CEO

4

STRESS TESTING AND ANALYTICS– A BRIEF INTRODUCTION

STRESS TESTS IN

THE N

EWS

“Of the 33 banks the Fed tested—which included the largest U.S. banks, like Bank of America, Citigroup, and Wells Fargo.. all were deemed strong enough to weather a severe economic meltdown without any help from the government.”

MO

DEL

VA

LIDA

TION

“Model risk is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports. “ [1]

“Model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses. ” [1]

Ref:[1] . Supervisory Letter SR 11-7 on guidance on Model Risk

Model Risk and Validation

STR

ESST

ESTING

AN

DM

OD

ELR

ISKM

AN

AG

EMEN

T

Regulatory efforts

EIOPA: Europe-wide stress test for the insurance sector

What’s driving Stress Testing and Model Risk Management efforts?

https://eiopa.europa.eu/Publications/Surveys/eiopa-14-215_stress_test_2014_specifications.pdf

STR

ESST

EST&

SC

ENA

RIO

TEST

Stress Tests and Scenario Tests

Figure courtesy: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf

DEFIN

ITION

SDefinitions

1. Scenarios :

“A scenario is a possible future environment, either at a point in time or over a period of time.”

“Considers the impact of a combination of events“

2. Sensitivity Analysis:

“A sensitivity is the effect of a set of alternative assumptions regarding a future environment. “

3. Stress Testing:

• Analysis of the impact of single extreme events (or risk factors)

• Extreme but plausible

Ref: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf

STR

ESST

ESTING

AN

DM

OD

ELR

ISKM

AN

AG

EMEN

T

Model verification:

• Migration of code from an older code-base/ from another language

• Regression Testing

Model Deployment:

• Model tests to be done prior to deployment

Model Replication:

• Models that replicate alternate models or functionality/strategy

Model Back testing:

• Test how a model performs under different conditions with historical data

What’s involved with Model Risk Management efforts?

MO

DEL

LIFECYC

LEM

AN

AG

EMEN

TK

EYTO

MR

MHow’s Model Risk Management implemented in the company?

Ref: The Decaloguehttp://quantuniversity.com/w9.html

CHALLENGES TO IMPLEMENTING MODEL RISK MANAGEMENT

CH

AR

AC

TERISTIC

SO

FS

TRESS

AN

DS

CEN

AR

IOTESTIN

GCharacteristics of Stress and Scenario testing

1. Difficult to build parametric models – Simulation driven approach necessary

2. Parameter space can explode easily

3. Large number (10s of thousands) of tests needed

4. Human intervention required

5. Company needs are different : Customization required

6. Significant coordination and engagement with multiple groups needed

Ref: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf

IMP

LEMEN

TING

SYSTEM

ATIC

STRESS

TESTING

ISC

HA

LLENG

ING

Challenges to systematic stress testing

1. Model Implementation

• Does it actually work for all intended use cases?

2. Model parameter testing

• How many parameters ?

• How many scenarios ?

3. Model Applicability

4. Model Benchmarking against Reference Implementation

• R vs Proprietary vs MATLAB

5. Model Migration (version)

• Regression Testing v1.0 to v2.0

6. Model use case validation

• Can we use the results to make decisions?

MO

DEL

LIFECYC

LEM

AN

AG

EMEN

TIS

STILLEV

OLV

ING

Challenges in Model Life Cycle Management

1. Model risk management considered as a regulatory/compliance task

2. No one-size-fits-all solution

3. Lack of transparency, standardization and centralized model management

4. Disparate and disconnected systems (SVN, Sharepoint, Excel sheets, custom systems)

5. Difficult to track and audit changes

6. Metrics to quantify model risk not established

MODELRISK ANALYTICS SOLUTION

LEV

ERA

GE

TECH

NO

LOG

YTO

SCA

LEA

NA

LYTICS

Leverage technology to scale analytics

1.64 bit systems : Addressable space ~ 8TB

2.Multi-core processors : Explicit and Implicit Multi-threading

3.Parallel and Distributed Computing : Leverage commodity/specialized hardware to scale problems

4.Micro services-based solution for fast deployment and replication

5.Cloud Computing

Ref: Gaining the Technology Edge: http://www.quantuniversity.com/w5.html

AD

DR

ESSING

STRESS

TESTIN

GC

HA

LLENG

ESHow can we address challenges in Stress testing?

1. Leverage hardware for scaling

1. Enable developers to plan for stress tests

2. Enable developers to scale without having to rewrite code

2. Enable developers to run their code

1. Locally

2. On Enterprise Servers

3. Cloud

AD

DR

ESSING

STRESS

TESTIN

GC

HA

LLENG

ESHow can we address challenges in Stress testing?

3. Facilitate exhaustive and robust testing without having to get lost in details

1. Enable testers to specify their tests and configurations

2. Enable benchmarking and comparisons with other models

4. Facilitate deployment of models built using different frameworks including legacy code

1. Provide a choice of models (which could be built in different languages)

2. Enable benchmarking and comparisons with models built using other languages

AD

DR

ESSING

MO

DEL

LIFEC

YCLE

MA

NA

GEM

ENT

CH

ALLEN

GES

How can we address challenges of Model Lifecycle Management?

1. Provide a single comprehensive environment to manage model validation tests

1. Enable integration with source control systems

2. Enable model validators to create, run and assess tests

2. Facilitate all stages of model lifecycle management in a single environment

1. Allow access to all model related artifacts (documentation, tests and results)

2. Provide ability to collaborate with all stakeholders responsible for model risk

3. Enable audit trails and analytics to manage model risk

MO

DELR

ISKAN

ALYTIC

SS

YSTEMModelRisk Analytics System

MO

DEL

RISK

AN

ALYTIC

SS

OLU

TION

Model Risk Analytics Solution

1. Distributed System to support Model Verification, Validation, Stress and Back Testing

2. Supports MATLAB and other open source packages like R, Python, SVN

3. Architecture leverages state-of-the-art asynchronous computing modes

4. Package management

1. Using Custom Client/ Images

2. Using Docker(Different containers for different apps)

5. Supports synchronous and asynchronous modes

6. Built for the cloud but can work on any cloud/internal infrastructure (private or public)

CASE STUDY

Value-at-Risk & Conditional Value-at-Risk

• VaR : The predicted maximum loss of a portfolio with a specified probability level (e.g., 95%) over a certain period of time (e.g. one day)

• CVaR (Expected Shortfall) : The expected value of the loss given that the loss exceeds VaR

Ref: Optimization Methods in Finance by Gerard Cornuejols, Reha Tutuncu, Cambridge University Press Image courtesy: http://www.imes.boj.or.jp/english/publication/mes/2002/me20-1-3.pdf )

How to Implement VaR and CVaR?

Methodology:

• Historical

• Variance-Covariance method

• Monte-Carlo simulations

Models are implemented in:

• MATLAB – Production Model

• Python – Alternate model developed by a 3rd-party vendor

• R – Reference/Benchmark model from a paper

Methods to compute VaR and CVaR• Historical method

Image Courtesy: http://www.investopedia.com/articles/04/092904.asp

1. Compute Daily Returns and sort them in ascending order

2. For a given confidence level (e.g. 95%) , find VaR α (X) such that:

P(X<= VaR α(X)) = α

3. Compute CVaR by taking the average loss of the tail

Methods to compute VaR and CVaR

• Variance-Covariance Method

Image Courtesy: http://www.investopedia.com/articles/04/092904.asp

1. Compute Daily Returns and fit a Normal distribution to obtain mean and Standard Deviation (µ & σ)

2. For a given confidence level (e.g. 95%) , find VaR α (X) such that:

P(X<= VaR α(X)) = αExample 95% => -1.65* σ

3. Compute CVaR by taking the average loss of the tail

(See Yamai and Yoshibahttp://www.imes.boj.or.jp/english/publication/mes/2002/me20-1-3.pdf )

Methods to compute VaR and CVaR

• Monte-Carlo Simulations

Image Courtesy: http://www.investopedia.com/articles/04/092904.asp

1. Compute Daily Returns and fit a Normal distribution to obtain mean and Standard Deviation (µ & σ)

2. Run n Monte-Carlo simulations with random numbers drawn from a normal distribution described by (µ & σ)

3. For a given confidence level (e.g. 95%) , find VaR α (X) such that:

P(X<= VaR α(X)) = α

4. Compute CVaR by taking the average loss of the tail

Model Summary

VaR Model

Historical Method

Python

R

MATLAB

Variance-Covariance

Method

Python

R

MATLAB

Monte-Carlo Simulations

Python

R

MATLAB

Given:1. Historical Daily price time series for a specified time period2. Constituents of a 3-asset long-only portfolio

Compute:VaR and CVaR

Model Verification criteria

1. Model Benchmarking• MATLAB production model vs Alternate Python model vs

Reference R model

2. Parameter sweeps• Different Confidence Intervals (90%, 95%, 99%)

3. Model Convergence• How many simulations needed ? (100, 500, 1000)

4. How do different methods compare?• Historical vs Variance-Covariance vs Monte-Carlo methods

VarModel.m

ModelRisk Studio facilitates easy structuring of tests

Input parameters can be specified for each model

ModelRisk Analytics allows you to run the model in multiple environments

The Model Testing results can be viewed in an interactive dashboard

Model Issues can be tracked, risks evaluated and addressed appropriately

FEA

TUR

ESComing soon!

ModelRisk Model Definition Language• To specify model parameters

ModelRisk Engine Optimization to leverage infrastructure in the cloud

• Resource constraints: • Budget & Time constraints

• Priority queues and jobs• Dynamic scaling and load balancing

ModelRisk Audit to enable periodic testing and ongoing monitoring of your models

MO

DEL

RISK

AN

ALYTIC

SB

ETAP

RO

GR

AM

If you are interested in being a part of the Beta program:

If you are interested in knowing more about the Model Risk Analytics systemOr Want to be part of the Beta program,

email: sri@quantuniversity.com to be added to the interest list

Model Risk Analytics Beta program

MO

DEL

RISK

MA

NA

GEM

ENT

WO

RK

SHO

PAlso available:

A 2-day workshop in Model Risk Management with hands-on case studies

Email: sri@quantuniversity.com to be added to the interest list

Stress Testing and Model Risk

Management for Quants

AN

OM

ALY

DETEC

TION

WO

RK

SHO

PComing soon:

A 2-day course in methods and best practices in Anomaly DetectionSeptember 19th, 20th , New York

Visit www.analyticscertificate.com/Anomaly for more details

Anomaly DetectionCore Techniques and

Best Practices

WH

ITEPA

PER

S

To get copies of these articles, go to:http://www.quantuniversity.com/articlerequest.html

OREmail: sri@quantuniversity.com

Thank you!

Sri Krishnamurthy, CFA, CAPFounder and CEO

QuantUniversity LLC.

srikrishnamurthy

www.QuantUniversity.com

Contact

Information, data and drawings embodied in this presentation are strictly a property of QuantUniversity LLC. and shall not bedistributed or used in any other publication without the prior written consent of QuantUniversity LLC.