Frank Masi, Ph.D., EVP Operations. Agenda Origins Model validation (OCC SR11-7) Regulatory pressure...

AML MODEL VALIDATIONA critical need in the new regulatory environment

Frank Masi, Ph.D., EVP Operations

Agenda

• Origins• Model validation (OCC SR11-7)• Regulatory pressure (exit letters, community banks)

– Where, community banks, U.S. wide• Conducting IVV

– Process– Areas

• Reporting• Independence• Vendor Due Diligence• Q&A

© 2015ARC Risk and Compliance

2

3

Model validation has been around for some time.

• Independent Software Verification and Validation (ISVV) derives from the application of IV&V (Independent Verification and Validation) to the software. Early ISVV application (as known today) dates back to the early 1970s when the U.S. Army sponsored the first significant program related to IV&V for the Safeguard Anti-Ballistic Missile System.

• By the end of the 1970s IV&V was rapidly becoming popular. The constant increase in complexity, size and importance of the software lead to an increasing demand on IV&V applied to software (ISVV).

• Meanwhile IV&V (and ISVV for software systems) gets consolidated and is now widely used by organizations such as the DoD, FAA, NASA[1] and ESA.[2] IV&V is mentioned in [DO-178B], [ISO/IEC 12207] and formalized in [IEEE 1012].

• Initially in 2004-2005, a European consortium led by the European Space Agency, and composed by DNV(N),[3] Critical Software SA(P),[4] Terma(DK)[5] and CODA Scisys(UK)[6] created the first version of a guide devoted to ISVV, called "ESA Guide for Independent Verification and Validation" with support from other organizations, e.g. SoftWcare SL (E) ( [7]

), etc.• In 2008 the European Space Agency released a second version, being SoftWcare SL was

the supporting editor having received inputs from many different European Space ISVV stakeholders. This guide covers the methodologies applicable to all the software engineering phases in what concerns ISVV.


Origins

http://en.m.wikipedia.org/wiki/U.S._Army

http://en.m.wikipedia.org/wiki/Anti-Ballistic_Missile

http://en.m.wikipedia.org/wiki/Independent_software_verification_and_validation


http://en.m.wikipedia.org/wiki/European_Space_Agency






4

• “Model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses. Effective validation helps ensure that models are sound. It also identifies potential limitations and assumptions, and assesses their possible impact.”

(Board of Governors of the Federal Reserve System (SR 11-7), 2011, p. 3)

Purpose of Model Validation


5

“Model risk occurs primarily for two reasons:

• The model may have fundamental errors and may produce inaccurate outputs when viewed against the design and objective and intended business uses…

• The model may be used incorrectly or inappropriately.”

(Board of Governors of the Federal Reserve System Office of the Comptroller of the Currency (SR 11-7a1), 2011, p. 3)

Model Issues


6

“All model components—inputs, processing, outputs, and reports—should be subject to validation; this applies equally to models developed in-house and to those purchased from or developed by vendors or consultants.”

(Board of Governors of the Federal Reserve System (SR 11-7), 2011, p. 3)

Model Components


Inputs Processes Outputs Reports

7

• “Evaluation of Conceptual Soundness. This element involves assessing the quality of the model design and construction, as well as review of documentation and empirical evidence supporting the methods used and variables selected for the model. This step in validation should ensure that judgment exercised in model design and construction is well informed, carefully considered, and consistent with published research and with sound industry practice.”

Key Elements of a Comprehensive Validation


8

• “Ongoing Monitoring. This step in validation is done to confirm that the model is appropriately implemented and is being used and performing as intended. It is essential to evaluate whether changes in products, exposures, activities, clients, or market conditions necessitate adjustment, redevelopment, or replacement of the model and to verify that any extension of the model beyond its original scope is valid. Benchmarking can be used in this step to compare a given model’s inputs and outputs to estimates from alternatives.”



• “Outcomes Analysis. This step involves comparing model outputs to corresponding actual outcomes. Back-testing is one form of outcomes analysis that involves the comparison of actual outcomes with model forecasts during a sample time period not used in model development at a frequency that matches the model’s forecast horizon or performance window.”

9



Regulatory Pressure

What form we are seeing this:• Exit letters (MRAs, MRIAs)• RFP/RFIs• Service solicitations

Where we are seeing this:

10

Where Who

NY City NJ Foreign Banks

Philadelphia Chicago Wholesale Banks

Ohio Miami Community Banks

Kansas City Tampa Trust Companies© 2015

ARC Risk and Compliance

11

CONDUCTING AN IVV


12

Parts of a Model Validation

IVV/Model Validation

Documentation Review

Policies

-Risk Assessment-BSA/AML and OFAC Policies-Monitoring Scenarios

Procedures

-Case Review-RFI-SAR Filing

BRD

-Data Mapping-Parsing-Controls

System Review

Input

-Data Mapping-Translated Data-Data Completeness-Data Accuracy-Truncation & Formatting

Process

-Baseline-Rules/Profiles-Configuration-Populations-Statistics

Output

-Alerts\Cases-Logs-Verification

Reporting

-Clear Identification Risks-False Positive Management


13

A Little Statistics


Sample Project

14

Week 1

Documentation Review•Review mapping documents;•Review branch risk assessment;•Audit reports;•AML/OFAC policies and procedures;•Product/System manuals;•Business requirement documents;•Functional design documents; and•Previous IVVs.

Week 2

Input Verification•Create and review data samples;•Review data for consistencies, accuracy, and appropriateness;•Controls;•Data Normalization;•Data standardizations;•Reconciliations;

•Data translations;•Data validation (mandatory, required, and supplemental);

Week 3

Process Verification•Review of aggregations, calculations, translations, thresholds and transformations;

•Statistical evidence validating thresholds, parameters, and categories;

Week 4

Output Verification•Demonstrable workflows and reviews;•Effective management reporting;•GAPs between AML policies and procedures;•Management of type 1 and type 2 errors;

Week 5-6

Report•Executive summary;•Top recommendations;•Demonstrate Policies and Procedure to System GAPs;•Document data analysis;•Document product analysis; and•Provide Observations and Recommendations.


Document Review (examples)

• Review mapping documents;• Review risk assessment;• Audit reports;• AML/OFAC policies and procedures;• Product/System manuals;• Business requirement documents;• Functional design documents; and • Previous IVVs.

15© 2015


Input Review (examples)

• Create and review data samples (focused sampling);• Review data for consistencies, accuracy, and

appropriateness (% populated, column confidence); • Data validation (mandatory, required, and

supplemental);• Review Controls (weak, medium, strong);• Data Normalization (convert to US $);• Data standardizations (US or European date formats);• Reconciliations (all transactions from source are

received and verified);

16© 2015


Process Review (examples)

• Data translations;– Data translations consist of a review and validation are

date format changes, text to currency, address parsing, calculated data, derived data, or any data manipulations within the software.

• Review of aggregations, calculations, translations, thresholds and transformations;

• Reports Analysis

17

0

200000000

400000000



18

Def-Class (Ex-

empt) 72

HighRisk 26

LowRisk

26

MediumRisk

30

Customers by Risk Class

For-eign

Banks 82

US Cor-pora-tion 28

US Bank 22

For-eign Cor-pora-tion 11

Foreign Owned US Corporation 10

Central Bank 1

Customers by Type

59

41

663 32

1 1 111 1

Customer Distribution by Country of Location

United States of America

CHINA

UNITED KINGDOM

HONG KONG

AUSTRALIA

SINGAPORE

KOREA, REPUBLIC OF

MACAO

PANAMA

JAPAN

BELGIUM

GERMANY

VIET NAM



19

Both Countries popu-lated2%

Missing Bene or Orig-inator Country

98%

Debit & Credit Country CodesIn All Wires Since Inception

Originator Country Codes Populated

16%

Origina-tor Coun-try Codes Missing

84%

Missing Originator Country CodesIn All Wires Since Inception

MT LC HR2 HR3 LR LLC BOX CE

Baseline 29 18 7 13 15 8 8 1

-20% 31 23 8 16 16 8 10 1

-40% 38 30 8 21 18 11 12 1

-60% 46 47 9 37 23 12 13 1

2.5

12.5

22.5

32.5

42.5

29

18

7

13 15

8 8

1

31

23

8

16 16

8 10

1

38

30

8

2118

11 12

1

46 47

9

37

23

12 13

1

2014 4th Quarter Alerts (Originator)(Baseline and Adjusted)

Baseline -20% -40% -60%

GB 1GU 4

HK 79 KH 191 SG 1

TW 2347

US 1405

VN 377

WT 1

2014 Transactions by Beneficiary Country

GB GU HK KH SG TW US VN WT


Output Review (examples)

• Demonstrable workflows and reviews;• Effective management reporting;• GAPs between policies and procedures; and• Management of type 1 and type 2 errors.

20

Type 1 Errors (False

Positives)

Type 2 Errors (False

Negatives)


21

VENDOR DUE DILIGENCE


Independence

Independence is measured by two factors:• Distance – how far you are removed from the original

project/model setup/changes.• Time – how long since the vendor was involved in the

project/model setup/changes.

A good rule of thumb is that the reviewer should not have been involved in the last setup/changes/review within the last 12 to 18 months.

22© 2015


23

• Independence• Knowledgeable about product or technology• Knowledgeable about compliance• Knowledgeable about process• Knowledgeable about business• Strong model validation methodology

Vendor Qualifiers


24

A strong model validation policy supports a strong governance program.

A strong model validation policy is risk mitigation.

Conclusion


References

Federal Financial Institutions Examination Council (2010). Bank Secrecy Act/ Anti-Money Laundering Examination Manual. Retrieved June 21, 2013: http://www.ffiec.gov/bsa_aml_infobase/documents/BSA_AML_Man_2010.pdf.

Board of Governors of the Federal Reserve System Office of the Comptroller of the Currency. (2011, April 4). Supervision and Regulation Letters (SR 11-7a1). Retrieved May 1, 2014, from Board of Governors of the Federal Reserve System: http://www.federalreserve.gov/bankinforeg/srletters/sr1107a1.pdf

Board of Governors of the Federal Reserve System. (2011, April 4). Supervision and Regulation Letters (SR 11-7). Retrieved April 30, 2014, from Board of Governors of the Federal Reserve System: http://www.federalreserve.gov/bankinforeg/srletters/sr1107.htm

25© 2015


THANK YOU

26

Questions

Contact Information:

Frank Masi, [email protected] ext. 102 http://www.arcriskandcompliance.com


Frank Masi, Ph.D., EVP Operations. Agenda Origins Model validation (OCC SR11-7) Regulatory pressure...

Documents

Transcript of Frank Masi, Ph.D., EVP Operations. Agenda Origins Model validation (OCC SR11-7) Regulatory pressure...