Risk Metrics-Management Tools How Choices in Risk Measurement Methods Affect the Presentation and...

Risk Metrics-Management Tools

How Choices in Risk MeasurementMethods Affect the Presentation and Quality of Risk ReportingApril 16, 2007

2PricewaterhouseCoopers April 2007

2007 FIRMA National Training Conference

Agenda

1 Risk Realities

2 Risk Measurement Approaches

3 Qualitative Risk Reporting

4 Semi-Quantitative Risk Reporting

5 Quantitative Risk Reporting

6

7

Compiling Risk Measurements and Generating Reports

Summary

Risk Realities



Risk Realities

• Senior Management, Board of Directors and Stakeholders are demanding comprehensive risk analysis from internal audit, compliance and risk management professionals.

• This has greatly increased the pressure for these groups to present insightful analysis on risk trends and provide predictions on how risk will change in the future.




Risk Realities

• The pros and cons of using different measurement techniques impact on how you collect data, perform analysis and generate risk reports.

- The choice of risk measurement techniques drives the extent a risk may be measured from quantitative, semi-quantitative or qualitative perspective.

• The choices in risk measurement methods affect the presentation and quality of risk reporting.


Risk Measurement Approaches




What is Measurement?

1. The act of measuring or the process of being measured. 2. A system of measuring: measurement in miles. 3. The dimension, quantity, or capacity determined by measuring: the

measurements of a room.

The American Heritage® Dictionary of the English Language, Fourth EditionCopyright © 2000 by Houghton Mifflin Company.Published by Houghton Mifflin Company. All rights reserved.





What is Risk Measurement?

Identifying risk metrics such as volatility, duration, dollar loss and then developing risk measures to calculate the risk metrics. The value obtained from the risk measure is risk measurement.





Risk Measurement Example

Risk Metric – Operational Loss

Risk Measure – Compensation paid to customers for interest lost when assets are not invested timely.





What are the Risk Measurement Approaches?

• Qualitative

• Semi Quantitative

• Quantitative





Qualitative Risk Measurement

• A means of comparing the relative priority, importance or significance of risks using descriptive scales

• Describes characteristics

• Measures are descriptive e.g., low/medium/high, vital/critical/important, good/bad

• Quantifiable data is not used

• Rankings are based on judgement





Semi-Quantitative Measurement

Quantitative scales are used together with scale descriptions The numbers associated with these descriptions do not have to bear

accurate relationship to the precise magnitude of the risk

The objective is to produce a more detailed assessment of risks than can be achieved using qualitative analysis without attempting to determine a definitive value for a risk





Quantitative Risk Measurement

• A technique that associates a numerical value with a risk and provides an impact value

• Uses complex calculations i.e., algorithm factors such as probability, uncertainty, exposure factor, frequency of occurrence, effectiveness factor, single loss expectancy/exposure


Qualitative Measurement:Risk Reporting Quality and Presentation Considerations



Qualitative Measurement EvaluationRisk Metrics-Management Tools

Approach Advantages Disadvantages

Qualitative • Calculations, if any, are uncomplicated and unsophisticated

• Areas of critical risks are referred to in general terms

• Typically requires much less time

• No numeric calculations• Process results are subjective and may not be repeatable

• Evaluating resulting risk management cannot be accomplished objectively



Reporting Content - Qualitative Measurement

• Because the methodology is straight forward, the target audience and stakeholders can easily understand the descriptive ratings and calculations

• The analysis is subjective and open to the interpretation of the evaluator's

judgment and industry knowledge• Demonstration of consistent application of rating definitions can be a quality

challenge• There can be disagreements in risk ratings between line management and

the risk evaluators

• Providing specific detail on magnitude of risk is limited at best because precise measurement are not possible

• The analysis is more anecdotal than objective. The target audience must understand how to interpret the assessments in the risk reports.




Reporting Format - Qualitative Measurement

Narrative descriptions help provide sufficient explanation to support ratings

• Consider a combination of narrative descriptions of key risks and risk management practices with a color-coded system for depicting the relative importance of key risks/risk measures

Disclosure using qualitative measures and explain their application limitations so the target audience does not arrive at unintended conclusions




Reporting Format - Qualitative Measurement

Present risks as statements of the risk and effect• If risks are written as management objectives, the risk concern may not be

clear

- EXAMPLE

• Objective-An effective Personal Transaction Reporting Program is established.

• Risk-Personal security transactions are not reported in accordance with 12CFR12




Reporting Format - Qualitative MeasurementRisk Metrics-Management Tools

High

Medium

Low

High Medium Low

LIKEIHOOD

INHERENT RISK

Sample Report



Reporting Frequency - Qualitative Measurement

Choosing a monthly, quarterly or annual reporting interval depends on:

• The magnitude of the risk implications• Time period of the data used for the analysis• Escalation of issues to senior management and the Board of Directors• Progress on management action plans to remediate risks


Semi-Quantitative Measurement:Risk Reporting Quality and Presentation Considerations



Semi-Qualitative Measurement EvaluationRisk Metrics-Management Tools


Semi-Quantitative

• Relatively easy to estimate impact and probability using formulae objectively

• Easy to adopt measurement technique

• Uses numerical values without the rigours of numerical analysis which may cause over-simplification of risks

• Requires significant input from specialists in the area under consideration to determine risk values



Reporting Content – Semi-Quantitative Measurement

The content is more objective since numeric data calculations are used to arrive at risk measures supporting the risk report evaluation

• Measures equate to a rating based on a range of values• The risk measures may not be precise enough to reflect the value risk

events• The magnitude of desire and undesirable outcomes are not calculated and

therefore, they could offset each other not yielding true value

There would continue to be some dependence on the knowledge of subject matter expert’s experience to interpret metrics.

• Some narrative would be necessary in the report to discuss the judgment portion of the metric




Reporting Format – Semi-Quantitative Measurement

In addition to considerations from Qualitative Risk Reporting:

Create visual reporting to illustrate impact and likelihood of risks• Heat Maps• Charts demonstrating comparison of business lines and/or risks

Disclosing time period for calculation of metrics so your audience understands the context of the risk discussion

Defining data parameters i.e. transactions, account types, system application from where the data was extracted that were used in the metrics




Reporting Format – Semi-Quantitative Measurement

- Higher Risk - Moderate Risk - Lower Risk - No Applicable Risk

Risk Categories

Business Area Credit Market Legal Operations Financial Compliance Strategic Reputation Technology Fraud Direction of

Risk

Private Banking

Wealth

Management

Employee Benefits

Trust Operations

Display with a color-coded system for depicting the relative importance of key risks/risk management practices




Reporting Frequency – Semi-Quantitative Measurement


• The magnitude of the risk implications• Time period of the data used for the analysis• Escalation of issues to senior management and the Board of Directors• Progress on management action plans to remediate risks


Quantitative Measurement:Risk Reporting Quality and Presentation Considerations



Quantitative Measurement EvaluationRisk Metrics-Management Tools


Quantitative • Processes and results are derived objectively, easily supported

• Calculations for frequency and impacts, countermeasures cost/benefit, etc. are consistent and repeatable

• Evaluating resulting risk management is easy

• Can be automated, quicker, easier to understand, repeatable more reliable

• Calculations are complex and require good explanations

• Information gathering is a considerable up-front task

• Could require up to 20 times the effort if done manually without automated tools



Reporting Content – Quantitative MeasurementRisk Metrics-Management Tools

In addition to considerations from Qualitative and Semi-Quantitative Measurement:

Quantitative Measurement is the most objective analysis of data• Precise measurement of metrics provides reliable information for decision making

• Critical to have computer tools to generate the calculations and evaluate the data

Historical data is needed for quantitative measure to provide calculation for tolerance limits and relevant rages for values.

Need to have methodology that validates the data gathered for analysis to ensure quality output

Data could be benchmarked against peer group to provide relative comparison to understand risks within the industry.



Reporting Format – Quantitative Measurement

The report can become too cluttered to read by the target audience with charts.• Some narrative text to provide background information to connect themes

and patterns can be helpful

Charts and Graphs need to have measurement scales clearly defined

Historical data should be clearly described• Refrain from displaying multiple historical figures to keep the focus on the

period under evaluation- Consider comparing year to date or prior quarter.




Reporting Format – Quantitative MeasurementRisk Metrics-Management Tools

“Residual” approach

The simplest operational risk measure, the residual approach assigns operational risk, based on the following equation: Operational Risk = Total capital – Market Risk – Credit Risk.

Activity-based

Activity-based measures estimate operational risk based on objective measures of the level of business (e.g., number of employees, operating expenses, trade volumes, etc.). This approach is also used as an allocation method for other top-down approaches.

Key risk indicators

These models identify factors and apply them as an early warning tool or to estimate expected losses, through the application of regression techniques or trend analysis. Key risk indicators include factors such as audit ratings, employee turnover, transaction volume or recent operational loss rates.

Operating leverage

This approach seeks to understand the relationship between cost structures (the mix of fixed and variable expenses) and revenues. Risk is the potential deviation between expected and actual expenses. The expected operating expenses are modelled using historical time series of revenues and expenses and regression techniques.

Examples of Quantitative Approaches



Reporting Frequency Quantitative Measurement


• The magnitude of the risk implications.• Time period of the data used for the analysis.• Escalation of issues to senior management and the Board of Directors• Progress on management action plans to remediate risks


Compiling risk measurements and generating risk reports

The risk measurement and reporting relationship



Risk Reporting Policy / ProceduresRisk Metrics-Management Tools

Pare down risk reportsStandard report format with key risk measuresDe-emphasize historical reporting

Streamline processEliminate excess reporting and paperReduce cycle time by actively reducing the number of reports

generated

Increasing usefulnessEmphasize change and trend informationInclude risk analysis by product, market, geography, and customerConsider comparisons against industry and / or competitors by product

/ service when data is availableUse a balanced set of measuresAlign reports with business and user requirementsEncourage changing management reports each year to reflect new

priorities



Reporting Reporting TechnologyRisk Metrics-Management Tools

Improve accuracy and timelinessDevelop flash reports after first day of closeDeliver reports in electronic format to usersEmphasize no re-keying of data

Increasing usefulnessReports available on-line to reduce ad hoc reporting requestsProvide on-line access to information with drill down capabilityAllow on-line feedback from users

Standardizing format and processImplement a single data warehouse for all financial and non-financial

dataDownload to PC based packages for consolidation / analysis for review

of variances / exceptionsAutomate both regulatory reports as much as possible to leverage this

information for risk reporting



Cost vs. Benefit of Generating Risk Reports


Know your cost of:

• Gathering and analyzing risk metrics• Cost of generating risk reports

- Personnel time and cost- Copying- Mailing

• Cycle time for receipt by target audience- Does your target audience have sufficient time to digest the

material before a meeting?



Cost vs. Benefit of Generating Risk ReportsRisk Metrics-Management Tools

Potential Reporting Metrics to Assess Costs:

• Total cost of management reporting / $1,000 revenue

• Personnel cost of management reporting / $1,000 revenue

• Percentage of time spent on data collection and manipulation

• Percentage of time spent on reviews and presentations

• Cycle time for senior management to get reports



Regulator Perspective on Risk Measurement and ReportingRisk Metrics-Management Tools

• The measurement methodology should be formally documented.

• The methodology should be tested to ensure the results derived are what is expected.

• The methodology should be presented to the appropriate oversight committee and/or Board of Directors for review and approval

• Presentations to oversight committees should be at a frequency commiserate with the level and materiality of risk

• Rationale for subjective assignment of risk ratings should be supported by written comments since the assessment is based on the evaluator’s judgment and experience.

• Risk Reports should present risks of the organizations activities in such a manner to convey potential impact of material loss

Summary The risk measurement and reporting relationship



Defining Performance Objectives

A disconnect between risk measurement and objectives can lead to reduced data quality and ineffective reporting when performance objectives are:

• General and do not reflect the business environment.Example-Drive growth by expanding client relationships (How, What, When, Where?)

• Not defined at the entity and activity levelsExample -Entity: Revenue goals at the Business Line level

• Wealth Management plans revenue growth from fee income by 5% in 2006 (increasing personal custody relationships)

-Activity: Revenue goals at the product level• Increase fee income from personal custody by 10% in 2006 (by

offering new online trade settlement service)




Distinction between Risk Measure vs. Risk Metrics

Risk measure is an operation for assigning a number to something.

A metric is your interpretation of the assigned number.

Remember• The measure is a mile and the metric is the distance traveled

• The use of the measure is the measurement




Factors that Drive Risk Reporting Quality and PresentationRisk Metrics-Management Tools

Complexityof the BusinessEnviroment to beMeasured

Sophistication of Risk Measurement



Risk Measure ApproachesSummary

Qualitative

Semi-Quantitative

Quantitative

Tailor to the measures appropriate for your risk environment. Consider using a mix of qualitative and quantitative approach based on your ability to assess materiality for a risk



Qualitative vs. Quantitative Measurement Techniques

A risk measurement methodology can be a combination of qualitative and quantitative techniques.

• Qualitative measures can used where risks do not lend themselves to quantification or when sufficient credible data is not practical

• The reasoning to use qualitative measures should documented along with the evaluation criteria employed to arrive at the rating for the measure

• The limitations of qualitative measures should be disclosed in risk reporting work products




Selecting a Risk Measurement Approach

It can be overwhelming to develop a high powered risk measurement approach

• Initially, focus on identification on your risk universe and then map the risks to your organization’s business lines

• Consider the cost of collecting data and analyzing data in comparison to the value realized for risk reporting

• Start small and then add complexity• Excel with your strengths and reliable data• Move to more sophisticated calculations as your skills develop

• As your risk environment changes, so will your measurement methods


Questions and Answers

© 2006 PricewaterhouseCoopers LLP. All rights reserved. "PricewaterhouseCoopers" refers to PricewaterhouseCoopers LLP (a Delaware limited liability partnership) or, as the context requires, other member firms of PricewaterhouseCoopers International Ltd., each of which is a separate and independent legal entity. *connected thinking is a trademark of PricewaterhouseCoopers LLP.

Valerie Nielsen, DirectorInternal Audit Services312.298.3524

[email protected]/internalaudit

© 2006 PricewaterhouseCoopers LLP. All rights reserved. "PricewaterhouseCoopers" refers to PricewaterhouseCoopers LLP (a Delaware limited liability partnership) or, as the context requires, other member firms of PricewaterhouseCoopers International Ltd., each of which is a separate and independent legal entity. *connected thinking is a trademark of PricewaterhouseCoopers LLP.

Risk Metrics-Management Tools How Choices in Risk Measurement Methods Affect the Presentation and...

Documents

Transcript of Risk Metrics-Management Tools How Choices in Risk Measurement Methods Affect the Presentation and...