Measuring Regulatory Excellence: Key Performance Indicators

2007 Annual Conference

Measuring Regulatory Excellence: Key

Performance IndicatorsBruce G. Matthews, P.Eng.

Professional Engineers Ontario

Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference

Overview• Project Definition and Scope

• What is a Key Performance Indicator?

• Project Process

• Challenges

• List of Potential KPIs

• Survey re: Relevance and Importance

• Recommended KPIs

• Aftermath

Project Definition and Scope

• Project arose from a Strategic Plan Objective: “Increase the public’s confidence in PEO as a regulator”

• Associated Strategic Initiative: “Develop and implement key performance indicators and targets of regulatory effectiveness, benchmarked against other comparable regulatory bodies.”

Project Definition and Scope con’t

• Literature review re: best practices

• Review of other regulators’ performance measurement practices

• Survey of stakeholders re: perceptions and expectations

• Assess availability of data for ongoing tracking / reporting of KPIs

What is a Key Performance Indicator?• A metric used to reflect the critical success

factors of an organization

• Can be financial or non-financial

• Nature of KPIs will differ depending upon the nature of an organization

• A KPI is a key part of a measurable objectives (which includes a direction, KPI, benchmark, target, timeframe)

Effective KPIs…• Should help define:

– The What – Are we doing the right things? What is our outcome in terms of impact and accomplishments?

– The How – Are we doing things right? Do we have the right inputs in terms of resources and processes?

Project Process• Literature Review

• Environmental Scan - Review performance measurement practices of other professional regulators

• Identify potential KPIs

• Survey stakeholders re: relevance and importance of potential KPIs

• Review pros and cons of potential KPIs

Literature Review re: Best Practices

• Performance measurement best practices– Balanced scorecard– UK National Consumer Council– “The Regulatory Craft”– “Red Tape to Smart Tape” International

Conference (2002)

Balanced Scorecard• Developed by Harvard professors Kaplan

and Norton

• Four categories– Customers– Internal Business Processes– Financial– Organizational Capabilities

• Weakness re: definition of “customer”

UK National Consumer Council

• Three steps to credible regulation– Objectives– Content– Structure and Governance

• www.ncc.org.uk

“The Regulatory Craft”• Book by Malcolm K. Sparrow (Brookings

Institution Press, 2000)

• Controlling risks, solving problems and managing compliance

• Matrix of “Regulatory Competencies” and “Business Results”

• Acknowledges “internal” versus “external” measures

Red Tape to Smart Tape Conference

• Sponsored by the Ontario Government in association with the Institute of Public Administration of Canada

• Core concept – Traditional business results measurements do not address public welfare issues and are not suitable for regulatory purposes

• www.smarttape.ca

Environmental Scan• Other engineering regulators in Canada

• Other professional regulatory bodies in Ontario and across Canada

• Most report key transactional statistics (e.g., number of licence applications)

• Little evidence of outcome measurement

Challenges• What does it mean to be an effective

regulator?

• Inability to benchmark due to lack of reporting by other “comparable regulatory bodies”

• Can’t use measures typical for professional associations

• KPIs must be evaluated in context

List of Potential KPIs• Competence programs

• Intrusions on regulatory mandate

• Complaints about licence holders or unlicensed individuals

• Licensing turnaround times

• Compliance action statistics

• Council evaluation

List of Potential KPIs con’t

• Fairness & dignity survey

• Complaints about PEO

• Employee engagement

• Percentage of licence holders with no reported violations

• Public confidence survey score

• Ratio of administrative overhead to program delivery cost

Survey re: Relevance and Importance

• PEO Council, Senior Management and selected external stakeholders

• Relevance – the applicability of a KPI as a measure of PEO’s effectiveness as a regulator

• Importance – the priority that should be given to a KPI as a measure of PEO’s effectiveness as a regulator

Survey re: Relevance and Importance

• Survey also included proposed short-term measures for each KPI, intended to cover the introductory transitional period

• Survey results were assessed in view of the best practices review– Need balance of internal and external

measures)– Need to ensure key regulatory competencies

are represented

Recommended KPIs• Competence programs

• Complaints about licence holders or unlicensed individuals

• Compliance action statistics

• Employee engagement

• Public confidence survey score

Competence Programs• Pros

– Competent members reflect a well regulated profession

– Competency is fundamental to professional regulation and easily understood by the public

• Cons– PEO currently has no competence programs

in place and none planned in the near term

Complaints about Licence Holders etc.

• Pros– Easily measured (transaction statistics

already reported)

• Cons– Frivolous or vexatious complaints could skew

the data– Does complaint volume really speak to

regulatory effectiveness?

Compliance Action Statistics

• Pros– Clear statement of enforcement activity– Some statistics already reported

• Cons– Need to define what qualifies as a

“compliance action”– Of greater interest to licence holders than to

the public

Employee Engagement• Pros

– Regular survey of employees already in place– Survey is widely recognized and used

• Cons– Is PEO willing to take action in deficient

areas?– Are the survey areas relevant to a

professional regulatory organization?

Public Confidence Survey Score

• Pros– Solid external measure of how PEO is

perceived– Survey would be conducted by an

independent research firm

• Cons– Frequency of measurement– Who is “the public”?

Implementation Issues• Currently no competence programs in

place at PEO

• Measurement data exists for only three of the recommended KPIs

• Public Confidence Survey still in development

• Issues of KPI reporting (timing, frequency, format) still to be determined

Aftermath• Project report received and considered by

PEO Council in November 2006

• Concern that recommended KPIs were all “lagging indicators” – there were no “leading indicators”

• Concern that recommended KPIs would drive policy and program development

• Concern about lack of competency programs

Aftermath con’t• PEO Council struck a task force to further

examine the question of KPIs for PEO

• Task force report (June 2007) recommends a new project with more research work re: critical success factors before defining KPIs

• Status as of September 2007: awaiting new project charter / terms of reference

Speaker Contact Information

Bruce G. Matthews, P.Eng.Manager, Complaints & DisciplineProfessional Engineers Ontario25 Sheppard Avenue West, Suite 1000Toronto, Ontario M2N 6S9T: 416-840-1076F: 416-224-9974E: bmatthews@peo.on.ca

2007 Annual Conference

Measuring Regulatory Excellence – A Self Audit Tool Focused on Entry to Practice

Jan RobinsonRegistrar & CEO

College of Physiotherapists of OntarioSeptember 8, 2007

Interesting times

• accountability

• transparency

• movement of internationally educated professionals

• public perspective

Politics in Ontario

Ontario Regulators for Access (ORA)

• Info Sharing

• Best practices

• Collaboration

“measures” ➪ government driven

report card to progress report

Key Factors Included• # of internationally trained applicants

(annual comparisons)• top five source countries• types of license granted• project initiatives

Focus on “10”

Thomson review

Key Recommendations

• fair registration practice code

• information for applicants

• criteria for decisions

• reasonable costs

• alternative evidence for unattainable documents

• training

• third party review

• self audits with reporting

• appeal procedures

2002 – ISO 17024

General requirements for bodies operating certification schemes

for persons

2006 Fair Access to Regulated Professions Act

Key features

➪ access centre ➪ Fairness Commissioner ➪ audit

Consider self audit tool

• gap analysis

• demonstrate commitment

• continuous improvement

• achieve best practice

Focus on entry to practice

Conversion to standards38 – registration10 - organization

Examples• The College has policies and procedures

related to confidentiality and privacy

• The College has clear processes for reviewing, evaluating and making changes to registration requirements

Examples• The College has uniform competence

requirements

• The College ensures that individuals with a conflict of interest are not involved in the decision-making process related to an applicant.

Format

• standard definition

• evidence

• analysis

• recommendation

• ISO reference

Key Findings

• identified no major gaps or concerns

• links between College and third party hard to evaluate

• Identified new areas for policy work activity

• identified areas where evidence weak

• tedious

• how often necessary?

Key Outcomes

• confidence in fairness and transparency of process

• evidence of performance all included in one place

• celebration of excellence

Next steps

• develop work plan to implement recommendations

• share with government and other regulators

Speaker Contact Information

Jan Robinson

Registrar & CEO

College of Physiotherapists of Ontario

e-mail : jrobinson @collegept.org

www.collegept.org

Measuring Regulatory Excellence: Key Performance Indicators

Documents

Transcript of Measuring Regulatory Excellence: Key Performance Indicators

National environmental indicators: measuring what matters?

Specific and Generic Performance Indicators for Measuring ...

Recommended Key Performance Indicators for Measuring ANSP ...€¦ · Recommended Key Performance Indicators for Measuring ANSP Operational Performance flight efficiency and predictability

Measuring Inequality with Asset Indicators - World Banksiteresources.worldbank.org/DEC/Resources/finaljpopmckenzie1.pdf · Measuring Inequality with Asset Indicators ... and a bootstrap

APPLICABLE INDICATORS FOR MEASURING THE LIQUIDITY OF ...

Measuring and Managing Research Excellence...0 Measuring and Managing Research Excellence HEPI Conference: Assessing impact, rewarding excellence: REF 2014 and beyondLondon, 22nd November

Measuring Urban Sustainabilitymedia2.planning.org/APA2012/Presentations/S487_Measuring Sustai… · Measuring urban sustainability Sustainability Indicators: Static, discrete indicators

Performance Indicators for Measuring Performance of - GUPEA

MEASURING AND MAPPING INDICATORS OF WATER RISK

MEASURING MUTUALITY INDICATORS FOR FINANCIAL MUTUALS

Indicators for Measuring Progress in Obesity Prevention/media/Files/Report Files/2013... · INDICATORS AUGUST 2013 ... Indicators for Measuring Progress in Obesity Prevention ...

Measuring Contraceptive Security Indicators 2013 – Summary Findings

Indicators for measuring library resources and services

Measuring Excellence with KnowAll Enquire

Measuring Capacity Indicators of Civil Society for ...

Latent constructs and composite indicators for measuring ...complexity.stat.unipd.it/system/files/CAPPAetal.pdfLatent constructs and composite indicators for measuring well-being of

Measuring Performance Excellence with Baldrige

Measuring Quality and Excellence: Identifying Indicators · 2019. 4. 30. · Why We Need Measurement Measurement allows identification of those with difficulties and monitoring of

Measuring and Benchmarking Key Performance Indicators:

Measuring Your Customer Satisfaction Excellence