Inspection, Evaluation, and Performance Auditing Presented By George F. Grob Center for Public...

Inspection, Evaluation, and Performance Auditing

Presented By

George F. GrobCenter for Public Program Evaluation

To the Association of Inspectors GeneralNew Orleans

November 2013

Center for Public Program Evaluation

Essentials of Evaluations and Inspections

• What are they?• Comparison to Audits• Other Related Professional

Disciplines• Evaluation Types and Approaches• Analytic Frameworks• Possible Organizational Locations• Managing an Evaluation Office• Quality Standards

2



3

What Is an Evaluation


Evaluation involves assessing the strengths and weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness.

4

Evaluation—What Is That?

American Evaluation Association


Examples of Government Sponsored Evaluations

• Early implementation of emergency response systems after 9-11

• Effects of programmatic earmarks for international HIV/AIDS relief programs

• Case study of Freddie Mac’s loss of $1.2 billion from Lehman Brothers bankruptcy

• Frequency of and criteria for home visits to foster care families

• Performance standards for homeless youth centers

5


More Examples• Unlicensed fire sprinkler contractors performing

inspections in Chicago

• Potential use of a fiscal intermediary by the Chicago department of public health

• Assessment of New Orleans' System of City Courts and Performance Review of New Orleans Traffic Court

• Evaluation of the City of New Orleans Delinquent Property Tax Collection Program

• Promising approaches for State and local government enforcement of prohibition on sale of cigarettes to youth

• Pharmacies with extreme outlier filling of prescriptions for dangerous, addictive drugs

6


Websites of OIG Evaluation ReportsFederal Agencies

City Government

• Department of Health and Human ServicesOIG Office of Evaluation and Inspections

• Department of CommerceOIG Audits and Evaluations

• Department of StateOIG Audits and Program EvaluationsOIG Inspections

• Department of Veterans Affairs

OIG Office of Audits and Evaluations Oversight Reports

• Department of Housing and Urban DevelopmentOIG Evaluations and Inspections

• Federal Housing Finance AgencyOIG Audits and Evaluations

• City of ChicagoAudits, Reviews, Advisories

• City of New OrleansAudits, Inspections, and Evaluations

7

U.S. Congress• Government Accountability Office (GAO)

Reports and Testimonies

Audits, Inspections, and Evaluations

https://oig.hhs.gov/reports-and-publications/oei/subject_index.asp

http://www.oig.doc.gov/Pages/Audits-Evaluations.aspx

http://oig.state.gov/lbry/audrpts/index.htm

http://oig.state.gov/lbry/isprpts/index.htm

http://www.va.gov/oig/apps/info/OversightReports.aspx

http://www.va.gov/oig/apps/info/OversightReports.aspx

http://www.hudoig.gov/reports-publications/inspections-evaluations

http://fhfaoig.gov/Reports/AuditsAndEvaluations

http://chicagoinspectorgeneral.org/publications-press/public-reports/

http://nolaoig.org/main/inside.php?page=reports_and_public_letters

http://www.gao.gov/browse/date/week

http://nolaoig.org/main/inside.php?page=reports_and_public_letters



8

Evaluation Standards


• Association of Inspectors General

• Council of the Inspectors General on Integrity and Efficiency

• Government Accountability Office

• Institute for Internal Auditors

• American Evaluation Association

9

Government and Professional Standards for Evaluations and Inspections


10

Association of Inspectors GeneralQUALITY STANDARDS FOR INSPECTIONS, EVALUATIONS, AND

REVIEWS

• General Standards Staff Qualifications Independence Due Professional Care

Qualitative Standards Quality Control Planning Data Collection and Analysis Evidence Timeliness Fraud and Other Illegal Acts Reporting Confidentiality Follow-Up

http://inspectorsgeneral.org/files/2012/06/IGStandards_revised_july2012.pdf


CIGIE Standards for Inspection and Evaluation

11

• Timeliness• Fraud, Illegal Acts, Abuse• Reporting• Follow-Up• Performance Measurement• Working Relationships and

Communications

• Competency• Independence• Quality control• Planning• Data Collection and

Analysis• Evidence• Records maintenance

http://www.ignet.gov/pande/standards1.html


GAO Standards for Performance Audits

• Chapter 3: General Standards▫ Independence▫ Professional Judgment▫ Competence▫ Quality Control

• Chapter 7: Field Work▫ Planning▫ Supervision▫ Evidence▫ Documentation

• Chapter 8: Reporting▫ Form▫ Contents▫ Quality Elements▫ Report Issuance

and Distribution

12

http://www.gao.gov/govaud/yb2003.pdf

http://www.gao.gov/govaud/yb2003.pdf

http://www.gao.gov/index.html

http://www.gao.gov/index.html


• Systematic Inquiry• Competence• Integrity/Honesty• Respect for People• Responsibilities for General & Public

Welfare

13

AEA Guiding Principles for Evaluators

http://www.eval.org/p/cm/ld/fid=51

http://www.eval.org/


•Standards for the Professional Practice of Internal Auditing

14

Institute of Internal Auditors

https://na.theiia.org/standards-guidance/mandatory-guidance/Pages/Standards.aspx

Discusses evaluation of risk and controls, and requires audit results to be based on analysis and evaluation.

No “Evaluation” standard as such


• OIGs may have responsibilities that go beyond audits and investigations.

• This additional work includes inspecting, evaluating, reviewing, studying, and/or analyzing government operations and programs for the purposes of providing information for decision-making, and of making recommendations to improve programs, policies, or procedures.

• The objectives of these processes include: providing a source of factual and analytical information, monitoring compliance, measuring performance, and assessing the efficiency and effectiveness of operations.

15

Inspections, Evaluations, and Reviews—AIG Definition

Source: Association of Inspectors General


Systematic and independent assessments of the design, implementation, and/or results of an Agency’s operations, programs, or policies.

They can be used to determine efficiency, effectiveness, impact, and/or sustainability of agency operations, programs, or polices.

16

Inspections and Evaluations—Federal IG Definition

Source: Council of the Inspectors General on Integrity and Efficiency


• They provide information that is timely, credible, and useful for agency managers, policymakers, and others.

▫ providing factual and analytical information; ▫ measuring performance; ▫ determining compliance with applicable law, regulation, and/or

policy ▫ identifying savings and funds put to better use; ▫ sharing best practices or promising approaches; ▫ assessing allegations of fraud, waste, abuse, and mismanagement.

17

Inspections and Evaluations—Federal IG Definition

Source: Council of the Inspectors General on Integrity and Efficiency



18

The Proving Professions


19

Auditors and Evaluators

EvaluationAudit


20


EvaluationAudit


21


EvaluationAudit

Performance Audit

Performance Evaluation

Financial Audits

Attestation Audits

Non-Audit Services

Issue Analysis

Inspections

Reviews

A Multitude of Analysis Types


22

Other Professional DisciplinesEvaluators/Inspectors Come From Many Professions

• Evaluation• Inspection• Audit• Social Science• Economics• Operations

Research Analysis

• Policy Analysis• Public Administration• Program Specialties

▫ Health Care▫ Housing▫ Defense▫ Trade▫ International Assistance


23

LegalAnalysis

Inspections

Economics

Evaluation

OperationsResearch

Applied Social

Research

Psychology

Statistics

Public Administration


24

LegalAnalysis

Inspections

Economics

Evaluation

OperationsResearch

Applied Social

Research

Psychology

Statistics



What About Investigators?

25

Evaluators Investigators?


Useful Interplay

26

Investigators

Evaluators

Fraud leads

Fraud trends

Tech

assist



27

Evaluation Types and Approaches


Evaluation Types and Approaches

• There are many legitimate and useful forms of evaluations

• Each has its purpose. The methodological approach must be adapted to it

• Generally accepted professional guidelines are available to guide the work.

28



29

As A Program MaturesThe Evaluation Questions Change

How is the implementation going?Any Issues?

Are service providers in compliance with program rules?

Are some service providers doing better than others?How so?

Are the required services being delivered?

Are impacted citizens or businesses satisfied with program results?

How do the services stack up to goals and standards?

Have conditions improved as intended?



30

And So Do the Analytic Approaches and Methods Used

Some inspections and evaluations need to be done fast

Others need more time

Some require more resources than others

All must be done professionally

Following are some evaluation and inspections types commonly used today.



31

Inspection/Evaluation Types

• Compliance Review• Performance Review• Landscape Survey• Early Implementation

Reviews• Impact Assessment• Report Card• Process Review



32

More Inspection/Evaluation Types

• “Just the Facts” Review• Report from “On the

Ground”• “Try it out” Reviews• Client Satisfaction Surveys• Before and After

Assessment• Control System Survey• Program Logic Model



33

Analytic Frameworks



34

The Standard IG Analytic Model

•FINDINGS -- Criteria -- Condition -- Effect -- Cause

•RECOMMENDATIONS


What the Federal IG Blue Book Says


35

“Findings often have been regarded as containing the elements of criteria, condition, effect, and, when problems are found, cause. However, the elements needed for a finding depend entirely on the objectives of the inspection. Thus, a finding or set of findings is complete to the extent that the inspection objectives are satisfied and the report clearly relates those objectives to the applicable elements of a finding.”


Two Analytic FrameworksPerformance Audit Performance Evaluation

• Findings▫ Criteria▫ Condition▫ Effect▫ Cause

• Evidence▫ Physical▫ Documentary▫ Testimonial▫ Analytic

• Recommendations

• Objective

• Questions to Be Answered

• Analytic Approach▫ Mixed Methods▫ Qualitative▫ Quantitative

• Findings

• Recommendations

36



37

METHODOLOGIES



38

The Fundamentals• The Correct Methods to Use Are

the Ones Most Compatible With the Purpose of the Study

• Always Use More Than One Method▫ At least one quantitative▫ At least one qualitative



39

Quantitative and Qualitative Methods• Quantitative methods involve things you can count

▫ Even if they are subjective answers to a closed ended questionnaire

• Qualitative methods are applied to the interpretation of statements▫ Especially from interviews▫ But also from observations and in documents

• Reviews are far more compelling if both methods are used▫ Otherwise people will say: “Numbers always lie” or “They always say that”



40

The Most Common Methods Used by Evaluators

• Background Search• On-Site Inspections• Review of Administrative

Files• Sampling• Data Collection

Instruments• Surveys• Focus Groups



41

Some Exercises


You Be the Evaluator

• Early implementation of emergency response systems after 9-11

• Frequency of and criteria for home visits to foster care families

• Pharmacies with extreme outlier filling of prescriptions for dangerous, addictive drugs

42


The Quick Start1. State the purpose

2. Ask two or three key questions 3. Choose the methods for each question

Quantitative Qualitative

4. Conduct the Field Work5. Analyze the data6. Construct Findings and Recommendations7. Draft the report


43


Discussion1. Would your approach work as either an

audit or evaluation?

2. Who will use the findings and recommendations?

3. How will you know that the recommendations have been acted upon?

4. What can you expect will be the result?

44



45

Organizing and Managing the Evaluation Function


Possible Organizational Location

46

Inspector General Evaluation

AuditInvestigatio

n Evaluation

Evaluation Evaluation


Managing an Evaluation Office

47

Work Planning

Operating Procedures

Quality Control

Quality Assurance

Professional Development

These are generally similar to

audit practices

Take Them Seriously !


Training Priorities for EvaluatorsHighest Priority Other Worthwhile Skills

• General Evaluation Concepts

• Writing to the Point

• Survey Design• Interviewing• Descriptive Statistics• Sampling• Spread Sheets• Graphs

• Focus Groups• Data Base Development• Randomized Control Trials

• Team Leadership• Programmatic Knowledge

48


Work Planning

•Combine Audit and Evaluation•Focus on risk and return on investment•Use a Tactical Approach

▫ To systematically fill gaps▫ To develop staff capacity▫ To keep up the attention on important areas

49



50

Program Area A

Program Area B

Program Area C

Program Area D

A strategic plan might consist of several families of program studies, some in the early stages, some further along, even in final stages


51

LegalAnalysis

Inspections

Economics

EvaluationAndAudit

OperationsResearch

Applied Social

Research

Psychology

Statistics


Listen to What They Have to Say


52

Good Luck!


George F. Grob

Center for Public Program Evaluation540-454-2888

[email protected]

53

mailto:[email protected]

Inspection, Evaluation, and Performance Auditing Presented By George F. Grob Center for Public...

Documents

Transcript of Inspection, Evaluation, and Performance Auditing Presented By George F. Grob Center for Public...