Inspection, Evaluation, and Performance Auditing
description
Transcript of Inspection, Evaluation, and Performance Auditing
![Page 1: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/1.jpg)
Inspection, Evaluation, and Performance Auditing
Presented By
George F. GrobCenter for Public Program Evaluation
To the Association of Inspectors GeneralNew Orleans
November 2013
![Page 2: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/2.jpg)
Center for Public Program Evaluation
Essentials of Evaluations and Inspections
• What are they?• Comparison to Audits• Other Related Professional
Disciplines• Evaluation Types and Approaches• Analytic Frameworks• Possible Organizational Locations• Managing an Evaluation Office• Quality Standards
2
![Page 3: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/3.jpg)
Center for Public Program Evaluation
Center for Public Program Evaluation
3
What Is an Evaluation
![Page 4: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/4.jpg)
Center for Public Program Evaluation
Evaluation involves assessing the strengths and weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness.
4
Evaluation—What Is That?
American Evaluation Association
![Page 5: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/5.jpg)
Center for Public Program Evaluation
Examples of Government Sponsored Evaluations
• Early implementation of emergency response systems after 9-11
• Effects of programmatic earmarks for international HIV/AIDS relief programs
• Case study of Freddie Mac’s loss of $1.2 billion from Lehman Brothers bankruptcy
• Frequency of and criteria for home visits to foster care families
• Performance standards for homeless youth centers
5
![Page 6: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/6.jpg)
Center for Public Program Evaluation
More Examples• Unlicensed fire sprinkler contractors performing
inspections in Chicago
• Potential use of a fiscal intermediary by the Chicago department of public health
• Assessment of New Orleans' System of City Courts and Performance Review of New Orleans Traffic Court
• Evaluation of the City of New Orleans Delinquent Property Tax Collection Program
• Promising approaches for State and local government enforcement of prohibition on sale of cigarettes to youth
• Pharmacies with extreme outlier filling of prescriptions for dangerous, addictive drugs
6
![Page 7: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/7.jpg)
Center for Public Program Evaluation
Websites of OIG Evaluation ReportsFederal Agencies
City Government
• Department of Health and Human ServicesOIG Office of Evaluation and Inspections
• Department of CommerceOIG Audits and Evaluations
• Department of StateOIG Audits and Program EvaluationsOIG Inspections
• Department of Veterans Affairs
OIG Office of Audits and Evaluations Oversight Reports
• Department of Housing and Urban DevelopmentOIG Evaluations and Inspections
• Federal Housing Finance AgencyOIG Audits and Evaluations
• City of ChicagoAudits, Reviews, Advisories
• City of New OrleansAudits, Inspections, and Evaluations
7
U.S. Congress• Government Accountability Office (GAO)
Reports and TestimoniesAudits, Inspections, and Evaluations
![Page 8: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/8.jpg)
Center for Public Program Evaluation
Center for Public Program Evaluation
8
Evaluation Standards
![Page 9: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/9.jpg)
Center for Public Program Evaluation
• Association of Inspectors General
• Council of the Inspectors General on Integrity and Efficiency
• Government Accountability Office
• Institute for Internal Auditors
• American Evaluation Association
9
Government and Professional Standards for Evaluations and Inspections
![Page 10: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/10.jpg)
Center for Public Program Evaluation
10
Association of Inspectors GeneralQUALITY STANDARDS FOR INSPECTIONS, EVALUATIONS, AND
REVIEWS • General Standards Staff Qualifications Independence Due Professional Care
Qualitative Standards Quality Control Planning Data Collection and Analysis Evidence Timeliness Fraud and Other Illegal Acts Reporting Confidentiality Follow-Up
http://inspectorsgeneral.org/files/2012/06/IGStandards_revised_july2012.pdf
![Page 11: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/11.jpg)
Center for Public Program Evaluation
CIGIE Standards for Inspection and Evaluation
11
• Timeliness• Fraud, Illegal Acts, Abuse• Reporting• Follow-Up• Performance Measurement• Working Relationships and
Communications
• Competency• Independence• Quality control• Planning• Data Collection and Analysis• Evidence• Records maintenance
http://www.ignet.gov/pande/standards1.html
![Page 12: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/12.jpg)
Center for Public Program Evaluation
GAO Standards for Performance Audits• Chapter 3: General
Standards▫ Independence▫ Professional Judgment▫ Competence▫ Quality Control
• Chapter 7: Field Work▫ Planning▫ Supervision▫ Evidence▫ Documentation
• Chapter 8: Reporting▫ Form▫ Contents▫ Quality Elements▫ Report Issuance
and Distribution
12
http://www.gao.gov/govaud/yb2003.pdf
![Page 13: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/13.jpg)
Center for Public Program Evaluation
• Systematic Inquiry• Competence• Integrity/Honesty• Respect for People• Responsibilities for General & Public
Welfare
13
AEA Guiding Principles for Evaluators
http://www.eval.org/p/cm/ld/fid=51
![Page 14: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/14.jpg)
Center for Public Program Evaluation
•Standards for the Professional Practice of Internal Auditing
14
Institute of Internal Auditors
https://na.theiia.org/standards-guidance/mandatory-guidance/Pages/Standards.aspx
Discusses evaluation of risk and controls, and requires audit results to be based on analysis and evaluation.
No “Evaluation” standard as such
![Page 15: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/15.jpg)
Center for Public Program Evaluation
• OIGs may have responsibilities that go beyond audits and investigations.
• This additional work includes inspecting, evaluating, reviewing, studying, and/or analyzing government operations and programs for the purposes of providing information for decision-making, and of making recommendations to improve programs, policies, or procedures.
• The objectives of these processes include: providing a source of factual and analytical information, monitoring compliance, measuring performance, and assessing the efficiency and effectiveness of operations.
15
Inspections, Evaluations, and Reviews—AIG Definition
Source: Association of Inspectors General
![Page 16: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/16.jpg)
Center for Public Program Evaluation
Systematic and independent assessments of the design, implementation, and/or results of an Agency’s operations, programs, or policies.
They can be used to determine efficiency, effectiveness, impact, and/or sustainability of agency operations, programs, or polices.
16
Inspections and Evaluations—Federal IG Definition
Source: Council of the Inspectors General on Integrity and Efficiency
![Page 17: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/17.jpg)
Center for Public Program Evaluation
• They provide information that is timely, credible, and useful for agency managers, policymakers, and others.▫ providing factual and analytical information; ▫ measuring performance; ▫ determining compliance with applicable law, regulation, and/or
policy ▫ identifying savings and funds put to better use; ▫ sharing best practices or promising approaches; ▫ assessing allegations of fraud, waste, abuse, and mismanagement.
17
Inspections and Evaluations—Federal IG Definition
Source: Council of the Inspectors General on Integrity and Efficiency
![Page 18: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/18.jpg)
Center for Public Program Evaluation
Center for Public Program Evaluation
18
The Proving Professions
![Page 19: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/19.jpg)
Center for Public Program Evaluation
19
Auditors and Evaluators
EvaluationAudit
![Page 20: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/20.jpg)
Center for Public Program Evaluation
20
Auditors and Evaluators
EvaluationAudit
![Page 21: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/21.jpg)
Center for Public Program Evaluation
21
Auditors and Evaluators
EvaluationAudit
Performance Audit
Performance Evaluation
Financial Audits
Attestation Audits
Non-Audit Services
Issue Analysis
Inspections
Reviews
A Multitude of Analysis Types
![Page 22: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/22.jpg)
Center for Public Program Evaluation
22
Other Professional DisciplinesEvaluators/Inspectors Come From Many Professions• Evaluation• Inspection• Audit• Social Science• Economics• Operations
Research Analysis
• Policy Analysis• Public Administration• Program Specialties
▫ Health Care▫ Housing▫ Defense▫ Trade▫ International Assistance
![Page 23: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/23.jpg)
Center for Public Program Evaluation
23
LegalAnalysis
Inspections
Economics
Evaluation
OperationsResearch
Applied Social
Research
Psychology
Statistics
Public Administration
![Page 24: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/24.jpg)
Center for Public Program Evaluation
24
LegalAnalysis
Inspections
Economics
Evaluation
OperationsResearch
Applied Social
Research
Psychology
Statistics
Public Administration
![Page 25: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/25.jpg)
Center for Public Program Evaluation
What About Investigators?
25
Evaluators Investigators?
![Page 26: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/26.jpg)
Center for Public Program Evaluation
Useful Interplay
26
Investigators
Evaluators
Fraud leads
Fraud trends
Tech
assist
![Page 27: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/27.jpg)
Center for Public Program Evaluation
Center for Public Program Evaluation
27
Evaluation Types and Approaches
![Page 28: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/28.jpg)
Center for Public Program Evaluation
Evaluation Types and Approaches
• There are many legitimate and useful forms of evaluations
• Each has its purpose. The methodological approach must be adapted to it
• Generally accepted professional guidelines are available to guide the work.
28
![Page 29: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/29.jpg)
Center for Public Program Evaluation
Center for Public Program Evaluation
29
As A Program MaturesThe Evaluation Questions Change
How is the implementation going?Any Issues?
Are service providers in compliance with program rules?
Are some service providers doing better than others?How so?
Are the required services being delivered?
Are impacted citizens or businesses satisfied with program results?
How do the services stack up to goals and standards?
Have conditions improved as intended?
![Page 30: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/30.jpg)
Center for Public Program Evaluation
Center for Public Program Evaluation
30
And So Do the Analytic Approaches and Methods Used
Some inspections and evaluations need to be done fast
Others need more time
Some require more resources than others
All must be done professionally
Following are some evaluation and inspections types commonly used today.
![Page 31: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/31.jpg)
Center for Public Program Evaluation
Center for Public Program Evaluation
31
Inspection/Evaluation Types• Compliance Review• Performance Review• Landscape Survey• Early Implementation
Reviews• Impact Assessment• Report Card• Process Review
![Page 32: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/32.jpg)
Center for Public Program Evaluation
Center for Public Program Evaluation
32
More Inspection/Evaluation Types
• “Just the Facts” Review• Report from “On the
Ground”• “Try it out” Reviews• Client Satisfaction Surveys• Before and After
Assessment• Control System Survey• Program Logic Model
![Page 33: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/33.jpg)
Center for Public Program Evaluation
Center for Public Program Evaluation
33
Analytic Frameworks
![Page 34: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/34.jpg)
Center for Public Program Evaluation
Center for Public Program Evaluation
34
The Standard IG Analytic Model•FINDINGS -- Criteria -- Condition -- Effect -- Cause
•RECOMMENDATIONS
![Page 35: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/35.jpg)
Center for Public Program Evaluation
What the Federal IG Blue Book Says
Center for Public Program Evaluation
35
“Findings often have been regarded as containing the elements of criteria, condition, effect, and, when problems are found, cause. However, the elements needed for a finding depend entirely on the objectives of the inspection. Thus, a finding or set of findings is complete to the extent that the inspection objectives are satisfied and the report clearly relates those objectives to the applicable elements of a finding.”
![Page 36: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/36.jpg)
Center for Public Program Evaluation
Two Analytic FrameworksPerformance Audit Performance Evaluation
• Findings▫ Criteria▫ Condition▫ Effect▫ Cause
• Evidence▫ Physical▫ Documentary▫ Testimonial▫ Analytic
• Recommendations
• Objective• Questions to Be
Answered• Analytic Approach
▫ Mixed Methods▫ Qualitative▫ Quantitative
• Findings• Recommendations
36
![Page 37: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/37.jpg)
Center for Public Program Evaluation
Center for Public Program Evaluation
37
METHODOLOGIES
![Page 38: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/38.jpg)
Center for Public Program Evaluation
Center for Public Program Evaluation
38
The Fundamentals• The Correct Methods to Use Are
the Ones Most Compatible With the Purpose of the Study
• Always Use More Than One Method▫ At least one quantitative▫ At least one qualitative
![Page 39: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/39.jpg)
Center for Public Program Evaluation
Center for Public Program Evaluation
39
Quantitative and Qualitative Methods• Quantitative methods involve things you can count
▫ Even if they are subjective answers to a closed ended questionnaire
• Qualitative methods are applied to the interpretation of statements▫ Especially from interviews▫ But also from observations and in documents
• Reviews are far more compelling if both methods are used▫ Otherwise people will say: “Numbers always lie” or “They always say that”
![Page 40: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/40.jpg)
Center for Public Program Evaluation
Center for Public Program Evaluation
40
The Most Common Methods Used by Evaluators
• Background Search• On-Site Inspections• Review of Administrative
Files• Sampling• Data Collection
Instruments• Surveys• Focus Groups
![Page 41: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/41.jpg)
Center for Public Program Evaluation
Center for Public Program Evaluation
41
Some Exercises
![Page 42: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/42.jpg)
Center for Public Program Evaluation
You Be the Evaluator• Early implementation of emergency
response systems after 9-11
• Frequency of and criteria for home visits to foster care families
• Pharmacies with extreme outlier filling of prescriptions for dangerous, addictive drugs
42
![Page 43: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/43.jpg)
Center for Public Program Evaluation
The Quick Start1. State the purpose
2. Ask two or three key questions 3. Choose the methods for each question
Quantitative Qualitative
4. Conduct the Field Work5. Analyze the data6. Construct Findings and Recommendations7. Draft the report
Center for Public Program Evaluation
43
![Page 44: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/44.jpg)
Center for Public Program Evaluation
Discussion1. Would your approach work as either an
audit or evaluation?
2. Who will use the findings and recommendations?
3. How will you know that the recommendations have been acted upon?
4. What can you expect will be the result?
44
![Page 45: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/45.jpg)
Center for Public Program Evaluation
Center for Public Program Evaluation
45
Organizing and Managing the Evaluation Function
![Page 46: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/46.jpg)
Center for Public Program Evaluation
Possible Organizational Location
46
Inspector General Evaluation
Audit Investigation Evaluation
Evaluation Evaluation
![Page 47: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/47.jpg)
Center for Public Program Evaluation
Managing an Evaluation Office
47
Work PlanningOperating Procedures
Quality ControlQuality Assurance
Professional Development
These are generally similar to
audit practices
Take Them Seriously !
![Page 48: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/48.jpg)
Center for Public Program Evaluation
Training Priorities for EvaluatorsHighest Priority Other Worthwhile Skills
• General Evaluation Concepts
• Writing to the Point
• Survey Design• Interviewing• Descriptive Statistics• Sampling• Spread Sheets• Graphs
• Focus Groups• Data Base Development• Randomized Control Trials
• Team Leadership• Programmatic Knowledge
48
![Page 49: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/49.jpg)
Center for Public Program Evaluation
Work Planning•Combine Audit and Evaluation•Focus on risk and return on investment•Use a Tactical Approach
▫ To systematically fill gaps▫ To develop staff capacity▫ To keep up the attention on important areas
49
![Page 50: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/50.jpg)
Center for Public Program Evaluation
Center for Public Program Evaluation
50
Program Area A
Program Area B
Program Area C
Program Area D
A strategic plan might consist of several families of program studies, some in the early stages, some further along, even in final stages
![Page 51: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/51.jpg)
Center for Public Program Evaluation
51
LegalAnalysis
Inspections
Economics
EvaluationAndAudit
OperationsResearch
Applied Social
Research
Psychology
Statistics
Public Administration
Listen to What They Have to Say
![Page 52: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/52.jpg)
Center for Public Program Evaluation
52
Good Luck!
![Page 53: Inspection, Evaluation, and Performance Auditing](https://reader035.fdocuments.in/reader035/viewer/2022081513/56815c5c550346895dca621e/html5/thumbnails/53.jpg)
Center for Public Program Evaluation
George F. Grob
Center for Public Program Evaluation540-454-2888
53