NSAA Peer Review Program: What to Know Before You Go! National State Auditors Association January...
-
Upload
juliet-garrett -
Category
Documents
-
view
225 -
download
0
Transcript of NSAA Peer Review Program: What to Know Before You Go! National State Auditors Association January...
NSAA Peer Review Program:What to Know Before You Go!National State Auditors AssociationJanuary 21, 2015
1
2
Opening Remarks
MODERATORR. Kinney PoynterExecutive Director,
NASACT
SPEAKERJohn Buyce
Director of State Audits,Office of the State Comptroller (NY)
SPEAKERWilliam (Brad) Blake
Chief Auditor, Center for Audit Excellence, Office of State Auditor (OH)
SPEAKERGreg Fugate
Performance Audit Manager, Office of the State Auditor (CO)
SPEAKERStaci Henshaw
Deputy Auditor, Virginia Office of the Auditor of Public Accounts (VA)
Agenda
• Overview, Objectives, and General Considerations
• Organization and Qualifications of the Review Team
• Performing Peer Reviews
• Finalizing the Peer Review
• Tips & Insights
3
Overview
• Purpose of peer review program – to provide independent assessments of state audit organizations to determine whether they have an adequately designed internal quality control system and are in compliance with that system
5
Overview
• Yellow Book began requiring peer reviews in 1989
• NSAA program started in 1985
• Currently, 54 eligible organizations participate in the program
6
Overview – Role of Committee
• Peer Review Committee has overall responsibility for the program– Provide guidance in the form of policies
and procedures– Resolve potential disputes that may arise
in the review process and ensure consistency of reviews
– Coordinate with other groups to ensure the adequacy of the review process
7
Overview – Role of NASACT
• Administer the review process on a daily basis
• Coordinate and assign review teams
• Train review team members
• Update policies and procedures for review by the Peer Review Committee
8
Objectives of the Review
• Evaluate whether the State Audit Organization’s system of quality control is…– suitably designed, adequately
documented and communicated; and– sufficiently complied with to provide
reasonable assurance of compliance with government auditing standards
9
Objectives of the Review
• Upon completion of the peer review, the peer review team issues a report
• Types of peer review reports– Pass– Pass with deficiencies– Fail
10
General Considerations
• Maintain confidentiality of state’s working papers and peer review work papers
• Be independent of state audit organization being reviewed
• Possess adequate knowledge and proficiency to perform review
11
Organization of Review Team
• Concurring reviewer– Most senior member of team
• Team leader– Overall responsibility for planning and
performing the review
• Team member– Responsible for performing tasks
assigned by team leader
13
Organization of Review Team
• Team leader and concurring reviewer identified 4-5 months prior to the actual date of the review
• Team members identified 2-3 months prior to the actual date of the review
14
Qualifications – Concurring Reviewer
• Must have served as team leader on a least one review under NSAA’s program
• Must have been recommended by former concurring reviewer
15
Qualifications – Team Leader
• Must have served as team member on a least one review under NSAA’s program
• Must have been recommended by former team leader and concurring reviewer
16
Qualifications – Team Member
• Must be recommended by audit organization head
• Must be in a supervisory or managerial role
• Must review audit documentation as part of job responsibilities
• Minimum of three years in supervisory role
17
Qualifications for Review Team Member – Other Issues
• Ensuring a proper match– NASACT tailors team skills and
experience to the SAO’s needs• Type of work performed (Financial,
Performance, Attest)• Size of the SAO and number of audits• Government Audit Quality Center (AICPA-
GACQ) membership• Federal reviewer for Single Audit
• Usually up to six reviewers selected
18
Preliminary Phase
• Understanding the Peer Review Program
• Obtaining/reviewing necessary SAO info including prior work papers
• Determining scope of review
• Sending audit staff questionnaire
• Selecting engagements
• Finalizing planning
21
Documents Used in a Peer Review
• Audit Organization Questionnaire
• Audit Staff Questionnaire
• Audit Organization’s Policies and Procedures and Review Guide
• Guide for Review of Audit Engagements
• Matters for Further Consideration (MFC)
• Findings for Further Consideration (FFC)
• Conclusions
22
Team Preparation: Understanding the Review Process
• Ensure each team member obtains an understanding of the process – NSAA External Peer Review Manual– Questions and Answers for All Team
Members – Administrative Matters
• Timing of review procedures• Travel arrangements• General expectations
23
Team Leaders: Review of Prior Work Papers
• Issues identified in prior peer review report– Prior peer review rating
• Read notes from exit conference
• Review other issues identified (MFCs and FFCs)
24
Audit Organization Questionnaire
• Completed by the SAO and provided to the Team Leader approximately 12 weeks prior to the review
• Provides the population of audits and audit hours within the review period
• Provides information about the state office’s staff, structure, and quality control system
25
Determining the Scope of the Review
• Determine whether the SAO performs– Financial Audits– Attestation Engagements– Performance Audits
• Engagements the SAO has stated to have been performed in accordance with government auditing standards are subject to the review
26
Determining the Scope of the Review
• Review should cover a current period of one year to be mutually agreed upon by the SAO and the review team
• Covers the quality control policies and procedures in effect and compliance for the year under review
• Includes reports issued during the year under review, or immediately thereafter if the work was substantially completed during the period
27
Determining the Scope of the Review
• The scope of the review does not cover– Non-GAGAS engagements– Audits done by others (e.g., contracted)
– Audit efficiency– State statute compliance– Administrative aspects, unless directly
required to satisfy applicable professional standards (e.g., Independence, CPE)
28
Other Administrative Matters
• Team Leader – Work Program– Prepare and send the engagement letter – Work with the SAO to establish deadlines
for receiving key documents– Initial communication with Concurring
Reviewer and Team Members
• NASACT – Get signed contract to Team Leader
29
Pre-Visit Work
• Team Leader– Select Audit Engagements for Review– Audit Staff Questionnaire– Preliminary Site Visit– Policy & Procedure Review– Finalizing the Review Plan– Administrative Logistics
30
Selecting Audit Engagements for Review
• Based on listing provided by the SAO• Factors to consider include:
– Spread across various supervisors, regions, etc.
– Spread across the review period– Different types of engagements (Cash
Basis vs GAAP, Single Audits, CAFRs, Performance, Attest)
• Select one “surprise” audit
31
Other Sample Selections
• Select a sample of audit staff to complete the Audit Staff Questionnaire
• While onsite, select a sample of audit staff for verification of CPE records and independence documentation
32
Audit Staff Questionnaire
• Work with the SAO to administer the questionnaire about 8 weeks prior to the review– Responses submitted directly to the Team
Leader about 6 weeks before the review
• Team Leader compiles the responses and communicates a summary to the team
• Evaluate responses for potential areas to concentrate review efforts
33
Preliminary Site Visit and Other Considerations
• Team Leader consults with the Concurring Reviewer to determine if a preliminary site visit is needed– Very rare– If one occurs, the preliminary site visit is
4-6 weeks in advance
• Team Leader has option to arrive on site at the SAO a few days prior to the rest of the team members
34
Team Members: Evaluation of the SAO’s Policies and Procedures
• Policies and Procedures Checklist – Separate checklists for financial audits,
performance audits, and attestation engagements
– Completed by the SAO and provided to the Team Leader along with the SAO’s policy and procedure manual, usually about 8 weeks prior to the review
35
Policies and Procedures Checklist
• Team Leader assigns each team member a section or sections of the policies and procedures to review
• Assignments are made to the team members approximately 8 weeks prior to the review
• Team Members are responsible for completing this work before arriving on site for the review
37
Finalizing the Review Plan
• Team Leader discusses plan for the review with the Concurring Reviewer
• Team Leader informs the SAO about sampled audit engagements, usually about 4-6 weeks prior to the review
• Team Leader only informs the SAO about the sampled “surprise audit” when the team arrives on site
38
Sampled Audits/Engagements
• Team Leader assigns each team member one or more audits/ engagements to review
• Team Members are responsible for completing some work before arriving on site for the review– Download and review reports– Complete reporting sections of the Audit/
Engagement Review Checklist
39
Administrative Logistics
• Team Leader will communicate about administrative logistics in the 1-2 weeks leading up to the review– Confirm travel, car, and hotel– Set time/place for initial meeting– Work plan/schedule for the review– Dress code and other information
• If something is unclear, be sure to ask!
41
Field Work Phase
• Complete the study and evaluation of QC policies and procedures
• Review compliance with policies and procedures
• Identify matters, findings, deficiencies, and significant deficiencies
• Aggregate and evaluate matters• Form conclusion on type of report to issue• Communicate conclusions at exit conference
42
Overall Fieldwork Schedule
• Five Day Reviews– Generally run Monday - Friday
• Ten Day Reviews– May run Monday to Wednesday or
Wednesday to Friday– Allows a few more days for review of
sampled audits
43
Arrival – Day Before the Review Begins
• Team Leaders often hold a team meeting at the hotel in the evening– Answer any questions, logistics, etc.– Results of policy & procedure reviews
• Team Members– Discuss policies noted in their assigned
area and their reviews of reports
• Some teams do a conference call during the week before the review
46
Day 1 – Let’s Get Started
• Entrance Conference
• Compliance testing for Policies & Procedures
• Get oriented to the audit workpapers, software, network, etc.
• Begin detailed review of first sampled audits
47
Team Leader – Day 1
• Run the Entrance Conference
• Discuss any outstanding issues with SAO officials
• Compliance testing for P&P
• Get organized
• Assist Team Members in starting their audit reviews
• Status meeting at day’s end
48
Team Members – Audit/Engagement Reviews
• Standard review guide for each type of audit/engagement– Financial, Performance, Attest
– Covers all phases of the audit/engagement from planning through reporting
– Cross referenced to individual audit standards, AU citations, etc.
• Use to test compliance with procedures by teams
49
Audit/Engagement Reviews
• Complete checklist for each audit/engagement– Seek clarification from audit teams– Start your second review if waiting for
information from others
• “NO” answers generate an MFC – given to Team Leader
51
MFC Form
• “Matters” generally will originate from “no” answers on checklist (either with design of, or compliance with, system of quality control)
• Can be cleared, discussed verbally with the audit organization, or carried forward to the Conclusions document
53
Day 2 – More Audit Reviews
• Team Leader– Answer questions, provide guidance, ensure
consistency– Organize MFCs, review completed work,
test items to workpapers– Get ready for the Concurring Reviewer to
arrive• Team Members
– Discuss issues with audit teams – In most cases, you should be on your
second audit in the afternoon
54
Day 3 – Start Wrapping it up
• Team Leader– Brief the Concurring Reviewer on
progress and issues– Continue organizing and grouping MFCs– Assist Team Members in completing
audit reviews
55
Day 3 – Start Wrapping it up
• Team Members– Continue completing review guides and
MFCs– Second audits should be all but done– If there’s a third, it should be well
underway in the afternoon• Prioritize the areas you want to review• Key on planning, execution, documentation
and common issues
56
Day 3 – Start Wrapping it up
• End of the Day– Status meeting
• Initial discussion of issues that may warrant additional work or be potential Findings
– Should have a sense of the overall rating
• Plan out work that needs to be completed and when
– Reassign work and assist each other as necessary
57
Day 4 – Pull it all together
• Team Leader– Organize MFCs for discussion– Manage completion of reviews by mid-
day– Lead team discussion to come to a
consensus on disposition of MFCs, Findings and rating
• Contact and consult with NSAA if necessary
58
Day 4 – Pull it all together
• Team Members– Complete all review guides and MFCs– Assist others to tie up loose ends
• Afternoon – Team Discussion– Review all MFCs and discuss the ones
you prepared– Work to consensus on items to elevate to
Findings based on significance and pervasiveness
59
Day 4 – End of the Day
• Decide on form of report and issues to discuss at exit conference– Complete the relevant Conclusion
documents and finalize a draft of the report
– Consult with NSAA as appropriate if report rating is other than “pass”
• Brief officials about the general conclusions of the report and any findings
60
How do you decide what type of report to issue?
• Based on professional judgment considering nature, causes, pattern and pervasiveness of matters and their relative importance to the organization’s system of quality control taken as a whole
61
Forms Used in Documenting Results
• Matters for Further Consideration (MFC)
• Conclusions Document
• Findings for Further Consideration (FFC)
62
Reporting Matrix
• Section II of the Peer Review Manual includes a reporting matrix to provide guidance on various reporting considerations when evaluating results
63
Pass
• Audit organization’s system of quality control has been suitably designed and complied with to provide the audit organization with reasonable assurance of performing and reporting in conformity with applicable professional standards in all material respects
66
Pass
• Design– Adequate, or inadequate for parts of one
or more standards
• Compliance– Sufficient, or insufficient for parts of one or
more standards
• Severity – Insignificant or Moderate
• Frequency – Isolated, but considered a Finding if recurring and pervasive
67
Pass with Deficiencies
• Audit organization’s system of quality control has been suitably designed and complied with to provide the audit organization with reasonable assurance of performing and reporting in conformity with applicable professional standards in all material respects with the exception of a certain deficiency or deficiencies that are described in the report
69
Pass with Deficiencies
• Design– Adequate overall, but inadequate for
substantially all of one standard or parts of several
• Compliance– Sufficient overall, but insufficient for one
standard or parts of several
• Severity – Serious• Frequency – Recurring & Pervasive
70
Fail
• Based on significant deficiencies described in report, audit organization’s system of quality control is not suitably designed to provide the audit organization with reasonable assurance of performing and reporting in conformity with applicable professional standards in all material respects
71
Fail
• Or based on significant deficiencies described in report, audit organization has not complied with its system of quality control to provide the audit organization with reasonable assurance of performing and reporting in conformity with applicable professional standards in all material respects
72
Fail
• Design– Inadequate to provide reasonable
assurance – several standards
• Compliance– Insufficient overall – several standards
• Severity – Severe
• Frequency – Recurring & Pervasive
73
Conclusions Document
• Used to determine the appropriate reporting for matters– FFC Form– Deficiency– Significant Deficiency
74
Findings for Further Consideration Form (FFC)
• Used to report findings not rising to the level of a deficiency or significant deficiency
• This becomes part of the working papers, but not part of the reporting process
76
Completion Phase
• Finalize the peer review report
• If rating is other than pass, audit organization should respond
78
Communication
• A successful review depends on good communication at all levels before and during the review
• Avoid making assumptions
• Ask questions
• Be responsive when asked a question or information is requested
80
Preparedness
• Understand the peer review process
• Establish and adhere to deadlines for pre-review and on-site work
• Review the state office’s website and work products
81
Audit Reviews – Planning, Execution and Documentation
• Work from the end inward (documentation)– Examine the report and supporting work
• Also work from the start (planning)– Look at the application of general
standards and the planning
• Meet in the middle (execution)– Did they follow procedures? – Do what they planned? – Properly report what they found?
82
Learn Something
• There is more than one way to comply with auditing standards
• The peer review process should be value-added– What can the state being reviewed learn
from the review team?– What can the review team learn from the
state being reviewed?
83
How can you get involved?
• Must have at least three years of supervisory experience
• Complete Team Member Qualifications form online
https://www2.nasact.org/QCR/2015_Team_Member_Qualifications.asp
– Info on your background, experience and the types of audits you supervise, as well as you availability and preferences for where you want to go
• Must update form each year• Currently about 300 participants
84
Don’t Get Discouraged
• Factors can limit your chance of selection – especially the first time– Number of reviews scheduled – Type of audits those states perform– Usually only one person per state on a
team– Usually no more than one first-time
reviewer– Your state’s balance in time credits
85
Any Other Questions?
Talk to your office’s peer review liaison or contact the Peer Review Coordinator, Kathleen Young, at [email protected] or (859) 276-1147
87
Questions
MODERATORR. Kinney PoynterExecutive Director,
NASACT
SPEAKERJohn Buyce
Director of State Audits,Office of the State Comptroller (NY)
SPEAKERWilliam (Brad) Blake
Chief Auditor, Center for Audit Excellence, Office of State Auditor (OH)
SPEAKERGreg Fugate
Performance Audit Manager, Office of the State Auditor (CO)
SPEAKERStaci Henshaw
Deputy Auditor, Virginia Office of the Auditor of Public Accounts (VA)