(The ERB)

2012 NASBA Regional Meetings

ERB is an independent body of NASBA responsible for providing annual assurance that the CPA examinations can be relied upon to meet state boards of accountancy statutory requirements.

The CPA Examination Review Board shall review, evaluate and report on the appropriateness of the policies and procedures utilized in the preparation, grading and administration of the Uniform CPA Examination and other examinations in general use by boards of accountancy for the licensing of certified public accountants; examine such records, and make such observations, inspections and inquiries as it deems necessary; and report annually to the Boards of Accountancy.

The ERB is composed of 5 rotating volunteers with various backgrounds principally in auditing. The ERB is assisted by two paid consultants in technology and psychometrics. There is also a dedicated three person NASBA staff to perform detailed procedures, document procedures, and carry out many of the administrative and procedural functions.

o  Sandra Wilson, CPA, CFE, CFFA, Chair o  Nick Mastracchio, CPA, Ph.D., Vice Chair o  Wes Johnson, CPA, Past Chair o  Charles (Chuck) Talbert, CPA o  David Vaudt, CPA

ERB members are appointed for three year terms by the NASBA Chair.

Steve Downing, Ph.D., Psychometrician Michael Harnish, CPA, CITP,CISA, CDP, EnCE, IT Specialist

Staff Toerien DeWit, CPA, Director of Operations Onita Porter, CPA, Audit Manager Shawn Jackson, CPA, CISA, Senior IT Auditor

The ERB was established in 1975 as a joint venture of NASBA and the AICPA. In 1977 it became an “independent” body of the NASBA Board of Directors. In 1993, the charge to the ERB was broadened to include the review and report on the International Qualifying Examination (IQEX) and any other examination provided by State Board of Accountancy. In 2010, the ERB was reorganized from 12 volunteers to the current 5 volunteer members and staff was expanded.

The primary objective of our review and evaluations is to express assurance to Boards of Accountancy: }  as to the presence and effectiveness of

internal controls over test development, administration and scoring

}  as to whether appropriate and adequate validity evidence exists in support of the examinations

Our program of work is developed from and follows the “Twelve Steps for Effective Test Development’ (Downing, Haladyna, 2006) which is based on the “Standards for Educational and Psychological Testing” ( AERA, APA, & NCME, 1999).

1.  Overall plan 2.  Content definition (practice analysis) 3.  Test specifications; blueprinting the test

(content specifications and skill specifications) 4.  Item development 5.  Test design and assembly 6.  Test production 7.  Test administration 8.  Scoring test responses 9.  Establishing passing scores (standard setting) 10.  Reporting test results 11.  Item banking 12.  Test technical reports

Procedures include inquiries, observations, walkthroughs and document inspections of the following: ◦  Internal controls over test development,

administration and scoring processes ◦  Validity evidence for the examinations ◦  Other procedures we consider necessary in the

circumstances

}  Scoring errors }  Launch of the CBTe }  Performance of new content items }  Pre-operational items }  Item errors

}  Access to Information Technology (IT) documentation

}  Business continuity and disaster recovery }  Change management policies }  New and updated proprietary software }  Data privacy }  Overall IT infrastructure

Our procedures include a review of the following: ◦  AICPA Internal Audit ◦  AUP performed by Withum Smith & Brown ◦  SOC2 Type 2 report on Business & Decision North

America (PA), Inc.

}  Board of Examiners (BOE) meetings }  Content Committee meetings }  Contract Steering Group meetings }  Content Preparation Subcommittee meetings }  Psychometric Oversight Committee (POC)

meetings }  Task Based Simulations (TBS) development

workshop }  TBS Internal Roundtable meetings }  International Qualification Examination (IQEX)

Committee meetings }  State Board Committee meetings

}  Procedures at Prometric are directed to the testing steps associated with Prometric’s involvement in the examinations: ◦  Test administration ◦  Item banks

}  We examine period specific issues: ◦  Launch of the CBTe ◦  International delivery of the Uniform CPA

Examination }  Our procedures include inquiries,

observations, walkthroughs and inspection of documents

}  Access to Information Technology (IT) documentation

}  Business continuity and disaster recovery }  Change management policies }  New and updated proprietary software }  IT infrastructure security }  Metrics proformance

}  The Exam Review Board visited 52 domestic testing centers

}  ERB visited 6 of the international testing centers

}  ERB staff viewed DVR tapings of the international testing centers prior to opening

}  Relied upon the reporting of State Boards of Accountancy site visits

}  Procedures at NASBA are directed to the testing steps associated with NASBA’s involvement in the examinations: ◦  Test administration ◦  Reporting of test results

}  We examine period specific issues: ◦  Launch of the CBTe ◦  Sale of PCS ◦  Candidate access to the National Candidate

Database International delivery of the Uniform CPA Examination ◦  Reorganization of NASBA’s IT Department

}  Access to Information Technology (IT) documentation

}  Business continuity and disaster recovery }  Change management policies }  New and updated proprietary software }  IT infrastructure security

}  We followed the same general planning and review process with the three parties:

◦  ERB planning meeting for upcoming year (August) ◦  Review and approved work programs (September) ◦  Provided written general plan for reviewing, evaluating

and reporting (October) ◦  Held organization planning meetings (November) ◦  Provided document request lists (December) ◦  Performed interim procedures and walkthroughs

(December) ◦  Performed ‘fieldwork’ procedures (January-February)

◦  Completed and reviewed the work programs (March) ◦  Provided parties with draft findings for

management letter and exit conference letter (March) ◦  Held exit conferences (April) ◦  Received responses to management letter items

(May) ◦  Finalized management letter (May) ◦  Approved the report to Boards of Accountancy (May) ◦  Reported to Boards of Accountancy at Regional

meetings (June)

}  In 2011 time spent: ◦  Volunteer members 189 days ◦  Staff 700 days ◦  Consultants 44 days

}  Or ◦  ERB meetings 224 days ◦  AICPA events 267 days ◦  Prometric events 250 days ◦  NASBA events 192 days

}  For a grand total of 933 days

To produce a ten minute video that explains what the ERB does in an informative and hopefully enjoyable format.

Thank you