Alberta Reliability StandardsCompliance Working Group Meeting

Minutes are noted on the summary slide for each section

June 19, 2012

Agenda Topics

1. Introduction

2. Presentation of discussion plan, method and scope

3. Presentation on compliance monitoring audit process issues

4. Initial discussion on solutions

5. Meeting debrief

Plan

• Review feedback and observations obtained from previous year’s audit experience

• Provide input on possible solutions to identified process issues

• Timelines

• 1st meeting (today)

• CWG review issues

• Gather input from CWG

• 2nd Meeting – DRAFT (end July or early August)

• AESO presents solutions based on first meeting input

• CWG refines solutions

• 3rd Meeting – FINAL (End August or early September)

• AESO presents updated solutions as per 2rd meeting refinements

• 4th meeting if required (End September or early October)

• Publish guidelines – one month following final meeting

Method, Scope

Method•Go over the issues in order of the compliance monitoring audit process sections (guideline process map)

• As a group: discuss each issue focusing on: impact, broadness and provide input on possible solutions

•Feedback will be used by the AESO to update the audit guideline

Scope•Scope is limited to:

• The compliance monitoring audit process

• Existing, clearly defined issues.

• Existing constraints (legislation and mandate)

•Issues outside of the scope will be tabled and brought forward at a later date or other forum.

Process sections

Identified Compliance monitoring audit issues (13)(numbers 7.x refer to the audit guideline sections)

Process Section Issues

GEN Information / Guidelines 3

7.1 Publish audit schedule 0

7.2 Changes to audit schedule 1

7.3 Workshop /Notification 3

7.4 Pre-audit conference 0

7.5 Gather audit evidence 0

7.6 Submit audit evidence 2

Process Section Issues

7.7 Review for completeness 3

7.8 Assess audit evidence 0

7.9 Issue draft report 0

7.10 Post-audit conference 0

7.11 Review draft audit report 1

7.12 Issue final audit report 0

7.13-14 Referral to MSA and retain evidence 0

Defined IssuesGeneral: Information/Guidelines

1. Applicability Assessments

Participants/CMP

- Process change to reflect updated applicability sections

- Complete

2. Acknowledgement and Consent

Participants/CMP

- Simplification to meet business need

- End August

3. Guideline overly complex

Participants

- Need to define specific issues

- CWG Input

Defined IssuesAudit Guideline - 7.2: Change to audit schedule

4. Request for deadline extensions

CMP

- Requests for deadline extensions were unreasonable

- CWG Review (from December 2011 CWG)

Defined IssuesAudit Guideline - 7.3: Workshop / Notification

5. Lessons learned and timing

Participants

- When can lessons learned be shared?

- CWG Input

6. Publish generic index template

Participants

- Can a generic index be published in advance?

- Related to #9

7. Asset identification

Participants/CMP

- Need to identify an asset list prior to audit

- CWG Input

Defined IssuesAudit Guideline - 7.6: Submit audit evidence

8. Bad index of evidence files

CMP

- Difficulties in finding evidence

- Created standard index template (Attachment 4)

- Related to #9 (from December 2011 CWG)

9. Audit index VS RSAWs

Participants/CMP

- What method can be used for indexing evidence?

- CWG Input

Defined IssuesAudit Guideline - 7.7: Review for completeness

10. Initial review too detailed

CMP

- Some assessment was done during completeness review

- CWG Review (From December 2011 CWG)

11. How can we help entities better manage information requests?

Participants/CMP

- Should we proposed tentative timelines at this stage?

- CWG Input

12. Additional request content is sometimes unclear

Participants

- Requests are not always clear resulting in multiple follow-up

- CWG Input

Defined IssuesAudit Guideline - 7.11: Review draft audit report

13. Re-issue of draft report

CMP

- Extent of commenting on re-issued draft report is unclear

- CWG Input

Additional issue brought forward by a participant

NEW: Difficulties with overlap of self-certification and audit

- What are the consequences of self-certifying compliance if there is a subsequent draft audit report showing a suspected contravention?

- Discussion was held at the meeting with follow-up at the next.

Discussion on issuesWORKSHEETS

Each worksheet will show:

1.Title

2. Background

3. Current status

4. Possible constraints

After a quick review of the issue we will begin discussions:

1. Gather thoughts on impact and broadness

2. Discuss possible solutions

Discussion on issuesSUMMARY

At the end of the discussion, we will:

1. Write a summary of the main points for all to see

2. Validate that the summary contains all pertinent discussions

NOTE: It is important that we validate the summary as these points will be used to create the solutions that will be presented at the 2nd meeting.

General

3

Guideline overly complex

BackgroundDuring an end of compliance monitoring audit survey a market participant stated that the

guideline was not used because it is overly complex.

Current status“Overly complex” is unclear and we need input

Possible ConstraintsThe purpose of the guideline is to define the procedures for planning, executing, and

reporting with respect to each compliance monitoring audit. It has the dual purpose of providing policy/rules and procedures (“how to”). During a compliance monitoring audit one is more likely to simply look for the procedure which is currently mixed with the policy/rules. It is possible to change the guideline however, both elements must remain.

3Audit Guideline overly complex

Our thoughts

The guideline(s) could be changed as follows:

1.Separate policy/rules from the “How to”

2.Create a simplified “how to” section

3.Use FAQs for missed items

- Update annually with FAQ transfer to guideline

4.Roll-out update to all participants (newsletter)

5.Send to new participants

Group discussion… What are the issues, impact and broadness?

3Audit guideline overly complex

SUMMARY

The CWG:

•Agrees with the approach to split the “how to” and policy

•There is always room for improvement in the audit guidelines

Audit guideline - 7.2: Change to audit schedule

4 (Review)

Requests for deadline extensions were unreasonable

BackgroundSome MPs requested extensions of all or most deadlines and did not provide sufficient

reasons. Guideline states “reasons beyond the market participant’s control” which is not clear. Most reasons were related to internal scheduling or change of staff.

Current statusAs discussed at the December 2011 CWG:

• The guideline is looking for “exceptional” reason.

• Entity needs to build a case

• We will communicate intent through compliance audit training sessions

• We will change guideline to provide better guidance

Audit guideline - 7.3: Workshops / Notification

5

Lessons learned and timing

BackgroundWe have been asked by entities to share lessons learned. “Lessons learned” are shared

during our audit training sessions - however, this is only useful to scheduled participants which may not be timely for all.

Current status“Lessons learned” are provided during the audit training of a scheduled entity.

Possible Constraints“Lessons learned” cannot be identified in the audit report - under rule 12, only suspected

contraventions are identified.

5Lessons learned and timing

Our thoughtsAs discussed at the December 2011 CWG meeting:

•We have modified training to focus on market participant needs

•Emphasis is placed on key areas (index, pointing to evidence, use of attestations and relating evidence to requirements/measures (location within a file)

Further to this, we will:

•Include a “lessons learned” module at the Stakeholder Information Sessions

•Provide a review of expectations at RSAW training

• New: Ad hoc

• Old: Request stakeholder input on interest and timing

Group discussion… Does this work?

5Lessons learned and timing

SUMMARY

The CWG:

•Agrees that a module at the stakeholder information sessions for lessons learned on the general audit process would be valuable

•There is a need for RSAW sessions to go over auditor expectations: • Start with statistically problematic

• When possible group by standards areas

• Possibly include AESO technical SME at the session

• Suggest once a year for problem standards

Audit guideline - 7.3: Workshops / Notification

6

Publish generic index template

BackgroundEntities are requested to fill out the index template when submitting audit evidence. The

index is sent out as an attachment to the notification letters. Some entities requested that a copy of the index template be published well in advance of the audit.

Current statusWe provide a copy filtered by entity type as an attachment to the notification letter. A

generic (non-filtered) copy is provided upon request.

Possible ConstraintsThis issue is related to #8 (difficulties for auditors in finding evidence) and with #9 (use of

RSAWs or Index)

6Publish generic index template

Our thoughtsWe have no issues with publishing a generic index – however, since the need for this depends on our decision to use RSAWs or not, we propose that 6, 8 and 9 be discussed as one issue.

Group discussion… Does this work?

6Publish generic index template

SUMMARY

The CWG:

•Agreed to discuss this issue with issue #9

Audit guideline - 7.3: Workshops / Notification

7

Need to identify an asset list prior to audit (Can extend to relationships and other information)

BackgroundThe need for information depends on individual standard requirements and on entity structure. As an example, a participant requested that we provide a list of assets that will be included in the audit.

Current statusInformation is requested and obtained during the audit.

Possible ConstraintsSince the need for information varies by standard and by entity structure, requests for lists cannot be generic. As an example, an entity may have different CIP procedures for different plants within their organisation – in such a case, a list of operating personnel is required for each plant (the asset) so that the yearly provision of the procedure can be verified.

7Need to identify an asset list prior to audit?

Our thoughtsLists are not limited strictly to assets. The lists are required to perform certain assessments. Requests depend on individual standard requirements and entity structure which may include but is not limited to:

•Functional entity type (who owns, who operates and who performs PP function)

•Asset list to verify inclusion or exclusion

Possible solutions:

•Introduce pre-audit questionnaire – similar concept to WECC.

•More information could be requested in the RSAW.

Group discussion… Impact (timing), broadness, alternatives?

7Need to identify an asset list prior to audit?

SUMMARY

The CWG:

•Is in favour of using a pre-audit questionnaire

•There will be a possible need for training

Audit guideline - 7.6: Submit audit evidence

8

Bad index of evidence files

BackgroundBased on the 2011 Q2 and Q3 audit experience, it was found that the suggested index

format as described in appendix B of the audit guideline, did not provide clear guidelines on what was required. Further to this, entities were allowed to submit a variety of index styles. Having a variety of index structures led to difficulties for auditors to find the actual evidence files and prolonged the audit schedule by necessitating requests for clarifications and re-submittal of additional evidence.

Current statusAs per the December 2011 CWG meeting, we now provide a index template filtered by entity

type as an attachment to the notification letter. A generic (non-filtered) copy is provided upon request.

Possible ConstraintsThe auditors need the entity to point to the evidence document/page for each applicable

requirement.

8Bad index of evidence files

Our thoughtsSince the need for an index depends on our decision to use RSAWs or not, we propose that 6, 8 and 9 be discussed as one issue.

Group discussion… Does this work?

8Bad index of evidence files

SUMMARY

The CWG:

•Agreed to discuss this issue with issue #9

Audit guideline - 7.6: Submit audit evidence

9

Audit index VS use of RSAWs

BackgroundIssue numbers 6 and 8 outlined the difficulties that the auditors were having in finding

evidence documents because of bad indexes. Entities suggested that perhaps the RSAW could be submitted. A pilot project enabling submission of both the RSAW and index (as back-up) was provided to requesting entities for 2012 Q1 and Q2.

Current statusAs per the December 2011 CWG meeting, we now provide an index template filtered by

entity type as an attachment to the notification letter. A generic (non-filtered) copy is provided upon request. As a pilot project, RSAW submission with index back-up is permitted when requested.

Possible ConstraintsThe intent is that the index or RSAW be used to point the auditor to the evidence

document/page related to the requirement/measure of the standard. For efficiency, only one method should be used.

9Audit Index VS use of RSAWs

Our thoughtsThe pilot project of using RSAWs was quite successful.

•There were no major difficulties in finding evidence.

•Most submitted RSAWs used imbedded document links which proved to be quite efficient.

• In one case the link did not work however the document was found using the index back-up. Alternatively an additional request could have been made.

•The “Description” column was sometime used to give or state evidence. The intent of the column is to describe the evidence (such as paragraph or page number).

•The purpose of using the RSAW should be the same as the index – to point the auditor directly to the evidence. The advantage of the RSAW is that the requirement, measure and assessment step are all aligned in one document and page row.

Group discussion… Index VS RSAW? Impact and broadness?

9Audit Index VS use of RSAWs

SUMMARY

The CWG:

•Agrees with using the RSAW instead of the index template

•Notes:• Language in RSAW could be clearer

• Separate the request from auditor functions

o Side note: New RSAW are formatted this way

Audit guideline - 7.7: Review of completeness

10 (Review)

Initial review too detailed

Background(Internal issue) The completeness review went beyond a simple check that all documents

were received and usable. Some assessment was being performed during the completeness review and then repeated in more detail during the assessment phase. Some entities were confused by the similar nature of requests during the completeness review and assessment phases.

Current statusAs discussed at the December 2011 CWG:

• The guideline will be changed to state “Administrative review” so as to align with language used in self-certification

• We will limit the “administrative review” to the intended checks

• We will better communicate the intent through training workshops

Audit guideline - 7.7: Review of completeness

11

How can we help entities better manage information requests?

BackgroundSome participants have asked us to give them an idea of when to expect additional

information requests in order to better plan SME resources during the audit assessment phase.

Current statusAlthough auditors try to provide an estimate of when additional requests will be available, it

is not part of the process to provide an audit assessment schedule

Possible ConstraintsInformation request timelines may depend on volume of material which is not known until

the initial “administrative review” is completed. Once the volume is known, a tentative schedule of when additional requests for each standard type is possible – however, there are too many variables to be able to determine in advance when the draft or final report will be completed.

11Need to establish an additional request schedule

Our thoughts•Establishing a schedule of when information requests will help entities and us to better prepare internal resources

•Any schedule would be on a tentative basis and could not be guaranteed

Possible inclusions in process:

•After the “administrative review”, the AESO will provide a tentative schedule of when the entity can expect information requests.

•The entity would provide any possible constraints

•The AESO will make “best effort” to accommodate entity needs.

Group discussion… Work for you? Impact and broadness?

11Need to establish an additional request schedule

SUMMARY

The CWG:

•Agrees with the approach of providing a schedule for additional requests:

• Schedule is a best guess

• Group by standard area

•Possibly update tentative schedule on known changesEx: Where it conflicts with the constraints

Audit guideline - 7.7: Review of completeness

12

Additional request content is sometimes unclear

BackgroundA participant stated that additional requests are not always clear and require additional

follow-up to be understood. This leads to difficulty in tracking requests, confusion and inefficiencies. Further to this it is not clear how an entity can request clarifications.

Current statusProcess is not sufficiently detailed.

Possible ConstraintsRequests are made so that the auditor is satisfied that all available evidence has been

received. Multiple requests may still be necessary.

12Additional request process not efficient

Our thoughtsEstablishing a more detailed process may help clarify request content.

•Auditors will use a template for requests

•Template to include all key information

• Standard / Requirement

• Background of request and any related documents

•Entities to use the template to respond

• Space provided for response

• Space provided to attach documents

Group discussion… Work for you? Impact and broadness?

12Additional request process not efficient

SUMMARY

The CWG:

•Agrees that using a template is good•Entity should communicate any need for clarifications and the AESO will re-word the request if required.

Audit guideline - 7.11: Review draft audit report

13

Extent of commenting on re-issued draft report

BackgroundThe guideline allows for re-issue of the draft report: “In the event a re-evaluation of

previously submitted audited evidence is deemed appropriate, and results in revised audit findings or significant changes to the draft audit report, a new draft audit report will be issued and the registered market participant will be provided ten business days to review and submit their comments in writing to the compliance monitor.” It is unclear however as to what extent comments may be provided.

Current statusNo changes have been made to section 7.11 of the guideline.

Possible ConstraintsNone identified

13Extent of commenting on Re-issued draft report

Our thoughtsComments on a re-issued draft report should be limited to items that have changed. The guideline wording would be changed to:

“In the event a re-evaluation of previously submitted audited evidence is deemed appropriate, and results in revised audit findings or significant changes to the draft audit report, a new draft audit report will be issued and the registered market participant will be provided ten business days to review and submit their comments on the changes in writing to the compliance monitor.”

Group discussion… Work for you? Impact and broadness?

13Extent of commenting on Re-issued draft report

SUMMARY

The CWG:

•Agrees with the approach of adding “on the changes” to this section of the audit guideline.

Next steps

• We will send power point summaries to all CWG members

• Create solutions based on today’s results

• Set next CWG meeting

• End July / Early August

• Date: Will send out survey to members on possible dates

• Time: Afternoon, mid-week is good.

• Purpose

• AESO to present solutions based on first meeting input

• CWG to refine

Thank You

Any questions or comments?