eManifest Review Review Engagement

December 2012

1

TABLE OF CONTENTS

1.0 INTRODUCTION ......................................................................................... 2

2.0 SIGNIFICANCE OF REVIEW ................................................................... 2

3.0 STATEMENT OF CONFORMANCE ........................................................ 2

4.0 KEY FINDINGS ............................................................................................ 3

5.0 SUMMARY OF RECOMMENDATIONS ................................................. 4

6.0 OVERALL MANAGEMENT RESPONSE ................................................ 4

7.0 FINDINGS ...................................................................................................... 5

7.1 Scope and user requirements ................................................................................................. 5 7.2 Project Plan ........................................................................................................................... 8 7.3 Cost Management ................................................................................................................ 10

7.4 Project Status Reporting ...................................................................................................... 12

APPENDIX A – ABOUT THE REVIEW ........................................................... 15

APPENDIX B – GLOSSARY ............................................................................... 17

APPENDIX C – ACRONYMS ............................................................................. 18

2

1.0 INTRODUCTION eManifest is a major crown project underway at the Canada Border Services Agency (CBSA). Once

fully implemented, eManifest will change the commercial import process. It will require members

of the trade community such as carriers, freight forwarders, and importers in all transportation

modes to electronically transmit data on cargo, conveyance and crew to the CBSA before shipments

arrive at the border.

The Internal Audit Division completed a review of the eManifest project which included controls

over the project scope, project management practices, cost management processes and project status

reporting. The review included interviews and examination of project documentation that took place

during the period from January 1, 2012 to August 30, 2012.

2.0 SIGNIFICANCE OF REVIEW

This review is significant because large government Information Technology (IT) projects like

eManifest are inherently complex, expensive and risky, and have had a history of cost overruns and

delays, and not delivering what had been planned. This review of eManifest will provide CBSA

management with information on the adequacy of its project management practices, in general and

for the eManifest project.

OBJECTIVE

The objective of this review was to determine the extent to which eManifest will meet the needs of

end users; the adequacy of the project plan with respect to managing the project’s scope, schedule

and resources; the management of costs; and the effectiveness of tools and processes for status

reporting.

3.0 STATEMENT OF CONFORMANCE

This review conforms to the Internal Auditing Standards for the Government of Canada, as

supported by the results of the quality assurance and improvement program. The approach and

methodology for this review followed the International Standards for the Professional Practice of

Internal Auditing as defined by the Institute of Internal Auditors and the Internal Auditing

Standards for the Government of Canada as required by the Treasury Board Internal Audit Policy.

This review was intended to provide a low level of assurance commensurate with a review.

3

4.0 KEY FINDINGS

Consultations are occurring with the future end users of eManifest. This is being done through

project working groups and governance committees. Some end users have expressed an interest in

having improved discussion on business requirements and feedback on decisions taken. Project

management understands the importance of good stakeholder consultations and considers this a low

risk in the overall context of the project.

The eManifest project has performed multiple types of testing on previous releases. However, at the

CBSA, standards for test plans have not been developed and final acceptance testing does not

include end users. Test results have not been shared with stakeholders to allow for a more informed

review and evaluation of the system as implemented. Ineffective acceptance testing could lead to

issues and change requests after the system is in the production environment.

The trade-off between project efficiency and end user involvement presents risks that need to be

considered and weighed. For example, the initial release of Build 1 for March 31, 2013 could be at

risk. The preliminary estimated effort to conduct testing was greater than the time available. If this

becomes an issue, management will have to weigh the trade-offs before making a decision. Project

governance committees at all levels are in place to facilitate the decision-making process.

For project planning and monitoring, management does not have a complete set of project

management information to monitor progress with respect to time, scope and cost. Each individual

manager is responsible for managing and reporting monthly on their piece of the project. There is

reliance on this reporting process and confidence that issues and risks will be identified and

addressed in a timely and effective fashion.

Earned value management (EVM) is a process that provides management with information on

investment performance and necessary data points to estimate probable completion costs and dates.

As part of the Agency’s cost management plan for eManifest, it had agreed to use EVM to measure

performance. While this work was initiated, it was discontinued due to the complexity and lack of

agency tools for the EVM process.

Multi-year projects are expected to forecast costs based on a costing template that aligns with the

Treasury Board Submission. eManifest is not following this process. Project managers are

negotiating their budget requirements with the eManifest project office without knowing the cost of

the detailed project plans. Cost forecasts to complete eManifest are based on the original Treasury

Board submission and workplans developed by project managers.

Comptrollership Branch has indicated that it is unable to provide reasonable assurance on the

project’s financial health because project costs and forecasts are not done by deliverable.

Comptrollership does monitor and discuss actual cost information with project management to

ensure it is consistent with corporate information.

Project management is reporting regularly on the status of the project. Clarification is required on

the information that should be reported, including the completeness of some of the dashboard

4

elements. This includes reporting by deliverable, rather than project phase. Oversight on the

dashboard could be strengthened to ensure all significant issues from CBSA stakeholders are

reported. The risk is that CBSA executive management may not be informed of significant issues

that could have broader implications for operations.

5.0 SUMMARY OF RECOMMENDATIONS

The review makes six recommendations to:

Review the current consultation process with end users to ensure it is understood and issues

are properly addressed.

Develop and implement CBSA project standards specific to test plans and user acceptance

testing (UAT).

Assess whether the system development life cycle and project management practices that

support agile techniques undertaken by the eManifest project are acceptable and applicable

to other Agency projects and update project management policies and guidance accordingly.

Develop and implement a process to demonstrate benefits realization of eManifest.

Provide assurance on the financial health and the remaining cost forecast of the eManifest

project and sign off as such on the dashboard.

Require that the Director General (DG) Projects Steering Committee reviews and discusses

all project risks and issues and include significant risks and issues on the eManifest project

dashboard.

6.0 OVERALL MANAGEMENT RESPONSE

The Programs Branch, Comptrollership Branch, and the Information, Science and Technology

Branch (ISTB) agree with this review and its findings.

Programs Branch will implement an improved consultation process with end users, will

demonstrate the benefits realization of eManifest, and will take steps to ensure that the DG Projects

Steering Committee is able to review and discuss all significant project risks and issues.

ISTB will assess whether the system development life cycle and project management practices that

support agile techniques, undertaken by the eManifest project are acceptable and applicable to other

Agency projects and will update project management policies and guidance accordingly. ISTB will

also move forward with a modernization and transformational proposal for a testing

framework/approach to address the recommendations.

Comptrollership, in consultation with Programs Branch (Office of Primary Interest), ISTB (Office

of Collateral Interest) and the Enterprise Project Management Office (EPMO) will implement a

process to strengthen the quality assurance of project financial information and the remaining cost

forecast of the eManifest project and sign off as such on the dashboard.

5

7.0 FINDINGS 7.1 Scope and user requirements

Project Management Objective: The project will meet its scope and end user requirements.

The success of a project depends on the effectiveness of the final project output in meeting its initial

business objectives and requirements of the business process owners and end users. Good project

management practices assist project managers to meet the project scope and end user requirements.

These practices include a sign-off of the scope definition and user requirements, having appropriate

safeguards in place to manage changes to project scope definition and user requirements, and

conducting user acceptance testing before the system is released into the production environment.

Project Scope:

Prior to 2010, sponsor branches (Admissibility, Enforcement and Operations branches) approved

business requirements. As of the April 1, 2010 reorganization, the Major Projects Directorate of

Programs Branch assumed the role of Office of Primary Interest (OPI) for the eManifest project.

Offices of Collateral Interest (OCI), including other directorates within the Programs Branch, and

other branches, now serve the role of Subject Matter Experts (SMEs). The purpose of this

reorganization was to improve the governance structure efficiencies and to streamline the escalation

process. As a result, internal end users (Operations Branch and Programs Branch, other than Major

Projects Directorate) are no longer required to sign off business requirements and they no longer

have the approval authority on policy issues and scope definition. The trade-off between efficiency

and end-user involvement presents itself with different risks that need to be considered and

weighed.

Currently, the Commercial Unit of the Pre Border Programs Directorate is tasked with coordinating

consultations on requirements between the project team and representatives of future end users. The

Commercial Unit identified relevant OPIs and provides relevant documentation for comments.

Once comments are received, the Commercial Unit will summarize them and attempt to find a

solution to issues. If an issue cannot be resolved, it is sent for a decision from the Director Policy

Steering Committee. Other avenues for feedback include an internal e-mail box, where users can

send issues for the policy team to review and action accordingly. The process has since evolved and

improved through the creation of the Managers Policy Committee in February 2012. This

committee was created to review new system functionality and discuss policy issues every two

weeks. These sessions should provide an opportunity to discuss and validate user needs. Although

consultations are occurring with future internal users, some end users expressed an interest in

having improved discussion on business requirements and feedback on decisions taken. Project

management understands the importance of good stakeholder consultations and considers this a low

risk in the overall context of the project.

6

Project Change Controls:

Regarding safeguards to manage changes to the project scope definition and end user requirements,

the eManifest project has adopted a more iterative approach to managing the project. The approach

used by the eManifest project team is inspired by agile project management concepts and is a

recognized approach to project management in the industry. Agile requirements elicitation requires

regular consultations with users through various methodologies to accommodate acceptable

business demands.

The eManifest project teams and information technology representatives must agree on which

project artefacts define the scope and the business requirements associated with each of the

remaining three builds that will form the basis for the development of systems. In most cases, the

business teams within the eManifest project are producing user requirements specifically aligned to

each build. An agreement on requirements forms the basis of the baseline, for which systems are

developed. The baseline for the first two builds was established in July 2012 creating a risk that IT

teams had been working using project artefacts that were not finalized. A requirements baseline is

an important element of change control, as all subsequent modifications to requirements after the

baseline is established require a formal change request. Without a baseline, user requirements may

fluctuate during the development process and cause unwanted consequences such as rework to

developed systems.

An additional consideration is the requirement for collaboration with future users during

requirements elicitation. As noted, the future three builds have incorporated the more iterative agile

approach to requirements definition. The current approach adopts aspects of an agile approach with

some important deviations; the eManifest project does not employ smaller release cycles, in which

the product is continuously ready for release. Additionally, continuous feedback between end users

and the development teams is not carried out. Consequently, there can be important lags between

when comments are received from stakeholders and when these groups will validate the

functionality developed.

User Acceptance Testing:

Typically, before system changes are released into the production environment, the application is

tested in a controlled environment to determine functionality and usability using scenarios that

resemble how the application will be employed by the end user.

The eManifest project has a testing plan and testing of previous releases has occurred. However, at

the CBSA, organizational guidance or standards specific to test plans or user acceptance have not

been developed.

With respect to testing activities for eManifest releases, areas to strengthen would include more

involvement of end users, and improved documentation and communication of test results.

Specifically, end users were not involved in acceptance testing and there was no separation of

duties between the individuals who defined the requirements and those who tested the requirement.

7

The test plan was insufficient in detail. Test plans should include details on the scope of testing,

schedule, test deliverables, release criteria and risks, and contingencies. The test results reviewed

were not adequately documented and not shared with all stakeholders to allow for an informed

review and evaluation of the system implemented by internal project stakeholders such as Operation

Branch, Border Programs and Pre-Border Programs.

In May 2012, a release pertaining to the Passage Deployment for Rail had a reduced testing period.

There were problems with the implementation of the system and resolution of these issues created a

delay of one month to the development schedule for the next release. The May 2012 release

proceeded to meet project time commitments. It is recognized that management needs to balance

project scope requirements, cost and schedule and manage the risks associated with them.

Summary

In regard to this project management objective as noted above, the Agency is exposed to risks that

system functionality may not meet the initial needs of end users and additional costs may be

incurred to make change requests to respond to users’ concerns. As well, expected results to

enhance employees’ efficiency may not be achieved and management may not be able to obtain the

information it requires to effectively manage the business process.

Within the project, there are different levels of risk tolerance. The trade-off between project

efficiency and end-user involvement presents risks that need to be considered and weighed by the

project governance committees.

RECOMMENDATION 1

The Vice-President of Programs Branch should review the current consultation process with end

users to ensure the process is understood, and end user feedback is considered and issues are

properly disposed of.

MANAGEMENT RESPONSE

Programs Branch agrees with this recommendation. The eManifest project team, in

consultation with Operations Branch, will enhance stakeholder consultations by conducting

focus groups with end users on end state passage and risk assessment systems. It will also

review and document the current process for consultation with end users, and will identify

any significant gaps in this process. By March 2013, Programs Branch will implement a

revised consultation process that will ensure the process is understood, the views of end

users are considered, and any issues identified by the process are addressed appropriately.

8

RECOMMENDATION 2

The Vice-President of ISTB in collaboration with the Vice-President of the Programs Branch

should develop and implement CBSA project standards specific to test plans and user acceptance

testing (UAT).

MANAGEMENT RESPONSE

The Information, Science and Technology Branch (ISTB) agrees with this recommendation.

ISTB will move forward with a modernization and transformational proposal for a testing

framework/approach to address standards specific to test plans and UAT. By April 2013, a

Testing Model Pilot will be implemented.

7.2 Project Plan

Project Management Objective: The project plan is sufficiently detailed to guide the project

throughout its lifecycle.

Project management guidance at the CBSA has established a common approach for managing

projects. Project plans are required documentation that includes scope, deliverables, work

breakdown structures, roles and responsibilities, cost estimates, milestones, key dependencies and

the identification of the project's critical path. An effective project plan establishes a baseline for the

project and enables management to track progress in terms of scope, time and costs. This guidance

developed by the Enterprise Project Management Office is largely based on traditional project

management principles.

The eManifest approach to project scheduling adopts some agile concepts to realize efficiencies.

The Agency’s guidance does not provide for alternative methodologies such as agile project

management, which is recognized as a valid project management methodology. In summary, the

eManifest project management approach includes:

A scope definition that aims to realize efficiencies. In December 2010, the eManifest

Strategic Deployment View Task Force was established to develop a high-level deployment

sequence for the remaining eManifest deliverables. This scope definition was amended to

allow deliverables to be released in parallel and realize efficiencies in the schedule.

A time box approach was adopted for project planning, where deadlines (end dates) related

to system implementation are fixed. The scheduled dates for tasks are permitted to fluctuate

to meet deadlines. Each project team is accountable for meeting its deadlines.

The master schedule contains the duration and estimated hours per task; a high number of

detailed plans are aggregated to the project master schedule. The granular project plans

contain the deliverables agreed upon in the scope definition document.

Three Builds for eManifest remain with release dates; for Build 1 – March 2013, Build 2 –

July 2014 and Build 3 – December 2014.

9

In project planning, a work breakdown structure helps accurately define and organize the scope of

the project by breaking down the project deliverables into work packages across different levels of

the project plans. Based on a review of project plans, the following was observed:

Project deliverables were indicated on detailed project plans.

Activities have been assigned an estimated duration, but did not provide specifics on

resource assignments, estimated effort or standard costs.

For detailed project plans, the 8/80 rule was not employed; this guideline stipulates that no

work package should be less than 8 hours or greater than 80 hours. This rule allows project

managers to govern the size and scope of the smallest chunks of work and apply consistent

rules across work breakdown structures.

A critical path method is used to predict project duration by analyzing the sequence of activities that

has the least amount of scheduling flexibility. For eManifest, the traditional critical path method

was not employed. The eManifest project schedule assumes no flexibility in delivery for builds.

The initial release management timelines for Build 1 could be at risk. The preliminary estimated

effort to conduct testing was greater than the time available to meet the delivery date of March

2013. Project governance committees will need to engage if potential issues arise in order to

facilitate decision making.

Summary

With respect to the current project plan for eManifest, management does not have a complete set of

project management information and as such it limits management's ability to monitor progress

with respect to time, scope and costs. Each individual manager is responsible for managing his/her

piece of the eManifest project including resources, scope and schedule. Managers are required to

report regularly on the status of their component. Once a month, the results from managers are

consolidated into the overall eManifest project dashboard. There is considerable reliance on these

management processes and confidence that issues and risks will be identified and addressed in a

timely and effective fashion.

RECOMMENDATION 3

The Vice-President of ISTB should assess whether the system development life cycle and project

management practices that support agile techniques, undertaken by the eManifest project are

acceptable and applicable to other Agency projects, and update project management policies and

guidance accordingly.

MANAGEMENT RESPONSE

The ISTB agrees with this recommendation. ISTB will assess whether the system

development life cycle and project management practices that support agile techniques,

undertaken by the eManifest project are acceptable and applicable to other Agency projects,

and update project management policies and guidance accordingly. The planned completion

date is April 2013.

10

7.3 Cost Management

Project Management Objective: Costs are monitored; deviations are identified and addressed in a

timely manner. Performance will be measured and validated.

Under Treasury Board Policy on the Management of Projects, projects are expected to achieve

value for money and deliver outcomes within cost constraints. The Agency also expects that

projects achieve value for money; the CBSA submitted a cost management plan as part of the

effective project approval that stated that the earned value management technique would be used to

ensure that costs were managed effectively, and performance was measured and validated.

Earned value management (EVM) is a process that provides project managers and stakeholders with

insight into investment performance, and provides the necessary data points to estimate statistically

probable completion costs and dates. The implementation of EVM is a recognized parameter of

good project management. It ensures that cost, schedule, and technical aspects of the investment are

integrated and:

• Relates time-phased budgets to specific tasks;

• Indicates work progress;

• Properly relates cost, schedule, and technical accomplishment to scope;

• Is valid, timely, and auditable;

• Allows for statistical estimation of completion costs;

• Supplies managers with information summarized at a practical level;

• Facilitates risk management; and

• Provides information on benefits realization.

For eManifest, EVM techniques are not currently being employed. Project managers indicated that

it was difficult to meet EVM requirements with the tools they presently have.

Currently, the eManifest project team is tracking actual costs through CAS. Different groups

however are reporting their costs differently. For example, ISTB employees use internal orders to

report their time. Other groups do not employ internal orders for their timesheets; rather they

perform a salary allocation to the project based on actual or estimated costs. The internal order code

is not a mandatory field in CAS. As such this creates a risk that costs are not distributed accurately

by projects.

At present, it is difficult to relate the full project cost to the actual and future project deliverables

and builds. Multiple systems are employed to support the cost management process such as

COMPASS and CAS but are not integrated with the eManifest project plans.

Multi-year projects are expected to forecast costs based on a costing template that aligns with the

Treasury Board submission. eManifest is not following this process. Project managers are

negotiating their budget requirements with the eManifest project office without knowing the cost of

the detailed project plans. Cost forecasts to complete eManifest are based on the original Treasury

Board submission and workplans developed by project managers.

11

As per the CBSA project guidance, the Comptrollership Branch and the EPMO have important

roles related to cost management. The EPMO is responsible for providing project oversight,

maintaining project management guidance, reviewing and analyzing the dashboard reports to ensure

accurate reporting on key project metrics, and validating and challenging the dashboard

information. Comptrollership is responsible for providing and completing the project financial

information and for performing quality assurance of project financial information. Comptrollership

is unable to provide assurance on project financial health because the project costs and forecasts are

not done by deliverables. Comptrollership does review, monitor and discuss with project

management the cost information to ensure the financial information aligns with CAS and

COMPASS.

Summary

In regard to the project management objective, the Agency is exposed to the risk that CBSA

management may not be made aware in a timely manner of potential cost overruns to the eManifest

project. Actual costs are monitored and the eManifest burn rate is tracked to the original eManifest

budget.

Management recognizes these challenges to integrate all costs and forecasts by deliverable and

relate them back to the project benefits. EVM is one approach to accomplish this but requires the

capacity, tools and discipline to implement.

RECOMMENDATION 4

The Vice-President of Programs Branch should develop and implement a process to demonstrate

the benefits realization of eManifest.

MANAGEMENT RESPONSE

Programs Branch agrees with this recommendation. The eManifest project will complete a

draft report demonstrating the benefits realization of the eManifest project by January 2013.

The project will then submit this report to the Vice-President-level Projects Steering

Committee for approval by March 2013.

RECOMMENDATION 5

The Vice-President of Comptrollership Branch should provide assurance on the financial health and

the remaining cost forecast of the eManifest project and sign off as such on the dashboard. This

includes a discussion with the Programs Branch on the financial information requirement to provide

the necessary assurance.

12

MANAGEMENT RESPONSE

The Comptrollership Branch agrees with this recommendation.

Since the eManifest Review, Comptrollership has worked collaboratively with key

stakeholders (OPI, OCI and EPMO) to establish the necessary elements to allow for the

provision of quality assurance on the financial health.

An Internal Order structure (broken down by Components and Builds) has been established

and agreed upon. Costs and timelines will be assigned to each Component and Build to

further enhance the ability to monitor and report on the financial health, commencing April

1, 2013. This measure will help the Agency advance towards the principle of Earned Value

Management (EVM).

The Branch Chief Financial Officer will be signing off, and/or providing comments, where

necessary, on the monthly dashboards to confirm that an assurance on the financial health,

including the remaining cost forecast, has been conducted.

7.4 Project Status Reporting

Project Management Objective: Tools and processes for status reporting are used appropriately.

eManifest project management is required to report the project status to CBSA senior executives

and the Treasury Board Chief Information Officer Branch on a monthly basis. This is accomplished

through a project dashboard designed to report on the core project metrics of cost, schedule, scope,

risks and issues. Data requirements are stipulated for each metric within the Guide to Executive

Project Dashboards for Treasury Board Oversight Projects. For example, to report on the project

schedule, project management specifies the approved completion date, forecasted completion date

and variance by deliverable. According to the Treasury Board Secretariat’s guide, the variance

indicator is calculated based on the difference between the revised completion date (approved) and

the forecasted completion date.

The status of each metric is colour-coded using green, yellow and red indicators. Green indicates a

variance of less that 10%, yellow is a variance between 10 and 20% and red is a variance of over

20%. The eManifest Project Support and Control Office completes the project reporting process by

collecting the information from various systems, and documents and validates the information with

the Project Director's Steering Committee before it is inserted on the dashboard. As noted above,

the quality assurance of financial information is to be provided by the Comptrollership Branch. The

EPMO reviews the reports and provides a challenge function on the accuracy and completeness of

the reporting. The final dashboard is signed off by the Major Project’s Director General.

The guidance on roles and responsibilities is sufficiently defined and key stakeholders understand

their roles and responsibilities for approvals and quality reviews.

13

For eManifest, there are some deviations from Treasury Board Secretariat’s guidance for reporting

on the project schedule. The schedule section of the status reports asks that variances be reported

for key milestones or deliverables based on the approved completion date and forecasted

completion date. For eManifest, the EPMO has raised concerns regarding the definition of a

milestone or deliverable. Currently, the project management uses project phases (e.g. construction

and testing) as their definition for milestones or deliverables. The EPMO requests that the

dashboard reflect key deliverables by the remaining Builds. This would be more effective in

monitoring progress and potential slippage.

Additionally, the calculation of variance (slippage as per the Treasury Board Secretariat and EPMO

guide) deviated from the policy. Project management currently uses a project schedule that includes

the entire project development time for eManifest (2007 to 2014). The current project schedule was

baselined and approved in January 2012 with final project implementation by December 2014.

Therefore, the slippage calculation should be based on the revised baseline project duration.

During the course of the review, various stakeholders raised a couple of issues and they appear to

be of significance for reporting on the project dashboard:

Key regulations that needed to come into force by November 2012 to enforce the mandatory

requirement for highway carriers to transmit information electronically are not in place. As a

result the anticipated benefits have been delayed.

Related to the first point, there is a slow uptake of highway carriers to employ electronic

data interchange. As of October 2012, 3,339 out of 15,000 active carriers were registered

which account for 22%, however it represents 77% of total trade volumes. Of the remaining

non-registered carriers, 5,651 carriers had averaged less than one release per month of which

1,381 had only one release during the year. This could potentially create a demand from

carriers when the mandatory requirement is put in place which the agency is unable to deal

with in a timely matter.

Regarding project risks, project risks are identified on the dashboard and they include mitigation

strategies. In reviewing some of the mitigation strategies, it is questionable whether the mitigation

will achieve a desired outcome in response to the risk. For example, a project risk identified is the

resource capacity of the ISTB to deliver on eManifest in addition to other Agency projects such as

the Beyond the Border Action Plan and the CBSA Assessment and Revenue Management initiative.

The mitigation was that eManifest is the foundation of many initiatives and the Agency’s

commitment is to continue to implement eManifest and secure required resources. The review team

has heard that this risk is now manifesting itself. As such this should be noted as an open issue and

managed accordingly.

Summary

In regard to this project management objective, project management is reporting regularly on the

status of the project and the tools and dashboards are being used. Clarification is required on the

information that should be reported in the dashboard and the completeness of some of the

dashboard elements, particularly the risk and issues section. Oversight on the dashboard reports

14

could also be improved to ensure all significant information from CBSA stakeholders is reported.

The risk is that CBSA executive management may not be informed of significant issues that could

have broader implications for operations.

RECOMMENDATION 6

The Vice-President of Programs Branch should require the DG Projects Steering Committee to

review and discuss all project risks and issues, and include significant risks and issues on the

eManifest project dashboard.

MANAGEMENT RESPONSE

Programs Branch agrees with this recommendation. By January 2013, detailed risk and

issue logs will be standing items on the DG Projects Steering Committee agenda where they

will be reviewed and discussed, and significant items will be included on the Project

Dashboard.

15

APPENDIX A – ABOUT THE REVIEW

REVIEW OBJECTIVES AND SCOPE

The objective of this review was to determine the extent to which eManifest will meet the needs of

end users; the adequacy of the project plan with respect to managing the project’s scope, schedule

and resources; the management of costs; and the effectiveness of tools and processes for status

reporting.

The review scope included all aspects of the eManifest project; there were no exclusions. The

engagement was a review and not an audit as detailed testing was not undertaken. The level of our

work was commensurate with a review, which provides a low level of assurance.

RISK ASSESSMENT

A number of risk areas were identified in the planning phase of the review:

Scope: In the absence of adequate controls over scope management there are risks that the

project will fail to meet business and user requirements leading to the rejection of delivered

capabilities. Additionally, there is a risk that insufficient testing will result in unsatisfactory

systems to business users.

Project Management: In the absence of reasonable project management practices, there are

risks of undetected deviations in scope, schedule, and costs. Additionally, without consistent

project management practices there are risks of inconsistency across interdependent

projects.

Cost Management: In the absence of sufficient controls over costs, there are risks of a lack

of control over project costs resulting in misspending of IT investments.

Project Reporting: In the absence of satisfactory controls over project reporting there are

risks that reporting is ineffective on project progress which may lead to unidentified issues,

undetected project showstoppers or undetected project risks. There is also a risk of loss of

focus on customer expectations and business needs.

APPROACH AND METHODOLOGY

The review was based mainly on interviews and documentation reviews using two frameworks:

Control Objectives for Information and Related Technology (COBIT) version 4.1; and

Project Management Body of Knowledge (PMBOK).

COBIT identifies the IT processes that should exist to ensure that IT is aligned with and supports

the business in an effective manner. COBIT and its supporting publications identify control

objectives, techniques and practices commonly required for each process.

PMBOK identifies the best practice process for project management, together with the knowledge

and techniques required for those processes to be effective. PMBOK can be designed to be applied

to any industry, including IT.

16

THE REVIEW CRITERIA USED FOR THE ENGAGEMENT PHASE WERE:

LINES OF

ENQUIRY

CRITERIA SUBCRITERIA

1. Scope and

user

requirements

The project will

meet its scope and

the requirements of

end users.

Project Scope: The project scope is defined and

formally approved by the program and project

sponsors.

Project Change Control: The project plan reflects

changes to scope for approved changes, a

baseline is established and the project plan is

updated to reflect approved change requests.

User acceptance testing: The system meets the

requirements of the business end users through

successful user acceptance testing.

2. Project Plan The project plan is

sufficiently detailed

to guide the project

throughout its life

cycle.

Integrated Project Plan: The scope, deliverables,

required resources and responsibilities, work

breakdown structures, estimates of resources

required, milestones, key dependencies, and

identification of a critical path are documented in

the integrated project plan.

3. Cost

management Costs are

monitored;

deviations are

identified and

addressed in a

timely manner.

Cost Management: A cost management process

has been implemented comparing actual costs to

budgets. Costs are monitored and reported, where

deviations are identified and assessed in a timely

manner.

4. Project Status

Reporting Tools and processes

for status reporting

are used

appropriately

Roles and responsibilities related to status

reporting are clear and understood.

Project Reporting: Project status reporting

complies with the relevant organizational

standards.

17

APPENDIX B – GLOSSARY

Acceptance testing: Applying performance and capability measures to project deliverables

to ensure that they meet specifications and requirements and satisfy the customer.

Agile Project Management: This is a highly iterative and incremental process, where

developers and project stakeholders actively work together to understand the domain,

identify what needs to be built, and prioritize functionality. Agile methods are used when

these conditions are present: project value is clear; the customer actively participates

throughout the project; the customer, designers, and developers are co-located; incremental

feature-driven development is possible; and visual documentation (cards on the wall vs.

formal documentation) is acceptable.

Baseline: Original Plan (for a project, work package, activity), plus or minus any approved

changes. May be used with a modifier (for example, cost baseline, schedule baseline,

performance measurement baseline).

Baseline Measurement: Set of approved estimates for scope, cost and schedule that forms

the basis for change management during project execution. A project is baselined at the end

of the planning phase.

Build 1: Functionality that allows freight forwarders in all modes of transportation to

transmit pre-arrival detailed data (i.e. House bill) is expected for March 2013.

Build 2: Functionality for commercial passage and core risk assessment is expected for July

2014. It will also include the functionality to allow importers to transmit Advance Trade

Data.

Build 3: Enhancements to Risk Assessment and Passage are expected in December 2014,

and will include enhancements to the commercial passage and risk assessment

functionality, as well as new functionality for crew risk assessment in all modes.

Milestone: 1. An event with zero duration and requiring no resources. Used to measure the progress of

a project or program and signify the completion or start of a major deliverable or other

significant metric such as cost incurred, hours used or payment made.

2. Identifiable point in a project, program or set of activities that represents a reporting

requirement or completion of a large or important set of activities.

Project Scope Management: Includes the processes required to ensure that the project

includes all the work required, and only the work required, to complete the project

successfully. Project scope management is primarily concerned with defining and

controlling what is and what is not included in the project.

18

APPENDIX C – ACRONYMS

CARM - CBSA Assessment and Revenue Management

CAS - Corporate Administrative System

CBSA - Canada Border Services Agency

COBIT - Control Objectives for Information and Related Technology

COMPASS - Compliance Measurement, Profiling and Assessment System

DG - Director General

EPMO - Enterprise Project Management Office

EVM - Earned Value Management

ISTB - Information, Science and Technology Branch

IT - Information Technology

OCI - Office of Collateral Interest

OPI - Office of Primary Interest

PMBOK - Project Management Body of Knowledge

SME - Subject Matter Expert

UAT - User Acceptance Testing