Auditing Large and Complex Auditing Large and Complex IT ProjectsIT Projects

Auditor GeneralAuditor General

State Of FloridaState Of Florida

OverviewOverview

Basic IT project risks.Basic IT project risks.

Example IT projects in Florida.Example IT projects in Florida.

Issues and challenges to auditing large IT Issues and challenges to auditing large IT projects.projects.

Checklist of suggested audit questions.Checklist of suggested audit questions.

Suggestions for further reading.Suggestions for further reading.

Basic IT Project RisksBasic IT Project Risks

Planning.Planning.

Procurement.Procurement.

Contract management.Contract management.

Development and Implementation.Development and Implementation.

Post-implementation.Post-implementation.

IT Project Risks - PlanningIT Project Risks - Planning

Lack of clearly defined goals, objectives, and Lack of clearly defined goals, objectives, and requirements.requirements.Lack of identification of project risks.Lack of identification of project risks.Poorly defined project management structure.Poorly defined project management structure.Unrealistic budget – time and dollars.Unrealistic budget – time and dollars.Lack of stakeholder & user buy-in.Lack of stakeholder & user buy-in.Extent and impact of business process Extent and impact of business process reengineering not sufficiently addressed.reengineering not sufficiently addressed.Inadequate basis for development of Inadequate basis for development of procurement criteria.procurement criteria.– No established baseline for measuring cost savingsNo established baseline for measuring cost savings

IT Project Risks - IT Project Risks - ProcurementProcurement

Poorly defined statement of work.Poorly defined statement of work.

Lack of basis for competitive solicitation.Lack of basis for competitive solicitation.

Flawed evaluation of proposals.Flawed evaluation of proposals.

Litigation after award.Litigation after award.

IT Project Risks – Contract IT Project Risks – Contract ManagementManagement

Contract terms that don’t adequately protect the state’s Contract terms that don’t adequately protect the state’s interest.interest.Poorly defined deliverables.Poorly defined deliverables.Lack of agreed upon performance measures.Lack of agreed upon performance measures.Payment without suitable performance.Payment without suitable performance.Lack of recourse for non-performance or poor Lack of recourse for non-performance or poor performance by contractors.performance by contractors.Vague ownership provisions – data, software, hardware.Vague ownership provisions – data, software, hardware.Vague or missing provisions regarding data security.Vague or missing provisions regarding data security.Poor or lacking termination & transition clauses to Poor or lacking termination & transition clauses to protect state’s interest should either party cancel.protect state’s interest should either party cancel.

IT Project Risks - Development IT Project Risks - Development and Implementationand Implementation

Implementation of software that does not Implementation of software that does not function properly & include good controls.function properly & include good controls.Poor data security.Poor data security.Inadequate training and knowledge transfer, Inadequate training and knowledge transfer, resulting in lack of sufficient knowledge to resulting in lack of sufficient knowledge to operate, maintain, and use the new systemoperate, maintain, and use the new systemCost overruns.Cost overruns.Failure to meet implementation deadlinesFailure to meet implementation deadlinesOver-customization of vendor software making Over-customization of vendor software making future upgrades costly to obtain and implement.future upgrades costly to obtain and implement.Loss of data integrity during conversion/cutover.Loss of data integrity during conversion/cutover.

IT Project Risks – Post IT Project Risks – Post ImplementationImplementation

Security vulnerabilities as a result of security not Security vulnerabilities as a result of security not being hardened after implementation.being hardened after implementation.Integrity of system compromised through poor Integrity of system compromised through poor maintenance and change control.maintenance and change control.On-going reliance on contractors.On-going reliance on contractors.In-house knowledge not sufficient to maintain In-house knowledge not sufficient to maintain system or infrastructure.system or infrastructure.Unable to turn off old systems that were to be Unable to turn off old systems that were to be replaced.replaced.Unable to sustain user buy-in.Unable to sustain user buy-in.

Example Florida ProjectsExample Florida Projects

My Florida Marketplace.My Florida Marketplace.

People First.People First.

MyFlorida Alliance.MyFlorida Alliance.

Aspire.Aspire.

Various educational entities.Various educational entities.

MyFlorida MarketplaceMyFlorida Marketplace

A Web-based e-procurement system for state A Web-based e-procurement system for state agencies.agencies.Developed by customizing Ariba software.Developed by customizing Ariba software.Application Service Provider – Accenture.Application Service Provider – Accenture.Responsible State agency: Department of Responsible State agency: Department of Management Services.Management Services.Total contract cost : $93.9 million.Total contract cost : $93.9 million.Contract term: October 2002 through November Contract term: October 2002 through November 2010.2010.Funded through 1% transaction fee on Funded through 1% transaction fee on purchases.purchases.

MyFlorida Marketplace – MyFlorida Marketplace – Planning and Procurement Planning and Procurement

IssuesIssuesCost-benefit and risk analysis not Cost-benefit and risk analysis not conducted prior to decision to outsource conducted prior to decision to outsource and issuance of ITN.and issuance of ITN.Insufficient involvement of key end-users Insufficient involvement of key end-users and stakeholders in development of ITN.and stakeholders in development of ITN.Lack of significant baseline data for Lack of significant baseline data for planning and analysis.planning and analysis.No mechanism to capture and track No mechanism to capture and track statewide costs associated with MFMP.statewide costs associated with MFMP.

MyFlorida Marketplace – MyFlorida Marketplace – Contract and Project Contract and Project Management IssuesManagement Issues

System not formally accepted prior to System not formally accepted prior to implementation.implementation.

Heavy reliance on ASP without sufficient Heavy reliance on ASP without sufficient monitoring of performance.monitoring of performance.– Limited monitoring by DMS.Limited monitoring by DMS.– Third-party monitor’s duties strayed from Third-party monitor’s duties strayed from

monitoring ASP.monitoring ASP.

MyFlorida Marketplace – IT MyFlorida Marketplace – IT Functionality and Control Functionality and Control

IssuesIssuesDeficient change control process.Deficient change control process.

System performance and capacity System performance and capacity management needed improvement; management needed improvement; performance issues existed.performance issues existed.

Disaster recovery plan not timely approved Disaster recovery plan not timely approved and lacked important provisions.and lacked important provisions.

Insufficient back-up provisions.Insufficient back-up provisions.

Deficiencies in security controls.Deficiencies in security controls.

MyFlorida Marketplace – IT MyFlorida Marketplace – IT Functionality and Control Functionality and Control

IssuesIssuesData integrity issues.Data integrity issues.– Problems with accounting system interface.Problems with accounting system interface.– Problems with attached scanned documents.Problems with attached scanned documents.

Declining use of system by State Declining use of system by State agencies.agencies.– System functionality issues.System functionality issues.– System performance issues.System performance issues.– Workflow inefficiencies.Workflow inefficiencies.

People FirstPeople First

An HR outsourcing initiative.An HR outsourcing initiative.

Includes a Web-based enterprise-wide Includes a Web-based enterprise-wide ERP system supporting:ERP system supporting:– HR administration.HR administration.– Benefits administration.Benefits administration.– Payroll administration.Payroll administration.– Staffing administration. Staffing administration.

People FirstPeople First

Service provider – Convergys.Service provider – Convergys.9-year contract.9-year contract.$349.9 million.$349.9 million.People First application built using SAP People First application built using SAP software.software.System phased in between May 2003 and System phased in between May 2003 and January 2005.January 2005.Convergys provides Florida with a SAS 70 Convergys provides Florida with a SAS 70 report on its service center.report on its service center.

People First – Planning and People First – Planning and Procurement IssuesProcurement Issues

Cost-benefit and risk analyses not Cost-benefit and risk analyses not performed prior to release of ITN.performed prior to release of ITN.

Inaccuracies in cost estimates within Inaccuracies in cost estimates within Business Plan.Business Plan.

No system to track statewide project cost.No system to track statewide project cost.

Deficiencies in evaluation and negotiation Deficiencies in evaluation and negotiation processes.processes.

People First – Contractual People First – Contractual IssuesIssues

Legal records retention requirements not Legal records retention requirements not included in contract.included in contract.

No provision for State to approve new or No provision for State to approve new or changes in subcontractors.changes in subcontractors.

No provision for subcontractors to obtain No provision for subcontractors to obtain background checks.background checks.

Many deliverables not timely provided.Many deliverables not timely provided.

Additional amounts paid to third-party monitor for Additional amounts paid to third-party monitor for performing services already required.performing services already required.

People First – Operational People First – Operational ProblemsProblems

System functionality problems and errors.System functionality problems and errors.

Lack of written security guidelines.Lack of written security guidelines.

Off-shoring of State employee personnel Off-shoring of State employee personnel data.data.

Planned system components not Planned system components not implemented, requiring workarounds by implemented, requiring workarounds by the agencies.the agencies.

MyFlorida AllianceMyFlorida Alliance

An effort of the State Technology Office to An effort of the State Technology Office to reengineer its IT functions and governance reengineer its IT functions and governance structure through outsourcing many of its structure through outsourcing many of its primary functions.primary functions.

STO was responsible for centralized STO was responsible for centralized management of IT for the executive management of IT for the executive branch of Florida government.branch of Florida government.

MyFlorida Alliance – Functions MyFlorida Alliance – Functions to be Outsourcedto be Outsourced

Enterprise e-communications.Enterprise e-communications.

Enterprise technology services desk.Enterprise technology services desk.

Enterprise applications management.Enterprise applications management.

Enterprise data center operations and Enterprise data center operations and consolidation.consolidation.

Enterprise portal, enterprise security, Enterprise portal, enterprise security, various others planned.various others planned.

MyFlorida AllianceMyFlorida Alliance

Two prime contractors – Bearing Point and Two prime contractors – Bearing Point and Accenture.Accenture.Contracts signed August 13, 2003.Contracts signed August 13, 2003.7 year term.7 year term.$324.7 million.$324.7 million.Procurement method: ITN.Procurement method: ITN.All agreements since terminated by STO.All agreements since terminated by STO.STO abolished in law effective 7/1/07.STO abolished in law effective 7/1/07.

MyFlorida AllianceMyFlorida Alliance

Inadequate planning & documentation to support Inadequate planning & documentation to support decisions to outsource or use ITN method.decisions to outsource or use ITN method.

Deficiencies in proposal evaluation and Deficiencies in proposal evaluation and negotiation, limiting fairness and competition.negotiation, limiting fairness and competition.

Contracts lacked provisions to protect the State Contracts lacked provisions to protect the State and did not pinpoint total cost to State.and did not pinpoint total cost to State.

Cost savings analyses questionable or not Cost savings analyses questionable or not available.available.

AspireAspire

Replacement for the State’s general ledger Replacement for the State’s general ledger accounting and cash management systems.accounting and cash management systems.

Project beginning date - 9/8/2003.Project beginning date - 9/8/2003.

Original planned rollout in three waves of Original planned rollout in three waves of agencies going live, from July 2005 through agencies going live, from July 2005 through December 2005.December 2005.

Rollout schedule amended five times.Rollout schedule amended five times.

Project suspended 5/17/2007, with $89 million Project suspended 5/17/2007, with $89 million spent to date.spent to date.

IT Projects at Florida’s IT Projects at Florida’s Educational EntitiesEducational Entities

University ERP implementations – for University ERP implementations – for example, University of Florida.example, University of Florida.– Insufficient system testingInsufficient system testing– Insufficient staff trainingInsufficient staff training– Functional system problemsFunctional system problems– System governance issuesSystem governance issues– IT security and control issuesIT security and control issues

Various community colleges and school Various community colleges and school districts.districts.

Issues and Challenges to Issues and Challenges to Auditing Large IT ProjectsAuditing Large IT Projects

When to audit.When to audit.– During the project?During the project?– Postaudit?Postaudit?

Impact of outsourcing.Impact of outsourcing.– Authority to audit contractor & subcontractorsAuthority to audit contractor & subcontractors– Contractor responsiveness to audit requestsContractor responsiveness to audit requests– Applicability of SAS 70 audits, if available, to non-financial audit Applicability of SAS 70 audits, if available, to non-financial audit

objectivesobjectives

Lack of sufficient entity knowledge of system.Lack of sufficient entity knowledge of system.– Difficulty in extracting data needed for auditDifficulty in extracting data needed for audit

Availability of entity staff to timely respond to audit Availability of entity staff to timely respond to audit requests during implementation or post-implementation. requests during implementation or post-implementation.

Suggested Audit ChecklistSuggested Audit Checklist

Planning.Planning.

Procurement.Procurement.

Contract Provisions and Management.Contract Provisions and Management.

Project Management.Project Management.

Suggested Further ReadingSuggested Further Reading

MyFlorida MarketplaceMyFlorida Marketplace– 2007-076 (IT Audit)2007-076 (IT Audit)– 2006-015 (IT Audit)2006-015 (IT Audit)– 2005-116 (Operational Audit)2005-116 (Operational Audit)

People FirstPeople First– 2007-087 (Operational Audit)2007-087 (Operational Audit)– 2005-047 (Operational Audit)2005-047 (Operational Audit)

Suggested Further ReadingSuggested Further Reading

MyFlorida AllianceMyFlorida Alliance– 2005-0082005-008

University of FloridaUniversity of Florida– 2006-040 (Operational Audit)2006-040 (Operational Audit)– 2006-145 (IT Audit)2006-145 (IT Audit)

Chapter 2007-115, Laws of FloridaChapter 2007-115, Laws of Florida– Abolishes STO and creates AEITAbolishes STO and creates AEIT

Questions?Questions?