CIWQS Review Phase II: Evaluation and Final Recommendations March 14, 2008
description
Transcript of CIWQS Review Phase II: Evaluation and Final Recommendations March 14, 2008
CIWQS Review Phase II:CIWQS Review Phase II:
Evaluation and Final Evaluation and Final
RecommendationsRecommendations
March 14, 2008March 14, 2008
BackgroundBackgroundBackgroundBackground Review Panel convened in response to Review Panel convened in response to
State Water Board’s request for external State Water Board’s request for external review of CIWQSreview of CIWQS
First Panel meeting held in May 2007First Panel meeting held in May 2007
Preliminary Panel report with Preliminary Panel report with recommendations issued July 2007recommendations issued July 2007
Water Board staff to consider Water Board staff to consider recommendations and demonstrate recommendations and demonstrate progress toward benchmarks through progress toward benchmarks through February 2008February 2008
Findings of Previous Findings of Previous ReviewReview
Findings of Previous Findings of Previous ReviewReview
Seven essential recommendations:Seven essential recommendations: Reduce the project’s scopeReduce the project’s scope Restructure CIWQS’ project managementRestructure CIWQS’ project management Validate the system’s requirementsValidate the system’s requirements Rebuild key constituenciesRebuild key constituencies Address data quality issuesAddress data quality issues Produce key reportsProduce key reports Improve user interfaceImprove user interface
Goals for Phase IIGoals for Phase IIGoals for Phase IIGoals for Phase II
Water Board staff will demonstrate Water Board staff will demonstrate progress by responding to Phase I progress by responding to Phase I recommendationsrecommendations
Including specific actions, timelines, and Including specific actions, timelines, and additional performance measures for the coming additional performance measures for the coming yearyear
Panel will review progress and prepare final Panel will review progress and prepare final report to:report to:
Judge adequacy of Water Board’s responseJudge adequacy of Water Board’s response Make final recommendations for future direction Make final recommendations for future direction
of the projectof the project
Overview of Final ReviewOverview of Final ReviewOverview of Final ReviewOverview of Final Review
CIWQS remains essential for the State Water Board to CIWQS remains essential for the State Water Board to fulfill its missionfulfill its mission
Terminating funding would be counterproductive Terminating funding would be counterproductive
CIWQS should be funded at an appropriate levelCIWQS should be funded at an appropriate level
The program’s new direction should lead to successThe program’s new direction should lead to success We are impressed with the remarkable turnaround since MayWe are impressed with the remarkable turnaround since May
We are particularly impressed with progress in We are particularly impressed with progress in reorganizing management and communicating with reorganizing management and communicating with usersusers
Overview (cont.)Overview (cont.)Overview (cont.)Overview (cont.)
The redefinition of project scope was The redefinition of project scope was appropriate and effectiveappropriate and effective
We have remaining concerns with some We have remaining concerns with some aspects of the technical approach, rate of aspects of the technical approach, rate of data cleanup, and report productiondata cleanup, and report production
Original RecommendationsOriginal RecommendationsOriginal RecommendationsOriginal Recommendations
Reduce the scope Reduce the scope
Restructure management Restructure management
Validate system requirementsValidate system requirements
Rebuild constituencyRebuild constituency
Address data issuesAddress data issues
Produce key reports Produce key reports
Improve user interfaceImprove user interface
Reduce the ScopeReduce the ScopeReduce the ScopeReduce the Scope
EvaluationEvaluation Achieved significant scope reductionAchieved significant scope reduction Increased probability of project successIncreased probability of project success Achieved buy-in from Steering Committee & usersAchieved buy-in from Steering Committee & users Phased eSMR implementation plan is reasonablePhased eSMR implementation plan is reasonable Implementation through eSMR Level 3 is desirableImplementation through eSMR Level 3 is desirable We concur with need for a separate federal interfaceWe concur with need for a separate federal interface
RecommendationRecommendation Consider trimming unneeded parts of the database to Consider trimming unneeded parts of the database to
meet reduced project scopemeet reduced project scope
Restructure ManagementRestructure Management Restructure ManagementRestructure Management
EvaluationEvaluation The new management structure positions CIWQS for The new management structure positions CIWQS for
successsuccess Effective new organization with leadership in Water Quality Effective new organization with leadership in Water Quality
not in DIT ties the project closer to the Board’s missionnot in DIT ties the project closer to the Board’s mission Buy-in from senior managers for the project’s importance Buy-in from senior managers for the project’s importance
and new directionand new direction
Validate System RequirementsValidate System RequirementsValidate System RequirementsValidate System Requirements
EvaluationEvaluation Fundamentally improved systems design & implementationFundamentally improved systems design & implementation Corrected design and approach based on testingCorrected design and approach based on testing The underlying structure can be made to workThe underlying structure can be made to work
Referential integrity issues are being addressedReferential integrity issues are being addressed Requirements better reflect user inputs & needsRequirements better reflect user inputs & needs
RecommendationsRecommendations Demonstrate a disciplined systems engineering processDemonstrate a disciplined systems engineering process
Document team members’ roles, responsibilities, activitiesDocument team members’ roles, responsibilities, activities Use test cases to demonstrate successful resolution of Use test cases to demonstrate successful resolution of
referential integrity issuesreferential integrity issues
Rebuild ConstituencyRebuild ConstituencyRebuild ConstituencyRebuild Constituency
EvaluationEvaluation Achieved remarkable cultural change that significantly Achieved remarkable cultural change that significantly
improved user communication and supportimproved user communication and support Developed effective user community structureDeveloped effective user community structure Users attended Panel meeting and strongly supported the Users attended Panel meeting and strongly supported the
projectproject Users strongly engaged in most areasUsers strongly engaged in most areas Issues with stormwater group illustrate cultural Issues with stormwater group illustrate cultural
improvements have further to goimprovements have further to go
RecommendationsRecommendations Should apply approach used for eSMR to all groups and Should apply approach used for eSMR to all groups and
issuesissues
Address Data IssuesAddress Data IssuesAddress Data IssuesAddress Data Issues EvaluationEvaluation
Less was accomplished than we had expectedLess was accomplished than we had expected Acknowledged the need for systematic QA/QCAcknowledged the need for systematic QA/QC Database integrity is being addressedDatabase integrity is being addressed Some progress on correcting existing data errorsSome progress on correcting existing data errors
RecommendationsRecommendations Devote more focused effort to QA/QC and data cleanupDevote more focused effort to QA/QC and data cleanup Develop coherent plan that prioritizes data cleanup and Develop coherent plan that prioritizes data cleanup and
identifies needed resources and timeframeidentifies needed resources and timeframe Create formal QA/QC mechanisms, including user Create formal QA/QC mechanisms, including user
participationparticipation Make QA/QC an integral part of database administration Make QA/QC an integral part of database administration
and operationand operation
Produce Key ReportsProduce Key ReportsProduce Key ReportsProduce Key Reports
Evaluation Evaluation Created a mechanism to identify and prioritize key reportsCreated a mechanism to identify and prioritize key reports Made some progress on developing canned reportsMade some progress on developing canned reports
RecommendationsRecommendations Demonstrate that required reports can be produced from Demonstrate that required reports can be produced from
the core CIWQS systemthe core CIWQS system Increase staff’s ability to create ad hoc reportsIncrease staff’s ability to create ad hoc reports Accelerate report productionAccelerate report production
Improve User InterfacesImprove User InterfacesImprove User InterfacesImprove User Interfaces
EvaluationEvaluation Created a mechanism for input and feedback from usersCreated a mechanism for input and feedback from users Created a successful and easy-to-use SSO moduleCreated a successful and easy-to-use SSO module Seem to be heading in the right directionSeem to be heading in the right direction
RecommendationsRecommendations Create a formal mechanism for validating and testing user Create a formal mechanism for validating and testing user
interfacesinterfaces Create context-specific drop-down menusCreate context-specific drop-down menus Create context-specific online helpCreate context-specific online help
Additional RecommendationsAdditional RecommendationsAdditional RecommendationsAdditional Recommendations
Build a non-CIWQS interface with ICIS-NPDESBuild a non-CIWQS interface with ICIS-NPDES
Develop a set of persistent end-to-end tests to Develop a set of persistent end-to-end tests to demonstrate that the data model and demonstrate that the data model and processes workprocesses work
Develop a deployment plan for the system and Develop a deployment plan for the system and datadata
Fund the system at an appropriate levelFund the system at an appropriate level
ICIS-NPDES Issue Must Be ResolvedICIS-NPDES Issue Must Be ResolvedICIS-NPDES Issue Must Be ResolvedICIS-NPDES Issue Must Be Resolved
ICIS-NPDES will not meet State Board’s needsICIS-NPDES will not meet State Board’s needs
There are different reporting needs for EPA and There are different reporting needs for EPA and for State Boardfor State Board
State system should cross-check violation State system should cross-check violation determinations between state and federal determinations between state and federal systems to quickly identify erroneous violationssystems to quickly identify erroneous violations
Download ICIS-NPDES data to CIWQS as needed Download ICIS-NPDES data to CIWQS as needed for State Board purposesfor State Board purposes
Evaluate Interface Options w/EPAEvaluate Interface Options w/EPAEvaluate Interface Options w/EPAEvaluate Interface Options w/EPA
Consider discharger submission usingConsider discharger submission using eDMR (file-oriented)eDMR (file-oriented) netDMR (interactive)netDMR (interactive)
Develop test plan and case(s) including EPA DMR Develop test plan and case(s) including EPA DMR data download to state systemdata download to state system
EPA has agreed to this using currently submitted data; EPA has agreed to this using currently submitted data; this should be formalized quickly to take advantage of the this should be formalized quickly to take advantage of the offeroffer
Develop End-to-End TestingDevelop End-to-End TestingDevelop End-to-End TestingDevelop End-to-End Testing
Develop and maintain a library of test Develop and maintain a library of test cases with known results that can be used cases with known results that can be used to test system and processes as changes to test system and processes as changes are madeare made
Should be implemented as part of all Should be implemented as part of all deployment plans on an ongoing basisdeployment plans on an ongoing basis
Develop Deployment PlanDevelop Deployment PlanDevelop Deployment PlanDevelop Deployment Plan
Develop a deployment plan for every Develop a deployment plan for every system update and releasesystem update and release
Should include software, hardware, data, Should include software, hardware, data, and quality control aspectsand quality control aspects
End-to-end testing should be integral part End-to-end testing should be integral part of the planof the plan
Each phase of the eSMR development Each phase of the eSMR development needs a deployment planneeds a deployment plan
Fund CIWQS at Appropriate LevelFund CIWQS at Appropriate LevelFund CIWQS at Appropriate LevelFund CIWQS at Appropriate Level
Panel is encouraged by progressPanel is encouraged by progress
Believe that changes make it likely the Believe that changes make it likely the CIWQS project will result in a system that CIWQS project will result in a system that serves the State Board’s needs for serves the State Board’s needs for accuracy, automation, and efficiency for accuracy, automation, and efficiency for the futurethe future
Encourage State Board and Legislature to Encourage State Board and Legislature to fund the development at levels that can fund the development at levels that can maintain and accelerate developmentmaintain and accelerate development
Next StepsNext StepsNext StepsNext Steps
Full report of findings and recommendations, with Full report of findings and recommendations, with performance metrics, within 2 monthsperformance metrics, within 2 months
Panel open to revisiting progress in 9 – 12 months if Panel open to revisiting progress in 9 – 12 months if askedasked
SummarySummarySummarySummary
The project has made substantial improvements and The project has made substantial improvements and is on a path to successis on a path to success
Management restructuring, rebuilding user constituencies, Management restructuring, rebuilding user constituencies, and system validation are clear successesand system validation are clear successes
Additional progress needed on data cleanup, reporting, user Additional progress needed on data cleanup, reporting, user interfacesinterfaces
Additional recommendations include:Additional recommendations include: Build a non-CIWQS interface with ICIS-NPDESBuild a non-CIWQS interface with ICIS-NPDES Develop end-to-end tests to demonstrate all functionalityDevelop end-to-end tests to demonstrate all functionality Develop comprehensive deployment planDevelop comprehensive deployment plan Fund CIWQS at an appropriate levelFund CIWQS at an appropriate level