| ©2012, Cognizant
©2012, Cognizant
CBC Practices Collaboration: Elizabeth Lerner, Partner – Program Management Consulting Susan Schanta, Associate Director - Program & Quality Consulting September 18, 2012
New Techniques to Elevate Quality Assurance to Program Assurance
| ©2012, Cognizant 1
Agenda
2 Introduction
3 Program Management
7 Program Assurance - Extending QA Principles
10 Program Schedule Standards
15 Managing Environmental Limitations
11 BRD Ambiguity Reviews
14 Architectural Reviews
17 UAT Strategy
18 Quality Measures
20 Balanced Scorecard
| ©2012, Cognizant
Introduction
• An IT-enabled Transformation should recognize that these are typically business-driven, not technology-led.
• A Program drives the conversion of
strategic intent into reality through the co-ordination of multiple projects focused on common objectives.
• Program Management stretches the project management model through a heightened level of cross-team coordination, cost management and quality controls to deliver a predictable outcome.
• Controls are needed to deliver
strategic, mission critical, and high risk programs of transformation successfully.
• Introducing a quality management
model into the program early reduces risk and increases flexibility to respond to challenges, and create efficiencies, preventing of do-over’s.
Program Management
Program Assurance
Trends Analysis
Value Management
Planning & Strategy
Architecture Project
Management
Implementation Management
Process Modeling
Applications Information
Infrastructure Planning
Gap Analysis
Project Monitoring
Resource Management
Risk Management
Strategy Formulation
2
| ©2012, Cognizant
Program Management
GOVERN
Build and implement best-practice project governance and controls while minimizing bureaucracy
SUPPORT
Help project teams by providing hands-on PM expertise. Coach and mentor project teams and become a center for project knowledge and learning
ASSURE
Provide independent challenge to projects. Assist
in the identification and management of cross-project
dependencies to ensure successful delivery
INFORM
Establish regular and accurate project reporting to provide ‘early warning’ management information
Program Management
Office
Deliver Business Solutions Successfully On Time, to Cost, and Quality
Enhance Organizational Capability Resource Management, Delivery, and On Demand Reporting
Driving the Strategic Agenda
A Project is an endeavor funded by finite resources with a finite duration and a specific outcome. A Program is a group of projects aligned by a single objective or outcome that can be better managed collectively than individually.
3
| ©2012, Cognizant
Program Management Dimensions
Review & Reporting
Risk & Issue Management
Program Assurance
Resource Management
Cost Management
Benefits Management
Communication & Business Input
Program Standards
Vendor Management
Governance & Organization Ensure projects are properly set up for success and business need is clearly articulated. Implement the overall, pragmatic, and consistent framework for managing the portfolio of client initiatives.
Consolidated view of projects. Maintain key management information from CFO level down. Report key actions and decisions required
Planning & Tracking Implement consistent approach for planning and tracking projects. Provide consolidated view of the portfolio roadmap
Implement portfolio-wide risk planning and management process to ensure the mechanism for dealing with issues is speedy / robust
Implement an independent challenge and assurance function that helps identify and resolve key issues of common concern and threats to benefits across the program
Build and maintain an overall resource plan to align the supply of resources and skills to the demand of the project portfolio
Program finances are managed, tracked, and reported across the Project Portfolio to ensure the cost of work remains in budget
Track and report the delivery of benefits arising from the portfolio and the value add to the organization
Implement the discipline of engaging and managing business input to projects. Develop and implement a communication plan
Set and maintain standard approach, tools, and techniques to support program delivery. Ensure standards adherence
Ensure rigor in the procurement and management of subcontracted third party services and products.
Program Management provides real management value by striking a balance between challenging and supporting projects.
Program scope clearly articulated. Implement consistent, pragmatic framework to management a portfolio of projects
4
| ©2012, Cognizant
Program Assurance
Plan > Assure > Control Purpose: Define quality standards in advance; review deliverables against standards; continuously question whether delivery is in line with the overall goals.
Benefit: Ensures that what the program delivers will achieve the overall aims
Agree
Agree to expectations up from to ensure that program deliverables are fit for purpose.
Record the agreement in a deliverable description record including what will be delivered, target date, format / template, and to what quality standard.
Program Quality Director helps the PM define the deliverable, apply knowledge of best practices, and ensure the end deliverable will meet the program’s requirements.
Question
Throughout the lifecycle and at all levels of the program, ask:
- Does this align with the program’s vision?
- How will this help achieve the program’s goals?
Test
When a deliverable is completed, it must be assessed against the quality standards set forth in the deliverable description.
The Program Quality Director will facilitate the process to ensure an appropriate independent reviewer has the opportunity to ‘test’ the deliverable and sign-off.
5
| ©2012, Cognizant
Polling Question
Program Assurance includes audits for
Project Schedules, Resource Management, Cost Management
Requirements Traceability, Environment Definition, Construction Readiness
Code Turnover Requirements, System Test Process
All of the Above
6
| ©2012, Cognizant
Program Assurance - Extending QA Principles
Program Assurance provides quality governance within the Program Management framework by introducing independent measurements to track and report against deliverables. Measurements are captured through frequent audits and risk assessments, making it possible to improve program execution.
Quality Assurance
• Requirements Traceability
• Test Coverage
• Test Execution Pass / Fail Rates
• Defect Management
Program Assurance
• Program Standards
• Planning and Tracking
• Cost Management
• Resource Management
• Environmental Limitations
• Risk / Issue Management
• Quality Gates
BRD Ambiguity Reviews
Construction Readiness
Architectural Reviews
• UAT Strategy
• Zero Uninspected Defects at Release
7
| ©2012, Cognizant
Techniques to Introduce and Maintain Controls
• Establish and maintain controls
• Ensure program management information is up-to-date and available
• Identify and manage business stakeholders
• Consolidate and analyze data across projects
• Produce regular and ad-hoc reports
• Track and realize program benefits
• Ensure all project plans are properly integrated
• Ensure plans remained aligned and dependencies and cross-project issues and risks are effectively managed
• Ensure project plans are complete, feasible, and all options are considered
• Ensure current status and outlook is realistic
• Ensure all issues are surfaced and addressed
• Advising and supporting individuals (especially PMs) in difficult or
unfamiliar circumstances
• Support the business through effective change management
• Conduct reviews (e.g. risk or reviews, reviews)
• Facilitating planning workshops
• Leading / participating in taskforces teams to resolve issues
Process and data
• Hard techniques; the most visible PMO deliverables
• Earns the right to do other things - necessary but not sufficient
• Set up right and get running smoothly
Content and people
• How experience is leveraged to add major value
• Earns the right to advise at a senior level
• The key to personal development through the project office
• Less obvious, need to search out
Controlling
Reporting
Integrating
Challenging
Coaching
Supporting
Controls are needed to deliver strategic, mission critical, and high risk programs of transformation successfully. Effective control is required to minimize disruptions and ensure that the program will meet its aims within time and cost constraints.
A set of controls handle the different challenges of program management, underpinned by a robust governance framework.
8
| ©2012, Cognizant
Project Lifecycle - Fundamental Building Block
Approval to proceed
Project Management Stages
Project Management * Deliverables
Program Related Processes
• Project Brief • Statement of Requirements
• Business Case
• Project Charter
• Management Plan
• Cost Plan
• Detailed Schedules
• Risk Assessment
• Quality Plan
• Create Balanced Scorecard
• Progress Reports
• Cost Reports
• Change Requests
• Risk / Issue Registers
• Implement Plan
• Quality Assurance
• Assurance Audits
• Post Implementation Review (PIR)
• Benefits Dashboard
Pre-feasibility Feasibility Management Planning
Execution and Delivery
Review and Closure
Start-Up
Work prioritization and approvals process (portfolio management)
Resource scheduling and allocation process
Annual budgeting / forecasting process
* Technical deliverables excluded from diagram
Approval to proceed
Approval to proceed
A consistent project lifecycle is the fundamental building block for control
Monitoring and Controlling
9
| ©2012, Cognizant
Program Schedule Standards
How do you measure? Consistent structure in project schedules results in consistent work effort measurement in the Integrated Project Plan. • Consistent Structure for Tasks
Across Phases - Requirements, Design, etc. By Team - Sub teams within departments By Artifact - SDLC documents
• Options
Tracking artifacts in separate matrix Separate stakeholder detailed project plans for
day by day tracking of low level tasks
Make Program Level Decisions Ensure a consistent measurement of program progress
10
| ©2012, Cognizant
BRD Ambiguity Reviews
Ambiguity Type
Search for What to Look for Comment Feedback
Term Appropriate Not quantifiable Ambiguous - Define appropriate
Term As required Not quantifiable Ambiguous - Define as required
Term Difficult Not quantifiable Ambiguous - Define difficult
Term e.g. Versus i.e.
Misuse of i.e. versus e.g. i.e. means "that is" I work most effectively at small coffee shops i.e. Starbucks… e.g. means "for example" I like melons e.g. honeydew, watermelon, cantaloupe, etc.
Should be i.e. which means for that is Should be e.g. which means for example
Term Error Error message or error reason not stated… What is the error message?
Term Error Error condition stated but not what causes the error condition to occur?
What is the error condition that causes this scenario?
Term Flexible Not quantified Ambiguous - Define flexible
Term If Applicable
Not quantified Ambiguous - What determines applicability?
Term ly Search for "ly" to find words that end in "ly" where the words are not quantifiable because each person can apply their own meaning. Examples: frequently, accordingly, efficiently, normally, similarly, commonly, etc.
Define _______ Examples: Ambiguous - Define frequently Ambiguous - Define efficiently
Term Maximum Not quantifiable Ambiguous - Define maximum
Term Minimum Not quantifiable Ambiguous - Define minimum
• Business Requirements define what will be delivered in the release.
• Ambiguity Reviews provide a measurement of whether scope documents are quantitative, clear, correct, and complete. Ambiguity reviews help eliminate defects in later phases by building quality into the scoping effort.
11
| ©2012, Cognizant
Sample Ambiguity Review
12
| ©2012, Cognizant
BRD Ambiguity Reviews
275
47 454
7
78
624
3
14
136
0
200
400
600
800
AmbiguousTerm
ClarificationNeeded
Field/Naming
Consistency
Glossary
Grammarand Wording
IncompleteRequirement
PossibleDuplicate
PossibleMissing
Requirement
UnclearRequirement
New Accounts BRD Ambiguity Review by Category
141
240
54
27 64
274 79 45 67
50
0 100 200 300 400 500
Ambiguous Term
Field/NamingConsistency
IncompleteRequirement
New Accounts Most Frequent Ambiguous Findings
(Ambiguous Adjective) All
Date/Time?
Define specific
Should Title be included?
SSN/TIN Field consistency
What are the valid values? Is this a limited list?
What are the valid values? Is this a limited list? What about Provincefor Canada?
• Can a non-SME understand the requirements?
• Are the requirements quantifiable?
• Can the design be measured for requirements fulfillment - Yes or No?
• Does the reader need to have all requirements available to understand the context of the requirement?
13
| ©2012, Cognizant
Architectural Reviews
Traditional Approach
• Architecture Documents Created
• Architectural Review Board during Design Phase High Level Design Functional Specifications Detailed Technical Design
Enhanced Approach - Construction Phase Reviews
• Selective Criteria Highest Impact on Failure Highest Volume Most Complex Algorithm
• Format White board walkthrough Q&A Session
14
| ©2012, Cognizant
Managing Environmental Limitations
• Establish working list of environmental limitations based on existing systems
• Meet with key stakeholders to document limitations
• Facilitate meetings to address solutions to limitations
• Escalate budget requirements for licenses, resources, etc. to Project Manager
15
| ©2012, Cognizant
Polling Question
When does UAT planning occur in large programs?
Incrementally throughout the lifecycle of individual projects
Only at the end of the program
Across all project lifecycle phases and at the end of the program
After Construction but before text execution
16
| ©2012, Cognizant
UAT Strategy
Shift User Acceptance Testing Earlier
Project Level: Early validation in System Test phase to confirm requirements fulfillment for deliverables such as reports, screens, calculations and business rules. This also offers an opportunity for the business to refine their workflow processes in response to the new or updated application.
Program Level UAT: Traditional E2E validation to validate the life of a transaction.
Event Calendar
A predefined schedule to help define, manage, and report on UAT test execution. The calendar supports the operational process of test execution by providing a shared understanding of the schedule and expectation for business user time commitments, scheduled system activities such as environmental updates and running batch jobs for electronic file processing. Run Books
Compilation of transactions that will be executed during a specific UAT cycle in the Event Calendar. Run Books assist the business users in organizing their test execution content and help control the order of processing. Run Books also provide a reference for the UATC and business to ensure everyone understands the goal of what will be executed in a given time period.
17
| ©2012, Cognizant
Quality Measures
18
| ©2012, Cognizant
Quality Measures
67% 33%
Project 1 Pass Fail
45% 55%
Project 2 Pass Fail
82%
18%
Project 3 Pass Fail
19%
81%
Project 4 Pass Fail
34%
66%
Project 5 Pass Fail
89%
11%
Project 6 Pass Fail
19
| ©2012, Cognizant
Balanced Scorecard
The Balanced Scorecard is a strategic performance management tool to track progress against plan.
• Point in time view • Mixture of financial and non-
financial measures • Tracks execution • Monitors quality • Highlights risks and issues
Getting Started
• Set expectations low • Define pain points and
milestones tracked across the program and / or project
• Determine available data inputs • Create initial scorecard at the
project level and build into a program level scorecard
• Use Excel to model appearance • Automate when stable
20
| ©2012, Cognizant
Balanced Scorecard
Program Level
• % Work Complete • Budget - Estimate vs. Actual • Program Tollgates - Final Dates • Project - Artifact Deliverables • Top 3 Risks • Top 3 Issues
Requirements
• Ambiguity Review Results • BRD to FSD Requirements Mapping • Requirements Assigned by Team • Test Case Coverage to Requirements • Program Completion of FSDs
Testing
• Test Execution % Complete • Test Case Pass / Fail Rates • Functional Modules Impact • Automation % Coverage • Performance Benchmark Results
Defect Management
• Defect Fix Rate Trend to Complete • Defect Root Cause • Defects by State • Defect Trend by Severity • Regression Defects • % Regression Defects - Automation
21
| ©2012, Cognizant | ©2012, Cognizant
Questions
22
| ©2012, Cognizant 23
Speakers Bios
Elizabeth Lerner, Partner - Program Management Consulting, Cognizant
Elizabeth (Liz) has spent 30 years managing large-scale change programs as a Strategic Global Program Director. In this capacity, she has been leading, managing, and delivering mission-critical results to top-tier global corporations. Liz brings with her a vast experience while working in various industries, successfully leading and delivering strategic change programs, overseeing initiatives of large-scale scope and complexity resulting in significant business value, top and bottom line growth, global industry leadership, business expansion, acquisitions and divestitures, operational efficiencies, and organizational restructuring. Liz has proven experience in strategy consulting for Fortune 100 clients. She is recognized for delivering large-scale programs on time and within budget and scope, thereby consistently meeting and exceeding customer expectations.
Liz can be contacted at [email protected].
Susan Schanta, Associate Director - Program & Quality Consulting, Cognizant
Susan has spent 20 years managing large-scale Program and Quality Assurance programs in new ventures for global Fortune 500 companies. She brings with her vast experience spanning the financial, healthcare, and retail domains. She has led several corporate initiatives in business optimization, onshore-to-offshore transitions, and QA enterprises including SDLC management, automation, and performance. Susan’s expertise in the industry-best practices and project execution has helped companies achieve their goals. Her expertise goes into niche skills such as implementing lifecycle disciplines through change management, sharply reducing budget variances, improving performance of tiered applications, introducing new disciplines for UAT execution to achieve improved quality and business workflow processes, and devising calibration methods to measure these achievements.
Susan can be contacted at [email protected].
| ©2012, Cognizant 24
References
Project Management Institute
http://www.pmi.org/
Project Management: 6 Best Practices for Ensuring Software Implementation Success
http://www.cio.com/article/683570/Project_Management_6_Best_Practices_for_Ensuring_Software_Implementation_Success
The Ambiguity Review Process
http://benderrbt.com/Ambiguityprocess.pdf
Eliminating Ambiguity from Your Requirements
http://www.modernanalyst.com/Resources/Articles/tabid/115/Default.aspx?ArticleType=ArticleView&ArticleID=989
Building & Implementing a Balanced Scorecard: Nine Steps to Success
http://www.balancedscorecard.org/BSCResources/TheNineStepstoSuccess/tabid/58/Default.aspx
Balanced Scorecard & The Project Manager
http://www.ittoday.info/Articles/Balanced_Scorecard_and_Project_Manager.pdf
Top Related