Day 201-20-201620-20-201705-20-20pearl-201-20-20deep-20kamal-20singh-131008015752-phpapp02
Transcript of Day 201-20-201620-20-201705-20-20pearl-201-20-20deep-20kamal-20singh-131008015752-phpapp02
1
Strategic approach to ascertain accurate decisions
after unplanned service outage in Telecom operations
Deep Kamal Singh
27 September 2013
2
Introduction
• Changes in System are Inevitable
• Uncertainty, Variations, Deviations are Un Avoidable
• Prime objective of Business – Profit & Cost Control
• Factors that control cost of variation/deviation
– Number of interdependent systems/services disrupted
– Knowledge Gap between cross functional domains
– Time spent in reaching a decision
• Exponentially proportional to Stake holders involved
• Directly proportional to number of services/systems disrupted
• Need of the hour, Business Need
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
3
• Organizations are facing a complex and
risky environment
– A “do more with less”
– “Tight” economic climate
– Expanding global priorities
– Necessity to enable innovation
The Strategic Approach
Identify All Services
Identify Revenue of
every service
Define Threshold
Form Crisis Management
Team
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
Identify All Impacted Services
Estimate outage & Calculate Revenue
Consolidate Options
Decide
• Delay
• GO Live
• Fall Back
Planning
Execution - Handling Crisis
Organizations are facing a
complex and risky environment
– A “do more with less”
– “Tight” economic climate
– Expanding global priorities
– Necessity to enable innovation
4Presentation Title
Knowledge Area
Processes
Initiating
Process
Group
Planning Process Group Executing Process GroupMonitoring &Controlling
Process Group
Closing
Process Group
Project Integration
Management
4.1 Develop
Project
Charter
4.2 Develop Project Management Plan 4.3 Direct and Manage Project
Execution
4.4 Monitor and Control
Project Work4.6 Close
Project or
Phase 4.5 Perform Integrated
Change Control
Project Scope
Management
5.1 Plan Scope Management 5.5 Validate Scope
5.2 Collect Requirements 5.6 Control Scope
5.3 Define Scope
5.4 Create WBS
Project Time
Management
6.1 Plan Schedule Management
6.7 Control Schedule
6.2 Define Activities
6.3 Sequence Activities
6.4 Estimate Activity Resources
6.5 Estimate Activity Durations
6.6 Develop Schedule
Project Cost
Management
7.1 Plan Cost Management
7.4 Control Costs 7.2 Estimate Costs
7.3 Determine Budget
Project Quality
Management8.1 Plan Quality Management 8.2 Perform Quality Assurance 8.3 Perform Quality Control
Project Human
Resource
Management
9.1 Plan Human Resource
Management
9.2 Acquire Project Team
9.3 Develop Project Team
9.4 Manage Project Team
Project
Communications
Management
10.1 Plan Communications
Management10.2 Manage Communications
10.3 Control
Communications
Project Risk
Management
11.1 Plan Risk Management
11.6 Monitor and Control
Risks
11.2 Identify Risks
11.3 Perform Qualitative Risk Analysis
11.4 Perform Quantitative Risk
Analysis
11.5 Plan Risk Responses
Project
Procurement
Management
12.1 Plan Procurement Management 12.2 Conduct Procurements 12.3 Control Procurements 12.4 Close
Procurements
Project Stakeholder
Management
13.1 Identify
Stakeholders 13.2 Plan Stakeholder Management
13.3 Manage Stakeholder
Engagement
13.4 Control Stakeholder
Engagement
5
Agenda
Introduction
Activities and Phases
Being prepared for uncertainty
Handling uncertainty
Theory in practice
In Conclusion
Q & A
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
6
Activities and Phases
• Complexity Assessment is first step to prepare for
uncertainty
• Concise definition of Decision makers for every stages
• Knowing Phases of activity – Strategic approach to take
next step under unplanned outage/failure depends upon
the phase
– 3 phases of any activity in simplistic view
• Preparation
• Execution
• Operation
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
7
Activity Complexity
1 2 3 4 5 6 7
Complexity 1 4 9 16 25 36 49
Teams 1 2 3 4 5 6 7
0
10
20
30
40
50
60
Co
mp
lex
ity
Activity Complexity Vs. Systems Involved
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
• Complexity of activity
– Exponentially proportional to
number of Inter-dependent
systems involved
– Directly proportional to
number of cross functional
teams involved
– Directly proportional to
Allocated duration versus
tasks list ratio
8
Impact of failure during Activity phases
Presentation Title
Planning Phase
• Delayed Activity timelines
• Budget overhead
• Resources re-alignment
• Impact on cost to business
• Man hours invested in
preparation, development,
planning, testing etc.
• Business loss due to delayed
launch
• Delay overhead cost on
logistics
• Cascading delay on other
planned activities
Execution Phase
• Increased service downtime
Sudden revenue loss
• Impact on cost to business
• Direct revenue loss
• Customer dissatisfaction
Unpredictable future
damages
• Increased calls to customer
care and support systems
• Resource working in
extended hours
Unpredictable future issues.
• If Fallback is decided Total
investment sunk.
Operation Phase
• Live Service downtime
Sudden revenue loss
• Impact on cost to business
• Direct revenue loss
• Customer dissatisfaction
Unpredictable future
damages
• Increased calls to customer
care and support systems
• Unplanned additional Man
hour investment
• Resource working in
extended hours
Unpredictable future issues.
9
Agenda
Introduction
Activities and Phases
Being prepared for uncertainty
Handling uncertainty
Theory in practice
In Conclusion
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
10
Activity Planning
• The present day Approach
– Preparation, Review and approval of detailed step wise
execution plan
• Estimation of planned outage duration.
– Business teams analyzes and calculate total cost of activity
to business
• Including cost of planned outages, consequential costs, logistics, HR overhead and so on.
– Every step of activity is executed, monitored and controlled if
everything goes ‘as per plan’
• Shortcomings – We can not predict what may go wrong and
thus we can not be 100% ready for unplanned outages
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
11
Planning for uncertainty
Being prepared for something that can not be planned – During
Preparation Phase
• Identification of all system and services
• Evaluation of revenue earnings of each services
• Formation of Crisis management team
• Unanimous view of Project managers, Activity owners, Team
leads and Business teams over
– Revenue potential of each service
– All factors that impact business for each service
• Documentation of the unanimous view – “ Service Impact
Matrix”
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
12
Preparing Service impact matrix
• “Service impact matrix” - single document of reference to assess
impact to business during outage
– List all services which are going to be affected due to the
planned activity
– Include subscriber/user/consumer base of every service
– Include time based distribution of revenue earnings of each
service,
– Include direct and indirect cost to business for each service if
failure is encountered at any time duration
• Service Impact Matrix must be finalized, reviewed and approved by
all stake holders.
– Saves time to estimate cost of outage to business during crisis
– Prevents lengthy review/approval process
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
13
Example Service impact matrix
SNO Service Name
Subscriber
Base
(In Mn)
Hourly Revenue Potential [in 100K Rs.] Indirect cost to business
H01 H02 H03 H04 H… H21 H22 H23 H24
Customer
Care
Overhead
Dependent
Services
impact
Customer
Satisfaction
Impact
1 Voice Calls 14.34 …
1.1 Local Home network 14.34 …
1.2 Local cross network 14.34 …
1.3 National Long Distance 09.60 …
1.4 International Long Distance 00.12 …
2 SMS 14.34 …
2.1 Local 02.21 …
2.2 National 02.21 …
2.3 International 00.01 …
3 Data Usage 01.90 …
4 Paper Recharges 08.14 …
5 ETOPUP 11.89 …
6 USSD Services 14.34 …
6.1 Subscriber Info 14.34 …
6.2 Subscription management 04.30 …
6.3 VAS Services 01.80 …
7 Subscriber Life Cycle 14.34 …
7.1 Subscriber Activation 0.03 …
7.2 Subscriber Churn 00.01 …
7.3 Service management 00.01 …
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
14
Agenda
Introduction
Activities and Phases
Being prepared for uncertainty
Handling uncertainty
Theory in practice
In Conclusion
Q & A
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
15
Identifying affected services and subscriber
base
In case of any unexpected outage we need to find out total impact
to business, thus
• First and foremost, get update from all teams and consolidate
list of services that are affected.
• Classify level of disruption for each impacted service
– Intermittent, Partial or Total
• Estimate outage duration and dependency factors for the
impacted service
• Consolidated outage stats for each impacted service in a single
view for comparison and correlation.
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
Considering the revenue impacting factors associated with the activity, below
equation can be deduced
If
R(S1), R(S2)…..R(Sn) = Avg. revenue per user per minute of Service1,2,…n
O(S1),O(S2)…..O(Sn) = Total estimated outage duration of Service1,2,…n
Sb(S1),Sb(S2)….Sb(Sn) = Estimated percentage of affected subscriber base
during outage of Service1, 2,….n
Su(S1), Su(S2)…..Su(Sn) = Total users of Service1, 2, ….n
Then
Total loss to business during service outage =
∑ [ R(Sn) x O(Sn) x Sb(Sn) x Su(Sn) / 100 ]
16
Deciding next course of action
• Identify impacted services
• Calculate Revenue Impact
– Consolidate outage stats of all affected
services
– Correlate available data with corresponding
details in Service Impact Matrix
• Assess Variable and Dependency factors
• Mathematically deduce all possible next steps
to formulate “The Decision Matrix”
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
17
Formulating Decision matrix
• The Decision matrix
– Mapping of Options vs. Risk vs. Revenue Impact vs. Variance factor
– Crisis Management team to prepare
• Cross functional managers and tech leads to tabulate
domain specific next step of actions
• Categorization of options
– Track 1 :: Go-Live
– Track 2 :: Delay
– Track 3 :: Fall back
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
18
Finalizing Decisions
• Ignore – impossible, unattainable, high risk options
– Rule out options which are
• least possible,
• have highest risk,
• have highest number of variable factors
• Final time estimation
– Get ‘Time to implement’ from respective teams
– Readiness Confirmation – technically and logistically
• Final out put of crisis management team:-
Decision matrix with final concluded set of options and details of
‘time to implement’ presented to apex decision makers
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
19
Agenda
Introduction
Activities and Phases
Being prepared for uncertainty
Handling uncertainty
Theory in practice
In Conclusion
Q & A
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
20
Case study : Activity Details
• Billing System upgrade in 14 telecom circles of a leading telecom operator in
India
• The activity requires complete outage of billing system for 8 hours, Since
billing system stays unavailable, all connected services (as listed) also face
subsequent downtime:– Voice calls – Local / National Long distance / International long distance
– SMS
– Data Browsing
– Real-time data charging
– Recharges – Voucher recharge and E-topup
– USSD
– Unified Subscriber life cycle management – Activation, Churn, daily jobs, and other offline process.
– Business reporting – MIS
• The complex activity involves changes at many functional systems
– High level of coordination and control required between Cross functional technical teams.
– Due to numerous Inter-dependent systems, outage of one system or service may result in consequential outage of other services also
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
21
Case study 1
• Subscriber : 5.4
• Activity: Core billing system upgrade
• Issue during execution: ETOPUP not working with new version
– Important to note here that multiple rounds of testing on test-bed and rehearsals were conducted which showed no sign of issues earlier.
• Technical teams started investigating and fix the issue ASAP.
• Business team kept waiting for updates from Technical team to decide next step.
• All possible next actions were greatly debated without reaching a final conclusion
– ETOPUP : DELAY
• Given estimated time to fix the issue was 2 Hours
– Revenue assurance team : FALL BACK
• Cancelling complete activity and reinstating old version billing system
– Billing system team : GO LIVE
• leaving ETOPUP team to fix “their” issue separately
• Business team: absence of conclusive data No decision was taken
• Finally after 95 minutes of extended downtime, ETOPUP issue was resolved and then
upgraded version of billing system was made live.
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
22
Evolution of strategic Approach
• Learning from failure – What took time to reach a decision?
– Lack of information to take a decision
• “What to do next” is Biggest Time eater
– Lack of predefined delay threshold for critical milestones
– Lack of designated managers to manage crisis
• Biased Decision Making is biggest Hindrance to consensus
• Absence of a dedicated Crisis management team.
– Lack of clarity on activity owners, team leaders, business owners and
decision makers
• Dedicated team lead/managers always suggest biased options which favor their own team
– Lack of methodical approach to ensure quick decision making
The learning drawn from events of site 1 became foundation for evolution
of strategic approach which is discussed in this presentation.
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
23
Case study 2
• Subscriber Base : 9.28 million
• Activity Details: Same activity as Case study 1
• The Issue: Once new version of billing system was made live for test traffic
– It was identified that some of call handling units were not functioning
• When call routed on those units voice calls, video calls, SMS, USSD
were failing
– Other units were functioning without any issues
• Crisis management team swung in action and as a first step it was deduced
that 20% of units were not functioning
– This meant Voice calls, SMS and USSD requests were impacted by 20%.
Important to note that the Crisis management team was doing all coordination
and fact gathering with cross functional technical and business teams, the
Technical teams were dedicatedly focused to resolve the issue.
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
Continued…
24
Case study 2
• Total Impact identified Service impact matrix referred
Assessment
– Revenue loss when Billing system is made live with 20% failures in 3 service
– Cost of delaying the launch – Which meant 100% subscriber base will not be able
to use ANY service until the issue is fixed
– Cost of fallback was also compared with above two costs
• 3 Tracks Options (Delay, Go Live, Fallback ) consolidated
– Presented to Business owners and Function heads
• Decision Taken
– Within the duration of 45 minutes allocated for system testing before launch –
Business team successfully reached a decision to remove those 20% units from
network (so call will not land on them) and make the system Live as per schedule
Even with unexpected variations encountered during critical launch
schedule, the strategic approach helped reaching a right decision
methodically
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
25
Agenda
Introduction
Activities and Phases
Being prepared for uncertainty
Handling uncertainty
Theory in practice
In Conclusion
Q & A
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
26
Identify impacted services
Identify impacted subscriber/user base
• Estimate probable service outage duration for each impacted service
Calculate total revenue impact against each impacted service
Evaluate indirect, additional and future cost to business due to service outages
Formulate list of possible next steps or possible options.
Identify variable (uncertain) factors and Risk associated with each concluded option.
Prepare Option vs. Risk vs. Revenue Impact vs. Variance factor matrix having each of 3 tracks
• Track 1 - Go-Live
• Track 2 – Delay
• Track 3 – Fall back
From every track rule out options which are
• Least possible
• Have highest risk,
• Have high variable factors.
Evaluate time to implement for every option.
Present Decision makers with Tracks containing concluded set of options and details of ‘time to implement’ for each
option.
Conclusion
27
Thank You
Deep Kamal Singh
Project Manager – BSS APAC
Comverse Network Sys.
+91-9971003585
Shay Shargal, PMP
CEO, RBS Projects
+972-52-23-11-11-0
www.rbsprojects.co.il
Past President – PMI Israel Chapter