19/15/2016

GE Healthcare R12 Upgrade Overview

- Umesh Date( Tech. Program Mgr.- R12 Upgrade)

GE HE ALTHCARE TODAY• $18 billion: global business unit of GE

• 46,000: number of employees worldwide

• $1 billion+ per year investment in R&D

• 35,800 patents filed by GE since 2000

• Core strengths in bio-sciences, medical imaging

& information technologies

• Headquartered in Chicago, USA

39/15/2016

BROAD SOLUTIONS FOR HE ALTHCARE

Diagnostic and Interventional Imaging

Patient Monitoring

Maternal and Infant Care

Anesthesia and Respiratory Care

Diagnostic Cardiology

Healthcare IT

Life Sciences

Healthcare Consulting

Global Services

CUSTOMERS

O U R K E Y C U S TO M E R S I N C LU D E :

• Healthcare systems, hospitals, and clinics

– Physicians

– Clinicians

– Nurses

– Executives and administrators

– IT managers

– Biomedical engineers

– Construction managers

• Government officials

• Pharmaceutical firms

• Genetics researchers

• Bio-science researchers

59/15/2016

GE Healthcare ERP Context

4 Pillar instances – SAP, PS & EBS ( 2)

Largest EBS instance upgraded to 12.2.4

80% of revenue transacted

53 countries

Approx. 10000 users

DB Size approx. 14 TB

150+ companies

Upgraded from 11.5.10 to 12.2.4

Our Upgrade Journey…..

Upgrade vs Re-implementR12 Technical Upgrade R12 Upgrade Plus R12 Reimplementation

Shortest

Tech upgrade to R12.2

• Customization remediation

• Integration and Bolt-on checks

• Regression & Performance Testing

• New Features: SLA, inv org, multi-org, role- based access controls

• Retire Customizations:

• Convert GL to Corporate COA

New R12.2 Instance

• Rationalize customizations

• Leverage new R12 features

• Potential for Phased Roll-out

Lowest

Lowest

Supportability

Real-time Patching

Timeline

Cost

Funct.Resource

Needs

BusinessBenefits

New

Features

Retire

Customizations

Corporate

COAs+ +

Medium - Long Longest

Medium - High Highest

Medium - High Highest

Supportability+ Re-engineering

Technical Upgrade allowed for other business priorities.

Incremental Change in Business process instead of Big Bang

89/15/2016

Initial Project Timeline

Code Remediation

2014 2015

Q3 Q4 Q1 Q2

CoE

Tech

Steady increase in program resourcing needs over time

Global

Teams

Core

Team

Planned Timeline

Resources

Leveraging a small core team to maximize value

R12 Upgrade Testing

99/15/2016

Global Functions

SVCGSC FIN Commercial ENG

IT Technical / Quality Team

GlobalManagementTeam

Senior LeadershipTeam

ProgramManagement

Region / P&L

Region/ Modality CIOs

Functional Program Manager

VP-Service

VP-ITVP- Supply

ChainCFO

CTO Regional GMsSupply chainG

SC

Reginal GMFinF

IN Sales GMC.E

.

GM Engg

EN

G

CIO

Fin Leader

IT Fin Leader

GSC Leader

IT GSC Leader

Service Leader

IT SVC Leader

ENG Leader

IT Eng Leader

Bus Integration

Business Integration Leader

Region / P&L Integration Leader

ITProgram Leader

Quality & Compliance Leader

Commercial ExLeader

Comm. IT Leader

Program Organization

SV

C

Service GMS

TechnicalProgram Manager

109/15/2016

Team Functional IT

Senior Leadership Team (SLT)

• Top Down Support – CAP / Communication

• Strategic Direction• Help with staffing engagement

• Top Down Support – CAP / Communication

• Strategic Direction• Help with staffing engagement

Guidance Management Team (GMT)

• Drive func leadership selection• Monitor progress, provide help

needed• Authorize major project variances• Manage conflicting priorities• Assist with functional staffing• Make the final Go / No-Go decision

• CIO engagement of functional leadership

• Drive IT leadership selection• Approve budget changes• Resolve cross program issues• Monitor progress, provide help needed• Make the final Go / No-Go decision

OverallProgramExecutive

• Manages SLT engagement• Organize and Lead GMT Sessions• Develops overall program strategy & organization• Manages top-level risk mitigation

ProgramManagers

• Organizes the overall program and mobilizes teams

• Drives overall program operating mechanisms, issues, and risk mitigation• Manages program Plan, risk mitigation, tollgates• Manages program budget• Manages communication plan

Business IntegrationLeader

• Creates Overall Program Test strategy• Establishes and drives test reporting• Drives cross functional integration

• Identifies simplification opportunities for future releases

Region IntegrationLeader

• Drives regional engagement through best practices and common work structure• Pivots between program, global processes and region / P&L leaders• Plans the work for the regional leads and monitors the overall region plan• Identifies top regional issues and drives to closure• Leads commercial staffing, supports functional staffing

Top Management Support

Staff Leaders

Make Go/No-go

LeadershipEngagement

Mobilize & Orchestrate

Cross-functionalConsistency &Integration

LeadRegionEngagement

Roles & Responsibilities

119/15/2016

Team Functional IT

Global Process Leader

• Recommend Go / No-Go• Assure functional coverage &

obtains resources• Validate functional test plan• Escalates critical risks• Organizational readiness

• Identifies process eco-system • Develop staffing coverage strategy and

resource plan• Staffs GE & contractor positions• Drives issue resolution• Liaison's to function

Global Track Leader

• Defines critical processes• Defines test scenarios• Manages test execution• Escalates critical issues

• Identifies testing coverage for customizations & interfaces

• Gathers data for testing• Drives fixing defects• Drives system documentation

maintenance

RegionalLeader

• Engages regional leadership and commercial teams for testing & readiness• Identifies local system integrations, country localizations, and CFD’s• Coordinates regional testing as needed – particularly where local tools are involved• Identifies regional risks and issues and drives to closure• Define the region go-live readiness (acceptance) criteria

IT TechnicalProgram Leader

• N/A • Plans & manages outage window execution

• Performs stress test & tunes system• Governs customization designs

• Drives Overall eco-system connectivity• Manages validation & compliance• Leads central break/fix for Oracle defects• Identify & modify BI fixes

OrganizationalReadiness

Process & ModuleReadiness

Run the business…Find the Bugs!

Run the Systems…Fix the Bugs!

Assure Regional Readiness

Upgrade Execution & System Performance

Roles & Responsibilities

129/15/2016

2014 2015 2016

Actual Program Timeline

NovOct Nov Dec Jan Feb Mar Apr May DecJun Jul Sep OctAug

Go LiveJan

CRP1 CRP2 UAT Validation Test

Development End to End Process Testing User AcceptanceTest

Compliance documentationTraining & Communication

DefectResolution

Defect Resolution

Key Activities:

CRP1 – Code remediation and functionality exploration

CRP2 – End to end testing and Bug fixes

UAT – User acceptance testing

Upgrade simulations to optimize the time

Back out planning and Cutover planning

Simulate Back out plans.

Communication & Training

Go-Live

6 Upgrade Simulations

139/15/2016

Upgrade completed on time – system came up 6pm Sunday as planned

Critical Business transactions exceeded daily averages (starting day 1 )

No post go-live outages

Successful January Close

Program Go-Live Highlights

Post Go-Live Executive Summary

“Just an awesome, awesome job!!!!” ~ CFO GEHC

“Best upgrade I have seen in 20 Years” ~CFO SC P&L

“As close to perfect as you could expect in an ERP upgrade” ~ VP &

CIO GEHC

149/15/2016

Key Learnings

159/15/2016

Key Learnings

ERP is not standalone 350 Interfaces ;Inbound and outbound

120 connecting systems

3 BI tools

Transactions flow from Source to ERP to destination systems

Code remediation and testing done across systems

…… But… How do we shutdown this web of systems?

Key is to understand dependencies and cross-impacts

How do we startup this web of systems?

A pilot can fly alone easily but when 15 pilots fly together, it’s a

different experience…. That’s how it feels like in the ecosystem !!

Think about your Ecosystem; not just the ERP

169/15/2016

Key Learnings

Use of tools to remediate code Automation of code remediation saved us months in the

project schedule

Code remediation activity could be run like a factory

Total number of code Objects remediated ~ 2300

…… But… We found that like all software, our tools had issues..

We missed code in ecosystem, missed SSWA objects,

missed embedded queries in apps like QA, Pricing etc.

Online patching compliance

Trust your tools but verify the results !

179/15/2016

Key Learnings

Test plan and Test coverage We executed 13000 test scenarios in CRP’s

Fantastic functional team engagement

Defect monitoring and root cause analysis helped

Exited UAT with 12 defects across all tracks.

Load testing and Regression Testing

…… But… Initially we did not test transactions which start in 11i but

end in R12. Practice period close. Get used to period close

analyzers.

Our root cause analysis did not look at underlying trends

and systemic defects…. We improved on that.

Start your Load testing early.

If using i-modules, please test them from outside the

firewall.

Test boundary conditions & exceptions !

189/15/2016

Key Learnings

Appreciate your team Any Oracle upgrade is very demanding on people

Work hours will be long and often at odd hours

Workload for different teams like developers, middleware team,

testers, users will be fluctuating.

Workload for DBA’s tends to be high all the times.

Keep morale up. Take pride in milestone accomplishments and

learn from every incident.

This is team work. Trust is key. Help in building it

…… But… Think about back ups

Think about shifting work hours

Think if you run a risk of knowledge concentration in a few

Think about your communication plan across the globe if you are

global

Remember DBA’s are the centerpiece of this project

199/15/2016

Key Learnings

Functional Users Need a Strong buy-in.

Knowledgeable super users are the key to success

Users need to test the system as they use it everyday

Reports, labels are the key

Spend ample time in communications, awareness and training.

Posters and banners at major hubs and Mfg/ Dist sites.

Think about having a dedicated communications leader

Involve them in sanity testing during simulations

Use your super users as your messengers and damage controllers

in case of small hick ups.

Think about your communication plan across the globe if you are

global

Project success = Happy Users

209/15/2016

Key Learnings

Upgrade Considerations from a DBA perspective Archive or Purge unwanted data

Refresh all MVs to reduce upgrade time

Concentrate on long running jobs

Parallelize tasks if possible

Clean up database objects before enabling ADOP (duplicate tables

etc.)

Implement SPMs and SQL profiles for long runners

…… But… Do end to end simulations

Plan for backups before critical steps

Check logs always for any unexpected errors

Follow the upgrade manual from Oracle. No Short cuts please !!

219/15/2016

Key Learnings

Cutover planning & execution We had a 1900 line project plan just for the cutover tasks

Understand the dependencies of tasks.

Understand dependencies of people. They need to trust each

other. Handshakes on tasks are important

Co-location helps a lot

Simulate like it is Production. Avoid the temptation of shortcuts.

…… But… Note every single incident for investigation later

Think user lock down is simple…it might not be.

Plan your simulations for every single failure. Don’t take things for

granted.

Practice, practice and practice….

229/15/2016

R12 Upgrade Outage planProvide high quality instance as efficiently as possible

What actions were taken?

• Practiced and improvised the shutdown process.

• Created save points which were key to BCP

• Worked with Oracle to get best practices

• Oracle fix was practiced and verified

• Conducted multiple simulations to iron out the process

• Added time for GS and introduced a functional checkout process

Save Points

What Did We Learn?

Go/No Go

Monday Tuesday Wednesday Thursday Friday Saturday Sunday24-Jan

Monday25-Jan

R12.2Code,

Patches Ch

eck

O

utSystem

shutdown

R12.2.4/Online Patching

3rd

Party,M/W

• Graceful shutdown adds 12 hours (flushing data through system)

• Issues need to be recreated. Painful.

• VCP first collection was slow

• Issues with R12.2.driver in the first 6 months vs the last 6 months

• Code migration, VCP collection, Sabrix install and Brazil localizations took 42 hrs.

• System startup was not smooth

VCP

What were the outcomes?

• Finally reduced it for 7 hrs.

• Gained ability to run multiple simulations at same time.

• VCP collections was not a bottleneck

• Reduced the time for the last phase to 27 hrs. from 42 hrs.

• Resulted in a very high user confidence

Practice to optimize the plan and understand dependencies

Outage at ~5 Days… additional time needed to ensure Quality

239/15/2016

Key Learnings

Where is your Plan B? We had a 300 line back out plan

Plan for back out from every step of the upgrade

Playbook for executive engagement in the worst case

Have Plan B incase of delays. Understand steps where you could

lose time and where you could gain time

Use the time gained wisely. Try to stay ahead and be ready for

contingencies.

Have playbooks for concurrent activities going out of sequence.

Practice your back out plans and Plan B’s to make sure they can

work.

Your Back out plans should cover the ecosystem not just the ERP.

Have a Plan B for every possible thing you can think about.

249/15/2016

Key Learnings

System Performance Comprehensive load testing

Verified performance between 11i & R12 up to the second

API and Interface performance is important too.

Watch out for first collection run for ASCP.

…… But… Watch out for performance post-upgrade

Allow time for post upgrade steps to improve performance

Simulate post upgrade performance to take corrective actions

Productions system will be faster than your test environments

Gather Stats and Table re-orgs will help. Ask your DBA’s to plan for

those pre and post Upgrade

Plan for post-upgrade performance. Don’t make it an after thought !

259/15/2016

Key Learnings

Post Go-live support Engage Production support teams during UAT

Pair the project SME’s with the support SME’s

Be ready for a high call volume

Have regional contact points by defect types for level 1 support

If you are changing java versions, be ready for login Issues.

Over communicate about go-live . Communicate not only during

the project but even after go-live.

Be ready with swat teams especially for finance.

All the best for your upgrade & hope that our learnings will make you smarter.

269/15/2016

279/15/2016

Operating Mechanism

Purpose / Objective Participants Frequency

Senior Leadership & GMT Review

Inform corporate officers of project status, issues, and help needed

Senior Leadership Team, aligned GMT members, selected team members, Global functional & IT leads, Gilles, Jason

Monthly

Program Leadership Forum

Strategic review, develop recovery plans, make course corrections

Program management, Global functions, Gilles, Region Leaders, Jason

Monthly

Program Review Review scorecard status at end of week; weekly status report preparation

Program management, Global functions, Gilles, Jason

Weekly

Global functional Review

Manage track progress, issue escalation & mitigation, update on program information

Global functions, track leads, region integration leader, break/fix manager

Weekly

Region LeadershipReview

Manage region progress, issue escalation & mitigation, update on program information

Region leader, Gilles, Brian,Optional: Functional leads

Weekly

Training & ImprovementForum

Add / update training & improvement lists Functional Leads, Gilles, Brian, Architecture leader

Bi-Weekly

IT Technical Integration

Session

Manage IT Platform progress, issue

escalation & mitigation, dependencies

IT Technical Team Weekly

Defect ManagementReview

Defect assignment, status, critical issues by module

Defect manager, IT track leads, BACOE architects, BI COE, GAMS

Weekly

Project Plan Review Progress vs plan update on activities Program management, Gilles, Jason

Weekly

Operating Mechanisms