PG&E CorporationIshaq Mohammed/Anil Saka

Online and Middleware Application Operations

2

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

3

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

4

Company Profile

PG&E SERVICE AREAIN CALIFORNIA

• Headquarters in San Francisco, California

• Provides energy to approximately 15 million people

• 70,000 square mile service territory

• Total Assets: $50 billion

• 2012 Net Income: $816 million

• 2012 Capital Expenditures: ~$4.5 billion

5

IT Overview - Infrastructure

• 280+ WebLogic Domains (versions 9.2 through 12.1.1)• SOA Suite & Oracle Service Bus (OSB)

• WebLogic Portal & Integration

• 970+ WebLogic Servers

• 205 Databases (versions 10g through 11.2.0.3)

• 500+ host servers running Linux and AIX

• Microsoft SQL Server, Apache HTTP Server 2.2

• 2,000 IT employees - 15 involved in middleware management

6

IT Overview – MyEnergy Business Application• MyEnergy application most

critical from business perspective

• Account management (e.g. customers can monitor /manage energy use, pay bills online)

• Promotion of energy-savings programs

• Scheduled outages of PG&E services

• 2.2 million active users

• 6,000 users logged in hourly during peak times

7

IT Overview – MyEnergy Architecture Diagram

8

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

9

Challenges

• High cost in manually building 30+ middleware environments each year (~$2.5 million)

• 20% increase in WebLogic landscape over next 12 months

• What worked in pre-production sometimes failed to work in production

10

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

11

Solution: Oracle Enterprise Manager to Provision Middleware Environments

• Modernizes the build process

• Guarantees speed to market delivery

• Eliminates errors

• Reduces labor cost and time spent

• Provides out-of-box monitoring

12

Oracle Enterprise Manager Cloud Control 12c Deployment

• Implemented four environments for Cloud Control 12c – dev, test, stage and production

• Production environment for Cloud Control 12c: Management Service 12.1.0.3 Release 3 on Redhat Linux with

Active-Standby high availability configured

Management Repository in two node RAC DB 11.2.0.3 on Redhat Linux

Management Agents 12.1.0.3 Release 3 on Redhat Linux and AIX

Fusion Middleware Plug-in 12.1.0.5

13

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

14

Middleware Use Cases

WebLogic Server 12c (WLS) version 12.1.1Oracle Service Bus 11g (OSB) version 11.1.1.6WebLogic Portal 10g (WLP) version 10.3.4Small, Medium Or Large Deployment

15

Prerequisites

• Install and start Oracle Management Agent on new server

• Install required packages on the OS, create OS user and group(webLogic)

• Enable passwordless (ssh keys) connection between Oracle Management Service and new VM

• Update the /etc/sudoers with: weblogicALL=(root)/u01/app/Oracle/product/agent12c/sbin/nmosudo 

16

Middleware Provisioning

• Discover reference Oracle Home/domain

• Create provisioning profile in software library

• Create directives in software library

• Edit predefined deployment procedure

• Test deployment procedure from console to capture input parameters

• Create wrapper script which calls emcli “submit_procedure”

• Enable end users to make requests using PG&E’s Provisioning Portal

17

Provisioning Profiles

18

Directives

• Provisioning Profiles Capture “Standard” Files and Artifacts of WebLogic Domain.

• Additional Files Specific to Line of Business must be moved to Target Host(s) with File Transfer Step or Directive.

• With Directives you upload required Files to Software Library along with script to move and/or Manipulate Files, called Main file

• There are other uses for Directives. However, this is how we primarily used them

• Be sure to check “Use Most Recent Version” check box, else your procedure keeps calling older version

19

Sample Directive

20

Customized Deployment ProcedureOracle-provided deployment procedure named "Provision Middleware" customized to accommodate PG&E requirements Custom Directives (Described Earlier), or Shell Script that Execute Shell Commands and/or call WLST We Generally Added One Directive and 3 to 4 Shell Scripts to Customized Procedures Shell Script Cannot Exceed 4096 Characters. Can Use Runtime Variables in Scripts, e.g. ${target.DOMAIN_HOME_DEST_ADMIN_HOST} Steps Can Have Conditions (e.g. execute only on remote servers)

21

Sample Deployment Procedure

22

Options to ProvisionProvision using console:• Several clicks and repetitive

process

Provision using EMCLI:• Capture once and execute many

23

EMCLI Wrapper Script• Generated Properties Files Made Generic by Substituting Actual Values with

Specific Keywords:

<Hostname> -> MyHostName <Domain Name> -> MyDomainName <DBSid> -> MyDBSid

• Wrapper Scripts Creates Copy of Properties Files and Substitutes New Values Passed on Command Line

• Removes Copy When Completed

• Provision New WebLogic Portal Domain Example: sh ./emcliWrapperDB.sh "DB_DATA" "S" “database-host“ "Linux" "WLPDBS" sh ./emcliWrapperWL.sh WLP S EIWLP wlp-mw-host database-host WLPDBS

24

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

25

PG&E Provisioning Portal

• Operating Systems

• SAP Environments

• Oracle Databases

• Oracle Fusion Middleware Environments

Capable of Provisioning:

26

PG&E Provisioning Portal Processes• PG&E has built a portal for request intake and complete orchestration of

the provisioning process

• Provides central interface to end users

• Makes REST calls to Pureflex System for OS provisioning

• Installs Management Agent

• Invokes EMCLI scripts to provision middleware environments

27

PG&E Provisioning Portal Features• Authentication against PG&E Active

Directory / SSO• Ability to notify Requestor/Approver

via email • System provisioning clients pay for

the resources • System provisioning clients are not

responsible for the maintenance of any part of the shared infrastructure

• Ability to monitor performance of their workloads to ensure SLAs are met

• Provides at least three deployment options for each artifact in terms of

sizing: small, medium, large

28

One Click!

• Provision Operating System

• Install Middleware

• Configure Domain • Discover Domain to obtain

out-of-box monitoring

29

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

30

Achieved Results and Improved Business

• Standardized OS, middleware and database images

• Reduced overall provisioning time by 75%

• Saved ~$1 million since inception

31

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

32

Future Plans

Extend provisioning to more use cases:• Oracle SOA Suite• Oracle Business Process Management Suite• Web Servers

33

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

34

Summary

• PG&E uses Oracle Enterprise Manager Cloud Control 12c deployment procedures with custom steps and scripts to provision all Oracle WebLogic Server, Portal and Service Bus environments

• PG&E has built an end-user portal to provision middleware infrastructure with a single click

• Provisioning brings PG&E a 75% reduction in time to market

• Many lines of businesses are enjoying the benefits of lower costs, faster delivery, and standardized deployments

35

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A