Post on 30-Dec-2015
Deploying E-Business Deploying E-Business Suit 11i on Oracle RACSuit 11i on Oracle RAC
11g on AIX11g on AIX
Oracle OpenWorldOracle OpenWorldSeptember 2010September 2010
IntroductionIntroduction
AgendaAgenda
• Bunnings at a glance Bunnings at a glance • Our Business Benefits of RAC Our Business Benefits of RAC • Infrastructure Capacity ManagementInfrastructure Capacity Management• Initial Build and Deployment ExperienceInitial Build and Deployment Experience• Our Build and DesignOur Build and Design• Preparation and Preparation and pre-requisitespre-requisites• Patching and Installation Patching and Installation • RAC conversion RAC conversion
– RDBMSRDBMS– Required Mid-tier changes Required Mid-tier changes
• OEM ConfigurationOEM Configuration• Issues and ResolutionIssues and Resolution• Useful References Useful References • Next stepsNext steps• Q&AQ&A
WIDEST RANGE
LOWEST PRICES
BEST SERVICE
0
1000
2000
3000
4000
5000
6000
7000
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Sales
Sales ($m)
Sales = IT ChallengesSales = IT Challenges
Our Business Benefits of RACOur Business Benefits of RAC
• Ensuring business core applications remain and have Ensuring business core applications remain and have the needed Flexibility and Agilitythe needed Flexibility and Agility
• High Availability 24x7 Australia and New Zealand High Availability 24x7 Australia and New Zealand OperationsOperations
• Deliver on Performance Deliver on Performance • Provide a level of Workload SegregationProvide a level of Workload Segregation
““Our Mission critical systems must be agile to support our Our Mission critical systems must be agile to support our business by improving operational effectiveness as well business by improving operational effectiveness as well as supporting decision making which is based on timely, as supporting decision making which is based on timely,
accurate and flexible information”accurate and flexible information”
User UniquenessUser Uniqueness
• PDT – orderingPDT – ordering
• Reporting – DiscovererReporting – Discoverer
• Store SupportStore Support
– FormsForms
– Self-serviceSelf-service
– Custom Forms and Self-serviceCustom Forms and Self-service
• StoresStores
– MC-75 PDA/PDT running self-serviceMC-75 PDA/PDT running self-service
– Touch screen and pen inputTouch screen and pen input
– WIFI and 3G WirelessWIFI and 3G Wireless
• Live reporting via XML Publisher and Discoverer 10gLive reporting via XML Publisher and Discoverer 10g
Our Mobile Procurement SolutionOur Mobile Procurement Solution
Infrastructure Capacity ManagementInfrastructure Capacity ManagementN
o. T
ran
sact
ion
s P
er M
on
th
Transactions Processed
** Chart is actual data from 2002 to 2009, covering total Transactions for Year on Year growth ** Chart is actual data from 2002 to 2009, covering total Transactions for Year on Year growth
Infrastructure Capacity ManagementInfrastructure Capacity Management
• Information Lifecycle Management is part of the solutionInformation Lifecycle Management is part of the solution
• EBS partitioning needs to be done early otherwise EBS partitioning needs to be done early otherwise special issues come up if regular purging cannot be special issues come up if regular purging cannot be undertaken due to business requirementsundertaken due to business requirements
– Inventory, GL, AP, AR, PO can grow quickly Inventory, GL, AP, AR, PO can grow quickly depending on usedepending on use
– Moving data in live systems is trickyMoving data in live systems is tricky
Metalink article 554539.1 (Using Database Partitioning with Oracle E-Business Suite )Has all the information needed to implement partitioning in EBS
Infrastructure Capacity ManagementInfrastructure Capacity Management
• RAC - scales very well… but some caveats!RAC - scales very well… but some caveats!
– OLTP/Warehouse type queries, Global Cache Service OLTP/Warehouse type queries, Global Cache Service (GCS) contention(GCS) contention
– Concurrent gatherstats across nodes can cause lock Concurrent gatherstats across nodes can cause lock upsups
– Any contention in a single node environment Any contention in a single node environment magnified in multi-node RACmagnified in multi-node RAC
Initial Build and Deployment ExperienceInitial Build and Deployment Experience
Full System Configuration:Processors – 58x 1.65GHz P5 (595)Memory – 512Gb RAM
11.5.10Application Server
11i EBS Database
Production Initial Phase :Our Early and Rapid
Deployment
Production Second Phase :
BLA
DE
(2x
3.6
Ghz
EM
64
T4
8G
b R
AM
)
BLA
DE
(2x
3.6
Ghz
EM
64
T4
8G
b R
AM
)
BLA
DE
(2x
3.6
Ghz
EM
64
T4
8G
b R
AM
)
11i EBS Database - RAC Node 3
Full System Configuration:Processors – 58x 1.65GHz P5 (595)Memory – 512Gb RAM
11i EBS Database - RAC Node 2
11i EBS Database - RAC Node 1
BLA
DE
(2x
3.6G
hz E
M6
4T
48G
b R
AM
)
BLA
DE
(2x
3.6G
hz E
M6
4T
48G
b R
AM
)
BLA
DE
(2x
3.6G
hz E
M6
4T
48G
b R
AM
)
BLA
DE
(2x
3.6G
hz E
M6
4T
48G
b R
AM
)
Clustered with Jserv& Servlet mode
Clustered with 0C4J
BLA
DE
(2x
3.6
Ghz
EM
64
T4
8G
b R
AM
)
EBS 11i Web Cache 10.1.2.3.0
EBS 11i Forms, Reports & Concurrent Managers
Initial Build and Deployment ExperienceInitial Build and Deployment Experience
Initial Build and Deployment ExperienceInitial Build and Deployment Experience
BLA
DE
(2x
3.6G
hz E
M64
T48
Gb
RA
M)
BLA
DE
(2x
3.6G
hz E
M64
T48
Gb
RA
M)
BLA
DE
(2x
3.6G
hz E
M64
T48
Gb
RA
M)
EBS 11i Forms, Reports & Concurrent Managers
Full System Configuration:Processors – 23x 4.14GHz P7 (780)Memory – 512Gb RAM
11i EBS Database - RAC Node 2
11i EBS Database - RAC Node 1
BLA
DE
(2x
3.6G
hz E
M64
T48
Gb
RA
M)
BLA
DE
(2x
3.6G
hz E
M64
T48
Gb
RA
M)
BLA
DE
(2x
3.6G
hz E
M64
T48
Gb
RA
M)
BLA
DE
(2x
3.6G
hz E
M64
T48
Gb
RA
M)
Clustered with Jserv& Servlet mode
Clustered with 0C4J
BLA
DE
(2x
3.6G
hz E
M64
T48
Gb
RA
M)
EBS 11i Web Cache 10.1.2.3.0
11i EBS Database - RAC Node 3
Full System Configuration:Processors – 21x 4.44GHz P7 (780)Memory – 512Gb RAM
Current Production – Physical Representation
RAC1
Radware Web Accelerator
Oracle Web Cache Oracle Web Cache
Clustered with OC4J
LBR with cookie
persistence
Form S2 Report S2Forms S1 Report S1
Concurrent Mgr 1 Concurrent Mgr 2
RAC2
(OLTP) (OLTP)
Clustered with Jserv/servlet mode
PCP
Primary/Secondary
Application Layer Linux
RH5
DatabaseLayer
AIX 5.3-08
Application Layer Linux
RH5
RAC3
(Read-Mostly)
Our Build and DesignOur Build and Design
Our Build and DesignOur Build and Design
• Our design and build phase experiencesOur design and build phase experiences
– Don’t try to do it all at onceDon’t try to do it all at once• Implement in multiple steps and give time to bed in changes and Implement in multiple steps and give time to bed in changes and
gauge their full impact before moving ongauge their full impact before moving on
– Follow stream deployment and testing strategyFollow stream deployment and testing strategy
• Start in your Sandpit/Test environments to work out the specific Start in your Sandpit/Test environments to work out the specific requirements of your environmentrequirements of your environment
• Most systems are integrated, CIT and SIT testing is next.. Most systems are integrated, CIT and SIT testing is next.. Beware of TNSNAME changes during RAC conversion for DB Beware of TNSNAME changes during RAC conversion for DB LinksLinks
Our Build and DesignOur Build and Design
• Our design and build phase experiencesOur design and build phase experiences
– Test thoroughlyTest thoroughly
• Don’t rush testing and thoroughly regression and load test Don’t rush testing and thoroughly regression and load test the new architecture, some issues only come out under load the new architecture, some issues only come out under load with custom codewith custom code
Testing ApproachTesting Approach
EBS Backend Load TestsEBS Backend Load Tests
•Swingbench ArchitectureSwingbench Architecture
•Custom class extensions to enable Custom class extensions to enable more controlmore control
•Junit integration with Swingbench to Junit integration with Swingbench to allow OA Framework testingallow OA Framework testing
Cluster Controller
Node 1 Node 3Node 2
EBS Environment
Co-ordinator
Workload simulated can include multiple concurrent transactions like TPC, Concurrent jobs and stored
proceedures
• Infrastructure testing using SwingBench bots deployed across our Infrastructure testing using SwingBench bots deployed across our varied network topology to calculate latency and load. varied network topology to calculate latency and load.
• Packeteer was installed at end points to enable granular monitoring of Packeteer was installed at end points to enable granular monitoring of
network utilisation. network utilisation.
NextG
3G
BOT(xN)
BOT(xN)
StoreWiFi
EBS Open Interface
BOT(xN)
BOT(xN)
Existing
Testing ApproachTesting Approach
• Our design and build phase experiencesOur design and build phase experiences
– First transformation was to move to ASM 11g/CRS11g First transformation was to move to ASM 11g/CRS11g 10g database10g database
• Requires a patch to solve locking issue when changes Requires a patch to solve locking issue when changes happen in ASMhappen in ASM
– Second transformation was to upgrade 10g RDBMS Second transformation was to upgrade 10g RDBMS to 11g RDBMSto 11g RDBMS
• Use best practices, prepare for changes in execution plans, Use best practices, prepare for changes in execution plans, baseline 10g system and compare, make sure you are fully baseline 10g system and compare, make sure you are fully patched up on 11gpatched up on 11g
Our Build and DesignOur Build and Design
Metalink document E-Business Suite 11i on RAC (Configuring Database Load balancing & Failover )ID 294652.1] and Metalink note 244366.1.
• Our design and build phase experiencesOur design and build phase experiences
– Third transformation was to go to RAC with 3 nodesThird transformation was to go to RAC with 3 nodes• More reliability and less impact if loss of one nodeMore reliability and less impact if loss of one node• Better ability to distribute workloadBetter ability to distribute workload• Mixed workload OLTP/ReportingMixed workload OLTP/Reporting• Discoverer and other reporting tools can be segregated to Discoverer and other reporting tools can be segregated to
specific nodesspecific nodes
Our Build and DesignOur Build and Design
Metalink document E-Business Suite 11i on RAC (Configuring Database Load balancing & Failover )ID 294652.1] and Metalink note 244366.1.
• Specialise workloads with RAC managed services and E-Specialise workloads with RAC managed services and E-Business suite application profile settings (load direction)Business suite application profile settings (load direction)
– We plan Active/asymmetric service descriptions, 2 We plan Active/asymmetric service descriptions, 2 nodes per service for HAnodes per service for HA
– TNSNAMES and JDBC are used for external access TNSNAMES and JDBC are used for external access such as reporting tools (Discoverer, Cognos, Portal) to such as reporting tools (Discoverer, Cognos, Portal) to isolate workloadsisolate workloads
Our Build and DesignOur Build and Design
Metalink document E-Business Suite 11i on RAC (Configuring Database Load balancing & Failover )ID 294652.1] and Metalink note 244366.1.
• Specialise workloads with RAC managed services and E-Specialise workloads with RAC managed services and E-Business suite application profile settings (load direction)Business suite application profile settings (load direction)
–Resource Manager Consumer Groups and Mapping are Resource Manager Consumer Groups and Mapping are used to manage resource utilisation which are then used to manage resource utilisation which are then mapped to servicesmapped to services
–Can go to application level via application profiles, dbc Can go to application level via application profiles, dbc and session_cookie_name settings which can be and session_cookie_name settings which can be changed to match RAC services changed to match RAC services
Our Build and DesignOur Build and Design
Metalink document E-Business Suite 11i on RAC (Configuring Database Load balancing & Failover )ID 294652.1] and Metalink note 244366.1.
Preparation and pre-requisitesPreparation and pre-requisites
• The minimum software levels for various EBS modules The minimum software levels for various EBS modules can be found on Metalink. can be found on Metalink.
Key PatchesKey Patches– Minipack 11i.AD.I.5 (233044.1)Minipack 11i.AD.I.5 (233044.1)– 11GR1 APPS Interoperability Patch (6328363) 11GR1 APPS Interoperability Patch (6328363) – 11i.ATG_PF.H RUP6 – (5903765) 11i.ATG_PF.H RUP6 – (5903765) – 11.5.10:SFM Unable to Process Orders in RAC 11.5.10:SFM Unable to Process Orders in RAC
Config (4022732) ( this can be applied before setting Config (4022732) ( this can be applied before setting up Oracle RAC) up Oracle RAC)
Metalink document : Using Oracle Real Application Clusters and Automatic Management with Oracle e-Business Suite Release 11i and Oracle 11g. Metalink note 455398.1Minipack 11i AD.1.5, Metalink note 233044.1
Patching and InstallationPatching and Installation
• Follow best practices for RAC and 11gFollow best practices for RAC and 11g– Separate homes for CRS, ASM, DBSeparate homes for CRS, ASM, DB– Patch up each component to latest patchset available Patch up each component to latest patchset available
on 11g starting with Clusterwareon 11g starting with Clusterware– Check key Metalink documents for known Check key Metalink documents for known
performance patches/issues with EBS 11iperformance patches/issues with EBS 11i– Shared file systems for directory objects (output Shared file systems for directory objects (output
directories), OCFS2, NFS, IBM GPFSdirectories), OCFS2, NFS, IBM GPFS– RAC memory overhead over single instance needs to RAC memory overhead over single instance needs to
be considered (about 10% more for buffer cache and be considered (about 10% more for buffer cache and 15% for shared pool)15% for shared pool)
Review the two key Metalink notes, 950896.1 and 244040.1 for known performance patches/issues with EBS 11i
RAC Conversion RDMBSRAC Conversion RDMBS
• CRS installation on each nodeCRS installation on each node• ASM conversion if requiredASM conversion if required• RAC conversion (follow Metalink)RAC conversion (follow Metalink)• Edit ConvertToRAC.xml Edit ConvertToRAC.xml
– Use RConfig utility and plan a few dry runsUse RConfig utility and plan a few dry runs– NetCA listener reconfiguration for cluster (RDMBS NetCA listener reconfiguration for cluster (RDMBS
level)level)– UNDO and TEMP tablespaces UNDO and TEMP tablespaces
• FLASH area may need adjustingFLASH area may need adjusting
– Enable AutoConfig on all database nodesEnable AutoConfig on all database nodes
EBS to RAC Conversion, Metalink note 455398.1 and 455398.1
RAC Conversion RDMBSRAC Conversion RDMBS
• StandbyStandby– Archive logs generated from each instanceArchive logs generated from each instance– Archive logs name changeArchive logs name change– RMAN scripts may need to be modifiedRMAN scripts may need to be modified
• Cloning implicationsCloning implications– RAC to RAC cloning, multiple to single node RAC or RAC to RAC cloning, multiple to single node RAC or
multiple to multiple adds complexity and will be multiple to multiple adds complexity and will be dependant on sitedependant on site
– Rapid Clone, Oracle Application Management Pack, Rapid Clone, Oracle Application Management Pack, manualmanual
– IBM SVC Metro Mirror and Flash-Copy, block level IBM SVC Metro Mirror and Flash-Copy, block level copying (high speed cloning)copying (high speed cloning)
Cloning Considerations and implications , refer to details on Metalink note 760637.1
RAC Conversion Mid-Tier changesRAC Conversion Mid-Tier changes
• Auto configuration, get ready to run it a few times on all Auto configuration, get ready to run it a few times on all nodes!nodes!
• Implement Load Balancing using context editor by pointing Implement Load Balancing using context editor by pointing to correct balance listenersto correct balance listeners– Tools OH TWO_TASK (s_tools_twotask)Tools OH TWO_TASK (s_tools_twotask)– iAS OH TWO_TASK (s_weboh_twotask)iAS OH TWO_TASK (s_weboh_twotask)– Apps JDBC Connect Alias (s_apps_jdbc_connect_alias)Apps JDBC Connect Alias (s_apps_jdbc_connect_alias)
• Parallel Concurrent Processing configurationParallel Concurrent Processing configuration– In multiple application tier onlyIn multiple application tier only
• Transaction managersTransaction managers
Known Issue, Service Manager Processes Are Not Starting After A Restart - usdsop cannot redirect standard output, Metalink note 733901.1
OEM configurationOEM configuration
• Oracle Enterprise Manager is an essential part of our Oracle Enterprise Manager is an essential part of our administration, control and support structureadministration, control and support structure
• Moving the instance from non-RAC to RAC had some Moving the instance from non-RAC to RAC had some impactsimpacts– Need to re-register the instance as RACNeed to re-register the instance as RAC– User Defined Metrics need to be moved from the User Defined Metrics need to be moved from the
instance level to Cluster level, we have a lot of these instance level to Cluster level, we have a lot of these and process was manualand process was manual
– Challenges in doing live performance management with Challenges in doing live performance management with OEM in RAC contextOEM in RAC context
• need to hone in with SQL scripts looking at session information need to hone in with SQL scripts looking at session information combined with various GV$viewscombined with various GV$views
• drill down in OEM goes from cluster to instance to session view drill down in OEM goes from cluster to instance to session view which makes it difficult to pinpoint cross instance locking => which makes it difficult to pinpoint cross instance locking => identifying library cache pin locks is a good exampleidentifying library cache pin locks is a good example
Issues and ResolutionIssues and Resolution
• Initial issues identified in testing addressed with patchesInitial issues identified in testing addressed with patches• Unique environment challenges Unique environment challenges • Testing stream alignment (UAT/SIT/CIT/SP)Testing stream alignment (UAT/SIT/CIT/SP)
– extra infrastructure required to reproduceextra infrastructure required to reproduce– 1 node RAC can be used for development and 1 node RAC can be used for development and
preliminary test environmentspreliminary test environments– Right sizing test data and environmentsRight sizing test data and environments– UAT must be sized to an adequate ratio to production UAT must be sized to an adequate ratio to production
to outline issues in contention and scalingto outline issues in contention and scaling
Next StepsNext Steps
• Extending EBS Environment ExternallyExtending EBS Environment Externally– OLMOLM– Procure to PayProcure to Pay– Credit Card (own - closed loop)Credit Card (own - closed loop)
• Further Growing Reporting RequirementsFurther Growing Reporting Requirements– Improve PerformanceImprove Performance– PartitioningPartitioning
• Continue with archiving Continue with archiving
Useful ResourcesUseful Resources
Oracle E-Business Suite Release 11i High Availability Documentation Roadmap – Metalink note 1072807.1
Oracle Applications Release 11i with Oracle 11g Release 1 (11.1.0) –Metalink note 452783.1 - Interoperability Notes
Using Oracle Real Application Clusters and Automatic Storage Management with Oracle E-Business Suite Release 11i and Oracle Database 11g – Metalink note 455398.1
John Olszewskijolszewski@bunnings.com.au
THANK YOUTHANK YOU