Post on 10-Apr-2018
8/8/2019 21 Oracle Rac With Asm on Linux
1/28
8/8/2019 21 Oracle Rac With Asm on Linux
2/28
8/8/2019 21 Oracle Rac With Asm on Linux
3/28
8/8/2019 21 Oracle Rac With Asm on Linux
4/28
RAC with ASM on
Linux x86-64Experience with an
extended cluster(stretched cluster)
8/8/2019 21 Oracle Rac With Asm on Linux
5/28
The Issue
Issue
Overloaded server Hundreds of applications running against a single
database (Oracle 9.2.0.6 on Sun Solaris).
Protected by a failover cluster (Veritas) Decision: New DWH apps on new DB server
Requirements
Meet cost, stability, availability and performancecriteria
Do not break internal standards (e.g. multipathing)
8/8/2019 21 Oracle Rac With Asm on Linux
6/28
The Requirements
Meet cost criteria
RAC on Linux with ASM is an option. Meet availability criteria
Use cluster technology.
Test in a PoC Meet stability and performance criteria
Customer provided data and queries of their internal systemsfor a DWH benchmark.
Benchmark: Oracle, , .
Test in a PoC
8/8/2019 21 Oracle Rac With Asm on Linux
7/28
The Benchmark
1 week at the HP-center in Bblingen
Hardware: 4 servers with 4 XEON 64Bit CPUs, 16G RAM.
EVA 8000 storage with 156 disks.
Result:
1. Oracle/
2.
Why Oracle?
8/8/2019 21 Oracle Rac With Asm on Linux
8/28
Proof of Concept (PoC)
Test extended RAC cluster configuration
Installation High Availability
Performance
Workload Management
Backup/Recovery
Maintenance Job Scheduling
Monitoring
8/8/2019 21 Oracle Rac With Asm on Linux
9/28
Benefits of an extended
cluster Full utilization of resources no matter where
they are located
Site A
All Work gets Distributed to All Nodes
One PhysicalDatabase
Site B
8/8/2019 21 Oracle Rac With Asm on Linux
10/28
Benefits of an extended
cluster Faster recovery from site failure than any
other technology in the market
Work Continues on Remaining Site
Site AOne Physical
Database
Site B
8/8/2019 21 Oracle Rac With Asm on Linux
11/28
PoC: Design Considerations
Connectivity: Redundant connections for
public, interconnect and IO traffic
Site A Site BDual SAN Connections
Dual Private Interconnects
Dual Public Connections
8/8/2019 21 Oracle Rac With Asm on Linux
12/28
PoC: Design Considerations
Disk (or better SAN-) mirroring: Host based
mirroring with ASM
8/8/2019 21 Oracle Rac With Asm on Linux
13/28
ASM setupSAN 1:
USP 600SAN 2:
USP 600
Diskgroup+DATA01
SPFILE; CONTROL_1DATAFILES, REDO_1
Diskgroup +FLASH01CONTROL_2
REDO_2ARCH REDO
BACKUPS
ASMFailuregroup
D01_02
ASMFailuregroupFlash01_02
ASMFailuregroup
D01_01
ASMFailuregroupFlash01_01
8/8/2019 21 Oracle Rac With Asm on Linux
14/28
PoC: Architecture
+asm2
CRS
ORA2
RAID1
SCRATCH
Raid 1
OS
ORA_HOME
ORACLE LOGS
+asm1
CRS
ORA1
RAID 1
SCRATCH
Raid 1
OS
ORA_HOME
ORACLE LOGS
+asm4
CRS
ORA4
RAID 1
SCRATCH
Raid 1
OS
ORA_HOME
ORACLE LOGS
Data
Center 1
SAN1OCR Standby/Backup
VOTING DISK Standby /Backup
DATAFILES
REDOLOGS
ARCH REDO
SPFILE
CONTROL FILE
SAN2OCR Active
VOTING DISK Active
DATAFILES
REDOLOGS
ARCH REDO
SPFILE
CONTROL FILE
PUBLIC
Network
PUBLIC
Network
PRIVATE
INTERCONNECT
PRIVATE
INTERCONNECT
Data
Center 2
+asm3
CRS
ORA3
RAID 1
SCRATCH
Raid 1
OS
ORA_HOME
ORACLE LOGS
8/8/2019 21 Oracle Rac With Asm on Linux
15/28
PoC: Network Layout
Data Center 1 Data Center 2
Node1
Node2
Node3
Node4
Router DC1/1
Router DC1/2
Router DC2/1
Router DC2/2
VLAN forPrivat NWwith dedicatedlines
PublicPrivat
ActivePassive
8/8/2019 21 Oracle Rac With Asm on Linux
16/28
Used Software
Linux SLES-9 SP2
Native Multipathing: Device Mapper 32Bit or 64Bit?
32Bit: Availability of versions and patches
64Bit: Addressable memory, no midterm-migration
Oracle for x86-64:
10.1.0.3 10.1.0.4
10.2.0.1
8/8/2019 21 Oracle Rac With Asm on Linux
17/28
Used Hardware
HP Proliant DL585 with 4 AMD Opteron 800
series processor running at 2.2 GHz, 32GBRAM.
2 Fibre-channels per server
Storage: 2 Hitachi USP 600
8/8/2019 21 Oracle Rac With Asm on Linux
18/28
Tests
Installation tests
High Availability Performance
Workload Management
Backup/Recovery
Maintenance
Job Scheduling Monitoring
8/8/2019 21 Oracle Rac With Asm on Linux
19/28
High Availability Tests
8/8/2019 21 Oracle Rac With Asm on Linux
20/28
High Availability Tests
Bonding
NIC failover tests successful. Multipathing
Device mapper available in Linux 2.6 (SLES9 SP2).
Multiple paths to the same drive are auto-detected, based ondrives WWID.
We successfully tested failover-policy (fault tolerance) andmultibus-policy (fault tolerance and throughput).
Check with your storage vendor on which multipath software(in our case Device Mapper or HDLM) and policy to use.
8/8/2019 21 Oracle Rac With Asm on Linux
21/28
High Availability Tests 10gR1
Crash Test Impact Service Main component BehaviorPower off Server TAF, CRS second instance is up and running
Disk failure - LUN_data RAID/ASM instances up and running
Disk failure - LUN_voting RAID instances up and running
Disk failure - LUN_ocr RAID instances up and running
Unplug fiber attachment Multipathing instances up and running
Kill Oracle Instance TAF, CRS second instance is up and running
Kill ASM Instance TAF, CRS second instance is up and running
Halt q on Linux TAF, CRS second instance is up and running
Unplug Interconnect cable Bonding instances up and running
Unplug all Interconnect cables TAF,CRS/Voting second instance is up and running
Power off - SAN (NoVoting) ASM instances up and running
Power off - SAN ( voting ) cluster down
8/8/2019 21 Oracle Rac With Asm on Linux
22/28
HA Design Considerations What about quorum?
What happens if the site with the quorum fails or allcommunications between sites are lost?
ThirdSite (10gR1)
Quorum
Quorum
Quorum
10gR2
8/8/2019 21 Oracle Rac With Asm on Linux
23/28
High Availability Tests 10gR2
Crash Test Impact Service Main component BehaviorPower off Server TAF, CRS second instance is up and running
Disk failure - LUN_data RAID/ASM instances up and running
Disk failure - LUN_voting RAID instances up and running
Disk failure - LUN_ocr RAID instances up and running
Unplug fiber attachment Multipathing instances up and running
Kill Oracle Instance TAF, CRS second instance is up and running
Kill ASM Instance TAF, CRS second instance is up and running
Halt q on Linux TAF, CRS second instance is up and running
Unplug Interconnect cable Bonding instances up and running
Unplug all Interconnect cables TAF,CRS/Voting second instance is up and running
Power off - SAN ( voting ) ASM, 3rd voting instances up and running
Power off - 3rd voting disk 2 remaining votes instances up and running
8/8/2019 21 Oracle Rac With Asm on Linux
24/28
High Availability Tests
Switch off SAN (i.e. switch off one HDS USP 600) Disk mount status changes from CACHED to MISSING in
ASM (v$asm_disk)
Processing continues without interruption
Switch on SAN again
Disks are visible twice: Mount status MISSING and CLOSED. Add disks again
alter diskgroup failgroup disk name force rebalance power 10;
Disks with status MISSING disappear. Disks with previousstatus CLOSED have status CACHED now.
Result: No downtime even with a SAN unavailability.
8/8/2019 21 Oracle Rac With Asm on Linux
25/28
Performance considerations
Need to minimize latency
Direct effect on ASM mirroring and cache fusionoperations
Its better to use direct connections. Additional
routers, hubs or extra switches add latency.
8/8/2019 21 Oracle Rac With Asm on Linux
26/28
Extended Cluster versus MAA
Extended RAC RAC + DG
Minimum Nodes Needed 2 3
Active Nodes All One Side Only
Recovery from Site Failure Seconds, No InterventionRequired
Manual Restart UsuallyRequired
Network Requirements High cost direct dedicatednetwork w/ lowest latency.Much greater networkbandwidth
Shared commercially availablenetwork. Does not have lowlatency requirements
Effective Distance Campus & Metro Continent
Disaster Protection Minor Site or LocalizedFailuresUser Errors
Database Corruptions
Wider DisastersUser Errors
8/8/2019 21 Oracle Rac With Asm on Linux
27/28
Summary
RAC on Extended Cluster with ASM
It works!
Good design is key!
Data Guard offers additional benefits
8/8/2019 21 Oracle Rac With Asm on Linux
28/28