How to Use the PowerPoint...

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Oracle RAC Mejores Prácticas Operacionales

Ricardo Gonzalez Senior Product Manager Real Application Clusters Agosto, 2015



Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

3

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 4

Mejores Prácticas Operacionales

Actualización

Instalación

t SI

http://www.slideshare.net/MarkusMichalewicz/oracle-rac-12c-collaborate-best-practices-ioug-2014-version

Clusters Genéricos

Cluster Extendido

Dedicados (OLTP / DWH)

Ambientes Consolidados

Storage

S.O.

Red

Cluster

DB

Área



















Agenda 1

2

3

4

Nuevo en Oracle RAC 12.1.0.2 (Instalación)

Clusters Genéricos

Cluster Extendido

Ambientes Dedicados


Apéndices A – D

5

5



Agenda 1

2

3

4


Clusters Genéricos

Cluster Extendido

Ambientes Dedicados


Apéndices A – D

6

5



12.1.0.1 Grid Infrastructure Management Repository (GIMR)

12.1.0.2 Grid Infrastructure Management Repository (GIMR)

• Crea una Base 12c

– Instancia única, Container Database (CDB),

con un Pluggable Database (PDB)

– El nombre del recurso es “ora.mgmtdb”

– Consolidación prevista para el futuro

– Instalado en uno de los (HUB) nodes

– Administrado como base de datos failover

– Almacena métricas del Sistema Operativo recogidas por el Health Cluster Monitor

– Almacenado en el primer Grupo de Discos creado en ASM

7

Nuevo en la instlación de 12.1.0.2 GIMR: Creado automaticamente


12.1.0.1 Creación de grupos de discos: Comience con el grupo de disco “GRID"

Recomendación: Cambio en la creación de los Grupos de Discos


12.1.0.2 Creación de Grupo de discos: comience con el grupo de disco “GIMR”

• El GIMR normalmente no requiere redundancia en el grupo de discos.

– Por lo tanto, no lo comparta con el grupo de discos GRID.

• Archivos de Clusterware (Voting Files y OCR) son fáciles de cambiar de grupo de discos

• Más información:

– How to Move GI Management Repository to Different Shared Storage (Diskgroup, CFS or NFS etc) (Doc ID 1589394.1)

– Managing the Cluster Health Monitor Repository (Doc ID 1921105.1)

9

Ejemplo en Apéndice A

Recomendación: Cambio en la creación de los Grupos de Discos


12.1.0.1 Use Standard Cluster 12.1.0.2: Use Flex Cluster (incluye Flex ASM por defecto)

10

Nuevo en 12.1.0.2: Recomendación de usar Flex Cluster


12.1.0.2: Use Flex Cluster (incluye Flex ASM por defecto)

11

Nuevo en 12.1.0.2: Recomendación: Usar Flex Cluster

Sí estuviera instalando un Cluster Oracle con

RAC Extendido ?

Más información en el Apéndice D

Use Standard Cluster

+ Flex ASM


Instale solo lo necesario Configure lo que desee (actualice después)

12

Nuevo Flexibilidad de Red en 12.1.0.2 – Recomendación

Más información en el Apéndice B


Automatic Diagnostic Repository (ADR)

ADR_base

diag

asm rdbms tnslsnr clients crs (others)

• Oracle Grid Infrastructure ahora soporta Automatic Diagnostic Repository

• ADR simplifica el análisis de archivos de log

• Centraliza la mayoria de logs en una estructura de directorio

• Mantiene histórico de los logs.

• Posee herramienta de línea de comando para gerenciar las informaciones de diagnóstico

Más información en el Apéndice C


Agenda 1

2

3

4


Mejores Prácticas Operacionales para

Clusters Genéricos

Clusters Extendidos

Ambientes Dedicados


Apéndices A – D

14

5


Mejores Prácticas Operacionales – Clusters Genéricos

Clusters Genéricos

Clusters Extendidos



Storage

S.O.

Red

Cluster

DB

Área


Clusters Genéricos

Clusters Extendidos



Storage Apéndice A

S.O.

Red

Cluster

DB

Área

16

Clusters Genéricos → Storage

Paso 1: Crear grupo de disco “GRID/GIMR”– Cluster Genérico

Paso 3: Mover archivos ASM: SPFILE / Password

Paso 2: Mover Archivos de Clusterware

Más información en el Apéndice A


Evite alto uso de memoria!

Use Memory Guard

Activado por defecto con 12.1.0.2

17

Use Discos de Estado Sólido (SSDs) para el swap

Detalles en “My Oracle Support” (MOS)

Nota MOS: 1671605.1 – “Use Solid State Disks to host swap space in order to

increase node availability”

Use HugePages para SGA (Linux)

Notas MOS: 361323.1 / 401749.1

Avoid Transparent HugePages (Linux6) Nota MOS: 1557478.1

Clusters Genéricos → Sistema Operativo → Memoria

brasil germany Oracle GI

Oracle RAC Oracle GI

Oracle RAC

brasil germany Oracle GI

Oracle RAC Oracle GI

Oracle RAC

Swapping


• OraChk

– Substituye RACcheck

– También conocido como ExaChk

• RAC Configuration Audit Tool

– Nota MOS: 1268927.1

• Verifica “Oracle” (DB):

– Base de Datos single instance

– Grid Infrastructure & Oracle RAC

– Uso de Maximum Availability Architecture (MAA) - (sí esta configurada)

– Configuración de Hardware Oracle

18

Clusters Genéricos → Sistema Operativo → OraChk y TFA

Trace File Analyzer

Nota MOS: 1513912.1


TFA – Eficiencia de la A a la Z


Clusters Genéricos → Sistema Operativo

Clusters Genéricos

Cluster Extendido



Storage Apéndice A

S.O. Configuración de Memoria + OraChk / TFA

Red

Cluster

DB

Área


Defina qué es “normal”

21

Configure el tamaño de Interconnect para throughput

agregado

Use redundancia (HAIPs) para balanceo de carga

Use subnets diferentes para el interconnect

Use Jumbo Frames cuando sea posible

Garantice soporte de toda la infraestructura

Clusters Genéricos → Red

Más información en Apéndice B

Receive()

brasil germany

8K Data Block

1500 byte MTU

Send()

Re-montaje Fragmentación

Oracle RAC Oracle RAC


Clusters Genéricos → Red

Clusters Genéricos

Cluster Extendido



Storage Apéndice A


Red Como discutido + Apéndice B

Cluster

DB

Área


Clusters Genéricos

Cluster Extendido



Storage Apéndice A


Rede Como discutido + Apéndice B

Cluster Apéndice D

DB

Área

Clusters Genéricos → Cluster

1: Instalar / mantener HUBs, agregue Leaf Nodes

3: Use Leaf nodes para casos de uso no-DB

2: Agregando nodos al Cluster


Agenda 1

2

3

4

Nuevo en Oracle RAC12.1.0.2 (Instalación)


Clusters Genéricos

Cluster Extendido

Ambientes Dedicados


Apéndices A – D

24

5


Mejores Prácticas Operacionales – Cluster Extendido

Clusters Genéricos

Cluster Extendido



Storage

S.O.

Red

Cluster

DB

Área


Oracle RAC Extendido

Desde el punto de vista de Oracle, una instalación de RAC extendido es utilizada cuando los datos (usando Oracle ASM) son “mirrored” entre storage

arrays independientes.

ER: abierto para hacer “ORACLE RAC EXTENDIDO" UNA CONFIGURACIÓN DISTINGUIBLE


Clusters Genéricos

Cluster Extendido



Storage Apéndice A Apéndice A



Cluster Apéndice D

DB

Área

Clusters Genéricos → Storage

Paso 1: Crear grupo de disco “GRID/GIMR”– Cluster Extendido

Paso 3: Mover Archivos ASM: SPFILE / Password

Paso 2: Mover Archivos de Clusterware

Paso 4: “srvctl modify asm –count all”


Cluster Extendido → Sistema Operativo

Clusters Genéricos

Cluster Extendido





Igual a los Clusters Genéricos


Cluster Apéndice D

DB

Área

Más información: Oracle Real Application Clusters on Extended Distance Clusters (PDF)

http://www.oracle.com/technetwork/database/options/

clustering/overview/extendedracversion11-435972.pdf

http://www.oracle.com/technetwork/database/options/clustering/overview/extendedracversion11-435972.pdf


Defina “normal”

El objetivo en una configuración de RAC extendido es esconder la distancia.

Cualquier aumento de latencia puede (!) afectar el desempeño de la aplicación.

29

VLANs son totalmente soportadas en Oracle RAC

Separación de sub-red vertical no es soportada.

Cluster Extendido → Red

Mais informações: Oracle Real Application Clusters on Extended Distance Clusters (PDF)

http://www.oracle.com/technetwork/database/options/ clustering/overview/extendedracversion11-435972.pdf

Mais informações: Oracle Real Application Clusters on Extended Distance Clusters (PDF)

http://www.oracle.com/technetwork/database/ database-technologies/clusterware/overview/ interconnect-vlan-06072012-1657506.pdf


Cluster Extendido → Red Clusters

Genéricos Cluster

Extendido Dedicados

(OLTP / DWH) Ambientes

Consolidados





Como Discutido +Apéndice B

Cluster Apéndice D

DB

Área


Cluster Extendido → Cluster Clusters

Genéricos Cluster

Extendido Dedicados


Consolidados






Cluster Apéndice D Igual a los Clusters Genéricos

DB

Área

Recuerde: El objetivo en una configuración de RAC extendido es esconder la distancia. .


Agenda 1

2

3

4



Cluster Genérico

Cluster Extendido

Ambientes Dedicados


Apéndices A – D

32

5


Mejore Prácticas Operacionales – Ambientes Dedicados

Clusters Genéricos

Cluster Extendido



Storage

S.O.

Red

Cluster

DB

Área

Apenas algunos items por considerar .


Ambientes Dedicados → Red

Clusters Genéricos

Cluster Extendido










DB

Área


Problema: Aplicación de patches y actualizaciones

35

Problema: Consumo de memoria Problema: Número de conexiones

Ambientes Dedicados → Base de Datos

brasil germany

Connection Pool

Oracle GI Oracle RAC

Oracle GI Oracle RAC

Solución: Rapid Home Provisioning Solución: Memory Caps Solución: varias, usando connection pools mayormente


Nuevo en Oracle Database 12c:

• SGA y PGA aggregated targets pueden ser limitados.

• Vea la documentación para “PGA_AGGREGATE_LIMIT”

36

Ambientes Dedicados → Base de Datos

[DB]> sqlplus / as sysdba SQL*Plus: Release 12.1.0.2.0 Production on Thu Sep 18 18:57:30 2014 … SQL> show parameter pga NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ pga_aggregate_limit big integer 2G pga_aggregate_target big integer 211M SQL> show parameter sga NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ lock_sga boolean FALSE pre_page_sga boolean TRUE sga_max_size big integer 636M sga_target big integer 636M unified_audit_sga_queue_size integer 1048576

1. No lidiar con Connection Storms, impedirlas.

2. Limitar el número de conexiones con la base de datos.

3. Usar Connection Pools siempre que sea posible: • Oracle Universal Connection Pool (UCP) -

http://docs.oracle.com/database/121/JJUCP/rac.htm#JJUCP8197

4. Cerciórese que las aplicaciones cierren sus conexiones • Sí el número de conexiones activas es bastante inferior al

número de conexioes abiertas, considere usar “Database Resident Connection Pooling” - docs.oracle.com/database/121/JJDBC/drcp.htm#JJDBC29023

5. Sí usted no puede impedir la Connection Storm, retardela • Use parametros del LISTENER para mitigar los efectos

secundarios negativos de una Connection Storm. La mayoria de estos parametros también pueden ser utilizados con SCAN.

6. Los servicios pueden ser atribuidos a una sub-red cada vez. Si usted controla la sub-red, usted controla el servicio.

Connection Pool

http://docs.oracle.com/database/121/JJUCP/rac.htm

http://docs.oracle.com/database/121/JJDBC/drcp.htm


Ambientes Dedicados → Base de Datos Clusters

Genéricos Cluster

Extendido Dedicados


Consolidados








DB Como Discutido

Área


Agenda 1

2

3

4



Cluster Genérico

Cluster Extendido

Ambientes Dedicados


Apéndices A – D

38

5


Melhores Prácticas Operacionales – Ambientes Dedicados

Clusters Genéricos

Cluster Extendido



Storage

S.O.

Red

Cluster

DB

Área

También tiene apenas algunos ítems por considerar

.


Consolidación de Base de Datos

• Varias instancias de base de datos en ejecución en un servidor

•Necesidad de administrar varias instancias de memoria

• Use Instance Caging y QoS (en cluster de RAC)

40

Use Oracle Multitenant

• Menor número de instancias para administrar (CDB)

• La “memory allocation” del servidor se simplifica

• Instance Caging puede ya no ser necesario

• QoS sigue benéfico para administración de los recursos

Ambientes Consolidados – Sin VMs 2 Opciones

Brasil germany

racdb1_3

Oracle GI

Oracle RAC

Oracle GI

Oracle RAC

brasil

chile germany Oracle GI | HUB Oracle GI | HUB

Oracle GI | HUB


italy Oracle GI | HUB

Oracle RAC

cons

Oracle RAC

cons1_2

cons1_1

CPU_Count=5

CPU_Count=3


Use Oracle Multitenant

• Puede ser operado como un ambiente dedicado,

• por lo menos a partir de la perspectiva del cluster,

• sí apenas 1 instancia CDB por servidor es usada

Ambientes Consolidados – Hágalos Dedicados …

Información adicional: http://www.oracle.com/technetwork/database/focus-areas/database-cloud/database-cons-best-practices-1561461.pdf

http://www.oracle.com/technetwork/database/options/ clustering/overview/rac-cloud-consolidation-1928888.pdf

Brasil germany

racdb1_3

Oracle GI

Oracle RAC

Oracle GI

Oracle RAC

brasil

chile germany Oracle GI | HUB Oracle GI | HUB

Oracle GI | HUB


italy Oracle GI | HUB

Oracle RAC

cons

Oracle RAC

cons1_2

cons1_1

CPU_Count=5

CPU_Count=3


Ambientes Consolidados – Resumen de Base de Datos

Caso de Uso Área

Clusters Genéricos

Cluster Extendido

Dedicado (OLTP / DWH)



Sistema Operativo

Configuración de Memoria + OraChk /

TFA

Lo mismo que para Clusters

Genéricos

Red Como

Discutido +Apéndice B



Como Dedicado + Como

Discutido

Cluster Apéndice A Como Genérico

Base de Datos Como Discutido Como Discutido

Especificamente para Oracle Multitenant en Oracle RAC, vea: http://www.slideshare.net/MarkusMichalewicz/oracle-

multitenant-meets-oracle-rac-ioug-2014-version

http://www.slideshare.net/MarkusMichalewicz/oracle-multitenant-meets-oracle-rac-ioug-2014-version
















Para más información

http://community.oracle.com/blogs/raclatino

[email protected]

@raclatino

mailto:[email protected]


Appendix A Creating “GRID” disk group to place the Oracle Clusterware files and the ASM files

Oracle Confidential – Internal/Restricted/Highly Restricted 44


Create “GRID” Disk Group – Generic Cluster

Use “quorum” whenever possible.


Create “GRID” Disk Group – Extended Cluster

• More information: http://www.oracle.com/technetwork/database/options/clustering/overview/e

xtendedracversion11-435972.pdf

• Use logical names illustrating the disk destination • Use a quorum for ALL (not only GRID) disk groups used in an ExtendedCluster

• Use Voting Disk NFS destination







Replace Voting Disk Location Add OCR Location

47

Move Clusterware Files

[GRID]> crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 8bec21793ee84fd3bfc6831746bf60b4 (/dev/sde) [GIMR] Located 1 voting disk(s).

[GRID]> crsctl replace votedisk +GRID Successful addition of voting disk 7a205a2588d44f1dbffb10fc91ecd334. Successful addition of voting disk 8c05b220cfcc4f6fbf5752b6763a18ac. Successful addition of voting disk 223006a9c28e4fd5bf3b58a465fcb66a. Successful deletion of voting disk 8bec21793ee84fd3bfc6831746bf60b4. Successfully replaced voting disk group with +GRID. CRS-4266: Voting file(s) successfully replaced

[GRID]> crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 7a205a2588d44f1dbffb10fc91ecd334 (/dev/sdd) [GRID] 2. ONLINE 8c05b220cfcc4f6fbf5752b6763a18ac (/dev/sdb) [GRID] 3. ONLINE 223006a9c28e4fd5bf3b58a465fcb66a (/dev/sdc) [GRID] Located 3 voting disk(s).

[GRID]> whoami Root

[GRID]> ocrconfig -add +GRID [GRID]> ocrcheck Status of Oracle Cluster Registry is as follows : Version : 4 Total space (kbytes) : 409568 Used space (kbytes) : 2984 Available space (kbytes) : 406584 ID : 759001629 Device/File Name : +GIMR Device/File integrity check succeeded Device/File Name : +GRID Device/File integrity check succeeded Device/File not configured ... Cluster registry integrity check succeeded Logical corruption check succeeded

Use “ocrconfig -delete +GIMR” if you want to “replace” and maintain a single OCR location.


Default ASM spfile location is in the first disk group created (here: GIMR)

Perform a rolling ASM instance restart facilitating Flex ASM

48

Move ASM SPFILE – See also MOS note 1638177.1

[GRID]> export ORACLE_SID=+ASM1 [GRID]> sqlplus / as sysasm …

SQL> show parameter spfile NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ Spfile string +GIMR/cup-cluster/ASMPARAMETER FILE/registry.253.857666347 #Change location

SQL> create pfile='/tmp/ASM.pfile' from spfile; File created.

SQL> create spfile='+GRID' from pfile='/tmp/ASM.pfile'; File created. #NOTE:

SQL> show parameter spfile NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ Spfile string +GIMR/cup-cluster/ASMPARAMETER FILE/registry.253.857666347

Use “gpnptool get” and filter for

“ASMPARAMETERFILE” to see updated ASM

SPFILE location in GPnP profile prior to

restarting.

[GRID]> srvctl status asm ASM is running on brasil,chile,germany [GRID]> srvctl stop asm -n germany -f [GRID]> srvctl status asm -n germany ASM is not running on germany [GRID]> srvctl start asm -n germany [GRID]> srvctl status asm -n germany ASM is running on germany [GRID]> crsctl stat res ora.mgmtdb NAME=ora.mgmtdb TYPE=ora.mgmtdb.type TARGET=ONLINE STATE=ONLINE on brasil

Perform rolling through cluster.

12c DB instances remain running!


Default ASM shared password file location is the same as for the SPFILE (here +GIMR)

Path-checking while moving the file (online operation)

49

Move ASM Password File

[GRID]> srvctl config ASM ASM home: <CRS home> Password file: +GIMR/orapwASM ASM listener: LISTENER ASM instance count: 3 Cluster ASM listener: ASMNET1LSNR_ASM

GRID]> srvctl modify asm -pwfile +GRID/orapwASM [GRID]> srvctl config ASM ASM home: <CRS home> Password file: +GRID/orapwASM ASM listener: LISTENER ASM instance count: 3 Cluster ASM listener: ASMNET1LSNR_ASM

[GRID]> srvctl modify asm -pwfile GRID

[GRID]> srvctl config ASM ASM home: <CRS home> Password file: GRID ASM listener: LISTENER ASM instance count: 3 Cluster ASM listener: ASMNET1LSNR_ASM

[GRID]> srvctl modify asm -pwfile +GRID PRKO-3270 : The specified password file +GRID does not conform to an ASM path syntax

Use the correct ASM path syntax!


Appendix B Creating public and private (DHCP-based) networks including SCAN and SCAN Listeners

50


Step 1: Add network Result

51

Add Public Network – DHCP

[GRID]> oifcfg iflist eth0 10.1.1.0 eth1 10.2.2.0 eth2 192.168.0.0 eth2 169.254.0.0

[GRID]> oifcfg setif -global "*"/10.2.2.0:public [GRID]> oifcfg getif eth0 10.1.1.0 global public eth2 192.168.0.0 global cluster_interconnect,asm * 10.2.2.0 global public Only in OCR: eth1 10.2.2.0 global public PRIF-29: Warning: wildcard in network parameters can cause mismatch among GPnP profile, OCR, and system.

[GRID]> su Password:

[GRID]> srvctl add network -netnum 2 -subnet 10.2.2.0/255.255.255.0 -nettype dhcp [GRID]> exit exit

[GRID]> srvctl config network -k 2 Network 2 exists Subnet IPv4: 10.2.2.0/255.255.255.0/, dhcp Subnet IPv6: Ping Targets: Network is enabled Network is individually enabled on nodes: Network is individually disabled on nodes:

[GRID]> crsctl stat res -t -------------------------------------------------------------------------------- Name Target State Server State details -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- … ora.net2.network OFFLINE OFFLINE brasil STABLE OFFLINE OFFLINE chile STABLE OFFLINE OFFLINE germany STABLE

…


Step 2: Add SCAN / SCAN_LISTENER to the new network (as required)

Result

52

Add Public Network – DHCP


[GRID]> srvctl update gns -advertise MyScan -address 10.2.2.20

# Need to have a SCAN name. DHCP network requires dynamic VIP resolution via GNS [GRID]> srvctl modify gns -verify MyScan The name "MyScan" is advertised through GNS.

[GRID]> srvctl add scan -k 2 PRKO-2082 : Missing mandatory option –scanname


[GRID]> srvctl add scan -k 2 -scanname MyScan [GRID]> exit [GRID]> srvctl add scan_listener -k 2

[GRID]> srvctl config scan -k 2 SCAN name: MyScan.cupgnsdom.localdomain, Network: 2 Subnet IPv4: 10.2.2.0/255.255.255.0/, dhcp Subnet IPv6: SCAN VIP is enabled. SCAN VIP is individually enabled on nodes: SCAN VIP is individually disabled on nodes: SCAN VIP is enabled. SCAN VIP is individually enabled on nodes: SCAN VIP is individually disabled on nodes: SCAN VIP is enabled. SCAN VIP is individually enabled on nodes: SCAN VIP is individually disabled on nodes:

[GRID]> srvctl config scan_listener -k 2 SCAN Listener LISTENER_SCAN1_NET2 exists. Port: TCP:1521 Registration invited nodes: Registration invited subnets: SCAN Listener is enabled. SCAN Listener is individually enabled on nodes: SCAN Listener is individually disabled on nodes: SCAN Listener LISTENER_SCAN2_NET2 exists. Port: TCP:1521 Registration invited nodes: Registration invited subnets: SCAN Listener is enabled. SCAN Listener is individually enabled on nodes: SCAN Listener is individually disabled on nodes: SCAN Listener LISTENER_SCAN3_NET2 exists. Port: TCP:1521 Registration invited nodes: Registration invited subnets: SCAN Listener is enabled. SCAN Listener is individually enabled on nodes: SCAN Listener is individually disabled on nodes:


oifcfg commands Result (ifconfig -a on HUB)

53

Add Private Network – DHCP

[GRID]> oifcfg iflist eth0 10.1.1.0 eth1 10.2.2.0 eth2 192.168.0.0 eth2 169.254.0.0 eth3 172.149.0.0

[GRID]> oifcfg getif eth0 10.1.1.0 global public eth2 192.168.0.0 global cluster_interconnect,asm * 10.2.2.0 global public Only in OCR: eth1 10.2.2.0 global public PRIF-29: Warning: wildcard in network parameters can cause mismatch among GPnP profile, OCR, and system.

[GRID]> oifcfg setif -global "*"/172.149.0.0:cluster_interconnect,asm [GRID]> oifcfg getif eth0 10.1.1.0 global public eth2 192.168.0.0 global cluster_interconnect,asm * 10.2.2.0 global public * 172.149.0.0 global cluster_interconnect,asm PRIF-29: Warning: wildcard in network parameters can cause mismatch among GPnP profile, OCR, and system.

BEFORE eth3 Link encap:Ethernet HWaddr 08:00:27:1E:2B:FE inet addr:172.149.2.7 Bcast:172.149.15.255 Mask:255.255.240.0 inet6 addr: fe80::a00:27ff:fe1e:2bfe/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:52 errors:0 dropped:0 overruns:0 frame:0 TX packets:17 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:20974 (20.4 KiB) TX bytes:4230 (4.1 KiB)

AFTER eth3 Link encap:Ethernet HWaddr 08:00:27:1E:2B:FE inet addr:172.149.2.7 Bcast:172.149.15.255 Mask:255.255.240.0 inet6 addr: fe80::a00:27ff:fe1e:2bfe/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1161 errors:0 dropped:0 overruns:0 frame:0 TX packets:864 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:720040 (703.1 KiB) TX bytes:500289 (488.5 KiB) eth3:1 Link encap:Ethernet HWaddr 08:00:27:1E:2B:FE inet addr:169.254.245.67 Bcast:169.254.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

HAIPs will only be used for

Load Balancing once at least the DB / ASM instances, of

not the node is restarted. They are considered

for failover immediately.


ifconfig -a on HUB – excerpt ifconfig -a on Leaf – excerpt

54

Side note: Leaf Nodes don’t host HAIPs!

eth2 Link encap:Ethernet HWaddr 08:00:27:AD:DC:FD inet addr:192.168.7.11 Bcast:192.168.15.255 Mask:255.255.240.0 inet6 addr: fe80::a00:27ff:fead:dcfd/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:9303 errors:0 dropped:0 overruns:0 frame:0 TX packets:6112 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:8344479 (7.9 MiB) TX bytes:2400797 (2.2 MiB)

eth2:1 Link encap:Ethernet HWaddr 08:00:27:AD:DC:FD inet addr:169.254.190.250 Bcast:169.254.255.255 Mask:255.255.128.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 eth3 Link encap:Ethernet HWaddr 08:00:27:1E:2B:FE inet addr:172.149.2.5 Bcast:172.149.15.255 Mask:255.255.240.0 inet6 addr: fe80::a00:27ff:fe1e:2bfe/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:4729 errors:0 dropped:0 overruns:0 frame:0 TX packets:5195 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1555796 (1.4 MiB) TX bytes:2128607 (2.0 MiB)

eth3:1 Link encap:Ethernet HWaddr 08:00:27:1E:2B:FE inet addr:169.254.6.142 Bcast:169.254.127.255 Mask:255.255.128.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth2 Link encap:Ethernet HWaddr 08:00:27:CC:98:C3 inet addr:192.168.7.15 Bcast:192.168.15.255 Mask:255.255.240.0 inet6 addr: fe80::a00:27ff:fecc:98c3/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:7218 errors:0 dropped:0 overruns:0 frame:0 TX packets:11354 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2644101 (2.5 MiB) TX bytes:13979129 (13.3 MiB) eth3 Link encap:Ethernet HWaddr 08:00:27:06:D5:93 inet addr:172.149.2.6 Bcast:172.149.15.255 Mask:255.255.240.0 inet6 addr: fe80::a00:27ff:fe06:d593/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:6074 errors:0 dropped:0 overruns:0 frame:0 TX packets:5591 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2262521 (2.1 MiB) TX bytes:1680094 (1.6 MiB)

HAIPs on the interconnect are only used by ASM / DB instances. Leaf nodes do

not host those, hence, they do not host HAIPs. CSSD (the node management

daemon) uses a different redundancy approach.


Step 1: Add network Result

55

Add Public Network – STATIC

[GRID]> oifcfg iflist eth0 10.1.1.0

eth1 10.2.2.0 eth2 192.168.0.0 eth2 169.254.128.0 eth3 172.149.0.0 eth3 169.254.0.0 #Assuming you have NO global public interface defined on subnet 10.2.2.0

[GRID]> oifcfg setif -global "*"/10.2.2.0:public [GRID]> oifcfg getif eth0 10.1.1.0 global public eth2 192.168.0.0 global cluster_interconnect,asm * 172.149.0.0 global cluster_interconnect,asm

* 10.2.2.0 global public PRIF-29: Warning: wildcard in network parameters can cause mismatch among GPnP profile, OCR, and system.


[GRID]> srvctl add network -netnum 2 -subnet 10.2.2.0/255.255.255.0 -nettype STATIC

[GRID]> srvctl config network -k 2 Network 2 exists Subnet IPv4: 10.2.2.0/255.255.255.0/, static Subnet IPv6: Ping Targets: Network is enabled Network is individually enabled on nodes: Network is individually disabled on nodes: [GRID]> crsctl stat res -t -------------------------------------------------------------------------------- Name Target State Server State details -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- … ora.net2.network OFFLINE OFFLINE brasil STABLE OFFLINE OFFLINE chile STABLE OFFLINE OFFLINE germany STABLE

…


Step 2: Add VIPs Result

56


[GRID]> srvctl add vip -node germany -address germany-vip2/255.255.255.0 -netnum 2 [GRID]> srvctl add vip -node brasil -address brasil-vip2/255.255.255.0 -netnum 2 [GRID]> srvctl add vip -node chile -address chile-vip2/255.255.255.0 -netnum 2 [GRID]> srvctl config vip -n germany VIP exists: network number 1, hosting node germany VIP Name: germany-vip VIP IPv4 Address: 10.1.1.31 VIP IPv6 Address: VIP is enabled. VIP is individually enabled on nodes: VIP is individually disabled on nodes: VIP exists: network number 2, hosting node germany VIP Name: germany-vip2 VIP IPv4 Address: 10.2.2.31 VIP IPv6 Address: VIP is enabled. VIP is individually enabled on nodes: VIP is individually disabled on nodes:

[GRID]> srvctl start vip -n germany -k 2 [GRID]> srvctl start vip -n brasil -k 2 [GRID]> srvctl start vip -n chile -k 2

[GRID]> srvctl status vip -n germany VIP germany-vip is enabled VIP germany-vip is running on node: germany VIP germany-vip2 is enabled VIP germany-vip2 is running on node: germany [GRID]> srvctl status vip -n brasil VIP brasil-vip is enabled VIP brasil-vip is running on node: brasil VIP brasil-vip2 is enabled VIP brasil-vip2 is running on node: brasil [GRID]> srvctl status vip -n chile VIP chile-vip is enabled VIP chile-vip is running on node: chile VIP chile-vip2 is enabled VIP chile-vip2 is running on node: chile


Step 3: Add SCAN / SCAN_LISTENER to the new network (as required)

Result

57


#as root [GRID]> srvctl add scan -scanname cupscan2 -k 2 [GRID]> exit [GRID]> srvctl add scan_listener -k 2 -endpoints 1522 [GRID]> srvctl status scan_listener -k 2 SCAN Listener LISTENER_SCAN1_NET2 is enabled SCAN listener LISTENER_SCAN1_NET2 is not running [GRID]> srvctl start scan_listener -k 2

[GRID]> srvctl status scan_listener -k 2 SCAN Listener LISTENER_SCAN1_NET2 is enabled SCAN listener LISTENER_SCAN1_NET2 is running on node chile [GRID]> srvctl status scan -k 2 SCAN VIP scan1_net2 is enabled SCAN VIP scan1_net2 is running on node chile


Appendix C Automatic Diagnostic Repository (ADR) support for Oracle Grid Infrastructure

58


• The ADR is a file-based repository for diagnostic data such as traces, dumps, the alert log, health monitor reports, and more.

• ADR helps preventing, detecting, diagnosing, and resolving problems.

• ADR comes with its own command line tool (adrci) to get easy access to and manage diagnostic information for Oracle GI + DB.

59

Automatic Diagnostic Repository (ADR) Convenience

ADR_base

diag

asm rdbms tnslsnr clients crs (others)


adrci adrci incident management

60

Some Management Examples

[GRID]> adrci ADRCI: Release 12.1.0.2.0 - Production on Thu Sep 18 11:35:31 2014 Copyright (c) 1982, 2014, Oracle and/or its affiliates. All rights reserved. ADR base = "/u01/app/grid“ adrci> show homes ADR Homes: diag/rdbms/_mgmtdb/-MGMTDB diag/tnslsnr/germany/asmnet1lsnr_asm diag/tnslsnr/germany/listener_scan1 diag/tnslsnr/germany/listener diag/tnslsnr/germany/mgmtlsnr diag/asm/+asm/+ASM1 diag/crs/germany/crs diag/clients/user_grid/host_2998292599_82 diag/clients/user_oracle/host_2998292599_82 diag/clients/user_root/host_2998292599_82

[GRID]> adrci ADR base = "/u01/app/grid" … adrci> show incident; ADR Home = /u01/app/grid/diag/rdbms/_mgmtdb/-MGMTDB: ************************************************************************* INCIDENT_ID PROBLEM_KEY CREATE_TIME -------------------- ----------------------------------------------------------- ---------------------------------------- 12073 ORA 700 [kskvmstatact: excessive swapping observed] 2014-09-08 17:44:56.580000 -07:00 36081 ORA 700 [kskvmstatact: excessive swapping observed] 2014-09-14 20:11:17.388000 -07:00 40881 ORA 700 [kskvmstatact: excessive swapping observed] 2014-09-16 15:30:18.319000 -07:00 …

adrci> set home diag/rdbms/_mgmtdb/-MGMTDB adrci> ips create package incident 12073; Created package 1 based on incident id 12073, correlation level typical

adrci> ips generate package 1 in /tmp Generated package 1 in file /tmp/ORA700ksk_20140918110411_COM_1.zip, mode complete

[GRID]> ls –lart /tmp -rw-r--r--. 1 grid oinstall 811806 Sep 18 11:05 ORA700ksk_20140918110411_COM_1.zip


Binary / Log per Node Space Requirement

Grid Infra. (GI) Home ~6.6 GB

RAC DB Home ~5.5 GB

TFA Repository 10 GB

GI Daemon Traces ~2.6 GB

ASM Traces ~9 GB

DB Traces 1.5 GB per DB per month

Listener Traces 60MB per node per month

Total over 3 months • For 2 RAC DBs • For 100 RAC DBs

• ~43 GB •~483 GB

• Flex ASM vs. Standard ASM Flex Cluster vs. Standard Cluster

– Does not make a difference for ADR!

61

Space Requirements, Exceptions, and Rules

gnsd

ocssd ocssdrim

havip

exportfs NFS

helper

hanfs

ghc ghs

mgmtdb

agent

APX

gns

Some

OC4J

Logs

Some

GI home

Logs


Appendix D Flex Cluster – add nodes as needed

62


Initial installation: HUB nodes only Add Leafs later (addNode)

63

Recommendation: Install HUB Nodes, Add Leaf Nodes


Add “brasil” as a HUB Node – addNode Part 1


Add “brasil” as a HUB Node – addNode Part 2


Add Leaf Nodes – addNode in Short Note: Leaf nodes do not

require a virtual node name (VIP). Application

VIPs for non-DB use cases need to be added

manually later.

Normal, can be ignored.


Database installer suggestion Consider Use Case

67

Continue to use Leaf Nodes for Applications in 12.1.0.2

Useful, if “spain” is likely to become a HUB at some point in time.


DBCA Despite running Leaf Nodes

68

Continue to use Leaf Nodes for Applications in 12.1.0.2

[GRID]> olsnodes -s -t germany Active Unpinned brasil Active Unpinned chile Active Unpinned

italy Active Unpinned spain Active Unpinned


Leaf Listener (OFFLINE/OFFLINE) Trace File Analyzer (TFA)

69

Some Examples of Resources running on Leaf Nodes

[grid@spain Desktop]$ . grid_profile [GRID]> crsctl stat res -t -------------------------------------------------------------------------------- Name Target State Server State details -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.ASMNET1LSNR_ASM.lsnr ONLINE ONLINE brasil STABLE ONLINE ONLINE chile STABLE ONLINE ONLINE germany STABLE

…

ora.LISTENER.lsnr ONLINE ONLINE brasil STABLE ONLINE ONLINE chile STABLE ONLINE ONLINE germany STABLE

ora.LISTENER_LEAF.lsnr OFFLINE OFFLINE italy STABLE OFFLINE OFFLINE spain STABLE ora.net1.network ONLINE ONLINE brasil STABLE ONLINE ONLINE chile STABLE ONLINE ONLINE germany STABLE

[GRID]> ps -ef |grep grid_1 root 1431 1 0 14:12 ? 00:00:19 /u01/app/12.1.0/grid_1/jdk/jre/bin/java -Xms128m -Xmx512m -classpath /u01/app/12.1.0/grid_1/tfa/spain/tfa_home/jlib/RATFA.jar:/u01/app/12.1.0/grid_1/tfa/spain/tfa_home/jlib/je-5.0.84.jar:/u01/app/12.1.0/grid_1/tfa/spain/tfa_home/jlib/ojdbc6.jar:/u01/app/12.1.0/grid_1/tfa/spain/tfa_home/jlib/commons-io-2.2.jar oracle.rat.tfa.TFAMain /u01/app/12.1.0/grid_1/tfa/spain/tfa_home

How to Use the PowerPoint...

Documents

Transcript of How to Use the PowerPoint...