My First 100 days with an Exadata (PPT)

My First 100 days with an Exadata

Presented by : Gustavo René Antúnez Oracle DBA The Pythian Group April, 2014

About Pythian •  Recognized Leader:

–  Global industry-leader in remote database administration services and consulting for Oracle, Oracle Applications, MySQL and Microsoft SQL Server

–  Work with over 250 multinational companies such as Forbes.com, Fox Sports, Nordion and Western Union to help manage their complex IT deployments

•  Expertise: –  One of the world’s largest concentrations of dedicated, full-time DBA

expertise. Employ 9 Oracle ACEs/ACE Directors.

–  Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC

•  Global Reach & Scalability: –  Around the clock global remote support for DBA and consulting, systems

administration, special projects or emergency response

About Me –  Oracle DBA –  Part of the First Oracle

Associate Group •  Started with Version 9.2 in 2004

–  With The Pythian Group for the last year

–  Movie Fanatic –  Music Lover –  Bringing the best from México

(Mexihtli) to the rest of the world and in the process photographing it :)

–  reneantunez.com –  @grantunez

•  #CLV14

Where do I come from?

How did I get to be a DBA

Happiest Job of 2014!*

*http://www.forbes.com/sites/susanadams/2014/03/20/the-happiest-and-unhappiest-jobs-in-2014/

Work-life balance

Relationship with boss and co-

workers

Daily tasks Job

resources

Field will grow by 15% between

2012 and 2022 DBA can be the

key driver of success

What is Oracle Exadata Database Machine? •  Optimized database machine designed by

Oracle. It is a combination of Oracle Exadata Storage Server software, Oracle Database software, completely redundant hardware and a high-bandwidth low-latency InfiniBand network that connects all the components inside.

My Version of an Oracle Exadata Database Machine

Storage Nodes

Database Nodes

Infiniband Leaf Switches

ILOM,KVM, Cisco Network

Switch

My Version of Rack Mount Options of an Exadata

1/8 1/4 1/2 1

X3-2 and X4-2

Compute or Database Nodes Grid

Infrastructure RDBMS

RAC

Linux or Solaris OS

Linux OS commands, SQL*Plus,

ASMCMD, CRSCTL

Storage or Cell Node Exadata

Storage Server Software

4 PCIe flash cards

12 rotating magnetic disks

12 rotating magnetic disks

Linux OS only

cellcli,dcli, Linux OS Commands

Abstraction of Disk 1 and 2 of the Cell Node

Physical Disk

Grid Disk

Grid Disk

Grid Disk

LUN

OS Storage Area (Disk1 and Disk2)

Cell Disk

Cell System Area (Disk 1 and Disk 2)

Faster (Outside)

Slower (Inside)

CellCLI> list cell detail name: exa1c114_net0

…

cellVersion: OSS_11.2.0.3.0_LINUX.X64_110929

cpuCount: 16

…

ipaddress1: 192.168.12.114/24

kernelVersion: 2.6.18-238.12.2.0.2.el5

locatorLEDStatus: off

makeModel: SUN MICROSYSTEMS SUN FIRE X4275 SERVER SAS

metricHistoryDays: 7

…

powerCount: 2/2

powerStatus: normal

releaseVersion: 11.2.2.4.0

…

status: online

…

upTime: 97 days, 21:53

General information about your cell

CellCLI> list physicaldisk detail name: 20:0

deviceId: 8

diskType: HardDisk

enclosureDeviceId: 20

errMediaCount: 0

errOtherCount: 0

foreignState: false

luns: 0_0

makeModel: "SEAGATE ST360057SSUN600G"

physicalFirmware: 0805

physicalInsertTime: 2010-02-05T10:31:54+00:00

physicalInterface: sas

physicalSerial: E03WV1

physicalSize: 558.9109999993816G

slotNumber: 0

status: normal

General Information about your Physical Disk

CellCLI> list lun detail name: 0_0

cellDisk: CD_00_exa1c114_net0

deviceName: /dev/sda

diskType: HardDisk

id: 0_0

isSystemLun: TRUE

lunAutoCreate: FALSE

lunSize: 557.861328125G

lunUID: 0_0

physicalDrives: 20:0

raidLevel: 0

lunWriteCacheMode: "WriteBack, ReadAheadNone, Direct, No Write

Cache if Bad BBU"

status: normal

General Information about your LUN Disk

CellCLI> list celldisk detail name: CD_00_exa1c114_net0

comment:

creationTime: 2011-06-14T02:25:45+00:00

deviceName: /dev/sda

devicePartition: /dev/sda3

diskType: HardDisk

errorCount: 0

freeSpace: 0

id: 00000130-8bf8-0381-0000-000000000000

interleaving: none

lun: 0_0

raidLevel: 0

size: 528.734375G

status: normal

General Information about your Cell Disk

CellCLI> list griddisk detail name: DATA_CD_00_exa1c114_net0

asmDiskGroupName: DATA

asmDiskName: DATA_CD_00_exa1c114_NET0

availableTo:

cellDisk: CD_00_exa1c114_net0

comment:

creationTime: 2011-06-14T02:26:18+00:00

diskType: HardDisk

errorCount: 0

id: 00000130-8bf8-867a-0000-000000000000

offset: 32M

size: 430G

status: active

General Information about your Grid Disk

View Name Description

V$CELL IP addresses assigned to the cells. This also includes the hash value for the cell which is included in the Exadata storage cell session waits in v$session_wait in the P1 column.

V$CELL_CONFIG Configuration information about all the levels in a storage cell.

V$CELL_REQUEST_TOTALS Snapshots of the requests made to each cell over the last 15 minutes

V$CELL_STATE Current statistics for each storage cell.

V$CELL_THREAD_HISTORY Detail of events over the previous 15 minutes for the storage cell.

Database Performance views to get information concerning the storage cells

Integrated Lights Out Manager (ILOM)

•  Cell and Compute Nodes come with a dedicated management channel for device maintenance called Integrated Lights Out Manager that works independently from these nodes

•  It supports various interfaces for accessing it:

–  Web Interface –  CLI (Command Line Interface) –  Remote console –  Intelligent Platform Management Interface (IPMI)


•  You can also login to the node via the ILOM home/antunez> ssh [email protected] Password:

…

Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved.

-> start /SP/console

Are you sure you want to start /SP/console (y/n)? y

…

test01db01.pythian.com login: root

Password:

Last login: Thu Jan 30 20:01:47 from 10.10.10.11

[root@test01db01 ~]# ps -eaf | grep pmon

grid 12167 1 0 21:42 ? 00:00:00 asm_pmon_+ASM1

root 13408 11206 0 21:42 ttyS0 00:00:00 grep pmon


•  You can also power on a server with the IPMI interface [root@test01db01 ~]# ipmitool -H test01db01-ilom -U root -P welcome1 chassis power on

•  Interact with the ILOM through the IPMI interface [root@test01db01 ~]# ipmitool sunoem cli "show /SP/network"

Connected. Use ^D to exit.

-> show /SP/network

/SP/network

Targets:

interconnect

ipv6

test

Properties:

commitpending = (Cannot show property)

dhcp_server_ip = none

…


•  http://tiny.cc/hw03dx

Disk Replacement

•  Common tasks done and that normally is not done on non-Exadata environment is a disk replacement.

•  The following two MOS documents should be revised before doing a disk replacement : –  1390836.1 (Predictive Failure) –  1386147.1 (Hard Failure)

Network Consolidated interconnect backbone

that serves all of the hardware

components.

iDB is used to ship SQL operations down to the cell nodes for execution

and to return query result sets to the database

kernel

It uses “zero copy”

Exachk: healtcheck for Exadata

•  MOS note 1070954.1 •  The tool is non-intrusive, when it completes it's collection

and analysis, it produces two reports, summary and detailed.

•  Advised to be run as a part of periodic maintenance operations on the exadata.

•  Configure oem12c to perform checkups on exadata (exachk) –  http://tiny.cc/scz3dx

Exachk: healtcheck for Exadata

Useful MOS Note and ASR •  Information Center: Troubleshooting Oracle

Exadata Database Machine (Doc ID 1346612.2) •  ASR is designed to automatically create Service

Requests when specific hardware fault events occur.

•  ASR requires the following software packages to be downloaded and installed: –  Oracle Automated Service Manager (OASM)

(SUNWsasm package) –  ASR Plug-in (SUNWswasr package)

Hybrid Columnar Compression (HCC)

•  HCC is optimized to use both database and storage capabilities on Exadata to deliver tremendous space savings

•  Data will only be compressed in HCC format when it is loaded using direct path loads types

BLOCK HEADER

CU HEADER

BLOCK HEADER BLOCK HEADER BLOCK HEADER

C3

C4 C1 C2

C7 C5

C6 C8 C8

Logical Compression Unit


•  HCC has four types of compression: –  Query Low. - It uses the LZO compression algorithm.

CREATE TABLE ... COMPRESS FOR QUERY LOW; –  Query High. - It uses the ZLIB (gzip) compression algorithm.

CREATE TABLE ... COMPRESS FOR QUERY HIGH; –  Archive Low. - It uses the ZLIB (gzip) compression algorithm as well,

but at a higher-level compression than Query High. CREATE TABLE ... COMPRESS FOR ARCHIVE LOW;

–  Archive High. – It uses Bzip2 compression

CREATE TABLE ... COMPRESS FOR ARCHIVE HIGH;


•  For new tables and partitions: CREATE TABLE emp (emp_id NUMBER, first_name VARCHAR2(128), last_name VARCHAR2(128)) COMPRESS FOR QUERY HIGH;

•  For existing tables and partitions

Online Redefinition (DBMS_REDEFINITION) ALTER TABLE ... MOVE COMPRESS FOR QUERY HIGH


Compression Method

Segment Size

High NDV Low NDV

MB Reduction MB Reduction

No Compression 3,109.00 3,008.00

Basic 2,769.00 10.90% 246.88 91.80%

OLTP 3,080.60 0.90% 280.94 90.70%

Query Low 2,466.90 20.70% 113.25 96.20%

Query High 1,637.40 47.30% 5.31 99.80%

Archive Low 1,632.80 47.50% 5.31 99.80%

Archive High 1,546.10 50.30% 5.31 99.80%

•  We created 2 tables consisting of only 12 columns and both with close to 20 billion rows. Many of the columns in table 1 have a very HIGH number of distinct values (NDV) and the second one with very LOW number of distinct values (NDV)


•  Oracle also provides an EHCC advisor, called GET_EHCC_CR, which can be found in MOS 1269846.1 SQL> EXEC GET_EHCC_CR('DATA','TST','EX_TAB_ORIGINAL',NULL,3); Compression Advisor self-check validation successful. select count(*) on both Uncompressed and EHCC Compressed format = 1000001 rows COMPRESSED_TYPE = "Compress For Archive Low"

COMPRESSED_BLOCKS = 10417

UNCOMPRESSED_BLOCKS = 19611

COMPRESSED_ROWS = 96

UNCOMPRESSED_ROWS = 51

COMPRESSION_RATIO = 1.8


•  Row level locking goes away when using HCC. •  Every update of a record stored in HCC format

results in a migrated row •  Non-direct path inserts will be loaded into OLTP

compressed blocks as opposed to HCC format. •  For the above reasons, mixing HCC with DML is

not recommended.

Smart Scan and Offloading

•  Processes queries at the storage layer, returning only relevant rows and columns to the database server.

•  Offloading refers to the concept of moving processing from the database servers to the storage layer

•  Will not use Smart Scan –  if the columns being requested include a database

large object (LOB) –  if a table is a clustered table –  If is an index-organized table


•  There are three basic requirements that must be met for Smart Scans to occur: –  There must be a full scan of an object.

•  These correspond to TABLE ACCESS FULL and INDEX FAST FULL SCAN operations of an execution plan.

–  The object must be stored on Exadata storage. –  The scan must use Oracle’s Direct Path Read

mechanism

•  The scan must use Oracle’s Direct Path Read mechanism.


Server Process

SQL Work Areas

Private SQL Area Session Memory

Database Buffer Cache

Smart Scan and Offloading •  With Smart Scan, come several optimizations:

–  Column Projection or Column Filtering •  The cell server will only return the columns

requested. –  Table that has 50 columns, and only 5 are selected and 2

involved in the join operation. The cell server will only return 7 columns.

Smart Scan and Offloading –  Predicate Filtering

•  The cell server will only return the rows requested instead of all the rows in a table.

–  Since iDB includes the predicate information in its

requests, this is accomplished by performing the standard filtering operations at the storage cells before returning the data.

Smart Scan and Offloading –  Storage Indexes

•  Cell Server In-Memory structure that its main goal is to reduce the number I/O’s from disk

•  Automatically created and maintained transparently by the Exadata Storage Server Software

•  Three “commonly” known hidden database parameters that deal with Storage Indexes

–  _kcfis_storageidx_disabled .- When set to TRUE it will turn Storage Indexes off

–  _kcfis_storageidx_diag_mode.- This will enable you to trace the Storage Index

–  _cell_storidx_mode.- Controls where Storage Indexes will be applied (Comparison operators)

Smart Scan and Offloading •  Should You Drop Indexes in Exadata?

–  http://tiny.cc/wr03dx

Exadata Patching •  You now have several components

–  The firmware of the nodes –  Operating System –  Infiniband Switches –  RDBMS and Grid Infrastructure binaries

•  MOS document 888828.1 will have the latest patches and release dates

•  Information Center: Upgrading Oracle Exadata Database Machine (Doc ID 1364356.2)

•  Exadata Database Server Patching using the DB Node Update Utility (Doc ID 1553103.1)

Different types of patches for Exadata Database Machine •  Storage Server software

–  Contains updates to firmware, operating system, and/or Exadata Storage Server

•  Rolling

•  Non-rolling

•  Database server –  Oracle Grid Infrastructure and Database software –  Operating system software and firmware

•  InfiniBand switch software •  Additional components

–  Ethernet switch, KVM, and PDU patches

Example of dbnodeupdate.sh [root@test01db01 ~]# ./dbnodeupdate.sh -u -l /u01/stage/p16432033/p16432033_112321_Linux-x86-64.zip …

(*) 2013-12-07 03:24:52: Backing up /etc/yum.repos.d/Exadata-computenode.repo in /etc/yum.repos.d.bak/071213032414 …

Active Image version : 11.2.3.1.0.120304

Active Kernel version : 2.6.18-274.18.1.0.1.el5

…

Automatic checks incl. : Issue 1.8 - Hotspare not reclaimed

: Issue 1.10 - Cell and Database image versions 11.2.2.2.2 or lower require workaround before patching …

Example of dbnodeupdate.sh … Manual checks todo : Issue 1.11 - Database Server upgrades to 11.2.2.3.0 or higher may hit network routing issues after the upgrade Note : After upgrading and rebooting run './dbnodeupdate.sh -c' to finish post steps

Continue ? [Y/n]

Y

(*) 2013-12-07 03:25:30: Verifying GI and DB's are shutdown

… (*) 2013-12-07 03:36:34: System will reboot automatically for changes to take effect

(*) 2013-12-07 03:36:34: After reboot run "./dbnodeupdate.sh -c" to complete the upgrade

Exadata and EM12c •  EM 12c offers an integrated approach to managing

Oracle Exadata Database Machine •  Use the Oracle Exadata Discovery Cookbook

–  http://www.oracle.com/technetwork/oem/exa-mgmt/em12c-exadata-discovery-cookbook-1662643.pdf

•  Pre discovery process and pre work will save you a lot of headaches –  exadataDiscoveryPreCheck.pl from MOS note 1473912.1 –  Need to have the ILOM user oemuser created –  Preferably to use the ASMSNMP user for ASM instances –  Make sure all the /etc/hosts are correctly set

Exadata and EM12c ************************************************************* * Enterprise Manager Exadata Pre-Discovery checks *

**************************************************************

…

Script used is /home/oracle/working/antunez/exadataDiscoveryPreCheck.pl

…

test1cel01 test1cel02 test1cel03 test2cel01 test2cel02 test2cel03 test2cel04

…

Verifying component versions...

-------------------------------

Verifying Exadata Storage Server Software version...

----------------------------------------------------

Verifying version for test1cel01 cell node...

Exadata Storage Server Software version is 11.2.3.2.1 ===> Ok

…

Press [Enter] to continue...

Verifying ILOM ipmitool version...

----------------------------------

ILOM ipmitool version is 1.8.10.4 ===> Ok

Exadata and EM12c

•  From the setup menu –  Select Add targets à Add targets manuallyàAdd

Non-HostàTargets using Guided process àOracle Exadata

•  Follow this order when discovering –  DataBase Machine –  Cluster and HA Services on each node –  Databases, Listeners and ASM

Exadata and EM12c

•  Recommended that you setup the target properties correctly when discovering

•  If Having Problems discovering a Database target, use the rcuJDBCEngine to test. [email protected] [oms] /opt/software/oms/Middleware/oms/bin oracle $ ./rcuJDBCEngine dbsnmp@"(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(Host=test01-scan)(Port=1521))(CONNECT_DATA=(SERVICE_NAME=testdb)))" JDBC_SCRIPT test.sql

…

Completed SQL script execution normally.

1 scripts were processed

Exadata and EM12c

Slideshare

•  Session ID 544 •  http://www.slideshare.net/

GustavoReneAntunez

Pythian at Collaborate 2014 Session ID

Session Date

Start Time Session Room Session Title

Presenter Name

157 4/9/2014 8:30 AM Level 3, Lido 3101B Database Private Clouds with Oracle Database 12c Marc Fielding

574 4/9/2014 8:30 AM Level 3, Lido 3005 DBA 101 : Calling all New Database Administrators Rene Antunez

14408 4/9/2014 4:30 PM Sands, Level 1 - 309 What's New in Oracle E-Business Suite R12.2 for Database Administrators?

Vasu Balla

158 4/10/2014 3:00 PM Level 3, San Polo 3403

Accelerate Your Exadata Deployment with the Skills You Already Have Marc Fielding

411 4/10/2014 4:15 PM Level 3, Murano 3306

Thou Shalt Not Steal: Securing Your Infrastructure in the Age of Snowden

Paul Vallee

544 4/10/2014 11:00 AM Level 3, San Polo 3403

My First 100 days with an Exadata Rene Antunez

135 4/11/2014 9:45 AM Level 3, Lido 3003 Ensuring Your Physical Standby is Usable Michael Abbey

194 4/11/2014 12:15 PM Level 3, Lido 3101B 12c Multi-Tenancy and Exadata IORM: An Ideal Cloud Based Resource Management

Fahd Chughtai

546 4/11/2014 8:30 AM Level 3, Lido 3103 RMAN in 12c: The Next Generation ReneAntunez

878 4/11/2014 9:45 AM Level 3, San Polo 3405

Practical Machine Learning for DBAs Alex Gorbachev

Thank you – Q&A To contact us

[email protected]

1-877-PYTHIAN

To follow us http://www.pythian.com/blog

http://www.facebook.com/pages/The-Pythian-Group/163902527671

@pythian

http://www.linkedin.com/company/pythian

We are hiring! Booth 1535

My First 100 days with an Exadata (PPT)

Technology

Transcript of My First 100 days with an Exadata (PPT)