D67016GC20 Exadata Workshop Part1

Exadata and Database Machine Administration Workshop

Student Guide

D67016GC20

Edition 2.0

January 2011

D71669

Copyright 2010, Oracle and/or it affiliates. All rights reserved.

Disclaimer

This document contains proprietary information and is protected by copyright and other intellectual property laws. You may copy and print this document solely for your own use in an Oracle training course. The document may not be modified or altered in any way. Except where your use constitutes "fair use" under copyright law, you may not use, share, download, upload, copy, print, display, perform, reproduce, publish, license, post, transmit, or distribute this document in whole or in part without the express authorization of Oracle.

The information contained in this document is subject to change without notice. If you find any problems in the document, please report them in writing to: Oracle University, 500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is not warranted to be error-free.

Restricted Rights Notice

If this documentation is delivered to the United States Government or anyone using the documentation on behalf of the United States Government, the following notice is applicable:

U.S. GOVERNMENT RIGHTSThe U.S. Governments rights to use, modify, reproduce, release, perform, display, or disclose these training materials are restricted by the terms of the applicable Oracle license agreement and/or the applicable U.S. Government contract.

Trademark Notice

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Authors

Peter Fusek

Jean-Francois Verrier

Mark Fuller

Dave Winter

Technical Contributors and Reviewers

Andrew Babb

Bharat Baddepudi

Maria Billings

Robert Carlin

Michael Cebulla

Nilesh Choudhury

Christian Craft

Ravindra Dani

Aslam Edah-Tally

Boris Erlikhman

Amit Ganesh

Ed Gilowski

Joel Goodman

Scott Gossett

Jim Hall

Roger Hansen

James He

David Hitchcock

Bill Hodak

Vimala Jacob

Martin Jensen

Kevin Jernigan

Caroline Johnston

Larry Justice

Vikram Kapoor

Bruce Kyro

Sumeet Lahorani

Publishers

Sujatha Nagendra

Giri Venugopal

Sue Lee

Juan Loaiza

Barb Lundhild

Varun Malhotra

Louis Nagode

Dan Norris

Michael Nowak

Sriram Palapudi

Umesh Panchaksharaiah

Sugam Pandey

Robert Pastijn

Marshall Presser

Georg Schmidt

Akshay Shah

Kam Shergill

Tim Shelter

Eric Siglin

Sundararaman Sridharan

Vijay Sridharan

Mahesh Subramaniam

Lawrence To

Alex Tsukerman

Kodi Umamageswaran

Douglas Utzig

Harald van Breederode

Mark Van de Wiel

Dave Winter

iii

Contents 1 Introduction

Course Objectives 1-2 Audience and Prerequisites 1-3 Course Scope 1-4 Course Contents 1-5 Terminology 1-6 Additional Resources 1-7 Practice 1 Overview: Introducing the Laboratory Environment 1-8

2 Exadata Overview Objectives 2-2 Traditional Enterprise Database Storage Deployment 2-3 Exadata Storage Deployment 2-4 Exadata Implementation Architecture Overview 2-6 Introducing Exadata 2-7 Exadata Hardware Details (Sun Fire X4270 M2) 2-8 Exadata Specifications 2-9 InfiniBand Network 2-10 Classic Database I/O and SQL Processing Model 2-11 Exadata Smart Scan Model 2-12 Exadata Smart Storage Capabilities 2-13 Exadata Smart Scan Scale-Out Example 2-16 Exadata Hybrid Columnar Compression 2-19 Exadata Hybrid Columnar Compression Architecture Overview 2-20 Exadata Smart Flash Cache 2-21 Exadata Storage Index 2-23 Storage Index with Partitions Example 2-25 Database File System 2-26 I/O Resource Management 2-27 Benefits Multiply 2-28 Exadata Key Benefits for Data Warehousing 2-29 Exadata Key Benefits for OLTP 2-31 Quiz 2-32 Summary 2-34

iv

Additional Resources 2-35 Practice 2 Overview: Introducing Exadata Features 2-36

3 Exadata Architecture Objectives 3-2 Exadata Software Architecture Overview 3-3 Exadata Software Architecture Details 3-5 Exadata Smart Flash Cache Architecture 3-7 Exadata Monitoring Architecture 3-9 Disk Storage Entities and Relationships 3-10 Interleaved Grid Disks 3-12 Flash Storage Entities and Relationships 3-13 Disk Group Configuration 3-14 Quiz 3-15 Summary 3-17 Additional Resources 3-18 Practice 3 Overview: Introducing Exadata Cell Architecture 3-19

4 Exadata Configuration Objectives 4-2 Exadata Installation and Configuration Overview 4-3 Initial Network Preparation 4-4 Configuration of New Exadata Servers 4-6 Answering Questions During the Initial Boot Sequence 4-7 Exadata Administrative User Accounts 4-11 Configuring a New Exadata Cell 4-12 Important I/O Metrics for Oracle Databases 4-13 Testing Performance Using CALIBRATE 4-14 Configuring the Exadata Cell Server Software 4-15 Creating Cell Disks 4-16 Creating Grid Disks 4-17 Creating Flash-Based Grid Disks 4-18 Configuring Hosts to Access Exadata Cells 4-19 Configuring ASM and Database Instances for Exadata 4-20 Configuring ASM Disk Groups for Exadata 4-21 Optional Configuration Tasks 4-22 Exadata Storage Security Overview 4-23 Exadata Storage Security Implementation 4-24 Quiz 4-26 Summary 4-29

v

Additional Resources 4-30 Practice 4 Overview: Configuring Exadata 4-31

5 Exadata Performance Monitoring and Maintenance Objectives 5-2 Monitoring Overview 5-3 Exadata Metrics and Alerts Architecture 5-4 Monitoring Exadata with Metrics 5-6 Monitoring Exadata with Metrics: Example 5-8 Monitoring Exadata with Alerts 5-9 Displaying Alert Examples 5-11 Monitoring Exadata with Active Requests 5-13 Monitoring SQL Execution Plans 5-14 Smart Scan Execution Plan Example 5-15 Predicate Offloading Considerations 5-16 Monitoring Exadata from Your Database 5-17 Monitoring Exadata with Wait Events 5-18 Monitoring Exadata with Enterprise Manager 5-19 Additional Monitoring Tools and Utilities 5-20 Cell Maintenance Overview 5-21 Automated Cell Maintenance Operations 5-23 Replacing a Damaged Physical Disk 5-24 Replacing a Damaged Flash Card 5-26 Moving All Disks from One Cell to Another 5-27 Using the Exadata Software Rescue Procedure 5-28 Quiz 5-30 Summary 5-32 Additional Resources 5-33 Practice 5 Overview: Monitoring Exadata 5-34

6 Exadata and I/O Resource Management Objectives 6-2 I/O Resource Management Overview 6-3 I/O Resource Management Concepts 6-5 I/O Resource Management Plans 6-6 IORM Architecture 6-7 I/O Resource Management Plans Example 6-8 Enabling Intradatabase Resource Management 6-11 Intradatabase Plan Example 6-12 Enabling IORM for Multiple Databases 6-13 Interdatabase Plan Example 6-14

vi

Category Plan Example 6-16 Complete Example 6-17 Using Database I/Os Metrics 6-20 Quiz 6-21 Summary 6-25 Additional Resources 6-26

7 Optimizing Database Performance with Exadata Objectives 7-2 Optimizing Performance 7-3 Flash Memory Usage 7-4 Compression Usage 7-6 Index Usage 7-8 ASM Allocation Unit Size 7-9 Minimum Extent Size 7-10 Quiz 7-11 Summary 7-13 Additional Resources 7-14 Practice 7 Overview: Optimizing Database Performance with Exadata 7-15

8 Database Machine Overview and Architecture Objectives 8-2 Introducing Database Machine 8-3 Database Machine X2-2 Full Rack 8-4 X2-2 Database Server Hardware Details (Sun Fire X4170 M2) 8-5 Start Small and Grow 8-6 Database Machine X2-8 Full Rack 8-7 X2-8 Database Server Hardware Details (Sun Fire X4800) 8-8 Database Machine Capacity 8-9 Database Machine Performance 8-10 Database Machine X2-2 Architecture 8-11 InfiniBand Network Architecture 8-13 X2-2 Leaf Switch Topology 8-14 Full Rack Spine and Leaf Topology 8-15 Scale Performance and Capacity 8-16 Scaling Out to Multiple Full Racks 8-17 Quiz 8-18 Summary 8-20

vii

9 Database Machine Configuration Objectives 9-2 Database Machine Implementation Overview 9-3 Configuration Worksheet Overview 9-5 Getting Started 9-6 Configuration Worksheet Example 9-7 Configuring ASM Disk Groups with Configuration Worksheet 9-11 Generating the Configuration Files 9-13 Other Pre-Installation Tasks 9-14 The Result After Installation and Configuration 9-15 Supported Additional Configuration Activities 9-17 Unsupported Configuration Activities 9-18 Quiz 9-20 Summary 9-22 Additional Resources 9-23

10 Migrating Databases to Database Machine Objectives 10-2 Migration Best Practices Overview 10-3 Performing Capacity Planning 10-4 Database Machine Migration Considerations 10-5 Choosing the Right Migration Path 10-6 Logical Migration Approaches 10-7 Physical Migration Approaches 10-9 Other Approaches 10-11 Post-Migration Best Practices 10-12 Quiz 10-13 Summary 10-15 Additional Resources 10-16 Practice 10 Overview: Migrating to Databases Machine using Transportable Tablespaces 10-18

11 Bulk Data Loading with Database Machine Objectives 11-2 Bulk Data Loading Overview 11-3 Preparing the Data Files 11-4 Staging the Data Files 11-5 Configuring the Staging Area 11-6 Configuring the Staging Area 11-7 Configuring the Target Database 11-10 Loading the Target Database 11-11

viii

Quiz 11-13 Summary 11-15 Additional Resources 11-16 Practice 11 Overview: Bulk Data Loading with Database Machine 11-17

12 Backup and Recovery with Database Machine Objectives 12-2 Backup and Recovery Overview 12-3 Using RMAN with Database Machine 12-4 General Recommendations for RMAN 12-5 Disk Based Backup Strategy 12-7 Disk Based Backup Configuration 12-8 Tape Based Backup Strategy 12-10 Tape Based Backup Configuration 12-11 Hybrid Backup Strategy 12-15 Restore and Recovery Recommendations 12-16 Backup and Recovery of Database Machine Software 12-17 Quiz 12-18 Summary 12-20 Additional Resources 12-21 Practice 12 Overview: Using RMAN Optimizations for Database Machine 12-22

13 Monitoring and Maintaining Database Machine Objectives 13-2 Monitoring Tools Overview 13-3 ILOM Overview 13-4 ILOM Example 13-6 DCLI Overview 13-7 DCLI Examples 13-8 InfiniBand Diagnostic Utilities 13-9 Database Machine Support Overview 13-11 Patching and Updating Overview 13-12 Maintaining Exadata Software 13-13 Maintaining Database Server Software 13-14 Maintaining Other Software 13-15 Quiz 13-16 Summary 13-18 Additional Resources 13-19 Practice 13 Overview: Using the distributed command line utility (dcli) 13-20

ix

A New Features in Update Release 11.2.1.3.1 Objectives A-2 New Features Overview A-3 Auto Service Request (ASR) A-4 The ASR Process A-5 ASR Requirements A-6 Oracle Linux 5.5 A-7 Enhanced Operating System Security A-8 Pro-active Disk Quarantine A-9 Other New Features A-10 Summary A-11

I t d tiIntroduction

Copyright 2010, Oracle and/or its affiliates. All rights reserved.

Course Objectives

After completing this seminar, you should be able to: Describe the key capabilities of Exadata and Database

Machine Identify the benefits of using Database Machine forIdentify the benefits of using Database Machine for

different application classes Describe the architecture of Database Machine and its

integration with Oracle Database, Clusterware and ASM Complete the initial configuration of Database Machine

D ib i d d h f i ti Describe various recommended approaches for migrating to Database Machine

Configure Exadata I/O Resource Management Monitor Database Machine health and optimize

performance


Exadata and Database Machine Administration Workshop 1 - 2

Audience and Prerequisites

This course is primarily designed for administrators who will configure and administer Oracle Exadata Database Machine.

Prior knowledge and understanding of the following is g g gassumed: Oracle Database 11g Release 2, including RAC and ASM. Linux and general network, storage and system

administration concepts. Recommended prior training:Recommended prior training:

Oracle Database 11g: Administration Workshop I Oracle Database 11g: Administration Workshop II Oracle 11g: RAC and Grid Infrastructure Administration Oracle Linux: Linux Fundamentals

Audience and Prerequisites This seminar is primarily designed for administrators who will configure and administer Oracle Exadata Database Machine


Exadata Database Machine.Please be mindful of the prerequisites because this course does not teach all aspects of the technologies used inside Database Machine. Rather it focuses on topics that are specific to Exadata and Database Machine. Prior knowledge and understanding of Oracle Database 11g Release 2, including Automatic Storage Management (ASM) and Real Application Clusters (RAC), is assumed. In addition, a working knowledge of Linux is assumed along with an understand of general networking, g g g g gstorage and system administration concepts.For students that do not meet these prerequisites, the recommended prior training includes the following courses: Oracle Database 11g: Administration Workshop I Oracle Database 11g: Administration Workshop II Oracle 11g: RAC and Grid Infrastructure Administration


g Oracle Linux: Linux Fundamentals

Course Scope

This course covers two main subject areas: Exadata Storage Server X2-2

This section focuses on the architecture and key capabilities of Exadata along with how to configure, monitor and optimize it.

Oracle Exadata Database MachineOracle Exadata Database Machine This section introduces students to Database Machine. The installation and configuration process is covered so that

students can make appropriate configuration decisions. Students also learn how to maintain, monitor and optimize

Database Machine after initial configuration.

Hardware is discussed during the course, however detailed hardware installation and maintenance is outside the scope of this course.

Course ScopeThis course covers two main subject areas:

The first section introduces students to Exadata Storage Server X2 2 (formerly known


The first section introduces students to Exadata Storage Server X2-2 (formerly known as Exadata Storage Server Version 2). Students learn about the architecture and key capabilities of Exadata along with how to configure, monitor and optimize it.

The second section introduces students to Oracle Exadata Database Machine. Students learn about the various Database Machine configurations. The installation and configuration process is covered so that students are equipped to make appropriate up-front configuration decisions. They also learn how to maintain, monitor and optimize Database Machine after initial configuration. Students are introduced to various options for migrating to Database Machine and learn how to select the best approach.

Although the hardware components of Database Machine are introduced and described to varying degrees throughout this course, you should consult the hardware documentation for specific hardware installation and maintenance details.


Course Contents

1. Introduction2. Exadata Overview3. Exadata Architecture4. Exadata Configuration5. Exadata Monitoring and Maintenance6. Exadata and I/O Resource Management7. Optimizing Database Performance with Exadata8. Database Machine Overview and Architecture9. Database Machine Configuration10 Migrating Databases to Database Machine10. Migrating Databases to Database Machine11. Bulk Data Loading with Database Machine12. Backup and Recovery with Database Machine13. Database Machine Monitoring and Maintenance

Course ContentsThe slide shows the ordering of lessons in this course.



Terminology

Unless otherwise indicated, Exadata refers to Exadata Storage Server. Typically a reference to Exadata refers to the combination of

software and hardware used in Exadata Storage Server. However at times there are specific references to ExadataHowever, at times there are specific references to Exadata hardware or Exadata software.

Unless otherwise indicated, Exadata X2-2 (formerly known as Exadata Version 2) is implied throughout the course. Exadata X2-2 is based on Sun hardware and is the only version of Exadata supported in Oracle Exadata Database MachineMachine.

Unless otherwise indicated, Database Machine refers to Oracle Exadata Database Machine. Typically, Database Machine refers to the entire system

including both hardware and software.

TerminologyThe slide indicates the conventions used throughout this course to abbreviate the formal product names for Exadata Storage Server and Oracle Exadata Database Machine


product names for Exadata Storage Server and Oracle Exadata Database Machine.


Additional Resources

Demonstrations (Viewlets) http://www.oracle.com/technetwork/tutorials/index.html Enter the Oracle Learning Library and conduct a search for

content in the Database Machine functional category. Look g yout for demonstrations with Exadata and Database Machine Version 2 Series in the title.

Oracle Technology Network (OTN) Exadata and Database Machine Page http://www.oracle.com/technetwork/database/exadata/index.

html OTN Exadata Discussion Forum

http://forums.oracle.com/forums/forum.jspa?forumID=829



Practice 1 Overview: Introducing the Laboratory EnvironmentIntroducing the Laboratory Environment

In this practice you will be introduced to the laboratory environment used to support all the practices during this course.



E d t O iExadata Overview


Objectives

After completing this lesson, you should be able to: Contrast the Exadata storage architecture with traditional

shared storage offerings Describe the hardware components of ExadataDescribe the hardware components of Exadata Outline the capabilities of Exadata Describe the main advantages of using Exadata compared

to traditional storage servers



Traditional Enterprise Database Storage DeploymentDeploymentDatabase Servers

Storage Arrays

Traditional Enterprise Database Storage DeploymentThe graphic in the slide illustrates the traditional deployment approach for multiple databases. Each database has an isolated allocation of storage resources and its bandwidth is limited by


Each database has an isolated allocation of storage resources and its bandwidth is limited by the hardware allocated to it. The isolation and dedication of hardware resources to individual databases can simultaneously lead to unused space and unused input/output (I/O) bandwidth for some databases, and overcommitted bandwidth with insufficient free space in others. The right balance is almost never achieved because real-world workloads are very dynamic. Large storage arrays are used today for many enterprise database deployments. These large storage arrays must be partitioned and have their bandwidth and space allocated across the d t b d li ti h i th t B th t hdatabases and applications sharing the storage array. Because these storage arrays house vast quantities of mission-critical data, they must be highly engineered, and consequentially very expensive, to deliver high levels of reliability and availability. Enterprise-class storage arrays are not only costly to procure, they also require highly specialized skills to manage and maintain. The result is a very high total cost of ownership when traditional large storage arrays are used in real-world enterprise database deployments.


Exadata Storage Deployment

Oracle Database 11g Servers

I/O Resource ManagementSmartstorageoperations

High performancestorage network

Storageconsolidation

(Transparent todatabases)

storage network

Data compression

Exadata Storage DeploymentThe graphic in the slide illustrates the general deployment approach with Exadata.

You can use Exadata to consolidate your storage environment Using Exadata multiple


p

You can use Exadata to consolidate your storage environment. Using Exadata, multiple databases can use storage from a single pool. Exadata uses Oracle Automatic Storage Management (ASM) to evenly distribute the storage load for every database across every available disk in the storage pool. Every database can use all the available disks to maximize performance. Exadata requires the use of Oracle Database 11g Release 2. Exadata works equally well with single-instance or Oracle Real Application Clusters (RAC) databases. Users and database administrators use the same tools and k l d th l d f ili ith B i b d i d t t d d tknowledge they are already familiar with. Being based on industry-standard components and technologies, Exadata is inexpensive to deploy. In addition, tight integration with the full suite of Oracle Database high-availability features, ensures that the reliability and integrity needs of mission-critical environments are met.

A key advantage of Exadata is the ability to offload some database processing to Exadata servers. With Exadata, the database can offload single table scan predicate filters and projections, join processing based on bloom filters, along with CPU-intensive


decompression and decryption operations. This ability is known as SQL processing offload or Smart Scan.

Exadata Storage Deployment (continued)In addition to Smart Scan, Exadata has other smart storage capabilities including the ability to offload incremental backup optimizations, file creation operations, and more. This approach yields substantial CPU memory and I/O bandwidth savings in the databaseapproach yields substantial CPU, memory, and I/O bandwidth savings in the database server resulting in potentially massive performance improvements.

Exadata includes Exadata Hybrid Columnar Compression. This feature provides very high levels of data compression implemented inside Exadata. Exadata Hybrid Columnar Compression allows the database to reduce the number of I/Os required to scan a table. For example, for data with a compression ratio of 10 to 1, the I/Os required to scan the data are reduced from 10 to 1 as well.

Exadata ensures that I/O resources are made available whenever, and to whichever, database needs them based on priorities and policies that you can define. The Database Resource Manager (DBRM) and Exadata I/O Resource Management (IORM) work together to manage intradatabase and interdatabase I/O resource usage to ensure that your defined service-level agreements (SLAs) are met when multiple applications and databases share Exadata storage.

Finally, even for queries that do not use Smart Scan, Exadata has many advantages overFinally, even for queries that do not use Smart Scan, Exadata has many advantages over conventional storage. Exadata is highly optimized for fast processing of large queries. It has been carefully architected to ensure no bottlenecks in the controller or in other components inside the storage server. It makes intelligent use of high-performance flash memory to boost performance and also uses a state-of-the-art InfiniBand network that has much higher throughput than conventional storage networks.


Exadata Implementation Architecture Overview

Oracle Database 11g Servers

Exadata Cell Exadata CellExadata Exadata

Linux OS Linux OS

DiskDisk

Exadatasoftware

Exadatasoftware

Exadata Implementation Architecture OverviewExadata is a self-contained storage platform that houses disk storage and runs the Exadata Storage Server Software provided by Oracle A single Exadata server is also called a cell A


Storage Server Software provided by Oracle. A single Exadata server is also called a cell. A cell is the building block for a storage grid. More cells provide greater capacity and I/O bandwidth. Databases are typically deployed across multiple cells, and multiple databases can share a single cell. The databases and cells communicate with each other via a high-performance InfiniBand network. Each cell is a purely dedicated storage platform for Oracle Database files although you can use Database File System (DBFS), a feature of Oracle Database, to store your business files i id th d t binside the database. Like other storage arrays, each cell is a computer with CPUs, memory, a bus, disks, network adapters, and the other components normally found in a server. It also runs an operating system (OS), which in the case of Exadata is Linux. The Oracle-provided software resident in the Exadata cell runs under this operating system. The OS is accessible in a restricted mode to administer and manage Exadata.


Introducing Exadata

High performance storage for Oracle Database Up to 1.8 GB/sec raw data bandwidth Up to 75,000 I/Os per second using flash

Exadata StorageServer

64 bit Intel-based Sun Fire Server Preinstalled software

Exadata Storage Server Software Oracle Linux x86_64 Drivers and Utilities Drivers and Utilities

Only available in conjunction with Database Machine

Introducing ExadataExadata is highly optimized for use with Oracle Database. Exadata delivers outstanding I/O and SQL processing performance for data warehousing and online transaction processing


and SQL processing performance for data warehousing and online transaction processing (OLTP) applications. Exadata is based on a 64 bit Intel-based Sun Fire server. Oracle provides the storage server software to impart database intelligence to the storage, and tight integration with Oracle Database and its features. Each cell is shipped with all the hardware and software components preinstalled including the Exadata Storage Server Software, Oracle Linux x86_64 operating system and InfiniBand protocol drivers. Since March 2010, Exadata is no longer offered as a standalone storage product. Now Exadata is only available for use in conjunction with Database Machine. Individual Exadata servers can still be purchased, however they must be connected to Database Machine. Custom configurations using Exadata are no longer supported for new installations.


Exadata Hardware Details(Sun Fire X4270 M2)(Sun Fire X4270 M2)

Processors 2 Six-Core Intel Xeon L5640 Processors (2.26 GHz)

Memory 24 GB (6 x 4 GB)Memory 24 GB (6 x 4 GB)

Local Disks 12 x 600 GB 15K RPM High Performance SASor 12 x 2 TB 7.2K RPM High Capacity SAS

Flash 4 x 96 GB Sun Flash Accelerator F20 PCIe Cards

Disk Controller Disk controller HBA with 512 MB battery backed cache

N t k T I fi iB d 4X QDR (40Gb/ ) tNetwork Two InfiniBand 4X QDR (40Gb/s) ports (1 dual-port PCIe 2.0 HCA)Four embedded Gigabit Ethernet ports

Remote Management 1 Ethernet port (ILOM)

Power Supplies 2 redundant hot-swappable power supplies

Exadata Hardware Details (Sun Fire X4270 M2)The slide shows a description of the Exadata Storage Server hardware.



Exadata Specifications

HP Disks HC Disks

Exadata Smart Flash Cache1 384 GB 384 GB

Raw Disk Capacity1 7.2 TB 24 TB

Uncompressed Data Capacity2 2 TB 7 TBUncompressed Data Capacity 2 TB 7 TB

Raw Disk Throughput (MBPS) 1,800 1,000

Effective Throughput with Flash (MBPS) 3,600 3,600

Disk I/Os per Second (IOPS) 3,600 1,440

Flash I/Os per Second (IOPS) 75,000 75,000p ( )

1 - Raw capacity calculated using 1 GB = 1000 x 1000 x 1000 bytes and 1 TB = 1000 x 1000 x 1000 x 1000 bytes.

2 - User Data: Actual space for uncompressed end-user data, computed after single mirroring (ASM normal redundancy) and after allowing space for database structures such as temporary space, logs, undo space, and indexes. Actual user data capacity varies by application. User Data capacity calculated using 1 TB = 1024 * 1024 * 1024 * 1024 bytes.

Exadata SpecificationsExadata is available in two configurations: with high performance (HP) disks or with high capacity (HC) disks The table in the slide lists the key capacity and performance


capacity (HC) disks. The table in the slide lists the key capacity and performance specifications for both configuration options.Note: MBPS stands for megabytes per second, IOPS stands for I/Os per second.Note: These metrics do not take into account compression. With compressed data, you can achieve much higher effective throughput rates. In all cases, actual performance will vary by application.


InfiniBand Network

InfiniBand: Is the Exadata storage network:

Provides highest performance available 40 Gb/sec each direction Is widely used in high-performance computing since 2002

Looks like normal Ethernet to host software: oo s e o a e e o os so a e All IP-based tools work transparently TCP/IP, UDP, HTTP, SSH,

and so on Has the efficiency of a SAN:

Zero copy and buffer reservation capabilities Is used for both storage and RAC interconnect:

Less configuration lower cost higher performance Less configuration, lower cost, higher performance Uses high-performance ZDP InfiniBand protocol (RDS V3):

Zero-copy, zero-loss Datagram protocol Open Source software developed by Oracle Very low CPU overhead

InfiniBand NetworkInfiniBand is the only storage network supported by Exadata because of its performance and proven track record in high-performance computing. InfiniBand works like normal Ethernet but


p g p p gmuch faster. It has the efficiency of a SAN, using zero copy and buffer reservation. Zero copy means that data is transferred across the network without intermediate buffer copies in the various network layers. Buffer reservation is used so that the hardware knows exactly where to place buffers ahead of time. These are two important characteristics that distinguish InfiniBand from normal Ethernet.InfiniBand is also supported as a unified network fabric for Exadata and the Oracle RAC interconnect. This facilitates easier configuration and fewer cables and switches. You can

l it f hi h f t l ti it h t t b kalso use it for high-performance external connectivity, such as to connect backup servers or ETL servers.On top of InfiniBand, Exadata uses the Zero Data loss UDP (ZDP) protocol. ZDP is open source software that is developed by Oracle. It is like UDP but more reliable. Its full technical name is RDS (Reliable Datagram Sockets) V3. The ZDP protocol has a very low CPU overhead with tests showing only a 2 percent CPU utilization while transferring 1 GB/sec of data.E h E d t i fi d ith d l t I fi iB d d d i d t b


Each Exadata server is configured with one dual-port InfiniBand card designed to be connected to two separate InfiniBand switches for high availability. Each InfiniBand link is able to carry the full data bandwidth of the entire cell, which means you can lose an entire network without losing any performance.

Classic Database I/O and SQL Processing Model

SELECT customer_idFROM ordersWHERE order_amount>20000;

Row returned1 6

Extents identified SQL processing:2 MB returned

2 5

I/O issued I/O executed:10 GB returned

3 4

Classic Database I/O and SQL Processing ModelWith traditional storage, all the database intelligence resides in the software on the database server To illustrate how SQL processing is performed in this architecture an example of a


server. To illustrate how SQL processing is performed in this architecture, an example of a table scan is shown in the graphic in the slide.

1. The client issues a SELECT statement with a predicate to filter a table and return only the rows of interest to the user.

2. The database kernel maps this request to the file and extents containing the table.3. The database kernel issues the I/Os to read all the table blocks.4 All the blocks for the table being queried are read into memory4. All the blocks for the table being queried are read into memory.5. SQL processing is conducted against the data blocks searching for the rows that satisfy

the predicate.6. The required rows are returned to the client.

As is often the case with the large queries, the predicate filters out most of the rows in the table. Yet all the blocks from the table need to be read, transferred across the storage network, and copied into memory. Many more rows are read into memory than required to


, p y y y qcomplete the requested SQL operation. This generates a large amount of unproductive I/O, which wastefully consumes resources and impacts application throughput and response time.

Exadata Smart Scan Model

SELECT customer_idFROM ordersWHERE order_amount>20000;

Row returned1 6

iDB command constructed

and sent to Exadata cells

Consolidated resultset built from allExadata cells

2 5

SQL processingin Exadata

2 MB returnedto server

3 4

Exadata Smart Scan ModelUsing Exadata, database operations are handled differently. Queries that perform table scans can be processed within Exadata and return only the required subset of data to the database


p y qserver. Row filtering, column filtering, some join processing, and other functions can be performed within Exadata. Exadata uses a special direct-read mechanism for Smart Scan processing. The above graphic illustrates how a table scan operates with Exadata:

1. The client issues a SELECT statement to return some rows of interest.2. The database kernel determines that Exadata is available and constructs an iDB

command representing the SQL command and sends it to the Exadata cells. iDB is a unique Oracle data transfer protocol that is used for Exadata storage communications.

3 The Exadata server software scans the data blocks to extract the relevant rows and3. The Exadata server software scans the data blocks to extract the relevant rows and columns which satisfy the SQL command.

4. Exadata returns to the database instance an iDB message containing the requested rows and columns of data. These results are not block images, so they are not stored in the buffer cache.

5. The database kernel consolidates the result sets from across all the Exadata cells. This is similar to how the results from a parallel query operation are consolidated.

6 The rows are returned to the client


6. The rows are returned to the client.Moving SQL processing off the database server frees server CPU cycles and eliminates a massive amount of unproductive I/O transfers. These resources are free to better service other requests. Queries run faster, and more of them can be processed.

Exadata Smart Storage Capabilities

Predicate filtering: Only the rows requested are returned to the database server

rather than all the rows in a table. Column filtering:g

Only the columns requested are returned to the database server rather than all the columns in a table.

Exadata Smart Storage CapabilitiesThe following database functions are integrated within Exadata: Exadata enables predicate filtering for table scans Rather than returning all the rows for


Exadata enables predicate filtering for table scans. Rather than returning all the rows for the database to evaluate, Exadata returns only the rows that match the filter condition. The conditional operators that are supported include =, !=, , =, IS [NOT] NULL, LIKE, [NOT] BETWEEN, [NOT] IN, EXISTS, IS OF type, NOT, AND, OR. In addition, many common SQL functions are evaluated by Exadata during predicate filtering. For a full list of functions that can be offloaded to Exadata, use the following query:

SELECT * FROM v$sqlfn_metadata WHERE offloadable = 'YES'; Exadata provides column filtering, also called column projection, for table scans. OnlyExadata provides column filtering, also called column projection, for table scans. Only

the requested columns are returned to the database server rather than all columns in a table. For tables with many columns, or columns containing LOBs, the I/O bandwidth saved by column filtering can be very large.

When used together, the combination of predicate and column filtering dramatically improves performance and reduces I/O bandwidth consumption. For example, when processing the following query, Exadata returns only the employee names that are longer than five characters:


SELECT name FROM employees WHERE LENGTH(name) > 5;Without predicate and column filtering, the storage subsystem would need to send all the rows and columns of the employees table to the database to evaluate.


Join processing: Simple star join processing is performed within Exadata.

Scans on encrypted data Scans on compressed data Scans on compressed data Scoring for Data Mining:

All data mining scoring functions are offloaded. Up to 10x performance gains.

Exadata Smart Storage Capabilities (continued) Exadata performs join processing for star schemas (between large tables and small

lookup tables) This is implemented using Bloom Filters which is a very efficient


lookup tables). This is implemented using Bloom Filters, which is a very efficient probabilistic method to determine whether an element is a member of a set.

Exadata performs Smart Scans on encrypted tablespaces and encrypted columns. For encrypted tablespaces, Exadata can decrypt blocks and return the decrypted blocks to Oracle Database, or it can perform row and column filtering on encrypted data. Significant CPU savings can be made within the database server by offloading the CPU-intensive decryption task to Exadata cells.

Smart Scan works in conjunction with Exadata Hybrid Columnar Compression so that column projection and row filtering can be executed along with decompression at the storage level to save CPU cycles on the database servers.

Exadata can perform scoring functions for data mining models. All data mining scoring functions, such as PREDICTION_PROBABILITY, are offloaded to Exadata cells for processing. This accelerates warehouse analysis while it reduces database server CPU consumption and the I/O load between the database server and Exadata.


p


Backups: I/O for incremental backups is much more efficient because

only changed blocks are returned to the database server. Create/extend tablespace:p

Exadata formats database blocks.

Exadata Smart Storage Capabilities (continued) The speed and efficiency of incremental database backups is enhanced with Exadata.

The granularity of change tracking in the database is much finer with Exadata With


The granularity of change tracking in the database is much finer with Exadata. With Exadata, changes are tracked at the individual Oracle block level rather than at the level of a large group of blocks. This results in less I/O bandwidth being consumed for backups and faster running backups.

With Exadata, the create/extend tablespace operation is also executed much more efficiently. Instead of formatting blocks in database server memory and writing them to storage, a single iDB command is sent to Exadata instructing it to format the blocks. Database server memory usage is reduced and I/O associated with the creation andDatabase server memory usage is reduced and I/O associated with the creation and formatting of the database blocks is eliminated with Exadata.


Exadata Smart Scan Scale-Out Example

DatabaseServer

dbs1

edsc1 edsc2 edsc14edsc13

InfiniBand Storage Network40 Gb/s Maximum

ExadataCell

Disks(12/cell)

Each cell can deliver 1.8 GB/s.

Total of 14 cells that can deliver 14 x 1.8 = 25.2 GB/s

Exadata Smart Scan Scale-Out ExampleThe example in the next three slides illustrates the power of Smart Scan in a quantifiable manner using a typical case in which multiple Exadata cells scale-out to share a workload


manner using a typical case in which multiple Exadata cells scale out to share a workload.The database server, depicted in the upper portion of the slide, is connected to the InfiniBand storage network, which can deliver a maximum of 40 gigabits per second (Gb/s). To keep the example clear and simple, assume that the InfiniBand storage network can deliver data at 40 Gb/s with no messaging overhead. We will also assume that a single database server has access to the full I/O bandwidth of all the Exadata cells.In this scenario, there are 14 Exadata cells. Assuming that each Exadata cell can deliver 1.8 ggigabytes (GB) of I/O throughput per second, the potential scanning power of all the Exadata cells is 25.2 GB per second.



DatabaseServer

select /*+ full(lineitem) */ count(*)from lineitem

where l_orderkey < 0;

dbs1Database asks to retrieve all blocks by doing a full table scan, and then

filters matching rows.


0 357 GB/s

If the table is 4800 GB in size, the complete scan would take approximately

16 minutes.

If the table is evenly distributedacross all disks, each cell

cannot send more than 40 / 14 = 2.85 Gb/s = 0.357 GB/s

to the database instance.

ExadataCell

Disks(12/cell)

0.357 GB/s

Disks are throttled by the network bandwidth!

Exadata Smart Scan Scale-Out Example (continued)Now assume a 4800 gigabyte table is evenly spread across the 14 Exadata cells and a query is executed which requires a full table scan As is commonly the case assume that the query


is executed which requires a full table scan. As is commonly the case, assume that the query returns a small set of result records.Without Smart Scan capabilities, each Exadata server behaves like a traditional storage server by delivering database blocks to the client database.Because the storage network is bandwidth-limited to 40 gigabits per second, it is not possible for the Exadata cells to deliver all their power. In this case, each cell cannot deliver more than 0.357 gigabytes per second to the database and it would take approximately 16 minutes to g g y p pp yscan the whole table.



DatabaseServer

select /*+ full(lineitem) */ count(*)from lineitem

where l_orderkey < 0;

dbs1 Database asks Exadata cellsto send back all matching rows.


If the table is 4800 GB in size, the complete table scan will complete in approximately

three minutes and ten seconds!

E h ll t

If the table is evenly distributedacross all disks, each cell

cannot send more than 40 / 14 = 2.85 GB/s = 0.357 GB/s

to the database instance.

ExadataCell

1 8 GB/s

Disks(12/cell)

Each cell can scan at aspeed of 1.8 GB/s,

and send its matchingrows to the database

instance. This represents a total scan at a speed

of 25.2 GB/s!

1.8 GB/s

Exadata Smart Scan Scale-Out Example (continued)Now consider if Smart Scan is enabled for the same query. The same storage network bandwidth limit applies However this time the entire 4800 GB is not transported across the


bandwidth limit applies. However this time the entire 4800 GB is not transported across the storage network; only the matching rows are transported back to the database server. So each Exadata cell can process its part of the table at full speed; that is, 1.8 GB per second. In this case, the entire table scan would be completed in approximately three minutes and ten seconds.


Exadata Hybrid Columnar Compression

Warehouse Compression

10 t i

Archival Compression

15 t i

Optimized for Speed Optimized for Space

10x average storage savings 10x scan I/O reduction Optimized for query performance

15x average storage savings Up to 50x on some data

Some access overhead For cold or historical data

Reduced Warehouse SizeBetter Performance

Can mix compression types by partition for ILM

Reclaim DisksKeep Data Online

Exadata Hybrid Columnar CompressionIn addition to the basic and OLTP compression capabilities of Oracle Database 11g, Exadata includes Exadata Hybrid Columnar Compression


includes Exadata Hybrid Columnar Compression.Exadata Hybrid Columnar Compression offers higher compression ratios for direct path loaded data. This compression capability is recommended for data that is not updated frequently. You can specify Exadata Hybrid Columnar Compression at the table, partition, and tablespace level. You can also choose between two types of Exadata Hybrid Columnar Compression, to achieve the proper trade-off between disk usage and CPU consumption, depending on your requirements: Warehouse compression: This type of compression is optimized for query performance,

and is intended for data warehouse applications. Online archival compression: This type of compression is optimized for maximum

compression ratios, and is intended for data that does not change frequently. You can use Exadata Hybrid Columnar Compression on complete tables or in combination with basic and OLTP compression by using partitioning.


Note: A compression advisor, provided by the DBMS_COMPRESSION package, helps you determine the expected compression ratio for a particular table with a particular compression method.

Exadata Hybrid Columnar CompressionArchitecture OverviewArchitecture Overview

Compression Unit (CU)

Block HeaderCU Header

C1

Block Header

C2C4

Block Header

C5

Block Header

C8C7

A compression unit is a logical structure spanning multiple database blocks.E h i lf t i d ithi i it

C1

C2C3 C4

C5 C6

Each row is self-contained within a compression unit. Data organized by column during data load. Each column compressed separately. Smart Scan is supported.

Exadata Hybrid Columnar Compression Architecture OverviewExadata Hybrid Columnar Compression is a new method for organizing data in database blocks Tables are organized into sets of rows called compression units (CU) Within a


blocks. Tables are organized into sets of rows called compression units (CU). Within a compression unit, data is organized by column and then compressed. The column organization of data brings similar values close together, enhancing compression ratios. Each row is self-contained within a compression unit.In addition to providing excellent compression, Exadata Hybrid Columnar Compression works in conjunction with Smart Scan so that column projection and row filtering can be executed along with decompression at the storage level to save CPU cycles on the database servers.Note: Although the diagram in the slide shows a compression unit containing four data blocks, it should not be assumed that a compression unit always contains fours blocks. The size of a compression unit is determined automatically by Oracle Database based on various factors in order to deliver the most effective compression result while maintaining excellent query performance.


Exadata Smart Flash Cache

High performance cache for frequently accessed objects Excellent for absorbing repeated random reads Allows optimization by application table

Hundreds ofI/Os per Sec

Tens of Thousands of I/Os per Second

Exadata Smart Flash CacheFor many years, a constraining factor for storage performance has been the number of random I/Os per second (IOPS) that a disk can deliver To compensate for the fact that even


random I/Os per second (IOPS) that a disk can deliver. To compensate for the fact that even a high performance disk can deliver only a few hundred IOPS, large storage arrays with hundreds of disks are required to deliver in excess of 60,000 IOPS.Exadata provides Exadata Smart Flash Cache, a caching mechanism for frequently accessed data. It is a write-through cache which is useful for absorbing repeated random reads, and very beneficial to OLTP. Using Exadata Smart Flash Cache, a single Exadata cell can support up to 75,000 IOPS, two cells can support up to 150,000 IOPS, and so on.Exadata Smart Flash Cache focuses on caching frequently accessed data and index blocks, along with performance critical information such as control files and file headers. In addition, DBAs can influence caching priorities using the CELL_FLASH_CACHE storage attribute for specific database objects.


Exadata Smart Flash Cache

High performance cache that understands different types of database I/O: Frequently accessed data and index blocks are cached. Control file reads and writes are cachedControl file reads and writes are cached. File header reads and writes are cached. DBA can influence caching priorities.

I/Os to mirror copies are not cached. Backup-related I/O is not cached. Data Pump I/O is not cached. Data file formatting is not cached. Table scans do not monopolize the cache.

Exadata Smart Flash Cache (continued)In more recent times, vast and expensive storage arrays have introduced equally expensive nonvolatile memory caches to improve performance However these caches know nothing


nonvolatile memory caches to improve performance. However, these caches know nothing about the applications using them, so their efficiency is limited when compared to their cost.With Exadata, each database I/O is tagged with metadata indicating the I/O type. Exadata Smart Flash Cache uses this information to make intelligent decisions about how to use the cache. This cooperation ensures the efficient use of Exadata Smart Flash Cache.For example, with ASM mirroring turned on, multiple copies of each data block must be written to disk to deliver the desired level of data protection. However, there is usually no p yneed to cache the secondary copies of a block because ASM will read the primary copy if it is available. A traditional storage array would not know about this characteristic leading to caching inefficiencies.Similarly, with traditional storage arrays, backups and exports will typically cause all the data to be loaded into the cache even though the operation will not read the data repeatedly. Exadata knows that there is no need to fill the cache with backup and export data. The same is true for data file formatting operations Finally Exadata does not flood the cache with data


is true for data file formatting operations. Finally, Exadata does not flood the cache with data from full table scans, as is the case with most storage arrays.

Exadata Storage Index

Storage Index in Memory

Region Index

B:1/5 B:3/8 E:a/jG:4/9

SELECT * FROM T1 WHERE B

Exadata Storage Index (continued)The storage statistics represent the data distribution (minimum and maximum values) of columns that are considered well clustered by Exadata. Exadata has heuristics to transparently determine hat col mns are cl stered eno gh to be incl ded in the storage indedetermine what columns are clustered enough to be included in the storage index.The storage index works best when the following conditions are true: The data is roughly ordered so that the same column values are clustered together. The query has a predicate on a storage index column checking for =, or some

combination of these.It is important to note that the storage index works transparently with no user input. There is no

d t t d t th t i d Th l t i fl th t i d i tneed to create, drop, or tune the storage index. The only way to influence the storage index is to load your tables using presorted data.Also, because the storage index is kept in memory, it disappears when the cell is rebooted. The first queries that run after a cell is rebooted automatically cause the storage index to be rebuilt.The storage index works for data types whose binary encoding is such that byte-wise binary lexical comparison of two values of that data type is sufficient to determine the ordering of those two values This includes data types like NUMBER DATE and VARCHAR2 However NLS datatwo values. This includes data types like NUMBER, DATE, and VARCHAR2. However, NLS data types are an example of data types that are not included for storage index filtering.


Storage Index with Partitions Example

ORDER# ORDER_DATE(Partition Key)

SHIP_DATE ITEM

1 2007 20072 2008 2008

Queries on SHIP_DATE do not benefit from ORDER_DATE partitioning: However SHIP_DATE is highly correlated with ORDER_DATE.

008 0083 2009 2009

Storage index provides partition pruning like performance for queries on SHIP_DATE: Takes advantage of ordering created by partitioning

Storage Index with Partitions ExampleThe example in the slide contains correlated columns. ORDER_DATE is highly correlated with SHIP DATE The dates are generally correlated because usually a ship date is close to an


SHIP_DATE. The dates are generally correlated because usually a ship date is close to an order date.If your table is partitioned by ORDER_DATE, and you execute a query using ORDER_DATE as a filter, then partition pruning is used to read only the relevant partitions. However, if you do a query using only SHIP_DATE in the WHERE clause, partition pruning cannot be used to optimize the query. However, if SHIP_DATE is part of the storage index, the storage index is used to skip all the blocks that do not correspond to your query. This filtering takes place at the storage level. The storage index helps the SHIP_DATE query to take advantage of the natural ordering implied by the ORDER_DATE partitioning and the natural correlation that exists between the ORDER_DATE and SHIP_DATE columns.


Database File System

Database File System (DBFS) enables the database to be used as a file system.

Files are stored as SecureFiles LOBs inside database tables that are stored in Exadata. Protected like any Oracle data ASM mirroring, Data Guard,

Flashback, and so on Shared storage for ETL staging, scripts, reports and other

application files 5 to 7 GB/sec file system I/O throughput capable on a full rack

Database Machine

Copy files to DBFSTransform and load into

database tables

Database File SystemOracle Database File System (DBFS) enables an Oracle database to be used as a POSIX-compatible file system on Linux DBFS is an Oracle Database capability that provides


compatible file system on Linux. DBFS is an Oracle Database capability that provides Exadata users with a high performance mechanism to load data into an Oracle database. DBFS can be used to stage your ETL files for example. Inside DBFS files are stored as SecureFiles LOBs. A set of PL/SQL procedures implement the file system access primitives, such as open, create, and so on. The dbfs_client utility enables the mounting of a DBFS file system as a mount point on Linux. It provides the mapping from file system operations to database operations. The dbfs_client utility runs

l t l i d i t t ith th k l th h th FUSE lib i f t tcompletely in user space and interacts with the kernel through the FUSE library infrastructure. Note: ASM Cluster File System (ACFS) is not supported over Exadata.


I/O Resource Management

TraditionalStorage Server

Y t

FIFO Disk Queue

H L H L L LRDBMS

I/O Requests

Exadata

You cannot influence theI/O scheduler.

High-priorityworkloadrequest

Low-priorityworkloadrequest

I/O scheduler based onExadata

I/O Requests

I/O scheduler based on prioritization scheme

L L L L

H HRDBMS L H H H

I/O Resource ManagementWith traditional shared storage, balancing the work of multiple databases sharing the storage subsystem is inherently difficult This issue is illustrated by the graphic at the top of the slide


subsystem is inherently difficult. This issue is illustrated by the graphic at the top of the slide, which shows how traditional storage servers handle I/O requests. In essence, they queue I/O requests in a first-in, first-out (FIFO) order, which makes no distinction between high-priority and low-priority requests. Exadata allows for allocation of I/O resources based on user-specified priorities and policies. This is illustrated in the graphic at the bottom of the slide where the Exadata I/O scheduler executes I/O requests based on a prioritization scheme. It does that by internally queuing I/O

t t t l i it b t i t i kl d f fl di th di krequests to prevent a low-priority but intensive workload from flooding the disks.I/O resource management is covered in more detail in the lesson titled Exadata and I/O Resource Management.


Benefits Multiply

Less with ExadataMultiple terabytes of user Even less withLess with Exadata Hybrid Column Compression

Multiple terabytes of user data normally requires

multiple terabytes of I/O

Even less withpartition pruning

Results in real-time on Database

Storage index skips worthless I/O

Smart scan means that only the results are returned to the

database

Database Machine

Benefits MultiplyThis is an example that shows you how the main Exadata features that were introduced in this lesson can work together to multiply the benefits of Exadata


lesson can work together to multiply the benefits of Exadata.Assume you have a multi-terabyte table and somebody runs a query that is interested in a small subset of the data, but causes a full table scan. Traditionally, the system would have to scan the terabytes of data.However, using Exadata Hybrid Columnar Compression could reduce the size of the table.If the table is partitioned, the optimizer could use partition pruning to eliminate a substantial proportion of the dataproportion of the data. Using storage indexes, Exadata might further reduce the amount of physical I/O that is executed.Finally, because of Smart Scan, the only data returned to the database is the data of interest to the query, some of which may have been cached inside Exadata Smart Flash Cache. This example shows how the various Exadata and Oracle Database features can work in harmony to improve the performance of a single operation using Database Machine


harmony to improve the performance of a single operation using Database Machine.

Exadata Key Benefits for Data Warehousing

Exadata uses more connections: Modular storage cell building blocks organized into

massively parallel grid Exadata has bigger network pipes:

InfiniBand network transfers data faster than Fibre Channel. Exadata transports less data between the storage and the

database: Query processing is moved into storage to dramatically

reduce data sent to servers while unloading server CPUs. Exadata Hybrid Columnar Compression reduces theExadata Hybrid Columnar Compression reduces the

number of physical I/Os for large table scans. In-memory parallel query provides a powerful alternative

query strategy that complements Exadata.

Exadata Key Benefits for Data WarehousingOne of the key benefits of Exadata is extremely enhanced performance for data warehousing applications By replacing your existing storage with Exadata it is possible to get up to 100


applications. By replacing your existing storage with Exadata, it is possible to get up to 100 times speedup for your data warehousing queries. The larger the data warehouse, the greater the speedup from using Exadata.Exadata addresses three key dimensions of database I/O that can hamper data warehouse performance. Exadata is based on a massively parallel architecture, which provides more connections

to deliver more data faster between the storage servers and the database servers. Exadata is built using wide network pipes that provide extremely high bandwidth

between the storage servers and the database servers. Exadata uses InfiniBand as the storage network ,which provides a throughput of 40 Gb/sec with very low latency. This is many times the bandwidth provided by traditional SAN storage networks.

Exadata is database-aware and can transport just the data required to satisfy SQL requests resulting in less data being sent between the storage servers and the database servers


servers.Basically, Exadata reduces the volume of data transported and moves data faster compared with other storage solutions.

Exadata Key Benefits for Data Warehousing (continued)In addition, Exadata introduces additional capabilities that can further enhance data warehouse performance.Exadata includes Exadata Hybrid Columnar Compression. This feature provides very high levels of data compression implemented inside Exadata. Exadata Hybrid Columnar Compression benefits large scale scans, commonly used in data warehousing, by efficiently scanning vast volumes of data using a fraction of I/Os. Compression ratios of 10 to 1 are common which means that a 10 TB table can be scanned using 1 TB of disk I/O.Exadatas tight integration with Oracle Database results in an intelligent platform for data warehousing The complete solution uses a range of technologies to deliver the best result notwarehousing. The complete solution uses a range of technologies to deliver the best result, not just relying on one approach to the problem. An example of this is the new in-memory parallel query feature of Oracle Database 11g Release 2. Normally, a Smart Scan would be used to execute portions of a query inside Exadata and return the minimum amount of data to the database server. In some cases, however, it may be more efficient to read all the required data into the memory on the database servers and process the query that way.In-memory parallel query enhances query performance by minimizing or even completely eliminating additional physical I/O for a particular query. Oracle automatically decides if an object being accessed using parallel execution benefits from being cached in the database buffer cache. The decision to cache an object is based on a well-defined set of heuristics including size of the object and the frequency that it is accessed. In-memory parallel query harnesses the aggregated memory across a database cluster for parallel operations enabling it to scale-out as the number of nodes in a cluster increases In anparallel operations, enabling it to scale-out as the number of nodes in a cluster increases. In an Oracle RAC environment, Oracle maps fragments of the object into each of the buffer caches on the active instances. By creating this mapping, Oracle knows which buffer cache to access to find a specific part or partition of an object. Using this information, Oracle Database will prevent multiple instances from reading the same information from disk over and over again, thus maximizing the amount of memory that can be used to cache the objects. In-memory parallel query nicely complements Exadata. Using this combination, some queries

b ffi i tl t d ith littl dditi l I/O b i i t bl i th d t b b ffcan be efficiently executed with little or no additional I/O by pinning tables in the database buffer cache whereas others can harness the power of Smart Scan inside Exadata.


Exadata Key Benefits for OLTP

Exadata uses more connections: Modular storage cell building blocks organized into

massively parallel grid Exadata has bigger network pipes:

InfiniBand net ork transfers data faster than Fibre Channel InfiniBand network transfers data faster than Fibre Channel. Exadata Smart Flash Cache:

Provides high-performance cache for frequently accessed objects

Is excellent for absorbing repeated random readsAll i i i b li i bl Allows optimization by application table

Hundreds ofI/Os per Sec

Tens of Thousands of I/Os per Second

Exadata Key Benefits for OLTPSome of the fundamental architectural characteristics of Exadata that are beneficial for data warehousing are equally relevant and beneficial for online transaction processing (OLTP)


warehousing are equally relevant and beneficial for online transaction processing (OLTP). The high-performance, low-latency, InfiniBand network used in conjunction with the massively parallel grid architecture of Exadata is ideal for supporting many thousands of simultaneous users.In addition, the introduction of Exadata Smart Flash Cache is of particular benefit to OTLP performance. Exadata Smart Flash Cache allows each Exadata cell to deliver up to 75,000 IOPS. In addition, Oracle Database and Exadata Smart Flash Cache work closely with each

th Thi ti ti i th f E d t S t Fl h C h th t l thother. This cooperation optimizes the usage of Exadata Smart Flash Cache so that only the most frequently accessed and performance-sensitive data is cached. Users have additional control over which database objects should be cached more aggressively than others, and which ones should not be cached at all.


Quiz

Exadata and Database Machine are two different names that designate the same thing.1. TRUE2 FALSE2. FALSE

Answer: 2



Quiz

What are the three unique benefits of Exadata compared to traditional storage servers?1. Larger disk sizes2 Smart storage capabilities2. Smart storage capabilities3. Higher storage network bandwidth4. Higher RAM capacity5. Integrated database I/O resource management

Answer: 2, 3, 5



Summary

In this lesson, you should have learned how to: Contrast the Exadata storage architecture with traditional

shared storage offerings Describe the hardware components of ExadataDescribe the hardware components of Exadata Outline the capabilities of Exadata Describe the main advantages of using Exadata compared

to traditional storage servers



Additional Resources

Lesson Demonstrations (Viewlets) Introduction to Smart Scan

http://st-curriculum.oracle.com/demos/db/11g/r2/dbmach/021ExadataSmartScanIntro/021exadatasmartscanintro_viewlet_swf.html

Introduction to Exadata Hybrid Columnar Compression http://st-

curriculum.oracle.com/demos/db/11g/r2/dbmach/022ExadataCompressionIntro/022exadatacompressionintro_viewlet_swf.html

Introduction to Exadata Smart Flash Cache http://st-

curriculum.oracle.com/demos/db/11g/r2/dbmach/023ExadataFlashCacheIntro/023exadataflashcacheintro_viewlet_swf.html

Smart Scan Scale Out Examplehtt // t http://st-curriculum.oracle.com/demos/db/11g/r2/exadatav2/smartscanscaleoutexample/smartscanscaleoutexample.swf

Storage Index http://st-

curriculum.oracle.com/demos/db/11g/r2/exadatav2/storageindex/storageindex.swf



Practice 2 Overview: Introducing Exadata FeaturesIntroducing Exadata Features

In these practices, you are introduced to four major capabilities of Exadata, namely: Smart Scan Exadata Hybrid Columnar CompressionExadata Hybrid Columnar Compression Exadata Smart Flash Cache Storage Index



Exadata ArchitectureExadata Architecture


Objectives

After completing this lesson, you should be able to describe: The Exadata architecture The relationship between the various storage abstractions

used in Exadataused in Exadata



Exadata Software Architecture Overview

DB Instance

ASM

DB Server

DBRM

DB Instance

DB Server

DB Instance

DB ServerSingle-instance DB RAC DB

EnterpriseManager

ASM ASMSingle ASM cluster

DBRMDBRM

LIBCELL LIBCELL LIBCELL

InfiniBand Storage Switch/NetworkiDB Protocol over

InfiniBand with Path Failover

Oracle Linux Oracle Linux Oracle LinuxCell Control

CLI(cellcli/dcli)

SSH

Exadata Server Exadata Server Exadata Server

Oracle Linux

CELLSRV

IORM

MS

RS

Oracle Linux

CELLSRV

IORM

MS

RS

Oracle Linux

CELLSRV

IORM

MS

RS

Exadata Software Architecture OverviewThe architecture of Exadata includes components on the database server and on the Exadata server. The overall architecture is shown in the slide. The following components reside on the


gdatabase server: Oracle Database communicates with Exadata using the Intelligent Database protocol

(iDB). iDB is implemented in the database kernel and LIBCELL. iDB is a unique Oracle data transfer protocol, built on Reliable Datagram Sockets (RDS), that runs on industry standard InfiniBand networking hardware. iDB provides data intelligence between the database and Exadata and enables ASM and database instances to utilize Exadata-specific features such as Smart Scan and I/O Resource Management iDB transparentlyspecific features, such as Smart Scan and I/O Resource Management. iDB transparently maps database operations to Exadata-enhanced operations. Single-instance or Oracle RAC databases access Exadata storage cells using iDB.

Automatic Storage Management (ASM) is required and provides a file system and volume manager optimized for Oracle Database.

Database Resource Manager (DBRM), in combination with Exadata I/O Resource Management (IORM), ensures that I/O resources are allocated based on defined priorities.


Note: The slide illustrates the recommended configuration where a single ASM cluster is used to consolidate storage for all of your databases. Alternatively, you can connect multiple separate ASM environments with separate disk groups to Exadata.

Exadata Software Architecture Overview

DB Server DB Server DB ServerSingle-instance DB RAC DB

EnterpriseManager

Single ASM cluster ASM ASM ASM

DB Instance

DBRM

DB Instance DB Instance

DBRMDBRM

LIBCELL LIBCELL LIBCELL

Oracle Linux

iDB Protocol over InfiniBand with Path

FailoverInfiniBand Storage Switch/Network

Oracle Linux Oracle Linux

Exadata Server

Oracle Linux

CELLSRV

IORM

MS

RS

Exadata Server Exadata Server

Cell ControlCLI

(cellcli/dcli)

SSH

Oracle Linux

CELLSRV

IORM

MS

RS

Oracle Linux

CELLSRV

IORM

MS

RS

Exadata Software Architecture Overview (continued)The software components that reside in Exadata include: Oracle Linux provides the Exadata server operating system.


p p g y Cell Server (CELLSRV) is the primary Exadata software component and provides the

majority of Exadata storage services. CELLSRV is a multithreaded server. CELLSRV serves simple block requests, such as database buffer cache reads, and Smart Scan requests, such as table scans with projections and filters. CELLSRV also implements I/O Resource Management (IORM), which works in conjunction with Database Resource Manager (DBRM), to meter out I/O bandwidth to the various databases and consumer groups issuing I/Os. Finally, CELLSRV collects numerous statistics relating to its operations. O l D t b d ASM t i t ith dOracle Database and ASM processes use LIBCELL to communicate with CELLSRV, and LIBCELL converts I/O requests into messages that are sent to CELLSRV using the iDB protocol.

Management Server (MS) provides Exadata cell management and configuration. It works in cooperation with the Exadata cell command-line interface (CellCLI). Each cell is individually managed with CellCLI. CellCLI can only be used from within a cell to manage that cell, however you can run the same CellCLI command remotely on multiple cells with the dcli utility In addition MS is responsible for sending alerts and collects some


the dcli utility. In addition, MS is responsible for sending alerts and collects some statistics in addition to those collected by CELLSRV.

Restart Server (RS) is used to start up/shut down the CELLSRV and MS services and monitors these services to automatically restart them if required.

Exadata Software Architecture Details

Exadata Cell Database Server

RDBMS instance ASM instance

dskm

SGA

dskmASMI/O

Proc

SGAData

SmartFlash Cache

ASMI/O

Proc

/opt/oracle/cell/cellsrv/deploy/

config

/etc/oracle/cell/network-config

diskmon

LIBCELL

iDB Protocol iDB Protocol

cellinit.oracellip.ora

bond0

cellsrv

RS

MSCellCLI

CELLSRVADR

adrci

cellinit.ora

cell_disk_config.xml

css

LIBCELL

InfiniBand switch

List localinterface IP

List accessibleExadata cells

MS internaldictionary

andCELLSRV internalparameters andlocal interface IP

Exadata Software Architecture DetailsDatabase-host side Exadata software: LIBCELL Library: Provides UNIX-like I/O primitives and is linked with ASM, RDBMS, and


C b a y o des U e /O p t es a d s ed t S , S, a dASM utilities. It uses the iDB Protocol to communicate with Exadata.

DISKMON (Network/Cell Monitor): Checks the network interface state and cell liveness. It uses a nodewide master process and one slave process (dskm) for each RDBMS or ASM instance. The master performs monitoring and propagates state information to the slaves. Slaves use the SGA to communicate with RDBMS or ASM processes. If there is a failure in the cluster, DISKMON performs I/O fencing to protect data integrity. Cluster Synchronization Services (CSS) still decides what to fence Master DISKMON starts withSynchronization Services (CSS) still decides what to fence. Master DISKMON starts with the clusterware processes. DISKMON also performs DBRM plan propagation.

Cell-side Exadata software: CELLSRV is a multithreaded server which provides the majority of Exadata storage

services. It provides smart storage capabilities, serves data blocks when offloading is not possible, and implements I/O Resource Management to meter out I/O bandwidth.

Management Server (MS) is an OC4J application that provides storage cell management


Management Server (MS) is an OC4J application that provides storage cell management and configuration functions, such as cell administration, and metrics and alerts generation. It also communicates with CELLSRV and the operating system.

Exadata Software Architecture Details (continued) Restart Server (RS): Monitors CELLSRV and MS and restarts them, if necessary. CellCLI: Executes user cell administration commands. The user must connect to the cell

to use CellCLI CellCLI communicates with MS using Web Servicesto use CellCLI. CellCLI communicates with MS using Web Services. ADRCI: CELLSRV uses the Automatic Diagnostic Repository (ADR) to log software errors.

An Exadata administrator may use the ADR viewer (ADRCI) to view and package ADR incidents.

InfiniBand provides a high-speed, high-bandwidth, and low-latency network fabric to support Exadata. InfiniBand is the only network fabric supported for communication between Exadata and database servers. The InfiniBand implementation in Exadata and Database Machine uses the open source RDS/Open Fabrics Enterprise Distribution (OFED). These packages are preinstalled in Exadata and Database Machine.Note: Exadata requires Oracle Database 11g Release 2 or later.


Exadata Smart Flash Cache Architecture

DB DB DB

est

1

est

1

Write Operation Read Operationon previously cached data

Read Operationon uncached data

cells

rv

Ack

now

ledg

emen

t

3 1

2 4ce

llsrvR

ead

Req

ue

3

2

cells

rvRea

d R

eque

3

2 42 4 2 4

Exadata SmartFlash Cache

Exadata Smart Flash Cache ArchitectureExadata Smart Flash Cache provides a caching mechanism for frequently accessed data on each Exadata cell Exadata Smart Flash Cache works in conjunction with Oracle Database to


each Exadata cell. Exadata Smart Flash Cache works in conjunction with Oracle Database to intelligently optimize the efficiency of the cache.Each database I/O is tagged with the following metadata: The CELL_FLASH_CACHE setting for the object associated with the I/O:

- DEFAULT specifies that Exadata Smart Flash Cache is used normally.- KEEP specifies that Exadata Smart Flash Cache is used more aggressively.

NONE specifies that Exadata Smart Flash Cache is not used- NONE specifies that Exadata Smart Flash Cache is not used. A cache hint, which is assigned by the database based on the reason for the I/O:

- CACHE indicates that the I/O should be cached. For example, the I/O is for an index lookup.

- NOCACHE indicates that the I/O should not be cached. For example, the I/O is for a mirrored block of data or is a log write.EVICT indicates that data sho ld be remo ed from the cache For e ample hen


- EVICT indicates that data should be removed from the cache. For example, when an ASM rebalance operation moves data between different disks, the cached copies that correspond to the original location are removed from the cache.

Exadata Smart Flash Cache Architecture (continued)In addition, Exadata Smart Flash Cache takes the following into consideration when processing I/O: I/O size: Large I/Os on objects with CELL FLASH CACHE set to DEFAULT are not cachedI/O size: Large I/Os on objects with CELL_FLASH_CACHE set to DEFAULT are not cached. Current cache load: Smart table scans are usually directed to disk. However, if the object

has a CELL_FLASH_CACHE setting of KEEP, some reads may be satisfied using Exadata Smart Flash Cache in order to best utilize the combined throughput of the disks and the cache.

Exadata Smart Flash Cache uses all of the aforementioned information to make intelligent decisions about which data is suitable for caching and which is not.Exadata Smart Flash Cache is a write-through cache. This means that for write operations, CELLSRV writes data to disk and sends an acknowledgement to the database so it can continue without any interruption. Then, if the data is suitable for caching, it is written to Exadata Smart Flash Cache. Write performance is not improved or diminished using this method. However, if a subsequent read operation needs the same data, it is likely to benefit from the cache.For read operations, CELLSRV must first determine if the requested data is already in Exadata S t Fl h C h CELLSRV i t i i h h t bl hi h it t i klSmart Flash Cache. CELLSRV maintains an in-memory hash table, which it uses to quickly determine which data blocks reside in Exadata Smart Flash Cache. If the requested data is cached, a cache lookup is used to satisfy the I/O request.For read operations that cannot be satisfied using Exadata Smart Flash Cache, a disk read is performed and the requested information is sent to the database. Then if the data is suitable for caching, it is written to Exadata Smart Flash Cache. When suitable data is inserted into a full cache a prioritized least recently used (LRU) algorithmWhen suitable data is inserted into a full cache, a prioritized least recently used (LRU) algorithm determines which data to replace. Objects with a CELL_FLASH_CACHE setting of KEEP are subject to a different cache retention policy than objects with a CELL_FLASH_CACHE setting of DEFAULT. KEEP objects have priority over DEFAULT objects so that new data from a DEFAULTobject will not push out cached data from any KEEP objects. To prevent KEEP objects from monopolizing the cache, they are allowed to occupy no more than 80% of the total cache size. Also, to prevent unused KEEP objects from indefinitely occupying the cache, they are subject to an additional aging policy which periodically purges unused KEEP object dataan additional aging policy, which periodically purges unused KEEP object data.


Exadata Monitoring Architecture

Exadata CellExadata Cell From the Enterprise

Enterprise Manager

agentOMS dcli

Exadata Cell

DataSmart

Flash Cache

eth0

SSH / CellCLI

cellsrv MS

CELLSRVADR

adrci

CellCLI

Network switch

eth0

Exadata Monitoring ArchitectureFor monitoring, there is an Enterprise Manager plug-in that you use in conjunction with Grid Control. Using this plug-in, you can monitor all the Exadata cells in your enterprise.


g g y yThe Enterprise Manager plug-in for Exadata does not require an agent on each Exadata cell. Instead, an existing Enterprise Manager agent uses SSH to connect to each cell and execute CellCLI commands. Using this architecture, monitoring information from numerous Exadata cells can be consolidated on to a single Enterprise Manager screen.The dcli utility facilitates centralized management across a group of cells. It can be used to execute CellCLI and other cell-level operating system commands across a group of cells and

id lid t d i f th t t Th d li tilit d lti l ll iprovide a consolidated view of the output. The dcli utility runs commands on multiple cells in parallel threads. The cells are referenced by their network name or IP address. Files can be copied to cells and command scripts can be executed on cells by using this utility. Finally, you can use the dcli utility to set up SSH user-equivalence to a cell or group of cells.Note: dcli is a Python script that is available on Exadata. You can copy it to your designated central management console and execute it from there. The dcli utility requires Python version 2.3 or later. dcli is discussed further in the lesson entitled Monitoring and Maintaining


g gDatabase Machine.

Disk Storage Entities and Relationships

Exadata Cell CellCLI> CREATE GRIDDISK ...

LUN CELLDISK ASM diskDisk GRIDDISK

DataFirst two

LUNs only

CellDisk

GridDisk

OR Visible to ASM

System Area

D67016GC20 Exadata Workshop Part1

Documents

Transcript of D67016GC20 Exadata Workshop Part1