Operational Impact of Deploying an Oracle Engineered ... · an Oracle Engineered System (Exadata)...

An Oracle White Paper October 2012

Operational Impact of Deploying an Oracle Engineered System (Exadata)

Oracle White Paper—Operational Impact of Deploying an Oracle Engineered System (Exadata)

Disclaimer

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


Introduction ..................................................................................................... 1

Objectives ....................................................................................................... 2

Audience (Who Can Benefit?) ........................................................................ 2

Executive Overview ......................................................................................... 3

Executive Architecture Refresher ................................................................... 5

Technology Components ............................................................................ 6

Another Oracle Database? ......................................................................... 7

It’s the Same . . . ......................................................................................... 7

. . . But Different .......................................................................................... 8

Administration and Management Tools ...................................................... 9

Organizational Impact ................................................................................... 10

Current Organizational Structure .............................................................. 11

DBA Team Capabilities............................................................................. 11

Impact on Resource Requirements .......................................................... 14

Operational Impact Observed where Exadata is Deployed...................... 14

Operational Model for Oracle Exadata Administration .................................. 15

Understanding the RASCI Acronym ......................................................... 16

Operational Model 1: Stay as You Are (Multiple Teams) ......................... 16

Operational Model 2: Database Machine Administration (DBMA) Team . 18

Operational Model 3: Exadata Database Administration (EDBA) Team .. 20

Ongoing Support ....................................................................................... 21

Conclusion .................................................................................................... 23

Appendix A: Staff Training—Oracle University (OU) .................................... 24

Appendix B: Suggested Reading .................................................................. 26

Appendix C: Operational Models for Exadata ............................................... 28


1

Introduction Oracle’s engineered systems are lead topics at Oracle Open World Conferences and have been included in keynotes devoted to Oracle Exadata, Exalogic, SPARC SuperCluster, Big Data, and Exalytics. An engineered system provides optimized hardware and software for specific workloads, resulting in improved performance and throughput. A significant benefit of the engineered system is the enablement of operational change that supports reduced complexity of systems management.

Traditionally, highly skilled and focused administrators have been responsible for managing individual portions of the technology stack. With engineered systems, lifecycle management becomes a more integrated and, ultimately, less complex environment. From a business perspective, the engineered system will result in improved performance and throughput along with reduced lifecycle management costs. As a consequence, this means, more junior technical resources can support the environment.


2

Objectives

What impact does an engineered system have on my day-to-day operations? How will it affect my staff? How do I manage and update these systems? What about training? These questions, and many others, are addressed throughout this document. These questions are relevant when considering a new technology that has both huge performance gains and operational impact to your organization. The objective of this paper is to provide some guidelines and recommendations on how to integrate an Oracle engineered system, specifically Oracle Exadata, into your overall IT operations process. We’ve also documented some lessons learned, and information gleaned, from customer experiences that should help you to have a successful Exadata project implementation.

Of course, every customer faces some challenges that are truly unique to their environment. This paper outlines some Oracle best practices, and practical approaches to the integration of Exadata into your current operations environment. Whether you are just dipping your toe in the water to get firsthand experience, or if you are considering a larger datacenter transformation project to truly leverage the power of the engineered systems platform, you should find this paper useful in helping you to lay the foundation for the integration and adoption of Oracle’s Exadata platform.

Audience (Who Can Benefit?)

Oracle’s engineered system platforms can represent a paradigm shift in the way you manage and maintain your systems throughout your datacenter. This can have an impact on the people, process, and methodologies that you have developed to effectively manage your operations today. This paper is targeted to provide information to today’s datacenter operations team whose primary function is to install, maintain, and update systems throughout the useful lifecycle in the datacenter. It can also be of benefit to those organizations that maintain central help desk operations and Network Operation Centers (NOCs).

Furthermore, this paper can be useful to systems and network architects who need to evaluate and consider technologies and process improvements or changes to address the impact of engineered systems to the organization. Integrating engineered systems into your current operations environment should be part of your deployment planning process and architectural review. Engineered systems can have a profound effect on improving your overall datacenter operations, but a thoughtful and planned integration will be critical for you to realize the performance and process improvements that engineered systems can bring.


3

Executive Overview

The Oracle Exadata Database Machine (Exadata) is a paradigm shift in database processing—one which is becoming more widely adopted. Drivers for moving to Exadata are improved performance and capacity, reduced cost of the storage, reduced server footprint, simplified high performance networking, and reduced Oracle software licensing. In addition, greatly improved business agility is a consequence of significantly reducing build, assemble, and deployment time. If multiple existing databases are being consolidated onto Exadata, administration and management costs are also positively impacted.

An IDC research paper states the following:

“Overall drivers for moving to integrated system functionality (e.g., Exadata) include reducing IT time associated with onsite system integration and maintenance, faster response to changing business conditions, ability to add capacity as needed, IT flexibility in deployment, and lower IT costs.”1

Like any new technology, Exadata requires planning for organizational, technical and environmental considerations. In choosing the applicable best practices for Exadata, each customer will need to consider their existing organizational procedures to determine the proper implementation strategy. This document provides a high level guide to the ongoing management including organizational impact of an Exadata solution.

The major conclusions are

• Oracle Exadata Is the Same, but Different. It is built from x86-based software and hardware components and runs standard Oracle 11gR2 database with RAC and ASM. However,

o The Exadata Storage Servers and Exadata Storage Server Software are game changing by enabling database query and analysis optimization to occur in storage, thus simplifying database tuning and improving performance.

o Components are designed, engineered and supported to work together in a number of pre-defined configurations. The architecture is actually simpler.

o Exadata is pre-integrated and requires less operational intervention on the part of administrative support teams.

1 Bozman, J., Eastwood, M., Scaramella J, Yezhkova N., “Real-time IDC Research opinion,” IDC LINK October 17, 2011, page 15.


4

• The Support Organization Need Not Change. Your existing administration team structure will be quite capable of managing Exadata. However, depending on your current situation and plans, you may want to move toward either

o An Exadata DBA (EDBA) team that owns all aspects of your Exadata systems, but coordinates experts from the existing specialist teams in carrying out non-database tasks.

o A single Database Machine Administration (DBMA) team that manages all aspects of your Exadata systems. This team should be centered on the current DBA team.

• Exadata Is Easier to Manage. Exadata is pre-defined, pre-integrated, and delivered by a single vendor. As such, it requires fewer tasks, less administration effort and far simpler tooling than large Oracle databases on traditional and more complex platforms. Oracle will provide periodic “system” updates that will allow you to update the entire Exadata system with pretested upgrade components. This makes Exadata inherently easier to manage than individual components from multiple suppliers.

• Recommended Tools. Oracle Enterprise Manager 12c (EM) is Oracle’s strategic systems management toolset for managing all Oracle systems and it is now fully Exadata-aware. Although we do not review these tools in detail within this paper, you should consider these management packs when deploying Exadata.

o The core EM infrastructure for Exadata is provided free of charge. Where necessary it integrates with lower-level tools such as ILOM, ASR, Ops Center, and Storage Server commands, as well as with your existing systems management and help desk tools.

o Optional EM Packs offer more automated and advanced functionality which address diagnostics, tuning and change management.

o Oracle Real Application Testing (RAT) is the ideal tool to consider when migrating an existing database to an Exadata platform. RAT can confirm and predict the performance of databases running on Exadata.

o Exachk is a tool designed to audit various configuration settings within Exadata— Database Servers, Storage Servers and InfiniBand Switches. It verifies key components of Exadata against supported version levels, recommended Oracle RAC settings and Exadata best practices. Exachk is available via MOS Doc Id. 1070954.1

• Training Recommendations. Whichever operational approach is selected, it is highly recommended that the core team responsible for overseeing the operational efficiency of the Exadata platform attends training offered through Oracle University. (Please refer to Appendix A for additional role-based training.)


5

Executive Architecture Refresher

This paper assumes that the reader is familiar with what Exadata is, why it is growing in popularity, and how it works. However, a quick review of the architecture may be useful as we will be referring to the major components throughout the document.

Exadata is not an appliance since, once it is plugged in, it needs looking after. Exadata is an engineered system that is hardware and software engineered to work together, and it does need an appropriate level of administration. However, Exadata can also be used as a database consolidation platform, and this typically results in reduced administrative effort compared with traditional platforms.

Figure 1. Exadata Database Machine (X3-2) high-level architecture

The overall architecture is summarized above, showing that it is clearly a combination of database servers, storage, and networking—designed, built, and supported together. There are two formats available currently with the only differences being the database server configurations.

• X3-2. 8 Database servers, each with two 8 core Intel processors. Total of 128 cores.

• X3-8. 2 Database servers, each with eight 10 core Intel processors. Total of 160 cores.

The X3-2 can be bought as an eighth, quarter, half, or full machine (or rack) and the X3-8 only as a full machine. For further scale up, up to 8 racks can be connected together. Storage expansion racks can be added (for example, to accommodate backups).


6

Technology Components

From a systems administration perspective, Exadata can be broken down into components as follows:

• Database Server Node. An x86-based server that runs the Database Management Software and Oracle Grid Infrastructure. The OS is most often Oracle Linux but, alternatively, can be Solaris. Each node contains Intel Xeon processors, Memory, Dual-Port QDR InfiniBand Host Channel Adapter, HBA with Battery Backed Write Cache, internal drives and dual hot swappable power supplies. There are 8 Database Server nodes per X3-2 Exadata Full Rack and 2 per X3-8 Full Rack. The following software runs on a Database Server node.

o Database Service. A logical database object which typically maps to a fixed group of users and database objects. It represents a workload which users and applications connect to rather than to a specific DB instance (or SID). One or more DB schemas with common users, attributes, and service levels can run on one or more compute nodes.

o Instance. Oracle memory, processes, and related structures and caches.

o Database. Physical customer data defined as one or more tables and stored physically on one or more disks across one or more storage cells. Also includes Data files, Control Files, Online Redo Log Files, Archived Redo Log Files, Parameter Files, Alert and Trace Log Files, and Backup Files.

o Oracle DBMS. Oracle software that manages the data. Consists of Oracle binaries (the “Oracle home”), processes, and configuration files. One or more per Exadata machine.

o Oracle Grid Infrastructure. Oracle software which ensures continuous access to the same data through multiple compute nodes. Includes RAC, voting disks, and file system.

o Oracle Automatic Storage Manager (ASM). Oracle software that manages how data is placed on disk. ASM has its own DB instance. ASM can be thought of as part of the Grid Infrastructure, although there are some separate management tasks that apply to it.

• Oracle Exadata Storage Server. A server that runs Oracle Enterprise Linux and the Oracle Exadata software. Each cell contains two x64 Intel Xeon processors, memory, dual port InfiniBand connectivity, management interface for remote access and dual-redundant hot-swappable power supplies. Storage Servers are the most highly controlled of all Exadata components in the sense that configuration files cannot be changed and no software can be installed on them. There are 14 Storage Servers per Exadata Full Rack.

o Disks. 12 SAS (high capacity) or SCSI (high performance) disks connected to a storage controller with battery-backed cache

o Flash Cache. 4 x 400 Gb Flash cards

• InfiniBand Network Switch. This is the network used for all internal communications and data shipping. Based on quad data rate (QDR) InfiniBand technology. The InfiniBand switch consists of software and hardware. There are three per full rack.


7

• Oracle Integrated Lights Out Manager (ILOM). ILOM is system management firmware that is pre-installed on all of Oracle’s x86-based servers. Oracle ILOM enables you to actively manage and monitor hardware and firmware components installed. Includes browser-based and command-line user interfaces, as well as SNMP and IPMI interfaces. The ILOM is independent of the operating system.

• Cisco Network Switch. Standard administrative 48-port Cisco 4948 Gigabit Ethernet switch allowing customers to connect their existing management network to the Exadata machine. The Cisco switch consists of software and hardware. This switch is optional and can be swapped by the customer for a preferred management switch (however such a switch would not be supported by Oracle).

• KVM and PDU. Keyboard, video screen, and mouse for direct management of one or more machines. Power Distribution Unit, which includes rack-level power metering capability with the ability to set threshold and alarm levels preventing overloaded circuits by providing a visible warning when the current power demand is close to the maximum. These are also optional.

Another Oracle Database?

Compared to traditional platforms, Exadata is the same but different. It is the differences that not only explain the astonishing performance and capacity increases, but also why the amount of administration required is less than for large Oracle database systems on traditional hardware platforms.

It’s the Same . . .

DBAs will be pleased to know that Oracle on Exadata is exactly the same database code version as Oracle on any other platform. Applications already running Oracle 11gR2 should require no changes to run against an Exadata database. However, depending on your experience, some elements of the architecture may be new to you. The standard elements include

• Oracle 11gR2 Enterprise Edition Database software with ASM.

o You may include any other database options, such as RAC, Partitioning, Advanced Compression, OLAP, Advanced Analytics, Advanced Security, Database Vault, Spatial Data and other Oracle 11gR2 Database features and options.

• Oracle Linux

• Oracle (Sun) x86 servers

• Cisco Management Network Switch

All of these components are installed, configured, and managed almost exactly the same as they would be on non-Exadata platforms. There are a few nuances, which will be noted later on.


8

If your databases are not yet on Oracle 11gR2, they will need to be upgraded either before or during the migration to Exadata. Oracle Exadata supports single and multi-instance database environments. Adequate planning is required since Exadata can be used for both single application migration and database consolidation. Prior versions of Oracle DBMS using file systems will be migrated to use ASM and the Oracle grid infrastructure.

. . . But Different

In general, there are two major differences (if you are already on 11gR2 using RAC and ASM).

1. Pre-engineered. All the components in the machine are pre-designed, pre-tested, pre-certified, and pre-installed. This has the following, important, implications;

• Compatibility. You do not have to work out and test which versions of which hardware and software are compatible—you simply get this information from My Oracle Support notes (such as MOS 888828.1). This saves time and reduces risk across many tasks.

• Performance Tuning. Customers will experience significant performance improvements without additional Exadata performance tuning. Exadata is pre-integrated and components are designed to scale and work together so you should expect minimal tuning.

• Patching Strategy. Oracle provides patches for all of the components within the Exadata database machine with the exception of the Ethernet switch (Cisco). Please refer to MOS note 888828.1 for the current available patches. The patches fit into the following categories:

o Database bundle patches

o Storage cell patches (which include a DB node component)

o Patches for the other components (KVM, PDU, and IB switches)

o Patching should be done on a regular basis with bundle patches being applied either quarterly or semi-annually and storage cells being patched semi-annually or annually. The other components can be patched either annually or as required.

• Data Guard Logical Standby. A Data Guard Logical Standby database should be considered for near-zero downtime database upgrades. When doing major upgrades (for example, 11gR2 to 12c), the only way to upgrade and fully test the new environment under realistic user loads is to utilize this method. Further information on Oracle Data Guard can be found in Appendix B.

2. New Components

• Oracle Exadata Storage Server. This runs the Exadata Storage Server Software on Oracle Enterprise Linux. The Software is the “secret sauce” which, in conjunction with the DBMS, handles cell offload processing. The major example of this is the Smart Scan (Query Offload), but it also optimizes other tasks such as backups, encryption, HCC, and


9

others. These storage servers should be considered closed systems as nothing can be (or needs to be) installed on them and all management tasks have to be carried out through the command line interface. There are 12 disks per storage server and they are largely self-managing. Additional disks cannot be added to a cell. Cells should be monitored and, although not normally required, some additional database tuning is possible. Patching will be required.

• InfiniBand Network. Used for all communications and data transfers within the Exadata machine, and requires monitoring but minimal administration and patching effort.

Administration and Management Tools

Oracle's primary administration and management toolset is Oracle Enterprise Manager (EM). In addition to providing systems management for Exadata, it also provides systems management capabilities for all Oracle applications, middleware, database, server, and storage across traditional and engineered systems platforms.

The core EM infrastructure and the Exadata plug-ins are free of charge.

EM 12c should be considered almost mandatory, although both EM 11g and EM 10g have Exadata capabilities.

EM 12c provides a uniquely powerful “machine-wide” view of Exadata operations that graphically combines hardware, software and network information. Many administration tasks are possible with just a couple of EM mouse clicks, significantly reducing the overall management effort required.

For further optimized and automated management of many database tasks, the following EM packs are strongly recommended (both for Exadata and all other Oracle environments)

• Diagnostics and Tuning

• Lifecycle Management, which addresses

o Configuration Management

o Provisioning

o Patching

o Schema Control

• Real Application Testing


10

Although not required, Oracle Enterprise Manager Ops Center (OC) can be used for additional Exadata hardware firmware and ILOM monitoring. If you have an existing Oracle (Sun) hardware/Solaris infrastructure it makes sense to ensure that Exadata is made known to OC. Use of OC in no way removes the need for EM 12c. (For additional information on OEM Ops Center, please refer to Appendix B.)

Organizational Impact

Exadata is a step forward in the people and process management of large database systems, and as such it is potentially game changing technology within your environment. Exadata simplifies the architecture of Oracle database systems and also simplifies the administration effort required. In deploying Exadata, prospective and existing customers should consider organizational changes to their existing support model to support Exadata more effectively.

The database on Exadata is mostly the same as it would be on any platform and although most of the components are industry standard items, they have been engineered to work together. There are also a couple of new components to be managed and tuned.

IT organizations have, in many cases, become “technology integrators”–teams of experts centered on the various technology layers. For many years, this has been a best practice in order to support the in-house design, procurement, assembly, test, and operation of highly complex and unique solutions.

As an engineered system, Exadata is a predefined system that speeds time to deployment and business solution implementation. Since Oracle designs, pre-integrates, tests, and upgrades Exadata as a single engineered system, customers have less technical support work to do which allows you to focus on how to best use the product to address business initiatives. The majority of work required is still standard DBA work.

In Oracle IT shops, the introduction and support of Exadata should require minimal training and management overhead while delivering maximum agility. Each organization has a different history and different priorities, so there is really not a single correct answer that will be appropriate for all customers.

In all cases, the use of a single centralized management framework by everyone involved will make the job simpler and easier. Oracle Enterprise Manager 12c with Exadata plug-ins is a recommended example.

Note: Implementation and management of Exadata can be done within your existing specialized teams and any changes you

decide upon can be a gradual evolution.


11

Current Organizational Structure

Since IT operations and support structures within many large organizations have served as in-house technology integrators, the complexity of the environments that they are required to support often includes a variety of tools and techniques for managing them. The diversity of experts needed has created an opportunity for operational inefficiencies.

Teams are typically aligned to specific applications (such as, email, financials, intranet) or to individual technology layers. There are specific teams to manage the physical servers, networks, disk storage, operating systems (perhaps specific teams for Windows and UNIX), security, database (DB2, Oracle, SQL Server), middleware, and applications. In addition, security and testing teams may be involved.

These teams have very deep expertise in their specific areas, sometimes having little involvement in other technologies. Although administration processes have been developed which normally enable these siloed teams to operate effectively, most organizations have experienced situations in which this operating model got in the way of the level of agility and responsiveness required by the business.

Such an approach can provide all the skills necessary to manage Exadata. However, this may not be the best approach since Exadata is a pre-configured system in which all components are designed to be tuned, monitored, and managed together in more efficient ways which may be slightly different to their traditional, non-Exadata, operation.

DBA Team Capabilities

In some cases (and especially in large organizations), the DBA team is highly specialized and does not get involved with the infrastructure on which the database sits.

Other DBA teams have considerably more skills than simply installing and running the database. From patching activities they already have a good knowledge of Linux or other operating systems. If Automated Storage Management (ASM) is used, they know something about storage architectures and techniques. Often, they are aware of basic networking concepts.


12

Figure 2. Administration balance for Exadata compared to traditional architectures

The total amount of administration work required is reduced significantly with the bulk of the savings related to greatly simplified management of the storage, network and OS components.

• Database. Roughly the same amount and type of work as for non-Exadata. Performance tuning is simpler because Exadata is pre-engineered to optimize and balance processing. There are some new concepts such as Smart Scan and Hybrid Columnar Compression to incorporate. Since Smart Scans provide query optimization in storage, traditional tuning techniques (such as indexing the database extensively to prevent full table scans during queries) are no longer required.

• Storage. Compared to traditional large-scale SAN infrastructures, administration is greatly reduced. This is due to the use of ASM, self-managing disks, and flash cache in the pre-configured storage cells. Once the grid disks and the disk groups for ASM are created, they will need minimal attention. Patching will still be required. For Exadata, the ongoing administration tasks are primarily managing the backups for the data stored in the databases. Many organizations also define an information lifecycle management strategy leveraging high- capacity disks in storage servers, the Exadata Storage Expansion Rack, ZFS Storage Appliance, or other media.

• OS. Since Exadata is a single-purpose machine (to run Oracle databases), management of the OS is simpler.

No management is required for the OEL OS on the Exadata Storage Servers other than patching. Nothing can be installed on it, all storage servers are identical in a rack, and the configuration should not be changed.

The OEL or Solaris OS running the Oracle Database Server nodes requires less administration than non-Exadata platforms because there is no need to install any additional programs on it, users do not need to be defined and managed and should be the same.

Organizations are aiming to have fewer people doing more things that are simpler and take less time and overall effort. Exadata

embraces this challenge.


13

• Network. The InfiniBand network is pre-configured and requires just basic monitoring and infrequent patching. It does not affect systems outside the Exadata environment, unless it is connected to other Oracle Engineered systems, such as Exalogic, Exalytics, or SuperCluster. When the machine has been configured, the IB (internal) network will require minimal attention. Network administration is then responsible for the connections to the public network with the associated DNS and IPs.

The Cisco management network also requires minimal management and no (Exadata-related) patching. It can be swapped for a switch from your preferred vendor if necessary.

• Hardware. The KVM and PDU require basic monitoring and minimal management, and in fact are entirely optional.

• Access to Patches. All Oracle patches are downloaded using the normal methods (typically through My Oracle Support (MOS) with a couple of exceptions. The Storage Server patches have a component associated with them (formerly the convenience or minimal pack) that must now be downloaded from the Unbreakable Linux Network (ULN) using a YUM server. YUM is the open-source standard utility for Linux operating systems’ package management. Complete instructions on how to setup a YUM server are included in Storage Server patch readme via references to the appropriate MOS notes. The other situation that might arise would be the use of a YUM server to facilitate Linux OS upgrades on the DB nodes.

As the DBA work is by far the largest proportion of the overall administration effort required, our recommendation is that the

DBA team is best positioned to be the central point of Exadata management.


14

Impact on Resource Requirements

Exadata is simpler to manage than existing complex database platforms. Savings will soon start to accrue because

• For each database system moved to Exadata, the comparative support and management cost will be reduced as the old system is turned off.

• As more databases are consolidated onto Exadata and traditional systems are turned off, mundane DB tasks are reduced, which allows your staff to focus on higher value functions. Staff realignment and reassignments from these older systems will more than offset any corresponding need for additional Exadata support staff—if any.

• When social, political, and governance policies are entrenched in opposition to organizational change, Gartner Key Findings recommends that you, “Create a data center within a data center to invest in integrated systems as a controlled business exercise”. 2

Operational Impact Observed where Exadata is Deployed

• Company A—Global Banking. Reduced tuning and maintenance burden and costs with Exadata’s automated tuning capabilities and found that it allowed the database administration team to reallocate development resources to help developers design their apps, as opposed to simply managing the database.

• Company B—Global Banking Vice president of infrastructure standards and governance states that in their organization the ideal role for an Exadata management lead is a “DMA” database machine administrator. This person is "60 percent Oracle RAC DBA, 20 percent storage administrator, 15 percent system administrator and 5 percent miscellaneous".

• Company C—Financial Services. Director of database services says: “I would recommend Exadata. My old team was doing nothing but firefighting in the old environment. Now they are all freed up to do new things."

• Company D. DBA resources have been reduced in support of a key system from five to two. The remaining DBAs have taken on more responsibility as they now manage the storage through ASM, the Linux OS and patching the entire platform. On occasion they coordinate with the other specialist admin teams.

2 Butler, Andrew, “How Ready Is Your Data Center to Be an Early Adopter of Integrated Systems?” Gartner Report G00218421: 15 November 2011, page 1.


15

• Company E. Consolidation onto Exadata has forced this organization to simplify. They now have one version of OS, database and storage to manage. "We have moved 10 of our large databases from older servers onto Exadata, and the savings in overall admin time are significant, not least of which is because everything is running the same stuff."

• Company F. Four major databases moved to Exadata overall support costs have been reduced by 23 percent.

• Oracle’s Deployment. Exadata has supported a period of exponential growth for Oracle, driven in large part by over 50 acquisitions. All internal application systems are underpinned by Oracle databases. So while the company’s size has more than doubled, the number of support staff has increased by only 10 percent.

Overall, there should be a significant reduction in the number of administrative FTEs needed if you move strategically to Exadata. At Oracle, we reduced from five administrators to two administrators, and DBA utilization increased as they work further down the technology stack. This also means that effort to support storage, OS and network administration teams has been reduced with a further saving of one to two FTEs. So Oracle could reduce five FTEs to only one where traditional systems migrate to Exadata. The end result would be fewer people doing more things that are simpler and faster using standardized and automated tools. As Oracle consolidates more systems onto Exadata, these benefits grow exponentially. By using administration staff more efficiently, unbudgeted overtime is also reduced and senior people are freed up to do higher value and more strategic work.

Operational Model for Oracle Exadata Administration

It is natural to think about and is quite possible to slot Exadata into your current administration structure just like a new server or a new version of Oracle. However, Exadata has many unique optimizations of standard components and one new major component (it’s the same but different). Being an engineered system, Exadata components should be tuned and managed together.

What we are looking for is a management structure that allows the introduction and ongoing support of Exadata with minimum overheads and maximum agility. To support the development of the options presented, examples of RASCI charts are provided.


16

Understanding the RASCI Acronym

For every step of your planning project, you should define the following:

R – Responsible The person responsible for delivering the project and task successfully.

A – Accountable The person with ultimate accountability and authority, and to whom “R” is accountable.

S – Supportive The person or team who are needed to do “the real work.”

C – Consulted The person whose input adds value and whose buy-in is essential for implementation.

I – Informed The person or group who need to be notified of results or actions taken, but don’t need to be involved in the decision-making process. We have identified three operational models.

1. Stay as you are (multiple specialist teams)

2. A Database Machine Administration (DBMA) team

3. An Exadata DBA (EDBA) team

For all options, you can maximize the value of your Exadata investment with

• The use of a standard, centralized administration toolset, such as Oracle Enterprise Manager to facilitate communication and hence encourage agility.

• Appropriate Training on Oracle Database 11gR2, RAC, ASM, EM 12c, and Exadata (see Appendix A – Oracle University).

• Oracle Consulting Services to advise on and assist with all aspects of Exadata readiness (including operational), architecture, implementation, and integration.

• Oracle Advanced Customer Support Services remote and on-site assistance with specific issues such as patching, networking, and advanced hardware support.

• Oracle Platinum Services for high availability support for engineered systems, including remote patching service.

Operational Model 1: Stay as You Are (Multiple Teams)

In some scenarios this approach will be the most appropriate, for example

• There are very well defined and well organized processes that enable effective administration of all aspects of the whole Oracle database infrastructure

• Business stakeholders are happy with the responsiveness of this operating model


17

• Exadata is being adopted tactically for a small number of specific projects. The majority of the Oracle environment is likely to remain based on traditional platforms

• The DBAs are very highly specialized and have no experience outside of pure DBA tasks.

• The current organization structure is very rigid and any changes would be extremely difficult.

However, this approach carries higher costs and provides limited agility.

RASCI CHART: STAY AS YOU ARE (MULTIPLE TEAMS)

TASK DBA OS SYSADMIN

NETWORK STORAGE CIO SECURITY

Day-to-day Operation R R R R A C

Testing R A C/I

Monitoring R R R R A C/I

Configuration Management R R R R A C/I

Tuning R R R R A C/I

Patching

Storage Cells ? ? A C/I

InfiniBand C R A C/I

Cisco Public Network C R A C/I

Db Compute Nodes OS C R A C/I

Database R A C/I

Backing Up Data R A C/I

Upgrading SW R R R R A C/I

Replacing HW R R R A C/I

Metering and Charging ? ? ? ? A C/I

Legend: R=Responsible, A=Accountable, S=Supportive, C=Consulted, I=Informed

IMPACT ON STAY AS YOU ARE TEAM

ACTIVITY IMPACT DESCRIPTION

TRAINING COSTS High All teams (DBA, OS, Network, Storage, and Security) will need some Exadata-related training, covering architecture, database, storage cells, and networking.

MANAGEMENT OVERHEADS

High Many tasks (such as performance tuning and patching) require that multiple Exadata components are addressed together. Therefore resources from multiple teams (with differing priorities) will frequently need to be scheduled and coordinated—requiring significant management overheads.

AGILITY Low The need to involve multiple teams may also hinder the speed with which problems can be resolved, new databases migrated onto Exadata, patches applied and, ultimately, business projects implemented. A common example is where access to the UNIX root passwords is needed. Here we could see multiple teams severely hampering efficient database management.


18

Operational Model 2: Database Machine Administration (DBMA) Team

This optimal approach enables the relative simplicity of Exadata to be reflected in a simpler, less costly, and more agile support organization. As Oracle pre-integrates and pre-tests all components together, the support team is able to focus more on business initiatives and outcomes, and less on technical administration.

As most of the administrative work is database-centric, this team should be based on the existing Oracle DBA team, with possibly a small number of additional network and storage staff so that it has the skills needed for all administration tasks for all technology components within Exadata. The DBMA team will own Exadata from a management perspective. Specialized expert assistance may be required from time to time, especially while experience with Exadata is being developed and the DBMA team is evolving.

If a single DBMA team is not possible, it may be seen as a target to move towards. It is a suitable approach where

• Exadata is seen as the strategic database platform and, over time, a considerable portion of the Oracle footprint will be consolidated onto it.

• The Oracle database infrastructure is already managed predominantly by the DBA team.

• There is a desire to make the database administration function more agile.

It is possible that the creation of this new team and their ownership of the new technology may cause some disharmony among existing technology support teams. This will have to be considered and managed on a case-by-case basis. Although some restructuring of the support organization will be required to create such a DMA team, from an Exadata perspective, there are major advantages to consider.


19

RASCI CHART: DBMA TEAM



Day-to-day Operation R A C

Testing R A C/I

Monitoring R A C/I

Configuration Management R C A C/I

Tuning R C A C/I

Patching

Storage Cells R A C/I

InfiniBand R C/I A C/I

Cisco Public Network R C A C/I

Db Compute Nodes OS R A C/I

Database R A C/I

Backing Up Data R A C/I

Upgrading SW R A C/I

Replacing HW R A C/I

Metering and Charging R A C/I


IMPACT ON THE DBMA TEAM


TRAINING COSTS Med Only the DBMA team will need training. This will cover Exadata architecture, Linux/Solaris, Storage Servers and InfiniBand networking.

New staff from other teams may need some training on Oracle.


Low The team will be self sufficient and require minimal coordination with other groups.

AGILITY High Planning and execution of all Exadata-related tasks can be carried out with no delays

The DBMA team could be created in one of two ways.

• Cross-Train Team Members. Train some or all of the DBAs to bring the team up to speed in other areas (Linux or Solaris, Networks and Exadata storage cells). Storage knowledge is the least significant as Exadata Storage Servers and Exadata Storage Server Software is new technology to anyone and is largely self-managing. Oracle University currently offers an Exadata Database Machine Administration Workshop and a Monitor Exadata Database Machine seminar. For those pursuing certification, an Oracle Certified Expert examination is available. Please refer to Appendix A for Exadata training.

• Add Team Members and Cross-Train. Add one or more people with a background in Linux/Solaris, networks, and possibly storage systems to the DBA team, then do cross-training.


20

Operational Model 3: Exadata Database Administration (EDBA) Team

In many cases, this is likely to be the most practical operating model, and offers the best of both worlds, as it enables the Exadata machines to blend in with existing systems from a management perspective.

Here we have an Exadata DBA team that is responsible for managing all aspects of Exadata. It should be based on the existing Oracle DBA team, but with Exadata capabilities so that it has the skills needed for basic Exadata administration tasks, but a clear mandate to call on experts in the existing specialized teams to carry out most of the non-database work.

Over time, this EDBA team is likely to become more self-sufficient and closer to the DBMA model described above.

This is a suitable approach where

• There is a significant existing Oracle database environment.

• There is no plan to consolidate the majority of Oracle databases onto Exadata. Although a number of critical systems will use it, there will still be widespread use of Oracle on traditional platforms.

• There is a well-defined structure in which multiple teams already provide effective administration of all aspects of the whole Oracle database infrastructure.

• Business stakeholders are happy with the responsiveness of this operating model.

• The EDBA team has direct and timely access to specifically named personnel in the existing support teams.

RASCI CHART: EDBA TEAM



Day-to-day Operation R S S S A C

Testing R S S A C/I

Monitoring R A C/I

Configuration Management R A C/I

Tuning R A C/I

Patching

Storage Cells ? ? A C/I

InfiniBand R S A C/I

Cisco Public Network R S A C/I

Db Compute Nodes OS R S A C/I

Database R A C/I

Backing Up Data R S A C/I

Upgrading SW R S S A C/I

Replacing HW R S S S A C/I

Metering and Charging ? ? ? A C/I



21

IMPACT ON THE EDBA TEAM


TRAINING COSTS Med The EDBA team will need some training, covering Exadata architecture and basic Linux / Solaris, storage and networking.

The named person in each of other teams should receive basic Oracle and Exadata training.


Med The team will be self sufficient on a day-to-day basis but coordination with other groups will be required for many tasks.

AGILITY Med Use of multiple teams with different priorities will inevitably involve some delays. However, these can be minimized by

• Pre-established contact points and procedures

• Use of Oracle Enterprise Manager across all teams

Please refer to Appendix C: “Operational Models for Exadata” for more information.

Ongoing Support

Utilizing Oracle or qualified third parties to assist with the administration of your Exadata environment is another option to be evaluated. Benefits of this option may include the following:

• Accelerated installation time

• Leveraging Oracle best practices

• Reduced risk with standard builds

• Pre-optimized configurations for intended use

• Establishing operational baselines and best practices

• Supporting existing staff in operational changes

• Knowledge transfer

There is a broad community of Oracle partners and Oracle, itself, that offers such services in a variety of models. The choice organizations make is generally due to their comfort with the skills that the partner demonstrates and the alignment of their offerings to identified needs. For example, to support an Exadata environment, Oracle’s Advanced Customer Support Services (ACS) provides a number of options based on fixed scope, time and materials, and advanced monitoring and resolution. These options include

• System and software installation and configuration

o Gather build-sheet data from the customer or Systems Integrator

o Research Oracle’s configuration best-practices based on the intended use and known reference architecture

o Update build-sheet to incorporate research results

o Install required software products and recommended patches


22

o Optimize configuration based on final build-sheet

o Run tests, verify configuration, and review with customer

o Conduct a review findings and recommendations

• Pre-production readiness and configuration review

• Advanced Monitoring and Resolution

o 24/7 health and performance monitoring from the database and operating system level down through the physical compute nodes, InfiniBand switches, and storage arrays

• Troubleshooting and resolution of alerts and incidents across the entire Exadata stack

• Proactive Oracle Database 11g support, including review of log files, provisioning, and cloning

o Reactive patch reviews, recommendations and change management, production, and performance diagnostics review

• Patch review and installation

Figure 3. Oracle Fixed Scope Services

Fixed scope services may be discrete or bundled into a support pack to supplement a Solution Support Center or Advanced Monitoring and Resolution.

Production readiness services help to ensure that you are operationally ready to support your system in production. These services include hardware and software installation and configuration services, as well as readiness reviews. Services are focused on enabling you to accelerate your ROI and reducing downstream risk by streamlining your deployment and ensuring Oracle best practices are implemented up front.

Production Optimization services are designed to help you achieve maximum availability, reliability, and performance. Many of these post-productions services are available on a recurring basis. For example, performance reviews and patching services. These services focus on the


23

continual improvement of mission critical production systems to maximize system availability and to optimize performance.

For additional information on Oracle Advanced Services and Oracle Consulting Services, please see Appendix B: Suggested Reading.

Conclusion

The Oracle Exadata Database Machine will have very positive productivity implications for IT and also help deliver better service levels to the business. Some roles will change within IT, including the following:

• Enterprise architects

• Management DBAs

• Development DBAs

• Storage managers

• Systems and network administrators

In each of these roles, there is a great opportunity to spend more time on delivery of new solutions and less time on day-to-day maintenance. The end result of understanding the change and operational management required when deploying Exadata will be better operations, management, minimized cost of ownership, improved team satisfaction, and an increased likelihood of success for everyone involved.

The deployment of one of Oracle’s engineered systems will create the need to rethink how you will approach the management and maintenance of an engineered system within your environment and may be different than your current systems management approach. It also provides you an opportunity to better align your datacenter operations resources around a growing industry trend in consolidation, virtualization, and cloud computing. By introducing an Oracle engineered system to your environment, you’ll take the steps to better aligning your operations resources toward the new datacenter paradigm of cloud computing.


24

Appendix A: Staff Training—Oracle University (OU)

There are a number of pre-defined learning paths available for Exadata, depending on your current knowledge level and experience. The chart below shows the learning path developed by Role within the IT Operations organization. The roles described in the chart are:

• Database Administrator for Exadata

• Linux System Administrator for Exadata

• Oracle Solaris System Administration for Exadata

• Oracle Solaris Network Administration for Exadata

• Oracle Solaris Storage Administration for Exadata

For details on the learning paths and the courses, please visit oracle.com/education/exadata

Figure 4. Oracle University learning paths by role


25

For certification details, please see oracle.com/certification. Use the Finder: select “Database Administrator”, select “Systems”.

You will see the webpage with details for Oracle Certified Expert, Oracle Exadata Database Machine Administration.


26

Appendix B: Suggested Reading

The Oracle public Website has an extensive amount of information on our engineered systems platforms. A simple search provides a large amount of material related to this subject.

However, we might suggest the following topics:

• Enterprise Manager 12c. You’ll want to checkout Oracle Enterprise Manager 12c to help you manage the Exadata platform:

• Exadata Blogs. You should find the blogs by Rene Kundersma helpful:

www.oracle.com/us/products/enterprise-manager/index.html

• Best Practices for the Oracle Exadata Database Machine.

https://blogs.oracle.com/XPSONHA/

www.oracle.com/us/solutions/exadata-maa-best-practices-155385.html

Oracle Exadata Start-Up Pack. Bundled services provided by Oracle Consulting and Oracle Advanced Customer Support www.oracle.com/us/support/library/exadata-start-up-pack-ds-192654.pdf

• Oracle Expert Services for Oracle Engineered Systems www.oracle.com/us/products/consulting/resource-library/engineered-systems-ds-1367830.pdfw

Remote Fault Monitoring and Update Services (Oracle Platinum Services www.oracle.com/us/support/premier/engineered-systems-solutions/platinum-services/overview/index.html

• Oracle Real Application Testing. www.oracle.com/au/products/database/options/real-application-testing/index.html

• On-Demand Webcast: Managing Oracle Exadata with Oracle Enterprise Manager.

• Oracle Data Guard: Disaster Recovery for Oracle Exadata Database Machine.

https://www.techwebonlineevents.com/ars/eventregistration.do?mode=eventreg&F=1003109&K=CAA1AC

www.oracle.com/technetwork/database/features/availability/maa-wp-dr-dbm-130065.pdf

• Exachk: Healthcheck for Exadata. https://blogs.oracle.com/XPSONHA/entry/exachk_healtcheck_for_exadata

http://www.oracle.com/us/products/enterprise-manager/index.html�

https://blogs.oracle.com/XPSONHA/�

http://www.oracle.com/us/solutions/exadata-maa-best-practices-155385.html�

http://www.oracle.com/us/support/library/exadata-start-up-pack-ds-192654.pdf�

http://www.oracle.com/us/products/consulting/resource-library/engineered-systems-ds-1367830.pdfw�

http://www.oracle.com/us/products/consulting/resource-library/engineered-systems-ds-1367830.pdfw�

http://www.oracle.com/us/support/premier/engineered-systems-solutions/platinum-services/overview/index.html�

http://www.oracle.com/us/support/premier/engineered-systems-solutions/platinum-services/overview/index.html�

https://www.techwebonlineevents.com/ars/eventregistration.do?mode=eventreg&F=1003109&K=CAA1AC�

https://www.techwebonlineevents.com/ars/eventregistration.do?mode=eventreg&F=1003109&K=CAA1AC�

http://www.oracle.com/technetwork/database/features/availability/maa-wp-dr-dbm-130065.pdf�

https://blogs.oracle.com/XPSONHA/entry/exachk_healtcheck_for_exadata�


27

The following links are provided as additional resources for Oracle Database High Availability Best Practices 11g Release 2 (11.2):

•

•

Understand Availability and Performance SLAs

•

Implement a High Availability Environment

•

Validate Your Performance and Availability SLAs

•

Set up and Use Security Best Practices

•

Establish Change Control Procedures

•

Provide a Plan to Test and Upgrade for Recommended Patches and Software

•

Use Proper Testing and Patching Practices

•

Execute Data Guard Role Transitions

•

Establish Escalation Management Procedures

•

Configure Monitoring and Service Request Infrastructure for High Availability

Check the Latest MAA Best Practices

http://docs.oracle.com/cd/E11882_01/server.112/e10803/operational_bps.htm#CHDBEJEG�

http://docs.oracle.com/cd/E11882_01/server.112/e10803/operational_bps.htm#CHDGJACA�

http://docs.oracle.com/cd/E11882_01/server.112/e10803/operational_bps.htm#CHDCJJEH�

http://docs.oracle.com/cd/E11882_01/server.112/e10803/operational_bps.htm#CHDFICJB�

http://docs.oracle.com/cd/E11882_01/server.112/e10803/operational_bps.htm#CHDBCGHG�

http://docs.oracle.com/cd/E11882_01/server.112/e10803/operational_bps.htm#CHDFDECG�

http://docs.oracle.com/cd/E11882_01/server.112/e10803/operational_bps.htm#CHDEBBID�

http://docs.oracle.com/cd/E11882_01/server.112/e10803/operational_bps.htm#CHDGGIAI�

http://docs.oracle.com/cd/E11882_01/server.112/e10803/operational_bps.htm#BACJBCBC�

http://docs.oracle.com/cd/E11882_01/server.112/e10803/operational_bps.htm#CHDEDHEG�

http://docs.oracle.com/cd/E11882_01/server.112/e10803/operational_bps.htm#CHDFGCJE�


28

Appendix C: Operational Models for Exadata

Operational Impact of Deploying an Oracle Engineered System (Exadata) October 2012 Contributing Authors: Jules Lane Vince Pulice David Rabo Scott Hamann Bob Skolnik Pete Werner Bob Stackowiak Heather Hughes Oracle Corporation World Headquarters 500 Oracle Parkway Redwood Shores, CA 94065 U.S.A.

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

1012

Operational Impact of Deploying an Oracle Engineered ... · an Oracle Engineered System (Exadata)...

Documents

Transcript of Operational Impact of Deploying an Oracle Engineered ... · an Oracle Engineered System (Exadata)...