Performance Tuning Oracle's BI Applications

Contact Us510.818.9480 | www.kpipartners.com

© KPI Partners Inc.

Start Here

Jeff McQuigg | Senior BI Architect

April 24, 2013

Performance Tuning Oracle’s BI Applications

2

Agenda

3

① Introducing The Performance Layer

② Building The Performance Layer

③ Mapping Into Oracle BI

④ Implementation Considerations

⑤ Q&A

ScheduleAgenda


4

Introducing ThePerformance Layer

5

Targeted At Organizations

Who Have:

Large Data Volumes

Custom Tables & Data Sets

Aggressive Performance Targets

Questionable Design Extensions

Slow Hardware

Today’s SessionIntroducing The Performance Layer


6

These performance concepts are applicable to any BI system

Custom Built Systems

Custom Stars in OBIA

SQL Server

Today’s SessionIntroducing The Performance Layer


7

Wider tables slow you down

Dashboards only need a few tables

Smaller is faster

For High Performance… Travel LightIntroducing The Performance Layer


vs.

8

Eliminate conflicting priorities

Singular focus on performance

Peak performance starts with design

Optimize Design For PerformanceIntroducing The Performance Layer


vs.

9

Great performance requires perfect design for how it is used

Mandate a top-down approach to tuning your BI application

Specialized design for specialized usage



vs.

10

Pre-built logic

Clean star models

Reduced data weight

Tables which match usage by Oracle BI



vs.

Top-Down design yields…

11Performance Tuning Oracle’s BI Applications

Introducing The Performance Layer >> Oracle’s View On Data Warehouse Architecture

12

Keep the BI Apps model mostly as-is & add a performance layer…

① Source data from BI Apps tables

② Bring only what you need

③ Denormalizations & pre-calculations

④ Use all database performance tools

*Optimize To Usage*

The Performance Layer

Pe

rfor

man

ceM

ini-

Fri

dge

BI A

pps

Da

ta F

ridge

ETL

Introducing The Performance Layer



① The Performance Layer is industry standard architecture

② Design is driven only by report performance improvement

③ Travel light

④ No need to alter BI Apps or DW

Introducing The Performance Layer - Takeaways

14

Building ThePerformance Layer

15

① Start with priority areas (select a Fact table)

② Identify the use cases (reports w/ prompts & data security)

③ Analyze resulting physical SQL

④ Try to tune the BI Apps model first!

12 Step Performance PlanBuilding The Performance Layer


16

⑤ Prototype a new data model to match those needs

⑥ Adjust SQL & benchmark (SQL handcrafting needed)

⑦ Map into Oracle BI & test (Unit & Regression)

⑧ Benchmark the Oracle BI report using prototyped tables



17

9. Build the tables using INFA & DAC - Complete Oracle BI RPD mapping

10. Formal Regression Test

11. Deploy

12. Enjoy praise from users



18

Reduce I/O with extreme prejudice

• Tune the BI Apps model first! It may work for you with low effort

• Employ techniques to eliminate I/O wherever possible

• Partition Elimination, Compression, Indexes, Aggregates, Star Transformations

• Let the Performance Layer do the work, not the report query

• Follow the KISS principle: Use a simple and clean Star. No Snowflakes!

• Ensure OBI is mapped properly and uses correct tables with perfect SQL

• Favor a general approach as opposed to a case-by-case approach

• A rising tide lifts all boats

Tuning PhilosophyBuilding The Performance Layer


19

There are 4 kinds of tables in the Performance Layer:1. Skinny Dimension and Fact tables

2. New Dimension tables

3. Mini-Dimension tables

4. Fact Aggregate tables

Built directly from the base BI Apps or DW tables

Goal: use these tables in as many reports as possible (80/20 rule)

Guiding principles and performance influences:1. Application use cases drive the layer’s design

2. Use minimal data for the job at hand

3. Aggregate Fact data when needed

4. Denormalize dimensions to eliminate extra joins

5. Pre-Build calculations to eliminate extra joins

6. Pre-Split data sets based on logical usage

Performance Layer

BI Apps or DW Perf. Layer

Building The Performance Layer


20

Partitioning & IndexingBuilding The Performance Layer


① Single column, local bitmap indexes on all Fact table FKs (_WIDs) and filter fields (DELETE_FLG)

② Single column bitmap indexes on all dimensional fields used in any sort of prompt or report filter

③ Special composite B-Tree indexes to assist Snowflaked areas

④ Composite B-Tree indexes on large dimensions for join backs (for list reports)

Query Indexing (4 Types)

For all Fact tables of a reasonable size (e.g., > 5M rows)

Usually partition on Month (Range or Interval)

The Database can easily eliminate the majority of the table

Allows for smaller, local indexes

Table Partitioning

Before Beginning:

Tune the OOTB Model

21

Skinny Tables are highly selective versions of the BI Apps or DW

tables• “Horizontal Aggregation” – use only 10 columns vs. 100 from the base table

Both Dimensions and Facts

Very easy to build and use

Goal: Reduce Avg. Row Length to 1/5th - 1/20th original size

Include only the columns you will need for top-down reporting analysis• If you don’t need Customer Address, don’t include it• Ignore Meta Data columns (e.g., INTEGRATION_ID, etc.)

Row sets are identical (1:1) with the base tables• For Dimensions use the same ROW_WIDs - can be used with existing fact tables easily

Skinny TablesBuilding The Performance Layer


22

Build using:1. Create Table as Select (CTAS)2. Insert /*+ APPEND */3. Materialized Views

Compress the table

Use Parallel hints & options

Don’t forget partitions

Enhance the tables with calculation logic

Database is very fast at these operations – expect only a few minutes for 100M rows

Skinny Tables (cont.)

CREATE TABLE WC_ACCT_BUDGET_SF COMPRESS NOLOGGING PARALLEL (DEGREE 8)PARTITION BY RANGE(PERIOD_END_DT_WID) INTERVAL(NUMTOYMINTERVAL(1, 'MONTH'))(PARTITION Part_01 VALUES LESS THAN (20100101)) AS SELECT /*+ PARALLEL(F,8) */

F.PERIOD_END_DT_WID,F.X_PERIOD_END_DT_WID,F.COMPANY_ORG_WID,F.GL_ACCOUNT_WID,F.X_POSTED_TOTAL_AMT,case when

GL_D."GL_ACCOUNT_NUM" = 'S250' then F."X_POSTED_TOTAL_AMT" end as PLAN_CASES,FROM W_ACCT_BUDGET_F F, W_GL_ACCOUNT_D GL_DWHERE F.GL_ACCOUNT_WID = GL_D.ROW_WID;



23

Enhance _SF tables with logic

Identify CASE WHEN statements which require other dimensions

• Potential great benefit if the join can be eliminated

• Don’t over do it – table will get less skinny with each column

Identify any data set splitting from the RPD

• HR Workforce Events table has both Events and Snapshots records but they are always used separately in the RPD

• Usage drives design: Split them out!

• Huge benefit for Event counting metrics (~10% of table)


case when GL_D."GL_ACCOUNT_NUM" = 'S250' then F."X_POSTED_TOTAL_AMT" end as PLAN_CASES,FROM W_ACCT_BUDGET_F F, W_GL_ACCOUNT_D GL_DWHERE F.GL_ACCOUNT_WID = GL_D.ROW_WID;

Create table WC_WRKFC_EVT_EVENTS_SF …… WHERE SNAPSHOT_IND = 0

Create table WC_WRKFC_EVT_MONTH_SNP_SF …… WHERE SNAPSHOT_IND = 1



24


Real Examples I/O Benefit

Sub Ledger (custom)

24X

Workforce Snap 11X

Workforce Events 56X

GL Balance 21X

Acct Budget 32X



Typical to get a 10X to 20X and even 50X I/O benefit in the _SF vs. the base _F in size

Reduced AVG_ROW_LEN COMPRESSION Record Set Splitting All without any aggregation

Skinny Dimensions also have benefits:

1. All I/O is a killer and slows down the entire system2. De-normalize into a Star (eliminate snowflakes & outer joins)3. Real World Ex, #1: Swap out 2 wide dims for 2 skinny dims improved query

time by 6X 4. Real World Ex. #2: One query going from 11s to 4s with one skinny dim5. Real World Ex. #3: GL Account Dimension: 37X I/O benefit

25

Pre-build major pieces of commonly used but complex logic into the Data Model

• Over-relying on the RPD or Reports for logic can harm performance

• Let the ETL for the Performance Layer do the work not the query

Example #1: Large binning and bucketingCASE WHEN FACT.ORDER_AMT BETWEEN 0 and 100 THEN ‘0-100’ ELSE CASE WHEN FACT.ORDER_AMT BETWEEN 101 and 200 THEN ‘101-200’ … END

• Build a new dimension table to hold these values – WC_CUST_ORDER_QTY_BAND_D

Example #2: Date format conversions – dynamically building a new column with a string concatenation statement:

substring(T66755."PER_NAME_MONTH" , 1, 4) , '') + '-' + isnull(right(T66755."PER_NAME_MONTH" , 2) , '')

• Build a new column in the W_DAY_D table & index it

Pre-build Common & Complex CalculationsBuilding The Performance Layer


26

Simply a higher level or levels of a larger dimension

• A combination of several Kimball concepts

• Granularities will be mixed

Make a new table from the large, base dimension

• Contains distinct combinations• Use only commonly used fields• Get cues from dashboard prompts,

column selectors, report filters

Create a new ROW_WID

Compress and index as normal, Parallel if needed for creation

Easy to build and map

Mini-DimensionsCreate table WC_EMPLOYEE_MD COMPRESS asselect

ROWNUM AS ROW_WID, W_ETHNIC_GRP_DESC,

WC_RACE_ETHNIC_DIVRSE_GRP_DESC, W_SEX_MF_CODE,

W_SEX_MF_DESC, WC_NON_EMPLOYEE_VENDOR_NAME

from (

select distinct W_ETHNIC_GRP_DESC,

WC_RACE_ETHNIC_DIVRSE_GRP_DESC, W_SEX_MF_CODE, W_SEX_MF_DESC, WC_NON_EMPLOYEE_VENDOR_NAME

from W_EMPLOYEE_D);This real world example created 5,400 records from a W_EMPLOYEE_D of 9+ Million rows.



27

_MD tables are used in the Performance Layer in two places:

1. Link into Skinny Facts • Use a separate FK in addition to the base _WID

• Fact table has both EMPLOYEE_WID and EMPLOYEE_MD_WID

Thus the _SF can join to all of the following:• New Mini Dimension (_MD) (~1% rows, some columns)

• New Skinny Dimension (_SD) (100% rows, some columns)

• Base BI Apps/DW Dimension (_D) (100% rows, 100% columns)

The OBI RPD can select which one is best for each query

Benefits of linking into the _SF1. The same set of fact rows are selected – no benefit2. Reduced dimension I/O, CPU and buffer space3. Faster join-back on list reports4. Very fast prompts, especially when constrained

Using Mini-Dimensions (1/2)

_D

_SD

_MD

Conceptual Size Differences



28

2. Use them for very high level Fact Aggregates• Build a Fact aggregate at the Mini-Dimension level

Allows greater field inclusion at no expense to aggregation compression ratios• Multiple fields are available - not just one

A good Mini Dimension and Skinny Fact/Aggregate can serve a large % of dashboard queries

MD’s & Fact Aggregates offer extreme performance:1. Real World Ex #1: From time-out after 10 minutes to 4 seconds

2. Real World Ex #2: GL Account MD: 131X I/O benefit

Using Mini-Dimensions (2/2)Building The Performance Layer


29

Aggregates are used when summary reports exist Pre-aggregate the dataset to make it smaller & faster Sometimes they are the only solution Typically a minimum of a 10:1 ratio is used Use all database tools as with any fact table

• Partitioning, Indexing, Star Transformations, Compression

For extreme needs, consider merging facts together• Ex: Monthly Actuals and Budgets

• Be mindful of gaps in datasets and non-conformed dimensions

Advanced implementations use partition management to build only changed data for faster load times

AggregatesBuilding The Performance Layer



① Building the underlying tables is relatively simple

② Take cues from dashboard, report & RPD configuration

③ Any savings in I/O helps the overall system

④ Use all of the available database performance tools

⑤ Tremendous benefits are possible with a few tables

Building The Performance Layer- Takeaways

31

Mapping Into Oracle BI

32

Link as much as possible to allow for the best performance across all scenarios

Performance Layer in OBI - Physical

BI Apps

Performance Layer

Best

Better

Base



33

The 3 Fact tables are mapped like any aggregate

The Skinny Fact (_SF) will have fewer dimensions and fewer metrics mapped to it

• Along the Employee dimension however it is the same as the base _F

OBI will prefer to use the _SF over the _F

• Uses regular aggregate navigation concepts

• Uses the _F when needed as a “backup plan”

Performance Layer in OBI - Facts

_A

_SF

_F



34

Raise the priority group on the base _D to have OBI prefer the _SDAs both LTS grains are identical, OBI needs more info to make a choice

Mapping Skinny Dimensions

_D _SD



35

Create a dummy hierarchy level and map the LTSs for the Mini Dimension and Fact Aggregate to it

The grain of the Mini Dimension is arbitrary

As long as OBI knows it is higher than the other LTSs it will be preferred (Priority groups not needed)

Mapping Dimension Aggregates

_MD

_A




Table MappingThe mapping of tables is straightforward

Link Tables As Much As PossibleLet Oracle BI make the best choice

Mapping Into Oracle BI - Takeaways

37

ImplementationConsiderations

38

The whole prototyping process can be done on a simple star in roughly two weeks

Allow for more time if you have:• Large data volumes• Difficult performance targets• More complex models and logic• Many disparate report patterns or lots of reports to consider• More stars are needed (e.g., Actuals and Budgets together)

Development effort depends on # new objects• Typically only another two weeks needed (ETL & OBI RPD)• A few more for regression test and deployment

Performance Layer Implementation Effort


Implementation Considerations

39

Use Production data volumes for accurate analysis

Use Production DDL, ETL code, OBI RPD and OBI Webcat

Quiet, unused machine for accurate benchmarking

Use hardware that is as similar to Prod as possible• KPI uses a database benchmarking tool to

compare environments

Ensure a Good Tuning Environment



40

Additional ETL & RPD Development• Use SQL scripts instead of Informatica mappings (less effort, faster

execution)

Additional Testing – Regression test is easy Additional ETL Run Time – may be critical Additional Database size - minor Customization Propagation / Impact Analysis

• True of any aggregate

Complex logic will be more difficult• Financial Analytics - snowflake with multiple segment hierarchies

• HR Workforce Event & Snapshot logic uses effective dates for future dated events

Considerations



41

Q & A

Who Is KPI Partners?www.kpipartners.com

The Leader In Oracle BI & EPM 42

Strategic Consulting | Systems Implementation | Training

Depot Repair Analytics

Fixed Asset Analytics

Manufacturing Analytics

Salesforce.com Analytics

Student Info Analytics

Subledger (SLA) Analytics

and more…

Transform Data Into Insight

Staff built from Oracle/Siebel/Hyperion engineering teams

On-site, off-shore and blended shore delivery models

Exclusive pre-built solutions for Oracle BI & E-Business Suite

Oracle BI

Hyperion

Endeca

Exalytics

43

Email: [email protected]

Web: kpipartners.com/contact

KPI World Headquarters39899 Balentine Drive

Suite #375

Newark, CA 94560

Phone: (510) 818-9480

Customer CareContact Us

The Leader In Oracle BI & EPM

New York, NY

Chicago, IL

Boston, MA

Minneapolis, MN

San Diego, CA

Greensboro, NC

North America Offices

Bangalore, India Hyderabad, India

Global Offices

mailto:[email protected]

http://www.kpipartners.com/contact

http://www.facebook.com/kpipartners

http://linkedin.com/company/kpipartners

http://vimeo.com/kpi

44

www.kpipartners.com

Performance Tuning Oracle's BI Applications

Technology

Transcript of Performance Tuning Oracle's BI Applications