T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
No Silver Bullets : OBIEE Performance in the Real World Robin Moffatt, Principal Consultant May 2014
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
The Requirement
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
The Problem
Lots of moving parts and complexity Overall solution crosses disciplines and job roles OBIEE developer Database developer DBA Server Admin Network SAN LDAP etc
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
The Additional Problem
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
The Only Answer
Evidence-based DESIGN and DIAGNOSTICS
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Performance Beyond the Best Practices
Tear down the reliance on Best Practice, but with a viable, better, alternative instead.
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Two-pronged approach
Do It Right, First Time What Makes A Performant OBIEE System? !
If Its Not Done Right, Know How To Figure Out What Is Broke Practical elements of diagnosing the causes of performance issues Methodical analysis - nose to tail
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Resp
onse
time
Number of concurrent users
Performance: Response time is slow for one user Capacity: Response time degrades as user concurrency increases
Adding Capacity alone will not fix your performance problems Performance improvements can alleviate Capacity problems
Is your problem Performance or Capacity?
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Nose to Tail Diagnostics
Request
Response
BI Server
BI Server
Presentation Services
Managed Server
Managed Server
JavaHost
Presentation Services
Web Browser
Web Browser
BI Plug-in
BI Plug-in
DWH
[ ... ]
[ ... ]
DWH
[ ... ]
[ ... ]
User WebLogic Server OBIEE system components Data Source(s)
Netw
ork
Netw
ork
Not all components listed, eg cluster controller
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Time Profile
To understand why is it slow, we first must understand where is it slow !
Approach championed by Cary Millsap / Method-R Thinking Clearly About Performance (2010)
Performance improvement is proportional to how much a program uses the thing you improved.
Amdahls Law
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Time Profile in Action
End-user reports a response time of 40 seconds. How can we make it faster?
Where did the time get spent?
Response
BI ServerManaged Server
Presentation ServicesWeb Browser BI Plug-in
User WebLogic Server OBIEE system components
DB Query 1 25 seconds
DB Query 2 5 seconds
Data Processing 10 secondsPage Generation
5 seconds
DWH
Data Source(s)DWH
Tim
e, s
econ
ds
Component
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Time Profile in Action
Step Action Response Time (s) %
1 Physical SQL 1 execute on DB 25.00 62.5
2 Physical SQL 2 execute on DB [5.00] 3 BI Server combines dataset from DB results 10.00 25.0
4 Presentation Services generates page 5.00 12.5
Total 40.00 100.0
DB Query 1
DB Query 2
Data Processing
Page Generation
Time profile shows clearly : 1.Improve performance of Query 1 2.Push work into single query
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Deep Dive into BI Server Time from nqquery.log
Elapsed
Response
Physical
Physical Query x
Physical Query 1 BI Server processing (federation, calculation, aggregation)
Send to client, wait for acknowledgement !Eg. Presentation Services generating pivot tables, table scrolling/paging, etc.
Com
pilat
ion
DB C
onne
ct
Logical Query Summary Stats: [...] Compilation time c (seconds)
Physical query response time p1 (seconds)
Physical query response time px (seconds)
Logical Query Summary Stats: Elapsed time e
Logical Query Summary Stats: [...] Response time r
Physical Query Summary Stats: [...] DB-connect time d (seconds) r - d - c - max(p) e - r
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
nqquery.log or Usage Tracking?
Step nqquery.log source Usage TrackingLogical Query Compilation c Logical Query Summary Stats: [...] Compilation time S_NQ_ACCT.COMPILE_TIME_SECDatabase connection d Physical Query Summary Stats: [...] DB-connect time n/aPhysical Query 1 response p Physical query response time S_NQ_DB_ACCT.TIME_SECPhysical Query 2 response p Physical query response time S_NQ_DB_ACCT.TIME_SECPhysical Query n response p Physical query response time S_NQ_DB_ACCT.TIME_SECLogical Query Response r Logical Query Summary Stats: [...] Response time S_NQ_ACCT.TOTAL_TIME_SECLogical Query Elapsed e Logical Query Summary Stats: Elapsed time n/aBI Server processing (federation, calculation, aggregation)
r-d-c-max(p) [derived] [derived]
Send to client, wait for acknowledgement (eg sawserver generating pivot tables etc)
e-r [derived] n/a
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Usage Trackings blind spot
Usage Tracking is good, but it omits the Total Elapsed Time !!!!!!!
This can hide problems:
Elapsed
ResponseSend to client, wait for acknowledgement
nqquery.log: Logical Query Summary Stats: [...] Response time r Usage Tracking: S_NQ_ACCT.TOTAL_TIME_SEC
nqquery.log: Logical Query Summary Stats: Elapsed time e Usage Tracking: [Not Available]
uh oh!
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Response
Iterative Approach
Be sure to account for all time, end-to-end (nose to tail) Expand time profile to focus on where the time is going in particular In reality, time profile based on BI Server alone will help in a lot of cases
Requestnqquery.logsawlog.log DB Explain plans, V$SQL, etcBrowser tools WLS logs
BI ServerManaged Server
Presentation ServicesWeb Browser BI Plug-in
User WebLogic Server OBIEE system components
DWH
Data Source(s)DWH
Netw
ork
Netw
ork
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Iterative Approach
Most components have extended logging levels Presentation Services WebLogic Server Browser Tools
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Digging Deeper with DMS Metrics
OBIEE supports Dynamic Monitoring System (DMS) !
Hundreds of low-level metrics !
Ranging from the obvious (BI Server Total Sessions) to the less obvious (Peak_DXEParameterizedImpl_Count) !
Once we have found WHERE the time has gone, DMS metrics are one way to help us find out WHY
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Accessing DMS Metrics
DMS Spy opmn EM obi-metrics-agent + graphite + grafana
http://ritt.md/oma-intro
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
DMS Case Study
Query is running slowly, but its not clear why
CPU is only at 50% utilisation Only a handful of rows returned from the DB
What is the BI Server up to for this time?
Step nqquery.log source Time %Logical Query Elapsed e Logical Query Summary Stats: Elapsed time 14Logical Query Response r Logical Query Summary Stats: [...] Response time 13Logical Query Compilation c Logical Query Summary Stats: [...] Compilation time 0 0%Database connection d Physical Query Summary Stats: [...] DB-connect time 0 0%Physical Query 1 response p1 Physical query response time 4 29%BI Server processing r-d-c-max(p) [derived] 9 64%Send to client, wait for cursor close e-r [derived] 1 7%
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
DMS Provides the Proof
The Connection Pool has a fixed Capacity Connections will start to Queue once Capacity is reached
Physical connections limit is reached for connection pool "Star_orcl_Connection Pool 01". Start delaying the queries
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Performance Diagnostics Approach
Capacity Examine capacity metrics (OS, DMS) over time Correlate with reported problems
Performance Build a time profile Dont know where to start? Try Usage Tracking or nqquery.log
-Long running queries -Logical queries returning lots of data to the client -Logical queries returning lots of data from the DB but small % to the client -Logical queries requiring lots of physical queries
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
So What Does Good Look Like?
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
This Is Not Best Practice
There is no one right way : It Depends c.f. ExtremeBI
The only best practice you should be using all the time is Use Your Brain. Steven Robbins / Tom Kyte
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
The Bucket List of Performance Gains
Dont Do It
Do It Only Once
Do It Less Often
Do It More Efficiently
(h/t Greg Rahn)
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Database Pushdown (Do It More Efficiently)
Reduce the amount of work/data handled further up the stack Federation across sources can be a challenge c.f. BI Server caching
BI Server
Managed Server
Presentation ServicesWeb Browser BI Plug-in
DWH
[ ... ]
User WebLogic Server OBIEE system components Data Source(s)
Netw
ork
Netw
ork
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Do You Need Database Pushdown?
BI Servers temp directory shows lots of work being done by BI Server Long time in BI Server processing phase Usage Tracking / Query logs : How much data the BI Server pulls back from the database as a percentage of rows returned to the user (low % is bad). How many database queries one report triggers - higher is generally less efficient
Step nqquery.log sourceLogical Query Elapsed e Logical Query Summary Stats: Elapsed timeLogical Query Response r Logical Query Summary Stats: [...] Response timeLogical Query Compilation c Logical Query Summary Stats: [...] Compilation timeDatabase connection d Physical Query Summary Stats: [...] DB-connect timePhysical Query 1 response p1 Physical query response timeBI Server processing r-d-c-max(p) [derived]Send to client, wait for cursor close e-r [derived]
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Feeding the Excel Monster (Do It More Efficiently / Dont Do it)
Whats being done with the data once its in Excel? Could it be done in OBIEE instead? !
Alternatives to Export from Analysis/Dashboard: Dump direct from the database Use Logical SQL against BI Servers ODBC/JDBC interface directly Oracle have specific recommendations (DocID 1558070.1 p.13) -Favour CSV export over Excel -Favour BI Publisher export over OBIEE Analysis Export
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Dont Do It / Do It Less Often
Filter Early Table Prompt != Filter
Report by Exception !
Make sure Dashboards have default/mandatory prompts(http://ritt.md/obi-prompts)
Yes, please do.
Be smart about Dashboard and Analysis design - dont cram everything into one page Less clutter : Better user experience & Better performance
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Not forgetting
Do It More Efficiently Aggregation
Do It Less Often BI Server caching !
Database optimisation (partitioning, indexing, parallelism, statistics, etc)
Balanced Hardware configuration
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
OBIEE Performance in the Real World
Evidence-based DESIGN and DIAGNOSTICS !
Do It Right, First Time ! If Its Not Done Right, Know How To Figure Out What Is Broke Methodical analysis - nose to tail
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
#EOF email: [email protected] web: http://www.rittmanmead.com/author/robin-moffatt/ twitter: @rmoff
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
References & Further Reading
Cary Millsap - Thinking Clearly About Performance http://queue.acm.org/detail.cfm?id=1854041
Blog series: OBIEE and Performance http://ritt.md/obiee-performance
obi-metrics-agent http://ritt.md/oma-intro
Greg Rahn - The Core Performance Fundamentals Of Oracle Data Warehousing Balanced Hardware Configuration http://wp.me/p3cJT-by
Oracle documentation - Oracle Database 2 Day + Data Warehousing Guide - Balanced Hardware Configuration E25555-03