The door to data clarity - interRel Consulting...Hybrid Aggregation Under the Covers •Enables the...
Transcript of The door to data clarity - interRel Consulting...Hybrid Aggregation Under the Covers •Enables the...
The door to data clarity
Oracle Analytics Partner of the Year
interRel Highlights and Company Overview
Our consultants average 9+
years of EPM experience
Focused on EPM & BI
since1997
1,000+ EPM projects completed, 12+ in
EPM Cloud
OTN tours, regional roadshows, weekly
webcasts, Play It Forward videos
(YouTube), newsletters, and blog posts!
#1 EPM Cloud Partner
Industries
Healthcare, Financial Services, Higher Ed, Retail & Wholesale, Tech & Telecom,
Consumer Goods, Energy, Insurance, & Manufacturing
Free Education
• 2019 Oracle Analytics Partner of the Year
• 2016 Oracle Global Partner of the year – Cloud EPM & BI
• 4-time Oracle Excellence Award• 9-time Inc. 5000 – list of fastest
growing U.S. private companies• Multiple Kscope Top Speaker
awards
AwardsCompany was founded on Essbase and has the
most experienced Essbase resources.
170+ Cloudspecializations, 60+
free Cloud webcasts, free Cloud videos, and
2 Cloud books!
Hybrid in Planning Cloud
Introduction
While this is a presentation on using Hybrid in EPBCS(Planning Cloud, Planning Business Process), we will talk about concepts and
differences between on-Premises Essbase, Planning, OAC and EPBCS.
Hybrid Defined
BSO
• Powerful calculation engine• Limited number of dimensions• Can be slow to aggregate • Ability to run procedural calculations• Uses Calc Script language • Data can be written to any level• Very structured. Dense and Sparse
dimensions are important
ASO
• Larger Dimensions• More Dimensions• No need to aggregate, Fast retrievals • Few Calculation options available• Uses Multi-Dimensional Expression
Language (MDX) for formulas• Data stored at Level 0 only• We don’t need no Dense/Sparse
configuration
If BSO and ASO had a Baby
ASO
BSO
Hybrid
Hybrid – The Love Child
• Like his Mom• Extensive Calculation capabilities• Calc Script Language• Utilizes Dense and Sparse dimensions
• Like his Dad• More and larger dimensions• Only stores data at Level 0
• Recessive Gene• Can be set to use ASO aggregations (Full or Partial)
What’s the Same? ASO vs. BSO
• Both types of databases are defined by their outline
• Most dimension and member properties like dimension type, data storage (store, never share, label only), consolidation tags and aliases
• Building dimensions
• Loading Data
• Certain rules files properties are database type specific but the overall interface and steps are the same.
• ASO – load to level zero members only
What’s the Same? End User Perspective
• End users won’t care whether their database is ASO or BSO
• The way users' access ASO are the same as BSO• Financial Reporting or Planning forms
• Smart View Add-in
• Repeat… no differences (just more dimensions)
Hybrid Cubes
• The issue with BSO• Large sparse dims can’t be
dynamically calculated• Stored sparse dims increase agg time
and database size
• The issue with ASO• Specifically designed for databases with
large sparse dims
• Missing powerful calc engine
• Can’t we have both? YES!
• Hybrid = BSO functionality + ASO performance for aggregation
HYBRID CUBES
Fast, dynamic
agg of dimensions
Powerful Calculation
Engine
How To Take Advantage of Hybrid
1. Enable hybrid
• On-premise Essbase & Planning
• Configuration setting “turned on” by cube in the Essbase.cfg
• OAC
• Enabled by default; no further action required
• Planning Cloud
• Required steps to convert to Enterprise and then Enable Hybrid
• More on this in just a bit
2. Set upper level members of sparse dimensions to “dynamic calc” data storage
Hybrid Aggregation Under the Covers
• Enables the full BSO functionality of a BSO outline, utilizing calculation order and calculation scripts
• Stores data in BSO data and index files
• Uses the ASO query processor whenever possible• When there is a query that the ASO query processor cannot handle, the BSO
query processor takes over and accomplishes the task
• Make upper level members in one or more sparse dimensions to leverage hybrid
• Oracle has a design guide to help with design & tuning recommendations
Hybrid Benefits = BSO Calcs + ASO Aggs
Reduces database/applicati
on size
Improves cube refresh
performance
Reduces time for data export/import
Reduces maintenance
window downtime
Improves performance of business rules
Improves usability
Calc Functions Not Supported in Hybrid
• Some calc functions aren’t supported for hybrid aggregation mode. Essbase will default to the slower block storage calculation mode if these are encountered - examples:• @ALLOCATE, @CREATEBLOCK, @XWRITE
• All functions are supported for hybrid in OAC not in 11.1.2.4 Essbase/Planning/EPM Cloud
• Notice entries in log:• Hybrid Aggregation Mode disabled for [??] due to [??]
Calc Functions Not Supported in Hybrid – OAC Essbase
• Earlier limitations with cross dims and two-pass calculations are eliminated in OAC Hybrid
• OAC Hybrid is more functional that on-prem Hybrid as it supports all functions while on-prem does not
• EPBCS hybrid will be the same as on-prem Hybrid
• The following functions are not supported for hybrid aggregation mode in OAC Essbase. If encountered, Essbase defaults to block storage execution for these functions.
• @ALLOCATE
• @CREATEBLOCK
• @IRREX
• @MDALLOCATE
• @MDSHIFT
• @MOVSUMX
• @PTD
• @SANCESTVAL• @STDEV• @STDEVP• @STDEVRANGE• @SYD• @TREND• @XWRITE
Calc Functions Not Supported in Hybrid – 11.1.2.4 Essbase
The following functions are not supported for hybrid aggregation mode in Essbase 11.1.2.4.
If encountered, Essbase defaults to block storage execution for these functions.
@ACCUM
@ALLOCATE
@ANCEST
@ANCESTVAL
@AVGRANGE
@COMPOUND
@COMPOUNDGROWTH
@CORRELATION
@CREATEBLOCK
@CURRMBR
@CURRMBRRANGE
@DECLINE
@DISCOUNT
@GROWTH@INTEREST@IRR@IRREX@MDALLOCATE@MDANCESTVAL@MDPARENTVAL@MDSHIFT@MEMBER@MOVAVG@MOVMAX@MOVMED@MOVMIN@MOVSUM@MOVSUMX@NPV
@PARENT@PARENTVAL@PTD@SANCESTVAL@SHIFT@SLN@SPLINE@STDEV@STDEVP@STDEVRANGE@SYD@TREND@XRANGE@XREF@XWRITE
Hybrid in Planning CloudRules and Regulations (a.k.a. Best Practices)
Hybrid Comes to Planning Cloud
• Not available for PBCS
• You can enable hybrid in EPBCS applications
• Enabled for all cubes within the application including the frameworks cubes
• Oracle has implemented “best practices” controls to ensure good performance with hybrid enabled cubes• More on this in just a moment…
Hybrid Dimension Design Scenarios
• BSO = All sparse members are set to Store/Never Share
• Hybrid Full Dynamic = All sparse parents are set to Dynamic Calc
• Hybrid Top Level = Top most sparse parents are set to Dynamic and the rest are set to Store/Never Share
• Hybrid Partial = Some sparse dimension parents are set to Dynamic Calc and some sparse dimension parents are set to Store / Never Share
• Hybrid BSO-ASO = Use Smart Push / ASO cubes for main aggregation; use Dynamic Calc for Sparse members where you need rollup in BSO (e.g. Plan Element)
What are “Best Practice” Validations?
• Validation checks that Oracle puts in place for hybrid applications to help manage potential performance issues
• You can’t “break” one validation rules
• When you try to refresh the application, if a “best practice” validation is not met, the cube will not refresh / hybrid will not be enabled
“Best Practices” Validations for Hybrid – both Custom & EPBCS Cubes – Warnings, Errors
“Best Practice” Validation
Block Size
# of Blocks
Number of dense dimensions
Max number of child members under any Dynamic Calc parent
Max number of child members under any Store parent
Flag parents with single child for level1 and above dimensions
Level 1 and above not set to Dynamic Calc or Label Only in Dense dimensions
Usage of Dynamic XREFs
Dimensions consolidating to Root
How To Enable Hybrid in Planning Cloud
Process to Use Hybrid For Existing Customers
• Enable Hybrid BSO AFTER 19.06 has been applied to the pod
• Do a cube refresh
• Then take an LCM backup of the application on your test environment and download it• This is because the service will be recreated with Essbase 5xx Version, and
the application including all snapshots will be deleted
• Please note that a successful cube refresh must be completed before taking the LCM backup.
Request SR
• Create SR
• You should also submit provide feedback and request the same
• Then support can open a bug from this feedback & attach the bug to the SR
• After we get confirmation and approval from you (through an SR), we will recreate the service in your POD with Essbase 5xx version
• If you need this on more than one environment, provide feedback from each of the pods and open one SR per pod
• All such requests (bugs) will be assigned to EPM Product Manager Mark Rinaldi for Approval. After the request is approved, the service will be recreated with Essbase 5xx.
Info to include in SR
• I <Your Name> authorize oracle to enable Hybrid mode on my EPBCS environment <Service URL>. I know that the environment will be recreated and the application will be deleted, and I will have to either import the backup LCM snapshot that I have or recreate the application from scratch.
• Support can then open a bug from the provided feedback and assign to 11677/General (if it is not done by default) with the following information:
• Subject: Enable Hybrid mode on My EPBCS environment
• Description: <Fills in the customer provided info in the SR>
Convert PBCS to Hybrid BSO in EPBCS/Enterprise for Existing Customers
• Hybrid is only available in EPBCS (for existing EPM Cloud customers)
• Will have to convert PBCS to EPBCS app• Must follow all of the steps to convert to EPBCS (documented here:
https://docs.oracle.com/en/cloud/saas/planning-budgeting-cloud/epbca/converting_app_to_enterprise_104xeebf37c1.html)
• You don’t have to enable the Frameworks though
• Application >> Overview >> Actions >> Convert to Enterprise
• Once your app is an Enterprise application, you are ready to enable!
Requirements when Converting PBCS to EPBCS / Enterprise
• You are commercially authorized to implement an application of type "Enterprise“ / you have an EPBCS subscription
• It does not use weekly, quarterly, or custom time periods
• You rename the Year dimension to Years prior to converting
• It uses simplified multicurrency (that is, it's not a standard currency application), if it's a multicurrency application.
• Artifact names don't use these prefixes:
• oep_—Prefix for artifacts common to all business processes
• ofs_—Prefix for Financials artifacts
• owp_—Prefix for Workforce artifacts
• opf_—Prefix for Projects artifacts
• ocx_—Prefix for Capital artifacts
Enabling Hybrid in Planning Cloud
• Hybrid can be enabled during application creation
• An existing application can also be converted to Hybrid by Enabling Hybrid Mode• The time for the hybrid conversion process depends on application / cube size
• Must be EPBCS application (hybrid not supported for PBCS)
• Before the application is enabled for Hybrid, Planning Cloud will perform a “best practices” validation check• Any failure must be fixed before the cubes are converted
• On successful validation the modules are reinitialized to bring in hybrid changes (though customizations are retained)
Enable Hybrid in Planning Cloud
Application >> Overview
Actions >> Enable Hybrid Mode
Confirm – Did you take your backups?
• Important!! You might need to get back to original app
• Strongly recommend you also take level 0 data exports via SUI for easy reload/recovery
Best Practice Validation Warnings & Errors
• Planning Cloud will display the warnings and errors that don’t follow the defined “best practices”
• You can’t move forward until errors are addressed
Downloaded Warnings and Errors
Expand to see the exact member names that failed
Once Errors are Resolved, Hybrid Enablement will begin
This could take a while depending on app size
Good Link on Cloud Customer Connect
https://cloudcustomerconnect.oracle.com/posts/92e42a9a01
Hybrid for New EPM Cloud Customers
• Hybrid availability changes with the new EPM Cloud pricing for new customers*
• Standard Planning Cloud• Max up to 8 cubes
• All BSO cubes are deployed in hybrid
• 5 seeded hybrid BSO cubes + 1 ASO for modules/frameworks
• Enterprise Planning Cloud• Same Hybrid & ASO cubes supported
• *still trying confirm this info
Tuning & Design TipsThe Pros
Which Dimensions to Define as Sparse / Dense?
• Goal: fewest number of densest blocks
• Good candidates for dense dimensions• Account & Period are usually dense
• Dimensions that aren’t very volatile / do not change often
• (If dense dimension changes require full database restructure)
• Dimensions with member formulas
• Good candidates for sparse dimensions• Attribute dimensions are only supported for sparse dimensions
• Really big, deep dimensions (product, entity, customer)
• Common sparse dims: Scenario, Version, Year, Entity
Block Size
• Goal:• Not too large
• Not too small
• “Just right”
• As dense as possible (cells are “filled up”)
• 2K – 200K
• Planning Cloud will do validations to ensure block size is in the recommended range
“Best Practice” Validation Warning Error
Block Size 350KB 550KB
Dimension Storage
• Balancing which members to set to “dynamically calc”
• Upper level dense members – almost always set to dynamic calc
• Upper level sparse members• As many as you can without impacting retrieval performance
• Test business rules that reference upper level sparse members
Dimension Ordering
Dense Dimension (Dynamic Non-Formula)
Dense Dimensions (Dynamic Formula)
Sparse Dimensions (Calculated and Stored)
Sparse Dimensions (Dynamic non-formula)
Sparse Dimensions (Dynamic formula)
Sparse Dimensions (Parallel Task dim)
Understanding Order of Calculations in Hybrid
• Hybrid was designed to follow exact calculation logic of classic BSO
• "Dynamic Calculation" changes the order in which Essbase calculates the values
• By changing dynamic settings and sparse/dense settings, users may change the calculation order of their cube, which can lead to some incorrect results
• Two ways to control calculation order in Hybrid:• Changing the dimension order
• Using solve order
Calculation Order for Dynamic Calculation
When Essbase dynamically calculates data values, it calculates the data in an order different from the batch database calculation order.
Batch Calculation Order
1. Dimension tagged as accounts
2. Dimension tagged as time
3. Other dense dimensions (in the order in which they appear in the database outline)
4. Other sparse dimensions (in the order in which they appear in the database outline)
5. Two-pass calculations
Dynamic Calculation Order
1. Sparse dimensions (in the order in which they appear in the database outline)
2. Dense dimensionsa. Dimension tagged as accounts, if dense
b. Dimension tagged as time, if dense
c. Time series calculations
d. Remaining dense dimensions
e. Two-pass calculations (Account 1st, Time 2nd)
3. Attributes
Other Considerations for Hybrid Calculation Order
• Attribute dimensions are dynamically calculated• Typically last step in the process
• Do not always apply time balance settings properly
• Asymmetric data / queries can cause differing calculation results• Symmetric data calculations produce the same results no
matter the dimension
• To address, if a member formula needs to “rollup”, set that member to Store
• E.g. Set “Sales” to Store with the member formula of “Units”*”Price”;
• Don’t set “Sales” to Dynamic calc
Query Solve Order
• Let Essbase run its default calculation order if possible
• However, if you need to control order of calculation / member formula, then set solve order• The higher the solve order setting, the later the member is calculated
• A formula with a solve order of 1 will be solved before a member with a solve order of 2
• Set per dimension / member to ensure accurate results and tuning the performance
BSO Tips That Still Apply
• Reduce fragmentation• Force restructure in Calc Manager
• Remove empty blocks
• Caches (but we can’t change that in Planning Cloud)
• Calc Parallel
Settings Not Supported in Hybrid
• SET CACHE, SET LOCKBLOCK, and SET MSG settings not allowed in business rules
• SET UPDATECALC is turned off by default
• The error you get in calc manager may not reflect the true problem (you are using one of these settings)
Hybrid in Planning CloudinterRel Findings
Findings Summary
• Improvement found in performance with new “under covers” Essbase version update
• No other changes made
• Hybrid can provide a significant performance and size improvement to traditional block storage cubes
• Applying dynamic calc settings to ALL sparse aggregating parent members did not provide positive retrieval results
• Mixed approach (1 stored dimension / top levels dynamic, rest sparse dynamic) provided best results
• Smaller and/or flatter aggregating sparse dimensions perform better within calculations and queries
• Larger and deeper aggregating sparse dimensions may not perform well in calculations and queries
Use Cases Tested
Example of Hybrid Testing
• For the FinPln and DtlPln Planning databases, the optimum performance based on existing test results is applying dynamic calc settings to the following dimensions
• Plan Element, Wedge, Sub Account, Company, Cost Center – ALL parent members
• Report Center – Gen 1 & 2 report center members in the Total Report Center hierarchy set to dynamic, all other report center parent members in the Total Report Center hierarchy stored
• Report Center is a large, deep hierarchy with multiple levels and although general agg performance could be improved by making it fully dynamic, query performance and dependent calculations on top level members were dramatically slower until the dependent members were stored
▪ Agg Performance over baseline
▪ Number of Blocks decreased 92%
▪ Database Size decreased 92%
▪ Agg calc time decreased 82%
▪ Query Performance over baseline
▪ Full RC Hierarchy (sparse) on rows increased 3%
▪ Full P&L Hierarchy (dense) on rows increased 205% (increase was approx. 2 sec to 6 sec)
▪ Calc Dependency on Top Sparse Members over baseline
▪ Calc time decreased 97%
Hybrid in PBCS / In GeneralThe Considerations
Hybrid is Enabled for All Cubes
Can’t enable hybrid by plan type / cube• So you are stuck with the “best practice” guidelines for all cubes including ASO
Confusing “Error” Messages when Really they are Warnings
• Didn’t really error out but you think that it did
• Warnings on hybrid validations but still refreshed successfully
• If the refresh completes, you are good to go!
Error Report Helpful
• Watch out – when you open in Excel, there is more info if you expand the row size
• You do get a list of the members that fail / have validation issues
Hybrid in EPBCSThe Considerations
Hybrid is Supported in the Frameworks
• Functionality works the same; business rules are updated to remove rollups for dimensions that are set to dynamic
• You can also make parents dynamic in other dimensions and update OOB rules to exclude those rollups
• Customizations are retained• need to review customizations of member formulas, business rules to see if there are
any calculation issues encountered with changing to hybrid
Dimension Storage Settings for Frameworks
• Certain sparse parent members are made dynamic in each framework cube
• In certain sparse dimensions, the top dimension member is set to Store / Never Share
• You can’t change • But you can make the hierarchies below gen 1 dynamic• See appendix for example
• Main issue• No data displays at top of the dimension in ad hoc retrieval• If users retrieve in ad hoc, they have to select a member from within the dimension;
will not see data at the top of the dimension
Dimension Storage Settings for Frameworks
Stored Gen 1 Members Required
Gen 1 required to be “Stored" for:
• Period
• Entity
• Scenario
• Year
• Version
• Account
Capital framework
• Asset
• Asset Class
• Custom dims in Capital framework
Custom dims in Financials Gen 1 can be dynamic calc
Dynamic Gen 1 Members Allowed
▪ Custom dims in Financials
Summary
Hybrid in Planning Cloud Summary
• Hybrid BSO is better than BSO
• Hybrid BSO does not replace ASO; ASO still makes sense for reporting cubes
• Likely “mixed approach” will be best; Some sparse hierarchies are dynamic while 1-2 others are stored
• Make sure to test and validate all calculation logic
• Think through best practices before jumping in, especially in the Financials framework
Helpful Links
https://www.essbaselabs.com/2018/05/essbase-hybrid-calculation-engine_97.html?spref=tw&m=1
https://docs.oracle.com/cd/E66975_01/doc.1221/essbase_db/frameset.htm?hybrid_aggregation.html
https://docs.oracle.com/cd/E51367_01/financialsop_gs/FADAG/dcadynca.html#dcadynca15843
https://cloud.oracle.com/opc/paas/ebooks/OAC+Essbase+Hybrid+BSO+Performance+Tuning.pdf
Videos to expand what you’ve learned here
Product Introductions
New Features in Oracle Analytics
Cutting-Edge Cloud updates
Expert-level videos for BI gurus
Technical Reference in video form
• To experience the education revolution first-hand, join our community at epm.bi/videos
Play It Forward
Best Selling Hyperion Technical Book Series
Upcoming Webcast
3-6
3-13
9-4
9-11
9-18
9-25
10-2
WHAT IS ORACLE PROFITABILITY & COST
MANAGEMENT?
WHAT IS ORACLESTRATEGIC MODELING?
WHAT IS ORACLEACCOUNT RECONCILIATION?
WHAT IS ORACLEDATA VISUALIZATION?
WHAT IS ORACLE ENTERPRISE DATA
MANAGEMENT?
Thank you.