Calc Scripts for Mere Mortals
-
Upload
parmitchoudhury -
Category
Documents
-
view
99 -
download
3
description
Transcript of Calc Scripts for Mere Mortals
-
459A
Calc Scripts for Mere Mortals
Presented by:Edward Roske, interRel Consulting
-
Agenda
Review the Basics Calc Script Commands Using Functions Controlling Execution Creating Blocks Organizing Calc Scripts Real World Calculations Optimization
-
Types Of Calculations
Outline consolidations Unary operators + - * / % Relies on hierarchy
Member formulas Allows complex calculations Can use functions and conditional processing
Calculation scripts Override outline operations Offer most control
-
Calculation Order
Two basic categories Outline order
Dense Sparse
Dimension order
-
Calculation Order All Dimensions First, Accounts Second, Time Third, all remaining dense dimensions Forth, all remaining sparse dimensions Last, Two-pass calculations
-
Calc Scripts
Reasons to use calc scripts: Override outline operators Specific calculation commands available Iterative calculations Variables Optimization
-
Calc Script Editor
Choice of the Essbase XTD Analytic Services Calc Script Editor or another text editor
If you use another editor: Save file with .csc extension Easier to perform some functions
Find/Replace See more lines of the script Printing options
Can mix and match
-
Essbase XTD Analytic Services Calc Script Editor Has a number of functions built in Command buttons simplify entries Validation function to detect errors Limited work area
-
Calc Script Editor Functions
Most simply enter the character that are on the face of the button
The is a notable exception Provides a list of available functions Grouped by function type Can include parameters
-
Member Selection
Associate the outline Can select members from the member list
Automatically adds quotes if needed Frequently easier/faster to just type the
member names
-
Online Help
Use menu option F1 does not work
Functions and commands are included Can also use Essbase XTD Analytic Services
documentation Technical Reference
-
Calculation Commands
Limited number of true calculation commands Not functions they are in another chapter Not Set statements they are covered under
optimization
-
Agg
Syntax AGG(Dimname, Dimname,);
Summary Aggregates dimension(s) according to the outline
operators Member formulas are ignored Very fast
Fewer than 6 levels
-
Array
Syntax Array ArrayName[DimName] = {initial values };
Summary Declares an array variable with as many
members as the dimension name that follows it Can be initialized with values if desired
Optional
-
Calc All
Syntax Calc All;
Summary Calculates all dimensions in the database Uses default calculation order This is typically the default calc Very efficient Can add Except to remove
dimensions/members
-
Calc First
Syntax Calc First;
Summary Calculates members tagged as TB First Does not update Time dimension, only Accounts
-
Calc Last
Syntax Calc Last;
Summary Calculates members tagged as TB Last Does not update Time dimension, only Accounts
-
Calc Average
Syntax Calc Average;
Summary Calculates members tagged as TB Average Does not update Time dimension, only Accounts
-
Calc Dim
Syntax Calc Dim(DimName, DimName,)
Summary Calculates the dimensions that are listed All dense dimensions are calculated first in the order
they appear Sparse dimensions are calculated last - also in the
order they appear If you need a different order, use multiple Calc Dim
commands Includes member formulas
-
Calc TwoPass
Syntax Calc TwoPass;
Summary Calculates members tagged as Two-Pass in the
Accounts dimension
-
CCONV
Syntax CCONV type->Currency name
Summary Special command reserved for currency
conversion Converts values to the currency name specified Only usable if you have the currency conversion
module
-
ClearBlock
Syntax ClearBlock ALL | UPPER | NONINPUT |
DYNAMIC Summary
Designed to clear blocks by replacing values with #Missing
Within a FIX block, may only clear a portion of a block
Options for which types of block to clear
-
ClearData
Syntax ClearData MemberName;
Summary Designed to set a member value to #Missing Works within a fix
-
ClearBlock vs. ClearData
ClearBlock is designed to clear an entire block At least a large portion of the block
No Member name, just block type ClearData is designed to clear a member
Usually before it is recalculated In many cases either one will work
-
DataCopy
Syntax DataCopy MemberName To MemberName;
Summary Copies data from one member to another All intersections are included Great for copying versions Will create blocks if they do not exist
-
If/Else/ElseIf/EndIf
Syntax IF (condition) ElseIf/Else EndIf;
Summary Condition must be if parenthesis In a calc script, associate with a member name
Does not really matter which member name Each If must end with EndIf Most effective on dense members
-
Fix/EndFix
Synatx Fix(MemberNames, Functions Etc.) ENDFIX
Summary Used to focus or limit a calculation Most effective on sparse members Used extensively
-
Loop/EndLoop
Syntax Loop (integer, break) EndLoop;
Summary Causes a block of code to be repeatedly
executed The integer determines the number of times the
loop runs The break value allows early exit Although it is a loop, its not a true programming
loop
-
Using Commands
Need to combine commands with functions Execute in the order they are written None by itself is that useful Used together, they provide a powerful tool
set to apply business rules to the database
-
Not Commands But
A member name by itself Calculates the member formula if present Calculates outline rollup if there is no formula
Example East;
Multiple members can be listed Member set functions can also be used
Example @IDESCENDANTS(2003)
-
Assignment Statements
Assign a value to a member Example
Budget = 0; Budget = Budget * 0.95;
There are other math operators More later
Every command ends with a semicolon (;) Okay, there are exceptions but most do
-
More on Member Names
All the members are calculated at once if possible
To force multiple passes of the database, group the member names in parenthesis One pass for each group in parenthesis Not usually what you want, so use with caution
-
Function Categories
Conditional Boolean Relationship Calculation Operators Mathematical Member Set Range and Financial
Allocation Forecasting Statistical Date and Time Miscellaneous Custom Defined
-
Calculation Operators
Nothing too exciting + - * / % Use parenthesis to control order of
operations
-
Conditional
Typically used within an If command >, >=,
-
Boolean
More traditional functions@ISACCTYPE@ISANCEST@ISCHILD@ISDESC@ISGEN@ISIANCEST@ISICHILD@ISIDESC@ISIPARENT
Typically used in If Statement Returns True/False
@ISISIBLING@ISLEV@ISMBR@ISPARENT@ISSAMEGEN@ISSAMELEV@ISSIBLING@ISUDA
-
Relationship Functions
Returns Values Frequently used to get other values in
database @ANCESTVAL@ATTRIBUTEBVAL@ATTRIBUTESVAL@ATTRIBUTEVAL@CURGEN@CURLEV@GEN
@LEV@MDANCESTVAL@MDPARENTVAL@PARENTVAL@SANCESTVAL@SPARENTVAL@XREF
-
Mathematical Functions
Most functions available@ABS@AVG@EXP@FACTORIAL@INT@LN@LOG@LOG10@MAX@MAXS
@MIN@MINS@MOD@POWER@REMAINDER@ROUND@SUM@TRUNCATE@VAR@VARPER
-
Member Set Functions
Returns a List of Member Names
@ALLANCESTORS@ANCEST@ANCESTORS@ATTRIBUTE@CHILDREN@CURRMBR@DESCENDANTS@GENMBRS@IALLANCESTORS@IANCESTORS@ICHILDREN
@MATCH
@IDESCENDANTS@ILSIBLINGS@IRSIBLINGS@IRDESCENDANTS@ISIBLINGS@LEVMBRS@LIST@LSIBLINGS
@MEMBER
@RDESCENDANTS
@MERGE@PARENT@RANGE
@RELATIVE@REMOVE@RSIBLINGS@SIBLINGS@UDA@WITHATTR
-
Range and Financial Functions
Accepts a range of members instead of individual members
@MINRANGE@MINSRANGE@NEXT@NEXTS@NPV@PTD@PRIOR@PRIORS@SHIFT@SLN@SUMRANGE@SYD
@ACCUM@AVGRANGE@COMPOUND@COMPOUNDGROWTH@CURRMBRRANGE@DECLINE@DISCOUNT@GROWTH@INTEREST@IRR@MAXRANGE@MAXSRANGE@MDSHIFT
-
Allocations Functions
Only two functions in this category Allocate MDAllocate
Used in simple situations to allocate an amount down to lower levels
-
Forecasting Functions
Similar to what you may have used in Excel @MOVAVG@MOVMAX@MOVMED@MOVMIN@SPLINE@TREND
-
Statistical Functions
Most common are available@CORRELATION@COUNT@MEDIAN@MODE@RANK
@STDEV@STDEVP@STDEVRANGE@VARIANCE@VARIANCEP
-
Date and Time
Just one to convert strings to a date ToDate
-
Miscellaneous
The catch-all category@CALCMODE@CONCATENATE@SUBSTRING@NAME
-
Using Functions
Frequently, functions are nested or used as parameters for another function
Functions are prefixed with @ Parameters are separated by commas Used in many places
Right of = FIX statements IFs etc.
-
Focusing Calculations
Two most common methods Fix If
Other methods: CrossDim operator MemberSet functions Relationship functions Variables
-
FIX
FIX limits the members that are processed Most effective when used on members of
sparse dimensions Only blocks that meet the Fix criteria are
processed Remaining blocks are simply skipped
Typical uses Only calculate one scenario Limit calc to current year
-
More on FIX
FIX works properly on dense members May result in all blocks being processed All statements between the FIX and ENDFIX
are executed as a block FIX statements an be nested Multiple members from the same dimension
are treated as or In other words, all members will be processed
-
What You Cannot Do With FIX
You cannot assign values to a member that is included in the FIX statement Actually, values cannot be assigned to any
member from a dimension in the FIX statement Example
Fix(Sales)Sales = 100;
ENDFIX Would result in an error
-
FIX On
Members from one or more dimensions Can list specific member names Can include member set functions Can include AND and OR
Use AND to get the intersection of two or more functions
Caution on empty sets
-
IF Statements
Conditional processing is often needed to process business rules
Can include complex conditions And, Or, Not, etc.
-
What You Can Check For
Most common are Members
Specific members MemberSet comparisons Use boolean functions
Values >, < = etc. Use #Missing for test
True is treated as 1 and false is 0
-
Syntax for IF
Place condition in parenthesis Associate with a member
Doesnt matter which member MemberName (
If (condition) Statements
EndIf ) Not required for member formulas
-
More on IF
Can have ElseIf (condition) Can Have Else Can have both Still terminate with single EndIf
Be sure to include ; All ElseIF/Else conditions have to remain
within parenthesis for the member association
-
To If or FIX
Generally, If works best on dense dimension members Particularly when ElseIF or Else are
includedBlock is only brought into memory once and all related
conditions are processedFix would bring block in multiple times
Conversely, FIX works best on sparse dimension members
Old saying Fix on Sparse, If on dense If youre dense, fix on sparse?
There are always exceptions
-
Other Method to Focus Calculations Cross Dim operator
Allows reference to specific cells Frequently in another block Technique for complex allocations
Relationship functions Can be used to get a value from another part of the
database Example is @ParentVal()
MemberSet functions Return member names Can be used to calculate those members or in a Fix/If
-
Loops
Can use the Loop/EndLoop to repeatedly execute calc commands
Not designed to increment variables and reference different members
Used to solve goal seeking problems
-
Why Create Blocks
Often need blocks that do not exist Typically when performing allocations
Actually, one of the most challenging aspects of writing calc scripts If a block does not exist, not processed by a calc script
Except outline calculations Frequent cause of calculation problems
ExampleIF (Budget > 100)
Actual = 50;ENDIF Assumes Scenario is sparse
-
Ways to Create Blocks
There are basically four methods to create blocks: Load data Calculate the database Use the DataCopy command Place a sparse member on the left side of an
assignment statement A little more complex than this
-
Load Data
During a data load, a block is created if it doesnt already exist Can use this to create blocks Create data records based on source information Load the file One trick is to use the Replace function of the
load rule
-
Calculate the Database
An outline rollup will create upper level blocks
Both Calc All/Dim and AGG will cause blocks to be created
Only blocks containing data are created Empty blocks are not written
-
DataCopy Command
Easiest way to create blocks If an intersection for the To member does
not exist, it is created Frequently, the only practical method to
create the needed blocks Danger in that it can create excessive
numbers of blocks
-
Assignment Statements
If The member on the left of the assignment
statement is sparseAnd
The database setting Create bocks on equations is selected
Blocks are created Not always possible to have a sparse
member on the left side of the assignment statement Try fixing on a dense member and assigning to
a sparse memberE.g. Fix on the account member and assign to another
May need to change dense/sparse settings
-
Changing Database Settings
A quick diversion When you change database settings, you
should stop and then start the database (or application) for the settings to take effect
-
Organizing Calc Scripts
Most calc scripts have common parts or sections Housekeeping Initial Calculation Mainline Rollup Back Calc
-
Housekeeping
The calculation environment is established Intelligent calculation is turned on/off Other Set commands are issue
Variables are declared Comments describing the purpose of the calc
script are entered Use /* and */ for comments
-
Key Set Commands
Set AggMissg ON | OFF Controls if #Missing values are aggregated If all children and #Missing, parent is et to
#Missing Upper level step-on
Caution needed if values loaded at upper levels SET MSG ERROR | WARN | INFO | SUMMARY |
DETAIL | NONE; Detail lists each block being processed Good for debugging Generates a large log and is slow
-
Initial Calculation
This is where business rules are applied Frequently include allocations
This usually requires a partial database rollup There may be many calculation blocks with
FIX statements Or If statements
Typically the most complex portion of the calc script
-
Mainline Calculation
This is the rollup of the database May be focused
e.g. FIX(Actual) Regardless, this is the major rollup of the
database
-
Back Calc
This is where two-pass members and any other members requiring special treatment are handled
Often requires recalculation of all or most of the blocks
Corrects percentages and ratios that may have been summed
-
Testing Calc Scripts
The following methodology works well: Use a test database Create test data in Excel
Keep it on its own sheet Clear the database Use Lock & Send to load test data Run the calc from Excel Retrieve onto a test sheet
-
More on Testing
Best to know expected outcome before retrieving
Can create a calc script to clear the database Minimizes switching between Excel and App
Manager The key is small amounts of data which
allows short test cycles Many people test with large amounts of data
resulting in long cycles and inability to get many tests in per day
When initial testing is complete, add to the test data set to make sure everything is working properly
-
Typical Issues
Intelligent Calc Prevents blocks from being calculated
FIX on too few or too many members Nested FIXs are ORs, not ANDs
Dynamic Calc runs after Batch Calc Dynamic calc members do not make good counters
Referring to the wrong block Remember, every intersection is processed unless
the calculation is focused
-
Real World Calcs
These exercises are derived from calculations created for customers but modified for the Sample application
-
Exercise
Simple Allocation Complex Allocation Comparable Sales Compute LQ (last quarter) in the scenario
dimension Compute 12 month rolling average Compute Market Share and Product Mix Get data from another database
-
Exercise (Continued)
Putting it together This is for the Budget dimension only Clear upper level blocks Add the member Insurance to the Measures
dimension and load a value at the top level Allocate the insurance expense to all products based
on Budgeted Sales Calculate Market Share in the calc script (may need
to change storage settings); insure it is correct at all levels
Include values for Last Quarter and a 6 month rolling average
Perform a rollup of the database
-
Goal
Make the calculation run faster Variety of settings/changes available to make
things run faster Many limiting factors
Existing reports/processes Source systems Hardware
-
Parallel Calculation
Apply multiple processors to a calc Only straight forward calcs can use this
If order dependent portions, calculated in serial Maximum of 4 processors Recommend 1 less than number on server Available in Essbase XTD Analytic
Services 6.5
-
Cache Settings
More is not always better Sometime Hyperion Essbase seems to spend
more time maintaining caches than calculating Usually find a point where increasing cache
does not improve performance Typically a trial and error process Set uncommitted / 0 on Transaction tab
-
Dense/Sparse Settings
This will make the largest difference Other factors influence decisions
Attribute dimensions If only a portion of the database is calculated
ideally you want to Fix on it as sparse members Reduces the number of blocks processed
May need to try a number of combinations Smaller blocks almost always calc faster
-
Review Dimension Order
Largest dense dimension to smallest Smallest sparse dimension to largest Hourglass shape Dimension calc order is influenced by order
of dimensions in the outline Goal is to process fewest blocks possible
-
Eliminate Database Fragmentation Previously calculated database takes longer
to calculate Faster to create blocks than to update them
Non-adjacent blocks can take longer to access As data is added to blocks, they grow and cannot
be rewritten to same place Export, ClearData (All), Import OR
Restructure Old DeFrag Command no longer available
Want to calc everything in the block at same time
-
Review Blocks that are Processed Can use SET MSG DETAIL Each block is listed as it is calculated
See application log Can determine number of times each block is
processed Can also see if expected blocks are being
processed Often, extra blocks are being read Correct by updating FIX
-
Control Calculation Environment Use SET Commands
Set CACHE HIGH; New cache Tracks the block used in the calc
Set CALCHASHTBL ON; Good for flat dimensions
Set FromBottomUp ON; Faster than TopDown Make sure you get correct results
-
Other Methods
Remove dimensions from the cube Can make a huge difference Some companies keep multiple databases
One with all dimensions and other with limited Partition
Move data that does not need to be calculated to another database
Intelligent Calc
-
More SET Commands
SET LOCKBLOCK HIGH; Allows more blocks to be locked Not really optimization but
SET CALCTASKDIMS n; Determines how many dimensions are used to
generate opportunities for the parallel calculation
-
Simulate the Calculation
Method to predict the actual calc time Based on the number of blocks that would be
created SET MSG ONLY; SET NOTICE HIGH; CALC ALL; Does not actually calc the database but
provides timings Run the calc, and use ratio of actual time to
simulated Not perfect but
-
Dynamic Calc
Can improve calc times significantly Make entire dimension dynamic calc
Never needs to be calculated Especially effective for Time dimension
Normally calculates very slowly All upper level members of dense dimension
Keeps block smaller Upper 1-2 levels of sparse dimension(s)
Be aware of how many blocks are needed to calc highest level
-
Custom Functions
Can write functions in Java Compile, save on server, and register Call from calc scripts Run slower than same calc command May run faster than complex series of
database passes required to perform the same thing
-
Custom Macros
Can predefine combinations of commands SumRange is actually a Macro
Combines Sum and Range functions Not necessarily faster but might avoid errors Need to define parameters May be possible to save a pass of blocks
-
Database Statistics
Not a optimization technique but a tool Can monitor calculation impacts
Number of blocks created Can refresh while calc is running
Block creation rate Not necessarily best for the calc
Copy and paste into Excel
-
Comments / Questions ?
Edward Roske
Office: 972-735-8716
Email: [email protected]
-
Please Complete Your Session Evaluation.
459ACalc Scripts for Mere Mortals
Presented by:Edward Roske,interRel Consulting
459AAgendaTypes Of CalculationsCalculation OrderCalculation Order All DimensionsCalc ScriptsCalc Script EditorEssbase XTD Analytic Services Calc Script EditorCalc Script Editor FunctionsMember SelectionOnline HelpCalculation CommandsAggArrayCalc AllCalc FirstCalc LastCalc AverageCalc DimCalc TwoPassCCONVClearBlockClearDataClearBlock vs. ClearDataDataCopyIf/Else/ElseIf/EndIfFix/EndFixLoop/EndLoopUsing CommandsNot Commands ButAssignment StatementsMore on Member NamesFunction CategoriesCalculation OperatorsConditionalBooleanRelationship FunctionsMathematical FunctionsMember Set FunctionsRange and Financial FunctionsAllocations FunctionsForecasting FunctionsStatistical FunctionsDate and TimeMiscellaneousUsing FunctionsFocusing CalculationsFIXMore on FIXWhat You Cannot Do With FIXFIX OnIF StatementsWhat You Can Check ForSyntax for IFMore on IFTo If or FIXOther Method to Focus CalculationsLoopsWhy Create BlocksWays to Create BlocksLoad DataCalculate the DatabaseDataCopy CommandAssignment StatementsChanging Database SettingsOrganizing Calc ScriptsHousekeepingKey Set CommandsInitial CalculationMainline CalculationBack CalcTesting Calc ScriptsMore on TestingTypical IssuesReal World CalcsExerciseExercise (Continued)GoalParallel CalculationCache SettingsDense/Sparse SettingsReview Dimension OrderEliminate Database FragmentationReview Blocks that are ProcessedControl Calculation EnvironmentOther MethodsMore SET CommandsSimulate the CalculationDynamic CalcCustom FunctionsCustom MacrosDatabase StatisticsComments / Questions ?Please Complete Your Session Evaluation.