05 Rollup and Scan Components

AccentureAccenture Ab Initio TrainingAb Initio Training 11

Introduction to Ab Initio

Prepared By : Ashok Chanda


Ab initio Session 5Ab initio Session 5

RollupRollup ScanScan


RollupRollup

Rollup evaluates a group of input records Rollup evaluates a group of input records that have the same key and then generates that have the same key and then generates data records that either summarize each data records that either summarize each group or select certain information from group or select certain information from each group. each group.

Rollup gives you more control over record Rollup gives you more control over record selection, grouping, and aggregation than selection, grouping, and aggregation than Aggregate . Rollup does not produce Aggregate . Rollup does not produce intermediate summary records. If you want intermediate summary records. If you want intermediate summary records, use Scan. intermediate summary records, use Scan.


Alphabetical List of Rollup Alphabetical List of Rollup ParametersParameters

check-sort check-sort key key key-method key-method limit limit logging logging log_input log_input log_intermediate log_intermediate log_output log_output log_reject log_reject max-core max-core ramp ramp reject-threshold reject-threshold sorted-input sorted-input transform transform


Parameter Descriptions for Parameter Descriptions for RollupRollup

sorted-inputsorted-input :(boolean, required) :(boolean, required) When set to When set to In memory: Input need not be In memory: Input need not be

sortedsorted, Rollup accepts ungrouped input, and , Rollup accepts ungrouped input, and requires the use of the requires the use of the max-coremax-core parameter. parameter.

When set to When set to Input must be sorted or Input must be sorted or groupedgrouped, Rollup requires grouped input, and , Rollup requires grouped input, and the the max-coremax-core parameter is not available. parameter is not available.

Default is Default is Input must be sorted or groupedInput must be sorted or grouped..



key-method key-method :(choice, optional) :(choice, optional) Method by which the component Method by which the component

groups records. Choose one of the groups records. Choose one of the following: following: Use key specifierUse key specifier — the component uses — the component uses

a key specifier. a key specifier. Use key_change functionUse key_change function — the — the

component uses the component uses the key_changekey_change transform function. transform function.



keykey :(key specifier, optional) :(key specifier, optional) Names(s) of the key field(s) Rollup can use Names(s) of the key field(s) Rollup can use

to group or define groups of data records. If to group or define groups of data records. If the value of the the value of the key-methodkey-method parameter is parameter is Use key specifierUse key specifier, you must specify a , you must specify a value for the value for the keykey parameter. parameter.

If you specify If you specify Use key_change functionUse key_change function for the for the key-methodkey-method parameter, the parameter, the keykey parameter is not available. parameter is not available.

For an alternate method Rollup can use to For an alternate method Rollup can use to define groups, see the define groups, see the key-methodkey-method parameterparameter



max-core max-core :(integer, required) :(integer, required) Maximum memory usage in bytes. Maximum memory usage in bytes. Only available when the Only available when the sorted-inputsorted-input parameter parameter

is set to is set to In memory: Input need not be In memory: Input need not be sortedsorted..

The default value of The default value of max-coremax-core is is 6710886467108864 (64 (64 megabytes). megabytes).

If the total size of the intermediate results Rollup If the total size of the intermediate results Rollup holds in memory exceeds the number of bytes holds in memory exceeds the number of bytes specified in the specified in the max-coremax-core parameter, Rollup parameter, Rollup writes temporary files to disk. writes temporary files to disk.


More on :The sorted-input More on :The sorted-input parameterparameter

Rollup can process either grouped or Rollup can process either grouped or ungrouped input, depending on the ungrouped input, depending on the setting of the sorted-input parameter setting of the sorted-input parameter .There are two choices for the sorted-.There are two choices for the sorted-input parameter:input parameter:

Input must be sorted or grouped Input must be sorted or grouped (default)(default)

In memory: Input need not be sortedIn memory: Input need not be sorted


Input must be sorted or Input must be sorted or groupedgrouped

When you leave the sorted-input When you leave the sorted-input parameter set to this default, Rollup parameter set to this default, Rollup requires data records grouped requires data records grouped according to the key parameter . If according to the key parameter . If you need to group the records, use you need to group the records, use Sort with the same key specifier that Sort with the same key specifier that you use for Rollup.you use for Rollup.



With sorted-input set to Input must be sorted or grouped, With sorted-input set to Input must be sorted or grouped, Rollup defines groups of data records in one of the following Rollup defines groups of data records in one of the following ways:ways:

According to the key parameter.According to the key parameter.

By the key_change transform function , if you set the key-By the key_change transform function , if you set the key-method parameter to Use key_change function.method parameter to Use key_change function.

Rollup only supports the key_change transform function when Rollup only supports the key_change transform function when the sorted-input parameter is set to Input must be sorted or the sorted-input parameter is set to Input must be sorted or grouped.grouped.

Rollup considers records as belonging to the same group if Rollup considers records as belonging to the same group if they have the same key value, or if the key_change transform they have the same key value, or if the key_change transform function returns 0 (false).function returns 0 (false).



The key_change transform function takes two arguments: The key_change transform function takes two arguments:

A previous input recordA previous input record

The current input recordThe current input record

The key_change transform function should return 0 (false) if The key_change transform function should return 0 (false) if the key values in these two records are the same, or non-0the key values in these two records are the same, or non-0

(true) if they are not the same.(true) if they are not the same.

If you want Rollup to treat all records as one group, specify If you want Rollup to treat all records as one group, specify the value in the key parameter as {}.the value in the key parameter as {}.


In memory: Input need not In memory: Input need not be sortedbe sorted

When you set the sorted-input parameter to this choice, When you set the sorted-input parameter to this choice, Rollup accepts ungrouped input, and groups all records Rollup accepts ungrouped input, and groups all records according to the key parameter .With sorted-input set to In according to the key parameter .With sorted-input set to In memory: Input need not be sorted, Rollup maximizes memory: Input need not be sorted, Rollup maximizes performance by holding intermediate results in main memory.performance by holding intermediate results in main memory.

The size of the table of intermediate results (until it reaches The size of the table of intermediate results (until it reaches the number of bytes specified in the max-core parameter) the number of bytes specified in the max-core parameter) depends on:depends on:

The number of distinct key valuesThe number of distinct key values

The size of the input record format The size of the input record format

The size of temporary_typeThe size of temporary_type


Built-in Functions for Built-in Functions for RollupRollup

The following aggregation functions The following aggregation functions are predefined and are only available are predefined and are only available in the rollup component:in the rollup component:

avgavg maxmax

countcount minmin

firstfirst productproduct

lastlast sumsum


Modes of Roll upModes of Roll up

Template ModeTemplate Mode

Use of functions such as sum() ,count() ,min() ,max()Use of functions such as sum() ,count() ,min() ,max()

Expanded ModeExpanded Mode

To find other details about min or max value record.To find other details about min or max value record.

Example : Example : you want to determine the price of you want to determine the price of the largest single purchase and the item that the largest single purchase and the item that was purchased was purchased


Rollup : Template Mode Rollup : Template Mode (Grid View)(Grid View)


Rollup : Template Mode Rollup : Template Mode (Text View)(Text View)

Out ::rollup(in) =begin out.cust_id :: in.cust_id; out.first_date :: min(in.dt); out.last_date :: max(in.dt); out.total_amount :: sum(in.amount);end;


Rollup Properties & Package Editor


Runtime behavior of Runtime behavior of RollupRollup

Rollup reduces each group of data records Rollup reduces each group of data records to a single aggregate output record, using to a single aggregate output record, using a series of transform functions as follows:a series of transform functions as follows:

With the first data record of each group, With the first data record of each group, Rollup creates a temporary aggregate Rollup creates a temporary aggregate record. record.

With each following data record of the With each following data record of the same group, Rollup updates the temporary same group, Rollup updates the temporary aggregate record. aggregate record.


Runtime behavior of Runtime behavior of RollupRollup

When you set sorted-input to Input must be When you set sorted-input to Input must be sorted or grouped, Rollup writes the sorted or grouped, Rollup writes the temporary aggregate record to the out port temporary aggregate record to the out port after processing the last data record of each after processing the last data record of each group, and repeats the preceding process group, and repeats the preceding process with the next group.with the next group.

When you set sorted-input to In memory: When you set sorted-input to In memory: Input need not be sorted, Rollup processes Input need not be sorted, Rollup processes all the data records, and then writes all the all the data records, and then writes all the temporary aggregate records to the out temporary aggregate records to the out port.port.


At runtime, Rollup executes At runtime, Rollup executes the following steps:the following steps:

Input selection : Input selection : optionaloptional Temporary initialization :Temporary initialization :optionaloptional RollupRollup FinalizationFinalization Output selection : Output selection : optionaloptional


Runtime : Input selectionRuntime : Input selection

If you have defined input_select, it filters the input If you have defined input_select, it filters the input records. records.

If the expression evaluates to 0 for a particular If the expression evaluates to 0 for a particular record , Rollup does not process the record (i.e., record , Rollup does not process the record (i.e., the record does not appear on any output port).the record does not appear on any output port).

If the expression produces NULL for a particular If the expression produces NULL for a particular record , Rollup:record , Rollup:

Writes the record to the reject port Writes the record to the reject port Writes a descriptive error message to the error Writes a descriptive error message to the error

portport



If you do not connect flows to the reject If you do not connect flows to the reject or error ports, Rollup discards the or error ports, Rollup discards the information. information.

If the expression evaluates to anything If the expression evaluates to anything other than 0 or NULL for a particular other than 0 or NULL for a particular record, Rollup processes the record.record, Rollup processes the record.

If you have not defined input_select, If you have not defined input_select, Rollup processes all records.Rollup processes all records.


Runtime : Temporary Runtime : Temporary initializationinitialization

Rollup passes the first record in each Rollup passes the first record in each group to the initialize transform group to the initialize transform function.function.

The initialize transform function The initialize transform function creates a temporary record for the creates a temporary record for the group, with record type group, with record type temporary_type. temporary_type.


Runtime :ComputationRuntime :Computation

Rollup calls the rollup transform Rollup calls the rollup transform function for each record in a group, function for each record in a group, using that record and the temporary using that record and the temporary record for the group as arguments.record for the group as arguments.

The rollup transform function returns The rollup transform function returns a new temporary record. a new temporary record.


Runtime :FinalizationRuntime :Finalization

Rollup repeats this procedure with each group.Rollup repeats this procedure with each group.

If you set sorted-input to In memory: Input need If you set sorted-input to In memory: Input need not be sorted:not be sorted:

After Rollup processes all the input records, it After Rollup processes all the input records, it calls the finalize transform function with the calls the finalize transform function with the temporary record for each group and an arbitrary temporary record for each group and an arbitrary input record from each group as arguments.input record from each group as arguments.

The finalize transform function produces an The finalize transform function produces an output record for each group.output record for each group.


Runtime :Output selectionRuntime :Output selection

If you have defined the output_select transform function, it If you have defined the output_select transform function, it filters the output records. filters the output records.

The output_select transform function takes a single The output_select transform function takes a single argument - the record produced by finalization - and argument - the record produced by finalization - and returns a value of 0 (false) or non-0 (true).returns a value of 0 (false) or non-0 (true).

Rollup ignores records for which output_select returns 0; it Rollup ignores records for which output_select returns 0; it writes all others to the out port. writes all others to the out port.

If you have not defined the output_select transform If you have not defined the output_select transform function, Rollup writes all records to the out port.function, Rollup writes all records to the out port.


Explanation of types and Explanation of types and transform functionstransform functions

temporary_typetemporary_type : Rollup requires temporary : Rollup requires temporary storage. The first section of the example defines storage. The first section of the example defines temporary storage as a record with a type named temporary storage as a record with a type named temporary_type: temporary_type:

type temporary_type type temporary_type recordrecord date (“DDMMYYYY") first_dt;date (“DDMMYYYY") first_dt; date (“DDMMYYYY") last_dt;date (“DDMMYYYY") last_dt; decimal(4) total_amt;decimal(4) total_amt; end;end;



initializeinitialize - The initialize transform function - The initialize transform function initializes temporary storage by setting the initializes temporary storage by setting the total_amount field of the temporary record to 0 and total_amount field of the temporary record to 0 and by saving the value from the dt field of the first by saving the value from the dt field of the first input record to the first_date and last_date fields of input record to the first_date and last_date fields of the temporary record:the temporary record:

Out :: initialize(in) =Out :: initialize(in) = beginbegin out.first_dt :: in.dt;out.first_dt :: in.dt; out.last_dt :: in.dt;out.last_dt :: in.dt; out.total_amt :: 0;out.total_amt :: 0; end;end;



rolluprollup - The rollup transform function fills the fields - The rollup transform function fills the fields in the temporary record for each group as follows:in the temporary record for each group as follows:

It accumulates, in the total_amt field, a sum of the It accumulates, in the total_amt field, a sum of the values in the amount field of the input records. values in the amount field of the input records.

It selects the smallest value of the dt field from the It selects the smallest value of the dt field from the group of input records, and inserts it in the first_dt group of input records, and inserts it in the first_dt field. field.

It selects the largest value of the dt field from the It selects the largest value of the dt field from the group of input records, and inserts it in the last_dt group of input records, and inserts it in the last_dt field.field.



Out :: rollup (tmp, in) =Out :: rollup (tmp, in) =beginbeginout.first_dt :: if (in.dt < tmp.first_dt)out.first_dt :: if (in.dt < tmp.first_dt) in.dtin.dt elseelse tmp.first_dt;tmp.first_dt;

out.last_dt :: if (in.dt > tmp.last_dt)out.last_dt :: if (in.dt > tmp.last_dt) in.dtin.dt elseelse tmp.last_dt;tmp.last_dt;out.total_amount :: tmp.total_amt + in.amount;out.total_amount :: tmp.total_amt + in.amount;end;end;



finalize finalize - The finalize transform function gathers the - The finalize transform function gathers the values of the customer_id field from the last record of the values of the customer_id field from the last record of the group, and the value of the first_date, last_date, and group, and the value of the first_date, last_date, and calculated total_amount fields from the temporary record calculated total_amount fields from the temporary record for the group, and produces the output record:for the group, and produces the output record:

out :: finalize (tmp, in) =out :: finalize (tmp, in) = beginbegin out.customer_id :: in.customer_idout.customer_id :: in.customer_id out.first_date :: tmp.first_dt;out.first_date :: tmp.first_dt; out.last_date :: tmp.last_dt;out.last_date :: tmp.last_dt; out.total_amount :: tmp.total_amt;out.total_amount :: tmp.total_amt; end;end;


SCANSCAN

Scan generates a series of Scan generates a series of cumulative summary records - such cumulative summary records - such as successive year-to-date totals - for as successive year-to-date totals - for groups of data records.groups of data records.

Scan produces intermediate Scan produces intermediate summary records. If you do not want summary records. If you do not want intermediate summary records, use intermediate summary records, use Rollup .Rollup .


Alphabetical List of Scan Alphabetical List of Scan ParametersParameters

check-sort check-sort key key key-method key-method limit limit logging logging log_input log_input log_intermediate log_intermediate log_output log_output log_reject log_reject max-core max-core ramp ramp reject-threshold reject-threshold sorted-input sorted-input transform transform


Parameter Descriptions for Parameter Descriptions for ScanScan

sorted-input : sorted-input : (Boolean, required) (Boolean, required) When set to When set to In memory: Input need not be In memory: Input need not be

sortedsorted, Scan accepts ungrouped input, and , Scan accepts ungrouped input, and requires the use of the requires the use of the max-coremax-core parameter. parameter.

When set to When set to Input must be sorted or Input must be sorted or groupedgrouped, Scan requires grouped input, and the , Scan requires grouped input, and the max-coremax-core parameter is not available. parameter is not available.

Default is Default is Input must be sorted or groupedInput must be sorted or grouped..



key-method : key-method : (choice, optional) (choice, optional) Method by which the component Method by which the component

groups records. Choose one of the groups records. Choose one of the following:following: Use key specifierUse key specifier — the component uses — the component uses

a key specifier. a key specifier. Use key_change functionUse key_change function — the — the

component uses the component uses the key_changekey_change transform function. transform function.



key : key : (key specifier, optional) (key specifier, optional) Name(s) of the key field(s) Scan can Name(s) of the key field(s) Scan can

use to group or define groups of data use to group or define groups of data records.records.

For an alternate method Scan can For an alternate method Scan can use to define groupsuse to define groups



max-core : max-core : (integer, required) (integer, required) Maximum memory usage in bytes. Only Maximum memory usage in bytes. Only

available when the available when the sorted-inputsorted-input parameter parameter is set to is set to In memory: Input need not be In memory: Input need not be sortedsorted. The default value of . The default value of max-coremax-core is is 1048576010485760 (10 megabytes). (10 megabytes).

If the total size of the intermediate results If the total size of the intermediate results Scan holds in memory exceeds the number Scan holds in memory exceeds the number of bytes specified in the of bytes specified in the max-coremax-core parameter, Scan writes temporary files to parameter, Scan writes temporary files to disk. disk.


Specifying a Transform for Specifying a Transform for ScanScan

Input selection : Input selection : optionaloptional Temporary initialization : Temporary initialization : optionaloptional RollupRollup FinalizationFinalization Output selection : Output selection : optionaloptional Key_change : Key_change : optionaloptional


Determining Whether to Determining Whether to Scan Sorted or Unsorted Scan Sorted or Unsorted

InputInput Scan can process either sorted or Scan can process either sorted or

unsorted input, depending on the unsorted input, depending on the setting of the setting of the sorted-inputsorted-input parameter parameter. . The setting can be one of the following: The setting can be one of the following:

Input must be sorted or groupedInput must be sorted or grouped (default) (default)

In memory: Input need not be In memory: Input need not be sortedsorted


Runtime behavior of Runtime behavior of ScanScan

Scan produces a series of n output records for each group of Scan produces a series of n output records for each group of n input data records, computing each output record as an n input data records, computing each output record as an aggregation of the input records processed so far, as follows:aggregation of the input records processed so far, as follows:

With the first data record of each group, Scan creates a With the first data record of each group, Scan creates a temporary aggregate record. temporary aggregate record.

With each following data record of the same group, Scan With each following data record of the same group, Scan updates the temporary aggregate record, producing an updates the temporary aggregate record, producing an aggregate record of the records processed so far for each aggregate record of the records processed so far for each group.group.

After processing each data record, Scan writes the aggregate After processing each data record, Scan writes the aggregate record to the out port , using a series of transform functions.record to the out port , using a series of transform functions.


At runtime, Scan executes At runtime, Scan executes the following steps:the following steps:

Input selection : Input selection : optionaloptional Temporary initialization :Temporary initialization :optionaloptional ComputationComputation FinalizationFinalization Output selection : Output selection : optionaloptional



If you have defined input_select, it filters the If you have defined input_select, it filters the input records. input records.

If the expression evaluates to 0 for a If the expression evaluates to 0 for a particular record , Scan does not process particular record , Scan does not process the record (i.e., the record does not the record (i.e., the record does not appear on any output port).appear on any output port).

If the expression produces NULL for a If the expression produces NULL for a particular record , Scan :particular record , Scan :

Writes the record to the reject port Writes the record to the reject port Writes a descriptive error message to the Writes a descriptive error message to the

error porterror port



If you do not connect flows to the reject If you do not connect flows to the reject or error ports, Scan discards the or error ports, Scan discards the information. information.

If the expression evaluates to anything If the expression evaluates to anything other than 0 or NULL for a particular other than 0 or NULL for a particular record, Scan processes the record.record, Scan processes the record.

If you have not defined input_select, Scan If you have not defined input_select, Scan processes all records.processes all records.


Runtime : Temporary Runtime : Temporary initializationinitialization

Scan passes the first record in each Scan passes the first record in each group to the initialize transform group to the initialize transform function.function.

The initialize transform function The initialize transform function creates a temporary record for the creates a temporary record for the group, with record type group, with record type temporary_type. temporary_type.


Runtime :ComputationRuntime :Computation

Scan calls the Scan transform Scan calls the Scan transform function for each record in a group, function for each record in a group, using that record and the temporary using that record and the temporary record for the group as arguments.record for the group as arguments.

The Scan transform function returns The Scan transform function returns a new temporary record. a new temporary record.


Runtime :FinalizationRuntime :Finalization

Scan can calls the finalize transform Scan can calls the finalize transform function once for every input record.function once for every input record.

Scan passes the temporary record and the Scan passes the temporary record and the input record to the finalize transform input record to the finalize transform function.function.

The finalize transform function produces an The finalize transform function produces an output record for each input record.output record for each input record.


Runtime :Output selectionRuntime :Output selection

if you have defined the output_select transform function, it if you have defined the output_select transform function, it filters the output records. filters the output records.

The output_select transform function takes a single The output_select transform function takes a single argument - the record produced by finalization - and argument - the record produced by finalization - and returns a value of 0 (false) or non-0 (true). returns a value of 0 (false) or non-0 (true).

Scan ignores records for which output_select returns 0; it Scan ignores records for which output_select returns 0; it writes all others to the out port. writes all others to the out port.

If you have not defined the output_select transform If you have not defined the output_select transform function, Scan writes all records to the out port.function, Scan writes all records to the out port.



temporary_typetemporary_type : Scan requires temporary : Scan requires temporary storage. The first section of the example defines storage. The first section of the example defines temporary storage for Scan as a record with a type temporary storage for Scan as a record with a type named temporary_type. This type contains a field named temporary_type. This type contains a field named amount_to_date, which provides a place for named amount_to_date, which provides a place for Scan to store the running total it calculates: Scan to store the running total it calculates:

type temporary_type =type temporary_type = recordrecord decimal(8.2) amount_to_date;decimal(8.2) amount_to_date; end;end;



initializeinitialize - The initialize transform - The initialize transform function initializes temporary storage by function initializes temporary storage by setting the amount_to_date field of the setting the amount_to_date field of the temporary record to 0 :temporary record to 0 :

temp :: initialize(in) =temp :: initialize(in) = beginbegin temp.amount_to _date :: 0;temp.amount_to _date :: 0; end;end;



scan scan - The scan transform function accumulates, - The scan transform function accumulates, in the amount_to_date field of the temporary in the amount_to_date field of the temporary record for each group, a running total of the record for each group, a running total of the values in the amount field of the input records values in the amount field of the input records from each group processed so far:from each group processed so far:

out :: scan(temp, in) =out :: scan(temp, in) = beginbegin out.amount_to_date :: out.amount_to_date ::

temp.amount_to_date + in.amount;temp.amount_to_date + in.amount; end;end;



finalize finalize - The finalize transform function gathers - The finalize transform function gathers the values of the customer_id and dt fields from the the values of the customer_id and dt fields from the current record, and the value of the current record, and the value of the amount_to_date field from the temporary record, amount_to_date field from the temporary record, and produces an output record:and produces an output record:

out :: finalize(temp, in) =out :: finalize(temp, in) = beginbegin out.customer_id :: in.customer_id;out.customer_id :: in.customer_id; out.dt :: in.dt;out.dt :: in.dt; out.amount_to_date :: out.amount_to_date ::

temp.amount_to_date;temp.amount_to_date; end;end;


Thank YouThank You

End of Session 5End of Session 5

05 Rollup and Scan Components

Documents

Transcript of 05 Rollup and Scan Components