Embed Size (px)
Transcript of Cache Informatica
Welcome to the World of Cache
The Hidden agendaa) Basics of Cache 1) Memory Cache 2) Where the cache files are created 3) Naming Conventions 4) Cache Calculations b) Advanced Cache 1) Look up Cache 2) Aggregator Cache 3) Joiner Cache 4) Ranker Cache
Lets get to the Basics:Cache is a combination of:
Index Cache: Server stores key values or condition values used to index values at a faster rate. Data Cache: Server stores output values.
Caching Storage Overview :
For Index Caches: a) Aggregators store group by values from Group-By ports. b) Rankers store Group-By values c) Joiners store index values for the master (Join condition columns) d) Lookups Stores lookup condition informationFor Data Caches: a) Aggregators store aggregate data based on Group-By ports (variable ports, output ports, non group by ports) b) Rankers store ranking based on Group-By port (output rows other than ranked column) c) Joiners store master table (Output columns not in Join condition). d) Look ups Stores stores lookup data that is not stored in the index cache.
Memory Cache : The server creates a memory cache based on size specified in the session properties which can be done manually based on certain calculations .By default, the PowerCenter Server allocates 1 GB to the index cache and 2GB to the data cache for each transformation instance. If the PowerCenter Server cannot allocate the configured amount of cache memory, it cannot initialize the session and the session fails. If the PowerCenter Server requires more memory than the configured cache size, it pages to the Disc. Since paging to disk can slow session performance, try to configure the index and data cache sizes to store data in memory.
Where are the Cache Files Created? The PowerCenter Server creates the index and data cache files by default in the PowerCenter Server variable directory, $PMCacheDir. If you do not define $PMCacheDir, the PowerCenter Server saves the files in the PMCache directory specified in the UNIX configuration file or the cache directory in the Windows registry. If the UNIX PowerCenter Server does not find a directory there, it creates the index and data files in the installation directory. If the PowerCenter Server on Windows does not find a directory there, it creates the files in the system directory. If a cache file handles more than 2 GB of data, the PowerCenter Server creates multiple index and data files. When creating these files, the PowerCenter Server appends a number to the end of the filename, such as PMAGG*.idx1 and PMAGG*.idx2. The number of index and data files are limited only by the amount of disk space available in the cache directory.
Three Instances when the Cache File exists even after Session completion: a) The session performs incremental aggregation. b) You configure the Lookup transformation to use a persistent cache. c) The session does not complete successfully.
Naming convention followed by Informatica Server: [ | _]_[partition index].[overflow index]For example,
PMLKUP8_4_2.idx,PMLKUP transformation type as Lookup, 8 the session ID 4 the transformation ID, 2 the partition index.
File Name Componen t
Cache file name prefix configured in the Lookup transformation.Describes the type of transformation: Aggregator transformation is PMAGG. Joiner transformation is PMJNR. Lookup transformation is PMLKUP. Rank transformation is PMAGG. Session instance ID number.
Session ID Transfor mation ID Partition Index
Transformation instance ID number.If the session contains more than one partition, this identifies the partition number. The partition index is zero-based, so the first partition has no partition index. Partition index 2 indicates a cache file created in the third partition. Identifies the type of file: Index file is .idx. Data file is .dat. If a cache file handles more than 2 GB of data, the PowerCenter Server creates multiple index and data files. When creating these files, the PowerCenter Server appends an overflow index to the filename, such as PMAGG*.idx.1 and PMAGG*.idx.2. The number of index and data files are limited by the amount of disk space available in the cache directory.
Aggregator: Index size: (Sum of column sizes in group-by ports + 17) X number of groups. Data Size: (Sum of column sizes of output ports + 7) X number of groups. Rank: Index size: (Sum of column sizes in group-by ports + 17) X number of groups. Data Size: (Sum of column sizes of output ports + 10) X number of groups + 20.
Joiner: Index Size: (Sum of master column sizes in join condition + 16) X number rows in master table. Data Size: (Sum of master column sizes NOT in join condition but on output ports + 8)X number of rows in master table LookUp: Index Size: # rows in lookup table [( S column size) + 16] * 2 Data Size: # rows in lookup table [( S column size) + 8]
DatatypeBinary Date/Time Decimal, high precision off (all precision) Decimal, high precision on (precision 18, 28) Decimal, high precision on (negative scale) Double Real Integer String Small integer
Aggregator, Rankprecision + 2 18 10 18 22 10 10 10 10 6ASCII mode: precision + 3
Joiner, Lookupprecision + 8 Round to nearest multiple of 8 24 16 24 32 16 16 16 16 16ASCII mode: precision + 9
Lookup Caches Overview The PowerCenter Server builds a cache in memory when it processes the first row of data in a cached Lookup transformation It allocates memory for the cache based on the amount you configure in the transformation or session properties. The PowerCenter Server stores condition values in the index cache and output values in the data cache The PowerCenter Server queries the cache for each row that enters the transformation. The PowerCenter Server also creates cache files by default in the $PMCacheDir If the data does not fit in the memory cache, the PowerCenter Server stores the overflow values in the cache files. When the session completes, the PowerCenter Server releases cache memory and deletes the cache files unless you configure the Lookup transformation to use a persistent cache.
Types of Lookup Cache When configuring a lookup cache, you can specify any of the following options: Persistent cache. You can save the lookup cache files and reuse them the next time the PowerCenter Server processes a Lookup transformation configured to use the cache Recache from source. If the persistent cache is not synchronized with the lookup table, you can configure the Lookup transformation to rebuild the lookup cache. Static cache. You can configure a static, or read-only, cache for any lookup source. By default, the PowerCenter Server creates a static cache. It caches the lookup file or table and looks up values in the cache for each row that comes into the transformation. When the lookup condition is true, the PowerCenter Server returns a value from the lookup cache. The PowerCenter Server does not update the cache while it processes the Lookup transformation. Dynamic cache. If you want to cache the target table and insert new rows or update existing rows in the cache and the target, you can create a Lookup transformation to use a dynamic cache. The PowerCenter Server dynamically inserts or updates data in the lookup cache and passes data to the target table. You cannot use a dynamic cache with a flat file lookup. For example, your lookup table is your target table. So when you create the Lookup selecting the dynamic cache what It does is it will lookup values and if there is no match it will insert the row in both the target and the lookup cache (hence the word dynamic cache it builds up as you go along), or if there is a match it will update the row in the target. On the other hand Static caches dont get updated when you do a lookup. Shared cache. You can share the lookup cache between multiple transformations. You can share an unnamed cache between transformations in the same mapping. You can share a named cache between transformations in the same or different mappings.
Calculating the Lookup Index Cache The lookup index cache holds data for the columns used in the lookup condition. The formula for calculating the minimum lookup index cache size is different than calculating the maximum size. For best session performance, specify the maximum lookup index cache size. Calculating the Minimum Lookup Index Cache 200 * [( S column size) + 16] Columns in lookup condition. The minimum size for a lookup index cache is independent of the number of source rows.
Calculating the Maximum Lookup Index Cache# rows in lookup table [( S column size) + 16] * 2 Columns in lookup condition.
Difference between Static and Dynamic CacheStatic cache:
U can insert rows into the cache as u pass to the target.The informatica server returns a value from the lookup table or cache when the condition is true.When the condition is not true, informatica server returns the default value for connected transformations and null for unconnected transformations.
You can use a relational or flat file lookup.
Dynamic cache : U can not insert or update the cache.
The informatica server inserts rows into cache when the condition is false.This indicates that the the row is not in the cache or target table. U can pass these rows to the target tableYou can use a relational look up only
Example: The Lookup transformation, LKP_PROMOS, looks up values based on the ITEM_ID. It uses the following lookup condition: ITEM_ID = IN_ITEM_ID1 ITEM_ID column size Column in lookup condition integer = 16
The lookup condition uses one column, ITEM_ID, and the table contains 60,000 rows. Use the following calculation to determine the minimum