dbfeeds .

Prepared by: Rajesh NarayananDate: 31/December/2004Reference:Version: Draft

dbFEEDS Project

Performance Tuning Guidelines– Informatica PowerCenter

of 25

Project: db FEEDSStage: Release 1Phase: DesignWork stream: InterfacesDocument: Performance Tuning GuideSubject: Informatica – Performance tuning

dbFEEDS Project


Version History:

Version Date By Changes

0.1 28/December RN First draft0.2 29/December RN Added UNIX portion (section 5)0.3 07/Jan RN Merged section2.2/2.3 from development

standards.

Contributors:

Name Role Location Remarks

Approval:

Name Role Location Remarks

Sanjay De CostaJoseph Houben

Reference Documents:

Name Author Version Date

Unix system (OS) tuning Informatica

of 25

dbFEEDS Project


Contents

1 DOCUMENT DESCRIPTION............................................................................................................5

2 DOCUMENT ORGANISATION........................................................................................................5

3 INFORMATICA PC PRIMARY GUIDELINES..............................................................................5

3.1 DATABASE UTILISATION..................................................................................................................5

3.2 LOCALISATION.................................................................................................................................5

3.3 REMOVAL OF DATABASE DRIVEN SEQUENCE GENERATORS...........................................................5

3.4 SWITCH OFF THE “COLLECT PERFORMANCE STATISTICS”...............................................................6

3.5 SWITCH OFF THE VERBOSE LOGGING...............................................................................................6

3.6 UTILISE STAGING.............................................................................................................................6

3.7 ELIMINATE NON-CACHED LOOKUPS.................................................................................................6

3.8 TUNE THE DATABASE.......................................................................................................................7

3.9 AVAILABILITY OF SWAP & TEMP SPACE ON PMSERVER..........................................................7

3.10 SESSION SETTINGS.......................................................................................................................7

3.11 REMOVE ALL OTHER APPLICATIONS ON PMSERVER...................................................................8

3.12 REMOVAL EXTERNAL REGISTERED MODULES..............................................................................8

4 INFORMATICA PC ADVANCED GUIDELINES..........................................................................8

4.1 FILTER EXPRESSIONS :.....................................................................................................................8

4.2 REMOVE DEFAULT’S:......................................................................................................................9

4.3 OUTPUT PORT INSTEAD OF VARIABLE PORT....................................................................................9

4.4 DATATYPE CONVERSION:.................................................................................................................9

4.5 STRING FUNCTIONS:........................................................................................................................9

4.6 IIF CONDITIONS CAVEAT...............................................................................................................10

4.7 EXPRESSIONS.................................................................................................................................10

4.8 UPDATE EXPRESSIONS FOR SESSION :............................................................................................10

4.9 MULTIPLE TARGETS / SOURCES ARE TOO SLOW :..........................................................................10

4.10 AGGREGATOR.............................................................................................................................10

4.11 JOINER........................................................................................................................................11

4.12 LOOKUPS....................................................................................................................................11

4.12.1 Lookups & Aggregators Fight...............................................................................................13

4.13 MAPLETS FOR COMPLEX LOGIC.................................................................................................13

4.14 DATABASE IPC SETTINGS & PRIORITIES...................................................................................13

4.15 LOADING....................................................................................................................................13

4.16 MEMORY SETTINGS....................................................................................................................14

4.17 REDUCE NUMBER OF OBJETS IN A MAP...................................................................................14

4.18 SLOW SOURCES - FLAT FILES....................................................................................................14

4.19 BREAK THE MAPPINGS OUT........................................................................................................15

4.19.1 Keep the mappings as simple as possible..............................................................................15

4.20 READER/TRANSFORMER/WRITER THREADS AFFECT THE TIMING.................................16

of 25

dbFEEDS Project


4.21 SORTING – PERFORMANCE ISSUES..............................................................................................16

4.21.1 Sorted Input Conditions.........................................................................................................16

4.21.2 Pre-Sorting Data....................................................................................................................17

4.22 WORKFLOW MANAGER..............................................................................................................17

4.22.1 Monitoring and Running a Session:......................................................................................18

4.22.2 Informatica suggests that each session takes roughly 1 to 1 1/2 CPU's...............................18

4.22.3 Place some good server load monitoring tools on the PM Server in development...............18

4.23 CHANGE DATABASE PRIORITIES FOR THE PMSERVER DATABASE USER..................................18

4.24 CHANGE THE UNIX USER PRIORITY...........................................................................................19

4.25 TRY NOT TO LOAD ACROSS THE NETWORK................................................................................19

4.26 BALANCE BETWEEN INFORMATICA AND THE POWER OF SQL AND THE DATABASE..................19

5 PERFORMANCE TUNING THE UNIX OS...................................................................................19

5.1 PROCESS CHECK.............................................................................................................................19

5.2 IDENTIFIYING & RESOLVING MEMORY PROBLEM..........................................................................20

5.3 IDENTIFYING AND RESOLVING DISK I/O ISSUES...........................................................................21

5.4 IDENTIFYING AND RESOLVING CPU OVERLOAD ISSUES...............................................................22

5.5 IDENTIFYING AND RESOLVING NETWORK ISSUES.........................................................................22

of 25

dbFEEDS Project


1 Document Description

This document describes the practices that can be followed by the ETL development

team, in order to get the best of Informatica PowerCenter (ETL). This document

mainly concentrates on optimising the performance of core ETL. In order to make

ETL to achieve the optimal performance; it is imperative to strike a good balance in

hardware, OS, RDBMS & Informatica PowerCenter 7.1.1. This document can be

used as reference by the development team & administration team.

2 Document Organisation

This document is divided into following parts

o Primary guidelines - Necessary for ETL to perform optimally,

fundamental approach for ETL design with Informatica PC 7.1.1

o Advanced guidelines - Guidelines can be applied on case-to-case basis,

Can be followed based on the problem scenario / environment

o Optimising Unix system – Performance tuning the OS (Unix/Linux

system)

3 Informatica PC Primary guidelines

3.1 Database utilisation

Utilise database for significant data handling operations, staging tables can be a

real benefit for parallelism in operations. It reduces the amount of processing time

by a significant amount.

3.2 Localisation

Try to localise the relational objects as far as possible. Try not to use synonyms for

remote database. Usage of remote links for data processing & loading certainly

slow the things down.

3.3 Removal of Database driven Sequence Generators

Usage of database oriented sequence generators proves to be a costly decision.

As this requires wrapper function / store procedure call, which might degrade the

performance by 3 times. Also it is not so easy to determine the bottleneck caused

by database sequence generators. If it is must to use database sequence

generators, then have a shared sequence generator & build a staging table from

of 25

dbFEEDS Project


the flat file, add a SEQ_ID column & call a POST TARGET LOAD procedure to

populate this column.

This requires a wrapper function / stored procedure call. Utilizing these stored

procedures has caused performance to drop by a factor of 3 times. This slowness is

not easily debugged - it can only be spotted in the Write Throughput column. Copy

the map, replace the stored proc call with an internal sequence generator for a test

run - this is how fast you COULD run your map. If we must use a database

generated sequence number, then follow the instructions for the staging table

usage. If we're dealing with GIG's or Terabytes of information - this should save you

lot's of hours tuning. IF YOU MUST - have a shared sequence generator, then build

a staging table from the flat file, add a SEQUENCE ID column, and call a POST

TARGET LOAD stored procedure to populate that column. Place the post target

load procedure in to the flat file to staging table load map. A single call to inside the

database, followed by a batch operation to assign sequences is the fastest method

for utilizing shared sequence generators.

3.4 Switch off the “Collect performance statistics”

This has an impact though it is minimal; removing this operation reduces reliance

on the flat file operations. However it may be useful to have this option switch ON

during tuning exercise.

3.5 Switch off the verbose logging

The session log has a tremendous impact on the overall performance of a session.

Over-ride the session log to NORMAL logging mode. In informatica logging

mechanism is not parallel; it is embedded into the operations. Also, this prevents

informatica metadata table from growing. Also, it is good idea to perform some

amount of automated house keeping which truncates the log from Informatica

metadata at regular intervals.

3.6 Utilise staging

If the source is flat file utilise a staging table. This way you can use SQL Loader,

Bulk-Load utility. Keep the basic logic in source load map; eliminate all lookups

from the code. At this juncture if the reader is slow, then check for following

o If there is an item in configuration file which sets a value to throttle reader,

it will limit the read throughput.

o Move flat file to local disk; don’t read from network or from RAID.

of 25

dbFEEDS Project


3.7 Eliminate non-cached lookups

Usage of non-cached lookups will hamper the performance significantly. Especially

if the lookup table is “growing” or “updated” target table. This show the indexes are

changing during the operation and optimizer looses the track of index & its

statistics. If possible use staging table - this allows using joiner also which can

increase the performance to large extent.

3.8 Tune the database

Estimate for small, medium and large source data set sizes, in terms of number of

rows / average bytes per row. Also estimate the throughput for each and

turnaround time for load. DBA should be provided with this information, along with

tables that are expected to be high read / write .DBA should assign the right table

to the right disk space that could make difference.

3.9 Availability of SWAP & TEMP space on PMSERVER

Having less disk space for SWAP & TEMP could potentially slow down the

performance of entire server. To monitor this one need to watch the disk space

while sessions are running. Without monitoring, it would be difficult to assess the

reason, especially if mapping contains Aggregates, or lookups that uses disk

cache or Joiner with heterogeneous sources.

3.10 Session Settings

Major chunk of tuning can be done in session. By switching on the “Collect

performance statistics” one will come to know the parameters to be set in session

level, or at least what has to be changed in database. Basically one should try to

achieve OPTIMAL READ, OPTIMAL THROUGHPUT and OPTIMAL WRITE. Over-

tuning one of these pieces can ultimately slow down the sessions.

Index Cache and Data cache are dynamically allocated first. As soon as the

session is initialised, the memory for data and index caches are setup. Their sizes

depend upon session settings

The Reader DTM also based on dynamic allocation algorithm, it uses the memory

available in chunks. Size of the chunk would be determined by the session setting

“Default Buffer block size”

of 25

dbFEEDS Project


Read the session throughput, then tune for the reader, see what the settings are,

and send the write output to a flat file for less contention. Check the Throttle reader

setting; increase the default buffer size by a factor of 64K each shot. If the reader

still appears to increase during the session, then stabilize, and then try increasing

Shared Session Memory from 12 MB to 24 MB. Check the writer throughput

performance statistics to make sure there is NO writer bottleneck. If you have slow

writer, change the map to single target table at a time to see which target is

causing the slowness and tune it.

NOTE: if the reader session to flat file just doesn't ever "get fast", then we have got

some basic map tuning to do. Try to merge expression objects, set the lookups to

unconnected (for re-use if possible), check the Index and Data cache settings if we

have aggregation, or lookups being performed. Etc... If we have a slow writer,

change the map to a single target table at a time - see which target is causing the

"slowness" and tune it. Make copies of the original map, and break down the

copies. Once the "slower" of the N targets is discovered, talk to DBA about

partitioning the table, updating statistics, removing indexes during load, etc...

There are many database things you can do here.

Remember the TIMING is affected by READER/TRANSFORMER/WRITER

threads. With complex mappings, don't forget that each ELEMENT (field) must be

weighed - in this light a firm understanding of how to read performance statistics

generated by Informatica becomes important. In other words - if the reader is

slow, then the rest of the threads suffer, if the writer is slow, same effect. A pipe is

only as big as its smallest diameter.... A chain is only as strong as its weakest

link. Sorry for the metaphors, but it should make sense.

3.11 Remove all other applications on PMServer

Except the database staging, PMServer plays well with RDBMS & its engine, but

doesn’t play well with application servers, in particularly JAVA Virtual Machines,

Web Servers, Security Servers, applications and Report Servers. All of these items

should be broken out to other machines; this is critical to improve performance on

PMServer machine.

3.12 Removal external registered modules

of 25

dbFEEDS Project


As far as possible, try to avoid the API’s which calls external objects, as this has

been proven slow. External modules might exhibit speed problems, instead try

using pre-processing / post processing with SED, AWK or GREP.

4 Informatica PC Advanced guidelines

4.1 Filter Expressions :

Create the filter (TRUE / FALSE) inside the port expression upstream. Complex

filter expressions slow down the mapping. However it acts faster in Expression

transformation with an output port for the result. Place the expression in

EXPRESSION Transformation upstream from filter. Compute a single numerical

flag: 1 for TRUE 0 for FALSE as output port. Push this data into the filter. This will

have positive impact on performance.

Use the Filter transformation early in the mapping.

To maximize session performance, keep the Filter transformation as close as

possible to the sources in the mapping. Rather than passing rows that you plan to

discard through the mapping, you can filter out unwanted data early in the flow of

data from sources to targets.

Use the Source Qualifier to filter

The Source Qualifier transformation provides an alternate way to filter rows. Rather

than filtering rows from within a mapping, the Source Qualifier transformation filters

rows when read from a source. The main difference is that the source qualifier limits

the row set extracted from a source, while the Filter transformation limits the row set

sent to a target. Since a source qualifier reduces the number of rows used

throughout the mapping, it provides better performance.

4.2 Remove Default’s:

Having a default value including “ERROR” slows down the session. It causes

unnecessary evaluation of values for every data element in the mapping. Best

method of allotting default value is to have variable in expression, which returns the

expected value on the condition. This will be faster than assigning default value.

4.3 Output port instead of Variable port

Variables are good for static and state driven, but slow down the performance time

as they are allocated each time a row passes through expression object. Try to use

Output port instead of variable port.

of 25

dbFEEDS Project


4.4 Datatype conversion:

Avoid performing implicit conversion of datatypes by connecting an Integer to string

or vice versa. Instead use the function that converts the data explicitly, this avoids

PMServer to decide on datatype conversion at run time.

4.5 String Functions:

String functions are costly on performance. E.g. ltrim, rtrim etc., as there involves

allocate & re-allocate of memory within READER thread. Also it would be imperative

to perform the string operations on the data, in which case following can be

considered.

Use varchar/varchar2 datatypes in database sources, if source is file then make it

delimited one. Try to use LTRIM/RTRIM functions on the data coming in from a

database SQL; this would be much faster than performing in ETL.

4.6 IIF Conditions caveat

As far as possible, make a logic that goes away from IIF, as IIF conditions are costly

in any language. IIF creates multiple path logic inside the application & uses the

decision to navigate. This might have an implication on performance as well. Other

option is to use Oracle DECODE in source qualifier.

4.7 Expressions

Expressions like IS_SPACES, ISNUMBER etc. affects the performance, as this is

the data validation expression that has to scan the entire string to determine the

result. Try to avoid using these expressions unless there is absolute requirement for

its usage.

4.8 Update Expressions for session :

In session if the option Update Else Insert is ON, then definitely performance will

slow down. As, Informatica has to performs 2 operations for each rows update w.r.t

PK, then if it returns 0 rows then perform Insert. As an alternative, Update Strategy

can be used where rows would be marked using DD_UPDATE or DD_INSERT

inside the mapping. In this case session settings can be INSERT & UPDATE AS

UPDATE or UPDATE AS INSERT.

4.9 Multiple targets / sources are too slow :

Mappings with Multiple targets can eat up the performance some time. If the

architecture permits then make one map per target. If the sources are from different

of 25

dbFEEDS Project


ftp locations & they are flat file, then ideal choice would be FTPing the file to source

to the ETL server & then process it.

4.10 Aggregator

If the mapping contains more than one aggregators, then the session will run slow,

unless the cache dir is fast & disk drive access speed is high. Placing aggregator

towards the end might be another option; however this will also bring down the

performance. As all the I/O activity would be a bottleneck in informatica.

Maplets are good source for replicating data logic, but if a maplet contains

aggregator still the performance of the mapping (that contains maplet) will affect.

Reduce the number of aggregators in the entire mapping to 1(if can), if possible,

split the mapping to several mappings for breaking down the logics.

Sorted input to aggregator will increase the performance to large extent, however if

the sorted input is enabled & the data passing to aggregator is not sorted, Session

will fail. Set the cache size to calculated amount using below mentioned formulae.

Index size = (sum of column size in group-by ports + 17) X number of groups

Data size = (sum of column size of output ports + 7) X number of groups

4.11 Joiner

Perform joins in a database. Performing a join in a database is faster than

performing a join in the session.

Use one of the following options:

o Create a pre-session stored procedure to join the tables in a database.

o Use the Source Qualifier transformation to perform the join.

Designate as the master source the source with the smaller number of records.

For optimal performance and disk storage, designate the master source as the

source with the lower number of rows. With a smaller master source, the data cache

is smaller, and the search time is shorter. Set the cache size to calculated amount

using below mentioned formulae.

Index size = (sum of master column size in join condition +16) X number of rows in

master table

Data size = (sum of master column size NOT in join condition but on output ports +

8) x number of rows in master table

of 25

dbFEEDS Project


4.12 Lookups

When caching is enabled, the PowerCenter Server caches the lookup table and

queries the lookup cache during the session. When this option is not enabled, the

PowerCenter Server queries the lookup table on a row-by-row basis.

Eliminate too many lookups. More the lookups means, the DTM

reader/writer/Transform threads are not left with enough memory to be able to run

efficiently (as it can). With too many lookups one need to trade in Memory

contention for Disk contention. The memory contention might be worse than disk

contention, as the OS ends up swapping in and out of TEMP/SWAP disk space,

with small block sizes to try to locate the lookup row, and as the row goes from

lookup to lookup, swapping becomes worse.

Both lookups and aggregators require memory space & each of them requires Index

& Data cache, ideally they share from the same heap segments. Hence, care should

be taken while designing mapping that consumes the memory.

In the case where a lookup uses more than one lookup condition, set the conditions

with an equal sign first in order to optimize lookup performance. In the case of a

cached lookup, an ORDER BY condition is issued in the SQL statement used to

create the cache. Columns used in the ORDER BY condition should be indexed.

The session log will contain the ORDER BY statement.

Tips on Caches:

Cache small lookup tables

Improve session performance by caching small lookup tables. The result of the

lookup query and processing is the same, whether or not you cache the lookup

table.

Use a persistent lookup cache for static lookup tables:

If the lookup table does not change between sessions, configure the Lookup

transformation to use a persistent lookup cache. The Informatica Server then saves

and reuses cache files from session to session, eliminating the time required to read

the lookup table. If the lookup table does not change between sessions, configure

the Lookup transformation to use a persistent lookup cache. The Informatica Server

then saves and reuses cache files from session to session, eliminating the time

required to read the lookup table.

Override the ORDER BY statement for cached lookups:

By default, the Informatica Server generates an ORDER BY statement for a cached

lookup that contains all lookup ports. To increase performance, you can suppress

of 25

dbFEEDS Project


the default ORDER BY statement and enter an override ORDER BY with fewer

columns

Place conditions with an equality operator (=) first.

If a Lookup transformation specifies several conditions, you can improve lookup

performance by placing all the conditions that use the equality operator first in the

list of conditions that appear under the Condition tab

Consider following for calculating caches for lookup

Attributes Method

Minimum Index Cache 200 * [( S column size) + 16] over all condition ports

Maximum Index Cache # rows in lookup table [(S column size) + 16] * 2 over all condition ports

Minimum Data Cache # rows in lookup table [(S column size) + 8] over all outputports (not condition port)

Maximum Data Cache 2 * minimum data cache

4.12.1 Lookups & Aggregators Fight.

The lookups and the aggregators fight for memory space as discussed above. Each

requires Index Cache, and Data Cache and they "share" the same HEAP segments

inside the core. Particularly in the 4.6 / 1.6 products and prior - these memory areas

become critical, and when dealing with many rows - the session is almost certain to

cause the server to "thrash" memory in and out of the OS Swap space. If possible,

separate the maps - perform the lookups in the first section of the maps, position the

data in an intermediate target table - then a second map reads the target table and

performs the aggregation (also provides the option for a group by to be done within the

database)... Another speed improvement...

4.13 Maplets for complex Logic

It’s good idea to break the complex logic into maplets. This allows managing the

mapping in much better & efficient way of breaking down the business logics.

Always remember shorter the distance between source and Target, better the

performance.

With complex mappings READER,WRITER & TRANSFORM threads affects by

timing, i.e. if the reader is slow, then rest of the threads suffer,similarily f he writer is

slow same is the effect.

of 25

dbFEEDS Project


4.14 Database IPC settings & Priorities

If PMServer & Oracle instance are running on the same server, use IPC connection

instead of TCP/IP connection. Try to change the protocol in the TNSNames.ORA

and Listener.ORA files, and restart the listener on the server .However this can be

used only locally, at the same time the speed increases between 2x and 5x .Another

option one can think is prioritizing the database login that informatica uses to

execute its task. These tasks when login to the database would override others.

This would be particularly helpful in increasing the performance especially when

bulk loader or SQL Loader is used.

4.15 Loading

Make sure indexes and constraints are removed before loading into relational

targets & this can be created as soon as the load is completed. This would help in

boost up the performance in bulk data loads.

Lesser the commit interval more the time for session to complete, set the

appropriate commit interval, anything above 50K is good. Partioning the data while

loading is another wise option. Following are the partitions Informatica provides

o Key Range

o Hash Key

o Round Robin

o Pass Through

When partitioning the individual transformation it is advisable to go for the following

o Aggregator Cache Use hash auto key

o Lookup Cache Use hash auto key partition type, equality condition

o Sorter Cache Use Hash auto key or Pass-through or Key range

4.16 Memory Settings

Session Shared Memory Size controls the total amount of memory used to buffer

rows internally by the reader and writer. Set session shared memory between 12MB

and 25MB, remember increasing the shared memory beyond this doesn’t guarantee

increase in performance rather it does decrease the performance.

Buffer Block Size controls the size of the blocks that move in the pipeline. Set

shared buffer block size around 128K, This would be used by informatica for

handling block of rows.

of 25

dbFEEDS Project


If the server has RAM over 12 GIG’s, then shared memory can be increased

between 1 and 2 GIG’s. Also the shared buffer block size should set relative to

shared memory settings.

The Informatica Server moves data from sources to targets based on workflow and

mapping metadata stored in a repository. A workflow is a set of instructions that

describes how and when to run tasks related to extracting, transforming, and

loading data. The Informatica Server runs workflow tasks according to the

conditional links connecting the tasks.

4.17 Reduce Number of OBJETS in a map

Frequently, the idea of these tools is to make the "data translation map" as easy as

possible. All too often, that means creating "an" (1) expression for each

throughput/translation (taking it to an extreme of course). Each object adds

computational overhead to the session and timings may suffer. Sometimes if

performance is an issue / goal, integrate several expressions in to one expression

object, thus reducing the "object" overhead. In doing so – it could speed up the map.

4.18 Slow Sources - Flat Files

If we've got slow sources, and these sources are flat files, we can look at some of the

following possibilities. If the sources reside on a different machine, and we've opened a

named pipe to get them across the network - then we've opened (potentially) a can of

worms. We’ve introduced the network speed as a variable on the speed of the flat file

source. Try to compress the source file, FTP PUT it on the local machine (local to

PMServer), decompress it, and then utilize it as a source. If you're reaching across the

network to a relational table - and the session is pulling many rows (over 10,000) then

the source system itself may be slow. We may be better off using a source system

extract program to dump it to file first, and then follow the above instructions. However,

there is something your SA's and Network Ops folks could do (if necessary) - this is

covered in detail in the advanced section. They could backbone the two servers

together with a dedicated network line (no hubs, routers, or other items in between the

two machines). At the very least, they could put the two machines on the same sub-net.

Now, if the file is local to PMServer but is still slow, examine the location of the file

(which device is it on). If it's not on an INTERNAL DISK then it will be slower than if it

were on an internal disk (C drive for you folks on NT). This doesn't mean a UNIX file

LINK exists locally, and the file is remote - it means the actual file is local.

of 25

dbFEEDS Project


4.19 Break the mappings out

One per target. If necessary, 1 per source per target. Why does this work? Well -

eliminating multiple targets in a single mapping can greatly increase speed... Basically

it's like this: one session per map/target. Each session establishes its own database

connection. Because of the unique database connection, the DBMS server can now

handle the insert/update/delete requests in parallel against multiple targets. It also helps

to allow each session to be specified for its intended purpose (no longer mixing a data

driven session with INSERTS only to a single target). Each session can then be placed

in to a batch marked "CONCURRENT" if preferences allow. Once this is done,

parallelism of mappings and sessions become obvious. A study of parallel processing

has shown again and again, that the operations can be completed sometimes in half the

time of their original counterparts merely by streaming them at the same time. With

multiple targets in the same mapping, a single database connection to handle multiplies

diverse database statements - sometimes hitting this target, other times hitting that

target. Think - in this situation it's extremely difficult for Informatica (or any other tool for

that matter) to build BULK operations... even though "bulk" is specified in the session.

Remember that "BULK" means this is your preference, and that the tool will revert to

NORMAL load if it can't provide a BULK operation on a series of consecutive rows.

Obviously, data driven then forces the tool down several other layers of internal code

before the data actually can reach the database.

4.19.1 Keep the mappings as simple as possible

Bury complex logic (if you must) in to a maplet. If you can avoid complex logic all

together - then that would be the key. The old rule of thumb applies here (common

sense) the straighter the path between two points, the shorter the distance... Translated

as: the shorter the distance between the source qualifier and the target - the faster the

data loads

4.20 READER/TRANSFORMER/WRITER threads affect the TIMING

With complex mappings, don't forget that each ELEMENT (field) must be weighed - in

this light a firm understanding of how to read performance statistics generated by

Informatica becomes important. In other words - if the reader is slow, then the rest of

the threads suffer, if the writer is slow, same effect. A pipe is only as big as its smallest

diameter.... A chain is only as strong as its weakest link.

of 25

dbFEEDS Project


4.21 Sorting – performance issues

We can improve Aggregator transformation performance by using the Sorted Input

option. When the Sorted Input option is selected, the Informatica Server assumes all

data is sorted by group. As the Informatica Server reads rows for a group, it performs

aggregate calculations as it reads. When necessary, it stores group information in

memory. To use the Sorted Input option, we must pass sorted data to the Aggregator

transformation. We can gain added performance with sorted ports when we partition the

session.

When Sorted Input is not selected, the Informatica Server performs aggregate

calculations as it reads. However, since data is not sorted, the Informatica Server stores

data for each group until it reads the entire source to ensure all aggregate calculations

are accurate.

For example, one Aggregator has the STORE_ID and ITEM Group By ports, with the

Sorted Input option selected. When you pass the following data through the Aggregator,

the Informatica Server performs an aggregation for the three records in the 101/battery

group as soon as it finds the new group, 201/battery:

STORE_ID ITEM QTY PRICE

101 ‘battery’ 3 2.99

101 ‘battery’ 1 3.19

101 ‘battery’ 2 2.59

201 ‘battery’ 4 1.59

201 ‘battery’ 1 1.99

If you use the Sorted Input option and do not presort data correctly, the session fails.

4.21.1 Sorted Input Conditions

Do not use the Sorted Input option if any of the following conditions are true:

The aggregate expression uses nested aggregate functions.

The session uses incremental aggregation.

Input data is data-driven. You choose to treat source data as data driven in the

session properties, or the Update Strategy transformation appears before the

Aggregator transformation in the mapping.

If we use the Sorted Input option under these circumstances, the Informatica

Server reverts to default aggregate behaviour, reading all values before

performing aggregate calculations.

4.21.2 Pre-Sorting Data

To use the Sorted input option, you pass sorted data through the Aggregator.

Data must be sorted as follows:

of 25

dbFEEDS Project


By the Aggregator group by ports, in the order they appear in the Aggregator

transformation.

Using the same sort order configured for the session.

If data is not in strict ascending or descending order based on the session sort order, the

Informatica Server fails the session. For example, if you configure a session to use a

French sort order, data passing into the Aggregator transformation must be sorted using

the French sort order.

If the session uses file sources, you can use an external utility to sort file data before

starting the session. If the session uses relational sources, we can use the Number of

Sorted Ports option in the Source Qualifier transformation to sort group by columns in

the source database. Group By columns must be in the exact same order in both the

Aggregator and Source Qualifier transformations.

4.22 Workflow Manager

The Informatica Server moves data from sources to targets based on workflow and

mapping metadata stored in a repository. A workflow is a set of instructions that

describes how and when to run tasks related to extracting, transforming, and loading

data. The Informatica Server runs workflow tasks according to the conditional links

connecting the tasks.

Monitor, add, edit, delete Informatica Server info in the repository

Stop the informatica Server

Configure database, external loader, and FTP connections

Manage sessions and batches - create, edit, delete, copy/move within a folder,

start/stop, abort sessions, view session logs, details, and session performance

details.

Source Server Target

of 25

dbFEEDS Project


Source Data Transformed

data

Instructions from Metadata

`

Repository

4.22.1 Monitoring and Running a Session:

When the Informatica Server runs a Session task, the Workflow Monitor creates session

details that provide load statistics for each target in the mapping. We can view session

details when the session runs or after the session completes.

Create a worklet to reuse a set of workflow logic in several workflows. Use the Worklet

Designer to create and edit worklets.

4.22.2 Informatica suggests that each session takes roughly 1 to 1 1/2

CPU's.

In keeping with this - Informatica play's well with RDBMS engines on the same machine,

but does NOT get along (performance wise) with ANY other engine (reporting engine,

java engine, OLAP engine, java virtual machine, etc...)

4.22.3 Place some good server load monitoring tools on the PM Server in

development

Watch it closely to understand how the resources are being utilized, and where the hot

spots are. Try to follow the recommendations - it may mean upgrading the hardware to

achieve throughput. Look in to EMC' s disk storage array - while expensive, it appears

to be extremely fast, it may improve the performance in some cases by up to 50%

4.23 Change Database Priorities for the PMServer Database User

Prioritizing the database login that any of the connections use (setup in Server Manager)

can assist in changing the priority given to the Informatica executing tasks. These tasks

of 25

dbFEEDS Project


when logged in to the database then can over-ride others. Sizing memory for these

tasks (in shared global areas, and server settings) must be done if priorities are to be

changed. If BCP or SQL*Loader or some other bulk-load facility is utilized, these

priorities must also be set. This can greatly improve performance. Again, it's only

suggested as a last resort method, and doesn't substitute for tuning the database, or the

mapping processes. It should only be utilized when all other methods have been

exhausted (tuned). Keep in mind that this should only be relegated to the production

machines and only in certain instances where the Load cycle that Informatica is utilizing

is NOT impeding other users.

4.24 Change the Unix User Priority

In order to gain speed, the Informatica UNIX User must be given a higher priority. The

UNIX SA should understand what it takes to rank the UNIX logins, and grant priorities to

particular tasks. Or - simply have the pmserver executed under a super user (SU)

command; this will take care of reprioritizing Informatica's core process. This should

only be used as a last resort - once all other tuning avenues have been exhausted, or if

we have a dedicated UNIX machine on which Informatica is running.

4.25 Try not to load across the network

If at all possible, try to co-locate PMServer executable with a local database. Not having

the database local means: 1) the repository is across the network (slow), 2) the sources /

targets are across the network, also potentially slow. If we have to load across the

network, at least try to localize the repository on a database instance on the same

machine as the server. The other thing is: try to co-locate the two machines (pmserver

and Target database server) on the same sub-net, even the same hub if possible. This

eliminates unnecessary routing of packets all over the network. Having a localized

database also allows us to setup a target table locally - which you can then "dump"

following a load, ftp to the target server, and bulk-load in to the target table. This works

extremely well for situations where append or complete refresh is taking place.

4.26 Balance between Informatica and the power of SQL and the database

Try to utilize the DBMS for what it was built for: reading/writing/sorting/grouping/filtering

data en-masse. Use Informatica for the more complex logic, outside joins, data

integration, multiple source feeds, etc... The balancing act is difficult without DBA

knowledge. In order to achieve a balance, we must be able to recognize what

operations are best in the database, and which ones are best in Informatica. This does

not degrade from the use of the ETL tool, rather it enhances it - it's a MUST if you are

performance tuning for high-volume throughput.

of 25

dbFEEDS Project


5 Performance tuning the UNIX OS

5.1 Process check

ps-axu:

Run to check for the following items:

o Are there any processes waiting for disk access or for paging, if so check

the I/O and memory subsystems.

o Processes that are using most of the CPU and Processes are using most

of the memory. This may help you distribute the workload better.

5.2 Identifiying & resolving memory problem

Run vmstat S 5 confirms memory problems and check for the following:

o Pages-outs occurring consistently? If so, you are short of memory.

o Are there a high number of address translation faults? (System V only)

This suggests a memory shortage.

o Are swap-outs occurring consistently? If so, you are extremely short of

memory. Occasional swap-outs are normal; BSD systems swap-out

inactive jobs. Long bursts of swap-outs mean that active jobs are

probably falling victim and indicate extreme memory shortage. If you don’t

have vmstat -S, look at the w and de fields of vmstat. These should

ALWAYS be zero.

If memory seems to be the bottleneck of the system, try following remedial steps:

o Reduce the size of the buffer cache, if your system has one, by

decreasing BUFPAGES. The buffer cache is not used in system V.4 and

SunOS 4.X systems. Making the buffer cache smaller will hurt disk I/O

performance.

o If you have statically allocated STREAMS buffers, reduce the number of

large (2048- and 4096-byte) buffers. This may reduce network

performance, but netstat-m should give you an idea of how many buffers

you really need.

of 25

dbFEEDS Project


o Reduce the size of your kernels tables. This may limit the systems

capacity (number of files, number of processes, etc.).

o Try running jobs requiring a lot of memory at night. This may not help the

memory problems, but you may not care about them as much.

o Try running jobs requiring a lot of memory in a batch queue. If only one

memory-intensive job is running at a time, your system may perform

satisfactorily.

o Try to limit the time spent running sendmail, which is a memory hog.

o If you don’t see any significant improvement, add more memory.

5.3 Identifying and Resolving Disk I/O Issues

Use iostat to check i/o load and utilization, as well as CPU load. Iostat can be used

to monitor the I/O load on the disks on the UNIX server. Using iostat permits

monitoring the load on specific disks. Take notice of how fairly disk activity is

distributed among the system disks. If it is not, are the most active disks also the

fastest disks

Following might help to rectify the problem due to I/O

o Reorganize your file systems and disks to distribute I/O activity as evenly

as possible.

o Using symbolic links helps to keep the directory structure the same

throughout while still moving the data files that are causing I/O contention.

o Use your fastest disk drive and controller for your root file system; this will

almost certainly have the heaviest activity. Alternatively, if single-file

throughput is important, put performance-critical files into one file system

and use the fastest drive for that file system.

o Put performance-critical files on a file system with a large block size:

16KB or 32KB (BSD).

o Increase the size of the buffer cache by increasing BUFPAGES (BSD).

This may hurt your systems memory performance.

o Rebuild your file systems periodically to eliminate fragmentation (backup,

build a new file system, and restore).

o If you are using NFS and using remote files, look at your network

situation. You don’t have local disk I/O problems.

of 25

dbFEEDS Project


o Check memory statistics again by running vmstat 5 (sar-rwpg). If your

system is paging or swapping consistently, you have memory problems,

fix memory problem first. Swapping makes performance worse.

If system has disk capacity problem and is constantly running out of disk space, try the following actions:

o Write a find script that detects old core dumps, editor backup and auto-

save files, and other trash and deletes it automatically. Run the script

through croon.

o Use a smaller block size on file systems that are mostly small files (e.g.,

source code files, object modules, and small data files).

5.4 Identifying and Resolving CPU Overload Issues

Use sar u to check for CPU loading. This provides the %usr (user), %sys (system),

%wio (waiting on I/O), and %idle (% of idle time). A target goal should be %usr +

%sys= 80 and %wio = 10 leaving %idle at 10. If %wio is higher, the disk and I/O

contention should be investigated to eliminate I/O bottleneck on the UNIX server. If

the system shows a heavy load of %sys, and %usr has a high %idle, this is

indicative of memory and contention of swapping/paging problems. In this case, it is

necessary to make memory changes to reduce the load on the system server.

When you run iostat 5 above, also observe for CPU idle time. Is the idle time

always 0, without letup? It is good for the CPU to be busy, but if it is always busy

100 percent of the time, work must be piling up somewhere. These points to CPU

overload.

o Eliminate unnecessary daemon processes. rwhod and routed are

particularly likely to be performance problems, but any savings will help.

o Get users to run jobs at night with any queuing system that’s available

always for help. You may not care if the CPU (or the memory or I/O

system) is overloaded at night, provided the work is done in the morning.

o Use nice to lower the priority of CPU-bound jobs will improve interactive

performance. Also, using nice to raise the priority of CPU-bound jobs will

expedite them but will hurt interactive performance. In general though,

using nice is really only a temporary solution. If your workload grows, it

of 25

dbFEEDS Project


will soon become insufficient. Consider upgrading your system, replacing

it, or buying another system to share the load.

5.5 Identifying and Resolving Network Issues

One can suspect problems with network capacity or with data integrity if users

experience slow performance when they are using rlogin or when they are

accessing files via NFS.

Look at netsat-I. If the number of collisions is large, suspect an overloaded network.

If the number of input or output errors is large, suspect hardware problems. A large

number of input errors indicate problems somewhere on the network. A large

number of output errors suggest problems with your system and its interface to the

network.

If collisions and network hardware are not a problem, figure out which system

appears to be slow. Use spray to send a large burst of packets to the slow system. If

the number of dropped packets is large, the remote system most likely cannot

respond to incoming data fast enough. Look to see if there are CPU, memory or disk

I/O problems on the remote system. If not, the system may just not be able to

tolerate heavy network workloads. Try to reorganize the network so that this system

isn’t a file server.

A large number of dropped packets may also indicate data corruption. Run netstat-

s on the remote system, then spray the remote system from the local system and

run netstat-s again. If the increase of UDP socket full drops (as indicated by

netstat) is equal to or greater than the number of drop packets that spray reports,

the remote system is slow network server If the increase of socket full drops is less

than the number of dropped packets, look for network errors.

Run nfsstat and look at the client RPC data. If the retransfield is more than 5

percent of calls, the network or an NFS server is overloaded. If timeout is high, at

least one NFS server is overloaded, the network may be faulty, or one or more

servers may have crashed. If badmixis roughly equal to timeout, at least one NFS

server is overloaded. If timeout and retrans are high, but badxidis low, some part of

the network between the NFS client and server is overloaded and dropping packets.

Try to prevent users from running me /O- intensive programs across the network.

The greputility is a good example of an I/O intensive program. Instead, have users

log into the remote system to do their work.

Reorganize the computers and disks on your network so that as many users as

possible can do as much work as possible on a local system. Use systems with

good network performance as file servers. If you are short of STREAMS data

of 25

dbFEEDS Project


buffers and are running Sun OS 4.0 or System V.3 (or earlier), reconfigure the

kernel with more buffers.

of 25

dbfeeds .

Documents

Transcript of dbfeeds .