Syntel_EAI_vs_ETL

download Syntel_EAI_vs_ETL

of 8

Transcript of Syntel_EAI_vs_ETL

  • 8/3/2019 Syntel_EAI_vs_ETL

    1/8

    A P P L I C A T I O N S

    A W h i t e P A P e r S e r i e S

    eAi And etL technoLogy hAve StrengthS

    And WeAkneSSeS ALike. there ArecLeAr boundArieS Around the tyPeS

    of APPLicAtion integrAtion ProjectS

    moSt APProPriAte for eAch technoLogy.

    EAI vs. ETL: Drawing Boundaries for Data Integration

  • 8/3/2019 Syntel_EAI_vs_ETL

    2/8

    Business enterprises invest millions of dollars to implemeand deliver Data Warehousing and Business Intelligence (BI) initiati

    that rely on consistent, accurate and reliable data. IT organizations in these enterprises

    must ensure that proper integration techniques are selected to address the data needs of the organiza

    Positioning a common enterprise-wide integration strategy with EAI is essential to establish a clear-

    cut partnership between business needs and IT solutions. Data integration, a function of ETL, is a

    prominent need as mediocre data at the foundation of any BI initiative fails to provide an accurate pi

    of the business. Thus the vital question: EAI or ETL?

    In this paper, we'll explore this question, comparing ETL against the data integration element of EA

    introduction

    cALL for eAi

    cALL for etL

    WhAt iS eAi?

    WhAt iS etL?

    comPAring eAi And etL

    diStinctive fActorS

    eAi vS. etL: A deciSion mAking guide

    drAWing boundArieS: eAi vS. etL

    the bottom Line

    2 0 0 7 S y n t e l , i n c .

    1

    2

    3

    4

    5

    6

    7

    8

    9

    eAi s. etl: Da Bdas Daa ia

  • 8/3/2019 Syntel_EAI_vs_ETL

    3/8

    1.

    cAll for eAi

    Most business activities involve multiple applica-tions and information sources; incompatibilitiesbetween these systems can cause delays and errorsthat prevent organization from achieving real-timebusiness. The key to increasing operational effi-ciency and maximizing the individual value of thesesystems is ensuring that they can communicate and

    interact in real time.Some of the challenges facing modernorganizations are:

    Giving the business complete, transparentaccess to informationEnabling seamless movement of informationfrom one application to another

    EAI, as a discipline, aims to alleviate many of theseproblems as well as create new paradigms for trulylean proactive organizations.

    2.

    cAll for etl

    ETL (Extract, Transform and Load) is the tech-nology with the focus for data integration, whetherin batch or real time for data stores/data ware-houses. It synchronizes data between diverse appli-cations and involves a lot more data manipulationthan simply moving data from point A to B. Thereis reconciliation, cross matching, de-duping, cleans-

    ing - all data-intensive tasks that lay the foundationfor facilitating analysis and reporting.

    These systems are no longer stand-alone and sepa-rate from operational processingthey are inte-grated with overall business processes.

    ETL is no longer nice to have, but is essential tosuccess.

    EAI, as a discipline, aims to createlean, proactive organizations.

    EAI LEvELS

    DATA-LEvEL EAI

    t aa-ll eAi qpls axa a lplappla aa ss saal xa, asa la (etL) qsa a pla aa was pl-s.

    MESSAgE-LEvEL EAI

    mssa-ll eAi aassa xa alpl applas s

    lal q sss.

    PrOCESS-LEvEL EAI

    Pss-ll eAi qs ssa-leAi la a w-lw aa apal p ssa lapal.

    Inteation Technoloies Wokin in Concet

    Figure 1. Example of integration

    technologies working together.

  • 8/3/2019 Syntel_EAI_vs_ETL

    4/8

    3.

    whAt iS eAi?

    Enteprise Application Integration is the process of

    aligning a businesss strategic vision with its infor-mation technology

    Enterprise Application Integration (EAI) solu-

    tions enable the automation of end-to-end business

    processes by coordinating sequences of tasks and

    resources (both systems and people) that perform

    them. EAI solutions support sophisticated excep-

    tion management and the dynamic modification of

    processes even when processes are underway.

    EAI involves developing a unified view of an

    enterprises business and its applications, seeing

    how existing applications fit into the new view, andthen devising ways to efficiently reuse what already

    exists while adding new applications and data.

    EAI provides packaged integration solutions

    to help the enterprise develop a consistent

    approach to integration for all applications.

    4.

    whAt iS etl?

    Extract, Transform and Load (ETL) provides data consolidation

    for building permanent databases used for analytics or reports,

    data federation for creating virtual dashboards or reports, and data

    propagation for the transfer of data between applications.

    These three database functions are combined into one tool to pull

    data out of source databases and place it into target databases.

    ETL is used to migrate data from one or more databases to others,

    to form data repositories, data marts, data warehouses and also to

    convert databases from one format or type to another.

    Extract - the process of reading data from source systems.

    Data can be extracted in schedule-driven pull mode or eventdriven push mode. Pull mode operation supports data consol-idation and is typically done in batch. Push mode operation isone online by propagating data changes to target data stores.Transform - the process of converting the extracted data fromits existing form into the format it needs to be in so that itcan be placed into other systems or databases. Transformationoccurs by using rules or lookup tables or by combining thedata with other data.Load - the process of creation and execution of workflows to

    write data into the target systems. Data loading may causea complete refresh of a target data store or may be done byupdating the target destination. Interfaces here include defacto standards like ODBC, JBDC, JMS, or application intefaces. Loads could be parallel, synchronized or sequenced;e.g., ETL tool support parallel execution which dramaticallyreduces response time for data-intensive operations on data

    warehouses/data stores.

    Figure 2.The EAI architecture has various layers that reflect an increasing level of matu

    rity in the integration environment with the overall enterprise application framework.

    EAI is the process of aligning a business's strategic vision with itsinformation technology

    Figure 3.The ETL process

  • 8/3/2019 Syntel_EAI_vs_ETL

    5/8

    The other services which form an integral part of the ETL

    framework are:

    Administration and Operation services - these services

    ensure effective utilization of resources in the data synchro-nization environment. They ensure effective administrationthrough job scheduling and tracking, metadata management,error recovery, etc.Transport services - the process of moving raw or trans-formed data from a source to a target system.Metadata services - Metadata is descriptive informationabout data and other structures, such as objects, businessrules, and processes that manipulate data. Metadata can begrouped into two categories:

    Technical metadata supports designers, developers, administratorsduring development, maintenance, and management of an informa-

    tion technology environment. It is the technical glue that links thetools, applications, and systems that together constitute a solution.Example of technical metadata: the schema design of a data ware-house is typically stored in a repository as metadata, which is usedto generate the scripts that build data warehouse tables.

    Business metadata, on the other hand, gives a clearer picture ofthe services of the enterprise environment to end-users. Examples ofbusiness metadata include: business requirements, timelines, busi-ness metrics, business process flows, and business terminology.Metadata authors enter information about the business applicationinto the metadata repository.

    5.

    compAring eAi AnD etl

    EAI tools are clearly most appropriate for process integration,

    which consists of multi-step business process management and

    real-time interactive processing when very large numbers of

    transactions are involved.

    ETL tools do not handle these processes well. ETL tools are not

    designed to handle discontinuous workflows, or to scale to mov-

    ing very large numbers of small transactional messages.

    EAI and ETL are not competing technologies. They each relyon the concept of a unified view and the definition of a mapping

    that allows data from many disparate sources to be projected

    onto that view.

    There are many situations where they can be used in conjunction

    with each other where ETL can act as a service to EAI. One of

    the main objectives of EAI is to provide transparent access to the

    wide range of applications that exist in an organization. An EAI-

    to-ETL interconnection could be built using a Web service or a

    message queue to give an ETL product access to this application

    data.

    Such an interconnection eliminates the need for ETL to develop

    point-to-point adapters for synchronizing applications data

    sources. EAI is focused on real-time processing, it can conse-

    quently act as a real-time event source or target by an ETL appli-

    cation.

    ETL tools allow developers to define

    ETL as Web services. These Web

    services can be invoked by EAI

    applications. This not only provides

    transformational power to the EAI

    environment, but also supports code and

    metadata reuse.

    In plain words, data integration (provid-

    ed by ETL) is a sub-set of process inte-

    gration (provided by EAI); a common

    functionality between ETL and EAI is

    data integration from disparate systems.

    It is important to note that data integra-

    tion using EAI is at a cost software,

    hardware, infrastructure, skills, licenses,

    heavy footprint.

    ETL is no longer "nice-to-have,"but is essential to success.

    AN EXAMPLE OF

    EAI/ETL OvErLAP

    epls s la plsa ps al-a sa pas awl ass.ts a as all as la a w.

    t aa pl s a ppaa a s alaps a a-a ss a.

    Sa s s a a ls ps a l a pl. h, eAs s as s w a -al as a etL ss p a a asas a .

    Data synchronization ETLInteractive processing (ETL

    or EAI)Multi-step processing EAI

    Batch and real-time data

    synchronization

    Point-to-point continuous

    processing

    Simple or no workflow

    BPM

    Multi-step process

    Figure 4. EAI/ETL overlap

  • 8/3/2019 Syntel_EAI_vs_ETL

    6/8

    6.

    DiStinctive fActorS

    Areas EAI ETL

    Definition Technology solution that enables systems tocommunicate

    Process designed by users to extract, transform, andload data from one or more sources to a target datarepository

    Performance

    Optimization

    System is aimed at reducing the response time for asingle user request or update

    System is aimed at reducing total time to create theunified historical record

    Integration Applications Data

    Focus Operational & Strategic Operational

    Business Case IT, e-business Better WorkflowData entry once

    Business IntelligenceDecision making

    Time Real Time Batch (moving to real time)

    Data Transactional-small Historical-enormous

    Metadata LimitedMessage metadata

    RichDimensional metadata

    Transformations Format orientedCode supported

    AnalyticJoinsAggregations

    Volume Single transactionsMessages/second (KB)

    Days or weeks of dataRecords per min (GB)

    Targets OLTP APICode supported

    Relational StructuresNative connectivityCodeless

    Extracts Data Using APIs Directly from database

    System Admin

    Involvement

    EAI requires no system administrator involvement.

    Once implemented, EAI is a technology solution thatis transparent to end users.

    ETL requires extensive system administrator

    involvement

    7.

    eAi vS. etl: A DeciSion mAking guiDe

    Syntel has developed a list of questions to help guide your orga-

    nization toward the best decision for the situation when decid-

    ing between EAI and ETL. This toolkit can be used as an aid

    to evaluate a project as process integration or a data integration

    project.

    Factors for consideration in the decision include:Costs of run-time processing and development.Proprietary nature of source or target systems. A situationwhere the source system can only be accessed via screenscrapping because the file layouts and key structures are

    part of package and source is not available. In such casesneither ETL nor EAI will work and a solution might haveto be developed on case to case basis.The state of data and load-time window available tomigrate data from source to target and vice versa which

    needs real-time movement of data.Complexity and mapping of source and target systems bydata elements and data quality in each system.Skills of staff relative to EAI and ETL tools.

    To determine if your solution should be EAI or ETL, answer the

    questions on the next page:

  • 8/3/2019 Syntel_EAI_vs_ETL

    7/8

    EAI s. ETL Decision-makin Toolkit

    YES NO

    Do you anticipate data coming from disparate target systemslying in silos that you need to integrate?

    Is your source data straight-forward and does it fit directly toyour target systems? (i.e. no data transformation required)

    Do you expect the tool to automatically analyze and executeoperations on your data?

    Is the migration a one-off event? (i.e. you do not anticipateadding additional systems in the future)

    In the event of a system or connection failure, do youexpect data rollback or data integrity checks to be executedautomatically?

    Do you have any logic involved or business decisions to bemade "on-the-fly" based on your source data?

    Do you have a large number of transactions to be completedand managed swiftly?

    Are you finished making EAI skill set and infrastructureinvestments?

    Do you need a workflow which will help streamline businessprocesses and decision-making?

    Do you anticipate future business growth, additional targetsystems, or business mergers which would require sharingthis data across systems?

    If you answered YES to the first four question

    the right choice for you is ETL.

    If the answer to the last 6 questions is YES,

    then an EAI tool is the solution for you. In th

    case, you should strongly consider bringing in

    enterprise architect to evaluate the possibility.

    The enterprise architect will ensure that the

    pieces of the wider puzzle fit together properly

    EAI ETL

    Reliability (guaranteeddelivery)Enables real-time businessdecisionsOut of box adapters for manyenterprise systems

    Metadata driven approachGUI tools for most tasks (littlecoding)Extremely efficient for largedata volumes

    High upfront costRelatively complex design

    patterns

    High upfront costsComplexity of tool

    Batch oriented

    Most suitable for real timedata needsHigh volume, low footprintdata exchangeMany consumers of the samedata

    Suitable for large volumes ofdataGenerally used to movedata between two or moredatabases/data repositories

    8.

    DrAwing BounDArieS: eAi vS. etl

    9.

    the Bottom line

    If data integration is the business pain point you

    are facing, the most effective solution will be ETL.

    However, if your real problem is process integration,

    you will be better off with an EAI implementation.

  • 8/3/2019 Syntel_EAI_vs_ETL

    8/8

    SYNTEL:ab

    s S l ' s w s a w w w . s l .

    SYNTEL

    525 e. B Ba, td ft, mi 48083

    248.619.3503

    [email protected]

    S ds s s ss gba 2000

    as. fdd 1980, S's ss

    ds Bpo, x aa d, aa,

    d , ad s aa a

    ss, as as -Bsss d ad a,

    ss ss, daa as, crm, ad erp.

    w axz s ss a s/-

    s gba D S, as

    x js a dd. S's ba aa as

    as a sa ad s a sd--a,

    bds, ad qa. w d a s d d a

    s a sass xs aza bsss

    as ad a a d as d

    aa d .