Using WSO2 ESB with SAP ERP (Retail)

by

Harsha SenanayakeHead of Enterprise Solutions

John Keells Holdings PLC

Agenda• John Keells Group / SAP ERP

• JKG : Enterprise Applications & Technology landscape / Vision

• KeellsSuper Business requirements

• Solutions Evaluated

• SAP Integration technologies

• Solutions Evaluated - Recommendation

• Solution Architecture Implemented

• Message flow and Enterprise Integration Patterns used

• Project team & plan

• Deployment architecture

• Lessons Learnt

• Q&A

John Keells Group

• Founded in 1870; Sri Lanka’s largest capitalized (1.6 billion USD ) company with shares listed internationally, with Global Depository Receipts (GDRs) listed on the Luxembourg Stock Exchange with a AAA+ credit rating

• Diversified into key sectors of the economy - Hospitality, Transportation and Logistics, Consumer Foods, Retail, Financial services, IT, Real estate and Property development and Plantation services

• 10,000+ employees

• Ranked among the “200 Best under a Billion” in Asia Pacific by Forbes

• Ranked the #1 Corporate in Sri Lanka by the Lanka Monthly Digest (LMD) 8 times in the recent past

• Signatory to the United Nations Global Compact

SAP ERP

• SAP (NYSE: SAP) is the market leader in enterprise application software with over 170,000 customers

• The company's best known products are its enterprise resource planning application (SAP ERP), SAP BusinessObjects software, and most recently and Sybase mobile products

• John Keells Holdings implemented SAP across its 70 subsidiaries in 2005 – the largest SAP implementation in Sri Lanka

JKG: Enterprise Applications & Technology

landscape

DW

Group ERP

ES

B

WCM ESS BI dashboards Composite apps Collab

Messaging

Standard contents

Search

Por

lets

(IT

S,

BS

P, W

eb

dyn

rpo

)N

ativ

e R

FC

(Ja

va E

SS

ap

ps)

cal

ls

DirectoryServices

Infrastructure – Servers, Storage, Network, Data Center services

CPM

ABAP - RFC data sourcesSAP Query

Industry specific LOB solutions

Infrastructure

ECM

Solution Lifecycle Management

MDM

Identity Management

& SSO

Business logic layer : ABAP, J2EE , .NET

Customer/Partners

EDI /ExtranetPortal

EDI / Web-services / WSRP

BE

x /

OD

BO

(O

LE D

B f

or

OL

AP

) M

DX

Fle

x /

IGS

/ X

ML

A (

MD

X)

SO

A g

over

nanc

e

Presentation layer :Enterprise Portal

Sync/Async Web ServicesXML/ IDOCS

Por

tlets

SAP GUI Rich /Web clients

Integration vision

[Gartner 2004 ] “An Enterprise Service Bus (ESB) is a new architecture that exploits

Web services, Message-oriented-middleware (MOM) Intelligent routing and transformation.

ESBs act as a light weight, ubiquitous integration backbone through which software services and application components flow”

Peace of mind : Once data is reliably delivered to the ESB, it is considered "safe" and out of the hands of the client

Enterprise service bus By David A. Chappell

Some of the Integration scenarios @ JKH

• Async Web services postings via SAP PI (EAI)

• AR/AP/GL DFinance_Document_POST -> XI IDOC->R/3

• Status retrieval by Polling with message GUID

• Sync web-service interfaces via SAP PI (EAI)

• Credit limit check WS -> BAPI -> R/3

• Banking Interfaces

• Custom ABAP programs

• Payment files – MT100, Encrypted propriety forms upload to bank sites

• Bank reconciliation - SWIFT MT940/ MultiCash

• Direct posting / reading of data via RPCs and WebAS Webservices

• RFC/BAPIs using .Net / DCOM / Jco connectors

• Companies – JMSL-POS, LMS-Sales app, BPOMate

• Direct upload of CSV files via FTP to SAP R/3 – AR/AP/GL Upload program ZF247

• ABAP BDC uploads

• Hotels

• PMS integration with Call accounting systems/ PBX, Door locking system, Minibar systems, In room internet systems, Voice

• CRS integration with GDS and OTAs

• Retail

• POS integration

• 3rd party settlement integration

• Transport and logistics

• EDI – CUSDEC

• SAP R/3 and RedPrairie WMS integration (XML<-> IDOC)

• Stock brokers

• Direct Market access via FIX

Group – SAP related Industry specific

KeellsSuper

• KeellsSuper is one of the leading supermarket chains in the country which has been in operation for the last 20 years

• KeellsSuper was Instrumental in popularising Modern trade in Sri Lanka

• Currently at 42 outlets and expanding

First Retailer to implement a leading ERP in Sri Lanka

Introduced the first Loyalty program in Retail with Integrated CRM and BI in Sri Lanka

Only online supermarket in Sri Lanka -KeellsSuper.com

Real-time stock taking using handheld scanners

Business requirement

• KeellsSuper has been running SAP ERP for 5 years and decided to migrate to SAP’s ERP Retail industry solution to benefit from the rich industry specific specific functionally available in SAP Retail ERP solution

Business Requirement

• Streamline the SAP and POS integration - Eliminate the performance issues and failures faced with the current SAP and POS integration

• Seamless integration with the online store – keellssuper.com.com

Business Requirements Required Integration scenarios

Direction Description

Article masters - new and changes

Condition masters - price changes

Vendor masters - new and changes

Customer master (credit customers)

Bonus Buy Conditions / Promotions

Download phys. inv. docs, upload phys. inv. count data

Credit l imits

Stock balances

Gift vouchers master data

Application acknowledgements

Financial transcations

Upload day-end closing POS

Upload sales data (compressed)

Inventory counts

Gift vouchers - issues

Inbound(POS - > Head offi ce)

Outbound(Head offi ce -> POS)

Business Requirements

• Guaranteed delivery - message queuing and guaranteed delivery semantics to ensure "unavailable" applications will get their data queued and delivered at a later time

• Unreliable and slow networks (specially in outstations), POS Server outages

• Quality of service (QOS) requirements

• Exactly Once In Order – E.g. Price changes

• Exactly Once – E.g. Sales data

• Prioritization of Messages

• Price changes Vs Article description change

• Better performance without impacting the ERP system

Solutions evaluated

1. Adopt the existing synchronous RFC based Point-to-Point integration solution developed using SAP DCOM / .Net Connector / Web services

2. Develop a Point-to-point integration solution based on SAP WS-RM enabled asynchronous Web services

3. Develop an IDOC based Point-to-point integration solution using SAP .Net connector

4. Implement an IDOC based ESB integration solution using SAP Netweaver PI

5. Implement an IDOC based ESB integration solution using WSO2 ESB

SAP Integration Technologies

RDBMS

NCo(.net)

RFC

BAPI

IDOC

SAP ERPBusiness logic

(ABAP)

Application

JCo(Java)

IDOC EDIIDOC XML

RPC

SOAP / WS-RM

SAP Application Server

JDBC/ODBC/ADO

SOAP Processing

Synchronous Vs Asynchronous services

• With synchronous services , clients invoke a request on a service and then suspend their processing while they wait for a response from the provider. The client cannot perform another task until a response or confirmation is received from the provider.

• With asynchronous services , clients initiate a request to a service and then resume their processing without waiting for a response. The service handles the client request in the provider system and returns a response at some later point, at which time the client can retrieve the response and proceed with its processing.

SAP Integration Technologies RFC/BAPIs

• An RFC is a synchronous interface method that calls and executes predefined functions

• A BAPI is a a RFC-enabled function that has been developed by SAP E.g :

• BAPI_PO_CREATE

• BAPI_PO_GETDETAIL

• Synchronous RFC/BAPI calls consume SAP ERP dialog processes and slow network connections can really hog ERP system resources

SAP Integration Technologies Web services

• The ABAP Web Application Server (WebAS 6.20+) can provide existing functions (BAPIs, RFCs) as Web services

• Since SAP NetWeaver 7.0, SP 14, Web Services Reliable Messaging (WS-RM) has been introduced as part of the ABAP stack. WS-RM provides reliable delivery of asynchronous messages using the SOAP protocol

• SAP was an early adopter of SOA for its business applications

• http://esworkplace.sap.com : The ES Workplace is the central place to view consolidated information about all available Enterprise Services delivered by SAP.

SAP Integration TechnologiesIDOCs

• SAP Intermediate Documents (IDOCS) are EDI like documents that are asynchronous in nature.

• The actual TRFC call to submit the IDOC to SAP is performed synchronously and very quickly, but the actual business processing can happen at some later time defined in the SAP system

• Schedule Report RBDAPP01 as a Background Job to process the IDocs (see SAP Note 399271).

• The outbound result (E.g. sales order confirmation) can also happen at some later time – Message type ALE AUDIT

• IDOCS offer additional queuing and retry capabilities

• IDOCs can be triggered using posting routines and Change pointers

• E.g At the time of Creating Purchase Order using ME21N

IDOCs available for the required integration scenarios Direction Message Type Interface description

ARTMAS Article masters - new and changes

COND_A Condition masters - price changes

CREMAS Vendor masters - new and changes

DEBMAS Customer master (credit customers)

WPDBBY Bonus Buy Conditions / Promotions

WVINVE Download phys. inv. docs, upload phys. inv. count data

ZFI_CRED Credit l imits

ZMM_STOCK Stock balances

ZSDGV Gift vouchers master data

ALEAUD Application acknowledgements

WPUFIB Financial transcations

WPUTAB Upload day-end closing POS

WPUUMS Upload sales data (compressed)

WVINVE Inventory counts

ZSDGVRE Gift vouchers - issues

Inbound(POS - > Head offi ce)

Outbound(Head offi ce -> POS)

Receiving and Sending IDOCs using .NET and Java

1.package com.sap.conn.idoc.examples;3.import com.sap.conn.jco.*;4.import com.sap.conn.idoc.jco.*;5.import com.sap.conn.idoc.*;6.import java.io.*;8.public class IDocClientExample {

10.public static void main(String[] args) {12.try38.// see configuration file BCE.jcoDestination provided in the installation directory.39.JCoDestination destination=JCoDestinationManager.getDestination("BCE"); 40.IDocRepository iDocRepository = JCoIDoc.getIDocRepository(destination);41.String tid = destination.createTID();42.IDocFactory iDocFactory = JCoIDoc.getIDocFactory();44.// a) create new idoc45.IDocDocument doc = iDocFactory.createIDocDocument(iDocRepository, "MATMAS02");46.IDocSegment segment = doc.getRootSegment();47.segment = segment.addChild("E1MARAM");48.// and so on. See IDoc Specification .....49.JCoIDoc.send(doc, IDocFactory.IDOC_VERSION_DEFAULT, destination, tid);56.destination.confirmTID(tid);58.}59.catch(Exception e)60.{61.e.printStackTrace();62.}63.System.out.print("program end");64.}65.}

Java IDOC Client (Sender)

.NET IDOC Server (Receiver)

Why IDOCs based integration was better suited for our requirement

• Synchronous BAPIs/RFC/WS calls are resource intensive

• All required WS-RM asynchronous outbound services are not available

• Tried and tested technology used by almost all SAP ERP Retail customers

• Only the changed data can be transferred from SAP

• ‘POS Interface Monitor’ functionality available to monitor IDOCs

Solutions evaluated – Recommendation

1. Adopt the existing RFC based Point-to-Point integration solution developed using SAP DCOM / .Net Connector / Web services

2. Develop a Point-to-point integration solution based on WS-RM enabled asynchronous Web services

3. Develop an IDOC based Point-to-point integration solution using SAP .Net connector

4. Implement an IDOC based ESB integration solution using SAP Netweaver PI

5. Implement an IDOC based ESB integration solution using WSO2 ESB

Synchronous RFC/BAPI/Web service calls consume SAP ERP dialog processes and slow network connections can really hog resources

Point-to-point spaghetti!

All required WS-RM asynchronous outbound services were not available Not tried and tested by SAP Retail customers, Point-to-point spaghetti!

Made sense but writing code for IDOC parsing , transformations and handling reliable messaging would have been LOT of effort!

And why reinvent the wheel? Separately licensed - SAP provides POS integration content out of

the box Dual stack (Java+ABAP) architecture makes PI very complex

and heavy - Java only ESB was introduced with NW 7.3 Light weight, Open source, Met our key requirements, Local

support, SAP adapter was being developed, Joint go-to-market strategy

WSO2 team claimed Ebay processes 1 billion transactions a day…

Implemented Solution architecture

Asanka Abeysinghe

SAPReceiver

ESB POSSender

SQL

ç (1) Query Sales line-items by Polling• Update on message successfully

accepted by the ESB - (Update TRANFFERED_TO_ESB =“TRUE”)

ç (3) Post IDOC ‘WPUUMS01’

è (4) Technical ack - IDOC created successfully /failedmessages if reprocessed in SAP.

(2) Transform/Map to IDOC structure- WPUUMS01 has limit on max line items

per IDOC therefore will need to be split into multiple IDOCs

è (7) Update on Application ack - UPDATE record:

• ALEAUD message status and update field TRANSFFERED_SAP _APPLICATION_ACK = “TRUE” and update field TRANSFFERED

è (5) Update on Technical ack - UPDATE record :

• TRANFFERED_TO_SAP_TEC_ACK=“TRUE”

Polling ConsumerSpilter Message translator

(XSLT)

è (6) Application ack - (ALEAUD)(status, message) – Multiple ALEAUD messages if reprocessed in SAP.

Store and Forward

Content-Based Router

Content-Based Router

Message flow and Enterprise Integration Patterns usedExample Scenario : POS Sales Data interface to SAP

Notations from the book Enterprise Integration Patterns - Gregor Hohpe

Project team

Head of PMO

SAP Functional

team

SD-POS Consultant

FICO Consultant

MM Consultant

SAP Technical

Team

ABAP Consultant with IDOC

skills

POS Team

.NET and SQL Developers

WSO2 Team

Architect

Project Manager

ESB Consultants

DS Consultants

QA Team

Core Team

Process leads

Admin / Infra Team

Project Plan

Project PlanJun-10 Jul-10 Aug-10 Sep-10 Oct-10 Nov-10 Dec-10 Jan-11 Feb-11 Mar-11

Project Kick off 1st JuneERP

Blueprint Document POS integration scenariosFinalise IDOCs and fields to required

RealisationConfigure standard IDOCsEnhance/develop IDOCs

WMSBlueprint Realisation

POS DevelopmentPOS Integration

Solution architectureWSO2 SAP Adapter developmentSizing and Deployment architectureIntegration scenarios / use casesField mapping and specificationsTrainingDevelop Transformation scriptsQA / Stress test

GoLive and Post Golive phaseGolive preparationGoLive 29th FebPost Golive support

Deployment architecture - Sizing

• 500k transactions on an average month

• 1 million on seasonal months

• Avg Size of a message - 100kb – 5mb

Direction Message Type # of IDOCs on May 2011

ALEAUD 4,569

ARTMAS 75,189

COND_A 325,517

CREMAS 214

DEBMAS 19

WPDBBY 646

WVINVE 4,110

ZFI_CRED 301

ZMM_STOCK 11,592

ZSDGV 491

WPUFIB 5,058

WPUTAB 1,339

WPUUMS 8,565

WVINVE 33,866

ZSDGVRE 4,140

Outbound(Head offi ce -> POS)

Inbound(POS - > Head offi ce)

Deployment architecture

Production Server Configuration DR

Server 1:-CPU:- 4 Cores

RAM:- 8 GB

Storage :- 50 GB

Server 2: CPU:- 2 Cores

RAM:- 4 GB

Storage :- 140 GB

Development Server Configuration

CPU:- 4 Cores

RAM:- 8 GB

Storage :- 105 GB

Lessons learnt• Avoid this:

• Always revalidate requirements to avoid over engineering – E.g. Do you really need EOIO for all integration scenarios?

Lessons learnt

• What worked in the Lab with sophisticated load test tools can break in the real environment

• When you have multiple components (SAP ERP, POS, 50+ end points over unreliable networks) it’s difficult to simulate a real production environment

• Strike a balance between offshore and onsite model

• Have frequent project / steering committee meeting to ensure EVERYONE is on the same page to avoid surprises

Q&A