QuestDirect.org

101970 - Leveraging Integration Broker To Increase Availability And Throughput

QuestDirect.org

Top Reasons to Use IB10 Endorsed by Prof. Axe Emel at Whatsamatta U. 9 Dealing with a Broker makes you feel richer 8 Same initials as Irritable Bowel 7 It’s better than a Disintegration Broker 6 Less Prone to Inappropriate comments than App Package 5 One more thing to pad your resume with 4 It’s sure to impress the babes 3 Moderately Cool Sounding 2 Buzz words have hypnotic effect on management 1 A bad day with IB is better than a good day with COBOL

QuestDirect.org

IntroductionsSam PredApplications System EngineerLocated in MinneapolisCurrently supporting Benefits and 9.2 upgrade

Wayne FullerApplications System EngineerLocated in MinneapolisTechnology Lead for 9.2 Upgrade

QuestDirect.org

Agenda• About Wells Fargo• High Volume• High Availability• Messaging Architecture• When to Message?

QuestDirect.org

Wells Fargo Population• Wells Fargo & Company (NYSE: WFC) Wells Fargo & Company is a

diversified financial services company providing banking, insurance, investments, mortgage, and consumer and commercial finance through more than 9,000 stores and more than 12,000 ATMs and the Internet (wellsfargo.com) across North America and internationally serving over 70 million customers.

• We employ over 282,000 team members in more than 35 countries across our 80+ businesses

• Current US population consists of • 265,000 full-time employees• 16,000 part-time employees• 1800 flexible employees• 38,000 Managers

QuestDirect.org

High Volume• Alternative to traditional Batch processing• Concurrent Processing• Reduced Contention• Restartability• Business Logic Reuse

QuestDirect.org

High Volume• Annual Merit Increase Load

CompEx File

PS HRMS

AppEngine IB

JOB

CI

QuestDirect.org

High Volume• Annual Merit Increase Load

– 200,000+ via JOB_DATA CI.– Approx. 5 seconds per transaction.– 11.5 days to single thread.– Reduced to 14 hours via messaging.– Respond to errors as they occur.

QuestDirect.org

High Volume• HRA Compensation Category

PS HRMS

JOB

AppEngine

SavePost Change

IB HRA Cat

QuestDirect.org

High Volume• HRA Compensation Category• ~280,000 processed in 168 minutes

– 1,667/min or 28/sec• Multiple sources of input (batch/real time)

QuestDirect.org

High Volume• Other Uses

– Automated Benefits Reprocessing (single thread 3 to 4 hours, down to 30 minutes)

– Quarterly ABBR Calc– Reorgs (dept/company changes)– Acquisition Loads

QuestDirect.org

High Availability• Staff Management

PS HRMS

Staff Mgmt

IBAsynch CI POS

Data

CI JOB Data

Handler

Email

QuestDirect.org

High Availability• Staff Management

– 38,000 managers– 24/7 Access– Decouple request from fulfillment– Asynch to Synch (commit points)

QuestDirect.org

High Availability• Employee Self Service

– W4, Direct Deposit (future)– No need to lock Team Members out– Improve satisfaction levels

QuestDirect.org

Messaging Architecture• Two dedicated servers

– Runs only messaging, services and gateway, no application servers– Gateway load balanced, allows sync services to run balanced– Node load balanced through Gateway setup– 2 CPU/2.27 gHz 16 core machines with 24 GB of memory– Average daily use < 15% of capacity

• Method to use multi-queues– Delivered doesn’t work– Helps with contention and partitioning the work– Only with unordered queues

• ig.connection property in integration.properties– Maximum cached connections, connection time is costly when hit heavily with synchronous services– Default is 10, changed to 200

QuestDirect.org

Messaging ArchitectureF5 – Load Balancer

Server A

Async Master

Sync Domain 1

Sync Domain 2

Physical Gateway

Server B

Async Slave

Sync Domain 1

Sync Domain 2

Physical Gateway

QuestDirect.org

Messaging Architecture• Current

– Async 7 dedicated servers on master, only default goes to slave

– Sync 4 domains load balanced with 2 gateways, 10 appservs each

• Future– Sync stays the same– Slave processing takes place of dedicated servers, will try

using only default

QuestDirect.org

Some Numbers• Synchronous services

– TurboTax W2 service• 1,000 to 2,000 daily, avg. of .5 seconds. Utilized Datapower transformation

– Job Agent service• 35,000 daily, avg < .3 seconds.

• Asynchronous– Identity management message

• 5000 daily, < .13 seconds– Guaranteed email

• 35000 daily– Staff Management

• 7500 daily

QuestDirect.org

When to Message?• High Volume Concurrent Processing

– Ability to pause processing, scheduling flexibility.

• High Availability (decoupling)• Transactions from multiple sources

(internal/external/batch/online).• Error Handling

– Can respond to errors while transactions continue processing

QuestDirect.org

Questions??

QuestDirect.org

Wells Fargo SessionsSession Time Room

Termination Processing at Wells Fargo W 2:00 Streeter

Enterprise Integration Using Third Party Transformation

Th 8:00 Schiller

Wells Fargo Upload Process Th 9:00 Rosemont 3

Leveraging Integration Broker To Increase Availability

Th 10:30 Rosemont 1

PeopleSoft Total Cost of Ownership Roundtable Th 11:30 Directors B

Automating Benefits Reprocessing at Wells Fargo Th 2:30 Rosemont 3

QuestDirect.org

APRIL 7-11, 2014Sands Expo and Convention CenterLas Vegas, Nevada

Save the Date

QuestDirect.org/COLLABORATE

COLLABORATE 14- Quest Forum is THE source for PeopleSoft roadmaps & news. It matters where you register! All PeopleSoft education and events run through Quest