Hw09 Large Scale Transaction Analysis
-
Upload
cloudera-inc -
Category
Economy & Finance
-
view
3.419 -
download
6
Transcript of Hw09 Large Scale Transaction Analysis
![Page 1: Hw09 Large Scale Transaction Analysis](https://reader030.fdocuments.in/reader030/viewer/2022020218/55a537ca1a28ab362c8b46ce/html5/thumbnails/1.jpg)
Hadoop World: NYC 1
Hadoop@Visa
Joe Cunningham Technology Strategy and InnovationOctober 2, 2009
![Page 2: Hw09 Large Scale Transaction Analysis](https://reader030.fdocuments.in/reader030/viewer/2022020218/55a537ca1a28ab362c8b46ce/html5/thumbnails/2.jpg)
Hadoop World: NYC 2
Agenda
1 Visa
2 Value-Added Information Products
3 Hadoop@Visa – research results
VisaNet – an overview
![Page 3: Hw09 Large Scale Transaction Analysis](https://reader030.fdocuments.in/reader030/viewer/2022020218/55a537ca1a28ab362c8b46ce/html5/thumbnails/3.jpg)
Hadoop World: NYC 3
Keeping the Brand Promise
ZeroLiability
goSecurity
Flexibility
![Page 4: Hw09 Large Scale Transaction Analysis](https://reader030.fdocuments.in/reader030/viewer/2022020218/55a537ca1a28ab362c8b46ce/html5/thumbnails/4.jpg)
Hadoop World: NYC 4
ZeroLiability
goSecurity
Flexibility
Keeping the Brand Promise
3
GlobalAcceptance
Reliability
GuaranteedPayment
![Page 5: Hw09 Large Scale Transaction Analysis](https://reader030.fdocuments.in/reader030/viewer/2022020218/55a537ca1a28ab362c8b46ce/html5/thumbnails/5.jpg)
Hadoop World: NYC 5
VisaNet Today
fast secure flexible reliable scalablefast secure flexible reliable scalable
• VisaNet is the world’s largest, most advanced payments network.
28 millionacceptancelocations
130 millionauthorizationsper day
1,600endpoints
Processes transactions in 1 second
16,300financial institutions
Less than 2 secondsunavailability per year
Processes in 175 currencies
1.4 millionATMs
![Page 6: Hw09 Large Scale Transaction Analysis](https://reader030.fdocuments.in/reader030/viewer/2022020218/55a537ca1a28ab362c8b46ce/html5/thumbnails/6.jpg)
Hadoop World: NYC 6
1 VisaNet – an overview
2
3
Value-Added Information Products
Hadoop@Visa – research results
Agenda
![Page 7: Hw09 Large Scale Transaction Analysis](https://reader030.fdocuments.in/reader030/viewer/2022020218/55a537ca1a28ab362c8b46ce/html5/thumbnails/7.jpg)
Hadoop World: NYC 7
Visa Processing Architecture
Security/Access Services
Message File Web
VisaNet Services Integration
Common Data Infrastructure
Authorization Clearing & Settlement
InformationDisputes Risk
![Page 8: Hw09 Large Scale Transaction Analysis](https://reader030.fdocuments.in/reader030/viewer/2022020218/55a537ca1a28ab362c8b46ce/html5/thumbnails/8.jpg)
Hadoop World: NYC 8
Value-Added Information Products
Information Services
Client • Portfolio Analysis• Visa Incentive Network
Accountholder• Transaction Alerts• Account Updater• Tailored Rewards
Risk Management Services
Authentication • Advanced Auth• Verified by Visa• Address Verification
Account Monitoring• Fraud Detection• Compromised Accounts• Consumer Alerts
Encryption• Dynamic Key Exchange• Dynamic Cryptograms
![Page 9: Hw09 Large Scale Transaction Analysis](https://reader030.fdocuments.in/reader030/viewer/2022020218/55a537ca1a28ab362c8b46ce/html5/thumbnails/9.jpg)
Hadoop World: NYC 9
1 VisaNet – an overview
2 Value-Added Information Products
3 Hadoop@Visa – research results
Agenda
![Page 10: Hw09 Large Scale Transaction Analysis](https://reader030.fdocuments.in/reader030/viewer/2022020218/55a537ca1a28ab362c8b46ce/html5/thumbnails/10.jpg)
Hadoop World: NYC 10
Hadoop SystemsVM System
Research Lab Setup
Relational Database
Custom Analytic Stacks
Encryption Processing
Management Stack
Hadoop #2~300Tb / 28 nodes
Hadoop #1~40Tb / 42 node
![Page 11: Hw09 Large Scale Transaction Analysis](https://reader030.fdocuments.in/reader030/viewer/2022020218/55a537ca1a28ab362c8b46ce/html5/thumbnails/11.jpg)
Hadoop World: NYC 11
Risk Product Use CaseCreate critical data model elements, such as keys and transaction statistics, which feed our real-time risk-scoring systems
Input: Transactions
Merchant Category
Country / Zip
Output: Key & Statistics
MCCZIP KeyStatistics related to account, transaction type, approval, fraud, IP address, tx. counts
HadoopCluster
Research Sample (Synthetic Transactions)
• 500 million distinct accounts
• 100 million transactions per day
• 200 bytes per transaction
• 2 years: ~ 73 billion transactions, ~ 36 TB
• One Month 13 minutes!
![Page 12: Hw09 Large Scale Transaction Analysis](https://reader030.fdocuments.in/reader030/viewer/2022020218/55a537ca1a28ab362c8b46ce/html5/thumbnails/12.jpg)
Hadoop World: NYC 12
Financial Enterprise Fit
Some key questions we are researching…
• What will the Hadoop Solution Stack(s) look like?File System
Transaction Sample System
Relational Back-End
Analytics Processing
• Internal vs. External cloudHow do I get data into a cloud in a secure way?
• How does HSM and security integration work in Hadoop?What are the missing pieces?
![Page 13: Hw09 Large Scale Transaction Analysis](https://reader030.fdocuments.in/reader030/viewer/2022020218/55a537ca1a28ab362c8b46ce/html5/thumbnails/13.jpg)
Hadoop World: NYC 13
Why Hadoop@Visa?
fast secure flexible reliable scalablefast secure flexible reliable scalable
• For our information products, we see promise in combining Visa’s data analytics capability with the power of Hadoop
• Analyze volumes of data with response times that are not possible today…
… and apply analytic models to individual client, not just client segment
• But, need to answer questions relating to stack configuration and support…
… and, eventually must fit with our financial enterprise paradigm:
![Page 14: Hw09 Large Scale Transaction Analysis](https://reader030.fdocuments.in/reader030/viewer/2022020218/55a537ca1a28ab362c8b46ce/html5/thumbnails/14.jpg)
Hadoop World: NYC 14
Keeping the Brand Promise
ZeroLiability
goSecurity
Flexibility
![Page 15: Hw09 Large Scale Transaction Analysis](https://reader030.fdocuments.in/reader030/viewer/2022020218/55a537ca1a28ab362c8b46ce/html5/thumbnails/15.jpg)
Hadoop World: NYC 15
ZeroLiability
goSecurity
Flexibility
Keeping the Brand Promise
3
GlobalAcceptance
Reliability
GuaranteedPayment