Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1...
Transcript of Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1...
![Page 1: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/1.jpg)
1
SMA 5506Database Technology
Summer 2005
Session L18
Web Aggregators and Aggregation Research:
Prof. Stuart Madnick ( [email protected] )
© S. Madnick & M. Siegel, 2005
2
Multiple Perspectives
Strategy and Policy
Aggregators
Global Financial Account Aggregation
Application
eCommerce & Financial Services
Technology
Web wrapper & Context mediator
![Page 2: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/2.jpg)
2
3
Overview: Aggregation – Focus on Information Re-use and Re-purposing
Aggregator definition, types & examples
Emerging Global Aggregation
Enabling Technologies
New Emerging Aggregators
Strategy, Legal, Cultural, and Policy Issues
Summary
4
Aggregator Definition
Transparently collects information
with or without prior source arrangement
resolves contextual differences
Performs analysis
not merely gather
Represents user
acts as agent for user
![Page 3: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/3.jpg)
3
5
Aggregator Types & Examples
Investment analysis, policy analysis
Department calendars
Complementary
Financial account aggregation
Customer Relationship Management (CRM)
Relationship
Shop for best prices (e.g., books)
Benchmarking divisions
Comparison
ExternalInternalSources
Types
6
www.amazon.com
![Page 4: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/4.jpg)
4
7
www.books.com *
* Acquired by Barnes & Nobles
8
www.mysimon.com
![Page 5: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/5.jpg)
5
9
www.evenbetter.com *
* Now called DealTime.com Comment: Need for Mega-Aggregators
10
Mega - Aggregators
Note: multiple book price aggregators
• using differing sources
• giving differing results !
What about an aggregator of aggregators
• A Mega- Aggregator … more later …
Aggregators not just about prices ...
![Page 6: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/6.jpg)
6
11
www.usair.com
12
www.maxmiles.com
![Page 7: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/7.jpg)
7
13
What is MaxMiles
You give it all your frequent flyer info, it . . .
Does mileage management for you . . .
Gathers all your frequent flyer info
Sends a personalized report via email
Updates reports on web site daily
Searches for missing mileage credits
Analyzes most current milage offers
14
From US Airways site"Access to this information is subject to the rules in the Dividend Miles Membership Guide. In addition, by using this site to access your Dividend Miles account, you agree that you will use this site in a manner consistent with the Dividend Miles Membership Guide and you further agree not to allow access to this siteto any third party by revealing your access code to any third party for any reason. Failure to comply with the foregoing restrictions on the use of this site shall be grounds for the termination of your access to this site and/or your membership in the Dividend Miles program."
![Page 8: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/8.jpg)
8
15
“Power of Attorney” on MaxMiles Site
16
Real time Tibco
Complementary AggregatorIntegrated Spreadsheet Sources
InternalExternal
![Page 9: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/9.jpg)
9
17
Everyone is impacted:If you are not an Aggregator,
you are an Aggregatee
Books.com, MySimon.com, EvenBetter.comBarnes and Noble, Borders, Amazon, . . .
MaxMilesUnited, American, TWA, US Air, . . .
Universal Financial Aggregator (UFA)your banks, your brokers, your billers , . . .
NOTE: Aggregatees -> Aggregators
18
Overview
Aggregator definition, types & examples
Emerging Global Aggregation
Enabling Technologies
New Emerging Aggregators
Strategy, Legal, Cultural, and Policy Issues
Summary
![Page 10: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/10.jpg)
10
19
Global Comparison Aggregator
http://soursop.mit.edu/global
20
Best global online price
![Page 11: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/11.jpg)
11
21
Global Price Comparison
USMexico
BrazilUK
FranceGermany
SpainItaly
NetherlandsDenmark
NorwaySweden
Japan500
1000
1500
2000
Pric
e of
DC
R-I
P5
($)
N= 52 22 22 18 12 8 2 3 3 6 6 16 2
22
Mega-Aggregator: Using Regional Aggregators
US Sweden France UK
![Page 12: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/12.jpg)
12
23
Global Complementary AggregationQ: How did CO2 emissions
(total, per GDP, per capita) change over time
(between 1990 and 2000) in Yugoslavia?
User 1: YUG as a geographic region bounded before the breakupUser 2: YUG as a legal autonomous state
Related effort: - Laboratory for Information Globalization and Harmonization Technologies (LIGHT)
24
2.07162SVN
2.0608.7MKD
4.5266.9HRV
3.913.6BIH
10.61627.823.7698.3YUG
PopGDPPopGDPCountry
20001990
SITTolarSVNSlovenia
MKDDenarMKDMacedonia
HRKKunaHRVCroatia
BAMMarkaBIHBosnia and Herzegovia
YUNNew Yug. Dinar
YUGYugoslavia
CurCodeCurrencyCodeCountry
225.93SVNUSD
64.757MKDUSD
8.089HRVUSD
2.086BIHUSD
67.26710.5YUGUSD
20001990ToFrom
3981SVN
3378MKD
5405HRV
1279BIH
1548035604YUG
20001990Country
CO2 Emission
4560
282
1.28
104.8
29523
2000
2800
535
1.5
66.5
35604
1990
User 2User 1
11002800GDP/Capita
640535CO2/GDP
1.461.5CO2/capita
24.266.5GDP
1548035604CO2
20001990Country
GDP in billions local currency; Population in millions
In 1000 tons per year
Total CO2 in 1000 tons per year; GDP in billions USD; CO2/Capita in tons per person; CO2/GDP in tons per million USD; GDP/Capita in USD per person
World Bank’s World Dev. Indicator DB; UN Statistic Division; Statistics Bureaus
OAK Ridge’s CDIAC DB; WRI; GSSD; EPAs Olsen (Web)
Many sources needed:Meanings in sources & users might differ
![Page 13: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/13.jpg)
13
25
Overview
Aggregator definition, types & examples
Emerging Global Aggregation
Enabling Technologies
New Emerging Aggregators
Strategy, Legal, Cultural, and Policy Issues
Summary
26
Emerging Technologies
Web wrappers
Extract selected information from web
Allows web to be treated as large database
Context mediators
Resolve semantic (meaning) differences
Enable meaningful aggregation
& comparison
![Page 14: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/14.jpg)
14
27
Overview
Aggregator definition, types & examples
Emerging Global Aggregation
Enabling Technologies
New Emerging Aggregators
Strategy, Legal, Cultural, and Policy Issues
Summary
28
User
UFA
CitigroupCredit Cards
FidelityBrokerage
FleetBanking
BancOneCredit Cards
Concept of a Universal Financial Aggregator ( UFA )
![Page 15: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/15.jpg)
15
29
COntext INterchange Project (context2.mit.edu/coin)
30
Universal Financial Aggregator (UFA) Demonstration
![Page 16: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/16.jpg)
16
31
Aggregated Report produced ( top part )
32
Aggregated Report produced ( bottom part )
![Page 17: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/17.jpg)
17
33
“Aggregation . . . The ‘killer app’ in the e-finance revolution”
http://www.tfconferences.com/conferences/AAG01/index.html
34
Current UFA Offerings (examples)
Account Aggregation Technology Suppliers• Yodlee• VerticalOne• eBalance
Account Aggregation Offerings• Chase, Fleet, Citibank (myCiti.com), . . .• Merrill-Lynch, Fidelity Investments, . . .• Yahoo, AOL, . . .
![Page 18: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/18.jpg)
18
35
Aggregatee
Aggregatee
Aggregator
Commercial Financial Account Aggregators
36
FI #2
FI #3
FI #1
User
UFA
First Call
Excel
Advisors
Wizards
Agents
Account Analyst
Asset Allocation
Financial Sources
Other Internet Sources
Intranet Sourcesand applications
Real Value: “After Aggregation” Services
UserInformation
![Page 19: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/19.jpg)
19
37
Advisors: Stock holdings re-aggregate
38
Wizards: Money Market (MMA) example
![Page 20: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/20.jpg)
20
39
MMA is also an aggregator, of MM rate sources
Better rates
40
Multi- Mega- Aggregators !
The Universal Financial Aggregator (UFA) :
Aggregates your financial information
several ways: totals and stock holdings
Also uses the Money Market Aggregator
which draws on multiple MM sources
including sources that are, in fact,
aggregators of some MM sources
![Page 21: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/21.jpg)
21
41
UFA Goals
Consumers and Financial Institutions:
• Can do what they do better
• Can do things they could not do before
• Redefine the use, role, and relationship of financial service
Business-to-Business examples also
42
Overview
Aggregator definition, types & examples
Emerging Global Aggregation
Enabling Technologies
New Emerging Aggregators
Strategy, Legal, Cultural, and Policy Issues
Summary
![Page 22: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/22.jpg)
22
43
Strategy, Legal, Cultural, and Policy Issues Regarding Aggregation
Relationships between Aggregators and AggregateesProtection of investment in data vs re-use
EU Database Directive4 proposed bills in US
Protection of consumer privacyLiability, consumer protection, jurisdiction
44
www.intershipper.net
![Page 23: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/23.jpg)
23
45
www.iship.com
46
Aggregator “states” example
Independent Agent ( BITS, Inc. ]
Independent Aggregator
Intershipper
iShip
Dependent Aggregator/Dominant ( UPS ]
Dependent Aggregator / Consortium
Partial Collaborative Aggregator
![Page 24: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/24.jpg)
24
47
Database Re-Use Lawsuits and Claims
Example CaseseBay vs. Bidder’s Edge
HomeStore.com vs. Bargain Network
First Union vs. PayTrust
mySimon vs. Priceman
Register.com vs. Verio
Ticketmaster vs. Tickets.com
Principles Used• Copyright infringement
• Trespass to chattels
• Breach of contract
• Misappropriation
• False advertising• Violation of the Computer
Fraud and Abuse Act• . . .
48
“ Trespass to chattels ”eBay vs. Bidder’s Edge
Injunction on May 24, 2000
Ticketmaster vs. Tickets.com
“it is hard to see how entering a publicly available web site could be called a trespass”
“ Copyright infringement ”Protects original works of authorship
Usually not applicable to data re-use
![Page 25: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/25.jpg)
25
49
Battle over Database Re-Use -continued
New Paradigm – “sweat of the brow”Collections of Information Antipiracy Act, H.R.354 (1999) – Coble bill
Consumer and Investor Access to Information Act of 1999, H.R. Rep.1858 (1999) – Bliley bill
Two other failed bills in 1996, 1997
EU Database Directive
50
International Differences
Decentralized
Self- regulation
No DB protection law
Opt-out for privacy
Centralized
Specific, strict regulation
DB Directive
Opt-in for privacy
U.S. E.U.
Reciprocity provision• Not protected if home country does not have similar law for DB
protection
• No data flow to countries without similar privacy protection law
Harmonization is needed• Safe Harbor Agreement between US and EU
![Page 26: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/26.jpg)
26
51
Overview
Aggregator definition, types & examples
Emerging Global Aggregation
Enabling Technologies
New Emerging Aggregators
Strategy, Legal, Cultural, and Policy Issues
Summary
52
Key Global Aggregation Insights & ImplicationsInsights• Everyone impacted
• Aggs not limited to price comparison
• Aggs can be combined and aggregated
• Aggs create new info spaces
• After-agg even more important
• New technologies emerging
• Business models and market conditions continue to change
Implications• Be part of strategy planning
• Identify new Agg applications and business opportunities
• Entirely new types of applications can be defined
• Who will own this space
• This must be exploited
• Exploit these technologies
• Must be prepared for changes
![Page 27: Web Aggregators and Aggregation Researchweb.mit.edu › smadnick › www › SMA › 2005 › L18-v1 2005 - BW Web … · 2 3 Overview: Aggregation – Focus on Information Re-use](https://reader033.fdocuments.in/reader033/viewer/2022042310/5ed75d3210199002b75616fa/html5/thumbnails/27.jpg)
27
53
Current Related Projects ( partial list )• Aggregation strategies (e.g., Open aggregation)• Regional & Global Aggregation: Cultural & legal• Multi-enterprise eBusiness: Trust & relationships
• Financial services/Internet support for SME’s• Future Financial Services (e.g., Wireless aggregator)
• Context Mediation & Web data extraction *• Data Quality & What XML will and will not do• Internal & Industry data standards efforts• Other new technologies (e.g., Mobile aggregation)
* Patents awarded in 1999
54
For Further Information and ContactsMIT Research:
– http://context2.mit.edu and http://context2.mit.edu/aggregation
– http://web.mit.edu/tdqm
MIT Technology Licensed Software:– http://web.mit.edu/smadnick/www/patents.html
Email Contact: – Stuart Madnick at [email protected]– Michael Siegel at [email protected]
Research funded, in part, by Fleet Bank, Merrill Lynch, Suruga Bank, BancoSantander (BSCH), Citibank, DoD Advanced Projects Agency (DARPA), MITRE Corporation, PricewaterhouseCoopers, Singapore-MIT Alliance (SMA), MIT-Malaysia University of Science and Technology (MUST) Collaboration