Building Data Integration Systems for the Web

Alon Halevy

Google

NSF Information Integration Workshop

April 22, 2010

Without (too much) Loss of Generality

Web Enterprise, Science projects, …

Information integration ≅ data management

A Few Principles

• Data management “in situ”– Data meaning is derived from its context– Manipulate data in its natural location

• Pay-as-you-go data management– Provide services before modeling is done– Data can be about any domain

• Collaboration should be built in– Query answering is only step the first step

Alex Labrinidis

@via Facebook

Structured Data & The Web

Discover

Manage,Analyze, Combine

ExtractPublish

Hard to query, visualize, combine data across organizations

Requires infrastructure, concerns about losing control

Hard to find structured data via search engines

Data is embedded in web page, behind forms

Outline

• Surfacing the Deep Web

• Searching tables on the surface Web

• Fusion Tables: a platform for data management on the Web.

What is the Deep Web?

store locationsused cars

radio stationspatents

recipes

• Deep = not accessible through general purpose search engines– Major gap in the coverage of search engines.

Tree Search

Amish quilts

Parking tickets in India

Horses

Solution Constraints

• Can’t design a solution that requires domain engineering– (unless you can make money in that

domain!)

• Boundaries between domains are fuzzy

• Solution needs to be integrated into general web search– Can’t assume special query syntax

Surfacing the Deep Web[Madhavan et al. VLDB 2008]

• Surfacing: – Find high-quality forms– Guess good queries to submit– Put the resulting HTML pages in the index

• ~3M sites, 50 languages, 700 domains.• 1000 queries per-second get results from the

deep web.• 400K forms served per day, 800K per week• Impact mostly on the long and heavy tail of

queries

Deep Web: The Future

• Still an opportunity to go deeper into the deep web:– E.g., map the user query into a form

submission.

• Key challenge: given a keyword query, map it to forms in any domain

• Understanding the meaning of forms is still hard (e.g. content, geo constraints).

Outline

Surfacing the Deep WebSearching tables on the surface Web

• Fusion Tables: a platform for data management on the Web.

Bad table

Vertical Tables

Sub-Header Rows

Winners of the Boston Marathon (but that’s nowhere in the table)

Schema Ok, but context is subtle (year = 2006)

WebTables: Exploring the Relational Web[Cafarella et al., VLDB 2008, WebDB 08]

• In corpus of 14B raw tables, we estimate 154M are “good” relations– Single-table databases; Schema = attr labels + types– Largest corpus of databases & schemas we know of

• The Webtables system:– Recovers good relations from crawl and enables search– Builds novel apps on the recovered data

(Web-scale) Schema Collection

name e-mail|email, phone|telephone, e-mail_address|email_address, date|last_modified

instructor course-title|title, day|days, course|course-#,course-name|course-title

elected candidate|name, presiding-officer|speaker

ab k|so, h|hits, avg|ba, name|player

sqft bath|baths, list|list-price, bed|beds, price|rent

With 2.6 million schemas you can do some very interesting things.

Synonym discovery

“KR”-Based Table Search [Wu, Madhavan, Miao, Pasca, Shen]

• Ideally, we describe every table:– Class of entities it contains– Properties being modeled– Context, quality, …

• Use Web-extracted knowledge bases– Extract isa-hierarchy using patterns:– “cities such as Paris and London”– “chemical elements including hydrogen and

oxygen”

Step 1: Find “Subject” of Table

Not always the left (or first non-number column)

Step 2: associate classes with subjectChemical elements

Most of the time, the class labels are not in attribute name

Leveraging Web-extracted Ontologies

• Given a query, e.g., (country, GDP)– Rank tables about countries that have GDP

somewhere in the schema. – Very high precision (~90%)

• Next challenge: understand binary properties and binary relationships.

• Domain specialization: – System should improve if given ontologies in a

particular domain.

25

Combine Search, Extraction, Cleaning and Integration

[Cafarella, Koussainova, H., VLDB 2009],

• Try to create a database of all“VLDB program committee members”

Outline

Surfacing the Deep WebSearching tables on the surface WebFusion Tables: a platform for data

management on the Web.

Data Management for the Web Era

• Integrate seamlessly with the Web:– Search, maps, …

• Easy to use:– Much broader user base, pay-as-you-go– Very simple data integration

• Provide incentives for sharing data

• Facilitate collaboration

Fusion Tables – our current attempt[Madhavan, Gonzalez, Langen, Shapley, Shen]

We store and leverage a large collection of tables.

Incentive

Incentive, Pay-..-Go

Coffee Production

Coffee Consumption

Seamless integration with other web tools

Toilet heat map…

Database functionality on map

Collaboration

Table Search

Show up in search results!

Data Integration

Merged Table

Carries attribution from both base tables. Owners maintain control of their own data.

Fine Grained Discussions

Example Uses of Fusion Tables

• Tracking potholes in Spain• Displaying bike routes (MTBGuru)• State of California statistics• Government data from data.gov• Data about voting locations in the USA• Brazilian beaches• Chicago homicides• Most requested pop songs by year

Conclusions

• Information integration “in situ”– Blur the boundary between structured and

unstructured data

• Combine search, extraction, cleaning and integration into a single experience

• Pay-as-you-go: introduce complexity as needed– Serve enterprises without IT depth

• OpenII – an open-source platform for information integration.

References• Fusion Tables:

– tables.googlelabs.com– SIGMOD, SOCC, 2010

• Deep-web crawling:– [Madhavan et al., VLDB 08]

• WebTables: – [Cafarella et al., VLDB 08]

• Octopus: – [Cafarella et al., VLDB 09],– [Elmeleegy et al, VLDB 09]