2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data...
-
Upload
ferdinand-harrington -
Category
Documents
-
view
216 -
download
2
Transcript of 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data...
![Page 1: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/1.jpg)
2012.11.08- SLIDE 1IS 257 – Fall 2012
Data Mining and the Weka Toolkitand Intro for Big Data
University of California, Berkeley
School of Information
IS 257: Database Management
![Page 2: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/2.jpg)
2012.11.08- SLIDE 2IS 257 – Fall 2012
Lecture Outline
• Announcements– Final Project Reports
• Review– OLAP (ROLAP, MOLAP)
• Data Mining with the WEKA toolkit• Big Data (introduction)
![Page 3: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/3.jpg)
2012.11.08- SLIDE 3IS 257 – Fall 2012
Final project
• Final project is the completed version of your personal project with an enhanced version of Assignment 4
• AND an in-class presentation on the database design and interface
• Detailed description and elements to be considered in grading are available by following the links on the Assignments page or the main page of the class site
![Page 4: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/4.jpg)
2012.11.08- SLIDE 4IS 257 – Fall 2012
Lecture Outline
• Announcements– Final Project Reports
• Review– OLAP (ROLAP, MOLAP)
• Data Mining with the WEKA toolkit• Big Data (introduction)
![Page 5: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/5.jpg)
2012.11.08- SLIDE 5IS 257 – Fall 2012
Related Fields
Statistics
MachineLearning
Databases
Visualization
Data Mining and Knowledge Discovery
Source: Gregory Piatetsky-Shapiro
![Page 6: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/6.jpg)
2012.11.08- SLIDE 6IS 257 – Fall 2012
The Hype Curve for Data Mining and Knowledge Discovery
Over-inflated expectations
Disappointment
Growing acceptanceand mainstreaming
rising expectations
Source: Gregory Piatetsky-Shapiro
![Page 7: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/7.jpg)
2012.11.08- SLIDE 7IS 257 – Fall 2012
OLAP
• Online Line Analytical Processing– Intended to provide multidimensional views of
the data– I.e., the “Data Cube”– The PivotTables in MS Excel are examples of
OLAP tools
![Page 8: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/8.jpg)
2012.11.08- SLIDE 8IS 257 – Fall 2012
Data Cube
![Page 9: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/9.jpg)
2012.11.08- SLIDE 9IS 257 – Fall 2012
Visualization – Star Schema
Dimension Table (Beers) Dimension Table (etc.)
Dimension Table (Drinkers)Dimension Table (Bars)
Fact Table - Sales
Dimension Attrs. Dependent Attrs.
From anonymous “olap.ppt” found on Google
![Page 10: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/10.jpg)
2012.11.08- SLIDE 10IS 257 – Fall 2012
Typical OLAP Queries
• Often, OLAP queries begin with a “star join”: the natural join of the fact table with all or most of the dimension tables.
• Example:SELECT *FROM Sales, Bars, Beers, DrinkersWHERE Sales.bar = Bars.bar ANDSales.beer = Beers.beer ANDSales.drinker = Drinkers.drinker;
From anonymous “olap.ppt” found on Google
![Page 11: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/11.jpg)
2012.11.08- SLIDE 11IS 257 – Fall 2012
Example: In SQL
SELECT bar, beer, SUM(price)FROM Sales NATURAL JOIN BarsNATURAL JOIN Beers
WHERE addr = ’Palo Alto’ ANDmanf = ’Anheuser-Busch’
GROUP BY bar, beer;
From anonymous “olap.ppt” found on Google
![Page 12: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/12.jpg)
2012.11.08- SLIDE 12IS 257 – Fall 2012
Example: Materialized View
• Which views could help with our query?• Key issues:
1. It must join Sales, Bars, and Beers, at least.
2. It must group by at least bar and beer.
3. It must not select out Palo-Alto bars or Anheuser-Busch beers.
4. It must not project out addr or manf.
From anonymous “olap.ppt” found on Google
![Page 13: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/13.jpg)
2012.11.08- SLIDE 13IS 257 – Fall 2012
Example --- Continued
• Here is a materialized view that could help:
CREATE VIEW BABMS(bar, addr,beer, manf, sales) AS
SELECT bar, addr, beer, manf,SUM(price) sales
FROM Sales NATURAL JOIN BarsNATURAL JOIN Beers
GROUP BY bar, addr, beer, manf;
Since bar -> addr and beer -> manf, there is no realgrouping. We need addr and manf in the SELECT.
From anonymous “olap.ppt” found on Google
![Page 14: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/14.jpg)
2012.11.08- SLIDE 14IS 257 – Fall 2012
Example --- Concluded
• Here’s our query using the materialized view BABMS:
SELECT bar, beer, salesFROM BABMSWHERE addr = ’Palo Alto’ AND
manf = ’Anheuser-Busch’;
From anonymous “olap.ppt” found on Google
![Page 15: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/15.jpg)
2012.11.08- SLIDE 15IS 257 – Fall 2012
Example: Market Baskets
• If people often buy hamburger and ketchup together, the store can:
1. Put hamburger and ketchup near each other and put potato chips between.
2. Run a sale on hamburger and raise the price of ketchup.
From anonymous “olap.ppt” found on Google
![Page 16: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/16.jpg)
2012.11.08- SLIDE 16IS 257 – Fall 2012
Finding Frequent Pairs
• The simplest case is when we only want to find “frequent pairs” of items.
• Assume data is in a relation Baskets(basket, item).
• The support threshold s is the minimum number of baskets in which a pair appears before we are interested.
From anonymous “olap.ppt” found on Google
![Page 17: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/17.jpg)
2012.11.08- SLIDE 17IS 257 – Fall 2012
Frequent Pairs in SQL
SELECT b1.item, b2.itemFROM Baskets b1, Baskets b2WHERE b1.basket = b2.basketAND b1.item < b2.item
GROUP BY b1.item, b2.itemHAVING COUNT(*) >= s;
Look for twoBasket tupleswith the samebasket anddifferent items.First item mustprecede second,so we don’tcount the samepair twice.
Create a group foreach pair of itemsthat appears in atleast one basket.
Throw away pairs of itemsthat do not appear at leasts times.
From anonymous “olap.ppt” found on Google
![Page 18: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/18.jpg)
2012.11.08- SLIDE 18IS 257 – Fall 2012
Lecture Outline
• Announcements– Final Project Reports
• Review– OLAP (ROLAP, MOLAP)
• Data Mining with the WEKA toolkit• Big Data (introduction)
![Page 19: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/19.jpg)
2012.11.08- SLIDE 19IS 257 – Fall 2012
More on Data Mining using Weka
• Slides from Eibe Frank, Waikato Univ. NZ
![Page 20: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/20.jpg)
2012.11.08- SLIDE 20IS 257 – Fall 2012
Lecture Outline
• Announcements– Final Project Reports
• Review– OLAP (ROLAP, MOLAP)
• Data Mining with the WEKA toolkit• Big Data (introduction)
![Page 21: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/21.jpg)
2012.11.08- SLIDE 21IS 257 – Fall 2012
Big Data and Databases
• “640K ought to be enough for anybody.”– Attributed to Bill Gates, 1981
![Page 22: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/22.jpg)
2012.11.08- SLIDE 22
Big Data and Databases
• We have already mentioned some Big Data – The Walmart Data Warehouse– Information collected by Amazon on users
and sales and used to make recommendations
• Most modern web-based companies capture EVERYTHING that their customers do– Does that go into a Warehouse or someplace
else?
IS 257 – Fall 2012
![Page 23: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/23.jpg)
2012.11.08- SLIDE 23IS 257 – Fall 2012
Other Examples
• NASA EOSDIS– Estimated 1018 Bytes (Exabyte)
• Computer-Aided design• The Human Genome• Department Store tracking
– Mining non-transactional data (e.g. Scientific data, text data?)
• Insurance Company– Multimedia DBMS support
![Page 24: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/24.jpg)
2012.11.08- SLIDE 24IS 257 – Fall 2012
Before the Cloud there was the Grid
• So what’s this Grid thing anyhow?• Data Grids and Distributed Storage• Grid vs “Cloud”
This lecture borrows heavily from presentations by Ian Foster (Argonne National Laboratory & University of Chicago), Reagan Moore and others from San Diego Supercomputer Center
![Page 25: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/25.jpg)
2012.11.08- SLIDE 25IS 257 – Fall 2012
The Grid: On-Demand Access to Electricity
Time
Qua
lity,
eco
nom
ies
of s
cale
Source: Ian Foster
![Page 26: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/26.jpg)
2012.11.08- SLIDE 26IS 257 – Fall 2012
By Analogy, A Computing Grid
• Decouples production and consumption– Enable on-demand access– Achieve economies of scale– Enhance consumer flexibility– Enable new devices
• On a variety of scales– Department– Campus– Enterprise– Internet
Source: Ian Foster
![Page 27: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/27.jpg)
2012.11.08- SLIDE 27IS 257 – Fall 2012
What is the Grid?
“The short answer is that, whereas the Web is a service for sharing information over the Internet, the Grid is a service for sharing computer power and data storage capacity over the Internet. The Grid goes well beyond simple communication between computers, and aims ultimately to turn the global network of computers into one vast computational resource.”
Source: The Global Grid Forum
![Page 28: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/28.jpg)
2012.11.08- SLIDE 28IS 257 – Fall 2012
Not Exactly a New Idea …
• “The time-sharing computer system can unite a group of investigators …. one can conceive of such a facility as an … intellectual public utility.”– Fernando Corbato and Robert Fano , 1966
• “We will perhaps see the spread of ‘computer utilities’, which, like present electric and telephone utilities, will service individual homes and offices across the country.” Len Kleinrock, 1967
Source: Ian Foster
![Page 29: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/29.jpg)
2012.11.08- SLIDE 29IS 257 – Fall 2012
But, Things are Different Now
• Networks are far faster (and cheaper)– Faster than computer backplanes
• “Computing” is very different than pre-Net– Our “computers” have already disintegrated– E-commerce increases size of demand peaks– Entirely new applications & social structures
• We’ve learned a few things about software
Source: Ian Foster
![Page 30: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/30.jpg)
2012.11.08- SLIDE 30IS 257 – Fall 2012
Computing isn’t Really Like Electricity
• I import electricity but must export data• “Computing” is not interchangeable but highly
heterogeneous: data, sensors, services, …• This complicates things; but also means that the
sum can be greater than the parts – Real opportunity: Construct new capabilities
dynamically from distributed services
• Raises three fundamental questions– Can I really achieve economies of scale?– Can I achieve QoS across distributed services?– Can I identify apps that exploit synergies?
Source: Ian Foster
![Page 31: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/31.jpg)
2012.11.08- SLIDE 31IS 257 – Fall 2012
Why the Grid?(1) Revolution in Science
• Pre-Internet– Theorize &/or experiment, alone
or in small teams; publish paper• Post-Internet
– Construct and mine large databases of observational or simulation data
– Develop simulations & analyses– Access specialized devices remotely– Exchange information within
distributed multidisciplinary teams
Source: Ian Foster
![Page 32: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/32.jpg)
2012.11.08- SLIDE 32IS 257 – Fall 2012
Why the Grid?(2) Revolution in Business
• Pre-Internet– Central data processing facility
• Post-Internet– Enterprise computing is highly distributed,
heterogeneous, inter-enterprise (B2B)– Business processes increasingly
computing- & data-rich– Outsourcing becomes feasible =>
service providers of various sorts
Source: Ian Foster
![Page 33: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/33.jpg)
2012.11.08- SLIDE 33IS 257 – Fall 2012
The Information Grid
Imagine a web of data• Machine Readable
– Search, Aggregate, Transform, Report On, Mine Data – using more computers, and less humans
• Scalable– Machines are cheap – can buy 50 machines with
100Gb or memory and 100 TB disk for under $100K, and dropping
– Network is now faster than disk
• Flexible– Move data around without breaking the apps
Source: S. Banerjee, O. Alonso, M. Drake - ORACLE
![Page 34: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/34.jpg)
2012.11.08- SLIDE 34IS 257 – Fall 2012
Tier0/1 facility
Tier2 facility
10 Gbps link
2.5 Gbps link
622 Mbps link
Other link
Tier3 facility
The Foundations are Being Laid
Cambridge
Newcastle
Edinburgh
Oxford
Glasgow
Manchester
Cardiff
Soton
London
Belfast
DL
RAL Hinxton
![Page 35: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/35.jpg)
2012.11.08- SLIDE 35IS 257 – Fall 2012
Current Environment
• “Big Data” is becoming ubiquitous in many fields– enterprise applications– Web tasks– E-Science– Digital entertainment– Natural Language Processing (esp. for
Humanities applications)– Social Network analysis– Etc.
![Page 36: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/36.jpg)
2012.11.08- SLIDE 36IS 257 – Fall 2012
Current Environment
• Data Analysis as a profit center– No longer just a cost – may be the entire
business as in Business Intelligence
![Page 37: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/37.jpg)
2012.11.08- SLIDE 37IS 257 – Fall 2012
Current Environment
• Ubiquity of Structured and Unstructured data– Text– XML– Web Data– Crawling the Deep Web
• How to extract useful information from “noisy” text and structured corpora?
![Page 38: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/38.jpg)
2012.11.08- SLIDE 38IS 257 – Fall 2012
Current Environment
• Expanded developer demands– Wider use means broader requirements, and
less interest from developers in the details of traditional DBMS interactions
• Architectural Shifts in Computing– The move to parallel architectures both
internally (on individual chips)– And externally – Cloud Computing
![Page 39: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/39.jpg)
2012.11.08- SLIDE 39IS 257 – Fall 2011
The Semantic Web
• The basic structure of the Semantic Web is based on RDF triples (as XML or some other form)
• Conventional DBMS are very bad at doing some of the things that the Semantic Web is supposed to do… (.e.g., spreading activation searching)
• “Triple Stores” are being developed that are intended to optimize for the types of search and access needed for the Semantic Web
![Page 40: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/40.jpg)
2012.11.08- SLIDE 40IS 257 – Fall 2012
Research Opportunities
• Revisiting Database Engines– Do DBMS need a redesign from the ground
up to accommodate the new demands of the current environment?
![Page 41: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/41.jpg)
2012.11.08- SLIDE 41IS 257 – Fall 2011
The next-generation DBMS
• What can we expect for a next generation of DBMS?
• Look at the DB research community – their research leads to the “new features” in DBMS
• The “Claremont Report” on DB research is the report of meeting of top researchers and what they think are the interesting and fruitful research topics for the future
![Page 42: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/42.jpg)
2012.11.08- SLIDE 42IS 257 – Fall 2011
But will it be a RDBMS?
• Recently, Mike Stonebraker (one of the people who helped invent Relational DBMS) has suggested that the “One Size Fits All” model for DBMS is an idea whose time has come – and gone– This was also a theme of the Claremont Report
• RDBMS technology, as noted previously, has optimized on transactional business type processing
• But many other applications do not follow that model
![Page 43: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/43.jpg)
2012.11.08- SLIDE 43IS 257 – Fall 2011
Will it be an RDBMS?
• Stonebraker predicts that the DBMS market will fracture into many more specialized database engines– Although some may have a shared common
frontend• Examples are Data Warehouses, Stream
processing engines, Text and unstructured data processing systems
![Page 44: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/44.jpg)
2012.11.08- SLIDE 44IS 257 – Fall 2011
Will it be an RDBMS?
• Data Warehouses currently use (mostly) conventional DBMS technology– But they are NOT the type of data those are
optimized for– Storage usually puts all elements of a row together,
but that is an optimization for updating and not searching, summarizing, and reading individual attributes
– A better solution is to store the data by column instead of by row – vastly more efficient for typical Data Warehouse Applications
![Page 45: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/45.jpg)
2012.11.08- SLIDE 45IS 257 – Fall 2011
Will it be an RDBMS?
• Streaming data, such as Wall St. stock trade information is badly suited to conventional RDBMS (other than as historical data)– The data arrives in a continuous real-time stream– But, data in RDBMS has to be stored before it can be
read and actions taken on it• This is too slow for real-time actions on that data
– Stream processors function by running “queries” on the live data stream instead• May be orders of magnitude faster
![Page 46: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/46.jpg)
2012.11.08- SLIDE 46IS 257 – Fall 2011
Will it be an RDBMS?
• Sensor networks provide another massive stream input and analysis problem
• Text Search: No current text search engines use RDBMS, they too need to be optimized for searching, and tend to use inverted file structures instead of RDBMS storage
• Scientific databases are another typical example of streamed data from sensor networks or instruments
• XML data is still not a first-class citizen of RDBMS, and there are reasons to believe that specialized database engines are needed
![Page 47: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/47.jpg)
2012.11.08- SLIDE 47IS 257 – Fall 2011
Will it be an RDBMS
• RDBMS will still be used for what they are best at – business-type high transaction data
• But specialized DBMS will be used for many other applications
• Consider Oracle’s acquisions of SleepyCat (BerkeleyDB) embedded database engine, and TimesTen main memory database engine– specialized database engines for specific applications
![Page 48: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/48.jpg)
2012.11.08- SLIDE 48IS 257 – Fall 2011
Some things to consider
• Bandwidth will keep increasing and getting cheaper (and go wireless)
• Processing power will keep increasing – Moore’s law: Number of circuits on the most advanced
semiconductors doubling every 18 months– With multicore chips, all computing is becoming parallel
computing
• Memory and Storage will keep getting cheaper (and probably smaller)– “Storage law”: Worldwide digital data storage capacity has
doubled every 9 months for the past decade
![Page 49: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/49.jpg)
2012.11.08- SLIDE 49IS 257 – Fall 2012
Research Opportunities-DB engines
• Designing systems for clusters of many-core processors
• Exploiting RAM and Flash as persistent media, rather than relying on magnetic disk
• Continuous self-tuning of DBMS systems• Encryption and Compression• Supporting non-relation data models
– instead of “shoe-horning” them into tables
![Page 50: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/50.jpg)
2012.11.08- SLIDE 50IS 257 – Fall 2012
Research Opportunities-DB engines
• Trading off consistency and availability for better performance and scaleout to thousands of machines
• Designing power-aware DBMS that limit energy costs without sacrificing scalability
![Page 51: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/51.jpg)
2012.11.08- SLIDE 51IS 257 – Fall 2012
Research Opportunities-Programming
• Declarative Programming for Emerging Platforms– MapReduce (esp. Hadoop and tools like Pig)– Ruby on Rails– Workflows
![Page 52: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/52.jpg)
2012.11.08- SLIDE 52IS 257 – Fall 2012
Research Opportunities-Data
• The Interplay of Structured and Unstructured Data– Extracting Structure automatically– Contextual awareness– Combining with IR research and Machine
Learning
![Page 53: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/53.jpg)
2012.11.08- SLIDE 53IS 257 – Fall 2012
Research Opportunities - Cloud• Cloud Data Services
– New models for “shared data” servers– Learning from Grid Computing
• SRB/IRODS, etc.
– Hadoop - as mentioned earlier - is open source and freely available software from Apache for running massively parallel computation (and distributed storage)
![Page 54: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/54.jpg)
2012.11.08- SLIDE 54IS 257 – Fall 2012
Research Opportunities - Mobile• Mobile Applications and Virtual Worlds
– Need for real-time services combining massive amounts of user-generated data
![Page 55: 2012.11.08- SLIDE 1IS 257 – Fall 2012 Data Mining and the Weka Toolkit and Intro for Big Data University of California, Berkeley School of Information.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d995503460f94a8369f/html5/thumbnails/55.jpg)
2012.11.08- SLIDE 55IS 257 – Fall 2012
Moving forward
• Much of this is already happening• Big Data is the new normal
– Terabytes are common Petabytes are Big• Hadoop and software based on it (Pig,
Hive, Impala, etc.) are becoming standard and well supported (e.g. Cloudera)
• Next Week we go for a deeper dive into Big Data and how it can be used– Tuesday – Tom Lento of Facebook– Thursday - ???