1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai...
-
Upload
isaac-jordan -
Category
Documents
-
view
217 -
download
0
Transcript of 1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai...
![Page 1: 1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai CEO, Hope Bay Technologies, Inc.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfa21a28abf838c9638c/html5/thumbnails/1.jpg)
1CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技
Big Data PhilosophyBen Jai
CEO, Hope Bay Technologies, Inc.
![Page 2: 1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai CEO, Hope Bay Technologies, Inc.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfa21a28abf838c9638c/html5/thumbnails/2.jpg)
2CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技
What is Big Data?
Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.
Translation: DB2/Oracle/MSSQL/MySQL can’t handle it
![Page 3: 1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai CEO, Hope Bay Technologies, Inc.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfa21a28abf838c9638c/html5/thumbnails/3.jpg)
3CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技
Difference from just Data?
Volume (amount of data) Velocity (speed of data in and out) Variety (range of data types and sources) Veracity (truthfulness of the data)
Translation: DB2/Oracle/MSSQL/MySQL can’t handle it
![Page 4: 1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai CEO, Hope Bay Technologies, Inc.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfa21a28abf838c9638c/html5/thumbnails/4.jpg)
4CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技
Is that It?
Big, fast, unstructured, uncertain, so?
Conventional wisdom: I can solve the problem if my computer is 1000 times bigger and faster!• I have a question looking for an answer.
Unconventional wisdom: I can find the pattern if my data set is 1000 times bigger!• I have a data set looking for useful information.
![Page 5: 1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai CEO, Hope Bay Technologies, Inc.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfa21a28abf838c9638c/html5/thumbnails/5.jpg)
5CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技
Example 1: Machine Translation
Conventional method: dictionary + grammar rules + exception database…• Good enough initially, but will not improve• Exceptions grow naturally and DB can’t follow
Statistical machine translation: compare same article in different languages and build statistical models• Need a LOT of data to see initial results but
will grow automatically without human knowledge of the languages
![Page 6: 1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai CEO, Hope Bay Technologies, Inc.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfa21a28abf838c9638c/html5/thumbnails/6.jpg)
6CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技
How big is BIG?
47.5
48.5
49.5
50.5
51.5
52.5
53.5
AE BLEU[%]
+weblm =219B words of web data!
BLEU% score – Arabic to English
![Page 7: 1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai CEO, Hope Bay Technologies, Inc.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfa21a28abf838c9638c/html5/thumbnails/7.jpg)
7CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技
Example 2: Spell Check
Conventional method: compare words against dictionary and find minimal editing distance. But:• Proper names are not in the dictionary• Word may be in the dictionary but not what the
user wants• Can’t break ties
Google method: look at what others do• Britney Spears brittany spears, brittney spears,
britany spears, britny spears, briteny spears, britteny spears, brine spears, …
![Page 8: 1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai CEO, Hope Bay Technologies, Inc.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfa21a28abf838c9638c/html5/thumbnails/8.jpg)
8CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技
300 Ways to Spell Her Name
![Page 9: 1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai CEO, Hope Bay Technologies, Inc.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfa21a28abf838c9638c/html5/thumbnails/9.jpg)
9CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技
Also Look at the Context
Kofi Annan is correct
Kofee Annan Kofi Annan
Kofee Shop Coffee Shop
![Page 10: 1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai CEO, Hope Bay Technologies, Inc.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfa21a28abf838c9638c/html5/thumbnails/10.jpg)
10CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技
Example 3: Detecting Defaults
Conventional method: look at FICO score MIT method: use machine learning to look at
100’s of variables in customer activities
NTU professor used a similar method to reduce return rate in TV shopping channel
![Page 11: 1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai CEO, Hope Bay Technologies, Inc.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfa21a28abf838c9638c/html5/thumbnails/11.jpg)
11CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技
Example 4: Who’s in Love?
![Page 12: 1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai CEO, Hope Bay Technologies, Inc.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfa21a28abf838c9638c/html5/thumbnails/12.jpg)
12CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技
Amazing Results
Big Data System Architecture
Data Source
Data Source
Data Source
Data Source
Data Storage Platform
Data Analysis Platform
Data Analysis Application
![Page 13: 1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai CEO, Hope Bay Technologies, Inc.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfa21a28abf838c9638c/html5/thumbnails/13.jpg)
13CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技
Levels of Big Data Processing
STORAGE is what you can ACCESS DATA is what you can PROCESS INFORMATION is what you can
UNDERSTAND INTELLIGENCE is what you can USE
![Page 14: 1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai CEO, Hope Bay Technologies, Inc.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfa21a28abf838c9638c/html5/thumbnails/14.jpg)
14CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技
Amazing Results
Intelligence
Data Storage Platform
Data Analysis Platform
Data Storage
Data Analysis Tool
Data Analysis ApplicationData Information
Big Data System Components
Data Source
Data Source
Data Source
Data Source
![Page 15: 1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai CEO, Hope Bay Technologies, Inc.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfa21a28abf838c9638c/html5/thumbnails/15.jpg)
15CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技
Who Can Use Big Data?
Those who owns a lot of data Those who can see the information Those who can convert the information
into intelligence Those who can monetize the intelligence
![Page 16: 1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai CEO, Hope Bay Technologies, Inc.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfa21a28abf838c9638c/html5/thumbnails/16.jpg)
16CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技
Who Can Make Money?
Those who owns a lot of data Those who can see the information Those who can convert the information
into intelligence Those who can monetize the intelligence
![Page 17: 1 CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技 Big Data Philosophy Ben Jai CEO, Hope Bay Technologies, Inc.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfa21a28abf838c9638c/html5/thumbnails/17.jpg)
17CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技
It’s Just the Beginning…
1952: UNIVAC I predicted the landslide victory of Dwight D. Eisenhawer