Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the...
-
Upload
domino-data-lab -
Category
Technology
-
view
24 -
download
1
Transcript of Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the...
![Page 1: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/1.jpg)
11 ZILLOW | TRULIA | STREETEASY | HOTPADS | NAKED APARTMENTS
Vincent Yates, Director of Analytics Engineering@VincentYates8
FOUNTAIN OF YOUTH OR POLLUTED SWAMP: IS YOUR DATA LAKE REVITALIZING YOUR BUSINESS OR ERODING THE FOUNDATION?
![Page 2: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/2.jpg)
2
One of these is worth $42,000 more
Finished sq-ft 2,602 2,602Lot Size 4,400 5,342Bathrooms 3 3Bedrooms 4 4Year Built 2004 2005Sale Price 861,000 819,000
![Page 3: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/3.jpg)
3
One of these is worth $164,000 more
Finished sq-ft 1,620 1,620Lot Size 1,620 1,620Bathrooms 2.5 3Bedrooms 3 3Year Built 2007 2007Sale Price 499,000 663,000
![Page 4: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/4.jpg)
4
One of these is worth >$10M annually
http://www.exp-platform.com/Pages/SevenRulesofThumbforWebSiteExperimenters.aspx
![Page 5: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/5.jpg)
55
DATA SCIENCE’S DIRTY LITTLE SECRET
![Page 6: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/6.jpg)
66
$3.1 TRILLION
IBM Big Data Hub
![Page 7: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/7.jpg)
7
Unknowns ≠ Seasonality
Seasonality
Seasonality
Seasonality
Seasonality
SeasonalitySeasonality
![Page 8: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/8.jpg)
88
SeriouslyDATA SCIENCE IS HARD
![Page 9: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/9.jpg)
9
Product &Communication
Programming
Statistics
![Page 10: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/10.jpg)
1010
24% of data scientists
UNSURE OF HOW MUCH OF THEIR DATA ARE INACCURATE
IBM Big Data Hub
![Page 11: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/11.jpg)
11
Errors Propagate in Dynamic Ways
![Page 12: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/12.jpg)
12
![Page 13: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/13.jpg)
1313
66% of data scientists
CLEANING DATA IS THE MOST TIME CONSUMING TASK
CroundFlower 2015 Data Science Report
![Page 14: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/14.jpg)
1414
My data is pretty good.DOES IT REALLY MATTER?
![Page 15: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/15.jpg)
1515
52.3% of data scientists
POOR DATA QUALITY IS THEIR BIGGEST HURDLE
CroundFlower 2015 Data Science Report
![Page 16: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/16.jpg)
1616
The cost of poor data quality
15-25% OF OPERATING PROFIT
Kaufman,Morgan: The Accuracy Dimension
![Page 17: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/17.jpg)
1717
Someone would have noticed and fixed itHOW DID WE GET HERE?
![Page 18: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/18.jpg)
18
Cracks start to show under pressure
Data Quality: The Accuracy DimensionThe Morgan Kaufmann Series in Data Management Systems
OperationalIntegration Replication
![Page 19: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/19.jpg)
19
Complexity/Agility is the scapegoat
Transaction applications,APIs, Third-party data producers
Transaction databases
Data Marts
Data Lake
![Page 20: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/20.jpg)
20
Complexity/Agility is the scapegoat
Transaction applications,APIs, Third-party data producers
Transaction databases
Data Marts
Data Lake
![Page 21: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/21.jpg)
21
Complexity/Agility is the scapegoat
Transaction applications,APIs, Third-party data producers
Transaction databases
Data Marts
Data Lake
![Page 22: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/22.jpg)
22
Complexity/Agility is the scapegoat
![Page 23: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/23.jpg)
23
Complexity/Agility is the scapegoat
![Page 24: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/24.jpg)
24
Complexity/Agility is the scapegoatData Marts
Data Lake
![Page 25: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/25.jpg)
25
Moral Hazard is the culprit
![Page 26: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/26.jpg)
2626
HOW DO WE GET OUT?A few simple tricks to head in the right direction
![Page 27: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/27.jpg)
2727
PROACTIVE NOT REACTIVEData scientist is not great under duress
![Page 28: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/28.jpg)
28
Get Back to Raw Data
![Page 29: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/29.jpg)
29
Centralize Definitions
![Page 30: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/30.jpg)
30
Model Where Possible
![Page 31: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/31.jpg)
3131
MODELING IS HARDBuild tools to make reactive easier
![Page 32: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/32.jpg)
32
![Page 33: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/33.jpg)
33
![Page 34: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/34.jpg)
34
Data Problems are as Old as Data
![Page 35: Fountain of Youth or Polluted Swamp: Is your data lake revitalizing your business or eroding the foundation?](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cf67461a28ab01358b5fc9/html5/thumbnails/35.jpg)
35
Many mistakes are required for catastrophe• Climate caused more icebergs
– Ignored Forecasts
• Tides sent icebergs southward– Poor/Wrong Measurement
• The ship was going too fast– Business needs over best data
• Iceberg warnings went unheeded– Data was Disregarded for Intuition
• The binoculars were locked up– Tools were behind lock and key
• The steersman took a wrong turn– Reactive action under stress lead to
wrong decisions
• The iron rivets were too weak– Cost savings over best data
• There were too few lifeboats– Marketing owned the message
http://cosmiclog.nbcnews.com/_news/2012/04/01/10970732-10-causes-of-the-titanic-tragedy