BigData€¦ · McKinsey Global Institute: Big data: The next frontier for innovation, competition,...
Transcript of BigData€¦ · McKinsey Global Institute: Big data: The next frontier for innovation, competition,...
-
MARTIN FOWLER
http://martinfowler.comhttp://thoughtworks.com@martinfowler
REBECCA PARSONS
[email protected]://thoughtworks.com
CTO
Big Data
Thursday, March 8, 2012
http://martinfowler.comhttp://martinfowler.com
-
MARTIN FOWLER
http://martinfowler.comhttp://thoughtworks.com@martinfowler
REBECCA PARSONS
[email protected]://thoughtworks.com
CTO
BigDataThursday, March 8, 2012
http://martinfowler.comhttp://martinfowler.com
-
MARTIN FOWLER
http://martinfowler.comhttp://thoughtworks.com@martinfowler
REBECCA PARSONS
[email protected]://thoughtworks.com
CTO
The Evolving Panorama of Data
Thursday, March 8, 2012
http://martinfowler.comhttp://martinfowler.com
-
Changing Nature of Data
Response
How we use data nowThursday, March 8, 2012
-
Changing Nature of Data
Response
How we use data nowThursday, March 8, 2012
-
Data is:
Thursday, March 8, 2012
-
Data is: Growing
Thursday, March 8, 2012
-
Data is: Growing
Thursday, March 8, 2012
-
Walmart: 1 million transactions per hour
Facebook: 40 billion photos
The Economist: Feb 25th 2010
Data is: Growing
Thursday, March 8, 2012
-
640K ought to be enough for anybody
Thursday, March 8, 2012
-
Lots of Traffic
Thursday, March 8, 2012
-
Thursday, March 8, 2012
-
Thursday, March 8, 2012
-
Data is:
Thursday, March 8, 2012
-
Data is: Distributed
Thursday, March 8, 2012
-
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
442 1,990 8,64040,223
127,942
356,191
616,308
853,698
1,080,872
1,287,537
1,482,824
Monthly Contributors to Wikipedia souce: wikipedia
Data is: Distributed
Thursday, March 8, 2012
-
Data is: Distributed
Thursday, March 8, 2012
-
Data is: Distributed
Thursday, March 8, 2012
-
Data is: Distributed
98% of internet access points in Africa are mobile
30 million networked sensor nodes
growing 30% per year
McKinsey Global Institute: Big data: The next frontier for innovation, competition, and productivity
Thursday, March 8, 2012
-
Data is:
Thursday, March 8, 2012
-
Data is: Valuable
Thursday, March 8, 2012
-
Data is: Valuable
$300 billion / year for US health care
60% increase in retail margins
McKinsey Global Institute: Big data: The next frontier for innovation, competition, and productivity
Thursday, March 8, 2012
-
Data is:
Thursday, March 8, 2012
-
Data is: Urgent
Thursday, March 8, 2012
-
Data is:
Thursday, March 8, 2012
-
Data is: Connected
Thursday, March 8, 2012
-
Data Silos
Thursday, March 8, 2012
-
Enterprise Data Model
App
App App
App
Thursday, March 8, 2012
-
Enterprise Data Model
App
App App
App
Thursday, March 8, 2012
-
Changing Nature of Data
Response
How we use data nowThursday, March 8, 2012
-
Changing Nature of Data
Response
How we use data nowThursday, March 8, 2012
-
Thursday, March 8, 2012
-
SQLSQL
Thursday, March 8, 2012
-
Thursday, March 8, 2012
-
Bigtable
Dynamo
Thursday, March 8, 2012
-
Thursday, March 8, 2012
-
"NoSQL"
Thursday, March 8, 2012
-
Definition of NoSQL
Thursday, March 8, 2012
-
of NoSQLCharacteristics
Thursday, March 8, 2012
-
of NoSQLCharacteristics
non-relational open-source cluster-friendly
21st Century Web schema-less
Thursday, March 8, 2012
-
Document
Graph
Graph
Key-value
Column-family
Thursday, March 8, 2012
-
Aggregate
Thursday, March 8, 2012
-
Eric Evans
Aggregate
Thursday, March 8, 2012
-
line items:
ID: 1001
customer: Ann
$4820321293533
$3910321601912
$5110131495054
$96
$39
$51
payment details:
Card: AmexCC Number: 12345expiry: 04/2001
orders
customers
order lines
credit cards
Thursday, March 8, 2012
-
line items:
ID: 1001
customer: Ann
$4820321293533
$3910321601912
$5110131495054
$96
$39
$51
payment details:
Card: AmexCC Number: 12345expiry: 04/2001
aggregate Order 1001
Thursday, March 8, 2012
-
line items:
ID: 1001
customer: Ann
$4820321293533
$3910321601912
$5110131495054
$96
$39
$51
payment details:
Card: AmexCC Number: 12345expiry: 04/2001
aggregate documentkey-valuecolumn family
Order 1001
Thursday, March 8, 2012
-
Order 1001Order 1004Order 1026Order 2843
Order 1001Order 1004Order 1026Order 3754
Order 1001Order 1004
Thursday, March 8, 2012
-
Document
Graph
Graph
Key-value
Column-family
Thursday, March 8, 2012
-
Document
Graph
Graph
Key-value
Column-family
Thursday, March 8, 2012
-
GraphThursday, March 8, 2012
-
GraphThursday, March 8, 2012
-
GraphThursday, March 8, 2012
-
Graph
Polyglot Persistence
Thursday, March 8, 2012
-
Billing
Inventory
Thursday, March 8, 2012
-
Billing
Inventory
Integration Database
Thursday, March 8, 2012
-
Billing
Inventory
Integration Database
Billing
Inventory
Thursday, March 8, 2012
-
Billing
Inventory
Integration Database
Billing
Inventory
Application Database
Thursday, March 8, 2012
-
Event Sourcing
Thursday, March 8, 2012
-
John Smith : Person
:Address
city = Portland
Thursday, March 8, 2012
-
John Smith : Person
:Address
city = Portland
change my address
Thursday, March 8, 2012
-
John Smith : Personchange my
address
:Address
city = Boston
Thursday, March 8, 2012
-
John Smith : Person
:Address
city = Portland
change my address
Thursday, March 8, 2012
-
John Smith : Person
:Address
city = Portland
change my address
:Event
change address
Thursday, March 8, 2012
-
John Smith : Personchange my
address
:Address
city = Boston
:Event
change address
Thursday, March 8, 2012
-
Log
Application State
Thursday, March 8, 2012
-
Log
Application State
Thursday, March 8, 2012
-
Log
Application State
Thursday, March 8, 2012
-
on-demand self-service
broad network access
resource pooling
rapid elasticity
measured service
special publication 800-145
Thursday, March 8, 2012
-
Data Sources
Thursday, March 8, 2012
-
Data Sourceswere
Thursday, March 8, 2012
-
Data Sourceswere will be
textimagevideo
connections
Thursday, March 8, 2012
-
Analytics
Thursday, March 8, 2012
-
Analyticswere
roll-ups
trends
variance
Thursday, March 8, 2012
-
Analyticswill be
pattern recognition
data mining
chasing connections
were
roll-ups
trends
variance
Thursday, March 8, 2012
-
Thursday, March 8, 2012
-
Thursday, March 8, 2012
-
Thursday, March 8, 2012
-
mapmapmapmap
mapmapmapmap
mapmapmapmap
per order
Thursday, March 8, 2012
-
mapmapmapmap reduce
mapmapmapmap reduce
mapmapmapmap reduce
per order per month
Thursday, March 8, 2012
-
mapmapmapmap reduce
mapmapmapmap reduce
mapmapmapmap reduce
reduce
per order per month
Thursday, March 8, 2012
-
Thursday, March 8, 2012
-
10,000 ft view (literally)
CodeCity by Richard Wettel
http://www.inf.unisi.ch/phd/wettel/codecity.html
Thursday, March 8, 2012
http://www.inf.unisi.ch/phd/wettel/codecity.htmlhttp://www.inf.unisi.ch/phd/wettel/codecity.html
-
Thursday, March 8, 2012
-
Changing Nature of Data
Response
How we use data nowThursday, March 8, 2012
-
Changing Nature of Data
Response
How we use data nowThursday, March 8, 2012
-
Data Scientist
Thursday, March 8, 2012
-
Data
Thursday, March 8, 2012
-
Data Journalist
Thursday, March 8, 2012
-
Thursday, March 8, 2012
-
Thursday, March 8, 2012
-
Thursday, March 8, 2012
-
Data Warehousing
Thursday, March 8, 2012
-
Data Warehousing
Thursday, March 8, 2012
-
Thursday, March 8, 2012
-
Thursday, March 8, 2012
-
http://ureport.ug/Thursday, March 8, 2012
-
http://ushahidi.com/
Thursday, March 8, 2012
-
http://libyacrisismap.net/Thursday, March 8, 2012
-
http://opendata.go.ke/Thursday, March 8, 2012
-
http://datawithoutborders.cc/Thursday, March 8, 2012
-
0
100
200
300
400
500
Supply Demand
Deep Analytical Talent in 2018
McKinsey Global Institute: Big data: The next frontier for innovation, competition, and productivityThursday, March 8, 2012
-
0
100
200
300
400
500
Supply Demand
Deep Analytical Talent in 2018
50% of supply
McKinsey Global Institute: Big data: The next frontier for innovation, competition, and productivityThursday, March 8, 2012
-
Thursday, March 8, 2012
-
Changing Nature of Data
Response
How we use data nowThursday, March 8, 2012
-
Changing Nature of Data
Response
How we use data nowThursday, March 8, 2012
-
What about us?
Thursday, March 8, 2012
-
Thursday, March 8, 2012
-
Thursday, March 8, 2012
-
Thursday, March 8, 2012
-
Thursday, March 8, 2012
-
MARTIN FOWLER
http://martinfowler.comhttp://thoughtworks.com@martinfowler
REBECCA PARSONS
[email protected]://thoughtworks.com
CTO
clip art from http://openclipart.org
Thursday, March 8, 2012
http://martinfowler.comhttp://martinfowler.com