Exploiting Big Data for Statistics: Some Implications for Policy

22
Exploiting Big Data for Statistics: Exploiting Big Data for Statistics: Some Implications for Policy Andrew Wyckoff* Directorate for Science, Technology & Industry Organisation for Economic Co-operation and Development Fb 25 February 2013 * The views expressed in this presentation are those of the authors and do not necessarily reflect the opinions of the OECD or its Membership.

Transcript of Exploiting Big Data for Statistics: Some Implications for Policy

Page 1: Exploiting Big Data for Statistics: Some Implications for Policy

Exploiting Big Data for Statistics:Exploiting Big Data for Statistics:Some Implications for Policy

Andrew Wyckoff*

Directorate for Science, Technology & IndustryOrganisation for Economic Co-operation and Development

F b 25 February 2013

* The views expressed in this presentation are those of the authors anddo not necessarily reflect the opinions of the OECD or its Membership.

Page 2: Exploiting Big Data for Statistics: Some Implications for Policy

Overview

• Some applications today

• Implications for public policy:– Economic & social policies;Economic & social policies;– Statistical policy

• Discussion

Page 3: Exploiting Big Data for Statistics: Some Implications for Policy

S & C

Mesh Networks / Ambient Computing

Supply Chain Management Security & Access Control

Work In Process Tracking Consumer ApplicationsWork In Process Tracking Consumer Applications

Asset Management Environmental Applications

3

Page 4: Exploiting Big Data for Statistics: Some Implications for Policy

Internet traffic flows continue to grow

90,000

100,000

MOBILEDATA

Monthly global IP traffic (Petabytes), 2005-2014

70,000

80,000MOBILE DATA

Consumer Internet video 

Consumer Voice over IP (VoIP) 

50,000

60,000Consumer Online gaming 

Consumer Video calling 

20,000

30,000

40,000Consumer Web, email, and data 

Consumer file sharing 

CONSUMERVOD

0

10,000

20,000 CONSUMER VOD

BUSINESS INTERNET / INTRANET

2010 2011 2012 2013 2014 2015

Source: CISCO VNI 2011

Page 5: Exploiting Big Data for Statistics: Some Implications for Policy

A lot of big data buzz

• “Data is the new oil.” Andreas Weigend, Stanford (ex Amazon)

• “The future belongs to companies and people that turn data into products”, Mike Loukides, O’Reilly Media

“The challenge–“Ten reasons why Big Data will

“Why big data is a big deal”InfoWorld 9/1/11

The challenge–and opportunity–of big data”McKinsey Quarterly 5/11

Big Data will change the travel industry”Tnooz 8/15/11

“Keeping Afloat in a Sea of 'Big

InfoWorld – 9/1/11 McKinsey Quarterly—5/11

“Getting a Handle on Big Data with

Tnooz -8/15/11

“The promise ofin a Sea of Big Data”ITBusinessEdge – 9/6/11

on Big Data with Hadoop”Businessweek-9/7/11

The promise of Big Data”Intelligent Utility-8/28/11

Page 6: Exploiting Big Data for Statistics: Some Implications for Policy

IT has always had an impact on Statistics

see www.abs.gov.au

Page 7: Exploiting Big Data for Statistics: Some Implications for Policy

Price Statistics MIT Billion Price ProjectMIT Billion Price Project

Page 8: Exploiting Big Data for Statistics: Some Implications for Policy

Demand for Jobs / SkillsHelp Wanted Statistics from the Conference BoardHelp Wanted Statistics from the Conference Board

Page 9: Exploiting Big Data for Statistics: Some Implications for Policy

New Job Starts / Job ChangesLinkedInLinkedIn

Page 10: Exploiting Big Data for Statistics: Some Implications for Policy

OECD Output ForecastsSWIFTSWIFT

Page 11: Exploiting Big Data for Statistics: Some Implications for Policy

Some economic policy implications

BenefitsBenefits

• Timeliness & now casting• Timeliness & now-casting• Robustness & granularity• Affordability & access• Democratisation & creativityDemocratisation & creativity

Page 12: Exploiting Big Data for Statistics: Some Implications for Policy

Unknown properties of Web Data

Source: www nature comSource: www.nature.com

Page 13: Exploiting Big Data for Statistics: Some Implications for Policy

Some economic policy implications

Ch llChallenges

•Unknown bias•Potential Instability•Quality

Page 14: Exploiting Big Data for Statistics: Some Implications for Policy

Some implications for NSOs: Will they get by-passed by “Big Data” ?Will they get by passed by Big Data ?

Page 15: Exploiting Big Data for Statistics: Some Implications for Policy

Some policy implications for NSOs

Good news

• Big Data tools are becoming widely availablebecoming widely available

• The Cloud can address infrastructure needs

• The “statistical commons” • The statistical commons grows

Page 16: Exploiting Big Data for Statistics: Some Implications for Policy

Some policy implications for NSOs

Challenges to AddressC a e ges o dd ess

A &O hi•Access &OwnershipP i•Privacy

•Liability•Liability•Skills•Skills

Page 17: Exploiting Big Data for Statistics: Some Implications for Policy

Access to and ownership of proprietary data

Page 18: Exploiting Big Data for Statistics: Some Implications for Policy

Privacy Issues

• image from http://wwwalthdatainnovation.com/sites/datawork.drupalgardens.com/files/styles/large/public/target.jpg

Page 19: Exploiting Big Data for Statistics: Some Implications for Policy

Liability

Page 20: Exploiting Big Data for Statistics: Some Implications for Policy

Skills “…the sexy job in the next 10 years

will be statisticians.”

Source: NYT, 5 August 2009

Page 21: Exploiting Big Data for Statistics: Some Implications for Policy

Possible new NSO roles

• Take on a new mission as a trusted 3rd party whose role would be to certify the statistical quality y q yof these new sources?

• Issue statistical “best practices” in the use of non-traditional sources and the mining of “big data”?g g

• Use non-traditional sources to augment (and g (perhaps replace) their official series?

Page 22: Exploiting Big Data for Statistics: Some Implications for Policy

Going ForwardG “A bl d d d t ld” b ildi • Groves: “A blended data world” – building on-top of existing surveys

lib i b d d– Calibrating web data to survey data

• Use of relative vs. absolute measures, Use of relative vs. absolute measures, now-casting

• Develop new methodologies

• Active Experimentation extracting • Active Experimentation, extracting lessons, devising “best practices”