UN Global Pulse: Big Data for a Better World (Strata Conf NYC)

“BEYOND TARGETED ADS BIG DATA FOR A BETTER WORLD”

Robert Kirkpatrick Director, UN Global Pulse

O’Reilly Strata Conference | New York | October 2012

www.unglobalpulse.org @unglobalpulse

25 October 2012 | www.unglobalpulse.org

Photo Credit: John Oyuke

Kenyan Farm Workers


Hyperconnectivity

Airplane Flights

Internet Traffic

Telephone Calls

Social Networking


20th-Century tools…


PRIVATE SECTOR Monitor operations…in real time Track market trends…in real time

Get customer feedback…in real time

GLOBAL DEVELOPMENT Unemployment? Food Security? Public Health?

Education? Migration?

Disaster Relief?

1.  Better early warning: Earlier detection of anomalies, trends and events allows earlier response.

2.  Real-time awareness: A more accurate and up-to-

date picture of population needs supports more effective planning and implementation

3.  Real-time feedback: Understanding sooner where

needs are changing -- or are not being met -- allows for rapid, adaptive course correction

BIG DATA IN REAL TIME: 3 OPPORTUNITIES


BIG DATA IS A HUMAN RIGHTS ISSUE

•  Never analyze personally identifiable information •  Never analyze confidential data •  Never seek to re-identify individuals


RESEARCH BACKGROUND: A GROWING BODY OF EVIDENCE

How Mobile Phone Carriers See the World

Call Detail Records (CDRs) •  Caller ID (hashed phone #) •  Caller Tower Location •  Receiver ID (hashed phone #) •  Receiver Tower Location •  Call Start Time •  Call Duration

Airtime Expense Records •  Caller ID (hashed phone #) •  Caller Tower Location •  Amount of Purchase •  Time of Purchase •  Balance at Time of Purchase


Modeling Behaviors in Mobile Data

Consumption variables • Number of calls, call duration, SMS/MMS/voice • Size, frequency, total number of airtime purchases • Handset Type and Features

Social variables • Degree of the social network • Weight of the contacts, frequency of communication

Mobility variables • Diameter of mobility and social network • Radius of gyration • Mobility Patterns

Source: Telefonica Research, 2011 25 October 2012 | www.unglobalpulse.org

EXAMPLE: AIRTIME CREDIT PURCHASE DATA


Higher household income

Lower household income

Average # of purchases / month

Aver

age

size

of p

urch

ase

SIZE AND FREQUENCY PREDICT HOUSEHOLD INCOME


Lower socioeconomic level

Higher socioeconomic level

CALLING PATTERNS AND ECONOMIC OPPORTUNITY


Men: •  Fewer calls •  Shorter calls •  Smaller social network •  More work-related calls

Women: •  More calls •  Longer calls •  Larger social network •  More personal calls

MEN AND WOMEN USE THEIR PHONES DIFFERENTLY


Tracking population movement to predict cholera

Source: Linus Bengtsson et. al., PLoS Medicine, 2011


A mobility index to evaluate H1N1 response in Mexico City

Source: Telefonica Research, 2011 See: http://www.unglobalpulse.org/publicpolicyandcellphonedata


dominican republic 11.7%

mexico 9.1% united kingdom 7.5%

canada 7.0%

guatemala 5.5%

ecuador 5.3%

jamaica 3.3%

india 3.0%

germany 2.5%

philippines 2.4%

TWITTER PREDICTS SPREAD OF INFLUENZA

r2 = .958

“You Are What You Tweet: Analyzing Twitter for Public Health. M. J. Paul and M. Dredze, 2011.” http://www.cs.jhu.edu/%7Empaul/files/2011.icwsm.twitter_health.pdf


Rumi Chunara et. al., American Journal of Tropical Medicine and Hygiene, 2012 86:39-45


GOOGLE SEARCHES FOR SYMPTOMS PREDICT DENGUE


2010 VS. 2011: INDONESIAN TWEETS ABOUT HIV

See: http://www.unglobalpulse.org/WorldAIDSDay-Part2 25 October 2012 | www.unglobalpulse.org

PROOF OF CONCEPT STUDIES

GLOBAL PULSE RESEARCH 2011

Online at: http://www.unglobalpulse.org/applyingbigdatatodevelopment

http://www.unglobalpulse.org/projects/can-social-media-mining-add-depth-unemployment-statistics

Online Discussions & Unemployment

Ireland


Online Discussions & Unemployment

United States


http://www.unglobalpulse.org/projects/twitter-and-perceptions-crisis-related-stress

Jakarta: nine million tweets per day

Map of Twi*er usage in Jakarta – by Eric Fischer 25 October 2012 | www.unglobalpulse.org

Tweets per day about food, during Ramadan in Indonesia

Start of Ramadan

End of Ramadan


Number of tweets per month commenting on the price of rice

Wordcloud1 (sunday)

Wordcloud2 (Sunday)

Food inflation

0.47

0.44

-0.85

1.49

2.81

2.21

-0.33

-1.94

-1.9

-0.28

1.27

1.84

1.07

-0.09

Data inflation

Tweets about the price of rice (per month)

Official Food Price Inflation (monthly from 25 cities)

Tweets predict food basket inflation (rice, chilies, fish, sugar, corn, cooking oil)


GLOBAL PULSE ABOUT

DIGITAL SERVICES AS HUMAN SENSOR NETWORKS: Observing fluctuations in well-being…in real-time

COPING STRATEGIES •  Buy cheaper foods •  Work longer hours •  Reduce energy use •  Draw down savings •  Sell assets •  Borrow from relatives

DIGITAL “SMOKE SIGNALS” •  Depletion of airtime credit •  Smaller mobile airtime

purchases •  Failure to repay microloans via

mobile financial services •  Changes in calling patterns •  Inbound money transfers •  Web searches for jobs, health •  Sales of livestock via mobile

trading network •  “Venting” on social media


Photo: Ministry of Foreign Affairs, Iceland


AGILE GLOBAL DEVELOPMENT?


•  This data may be less accurate that official sources.

•  But it’s faster. •  And it’s cheaper to

collect. •  How can we leverage

the speed to change the outcome?

Integrating real-time data into an institution

USGS Twitter Earthquake Detector (TED)


THE PROBLEM WITH TELESCOPES…

There’s a universe of data that we can’t see.

…AND MACROSCOPES


Data Philanthropy?

A global

real-time public/private

data commons?

EXAMPLE R&D PROJECT: Mobile Networks as Drought Sensors in the Sahel

Proposal •  Obtain 2011-2012 mobile CDRs and airtime purchases. •  Derive mobility, consumption, and social variables. •  Correlate variables with precipitation levels, survey

data. •  Identify signatures of drought impacts in 2011. •  Identify signatures of aid impact in 2012. •  Develop and evaluate prototype during next drought. •  Release open source “appliance” through GSMA.


PULSE LABS

Joint Research | Rapid Prototyping | Capacity Building


Pulse Lab Network

Pulse Lab Jakarta…………October 2012 Pulse Lab Kampala……….January 2013 Other locations…………...???


PULSE LABS R&D INNOVATION STRATEGY

1.  Partner with governments to establish Pulse Labs 2.  Build world-class teams of data scientists,

engineers, and policy experts 3.  Partner with private sector for real-time data and

cutting edge technology 4.  Work with UN agencies and academia to conduct

research around challenges in 5.  Build open source prototypes of tools to automatic

real-time monitoring 6.  Support broad adoption of useful innovations 7.  Share everything we learn and build


SO HOW DO I GET INVOLVED? ARE YOU.. •  A company with powerful data you think

could make the world a better place? •  A technology provider with screaming fast

computing or killer analytics? •  A whiz data scientist interested in hard

problems, positive impact, and global scale? •  A big data privacy expert who understands

that we cannot help unless we also protect? 25 October 2012 | www.unglobalpulse.org

Research Tool 1 Crimson Hexagon: ForSight


Food Prices: What a real crisis looks like

14 -‐ 21 Aug Ramadhan / Idul Fitri

23 July -‐ 2 Aug ‘tempeh’ and ‘tofu’ hot debate


Comparing Crises

23 July -‐ 2 Aug ‘tempeh’ and ‘tofu’ hot debate

during soybean shortage

18 Mar -‐ 7 Apr Fuel subsidy cut plan and

protests against it

Tweets about food

Research Tool 2 SAS Social Media Analytics and SAS Text Miner


Analytic Workflow

Internet Conversation

Global Pulse Relevance

Filter

Sentiment, Mood & Influence

Topic & Geography Categories

Interactive Dashboard

1) Over 200,000 new Indonesian language documents per day

3) Capture senKment and mood for Bahasa

2) Extract conversaKons about rice, cooking oil, fuel, employment, etc.

4) Detect locaKon, price, availability, specific govt. programs, etc.

5) Explore results and correlate with official staKsKcs to official BPS staKsKcs : Consumer Price Index (CPI) for 12 common foods

anxious, confident, confused, hosKle, sad, happy (-”-) ;-) ((+_+)) :@ :( :)

What’s the deal with Indonesians and eggs?

For every 5000 more tweets about eggs… …we see a 2-3% decrease in food CPI?

The Signals Are Getting Stronger

0

5000

10000

15000

20000

25000

30000

35000

40000

minyak (oil) ketahanan pangan (food security) budidaya (culKvaKon) telur (eggs)

è Big increase in volume of relevant conversations over 18 months

Indonesians are increasingly using social media to discuss basic needs


So Are the Temporal Correlations è Listening to social conversations provides insight on official data

-‐2.5

-‐2

-‐1.5

-‐1

-‐0.5

0

0.5

1

1.5

2

2.5

Social Media Food Index

BAPPENAS Food Price Index


Next up for Pulse Lab Jakarta research: 1 year of anonymized Indonesian CDRs

•  4 largest carriers •  170 million subscribers •  200 billion call records •  80 terabytes of data


Image credit: Aaron Koblin 24 hours of AT&T phone calls and Internet

traffic flowing through New York City

ROBERT KIRKPATRICK Director UN Global Pulse www.unglobalpulse.org [email protected] +1 (650) 796-5709

UN Global Pulse: Big Data for a Better World (Strata Conf NYC)

Documents

Transcript of UN Global Pulse: Big Data for a Better World (Strata Conf NYC)