The Gory Details Of Datajournalism: what went wrong and who was responsible by Henk van Ess

Data Reality CheckHenk van Ess [email protected]

chairman WWW.VVOJ.NL

Sunday, March 25, 12

mailto:[email protected]

mailto:[email protected]

http://WWW.VVOJ.NL

http://WWW.VVOJ.NL

Andrew Lehren, NYT

Age of Data by Geoff McGhee Paul Bradshaw

Digital Diggers ProjectData Driven Stories Book

http://www.vvoj.nl

This year: a new book & site about data journalism


http://www.vvoj.nl

http://www.vvoj.nl

Let’s cut the crap


Let’s cut the crap

Who was in charge and what went wrong?


The CoderThe ReporterThe Designer





Showing what is possible, not what is sensible




Assembly line syndrom: no place for new workflows


Do you use tools like:


Do you use tools like:before



List of dangerous chemical companies

before




before

after



Adobe Captivate


before

after


There are many new dedicated tools



Timeline the bad way



Timeline the bad way Timeline with Timeline 3D



Timeline the bad way Timeline with Timeline 3D

This was done in 30 minutes




Mind boggling presentations.. leaving your mind boggled




Showing problems, not explaining them


when no one is in charge (anymore)



empty pages



broken databases empty pages


Datajournalism is


Finding information

Datajournalism is


Finding information

Processing information

Datajournalism is


Finding information


Presenting information

Datajournalism is


Put like that, it seems the job description for any reporter

Finding information


Presenting information

Datajournalism is


To be more specific, you need:



A fine researcher



A fine researcherA smooth talker



A fine researcherA smooth talkerA good number cruncher



A fine researcherA smooth talkerA good number cruncherA great story teller



A fine researcherA smooth talkerA good number cruncherA great story tellerA brilliant designer/ GUI specialist



A fine researcherA smooth talkerA good number cruncherA great story tellerA brilliant designer/ GUI specialistFantastic communicator




“overlapping set of competencies drawn from disparate fields” http://bit.ly/nLlbYn


http://bit.ly/nLlbYn




You offer:“overlapping set of competencies drawn from disparate fields” http://bit.ly/nLlbYn




Your problem

Data journalism is a “baggy monster of a field”5 tips for getting started in data journalism

http://www.poynter.org/how-tos/digital-strategies/147734/5-tips-for-getting-started-in-data-journalism/




We can learn from Nokia, for once :

Go for teamwork!


The gory details


The problem


The problem

Reporters bored with Wikileaks


WHY?

Most explosive documents were already published


WHY?

Finding new leads in new documents was hard


Colleagues told me: ‘It isn’t worth our time’


Me, being shocked


I said:

Me, being shocked


I said:

If you don’t check this, who does?????Me, being shocked


How we started:


No inhouse coders

How we started:


No inhouse codersNo designers

How we started:


No inhouse codersNo designersNo funding

How we started:


No inhouse codersNo designersNo fundingNo technical infrastructure

How we started:


How it ended:

http://cablesearch.org/?page_id=46


http://screencast.com/t/1scMTaBgZytF


So what happened here?


Text

www.cablesearch.org


http://www.cablesearch.org

http://www.cablesearch.org

How it started





Why did this succeed?


It solved a problem



It solved a problem


For news beats


It solved a problem


For news beats

By tags, for specialist & background stories


It solved a problemSharing with public gave us great new ideas



It solved a real problemCrowdsource!The Business plan paid for infrastructure



Even more gory details


Volkskrant:


Volkskrant:

• Newspaper demanded tons of data about the state of elderly homes in The Netherlands


The response


Volkskrant


Volkskrant• Pushed over and over again to get the

data


The response


Volkskrant• Invoked the freedom of information act


The response


Volkskrant• They kept pushing

39


After one year


After one year Finally. Succes!


A cd full of data


• 270.000 questions• 320.000 answers• Devided in 9 categories • More then 12 subcategories


Overwhelmed by data:• 270.000 questions• 320.000 answers• Devided in 9 categories • More then 12 subcategories


Minor detail: all 4500 forms are PDF


So what now?


It’s 2012. Don’t despair


Anatomy of a projectI posted the pitch for the project:

“Need coder for converting 4500 PDF’s”


One minute later


The Whole World Is Bidding


And Bidding


The result was:


The result was:• Over 60 programmers wanted to convert the

data into Excel and Web (MySql) for a price of $100 -$3000

• Choose the ones that are in the top 300 (out of 200.000 coders) to avoid surprises


Topcoder for $500


Out of 9 projects


Out of 9 projects

• 8 were a success• 1 was a miserable failure • Total number of publications: 7, 1 pending


Don’t forget

Always 2x coders for the same jobPerfect Pitch is ParamountDataminded reporter should do the pitch


Project lessonsHire coders, don’t become oneDo get the concepts behind data mining so you can instruct the coder Outsource smaller projects can be done


Back to the Volkskrant


The first job was done in 5 days


The first job was done in 5 days• PDF converted to normal text


Other steps:


Other steps:

• Text converted to Excel• Excel converted to MySql• Search engine for reporters enabled

them to ‘Google’ data of the homes - later used as public search


FINALLY SOME ANSWERS


• How many homes have financial troubles?• How many aggressive people caused

trouble?• How many actions of non-medical

employees were not backed up by a doctor?

• Which elderly home is the best/worst?

FINALLY SOME ANSWERS


Unexpected questions:

• Why are homes with a high mortality rate?• Why are homes with a sickness absence

of 15-20% ?


Results

• Two lead news stories on the front page• A special section about the best/worst

homes• ‘Top 300’ on the web for consumers:

where do I put daddy away• A lot of goodwill


The lessons:


The lessons:

• 1.Datamining is the start of the project, not the end

• 2 Reporters write the story• 3 Public comments on story - new stories


School dataThe 2x 500 euro Solution


What ifYou want to report on a regular basis on the same

subject?


Spidering or custom programming ?

Roughly 7500 records in two spreadsheets


How it was done

Brin codes identify school

http://functiemix.minocw.nl/resultaten.aspx?criteria=1&type=1&brin=00AH


How it was done: securing school codes

List of BRIN codes

ScreenScraper

BuildURL

Req.Page

GetPage

Build Report


How it was done: Building URL (from list)

List of BRIN codes

ScreenScraper

BuildURL

Req.Page

GetPage

Build Report


How it was done: Request web page

List of BRIN codes

ScreenScraper

BuildURL

Req.Page

GetPage

Build Report


How it was done: Get & parse web page

List of BRIN codes

ScreenScraper

BuildURL

Req.Page

GetPage

Build Report


How it was done: Build report

List of BRIN codes

ScreenScraper

BuildURL

Req.Page

GetPage

Build Report


How it was done: REPEAT until done

List of BRIN codes

ScreenScraper

BuildURL

Req.Page

GetPage

Build Report


Things to consider


Things to consider• Creates competitive advantage


Things to consider• Creates competitive advantage• - Saves time


Things to consider• Creates competitive advantage• - Saves time• - More accurate


Things to consider• Creates competitive advantage• - Saves time• - More accurate• - Not tedious, no insult to intelligence


Things to consider• Creates competitive advantage• - Saves time• - More accurate• - Not tedious, no insult to intelligence• Legal ramifications



– Always be respectful of your source



– Always be respectful of your source – You can’t copyright facts



– Always be respectful of your source – You can’t copyright facts– You can copyright how facts are presented



– Always be respectful of your source – You can’t copyright facts– You can copyright how facts are presented– Don’t make yourself conspicuous


LessonsBuild once, use often

Generic & important: zip code / geodata / feedback


The 300 euro project


11/20/08 12:51 Henk van Ess

digital digging: converting data

Source: Funda

Technique: ‘datascraping’

Makelaars manipuleren woningaanbod ROTTERDAM, 3 JAN. NVM-makelaars en huizenverkopers manipuleren met te koop staande huizen, door ze een paar dagen van de markt te halen en daarna als nieuw aan te prijzen. De afgelopen drie maanden is dat over heel Nederland verspreid meer dan 2.700 keer gebeurd


• Script was written to download the data of more then 300.000 houses during three

months

• The results were put in an Excel file.

• Reporters browsed manually for anomalies


Results


Results• Two frontpage stories• 1 Appendix• Higher circulation


Nice, but..

Can’t you do it cheaper?


The 0 euro-projects


european databases


Creative thinking


Creative thinking

• What databases are there online in Europe?

• Hmm, there is a register of all people who attended meetings, a presence register.


11/20/08 12:51 Henk van Ess

So what??• What can you do with such a register?• Study which members are there and which not• Study which countries are more absent than others• How big is the effect if certain members of countries

aren’t voting at all?• In reality some countries have more influence then

you would expect from the actual number of seats


11/20/08 12:51 Henk van Ess

CAR: do try this at home! The Netherlands has

more power

Source: EU: database, available in 20

languages


Elaborate work..


Healthcare regulators


11/20/08 12:51 Henk van Ess11/20/08 12:51 Henk van Ess [email protected]



Tuchtcollege voor de Gezondheidszorg



Verdicts against doctors• Uitspraken van het CTG• <http://www.tuchtcollege-gezondheidszorg.nl/recent_files/recent.htm>De meest

recente uitspraken<http://www.tuchtcollege-gezondheidszorg.nl/recent_files/recent.htm>

• <http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/huisarts.htm>Uitspraken over huisartsen en basisartsen<http://www.tuchtcollege-

gezondheidszorg.nl/huisarts_files/huisarts.htm>• <http://www.tuchtcollege-gezondheidszorg.nl/specialisten_files/

SPECIALISTEN.HTM>Uitspraken over medische specialisten<http://www.tuchtcollege-gezondheidszorg.nl/specialisten_files/SPECIALISTEN.HTM>

• Uitspraken over psychiaters<http://www.tuchtcollege-gezondheidszorg.nl/psychiater.htm>

• <http://www.tuchtcollege-gezondheidszorg.nl/tandarts_files/Tandartsen.htm>Uitspraken over tandartsen<http://www.tuchtcollege-

gezondheidszorg.nl/tandarts_files/Tandartsen.htm>• <http://www.tuchtcollege-gezondheidszorg.nl/verpleegkundigen.htm>Uitspraken

over verpleegkundigen<http://www.tuchtcollege-gezondheidszorg.nl/verpleegkundigen.htm>

• <http://www.tuchtcollege-gezondheidszorg.nl/verloskundigen.htm>Uitspraken over verloskundigen<http://www.tuchtcollege-gezondheidszorg.nl/verloskundigen.htm>

• <http://www.tuchtcollege-gezondheidszorg.nl/apotheker.htm>Uitspraken over apothekers<http://www.tuchtcollege-gezondheidszorg.nl/apotheker.htm>



Total chaos•

2005.009 en 046<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.009%2520en%2520046.htm>2005.012<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.012.htm>2005.013<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.013.htm>2005.014<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.014.htm>2005.015<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.015.htm>2005.016<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.016.htm>2005.017<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.017.htm>2005.018<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.018.htm>2005.019<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.019.htm>2005.020<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.020.htm>2005.032<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.032.htm>2005.033<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.033.htm>2005.034<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.034.htm>2005.055<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.055.htm>2005.109<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.109.htm>2005.110<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.110.htm>2005.151<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.151.htm>2005.152<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.152.htm>2005.155<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.155.htm>2005.178<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.178.htm>2005.211<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.211.htm>2005.223<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.223.htm>2005.224<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.224.htm> 2004.005<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.005.htm>2004.012<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.012.htm>2004.014<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.014.htm>2004.021<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.021.htm>2004.022<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.022.htm>2004.023<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.023.htm>2004.024<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.024.htm>2004.025<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.025.htm>2004.026<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.026.htm>2004.027<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2004.027.htm>2004.028<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/



Step 1: download the site



Step 2



Step 2• Download a desktop

search tool



Step 3



Step 3 Start typing questions as you do in Google



Like this:


DIFFERENCES BETWEEN COURT RULINGS


DIFFERENCES BETWEEN COURT RULINGS

Coverstory


This presentation:

HENK VAN ESSwww.searchbistro.com

[email protected]@henkvaness

http://www.slideshare.net/searchbistro






















The Gory Details Of Datajournalism: what went wrong and who was responsible by Henk van Ess

Education

Transcript of The Gory Details Of Datajournalism: what went wrong and who was responsible by Henk van Ess