1

Open Data, Visualization &Usability for Online News Delivery a presentation by Mohan & Sze

2

Background

3

• Lack of structured access to information on local news sites

• POV of two technologists whose interests in technology and current issues/activism overlap

• Propagation of more aware and savvy communities

Who are we?

4

Questions

5

• know HTML and/or CSS?

• know Javascript?

• know server side programming?

• maintain a news site / website?

• monetize your content?

How many of you…

6

Open data

Visualisation

7

Your data wants to be free

8

Why?

You cannot define how people will consume your information, leverage them

• Don’t turn your user base into pirates

Why?

Monetization cannot solely be a result of exclusivity but a result of mass access and authority

• The more people use your data, the more valuable you become (advertising)

• How are you making your data (not just website) more accessible

Why?

The NY Times said the project had met expectations, drawing 227,000 paying subscribers — out of 787,000 over all — and generating about $10 million a year in revenue.

“But our projections for growth on that paid subscriber base were low, compared to the growth of online advertising...”

“...what wasn’t anticipated was the explosion in

how much of our traffic would be generated by Google, by Yahoo and some others.” Vivian L. Schiller, senior VP and GM of NYTimes.com

If you ain’t the top dog, use openness as a way to partner, grow market share and relevance

• Even Google does it - OpenSocial

• Leverage exiting platforms - don’t roll another SNS

Why?

No matter how smart you are, there are smarter people out there

• Do you want them to be your competitor or partner

• Who will be making your killer Facebook application ?

Why ?

17

How?

How

• Provide a way to access data without presentation clutter. Turn your sites into information services

• Make it structured & machine readable - Web APIs, Microformats

How

• Provide simple ways to export and import data into your system

• Syndication, public wikis

• Provide ways to simplify the querying of your indexed data

• Opensearch, Better back end search

How

21

Microformats

• Small bits of HTML that represent things like people, events, tags, etc. in web pages. (annotations)

• These small bit of standardized markup adds semantic meaning to data in a page in a non-intrusive and machine readable manner, data becomes more structured

• A bottom up approach - community defines what and how the markup will be

Microformats

23

Let’s jump right in!

<table class="contentpaneopen"> <tbody> <tr> <td colspan="2" class="createdate" valign="top">Wednesday, 28 May 2008 </tr> <tr> <td colspan="2" valign="top"> <p><img src="forum_location_files/pjmap.jpg" alt="Image" title="Image" border="0" height="367" hspace="6" width="497"></p> <p>Shah Village Hotel<br>No. 3 &amp; 5 Lorong Sultan , 46200 Petaling Jaya, Selangor D.E. Malaysia </p> <p>Hotel Information: <a href="http://www.hspj.com.my/HSGS/index.php">Singgahsana Hotel</a>,&nbsp;<a href="http://www.shahsresorts.com/petalingjaya/default.htm">Shah's Village Hotel</a></p> </td> </tr> <tr> <td colspan="2" class="modifydate" align="left">Last Updated ( Friday, 06 June 2008 )</td> </tr> </tbody></table>

<table class="contentpaneopen"> <tbody> <tr> <td colspan="2" class="createdate" valign="top">Wednesday, 28 May 2008 </tr> <tr> <td colspan="2" valign="top"> <p><img src="forum_location_files/pjmap.jpg" alt="Image" title="Image" border="0" height="367" hspace="6" width="497"></p> <p>Shah Village Hotel<br>No. 3 &amp; 5 Lorong Sultan , 46200 Petaling Jaya, Selangor D.E. Malaysia </p> <p>Hotel Information: <a href="http://www.hspj.com.my/HSGS/index.php">Singgahsana Hotel</a>,&nbsp;<a href="http://www.shahsresorts.com/petalingjaya/default.htm">Shah's Village Hotel</a></p> </td> </tr> <tr> <td colspan="2" class="modifydate" align="left">Last Updated ( Friday, 06 June 2008 )</td> </tr> </tbody></table>

<div class=“vcard”>

Shah Village Hotel<br>No. 3 &amp; 5 Lorong Sultan , 46200 Petaling Jaya, Selangor D.E. Malaysia

</div>

<a href="http://www.shahsresorts.com/petalingjaya/default.htm">Shah's Village Hotel</a>

<div class="vcard">

Shah Village Hotel

No. 3 &amp; 5 Lorong Sultan , 46200 Petaling Jaya, Selangor D.E. Malaysia

</div>

<a href="http://www.shahsresorts.com/petalingjaya/default.htm">Shah's Village Hotel</a>

<div class=“vcard”>

Shah Village Hotel<br>No. 3 &amp; 5 Lorong Sultan , 46200 Petaling Jaya, Selangor D.E. Malaysia

</div>

<a href="http://www.shahsresorts.com/petalingjaya/default.htm">Shah's Village Hotel</a>

<div class="vcard">

<div class="fn org"> Shah Village Hotel</div>

No. 3 &amp; 5 Lorong Sultan , 46200 Petaling Jaya, Selangor D.E. Malaysia

</div>

<a href="http://www.shahsresorts.com/petalingjaya/default.htm">Shah's Village Hotel</a>

<div class=“vcard”>

Shah Village Hotel<br>No. 3 &amp; 5 Lorong Sultan , 46200 Petaling Jaya, Selangor D.E. Malaysia

</div>

<a href="http://www.shahsresorts.com/petalingjaya/default.htm">Shah's Village Hotel</a>

<div class="vcard">

<div class="fn org">Shah Village Hotel</div>

<div class="adr"> No. 3 &amp; 5 Lorong Sultan , 46200 Petaling Jaya, Selangor D.E. Malaysia

</div>

</div>

<a href="http://www.shahsresorts.com/petalingjaya/default.htm">Shah's Village Hotel</a>

<div class=“vcard”>

Shah Village Hotel<br>No. 3 &amp; 5 Lorong Sultan , 46200 Petaling Jaya, Selangor D.E. Malaysia

</div>

<a href="http://www.shahsresorts.com/petalingjaya/default.htm">Shah's Village Hotel</a>

<div class="vcard"> <div class="fn org">Shah Village Hotel</div> <div class="adr"> <div class="street-address"> No. 3 &amp; 5 Lorong Sultan </div>,<span class="postal-code"> 46200 </span> <span class="locality"> Petaling Jaya </span>, <span class="region"> Selangor D.E. </span> <span class="country-name"> Malaysia </span> </div> </div><a href="http://www.shahsresorts.com/petalingjaya/default.htm">Shah's Village Hotel</a>

<div class=“vcard”> <a href= "http://www.shahsresorts.com/petalingjaya/default.htm" class=“url fn org”>Shah Village Hotel</a> <div class="adr"> <div class="street-address"> No. 3 &amp; 5 Lorong Sultan </div>, <span class="postal-code"> 46200 </span> <span class="locality"> Petaling Jaya </span> , <span class="region"> Selangor D.E. </span> <span class="country-name"> Malaysia </span> </div> </div>

<div class=“vcard”><abbr title="3.102233488513198;101.64600670337677"class="geo"><a href= "http://www.shahsresorts.com/petalingjaya/default.htm" class=“url fn org”>Shah Village Hotel</a> </abbr> <div class="adr"> <div class="street-address"> No. 3 &amp; 5 Lorong Sultan </div>, <span class="postal-code"> 46200 </span> <span class="locality"> Petaling Jaya </span> , <span class="region"> Selangor D.E. </span> <span class="country-name"> Malaysia </span> </div> </div>

32

Second example

• Search engine support

• http://kitchen.technorati.com/search/

• google creative commons search

• yahoo search support

Why?

34

• Machine readable

• Cheap API - Low barrier for community to build add ons to your site tools in (Javascript - FF3 , PHP etc.)

Why?

35

• End user application support (Firefox3 with extension,IE8 OpenService - WebSlices , soon more apps)

Integrate into news sites:

•Listings of classifieds - hListings

•News stories - markup with hAtom, markup authors,personalities and organizations with hCard, markup locations with geo and adr

•Reviews of products,restaurants, movies - hReview

e.g. http://www.australianit.news.com.au/contactus

How to incorporate

How to incorporate

Build tools internally that leverage the markup

•Internal search engine that is markup aware for your site

•Build visualisation tools/widgets that leverage the markup

How to incorporate

Build communities that leverage the markup

•Have a developer section on your site

• DreamweaverExtension - http://www.webstandards.org/action/dwtf/microformats

• Textpattern Microformats Plugin (http://placenamehere.com/TXP/pnh_mf/)

• Wordpress Microformat plugins - http://microformats.org/wiki/wordpress

• TYPO3 - tt_address , TIMTAB Weblog plugins

• Joomla - No support

• Drupal - Drupal Upcoming.org syndication module-http://hybernaut.com/upcoming-hcal - Weak support

• PHP Lib - http://enarion.net/phpmicroformats/ (Note: Search for otherlanguage libs to generate microfrmats)

Produce

• Client end

• Microformat bookmarklet - http://leftlogic.com/lounge/articles/microformats_bookmarklet

• Firefox add on - Operator-http://www.kaply.com/weblog/operator/

• Server end

• Python based microformat parser - http://code.google.com/p/aump/

• PHP based microformat parser - http://code.google.com/p/hkit/

• TransFormr -A Simple set of XSLT and PHP tools for Transforming

• Microformats - http://code.google.com/p/transformr/

• http://suda.co.uk/projects/x2v - webservices

Consume:

• Weak tool-ing for production in open source components such as Joomlaand Drupal

Challenges:

SyndicationAccess of content + metadata through feeds. A data API for your site.

Users will consume your data in the way they find most comfortable. Syndication allows developers/users to build the right method for themselves. (Mobile readers, Screen readers). Bottom line - increase in circulation.

Most Malaysian news sites do it badly

Why?

Why?

Additional metadata contained within a feedallows for remixing and discovery. Categories and tags (metadata) should be taken seriously. (Technorati, wordpress,yahoo pipes )

Why?

Allows for 3rd party applets/widgets to be built on top of your content

• Integrate into news sites:

• Syndicate everything on your site

• All sections , all columns ,search results, listings (classifieds etc.)

How:

• Drupal Atom module - http://drupal.org/project/atom - Overall good support

• Joomla - DS-Syndicate-http://extensions.joomla.org/component/option,com_mtree/task,viewlink/link_id,2497/Itemid,35/(Only Joomla 1.0 though) - Bad support

• Wordpress - Good support

• Universal Feed Generator -http://www.phpclasses.org/browse/package/4427.html - Roll your own

Produce

•Simplepie PHP library - http://simplepie.org/development/

•Drupal - FeedAPI - http://drupal.org/project/feedapi

Consume

•Use Atom 1.0

•Cache, cache, cache (Support conditional GETs, Slugs etc.)

•Autodiscover links, on all pages - leverage support in browsers

•Use service to redistribute - such as Feedburner

Best practice:

Monetization:

If people aren't returning to my site for content, how do I monetize ?

Challenges:

A collection of simple formats that help searchengines share their search results.

3.0 OpenSearch

Browser support is already available

• Firefox, IE7 onwards.

Why?

Why?

Leverage search aggregation, deep search

• Growing field - A9, Technorati, MS Search Server 2008 OpenSearch

Why?

Weak search access results in searching through Google or Yahoo?

• Your customers are looking for your data on other sites - hence you are loosing usage metrics as well as monetization opportunities.

• Integrate into news sites:

• Make search a 1st class interface to your website.

• Why are you not using this browser UI element ?

How

62

63

• Ensure everything is indexed and can be searched

• Elevate search from an afterthought into a first class access tool of your site

How

• Textmate - wet_opensearch plugin

• Drupal - OpenSearch Plugin - - http://drupal.org/project/opensearchplugin

• Online creator tool - http://ready.to/search/en/

• Wordpress - plugin - http://wordpress.org/extend/plugins/open-search-document/

Produce

Drupal - OpenSearch client - http://drupal.org/project/opensearchclient

Consume

1. Use autodiscovery links

2. Write informative OpenSearch description documents

3. Provide atom + microformat + extensions (geo) based search resultswhere applicable

Best practice

67

Open data

Visualisation

68

Bad design obscures.

Good design reveals.

69

• Design is about constraints – understanding, respecting and breaking.

• An effective design allows users to get what they’re looking for in an easy, trusted and fast manner.

So, what’s the deal with design?

70

Traditional vs social mediaHow are they different?

71

72

73

1. Dynamic content (static vs changeable)

2. Interactivity (comments)

3. Relevance / Popularity (unique visits, comments, linkbacks)

4. Accessibility (archives)

5. Mashup (mix media, mix services – ads, widgets)

6. Creative freedom

7. Continuous presence

8. Reusable (syndicatable, linkable, tagable)

74

Social media = interactions between people using

applications / interfaces

75

• improve the interactions between audience

• prioritize information – hierarchical structure – popularity, relevance, recency et. al.

• allow users to reorganise and remix content and represent content in a manner they desire.

Why?

76

SIMILE Timelinea Javascript-based widget for visualizing time-based information

77

• linear narrative – visual representation of chronological events

• allows users to be able to appreciate and interact with news in a more fluid, continuous form – better context to events & stories

• Mashup – information can be easily extracted from different sources (e.g. http://www.bbc.co.uk/history/british/launch_tl_british.shtml )

Why?

78

• http://simile.mit.edu/timeline/

• Timeline is similar to Google Maps: zero application installation, server-side/client-side

• Data is easily populated into Timeline using: RDF, RSS, JSON, Atom, XML

How

79

<html>

<head>

...

<script src="http://simile.mit.edu/timeline/api/timeline-api.js“ type="text/javascript"> </script>

...

</head>

<body>

...

</body>

</html>

80

<html>

<head>

...

<script src="http://simile.mit.edu/timeline/api/timeline-api.js“ type="text/javascript"> </script>

...

</head>

<body>

...

</body>

</html>

81

82

Timestampinga Javascript-based widget for visualizing time-based information

83

• Video markers create easy transitions for readers to move between different paragraphs of a news article and its corresponding video clip

• Limited time - short attention span – users has the freedom to view specific portions of a clip

Why?

84

• Interactive markers – viddler.com allows users to

1. attach comment markers at any point of the video clip;

2. include tags in the video timeline to describe elements in the video clip

How:

85

86

87

88

89

Google MapsBuilding humane interfaces by respecting standards and

habituation

90

• Topographical view of how news is scattered allows for readers to focus on hyper local content

• Maps also serve as infographics to augment readers’ experience in relating or understanding news, especially if there is a location specific context e.g. Cyclone Nargis (http://www.nytimes.com/interactive/2008/05/08/world/20080508_MYANMAR.html)

Why?

91

• zero application installation, server-side/client-side

• data is easily populated with XML

How:

92

• Human readable also machine readable

• Better manipulate the data - move towards machines autonomously aggregating, remixing and manipulating data

• Make your data accessible - make yourself important

• Monetize by selling attention -but don’t lose relevance

• Data + Visualization = More attention

Summary

Contact details

• Mohan

• mohangk@gmail.com

• Sze

• szeying@gmail.com