www.london.ac.uk

RSS feeds using Millennium data

Andrew Preater, University of London Research Library Services

Presented at EIUG 2010, 15th June 2010

A short break in County Durham

I work for University of London Research Library Services, at Senate House

But I will talk about my previous development work at Durham University Library

Introduction

The problem is the new books list

We use these to list new items for readers as a current awareness tool

Various ways to do this...

A traditional approach

Durham’s new books featured list

Problems

High maintenance

Not split by subject; not easily ‘mashable’

Usage next to nothing by 2007-08

10 hits!

Puts our metadata where the reader is

Much less work for library staff

Standards-based XML data, can be reused elsewhere or mashed up

RSS feed improvements

RSS feed icon from www.feedicons.com

Low-risk pilot for automated export and processing of Millennium data

Demonstrates this utility for future projects

Quickest and easiest example using this approach

Project as proof of concept

Automated as much as possible

Minimal effort by non-systems staff to maintain

No special software – no budget!

Stable and reliable, ‘just works’

Desired outcomes

Other than Millennium...

1. Linux server with Perl installed

2. MySQL database

3. Web server running PHP

Software used

A featured list was created each week based on changing book item status to ‘d’

So a ‘new books’ review file was being made...

Basic idea

New step added: export the contents of the review file and reuse it

BIB MARC 245 $aBIB MARC 245BIB AUTHORBIB IMPRINTBIB SUBJECTBIB RECORD #ITEMFUND CODEITEMSHELFMARKITEMLOCATION

Export these fields

"Dead white men and other important people :"~"Dead white men and other important people : sociology's big ideas / Ralph Fevre and Angus Bancroft."~"Fevre, Ralph, 1955-"~"Basingstoke : Palgrave Macmillan, 2010."~"Social sciences -- Philosophy.";"Sociology."~"b25978974"~"bgsoc"~"300.1 FEV"~"main4"

Example single item

Perl script run every 15 minutes by cron:

1. Checks if there is a new file

2. Processes the data

3. Loads it into a MySQL database

4. Cleans up

Processing this list

1. Replace & with &

2. Insert RFC822-compliant date

3. Strip quotation marks around fields

4. Strip trailing non-alphanumeric character in 245 $a

5. Lowercase fund codes

Step 2: tidying up the data

Step 2: example single item

|Dead white men and other important people|Dead white men and other important people : sociology's big ideas / Ralph Fevre and Angus Bancroft.|Fevre, Ralph, 1955-|Basingstoke : Palgrave Macmillan, 2010.|Social sciences -- Philosophy.";"Sociology.|b25978974|bgsoc|300.1 FEV|main4|Mon, 07 Jun 10 12:31:01 BST|Mon, 07 Jun 10 12:31:01 BST|

Step 2: example single itemDead white men and other important people 245$aDead white men and other important people : sociology's big ideas / Ralph Fevre and Angus245 Bancroft.Fevre, Ralph, 1955- AuthorBasingstoke : Palgrave Macmillan, 2010. ImprintSocial sciences -- Philosophy.";"Sociology. Subjectb25978974 Record #bgsoc Fund code300.1 FEVShelfmarkmain4 LocationMon, 07 Jun 10 12:31:01 BST Date

Two tables are used:

items is refresh weekly: contains our books information

fundmap maps Millennium fund codes to subjects. Export is automated but doesn’t need to run weekly

Database

foobar

fundmap exampledeptcode

fundcode deptname site

ECON bcecoEconomics & Finance

DURHAM

HIST bchis History DURHAM

MEIS bbcmeGovt & Intl Affairs/IMEIS

DURHAM

MEIS bxabcGovt & Intl Affairs/IMEIS

DURHAM

CTV ctvl1Trevelyan College Library

DURHAM

Web front end

PHP script hosted on IT Service Web server will serve the feeds

http://www.dur.ac.uk/reading.list/newitems.php?dept=HIST

Parameter is ‘all’ or a subject code

1. Select items from database

2. Writes beautiful, valid RSS

3. Serves it up to the browser

What it does

A bit more detail...

Generating RSS feed XMLWrite <title>, <description> <link>, <image> once

Item <description> author, shelfmark and subjects hyperlinked to catalogue search.

Item <title> is 245 $a and links to catalogue bib record

Item <description> contains full item data. Can include encoded HTML

For each database line, write one RSS news <item> ...

Finished product - I

Shown in Akregator feed reader

Running happily since August 2007

Finished product - II

HTML version of RSS feeds on Library Web site

Also: in-house PC screensavers, plasma displays...

SummaryMillennium review file

Exported flat file

Display with Web front end

Process and load into database

Easiest to use Unicode everywhere

Write valid RSS 2.0 or Atom, use http://feedvalidator.org for hints

Few complaints; change uncovered a tiny hard core of featured lists fans

Lessons learned

That said...

You can automate much of it with Expect or AutoIt

Recommend Marc Dahl’s presentation on Expect for Innopac: http://bit.ly/dahl-expect

“Couldn’t you automate this?”

Automated export and processing used for:

Exporting Course Reserves to Blackboard

Display of e-resources data in CMS

Sending fines data to Oracle Financials

Following on from this...

Thank you!

Any questions?

Contact meEmail: andrew.preater@london.ac.ukTwitter: @preater