Alma 1 year after STP: implementing batch services

IGeLU Budapest Sep 2, 2015

Bart PeetersHead Operations LIBIS

Agenda

• Introduction• The LIBISnet Consortium• 3 main areas of improvement for batch services:

A. Internal Workflow Automation IssuesB. Import-Export IssuesC. Consortial Model Issues

• Conclusion – Wrap Up

Agenda

• Introduction• The LIBISnet Consortium• 3 main areas of improvement for batch services:

A. Internal Workflow Automation IssuesB. Import-Export IssuesC. Consortial Model Issues

• Conclusion – Wrap Up

Introduction

• Some areas where batch services are needed:– Import– Export– (Bibliographic) maintenance

• Consortial context (IZ – NZ)• Existing tools• Use cases• Suggestions for enhancements

Introduction

• Existing tools

Agenda

• Introduction• The LIBISnet Consortium• 3 main areas of improvement for batch services:

A. Internal Workflow Automation IssuesB. Import-Export IssuesC. Consortial Model Issues

• Conclusion – Wrap Up

The LIBISnet Consortium• 32 Alma Institutions• > 85 sublibraries• Approx. 6 000 000 shared bibliographical records:

– 60% books– 15% articles– 8% dissertations– 4.5% Journal Titles– 2.5% Images– 10% Other (Games, Schoolbooks, Norms, Text Resource, Audiovisual, Video, Audio,Rare

Books, Scores, Maps, Websites, Chapters, Manuscripts, Other)

• Approx. 75% physical titles• Approx. 25% electronic titles• Approx. 7 000 000 physical items• Approx. 1 500 000 electronic portfolios

The LIBISnet Consortium

The LIBISnet Consortium

The LIBISnet Consortium

The LIBISnet Consortium

The LIBISnet Consortium

Agenda

• Introduction• The LIBISnet Consortium• 3 main areas of improvement for batch services:

A. Internal Workflow Automation IssuesB. Import-Export IssuesC. Consortial Model Issues

• Conclusion – Wrap Up

A) Internal workflow automation issues

• General principle: support full automation of end-to-end workflows via scripting, requiring no manual startup or interventions

• Use case: update of odis subjects

A) Internal workflow automation issues

• Use case: update of odis subjects

A) Internal workflow automation issues

• Current method:– Weekly file with subject updates from odis– Full export of KADOC BIB records (650_7 $$6 is not indexed)– Binary MARC => (Aleph) sequential format (easy ‘grep’)– File: Alma mms id | odis nbr– Match in UNIX with odis file– Result: Alma mms id | updated odis subjects– Old odis subjects replaced– Import in Alma via merge routine=> Not all these steps can be completely automated, manual intervention needed in Alma UI

A) Internal workflow automation issues

• Suggestions:1. APIs to interact with the job management system (“job

management APIs”)• Supply job parameters• Upload file(s)• Submit job• Retrieve status of job (also while job is running)• Retrieve job report• Download files

A) Internal workflow automation issues

• Suggestions:2. APIs to interact with the set management system (“set

management APIs”)• Supply set creation parameters• Upload file(s) with Ids (only for itemized sets)• Create itemized or logical set• Retrieve set contents (list of Ids)• Delete set (could be temporary set with search results)

• Other use case: automatic import of electronic dissertations from SAP to Alma and Rosetta

Agenda

• Introduction• The LIBISnet Consortium• 3 main areas of improvement for batch services:

A. Internal Workflow Automation IssuesB. Import-Export IssuesC. Consortial Model Issues

• Conclusion – Wrap Up

B) Import / export issues

• General principle: provide enough features to avoid extreme use of APIs– Slow– Tresholds:

• Daily API request treshold (500 calls per number of named user licenses)

• Concurrent API request treshold (10 calls per institution per second)

• Use case: import of a new collection/library with BIB, HOL and ITEM data. Issue: HOL data.

B) Import / export issues

• Current method:– Creation of 1 file with BIB and ITEM data (in 9XX field with

different subfields mapped to item fields). – Import in Alma, HOL records are created automatically, only

852 field, we need more ...– Via API retrieval of HOL record nbrs– Via API export of HOL records– Update HOL records– Via API update of HOL records

B) Import / export issues

• Suggestions:3. Job for matching bib records in input file with bib records

in Alma • with reporting (matched | not matched | multiply matched

records)• Possibility to create sets of matched | not matched | multiply

matched records• Integrated in import jobs (with option “do not import” upon match

| upon no match)

B) Import / export issues

• Suggestions:4. Full support of physical HOL records (same level as BIB

records)• Sets of HOL records (~ set of ITEM records)• Importing job for MARC HOL (new, updates)• Exporting job for MARC HOL

5. Export BIB records with fields from HOL records, ITEM records, e-portfolio records, ... (expand option)

6. Import BIB records together with HOL records, ITEM records, e-portfolio records, ...

Agenda

• Introduction• The LIBISnet Consortium• 3 main areas of improvement for batch services:

A. Internal Workflow Automation IssuesB. Import-Export IssuesC. Consortial Model Issues

• Conclusion – Wrap Up

C) Consortial model issues

• General principle: remove “the wall” between the institution zones and the network zone.

• Use case: “batch” update BIB records.– Set are often created in the institution zone(s) by making use

of HOL and ITEM indexes– Updates need to be done in the network zone– IZ MMS ID ≠ NZ MMS ID

C) Consortial model issues

• Current method:– Creation of IZ set (in Alma UI by librarian)– Export of the IZ set *– Retrieval via Primo API => PNX records returned that include

the NZ MMS ID *– Creation of NZ set (via upload of file with NZ MMS IDs, no

more limit of 5000 records )– Apply normalisation rule on set or – Import with update

C) Consortial model issues

• Suggestions:7. Include NZ MMS IDs when exporting IZ records8. Include NZ MMS IDs when API getting record from IZ in a

way that works for all records (new Alma records and migrated ALEPH records)

9. Include IZ MMS IDs when exporting record from NZ for all or some of the member institutions

10. API to get list of institutions having local bib records for given bib mms id and (optionally) the mms ids of these local records

C) Consortial model issues

• Suggestions:11. Creating set in NZ from set in IZ12. Creating set in IZ(s) from set in NZ13. Export NZ BIB records with information from IZ records

• local bib recs, holding recs, item recs, e-portfolio recs, ... • for all or some of the member institutions

14. Import NZ BIB records together with new IZ records • local bib recs, holding recs, item recs, e-portfolio recs, ... • for all or some of the member institutions

C) Consortial model issues

• Other use cases:– Exports of NZ BIB records with IZ HOL info:

• Antilope (union catalogue of periodicals in Belgian academic and special libraries)

• bib.belgium.be (the common catalogue of the Belgian federal libraries)

– Import of journal articles in NZ and linking to IZ

Agenda

• Introduction• The LIBISnet Consortium• 3 main areas of improvement for batch services:

A. Internal Workflow Automation IssuesB. Import-Export IssuesC. Consortial Model Issues

• Conclusion – Wrap Up

Conclusion

• More use cases• Suggestions are not beatific or the only solutions

possible• APIs: offer great potential, but:

– Tresholds– Relatively slow– When something goes wrong ...

• Do you have suggestions?

Bart.Peeters@libis.kuleuven.beDirk.Kinnaes@libis.kuleuven.be

Thank you!Questions?