Working out the Future of Library Resource Discovery

Marshall BreedingIndependent Consultant,Founder and Publisher, Library Technology Guideshttp://www.librarytechnology.org/http://twitter.com/mbreeding

October 5, 2015NISO Event: Future of Library Resource Discovery

http://www.niso.org/publications/white_papers/discovery/

DescriptionMarshall Breeding will highlight some of the key findings of the white paper he developed for the NISO Discovery to Delivery topic committee. The presentation will include some updated information on the state of the current arena of commercial and open source discovery services, including trends in adoption and new technical and functional capabilities. Looking forward, Breeding will mention some longer-term possibilities and opportunities for discovery services to move beyond the current models of centralized indexes, including greater reliance on semantic technologies and linked data.

Library Discovery Past

Bound Catalog

National Library of Colombia

Card Catalog

National Library of Argentina

Online Catalog

Books, Journals, and Media at the Title Level

Not in scope: Articles Book Chapters Digital objects

Scope of SearchSearch:

Search Results

ILS Data

NOTIS: MDAS Multiple Database Access System Released in 1989 Article-level indexing (Mostly Wilson

Databases) Grant supported by Pew Charitable Trusts Development Partners: NOTIS and

Vanderbilt UniversitySee: Steffey, RJ. “NOTIS multiple database access system: a look behind the scenes” Online , v14 n5 p46-49 Sep 1990

Next-gen Catalogs or Discovery Interface

Single search box Query tools

Did you mean Type-ahead

Relevance ranked results Faceted navigation Enhanced visual displays

Cover art Summaries, reviews,

Recommendation services

Books, Journals, and Media at the Title Level

Other local and open access content

Not in scope: Articles Book Chapters Digital objects

Scope of Search

Discovery Interface search modelSearch: Digital

Collections

ProQuest

EBSCOhost

…MLA

Bibliography

ABC-CLIO

Search Results

Real-time query and responses

ILS Data

Local Index

MetaSearch Engine

Web-scale Index-based DiscoverySearch:

Digital Collections

Web Site Content

Institutional

Repositories

…E-Journals

Reference Sources

Search Results

Pre-built harvesting and indexing

Consolidated Index

ILS Data

Aggregated Content packages

(2009- present)

Usage-generate

dData

Customer

Profile

Open Access

Evaluating the Performance of Index-based Discovery Services

Intense competition: how well the index covers the body of scholarly content stands as a key differentiator

Difficult to evaluate based on numbers of items indexed alone.

Important to ascertain how your library’s content packages are represented by the discovery service.

Important to know what items are indexed by citation, which are full text, and how A&I content is handled

Open Discovery InitiativeLibraries

Publishers

Service Providers

14

Marshall Breeding, Vanderbilt UniversityJamene Brooks-Kieffer, Kansas State University Laura Morse, Harvard UniversityKen Varnum, University of Michigan

Sara Brownmiller, University of OregonLucy Harrison, College Center for Library Automation (D2D liaison/observer)Michele Newberry

Lettie Conrad, SAGE PublicationsRoger Schonfeld, ITHAKA/JSTOR/PorticoJeff Lang, Thomson Reuters

Linda Beebe, American Psychological AssocAaron Wood, Alexander Street Press

Jenny Walker, Ex Libris GroupJohn Law, Serials SolutionsMichael Gorrell, EBSCO Information Services

David Lindahl, University of Rochester (XC)Jeff Penka, OCLC (D2D liaison/observer)

The Context for ODI Based on a meeting at ALA Annual Conference in

New Orleans on Sunday, June 26, 2011. Recognition of the following trends and issues: Emergence of Library Discovery Services solutions

Based on index of a wide range of content Commercial and open access Primary journal literature, e-books, and more

Adopted by thousands of libraries around the world, and impact millions of users

Agreements between content providers and discovery providers ad-hoc, not representative of all content, and opaque to customers.

15

ODI deliverables Standard vocabulary NISO Recommended Practice:

Data format & transfer Communicating content rights Levels of indexing, content availability Linking to content Usage statistics Evaluate compliance

Inform and Promote Adoption

16

ODI Recommended Practices Published June 25, 2014 NISO RP-19-2014

http://www.niso.org/workrooms/odi / http://www.niso.org/workrooms/odi/publications/rp/rp-19-2014

Metadata elements for content providers to contribute to discovery service providers

Content providers disclose extent to which they participate with each discovery service

Discovery Service providers disclose what content is represented in index

Discovery services disclose any bias in search results or relevancy relative to business relationships

Discovery services provide use statistics

NISO Discovery White Paper Commissioned by NISO Discovery to

Delivery Topic Committee First Draft Nov 2014

Revised based on feedback from D2D Published Feb 20, 2015 Launched at ER&L

NISO Discovery Paper Outline General Background Integration between Discovery Services and

Management Systems Linked Data Gap Analysis Opportunities for Future Enhancements in discovery Discovery Beyond Library-provided Interfaces Open Discovery Initiative: recommendations for

Phase II Longer term prospects

Library Discovery Present

Library Perspective Strategic investments in subscriptions Strategic investments in Discovery Solutions to provide

access to their collections Expect comprehensive representation of resources in

discovery indexes Problem with access to resources not represented in index Encourage all publishers to participate and to lower

thresholds of technical involvement and clarify the business rules associated with involvement

Need to be able to evaluate the coverage and performance of competing index-based discovery products

Value and Economy Academic and research libraries spend far more

of their budgets on content than resource management or discovery technologies

Discovery represents essential infrastructure to maximize impact of library collections

Resource management represents essential infrastructure to assemble and assess optimal collection to support library mission

Ever increasing costs of content exert pressure on budgets and demand more effective discovery and more efficient management

Role of the library in discovery Acquisition and Management of

resources Integrate content into campus enterprise

infrastructure and information architecture

Provide general and specialized interfaces

Participate in production and publication Participate more deeply in research

process Manage content on behalf on the

institution in ways that optimize access and discovery.

Web-scale Index-based DiscoverySearch:

Digital Collections

Web Site Content

Institutional

Repositories

…E-Journals

Reference Sources

Search Results

Pre-built harvesting and indexing

Consolidated Index

ILS Data

Aggregated Content packages

(2009- present)

Usage-generate

dData

Customer

Profile

Open Access

Bento Box Discovery ModelSearch:

Digital Collections

Web Site Content

Institutional

Repositories

E-JournalsSearch Results

Pre-built harvesting and indexing

Consolidated IndexILS Data

Aggregated Content packages

Open AccessVuFind /

Blacklight

State of Discovery indexes Very strong coverage of primary

publishers of scholarly materials Especially English and other Western

Languages Weaker coverage of scholarly content in

other international regions Asian languages, Arabic, etc.

Mixed coverage of A&I resources Mixed converge of non-textual resources

Some Key Areas for Publishers1. Expose content appropriately2. Trust that access to material will be

controlled consistent with subscription terms

3. “Fair” Linking4. Materials not disadvantaged or

underrepresented in library discovery implementations

5. Usage reporting

Representation of A&I Important to understand how a discovery

service incorporates A&I resources Does it receive content from the A&I

provider directly and make use of value-added terminology

If not: citations or full-text indexing of some portion of the titles represented in the A&I product

NOT the same, and possibly misleading

28

A&I Content in Discovery Services

What is the place for A&I services in the discovery ecosystem

Are there technology solutions capable of substituting for A&I content? Specialized and scoped search

methodologies Clustering, term extraction, etc.?

Specialized vocabulary and other metadata make positive contributions to the discovery process

Researchers value A&I tools

ODI Standing CommitteeLibraries

Publishers

Service Providers

30

Marshall Breeding, Independent Consultant

Laura Morse, Harvard UniversityJason Price, SCELC

Ken Varnum, University of MichiganDave Whisenant, Florida Virtual Campus

Lettie Conrad, SAGE PublicationsMichael McFarland, CredoReferenceJill O’Neill, NFAIS

Elise Sassone, Springer

Aaron Wood, Ingram Content GroupJulie Zhu, IEEE

Scott Bernier, EBSCO Information ServicesSteven Guttman, Proquest

Rachel Kessler, Ex LIbris

John McCullough, OCLC

ODI Standing CommitteeThe Open Discovery Initiative Standing Committee was formed following approval of the Recommended Practice published by NISO on June 25, 2014

We are charged with the following tasks:

• Promotion and education of ODI Recommended Practice for all stakeholders• Provide support for content providers and discovery service providers during adoption and completion of conformance checklists• Provide a forum for ongoing discussion related to all aspects of discovery platforms for all stakeholders• Consider next steps for items deemed out scope from the original ODI Work Group Recommended Practice • Identify emerging needs in the open discovery space and determine appropriate courses of action• Make recommendations to the D2D topic committee on further work items required to fulfill the goals of the Open Discovery Initiative

31

Current issues and areas of development

Challenge for Relevancy Technically feasible to index hundreds of millions

or billions of records through Lucene or SOLR Difficult to order records in ways that make

sense Expectation that relevancy be neutral relative to

content source or publisher Many fairly equivalent candidates returned for

any given query Must rely on use-based and social factors to

improve relevancy rankings

Relevancy Ever-improving, yet flaws remain Increased use of use data and

personalize context to identify and order search results

State of the art improving via more sophisticated search and retrieval technology, increased use of aggregated contextualized data, and other factors

Socially-powered discovery Leverage use data to increase

effectiveness of discovery Usage data can identify important or

popular materials to inform relevancy engines

Identify related materials that may not otherwise be uncovered through keyword matching

Be careful to avoid introducing bias loops

Externalizing functionality Provide tools and widgets in course

management platforms Reading list management Improving presentation via mobile

devices

Open access content Only a minority of scholarly resources

available through open access licenses Difficult to identify open access versions

available Often presented proprietary content

when open access is also available

Interoperability of Discovery Services and Management Platforms Discovery and Management solutions offered as matched sets

Ex Libris: Primo / Alma ProQuest: Summon / Intota OCLC: WorldCat Discovery Service / WorldShare Platform

Independent Discovery and Management Kuali OLE: no discovery component EBSCO Discovery Service: Works with any Resource management

system Both product categories depend on an ecosystem of

interrelated knowledge bases API’s exposed to mix and match, but are efficiencies and

synergies are lost? Recommendation to explore expectation regarding

interoperability between these two product categories

Discovery Service Installations

Product 2007 2008 2009 2010 2011 2012 2013 2014 InstalledEBSCO EDS 1774 2634 8246Primo 12 37 53 506 111 101 98 88 1528AquaBrowser 55 339 64 69 74 58 81 6 89

Encore 72 72 109 56 72 36 346

BiblioCommons 41 ~200

Summon     50 164 214 158 238 195 697Enterprise   16  75 100 102 123 150 538

Infor Iguana     18 74

Axiell Arena     61 57 33 35 95 404

Gap Analysis Many resources still not addressed in central indexes

Especially A&I products Better coverage of open access materials Better support for internationalization and

multilingual search and retrieval Improved capabilities for precise search, known

items, browsing Improved and more transparent relevancy rankings Non-textual content and retrieval mechanisms Better integration with learning management

systems

Opportunities for Enhancements in Discovery

Improved delivery of APIs More coherent ecosystem of APIs among

discovery services and with resource management systems

Social features and scholarly collaboration

Address research data Special Collections and archival

materials: hierarchical discovery and browsing

Expanded Analytics and Altmetrics

Beyond Index-based Discovery

Library Discovery Future

The future of Resource Discovery More comprehensive discovery indexes Stronger technologies for search and

retrieval Discovery beyond library-provided

interfaces Linked Data to supplement discovery

indexes

Universal participation Barriers to participation soften as mutual interest

prevails over competitive conditions Advantage to content providers to maximize

exposure of resources Discovery providers gain value in functionality as

metadata becomes increasingly commoditized Essential to preserve value of indexing and

abstracting services Content providers see discovery as a essential

channel for distribution

More Distributed Discovery Address the reality that discovery takes

place outside of library provided interfaces

Optimized exposure in the ecosystem of search engine and social network

Not Concentrated on the Library web site Expression of discovery services via

other campus tools and portals and beyond

Multi-layered discovery Native interfaces of specialized content

services Disciplinary aggregations General library discovery tools Global Internet-based discovery

Discovery beyond Library Interfaces

Improved performance of library content through Google Scholar Same expectations for transparency?

Better exposure of library-oriented content Schema.org or other microdata formats

Better exposure of scholarly resources Open access & Proprietary

Embedded tools in other campus interfaces

Part of the General Internet Infrastructure

Scholarly content will be promoted via similar mechanisms as commercial content

Additional levels of infrastructure to protect privacy

Resource management and/or discovery tools expose content items as open linked data

Library opts out of Discovery Utrecht University Library Decision to not implement a discovery service

but to rely entirely on Google Scholar and other general and scholarly search engines

http://www.uu.nl/en/university-library/searching-for-literature/searching-for-articles-books-theses

Kortekaas , Simone. “Thinking the unthinkable: a library without a catalogue — Reconsidering the future of discovery tools for Utrecht University library.” LIBER General Annual Conference 2012

Linked Data Major trend toward information systems based on

linked data Many projects now based on linked data Area of peak interest for Library of Congress, OCLC, etc BIBFRAME

Potential to transform how libraries approach discovery

Likely interim hybrid models: central indexes + Linked Data

Current opportunities in making library content more discoverable

Library adoption of Linked data architecture

Not yet a fully operational method for library-oriented content Increasing representation of bibliographic

resources BIBFRAME stands to make great impact

Universe of scholarly resources not well represented

Will current expectations for content providers to make metadata or full text available for discovery expand to exposure as open linked data?

Hybrid models Can index-based search tools be

improved through Linked Data Browse to related resources Add additional hierarchies of structure to

search results

Will linked data models prevail? Possibility that open linked data may

eventually supplant index-based products?

Index technology supplements fundamental architecture based on linked data

Possibilities for Open Access discovery index Open source tools exist for discovery Interfaces:

VuFind Blacklight

No open access discovery indexes High threshold of expense and difficulty to build

index Platform costs Software development Publisher relations Billions of content items to index and maintain

Current model requires massive resources

Threshold of resources required currently too high for open access central discovery index

Assessment might change if options narrowed

Opportunities to lower barriers to entry? More open model more likely to come

through linked data discovery model

Commoditization of Central Indexes Knowledgebases of e-resource coverage

commoditized via KBART and other factors

Central index content likewise will eventually become commoditized

Limited number of discovery service platforms?

Value found in the synergies between library resource management and optimized discovery and delivery

Value in open scholarship Hopefully the future will be based on

open access to scholarly research Mandates from funding organizations will

transform scholarly communications Current discovery models based on

preponderance of proprietary content Future discovery must assume

dominance of open access publishing and underlying data sets

Future of discovery service products Remain one of the essential components of library

technology infrastructure Loosely or tightly tied to resource management Increased sophistication in direct discovery and

delivery functionality Increased expectation to syndicate content to local

and global discovery context Investments made in creation of discovery service

platforms will provide leverage into each next phase of scholarly information infrastructure

Scholarly publishing arena may change dramatically in next decade.

Open Discovery Initiative: recommendations for Phase II

Address A&I concerns to improve participation

Data exchange mechanisms: metadata + content Lower threshold of participation

Interoperability with resource management systems

Potential Opportunities for NISO Convene a second phase of the Open Discovery

Initiative Launch research project on open linked data in

scholarly publishing sector to facilitate new models of discovery and access

Expand scope of Altmetrics group to address their integration in discovery service ecosystem

Possible new workgroup to explore recommended practices for improving discoverability of resources via open linked data, schema.org, and other mechanisms.

Longer term prospects Opportunities for discovery directly tied

to realities in scholarly publishing Dominance of proprietary publishing

requires index-based discovery Future to open access and exposure as

open linked data will enable additional models of discovery

An ongoing conversation Now in a critical point for discovery

Current products evolve Reaching limits of the prevailing

architecture? Current set of products and services an

interim step Important for stakeholders to engage in

defining the future of library resource discovery

Future products must address expected changes in scholarly publishing, library priorities, and institutional strategies.

Questions and discussion