HathiTrust Research Center: The Fast Version

7
HathiTrust Research Center The Fast Version Robert H. McDonald | @mcdonald Executive Committee-HathiTrust Research Center (HTRC) Deputy Director-Data to Insight Center Associate Dean-University Libraries Indiana University

description

These are my pecha kucha slides for the Library of Congress Storage meeting held on Sept 21, 2012 in Washington, DC.

Transcript of HathiTrust Research Center: The Fast Version

Page 1: HathiTrust Research Center: The Fast Version

HathiTrust Research Center The Fast Version

Robert H McDonald | mcdonaldExecutive Committee-HathiTrust Research Center (HTRC)

Deputy Director-Data to Insight CenterAssociate Dean-University Libraries

Indiana University

HTRC Mission

The HathiTrust Research Center (HTRC) is a collaborative research center launched jointly by Indiana University and the University of Illinois to act as the public-facing research arm of the massive HathiTrust Digital Library

The HTRC is mandated to help researchers from around the world surmount the difficulties associated with processing and analyzing terascale amounts of digital text Thus the scholarly developers at HTRC work to develop cutting-edge software tools and cyberinfrastructure to enable advanced computational access to the growing digital record of human knowledge HTRC began its efforts July 2011

HTRC Non-Consumptive Research Paradigm

bull No action or set of actions on part of users either acting alone or in cooperation with other users over duration of one or multiple sessions can result in sufficient information gathered from collection of copyrighted works to reassemble pages from collection

bull Definition disallows collusion between users or accumulation of material over time Differentiates human researcher from proxy which is not a user Users are human beings

HTRC Current Infrastructure

bull Serversndash 14 production-level quad-core servers (virtual

machines)bull 16 ndash 32GB of memorybull 250 ndash 500GB of local disk each

ndash 6-node Cassandra cluster for volume storendash Ingest service and secure Data API access point

bull Storage (IU University Infrastructure)ndash 13TB of 15000 RPM SAS disk storagendash Increase up to 17TB by end of 2012ndash 500TB available in late year 2-year 3

HTRC Architecture

Data API access interface

Portal Access

Direct programmatic

access (by programs running

on HTRC machines)

Security (OAuth2)

Audit

Cassandra cluster volume store

Solr index

Algorithms

Result Sets

Meandre Workflows

Registry (WSO2)

Compute resources Storage resources

Agent

Job Submission

Collection building

Collections

Blacklight

Solr Proxy

HTRC Next Steps

bull Phase 2 Begins Jan 2013hellip

bull Thanks to

Contact Information

bull Robert H McDonaldndash Email ndash

robertindianaedundash Chat ndash rhmcdonald on

googletalk | skypendash Twitter - mcdonaldndash Blog ndash

httpwwwrmcdonaldnetndash Twitter Hashtag

HTRC12ndash Web

httpwwwhathitrustorghtrc

httpslideshareQCOrIX

  • HathiTrust Research Center The Fast Version
  • HTRC Mission
  • HTRC Non-Consumptive Research Paradigm
  • HTRC Current Infrastructure
  • HTRC Architecture
  • HTRC Next Steps
  • Contact Information
Page 2: HathiTrust Research Center: The Fast Version

HTRC Mission

The HathiTrust Research Center (HTRC) is a collaborative research center launched jointly by Indiana University and the University of Illinois to act as the public-facing research arm of the massive HathiTrust Digital Library

The HTRC is mandated to help researchers from around the world surmount the difficulties associated with processing and analyzing terascale amounts of digital text Thus the scholarly developers at HTRC work to develop cutting-edge software tools and cyberinfrastructure to enable advanced computational access to the growing digital record of human knowledge HTRC began its efforts July 2011

HTRC Non-Consumptive Research Paradigm

bull No action or set of actions on part of users either acting alone or in cooperation with other users over duration of one or multiple sessions can result in sufficient information gathered from collection of copyrighted works to reassemble pages from collection

bull Definition disallows collusion between users or accumulation of material over time Differentiates human researcher from proxy which is not a user Users are human beings

HTRC Current Infrastructure

bull Serversndash 14 production-level quad-core servers (virtual

machines)bull 16 ndash 32GB of memorybull 250 ndash 500GB of local disk each

ndash 6-node Cassandra cluster for volume storendash Ingest service and secure Data API access point

bull Storage (IU University Infrastructure)ndash 13TB of 15000 RPM SAS disk storagendash Increase up to 17TB by end of 2012ndash 500TB available in late year 2-year 3

HTRC Architecture

Data API access interface

Portal Access

Direct programmatic

access (by programs running

on HTRC machines)

Security (OAuth2)

Audit

Cassandra cluster volume store

Solr index

Algorithms

Result Sets

Meandre Workflows

Registry (WSO2)

Compute resources Storage resources

Agent

Job Submission

Collection building

Collections

Blacklight

Solr Proxy

HTRC Next Steps

bull Phase 2 Begins Jan 2013hellip

bull Thanks to

Contact Information

bull Robert H McDonaldndash Email ndash

robertindianaedundash Chat ndash rhmcdonald on

googletalk | skypendash Twitter - mcdonaldndash Blog ndash

httpwwwrmcdonaldnetndash Twitter Hashtag

HTRC12ndash Web

httpwwwhathitrustorghtrc

httpslideshareQCOrIX

  • HathiTrust Research Center The Fast Version
  • HTRC Mission
  • HTRC Non-Consumptive Research Paradigm
  • HTRC Current Infrastructure
  • HTRC Architecture
  • HTRC Next Steps
  • Contact Information
Page 3: HathiTrust Research Center: The Fast Version

HTRC Non-Consumptive Research Paradigm

bull No action or set of actions on part of users either acting alone or in cooperation with other users over duration of one or multiple sessions can result in sufficient information gathered from collection of copyrighted works to reassemble pages from collection

bull Definition disallows collusion between users or accumulation of material over time Differentiates human researcher from proxy which is not a user Users are human beings

HTRC Current Infrastructure

bull Serversndash 14 production-level quad-core servers (virtual

machines)bull 16 ndash 32GB of memorybull 250 ndash 500GB of local disk each

ndash 6-node Cassandra cluster for volume storendash Ingest service and secure Data API access point

bull Storage (IU University Infrastructure)ndash 13TB of 15000 RPM SAS disk storagendash Increase up to 17TB by end of 2012ndash 500TB available in late year 2-year 3

HTRC Architecture

Data API access interface

Portal Access

Direct programmatic

access (by programs running

on HTRC machines)

Security (OAuth2)

Audit

Cassandra cluster volume store

Solr index

Algorithms

Result Sets

Meandre Workflows

Registry (WSO2)

Compute resources Storage resources

Agent

Job Submission

Collection building

Collections

Blacklight

Solr Proxy

HTRC Next Steps

bull Phase 2 Begins Jan 2013hellip

bull Thanks to

Contact Information

bull Robert H McDonaldndash Email ndash

robertindianaedundash Chat ndash rhmcdonald on

googletalk | skypendash Twitter - mcdonaldndash Blog ndash

httpwwwrmcdonaldnetndash Twitter Hashtag

HTRC12ndash Web

httpwwwhathitrustorghtrc

httpslideshareQCOrIX

  • HathiTrust Research Center The Fast Version
  • HTRC Mission
  • HTRC Non-Consumptive Research Paradigm
  • HTRC Current Infrastructure
  • HTRC Architecture
  • HTRC Next Steps
  • Contact Information
Page 4: HathiTrust Research Center: The Fast Version

HTRC Current Infrastructure

bull Serversndash 14 production-level quad-core servers (virtual

machines)bull 16 ndash 32GB of memorybull 250 ndash 500GB of local disk each

ndash 6-node Cassandra cluster for volume storendash Ingest service and secure Data API access point

bull Storage (IU University Infrastructure)ndash 13TB of 15000 RPM SAS disk storagendash Increase up to 17TB by end of 2012ndash 500TB available in late year 2-year 3

HTRC Architecture

Data API access interface

Portal Access

Direct programmatic

access (by programs running

on HTRC machines)

Security (OAuth2)

Audit

Cassandra cluster volume store

Solr index

Algorithms

Result Sets

Meandre Workflows

Registry (WSO2)

Compute resources Storage resources

Agent

Job Submission

Collection building

Collections

Blacklight

Solr Proxy

HTRC Next Steps

bull Phase 2 Begins Jan 2013hellip

bull Thanks to

Contact Information

bull Robert H McDonaldndash Email ndash

robertindianaedundash Chat ndash rhmcdonald on

googletalk | skypendash Twitter - mcdonaldndash Blog ndash

httpwwwrmcdonaldnetndash Twitter Hashtag

HTRC12ndash Web

httpwwwhathitrustorghtrc

httpslideshareQCOrIX

  • HathiTrust Research Center The Fast Version
  • HTRC Mission
  • HTRC Non-Consumptive Research Paradigm
  • HTRC Current Infrastructure
  • HTRC Architecture
  • HTRC Next Steps
  • Contact Information
Page 5: HathiTrust Research Center: The Fast Version

HTRC Architecture

Data API access interface

Portal Access

Direct programmatic

access (by programs running

on HTRC machines)

Security (OAuth2)

Audit

Cassandra cluster volume store

Solr index

Algorithms

Result Sets

Meandre Workflows

Registry (WSO2)

Compute resources Storage resources

Agent

Job Submission

Collection building

Collections

Blacklight

Solr Proxy

HTRC Next Steps

bull Phase 2 Begins Jan 2013hellip

bull Thanks to

Contact Information

bull Robert H McDonaldndash Email ndash

robertindianaedundash Chat ndash rhmcdonald on

googletalk | skypendash Twitter - mcdonaldndash Blog ndash

httpwwwrmcdonaldnetndash Twitter Hashtag

HTRC12ndash Web

httpwwwhathitrustorghtrc

httpslideshareQCOrIX

  • HathiTrust Research Center The Fast Version
  • HTRC Mission
  • HTRC Non-Consumptive Research Paradigm
  • HTRC Current Infrastructure
  • HTRC Architecture
  • HTRC Next Steps
  • Contact Information
Page 6: HathiTrust Research Center: The Fast Version

HTRC Next Steps

bull Phase 2 Begins Jan 2013hellip

bull Thanks to

Contact Information

bull Robert H McDonaldndash Email ndash

robertindianaedundash Chat ndash rhmcdonald on

googletalk | skypendash Twitter - mcdonaldndash Blog ndash

httpwwwrmcdonaldnetndash Twitter Hashtag

HTRC12ndash Web

httpwwwhathitrustorghtrc

httpslideshareQCOrIX

  • HathiTrust Research Center The Fast Version
  • HTRC Mission
  • HTRC Non-Consumptive Research Paradigm
  • HTRC Current Infrastructure
  • HTRC Architecture
  • HTRC Next Steps
  • Contact Information
Page 7: HathiTrust Research Center: The Fast Version

Contact Information

bull Robert H McDonaldndash Email ndash

robertindianaedundash Chat ndash rhmcdonald on

googletalk | skypendash Twitter - mcdonaldndash Blog ndash

httpwwwrmcdonaldnetndash Twitter Hashtag

HTRC12ndash Web

httpwwwhathitrustorghtrc

httpslideshareQCOrIX

  • HathiTrust Research Center The Fast Version
  • HTRC Mission
  • HTRC Non-Consumptive Research Paradigm
  • HTRC Current Infrastructure
  • HTRC Architecture
  • HTRC Next Steps
  • Contact Information

HATHITRUST A Shared Digital Repository How Can Digital Collections Support Shared Print Initiatives? The HathiTrust Print Monograph Archive Planning Task.

HATHITRUST A Shared Digital Repository How Can Digital Collections Support Shared Print Initiatives? The HathiTrust Print Monograph Archive Planning Task.

This Library Never Forgets Preservation, Cooperation, and the Making of HathiTrust Digital Library Jeremy York Project Librarian HathiTrust Digital Library.

This Library Never Forgets Preservation, Cooperation, and the Making of HathiTrust Digital Library Jeremy York Project Librarian HathiTrust Digital Library.

HATHITRUST A Shared Digital Repository Collaborating Globally, Planning Locally HathiTrust and New Opportunities in Collection Management GWLA/UNM: Emerging.

HATHITRUST A Shared Digital Repository Collaborating Globally, Planning Locally HathiTrust and New Opportunities in Collection Management GWLA/UNM: Emerging.

HathiTrust Research Center Architecture Overview Robert H. McDonald | @mcdonald Executive Committee-HathiTrust Research Center (HTRC) Deputy Director-Data.

HathiTrust Research Center Architecture Overview Robert H. McDonald | @mcdonald Executive Committee-HathiTrust Research Center (HTRC) Deputy Director-Data.

HATHITRUST A Shared Digital Repository HathiTrust METS and PREMIS October 25, 2011 Jeremy York Project Librarian, HathiTrust.

HATHITRUST A Shared Digital Repository HathiTrust METS and PREMIS October 25, 2011 Jeremy York Project Librarian, HathiTrust.

HATHITRUST A Shared Digital Repository More, Better, Together: HathiTrust Accomplishments and Aspirations The Researcher of Tomorrow Universidad Complutense.

HATHITRUST A Shared Digital Repository More, Better, Together: HathiTrust Accomplishments and Aspirations The Researcher of Tomorrow Universidad Complutense.

HathiTrust 2017 Member Meeng Charng our Strategic Direcons · HathiTrust: Collaborang to advance research, teaching, and learning using the digital record of human knowledge. HathiTrust

HathiTrust 2017 Member Meeng Charng our Strategic Direcons · HathiTrust: Collaborang to advance research, teaching, and learning using the digital record of human knowledge. HathiTrust

Plale HathiTrust El Colegio de Mexico May2014

Plale HathiTrust El Colegio de Mexico May2014

HATHITRUST A Shared Digital Repository HathiTrust Overview: Partnership and Services Jeremy York Wesleyan University Web Presentation February 18, 2014.

HATHITRUST A Shared Digital Repository HathiTrust Overview: Partnership and Services Jeremy York Wesleyan University Web Presentation February 18, 2014.

HATHITRUST A Shared Digital Repository HathiTrust Update January 30, 2015 Mike Furlough, Executive Director Sharon Farb, HathiTrust Collections Committee.

HATHITRUST A Shared Digital Repository HathiTrust Update January 30, 2015 Mike Furlough, Executive Director Sharon Farb, HathiTrust Collections Committee.

LCA HathiTrust Amicus Brief vs Preliminary Judgment

LCA HathiTrust Amicus Brief vs Preliminary Judgment

HATHITRUST A Shared Digital Repository HathiTrust and the Future of Research Libraries American Antiquarian Society March 31, 2012 Jeremy York, Project.

HATHITRUST A Shared Digital Repository HathiTrust and the Future of Research Libraries American Antiquarian Society March 31, 2012 Jeremy York, Project.

Opportuni)es,and,Challenges,of,Text, Mining,HathiTrust ...#HTRC''@HathiTrust' HathiTrust repository is a latent goldmine for text mining analysis, analysis of large-scale corpi through

Opportuni)es,and,Challenges,of,Text, Mining,HathiTrust ...#HTRC''@HathiTrust' HathiTrust repository is a latent goldmine for text mining analysis, analysis of large-scale corpi through

furlough Hathi WRLC Library MAR82015 all - HathiTrust › documents › HathiTrust-WRLC-20150310… · HATHITRUST A Shared Digital Repository HathiTrustand** Collec2ve*Stewardship*atScale*

furlough Hathi WRLC Library MAR82015 all - HathiTrust › documents › HathiTrust-WRLC-20150310… · HATHITRUST A Shared Digital Repository HathiTrustand** Collec2ve*Stewardship*atScale*

HathiTrust Supplemental Metadata

HathiTrust Supplemental Metadata

Institutional Uses of HathiTrust

Institutional Uses of HathiTrust

HATHITRUST A Shared Digital Repository HathiTrust: Key Concepts and Issues in Managing the Digital Archive ICPSR Summer Workshop “Curating and Managing.

HATHITRUST A Shared Digital Repository HathiTrust: Key Concepts and Issues in Managing the Digital Archive ICPSR Summer Workshop “Curating and Managing.

HATHITRUST A Shared Digital Repository A Preservation Infrastructure Built to Last: Preservation, Community, and HathiTrust UNESCO Memory of the World.

HATHITRUST A Shared Digital Repository A Preservation Infrastructure Built to Last: Preservation, Community, and HathiTrust UNESCO Memory of the World.

Minnesota Digital Library and HathiTrust

Minnesota Digital Library and HathiTrust

HathiTrust Research Center Data Capsule Overview 09.10.14

HathiTrust Research Center Data Capsule Overview 09.10.14