CWSpaceArchiving Courseware Websites to DSpace

Using a Content Packaging Profile

& Web Services

©

Digital Library Federation Spring Forum, 2006

April 11, 2006 — Austin, Texas

William Reilly, Rob Wolfe — MIT Libraries

2006

DLF Spring 2006 MIT's CWSpace 2

Project Overview

Vision: Increase value in Institute publication: OCW– Also: Increase

value in Open Source platform: DSpace

* CLE = Collaborative Learning Environment

Goal: Archive OCW to MIT’s DSpace– Also: Make content

available to CLEs *

CWSpace: MIT iCampus project, nearing completion

Method: Interoperability of systems– Use Content Packaging standard– Use Web Services standard

DLF Spring 2006 MIT's CWSpace 3

Perceived Need: inter-operability™

• Obvious observation: Higher Ed. Technologies increasingly seeking integration with Library & Repository holdings

• Not so obvious(?): Libraries seeing their charter to include collection, preservation, dissemination of Teaching & Learning materials (?)

DLF Spring 2006 MIT's CWSpace 4

InterOp: What’s Available

DLF Spring 2006 MIT's CWSpace 5

InterOp: What’s Achievable 1 of 3

DLF Spring 2006 MIT's CWSpace 6

InterOp: Challenges 2 of 3

DLF Spring 2006 MIT's CWSpace 7

InterOp: Possible Resolution? 3 of 3

DLF Spring 2006 MIT's CWSpace 8

OCW Materials in DSpace = New Ground

Educational Materials Digital Archive

New kind of content: educational, teaching, learning

POLICY - Cost/merit evaluation of archival treatment. Preservation and IP considerations

New emphasis (ideally) on re-use, re-purposing: aggregation, disaggregation

GRANULARITY - Challenge to digital archive; how flexible can it be? (cf. CMS)

New kind of structure, composition: website, compound digital object, rich metadata

COMPLEXITY - Need to package multi-file content, plus metadata

Content PackagingNew needs for system-to-system access: CLEs, Image Tools

BACK-END / FRONT-END - Need for networked access

Web Services

DLF Spring 2006 MIT's CWSpace 9

Content Packaging

DLF Spring 2006 MIT's CWSpace 10

Content Packaging: Steps Toward a Profile

• Model OCW Content

• Choose Packaging Specification

• Profile That Specification

DLF Spring 2006 MIT's CWSpace 11

Content Package

Package Interchange File (PIF)

• Usually a .zip file• Contains a manifest• Manifest references

files (some in, some out of the package)

DLF Spring 2006 MIT's CWSpace 12

Object Models

• Mapping one world to another

• Content Package design considerations

• Use, re-use, subsequent disseminationOCW DSpace

SIP DIP

…

…

DLF Spring 2006 MIT's CWSpace 13

OCW to DSpace mapping

DLF Spring 2006 MIT's CWSpace 14

Which Packaging Spec.?

• Content packaging specifications are generic loose envelopes to carry content, metadata

• Choice of package is largely determined by user community

• Dspace to use METS internally• Higher Education uses IMS

Content Packaging

• METS• IMS-CP• MPEG-21

DIDL• XFDU

DLF Spring 2006 MIT's CWSpace 15

METS and IMS-CP

DLF Spring 2006 MIT's CWSpace 16

IMS-CP & OCW Content Model

DLF Spring 2006 MIT's CWSpace 17

IMS-CP & DSpace Object Model

DLF Spring 2006 MIT's CWSpace 18

OCW to IMS-CP to DSpace

DLF Spring 2006 MIT's CWSpace 19

Profiling Package Standards

Every Implementation of IMS-CP or METS will:

• Constrain the Standard• Extend the Standard• Interpret the Standard

The sum of these adjustments is the implementation profile

The Standards themselves encourage profiling in a systematic way:

IMS Application Profile Guidelines Technical Manual

http://imsglobal.org/ap/apv1p0/imsap_techv1p0.html

METS Profile Schema and Guidelines

http://www.loc.gov/standards/mets/mets-profiles.html

DLF Spring 2006 MIT's CWSpace 20

Application Profiles

DLF Spring 2006 MIT's CWSpace 21

Application Profiles

DLF Spring 2006 MIT's CWSpace 22

Application Profiles

DLF Spring 2006 MIT's CWSpace 23

Application Profiles

DLF Spring 2006 MIT's CWSpace 24

Application Profiles

DLF Spring 2006 MIT's CWSpace 25

Profiled Content Package…

Now with Package defined…

…need to move that across the network from producer to consumer… DSpace

Content Provider

DLF Spring 2006 MIT's CWSpace 26

Web Services

DLF Spring 2006 MIT's CWSpace 27

DSpace dev: LNI & PKGers

DLF Spring 2006 MIT's CWSpace 28

Lightweight Network Interface

0. What Was The Problem?

1. What Is the LNI?

2. DSpace & WebDAV

3. Example of Use: Submit OCW

4. Future Development…

DLF Spring 2006 MIT's CWSpace 29

What was the Problem…?Content Providers wanted…• Networked Access: Programmatically access the DSpace object model, but not

running on the DSpace server

Basic (Lightweight Network Interface (LNI))

[ Access, Deposit, Limited Mods. ]

• Submit

• Retrieve Item w. Metadata

• Build own U/I

• Collection Mgt, Map Items

Advanced (possible future LNI dev.)

[ Replace, Withdraw, Delete ]

• Specialist metadata

• Automated metadata

• Quality control checks

• Format migration …

DSpace

Content Provider

DLF Spring 2006 MIT's CWSpace 30

What Is the LNI?

The "Lightweight Network Interface" is:

• A remote API to control DSpace

• It's essentially WebDAV: Extensions to HTTP

– A SOAP (with WSDL) equivalent is provided

• It's a mapping: —

– a "complete and comprehensive" view of the DSpace public object

API…

(the "business logic" layer)

– …as mapped to WebDAV semantics

DLF Spring 2006 MIT's CWSpace 31

DSpace: Model• DSpace digital repository• Differs from file system• Submit URI not known• Workflow• …

DLF Spring 2006 MIT's CWSpace 32

WebDAV: Model, Basics• Each Resource

named by a URL

• Properties are "name value" pairs

• Properties are extensible

• Property values can be found, and modified

DAV = "Distributed Authoring and Versioning"

URL

DLF Spring 2006 MIT's CWSpace 33

WebDAV: Model, Fuller• Resources

can be Collections or leaf nodes

• Collections can nest

• Everything can be discovered descending from the DAV server's root '/' resource

DLF Spring 2006 MIT's CWSpace 34

DSpace Mapped to WebDAV• WebDAV

DLF Spring 2006 MIT's CWSpace 35

DSpace Implemented as WebDAV

1. Properties– WebDAV– Custom for DSpace

2. LNI URLs

3. WebDAV Methods

DLF Spring 2006 MIT's CWSpace 36

Properties: WebDAV

• DAV:creationdate• DAV:displayname• DAV:getcontentlength• DAV:getcontenttype• DAV:getlastmodified• DAV:resourcetype• …

http://www.webdav.org/specs/rfc2518.html

Akin to file system properties,with add'l. metadata

DLF Spring 2006 MIT's CWSpace 37

Properties: Custom DSpace (1 of 3)

• ALL DSpace Objects:– dspace:type

• "BITSTREAM", "ITEM", "COMMUNITY" etc.

– DAV:resourcetype • "COLLECTION" etc.

– dspace:displayname– dspace:current-user-privilege-set

DLF Spring 2006 MIT's CWSpace 38

Properties: Custom DSpace (2 of 3)

• Community DSpace Objects– dspace:logo– dspace:short_description– dspace:side_bar_text– dspace:handle– …

DLF Spring 2006 MIT's CWSpace 39

Properties: Custom DSpace (3 of 3)

• Bitstream DSpace Objects– DAV:getcontentlength = getSize()– dspace:format = getFormat().getID()– dspace:sequence_id = getSequenceID()– …

• Many more DSpace Objects (some 10)– With many more Properties (some 87)

http://wiki.dspace.org/LightweightNetworkInterface

DLF Spring 2006 MIT's CWSpace 40

URLs: DSpace LNI• Community

– http://uni.edu/dspace/dav/dso_1721.1$46

• Collection– http://uni.edu/dspace/dav/dso_1721.1$349

• Item– http://uni.edu/dspace/dav/dso_1721.1$5543

• Bitstream– http://uni.edu/dspace/dav/dso_1721.1$5543/bitstream_13

The LNI URLs for DSpace objects are only for use with the LNI.

• They are therefore not to be used:

• as persistent URLs (e.g. bookmark, store for later use)

• as interactive URLs (e.g. browser)

DLF Spring 2006 MIT's CWSpace 41

WebDAV Methods in LNI• PROPFIND - very powerful request with multiple functions:

– list names of properties– return property values– recurse through 1 or more levels of a "collection" resource.

• PROPPATCH - changes value of or deletes properties on one resource.• GET - retrieves contents of a resource• PUT - replace or add a new resource to a collection• COPY - map a resource ("Item") to a different Collection.

As noted, equivalent SOAP methods are provided

Not Implemented: Locking, Versioning.

DLF Spring 2006 MIT's CWSpace 42

Current DSpace (pre-LNI)Content Provider, currently

• Batch import (& export)

• End user Web U/I

• dspace-admin Web U/I

• SRW, OAI-PMH

DSpace

Batch Importer

OCW Comm.

"Biology (7)"

"Physics (8)"

DSpace Object API

Web U/I Web, butNon Auto-

mated

Non Std.Non Web

Workflow Adm

Item

DLF Spring 2006 MIT's CWSpace 43

DSpace with LNIContent Provider using LNI:• LNI SOAP end-point URL

• LNI WebDAV URLs

• dspace-admin (partial)

• SRW, OAI-PMH– (Not replicated in LNI)

• Still available: – Batch import (& export)– End user Web U/I

DSpace

DSpaceLNI.class

OCW Comm.

"Biology (7)"

DSpace Object API

DSpace LNI WebDAV

- Web.- Stds.

- Batch-able.

"Physics (8)"

Item

DLF Spring 2006 MIT's CWSpace 44

Submission: OverviewOpenCourseWare Client

Needs 3 Values:

1. LNI SOAP end-point URL http://userid:password@dspace.mit.edu/dspace-lni/lni/DSpaceLNI

2. Handle for OCW Community = hdl:1721.1/125

3. Dept. Name (MIT "Course" Number) for Course to be submitted = "Physics (8)"

DSpace

DSpaceLNI.class

OCW Comm.1721.1/125

"Biology (7)"

DSpace Object API

DSpace LNI WebDAV

Lang. -Indep.

Platform -Indep.

"Physics (8)"

Item

DLF Spring 2006 MIT's CWSpace 45

Submission: 1. LookupRequest: OCW Comm. HANDLE

LNI lookup() = http://dspace.mit.edu/dspace-lni/dav/lookup/handle/1721.1/125

Response: OCW Comm. LNI URL http://dspace.mit.edu/dav/dso_1721.1$125

DSpace

DSpaceLNI.class

OCW Comm.1721.1/125

dso_1721.1$125

"Biology (7)"dso_1721.1$197

"Physics (8)"dso_1721.1$253

DSpace Object API

DSpace LNI WebDAV

Lookup

DLF Spring 2006 MIT's CWSpace 46

Submission: 2. CollectionsRequest: PROPFIND on LNI URL for children DSpace

Collections' "displaynames":<propfind>

<prop>

<DAV:displayname/>

</prop>

</propfind>

Response: MULTISTATUS<multiStatus>

<resource>

<href URI="dso_1721.1$197"/>

<displayname>Biology (7)</displayname>

</resource>

<resource>

<href URI="dso_1721.1$253"/>

<displayname>Physics (8)</displayname> </resource>...

DSpace

DSpaceLNI.class

OCW Comm.1721.1/125

dso_1721.1$125

"Biology (7)"dso_1721.1$197

"Physics (8)"dso_1721.1$253

DSpace Object API

DSpace LNI WebDAV

Iterate…

DLF Spring 2006 MIT's CWSpace 47

Submission: 3. Submit (PUT)Request: PUT to DSpace Collection LNI URL,

including specification of Packager Type (IMS-CP):

PUT /dspace/dav/dso_1721.1$253?package=OCW-IMSCP

....package contents in body...

Response: LNI URL for DSpace Item in "Location" Header:

HTTP/1.1 201 OK

Location: http://dspace.mit.edu/dspace/dav/dso_1721.1$836

....other headers....

DSpace

DSpaceLNI.class

OCW Comm.1721.1/125

dso_1721.1$125

"Physics (8)"dso_1721.1$253

DSpace Object API

DSpace LNI WebDAV

IngestItemdso_1721.1$836

DLF Spring 2006 MIT's CWSpace 48

Submission: 4. Item HandleRequest: PROPFIND on DSpace Item LNI URL for the DSpace

Handle:

<propfind>

<prop>

<dspace:handle/> </prop>

</propfind>

Response: Properties Requested

=== PROPERTIES Successfully returned:

dspace:handle = "hdl:1721.1/836"

DSpace

DSpaceLNI.class

OCW Comm.1721.1/125

dso_1721.1$125

"Physics (8)"dso_1721.1$253

DSpace Object API

DSpace LNI WebDAV

Reverse Lookup

Itemdso_1721.1$836

1721.1/836

OCW Content Mgt. System

Stores DSpace Handle

SQL Server Table: hdl:1721.1/836

DLF Spring 2006 MIT's CWSpace 49

Submission: SummaryIn this 4-step OpenCourseWare

submission example, we've seen:• SOAP endpoint URL• Dspace Handles• Dspace LNI URLs• LNI Lookup() function• WebDAV Properties• WebDAV Methods• PROPFIND (LNI request)• MULTISTATUS (LNI response)• LNI "Reverse Lookup"• WebDAV Headers (Location;

Success/Fail)

DSpace

DSpaceLNI.class

OCW Comm.1721.1/125

"Physics (8)"1721.1/253

DSpace Object API

DSpace LNI WebDAV

Item1721.1/836

DLF Spring 2006 MIT's CWSpace 50

Future Development

• Other Use Cases, Needs– Basic, Advanced …

• Other DSpace objects– Bitstream Format Registry– Metadata Registry …

• Other WebDAV methods– PUT (for Replacement)– DELETE– MKCOL …

CWSpaceArchiving Courseware Websites to DSpace

©

Digital Library Federation Spring Forum, 2006

April 11, 2006 — Austin, Texas

William Reilly, Rob Wolfe — MIT Libraries

2006

Questions, & Etc.

http://cwspace.mit.edu/docs/ProjectMgt/Reports/MIT-CWSpace-DLF-Spring2006.ppt

V. 20060410_1510