Connecting Applications with the NSDL Technical Network Services.

Connecting Applications with the NSDL

Technical Network Services

Annual Meeting 2008

NDR API

NCS

Presentation overview

• Introduction (Jim Blake)• Collaborative Tools

– NSDL Wiki (Katy Ginger)– ExpertVoices (Sharon Clark)– OnRamp (Lynette Rayle) – NSDL Collection System (Jonathan Ostwald)– Strand Map Service (John Weatherly)

• Backend Services– NDR API (Aaron Birkland) – Search Service (Jim Blake)

• Conclusion (Jim Blake)

Annual Meeting 2008

NDR API

NCS

TNS operations and services

• Operate infrastructure and library– Provide NCore platform for use by NSDL

grantees– Operate NSDL.org to support

dissemination of grantee collections

* Swiped out of context from yesterday's presentation by Lagoze, Krafft, Sumner and Wright

Annual Meeting 2008

NDR API

NCS

TNS operations and services – (paraphrased)

• Mix and Match software stack– "multi-level commoditization"

• Discoverability

Annual Meeting 2008

NDR API

NCS

The stack

Fedora

NDR API

NDR Toolkit

WordPressplug-ins

WordPress

Fedora

NDR API

NDR Toolkit

WordPressplug-ins

yourWordPress

Fedora

NDR API

NDR Toolkit

yourapplication your

application

Fedora

NDR API

Annual Meeting 2008

NDR API

NCS

Discoverability

• More than a repository, it's a brand name.

Annual Meeting 2008

NDR API

NCS

Discoverability

• More than a brand name, it's a web site with high page rank:

http://wiki.nsdl.org/index.php/How_To_Improve_Rankings_With_Blog_Posts

Annual Meeting 2008

NDR API

NCS

Discoverability

• Less obvious– Improved page rank

• All resources are reachable from the site maps at nsdl.org

– More hits• Site maps are crawled by Google and Yahoo.

Annual Meeting 2008

NDR API

NCS

Case study 1: NSDL.org

• Most of the NCore software was initially written to support NSDL.org

• Changing viewpoint:– from writing NCore for NSDL.org– to using NCore in NSDL.org

Annual Meeting 2008

NDR API

NCS

The NCore platform

Search Service

Strand Map

Service

NSDLCollection

System

NSDL Data Repository

NDR-API

Fedora: Native Interface

ExpertVoices

NSDLWiki

OAI/RSSIngest

OAIServer

OnRamp

Annual Meeting 2008

NDR API

NCS

Case study 2: IPY

• E-Zine

• Expert Voices blog

• Funded development for – OnRamp functionality– Expert Voices theme– etc.

Annual Meeting 2008

NDR API

NCS

e-zine via OnRamp

Annual Meeting 2008

NDR API

NCS

blog via Expert Voices

Annual Meeting 2008

NDR API

NCS

Case study 3:Curriculum Customization Service

• Digital library services w/o the NSDL Data Repository• Collection management with the NCS• Strand maps• DDS search (Digital Discovery System)

Annual Meeting 2008

NDR API

NCS

Concept structure via NCS

Annual Meeting 2008

NDR API

NCS

Education Standards via SMS

Annual Meeting 2008

NDR API

NCS

related resources via DDS

Annual Meeting 2008

NDR API

NCS

the "stack"

NCS SMS DDS

Annual Meeting 2008

NDR API

NCS






Annual Meeting 2008

NDR API

NCS

NSDL Wiki Features

• Collaborative online environment to organize, create, & annotate resources

• Create links directly to NSDL resources on your Wiki pages• Group specific spaces (restricting to a group is possible)• Individual user accounts• Search extension enables linking to NSDL

– find resources to talk about– insert links to resources

• NDR Extension enables the addition of:– new resources to the library– referenced resources to the library– metadata about resources

Annual Meeting 2008

NDR API

NCS

NSDL Wiki Uses• NSDL’s Classic Articles in Context (CAC)

– Landmark works of scientific inquiry via dynamic learning spaces for teachers and learners

– Links to related NSDL teaching resources and bibliographies of contemporary articles that cite classic papers

– A venue for collaborative discussion and contribution directed by its users

• NCORE documentation• MatDL – soft matter wiki is a local implementation by

an NSDL Pathway

Annual Meeting 2008

NDR API

NCS

Create DiscoverDescribe

Identify AnnotateRelateContributeRepresent

Aggregate

Annual Meeting 2008

NDR API

NCS

NDR API

Create DiscoverDescribe

Identify AnnotateRelateContributeRepresent

Aggregate

Annual Meeting 2008

NDR API

NCS

NSDL Wiki Technical• Based on MediaWiki 1.11 and 1.12

• Extensions are available to download from SourceForgehttps://sourceforge.net/project/showfiles.php?

group_id=204090&package_id=268147

• NSDL Wiki Documentation:https://wiki.nsdl.org/index.php/Community:NSDLWiki

Annual Meeting 2008

NDR API

NCS






Annual Meeting 2008

NDR API

NCS

Expert Voices• A blogging tool with integrated NSDL search,

resource linking, and publication to NDR

• Moderated conversations are designed to tie NSDL resources to science news and add context for resources that enhance discovery, selection, and use

EV has two interrelated goals: • provide a community discussion/presentation forum• create context for science resources in the library

Expert Voices Gateway: http://expertvoices.nsdl.org

Annual Meeting 2008

NDR API

NCS

Features

• Wordpress Multi-User• Published entries can become NSDL resources• Blog entries can have linked references to NSDL

resources• Levels of authority (Roles)• Integrated with NSDL Shibboleth-based community sign-

on (Wordpress plug-in)• Blog(s) available as RSS feed(s)

Annual Meeting 2008

NDR API

NCS

Example Blog Uses

• Connecting current news to related library resources http://expertvoices.nsdl.org/connectingnews/

– Encourage teachers to use current events as teaching opportunities– Contextual links to NSDL resources– Includes related NSES standards – RSS from the blog is used on the MSP website

• Topic-based discussionhttp://expertvoices.nsdl.org/polar/

– Provide ideas for connecting science and literacy topics– Generates the News Section for Beyond Penguins and Polar Bears eZine

• NSDL community members as contributors http://expertvoices.nsdl.org/pats-picks/

– Link and review web resources for STEM educators

• Teacher/student discussion forum http://expertvoices.nsdl.org/cornell-info204/

– supplemental blog for a Cornell University course

http://expertvoices.nsdl.org/connectingnews/



http://expertvoices.nsdl.org/polar/



http://expertvoices.nsdl.org/pats-picks/

http://expertvoices.nsdl.org/cornell-info204/

Annual Meeting 2008

NDR API

NCS

NSDL Resource LinksProvides more information about a resource in the library

Annual Meeting 2008

NDR API

NCS

NSDL NDR ConnectionAdd a blog post and its associated resources to the NSDL

Annual Meeting 2008

NDR API

NCS

NSDL NDR ConnectionFind blog post and its referenced resources in NSDL search

Blog Post

Referenced Resource

Annual Meeting 2008

NDR API

NCS

Referenced Resources

<dct:references> in metadatahttp://ndr.nsdl.org/api/get/2200/20070314153845593T/format_nsdl_dc

Relationships in objectshttp://ndr.nsdl.org/api/get/2200/20070314153845593T

... <dct:references xsi:type="dct:URI">

http://www.polarbearsinternational.org/in-the-news/threatened-species/</dct:references> <dct:references xsi:type="dct:URI">

http://nsf.gov/news/special_reports/arctic/index.jsp</dct:references> ...

... <relationships> <nsdl:annotates>2200/20070314154210749T</nsdl:annotates> <nsdl:metadataProvidedBy>2200/20070314153843382T</nsdl:metadataProvidedBy> <nsdl:metadataFor>2200/20070314153844301T</nsdl:metadataFor></relationships>...

Annual Meeting 2008

NDR API

NCS

NCore WPMU Plugins

Plugins available to integrate other WPMU installations with NCore data repository

• NSDL Search Popup (nsdl-searchPopup): Enhances the URLs in your post with NSDL resource links.

• NSDL Site Admin (nsdl-admin): Provides admin tools to add blogs as resources to the NSDL Data Repository.

• NSDL NDR Popup (nsdl-ndrPopup): add a resource and its metadata to the NSDL Repository.

http://wiki.nsdl.org/index.php/Community:ExpertVoices



Annual Meeting 2008

NDR API

NCS






Annual Meeting 2008

NDR API

NCS

The NSDL Collection System (NCS)

• Web application for creating, editing, managing and developing collections.

• Communicates directly with NDSL data repository (NDR)

• Features include

– Metadata Editor

– Educational Standards– Collection Management– Web services

NCS Metadata Editor

Chunked records

Incrementalvalidation

Controlledvocabularies

RequiredFields

Bestpractice

s

Projectlogo

User Interface programmatically constructed from XSD schema

Annual Meeting 2008

NDR API

NCS

Collection Management

• Web-based access supports distributed cataloging and collection management

• Per-collection workflow support enables coordinated management processes

• Roles and permissions allows managed access to information and system functionality

Annual Meeting 2008

NDR API

NCS

Web Services

MetadataEditor

MetadataEditor

Search / Browse, Manage

Search / Browse, Manage

LocalRepository

LocalRepository

IndexIndex

PutRecord

PutRecord

SearchSearch

OAIOAI

StandardsSuggestorStandardsSuggestor

NDRNDR

NCS

Annual Meeting 2008

NDR API

NCS

Use Case: NSDL

• Collection Management– NCS manage collections metadata for each NSDL

collection– 20 out of 150 plus NSDL collections are directly

managed in the NDR via the NCS

• NSDL.org – Recommend a Resource inserts recommendations

into the NCS using the put record web service– Collections and Pathways views pull data from NCS

web service

Annual Meeting 2008

NDR API

NCS

Use Case: Curriculum Customization Service (CCS)

Learning application in support of Science Curriculum for the Denver Public School District

Curriculum objects are cataloged in the CCS NCS

Complex and dynamic displays pull data from CCS NCS via web service

Annual Meeting 2008

NDR API

NCS

NCS Technical• Java-based Application runs on Linux, PC, Mac

• NSDL Wiki Documentation• https://wiki.nsdl.org/index.php/Community:NCS

• Local Install• Download link is found on wiki page

• To create a collection in the NCS hosted by NSDL at Cornell University, contact NSDL at http://nsdl.org/about/contactus/

https://wiki.nsdl.org/index.php/Community:NCS

http://nsdl.org/about/contactus/

Annual Meeting 2008

NDR API

NCS






Annual Meeting 2008

NDR API

NCS

The Strand Map Service• A service for creating interactive applications that enable

educators and learners to– Explore the relationships and interconnections between

important STEM concepts and learning goals– Locate resources and other curriculum components

supporting specific concepts and learning goals– Enhance science content knowledge, pedagogical content

knowledge (educators), and preparation for learning (learners)

– Design coherent instructional materials supporting recognized learning goals and informed by research on student learning (educators)

• Service APIs let developers embed the maps in Web sites and display educational resources and other information in the maps.

Annual Meeting 2008

NDR API

NCS

Content from Project 2061/AAAS

• Benchmarks: Describe what learners should know, or be able to do, at key stages in their education across the STEM disciplines

• Strand maps: Learning progression diagrams illustrating how student understanding changes over time

• Curriculum components: Materials and information to improve teachers’ science content knowledge, pedagogical content knowledge, and instructional practices

Annual Meeting 2008

NDR API

NCS

SMS Service

• Integrate the AAAS Atlas on-line

Annual Meeting 2008

NDR API

NCS

Technical Components of the Service

• Computational model of interconnected concepts– Depict how ideas change over time -> Learning Progressions and

Knowledge Maps

– Integrate curriculum components -> prior research, learning resources, standards, misconceptions, assessments

• Visualization algorithms incorporating AAAS graphical conventions

• JavaScript and REST service APIs for use by developers

Annual Meeting 2008

NDR API

NCS

SMS Architecture

Annual Meeting 2008

NDR API

NCS

JavaScript and REST APIs

Developers can:• Place the interactive Strand Maps in any Web

page• Display educational resources and other

information in the maps– Associate resources with learning goals

• Access and display map visualizations– Interactive JavaScript, SVG, PDF, JPG, PNG,

TIFF• Access and traverse the data graph: chapters,

maps, grades, strands, benchmarks– XML, JSON

APIs used to place interactive maps in web pages, embed content and

control map features

Benchmark details

Related benchmarks

Map Navigator

NSES standardsEmbedded

content

Embedded content

API is used to insert content in the bubble and tabs

Student misconceptions

Search and browse for maps

Use Case: Curriculum Customization Service• Enhances key concepts in the Denver Public Schools

curriculum with the benchmarks

Use Case: Educational Benchmarks Collection• NSDL collection containing each benchmark with related

educational resources

Use Case: NSDL Science literacy Maps• Connects NSDL resources with the benchmarks

Annual Meeting 2008

NDR API

NCS

Using the Strand Map Service

• Available for developers: – Web service APIs– Documentation– Working code examples

Go to:http://strandmaps.nsdl.org/cms1-2/docs/

Annual Meeting 2008

NDR API

NCS






Annual Meeting 2008

NDR API

NCS

The Repository

• Central storage, organization, and access platform for data

• Serves as the “patch bay” through which NCore components share data and interoperate.

• Has an object model specializing in organizing data from multiple sources around shared resources.

• Access is through a model-centric web service API

Focus on this section is on how data flows through the repository: From an OAI harvest, blog post, or wiki page to Search discovery.

Annual Meeting 2008

NDR API

NCS

Into the RepositoryOAI Harvest

Service

Pathway or Project

OAI

Repository API

Repository

NCS

Annual Meeting 2008

NDR API

NCS

Out of the repository

OAI

Search

API

Strand maps

Nsdl.org, Wiki, EV

NCS

Annual Meeting 2008

NDR API

NCS

A Closer LookMetadata

Propertiesoai:id = abc123

Datastreams

Relationships

format_nsdl_dc

nsdl:metadataFor

nsdl:providedBy

Resource

MDProviderBlob of nsdl_dc xml, same as OAI metadata payload in the case of Harvest service

Required fields are populated by harvest service via API

Annual Meeting 2008

NDR API

NCS


Propertiesoai:id = abc123ncs:status = swell

Datastreams

Relationships

format_nsdl_dcformat_ncs_itemformat_dcs_data

nsdl:metadataFor

nsdl:providedBy

Resource

MDProviderNCS, for example, stores its “full” metadata and audit trails in additional datastreams

Applications may introduce new elements for their own use in the NDR object

Annual Meeting 2008

NDR API

NCS



Datastreams

Relationships


nsdl:metadataFor

nsdl:providedBy

Resource

MDProvider

Content

The API allows read/write access to all parts of every object (subject to permissions and access rights).

...of course, elements of the object that aren’t part of the core model are just along for the ride, useful to applications that understand them, benign to others.

API

Annual Meeting 2008

NDR API

NCS

SearchingMetadata


Datastreams

Relationships


nsdl:metadataFor

nsdl:providedBy

Resource

MDProvider

Content

The API allows searching over properties and datastreams of all objects

API search results are repository objects..

API

Annual Meeting 2008

NDR API

NCS

SearchingMetadata


Datastreams

Relationships


nsdl:metadataFor

nsdl:providedBy

Resource

MDProvider

Content

The search service allows search over nsdl_dc metadata, and resource content (and a few other facts, such as identity of the resource’s collection).

Search results are, for the most part, all nsdl_dc metadata describing a single resource.

Search

Annual Meeting 2008

NDR API

NCS

Searching and Indexing

Metadata


Datastreams

Relationships


nsdl:metadataFor

nsdl:providedBy

Search

API

Other datastream content may be searched by externally-deployed applications, such as DDS, or custom indexes.

Annual Meeting 2008

NDR API

NCS






Annual Meeting 2008

NDR API

NCS

The NSDL Search Service

• A Lucene search index, containing over 2 million records, exposed as a web service.– One record for each resource in the repository

(active resources with metadata)• A Nutch database of crawled web pages, exposed as

a web service.– parsed text from over 800,000 pages

(restricted by robots.txt and non-text formats)• A GUI front-end webapp to assist with queries and to

display results.

Annual Meeting 2008

NDR API

NCS

Fully exposed Lucene query syntax

• http://ndrsearch.nsdl.org/search?q=frogs

• Default seach hits all text fields:q=frogs

• Search specific fieldsq=dc\:title:frogs

• Boolean combinationsq=frogs AND spaghetti NOT ellipsoids

• Proximity searchesq="deformed frogs"~5

Annual Meeting 2008

NDR API

NCS

The stack

Fedora

NDR API OAIserver

SearchService

Nutchdatabase

Searchfront end

Annual Meeting 2008

NDR API

NCS

Reaching behind the GUI

Annual Meeting 2008

NDR API

NCS

Search Technical

• Lucene index:– 2,000,000 records in 20 gigabytes– updated nightly from the repository and the Nutch

database• Nutch database with parsed text from over 800,000

pages– crawl for several hours daily, complete every 30

days, approx.• NSDL Wiki documentation:

https://wiki.nsdl.org/index.php/Community:Search

Annual Meeting 2008

NDR API

NCS






Annual Meeting 2008

NDR API

NCS

The challenge moving forward

1. How to best use what we have2. Where to go from here

Annual Meeting 2008

NDR API

NCS

How to use what we have?

• A change in mindset:– from "NCore is software developed in order to

implement the NSDL"– to "NSDL is a digital library implemented using

NCore software

Annual Meeting 2008

NDR API

NCS

How to use what we have?

• Generifying, modularizing, adding configurability– How can we make these tools more widely useful?

• Focus on all levels of documentation– Adopt a user-centric approach

• Eat our own dog food– Install from our documents– Use released NCore software on our production

servers.

Annual Meeting 2008

NDR API

NCS

Where to go from here?

• Building the roadmap• Are you our next use case?

Connecting Applications with the NSDL Technical Network Services.

Documents

Transcript of Connecting Applications with the NSDL Technical Network Services.