Workshop onQuality/Selectivity of the DLESE

Collections

• Framing the Question• History of the Discussion

Kim Kastens, June 30, 2003

Framing the Question

• DLESE has Broad Collection and a Reviewed Collection. We are (mostly) talking about the Broad Collection.

Framing the Question

• Resources enter the DLESE Broad Collection via two routes:

– Individually, via the DLESE Cataloging Tool (the “Community Collection”)

– As part of an aggregated or themed collection, a collection accessioned into DLESE in its entirety.

• We are concerned with quality and relevance of resources entering via both routes.

Framing the Question

• Anyone can submit a resource to DLESE via the cataloging tool, which is an open set of web forms.

• This has given rise to concerns that “junk” could get into the DLESE Collections.

Framing the Question

• This workshop needs to make recommendations on two issues:

– What should be the criteria by which resources are approved for inclusion in the DLESE Broad Collection?

– What should be the procedures by which these criteria are implemented?

Framing the Question: Criteria

• Resources submitted for the DLESE Broad collection currently must meet two criteria:

– The resource is relevant to Earth System Education

– The resource works (i.e. it has no conspicuous bugs).

Framing the Question: Criteria

• Other possible criteria that have been suggested for the DLESE Broad Collection:

• No cost or low cost for educational users

• Resource is in English

• No commercial message

• No intrusive advertising

• No blatant religious message

• No blatant political message

• No blatant errors of fact

• Educational effectiveness

• Well documented

Framing the Question: Procedures

• With respect to procedures, we have two issues:

– By what process shall we identify problematic resources?

– What shall we do when we find a problematic resource?

Framing the Question: Procedures

• By what process shall we identify problematic resources?

– Ask the resource contributor (current system)?

– Screening by the community? – Screening by paid staff?

Framing the Question: Procedures

• What shall we do when we find a problematic resource?

– Exclude it from the collection?

– Include it in the Broad Collection but annotate it?

This is what an annotation might look like in theDiscovery System:

Framing the Question: Procedures

Screening bypaid staff

Screening bycommunity

Problematicresourcesexcluded fromlibrary

1 2

Problematicresourcesannotated inDiscoverySystem

3 4

History of the Discussion

• Coolfont: August 1999

– Collections Policy drafted

– There shall be a Reviewed Collection and an “Unreviewed” Collection

– Collection Committee established

History of the Discussion

• Coolfont: August 1999 (cont’d)

– Rationale for Reviewed Collection: • Users’ Perspective: “…. recognized, efficient source for

quality teaching and learning materials.”

• Creators’ Perspective: “…. a recognized stamp of professional approval at the level of publication in a peer-reviewed journal.”

– Rationale for the “Unreviewed” Collection:• “Users are seeking materials on a huge range of topics.

The DL provides added value by being inclusive while providing powerful search and classification capability.”

History of the Discussion

• Coolfont: August 1999 (cont’d)

– Criteria for Reviewed Collection:

• Accuracy, as evaluated by scientists• Importance/significance• Pedagogical effectiveness. • Well documented. • Ease of use for students and faculty• Inspirational or motivational for students• Robustness/sustainability

History of the Discussion

• Coolfont: August 1999 (cont’d)

– No Criteria established for “Unreviewed” Collection

– After debate, it was decided that there would be a human-mediated step between submission of resource and ingestion into library.

History of the Discussion

• Spring 2000: Academic Career Recognition Task Force Web Survey

– Seven selection criteria for the Reviewed Collection met approval of prospective DLESE users, resource creators, and department Chairs.

History of the Discussion

• Mid-late 2000: Collecting began

– DPC: testbed collection for exercising metatdata framework

– Montana State: Dave Mogk & students

– Foothill College: Chris DiLeonardo & students

History of the Discussion

• October 2000: Collections Meeting at Boulder:

– DLESE Community Cataloger tool introduced to non-DPC collecting groups (AGI, Montana State, others?)

History of the Discussion

• November 2000: Steering Committee Meeting at Lamont:

– Contentious discussion about “filters” at the gateway to the Broad Collection

– Agreement on only two of the discussed “filters”: (1) relevant to Earth System Education(2) “It works”, e.g. no conspicuous bugs

- Contentious discussion of how to apply “filters”; clarity seemed to emerge when John Snow described a “holding tank” system used in his history group.

History of the Discussion

• Nov-Dec 2000: Steering Committee Meeting at Lamont (cont’d):

- Meeting Minutes: • “The general concept of a 30-day public comment period on new

resources was agreed to. This will allow a time for the community to review resources….”

• “In the short term, partners collecting resources …. will review them to make sure they are appropriate

• “….the Collections Committee, collection proposal team and the DPC will work together to investigate mechanisms for encouraging review….”

History of the Discussion

• February 2001 Collections Meeting:

– Joint meeting of Collections Committee, “Collections Partners”, and Community Review System Editorial Review Board

– DLESE Community Cataloging Tool open to the world

– Collections Committee drafted Deaccession Policy

History of the Discussion

• February 2001 Collections Meeting (cont’d):

– Collections Committee discussed “filters” at gateway to Broad Collection. Imperfect consensus:

• Relevance Filter – Is the resource relevant to Earth System Science education?

• Integrity Filter – Are there no blatant errors of fact in the resource? – Are there no blatant political, religious, or commercial messages in

the resource? – Does it function reasonably; i.e., seem to be basically bug-free?

History of the Discussion

• April 2001 Steering Committee meeting at Biosphere 2:

– Collections Committee/DPC Collections group presented fleshed out version of the “holding tank” or “provisional status” plan.

– Many questions and issues. Who are reviewers? How mobilized and overseen? No $ to oversee the “army of filterers.”

– No resolution.

History of the Discussion• July-Aug 2001 Steering Committee meeting at Flagstaff:

– 850 resources in library. Metadata QA streamlined.

– “Mike Mayhew indicated a concern …about the broad collection. ….Where is the quality control in developing the collection? Do we dilute the value of library with variable quality?”

– Holding tank idea revisited, in simpler form without “designated reviewers”

– Action item: “Boyd …. will develop a draft proposal/set of guidines to implement a holding tank in which resources are discoverable in the system and identified as accessioned within a 30-day period with some mechanism to accept comments. The proposal for implementation will not include a designated reviewer”

History of the Discussion

• February 2002 Steering Committee meeting at Boulder:

– Draft Collections Accession Policy presented

– Revised throughout spring

– DLESE oversight would be review of review process, rather than review of individual resources

History of the Discussion

• July 2002 Steering Committee & Annual meeting at Cornell:

– Deaccession Policy approved– Interim Collection Accession Policy approved– First annotation service demo’d within DLESE– Faulker reported that NSDL content philosophy was:

“Educational value …to be manifest in capabilities for annotation and selective filtering, rather than an accession threshhold”

– Possibility raised that annotation option might be solution to ongoing dilemma about quality of DLESE Broad Collection.

History of the Discussion

• Fall 2002:

– Sumner et al focus group study of Educators’ perceptions of Quality.

– Best Practices for Resources summited to the DLESE Reviewed Collection begins to take shape.

History of the Discussion

• Spring 2003:

– Ad hoc Collections group met in Boulder, worked on how to implement Interim Collections Accession Policy and on Pathways to Reviewed Collection document

– June 13: 12 collections met documentation requirements to be accessioned as collections.