National and University Library of Slovenia

University of Ljubljana, Faculty of Civil and Geodetic Engineering

User-centred evaluation of digital User-centred evaluation of digital repositories: reUSE experiencesrepositories: reUSE experiences

LIDA Conference, Dubrovnik 30. May – 3. June 2005

by Mateja Šmid, Darko Majcenovič, Špela Zupanc, Alenka Kavčič Čolić

Introduction Objectives Evaluation User survey Conclusion

Project reUSE: reuse digital master files of printed materials!

Financial source: EU eContent (nr. 11173) Partners: 9 partners University and national

libraries from 4 different countries: Austria, Estonia, Germany, and Slovenia.

Duration of the project: June 2004 - June 2006

INTRODUCTIONINTRODUCTION

Introduction

Introduction Objectives Evaluation User survey Conclusion

UBER – Humboldt University Berlin repository

ONB – Austrian national library ALO – Austrian literature online NLE – national library of Estonia

Each implementers will establish its own digital trusted repository with its own restrictions, services and policies.

REPOSITORY IMPLEMENTERSREPOSITORY IMPLEMENTERS

Introduction

Introduction Objectives Evaluation User survey Conclusion

Repository = organization that intends to maintain information for access and use.

Trusted repository = has the mission to provide reliable long-term access of managed digital resources to its designated community, now and in the future.

TRUSTED DIGITAL REPOSITORIESTRUSTED DIGITAL REPOSITORIES

Objectives

Introduction Objectives Evaluation User survey Conclusion

To:

build useful systems systems optimatization improve services to users measure progress enhance the role of digital repositories

OVERALL TO MAKE USER-CENTRED REPOSITORIES

PURPOSE OF reUSE EVALUATIONPURPOSE OF reUSE EVALUATION

Evaluation

Introduction Objectives Evaluation User survey Conclusion

EVALUATION FRAMEWORKEVALUATION FRAMEWORK Evaluation approach

Methodology: multi-atribute system and SWOT analysis focused on: Organizational aspects Technical aspects User perspective

We will use interviews and questionnaires to gain a quantitative and qualitative comprehensive study.

Evaluation

Introduction Objectives Evaluation User survey Conclusion

ORGANIZATIONAL ASPECTSORGANIZATIONAL ASPECTS External and internal context in which the

repositories were developed and their main functions in it. It emphasizes the concept of trustfulness through: The organizational strategy which defines the goals

and mission of the repository, The stakeholders interests in its maintaining and

financing, Favourable organizational culture to its development, The long-term preservation strategy and policy.

Evaluation

Introduction Objectives Evaluation User survey Conclusion

TECHNICAL ASPECTS OF THE REPOSITORIESTECHNICAL ASPECTS OF THE REPOSITORIES Technical specifications which can show us

to what extent the repositories satisfy users requirements and needs.

Several subgroups will be evaluated: General technical information OAIS compliance Technical and procedural suitability, security

and procedural accountabillity User friendliness and services

Evaluation

Introduction Objectives Evaluation User survey Conclusion

Digital libraries are useful when

are in harmony (Bishop, Van House, Butterfield: Digital library use, 2003)

The usersThe

documents

The system

The DLsThe DLsare seen as:are seen as:

sociotechnical systemsociotechnical system

networks of technologynetworks of technology

information artifactsinformation artifacts

people and people and practices practices interacting with the larger interacting with the larger world of work and societyworld of work and society

Evaluation

Introduction Objectives Evaluation User survey Conclusion

Happy user is the main goal.

The fact that the user predetermine the collection should be highly considered. It is imporant WHO WHO WILL ACCESSWILL ACCESS and for WHAT PURPOSEWHAT PURPOSE.

User survey

Introduction Objectives Evaluation User survey Conclusion

Purpose: to evaluate repositories from user perspective

Form: questionnaire is in national language one sheet in paper form

Based on: ISO 9241 Usability standard ISO 13407 Human-centred design processes for interactive

systems some specific questiones for reUSE repositories …

ADDITIONALY ADDITIONALY other methods such as simple other methods such as simple observation and logs and observation and logs and statistics will be carried out statistics will be carried out (depands on certain institution)(depands on certain institution)

USER PERSPECTIVEUSER PERSPECTIVE

User survey

Introduction Objectives Evaluation User survey ConclusionUser survey

End-user questionnaire consists of 2 End-user questionnaire consists of 2 parts:parts:1. general (demographic) information1. general (demographic) information2. questiones2. questiones

Introduction Objectives Evaluation User survey Conclusion

DIFFERENT SET OF QUESTIONES:DIFFERENT SET OF QUESTIONES:

content user’s satisfaction with material in the digital repository

recognition understandability of the interface

performance and efficiency efficiency of the repository

personal and subjective perception personal view and satisfaction with the repository

error tolerance possible effects on further search

User survey

Introduction Objectives Evaluation User survey Conclusion

2 periods of end user’s evaluation. June 2005 after repositories are set up September 2005 after strong dissemination will be done

To get stable and representative sampling around 100 questionnaires per repository will be needed.

Every 10th visitor will be asked:Whether he/she has ever used the repository?

If yes, he/she is pleased to fill in the questionnaire for end-user. If no, the person is pleased to fill in short questionnaire for non-

user.

Perception/opinion of non-users is also important.

ConclusionUser survey

REALIZATION OF THE SURVEYREALIZATION OF THE SURVEY

Introduction Objectives Evaluation User survey Conclusion

Different set of questiones will give us several answers.

Important feedback from end-users and non-users as well. implementers might redesign certain part(s) of repository if it is

necessary clarification of what kind of content is expected by users

Based on those answers we will be able to compose a broader picture about user’s perspective on certain repository.

Usage habits, user’s wish, a need for training,… answers on such questiones will be given.

… Cultural differencess might also surface (different backgrounds of national/university libraries, different participating countries…)

User survey

EXPECTED RESULTSEXPECTED RESULTS

Introduction Objectives Evaluation User survey Conclusion

CONCLUSIONCONCLUSION

At the end of the evaluation a comprehensive SWOT analysis for each implemented repository will be made.

The evaluation should have the retroactive effect on the awareness and further dissemination valuable input for the implementers to

improve the repositories if necessary to improve services to users

Conclusion

Introduction Objectives Evaluation User survey Conclusion

THANK YOU FOR ATTENTION!THANK YOU FOR ATTENTION!

Mateja Šmid (msmid@ikpir.fgg.uni-lj.si)

Darko Majcenović (darko.majcenovic@nuk.uni-lj.si)