Post on 28-Dec-2015
National and University Library of Slovenia
University of Ljubljana, Faculty of Civil and Geodetic Engineering
User-centred evaluation of digital User-centred evaluation of digital repositories: reUSE experiencesrepositories: reUSE experiences
LIDA Conference, Dubrovnik 30. May – 3. June 2005
by Mateja Šmid, Darko Majcenovič, Špela Zupanc, Alenka Kavčič Čolić
Introduction Objectives Evaluation User survey Conclusion
Project reUSE: reuse digital master files of printed materials!
Financial source: EU eContent (nr. 11173) Partners: 9 partners University and national
libraries from 4 different countries: Austria, Estonia, Germany, and Slovenia.
Duration of the project: June 2004 - June 2006
INTRODUCTIONINTRODUCTION
Introduction
Introduction Objectives Evaluation User survey Conclusion
UBER – Humboldt University Berlin repository
ONB – Austrian national library ALO – Austrian literature online NLE – national library of Estonia
Each implementers will establish its own digital trusted repository with its own restrictions, services and policies.
REPOSITORY IMPLEMENTERSREPOSITORY IMPLEMENTERS
Introduction
Introduction Objectives Evaluation User survey Conclusion
Repository = organization that intends to maintain information for access and use.
Trusted repository = has the mission to provide reliable long-term access of managed digital resources to its designated community, now and in the future.
TRUSTED DIGITAL REPOSITORIESTRUSTED DIGITAL REPOSITORIES
Objectives
Introduction Objectives Evaluation User survey Conclusion
To:
build useful systems systems optimatization improve services to users measure progress enhance the role of digital repositories
OVERALL TO MAKE USER-CENTRED REPOSITORIES
PURPOSE OF reUSE EVALUATIONPURPOSE OF reUSE EVALUATION
Evaluation
Introduction Objectives Evaluation User survey Conclusion
EVALUATION FRAMEWORKEVALUATION FRAMEWORK Evaluation approach
Methodology: multi-atribute system and SWOT analysis focused on: Organizational aspects Technical aspects User perspective
We will use interviews and questionnaires to gain a quantitative and qualitative comprehensive study.
Evaluation
Introduction Objectives Evaluation User survey Conclusion
ORGANIZATIONAL ASPECTSORGANIZATIONAL ASPECTS External and internal context in which the
repositories were developed and their main functions in it. It emphasizes the concept of trustfulness through: The organizational strategy which defines the goals
and mission of the repository, The stakeholders interests in its maintaining and
financing, Favourable organizational culture to its development, The long-term preservation strategy and policy.
Evaluation
Introduction Objectives Evaluation User survey Conclusion
TECHNICAL ASPECTS OF THE REPOSITORIESTECHNICAL ASPECTS OF THE REPOSITORIES Technical specifications which can show us
to what extent the repositories satisfy users requirements and needs.
Several subgroups will be evaluated: General technical information OAIS compliance Technical and procedural suitability, security
and procedural accountabillity User friendliness and services
Evaluation
Introduction Objectives Evaluation User survey Conclusion
Digital libraries are useful when
are in harmony (Bishop, Van House, Butterfield: Digital library use, 2003)
The usersThe
documents
The system
The DLsThe DLsare seen as:are seen as:
sociotechnical systemsociotechnical system
networks of technologynetworks of technology
information artifactsinformation artifacts
people and people and practices practices interacting with the larger interacting with the larger world of work and societyworld of work and society
Evaluation
Introduction Objectives Evaluation User survey Conclusion
Happy user is the main goal.
The fact that the user predetermine the collection should be highly considered. It is imporant WHO WHO WILL ACCESSWILL ACCESS and for WHAT PURPOSEWHAT PURPOSE.
User survey
Introduction Objectives Evaluation User survey Conclusion
Purpose: to evaluate repositories from user perspective
Form: questionnaire is in national language one sheet in paper form
Based on: ISO 9241 Usability standard ISO 13407 Human-centred design processes for interactive
systems some specific questiones for reUSE repositories …
ADDITIONALY ADDITIONALY other methods such as simple other methods such as simple observation and logs and observation and logs and statistics will be carried out statistics will be carried out (depands on certain institution)(depands on certain institution)
USER PERSPECTIVEUSER PERSPECTIVE
User survey
Introduction Objectives Evaluation User survey ConclusionUser survey
End-user questionnaire consists of 2 End-user questionnaire consists of 2 parts:parts:1. general (demographic) information1. general (demographic) information2. questiones2. questiones
Introduction Objectives Evaluation User survey Conclusion
DIFFERENT SET OF QUESTIONES:DIFFERENT SET OF QUESTIONES:
content user’s satisfaction with material in the digital repository
recognition understandability of the interface
performance and efficiency efficiency of the repository
personal and subjective perception personal view and satisfaction with the repository
error tolerance possible effects on further search
User survey
Introduction Objectives Evaluation User survey Conclusion
2 periods of end user’s evaluation. June 2005 after repositories are set up September 2005 after strong dissemination will be done
To get stable and representative sampling around 100 questionnaires per repository will be needed.
Every 10th visitor will be asked:Whether he/she has ever used the repository?
If yes, he/she is pleased to fill in the questionnaire for end-user. If no, the person is pleased to fill in short questionnaire for non-
user.
Perception/opinion of non-users is also important.
ConclusionUser survey
REALIZATION OF THE SURVEYREALIZATION OF THE SURVEY
Introduction Objectives Evaluation User survey Conclusion
Different set of questiones will give us several answers.
Important feedback from end-users and non-users as well. implementers might redesign certain part(s) of repository if it is
necessary clarification of what kind of content is expected by users
Based on those answers we will be able to compose a broader picture about user’s perspective on certain repository.
Usage habits, user’s wish, a need for training,… answers on such questiones will be given.
… Cultural differencess might also surface (different backgrounds of national/university libraries, different participating countries…)
User survey
EXPECTED RESULTSEXPECTED RESULTS
Introduction Objectives Evaluation User survey Conclusion
CONCLUSIONCONCLUSION
At the end of the evaluation a comprehensive SWOT analysis for each implemented repository will be made.
The evaluation should have the retroactive effect on the awareness and further dissemination valuable input for the implementers to
improve the repositories if necessary to improve services to users
Conclusion
Introduction Objectives Evaluation User survey Conclusion
THANK YOU FOR ATTENTION!THANK YOU FOR ATTENTION!
Mateja Šmid (msmid@ikpir.fgg.uni-lj.si)
Darko Majcenović (darko.majcenovic@nuk.uni-lj.si)