ARL Statistics and Measurement Program Effective, Sustainable and Practical Assessment Martha...
-
Upload
alexia-carr -
Category
Documents
-
view
236 -
download
0
Transcript of ARL Statistics and Measurement Program Effective, Sustainable and Practical Assessment Martha...
ARL Statistics and Measurement ProgramEffective, Sustainable and Practical Assessment
Martha Kyrillidou Director, ARL Statistics and Service Quality Programs
Association of Research Libraries
Steve Hiller Director, Assessment & Planning, University of Washington Libraries
ARL Visiting Program Officer
Jim SelfDirector, Management Information Services, University of Virginia Library
ARL Visiting Program Officer
University of Cape Town Libraries16 August 2007
ARL www.arl.org
ARL Mission
• Non-profit organization of the libraries of research institutions in North America
• Forum for exchange of ideas
• Agent for collective action
Mission: Shaping the future of research libraries in the changing environment of public policy and scholarly communication.
Members: 123 major research libraries in North America.
Ratios: 4 percent of the higher education institutions providing 40 percent of the information resources.
Users: Three million students and faculty served.
Expenditures: $3.4 billion annually, $1.1 billion for acquisitions of which 37 percent is invested in access to electronic resources.
www.arl.org
What makes a research Library?
• Breadth and quality of collections and services• Sustained institutional commitment to the library• Distinctive resources in a variety of media• Services to the scholarly community• Preservation of research resources• Contributions of staff to the profession• Effective and innovative use of technology• Engagement of the library in academic planning --from ARL ‘Principles of Membership’
The Beauty and the Beast
• What makes a quality research library?
“Quality much like beauty is in the eye of the beholder”
The Beauty and the Beast
• What makes a quality research library?
“Quality much like beauty is in the eye of the beholder”
The Beauty and the Beast
•What makes a quality research library?
“Quality much like beauty is in the eye of the beholder”
A gateway to assessment tools: ARL StatsQUAL™:
ARL Statistics -- E-Metrics
LibQUAL+®
DigiQUAL™
MINES for Libraries™
•Library Assessment Conferences
•Service Quality Evaluation Academy
•Library Assessment blog
•Making Library Assessment Work
•ESP Assessment
– Effective, Sustainable, Practical
Assessment at ARL
www.arl.org
StatsHome
LibQUAL+DigiQUAL(/digiqual)
MINES(/mines)
SAILS(/sails)
E-Metrics(/emetrics)
ARL Statistics(/arlstats)
Interactive Statistics
(/interactive)?
Login
UserProfile
InstitutionProfile
www.libqual.org
SurveyManagement
StatsQUAL™
www.statsqual.org
Updating the Traditional ARL Statistics
• E-Metrics = ARL Supplementary Statistics– On going efforts to update and refine core data.
– Exploring feasibility of collecting e-metrics.
• ARL Task Force on New Ways of Measuring Collections :– Growing concern with utility of membership index.
– Study ARL statistics to determine relevance.
– Develop Profile of Emerging Research Libraries.
www.statsqual.org
Tradition
• ARL membership criteria index:– Volumes held– Volumes Added Gross– Current Serials– Total Expenditures– Professional plus support staff
Task Force Recommendations
• Reserve use of the current membership criteria index to those occasions when it is needed for consideration of membership issues
• Implement an expenditure-focused index
• Use the new expenditure-focused index for any public reports, such as the Chronicle of Higher Education
• Begin to develop a services-based index that combines the following three factors: collections, services, and collaborative relationships
• Revise definitions for collections-related data categories, such as serials, and experiment with a variety of new measures, including usage data, strength of collections, and service quality measures to develop a richer set of variables for potential inclusion in the three factor alternative index
• Collect qualitative data to develop a profile of ARL member libraries
The power of narrative … (Peter Brophy, Yvonna Lincoln, Colleen Cook …..)
The focus on economics
• ARL Expenditures focused index:– Total library expenditures– Expenditures for Professional staff– Expenditures for Library materials– Professional plus support staff
Linear Regression
0 25 50 75 100
Rank of index03
0
25
50
75
100
Ran
k o
f ex
pin
d03
Rank of expind03 = 3.24 + 0.94 * rindx03R-Square = 0.89
Linear Regression
0 25 50 75 100
Rank of index05
0
25
50
75
100
Ran
k o
f ex
pin
d05
Rank of expind05 = 2.50 + 0.94 * rindx05R-Square = 0.89
Thinking Strategically About Library Futures
• What is the central work of the library and how can we do more, differently, and at less cost?
• What important set of services does the library provide that others can’t? What new roles are needed?
• What advantages does the research library possess?
• What will be the most needed by our community of users in the next decade? How is user behavior changing?
• What should our libraries aspire to be ten years from now? What are the implications of technology driven change?
• What are the essential factors responsible for the success of the library?
www.arl.org
Defining Success in a Digital Environment
• Crafting new measures of success.
• Moving from measuring inputs to outputs.
• Understanding impact of library roles and services.
• Agreeing on qualitative measures of success: user perceptions, user success, creating value, advancing HE goals.
• Reallocating and managing capabilities to focus on new definitions of success.
www.arl.org
• "You're going to be able to go back and forth from Google Book Search into Harvard, or from Harvard's catalog into Google,“ reminding users that it was important to keep the HOLLIS catalog the "starting point for their research," since only HOLLIS can give a full picture of the 15.5 million books in Harvard's collection, the nation's largest …Harvard would look to scan all of its-out-of copyright books, about one million, over time but that
ultimately "the ideal" is one big digital ultimately "the ideal" is one big digital library of books held in libraries around the library of books held in libraries around the world. world. ”It shouldn't matter whether it came from Harvard or Michigan"
- Sidney Verba, Harvard library director, - Library Journal Academic Newswire – April 24, 2007
old.libqual.org
A LibQUAL+® Update
The LibQUAL+® premise, dimensions, and methodology
LibQUAL+® resultsLibQUAL+® in action
old.libqual.org
Dimensions ofLibrary Service Quality
Information Control
LibraryServiceQuality
Model 3
Self-Reliance
Equipment
Timeliness
Ease of Navigation
Convenience
Scope of Content
Affect of Service
Library as Place
Reliability
Assurance
Responsiveness
Empathy
Refuge
Symbol
Utilitarian Space
old.libqual.org
…and a Box”
Why the Box is so Important:• About 40% of participants provide open-
ended comments, and these are linked to demographics and quantitative data
• Users elaborate the details of their concerns• Users feel the need to be constructive in their
criticisms, and offer specific suggestions for action
Background: ServQUAL LibQUAL+® DigiQUAL™
LibQUAL+® Dimensions of Service Quality:• Affect of Service• Information Control• Library as Place
Developing DigiQUAL™ Survey Items
DigiQUAL™12 themes of service quality:• Accessibility• Navigability• Interoperability• Collection building• Resource Use• Evaluating collections• DL as community for users• DL as community for developers• DL as community for reviewers• Copyright• Role of Federations• DL Sustainability
www.digiqual.org
Assessing the Value of Networked Electronic Services:
Measuring the Impact of Networked Electronic Services (MINES) - MINES for Libraries™
www.arl.org/stats/newmeas/mines.html
The MINES survey
• A research methodology consisting of a web-based survey form and a sampling plan.
• Measures who is using electronic resources, where users are located at the time of use, and their purpose of use.
• Adopted by the Association of Research Libraries (ARL) as a part of the “New Measures” toolkit May, 2003.
• Different from other electronic resource usage measures that quantify total usage (e.g., COUNTER, EQUINOX, E-Metrics, ICOLC guidelines, ISO and NISO standards) or measure how well a library makes electronic resources available (LibQUAL+®, DigiQUAL™).
What is MINES?
• How extensively do sponsored researchers use the new digital information environment?
• Are researchers more likely to use networked electronic resources from inside or outside the library?
• Are there differences in usage of electronic information based on the user’s location (e.g., in the library; on-campus, but not in the library; or off-campus)?
• What is a statistically valid methodology for capturing electronic services usage both in the library and remotely through web surveys?
• Are particular network configurations more conducive to studies of digital libraries patron use?
Questions Addressed
Where are the most critical assessment needs and opportunities?
• Complementing LibQUAL+® with additional measures.
• Developing impact studies on user success, economic value, and community return on investment.
• Moving target: what is a digital library?
• E-Resources: understanding usage.
• Gaining acceptance and use of standard measures for e-resources.
• Building a climate of assessment throughout library.
What are the lessons learned?
• Understanding changes in users approach to information resources.
• Service quality improvement is a key factor.
• Understanding the impact of e-resources on library services - TRL.
• Learning how to compete and/or collaborate with Google.
• Upfront investment in design and development.
• Making the assessment service affordable, practical, & effective.
• Assessment needs to be satisfying and fun.
In Closing
• As higher education is challenged on accountability and effectiveness issues so will libraries.
• A growing appreciation of need for fresh assessment measures, techniques, and processes - old arguments don’t work.
• Basic questions of role, vision, and impact must be answered by library community.
What’s in a “Library”
• A word is not crystal, transparent and unchanged; it is the skin of a living thought, and may vary greatly in color and content according to the circumstances and time in which it is used. – --Justice Oliver Wendell Holmes
Selected References
• Kyrillidou, Martha and Sarah Giersch. “Developing the DigiQUAL Protocol for Digital Library Evaluation.” Paper Presented at JCDL - Joint Conference on Digital Libraries, Denver, CO, June 6-11, 2005. [Available at http://www.libqual.org/documents/admin/digiqual-jcdl05-v5.pdf]
• Kyrillidou, Martha, Toni Olshen, Brinley Franklin, and Terry Plum. “MINES for Libraries(tm): Measuring the Impact of Networked Electronic Services and the Ontario Council of University Libraries' Scholar Portal, Final Report.” Presented at the 6th Northumbria International Conference on Performance Measurement in Libraries and Information Services, Durham, England, Aug. 23, 2005. [Available at http://www.libqual.org/documents/admin/FINAL%20REPORT_Jan26mk.pdf]
• Franklin, Brinley and Terry Plum. "Library usage patterns in the electronic information environment" Information Research, 9(4) paper 187 (2004). [Available at http://InformationR.net/ir/9-4/paper187.html]
• Franklin, Brinley, and Terry Plum. "Documenting Usage Patterns of Networked Electronic Services." ARL: A Bimonthly Report on Research Library Issues and Actions from ARL, CNI, and SPARC, 230/231 (2003): 20-21. [Available at http://www.arl.org/newsltr/230/usage.html].
• Cook, Colleen, Fred Heath, Martha Kyrillidou, Yvonna Lincoln, Bruce Thompson, and Duane Webster. “Developing a National Science Digital Library (NSDL) LibQUAL+™ Protocol: An E-service for Assessing the Library of the 21st Century” Submitted for the Developing an Evaluation Strategy for the Educational Impact of the National Science Digital Library Workshop, Washington DC, October 2-3, 2003. [Available at http://www.libqual.org/documents/admin/NSDL_workshop_web1.pdf]
• Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “Evaluating the NSF National Science Digital Library Collections.” Paper presented at the Multiple Educational Resources for Learning and Online Technologies (MERLOT) Conference, Costa Mesa, California, August 3-6, 2004. [Available at http://www.libqual.org/documents/admin/MERLOT%20Paper2_final.pdf]
• Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “User Perspectives Into Designs for Both Physical and Digital Libraries: New Insights on Commonalities/Similarities and Differences from the NDSLDigital Libraries and LibQUAL+™ Data Bases.” 7th ISKO-Spain Conference, The human dimension of knowledge organization, Barcelona, Spain July, 6-8, 2005. [Available at http://www.libqual.org/documents/admin/ISKO.PDF]
Selected References