Beyond LibQUAL+ ® : Effective, Sustainable, and Practical Assessment Other tools: ClimateQUAL ®,...
-
date post
18-Dec-2015 -
Category
Documents
-
view
215 -
download
0
Transcript of Beyond LibQUAL+ ® : Effective, Sustainable, and Practical Assessment Other tools: ClimateQUAL ®,...
• Beyond LibQUAL+®: Effective, Sustainable, and Practical Assessment
• Other tools: ClimateQUAL®, MINES for Libraries®, annual statistics
• Other approaches: Return on Investment, Value & Impact
Martha Kyrillidou, Glasgow, May 24 2010
ARL Mission
• Non-profit organization of the libraries of research institutions in North America
• Forum for exchange of ideas
• Agent for collective action
What makes a research Library?
• Breadth and quality of collections and services• Sustained institutional commitment to the library• Distinctive resources in a variety of media• Services to the scholarly community• Preservation of research resources• Contributions of staff to the profession• Effective and innovative use of technology• Engagement of the library in academic planning --from ARL ‘Principles of Membership’
What’s in a “Library”
• A word is not crystal, transparent and unchanged; it is the skin of a living thought, and may vary greatly in color and content according to the circumstances and time in which it is used. – --Justice Oliver Wendell Holmes
Thinking Strategically About Library Futures
• What is the central work of the library and how can we do more, differently, and at less cost?
• What important set of services does the library provide that others can’t? What new roles are needed?
• What advantages does the research library possess?
• What will be the most needed by our community of users in the next decade? How is user behavior changing?
• What should our libraries aspire to be ten years from now? What are the implications of technology driven change?
• What are the essential factors responsible for the success of the library?
www.arl.org
A gateway to assessment tools: ARL StatsQUAL™:
ARL Statistics -- E-Metrics
LibQUAL+®
ClimateQUAL™
DigiQUAL™
MINES for Libraries™
•Library Assessment Conferences•Service Quality Evaluation Academy•Library Assessment blog•Effective, Sustainable, Practical Assessment•Profiles … Scenarios …. Balanced Scorecard …. LibValue
Assessment at ARL
www.arl.org
http://www.climatequal.org
ARL Tools for Library Assessment
As we look to the future…
New Tool New
Tool
New Tool
Common Database
StatsQUAL®
Common User Interface
Unified Data Structure
Enhanced Data Mining/Warehousing Opportunities
Faster Development Cycle for New Tools
Common Workflow
Users
http://www.climatequal.org
ARL Tools for Library Assessment
As a result of the work of the New Measures and Assessment Initiative (1999)…
ARL Statistics™
Since 1907-08
LibQUAL+®
Since 2000
MINES for Libraries™
Since 2003
DigiQUAL®
Since 2003
ClimateQUAL™
Since 2007
Partners
2008• Arizona State
University• Cornell University• Duke University• Emory University• Kansas State
University• New York University• Northwestern
University• University of
Houston• University of
Maryland• University of
Massachusetts, Amherst
2007 • Texas A&M
University• University of
Arizona• University of
Connecticut• University of Iowa• University of
Kansas
2009 George Mason
University Illinois State
University Johns Hopkins
University Oberlin College University of
California, Berkeley University of Hawaii
at Manoa University of
Nebraska-Lincoln University of
Wyoming
http://www.climatequal.org
Goals & Research Question for Project
• Develop a tool that assesses the “health” of your library.– Whether the policies, procedures and practices of your
library are supporting your mission and facilitating meeting current and future customer needs.
• Develop large database of norms to help libraries interpret their results.
• Develop an active community of libraries that share experiences and interventions to improve the effectiveness of libraries participating in community
• Track changes in libraries over time to permit objective feedback regarding effectiveness of attempted practices/interventions.
• Empirical validation of the healthy organization theory.
http://www.climatequal.org
Core Concepts
• Climate for Diversity• Climate for Fairness
– Distributive Justice– Procedural Justice– Informational Justice– Interpersonal Justice
• Climate for Innovation• Climate for Continual Learning• Climate for Teamwork• Climate for Psychological Safety• Climate for Customer Service• Immediate Supervisor Scales
– Authentic Leadership; Leader-Member Relationship Quality• Work Group Conflict• Worker Attitudes
http://www.climatequal.org
2000 U. Maryland Survey
Results• Issues identified:
– Managerial training– Mentoring– Standardization of procedures– Decision-making practices, etc.
• Recruitment/selection of ethnic minorities• Performance review• Rewards and recognition
Transformative Changes
• Holding All-Staff Meetings
• Emphasis on learning
• Creating training for supervisors
• Created new position - Coordinator of Personnel Programs
• Focus on team building
http://www.climatequal.org
2004 U. Maryland Survey
Results• Support of diversity and
communication practices• Employees feel they are
treated fairly• Teamwork is valued• Increase in
Organizational Withdrawal from 2000 to 2004
• Indication of consistent ethnic and divisional differences
Transformative Changes
• Conducted focus groups within two divisions
• Groups and teams identifying strategies
• Developing supervisor core competencies
• Incorporating Organizational Citizenship Expectations into work plans
• Beginning workforce planning
http://www.climatequal.org
Community Building
• No one-size-fits-all method of interpreting results or implementing changes
• Creation of intimate community of participants
• In-person events, an online shared workspace, and conversations
• Allows for shared experiences concerning improvement strategies
Assessing the Value of Networked Electronic Services:
Measuring the Impact of Networked Electronic Services (MINES) - MINES for Libraries™
www.arl.org/stats/newmeas/mines.html
The MINES survey
• A research methodology consisting of a web-based survey form and a sampling plan.
• Measures who is using electronic resources, where users are located at the time of use, and their purpose of use.
• Adopted by the Association of Research Libraries (ARL) as a part of the “New Measures” toolkit May, 2003.
• Different from other electronic resource usage measures that quantify total usage (e.g., COUNTER, EQUINOX, E-Metrics, ICOLC guidelines, ISO and NISO standards) or measure how well a library makes electronic resources available (LibQUAL+®, DigiQUAL™).
What is MINES?
• How extensively do sponsored researchers use the new digital information environment?
• Are researchers more likely to use networked electronic resources from inside or outside the library?
• Are there differences in usage of electronic information based on the user’s location (e.g., in the library; on-campus, but not in the library; or off-campus)?
• What is a statistically valid methodology for capturing electronic services usage both in the library and remotely through web surveys?
• Are particular network configurations more conducive to studies of digital libraries patron use?
Questions Addressed
Effective, Sustainable and Practical Assessment
• Association of Research Libraries (ARL) project “Making Library Assessment Work” began in 2005 to “Assess the state of assessment efforts in individual research libraries, identify barriers and facilitators of assessment, and devise pragmatic approaches to assessment that can flourish in different local environments”
• Funded by participating libraries• Conducted by Steve Hiller and Jim Self
– 1.5 day site visit– Presentation and best practices– Interviews and meetings – Report to the library with recommendations
Library Assessment Process
• Focuses on customer needs, defining measurable “outputs” and offering services that meet those needs
• Collects, analyzes and uses data for management, program development, and decision-making
• Emphasizes ongoing communication with customers, opportunities for collaboration, qualitative measures and circular process of continuous improvement
MLAW/ESP:Data Collection Methods
Pre-Visit• Survey on assessment activities, needs etc.• Telephone follow-up• Mining library and institutional web pages
Visit (1.5 days)• Presentation on effective assessment • Group meetings
Follow-up and report• Pursue leads and additional information
Commonly Identified Assessment Needs (30 Libraries)
67%
70%
80%
87%
90%
97%
0% 25% 50% 75% 100%
Build AssessmentCulture
Learning Outcomes
Staff Data Use Skills
Data Analysis
Data Collection
Use Data Effectively
Organizational Indicators of Effective Assessment
• Library leadership/management “truly” supportive
• Customer focus is a shared library value
• Organizational culture receptive to change & improvement
• Assessment responsibility recognized and supported
• Library has strategic planning process and prioritizes
• Evidence/Data used to improve services/programs
– Web sites (usability)
– Facilities (qualitative methods)
– Serial subscriptions (emetrics)
– LibQUAL+™ results are followed-up
Building a Community of Practice
• Biennial Library Assessment Conference– 220 registrants for 2006 conference in Charlottesville, VA– 380 registrants for August 2008 in Seattle, WA
• Workshops– Biennial Service Quality Evaluation Academy– Full day and half day workshops
• Library Assessment SPEC Kit (December 2007)– SCONUL/ARL collaboration for UK and Irish libraries
• Assessment tools– LibQUAL+® (“Millions served”)– MINES for Libraries®– ClimateQUAL™, DigiQUAL™ and more
Customer Needs:
Internal Processes
Marketing & Communications
Library Services Operational Efficiency
LearningAndGrowth
Customer Perspective Financial Perspective
Recruit, develop, and retain productive, highly qualified
staff
Recruit, develop, and retain productive, highly qualified
staff
Strategic Skills / Workforce Infrastructure
Implement space and technology upgrades to
support cutting-edge services and initiatives
Implement space and technology upgrades to
support cutting-edge services and initiatives
CulturePromote culture of
assessment and accountability
Promote culture of assessment and accountability
Develop “community” model of leadership and organizationDevelop “community” model
of leadership and organization
The intellectual crossroads where The intellectual crossroads where research and teaching, tradition and research and teaching, tradition and innovation, faculty and students all innovation, faculty and students all
intersectintersect..Facilitate world-class research, teaching and learning through leadership in the creation, access, use, and preservation of
information and knowledge for scholars at the University of Virginia
UW Strategy Map Based on Strategic Plan Mission
Enhance undergraduate
services
Enhance undergraduate
services
Reshape library spaces
Reshape library spaces
Customer Perspective
Demonstrate library value to research
productivity
Demonstrate library value to research
productivity
Optimize organizational structure and
efficiency
Optimize organizational structure and
efficiency
Financial Perspective
Internal Perspective
Learning and Growth Perspective
Marketing & Communications
Library Services Operational Excellence
Communicate and market library services &
resources
Communicate and market library services &
resources
Develop and promote scholarly communication
plan
Develop and promote scholarly communication
plan
Integrate resources & services into user
environments
Integrate resources & services into user
environments
Strengthen library liaison
program
Strengthen library liaison
program
Build a management information structure Build a management
information structure
Create a workplace of choice
Create a workplace of choice
Enriching the Quality of Life and Advancing Intellectual Discovery by Connecting People with Knowledge
Enriching the Quality of Life and Advancing Intellectual Discovery by Connecting People with Knowledge
Develop staff expertise that supports current & future
TLR
Develop staff expertise that supports current & future
TLR
Strengthen digital and
physical delivery services
Strengthen digital and
physical delivery services
Provide technological infrastructure to support
staff and user needs
Provide technological infrastructure to support
staff and user needs
Culture Infrastructure Strategic SkillsStrategic Skills
Assess effectiveness of services & programs Assess effectiveness of services & programs
Provide access to information resources for world-class TLR
Provide access to information resources for world-class TLR
Strategy Map
Improve discovery of and
access to scholarly resources
Improve discovery of and
access to scholarly resources
Create world-class teaching
& learning environments
Create world-class teaching
& learning environments
Strive for exemplary
service that is responsive to
user needs
Strive for exemplary
service that is responsive to
user needs
Marketing and Communications
Services Operational Effectiveness
Promote the Library’s role in the discovery, dissemination,
and preservation of knowledge
Promote the Library’s role in the discovery, dissemination,
and preservation of knowledge
Integrate the Library into the
University’s teaching,
learning, and research mission
Integrate the Library into the
University’s teaching,
learning, and research mission
Develop highly-trained,
technologically-fluent
superlative staff
Develop highly-trained,
technologically-fluent
superlative staff
Nurture a healthy,
collaborative, and dynamic organization
Nurture a healthy,
collaborative, and dynamic organization
Grow an evidence-
based culture
Grow an evidence-
based culture
Align the Library’s budget with the
University’s mission
Align the Library’s budget with the
University’s mission
Increase alternate sources of revenue
e.g. fundraising, grants and revenue
generation
Increase alternate sources of revenue
e.g. fundraising, grants and revenue
generation
Mission: advance teaching, learning and research at McMaster by teaching students to be successful, ethical information seekers, facilitating access to information resources, providing welcoming spaces for intellectual discovery and promoting the innovative adoption of emerging learning technologies
Track efficiency and effectiveness of Library programs and services
Track efficiency and effectiveness of Library programs and services
Enhance the Library’s commitment to strategic
planning
Enhance the Library’s commitment to strategic
planning
Encourage innovation and
risk taking
Encourage innovation and
risk taking
Mission: To advance research, teaching, learning at Johns Hopkins University by providing relevant and significant collections, services, and instruction for faculty, students, and staff. The Sheridan Libraries through the University Libraries Council partners with other Johns Hopkins Institutions
libraries to maximize support for our constituents while minimizing costs.
Mission: To advance research, teaching, learning at Johns Hopkins University by providing relevant and significant collections, services, and instruction for faculty, students, and staff. The Sheridan Libraries through the University Libraries Council partners with other Johns Hopkins Institutions
libraries to maximize support for our constituents while minimizing costs.
Customer Needs Provide
productive, user-centered
workspaces (virtual and physical)
Provide productive, user-
centered workspaces (virtual and physical)
Ensure fast, easy access to resources and services needed
for research and teaching
Ensure fast, easy access to resources and services needed
for research and teaching
Financial Perspective
Develop strong support base
Develop strong support base
Internal Processes
Communications Programs Operational Efficiency
Promote resources
and services
Promote resources
and services
Expand and preserve access to information
including digital library development
Expand and preserve access to information
including digital library development
Create productive research, learning, study, and social
space
Create productive research, learning, study, and social
space
LearningAndGrowth
Recruit, develop, and retain
productive highly qualified staff
Recruit, develop, and retain
productive highly qualified staff
Promote an adaptive, collaborative, diverse, engaged,
innovative workplace
Promote an adaptive, collaborative, diverse, engaged,
innovative workplace
Create a culture of assessment and accountability
Create a culture of assessment and accountability
JHU Library Strategy Map
Strategy Map
Minimize costsMinimize costs
Identify and improve processes and
infrastructure with greatest impact
Identify and improve processes and
infrastructure with greatest impact
Integrate and strengthen
instructional and research support
services and programs
Integrate and strengthen
instructional and research support
services and programs
ARL Profiles
• Describes the here and now and strategies
• 86 qualitative descriptions of research libraries• Five pages each• Address issues related to collections, services and
collaborative relations• Preliminary report available: directors.arl.org• Working on identifying key areas and defining an
ARL Statistics 2.0 survey
ARL Scenarios
• Process led by Karla Hahn, Associate Executive Director for Transforming Research Libraries, with external consultant
• Interview and focus groups on out of the box thinking attempting to describe different scenarios of the future of research libraries in 20 years or so
• A visioning tool• FAQ on ARL website
IMLS Lib-Value Grant
Partner Institutions: U of Tennessee, U of Illinois at Urbana-Champaign and ARL
Primary Goals• Establish a fact-based articulation of the value
and ROI of the university library resources and services within the wider mission of university administration.
• Develop a model for ROI and tools that implement this model which can be used by other academic libraries.
Impact
• More informed librarians who understand how to conduct and present ROI studies for their libraries and who can measure all aspects of library services and collections
• Improved understanding by university library funders of the ROI and value of university libraries to the overall mission of the university and what products and services are the best investments
01/04/2010
Introductions - Management Team
• Carol Tenopir – UT ([email protected])
• Paula Kaufman – UIUC ([email protected])
• Martha Kyrillidou – ARL ([email protected])
• Donald King - UNC ([email protected])
• Bruce Kingma -Syracuse ([email protected])
• Tina Chrzastowski – UIUC ([email protected])
• Gayle Baker – UT ([email protected])
• Ken Wise – UT ([email protected])
• Rachel Fleming-May – UT ([email protected])
• Regina Mays – UT ([email protected])
• Crystal Sherline – UT ([email protected])
• Andrea Baer – UT ([email protected])
01/04/2010
June Workshops
In Washington DC ….
Return on Investment – Lib-Value with Paula Kaufman, Don King and Bruce Kingma
Value and Impact with Stephen Town