10/10/2016 - TSLAC...•Assign responsibility •Share findings •Block off time in your schedule...
19
10/10/2016 1 Texas High School Librarians Assessing Library Programs and Collections in Academic Libraries Jacqueline Bronicki Assessment and Statistics Coordinator, University of Houston Libraries, Houston, TX Krystal Wyatt-Baxter Head of Assessment, University of Texas Libraries, Austin, TX
Transcript of 10/10/2016 - TSLAC...•Assign responsibility •Share findings •Block off time in your schedule...
10/10/2016
1
Texas High School Librarians Create College Ready Students
Assessing Library Programs and Collections in Academic Libraries
Jacqueline Bronicki Assessment and Statistics Coordinator,
University of Houston Libraries, Houston, TX
Krystal Wyatt-Baxter Head of Assessment,
University of Texas Libraries, Austin, TX
10/10/2016
1
Programmatic Assessment in Academic Libraries
Krystal Wyatt-Baxter
Head of Assessment, University of Texas Libraries
University of Texas at Austin
Guiding questions
• What is programmatic assessment?
• Why do it?
• How do you get started?
• What if you don’t have an assessment librarian?
• How do you keep it going?
• Where can you learn more?
What is programmatic assessment?
• Cyclical process
• Focused on entire program
• System of setting goals and making improvements
10/10/2016
2
Why do it?
• SACS accreditation
• Helps prioritize work
• Decision-making
• Impact
How do you get started?
• Plug in to existing efforts
• Ask yourself (and your team) some questions
• What are your unit’s main goals?
• What does successfully achieving those goals look like?
• What big decisions do you anticipate making? What information will help you make those decisions?
• What do you need to know in order to make improvements in your work?
• How does your unit’s work connect to your library’s mission? Your institution’s mission?
• What assessment do you already routinely perform?
Breaking down the assessment loop
Goals
Outcomes
Strategies Methods & Targets
Findings
Next Steps
10/10/2016
3
Goals
• Broad overarching statements that address what the program/unit/process is trying to achieve
• Why does the program exist?
Goals
Outcomes
Strategies Methods & Targets
Findings
Next Steps
Example goals
Teaching unit:
• Support undergraduate learning of foundational information literacy skills through direct contact with students and collaborative partnerships with faculty and teaching assistants.
Assessment unit:
• Provide staff with access and support for using data to assess programs and make decisions
Outcomes
• Backbone of your plan
• Break goals down into specific, measurable pieces
• Statements that describe what, specifically, you want to happen as a result of the work you do
Goals
Outcomes
Strategies Methods & Targets
Findings
Next Steps
10/10/2016
4
Example outcomes
Teaching unit:
• Signature Course students will be able to critically evaluate information
Assessment unit:
• Assessment unit will deliver timely, accurate analysis of survey results.
Strategies
• What are you doing to ensure that outcomes are achieved?
• Basically, your work
• Recognizes that many things are out of your control and you won’t always achieve 100% success
Goals
Outcomes
Strategies Methods & Targets
Findings
Next Steps
Example strategies
Teaching unit:
• Teach instruction sessions
• Assignment design
• “Train the trainer”
• LibGuides
Assessment unit:
• Analyze surveys
• Use project management software to plan workflow of analyses
• Meet with statistical consultants as needed
10/10/2016
5
Methods & Targets
Finally, the assessment part:
• How will you determine whether outcomes have been achieved?
• What level of success are you aiming for?
Goals
Outcomes
Strategies Methods & Targets
Findings
Next Steps
Example methods & targets
Teaching unit:
Method
• Pre-and post-test delivered via email
Target
• 70% of Signature Course students score a 3 or higher on the test
Assessment unit:
Method: Timeline tracking of survey analyses
Target: Each analysis delivered within 2 weeks of survey close
Method: Quality check of analyses
Target: 90% of analyses delivered without mistakes
More on methods
• Write outcomes first and use them to guide method choice
• Direct/indirect – try for a mix
• Start with what you already have
10/10/2016
6
Doable methods that answer common questions
• Rubrics
• AAC&U VALUE rubrics
• RAILS rubric bank
• Procedure/policy tracking
• Surveys/feedback forms
• Project SAILS
• NSSE Information Literacy Module
• Research that is “just good enough”
More methods
Closing the loop
• Findings (analysis)
• Next steps (change) – making sure it’s not just on paper
• Share and communicate
Goals
Outcomes
Strategies Methods & Targets
Findings
Next Steps
10/10/2016
7
Examples of loop closing
Teaching unit:
• Found that students weren’t making much improvement in keyword brainstorming
• Decided to focus in on how we were teaching keyword brainstorming by doing individual class activity assessments
Assessment unit:
• Found that survey results weren’t being delivered on time
• Decided to collapse several surveys together to reduce redundancy and streamline workflow
Communicating findings
“I love it so much. So beautiful. Makes me want to cry. A kitchenette? Wow. These sofas? Wow. These OUTLETS. YESSSSS.”
What if you don’t have an assessment librarian?
• Start small
• Look at what you already have
• Find resources on campus
10/10/2016
8
How do you keep it going?
• Timeline and annual processes for planning and closing the loop
• Formal, documented plan
• Assign responsibility
• Share findings
• Block off time in your schedule
Where can you learn more?
• Library Assessment Conference
• Past Proceedings
• Library Assessment blog & listserv
• LLAMA Assessment Section
• TLA Assessment Interest Group
• Individual library & institutional assessment sites
Further questions?
10/10/2016
1
Effective Library and Collection Assessment: An Evidence-Based Approach
University of Houston Libraries Jackie Bronicki, Assessment and Statistics Coordinator
What I’ll try to cover today (both Library and Collection Assessment)
Overview Step 1 – Planning Phase
Step 2 – Data Collection Phase
Step 3 – Analytical Phase
Step 4 – Data Visualization and Reporting Out Phase
Lessons Learned (the hard way)
10/10/2016
2
1 – Planning Phase
1.Project Proposal
• Timelining • Stakeholders • Relationships • Analytical Plan
Example: Library Assessment – 4 year Proposal
Example: Library Assessment – Collection Gap Analysis Example (2013-2014)
10/10/2016
3
10/10/2016
4
10/10/2016
5
Gap Analysis – Analytical Plan for Monograph Data
1. Distribution and Frequency of Call Numbers by LC Class for Print Monographs
2. Ratio – Circulation of print and ILL Calculations
3. LC Subset B – further frequency and distribution by two letter subsets
Show the variance that can exist at subset level
4. Usage by Call Number Range
5. YTD Usage vs. All Usage
6. Age (Publication Year) of collection by call number
7. Usage by age (Stratify by call number)
2 - Data Collection
•Have to understand your systems
•Unique Identifiers
•Cleaning
•Version Control
•Type of Data
Multiple Data Sources: • Catalog – OPAC • Link Resolver • Electronic Resource Management
System (ERM) • Google Analytics • EZProxy
10/10/2016
6
File-Naming is the
3 – Analytical phase
10/10/2016
7
If PEU>1 Overused If PEU<1 Underused
Mean RBH=1.54
GAP ANALYSIS
LC Subclass
Percent of Holdings
Percent Usage PEU
Holdings Usage
Percent of ILL Borrowing RBH ILL Usage
B 1.32% 1.43% 1.08 Overused 0.79% 0.60 Underused
BC 0.09% 0.08% 0.82 Underused 0.05% 0.51 Underused
BD 0.24% 0.20% 0.84 Underused 0.24% 1.01 Underused
BF 1.22% 1.78% 1.46 Overused 2.00% 1.64 Overused
BH 0.07% 0.09% 1.29 Overused 0.05% 0.68 Underused
BJ 0.22% 0.27% 1.21 Overused 0.18% 0.79 Underused
BL 0.42% 0.65% 1.56 Overused 0.69% 1.65 Overused
BM 0.10% 0.07% 0.67 Underused 0.09% 0.95 Underused
BP 0.13% 0.26% 1.95 Overused 0.34% 2.57 Overused
BQ 0.04% 0.10% 2.63 Overused 0.32% 8.05 Overused
BR 0.36% 0.33% 0.91 Underused 0.70% 1.96 Overused
BS 0.22% 0.16% 0.73 Underused 0.36% 1.62 Overused
BT 0.16% 0.13% 0.85 Underused 0.40% 2.53 Overused
BV 0.18% 0.15% 0.86 Underused 0.44% 2.49 Overused
BX 0.52% 0.29% 0.56 Underused 1.69% 3.23 Overused
If PEU>1 Overused If PEU<1 Underused
If RBH>Mean RBH Overused If RBH<Mean RBH Underused
Mean RBH=1.54
LC Subclass Holdings Usage ILL Usage Action
B Overused Underused No Changes BC Underused Underused Ease Off BD Underused Underused Ease Off BF Overused Overused Growth Opportunity
BH Overused Underused No Changes BJ Overused Underused No Changes BL Overused Overused Growth Opportunity
BM Underused Underused Ease Off BP Overused Overused Growth Opportunity
BQ Overused Overused Growth Opportunity
BR Underused Overused Change Purchasing
BS Underused Overused Change Purchasing
BT Underused Overused Change Purchasing
BV Underused Overused Change Purchasing
BX Underused Overused Change Purchasing
ILL underused, but what we have is used
heavily
Collection proportionally underused with little
outside demand
Collection proportionally overused and
demands a lot of ILL support
Existing collection is underused but has
overuse of ILL
Using the Data to Make Collection
Development Recommendations
10/10/2016
8
4 – Data Visualization
And Reporting out
• Tell the story quickly • Choose the right software to tell that
story • Use Colors and Size to tell the story • Don’t Data Dump – this is hard for me! • Report out often to Stakeholders (both
during and after the project)
SPAN YEARS ANALYSIS
10/10/2016
9
5.70%
3.80%
90.40%
3.00%
1.90%
95.10%
4.00%
3.10%
92.90%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
No, and I don’t plan to purchase one.
No, but I plan to purchase one.
Yes, I currently own one.
Do you own a laptop?
University of Houston DR Public Institutions All US institutions
Jackie Bronicki Assessment & Statistics Coordinator
10/10/2016
1
Texas High School Librarians Create College Ready Students
Assessing Library Programs and Collections in Academic Libraries
Jacqueline Bronicki Assessment and Statistics Coordinator,
University of Houston Libraries, Houston, TX [email protected]
Krystal Wyatt-Baxter Head of Assessment,
University of Texas Libraries, Austin, TX [email protected]
