Post on 28-Dec-2015
The Required Library Component: Assessing First-Year Teaching in the
Small Academic Library
Susan von Daum Tholl, PhD, DirectorDiane Zydlewski, Head of Reference
Anne Hancock, Collection Development Librarian
Scope of our discussion
• Setting the stage—gaining administration approval for a required library component
• Development of the library curriculum within the FYS; one-on-one faculty collaboration and attendance at FYS faculty meetings
• Hybrid approach and revision of our online tutorial
• Continual modifications of assignments and library workshop
• Data gathering; faculty collaboration
• Statistical analysis, which is ongoing
• Use of our results to mold the future. . . .
• Pre-2005 — Basic info lit session offered in non-credited venues
• Fall 2005 — First-Year Seminars required Library Component offered in both fall and spring semester
• 2005-Spring 2007 — Worked 1 on 1 with faculty on assignments
• 2007-2008 — Piloted customized assignments and pre- and post- testing by individual seminar
• Fall 2008 — Applied to IRB; approval received January 2009
• Fall 2009 — 3-Year Study began; all seminars offered in Fall semester
• Fall 2011 — Study completed
Timeline of Development
Stated Goal of FYS
“engage first-year students in research
and learning the basic elements of
conducting research specifically using
the resources of the library”
• Pre-2005 — Basic info lit session offered in non-credited venues
• Fall 2005 — First-Year Seminars required Library Component offered in both fall and spring semester
• 2005-Spring 2007 — Worked 1 on 1 with faculty on assignments
• 2007-2008 — Piloted customized assignments and pre- and post- testing by individual seminar
• Fall 2008 — Applied to IRB; approval received January 2009
• Fall 2009 — 3-Year Study began; all seminars offered in Fall semester
• Fall 2011 — Study completed
Timeline of Development
Pre-TestOnline Tutorial and Welcome Assignment
Workshop & Workshop
Assignment – Clickers (Year 3)
Evaluation Assignment (Years 1 & 2)
Post-Test
Substance of the
Library Component
from fall 2009 to fall 2011
Criteria for eligibility of participants:
• First-Year entering students
• Age (over 18 years of age at the pre-test)
• Completion of both pre- and post-test
• Consent letter explaining the process for opting out
Data collected for individual students:
• Pre- and Post-test scores
• Assignment grades
• Number of assignments completed
• Gender
YearEligible
Participants/Enrollments
% students showing positive
change pre-to post-test for all participants
% increase pre- to pos-test for all
participants
Average pre-test grade
Average post-test
grade
Fall 2009 324/366(90.00%)
88.89% 45.01%MIN=-50.00%MAX=300.00%
10.00 MIN=2.00 MAX=18.00
13.65 MIN=1.00 MAX=18.00
Fall 2010 411/542(75.83%)
88.10% 48.65%MIN=-30.00%MAX=550.00%
10.11 MIN=2.00MAX=17.00
13.82 MIN=6.00 MAX=18.00
Fall 2011 443/551(80.40%)
97.97% 54.38%MIN= -28.75%MAX=500.00%
10.18MIN=2.00MAX=17.00
14.81MIN=5.00MAX=18.00
Rates of Improvement during the Study
A Hybrid Approach:
Combining a Hands-On Workshop and an Online Tutorial
Diane Zydlewski, Head of Reference
The Evolving Role of the Tutorial
Fall 2003Online PowerPoint Presentation accompanying a live presentation.
Fall 2005Web-based Tutorial completed independently or in a workshop.
Spring 2008Web-based Tutorial completed during a required workshop.
Fall 2008Combination of the tutorial and hands-on activities in the workshop.
Pre-Test
Online Tutorial and
Welcome Assignment
Workshop with clickers
and Workshop /Evaluation
Assignment
Post-Test
Current Role of the Tutorial within the Library Component of FYS Program
Scope of the Tutorial
Pop-up alert appears if the
wrong answer is selected.
Technical and Design Aspectsof the tutorial
(See also the best practices handout.)
Fall 2005
Simulation of Library Web Site.
Fall 2011
Spring 2007
Fall 2011
Assignments: A Dynamic Information Literacy Tool
Anne Hancock, Collection Development Librarian
WelcomeAssignment
• Complete one week before workshop
• Register for Library Barcode
• Questions covered:• basic library info• creating search
strings• online catalog• online databases
WorkshopAssignment
• Hand out during Library Workshop
• Identify research topic
• Create search string
• Find book, article, and website
• Complete outside of class, due 1 week later
• Evaluate resources found in Workshop for content quality
• Create citation
Do
ne
Concept & Design : A “Stepped” Approach
Evaluation & Citation
Assignment
PROBLEMS
• Too much material
• Assignment questions unclear
• Faculty confusion about assignment
• Students not reading instructions
The Welcome Assignment
SOLUTIONS
• Pared down scope
• Conformed assignment questions to tutorial language and SIMPLIFIED.
• Reinforced assignment process with faculty
• Made instructions stand out more
(increased spacing and font)
PROBLEMS
• Students did not have enough time!!!!
• Workshop sessions of different lengths (50 or 75 minutes)
• Number and types of resources a challenge
• Search String CONFUSION!!!
The Workshop Assignment
SOLUTIONS
• Pared down number of resources we asked students to find.
• Simplified search string portion of assignment.
PROBLEMS
• Questions open-ended
• Student answers incomplete
• Difficult to grade subjective questions in a standardized way
• Citations
• Faculty and Student perceptions
The Evaluation & Citation Assignment
SOLUTIONS
• Rewrote assignment
• Used guided evaluation questions
• Removed open-ended questions
• Removed formal citation and asked bibliographic questions instead.
• Eliminated assignment in 2011
Prior to IRB studyFaculty
Last year of IRB studyOne Librarian
Grading Methods
Data Gathering & Analysis
Data Collected:
• Pre- and Post-test scores and question by question answers
• Assignment grades
• Number of assignments completed
• Gender
Data Analysis:
• Created IBM SPSS database for each year
• Data entered into IBM SPSS
• Using IBM SPSS software to run statistical tests
All Welcome only Workshop only Evaluation only None0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Assignment Completions by Year
2009
2010
2011
Assignments Completed
All Welcome only Workshop only Evaluation only None0%
20%
40%
60%
80%
100%
120%
140%
Average % Change Pre to Post-Test by As-signment Completed
2009
2010
2011
Assignment Completed
Pe
rce
nta
ge
Ch
an
ge
YearEligible
Participants/Enrollments
% students showing positive
change pre-to post-test for all participants
% increase pre- to pos-test for all
participants
Average pre-test grade
Average post-test
grade
Fall 2009 324/366(90.00%)
88.89% 45.01%MIN=-50.00%MAX=300.00%
10.00 MIN=2.00 MAX=18.00
13.65 MIN=1.00 MAX=18.00
Fall 2010 411/542(75.83%)
88.10% 48.65%MIN=-30.00%MAX=550.00%
10.11 MIN=2.00MAX=17.00
13.82 MIN=6.00 MAX=18.00
Fall 2011 443/551(80.40%)
97.97% 54.38%MIN= -28.75%MAX=500.00%
10.18MIN=2.00MAX=17.00
14.81MIN=5.00MAX=18.00
Rates of Improvement during the Study
Use of Results to Mold the Future Library Component. . .
• Statistical analysis is ongoing.
• Analyze results of the pre- and post-test question-by- question to identify patterns of item difficulty to indicate what our teaching must address.
• We are looking at completion rates for the Assignments as they relate to individual student learning outcomes on the post-test.
The Future . . .
• Rework the pre- and post-tests, clarifying some language and creating distractors to develop a stronger test.
• Aside from the IRB Study, anonymous “clicker” data on self-efficacy and student satisfaction from the workshops needs to be compiled and analyzed.
The Study as a springboard for new initiatives . . . .
• Use statistical analysis to help to inform the various levels of our Information Literacy Program
• Formalize our required Library Workshop in the First- Year Writing Program, whose topic is a more advanced look at evaluation, especially online resources
• Apply to IRB for a follow-up study of participants in Years 2 and 3 of the FYS Study when they are Seniors
• Develop focus groups with the Instructional Assistants
• Fill a new FT position – Instructional Librarian
• Create a faculty development program
Thank you!
Our Contact information:
Susan von Daum Tholl, PhDtholl@emmanuel.edu
Diane Zydlewskizydlewsd@emmanuel.edu
Anne Hancockhancoan@emmanuel.edu
http://www1.emmanuel.edu/library