Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State...
-
Upload
avis-walsh -
Category
Documents
-
view
215 -
download
1
Transcript of Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State...
![Page 1: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/1.jpg)
Assessing Information Literacy Skills: A Look at Project SAILS
Joseph A. Salem, Jr.Kent State University
ARL New Measures Initiatives CREPUQFebruary 11, 2005
![Page 2: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/2.jpg)
Context
Explosion of interest in information literacy Accountability Assessment
Formative: for planning/improvement Summative: evidence/documenting
![Page 3: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/3.jpg)
What Is an Information Literate Person? Information Power - American Association of
School Librarians 9 standards
Big6 Information Competency Standards for
Higher Education – ACRL 5 standards, 22 performance indicators, 87
outcomes, 138 objectives
![Page 4: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/4.jpg)
Our Questions
Does information literacy make a difference to student success?
Does the library contribute to information literacy?
How do we know if a student is information literate?
![Page 5: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/5.jpg)
The Idea of SAILS
Perceived need – No tool available Project goal – Make a tool:
Programmatic evaluation Valid Reliable Cross-institutional comparison Easy to administer for wide delivery Acceptable to university administrators
![Page 6: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/6.jpg)
The Project Structure
Kent State team Librarians, programmer, measurement expert
Association of Research Libraries partnership Ohio Board of Regents collaborative grant
with Bowling Green State University (part for SAILS)
IMLS National Leadership Grant Working with many institutions
www.projectsails.org
![Page 7: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/7.jpg)
Project Parameters
Test based on ACRL document Test development model: Systems
design approach Measurement model: Item Response
Theory Tests cohorts of students (not
individuals) – programmatic assessment A name
Standardized Assessment of Information Literacy Skills
![Page 8: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/8.jpg)
Test Development
Systems design approach:
1. Determine instructional goal
2. Analyze instructional goal
3. Analyze learners and contexts
4. Write performance objectives
5. Develop assessment instrument
The Systematic Design of Instruction. 6th ed. By Walter Dick, Lou Carey, James O. Carey. Boston: Pearson/Allyn and Bacon, c2005.
![Page 9: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/9.jpg)
ACRL Standards – Significant Challenges Breadth and depth Objectives
Multi-part Multi-level
Habits/behaviors versus knowledge
![Page 10: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/10.jpg)
Consider Skill Sets
Regrouping the ACRL objectives (and some outcomes)
12 sets of skills organized around activities/concepts
More closely mirrors instructional efforts?
![Page 11: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/11.jpg)
Item Development Process
Review competencies and draft some items:
"How can I know that a student has achieved this competency?"
Formulate a question and answers. This may take several iterations.
Develop additional responses that are incorrect, yet plausible. Aim for five answers total.
![Page 12: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/12.jpg)
Testing the Test Items
Conduct one-on-one trials Meet with individual students, talk through test
items Conduct small group trials
Administer set of items to group, engage in discussion after
Conduct field trials Administer set of items to 500+ students, analyze
data
![Page 13: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/13.jpg)
Measurement Model
Item Response Theory Also called Latent Trait Theory Measures ability levels Looks at patterns of responses
For test-takers For items
Rasch measurement using software program “Winsteps” (www.rasch.org)
![Page 14: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/14.jpg)
Response Pattern Example
Easier questions Harder questions
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
Person A C C C C C
Person B C C C C C
Person C C C C C C
Person D C C C C C C C C
C = gave correct answer
![Page 15: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/15.jpg)
Data Reports
Based on standards and skill sets Looking at cohorts, not individuals Show areas of strength and areas of
weakness
![Page 16: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/16.jpg)
The Person-Item Map
Plots items according to difficulty level Plots test-takers according to their patterns of
responses Can mark average score for cohorts
Cross-institutional average Specific institution average
![Page 17: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/17.jpg)
The Bar Chart
Another representation of the information Group averages
Major, class standing, etc. Which groups are important to measure?
How do you know which differences in means are important?
![Page 18: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/18.jpg)
Current Instrument Status
158 items developed, tested, and in use Most ACRL learning outcomes covered
Not Standard 4: Uses information effectively to accomplish a specific purpose.
12 skill sets developed based on ACRL document
![Page 19: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/19.jpg)
IMLS Grant Status
Phase I complete 6 institutions participated Feedback from institutions
Phase II underway 36 institutions participated
Phase III started June 2004 About 70 institutions participating Wrap-up summer 2005
![Page 20: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/20.jpg)
Project Highlights
Discipline-specific modules Canadian version of the instrument Automated survey generation Automated report generation
![Page 21: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/21.jpg)
Next Steps for SAILS
IMLS grant period ends on September 30, 2005
Stop administering SAILS to allow analysis of the instrument Does the instrument measure what we
want it to? Are institutions getting what they need?
![Page 22: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/22.jpg)
Next Steps for SAILS
Analyze data and input from institutions
Validate the instrument Factor analysis and skill sets Outside criterion testing through
performance testing Test-taker characteristics
Sex, ethnicity, class standing, GPA Test administration methods
![Page 23: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/23.jpg)
Next Steps for SAILS
Re-think how results can be presented or used Scoring for the individual Pre and post-testing Cut scores
Administrative challenges Automate data analysis Re-engineer administrative tools Create customer interface
Test development Develop new items
![Page 24: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/24.jpg)
Summary
Vision: Standardized, cross-institutional instrument that
measures what we think it does To answer the question:
Does information literacy make a difference to student success?
![Page 25: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/25.jpg)
For More Information
www.projectsails.org [email protected]
Joseph Salem, [email protected]
Mary Thompson, project [email protected]; 330-672-1658
![Page 26: Assessing Information Literacy Skills: A Look at Project SAILS Joseph A. Salem, Jr. Kent State University ARL New Measures Initiatives CREPUQ February.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649e0e5503460f94af775a/html5/thumbnails/26.jpg)
Questions?