ASSESSMENT OF KNOWLEDGE AND SKILLS IN NURSING AND ALLIED HEALTH STUDENT INFORMATION LITERACY INSTRUCTION

A systematic review of the literature

Presented by Pamela Harrison on behalf of Jill Boruff & Pamela HarrisonJune 22, 2015

Background

•Primary researcher: Jill Boruff•Offering course-integrated instruction in PT/OT since 2008• Boruff & Thomas, 2011

•Currently, assessment in PT/OT takes the following forms:• Clicker-based assessment • Self-assessment (confidence scale, short answer)• Assignment: include clinical question, search strategy and selected resources

@jillyb3

#ILassess

What others are saying: Assessment•Oakleaf, 2008: overview of IL assessment methods•Oakleaf, 2009: Information Literacy Instruction Assessment Cycle (English 101)•Schilling & Applegate, 2012: assessment methods used by librarians (multidisciplinary)•Brettle, 2007: systematic review of measures for information skills training (health settings)#ILassess

What others are saying: Instruction

•Maggio & Kung, 2014: information retrieval instruction (UGME); includes information on librarian involvement in teaching/assessment cycle• Just, 2012: information retrieval instruction and assessment (medical students and residents); data on intervention, assessment tool & target•Koufogiannakis & Wiebe, 2006: systematic review of IL instruction (undergraduates)#ILassess

What we hope to do

•Provide a better understanding of how information literacy instruction is being assessed in nursing and allied health•Provide librarians and other educators with suggestions on how to improve assessment within IL instruction

#ILassess

Research question

How is assessment used in IL instruction in Nursing and Allied

Health?

Forms of assessme

nt

Target of assessme

nt

Instructional

context

#ILassess

• Tests of knowledge or skill• Examples: Multiple choice tests, short-answer tests

• Self-assessment or attitudinal surveys• Examples: Quizzes, focus groups

• Course assignments or products• Examples: citation analyses, bibliographies

• Is the assessment formative or summative?

Forms of assessment

#ILassess

Target of assessment 4 assessment areas identified (Khan & Coomarasamy, 2006; Schilling & Applegate, 2012):

Knowledge

• Recall and understand concepts taught

Skills• Ability to apply knowledge in context

Attitudes• Recognize the need to use acquired knowledge and skills

in scenarios, and intend to use those skills in practice

Behaviour• Use acquired knowledge and skills in practice

#ILassess

Instructional context

•Course-integrated, librarian-led workshops•One-shot librarian-led workshops• Instructor-led instruction•Other forms of instruction

(Sources: Khan & Coomarasamy, 2006; Maggio & Kung, 2014)#ILassess

Methodology: Search details

• Search: 1990 to January 16, 2015• Ovid Medline• EMBASE• CINAHL (EBSCO)• ERIC (EBSCO)• LISA• LISTA• ProQuest Theses & Dissertations

•Documentation: PRISMA flow chart (Source: Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., & PRISMA Group. (2009))#ILassess

Search concepts

Nursing or Allied Health

Educational

program

IL Instructio

n or Assessme

nt

#ILassess

Methodology: Inclusion criteria

✔Nursing or Allied Health program✔Degree-seeking students✔Assessment of information literacy skills✔Question formulation✔Searching/resource selection✔Critical appraisal (if a librarian is involved)

#ILassess

Exclusion criteria✗Exclusively about medical students, medical residents, or non-degree programs

✗Continuing education programs or workplace/practitioner training programs

✗Exclusively about accreditation standards or competencies

✗Exclusively about faculty attitude toward, or use of, resources

✗Commentaries: position or policy statements; editorials; product reviews

✗Lists#ILassess

PRISMARecords identified through database

searching(n = 4366 )

Records after duplicates removed

(n = 2965)

Records published from 1990 onwards, and screened (n =

2724)

Additional records identified through other

sources (n = TBD)

Records excluded(n = 2143)

#ILassess

PRISMA continued…Records screened

(n = 2724)Records excluded

(n = 2143) (from previous slide)

Full-text articles assessed for eligibility (n = 605)

Studies included in qualitative synthesis (n=

130)

Full-text articles excluded (n = 475)• No librarian and no IL skills (n =

166)• No assessment described (n =

134)• No program described (n = 84)• Reasons from abstract level (n

= 34)• Review articles (n = 4)• Article not available (n = 11)• Language (n = 33)• Duplicates (n = 6)• Other (n = 3)

#ILassess

Preliminary results: (n = 130) # of included papers, by discipline

PT / OT SLP Nursing Mix Nurs/AH0

10

20

30

40

50

60

70

80

90

100

#ILassess

Preliminary results: (n = 22)PT/OT/SLP IL assessment snapshot

Assessment # Validated ( ✔ )

Assignment(s) 10

Adapted Fresno Test 3 ✔

Dutch Modified Fresno Test 1 ✔

Evidence-Based Practice Profile Questionnaire

1 ✔

Research Readiness Assessment 1 ✔

Vignette Skill Exercise 1

Survey/test: Self-efficacy 2

Survey: Satisfaction 3

Pre-/Post test: Knowledge 2

Survey/test: Behaviour and Attitudes 4

#ILassess

Next steps•Additional material:• For screening and data extraction: studies cited in review articles• For data extraction only: assessment tools or additional data (will contact authors)

•Data extraction for accepted items•Analysis: • What are the most common forms of assessment? Are any of them validated? • Which assessments are being used for which EBP skills (i.e. question formulation or searching)?

#ILassess

Observations and lessons learned•Full text screening for qualitative data takes a long time•Multi-purpose terms result in many false drops•Learning assessment vs. usability testing •Variation in depth of descriptions suggests that the literature would benefit from guidelines on reporting assessment methods#ILassess

Questions?Jill Boruff, Primary Investigator

McGill University jill.boruff@mcgill.ca

Pamela Harrison, Co-Investigator McGill University

pamela.harrison@mcgill.ca

@jillyb3

@tweet_nothings_

References• Boruff, J. T., & Thomas, A. (2011). Integrating evidence-based

practice and information literacy skills in teaching physical and occupational therapy students. Health information and libraries journal, 28(4), 264.

• Brettle, A. (2007). Evaluating information skills training in health libraries: a systematic review. Health Information & Libraries Journal, 24(s1), 18.

• Just, M. L. (2012). Is literature search training for medical students and residents effective? a literature review. Journal of the Medical Library Association: JMLA, 100(4), 270.

• Khan, K. S., & Coomarasamy, A. (2006). A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. BMC Medical Education, 6(1), 59. doi:10.1186/1472-6920-6-59

• Koufogiannakis, D,. & Wiebe, N. (2006). Effective methods for teaching information literacy skills to undergraduate students: A systematic review and meta-analysis. Evidence Based Library and Information Practice, (1)3: 3.

• Continued…

References (continued)• Maggio, L. A., & Kung, J. Y. (2014). How are medical students trained

to locate biomedical information to practice evidence-based medicine? a review of the 2007–2012 literature. Journal of the Medical Library Association: JMLA, 102(3), 184.

• Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & PRISMA Group. (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLOS Medicine, 6(7), e1000097.

• Oakleaf, M. (2008). Dangers and opportunities: a conceptual map of information literacy assessment approaches. portal: Libraries and the Academy, 8(3), 233.

• Oakleaf, M. (2009). The information literacy instruction assessment cycle: A guide for increasing student learning and improving librarian instructional skills. Journal of Documentation, (65)4, 539.

• Schilling, K., & Applegate, R. (2012). Best methods for evaluating educational impact: A comparison of the efficacy of commonly used measures of library instruction. Journal of the Medical Library Association: JMLA, 100(4), 258.