Rubric Assessment of Student Responses to an Information Literacy Tutorial

5/21/2018 Rubric Assessment of Student Responses to an Information Literacy Tutorial

1/43

Rubric Assessment of Student

Responses to an InformationLiteracy Tutorial

Megan OakleafLibrarian for Instruction & Undergraduate

Research

Steve McCannNCSU Libraries Fellow


2/43

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004

Presentation Objectives

Share our experiences.

In changing the way we assess our program

In adapting ACRL outcomes to our project

In selecting a learning artifact to assess

In piloting our assessment plan

Facilitate your experience with this

type of assessment.

Provide take away ideas.


3/43


4/43


5/43

NCSU Libraries

Instruction Program


6/43


Information Literacy

Information literacy is a set of abilities

requiring individuals to "recognize when

information is needed and have theability to locate, evaluate, and use

effectively the needed information."

Association of College and Research Librarieshttp://www.ala.org/ala/acrl/acrlstandards

/informationliteracycompetency.htm


7/43M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004

How have we

measured our success?



The Decision to Change

Goal: To assess student learning of

information literacy skills using

outcomes-based assessment.

Need 2 things

An Artifact to Assess

Outcomes to Measure



What artifact will we assess?

Interactions with students that could

yield assessment artifacts

50-minute one-shot workshops

Library Online Basic Orientation (LOBO)



Disadvantages of Using

Workshops for Assessment Perceived lack of time for open-ended

responses, only m.choice and T/F are

options. Taught by numerous librarians who lack

assessment knowledge.

Inconsistent audiences & content. Incomplete spectrum of outcomesaddressed.



Advantages of Using

LOBO for Assessment Forms basis for IL instruction at NCSU.

Reaches virtually all incoming freshmen.

Recently redesigned, includes open-

ended questions.

Captures student responses in a

searchable database.

Potential for rich data.


12/43


13/43


14/43



What outcomes will we assess?

Information Literacy Competency

Standards for Higher Education

5 Standards

22 Performance Indicators

87 Outcomes




Objectives for Information Literacy

Instruction: A Model Statement for

Academic LibrariansStandards = 5

Perform. Indicators = 14

Outcomes = 35

Bullets = 133


17/43




LOBO Objectives & Outcomes

Objectives = 5

Measurable

Outcomes= 45



First Steps

Set up DB to access student answers.

Match outcomes to questions.How will we know the outcomes been met?

Beginning, Developing, Exemplary

Pilot test a section of LOBO.



The Pilot Test

Problem: Students use web sources for

academic purposes without evaluating

their quality.

Are they duped by low-quality sites?

Can we teach them to be morecritical consumers of information?


21/43


22/43



Evaluating Web Sites

What criteria are you looking for?

What clues can you find?

What specific example can you give

from the web site at hand?

Is the web site a good one for

you to use?


24/43

Evaluating Web Sites


25/43


26/43


27/43


Pilot Test Results

What we found out from 50 randomly

selected student accounts


28/43


LOBO Sections - % Completed

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%100%

1. The

ResearchProcess

2. Defining

ResearchNeeds

3. Developing

a ResearchStrategy

4. Conducting

the Search

5. Evaluating

Resources

6. Using

Resources

Lobo DataJan-April, 04 1,830 total accounts new

accounts

46% of total questions

answered

Evaluated 50 accounts

Sections Total Answers Ttl Quest. Poss. Ans. % Complete

1. The Research

Process7,031 7 13,230 53%

2. Defining Research

Needs9,056 10 18,900 48%

3. Developing a

Research Strategy9,798 11 20,790 47%

4. Conducting the

Search11,984 12 22,680 53%

5. Evaluating

Resources11,582 16 30,240 38%

6. Using Resources 4,692 6 11,340 41%


29/43


Objective 5: Evaluating Resources

#3 Evaluate Web Sites questions:1. Locate a website

2. Evaluate websites authority

3. Evaluate recency/currency4. Identify Bias/Point of View

Section 5: Evaluating Resources

Task Answered Null Possible % Complete1. Evaluate Books 4,455 8,824 13,279 34%

2. Evaluate Articles 1,570 2,224 3,794 41%

3. Evaluate Web Sites 5,557 7,722 13,279 42%


30/43


Q1: Locate a Website

4 points possible: Average score 2.7

Question Text: Type the title and URL (web address) of the web site you will

evaluate here:

Q1: Cumulative Count of Scores

0

5

10

15

20

25

30

35

40

Total 0 3 36 0 11

Count-0 Count-1 Count-2 Count-3 Count-4

Q1: Count of Scores by Outcome

0

5

10

15

20

25

30

35

40

45

Beginning Developing Exemplary Beginning Developing Exemplary

Completion Accuracy


31/43


Q2: Evaluate a Websites Authority


Question Text: Answer the questions above for the web site you're evaluating.

Overall, does what you know about the URL of the web site indicate that it's a good

resource?

Answer the questions above for the web site you're evaluating. Overall, does what

you know about the authorship of the web site indicate that it's a good resource?


0

2

4

6

8

10

12

14

16

18

Total 0 5 1 2 6 16 9 7 4

Count-

0

Count-

1

Count-

2

Count-

3

Count-

4

Count-

5

Count-

6

Count-

7

Count-

8


05

101520

25303540

Beginning

Developing

Exemplary

Beginning

Developing

Exemplary

Beginning

Developing

Exemplary

Beginning

Developing

Exemplary

Us es Criteria Terms Cites Clues Cites Ex amples from Sourc e Judges Whether or Not To

Use


32/43


Q3: Evaluate a Websites Currency



Overall, does what you know about the currency of the web site indicate that it's a

good resource?


0

2

4

6

8

10

12

14

16

Total 0 6 0 5 4 2 12 6 15

Count-

0

Count-

1

Count-

2

Count-

3

Count-

4

Count-

5

Count-

6

Count-

7

Count-

8


05

1015

20253035

Beginning

Developing

Exemplary

Beginning

Developing

Exemplary

Beginning

Developing

Exemplary

Beginning

Developing

Exemplary

Us es Criteria Terms Cites Clues Cites Ex amples f rom Sourc e Judges Whether or Not To

Use


33/43


Q4: Identify a Websites Bias



Overall, does what you know about the bias of the web site indicate that it's a good

resource? Overall, is this web site a good resource to use for your assignment?


0

2

4

6

8

10

12

Total 1 8 3 4 10 11 8 3 2

Count-

0

Count-

1

Count-

2

Count-

3

Count-

4

Count-

5

Count-

6

Count-

7

Count-

8


05

101520

25303540

Beginning

Developing

Exemplary

Beginning

Developing

Exemplary

Beginning

Developing

Exemplary

Beginning

Developing

Exemplary

Uses Criteria Terms Cites Clues Cites Examples from Sourc e Judges Whether or Not To

Use


34/43


Pilot Test Findings

Students proven successful with

mechanical tasks like checking currency

and identifying URLs.

Students are shown as developing

with judgment tasks such as authority

and bias.

Bias is a potential target area.


35/43


How are we using our

assessment results? Changes to LOBO

Add and Reorganize Content

Improve Question Format Enlarge Response Space

Make Rubrics Available?

Train Course Instructors


36/43

Changes to LOBO

Answer these questions about the web site youre evaluating in the space below:

Who created the site? What point of view do they represent?What organizations support the site? What biases might they have?Are links included that point to other viewpoints?Are there signs of bias included in the site?Are you biased toward the site?

Overall, does what you know about the bias of the web site indicate that its a good resource?

ReplaceQuestions

with

Content

Sample StudentAnswers

Get Help with Your Answer

Fix

Questions,

Add Links,

Enlarge

Answer

Space

Add Viewlet

to Model

Applicationof Content

Changes Coming Soon!


37/43

Changes for Instructors

Share Rubrics,

Continue OngoingTraining,

Add Lesson Plans


38/43


How are we using our

assessment results? Changes to the Instruction Program

Including assessment in departmental 3-yr goals

Sharing data with subject-specialist librarians Initiating rubric assessment of advanced

instruction

Reporting results to library administration

and colleagues on other campuses


39/43


Key Learning

Rubrics require lots of revision! Rubrics are effective in measuring

higher-level thinking skills.

Rubrics provide informationadministrators can use for reportingand instructors can use to improveteaching and learning.

Our colleagues are interested inour progress.


40/43


Take Aways

1. Can higher-level thinking skills likeinformation literacy or critical thinking beadequately described in a rubric?

2. Are rubrics good tools for assessing studentresponses to tutorials?

3. What problems did you find in your practicewith these rubrics?

4. What part of this process/project could beapplied at your institution?


41/43


Muddiest Points?


42/43

Contact Information

Megan Oakleaf

Librarian for Instruction &Undergraduate Research

[email protected]

Steve McCann

NCSU Libraries Fellow

[email protected]


43/43

Rubric Assessment of Student

Responses to an InformationLiteracy Tutorial

Megan OakleafLibrarian for Instruction & Undergraduate

Research

Steve McCannNCSU Libraries Fellow

Rubric Assessment of Student Responses to an Information Literacy Tutorial

Documents

Transcript of Rubric Assessment of Student Responses to an Information Literacy Tutorial