NCLA Learning Outcomes Assessment Workshop

53
1 Assessing for Improvement Diane Harvey NCLA-BIG Workshop May 21, 2010

description

Workshop for North Carolina Library Association Bibliographic Instruction Group, May 21, 2010, UNC-Greensboro

Transcript of NCLA Learning Outcomes Assessment Workshop

Page 1: NCLA Learning Outcomes Assessment Workshop

1

Assessing for Improvement

Diane HarveyNCLA-BIG WorkshopMay 21, 2010

Page 2: NCLA Learning Outcomes Assessment Workshop

2

Learning outcomes for today

At the end of this session, participants will be able to:

1. Describe the learning outcomes assessment process

in libraries.

2. Write learning outcomes for an information literacy instruction session.

3. Choose appropriate assessment methods.

4. Define ways to work with assessment results.

Page 3: NCLA Learning Outcomes Assessment Workshop

Assessment climate in higher education

3

Spellings Commission http://www.ed.gov/about/bdscomm/list/hiedfuture/index.html

Voluntary System of Accountability (VSA) http://www.voluntarysystem.org/

Accrediting Agencies http://www.sacs.org/

Page 4: NCLA Learning Outcomes Assessment Workshop

SACS standards

3.3.1 …identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results

3.8.2 …ensures that users have access to regular and timely instruction in the use of the library and other learning/information resources.

4

Page 5: NCLA Learning Outcomes Assessment Workshop

Assessing for improvement5

We don’t assess to prove…

…but to improve.D. Stufflebeam

Page 6: NCLA Learning Outcomes Assessment Workshop

Assessment in libraries6

Focus on collections and services (e.g. ARL LibQUAL)

Focus on standards (e.g. ACRL IL Competency Standards)

Page 7: NCLA Learning Outcomes Assessment Workshop

What is LOA: learning outcomes assessment?7

Systematic look at what students are learning

Moving from “What am I going to teach today?” to “What do I want students to learn today?”

Page 8: NCLA Learning Outcomes Assessment Workshop

What LOA is NOT8

NOT evaluation of teaching

NOT evaluation of program

Page 9: NCLA Learning Outcomes Assessment Workshop

Why assess student learning in libraries?9

Affirm commitment to student learning

Improve effectiveness of instructional programs

Align with campus, disciplinary, and higher ed efforts

Be recognized on campus for contributions to student learning

Page 10: NCLA Learning Outcomes Assessment Workshop

10

Assessment loop

Develop student learning

outcomes

Set criteria

Devise assessment measures

Do assessment

Work with results

Page 11: NCLA Learning Outcomes Assessment Workshop

11

Levels of assessment

Institutional (University wide)

Program (e.g. Department)

Course

Session (e.g. IL instruction)

Page 12: NCLA Learning Outcomes Assessment Workshop

LOA in libraries: constraints12

Lack of consistent, reliable access to students

IL instruction: is it a “program”?

Program assessment vs. session assessment

Difficult to isolate effects of library instruction

Page 13: NCLA Learning Outcomes Assessment Workshop

Know your IL program!13

Do you offer a for-credit IL course?

Do you provide IL instruction for large programs (e.g. freshman writing)?

Is IL integrated into the curriculum?

Do you offer IL for capstone, thesis or senior seminar courses?

Page 14: NCLA Learning Outcomes Assessment Workshop

Map your IL program14

First Year Writingand Freshman Seminars

Subject specific instruction

Honors capstone seminars

Page 15: NCLA Learning Outcomes Assessment Workshop

Where will you assess student learning?15

On the instruction session level?

In a for-credit library skills course?

Across a series of library instruction sessions?

Broadly across the student population?

Page 16: NCLA Learning Outcomes Assessment Workshop

Remember…16

Library instruction doesn’t work the same way as instruction in an academic department.

Know the shape of your program – be able to explain how and why you assess student learning.

Page 17: NCLA Learning Outcomes Assessment Workshop

17

Page 18: NCLA Learning Outcomes Assessment Workshop

18

Page 19: NCLA Learning Outcomes Assessment Workshop

Some simple approaches19

Develop learning outcomes for your instruction program, and look at where they are taught – “snapshots”

Focus on large programmatic efforts e.g. first year writing

Highlight specific academic departments

Assess individual library instruction sessions – develop shared learning outcomes, compare/contrast

Page 20: NCLA Learning Outcomes Assessment Workshop

Snapshots20

Outcome: Students will be able to distinguish between popular and scholarly journals.

Assessment: pre/post test in freshman English, one minute paper in senior capstone

Page 21: NCLA Learning Outcomes Assessment Workshop

21

What is a learning outcome?

A learning outcome is one sentence that indicates what students should represent, demonstrate or produce as a result of what they learn.

- source: Peggy Maki

Page 22: NCLA Learning Outcomes Assessment Workshop

22

Good learning outcomes:

Focus on what students will learn/know/be able to do

Describe actions or behaviorsAre results orientedAre observable and measurableInclude a time frame

Page 23: NCLA Learning Outcomes Assessment Workshop

23

ACRL Standards

3 levels: standards, performance indicators, outcomes

Can be used for LOA

Examples: (1)Identifies keywords, synonyms and related terms for the information needed. (2) Selects an appropriate documentation style and uses it consistently to cite sources

Page 24: NCLA Learning Outcomes Assessment Workshop

Writing outcomes24

Today, we’ll write learning outcomes for a library instruction session.

Page 25: NCLA Learning Outcomes Assessment Workshop

25

Exercise: What do you cover?

Think of an information literacy instruction session that you teach on a regular basis.

List what you would cover in that session.

Page 26: NCLA Learning Outcomes Assessment Workshop

26

Bloom’s Taxonomy

Classification of educational objectivesPublished in 1956, revised in 2001 (changes: noun to

verb, synthesis/creating becomes highest level)

Taxonomy = classificationCognitive levels (lower higher)Provides way to express outcomes

Page 27: NCLA Learning Outcomes Assessment Workshop

27

Bloom’s levels (lowest highest)

Knowledge/RememberingComprehension/UnderstandingApplication/ApplyingAnalysis/Analyzing Evaluation/EvaluatingSynthesis/Creating

Page 28: NCLA Learning Outcomes Assessment Workshop

28

Verbs for information literacy

Knowledge/Remembering: define, list, recognize Comprehension/Understanding:characterize, describe,

explain, identify, locate, recognize, sort Application/Applying: choose, demonstrate, implement,

perform Analysis/Analyzing: analyze, categorize, compare,

differentiate Evaluation/Evaluating: assess, critique, evaluate, rank, rate Synthesis/Creating: construct, design, formulate, organize,

synthesize

Page 29: NCLA Learning Outcomes Assessment Workshop

Verbs to avoid29

Understand

Appreciate

Know about

Become familiar with

Learn about

Become aware of

Page 30: NCLA Learning Outcomes Assessment Workshop

30

Learning outcomes formula

1. Time frame2. Student focus3. Action verb4. Product/process/outcome

Page 31: NCLA Learning Outcomes Assessment Workshop

31

Using the formula

Time frame: “At the end of the library session…”

Student focus: “…students will be able to…”Action verb: “…identify…”Product/process: “…a relevant database for

their term paper research.”

Page 32: NCLA Learning Outcomes Assessment Workshop

32

Another example:

Time frame: “After completing the online tutorial…”

Student focus: “…students will be able to…”Action verb: “…differentiate between…”Product/process: “…scholarly journals and

popular magazines.”

Page 33: NCLA Learning Outcomes Assessment Workshop

33

One more example:

Time frame: “By the time they complete their undergraduate program…”

Student focus: “…students will be able to…”Action verb: “…construct…”Product/process: “…a research question that

can be investigated using primary archival resources.”

Page 34: NCLA Learning Outcomes Assessment Workshop

34

Examples: bad and better

Bad: Students will understand how to use social science databases.

Better: Students will perform a search in Social Sciences Abstracts that retrieves relevant items.

Bad: Students will be able to search the catalog.Better: Students will use the catalog to identify a book on their

topic.

Bad: Students will appreciate the importance of correct citation.Better: Students will produce citations in correct APA format.

Page 35: NCLA Learning Outcomes Assessment Workshop

35

Now, write your own!

Choose two items from the list of what you cover in an information literacy session.

Transform them into learning outcomes, using the formula.

Find a partner and critique your outcomes using the following checklist.

Page 36: NCLA Learning Outcomes Assessment Workshop

36

Checklist for learning outcomes

Includes a time frame?Focuses on students?Uses action verbs?Names a product or process?Is measurable/observable?Prompts a measure/method?Will be useful for you to assess?

Page 37: NCLA Learning Outcomes Assessment Workshop

Sample size37

Use any convenient sample that will give you useable information.

Make sure that groups of interest are represented.

Sample size is important only if you plan to publish your results.

Do you want to generalize to the entire student population?

Page 38: NCLA Learning Outcomes Assessment Workshop

Assessment Methods

Ask yourself

“How will I know?”

38

Page 39: NCLA Learning Outcomes Assessment Workshop

39

Assessment methods

Knowledge testOne Minute paper & variationsBibliography analysisConcept InventoryStandardized test

Page 40: NCLA Learning Outcomes Assessment Workshop

Knowledge tests40

Tests knowledge and/or skills before and/or after library instruction session.

Can be given at end of library session or later in semester.

Can use clickers to gather data during library session.

Sample questions:1. What is the difference between a library catalog

and a database?2. The Boolean operator “or” narrows a search statement (true/false).

Page 41: NCLA Learning Outcomes Assessment Workshop

One minute paper & variations41

Sample questions:

1. What is the most important thing about library research you learned today?

2. 3-2-1 (three things you learned, two things you’re still confused about, one thing you’d change about session)

3. What is one question you still have?

4. In your research, what will you do differently after today’s session?

Page 42: NCLA Learning Outcomes Assessment Workshop

Bibliography analysis42

Look for citations from scholarly/peer reviewed journals

Look for citations for books and journals owned by your library

Look for articles retrieved from your databases.

Page 43: NCLA Learning Outcomes Assessment Workshop

Concept inventory43

Make a checklist of 3-12 important concepts students need to master.

Ask students to explain each concept in a sentence or two. If a concept is unfamiliar, they should leave blank.

Count good responses for each concept, then plan future instruction.

Page 44: NCLA Learning Outcomes Assessment Workshop

Standardized tests44

I-Skills

ILAT (James Madison University)

SAILS

Page 45: NCLA Learning Outcomes Assessment Workshop

I-Skills45

Developed by ETS7 ICT proficienciesTracked to ACRL standardsTask based, 2 sections, 75 minutes

Page 46: NCLA Learning Outcomes Assessment Workshop

SAILS: Standardized Assessment of Information Literacy Skills

46

45 questions 35-50 minutes Multiple choice Can require cooperation of faculty and administrators Sample question: If you wanted to search for a topic that

has several components, such as nutrition for pregnant women, which operator would you use? (and, or, not, adj)

Source: Project Sails https://www.projectsails.org/abouttest/samples.php

Page 47: NCLA Learning Outcomes Assessment Workshop

ILAT – James Madison University47

Web based60 itemsTests ACRL standards 1,2,3 and 5

Page 48: NCLA Learning Outcomes Assessment Workshop

48

Using assessment information

Rewrite learning outcomesChange what you do/how you teachWork collaboratively with colleagues

(librarians and faculty)Revise assessment measuresShare the news in the library and on campus

Page 49: NCLA Learning Outcomes Assessment Workshop

49

Learning outcomes are:

Fundamental components of a learning assessment program

Focused on student learningIndicate what students will know/be able to doMeasurable, observable, overt

Page 50: NCLA Learning Outcomes Assessment Workshop

Learning Outcomes Assessment 50

A tool to help librarians

improve student learning

Page 51: NCLA Learning Outcomes Assessment Workshop

51

LOA resources

Angelo, Thomas (1993). Classroom assessment techniques. Maki, Peggy (2004). Assessing for learning : building a

sustainable commitment across the institution. Middle States Commission on Higher Education(2003).

Developing research & communication skills: guidelines for information literacy in the curriculum http://msche.org/publications/devskill050208135642.pdf

Neely, Teresa Y(2006). Information literacy assessment : standards-based tools and assignments.

Radcliff, Carolyn et.al (2007). A practical guide to information literacy assessment for academic librarians.

Student Learning Outcomes Assessment (University of Virginia)http://www.web.virginia.edu/iaas/assessment/outcomes.htm

Page 52: NCLA Learning Outcomes Assessment Workshop

52

Let’s assess this workshop!

Page 53: NCLA Learning Outcomes Assessment Workshop

53

Thank you!

Diane Harvey

Head, Instruction & Outreach

Perkins Library

Duke University

[email protected]