LIGNING LEARNING OUTCOMES AND GRADING STANDARDS...

LESS ASSESSMENT, MORE LEARNING ALIGNING LEARNING OUTCOMES AND GRADING STANDARDS WITH

PROGRAM AND COURSE-LEVEL ASSESSMENT AND FEEDBACK

Materials for the Keynote Session in

MAKING PROGRAM ASSESSMENT WORK FOR YOU

The Ohio State University

9:30 to 10:45 AM, Friday 8 March 2013

Dr. Tom Angelo, Adjunct Professor of Higher Education La Trobe University – Melbourne, Australia

Some terms/concepts that might be of use . . .

Surface and Deep learning

Intended Learning Outcomes (ILOs)

Backward Design

Strategic Alignment

Bus Test, Parrot Test, Car Park Test

Cognitive load

Metacognition

Formative and Summative assessment

Novice-Expert differences

The 80/20 Rule (aka, the Pareto Principle)

Less Assessment, More Learning — Keynote at OSU Program Assessment Conference – 8 March 2013 page 2

Tom Angelo – [email protected] – 8 March 2013

FIVE DIMENSIONS OF HIGHER LEARNING

Approximate percentage What percentage of their of the instruction you got instruction your undergrad in your undergraduate years in . . . students need now in . . .

DECLARATIVE LEARNING Learning What Learning facts and principles

PROCEDURAL LEARNING Learning How Learning skills and procedures

CONDITIONAL LEARNING Learning When and Where Learning applications

REFLECTIVE LEARNING Learning Why Learning to understand

one's self and others

METACOGNITIVE LEARNING Learning How to Learn Learning to direct and manage one's own learning



Intended Learning Outcomes (ILOs)

What & how well are students expected to know and be able to

do by end of course?

Diagnostic Assessments

How will you find out what students already know and can do – and how well – re: ILOs at start of course?

Major Learning Assignments

What work will students actually do in order to develop/practice/ master the course ILOs?

Formative Assessments

How & when will you provide feedback for learning before high-stake assessments?

Summative Assessments

How & when will you assess and grade students’ individual achievement of the ILOs?



Clarifying Intended Learning Outcomes (ILOs)

Examples to consider, critique, and perhaps improve from Phrenology 101

1. On completion of this course, you should be able to:

A. Demonstrate enhanced knowledge of the basic tenets of phrenology and its history

B. Demonstrate understanding of what was current best practice of phrenology, as it was practiced in England of the 1840s

C. Appreciate the relationship of phrenology to neuroscience

2. When you have completed this course, you should be able to:

A. List the six basic tenets of Gall’s phrenological system

B. Identify, locate, and explain the functions of at least 30 of the “organs” of the brain

C. Explain the significance of organ size and shape

D. Identify and summarize the key contributions of at least six major figures in the history of phrenology

3. To successfully complete this course, you must demonstrate that you can:

A. Correctly locate and label all 35 organs on a map of the skull

B. Phrenologize three subjects in one hour, summarize your analyses of all three in writing in the second hour, and achieve at least 85% agreement with expert analyses

C. Prepare a character analysis and related career and marriage advice for a fourth subject, achieving at least 85% agreement with the expert responses

D. Develop a 20-minute talk on your case study (C above), complete with visuals, for presentation at the OSP (Ohio Society of Phrenologists) and evaluation by the members.

[Presentation quality must be rated “Very Good” or “Excellent” by at least 80% of those OSP members in attendance].



A Simple Grading Grid/Minimal Rubric for Low-Value Assignments

Grading grid for Session Preparation Assignments (SPAs) in EDU4CTL

5 Points [Excellent overall]

Meets all the criteria for “Good work” below, and:

Goes beyond expectations in making/analyzing connections, connecting examples to theory/research, identifying several potential applications/ implications for practice, and/or

Provides new information/ideas/concepts that, when shared, will contribute to the learning of other Foundations participants

4 Points [Good overall]

Meets all the criteria for “Satisfactory work” below, and, in addition:

Makes explicit relevant links/connections between personal experience and readings in nearly all/all appropriate responses

Explains the meaning/importance of at least some of those links/connections

Identifies at least one potential application/implication for practice

3 Points [Satisfactory overall]

Responds to key requirements of all prompts/questions

Provides relevant examples of personal/professional experience as required

Makes explicit some relevant links/connections between personal experience and assigned readings

Cites and references correctly 90%-100% of the time

Writes clear, coherent prose that is appropriate to the task in grammar, tone and diction

2 Points [Not yet satisfactory]

Fails to meet one or two key ‘Satisfactory’ criteria

1 Point [Incomplete overall]

Fails to meet three or more key ‘Satisfactory’ criteria

0 Points [Not yet submitted]

May be submitted until 28 May 2012 with lecturer’s permission



FEEDBACK ON ASSESSMENT TASK: SOCIOLOGY 1 AAB

Title of assessment task: Main Essay (Assignment 2, Task 2) 35 % of final mark

Student’s name: Year: 1

Tutor:

MARKING CRITERIA

Disciplinary Knowledge Excellent Good Satisfactory Passable Poor

Used and applied key concepts of the subject where applicable as per the lists on the lecture summaries (e.g., insider/outsider; hegemony; antipodes, culture, cultural traffic, culture industry, mcdonaldization, globalization, dissimulation, class, status, party, new world settler society etc.)

Integrated place, time and division dimensions of the topics being critically analysed

Used and integrated into your argument and discussion at least 10 references including some from the textbook esp. chapters 64/65 (Papastergiardis/Hage).

Graduate Capabilities EXCEEDED MET NOT MET

Writing

Responded to the question with a clear argument

The writing is coherently structured

Extent of grammar and spelling errors

Referencing: Sources and evidence used appropriately

Creative Problem Solving

Identified what needed to be understood about the problem

Applied relevant disciplinary concepts/methods to the problem

Submission Requirements MET NOT MET

Submitted to Turnitin

Within the word limit

Submitted on time No. of days late:

FINAL MARK: /35

COMMENTS:



Macroeconomics Essay Grading Grid Assignment: Write a well-structured, enlightened critical essay about current economic conditions that demonstrates command of existing economic knowledge, appropriate interpretation and application of that knowledge, and demonstrates appropriate use of data and argumentation to support well-reasoned policy recommendations. Basic Questions: What is the current macroeconomic situation in the U.S.?

What is the likely prognosis for the next 12 to 24 months? What are your economic policy recommendations?

Criterion

Dist

ingu

ishe

d

Exce

eds

Expe

ctat

ions

Mee

ts a

ll

Expe

ctat

ions

Mee

ts s

ome

Expe

ctat

ions

Unsa

tisfa

ctor

y

Mis

sing

or f

ails

min

imum

req.

Structure 10 8 6 4 2 0

Meets all minimum requirements

Executive summary is effective & concise

Introduction clearly lays out a roadmap for

the paper and places the information in

context

Body addresses all the Basic Questions,

includes the argumentation and data

Conclusion provides summary and closure

Content

Knowledge 30 24 18 12 6 0

Command of existing economic knowledge

Use of terms, theories, and data are

Informed judgment demonstrated by

selection of terms, theories and data (shown

by the exclusion irrelevant and inclusion of

relevant issues)Argumentation 40 32 24 16 8 0

Argument flows logically so that early

statements lay the foundation for later

statements and the reader is guided through

the arguments

Appropriate application of theory is used to

make argument; clearly links theory and data

to conclusions

Arguments are persuasive focuses on key

points, does not wonder, uses no

unnecessary verbiage

Alternative policies, arguments, conclusions

and generalizations are noted where they exist

and addressed; differences of opinion,

supported by evidence, are also discussed

Data used is reliable, valid, and pertinent; it

provides effective support; no superficial

information or tangential data muddies the

argument

Striving for Excellence and Creativity 20 16 12 8 4 0

Presentation is neat and professional; all

visuals used are well labeled, clear, and

effective conveying information better than

words; text contains no errors and is easy to

read & understand

Creativity – the paper clearly holds the

imprint of the author. Original thought is

demonstrated by innovative organization, the

integration of concepts and ideas, the use of

new approaches, the novel use of visuals, or

other tangible traits

Thanks to Dr. Richard Stratton of the University of Akron for permission to use this example.



Reflective Questions on Designing and Assessing Courses/Programs to Promote Critical Thinking

Thinking of a course/program you know well . . .

When, how, and why do students need to engage in critical thinking to succeed in that course/program?

Identify and describe 2-3 important assignments, projects, tests, etc. that require critical thinking of your students

What key learning objectives/outcomes are those assignments meant to help students achieve?

How/when do you determine (assess) whether students have the necessary dispositions, knowledge and skills to begin doing that critical thinking?

How/when do you teach and help students develop the necessary dispositions, knowledge and skills if they haven’t already mastered them sufficiently?

How well does your assessment and grading support the importance of the critical thinking work you require?



Critical Thinking: How Some Authors Define It

“Critical thinking, as we define it here, means reviewing the ideas we have produced, making a tentative decision about what action will best solve the problem or what belief about the issue is most reasonable, and then evaluating or refining that solution or belief.”

V.R. Ruggiero (1991). The Art of Thinking: A Guide to Critical and Creative Thought, p. 149.

[Critical thinking is] . . . “an investigation whose purpose is to explore a situation,

phenomenon, question or problem [in order] to arrive at a hypothesis or conclusion that integrates all available information and that can therefore be convincingly justified. In critical thinking, all assumptions are open to question, divergent views are aggressively sought, and the inquiry is not biased in favor of a particular outcome.”

J.G. Kurfiss. (1988). Critical Thinking: Theory, Research, Practice, and Possibilities, p. 2.

“ . . . critical thinking appears to stress the individual’s ability to interpret, evaluate, and make informed judgments about the adequacy of arguments, data , and conclusions.”

E.T. Pascarella & P.T. Terenzini. (1991). How College Affects Students: Findings and Insights from Twenty Years of Research, p. 118.

“ . . . most formal definitions of critical thinking include the intentional application of

rational, higher-order thinking skills such as analysis, synthesis, problem-recognition and problem-solving, inference, and evaluation.”

T.A. Angelo. (1995). “Classroom assessment for critical thinking.” Teaching of Psychology, 22(1), p. 6.

“Critical thinking is not simply being highly critical of everyone else’s thinking but your own.”

Anonymous. (2002).



Fostering Critical Thinking Approaches Well-Supported by Research

Step-by-step Guided Practice

Authentic Problem-solving

Structured Collaboration

Focused Communication

Formative Feedback

Guided Inquiry/Research

Hindering Critical Thinking Approaches Contra-indicated by Research

Focus on rote learning

Information overload

One-shot assignments/assessments

Meaningless assignments

Norm-referenced (curved) marking

Assessment fatigue

In other words, we are more likely to help students develop critical thinking skills if they are actively engaged in –

Learning and practicing explicit processes for reasoning and communicating – whether the scientific method, the writing process model, a particular historical method, the nursing method, or __________

Working in structured small groups to talk about and solve messy, authentic problems – with feedback, guidance, and evaluation from experts

Making their assumptions, beliefs, and ideas explicit to themselves and others through writing and speaking

Evaluating, explaining, supporting, and justifying their ideas – and those of others – through writing and speaking directed at relevant audiences



Seven Common Misperceptions about Assessment and Seven Reasonable Responses

1. We’re doing just fine without it.

OK, then let’s use assessment to find out what works, and to help us document and build on our successes.

2. We’re already doing it.

OK, then let’s audit all the assessments we already do to discover what we know and what we don’t.

3. We’re far too busy to do it.

OK, but since we’re already doing it, let’s use assessment to see where and how we can save time and effort.

4. The most important things can’t/shouldn't be measured.

And not everything measurable should be measured, but let’s see if we can agree on how we can tell when we’re succeeding in these most important things.

5. We’d need more staff and lots more money.

Since we're unlikely to get more resources: how, what, and where can we piggyback, embed, and substitute?

6. They’ll use the results against us.

They might. So, let’s build in strong safeguards against misuse before we agree to assess.

7. No one will care about or use what we find.

To avoid that, let's agree not to do any assessments without a firm commitment from stakeholders to use the results.



Applications Card DIRECTIONS: Please take a moment to recall the ideas, techniques, and strategies we've discussed – and those you've thought up – to this point in the keynote. Quickly list as many possible applications as you can. Don't censor yourself! These are merely possibilities. You can always evaluate the desirability and/or feasibility of these possible applications later.

Interesting Some possible IDEAS/TECHNIQUES APPLICATIONS of those from this session ideas/techniques to my work

Reference: Angelo, T.A. & Cross, K.P. (1993). Classroom Assessment Techniques: A Handbook for College Teachers, 2nd edition. San Francisco: Jossey-Bass.



A FEW USEFUL REFERENCES ON ASSESSMENT AND FEEDBACK

Hattie, J. & Temperley, H. (2007). The power of feedback. Review of Higher Education, 77(1), 81-112.

Huba, M.E. & Freed, J.E. (2000). Learner-Centered Assessment on College Campuses: Shifting the Focus from Teaching to Learning. Needham Heights, MA: Allyn & Bacon.

Light, R. J. (2001). Making the Most of College: Students Speak Their Minds. Cambridge, MA: Harvard.

Mentkowski, M. & Associates. (2000). Learning that Lasts: Integrating Learning, Development, and Performance in College and Beyond. San Francisco: Jossey-Bass.

Nichol, D.J. & Macfarlane-Dick, D. (2007). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218.

Price, M., et al. (2012). Assessment Literacy: The Foundation for Improving Student Learning. Oxford: Assessment Standards Knowledge Exchange (ASKe)/Oxford Brookes University.

Suskie, L. (2009). Assessing Student Learning: A Common Sense Guide, 2nd ed. San Francisco: Jossey-Bass.

Walvoord, B.E. (2004). Assessment Clear and Simple: A Practical Guide for Institutions, Departments, and General Education. San Francisco: Jossey-Bass.

Wiggins, G. (1998). Educative Assessment: Designing Assessments to Inform and Improve Student Performance. San Francisco: Jossey-Bass.

. . . and on Improving Teaching and Learning Quality

Ambrose, S.A., et al. (2010). How Learning Works: Seven Research-based Principles for Smart Teaching. San Francisco: Jossey-Bass.

Anderson, L. & Krathwohl, D.R. (Eds.) (2001). A Taxonomy for Learning, Teaching, and Assessment: A Revision of Bloom’s Taxonomy of Educational Objectives (Abridged Ed.). New York: Allyn & Bacon.

Angelo, T. (2012). Designing subjects for learning: Practical, research-based principles and guidelines. In Hunt, L. & Chalmers, D. University Teaching in Focus: A Learning-centred Approach. London: Routledge, Taylor & Francis, 93-111.

Biggs, J. & Tang, C. (2011). Teaching for Quality Learning at University: What the Student Does. 4th ed. Berkshire, England: McGraw-Hill/Open University Press.

Bransford, J.D., Brown, A.L, & Cocking, R.R. (Eds.). (2000). How People Learn: Brain, Mind, Experience, and School, Expanded ed. Washington, DC: National Academy Press.

Davis, B.G. (2009). Tools for Teaching, 2nd ed. San Francisco, CA: Jossey-Bass.

Hattie, J. A. C. (2009). Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. New York: Routledge.

Kuh, G.D. et al. (2006). What Matters to Student Success: A Review of the Literature. Washington, DC: National Postsecondary Education Cooperative, U.S. Dept. of Education Institute of Education Sciences.

Pascarella, E. & Terenzini, P. (2005). How College Affects Students: A Third Decade of Research. San Francisco, CA: Jossey-Bass.

Ramsden, P. (2003). Learning to Teach in Higher Education, 2nd ed. London: RoutledgeFalmer.

Svinicki, M.D. (2004). Learning and Motivation in the Postsecondary Classroom. Bolton, MA: Anker.

Svinicki, M. & McKeachie, W.J., et al. (2011). Teaching tips: Strategies, research, and theory for college and university teachers, 13th edition. Wadsworth/Cengage Learning: Belmont, CA, USA.

LIGNING LEARNING OUTCOMES AND GRADING STANDARDS...

Documents

Transcript of LIGNING LEARNING OUTCOMES AND GRADING STANDARDS...