Katharina StöttingerISBE Berlin 2013 Measuring Vocational Competence Written Performance...

Katharina Stöttinger ISBE Berlin 2013

Measuring Vocational Competence

Written Performance Assessments beyond Multiple Choice

Katharina Stöttinger

Department for Business Education and DevelopmentKarl-Franzens-Universität Graz

Berlin, August 2013


How Students Feel About Tests

* Probably not an actual

Einstein quote


Objectives for this Session

• Gain insight into the challenges involved in developing an instrument to measure occupational skills

• Get inspiration for designing competence-based learning opportunities for students of commercial subjects/students undergoing vocational training

• Gain insights into ensuring a satisfactory level of consistency when it comes to the rating of open-ended test items.


Outline

• General requirements for competence based assessments

• Construction of suitable items• Construction of rating guidelines• Implication for teachers


General Reqirements for Competence Based Assessments

Items should • represent authentic challenges grounded in real-world working

situations• be open-ended• activate complex cognitive functions, requiring students to

– reproduce factual knowledge– contextualize it– apply it to a specific situation – and potentially transfer it to new problems with a similar structure.

• trigger responses that can be rated with a satisfactory level of consistency among various experts.


Case Study: A Comparison of Offers

You work for FunInTheSun, a manufacturer of sporting equipment who sponsors the regional beach-volleyball tourney.

Your boss Mrs. Kofler wants to use this occasion to distribute goody-bags with promotional freebies to V.I.P guests. Each goody-bag should contain a memory stick with the name of your company printed on it.

Now Mrs. Kofler has asked you to compare offers from four manufacturers of promotional freebies, choose the best offer and write her a memo explaining your choice.


Case Study – A Comparison of Offers

Consider the following context:

• Today is July, 18th. The beach volleyball tourney will take place on July, 27th. Mrs. Kofler has asked you to hurry with your decision, since she doesn’t want to assemble the goody-bags at the last minute.

• The guest list for the event contains 46 V.I.P guests.

• Your budget for the memory sticks amounts to EUR 300,-.


Case Study – A Comparison of Offers

Price per unit

Production Time

Minimum order quantity

Bulk Discount

Cash Discount

Expiration Date


Case Study - Potential Solutions

Cheapest option is also supplier with longest production time

• Prioritize cost minimization and trust that there won’t be any delays • Choose the second cheapest supplier, who is only marginally

more expensive than the cheapest option and could deliver a lot faster

• Contact a third supplier, whose offer has already expired but who would be cheaper and faster than both options and ask whether the offer could be extended

Answer should ideally demonstrate an awareness of trade-offs


Case Study - Competences

Competence Revealed through

Reading Comprehension

& Diligence

•picking out the relevant details from a mass of materials•noticing expiration dates and mininum order requirements•noticing differences in what is covered in the price per unit (VAT; costs of imprinting the company‘s logo)

Pertinent factual knowledge

•Applicaton of technical terms and business conventions•knowing the order of discounts•knowing the meaning of VAT


Case Study - Competences

Competence Revealed through

Mathematical Skills

•calculating total costs•accounting for tax and discounts

Rhetorical Skills •Verbalizing a thought process•Presenting a convincing argument

Critical Thinking Skills

•awareness of trade-offs•considering the wider context


Case Study – Common Misapprehensions

• Students only compare the price per item

• Minimum order requirements as a knock-out criterion

• Failure to take advantage of discounts

• General lack of insight into the interplay of quantity, price per item, discounts and percentages


Case Study – Learning Objectives

• Make sure that things are compareable before you compare them. (Some prices contain VAT and costs for personalizations, some don‘t).

• Price per unit is not the only factor that determines total cost.

• Order quantity required is not the only factor that determines total cost.

• Take advantage of discounts



• Know the conventions of business (eg. Bulk discounts are subtracted before cash discounts)

• Be aware that certain conditions might be subject to negotiation (eg. Sometimes expired offers might be extended on request.)

• Cost is not the only criterion – time and quality matter as well.

• Be aware of trade-offs.



Transferable skills:

Before making any decision• Make sure you know all the factors that

influence the potential outcome.

• Make sure you know how those factors influence each other (eg. the relationship between quantity and discounts)

• Never look at a single factor in isolation.



Transferable skills

When trying to convince others of your choice• Make your choice as transparent as possible• List all the factors that have influenced your

choice• Point out any trade-offs involved• Explain your priorities with reference to the

situation at hand.

(=> Respect the complexity of a problem. Demonstrate awareness of multicausality, interdependencies, trade-offs and wider context.)


General Reqirements for Competence Based Assessments

Items should • represent authentic challenges grounded

in real-world working situations

• be open-ended

• activate complex cognitive functions

• trigger responses that can be rated with a satisfactory level of consistency among various experts.


Case-Study – Rating Guideline

Rating Guideline based on an analysis of • competences involved in a useful approach to the

problem• common misconceptions and mistakes inherent in

less competent approaches• learning objectives derived from these differences.


Case Study – Rating Guideline

Item The Student Rating

Calculation Checks whether any given offer is covered by the budget

0 …no

1 … yes

Takes VAT into account 0 … no

1 … yes

Takes minimum order

requirement into account

0 … no

1 … yes

Takes advantage of bulk discount 0 … no

1 …yes, with errors

2 .. .yes, no errors

Takes advantage of cash discount 0 … no

1 …yes, with errors

2 .. .yes, no errors


Case Study – Rating Guideline

Item The Student Rating

Choice Makes a choice and provides an explanation

0 …no

1 … yes, justification is questionable

2 … yes, justifaction is consistent

Considers expiration date of offers

0 … no

1 … yes

Considers costs 0 … no

1 … yes

Considers production time 0 … no

1 …yes

Considers possibility of re-negotiation

0 … no

1 … yes


Personal Mission Statement as a Teacher of Commercial Subjects

• I help people learn how to rationalize their choices.• I help people learn how to appreciate complexity.

Guiding Principles:• There is usually more than one valid approach to anything. • Still, some approaches might make more sense than others. • The best approach is the one that makes the most sense to all the people

involved in its implementation.• In order to help others make sense of your choices, make them transparent.• The goal is not objectivity, because objectivity is an illusion as soon as trade-

offs are involved. • The goal is inter-subjectivity.• There is always more than one factor to consider.• Context matters.


Rating Guidelines as a Tool for Teaching

Rating Guidelines• Require teachers to thouroughly reflect upon the

challenges inherent in any given learning opportunity and students‘ potential missapprehensions in those regards

• Make assessments more transparent and promote a perception of fairness

• Provide a good framework for face-to-face feedback for students, promoting students‘ self reflection

• Facilitate team-teaching

Katharina StöttingerISBE Berlin 2013 Measuring Vocational Competence Written Performance...

Documents

Transcript of Katharina StöttingerISBE Berlin 2013 Measuring Vocational Competence Written Performance...