Item: Item: A question used to test a piece of A question used to test a piece of knowledge. knowledge. Instruction: Instruction: An explanation of how to complete a task. It contains a meaningful It contains a meaningful situation and the task.situation and the task. Stem: Stem: The beginning of a multiple choice test The beginning of a multiple choice test item. In multiple choice items, it leads into four item. In multiple choice items, it leads into four response options, made of three distracters and response options, made of three distracters and one key.one key. Cue: Cue: The item’s response options.The item’s response options. Key: Key: The response option that is correct.The response option that is correct. Distracter: Distracter: The response option that is not The response option that is not the correct answer the correct answer

Normal item: Normal item: The format describing an item that includes one simple positively stated stem that directly leads to the response options. It is the recommended format.

Not item: Not item: The format describing an item with a negatively stated stem. It typically asks for what is “not,” least” or “worst” or use words like “except” or “but.” It should be avoided particularly when participants are taking the test in their native language.

Poor distracter: Poor distracter: A response option selected by less than 10 percent of the respondents to the items. Under-selected distracters indicate either that the distracters were ineffective or that the content areas behind the distracters were well understood by the respondents so that these distracters were not “distracting” at all.

Before you start designing your items, answer the Before you start designing your items, answer the following 2 questions:following 2 questions:

What am I going to assess? (aspects of language) How am I going to assess it? (test formats)

Write your items considering the following 4 aspects:Write your items considering the following 4 aspects:

Task (aspect assessed) Context (meaningful) Instructions (clear and concise) Stem (question or statement) Options (cues and key)

Depending on the nature of the content being Depending on the nature of the content being assessed, choose the most appropriate test formatassessed, choose the most appropriate test format

READING AND LISTENINGREADING AND LISTENING

Gap-filling (open/with options)Multiple choiceMatchingTrue/false (don’t know/not mentioned)Open-ended questionsClosed questionsTable completionLinkingDiscrepanciesListing/Sequencing

WRITING AND SPEAKINGWRITING AND SPEAKING

Essay questionsGuided writingPunctuationSummaryNote takingDictationFree interviewPicture descriptionInformation transferRole-playOral Presentation

LANGUAGE COMPONENTSLANGUAGE COMPONENTS

Cloze exercise (blank every th word)Gap-fillingSentence arrangementSentence writingTable completionMatching sentence halvesMatching (words/definitions)Multiple choiceOpen-ended questionsClosed questionsDialogue completion

Problem: Problem: For every item, all cues must be grammatically and/or semantically consistent and homogeneous.

Wrong: Wrong: John _________ flowers to the party last night. a)Carriesb)Carriedc)Liftsd)Lifted

Better: Better: John _________ flowers to the party last night. a)Carriesb)Carriedc)Has carriedd)Is carrying

Problem: Problem: The stem should not introduce the correct option.

Wrong: Wrong: He wrote anan _________ in his laptop. a)Letterb)Post cardc)E-maild)Memo

Better: Better: He wrote a brief _________ in his laptop. a)Letterb)Post cardc)E-maild)Memo

Problem: Problem: In communicative tests, instructions should not be based on grammar, but on meaningful contexts.

Wrong: Wrong: Write sentences using make, let, make, let, and be be allowed.allowed.

Better:Better: You got admitted in a university away from for hometown so you have to move to a student residence. Write sentences expressing the things that are prohibitedprohibited or permitted permitted in this context.

Problem: Problem: The cues in a multiple-choice format should be all grammatically possible.

Wrong: Wrong: Katherine is a very _________ student. She gets nothing but straight A’s. a)a)SmartlySmartlyb)Intelligentc)Preoccupiedd)Considerate

Better: Better: Katherine is a very _________ student. She gets nothing but straight A’s. a)Thoughtfulb)Intelligentc)Preoccupiedd)Considerate

Problem: Problem: It is never suggested to use made-up words or grammatical structures as distracters.

Wrong: Wrong: People say that my children are malraised/spoiled malraised/spoiled because I buy them whatever they ask for.

Better: Better: People say that my children are consented/spoiled consented/spoiled because I buy them whatever they ask for.

Problem: Problem: Language tests assess overall language proficiency, not cognitive skills or creativity.

Wrong: Wrong: Unscramble the jumbled letters below to form English words.a)MEAXb)GMAEUZc)TACd)ALAVGI

Better: Better: Unscramble the jumbled words below to form sentences expressing frequency.a)gym often go I the to _____________________________b)she ever television hardly watches___________________c)my mom home never at is _________________________d)cooks Franklin dinner always_______________________

Problem: Problem: The difficulty in an item should lie in the task itself, not in the instructions or the stem.

Wrong: Wrong: When a police officer arrests someone, the officer must inform the person of a certain rights, including the right to remain silent and the right to an attorney. Why, according to the text, is this required?

Better: Better: According to the text, why must police officers inform suspects of their rights during an arrest?

Problem: Problem: For every item, there should always be one (and only one) correct response.

Wrong: Wrong: The person in the listening said he had left his land because he… a)Wanted to seek his fortuneb)Wished to avoid his creditorsc)Preferred the new landd)All of the abovee)None of the abovef)A and B, but not Cg)B and C, but not A

Better: Better: The person in the listening said he had left his land because he… a)Wanted to seek his fortuneb)Wished to avoid his creditorsc)Preferred the new landd)Had his family abroad already

Problem: Problem: All cues should be about the same length. Another option is having two short cues and two long ones.

Wrong: Wrong: Based on the recording, what is the best way to reduce inflation?a)Cutting on income taxes.b)Increasing government expenses.c)Reducing the number of bank operating all over America.d)Raising local interest rates.

Better: Better: Based on the recording, what is the best way to reduce inflation?a)Cutting on income taxes.b)Increasing government expenses.c)Reducing the federal deficit. d)Raising local interest rates.

Problem: Problem: The correct answer should be based on the content of a passage and not on previous knowledge or common sense.

Wrong: Wrong: Read the text and say if these sentences are TRUE or FALSE.a) The three basic principles to be environmentally friendly are reduce, reuse reduce, reuse and recyclerecycle. ______ 

Better: Better: Read the text and answer the following questions.a) What are the three basic principles to be environmentally friendly? _________________________________________

Once the item is designed, it is important to Once the item is designed, it is important to determine its appropriateness in terms of…determine its appropriateness in terms of…

VALIDITY: VALIDITY: Items test what we want to test.

RELIABILITY: RELIABILITY: Items’ results are consistent and replicable.

PRACTICALITY: PRACTICALITY: Items are easy to design, complete and correct.

BACKWASH: BACKWASH: Items are meaningful and motivating.

An item is VALID if it assesses authentic An item is VALID if it assesses authentic communication and interaction, and also reflects communication and interaction, and also reflects what students do in class. It has POSITIVE what students do in class. It has POSITIVE BACKWASH if students feel they have been tested BACKWASH if students feel they have been tested fairly and become aware of the importance of fairly and become aware of the importance of language proficiency.language proficiency.

Wrong: Wrong:  Dictation. Listen to the instructions to start a weblog and write the information on the space provided. Then translate the information into your native language.

Better: Better: You want to create a weblog and a friend of yours is giving you the information to set up an account. Listen Listen to the instructions and fill in the missing information to to the instructions and fill in the missing information to learn how to start your own blog. learn how to start your own blog.

An item is RELIABLE and if it measures students’ An item is RELIABLE and if it measures students’ performance consistently. It is PRACTICAL if it takes performance consistently. It is PRACTICAL if it takes little time to design, organize and mark, and involves little time to design, organize and mark, and involves resources that are easily available. resources that are easily available.

Wrong: Wrong: In your opinion, how can computers help us?_______________________________________________________________________________________________________________________________________

Better: Better: Mention three ways in which computers can help firefighters in case of an emergency?1)____________________________________________2)____________________________________________3)____________________________________________

Integrative and open-ended test formats are Integrative and open-ended test formats are usually very VALID and also have a POSITIVE usually very VALID and also have a POSITIVE BACKWASH, since they involve communication BACKWASH, since they involve communication and interaction in real-like situations. and interaction in real-like situations.

Example: Example: While reading today’s newspaper, you find two interesting articles. Express your Express your opinion about them. opinion about them.

However, integrative and open-ended test However, integrative and open-ended test formats are often formats are often impracticalimpractical (difficult to (difficult to interpret) and interpret) and unreliableunreliable (open and subjective). (open and subjective).

On the other hand, discrete item test formats are On the other hand, discrete item test formats are usually very RELIABLE and PRACTICAL, usually very RELIABLE and PRACTICAL, BECAUSE they provide enough information, BECAUSE they provide enough information, correct answers are limited and they are quick correct answers are limited and they are quick and easy to mark. and easy to mark.

Example: Example: It is said that pit-bulls are a breed/racebreed/race of dog that is particularly violent.

Nevertheless, discrete item formats frequently Nevertheless, discrete item formats frequently lack lack validityvalidity (meaningful communication) and (meaningful communication) and positive backwash positive backwash (luck is as important as hard (luck is as important as hard work). work).