Outcome Based Assessment Methods

Outcome Based Assessment Methods

Adopted from ABET IDEAL (Institute for the Development of Excellence

in Assessment Leadership) Workshop material (August 2009)

ByWajih Abu-Al-Saud

8 December 2009

Dr. Wajih Abu-Al-Saud 2

Outline

• Classification of Assessment Methods• Assessment Methods

– Written Surveys and Questionnaires– Exit and other Interview– Commercial Norm-Referenced Standardized Examinations– Locally Developed Exams– Focus Groups– Portfolios– Simulations– Performance Appraisals– External Examiners– Archival Records– Oral Examinations– Behavioral Observations

8 December 2009


Classification of Assessment Methods

8 December 2009

Direct IndirectStandardized Exams Written Surveys and

QuestionnairesLocally Developed Exams

Portfolios Exit and Other Interviews

Simulations

Performance Appraisals Archival Records

External Examiners

Oral Exams Focus Groups

Behavioral Observations


1. Written Surveys and Questionnaires

DefinitionAsking individuals to share their perception about the target being studied (such as their own skills or knowledge, or attributes or qualities of a program or course) by filling survey/questionnaire forms

8 December 2009



Advantages• Provide accessibility to individuals who otherwise would

be difficult to include in assessment efforts (alumni, employers)

• Can cover a broad range of attributes within a brief period of time

• Results are usually very clear even to a lay person• Typically produce students’ and alumni’s perspective of

the institution (which lead to changes beneficial to relationship with these groups)

8 December 2009



Advantages (when third-parties [parents, employers] are completing surveys)

• Enable recognition and contact with important but often under-valued constituents (relations may improve by just asking for their input)

• Offer different “less-biased” opinion then students• Can increase validity of results (through triangulation) • Provide unique and valuable stakeholder input (how is a

course/program/college serving their purposes)

8 December 2009



Disadvantages• Results tend to be highly dependent on wording and

organization of items (good surveys are difficult to create)

• May be biased because they often rely on volunteer participants

• Mail or Email surveys tend to have low response rates• Commercially prepared surveys may not be entirely

relevant to individual institutions• Multiple-choice surveys may not provide opportunities

for respondents to express their true opinions

8 December 2009



Disadvantages (when third-parties [parents, employers] are completing surveys)

• Very difficult to anticipate and detect bias of third-parties• Expensive because of logistics (identifying sample,

making contact, getting useful responses) • Very low response rates because of low investment by

third parties in assessment process• If information about specific individuals is requested,

confidentiality becomes an issue

8 December 2009



Ways to Reduce Disadvantages• Use only carefully construct surveys that have been reviewed by

survey experts• Include open-ended items along with multiple choice items• Use pilot samples and request feedback from respondents on clarity,

sensitivity, and format of items• Obtain response from the larges sample size possible and follow-up

with non-respondents (in person or by phone)• If using commercially prepared surveys, add locally developed items

of relevance to institution• If using locally prepared surveys, add externally-referenced items

(e.g. from surveys for which national data is available)• Attain consent in cases where information about specific individuals

is being requested• Coordinate contacts with other organizations contacting the same

group to increase response rates 8 December 2009


2. Exit (and other) Interviews

DefinitionAsking individuals to share their perception through face-to-face dialogue about the target being studied (such as their own skills or knowledge, or qualities and attributes of a program or course)

8 December 2009



Advantages (similar to written surveys but require direct contact limiting accessibility to certain populations)

• Sometimes yield benefits other than data collection stemming from interaction with students

• Allow more individualized questions and follow-ups questions based on responses of interviewees

• Provide immediate feedback to interviewer• Can include a greater variety of items than surveys

8 December 2009



Advantages (when third-parties [parents, employers] are being interviewed)

• Provide unique and valuable stakeholder input (how is course/program/college serving their purposes)

• Offer different “less-biased” opinion then students• Enable recognition and contact with important but often

under-valued constituents (relations may improve by just asking for their input)

• Can increase validity of results (through triangulation)

8 December 2009



Disadvantages• Require direct contact, which may be difficult to arrange• May be intimidating to interviewees, thus biasing results

in the positive direction• Results tend to be highly dependent on wording of items

and the manner in which interviews are conducted• Time consuming, especially if large number of persons

are to be interviewed

8 December 2009



Disadvantages (when third-parties [parents, employers] are being interviewed)

• Very difficult to anticipate and detect bias of third-parties• Expensive because of logistics (identifying sample,

making contact, getting useful responses)• If information about specific individuals is requested,

confidentiality becomes an issue

8 December 2009



Ways to Reduce Disadvantages• Plan interviews carefully with assistance from experts• Provide training for interviewers that include guidance in

putting interviewees at ease and related interview skills• Coordinate contacts with other organizations contacting

same group to increase rate of acceptance for interviews• Conduct pilot testing of interview and request feedback

from interviewee to improve interview process• Interview purposeful samples of students when it is not

possible to interview all• Conduct phone interviews when face-to-face interviews are

not possible• Utilize focus groups when individual interviewing is not

possible or is too costly8 December 2009


3. Commercial Norm-Referenced Standardized Exams

DefinitionGroup administered exams, mostly or entirely multiple-choice, “Objective” tests in one or more areas. Scores are based on comparison with a reference or norm group. Typically must be purchased from a private vendor. Used primarily on students in individual programs or courses.

8 December 2009



Advantages• Can be adopted and implemented quickly• Objective scoring• Reduce/eliminate faculty time demands in development

and grading (relatively low frontload and backload)• Provide for externality validity of measurements (degree

to which the conclusions in your study would hold for other persons in other places and at other times)

• Very beneficial for benchmarking and cross-institutional comparison studies

8 December 2009



Disadvantages• Eliminate process of learning and clarification of goals and

objective associated with locally developed tools• Unlikely to completely measure or assess specific goals and

objectives of a program, department, or institution• May limit what can be measured• Norm-referenced data depends on comparison group and method

of selecting students to be tested (most norm-referenced tests in higher education do not utilize randomly selected sample)

• Multiple-choice tests generally include a high degree of error resulting from guessing correct answers

• Results unlikely have direct role on program improvement• Either students or program have to pay for exam• If used repeatedly, faculty may teach to the exam

8 December 2009



Ways to Reduce Disadvantages• Choose test carefully, and only after reviewing all

test components and finding a satisfactory degree of match with the learning outcomes of the program

• Request and review technical data, especially reliability and validity data and information on normative sample from test publishers

• Insure that such tests are only one aspect of multi-method approach in which no firm conclusions based on norm-referenced data are reached without validation from other sources (triangulation)

8 December 2009


4. Locally Developed Exams

DefinitionObjective and/or subjective tests designed by faculty of the program or course sequence being evaluated

8 December 2009



Advantages• Content and style can be geared to specific goals and

objectives• Local grading by faculty can provide relatively rapid

feedback• More direct implication of results for program

improvements • Greater faculty/institutional control over interoperation

and use of results • Process of development can lead to

clarification/crystallization of what is important in the process of student learning

8 December 2009



Disadvantages• Require considerable leadership/coordination, especially

during the various phases of development• Costly in terms of time and effort (more “frontloaded”

effort for objective exams; more “backloaded” effort for subjective exams)

• Cannot be used for benchmarking, or cross-institutional comparisons

• Demand expertise in measurement to assure validity, reliability, and utility

8 December 2009



Ways to Reduce Disadvantages• Join with other programs, departments, or institutions

with similar goals and objectives as means of reducing costs of developing exams and adding externality at the same time

• Utilize on-campus measurement experts whenever possible for test construction and validation

• Incorporate outside experts and community leaders into development and grading process

• Embed in program requirements for maximum relevance with minimum disruption (insert into capstone course),

• Validate results through use of multi-method approach (triangulation)

8 December 2009


5. Focus Groups

DefinitionTypically conducted with 6-12 individuals who share certain characteristics that are related to a particular topic linked to a research or evaluation question. Group discussions are conducted by a trained moderator with participants having the task of identifying trends/patterns. Moderator’s purpose is to provide direction and set the tone for the group discussion, encourage active participation from all group members, and manage time. Moderator must not allow own biases to influence discussion.

8 December 2009


5. Focus Groups

Advantages• Useful to gather ideas, details, new insights and to

improve question designs of tests• Helpful in the design of surveys and interpreting results

of and getting more in-depth information on issues identified by a survey

• Interaction among focus group participants often leads to new insights

• Allows the moderator to probe and explore unanticipated issues

8 December 2009


5. Focus Groups

Disadvantages• Groups can be difficult to assemble• Not suited for generalization about population being

studied• Conducting effective focus groups requires training of

moderators• Moderator has less control than in individual interviews• Differences in the responses between/among groups can

be problematic• Data provided by focus groups can itself be difficult to

analyze

8 December 2009


5. Focus Groups

Ways to Reduce Disadvantages• Offer an incentive for participants if possible to

increase participation• Train moderators to use open-ended questions,

probes, and learn when and how to move into new topic areas

8 December 2009


6. Portfolios

DefinitionCollection of multiple student work samples usually compiled over time and rated using rubrics. The design of a portfolio is dependent upon how the scoring results are going to be used

8 December 2009


6. Portfolios

Advantages• Can be used to view learning and development

longitudinally (samples of student work can be collected over time), which adds a very useful perspective

• Multiple components of a curriculum can be measured at the same time

• Economical in terms of student time and effort, since no separate “assessment administration” time is needed

• Samples in portfolios generally reflect student ability more than test results when pre-planning

• Avoids or minimizes “test anxiety” and other “one shot” measurement problems

• Increases student participation in the assessment process

8 December 2009


6. Portfolios

Disadvantages• Can be costly in terms of evaluator time and effort

(management of the collection, and scoring process, including establishment of reliable and valid scoring rubric may be challenging)

• If samples to be included have been previously submitted for course grades, faculty may be concerned that a hidden agenda of the process is to validate their grading

• May not provide for externality• Security concerns may arise as to whether submitted

samples are the students’ own work

8 December 2009


6. Portfolios

Ways to Reduce Disadvantages• Consider having portfolios submitted as part of the

requirements of a specific course requirements• Have more than one rater for each portfolio; establish

inter-rater reliability through piloting designed to fine-tune rating criteria

• Investigate the use of electronic portfolios as means to increase process efficiency

• Utilize portfolios from representative samples of students rather than having all students participate

• Provide training for raters• If portfolios are selected by students, understand that this

is probably their best work

8 December 2009


7. Simulations

DefinitionA competency-based measure where a person’s abilities are measured in a situation that approximates a “real world” setting. Simulation is primarily used when it is impractical to observe a person performing a task in a real world situation (e.g. on the job)

8 December 2009


7. Simulations

Advantages• Better means of evaluating depth and breadth of student

skill development than test or other performance-based measures

• More flexible; some degree of simulation can be arranged for virtually any student target skill

• For many skills, can be group administered, thus providing an excellent combination of quality and economy

8 December 2009


7. Simulations

Disadvantages• For difficult skills, the higher the quality of simulation

the more subjective the rating of student will be compared to standardized tests

• Usually require considerable frontloading effort for planning and preparation

• More expensive than traditional testing options in the short run

8 December 2009


7. Simulations

Ways to Reduce Disadvantages• Limit complexity degree of simulation situations to those

that can be rated more objectively• Administer group simulations as much as possible to

reduce cost and increase number of skills being rated

8 December 2009


8. Performance Appraisals

DefinitionA competency-based method whereby abilities are measured in most direct, real-world approach

8 December 2009



Advantages• Provide a more direct measure of what has been learned

(presumably in the program) • Go beyond paper-and-pencil test and most other

assessment methods in measuring skills• Preferable to most other methods in measuring the

application of learning to specific setting and situations• Particularly relevant to the goals and objectives of

professional training programs and disciplines with well defined skill development

8 December 2009



Disadvantages• Requires considerable time and effort (especially

frontloading effort), thus being costly• Samples of behavior observed or performance appraised

may not be typical, especially because of the presence of observer

• Rating of student performance is typically more subjective than standardized tests

8 December 2009



Ways to Reduce Disadvantages• Conduct observations/appraisals in the least intrusive

manner possible (e.g. using one-way observational mirrors, videotaping)

• Conduct pilot-testing in which rate of agreement (inter-rater reliability) between observers/appraisers is determined

• Provide training for observers/appraisers• Develop measurable criteria for observing and appraising

performance• Cross-validate results with other measures

8 December 2009


9. External Examiners

DefinitionUsing an expert in the field from outside your program, usually from a similar program at another institution to conduct, evaluate, or supplement assessment of your students. Information can be obtained from external evaluators using many methods including surveys, interviews, … etc.

8 December 2009



Advantages• Increases impartiality by having third party objectivity• Outsiders can see attributes to which insiders have grown

accustomed• Students may disclose to an outsider what they might not

otherwise share• Feedback useful for both student and program evaluation

and provides valuable program consultation• May serve to stimulate other collaborative efforts

between department/institutions• Evaluators may have skills, knowledge, or resources not

otherwise available

8 December 2009



Disadvantages• Risk of misfit between examiner’s expertise and

expectations and program outcomes• Can be very costly and time consuming for individualized

evaluations in large programs

8 December 2009



Ways to Reduce Disadvantages• Share program philosophy and objectives with the

external examiner and agree on assessment criteria before the assessment

• Form reciprocal external examiner “consortium” among similar programs to minimize costs, swapping external evaluations back and forth

8 December 2009


10. Archival Records

DefinitionBiographical, academic, or other filed data available from the college or other agencies and institutions

8 December 2009



Advantages• Tend to be accessible, thus requiring minimal effort• Constitute non-intrusive measurement not requiring

additional time or effort from students or other groups• Can be cost efficient if required data is readily retrievable

in desired format• Build upon data collection efforts that have already

occurred• Very useful for longitudinal studies• Good way to establish a baseline for before and after

comparisons

8 December 2009



Disadvantages• Especially in large institutions, may require considerable

effort and coordination to determine exactly what data are available campus-wide and to then get that information in desired format (may require a lot of data mining)

• To be most helpful, data from different sources may need to be combined. This may require the ability of designing good databases

• Availability of data may discourage the development of other more appropriate measure or data source

• May encourage attempts to find ways to use data rather than assessment related to specific goals and objectives

8 December 2009



Ways to Reduce Disadvantages• Early-on in the development of an assessment program,

conduct a comprehensive review of data typically being collected through the institution and its units

• Only use archival records that are relevant to specific goals and objectives of learning and development to reduce demand on database design

8 December 2009


11. Oral Examinations

DefinitionAn assessment of student knowledge levels through a face-to-face dialogue between the student and an examiner (usually a faculty)

8 December 2009



Advantages• Content and style can be geared to specific goals, objectives, and

student characteristics of the program• Provide process evaluation of student thinking and speaking skills

along with knowledge content• Dialogue format decreases miscommunications and

misunderstandings, in both questions and answers• Allow measurement of student achievement in considerably greater

depth and breadth through follow-up questions, probes, encouragement of detailed clarifications

• Greater faculty/institutional control over interpretation and use of results

• More direct implication of results for program improvements• Non-verbal reactions of students aid in interpretation of student

responses• Local grading by faculty can provide immediate feedback8 December 2009



Disadvantages• Requires considerable leadership/coordination, especially during

the various phases of development• Can be difficult to document by note-taking and providing student

feedback with a grade• Costly in terms of time and effort (more “frontloaded” effort for

objective exams; more “backloaded” effort for subjective exams)• May not provide for externality (objectivity associated with

comparisons external to the program or institution)• Requires considerably more faculty time since oral exams must

be conducted one-to-one or with a small group of students at most• Can be inhibiting on student responsiveness due to intimidation,

face-to-face pressures, Oral versus written mode. May have similar effect on faculty

8 December 2009



Ways to Reduce Disadvantages• Prearrange “standard” questions, most common follow-

up probes, and how to deal with typical students problem responses using “Pilot” testing and training

• Electronically record oral exams for more detailed evaluation later

• Take time to establish open, non-threatening atmosphere for testing

8 December 2009


12. Behavioral Observations

DefinitionMeasuring the frequency, duration, or any other feature of student actions, usually in a natural setting with non-interactive methods. For example, formal or informal observations of a classroom. Observations are mostly made by an individual and can be augmented by audio or video tapes

8 December 2009



Advantages• Best way to evaluate degree to which attitudes, values,

etc. are really put into action• Least intrusive assessment option, since purpose is to

avoid any interference with typical student activities• Catching students being themselves is the most “natural”

form of assessment

8 December 2009



Disadvantages• Always some risk of erroneous results due to “observer

effect” (i.e. subjects may behave atypically if they know they are being observed)

• Depending on the purpose of the data gathering, there may be socially, professionally, or legal consideration to be dealt with (e.g. invasion of privacy)

• May encourage “Big Brother” perception by students/faculty being observed of the assessment process or institution

8 December 2009



Ways to Reduce Disadvantages• Avoid using this method when studying socially or

ethically sensitive issues (e.g. do students adhere to a code of ethics)

• Utilize electronic “observers” (i.e. audio and video recorders) whenever possible, for highly accurate, reliable, permanent observation record (although this may increase assessment cost)

• Strictly adhere to ethical guidelines for the protection of subjects being observed

8 December 2009


Conclusion

In this presentation, 12 assessment methods along with their advantages, disadvantages and methods for reducing the disadvantages were discussed

8 December 2009

Outcome Based Assessment Methods

Documents

Transcript of Outcome Based Assessment Methods