Testing and Assessment in Vocational Education ontents 1 Purposes of the Background Paper2 Evolution...

Testing and Assessment in VocationalEducation

March 1994

OTA-BP-SET-123NTIS order #PB94-156262

GPO stock #052-003-01367-3

Recommended Citation:U.S. Congress, Office of Technology Assessment, Testing and Assessment inVocational Education, OTA-BP-SET-123 (Washington, DC: U.S. GovernmentPrinting Office, March 1994).

For sale by the U.S. Government Printing OfficeSuperintendent of Documents, Mail Stop: SSOP, Washington, DC 20402-9.328

ISBN 0-16 -043127-1

Foreword

T he world of work is changing. As this occurs, educators must rethinkhow to prepare young people for employment. For young people whodo not plan to obtain a college degree, it is no longer clear what typeof employment will provide satisfying work that can lead to financial

independence. And it is perhaps even less clear how students can be preparedfor a work environment characterized by change.

As part of its growing concern for noncollege bound youth, Congress hasbegun to revisit and revise legislation that supports vocational education. In the1992 revision of the Perkins Act, the House Committee on Education and Laborasked that OTA provide information on the types of tests and assessmentinstruments used to define the outcomes and effectiveness of these programs.By looking at these instruments, Congress can obtain insight. What we test iswhat we teach.

Accordingly, OTA has compiled this background paper on tests andassessments for secondary vocational education. It includes a state-by-statesurvey of assessment instruments, as well as information on new instrumentsunder development by test vendors. Additionally, OTA has reviewed theemerging theories attempting to define ‘‘broad technical skills. ’

The survey data suggest some shifting away from the traditional,performance-based measurements of vocational education. This is occurring ata time when performance-based instruments are being introduced in otherschool subjects, in order to give a more useful and accurate indication of studentachievement. OTA also found that there is no consensus on the meaning of“broad technical skills,’ and that a number of approaches are being used to tryand prepare vocational education students for a changing workplace.

This background paper should help to inform the debate on vocationaleducation and the school to work transition. These topics will be of substantialimportance to Congress and the nation in the coming years, and OTA is gladto contribute to this discussion.

Roger C. HerdmanDirector

. . .Ill

Preject StaffJohn Andelinl

Assistant DirectorScience, Information, and Natural

Resources Division

Peter Blair2

Assistant DirectorIndustry, Commerce, and

International Security DivisionDivision

Nancy CarsonProgram ManagerScience, Education, and

Transportation

CONTRACTORS

Richard KimbellGoldsmiths’ CollegeUniversity of London

Nancy KoberCharlottesville, VA

PRINCIPAL STAFF ADMINISTRATIVE STAFF

JOHN WIRT Marsha FennProject Director Technical Editor

OTHER CONTRIBUTING STAFF Tamara KowalskiResearch Secretary

Elizabeth EisnerDetailed, U.S. Department of Gay Jackson

Education PC Specialist

C. Maxine WilliamsIn-house contractor

i~rough August 1~3.

2AfterAugust 1993.

iv

workshop Participantsand Reviewers

WORKSHOP ON CAPABILITY FOR USING TECHNOLOGY, MARCH 9, 1993

Sherrie Gott, ChairpersonAir Force Human Resources Institute

Henry BraunEducational Testing Service

Larry HirschornCenter for Applied Science

Richard Kim bellUniversity of London

REVIEWERS

Barbara BorderEducational Leadership Consultants

Neal B. CareyCenter for Naval Analysis

Cheryl Fields TylerAmerican Electronics Association

Brenda HattawayVocational-Technical Education Consortium

of the States

Gary HoachlanderMPR Associates

Roger ShankNorthwestern University

Joyce ShieldsHay Systems, Inc.

Ronald ToddDrexel University

Shoshana ZuboffHarvard Business School

Richard JonesNew York State Department of Education

Darrell ParksOhio State Department of Education

Evelyn WellsNational Occupational Testing Institute

Joel WestAmerican College Testing

Lauress WiseManpower Data Systems Center

NOTE: OTA appreclafes and is grateful for the \aluable assistance and thoughtji(l critiques proi’ided by the uwkshop participants, re}vew’ers.and contributors. The participants, re~’iewers, and contributors do not, howe~’er, necessarily apprw’e, disapprw’e, or endorse this report. OTAassumes Jill responsibility for the report and the accuracy of Its contenm.

v

contents

1 Purposes of the Background Paper 2Evolution of Performance-Based Accountability in Federal

Vocational Education Law 4Testing and Assessmen tResources in Vocational Education 6Current State Assessment Policies in Vocational Education 11Emerging State Responses to Perkins Performance Standards 14Defining and Assessing Broad Technical Skills 17

2 The Evolution of Performance-Based AccountabilityRequirements in Federal Vocational Education Law 21The Current Requirements 22Legislative History 26Changing Themes 32

3 State Policies on Testing and Assessment 35Design and Methodology of the State Survey 37Origins of Testing and Assessment Policy in Vocational

Education 38Differences Among the States in Policy on Testing and

Assessment 41State Resources for Testing and Assessment 42What Skills are Assessed’? 44Where in the Curriculum are the Skills Assessed? 48Why are the Skills Assessed? 51What Form of Measurement is Used? 53How are Practices of Testing and Assessment Changing? 55What Skills Will Be Assessed? 59

4 Case Studies: Three Vendors of Tests for OccupationalCompetencies 67Work Keys 67Vocational-Technical Education Consortium of the States 72National Occupational Competency Testing Institute 76Findings 80

vii

5 Broad Technical Skills 83Vocational Aptitudes 84Occupational Maps 91Core Occupational Skills 94Design and Technology 95Cognitive Skills 97Final Note 101

6 Implementation of Performance Standardsand Measures 103Technical Assistance Efforts 104ED Regulations and Monitoring 105Research and Demonstration 107Business and Education Standards 107Goals 2000: Educate America Act 108Conclusion 109

A LegisIative Milestones Influencing Accountability inFederal Vocational Education Programs 111

B List of Acronyms 114

Photographs supplied by the American Vocational Asswiafion and Future Business Leaders of America.

—. —

Summary

I n 1990, Congress enacted amendments to the Carl D.Perkins Act requiring states to measure the effectiveness ofvocational education and to set performance standards forvocational programs. The law requires using these standards

to evaluate the effectiveness of local vocational educationprograms and stimulate “program improvement’ efforts wheredeficiencies are found.

The requirements for standards and measures are part of alarger thrust in the 1990 amendments to strengthen accountabil-ity for vocational programs. Other reforms advocated in thelegislation include integrating academics and vocational educa-tion, and creating “tech prep” programs. At the high schoollevel, both involve reorganizing vocational programs aroundbroader sets of academic and technical skills leading to advancedtraining and highly skilled jobs.

The requirements for standards and measures are likely to havea significant impact on the amount and type of testing andassessment occurring in vocational education. High-qualityassessments are crucial; incomplete or inaccurate assessmentresults could lead to unfair or misleading conclusions aboutwhich local programs need improvement. And depending onwhich type of tests are used and how they are applied, theimplementation of performance standards could either promoteor impede the other reform goals of the 1990 law.

To help Congress determine whether better assessmentinstruments should be and can be developed for vocationaleducation, section 423 of the Perkins Act directed the Office ofTechnology Assessment (OTA) to study tests used in vocationalprograms, particularly those designed to measure broad technical

1

2 Testing and Assessment in Vocational Education

skills. Congress asked OTA to analyze the uses ofthese tests, identify trends in vocational assess-ment, and identify policy issues relevant toimproving test development and quality.

PURPOSES OF THE BACKGROUND PAPERThis background paper responds to the con-

gressional request by providing a picture ofgeneral progress in vocational education assess-ment instruments and policies. The paper has sixmajor purposes, each explored in greater detail ina subsequent chapter:

●

●

●

●

●

●

to trace the evolution of federal accountabil-ity requirements in vocational education lawand to explain the intent of the 1990 provi-sions (ch. 2);to profile current state testing and assess-ment policies in vocational education (ch.3);to analyze how state assessment policies andpractices are changing in response to PerkinsAct requirements and to consider how thesechanges could affect both other reforms invocational education and the nature of learn-ing and instruction in vocational educationitself (ch. 3);to describe some of the testing and assess-ment resources available to measure variouskinds of occupational skills, including testsdeveloped by three main vendors (ch. 4);to explore alternative approaches for defin-ing, teaching, and measuring broad technicalskills and to highlight issues that should beconsidered in moving toward assessmentsthat meet needs identified by Congress (ch.5); andto describe how the Department of Educa-tion has implemented performance stand-ards (ch. 6).

In the course of describing this progress, thepaper also raises a number of broader issues ofhow the policies of testing and assessmentadopted by states and localities may turn out to

affect the implementation of performance stand-ards, accountability in the federal legislation, andultimately the nature and content of vocationaleducation. Understanding these relationships isparticularly important, because the purpose of thestandards is to make decisions about the effective-ness of programs at the state and local levels.

It is also important to note that testing andassessment is conducted in vocational educationfor many purposes. These include instructionaldiagnosis, grading student performance, certifica-tion of student competence for employers, andvarious other policy purposes at the state andlocal levels. OTA’s survey covered all practicesof testing and assessment affected by state policy;not just testing and assessment tied to perform-ance standards.

Defining Types of SkillsSeveral kinds of skills can be assessed in

vocational programs, reflecting the multiple goalsof vocational education. For purposes of thispaper, OTA has grouped them into four types (seebox l-A). The first type are academic skills,primarily the areas of reading, writing, andmathematics. The other three types are variouskinds of occupational skills: vocational skills,which tend to be job specific; generic workplaceskills, which encompass employability skills suchas positive work attitudes, as well as teamwork,effective communication, and other kinds ofgeneral workplace competencies; and broad tech-nical skills, which are the core skills and under-standings of technology, information, and organi-zation needed to perform effectively within anindustry or range of occupations.

Methodology and Limitations of This StudyThis background paper draws on several re-

sources. Although assessment practices in voca-tional education have not been the subject ofmuch prior research, OTA reviewed the availablestudies. Of particular relevance were a 1992 studyof state implementation of performance standards

Chapter 1: Summary 3

Box I-A–Types of Skills Assessed in Vocational EducationAcademic Skills

The knowledge and skills in the traditional academic subject areas of English, mathematics, science, history,and so forth that students are expected to acquire in school. In vocational programs, these skills maybe testedfor or assessed directly using standard methods or in the context of occupational problems or situations.

Occupational Skills

As used in this report, occupational skills refer to knowledge and skills other than academic needed to performeffectively in jobs; in other words, either vocational, generic workplace, or broad technical skills

Vocational Skills-The specific knowledge and skills required in the performance of particular jobs or groupsof jobs within a certain occupational area.

Generic Workplace Skills-There are two types: employability skills, such as positive work attitudes andknowledge of how to find a job; and general competencies, such as ability to work in teams, communicate wellwith others, and solve problems. One definition of these general competencies is the five workplace competenciesof the Secretary’s Commission on Achieving Necessary Skills (SCANS).l

Broad Technical Skills-The core skills and understandings of technology, information, organization, andeconomics needed to perform effectively within an industry or group of industries.

1 The five workplace competencies identified by SCANS are that effective workers can productively use resources,interpersonal skills, information, systems, and technology. Secretary’s Commission on Achieving Necessary Skills, WhatkWk Requires of Schools (Washington, DC: U.S. Government Printing Offioe, June 1991).

by MPR Associates;1 a survey of state compe- tions for future policy analyses. Many of thetency testing in vocational education by Missis-sippi State University;2 and a survey of stateinitiatives for industry skill standards by theNational Association of State Directors of Voca-tional Education.3

To obtain more information on current statetesting practices and proposed changes, OTAconducted a survey of the 50 states and theDistrict of Columbia, following up with tele-phone interviews. OTA also conducted casestudies of three major test vendors in vocationaleducation.

This paper is exploratory, intended to providebasic descriptive information and point to direc-

issues raised are complex and will require furtherstudy. For example, data on test use are notavailable for special populations of students, anarea of keen congressional interest. More exten-sive study of local assessment practices in voca-tional education would also be beneficial. Be-cause states vary greatly in their influence onlocal assessment practices, state surveys canprovide only a partial view of local practices.

Over time, the testing and assessment ap-proaches and instruments chosen by states andlocalities will vary. Careful attention to the effectsof these choices on the implementation of per-formance standards and on the nature of curricu-

‘ Mikala L. Rahn et al., MPR Associates, Inc., “State Systems for Accountability in Vocational Education, ” prepared for the U.S.Department of Education, Office of Vocational and Adult Education, December 1992.

z Rebecca lmve-Wilkes and Ronda Cummings, Research and Curriculum Unit for Vocational, Technical, and Adult Education, MississippiState University, ‘‘ 1990 State of the Art Report on Statewide Student Competency Testing in Vocational and Technical Education, ’ preparedfor the National Network for Curriculum Coordination in Vocational-Technical Education and the Southeast Curriculum Coordination Center,October 1990.

~ Barbara Border, Educafion-Dri}’en Skill S(andurds Systems in the United Slates, prepared for the U.S. Department of Education(Washington, DC: National Vocational Technical Education Foundation, October 1993).


lum and instruction in vocational education willbe required to determine whether the legislativegoals have been reached.

EVOLUTION OF PERFORMANCE-BASEDACCOUNTABILITY IN FEDERALVOCATIONAL EDUCATION LAW

The current requirements for performance stand-ards and outcome measures are the most recentstage in the evolution of accountability require-ments in the federal vocational education law.Every major vocational education law sincethe Smith-Hughes Act of 1917 has includedaccountability requirements, but the tools forachieving accountability have become moreoutcome-oriented over time.

In the early years of federal support, when theprimary goal was to encourage growth in voca-tional education, accountability was enforced byregulating program “inputs.’ States were di-rected to establish and adhere to minimumrequirements for vocational teacher qualifica-tions, classroom equipment, and instructionalhours per week. These requirements gave federalprogram administrators a tool that they wieldedaggressively to shape the growth of vocationaleducation, thereby helping to establish the con-cept of federal funds as carrot and stick.

The first attempt to define and look system-atically at outcomes of vocational educationoccurred with the Vocational Education Act of1963, which introduced a requirement for peri-odic evaluation of local programs. Program offer-ings were to be reviewed in terms of . . currentand projected manpower needs and job opportu-nities . . .’ in relevant occupational fields;however, the law did not specify how thesereviews were to be conducted. The 1968 amend-ments took another step toward defining out-comes by limiting federal support to local pro-grams that could be demonstrated “. . . to prepare

students for employment . . . or be of significantassistance [to students] in making an informedand meaningful occupational choice. ’ In thisway the amendments emphasized that the primarypurpose of vocational programs was to providestudents with the specific skills needed for realjobs, not just with general learning in the manualarts.

The 1976 law further sharpened the focus onoutcomes by specifying that the mandated localprogram reviews should examine the extent towhich program completers and leavers: a) foundemployment in occupations related to their train-ing, and b) were considered by employers to bewell trained.

The Perkins Act of 1984 explicitly directedstates to develop measures of program effec-tiveness, such as the occupations to be trained for,the levels of skills to be achieved, and the ‘‘. . .basic employment competencies to be used inperformance outcomes, which will reflect thehiring needs of employers . . .“ (section 11 3).Foreshadowing the current movement to defineskill standards for various industries, the 1984law also required states to establish technicalcommittees of business and industry representa-tives; these committees were to develop invento-ries of skills for “priority occupational fields, ’which could be used to ‘define model curricula.

By 1990, Congress had concluded that priorcalls for change had not spurred significantimprovements in the quality of vocational educa-tion. Influenced by experiences with outcome-based accountability in other federal educationand training programs, Congress amended thePerkins Act to require states, within 1 year, todevelop and implement statewide systems of‘‘core standards and measures’ that defined thestudent outcomes expected in local programs.

In delineating the types of outcomes that statescould select, Congress endorsed a broad view ofthe purposes of vocational education that encom-

477 Stat. 406.

s 82 Stat. 1076.


passed academic achievement, dropout preven-tion, and preparation for higher education, as wellas job preparation. Thus, the law states that thestandards must include a minimum of two out-come measures: 1 ) a measure of learning andcompetency gains, including student progress inbasic and more advanced academic skills; and 2)a measure of one or more of the following—jobor work skill attainment or enhancement, schoolretention or secondary school completion, place-ment into a job, additional training or education,or military service. State systems also are re-quired to include incentives or adjustments thatencourage service to special populations, al-though the legislation offers no guidelines on howthis should be done.

By including academic outcomes in the per-formance standards and placing priority in otherparts of the legislation on integrating academicand vocational education, Congress signaled amajor, new direction of federal policy on voca-tional education. The intent of the policy is thatstudents who take vocational education shouldhave the same academic knowledge and skills asother students. It is a statement that students whograduate from vocational programs should be aswell equipped for their future lives of work andlearning as other students.

The 1990 legislation marks a significantturning point in federal accountability byexplicitly tying the process of state and localreview to standards based on outcomes. Begin-ning after 1993-94, each local recipient of PerkinsAct basic grant funding must use the statewidestandards and measures to evaluate annually theeffectiveness of its vocational programs. (Localrecipients may use federal funds, to a reasonableextent, to conduct these reviews.) Eventually,local recipients who are not making substantialprogress toward the standards must develop a‘‘progam improvement’ plan identifying changesto be made in the following year.

The requirement for standards is also signifi-cant as much for what it does not require as forwhat it does. First, Congress did not authorize theSecretary of Education to issue national standardsand measures, but instead gave states consider-able flexibility to design their own systems andselect from a range of outcomes. Only twooutcome measures are required. (In practice,however, most states have adopted multiplestandards and measures. )

Second, the main purpose of the performancestandards is to make decisions about programs.The results of the performance standards arespecifically tied to the annual review require-ment, which has been in the federal legislation forsome time. The standards are not intended tocertify or credential individuals. (This is incontrast to current proposals for industry skillstandards, which would be used for individualcredentialing.) Although testing of students willbe necessary to satisfy some of the performancestandards developed by the states, the law makesclear that the results are to be used primarily toevaluate and improve local programs. It is up tothe state to decide whether to implement perform-ance standards as a system of student certificationor to incorporate student certification functionsinto their overall plan. According to the OTAstate survey, at least two states, Pennsylvania andNew Jersey, have chosen to implement perform-ance standards as systems of student certification.

Third, Congress chose not to link the voca-tional education performance standards to federalfunding or any other incentives or sanctions. AsHouse and Senate reports make clear, no authorityexists for states ‘‘. . . to apply sanctions inconnection with the utilization of measures andstandards at the local level."6 The mild conse-quence that was attached to the local programevaluations—state intervention through joint state-local program improvement plans—was not in-tended ‘‘. . . to be punitive in nature, but rather to

b U.S. Congress, House Committee on Education and Labor, “Applied Technology” Education Amendments of 1989, ” H. Rept. 101-41,Apr. 28, 1989, p. 4.


encourage an infusion of resources from the state[for] programs that are in need of assistance andimprovement.

Fourth, the legislation is not intended toposition Congress as the distant, final judge oflocal processes. Thus, the local level, not thefederal level, is the primary arena for conductingevaluations, reviewing evaluation data, and carry-ing out program improvement (with the involve-ment of the state if needed). The act does notrequire recipients to submit the results of evalua-tions to the Secretary, nor does it direct evaluationresults to be structured so as to yield a nationaldatabase on program effectiveness or a nationalsystem of performance standards. National infor-mation needs are to be met though other mandatedstudies and research.

In passing the legislation, there was somesupport in Congress for encouraging or evenrequiring the development of a national system ofperformance standards for vocational educationbased on the performance standards to be devel-oped by the states. Congress decided to providestates with a great deal of flexibility in definingtheir performance standards and not require thedevelopment of a national system. The issue wasresolved by including in the final legislation astudy to evaluate the quality and comparabilityacross states of the performance standards andmeasures adopted by the states. The presence ofthe study suggests that, in the future, considera-tion could be given to forming a national systemof performance standards for vocational educa-tion. In considering such a step, two of theimportant criteria could well be the extent ofagreement among the states on vocational out-comes and their capabilities for measuring thoseoutcomes.

Both the flexibility given the states and thepossibility of expanding the performance stand-ards into a national system lead to a number of

important future policy questions. States willhave to make difficult decisions on outcomeswith a great deal of latitude and not muchexperience. It is an open question whether infor-mation from student testing and assessment willprove to be useful for making decisions aboutimprovements in local programs. It is also anopen question whether the performance standardsand measures developed for local program improve-ment could or should be used to develop acommon core of competencies and indicators atthe national level. Finally, it is an open questionwhat the effects on the nature and content ofvocational education may be from using testingand assessment information for purposes ofaccountability, especially given the imperfectquality of the available methods and instrumentsof testing and assessment. At this point, theeffects are impossible to predict.

TESTING AND ASSESSMENT RESOURCESIN VOCATIONAL EDUCATION

The federal legislation is silent about the typesof testing and assessment resources needed tomeasure student outcomes and implement per-formance standards. Nevertheless, the new re-quirements will place substantial burdens on stateand local testing and assessment instruments,programs, and practices. It seems clear that thecapacity for testing and assessment must increasein response, but by how much and in whatdirection is uncertain. Because there has beenlittle systematic research on vocational educationtesting and assessment issues, it is not even clearwhat resources currently exist, what the range oftesting practices is, and how these practicescompare with the rest of education. As a startingpoint, OTA pulled together existing informationon testing resources in vocational education andconducted a survey of state practices.

7 U. S. Congress, Senate Committee on Labor and Human Resourees, ‘‘Carl D. Perkins Vocational Education Act Amendments of 1989,’S. Rept. 101-22I, Nov. 21, 1989, pp. 22-23.


Overview of Testing inVocational Education

After reviewing evidence from its state surveyand other research, OTA concludes that testingand assessment practices in secondary voca-tional education differ considerably from therest of education in their features, origins, andapplications. In fact, the best assessment prac-tices in vocational education resemble thealternative forms of assessment just now beingexplored for the rest of education. However, thequality of these assessment practices varies greatlyamong states and localities.

Like the rest of education, vocational educationprograms use both written, short-answer forms oftesting and diverse methods of performanceassessment, but vocational education relies muchless heavily on short-answer methods of testingand more on assessment. The answer formats ofthe written tests are typically matching or multi-ple choice. A growing amount of this writtentesting, as it will be called, is done using eitherinstruments that are centrally developed by statesor test vendors, or locally adapted forms of thoseinstruments (see box 1 -B). The centrally devel-oped instruments are produced through iterativecycles of writing and revision, but the resultinginstruments are typically much less standardizedand easier to adapt than are the highly standard-ized and secure tests that are so common inacademic education. Further complicating thepicture, most of these centrally developed instru-ments include both written and performanceexercises: however, when they are used, thewritten portions generally predominate. At thelocal level, teachers may also prepare and usetheir own written tests. The centrally developedwritten tests and the adapted versions of themproduced at the state and local levels are the focusof this report rather than locally produced writtentests.

The diversity of assessment methods utilized isbroad. The range includes the preparation ofstudent profiles and portfolios, structured teacherratings of student capabilities demonstrated in thecourse of regular classroom work, evaluatedstudent projects, and even organized competitiveevents.

In contrast to the rest of education, both thewritten testing and the diverse forms of assess-ment used in vocational education are nearly allcriterion-referenced rather than norm-referenced—meaning that they are designed to measurewhether students have the knowledge and skillsneeded for particular jobs rather than how theyperform relative to other students.

The testing and assessment done in vocationaleducation stems from very different origins thantesting and assessment in academic education.The roots of standardized academic testing lie inthe mental testing movement in psychology andeducation. 8 The source of testing and assessmentin vocational programs is the competency-basedmovement in vocational training. In properlyconducted competency-based vocational educa-tion, the curriculum content, test items, andperformance exercises are derived from analysesof actual tasks performed by people in specificjobs or occupational areas. Ideally, there is a veryclose alignment of instruction, assessment, andjob tasks. As a result, teachers have a much morecentral role in judging student performance thanin standardized academic testing.

In the best competency-based programs, it isskills learned, not time spent, that drives the paceof instruction for individual students. In thisrespect, the philosophy of competency-basedinstruction and assessment is wholly differentfrom the philosophy of whole-class instructionwith mass testing at fixed points in the curricu-lum. In vocational education, testing and as-sessment are not after the fact, external proc-esses of inspection but integral parts of the

x U.S. Ctmgress, Office of Technology Assessment, Testing in American .$choo/s,” Asking the Righ/ Quesfions, OTA-SET-5 19 (Wash ingt{m,DC L’.S. GfJ\crnmmt Print]ng Office, February 1992), ch. 4.

8 I Testing and Assessment in Vocational Education

Box 1-B--Glossary of Testing Terms

Competency Testing

Competency testing is the administration of written tests to determine whether students have the knowledgeand skills required to perform effectively in a job or occupational area. The individual items of the test are derivedfrom analyses of the specific tasks involved in performing those jobs. The answer format of these tests is typicallyclosed ended, that is, multiple choice or matching. The results of the testing maybe used for purposes of improvinginstruction or instructional programs, documenting or reporting student achievement, or certifying student

capabilities to employers.

competency Assessment

This type of assessment uses one or more methods of observation, rating, and recording of actualperformances to determine the capabilities of students for performing well in a job or occupational area andconveying the results to others. The performances observed and rated may be part of the student’s regularlyassigned classroom or project work, an organized event of some kind, or occur in response to a problem or tasksituation especially assigned for the purpose of assessment. Student performances are typically rated or evaluatedaocording to a structure of valued competencies derived from the analyses of the tasks actually performed on thejob and a scale of performance levels. The preparation of student profiles, portfolios, or other forms of documentingthe results of the assessment and explaining the students’ performances maybe part of the assessment process.The results of the assessment may be used for purposes of improving instruction or instructional programs,documenting or reporting student achievement, or certifying student capabilities to employers.

Academic Testing

In academic testing, written instruments are used to measure the knowledge and skills of students in thetraditional academic subject areas of mathematics, writing, science, literature, history, and so forth. The answerformat of these tests is typically multiple choice, matching, or some other such closed-ended form of response.

process of education—a goal only now being For a number of years, interest has also beenadvanced in academic education. growing in vocational education

In the competency-based tradition, assessment providing students with profilesincludes carefully designed performance exer- petencies and encouraging themcises or tasks, instructor or juried assessment of folios of their accomplishmentscompleted student projects, and teacher assess- seeking.

in the idea ofof their com-to build port-to use in job

ment of regular classroom work using systematicmethods of rating. There is also a strong traditionof organized events in which students compete forrecognition and reward. The competency-basedmodel attempts to systematize these variousforms of performance assessment by providingthe instructor or judge with performance scalesand lists of valued competencies that can be“checked off’ or rated as students performvarious tasks on demand or over a period of time.

Conclusions cannot yet be reached about whetherthese various methods of performance assessmentused in vocational education are more or lessreliable or valid than the written testing that isdone. Before conclusions can be drawn aboutwhich methods are best for which purposes,closer investigations must be conducted in voca-tional education of the consistency and relevanceof different assessment methods, and their actualapplications in vocational education.


Standardized Testing

Standardized tests use written instruments that are developed according to principles of test theory andadministered according to uniform procedures to provide comparability y of results among test takers. Standardizedtesting is principally used to measure academic achievement.

Performance Assessment

In performance assessment, student performances are rated in response to an “on-demand” task; that is,all students respond to the same task (or prompt) that has been given to them expressly for the purpose ofobserving and evaluating their capabilities for performance in certain areas. The tasks are developed throughiterative cycles of trial and revision. Performance assessments maybe academically or occupationally oriented.The tasks imposed are generally situated to simulate real environments and are open ended in that students mayrespond to them in many different ways. They also typically involve more complex levels of thinking and doing thanare possible with closed-ended testing.

Criterion-Referenced Tests or Assessments

These tests or assessments focus on”... what test takers can do and what they know, not how they compareto others.” Criterion-referenced tests or assessments are designed to show how a student is doing relative tocompetencies required on the job or specified educational goals or objectives. l

Norm-Referenced Tests

These tests are designed to compare one student’s performance with the performances of a large group ofstudents. Norm-referenced tests are developed to make fine distinctions between students’ performances andaccurately pinpoint where a student stands in relation to a large groups of students. Assessment is almost bydefinition not norm referenced.2

1 U.S. Congress, Office of Technology Assessment, Testing in America’s Scl)oo/s: Asking the Flight Questions,OTA-SET-519 (Washington, DC: U.S. Government Printing Office, February 1992), pp. 169-170.

p Ibid., pp. 168-169.

The critical issues in performance assessment generally thought to be best for measuring factualare the comparability of judgments from instruc- knowledge and certain forms of cognition and(or to instructor and program to program, and thecorrespondence of those judgments with anystandards that have been set. In some cases,business representatives or parents may be in-volved. With sufficient training for judges, groupmethods of judging, and statistical checks on theconsistency of ratings. it is possible to achievesatisfactory levels of consistency in rater judg-ment across units in performance assessment.

The critical issues in written testing are therelevance of test items to capabilities for actualjob performance and the long-term effects of thetesting method on teaching and learning invocational programs. Written test formats are

reasoning, which may or may not be closelyrelated to ‘‘know-how’ and capabilities neededfor complex and extended performance in theworkplace (and in life).

Resources From Test VendorsTest vendors supply some of the testing and

assessment materials used in vocational educa-tion. Three of the best known testing organiza-tions in vocational education are the Vocational-Technical Consortium of the States (V-TECS),the National Occupational Competency TestingInstitute (NOCTI), and American College Testing(ACT). NOCTI produces the Student Occupa-


tional Competency Achievement Testing (SOCAT)materials. ACT is in the early stages of marketinga new testing program called Work Keys, whichmeasures general workplace competencies. Thetesting and assessment products of these threeorganizations are distinctly different.

V-TECS and NOCTI utilize structured meth-ods of job analysis to develop competency-basedmaterials for vocational education assessment.Both organizations produce performance exer-cises as well as short-answer written test items,although more extensive resources are availablefor written testing.

NOCTI is the more conventional of the two.The tests it develops and sells to states and localprograms are secure, consisting of fixed sets ofitems derived from job analyses in relevant fields.The SOCAT tests produced by NOCTI areavailable for 71 specific occupations.

V-TECS, by contrast, does not sell “tests,’ butinstead provides its 23 state members and constit-uent local programs with 35 different ‘‘test itembanks, ’ which they may use to construct testsreflecting their own state or local priorities.(V-TECS testing materials are also available forpurchase by nonmembers.) Each V-TECS itembank is specific to a job area or occupation. Itemsare scrutinized for occupational relevance andpossible bias and pilot-tested for consistency ofresponse. The initial development of V-TECSmaterials is done by vocational education agen-cies in member states.

V-TECS also makes available lists of compe-tencies and performance standards by occupation.These V-TECS catalogs, as they are called, areavailable for over 200 occupations.

In the OTA state survey, state personnelfrequently reported devoting substantial efforts toadapting, redeveloping, and expanding V-TECScatalogs and item banks, or using them inconjunction with competency lists, tests, or itemsfrom other sources. The most common reasongiven for doing so is that neither the V-TECS

materials nor those from other sources adequatelyreflect state and local priorities among differentareas of knowledge and skills. Whether thisreinvention and adaptation is genuinely useful ormerely duplicative is impossible to say from thedata available. Local priorities undoubtedly differfrom state and national ones. Moreover, severalstudies have found that the process of reinventionis essential to the thorough implementation ofinnovations—’ ‘to understand is to invent. ’ Still,questions remain about whether this reinventionaffects the comparability of assessment resultsfrom place to place and how much of it is reallynecessary.

The new battery of Work Keys tests beingdeveloped by ACT differs from the V-TECSmaterials in some important respects. The WorkKeys tests generally fit the basic model of“‘written testing, ’ because of the thoroughnesswith which they are being developed, theirrequirements for standardized administration, andtheir centralized scoring and secure nature.9 Butseveral of the Work Keys tests involve innovativemethods of response, such as listening to audio-tapes and transcribing what is heard, viewingvideos to provide a context for extended forms ofmultiple-choice questions, and watching demon-strations to assess learning from observation. TheWork Keys system is just now being implementedin several states and local programs.

The main innovation of Work Keys is its focuson general workplace competencies, such as‘‘applied technology” and “teamwork,” ratherthan on job-specific skills. ACT plans to providea means to compare profiles of skills needed indifferent job areas with profiles of knowledge andskills demonstrated by test takers. In short, WorkKeys uses a different approach from V-TECS andSOCAT to link test content with job skillrequirements.

How much influence do these three majorvendors have on testing and assessment prac-tices in vocational education? Available evi-

‘) ACT plans w offer local scoring of the Work Keys tests as an option in the future.

dence suggests that their impact is limited sofar. V-TECS currently appears to be having thegreatest influence through its deliberate strategyof modeling good competency testing practicesand providing resources that states and localprograms can use to develop their own assess-ment programs. Not all states belong to V-TECS,however, and V-TECS has tests for only 35 of themore than 200 occupational areas in which jobcompetencies have been defined.

The most concrete estimates of the number ofstudents taking vendor tests are available for theSOCAT tests, which are returned to NOCTI forscoring. Although NOCTI has many other clientsfor its testing products, the number of SOCATtest takers in schools is not large. For 1992,NOCTI reports that 9,015 secondary and post-secondary students took SOCATs; in that sameyear about 720,000 high school seniors werevocational students. 10

Work Keys is too new to say how extensive itsimpact will be, but at least two states, Ohio andTennessee, have adopted portions of it for state-wide use, and many more are considering it.

CURRENT STATE ASSESSMENT POLICIESIN VOCATIONAL EDUCATION

State vocational education agencies are anothersource of testing and assessment resources forvocational programs. Through its survey andother studies, OTA has collected basic descriptiveinformation about state assessment policies forvocational education as another means of deter-mining the resources available for vocationaleducation assessment.

Based on its survey and other evidence, OTAfinds that state testing policies for vocationaleducation are quite different from state testingpolicies for elementary and secondary educationin general.

At the elementary and secondary levels, stateDepartments of Education commonly fund and

operate programs forvarious grade levels


mass testing of students atin various subjects. The

purpose of much of this testing is to demonstrateaccountability, and results are reported to thepublic.

By contrast, no state vocational educationagencies directly administer a program of masstesting or assessment of all students at a fixedpoint in time. In most states, the primary assess-ment responsibility of the agency is to set policiesfor local programs to follow. Most state agenciesalso provide assessment resources to local pro-grams, such as competency lists, test item banks,and tests with instructional materials. The mainpurposes of these state policies are to evaluateprograms and courses, and assess and certify stu-dent progress—not to demonstrate accountability.

Categories of State PoliciesResponses to the OTA survey reveal a variety

of testing instruments and policies among thestates. The 50 states and the District of Columbiareported a total of 92 different components oftesting or assessment for academic skills and the3 different kinds of occupational skills in theirstate programs, or an average of about 2 per state.Generally one of these components is for academ-ic skills and the other is for occupational skills.Some states have more than one component oftesting or assessment in each of these areas.

OTA finds that state assessment policies can begrouped into four distinct categories:

1. Eighteen states mandated or strongly en-couraged written forms of competency testingfor occupational skills in local programs in the1992-93 school year.

● All of these states favor written methods oftesting over alternative forms of assessmentfor occupational skills.

. All provide local programs with competencytests or access to a competency test itembank.

I () \ocatlonil]” ~tu~ent$ ~re ~cfine~ as s[u~en[s who took” nlore than four credits of v(~ationa] educatit)n in [heir high school” careers.

1 2

●

●

●

Testing and Assessment in Vocational Education

These states enroll about one-half of all highschool students.Approximately one-half of them are knownas leaders in the development of competency-based testing in vocational education.The sophistication and comprehensivenessof the testing programs in these 18 statesvaries greatly. Some have comprehensiveprograms of testing and assessment consist-ing of three or four components, such as acompetency-based test item bank and sup-porting resources for over 50 occupations;student profiles or portfolios; and a strategyfor obtaining the test scores of vocationalstudents from the statewide academic testingprogram. Other states in the category mayonly offer local programs a single writtentest of employability skills.

2. Fifteen states mandate assessment of occu-pational skills in local programs withoutspecifying how this should be done.

●

●

●

These states tend to encourage a diversity ofapproaches to assessment without favoringsome methods over others. Various forms ofassessment, rather than written testing, aregenerally encouraged.All of these states require assessment of theoccupational skills of students, but allowlocal programs to choose their own methodor methods.These states are much less likely to providelocal programs with competency tests oritem banks than states in category 1, and aremore likely to provide materials for develop-ing student profiles or portfolios.These states enroll about one-quarter of allhigh school students.

3. Ten states encourage assessment of occupa-tional skills in local programs without specify-ing how this should be done.

. Like the previous category, these statesencourage diverse approaches to assessmentrather than written testing.

●

●

●

Testing and assessment are only encouraged,not required. The encouragement given totesting and assessment is generally notstrong.The only assessment resource that thesestates are likely to provide local programs istesting or assessment materials that comewith instructional resources for a competency-based curriculum.These states enroll about one-eighth of allhigh school students.

4. Eight states had no specific policy or programin school year 1992-93 to encourage or requiretesting or assessment of occupational skills.

●

●

●

In 1992-93, these states provided no re-sources to local programs for assessingoccupational skills.Most of these states have established per-formance standards to comply with thePerkins Act through using measures that donot require information from testing orassessment, or by deferring adoption to theirlocal programs.These states enroll about one-eighth of allhigh school students.

Types of Skills AssessedState assessment policies for vocational educa-

tion cover the four types of skills describedearlier-academic, vocational, generic workplace,and broad technical skills (see box 1-A).

Vocational skills—in other words, job-specificskills—are the type most commonly assessed,followed by academic skills. All 43 states with anoccupational assessment policy (those in the firstthree categories above) have a policy for requir-ing or encouraging assessment of vocationalskills.

In the first three categories, 31 states also havein place a policy for assessing the academic skillsof vocational students. In addition, all eight statesin the fourth category have policies for assessingthe academic skills of vocational students. Stateshave apparently responded rapidly to the Perkins


Act requirements for academic outcome meas-ures. The OTA survey was conducted in 1992-93,the school year when most states began imple-menting standards and measures as directed underthe Perkins Act.

Far less frequently addressed are genericworkplace skills. Only seven states in the firstthree categories had policies for assessing genericworkplace competencies, and these assessmentswere typically administered along with assess-ments of vocational skills rather than beingconducted separately.

Broad technical skills are scarcely addressedat all in state policies; only one state empha-sized assessment of these skills in the 1992-93school year.

Use of Assessment ResultsStates use the results from assessments of

occupational skills differently than they useresults from assessments of academic skills,according to respondents to the OTA survey. Foroccupational skills, testing and assessment infor-mation are used most often to evaluate instructionand progress in learning, or assess student attain-ment for course or program completion or certifi-cation. The second most frequent use is foraccountability, including accountability under thePerkins Act. The third most frequent use is formaking decisions about the improvement ofcourses, programs, or schools.

For academic skills, information from testingand assessment is used most often to meetaccountability requirements included under thePerkins Act; second most often for studentassessment or credentialing; and third most oftento improve programs, courses, or schools. Virtu-ally no assessment components for either aca-demic or occupational skills are being used tomeet accountability requirements other than thoseunder the Perkins Act.

There are two important conclusions here.First, for-both academic and occupational skills,the least likely use of testing and assessment

information is to improve programs-eventhough this is the main purpose of the Perkinsperformance standards. The reason is unclear;perhaps the information being used is not in auseful form for program improvement or theremay be a lack of knowledge about how theinformation can be used and experience in doingso. Also, data from performance standards willnot start to become available in most states until1993-94, after the first year of operation.

Second, information about students’ aca-demic skills is substantially more likely to beused for purposes of accountability alone thanit is for assessing student progress or improv-ing programs. This finding suggests that statepolicies for assessing academic skills may havebeen adopted primarily to comply with thePerkins Act. This indicates that either policies oftesting and assessment for academic skills are stillin the early stage of implementation, or theacademic information being obtained is even lessuseful than the occupational results for improvingprograms.

Strategies for Obtaining AssessmentInformation

Strategies to obtain assessment informationvary among states and according to whetheracademic or occupational skills are being tested.In general, assessments of occupational skills aremuch more closely tied to local vocationalcurricula and instruction than assessments foracademic skills.

To obtain information about academic skillsof vocational students, most states have chosento use scores from their centrally administeredstatewide testing program rather than developnew, locally based strategies more closelyrelated to their vocational programs. Of the 31states that test for academic skills, most use eithera state minimum competency exit examination oranother test administered statewide at a particulargrade level; 26 of the 40 different state academictesting components reported in the OTA survey


were of these two types. Typically these examsare administered in the 9th, 10th, or 11th grade;however, some states are using test score informa-tion from the 8th grade. Under these circum-stances, it is very hard to see how one could usethe information about students’ academic skills toimprove vocational programs. Most vocationalcourses are in the 11th and 12th grades.

In contrast, information about occupationalskills comes from assessment programs thatare either tied to students’ completion of acourse or course sequence, or are ongoing inthe vocational curriculum; 31 of the 54 occupa-tional testing components reported in the OTAsurvey fit this model.

The relationship between the occupationalassessment program and the vocational curricu-lum varies significantly by state. States thatencourage occupational testing—those in cate-gory 1 above—strongly tend to focus their testingon completion of courses or course sequences(mostly the latter). States that mandate occupa-tional assessment--category 2—are split be-tween those that focus on the course completionand those in which assessment is ongoing. Statesthat encourage occupational assessment—category 3—tend to use ongoing assessment.

Questions arise as to how states will coordinatedissimilar information from academic and occu-pational testing to carry out the new programimprovement requirements.

EMERGING STATE RESPONSES TOPERKINS PERFORMANCE STANDARDS

The OTA survey also asked states about theirplans for expanding, contracting, or continuingtheir current assessment policies for all four kindsof skills over the next 3 years. Questions eliciteddetails on new components to be added, changesin skills to be assessed, and populations andprograms to be tested or measured. The stateswere also asked about the extent to which they areresponding in their expansion plans mainly to thePerkins mandate for performance standards, state

educational reform initiatives, state workforceinitiatives, or other sources. Questions werestructured to determine whether the plans aredefinite or tentative.

Expansion of Testing and AssessmentThe results show that states clearly are plan-

ning substantial expansion of their assessmentprograms and policies. Most of the increase willbe due to expansion of existing testing andassessment components, although some stateswill also add new components. Forty-eight of the92 testing and assessment components currentlym effect are slated for expansion by 1995 and 20new components will be added. The remainingcomponents will stay the same or nearly the same.

The nature of the changes proposed variesgreatly. Some states are planning only minoradditions or modifications to existing programs,while others are planning comprehensive newsystems, Ohio, for example, will implement anambitious expansion of its 28-year-old OhioVocational Competency Assessment Program(OVCAP). OVCAP will be expanded to includethree new tests for generic workplace skills fromthe Work Keys system (which the state piloted in1992-93), along with new or revised competency-based tests in at least 63 occupational areas. Ohiois also changing its policy to strongly encouragecompetency-based tests in all local programs; inthe past, these tests have been made availableonly to local programs that requested them. Inaddition, the state office of vocational educationwill obtain scores of vocational students on theStatewide Ninth Grade Proficiency Exam. At theother end of the spectrum, several states areplanning only to increase the number of occupa-tional areas covered by their current assessmentprograms.

Changes in Written Testing andPerformance Assessment

Several states intend to shift to writtentesting for occupational skills and away from


assessment. This trend may be a significantand apparently unintended consequence of thePerkins Act mandates for performance stand-ards.

There will be eight new components of writtentesting for occupational skills among the statesand three new components of assessment; how-ever, two existing occupational assessment com-ponents will be eliminated for a net gain of onenew component of assessment and eight ofwritten testing. Essentially this means that occu-pational skills will be measured more like aca-demic skills are now, largely through short-answer tests.

A major issue is how this shift toward writtentesting will affect the character and content ofvocational education. Vocational programs areone of the last places where written testing wouldseem to provide better information than perform-ance assessment for most purposes. For example,the results of performance assessment consist-ently have been found to be more directly relatedto competence on the job than competencytesting.

Research also has shown that the imposition ofhigh-stakes, standardized testing for basic skillscan narrow the range of skills taught by teachersand learned by students in regular academicclassrooms. Presumably the same kinds ofeffects could occur in vocational educationover the long term, with the content of teachingand learning shifting away from developmentof more complex skills toward acquisition offactual knowledge, and away from activestudent production toward classroom lecture.The effects on the character of vocationaleducation could be profound.

There is very little good research on theinstructional effects of written testing versusperformance assessment in learning environ-ments like those found in vocational programs, soit is not possible to come to any firm conclusionsabout the issue. One of the best known pieces ofresearch is an article on training in the military,which gives examples of dramatic effects on

learning goals and activities that occurred whenperformance assessment was substituted for writ-ten testing in training environments. When thishappened, classroom lecturing on topics such asthe muzzle velocity of guns greatly diminished,desks were removed, and students spent muchmore of their time repairing equipment that theywere supposed to be learning how to repair. Theperformance of students in repairing equipmentsharply increased.

It is ironic that written testing methods appearto be expanding in vocational education at thevery time questions are being raised about theeffectiveness of standardized testing in the rest ofeducation, and experimentation with performanceassessment is flourishing. One of the reasons forthis is to assess more complex cognitive skillsthan can be assessed with written testing. Anotherimportant issue is how states will coordinatedissimilar information from academic and occu-pational testingment.

Changes in

for purposes of program improve-

Skills AssessedState policies are also changing with regard to

the types of skills to be measured (see box 1 -A).The greatest expansion will occur in testing orassessment of vocational skills—the job-specific skills conventionally taught in voca-tional programs. Testing for vocational skillswill expand in 35 of the 50 state components thataddressed these skills in the 1992-93 school year.In addition, five new components for vocationalskills will be added by 1995.

Assessments of generic workplace skills areexpected to increase at a high rate from 8components to 22, although 17 of the 22 willassess generic workplace skills in combinationwith other occupational skills.

Assessment of broad technical skills willexpand five-fold but will still be the smallestcategory of testing. There are several reasons forthis. One is the lack of clarity about the nature ofthese skills and the scarcity of instruments and


methods for assessing them. Another is that moreemphasis has been placed on integrating academ-ic and vocational education than on reorganizingvocational curricula around broad technicalskills. Interest in broad technical skills is mount-ing, however. The 1990 amendments identified“applied technology, ’ or applied academics, asa theme for vocational education and stressed theneed for vocational students to learn about “allaspects of industry. ’ The strongest indication isin the legislation pending before Congress todevelop voluntary national industry skill stand-ards, where emphasis would be placed on orient-ing skill training around clusters of broad occupa-tional skills; supporters contend that this willimprove the quality of training, with long-termpayoffs for workers and the economy as a whole.

The area with the slowest growth of testingand assessment will be academic skills. Tenstates will add academic skills for the first time,but in states that already assess academic skills,expansion is planned for only 17 of 41 compo-nents; the other 24 components are not slated forchange. This is somewhat surprising, in light ofthe new Perkins provisions, especially the direc-tive for performance standards to measure gainsin student academic skills over time. Test scoregains are generally much more difficult to meas-ure than achievement at a single point in time,especially in a mass testing program. It is hard tosee how, as many states are doing, test score datafrom a minimum competency exam given at onepoint in time or from a state test administered ina single grade can be used to show gains instudent academic skills.

1 Effects of New Policies onVocational Reform

A key issue is how these state changes intesting and assessment changes will promote orimpede the other reform goals of the revisedPerkins Act, such as integrating academic andvocational education and broadening preparationin technical skills. It may well be the case that

academic and vocational education are beingdriven further apart rather than closer to-gether by the responses of states to perform-ance standards. Resolving this problem willrequire efforts by the states to develop assessmentmethods that are more compatible and consistentwith the goal of academic integration.

As noted above, states are using differentapproaches to measure academic and occupa-tional skills. Most states are relying heavily onnorm-referenced, standardized tests to measureacademic skills, and on locally adapted andcriterion-referenced tests to assess occupationalskills. Testing for academic skills is also predom-inantly written, while occupational skills arebeing measured through a mix of performanceassessment and written testing. Academic testingis centralized and conducted statewide at fixedgrade levels, while occupational testing is highlydecentralized and tied to the local structure ofcourse and program completions. In addition,most of the academic testing occurs in the 9th,IOth, and 1 lth grades, while the majority ofoccupational testing occurs as students completecourses or programs, mostly in the 11th and 12thgrades. It is hard to see how academic testinformation collected in grades 9 through 11 canbe used to monitor and/or support the integrationof academic and vocational education at grades11 and 12.

In addition, the information from standardizedacademic tests is often in the form of ‘‘rank inclass, ’ while the information on occupationalskills may be much more performance orientedand competency specific. It is hard to see howsuch information can be used systematically forthe highly localized school-by-school, progranl-by-program, and teacher-by-teacher nature of theefforts required to integrate academic and voca-tional education.

In short, there are formidable problems inreconciling and properly interpreting test scoresfor purposes of monitoring and improving voca-tional-academic integration. Inattention to thesedifficulties could lead to serious misuse of test

Chapter 1: Summary I 17

results. If the content of the statewide academictests does not reflect the academic content of theprograms being integrated, the tests results couldgive a misleading impression of the success of theefforts.

DEFINING AND ASSESSING BROADTECHNICAL SKILLS

The implications of organizing vocational pro-grams around broad technical skills are farreaching. Large segments of vocational educationare now competency based and oriented topreparing students for specific jobs or job areas.Broad technical skills, by contrast, tend to beclustered around an industry or industry group-examples include health care, financial services,manufacturing, hospitali~~, and a@business. Broadtechnical skills could even be defined to includehistorical knowledge of the social and economicdevelopment of the industry. Thus, organizingvocational education around broad technicalskills could direct more of the vocational educa-tion effort to preparing people for careers andon-the-job learning, rather than for specific entry-level jobs.

Data from the OTA survey indicate thatorganizing vocational education around broadtechnical skills is a relatively low priorityamong state and local programs; most voca-tional programs continue to be oriented to-ward occupationally specific competencies.Only 1 state assessment component out of 92current components is oriented primarily to broadtechnical skills; by comparison, 51 componentsfocus on vocational skills. Only four more com-ponents in broad technical skills are planned forthe next few years.

A major reason for this low priority is the lackof existing models to illustrate what broad techni-cal skills are and how they can be taught. Thereare no clear alternatives to the job competencymodel, which drives prevailing perceptions of thenature of skill, provides the basis for developing

curricula and tests, and generally frames voca-tional education and much of the training enter-prise. Concepts of broad technical skills must bedefined before assessment programs and method-ology can be developed and validated.

As a first step toward developing concepts ofbroad technical skills, OTA has identified fivealternative approaches, founded on substantiallydifferent assumptions about the relationshipsbetween general and specific skills and betweenfoundation and more advanced skills. These fivealternatives are vocational aptitudes, core occu-pational skills, occupational maps, design andtechnology, and cognitive skills.

I Vocational AptitudesVocational aptitude methods reflect a theory

that people perform best in jobs for which theyhave strong abilities, and that these abilities areidentifiable through tests. Vocational aptitudetests have been developed by commercial pub-lishers, the military, and others; these tests areused to select people for jobs or training programsor to guide career counseling. They are developedby postulating general and specific abilities thatmight be good predictors of performance or careeroutcomes in a range of occupations, and thenselecting tests measuring those abilities. Certaindomain-specific abilities used in the final test canbe viewed as definitions of broad technical skills.

A good example of this type of test is theArmed Services Vocational Aptitude Battery(ASVAB). The ASVAB consists of seven majorcontent areas of general academic and intellectualability and three content areas of technical ability.The general abilities include verbal compre-hension, arithmetic reasoning, and coding speed.The technical abilities are mechanical com-prehension, electronics information, and autoand shop information. The three measures oftechnical skills have been shown through valida-tion research to be significantly related to per-formance.


1 Occupational MapsA second concept of broad technical skills is

emerging in some of the industry skill standardsprojects being supported by the U.S. Departmentsof Education and Labor. Grantees are urged toorganize their standard-setting efforts around‘‘broad occupational areas. ’ The skill definitionsare task-oriented, as in the job competencyapproach used by V-TEC!$, but are defined acrossa broad range of similar jobs within an industry.

The American Electronics Association, forexample, is considering a structure for its skillstandards project that wo~ld organize 60 percentof electronic industry jobs that do not require abachelor’s degree into three areas: manufacturingspecialist, pre- and post-sides analyst, and admin-istrative/informat ion serv i ces spec ial ist. The map-ping process involves identifying the majorpurpose of work in each broad area and defininga limited number of criticid functions. For each ofthese critical functions, generic activities andexplicit criteria for good performance are speci-fied. These activities and criteria provide a clearerbasis for setting performance-level standards thanthe job competency methc)d. Categories of knowl-edge and cognitive skills underlying job perform-ance can be defined as well. For example, formanufacturing specialists, the purpose of the jobis to develop, manufacture, deliver, and improveelectronics products and processes that meet orexceed customer needs. The initial functions areto:

. ensure the production process meets busi-ness requirements,

. initiate and sustain communications,

. establish customer needs,

. determine design manufacturability,

. use human resources to manage work flow,

. select and optimize equipment to meetrequirements, and

. make products that meet customer specifica-tions.

1 Core Occupational SkillsIn vocational education. the basic approach to

broadening technical skills has been to groupvocational programs into clusters of occupationsand adopt a guiding common core of requiredoccupational knowledge and skills. At the intro-ductory levels of instruction, all students in alloccupational areas take the same courses, whichare directly organized around the core skills. Ateach more advanced level, students take coursesin one of the cluster areas. Instruction is organizedaround the same set of core skills but they becomemore specialized and embedded in more specificoccupational content. In New York, where coreoccupational skills strategy has been implem-ented, the skill areas are: a) personal develop-ment, b) social/economic systems, c) informationskills, d) resource management, and e) technol-ogy.

1 Design and TechnologyA fourth approach focuses on development

capability for designing technological systems.Advocates assert that all students would benefitfrom becoming proficient in the design of sys-tems, not only those headed for work or 2-yearcollege programs. The view is that the develop-ment of proficiency in designing and buildingtechnological systems and learning about tech-nology as a product of human civilization shouldbegin in the early grades. A few such content-oriented concepts could provide a competency-oriented but comprehensive definition of broadtechnical skills.

Some design and technology courses are taughtin the United States at the high school and collegelevels, but at the high school level the concept hasbeen developed much further in Great Britain andother foreign countries. Over the past 20 years,the British have developed a sequence of coursesthat now are among the 10 major strands in thenew national curriculum, along with mathemat-

Chapter 1: Summary I 19

ics, science,jects. Design

foreign languages, and other sub-and technology is taken by boys and

girls and high and low ability students, beginningin 1st grade. It is conducted to appeal both tostudents who are interested mainly in the humani-ties and those who are more scientifically ortechnically inclined.

Design is the more significant of the twocomponents; students become designers. Theyacquire the procedural knowledge of learninghow to design; the capability to communicatecomplex ideas with clarity, confidence, and skill;and the conceptual knowledge and understandingof materials, energy, aesthetics, and people andtheir needs required to create and build effectivetechnological systems.

The procedural knowledge of learning how todesign involves learning how to think concretelyin complex situations, make choices, and useknowledge to produce better designs. Studentsweigh the desirability of alternative designs fromsocial, economic, aesthetic, and human stand-points, as well as from the perspective of produc-ibility.

B Cognitive SkillsA fifth approach defines broad technical skills

in terms of cognitive skills, This approach isbased on research from cognitive science thatidentifies skills needed to troubleshoot equipmentand solve problems in a range of occupations,apprenticeship situations, and academic learning.

Much of this research focuses on explaining thedifferences between the cognitive skills of expertsand novices. Research has shown that people whoare expert in a domain have acquired largecollections of schematically and meaningfullyorganized factual, conceptual, and proceduralknowledge that bears on the technological de-vices or complex systems in their field. Much ofthis knowledge is highly specific to these devicesand the contexts in which they are used. Thesestructures of knowledge enable experts to under-stand the complex relationships necessary for

skilled performance. Experts differ profoundlyfrom novices in the speed and flexibility withwhich they can a) access these structures ofknowledge, and b) think “metacognitive] y””-that is, set goals, apply procedural skills flexibly,and learn from experience in solving problems.

Cognitive skills are acquired in stages. Initiallya skill is heavily dependent on declarative, orverbal, knowledge. In the declarative stage, thelearner either encounters or is taught the facts andprocedures relevant to executing a particular skill.These facts are stored in memory as ‘‘state-merits, ’ which can be verbalized and recalledone-by-one in the steps required by the cognitiveskill. The second stage is skill automation, orcompilation of the cognitive skill. In this process,the factual and conceptual knowledge acquired inthe declarative stage is transformed gradually intoa highly organized procedural form that can beaccessed and used with minimal conscious rea-soning activity. The third stage is skill refine-ment, or procedural ization. In this stage, perform-ance of the skill is speeded up by weeding outnonessential steps and strengthening associationsbetween possible occurrences and effective re-sponses. As a result, performance of the skillbecomes much more sensitive to small but criticalsituational differences, and the flexibility ofresponse to unexpected situations or new datagreatly increases. This model has been applied ina wide range of domains, from power plantcontrol to financial planning to tennis.

These basic concepts from cognitive sciencebegin to suggest how the various approaches tobroad technical skills are related to each other;they also begin to point the way toward poten-tially more powerful definitions of broad techni-cal skills. The concepts imply that broad technicalskills could be defined in terms of the attributes ofthinking and performance that enable individualsto perform in expert-like ways within suitablydefined occupational domains in comparison toindividuals who are not so expert.

Broad technical skills might be described asskills that are deep in selected areas, but robust


and flexible across broader domains. They consist The ability to utilize expertise in novel situa-of tightly integrated structures of contextual, tions and in new domains needs to be measured byconceptual, and procedural knowledge that are both capacity for responding to new tasks anddemonstrated and expressed through a variety of evidence of general learning aptitudes.verbal, visual, behavioral, and other more tacitways.

The Evolution ofPerformance-Based

Accountability Requirementsin Federal Vocational

Education Law 2

I n 1990 Congress enacted amendments to the Carl D. PerkinsVocational Education Act that required states to developand implement standards and measures for the outcomesexpected of vocational education students. These standards

and measures are to be used to evaluate the effectiveness of localvocational education programs and to stimulate ‘‘programimprovement efforts where deficiencies are found.

These provisions for performance standards, which states andlocal institutions are now implementing, are the most recentstage in the evolution of accountability requirements in federalvocational education law. Accountability means different thingsto different people, but in the federal context it usually entailssome or all of the following aims: assuring that federal funds areused for their intended purposes and beneficiaries, determiningwhether the federal investment is yielding results, generatingpolitical support for federal aid, encouraging more effectiveprograms, and acquiring information for planning and policydecisions. ’ In the past, various tools have been used to achieveaccountability, including content and staffing standards; stateplans and assurances; detailed guidelines for program adminis-tration and service delivery; mandates for local, state, and federalevaluations; federal and state monitoring; and assessments byoutside bodies.

‘ Another definiti{m of accountability is that: ’“1( is a process to help people whoexpect specific benefits from a particular activity (and whose support is essential to itscontinuation) judge the degree to which the activity is working in their interest so that theymight sustain it, modify it, or eliminate it. See Paul T. Hill et al., ‘‘Pand(lra’s BOX:Accxmntabil ity and Performance Standards in Vocational Education, ’ paper prepared forthe Nati(mal Center for Research in Vocational Education, December 1992, p. 9.

21

22 I Testing and Assessment in Vocati~al Education

This chapter seeks to illuminate congressionalintent in the 1990 amendments by analyzing theevolution of accountability and evaluation re-quirements in federal vocational education lawand related statutes. The first section of thechapter summarizes the current accountabilityrequirements of the Perkins Act. The secondsection traces the legislative history of account-ability through several decades of vocationaleducation laws. A final section identifies chang-ing themes.

THE CURRENT REQUIREMENTSThe Carl D. Perkins Vocational and Applied

Technology Education Act Amendments of 1990,Public Law 101-392, went well beyond a “. . .routine, business as usual reauthorization . . .‘by substantially revising the Perkins Act of 1984,the main law governing federal aid to vocationaleducation. Among the more consequential changeswere interlocking revisions in the law that havetipped the balance of accountability provisionsfrom funding and planning requirements to thedirect assessment of students’ outcomes.

1 Rationale for Outcome-BasedAccountability

Several developments spurred the IOlst Con-gress to move toward outcome-based accounta-bility in the Perkins Act. First, national studiesindicated ‘‘. . . the need for a massive upgradingin the quality of secondary vocational educa-tion.’ The school reform movement of the 1980s

had largely “. . . bypassed vocational educa-tion. ’ ‘q Concern was prevalent among businessleaders that many high school graduates (includ-ing, but not limited to. vocational programgraduates) lacked the academic, employability,and occupational skills needed to compete in aglobal economy. And all too often, the poorestquality vocational programs were found inschools serving low-income students.4

Second, evidence suggested that the 1984Perkins Act had not been as effective as hoped.sThe congressionally mandated National Assess-ment of Vocational Education concluded that thePerkins Act was a weak mechanism for improv-ing program quality and increasing access ofspecial populations to good programs; the studysuggested performance indicators as a possibleway to strengthen the act.b

Third, attitudes about how to judge the effec-tiveness of human resource programs hadchanged considerably since 1984. “Outcome-based” systems of accountability, which placedrelatively less emphasis on whether programswere adhering to procedural requirements andrelatively more on whether they were producingthe intended results, were gaining popularity atthe federal, state, and local levels.7

Federal precedent for outcome-based evalua-tion already had been established through 8 yearsof experience with Job Training Partnership Act(JTPA) performance standards, new requirementsfor performance-based evaluation in the JobOpportunities and Basic Skills (JOBS) trainingprogram for welfare recipients, and new “pro-

2 Augustus F. Hawkins, Congressional Record, U.S. Congress, House, 101st Cong., 1st sess., May 9, 1989, H 1709.

3 John G. Wirt, National Assessment of Vocational Education, testimony at hearings before the House Committee (m Education and Labor,Subcommittee on Elementmy, Secondary, and Vocational Education, Mar. 7, 1989, p. 5.

4 Ibid., p. 7.

~ Among the studies cited as influential by House and Senate Committee reports were U.S. Department of Education, National Assessmentof Vocational Education, Final Report, Volume 1: Summury of Findings and Recommendation.s (Washington, DC: 1989); U.S. GeneralAccounting Office, Vocational Education: Opporrunily 10 Preparefor Ihe Future, GAO/HRD 89-55 (Washington, DC: 1989); and AmericanSociety for Training and Development, Training America: fzarning 10 Workfor /he 2]sr Cenrury (Washington, DC: 1988).

c National Assessment of Vocational Education, op. cit., f(xmmte 5, p. ix.7 Richard N. Apling, Congressional Research Service, Education and Public Welfare Division, ‘ ‘Vocational Educati(m Perf(mnance

Standards,” 89-440 EPW, July 6, 1989, pp. 3-4.

Chapter 2: The Evolution of Performance-Based Accountability Requirements 123

gram improvement’ requirements in the Chapter1 legislation for disadvantaged children. Voca-tional education seemed a good candidate for thisapproach, in light of its extensive use of competency-based curricula and its ‘*somewhat ill-definedtradition’ of examining such outcomes as jobplacement and employer satisfaction.g

Fourth, as the Perkins reauthorization wasbeing considered, the National Governors’ Asso-ciation and the administration were engaged in abroader discussion about the desirability of na-tional standards for what students should knowand be able to do in key academic subjects.9Around the same time, business panels werebeginning to consider the merits of nationalindustry-based skill standards. 10

Amid this climate, Congress reached a conclu-sion about vocational education: ‘‘What wasacceptable as good or adequate before is notacceptable today if our country is to competeeconomical y. 1‘ Not only did vocational educa-tion need to be improved on its own terms, wroteHouse members, but it also needed to be “. . . tiedin much more closely with academic course work. . . ‘‘ and with the reforms occurring across thepublic educational system. ’z

The resulting legislation had several inter-related aims: to target ‘‘. . . money more carefullyon programs that produce results . . .? ‘‘ to inte-

grate academic and vocational education, toimprove programs for special populations ofstudents, and to ease regulatory burdens. Con-gress seemed to view the law’s major provisionsas an interdependent system of checks andbalances that together would “. . . reassure criticsthat the legislative supporters of vocational edu-cation were serious about quality. ’ ‘A

Accountability based on standards and meas-ures appears to have been an important compo-nent of this system of checks and balances—atradeoff for relaxing or eliminating certain proce-dural requirements and funding set-asides. ‘ ‘In-stead of spending money and staff time regulatinglocal programs, ” one senior congressional staffperson explained, ‘‘state agencies will now befree to concentrate on securing results. ”

I Overview of 1990 AccountabilityProvisions16

STATE STANDARDS AND MEASURESAt the heart of the new accountability provi-

sions is a requirement for state boards of voca-tional education to develop and implement, bySeptember 25, 1992, “. . . a statewide system ofcore standards and measures of performance forsecondary and postsecondary vocational educa-tion programs . .“ (section 1 15).17 Congress

X E. Gareth Ht)achlander, ‘‘Perft)mlance Based Policies in Vocational Educati(m, paper prepared for the Research ~(mfcrencc of theAsw)ciati[m for public Policy and Management, Seattle, WA, October 1988.

‘) This debate followed” the 1989 adoption of six National Education Goals as the Governors and president Bush sought w a}s to measureprt)grcss toward the Goals.

I{) Conlnllsslon ” (m the Ski]]s of the American Wt)rkforcc, Ameri[a’s Choice.” I{lx}I Ski/is or L<m Wa~e.s (Rochester, NY’: Nati(mal center(m Educati(m and the Ec(monly, June 1990).

1 I U,s, congress, Hou~e C{)nlnll[tee on Educa[l{)n a~~ Labor, ‘ ‘Applied Technology” Educati(m Arnendmcnts of 1989. ’ H. Rcpt. 101-41,Apr. 28, 1989, p. 4.

11 Augustus F , Hawk Ins, L’onRrer$fona/ Re(.ord, LJ. S. ctmgrcss, House, 101 St Cong., I st sess.. Jan. ~! 1989. ’22

13 J{)hn F. Jennings, ‘‘C(mgressi(mal Intent, ” Vo~o[ional Edlttation Jolirnol, February 1991, p. 18.14 Hll] Ct. a], op. ClI., f(N)tnt)[e I, p. ~’.

] $ Jmnlngs, op. cit., f(x~tm)te I 3, p. 19.16 Sectl{)n references ,n this pan refer t. sections of [he Car] D per~ins \’ocatlonal” and App]i~~ Ttxhrlo](lgy Educii[i(m Act (20 U.S.C.

ZN) ] -z47 I ) aS arnendcd by Puhl ic Law 101-392.

17 For the ] 990 anl~ndnlen[s, a measure means ‘“a descrip[i(m of an OUtConlc,’” ” while a standard means ‘‘the le~ cl {)r rate {~f that t~utconw. ”See 32 CFR 400.4.


intended that these systems would ‘‘. . . apply toall the programs assisted under this Act . . .’ andbe developed with the advice of a broadlyrepresentative state committee of practitioners.

The state systems are required to include aminimum of two outcome measures:

1. a measure of learning and competencygains, including student progress in basicand more advanced academic skills; and

2. measures for one or more of the following—job or work skill attainment or enhance-ment; school retention or secondary schoolcompletion; or placement into a job, addi-tional training or education, or militaryservice.

By specifying outcomes that encompass aca-demic improvement, dropout prevention, andhigher education enrollment, as well as jobpreparation, Congress endorsed a broad view ofthe purposes of vocational education. The statesystems are also required to include incentives oradjustments that encourage service to specialpopulations (and are consistent with individual-ized education plans for disabled students).

LOCAL EVALUATION ANDPROGRAM IMPROVEMENT

The Perkins standards and measures are de-signed to derive their real accountability ‘‘teeth’from their interaction with new requirements forlocal evaluation and program improvement (sec-tion 117). Beginning with school year 1992-93and each year thereafter, every local recipient ofPerkins Act basic grant funding must use thestatewide standards and measures to evaluateannually the effectiveness of its vocational pro-grams. This annual evaluation must also assesshow well programs are providing students withstrong experience in and understanding of theindustry they are preparing to enter. Local recipi-ents may use Perkins Act funding, to a reasonable

and necessary extent, to conduct the mandatedevaluations.

ACCOUNTABILITY FOR SPECIAL POPULATIONS

The 1990 amendments also changed the ap-proach to accountability for special populations.As summarized by one congressional sponsor:

For 30 years wc have talked about access [ofspecial populations] and rightfully so. . . . Whenwe began writing this bill, however, we asked thequestion: Access to what? And if we could notanswer ‘‘access to quality’ or ‘‘access to excel-lence,’ then access was not good enough.19

Toward this end, most of the funding set-asides-the mechanism favored since 1968 for servingdisadvantaged and disabled students and otherswith special needs—were e] iminated, as weresome of the more restrictive process requirernentsgoverning services to special populations. Inexchange, the 1990 law revised the basic grantfunding formula to concentrate more dollars inpoor schools and to target federal support onvocational programs that were coherent andsequential, integrated with academic education,and sizable enough to promise meaningful re-sults—in other words, programs with featureslikely to be effective for all students.

STATE PLANNING AND ASSESSMENTTwo other state-level accountability mecha-

nisms deserve mention. First, the 1990 amend-ments continued the longstanding requirement forstates to develop 5-year plans, subject to federalapproval, that describe and make assurancesabout their vocational programs (section 11 3).Second, prior to submitting their plans, statesmust conduct an assessment of program quality66 , . . using measurable objective criteria devel-oped by the state board . . .’ (section 11 6). Thisassessment is intended to provide baseline infor-

IX 32 cm 403.191.I (, w i] ] ianl F. G()()dl;n~, (longres.sional )?ccord, U.S. Congress, House, 101 St C(mgress, 1st sess., May 9, 1989, H 1703.

Chapter 2: The Evolution of Performance-Based Accountability Requirements I 25

mation about student needs and existing programcapabilities for use in state planning.

Congress considered but rejected provisionsthat would have encouraged or required expan-sion of the performance standards and measuresadopted by the states into a national system ofperformance standards and reporting for voca-tional education. The issue was resolved byincluding in the legislation a study of the compa-rability across states of the standards adopted andthe quality of information available about out-comes. In effect, the study suggests that in thefuture Congress might consider expanding theperformance standards adopted by the states intoa national system.

NATIONAL ACTIVITIES

The mandates for outcome-based evaluation atthe state and local level are reinforced by comple-mentary activities at the federal level. The lawdirected the Secretary of Education to:

●

●

Q

●

provide technical assistance to the states asthey develop their systems of standards andmeasures (section 11 5);submit a report to Congress describing andanalyzing state systems and evaluating thedegree of comparability across states (sec-tion 11 5);consider the implementation of programevaluations and improvements when ap-proving state plans (section 11 4); andconduct research on the development andimplementation of performance standardsand their effects on student participation andstudent outcomes, especially for specialpopulations (section 402).

The amendments also authorized a new na-tional program with future implications for statestandards and measures: the Business and Educa-

tion Standards program (section 416). Under thisauthority, the Secretary of Education has madegrants to trade associations, labor organizations,and comparable national organizations to developnational standards for competencies in variousindustries and trades.zo

SIGNIFICANCE OF THE NEW PROVISIONS

Realizing, perhaps, that the requirements forstandards and measures were breaking newground and that the cooperation of states and localrecipients was vital to the success of this en-deavor, Congress took a cautious and incrementalapproach, remarkable as much for what it does notrequire as for what it does.

First, Congress chose not to link vocationaleducation performance standards to funding orother incentives and sanctions, as is done in theJTPA. No authority exists for states “. . . to applysanctions in connection with the utilization ofmeasures and standards at the local level. ’ ‘z’Furthermore, the mild consequence that wasattached to evaluations—state interventionthrough joint program improvement plans—wasnot intended ‘‘. . . to be punitive in nature, butrather to encourage an infusion of resources fromthe state for programs that are in need ofassistance and improvement. ‘ZJ

Second, the legislation did not authorize theSecretary to issue national standards and meas-ures, but rather gave states considerable flexibil-ity to develop their own systems. Only twostandards are actually required by law, one forstudent learning and the other for attainment ofspecific outcomes. Within these broad param-eters, states could choose the particular measuresfor each outcome, add more outcomes, anddevelop different standards for secondary andpostsecondary programs.

20 ~,~ ~rogran, is being ~)p.rate~ in [an~enl ~ ith a Slnll]ar s~l}] s[an~ards and ceflifica[lon pr(~gram in the Depaflnlent of Labor.

1 I House Conlnlltte~ on Education and Lab(w, op. cit., ftx~mote I 1, p. 14.

‘~ U.S. C(mgress, Senate C(mmlittee (m Labor and Human Resources, “Carl D. Perkins Vocatitmal Educatl(m Act Amendments of 1989, ”S. Rcpt. 101 -~~ ] , Nov. ~ i , 1989, pp. ~~-~~.


Third, it was not the intent of Members ofCongress “. . . to set themselves up as the remote,ultimate judges of local processes. ’ ‘2s Thus, thelocal level, nof the federal level, is the primaryarena for conducting evaluations, reviewing eval-uation data, and carrying out program improve-ment (with the involvement of the state, ifneeded). “The Act does not contemplate requir-ing a recipient to submit the results of theevaluation to the Secretary, ’ ’24 nor are the evalua-tion requirements structured to produce a nationaldatabase on program effectiveness. National in-formation needs are to be met through othermandated studies and research.

Fourth, recognizing that students enroll invocational programs for different reasons, thestate system of standards and measures . . is notnecessarily intended to apply in its entirety toeach individual within the program. ’25

LEGISLATIVE HISTORYEvery major vocational education law since

1917 has included accountability requirements.The form, scope, targets, overseers, and level ofdetail of these mandates have changed consider-ably over time, however (see appendix A).

1 The Smith-Hughes ActAlthough the Smith-Hughes Act of 191726 did

not require program evaluations, it introducedthree principles that laid the ground work foraccountability and evaluation requirements inlater laws. First, Smith-Hughes instituted the

concept of using federal funds as a carrot and astick to stimulate improvement in vocationaleducation and influence state and local policy.Given the newness of the field, improvement wasdefined mostly in terms of growth: more profes-sional teachers, more and better equipment andfacilities, and longer instructional time.

Toward this end, states that desired Smith-Hughes funding had to establish qualifications forvocational teachers, supervisors, and directorspaid from federal funds, and minimum require-ments for plant and equipment in federallysupported vocational classes. The law also pre-scribed minimum instructional hours per weekand per year and required that half the time intrade, industrial, and agricultural programs bespent in practical work experience.27

These requirements gave federal administra-tors a tool that they could wield aggressively toshape the structure and content of vocationaleducationo*g In fact, federal monitoring of state

compliance with federal directives was a primaryaccountability mechanism during the program’searly years.

Second, Smith-Hughes introduced the stateplan for vocational education, thereby inaugurat-ing a “0 . . clear-cut, systems management oraccountability model which became common-place as a federal strategy in subsequent decadesbut was novel for that time. ”29 Specifically, theact required state boards to develop plans—subject to approval by a Federal Board forVocational Education (FBVE)—that described

23 Hi]] et a],, op. cit., footnote 1, p. vi.

2457 Feder~/ Register 36842 (Aug. 14, 199*).

z~ House Conlmittee (m Education and Labor, 0p. cit., fOOmOte 11, p. 14.

’26 us. statures al hrge, w)]. 39, part 1, ch. 114, PP. 9*9-936.

27 me Smith-Hughes requirenlents for mlnlrna] teacher qua]lfica[lons continued until the enactment of the 1976 amendments to the

Vocati(mal Education Act of 1963. The requirements for equipment, program format, and instructi(mal time were not included in the 1963 actand were formally repealed in the 1968 amendments.

2s Larry Cuban, ‘Enduring Resiliency: Enacting and Implementing Fedelal Vocational Education Legislation, ’ Work, YouIh and Sthoo/ing:Historica/ Per.$pec(i)es on Vocationa/ Educa/ion, Harvey Kanterand David Tyack (eds.) (Stanford, CA: Stanford University Press, 1979), pp.80-8 I.

Z’J Ibid., p. 106.


the programs, schools, equipment, and courses tobe funded, the instructional methods to be used,the teacher training to be provided, and thequalifications of teachers and supervisors. Theeffects of this requirement were far-reaching;federal officials produced detailed guidelines forthe content of the plans, participated actively intheir development and review, and often calledfor revisions.so As one analyst summarized:

The creation of a State Plan signed by bothstate and federal authorities, its review by theFBVE and the staff-written regulations thatfollowed made it possible for the Federal Boardto influence directly state vocational programswhile contributing modest financial support.31

Third, the act enunciated one of the key goalsof vocational education that would appear (inupdated wording) in all subsequent laws andwould eventually be used as a standard formeasuring program success: ‘‘. . . The controllingpurpose of [vocational] education, ” the legisla-tion read, ‘‘shall be to fit for useful employ-ment. . . .’

1 Vocational Education Act of 1963The Vocational Education Act of 1963 is

generally regarded as the beginning of the modernfederal role in vocational education.32 Enactedamid a period of rising unemployment, PublicLaw 88-210 attempted to respond to criticismsthat vocational education—with its emphasis onthe traditional areas of agriculture, home econom-ics, and trades—had failed to keep pace with . .the more sophisticated economy of the 1960’ s.’ ‘sqThe 1963 act sought to improve vocational

education and provide access to training that was“of high quality” and ‘‘realistic” in light ofemployment opportunities.

The 1963 act introduced the concept of pro-gram evaluations. States were required to conductand give ‘‘. . . due consideration . . . [to] periodicevaluations of state and local vocational educa-tion programs and services in light of informationregarding current and projected manpower needsand job opportunities. ’ ‘s4 Decisions about how toconduct and fund these evaluations and what theywould examine were left entirely to the state, solong as some use was made of labor marketinformation.

The law also created an ad hoc nationaladvisory council that would review and recom-mend improvements in vocational education—the first of several outside bodies charged withstudying vocational programs. As with Smith-Hughes, the concept of “improvement” in 1963was viewed largely in terms of expanded infra-structure and better services.~s This advisorycouncil was later formalized in the 1968 amend-ments. It continued in the legislation up until the1990 amendments, when it was terminated.

Another key provision reserved at least 3percent of each state’s grant for ancillary services,defined to include program evaluation, teachertraining, demonstration and experimental pro-grams, materials development, and state adminis-tration and leadership.

9 Vocational Education Amendments of 1966By 1968, federal support had helped fuel

tremendous expansion of vocational enrollments

.W ~ether [his federal influence he]ped or hindered education is an area of disagreement among the limited b(tiy of research about the early

years of vocational education. See ibid.

31 Ibid., p. 105.~Z 1ntemledla[e Iegl$]atlon affec[lng V(xa[lonal educa(ion included the George-Reed Act of 1929, the George-men Act of 19~6T and ‘he

George-Barden Act of 1946.?~ u s congress, Senate Committee on Labor and ~b]ic Welfare, Vocational Educatitm and National Institute of Education Amendments. .

of 1976, S. Rept. 94-882, May 14, 1976, p. 42.

~A 77 Stat. 406.~~ u s congress, House Comnlittee on Education and Lab(~rt. . “Vocational Education Act of 1963, ” H. Rept. 88-393, June 18, 1963, p. 6.


and expenditures. Even so, concerns persistedthat vocational programs were not adequatelypreparing students for growth occupations andwere maintaining outdated programs.~G Enunciat-ing a theme that would reemerge in 1990, theHouse Committee concurred with national advi-sory council findings that “. . . any dichotomybetween academic and vocational education isout moded, ’ and that the development of atti-tudes, basic educational skills, and work habitswas as important as training in specific occupa-tional skills.q7

The Vocational Education Amendments of1968, Public Law 90-576, sought to address theseconcerns through revised accountability require-ments and other means. The requirement wasextended for states to conduct periodic evalua-tions based on labor market needs and considerthe results in program planning. A new authoriza-tion of funding for state administrative activitieswould help support evaluation.

The most significant change in the 1968amendments was a new provision that limitedfederal support to programs that could be demon-strated “. . . to prepare students for employmentor . . . for successful completion of such aprogram, or be of significant assistance to indi-viduals enrolled in making an informed andmeaningful occupational choice. ‘s8 (Homemak-ing education was exempted.) The amendmentscontained no guidance, however, about how localprograms might demonstrate these outcomes tothe state, nor were they construed by states tomean that a formal outcome-based evaluation wasnecessary.

I Education Amendments of 1976Approaching reauthorization in 1976, Con-

gress pointed to several dramatic changes result-ing from federal support for vocational education,all based on ‘‘inputs’ increased enrollments,higher expenditures from all sources, construc-tion of area vocational schools, more and bettertrained teachers, and a greater number and varietyof course offerings. As the House Committeenoted, however, evidence of program outcomeswas sorely missing:

Measures of the outputs—the success of theprogram completers and leavers in finding andkeeping jobs—are more difficult to find, . . .[T]here is some scattered evidence of placementsuccess for a number of local programs. But interms of periodic and extensive reviews of theseprograms, in terms of the success of their studentsin obtaining and keeping jobs, little can be foundat present. For that reason, the Committee hasrecommended amendments to provide this typeof information for measuring the effectiveness ofthe programs.s9

Accountability was found wanting on severalother scores. A General Accounting Office reportcriticized the Department of Health, Education,and Welfare for failing to follow up on evaluationdocuments submitted by states, for conductingperfunctory reviews of state plans, and for inade-quately monitoring state programs.a In addition,the lack of reliable national occupational informa-tion and local employment data made it hard forstates and local recipients to conduct solid evalua-tions. Perhaps the greatest failing was in followup of job placement and employer satisfaction,

36 u s Congre55, House Committee on Education and Labort. . “Vocational Education Amendments of 1968, ” H. Rep[. 90-1647, July 8,1968, p. 2. The criticism of using federal funds to maintain outdated or poor quality programs would surface in subsequent reauthorizations,until the use of funds for program maintenance was restricted somewhat in 1984.

37 Ibid.

3X 82 Stat. 1076.

w us, Confless, House committee on Education and Lab{)r! “Education Amendments of 1976,” H. Rept 94-1085, May 4, 1976, p. 12.

m Ibid., p. 17.

‘] Ibid., p. 20.

Chapter 2: The Evolution of Performance-Based Accountability Requirements ! 29

which congressional witnesses testified ‘‘. . . wasvery sporadic and extremely uneven. ’ ’41

Concluding that existing mandates seemed66 . . . to be having little effect,”42 Congresssignificantly strengthened the accountability andevaluation requirements in Public Law 94-482,the Education Amendments of 1976. The 1976law for the first time contained a separate sectionon evaluation (section 11 2), with the aim of”. . .assisting] states in operating the best possibleprograms of vocational education. ” Within the5-year period of the state plan, every state had toevaluate the effectiveness of each local programand use the results in revising state programs andplans. For every program purporting to impartentry-level job skills, the evaluations are requiredto show the extent to which program completersand leavers: a) found employment in occupationsrelated to their training or pursued additionaleducation; and b) were considered by employersto be well trained. These two gauges of effective-ness were specified ‘‘. . . because in [the Commit-tee’s] opinion they show most clearly whetherpersons trained in vocational programs are show-ing the results of such training. ’ ‘4s Programs thatwere ‘‘. . . purely introductory or preparatory toactual job training . . .’ were excluded, and datawas to be collected by sampling wherever possi-ble to reduce the burden.w

The 1976 amendments signified a high-watermark for the use of plans, applications, andreports as accountability tools. Public Law 94-482 required a 5-year state plan, developed withthe involvement of 10 representative groups; anannual program plan; an annual state accountabil-ity report that summarized evaluation findingsand described how they were used to improve

programs; and local applications. As part of the5-year state plan, an assessment of current andfuture state needs for job skills was also man-dated.

How well the 1976 evaluation requirementswere implemented was a topic addressed by theNational Institute of Education (NIE) study ofvocational education. ‘‘State and local vocationaleducators objected strenuously to the evaluationrequirements . . .’ and felt overwhelmed by theprospect of implementing the “. . . new, complex,and costly . . .’ procedures.45

Despite state complaints, the NIE study foundthat the 1976 evaluation provisions did stimulateimprovements in evaluation. In 1976, few, if any,states had adopted evaluation procedures ascomprehensive as those called for in the act.Program reviews were the most frequently imple-mented activity resulting from the amendments.At the same time, the study identified practicalproblems with the reliability and consistency ofthe followup data and also found that employersatisfaction data was being collected much lessfrequently than student placement data.% Manyof the pieces were in place, however, and theamendments helped draw together and systema-tize these discrete elements.

1 Carl D, Perkins Vocational Education ActThe 1984 Carl D. Perkins Vocational Educa-

tion Act, Public Law 98-524, replaced the Voca-tional Education Act of 1963. In some respects, itrepresented the apex of federal prescriptiveness invocational education, especially regarding serv-ices for special populations, which were fundedthrough set-asides totaling 57 percent of statebasic grant funding. The remaining 43 percent of

42 Ibid., p. 38.

43 Ibid., p. 38.

u Ibid., p. 39.

45 State and I(xal adnlinls(ra(ors argued (hat the emphasis (m s[udent placement failed [o reflect the broad goals of Vocational education,overlooked ec(momic forces (mtside the c(mtrol of schools, and might diminish service for hard-to-place students. See U.S. Department ofEducation, National Institute of Education, The Votariona/ Educarwn Sfudy: lnlerirn Reporr (Washington, DC: 1980), pp. V-5 to V-8.

m Ibid., p. V- I 1.


the basic grant was targeted on “program im-provement’ —which in the 1984 context meantmaking programs more responsive to contempo-rary labor market needs, especially high-technology fields, and updating equipment, cur-riculum, and staff. The 1984 reauthorizationoccurred in a climate in which the very existenceof high school vocational education was beingquestioned. 47

As one of several amendments aimed at im-proving program quality, the 1984 Perkins Actdirected states to develop measures of programeffectiveness, such as the occupations to betrained for, the levels of skills to be achieved, andthe “. . . basic employment competencies to beused in performance outcomes, which will reflectthe hiring needs of employers . . .’ (section 113).

The U.S. House of Representatives soughtmore specific and outcome-based evaluationrequirements than those that found their way intofinal law. The House wanted the states to developobjective standards for the outcomes of occupa-tionally specific programs, which included ad-justment factors for local situations, and apply thestandards to the approval of local plans and thedirection of technical assistance to improve localperformance. Although the House Committeetook pains to distinguish that these expectedoutcomes were not performance standards, theyforeshadowed the requirements for standards thatwould be adopted 6 years later.

Foreshadowing the Business and EducationSkill Standards program, the 1984 law alsorequired states to establish technical committees,composed of business and labor representatives,to develop inventories of skills for priorityoccupational fields, which could be used to definemodel curricula. These technical committeesinfluenced vocational education to adopt thejob-competency model, where instruction and

testing and assessment are closely tied to thespecific skills needed for individual jobs.

1 Other Influential StatutesPrior to 1990, Congress had already enacted

requirements for performance-based accountabil-ity and program improvement in other federaleducation and training legislation. The approachesvaried, but several influenced the Perkins Actamendments. Two of the most important pro-grams are the Job Training Partnership Act andthe Chapter 1 Program of the Elementary andSecondary Education Act.

JOB TRAINING PARTNERSHIP ACT OF 1982The JTPA was a trailblazer in performance-

based evaluation and continues to be a stronginfluence on other federal human resource pro-grams, including vocational education. Unlikevocational education, the JTPA is a whollyfederal creation, completely federally funded, andis mostly directed to low-income individuals.

Performance standards are established accordi-ng to a hierarchical process, starting with thedefinition of certain broad initial outcomes in theauthorizing legislation.~ For adults in Title II

programs, these outcomes are placement in un-subsidized jobs, retention in unsubsidized em-ployment, increased earnings, reduced welfaredependency, and acquisition of employability andbasic skills (including receipt of a high school orequivalent diploma). For Title 11 youth programs,all of the adult outcomes are applicable, plusattainment of employment competencies, second-ary and postsecondary school completion, drop-out prevention and recovery, and enrollment inother training, postsecondary education, or mili-tary service. Based on these broad parameters inthe statute, the Secretary of Labor providesfurther detail, selecting specific outcomes that

47 U.S. Congess, Senate Committee on Labor and Human Resources, ‘‘Carl D. Perkins Voea[ional Education Act of 1984,’ S. Rept. 98-507,June 7, 1984, p. 2.

0 see ~ectjon ] 06 of me Job Traini~~ pa~nemhip Act of 1982, ~b]ic Law 97-300, enacted OCt. 13, 1982; amended Lkc. ~ 1, 1982, Public

Law 97-404, sec. I (b).


conform with the statutory intent, establishingcore standards and quantitative measures for eachone, determining the time period of measurement,and suggesting cost-effective ways for obtainingthe remaining data.

In the early years of the program, thesestandards focused primarily on short-term out-comes attained immediately on completion oftraining. Recently, the Department of Labor hastried to extend the time period to 13 weeks afterprogram termination.

The law gives governors the flexibility todetermine the relative importance of differentstandards and to prescribe variations in standardsbased on state economic, geographic, and demo-graphic conditions, as well as characteristics ofthe populations served. Governors may alsoprescribe additional standards, develop incen-tives, and sanction policies, including incentivesfor hard-to-serve individuals. Local Private In-dustry Councils make the final translation intobottom-line criteria for service providers.

Unlike vocational education, the JTPA at-taches sanctions and incentives to the standards.Programs that fail to meet standards for 2 years,after receiving technical assistance from thegovernor, are subject to a state-imposed reorgani-zation plan, which may shift funds to anotherlocal administrative entity. In certain cases, theSecretary can withhold up to one-fifth of the localrecipient’s funds for technical assistance.

The JTPA experience with performance stand-ards shows both their benefits and their potentialpitfalls. “In conjunction with clearly identifiedclient and service goals, performance standardsappeared to have their intended effects of increas-ing efficiency and accountability, ’ a 1988 studyfound.J9 At the same time, JTPA performancestandards have been criticized for encouraging‘‘creaming “— focusing services on clients whoare easiest to place rather than the most disad-vantaged—because the prior standards measured

success primarily through high placements andlow cost.

In summary, while the vocational educationstandards address some of the same outcomes asthe JTPA standards and, like the JTPA, allow foradjustments for conditions, they differ from theJTPA in that they are not tied to funding, othersanctions, or incentives, are not based on nationalnumerical measures, and do not address costIssues.

CHAPTER 1, ELEMENTARY AND SECONDARYEDUCATION ACT

Chapter 1 (formerly Title I) of the Elementaryand Secondary Education Act of 1965 was one ofthe first federal education programs to mandateevaluations of student outcomes. In this and otherrespects, it has substantially influenced the Perk-ins Act. Yet Chapter 1 differs from vocationaleducation in many ways, not the least of which isin the evaluation system that has arisen from the1965 requirement.

Chapter 1 is completely federally funded,supplementary to the regular school program,aimed at a distinct target population, and focusedprimarily on academic skills. Evaluation in Chap-ter 1 is governed by a national system, the Title IEvaluation and Reporting System (TIERS). Un-like vocational education, Chapter 1 uses a singleoutcome measure—the scores of participatingstudents on standardized achievement tests—toevaluate program effectiveness. Local schooldistricts (the recipients of Chapter 1 funds) mustconduct pretesting and post-testing of Chapter 1students and report the scores to the state inaccordance with a complex set of TIERS techni-cal guidelines. The state aggregates and reportsthe test scores to the federal government, whichfurther aggregates the results into a nationalpicture of Chapter 1 student achievement.

Chapter 1 evaluation data took on greatersignificance with the 1988 enactment of new

~~ Katherine p. Dick Inson C( a],, E\a/lffl/lOn {Jfthe Eficfts oj’JTPA Perji)rrnancc Standards on Clients, .$er~>lce.$, and COSIS (Washington, ~“

Nati(mal C(mmiss](m for Employment P(~llcy, 1988), p. 4.


program improvement provisions (Public Law100-297), which were spurred by concerns aboutstagnating program quality and were accom-panied by a loosening of process requirements inother parts of the legislation.so Under the 1988amendments, local school districts must conductannual evaluations of Chapter 1 student performa-nce and must implement program improvementplans in project schools that do not exceed stateminimum requirements for Chapter 1 aggregatestudent achievement. If, after a year, schoolperformance still does not improve, then the statebecomes involved through a joint improvementplan. The process continues until improved per-formance is sustained for more than 1 year.

Federal regulations have set a minimum stand-ard for annual test score gains, but states areencouraged to establish higher standards. Statesand local school districts are also encouraged toevaluate local programs on the basis of “desiredoutcomes’ other than standardized test scores,but there is little incentive to do so, since moreoutcomes mean additional hurdles for programimprovement. Chapter 1 also requires schools toexamine the progress of individual children and toconduct a thorough assessment of program needsfor children who do not meet minimum standards.

The Perkins Act program improvement provi-sions drew some key features from Chapter 1: theauthority for states to develop standards, therequirement for consultation with a committee ofpractitioners, and the process for local and stateprogram improvement plans. Unlike Chapter 1,however, vocational education standards do notrely heavily on a single measure and are not tiedto a national reporting and evaluation system.

CHANGING THEMESSeveral themes emerge from the legislative

history of vocational education that help to clarifycongressional intent about the new requirementsin the Perkins Act for accountability, show the

shifts in federal policy on accountability invocational education, and highlight issues likelyto arise during implementation or future re-authorizations.

The mechanisms for accountability in voca-tional education have changed substantively overtime, as the federal government has sought betterways to improve program effectiveness andachieve federal goals in a field with an increas-ingly strong state and local presence.

. In the early years of the program, from 1917to roughly 1963, accountability was en-forced largely through federal approval ofstate plans and federal monitoring of stateprograms.

● State planning peaked as an important ac-countability tool in 1976 with mandates formultiple plans and accountability reports.

. Since 1963, mandated reviews of the qualityof local programs and the access of specialpopulations to them has been a linchpin ofaccountability. Initially, these evaluationswere oriented to the state review of thequality of local programs using criteria otherthan student outcomes. The initial steptoward student outcomes was made in 1976,with”requirements for followup informationon job placement and employer satisfaction.The 1990 requirement for performance stand-ards sharpens the focus on student outcomes.

. Responsibility for conducting program re-views has shifted in recent years from statesto local recipients, bringing the activitycloser to those with the greatest stake in theoutcomes and the greatest likelihood ofusing them to revise programs. The 1990amendments require reviews of the qualityof local programs both by the states usingtheir performance standards measures andby local programs themselves. The localreviews are to be conducted annually.

X) U.S. Congress, House C(m}mittee on Education and Labor, ‘‘School Improvement Act of 1987,”H. Rept. 10095, May 15, 1987, p. 22.

Chapter 2: The Evolution of Performance-Based Accountability Requirements I 33

●

●

Through 1976-or through 1984, on someissues—federal procedural requirements andfunding set-asides became more numerousand detailed, In recent years, however, thebalance has shifted somewhat away fromthese mechanisms and toward outcome-oriented standards.Throughout the legislative history, the fed-eral government has frequently turned toquasi-independent bodies, such as national,state, and local advisory councils and theNational Center for Research on VocationalEducation to conduct evaluations, providetechnical support, and solicit business com-munity advice on vocational education.

1 Reasons for Shifts inAccountability Requirements

Several reasons underlie these shifts in theaccountability provisions of the federal legisla-tion. The reasons include persistent concernsabout the effectiveness and relevance of voca-tional programs, changing definitions of quality,state and local backlash against federal prescrip-tion, and the strong desire of Congress to maintainthe viability of the federal program. Still, tyingvocational education to student outcomes isdifficult because of its multiple goals, which arehard to measure, and variation in prioritiesaccorded them by states and local communities.

One of the most important reasons is that theaccountability requirements of the federal legisla-tion often do not seem to Congress to haveproduced the desired results. Concerns about theeffectiveness of vocational education programsand their relevance to labor market needs havepersisted through almost every reauthorization ofvocational education back at least to the Voca-tional Act of 1963. As one researcher noted in1979:

[After] sixty years of school programs andafter billions of federal, state, and local dollars, no

legislator, educator, or lobbyist can prove thatvocational programs do precisely what federallegislation promised. On the contrary, in the lastforty years blue-ribbon committees and govern-ment financed studies have pointed out repeat-edly serious shortcomings in vocational educa-tion allocations, operations, productivity, andimpact within schools.s’

Second, as the field of vocational education hasmatured, the definition of a quality program haschanged from infrastructure (e.g., adequate facili-ties, equipment, and professional staff), to equity(e.g., access for special populations), to moderni-zation (e.g., market-relevant courses and updatedcurricula), and eventually to student impacts (e.g.,job placement and competency attainment). Asdefinitions have changed, so have the means forenhancing quality. The specific, overall goal ofvocational education addressed most often inevaluation requirements in recent years is that ofpreparing students for employment. From aninitial directive in 1963 for evaluations based on‘‘manpower needs and job opportunities, ’ thelegislation has become more specific about howprogress toward employment goals should bemeasured. Other goals—such as dropout preven-tion and academic achievement—have not beentargeted for evaluation until quite recently.

Finally, it is telling that policy makers haveresponded to evidence of shortcomings in voca-tional education by devising new, different, orstronger accountability mechanisms rather thanby eliminating federal support or merging voca-tional education with job training, as some haveproposed. This decision suggests that legislatorsare committed to maintaining vocational educa-tion as a viable system and that they believe thefederal government can influence state and localpolicy even with a very limited share of federalfunding.


1 Balancing Accountability and FlexibilityFor much of the 1960s and 1970s, the federal

government relied on detailed process require-ments to ensure that human resource programsreached the intended beneficiaries and producedthe desired results. In the early 1980s, as state andlocal criticism heightened about burdensome,counterproductive, and overly prescriptive fed-eral mandates, Congress responded by relaxingrequirements in many programs, from the Com-prehensive Employment Training Act (CETA) toChapter 1.

Vocational education represented a somewhatspecial case in this debate. The early years of theprogram were characterized by a high degree offederal influence and dependence on federalfunding. During the 1960s, however, state flexi-bility actually increased, as the legislation elimi-nated some rigid funding categories and gavestates more discretion over which programs tosupport and how to use their federal grants.Federal efforts to exert stronger influence in 1968and 1976 were undercut somewhat, as the federalshare declined and states channeled their ownresources into programs that were not alwaysconsistent with federal priorities. In 1984, asCongress was relaxing requirements in othereducation programs, it continued to seek strongermechanisms for enforcing federal priorities invocational education. The results were mixed.

In 1990, Congress changed course and movedto require outcome-based evaluation in exchangefor increased flexibility. The tradeoff, however,was not as tidy as congressional rhetoric suggests.The 1990 amendments, though less prescriptivethan the 1984 Perkins Act, are still rather detailed.Although many process and set-aside require-ments were eliminated, other new requirementswere added, governing funds distribution, pro-gram content, and the integration of vocationaland academic education. However, the emphasison performance-based accountability has beencautious compared to the JTPA.

Through several reauthorizations, Congresshas also tried to balance tensions that arosebetween the goals of access and excellence and toensure that vocational education services tospecial populations were of high quality. The1990 amendments went several steps further byeliminating the set-asides and requiring high-quality services for special populations.

As the JTPA experience shows, performancestandards do not necessarily resolve the tensionbetween these two goals and may create newchallenges. Learning from the JTPA, the 1990amendments sought to build in safeguards toensure that standards and measures would includeadjustments for serving special populations andthat local evaluations would include a review ofthe progress of special populations.

1 Cautious ApproachAs noted above, the 1990 legislation took a

c~autious approach to performance outcomes in~~ocationa] education. The standards me state-

cleveloped, not nationally developed. There is noprovision for reporting local evaluations to thefederal government, nor are funding sanctionsattached to the results. The program improvementprocess is meant to be helpful, not punitive. Thereare also several provisions for additional research,technical assistance, demonstration, and datacollection regarding implementation of the per-formance standards.

In short, Congress built in several opportunitiesto monitor the progress of implementation i~ndkeep informed of difficulties that may arise. Thissuggests that the new accountability provisionsare conceived as a first step, to be reviewedcarefully before the next reauthorization. Thisdeliberate approach to performance standards isalso reflected in other programs, such as thephase-in periods for standards in the JOBS andFood Stamp Education and Training programs.

State

Testing andAssessment 3

A s with any new policy, the ultimate effects of the shift infederal policy on accountability to performance stan-dards will depend on how it is implemented in state andlocal programs of vocational education. The role of the

states will be pivotal in the implementation process because oftheir responsibilities for setting the performance standards andadopting policies of testing and assessment measuring progress.

Beyond the initial definition of the performance standards andmeasures for a state, successful implementation will require thedevelopment of substantial state resources for assessing theacademic and occupational skills acquired by students. In manystates, new resources will be needed.

As indicated in chapter 2, the legislation requires the adoptionof outcome measures for learning and competency gains, and atleast one other area of competency attainment or employmentoutcomes. Most states have adopted sets of standards andmeasures in at least four or five areas, involving some combina-tion of gains in or attainment of academic skills, gains in andlorattainment of occupationally specific skills, attainment ofgenera! work or employability skills, rates of program comple-tion, rates of job placement, and status of employment or furthereducation. ’ In most cases, all except the last three types of out-comes will require information about the performance ofindividual students from some form of testing or assessment. TheNational Center for Research in Vocational Education’s (NCRVE)recent tabulation of the standards adopted by the states shows

I Mikala L. Rahn et al., $‘ State Systems for Accountability in Vocational Education, ’paper prepared for [he U.S. Department of Education, office of Vocational and AdultEducation, December 1992.

35


that a majority of them at least will require someinformation from testing and assessment.2

Consequently, the eventual success of perform-ance standards in stimulating reform and im-provement in vocational education will dependon the resources for testing and assessment withinstate and local programs. If no resources exist orthe quality of the information from testing andassessment is low, conclusions derived will befaulty. The testing or assessment process mightnot be focused on the most important outcomes ofthe local programs, or the results may not bedependable because of the instruments employed,how they were presented to students or adminis-tered, or how the responses of students were ratedor scored. The problems could be random, inwhich case they might not be threatening to theintegrity of the resulting information, or theycould be systematic, in which case they would bea problem. Evidence from academic educationshows that when the stakes for testing andassessment are sufficiently high, the process ofinterpreting the results can become highly politi-cized and the results can be distorted.q

This chapter presents the results of a “firstcut’ effort by the Office of Technology Assess-ment (OTA) to describe the policies of states ontesting and assessment for the academic andoccupational skills of vocational education stu-dents, and the state plans for expanding theirpolicies on testing and assessment by 1995. Thedata presented come from a survey conducted byOTA, along with interviews with state personneland others who are knowledgeable about prac-tices of testing and assessment in vocationaleducation.

The results presented serve three main pur-poses. One is to describe the range of currentpractices of testing and assessment in vocationaleducation. Partly, this is done through comparing

current practices with practices of testing andassessment in academic education. There areclear differences between the philosophies andorigins of testing and assessment in academic andvocational education that are important to keep inmind. The second purpose is to describe the plansof states for expanding their resources for testingand assessment in response to the requirementsfor performance standards in the Perkins Amend-ments, and other forces at work at the state andlocal levels in vocational education. The third isto consider the correspondence, or lack thereof,between the emerging policies of states on testingiind assessment and two reform goals: integratingiicademic and vocational education, and broaden-ing the technical skills around which vocationaleducation is organized.

The testing and assessment policies of statesare described in terms of two dimensions. Onedimension is the four types of academic andoccupational skills, and the other is the extent towhich methods of written testing or assessmentare emphasized. These categories of academicand occupational skills are important becausethey are related to the integration of academic andvocational education, and broadening the techni-c~al skills around which vocational education isorganized. Whether the state policy emphasizeswritten testing or assessment is important becausec)f the potential effects on the content andcharacter of instruction in vocational education,and because of validity and reliability issues.

For the purposes of this study, wrilten Ies/ingwill be defined to be any method of examiningstudents in which the format of the answers to thequestions asked are multiple choice, matching, orsome other method of filling in a small blank orselecting the correct response from a given list ofresponses. Such measuring instruments areclosed ended. Assessmen/ will be defined as the

z Ibid., table 2.

3 U.S. Congress, Office of Technology Assessment, 7’esring in American Schoo/s: Asking (he Right Quesrion.s, OTA-SET-5 19 (Washington,DC: U.S. Government Printing Office, February 1992), ch. 2; and Daniel Koretz, “Arriving in Lake Wobegon: Are Standardized TestsExaggerating Achievement and Distorting Instruction’?” American Educator, w)]. 95, No. 2, 1988, pp. 46-52.

Chapter 3: State Policies on Testing and Assessment I 37

observation and rating or judging of studentperformances to ascertain academic or occupa-tional skills, following some systematic proce-dure of measurement.

DESIGN AND METHODOLOGYOF THE STATE SURVEY

The OTA survey was conducted by telephonefrom February through April of 1993. It had twoparts: 1 ) a series of closed-form questions toobtain basic descriptive information about theindividual components of each state’s program oftesting and assessment, and 2) further discussionwith the state respondents to develop a brief,written description of the nature, form, andpurposes of each of the components and how theyare related to each other. Both the coded question-naire responses and the qualitative descriptionswere sent back to each state for verification of theaccuracy and completeness of the information.The respondents were the state directors ofvocational education in each of the 50 states andthe District of Columbia, or a person designatedby the state directors. All 51 responded. Ques-tions were asked only about secondary vocationaleducation.

The survey was organized around the collec-tion of data for each of the major, individualcomponents of each state’s program of testingand/or assessment, not the program as whole. Ina state with a well developed program of testingand assessment, these components could be, forexample:

1.

2.

a state policy or guideline of requiring localprograms to monitor the academic achieve-ment of all students who complete a voca-tional program through the administrationof a commercially available test, such as theTest of Adult Basic Skills (TABE);a state policy or guideline of making acompetency-based test item bank available

3.

to local programs and strongly encouragingthem to construct their own tests for meas-uring the vocational skill attainments of allstudents on completion of vocational pro-grams; anda state policy of strongly encouraging localprograms to provide all students who com-plete vocational programs with a profile orcumulative portfolio showing their accom-plishments and competencies to use inseeking employment,

Data was collected by components to be asprecise as possible about the nature and extent ofeach state’s policies of testing and assessment,and plans for the expansion or contraction ofthose policies through 1995. These componentsof testing and assessment are the basic unit ofanalysis of the study.

Although the data describe only the policies ofstates on testing and assessment, the informationalso appears to provide a reasonably accuratepicture of policies and practices at the local levelin many states. Local conformance is mandated insome states and in many others there are strongtraditions of following the lead of the stateagency, even if policy is not mandated.4 In theremaining states, the range of local policies andpractices is broad.

The survey was designed by OTA to describeall components of each state’s policies andpractices of testing and assessment, rather thanonly the components employed for implementingperformance standards and measures. This pro-vides a more accurate basis for describing prac-tices of testing and assessment in vocationaleducation, and examining the effects on thesepractices of the requirements for performancestandards in the federal legislation. Testing andassessment are conducted for many reasons otherthan performance standards. Insofar as possible,the information collected by OTA was compared

4 Lawrence Cuban, “Enduring Resiliency. Enacting and lrnplementing Federal Vt~atitmal Educati(m hgislati(m, ” Work, Youth, and.%hoo/ln<t: I//r/or/(a/ Per.~pcttl\e.~ on Vocatiwmli.sm in Ameri(wn Education (Slanf(~rd, CA: Stanford University Press, 1982), pp. 45-78.


item-by-item with the data on the plans of statesfor implementing performance standards andmeasures that have been collected by NCRVE.5Followup calls were made to resolve all differ-ences. All data was obtained from the states bytelephone but then sent back to them for theirreview and explicit approval.

For each component, questions were asked inthe telephone survey and interviews to determine:

●

●

●

●

●

●

What skills are assessed within the fourbroad categories of academic, vocational,generic workplace, and/or broad occupa-tional skills.How information resulting from testing andassessment is used by the states.Where in the curriculum testing or assess-ment is conducted (e.g., by grade level, inintroductory vocational education courses,or at the end of a sequence of occupationallyspecific vocational courses).Whether all local programs are required as amatter of state board policy or legislation toconduct testing and assessment or are onlyencouraged by the state to conduct it.What resources for testing and assessmentare made available to local programs by thestate as part of the state’s policy on testingand assessment.What the state’s plans are for expanding,contracting, or adding to their program oftesting and assessment through 1995-96.

Practices of testing and assessment in voca-tional education have not previously been thesubject of much research. OTA asked the obviousand most simple questions about testing andassessment practices in vocational education in

order to provide basic descriptive information andraise policy questions. The most extensive exist-ing study is the recent NCRVE report on theimplementation of performance standards. TheNCRVE study lists the performance standardsthat have been adopted by the states and some ofthe measuring instruments that will be used.b Asurvey of state practices in competency-basedtesting has also occasionally been conducted bythe Research and Curriculum Unit for Vocational,Technical, and Adult Education of MississippiState University. 7 The most recent report de-scribes the states that have programs of competency-based testing for vocational skills and makecompetency-based tests or test item banks avail-able to their local vocational programs. TheNational Association of State Directors of Voca-tional Education has also recently completed aone-time survey of the industry skill standardsinitiatives of states that contains some informa-tion on assessment practices.8

O R I G I N S O F T E S T I N G A N D A S S E S S M E N T

P O L I C Y I N V O C A T I O N A L E D U C A T I O N

The origins and current practices of testing andassessment in secondary vocational education aredifferent from the rest of education in severalimportant respects. The phrase “testing andassessment’ that has been repeatedly used abovebegins to reveal some of these differences.Roughly speaking, there are two related traditionsof testing and assessment in vocational education.These two traditions have common origins in thecompetency-based movement. They differ in theemphasis placed on the need for written testing toprovide reliable measurement, as opposed toallowing the use of a broad range of methods of

s Rahn et al., op cit., f(xmwte 1, appendix.

b Ibid.7 National Network for Curriculum C(x)rdination in Vocational Technical Education, Research and Curriculum Unit for Vocational,

Technical, and Adult Education, Mississippi State University, “ 1990-91 State of the Art Report on Statewide Student Competency Testingin Vocational and Technical Education, ’ unpublished report October 1990.

X Barbara Border, Educution-Dri\’en Ski// S[andards Systems in rhe United States, prepared for the U.S. Department of Education(Washingt(m, DC: National Vocational Technical Education Foundation, October 1993).

Chapter 3: State Policies on Testing and Assessment] 39

observation and evaluation, some of which mayinvolve a substantial amount of judgment on thepart of instructors or others.

The written approach to competency testing invocational education involves the administrationof tests in which the questions are keyed directlyto specific items of knowledge and skill neededon the job, and the answers are provided by somemethod of checking off the correct response froma given list of responses. In vocational education,most of these written tests are matching ormultiple choice.

The range of competency-based methods ofassessment used in vocational education is broad.Included is the administration of carefully de-signed performance exercises, the summary eval-uation of projects undertaken by students, organ-ized events or competitions in which studentscompete for recognition and rewards, and eventhe subjective rating of regular classroom workwithin some framework of performance elements,rating scales, and rating procedures. There arealso longstanding practices of providing studentswith profiles and/or encouraging them to accumu-late portfolios of their schoolwork to use inseeking employment.

For the purposes of this report, assessment isdefined as a process where student responses to atask or variety of tasks over a period of time arecarefully observed and evaluated or interpretedaccording to some set of agreed on criteria ordimensions. The tasks may be presented to thestudent by the teacher or a test administrator in theform of specific “prompts” or statements ofproblems to be solved. They could alternativelybe initiated by the student in response to generalinstructions, Students may respond to the promptsor problem situations on demand-that is, theyare assigned tit a certain point in time and must beresponded [o within a limited period of time; orperformance may occur over an extended periodof time, as a part of the regular program ofinstruction and student work.

The differences between competency testingand assessment in vocational education parallel insome respects the current debate in academiceducation over the future of standardized testingand performance assessment. There are, how-ever, virtually no traditions in vocationaleducation of reliance on the kinds of norm-referenced, standardized tests of academicskills that are so prevalent in academic educa-tion. Testing in vocational education stems fromentirely different origins than standardized testingin academic education. Competency testing invocational education stems from the competency-based movement in vocational training, whilestandardized testing in academic education isdescended from the mental testing movement thatbegan in psychology around the turn of thecentury and has resulted in the concept of abilitytesting.9

One of the major differences between thecompetency-based, written testing done in voca-tional education and the standardized testing donein academic education is that in vocationaleducation the tests are constructed to measurewhether students have the skills needed to per-form particular jobs, rather than how their per-formance compares with other students taking thesame test. In the language of test theory, thewritten tests in vocational education are criterion-referenced rather than norm-referenced.

In properly conducted competency-based vo-cational education, both the content of the testsand the curriculum of instruction are criterion-referenced—that is, they are derived from analy-ses of the tasks actually performed by peopleworking in specific occupational areas. Therelationship between the job tasks identified andthe content of instruction and individual test itemsis close.

Competency testing and assessment are a keyaspect of the concept of open entry/open exit thatis followed in many vocational programs. Thismethod of or~anizin~ instruction is basic to the


competency-based approach, but not all programsare organized in this way. In open entry/open exitinstruction, students are tested or assessed whenentering programs to determine their skills. Theythen work to acquire the particular skills requiredto achieve their employment goals; they leave theprogram when they have demonstrated mastery ofthose skills, according to the results of tests orassessments. Students learning alongside eachother may be learning very different skills inhighly individualized programs of instruction.

The traditions of competency-based assess-ment in vocational education are older thancompetency-based testing but stem from the sameorigins in the competency-based movement. Thespecific lists of job competencies are employed ascontent and performance-level standards for ob-serving and recording the capability demon-strated by students in different kinds of assess-ments. The content standards define the skills thatstudents should demonstrate. There is also usu-al] y a categorized scale of performance levels thatdefines degrees of proficiency in performance.These performance levels are defined accordingto a scheme showing that the student is, forexample, ‘‘skilled, “ ‘‘moderately skilled, ” “un-skilled, ’ or ‘‘has not been exposed’ to a particu-lar task or sets of tasks.

The philosophy and methods of both compe-tency testing and competency assessment are thuswholly different from the academic tradition ofwhole class, teacher-dominated instruction withtesting at fixed points in the curriculum. Skillslearned and not time spent are what drives thepace of instruction and assessment. Testing andassessment are not after the fact, external proc-esses of inspection but integral parts of theprocess of education. In all of these respects, thetraditions of testing and assessment in vocationaleducation resemble what is being advocatedelsewhere in the rest of education to replace

standardized testing with alternative forms ofassessment.

Whether the written methods of competencytesting are any more or less reliable and more orless valid than the methods of assessment used invocational education is impossible to say merelyaccording to differences between the methodolo-gies themselves, The critlcal issue with assess-ment is the comparability of judgments madeacross instructors and from program to program.This comparability concerns both the ratings orevaluations given for similar performances andthe level of performance considered sufficientlyhigh to meet the standards. With sufficient effortsto develop consistency in rating processes[hrough procedures of training and group judging,i~nd to utilize available techniques for statisticallychecking on the consistency of ratings, highlevels of consistency in raters’ judgments can beachieved in performance assessment.

With written testing, the two most criticalissues are the relevance of what can be measuredto capabilities for performance, and the long--runeffects of closed-form testing instruments on thecontent and methods of instruction. Written formsof testing are generally thought to be best formeasuring factual knowledge and certain formsof reasoning. The “know-how” and capabilitiesfor more complex and extended performance thatare critical in the workplace (and in life) cangenerally not be measured as well with writtenforms of testing. Written tests suffer from the factthat the knowledge and powers of reasoning thatcan be measured with the greatest internal consis-tency of individual test items, which is thenecessary criterion of a sound written test, typi-cally do not include some of the most importantcapabilities for occupational preparation.

The crucial point is to not assume that methodsc)f written testing are sound simply because oftheir written format and that methods of asscss-

I o ~e nlajor source of va~a(ion in perfomlance assessment has generally been found to be task variety rather than inconsistency in raters’

judgments. See, for example, Richard Shavelson, “Generalizability l’heory and Military Perfomlance Measurements: lndiv~dualPctftJm~ancc, ” Perjbrnwnce A.$sessmenl in I}W Work P/ace (Washington, DC: National Academy Press, 1991 ), pp. 207-257.

Chapter 3: State Policies on Testing and Assessment 141

ment are unsound simply because they involveelements of judgment in scoring performances.Much closer investigation of the consistency andrelevance that are possible with different forms oftesting and assessment, and how they are con-ducted in vocational education, is needed beforeconclusions can be drawn about which methodsare best for what purposes.

It is important to point out that assessment invocational education may include written testingas one of several methods of measurementemployed. The emphasis in assessment is onusing different methods of measurement fordifferent kinds of knowledge and skill rather thanheavily emphasizing written testing.

DIFFERENCES AMONG THE STATES INPOLICY ON TESTING AND ASSESSMENT

One major difference among the states is theemphasis that they place on written testingcompared to assessment. Many advocates ofcompetency-based methods of testing and assess-ment have encouraged the development of stateresources for written competency testing for sometime; certain states are much further along in thedevelopment of this capability than others. TheVocational-Technical Education Consortium ofthe States (V-TECS), described in chapter 4, hasbeen one of the important results of this develop-ment.

Another important difference among the statesis the extent to which local programs are required,or mandated, to follow the policy on testing andassessment or are only encouraged to follow it asa matter of state board policy, state administrativepolicy, or state legislation. A few states have nopolicy or program of testing and assessment forvocational skills but most do, and either require orencourage local programs to follow it.

Using these two dimensions of difference, the50 states and the District of Columbia can begrouped into four different types of environmentsfor testing and assessment:

1.

2.

3-.

4.

A

States that encourage written testing foroccupational skills in local programs, in-cluding competency-based or other forms ofwritten testing. The encouragement of test-ing in most of these states is strong.Stutes that mandate assessment for occupa-tional skills in local programs withoutspecifying what methods should be used.These states typically encourage the use ofmultiple methods of assessment and testingin local programs.States that encourage assessment for occu-pational skills in local programs withoutspecifically encouraging one method overothers. These states also typically supportthe use of multiple methods of assessment.The encouragement given to assessment inthese states is generally not strong.States that have no specific policy orprogram of encouragement or requirementfor either the testing or assessment ofoccupational skills.

fifth category of “mandated testing” turnedout to include only two states in the 1992-93school year so it was combined into the firstcategory (states that encourage testing). One ofthese states is a small state that mandated oneform of testing for occupational skills in 1992-93and the other is New York, which has two typesof statewide tests for occupational competencies.

The fact that only two states have mandatorytesting shows one clear difference already be-tween vocation education and the rest of ele-mentary and secondary education. In academiceducation, most states have a large, statewide,mandatory program of testing for academic skills.For example, there were 39 states where anorm-referenced, standardized, written test ofacademic skills was administered statewide to allstudents in one or more subject areas (e.g.,


mathematics) in the 1992-93 school year. ] 1 Innearly all cases, these tests are administered at acertain grade level in those subject areas. No suchmass, mandatory, statewide testing of all thestudents at a grade level or any other fixed pointin time is conducted in vocational education.Furthermore, in many states this mass testing isadministered by the state (or a contractor to thestate) rather than local school districts. In voca-tional education, most of the testing and assess-ment is actually conducted or administered bystaff of the local programs.

All 50 states and the District of Columbia wereclassified into the four categories defined aboveusing data from the survey of states conducted byOTA. As shown in figure 3-1, 18 states areclassified’2 as encouraging testing for occupa-tional skills in 1992-93, 15 as mandating assess-ment for occupational skills, 10 as encouragingassessment of occupational skills, and 8 hadeffectively no policy on vocational testing orassessment.

The 18 states in the first category of encourag-ing testing enrolled about one-half of all highschool students. The 15 states in the secondcategory of mandating assessment enroll aboutone-quarter of all high school students. The lasttwo categories each enroll about one-eighth of allhigh school students.

It is important to point out that the statesclassified as having no policy on testing orassessment for occupational skills may still haveadopted performance standards and measuresbased on other kinds of information, such as ratesof program completion or placement. Some ofthese states also chose to meet the performancestandards requirements in the 1990 amendmentsby adopting a policy of allowing local programsto use their own performance standards andmeasures, which were then considered to be thestate’s performance standards and measures.

Figure 3-1: State Policies on Testing andAssessment for Occupational Skills (in 1992-93)a

Encourages testing718

Mandates assessment 15

Encourages assessment 10

Had no policy on test- ~

ing or assessment

Io 5 10 15 20

Number of statesa Flfly states pILJS the Dlstrlct Of Cdumbm.

SOURCE: Off Ice of Technology Assessment, 1993.

STATE RESOURCES FOR TESTINGAND ASSESSMENT

States encouraging testing for occupationalcompetencies differ substantially in the extent ofencouragement they provide, but there is no wayof distinguishing among them using the datacollected in the OTA survey. At one extreme arestates like Oklahoma, where substantial invest-ments have been made in competency testing andassessment in over 100 job-specific areas over anumber of years. Oklahoma even has its own testdevelopment center, the Oklahoma OccupationalTesting Center (OOTC). The state makes threekinds of instruments available to local programs:

1.

2.

tests for measuring the gains in the specificoccupational competencies taught in spe-cific courses,tests for measuring the levels of compe-tency achieved by students who completesequences of courses, and

I I Nofih Central Regional Education ~bt)ratt)ry, State Student Assessment frograrn Database, /992-93 (Oak Br(x)k, IL: Regional Policy

Information Center, September 1993), table 3.7, p. 141.11 states are ~]a~sifjed on the basis of their major compment of testing and assessment fOr OWupatiOnal skills.


Figure 3-2: State Resources for Testing or Assessment of Occupational Skills (in 1992-93)

In 18 states that encourage testing151

I12-

9-

6

3

0

13

3

-ElNumber of states

In 10 states that encouraqe assessment7-,

6i

5-

4 {

2-

5

Number of states


3. profiles displaying those competencies thatare given to students when leaving localprograms to use in seeking employment.

At the other extreme is a state that only makes asingle, state-developed test of ‘core occupationalcontent’ available to local districts to use as theywish.

Further differences among the four types ofstates can be illustrated by showing the kinds ofresources for testing and assessment they provideor strongly recommend to local vocational pro-grams. As shown in figure 3-2, four types ofresources are m-ovided:

In 15 states that mandate assessment

8{

4{ 3

4

5

h

3

Number of states

Resources provided by states to local programs:

m

n

m

Test-item bank

Other competency tests

Student profile or portfolio

Competency-based curriculum materials

No testing or assessment resources

1. Competency test item banks. Using these

2.

3.

test item banks, local programs construct orwrite their own criterion-referenced compe-tency tests reflecting the specific needs oftheir own programs of instruction.Other forms of competency tests. These maybe developed by the states themselves,purchased from vendors such as the Na-tional Occupational Competency TestingInstitute, or obtained from industry groupsfor use in certifying students for particuktrjobs.Student projiles or portfolios. These re-sources are used to provide students with a


4.

means of reflecting on their own progressand communicating their accomplishmentswhen seeking employment, further educa-tion, or for other purposes.Competency-based curriculum materials.These materials frequently include testinginstruments, assessment ideas, assessments,written testing, and/or checklists of compe-tencies that are suitable for purposes of bothteaching and conducting assessments. is

As indicated in figure 3-2, all 18 of the statesthat ‘‘encourage testing’ either recommend theuse of, or make available to local programs, acompetency-based testing instrument or test itembank. Some of these states also provide the othertwo types of resources for assessment, but notnearly to the same extent as states that mandateassessment or encourage assessment. On the otherhand, only four of the states that mandateassessment make resources for competency-based testing available to local programs. Thesestates apparently do so without specifically en-couraging the use of testing over alternative formsof assessment. Both the states that encouragetesting and those with mandatory assessment tendto place a strong emphasis on competency-basedvocational instruction and embed assessment inthe instructional process. These two types ofstates differ, however, in the extent to which theystress the need for written methods of testing, asopposed to more qualitative methods of assess-ment. In the states with mandatory assessment,local programs are required to conduct some formof assessment, but may use methods that best suitthe philosophy and needs of their program.

The extent to which local programs are compe-tenc y based differs among the four types of states.As shown in figure 3-3, states that mandateassessment tend to report higher percentages ofcompetency-based programs than do states that

encourage assessment or even those that encour-age testing. This is true even among the subset ofnine states where extensive test item banks forcompetency-based instruction have been devel-oped. It may be that competency testing is beingused in these states as a mechanism for forcinglocal programs to employ competency-basedinstruction and it is not working as well as themethods that are being used in states that aremandating assessment.

In fact, states with no policy of testing orassessment for occupational skills appear to havenearly the same incidence of competency-basedinstruction as states that encourage testing. Thisfourth group of states includes about equalnumbers of very small states with no policy ontesting or assessment for occupational skills, andlarger ones where significant statewide educa-tional reforms are under way that involve voca-tional education. (The number of states in thefourth category is very small, so that the apparentsimilarity with the first category may not besignificant.)

WHAT SKILLS ARE ASSESSED?One of the most important aspects of the 1990

amendments is the substantive priorities for thereform of vocational education. Chief amongthese reforms is the integration of academic andvocational education, or combining the teachingof academic and vocational skills in the schoolcurriculum. A second reform in the legislation isto broaden the occupational skills around which\’ocational programs are organized. This priority,which will be called organizing vocational educa-tion around broad ?echnical skills, is less welldeveloped in the legislation than the priority onintegrating academic and vocational education,but is evident in the language of “appliedtechnology” and teaching “all aspects of indus-try’ ‘ in vocational programs that appears in the

13 Because Of {he way that [he questions were asked in the OTA survey, the data in figure 3-2 represent the main resources provided 10 kal

programs rather than all of the resources provided.

14 The Iestlng itenls in figure 3-2 include ‘ ‘Test-Item Banks” and ‘*Other Competency Tests. ”


Figure 3-3: Use of Competency-Based Curricula in Local Programs (in 1992-93)

- In 18 states that encourage testingIu

8

6

4

2

0

6

5

4

3

2

1

0

8

I1 0 0

Number of states

In 10 states that encourage assessment

o

5

3I

2

0

Number of states

8

7

6

5

4

3

2

1

0


4

7

3

1

0

Number of states

In 8 states with no policy on theassessment of occupational skills

4

3

2

1

0

2 2

3

1

0

Number of states

Percentage of local programs that are competency based:

D All 50-100 p e r c e n t D L e s s t h a n 5 0 p e r c e n t m N o n e _ N o t a v a i l a b l e

SOURCE: Office of Technology Assessment, 1993.

1990 amendments. It is also evident in the requestfrom Congress for information from OTA aboutthe availability of instruments for assessing thebroad technical skills of vocational students. Athird kind of reform that has been recommendedby various outside commissions and studies isthat preparation for work in vocational educationand all kinds of training programs should be

organized around the development of what in thisreport are called generic ~wrkplace cornpetenciesor skills.

A major new development is the plan to createa National Skill Standards Board in the Goals2000: Educate America legislation currently beingconsidered by Congress. This calls for broadeningthe technical skills taught in vocational and other

I f Anthony carIWXi]~ ‘i al. ! Workp/ace llLIsIts (San Francisco, CA: Jossey Bass, 1990); and Secretary’s Cf)nm~issi(m tm AchievingNecessary Skills, Whal Work Requ~rc.s oj Sthool.s (Washingt(m, DC: U.S. Government Printing office, June 1991).


kinds of work-related education and trainingprograms. The board would do this by definingthe skills required by industry in “broad occupa-tional areas” of the economy and supporting thedevelopment of a national system for certifyingthe competence of individuals in those areas. Thegoals reflect recommendations that have beenmade by a number of outside commissions andstudies. ‘b

These proposals for the reform of vocationaleducation (and other forms of skill training andeducation) provide a skills framework for describ-ing the kinds of change that are occurring in thetesting and assessment programs of states.Change in the types of skills being tested for orassessed by the states provides some indication oftheir priorities for the reform. The four types ofskills are:

1{

2,

3,

4.

Vocational skills, which consist primarilyof job-specific skills determined throughjob analysis and other tools of competency-based vocational education.Academic skills, which, among the statevocational testing and assessment programssurveyed, are primarily reading, writing,and mathematics.Generic workplace skills, which are of twotypes: a) employability skills, such as workattitudes and knowledge of how to find ajob; and b) workplace competencies, suchas ability to work in teams, communicateinformation, solve problems, and manageresources.Broad technical skills, which are the coreskills and understandings of technology,information, and organization, and evenhistory, needed to perform effectively withina range of occupations in an industry, suchas manufacturing, finance, or hospitality.

The first, third, and fourth of these types ofskills together (vocational skills, genericworkplace skills, and broad technical skills) willbe called occupational skills.

The number of states with a component oftesting and assessment for each of these four typesof skills is shown in figure 3-4. The percentagesof states having at least one component of testingor assessment are very similar for each type ofskill—that is, the percentage of all states in thefirst category that test for or assess academicskills is about the same as the percentage of statesin the second and third categories of states thattest or assess for academic skills, and so forth. All43 states in the first three categories of states haveat least one component of testing or assessmentfor vocational skills. In the first three categories,31 states have at least one component of testing orassessment for academic skills. Seven states inthe first three categories have components oftesting or assessment for generic workplaceskills. Only one state in the first three categctriessupports testing for broad technical skills. Instates with no policy of testing or assessment foroccupational skills (the fourth category), onlyacademic skills are assessed.

It is important to point out that the 1992-93school year was actually the first year in whichstates were expected to operate their new per-formance standards under the 1990 amendmentsto the Perkins Act. This affects mainly the numberof states with a component of testing or assess-ment for academic skills in subsequent years. Ifthe baseline year of the OTA survey had been 1year earlier, the number of states conductingtesting or assessment for academic skills would inall likelihood have been much smaller than the 31found in 1992-93, while the number conductingtesting or assessment for vocational skills would

lb ConlnllSSlon” on the Skills of the American Workforce, High Skills or Luw Wages! (Rochester, NY: Nati(mal Center on Education andthe Economy, 199 I); and Gene Bottoms, Redesigning and Rejixusing High Schou/ Vocationa/ Smdies (Atlanta, GA: Southern Regi(malEducation Board, 1993).


Figure 3-4: Skills Included in Current State Policies on Testing and Assessment (in 1992-93)

In 18 states that encourage testing20 ~

181 8

1 6

1 4

12

10

8 -

6

4

2

0

14

3

11

Number of states

In 10 states that encourage assessment1 2 ~

10

8

6

4

2

0 I

10

7

Number of states

SOURCE. Off Ice of Technology Assessment, 1993.

0

have been about the same as shown in figure3-4.17 The reason is simply that most statesessentially employed their existing policies oftesting or assessment for vocational skills inimplementing performance standards. Many fewer

18

15

12

9

6

3

0


15

10

o

Number of states

In 8 states with no policy on theassessment of occupational skills

8

6

4

2

0

mm

4

Vocational skills

Academic skills

Generic workplace skills

Broad technical skills

Number of states

states initially had any capabilities in place fortesting or assessing academic skills. ’8 This indi-cates a substantial response by the states to therequirements of the 1990 amendments for per-formance standards.

I T In ~1 I ~~) I surk ~y. [he National Center for Research in Vocational Education found that 24 percent of states had previously used specific

W>rf(mnancc standards and metisures for academic skills at some time in the past. E. Gareth Hoachlander, f’ cr@mIarrcc Mc(]surcs a nd Smrrdardsjor \’o{fJ//fm~i/ ~ldufa[/~M: /99/ .$ur~ey Re.su/rs, MDS-388 (Berkeley, CA: Nati(mal Center for Research in Vocational Educatitm, January1 992), figure 1.

I h ~c nLITllber ~}f S[:ites With a Statewide acad~nl]c testing program is large but the number of stale \ ocational education WJCnCICS that w~r~

rcgul:lr!y obt:iln]ng data fr~lnl those programs for w)cati(mal students was und(mbtedly small.


WHERE IN THE CURRICULUM ARETHE SKILLS ASSESSED?

A second way of describing the testing andassessment policies of states is to ask where in theschool curriculum the different types of skills areassessed and for what reasons. The answers shedsome light on relationships between performancestandards and reform.

The survey data collected by OTA show thatthe testing and assessment for academic skillsincluded in state programs is highly separatedfrom the testing and assessment for occupationalskills. This separation appears to be undercuttingthe priority in the legislation on integratingacademic and vocational education. The separa-tion occurs because of major differences in themethods of testing and assessment being used formeasuring academic and vocational skills.

As indicated in figure 3-5, most states havechosen to measure the academic skills of voca-tional students through obtaining test results fromeither their statewide academic testing program ortheir statewide minimum competency exit exami-nation. Twenty-six of the 40 components oftesting or assessment for academic skills in1992-93 used these measures. These 26 compo-nents cover 25 of the 35 states where academicskills were measured in 1992-93, or 71 percent ofall states. Fifteen of these 26 components useresults from the regular statewide program ofacademic testing or assessment, and 11 use theminimum competency exit exam. The remaining14 of the 40 components of testing or assessmentfor academic skills are much more closely tai-lored to students’ patterns of enrollment invocational and academic programs.

The problem with relying on exit exams andstatewide academic tests is that they are adminis-tered centrally without any relation to whenstudents enroll in vocational education, whiletesting and assessment for occupational skills isdone locally and is closely tied to individualcourses or the completion of vocational andacademic courses.

Figure 3-5: Location of Testing and Assessment in~he Secondary School Curriculum (in 1992-93)

Number of testing and assessment components

Exitexam

Gradelevel L------

End ofsequence

Bycourse

Ongoing 22

0 5 10 15 20 25

NOTE: occupational skills include vocational, genenc workplace, andbroad techmcal skills.


Furthermore, the bulk of this academic testingoccurs in the 9th, IOth, and 1 lth grades, althoughsome of the academic testing occurs in the 12thgrade. (Some of it actually occurs as early as the8th grade.) Minimum competency exit exams aretaken for the first time in most states in the 9th orIOth grade and students keep taking them untilthey pass in order to graduate with a diploma.

In contrast, very little of the testing andassessment for vocational skills among the totalof 55 components of testing and assessment foroccupational skills shown in figure 3-5 occurs inthe 9th or IOth grades. Most of the testing andassessment is conducted in the more advancedoccupationally specific coursework that stuclentstypically take in the 1 lth and 12th grades. Only 6of the 55 components shown cover vocationalcourses at the general introductory level, whichare typically taken by students in the 9th or 10thgrades. The remaining 50 components are foroccupationally specific vocational education.

Another difference between academic andoccupational approaches is that vocational skillsmeasurement is much more closely related tospecific courses and sequences of courses in the


vocational curriculum than is academic testing.As shown in table 3-1, in the states that encouragetesting for vocational skills, the most frequentplace in the curriculum for testing and assessmentis at the end of a sequence of occupationallyspecific courses. This is the case by a substantialmargin over all of the other possibilities inpercentage terms. The second-most frequent pat-tern is to test or assess for skills in relation tospecific vocational courses. These two uses oftests and assessment appear to reflect the closerelationship between testing and competency-based vocational instruction that exists in stateswith strong policies of testing or assessment forvocational skills. The purpose of the testing andassessment at the end of a sequence of courses isgenerally to determine what students havelearned.

The third basic pattern is ongoing testing andassessment for vocational skills, which meansthat local programs are encouraged to conducttesting and assessment as an integral part ofinstruction, with perhaps summative assessmentsor other forms of evaluation at certain points,without any specific guidelines on when theassessment should take place or for what specificpurposes. Especially in the states with policies ofmandatory assessment, this ongoing assessmentis typically done within a framework of competency-based vocational education. The emphasis inthese states is on encouraging local programs toconsider multiple approaches to assessment andto develop their own programs.

The policies of testing and assessment foracademic skills in these same (mandatory assess-ment) states are similarly ongoing, as shown intable 3-1. Few of these states have formalstatewide testing programs like the states that areobtaining information about academic skills bygrade level or from a minimum competency exitexam. How frequently local programs in thesestates are finding ways of linking academic andvocational assessment in their ongoing programsof testing and assessment is impossible to tellfrom the data OTA collected.

The potential for assessing academic andvocational skills together is much greater in thestates where the policies of testing and assess-ment for both academic and vocational skills arehighly localized than in states where the data arebeing obtained from a centralized, statewideacademic testing program. None of the states inthe two categories of ‘‘mandatory assessment’and ‘‘encourage assessment, ’ where most of theongoing testing and assessment occurs, indicatedthat a priority has been placed on the integra-tion of academic and vocational skills in thepractices of testing and assessment. They maybedoing this but it is not possible to tell from theOTA survey.

There are only 14 components in 12 stateswhere close course-level connections are appar-ently being drawn between academic and occupa-tional skills assessment. These are shown in thetop portion of columns three, four, and five oftable 3-1. There are three basic patterns of testpolicy and use among these components. One isthat a commercially available academic test, likethe Test of Adult Basic Skills, is being directlyadministered to vocational students as they enteror complete specific vocational courses or se-quences of courses. Gain scores are obtained in afew of these cases by correlating test scores fromprogram or course completion with scores on thesame test taken at an earlier point in the student’sstudies. This takes a substantial degree of coordi-nation. The second pattern is states where aca-demic skills have been incorporated into the listsof occupational competencies around which test-ing or assessment for occupational skills is beingconducted. There is only one state (Oklahoma)where a sufficient priority has been accorded tothis strategy to have indicated it on the question-naire. However, pilot projects to accomplish thissame strategy are currently under way in severalother states. The third pattern is states where testsfor academic skills that are contextualized tospecific occupational areas have been developed.Arizona’s Academic Test Item Bank (ATIB) isthe clearest example of this. The ATIB is a


Table 3-1: Location of Testing and Assessment for Vocational and Academic Skillsin the Secondary School Curriculum in 1992-93

(total number of testing or assessment components by type of state)

Location in the curriculum

Minimum competency Sequence of By vocationalType of skills and state exam By grade Ievei vocational courses course Ongoing

Academic skills:Testing encouraged . . . . . . 4Mandatory assessment . . . 2Assessment encouraged . 3No current policy . . . . . . . . 2

Total . . . . . . . . . . . . . . . . 11

Occupational skills:Testing encouraged . . . . . . 0Mandatory assessment . . . 0Assessment encouraged . 0No current policy . . . . . . . . 0

Total . . . . . . . . . . . . . . . . 0

6522

15

00000

31116

14610

21

20002

9210

12

13206

41080

22

NOTE: The total numbers of components for academic and occupational skills are 40 and 55, respectwely.


resource bank of written test items for academicskills that are criterion-referenced to occupationalskills and set in vocational problem situations.

Problems of integrating academic and voca-tional skills testing and assessment will bedifficult to overcome in states that rely on testscores from their large-scale, statewide standard-ized testing programs for the assessment ofacademic skills. One problem with using thisstatewide data is that the grade Zevels whereacademic test score information is being collectedare different from the grade levels where testingand assessment for occupational skills is beingconducted. The fact that most of the academictesting is done before students take most of theirvocational education suggests that academic test-ing could turn out to be used primarily as a screenfor entrance into vocational programs, or as a prodto students’ teachers in earlier grades, rather thanas a means of verifying the academic skillattainments of vocational students when theyleave their vocational programs or for actuallyimproving the integration of academic and voca-tional learning.

There will also be difficult logistical problemsin matching the statewide test data with the highly

localized vocational testing and assessment on astudent-by-student basis. Merging or comparingpre- and post-test score data for the same studentfrom these two sources and relating any gains orlosses observed to the very complex patterns oflocal enrollment in vocational courses and pro-grams will be extremely difficult. The most likel yuse of the centralized academic data will be incoming to broad conclusions about the generallevels of academic performance of students invocational programs compared to other parts ofthe school curriculum. Whether the higher orlower scores of the vocational students are due tothe vocational curriculum, the success or lackthereof in integrating academic and vocationalinstruction, or processes within schools of chan-neling students into different curriculum areas inthe first place will be impossible to determine.The best solutions to these kinds of problems arelikely to lie in the decentralizing of testing andassessment for academic skills to local programsand embedding it in instruction. The challengewill be to find a way of doing this whilemaintaining a sufficient degree of comparabilityacross programs.


Why most of the states have chosen to importtest data from their large-scale, statewide aca-demic testing programs is reasonably clear. Thedata provide the simplest and cheapest means ofcomplying with the requirements for standardsand measures of performance ‘‘including aca-demic skills” in the 1990 amendments. All othersolutions are complicated and more difficult toimplement.

WHY ARE THE SKILLS ASSESSED?Another issue is how states are using informat-

ion from their programs of testing and assess-ment to meet performance standards require-ments. These could include using the results inmaking decisions at the state and local levelsabout the improvement of schools or programs,diagnosing student learning and modifying in-struction at the classroom level on a regular basis,monitoring student attainment of skills in coursesor programs, and certifying students’ capabilities.For performance standards, the issue is whetherthe information is actually being used to makedecisions about programs, and not just beingcollected in order to comply with the require-ments.

The purpose of the performance standards is tomake decisions about local programs, but it ispossible to simply collect information about localprograms and compute whether they are meetingthe standards without doing anything with theresults. In the OTA survey, the state respondentswere asked to indicate all of the purposes forwhich each of the separate components of theirtesting and assessment programs were being usedat the state and local levels. For each componentof their program of testing and assessment, staterespondents were asked to check off on thequestionnaire which of the following uses werebeing made of the information:

. T(J satisfy requirements for accountability,not including accountability or performancestandards under the Perkins Act.

● To satisfy requirements for accountabilityincluding accountability or performance stan-dards under the Perkins Act.

● To make decisions about the improvement ofprograms, courses, or schools at the state orlocal levels.

. To assess students for program or coursecompletion, certification, or progress inlearning.

Respondents were instructed to check all usesthat occur at the state and local levels with somefrequency. They checked an average of 2.07 usesper testing and assessment component for voca-tional skills and 1.98 uses per component foracademic skills.

One important finding from this question isthat the patterns of use are opposite for academicand occupational skills. For academic skills, themost frequent use of information resulting fromtesting and assessment at the state and local levelsis compliance with the requirements of the 1990amendments. The most frequent use of informa-tion about occupational skills is monitoring theskill attainments of individual students or certify-ing student accomplishments. This informationalso informs teachers on the effectiveness ofinstruction.

The second main finding is that the least likelyuse of the information for both academic andoccupational skills is making decisions about theimprovement of programs, schools, or courses.This is potentially a problem since performancestandards are supposed to be used in reviewingthe effectiveness of local programs.

Further indication of the seriousness of theproblem is the very low frequency with whichinformation from testing and assessment is beingused at the state and local level for reasons ofaccountability other than meeting the Perkinsrequirements. Information from testing or assess-ment is apparently not a very important basis foraccountability at the state and local levels. Thismeans that the test-based forms of accountabilityin the 1990 amendments are substantially differ-


ent from whatever the principal mechanisms ofaccountability currently are. Using informationfrom testing or assessment for purposes of ac-countability and making decisions about theimprovement of programs will be relatively newfor most states.

States do have considerable experience incollecting data on rates of program completion orfollowup rates of employment and earnings.Whether this information is merely collected oractually used for purposes of improving programsor making decisions about them is the importantquestion.

There are several possible reasons for the lowrates at which information from testing andassessment is being used for program improve-ment. One is that states were not required underthe legislation to link their performance standardswith the review of local programs until the1993-94 school year. Perhaps this year thefrequency with which information from testingand assessment is used for program improvementwill increase substantially. Another reason maybe that states simply do not have enough experi-ence yet to know precisely how information fromtesting and assessment can be used for makingdecisions about programs.

However, it may also be that the testing andassessment information simply turn out not to bevery useful for making decisions about programs.The issues in appropriately using informationfrom testing and assessment for making high-stakes decisions about programs are complex anddifficult. Data are easy to come by. Informationthat really means something and measures accu-rately the outcomes of programs is much moredifficult to obtain. Accurate information is evenmore difficult to obtain when the consequences ofproviding it to a state authority may be untoward.The examples given in the previous section aboutcoordinating academic and occupational test in-formation from different sources indicate howcomplicated and difficult the careful use ofinformation from testing and assessment will bein many cases.

It is crucially important to recognize that thedata presented in figure 3-6 can only be inter-preted qualitatively because of the way thatquestions were asked in the OTA survey. Forexample, the data cannot be interpreted to say thatinformation from testing and assessment foroccupational skills is used ‘‘x’ percent moreoften for working with students than is informat-ion about the academic skills of students. Theonly comparison that can be made from the datapresented in figure 3-6 is that testing and assess-ment are used more frequently for some purposesthan others, as discussed above, within the twocategories of skills. In order to measure use inabsolute terms, scales for recording perceptionsof the importance of different uses of informationfrom testing and assessment and combining themwould have to be developed.

The data were checked to see if the basic profileshown in figure 3-6 differs among the four typesof states. It does not. All four groups of states havebasically the same profile.

Some differences exist in how informationabout occupational skills is used depending onwhere in the curriculum it is generated. As shownin table 3-2, the frequency of different uses isabout the same for all three ways in which testingor assessment information is generated in thecurriculum (at the end of sequences of courses, bycourse, or through ongoing assessment). Themain exception to this is that information fromongoing assessment is significantly more likely tobe used for purposes of student assessment thanthe information generated through the other twomodes. Basically, the ongoing form of testing andassessment appears to be more student orientedthan the two other forms of assessment. As will beshown in the next section, ongoing testing andassessment also involves less use of tests andmore use of assessments than testing and assess-ment that is related to courses or sequences ofcourses. In short, testing appears to be oriented toaccountability and assessment to student learning.

a

ba)u)op

:

c

d

KEY


Figure 3-6: Purposes of Testing and Assessment in 1992-93

Occupational skills

Total number of components = 55r I

C2’36

4 8

1 1 1 1 1 10 5 10 15 20 25 30 35 40 45 50

Number of states

a = Used for account ablllty, not mcludmg Perkins

Academic skills

Total number of components = 40

aII

3

b 32a)v)on

0 5 10 15 20 25 30 35 40

Number of states

b = Used for accountablllt y, mcludmg Perlansc = Used for program, course, or school Improvementd = Used for the assessment of students for pqram or course completion, certification, or morutonng of Iearnmg progress


WHAT FORM OF MEASUREMENT IS USED?The extent of reliance on written forms of

testing as opposed to assessment differs greatlyamong the states, as shown in figure 3-7. Nearlyall of the testing and assessment for academicskills encouraged or required by the states is doneusing standardized, written tests; only 7 out of the40 components of testing or assessment foracademic skills involve the use of instruments ormethods other than standardized tests. In most ofthese cases, the standardized tests are developedby the states themselves, although some arecommercially developed. The seven componentsof testing or assessment for academic skills arewide ranging. In Arizona, the vocational programwill use results from the assessments for aca-demic skills that each local school district isrequired to conduct under state policy. Some ofthese districts employ various forms of perform-

ance assessment, while others rely on writtentests. Arizona has an item bank for assessingacademic skills in the context of vocational skills.The vocational program in California will useresults from their large, new, statewide system ofperformance assessment in the subject areas ofreading, writing, mathematics, science, socialstudies, and history. In the four other states usingassessment for academic skills, the state hasturned directly to their local vocational programand expects them to develop means of conductingassessments for academic skills.

For occupational skills, the differences amongthe states are much greater. As shown in figure3-7, the proportion of components that involvewritten testing is much higher in the 18 states thatencourage testing for occupational skills than inthe states that mandate or encourage assessment.In fact, 20 of the total of 25 existing componentsof testing or assessment for occupational skills in


Table 3-2: How the Results From Testing and Assessment for Occupational Skills are Used Depending onWhere in the Curriculum They are Conducted (total number of uses)

Uses of the resulting testing and assessment information about occupational skills

Location of the testing For accountability For accountability y For program, For student

or assessment in the purposes, not including purposes including school, or course assessmentvocational curriculum Perkins Perkins improvement and/or certification

At the end of sequences ofvocational courses . . . . . . . . . . . 1 15 12 16

During and/or at the endof individual courses . . . . . . . . . 1 8 7 10

Ongoing in the vocationalcurriculum . . . . . . . . . . . . . . . . . . 0 13 10 22

Total . . . . . . . . . . . . . . . . . . . . 2 36 29 48


these 18 states involve written testing and only 5others involve assessment. Conversely, only 3 ofthe total of 27 components in states that encour-age or mandate assessment involve some form ofwritten testing. The other 24 components involvesome form of assessment. Generally, the model ofassessment in these states is:

1.

2.

3.

4.

sets of core occupational competenciescovering as many as 50 or more occupation-ally specific job areas are adopted by thestate;local programs are either encouraged orrequired to utilize these sets of competen-cies in organizing instruction and conduct-ing assessments;local programs are also typically encour-aged or required to provide students with aprofile of their competencies using the samecompetency frameworks; andassessment tied to specific courses, programcompletion, or ongoing assessment is stronglyencouraged or required.

Especially in states that mandate assessment,local

5.

programs are furthermore required to:

submit evidence to the state confirming thatthe required assessments have been con-ducted and describing how they were con-ducted.

In the information about state programs that OTAobtained, only one state volunteered that localprograms are annually reviewed using informa-tion from testing and assessment. (However,OTA did not specifically ask this question.)

While the information in figure 3-7 and datareported above distinguish sharply between writ-ten testing and assessment, it needs to be under-scored that the line of separation between the twois blurred. Even in states where competencytesting for occupational skills is encouraged, themain component of testing for occupational skillstypically includes elements of both testing andassessment. This is also true for the major vendorsof occupational tests, as described in the nextchapter. In short, the components of “testing’that are shown in figure 3-7 frequently includesome elements of assessment. Furthermore, statesthat encourage testing in their main component ofmeasurement for occupational skills may alsohave other components primarily oriented toassessment. One example of this would be severalstates with both a policy of encouraging compe-tency testing in their main component of testingor assessment for occupational skills, and provid-ing students with profiles or portfolios. In fact, invocational education there tends to be a substan-tial emphasis in most states on using multipleforms of assessment.


Figure 3-7: Methods of Testing and Assessment Employed by the States (in 1992-93)

In 18 states that encourage testing In 15 states that mandate assessment

14

20

913

A 1 52

Occupational skills Academic skills Occupational skills Academic skills

In 8 states with no policy on theassessment of occupational skillsIn 10 states that encourage assessment

~ Written tests

m Assessment

10

5

6

n 2

0

Occupational skills Academic skills


Occupational skills Academic skills

The experience in most states where writtenforms of testing for occupational skills is encour-aged is that the written tests end up being used toa greater extent than methods of assessment inmeasuring occupational skills. Once started, theiruse frequently tends to grow disproportionatelycompared to assessment. Even V-TECS, which isone of the key organizations involved in develop-ing competency-based testing for vocational edu-cation, has performance items for only two-thirdsof the 35 item banks where both competency lists

and testing items are available, and the perform-ance items comprise only a small proportion ofeach item bank.

HOW ARE PRACTICES OF TESTING ANDASSESSMENT CHANGING?

The next question is how testing and assess-ment in vocational education are changing inresponse to the 1990 amendments. The mostlikely direction of change is that the performance

{Y Brenda Hattaway, assistant director, V-TECS, pers(mal cmmnunicalion, Sept. ~0. 1993.


standards that states have adopted will requireexpansion of testing and assessment.

OTA asked the states to describe their plans forthe expansion, contraction, or continuation oftheir existing components of written testing andassessment for both academic and occupationalskills over the 3 years between 1992-93 and1995-96, and what new components of writtentesting or assessment for academic and occupa-tional skills will be added. Questions were askedin some detail about changes in the skills to beassessed, the populations and programs to betested or measured, and even the extent to whichtheir plans for expansion are responding to therequirements for performance standards in thePerkins amendments. Questions were structuredto determine if the plans are definite or are onlybeing considered. Most states responded that theirplans for expansion are definite.

The results show that a substantial increase intesting and assessment will occur in vocationaleducation by 1995-96 (see figure 3-8). Most of theincrease will occur in the expansion of existingcomponents of testing and assessment rather thanthe creation of new ones, but some new compo-nents will also be added. In 1992-93, there werea total of 92 components of testing and assess-ment for academic and occupational skills amongthe states. Of those existing components, 48 areslated for expansion by 1995 and 20 new ones willbe added for a total of 109 components by 1995.20

The remaining 41 components will stay the sameor nearly same through 1995.

States indicated in their questionnaire re-sponses that the requirements of the Perkins Actfor performance standards have been an importantif not the deciding factor in over 80 percent of allthese planned expansions of testing and assess-ment for occupational skills and over 70 percentof the planned expansions for academic skills.

States’ plans for the expansion of their policiesof testing and assessment through 1995 cover awide range. Some states are planning only minoradditions or modifications to existing programs.Others involve the implementation of compre-hensive new systems.

Ohio, for example, will implement an ambi-tious expansion of their 28-year-old Ohio Voca-tional Competency Assessment Program to in-clude: a) three subtests from the new Work Keyssystem, which were being piloted in the stateduring 1992-93 (Work Keys is described inchapter 4); b) new or revised competency tests forboth occupationally specific and employabilityskills in over 60 occupational areas; c) and a shiftin policy from simply making competency testsavailable to local programs to strongly encourag-ing their use in local programs. In effect, acomprehensive statewide system of competencytesting for vocational and generic workplaceskills is being implemented. Ohio will alsocontinue its longstanding statewide program ofperformance competitions for students.

Texas plans to add student portfolios to theircurrently required profiles of student competen-cies and to launch a new performance-basedassessment program for generic workplace com-petencies. South Carolina plans to expand thenumber of occupational areas where test banksare available beyond the 41 areas where they arecurrently in use. Iowa plans to stiffen theirrequirements for assessing student competenciesin local programs. West Virginia plans to addstudent profiles to their competency testing pro-gram in the 1993-94 school year and make therequirement mandatory for local programs. Kan-sas will incorporate generic workplace skills intotheir profiles of student competencies after pilot-ing them this year. At the other end of thespectrum, there are several states where the onlyplans for expansion are to increase the number of

20 ~ree of th~=~ new c[)mP)nents Wll] be exis[ing ct)rnp)nents that change from assessment to testing. Tbtxc three conqxmen(s are classified

as “new’ in figures 3-8 and 3-9. This explains why the 20 new components added to the 92 components that existed in 1992-93 increases thetotal number of cmnponents in 1995-96 to only 109. Two of these three “new’ components are occupational and one is academic.

.


Testing

Figure 3-8: Planned Expansion of Testing v. Assessment by 1995-96

Occupational skillsAssessment

1 ‘p ~ ~II II l’”” ‘“

t-

4

I , 1 I0 2 4 6 8 10 12 14 16 18 20 22

Number of components

l“”

r 1 1 10 2 4 6 8 10 12 14 16 18 20 22

Number of components

Academic skillsTesting AssessmentI 1 r

II

‘ F;”,, \ ~ ,

II 1 . ’ II

. . .

II II

,<,

,,,

:2Viz

%3 : , II<’ _ No Changea

II II II U ExpansionII ,! 1!II 4 ’ ,,11 ~ New

,!1 III 1 I I r 4 \ 1 I I I0 2 4 6 8 10 12 14 16 18 20 22 0 2 4 6 8 10 12 14 16 18 20 22

Number of components Number of components

1 = Encourage testing 2 = Mandate assessment 3 = Encourage assessment 4 = No policy

a One of these components will contract.


occupational or job areas covered in their existingprogram.

The most significant shift that will occur in thenear future is from assessment for occupationalskills to written testing. There will be eight newcomponents of written testing for occupationalskills among the states and three new componentsof assessment. However, two of the existingcomponents of assessment for occupational skillswill disappear for a net gain of only one new

component of assessment. All eight of these newcomponents of written testing will be in states thatare currently in the category of either mandatingor encouraging assessment, or do not have apolicy on testing and assessment for occupationalskills.

This means that the number of states in thecategory of encouraging testing for occupationalskills will increase from 18 to 26. Those in theother three categories of mandating or encourag-


ing assessment, or not having a policy on testingand assessment, will decline from 33 states to 25states.

Three-quarters of the existing components ofwritten testing in states that encourage testing foroccupational skills are also slated for expansion,compared to just over 50 percent of the existingcomponents of assessment (17 of 23 componentsof testing compared to 16 of 29 existing compo-nents of assessment) .2’

The net effect of all these changes will be toshift the balance between testing and assessmentin state policy from 23 components of testing foroccupational skills and 29 for assessment in1992-93 to 31 components of testing and 30 ofassessment by 1995.

The major issue is what the long-term effects ofthis shift toward written methods of testing arelikely to be on the character and content ofvocational education. Programs of vocationaleducation and training seem to be one of the lastplaces where written testing should be preferredover methods of performance assessment. Forexample, methods of work sample testing haveconsistently been found to be more directlyrelated to competence on the job than competencytesting. 22 The long-term effects of written testing

could be to shift teaching and learning in voca-tional education away from the development ofmore complex skills toward a greater emphasis onacquiring factual knowledge of various kinds, andaway from constructive forms of student learningtoward teacher-dominated modes of instruction.The effects on the character of vocational educa-tion could be profound.

There are several factors driving this shifttoward increased written testing. One is the lower

cost of written testing compared to performanceassessment in time, materials, and instructoreffort. This tends to produce strong incentives forthe growth of written testing and the decline ofassessment. Another is the widespread perceptionthat written testing is more reliable and fair thanmethods of assessment where the judgments ofinstructors or evaluators are generally involved.In fact, with sufficient training and moderation ofjudgments, differences in rating among raterspresents no problem.23 The third is that the kindsof factual knowledge and information that aremost readily tested tend to favor lecture methodsof instruction over active methods of learning andhands-on laboratory and field work. This willtend to drive instruction in vocational educationtoward teacher lecturing and away from hands-onlearning. For example, factual information aboutbuilding codes and handbook information aboutthe sizes and strengths of materials is much easierto incorporate into a written test and easier toteach with lecture methods than are concepts ofdesigning and installing an electrical system for ahouse. Both are important but the issue is thebalance between the two.

Unfortunately, there is very little good researchon the long-term effects of written testing com-pared to assessment on the character and contentof instruction in active learning environments likevocational education, so firm conclusions on thisissue cannot be drawn. One of the best knownstudies is an article on training in the military,which gives one example of the dramatic effectsthat can occur. In the example, the substitution ofperformance assessment for written testing pro-

Z1 Figure 3.8 shows 1 ] components of assessment that will no[ chang(. and 16 that will expand. In addition, there arc the Z ct)nlponcnts of

assessment that will be converted to written testing for a total of 29 components.

22 Alexandra Wigdt)r and Ben F. Green, Jr. (eds. ), Performance Assessment in the Workplace, Vo/. / (Washington, ~“ National AcademyPress, 1991 ); and John Hunter, ‘Causal Analysis, Cognitive Ability, Job Knowledge, Job Perfomlance, and Supervisor Ratings, ” Per-/w-manccMeasw-e and Theory, S. Lundy (cd.) (Hillsdale, NJ: Lawrence Erlbaum, 1983).

23 Shavelson, op. cit., footnote 10.


duced many changes.24 When written testing was

stopped and students were assessed instead onperformance, classroom lecturing on topics suchas the muzzle velocity of guns and the functionsof breech block locking bolts greatly diminished,desks were removed from the training rooms,equipment that the students were learning how torepair was brought in, and students began tospend much more of their time in class actuallyrepairing equipment. The instructors complainedthat the new tests were too hard and they wereright. The students’ scores were very low on theperformance tests. But their scores increasedrapidly with each successive class of students inthe program. Performance on the new assess-ments soon became the best predictor of grades inthe class and correlations of the written tests withgrades dropped precipitously. The only thing thatwas changed was the substitution of performanceassessment for the written tests.

There is also a considerable amount of researchshowing that the imposition of high-stakes, stan-dardized testing for basic skills on instruction inregular education tends to narrow the range ofskills taught and learned by students.25

The irony is that written methods of testingappear to be expanding in vocational educa-tion at the very time that questions are beingraised in the rest of education about theeffectiveness of standardized testing, and agreat deal of experimentation is under waywith methods of assessment, student profiles,and the like.

There should be no mistake about what is beingsaid here. Much of the expansion of testing andassessment that is now occurring in vocationaleducation is in areas of assessment. A goodexample of one of the major new efforts invocational assessment is the California Career-Technical Assessment Project (C-TAP—see box

3-A). C-TAP, which is in the process of beingdeveloped and tested, is a comprehensive systemof assessment procedures and student portfoliosorganized in part around newly defined broadtechnical and occupational skills. But, in fact, thisnew system of assessment is only one of threenew components of assessment planned for voca-tional education. And, meanwhile, two other newcomponents of assessment in other states will betransformed into programs of written testing. Thegreatest expansion overall is occurring in meth-ods of written testing.

WHAT SKILLS WILL BE ASSESSED?By 1995, states are planning considerable

change in the skills that will be measured. Thischange has implications for reform in vocationaleducation.

How the mix of skills assessed will change isshown in figure 3-9 for the four basic categoriesof skills that have been defined: vocational skills,generic workplace competencies, broad technicalskills, and academic skills. As in the previousfigure, the pattern is basically one of growth.There will be expansion of the skill areas oftesting and assessment.

Before discussing figure 3-9 in detail, it shouldbe noted that change in testing and assessment isrepresented in both figures 3-8 and 3-9 by thecomponents and subcomponents of states’ poli-cies of testing and assessment that are new,expanding, or staying the same. The data pre-sented provide a proxy for more direct measuresof the change that is occurring in testing andassessment at the local level, such as in thenumbers of students who are being tested orassessed and the amount of time they are spendingin testing or assessment. (Data like this would beextremely difficult to obtain.) The expansion ofstates’ policies frequently involves, for example,

2J Nf)m]an Frederiksen, “The Real Bias in Tes[ing: Influences on Teaching and lxarnlng, ” American Psychologist, vol. 39, No. 3, March1~8~, pp. 193-202.

~< George Madaus, “The Influence of Testing on Curriculum,’ Cri(ica/ Issues in Curricu/~~m, 87th YearbtKA of the Nati(mal .N)ciety forthe Stud} of Educati(m, Laurel N. Tanner (cd. ) (Chicago, IL: University of Chicago Press, 1988), pp. 83-121.


Box 3-A: Career-Technical Assessment Project (C-TAP)For the California State Department of Education, the Far West Laboratory for Education Research and

Development is currently in the process of developing, pilot testing, and validating a statewide systemof authenticstudent assessment for vocational-technical programs in the state. The system will be an integral part of the CareerVocational Division’s occupational clusters strategy for vocation-technioal education. The plan is to implement thesystem in high schools, adult education programs, and regional occupation programs/centers throughout the state.

Student mastery of core occupational content, as well as career skills and appropriate academic standards,will be assessed in an integrated way. The system involves the adoption of three kinds of standards: 1) a“framework” of occupational content standards in each of the cluster areas, 2) a series of so-called “careerperformance standards,” and 3) standards for academic skills that are needed in the workplace and underlie thecareer performance standards. Two examples of these occupational dusters are animal science in agriculture andcomputer science and information processing in business. Career performance standards will be set in the areasof personal skills of attitude and time management, interpersonal skills, thinking and problem-solving skills,communication skills, occupational safety, employment literacy, and technology literacy.

Students will be certified as “work ready” for an occupational cluster. Student certification will signify thatstudents have the knowledge, skills, and abilities to succeed in an occupational field; the system of assessmentis primarily intended to certify students for further learning in an occupational area or an academic program ofstudy. The curriculum standards may be changed in California so that all students will decertified as ready for work.

Two different kinds of assessments are being employed in the C-TAP system-cumulative and administered.

Cumulative Assessments

The cumulative assessments provide each student with a record of their accomplishments and levels ofachievement in high school to use in seeking employment. Most of the assessment products are highly dependenton initiative taken by the students. The three main components of cumulative assessment are a SupervisedPractical Experience (SPE), an assessment project, and a portfolio.

Supervised Practical Experience. Students who are enrolled in an occupational program requiring an SPEhave their work supervisor complete an evaluation form rating their skills in the seven career performance areas.Additionally, students are rated on skills specific to their vocational program required “on the job.’’The completedSPE evaluation form is included in the portfolio.

Assessment Project. Students complete and present an approved assessment project during the course oftheir program. Projects involve either the planning and development of a tangible end product a written description

increasing the number of occupational program students who are being tested versus those whoareas where testing or assessment will be con-ducted, revising the testing instruments that willbe available from the state and expanding theirnumbers, adding a new category of skills to thestate’s testing or assessment program, or makingtesting or assessment a requirement (mandatory).All such changes in policy would increase theimpact on students. Most of the changes plannedby states are of these kinds. Essentially then,figure 3-8 represents the change in the numbers of

are being assessed, while figure 3-9 represents thegrowth that will occur in the number of studentswho will be assessed in different categories ofskill.

It is important to point out a major differencebetween figure 3-8 and 3-9. Figure 3-8 shows thenumbers of components of testing and assessmentfor academic and occupational skills. These arethe same components as in figure 3-5, 3-6, and3-7. In figure 3-9, each of the components of


and analysis of a significant “real world” experience in the student’s field, or both. Project ideas will be developedjointly by the student and teacher according to specific guidelines provided by the state for each occupationalcluster. Ideally, parents and business and industry people are involved in the development of the students’ work.At the conclusion of the project, all students describe their learning experiences through an oral presentation.

Structured Portfolio. Students assemble a structured portfolio during the course of their high school program.The portfolio helps students organize and present a collection of their work for purposes of assessment and forpresentation to prospective employers and/or advanced training institutions.

The portfolio can include a letter of introduction, a career development package, a research write-up, acompleted SPE evaluation form, the assessment project results, and four or more work samples. The careerdevelopment package will include a resumé, an employment or college application, and a letter ofrecommendation. The research write-up is accompanied by an original outline and at least one early draft that hadbeen submitted for teacher review. The topics of the research write-up maybe selected by the student from choicesrelated to safety, social consequences, business practices, and future trends in the student’s chosen field. Thework samples are evidence of the student’s best work and maybe related to the student’s certification projectorSPE. Work samples are selected for inclusion by the students with guidance from the teacher according to criteriaestablished by the state. Work samples may be represented by actual products, photographs, videotapes,performances, written descriptions, or any other reasonable and appropriate means.

Administered Assessments

The administered assessments are structured exercises that are given to students at a certain time in orderto assess their capabilities for performance.

Project Presentation. Students present details of their assessment project to a review panel and respond toquestions from the panel. The panel may consist of teachers, parents, students, and industry representatives. Thestudent’s presentation is evaluated according to specified criteria including oral presentation skills and ability toreflect on the project experience

Written Scenario. Students are presented with a written scenario representing a complex and realisticproblem from their vocational area. Students have 45 minutes to respond in writing. They are judged on ability toapply content knowledge to address the problem presented in the scenario.

On-Demand Test Students take an on-demand test focusing on the career performance standards.Enhanced multiple-choice items, including interpretative exercises and open-ended responses, are emphasized.SOURCE: Adapted from Far W@ Laboratory for Educational Reaearch and Development, Career-Tbchrrica/ Assessment Project (SanFrancisco, CA: Nov. 2, 1992).

occupational skills are broken into subcompo- Included in figure 3-9 are 10 cases where bothnents according to the number of different occu-pational skills that are assessed. Many of thecomponents for assessing occupational skillsinvolve the measurement of two, or in a few casesall, of the occupational skills—that is, vocational,generic workplace, and/or broad technical skills.Each of these subcomponents are represented asa single count in figure 3-9 even though it mayonly be part of a more extensive component oftesting or assessment for occupational skills.

academic and occupational vocational skills willbe assessed together by 1995-96.

Academic SkillsThe most surprising result in figure 3-9 is that

the slowest rate of expansion in state programs oftesting and assessment will occur in the area ofacademic skills. State programs of testing andassessment for academic skills will not change in24 of 41 components (59 percent) in which they


Figure 3-9: Skill Areas of Planned Expansion in Testing and Assessment by 1995-96

Academic skillsI 1

1I

) 1 I I !o 5 10 15 20 25 30

Number of subcomponents

General workplace skillsI 1

I I

I

I I I I I 10 5 10 15 20 25 30

Number of subcomponents

Vocational skillsr

1’”(’

0 5 10 15 20 25 30Number of subcomponents

Broad technical skillsI

_ No changea

D Expansion

m New

0

1 = Encourage testing 2. Mandate assessment 3 =

a one of these components of academic testing and assessment WIII contract.


were assessed in the 1992-93 school year.26 The means

4

5 10 15 20 25 30Number of subcomponents

Encourage assessment 4 = No policy

for testing or assessing academic skills inother 17 components (or 41 percent) will expand. place for 1992-93. These percentages of expand-Thirteen new components will also be added by ing, unchanging, and new components of testing1995, which is a 27 percent increase in the total and assessment for academic skills will benumber of components by 1995. These new approximately the same across all four types ofcomponents will be mostly in states that had no states.

M One conlP)nent of Iestlng or assessmnt for academic skills will contract. TO simplify figures 3-8 and 3-9, this component is included

in the data for c(mlpments where there will be ‘‘no change. ’

Chapter 3:

Vocational SkillsIn contrast to the 41 percent rate of expansion

for academic skills, testing and assessment forvocational skills will expand in 36 of the 51subcomponents of state policy where they wereassessed in 1992-93 school year, or at a rate of 70percent of the existing subcomponents. In addi-tion, testing or assessment for vocational skillswill occur in six new components by 1995 for anincrease of 12 percent.27 The overall rate ofexpansion (including new subcomponents) forvocational skills will be even faster for genericworkplace skills and broad technical skills, butthe total amounts of testing or assessment forthese two categories of skills in 1992-93 will stillbe considerably smaller than for vocational skillsbecause of the smaller base. This means thatamong the four types of skills the largest totalincrease in testing and assessment will be for theconventional, occupationally specific skills ofvocational education.

Integration of Academic andVocational Education

This slow rate of growth for testing andassessment for academic skills is surprising forseveral reasons. One is that academic skills are anew area for testing and assessment in vocationaleducation, and another is the strong indications inthe 1990 amendments that gains in academicskills should be at least one of the two perform-ance standards established by the states. Testscore gains are generally much more difficult tomeasure than attainments at any point in time andespecially in a statewide, mass testing program.The fact that many of the states are relying on testscore data from their state-level minimum compe-tency exit examinations or some other statewidetest, which is administered at a certain grade level,makes it very hard to see how they will be able to

State Policies on Testing and Assessment 63

show gains in students’ academic skills over thecourse of their enrollment in vocational educa-tion. These difficulties indicate that state plans forexpanding the capabilities for academic skillstesting and assessment should be substantial.

Another apparent barrier to the integration ofacademic and vocational education is the substan-tial differences that exist between the methods oftesting and assessment being used by states foracademic skills compared to occupational skills,as discussed above. One of these differences is theheavy reliance on norm-referenced, standardizedtesting for academic skills compared to thecriterion-referenced nature of testing and assess-ment for occupational skills. Another differenceis that on the academic side, most of the testingand assessment is predominantly written, whileon the occupational side, there is a mix ofassessment and written testing. Furthermore, thetesting for academic skills is typically centralizedat the state level and conducted statewide on amass basis at a given grade level, while on theoccupational side the process is highly decentral-ized and tied to the local structure of courses andprogram completions.

A few statistics from the OTA survey tell thisstory of the divorce of testing and assessment foracademic skills from occupational skills in starkterms. Only 10 of the 54 subcomponents oftesting and assessment for academic skillsplanned by the states will be closely related to thetesting and assessment for occupational skills atthe local level—that is, in relation to the structureof courses or program completions, or in anongoing program of assessment.28 Various meth-ods of doing this are being tried in the statesinvolved but in only 5 of the 54 subcomponentsare academic skills actually being tested for orassessed together with occupational skills in thecontext of the students’ vocational and academicprograms. In the other cases, a commercially

27 While the rate of increase is only 12 percent, fewer states were without any capacity for testing or assessing vocati(mat skills than academicskills. S(J the expected rate of new c(~mponents of academic skills should be higher.

2X me I o su~onlp)nents are not explicitly shown in figure 3-9.


developed academic test is being separatelyadministered to students but in relation to theirvocational course work.

The problems of reconciling academic andoccupational test score and assessment informa-tion, and meaningfully interpreting it for purposesof monitoring or improving programs are formi-dable. Not paying close attention to these difficul-ties could lead to serious problems of misuse oftest information and assessment results. If thecontent of the statewide academic tests does notreflect the academic content of the vocational oracademic programs that are being integrated, thenthe resulting readings about the success of theefforts will be false. Also, the information fromthe academic tests may be in the form of “rank inclass, ’ as is frequently the case with standardizedwritten testing, while the information aboutoccupational skills will tend to come from assess-ments and again be much more competencyspecific.

Considering all of these difficulties, and thepriority in the legislation on integrating academicand vocational education, it is surprising howlittle effort states are planning to devote torevising and expanding their policies of testingand assessment for academic skills compared toexpanding their resources for measuring occupa-tional skills. The ways in which testing andassessment for academic skills is separated fromtesting and assessment for occupational skillsindicates that academic and vocational educationare possibly being driven further apart rather thancloser together. It is very hard to see how thetesting and assessment information being pro-duced by the states could easily be used forpurposes of the highly localized, school-by-school, program-by-program, and teacher-by-teacher efforts that are required to integrateacademic and vocational education. The slowerexpansion of testing and assessment for academicskills that is planned, together with the fact thatacademic and occupational skills assessment willbe combined in only 10 cases, implies that theexpansion of testing and assessment for voca-

tional skills has higher priority for 1995 than theintegration of academic and vocational education.

The inherent difficulties may explain a goodpart of the lack of plans for expanding academicskills assessment in the implementation of Perk-ins performance standards. Vocational educatorsmay be understandably reluctant to initiate majorcoordination efforts with their statewide officesfor academic testing, when this could involvesubstantial change in how testing and assessmentis done and intrude in an area (the statewideacademic testing program) where they have notbeen involved. The statewide office of academictesting may similarly be reluctant to coordinatetheir efforts with vocational education because itis unfamiliar territory and might involve substan-tial change in their approaches to testing andassessment. California’s new portfolio-based sys-tem of testing and assessment for occupationalskills, which includes some assessment of aca-demic competencies, and where efforts will bemade to coordinate with the new, statewidesystem of performance assessment, provides onemodel of how statewide assessment of academicand occupational skills can potentially be coordi-nated.

Generic Workplace SkillsA rapid expansion of testing and assessment for

generic workplace skills is clearly in store, Asshown in figure 3-9, the number of new subcom-ponents of testing or assessment for generic skillswill exceed the number of existing componentsby nearly a factor of two for an increase of 175percent. Most of the components of testing andassessment for generic workplace skills will benew in 1995. However, the large numerical size ofthis increase probably overstates the actual effectson learning and instruction in vocational educa-tion compared to effects of the planned expansionof testing and assessment for vocational skills,because generic workplace skills are the primaryfocus of testing or assessment in only 5 of the totalof 22 cases. In the 17 other cases, the assessment


of generic workplace skills will be combined withthe assessment of skills in one of the othercategories of skills and be in a secondary rolerelative to those skills. Twelve of these 17combinations are with vocational skills.

Broad Technical SkillsEven after the planned fivefold expansion, the

smallest category of capacity for testing andassessment will be for broad technical skills.Several reasons for this can be surmised. One isthe lack of clarity about the nature of these broadtechnical skills, and instruments and methods forassessing them. Another is that the reorganizationof vocational curriculum around broad technicalskills has not been the object of as much reformpressure as has been the concept of integratingacademic and vocational education. However, thepriority on broadening occupational skills ap-

pears to be growing. Still, the greatest totalexpansion of testing and assessment in absoluteamount will occur in the traditional area ofvocational skills, which tend to be very jobspecific (see figure 3-9). At least so far, perform-ance standards appear to be working against thereform of broadening occupational skills.

The numbers of subcomponents where morethan one category of skills are included isgrowing. For example, in figure 3-9, the totalnumber of subcomponents shown for the 109components of testing and assessment planned bythe states for 1995 is 138. This is an average of1.26 skill areas per component of testing orassessment. The comparable number for the1992-93 school year was 99 skill areas for the 92components of testing or assessment, or 1.08 skillareas per component.29

29 [n figure 3.8, State progmm5 that involve the lntegmtion of testing and assessment for academic and (~cupati(mal skills are treated as tw~~

separate components. Components of testing are ‘‘new’ if they did not exist in 1992-93, or they will change by 1995 from either testing toassessment or vice versa. Subcornponents of testing or assessment for skill area are classified in the data as “‘new” tmly when the component

is new. The addition of a new subcomponent of, for example, generic workplace skills to a program of testing (w assessment for occupationalskills is classified as a ‘ ‘new aspect of an existing component.

CaseThree

Studies:Vendors

of Tests forOccupationalCompetencies

A nother perspective on the capacity for testing andassessment in vocational education is provided bydescribing three main vendors who supply states andlocal programs with resources for occupational testing.

The vendors represent the mainstream of current practices invocational education and one direction of future development—generic workplace skills.

The case studies include:the Work Keys System being developed by AmericanCollege Testing (ACT);the Vocational-Technical Education Consortium of theStates (V-TECS), which supports a system of test itembanks for competency-based testing and assessment; andthe Student Occupational Competency Achievement Test-ing (SOCAT) program of the National Occupational Com-petency Testing Institute (NOCTI).

first assessment program is just now being developed andimplemented. The latter two programs have been in operation fornearly 20 years.

WORK KEYS1

Origin of ACT and Work KeysWork Keys is a system being developed by ACT for teaching

and testing general workplace competencies and employabilityskills. The system is well along in development. When fully

4

I Joel West, Joy McClarity, Kate Ulmer Sottong, and Barry Mason, American CollegeTesting, OTA interview, Jan. 28, 1993.

67


operational, as expected in early 1994, WorkKeys will include the following four interrelatedcomponents for each of eight skill areas: 1 ) testssuitable for large-scale, high-stakes testing situa-tions; 2) a job profiling component for analyzingthe skill levels required in eight areas in real-lifejobs; 3) instructional materials related to the skillsassessed; and 4) a reporting service. Portions ofall the components are available now.

Work Keys represents a broadening of missionfor ACT, an independent, not-for-profit organiza-tion founded in 1959, which provides programsand services in the areas of college admissions,career and educational planning, financial aid,continuing education, professional certification,and licensure. ACT is best known for its testingprogram for college entrance. More than a millionhigh school students take the ACT tests each year;college admissions officers use the scores inmaking admissions and placement decisions. In1991, ACT decided to expand its services toencompass students bound directly for theworkforce.

The Work Keys SystemThe broad goal of the Work Keys system,

according to ACT, is to help strengthen theworkplace skill competencies of all individualswho are bound for work or are already injobs—not just vocational students. ACT hopesthat schools will use Work Keys to help studentssee the connection between the skills acquired inschool and those needed on the job.

The current design for Work Keys focuses on12 different skills, each of which will eventuallyhave its own separate test.2 ACT identified theseskills by surveying the literature on workplaceskills and consulting with employers, teachers,and employment and training experts; the aimwas to identify skills that are both measurable andteachable and that are viewed as important by

employers and educators. When fully operational,Work Keys will enable test takers to evaluate theirskills in a general way or compare their skilllevels with those required by employers forspecific jobs.

The four linked components of Work Keys canbe summarized as follows:3

1.

2.

3.

4.

Testing Component. This will include atleast 12 workplace skills tests or assess-ments that will be criterion-referenced andrequire written or other kinds of responses.The instruments will measure the level ofcompetency demonstrated by the individ-ual, or “how much” of the workplace skillthey can demonstrate. The tests will beadministered in a variety of formats such asmultiple choice, constructed response, andcomputer adaptive. Necessary materials willrange from paper and pencil to audiotapesand videotapes.Job-Profiling Component. This componentwill enable a company to profile the compe-tency or skill levels required for an em-ployee to perform selected jobs success-fully. The job profiling system was releasedin the fall of 1993.Instructional Support Component. Instruc-tional materials and accompanying refer-ence guides will help learners and theirteachers take steps to improve and broadenlearner skills, so that people can obtain thejobs they want.Reporting and Recording Component. Acomprehensive recording and reporting serv-ice will provide informative reports onassessment results to students, teachers, andemployers. For example, teachers may usethe service to see how many of theirstudents have strong workplace skills orevaluate instructional programs. Educators

J Although ACT plans to develop Work Keys tests fw 12 skills, this number may change during the development process.

~ The whole system is being implemented in phases and is currently operational for the initial skill areas; additional programs, services,and skill areas will be added over time.

Chapter 4: Case Studies: Three Vendors of

and employers will be interested in compar-ing the test results to job profiles.

Purchasers will be able to buy and use anycombination of the four components, althoughACT will encourage users to view the system asa whole and to use all the parts for which theyhave a need.

The Work Keys AssessmentsAll of the Work Keys assessments aim to

measure ‘‘work-related basic skills, ’ with anemphasis on workplace applications of skillsrather than academic applications. In addition, allof the assessments are criterion-referenced (notnorm-referenced), meaning that an examinee isevaluated on his or her performance relative to thecontent and level of the test items and not theperformance of other test takers.

Each test will include questions across a rangeof levels of difficulty, from four to six dependingon the assessment. For each assessment, the rangeof levels reflects the skills required from thelowest level likely to be assessed to just below thelevel at which specialized training would berequired. The levels are hierarchical. For exam-ple, an examinee who scores at the fourth level onthe Applied Mathematics assessment should alsobe able to perform well on exercises at all levelsbelow, but not levels above. Because the tests arecriterion-referenced, a specific level on one as-sessment does not correspond to the same level onanother assessment.

Six tests were released in September of 1993:

1. Reading for Information. This “. . . meas-ures the examinee’s ability to read andunderstand work-related instructions andpolicies. Reading selections and questionsbased on the actual demands of theworkplace appear in the form of memos,bulletins, notices, letters, policy manuals,and governmental regulations. ’ Questionson the test fall across five levels of diffi-culty, arranged from easiest to the most

Tests for Occupational Competencies I 69

2.

3.

4.

5.

6.

difficult, and are followed by multiple-choice questions.Listening and Writing. This assessmentmeasures listening and writing skills to-gether, simulating how they are often usedin the workplace. The examinee listens toaudio-messages on tape, takes notes, andcomposes a written message. The writtenmessages are scored in two different ways:a listening score measures accuracy andcompleteness of information, and a writingscore addresses grammar.Applied Mathematics. This measures thetest taker’s skill in setting up and solvingwork problems using ‘‘mathematical rea-soning skills. ’ Examinees are allowed touse calculators, just as in the workplace.Applied Technology. This paper-and-pencil,multiple-choice test measures an individ-ual’s skill in solving technological prob-lems, covering the basic principles of me-chanics, electricity, fluid dynamics, andthermodynamics as they apply to machinesand equipment found in the workplace. Thetest emphasizes skills in identifying andanalyzing relevant parts of problems, evalu-ating potential solutions, selecting materi-als and solutions, and applying methods tonovel challenges and circumstances.Teamwork. This test measures an exam-inee’s ability to ‘‘. . . choose behaviorsand/or actions that simultaneously supportteam interrelationships and lead towardaccomplishment of work tasks. Test takerswill watch a videotape of teams of workersperforming tasks and will be asked multiple-choice questions about the teamwork sce-narios.Locating Information. This multiple-choicetest will measure the ability to use graphicdocuments to insert, locate, compare, andsummarize information. The types of graph-ics used on the test include diagrams,blueprints, floor plans, tables, forms,graphs, and instrument gauges.

70 Testing and Assessment in

ACT plans to complete at least

Vocational Education

five additionaltests in the next several years. One, Observations,is scheduled for release in September 1994. Theothers are currently in the design and earlydevelopment phases.

●

●

●

●

•

Speaking. This test will measure whether aperson can convey a spoken message clearly.Observation. This test will measure a per-son’s ‘‘watching skills’ ‘--the ability to learnfrom demonstrations.Motivation. This test will measure dependa-bility and responsibility, and will focus onwork habits that can be taught (rather thanattributes of personality).Learning. This test will measure a person’sskill at adjusting to changes in a job situa-tion, such as those resulting from a new jobor new technology.Managing Resources. This test will measurea person’s skill in scheduling, budgeting,and allocating resources.

The Assessment Development ProcessACT is undertaking several steps to ensure that

Work Keys will be responsive to the needs ofemployers and educators and that the assessmentsdeveloped will be reliable, valid, and fair. Early inthe development process, ACT created a consor-tium of six “charter states’ with a demonstratedinterest in new vocational assessments to giveadvice and help pilot the system.4 No state isobliged, however, to use the Work Keys assess-ments once completed. Each charter state also hasa Work Keys advisory panel, composed oftwo-thirds business and industry representativesand one-third educators. The advisory panels helpACT with the conception and development ofWork Keys components, with prototype testingand pretesting, and with marketing the system. Inaddition, the panels are expected to help facilitatethe use of the Work Keys assessments in theirown states.

ACT follows a typical process of objective testdevelopment to ensure that test items and assess-ments meet high professional standards of relia-bility, content validity, and fairness. First,‘‘constructs, ’ or definitions, are identified anddeveloped for each skill, with the help of tworepresentatives from each charter state. Onceconstructs are developed, ACT attempts to defineand describe a hierarchy of content levels for eachskill to provide a set of criteria for test construc-tion.

The second step is to draft items that corre-spond to each level of difficulty; ACT hires itemwriters to help with this process. All items mustbe written to the specifications developed by ACTand the advisory panels, and all items are editedby ACT staff.

The third phase is “prototype testing. ’ Inprototype testing, the draft items are administeredto small samples of students and employees. ACTdetermines whether the draft items appear tocorrespond with the expected level of difficultyand satisfy the content criteria. Based on thefindings, ACT rewrites and redevelops items foreach level of difficulty.

Fourth, a large number of items are written,edited, and reviewed by experts for contentvalidity and fairness. These items are pretested ona large sample of individuals. For the multiple-choice Work Keys tests, this sample consists ofover 1,600 student and employee volunteers. ‘Thepretest results are analyzed for consistency andreliability. High-quality items meeting all contentand statistical criteria are selected to produce thefinal tests.

Test Administration and ReportingThe Work Keys tests are designed to be

administered by teachers or guidance counselors,in accordance with procedures proscribed byACT. Currently, all tests completed by students

4 The six charter states are Wisconsin, Tennessee, Michigan, Iowa, Oregon, and Ohio.

Chapter 4: Case Studies: Three Vendors of Tests for Occupational Competencies 71

are returned to ACT for scoring; local onsitescoring will be available in the future.

For each test purchased, five different types ofreports will be generated by ACT and sent backto the client.

. Chat-r Essay Report. This report providessome general descriptive information abouthow various groups of examinees scored oneach test. It is organized around a standardset of questions such as: ‘‘Do the scores ofmales and females differ on the Reading forInformation test?’ For each question, a pageof the report provides information about thepercentages of examinees achieving differ-ent score levels in both bar and table form.The report is produced for each test adminis-tered.

. Individual Reports. Each of the examineeswill receive a multipart report describing hisor her performance on each test with sugges-tions of learning activities that might beundertaken to improve skills. The first partgives the examinee’s scores on the assess-ment, a few tasks illustrating the levels ofperformance associated with the scores, andsuggested learning activities. The secondpart contains the examinee’s scores alongwith demographic and other informationprovided by the examinee during the testingprocess. The third part summarizes theinformation in a form suitable for attachmentto job or school admission applications. Theinformation in the report is designed to beused for course planning, career guidance,and individual goal setting.

. Roster Report. One copy of this report,which lists the name and four lines ofinformation on each examinee, will be pro-vided for each client order. The informationwill include assessment scores and demo-graphic and job-related information (e.g.,job-seeking and career choice information).

● Vocational Information Reports. This reportis designed to be used in determining the

career goals of a group of examinees andwhether those goals match the occupationalopportunities in a given region or city. Itshows the percentages of examinees holdingand expressing interest in particular jobsselected from a given list.

. Local Items Report. Clients may also includelocally generated items in the Work Keyssystem. These questions are primarily in-tended to obtain information about an exam-inee’s instructional experiences (e.g., inusing calculators). This report tabulates theresponses of examinees to those questions.

ACT is considering offering electronic re-sumes for interested individuals who take theWork Keys assessments. This resume wouldcontain cumulative information on an individ-ual’s skill levels over time, and would be madeavailable to prospective employers at the individ-ual’s request.

Job ProfilingThe job profiling component of Work Keys

will enable employers to identify the nature andlevel of work-related basic skills required for jobsin their companies. By following the Work Keysprocedure, an employer will be able to determinethe level of each of the 12 Work Keys skillsrequired for every job profiled, Analysts trainedand certified by ACT will conduct the jobprofiling. The profiling procedure is being devel-oped in a joint effort with a number of companies.

Individuals who participate in Work Keys willalso be able to develop their own skill profilesbased on their assessment score reports. A studentcould then compare his or her personal profile tothe job profile for their desired occupation. Bothlearners and employers will be able to see theextent to which an individual has the skills neededto qualify for a particular job. ACT is developinga database of job profiles (without employeridentification) that can provide employers andeducators with a general picture of skill require-ments for different occupations.


The instructional component will consist ofmaterials describing the Work Keys skills ingreater detail and illustrating workplace applica-tions of those skills. ACT may also offer work-shops for educators and employees to discussstrategies for building the skills needed in theworkplace.

Implementation of Work KeysACT began marketing the Work Keys system

in early 1993. A number of states have decided touse or are considering use of the Work Keyssystem in various ways. Several states intend touse Work Keys for measuring ‘ ‘basic or moreadvanced academic skills” as required in thePerkins Act. Others may administer it as a pretestat the beginning of 11th grade and a post-test at

the end of 12th grade to assess student gains inworkplace skills. Still others may use Work Keysas a program completion examination at the endof grade 14, after the final years of an integratedsecondary-postsecondary ‘‘tech-prep’ program.

VOCATIONAL-TECHNICAL EDUCATIONCONSORTIUM OF THE STATES5

Origin and Purpose of V-TECSV-TECS is a consortium of 23 states with the

goal of promoting competency-based vocational-technical education. Since its inception in 1973,V-TECS has been a unit within the SouthernAssociation of Colleges and Schools, the mainaccrediting agency in the South. Full membershipin V-TECS is limited to state agencies responsi-ble for administering the Perkins Act programs.Member agencies provide proportional supportfor V-TECS administrative and product develop-ment costs. Associate membership is open to . .

the military services, federal, state and localgovernmental agencies, international entities, andother organizations. . .’ with demonstrated inter-est in performance-based education.6

V-TECS aims to accomplish its goal of pro-moting competency-based education through co-operative research and development (R&D) ef-forts in four main areas: 1 ) analyzing jobs; 2)organizing job-related information; 3) developingcomponents for assessing student achievement;and 4) designing, developing, and/or acquiringinstructional materials that link teaching with theskills required for jobs.

These four efforts are interrelated, with the firsteffort, occupational analysis, providing the foun-dation for the other three. The founders ofV-TECS felt that the improvement of curriculafor vocational-technical education should beginwith occupational analysis––ascertaining the spe-cific tasks and duties performed by workers incertain jobs and building a curriculum aroundthem. The task of developing the occupationalanalysis was divided among participating states,with each state taking responsibility for certainoccupations and sharing their findings with othermembers.

Assessment did not become a major focus ofV-TECS until 1986, when banks of test items forstates or other V-TECS members to use inconstructing their own competency-based testswere developed to respond to the growing interestin better assessment and credentialing for voca-tional students.7 As with curriculum, V-TEJCSmembers felt that assessment should be based onwhat students would be required to do in theoccupations for which they were trained. Thus,every test item selected for the banks is tied to aspecific task or duty in a specific job area.

5 Information in this case study is based on Brenda Hattaway, assistant director, V-TECS, OTA interview, Jan. 12, 1993.

b The technical training commands of the Army, Air Force, Navy, and Marine Corps are associate members and suppwt V-TECS throughsuch means as sharing task lists with member states. Other federal agencies, such as the Bureau of Prisons and the International Labor AffairsDivisi(m of the U.S. Department of Labor, are associate members.

7 According to Hattaway (op. cit., footnote 5), the main purpose of the tests has been to improve instruction.


Paralleling its four R&D efforts,offers four main products and services:

Analytical Tools. First developedthese are called “catalogs” andover 200 job areas. 8 For each job

V-TECS

in 1973,exist forarea, the

catalog consists of lists of the duties andtasks involved, along with the tools, equip-ment, and work aids needed to performthem, Finally, for each of the duties andtasks there is a list of performance objec-tives and the steps that the worker must takein performing them.9

Instructional Tools. These, too, are organ-ized by job area. For each job area, theyinclude instructional worksheets, lists ofinstructional resources and activities, listsof enabling competencies, and lists ofrelated academic skills. Development ofinstructional tools began in 1984, and theynow exist for 66 job areas.Assessment Tools. V-TECS has also de-veloped or acquired banks of test items.Each of the test items is criterion-referencedto a specific duty and task. A test item bankis available for 35 of V-TECSs job areas.The banks include both written (matchingor multiple-choice) and performance-baseditems.V-TECS DIRECT. This is a software pack-age designed for storing and retrieving theV-TECS materials that make up the otherthree components. As of fall 1992, catalogsand instructional tools for about 70 occupa-tions were available on disk. All test itembanks have been available on disk fromtheir initial release.

Development Process for V-TECSCatalogs and Instructional Tools

V-TECS follows a multistage process in devel-oping all of its catalogs and instructional tools.First, the consortium determines the priorities forjob analyses,

10 new product development, and

revisions by collecting statistical informationfrom such sources as the U.S. Department ofLabor and surveying the needs of member states.

Second, V-TECS identifies competency-basedcurriculum materials and other materials for eachjob area that it plans to analyze. These materialshelp V-TECS develop task lists and lists of tools,equipment, and work aids. V-TECS refines thelists by identifying a target population of workersin the particular occupational domain beinganalyzed. A sample of this population is inter-viewed and observed. The tasks are then organ-ized under duty statements.

The task and duty lists and the tools, equip-ment, and work aids lists are then sent to a largersample of the target population in survey form.Surveyed workers are asked whether they per-form the tasks listed on the V-TECS lists andwhether they use the tools, equipment, and aidslisted, V-TECS analyzes the survey data todetermine the percentage of incumbent workersthat performs each task and uses the various tools,equipment, and work aids. Based on these per-centages, final lists are developed.

V-TECS has recently taken steps to improvethe development of task lists, so that they will betruly national in scope. In prior years the listswere validated by surveying people in only a fewstates. However, V-TECS has expanded its sur-veys to include industry people in many states.

X V-TECS estimates that its catalogs cover at least 90 percent of the vt~ati(mal education proglarn areas that exist in the nati(m.

‘> According to the V-TECS Technical Re]krence Handbook: “. . . each ‘task’ is a statement of measurable behavior which describes ameaningful unit of work having a definite beginning and ending point. A ‘duty’ is a broad descriptor under which similar tasks are organ ized. ’The perft~mlance objectives consist of a list or description of the c(mditi(ms under which a task is perftmned, a descripti(m of the task to bePerf(mmcd, and a Perfomlance standard. This standard is “. . . an observable or measurable standard of perf(}mlance deemed appropriate forsuccessful c(m~pleti(m of the task by incumbent workers.

It) A job are:i nla} Include (me or more Dictionary of occupati(mal Title (DOT) job classifications.


Third, V-TECS selects a group of workers toserve on a writing team; the team is responsiblefor specifying performance standards and identi-fying the steps involved in performing eachindividual task in a duty area. Following thisphase, subject matter experts are asked to identifythe cognitive knowledge, psychomotor skills, andwork-related behaviors that are critical to theperformance of each of the tasks. The “enablingcompetencies" and “related academic skills’that eventually appear in the instructional toolsare derived from these knowledge, skills, andbehaviors.

The V-TECS Item BanksAll test items in the V-TECS banks are

developed by member states in accordance witha standard V-TECS development model. Elevenstates have contributed to the development of testitem banks so far, with the V-TECS central officemonitoring quality. If a member state wishes tohave its own state-developed test item banklabeled as a V-TECS item bank, it must documentthat it has gone through the standard V-TECSdevelopment process. V-TECS staff also reviewand edit the state-developed test item banks, onoccasion sending them back to the states forfurther editing. 11

The V-TECS process for developing test itemsincludes several steps:

. Validating Task Lists, Performance Objec-tives, and Performance Steps. The first stepin developing test item banks is to review thecontinued validity of the task lists, perform-ance standards, and performance steps con-tained in the catalog of the job area for whichthe test is to be developed. This is accom-plished by having five to eight workers from

●

●

●

�

different-sized companies and employerswho are experts in the job area review thelists and make minor changes as they seeappropriate.Writing Test Items. V-TECS selects a [earnof test item writers. The team must includeinstructors with recent work experience, thestate technical coordinator or project direc-tor, workers from the domain area, and aV-TECS central office representative, whenavailable. After receiving special training,the writing teams develop both written andperformance items by reviewing the com-pleted task analyses and using V-TECSguidelines to ensure that test items match thetasks.Reviewing and Editing Test Items. The testitems are reviewed by four groups of expertsand revised if necessary. The four groupsinclude the writing team, test item construc-tion experts, subject matter experts, and asample of workers. V-TECS central officestaff review the test items to make sure theymatch the duties and tasks of each of theoccupations and to make sure they areformatted correctly. V-TECS staff then editthe test items and compile the test item blink.Field Testing of Item Bank. All test items arefield tested in schools to check whether theitems are clear, reliable, and free of sexual orracial bias and whether the directions areclear. Test administrators are asked to pro-vide feedback on any difficulties encoun-tered during the field test. Item response datafrom field testing are scored and analyzed foritem difficulty, item discrimination, anddistracter response. 12 These three forms ofanalysis help ensure that test items discrimi-

I I staleS ~]S() hale the ~)p[ion of sharing their state. deve]()~d item banks without having tlle~l labc]e~ as V-TECS item banks. ]n [his case,

V-TECS will help make the banks available to other states, clarifying that these materials may not have been developed according to the standardV-TECS process.

I z Multlp]e-choice ~ue$tions me IYpical]y constmcted I() inc]ude three distract(w respmses, which are p]ausibk? a]lUTMtiV~s I() ~h~ c(’~~~t

answer. There is (rely one correct response. Test takers’ responses to these distractfms are analyzed in the cxmrse of test cxmstructi(m to check(m the qua] ity of the questions and answers provided.


nate between examinees who have receivedinstruction in the area being tested andexaminees who have received no such in-struction.

● Editing and Completing Final Item Bank.The item bank is revised based on the fieldtest results.

This process of ongoing, multistage review byworkers, V-TECS staff, and other experts helps toensure the content validity of the items—that theyare tied directly to real occupational tasks andduties.

As mentioned above, the banks include bothwritten and performance items. The written testsmeasure knowledge and skills, with the emphasison knowledge. There are two categories of writtenitems: those that measure the examinee’s abilityto recall information, and those that test theexaminees’s ability to analyze information. V-TECS supplies the correct answer for the writtenitems.

The performance items also measure bothknowledge and skills but emphasize skills. Someitems focus on the process the student uses, othersfocus on the product that results from the per-formance, and some measure both process andproduct.

Each performance item comes with a descrip-tion of the task that the student must perform. Anevaluator observes student performance using thechecklist, checking off actions that the examineecompletes successfully.

V-TECS does not supply explicit guidelines onwhat a successful completion of a task or piece ofa task should be like. To understand what isexpected of an examinee, the evaluators mustrefer to the ‘‘performance objectives and stand-ards’ for each task; which are contained in theV-TECS Analytical Tools. Agencies administer-ing the test can decide on a minimum level ofperformance required in order to ‘pass. ’ Scoringis thus performed by the user rather than a testingorganization, as in the case of Work Keys.

Use and Availability of Test Item BanksV-TECS test item banks are used by states for

a variety of purposes, including evaluatingschools and programs and certifying program orcourse completion for individual students. Privatecompanies also use some of the V-TECS materi-als, but not as extensively as schools. Entitiesusing a V-TECS item bank select a number ofitems from it for each competency to be assessed,and combine them into a test or assessment for usein classrooms.

V-TECS does not maintain records on thenumber of states, schools, or school districtsusing its materials, nor does the organizationmaintain information on which states are usingtest item banks to meet Perkins Act evaluationrequirements. These types of records would bedifficult to maintain because the materials are soeasily shared.

The V-TECS materials are widely available.Members receive two copies of all V-TECSmaterials. Nonmember states can purchase thematerial at a higher cost. In fact, V-TECSmaterials, including the test item banks, can bepurchased by any entity. The only form ofmarketing V-TECS does is through exhibitingtheir materials at national conferences. For exam-ple, they exhibit at the annual meetings of theAmerican Vocational Association and at theAmerican Society for Training and Development.

The V-TECS organization provides technicalassistance to members and other users of itsmaterials. Based on requests from people inmember states, V-TECS conducts workshops onsuch topics as how to write and review items andhow to interpret student performances and prod-ucts. The organization also offers workshops oncompetency-based vocational education at therequest of a state agency or a group of schools.

In addition, V-TECS sponsors periodic na-tional conferences on competency-based assess-ment and performance standards for vocationaltechnical education. At least once a year, V-TECSholds a workshop for its technical coordinators-


the state agency employees from member stateswho oversee development of V-TECS material intheir state, promote V-TECS materials, and workwith the state educators and business people.

The Future of V-TECSV-TECS is considering whether it should offer

a testing service that would actually constructtests for states. In addition, the organization iscontinuing its efforts to ‘‘fill in the gaps’ in itscurrent titles. For example, in 1991 V-TECSbegan identifying related academic skills, a pro-cess that it plans to continue for all occupations.Another priority is developing test items for alljob areas addressed in V-TECS materials.

NATIONAL OCCUPATIONAL COMPETENCYTESTING INSTITUTE13

Overview of NOCTI and theSOCAT Testing Program

NOCTI was created in the early 1970s as anational, not-for-profit educational corporation.All U.S. states and territories are members,although voting privileges are reserved for statesthat have purchased $100 of NOCTI goods andservices during the past year. States are repre-sented through a consortium of vocational educa-tion officials. These officials are appointed bystate departments of education, state-approvedteacher education institutions, and other agenciessanctioned by the NOCTI Board of Trustees.Member states play a role in developing andmarketing tests.

The original mission of NOCTI was to developexaminations to assess the occupational compe-tencies of vocational education teachers, an areawhere there was a paucity of assessment instru-ments. Today NOCTI is the nation’s primaryprovider of vocational teacher competency exams,

offering 60 specific Teacher Occupational Com-petency Testing (TOCT) examinations.

In the late 1970s, several trends converged toconvince the NOCTI board that there was a needfor competency tests for vocational students.First, many people felt that competency-basedtesting would help to improve vocational educa-tion and demonstrate its value to employers andthe public. There was also emerging interest indeveloping national standards for vocational stu-dent certification.

Responding to these trends, NOCTI formed aconsortium of states to develop a system ofStudent Occupational Competency AchievementTesting (SOCAT) examinations. Since 1978,NOCTI, working with its member states, hasoverseen the development of 71 SOCAT tests.Each test is tied to a specific occupation, includ-ing many of the same occupations covered by theteacher tests. Most of the occupations covered fallwithin one of the traditional vocational educationprogram areas of business, agriculture, homeeconomics, trade and industry, technical andcommunications, marketing and distribution, orhealth.

Using the same methodology, NOCTI alsodevelops Industrial Occupational CompetencyTesting (IOCT) examinations for specific compa-nies. Industries use the IOCT to conduct preem-ployment testing, identify in-house training needs,design specific training programs, certify skills,promote employees, and carry out other purposes.Other customers of NOCTI are the U.S. Job Corpsand the military.

Somewhat over 50 percent of NOCTI’s totalrevenues come from the IOCT tests for industry.Less than 15 percent of total revenues comes fromthe student tests (SOCATs), because NOCTI triesto keep the cost of SOCAT tests affordable forschools. The remaining 35 percent or so ofrevenues comes from the teacher tests and specialprojects.

13 me infomallon” in this Ca$e study is based (m OTA interviews with Scott whitener, presidentichief executive officer, NOCTI, and Evelyn

Wells, assessment specialist, NOCT], January and February 1993.

Chapter 4: Case Studies:

The SOCAT and TOCT Tests

Three Vendors of Tests for Occupational Competencies 77

SOCAT test items are based on core competen-cies considered to be critical for job readiness.These cornpetencies are derived from analysis ofthe specific tasks performed in a particular job forwhich the test is being developed. The content ofthe SOCAT and the TOCT are basically the samefor the same job, except for the level of compe-tency. The student tests measure the skills andknowledge expected of a‘ ‘job-ready’ entry-levelworker, whereas the TOCT tests measure theskills and knowledge expected of an “experi-enced’ worker. The tests are designed to be usedat both the secondary and postsecondary levels.

The SOCAT tests have two components, awritten part and a performance part. Both parts aretied to the knowledge and competencies requiredof job-ready entry-level workers in a givenoccupation. NOCTI encourages states to use boththe written and performance components, be-cause it feels that the written test alone gives anincomplete picture of a student’s skills andknowledge. The written tests are multiple choiceand are designed to assess whether a studentunderstands and can apply the information neededto perform various tasks. The performance testsrequire a student to perform various tasks that anindividual must be able to do to be consideredjob-ready. The instructions for doing the perform-ance tests call for having evaluators/administra-tors from industry judge the quality of both theperformance process and the resulting product.

NOCTI also supplies a third test of thinkingability, if requested by the schools. This Test ofCognitive Skills was developed by CTB/McGraw-Hill and is marketed by them. The test measurescapabilities of analyzing patterns or sequences ofletters, figures, and numbers; recognizing rela-tionships and classifying objects and conceptsaccording to common attributes; recalling previ-ously presented information; and reasoning logi-cally. The items on this test are not tied directlyto the SOCAT tests, but are scored by NOCTI and

reported back to users as part of the SOCAT scorereport.

A major difference from V-TECS is that theSOCAT tests are fixed and secure. V-TECS is anitem bank. The SOCATs are fixed in that theitems are preselected by NOCTI and are the samefor all test takers. Clients return completed teststo NOCTI for scoring and do not receive anyinformation about the correct answers. In thisrespect, the SOCATs are similar to standardizedacademic tests.

Test Development ProcessThe process for developing the TOCT and

SOCAT tests begins with the analysis of occupa-tional tasks. For each job or occupation ad-dressed, NOCTI assembles a test developmentteam. The team consists of secondary and post-secondary vocational educators from at least twostates and industry representatives who are highlycompetent workers familiar with the tasks re-quired in a given occupation. NOCTI selectssome of the industry representatives; state educa-tional agencies select the remaining industryrepresentatives and all of the educators.

These teams then proceed to identify thespecific tasks involved in performing the targetjob or occupation in question by following theDACUM (Developing a Curriculum) process orreviewing existing task lists and merging theminto one. Generally, the DACUM method is usedonly when high-quality task lists do not alreadyexist for an occupation. After prioritizing thetasks, the team members determine which itemsare best evaluated through a performance examand which are best done through a written exam.The actual test items are then written either by theteam members or subject matter specialists hiredfor the task.

Once written, the items are reviewed byNOCTI’s in-house testing experts. A bias reviewis conducted and reading experts ensure that thereading level of the exam matches the levelrequired for the occupation in question. This is


accomplished by comparing the test items withactual written materials that workers use on thejob.

The draft exam is then pilot- and field-tested ina two-stage process. Teachers or schools firstvolunteer to administer the test in a small numberof schools; NOCTI monitors the administration tomake sure the test is operationally sound and thattime limits and other administrative features areappropriate. Then, the exam is field tested on alarger scale in at least two states. Through thistesting, NOCTI seeks to learn whether the itemsare free of colloquialisms or regional biases,whether the test items actually match the skillsrequired for the job, whether the time limits areappropriate, and whether the tasks are up to date.This process helps to ensure the validity of thetest.

NOCTI also conducts an item analysis todetermine which items are not good discrimina-tors among the students—that is, are too easy ortoo hard for them to answer. Suspect items arereviewed by a subject matter or test expert andpossibly revised. Once final refinements andrevisions are made, the test is ready to be soldnationally.

SOCAT tests are reviewed annually by NOCTI.After each annual administration, NOCTI asks forfeedback from industry representatives, teachers,and students involved in the testing. The feedbackis cataloged, and if there is significant criticismof a test, NOCTI reinstitutes a committee ofindustry representatives and educators to reviewthe test and determine whether revision is neces-sary.

The technical quality of the test is addressed atseveral points in the development process. Con-tent validity is achieved by having real workersdevelop task lists and provide feedback on testcontent after they are administered. (NOCTI doesnot check for predictive validity, although theorganization would like to do so if more resourceswere available.)

NOCTI calculates reliability statistics for thewritten tests using the Kuder-Richardson methodfor determining the internal consistency of testitems. Furthermore, as explained in greater detailbelow, NOCTI has developed scoring guidelinesfor the evaluators/administrators of the perform-ance tests.

Use of SOCAT TestsSOCAT test services maybe purchased by any

educational agency, including state agencies orindividual schools. In 1992, the SOCAT test wasadministered to 9,015 secondary and postsecon-dary students in 24 states, with the number perstate ranging from 1 in Delaware to 3,435 inPennsylvania. It is estimated that about one-halfof the states using SOCAT tests administer allthree components: the written, performance, andcognitive skills tests.

This number of secondary and postsecondarystudents taking the SOCAT tests is not large inrelation to the size of vocational education. In1992, there were about 720,000 high schoolseniors who took four credits or more of voca-tional education in their high school careers andtherefore can be considered to be “vocationalstudents." 14

14 ~15 e5tlnlate was obtained a5 fo]]ows.” ‘T’he number of high school seniors in the 199 I -92 schtwl year was 2.4 million. 1[ is reasonable

to assume that the number of seniors was about the same in the 1992-93 school year, since the increase in o~era]l high school enrollmentsbetween 1991-92 and 1992-93 was less than 2 percent. It is also reasonable to assume that 30 percent of these 1992 seniors were “vocationalstudents. ” This is the percentage of 1987 high school seniors who took ibur credits or more of vocational educati(m courses by the time ofgraduati(m, and this has been a fairly stable number since the high school class of 1982. A reasonable estimate of the total number of high sch(w]seniors who might have taken vocational tests in the 1992-93 school year is therefore 720,000. U.S. Department of Education, Office ofEducational Research and Improvement, National Center for Educational Statistics, Digesr oj”Education Statisric.~, /993 (Washingt(m, DC: U.S.Government Printing Office, 1993), tables 2 and 44; and U.S. Department of Education, Office of Educati(mal Research and Improvement,National Center for Education Statistics, Vwwional Education in Ihe Unifed S/ales, 1969-1990 (Washington, DC: U.S. Government Frmtingoffice, April 1992), table 13.


In some cases, the written SOCAT test isadministered as a pretest, and the written andperformance tests are used as post-tests to meas-ure the occupational competency gains of stu-dents who exit a program, as specified in thePerkins Act. Although NOCTI has traditionallydiscouraged the use of the written test alone, in1992 the organization began making the writtentest available for pretesting because of acceler-ated interest in using it to fulfill Perkins require-ments. NOCTI believes that when the written testis used as a pretest and both the written andperformance tests are used as post-tests, valuableinformation about competency gains can begenerated.

In other states, students who do well on theSOCAT test receive special recognition certifi-cates or credentials from the state. These awardsare in addition to the SOCAT certificate, regu-larly provided by NOCTI for each test taker,which carries a printed report of the student’sscores on the back. Both the state-issued certifi-cate and the SOCAT certificate are intended foruse in student portfolios for interviewing withprospective employers.

In other cases, successful completion of ajob-ready SOCAT test may be used to receiveadvanced standing in college or university pro-grams.

In order to use the SOCAT tests, teachers mustensure that they are teaching the skills andknowledge that the tests measure. Since SOCATtests are secure, NOCTI regularly providesschools with support materials and test specimensets to help facilitate test selection.15

Sometimes NOCTI helps states to determinewhether there is a fit between the SOCAT testsand their state vocational curriculum. For exam-ple, NOCTI is currently comparing how therevised task lists developed by a specific statematch the SOCAT tasks. In most cases, however,states are responsible for determining whether the

tests are compatible with their vocational pro-grams.

Administration of the SOCATsMember states that use SOCAT tests receive

test booklets, test manuals, and videotapes dem-onstrating the procedures for administering thewritten and performance examinations. Users canalso contact NOCTI for additional technicalassistance; NOCTI has a test center in every statewith a knowledgeable staff person who cananswer questions about the SOCAT tests. In thewords of the NOCTI president: “When a schoolor a state administers the SOCAT tests, they havethe whole organization behind them. ’

The SOCAT tests are designed to be adminis-tered by local personnel at local school sites. Anyteacher, guidance counselor, or test administratorcan administer the written multiple-choice testand the cognitive skills test. However, as men-tioned above, NOCTI recommends that a ‘‘. . .journey worker, trades person. or business repre-sentative with technical expertise in the occupa-tion should administer the performance test. ’

After test administration, a school sends thetest results back to NOCTI for machine-scoring.Within 2 weeks of receipt, NOCTI ships twotypes of reports to the user, individual studentreports and teacher reports.

Each test taker receives a student report printedon the back of a certificate, which includesseparate scores for the cognitive skills, written,and performance tests. Scores from the cognitiveskills and written tests are presented as thepercentage of questions answered correctly.Scores from the performance test are presented asthe percentage of possible points earned. Thestudent receives an overall percentage score andsubscores for different duty areas or tasks. Forexample, the Electronic Technology written testcontains subscores for “schematic symbols, ’‘‘safety, ‘‘ ‘‘soldering, ’ ‘‘components, ‘ ‘power

I $ ]n tho~c Sltll:itlons ~ h~rc the SOCAT tests do not nl~tch what is being taught, the NOCTI president contends that the pr(~tkll is ~’ith the

currIculum, txcauw SO CAT tests are built around the skills, Imowledge, and c(mnpetencies identified b} indus[rj as tmng rwccssary.


supplies, ’ and other subjects. The student’swritten test scores are also compared, by percent,to the scores of other students in the same class,school, and state and with the whole nation.

For the performance tests, students receiveratings for both their performance on individualtasks and the product(s) they produce. The ratingsspan a scale of 1 through 5, with 1 and 2unsatisfactory, and 3 through 5 satisfactory. Theperformance task report lists all of the tasksperformed with a single number rating next toeach task.

NOCTI has recently developed explicit guide-lines and criteria that examiners can use to assignprocess and product ratings to the test takers’performances. For example, if the student is askedto clean a car assembly, NOCTI now suppliesguidelines on how clean the rotor must be for thestudent to receive a certain score. These guide-lines, which are intended to provide inter-raterreliability, are currently available for about one-half of the performance tests. For the remainder ofthe tests, the evaluator must use subjectivejudgment to rate student performance.

Teachers receive a class report consisting of acomposite of the student reports analyzed byclass, school, state, and the nation, along withstandard deviations and standard error rates.

Future PrioritiesNOCTI is currently developing SOCAT tests

for additional occupational areas and plans tocontinue such expansion. One of the biggestchallenges is keeping the existing tests up to datein view of the tremendous changes occurring inindustry and in occupational skill requirements.

FINDINGSThe testing products of these three organiza-

tions are distinctly different from each other andrepresent the range of testing practices in voca-tional education reasonably well. Work Keys isthe newest of the three systems and is at one endof the range. V-TECS and the SOCAT tests fit the

mold of competency-based testing for job-specific skills, in that both written tests andperformance exercises are included, and the skillsassessed are job specific. Work Keys is focusedon generic workplace skills and is similar to aconventional standardized academic test in themethods of test development that are beingemployed, the strict procedures of test adminis-tration that must be followed, and the closed-ended nature of most of the test items. TheSOCAT tests fall in the middle of the range. LikeWork Keys, each test consists of a fixed set ofitems and must be administered according tostandardized procedures in order for the resultingtest scores to be comparable among test takers.Unlike Work Keys, each test item is directlyrelated to specific work tasks in a particular jobarea. Work Keys tests employ multiple items tomeasure one competence that is generally relatedto all job areas.

Both V-TECS and NOCTI utilize structuredmethods of job analysis to develop their competency-based materials. Work Keys employs a separatesystem of job profiling to ascertain the relevanceof each general competence to a particular job orgroup of jobs. V-TECS is different from NOCTI’sSOCATs and Work Keys in that it is more asystem of resources for competency testing andassessment than a test. V-TECS provides statemembers of the consortium and local programswithin those states with a number of so called“test item banks,’ from which they constructtheir own tests reflecting their own local priori-ties, rather than providing them with tests that aresecure and consist of a fixed set of items. Each testitem bank is specific to a job area or occupation.Both the NOCTI and V-TECS products includeboth short answer, written test items, and someperformance exercises; however, in both cases thewritten tests and the performance items arepackaged separately and the written componentsgenerally predominate.

V-TECS, therefore, does not sell tests butrather models good practices of competencytesting and makes testing resources available to

Chapter 4: Case Studies: Three Vendors of Tests for Occupational Competencies I 81

its 23 member states and local programs withinthose states for them to follow and use inconstructing their own tests.

No firm conclusions can be drawn about thevalidity and reliability of decisions made at thestate and local levels using the tests or testingresources produced by these three vendor organi-zations simply on the basis of the kinds of teststhey produce or the methods of test developmentthey follow. Work Keys will have the most dataof the three vendors from pilot tests showing thereliability of their instruments. On the other hand,research reviews have found that competencytests of the kind produced by V-TECS andNOCTI are significant y correlated with scores onwork sample and hands-on performance tests.16 Itis important to point out, however, that the skillsmeasured by Work Keys are intended to begeneral across jobs, and the skills measured byV-TECS and the SOCATs are specific.

The extent to which the testing resourcesproduced by these three vendor organizations arecurrently being used in vocational educationdiffer substantially. The most concrete estimatesare for the SOCAT tests, since they must bereturned to NOCTI for scoring. The numbers ofSOCAT test takers are not large. In fact, NOCTIreports that SOCATs were taken by 9,015 second-ary and postsecondary students in 1992. Incomparison, the number of high school seniorswho were vocational students was about 720,000in 1992.

The Work Keys system is much newer and nofirm estimates of the number of test takers areavailable yet. At least two states have adoptedparts of the Work Keys system, and more areconsidering the possibilities. Because Work Keysis so different from the SOCAT and V-TECS

products in the skills tested and the methods oftesting, the effects of Work Keys on testing invocational education could be substantial, if asignificant number of states decide to adopt it.The requirements of the 1990 amendments forperformance standards are an important source ofstates’ interest in purchasing Work Keys, asindicated in the interviews conducted by OTA inproducing this chapter.

In the state survey conducted by OTA, de-scribed in chapter 3, state personnel frequentlyreported that substantial efforts are devoted toadapting, redeveloping, and/or expanding thecompetency lists and testing resources producedby V-TECS. The V-TECS materials are used invarious ways in these efforts, along with compe-tency lists and competency tests (or test items)from many other sources. The reason commonlygiven as to why these efforts to adapt and revisematerials obtained from elsewhere are necessaryis that neither the V-TECS materials nor thematerials available from other sources adequatelyreflect the priorities among different areas ofknowledge and skills that are most important inthe state or local program area.

How much genuine need exists for this rein-vention and adaptation of materials developedelsewhere and how much of it is unnecessaryduplication of effort is impossible to say from thedata available to OTA. Local priorities amongdifferent areas of knowledge and skill undoubt-edly differ from state and national priorities, andprocesses of reinvention have frequently beenfound to be essential for the thorough implemen-tation of innovations. “To understand is toinvent’ is perhaps the clearest way of expressingthis frequent finding in studies of implementa-tion. ]7 On the other hand, questions can be raised

lb Alexandra K. Wigd{)r and Bert F. Green, Jr. (eds. ), Per@matrce Assessment jbr the Workp/ace, vtJ1. I (Washington, DC National

Academy press, 199 I), ch. 8, and J.E. Hunter, “Causal Analysis, Cognitive Ability, Job Knowledge, Job Perfomlance, and SupervisorRatings, ” S. Lund) et al. (eds. ), Per@rmance Measure and Theory (Hillsdale, NJ: Lawrence Erlbaum, 1983); R.R. Reilly, “The Va\id~ty andFairness [}f Alternatives to Cognitive Tests,”’ Po/icy /ssues in Emp/oymen? Tes~ing (Boston, MA: Kluwer Academic Publishers, 1993).

I T Paul Bemlan and Ml]brcy #la]] in McLaughlin, Federal Progranls supporting Edu(,atl~~/ cha~ge, VO/. ]V: 7’}Jc Frndirr~.T in /?(?\’ltW’,

prepared for the U.S. office of Educatifm, Department of Health, Educati(m, and Welfare, R- 1589/4-HEW (Santa M(mlca, CA. Rand Corp.,April 1975).


about the consequences of this process of reinven-tion for the comparability of assessment resultsfrom place to place and just how necessary anduseful it is.

The main conclusion, though, is that theinfluence of the products of these three vendors ofvocational tests on testing and assessment prac-tices in vocational education is limited, at least inrelation to all states and all students enrolled invocational education. V-TECS appears to havethe greatest influence through its deliberate strat-egy of modeling good competency testing andassessment practices for states and local programsto follow, and providing them with competencylists and test item banks to be used as resources indeveloping their own programs of testing andassessment. However, only 23 states are membersof the V-TECS consortium and test item banks areavailable for only 35 of the over 200 occupationalareas in which competency lists are available.

V-TECS sells its testing materials to any state oranyone who wishes to buy them. While NOCTI asan organization has many other clients andcustomers for its testing products, the number ofstudents currently taking their SOCAT test is verylimited. Work Keys is too new to know howextensive its impact will actually be, but at leasttwo states (Ohio and Tennessee) have adoptedportions of it for statewide use and many more areconsidering its use.

It is also important to point out that someindividual states, such as Oklahoma, which has anextensive program of test development and distri-bution, also provide competency tests and re-sources for testing to other states in various ways.The three vendors described here are the mostvisible vendors of testing resources in vocationaleducation but not necessarily the only suchsource.

I

BroadTechnical

Skills 5

T he world of work is changing rapidly, and in ways thatare difficult to anticipate. In response to this change andin an effort to make vocational education a method ofequipping young people for more than one occupational

future, policymakers and educators have begun to seek skilltraining and preparation that applies to many jobs. The languageof the 1990 amendments to the Carl D. Perkins VocationalEducation Act refers to ‘ ‘broad technical skills, ” The Office ofTechnology Assessment (OTA) was asked to determine howmuch testing and assessment was aimed at measuring themastery of these skills. This concept is a new one, however, andno agreement exists on what skills are truly lifetime occupationalskills, or on how those skills are best acquired.

Very few states are moving toward organizing their vocationalprograms around broad technical skills. One of the principalreasons for this is the lack of research and proven experienceshowing what broad technical skills are, how they can be taughtand assessed, and what the advantages and disadvantages of sucha change would be.

The implications of organizing vocational programs aroundbroad technical skills are potentially far reaching. Large seg-ments of the training enterprise in this country are oriented toproviding people with just the skills needed for particular jobsinstead of careers. By organizing more of training around thedevelopment of broad technical skills, a greater proportion of thetotal training effort might be directed to the preparation of peoplefor careers and learning on the job rather than for specificentry-level positions. This lengthening of the ‘ ‘skill horizon’ ofvocational education and other training efforts would require

83


placing greater emphasis on developing capabili-ties for thinking, responding to the unexpected,and taking initiative, in addition to acquiring theskills needed for performing specific occupa-tional tasks.

Broadening the technical and occupationalskills around which vocational education is or-ganized could also help to facilitate the integra-tion of academic and vocational education. Byshifting the content of vocational preparationtoward the development of capabilities that re-quire more thinking and knowledge application,opportunities should expand for connecting occu-pational and academic content.

Whether the potential advantages from broad-ening the technical skills concept can be realizedis an open question. Certain major fields ofvocational education have historically been muchmore broadly oriented than others. Agriculture isa clear example, while the trades have been morespecialized. The recent pressures on all fields,including agriculture, have been to become morecompetency based and job oriented.

The main purpose of this chapter is to suggestsome alternative ways of beginning to definebroad technical skills. Much more thought needsto be given to this basic concept than has beengiven to it so far.

Broad technical skills are different from ge-neric workplace skills in certain key respects. Themain difference is that generic workplace skillshave been defined as essentially the same acrossindustries. Whether people who have these ge-neric workplace skills in one industry or occupa-tional area in fact demonstrate them with equalproficiency in other occupational or industryareas has not been shown.

The concept of broad technical skills reflectsmuch more directly the content of technology,information, and methods of organization withinan industry or group of industries. It is possiblethat these industry groups could be as broad as

health care, financial services, manufacturing,hospitality, and agribusiness; or it may be neces-sary to limit them to occupational areas withinindustry groups. Both broad technical skills andgeneric workplace skills may be important forproductivity in the workplace and career develop-ment. Broad technical skills will be defined in thisreport to include not only the specific technicalcontent of job performance but background knowl-edge of the technological, social, and economicdevelopment of the industry.

Five different approaches to defining broadtechnical skills are described in this chapter. Theapproaches are: vocational abilities and aptitudes,occupational maps, core occupational skills, de-sign and technology, and cognitive problemsolving. They are founded on fundamentallydifferent assumptions about the nature of skills,relationships between skills and the contexts inwhich they are used, and relationships betweengeneral and specific skills. The approaches differgreatly in the extent to which they are supportedby research; some reflect extensive analysis andothers exist primarily as practices or even ideasfor improvements in practice, These approaches ●

do not exhaust the range of possibilities.

VOCATIONAL APTITUDESOne approach to defining broad technical skills

is to think of them as vocational aptitudes. Thethesis of vocational aptitude testing is that peopleperform best in the jobs or occupational fields forwhich they have the most “abilities’ and thatthese abilities can be identified through tests. Theinstruments developed for assessing vocationalaptitudes employ the techniques of testing forachievement and multiple abilities developedfrom research in psychology on mental testingand intelligence. 1 Aptitude tests differ from testsof achievement mainly in the uses that are madeof the results rather than the abilities measured

] be J. Cmnbach, L“ssentiu/s of Psychological Te.wing (New York, NY: Harper and Row, 1990), ch. 10.

Chapter 5: Broad Technical Skills I 85

and the nature or content of the tests employed.2

Aptitude tests are constructed to measure thelikelihood that a person will perform well in thefuture in different kinds of jobs or occupationalfields, while achievement tests are primarilyintended to measure what a person has learned inresponse to education, such as a training programfor computer technicians or mathematics up to acertain grade in high schools

Most aptitude tests consist of several individ-ual tests for different areas of knowledge orability. The tests are selected because of theircontribution to the accuracy with which perform-ance in a subsequent job or occupation can bepredicted. These component tests of the overalltest ‘‘battery’ typically include measures of whatcan be considered pure cognitive ability, such asmemory span, spatial reasoning ability, or percep-tual speed; academically related achievement,such as vocabulary, reading comprehension, orarithmetic problem solving; perceptual-motorskills, such as manual dexterity or eye-handcoordination; and other areas of knowledge orskill more specific to the occupations of interest.4

These four domains of ability are different froma person’s vocational interests and personalitycharacteristics, which can also be measuredthrough another kind of test called an ‘‘interestinventory, such as mechanical comprehension

or computer knowledge. Both vocational aptitudetests and interest inventories have been shown tobe significantly related to job performance.s

Tests of vocational aptitude are developedprimarily for purposes of career counseling orscreening people for selection into jobs or occu-pational areas. Because of their origins in mentaltesting, the format of most aptitude tests iswritten, except for measures of perceptual-motorskills. Because of the written format, aptitudetests can be group administered and are thereforerelatively inexpensive. This is essential given theprimary use of this kind of test, which is screeninglarge numbers of job applicants.

One view is that the knowledge and abilitiesmeasured through vocational aptitude testing areprerequisites for success in different kinds of jobs.Another interpretation is that measurement ofknowledge and abilities provides an indication ofthe individual’s capability for learning new tasksthat are similar to those that will be performed onthe job. This latter interpretation opens a door tomaking connections among the different ap-proaches to broad technical skills.

Vocational aptitudes differ from job compe-tencies in that they are presumed to underliepeople’s performance of tasks on the job. Jobtasks describe specific behaviors that peopleexhibit in response to certain work goals or

2 ‘‘Achievement and aptitude tests are not fundamentally different. . . .Tests at one end of the aptitude-achievement c(mtmuum can bedistinguished from tests at the other end primarily in terms of purpose. For example, a test for mechanical aptitude [c(mld] be included m abattery of tests for selecting among appl icants for pilot training since knowledge of mechanical principles has been found to be related I(J successin flylng. A sirmlar test [could] be given at the end of a course in mechanics as an achievement test intended to measure what was learned inthe course.” Nati(mal Academy of Sciences Committee on Ability, Abilily Tes(in,g: Uses, Consequences, and Confrot’ersies: Parr 1: Repor!oj’lhe Comm~l/ee (Washington, DC: Nati(mal Academy Press, 1982), p. 27.

~ Anne Anastasi, P~?chologica/ Tes~ing, 6th ed. (New York, NY: Macmillan Publishing Co., 1988), p. 412.4 Psychologists” frequently distinguish between “fluid” intelligence, such as spatial ability, visualizati(m ability, sh(m-term mcrmwy, and

so f(mh, and ‘‘crystal] ized’ intelligence, which is the product of experience, such as arithmetic reasoning and word knowledge. The theoryis that people use their fluid abilities to obtain crystallized abilities. See, for example, Richard E. Snow, “Aptitude Theory: Yesterday, Today,and Tomorrow, fi’duca~ional Psycho/ogi.~t, vol. 27, No. 1, 1992, pp. 5-32. A wide range of predictor variables for vocational aptitude testingis discussed in Nommn G. Peters(m et al., ‘‘project A: Specification of the Predictor Domain and Development of New Selecti(m Class ificati(mTests, ” Personne/ Psycho/og), vol. 43, No. 2, summer 1990, pp. 247-276.

$ Generally, the findings are [hat cognitive and perceptual-psychomotor ability tests provide the best prediction of job-spec]fic and generaltask proficiency, while measures of temperament and Persimality are the best predictors of giving extra effort, suppmting peers, and exhibitingPers(mal discipline. Scores derived from interest inventories c(wrelate more highly with task proficiency than with dmm)nstrating effort andpeer suppmt. Jeffrey J. McHenry et al., ‘‘project A Validity Results: The Relationship Between Predictor and Criteri(m Il)mains, PersonnelP\ytholog), vOI. 43, N(). 2, summer 1990, pp. 33 S-354.


demands of the job environment, and job compe-tencies are tasks that people have demonstratedthat they are able to perform repeatedly. Theconcept of vocational aptitudes is that the per-formance of these specific tasks requires a certaincombination of abilities, or traits, as they aresometimes called, and that different tasks requiredifferent combinations of these abilities or traits.Their mental basis is what lends these traits orabilities their presumed generality of applicationacross many different kinds of specific tasks. Jobcompetencies, on the other hand, are not generallypresumed to be transferable across jobs or worksituations in the ways that they are usuallydefined. There is no model of transfer built intothe job competency model. Aptitudes are thus atone extreme essentially fixed (though learnable)qualities of the mind, while job competencies (orjob tasks) describe highly specific, observable jobbehaviors that vary greatly from job to job. A“giant leap” of faith is required to translatebetween the two.6

Two approaches to filling this gap are de-scribed later in this chapter. One is occupationalmapping, which involves broadening the defini-tion of job tasks to cover whole clusters ofoccupations and revising the nature of the taskdescriptions to include thinking as well as behav-ior. The other approach is the cognitive skillsmodel, which provides a detailed description ofthe thinking processes involved in acquiring theknowledge required in a job domain, and becom-ing expert at performing a range of tasks in thedomain and responding to a variety of new taskdemands.

Vocational aptitudes are important to considerin defining broad technical skills because of theway the tests are developed. A number ofdifferent areas of knowledge and abilities are

hypothesized to be significant predictors of per-formance in a range of jobs or occupational areas.Each of the abilities is then described in terms ofa “construct’ or hypothesized relationship be-tween processes of thinking and observableaspects of performance. Subtests for each of theseconstructs are then developed and assembled intoa larger test for administration to a sample ofindividuals to obtain information about how theirscores on the subtests are related to measures oftheir performance. Methods of statistical infer-ence are employed to determine which of thesubtests provide the greatest capability for distin-guishing between high and low performers acrossthe range of jobs or occupations selected. Con-structs found through this process to be highlycorrelated with performance that is ‘‘technicalin nature can then be considered candidates forbroad technical skills.

One important example of a vocational apti-tude test is the Armed Services VocationalAptitude Battery (ASVAB). The ASVAB is usedby the military for selecting personnel into thearmed forces and assigning them to jobs andinitial training. It consists of 10 subtests, as shownin table 5-1.

The military also provides the ASVAB free ofcharge to high schools as the main part of anationwide student career counseling program.Over a million high school students take theASVAB every year (out of, for example, theapproximately 2.4 million seniors in 19927). TheASVAB is administered in the schools by person-nel from the Military Entrance Processing Com-mand and scored by them. Each school receivesin return a report on students’ performance andeach student a complete profile of his or herscores, a guide to military careers, and carefully

G Alexandra K. Wigdor and Bert F. Green, Jr. (eds. ) Perji)rnIanteA ssessmenf in Ihe Workplace, Vol. I (Washington, DC: Natitmal AcademyPress, 1991 ), p. 85.

7 Sec f(x)tnote 14 in ch. 4 of this report.

Chapter 5: Broad Technical Skills 87

Table 5-1-Subtest Content Areas of the Armed Services Vocational Aptitude Battery

Subtest content area Description of the subtest

General science (GS)Arithmetic reasoning (AR)Word knowledge (WK)

Paragraph comprehension (PC)

Numerical operations (NO)

Coding speed (CS)

Auto and shop information (AS)

Mathematics knowledge (MK)Mechanical comprehension (MC)Electronics information (El)

Knowledge of the physical and biological sciencesWord problems emphasizing mathematical reasoning rather than mathematical knowledgeUnderstanding the meaning of words, i.e., vocabularyPresentation of short paragraphs followed by multiple-choice itemsSpeeded test of four arithmetic operationsSpeeded tests of matching words and four-digit numbersKnowledge of auto mechanics, shop practices, and tool functions in verbal and pictorial

termsKnowledge of algebra, geometry, and fractionsUnderstanding of mechanical principles, such as gears, levers, and pulleysKnowledge of electronics and radio principles in verbal and pictorial terms

SOURCE: John R Welsh and Susan K. Kuclnkas, Armed Serwces VocatJona/ Battery: /rrtegratlve Review of Va/dlty Stud/es, AFHRL-TR-90-22

(Brooks Air Force Base, TX: Human Resources Laboratory, July 1990), table 2.

prepared instructions explaining how they can usethe information to decide which military occupa-tions they are best qualified to pursue.8

Scores on the ASVAB have been shown topredict job performance reasonably well using anumber of different criteria: grades in trainingschool; scores on job-specific, hands-on perform-ance assessments; scores on job knowledge tests;9

supervisor ratings; field proficiency ratings; andmany other variables. 10 The multiple correlationbetween ASVAB scores and normalized scores

on the job performance assessments is typically inthe neighborhood of 0.6, which the NationalAcademy of Sciences has concluded is a “usefulamoun t .11 This correlation of 0.6, with somevariation higher and lower, has held across manydifferent kinds of military jobsl2 and to someextent equivalent civil i an jobs. 13 The total amountof variance in the criterion variable explained bythis correlation of 0.6, whether it is grades intraining school or hands-on job performance, is

~ [1 ,S, ~.p:i~lllent of ~. fcn~e, ~.fcnse ManpJ~,~r Data Center, C~l{nSe/~r.~ ~anU41/~~~r (he Armed Ser),i(,es Vo(’ational Apt[[udc ~allcr>t,

DOD 1304. 12-L-ASTP-CM (Ntmh Chicago, IL CJ.S. Military Entrance Pn)cessing C(mm~and, July 1992).

‘) Jtjb Anew ledge tests at-e essentially the same type of test as the job competency test discussed in ch. 3.

I [j ch:~r]()[t~ C:imphell et al., ‘‘Devel(qment of Multiple Job Performance Measures in a Representative Sample of Jobs,’” Personnelpl).<./lo/oq)”, ~01, ~~, N(}. ~, ~ulllnler 1990, pp. 277-300. See also Paul W. May ber-ry and Neil B. Carey, Rc/ati(~n.T~llp ~ef~icen J’~L~~fl~ and4.Mc(han/(a/ ,W(~irl[en(~n(c J(Jh Pcrjimnance, CRM 92-1929 (Alexandria, VA: Center for Naval Analysis, March 1993). An cxtcnslvc set (}fhands-on I(Ib performance measures was dcvcl(pxi for the joint service Ji~b Perfomlance Measurement project (~~crseen by the Nati(malAcademy of Science\, v.hlch are utilized in these and many of the other studies reported in this chapter.

I I Wlgdor ~n~ ~rccn (cds. ), op. cit., fotm)te 6, p. 163.

I ~ [bl~, p 15 ] Ml]t{)n Maler, ,~f[/llar}, Apf//f44je 7k,Yfing: The PUS( Fifil Years, DMDC Technical Report 93-007 (MonI~r~y, CA ~’f~ns~

Manp)wcr Data Center, June 1993), p. 6; and Paul W. May berry, Validation of ASWAB Against Irrtantrj’ Job Perjurrnance, CRM 90-182(Alcxandr]a, L’A Center for Natal Analysis, December 1990). The predictive validity ranges from 0.2 or higher to over 0.70, but in most jobsIS abou[ ().6. lbld., p. 1263, These c(u-rclati(ms are all c(wrected for ‘ ‘restriction of range, ’ according to the prx)ecdures recommended in StephanDunbar and R[Jbmt Llnn, ‘ ‘Range Restricti(m Adjustments in the Prediction of Military Job Perfom~ance,” Pet-jl)rrnance As.ve.ssrmnt in theWi)rLp/a(c, ki)/. //, Alcxamlra K. Wlgdor and Bert F. Green, Jr. (eds. ) (Washingt(m, DC: National Academy Press, 1991), pp. 127-157. Thec(m-ectl(~n f~ )r rcstrlctltm of range ln~ ol~es adjusting the computed c(melati(m between ASVAB scores and the criteri(m variable for the factthat pcr[(~rmancc sc~mx :ire ai ailable (rely for Individuals who have been selected fm the military. Without c(wrecting for the restrictltm of range,the pruficti~ c \ alidtty of [he AS\’AB ]s in [he neighbt)rh(x)d t)f 0.4. corrections” are also sometimes made for unrcllabillt~ of the crlttn-i(mmezisure In s(~nw anal~ ws of prcdictik e validity.

i 1 R 1 ~{, )lrllgren ~ind MR. Dal]do~, A v~/id~[lor? ~jt [/le A,WAIJ Againsf $’uperl’isors’ Rafings in Cl\’1/lan 0~’~’11/MtI’On.$ (palo ‘]to, CA. .Amcrlcan ]nstltutcs f{}r Research. 1993),


generally about 25 percent.14 The predictivevalidities of the ASVAB are somewhat higher forgrades in training school than for hands-onperformance assessments (0.1 to 0.2 higher), andsomewhat lower but much more variable for fieldproficiency ratings and supervisor ratings. ’s

Research in the Marines shows that the predic-tive value of the ASVAB continues after the firstterm of enlistment for both infantry and leader-ship personnel. The main finding is that over 3years of experience are required for personnelwith low tested aptitude on the ASVAB to reachthe same level of proficiency on the job perform-ance assessments as high aptitude personneldemonstrate in their first year.l6

The critical question for the purposes of thischapter is the contribution of the three technicallyoriented aptitudes to job performance in compari-son to the academically and cognitively orientedaptitudes, and the extent to which these technicalaptitudes are more critical for performance insome occupations than others.

Factor analysis shows that scores on the 10different tests of the ASVAB can be reduced tofour intercorrelated factors: verbal aptitude (gen-eral science, word knowledge, and paragraphcomprehension), mathematical aptitude (arithme-tic reasoning and mathematics knowledge), cog-

nitive speed (numerical operations and codingspeed), and technical aptitude (auto and shopinformation, mechanical comprehension, and elec-tronics information). ’7 The issue for this report ishow much ability in this fourth domain oftechnical knowledge and skills contributes to thepredictive validity of the ASVAB beyond what iscontributed by the more academically and cogni-tively oriented domains.ls

The actual content of the tests for technicallyoriented aptitudes is illustrated in box 5-A. Thetest questions shown indicate the nature of theknowledge and skills tested in these areas.

Hunter et al. have asserted that the 10 aptitudedomains of the ASVAB can be reduced to only 1factor of general ability, because they do notcontribute separately to the predictive power ofthe ASVAB across different military occupa-tions.19 Their research shows that statistical

differences in the numbers of individuals sampledand other such ‘artifacts’ of the research designsaccount for all of the differences previously foundin the ability of subtests of the ASVAB to predictperformance among occupations. By correctingfor all of these statistical artifacts, no morevariance in performance scores could be ex-plained using separate scores on the 10 differentsubtests than could can be explained by a single

I Ll D. R. Divgl and pau] w. May berry, 7’he R~le of Aptitude Fa~,(~rs in predictin~ Hands-on Job Perjbrmance, CRM 69-215 (Alexandria,

VA: Center for Naval Analysis, March 1990), p. 5; and Lauress Wise. “The Validity of Test Scores for Selecting and Classifying EnlistedRecruits, ” Test Policy in Defense: Lessons From the Military for Education, Trainin~, and Employment, Bernard Gifford and Linda C. Wing(eds.), National Commission on Testing and Public Policy (Boston, MA: Kluwer Academic Publishers, 1994), table 15, pp. 221-260.

IS John E. Hunter, “A Causal Analysis of Cognitive Ability, Job Knowledge, Job Performance, and Superior Ratings,” PerjimnanceMeasurement and 7’heory, F. Landy et al. (eds. ) (Hillsdale, NJ: Erlbaum, 1993). See also WigdOr and Green (eds. ), op. cit., footnote 6, table8- 10; and May berry, op. cit., footnote 12, table 3.

16 pau] w. May~q, Ana/y~is ad Predit,tion of Infantry Unit Leaders’ Performance, CRM 91-99 (Alexandria, VA: Center for Naval

Analysis, June 1991 ), figure I; and May bemy, op. cit., footnote 12, figure 1.

17 peter H. Sto]off,” Fa(.t~r Analysis of A~AB Form &I In the 19&l DOD Reference p~pulotiun, CNA Memorandum 83-3135 (Alexandria,

VA: Center for Naval Analysis, August 1983).IH The ASVAB Cumen[ly has n{) test for ~rceptual-psychon~()[()r or ~patial abilities, as do other aptitude tests like the General Aptitude Test

Battery. Consideration is being given to adding “assembling objects ‘‘ as a spatial reasoning test to the ASVAB.

1~ John E. Hunteret al., ‘‘The Validity of the Armed Services Vocational Aptitude Battery (ASVAB) for Civilian and Military Occupations, ’U.S. Air Force Contract NW F416689-83-C-0025, August 1985. See also Lee J. Cronbach, “Five Perspectives on Validity Argument, ” TestVa/idity, Howard Wainerand Henry J. Braun (eds. ) (Hillsdale, NJ: Lawrence Erlbaum Associates, 1988), pp. 1- 18; Cronbach, op. cit., footnote1, pp. 293-398; and Malcolm Jones and James A. Eacles, “Predicting Training Success: Not Much More Than g,” Personnel P.s>chology, vol.44, 1991, pp. 321-332.


Box 5-A-Examples of the Content of Test Items on Subtest of the ASVABfor Broad Technical Skills

Auto and Shop lnformation-. A car uses too much oil when which parts are worn?

(a) pistons (b) piston rings (c) main bearings (d) connecting rods. The saw shown above is used mainly to cut:

(a) plywood (b) odd-shaped holes in wood (c) along the grain of the wood(d) across the grain of the wood

Mechanical Comprehension-. In this arrangement of pulleys (a system of pulleys with different diameters is shown), which pulley turns

faster?(a) pulley A (b) pulley B (c) pulley C (d) pulley D

. Which post holds up the greater load? (a load atone end of a beam supported by two posts is shown)(a) post A (b) post B (c) both equal (d) not clear

E/ectronics /nformation-. Which of the following has the least resistance?

(a) wood (b) iron (c) rubber (d) silver. In the circuit shown (a battery connected to a resistor) the resistance is 100 ohms and the current is 0.1

amperes. What is the voltage?(a) 5 (b) 10 (c) 100 (d) 1,000 Volts

SOURCE: U.S. Department of Defense, Defense Manpower Data Center, Counselor Manua/ for ttre Armed Services Vocationa/Aptitude6attery, DOD 1304.12-L-ASTP-CM (North Chicago, IL: U.S. Military Entrance Processirrg Command, July 1992), pp. 131-135.

factor of general ability. This is called the validity the total variation in performance explained bygeneralization hypothesis.

More recent research using new hands-onmeasures of job performance contradicts thishypothesis with regard to technical aptitudes.This newer research shows that the contributionof technical aptitude to the predictive validity ofthe overall test battery is significant. This impliesthat the technical aptitude measured by theASVAB is different from the more purely cogni-tive and academically oriented areas of the test.Using new testing methodologies developed forthe Marine Corps Job Performance MeasurementProject, which was overseen by a committee ofthe National Academy of Sciences, Divgi andMay berry have recently found in a study ofMarine personnel that the technical aptitudecomposite of the ASVAB explains 10 percent of

the whole test.20 The statistical model they usedfirst accounts for all of the variation in perform-ance that can be accounted for by one generalfactor of ability and then, secondarily, for all ofthe additional variation that can be accounted forby a composite of the three technical domainsconsidered together (auto and shop information,mechanical comprehension, and electronics in-formation). This is a conservative estimate of theseparate effects of the technical aptitude compos-ite on performance. They conclude that one ofmain reasons why Hunter et al. found that generalability accounts for all of the variance in perform-ance scores is that they used grades in trainingschool as their criterion variable rather thanhands-on performance tests. When grades intraining school are used as the criterion variable,

10 Di\,gl and May b~my, op. cit., footnote 14, tab]e z.


Divigi and May berry find that the technicalcomposite adds little to the predictive validity ofthe ASVAB.21

Similar results were found in another study ofMarine infantry by the National Academy ofSciences. When job knowledge tests were used asthe criterion variable, the three major, “non-speeded’ factors of the ASVAB have about thesame predictive validities of about 0.73 (the threenon-speeded composites are mathematical, ver-bal, and technical aptitude) .22 When hands-onperformance tests are used, the predictive validi-ties of the same three factors varied significantly.The correlation of technical aptitude with thehands-on performance test results was the highestamong the three composites at 0.65, while theother two were about 0.57.

The validity of the ASVAB for predictingperformance in mechanical occupations in theMarines has been studied by Mayberry and Carey.They find that, for hands-on performance tests,the predictive validity of the mechanical mainte-nance composite of the ASVAB used by theMarines 23 averaged 0.068 higher than the nextbest composite, which is General Technical(GT),24 and over 0.15 higher than the ArmedForces Qualification Test (AFQT), which is ameasure of general ability used by the military .25These translate to an average of 11 percent aboveGT and 30.6 percent above the AFQT. The othercomposites were clerical and electronics.26 Nosuch differences were found in predicting jobknowledge test scores or grade point averages in

training school. In other words, the technicalaptitude measured through performance tests isdifferent from general intelligence, while thetechnical aptitude measured through written jobcompetency tests or grades in training school isnot.

Another source of evidence for the differentialvalidity of the ASVAB can be obtained bycomparing the predictive validity of the subtestsacross occupations, as shown in figure 5-1. Thevalidities of the different subtests are higher inoccupations where the nature of the job taskscorresponds with the nature of the subtest. Evi-dence like this does not prove the differentialvalidity of the ASVAB but strongly suggests thatit exists.27 For infantryman, there are few differ-ences in predicted hands-on performance amongthe 10 individual subtests, except for mechanicalcomprehension, which is noticeably higher thanthe rest. This agrees with the findings of May-berry and Carey. For administrative occupations,the validities of arithmetic reasoning and mathe-matical knowledge are substantially higher thanfor all of the other subtests and the otheroccupations. For mechanical jobs, the technicalcomposite is by far the highest and all of the othervalidities are lower than for all other occupations.

These results from psychometric studies havebeen confirmed using wage data for civilianoccupations. Using the National LongitudinalStudy, one researcher found that both wages andearnings in civilian occupations are significantlycorrelated with scores on the mechanical, elec-

ZI Ibid., p. 9.

~z Wigdor and Green (eds.), op. cit., footnote 6, figure 8-4. The study was based on P.W. May berry, /nterim Result.rjor Mmne L“orps JobPerjimnance Measurement Project, Research Memorandum 88-37 (Alexandria, VA: Center for Naval Analysis, 1988).

zl me mechanical nlalntenance conlP)sitc consists of the total AS VAB sc(wes for arithmetic reasoning, auto and shop infomlatitm,

mechanical comprehension, and electronics infornlation.2 4 me General Technical Conlp)slte consists of arlthn~etic reasoning, word knowledge, p a r a g r a p h C(mpretwnskm, a n d mech,lnical

comprehension.

25 May berry and Carey, op. cit., footnote 10, table 13. The predictive validities of the mechanical composite were 0.64 or greakr ft~r f(mrof the five mechanical (~cupations studied and one validity was 0.70.

lb Clerical consists of the mathematics knowledge, word km~wledge, paragraph c(mlprehension, and code speeding subtests, and clectr(micsis arithmetic reasoning, general science, mathematics knowledge, and elt:ctronics infomlation.

27 Wigdor and Green (eds. ), op. cit., f(xxm)te 6, p. 171.

Figure 5-l-Correlations of ASVAB Tests WithHands-On Performance Test Scores

(Army occupations)

0.41I Infantry ~ “---”~I 4- Administration

0.3 + ● Mechanic

0.1

0 I I I I IGS WK PC AR MK AS MC El

ASVAO subtest

KEY:

GS = General science MK . Mathematics knowledgeWK = Word knowledge AS = Auto/shopPC . Paragraph comprehension MC . Mechanical comprehensionAR = Arithmetic reasoning El = Electronics informatlon

SOURCE: Alexandra K. Wlgdor and Bert F. Green, Jr. (eds.),Performance Assessment for the Workplace, Vol. / (Washington, DC:National Academy Press, 1991 ), p. 171.

tronics, and shop information subtests of theASVAB.28 No effects were found for scores onthe academic and cognitive subtests, but they arehighly correlated with years of schooling, whichwere included in the analysis as a control variable.The independent correlation of the technicalaptitude with wages provides strong indication ofits independence of the academic and cognitiveaptitudes of the ASVAB.

Considered altogether, these studies providesubstantial evidence that the technical aptitudemeasured by the ASVAB, whose content isrepresented by the test questions in box 5-A, issignificantly related to job performance andsubstantially different from academic and purelycognitive aptitudes.


OCCUPATIONAL MAPSA second concept of broad occupational skills

is being developed by the American ElectronicsAssociation (AEA) with support from the U.S.Department of Labor. The effort is one of severalpilot projects being supported by the Departmentsof Labor and Education to develop and demon-strate the feasibility of industry skill standards. Inthese projects, grantees are urged to organize theirstandard-setting efforts around what the govern-ment has called ‘‘broad occupational areas. Forthe purposes of this report, the approach will becalled “occupational mapping. ’

AEA’s approach to skill standards involvesreconstructing the job competency model aroundgenerically defined job tasks that cut across arange of occupations, and including categories ofrelated knowledge and technical skills. The newmodel is intended to provide the industry with themeans to ‘‘speak with one voice’ on the skillsneeded for careers within industry and yet reflectthe major differences in specific contexts ofemployment and jobs. Under the approach, itmust be possible to demonstrate the technicalaccuracy of the resulting skill standards—that thestandards are related to high performance in theworkplace. The skill standards are also expectedto be useful for a wide variety of human resourceplanning and development needs within theindustry other than certification of the compe-tence of individuals.

The orientation of the AEA approach to defin-ing job tasks is borrowed from the job compe-tency model. However, the tasks are identifiedfrom the top down within broad clusters ofindustry jobs rather than the bottom up withinnarrow categories as in the job competencymodel. For the AEA project, three broad clustersof occupations have been chosen: manufacturingspecialist, pre- and post-sales analyst, and ad-ministrative/information services support ana-lyst. AEA estimates that employment in these

lx Jt~hn Bishop, Center for Advanced Human Resource Studies, c(~rne]i university, “Educati(mal Refornl and Technical Educati(m, ”wt)rklng papr N(). 93-04, 1993.


Box 5-B--Broad Occupationai Tasks for Manufacturing SpecialistsKey Purpose

Develop, manufacture, deliver, and improve electronics-related products and processes that meet or exceedcustomer needs.

Critical FunctionsEnsure production process meets business requirements.Initiate and sustain communication processes and procedures.Establish customer needs.Determine design workability and manufacturability.Use human resources to manage work flow.Select, obtain, and optimize available machines and equipment to meet product process requirements.Make products that meet customer specifications.

Includes Jobs Such asProduction associate, operator, production technician, and assembler.

ScenarioManufacturing specialists are on the front line of the manufacturing operation. They work as individuals, or

increasingly, as members of “self-managed” teams. They often play a key role in work scheduling and resourcemanagement within their work group.

They use a variety of quality, cost-reduction, and cycle-time productivity tools and processes. Their directcommunication with internal and external customers and other specialist staff is increasing.SOURCE: American Electronics Association, draft internal memorandum, November 1993.

three occupational areas covers about 60 percent particular jobs within the three broad industryof all jobs at skill levels not requiring a baccalau-reate degree in the computer, software, semicon-ductor, consumer electronics, and telecommuni-cations industries.

This “mapping” of the broad occupationalclusters into tasks is done by industry representa-tives. The process involves first identifying themajor purpose of work in each area and a limitednumber of ‘‘critical functions’ defining thecontent of work output. An example of the majorpurpose and the critical functions for manufactur-ing specialists is shown in box 5-B.

The next step in the mapping process is toidentify a limited number of activities that mustbe performed in order to achieve each of thecritical functions in the cluster area and explicitcriteria for knowing when the activities are beingperformed well. These criteria will provide thebasis for setting performance-level standards for

clusters. The specific standards adopted willdiffer among job areas and segments of theindustry, but will remain consistent within thetask-oriented framework. In the end, the mappingprocess will result in skill definitions similar tothe job competency model, but guided by aframework of critical functions and performancecriteria. Each critical function, together with theassociated set of activities and performancecriteria, is called a competency module. Thesecompetency modules will be the basic unit of skillcertification. An example of the competencymodule for the second critical function of manu-facturing specialists is shown in box 5-C.

The criteria defining when critical functionsare performed well are one of the major ways inwhich the occupational mapping strategy of AEAdiffers from the job competency approach. In thejob competency model, the criteria are frequently


Box 5-C--Activities and Performance Criteria for Manufacturing Specialists:Ensure Production Process Meets Business Requirements Competency Module

Integrate Improvement Process Into Each Critical Function● Quality monitoring and improvement processes are performed and are documented according to company

procedures.. Deviation and root cause of deviation are identified from ongoing analyses of processes.. Recommendations for process improvement are documented, approved, and implemented.

Meet Health, Safety, and Legal Requirements With Regard to Process, Product, and People. Health and safety requirements and procedures are implemented and followed at all times.● Potential health and safety hazards are identified through continuous safety review.. Confidentiality of proprietary information is protected according to company policy.. Company standards of business conduct are followed.

Select Setup and Perform Diagnostic Tests. The selected test method meets product specifications and customer requirements.. The test method is safe, cost-effective, and meets time needs.. Equipment setup conforms to required test and space specifications.● Test equipment is calibrated correctly and functions according to specifications.. Proper handling procedures are followed.. Tests and test documentation are completed according to prescribed sequence, time, and quality

requirements.. Test results and serialization of tested products are accurately documented.

Analyze and Interpret Test Data for Problems That Require Corrective Actions and for Compliance WithSpecifications

● Products or process deviations and root causes of deviations are accurately identified and documented,and corrective action is initiated.

. Systems for evaluating remedial action are established.

. Corrective action and appropriate recommendations are documented.

. Products forwarded to customer on a “conditional accept” basis are accompanied by accuratedocumentation completed to customer specifications.

. Tests are in compliance with legal requirements, company policy, and customer specifications.SOURCE: American Electronics Association, draft internal memorandum, November 1993.

only restatements of the tasks,29 or a set of simple reflect the basic concepts of high-performanceterms such as “skilled,’ ‘‘moderately skilled, ’or “unskilled,’* as was discussed in chapter 3.This aspect of the occupational mapping ap-proach provides a clearer methodology for settingperformance-level standards than does the jobcompetency method.

As indicated by the manufacturing specialistexample, the broad task structures being identi-fied by AEA are also different from the jobcompetency model in the extent to which they

work systems. The overall structure of the criticalfunctions for manufacturing specialists clearlyreflects, for example, the responsibilities beinggiven to front-line workers for monitoring qual-ity, continuous improvement, and dealing withcustomers.

The next phase of the AEA project will be toidentify the categories of technical skills, knowl-edge, and understanding that underlie perform-ance of the critical functions. In these areas, AEA

~~ For exarllp]e, if [he task is ‘‘prepares check] ists, the criteria is ‘‘prepared checklists. ’


will incorporate methodologies for measuringskills and abilities, as discussed in other sectionsof this chapter. These could include ways tomeasure the technical knowledge or aptitudes ofindividuals, or their cognitive skills. This knowl-edge and underlying capabilities aspect of occu-pational mapping indicates how the techniquemay provide a framework for integrating differentapproaches to broad occupational skills. It also iswhere AEA is considering how the skills structurecan relate to the K-12 system of education.Similar approaches to the definition of broadoccupational skills have been followed in GreatBritain and Australia.

CORE OCCUPATIONAL SKILLSAs shown in figure 3-9 in chapter 3, five states

are planning to adopt or have adopted somestrategy for organizing their programs of voca-tional education around broad technical skills. Inmost of these cases, the strategy adopted by thestate is to divide vocational education into broadclusters of occupations and, in the early years ofpreparation, organize students’ programs aroundthe development of some set of core occupationalskills. Each core area of knowledge and skillsthen branches at the next step into sequences ofclusters of courses that become increasingly morespecialized. Some examples of these broad clus-ter areas are the traditional areas of business,agriculture, and health. States typically have 6 to10 cluster areas covering all industries.

New York was the first state to adopt such astatewide policy, starting in 1983.30 Through along process of planning and development, a‘‘continuum’ of occupational education wasadopted. A key aspect of the continuum is that,starting in the earliest grades and continuingthroughout high school, the occupational curricu-

lum is guided by learning objectives in five broadareas:

1.

2.

3.

4.

5.

personal development (e.g., the develop-ment of a positive self-concept, learning towork in groups, and learning how to find ajob);social systems (e.g., understanding the eco-nomics of business and the legal system);information skills (e.g., communicating orallyand using computers to present informa-tion);resource management (e.g., managing moneyand time); andtechnology (e.g., understanding systems inmanufacturing, construction, communica-tions, and agriculture).

Occupational education starts in the elementarygrades with courses in the use of computers. Bythe end of the 8th grade, all students must havetaken required courses in keyboarding and com-puter literacy, home and career skills, and Intro-duction to Technology.3l

All students who decide to major in vocationaleducation are then required to take a 1-year coursein the Introduction to Occupations. This course isorganized around the five core occupationalcompetencies, with an emphasis on employabilityand other generic workplace skills in the first halfof the course and more occupationally specificknowledge and skills in seven occupationalcluster areas in the second half. In the first half ofthe Introduction to Occupations course, studentstake units in the economics of business andpersonal finances, human relations, decision mak-ing and problem solving, energy systems, andother topics. The seven cluster areas that start inthe second half are agricultural education, business/marketing education, health occupations educa-tion, home economics education, technical educa-

w This description of the New York Progranl is based on Richard .lones, former Director of the Division of occupational Education

instruction, personal communication, Aug. 24, 1993.

3 I ~c Intr(~uc[ion t. Technology” course is similar to the curriculum described in the next section on Design and Technology, but the f(~us

is (m learning about [hc systems of technology.”


tion, technology education, and trade and indus-trial education. In agricultural education, forexample, there is a module in the second half ofthe Introduction to Occupations course on plantscience that covers the basic biology and agron-omy of plants (seeds, roots, leaves, stems, andflowers), life supporting processes and reproduc-tion, environmental factors, soil and fertilizers,plant pests, and so forth. Following the Introduc-tion to Occupations course, students take asubject area core course and sequences of three ormore increasingly more specialized courses inone of these seven cluster areas. Instruction atthese more advanced levels is still guided by thesame core occupational skills but at more ad-vanced levels of learning.

An assessment program has been implementedto sustain the teaching of the core occupationalcompetencies throughout the curriculum. The testfor the Introduction to Occupations course con-sists of a locally administered but state-developedcompetency test. The state requires that allstudents who complete the course take the test.The state also has competency tests in four of theseven occupational cluster areas, which are takenon completion of required coursework in thoseareas.

California is in the process of developing andimplementing a similar approach to organizingvocational education around core skills and occu-pational clusters. For example, in business educa-tion, which is serving as a model for the othercluster areas, a framework of curriculum stand-ards will soon be adopted for the first year courseof the program. This course is called the businesstechnology core. After completing the businesstechnology core, students will choose one of four‘‘career path clusters’ in computer science andinformation systems, business administration,accounting and finance, or marketing. Threesequential levels of learning outcomes areplanned within each cluster; these will be adoptedas a series of standards. There will be career pathcluster standards, career path specialization stand-ards, and an entrepreneurship standard. The

business technology core will also be linked toinstructional units or courses in business explora-tion planned in the upper elementary grades.

Cutting across all of these cluster areas in thebusiness program will be the career performancestandards in communication skills, employmentliteracy, interpersonal skills, occupational safety,personal skills, technological literacy, and think-ing and problem-solving skills that are describedin box 3-A in chapter 3. The assessment of thesecareer performance standards will be part of thenew statewide system of assessment for voca-tional education that will consist of studentportfolios and on-demand assessment (also de-scribed in box 3-A).

This core occupational skills model of NewYork and California is similar to the reorganiza-tion of the German apprenticeship system that hasoccurred over the past 10 years or so to broadenthe skill content of the training. A similarbranching structure of broad occupational areasthat divide into specialties over the 3-year courseof an apprenticeship program has been adopted.

DESIGN AND TECHNOLOGYThe fourth approach to defining broad techni-

cal skills is to view technology itself as somethingthat is worthwhile for all students to learn about,and not just students who are headed for work or2-year programs. From this viewpoint, learningabout technology as a product of human civiliza-tion as well as developing the capability foractually using it to accomplish practical purposesshould be part of the education of all students.starting in the early grades. Involving all studentsin learning about technology and how to use itcould help to erase some of the distinctions oftenmade in schools between students headed forcollege and working with their minds, and thosedestined for working with their hands. Increas-ingly, technology itself is blurring this distinc-tion. Teaching technology to all students in thisway frames the issue of who needs what technicalskills in the broadest possible terms; capabilities


for design and technology are needed to someextent by everyone. At the same time, if startedearly enough, technology education could help tomotivate students toward pursuing technical ca-reers and provide them with foundations ofknowledge and capabilities for convertingthought into action.32

A variety of different forms of technologyeducation exist in the United States and abroad.One version may offer considerable potential forhelping all students develop broad technicalskills. In Great Britain, this new form of technol-ogy education is known as design and technology.It has developed over the past 20 years and is now1 of the 10 major subjects of the new nationalcurriculum of the British schools.

The significant term is ‘‘design. ’ In design andtechnology, students learn as much or more aboutbecoming a designer as they do about theworkings of the technology itself. Becoming adesigner involves acquiring three kinds of knowl-edge and capabilities:

1.2.

3.

The

procedural knowledge of how to design;capability of communicating complex ideaswith clarity, confidence, and skill; andconceptual knowledge and understandingof the use of materials, energy systems, andaesthetics in designing and building techno-logical systems—along with knowledgeand awareness of people and their needs.

procedural knowledge of design involveslearning how to discern needs and thoroughlyinvestigate them as an architect or engineerwould, prepare a design brief or plan for meetingthose needs, generate ideas and develop them intoa design, build the design, reflect on the quality ofthe result in meeting the original needs, and thenact on the reflection to produce a better design.The subject is taken by both boys and girls, andboth high and low ability students, beginning inthe first grade. It is conducted to appeal to all

students—those oriented mainly to the humani-ties, and those more scientifically or technicallyinclined.

Learning how to design involves learning howto think concretely in complex situations. Itinvolves using knowledge and logical thinking toproduce better designs. It also involves makingchoices and weighing the desirability of alterna-tive designs from social, economic, aesthetic, andhuman points of view, as well as from theperspective of producibility.

A thoroughly developed performance assess-ment in England has shown the effectiveness ofthe design and technology curriculum for teach-ing students how to become designers. Normally,girls perform better than boys on the design tasksof investigation and reflection, while the boysexcel at developing designs and building theminto final products. Lower ability students tend tobe better at design tasks that are tightly structured,and higher ability students tend to be better onmore openly structured tasks. However, a clearfinding of the assessment is that the more studentshave been exposed to design and technology, thesmaller these differences become.

The assessment also showed the interdepen-dence of thinking and doing in technology.Students given the same set of five assessmenttasks but in different order were much better ableto identify the strengths and weaknesses of anoriginal design when they had previously beengiven a task of actually designing an improvedproduct. Providing students with a brief period ofdiscussion midway in the development of adesign, where the students described their plansand received peer comments, immensely im-proved the quality of their final products. Theassessments showed no simple relationships be-tween sequences of design steps and thoughtprocesses that lead to quality results, and thosethat do not. In other words, the process of design

sz This section draws on Richard Kimbell, University of bmdon, Goldsmith’s College, “Technology in the School Curriculum, ” OTAcontractor report, October 1993. This contractor report is available from OTA’S Education and Human Resources Program.


cannot be reduced to a checklist or a methodicalseries of steps.

COGNITIVE SKILLSFinally, a fifth approach to defining broad

technical skills is the concept of cognitive skills.This concept comes from cognitive science re-search on problem solving and trouble shooting ina range of occupations, apprenticeship learning,and learning in academic subjects. Much of theresearch has been generated by efforts to explainthe differences between the cognitive skills ofexperts and novices,

This research shows that experts draw ontightly integrated structures of procedural andconceptual knowledge. This knowledge is alsohighly contextualized; it is specific to the particu-lar organization and technological environmentof the expert. Over time, this knowledge becomesschematically organized, according to specificproblems and situations—it is not abstract. Thisschematic knowledge enables experts to under-stand the complex relationships necessary forskilled performance. Experts differ profoundlyfrom novices in the speed with which they canaccess this knowledge; a novice must try to solveeach problem de novo.33

Substantial proportions of the knowledge ofexperts are procedural and tacit; experts knowwhat to do and when, but they cannot alwaysexpress what they do exactly. Their proceduralknowledge, together with the conceptual andcontextual knowledge underlies what scientistscall metacognition—the ability for goal setting,planning, adaptation, and learning that allow forexpert problem solving.34

Cognitive skills are acquired in stages. Ini-tially, a skill is heavily dependent on declarative(verbal) knowledge. In the declarative stage, thelearner either encounters or is instructed in thefacts and procedures relevant to the execution ofa particular skill.3s These facts are stored inmemory as statements, which can be verbalizedand recalled one-by-one as required. The noviceuses this knowledge interpretively-that is, thenovice might say: “The situation is ‘a,’ thereforeI should do ‘b. ’ “ The novice then would be ableto do “b.’ ‘ In this declarative stage, generalproblem-solving strategies are employed by thenovice to organize these steps and bring each oneinto play until the problem is solved or the goal isreached. An example of such a general, or ‘‘weakmethod, ’ strategy is solving problems by anal-ogy, or mimicking the steps that were followed insuccessfully solving an apparently similar prob-lem. The strategy is general because it does notdepend on knowledge of the area in which it isbeing applied. “Strong methods” are specific toa domain and are heavily dependent on knowl-edge of it.

The second stage is “skill automation” orcompilation of the cognitive skill. In skill automa-tion, weak methods of problem solving aretransformed into strong methods.36 This occursthrough gradual transformation of the factual andconceptual knowledge acquired in the declarativestage into a procedural form that is highlyorganized so that it can be accessed and used withminimal conscious reasoning activity.37 The skillbecomes “automatic” and the load on workingmemory of recalling the specific steps involvedbecomes much less. The skill can be performed

33 Robefi Glaser et al., b’lrnplications of Cognitive Psychology for Measuring Job Perfomlance, ” in Wigdor and Green (eds. ) op. cit.,footnote I 2, pp. I -2.

34 Ibid.ls James M. Royer et al. T ‘ ‘Techniques and Procedures for Assessing Cognitive Skills, Re\’iw oj” Ed/(cational Reseor(h, vol. 63, N(). 2,

summer 1993, p. 204.

36 J.R. Anderson, ‘‘Skill Acquisition: Compilation of Weak-MethO& Problem Solutmns,” P.r?(}rolo~i[o/Retie)is,” w)]. 94, N(). 2, 1987, pp.192-210.

37 Royer et al., op. cit., footnote 35, p. 2@5.


without consciously thinking about it and thespeed increases. Studies of apprenticeship haveshown that this compilation process is aided by:a) situated learning, where students execute tasksand solve problems in a real environment wherethe purposes of the knowledge are clear; b)external support, or “scaffolding,’ availablefrom a tutor or master to model the idealperformance, support learning, and supply miss-ing pieces of knowledge; c) fading, or withdrawalof support, as the skill is acquired; and d) learningactivities that are carefully sequenced to be bothsensitive to student needs and robust enough tofoster integration and generalization of the skill.38

The third stage is skill refinement, or procedu-ralization, which is a continuation of the compila-tion process of the second stage. In this stage,performance of the skills is speeded up byweeding out nonessential steps and strengtheningassociations between events that may occur inperforming the skill and effective actions in

39 As a result, performance ‘f

response to them. the skill becomes much more sensitive to smallbut critical situational differences and the flexi-bility of response to unexpected situations or newdata greatly increases.

This model of skill acquisition has been shownto account for many of the differences in thecognitive skills of novices and experts, and toaccurately describe the learning processesthrough which the skills are acquired.40 Themodel has been applied in a wide range ofdomains from electronics troubleshooting, powerplant control, and financial planning to tennis.

The knowledge of experts is thus highlyprocedural and integrated in nature rather thanconceptual and detached from contexts of use, asis so much of the learning process and theknowledge acquired in school. Facts, routines,

and concepts are bound together with rules fortheir application and to conditions under whichthe knowledge is useful.41 The knowledge ishighly goal oriented and conditional. It includesknowledge of specific goals and interrelatedsubgoals, methods that are employed in pursuingthose subgoals, selection rules for choosingsubgoals, evaluating methods based on experi-ence, and prerequisite cognitive skills. For exam-ple, the goal of the relatively simple cognitiveskill in word processing of moving text from onepoint in a document to another point can bebroken down into a conditional series of subgoalsand procedural steps. (To move a block of textfrom A to B, first block the text to be moved, thenchoose the ‘‘move’ operation from the menu, andso forth). Experts have been shown to be muchbetter at identifying the hierarchy of subgoals, orintermediate steps, for accomplishing such tasksthan novices. In the language of cognitive sci-ence, experts are much better than novices atmapping the ‘‘problem space’ of procedures toemploy in solving a problem or generating an ideaand deciding on the order. Novices, for example,may not be able to identify the problem space orstructure of goals, and simply try the few proce-dures they know one by one to see which one maywork. Selection rules refer to the knowledgeacquired by the expert of the particular conditionsunder which specific procedures should be em-ployed. Prerequisite skills could be, for example,capabilities for performing all of the measure-ments needed in executing a procedure.

Cognitive research has shown that the organi-zation of the knowledge structures of experts iswhat explains the flexibility and speed withwhich they are able to access knowledge and useit. One of the profound aspects of the knowledgeorganization of experts is their ‘depth of problem

‘x Sher-r-k P. Gtm, “Apprenticeship Instruction for Real-Wodd Tasks: The C()(wdinati(m of Pr(mdures, Mental Models, and Strategies, ”Re\fIeH oj’Re.rearch In Education, v(d. 15, 1988, p. 99.

1{} Rt~yer ct al., op. cit., footnote 35, p. 206.

~) G]aser et a],, op. cit., f(x)tno(t? ~~.

‘1 Ibid., p. 7.


representation. ● ‘ Using their stored knowledge,experts are able to rapidly induce principles fromthe given features of problems and then representthe problems in terms of these principles. Thisgives them quick access to patterns of events andassociated sequences of preferred moves andprocedures. They are able to do this because theydo not have to first load this knowledge into theirworking memory before they are able to access itand use it. The problem representations of nov-ices, on the other hand, tend to be expressed interms of literal objects, the surface features ofproblems, or the events given explicitly in aproblem statement.42

One example of the depth of problem represen-tation of experts is ‘‘chinking. ’ For example,expert electronics technicians have been shown tobe much better at reproducing circuit diagramsflashed before them for a brief period of time thannovices, but only when the circuits are correct.When the circuits are wrong, there is no differ-ence between experts and novices in their abilityto reproduce the diagrams.43 Skilled electronicstechnicians also tend to reconstruct symbolicdrawings of circuit diagrams they are workingwith according to the functional nature of theelements in the circuit, while novices tend toreproduce the circuits based on the spatial prox-imity of the elements. The “chunks’ of knowl-edge they draw on to produce these diagrams tendnot to reflect knowledge of function.44

In addition to the contextual and proceduralknowledge in the knowledge structures of expertsis a third component, the capability for visualiz-ing or mentally modeling the features or operationof the technological devices or systems, or

representing the problem space of interpretationsand conceivable actions. These mental modelsprovide essential connections between the proce-dural and contextual knowledge, enabling theexpert to relate knowledge of one form to theother and build representations of the situations athand. 45 These visual representations help toovercome the limiting serial nature of languageand procedural knowledge to provide more ade-quate means of explaining or interpreting com-plex and/or dynamic phenomena encountered.%

An important finding in studies of the diagnosisof x-rays by physicians, for example, shows thatexperts have a reservoir of preexisting schematain their heads of normal and abnormal configura-tions of the organs or tissues that are triggeredearly in a diagnosis.47 The diagnosis i nvo lves

fitting the schemata to the present case until themost consistent explanation is found.

Mental models are essential for effective causalreasoning about the sources of problems andpotential solutions. Studies of experts have shownhow much better they are than novices at visualiz-ing the systems with which they are working atdifferent levels of abstraction and working backand forth across these levels of abstraction toinvoke the level of analysis most appropriate forthe diagnosis or task at hand.48 For example, indiagnosing a broken piece of equipment, expertsmay employ their schematic knowledge of thefunctions of the equipment and its components,factual knowledge about the operational charac-teristics of the individual components, and order-ings of how observed symptoms of one kind areconditionally related to others. Much of thisreasoning about problems tends to be qualitative

42 Royer ~t ~],, f)p, Cl[., footnok 35, pp. 2 17 “ 2 2 2 .

43 Ibid., p. 2 I 8.

U R[)bu-t Glascr, ‘Expertise and Assessnwn(, s Cognition and Tesfing, M.C. Wittrock and E.L. Baker (eds. ) (Englcw(Md Cl]ffs, NJ PrenticeHall, 1~~] ), pp. ] 7-30.

~~ Glaser et a],, op. cit., footnote ~~, p. ~.

M Gott, op. cit., footnote 38, p. 123.

JT G]aser et al,, op. c[t,, fo{)tn(}tc ~~, p. 9.

+ Gott, op. cit., f(}(~tnotc 38, p. I 24.


in nature rather than quantitative (e.g., “if thesignal is present there, then it should be presenthere’ ‘). Overemphasis on quantitative reasoningin the early stages of acquiring skills has consist-ently been shown to inhibit understanding of thecausal principles involved.49 For example, stu-dents in the physical sciences who have mainlystudied in the traditional modes of highly quanti-tatively oriented instruction have consistentlyshown deficiencies and mistakes in their under-standing of the underlying causal principles in adomain. 50

A number of experimental efforts have beenundertaken to build computer-based tutors fortraining experts based on these concepts of thenature and acquisition of cognitive skills. Evalua-tion of the effectiveness of these tutors has shownthe strongest results for systems that coordinatethe teaching of contextual and procedural knowl-edge so that the uses of visualization and model-ing in problem solving are made explicit.51 Tutorsthat mainly support the acquisition of proceduralknowledge yield trainees who are less able to dealwith complexities and uncertainty. Such traininghas usually been found to be even less effectiveand efficient than methods of informal observa-tion on the job. A related finding is that thepresentation of “. . . abstract representations ofdevice interdependencies for learners to inspectand manipulate seems central to the developmentof envisioning capabilities. ’ ’52

This model of the acquisition of cognitiveskills has important implications for broad techni-cal skills. The cognitive model supports the jobtask and procedural orientation of job compe-tency approaches over attempting to teach orassess general abilities directly, as implied by thevocational aptitude and core occupational skillsapproaches. At the same time, the model implies

that the limitation of learning to the rote memori-zation of procedures within a fixed and narrowtask environment without the simultaneous intro-duction of concepts and support for modeling andvisualization will inhibit the development ofexpertise and result in trainees who are notcapable of responding flexibly to new problems.The model therefore suggests that the teachingand assessment of broad technical skills shouldinclude methods of eliciting respondents’ capa-bilities for visualizing and conceptualizing prob-lems in context, and not just recalling facts frommemory even if the facts are closely related tothose contexts and demonstrate mastery of proce-dures. The model does not support decontextual-ized approaches to the assessment of broadtechnical skills, like those employed in traditionaltesting for vocational aptitudes or multiple abili-ties, and especially not at expert levels ofperformance. The decision orientation of themodel strongly suggests that the assessment ofbroad technical skills should include strategiesfor assessing metacognition—the capability tochoose among goals and methods and alternativeinterpretations of evidence in different contexts.

The cognitive model also strongly suggeststhat assessment should focus on both the proce-dural and conceptual aspects of broad technicalskills in authentic contexts, rather than the recallof isolated bits of knowledge on demand. Reli-ance should be placed on using multiple methodsof assessment rather than written testing, al-though both may be needed. The highly inte-grated and contextualized nature of the process ofknowledge compilation lying at the heart of thecognitive model implies that learning broadtechnical skills must also be active, so thatstudents are constructing knowledge for them-

49 Ibid., p. 127.

M Ibid Also see Howard Gardner, The uns(.h~~]ed Mind: l!ow Children Think, and I{ow’ Schools Sh~l~ld Teach (New York, NY: Basic

Books, 1991 ).s I Gott, ~)p$ cit., footnote 38t P. 16]”

52 Ibid.


selves, gaining facility in integrating it fromdifferent sources, and using it for practical ends.

Finally, the cognitive model raises a cautionflag against the possibilities of broadening skilldefinitions and methods of assessing and certify-ing them simply through expanding indefinitelythe job and task environments in which theyapply. The reason for this skepticism is theprominent role of contextual knowledge in thedevelopment of expertise, and the highly organ-ized and tightly coupled ways in which theknowledge of experts is structured.

A different model of how to achieve breadth ofskill is suggested by the cognitive model in whichbreadth is achieved through depth. The view isthat the best way to achieve breadth of skill in thelong run is through deepening knowledge devel-opment within a limited number of areas ofspecialization, rather than attempting to take onever wider circles of problems. The concept isachieving “breadth through depth” rather thanbreadth alone. The problem with the breadththrough depth approach is that expertise devel-oped through specialization and depth of pursuitcan be ‘ ‘brittle; it does not transfer well to newsituations and new kinds of problem environ-ments. When confronted with new problems inunfamiliar areas, the narrowly trained learner canbe immobilized and incapable of respondingproductively. In order to achieve breadth, thestrategy of developing and assessing depth mustbe coupled with considerable experience in deal-ing with new problems and novel situations.

Cognitive theory is not very helpful so far inprescribing exactly how transfer occurs andbeyond that how learning should occur in order tomaximize capacity for transfer, except to say thatthe knowledge and skills acquired will be brittleif learners are not confronted with novel problemsto solve within their domains of expertise and inrelated domains. In this light, breadth of skillbecomes the individual’s capability for usingknowledge from domains in which one is expertin new domains. Whether knowledge and skills

from domains in which one is expert are appliedin the new domains as ‘‘finite elements’ orinstead facilitate the acquisition of knowledgeand skills in the new areas is an unresolved issue.

FINAL NOTEThe aptitude model suggests that the ability to

transfer expertise in novel situations and to newdomains needs to be defined in terms of bothdemonstrated capacity for responding cognitive] yto new situations and evidence of broader generalaptitudes that strongly relate to the ability totransfer and learn in new situations. This meldingof two approaches is also evident in the additionof categories of knowledge and skill to the broadoccupational tasks that are identified in theapproach of occupational mapping.

In the occupational mapping approach, breadthof technical skill is described through identifyingtasks presumed to define competencies that cutacross a range of occupations and form the core ofexpertise. Their manifestations may be somewhatdifferent in different job contexts and industrysegments but in outline they are similar. Thecritical role of contextual knowledge in expertise,as described in the section on the cognitive model,indicates that it remains an open question whetherindividuals who are found to be competent in onejob context according to the occupational map-ping approach would be able to demonstrate equallevels of the same competencies in other jobcontexts.

These five approaches to defining broad tech-nical skills provide starting points for moving toconcepts that are at once deeper, more unifying,and more specific than the concepts underlyingthe various approaches. Another need is tobecome clearer about the content of technicalskills and how they differ from other capabilities,such as interpersonal and communications skills,motivation, and academically oriented compe-tence.

Implementationof PerformanceStandards and

Measures 6

u rider the 1990 Perkins Act Amendments, Public Law101-392, the primary responsibility for planning andimplementing vocational education standards and meas-ures rested with the states. Federal law established only

minimum content and process requirements for statewidesystems, leaving key decisions to the discretion of state boards. ]This decentralization of authority allowed for variation in statesystems, consistent with the reality that states differed greatly intheir structures for vocational education and in their expertise inperformance-based accountability.2 Further, it reflected theabsence of national consensus about standards for evaluatingvocational education.

In keeping with this decentralization, federal administrativeresponsibilities regarding performance standards are limited butstill quite important. The amended Perkins Act charged the U.S.Department of Education (ED) with providing technical assist-ance to states for the development of accountability systems(section 115) and with conducting or overseeing research onstandards and measures (sections 402, 403, 404, and 422). Fur-nishing guidance on performance standards was also an implicitpart of ED’s obligation to issue regulations for the Perkins

1 An]tm: [he dcc]s]tms Icft to states were which standards and measures would bcused, how local prt~granl quality W(NIM be judged, and what type of assistance W<(mid bepr~)i Idcd tt) progr:ims makln: Insufficient progress t(nvard s[andards.

~ A natit~n;il sum cy cxmductcd in 1991, prit)r m the Perkins Act deadline fordc~ cloplng pv-f(mnancc ~ystcrns, f(mnd that about one-half of the states had used svcificpcrfl~rn]ance st;indarcls [Jr measures for v{~cati(mal education in the past, E. GarcthHtnchlandtr and Mikala L. Rahn, Per/ormancc Measures and Sk?ndmdsjor Vo(atlonalli(lu< (IIion: 1991 .5ur~e> Result$ ( B e r k e l e y . CA Nati(mal Center for Research InLroc:itlt)nal Educat]on. I 992). p, 4.

\

103


Act and to conduct regular oversight and monitor-ing of federal vocational education programs.Administering a federal competitive grant pro-gram to develop national skill standards forparticular industries and trades (section 416) isanother related component of ED’s role.

Some of these responsibilities are being carriedout directly by ED’s Office of Vocational andAdult Education (OVAE); some are being con-ducted through other entities, particularly thefederally supported National Center for Researchin Vocational Education (NCRVE).

TECHNICAL ASSISTANCE EFFORTSProviding technical assistance is one of ED’s

most critical duties concerning performance stan-dards. The move toward performance-based ac-countability is a pivotal change for vocationaleducation. In developing their systems, states—especially those with little prior experience inperformance-based evaluation—were likely toencounter several difficult and highly technicalissues for which additional guidance would behelpful. Recognizing the complexity of the task,the law gave states 2 years, until September 25,1992, to put in place their new systems andcharged ED with furnishing technical assistance.

ED has delegated much of the responsibility forproviding technical assistance to NCRVE. Forseveral reasons, NCRVE was a logical choice forthis assignment. Research, dissemination, andtraining on issues of performance standards werealready part of the center’s mission under therevised Perkins Act. In addition, ED concludedthat the center was more likely than OVAE to

offer the necessary expertise in technical issues ofevaluation, testing, and measurement, especiallygiven OVAE’s current staffing levels.3

Under the present arrangement, OVAE handlesclay-to-day communications with states and over-sees implementation of state accountability plans.For additional guidance on performance andevaluation issues, OVAE often refers states toNCRVE. 4 In providing technical assistance, NCRVEhas undertaken five special efforts to help statesimplement the new accountability requirements:

●

●

●

●

●

three regional workshops held in Februaryand March of 1992, and one national techni-cal assistance session held in July 1992, alljointly sponsored with OVAE;5

operation of a clearinghouse;publication of a practitioner’s guide onaccountability; 6

research studies on the implementation ofstate standards and measures; anda national conference on performance stan-dards in vocational education in the summerof 1993.

NCRVE has received some special funding tohelp carry out these activities. From an additional$2 million appropriated by Congress to NCRVE,ED earmarked approximately $200,000 for tech-nical assistance on performance standards.

The core of NCRVE’s initial technical assist-ance efforts was the 1992 workshop series; abouttwo-thirds of the $200,000 supported this activ-ity. During the workshops, state officials respon-sible for developing the new accountability sys-tems had the opportunity to share practices andreceive expert advice on issues related to stan-

~ Joseph Casello, branch chief, Program Analysis Branch, Office of Vocational and Adult Education, U.S. Department of Education,personal c(mlmunication, June 2, 1993; E. Gareth Hoachlander, director, Plaming and Evaluation, National Center for Research in VocationalEducati(m, personal communication, June 1, 1993; and Debra Nolan, project director, Business and Education Standards Program, Divisionof National Programs, Office of Vocational and Adult Education, U.S. Department of Education, Personiil communication,” June 9, 1993.

4 Casello, op. cit., footnote 3.

~ Regional workshops of 2+ days each were held in Washington, DC, St. Lmis, and San Francisco; the national workshop was held inMinneapolis.

6 E. Gareth Hoachlander et al., Accounrabi/i(y jtir Vocalionul Educa(ion: A Pracfitloner’s Guide (Berkeley, CA: National Center forResearch in Vocational Education, 1992).

Chapter 6: Implementation of Performance Standards and Measures 105

dards, measures, and assessments. OVAE staffparticipated in all the workshops and weremembers of the NCRVE advisory group chargedwith planning the workshops and reviewingNCRVE materials regarding performance stan-dards.

The remainder of the $200,000 has helpedsupport a clearinghouse, which gathers currentinformation about state accountability plans andprovides telephone technical assistance.

Through its Practitioner’s Guide and othermaterials, NCRVE has amplified the limitedguidance contained in the law and regulationswith detailed suggestions, examples, and recom-mendations. Among them are concrete examplesof standards, learning measures, labor marketoutcomes, assessment methods, and strategies forevaluating access of special populations.

NCRVE also identified six basic features thatshould be incorporated into a well-designedaccountability system and offered expert opinionson several key issues. For example, NCRVErecommends that state accountability systems gobeyond the 2 minimum measures required by lawand include from 6 to 12 measures.7 The centeralso suggests that student performance be as-sessed both in terms of gains over time andabsolute attainment, so that growth by studentswho began at a low level is not overlooked.8

NCRVE has further encouraged states to phase inelements of a more ambitious system over timeand to continue monitoring and modifying theiraccountability systems as they gain experience.9

To meet the need for more sophisticatedguidance on technical issues, both NCRVE andOVAE plan to continue offering technical assist-ance, maintaining the clearinghouse, and con-ducting workshops and training session. A follow-

up conference jointly sponsored by NCRVE andOVAE was conducted in July 1993. The purposeof the meeting was to provide state administratorswith a forum to share experiences and help EDdetermine what additional technical assistance isneeded .

Participants in the meeting identified the mainissues where further technical assistance is neededby the states in implementing performance stand-ards. Discussion focused on: 1 ) using the informa-tion that will be generated in local programreviews for purposes of improvement, and 2)what program improvement plans should be likein order to be really helpful. Many states are alsolooking for assistance in how to set standardsusing rigorous methods. Concerns were expressedthat not enough is known about the reliability andvalidity of the skill standards and methods ofassessment that are being used. Issues of reliabil-ity and validity will grow in importance asperformance standards are implemented.

As yet, no technical assistance efforts havebeen directed specifically toward testing andassessment for the implementation of perform-ance standards.

ED REGULATIONS AND MONITORINGIssuing regulations and monitoring state com-

pliance for federal vocational programs are amongthe major administrative responsibilities of ED,and both affect state implementation of perform-ance standards.

Evaluation and accountability requirementsbecame a highly controversial issue during thedevelopment of regulations for the Perkins Act. 11

In October and November of 1990, ED heldregional meetings, as required by section 504 ofthe Perkins Act, to obtain public input prior to

7 lbld., p. 7.X Ibid., p. 13.

‘} Ibid., p. 1 I 2.

1~1 cas~]lt), op. ci[., footnote 3.

I I II is no[ewofihv” tha[ evaluation, standards, and measures were not particularly controversial during congressional consideration,t)vershad(}wcd by other heated Issues, such as allocation fommlas and the distribution of funds between secondary and postsecondaxy programs.


drafting regulations.12 Following these meetings,

the Secretary, as further required by law, selectedthree major issues to undergo the process ofnegotiated rulemaking, which provides state andlocal representatives with a forum to discuss andmake recommendations to the Department onissues of complexity, contentiousness, or sub-stantial consequence. One of the issues chosenwas whether state and local vocational agenciesshould apply program evaluations to all of theirvocational education programs or only to thefederally funded components or projects.13

On October 11, 1991, ED published proposedregulations that took the position that localentities receiving Perkins basic grant funding hadto evaluate the effectiveness of all their voca-tional education programs, not just those receiv-ing Federal funding.

14 A surge of public comment

opposed this interpretation, many arguing that itwas unduly costly and burdensome, that it wasinconsistent with the language in the law, or thatit represented unwarranted federal intrusion. 15

Final regulations were not issued until August14, 1992—less than 6 weeks before the statestandards and measures were due, and well pastthe deadline for ED regulations specified in theGeneral Education Provisions Act. 16 Debate withinthe administration over the evaluation issueseems to have been the main reason behind thedelay .17 Thus, states were compelled to proceedwith development of standards and measures in

the absence of a definitive interpretation abouttheir scope.

The final regulations narrowed the evaluationto encompass only “. . . the particular projects,services, and activities . . .’ receiving PerkinsAct basic or special funding, unless the statedetermined that a broader evaluation was needed. 18

The final rules also contained other importantclarifications. When conducting evaluations, localvocational agencies could rely on representativesampling techniques. In addition, grantees coulduse Perkins funds to pay for mandated evalua-tions, without regard for the 5 percent ceiling onlocal administrative costs.19

Mirroring the legislation, the regulations avoidedfurther specificity on such issues as what “com-petency gains” mean and how they should bemeasured, whether separate standards for second-ary and postsecondary programs are encouragedor desirable, how special populations should beaddressed, and how basic and advanced academicskills should be measured. These and otherspecialized issues were left for states to decide,with advice from NCRVE and other sources.

According to OVAE officials, every stateappears to have met the minimal requirements fora statewide performance system; many states planto phase in or expand the components of thesystem over time. As a next step, OVAE; iscurrently conducting onsite reviews of stateimplementation .20

i 1 Meetings were he]d in Philadelphia, Atlanta, Kansas City, and San Fmncisct).

I T paul M, ]Wln and Richard N, Ap\]ng vO(.~l~O~/ h’dwotiim: Major Provisions oj’ Ihe 1990 Amendments (P.L. 10I -392) (Wmhinglm,DC: congressional” Research Service, 1991 ), p. 18.

I ~ 56 F-Fdera/ Regis/er 51448 (Oct. ] i, 1991 ).

] ~ 57 Federal Register 36841 (Aug. 14, 1992).

I (> SectIon 43 ] (g) of the General Education ~ovisions” Act (20 U.S.C. ] 232) r’equ]res final regulatlms for Department of Education pN)gHMllS

to be promulgated within 240 days of enactment of the applicable authorizing statute, unless a revised schedule is approved by the cong~ssionalauth(wizing committees. It is not uncommon, however, for the Department to have missed this deadline.

17 John F, Jcnnlngs, ~ounse], House Conln}i[[ee on Education and Labor, persona] cornrmmlcatk)n, Feb. 23, 1993; and ‘ ‘ED Set to ]SSLK VOC

Ed Rules After Ford Increases Heat,’ Education Daily, Aug. 12, 1992, p. 3.

IX 45 CFR 403.191 and 403.201.

l’) 45 cm 403.191.l“ Casclh}, op. cit., footnote 3.

Chapter 6: Implementation of Performance Standards and

RESEARCH AND DEMONSTRATIONSeveral federally sponsored studies are under

way that address the issues of performancestandards, skill standards, and accountability.

As required by law, NCRVE is conductingresearch on performance standards and accounta-bility systems. Findings of a baseline survey ofprior state experience with vocational educationmeasures and standards were published in March1992 .21 A second study describes and analyzesthe state accountability systems to date.22 A thirdstudy, conducted jointly with the Rand Corp. andscheduled for completion in 1994, is examiningthe effects of performance accountability y systemson state and local policy.

OVAE has also awarded a contract to study theperformance standards adopted by the states, ascalled for in Section 1 15(f) of the 1990 amend-ments. The purpose of the study is to evaluate andreview the systems developed by the states.Through a mail survey and case studies in ninestates, the study will:

●

●

●

The

describe, in some detail, the status of eachstate’s system of performance standards andmeasures,assess the reasonableness and appropriate-ness of performance standards and measuresfor specific populations, andexamine the comparability of the perform-ance standards across states to determine thefeasibility of establishing a common core ofindicators.

study is being performed by the BattelleHuman Affairs Research Centers.23

BUSINESS AND EDUCATIONAnother federal activity with

Measures I 107

STANDARDSrelevance to the

new Perkins accountability requirements is thejoint Department of Education-Department ofLabor (DOL) initiative to develop voluntary,national skill standards in various industries andtrades. The ED component, called the Businessand Education Standards Program, is authorizedby section 416 of the Perkins Act. The DOLcomponent, called the Voluntary Skill Standardsand Certification Program, is being supportedwith departmental discretionary funds.24 BothDepartments have made competitive grants toprojects to organize and operate business-labor-education partnerships, which will in turn de-velop national skill standards and certificationsfor competencies in industries and trades.

Eligible grantees under the ED program in-clude industrial trade associations, labor organi-zations, national joint apprenticeship committees,and comparable national organizations. The stan-dards developed through ED grants must addressat least:

1. major divisions or specialty areas withinoccupations;

2. minimumtent;

3. minimum4. minimum

staff; and5. minimum

study.25

hours of study to become compe-

tools and equipment required;qualifications for instructional

tasks to be included in a course of

The ED program was funded with appropria-tions of $3.5 million annually from fiscal years1991 and 1992. DOL reserved $1.2 million from

2 I Hoachlan&r and Rahn, op. cit., foolno(c ~.

~J Mihala Rahn et al., .Srate .Sys[em.$,/Ur Accourr(abi/i[y in Vocational E“ducalion, MDS-491 (Berkeley, CA N:itit)niil Center t(~r Research

in V(~ati(mal Educati(m, December 1992).23 Bat[e]I~ Hun~an Affairs Research c~nters, “‘Performance Standards and Measures: Evaluati(m Stud}, “‘ repmt t-wing prepared f(w (he U.S.

Department of Educati(m, Office of Vocati(mal and Adult Educati(m, in progress.

~d Michacla Meehan, p)licy analyst, U.S. Dcpartrmmt of Labor, pers(mal communicati(m, June 15, 1993.

2557 F“ederoI Rc,gt ~!er 45] 46 (Sepl. 3~, 1992).


fiscal year 1992 discretionary funds. Granteesmust match federal funds dollar for dollar.26

In the first round of grants, awarded in Septem-ber 1992, ED supported seven projects and DOLsix. In the second competition, ED awarded ninenew projects.27 The grant per iod for the ‘D

projects is 18 months, with one continuation, andthe grant period for DOL projects is 1 year.28

Both ED and DOL recommend that standardsbe benchmarked to world-class levels of industryperformance and “. . . tied to measurable, per-formance-based outcomes that can be readilyassessed. ’29 The ultimate aim is for the projectsto yield standards that could be adopted and usedby education and training institutions, labororganizations, business and industry, employers,individuals, and government.

From DOL’s perspective, it is particularlycritical that the standards developed have supportfrom business and industry; if this occurs, theneducation institutions and other communities arelikely to follow suit.3o

How the skill standards might mesh with stateaccountability systems required by the PerkinsAct or with Job Training Partnership Act (JTPA)performance standards is a critical issue. Accord-ing to the ED Business and Education programdirector, she and the OVAE administrators of thestate vocational programs consult regularly, andED is taking several steps to ensure that the twoefforts are coordinated. Regulations and guidance

for the Business and Education program empha-size the importance of dissemination and adop-tion of standards by state officials and otherentities. In addition, ED has encouraged granteesto examine existing standards, including thosedeveloped by the state technical committeesauthorized in the original Perkins Act, whendeveloping national standards. Further, almostevery partnership includes involvement of statevocational education directors.31

The DOL program administrator, however,emphasized that the ultimate goal of the DOLstandards program is to produce national stand-ards and overcome the fragmentation that couldoccur if each state proceeds with its own stand-ards. Over the long term, DOL also hopes to beable to use national skill standards to evaluateoutcomes for JTPA and other Department pro-grams. 32

GOALS 2000: EDUCATE AMERICA ACTProgress toward national skill standards may

be further stimulated if the Goals 2000: EducateAmerica Act—a primary education initiative ofthe Clinton Administration-is enacted.33 E Dand DOL have consistently stressed the relevanceof their respective skill standards programs toNational Education Goal No. 5 (adult literacy,citizenship, and ability to compete in theworkplace). This link would be solidified by TitleIV of Goals 2000, which is currently being

XI ~bra Nt)]an, U.S. Department of Education, ‘‘Project Abstracts for the Business and Education Standards Program, ’ unpublished repmt,1992.

27 First. r(~und pro@tS Will develop standards in the following fields: health sciences and technology; electronics; computer-aided drafting;air conditioning and refrigeration; biotechnical sciences; printing; automotive technicians; industrial laundering; tourism, travel, andhospital ity; metalworking; electrical construction; and retail trade. Second-round projects focus on human services occupations, heavy highwayconstruction and demolition, chemical process industries, hazardous materials management, photonics, agriscience, welding, forest and woodproduction and manufacturing, and food marketing.

2s A decision is pending on whether additional D(IL discretionary money will be provided to Continue the existing ProJ@s for another Year

and to make a new round of grants,

2957 Federal Register 9490 (Mar.

M Meehan, Op. cit., footnote 24.

I I Nolan, Op. Cit., footnote 3.

lz Meehan, op. Cit., f(~)tnote 24-

33 HR. 1804 and S. 846.

18, 1992),

Chapter 6: Implementation of Performance Standards and Measures 109

considered by Congress. The bill would establisha national skill standards board, charged withidentifying broad occupational clusters and en-couraging the development of voluntary, nationalskill standards for each cluster. The legislationauthorizes $15 million annually for this purpose.

CONCLUSIONThe Department of Education took seriously its

mandate to help states implement new accounta-bility systems for vocational education and, withthe help of NCRVE and other entities, hasprovided a reasonable level of technical support

on this issue, especially considering the limitedstaffing capacity of OVAE and the complexity ofthe issues involved. It is likely that in the futuremore attention will be required to establishvalidity and reliability of the methods of testingand assessment being employed in vocationaleducation.

The relationship among state performancestandards, the ED and DOL skill standardsprograms, and new legislation affecting nationalskill standards raises important issues warrantingcontinued attention.

Appendix A:Legislative Milestones

Influencing Accountabilityin Federal VocationalEducation Programs A

1917: Smith-Hughes ActRequired states to adopt minimum standards forteacher qualifications, plant and equipment, andhours of instruction in federally supported voca-tional programs.Required states to develop plans with approval ofa Federal Board for Vocational Education.Identified the outcome of vocational educationprograms as preparing students to be fit for usefulemployment.

1963: Vocational Education Act●

●

●

●

Required states to conduct and use periodicevaluations of state and local vocational educa-tion programs in light of current and projected‘‘manpower’ needs and job opportunities.Reserved 3 percent of state grants for ancillaryservices, including program evaluation.Established an ad hoc federal advisory council toreview and recommend improvements in voca-tional education programs.Revised state plan provisions to require moredetailed assurances and descriptions.

1968: Vocational Education Amendments. Continued requirements for evaluations based on

labor market needs.. Authorized funds for evaluation and other activi-

ties related to state administration.

111

Limited federal funding only to vocational pro-grams that could be “. . . demonstrated to preparestudents for employment or for successful com-pletion of such a program or be of significantassistance to participants in making an informedand meaningful occupational choice. ’Created a standing national advisory council toevaluate program effectiveness.Established state advisory councils with programevaluation duties.Required states to develop both long-range andannual plans.Mandated local plans.

1976: Vocational Education Amendments●

●

●

●

●

Directed states to evaluate the effectiveness ofeach local program within the 5-year duration ofthe state plan.Required programs that sought to impart entry-level job skills to be evaluated according to theextent to which program completers and leaversfound employment in occupations related to theirtraining (or pursued additional education) andwere considered by employers to be well trained.Expanded the evaluation duties and enhanced theindependence of state advisory councils.Created local advisory councils.Authorized technical assistance and siudies froma national advisory council and a new nationalresearch center, and authorized a national dataand occupational information system.

112

●

●

●

Testing and Assessment in Vocational Education

Clarified and strengthened the evaluation andmonitoring responsibilities of the Department ofHealth, Education, and Welfare.Expanded state planning duties.Required states to summarize evaluation findingsin an annual accountability report and to assesscurrent and future job needs within the state.

1982: Job Training Partnership Act. Pioneered use of performance standards to select

service providers, encourage quality and effi-ciency, and target programs for rewards orsanctions.

1984: Carl D. Perkins VocationalEducation Act

Charged states with developing measures ofprogram effectiveness in such areas as occu-pational skills and basic employment competencies,with separate measures for disabled individuals.Required states to evaluate not less than 20percent of local recipients in each fiscal year.Continued requirements for advisory councilevaluations, national data on completers andleavers, state needs assessments, and a nationalassessment of vocational education.Deleted requirement for states to evaluate jobplacement and employer satisfaction for programcompleters and leavers.Simplified state planning and reporting requirements.Established state technical committees to developskill inventories for specific occupations.

1988: Hawkins-Stafford Elementaryand Secondary SchoolImprovement Amendments

. Amended the Chapter 1 program for disad-vantaged students to require schools that did notmeet standards for increased achievement amongparticipating children to develop and implement“program improvement” plans.

1988: Family Support Act● Created the Job Opportunities and Basic Skills

(JOBS) program for welfare recipients, which

required the Secretary of Health and HumanServices to develop outcome standards for judg-ing individual progress by October 1, 1993.

1988: Hunger Prevention Act. Amended the Food Stamp Employment and

Training Program to require the Secretary ofAgriculture to establish performance standardsfor measuring the employment outcomes ofparticipants.

1990: Carl D. Perkins Vocational and AppliedTechnology Education Amendments

Required states to develop standards and meas-ures of performance for vocational educationprograms, reflecting attainment of job or aca-demic skills, school retention and completion, orplacement.Required local recipients to: conduct an annualevaluation of vocational programs based on statestandards and measures; review the progress ofspecial populations; and assess student knowl-edge about the industry they are preparing toenter.Required local recipients that did not showsubstantial progress to develop and implementprogram improvement plans.Directed states to develop measurable goals andaccountability measures for special populationsand required local recipients to describe howaccess to programs of high quality will beprovided for special populations.Required states to conduct an initial assessment ofprogram quality and needs using measurableobjective criteria.Authorized technical assistance, advice, and datacollection on performance standards and evalua-tion by the Department of Education, the NationalCenter for Research on Vocational Education, theNational Occupational Information CoordinatingCommittee, and state advisory councils.Authorized grants to business-education com-mittees to develop skill standards in trades andindustries.

ACT — American College TestingAEA — American Electronics AssociationASVAB — Armed Services Vocational Aptitude

BatteryATIB — Academic Test Item Bank (Arizona)C-TAP —Career-Technical Assessment project

(California)DOL — U.S. Department of LaborED — U.S. Department of EducationFBVE — Federal Board for Vocational EducationIOCT — Industrial Occupational Competency

TestingJOBS — Job Opportunities and Basic SkillsJTPA — Job Training Partnership ActNCRVE — National Center for Research in

Vocational EducationNIE — National Institute of EducationNOCTI — National Occupational Competency

Testing Institute

Appendix B:List

ofAcronyms B

OOTC — Oklahoma Occupational Testing CenterO V A E — Office of Vocational and Adult

Education (U.S. Department ofEducation)

OVCAP — Ohio Vocational CompetencyAssessment Program

R&D — research and developmentSCANS — Secretary’s Commission on Achieving

Necessary SkillsSOCAT — Student Occupational Competency

Achievement TestingT A B E — Test of Adult Basic SkillsT IERS — Title I Evaluation of Reporting SystemT O C T — Teacher Occupational Competency

TestingV-TECS — Vocational-Technical Education

Consortium of the States

113

Index

Academic skillschanges in skills assessed, 16, 61-62definition, 2, 3, 46norm-referenced assessment, 7, 9, 16, 39, 42sources of skill information, 13-14, 48state assessment policies, 12, 46, 63-64statewide academic testing programs, 39, 42, 63strategies for obtaining assessment information, 13-14, 64

Academic Test Item Bank, 49-50Accountability y

changing themes, 4-6, 32-34current regulations, 22-26legislative history, 4-6, 26-32mechanisms, 32-33performance-based, 4-5, 21-34requirements, 2, 33-34special populations, 3, 24

Acronym list, 114ACT. See American College TestingAEA. See American Electronics AssociationAFQT. See Armed Forces Qualification American College Testing, 10, 67

assessment development process, 70job profiling, 71 -72

American Electronics Association, 18, 9

Test

-94American Society for Training and Development, 75American Vocational Association, 75Annual review requirement, 5Applied mathematics test, 69Applied technology test, 69Aptitude tests, 84-85Arizona, 49, 53Armed Forces Qualification Test, 90Armed Services Vocational Aptitude Battery, 17,86-91Assessment. See also specific skills assessed

definition, 7, 36-37, 39differences in state policies, 11 -12, 41-42

methods, 7, 39origins of assessment practices, 7, 38-41resources, 2, 9-11, 42-44state policies, 2, 11 -12, 24-25, 41-42use of results, 13, 51-52

ASVAB. See Armed Services Vocational Aptitude BatteryATIB. See Academic Test Item Bank

Broad technical skills, 44, 65changes in skills assessed, 15-16, 65cognitive skills, 19-20, 97-101core occupational skills, 18, 94-95definition, 2, 3, 16, 44, 46, 84design and technology, 18-19,95-97lengthening the skill horizon, 36, 83occupational maps, 18, 91-94state assessment policies, 13, 65vocational aptitudes, 17, 84-91

Business and education standards, 107-108. See a/so Industryskill standards

C-TAP. See California Career-Technical Assessment ProjectCalifornia, 53, 59,95California Career-Technical Assessment Project, 59, 60-61Cad D. Perkins Act. See Perkins Act of 1984Certification of student competence for employers, 2, 5, 13,

51CETA. See Comprehensive Employment Training ActCognitive skills, 19-20,97-101Competency assessment

definition, 8methods, 39-41shift away from, 15, 57-58

Competency testingdefinition, 8, 39item banks, 43, 74-75, 75-76shift toward, 14-15, 57-58written, 39

115


Competitive events, 8, 39Comprehensive Employment Training Act, 34Computer-based tutors for training experts, 100Core occupational skills, 18, 94-95Criterion-referenced assessment, 7, 16, 43,49, 63

definition, 9, 39NOCTI, 77VTECS, 73Work Keys, 69-70

Curriculummaterials, 11-12, 43-44, 72, 73where skills are assessed, 48-51, 54

DACUM process, 77Delaware, 78Design and technology, 18-19, 95-97Disabled students, 24DOL. See U.S. Department of Labor

ED. See U.S. Department of EducationEffectiveness of vocational programs

local reviews of, 1, 13, 21, 24, 27, 32levels of skills to be achieved, 4need for improvement, 4-5, 6occupations to be trained for, 4outcome measures, 5

Elementary and Secondary Education Act of 1965, 31Exit exams, 13-14, 48-50, 63Expansion of testing and assessment programs, 14-17,55-65

Family Support Act of 1988, 112FBVE. See Federal Board for Vocational EducationFederal Board for Vocational Education, 27Food Stamp Education and Training program, 34

General Education Provisions Act, 106Generic workplace skills, 45

changes in skills assessed, 16, 64-65detinition, 2, 3, 46, 84state assessment policies, 13, 46

Glossary of testing terms, 8Goals 2000: Educate America Act, 45-46, 108-109Grade levels of academic and vocational testing, 16,42,48,

50Great Britain, 96GT. See General Technical

Hawkins-Stafford Elementary and Secondary SchoolImprovement Act

Amendments, I I 2Hunger Prevention Act of 1988, 112

Industrial Occupational Competency Testing, 76Industry skill standards projects, 17, 23

demonstration projects, 18, 25, 91

purposes of federal demonstrations, 107state directors’ study, 3, 38skill standards board, 45, 109technical committees and, 4, 30

Instructional diagnosis, 2, 8, 11, 13, 30

IOCT See Industrial Occupational Competency TestingIowa, 56Item banks, 43

Arizona Test Item Bank, 49state resource, 1 I- I 2, 37, 43, 44use and availability, 38, 55, 82VTECS, 18, 74-76, 81

Job Opportunities and Basic Skills training program forwelfare recipients, 22-23, 34

Job placementand performance standards, 5, 24, 25, 42evidence of, 23, 28-29JTPA, Title II, 30-31evolution of legislation, 32

Job profiling, 71-72Job Training Partnership Act of 1982, 22-23, 25, 30-31, 34JOBS. See Job Opportunities and Basic Skills training

program for welfare recipientsJTPA. See Job Training Partnership Act

Kansas, 56

Learning or competency gainsand performance standards, 5, 24, 35, 105, 106difficulty of measuring, 16-17, 50, 63Chapter I, 32NOCTI, 79Oklahoma testing program, 42Work Keys, 72

Learning test, 70Legislation. See specific lawsListening and writing test, 69Location of information test, 69

Marine Corps Job Performance Measurement Project, 89Math. See Applied mathematics testMethodology

state survey, 2-4, 37-38Military Entrance Processing Command, 86Mississippi State University, 3, 38Motivation tests, 70, 102MPR Associates, 3

National Academy of Sciences, 87, 89National Assessment of Vocational Education, 22National Association of State Directors of Vocational

Education, 3

Index 117

National Center for Research in Vocational Education,35-36, 104

National Governors’ Association, 23National Occupational Competency Testing Institute, 9-11,

43, 67, 76National Skills Standards Board. 45-46. See also Industry

skills standardsNational system of performance standards. See Performance

standardsNCRVE. See National Center for Research in Vocational

EducationNew Jersey, 5New York, 41, 94, 95NOCTI. See National Occupational Competency Testing

Institute

Observation test, 70Occupational assessment, 3, 16

form of measurement, 53-55shift to written testing, 14, 58

skills assessed, 44-47, 59-6 Istate assessment policies, I 2, 41

state resources, 42-44why skills are assessed, 5 I -53

Occupational maps, 18, 91-94Occupational skills. See Core occupational skills

changes in skills assessed, 55-59definition, 2, 46differences compared to academic skills, 13shift to written testing, 15sources of skill information, 13-14, 16, 48-49state assessment policies, 12- 13, 35, 41state testing resources for, 43why skills arc assessed, 13-14, 5 I -53

Office of Technology Assessment, 1-2, 7, 11, 17, 36, 83Office of Vocational and Adult Education, 104, 107Ohio, 14, 56, 82Ohio Vocational Competency Assessment Program, 14, 56Oklahoma, 42, 49, 82Oklahoma Occupational Testing Center, 42-43OOTC. See Oklahoma Occupational Testing CenterOutcome-oriented assessment, 4

measures, 5national activities, 25

OVAE. See Office of Vocational and Adult EducationOVCAP. See Ohio Vocational Competency Assessment

Program

Pennsylvania, 5, 78Performance assessment

and written testing, 58-59, 85

critical issues, 9, 39definition, 7,9effects on instruction, 15, 58-59

hands-on performance tests, 7, 10, 73, 75, 77-79, 90statewide systems of, 53, 64

Performance-based accountabilitychanging themes, 32-34current requirements, 22-23evolution of legislation, 4-6federal regulations, 105-106legislative history, 26-32overview of 1990 amendments, 23-26

Performance standards, 2link to federal funding, 5, 25national system, 5-6, 25-26purpose, 5, 51special populations, 5technical assistance to states, 104-105

Perkins Act of 1984, 4, 29-30, 51, 1121990 amendments, 1-2, 46-47, 83, 103, 112

effects, 16overview, 23-24

effectiveness, 22evolution, 29-30special populations, 30state responses, 13-17

Program improvement. See Program effectiveness

Reading for information test, 69Research and demonstration, 107Resource management test, 70Reviews of local program effectiveness. See Effectiveness of

vocational programs

Smith-Hughes Act of 1917,4, 26-27, 111instructional hours per week, 4, 26

SOCAT. See Student Occupational CompetencyAchievement Testing

South Carolina, 56Speaking test, 70Special populations

and performance standards, 3, 5, 105balanced approach, 30, 34federal priority, 22, 24, 25

Standardized testing, 16, 39. See also Academic skillsdefinition, 9effectiveness, 59

State surveydesign and methodology, 37-38

State-local program reviews. See Effectiveness of vocationaleducation

Student Occupational Competency Achievement Testing, 67administration, 79-80future, 80overview, 76use, 78

Student Occupational Competency Testing Institute, 10-1 I


Student profiles and portfolios, 7, 43-44

Teacher Occupational Competency Testing, 76, 77,78Teacher qualifications, 4Teamwork test, 69Technical skills. See Broad technical skillsTennessee, 11.82Test administration and reporting, 70-71Test of Adult Basic Skills, 37, 49Testing. See also .speci]c tests by name

differences in state policies, 41glossary of terms, 8mandatory. 41-42policies. 2, 38-41resources, 2, 6-7, 10- I 1, 42-44state policies, 35-65

Texas, 56TIERS. See Title I Evaluation and Reporting SystemTitle 1 Evaluation and Reporting System, 31-32TOCT. See Teacher Occupational Competency TestingTypes of skills, 2, 3

changes in skills assessed, 15-16mix of skills assessed, 59-61

U.S. Department of Education, 2industry skill standards projects, 18initiative to develop standards, 107-108, 109regulations and monitoring, 105-106role, 103-104technical assistance efforts, 104-105

U.S. Department of Labor, 31industry skill standards projects, 18, 91initiative to develop standards, 107-108

V-TECS. See Vocational-Technical Education Consortiumof the States

Vocational-acadcmic integration, 16-17,44,50, 63-64Vocational aptitudes, 84-91, 85

aptitude versus achievement, 85types of vocational aptitude tests, 17

Vocational Education Act of 1963, 4, 27-28, 1111968 amendments, 4, 28, 1111976 amendments, 4, 28, 111

Vocational skills, 63changes in skills assessed, 15, 63definition, 2, 3, 46grade levels of, 48integrating with academic skills, 63sources of skill information, 49state assessment policies, I 1-12, 17, 38

Vocational-Technical Education Consortium of the States,10-11, 41-42, 67

development process for catalogs and instructional tools,73-74

future, 76item banks, 74-76origin and purpose, 72-73

Voluntary Skill Standards and Certification Program. SeeIndustry skill standards

Wageseffects of technical skills on, 90

West Virginia, 56Work Keys, 10, 14, 56

assessment development process, 70assessments, 69-70design, 68-69implementation, 72job profiling, 71-72origin, 67-68

Written testing, 7, 53components, 58critical issues, 9definition, 36effects on instruction, 85listening and writing tests, 69state assessment policies, 12, I 5

Testing and Assessment in Vocational Education ontents 1 Purposes of the Background Paper2 Evolution...

Documents

Transcript of Testing and Assessment in Vocational Education ontents 1 Purposes of the Background Paper2 Evolution...