Elizabeth L. Shoenfelt, Ph.D.Alicia Turner, M.A. CandidatePatricia Slack, M.A. Candidate

David Normansell, M.A. Candidate

Department of PsychologyWestern Kentucky University

ASSESSMENT of the CONTENT VALIDITY

of the WATER TRAINING INSTITUTE (WTI)

CURRICULUM

OVERVIEW

Content Validation – Training Program Evaluation Methods

Criterion Phase Participants Procedure

Content Phase Participants Procedure

Analysis

Background: WTI

2 year associate program2 certification programs (Water and

Wastewater)Curriculum: general requirements and 2

specialized tracks (Water and Wastewater)4 projected entry level jobs that WTI graduates

will most likely enter: o Water Treatment Operatoro Wastewater Treatment Operatoro Distribution Systems Operatoro Collection Systems Operator

Background: NSF Grant

Grant evaluation includes an assessment of the content validity of the WTI Curriculum

Will examine the 4 courses currently in place to ensure they are preparing students with the Knowledge Skills and Abilities (KSAs) they need to successfully enter the workforce in any of the 4 identified entry level jobs

Content Validity

CONTENT VALIDITY: The extent to which the material taught in the training course reflects the actual KSAs required for effective job performance

The more similar the WTI training program content is to the job, the more effective it should be in preparing WTI program graduates.

Content Validation – Training Evaluation Methods

Matching Technique (Ford & Wroten, 1984) Analyzed the effectiveness of a Police department training program by identifying Knowledge, Skills, and Abilities (KSAs) considered important to job performance and the KSAs that currently taught in the training program (i.e., time spent)

• Used a matching matrix to graph the data to identify Hits and Misses• Hit: training emphasis reflects training needs (i.e., high importance KSA receives high emphasis in program)• Miss: training emphasis does not reflect training needs

• o Deficiency: high importance KSA receives low emphasis

• o Excess: low importance KSA receives high emphasis

Matching Matrix

Content Validation – Training Evaluation Methods

Linking Technique (Teachout, Sergo, & Ford, 1997) Used the matching matrix but also linked training emphasis to difficulty of learning

More effective picture: High Importance KSA with Low Emphasis – is it a Deficiency or does it have Low Difficulty of Learning?

Current Evaluation

Will examine the content validity of the WTI curriculum

Will use combination of the Matching and Linking Techniques

2 Phases: Criterion (Job)Content (WTI Curriculum)

Criterion Phase: Procedure

Developed preliminary Job Knowledge Surveys (JKS) based on materials received from the Associate Director of the Center for Water Resource Studies at WKU that included the KSAs needed for the 4 entry level jobs

For each element of job information, JKS included ratings for: a) Time Spent on job

b) Importance to jobc) Difficulty of Learningd) When learned (before hire/formal trn/on the

job)e) Should it be taught in WTIf) Is it needed for certification

Job Knowledge Survey: Sample

Criterion Phase: Participants

WTI Steering Committee Members

Knowledgeable individuals about Water and Wastewater Industry for both Kentucky and Tennessee.

From a variety of organizations from Kentucky and Tennessee such as Kentucky Rural Water Association and Tennessee Association of Utility Districts

JKSs were distributed to steering committee members during their WTI workshop/conference in Louisville, KY. The members pilot tested the surveys and provided feedback to refine surveys

Criterion Phase: Procedure

The JKSs were refined based on Steering Committee feedback

Steering Committee members collectively identified 40 water and wastewater incumbents to serve as job content Subject Matter Experts (SMEs) to complete the JKSs(Approximately 10 per entry level job)

The JKS were mailed to the Steering Committee Members for distribution to the identified job incumbent SMEs with a return deadline of October 30, 2009

The JKS data have been entered.We are now in the process of analyzing the JKS data.

Content Phase: Participants

Second Phase is a content analysis of the WTI Curriculum

We will develop Course Content Surveys to determine KSAs taught and training emphasis (time)

WTI Curriculum Content SMEs:o Bowling Green Community College WTI Instructoro Teaching Assistant for the Center for Water Resource

Studies at WKUo The 4 current WTI students also will complete the Course

Content Surveys

Content Phase: Procedure

The SMEs will develop a list of KSAs that are taught in each of the current 4 courses through a “brainstorming” panel discussion

At the end of the discussion, Curriculum SMEs will be provided with the course syllabi and any relevant training materials to further identify KSAs taught

The KSAs identified will be used to develop a Course Content Survey. Each KSA taught in the WTI course will be rated on the time spent teaching the KSA.

The SMEs (including students) will complete the survey

Analysis

• Matching Matrix will be used to compare: • Importance ratings vs. Time Spent Training ratings• Difficulty of Learning ratings vs. Time Spent Training

ratings

• Identify hits and misses in the courses:• Hits – appropriate emphasis in curriculum• Deficiencies – where more emphasis is needed• Excesses – where time on topic can be devoted

elsewhere

Matching Matrix

Formative Evaluation

Data from the Content Validation can be used:

• To revise current WTI courses (if needed) to ensure appropriate emphasis of job-related KSAs

• In developing other WTI courses to ensure they contain appropriate emphasis of job-related KSAs

References

Primary References

Bownas, D., Bosshardt, M., & Donnelly, L. (1985). A Quantitative Approach to Evaluating Training Curriculum Content

Sampling Adequacy. Personnel Psychology, 38. 117-131.

Ford, J., & Wroten, S.(1984). Introducing New Methods for Conducting Training Evaluation and for Linking Training

Evaluation to Program Redesign. Personnel Psychology, 37. 651-665.

Sproule, C. Rationale and Research Evidence Supporting the Use of Content Validation in Personnel Assessment.

Retrieved from International Personnel Assessment Council Website: http://www.ipacweb.org

Teachout, M., Sego, D., & Ford, J. (1997). An Integrated Approach to Summative Evaluation for Facilitating Training

Course Improvement. Training Research Journal, 3. 169-184.

Questions?

Content Validity Ratio ((N imp – N not-imp) / N)