Singing from the Same Hymn SheetA VALIDITY STUDY ON ASSESSMENT OF THE BARBADOS

NVQ-B LEVEL I IN AMENITY HORTICULTURERoxanne Waithe

The University of the West Indies, Cave Hill Campus4th October 2016

Background to the studyoBarbados National Vocational Qualifications (NVQ-Bs) were established in 2003 but up until 2008 only the NVQ-B in Amenity Horticulture was administered

oNVQ-Bs for six occupations – organizations were not using them. Competency-based qualifications……Why?

oNVQ-B assessments – measurement of competency for the NVQ-B in Amenity Horticulture

Presentation OutlinePurpose of the studyResearch questionsReview of the LiteratureMethodologyResults Implications and conclusions

Purpose of the study1. Examine how the NVQ-B was being used to

measure employee competence in Amenity Horticulture

2. Establish the validity of assessment in the NVQ-B Amenity Horticulture Level I

Research questions - Phase 1Examine how the NVQ-B is used to measure employee competence based

on experiences and perceptions of key usersa) How are occupational standards for the NVQ-B used in compliant

organizations?b) How does the assessment process for the NVQ-B establish competence

in relation to occupational standards?c) What are some of the challenges associated with administering the

NVQ-B?d) What is the perceived influence of NVQ-Bs on the accomplishment of

competency?

Research questions – Phase 21) Establish the validity of assessment in the NVQ-B amenity horticulture

level 11. How does NVQ assessment reflect the evidence requirements outlined

in the national occupational standards for Amenity Horticulture?1) How congruent are assessors’ interpretations of evidence for the NVQ in

Amenity Horticulture?2) To what extent does the evidence collected during assessments for the

NVQ-B Level I in Amenity Horticulture establish the competencies outlined in the occupational standards?

Review of the LiteratureLITERATURE REVIEW I

Making NVQ assessment

decisions• Perceptions on the Proficiency of NVQs

Evolution of NVQs

LITERATURE REVIEW II

Approaches to

Assessment Validation

Modern Validity Theory

Defining validity

Underlying assumptions &

threats to validity

Are NVQs Valid?

Validity theory in practice

Seminal Sources - Literature Survey I Swailes & Roodhouse (2003) – Key features of NVQ qualifications Hager & Butler (1996) - Comparison of assessment in academic and vocational qualifications Qualifications and Curriculum Authority (2006) – Comparability study of NVQ assessment in practice Roe et al. (2006) Perceptions and use of NVQs

Competence: the ability to apply knowledge, understanding, practical, and thinking skills to achieve affective performance to the standards

required in employment (QCA, 2006)

Seminal Sources - Literature Survey II Greatorex & Shannon(2003) – Evaluation of validity of NVQs Messick (1989, 1995, 1996, 1998) - Validity theory Kane (1992, 2001, 2004, 2006, 2009, 2013) – Validity process and validity evidence Johnson & Blinkhorn (1992) Model for validating NVQ assessmentValidity: the extent to which an assessment decision about a candidate,

based on performance by the candidate, is justified. (TVET Australia, 2010)

Philosophical assumptionsInterpretive

NVQ-B users Multiple sources of evidence

• Interpretivist philosophy – facilitate experiences and perceptions of all key NVQ-B users (employers, administrators, candidates, assessors)

• Interpretative approach to assessment validation – multiple data sources at various assessment stages

• Interpretive inquiry processes serve to expand validation practice to include diverse evidences for the generation of multiple-perspective validity arguments. (DeLuca, 2011)

Case Study Methodology1. Case study research was compatible with my philosophical

viewpoint – interpretivist orientation toward data collection and analysis

2. Case study is ideal when the research questions take the form of ‘how’ and ‘why’. (Yin, 1984)

3. Case study is an appropriate vehicle for interpretive investigations because of its ability to accommodate different research techniques, both qualitative and quantitative (Yin, 2003)

Case Study MethodologyCASE: NVQ-B AMENITY HORTICULTURE

LEVEL IPhase I:

Exploratory study (How does assessment

work?)

Phase II: Assessment Validation

Assessment system

(processes)

Assessment content

Assessment decisions

Data collection – Phase I

Participants• NVQ candidates (6)• NVQ assessors (6)• TVET technical officers

(3)• Employers (25)

Methods• Interviews• Document analysis• Surveys

Data• NVQ system

(Operational context)• NVQ assessment• NVQ perceptions

Key findings from the research – Phase I

1. The NVQ-B qualification is patterned after the traditional British NVQ model. NVQ-B compliant organizations use occupational standards for the purposes of training, developing job descriptions, and conducting employee interviews.

2. NVQ-B Assessment Process – involves NVQ assessors gathering evidence of candidates’ competence through observation, questioning, photography or witness testimony

3. NVQ-B Challenges – were mostly assessment related including timely record-keeping for evidence of competence, and practical difficulties associated with assessing people in the workplace

4. NVQ-B Perceptions -

Improves employee

performance

Improves self-efficacy

Too much paperwork

What NVQs?

Data collection – Phase II

Participants• NVQ assessor trainers (2)• NVQ assessors (3)• TVET technical officers (3)• NVQ candidate trainers (3)• NVQ Coordinators (4)• External verifier (1)

NVQ Assessment Validation Framework• Interpretive argument• Validity argument• (Data collection: interviews,

document analysis, two focus groups)

Data• Assessment process• Assessment content• Assessment decisions

Assessment Validation for the NVQ-B in Amenity Horticulture Level I

The validation process is the assembly of an extensive argument for the claims that are made about an assessment. (Kane, 2006)

Purpose of the assessment validation was to produce validity evidence for assessment including

representativeness of content, coherence between occupational standards and assessment tasks, and the level

of congruence of assessment decisions under different conditions

Assessment Validation Framework

Present interpretive arguments about the

NVQ-B Amenity Horticulture Level I (Use

research questions)

Collect validity evidence to support or refute claims: document review, interviews,

focus groups

Systematic evaluation of evidence concerning assessment process, assessment content,

assessment decisions

Build a validity argument based on evaluation of

evidence (Messick, 1989; Kane, 2006, 2009)

Interpretive Argument 1 Assessment of the NVQ in Amenity Horticulture provides a

reasonable indication of competence in the occupational standardsSupporting claims

The content coverage of the NVQ assessment adequately represents the occupational standards for amenity horticulture

The NVQ-B assessment system uses a rigorous methodology for providing evidence of competence (knowledge, understanding and skills) in amenity horticulture

Interpretive Argument 2 Assessments for the NVQ in Amenity Horticulture Level I provide meaningful data on candidates’ readiness to perform in this field

Supporting claimsRules for assessment judgement are consistently and accurately appliedCandidates deemed competent in Amenity Horticulture Level I are likely to be given the same decision regardless of assessor or assessment locationA decision of ‘competent’ in Amenity Horticulture is a reliable indicator of performance in the occupational area

Key findings from the research – Phase II

Validity Evidence for assessment contentGenerally good coherence between NVQ assessment content and evidence requirements outlined in national occupational standards for amenity horticulture.

NVQ coordinators from the four registered training organizations concur that assessments adequately cover the content standards that they are designed to measure.

Threats to validity evidence

Measurement of some constructs related to health and safety that are critical to occupational competence are not sufficiently covered in practical assessment tasks

The evidence guide for assessors do not provide rules for decision-making or assessment exemplars that would allow comparison of content across parallel assessment events (nursery/golf course/cricket pitch/farm)

Key findings from the research – Phase II

Validity Evidence for assessment processThe NVQ-B system conducts assessments for amenity horticulture using a systematic method for collecting evidence of competence and making a judgement that the candidate can perform work in accordance with the occupational standards.

The assessment process is administered in four stages by TVET Council and the RTOs described to some extent in the NVQ-B user guide.

Threats to validity evidence

Blueprint for NVQ-B assessment not fully documented

The rating scale of 1 – 5 which was introduced to assess skills and competence is somewhat ambiguous and could lead to errors in judgement

Key findings from the research – Phase II

Validity Evidence for assessment decisionsAssessors have been trained to judge candidates’ performance against occupational standards.

High level of inter-rater agreement amongst assessors was recorded for a mandatory unit of assessment.

Threats to validity evidence

Rating scale for performance criteria does not provide enough information

Assessors are experts in either one or two areas of amenity horticulture

Validity evaluation for NVQ Amenity Horticulture Level IINTERPRETIVE ARGUMENT CLAIMS VALIDITY EVALUATION

1. Content coverage of the NVQ adequately represents the occupational standards for amenity horticulture

Evidence from competency mapping and participant interviews support the claim for content

2. The NVQ-B assessment system uses a rigorous methodology for providing evidence of competence in amenity horticulture

Evidence from document review and participant interviews support the claim for rigor

3. Rules for assessment judgement are consistently and accurately applied

Evidence from document review and participant interviews do not support the claim for consistent assessment judgement

4. Candidates deemed competent are likely to be given the same decision regardless of assessor or location

Research evidence does not support claim for equivalent assessor decisions

5. A decision of ‘competent’ is a reliable indication of performance in the occupational area

Evidence supports claim for ‘competent’ decisions where assessors assess in their area of specialization

Validity Arguments for NVQ-B in Amenity Horticulture

VALIDITY ARGUMENT IAssessment of the NVQ-B in Amenity Horticulture Level I provides a reasonable indication of competence in specific areas of Amenity Horticulture outlined in the occupational standards

VALIDITY ARGUMENT IIAssessments for the NVQ-B in Amenity Horticulture Level I provide meaningful data on candidates’ readiness to perform in their specific area of amenity horticulture.

Research implications How did the results compare with what was presented in the literature review?

1. Compared to other more commonly known qualifications, the NVQ-B is not easy to interpret. Cox (2007) review of employer and employee perceptions; Qualifications and Curriculum Authority (2010) comparability study of NVQ assessment in practice

2. The validity of assessors’ judgements for the NVQ-B in Amenity Horticulture is definitely influenced by assessor experience and training. Johnson (2008) judging vocational portfolios; Wolf (1995) competence based assessment

3. Assessment in the NVQ-B Amenity Horticulture tends to focus on adherence to the assessment process more than on standardizing assessment decisions. Greatorex and Shannon (2003, 2008) standardizing assessors’ judgements

Singing from the same hymn sheet• All NVQ-B users refer to the occupational standards

for Amenity Horticulture Level I as the primary resource to guide assessment

• Interpretation and use of the occupational standards depends on two factors:

1. Role in the NVQ assessment process2. Context in which the assessment takes place

Thank you.