Integrated Performance Assessment: Linking Individual Development to Industry Standards

1

Transforming Professional DevelopmentTransforming Professional Development© 2007 IPA Research Foundation© 2007 IPA Research Foundation2007 International AECT Conference, 10/27/2007

Integrated Performance Assessment (IPA)

Linking Individual Development to Industry Standards

Tammé McCowin Doctoral CandidateUniversity of Phoenix

2


Agenda

Topic Time (Mins.)

Literature Review 10

ISD Performance Inventory 5

Scale Development Process 10

Peer Activity: Using the ISD Performance Inventory

40

Research Participation 5

Questions and Answers 5

3


Literature ReviewTopics• Symptoms and Issues• Problem Statement• Research Purpose• Research Question• Literature Search• Research Findings• IPA Research Methodology

Topics• Symptoms and Issues• Problem Statement• Research Purpose• Research Question• Literature Search• Research Findings• IPA Research Methodology

4


Symptoms and Issues

• Instructional technology (IT) professionals must continually update and develop their skills in an ever-changing world market due to global competition, new methods and tools, fluctuations in the market, business acumen, strategy, positioning, and technological innovations

• There is a demand for IT professionals to continually adapt to agile work environments while acquiring new competencies in a rapidly changing industry

(Lui, Gibby, Quiros, & Demp, 2002; Katz, 2005)

(Lui, Gibby, Quiros, & Demp, 2002; Katz, 2005)

(Curtis, Hefley, & Miller, 2002; Lui et al., 2002; Lui, Jones, & Hempstreet, 1998; Katz,

2005)

(Curtis, Hefley, & Miller, 2002; Lui et al., 2002; Lui, Jones, & Hempstreet, 1998; Katz,

2005)

5


Symptoms and Issues

• IT professionals “have a responsibility to keep their skills current”

• It is an important and essential competency to update and improve one’s knowledge, skills, and abilities

• Skill utilization, extension, and versatility are critical components for sustaining professional success and technical professionals must continually extend their knowledge and skill capabilities to grow, innovate, and lead

(Katz, 2005)(Katz, 2005)

(Richey, Fields, Foxon, Roberts, Spannaus, & Spector, 2005)

(Richey, Fields, Foxon, Roberts, Spannaus, & Spector, 2005)

(Rothwell & Kazanas, 2004, p. 386)

(Rothwell & Kazanas, 2004, p. 386)

6


Problem Statement

• Effective job performance can be mitigated when no means is available to accurately and objectively assess the performance of IT professionals on industry defined competency standards

Instructional Technology (IT)

7


Research Purpose

• Career Planning and Professional Developmento To provide a valid and reliable measurement

methodology to guide career planning and professional development activities

• Leadership Developmento To develop technical leadership by focusing on

professional competencies, maturation, experience, and context

• Selection and Guidance Counselingo To provide an effective and objective measurement

methodology to aid employers and educators with making better decisions about job and student applicants (e.g. classification, placement, educational programming, recruitment, and selection)

8


Research Question

• What are the valid and reliable competencies, methods, and tools for assessing the preparation and performance of IT professionals?

Instructional Technology (IT)

Why ask this question?What someone thinks he or she is capable of doing, how

others know that he or she is capable, and how that capability may be demonstrated is not always in

alignment.

Why ask this question?What someone thinks he or she is capable of doing, how

others know that he or she is capable, and how that capability may be demonstrated is not always in

alignment.

9


Literature Search

0

9

2

5

4

6

0

1

2

3

4

5

6

7

8

9

Questia Pro/Psy ERIC Dissertations ETR&D PIQ

26 reference sources found26 reference sources found

Literary Sources

Tota

l

Number of References

10


Literature Search

11

4

1 1 1

0

2

4

6

8

10

12

ISD Practice CD MD TD Other

18 research studies found18 research studies found

Study Type

Tota

l

Number of Research Studies

Competency Development (CD), Model Development, (MD), Tool Development (TD)

11


Study Findings

• ISD Practice Studieso Software tools for empowering instructional developerso What do instructional designers actually do? An initial

investigation of expert practiceo Designer thinking: How novice and experts think about

instructional designo Context sensitive instructional design models: A

response to design theory, practice, and criticismo The present and future of ID practiceo Interactive multimedia design and production processes

o A history of instructional design and technology: Part I: A history of instructional media

(Rowland, 1992)(Rowland, 1992)

(Wedman & Tessmer, 1995)(Wedman & Tessmer, 1995)

(Winer & Vazquez-Abad, 1995)(Winer & Vazquez-Abad, 1995)

(Perez & Emery, 1995)(Perez & Emery, 1995)

(Lui & Hempstreet, 1998)(Lui & Hempstreet, 1998)

(Reiser, 2001a)(Reiser, 2001a)

(Gayeski, 1991)(Gayeski, 1991)

12


Study Findings

• ISD Practice Studieso A history of instructional design and technology: Part II: A

history of instructional designo The challenge of being an instructional designer for new

media development: A view from the practitioners

o Paradigms in the theory and practice of education and training design

o Instructional design practice: Career environments, job roles, and a climate of change(Larson & Lockee, 2004)(Larson & Lockee, 2004)

(Visscher-Voerman & Gustafson, 2004)(Visscher-Voerman & Gustafson, 2004)

(Reiser, 2001b)(Reiser, 2001b)

(Lui, Gibby, Quiros, & Demps, 2002)(Lui, Gibby, Quiros, & Demps, 2002)

13


Study Findings

• CD Studieso Roles and competencies of instructional design as

identified by expert instructional designers: a qualitative study*

o An examination of the instructional design competencies written by the International Board of Standards for Training, Performance, and Instruction*

o Instructional design competencies: The standards (3rd ed.)

o Building a foundation for the tomorrow: Skill standards for information technology 2003 edition

(Song, 1998)(Song, 1998)

(Richey, Fields. Foxon, Roberts, Spannaus, & Spector, 2001)(Richey, Fields. Foxon, Roberts, Spannaus, & Spector, 2001)

(Atchison, 1996)(Atchison, 1996)

(NWCET, 2003)(NWCET, 2003)

Competency Development (CD) * Denotes dissertation study

14


Study Findings

• MD Studieso ID model construction and validation: A multiple

intelligences case o The construction and validation of an instructional

systems design model incorporating multiple intelligences*

• TD Studieso The validity of a multiple-choice, paper and pencil

instrument in discriminating between masters and nonmasters of instructional design*

• Othero The comparative study of information competencies-

using Bloom’s taxonomy

(Stepp, 1990)(Stepp, 1990)

Model Development, (MD), Tool Development (TD)

(Tracey & Richey, 2007) (Tracey & Richey, 2007)

(Tracey, 2001) (Tracey, 2001)

* Denotes dissertation study

(Ven & Chuang, 2005)(Ven & Chuang, 2005)

15


IPA Research Methodology

ISD PerformanceInventory

Study

ISD PerformanceInventory

Study

Multiple IntelligencesISD Construct

Map Study

Multiple IntelligencesISD Construct

Map Study

Integrated SkillsAssessment

Study

Integrated SkillsAssessment

Study

IntegratedPerformance

Assessment Study

IntegratedPerformance

Assessment Study

11 22 33 44Perceived

RQWhat are the valid

and reliable competencies for

assessing the preparation and

performance of IT professionals?

RQ1What is the

relationship between MI Constructs and ISD competencies?

H1IT professionals who are able to correctly respond to a large

number of competency-

intelligence items will exhibit skill

integration.

RQWhat is the

relationship between perceived assessed and demonstrated

performance?RQ2What combination of

MI Constructs and ISD competencies

influence an IT professional’s skill

integration?RQ3

What combination of MI Constructs and ISD competencies

influence an IT professional’s skill

imbalance?

H2IT professionals who

are not able to correctly respond to a large number of

competency-intelligence items will exhibit skill

imbalance.

Research Question (RQ), Hypotheses (H), Instructional Systems Design (ISD)

assessed demonstrated

16


ISD Performance Inventory

Topics• Scale Composition• Rating Scale• Sample Professional Foundation

Scale

Topics• Scale Composition• Rating Scale• Sample Professional Foundation

Scale

17


Scale Composition

Scale C PS SIProfessional Foundations 5 26 63

Planning and Analysis 7 30 59Design and Development 6 32 58

Implementation and Management 5 34 56Digital Media 5 35 63

Technical Writing 5 30 52Web Development and

Administration8 55 92

Total 41 242 443

C - Competencies, PS - Performance Statements, SI - Scale Items

*Scale items were written to align to industry standards.

*Scale items were written to align to industry standards.

*International Board of Standards for Training Performance and Instruction (IBSTPI)*National Workforce Center for Emerging Technologies (NWCET)

18


Rating Scale

Main Scale Equivalent Scale

1. Absolutely need 1. Novice2. Strongly need 2. Advanced

Beginner3. Definitely need 3. Competent4. Somewhat need 4. Proficient5. No need 5. ExpertRating oneself as an expert on an individual item does not imply that you cannot or do not need to ever make improvements. Rather it is an

indication of where you believe you are to date regarding where you are in your career and the current state of the industry and based on the

standards that are relevant right now.

Rating oneself as an expert on an individual item does not imply that you cannot or do not need to ever make improvements. Rather it is an

indication of where you believe you are to date regarding where you are in your career and the current state of the industry and based on the

standards that are relevant right now.

19


Sample Professional Foundations Scale

Performance Statement

Scale Item Rating Scale

Create messages that accommodate learner needs and characteristics, content, and objectives

When designing instruction, write learner centric content that is factual, clear, unambiguous, and tied to relevant supporting visual aids to improve learning and comprehension of subject matter.

1 2 3 4 5

When designing instruction, write instructional objectives that clearly describe a measurable performance.

1 2 3 4 5

When designing instruction, write learner centric content that maps directly to documented instructional objectives.

1 2 3 4 5

Write and edit text to produce messages that are clear, concise, and grammatically correct.

When designing instruction, write learner centric content that is clear, concise, and grammatically correct.

1 2 3 4 5

When designing instruction, edit learner centric content that is clear, concise, and grammatically correct.

1 2 3 4 5

C1: Communicate Effectively in visual and oral form

1 – Absolutely need, 2 – Strongly need, 3 – Definitely need, 4 –Somewhat need, 5 – No need

20


Scale Development ProcessTopics• Four Step Scale Development

Process• Construct Delineation• Item Generation• Pilot Studies• Validation Studies• Scale Sample Sizes

Topics• Four Step Scale Development

Process• Construct Delineation• Item Generation• Pilot Studies• Validation Studies• Scale Sample Sizes

21


Four Step Scale Development Process

ConstructDelineationConstruct

DelineationItem

GenerationItem

Generation Pilot TestPilot Test Validation StudyValidation Study

11 22 33 44

Create higher order factors

Create factors

Create variables

Create control variables

Develop item pool

Create response format

Conduct expert reviews

Selects small study group samples

Administer instrument to

samples and collect data

Conduct factor and reliability analysis

Refine instrument

Select random sample

Administer instrument to

samples and collect data

Conduct factor and reliability analysis

Conduct concurrent validity analysis

22


Factor-Item Structure Relationships

Knowledge Domains

(7)

Knowledge Domains

(7)

Higher OrderedFactors

(7)

Higher OrderedFactors

(7)

ScalesProfessional Foundations

Planning and AnalysisDesign and Development

Implementation and ManagementDigital Media

Technical WritingWeb Development and Administration

ScalesProfessional Foundations

Planning and AnalysisDesign and Development

Implementation and ManagementDigital Media

Technical WritingWeb Development and Administration

11 22 33

Performance Statements (PS), Instructional Systems Design Competencies (ISDC)

CompetencyStatements

(41)

CompetencyStatements

(41)

PerformanceStatements

(242)

PerformanceStatements

(242)

Factors(41)

Factors(41)

Variables(242)

Variables(242)

=

=

=

ISDC(41)

ISDC(41)

Scale Items(443)

Scale Items(443)

=

=

=

23


Item Generation

Qualitative ReviewStep 1: Determine statement’s behavioral characteristics Step 2: Determine statement’s unitary valueStep 3: Determine statement’s clarityStep 4: Determine statement’s duality focusStep 5: Determine statement’s functional purposeStep 6: Determine statement’s face validity

Quantitative ReviewStep 7: Determine inter-rater reliability of initial item poolStep 8: Determine statement’s measurement typeStep 9: Determine statement’s skill level

Nine-Step Review Process

24


Pilot Studies

•Two sample groups will be pulled to complete pilot studies

•Group 1 will be used to test the skill inventory for reliability and factor structure relationships

•Group 2 will be used to refine the skill inventory for construct validity and to further confirm factor structure relationships

Group Samples

Phase One Instrument Development

• Study 1: 10-20 participants• Study 2: 10-20 participants

Group Samples

Phase One Instrument Development

• Study 1: 10-20 participants• Study 2: 10-20 participants

25


Validation Studies

•There are two sample study groups needed to complete validation studies

•Study 3 is to finalize the skill inventory and confirm test-retest reliability

•Study 4 is to establish concurrent and predictive validity

Group Samples

Phase Two Instrument Validation• Study 3: 276-460 participants• Study 4: 75 participants

Group Samples

Phase Two Instrument Validation• Study 3: 276-460 participants• Study 4: 75 participants

All participants who register to participate will be randomly placed in one or more study

groups. Email communication will be sent to you at the start of

study 1 to explain the details and list which group or groups

you have been randomly placed in.

All participants who register to participate will be randomly placed in one or more study

groups. Email communication will be sent to you at the start of

study 1 to explain the details and list which group or groups

you have been randomly placed in.

26


Scale Sample Sizes

Scale C PS SI SampleProfessional Foundations 5 26 63 189-315

Planning and Analysis 7 30 59 177-295Design and Development 6 32 58 174-290

Implementation and Management 5 34 56 168-280Digital Media 5 35 63 189-315

Technical Writing 5 30 52 156-260Web Development and

Administration8 55 92 276-460

Total 41 242 443 276-460

C - Competencies, PS - Performance Statements, SI - Scale Items

*International Board of Standards for Training Performance and Instruction (IBSTPI)*National Workforce Center for Emerging Technologies (NWCET)

The sample size of a scale is generally five times the size of a scale (DeVellis, 2003; Netemeyer, Bearden, & Sharma,

2003)

The sample size of a scale is generally five times the size of a scale (DeVellis, 2003; Netemeyer, Bearden, & Sharma,

2003)

27


Peer Activity: Using the ISD Performance

InventoryTopics• Reviewer Types• Instructions to Self Raters• Instructions to Third-party

Raters• Peer Activity: How to Use the


Topics• Reviewer Types• Instructions to Self Raters• Instructions to Third-party

Raters• Peer Activity: How to Use the


28


Reviewer Types

•Self Rater– An individual who voluntarily uses the ISD

Performance Inventory to rate themselves to obtain performance feedback

•Third-party Rater– An individual who voluntarily uses the ISD

Performance Inventory to rate another person to provide performance feedback (e.g. peer, teacher, manager, team member, etc.)

29


Instructions to Self-Raters

•You will be presented with items for each scale

•Rate the level of professional development you require on each item using the rating scale

•Use the ISD Skill Level Response Format Descriptions to judge your professional development on each item relative to the rating scale

30


Instructions to Third-party Raters

•Use the interview rubric to interview your partner

•Rate the level of professional development your partner requires on each item using the rating scale

•Use the ISD Skill Level Response Format Descriptions to judge his or her professional development on each item relative to the rating scale

31


Peer Activity• Select a partner and

determine your role (5-mins)

• Third-party rater interviews his or her partner (15-mins)

• Self rater and third-party raters complete the inventory (10-mins)

• Compare and share findings (5-mins)

• Reporting and analytics (5-mins)

How to Use the ISD Performance Inventory

How to Use the ISD Performance Inventory





1 2 3 4 5


1 2 3 4 5


1 2 3 4 5


When designing instruction, write learner centric content that is clear, concise, and grammatically correct. 1 2 3 4 5When designing instruction, edit learner centric content that is clear, concise, and grammatically correct. 1 2 3 4 5





1 2 3 4 5


1 2 3 4 5


1 2 3 4 5


When designing instruction, write learner centric content that is clear, concise, and grammatically correct. 1 2 3 4 5When designing instruction, edit learner centric content that is clear, concise, and grammatically correct. 1 2 3 4 5

C1: Communicate Effectively in visual and oral form

40-mins40-mins

32


Research Participation

Topics• Eligibility Requirements• Participation Benefits• How to Participate

Topics• Eligibility Requirements• Participation Benefits• How to Participate

33


Eligibility Requirements• Anyone who falls into one of the

instructional design roles or both and has performed the functions of any of these roles

content developer, eLearning developer, instructional developer, information architect, information developer, programmer, project manager, media designer/specialist, multimedia author/specialist, multimedia developer, training developer, user interface developer, web developer, web media developer, technologist, teachers, students

content developer, eLearning developer, instructional developer, information architect, information developer, programmer, project manager, media designer/specialist, multimedia author/specialist, multimedia developer, training developer, user interface developer, web developer, web media developer, technologist, teachers, students

analyst, course developer, content editor, eLearning specialist, electronic publisher, evaluator, communications, designer, creative writer, document specialist, instructional designer, interaction designer, online publisher, project manager, technical communicator, technical writer, trainer, training designer/specialist, usability designer, user interface designer, web designer, teacher, students

analyst, course developer, content editor, eLearning specialist, electronic publisher, evaluator, communications, designer, creative writer, document specialist, instructional designer, interaction designer, online publisher, project manager, technical communicator, technical writer, trainer, training designer/specialist, usability designer, user interface designer, web designer, teacher, students

Instructional Designer Role

Instructional Developer Role

34


Participation Benefits

•When you register and fully participate in the research study you will

– Receive your choice of one online training course to promote professional development toward linking your skills to industry standards

– Contribute to the research and validation of a online for performance assessment tool professional development

– Earn a chance to win either one 30GB video iPod or one of two 4GB iPod Nanos

35


How to Participate

• To participate you must:o Register at the research website, provide your

name, email address, contact number, and some general demographic information

o Acknowledge and give your formal consent to participate by selecting the terms and conditions check box on the registration form

o Finalize your registration by clicking the link in the confirmation email that will be sent to you

http://www.iparesearch.orghttp://www.iparesearch.org

Let’s transform professional development together!Share the link with your colleagues and friends!

http://iparesearch.blogspot.comhttp://iparesearch.blogspot.com

36


Questions and Comments?

Integrated Performance Assessment: Linking Individual Development to Industry Standards

Technology

Transcript of Integrated Performance Assessment: Linking Individual Development to Industry Standards