Irt-mirt for Pemea (David, A.)

8/9/2019 Irt-mirt for Pemea (David, A.)

1/19

Unidimensional and Multidimensional Unidimensional and Multidimensional IRT Analysis for a Competency Test inIRT Analysis for a Competency Test in

Guidance & Counseling Guidance & Counseling

Adonis P. DavidAdonis P. DavidPhilippine Normal UniversityPhilippine Normal University


2/19

Wh at to Expect:Wh at to Expect:

1. Brief overview of t h e unidimensional and 1. Brief overview of t h e unidimensional and multidimensional approac h in IRT analysis.multidimensional approac h in IRT analysis.

2. Introduction to t h e development of a competency 2. Introduction to t h e development of a competency test in guidance and counseling.test in guidance and counseling.

3. Examination of t h e test using unidimensional and 3. Examination of t h e test using unidimensional and

multidimensional approac h es.multidimensional approac h es.


3/19

Background:Background:

In the Philippines, there have been recent efforts to developcognitive measures like achievement and aptitude testsusing the framework of Item Response Theory ( IRT).

There are instances were a measure is not measuring one

single trait or ability (unidimensional) but a set of traits or abilities (multidimensional).

The probability of successfully answering a test item in aunidimensional model depends on one underlying abilitydimension, while the probability of success in amultidimensional model depends on multiple abilitydimensions (Hartig & Hohler, 2009).


4/19

Unidimensional IRT AnalysisUnidimensional IRT Analysis

Refers to the traditional and standard IRTmodels commonly used in the development of cognitive measures.

In this approach, items are expected or assumed to be unidimensional- that is, itemsare supposed to measure a similar, singlelatent trait.

In Rasch analysis, a single latent trait isassumed to determine individuals'performances on a test (Wang, Yao, Tsai,Wang, & Hsieh, 2006).


5/19

Unidimensional IRT Approac hUnidimensional IRT Approac h

ABILITY

ITEM1

1

ITEM2

ITEM3

ITEM4

ABILITY 1

ITEM11

ITEM2

ABILITY 2

ITEM31

ITEM4

1A 1B

Composite Approach

Consecutive Approach

Composite Approach

Consecutive Approach


6/19

Multidimensional IRT AnalysisMultidimensional IRT Analysis

An extension of the traditional IRT models.

MIRT has been described as a highlyinteresting methodology for assessing

competencies within educational contextssince it is capable of taking into accountmultiple abilities simultaneously (Hartig &Holer, 2009).

Done through assuming correlations andsimultaneously calibrating all the items and allthe subtests of a given measure.


7/19

Multidimensional IRT Approac hMultidimensional IRT Approac h

Between

Between- -

Item Within

Item Within- -

Item

Item

ABILITY 1

ITEM11

ITEM2

ABILITY 2

ITEM31

ITEM4

2A 2B

ABILITY 1

ITEM11

ITEM2

ABILITY 2

ITEM31

ITEM4


8/19

The Guidance and C ounselingThe Guidance and C ounselingC ompetency Test (G CC T)C ompetency Test (G CC T)

a. A competency test for undergraduate andgraduate guidance and counseling & psychologystudents.

b.A paper-and-pencil test in a multiple-choiceformat.

c. Has six subtests corresponding to the six areascovered by the licensure examination in guidanceand counseling.

d. Has a total of 50 items.e. Designed to measure evidence of minimum

competency expected in a guidance counselor.


9/19

Development of the G CC TDevelopment of the G CC T

REV IEW OF L ITERATURE/REV IEW OF L ITERATURE/SURVEY OF EX IST INGSURVEY OF EX IST ING

LIC ENSURE/A C HIEVEMENT TESTLIC ENSURE/A C HIEVEMENT TESTIN GU IDANC E AND C OUNSEL INGIN GU IDANC E AND C OUNSEL ING

DEVELOPMENT AND EXPERTDEVELOPMENT AND EXPERTVALIDATON OF THE INITIALVALIDATON OF THE INITIAL

ITEM POOLITEM POOL(100 ITEMS)(100 ITEMS)

GCC TGCC TFORFOR

IRT ANALYS

IS

IRT ANALYS

IS(50 ITEMS)(50 ITEMS)


10/19

P articipants:P articipants:

Ninety-six (96) college and masters level students from twouniversities in Manila participated in this study(M age=25.23 yr.); 64 (66.7%) were females, 32 (33.3%) weremales. All of the college level participants were psychologymajors while majority of the masters level participantswere guidance and counseling majors; the others are fromallied courses (e.g. school psychology).


11/19

IRT ANALYSIS:IRT ANALYSIS:

The program C onstruct Map, a graphical, menu-drivensoftware package combining a multidimensional IRTengine for estimating item and person parameters(Kennedy, Wilson, Draney, Tutunciyan, & Vorp, 2008), wasused in all IRT analysis.

In the unidimensional model, all 50 items of the G CC T wereassumed to load in one single latent dimension (compositeapproach).

In the multidimensional model, the items were assigned intoone particular dimension corresponding to one of thesix subtests of G CC T (between-item multidimensionality).


12/19

F INDINGS:F INDINGS:

1. The item response data was represented by themultidimensional model better than theunidimensional model.


13/19

MODEL F IT:MODEL F IT:

Deviance Number of

Parameters

UnidimensionalUnidimensional 5,484.5785,484.578 5151

MultidimensionalMultidimensional 5,444.6575,444.657 7171


14/19

ADDITIONAL ANALYSIS: ADDITIONAL ANALYSIS:

Given that the multidimensionality of the G CC T was justified, the six subtest of the G CC T wereexamined in terms of their item and abilityestimates.

Unidimensional modeling refers to estimation of items that load to one specific dimensionindependent of the other dimensions (consecutiveapproach).

Multidimensional modeling of the six dimensionswhere the dimensions are assumed to load toone latent trait or competency.


15/19



2. The estimated difficulty of the items in eachdimension was similar regardless if the dimensionis considered as unidimensional or multidimensional.


16/19

DimensionDimensionModelModel

ItemItemDifficultyDifficulty

StandardStandardErrorError

Item FitItem Fit No. of No. of MisfitsMisfits

CounselingCounselingUnidimensionalUnidimensionalMultidimensionalMultidimensional

.03.03

.03.03.15.15.16.16

1.021.021.061.06

4455

Group ProcessGroup ProcessUnidimensionalUnidimensionalMultidimensionalMultidimensional

--.10.10--.10.10

.17.17

.17.171.011.01.99.99

1100

TestingTestingUnidimensionalUnidimensionalMultidimensionalMultidimensional

.11.11

.11.11.17.17.17.17

.95.95

.97.972222

FoundationsFoundationsUnidimensionalUnidimensionalMultidimensionalMultidimensional

.17.17

.18.18.15.15.16.16

.95.951.001.00

1100

Career GuidanceCareer GuidanceUnidimensionalUnidimensionalMultidimensionalMultidimensional

.29.29

.31.31.17.17.17.17

.95.95

.99.991111

OrganizationOrganizationUnidimensionalUnidimensionalMultidimensionalMultidimensional

.30.30

.30.30.19.19.17.17

1.011.01.93.93

1111

Comparison of Item Difficulty and Item F it StatisticsComparison of Item Difficulty and Item F it Statistics


17/19



2. The estimated difficulty of the items in eachdimension was similar regardless if the dimensionis considered as unidimensional or multidimensional.

3. The results obtained showed moderate tosubstantial difference in the ability estimation of several cases.


18/19

Student 1 Student 2 Student 3

GCCTGCCT

DimensionDimension

RawScore

UD MD RawScore

UM MD RawScore

UD MD

CounselingCounseling 1212 .37.37 1.151.15 88 --.07.07 .15.15 22 --.89.89 --1.531.53

GroupGroup 55 --.36.36 --.02.02 22 --.99.99 --.88.88 44 --.56.56 --.52.52

TestingTesting 44 --.54.54 --.31.31 33 --.87.87 --.57.57 44 --.54.54 --.31.31

FoundationsFoundations 33 --.01.01 .32.32 44 .04.04 .58.58 44 .04.04 .89.89

CareerCareer 22 --.87.87 --.12.12 22 --.87.87 --.22.22 22 --.87.87 --.18.18

OrganizationOrganization 11 --2.232.23 --.82.82 11 --2.232.23 --.82.82 00 --2.392.39 --.99.99

Comparison of t h e Unidimensional and Multidimensional Comparison of t h e Unidimensional and Multidimensional

Ability Estimates of Selected P articipants Across t h e 6 Dimensions Ability Estimates of Selected P articipants Across t h e 6 Dimensions .


19/19

CONCLUSIONS:CONCLUSIONS:

T h e analysis provided evidence on multidimensionality of t h e GCCT.

T h e GCCT would benefit from structural validation (e.g.Confirmatory F actor Analysis)

T h ere were some observed differences in t h e outcomes of t h e IRT analysis as a function of t h e type of modeling approac h (unidimensional vs multidimensional).

T h ere is evidence t h at results of IRT analysis depend a lot on t h e c h oice of model.

Irt-mirt for Pemea (David, A.)

Documents

Transcript of Irt-mirt for Pemea (David, A.)