Scenario-based scales: Integrating Guttman facet theory and Rasch measurement

Larry H. Ludlow, Ph.D.Boston College

Lynch School of EducationEducational Research, Measurement, and Evaluation

ludlow@bc.eduhttp://www.bc.edu/content/bc/schools/lsoe/academics/departments/erme.html

Prepared for:Institute of Education

Higher School of EconomicsMoscow, RussiaOctober, 2014

______________________________________________________________ Teachers with AfL most embedded AfL Practices Least Embedded (More highly skilled) (Less common) 4 + | | . | | 3 . + | . | .## | .## | 2 .#### + ###### | PSA5 .###### | .######## | ########### | PSA1 PSA2 1 .########### M+ .########## | PSA4 .########## | LISC5 PSA3 ########### | FB5 FB4 .###### | QCD3 0 ### + LISC4 .## | FB2 LISC1 QCD5 .# | FB3 . | QCD4 . | FB1 LISC2 -1 . + QCD1 | | LISC3 | | QCD2 -2 + | Teachers with AfL least embedded AfL Practices Most Embedded (Least skilled) (More common) ______________________________________________________________ Figure 1. Variable Map for the AfLMi

Lo

git

Valu

e

Variable Map for AfL

Peer-and Self-Assessment

Learning Intentions and Success Criteria Feedback

Questioning and Classroom Discussion

Levels of Practice

2

BUT what is the “meaning” for a given teacher?

Did this work roughly enough as intended?

Average |P=50% 1 |P=50% at Average|P=50% 5 Item Response | | on each Item |

6 + + + Embedded | | | | | | | | | + + + | | | | | | | | | PSA5 + + + | | | | | | PSA1 PSA2 . | | | | | | PSA4 . + + + LISC5 PSA3 | | | FB5 FB4 . | | | QCD3 .## | | | LISC4 .## | | | FB2 LISC1 QCD5

5 .#### + + + FB3 Established ###### | | PSA5 | QCD4 .###### | | | FB1 LISC2 .######## | | | QCD1 ########### | | PSA1 PSA2 |

4 .########### + + + LISC3 Emerging .########## | | PSA4 | .########## | | LISC5 PSA3 | QCD4 ########### | | FB5 FB4 | .###### | | QCD3 | ### + + LISC4 +

3 .## | | FB2 LISC1 QCD5 Sporadic .# | | FB3 | . | | QCD4 | . | | FB1 LISC2 | . + + QCD1 + | | | | | LISC3 | | | | | | QCD4 |

2 + PSA5 + + Never | | | | | | | PSA1 PSA2 | | | | | + PSA4 + + | LISC5 PSA3 | | | FB5 FB4 | | | QCD3 | | | LISC4 | | + FB2 LISC1 QCD5 + + | FB3 | | | QCD4 | | | FB1 LISC2 | | | QCD1 | | + + + | LISC3 | | | | | | QCD4 | | | | | + + + | | | | | | + + + | | | | | |

1 | | | Don’t understand

Fortunately, no one is at this level—50% prob of a “1”.

Most of the teachers are at these two levels.

This is the level to which we would like to see them advance—50% prob of a 6.

3

A few teachers are at this level.

Now we can show “average” response levels showing how teachers responded.

Teachers’ AfL Benchmark

Descriptions

04/20/23 Ludlow-ERME 4

Very useful description of how to interpret average scores.

Why not make a “scenario description” exactly like we have tried to do here?

Personal Engagement

According to Kahn (1990):

• Personal engagement represents “the simultaneous employment and expression of a person’s ‘preferred self’ in ways that promote deep connections to one’s work (whatever form that might take, paid or unpaid) and to other people.

• Such engagement leads to active, full, and satisfying involvements rather than obligatory, passive, or emotionally anemic ones.

•We become personally engaged when we find that it is meaningful and safe to express our full selves and when we are psychologically ready to do so. (p.700).

5

Our Broader Definition of Engagement•A positive, enthusiastic, and affective connection with a role that both motivates individuals to invest their valuable resources and simultaneously energizes them.

Ranges across lower levels through higher levels.

•Characterized by persistence, energy, focus, and interest experienced while enacting a role.

Highly correlated and range from lower to higher levels.

•A unidimensional construct that cuts across different adult roles we may assume.

6

What is psychological engagement?

• What does the concept of “full” engagement or “less than full” engagement look and feel like?

• Is it possible to be “disengaged”? • Is this construct on a continuum? And, if so

what is on each end of that continuum? • How do you measure engagement? • Can it be measured across roles or is this a role-

specific concept?

8

Conceptual Model

Productive Engagement in Later LifePaid Employment Caregiving

Currently working for pay Currently involved in helping a friend or relative age 18 or older who has trouble taking care of themselves because of a physical or mental illness, disability, or for some other reason--includes caring for them directly or arranging for their care by others

Volunteering Informal Helping

Did volunteer work in the last 4 weeks for any of the following: • a church, synagogue or other religious organization• a school or educational organization• a political group or labor union• a senior citizen group or related organization• any other national or local organization, including United Fund, hospitals, and the like

Provided unpaid help in the last 4 weeks to someone who does not reside with them (excluding ill/disabled), including:• providing transportation, shop, run errands• helping with housework, car maintenance, etc. • providing childcare 

9

Sample

• Conducted in September 2012 through Survey Sampling International (SSI).

• Stratification for activity involvement resulted in equal numbers of respondents per activity category: • 120 paid workers • 120 volunteers • 120 caregivers • 120 informal helpers

• 480 individuals age 50 to 89 • Mean age 63.18 (SD:8.32)• 60.6% Female• 38.8% Bachelors degree or higher • 90.9% Caucasian• 57.9% Married/cohabiting

• 92.1% Live independently 10

Classical Test Theory (CTT) vs Item Response Theory (IRT)

Classical Test Theory

•“Traditional” approach since 1904

•Based on notion of “true score” X = T + E

•Emphasis is on use and application of “total score”

•Typical procedures are Cronbach alpha, test-retest reliability, factor analysis

•Simple items, often using Likert scoring—SA to SD

Item Response Theory

•“Modern” approach since 1980’s

•Based on probability notion of “expected response” for a person to an item

•Emphasis on meaning and interpretation of individual items

•Many different statistical models for determining “expected responses”

•A particularly powerful model for item/scale development is the Rasch measurement model.

11

Rasch Model Measurement Principles

When measuring a construct, items should define a construct with these characteristics:

• Variation: wide range of beliefs, attitudes, opinions, abilities

• Unidimensional: address a single, common attribute of the construct

• Hierarchical: progress from “easier-to-difficult” to endorse or accomplish items/tasks

• Continuum: form a uniform continuum across this hierarchical, unidimensional variation

12

Ludlow-ERME13

Structure of a “variable”

Every variable stretches across a continuum: levels of knowledge, cognitive abilities, affective characteristics (e.g. engagement)

We construct a deliberate hierarchical sequence of items, like steps on a ladder, that show where a person is located and what can then be expected to advance along that continuum (or up the ladder).

High Engageme

nt

Low Engagement

How to turn Rasch principles into scenarios?Facet Theory (L Guttman)

• Methodological approach to reveal how we form beliefs, attitudes, perceptions such as “engagement”.

• Each such perception is based on many influences, circumstances, factors, dimensions, considerations.

o Each of these influences is called a “facet”.o Each facet is made up of different levels/elements.

• In essence, FT or facet design, reveals the facets and their elements.

14

Sentence mapping (Guttman)

• A visual, grammatical device to link the elements of the facets.• All possible combinations of one element from each facet are

formed.• We have 4 facets, each with 3 levels for a total of 3 x 3 x 3 x 3 =

81 combinationso These combinations are used to form sentences.o These sentences are paired to become our scenarios.

Our facets and elements are:

15

X’s Role Engagement Level Captured by His/Her

Interest Focus Energy Perseverance

(fascinated) (difficult to tear away)

(gets energized)

(gives all)

(moderate) (moderate) (moderate) (moderate)

(does not care)

(completely checked out)

(does not invest)

(almost always gives up)

Choosing Scenarios

Level Interest Focus Energy Perseverance

3 (fascinated) (difficult to tear away)

(gets energized)

(gives all)

2 (moderate) (moderate) (moderate) (moderate)

1 (does not care) (completely checked out)

(does not invest)

(almost always gives up)16

All 81 combinations are impractical, how do you choose?• Randomly select from the 81 combinations of Ii,Fi,Ei,Pi.• Extreme groups contrast procedure (B Bloom).

We chose the following combinations3 from the extreme positive (I1, F1, E1, P1) group, 3 from the extreme negative (I3, F3, E3, P3), and; 3 from the middle (I2, F2, E2, P2).

How did we start thinking about the enagagement construct?

Level Interest Level Focus Level Energy Level Perseverance Person total

3 Identifies strongly

3 Difficult tear self away

3 Gets Energy 3 Gives All Anan (9) 12

3 Fascinated 3 Intensely focused 3 Invests much energy

2 Persists when difficulties arise

Melissa (7) 11

3 Enthusiastic 2 Pretty focused 3 Goes above and

beyond2 Deals with

challenges Tina (1) 10

2 Interested 2 Mind wanders

occasionally but pays attention

2 Does what it takes

2 Handles difficulties

Clair (6) 8

2 Somewhat

interested2 Generally

focuses2 Does what is

required2 Keeps at it when

difficulties ariseStan (2) 8

2-1 Indifferent 1 Often thinks

other things2 Little desire do

more than required

2 Forces self keep going when difficult

Elyssa (4) 7

1 Tired of work 1 Thinks about

other things2 Does not invest

much energy1 Does not go out

of way when difficulties arise

Jackie (8) 5

1 Unenthusiastic 1 Usually thinking

about other things

1 Does not go out of way

1 Gives up when any effort required

Larry (5) 4

1 Does not care 1 Completely checked out

1 Does not invest any energy

1 Almost always gives up when any effort required

Jaime (3) 4

How did we think about the “level” of the scenarios?

Scenarios (using Work as the “Role”)

9) Anan identifies strongly with his work (in a positive way) and sometimes gets so wrapped up in what he is doing that it is difficult to tear himself away. He gives all of himself to his work and finds that he gets energized from doing so.  

7) Melissa is fascinated by her work and is usually intensely focused on whatever she’s working on. She is willing to invest much energy in order to do a good job and she persists when difficulties arise.

1) Tina feels enthusiastic about her work and is pretty focused on the task at hand. She goes above and beyond what is required and when challenges arise, she deals with them.

6) Clair is interested in her work and pays attention to whatever she’s working on. She does what it takes and handles difficulties when they arise, though her mind wanders occasionally.

2) Stan is somewhat interested in his work and generally focuses on whatever he’s working on. He does what is required and keeps at it when difficulties arise.

4) Elyssa feels indifferent about her work and often thinks about other things. She has little desire to do more than is required and has to force herself to keep going when things get difficult.

8) Jackie is tired of her work and usually thinks about how much she would rather be doing something else. She does not invest much energy in what she does and doesn’t go out of her way when difficulties arise.

5) Larry feels unenthusiastic about his work and is usually thinking about other things. He does not go out of his way to get tasks done and tends to give up when any effort is required.

3) Jamie does not care about his work and is completely checked out. He does not invest any energy at all in what he does and almost always gives up when effort is required.

On a typical day , are you…o Much more engaged than Xo More engaged than Xo About as engaged as Xo Less engaged than Xo Much less engaged than X

19

MEASURE Person - MAP - Item <more>|<rare> 8 # + | | | | 7 + | | | | 6 + | | | | 5 + | | | | 4 + .# | | .## | # | 3 + ## | | ########## | SC9 ######## | 2 + .########## | # M| SC1 SC7 ##### | .### | 1 .## + | | ### | SC2 .## | SC6 0 .# +M # | | . | . | -1 . + | SC4 SC8 | .# | | SC5 -2 . + SC3 | | | | -3 + <less>|<frequent>EACH "#" IS 2. EACH "." IS 1.

Work9) Identifies strongly with “X” (in a positive way)…gets so wrapped up it is difficult to tear self away…gives all of self to and gets energized from doing so.

7) Fascinated and intensely focused…willing to invest much energy…persists when difficulties arise.

1) Enthusiastic and pretty focused…goes above and beyond and deals with challenges

2) Somewhat interested and generally focused…does what is required and keeps at it when difficulties arise.

6) Interested and pays attention…does what it takes and handles difficulties, though mind wanders occasionally.

4) Feels indifferent and thinks about other things…little desire to do more than is required…forces self to keep going when things difficult.

8) Tired and usually thinks about how much rather be doing something else…does not invest much energy and doesn’t go out of way when difficulties arise.

5) Unenthusiastic and usually thinking about other things…does not go out of way…gives up when effort is required.

3) Does not care…completely checked out…does not invest any energy…gives up

High scoring people

Low scoring people

“Variable Map”

20

Higher level of engagement

Lower level of engagement

Harder scenarios

Easier scenarios

MEASURE Person - MAP - Item <more>|<rare> 8 # + | | | | 7 + | | | | 6 + | | | | 5 + | | | | 4=41 + .# | | .## | # | 3=39 + ## | | ########## | SC9 ######## | 2=35 + .########## | # M| SC1 SC7 ##### | .### | 1=31 .## + | | ### | SC2 .## | SC6 0=27 .# +M # | | . | . | -1=22 . + | SC4 SC8 | .# | | SC5 -2=19 . + SC3 | | | | -3 + <less>|<frequent>EACH "#" IS 2. EACH "." IS 1.

Work

21

9) Identifies strongly with “X” (in a positive way)…gets so wrapped up it is difficult to tear self away…gives all of self to and gets energized from doing so.

4) Feels indifferent and thinks about other things…little desire to do more than is required…forces self to keep going when things difficult.

8) Tired and usually thinks about how much rather be doing something else…does not invest much energy and doesn’t go out of way when difficulties arise.

But what does a score actually mean?

For example, what is the difference between a person who scored 38 and one who scored 22?

Where are those scores and what is their “engagement difference?

MEASURE Person - MAP - Item

WORK 8 # + | | | | 7 + | | | | 6 + | | | | 5 + | | | | 4 + .# | | .## | # | 3 + ## | | ########## | SC9 ######## | 2 + .########## | # M| SC1 SC7 ##### | .### | 1 .## + | | ### | SC2 .## | SC6 0 .# +M # | | . | . | -1 . + | SC4 SC8 | .# | | SC5 -2 . + SC3 | | | | -3 + EACH "#" IS 2. EACH "." IS 1.

Person - MAP - Item

CAREGIVING + | | | | + | | | | + | | | | + . | | . | | + . | | | | # + ## | | .### | ##### | SC9 .##### + . | ##### | SC7 .#### | SC1 .#### M| .#### + ##### | ## | | SC6 ## | SC2 ##### +M . | .# | # | . | + . | SC4 SC8 .# | # | SC3 SC5 | # + . | | . | | + EACH "#" IS 2. EACH "." IS 1.

Person - MAP - Item

INFORMAL HELPING + | | | | + . | | | | + | | . | | . + | | | | # + | ######### | | SC9 | .#### + | ####### | | SC1 SC7 .####### | M+ .###### | ### | | ##### | + .## | SC2 ### | | SC6 .# | #### +M . | | .# | . | + | | . | . | + SC8 . | SC4 | | SC3 SC5 | + EACH "#" IS 2. EACH "." IS 1.

Person - MAP - Item

VOLUNTEER .## + | | | | + | | | | . + | | | | ### + | .## | | | .## + | ######### | | SC9 ########## | . + ###### | M| | SC1 SC7 .###### | .### + | .## | . | .### | .# + | SC2 .## | . | SC6 | # +M | | . | | + | | . | | + SC8 | SC4 . | | SC3 SC5 | + EACH "#" IS 2. EACH "." IS 1.

22

GAP

GAP

GAP

GAP

GAP

GAP

GAP

GAP

How well did our predicted scenario order come out?

MEASURE Person - MAP - Item

WORK 8 # + | | | | 7 + | | | | 6 + | | | | 5 + | | | | 4 + .# | | .## | # | 3 + ## | | ########## | SC9 ######## | 2 + .########## | # M| SC1 SC7 ##### | .### | 1 .## + | | ### | SC2 .## | SC6 0 .# +M # | | . | . | -1 . + | SC4 SC8 | .# | | SC5 -2 . + SC3 | | | | -3 + EACH "#" IS 2. EACH "." IS 1.

Person - MAP - Item

CAREGIVING + | | | | + | | | | + | | | | + . | | . | | + . | | | | # + ## | | .### | ##### | SC9 .##### + . | ##### | SC7 .#### | SC1 .#### M| .#### + ##### | ## | | SC6 ## | SC2 ##### +M . | .# | # | . | + . | SC4 SC8 .# | # | SC3 SC5 | # + . | | . | | + EACH "#" IS 2. EACH "." IS 1.

Person - MAP - Item

INFORMAL HELPING + | | | | + . | | | | + | | . | | . + | | | | # + | ######### | | SC9 | .#### + | ####### | | SC1 SC7 .####### | M+ .###### | ### | | ##### | + .## | SC2 ### | | SC6 .# | #### +M . | | .# | . | + | | . | . | + SC8 . | SC4 | | SC3 SC5 | + EACH "#" IS 2. EACH "." IS 1.

Person - MAP - Item

VOLUNTEER .## + | | | | + | | | | . + | | | | ### + | .## | | | .## + | ######### | | SC9 ########## | . + ###### | M| | SC1 SC7 .###### | .### + | .## | . | .### | .# + | SC2 .## | . | SC6 | # +M | | . | | + | | . | | + SC8 | SC4 . | | SC3 SC5 | + EACH "#" IS 2. EACH "." IS 1.

23

Where is the average level of engagement?

MEASURE Person - MAP - Item

WORK 8 # + | | | | 7 + | | | | 6 + | | | | 5 + | | | | 4 + .# | | .## | # | 3 + ## | | ########## | SC9 ######## | 2 + .########## | # M| SC1 SC7 ##### | .### | 1 .## + | | ### | SC2 .## | SC6 0 .# +M # | | . | . | -1 . + | SC4 SC8 | .# | | SC5 -2 . + SC3 | | | | -3 + EACH "#" IS 2. EACH "." IS 1.

Person - MAP - Item

CAREGIVING + | | | | + | | | | + | | | | + . | | . | | + . | | | | # + ## | | .### | ##### | SC9 .##### + . | ##### | SC7 .#### | SC1 .#### M| .#### + ##### | ## | | SC6 ## | SC2 ##### +M . | .# | # | . | + . | SC4 SC8 .# | # | SC3 SC5 | # + . | | . | | + EACH "#" IS 2. EACH "." IS 1.

Person - MAP - Item

INFORMAL HELPING + | | | | + . | | | | + | | . | | . + | | | | # + | ######### | | SC9 | .#### + | ####### | | SC1 SC7 .####### | M+ .###### | ### | | ##### | + .## | SC2 ### | | SC6 .# | #### +M . | | .# | . | + | | . | . | + SC8 . | SC4 | | SC3 SC5 | + EACH "#" IS 2. EACH "." IS 1.

Person - MAP - Item

VOLUNTEER .## + | | | | + | | | | . + | | | | ### + | .## | | | .## + | ######### | | SC9 ########## | . + ###### | M| | SC1 SC7 .###### | .### + | .## | . | .### | .# + | SC2 .## | . | SC6 | # +M | | . | | + | | . | | + SC8 | SC4 . | | SC3 SC5 | + EACH "#" IS 2. EACH "." IS 1.

24

38 38

38 38

Where is the same score across the 4 roles?

Conclusions & Implications

• Was it possible to offer a new definition of engagement?

Yes!

• Was it possible to construct meaningful and useful scenario items that captured one’s level of engagement?

Yes!

• Was it possible to define engagement and construct scenario-based scales that were invariant across the adult roles of work, caregiving, informal helping and volunteering?

Yes! 25

Patterns of Practice for Equity (POPE)The literature clearly identified the importance of teachers:•Selecting worthwhile content and designing learning opportunities aligned to valued outcomes•Connecting to students’ lives and experiences•Creating learning-focused, respectful and supportive learning environments•Taking an inquiry stance through using evidence to scaffold learning, and improve teaching,•Taking responsibility for further professional engagement and learning, and•Challenging inequities.

Teaching for Equity Enactment ScenariosWe potentially have six interconnected components (aspects/facets) of patterns of practice for equity but this number may be modified. We will initially construct scenario-type items for each selected pattern of practice for equity facet, defining enactment of that dimension from excellent, through moderate, to limited (high, medium, low).

We will use the following steps to develop and test the Rasch tool:•Select the patterns of practice for equity facets to be included in the tool and identify the elements of each facet. Typically the identification of facets and their elements requires considerable iterations and revisions.•Select an exemplar term, plus synonyms for this term, for each pattern of enactment (excellent, moderate, limited) of the selected facets. •Construct mapping sentences. These contain the formal elements of the facets and the informal components of natural language in order to contextualize the facets.•Determine which patterns of enactment will form part of the tool. We will combine the exemplar terms from one scale with those of the other facets. If we select 6 facets, each with three levels, there are potentially 36 = 729 possible combinations. We will, instead, use an “extreme groups contrast” procedure with three scenarios from the higher facet level, three from the moderate level and three from the lower level. We intend to follow this approach as it maximizes the possibility of working with scenarios that capture the boundaries of patterns of practice for equity enactment levels.