Evaluating the TCI questionnaire using SEM

Evan KontopantelisStephen Campbell

David Reeves

Overview• A warning… • Part one

– Organizational (Team) Climate– Team climate in primary care– Team climate concerns– Team Climate Inventory– Data structure of the TCI– Validation of the TCI– So, what’s the problem?

• Part two– Data collection– What we hope to achieve & how– Structured Equation Modelling

• A few notes• Results: Items models• Results: Scores models

– Exploratory Factor Analysis– Conclusions– Future work

A warning…

• "Oh, people can come up with statistics to prove anything, Ken. 14% of people know that“

Part one

Formulation of the TCI questionnaire and prior

analyses

Organizational (Team) Climate

• Cognitive schema approach: “…individuals’ cognitive representation of proximal environments…expressed in terms of psychological meaning and significance to the individual…” (James & Sells 1981)

• Shared perceptions approach: “…the shared perception of the way things are around here” (Reichers & Schneider 1990)

• These approaches are compatible, in principle

Team climate in primary care

• Healthcare professionals work in integrated teams in order to improve task effectiveness, morale and team viability

• Structural changes may only be translated to positive outcomes if team-level processes are effective

• Team climate is a concept that may be of relevance to team processes, hence we “measure” and relate it to performance(Bower, Campbell, Bojke, Sibbald, 2003)

Team climate concerns

• Can TCI be considered to quantify team climate (and hence become a useful measure/predictor of performance)?– A large variability of responses within a

practice would indicate that the “shared perception” theory cannot really be applied

– Can there ever really be only one team climate in a big multi-professional general practice?

Team Climate Inventory

• 65 item measure with six subscales that attempts to quantify the team climate within a practice, based on the shared perceptions approach (Anderson & West, 1994)

• All items on a scale of 1 to 5• The six subscales (factors) are:

Participation Task orientation

Support for innovation

Reflexivity

Clarity of objectives Teamworking

Data structure of the TCI

• Although it’s a 3-level structure ( items, respondents & practices), only two level constructs can be examined (R:P)

• We either use all the items for each respondent (items models) or the average scores of the subscales (scores models)

• Averaging the responses within a practice, for each item, data is condensed to single level

Validation of the TCI• Initially a four-factor theory (Anderson & West,

1994)• Exploratory and Confirmatory Factor

Analysis indicated that a five-factor construct was more suitable but…

• Although “…examining item statistics at the individual level avoids additional problems of dealing with summed data at the team level…” analyses use group level sums for each item (Anderson & West 1998)

• The five factor construct was again verified with Norwegian data using group level sums (Mathisen et al 2004)

So, what’s the problem?

• Exploratory factor analyses have been used on unaggregated data, ignoring the nested structure

• Confirmatory analyses using SEM have only been used with group averages, ignoring various statistical issues

• No published analyses on the six subscale measure which is based on the earlier versions

Part two

NPCRDC data and preliminary analyses

Data collection

• The questionnaire was distributed to all clinical, nursing and administration staff working in a sample of 60 practices in 1998 and 42 of the same practices in 2003

• Response rates varied greatly by practice

1998 2003number of practices 60 42

average respondents per pract 9.5 12.2

average resp rate per pract 63.1% 65.1%

What we hope to achieve & how

• What? – Validate the usage of TCI as an independent

variable in QuIP analyses• How?

– verify the validity of the 6 dimension construct on which the questionnaire was based taking into account the nested structure of the data.

– compare the full-item structure with the score one, in terms of complexity and information provided (+)

– use Exploratory Factor Analysis to see if there is only one factor per dimension

A few notes on Structured Equation Modelling

• Hybrid technique that encompasses aspects of confirmatory factor analysis, and regression

• It encourages confirmatory rather than exploratory modelling, hence it is better suited for theory testing rather than theory development

Observed variables

Latent variables

Causality (reg equations)

Error terms

Correlations

Estimated parameters

Results - Scores models

• We estimated the performance of a single-factor and a two-factor model

• The 6 aggregated variables seem to comprise a single latent variable, which we call “team climate”

Results - Items models• We estimated the

performance of models with 1, 2, 6 and 7 factors

• Some path weights (2 for reflex & 5 for work) are close to zero, hence they don’t contribute to the calculated factors.

• This is an indication that each of these two dimensions should be probably described by more than one factor.

• Fit was exceptional if error-term correlations were included (an indication that something is amiss)

Results - EFA (a)

• For each TCI dimension, Exploratory Factor Analysis was executed

• Both datasets agreed on the number of factors we needed to construct, for each dimension.

• The factor loadings verified that the datasets agreed not only on which questions “made up” which factors, but also on the amount that each question contributed to the total variance.

Results - EFA (b)

• Reflex’s factors seem to be discussion & consideration of methods and actual changes taking place.

• Work’s factors are likely to be team evaluation, personal evaluation in relation to the team and interdependence.

Dimension

# of factors

Part 1

Supinv 1

Obj 1

Task 1

Reflex 2

Work 3

Conclusions

• The Items models appear to perform better than the Scores ones, although there isn’t a solid measure available that can compare models that are that different.

• Exploratory factor analysis indicated that there may be more than one factor in certain dimensions. This finding was verified by the Items SE Models. Using a single aggregate variable for each of those dimensions is unsuitable.

Future work

• Data collected for 2005 with which the models will be validated

• Different model structure if information on respondent IDs becomes available

• Alternative Items’ models will be created, taking into account the results of EFA (3 and 2 latent variables for work and reflex respectively).

• Overall EFA performed (not for each subgroup separately)