Dissertation Defense Presentation - December 4, 2015 - JASheldon - Final Slides

EVALUATION AS SOCIAL INTERVENTION: AN EMPIRICAL STUDY OF EMPOWERMENT

EVALUATION PRACTICE AND PRINCIPLE EFFECTS ON PSYCHOLOGICAL EMPOWERMENT AND SELF –

DETERMINATION OUTCOMES

Jeffrey Andrew Sheldon

Claremont Graduate University2015

A dissertation submitted to the Faculty of Claremont Graduate University in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Psychology.

Overview• Conceptual Primer

•Concerns About Empowerment Evaluation from Evaluation Theorists and Commentators

•The Need for Research on Empowerment Evaluation

•Research Questions

•Methodology and Analytical Strategy

•Survey Design

•Sampling Rationale and Procedures

•Participant Characteristics

•Results and Take-Aways for Research Questions 1 - 3

•Results and Implications for Practice and Policy from Research Question 4 ($)

•Strengths and Limitations of this Study

•Recommendations for Future Research on Empowerment Evaluation

•ConclusionsEmpowerment Evaluation Practice, Principles, and Outcomes

2015

2

Conceptual Primer: Empowerment Theory

• A long-term process individuals, organizations, and communities undertake for themselves rather than something done to or for them (Lather, 1991, 1992; Claridge, 1996)

• Empowered outcomes are evidenced by individuals or aggregate bodies of individuals engaging in behaviors that permit effective pursuit of planned social change: working with others, managing resources, learning decision-making skills, influencing policy, accessing government, sharing leadership, obtaining needed resources, being networked to others, etc. (Schulz et al., 1995; Zimmerman, 2000).

• Three interrelated components (Zimmerman, Israel, Schulz, & Checkoway, 1992).

• Interactional: Developing knowledge, skills, and strategies to become leaders/advocates for change.

• Intrapersonal: Appraisal/critical awareness of, and the motivation and agency to change one’s life circumstances (i.e., self-efficacy).

• Behavioral: Individual-level actions requiring leadership that benefits individuals and communities by influencing social policy.

Empowerment Evaluation Practice, Principles, and Outcomes 2015

3

Conceptual Primer: Self-Determination Theory

• A general, organismic theory of human motivation organized around three sets of motivational processes—intrinsic, extrinsic, and amotivational —and their relationship to self-determined behaviors - free will and determination of one’s own fate or course of action without compulsion (Ryan & Connell, 1989).

• To be self-determined, experience well–being, and have optimal health (e.g., self-esteem, self-actualization, vitality, positive affect, and work engagement) people have to satisfy three basic psychological needs endowed by nature (Koestner, Ryan, Bernieri, & Holt, 1984):

• Competence: efficacious performance of behaviors based on positive feedback and communication.

• Autonomy: choice in how to perform a task is provided, the task is recognized as being important, and feelings about the task are acknowledged.

• Relatedness: meaningful connections to others and a sense of belonging; the opposite is alienation and inauthenticity.

• The collective research on SDT suggests that when basic psychological needs are supported, intrinsic motivation and integrated extrinsic motivation are most likely to be evident.Empowerment Evaluation Practice, Principles, and Outcomes

2015

4

Conceptual Primer: Social Intervention

• An emancipatory process that identifies and eradicates factors contributing to disempowerment and increases self-determination through a sense of connectedness to a larger social collective (Bennett, 1987; Lennie, 2005; Narayan, 2005, 2005).

• Committed to a vision of social justice (Vanderplaat, 1995).

• Fundamental goal: “ensure individuals and groups have the power to influence the direction of their lives and of their social institutions” on their own behalf (Bennett & Hallman, 1987, p. 93; Vanderplaat, 1995, 1998; Vanderplaat, Samson, & Raven, 2001).

• Characteristically participatory/a collective social activity that encourages person-to-person interaction and mutual support rather than person-to-program interaction through which a collective identity emerges that is communicatively secured and, in turn, makes possible the emergence of collective political agency (Vanderplaat, 1995).

• An evaluation serving as a social intervention is a communicative bridge between the collective needs and interests of a group and social policy formation (Vanderplaat, 1995).


5

Conceptual Primer: Empowerment Evaluation

• Derived from the ideological evaluation traditions, is by nature democratic, and facilitates examination of important issues in an open, communicative, and collective forum; the evaluator and evaluation participants learn from each other as equals.

• Comes under the umbrella of stakeholder-involved evaluation approaches (e.g., participatory, collaborative, democratic, and utilization-focused) and resonates with many in the evaluation community (Miller & Campbell, 2006).

• Has an overt political agenda of changing power differentials (Mertens, 1999, 2003; Smith, 2007).

• A process that facilitates the development of control over the fate of people joined by their relationship to a social program (Fetterman & Wandersman, 2005, 2007).

• Its intended outcome is increasing evaluation participants’ perceptions of their own empowerment and self-determination, less so assessing program merit or worth (Bennett, 1987; Smith, 1999).

• The means by which empowerment and self-determination are fostered, illumination generated, and liberation actualized is through the assessment of program merit or worth and subsequent plans for program improvement (Fetterman, 1995).

• People are able to help themselves as they build evaluation capacity and a sense of community referred to as communities of practice (Fetterman, 1996, 2001, 2002).

• Two purported advantages: self-sufficiency and liberation from traditional and preexisting expectations, roles, and constraints imposed by others – both of which speak to psychological well-being (Fetterman, 1994).


6

Conceptual Primer: Empowerment Evaluation Cont’d…• Distinguished from other stakeholder-involved evaluation approaches by 10 guiding principles derived from

Community Psychology and Participatory Action Research.

• Process (6): Community Ownership, Inclusiveness, Democratic Participation, Community Knowledge, Evidence –Based Strategies, and Accountability.

• Outcome (4): Improvement, Organizational Learning, Social Justice, and Capacity Building.

• From its inception, Fetterman (1994, 1996) has recommended specific steps in empowerment evaluation practice. Currently, there are three models (from earliest to most recent):

• Three-step – 1994

• Ten-step (a.k.a., Getting To Outcomes) - 2003

• Five –tool - 2010

• Each model has a significant degree of specificity, each incorporates Fetterman’s original steps, and each provides the basis for contemporary empowerment evaluation practice (Fetterman & Wandersman, 2007).


7

Concerns About Empowerment Evaluation

• Vague contingencies/prescriptives for actual practice (Cousins, 2005; Smith, 1999; Worthington, 1999).

• An absence of rigorous evidence to show it is empowering and leads to empowered outcomes (Patton, 2005; Worthington, 1999).

• Evidence of the effectiveness of empowerment evaluation is suggestive, not preponderant, and inconclusive (Smith, 2007).

• Outcomes are difficult to operationalize and assess, and even if measurable, they may be unrealistic for some smaller-scale or shorter-term projects (Campbell, Dorey, Naegeli, Grubstein et al., 2004; Patton, 1997).

• How to measure the success of an empowerment evaluation remains a challenge (Levin, 1996).

• Most practitioners have focused on whether the participating organizations can learn evaluation skills and successfully develop, launch, and utilize evaluations of their programs rather than on measuring empowerment and self – determination outcomes (Lennie, 2005).


8

The Need for Research on Empowerment Evaluation

• The empowering and self – determining outcomes of empowerment evaluation, has been widely acknowledged, although its study has often been superficial or ignored (Diaz – Puente, 2007; Diaz – Puente, Yague, & Alfonso, 2008; Diez 2001; Floc’hlay and Plottu 1998; Gregory 2000; Potter 2005; Saraceno 1999).

• Cousins (2005) noted that the quality of empowerment evaluation implementation resides in the extent to which persuasive evidence can be provided to examine the claims it makes. Further, the concept of empowerment evaluation “still remains a little bit fuzzy” which calls for stronger evidence of empowerment evaluation success from actual studies (p. 207).

• According to Miller (2005) a thorough account of how obtaining evaluation skills is empowering, to whom it is or ought to be empowering, and what role evaluators play in the process of shifting power is needed.

• Anderson (1996), Gore (1992), Lennie, (2002), and Lennie et al. (2003) have called for closer examination of research and evaluations that claim to have produced empowerment, and by extension, self-determination.

• Small number of studies (original research) on stake-holder involvement reflects a dearth of empirical research on evaluation, specifically empowerment evaluation (Brandon & Fukunaga, 2014)


9

Research Questions: 3 Descriptive, 1 Inferential


10

1. What was the extent to which evaluation practitioners adhered with fidelity to the three-step model, the 10-step model, or the five-tool model?

2. What was the extent to which the 6 process principles (i.e., community ownership, inclusion, democratic participation, community knowledge, evidence-based strategies, and accountability) were reported to be in evidence during the evaluations reported on by evaluation practitioners?

3. What was the extent to which the 4 outcome principles (i.e., improvement, organizational learning,

social justice, and capacity building) were reported to have resulted from the evaluations reported on by evaluation practitioners?

4. To what extent could variation in empowerment and self-determination be explained by:

a) The interaction between empowerment evaluation model fidelity and percentage of activities implemented?

b) The interaction between empowerment evaluation model fidelity and percentage of activities

implemented during the early, mid, and late phases of the empowerment evaluations?

c) The process principles reported to be in evidence during the empowerment evaluations?

d) The outcome principles that resulted from the empowerment evaluations?

e) Empowerment evaluator demographic, academic, and professional characteristics (e.g., gender, number of years conducting empowerment evaluations, highest earned degree, etc.)?

Methodology

• 92 item, self–report, on–line survey consisting of Likert – type and open–ended items.

• Item development based on:

— Constructs found in the literature regarding the three known empowerment evaluation models and their respective implementation steps and activities.

— The 10 principles of empowerment evaluation as outlined in the literature.

— The purported empowerment and self – determination outcomes for individuals and organizations engaged in the process of an empowerment evaluation discussed in the literature.

— Constructs found in the social psychology literature regarding empowerment theory and self–determination theory.

• Presented an early version to a graduate-level survey research methods course to deal with design flaws, problems with wording and comprehension, issues in cognitive overload, and potential issues in internal consistency through elimination of similar items.

• Worked with a university–based survey research expert on culling out items, refining item tags and response choices, overall survey design, and on-line beta testing.

• Two university-based experts in empowerment evaluation each performed an independent content analysis.


11

Survey Design


12

Section

Items

Scale(s) Developed

Research Question(s)

Your evaluation activities

29

Empowerment Evaluation Models

RQ 1, RQ 4

Process Principles RQ 2, RQ 4

Evaluation participant activities 12 Process Principles RQ 2, RQ 4

Changes you observed in individual values 5 Process Principles RQ 2, RQ 4

Outcome Principles RQ 3, RQ 4 Changes you observed in individual behaviors

16

Outcome Principles

RQ 3, RQ 4

Individual Empowerment RQ 4

Individual Self-Determination RQ 4 Changes you observed within the organization

14

Outcome Principles

RQ 3, RQ 4

Organizational Empowerment and Self-Determination

RQ 4

Inclusiveness 1 Process Principles RQ 2, RQ 4

Evaluator accountability 2 Process Principles RQ 2, RQ 4

The evaluation model you used 1 -- -- About you: demographic information

12

--

RQ 4

Analytical StrategyUnit of Analysis: The evaluation practitioner – creates the evaluation context conducive to development of empowerment and self-determination and adheres to principles.

Descriptive statistics used to determine:

• The extent to which evaluation practitioners adhered with fidelity to the steps of the empowerment evaluation model they implemented.

• The extent to which the process principles were reported to be in evidence during the evaluations reported on by evaluation practitioners.

• The extent to which the outcome principles were reported to have resulted from the evaluations reported on by evaluation practitioners.

Multiple regression techniques used to determine the extent to which variation in the four empowerment sub-constructs, the individual empowerment construct, the three self – determination sub-constructs, the individual self-determination construct, and the organizational empowerment and self – determination construct could be explained by:

• The interaction between empowerment evaluation model fidelity and percentage of activities implemented.

• The interaction between empowerment evaluation model fidelity and percentage of activities implemented during the early, mid, and late phases of an empowerment evaluation.

• The process principles reported to be in evidence during the empowerment evaluations.

• The outcome principles that resulted from the empowerment evaluations.

• Empowerment evaluator demographic, academic, and professional characteristics (e.g., gender, number of years conducting empowerment evaluations, highest earned degree, etc.) selected through a correlational analysis.


13

Sampling Rationale & Procedure

• Non – probabilistic (i.e. self – selected), purposive, and not dependent on the rationale of probability theory.

• Criterion for inclusion: evaluation practitioner had to have conducted an empowerment or empowering evaluation in the past five years.

• Building the Sample:

• Initial sample built from the American Evaluation Association (AEA) Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group membership (with help from the moderator), extant evaluation literature, and snowball sampling (n = 232).

• Researcher’s professional networks, global evaluation networks, global evaluation associations and

societies, AEA eLibrary, AEA365 blogs, and evaluation journals used to further identify potential study participants.

• Names and contact information obtained through networks, etc. were cross-checked against the initial sample to update contact information and eliminate duplicates.

• Upon reconciliation of names and contact information the sample included 615 evaluation practitioners.


14

Recruitment


15

Groups Potential Number of Recipients

3 AEA eGroups (i.e., CPE, ICCE, OL-ECB) 1,416

4 Listserves (e.g., EvalTalk, SCRA, CommunityPsychUk, etc.) 7,800+

19 LinkedIn Groups (e.g., SAMEA, Evaluators Group, etc.) 83,531

5 Yahoo Groups (e.g., MandENews, XCeval, AfRea, etc.) 5,850

39 Facebook Communities/Interest Groups (e.g., InDEC, M & E, etc.) 57,810

Total* 156,407

*Note: The total is likely duplicated head-count as people often belong to multiple professional groups.

Data Collection and Response RateData Collection

• Dillman’s (2000) four-step process guided implementation of the survey.

• Study officially launched one week after the recruitment phase ended by directly contacting potential study participants through email, LinkedIn (In Mail), and Facebook messaging, and diffusely contacting potential study participants via groups used in recruitment with the survey link resulting in 512 total respondents.

• From study launch to first reminder 112 respondents.

• From first to second reminder 140 additional respondents.

• From second to final reminder, 107 additional respondents.

• From final reminder to close of survey, 162 additional respondents.

Response Rate

• Difficult to calculate because both direct and diffuse recruitment methods were used.

• Response rate considered to be 25% as 131/521 provided usable data.


16

Participant Characteristics99 of 131 evaluation practitioners (76%) reported demographic, academic preparation, and professional characteristics:

• 58.3% English speaking.

• 59.8 % White, Non – Hispanic.

• 67 % Female.

• An average of 15 years in the field of evaluation.

• An average of 11 years conducting empowerment evaluations.

• 52% lived in a North American country (i.e., United States, Canada, or Mexico).

• 51.1% conducted empowerment evaluations in a North American country.

• 20% conducted empowerment evaluations in an African country.

• 48.5% held a Ph. D.

• 38.4% held a Master’s Degree.

• Academic degree fields: 46.8%s social sciences, 27.4% monitoring and evaluation, 17.9 % research, 12.8% public health, and 12.6% other academic (e.g., international development).

• Professional content area of expertise: 21.1% monitoring, evaluation, and research, 12.6 % respectively development and social justice, and 11.6 % respectively action and community – based research, health, and social science.


17

Results: Research Question 1 – Model Fidelity


18

Fidelity to the Three-Step Model Frequency

Valid Percent

No Fidelity

10

7.8

Fidelity 119 92.2

Total 129 100.0

Fidelity to the Ten-Step Model

Frequency

Valid Percent

No Fidelity

12

9.4

Fidelity 115

90.6

Total 127 100.0

Fidelity to the Five-Tool Model Frequency

Valid Percent

No Fidelity

26

20.3

Fidelity 102 79.7

Total 128 100.0

Results: Research Question 2 – Process Principles in Evidence


19

Community Ownership Process Principle Frequency

Valid Percent

No Evidence

8

6.3

Evidence 120 93.8

Total 128 100.0

Inclusiveness Process Principle Frequency

Valid Percent

No Evidence 25 25.8 Evidence 72 74.2 Total 97 100.0

Democratic Participation Process Principle Frequency

Valid Percent

No Evidence

20

17.5

Evidence 94 82.5

Total 114 100.0

Results: Research Question 2 Cont’d…


20

Community Knowledge Process Principle

Frequency Valid Percent

No Evidence

24

20.7

Evidence

92

79.3

Total 116 100.0

Evidence-Based Strategies Process Principle

Frequency

Valid Percent

No Evidence

25

25.8

Evidence

72

74.2

Total

97

100.0

Accountability Process Principle

Frequency

Valid Percent

No Evidence

28

29.8

Evidence 66 70.2

Total 94 100.0

Results: Research Question 3 – Outcome Principles in Evidence


21

Improvement Outcome Principle

Frequency

Valid Percent

No Evidence

10

9.7

Evidence 95 90.5

Total 105 100.0

Organizational learning Outcome Principle

Frequency

Valid Percent

No Evidence

14

14.1

Evidence 85 85.9

Total 99 100.0

Results: Research Question 3 Cont’d…


22

Social Justice Outcome Principle

Frequency

Valid Percent

No Evidence

13

13.8

Evidence 81

86.2

Total 94 100.0

Capacity Building Outcome Principle

Frequency

Valid Percent

No Evidence

41

42.3

Evidence 56 57.7

Total 97 100.0

Research Questions 1 – 3 Take-AwaysResearch Question 1

• Based on the literature, I thought that the majority of evaluation practitioners would not adhere with fidelity to the steps of the empowerment evaluation model they implemented. However, the average fidelity score index revealed a majority of the study participants adhered with fidelity to the three-step, 10-step, and five-tool models.

Research Question 2

• I thought that a majority of evaluation practitioners would report the readily observable process principles of community ownership, inclusion, and democratic participation were evident to them, but the community knowledge, evidence-based strategies, and accountability process principles were not.

• The average evidence score index indicated evidence of the community ownership, inclusion, and democratic participation process principles. However, the average evidence score index also indicated evidence of the community knowledge, evidence-based strategies, and accountability process principle.

Research Question 3

• I thought that a majority of evaluation practitioners would report the outcome principles of improvement and capacity building were evident to them, but the more distal outcomes of organizational learning and social justice outcome principles were not.

• The average evidence score index indicated evidence of the improvement and capacity building principles. However, the

average evidence score index also indicated evidence of the organizational learning and social justice principles.


23

Results: Research Question 4 – Reject or Fail to Reject H0


24

Research Question 4.1: To what extent could variation in empowerment and self-determination be explained by the interaction between empowerment evaluation model fidelity and percentage of activities implemented? H0 = The interaction between empowerment evaluation model fidelity and percentage of activities implemented would not be able to explain variation in the sub-constructs and constructs of individual empowerment and self – determination, and organizational empowerment and self – determination. In nine of 10 models I failed to reject the null hypothesis, but I rejected the null hypothesis in one of 10 models because the interaction between model fidelity and percentage of activities implemented was able to explain variation in evaluation capacity; notably it was model fidelity that explained most of the variation. Interaction: R Square = .081, F = 4.158, p = .019

Model Fidelity: Beta = .251, t = 2.493, p = .014

Results: Research Question 4 – Reject or Fail to Reject H0


25

Research Question 4.2: To what extent could variation in empowerment and self-determination be explained by the interaction between empowerment evaluation model fidelity and percentage of activities implemented during the early, mid, or late phases of an empowerment evaluation?

H0 = The interaction between empowerment evaluation model fidelity and percentage of activities implemented would not be able to explain variation in the sub-constructs and constructs of individual empowerment and self – determination, and organizational empowerment and self – determination during the early, mid, and late phases of an empowerment evaluation.

In nine of 10 models I failed to reject the null hypothesis, but I rejected the null hypothesis in one of 10 models because the interaction between early evaluation model fidelity and percentage of activities implemented, and mid-evaluation model fidelity and percentage of activities implemented was able to explain variation in evaluation capacity; notably it was model fidelity during the mid-evaluation phase that explained most of the variation.

Interaction: R Square = .097, F = 2.484, p = .049

Mid-evaluation Model Fidelity: Beta = .308, t = 2.076, p = .041

Results: Research Question 4 - Reject or Fail to Reject H0 Cont’d…

Research Question 4.3: To what extent could variation in empowerment and self-determination be explained by the process principles reported to be in evidence during the evaluations reported on by evaluation practitioners? H0 = The community ownership, inclusiveness, and democratic participation process principles would not be able to explain variation in the sub-constructs and constructs of individual empowerment and self – determination, and organizational empowerment and self – determination. In six of 10 models I failed to reject the null hypothesis because community ownership and democratic participation were unable to explain variation, but I was able to reject the null hypothesis in four of 10 models because the inclusiveness process principle was able to explain variation in evaluation knowledge, evaluation capacity, individual empowerment, and evaluation competence.

Evaluation knowledge - Inclusiveness: R Square = .054, F = 4.068, p = .047; Beta = -.233, t = -2.017, p = .047

Evaluation Capacity - Inclusiveness: R Square = .082, F = 6.310, p = .014; Beta = -.286., t = -2.512, p = .014

Individual Empowerment - Inclusiveness: R Square = .057, F = 4.317, p = .041; Beta = -.238, t = -2.078, p = .041

Evaluation Competence - Inclusiveness: R Square = .059, F = 4.552, p = .036; Beta = -.244, t = -2.134, p = .036


26

Results: Research Question 4 - Reject or Fail to Reject H0 Cont’d…

Research Question 4.4: To what extent could variation in empowerment and self-determination be explained by the outcome principles that resulted from the evaluations reported on by evaluation practitioners? H0 = The improvement and capacity building outcome principles would not be able to explain variation in the sub-constructs and constructs of individual empowerment and self – determination, and organizational empowerment and self – determination. In all 10 models I failed to reject the null hypothesis because evidence of the improvement and capacity building outcome principles were unable to explain variation.


27

Results: Research Question 4 - Reject or Fail to Reject H0 Cont’d…Research Question 4.5: To what extent could variation in empowerment and self-determination be explained by empowerment evaluator demographic, academic, and professional characteristics? H0 = The characteristics gender, professional expertise, and where empowerment evaluations were conducted would be unable to explain variation in the sub-constructs and constructs of individual empowerment and self – determination, and organizational empowerment and self – determination.

In nine of 10 models I failed to reject the null hypothesis because gender was unable to explain variation, but I was able to reject the null hypothesis in one of 10 models because gender was able to explain variation in individual empowerment. Individual empowerment - Gender: Beta = -.293, t = -2.331, p = .024 In all 10 models I failed to reject the null hypothesis because professional expertise was unable to explain variation. In five of 10 models I failed to reject the null hypothesis because where empowerment evaluations were conducted was unable to explain variation, but in the other five models I was able to reject the null hypothesis because where empowerment evaluations were conducted was able to explain variation in evaluation knowledge, individual empowerment, evaluation competence, evaluation autonomy, and feelings of relatedness to others. Evaluation Knowledge - Work in an African Country: Beta = .627, t = 2.090, p = .041Individual Empowerment - Work in an African Country: Beta = 699, t = 2.343, p = .023 Evaluation Competence - Work in an African Country: Beta = .715, t = 2.407, p = .019 Evaluation Autonomy - Work in an African Country: Beta = .630, t = 2.121, p = .038Feelings of Relatedness to Others - Work in a Non-African Country: Beta = -.361, t = -2.895, p = .006; - Work in an African Country: Beta =.645, t = 2.306, p = .025


28

Results: Research Question 4 - Reject or Fail to Reject H0 Cont’d…Although not tested against the null hypothesis, variation in evaluation knowledge, individual empowerment, evaluation competence, and evaluation autonomy was explained by where evaluation practitioners lived, specifically an African country.

• Evaluation Knowledge - Live in an African Country: Beta = -.675, t = -2.395, p = .020

• Individual Empowerment - Live in an African Country: Beta = -.787, t = -2.850, p = .006

• Evaluation Competence - Live in an African Country: Beta = -.634, t = -2.270, p = .027

• Evaluation Autonomy - Live in an African Country: Beta = -.684, t = -2.454, p = .017

Variance accounted for with all 10 demographic, academic, and professional characteristics in the model:

• Evaluation Knowledge: R Square = .283, F = 2.214, p = .030

• Individual Empowerment: R Square = .348, F = 2.829, p = .007

• Evaluation Competence: R Square = .297, F = 2.369, p = .020

• Evaluation Autonomy: R Square = .299, F = 2.390, p = .019

• Feelings of Relatedness to Others: R Square = .385, F = 3.188, p = .003

Empowerment Evaluation Practice, Principles, and Outcomes

2015

29

Research Question 4 - Implications for Evaluation Practice and PolicyIf building evaluation capacity is important:

• Implement a high percentage of activities to ensure model fidelity especially during the mid-phase of an empowerment evaluation.

• Attend to the inclusiveness process principle to ensure everyone who wants to be fully engaged in the evaluation is included.

If building evaluation knowledge is important: • Attend to the inclusiveness process principle to ensure everyone who wants to be fully engaged in the evaluation is included. • Attend to the community knowledge process principle so that everyone who is engaged in the evaluation can use their collective

wisdom to develop evaluation tools, evaluation procedures, interpreting data, etc. If building evaluation competence is important: • Attend to the inclusiveness process principle to ensure everyone who wants to be fully engaged in the evaluation is included. If individual empowerment is important: • Attend to the inclusiveness process principle to ensure everyone who wants to be fully engaged in the evaluation is included.


30

Strengths of this Study: Addressing Concerns

• Determining which steps/activities of the three empowerment evaluation models have been applied in actual evaluation practice.

• Operationalizing the steps of the three empowerment evaluation models as evaluation practitioner and evaluation participant activities.

• Defining empowerment evaluation model fidelity, developing a model fidelity index, and measuring model fidelity.

• Operationalizing the empowerment evaluation process and outcome principles as evaluation practitioner and evaluation participant activities, developing an index for determining evidence of the principles, and determining whether the principles were in evidence during and as a result of the empowerment evaluations reported on by evaluation practitioners.

• Defining the sub-constructs and constructs of empowerment theory and self-determination theory for and adapting them to empowerment evaluation.

• Explaining variation in empowerment and self-determination by empowerment evaluation model fidelity, evidence of the empowerment evaluation process and outcome principles, and evaluation practitioners’ demographic, academic, and professional characteristics.


31

Limitations Most Impacting this Study

• Self - Selection

• Calls into question the extent to which the evaluation practitioners in the sample were equivalent to other evaluation practitioners who conduct empowerment evaluations.

• Social Desirability

• Calls into question the accuracy of responses (see for example, model fidelity scores).

• Study participants claim and over-report activities which are socially desirable (Krumpal, 2013).

• Self-deception: A respondent believes a statement to be true of herself even though it is inaccurate.

• SEEPPO Reliability

• Although the SEEPPO appeared to have strong content validity there were not enough observations to conduct an adequate reliability analysis.

• Further refinement and testing, including a factor analysis, to establish its reliability and predictive validity is necessary.


32

Recommendations for Research on Empowerment Evaluation • Develop and validate, or adapt existing empowerment and self-determination scales so evaluation practitioners can measure the empowerment

and self-determination of individuals and organizations with whom they work; instruments should be able to discriminate between individual empowerment and self-determination sub-constructs and organizational empowerment and self-determination sub-constructs.

• Measure the empowerment and self-determination of empowerment evaluation participants before, during, and post-evaluation as well as the distal outcomes of empowerment evaluation (e.g., the extent to which social policy was influenced towards favorable outcomes, the extent to which power differentials were equalized, etc.)

• Create and test a theory of change model, including moderators and mediators, showing the likely pathways from empowerment evaluation model implementation and the process and outcome principles to individual and organizational empowerment and self-determination.

• Test whether there is a relationship between models and principles, the extent to which a relationship exists, and whether model fidelity (along a continuum) or evidence of the principles (along a continuum) attenuates that relationship.

• Explore the role of contextual variables in the development/enhancement of empowerment and self-determination by comparing the outcomes of empowerment evaluations conducted in different countries.


33

Conclusions• To a limited extent individual empowerment and self-determination were the likely outcomes of the

empowerment evaluations reported on by this sample of 131 evaluation practitioners; model fidelity, the inclusiveness and community knowledge process principles, and evaluation practitioner characteristics appeared to have been able to explain variation.

• Given evidence of all 10 guiding principles it can be said that the evaluations reported on were likely to be empowerment evaluations as defined by Fetterman and Wandersman.

• The evaluation practitioners in this study largely adhered with fidelity to the steps of the models they implemented; no model was privileged over another as steps were chosen from across all three models (96.2%, n = 126).

• Evaluation practitioner’s personal, academic, and professional characteristics did not fully factor into the empowerment of individuals and organizations with whom they worked. However, evaluation knowledge, individual empowerment and all three individual self-determination sub-constructs (i.e., competence, autonomy, and relatedness) appeared to have been correlated (both + and -) with empowerment evaluations conducted by evaluation practitioners who lived and worked in an African country, who were female, and who conducted empowerment evaluations in another country.

• The SEEPPO was a successful first attempt at providing a practical, largely quantitative tool to empowerment evaluators interested in assessing the process and outcomes of their work, less so actual empowerment and self-determination.


34

Dissertation Defense Presentation - December 4, 2015 - JASheldon - Final Slides

Documents

Transcript of Dissertation Defense Presentation - December 4, 2015 - JASheldon - Final Slides