Nonparticipant to Participant a Methodological Perspective on Evaluator Ethics

http://aje.sagepub.comAmerican Journal of Evaluation

DOI: 10.1177/1098214005284979 2006; 27; 98 American Journal of Evaluation

Scott R. Rosas Nonparticipant to Participant: A Methodological Perspective on Evaluator Ethics

http://aje.sagepub.com The online version of this article can be found at:

Published by:

http://www.sagepublications.com

On behalf of: American Evaluation Association

can be found at:American Journal of Evaluation Additional services and information for

http://aje.sagepub.com/cgi/alerts Email Alerts:

http://aje.sagepub.com/subscriptions Subscriptions:

http://www.sagepub.com/journalsReprints.navReprints:

http://www.sagepub.com/journalsPermissions.navPermissions:

http://aje.sagepub.com/cgi/content/refs/27/1/98 Citations

by kimbao bao on April 13, 2009 http://aje.sagepub.comDownloaded from

10.1177/1098214005284979American Journal of Evaluation / March 2006Rosas / Nonparticipant to Participant

Nonparticipant to ParticipantA Methodological Perspective on Evaluator Ethics

Scott R. RosasNemours Health and Prevention Services

Evaluators of human services initiatives are often confronted with the prospect of alteringtheir methodological choices during the course of an evaluation. Usually, this needemerges from circumstances that challenge the ability to carry out the evaluation, such as accessto respondents, the attrition of participants, the availability of resources, and constraints ontime. The present scenario provides an opportunity to examine potential ethical challengesassociated with a shift from one methodological approach to another, as the evaluator adjuststhe extent of his participation in the program. At the core of this examination is a realization ofhow the change in method, from nonparticipant observer to participant observer, affects theevaluation and what set of standards and principles guides the evaluators response as he pro-ceeds. It is clear that this scenario exposes a set of thorny ethical issues for the role of the evalua-tor in relation to the program, particularly those issues associated with power, control, andinfluence. This response, however, addresses the less obvious considerations of the princi-ples that guide an evaluators conduct in the event of a decision to alter the methodologicalapproach.

Smiths (2002) eloquent meta-analysis of past ethical challenges responses in the AmericanJournal of Evaluation suggested that there has been some question as to what represents an eth-ical predicament in evaluation and that methodological problems do not necessary constitutemoral problems. However, Morris (1998) argued that issues of methodology and design canhave significant ethical implications for an evaluators work (p. 381). Taking this perspective,it is not clear what ethical conflicts, if any, face the evaluator in the current case as he pondersthe methodological path on which to travel. Within this particular context, the dilemma to stepoff of the current path is one of method; the dilemma of what happens after stepping off the pathis one of ethics. At issue here, as Morris highlighted, is how the decision to alter the method hasethical implications for the evaluators work and ultimately affects the quality of the evaluation.

On the surface, at least, it appears that the evaluators decision to fully engage as a participantin the program was not driven by methodological considerations. Instead, the shift in methodwas the result of an interest by both the client and the evaluator in having the evaluator assume amore active, participatory role in the program. Regardless of the means by which the changeoccurred, as the evaluator proceeds with the evaluation in the participant observer role, impor-tant implications for the evaluation arise stemming from the shift in method. Rather than argu-

Scott R. Rosas, Center for Childrens Health Innovation, Nemours Health and Prevention Services, ChristianaBuilding, 252 Chapman Road, Suite 200, Newark, DE 19701; phone: (302) 444-9131; fax: (302) 444-9200; e-mail:[email protected].

American Journal of Evaluation, Vol. 27 No. 1, March 2006 98-103DOI: 10.1177/1098214005284979 2006 American Evaluation Association

98


ing for or against the merits of the evaluators decision, this examination of methodologicalconsiderations that emerge in this scenario focuses on the ethical next steps for the evaluator. Inthis commentary, I discuss the ethical tensions that are created by the methodological switch,including issues associated with the relationship between the evaluator and the client, the ex-pectations for the evaluation on the part of the participants, and the methods used to establishthat the evaluation findings are credible.

Key Method Considerations

The change in method from nonparticipant observer to participant observer can be subtle,especially to stakeholders who are not evaluators. Although the two methodological strategiesare distinguished by their assumptions about the position of an evaluator and his or her relation-ship to the object of interest, the stakeholders in this scenario might have difficulty recognizingthe potential impacts of the shift upon the evaluation. It is critical for an evaluator to understandthe distinctions, because a failure to identify, communicate, and account for differences couldlead others to view the evaluator as incompetent, inexperienced, unethical, or, worse, a com-bination of all three (Smith, 2002). This commentary on the To Participate or Not to Partici-pate scenario discusses the ethical tensions that are created by the methodological switch,including issues associated with the relationship between the evaluator and the client, theexpectations for the evaluation on the part of the participants, and the methods used to establishthat the evaluation findings are credible.

The degree of distance, both in terms of the evaluators relationship to the object of interestand the perception of stakeholders regarding that relationship, is a key consideration in themethodological shift. Consistent with a nonparticipant observation approach, the evaluator inthis scenario initially sought to remain as unobtrusive as possible as he recorded the commit-tees behavior and activities. In situations in which the evaluator interacted with the client andstakeholders in the expected role as evaluator (i.e., observations, in-depth interviews), hisbehavior and actions communicated his intent to maintain a nonparticipatory position andestablished a set of tacit roles for both the evaluator and stakeholders. Whether the expectationsfor evaluator and stakeholder behavior in relation to the method were implicit or explicit mat-ters little. The evaluator needs to be cognizant of the expected behaviors and roles individualsassume in data gathering based on the particular method being used, as well as how changesalter those expectations.

In this case, the evaluator established an approach whereby data collection through nonpar-ticipant observations was separated from data collection through interviewing. Stakeholderswere engaged in a data-gathering process, functioning in roles consistent with the nonpartici-pant method. The shift from nonparticipant observer to participant observer changes the datacollection process, in which informal interviewing and observation of behavior are occurringjointly, in a dynamically recursive process. Consequently, stakeholders may become confusedas they seek to understand their revised role and what information is considered in bounds orout of bounds in the context of the evaluation. The movement of the evaluator from nonpartic-ipant to participant observer is likely to raise concerns, voiced or not, about how informationshared previously in confidence will be used over the course of the evaluation. Given the con-tentious nature of the interactions between the committee members, the evaluators knowledgeof individual stakeholder perceptions of the program could easily place him in a politicallyprecarious situation, of which he should be mindful.

Commensurate with a specific observational approach, the evaluator assumes a particularmeaning-making role relevant to that social context. Evaluators conducting nonparticipantobservations do so with a detached perspective whereby the emphasis is on watching rather

Rosas / Nonparticipant to Participant 99


than taking part in the social milieu. This removed point of view enables evaluators to focus onspecific situations and occurrences rather than attempting to immerse themselves in the entirecontext. Inferences as to what is occurring are based on the observable behavior, with less atten-tion to intent, attitudes, emotions, or thoughts. The shift in method in this case emphasizes theneed for the evaluator to attend to the subtle changes in the axiological and ontological assump-tions that underpin both nonparticipant and participant observation methods.

In an attempt to reconstruct the realities of informants in qualitative inquiry, the evaluatornegotiates meanings and interpretations with human data sources (Lincoln & Guba, 1985).This reconstruction and negotiation process is operationalized differently for the two methodsas a function of their axiological orientations. As a nonparticipant observer, the evaluator filtersand interprets reality as he witnesses actions and behaviors. Alternatively, in a participantobservation approach, reality is coconstructed with others, as the evaluator becomes an ac-cepted part of the context under study. Thus, as a participant, the evaluators behavior, attitudes,and emotions play a more prominent role in shaping the understanding of actions and eventsincluded in the evaluation. Although the degree to which the actions of the evaluator influencebehavior in the setting is likely to wane over time, initial reactivity is a consideration. As a par-ticipant observer, the evaluator needs to widen his perspective to more explicitly monitor howmuch and in what ways he is influencing the phenomenon he is observing. A failure of the eval-uator to shift perspective and focus could lead to an inappropriate analysis of the data collectedin the evaluation through the introduction of unaccounted-for and unchecked bias.

Finally, the recognition that a nonparticipant observational approach does not take as long oris as labor intensive as a participant observer approach highlights investment considerationsrelated to the method of choice. Because the evaluator needs to be accepted as a part of the con-text under study to ensure that the observations are of the natural phenomenon, extensive workover an extended period of time is required. Thus, a corresponding increase in time, energy, andcommitment would likely need to occur. The increase in investment not only has workloadimplications for the evaluator but also puts the client in a potentially compromising position ofneeding to allocate additional resources to meet the demands of the approach. The inability ofthe client to meet the potential increase in the resource burden could affect the quality of theevaluation by limiting the ability to collect the necessary observational data.

Changes in the methodological approach will undoubtedly affect the originally negotiatedexpectations held by the client and evaluator. As Patton (1986) suggested, for those adoptingqualitative evaluation strategies, the ideal is to negotiate and adopt the level of participation thatwill produce the most meaningful data about the program given the participant characteristics,the nature of interactions and relationships, and the overall context of the program. Assumingthat the evaluator clearly articulated the specifics of the approach, as well as what is likely toemerge from such an approach, in the original negotiation, a change in method would affect theoriginal agreement, even if the clients and stakeholders perceive the change to be minor. More-over, a failure to inform the client of the new requirements brought about by the methodologicalshift could result in the contribution to the belief on the part of the client that the resources,scope, and expectations of the new methodological approach remain unchanged. A potentialconflict could arise, because the demands of a participant observation method might be wellbeyond the ability (or interest) of the client to support. Thus, the evaluations time frame andcommitment originally agreed on would need to be renegotiated.

What Can Be Done?

Rossi (1995) argued that the Guiding Principles for Evaluators (American Evaluation Asso-ciation, Task Force on Guiding Principles for Evaluators, 1995) are purposefully vague so as to

100 American Journal of Evaluation / March 2006


be relevant to a broad range of evaluation practitioners. Similarly, House (1995) indicated thatthe Guiding Principles for Evaluators provide a starting point for discussion of how evaluatorsmight seek to conform to accepted professional standards of conduct rather than a definitivestatement of practice. Because of the wide range of methodological approaches used by evalua-tors, neither the Guiding Principles for Evaluators nor The Program Evaluation Standards(Joint Committee on Standards for Educational Evaluation, 1994) offers much guidance whendiscussing the ethical issues related to methodological choices. They are, however, useful as abackdrop in the following discussion of how the evaluator in the scenario could respondethically to the shift from nonparticipant to participant observation.

Apply Principles and Standards

Both the guiding principle on systematic inquiry (American Evaluation Association, TaskForce on Guiding Principles for Evaluators, 1995) and the Joint Committee on Standards forEducational Evaluations (1994) accuracy standard emphasize the need to maintain high tech-nical standards and underscore the importance of the awareness of the methodological differ-ences relevant to an evaluation. Similarly, Chelimsky (1995) recognized that attempts to dis-credit evaluation findings have often focused on methods, evaluator skills, and objectivity withrespect to the object under study. Thus, as the evaluator moves forward with the evaluation in aparticipant observer role, he should carefully document and describe the details of the shift inmethod, making transparent his understanding of the issues related to the methodological shift,as well as demonstrating his understanding of the methodological technique. The evaluatorshould explain, in ways that are technically and contextually sensitive, the shortcomings andstrengths of the new methodological choice, including how the new set of assumptions willaffect the type of data collected, the interpretation of the data collected, and the conclusions thatare drawn from the data. These steps should help the evaluator avoid misrepresenting the proce-dures, data, or findings and help ensure that results emerge from sound methodological prac-tice. It is critical that the evaluator recognize that the responsibility to discuss and clarify thesematters rests solely with him, not with the client. The inability of the evaluator to fully explorethis issue could contribute to a misalignment in the method-expectation match and affect thequality of the evaluation.

The competence principle states that evaluators should possess the education, abilities, andexperience appropriate to undertake the tasks proposed in the evaluation and practice withinthe limits of their professional training and competence (American Evaluation Association,Task Force on Guiding Principles for Evaluators, 1995, p. 22). Although Rossi (1995) pointedout the lack of clarity as to what is competence in evaluation, critical reflection by the evaluatoris necessary in the event of a methodological shift. If the evaluator (or the client for that matter)has concerns as to whether he is competent in participant observation, he should seek to gainthe competence directly through the assistance of others who possess the required expertise(American Evaluation Association, Task Force on Guiding Principles for Evaluators, 1995,p. 23). The evaluator in this scenario could engage a colleague with experience in participantobservations to provide guidance and coaching through the course of the evaluation. Establish-ing an appropriate level of technical support could provide an ideal context for the evaluator toimprove his competencies to provide the highest level of performance in this and future eval-uations. Competence is also necessary to establish evaluator credibility (Joint Committee onStandards for Educational Evaluation Standard U2), which can facilitate the acceptance of theevaluation findings by program stakeholders.

In keeping with the principles of integrity and honesty, the evaluator should renegotiate withclients and stakeholders relevant points associated with the methodological shift. A new agree-



ment needs to be reached as to the costs, tasks, limitations, and expectations about the likelyresults. Moreover, the evaluator should take care to record all changes made in the originallynegotiated project plans and the reasons why the changes were made (American EvaluationAssociation, Task Force on Guiding Principles for Evaluators, 1995, p. 23). Although the com-mittee insisted on the evaluators participation, clearly, the evaluator already had an interest inthe program beyond that of the evaluation. It is important that the evaluator be explicit abouthow his investment in and commitment to the program, and consequently his interest in theevaluation, changes with the methodological shift. This attention is necessary to protect againstdistortions caused by personal feelings and biases (Joint Committee on Standards for Educa-tional Evaluation Standard A11).

The guiding principle of respect for people admonishes evaluators to respect the secu-rity, dignity, and self-worth of respondents, program participants, clients, and other evaluationstakeholders (American Evaluation Association, Task Force on Guiding Principles for Evalu-ators, 1995, p. 24). To adhere to this principle, the evaluator in this scenario will likely need toreiterate his intent to maintain the confidentiality of the interview data in the context of themethodological shift. Emphasis on respecting and protecting the rights and welfare of subjectsin the evaluation (Joint Committee on Standards for Educational Evaluation Standard P3) com-municates to clients and stakeholders the sensitivity of the evaluator to the information man-agement issues affected by the shift in method.

Strengthen the Approach

Ultimately, in qualitative inquiry, an evaluator seeks believability on the basis of coherence,insight, instrumental utility (Eisner, 1991), and trustworthiness (Guba & Lincoln, 1988). Oneof the ways in which the evaluator in this scenario could help minimize ethical problems associ-ated with a shift in method is to emphasize quality as outlined for qualitative inquiry (Guba &Lincoln, 1988). Thus, in the pursuit of quality, the evaluator might protect against distortionsand biases based on his newly emerging presence through prolonged engagement in the contextbeing studied. He could use techniques such as triangulation, peer debriefing, and memberchecks to enhance the likelihood that the findings are an accurate reflection of the experiencesof the participants and the context of the inquiry. Furthermore, the evaluator could establishdependability and confirmability audit trails to record key evaluation processes and ensure thatconclusions, interpretations, and recommendations are grounded in defensible data. When re-sources dictate, the evaluator in this scenario might suggest an audit or metaevaluation to assessthe quality of the evaluation; provide guidance for improving methods and approaches; andmaintain credibility with clients, stakeholders, and other evaluators (House, 1987; Stufflebeam,2001). Although rigorously applied methods do not resolve what House (1995) saw as themethodological fallacy of competently executed methods alone solving ethical problems, theycan help the evaluator maintain high technical standards, demonstrate competence in theapproach, and manage objectivity throughout. Using specific strategies in support of the guid-ing principles and standards, the evaluator could not only maintain a high level of ethicalbehavior but also exhibit transparency in the methodological approach.

Anticipate

Although the next steps suggested above are in response to the shift in method borne out of adecision to enter the evaluation as a participant observer, it might also be useful to consider howevaluators could be proactive about such situations. Given the frequency that shifts between

102 American Journal of Evaluation / March 2006


nonparticipant and participant observer roles might actually occur in evaluation practice, itmight behoove evaluators in their negotiations to highlight at the outset the potential tensionbetween the two methodological approaches. Anticipating the possibility of such a shift, theimplications for the evaluation could be discussed. For example, the evaluator might use a caseexample of a situation similar to this to illustrate the possible issues associated with a shift inmethod. It is within this context that the strengths and limitations of the two methods, antici-pated roles and expectations shaped by each method, and the potential effects of a change inmethod on the interpretation of the evaluative findings could be explored. Extending this sug-gestion further, these discussions might need to include the limits of participation consistentwith the method of choice and specific ways the evaluator will seek to maintain those limitsthroughout the course of the evaluation. All things considered, evaluators need to be awarewhen shifts in methods are in conflict with originally negotiated limits, in particular when theypotentially alter expectations of clients and stakeholders and the conclusions that may bereached as a part of the evaluation process.

References

American Evaluation Association, Task Force on Guiding Principles for Evaluators. (1995). Guiding principle for eval-uators. In W. R. Shadish, D. L. Newman, M. A. Scheirer, & C. Wye (Eds.), Guiding principles for evaluators (Newdirections for program evaluation, Vol. 66, pp. 19-26). San Francisco: Jossey-Bass.

Chelimsky, E. (1995). Comments on the guiding principles. In W. R. Shadish, D. L. Newman, M. A. Scheirer, & C. Wye(Eds.), Guiding principles for evaluators (New directions for program evaluation, Vol. 66, pp. 53-54). San Fran-cisco: Jossey-Bass.

Eisner, E. W. (1991). The enlightened eye: Qualitative inquiry and the advancement of educational practice. New York:Macmillan.

Guba, E. G., & Lincoln, Y. (1988). Do inquiry paradigms imply inquiry methodologies? In D. M. Fetterman (Ed.), Qual-itative approaches to evaluation in education (pp. 89-115). New York: Praeger.

House, E. R. (1987). The evaluation audit. Evaluation Practice, 8(2), 52-56.House, E. R. (1995). Principled evaluation: A critique of the AEA guiding principles. In W. R. Shadish, D. L. Newman,

M. A. Scheirer, & C. Wye (Eds.), Guiding principles for evaluators (New directions for program evaluation, Vol. 66,pp. 27-34). San Francisco: Jossey-Bass.

Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards (2nd ed.). Thou-sand Oaks, CA: Sage.

Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Newbury Park, CA: Sage.Morris, M. (1998). Ethical challenges. American Journal of Evaluation, 19, 381-382.Patton, M. Q. (1986). Utilization-focused evaluation (2nd ed.). Beverly Hills, CA: Sage.Rossi, P. H. (1995). Doing good and getting it right. In W. R. Shadish, D. L. Newman, M. A. Scheirer, & C. Wye (Eds.),

Guiding principles for evaluators (New directions for program evaluation, Vol. 66, pp. 55-60). San Francisco:Jossey-Bass.

Smith, N. L. (2002). An analysis of ethical challenges in evaluation. American Journal of Evaluation, 23, 199-206.Stufflebeam, D. L. (2001). The metaevaluation imperative. American Journal of Evaluation, 27, 183-209.



Nonparticipant to Participant a Methodological Perspective on Evaluator Ethics

Documents

Transcript of Nonparticipant to Participant a Methodological Perspective on Evaluator Ethics