Trustworthiness

Clara Giraldo. Ps. MSC,Ph.D.

Maestría En Salud Colectiva

Seminario de Investigación II: InvestigaciónCualitativa

Alternative Criteria for Judging Quality

Diverse approaches of qualitative inquiryDiverse approaches of qualitative inquiryagree that the issue of credibility andagree that the issue of credibility andquality are intersect with the audience andquality are intersect with the audience andthe intended inquiry purpose.the intended inquiry purpose.

Patton identifies five contrasting sets of criteriafor judging the quality of qualitative inquiry, withdifferent philosophical frameworks. Some of theare overlapping , some of them have subtledifferences. The five criteria are:

Alternative Criteria for Judging Quality II

Traditional scientific research criteria: Derives from positivism, realism and analytic inductive

approaches.

Social Construction and constructivism criteria Derives from constructivism that states that the human

world is a constructed world and should be study as suchworld is a constructed world and should be study as such

Artist and evocative criteria Derivate from the autoDerivate from the auto--ethnography and evocative formsethnography and evocative forms

of inquiry.of inquiry.

Critical criteria Derives from critical theory, feminist theory and activist

research

Evaluations standards and principles. Derives from evaluation research standards

Alternative Criteria for Judging Quality III

Some of the confusion that people have inassessing QLR steams from thinking itrepresenting an uniform perspective , especially incontract with quantitative research.

Different perspectives about such things asDifferent perspectives about such things astruth and the nature of reality constitutetruth and the nature of reality constitutetruth and the nature of reality constitutetruth and the nature of reality constituteparadigms or worldview based on alternativeparadigms or worldview based on alternativeepistemologies andepistemologies and ontologiesontologies..

People see qualitative finding thought differentparadigmatic lenses.

Try to match the criteria of different perspectivescan be complex.

Traditional Sciences (TS) ResearchCriteria

One way to increase the credibility and thelegitimacy of qualitative inquiry among those whoplace priority to traditional scientific research is toemphasizes those traditional criteria.

TS emphasize objectivity, rigorous forms of data TS emphasize objectivity, rigorous forms of datacollection, cross-checking, cross-validation by usingmultiple coders and calculating inter-coderconsistency to establish the validity and reliability.

Search causal explanation and generalizaibility.

Social Constructionism and ConstructivistCriteria This perspective hasThis perspective has

generated a newgenerated a newlanguages andlanguages andconcepts to distingueconcepts to distinguequality in QLR.quality in QLR.

Conventionalterms

Naturalistic terms

internal validity Credibility

They encouragedialogues amongperspectives.

Uses de concept ofpraxis and reflexivity

external validity Transferability

Reliability Dependability

Objectivity Confirmability

Artistic and Evocative Criteria

They emphasize the called “narrative turn”.

They focuses on aesthetics, creativity,They focuses on aesthetics, creativity,interpretative vitality and expressiveinterpretative vitality and expressivevoices.voices.voices.voices.

The Vagina Monologs is a prominentexample or how artistic expression canprovide expressive voices and become acentral to the work as depictions of thephenomenon.

Critical Changes Criteria

Some people engage in QR as form to make aSome people engage in QR as form to make acritical analysis of social and political changescritical analysis of social and political changesand they put aside openand they put aside open--mindedness andmindedness andobjectivity and take an activist stances.objectivity and take an activist stances.

They always ask who is benefited and who is harmby the validity of the assessment.by the validity of the assessment.

They explain the political agenda of the researchthat is maintaining economic power and socialinequities. Influenced by Marxism the believe thatclass conflict is central to understand communitiesand societies.

Evaluation Standards and PrinciplesCriteria

The evaluation profession has adopted standardsthat call for evaluations to be useful, practical,to be useful, practical,ethical, and accurate.ethical, and accurate.

The AEA added the following principles:systematic inquiry, evaluator competence,The AEA added the following principles:systematic inquiry, evaluator competence,integrity/honesty, respect for people (fairness) andthe responsibility to generate a welfare( taking inaccount diversity of interest and values).

Many different standards and viewpoints coexistunder the broad umbrella.

Mixing and Changing Perspectives

These perspectives are angles of vision not only tocriticize but also to undertake them.

Mixing them can be useful, although it makecreates some tension among them.

What criteria you chose to emphasize in yourWhat criteria you chose to emphasize in yourresearch deepens on your inquiry , the valuesresearch deepens on your inquiry , the valuesand perspectives of the audiences for yourand perspectives of the audiences for yourwork .work .

Addressing Trustworthiness in QLR

The basic question addressed by the notion of trustworthiness,according to Lincoln and Guba, is simple:

"How can an inquirer persuade his or her audiences that theresearch findings of an inquiry are worth paying attention to?"(1985, p. 290).

When judging qualitative work, Strauss and Corbin (1990)believe that the "usual canons of ‘good science’…requirebelieve that the "usual canons of ‘good science’…requireredefinition in order to fit the realities of QLR" (p. 250).

Lincoln and Guba (1985, p. 300) have identified one alternativeset of criteria that correspond to those typically employed tojudge quantitative work (see Table 1).

Comparison of criteria for judging thequality of quantitative versus QLR

Conventional terms Naturalistic terms

internal validity Credibility

external validity Transferability

Reliability Dependability

Objectivity Confirmability

Critic to Lincoln and Guba’s QualityCriteria 1

Smith and Heshusius (1986) sharply criticize thosewriters, like Lincoln and Guba, who they believe haveadopted a stance of "detente" with rationalists.

They are particularly annoyed by Lincoln and Guba’sThey are particularly annoyed by Lincoln and Guba’suse of "comparable criteria," which to their eyes lookuse of "comparable criteria," which to their eyes lookuse of "comparable criteria," which to their eyes lookuse of "comparable criteria," which to their eyes looklittle different than the conventional criteria theylittle different than the conventional criteria theysupposedly replace.supposedly replace.

In either case, there must be a "belief in the assumptionthat what is known—be it an existent reality or aninterpreted reality—stands independent of the inquirerand can be described without distortion by the inquirer"(p. 6).

Critic to Lincoln and Guba’s QualityCriteria 2 Smith and Heshusius claim that naturalistic researchSmith and Heshusius claim that naturalistic research

can offer only an "interpretation of thecan offer only an "interpretation of theinterpretations of others, " and that to assume aninterpretations of others, " and that to assume anindependent reality is "unacceptable" for the QLRerindependent reality is "unacceptable" for the QLRer(p. 9).(p. 9).

Their stance is a strong one, because the only reality it Their stance is a strong one, because the only reality itaccepts is a completely mind-dependent one, which willvary from individual to individual; in other words, forSmith and Heshusius, there is no "out there" out there.

For these researchers, it would not be possible tochoose a best interpretation from among the manyavailable, because no technique or interpretation can be"epistemologically privileged" (p. 9).

Critic to Lincoln and Guba’s QualityCriteria 3 To maintain this stance would seem to negate theTo maintain this stance would seem to negate the

value of doing research at all, because it prohibitsvalue of doing research at all, because it prohibitsthe possibility of reconciling alternativethe possibility of reconciling alternativeinterpretationsinterpretations.

Therefore, it is important to determine which criteria areconsistent with the naturalistic paradigm, yet which allowconsistent with the naturalistic paradigm, yet which allowfor a declaration that "good science" has been carriedout.

In the following sections, conventional and naturalisticcriteria will be discussed, with the goal of selectingcriteria which are appropriate for judging the overalljudging the overalltrustworthiness of a qualitative studytrustworthiness of a qualitative study..

Internal Validity versus Credibility 1

In conventional inquiry, internal validity refers to theextent to which the findings accurately describe reality.

Lincoln and Guba (1985) state that "the determination ofsuch isomorphism is in principle impossible" (is in principle impossible" (p. 294p. 294),),because one would have to know the "precise nature ofbecause one would have to know the "precise nature ofbecause one would have to know the "precise nature ofbecause one would have to know the "precise nature ofthat realitythat reality" and, if one knew this already, there would beno need to test it (p. 295).

The conventional researcher must postulaterelationships and then test them; the postulate cannot beproved, but only falsified.

The naturalistic researcher, on the other hand,The naturalistic researcher, on the other hand,assumes the presence of multiple realitiesassumes the presence of multiple realities andattempts to represent these multiple realities adequately.Credibility becomes the test for this.Credibility becomes the test for this.

Internal Validity versus Credibility 1

Credibility depends less on sample size than on theCredibility depends less on sample size than on therichnessrichness of the information gathered and on theand on theanalytical abilities of the researcheranalytical abilities of the researcher (Patton, 2002).

It can be enhanced through triangulationenhanced through triangulation of data. Patton identifies four types of triangulation: 1) methodsmethods triangulation; 1) methodsmethods triangulation; 2) datadata triangulation; 3) triangulation through multiple analystsmultiple analysts; and 4) theorytheory triangulation. Other techniques for addressing credibility include

making segments of the raw data available for others toanalyze, and the use of "member checksmember checks," in whichrespondents are asked to corroborate findings (Lincolnand Guba, 1985, pp. 313-316).

Internal Validity versus Credibility 13

Lincoln y Guba (1985) afirman que otras técnicas pararevolverse el problema de la credibilidad incluyen esincluir segmentos que ponen a disposición los datoscrudos para que otros las analicen, y el uso "decontroles de miembros," en cuales se les pide a losparticipantes que confirmar conclusiones, 1985, ps 313-participantes que confirmar conclusiones, 1985, ps 313-316).

Lincoln yLincoln y GubaGuba proponen una medida que podríaproponen una medida que podríarealzar la seriedad de la investigación cualitativa querealzar la seriedad de la investigación cualitativa quees el uso "de una de auditoría de preguntas," en quees el uso "de una de auditoría de preguntas," en quelos revisores examinan tanto proceso como ellos revisores examinan tanto proceso como elproducto de la investigación mirando laproducto de la investigación mirando laconsistencia.consistencia.

External Validity / Generalizabilityversus Transferability 1 In conventional research, external validity refers to the

ability to generalize findings across different settings. Making generalizations involves a trade-off between

internal and external validity (Lincoln and Guba, 1985).That is, in order to make generalizable statements thatapply to many contexts, one can include only limitedaspects of each local context.apply to many contexts, one can include only limitedaspects of each local context.

Lincoln and Guba (1985) admit that generalizability isgeneralizability is"an appealing concept,""an appealing concept," because it allows asemblance of prediction and control over situations (pp.110-111).

Yet they suggest that the existence of local conditionsthe existence of local conditions"makes it impossible to generalize"makes it impossible to generalize" (p. 124).Cronbach (1975) discusses the problem by saying:

External Validity / Generalizabilityversus Transferability 2 The trouble, as I see it, is that we cannot store up

generalizations and constructs for ultimate assembly intoa network.

It is as if we needed a gross of dry cells to power anengine and could only make one a month. The energywould leak out of the first cells before we had half thebattery completed (p. 123).would leak out of the first cells before we had half thebattery completed (p. 123).

According to Cronbach, "when we give proper weight tolocal conditions, any generalization is a workinghypothesis, not a conclusion" (p. 125).

In the naturalistic paradigm, the transferability of aIn the naturalistic paradigm, the transferability of aworking hypothesis to other situations depends onworking hypothesis to other situations depends onthe degree of similarity between the originalthe degree of similarity between the originalsituation and the situation to which it is transferredsituation and the situation to which it is transferred.

External Validity / Generalizabilityversus Transferability 3 The researcher cannot specify the transferability ofThe researcher cannot specify the transferability of

findings; he or she can only provide sufficientfindings; he or she can only provide sufficientinformation that can then be used by the reader toinformation that can then be used by the reader todetermine whether the findings are applicable to thedetermine whether the findings are applicable to thenew situationnew situation (Lincoln and Guba, 1985).

Other writers use similar language to describetransferability, if not the word itself.

Other writers use similar language to describetransferability, if not the word itself.

For example, Stake (Stake (19781978) refers to what he calls) refers to what he calls"naturalistic generalization"naturalistic generalization" (p. 6).

Patton suggests that "extrapolation" is anPatton suggests that "extrapolation" is anappropriate term for this processappropriate term for this process (2002, p. 489).

Eisner says it is a form of "retrospective generalization"that can allow us to understand our past (and future)experiences in a new way (1991, p. 205).

Reliability versus Dependability 1

Kirk and Miller (1986) identify three types of reliabilityreferred to in conventional research, which relate to:

1) the degree to which a measurement, givenrepeatedly, remains the same;

2) the stability of a measurement over time; and 3) the similarity of measurements within a given time 3) the similarity of measurements within a given time

period (pp. 41-42). They note that "issues of reliability have received littlereliability have received little

attention" from QLRersattention" from QLRers, who have instead focused onachieving greater validity in their work (p. 42).

Reliability versus Dependability 2

Although they give several examples of how reliabilitygive several examples of how reliabilitymight be viewed in qualitative workmight be viewed in qualitative work, the essence ofthese examples can be summed up in the followingstatement by Lincoln and Guba (1985): "Since there"Since therecan be no validity without reliability (and thus nocan be no validity without reliability (and thus nocredibility without dependability),credibility without dependability), a demonstration ofcredibility without dependability),credibility without dependability), a demonstration ofthe former is sufficient to establish the latter" (p. 316).

Nevertheless, Lincoln and Guba do propose oneGuba do propose onemeasure which might enhance the dependabilitymeasure which might enhance the dependability ofQLR.

That is the use of an "inquiry audit," in which"inquiry audit," in whichreviewers examine both the process and the productreviewers examine both the process and the productof the research for consistencyof the research for consistency (1985, p. 317).

Objectivity versus Confirmability 1

Conventional wisdom says that research whichrelies on quantitative measures to define asituation is relatively value-free, and thereforeobjective.

QLRQLR, which relies on interpretations and isisadmittedly valuevalue--bound, is considered to bebound, is considered to beQLRQLR, which relies on interpretations and isisadmittedly valuevalue--bound, is considered to bebound, is considered to besubjective.subjective.

In the world of conventional research,subjectivity leads to results that are bothunreliable and invalid.

Objectivity versus Confirmability 2

There are many researchers, however, who call intoquestion the true objectivity of statistical measures and,indeed, the possibility of ever attaining pure objectivity atall (Lincoln and Guba, 1985; Eisner, 1991).

PattonPatton (1990) believes that the terms objectivity and PattonPatton (1990) believes that the terms objectivity andsubjectivity have become "ideological ammunition in theparadigms debate." He prefers to "avoid using eitheravoid using eitherwordword and to stay out of futile debates about subjectivityversus objectivity." Instead, he strives for "empathiche strives for "empathicneutrality" (p. 55).neutrality" (p. 55).

While admitting that these two words appear to betwo words appear to becontradictorycontradictory, Patton points out that empathy "is aempathy "is astance toward the peoplestance toward the people one encounters, whileneutrality is a stance toward the findingsneutrality is a stance toward the findings" (p. 58).

Objectivity versus Confirmability 3 A researcher who is neutral tries to be nonresearcher who is neutral tries to be non--

judgmental, and strives to report what is found in ajudgmental, and strives to report what is found in abalanced waybalanced way.

Lincoln and Guba (Lincoln and Guba (19851985) choose to speak of the) choose to speak of the"confirmability"confirmability" of the research.

In a sense, they refer to the degree to which thedegree to which theresearcher can demonstrate the neutrality of theresearcher can demonstrate the neutrality of theresearch interpretations, through a "confirmabilityresearch interpretations, through a "confirmabilityresearch interpretations, through a "confirmabilityresearch interpretations, through a "confirmabilityauditaudit."

This means providing an auditproviding an audit trail consisting of 1) raw dataraw data; 2) analysisanalysis notes; 3) reconstruction and synthesis productssynthesis products; 4) process notesprocess notes; 5) personal notespersonal notes; and 6) preliminary developmental informationpreliminary developmental information (pp. 320 -

321).

Objectivity versus Confirmability 4

With regard to objectivity in QLR, it may be useful toWith regard to objectivity in QLR, it may be useful toturn to Phillips (turn to Phillips (19901990), who questions whether there), who questions whether thereis really much difference between quantitative andis really much difference between quantitative andQLRQLR:

Bad work of either kind is equally to be deplored; andgood work of either kind is still—at best—only tentative.good work of either kind is still—at best—only tentative.

But the good work in both cases will be objective, in thesense that it has been opened up to criticism, and thereasons and evidence offered in both cases will havewithstood serious scrutiny.

The works will have faced potential refutation, andinsofar as they have survived, they will be regarded asworthy of further investigation (p. 35).