Cap. 8

P1: KAE052184097Xc08 CB1040B/Ericsson 0 521 84087 X May 22 , 2006 14 :15

C H A P T E R 8

Observation of Work Practicesin Natural Settings

William J. Clancey

Keywords: Ethnography, Workplace Study,Practice, Participant Observation, Ethnome-thodology, Lived Work

Introduction

Expertise is not just about inference appliedto facts and heuristics, but about being asocial actor. Observation of natural settingsbegins not with laboratory behavioral tasks –problems fed to a “subject” – but with howwork methods are adapted and evaluated byexperts themselves, as situations are expe-rienced as problematic and formulated asdefined tasks and plans. My focus in thischapter is on socially and physically locatedbehaviors, especially those involving conver-sations, tools, and informal (ad hoc) interac-tions. How an observer engages with practi-tioners in a work setting itself requires exper-tise, including concepts, tools, and methodsfor understanding other people’s motivesand problems, often coupled with methodsfor work systems design.

By watching people at work in everydaysettings (Rogoff & Lave 1984) and observ-

ing activities over time in different cir-cumstances, we can study and documentwork practices, including those of proficientdomain practitioners. This chapter intro-duces and illustrates a theoretical frameworkas well as methods for observing work prac-tices in everyday (or natural) settings in amanner that enables understanding and pos-sibly improving how the work is done.

In the first part of this chapter, I explainthe notion of work practices and the his-torical development of observation in nat-ural settings. In the middle part, I elabo-rate the perspective of ethnomethodology,including contrasting ways of viewing peopleand workplaces, and different units of analy-sis for representing work observations. In thefinal part, I present methods for observationin some detail and conclude with trends andopen issues.

What are Work Practices ina Natural Setting?

Every setting is “natural” for the people whofrequent it. A laboratory is a natural work

12 7


12 8 the cambridge handbook of expertise and expert performance

setting for some scientists, whereas expe-dition base camps are natural for others.The framework provided here is intendedto be applicable to any setting, includ-ing school playgrounds, churches, interstatehighways, and so on. But we focus onworkplaces, where people are attempting toget some work done, for which they havebeen prepared, and have sufficient expe-rience to be acknowledged as experts byother people with whom they interact. Thiscan be contrasted with studies of every-day people being expert at everyday things(e.g., jumping rope, car driving) or eventspurposely arranged by a researcher in alaboratory.

In studying natural settings, one viewsthem broadly: Consider a teacher in a schoolwithin a community, not just a classroom.Seek to grasp an entire place, with its nestedcontexts: Rather than focusing on a physi-cian in a patient exam room, study the clinic,including the waiting room.

Heuristically, one can view an expert’sperformance as a play, identifying the stage,the “acts,” roles, and the audience. But alsoview the play as having a history, whosenature is changing in today’s performance:What are the actors’ long-term motives?How is this performance challenging orinfluenced by the broader community of prac-tice (Wenger 1998) (e.g., other clinics andnurses)?

Also inquire more locally about thechronology and flow of a performance: Howdo people prepare, who assists them (thinkof actors), how do they get informationabout today’s work, when and where dothey review and plan their work, how areevents scheduled? Look for informal placesand off-stage roles – backrooms and prepara-tion areas, dispatchers, janitors, and supportpersonnel. All of this is part of the exper-tise of getting a job done, and multiple partsand contributions need to be identified if thefundamental question about work is to beanswered: What affects the quality of thisperformance? What accounts for its success?As a heuristic, to capture these contextualeffects, one might frame a study as being

“a day in the life” of the people – and thatmeans 24 hours, not the nominal work day.

Thus, a study of work practices is actu-ally a study of a setting; this context makesthe observed behavior understandable. Forexample, consider understanding clowns:

If we had a film of a clown doing som-ersaults, and nothing else (i.e., we knewnothing about circuses, about the history ofclowns and so on), then the film would nottell us what we need to know to make senseof what the clown was doing. . . . One wouldneed to know something about how theyare part and parcel of circuses, and howtheir somersaulting is viewed [by manyobservers] as a kind of sentimental self-mockery. (Harper 2 000, pp. 2 44–2 45 ;attributed to Gilbert Ryle)

To understand a setting, it is useful toview all workers (not just performers onstage) as social actors. When we say thatwork is socially recognized as “requiring spe-cial skills or knowledge derived from exten-sive experience” (Hoffman, 1998, p. 86), wemean that people are visibly demonstrat-ing competency, in how they make inter-pretations, conduct business, and produceresults that are “recognizably accountable”(of agreeable quality) to institutional andpublic audiences (Heritage 1984 ; Dourish &Button 1998). This perspective has the dualeffect for expertise studies of considering theworker as an agent who, with other agents,coconstructs what constitutes a problem tobe solved and how the product will be eval-uated. Methods for applying this theoreti-cal perspective, called ethnomethodology, arepresented in this chapter.

Observing people in a natural setting iscommonly called fieldwork. Besides watch-ing and recording and asking questions, field-work may include interviewing, studyingdocuments, and meeting with the peoplebeing studied to analyze data together andpresent findings (Forsythe 1999, p. 128).Fieldwork is most often associated with thebroader method of study called ethnogra-phy (Spradley 1979; Fetterman 1998 Harper2000, p. 239), literally, the written study


observation of work practices in natural settings 12 9

of a people or culture. Neither fieldworknor ethnography are specific to any disci-pline. Originally associated most stronglywith anthropology, the methods todayare commonly used by linguists, sociolo-gists, computer scientists, and educationalpsychologists.

The actual methods of observation –spending time in a natural setting andrecording what occurs – may at first appearas the defining characteristic of an ethno-graphic study, but the difficult and less obvi-ous part is being able to understand workpractice. For example, outsiders are oftenunaware of the inherent conflicts of a worksetting (e.g., to physicians, dying people area source of money; to police, crime statis-tics a source of political trouble), which limitwhat can be done, making it necessary to cre-atively interpret procedures and regulations.

This chapter focuses on how to see whatis happening, how to apply ethnomethodol-ogy concepts to analyzing everyday actions.Starting the other way around – with cam-era at hand and a poor theoretical back-ground – could be like bringing an aquariumfish net to the deep sea, collecting a hodge-podge of anecdotes, narratives, and interest-ing photographs, with little understanding ofpeople’s practices (Button & Harper 1996,p. 267). Furthermore, a planned analytic pro-gram is important when studying work prac-tice for design, “otherwise observations canbe merely invoked at will for particular pur-poses such as, for example, to legitimizedesign decisions already made” (p. 267).

An observational study is itself modu-lated by the observer’s purpose and relationto the organizational setting. Intending totransform the setting (e.g., as a consultant)requires engaging as an observer in a partic-ular way, not merely recording and note tak-ing. A helpful, reflective activity called par-ticipatory design (Greenbaum & Kyng 1991,p. 7; Beyer & Holtzblatt 1998) involves nego-tiating and codiscovering with the workerswhat is to be investigated (e.g., setting up a“task force group”; Engestrom 1999, pp. 71–73). In settings such as hospitals and businessoffices, this developmental perspective com-

monly focuses on software engineering andorganizational change.

Historical and ContemporaryPerspectives

This section reviews how observation innatural settings developed and was shaped,especially by photographic tools, and howit relates to the psychological study ofexpertise.

Scientific Observation in Natural Settings

In studies of culture, surveying “informants”on site goes back to the earliest days of 19th-century anthropology (Bernard 1998, p. 12).Several articles and books provide excellentsummaries of the theoretical backgroundand methods for observation in natural set-tings, including especially Direct SystematicObservation of Behavior (Johnson & Sackett1998) and Participant Observation (Spradley1980; Dewalt & Dewalt 2002).

As the ethnomethodologist stresses,observation in natural settings is inherent insocial life, for it is what people themselvesare doing to organize and advance theirown concerns. But perhaps the tacit, uncon-trollable, and mundane aspect of everydaylife led psychologists to set up experimentsin laboratories and anthropologists to setup camp in exotic third-world villages.Moving studies of knowledge and expertiseto modern work settings developed over along period of time, starting with cognitiveanthropologists and socio-technical analysts(Emery 1959), and progressing to the“Scandinavian approach” to informationsystem design (Ehn 1988; Greenbaum &Kyng 1991). But today’s methods of obser-vation began with the invention of – andmotivations for – photography.

Visual Anthropology

Photographs and video are indispensable forrecording behavior for later study. The visualrecord allows studying how people structuretheir environment, providing clues about


130 the cambridge handbook of expertise and expert performance

how they are relating to each other andstructuring their work life. Using photogra-phy for close observation dates to the late19th century. Eadweard Muybridge’s famousearly motion pictures (Galloping Horse[1878], Ascending Stairs [1884]) demon-strate the early motivation of using film tostudy animal and human movements whosespeed or structure elude direct observation.

Margaret Mead and Gregory Bateson pio-neered the use of film for capturing non-verbal behavior. Their work was influen-tial in treating photography as primary data,rather than as only illustrations (El Guindi1998, p. 472). Today the use of photo-graphic methods is fundamental in observa-tion of natural settings, and is termed videoethnography or interaction analysis (Jordan& Henderson 1995).

An integral part of any observationalstudy in a natural setting considers howphysical space, including furniture anddesigned facilities, is used “as a specializedelaboration of culture” (Hall 1966), calledproxemics. This study broadly relates ethol-ogy (Lorenz 1952) to analyses of physical-perceptual experience (e.g., kinesics, Bird-whistell 1952), including “body language”(Scheflen 1972), personal and public kindsof space, nonverbal communication (Hall1959), and culture differences. Using time-lapse video, Whyte (1980; PPS 2004) studiedhow people used public plazas at lunchtime,a striking everyday application of proxemicsfor architectural design.

Visual analysis considers posture, ges-tures, distance and orientation of bodies,territoriality, habitual use of space (e.g.,movement during the day), relation ofrecreational and work areas, preferencesfor privacy or indirect involvement (opendoors), and so on. For example, referring toFigure 8.1, how would you group the peo-ple, given their posture and behavior? Whatactivities occur in this space? What do bodypositions reveal about people’s sense of tim-ing or urgency? Even a single image canreveal a great deal, and will provide evidencefor broader hypotheses about relationships,complemented by living with these peoplefor several weeks.

Figure 8.1. “The area between the tents” at theHaughton-Mars Base Camp 1999.

The Development of Natural Observationin Expertise Studies

Analysts seeking improved efficiency in pro-cedures and designing automation stud-ied workplaces throughout the 20th cen-tury. Developmental psychology primarilyfocused on schools, whereas organizationallearning (Senge 1990) chose business set-tings. Computer scientists brought domainspecialists into their labs to develop expertsystems in the model-building process calledknowledge acquisition (Buchanan & Shortliffe1984). Human-factors psychologists took upthe same analytic concepts for decompos-ing work into formal diagrams of goalsand methods, called cognitive task analysis(Vicente 1999), and characterized decisionmaking as probabilistic analyses of situationsand judgmental rules (Chi et al. 1988). Atthe same time, social scientists were beingdrawn by colleagues designing computersystems, motivated largely by labor forcesin Europe (Ehn 1988), forming subfieldssuch as business anthropology and work-place studies (Luff et al. 2000).

By the 1990s, industrial engineers andsocial scientists already in the workplacewere joined by computer scientists and psy-chologists, who had transitioned from labo-ratory interviews and experiments to “designin the context of use” (Greenbaum & Kyng


observation of work practices in natural settings 131

1991). The work of studying knowledge andlearning moved to everyday settings such assupermarkets (Lave 1988), insurance offices,and weather bureaus (Hoffman et al. 2000).The discipline of human-computer interac-tion (HCI) became a large, specialized sub-field, a consortium of graphics artists, socialtheorists, psychology modelers, and softwareengineers (Nardi 1996; Blum 1996; Kling &Star 1998).

Broadly speaking, HCI research has pro-gressed from viewing people as computerusers – that is, asking questions such as “Whathappens if people are in the loop?” – to view-ing people, computers, documents, facilities,and so on as a total system, and understandingthe processes holistically. In some respects,this approach began with socio-technicalsystems analysis in the 1950s–1970s (Corbetet al. 1991, p. 9ff). Hutchins (1995) providesespecially well-developed examples of howtools, interfaces, and distributed group inter-actions constitute a work system.

Expertise in Context: Learning to See

Observing and systematically studying awork place is sometimes treated as easy bynon-social-scientists, who might perform thework sketchily or not actually analyze prac-tices (Forsythe 1999). The spread of theanthropological and social perspectives tocognitive science was at first limited, at bestshifting the analysis to include the socialcontext. For example, only one chapter inExpertise in Context (Feltovich et al. 1997)explicitly involved an observational study ofa natural setting (Shalin’s video analysis in ahospital). Ericsson and Charness used diariesfor studying violinists, without investigatingtheir home setting. Other researchers con-sidered experts as socially selected (Agnewet al. 1997) and more broadly serving andpart of market, organization, or commu-nity networks (Stern 1997); or viewed exper-tise as part of cultural construction (Collins1997, Clancey 1997).

An edited volume from a decade ear-lier, The Nature of Expertise by Chi et al.(1988), focused even more narrowly onmental processing of text: Documents were

provided to subjects to read, to judge, totype, or learn from. Expertise was viewednot about competence in settings (i.e.,situated action), but decision making, rea-soning, memory retrieval, pattern match-ing – predominantly aspects of the assumedinternal, mental activity occurring in thebrain. For example, a study of restaurantwaiters (p. 27) was reduced to a study ofmemory, not the “lived work” of being awaiter. A study of typing concerned timingof finger movements, nothing about officework. Of the twelve studies of experts,only one included “naturalistic observation”to “fashion a relatively naturalistic task”(Lesgold et al. 1988; p. 313), namely, dictat-ing X-ray interpretations.

This said, one of the most influential anal-yses of the contextual aspects of behav-ior, Suchman’s (1987) Plans and SituatedActions, also did not involve the study ofpractice. Suchman studied two people work-ing together who had never used a photo-copier before (p. 115) – a form of puzzle solv-ing in which a predefined task is presentedin “the real world” (p. 114). Suchman’s studyis an example of ethnomethodological analy-sis because it focuses on mutual, visible con-struction of understanding and methods, butit is not carried out using the ethnographicmethod (Dourish and Button 1998, p. 406)because this was not a study of establishedpractices in a familiar setting.

In summary, a participatory design pro-ject uses ethnography to study work prac-tice, which may be analyzed from anethnomethodological perspective (Heritage1984). More generally, ethnography mayinvolve many other analytic orientations,emphasizing different phenomena, topics,and issues (Dourish & Button 1998, p. 404).Ethnographic observation involves a rigor-ous commitment to confronting the worldsof people as they experience everyday life,to understand how problematic situationsactually arise and are managed. Workplacestudies, contrasted with the study of knowl-edge and experts in the 1970s and 1980s (Chiet al. 1988), signify a dramatic change in howexpertise is viewed and studied, often withentirely different motivations, methods, and



partnerships, and having a significant affecton the design of new technologies.

Work Systems Design Project Examples

Here I present two representative exam-ples of work systems design projects to illus-trate the relation of methods and the resultsachieved.

A three-year ethnographic study ofa reprographics store was conducted toimprove customer service (Whalen et al.2004).1 The data were collected in threephases. First, the researchers made ethno-graphic observations, shadowing and inter-viewing employees as they worked. Sec-ond, the team collected over 400 hoursof video recordings in the store frommultiple simultaneously recording cameras.The videotapes were digitized and dividedinto distinct episodes, consisting of morethan 500 customer-employee interactions,some of which were transcribed and ana-lyzed. Finally, three research team mem-bers became participant observers in thestores, working as employees, serving cus-tomers, and operating the printing and copy-ing equipment.

The study resulted in the developmentof a “customer service skill set,” a set ofweb-based instructional modules designedto raise employees’ awareness of the orga-nization of customer-employee interactions.Topics include how to listen to what the cus-tomer wants during initial order taking, howto talk about price, and the importance oftaking the time to review the completed jobwith the customer. The modules were co-developed by the research team and six storeemployees who met once a week for twomonths. For example, the common question“When do you need it?” is practically unan-swerable by the customer because they don’tknow the work load and scheduling con-straints of the store, so they reply, “Whencan you have it?” The employees were askedto experiment with ways of opening up thediscussion about due time (e.g., “Is this anurgent job?” or “Would you like to pick thisup tomorrow afternoon?”), and they noticeda useful change in customer responses. These

analyses inform further reconsideration ofthe burden placed on customers in justify-ing the need for “full service” and the deli-cate balance of providing assistance to “self-service” customers.

The second example illustrates system-atic design and adaptation to most aspectsof a work system – organization, facili-ties, processes, schedules, documents, andcomputer tools. For three-and-a-half years,NASA Ames’ researchers worked closelywith the Mars Exploration Rover (MER)science and operations support teams atthe Jet Propulsion Laboratory, in Pasadena,California. The project included the designand training phase of the mission (start-ing January 2001), as well as the surfaceoperations phase that began after the suc-cessful landing of two rovers on the Mar-tian surface (January 2004). Observationfocused on the interactions between thescientists, computer systems, communica-tion network (e.g., relay via Mars satellites),and the rovers, using ethnography to under-stand the successes, gaps, and problem areasin work flows, information flows, and tooldesign in operations systems.

Research data included field notes, mis-sion documents and reports, photographs,video, and audiotape of the work of mis-sion participants. Two to four researcherswere present during all of the premissiontests (2001–2003), all but one of the sci-ence team’s twice-yearly meetings (2001–2003), and the majority of the science team’sweekly conference calls. Learning the intri-cacies of the rover instruments and theiroperation was necessary to understand thetelerobotic work. Ongoing findings in theform of “lessons learned” with recommenda-tions for improving mission work processeswere presented to operations teams severaltimes each year. Data analysis focused on thelearning of the science team as a work prac-tice developed that moved the daily rover-operations plan from team to team across thethree-shift mission timeline. The researchersidentified and categorized types of informa-tion and working groups, and defined workflows, communication exchanges, scien-tists’ work practice and scientific reasoning



process, and the interactions of work prac-tice between scientists and rover engineers.Over time many scientists and managersbecame informally involved in assisting theobservation and documentation process andrefining design recommendations.

The researchers developed a naming con-vention and ontology for objects on Mars,a prioritization scheme for planning roveractivities, and a method of documentingthe scientific intent of telerobotic opera-tions to facilitate communication betweenoperations shifts and mission disciplines.They trained MER scientists in these pro-cedures and associated tools during simu-lated missions. During the mission in 2004 ,two researchers moved to Pasadena; sixresearchers rotated to cover the shifts thatmoved forty minutes later each day in syn-chrony with local Mars time. The teamthen developed operations concepts for an“extended mission phase,” during whichscientists worked from their home cities,and rover planning was compressed andsimplified to reduce work on nights andweekends.

Overall, this work systems design projecthelped define and enhance the teleroboticscientific process and related mission sur-face operations, including design of facili-ties for science meetings. Researchers con-tributed to the design of four computersystems used for rover planning and scien-tific collaboration that were being developedsimultaneously by NASA Ames colleaguesand JPL. The MER work systems design andthe methods employed are influencing oper-ations concepts and system architectures forsubsequent missions.

Ethnomethodology’s AnalyticPerspective

In this section, I explain how the “method-ology” being studied in a workplace is notjust a technical process for accomplishing atask, but incorporates social values and cri-teria for judging the quality of the work.This idea originated in Garfinkel’s discovery

in the mid-1950s of jurors’ “methodological”issues:

. . . such as the distinction between “fact”and “opinion,” between “what we’re enti-tled to say,” “what the evidence shows,” and“what can be demonstrated” . . . These dis-tinctions were handled in coherently orga-nized and “agree-able” ways and the jurorsassumed and counted on one another’sabilities to use them, draw appropriateinferences from them and see the sense ofthem . . . common-sense considerations that“anyone could see.” (Heritage 1984 , p. 4)

Ethnomethodology thus emphasizes thecommonsense knowledge and practices ofordinary members of society, as they “makesense of, find their way about in, and act onthe circumstances in which they find them-selves” (p. 4). However, formalizing theseassumptions, values, and resulting proce-dures is not necessarily easy for people with-out training (Forsythe 1999).

Ethnomethodology has led researchersto reconceive how knowledge and actionare framed, “wresting . . . preoccupation withthe phenomenon of error” prevalent inhuman factors research (Heritage 1984). Thefocus shifts to how people succeed, howthey construct the “inherent intelligibilityand accountability” of social activity, placingnew emphasis on the knowledge people use“in devising or recognizing conduct” (p. 5).Button and Harper (1996) provide a cogentexample about how “Decisions about whatcrimes are reported by police are intimatelytied up with questions of what is practical forthe reporting officer and what is in the inter-ests of the police organization as a whole”(p. 275).

Contrasted with technical knowledge(Schon 1987), this aspect of work methodsis reflective and social, concerning how one’sbehavior will be viewed, through under-stood norms and social consequences. Eth-nomethodology thus provides a kind oflogical, systemic underpinning to how activ-ity becomes coordinated – “how the actorscome to share a common appraisal of theirempirical circumstances” (Heritage 1984 ,p. 305) – that is, the process by which they



come to cooperate and their methods forresolving conflicts.

The idea of “intelligibility and account-ability” means that the work activity is“organized so that it can be rationalized”(Dourish & Button 1998, p. 415), that is,so that it appears rational. For example,the Mars Exploration Rover’s (MER) oper-ations (Squyres et al. 2004) were plannedand orchestrated by the science team so theexploration could be recognizable to oth-ers in perpetuity as being science, especiallythrough the method of justifying instrumentapplications in terms of hypothesis testing.In practice, geologists will often just strikea rock to see what is inside. In MER, theapplication of the rock abrasion tool wasoften explained within the group and tothe public as looking for something spe-cific. As the mission continued on for manymonths, the need for such rationalizationdiminished, but as the scientists were boundat the hip, with one rover to command (ateach site), they continued to justify to eachother why they would hit a particular rockand not another – something that would beinconceivable in their activity of physicallywalking through such a site with a hammerand hand lens. Thus, the practice of geologychanged during the MER mission to adaptto the circumstances of a collective, his-torical, public, time-pressured activity; andproduction of accounts of what should bedemonstrably scientific action were adaptedto fit this situation (cf. Dourish & Button1998, p. 416).

One must avoid a misconception thattechnical knowledge is just being selectivelyapplied in social ways. Rather, what counts asexpertise – the knowledge required to iden-tify and solve problems – reflectively devel-ops within the setting, which Collins calls “themutual constitution of the social and con-ceptual” (Collins 1997, p. 296). During theMER mission, a cadre of scientists and engi-neers capable of doing science with rovershas developed new expertise and meth-ods of working across disciplines in a time-pressured way.

In summary, expertise is more than facts,theories, and procedures (e.g., how to be ageologist or policeman); it includes practi-

cal, setting-determined know-how in beinga recognizably competent social actor. Eth-nomethodology reveals the reflective workof constructing observable (nonprivate) cat-egorizations (e.g., deciding which Mars rocksto investigate). Thus, an essential task forthe outside observer is to learn to see theordered world of the community of practice:“Human activity exhibits a methodicalorderliness . . . that the co-participants canand do realize, procedurally, at each andevery moment. . . . The task for the analystis to demonstrate just how they do this”(Whalen et al. 2004 , p. 6). The followingsection provides some useful frameworks.

What People Do: ContrastingFrameworks

Social-analytic concepts for understandinghuman behavior in natural settings are con-trasted here with information processingconcepts that heretofore framed the study ofknowledge and expertise (Newell & Simon1972).

Practice vs. Process

Practice concerns “work as experienced bythose who engage in it” (Button & Harper1996, p. 264), especially, how “recognizablecategories of work are assembled in the real-time actions and interactions of workers”(p. 264), memorably described by Wynn(1991):

The person who works with informationdeals with an “object” that is more diffi-cult to define and capture than informationflow charts would have us imagine. Theseshow “information” in little blocks or trian-gles moving along arrows to encounter spe-cific transformations and directions alongthe diagram. In reality, it seems, all alongthe arrows, as well as at the nodes, thatthere are people helping this block to bewhat it needs to be – to name it, put itunder the heading where it will be seen asa recognizable variant, deciding whether toleave it in or take it out, whom to convey itto. (pp. 56–57).



Button and Harper (1996, p. 265) give theexample of people analyzing interviews:“The coders would resort to a variety of prac-tices to decide what the coding rules actu-ally required of them and whether what theywere doing was actually (or virtually) in cor-respondence with those rules.”

Practice is also called “lived work” – “whatwork consists of as it is lived as part of orga-nizational life by those who do it” (Button& Harper 1996, p. 272). Practice is to becontrasted with formal process specificationof what work is to be done. In the work-place itself, processes are often idealized andconstitute shared values – “crimes should bereported to the bureau as soon as possible”(p. 277). Narratives that people record orpresent to authorities cater to these avowedpolicies or preferences, creating an inherentconflict in the work system between whatpeople do and what they say they do. Thepoint is not just that the documents andbehavior may disagree, but rather, for exam-ple, the records may reveal workers’ under-standing of how their practices must be rep-resented to appear rational.

Two fundamental concepts related to thepractice–process distinction are behavior–function and activity–task. Process mod-els (e.g., information processing diagrams)are idealized functional representations ofthe tasks that people in certain roles areexpected to do. Practice concerns chrono-logical, located behaviors, in terms of every-day activities, for example, “reading email,”“meeting with a client,” and “sorting throughpapers.” Activities are how people “chunk”their day, how they would naturally describe“what I am doing now.” Tasks are discovered,formulated, and carried out within activities,which occur on multiple levels in parallel(Clancey 2002).

Putting these ideas together, one mustbeware of identifying a formalized scenario(cf. Feltovich et al. 1997, p. 117) with thephysical, interactive, social context in whichwork occurs. The work context is fun-damentally conceptual (i.e., it cannot beexhaustively inventoried in descriptions ordiagrams) and dynamically interpreted, inwhich the actor relates constraints of loca-tion, timing, responsibility, role, changing

organization, and so on. Scenarios used forstudying expertise often represent an exper-imenter’s idealized notion of the “inputs,”and thus working a scenario may be morelike solving a contrived puzzle than inter-acting with the flow of events that an actornaturally experiences.

Invisible vs. Overt Work

Observing work is not necessarily as easyas watching an assembly line. Work may beinvisible (Nardi & Engestrom 1999, p. 2)to an observer because of biases, because itoccurs “back stage,” or because it is tacit,even to the practitioners. These three aspectsare discussed here.

First, preconceptions and biased methodsmay prevent the ethnographer from seeingwhat workers accomplish. For example, in astudy of telephone directory operators, theresearchers’ a priori “notion of the ‘canon-ical call’ rendered the variability of actualcalls invisible and led to a poor design fora partially automated directory assistancesystem” (p. 3). A related presumption isthat people with authority are the experts(Jordan 1992). For example, the Mycin pro-gram (Buchanan & Shortliffe 1984) wasdesigned in the 1970s to capture the exper-tise of physicians, but no effort was madeto understand the role of nurses and theirneeds; in the study of medical expertise,nurses were “non-persons” (Goffman 1969;Star & Strauss 1999, p. 15).

The second aspect of invisibility arisesbecause “many performers – athletes, musi-cians, actors, and arguably, scientists – keepthe arduous process of preparation for pub-lic display well behind the scenes” (Star &Strauss 1999, p. 21), which Goffman called“back stage.” One must beware of violat-ing autonomy or not getting useful informa-tion because of members’ strategic filteringor hiding of behavior (Star & Strauss 1999,p. 22).

The third form of invisible work is tacit“articulation work” – “work that gets thingsback ‘on track’ in the face of the unexpected,and modifies action to accommodate unan-ticipated contingencies.” (Star & Strauss1999, p. 10). These may be steps that people



take for granted, such as a phone call to acolleague, which they wouldn’t necessarilyelevate to being a “method.”

Participatory design handles the variousforms of invisible work by using ethnogra-phy to identify stakeholders and then involv-ing them in the work systems design project(e.g., see the examples in Greenbaum &Kyng 1991).

Members’ Documentationvs. Literal Accounts

The production of documentation is partof the lived work of most business, gov-ernment, and scientific professions. To dealwith the nonliteral nature of documentationmentioned previously, one should study theactivity of reporting “involved in sustainingan account of the work as a formal sequen-tial operation” (Button & Harper 1996,p. 272) as a situated action with social func-tions. For example, in Mars habitat simula-tions (Clancey 2002 , in press), one can learnfrom daily reports what the crew did. Butone must also inquire how the reportingwas accomplished (e.g., contingencies suchas chores, fatigue, power failure, etc. thatmade reporting problematic), what account-ability concerned the crew (e.g., the publicimage of the Mars Society; hence what wasemphasized or omitted), and why report-ing was given such priority (e.g., to adhereto scientific norms). What people write maynot be what they actually did, and interviewsmay present yet another perspective on whythe reports even exist.

Managing Inherent Conflictsvs. Applying Knowledge

One view of expertise is that people applyknowledge to accomplish goals (Newell &Simon 1972). Yet, goals are not simply thelocal statement of a task, but relate to long-term social-organizational objectives, suchas later “work load and responsibilities”(Button & Harper 1996, p. 277). For exam-ple, the chair of NASA’s Mission Manage-ment Team during the Columbia mission(which was destroyed on re-entry by wingtiles damaged by broken tank insulation

foam during launch) didn’t classify foamdamage on the prior mission, STS-112 inDecember 2002 , as an “in-flight anomaly” –the established practice. Doing so could havedelayed a subsequent mission in Februarythat she would manage (CAIB 2003 , p.138–139). Thus, a recurrent considerationin how work is managed is “what-this-will-mean-for-me-later-on” besides “what-can-I-do-about-it-now.” The organizational con-text of work, not just the facts of the case,affects reporting a mishap event (Button &Harper 1996, p. 277).

In summary, the view of rationality as“applying knowledge” can be adapted to fitnatural settings, but the goal of analysis mustinclude broad organizational factors thatinclude role, identity, values, and long-termimplications. The expert as agent (actor,someone in a social setting) is more than aproblem solver, but also an expert problemfinder, avoider, delegator, prioritizer, refor-mulater, communicator, and so on.

We have now considered several con-trasts between information processing andsocial-analytic concepts for understandinghuman behavior in natural settings. Buthow does one apply an analytic perspectivesystematically?

Unit of Analysis: The Principle ofMultiple Perspectives

A fundamental aspect of ethnography is totriangulate information received from dif-ferent sources at different times, includingreinterpreting one’s own notes in light oflater events, explicitly related to previousstudies and analytic frameworks (Forsythe1999, pp. 127–128). In conventional terms,to make a study systematic, one gathers datato model the work from several related dif-ferent perspectives:

� Flows: Information, communication,product

� Independent variables: Time, place, per-son, document

� Process influences: Tool, organization/role, facility, procedure



To provide a suitable social framing and orga-nization of these data categories, this sec-tion suggests the following units of analysis:activity system, temporality, and collectives.

Activity System

Activity theory (Leont’ev 1979) providesessential analytic concepts for understand-ing what is happening in a natural setting(Lave 1988; Nardi 1996; Engestrom 2000).Psychologically, activity theory suggests howmotives affect how people conceptuallyframe situations and choose problem-solvingmethods (Schon 1979). People broadlyunderstand what they are doing as identity-related activities (e.g., “exploring an Arcticcrater as if we were on Mars”). Career, social,or political motives and identities may influ-ence how procedures are interpreted andtasks enacted. Engestrom (1999) provides anexemplary activity theory analysis of a hos-pital setting.

Temporality: Phases, Cycles, Rhythm

A second unit of analysis is temporality:How does the work unfold during the courseof a day or a week? Does it vary seasonally?Is a given day typical? One might observean individual at different times and set-tings, and look for disparities between inter-views and what people say about each other(Forsythe 1999, p. 138). An essential, recur-rent organizing conception is the separatingof work into categories such as “‘someone-now,’ ‘me-when-I-can,’ ‘what-is-mine,’ and‘everyone’s-concern’ to prioritise . . . work”(Button & Harper 1996, p. 276). Thus,expertise transcends how individual tasks areaccomplished, to involve how time is madeaccountably productive.

Collectives

The third unit of analysis is the collective,the people who are interacting in a setting,as well as the conceptualized audience ofclients, managers, and the community ofpractice. The collective might consist of peo-ple who don’t directly know each other:“The occupational community [of photo-

copy machine technicians] shares few cul-tural values with the corporation; techni-cians from all over the country are muchmore alike than a technician and a sales-person from the same district” (Orr 1996,p. 76).

How is the study of a collective relatedto individual expertise? Lave (1988) con-trasts the view that culture is a collection ofvalue-free factual knowledge with the viewthat society and culture “shape the partic-ularities of cognition and give it content”(p. 87). Thus, the study of culture is insepa-rable from a study of how expertise is identi-fied, developed, exploited, organized, and soon. Orr’s study reveals that “The techniciansare both a community and a collection ofindividuals, and their stories celebrate theirindividual acts, their work, and their individ-ual and collective identities” (p. 143), suchthat storytelling has a social-psychologicalfunction with many practical and institu-tional effects.

Methods for Observationin Natural Settings

In considering methods of observation, oneshould not rush to the recording parapher-nalia, but first focus on how the study isframed, the nature of engagement of theobserver in the setting, and the work plan.This section of this chapter surveys usefulhandbooks, then summarizes key considera-tions and methods.

Handbooks for ObservingNatural Settings

The following handbook-style guides aresuggested for learning more about how toobserve natural settings. These fall on a spec-trum from observational science to rigorousengineering design.

Handbook of Methods in Cultural Anthro-pology (Bernard 1998) provides a balancedtreatment of the history and methodsof anthropology, with tutorial-style chap-ters on epistemological grounding, partic-ipant observation, systematic observation,



structured interviewing, discourse and textanalysis, and visual analysis.

Design at Work: Cooperative Design ofComputer Systems (Greenbaum & Kyng1991) is a primer of examples, theory, andmethods for participatory design. It repre-sents especially well the Scandinavian per-spectives that have defined change-orientedobservational studies of workplaces as amorally driven, industrially funded, and the-oretically grounded activity.

Contextual Design (Beyer & Holtzblatt1998) may be used as a beginner’sguidebook for conducting a “contextualinquiry,” including how to observe and workwith customers (with unusually detailedadvice about how to conduct interviews);how to model work (organizational flow,task sequences, artifacts such as docu-ments, culture/stakeholders, and physicalenvironment); and how to redesign work(including storyboards, paper prototypes).

Cognitive Work Analysis (Vicente 1999)provides another program for design-ing computer-based information systems,based on detailed mapping of informationflows, task constraints, and control pro-cesses. This book presents the method-ology and perspective of Jens Rasmussenand his colleagues (Rasmussen, Pejtersen, &Goodstein 1994): Work models must bedetailed for tool design, and hence obser-vation must be systematically organized tounderstand the domain (see also Jordan1996). In particular, analysis of fields –the physical-conceptual spaces for possibleaction – is generalized from observations ofparticular trajectories or behaviors in thisspace (Vicente 1999, p. 179).

Framing the Study: Purpose, Genre,Timing, and Biases

Every study of expertise occurs in its owncontext, which shapes the observer’s inter-ests, targeted product (a publication? adesign document?), and the pace of work.Researchers therefore find it useful to havea variety of different approaches that can beadapted, rather than imposing one rigorousmethod on every setting.

Observation of expertise in naturalsettings has been undertaken as a scientificendeavor (studying decision making, cre-ativity, etc.); to develop training strategies;or, typically, to redesign the workplaceby automating or facilitating the workprocesses (Blomberg et al. 1993 ; Nardi &Engestrom 1999; Jordan 1993 , 1997; Rosset al. Chapter 23).

Dourish & Button (1998) summarize therelation of ethnography and ethnomethod-ology to technological design, emphasizinghuman-computer interaction. Luff, Hind-marsh, and Heath (2000) provide anupdated collection of detailed workplacestudies related to system design. More gen-erally, workplace studies may be part of abroader interest in organizational develop-ment (Engestrom 1999; Nardi & Engestrom1999, p. 4).

Before a study begins, one should makeexplicit one’s interests, partly to approachthe work systematically, and partly to exposebiases so others may better evaluate anduse the results (e.g., provide a compara-tive survey of related studies, Clancey inpress). Throughout a study, one shouldalso question conventional metaphors thatpredefine what is problematic. For example,the term “homelessness” could lead to focus-ing on housing, rather than studying howsuch people view and organize their lives(Schon 1979). The underlying nature of asetting may clarify as change is attempted(Engestrom 1999, p. 78).

Observer Involvement

For many researchers, participant observa-tion is the ideal way to study people, infor-mally learning by becoming part of the groupand learning by watching and asking ques-tions. But participant observation is not nec-essary and may not be possible, for instance,in highly technical or risky work such as airtraffic control (Harper 2000, p. 258).

Observation should be a programmaticstudy (p. 240–241), with demonstratedsincerity and probity (p. 251). Ethnographyis not a haphazard hanging around or shad-owing, as if anything is of interest (p. 254).



Rather, the observational work must be a sys-tematic investigation, with some sequentialorder (though often dynamically replanned)that covers a related set of roles, places,situations, and timelines. For example, instudying MER rover operations mentionedpreviously, the researchers were confrontedwith a 24-hour operation in three floorsof a building, involving three shifts of dis-tinct engineering and scientist teams. Givenaccess constraints, the group focused on oneroom at first, where the scientists met threetimes during the day and worked out fromthat group and place to understand howinstructions were prepared for the rover’snext-day operations and then how Mars datawas received and stored for the scientiststo access.

To stimulate inquiry and make learningprogressive, the observer should keep a jour-nal and review it periodically for issues torevisit. Another method is to review pho-tographs and ask about every object, “Whatis that? What is it for? Who owns it? Whereis it used and stored?” This can be doneeffectively via email with colleagues whoare not at the study site, encouraging themto ask questions about what they see inthe photos.

The ideal in participatory design is to findat least one person in the setting who canbe a champion for the inquiry, explainingthe study to others, getting access, and mak-ing the observational activity legitimate. Bythis conception, people in the workplace arepartners in a cooperative activity, and notreferred to as “subjects,” “users,” or “opera-tors” (Wynn 1991, p. 54). Probably no otherphilosophical stance is more fundamental tothe observer’s success. Data are discussedwith the workers (in appropriate forums);report outlines are circulated for comment;related local expertise responsible for mod-eling the workplace is solicited for advice;documents about the work may even becoauthored with organizational champions.

Program of Work

For an observational study to be systematic,there must be an explicit program or plan for

what, where, and how to study the setting(Harper 2000, p. 248). For example:

� Map out the key processes of the organi-zation.

� Understand the diversities of work.� Understand how different sets of persons

depend on one another.� Determine salient junctures in the infor-

mation life cycle.

A plan will specify particular kinds of recordskept over a certain period, and how theywill be created, as described in subsequentsections.

Person, Object, Setting, Activity,Time-oriented Records

To be systematic, the observer must deliber-ately adopt a perspective and keep recordsorganized accordingly. Jordan (1996) sug-gests the perspectives person, object (e.g.,documents), setting, and task or process.More generally, an activity-oriented recordincludes any recurrent behavior, includingboth predefined work tasks (e.g., processingan order) and behaviors that may not be partof a job description (e.g., answering a phonecall). Time is an orthogonal dimension. Forexample, one could check to see what peo-ple in a work area are doing every 15 minutesor observe a given setting at the same timeevery day. Time-lapse video can be used torecord when people enter and leave a partic-ular place (Clancey 2001).

Anthropologists make a distinctionbetween two kinds of data: Emic categories(after phonemic) are used by participants;etic categories (after phonetic) are formaldistinctions from an analyst’s perspective(Jordan 1996). The basic systematic unitsmentioned in this section are etic: activities,roles, objects, persons, places, durations,etc.; in Western European and NorthAmerican business settings these often fitemic distinctions.

Study Duration

Observational studies may last from weeksto years. The duration depends on the



logistics and natural rhythm of the setting,technical complexity, and the study’s pur-pose. Generally speaking, long-term involve-ment is preferable to follow the devel-opment of work practice. However, afew months of regular observation can besufficient; a few weeks of daily participationusually enables a proficient analyst to forman understanding that can be a launchingpoint for more focused interviews and designsessions. Indeed, one aspect of a study is toidentify periodicities and historical develop-ments, that is, to locate observations withinoverarching cycles and trends.

Recording Methods and Logistics

Data from natural settings is recorded usingtools varying from paper and pen to elec-tronic tracking devices. The standard mediaare texts (e.g., field notes, documents foundin the setting), video and audio recordings,photographs, and computer models (e.g., theBrahms work practice simulation system,Clancey et al. 1998; Sierhuis 2001). Record-ing has enabled “repeated and detailedexamination of the events of an interac-tion . . . permits other research to have directaccess to the data about which claims arebeing made . . . can be reused in a varietyof investigations and can be re-examined inthe context of new findings” (Heritage 1984 ,p. 238). Having a body of such data is the sinequa non for being a researcher who studiesnatural settings.

Recordings must be labeled, indicatingat least the setting, date, and time. Expe-rienced ethnographers suggest the follow-ing procedures: Collect photographs in acomputer catalog, where they can be sortedby categories into folders. Transcribe fieldnotes (not necessarily journals) in an elec-tronic form so that they can be sharedand searched. Organize computer files infolders, separating preparatory/logistic infor-mation, miscellaneous graphics, documentsacquired, photographs, field notes, presen-tations and reports, press stories, email, andso on.

When recording outdoors, wirelessmicrophones can be used to avoid wind

interference. An audio mixer with severalmicrophones enables combining differentsources (e.g., computer speech output,“ambient” remarks, radio or telephone con-versations). Typically, observation revealssettings where interpersonal interactionoccurs, from which one chooses “hot spots”(Jordan 1996) for systematic video record-ing. The following methods are suggested:Use a tripod and wide angle lens, andmultiple cameras for different view pointsif possible. Take systematic photographicrecords (e.g., the same place each day, suchas a whiteboard) or take a rapid sequence tocreate a “film strip” that captures changingpostures and positions as people interactwith materials and each other. Interviewscan be audio recorded, but video (on atripod off to the side) provides moreinformation.

Written records can include a pocketnotebook (for jotting down phrases or not-ing things to do), a daily journal (oftenhandwritten) that describes one’s personalexperience, and field notes (perhaps usingan outline-based note-taker), with differentsections to elaborate on observations, raisequestions, and interpret what is happening.Surveys given before, during, and after obser-vation are recommended. View a survey as away to prompt conversations and to encour-age people to reflect with you on what isimportant, including their sense of account-ability and how they evaluate their own per-formance (see Clancey, in press). Finally, ifthe circumstances of privacy and intellectualproperty allow, one may learn a great dealfrom documents found in garbage cans.

Data Analysis

Experienced researchers suggest flexible useof computer tools for representing work(Engestrom 1999, pp. 85–90). Analysismethods are detailed in the handbooks citedabove. Key pointers are provided here.

First, video data must be inventoriedor will probably never be analyzed. Use aspreadsheet or outline to list the general con-tent for each recording, and as you watchloosely transcribe material of special inter-



est. For an extensive video collection of verydifferent settings, create a catalog of illustra-tive frames. Video to be analyzed should bereformatted if necessary with the time anddate displayed.

Social scientists often use some formof conversational analysis (CA), includinggaze and gestures (Heritage 1984 , p. 233).This method has revealed that behavior in“naturally occurring interactions” is stronglyorganized to great levels of detail. In pureform, CA eschews uses of interviews, fieldnotes, and set-up situations in real worldenvironments (p. 236). CA emphasizes“conversation as social action, rather thanas the articulation of internal mental states”(Dourish & Button 1998, p. 402 ; Whalenet al. 2004).

Video-based interaction analysis (Green-baum & Kyng 1991; Jordan 1996; Jordan &Henderson 1995) is a method for examiningdata in which scientists from different dis-ciplines may spend hours discussing a care-fully chosen, transcribed five to ten minutesegment.

Besides narratives and verbal analyses,data may be collected in spreadsheets (e.g.,time vs. person/place/activity), flowcharts,concept networks, timelines, and graphs(generated from the spreadsheets) (Clancey2001, in press). If the data have been gath-ered systematically, it will be possible tocalculate summary statistics (e.g., how longpeople did various activities in differentplaces). Such information may prompt fur-ther questioning and reveal patterns thatwere not noticed by the ethnographeron site.

Social scientists use a wide variety of met-rics. However, some studies never measureor count anything, as statistics are viewedmerely as an attempt to quantify everything(Forsythe 1999, p. 139) or as being misleading(Nardi & Engestrom 1999, p. 1). Researchersengaged in design projects are more likelyto seek a balance. The real issue is whetherthe measurements are meaningful (Bernard1998, p. 17). As a stimulus for furtherinquiry, it may be useful to quantify mem-bers’ concerns (e.g., “I’m interrupted toomuch”).

Perspective: Improving EthnographicPractice

Observation in natural settings is a valu-able, and some say necessary, way to sys-tematically learn about practical knowledge,that is, to understand how people, places,activities, tools, facilities, procedures, andso on relate. One can learn about techni-cal knowledge from textbooks or lectures,or even get important insights from surveysor by designing experiments in a labora-tory. But expertise has a subjective, impro-visatory aspect whose form changes withthe context, which is always changing. Thiscontext includes the workers’ conception ofpersonal and organizational identity (includ-ing motives and avowed goals), economictrends, physical environment, and so on.

Observation in natural settings may bearduous because of the time required, equip-ment maintenance, the amount of data thatis often generated, personal involvementwith the people being studied, and politi-cal and power concerns of the organizationalsetting. Some conflicts are inherent, with noeasy solution:

� Ethics, privacy, and confidentiality� Distribution and simultaneity of collec-

tive work� Long-cycle phases and off-hours commit-

ments� Representativeness and systematicity of

the data (vs. details of specific situations)� Exposing invisible work (e.g., practices

deviate from legally proscribed routines)� Point-of-view and authoritative biases

Using ethnography for design of work sys-tems is problematic: One often seeks a large-scale system design, but the study focuses onthe “small-scale detail of action” (Dourish& Button 1998, p. 411). Observation natu-rally focuses on what is; how does one moveto what might be? (see Greenbaum & Kyng1991; Dekker, Nyce, & Hoffman 2003 .)

Just like other work, ethnographies inpractice do not always measure up to theespoused ideal: “Social scientists have for



one reason or another failed to depict thecore practices of the occupational worldswhich they have studied” (Heritage 1984 ,p. 300). For example, the MER missionstudy was limited in practice by the num-ber of observers and their stamina. Out-side the defined workflow of mission oper-ations, the scientists were also participatingin parallel activities of grant writing, pub-lic affairs, paper preparation, and so forth.Some of these unobserved activities directlyaffected science operations (e.g., prepar-ing a comparative graph for a conferencepresentation might require additional datafrom Mars).

The role of simulation for driving obser-vation and formalizing data is unclear(Sierhuis 2001; Seah et al. 2005). As onedelves into individual behaviors of spe-cialists, are approaches recurrent or justidiosyncratic? To what extent does a col-lective have uniform methods? Should asimulation be broad (e.g., several weeks)or deep (e.g., modeling computer systeminterfaces)?

Finally, social scientists, like other work-ers, may find it difficult to articulate theirown methods: There is “no stable loreof tried and trusted procedures throughwhich, for example, taped records . . . can bebrought to routine social scientific descrip-tion” (Heritage 1984 , p. 301). Researchersoften have different disciplinary interests, soa group of ethnographers at one site mightnot collaborate until they write a reportfor the host organization. At this point,the problem of indexing and sharing databecomes visible, both within the group andto others seeking to better understand astudy. Effectively, in documenting observa-tional studies, the work practice researcheris caught up in all the familiar issues oflived work, accountability, and contingentmethods.

Acknowledgments

I am especially indebted to my colleaguesat the Institute for Research on Learning(1987–1997) for demonstrating through

their work the ideas presented here. TheMER Human-Centered Computing ethnog-raphy team from NASA/Ames includedRoxana Wales, Charlotte Linde, ZaraMirmalek (University of California, SanDiego), Chin Seah, and Valerie Shalin(Wright State University). Topic sugges-tions and editorial advice for this chapterwere also provided by Robert Hoffman,Patty Jones, Brigitte Jordan, Mike Shafto,Maarten Sierhuis, Marilyn Whalen, andJudith Orasanu. This work has beensupported in part by NASA’s Comput-ing, Communications, and InformationTechnology Program.

Footnote

1. The description of this project has been pro-vided by Peggy Szymanski at PARC.

References

Bernard, H. R. (Ed.). (1998). Handbook of meth-ods in cultural anthropology. Walnut Creek, CA:AltaMira Press.

Beyer, H. & Holtzblatt, K. (1998). Contextualdesign: Defining customer-centered systems. SanFrancisco: Morgan-Kaufmann.

Birdwhistell, R. (1952). Introduction to kinesics.Louisville: University of Louisville Press.

Blomberg, J., Giacomi, J., Mosher, A. & Swenton-Wall, P. (1993). Ethnographic field methodsand their relation to design. In D. Schuler &A. Namioka (Eds.), Participatory design: Princi-ples and practices, (pp. 123–155). Hillsdale, NJ:Erlbaum.

Blum, B. I. (1996). Beyond programming: To anew era of design. New York: Oxford Univer-sity Press.

Buchanan, B. G., & Shortliffe, E. H. (Eds.).(1984). Rule-based expert systems: The MYCINexperiments of the Heuristic ProgrammingProject. Reading, MA: Addison Wesley.

Button, G., & Harper, R. (1996). The relevance of“work-practice” for design. Computer SupportedCooperative Work, 4 , 263–280.

Chi, M. T. H., Glaser, R., & Farr, M. J. (1988).The nature of expertise. Hillsdale, NJ: Erlbaum.



Clancey, W. J. (2001). Field science ethnography:Methods for systematic observation on anarctic expedition. Field Methods, 13 , 223–243 .August.

Clancey, W. J. (2002). Simulating activities:Relating motives, deliberation, and attentivecoordination. Cognitive Systems Research, 3 ,471–499.

Clancey, W. J. (in press). Participant observationof a Mars surface habitat mission simulation.To appear in Habitation.

Clancey, W. J., Sachs, P., Sierhuis, M., & vanHoof, R. (1998). Brahms: Simulating prac-tice for work systems design. Int. J. Human-Computer Studies, 49, 831–865 .

Collins, H. M. (1997). Rat-Tale: Sociology’scontribution to understanding human andmachine cognition. In P. J. Feltovich, K. M.Ford, & R. R. Hoffman (Eds.), Expertise incontext (pp. 293–311). Cambridge, MA: MITPress.

Columbia Accident Investigation Board(CAIB). (2003). CAIB Report, Volume 1.NASA. [August 2005] http://CAIB.NASA.GOV/news/report/volume1/default.html.

Corbet, J. M., Rasmussen, B., & Rauner, F. (1991).Crossing the border: The social and engineeringdesign of computer integrated manufacturing sys-tems. London: Springer-Verlag.

Dekker, S. W. A., Nyce, J. M., & Hoffman, R. R.(2003). From contextual inquiry to designablefutures: What do we need to get there? IEEEIntelligent Systems, March–April, 74–77.

Dewalt, K. M., & Dewalt, B. R. (2002). Participantobservation. Lanham, MD: Rowman & Little-field.

Dourish, P., & Button, G. (1998). On“technomethodology”: Foundational relation-ships between ethnomethodology and systemdesign. Human-computer interaction, 13 , 395–432 .

Ehn, P. (1988). Work oriented design of computerartifacts. Stockholm: Arbetslivscentrum.

Emery, F. E. (1959). The emergence of a newparadign of work. Canberra, Australia: Aus-tralian National University, Centre for Contin-uing Education.

El Guindi, F. (1998). From pictorializing to visualethnography. In H. R. Bernard (Ed.), Handbookof methods in cultural anthropology (pp. 459–511). Walnut Creek, CA: AltaMira Press,

Engestrom, Y. (1999). Expansive visibilizationof work: An activity-theoretical perspective.

Computer Supported Cooperative Work, 8,63–93 .

Engestrom, Y. (2000). Activity theory as a frame-work for analyzing and redesigning work.Ergonomics, 43 , 960–974 .

Feltovich, P., Ford, K., & Hoffman, R. (Eds.).(1997). Human and machine expertisein context. Menlo Park, CA: The AAAIPress.

Fetterman, D. M. (1998). Ethnography (2nd Ed.).Thousand Oaks: Sage Publications.

Forsythe, D. E. (1999). “It’s just a matterof common sense”: Ethnography as invisiblework. Computer Supported Cooperative Work, 8,127–145 .

Goffman, E. (1969). The presentation of self ineveryday life. London: Allen Lane.

Greenbaum, J., & Kyng, M. (Eds.). (1991). Designat work: Cooperative design of computer systems.Hillsdale, NJ: Erlbaum.

Hall, Edward, T. (1966). The hidden dimension.New York: Doubleday.

Hall, Edward, T. (1959). The silent language. NewYork: Doubleday.

Harper, R. H. R. (2000). The organisation ofethnography: A discussion of ethnographicfieldwork programs in CSCW. Computer Sup-ported Cooperative Work, 9, 239–264 .

Heritage, J. (1984). Garfinkel and ethnomethodol-ogy. Cambridge, UK: Polity Press.

Hoffman, R. R. (1998). How can expertise bedefined? Implications of research from cogni-tive psychology. In R. Williams, W. Faulkner, &J. Fleck (Eds.), Exploring expertise (pp. 81–100).New York: Macmillan.

Hoffman, R. R., Coffey, J. W., & Ford, K. M.(2000). STORM-LK: System to organize repre-sentations in meteorology. Deliverable on thecontract, “Human-centered system prototype,”National Technology Alliance.

Hutchins, E. (1995). Cognition in the wild.Cambridge: MIT Press.

Johnson, A., & Sackett, R. (1998). Direct sys-tematic observation of behavior. In H. R.Bernard (Ed.), Handbook of methods in culturalanthropology. Walnut Creek: AltaMira Press,pp. 301–331.

Jordan, B. (1992). Technology and socialinteraction: Notes on the achievement ofauthoritative knowledge in complex settings.IRL Technical Report No. 92-0027. [December2004] http:// www. lifescapes. org/ Writeups.htm.



Jordan, B. (1993). Ethnographic workplace stud-ies and computer supported cooperative work.Proceedings of the interdisciplinary workshop oninformatics and psychology. Scharding, Austria,June. Amsterdam: North Holland.

Jordan, B. (1996). Ethnographic workplace stud-ies and computer supported cooperative work.In D. Shapiro, M. Tauber, & R.Traunmuller(Eds.), The design of computer-supported cooper-ative work and groupware systems (pp. 17–42).Amsterdam: North Holland/Elsevier Science.

Jordan, B. (1997). Transforming ethnography –reinventing research. Cultural AnthropologyMethods, 9, 12–17, October.

Jordan, B., & Henderson, A. (1995). Interactionanalysis: Foundations and practice. The Journalof the Learning Sciences, 4 , 39–103 .

Kling, R., & Star, S. L. (1998). Human centeredsystems in the perspective of organizationaland social informatics. Computers and Society,28, 22–29, March.

Lave, J. (1988). Cognition in practice. Cambridge:Cambridge University Press.

Leont’ev A. N. (1979). The problem of activity inpsychology. In Wertsch, J. V. (Ed.), The conceptof activity in soviet psychology, Armonk, NY: M.E. Sharpe, (pp. 37–71).

Lesgold, A., Rubinson, H., Feltovich, P., Glaser,R., Klopfer, D., & Wang, Y. (1988). Expertisein a complex skill: Diagnosing x-ray pictures. InM. T. H. Chi, R. Glaser, and M. J. Farr, (Eds.),The nature of expertise. (pp. 311–342) Hillsdale,NJ: Erlbaum,

Lorenz, K. (1952). King Solomon’s ring. New York:Crowell.

Luff, P., Hindmarsh, J., & Heath, C. (Eds.).(2000). Workplace studies: Recovering workpractice and informing system design.Cambridge: Cambridge University Press.

Nardi, B. (1995). Context and consciousness:Activity theory and human-computer interaction.Cambridge, MA: MIT Press.

Nardi, B., & Engestrom, Y. (Eds.) (1999). A webon the wind: The structure of invisible work.Computer Supported Cooperative Work, 8(1–2),1–8.

Newell, A., & Simon, H. A. (1972). Human prob-lem solving. Englewood Cliffs, NJ: Prentice-Hall, Inc.

Orr, J. E. (1996). Talking about machines. Anethnography of a modern job. Cornell UniversityPress.

PPS (“Project for Public Spaces”). [December2004] http://www.pps.org/info/placemakingtools/placemakers/wwhyte.

Rogoff, B., & Lave, J. (1984). Everyday cognition.Cambridge: Harvard University Press.

Rasmussen, J., Pejtersen, A. M., & Goodstein, L.P. (1994). Cognitive systems engineering. NewYork: Wiley.

Scheflen, A. E. (1972). Body language and thesocial order. Englewood Cliffs: Prentice-Hall.

Schon, D. A. (1979). Generative metaphor: Aperspective on problem-setting in social polity.In A. Ortony (Ed.), Metaphor and thought(pp. 254–283). Cambridge: Cambridge Univer-sity Press,

Schon, D. A. (1987). Educating the reflectivepractitioner: Toward a new design for teach-ing and learning in professions. San Francisco:Jossey-Bass.

Seah, C., Sierhuis, M., & Clancey, W. J. (2005).Multi-agent modeling and simulation approachfor design and analysis of MER mission oper-ations. Proceedings of 2 005 international con-ference on human-computer interface advancesfor modeling and simulation (SIMCHI’05),ISBN 1-56555 -288-1.

Senge, P. M. (1990). The fifth discipline: The artand practice of the learning organization. NewYork: Doubleday.

Sierhuis, M. (2001). Modeling and simulatingwork practice. Ph.D. thesis, Social Science andInformatics (SWI), University of Amsterdam,The Netherlands, ISBN 90-6464-849-2 .

Spradley, J. P. (1979). The ethnographic interview.New York: Holt, Rinehart and Winston.

Spradley, J. P. (1980). Participant observation. FortWorth: Harcourt Brace College Publishers.

Squyres, S. W., Arvidson, R. E., Bell III, J. F.,et al. (2004). The Spirit Rover’s Athena sci-ence investigation at Gusev Crater, Mars, Sci-ence, 305 , 794–800.

Star, S. L., & Strauss, A. (1999). Layers of silence,arenas of voice: The ecology of visible andinvisible work. Computer Supported Coopera-tive Work, 8, 9–30.

Suchman, L. A. (1987). Plans and situatedactions: The problem of human-machine com-munication. Cambridge: Cambridge UniversityPress.

Vicente, K. J. (1999). Cognitive work analysis:Toward safe, productive, and healthy computer-based work. Mahwah, NJ: Erlbaum.



Webster’s Third New International Dictionary,Unabridged. Merriam-Webster, 2002 . http://unabridged. merriam-webster. com (13 Dec.2004).

Wenger, E. (1998). Communities of practice:Learning, meaning, and identity. New York:Cambridge University Press.

Whalen, M., Whalen, J., Moore, R., Raymond,G., Szymanski, M., & Vinkhuyzen, E. (2004).Studying workscapes as a natural observationaldiscipline. In P. LeVine & R. Scollon (Eds.),

Discourse and technology: Multimodal discourseanalysis. Washington, DC: Georgetown Uni-versity Press (pp. 208–229).

Whyte, W. H. (1980). The social life of smallurban spaces. Washington, DC: The Conserva-tion Foundation.

Wynn, E. (1991). Taking practice seriously.In J. Greenbaum & M. Kyng (Eds.),Design at work: Cooperative design ofcomputer systems. Hillsdale, NJ: Erlbaum,pp. 45–64 .


146

P1: JZG052184097Xc09 CB1040B/Ericsson 0 521 84087 X May 22 , 2006 14 :47

C H A P T E R 9

Methods for Studying the Structure ofExpertise: Psychometric Approaches

Phillip L. Ackerman & Margaret E. Beier

“Psychometrics” refers to the scientific dis-cipline that combines psychological inquirywith quantitative measurement. Thoughpsychometric theory and practice pertain toall aspects of measurement, in the currentcontext, psychometric approaches to exper-tise pertain to the measurement and pre-diction of individual differences and groupdifferences (e.g., by gender, age) and, inparticular, high levels of proficiency includ-ing expertise and expert performance. Thescientific study of expertise involves sev-eral important psychometric considerations,such as reliability and validity of measure-ments, both at the level of predictors (e.g., interms of developing aptitude measures thatcan predict which individuals will developexpert levels of performance), and at thelevel of criteria (the performance measuresthemselves). We will discuss these basicaspects of psychometric theory first, andthen we will provide an illustration of psy-chometric studies that focus on the predic-tion of expert performance in the contextof tasks that involve the development andexpression of perceptual-motor skills, andtasks that involve predominantly cognitive/

intellectual expertise. Finally, we will discusschallenges for future investigations.

Before we start, some psychologicalterms need to be defined. The first terms are“traits” and “states.” Traits refer to relativelybroad and stable dispositions. Traits can bephysical (e.g., visual acuity, strength) orpsychological (e.g., personality, interests,intelligence). In contrast to traits, statesrepresent temporary characteristics (e.g.,sleepy, alert, angry). The second set ofterms to be defined are “interindividualdifferences” and “intraindividual differ-ences.” Interindividual differences referto differences between individuals, suchas the difference between the heights ofstudents in a classroom or the speed ofdifferent runners in a race. Intraindividualdifferences refer to differences withinindividuals, such as the difference betweenthe typing speed of an individual measuredat the beginning of typing class and thatsame individual’s typing speed at the endof a year of practice in typing. Studies ofthe development of expertise during skilltraining can focus on interindividual dif-ferences (e.g., the rank ordering of a group

147

Cap. 8

Documents

Transcript of Cap. 8