Field vs Lab

8/3/2019 Field vs Lab

1/30

A Com parison of Laboratoryand Field Research in the S tudyof Electronic Meeting SystemsALAN R. DENNIS, JAY F. NUNAMAKER. JR..and DOUGLAS R. VOGELA L A N R. D E N N I S is a doctoral candidate in M IS at the University of Arizona. Hereceived a Bachelor of Co mp uter Science from Acadia University and an M.B .A. fromQ uee n's University in Kingston, Ontario, and was a winner of the AACSB NationalDoctoral Fellowship. Prior to entering the Arizona doctoral program , he spent threeyears as a faculty member of the Queen's University School of Business. He haspublished several conference proceedings, book chapters, and journal articles (in,among others, MIS Quarterly, Information & Management, Data Base, DecisionSupport Systems, C omputers & Graphics, IEEE Transactions on Systems, Man andCybernetics, and International Journal of Ma n-Mach ine Studies). His current re-search interests include electronic meeting systems, decision support systems, andbusiness graphics.D O U G L A S R . V O G E L is Assistant Professor of MIS. College of Business and PublicAdm inistration, University of A rizona, Tucson. He has been involved with co mpu tersand computer systems in various capacities for over 20 years. He received his MS incom puter science from U CL A, and his Ph.D. in MIS from the University of Minnesota,where be w as also coordinator for MIS Research Center. His current research interestsbridge the business and academic communities in addressing questions on the impactof management information systems on aspects of interpersonal communication,group decision m aking, and organizational productivity.ABSTRACT: Research into the use of Electronic Meeting Systems (EMS) has rapidlyincreased over the past few ye ars. However, EMS laboratory experiments have oftendrawn very different conclusions about the effects of EMS use than have EMS fieldstudiesconclusions that at first appear inconsistent. By examining the differencesin the design of prior EMS studies, we attempt to better understand tbe factors thataffect the use of EMS techno logy. It is our contention that these differences in findingsare not inconsistent, but rather they reflect different situations that researchers havestudied. We identify 24 potentially important differences in organizational contexts,group characteristics, tasks, and EMS environ ments. In planning future re search , EMSresearchers need to make explicit design decisions for each of these aspects, and toconsider how those choices may affect research results. An analysis of these differ-ences suggests several approaches to enhance the design of future laboratory experi-ments and field studies.An earlier version of this paper was originally published in the Proceedings ofthe Twenty-ThirdHawaii International C onference on System Sciences (IEEE Computer Society Press, 1990).


2/30

tO8 DENNIS, NUNAMAKER. AND VOGEL

KEY WORDS AND PHRASES; electronic tneeting systems, decision support systems,research methodologies.ACKNOWLEDGMENT We would like to thank Terry Connolly, Joey George, ScottPoole, Daryl Sabers, Joe Valacich, Gary Wagner, Mary Anne Winniford, and BobZm ud for helpful comm ents on earlier drafts of this paper. This research w as partiallystjpported by the IBM M anagement of Information Systems Prog ram, with additionalfunding from the Social Sciences and Hum anities Research Council of Canad a.IntroductionIN RECENT YEARS, THERE HAS BEEN A GROWING INTEREST in the use of informationtechnology lo support group w ork. As the scope of problems supported and the natu reof this technology h as evolved, it has been known by m any names . W e prefer the termElectronic M eeting System (EMS) [9]. Curre ncy, the results from different types ofEMS research tell us different things. As shown in the next section, many experim entsin laboratory settings have found that, while the use of F.MS software can improve thequality of meeting o utcom es, it can also require slightly longer meeting times and leadto decreased participant satisfaction. In contrast, many field studies of EMS use havefound that the use of EMS produces high er quality o utcom es in less time, with highlysatisfied participants. At first glance, Lhese results appear inconsistent. However, it isour contention that they are not inconsistent, but rather reflect the different contex ts,groups, task s, and EMS that researchers have studied.

The purpose of this paper is to examine previous laboratory ex perimen ts and fieldstudies to try to understand w hat may account for differences in findings. By exam-ining the differences in contexts, group s, tasks, and EMS among prior studies, we canidentify som e of the factors that poten tially influence th e effects of EMS use. Thisanalysis should provide four b enefits. First, it will help researchers and practitionersunderstand the bou nds to which specific research findings can be generalized. Sec ond,it will encourage researchers to documen t critical aspects of their studies, so that resultsfrom different studies can be better comp ared. Third , it will highligh t factors that needto be addressed in the future emp irical evaluation and developmen t of EMS technology.Finally, it will begin to lay the foundation for research leading to the developm ent oftheories of EMS.

The next section briefly review s previou s EMS laboratory and field studies. The thirdsection h ighlights the key design differences between these two classes of study, andconsiders how these differences could account for the contrasting fmdings. The fmalsection d iscusses implications for researchers.P r e v i o u s E M S R e s e a r c hIN TfflS PAPER, WE FOCUS EXCLUSIVELY ON PUBUSHED RESEARCH (a s of th e Su m m erof 1990) thatevaluatestheeffectsof using EMS meeting rooms where groups m eet atthe same time and place. Obviously, this excludes some valuable dissertations and


3/30

LABORATORY VS. FIELD RESEARCH ON EM S 109

was not the evaluation of EMS technology [31]. Our focus on EMS meeting rootns isnot to suggest that all such meetings are similar; indeed, as we shall sec in the nextsection, such meetings have varied considerably among studies. The purpose of thissection is to provide a very brief review of the findings from previous research (seealso [9 , 41 , 53]).

There are a variety of research methodologies available to researchers in theManagement Information Systems (MIS) area, as recent discussions in MIS joumalshave highlighted [cf. 3 ,1 6 ,2 0, 3 4 ]. To date, laboratory experiments and field studieshave been the most commonly used methodologies in the study of EMS. The applica-bility of conc lusions from laboratory studies to the organizational u se of Mis has beenchallenged by field researchers, as the laboratory setting, the use of students assubjects, and the nature of experimental tasks are often different from those encoun-tered in the organizational use of MIS. Ga lliers and Land note that "although theexperimental design of such is research may well be academically acceptable andintemaliy co nsistent, all too often it leads to inconclusive or inapplicable resu lts" [20,p. 90 0]. T he question is how to interpret behavior exhibited in laboratory se ttings andapply conclusions from it to behavior in natural settings. That is, while laboratorystudies may have intemal validity, they may lack extemal validity.

In contrast, the difficulty in attributing the observed ou tcomes to the use (or nonuse)of MIS technology in field settings has caused the use of field studies to come underfire. Th e different stand ards of evidence used in field studies have been ch allenged byexperimentalists: "The ideal of science is the controlled experiment. . . . The mainreason for the preeminence of the controlled experiment... is that researchers canhave more confidence that the relations they study are the relations they think theyare" [40, p. 293]. Field studies typically involve the examination of open systems;that is, systems that perm it other factors n ot related to the research design to affect theobserved outcomes [7]. Separating the effects of the factors of interest from those ofother factors can he difficult. That is, while field studies may have extemal validity,they may lack intemal validity.

In this paper, we define EMS field studies as stud ies of the use of EMS technologyby organizational groups (public or private sector) addressing problems of their ownchoosing at an EMS facility at their organization or another institution, wh ere data arecollected to develop theory or evaluate research p ropositions. Laboratory experimentsinvolve groups using EMS technology to address tasks prescribed by a researcher forthe purpose of exam ining the effects of the use of the technology, often using studen tsubjects. The field experim ent by Jarvcnpaa et al . [36] largely parallels this design andis grouped with the laboratory experiments.Laboratory ExperimentsExperimen tal research co nducted in EMS meeting room s has produced a variety ofresearch findings for effectiveness, efficiency, and group member satisfaction (seeTable 1). Studies that did not measure one or more of these constructs are indicated


4/30

110 DENNIS. NUNAMAKER. AND VOGEL

Table 1 Outcom es (Compared to No EMS Support)Experiments

[4] BuiEffectivenessincreased

[5] Chidambarum n/a[6] Connolly[8] Dennis

[12] Dennis[13] DeSanctis[17] Easton[18] Baston[21] Gallupe[22] Gallupe[23] Gallupe[24] George[30] Ho[34] Jarvenpaa[37] Jessup[38] Jessup[43] Um[44] Loy

n/an/an/an/aincreasedn/ano changeincreasedno changeno changen/aincreasedn/an/an/an/a

[58] Sambamunhy n/a[59] Sharda[63] Steeb[64] Valacich[65] VanShaik[68] Watson[69] ZigursField studies

[2 ] Adelman[10] Dennis[11] Dennis[47] McCartt[50] Nunamaker[51] Nunamaker[52] Nunamaker[64] Valacich[66] Vogel[67] Vogel

increasedno changen/ano changen/an/aEffectivenessincreasedincreasedincreasedin/decreasedincreasedincreasedincreasedincreasedincreasedincreased

Efficiencydecreasedrt/an/an/an/an/adecreasedn/an/ano changedecreaseddecreasedn/an/an/an/an/anAin/adecreaseddecreasedn/an/an/an /aEfficiencyincreasediiKTeasedincreasedin/decreasedincreasedincreasedincreasedincreasedincreasedincreased

Member satisfactionin/decreasedn/an/an/an/an/aincreasedn/ano changedecreasedno changedecreasedn/ano changen/an/an/an/an/an/aincreasedn/an/adecreasedn/aMember satisfactionincreasedincreasedincreasedin/decreasedincreasedincreasedincreasedincreasedincreasedincreased

Some experimental studies found the use of EMS to improve meeting effectiveness(e.g., decision qtiality) [4,17,22,34,59], while others found no effect on effectiveness


5/30

LABORATORY VS. HEL D RESEARCH ON EMS U I

[22]. Som e studies found that EMs use increased grou p mem ber satisfaction [4, 17.63] or decreased satisfaction [4 ,2 2 ,2 4, 68 ] or had no effect [2 1, 2 3, 34 ]. It is difficultto see any clear pattem in these findings.

Field Study ResearchCom paratively fewer field studies have yet reached pubhcation (see Table 1) [2 ,1 0,11 ,47 ,50 ,51 ,5 2, 64 ,66 ,67 ]. None of these studies have directly compared EMS- andnon-EMS-supponed groups, although most have asked participants to com pare the twoindirectly. Many of these field studies could also be tenned action research, as theresearchers w ere also participants (i.e., the facilitator). In these cases, the researcher(s)had two objec tives: to develop a better understanding of the effects of EMS technology,and to assist the group in performing its task. An exception to this are two studiesreported by Nunamaker e ta l . [52] and Vogel eta l. [66] in which researchers designedthe data collection plan and instrum ents, but played no role in the mee tings. In vin uallyall cases (some exceptions are noted in McCant and Rohrbaugh [47]), participantsfound the use of EMS technology improved meeting effectiveness, efficiency, andmember satisfaction. In most cases, these time improvements were postmeetingperceptions, but in one case [52] they were differences between premeeting timeestimates and actual meeting times. The pattem here is clear: field studies indicateimprovements in effectiveness, efficiency, and satisfaction for EMS use.

Key Differences in Des ignI N T in s SECTION, WE ffiGHUGHT DESIGN DIFFERENCES IN PREVIOUS RESEARCH. A fe wof the differences we identify are inherent and caused by the choice of researchmethodology, whether laboratory or field. Many other differences are not inherent,but rather reflect design decisions quite amenable to change in futtire studies. Forexample, in the following sections we note that previous laboratory experimentstypically studied the use of EMS technology by small groups addressing operational-level decisions, while fteld research typically studied larger groups addressing strate-gic-level problems. There is no inherent reason for this focus; future studies couldquite easily have a different focus. As we will show be low, there is reason to expectthat altem ative focus may yield different fmdings.

We have argued that the processes and outcomes of group woric depend on theinteractions among four groups of variables: organizational context, group character-istics, task, and EMS technology [9] . W hat variables within these four groups havesignificant effects remains an open question. We have selected 24 variables; clearlythere are many others that could be considered. How ever, the number of variables alsoprecludes m eta-analysis as there are too many variables for the number of studies. Ourselection was motivated both by theoretical arguments (i.e., there is some a priorireason to suspect that the variable might affect pro cesses and outcom es) and em pirical


6/30


7/30

LABORATORY VS. FIELD RESEARCH ON EMS 113

Organizational Context"The embedding organization is a crucia] part of any work gro up's c ontext, and thepattem of mutual inter-depcndence between the group and the surrounding organiza-tion is a crucial part of the substantive phenom ena to be explored" [49, p. 385]. Wenote four contextual factors that may be impo rtant (see Table 2). First, organizationalculture and behavior norms serve as a guide to the meeting process for organ izationalgroups in field studies. Norm s may be lacking in laboratory groups formed for thepurpose of an experiment; an assembled group of individuals m ay be a group in body,but not in spirit. In preexisting experimental groups, contextual norms may simply bedifferent from the norms of organizational groups. For example, in one laboratoryexperiment using preexisting groups, the norms of one group were such that it choosenot to use the EMS [69]. Second, in organizational groups, group members haveincentives to perform. Accomplishing the task successfully means recognition andreward for the group. In some experim ents, where the tasks are such that performancecan be measured objectively, this has been provided by pay and incentives based onexperimental performance [e.g., 22].

Third, m embers of organizational groups may not always have consistent goals andobjectives; there may be political elements such that the "best" outcome for somegroup member(s) is different from that for other group member(s). Tasks in experi-ments have traditionally presumed the rational model, "where organizational deci-sions are consequences of organizational units using information in an intendedlyrational manner to make choices on behalf of the organization" [32, p. 3], althoughsome [e.g., 68] have involved what are essentially barg aining tasks that have no rightanswers.

In field studies, objectives have not always followed the rational m ode l. They h aveoften had a political com ponen t, "where organizational decision s are consequ ences ofthe application of strategies and tactics by units seeking to infiuence decision processesin directions that will result in choices favorable to them" [32, p. 3]. Thus the goals(or incentives) for participants in field studies have not always been congruentor-thogonal or in direct competition on occasionin sharp contrast to the goals ofparticipants in most laboratory studies, whose goal has often been to make the "best"decision for the organization in the experimental case.

Finally, for organizational grou ps, issues and problem s are interrelated. "Basically,every real world problem is related to every other real world problem" [46, p. 4,original emphasis]. Thus every time an organizational group attempts to resolve aparticular problem, it needs to consider the problem's potential relationship with allother problems. This generally has not been a concem for groups in laboratoryexperiments.

In summary. Table 2 shows that most previous laboratory experiments haveexamined student-related organizational cultures without perfonnan ce incentives butwith common objectives without interrelated problems. Most field studies have


8/30

114 DEN NIS. NUNAMAK ER, AND VOGEL

experiments finding EMS use to improve performance provided incentives to subjectsor studied a nonstudent organizational culttire.

Group CharacteristicsThere have been many differences between the groups used in experimental researchand those in the field that may accoun t for differences in fmdings; w e focus on seven(see Table 3). First, most experimental groups have been com posed of students, whileorganizational groups have been composed of m anagers and professionals. Individualcharacteristics of the two populations may be different. For example, one individualcharacteristic of interest is prior experience w ith grou p m eetings; most manag ers andprofessionals w ill have had m ore experience with group meetings than undergradu-ates. While the use of senior and junior undergradtiate students in experimentalresearch is a common practice, studying one population (e.g., undergraduates) mayprod uce different res ults than studying another (e.g., mana gers and professionals) [7] .Based on a review of 32 (non-Mis) studies comparing undergraduates to managers,Go rdon , Slade, and Schm itt [26] conclude that there are signiftcant differencesbetween undergraduates and managers. In M I S , Remus [55, 56] has also found thatthere can be important differences betw een m anagers and underg radua tes in decision-making processes and performance. Of course, all groups of students or groups ofmanagers and professionals are not the same; some undergraduates may have priorwork e xperience, w hile participants in field studies may be recent graduates with littleor no experience. Reporting subject dem ographics becomes important.A second potential factor is the familiarity of group members with the task.M emb ers of organizational groups typically have had more experience with the task,as, in general, they address the tasks faced in the ongoing management of theorgan ization. In con trast, experim ental group s have often had less familiarity with thetask they have been assigned.

Third, experimental groups have typically been ad h oc, formed for the sole purpo seof the experiment, and have no past history or foreseeable future. Field studies havetypically used established groups, for whom the meeting under study is just one in along series of meetings. Participants in field studies need to associate with each otherand live witb the meeting outcome long after the meeting is over, in contrast withmany experimental groups. Differences have been observed between ad hoc andesiablisbed/ongoing groups in non-EMS-supported research [e.g., 28] and in EMSresearch [8].

Fourth, groups in experimental studies have generally been peers. While someexperiments have studied the effects of an emerging leader or have temporarilyassigned a leader for the duration of the experimental session [e.g., 24], this form ofleadersh ip can be different from that in organization s. Gro ups in previous field stud ieshave gen erally had a distinct hierarchy and/or differing social status am ong me mb ers;the leader w as the leader before, dur ing, and after themee ting. The leader(s) promotesand rewards participants. His/her presence was felt during the meeting, although


9/30


needs than groups in which participants have similar power and status. Different groupstructure and leadership has been shown to affect non-EMS-supported mee tings [60].

Fifth, participants in experiments have often been fyst-time users of the EMStechnolo gy. Pa rticipants in field studies have also often been fu-st-time users, but bythe end of the study they have often logged many hours of use; they are moderatelyexperienced EMS users. Observations drawn from the study of inexperienced users ofEMS technology may be useful, but may apply only to inexperienced users. General-izing the results of these studies to the ongo ing organ izational use of EMS may not beappropriate.

Sixth, researchers have speculated that EMSmay prove more effective for largerrather than smaller groups [9 ,14 ] . Most experimental research to date has focused onsmall groups (often 3-6 members). In contrast, most field study groups have beenlarger (typically 10 or more m em bers). Group size, of course, has been shown to havesignificant im pacts on non-EMS-supported g roup w ork [e.g., 29 ].

Finally, we consider the logical size of the group in addition to the physical size ofthe group. Groups can be considered logically small if there is a high overlap in theparticipants' domain know ledge and skill; while there may be m any participants, theircombined knowledge and skills do not extend very far beyond those of the mostknowledgeable and skilled person in the group. Logically, large groups have lessovCTlap in knowledge and skills; thus the group has a wider range of knowledge andskills than any o ne person in the group. The overlap or lack of overlap of skills, traits,and ab ilities has been show n to hav e different effects in studies of non-EMS-supportedgroup work [e.g., 60] particularly in the pop en sity to participate [15].

In organ izational gro up meeting s, participants have typically come from differentareas. They bring different sk ills and domain know ledge that can be used by the groupin resolving the issue at hand. Organizational groups have therefore been logicallylarge. In contrast, experimental groups have often been logically small, having beendrawn from one population, typically mem bers of an MIS course or other course in th ebusiness school.

In summ ary. Table 3 shows that the majority of previous laboratory experimentshave studied physically and logically small ad hoc groups of undergradtiate students,without a formal hierarchy, who were unfamiliar with the task domain and the EMS.The majority of field studies have examined physically and logically medium- andlarge-sized established groups of managers, with a fonnal hierarchy, who werefamiliar with the task domain and had more experience with the EMS.The TaskThe task performed by groups is "an especially important variable, often accountingfor as much a s 50 percent of the variation in group performance" [54, p. 88]. We noteeight potentially important differences in task (see Table 4). The fu^t is the type oftask. There are many useful ways of classifying type of tasks [e.g., 48] , such as choice


10/30

116 DENN IS, NUNAMA KER. AND VOGEL

the type of tasks between experimental groups and organizational groups that mayhave had a major impact on the conclusions drawn from different studies.

Second, task complexity has been shown to influence group member participationin non-EMS-supported group work [e.g., 15]. Experimental tasks must be appropriatefor the experimental group and the time for which they have agreed to participate.Most tasks last one sessionat most a few hours. In contrast to the tasks in experi-mental sessions, tasks faced by organizational groups have often been "wickedproblems" [46,57] . Wicked problems cannot be exhaustively formulated and writtendown on paper. They have no end; the task is never complete. They cannot bereproduced and are essentially unique. They do not have an enum erable set of potentialsolutions or operations that could be included in the solution plan. In many ca ses, theproblem cannot be stated separately from the solution, as understanding the task issynonym ous with solving it. Finally, there is no imm ediate and no ultimate test of the"goodne ss" of a solution. W hile these tasks occur in field studies, they have typicallynot been studied in experimentsfor obvious reasons!

Third, this complexity is often reflected in the time required to complete the task.Tasks faced by organ izational grou ps in prior field studies have often required severalsessions to address, wh ile those in the laboratory have typically required seve ral hou rsat most.

Fourth, the clarity of the task has often varied among studies. In many cases,participants in organizational groups, particularly logically large groups, do not havea common understanding of the problem or may hold different assumptions about it.A group member's assumptions about, or conception of, the situation may differsignificantly from the actual situation and from that of other group members [1,46] ,which can have a significant effect on the outcomes. However, in laboratory experi-ments, participants have typically read the same task instructions, and thus have beenmore likely to share common assumptions and conceptions.

Fifth, similar differences have been common in the symmetry of informationpertaining to the task. Members of organizational groups in the field have often hadasymmetrical (even conflicting) task information. Experimental groups, who in pre-vious EMS studies have generally worked from a common description of the experi-mental task, would have been more likely to have a common set of infonnation.Non-EMS-supported research has show n that interventions into the group process havedifferent effects, depending upon whether all group members have the same informa-tion or different infonnation [e.g., 27].

Sixth, the infonnation management needs of experimental groups and organiza-lional groups typically have been different. Experimental tasks have tended to be"clean," well-defined tasks that arc neatly packaged and small in scope. All informa-tion is generally prov ided in one sou rce, a case package. In contrast, the tasks used byorganizational groups are usually ill-defined and "messy" [1,46]. "M essine ss" in thissense refers to the degree that the needed information comes from many fragmentedsources with varying degrees of objectivity and accuracy. Data may be intemal


11/30

LABORATORY VS. FIELD RESEA RCH ON EMS 117necessary rep orting of informatioti to experimental participants rem oves their need toaccess these messy data from mu ltiple sources a need often keenly felt by organiza-tional groups, and thus effects on processes and outcomes may have differed.

Seven th, with organizational gro ups there is a need to share information b eyond thegroup to other individuals and groups in the organization. Virtually all organizationalgroups have taken minutes of meetings for future reference. As a rule, experimentalgroups don't. The EMS helps ensure key information that will form the basis of themeeting's minutes is recorded error free, without requiring one of the participants todevote time to the recording processand potentially lose opportunities to participate.As the EMS record s this infonnation, it also promo tes greater accu racy; a personrecording information from which the minutes will be prepared has a greater chanceto mis-hear or misinterpret spoken com men ts.

Finally, the information needs of organizational groups have usually includedinformation from previo us m eetings, whether EMS-supported or not. As more than onemeeting may b e needed to address the task, it becom es m ore important to manage theinformation arising in sessions to ensu re that it is accessible in subsequent ses sions. Itmust be available for members to refer to, and it must also be available to memberswho have missed prior meetings to ensure they are briefed on the missed session(s).These information managem ent needs have typically not been addressed by experi-mental tasks requiring one m eeting.

In summary. Table 4 shows that the majority of laboratory experiments haveexam ined low-co mp lexity, high-clarity, choice tasks with high information symm etryand few information sources, requiring one to two hours to perform, and no informa-tion to be shared from previous meetings or to subsequent meetings. The majority offield studies have examined medium-to-high complexity generation and planningtasks of varying clarities with low to medium information symmetry and manyinformation sources requiring 1-2 days to perform, with no information shared fromprevious m eetings and to subsequen t meetings . The laboratory studies finding g reatereffectiveness with EMS use were more likely to have used tasks including somegeneration activities of higher complexity that required longer to perform.EMS EnvironmentIf we examine the EMS environments used in previous studies, we see at least fourdifferences reflecting the different EMS software used at various research institutions(see Table 5). First, an EMS environm ent can prom ote at least three different styles ofmeeting, each of which may have different effects: a chauffeuredprocess, a supportedprocess, and an interactive process. In using these term s, we refer to styles of meetingprocesses that combine differing amounts of verbal and electronic communication.Each style can be combined with the others and with non-EMS-supported verbaldiscussion at different stages of any meeting.

With a chauffeured process, only one person uses the EMS, either a group memberor a meeting leader/facilitator. A workstation is connected to a public display sc reen,


12/30

118

!3UI

^ .2 E E o of f

g g S8

a:-I.g

'S io ^ o ^ 0 < S o O 0 0 0 O O < U O O O ( S e 3rt>sE3Bc:eccn>iBBe^>-.

I I E . 2 E . 2 - 2 - 2 . 2 . 2 . 2 . 2 . 2 E E - 2 - 2 . 2 -au :2a -^ 3

V) in a> w M uj24 M 4 ^ > ^ K ^ H N - ^ 1 ^ 1 ^U ) bJD ^ dO Ofi Dfl &J) OU bJ) ~


13/30

119

l A .^ c n E AI

u


14/30

120

II I I

g o o o o o o o o o o p g o S o p o o p g o

i ; !-i o o E fe E o o o o o o E E E o o o o o

S Ea Ea

^ .c^ .^ .g -o ^ ^^ 2 :a 3 E 3 E4J

w: E Eis -a ( N O31 if '5 -Si I 'E 'E -r 'o o o o o

o .S ,JP E E S .E:i ro m m" ^ OC T v " ^ ' - - H

sE

S . 2 Eo.a

S u u u uM IA lA nO O O O OF M F ^ " C " C rCU U O U U

oou

o

III 3^ 3J 3^ uE- & g- w) (^


15/30

O O O Oc c c c

4 ^ 5 E E E E E E E E E E

121

o j= j= j=U DO 00 00E X 2 S

1 . T 3 T) TS -o -o -a T3W U ^ O O O 4Jf f f f f ? ^V . 2 . 2 . 2 . 2 . 2 . 2 . 3 . 2 . 2 . 3-q j= x:

oi) oi) oj) 00 00 oi) oi)

E S . 2 . 2 . 2 . 2 . 2 . 2 . S E

o7 .so 55 j ^ - ^ - ^ f O - ^ - ^ f S O O f O O O

I E2gOi)

u'=4ja

QJ Su . GU

g g

c - 5 1 y 3 3 ^ 0 0


16/30

122

"B

l

1 11 1 11tA M IA t/1

a, c c c

o -p o - p - p^ ^ .L ^ ^K 4 ^ . ^ Jfc^

9

a a to Vat > > >J? a -a -aq u ow a s a.S .5 .5 .S


17/30

123

I

a -S iS -S iS s s s sa -a o -p o o o p o

ao BO BOa 3 s

3o

>ic

a.s

uaM

< ^.53

J=

x :

.s

3oo.

.s

J 5

a.

.s

3

(A

S6

3 y Q J 4 J13 o o


18/30

124 DEN MS, NUNAHAK ER. AND VOGEL

and structure information. A supported proce ss is similar to a chauffeured meetingproce ss, but differs in that each mem ber has a com puter w orkstation that provides aparallel, anon ymo us electronic comm unication chan nel. The electronic blackboard isstill used to present and structure information, but with members able to add itemselectronically from their wo rkstations. The meeting proceeds using a mixture of verbaland electronic interaction. With an interactive process, the parallel, anonymouselectronic communication channel is used for almost all group communication.Virtually no one speaks. While an electronic blackboard may be provided, the groupmemory is typically too large to fit on a screen, and thus it is maintained so that allmem bers can access it electronically at their wo rkstations.

Second, the degree of structure provided by the EMS in terms of the process and,third, the task, has also varied. Som e studies have provided high proce ss structure, inwhich the groups followed a predetermined agenda using predetermined software,while in others group members have been free to select their own agenda and tools.Som e studies have provided high task structure (e.g., providing mathem atical m odelsor conceptual techniques) to assist the group in resolving the task, while others haveprovided no task structure.

One final difference has been facilitation. Laboratory experiments have typicallyused EMS environm ents w ithout facilitators or with only a technical facilitator who serole was essentially passive (fearing possible confounding effects of the facilitator ontreatments). Field studies have typically examined environments in which the facili-tator took an active role as an integral part of the normal EMS processas importantto the outcome of the meeting as any other aspect of the EMS technology, such asindividual workstations or electronic blackboards. Indeed, some EMS environmentsare based on the prem ise that the facilitator is more important to the normal operationof the environment than individual w orkstations [e,g. ,47]. The perceptions of partic-ipants suggest that use of a facilitator can affect meeting ou tcomes at least as much asany other component in the EMS environment.

In summary. Table 5 shows that the majority of laboratory experime nts used highlystructured processes with more verbal communication (i.e., supported or chauffeured),with less task structure, and with a passsive facihtator or no facilitator. T he majorityof field studies used moderately or highly structured processes w ith a more even mixof verbal and electronic comm unication (i.e., interactive, supported, and chauffeuredprocesses) w ith a variety of task structure and an active process facilitator.

I m p l i c a t i o n sTHE FIRST AND MOST BASIC IMPUCATION PROM THESE DIFFERENCES is that it isimportant to acknowledge explicitly the limitations of conclusions drawnfromEM Sresearch. Results from both experiments and field studies should not be freelygeneralized to all organizational contexts, all groups, all tasks, all F-MS environments.Research m ust report sufficient contextual, group, task, and EMS environment infor-


19/30

LABORATORY VS. HE LD RESEARCH ON EMS 12S

Second, we have proposed 24 differences that may (or may not) have importantimpacts on the processes and outcom es of EMS-supported m eetings. The im portance ofeach of these differences remains an em pirical question. Do any or all have significantimpacts on effectiveness, efficiency, or satisfaction in EMS-supported group work? Th eneed (or, conve rsely, the opportunity) for future research is enorm ous.

Finally, we beUeve that both laboratory and field researchers need to considerexplicitly these issues in designing future research, as these factors have the potentialto change research fmdings. The key for experimental researchers is to think of thelaboratory experiment as a challenge, as an opportunity to mode! the "real" world oftheir chosen target environment a s closely a s possible, in order to m aximize the abilityto generalize findings to the organizational use of EMS. While it has been argued thatsuch "extemal validity" issues are unimportant in theoretically motivated research [7],we disagree. Instead, we have argued that processes and outcomes depend upon theinteraction among many variables. Such an interactionist perspective argues thatsupport for a theory developed in one environment provides no evidence that that theorywill hold in a different environment; indeed, the opposite may be true (e.g., classi-cal/Newtonian physics versus subatomic/quantum theory). In our opinion, developingsuch context-free theories is inappropriate, as few such universal "truths" ex ist.

In contrast, the inherent problem in field research is the lack of con trol, and the lackof precision and accuracy in measurement. The challenge for EMS researchers con-ducting field study research is to improve these issues, e.g., by the increased use ofrandom assignment and "hard" performance measures rather than "perceived" mea-sures. Several recent papers have presented insightful observations about the designof MIS field studies [42, 45, 70]. Rather than take the general perspective of thesepapers, we will focus on issues particularly relevant to EMS field studies.

In the rest of this section, we present the im plications for design d ecisions in futureresearch (see Table 6). The se decisions are listed under seven heading s, correspondingto some degree with the areas identified previously: building groups, established/on-going groups, group size, task, information management, incentives, and EMS envi-ronment. We expect that other researchers with different interests will draw otherconclus ions and choose different altem atives for these research design decision s. Theimportant point is that these design decisions should be made explicit and that thechoices are appropriate for the goals of the researcher within the constraints faced.Building GroupsIdentifying a pool of subjects to form experimental groups is often an early step in theresearch design of a laboratory experim ent Ideally, subjects will be the same managersand professionals who could use EMS in the organizational environment. Unfortunately,this is not usually a practical altemative; of necessity, Uie use of student subjects is acommon practice. Work by Gordon et al. [26] and Remus [55, 56] suggests that, ifstudents are to be used, the use of more experienced, more m ature students (e.g., graduatestudents) is strongly preferred to less experienced, less mature students (e.g., typical


20/30

126 DENNIS, NUNAM AKER, AND VOGEL

Table 6 Key Design Decisions and Some Options

Laboratory experimentsBuilding groups Use a diverse subject pool Use more mature graduate students Build logically large groups

Establishedlongoing groups Use existing groujK that hav e a past

and a future of working togetherGroup size

Study larger groups

Task Use preexisting tasks with which

the subjects are familiar Use very information-rich cases

Information management Provide multiple information sources Provide different information to

different subjects

Field studies Attempt to find and study "contior' groups Select groups with desired characteristics Understand group me mb ers' backgroun ds

Study established/on going projects aswellas individual meetings

Study smaller groups Examine fit of the group, task, technology

Track task characteristics Examine participants'task comprehension Monitor participa nts'interest

Provide information integration andmonitor usage

EM S environment Use environments appropriate for Examine differences in technology across

organizations sites Monitor the type of meeting process. Monitor the type of meeting process,

degree of task and process stmcture, and degree of task and process structure, andthe level of facilitation the level of facilitation

Incentives Give incentives tomotivate subjects Give different ince ntives to different

subjectsUnderstand organizational incentives

subjects. StudyinggroupsofundergraduateMlSorbusiness Students, for example, tnaynot lead to the same conclusions as studying diverse groups whose training, experi-ence, and interests are not exclusively in MIS or business. Drawing subjects from anarrowly defined population may also lead to the formation of groups whose membershave the same basic skills and knowledge domain; in other words, it may lead to theformation of logically small groups, not the logically large groups often found inorganizations. One solution may be the use of stratified random sampling from several


21/30


subject from an organizational behavior course, one from an engineering course, andso on. This has the added benefit of increasing the generalizability of the study [7],but it also has the potential to increase the error variability of statistical measures [40 ].Another step toward providing an experimental context more similar to that oforganizations is to give each subject different information about the case, to betterreflect the differing views held by the actors or groups in the case.

The selection of groups to study in field studie s is equally important. The desirabilityof simultaneo usly studying random ly selected treatment and control groups is obvious.How ever, such a luxury is not com mo n in fteld studies. Instead, opportunistic analysisof naturally occurring organizational events is more likely. It becomes important tounderstand w hy groups have chosen to use EMS technology, as the motives, needs, andexpectations of early innovators may differ from those of subsequent groups. How-ever, opportunities to select groups randomly may arise more com monly than antici-pated. For example, resource con straints can make it impossible for all possible groupsin an organiza tion to use EMS technology or, alternately, only part of an organizationmay be targeted for the initial installation of EMS technology. In these cases, it maybe possible to study otherwise similar treatment and nontreatment groups addressingsimilar tasks in similar contexts [7].

Other opportunities also exist. Markus [45] argues that in order to disconfum atheory, it is necessary to define a set of cond itions in which the theory is m ost likelyto hold. By selecting a case that satisfies these conditions, and then proceeding todisconfirm a theory, tbe theory will be "quite decisively disconfirmed" (p. 10).Therefore, in the study of single groups using EMS, is is necessary to define thecharacteristics of a desired group, and then identify a group that meets those charac-teristics.

Field studies can also benefit from data gathered on the background of groupmembers. Such information is useful in determining the appropriate technology andtools, as well as assisting in developing theories to explain why seemingly similargroups have dissimilar EMS experiences. Hypotheses from such observations can thenbe tested experimentally. The combined knowledge base of experimental and fieldexperience can encourage the development of em pirically based guidelines for appro-priately matching technology to task and group characteristics.Established/Ongoing GroupsIn order to better model the established/ongoing groups typically found in organiza-tions (who have a past and future together), Zigurs et al. [69], for example, usedpreexisting groups of students. These groups were taken from an MIS course thatrequired groups of students to work together for the duration of the course. Theexperimental session (unrelated to the course material) occurred midway through theterm. The groups had no prior experience with EMS, which Zigurs et al. conclude w asa major factor in one gro up 's refusal to use the EMS; they did not want to change their


22/30

128 DENN IS, NUNAM AKER. AND VOGEL

making them similar to organizational groups. However, these norms can increaseerror, thus requiring m ore groups to achieve statistical significance [8].

Field studies as well as laboratory studies should focus not only on the meetingsession as a unit of analysis, but also on the project as a unit of analysis, where a projectmay consist of a num ber of meeting s. Group mem bership and linked use of informa-tion across sessions should be monitored. Particular attention should be given totracking changes in group member perceptions and performance over a series ofsessions. Comparisons can be made on a longitudinal basis within a project or seriesof sessions as well as between projects.Group SizeA host of non-EMS-supported research has shown that the process and outcome ofgroup meetings is different among groups of different sizes [e.g., 29]. Importantdifferences have also been o bserved am ong different sized EMS-supported group s [1 2].As organizational groups have tended to be larger than experimental groups inprevious studies, one obvious suggestion is to increase the size of groups used inexperimen tal studies, and for field studies to exam ine the effects of EMS use on smallergroups.

Field studies, as well, can benefit from careful tracking of group size as a functionof impact on the process and outcome of sessions. Much can be leamed by takingadvantage of situations that suggest v arious group sizes, as well as situations in wh ichthere appears to be a mismatch of the group size with the task. One goal is to com pareequivalent grou p sizes in experimental and field settings .The TaskSelecting the task is arguably the most important pan of the experimental researchdesign. The task should be both appropriate for the subjects, and similar to the tasksaddressed by organizational groups in whatever target environm ent the researcher haschosen. While most laboratory experiments have studied decision-making tasks (i.e.,choice tasks or generate and choose tasks), most field stud ies have examined generatetasks and planning tasks. This suggests that future experiments should seek toinvestigate generate/planning tasks, while future field studies should exam ine decisionmaking.

As w ell as providing an issue to be addressed by the group , the task in the laboratorymust provide the backdrop for the group discussion. The task domain should beappropriate for the subjects, and, ideally, familiar to them [26]for example, theparking problem common to most universities [6]. Such tasks are readily u nderstood ,and provide a rich meeting environment. Participants bring assumptions to themeeting. Different perceptions are likely. It may even be possible to measure theimproved u nderstandin g of the issues gained by each individual.

If such preexisting tasks are not used, sufficient background information must be


23/30


Subjects should be actively encouraged to consider all relevant components of theissues at hand, as would b e the case in the tasks facing o rganizational grou ps.

Field studies should likewise be concemed with task issues from several perspec-tives. Different measures of the nature of the task should be collected, to helpresearchers understand ihe effects of EMS for different types of task. Such measuresinclude task difficulty, familiarity with the task, etc. [60]. In m any cas es, there maybe a lack of consensus on what the task is; group members may not share a commonunderstanding of the goals and objectives of the group. These issues need to beconsidered, possibly by comparing processes and outcomes w hen a group does a "real"task instead of an "artificial" one developed by the researchers that differs in theseaspects.Infonnation ManagementInformation for experimen tal sessions could be provided in a variety of forms. Ratherthan being neatly condensed into one case package, it could be spread across a varietyof "organizational" reports and databases, as it could be with organizational groups.The information can then be made available in either paper or electronically accessiblecopy during the group's deliberations. Different information (either conflicting orcom plemen tary) co uld be provided to different participants. The end result is a settingmore typical of the way organizational groups use EMS. Further, the availability (orlack thereoO of information can be used to reinforce the group's charter, as well asevaluate aspects of the impact of introduction of information extemal to the groupduring the decision-making proces s. Again , such information dynam ics are typical oforganizational grou p use of EMS.

From a field research perspective , it is important to understand how the differencesin information held by the participants affect the processes. Links to corporatedatabases and integration into organization infonnation systems are also important,and thus systematically monitoring and tracking the use of extemal informationbecom es possib le. Provision should also be made for integrating information ^ r o s sgroup sessions and among groups to further provide a measure of effective groupsupport and provide a foundation for eva luation.IncentivesPerformance incentives should be provided to experimental subjects to better ensurethat their participation behavior is appropriately motivated; without incentives, re-searchers can be less sure that the behavior exhibited is "real," as subjects have nostake in the outcome [62]. Subjects in groups achieving "better" meeting outcomescan be rewarded, even for tasks that do not provide "objective" outcomes. In organi-zational groups, incentives are not necessarily tied to the "best" outcome for theorganization as whole, and thus similar incentives need not be provided to eachsubject. To model the vested interests and political nature of organizational groupmeeting s, each su bject's rew ard could be be determined on a different basis, such as


24/30

130 DENNIS, NUNAMAKER. AND VOGEL

administration of performance incentives in the organization being studied and thedegree of organizational politics. Members may also be more or less motivated toparticipate in the stated task. Although this is true of field groups with or withoutprovision of automated support, it becomes particularly important to track groupmem ber perceptions of the task to facilitate com parisons ac ross groups and tasks. Pre-and postmeeting questionnaires, as well as systematic tracking of group memberinteraction during EMS sessions, is helpful to evaluate incentive issues. A dditionalinformation can be obtained in interviews with group leaders, facilitators, managers,and executives knowledgeable in the incentive structure and politics of the organiza-tion to determine the degree to which they could affect processes and outcom e.EMS EnvironmentAs each type of meeting process (chauffeured, suf^rted, or interactive) may havedifferent effects on meeting outeo mes, it is essential to docum ent the processes groupsuse at each point in the me eting. Som e processes are likely more effective than othersfor certain activities, so experiments comparing different p rocesses for the same tas ksare needed.

Research data can be collected unobtrusively ftom a variety of sources. EMSenvironments can be designed to provide m ore detailed process information to gaininsight into the EMS-supported meeting process. Software can provide detailed activitylogs. Audio-video recordings of sessions can be used for detailed analysis. Groups c ^be m ore carefully observed before and after the EMS-supported m eeting(s) to providea better understanding of exactly how the EMS altered the group work proc ess.

EMS is a state-of-the-art techn ology. Unlike other technologies in MIS research (suchas end-user computing or system development techniques), it cannot yet be widelystudied in the hands of practitioners. Only a few organization s use EMS technology a sresearchers know it today [41]. This has several implications. First, researchers m ustdevelop their own EMS (or use EMS technology developed by other researchers). W hileit is tempting to develop only the m inimum EMS environment necessary for laboratoryresearc h, this lessens the generalizability of findings from these enviro nm ents to EMSenvironments in organ izations.

Second, a major concem of field studies lies in comparison of results across sites.Rarely are sites identical. Rather, each site, for historical or practical reasons, or aconscious d esire to evaluate different physical configurations, ten ds to have a certaindistinctiveness. This can seriously confound the ability to com pare results across sites,even when ignoring cultural and other organizational differences that may also varysignificantly, even within the same organization. The site characteristics should becarefully recorded and analyzed to better field research results leading to the devel-opment of new theories.

Finally, a major research contribution that EMS researchers can make is to developguidelines for the successful implementation of EMS technology in organizations.Practitioners in the early 1980s were relatively unsuccessful in their attempts to


25/30


is to guide practitioners in the successful ado ption of EMS technology, which requiresthe use of EMS environments similar to those required by o rganizations.

T o w a r d a M o r e P e r f e c t W o r l dW E BELIEVE THAT, WHILE FINDINGS FROM PREVIOUS EMS laboratory experiments andfield studies have been different, they are not inconsistent Findings have beendifferent because EMS use in previou s field studies has been distinctly different fromEMS use in previous laboratory exp erime nts. We noted 24 specific differences in fourareas. To paraphase Huber [33, p. 571], while we believe that each of these differenceshas merit, even if only one or two has m erit, the conclusion has merit that the use ofEMS in experiments has been sufficiently different from the use of EMS by organiza-tional groups to suggest that researche rs have studied different co ntex ts, groups, tasks,and EMS, thus findings are not necessarily inconsistent.

This also suggests that previous laboratory results shotild not be generalized tocurrent field use of EMSand that previous field studies should n ot be generalized tocurrent laboratory environmentsalthough they could be generalized to similarenvironments.

In our opinion, these issues do not lessen the value of previous experim ental or fieldstudy research. Both have been key com ponents in advancing the state of know ledgeof EMS-supported group w ork. Ho wever, each has been, and w ill continu e to be, onlyone componen t. Many of these past differences reflect research design issues that canbe addressed in futtire studies. For example, future field studies can examine smallergroups while future laboratory studies can examine larger groups. Other differences,however, are more fundamental to these research methodologies, and will not becompletely am eliorated (e.g., task complexity). Regardless of researche rs' attem pts,there will continu e to be important differences between the laboratory and the field.

In a more perfect world, these fundamental differences would be widely recogn ized.We would realize that neither laboratory experiments nor field studies providecomplete insight into the use of EMS, and that an overreliance on one approach couldbe m isleading. W e would strive to gain firsthand u nderstanding from both perspe ctivesin order to better integrate findings. In a more perfect w orld, field re se ^c he rs wouldrecognize and accept that ihe laboratory is not the Field, and that certain aspects oflaboratory exp eriments m ay never be "rea l." Likew ise, laboratory researchers wouldrecog nize and accept that the field is not a laboratory, and that different standards ofevidence are used to assign cause and effect.

In a more perfect world, we would strive to understand the effects that thesedifferences have on the processes and outcomes of group meetings. Is EMS morebeneficial in politically charged contexts, to larger groups, for generation tasks, orwhen it provides a highly structured process? We could then use this understandingto interpret and apply the conclusions of experiments to the use of EMS by business


26/30

132 DENNIS. NUNAMAK ER. ANDVOGEL

new research designs: field studies with greater rigor and experiments with greaterrelevancy to the organizational use of EMS.

In a more perfect world, we would recognize that these design decisions directlyaffect the processes and outcomes observed and the subsequent conclusions drawnabout the effects of EM S use. Such decisions would be made only after a carefulconsideration of their potential impact on the final conclusions and in a manner thatbest suited the research questicms. We have presented several approaches to enhancethe design of EMS experiments and field studies. These suggestions certainly do notapply to all research programs, as different programs have different objectives. Norare these suggestions without cost. However, as suggested by Javenpaa, Dickson, andDeSanctis [35] with regard to another research area (management graphics), a lesserattention to such details will not advance our understanding of EMS; it will only serveto cloud the issues.

NOTES

1. Although Kerlinger is speaking about the social sciences in general, weand others [e.g.,25 ,39 ] feel thai this view is aJso relatively prev alent in the MIS domain.2. In using the tenn "organizational gro ups" throughout this paper, we are referring to thegroups from both the public and the private sector who have used EMS technology aspart oftheir ongoing organizational activities.REFERENCES

1. Ackoff, R. L. Creating the Corporate Future. New York: John Wiley and Sons, 1981.2. Adelman , L. Real time computer support for decision analysis in a group setting. Inter-faces, 14 , 2 (1984), 75-83.3. Benbasat, I.; Goldstein, D. K.; and Mead, M. The case research strategy in studies of in-formation systems. M/^guorrer/^i, 11 , 3 (1987), 369-386.4. Bui, T., and Sivasankaran, T. R. Relation between GDSS use and group task com plex-ity. Proceedings of the Twenty-Third Hawaii International Conference on Systems Sciences(1990), n i: 69-7 8.5. Chidambarum, L.; Bostrom, R. P.; and Wy nne, B. E. An empirical investigation of the

impact of computer support on group developm ent. Proceedings of the Twenty-Third HawaiiInternational Conference on Systems Sciences (1990), III: 3-12.6. Connolly, T.; Jessup, L. M.; and Valacich, J. S. Idea generation in a GDSS: effects of an -onymity and evaluative tone. Management Science, forthcoming.7. Cook, T. D ., and Campbell, D. T. Quasi-Experimentation. Boston: Houghton Mifflin,1984.8. Dennis, A. R.; Easton, A. C; Easton, G.K.; George, J. F.; and Nunamaker, J. F., Jr. Adhoc versus established groups inan electronic meeting system environm ent. Proceedings ofthe Twenty-Third Hawaii International Conference on Systems Sciences (1990), HI: 23-29.9. Dennis, A. R.; George, J. F.; Jessup, L. M.; Nunamaker, J. F. Jr.; and Vogel, D . R. In-fonnation technology to support group work. MIS Quarterly, 12 (1988), 591-624.10 . Dennis, A. R.; Heminger, A. R.; Nunamaker, J. F., Jr.; and Vogel, D . R. Bringing auto-mated support to large groups: the Burr-B rown experience. Information & Management, 18,3 (1990). 111-121.


27/30

LABORATORY VS. HE LD RESEARCH ON EMS 133of group size in an electronic meeting system environment. IEEE Transactions on Systems,Man, and Cybernetics, forthcoming.

13. DeSanctis, G.; D'Onofrio, M. J.; Sambamtirlhy, V .; and Poole, M. S. Com prehensive-ness and reslrictiveness in group d ecision heu ristics: effects of com puter support on consen-sus decision making. ICIS (1989), 131-140.14. DeSanctis, G., and Gallupe, R. B. A foundation for the study of group decision supportsystems. Management Science, 33 (1987), 589-609.

15 . Diehl, M., and Stroebe, W. Productivity loss in bralnstorming groups: towaid the solu-tion of a riddle. Journal of Personality a nd Social Psychology, 53 (1987), 497-509.16. Dutton, W. H. Letter to the editor. MIS Quarterly, 12,4 (1988), 521.17. Easton, A. C ; Vog el, D. R.; and Nun ama ker, J. F., Jr. Stakeholder identification and as-umption surfacing in small groups: an experimental study. Proceedings o f the Twenty-Sec-ond Hawaii International Co nference on Systems Sciences (1989), III: 344-352 .18. Easton, G.; G eorge, J. F.; Nun ama ker, J, F., Jr.; and Pcndergast, M . O. Using tw o differ-ent electronic meeting system tools for the same task: an experimental com parison. Journalof Management Information Systems, 1990.19. Fellers, J. W. Th e Effect of Group Size and Computer Support on G roup Idea Gen era-tion for Creativity Tasks: An E xperimental Evaluation Using a Repeated M easures De sign.Unpublished Ph.D. dissertation, Indiana University, 1989 .20. Galiiers, R. D., and Land, F. F. Choosing ^jpropriate information systems researchmethodologies. Viewpoint, Communicat'ionsoftheACM, 30, 11 (1987), 900-9 02.21. Gallupe, R. B . Suppressing the contribution of the grou p's best mem ber: isGDSS use ap-propriate for all group tasks? Proceedings of the Twenty-Third Haw aii International C onfer-ence on Systems Sciences {\^^\^S1\ Yh-ll.Tl. Gallupe, R. B.; DeSanctis, G.; and Dickson, G. W . Computer-based support for groupproblem finding: an experimental investigation. MIS Qua rterly, 12, 2 (1988), 277-29 6.23. Gallupe, R. B., and McKeen, 3. D. Beyond com puter-mediated comm unication: an ex-

perimental study into the use of a group decision support systems for face-to-face versus re-mote meetings. Information &. Management, 18, 2 (1990).24 . George, J. F.; Easton, G. K.; Nunamaker, J. F., Jr.; and Northcraft, G. B. A study of col-laborative group work with and w ithout computer based support. Information Systems Re-search, forthcoming.25. Goldstein, D.; Markus, M. L.; Rosen, M.; and Swanson, E. B . Use of quantitative m eth-ods in MIS research. Proceedings of ICIS (December 1986), pp. 338-339.26 . Gordon , M. E.; Slade, L. A.; and Schmitt. N. The "science of the soph om ore" revisited:iiomconiectmetoempkicism. Academ y of Mana gement Review, 11,1 (1986), 191-207.27. Hackm an, J. R., and Kaplan, R. E. Interventions into group process: an approach to im-proving the effectiveness of groups. Decision Sciences, 5 (1974), 459-480.28. Hall, J., and Williams, M. S. A comparison of decision-making perfonn ance in estab-

lished and ad hoc groups. Journal of Personality and Social Psychology, 3 (1966), 214-222.29 . Hare, A. P. Group size. American Behavioral Scientist, 24, 5 (1981), 695-70 8.30 . Ho, T. H.; Ram an, K. S.; and W atson, R. T. Group decision support systems: the cul-tural factor./C/5, 1989,119-129.31. Hoffer, J . A.; An son, R,; Bostrom , R. P.; and Michae le, S. J. Identifying the root causesof data and systems planning p roblems: an application of the PLEXSY S electronic m eetingsupport system. Proceedings of the Twenty-Third Haw aii International Conference on Sys-tems Sciences (1990), m : 30 39.32 . Huber, G. P. The nature of organizational decision making arxl the design of decisionsupport systems. MIS Quarterly, 5, 2 (1981 ), 1-10.33. Hub er, G. P . Cognitive style as a basis for Mis and DSS designs: much ado about noth-ing'} Management Science, 29, 5 (1983), 567-579.34. Jarvenpaa, S. L. The importance of laboratory experimentation in IS research. TechnicalCorrespon dence, Coffimurticanon5o///ie.4CA/, 3 1, 12 (1 98 8) , 1502-1504.


28/30

134 DENN IS. NUNAMAKER . AND V0C5ELdium-sized g roups w orking on unstructured problems: a field experimen t. MIS Quarterly,12,4 (1988), 645-666.37. Jessup, L. M.; Connolly, T.; and Galegher, J. The effects of anonymity o n group pro -cess in an idea-generating task. MIS Quarterly, forthcoming.

38 . Jessup, L. M., and Tansik, D. A. Decision making in an autom ated environment: the ef-fects of anonymity and proximity on group process and outcome w ith a group decision sup-port system. Decision Sciences, forthcoming.39. Kaplan, B., and Duchon, D . Com bining qualitative and quantitative methods in infor-mation systems research: diczsc study. MIS Quarterly, 12,4 (1988), 571-586.40 . Kerlinger, F. N. Foundations of Behavioral Research, third edition. New York: Holt,Rinehart and Winston, 1986.41. Kraemer, K. L., and King. J. L. Computer-based systems for cooperative w ork. Comput-ing Surveys, 20 (19^1 115-146.42. Lee, A. S. A scientific m ethodology for MIS case studies. MIS Quarterly, 13, 1 (1989),33-50.43. Lim, L. H.; Raman, K. S.; and W ei, K. K. Does GDSS promote more democratic deci-sion-making?the Singapore experiment. Proceedings of the Twenty-Third Hawaii Intema-tional Conference on Systems Sciences (1990), HI: 59-6 8.44 . Loy, S. L.; Pracht, W. E.; and Courtney, J. F., Jr. Effects of graphical problem structur-ing aid on small group decision mak ing. Proceedings of the Twentieth Hawaii InternationalConference on System Sciences (1987), 566-574.45. Markus, M. L. Case selection in a disconiimiatory case study. Presented at the New Ap -proaches in MI S Research Symposium, National Academy of Management Meeting, August9,198 8, Anaheim, CA.46. Mason, R. O., and Mitroff, 1.1. Challenging Strategic Planning Assumptions. NewYork: John Wiley and Sons, 1981.47. McCartt. A. T., and R ohrbaugh. J. Evaluating grou p decision support system effective-

ness: a performance study of decision confeiencmg. Decision Support Systems, 5 (1989),243-254.48. McGrath, J. E. Groups: Interaction and Performance. Englewood Cliffs, NJ: Prentice-Hall, 1984.49. McGrath. J. E. Studying groups at work: ten critical needs for theory and practice. In P.S. Goodman and Associates (eds.). Designing Effective W ork Group s. San Francisco: Jossey-Bass. 1986.50 . Nunam aker. J. F., Jr.; Applegate, L. M.; and K onsynski, B. R. Facilitating grou p cte-ativity vj'iih GDSS. Journal of Managem ent Information Systems, 3 (1987). 5-19.51. Nunam aker, J. F., Jr.; Applegate, L. M.; and Konsynski, B . R. Computer-aided delibera-tion: model management and group decision support. Journal of Operations Research(1988), 826-848.52. Nunam aker., J. F., Jr; Vogel, D.; Heminger, A.; Martz, B. ; Grohow ski, R.; and McGoff.C. Ex periences at IBM with group support systems: a field study. Decision Support Systems,5,2(1989), 183-196.53. Pinsonneault, A., and Kraemar, K. L. The impact of technological support on groups:an assessment o( i\e.empirical research. Decision SImport Systems, 5, 2(198 9), 197-216.54 . Poole, M. S.; Siebold, D. R.; and M cPhee, R. D. Group decision-making as astructurational proce ss. Quarterly Journal o f Speech, 71 (1985), 74-102.55. Rem us, W. E. An em pirical test of the use of graduate students as surrogates for m anag-ers in experimen ts on business decision making. Journal of Business Research, 14 (1986),20-30.56. Rem us, W. Using students as subjects in experiments on decision support system s. Pro-ceedings of the Twenty-Second H awaii International Conference on Systems Sciences

(1989),IV:176-180.57 . Rittel, H. W. J., and Webber, M. M . Dilemmas in a general theory of planning.


29/30

LABORATORY VS, FIELD RESEARCH ON EMS 135

59. Sharda, R.; Barr, S. H,; and McD onnell, Decision supp on system effectiveness: a re-view and an empirical test. Management Science, 34,2 (1988). 139-159.60. Shaw, M. Group D ynamics: The Psychology of Small Group Behavior, third edition.New York: McGraw HiU, 1981.61. Siegel, J. S,; Dubrovsky, V.; Kiesler, S,; and McGture, T, Group processes in com puter-mediated communication. Organizational Behavior and Human Decision Processes, 37(1986), 157-187,62 . Smith, V. L. Micro econom ic systems as an experimental science. American EconomicReview, 72 (198 2), 923-9 55,63. Steeb, R,, and Johnston, S. C. A computer-based interactive system for group decisionmaking, IEEE Transactions on Systems, M an, and Cybernetics, SMC-11 (1981), 544 55 2.64. Valacich, I S.; Den nis, A. R,; and Nun amak er, J. F., Jr. Electronic meeting sup port: theGroupSystems concept. International Journal of Man-M achine Studies, forthcoming.65. Van Shaik, F. D,, and Sol, H. G, Effectiveness of decision support systems. Proceed-ings of the Twenty-Third Hawaii Intemational Cor^erence on Systems Sciences (1990), UI:50-58.66. Vogel, D . R,; Martz, W, B,; Nunamaker, J. F., Jr.; Grohowski, R. B.; and McGoff, C,Electronic meeting system experience at IBM, Journal of Managem ent Information Systems(1990).67. Vogel D, R,, and Niinamakcr, J, F., Jr, Health service group use of automated planningsuppoTL Administrative Radiology (Septonber 1989),68. Watson, R. T.; DeSanctis, G,; and Poole, M, S, Using a GDSS to facilitate group consen-sus: some intended and tinintended consequences, MIS Quarterly, 12, 3 (1988), 463-47 8.69. Zigurs, I.; Poole, M, S,; and DeSanctis, G, A study of influence in computer-mediatedcommtmication. MIS Quarterly, 12 ,4 (1988), 625-644.70 . Zm ud, R, W ., and Hauser, R. Field experimentation in MIS research. Working Paper,College of Bu siness, Florida State University, June 19 88.


30/30

Field vs Lab

Documents

Transcript of Field vs Lab