EMPIRICAL RESEARCH IN INFORMATION … RESEARCH IN INFORMATION SYSTEMS: THE PRACTICE OE RELEVANCE^...

Benbasat & Zmud/Practice of Relevance

EMPIRICAL RESEARCH IN INFORMATION SYSTEMS:

THE PRACTICE OE RELEVANCE^

By: Izak BenbasatUniversity of British ColumbiaFaculty of Commerce and Business

Administration#452-2053 Main MallVancouver, [email protected]

Robert W. ZmudMichael F. Price College of BusinessUniversity of OklahomaNorman, OK [email protected]

AbstractThis commentary discusses why most IS acade-mic research today lacks relevance to practiceand suggests tactics, procedures, and guidelinesthat the IS academic community might follow intheir research efforts and articles to introduce rel-evance to practitioners. The commentary beginsby defining what is meant by relevancy in thecontext of academic research. It then explainswhy there is a lack of attention to relevance with-in the IS scholarly literature. Next, actions thatcan be taken to make relevance a more centralaspect of IS research and to communicate impli-cations of IS research more effectively to IS pro-fessionals are suggested.

Keywords: Relevance, rigor, academic research,applied research

ISRL Categories: AI0104, AI03, AI05

Introduction

1 Lynda Applegate was the accepting senior editor forthis paper.

"Is research in the Ivory Tower 'Fuzzy, Irrelevant,Pretentious?'" {Business Week 1990), The point-ed question raised in the title of this BusinessWeek article is not an isolated, off-hand observa-tion. Instead, it represents the views of many ofthe stakeholders collectively holding the largessof business school faculty: students; recruiters;funding, grant, contract, and gift sources; con-tacts enabling access to resource sites; and busi-ness school deans, Scott Cowen, then dean ofCase Western Reserve University's WeatherheadSchool of Management, stated "As much as 80%of management research may be irrelevant"(Business Week 1990, p, 62) and Richard West,New York University's business school dean atthe time, was even more critical in his assess-ment of academic articles in scholarly journals,"[Business academics] say nothing in these arti-cles and they say it in a pretentious way"(Business Week 1990, p, 62), While theseremarks are somewhat dated, they most likelywould be upheld, or perhaps even exaggerated,today.

The criticisms expressed above have also beendirected to published information systems (IS)research (Galliers 1994; Saunders 1998; Zmud1996a, 1996b), That IS research has a credibilitygap within the business community is certainly

MIS Quarterly Vol. 23 No. 1, pp. 3-16/March 1999 3


evidenced by the 1995 decision of the Societyfor Information Management (SIM) Internationalto terminate their long-standing practice ofincluding a subscription to the MIS Quarterly aspart of their members' dues. And, when providedwith the opportunity on their dues statement toreceive (at additional charge) the Quarterly at adiscounted price, few members have chosen todo so—even though the journal has as a statededitorial mission of publishing research targetedat information system managers.

It certainly is not a surprise to most IS academicsthat the business community would question thepractical relevance of IS research published inthe leading journals of our field. Does IS researchproduce the knowledge that today's IS profes-sionals can apply in their daily work? Does itaddress the problems or challenges that are ofconcern to IS professionals? Does it focus on cur-rent technological and business issues? Are ISresearch articles accessible to IS professionals? Itis our view that the answers to these questions donot shed a favorable light on IS academic work.The aforementioned fact that the MIS Quarterly,which has made a conscientious attempt over theyears to cater to practitioners, has lost the major-ity of its SIM readership tangibly supports such aview. For much "softer" evidence, how oftenhave IS practitioners commented to you about anarticle published in an IS scholarly journal? Oneof this article's authors regularly interacts with anelite group of information systems executives inhis role as research director of SIM's AdvancedPractices Council, In the rare instance when areference to published academic work is madeby these executives, it invariably was publishedin Harvard Business Review or SloanManagement Review.

We do not wish to imply through this essay thatall IS research should strive to meet the (oftencapricious) needs of practicing IS professionals.We value the role theory development and basicresearch play in the advancement of the field.This paper is targeted to IS academics who arecommitted to both applying rigorously themethodology best suited to their research goalsand better accommodating practical relevancewithin their research endeavors. We consider ISprofessionals (including analysts, user represen-tatives, and IS managers) as well as managers

with an interest in information technology (IT)deployment and utilization to be the consumersof IS research that is relevant.

The intent of this article is two-fold: to explainwhy one tends today to observe a lack of rele-vance to practice in IS research and to suggesttactics, procedures, and guidelines that the ISacademic community might follow to introducerelevance into their research efforts and articles.We begin by defining what is meant by relevan-cy in the context of academic research. Then, weprovide an explanation for the lack of attentionto relevance within the IS scholarly literature.Finally, we recommend actions to be taken tomake relevance a more vibrant aspect of ISresearch efforts and to communicate the fruits ofIS research more effectively to IS professionals.

It is important to mention at the start that theviews expressed in this paper are those of twoNorth American IS academics who have mainlyespoused a positivist research tradition.Colleagues who prefer to see the world throughother lenses, e,g,, advocates of "action research,"are likely to offer their own approaches to attainrelevance. In addition, we focus here on the con-tributions of empirical research to relevance;"design" is outside the scope of our discussion.Our answers here are obviously not the onlyones feasible, but we hope that they will initiatea discussion on how to obtain the same goalsthrough multiple means.

The Nature of RelevantResearch • • • • • • • • • • • • ^ ^ ^Just what exactly is research that is "relevant topractice"? It is possible to gain insights regardingwhat managers find to be interesting and impor-tant (Price 1995; Sears and Pickler 1996), Thetopics being addressed are clearly very influen-tial in determining an article's relevance in theeyes of practitioners. Not surprisingly, articlesthat address enduring (or current) organizationalproblems, challenges, and dilemmas as well asarticles that address timely business issues tendto be well received by practice.

4 MIS Quarterly Vol. 23 No. 1/March 1999


Relevancy is not assured, however, through theselection of a "relevant" topic. Unless an article'simplications are implementable, i,e,, prescribedin a manner that could be put to use (to someextent) in practice to exploit an opportunity or toresolve a problem, practitioners are unlikely tocharacterize it as being of interest. Not all (per-haps, even, quite few) scholarly articles, howev-er, manage to produce such outcomes. Does thismean that few scholarly articles have the poten-tial to be relevant? It does not, as two other typesof articles are also valued, though not as univer-sally, by practicing managers. They are articlesthat synthesize an existing body of research (i,e,,which classify, categorize, and summarize majorthemes and findings), and articles that stimulatecritical thinking by challenging the reader'scausal assumptions (Davis 1971) or by identify-ing emerging trends, structural changes, or para-digms. The former provide a fairly "painless" wayfor managers to acquaint themselves with the"state of knowledge" regarding specific domains,while the latter have the potential to enhance orrestructure the mental models managers apply intheir practice.

Just as important as, if not more important than,an article's content is its style and tone. Statedsimply, articles that are not read, regardless oftheir content, are not relevant. Articles that tendto be read by IS professionals are those that

• are shorter• use more exhibits• use everyday language, rather than esoteric or

stilted language• have less discussion of related literature• have less discussion of a study's methods• have more contextual description• have more prescriptions

Note that the above attributes omit commentaryon an article's rigor, i,e,, the correct use of meth-ods and analyses appropriate to the tasks-at-hand. Relevancy does not imply that researchneeds to be carried out in a less rigorous fashion.In fact, managers respect and value rigor (as itoften proves to be a key discriminator betweenacademic and consultancy contributions). Whatshould be refrained from are the "trappings" ofrigor: lengthy homages to the "literature," too"scholarly" a tone, and elongated descriptions ofa study's methodological and analytical proce-

dures that could be kept in the background inappendices.

To summarize, we suggest that authors who striveto craft relevant articles for practitioners must, ata minimum, focus on the concerns of practice,provide real value to IS professionals, and applya pragmatic rather than academic tone. Ideally,they would also describe how the ideas dis-cussed or actions suggested would be imple-mented in practice, allowing for contextual dif-ferences that are important to individual readers.

Why Much IS ResearchLacks Relevance ^ ^ ^In this section, we discuss the reasons that muchof the current IS literature lacks sufficient rele-vance. If these are structural (i,e,, reflective ofinstitutional or environmental forces), then reso-lutions of such structural elements must beembedded within the tactics we propose forincreasing the relevancy of IS research.

We have identified five such explanations. Twoare associated with the nature of IS scholarship:an emphasis on rigor over practical relevanceand a lack of a cumulative research tradition, Athird reflects the dominant attribute of thedomain in which our research takes place: therapid and continuous rate of change associatedwith information technologies. The fourthreflects the limited extent to which IS academi-cians are exposed to the business and technolog-ical contexts in which IS phenomena transpire.The fifth is associated with institutional and envi-ronmental constraints that influence the freedomof action within academia.

An Emphasis on Rigor Over Relevance

In order to establish IS as an academic disciplineand to gain the respect of more established acad-emic disciplines in business schools, ISresearchers and the editors of top IS academicjournals have tended to emphasize rigor over rel-evance in their journals and in their views ofappropriate promotion and tenure criteria. Such astance was taken as a reaction to the criticism lev-eled at the field since its inception, (See Benbasatand Weber [1996] for a historical perspective,)

MIS Quarterly Vol. 23 No. 1/March 1999 5


Throughout the 1970s, it became quite evidentthat the quality of IS research, in comparison tothat of other business disciplines, was foundwanting.

In response. North American IS academics beganlaying the foundations of a research infrastruc-ture with new journals (MIS Quarterly), confer-ences (International Conference on InformationSystems), and workshops on research methods(Harvard Business School colloquia on qualita-tive, experimental, and survey research). By the1990s, this continuing emphasis on performingrigorous research had finally paid off. Today, thequality of IS research matches that of its sisterbusiness school disciplines. But the price wehave paid is that the practical relevance of ourresearch has been relegated to a secondary role.

Lack of a Cumulative TraditionGenerally, IS researchers have been less success-ful than their colleagues in other business schooldisciplines in developing a cumulative researchtradition (Keen 1980), Without such cumulativeresults, it becomes difficult, if not impossible, todevelop and assess strong theoretical modelssuch that prescriptive actions can confidently besuggested for practice.

At least three explanations arise to explain thislack of a cumulative tradition. First, in IS, a mul-tiplicity of theoretical frames exists for most phe-nomena being studied. For example, an exami-nation of interorganizational linkages could verywell be approached from a number of quite dis-tinct perspectives (I/O economics, computer net-work or data architectures, sociological net-works, organization design, interpersonal rela-tionships, population ecology, etc), with resul-tant articles potentially publishable in the sameIS journal, IS scholars reflecting each of these dis-parate perspectives often have difficulty under-standing, let alone incorporating, work per-formed within others. Second, as a community,IS scholars have, until recently, been reluctant tovalue the existence of a well-defined collectionof research constructs and instruments. Beingtechnophiles at heart, IS researchers would muchrather invent than adopt! But without commontools and a shared language, it becomes difficultto evolve streams of research that build rich con-

ceptualizations and understandings of the phe-nomena populating the IS domain. Third, therehas been a significant proliferation of IS journals,as with many other disciplines, over the last twodecades. As a consequence, it is becoming hard-er and harder to locate and access, let alonebuild upon, the work of others.

The Dynamism of InformationTechnologyMany IS academics (as well as practitioners) findthe dynamic technological environment of the ISfield to be one of the more compelling motivesfor working in this field. But this dynamism also(1) adds much complexity and uncertainty to ourresearch endeavors, (2) results in our chasingafter practice rather than leading practice, and (3)typically leads to reporting results from (rigorous)studies involving new technologies years afterthe technology's acceptance (and, occasionally,its rejection) by practice. As a result, many of thearticles published in current issues of IS journalsread—in terms of the technical and business con-texts being described—as if they could havebeen written years ago (in many instances, thefirst drafts probably were!).

As a community, IS scholars just have not dealtwell with the dynamism of our field. Too often,studies that focus on new technologies are pub-lished far past the date when the technologycould be considered "new," Needless to say, pro-nouncements in the future about today's techno-logical and associated business challenges arejust not going to be considered relevant by mostpractitioners. This structural problem has clearlynot been handled well and will be addressedlater in this article in the recommendations toauthors on topic selection.

Limited Exposure to Relevant ContextsIn order that IS research be relevant, ISresearchers must in some form or another beexposed to the practical contexts where IT-relat-ed usage and management behaviors unfold. Formany IS academicians, such exposure tends tooccur infrequently and, when it does occur,tends to be insufficiently targeted, insufficientlyrich, or both.



We believe there are three reasons for this. First,academics are very busy with teaching, research,and administrative service leaving very little timeto gain exposure, on a regular basis, to the contextswith which practitioners must contend. Second,when such exposure does occur, it often entailseither recruiting or consulting (including continu-ing education) relationships. With recruiting con-texts, the majority of the discussion usually focus-es on curriculum issues, skill set development, andassessments of specific students, and emphasizestechnical, rather than managerial, issues. Withconsulting relationships, IS academicians oftenfind themselves involved in delivering predefinedcontent (i,e,, continuing education) or a prede-fined output (a design, a program, etc) In bothcases, what tends to be absent are rich, loosely-structured dialogues of the opportunities and prob-lems being experienced in practice and discus-sions of how these might be examined throughacademic research. Third, all IS academics lack asufficient exposure to current and future techno-logical environments. Many, if not most, academ-ic IS departments are laggards regarding theircapabilities to maintain a current, let alone lead-ing-edge, hands-on technological environmentdue to a lack of financial and human resources.

Institutional and Political FactorsThe patronage and power structures within aca-demic institutions wield significant influence onthe degrees of freedom academics have in pursu-ing relevance. To be "legitimate," IS researchershave to adhere to the rules and conventionsimposed upon them by the academic patronagesystem (King and Applegate 1997), The majorityof the leading North American universities andbusiness schools are research-oriented institu-tions. Many of the top business schools stresstheory-based empirical research, economics, ormathematical modeling based work in their pro-motion and tenure decisions. In the past, internal(university) and external (federal) granting agen-cies have emphasized the pursuit of rigor as thestandard for awarding research grants. Therefore,our institutional environment has not encour-aged the pursuit of relevance. Interestingly, Kingand Applegate comment that the increasing shiftfrom government to corporate funding may befueling interest in the relevance of research.

SummaryAll of the five reasons discussed above haveaffected the topics studied in academic ISresearch, whereas the emphasis on rigor, institu-tional factors, and limited exposure to relevantcontexts have been mainly responsible for influ-encing the language and argumentationemployed in academic IS articles. The IS disci-pline has made significant progress in two areas.Today, the quality of most IS research work iscomparable with that of other business schooldisciplines. Since we have managed to achieve ahigh standard of rigor, we can afford to shift someof our attention toward relevance without undueconcern about being criticized by others,although the forces of institutional patronage stilllimit the scope of such a shift. And, over the lastdecade, we have begun to observe cumulativetraditions develop in a number of the theoreticalstreams that comprise the IS discipline. Theremaining three areas—the fast changing IT envi-ronment, limited exposure to relevant contexts,and the institutional context—continue to posesignificant challenges for the IS community.

Suggestions for Increasing theRelevance of IS ResearchThe primary responsibilities for publishing morerelevant IS research falls jointly on the authorswho produce manuscripts and the journal editorswho decide which of the submitted manuscriptswill be published. Authors need to rethink theirbehaviors regarding the topics being studied andwritten about, the purpose of the research pro-jects, and the readability of their manuscripts.Journal editors need to rethink their behaviorsregarding acceptance/rejection criteria and relat-ed signals (reviewer feedback, editorial com-ments, and published articles) sent to current andfuture authors. Here, we develop each of thesepathways for increasing the relevance of ISresearch and offer specific recommendations.

Authors: Topic SelectionToo often, an individual's or group's interestregarding a particular issue, and not the intendedoutputs of the project, becomes the sole basis for


Benbasat & Zmud/Rractice of Relevance

initiating research. While interest and an abilityto perform are required, they should be neces-sary rather than sufficient determinants of aresearch project. The key criterion is the poten-tial value and interest that stakeholders (journals,colleagues, practitioners) are likely to derivefrom the outcomes of the research project, out-comes most likely to appear in the literaturethree to five years in the future.

Recommendation 1 : The foremost criterion to beapplied in selecting research topics should bedirectly related to the future interest that keystakeholders (journals, colleagues, and practi-tioners) are likely to hold in a topic.

We suggest a two-pronged approach to the selec-tion of relevant IS topics. The first is to be moreattuned to the needs of practitioners by payingmore careful attention to the expressed areas ofinterest to practitioners. The second is for the IS aca-demic community to take a more proactive role indiscussing the key research areas of the disciplineby taki ng i nto account the benefits that wi 11 accrueto practitioners from research in these areas.

All too often, however, IS academics discovertheir research topics from the IS academic litera-ture. Using the IS academic literature as the pri-mary source for research topics is problematic forat least two reasons: the articles are dated whenpublished, and few of these articles were likelymotivated explicitly by the concerns of practice.

Recommendation 2: IS researchers should lookto practice to identify research topics and lookto the IS literature only after a commitmenthas been made to a specific topic.

There are many ways to find out what is of inter-est to practitioners: going to practitioner confer-ences, talking to practitioners, reading practition-er and general management journals, teachingexecutive education courses, engaging in con-sulting activities, etc Another effective vehicle is"key issues" surveys, where practitioners identifythose issues of most importance to them.Admittedly, time delays are present between theconduct of these surveys and their publication.For example, the latest MISRC/SIM key issuessurvey published in MIS Quarfer/y (Brancheau etal, 1996) presents key issues identified by practi-tioners in 1994—even with an expedited reviewprocess. Nevertheless, while "key issues" are

expected to vary in their importance over time,they historically have exhibited a reasonabledegree of stability. Still, the best strategy for anyIS researcher is to develop personal relationshipswith individual practitioners, both as a source ofideas for future projects and as a source of feed-back on current projects.

Even with the identification of what is of interestto practitioners, we obviously still face the prob-lem of reaction time. If, on average, it takes threeor more years from the start of a study to eventualpublication, then the "sense and react" cycle ismuch too long. Therefore, rather than "intelli-gence" regarding what was important two yearsago (from "key issues" surveys) or what is impor-tant today (from the current IT press), what weneed is to be focusing on those fundamentalissues likely to be important three years from now!This is where we think the IS academic communi-ty should and could play a more active role.

Consider, for example, a concern of great impor-tance to IS practitioners and a topic for whichconsiderable academic research has been per-formed: the adoption and adaptation of newinformation technologies. Over time, ISresearchers (along with colleagues from relateddisciplines) have had success in defining thenature of this phenomenon and developing a richunderstanding of the fundamental concerns asso-ciated with it. But for how many other phenom-ena germane to IS contexts could a similar pic-ture be painted? We think surprisingly few. Whileindividual intellectual creativity and serendipityin identifying research topics will (and must)always exist, we can not shirk our collectiveresponsibility to bring more focus to the IS disci-pline. Without an understanding of the funda-mental phenomena that comprise our field, howcan we be proactive and have, even partially,answers to assist practitioners when new prob-lems arise? We need to reach some degree ofconsensus as a community regarding the core,enduring phenomena associated with informa-tion systems planning, design, development,implementation, and use. And such a consensusmust be biased toward the future while remain-ing cognizant of the past. We realize that a callfor consensus has both proponents (Benbasat andWeber 1996) and opponents (Robey 1996), Whatwe are suggesting here is agreement on the



nature of these core phenomena, not conformityon how to investigate them.

Recommendation 3: More discussion on the coreresearch issues of the IS field that have thepotential to influence practice is needed.Members of the IS research community mustprepare manuscripts that define the importantphenomena for the various segments of the ISdiscipline, and editors of the leading IS jour-nals, as well as other major outlets such asISWorld Net, must work with these authors toensure that such information is disseminated.

Authors: Article's Purpose

Research projects are undertaken for many dif-ferent reasons, and once they have been com-pleted, articles describing these projects canagain be written with quite distinct purposes inmind. Often IS research projects are initiatedbecause authors wish to communicate how theyhave conceptualized and dealt with a challeng-ing problem rather than communicate a com-pelling and implementable resolution for theproblem. Articles based on the former projectsare written primarily to describe the completedwork for publication purposes rather than tocommunicate a well-honed message to a well-defined readership. Generally, IS researcherstend to focus on the "inputs" (e,g,, conceptual-izations and methods) of their research projectsrather than the "outputs" of these efforts.

Recommendation 4: When deciding whether ornot to begin a new research project or a newmanuscript, IS researchers should focus onthe likely outcomes, rather than the inputs, ofsuch efforts.

Let's consider these "outputs," focusing on thethree basic ways in which organizationalresearchers could influence practice (Astley andZammuto 1992),^

Direct UtilizationAcademic work could impact practice throughthe development of tools, techniques, and prac-tices (Astley and Zammuto 1992), As such anoutput holds immediate and real value to prac-

^Astley and Zammuto label them as the instrumental,conceptual, and symbolic modes.

tice, it tends to be the primary criteria applied bythose who criticize the relevance of academicresearch. Such research contributions are infre-quently observed since "researchers and usersbelong to separate communities with very differ-ent values and ideologies and these differencesimpede utilization" (Beyer and Trice 1982, p,608), Academics doing theory-based empiricalresearch normally strive to develop parsimonioustheories with a limited number of variables thatexplain phenomena across a wide range of orga-nizations, settings, and contexts (Daft and Lewin1990), However, what practitioners desire arerich prescriptions to be applied in (their) specificsituations that capture the uniqueness and com-plexity of their own organizational settings.Therefore, the potential for direct applicability istypically quite low.

Still, it is possible for some academic researchto contribute to practice in a direct, imple-mentable mode. Once a sizable body of litera-ture exists regarding a phenomenon, it doesbecome possible to synthesize this literature,e,g,, as a state of the art review, to developusable prescriptions. And, as this body of liter-ature evolves, authors can place the insightsgained from their specific studies within thisbroader body of knowledge to again produceuseable prescriptions.

Recommendation 5: In order for IS researchers tobe more proactive in a direct sense, it isimperative that the IS research communityproduce cumulative, theory-based, context-rich bodies of research.

What are these theories, if any, which form thebasis for these cumulative bodies of research?While the IS discipline has had little success increating its own theories (the on-going debateabout whether or not this is desirable for the ISdiscipline lies outside the scope of this essay), ithas been fairly successful in applying theoriesdeveloped in related disciplines to important ISphenomena. One exemplar of such practicesinvolves the efforts of a number of IS researchesto adopt and adapt the Theory of ReasonedAction (Fishbein and Ajzen 1975) and its exten-sions, such as the Theory of Planned Behavior(TPB) (Ajzen 1991), to the study of IT adoption,implementation, and use.



TPB posits that an individual's use of a specific ITwill be influenced by: (1) aff/fudes formed by theperson's beliefs about the expectations of out-comes associated with IT use, (2) subjectivenorms formed by the person's beliefs about howimportant others expect the person to behaveregarding IT use, and, (3) perceived behavioralcontrol formed by the person's beliefs about theextent to which the person is capable of actuallyusing the IT, While TPB does not describe theprocess of implementation in a specific context,it has experienced a high degree of predictivevalidity and can be used to identify areas of con-cern for a specific context (see Taylor and Todd1995 for an overview). As such, it can serve as aneffective diagnostic tool.

Now let's examine how the body of published ISresearch applying TPB fares in terms of its "uti-lizability" (Cheng and McKinley 1983), i,e,, itspractical relevance, the applicability of its find-ings, and its specificity. First, TPB can certainlybe conceived as possessing practical relevanceas it focuses indirectly on improving organiza-tional performance through its predictionsregarding IT usage. Second, prior to implement-ing a particular IT, the potential users of this ITcould be surveyed to determine their likelihoodof using the IT, their attitudes toward using the IT,their subjective norms and behavioral control.Based on the feedback, system implementerscould take appropriate corrective action (e,g,,improving attitudes by making the new systemmore compatible with current practices,) As TPBcontain variables that can be manipulated bypractitioners, its findings are applicable. Third,one of the strengths of the TPB is that while it isa general theory, it could be applied to a widevariety of contexts for predicting the adoption ofdifferent types of IT, Its major constructs reflectthe key variables that have been identified asinfluential in previous implementation research,such as top management support, and are flexi-ble enough to subsume situation-specific factors;thus it has the characteristic of specificity.Considering all three criteria together, TPB doesseem to be a theory very likely to produce utiliz-abte outcomes.

In addition to a theory's usability, there is anoth-er major way academic researchers in IS caninstrumentally contribute to practice. These are

contributions in the area of methodology andmeasurement. Practitioners are very ofteninvolved in measurement. Consider, for example,the needs of an individual assigned to undertakea post-implementation audit of an informationsystem that involves assessing users' satisfactionwith the quality of the implemented system.Another common measurement need involvesuser perceptions of the usability and usefulnessof an application's interface, especially during aprototyping development effort.

To answer these questions correctly requiresproper measures. However, measurement is acomplex endeavor, and poor measures are worsethan no measures as they provide incorrect datafor prediction, monitoring, and evaluation. Theaforementioned emphasis on rigor has influ-enced IS academics to develop a large number ofhigh-quality measurement tools. Recently, forexample, the May 1997 issue of MIS Quarterlypublished three research notes on the utilizationof the SERVQUAL instrument to measure IS ser-vice quality. Many such instruments, and evi-dence of their quality, are available from a website for IS measurement instruments(http://www,ucalgary,ca/~newsted/surveys,html).

Frames of ReferenceBased on their research findings, academicianssupply new concepts, which then alter the per-ceptions and mental models that practitionersapply in their work lives. What are some of theIS-related conceptual developments that havehad an impact on practice? End-user computing,critical success factors, business process reengi-neering, Nolan's model of stages of growth.Porter's value chain as well as his competitiveadvantage framework, and transaction cost theo-ries are just a few that immediately come tomind. What kind of conceptual development ismost useful for practice? Generally, those thatreorganize phenomena such that they seem lesscomplex (Davis 1971) or that enable contin-gency-based action. Descriptive case studies,together with the author's interpretations ofevents taking place in a specific organization,often prove to be effective means for communi-cating such contributions to practice.

By imputing understandings to managerialevents, [academics] perform a quasi-



journalistic role by communicating andreporting interpretations that disseminatethroughout the managerial community(Astley and Zammuto 1992, p, 456,)

An excellent example of this conceptual propa-gation is provided by the Strategic Grid(Applegate et al, 1996), Here, a 2-by-2 matrixrepresents the strategic relevance and impact ofan organization's IT portfolio, along with intu-itively understood labels like factory, support,strategic, and turnaround. Such labeling is criti-cal as it becomes the vehicle (i,e,, the "hooks")for contingency models that allow managers tothen interpret these concepts from the context oftheir own organizations. Similarly, Nolan's(1973) "Stages Hypothesis," through its initiation,contagion, control, and integration stages, crisplyand clearly describes the phases that essentiallyall organizations follow as they increase theirinvestment in and dependence on IT, Why is itthat the "Stages Hypothesis" continues to bereceived well by practitioners, in spite of havingbeen criticized by some academic researchers ontheoretical and empirical grounds? It is becauseit provides a conceptual language enabling an ISmanager to identify where his/her firm is posi-tioned in a "stages" sense. This allows the man-ager to better grasp the current challenges facingthe firm and the appropriate tactics for overcom-ing them, to predict what is likely to happen asthe firm transcends to successive stages, and,most importantly, to communicate these notionsto other executives.

Recommendation 6: In order for IS research to bemore relevant, it is important that authorsdevelop frames of reference which are intu-itively meaningful to practitioners to organizecomplex phenomena and to provide contin-gency approaches to action.

JustificationTheories, concepts, and findings from IS researchcould be used by practitioners to legitimate, ratio-nalize, and justify courses of action taken (Astleyand Zammuto 1992), Over the last decade, numer-ouspractitionershaveundoubtedlyusedthe"bible"of academic research (along with the presence ofacademic consultants) to justify IT-based initiativeson the basis of "reengineering" or gaining "compet-itive advantage," For example, ITexpenditu res may

be defended by claiming that they are needed toreengineer the firm for better customer service andimproved employee morale through empower-ment, asclaimedinthe academic literature,

A particular form of IS research often used in jus-tifying action is the benchmark study. Becausesuch studies (e,g,, the percentage of IT budgetthat goes toward new development, the compo-sition of firms' application portfolios) enablefirms to compare themselves with others, theytend to be of great interest to practice. There isunderstandably a desire to evaluate yourselfagainst your peers, for budgeting purposes aswell as for use in justifying why particular actionswere taken. Typically, such information is actual-ly a by-product rather than the primary outcomeof many academic research studies. However, byincluding such material within an article, the arti-cle's relevance can be greatly enhanced.

Recommendation 7: In order for IS research to bemore relevant, IS academics should portraythe outputs of their research in ways such thatit might be utilized by practitioners to justifyand rationalize IT-related initiatives.

Authors: Article's ReadahilityA principal means (and perhaps the least diffi-cult) for enhancing the relevance of IS articleslies with improving the language of communica-tion, A practitioner member of the Academy ofManagement makes this point strongly:

Journal and Review seem written by acade-mics looking to impress their fellow acade-mics with their ability to use polysyllabicjargon to confuse and obfuscate, I some-times show articles to colleagues as a JOKE,Most can't make it through the abstract(Sears and Pickler 1996, p, 25),

A number of years ago, the MIS Quarterlyresponded to such criticism by its managerialreadership (1) by incorporating reader-friendly"executive overviews" for published articles as ameans of increasing practitioners' motivation toactually read the articles and (2) by proactivelyworking with authors to improve the readabilityof their manuscripts.

The tendency of academics to communicatetheir ideas through a dense and dry style rests

MIS Quarterly VoL 23 No. 1/March 1999 11


partly on the belief that reviewers expect to seeevidence of academic rigor. But how exactlydoes one demonstrate academic rigor:

• Does scientific rigor demand complex,detailed, and often redundant explanation?

• Would scientific rigor benefit from clear, sim-ple, and concise explanation?

Too often, while the former view prevails, the lat-ter view may actually communicate an article'srigor—to both practitioners and academics—much more effectively. As our academic subdis-ciplines become more and more specialized,even IS academicians are having difficulty com-municating with one another!

Recommendation 8: The vast majority of ISresearch articles should be crafted in a clear,simple, and concise manner such that theyare accessible by alt the potential readershipof a journal.

The exceptions to this recommendation mightinvolve manuscripts whose primary contributionto the IS field is based on technical/analyticalsophistication and elegance, articles dealing withresearch methods/rnethodologies, and philo-sophical discussions of the foundations of the ISfield.

Journals: Reviewing Processes

We have made a number of suggestions toenable and encourage IS academicians to under-take research projects on topics of interest topractitioners and to then prepare articles relevantfor practice based on this research. We do recog-nize, however, that the primary force influencingIS academicians regarding their research, manu-script preparation, and journal selection behav-iors are the signals provided them by leading ISjournals: mission statements, review comments,and editorial decisions. If the IS discipline is tobecome more relevant, this change process mustbe fostered by our leading journals,

A much better balance among rigor and rele-vance is needed in the editorial review process.Our own experiences in a variety of editorialroles suggest that the most difficult entity to influ-ence regarding the reviewing processes is thebehavior of reviewers. Thus, journal editors andboard members must adopt proactive stances in

their support of relevance. Conscious effortshould be spent to (1) select referees likely tovalue both rigor and relevance and (2) ensurethat review commentaries and decisions addressa manuscript's relevance as well as its rigor.

Recommendation 9: Editors and editorialboards of all IS journals need to criticallyexamine their current postures, reviewingprocedures, and editorial decisions concern-ing the balance between rigor and relevancewith the goal of publishing manuscripts thatare characterized by both.

Conclusions

In our commentary, we have attempted to offersome specific recommendations to increase therelevance of research published in IS journals. InTable 1, we summarize the key dimensions ofrelevance and in Table 2, recommendations as tohow to achieve them. Note that in Table 1 we donot have a dimension labeled "useful," In ourview, as outlined in the section entitled "TheNature of Relevant Research," an article describ-ing research that is interesting, applicable, andcurrent has the potential to be useful for practi-tioners. This potential is fulfilled when the articleis accessible, i,e,, understood by practitioners.Therefore, our definition of relevant research isone that is potentially useful for, as well as acces-sible by, its intended audience.

Our view accommodates the need for rigor whileclearly distinguishing the "relevant academic"from the consultant. While both academicresearchers and consultants value the discoveryand application of new ideas and solutions, it isthe academic researcher who is more concernedwith issues of justification (i,e,, insuring that whatis being discovered and applied is in fact "cor-rect"). While practitioners seem to feign littledirect interest in justification, most do recognizethat justification allows them, even if indirectly,to better assess whether or not the prescriptionsthey receive are based on solid foundations.

The world of practice certainly has much to offerthe IS academic researcher. Practice providesstrong signals regarding what we should be study-ing. Invariably, it is practitioners, as they strive to



Table 1. Dimensions of Relevance

Category

Article's Content

Article's Style

Dimensions of Relevance

Interesting

Applicable

Current

Accessible

Description

Does IS research address the problenns orchallenges that are of concern to ISprofessionals?

Does IS research produce the knowledgeand offer prescriptions that can be utilizedby practitioners?

Does IS research focus on current, at thetime of publication, technologies andbusiness issues?

Are IS research articles able to beunderstood (in terms of tone, style,structure, and sennantics) by ISprofessionals? Are they written in a style thatprofessionals would enjoy reading?

stay ahead of their competitors, who discoverthrough trial and error (if only within a single con-text) the value of emerging IT-related innovations.Clearly, a strong symbiotic relationship existsbetween practice and research. Theories andmodels are judged on their predictive power forguiding practitioners, the quality of theories areenhanced by testing and revising them with datafrom the world of practice, and the cues andinsights offered by practitioners are assessedthrough rigorous examination across a variety ofcontexts.

Academic research is time consuming, due inpart to the necessity to adhere to rigorous stan-dards. We have no doubt that information sys-tems practitioners understand this well, sincethey too face the necessity of devising techni-cally correct, yet resilient, resolutions to com-plex issues for which there are no simple solu-tions. And, like IS practitioners, academics facemore interesting and important (research) chal-lenges than they can tackle given their limitedtime, effort, and monetary resources. So, howdo we proceed? We, as a community, mustengage in a dialogue with practitioners about:(1) their "critical success factors"; (2) the impor-tant challenges they confront on a regular orperiodic basis; (3) the important questions they

have been unable to find answers for; and (4)the issues that will be important to them threeto five years from now. Similarly, we as acade-mics, with insights from practitioners, mustdefine the "fundamental issues" of our disci-pline. We are just not convinced that a laissez-faire climate of "let a thousand flowers bloom"is in the best long-term interest of the IS acad-emic community.

We must make a concerted effort to communi-cate to practitioners how our research would berelevant to them. As well, we need to demon-strate, or at least describe, the extent to which theoutcomes of IS research have been and are usedby practitioners. Unfortunately, the IS field doesnot possess the evidence with which to illustratethe impact of its research. While other adminis-trative science disciplines have investigated howacademic research has diffused into the world ofpractice and influenced its inhabitants, we knowof no work in IS that has done so. This is animportant question that IS academics shouldinvestigate.

In closing, we wish to end on a positive note. Toreiterate, we are committed to and interested indoing good research. We strongly believe that thecontributions of IS academics to practice can and



Table 2. Recommendations to Attain Relevance

Topic Selection

Article's Purpose

Article's Readability

Editorial Process

Recommendations

1, Focus on future interests of key stakeholders.

2, Identify topics from IS practice.

3, Identify, as an academic community, the coreresearch issues that can influence practice inthe future.

4, Focus on the likely outcome (that can influencepractice) rather than on inputs (academic andintellectual challenges) when choosing a researchproject.

5, Develop cumulative, theory-based, context-richbodies of research to be able to make prescriptionsand be proactive.

6, Develop frames of references to organizephenomena and provide contingency approachesto managerial action.

7, Portray research outputs in ways practitioners canutilize to justify and rationalize IT related decisions.

8, Use clear, simple, and concise style in the write-up.

9, Set the goal of publishing manuscript as being bothrigorous and relevant.

Dimensions ofRelevanceAddressed

Interesting

Interesting

Current

InterestingApplicable

Applicable

Applicable

Applicable

Accessible

InterestingApplicableCurrentAccessible

should rise. The IS academic community has astrong research infrastructure, highly respectedjournals, a growing body of good measurementinstruments, and, as more cumulative work isdone, an expanding portfolio of versatile theoriesto enable us to react more quickly and provideprescriptions to practice. Furthermore, in the lastdecade. North American IS journals have beenless dogmatic about their emphasis on positivist(or "scientific") research. They are now more wel-coming of qualitative and case-oriented studiesthat are more likely to produce the contextual andlinguistic "hooks" so valuable in conveying theoutcomes of rigorous IS research such that itsvalue to practice is more easily recognized. For all

these reasons, and with some extra effort andattention by IS academicians, we foresee in thefuture even stronger cooperation between the twosolitudes of practice and academia.

AcknowledgementsWe thank Genevieve Basselier for her valuablecomments on this manuscript and LyndaApplegate for her excellent editorial guidance.

ReferencesAjzen, I, "The Theory of Planned Behavior,"

Organizational Performance and HumanDecision Processes (50), 1991, pp, 79-211,

14 MIS Quarterly VoL 23 No. 1/March 1999


Applegate, L, M,, McFarlan, F, W,, andMcKenney, J, L, Corporate Information SystemsManagement, Irwin, Chicago, 1996,

Astley, W, C , and Zammuto, R, F, "OrganizationScience, Managers, and Language Games,"Organization Science (3:4), November 1992,pp, 443-460,

Benbasat, I,, and Weber, R, "Rethinking Diversityin Information Systems Research," InformationSystems Research (7:4) , December 1996, pp,389-399,

Beyer, J, M,, and Trice, H, M, "The UtilizationProcess: A Conceptual Framework andSynthesisof Empirical Findings,"/\c/m/n/sfraf/VeScience Quarterly {27), 1982, pp, 591-622,

Brancheau, J, C, Janz, B, D,, and Wetherbe, J, C,"Key Issues in Information SystemsManagement: 1994-95 SIM Delphi Results,"M/SQuarter/y (20:2), June 1996, pp, 225-242,

Business Week. "Is Research in the Ivory Tower'Fuzzy, Irrelevant, Pretentious?'" October 29,1990, pp, 62-66,

Cheng, J, L, C, and McKinley, W, "Toward anIntegration of Organization Research andPractice: A Contingency Study of BureaucraticControl and Performance in ScientificSettings," Administrative Science Quarterly(28), 1983, pp, 85-100,

Daft, R, L,, and Lewin A, Y, "Can OrganizationStudies Begin to Break out of the NormalScience Straightjacket? An Editorial Essay,"Organization Science (1:1), 1990, pp, 1-9,

Davis, M, F, "That's Interesting: Towards aPhenomenology of Sociology and a Sociologyof Phenomenology," Philosophy of SocialScience (1), 1971, pp, 309-344,

Dearden, J, "MIS is a Mirage," Harvard BusinessReview, January-February 1972, pp, 90-99,

Fishbein, M,, and Ajzen, I, Belief, Attitude,Intention and Behavior: An Introduction toTheory and Research, Addison-Wesley,Reading, MA, 1975,

Galliers, R, D, "Relevance and Rigour inInformation Systems Research: Some PersonalReflections on Issues Facing the InformationSystems Research Community," in BusinessProcess Re-Engineering, Proceedings of theIFtP TC8 Open Conference on BusinessProcess Re-Engineering, B, C, Glasson, I, T,FHawryszkiewycz, B, A, Underwood, R, A,

Weber (eds,), Queensland, Australia, 1994,pp, 93-101,

Keen, P, W, "MIS Research: Reference Disciplinesand a Cumulative Tradition," Proceedings ofthe First International Conference onInformation Systems, E, McLean (ed,),Philadelphia, PA, 1980, pp, 9-18,

King, J, L,, and Applegate, L, M, "Crisis in the CaseStudy Crisis: Marginal Diminishing Returns toScale in the Quantitative-Qualitative ResearchDebate," Information Systems and QualitativeResearch, A, S, Lee, J, Liebenau and J, I,DeGross (eds,). Chapman & FHall, London,1997, pp, 28-30 (accessible from http://www, hbs,edu/app legate/cases/research/paper,html#patronage),

Nolan, R, L, "Managing the Computer Resource:A Stage FHypothesis," Communications of theACM (16:7), July 1973, pp, 399-405,

Price, R, L, "A Customer's View ofOrganizational Literature," in Publishing in theOrganizational Sciences, 2nd ed, L, L,Cummings and P, J, Frost (eds,). SagePublications, Thousand Oaks, CA, 1995, pp,98-107,

Robey, D, " Diversity in Information SystemsResearch: Threat, Promise and Responsibility,"Information Systems Research (7:4), December1996, pp, 400-408,

Saunders, C, "The Role of Business in ISResearch," Information Resources ManagementJournal, Winter 1998, pp, 4-6,

Sears, P,, and Pickler, L, "A Preliminary Study ofthe Practitioner Members of the Academy ofManagement," working paper, BaldwinWallace College, Summer 1996,

Taylor, S,, and Todd, P, "UnderstandingInformation Technology Usage: A Test ofCompeting Models, Information SystemsResearch (6:2), June 1995, pp, 144-176,

Zmud, R, "Editor's Comments," MIS Quarterly(20:2), June 1996a, pp, xxi-xxiii,

Zmud, R, "Editor's Comments," MIS Quarterly(20:3), September 1996b, pp, xxxvii-xxxviii.

About the Authors

Izak Benbasat is CANFOR Professor ofManagement Information Systems and associatedean at the Faculty of Commerce and BusinessAdministration, the University of British



Columbia, Vancouver, Canada, He holds a Ph,D,in management information systems from theUniversity of Minnesota, FHis research interestsare in understanding the managerial policiesinstrumental in the successful adoption of newinformation technologies, measuring informationsystems competence in line managers, evaluat-ing human-computer interfaces, measuring theeffectiveness of decision support tools, investi-gating the role of explanations in expert systems,and comparing methods for conducting informa-tion systems research.

Robert W. Zmud is Michael F, Price Professor ofManagement in the College of BusinessAdministration at the University of Oklahoma,His research interests focus on the impact ofinformation technology in facilitating a variety oforganizational behaviors and on organizationalefforts involved with planning, managing, anddiffusing information technology. He holds aPh,D, (management) from the University ofArizona, an M,S, (management) from M,I,T,, anda B,A,E, (aerospace engineering) from theUniversity of Virginia,

16 MIS Quarterly VoL 23 No. 1/March 1999

EMPIRICAL RESEARCH IN INFORMATION … RESEARCH IN INFORMATION SYSTEMS: THE PRACTICE OE RELEVANCE^...

Documents

Transcript of EMPIRICAL RESEARCH IN INFORMATION … RESEARCH IN INFORMATION SYSTEMS: THE PRACTICE OE RELEVANCE^...