A Tale of Two Manuscripts Joey F. George College of Business Florida State University, USA August...

A Tale of Two Manuscripts

Joey F. GeorgeCollege of Business

Florida State University, USAAugust 2006

© 2006 Joey F. George

Overview

The Research Process The Publication Process Two Examples:

Biros, George & Zmud, MISQ, 2002 George, Marett & Tilley, under review

Closing Thoughts

Thanks to Carol Saunders for letting me borrow her presentation title.


The Research Process

Motivation Getting Started Planning Execution Data Analysis


Motivation

What do you want to study? Why? What’s your Research Question? “Nothing is as practical as a good theory”

Ascribed to Kurt Lewin A good theory makes life so much easier Theory drives literature review, research

model, hypotheses, & most importantly the interpretation of the results


Getting Started

Internal Review Board (IRB) / Human Subjects Committee (HSC) approval

Grant preparation & pursuit (if applicable) Initial research design Gaining access to:

Sites Subjects Software


And the rest….

Planning Completing the research design Logistics

Execution Pilot studies Data collection

Data Analysis


Research & Publishing

We often focus on publishing because that is what counts

Academic work is product-oriented, not process-oriented, & publications are the product

But without good research on interesting and timely topics, there is no meaningful publication….


The Publication Process

Learn from your own reviewing: Understanding how the process works Getting insights into what is good & is not good

Polish: Presentations & peer review Polish: Consider hiring a language editor Targeting & Selling Submission Monitoring the review process


Targeting & Selling

Targeting a journal (or conference) Checking out various publication outlets Read the articles in the outlet you choose

Framing the question (& the answer) Identify your audience Write for your audience


Submission

Follow instructions for authors & format accordingly

Choose reviewers & editors if you are allowed Get an acknowledgment of receipt


Monitoring the Review Process

Some journals allow you to check the status of your submission in near real-time

Some do not: Contact the editor if you have not heard anything in 3 months This is not rude – it is the editor’s job to keep

things moving Editors are human too – sometimes things get

lost or don’t see action unless the editor is reminded


Two Examples: Manuscript One

Biros, D., George, J F., and Zmud, R.W. “Inducing Sensitivity to Deception in Order to Improve Decision Making Performance: A Field Study.” MIS Quarterly, 26(2), June 2002, 119-144


Timeline

Data collected Winter 1997-98

Biros defends June 1998

MS first sent to MISQ October 1999

SE returns paper w/o review November 1999

Revised MS back to MISQ March 2000

Reviews back: R&R End of May 2000

Revised MS (r1) back to MISQ November 2001

Reviews back: R&R January 2002

Revised MS (r2) back to MISQ March 2002

SE conditionally accepts paper March 2002 (one week later)

SE accepts paper April 2002

Paper published June 2002 (MISQ 26, 2)*

* very unusual


Two Examples: Manuscript Two

George, J.F., Marett, K., and Tilley, P. “The Effects of Warnings, Computer-based Media, and Probing Activity on Successful Lie Detection.” Submitted (now) to the Western Journal of Communication


Timeline

Data collected Winter 2002-03

MS first sent to MISQ January 2004

Reviews back: R&R May 2004

Revised MS (r1) back to MISQ August 2004

Reviews back: R&R February 2005

Revised MS (r2) back to MISQ May 2005

Reviews back: R&R September 2005

We withdraw the paper November 2005

New version of paper to CM December 2005

Reviews back: Rejection March 2006

New version of paper to WJoC May 30, 2006


Timelines Comparison

MISQ 2002 paper

Resume study paper


Biros Paper Study

RQ: Do different strategies affect success of detecting bad data in a database?

3 x 4 design; quasi-experimental field study 3 levels of expertise: novice, officer, specialist Control & 3 treatments: warned, trained, warned

& trained Conducted at USAF training base N = 205 20 scenarios 15 scenarios 8 scenarios


Biros Paper: 1st Model

DeceptionDetectionAbility

H1-

Arousal

UserExperience

in the Domain

DetectionTraining

ArtifactTruth Bias H2

H3

H4

+

+

+


Biros Paper: Initial Result

The manuscript could have been rejected, but instead the SE (consulting with an AE) sent the paper back & asked for a new version to be submitted

Detailed 5 page review on what to fix At this point, in revising, we switched from

ANOVA-based analysis to PLS


Biros Paper: 2nd Model

Sensitivity to Deception

Domain Experience

Task PerformanceSuccessful Deception Detection

H1

H5

H2

H3

H4

False Alarms

H6

H7


Biros Paper: 1st Review

Reviewers: R1: reject R2: major revisions R3: minor revisions AE & SE: revise & resubmit

Switch from 15 scenarios to 8 R2 wanted to see “discriminability” added Revision took 18 months


Biros Paper: From the SE

“Could I please indicate, however, that you need to evaluate carefully whether you should proceed with a revision and re-submission to the MIS Quarterly. I agree with the Associate Editor that it is unclear whether you can address some of the concerns raised by the Reviewers satisfactorily.”

“You need to take care, however, that a revised paper does not end up looking like a response to reviewers because it is filled with caveats and perhaps incidental/ancillary information. ”


Biros Paper: More from the SE

“Please be assured that I strongly believe the absence of statistical significance is not a cause for rejecting a paper, but the onus is on the researchers to show that lack of support for the hypotheses does not reflect experimental inadequacies.”

“Please give some careful thought to Reviewer 1's comments about the lack of an "ah hah" factor.”

“…the revisions are high risk.”


Biros Paper: 3rd Model

Sensitivity

Detection Success Ratio

Experience

False Alarms Proportion

Average time per scenario

Number CorrectRatio

R2 = .193 R2 = .138

R2 = .114

R2 = .198

.439***

.298**

.220*

.303***

-.181 *

-.150*

.337***


Biros Paper: 2nd Review

Reviewers: R1: Revise R2: Publish R3: Minor revisions AE/SE: Revise & resubmit

One construct, sensitivity, gets split into 3 Now have 14 hypotheses Revision prepared in only 2 months


Biros Paper: From the SE

“In paragraphs 3-6 of his/her report, the AE summarizes your findings and challenges you to show that they add sufficient value to our existing knowledge to warrant publication in the MISQ. I understand his/her concerns, and I agree that the ‘value-add’ provided by your current findings is still somewhat problematical. I am willing to ‘live with’ this concern, however, because with suitable revisions I believe your paper can still make sufficient contribution to warrant publication.”


Biros Paper: Final Model


Important Considerations

Review excerpts illustrate 3 common issues: Does the paper make enough of a

contribution? Are the findings interesting & novel (ah-hah)? Is there “enough” statistical significance to

warrant publication? Also, take care to revise a manuscript so it

does not become more of “response” to the reviewers than a research paper


George Paper Study

RQ: How does deception detection success vary with media and suspicion?

4 x 2 factorial experimental study 4 types of computer media (e-mail, chat, chat

with audio, and audio only) 2categories of induced suspicion, present or

absent 78 dyads (156 individuals)


George Paper: 1st Model

No model, just 2 hypotheses:

media

suspicion

detectionaccuracy

NOTE: Model not in the manuscript, but this is what it would have looked like.


George Paper: 1st Review

Reviewers: R1: reject R2: minor revisions R3: major revisions AE & SE: revise & resubmit

In response, we greatly expanded coverage of the deception literature

Added 3 more “exhibits” for a total of 5 We separated out the dependent measures of

accuracy to become awareness & detection


George Paper: From the AE

“I believe that the problems that exist are quite fixable.”

“Reviewer 1 questions the purpose of including the warning construct in the model…. It would also be appropriate to develop such a hypothesis of moderation by/with media, given the submission of the manuscript to an MIS journal.”

“Reviewers 1 and 3 raise concerns with regard to both metrics employed to assess deception detection.”


George Paper: 2nd Model

No model, just 5 hypotheses:

NOTE: Model not in the manuscript, but this is what it would have looked like.

media

warnings

false alarms

successfuldetection

awareness


George Paper: 2nd Review

Reviewers: R1: Major revisions R2: We lost R2 in this round R3: Reject AE & SE: Revise & resubmit

In response to the AE, we introduced a totally new model

Changed measures from proportions to counts & used ANCOVA analysis



“Both reviewers concede that the manuscript has the potential to make a contribution to the literature. I myself see this as a strong potential and would like to see it come to fruition.”

“R1 is concerned that three of the five hypotheses are unrelated to media characteristics…. R1 feels that the contribution offered by this work is quite limited.”

“Dimensions [of media] may be combined into just two sets of distinctions – variety/feedback (audio higher than email and chat) and rehearsal/permanence (email and chat higher than audio).”


New Models….

AwarenessDetection

False Alarms

Variety/ Feedback Rehearsal/

Permanence

+ + +

–x x

Warning+

Awareness

Detection

False Alarms

Variety/ Feedback

Rehearsal/ Permanence

+ ++

–x x

Suggested by the AE

OR


George Paper: 3rd Model


George Paper: 3rd Review

Reviewers: R1: Reject R3: Minor revisions AE & SE: Revise & resubmit

The reviewers essentially reverse their decisions from the prior version

The AE calls for yet another new model We decided to withdraw the paper



“I think that there are still significant flaws in the conceptualization and I too wonder about the appropriateness of the manuscript (in its current form) for an IS journal.”

“I think that this needs to be a really good revision - with considerable thought going into the conceptualization. Otherwise, I feel that I will be unduly imposing on the reviewers’ goodwill by asking them to look at the manuscript yet again.”

And the AE suggests yet another new model…


The Next New Model

Warnings Awareness Detection Accuracy

Media Characteristics

ReprocessabilitySynchronicity


Reworking the George Paper

In trying to determine if we could revise the paper according to the MISQ reviews, we re-thought and re-conceptualized the entire effort

Changed to SEM analysis with PLS We had an essentially new paper, and we

decided it was not suitable for an MIS journal We sent it instead to Communication

Monographs


George Paper: 4th & 5th Model

warnings

medium

probing

perceivedhonesty

lie detection

H1

H2

H3

H4

H5

H6

H7


George Paper: 4th Review

Different review system & structure Two reviewers, no AE R1 is ambivalent & suggests a revision R2 says the paper should be rejected because

“the results are inconclusive” R2 also says we should have used SIDE

theory instead of MRT EIC rejects the paper but says it could be

published at other communication journals


George Paper: From the EIC

“The context of this research is quite interesting and you have some intriguing findings…. Overall, the study seems carefully designed and executed.”

“One review is ambivalent -- the reviewer leaves open the possibility of a revision…. The other review recommends against publication in CM.”

“[As] the rationale does not take into account the current status of debate over channel effects, the theoretical contribution of the findings is unclear.”


Important Considerations

Importance of the review team: Senior editors and associate editors do not

always provide strong, consistent guidance Some reviewers are tenacious and will not let go

of a perceived problem There is always the issue of contribution Dangers of interdisciplinary work Journals in other fields do not follow the same

review processes (or philosophies) we do


Closing Thoughts

Do the research first Never submit a first or early draft The review team – something you have little

control over – is a key factor in publishing success

Never give up on something you believe in

A Tale of Two Manuscripts Joey F. George College of Business Florida State University, USA August...

Documents

Transcript of A Tale of Two Manuscripts Joey F. George College of Business Florida State University, USA August...