A Research Publishing Checklist

Presented at the Research Paper Publishing & Presenting Workshop,At the Sixth Wuhan International Conference on E-Business (WHICEB

2007) , Wuhan, P.R. China, May 26-27 – Final Version

See: http://brianwhitworth.com/papers.html

Brian Whitworth, Massey University, Albany New Zealand

PhD Publishing

• Once PhD students first graduated then published

• Now most publish then graduate• Lets new researchers “cut their teeth”• Gives advisor and student a common

concrete goal• Good for CV and jobs

Research is …

• Complex – balances many aspects• Abstract – needs interpretation• Dynamic – a process not a productThe checklist approach:• “Chunks” knowledge into elements• Summary statements/examples• Academic document layout

Research Publishing Checklist

• For students and advisors

• Presents structured knowledge elements

• Not every element applies to everyone

• User must select what elements applies

• Some elements marked * are local, e.g. just for quantitative research

Respect the scientific community by reading, referencing and building on

other’s work

• Research is a social activity as well as an intellectual one

• Community members resolve disagreements by appeals to physical world data

• Critical to this community is respect for research colleagues by reading and quoting their research

OVERVIEW: Collegiality

Give reasons before conclusions, & derive conclusions from what

went before • Rationalization Intellect thinks backwards giving

“excuses” after results• Scientific argument Use intellect to argue

forwards, from present “facts” to conclusions. • We “follow the logic” wherever it leads. • Drive the arrow of thought forwards, create new

ideas, and predict the future

OVERVIEW: Scientific thinking

Academic form connects ideas together in a logical, consistent argument

– Introduce Topic– Review Theory – Describe Method – Analyze Data – Discuss Results

OVERVIEW: Academic Format

LogicalProgression

OVERVIEW: Academic Format

Topic/Problem

ABSTRACTWORLD

Research Method

Figure 1. A Research Logic

CONCRETEWORLD

Theory Review

Research Question

DataAnalysis

Discuss Results

• Use introduction to define theory scope.

• Use theory review to derive a research question

• Use research question to choose a method.

• Use method to create physical based data

• Use data analysis to form conclusions that answer the research question

• Use conclusions to make general theoretical and practical implications

OVERVIEW: Academic Format

Check for and fix spelling and grammar errors, especially early on

• Dont judge a book by its cover, but many do• Some assum those who canot spell correcty also

cannt think correct • Spelling or grammar errors, especially early on, as

they may bias reviewers to find faults. • Run a computer spell-check last thing, and have a

native language speaker read the document

OVERVIEW: Correct language

Check for and fix “red flag” statements that cannot be defended

against critical review

• Red flags - indefensible statements that cannot be sustained – e.g. “Technology improves efficiency” - some does not.

• Reduce indefensible statements– Provide more support - supporting references– Reduce statement strength e.g. “Technology can improve efficiency” – Moderate absolutes, e.g. “Some technology improves efficiency”– Hedge statement, e.g. “Many argue that technology improves

efficiency”– Make it a question, e.g. “Does technology improve efficiency?”

OVERVIEW: Indefensible statements

INTRODUCTION

Topic/Problem

Research Method

Theory Review

Research Question

DataAnalysis

Discuss Results

A title should describe the topic in an interesting way and invite the

reader to read on

• Title briefly conveys what the paper is about, e.g. research topic, dependant variable or contribution– ‘Websites” - too vague

– “The Usage of Websites”- describe the dependant variable

– “A Study of Factors Influencing Website Usage” - adds research contribution

• To also attract reader interest?– “Website Usage: Why people Come”

INTRODUCTION: Title

List authors in order of contribution not seniority, where each could present the

publication at a conference • First author - the person who did most of the work• Author order is based on contribution not seniority• Senior professors may list after their students • All co-authors should understand the paper well

enough to adequately present it at a conference

INTRODUCTION: Author(s)

Abstract should tell the reader the paper’s main purpose, results

and implications • Abstract should convey meaning

– e.g. NOT “This paper gathers data relevant to current theories

and analyses it to draw several critical conclusions” – Don’t state “conclusions were made”, state the conclusions!

1. Purpose e.g. “To study a new technology acceptance variable”

2. Findings e.g. “User control significantly increased user acceptance.”

3. Implications e.g. “Systems with user control are more accepted.”

INTRODUCTION: Abstract

State a practical problem which the research addresses

• Problems agitate and activate people and highlighting a problem invokes reader interest

• If there is no problem, why was this research done?

• Stating the problem your research addresses is a natural publication start point– e.g. cancer research is based on the problem of

cancer

INTRODUCTION: Problem

Choose one topic, and refer to it consistently through the paper

• State topic area early • Topic tells readers if your paper is in their

area of interest• Choose one topic, and use consistent words

to refer to it in the paper– Technology acceptance, technology diffusion

and technology evaluation are different topics

INTRODUCTION: Topic

State the research purpose in a single sentence within the first few pages

• Purpose statement builds on the topic/problem• States what the research aims to achieve• Editors expect to find a purpose statement within a

page or two – “This paper aims to …”

– “The goal of this research is …”,

• Have one purpose stated one way – not several differently stated purposes

INTRODUCTION: Purpose

State publication type (e.g. review, proposal or empirical study) early so readers know what to expect

• Scientific value sources: Data results, Logical analysis and Past research

• Theory papers suggest new theory models

• Reviews summarize the current state of the art

• Empirical papers present new research data

• Research proposals propose a method

INTRODUCTION: Publication Type

Give a brief background history introducing the research

• Background history helps “warm-up” and historical facts can “set the stage”

• Good papers always “tell a story”

INTRODUCTION: Background

State the target audience, i.e. who is expected to read this

paper • Research relevance is indicated by the people

interested in it, and distinct from its rigor or scientific quality

• Target audience statements like ‘This paper will interest software users.” help relevance

• Knowing your target audience also helps define the way the paper is written, e.g. is it for a technical or end-user audience?

INTRODUCTION: Target Audience

THEORY

Topic/Problem

Research Method

Theory Review

Research Question

DataAnalysis

Discuss Results

Define the boundaries of the research study and stay within that scope

• A paper’s scope is like a line around a territory, defining what is in and out of the paper

• Also state what the research is not about, and does not cover.

• Do not “wander freely” in areas of little relevance to your topic

THEORY: Scope

Analyze the literature by issues (agreements/contradictions), not as a sequential list of other’s research

• Honestly summarize current research in the field• Don’t give a sequential list of what others did, e.g.

A did this, B did that, C did the other, etc. • Integrate past research by finding themes,

agreements, conflicts, contradictions and gaps, e.g. compare and contrast many authors on some issue, then review another issue

THEORY: Literature Review

Clearly define the main topic construct/variable, which the research is primarily about..

• Make topic construct as clear as possible

• Quantitatively measured constructs are variable(s), e.g. the abstract construct “Usage” measured by number of web hits or time on site, is a variable

THEORY: Topic construct/variable

Distinguish what is causing from what is being caused in casual relations between constructs

• In causal effects, where constructs/variables affect each other– Those that are changed are dependant, e.g. web

usage– Those that cause change are independent,

• e.g. web site usage has causes (users, the site) and itself causes other effects (customer attitude – which in turn can also affect usage)

THEORY: Causal Effects

State the theory base (or conceptual framework) this research will use,

ideally with a diagram • Research without theory is weak • After presenting other’s work, choose the theory

base your research will use • Present the theory you will use:

– An existing theory

– An existing theory combination

– An existing theory modified by you

– A new theory of your own

THEORY: Theory base

A research question is a single sentence (clear), with many

answers (falsifiable), that allows feasible data collection (testable) • RQ is the conclusion of the literature review• Defined before the method – for one research

question one choose many methods• A single question • Puts a real choice, e.g. “Does fear make one

afraid?” is not a real question

THEORY: Research Question

Choose a suitable research type (exploratory, descriptive, correlational or explanatory) that is feasible to do

Exploratory, e.g. grounded theory - new topic

Descriptive, e.g. case study - evolving areas.

Correlational, e.g. longitudinal study - known but not controlled area

Explanatory research, e.g. controlled experiment – known and controllable area

Choose the right type for your topic

METHOD: Research Type

Control

Realism

A hypothesis is a statement whose opposite the data results can falsify

• Hypotheses restate the RQ as statements whose opposite research results can be falsified, e.g.– RQ: “Do subjects prefer polite software?”

– Hypoth: “Subjects prefer polite to impolite software”

– Null hypoth: “No response difference between polite and impolite software”

– Null hypothesis assumes a random data distribution, so one can estimate how unlikely the actual data result is

– If very unlikely (less than 5% or 1 % probability), the null hypothesis is rejected, and hypothesis is “supported”.

THEORY: Hypothesis*

METHOD

Topic/Problem

Research Method

Theory Review

Research Question

DataAnalysis

Discuss Results

Only present methodological theory for a new method, but briefly

• Briefly present theory of an uncommon method, e.g. – Interpretive method theory

– Conjoint analysis theory

• While your research theory needs extended coverage, your method theory does not– Readers familiar with factor analysis don’t want pages of

factor analysis theory copied from a textbook

METHOD: Methodological theory

Use qualitative and quantitative data (mixed-method approach) ..

• The principles of science apply to qualitative (e.g. language) and quantitative (numeric) data

• Constructs (qualitative) and Variables (quantitative) are different names for abstraction

• Qualitative data - better meaning but less precision• Quantitative data - more precise but less meaning• Justify your data type (based on the research goal)• Gather a complementary data type (mixed-method)

METHOD: Qualitative vs. Quantitative

Report how pilot testing improved the research method or tools

• New research is dangerous - one miscalculation can cause failure

• Pilot study tries out the method and research tools on a few cases to uncover problems

• Report how pilot data modified the method– e.g. how research interview script was improved

based on the pilot data

METHOD: Pilot Study

Describe the method logic that ensures the data really answers the

research question • Research design is the method logic that ensures that the

data collected answers the research question correctly, e.g. case study, repeated measures, longitudinal design, etc

• Each design has different requirements, e.g. repeated measures subjects might try first one software product then another – this requires the product order be randomized, to avoid order effects.

METHOD: Research Design

Randomly allocate subjects to a control group to reduce spurious

effects• Control group are subjects who get everything the

same except the treatment, e.g. medical research controls get a sugar pill instead of the new drug

• Controls for spurious effects• Control subjects must be randomly allocated, i.e.

don’t let subjects choose to be in the control group• Is important where subjects know they are being

observed, e.g. the “Hawthorne effect”

METHOD: Control Group*

State how you measured what you are trying to measure, and why you

chose this way to gather data• State how you actually measured or

investigated each variable/construct, as there are many ways to measure something

• Justify your measurement tools or method, whether taking qualitative or quantitative measures

METHOD: Measurement

Argue why the data is reliable, e.g. by precedence, test-retest or split-half checks

• Reliable measurements change little if measured at different times or with different people

• Qualitative studies use standard methods, keep detailed journals and analyze self-consistency to get reliability

• Test-retest reliability checks if a measure changes if done again, e.g. give a test then give it again

• Split-half reliability checks a measure’s internal consistency e.g. compare the first half with the second half of a test

METHOD: Reliability

Argue why the data is valid, with

content, criterion or construct validity • Measure’s validity - degree it measures what it is

supposed to, e.g. foot size is an invalid measure of IQ. – 1. Content validity - construct content is self evident in the

measure, e.g. “Did you enjoy the web site?” measuring web site enjoyment.

– 2. Criterion validity - validate measure against a valid one, e.g. validate web clicks against purchase data.

– 3. Construct validity - measure “hangs together” (convergent) and separates from other measures (discriminant).

• OR - use a tool someone else has tested and validated

METHOD: Validity

Specify the unit of research of your study, i.e. one data gathering “case” • The unit of research, or case, is the data collection

unit, e.g. for online group voting case could be:– The vote (choice)

– The individual (satisfaction)

– The group (agreement)

• Raw data tables - variable columns and row cases• Case affects N, e.g. 90 subjects could give 900 votes

(N=900), 90 satisfaction ratings (N=90), or 18 five person agreement ratings (N=18)

METHOD: Unit of Research

Describe in sequence the data collection procedure, with any tools

used, e.g. introduction script • Describe step by step data collection procedure e.g.

– “After entering the room subjects were read the study instructions (Annex A), then signed the consent form (Annex B)…”

• Procedure can be face-to-face, by telephone, by self-administered questionnaire, by website, etc.

• Describe procedure deviations, e.g. subjects who left the room or talked to a friend

METHOD: Procedure

Describe the task and any instructions involved

• Describe exactly what subjects had to do

• Describe any training given (to reduce initial skill variations)

• Describe the script read out or given to all subjects

METHOD: Task*

Design questions that are understandable, unambiguous,

unbiased, answerable and not offensive

• Good questions reduce error variance from subjects who just “tick the boxes”– 1. Understandable, e.g. simple words and grammar– 2. Unambiguous, e.g. don’t use double-barreled questions– 3. Unbiased, e.g. “Don’t you agree that …?” is biased– 4. Answerable, e.g. an ordinary person may not know

“How good is the national budget?”– 5. Not offensive, e.g. some may not answer some

questions

METHOD: Question Design*

Design response scales that are easy to use, unbiased, exhaustive and

sensitive • Multi-choice response scales should be

– 1. Easy to use– 2. Unbiased (do not imply a preferred answer)– 3. Exhaustive (cover all answer choices)– 4. Sensitive (produce varied responses)

• Above about 5-7 options, scale sensitivity and reliability increase little

METHOD: Response Scales*

State why sample data may have population implications, e.g. sample is unbiased, big enough, and represents the

population • Collect data from a sample, but conclude about a

population, e.g. study of students concludes about

“people” • To generalize from sample to population, it must be:

– 1. Big enough,

– 2. Representative of the population

– 3. Unbiased by the researcher, e.g. subjects chosen by random numbers

METHOD: Sample selection

Check the sample is big enough for the question(s) asked of the data

METHOD: Sample size*

Perceived Usability

High Medium Low

Male

Female

A 2x3 study with 6 cells needs at least 6x15 = 90 subjects

Describe your research method well enough so another researcher could

repeat it• Replicability means the research can be

repeated by another researcher

• Method must be complete

• Give copies of any research tools used, like questionnaire, task script, or subject instructions

METHOD: Replicability

RESULTS

Topic/Problem

Research Method

Theory Review

Research Question

DataAnalysis

Discuss Results

Use research rigor to avoid Type I false claim errors, and research sensitivity to avoid Type II null

result errors • Type I error- a false result from insufficient research rigor,

i.e. an error of commission• Type II error misses a true result from insufficient power, i.e.

an error of omission• Research must be powerful to gain true results and rigorous

to avoid false results. Improve research power by:– Enhancing responses (e.g. motivating subjects with a competition)– Reducing subject error (e.g. subject training or clearer questions)– More powerful statistical methods.

• Unfortunately reducing one error type increases the other

RESULTS: Error Type

State how raw data was converted to descriptive data, and how missing values were handled

• Describe how the study raw data converted into descriptive data, e.g. transcribing tapes, translating computer data

• What was left out? Describe number of missing values and how dealt with

• Missing values can be Nil Response (NR) or did not reply, or Not Applicable (NA) response or could not reply

• e.g. “Why did you buy your mobile phone?” – NR - Ignores and walks away – NA – “I don’t have a mobile phone”

RESULTS: Data Conversion

Present sample demographics compared to population

demographics • To show the sample is representative, give

sample demographics, e.g. number of males vs. females etc.

• Compare the properties of the sample to the properties of the population

RESULTS: Sample Demographics

Analyze data according to type (qualitative, interval, ordinal, categorical)

• Qualitative data summarized by human interpretation• Quantitative data can be

– 1. Interval, a continuous number series like 1,2,3,… – 2. Ordinal, a series of ranks, like 1st, 2nd, 3rd, etc – 3. Categorical, set of categories - PC, Laptop and Handheld.

• Summarized by frequency, central tendency, variability

• Data type affects data power– e.g. measure computer experience in years (direct - interval

data), or self-rated Expert, Competent, Novice categories

RESULTS: Data Type

Give a descriptive summary of the data (e.g. N, mean and variance) and

comment on it • First summarize data descriptively - as descriptive

quotes or numbers • Thousands of data points can be described by

frequency (N), mean and variance• Deduce possible effect direction from descriptive

data, e.g. that people prefer more usable computer systems

• Round off the data appropriately to error, e.g. 3.74 not 3.742356

RESULTS: Descriptive summary

Select analytic statistics for strength (e.g. Pearson’s r, Eta, Gamma, Phi,

Lambda) and significance (e.g. t-test, F-test, Chi-squared) based on data type

• Analytic statistics can show strength or significance

• For a two-variable relation– significance is whether they correlated by

chance, ie is there an effect?– strength is how much the variables correlate

RESULTS: Analytic statistics*

State and justify analysis assumptions

• All statistics have assumptions

• e.g. the F-test assumes:– Responses are normally distributed– Measure variances are the same (homogenous)– The measure errors are independent.

RESULTS: Assumptions

Argue each finding individually from the data analysis

• At each step of the data analysis, various findings are concluded, e.g. a list of hypotheses supported or not supported.

• Argue each finding individually

RESULTS: Findings

Tables need a clear title giving topic and other variables, have row/column

headings, and to be referenced in the text

• Table title: Usually topic variable by one or more causal variables, e.g. “Usage by Experience”

• Need row/column headings • Must reference in the text, e.g. See Table 1• Two types of result table

– 1. Frequency Tables. Table cells show frequencies

– 2. Mean Tables. Table cells show response means

RESULTS: Tables*

RESULTS: Frequency Tables*

Group Disagreement

NInformation Exchanged 0.0 0.4 0.6 0.7 0.8 0.9

I Blind 8.8% 18.1% 25.9% 20.8% 22.2% 4.2% 216

II Group Aware 66.7% 22.1% 7.8% 3.4% 0.0% 0.0% 204

III Confidence Aware 65.3% 23.1% 6.9% 3.7% 0.9% 0.0% 216

Table1. Group disagreement by information exchanged

In frequency tables, show the topic variable frequency percentages across each row, and varying between rows

RESULTS: Mean Tables*

In mean tables, show the topic variable as a cell mean, varying by row/column variables

Confidence

Disagreement (D)1

Very Not Confident 2

Not Confident

3Fairly Confident

4Confident

5Very Confident

Agree with everyone 0% 0% 2.5% 0.8% 0%

Disagree with one 0% 2.3% 3.4% 0.9% 2.0%

Disagree with two majority

15.4% 4.7% 5.9% 7.0% 5.0%

Disagree with three minority

52.6% 43.9% 40.2% 22.2% 28.8%

Disagree with all 87.5% 76.0% 72.7% 64.9% 62.0%

Table 2. Vote change by prior vote confidence and disagreement

Use graph types (line, bar or pie) .. depending on data type, and label

each graph axis clearly

• Graph type depends upon data types– Both variables interval type - line graph, with

horizontal X-axis the independent variable and vertical Y-axis the dependant variable

– If an Interval dependant variable broken down by a nominal IV - a bar graph

– Pie graph is for categories that add up to 100%.

RESULTS: Graphs*

00.10.20.30.40.50.60.70.80.9

Blind Group Aware ConfidenceAware

D

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Vote 1 Vote 2 Vote 3

D

Summarize the main conclusions of your study in a numbered or

bulleted list, and answer the research question

• Summarize the main conclusions to lead into the general discussion to follow

• It should clearly answer the research question put earlier

DISCUSSION: Summarize conclusions

DISCUSSION

Topic/Problem

Research Method

Theory Review

Research Question

DataAnalysis

Discuss Results

Distinguish the value this research adds, apart from what others have

done• Paper should be clear on its research

contribution, i.e. what it adds that is new• Don’t confuse your work with past research by

others:– To add knowledge currently lacking– To resolve a theory conflict– To specify more detail on that already known– To summarize current work

• Answer what value does this paper add?

DISCUSSION: Research Contribution

Declare rather than hide your research limitations, i.e. honestly

criticize your own research• Limitations are the research weaknesses, the

assumptions of your scientific argument • All research has limitations, as every argument has

assumptions- a “foolproof” argument is simply one with undeclared assumptions

• Declare limitations than to hide them, so then they can be taken into account.

• Knowledgeable readers will see the limitations anyway, and wonder why you did not mention them

DISCUSSION: Limitations

Discuss probable and possible implications of the findings for

both theory and practice • Based on your conclusions, expand your

research into the big theory picture, noting its implications:

– To challenge some theories or support others– To suggest new points of view– To change practice by enabling new applications or

practices.

• How does this research fit with current trends?

DISCUSSION: Implications

Suggest future research possibilities to help other

researchers• Help the research community by

suggesting how others could build upon your work

• Imagine future research possibilities and envision future outcomes.

DISCUSSION: Future research

Acknowledge those who have helped you

• It is considerate to acknowledge anyone who helped you, whether an assistant who helped gather data, or a colleague who gratuitously commented

• It can be important to acknowledge any grant support.

DISCUSSION: Acknowledgements

Ensure the references reflect the quality, breadth and up to date

nature of your publication• Reviewers may judge publication quality

by the references:– Are journal article page numbers given? – Are author names spelled correctly? – Are major authors in the field referenced? – Are the references up to date? – Are there journal references, not just books,

web sites or magazines?

DISCUSSION: References

Final Points

• Is the publication relevant? –Practically/theoretically useful? (Introduction, Theory)

• Is the publication rigorous? –Scientifically correct? (Method, Results)

• Is the publication generalizable? –Widely applicable? (Theory, Discussion)

• Is the publication logical? – Consistent and logical sequence of ideas? (All)

• Is the publication well written? – Interesting and understandable? (All)

DISCUSSION: Checklist

Ways to use the Checklist

• Give to students along with sample papers• Focus on a particular element in discussion

– Discuss element, then review the paper with respect to it.

• If student’s selectively ignore a difficult or inconvenient academic issue (for them), use the checklist to point it out

• Use to structure thesis discussions, and give student a sense of progress

• Shows how early elements lead to later ones, so students can start to write a “thesis.doc” right away