Seminar on Enterprise Software - ut

Seminar on Enterprise Software

Fabrizio Maria Maggi

[email protected]

(based on lecture material by Raimundas Matulevičius and Dietmar Pfahl)

Writing Good Software Engineering

Research Papers

by Mary Shaw

Proceedings of the 25th International Conference on Software

Engineering, IEEE Computer Society, 2003, pp. 726-736.

presented by Fabrizio Maria Maggi

University of Tartu, Estonia

A good research paper or thesis should

answer a number of questions• What, precisely, was your contribution?

– What question did you answer?

– Why should the reader care?

– What larger question does this address?

• What is your new result?

– What new knowledge have you contributed that the reader can use elsewhere?

– What previous work (yours or someone else’s) do you build on? What do you provide a superior alternative to?

– How is your result different from and better than this prior work?

– What, precisely and in detail, is your new result?

• Why should the reader believe your result?

– What standard should be used to evaluate your claim?

– What concrete evidence shows that your result satisfies your claim?

Shaw M.: Writing Good Software Engineering Research Papers,

Proc. of the 25th International Conference on Software Engineering,

IEEE Computer Society, 2003, pp. 726-736.

3

A good research paper or thesis should

answer a number of questions• What, precisely, was your contribution?

– What question did you answer?

– Why should the reader care?

– What larger question does this address?

• What is your new result?

– What new knowledge have you contributed that the reader can use elsewhere?

– What previous work (yours or someone else’s) do you build on? What do you provide a superior alternative to?

– How is your result different from and better than this prior work?

– What, precisely and in detail, is your new result?

• Why should the reader believe your result?

– What standard should be used to evaluate your claim?

– What concrete evidence shows that your result satisfies your claim?

� If you answer these questions clearly

� You will probably communicate your result well

� Your result represents an interesting, sound, and

significant contribution

� You will have a good chance of getting thesis accepted

for publication in a conference or journal

� You will have a good chance of publishing the paper (or

few) in a prestigious conference or journal

� If you answer these questions clearly

� You will probably communicate your result well

� Your result represents an interesting, sound, and

significant contribution

� You will have a good chance of getting thesis accepted

for publication in a conference or journal

� You will have a good chance of publishing the paper (or

few) in a prestigious conference or journal




4

What, precisely, was your

contribution?




5

What kinds of questions do software

engineers investigate?




6

What do thesis committees look for?

• A clear statement of the specific problem you solved

• The question about software development you answered

• An explanation of how the answer will help solve an important software engineering problem




7

What is your new result?




8

What kinds of results do you produce?




9

What do thesis committees look for?

• Interesting, novel, exciting results that

Significantly enhance the ability

• to develop and maintain software

• to know the quality of the software

• to recognize general principles about software

• to analyze properties of software




10




11

What’s new here?




12

What has been done

before?

How is your work

different or better?

Questions that committee asks about

your thesis

What, precisely, is the result?

• If you introduce a new model

• be clear about its power

• empirical data, formal semantics, mathematical principles

• If you introduce a new metric

• define it precisely

• If you introduce a new architectural style, design pattern, or

similar design element,

• treat it as if it were a new generalization or model

• If your contribution is principally the synthesis or integration of

other results or components

• be clear about why the synthesis is itself a contribution




13

Questions that committee asks about

your thesis

What, precisely, is the result?

• If your paper is chiefly a report on experience applying

research results to a practical problem

• say what the reader can learn from the experience

• If a tool plays a featured role in your paper

• what is the role of the tool?

• If a system implementation plays a featured role in your paper

• what is the role of the implementation?

• archiecture, implementation technique, capabiliy, performance,

system is a result itself




14

Why should the reader believe your

result?




15

What do committees look for?Why should the reader believe your result?

• If you claim to improve on prior art

– compare your result objectively to the prior art

• If you used an analysis technique

– follow the rules of that analysis technique

• If the technique is not a common one

– explain the technique and standards of proof, and be clear about

your adherence to the technique




16

What do committees look for?Why should the reader believe your result?

• If you offer practical experience as evidence for your result

– establish the effect your research has

• If you performed a controlled experiment

– explain the experimental design

• If you performed an empirical study

– explain what you measured, how you analyzed it, and what you concluded

• If you use a small example for explaining the result

– provide additional evidence of its practical use and scalability




17

Useful Material for Section: Related Work

Systematic Literature Survey – Protocol

• An example: https://arxiv.org/pdf/1705.02288.pdf

• Background: The rationale for the survey.

• Research questions that the review is intended to answer.

• Search strategy that will be used to search for primary studies

– including search terms and

– resources to be searched.

Resources include

– digital libraries,

– specific journals, and

– conference proceedings.


• Background: The rationale for the survey.

• Research questions that the review is intended to answer.

• Search strategy that will be used to search for primary studies

– including search terms and

– resources to be searched.

Resources include

– digital libraries,

– specific journals, and

– conference proceedings.

Software Engineering Digital Libraries

•Scopus

•Web of Science

•Google scholar

•IEEExplore

•ACM Digital library

•SpringerLink


• Study selection criteria

– Study selection criteria are used to determine which

studies are included in, or excluded from, a systematic

review


• Study quality assessment checklists and procedures

– The researchers should develop quality checklists to

assess the individual studies

• Data extraction strategy

– Design data extraction forms to accurately record the

information researchers obtain from the primary studies

– Linked to the research questions


• Synthesis of the extracted data

• Collating and summarising the results of the included primary studies

– Descriptive (non-quantitative)

– Quantitative

How bibliography needs to be organized?

References should appear as

• Book with one author:

– Author, A. A. (2005). Title of work.

Location/City, State: Publisher.

• Book with two authors:

– Author, A. A., & Author, B. B.

(2005). Title of work.

Location/City, State: Publisher.

• Book with more than two

authors:

– Author, A. A., Author, B. B., &

Author, C. C. (2005). Title of

work. Location/City, State:

Publisher.

• Journal article:

– Sawyer, S., & Tapia, A. (2005).

The sociotechnical nature of

mobile computing work: Evidence

from a study of policing in the

United States. International

Journal of Technology and

Human Interaction, 1(3), 1-14.

• A publication in press:

– Junho, S. (in press). Roadmap for

e-commerce standardization in

Korea. International Journal of IT

Standards and Standardization

Research.


References should appear as

• Edited book:

– Zhao, F. (Ed.). (2006). Maximize

business profits through e-

partnerships. Hershey, PA: IRM

Press.

• Chapter in an edited book:

– Jaques, P. A., & Viccari, R. M.

(2006). Considering students’

emotions in computer-mediated

learning environments. In Z. Ma

(Ed.), Web-based intelligent e-

learning systems: Technologies

and applications (pp. 122-138).

Hershey, PA: Information Science

Publishing.

• Report from a university:

– Broadhurst, R. G., & Maller, R. A.

(1991). Sex offending and

recidivism (Tech. Rep. No. 3).

Nedlands, Western Australia:

University of Western Australia,

Crime Research Centre.

• Published proceedings:

– Deci, E. L., & Ryan, R. M. (1991).

A motivational approach to self:

Integration in personality. In R.

Dienstbier (Ed.), Nebraska

Symposium on Motivation: Vol.

38. Perspectives on motivation

(pp. 237-288). Lincoln: University

of Nebraska Press.


References should appear as• Unpublished doctoral

dissertation or master’s

thesis:

– Wilfley, D. (1989). Interpersonal

analyses of bulimia: Normal-

weight and obese. Unpublished

doctoral dissertation, University of

Missouri, Columbia.

• A presented paper:

– Lanktree, C., & Briere, J. (1991,

January). Early data on the

Trauma Symptom Checklist for

Children (TSC-C). Paper

presented at the meeting of the

American Professional Society on

the Abuse of Children, San

Diego, CA.

• Web site:

– VandenBos, G., Knapp, S., &

Doe, J. (2001). Role of reference

elements in the selection of

resources by psychology

undergraduates. Journal of

Bibliographic Research, 5, 117-

123. Retrieved October 13, 2001,

from http://jbr.org/articles.html


In-Text Citations• In-text citations should appear with author surname followed by

publication year in parentheses

(Brown, 2002)

• Citing several references in-text:

– In most organizations, data resources are considered to be a major

resource (Brown, 2002; Krall & Johnson, 2005; Smith, 2001).

– Brown (2002) states that the value of data is recognized by most

organizations

– “In most organizations, data resources are considered to be a major

organization asset” (Smith, 2001, pp. 35-36) and must be carefully

monitored by the senior management.

– Brown (2002) states that “the value of data is realized by most

organizations” (p. 45).


In-Text Citations

• If you have organized the citations with number in

brackets

– In most organizations, data resources are

considered to be a major resource [15; 30; 84].

Useful Material for Section: Evaluation

Research Methods

• Experiment

– Controlled Experiment

– Quasi-Experiment

• Case Study

– Descriptive Case Study

– Exploratory Case Study

– Confirmatory Case Study

• Survey

– Questionnaire-based

– Literature-based

Case Study – Variants

• Descriptive Case Study

– Purely observational / Focus on “What happens?”

• Explorative Case Study

– Initial investigation of some phenomena to derive new

hypotheses and build theories / Focus on “What and Why?”

• Confirmatory Case Study

– Start out with a given theory and try to refute it, ideally with a

series of case studies covering various contexts

More on Case Study design (SE Group at Lund University):

http://serg.cs.lth.se/education/case_study_research/

Case Study – Characterisation

� Characteristics:

– When to use? --> When 'rich'

information is requested

– 'Rich' (in-vivo=in the living),

– More focus on qualitative data -->

allows for better understanding of

conditions under which a

technique/tool works

� Issues:

– Important: Proper case selection /

clearly stated research question /

clearly defined framework for

interpreting the observations

– 'Generalisability' (1 case --> only 1

context)

� Definition:

– An empirical enquiry that

investigates a contemporary

phenomenon within its real-life

context (in-vivo=in the living),

especially when the boundaries

between phenomenon and context

are not clearly evident.

� Example:

– Experienced practitioners

communicate about software

design in many different ways (and

other than, say, undergraduate

students). Thus, one may conduct

a case study in a local company.

Research Methods

• Experiment



• Case Study




• Survey



Survey – Characterisation

• A survey is a data collection method or tool used to gather

information about individuals in order to identify the

characteristics of a broad population

• The defining characteristic is the selection of a representative

sample from a well-defined population with the aim to

generalise from the sample to the population.

• Usually conducted with questionnaires, but can also involve

structured interviews or data logging techniques

• Example:

– Investigate to what extent, for which purpose, by which

companies, and by whom within the companies, UML

diagrams are used.

Survey – Characterisation (cont'd)

When to use it?

– Either at start of research to get an understanding of the

current situation (

– or at the end of a research phase to see the

impact/acceptance/etc. of a new method/technique/tool

Issues:

– 'Superficial' --> no explanation / no causality --> not

suitable for hypothesis testing

– 'Generalisability' of results depends on the choice of

population and 'response rate', as well as validity and

reliability of the data collection instrument

Survey – Example

What?

Research Questions:

- How is Agile practiced at Microsoft?

– i.e. What do engineers do?

- How do engineers feel about it?

– i.e. Do they like it?

Who, Where, and When?

Microsoft (worldwide, 2006)

Anonymous survey sent to 2821 engineers

• 10% random sampling of all developers, testers, program managers at Microsoft in October 2006

487 valid responses

• 44% developers, 28% testers, 17% program managers

Source: Andrew Begel and Nachiappan Nagappan,

Usage and Perceptions of Agile Software Development

in an Industrial Context: An Exploratory Study, in First

International Symposium on Empirical Software

Engineering and Metrics, IEEE Computer Society,

September 2007

Why?

• Many agile

approaches

exist – what's

in it for

Microsoft?

Survey – Example (cont'd)

Agile practice

penetration at

Microsoft

Survey – Example (cont'd)Qantitative Results (Highlights)

• 33% of respondents (spread across

divisions) report their team uses Agile

methodologies.

• They mainly use Scrum (68%).

• Used for many legacy products.

• Agile usage does not appear to depend

on team co-location.

• Test-driven development and pair

programming are not very common.

Qualitative Results (Highlights)

• MS engineers who have used Agile like it

for their local team, but not necessarily

for their organization.

• They worry about scale, overhead, and

management buy-in.

Perceived benefits (687 comments, 44 themes)

Perceived problems (565 comments, 58 themes)

Research Methods

• Experiment



• Case Study




• Survey



Karli Oruste, Process Mining in Industry, 2017

Allar Soo, Automated Process Discovery: A

Literature Review and a Comparative Evaluation

with Domain Experts, 2017

Research Methods

• Survey



• Experiment



• Case Study




Controlled Experiment – Characterisation

• An investigation of a testable hypothesis where one or more

independent variables are manipulated to measure their effect on one

or more dependent variables.

• In Software Engineering, typically, experiments require human

subjects to perform some task.

… …

Treatments

(Interventions)

Independent

Variables

Dependent

Variables

E

C

Controlled Experiment – Simple Example

• Independent Variable: UML diagram usage (yes or no)

• Dependent Variable: Design Quality

• Treatments: E = use UML / C = don’t use UML

E

Treatments

(1 Factor / 2 Levels)

Independent

Variable

Dependent

VariableC

NB: Design can be

within-subject or

between-subject

Controlled Experiment vs. Quasi-Experiment

Randomization is a

prerequisite for a

controlled experiment!

Experiment – Characterisation (cont'd)

�Characteristics:

–When to use? --> When in-depth understanding about causal relationships is requested

–Focus on quantitative data --> allows for use of statistics for hypothesis testing

–Due to control over independent variables (in vitro = 'in glass', i.e. laboratory), helps detect causal relationships (--> tests hypotheses --> theory building)

� Issues:

–Expensive, 'narrow' (in-depth)

–'Generalisability'? ('artificial' lab setting --> test subjects, tasks, artifacts)

Experiment – Example

What?

Research Question:

• What is best – Pair Programming or Solo Programming?

Who, Where, and When?

Norway, 2007

295 junior, intermediate and senior

professional Java consultants from 29

companies were paid to participate (one work

day)

99 individuals; 98 pairs

The pairs and individuals performed the same

Java maintenance tasks on either:

• a ”simple” system (centralized control style), or

• a ”complex” system (delegated control style)

They measured:

• duration (elapsed time)

• effort (cost)

• quality (correctness) of their solutions

Source: E. Arisholm, H. Gallis, T. Dybå, and D. Sjøberg, “Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise,” IEEE Transactions on Software Engineering, 2007, 33(2): 65-86.

Why?

Many studies with contradicting

results – mostly conducted with

students (not with professional

developers)

Total Effect of PP

84 %

7 %

-8 %

-40 %

-20 %

0 %

20 %

40 %

60 %

80 %

100 %

120 %

140 %

160 %

Duration Effort Correctness

Dif

fere

nc

e f

ro

m in

div

idu

als

Experiment – Example (cont'd)

Effect of PP for Juniors

5 %

111 %

73 %

-40 %

-20 %

0 %

20 %

40 %

60 %

80 %

100 %

120 %

140 %

160 %


Dif

fere

nc

e f

rom

in

div

idu

als

Moderating Effect of System Complexity for Juniors

4 %

109 %

32 %

6 %

112 %

149 %

-40 %

-20 %

0 %

20 %

40 %

60 %

80 %

100 %

120 %

140 %

160 %


Dif

fere

nc

e f

rom

in

div

idu

als

CC (easy)

DC (complex)

Moderating Effect of System Complexity for Seniors

55 %

-13 %

8 %

115 %

-23 %

-2 %

-40 %

-20 %

0 %

20 %

40 %

60 %

80 %

100 %

120 %

140 %

160 %


Dif

fere

nc

e f

rom

in

div

idu

als

CC (easy)

DC (complex)

Effect of PP for Seniors

-9 %

83 %

-8 %

-40 %

-20 %

0 %

20 %

40 %

60 %

80 %

100 %

120 %

140 %

160 %


Dif

fere

nc

e f

rom

in

div

idu

als

Experiment – Example (cont'd)

So, when should we use PP?

The question of whether PP is best, or not, is meaningless!

One should ask: In which situation is PP best to achieve a defined goal?

Importance of Context:Helps construct/refine theory about when and how to do 'Pair Programming'

Selecting the Research Method (to be

submitted)

• The choice of method depends among other things on:

– Suitable study subject (e.g., do participants have enough experience?)

– Possibility to control the environment

– The size/scale/cost of the study

– The need for generality in the results

– Availability of information/data and other resources

– What is the purpose of the study? (exploration, prediction,

understanding of cause-effect relations, applicability of results in

industry, ....)

• Difficult to provide general recommendation with respect to choice of

method

Experiment or Case Study or Survey?

• Experiments give the researchers freedom to isolate a defined effect and

to hold other things constant (-> research in-depth)

– Often difficult to avoid that people respond differently than they would have

done in a natural environment

– Also, you might have a too restricted setting and omit important influencing

factors that play a role in a natural environment

• Case-studies have the advantage that you observe people doing what

they are actually doing in their natural environment (-> research in-the-

typical)

– It is limited what you can control without interfering with natural activity

• Surveys provide researchers with information about what many people

(think they) are doing (-> research in-breadth)

– No control whatsoever (issues relate to data validity and representativeness)

Validity & Reliability of Empirical Studies

• Construct Validity

– Concepts being studied are operationalized and measured correctly (do the measures used actually represent the concepts you want to measure?)

• Internal Validity

– Establish a causal relationship and sort out spurious relationships (exclude confounding variables / by: random sampling)

• Conclusion Validity

– Do proper statistical inference

• External Validity

– Establish the domain to which a

study’s findings can be

generalized (precisely describe

the population and

experimental conditions)

• Reliability

– The study can be repeated (i.e.,

by other researchers) and

yields the same results

– The measurement instrument is

reliable (interrater agreement)

Example (page 12):

http://ieeexplore.ieee.org/document/6776520/

Empirical Research Guidelines

Empirical Research Guidelines

1. Experimental Context

2. Experimental Design

3. Data Collection

4. Analysis

5. Presentation of Results

6. Interpretation of Results

To be submitted

Example:

https://link.springer.com/chapter/10.1007%2F978-

3-319-23063-4_21C

1. Experimental Context

• research questions, hypotesis and variables (see Section 4

in the example)

Example:


3-319-23063-4_21C

2. Experimental Design

• Subjects/objects, datasets, procedure and metrics

(see Section 4.1, 4.2 and 4.3 in the example)

Example:


3-319-23063-4_21C

3. Data Collection

• Ensure that the data collection process is well

defined

• Monitor the data collection and watch for deviations

from the experiment design

4. Analysis

• Ensure that the collected data from the experiment

is analyzed correctly

• Monitor the data analysis and watch for deviations

from the experiment design

5. Presentation of Results

• Ensure that the reader of the results can

understand the objective, the process and

the results of experiment

6. Interpretation of Results

• Ensure that the conclusions are derived

merely from the results of the experiment

Presentation

• Present the following:

– The problem statement

– Background

– Review of the state of the art

– Identified gaps in the state of the art

– The expected contribution of the thesis given the identified gaps (objectives)

– Experimental design and research design

• Each student has a time-slot of 15 minutes: 12 minutes for the talk + 3 minutes

for Q&A

• After each session I will organize a short feedback session to give suggestions

to the presenters

60

How not to do presentation

• Don McMillan: Life After Death by PowerPoint

– http://www.youtube.com/watch?v=WGiePuNFXwY

– http://www.youtube.com/watch?v=zDvm1PVtgWo

– http://www.youtube.com/watch?v=lpvgfmEU2Ck

Seminar on Enterprise Software - ut

Documents

Transcript of Seminar on Enterprise Software - ut