Reviewers comments on our paper submitted to 35th ASEE/IEEE Frontiers In Education Conference

36
Reviewers comments on our paper submitted to 35th ASEE/IEEE Frontiers In Education Conference K. Schützler, K. Bothe

description

Reviewers comments on our paper submitted to 35th ASEE/IEEE Frontiers In Education Conference. K. Schützler, K. Bothe. Introduction. In 2003 first presentations of current JCSE project state: Information Technology Interfaces, Cavtat, Croatia Eurocon, Ljubljana, Slovenia - PowerPoint PPT Presentation

Transcript of Reviewers comments on our paper submitted to 35th ASEE/IEEE Frontiers In Education Conference

Page 1: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Reviewers commentson our paper submitted to 35th ASEE/IEEE

Frontiers In Education Conference

K. Schützler, K. Bothe

Page 2: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 2

Introduction

• In 2003 first presentations of current JCSE project state:– Information Technology Interfaces, Cavtat, Croatia– Eurocon, Ljubljana, Slovenia– 1st Balkan Conference in Informatics, Thessaloniki,

Greece

• 2005: submitted a new paper with current state to 35th IEEE/ASEE Frontiers In Education Conference, Indianapolis

Page 3: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 3

Homepage of FIE 2005

Page 4: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 4

Topics of interest on FIE 2005• Accreditation and assessment • Active learning • Capstone and senior design

experiences • Computer and Web-based

software • Creative design experiences • CSET educational research • Distance learning: Methods,

technologies, and assessment • Diversity: Valuing it, achieving it,

and teaching it • Entrepreneurship programs • Ethics: Creative ways to teach

and assess it • Faculty development • First-year courses and programs • Globalization: Preparing faculty

and students • Innovative degree programs and

curricula

• Innovative uses of technology in the classroom

• K-12 initiatives and partnerships • Laboratory experiences: On-site

and at a distance • Learning models • Lifelong learning • Nontraditional students • Partnerships (industry,

government, university, international)

• Service learning • Software engineering • Student retention and persistence • Teaming • Undergraduate research

experiences • Women in CSET education • Other (You may submit abstracts

and proposals on other topics that address issues at the frontiers in CSET education.)

Page 5: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 5

K. Bothe, K. Schützler, Z. Budimac, K. Zdravkova:Collaborative Development of a Joint Web-Based Software

Engineering Course Across Countries

K. Bothe, K. Schützler, Z. Budimac, K. Zdravkova:Collaborative Development of a Joint Web-Based Software

Engineering Course Across Countries

Page 6: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 6

Aims of this presentation

• Document the serious and detailed review process of FIE 2005

• Make reviewers comments available for all project participants

• Draw some conclusions

Page 7: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 7

January 10, 2005: Abstract submission

February 20, 2005: Acceptance/Rejection of abstract

March 21, 2005: Full paper submission

April 30, 2005: Notification on review results

May 30, 2005: Submission of revised paper with modification notes

Page 8: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 8

Excerpt from first accept notification email

Dear Klaus, The final recommendation for FIE paper 1106 is:ACCEPT with major changes

In the comments below, you will see numerical scores for differentaspects of your presentation. The scale was:1 - poor 2 - needs work 3 - good 4 - very good 5 - excellent

That means: Each comment of the reviewers must be addressed(even if considered inappropriate)

Page 9: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 9

Two reviewers raise a number of questions/issues that require greater revision than the reviewers indicated - thus my recommendation. As you do have an amount of space left in the paper to addresses these, I recommend that you focus on those that would broaden the usefulness of the paper to a wider audience. In an email to me, please indicate how and where the issues raised by the reviewers are addressed. If you feel that the comments are not appropriate, please indicate that to me as well.

Note from Heidi Diefes-Dux(from the organizer‘s site):

Page 10: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 10

Reviewer's recommendation: accept_as_isNew ideas: yesComments on new ideas: It addresses collaborative development of course materials across

several countries.Soundness of work: yesContribution: 3Comments on work contribution: project management issuesPresentation quality: 4, Writing quality: 4, References: yesGender neutral language: yes, Should anything be deleted: noCompleteness of work: noComments on completeness: still somewhat of a work in progress

Reviewer 1

That’s all w

e rece

ived

from th

is revie

wer

That’s all w

e rece

ived

from th

is revie

wer

Page 11: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 11

Reviewer's recommendation: accept_revise

New ideas: yes

Comments on new ideas:

Describes experience with a shared course in software engineering with participants in multiple countries. Approach could be useful for technical education in many locations around the world.

Soundness of work: yes

Reviewer 2

Page 12: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 12

Contribution: 4Comments on work contribution: Describes a successful project in which many

participants not sharing a language use common course materials to support a course in software engineering. Insight and experience from this project could be of broad general interest, particularly in developing countries and countries partnering with them in education.

Presentation quality: 4, Writing quality: 4, References: yesGender neutral language: yesShould anything be deleted: no, Completeness of work: yes

Reviewer 2 (cont.)

Page 13: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 13

This is interesting work and the quality of the writing is fairly good. My questions address relatively minor points, but I encourage the authors to read over the paper again carefully to catch a few remaining awkward sentences, typos, and misused words.

We went through the paper as requested and indeed found a few mistakes and even one incomplete sentence.

Reviewer 2 commentsWriting quality

The reviewer’s comment

The reviewer’s comment

Our answer

Our answer

Page 14: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 14

You describe the course as web-based, but that applies more to the instructors than student participants. (That is, students apparently attend lectures where they see powerpoint slides rather than use machines on a network.) This seems to be inconsistent with the usage of the term in my experience.

Reviewer 2 commentsWeb-based course?

We agree with reviewer's description of our course, but we still call the course web-based. In section "Overview of project management and course materials" we added "Although students are expected to attend regular lectures, the course as a whole could be called web-based. All documents the students need are published at the course website: the slides in PDF-format, case studies, assignments, tool information, literature recommendations, schedule, etc.

skip

Page 15: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 15

The abstract mentions a "plethora of interesting contributions" from the diff-erent participants across various universities and countries. Could you give some examples? I'm curious how the course has evolved over time. You allude to such changes but give essentially no concrete examples or details.

Reviewer 2 commentsContributions from all participants?

We added a paragraph in section that gives an example how the contributions changed the course.

Besides that, we would like to draw the attention to Figure 3 for additional information on the contributions of all project partners.

Translate materials to English

Add new slides and other materials

Include new topics

Write review reports

Develop slide style guides

Write lecture notes

Design and administrate the project web-site

Contribute to the assignments

Use slides in lectures and give usage reports

Contribute to the pool of case studies

Translate materials to native language

Course development

Figure 3: Contributions to the project

Page 16: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 16

You don't say much about some of the links under course materials, specifically the examinations, tools, literature, and duration links. Are copies of all examinations shared with other participants? What about translation overhead? What kind of tools are currently available, or do you plan to make available in the future? Will the literature link simply give citations of relevant papers, or does this describe internal documents for the course? What duration information is included: just the time required in class to cover each topic or to cover each lecture?

Reviewer 2 commentsOne comment – many questions:

examinations, tools, literature, duration

We extended section "Course materials" with the new subsections "V. Literature" and "VI. Tools". It covers now all relevant aspects of the materials on the project website. In section "Overview of project management and course materials" we extended the information concerning the Duration link.

skip

Page 17: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 17

Are the assignments linked in (under course materials) labs or homework style problems? What kinds of assignments have been included in the course so far, and what kind of tools or documentation do they require?

Again, the missing information is addressed in two new paragraphs in subsection "IV. Assignments" of section "Course materials".

"Typical tasks of the assignments include: reviews of requirements specifications, object-oriented modeling practices, performing cost estimations, applying test case determination methods, automated regression testing, and evaluating software metrics."

Reviewer 2 commentsAssignments

Page 18: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 18

It seems challenging to let all participants update course mate-rials but to store only a single current file. What if you don't like the changes someone else makes?

Reviewer 2 commentsConcurrent updates?

To address this question we added an extra paragraph at the end of subsection "IV. Update management" in section "Project management to organize collaborative work":

"Of course, it is necessary to assess the results of the update process. Modified topics are reviewed by their original author. If there are any issues raised, they will be discussed and solved in a cooperative way between author and modifier."

skip

Page 19: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 19

Do you have any way of measuring the effectiveness of the class or of this approach to managing a class? To what extent has it reduced the overhead for new participants in creating a new class and getting it up and running?

Reviewer 2 commentsCollaboration success measured?

The "Conclusions and future work" section has been extended: "Our original goals to disseminate experience and to reduce efforts in creating a new class on software engineering were reached successfully. Participants at several universities could introduce an approved and mature course with all of its parts (lectures, assignments, case studies, etc.) Both, the many years of experience and the produced materials could be reused. The partner universities in southeast Europe experienced significant reduces (compared to earlier attempts of building new courses from scratch) of the managerial efforts for introducing the course, a benefit that can hardly be overestimated."

skip

Page 20: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 20

Reviewer's recommendation: accept_revise

New ideas: no

Comments on new ideas:

Multi-national collaboration and material sharing.

Soundness of work: yes

Contribution: 2

Comments on work contribution:

Giving confidence to others to follow this pattern of working.

Reviewer 3

Page 21: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 21

Presentation quality: 5

Writing quality: 4

References: yes

Gender neutral language: yes

Should anything be deleted: no

Completeness of work: no

Reviewer 3 (cont.)

Page 22: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 22

Extra commentaries by the participants and a retrospective an-alysis of the effectiveness of the transfer process would have been useful.

Reviewer 3 comments Missing information

As mentioned above, we discuss the effectiveness of the tran-sfer in our new version of the paper. We do not think that com-mentaries would fit in the structure of the paper. In our opinion, the paper as a whole presents the common experiences of all project partners.

Page 23: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 23

The paper was well prepared and presented, expressed in good English and easy to read, with sufficient reference to earlier activities and relevant publications. But how was the direction and management of this project influenced by the earlier work?

Reviewer 3 commentsInfluence of related work

To clarify, why our project could not be a simple repetition of the referenced projects, we added a paragraph:

"Compared to similar approaches [2, 3, 5, 6] for collaborative preparation and usage of joint course materials, our project contains a couple of unique features: multi-lingual project partners, a strict style for lecture notes, and a strong strategy for project management with review reports, update manage-ment, style guides, etc."

Page 24: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 24

I was disappointed in the lack of critical appraisal, or pedagogic discussion concerning the effectiveness of the transfer process. I do not think, if I were to embark on a similar multi-national col-laboration, that I would rush to reread the paper for advice and guidance. There is little here which challenges common sense, and no experiential commentary.

Reviewer 3 commentsAddressed the right issues?

As mentioned in section "Conclusion and future work", more and more partners joined us over the years and will join us in the future. From our experience the questions raised by new partners are indeed the ones addressed in our paper. Thus, we do not find this comment appropriate.

skip

Page 25: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 25

The topic and information about how to run similar projects could be of genuine value for many teachers in HE across Europe, and may interest US academics too. What key information do the authors hope to relate to readers? What are the principal benefits to be gained from reading this paper?

Reviewer 3 commentsKey information of the paper

We added a paragraph in section "Conclusions and future work" to explicitly state that our project could serve as an guiding example to similar settings:"Besides that, our project shows that the process of transfer-ring an existing course to other environments (even with dif-ferent languages) can be performed successfully, as long as the issues we presented here are addressed appropriately."

Page 26: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 26

Reviewer 3 commentsStudents cooperation

Are students at the partner institutions encouraged to communicate or even cooperate on case studies and assignments?

"Up to now, students do not communicate or cooperate across universities."

Page 27: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 27

Was any appraisal undertaken to gauge the differences between the effectiveness of the course material when used in the original insti-tution, and that achieved in the acceptor universities? Did student feedback get compared? Were graded assignments and exam ques-tions shared? How was success/failure judged?

Reviewer 3 commentsEffectiveness of course at partner sites

We extended our paper with another paragraph:"Up to now, students do not communicate or cooperate across univer-sities. However, there is already some kind of indirect cross-university influence from the students. Their solutions to the assignments are compared throughout the whole project and they are evaluating their respective course using a project-wide feedback form. The compari-son of these feedback forms is subject to future work."

skip

Page 28: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 28

How was the course material translated from German to English? Who paid for this time consuming work? How were the results checked? How long did it take? Presumably it was an irksome task because of the later reference to automating the work.

Reviewer 3 commentsTranslation to English

We changed the first paragraph of section "Project management to organize collaborative work" to cover more details on the managerial background of the project:"Because our project received substantial funding from DAAD, yearly workshops and coordination meetings were possible. Partners from SEE were enabled to stay at the German university several times in order to work on project tasks like translating existing and preparing new materials."

Page 29: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 29

As the courses were ACM compliant, would it have been possible to use an alternative course, already available in the English language?

Reviewer 3 commentsMobility issues and material reuse

We do not think that using other existing courses would have been appropriate, because the partners explicitly expressed their interest in the particular course available at Humboldt University. We changed the respective sentence in the first section to reflect this situation more.

How much travel, face-to-face meetings for academics, or guest-lecturing was possible during this project?

The change of the first paragraph of section "Project management to organize collaborative work" does apply to this question, too.

skip

skip

Page 30: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 30

Reviewer 3 commentsMisunderstandings

"the aims changed over time" - how, why and in what manner?

We think, that this issue has been addressed already in the directly fol-lowing sentences. We made it a bit clearer now, by putting a colon instead of a dot. "…over time, the targets … of the project have changed considerably: With the contributions from participating partners, new materials have been developed as a response to the feedback and suggestions of professors and their students. Due to the additional information the project had to face and overcome organizational challenges to coordinate the work."

"special course material problems" - what were these?

We assume that this results from a misunderstanding of our bad English. We changed it to "special problems concerning course materials" and again replaced a dot by a colon.

skip

Page 31: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 31

The paper appears to have been produced by the senior part-ner, with little evidence of reported experience from the junior, accepting institutions. Views more clearly from all the instituti-ons would have been welcome.

Reviewer 3 commentsPaper from one partner only?

We do not consider this comment as being appropriate. We are not able to imagine what kind of evidence the reviewer is requesting. The paper has been produced together by the senior partner and two of the junior partners and contains experience of both sides.

Page 32: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 32

Well chosen case studies are often the most useful material, "borrowed" by colleagues from different universities. Little was said on this subject. Were the two original scenarios specially drafted, or inherited from the original, parent course? Mention is made of the tutors' commercial experience being brought to bear locally for the benefit of the students. How can this resource be distributed successfully to other sites?

Reviewer 3 commentsCase studies

In subsection "III. Case studies" we added a paragraph:"Presently, we are including two case studies reused from the original course, one from an administrative application and the second from a technical application. The project participants contributed additional well suited case studies. These case studies are not included in the course so far, because teaching materials for them still have to be prepared."

Page 33: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 33

For the reviewer, the project URL, reference [7], did not work when accessed. An email was dispatched to the principal author.

We fixed that problem. Instead of ".../index.html" it has to be ".../index.htm".

Reviewer 3 commentsTechnical problems and recommended literature

What text books were recommended for use by students?

As mentioned above, we added subsection "V. Literature" to section "Course materials" to give more information on the literature recommendations in our course.

skip

Page 34: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 34

Was any project source file control system (CVS?) employed to maintain project consistency?

This question is answered in the last paragraph of subsec-tion "IV. Update management" in section "Project manage-ment to organize collaborative work“:"Since most materials are in binary formats and are not changed concurrently there was no urge to apply a complete configuration management. However, previous versions are archived to be used as fallbacks in case of problems."

Reviewer 3 commentsConfiguration management

Page 35: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 35

Klaus Bothe -

Thank you for making those changes. I am very satisfied with your responses to the reviewers and I am looking forward to seeing you and your presentation in Indy!

Heidi************************************************************* Heidi A. Diefes-Dux, Assistant Professor Purdue University Department of Engineering Education *************************************************************

Final note from Heidi Diefes-Dux(from the organizer‘s site):

Page 36: Reviewers comments on our paper submitted to 35th ASEE/IEEE  Frontiers In Education Conference

Kay Schützler, Baile Herculane, 28.08. - 03.09.05 36

Conclusions

• Our project is interesting and useful for third parties

• FIE reviewers are interested in quantifiable results

• Commonly used real-life case studies are challenging– Still a somewhat open question in our project

• Cooperation between students across countries is to be considered seriously

• Feedback from partner universities is highly recommended – especially critiques