WCCE 2009 – World Conference on Computers in EducationBento Golçalves-RS, Brazil, July 27-31, 2009, www.wcce2009.org

Growing a Peer Review Culture among Graduate Students

Vinícius M. Kern1,2, Osmar Possamai1, Paulo M. Selig1, Roberto C.S. Pacheco1, Gilberto Corrêa de Souza1, Sandro Rautenberg1,3, and Renata Tavares da Silva Lemos1

1 Graduate Program in Knowledge Engineering and Management, Federal University of Santa Catarina, Brasil

kern@egc.ufsc.br (contact author), www.egc.ufsc.br2 Instituto Stela, Brasil, www.stela.org.br3 Universidade Estadual do Centro-Oeste (Unicentro), Brasil, www.unicentro.br

Outline of the presentation

Introduction A Systemic View of Graduate Learning Approach to Growing a Peer Review Culture

Results and Discussion Concluding Remarks

2

Introduction

• Common processes for pursuing graduate education excellence:

• candidate selection• teaching and coursework• research (conducted by students under supervision)• thesis defense

• Our complementary approach: the growing of a peer review culture among graduate students

• Grad. Prog. Knowledge Eng. & Mgmt (KEM or EGC in Portuguese), Federal U. of Santa Catarina

• Started in 07/2004, multidisciplinary (Eng., IT, Mgmt, Media)• Research object: “knowledge as a production factor”.• High candidate rates: 322 to 423 candidates for about 60 no-

scholarship annual openings, since 2004.

3

Introduction (II)• EGC/UFSC (cont.)

• Faculty: ~40 from 10 departments, different worldviews/cultures (multidisciplinary); interdisciplinary character of the research object

• Risk of being “multidisciplinary, hence potentially dispersive, rather than interdisciplinary, hence cohesive” (Bunge, Emergence & Convergence)

• Interdisciplinarity requires excellent communication

• Need for enhanced communication >>motivation for our first annual workshop in 2004

• Students presented proposals; template provided

• 2005 on: Research Seminars, mandatory, no credits, then the workshop• Proposals with peer review: 49 (2005), 67 (2006), 54 (2007), 62 (2008)

• Next: systemic analysis and description of our peer review approach to growing a peer review culture; future work

4

Our Systemic View of Graduate Learning - Concepts

• DISCLAIMER: No holism. “When rigorous contemporary social scientists hear the word 'system,' they are likely to draw their intellectual guns”

• “Systems have systemic (emergent) features that their components lack”

• Everything is a system or an actual or potential component of a system / There are no permanent strays or isolates

• Any concrete system σ can be modeled as μ(σ) = <C(σ), E(σ), S(σ), M(σ)>

• Composition: collection of all the parts of σ• Environment: collection of (other) items that act on or are acted upon by some or

all components of σ• Everything has an environment except for the universe as a whole.

• Structure: collection of relations, in particular bonds, among components of σ (endostructure) or among these and items of the environment (exostructure)

• Mechanism: collection of processes in σ that make it behave the way it does – that generate qualitative novelty, emergence.

• Abstract systems have no mechanism

SOURCES: Bunge’s E&C (2003) and ‘Mechanism and explanation”, Phil Soc Sci (1997)

5

Our Systemic View of Graduate Learning – Systems…

System

CComposition

EEnvironment

SStructure

MMechanism

Linguistic community

People who speak the same language.

Culture(s) where the language is used.

Collection of linguistic communication relations.

Production, transmission, and reception of symbols.

Firm Personnel and management.

Market and government.

Work relations among the firm's members;member-environ. item.

Activities that end up in the firm's products.

Organism Physical and chemical micro- and mesosystems, in partic. water, carbohydrates, lipids, proteins, and nucleic acids.

Medium rich in nutrients and energy fluxes, with variables (pressure, temperature etc.) within narrow intervals.

All the bonds (direct/indirect, physical/chemical, covalent/non-cov.)

that keep together plus all the ties - phys., chem., biol. - with environmental items.

Metabolism and accompanying processes that allow for maintenance and self-repair, expression and repression of genes etc.

Size B

ond strength

– Structure (bonds of…): [Endostr.] collaboration, communication, co-authorship/cooperation in projects, advising, teaching, peer pressure, [Exostr.] submission to rules, funding, refereeing, cultural influence, affiliation to UFSC, reputation-networking-partnership-affiliation with other organizations

– Mechanism: study+research, advising, project/publication cooperation, communication* (scientific, objective criticism and argumentation)

6

Our Systemic View of Graduate Learning – Finally

Funding agencies

UFSC (host institution), staff

Norms,regula-tions,laws

faculty

studentsOrganizations (firms, research institutes, univ., NGOs, govern-mental, inter-national)

Culture, Brazilian graduate education community

7

Approach to Growing a Peer Review Culture among Students

• Peer review in education by first author• 1994 as student (Virginia Tech), then since 1997 as professor• More than 1,000 ‘victims’ http://kern.ispeople.org/par_en.html

• Reasons to undertake: cooperation, written expression, critical thinking, professional responsibility (working on bonds and skills…)

• Initiatives by several other scholars around the world (see refs)

• Our research seminars since 2005 (mandatory, no grade)• About 8 4-h encounters: proposal preparation, live presentations and

professors’ feedback, advice for refereeing (including sample referee forms)

• Usually 2+ professors in class• Feedback on form and on internal and relational coherence of

proposal sections; no criticism on merit (because of methodology)

• Single-round, double-blind peer review (OJS)

• Each student gives and receives 3 feedbacks (ideally)

• Full feedback to all students and advisors (spreadsheet summary)

• December: annual workshop (all freshmen present proposals)

8

Approach to Growing a Peer Review Culture among Students (II)

Objectives

• For the students…• Experience, acquaintedness: To know the essential parts of a thesis

proposal, then re-elaborate the proposal presented for entrance in the graduate program and participate in a peer review round.

• Maturing: To start developing competence to give and receive professional, objective critique of scientific work.

• For the graduate program• Culture: To create and disseminate a culture of objective,

interdisciplinary scientific criticism.• Spur thruput: To serve as catalyst of the advising process.• Interdisciplinarity: To stimulate interdisciplinary scientific

interchange.

9

Approach to Growing a Peer Review Culture among Students (III)

• Threats to the process• Without strong guidance, it becomes an exchange of opinions (at best)

• Upper management support desperately needed (ok)

• Because we are so multidisciplinary (not inter-), Method is out (I know, that’s awful…)

• What we did to face the threats• Adopted one successfully-in-use set of guidelines (2nd author, positivist

approach, engineering method)

• GP dean is always one of the professors• Required advisor involvement• Started a collaboration to tame methodology

• 2007: 2 collaborators gave lectures• 2008: 5 professors in 2 diff Method courses (elective quanti, quali)

• 2009: Intro to research methodology: 7 professors plus another one who attended all classes (then 3 complementary courses)

• Worldviews/main classes of paradigms; Methodological tour

10

Approach to Growing a Peer Review Culture among Students (IV)

• Template for proposals – usual sections• Abstract• Motivation• Problem statement and/or question• Objectives [might be split into general and specifics]• Relevance• Scope• Main references• General appraisal

• Software choices• Conference management choices: either costly/unknown or too

limited• Open Journal System: free, widespread, allows attachment• Attachment: Referee form spreadsheet (Excel, controlled content, easy to

import to MySQL)

• Referee form (next)

11

A

b

App

roac

h to

Gro

win

g a

Pee

r R

evie

w

Cul

ture

am

ong

Stu

dent

s (V

– r

efer

ee f

orm

)

Área: Faço... doutoradoSuponho pertencer...

No.: (Digite o número do documento, para que o organizador possa verificar)

Título: (Preencha com as primeiras palavras do título - isto é uma dupla verificação da identidade da proposta revisada)

Área: Nível:

Comen-tários?

Comen-tários?

Apresentação do problema de pesquisa (Contextualiza com referências relevantes?)

Objetivos Específicos (São mensuráveis? Contribuem para atingir o objetivo geral?)

Comen-tários?

Comen-tários?

Objetivo geral (É coerente com o problema definido?)

Pergunta de pesquisa (Define o problema de forma clara e precisa, a partir do gap de conhecimento?)

Apreciação global (Atribua um escore para a proposta como um todo)

Escopo (Delimita o que abrange e o que não abrange o estudo teórico?)

Referências (Usa as fontes adequadas? Lista apenas as obras citadas no texto, conforme ABNT 6023?)

Resumo (Resume = apresenta sucintamente: problema, abordagem, resultados esperados e o significado destes?)

SEÇÃO I - VISÃO GERALResuma a proposta em uma sentença ou parágrafo.

SEÇÕES II E III - AVALIAÇÃO OBJETIVA E COMENTÁRIOS

INFORMAÇÕES GENÉRICAS SOBRE O(A) REVISOR(A)

... ao terço de alunos mais conhecedores do tópico da proposta.

Engenharia do Conhecimento

INFORMAÇÕES GENÉRICAS SOBRE A PROPOSTA DE TESE OU DISSERTAÇÃO REVISADA

Comen-tários?

Comen-tários?

Comen-tários?

Comen-tários?

Referee research area (KE/KMa/KMe), level (Dr/MSc), and self-declaration of expertise

Proposal # and title, author area and level (all proposals from same level of referee’s, 2 out of 3 from same area)

1-paragraph summary of proposal

Grades and

comments for each proposal section

Grades and comments for proposal as a whole

12

Results and Discussion

• Mostly anedoctal (and some exploratory research)

• Proposals processed: 2005-49, 2006-67, 2007-54, 2008-62

• Additional students allowed for refereeing only

• Numbers for 2008 (4 extra referees 198 allocations, 192 delivered)

• Commitment and depth of feedback vary• While it may be related to “mandatory, no credits, no grade”, strategy for

improvement has been: raising awareness (related to communication bonds)

13

Results and Discussion (II)

• Publication of full and aggregate results started in 2008 (Why: very limited resources; Before 2008: “here are your forms”)

• Aggregates: by [area, level, gender—not in the paper], percents of self-declaration of expertise

• For each proposal• For each referee report (typically 3 for proposal)

• Research area of the referee• Referee’s self-declaration of expertise• Grades (0-10) and comments on the topics reviewed

• Report allows students to…• See all referee reports about own proposal• Analyze, compare reports, reflect about quality of own work.• See what colleagues said about the same proposals refereed• Reflect about own skills as referee and about general quality of

communication* bonds

14

Results and Discussion (III-Aggregates 2008)

15

Results and Discussion (IV-Aggregates)

• Grades• 0-3 (reject), 4-5 (weak reject), 6-7 (weak accept), 8-10 (accept)

• Given: From 6.67 (Scope, average) to 7.37 (General objective, avg)

• Self-declaration of expertise• Even though students meet in 3+ mandatory courses plus the

Seminars, they have a poor appraisal of their comparative expertise in the proposals’ topics…

• … or maybe this is a cultural issue (you can’t claim to be so good)

16

Results and Discussion (V)

• Opportunities for Improvement and R&D Issues• Instrumental issues

• Better processing of peer review bureaucracy• (Semi-)automation of knowledge-intensive tasks, e.g.

• referee allocation• referee rating• process reliability and validity• evaluation with nonlinear dynamics (Losada: connectivity)

• Methodological issues• Articulation of Seminars with new mandatory course on

Methodology (toward interdisciplinarity)

• Now we have some preparation to discuss paradigm and method choice

• Investigate mechanisms (Bunge) that create emergent properties in our graduate learning system

• Multilevel analysis (macro-micro systems)

17

Concluding Remarks

• Experience report (mostly anedoctal; empirical research to come)

• Computer use still very marginal

• Students experience the main scientific method for quality control and have an opportunity to sharpen their knowledge and strengthen their (scientific, rigorous, objective) communication bonds with their peers and professors, adviser included

• Part of our quest for interdisciplinarity through strengthening communication bonds

• Aim at contributing, as well, to establish peer review as a replicable, scalable educational approach (Kern et al., 2007)

18

ReferencesAngelov, C., Melnik, R.V.N., Buur, J.: The synergistic integration of mathematics, software engineering, and user-centered design: exploring new

trends in education. Fut. Gen. Comp. Sys. 19(8), 1299-1307 (2003)Araújo, L.H.L.: Uma aplicação da dinâmica não-linear para a avaliação de desempenho de comunidades virtuais de aprendizagem [An

application of nonlinear dynamics to performance evaluation of virtual learning communities]. Masters dissertation, Universidade Católica de Brasília, Brasília-DF, Brasil (2004)

Bunge, M.: Emergence and convergence: Qualitative novelty and the unity of knowledge. Uni. Toronto Press, Toronto (2003)Bunge, M.: Mechanism and explanation. Phil. Soc. Sci. 27(4), 410-465 (1997)Bunge, M.: Systemism: the alternative to individualism and holism. J. Socio-Econ. 29(2), 147-157 (2000)Cunningham, S.J.: Using a computer conferencing system to support writing and research skill development. ACM SIGCSE Bul. 26(4), 5-8

(1994)Davies, R., Berrow, T.: An evaluation of the use of computer supported peer review for developing higher-level skills, Comput. Educ. 30(1/2),

111-115 (1998)Denning, T., Kelly, M., Lindquist, D., Malani, R., Griswold, W.G., and Simon, B.: Lightweight preliminary peer review: does in-class peer review

make sense? ACM SIGCSE Bul. 39(1), 266-270 (2007)Gehringer, E.F.: Electronic peer review and peer grading in Computer-Science courses. ACM SIGCSE Bul. 33(1), 139-143 (2001)Hafen, M.: Developing writing skills in computer science students. ACM SIGCSE Bul. 26(1), 268-270 (1994)Hartman, J.: Writing to learn and communicate in a data structures course. In: 20th SIGCSE Technical Symposium on Computer Science

Education, pp. 32-36 (1989)Kern, V.M., Pacheco, R.C.S., Saraiva, L.M., Pernigotti, J.M.: Peer review in Computer Science education: Requirements for continuous, large

scale application. In: F.M. Mendes Neto and F.V. Brasileiro (Eds.), Advances in computer supported learning, pp. 46-65, Idea Group Publishing, Hershey (2007)

Kern, V.M., Saraiva, L.M., Pacheco, R.C.S.: Peer review in education: promoting collaboration, written expression, critical thinking, and professional responsibility. Educ. Inf. Technol. 8(1), 37-46 (2003)

Liu, E.Z., Lin, S.S.J., Chiu, C., Yuan, S.: Web-based peer review: the learner as both adapter and reviewer. IEEE Trans. Educ. 44(3), 246-251 (2001)

Losada, M., Heaphy, E.: The role of positivity and connectivity in the performance of business teams: A nonlinear dynamics model. Am. Behav. Scient. 47(6), 740-765 (2004)

Losada, M.: The complex dynamics of high performance teams. Math. Comput. Model. 30(9-10), 179-192 (1999)Moreira, D.A., Silva, E.Q.: A method to increase student interaction using student groups and peer review over the internet. Educ. Inf. Technol.

8(1), 47-54 (2003)Riggs, T., Wilensky, R.: An algorithm for automated rating of reviewers. In: 1st ACM/IEEE-CS joint conference on digital libraries, pp. 381—

387. Roanoke, (2001)Schreiber, G. et al.: Knowledge engineering and management: the CommonKADS methodology. MIT Press, Cambridge MA (2000)Sitthiworachart, J., Joy, M.: Web-based peer assessment in learning computer programming, In: 3rd IEEE International Conference on

Advanced Learning Technologies (ICALT’03), (2003)Smith, A.J.: The task of the referee. IEEE Computer, 23(4), 46-51 (1990)

19

Thank you!AcknowledgmentsAuthors GCS, SR, RTSL (volunteer

students)

Instituto Stela (network and systems administration services)

Questions?

Contact author:

Vinícius Medina Kern, kern.ispeople.org (see menu ‘Presentations…’ for these slides)

Researcher at Instituto Stela, www.stela.org.br

Professor at EGC/UFSC, www.egc.ufsc.br

(Both in Florianópolis-SC, Santa Catarina island)