How can research and artistic How can research and artistic performance of university staff performance of university staff

members be measured ICT-based?members be measured ICT-based?

Harald LothallerHarald Lothaller

University of Music and Performing Arts Graz University of Music and Performing Arts Graz (Austria)(Austria)

CRIS 2010 - Current Research Information Systems, Aalborg University, Denmark, June 2nd - 5th 2010

Background Design process Technical implementation Accompanying measures Conclusions

OvervieOvervieww

Development following DIRKS

University of Music and University of Music and Performing Arts GrazPerforming Arts Graz

One out of 22 public universities in Austria One out of 6 public universities of arts in

Austria One out of 4 public universities in the city of

Graz

BackgroundBackground

University of Music and University of Music and Performing Arts GrazPerforming Arts Graz

About 420 staff members in teaching, arts, and research

About 150 staff members in administration and services

Nearly 2300 students from 59 countries Fields of studies: Church music, Composition

and music theory, Conducting, Electrical engineering/Sound engineering, Instrumental studies (classical music), Jazz, Music education, Musicology, Performing art, Stage design, and Voice as well as Doctoral study programs and Non-degree programs

BackgroundBackground

Austrian Universities act 2002Austrian Universities act 2002 §13: …the Intellectual Capital Report (ICR) “shall,

as a minimum, present in itemised form (1) the sphere of action, social goals and self-imposed objectives and strategies; (2) its intellectual capital, broken down into human, structural and relationship capital; (3) the performance processes set out in the performance agreement, including their outputs and impacts”.

§ 14: “The universities shall develop their own quality management systems in order to assure quality and the attainment of their performance objectives.”

BackgroundBackground

Communication of universitiesCommunication of universities Middaugh, 2001, p.1:

“Colleges and universities have done a horrible job of communicating to both internal and external groups what faculty do and how well they do it.”

Middaugh, 2001, p.54: …information about the outcomes of research or public service activities “are integral to a full understanding of what faculty do”.

BackgroundBackground

Staff members’ viewStaff members’ view The main activities are teaching, research, and

the advancement and appreciation of arts Additional administrative duties are out-of-favor

and often disregarded The work, its quality, and its outcomes can

hardly be measured and this holds for quantitative approaches in particular (cf. Middaugh, 2001, p.80ff.)

Institutional rules sometimes are takes as conflicting with “[…] the freedom of scientific and artistic activities […]” (Austrian Universities act 2002, §2)

BackgroundBackground

Two main questionsTwo main questions

How can the research and artistic performance of university staff members be measured?

How can the staff be made to participate in an online tool designed to measure their performance?

BackgroundBackground

Our approachOur approach

“a structured and rigorous approach designed to ensure that records and information management is firmly based in the business needs of the organization” (National Archives of Australia, 2001a, p.5) a)Preliminary investigationb)Analysis of business activityc)Identification of recordkeeping requirementsd)Assessment of existing systemse)Identification of strategies for recordkeepingf)Design of a recordkeeping systemg)Implementation of a recordkeeping systemh)Post-implementation review

DIRKS DIRKS methodologymethodology

Preliminary investigationPreliminary investigation

…it was inevitable to get an overview concerning the role of the “organization, its structure, the business, regulatory and sociopolitical environments in which it operates, and major factors affecting its recordkeeping practices” (National Archives of Australia, 2001, A-4).

DIRKS – step ADIRKS – step A

Analysis of business activitiesAnalysis of business activities

Austrian University act 2002 explicitly points to eleven tasks universities have to fulfill within their sphere of action:

“1. advancement of sciences (research and teaching), and the advancement, appreciation and teaching of the arts

[…]11. provision of public information on the

performance of the tasks of the universities” (§3)

DIRKS – step BDIRKS – step B

Identification of recordkeeping Identification of recordkeeping requirementsrequirements

Clear focus on the measurement of research and artistic performance, but not teaching or financial aspects…

Two main aims: 1) The new tool should represent the wide-

spread range of activities of the university members.

2) The new tool should fulfill all requirements from the Austrian Universities act 2002 with regard to quality management and the ICR in particular

DIRKS – step CDIRKS – step C

Assessment of existing systemsAssessment of existing systems Paper & pencil approach for some years

Did not fully comply with the new requirements from the ICR

Was often criticized by staff members as to narrow

Contained a categorization system that gave us a starting point for designing the new tool

DIRKS – step DDIRKS – step D

Identification of strategies for Identification of strategies for recordkeepingrecordkeeping

“The purpose of Step E is to determine the most appropriate policies, practices, standards, tools and other tactics that your organisation should adopt to remedy weaknesses identified in Step D and ensure that they meet recordkeeping requirements identified in Step C” (National Archives of Australia, 2001, E-4).

DIRKS – step EDIRKS – step E

Design of a recordkeeping systemDesign of a recordkeeping system

DIRKS – step FDIRKS – step F

Step 1: revised system

Step 2: external

discussionsStep 3: internal

discussions

Project owner: ICR and management requirements in mind

Step 4: list of

attributes

Four steps:

Design of a recordkeeping systemDesign of a recordkeeping system

DIRKS – step FDIRKS – step F

At present, we use eight groups of categories:Artistic activities (11 categories)Research-related activities (6 categories)Pedagogic activities (6 categories)Publications (17 categories)Projects (3 categories)Functions and activities (16 categories)Awards (5 categories)Administration and service (8 categories)

Implementation of a Implementation of a recordkeeping systemrecordkeeping system

DIRKS – step GDIRKS – step G

Campus management system: CAMPUSonline From campus and elsewhere

(http://online.kug.ac.at) General information and internal area (with log-

in) New feature „Leistungen“ in 2007 Administration area for flexible configuration

• Transfer of categorizational system into CAMPUSonline

• Configuration on 4 levels

Implementation of a Implementation of a recordkeeping systemrecordkeeping system

DIRKS – step GDIRKS – step G

Level 2

Level 3

Level 4

Level 2

Implementation of a Implementation of a recordkeeping systemrecordkeeping system

DIRKS – step GDIRKS – step G

Transfer into CAMPUSonline was done all at once Subsequent testing phase:

• Test takers from different scientific and artistic fields

• Test takers with wide range of computer-handling skills

Adaptations and reductions were necessary• with regard to complex data entering possibilities• with regard to visibility of icons and order of

details Testing several alternatives gradually Theoretical importance versus practical situation!

Implementation of a Implementation of a recordkeeping systemrecordkeeping system

DIRKS – step GDIRKS – step G

Stähler, 2002, p.290f.: Innovation does not necessarily succeed • Even if there is a benefit for the customers or

users• Acquirement of knowledge is linked to time

(restricted factor)

Acceptance of innovation is high with• High benefits ( highlighted in

communication)• Low time demands for acquiring knowledge (

support)

Implementation of a Implementation of a recordkeeping systemrecordkeeping system

DIRKS – step GDIRKS – step G

Roll-out to all staff members in January 2008 Mainly three kinds of accompanying support:1) Detailed script of more than 60 pages

• Handed out in trainings and available for download

2) Small-group trainings • Up to 8 persons• From same department• Rather similar computer-handling skills

3) Personal assistance on demand by email or telephone

Implementation of a Implementation of a recordkeeping systemrecordkeeping system

DIRKS – step GDIRKS – step G

Communication is of high importance!• Following the given structures• Users as multipliers in addition• Ok, but could have been better…• Internal report after 6 month (and then every

6 month)

Highlighting two aspects in communication:1) Requirements from University act2) Personal advantages (e.g., online self-

presentation)

Post-implementation reviewPost-implementation review

DIRKS – step HDIRKS – step H

Presentation of persons, departments or the university to the respective community and to the public

Intellectual Capital Report Requirements are met well Better data in quality and quantity Analyses much easier

Internal reports every 6 month Evaluations of single persons or departments

Facilitates self-reports Presentations are more traceable for peer reviewers

etc. Provides quality information in addition to quantity

Post-implementation reviewPost-implementation review

DIRKS – step HDIRKS – step H

End of 2009: > 10.000 data entries Participation rate: 2/3 of academic staff

Enhancements are going on:• Adjustments due to user feedback• Expecting changes due to new governmental

rules for ICR• Quality assurance• Export function• FAQs via video/podcast

Post-implementation reviewPost-implementation review

DIRKS – step HDIRKS – step H

Cooperation with the University for Art and Industrial Design Linz (Austria)• CAMPUSonline too• Adjusting our system to their specific needs

(i.e. their scientific and artistic fields in particular)

• Transferring the system• Adjusting or building respective queries and

analyses• Successfully completed in June 2009

Successful implementationSuccessful implementation

ConclusionsConclusions

Tool provides fundamental data for performance statistics

Following DIRKS

Comply with quality Comply with practical standards requirements

Lessons learntLessons learnt

ConclusionsConclusions

To summarize our experiences, we have learned that (a)high effort for involving a wide range of persons pays off twice afterwards, (b)sometimes research and management interests in data collection have to be pared down in favor of usability in practice, and (c)extensive instructions, personal assistance, and highlighting beneficial features can make even skeptics or unskilled computer users participate in an ICT-based performance recording.

Harald Lothaller

Quality Management & ReportingUniversity of Music and Performing Arts

Graz (Austria)

Email: harald.lothaller@kug.ac.atHomepage: www.kug.ac.at

ContactContact