A Quality Model for Web-Services Evaluation in Cultural

Heritage Management

ICCSA 2011Santander, 20-23 June 2011

Aline Chiabai

2

Background and motivations

Case study

Theoretical framework

Methods

Survey results

Conclusions and further steps

Outline

3

ISAAC “Integrated e-Services for Advanced Access to Heritage in Cultural Tourist Destinations” (FP6)

User relevant integrated platform and e-services in urban cultural destinations to valorize the use of EU heritage taking into account stakeholders’ perspectives

Case study: e-governance model for cultural heritage management in the city of Genoa (Italy)

ISAAC project

4

First phase:

• Construction of a user-friendly geo-referenced Web system (www.isaac-genovaculture.eu) as a tool to facilitate communication and participation among the different stakeholders (for CH management)

Second phase:

• Activation of the participatory process (Blended Focus Groups): bottom-up approach

• Stakeholders: residents, tourists and service providers

• Fine-tuning of the system according to users’ expectations

• SERVQUAL analysis

Integrated approach

e-Participation website prototype

Application: Blended focus groups

Residents TouristsLocal

service-providersExternal

service-providers

SERVQUAL

TEST

Satisfaction analysis

Stakeholders’ involvement

Evaluation of users expectations

Recursive cycle “planning-action-revision”

http://www.isaac-genovaculture.eu/

Information Inform and sensitise people about cultural resources

CommunicationUse new technologies as scaffolding to favour communication and critical reflection between citizens about cultural tourism and cultural resources

ParticipationDesign scenarios for sustainable urban destinations through participatory tools

9

Generally, the management of CH does not account for users’ preferences

Traditional top-down approaches often not effective and even in contrast with real users’ needs

Innovative forms of CH management have to be built on users’ expectations and satisfaction for service optimization and diversification

Rationale of the study

10

• SERVQUAL original version developed by marketing expert (Zeithaml, Parasuraman e Berry, 1991)

• Objective: provide enterprises with a tool to evaluate the service quality, customers’ opinions about products/services supplied

• Service quality: the degree of discrepancy between the expectations or desires of the customers and their perceptions

• More difficult to evaluate service quality (and specifically e-services) than product quality

SERVQUAL ANALYSIS

11

The SERVQUAL model

12

The quality service life cycle

13

Object to evaluate: website comprising a set of web-services

The following “service quality dimensions” have been identified, characterising the web-services:

• design, layout and usability (navigation, visualization)

• content and information updating

• e-participation

• self-construction topics (interaction with web design staff)

Within each group, a number of more specific features (sub-services) have been identified

Survey design

14

Service quality dimensions

Service quality dimension Specific features

1. Design, layout and usability: pleasant graphical aspect and clear layout. User-friendliness navigation and speed loading of the pages.

Layout, easy reading, information easy to find, use of images and photos to support information.

2. Content and information updating: the ability of the web portal to offer information that is comprehendible, reliable, precise and updated.

Frequent update, tourist and historical info, clearness.

3. e-participation: the presence of tools enabling users to communicate and debate.

Thematic discussion, forum/blogs, new themes suggestions, interaction with municipality.

4. Self-construction topics: possibility of interactions with the editorial staff to have personal assistance in navigation or to suggest improvements of the web site.

Moderator, contribution to content, assistance for navigation, suggest improvement, communication to users.

15

- Section 1: Socio-demographic questions (gender, age, education) and internet behaviour (frequency and location of use)

- Section 2: Users’ expectations with respect to a generic and hypothetical website that delivers services similar to the one under valuation.

- Section 3: Users’ perceptions (satisfaction) of the specific website under valuation

Questionnaire strucutre

EXPECTATION ANALYSIS: EXAMPLE

1 An excellent web portal about cultural goods, should be graphically attractive and have a pleasant aspect, even if visualisation is slower.

1 2 3 4 5 6 7

Completely disagree

Neither agree nor disagree

Completely agree

2 An excellent web portal about cultural goods, provides specific areas for thematic discussion reserved to users

groups.

1 2 3 4 5 6 7

• For each of the 4 macro areas, a set of statements have been proposed to the respondents

• Respondents have been asked to assign a score to each specific feature, using a likert scale from 1=completely disagree, 4=neither agree or disagree, 7=completely agree

PERCEPTION ANALYSIS: EXAMPLE

1 www.isaac-genovaculture.eu is graphically attractive and has a pleasant aspect, even if visualisation is a little slow.

1 2 3 4 5 6 7

Completely disagree

Neither agree nor disagree

Completely agree

2 www.isaac-genovaculture.eu provides specific areas for thematic discussion reserved to group of users.

1 2 3 4 5 6 7

18

(1) GAP = P-E P = perception score (perceived quality)

E = expectation score (expected quality)

Scores are assigned based on the response to the questionnaire (seven point likert scale)

The largest negative gaps identify the service features where to prioritize intervention for improvement

Positive gap if expectations have been exceeded, which is not necessarily positive (over-supplying)

Calculation of SERVQUAL score

19

(2) GAPw = (P-E)x(D+100)/100

P = perception score

E = expectation score

D = score assigned to the service dimension (based on a total 100 score)

The weighted SERVQUAL scoreRespondent is asked to allocate a total score of 100 points among the 4 service dimensions, according to the relative importance assigned to them:

(a) Layout/usability, (b) content, (c) e-participation, (d) self-construction topics

20

Timing: February-April 2008

Place: Genoa

Recruitment: respondents randomly selected from a list of users subscribed to the FEEM Culture Factory of Genoa

Sample: 89 compiled questionnaires

Quota sampling

Survey administration

21

Results: users’ preferences for service quality dimensions

22

Results for the broad service dimensions

User expectations and perceptions

4.815.12

5.88 5.87 5.354.92

5.124.70

1

2

3

4

5

6

7

Layout and usability Content E-governance Interaction with staff

Av

era

ge

sc

ore

Expectations Perceptions

1 2 3 4 5 6 7

Layout

Easy to read

Easy to find info

Photos & images

Frequent update

Tourist info

Clear info

Historical info

Thematic discussion

Forum, blogs

New themes suggest

Municipality interaction

Moderator

Contribution to content

Assistance for navigat

Suggest improvement

Communic to users

4,7

5,86,56,5

6,4

4,36,5

6,2

5,2

5,25,3

5,14,9

5,25,6

5,45,2

Average score

Expectations for a website focusing on cultural tourism

LAYOUT & USABILITY

E-PARTICIPATION

CONTENT

INTERACTION WITH STAFF

Analysis of users’ expectations about specific Web features

24

Results at sub-service level

1.00 2.00 3.00 4.00 5.00 6.00 7.00

Layout

Easy to read

Easy to find info

Photos & images

Frequent update

Tourist info

Clear info

Historical info

Thematic discussion

Forum, blogs

New themes suggest

Municipality interaction

Moderator

Contribution to content

Assistance for navigat

Suggest improvement

Communic to users

4.73

5.76

6.52

6.52

6.44

4.31

6.51

6.21

5.16

5.18

5.26

5.10

4.90

5.22

5.57

5.43

5.17

3.92

5.39

5.01

5.37

5.67

3.13

5.13

5.31

5.93

6.13

5.89

3.29

4.36

5.37

4.92

5.22

3.27

Average value of the answers

Pro

po

sit

ion

pro

po

se

d to

th

e in

terv

iew

ed

Comparison between perception and expectation for each proposition

EXPECTATION AVERAGE FOR EACH PROPOSITION PERCEPTION AVERAGE FOR EACH PROPOSITION

Fig. 5. Comparison user expectations and perceptions at sub-service level

25

Results: SERVQUAL simple gap expectations/perceptions

-1.90

-0.20

-0.65

0.15

-0.54

-1.81

0.63

0.17

0.78

-0.90

-1.37

-1.18

-0.76

-1.15

-1.51

-0.37

-0.81

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6

AVERAGE GAP

Layout

Easy to read

Easy to find info

Photos & images

Frequent update

Tourist info

Clear info

Historical info

Thematic discussion

Forum, blogs

New themes suggest

Municipality interaction

Moderator

Contribution to content

Assistance for navigat

Suggest improvement

Communic to users

GAP BETWEEN EXPECTATIONS AND PERCEPTIONS

LAYOUT & USABILITY

CONTENT

E-GOVERNANCE

INTERACTION WITH STAFF

26

Results: SERVQUAL simple and weighted gap

-2.40 -2.10 -1.80 -1.50 -1.20 -0.90 -0.60 -0.30 0.00 0.30 0.60 0.90

LayoutEasy to read

Easy to find infoPhotos & images Frequent update

Tourist infoClear info

Historical infoThematic discussion

Forum, blogsNew themes suggest

Municipality interactionModerator

Contribution to content Assistance for navigatSuggest improvement

Communic to users

-0.81-0.37

-1.51

-1.15

-0.76

-1.18

-1.37

-0.90

0.780.17

0.63

-1.81

-0.54

0.15

-0.65-0.20

-1.90

-1.11

-0.51

-2.02

-1.57

-1.06

-1.67

-1.91

-1.22

0.830.10

0.69-2.15

-0.62

0.14

-0.74-0.24

-2.17

GAP VALUES

Gap comparison

GAP AVERAGE FOR EACH PROPOSITION WEIGHTED GAP AVERAGE FOR EACH PROPOSITION

Fig. 7. SERVQUAL simple average gaps and weighted average gaps

27

The most important features for a CH website (higher expectations) are layout, design, usability and content, in particular:

Information easy to find and supported by images, clear information and frequently updated

Inclusion of e-Participation services in web-based cultural city services is of less demand from the citizens compared to other attributes

Conclusions (1)

28

CNSIS report (2010) examines the relationship of the Italians with ICT and concludes that only 43% of the web users totally trust this communication means (might explain the low desirability of e-participation)

Interacting with public administration to improve cultural services is currently seen as insignificant, confirming that the Web is yet to be taken by the citizens as a true instrument for participation in collective decision-making

Conclusions (2)

29

Small sample size, does not allow for differentiation among stakeholders types

Study the relationship between respondents answers and their socio-demographic profile (age, gender, education, …)

Include analysis of service providers and compare with expectations/perceptions of users

Limitations and further research

30

THANK YOU

aline.chiabai@bc3research.org

Basque Centre for Climate Change (BC3) Bilbao, Spain