Quality of Experience: What End-users Say About

Web Services?

Bipin Upadhyaya

Introduction

• Web services provided by different vendors have different qualities

• Quality of Service(QoS) refers to non-functional requirements, such as performance or availability requirements

• QoS deals with how a service behave at run-time

2

Service Selection Approaches

QoS aware architecture filters the low quality web services from a set of functionally equivalent services

3

Limitation of QoS

• Static Release• Not frequently updated • Environment & platform specific

• Runtime Monitoring• Evaluation of QoS at client-side • resource intensive, time consuming &

expensive

End users barely understand QoS attributes

4

User Experience

On average, users tell 15 people about good experiences and 24 people about bad experiences

A word-of-mouth recommendation is the primary factor behind

20% to 50%of all purchasing decisions

Ref: http://www.mckinsey.com/insights/marketing_sales/a_new_way_to_measure_word-of-mouth_marketing

Reviews

• Web 2.0 has enabled people to broadcast their experience to the mass

• Online user reviews are examples of such phenomenon

• Credible source of information

6

Heterogeneity in Reviews

7

Extract Relevant Information from Reviews

• Extract service features such as• Parts and Properties

• Opinions about the features are expressed as• Adjectives and Adverbs

#Hotel

Pool

ElevatorFood

Breakfast

Staff

Location

Room

8

Overview of Our Approach

9

POS Tagged Sentence

Extract

Sentence

& Tag POS

QoE Attributes & Opinions

Extract

QoE &

Opinion

Aggregated QoE

Attributes

Group

Similar QoE

Attributes

POS Tag for Review Sentences

Dropbox has great synchronization and folder

sharing capability.

Dropbox/NNP has/VBZgreat/JJ synchronization/NN&/CC folder/NN sharing/NN

capability/NN ./.

POS Tagger

10

Extract QoE Attributes and Opinion

(great, synchronization) (great, Folder sharing capability)

Dropbox/NNP has/VBZ great/JJsynchronization/NN &/CC folder/NN

sharing/NN capability/NN ./.

Detect negation and reverse adjectivesDetect POS patterns

11

Polarity of Opinion

• SentiWordNet is used to assign a score of opinion• Each word in WordNet is assigned a score• Allocates a sentiment score of positive,

negative or objective to words

12

(great, Synchronization) (great, Folder sharing capability)

Great –> 0.75

Cluster QoE Attributes

Performance (6)

Download (3)

Install (2)Ease to Use(4)

Operation (4)

Sync (3)

Synchronization(4)

Back-up (5)

Store (3)

Fast (4)

Upload (3)

Download (3)

Install (2)Ease to Use (4)

Setup(2)

Sync (3)

Synchronization(4)

Back-up (5)

Store (3)

Performance (6)

Operation (4)

Fast (4)Upload (3)

Setup(2)

Clustering

13

Select Candidate Element

• Rank each word in a cluster based on frequency and semantic similarity with other words

• Candidate elements can be used by users to specify preferences

Performance (6)

Operation (4)

Fast (4)

Upload (3)

QoE Attribute Rank

Performance 7.04

Operation 5.168

Upload 3.66

Fast 5.982

14

Research Questions

RQ1. Can our approach extract QoE from online

reviews?

RQ2. Does user sentiment about QoE attributes

changes over time?

RQ3. How do QoE attributes relate with QoS

attributes?

15

Case Study Setup

448 440 422 419

6980

5866

6611

4110

0

1000

2000

3000

4000

5000

6000

7000

8000

Trip Shopping Storage MappingService

Sentences containing QoE attributes and opinions

Sentences without QoE attributes and opinions

• Collect reviews for a periods of 13 months• Collect QoS attributes such as execution time

and availability for all services

16

RQ1. Can our approach extract QoE from online reviews?

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Travel Shopping Storage MappingService

Recall

Precision

Extract QoE attributes with high precision and recall

17

90%

72%

RQ2. Does user sentiment about QoE attributes change over time?

18

User’s sentiment about QoE attributes changes over time

RQ3. How do QoE attributes relate with QoS attributes?

High correlation between QoS and QoE Attributes indicates the possibility of using

QoE attributes for service selection. 19

20