Software Quality Metrics

1

Software Quality Metrics

Analyzing & MeasuringCustomer Satisfaction

(Chapter 14)

ByZareen Abbas

Reg# 169/MSSE/F07

Usman ThakurReg# 181/MSSE/F07

2

Overview-Quality

Product quality and customer satisfaction together form the total meaning of quality. Customer satisfaction is the ultimate validation of quality.

Quality is combination of two things:1. Product Quality and2. Customer Satisfaction

Small q and Big Q

3

Overview-Quality

4

Customer buy not product but assurance Producer sell their assurance not product

TQM-Total Quality Management sole focus

is linking quality with customer satisfaction

Overview-Quality

5

What is Customer Segment

Every business must decide "what market it is in" - what products and services it offers, to whom they will be offered, in which geographic area and which firms are its competitors

Once these strategic decisions are made, the business can refine "to whom it offers" its products and services into a set of customer or market segments

6

Process Model Approach

7

Why Customer Satisfaction

8


Many studies suggested that enhancing customer satisfaction is the bottom line of business success: Ever-increasing market competition Only way to retain the Customers To expand market share To gain more profit To enhance/ improve product satisfaction level

eg. 90% to 95%

Studies show that it is five times more costly to recruit a new customer than it is to keep an old customer:

Why is it costly?

9


It is fact that dissatisfied customers tell 7 to 20 people about their experiences

While satisfied customers tell Only 3 to 5 people

11

ISO Quality Policy and Customer Focus

12C:\Documents and Settings\Administrator\Desktop\Prest Customer satisfaction\Customer Metrics 411 Customer Satisfaction gooood.mht

13

Customer Satisfaction is the crucial requirement for a “good reputation”. Those who have it, will find all doors open

But those who have lost it, frequently perish or have to struggle hard to have their names associated with that magic word “Quality”

14

Customer Satisfaction

Total customer satisfaction is the primary quality issue.

Customers are the only people who can determine total customer satisfaction.

To achieve total customer satisfaction, the organization must know the customer, itself, its product, and its competition.

16

Metrics for Customer Satisfaction

Surveys

17

Metrics for Customer Satisfaction

Surveys

TQM, more and more companies are conducting surveys to measure their customers' satisfaction.

We will discuss customer satisfaction surveys and the analysis of survey data.

As an example, we describe an analysis of the relationship between overall customer satisfaction and satisfaction with specific attributes for a software product.

18

Why to Surveys for Customer Satisfaction

There are various ways to obtain customer feedback with regard to their satisfaction levels with the products and the company.

Like, telephone follow-up regarding a customer's satisfaction at a regular time after the purchase is a frequent practice by many companies.

Other sources include customer complaint data, direct customer visits, customer advisory councils, user conferences, and the like.

To obtain representative and comprehensive data, representative of the entire customer base.

19

Methods of Survey Data Collection

There are three common methods to gather survey data:

1. Face-to-face interviews (Personal)

2. Telephone interviews

3. Mailed questionnaires

The personal interview method requires the interviewer to ask questions based on a pre-structured questionnaire and to record the answers. The primary advantage of this method is the high degree of validity of the data.

20

Advantages and Disadvantages of Three Survey Methods

21

Sampling Methods

When the customer base is large, it is too costly to survey all customers. Estimating the satisfaction level of the entire customer population through a representative sample is more efficient.

To obtain representative samples, scientific probability sampling methods must be used. There are basic types of probability sampling:

Simple random sampling, Systematic sampling, Cluster (group) sampling.

22

How large a sample is sufficient?

The answer to this question depends on the confidence level we want and the margin of error we can tolerate.

The higher the level of confidence we want from the sample estimate, and the smaller the error margin, the larger the sample we need, and vice versa.

For each probability sampling method, specific formulas are available for calculating sample size, some of which (such as that for cluster sampling) are quite complicated.

The following formula is for the sample size required to estimate a population proportion (e.g., percent satisfied) based on simple random sampling:

23

Formula for sampling

24

Formula for sampling

N: Population = 10,000 Z = confidence level 90 % = 1.65 p = satisfaction level = 80 % B: margin of error = 5 %

n = 30,000*(1.65)2 x 80(1-80) .

30,000*52 x [1.652 x 80(1-80)]

n = ???

25

Sample Size (for 10,000 customers) in Relation to Confidence Level and Error Margin

26


27


28


29


30

Analyzing Satisfaction Data

31


Customer satisfaction is often measured by customer survey data via the five-point scale: Very satisfied Satisfied Neutral Dissatisfied Very dissatisfied.

32


Satisfaction with the overall quality of the product and its specific dimensions is usually obtained through various methods of customer surveys.

For example, the specific parameters of customer satisfaction by IBM include CUPRIMDA:

Capability Functionality Usability Performance Reliability Installability Maintainability Documentation/information Availability

HP include FURPS: functionality, usability, reliability, performance, andservice

33


The data are usually summarized in terms of percent satisfied. In presentation, run charts or bar charts to show the trend of percent satisfied are often used.

We recommend that confidence intervals be formed for the data points so that the margins of error of the sample estimates can be observed immediately

34

Quarterly Trend of Percent Satisfied with a Hypothetical Product

Traditionally, the 95% confidence level is used for forming confidence intervals and the 5% probability (p value) is used for significance testing.

35



36



37

Approaches for Analyzing

Satisfied Non-satisfied

Usually the 1st metric, percent satisfaction, is used. In

practices that focus on reducing the percentage of non-

satisfaction, much like reducing product defects, non-satisfied

metric is used.

38

Non-satisfied metrics

Although percent satisfied is perhaps the most used metric, some companies, such as IBM, choose to monitor the inverse, the percent non-satisfied.

Non-satisfied includes the neutral dissatisfied, and very dissatisfied in the five-point scale.

The rationale to use percent non-satisfied is to focus on areas that need improvement. This is especially the case when the value of percent satisfied is quite high.

Figure 12.3 in Chapter 12 shows an example of IBM Rochester's percent non-satisfied

39

In addition to forming percentages for various satisfaction or dissatisfaction categories, the weighted index approach can be used.

For instance, some companies use the Net Satisfaction Index (NSI) to facilitate comparisons across product.

The NSI has the following weighting factors: Completely satisfied = 100% Satisfied = 75% Neutral = 50% Dissatisfied = 25% Completely dissatisfied = 0%

40

Specific Attributes and Overall Satisfaction

Co-relationship between Attributes and overall satisfaction

41

Specific Attributes and Overall Satisfaction

The major advantage of monitoring customer satisfaction with specific attributes of the software, in addition to overall satisfaction, is that such data provide specific information for improvement.

The profile of customer satisfaction with those attributes (e.g., CUPRIMDA) indicates the areas of strength and weakness of the software product. One easy mistake in customer satisfaction analysis, however, is to equate the areas of weakness with the priority of improvement, and to increase investment to improve those areas.

Example Documentation and Reliability

42

On the other hand, customers may not like the product's documentation. To answer the question on priority of improvement, therefore, the subject must be looked at in the broader context of overall customer satisfaction with the product.

Specifically, the correlations of the satisfaction levels of specific attributes with overall satisfaction need to be examined. After all, it is the overall satisfaction level that the software developer aims to maximize

43

Variables

Dependent variable : Overall customer satisfaction

Independent variables: Attribute (Reliability)

44

How to see relationship of attributes with overall satisfaction

Two statistical approaches

Least-squares multiple regression (Ordinal)

Logistic regression

45

Logistic Regression

In statistics, logistic regression (sometimes called the logistic model) is used for prediction of the probability of occurrence of an event by fitting data to a logistic curve.

It makes use of several predictor variables that may be either numerical or categorical. For example, the probability that a person has a heart attack within a specified time period might be predicted from knowledge of the person's age, sex and body mass index.

Logistic regression is used extensively in the medical and social sciences as well as marketing applications such as prediction of a customer's propensity to purchase a product or cease a subscription.

46

Logistic Regression

For the logistic regression approach, we classified the five-point scale into a dichotomous variable (categorizes data into two groups)

1. Very Satisfied 2. Satisfied 1,2,3 = 03. Neutral vs4. Dissatisfied 4,5 = 15. Very dissatisfied

Very satisfied and satisfied (4 and 5) versus non-satisfied (1, 2, and 3). Categories 4 and 5 were recoded as 1 and categories 1, 2, and 3 were recoded as 0. The dependent variable, therefore, is the odds (probability) ratio of satisfied and very satisfied versus non-satisfied.

DV: A variable that categorizes data into two groups such as bankrupt versus solvent or energy company versus not energy company. Also called Dummy Variable

47

Logistic Regression

The odds ratio is a measurement of association that has been widely used for categorical data analysis.

48

Ordinary regression

Both approaches overall customer satisfaction is the dependent variable, and satisfaction levels with UPRIMD-A are the independent variables.

The purpose is to assess the correlations between each specific attribute and overall satisfaction simultaneously. For the ordinary regression approach, we use the original five-

point scale. The scale is an ordinal variable. Sensitivity research in the literature, however, indicates that if the sample size is large (such as in our case), violation of the interval scale and the assumption of Gaussian distribution results in very small bias. In other words, the use of ordinary regression is quite robust for the ordinal scale with large samples.

49

Analysis: Ordinary regression

50


51


52

Results of Ordinal Regression Analysis

53

Usability with reliability, installability, and documentation, and performance with availability are the more notable ones. In relation to overall satisfaction, reliability, usability, and installability have the highest correlations.

Reliability being the most significant variable implies that although customers are quite satisfied with the software's reliability (93.8%), reliability is still the most determining factor for achieving overall customer satisfaction.

54

Results of the Logistic Regression model

The most striking observation is that the significance of availability in affecting customer satisfaction is in contrast to findings from the ordinary regression analysis.

Now availability ranks third, after reliability and usability, in affecting overall satisfaction. The difference observed from the two models lies in the difference in the scaling of the dependent and independent variables in the two approaches.

Combining the two findings, we interpret the data as follows:

55

Results of Logistic Regression Analysis

56

Combining the two findings, we interpret the data as follows:

Availability is not very important in influencing the average shift in overall customer satisfaction from one level to the next (from dissatisfied to neutral, from neural to satisfied, etc.).

However, availability is very important in affecting whether customers are satisfied versus non-satisfied.

Therefore, availability is a sensitive factor in customer satisfaction and should be improved despite its level of high satisfaction.

57

2nd Part

58

Conditional Probabilities

Dependent variable of the logistic regression model (satisfied versus non-satisfied) is more appropriate for our purpose, we use the results of the logistic model for the rest of our example

59

Conditional Probabilities

60

Logistic Regression coefficients versus satisfaction Levels

61

Satisfaction with Company

Technical solutions: quality/reliability, availability, ease of use, pricing, installation, new technology

Support and service: flexible, accessible, product knowledge

Marketing: solution, central point of contact, information

Administration: purchasing procedure, billing procedure, warranty expiration notification

Delivery: on time, accurate, post delivery process Company image: technology leader, financial

stability, executives image.

62

Satisfaction with Company

It is apparent that customer satisfaction at both the company level and the product level needs to be analyzed and managed.

Knowledge about the former enables a company to take a comprehensive approach to total quality management; knowledge about the latter provides specific clues for product improvements.

Yet another type of analysis centers on why customers choose a company's products over other companies', and vice versa.

This kind of analysis requires information that is not available from regular customer satisfaction surveys, be they product level or company level. It requires data about customers' decision making for purchases and requires responses from those who are not the company's current customers as well as those who are. This type of analysis, albeit difficult to conduct, is worthwhile because it deals directly with the issue of gaining new customers to expand the customer base.

63

How good is good enough

Should be 100% for long term. Should consider competitors. Even if there are no competitors unsatisfied

customers would look for different alternatives. That becomes a trigger for emerging competitions.

So total customer satisfaction should be the focus. Should my company invest $2,000,000 to improve

satisfaction from 85% to 90%? Given that my company's customer satisfaction is at 95%, should I invest another million dollars to improve it or should I do this later?

64

Satisfaction level vs Market share

Algorithm for market share.

65

Example Comparison:

66

the answer to the "how good is good enough" is obvious: You have to be better than your competitors. Therefore, it is important to measure not only one's customer satisfaction level, but also the satisfaction level of one's competitors. Indeed, many companies have been doing exactly that. Do Benchmarking

67

Recommendations!

Total quality management (TQM) literature recommends three key elements of a good customer satisfaction management program: a post purchase call back program, a complaint management process. And a customer satisfaction survey

program.

68

For Small Organizations

For small organizations that don't have a customer focus program in place, It is recommended to integrate one or more of these practices into their development process.

Good requirements. Design FVT (Functional Verification Test) Customer like Testing Internal Beta Customer Beta Validation and feedback for further improvement Special Inbound customer testing.

69


Requirements: Good requirements-gathering and analysis techniques always

involve the users closely. For long-term product strategy, customer advisory councils usually provide good directions

For Extreme Programming (XP), customer involvement is regarded as so important that an onsite customer is part of the development team and is responsible for domain expertise and acceptance testing. For object-oriented processes in general, the role of use cases in requirements analysis is another illustration of the importance of user involvement.

Design: The prototyping approach is a good way to get customers'

feedback and to ensure that the product satisfies requirements. In recent years, some companies use the term user-centered design (UCD) to signify the heavy user involvement in their design process.

70


Functional verification test (FVT): Customer testing can start early, when FVT is near completion.

This early customer testing is not a beta program, and is called a hackers' invitational or customer invitational program. A selected small number of customer experts who know the product well are invited to the vendor's development laboratory to test the new function and to attempt to break the system.

Customer like testing: Unit tests and functional verification tests are normally for

functional defect removal. For product-level testing or system testing, which are normally conducted by an independent test team, many software organizations adopt a customer like testing strategy. The testing environments and the test scenarios simulate customer operations, and therefore offer the chance to find latent defects that are likely to be encountered by customers.

71


Internal Beta, or eating one's own cooking: When product development is about complete, and independent testing

about to start, the development organization can put the product into production mode running its own business operations. This strategy puts the development team into the customer's shoes and can really motivates the team's quality focus.

Customer beta: This is a broad-based customer validation program. It usually starts

when final testing is about done and before the product is released to the market. The product has to be at a fairly good quality level at this stage. The main purpose is to get customer validation and feedback for further improvement. Customer beta programs should not be used as an extended development phase as a major activity for defect removals.

Special inbound customer testing: Unlike the broad-based beta programs, this strategy is to invite a small

set of one's core customers to the development laboratory to test applications before releasing the product to the market. This approach focuses on depth, and is relevant for a small set of customers who depend on the software for their mission-critical business applications

72

Conclusion

Finally, we emphasize that measuring and analyzing customer satisfaction is but one element of customer satisfaction management. A good customer satisfaction management process must form a closed loop of measurement, analysis, and actions. While it is not the intent of this chapter to cover the customer satisfaction management process, we recommend that such a process cover at least the following elements:

Measure and monitor the overall customer satisfaction over time, one's own as well as key competitors'.

Perform analyses on specific satisfaction dimensions, quality attributes of the products and their strengths, weaknesses, prioritization, and other relevant issues.

Perform root cause analysis to identify inhibitors for each dimension and attribute.

Set satisfaction targets (overall and specific) by taking competitors' satisfaction levels into consideration.

Formulate and implement action plans based on the above.

73

Extra

74

Outcomes of measuring customer satisfaction

Measuring customer satisfaction provides you with valuable information from the one group most critical to your business success - your customers.

If customer satisfaction surveys are conducted accurately, you should be able to discover the following key insights:

1. Specific areas where customer service improvements are necessary

2. Expectations of your customers regarding customer service

3. Specific reasons for customer dissatisfaction

4. How to improve retention of existing customers

5. How to make customers feel valued

6. Missed opportunities to demonstrate your capacity to solve problems

75

Why Include Customers of Competitors

The rationale for including non-customer (customers of competitors) benchmarking is that without the non-customer data customer satisfaction levels are arbitrary.

Both sets of data allow an organization to exploit its strengths, and put initiatives in place to narrow and eliminate any gaps between expectations and performance.

In fact, the best performing organizations benchmark themselves against: their best competitor the industry average a world class supplier in a similar industry

76

How Exit InterviewsCan Translate into Huge Profit Increases

By measuring only customer satisfaction levels, organizations miss former customers who have left ... because they no longer had the need for, or were unhappy with the products or services being offered.

Measuring customer retention, on the other hand, relates directly to the bottom line. Long term customers spend more, refer new clients and are less costly to do business with.

Ironically, past customers present every company with an opportunity. They can tell the organization exactly what parts of the business to fix in order to reduce the number of customers at risk. This improves customer retention and, subsequently, profitability.

An average organization loses about 15% of its customers every year. But if this can be reduced to 10%, bottom line profits improve 35% to 85%.

Finding out why customers leave can often be difficult since the majority of unhappy customers don't complain, they simply quit. Exit interviews solve this problem.

77

Achieving Continuous Quality Improvement

Continuous quality improvement begins by identifying customer expectations for all key "moments of truth" - the critical interactions customers have with the organization.

The best way to understand customer expectations is to listen to customers using qualitative research techniques.

This usually requires skillful probing by someone practiced in customer satisfaction measurement.

After identifying expectations, customer satisfaction can readily be measured, how?

However, this requires the customer to answer specific questions about how he or she feels about the company's performance.

This is why it is so important to capture their interest and build the credibility needed to gain their cooperation. The task is made considerably easier by speaking the customer's language and presenting only issues that are truly significant.

78

Customer Focus

79

A Predictor of Financial Performance

Measured the right way, customer satisfaction is a leading indictor of a company’s financial performance. Managed scientifically using a proven, standard metric, customer satisfaction drives loyalty, profitability and growth. The American Customer Satisfaction Index (ASCI) is one of the best-in-breed solutions for customer satisfaction measurement that is tied directly to financial performance.

80

Identify customer needs and expectations

The identification of customer needs and expectations requires systematic, thorough, and continuous communication. The most important aspect of this process is to listen to the customer.

Once customer needs are identified, these needs must be monitored continuously to ensure that the product and/or service still satisfies them.

81

Develop customer relationships

Customer relationships are the core of customer-driven project management. This relationship ensures continuous customer satisfaction.

Customer relationships require communication, support, and responsiveness.

Trust is absolutely essential for any relationship and is necessary to ensure full participation of all members on the customer-driven team.

82

Thank you

Q & A

Software Quality Metrics

Documents

Transcript of Software Quality Metrics