Global Business Research Journals - Effects on Customers’...

Global Review of Research in Tourism, Hospitality and Leisure Management (GRRTHLM)

An Online International Research Journal (ISSN: 2311-3189)

2018 Vol: 4 Issue: 1

485 www.globalbizresearch.org

Effects on Customers’ Emotion via Changes of Ambient Light and

Music in Coffee Shop from Service Design View

Mei-Hua Hsih,

Graduate Institute of Design Science, Tatung University,

Director of Ikari Coffee Chain Stores, Taiwan, R.O.C.

E-mail:[email protected]

Chen-Chiung Hsieh,

Department of Computer Science and Engineer,

Tatung University, Taiwan, R.O.C.


Jian-Xin Yang, Department of Computer Science and Engineer,



Yun-Maw Cheng, Graduate Institute of Design Science,



___________________________________________________________________________

Abstract

The study explored the critical needs of external customers and internal customers of Ikari

coffee chain stores in Taiwan from a service design perspective. By using semi-structured

interviews, induction questionnaires, KJ mining method, and brainstorming, all of them

generated ideas for service design. After compiling, classifying, inductively reasoning and

analyzing collected data, this study has identified critical needs. Among them, we found that

the atmosphere of the coffee shop such as ambient light and music in the service design is the

critical need for customers. We hope customers’ emotion is joyful when they enjoy their meal

at Ikari coffee chain stores. The aim is to have positive effects on customers’ emotion. To

achieve this, we adopt Internet of things (IoT) techniques to change both lights and music

according to the detected facial expressions of customers. Firstly, customer image is sampled

every second to detect face and extract features related to expression. Secondly, extracted

features are fed into an artificial neural network (ANN) to predict the face expression of

customers which are stored to a database server. Then, the server checks the numbers of each

kind of expressions of customers at a regular time interval to pick the maximum as the base to

change light and music styles. The color of spotlight suiting most customers are sent to the

network bridge of light control, and it changes colors of the light bulbs through ZigBee.

Similarly, suitable music is selected and played. The experiment found that the number of

customers who were happy grew, and the number of negative emotions decreased. The results

successfully demonstrated the capabilities of our proposed IoT system. Furthermore, we

verified the explored critical needs of external customers and internal customers in service

design point of view to create a more valuable and outstanding brand for Ikari coffee chain

stores.

__________________________________________________________________________

Key Words: Service design, expression recognition, ambient light and music, neural network,

IoT.

JEL Classification: Z3, Z31





1. Introduction

As the Taiwan coffee chain market expansion since 1990, the ritual of drinking coffee has

been ushered into Taiwan. To run a business with a coffee chain brand, we need to preserve

continuously competitiveness. Not only increase the number of stores to reach a solid

economical scale but also commit themselves to innovation, differential marketing to

cultivate the brand, striving to create high-quality environment, and providing expertise in the

field to win the hearts of customers.

Especially in the latest three years, coffee chain stores faced with changes in economy,

politics, and technology. Based on this, aside from price competition considerations, they also

should be aware of customers’ needs. Products and services are the most basic needs of the

customer. However, what exactly is a good product? What on earth is good service? They

both depend on the customer’s experience, value judgment of quality, and service quality of

the product.

To represent the image of brand name, in addition to products and services, what are

customers’ needs and wants? It must be the one that is compatible with the customers. It takes

into account all the customers’ touch point and examines whether service design meets their

needs from the customer's point of view. And then, we can create a mutual value and give

birth to the ideal coffee shop that customers expect.

In this study, Section 2 gives some popular adopted methodologies to investigate the

critical needs of customers. Some environmental factors often used to change customers’

emotion are also given. Section 3 defines our service design and identifies customers’ critical

needs from external and internal customers. After that, two customer’s critical needs are

chosen to tie with customer emotion. Section 4 carried out the proposed internet of things

(IoT) system architecture for emotion detection and mood changing mechanisms. All the

implementation details are given. Section 5 describes the experimental results and presents

discussion. Conclusions and future works are given at last.

2. Literature Review

2.1 Service Design

Service design in a narrow sense refers to only "service" design and "process" design. In

the restaurant, the "service" design means the table service (stand for customer ordering, or

squat down to a meal), self-service, or half self-service. "Process" design means service

blueprint from customers greeting, eye contact with a face smile in 5 seconds, 15-degree bow,

gesture guiding seats, to introducing the meal, body language, customers’ enjoying a meal till

customers leave, and thanks. All customer contact steps are covered. Service design varies in

broad sense. In addition to the narrow service design and process design, in single enterprise’s

management, it integrates into the design, production, marketing, research and development,





human resources, and financial management. The creative process is not only complicated,

but also the results create a lot of presents. These will let internal customers and external

customers show high satisfaction, high loyalty, and then sublime to create a happy society [1-

3].

External customer means organizations or individuals who accept the product or service,

including the final customers, users, beneficiaries or purchasers; internal customer means

personnel and departments within the organization, including shareholders, operators, and

employees. The critical needs refer to in any enterprise in the professional field meet the

requirements customers care about most so that enterprises have a better competitive

advantage [4, 5].

The KJ mining method (affinity diagram) [6] was developed by Kawakita Jiro in 1953 and

used to organize data into useful categories or transform data into information in other words.

It adopts the bottom-up sorting process and is very useful for classifying data. To create an

affinity diagram, a group of people, index cards or sticky notes, and physical space are

required. The steps in the KJ method are (1) determine the theme, (2) gather data, (3) sort data

into groups, (4) create header cards, and (5) draw finished diagram. In step (3), people first

transform data into index cards or sticky notes; scatter the cards on a table or post the notes on

a wall; and then arrange the cards according to related ideas, issues, or topics.

2.2 Related Service Design using Lighting and Music

Researches [7, 8] show that music can alter people moods and emotions. People rely on

music to produce changes in experienced affective processes. Chang [9] found that lighting,

color, and events can be used to create atmosphere and influence the feelings of subjects.

Lighting can influence mental quality and sense by changing color and other characteristics.

Variations in color shades can satisfy consumers in their current entertainment context, so that

light can interact with human emotions with a new definition. Yeh [10] investigates the

lighting techniques and skills that employed to the commercial spaces of chained coffee

shops. Specifically, front projection lighting produced a sense of peacefulness in the subjects;

back projection lighting did not create considerable emotional reactions in the subjects; and

down projection lighting evoked the maximal levels of pleasure and peacefulness to the

subjects.

The structures and reaction mechanism of human bodies are complicated and customers’

moods would easily be affected by vibes around us, such as lighting, music, temperature,

climates, and so on. Recently, more and more visual products have surfaced like monitoring

software that detects human faces and to identify who they are and consequently replaces the

traditional clock-punching system. We try to transform customers’ depressive moods into

pleasant ones, or from negative to positive through the changing of lights and music.





3. Customers’ Critical Needs Investigation

In this research, Ikari coffee chain, Taiwan, founded in 1994, and owns 58 stores [11] is

the subject for the study. We use qualitative case study to explore the critical needs of the

external and internal customers from service design perspective. The external customers were

divided into two groups, the family group and the office worker group, in order to enable their

respondents to freely express their opinions. We adopted semi-structured interviews with

fifty-six guests that including twenty-eight family guests, twenty-eight office workers;

twenty-two male guests, thirty-four female guests; forty-four guests’ consumption frequency

of more than 1-2 times per week including twenty-two family guests, twenty-two office

workers, or nineteen for male guests and twenty-five female guests. The age statistics were

forty-six guests above forty years old, ten guests under twenty-nine years old as shown in Fig.

1.

Figure 1: Distribution of customer age

Figure 2: Questionnaires A (green) and B (light yellow)

First, we investigated the incentives of customers to come to Ikari stores and then

interviewed with the four spindles of interpersonal interaction design, space design, operation

flow design and the design of the ideal coffee shop using questionnaire A as shown in Fig. 2.

Some customers used the questionnaire B on the 60 questions to guide customers to answer if

they were unable to fully express their own opinion. Finally, the contents of the interview will

be collected and classified to find out the critical needs. We found out that the light and music

are the critical needs for customers in space design as shown in Fig. 3.





Figure 3: Critical needs of customers in space design

Internal customers include the supervisory level and the senior members. Arranged in a

two-day and one-night off job training camp, the internal customers were divided into two

echelons. Each echelon consisted of 35~70 persons that were divided into five groups. First,

we used the KJ mining method to start the definition of service design. Each group has to

report and analyzes the reasons by brainstorming method. Secondly, classify the items into

major branches. Third, by collecting all of the critical service design to the six major

dimensions: partner function development, environment, operating process, job design, brand

marketing, and intersectional cooperation dimensions. Each dimension has the final

conclusion that includes five critical needs as shown in the service design fishbone diagram of

Fig. 4. We also found out the light and music were the critical needs for customers in the

environmental dimension.

Figure 4: Fishbone diagram from service design view

4. IOT System Design for Emotion Changing

The Internet of Things (IoT) [12, 13] is an emerging global Internet-based information

architecture facilitating the exchange of goods and services. The IoT will serve to increase





transparency and enhance the efficiency of global supply chain networks. Extending the

initial application scope, the IoT might also serve as backbone for ubiquitous computing,

enabling smart environments to recognize and identify objects, and retrieve information from

the Internet to facilitate their adaptive functionality. The flow chart as shown in Fig. 5 is our

study methodologies. Left is the investigation of customer critical needs during service design.

Center is the proposed IoT flow chart for Ikari chain store. Right is to reach our study

conclusions.

Figure 5: Study flow chart

4.1 Facial Expression Recognition

Facial expression recognition is a process performed by humans or computers, which

consists of (1) locating faces in the scene or describing the texture of the skin in a facial area;

this step is referred to as face detection and facial features extracted from the detected face

region, (2) analyzing the motion of facial features and/or the changes in the appearance of

facial features and classifying this information into some facial-expression-interpretative





categories such as facial muscle activations like smile or frown, emotion categories like

happiness or anger, attitude categories like (dis)liking or ambivalence [14].

OpenCV [15] is a collection of ways of computing that includes computer vison, mechanic

learning, and vision- processing to build our system. It provides 2500 ways of computing and

is a free, open, and cross-platform BSD License. Computer vision computing can be used to

detect and identify human faces and moving objects to analyze their behaviors, trace moving

objects, reconstruct 3D objects/scenes, stitch images of surround view, retrieve similar images

from database, trace eyeball, and create augmented reality.

Here, we deploy functions such as face detection and ASM features points extraction [16].

ASM face model consists of 75 facial features points that are used to calculate the quantity of

facial expression movements. Figure 6(a) shows the distribution of these facial feature points.

Some regions of interest (ROI) on face are defined for expression identification. The ROIs are

eyebrows, wrinkled brow, wrinkled nose prints, left smile folds (nasolabial), right smile folds,

mouth, and mouth corners as shown in Fig. 6(b) and 6(c). We then calculate the quantity of

the changes of these seven ROIs and the degree of the depth of facial wrinkles.

Figure 6

(a) (b) (c)

Figure 6. (a) ASM 75 facial feature points. (b) and (c) are the selected seven ROIs used for

expression identification.

Inside of each ROI, Gabor filter [17, 18] is used for edge detection. The representation of

the filter’s frequency and direction is close to that of human visual system. The representation

can also serve as descriptors for wrinkles. Gabor filter is self-similar which can be produced

from a small original wave after expansion and rotation. In practical applications, Gabor filter

parameters include wave length, direction, phase, and band width, as shown in Figure 7(a),

7(b), 7(c), and 7(d), respectively. These filter can be used to extract related features but

different band width and directions [17-18].





Figure 7

(a) (b)

(c) (d)

Figure 7. Gabor filter. (a) Wave length 5, 10, and 15. (b) Angle 0, 45, 90, 135, and 180

degrees. (c) Phase offset 0, 180, -90, and 90. (d) Band width 0.5, 1, and 2.

Figure 8: Nasolabial of (a) 45 degrees and (b) 135 degrees

The direction and depth of wrinkles of nasolabial and mouth corners in the respective ROIs

are detected by Gabor filter. Wrinkles of left nasolabial and left mouth corner are detected by

45-degree Gabor filter, while those of right nasolabial and right mouth corner are detected by

135-degree Gabor filter. Figure 8 shows the detected nasolabial. Once the Gabor edge exists

in the ROI, the feature value is true else false.

Artificial Neural Network (ANN) [19] is then used to do the facial expression recognition.

The input layer is the feature values of the seven facial ROIs. ANN training process is to feed

these extracted feature values into the ANN for automatically updating the neural weights.

After that, the trained ANN could be used for recognition.

4.2 IoT System Implementation

Figure 9 shows the system architecture which is used for the implementation. It uses video

camera to capture person’s face and then transmitted to server for expression recognition. All

the recognition results would be saved to a database and a mechanism, emotion state

transition diagram (ESTD) designed in Sec. 4.3, could be used to control the music and light.





Figure 9: System architecture

For safety reason, database server presumably is set not allowed to be connected except

the localhost so that we need to reconfigure it. Firstly, turn off server’s firewall to allow

outside packet to pass. Because DB server can only be monitored locally in default and it

refuses other IPs and thus we need to modify my.cnf to allow outside visits as shown in

Figure 10. Then, we need to add new users via phpmyadmin to allow outside users to access

the database.

Figure 10: Modification of my.cnf file

JavaScript is used to implement some functions to control smart light bulbs by using

built-in APIs. These functions define bulb-related setups such as the IP Address of Network

Bridge and the color/saturation of light. These functions embedded in index.php could refresh

the homepage every 30 seconds and reads the latest data from the database server. Here we

deploy JavaScript functions to send the color/brightness of light to the corresponding bulb id

via Network Bridge. The index.php for light and music are as shown in Figure 11(a) and

11(b), respectively. After Network Bridge receives commands, it changes the colors of bulbs

by ZigBee. Capture the latest expressions of subjects and use MySQL command as below to

select the music suitable for most people from database as shown in Figure 11(c). The music

is played by use of the Audio label of HTML5 and the page is rearranged and the next music

is played again after 2 minutes.





Figure 11

(a) (b)

SELECT `data` FROM `music_data_mood A` WHERE `data` in

(SELECT `data` FROM `music_data_mood B`) WHERE `data` in

(SELECT `data` FROM `music_data_mood C`)

(c)

Figure 11. (a) Index.php for music. (b) Index.php for light. (c) The SQL command used to

select the music suitable for music_data_mood A, B, and C, where A, B, C are the top three

moods of guests.

4.3 Emotion State Transition Diagram

We adopt wisdom ambient light control as shown in Figure 12 by Philip hue API and

classify music into four types: classical, jazz, pop, and light music. We summarize the above

mentioned researches in Sec. 2.2 to propose the emotional state transition diagram (ESTD) as

shown in Table 1 and Fig. 13. ESTD utilizes the RGB values [20] and C.I.E. chromaticity

diagram [21] to change the ambient light and music to affect human expression. There are

totally nine colors adopted to change customer’s emotion.

Figure 12: Color lighting effects





Table 1: Detail description of the ESTD





Figure 13: The emotional state transition diagram (ESTD)

5. Experimental Results and Discussion

5.1 Experimental Setup

We did much preparation, practice, and arrangement. At the pretest, experimented with

designing the laboratory atmosphere as a restaurant, in addition to ambient light and four

different genres of music, also included tables and chairs, full service catering, service staff,

questionnaires, and the arrangement of the whole process as shown in Fig. 14(a), 14(b), and

14(c). A review of pre-test, the following four experiments were changed to as a simple

afternoon tea mode. The experimental group is under the variation of music and ambient light

while the contrast group is without changing music and ambient light.

The camera captures customer’s face and then doing facial expression recognition once

every second. The system sums up the identified expressions every 30 seconds to avoid

changing light frequently to feel uncomfortable. However, the music shifts to the next style

every 2 minutes for the average music length. That is, we get data every 2 minutes to change

music. The expressions showing up the most number of times is selected and sent to database

server.

5.2 Facial Expression Recognition Results

Figure 15 shows the expression recognition of the seven defined expressions. Twelve

persons, six males and six females, were invited to join the experiments. Each kind of

expression is actually tested 14 to 15 times for every subject. The data amount is more than

1,000 in total for the seven kinds of expressions and the recognition rate is 79.28% in average.





Figure 14

(a)

(b)

(c)

Figure 14. (a) Pre-test preparation. (b) Color light bulbs and cameras placed in the flower pot.

(c) Some participants in the experiments.

Figure 15: Recognition rates of the proposed facial expression recognition method





5.3 Effects on Customer Emotion

Each experiment took about an hour. This study uses the change quantity of facial

movements and the depth change of facial wrinkles as the base of expression identification.

However, in fact, not all human emotions are reflected on faces. Everybody has different

ways to express emotions because they have different personal traits; they have different

wrinkle depths due to different ages. For example, introvert and shy people are not good at

exaggerated facial expressions. We selected 25 video clips that were less affected by noises

and appeared normal from over 50 video clips.

In contrast, subjects did the experiment again with light and music control in the

environment. We compared the numbers of subjects who were happy without changing light

and music to that with changing light and music. The number of subjects who were happy

grown from 9 to 14 which proved that light and music could substantially change guests

emotions. As shown in Fig. 16, the results successfully met our original research goal

according to the subjective description of subjects and the objective observation of guest

facial expressions.

Figure 16: The number of happy subjects increased if ambient light and music were deployed

6. Conclusions and Future Works

Ikari coffee chain, already 24 years old, has gone through growth, maturity, continuous

innovation and change during this period, and now is still stably expanding. It not only wishes

to have a good presentation to internal customers and external customers, but it also needs to

shoulder the social responsibility in the ever-changing global environment. Guests 24 years

ago are mostly middle-aged now. To attract the younger generation is becoming increasingly

necessary. Therefore, besides the company's positioning, service design should turn the trend

toward versatility and tastes of the youngsters. Popular tools of service design are available

and many have been used in this study. We may try to use the others in the next study.

In the service profit chain, the quality of internal service covers job design, service design,

the reward for internal customers and development. Satisfied and blessing-feeling internal





customers often not only affect the productivity and retention rates, but they also showcase

the value of external service to consequently create high loyalty and promote the revenue and

profit growth. This time, after discussing the critical service design needs of 70 internal

customers, Ikari coffee hopes to really do a good job about the six dimensions and the 30

items. It also deeply wishes that internal customers will have a high level of satisfaction, and

then the company will sublimate into a blessed business.

In the experiments using proposed IoT system, we could not limit subjects’ movements and

ask them to face the cameras all the time. In addition to ambient light color could affect

person mood, but also different periods of music playback that may have effects. However,

not all subjects would have the same affection for the given music. These could also be listed

for the future works. From technical point of view, the whole system could be implemented

into mobile devices on Android and iOS. Ultimately, it saves lots of cost and lifts the limits of

equipment.

Though this study, still there is need to be improved in real applications. If the proposed

IoT system is equipped in cafes, bookstores, art galleries and yachts, customers will feel the

differences the changes of ambient light and music bring. The differences also vary according

to venues and activities they are doing. The bigger the venue is, the huger the impact is. This

study will be test extensively in Ikari coffee chain store after we have finished the revision of

some processes and solved some hardware/software integration problems.

References

B. Mager, S. Miettinen, and M. Koivisto (Eds.). (2009). Designing Services with Innovative Methods.

Chen Xi Yuan. (2009). Research Property Management. Service Quality, Customer Demand and

Customer Satisfaction.

Daniel Västfjäll. (2001). Emotion Induction through Music: A Review of the Musical Mood Induction

Procedure. Musicae Scientiae, 5(1), pp. 173 – 211, 2001.

David Matsumoto and Hyi Sung Hwang. (2011). Basic Research Leads to Training Programs that

Improve People’s Ability to Detect Emotions. Psychological Science Agenda, 2011.

Emmanuel Bigand, Suzanne Filipic, and Philippe Lalitte. (2005). The Time Course of Emotional

Responses to Music. The New York Academy of Sciences, pp.429–437, 2005.

H. Hsu and M. Wang. (2012). Detection of Copy-Move Forgery Image using Gabor Descriptor. Anti-

counterfeiting, Security, and Identification, pp. 1-4, 2012

Hue Developer Program: http://www.developers.meethue.com/documentation/hue-xy-values.

Hugh Dubberly. (2008). On Modeling: Learning Curves for Design. Interactions 15(4), pp. 13-16, July

2008. DOI: https://doi.org/10.1145/1374489.137449

Ikari Coffee Chain Sotores: http://www.ikari.com.tw.

Jen-Hsuan Yeh. (2016). The Study of Emotional Reactions Toward Different Lighting in a Restaurant

Space. Unpublished.

Marc Stickdorn and Jakob Scheneider. (2011). This is Service Design Thinking: Basics, Tools, and

Cases. Wiley; 1st edition, January 11, 2012.





Mou-Hsien Chang. (2015). A Case Study of the Product Development of Interactive LED Wireless

Sensor Lighting Products for Home Entertainment. Master Thesis of the Department of Industrial

Design, Shin-Chien University, Taipei, Taiwan, 2015.

Nasser H. Dardas and Nicolas D. Georganas. (2011). Real-Time Hand Gesture Detection and

Recognition Using Bag-of-Features and Support Vector Machine Techniques. Transactions on

Instrumentation and Measurement, 60(11), pp 3592 – 3607.

Peter A. RHODES. (2013).Rendering the CIE 1931 Chromaticity Diagram.

R. H. Weber and R. Weber. (2010). Internet of Things, DOI 10.1007/978-3-642-11710-7.

S. Haller, S. Karnouskos, and C. Schroth. (2008). The Internet of Things in an Enterprise Context, pp.

14-28.

Shizoom Wang and Simon Wong. (2015). OpenCV with Microsoft Visual Studio, ISBN:

9789864340279.

Stephan Dreiseitl and Lucila Ohno-Machado. (2003). Logistic Regression and Artificial Neural

Network Classification Models: a Methodology Review.

Tung-lin Wang. (2009). Patched Image Inpainting Based on Texture Segmentation and Structure

Reconstruction, Master Thesis of the Dept. of Electric Engineering, National Taiwan University of

Science and Technology, 2009.

Xiao Shang Hua. (2011). Customer Demand for Hualien. B & B Website Connotation Research

Investigation.

Ying-Mei Cheng and Sou-Sen Leu. (2010). Integrating Data Mining with KJ Method to Classify

Bridge Construction Defects. Expert Systems with Applications 38, pp. 7143–7150, 2011.

Global Business Research Journals - Effects on Customers’...

Documents

Transcript of Global Business Research Journals - Effects on Customers’...