Automated Triage in Digital Primary Care

IN DEGREE PROJECT INDUSTRIAL ENGINEERING AND MANAGEMENT,SECOND CYCLE, 30 CREDITS

, STOCKHOLM SWEDEN 2020

Automated Triage in Digital Primary CareAssessing the Potential of Using Multi-Criteria Decision-Making Models

ALBIN GRANELL

CHRISTOFER BORÉN

KTH ROYAL INSTITUTE OF TECHNOLOGYSCHOOL OF INDUSTRIAL ENGINEERING AND MANAGEMENT

Automatiserat Triage i Digital Primärvård

Utvärdering av potentialen att använda Multi-Criteria Decision-Making-modeller

av

Albin Granell Christofer Borén

Examensarbete TRITA-ITM-EX 2020:285 KTH Industriell teknik och management

Industriell ekonomi och organisation SE-100 44 STOCKHOLM

Automated Triage in Digital Primary Care

Assessing the potential of using Multi-Criteria Decision-Making models

by

Albin Granell Christofer Borén

Master of Science Thesis TRITA-ITM-EX 2020:285 KTH Industrial Engineering and Management

Industrial Management SE-100 44 STOCKHOLM

Examensarbete TRITA-ITM-EX 2020:285

Automatiserat Triage i Digital Primärvård

Albin Granell

Christofer Borén

Godkänt

2020-06-05

Examinator

Jannis Angelis

Handledare

Anna Svarts Uppdragsgivare

Kontaktperson

Sammanfattning

Det ökande underskottet av sjukvårdsresurser gör effektivitetsförbättringar i sjukvårdsbranschen nödvändigt för att säkerställa säker och tillgänglig sjukvård för alla. Digitalisering förväntas fylla en fundamental roll i denna transformation och digitala vårdgivare i primärvården har under de senaste åren växt till en betydande del av den svenska primärvårdssektorn. Flertalet av dessa har byggt lösningar för automatiserat triage, där triagefunktionärens roll ersätts av en automatiserad process där en triagealgoritm direkt hänvisar patienten till den lämpliga vårdnivån.

Trots tillväxten av digitala vårdgivare i primärvården och deras automatiserade triagesystem i primärvården är forskning kring effekterna av att automatisera triageprocessen i primärvården begränsad. Denna studie strävar efter att utvärdera potentialen i att använda MCDM-modeller för automatiserat triage i den digitala primärvården genom en casestudie på en av de ledande digitala vårdgivarna i primärvården. Studien är uppdelad i två delar. I del ett genomförs intervjuer för att kvalitativt fastställa vilka faktorer som bör inkluderas i en automatiserad MCDM-modell för triage. I del två simuleras den resulterande MCDM-modellen för att utvärdera dess resultat jämfört med den traditionella triagemodellen i vilken alla patienter har ett inledande möte med en sjuksköterska.

Studien visar att en automatiserad MCDM-modell för triage kan förbättra kostnadseffektiviteten i termer av lönekostnader och produktivitet i termer av färre konsultationer per patient, jämfört med den traditionella triagemodellen. Däremot visar den traditionella triagemodellen högre effektivitet i termer av att enbart utnyttja läkarresurser för patienter i absolut behov av läkarvård.

Nyckelord

Triage, Sjukvård, Primärvård, Digital sjukvård, Multi-Criteria Decision-Making

Master of Science Thesis TRITA-ITM-EX 2020:285

Automated Triage in Digital Primary Care

Albin Granell

Christofer Borén

Approved

2020-06-05 Examiner

Jannis Angelis Supervisor

Anna Svarts Commissioner

Contact person

Abstract

The increasing global deficit of healthcare resources makes efficiency improvements in the healthcare industry a complete necessity to assure safe and available healthcare for everyone. Digitalization is expected to play a fundamental role in this transition and digital primary healthcare providers have in recent years developed into a substantial part of the Swedishprimary care sector. Several of those have built solutions for automated triage, where the role of a triage officer in traditional primary care is replaced by an automated process, in which an triage algorithm directly refers the patient to the appropriate level of care.

Despite the rise of digital healthcare providers and automated primary care triage systems in particular, research on the implications of automating the triage process in primary healthcare is scarce. This study aims to assess the potential of using MCDM models for automated triage in digital primary care, by conducting a single case study at one of the leading digital healthcare providers. The study is separated into two phases. In phase one, interviews are conducted to qualitatively determine what set of factors to include in an automated MCDM triage model.In phase two, the resulting model is simulated to evaluate the performance compared to the traditional triage model in which all patient journeys start with an initial nurse meeting.

The study shows that an automated MCDM triage model can improve cost efficiency in terms of clinician salary costs and productivity in terms of fewer consultations per patient, compared to the traditional triage model. However, the traditional triage model is shown to be more efficient in terms of only utilizing doctor resources for patients in absolute need of doctor care.

Key-words: Triage, Healthcare, Primary care, Digital healthcare, Multi-Criteria Decision-Making

Contents

1 Introduction 1

1.1 Background to Digital Triage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Objective & Research Question . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2.1 Evaluating Efficiency, Productivity and Patient Experience . . . . . . . . 3

1.3 Delimitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.4 Expected Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.5 Report Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Triage Literature 7

2.1 Formal Definition of Triage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2 Previous Research on Triage Models . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.2.1 Triage in Emergency Care . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.2.2 Triage Judgment Frameworks in Swedish Primary Care . . . . . . . . . . 10

3 Empirical Context 11

3.1 The Swedish Primary Care System . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.1.1 The Governance System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.1.2 The Act on System of Choice in the Public Sector . . . . . . . . . . . . . 12

3.1.3 Public Reimbursement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.1.4 Public Reimbursement of Non-Resident Patients . . . . . . . . . . . . . . 14

3.1.5 Triage As a Requirement For Reimbursable Digital Healthcare . . . . . . 15

3.2 Guiding Principles in Swedish Healthcare . . . . . . . . . . . . . . . . . . . . . . 15

3.2.1 Prioritization and the Ethical Platform . . . . . . . . . . . . . . . . . . . 15

3.2.2 Principles of Cost Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.3 Digital Primary Care Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.3.1 Major Players and Triage . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.3.2 Criticism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4 Theoretical Background 21

4.1 Multi-Criteria Decision-Making . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4.1.1 MCDM Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

4.1.2 Evaluating and Comparing MCDM Methods . . . . . . . . . . . . . . . . 24

4.2 Simulation as a Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.3 Performance Evaluation for Predictive Modeling . . . . . . . . . . . . . . . . . . 25

5 Method 27

5.1 Research Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

5.2 Phase 1: Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

5.2.1 Selection of Interviewees . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

5.2.2 Interview Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5.2.3 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

5.3 Phase 2: Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

5.3.1 Simulation Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

5.3.2 Simulation Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

5.3.3 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

5.3.4 Obtaining the results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

5.3.5 Triage Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

5.3.6 Entering the Decision Matrix Values of the MCDM Model . . . . . . . . . 42

5.3.7 Determining the Relative Weights of the MCDM Model . . . . . . . . . . 44

5.3.8 Effect of Systematic Errors in the Digital Triage Assessment . . . . . . . 45

5.4 Evaluation of Research Method Quality . . . . . . . . . . . . . . . . . . . . . . . 46

5.4.1 Validity, Reliability and Generalizability . . . . . . . . . . . . . . . . . . . 47

5.4.2 7 Criteria of Mixed Method Research Undertaking Quality . . . . . . . . 48

6 Empirical Findings 50

6.1 Phase 1 - Qualitative Interviews (RQ1) . . . . . . . . . . . . . . . . . . . . . . . 50

6.1.1 Input Factors Included in Phase 2 Simulations . . . . . . . . . . . . . . . 51

6.1.2 Input Factors Not Included in Phase 2 Simulations . . . . . . . . . . . . . 52

6.2 Phase 2 - Simulations Comparing Triage Models (RQ2) . . . . . . . . . . . . . . 53

6.2.1 Overview of Simulation Results . . . . . . . . . . . . . . . . . . . . . . . . 53

6.2.2 Cost Efficiency - M1 Clinician Salary Costs . . . . . . . . . . . . . . . . . 54

6.2.3 Cost Efficiency - M2 LEON Enactment . . . . . . . . . . . . . . . . . . . 57

6.2.4 Healthcare Productivity - M3 Consultations per Patient . . . . . . . . . . 61

6.2.5 Healthcare Productivity - M4 Clinician Idle Time . . . . . . . . . . . . . . 64

6.2.6 Patient Experience - M5 Patient Waiting Times . . . . . . . . . . . . . . 65

6.3 Phase 2 - Simulations Evaluating Systematic Assessment Errors (RQ3) . . . . . . 66

6.3.1 Cost Efficiency - M1 Clinician Salary Costs . . . . . . . . . . . . . . . . . 67

6.3.2 Cost Efficiency - M2 LEON Enactment . . . . . . . . . . . . . . . . . . . 69

6.3.3 Healthcare Productivity - M3 Consultations per Patient . . . . . . . . . . 72

6.3.4 Healthcare Productivity - M4 Clinician Idle Time . . . . . . . . . . . . . . 75

6.3.5 Patient Experience - M5 Patient Waiting Times . . . . . . . . . . . . . . 75

7 Discussion 78

7.1 RQ1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

7.2 RQ2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

7.3 RQ3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

7.4 Reflection on Sustainability Aspects . . . . . . . . . . . . . . . . . . . . . . . . . 81

8 Conclusion 82

8.1 Summary and Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

8.2 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

8.3 Limitations and Further Research . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

References 89

A Empirical Nursability Distribution 90

B Empirical Meeting Length Distributions 90

List of Figures

1 MTS flow chart for traumatic injuries [15] . . . . . . . . . . . . . . . . . . . . . . 9

2 2x2 contingency table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3 Overview of the applied mixed method research process . . . . . . . . . . . . . . 28

4 Overview of the simulated patient flow . . . . . . . . . . . . . . . . . . . . . . . . 32

5 Illustration of patient flow in simulation model . . . . . . . . . . . . . . . . . . . 37

6 RQ2: Staffing distribution per model ID . . . . . . . . . . . . . . . . . . . . . . . 54

7 RQ2: Boxplot of hourly staffing cost in SEK . . . . . . . . . . . . . . . . . . . . . 55

8 RQ2: Boxplot of P3 Final Doctor Meeting End Time . . . . . . . . . . . . . . . . 55

9 RQ2: Boxplot of P4 Final Nurse Meeting End Time . . . . . . . . . . . . . . . . 56

10 RQ2: Boxplot of P5 Clinician Salary per Patient Journey (excl. cost for idle) . . 57

11 RQ2: Boxplot of P6 True Positive Rate . . . . . . . . . . . . . . . . . . . . . . . 58

12 RQ2: Boxplot of P7 True Negative Rate . . . . . . . . . . . . . . . . . . . . . . . 59

13 RQ2: Boxplot of P8 Positive Predictive Value . . . . . . . . . . . . . . . . . . . . 59

14 RQ2: Boxplot of P9 Negative Predictive Value for MCDM models . . . . . . . . 59

15 RQ2: Boxplot of P9 Negative Predictive Value for traditional models . . . . . . . 60

16 RQ2: Boxplot of P10 Accuracy for MCDM models . . . . . . . . . . . . . . . . . 60

17 RQ2: Boxplot of P10 Accuracy for traditional models . . . . . . . . . . . . . . . 60

18 RQ2: Boxplot of P11 Nurse Triage Rate . . . . . . . . . . . . . . . . . . . . . . . 61

19 RQ2: Boxplot of P12 Doctor Resource Management Rate . . . . . . . . . . . . . 61

20 RQ2: Boxplot of P13 Number of Patients . . . . . . . . . . . . . . . . . . . . . . 62

21 RQ2: Boxplot of P14/P13 Doctor Meetings per Patient . . . . . . . . . . . . . . 62

22 RQ2: Boxplot of P15/P13 Nurse Meetings per Patient . . . . . . . . . . . . . . . 63

23 RQ2: Boxplot of (P14+P15)/P13 Meetings per Patient for MCDM models . . . 63

24 RQ2: Boxplot of (P14+P15)/P13 Meetings per Patient for traditional models . . 63

25 RQ2: Boxplot of P18 50th Percentile of Waiting Time . . . . . . . . . . . . . . . 65


27 RQ2: Boxplot of P20 100th Percentile of Waiting Time . . . . . . . . . . . . . . 66

28 RQ3: Boxplot of hourly staffing cost in SEK . . . . . . . . . . . . . . . . . . . . . 67

29 RQ3: Boxplot of P3 Final Doctor Meeting End Time . . . . . . . . . . . . . . . . 68

30 RQ3: Boxplot of P4 Final Nurse Meeting End Time . . . . . . . . . . . . . . . . 68

31 RQ3: Boxplot of P5 Clinician Salary per Patient Journey (excl. cost for idle) . . 69

32 RQ3: Boxplot of P6 True Positive Rate . . . . . . . . . . . . . . . . . . . . . . . 70

33 RQ3: Boxplot of P7 True Negative Rate . . . . . . . . . . . . . . . . . . . . . . . 70

34 RQ3: Boxplot of P8 Positive Predictive Value . . . . . . . . . . . . . . . . . . . . 70

35 RQ3: Boxplot of P9 Negative Predictive Value . . . . . . . . . . . . . . . . . . . 71

36 RQ3: Boxplot of P10 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

37 RQ3: Boxplot of P11 Nurse Triage Rate . . . . . . . . . . . . . . . . . . . . . . . 71

38 RQ3: Boxplot of P12 Doctor Resource Management Rate . . . . . . . . . . . . . 72

39 RQ3: Boxplot of P13 Number of Patients . . . . . . . . . . . . . . . . . . . . . . 73

40 RQ3: Boxplot of P14/P13 Doctor Meetings per Patient . . . . . . . . . . . . . . 73

41 RQ3: Boxplot of P15/P13 Nurse Meetings per Patient . . . . . . . . . . . . . . . 74

42 RQ3: Boxplot of (P14+P15)/P13 Meetings per Patient . . . . . . . . . . . . . . 74

i



45 RQ3: Boxplot of P20 100th Percentile of Waiting Time . . . . . . . . . . . . . . 77

46 Empirical mass distribution of nursability scores . . . . . . . . . . . . . . . . . . 90

47 Empirical meeting length (minutes) distribution for nurse meetings with outcome

patient helped . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

48 Empirical meeting length (minutes) distribution for nurse meetings with outcome

referral to doctor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

49 Empirical meeting length (minutes) distribution for doctor meetings . . . . . . . 91

ii

List of Tables

1 Mapping of Principles of Cost Efficiency in Primary Care . . . . . . . . . . . . . 17

2 Overview of interview sample groups . . . . . . . . . . . . . . . . . . . . . . . . . 29

3 List of interviewees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4 Interview themes and topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

5 Expected meeting lengths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

6 Collected parameters for each simulated day . . . . . . . . . . . . . . . . . . . . . 39

7 The ideal triage models simulated . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

8 The traditional triage models simulated . . . . . . . . . . . . . . . . . . . . . . . 41

9 Parameters used for expected cost calculations . . . . . . . . . . . . . . . . . . . 44

10 The MCDM models simulated with their respective relative weight combinations 45

11 The systematic errors simulated . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

12 Table of factors mentioned in interviews . . . . . . . . . . . . . . . . . . . . . . . 50

13 RQ2: Summary of average daily results for the evaluated triage models . . . . . 53

14 RQ2: Average values of M2 LEON enactment parameters . . . . . . . . . . . . . 58

15 RQ2: Average values of M3 Consultations per Patient parameters . . . . . . . . 61

16 RQ2: Average values of M4 Clinician Idle Time . . . . . . . . . . . . . . . . . . . 64

17 RQ3: Average values of M2 LEON enactment parameters . . . . . . . . . . . . . 69

18 RQ3: Average values of M3 Consultations per Patient parameters . . . . . . . . 72

19 RQ3: Average values of M4 Clinician Idle Time . . . . . . . . . . . . . . . . . . . 75

iii

Acknowledgements

We would like to express our utmost gratefulness to our supervisor Anna Svarts at the Institution

of Industrial Economics and Management, who despite the COVID-19 situation has found ways

to continuously give us valuable in-depth feedback and quickly respond to questions whenever

needed.

We would also like to thank our partner company and all colleagues and interview participants

who allocated time to contribute to this thesis. The supporting environment that you have

provided has been a pleasure to work in. An extra expression of appreciation goes out to our

supervisor at the partner company.

iv

1 INTRODUCTION 1

Section 1: Introduction

In this chapter, the reader is introduced to the subject and objective of the study. The research

questions are presented, together with the groups of performance indicators used to evaluate

RQ2 and RQ3. Further, the delimitation and expected contribution of the study is presented.

1.1 Background to Digital Triage

The World Health Organization estimates that by 2035, there will be a global deficit of about 12.9

million healthcare professionals [1]. When there exists a scarcity of healthcare resources, decisions

must be made about how to allocate these resources. The ability to use available healthcare

resources cost efficiently is, and will continue to be of utmost importance. Digitalization of

healthcare services is projected to play a fundamental role in improving healthcare cost efficiency.

Known as triage from the French word trier ’to sort’, the practice of allocating resources to

patients originates from the battlefields of war in the beginning of the 18th century. At the time,

Napoleon’s army recognized a need to categorize wounded soldiers in order to prioritize treatment

for those who needed the most urgent medical attention. Today, triage is most prominently found

in emergency care units where the need of prioritizing the most critical patients reemerges.

When it comes to triage within the non-emergent primary care sector, research is limited and

official guidelines are few compared to those applicable for emergency care. Still, each Swedish

citizen visits primary care on average 1.33 times every year, indicating a high volume of

healthcare consultations that must be conducted efficiently. The role of triage is to enable such

cost efficient operations. In a systematic literature review, it was found that the available

evidence indicates that gate-keeping access to specialized levels of care at the primary care

level was associated with lower utilization of health services and lower expenditure [2].

Nonetheless, in the context of modest urgency, rather than to determine the urgency of

treatment, the objective of primary care triage in Sweden is to guide the patients to the

appropriate level of care. In traditional healthcare, patients calling or visiting their healthcare

center are given an initial assessment by the triage officer, usually a nurse, who guides the

patients to the appropriate level of care.

As Swedish healthcare is regulated by the Health and Medical Services Act on a nation-wide

level, healthcare must be performed so that priority is given to those with the greatest needs of

medical services. Furthermore, the law stipulates that publicly reimbursed healthcare must be

organized such that it encourages cost efficiency. One commonly cited triage policy is the LEON

principle (Lowest Efficient Level of Care) which guides triage officers to guide patients to the

lowest (i.e. least costly) level of care at which they can be efficiently helped. However exactly

how cost efficiency is supposed to be measured remains a debated topic.

1 INTRODUCTION 2

The practice of digital primary care has grown rapidly in Sweden since 2015. This enables pa-

tients to meet healthcare professionals for assessments and treatments through digital channels

such as text or video. As of 2020, the major digital primary care providers in Sweden employ

doctors, nurses and psychologists for these services. Having multiple professions available, just

like in traditional primary care, requires that some triage system guides patients to the appro-

priate level of care.

At several of the major digital healthcare providers, triage has become an automated process

without synchronous contact between the patients and the triage officers. Instead of describing

their symptoms to nurses, patients fill out a questionnaire and are directly guided to the

appropriate level of care based on their answers. Consequently, an accurate triage system could

potentially save salary costs, should it be able to perform the role of the triage officer. Nurses

currently employed as triage officers could also use their time to help more patients if such an

automated triage system was to be employed. However, if the automated triage system would

be less accurate in determining the appropriate level of care than nurses as triage officers, it

could lead to an overflow of patients being guided to a higher level of care than required which

is costly and a misuse of healthcare resources. Limited research has been done regarding these

potential savings and what requirements such an automated triage system must fulfill in order

to be beneficial.

Nevertheless, this first-iteration of automated triage employed today is still simple in its essence

and fails to account for many important factors such as patient experience, policies and societal

economics. For example, imagine the scenario where a patient who is deemed to be suited for

nurse care is placed in line for a video meeting with the next available nurse while there are

doctors ready to have immediate consultations with the patient. The patient experience of a

longer waiting time is not optimal and neither is the fact that a doctor could have helped the

patient, making room for more consultations for patients in need.

1.2 Objective & Research Question

Designing a digital triage system for primary care presents two main challenges. Firstly, the

system must be able to assess the patients’ medical needs and secondly, it must be able to make

a judgment based on certain criteria to guide patients to the appropriate level of care. In its

most simple form, one such system may simply ask patients what level of care they believe to be

in need of and consequently guide patients to that very level of care. In a more complex form,

such a system could generate an estimation of the chance that a certain level of care would be

able to fill the patient’s healthcare needs. In turn, the more complex system would take that

assumption into consideration alongside several other quantitative data points to produce its

judgment.

This study aims to investigate a triage system in the more complex setting, where the digital

service generates an estimation of the chances that the different potential levels of care could fulfill

the patient’s healthcare needs. As briefly previously discussed, the currently employed models of

1 INTRODUCTION 3

digital triage at some of the major digital healthcare providers today fail to account for many of

the factors that a triage officer would in the traditional sense of triage. Taking multiple factors

into account to produce a decision could be difficult if two different factors favors two different

outcomes. In operations research, the topic of Multi-Criteria Decision-Making (MCDM) covers

evaluation of multiple criteria to produce a decision where there does not exist a unique optimal

solution as some criteria might be conflicting. Employing MCDM in a digital triage setting is

one way of enabling an expansion of the information upon which to base a triage decision.

Based on this objective and previous background, this study aims to investigate whether digital

primary care enables usage of more complex MCDM-based triage policies with comparative

advantage to the traditional triage policy. This is performed by answering the three questions

below.

At first, a qualitative investigation is needed to outline the set of factors to construct the MCDM

algorithm upon.

RQ1 - What set of factors should be considered in an automated digital triage system?

Secondly, the performance of a complex MCDM-based triage model must be investigated and

evaluated through a set of relevant criteria.

RQ2 - Is it possible, using Multi-Criteria Decision-Making models to improve cost

efficiency, healthcare productivity and patient experience?

Finally, to understand the limitations of an automated triage system, the investigation seeks to

explain the impact of a triage system based on bad data input.

RQ3 - What is the impact of a systematic error in the digital triage assessment in terms

of cost efficiency, healthcare productivity and patient experience?

1.2.1 Evaluating Efficiency, Productivity and Patient Experience

The cost efficiency, healthcare productivity and patient experience of the healthcare system have

been evaluated by five groups of performance indicators. These groups have been decided upon

on the basis of Sweden’s national indicators and fundamental principles for good-quality health

and medical care, as set forth by the National Board of Health and Welfare (NBHW) [3]. We

determined which of the national indicators that have relevance within the scope of this study

and adopted them into the appropriate measurements. They are, under each respective field of

monitoring:

Cost Efficiency

1 INTRODUCTION 4

M1 - Clinician salary costs

M2 - LEON enactment

Healthcare Productivity

M3 - Consultations per patient

M4 - Clinician idle time

Patient Experience

M5 - Patient waiting times

1.3 Delimitations

At first, it should be noted that the scope of the study has been confined to the Swedish primary

care. That does not necessarily mean that the results obtained are not applicable to healthcare

systems in other countries, however the study has been designed based on Swedish healthcare

policy. Furthermore, the study will assume a primary care setting in which patients are to be

initially guided to one of two levels of care. The lower level of care is represented by nurses

and the higher level of care is represented by doctors. This is the most impactful delimitation

of this study since this greatly differs from a real setting in which patients may potentially see

healthcare workers of many different specialties and professions. However, as patients visiting

physical primary care generally start their journey by a nurse meeting regardless of where they

eventually get helped, we argue that such a delimitation does not limit our ability to answer the

research questions.

This study has been conducted in partnership with one of Sweden’s leading digital healthcare

providers. As such, the quantitative input data that has been used in the study to model patients’

needs has been generated by patients’ needs when seeking care at a digital healthcare provider.

Therefore, it can be assumed that the results of the study are best applicable for patients with

primarily symptoms suitable for digital care. Furthermore, for some patients present in the input

data figures, a previous assessment could potentially have taken place with a nurse by phone

which would impact the type of patients and symptoms making up the data.

The nature of the quantitative input data also means that the study disregards seasonal patterns

of the healthcare needs of Swedish primary care patients. As the conditions themselves may

change, it it however believed that the severity of the treated conditions has a small seasonal

effect and may be disregarded.

The study also assumes that all patients who seek care are in need of care and in fact can be

helped through a digital consultation. In reality, there will unavoidably be patients who visit

healthcare centers, physical and digital, that do not need healthcare or that need to be referred

to another healthcare institution. From a triage point-of-view however, these patients should be

referred to the appropriate institution before determining the appropriate level of care within

1 INTRODUCTION 5

the institution. Further, contact with experienced healthcare professionals and the analytics

department at our digital healthcare partner has confirmed that this share of patients is small.

As such, we will disregard these patients in the scope of our thesis.

Regarding the role of triage to distribute healthcare resources in order of patient needs, we will

assume a setting of digital healthcare in which patients have a comparable need for care. That

means that we study a triage system in which no patient is given priority over another patient

based on the urgency of the two patients’ healthcare needs. Rather, we assume a situation where

the role of the triage system is solely to determine the appropriate level of care. This is a suitable

delimitation when studying a digital healthcare setting as the patients seeking digital primary

care all tend to have concerns with little or no urgency.

It should be emphasized that this study does not focus on improving the assessment of patients,

but rather the judgment such an assessment should render. These stages of the triage process

is more formally defined in the next section. Moreover, when comparing digital tools for triage

with traditional judgments performed by a nurse, this study will assume that the judgment of

the nurse is completely accurate and that a nurse always will guide a patient to the lowest level

of care that can treat the patient.

1.4 Expected Contribution

The healthcare resource deficit makes efficiency improvements within the healthcare sector a

complete necessity to assure available and safe healthcare for everyone in the future. In a state

public report from 2019, it is concluded that tools for automated anamnesis and triage have a

great potential of unlocking healthcare resources by facilitating patients being directly referred to

the the most appropriate level of care [4]. Further, following a research request from Stockholm

region, Lagerros et. al. published a report in December 2019 studying the impact of digitalizing

primary healthcare [5]. It was concluded that automated triage systems show great potential,

but emphasized that there is limited research on the subject and that evidence based knowledge

about their efficiency, patient safety and resource utilization currently is limited and would be

of great value.

This study aims to bridge the knowledge gap about the implications of automated triage in

digital primary care. Historically, most triage research has focused on emergency healthcare,

but given the current state of the healthcare sector with and increasing resource deficit, there is

a great need of efficiency improvements in the primary care sector. By studying the potential of

using MCDM models as a tool for automated triage, we hope to contribute with evidence based

insights on the efficiency of automated triage systems digital primary care.

1 INTRODUCTION 6

1.5 Report Outline

The thesis proceeds as follows:

1) Introduction - In this chapter, the reader is introduced to the subject and objective of

the study. The research questions are presented, together with the groups of performance

indicators used to evaluate RQ2 and RQ3. Further, the delimitation and expected contri-

bution of the study is presented.

2) Triage Literature - This chapter presents the findings of previous and relevant research

conducted within the field of triage. Given the limited previous research conducted on

triage in primary healthcare, it is mostly focused on triage research in an emergency

healthcare setting. However, it also presents two triage decision support tools developed

for Swedish primary care.

3) Empirical Context - This chapter provides a description of the Swedish primary

healthcare system, including governance, laws, guiding triage policies and the emerging

sector of digital primary healthcare.

4) Theoretical Background - This chapter presents the theoretical background of the

MCDM model. It proceeds with the theoretical aspects of using simulation as a tool and

presents the performance evaluation framework used to analyze the simulation outcome.

5) Method - This chapter presents the method of the study. It accounts for methodological

choices and provides details of the processes associated with the interviews and simulation.

It is concluded with a discussion of the quality of research

6) Empirical Findings - This chapter presents the findings of the study. It is structured

in chronological order, starting with a presentation of the interview findings and then

proceeding with the simulation results. The simulation results are structured according to

the performance measurements presented in section 1.2.1.

7) Discussion - This chapter discusses the empirical results and is structured according to

the research questions. It also includes a discussion of the sustainability aspects of the

study.

8) Conclusion - This chapter presents the final conclusion to the stated research questions,

its contributions to science and suggestions for further research.

2 TRIAGE LITERATURE 7

Section 2: Triage Literature

This chapter presents the findings of previous and relevant research conducted within the field

of triage. Given the limited previous research conducted on triage in primary healthcare, it is

mostly focused on triage research in an emergency healthcare setting. However, it also presents

two triage decision support tools developed for Swedish primary care.

2.1 Formal Definition of Triage

Throughout this thesis, the term triage in its traditional sense will be used in accordance with

the definition published by Iserson & Moskop [6]. This requires three conditions to be satisfied:

1. At least a modest scarcity of healthcare resources exists. In circumstances where resources

always are sufficient to meet the needs of patients with immediate attention, no triage

is needed. At the other extreme, without any available healthcare resources, triage is

irrelevant. In a primary care setting where triage guides patients to either doctors or

nurses, we interpret this condition as focusing on scarcity of the higher level of care, i.e.

doctors.

2. A healthcare worker (often called a “triage officer”) assesses each patient’s medical needs,

usually based on a brief examination. This emphasizes that triage is distinguished as a

process of allocation on a per-unique-individual basis.

3. The triage officer uses an established system or plan, usually based on an algorithm or

a set of criteria, to determine a specific treatment or treatment priority for each patient.

Compared to purely ad hoc or arbitrary decisions, this underlines that triage systems by

definition contain systematics.

In regards to digital triage, no formal definition has been published. We argue that the first

of Iserson & Moskop’s conditions remains unaltered as it acts as a prerequisite rather than a

constraint of what triage is. As for the second criterion, the main aspect of individual assessment

is retained. However, digital triage substitutes the healthcare worker with a digital service that

performs an assessment without involvement from any individual. We will refer to this as digital

assessment. At the same time, this digital assessment depends on the patient’s ability to be

able to communicate through the employed digital channel and the service’s ability to quantify

the patients’ needs. In its most basic form, this assessment could potentially be one single,

fixed questionnaire. In contrast, a digital assessment could also mean customized evaluations

interpreted through text or speech, based on the patients’ symptoms, demographic factors and

previous contacts with health services, as well as digitally monitored vital parameters (e.g. blood

pressure and heart rate). From such a condition, the third of Iserson & Moskop’s criterion is

easily altered to define digital triage such that the triage system, rather than a triage officer,


determines the features of the future treatment. As such, we will assume the following conditions

for usage of the term digital triage throughout this thesis:

1. At least a modest scarcity of healthcare resources exists.

2. A digital service assesses and quantifies each patient’s medical needs, usually based on brief

self-assessed statements by the patient.

3. The digital service uses an established system or plan, usually based on an algorithm or a

set of criteria, to determine a specific treatment or treatment priority for each patient.

It should be emphasized that our use of the term digital triage incorporates that the assessment

and judgment of patients are automated processes, in addition to being digital. This means that

no individual assessment or judgment is performed for each patient by a healthcare worker.

2.2 Previous Research on Triage Models

Existing research on triage is largely focused on emergency healthcare systems and how to ensure

clinical justice for patients at emergency departments [7] [8] [9]. A search on the Web of Science

database on the topic “emergency” AND “triage” AND (”healthcare” OR ”health care”) in the

time span 1975-2019 yields 1,270 results, whereas a search on (”automated” OR “digital”) AND

“triage” AND (”healthcare” OR ”health care”) only yields 57 results. Out of these published

articles not specifically focused on triage in emergency departments, most research focus on

the medical adequacy of digitizing the triage process within specific medical areas, such as

dermatology, oncology or other medical specialities [10] [11] [12].

2.2.1 Triage in Emergency Care

As previously mentioned, the purpose of triage in emergency healthcare is to determine the

priority of treatment for emergency patients based on the severity of their condition. Triage

systems serve as a method for systematic prioritization and are often based on trigger tools

for vital signs, usually including systematic questionnaires adapted for different settings and

conditions. Research on emergency healthcare often refers to three phases of triage [13];

• Pre-hospital triage - Aiming at allocating and dispatching ambulance and pre-hospital care

resources

• Triage at scene - Performed by the first clinician attending the patient, giving a first

in-person judgement of the patient’s condition

• Triage on arrival - Performed as patient arrives to the emergency department or receiving

hospital and aimed at prioritizing the patient’s treatment according to the urgency in need

of care


Research has been performed on all different phases of triage, but most established triage systems

are those developed during the 1990s and 2000s for use in the emergency departments, i.e.

in the triage on arrival phase. The Australasian Triage Scale (ATS), Canadian Triage and

Acuity Scale (CTAS), Manchester Triage System (MTS), Emergency Severity Index (ESI) and

Medical Emergency Triage and Treatment System (METTS) are some of those, which all have

disseminated and been implemented at emergency departments all over the world [14]. In Europe,

MTS is the most commonly used system in emergency departments. In this system, orthopedic

disorders are divided into five groups; traumatic injuries, joint pain, vertebral pain, postoperative

disorder, and musculoskeletal infection. For each group, a flow chart has been developed to

help the triage officer assign the patient one of five different urgency categories, each with a

standardized colour tag and a maximum waiting time ranging from immediate (0 min) to non-

urgent (240 min) [15]. Below, an example of a flow chart in the MTS is illustrated:

Figure 1: MTS flow chart for traumatic injuries [15]

MTS is one of many different triage systems, but the concept of urgency categories based on

a severity criteria is similar for most systems [13]. There are numerous studies evaluating and

comparing the performance of different emergency triage systems, but differences in study design,

study populations, reference standards etc. have shown to have great impact on the results.

Hence, up until today there is no established consensus on the most reliable and efficient triage

system in emergency care [8].


2.2.2 Triage Judgment Frameworks in Swedish Primary Care

Even though there, in terms of evaluating patients’ need of care to allocate healthcare resources,

are some similarities between triage in emergency and primary care, the prerequisites and the

triage systems differ a lot. Unlike in emergency departments, primary care patients are usually

not in urgent need of care. Therefore, evaluation of the severity of the patients’ condition to

prioritize who gets help first plays a vital role in emergency care, while not being as important

in primary care triage. Instead, primary care triage aims at utilizing healthcare resources in the

most efficient way by sending the patient to the lowest (i.e. least costly) level of care at which

they can be efficiently helped [4].

Triage systems in primary care are far less developed than the widely researched emergency

triage systems previously presented. However, some of the Swedish regions have developed their

own triage handbooks, aimed at supporting triage officers in primary healthcare with guidelines

on how to decide needed level of care for primary care patients. Region Skane has developed

”Triagehandboken”, a decision support tool developed and reviewed by specialist representatives

from various medical fields. Just like many of the established emergency triage systems it applies

a symptom based approach, where judgment of vital functions is a central part of the triage

process. For each symptom, a systematic guide has been developed to help the triage officer

gather relevant information, such as pain intensity, duration and other simultaneous factors, e.g.

age or other complicating diagnoses. The answer to those questions helps the triage officer to

decide the most appropriate level of care.

In a similar way, Narhalsan, the public provider of primary care in the Vastra Gotaland region,

has developed ”En handbok for Narhalsans sjukskoterskor”, a triage guide for nurses in Vastra

Gotaland. It has many similarities with ”Triagehandboken”, containing instructions on how to

systematically assess vital functions to increase understanding of the symptoms and

consequently, be able to triage the patient to the most appropriate level of care.

3 EMPIRICAL CONTEXT 11

Section 3: Empirical Context

This chapter provides a description of the Swedish primary healthcare system, including

governance, laws, guiding triage policies and the emerging sector of digital primary healthcare.

3.1 The Swedish Primary Care System

Primary care in Sweden is a part of the open care which serves patients’ day-to-day healthcare

needs. At 55.9 billion SEK, the primary care sector accounts for 10.6% of Sweden’s 526.2

billion SEK annual healthcare bill as of 2018 [16] [17]. In most cases, primary care acts as the

first point-of-contact for patients’ non-emergent healthcare services, coordinating and referring

patients to specialist care when needed. To the Swedish public, healthcare centers serve as their

primary care institutions which employ healthcare professionals in several categories such as

general medicine, psychology and physiotherapy.

3.1.1 The Governance System

The Swedish healthcare system is mainly funded by tax money. As such, healthcare providers

are regulated by law and policy in regards to how to treat patients and how to get reimbursed

for their provided care. The general governance structure of Sweden’s public institutions is

divided into three levels; national, regional and local. On a nation-wide level the Ministry of

Health and Social Affairs oversees all questions regarding social welfare. Covering the full range

of healthcare from primary care to hospital care, the National Board of Health and Welfare

(NBHW) operates under the Ministry on a nation-wide level. NBHW works with establishing

standards, principles and guidelines for Swedish healthcare practices. Furthermore, on a nation-

wide level, the government decides on legislation and the state budget from which money is

being directed to the regions and municipalities in state grants.

Healthcare organized in the 21 regional county councils totals 313.6 billion SEK annually as

of 2018 and in terms of governance of primary care, they are the most important institutions.

The county councils are responsible for delivering and organizing much of the healthcare in each

region and they have authority to individually regulate primary care in its geographical area.

This leads to primary care in Sweden being conducted based on 21 different preconditions. On

the local level, 290 local municipalities have governing power, however their role in terms of

healthcare is marginal and of no essence in terms of primary care. In coordinating the county

councils and the municipalities, all regional and local organizations are members of the Swedish

Association of Local Authorities and Regions (SALAR). SALAR supports the county councils

and municipalities through coordination in e.g. collective agreements or knowledge exchange in

e.g. best-practice guidance.


Returning to nation-wide law, the main governing factor of Swedish primary care on a broad

level is the Health and Medical Services Act. The law stipulates that primary care should

be provided without delimitation in terms of illnesses, age or patient groups. Furthermore,

according to law, the responsibilities of primary care comprise treatment, caregiving, preventive

efforts and rehabilitation where hospital care or other specialized care is not required.

3.1.2 The Act on System of Choice in the Public Sector

According to the Health and Medical Services Act, regions are obliged to organize the primary

care as a system of choice. This means that citizens must be allowed to freely choose between

all available primary care providers and that the county councils must treat all providers within

each of the respective regions equally. As citizens freely choose where to satisfy their healthcare

needs, the public reimbursement from the county council will also follow the citizens and go to

their choice of healthcare provider. One important effect of this system of choice is that privately

operated healthcare providers can compete with publicly operated healthcare providers on the

same conditions according to law. This was enabled in the beginning of 2009 as the Act on

System of Choice in the Public Sector was adopted into law.

Mandated by the Health and Medical Services Act as the regulation to apply in enabling a system

of choice, the Act on System of Choice in the Public Sector dictates the tendering process in

primary care. As such, each region is required to publish a tender document that outlines

the terms of entering into an agreement with the county council which eventually enables any

healthcare provider to be publicly reimbursed. Such a tender document regulates the scope of

the assignment in terms of services, requirements and reimbursement. All healthcare providers

that meet the requirements of the tender document in a specific region have the right to establish

themselves in primary care with public reimbursement and serve the citizens of that region. As

of 2017, 43% of primary healthcare centers in Sweden are privately operated having fulfilled the

respective requirements of the county councils according to the Act on System of Choice in the

Public Sector [18]. In doing so, these healthcare providers take part in fulfilling the regions’

responsibility of providing primary care to the Swedish population. One important factor when

observing the market competition however, is that all healthcare providers that are publicly

reimbursed can not compete through pricing as patients’ co-pay and the reimbursement itself

are standardized within each region.

3.1.3 Public Reimbursement

When outlining the public reimbursement system of primary care in Sweden, the circumstance

of regional governance should be underlined. As a consequence of the regional sectioning, there

are 21 different bases of how healthcare providers are reimbursed wherein no reimbursement

model is the same. However, there are three general themes of how primary care providers are

reimbursed: capitation, fee-for-service and performance-based.


Capitation accounts for the largest share of the total public reimbursement. The exact share

of capitation reimbursement relative to the total reimbursement is in the range 50-100 percent,

but varies between different regions [19]. The fundamental policy of capitation is that each citizen

can choose to enlist with a specific healthcare provider. The caregiver is then responsible to carry

out the enlisted patients’ healthcare needs. For this responsibility, the healthcare providers are

paid a monthly sum for each enlisted patient, regardless of how much care the enlisted patients

seek. This is what’s known as the capitation reimbursement. The actual sum that is paid

out for each patient differs, both between regions and within regions as there are three main

components that are used to calculate the capitation reimbursement for each provider’s patients.

These are 1) Adjusted Clinical Groups (ACG), 2) Care Need Index (CNI) and 3) the belonging

of patients into different age-groups. ACG is a system developed by researchers at the John

Hopkins University in Baltimore, which assigns patients an ACG value based on their previously

recorded medical diagnoses. CNI is an index based on seven socio-economic factors which have

been found to relate to patients’ healthcare needs. Both the adoption of, and eventual weight

assigned to each one of these three components differ between regions.

A reimbursement structure that relies on capitation creates incentives for healthcare providers

to increase cost efficiency, patient safety and preventive care [20]. In terms of triage, this means

that capitation reimbursement also incentivizes that triage should be performed such that the

resulting care provided to the patient is as cost efficient as possible.

Fee-For-Service reimbursement is paid to the healthcare provider for each meeting. As the

name suggests, this component of public reimbursement is paid to healthcare providers for each

completed consultation. In many of the regions, patients themselves bear a part of the cost

for their consultation. If such a co-pay is paid, this is part of fee-per-service reimbursement.

Apart from regional variations, the sum of the fee-for-service reimbursement in many cases

differ between healthcare professions and whether the patient has enlisted at the provider or

not. For example, as of 2019, one third of the county councils do not pay any fee-per-service

reimbursement for enlisted patients [21]. The fee-for-service reimbursement can however also

reduce the public reimbursement to the healthcare providers. In several regions, the county

councils apply a penalty fee for providers whose enlisted patients have consultations with other

caregivers.

The implications of the fee-for-service reimbursement in terms of financial incentives in

conducting triage are not as straight-forward as the ones of the capitation reimbursement.

There are no general conclusions to be drawn in terms of triage. As different regions apply

different reimbursement policies, the most beneficial level of care in terms of gross profits will

differ between regions. More specifically, the determining factor in terms of financially

incentivizing triage to a certain level of care is found in the difference of reimbursement sums

for consultations at different levels of care. Regarding this, there are insights to be found when

comparing the reimbursement policies of different regions. As one example, Region Stockholm

pays a fee-per-service reimbursement for doctor meetings of 260 SEK and for nurse meetings of

230 SEK. This difference of 30 SEK will, at the current market levels, never outweigh the


higher salary cost for doctors. Thus, the reimbursement policy in Region Stockholm

incentivizes triage to nurses, which is in line with the county council’s guidelines in their tender

document where healthcare providers are told to treat patients according to the LEON

principle, which suggests nurses as the primary endpoint of a triage system. In contrast,

Region Ostergotland, instructing providers to employ the MEON principle pays 250 SEK more

for doctor meetings than for nurse meetings, approximately equalling the gross profit of the

two respective meeting types at different levels of care.

Performance-Based is the third major theme that makes up the public reimbursement

system. Relative to the other two major themes, performance-based public reimbursement

generally constitutes a small share of the total reimbursement. The sum of performance-based

reimbursement, just like the two other major themes differs between regions, but can be based

on patient satisfaction, care coordination, continuity, enrollment in national registers or

compliance with evidence-based guidelines to name a few examples. Many regions employ

coverage ratio as one performance measure, aiming at incentivizing healthcare providers to

treat patients in primary care when possible rather than referring them to specialist care.

Two-thirds of regions also reimburse healthcare providers on geographical basis, for example

compensating players in areas of low population density or with long distances to the closest

hospitals. Note that this reimbursement in Region Ostergotland is only paid for consultations

with patients enlisted at another healthcare provider. The fee-per-service reimbursement of

enlisted patients’ consultations is 0 for all levels of care, thus incentivizing nurse meetings

based on gross profit.

3.1.4 Public Reimbursement of Non-Resident Patients

All the above themes of reimbursement however only applies to their full extent when providers

treat patients within the one of the 21 regions that they are established in. Since patients are

still legally allowed to seek care in regions outside their home region, the regions through

SALAR have established a common policy for reimbursement of non-resident patient

consultations. This policy enables healthcare providers to be reimbursed on a fee-for-service

basis for treating non-resident patients. Since digital healthcare meetings do not require

physical presence at a certain location by neither the healthcare professional not the patient,

this is what essentially has enabled digital providers to operate on a national level with public

reimbursement. Through establishing themselves as a healthcare provider in one region

according to the Act on System of Choice in the Public Sector, digital healthcare providers

have been treating patients from outside their established region as non-resident patients. As

such, the digital practice has enabled a national reimbursement model not intentionally created

for this purpose. As of 2020, the public reimbursement for digital meetings with non-resident

patients according to the agreement in SALAR are as follows: [22]

• Consultations with doctors: 500 SEK


• Consultations with psychologists, councelors or psychotherapists: 425 SEK

• Consultations with other care professions: 275 SEK

3.1.5 Triage As a Requirement For Reimbursable Digital Healthcare

In the wake of digital healthcare consultations emerging in Sweden, questions have risen regarding

the ability of digital consultations to substitute physical ones. To ensure that the tax money

spent on publicly reimbursed digital healthcare meetings serve its purpose, SALAR has set up

five criteria which establish the principles for reimbursing digital healthcare. [23] The principles

were then attached to the recommendations sent to the county councils regarding reimbursement

of non-resident patient consultations. The five principles are:

• The meetings must be preceded by ID check through strong authentication.

• The meetings must be preceded by an assessment to exclude symptoms and diagnosis that

should be taken care of by physical care or that no not need medical attention.

• The meeting must constitute of “eligible healthcare” according to the definition by NBHW,

and thus not be a question of advisory services.

• The meeting must fulfill the same requirements on EMR usage and reporting as equivalent

meetings in primary care according to the criteria set out by the county council.

• The digital healthcare provider is liable to pay for and to have routines in place for referring

patients in need of lab tests and other medical services.

In terms of digital triage, the second criteria requires at least some form of triage to occur before

a digital meeting in order to be eligible for reimbursement.

3.2 Guiding Principles in Swedish Healthcare

3.2.1 Prioritization and the Ethical Platform

According to a government bill adopted by the Swedish Parliament in 1997, prioritizing in

Swedish healthcare must be performed based upon an ethical platform of three principles. These

are, in hierarchical order: [24]

1. The Principle of Human Rights, meaning that all human are equal regardless of

personal characteristics or societal function.

2. The Principle of Need and Solidarity, meaning that resources should be distributed

according to needs.


3. The Principle of Cost Efficiency, meaning that when faced with a choice between

different actions, a reasonable relationship between costs and effect should be strived after.

Effects are to be measured by improved health and improved quality of life.

As mentioned previously, patients within digital healthcare tend to have a comparable need of

care, meaning that no prioritization must be done according to needs. As such, prioritization

of healthcare resources, i.e. primary care triage in our setting of digital healthcare, should

be performed whilst striving after a reasonable relationship between costs on one side, with

improved health and improved quality of life on the other side.

3.2.2 Principles of Cost Efficiency

Returning to the Health and Medical Services Act, Swedish law stipulates that publicly

reimbursed healthcare should be organized such that it promotes cost efficiency. The NBHW

also lists efficient healthcare as one of six areas which constitute good-quality health and

medical care in Sweden [25]. Efficient healthcare according to NBHW is defined to describe

that ”available resources are utilized in the best possible way to reach intended targets”. This

means, NBHW continues, that ”healthcare is organized and supplied in cooperation between

the the players in the healthcare system based on the severity of the patient’s illness and the

cost efficiency of the treatment”. Distinguishing efficiency from productivity, the NBHW

underlines that efficiency sets results in relation to costs whilst productivity sets efforts in

relation to costs and thus, is only one of multiple factors that constitute efficiency.

One of the nation-wide organizational measures to promote cost efficiency is the capitation

reimbursement model previously discussed. Another cost efficiency measure which could have

large effects on triage systems is the LEON principle. An acronym for the Swedish translation

of Lowest Efficient Level of Care, the LEON principle could serve as a guiding principle for

what level of care a patient should be referred to. In the 2016 final report of the government

inquiry into efficient care, led by Goran Stiernstedt, the LEON principle is described as what

should be the ”obvious strategy” in healthcare. The report further defines the principle as the

process of ”directing duties to the profession that can perform them for the lowest total cost with

maintained or improved quality”. At the same time, the report also notes that this principle

does not seem to be a given fact in regionally funded healthcare such as primary care. [26]

The lack of adopting the LEON principle in Swedish primary care is reflected by a mapping

of the currently valid tender documents as written by the county councils. Only in Stockholm

are healthcare providers explicitly mandated to treat patients according to the LEON principle.

However, such a mapping also reveals that other principles of cost efficiency have been adopted

within Swedish primary care. The following alternatives of the LEON principles are currently

present in the regions. Note that Varmland only applies BEON as a policy in publicly operated

primary care, meaning that the policy is not present in the tender documents.


Policy... Efficient

Level of Care

# of County

CouncilsCounty Councils

LEON Lowest 1 Stockholm

MEON Most 3 Kronoberg, Vasterbotten, Ostergotland

BEON Best 2 Blekinge, Varmland

NEON Closest 1 Jamtland/Harjedalen

Table 1: Mapping of Principles of Cost Efficiency in Primary Care

The consequences of the different policies are unclear as no tender document presents a clear

definition of their policy. According to the definition of the LEON principle presented in the

government inquiry final report, one could argue that all policies essentially are the same without

any further explanation. There does however seem to be a distinction made amongst healthcare

professionals, especially between the most well-known LEON principle and the MEON principle,

most frequently present in the tender documents from the county councils. For example, in 2019

the founder and CEO of Min Doktor Magnus Nyhlen wrote a debate article on the Swedish

debate platform Dagens Samhalle Debatt, arguing for substituting the LEON principle for the

MEON principle, in opposition to the previously released report by the government inquiry [27].

As there are no commonly adapted definitions of neither the LEON, the MEON, the BEON nor

the NEON principles, it is difficult to draw any further conclusions about their consequences on

triage in primary care. What can be said with certainty is that in this debate, there are two main

schools of thought when applied to primary care. One perspective is held by those who argue

that nurses should serve as the first point of contact [28]. The other side argues that patients

shouldn’t have a designated first point of contact but rather be referred to the most appropriate

level of care immediately when such a judgment can be made ahead of the first meeting with a

healthcare professional [29] [30].

3.3 Digital Primary Care Practice

In June 2016, a report was published by McKinsey which concluded that digitalization of the

Swedish healthcare system potentially could lead to 25% cost savings by 2025 [31].

Approximately 11% of the potential cost savings were attributed to consultations at distance.

The practice of digital primary care has grown rapidly during the second half of the 2010’s.

Although many of the privately operated digital healthcare services were founded some years

before, the starting point for nation-wide expansion was when Region Jonkoping equated

digital healthcare consultations with physical ones in the spring of 2016. The digital healthcare

services could then partner up as subcontractors to healthcare providers in Region Jonkoping

with agreements in place according to the Act on System of Choice in the Public Sector. By

doing so, their digital healthcare services could be used by any Swedish patient according to

the Health and Medical Services Act, allowing all citizens to freely choose between all available


primary care providers [32]. Consultations with patients residing outside Region Jonkoping

would then be reimbursed according to the policy on non-resident patients. During 2018, 4.6

percent of all primary care consultations with doctors in Sweden were performed through

privately operated digital services, with the patient formally treated as a non-resident patient.

3.3.1 Major Players and Triage

As the digital healthcare providers have grown in number of yearly consultations, the diversity

of said providers has also grown. For example, the digital healthcare practice has been adopted

by many county councils in charge of their publicly operated healthcare providers. When the

digital healthcare practice first started, doctors were the only healthcare profession available

through the digital services. The services today involve more professions than just doctors,

such as nurses, psychologists and midwives. As of 2019, the three largest privately operated

digital healthcare providers in terms of number of digital meetings are KRY, Min Doktor and

Doktor.se. There are a few similarities between all three players. First of all, they all operate

using a smartphone application as their primary contact method. Also notably, they all operate

on a nation-wide level through Region Sormland, much due to the fact that the county council

in Sormland enables the digital healthcare providers to offer their publicly reimbursed services

without co-pay for the patients. Furthermore, the three all employ at least both doctors and

nurses, and thus, use some triage system to guide patients to the appropriate level of care.

KRY is currently the largest player in the digital healthcare industry. The majority of the

digital consultations at KRY are conducted through synchronous video meetings between

patients and healthcare professionals. In addition to their digital service, KRY has since

December 2018 established physical presence, enlisting patients in two regions; Region Skane

and Region Sormland. In terms of triage, KRY claims to have developed a digital triage

system in which the assessment is performed on the basis of artificial intelligence [33]. When

initially launching the triage system, the assessment relied on pre-defined rules intended to

imitate the judgment of a nurse, similar to the triage handbooks presented in Section 2.2.2.

Gathering data from their service, KRY today aims to use machine learning to teach the triage

system what level of care that is most appropriate, based on the patients’ previous healthcare

consultations, demographics and recorded outcomes for similar symptoms. With such a triage

system, KRY looks to facilitate processes where patients always are guided to the appropriate

level of care at the first consultation.

Doktor.se is the most newly founded of the three major players, launched 2016. Similarly to

KRY, Doktor.se conducts the majority of their digital healthcare consultations synchronously,

although conducting text-based chat-like consultations and audio-calls without video to a larger

extent than their competitor. Doktor.se operates physical healthcare centers in three regions;

Region Skane, Region Sormland and Region Uppsala. Furthermore, they have physical presence

through a cooperation with the pharmacy Apoteket Kronan, operating a handful of small clinics


at the pharmacies. These clinics are not on their own complete healthcare centers with ability

to enlist patients. Regarding triage, Doktor.se advocates for a system with more traditional

triage performed by a triage officer, i.e. a nurse. As such, nurses are always the initial level of

care when seeking healthcare at Doktor.se and consultations with doctors are only booked by a

nurse. When Region Stockholm in 2019 requested information from the industry regarding how

to act in order to ensure that digital offerings became an integrated part of the healthcare in the

region, Doktor.se was one of many respondents. In the response, Doktor.se suggested that all

triage had to be performed by a nurse and that using doctors as the first level of care should be

made non-applicable for public reimbursement. Furthermore, Doktor.se suggested in the same

document that the region should institute a public reimbursement policy for only the triage, i.e.

guiding patients to the appropriate level of care without treatment.

Min Doktor was initially founded with a focus towards insurance patients as there were no

path towards publicly reimbursed digital healthcare when the company started up. Min Doktor

performs the major part of the digital healthcare consultations asynchronously through text,

but consultations through audio or video also occur. In Region Sormland, Min Doktor has

one physical healthcare center but the physical footprint of Min Doktor is mainly focused to

pharmacies. In 2018, retailer ICA Gruppen bought a 42 percent share of Min Doktor through

their pharmacy subsidiary Apoteket Hjartat. Since then, Min Doktor has operated Apoteket

Hjartat’s local clinics located in proximity to ICA’s grocery stores. Besides light healthcare

services, Min Doktor clinics also perform a number of tests and vaccinations. In guiding patients

to either nurses or doctors, Min Doktor publicly endorse the MEON principle, aiming to help

the patients in short and efficient care chains. Equal to KRY, patients to Min Doktor answer a

set of predetermined questions based on their symptoms before the meeting which is the basis of

the assessment in the triage system. The specifics of the assessment and judgment in the triage

system are not publicly known, however the use of asynchronous meetings certainly allows for a

less automated solution.

3.3.2 Criticism

The digital healthcare practice has been subject to much criticism. A wide range of critical

opinions have been heard in the public forum. The main factors of criticism have regarded the

medical quality of the services, the reimbursement policy used by digital healthcare providers and

that digital healthcare does not follow the principle that healthcare should be given according

to needs [34] [35].

Related to triage, the key policy risk for digital healthcare providers is the consequence of

a triage system where patients are guided to an inappropriately high level of care. In such

cases, questions could be raised whether the digital healthcare providers adhere to the law of

organizing the publicly reimbursed healthcare on principles of cost efficiency. As much of the

criticism towards the digital practice also has revolved around providers draining the healthcare

system of financial resources, a faulty triage system could further fuel such opinions since doctor


meetings have a higher reimbursement sum than nurses at a lower level of care.

4 THEORETICAL BACKGROUND 21

Section 4: Theoretical Background

This chapter presents the theoretical background of the MCDM model. It proceeds with the

theoretical aspects of using simulation as a tool and presents the performance evaluation

framework used to analyze the simulation outcome.

4.1 Multi-Criteria Decision-Making

The intellectual challenge of making decisions in complex environments is as old as mankind

and something that most people encounter, both in work and everyday life. The approaches to

complex decision-making have varied through history and historically, a common approach has

been to seek advice from from oracles, kings or priests. However, simultaneously with the last

century’s development of scientific disciplines, the old methods have been replaced with modern

science and technology. Today, there are several theories such as linear programming, dynamic

programming and inventory optimization amongst others, that have been widely researched and

that all have the common element of acting as a tool in search for optimal solutions, or decisions.

One of those methods, that has captured a lot of attention in recent years, is the Multi-Criteria

Decision-Making (MCDM) model [36].

Multi-Criteria Decision-Making models have during the last decades become a widespread

strategic decision-making tool within several different areas and disciplines. They are used to,

given a set of alternatives and decision criteria, numerically rank and evaluate the different

alternatives to in the end, be able to choose the best one. They are often used in settings with

contradicting criteria, e.g. when balancing decreasing cost and maintaining quality. All

decision-making models including numerical analysis of alternatives involve three steps [36]:

1. Determine the relevant criteria and alternatives.

2. Attach numerical measures to the relative importance of the criteria and to the impacts of

the alternatives on these criteria.

3. Process the numerical values to determine a ranking of each alternative.

Zimmermann [37] divides MCDM into two categories, Multi-Objective Decision-Making

(MODM) and Multi-Attribute Decision-Making (MADM). MODM studies decision problems

with a continuous decision space, where mathematical programming problems is a typical

example. MADM, which is often referred to as MCDM, focus on decision problems with finite,

discrete and predetermined decision alternatives. In the setting of this study where there are

only two decision alternatives, send the patient to a doctor or to a nurse, MODM is not

relevant and will therefore not be further introduced.


4.1.1 MCDM Definition

MCDM problems are easily expressed in matrix format. Assume we have m alternatives Ai

(where i = 1, 2, . . . ,m) and n criteria Cj (where j = 1, 2, . . . , n). Let A be a (m× n) matrix, in

which each element aij represents the performance of alternative Ai with respect to criterion Cj .

Further, assume weights wj (j = 1, 2, . . . , n) representing the relative weight of each criterion.

Then, a typical decision matrix is constructed as follows [37]:

Criterion

CCC1 CCC2 CCC3 . . . CCCn

(w1 w2 w3 . . . wn)

AAA1 a11 a12 a13 . . . a1n

AAA2 a21 a22 a23 . . . a2n...

......

.... . .

...

AAAm am1 am2 am3 . . . amn

Given a decision matrix as per above, the difference between the different MCDM methods is how

to process the numerical values in the matrix to determine the rank of each alternative. Some

examples of widely used methods are the weighted sum model (WSM), weighted product model

(WPM), analytical hierarchy process (AHP) and revised analytical hierarchy process (revised

AHP). The following definitions will assume that all criteria are to be minimized. However, this

could be replaced by a maximization case.

WSM The WSM model is the most commonly used MCDM approach, especially in single

dimensional problems. In this model, the best alternative AAA∗ is the alternative i that minimizes

(given that it is a minimization case) the weighted sum:

AAA∗ = mini

n∑j=1

aijwj , for i = 1, ...,m (1)

A vital assumption for this model is the additive utility assumption, which requires all criteria

to be measured in the same unit. Hence, in multi-dimensional problems with criteria measured

in different units, the WSM model is not applicable.

WPM Unlike the WSM model, WPM eliminates any units of measure and is therefore

sometimes referred to as dimensionless analysis. By using relative values rather than absolute

values, units are eliminated and hence, it is well suited for both single- and multi dimensional


problems. To compare two alternatives AK and AL, the following ratio is calculated [38]:

R(AK/AL) =

n∏j=1

(aKj

aLj

)wj

(2)

Each ratioaKj

aLjis intended to represent the relative advantage in the pairwise comparison between

the two alternatives, with respect to criterion Cj . If the term R(AK/AL) is greater than one,

alternative AL is better than alternative AK (in the minimization case). If the number of

alternatives is greater than two, the ratio for all pairwise combinations of alternatives must be

calculated and the best alternative is then the one which is is better than or at least equal to

all other alternatives. Hence, the number of ratios to be calculated given a decision matrix with

m alternatives is m(m− 1)/2, causing the number of computations to increase rapidly with the

number of alternatives in the model.

AHP In the AHP model, each element a∗ij in the decision matrix is given by:

a∗ij = aij/

m∑k=1

akj , for k = 1, ..., n (3)

where aij and akj are actual values. Hence, the entry a∗ij represents the relative value of

alternative Ai considered in terms of criterion Cj and the sum∑n

i=1 a∗ij is always equal to one.

Just like the WPM, AHP is dimensionless and can be used for both single- and multi

dimensional problems.

Once the elements in the decision matrix has been transformed as per equation 3, the procedure

for determining the best alternative is the same as in the WSM model; the best alternative i is

the one that minimizes (given a minimization case) the weighted sum, i.e. [39]:

AAA∗ = mini

n∑j=1

aijwj , for i = 1, ...,m (4)

Revised AHP A problem with the AHP model is that a ranking inconsistency may occur

when new identical alternatives are introduced in the model, which Belton and Gear [40] illustrate

with an numerical example in their article On a Short-Coming of Saaty’s Method of Analytic

Hierarchies. They claim that the cause of this problem is that the relative values a∗ij for each

criterion Cj sum up to one. Instead, they suggest that the relative values in the decision

matrix should be derived by dividing each absolute value with the largest absolute value for the

corresponding criterion. I.e. that the denominator in equation 3 should be changed to the largest

value akj for k = 1, ...,m, relaxing the constraint that the relative values for each criterion should

sum up to one. This method which, except for the derivation of the elements in the decision

matrix, is identical to the AHP model, has been named the revised AHP model.

Even though the revised AHP model overcame the ranking inconsistency problem illustrated

by Belton and Gear, it has been criticized by Saaty [41], who claims that identical alternatives


should not be considered in the decision process. It has also faced criticism from Triantaphyllou

and Mann [42], who demonstrated that similar inconsistency problems as the one illustrated by

Belton and Gear may occur in both the AHP and revised AHP, even when non identical new

alternatives are introduced to the models. Hence, there still is no established consensus on the

reliability of the AHP and revised AHP models.

4.1.2 Evaluating and Comparing MCDM Methods

As previously mentioned, numerous different numerical MCDM methods have been developed.

However, a paradox occurs when trying to determine the best one - what decision-making method

should be used to choose the best decision-making method? Evangelos demonstrates that it is

impossible to precisely determine the best MCDM method, since for to do so, one needs to use

the best MCDM method [36]. However, Triantaphyllou and Mann [42] propose two evaluative

criteria, which can be used as a framework to compare and evaluate different MCDM methods.

The evaluative criteria are constructed as follows:

• First evaluative criterion - ”An MCDM method that is accurate in multi-dimensional

problems should also be accurate in single-dimensional problems”

• Second evaluative criterion - ”An effective MCDM method should not change the

indication of the best alternative when an alternative (not the best) is replaced by another

worse alternative (given that the relative importance of each decision criterion remains

unchanged)”

Evangelos [36] uses the two evaluative criteria proposed by Triantaphyllou and Mann to evaluate

the four previously introduced MCDM models. In terms of the first criterion, he uses the WSM

model as a benchmark to map the contradiction rate with the other models for different numbers

of criteria and alternatives. For both low and high numbers of alternatives and criteria, the

contradiction rate is lowest for the revised AHP model. However, for low numbers the difference

between the models is small.

In terms of the second criteria, it can easily be shown that the mathematical structure of the

WPM model eliminates the possibility of contradictory results. Between the AHP and revised

AHP model, the revised AHP model showed slightly higher rate of change than the AHP model

for most tested combinations of number of alternatives and criteria.

4.2 Simulation as a Tool

Throughout this section, the presented theory is based on the published work Simulation with

Arina by D. Kelton [43]. Simulation according to Kelton refers to a broad collection of methods

and applications to mimic the behaviour of real systems. Simulation is used as a tool when


experimenting directly with the system can’t or shouldn’t be made. In such situations, one

might build a model which serves as a stand-in to answer questions about the outcomes, should

the experiments have been performed on the actual system. The role of a model upon which

simulations are made is to mimic the behaviour of the actual system and as such, the model

has to be built with enough detail in order to obtain relevant results from the simulations. The

learnings obtained by the model must not significantly be different from what one would have

learned about the system by experimenting with it directly. This is known as the model validity.

However, just like Kelton writes, simulation isn’t quite paradise. Real systems are almost always

affected by uncontrollable and unknown events and as such, many simulation models involve

some form of stochastic component, causing randomness in the output of the model as well.

With such models there is need to run the simulation multiple times to obtain steady results,

but it can be difficult to determine how many times are enough to obtain reliable results.

As simulation models are rarely able to fully capture the complexity of reality, simulation models

are often simplified to only include the key components of the phenomena studied. This yields

a trade-off between complexity and accuracy which, if not carefully considered, can have severe

impact on the reliability of the simulation [44]. A general simulation study would tend to follow

several of the following steps:

1. Understand the system

2. Formulate clear goals

3. Formulate the model representation

4. Translate into modeling software

5. Verify that the simulation model represents the conceptual model faithfully

6. Validate the model

7. Design the experiments

8. Run the experiments

9. Analyze the results and get insights

4.3 Performance Evaluation for Predictive Modeling

In binary classification problems such as the triage judgment problem in this study, the primary

source of performance measurements is the 2x2 contingency table. [45] Figure 2 shows the 2x2

contingency table.


Figure 2: 2x2 contingency table

Adapting a contingency table with appropriate classification enables use of the common metrics

used that can be calculated from the contingency matrix. Some of these metrics are:

True Positive Rate measures the proportion of actual positives that are classified as such.

TP

TP + FN(5)

True Negative Rate measures the proportion of actual negatives that are classified as such.

TN

TN + FP(6)

Positive Predictive Value measures the proportion of classified positives that are actually

indeed positives.TP

TP + FP(7)

Negative Predictive Value measures the proportion of classified negatives that are actually

indeed negatives.TN

TN + FN(8)

Accuracy measures the proportion of correct classifications, both positive and negative, among

the total number of examinations.

TP + TN

TP + TN + FP + FN(9)

5 METHOD 27

Section 5: Method

This chapter presents the method of the study. It accounts for methodological choices and

provides details of processes associated with the interviews and simulation. It is concluded with

a discussion of the quality of the research.

5.1 Research Design

This thesis aims to assess whether a Multi-Criteria Decision-Making (MCDM) model used in

digital triage could increase cost efficiency, improve healthcare productivity and strengthen the

patient experience in Swedish digital primary healthcare. Furthermore, the study looks to find

what factors such a triage model should consider when deciding level of care for patients as

well as evaluate the effects of a systematic error in the digital triage assessment. Due to the

novelty of the field, previous research on automated triage in primary care is scarce and there is no

established research on how to conduct digital triage. Digital triage in primary care has not been

implemented at scale for many years knowledge and data about the implications of digital triage

is therefore believed to be most developed among a few healthcare providers with experience

from using it in their daily business. Hence, it was decided to conduct this research through a

single case study at one of the major Swedish digital healthcare providers. The case study gave

in-depth knowledge of the complexity that the digital healthcare providers face and how they

would view the inter-dependencies of the factors linked to digital triage. It also provided an

opportunity to study digital triage from within a digital healthcare provider and bring clearness

to the gap in research on automated triage for digital primary healthcare providers.

The case study used mixed method research, combining quantitative and qualitative studies.

Mixed method research has historically been widely used in healthcare research and it has

been argued that it can be particularly useful in healthcare research since a broad range of

perspectives is needed to fully capture the complexity of the phenomena studied [46]. As the

field of automated triage in primary healthcare lacks established theory to originate its findings

from, it was decided to split the study into two phases. The first phase was conducted through

a qualitative study with an inductive approach, aimed at determining what set of criteria to

include in a MCDM triage model and in doing so, answering RQ1. This phase consisted of

multiple interviews, where the research findings were let to emerge from the dominant and

significant themes, without the restraints imposed by structured methods [47] [48]. Furthermore,

the interviews were complemented by the study into the empirical context previously presented.

Based on the findings in the first phase of the study, a MCDM triage model was developed. In

phase two, to answer RQ2 and RQ3, this model was quantitatively evaluated by simulation,

testing the hypothesis that an automated MCDM triage model can increase efficiency and

improve productivity as well as the patient experience in primary care. An overview of the

research process in illustrated in Figure 3.

5 METHOD 28

Figure 3: Overview of the applied mixed method research process

5.2 Phase 1: Interviews

As previously mentioned, research on automated triage in digital primary care is scarce and

despite an extensive search process, no previous research on factors to consider in an automated

triage system for primary care was found. Hence, the aim of phase 1 was to qualitatively

determine what factors to include in an MCDM triage model, by conducting interviews at the

partner company and connect this to the study into the empirical context.

5.2.1 Selection of Interviewees

There are different opinions on optimal numbers of interviews in qualitative research. According

to Kvale and Brinkmann [49], the number of interviews is usually in the range 5− 25, whereas

Blomkvist and Hallin [48] recommend to keep it between 10 − 15 interviews. However, both

emphasize that this is dependent on the quality of the interviews and in the end, it is up to

the researchers to decide when the interviews have reached empirical saturation, i.e. when the

interviews no longer yield new, relevant information and insights. Hence, no predetermined

sample size was set before selecting interviewees.

In total, 7 semi-structured interviews were conducted with employees from various positions

at the case company. To get a holistic view of automated triage and to ensure that insights

from all different departments at the case company were captured, four study sample groups

were formed. These four study groups represent the four areas of expertise involved in making

decisions about the triage system in the organisation. An overview of the sample groups and how

their corresponding area of expertise is relevant to triage is presented in Table 2. Within each

group, at least one interviewee with a minimum of one year of experience within digital healthcare

was selected. Further, a snowball sampling technique was applied, where more interviewees were

selected based on referrals from previous interviewees. This technique is efficient when it is hard

to identify the best suited members of the desired population and it also increase chances of

finding candidates with relevant insights and experiences [50]. In Table 3, a complete list of

interviewees and their area of expertise is presented.

5 METHOD 29

Sample Group Group Code Sample Size Relevance in Automated Triage

Policy P 1 Knowledge of laws and regulations

Tech T 3 Implementation and patient experience

Operations O 1 Staffing and clinician experience

Medical M 2 Medical expertise

Table 2: Overview of interview sample groups

Interviewee Code Title at Case Company Interview Length Date

P1 Policy Director 45 min 30.01.2020

T1 Chief Technical Officer 22 min 20.01.2020

T2 Product Manager, Patient Queue 43 min 17.01.2020

T3 Product Manager, Data 51 min 04.02.2020

O1 Head of Digital Operations 26 min 22.04.2020

M1 Medical Doctor 27 min 20.01.2020

M2 Head of Patient Safety 33 min 20.01.2020

Table 3: List of interviewees

The initial plan was to also conduct interviews with representatives from SALAR, but due to

the COVID-19 situation during 2020, SALAR representatives were not able to allocate time

to this research. Those interviews were intended to give a wider perspective, but even though

no SALAR representatives were interviewed, the interviews with case company representatives

reached empirical saturation and contributed with high quality insights. Hence, the impact on

the overall quality of the research is believed to be minor.

5.2.2 Interview Design

Due to the exploratory nature of phase 1, semi-structured interviews were the preferred choice

of data collection. It is a common method for data collection in research of exploratory nature,

since it allows the researcher to be flexible and seek new insights given a specific organization

and context [50]. The themes and questions covered varied from interview to interview and was

adopted to the expertise of each interviewee in order to capture as much as possible of their

professional knowledge [51]. In Table 4, a schedule of the themes covered in the interviews, split

by sample group, is presented. The themes and questions were not necessarily read in verbatim

and order, but rather seen as a guide to ensure that relevant topics were covered [52]. Deviations

from the guide were not seen as a problem, but rather encouraged, since it captured unexpected

but relevant areas that emerged throughout the interviews [53].

5 METHOD 30

Sample Group Theme / Topic

Information about the interviewee

Factors to consider in a digital triage system

Policy risks with a digital triage systemPolicy

Laws and guidelines regulating triage



The trade-off between impact and effort when building a digital triage systemTech

The role of the patient in a digital triage system



Monitoring of KPIs to ensure efficient operationsOperations

Staffing of clinicians depending on the triage system



Considerations for a triage officer in traditional triage

Experiences from the clinicians’ point of view:

Similarities and differences between traditional triage and digital triage

Medical

Patient safety in a digital triage system

Table 4: Interview themes and topics

All interviews were carried out face-to-face in Stockholm. Interviewees working at the case

company were contacted through the internal communication system Slack and briefly introduced

to the research subject before the interview. No regard was put on the order of the interviews,

although the interview with a representative from Operations took place after a 2-month-gap in

interviews. This was mainly due to two reasons. One reason was that the COVID-19 situation

led to the operations department at the partner company being unable to allocate time for this

interview during March. The other reason is that the need of insights from the Operations

department did not prove vital until at a later stage in the research process.

The reasoning behind only conducting face-to-face interviews was that the interpersonal

chemistry is vital to generate interest and face-to-face interviews also enables use of visual aids,

such as reading body language and facial expressions, which gives a more comprehensive

understanding of the interviewee [54]. All interviews gave consent to recording using a mobile

phone and in addition, notes were taken throughout the interview to ensure that all relevant

information was captured. Both researchers participated in all interviews, where one was

assigned responsibility for taking notes and the other was assigned responsibility of conducting

the interview. These responsibilities shifted between interviews.

5 METHOD 31

5.2.3 Data Analysis

In addition to the notes taken throughout the interviews, all interviews were listened through and

transcribed within a week in order to maintain the contextual interpretation. To decrease time in

transcribing the interviews, a method proposed by Saunders [50] was used, where only the parts

perceived as contributing to the research were transcribed. This allowed for a condensation of

meaning where conversations were broken down to a summary of key points emerging from the

interviews. Compressing the transcriptions from long conversation to brief statements allowed

to analyze the data in a more comprehensible way, where patterns, repeated concepts and ideas

were easier to identify.

Analysis of the data collected followed a grounded approach, where inductive reasoning was used

to identify repeated concepts and ideas [55]. After condensing and summarizing the data, all

different thoughts and ideas regarding factors to consider in an automated triage model were

listed. A frequency analysis was performed and pieces of data related to each listed factor was

mapped to identify further relationships and understand the meaningfulness of the collected

data. To evaluate the appropriateness of including a factor to consider, the study into the

empirical context was used to find policies that would support or oppose such a factor when

necessary.

5.3 Phase 2: Simulation

In order to evaluate the effect of a triage system for answering the research question, there

was a need to time-efficiently assess how different triage models would perform in a realistic

setting. Assigning R&D teams to build and implement these policies for actual patients without

knowledge about their impact would introduce a substantial risk in terms of patient safety,

development costs and time [56]. As such, it was chosen to model the patient flow, triage

function and queuing system of a digital healthcare provider for simulation. The simulation

methodology was based on the step-by-step procedure presented in Section 4.2 [43]. The results

are gathered on a day-by-day basis and scoped in accordance with the 5 groups of performance

indicators presented in section 1.2.1.

5.3.1 Simulation Strategy

The simulation model was constructed using the programming language Python (v.3.8.2).

According to the pre-set delimitations of the study, all patients in the simulated flow would

eventually be helped digitally by either a nurse or a doctor. Furthermore, all patients could be

helped by a doctor, however for not all patients help by a nurse would satisfy their needs.

Based on these criteria, a patient flow was designed according to the overview presented in

Figure 4.

5 METHOD 32

Figure 4: Overview of the simulated patient flow

As previously noted, simulation models are often simplified to only include the key components of

the phenomena studied. As such, the model was constructed so that only the judgment module

of the patient flow would be altered when comparing different triage models. Certain factors not

affecting triage in a real setting were also simplified within the other parts of the patient flow.

The set design of the patient flow model is now presented step-by-step in the following section.

5.3.2 Simulation Modules

Patient Inflow / Arrival Time Patients arrive into the system according to a Poisson

process, in which the time between each arriving patient follow a exponential distribution with

parameters such that the expected rate of patients was 3 per minute. The exponential

distribution is often used to model arrival times into a system as independent arrivals with a

constant arrival rate are exponentially distributed. The distribution is characterized by the

single arrival rate parameter. Furthermore, an important attribute of the exponential

distribution is that it is memory-less, meaning that the probability of an event (i.e. patient

arriving) during an upcoming period set of time is the same regardless of when the last event

occured. In other words, the probability of a patient arriving during the next minute after

waiting for one minute without a patient arriving, is the same as the probability was a minute

ago of a patient arriving during the minute that just passed. Formally, this means that the

inter-arrival time X has the property P (X ≤ t+ s|X > s) = P (X ≤ t) [57].

To generate the random arrival times, inter-arrival times were simulated starting at time 0

with the first patient arriving after one inter-arrival time. Each patient after that arrive one

random inter-arrival time after the prior. The last patient is the one to arrive before the

cumulative inter-arrival time exceeds 24 hours. The random variables were generated using the

numpy.random module for random sampling within the numpy package. Notably, the patient

volume is generated during 24 hours. Meanwhile, serving these patients could potentially

exceed 24 hours as the queue of all generated patients is served before the simulation ends.

The patients were organized in a dataframe from the pandas package which store the patient’s

arrival time as well as their simulation-ID which represented the order in which they arrived

5 METHOD 33

into the system, with the first patient getting ID 1. The patients themselves are constructed as

a class object, enabling attributes to be attached to each patient.

Assessment / Nursability The initial part of a triage function is the assessment stage. As

previously noted, the assessment of patients is not the factor to evaluate in this study. As such,

the simulation model did only need to generate the assessment outcome rather than the input

factors going in to the assessment of a patient. The outcome of patient assessment took the form

of a number between 0 and 1, called nursability score, which represented the probability that a

nurse would be able to satisfy the healthcare needs of the patient. This should be viewed as a

representation of an assessment of the patient’s symptoms.

The generated nursability scores in the simulation model were based on historical values at

the partner company presented in Appendix A. At first, empirical mass distribution histograms

were calculated in ranges of 10 percentage points based on the historical values. To generate a

simulated patient’s nursability score, a discrete random variable was created using the rv discrete

class of the scipy.stats module. Generating an observation from this random variable finds the

bracket of 10 percentage points for the patient. The final nursability score assigned to the patient

was then generated by a uniformly distributed random variable on the interval of 10 percentage

points. The nursability score was attached to the patient as an attribute to the patient object.

In the input data, the nursability scores do not exceed 0.9 and is therefore reflected in the

simulation.

Judgment This is the most essential module of the simulation model where patients were sent

for either a nurse consultation or a doctor consultation. The details and set of triage models

evaluated in this module are presented in Section 5.3.5.

Queues / Waiting Times After the triage is completed, patients are placed into one of two

queues to either have their consultation with a doctor or a nurse. The queues are based on

the dataframe from the pandas package which lists each patient’s arrival time, actual meeting

start time and actual meeting length, the last one being simulated in the meeting module of the

model.

Apart from calculating the actual start time of each patient’s meeting, the queue system also

estimate a waiting time for both professions before assigning a patient to the end of any queue.

These estimates may be used as a parameter in the judgment module preceding the queue system.

These estimates were based on the expected value of the meeting length for each profession and

are according to Table 5. By these assumptions, the model calculated what estimated time

remains of the ongoing meetings and what total estimated time the unstarted meetings would

consume.

5 METHOD 34

Description Value

Expected meeting length nurse - patient helped 11.39 min

Expected meeting length nurse - patient not helped 9.87 min

Expected meeting length doctor 12.06 min

Table 5: Expected meeting lengths

As well as estimated waiting times, this module calculates an actual waiting time in order to

correctly set a meeting start time in the dataframe. This is based on simulated meeting lengths

in the meeting module which feed into the queue module. The actual waiting time is set to

end depending on the length of the queue; if there is available staffing capacity at the time of

the patient arriving into the system, the waiting time is 0 and the arrival time to the system

is equal to the meeting start time. On the other hand, if all clinicians are occupied, the actual

meeting start time will be set to when the n:th last patient in the queue is known to end their

meeting where n represents the number of staffed clinicians serving the queue. In reality, this

value is obviously not known when assigning the patient to the queue system. Therefore the

actual waiting time is only used as a tool in the simulation rather than a factor to use when

assigning patients in the triage system.

Staffing Staffing of nurses and doctors is not a module in the flow for each patient. However

when evaluating triage modules which interact with waiting times, the staffing policy of doctors

and nurses play an important role. As different triage models may have different tendencies

to guide patients to doctors and nurses, respectively, staffing would in a realistic setting be

distributed accordingly.

Furthermore, in triage models which take waiting times into account when judging between

sending patients to nurses or doctors, staffing will have an impact on the triage system itself.

With a constantly higher waiting time to one of the professions due to unproportionally

distributed clinician resources, the triage system would be affected by this constant factor

potentially impacting patient safety and cost efficiency. Due to this phenomena, the aim in the

staffing module is to eliminate the impact of staffing on the judgment of the triage system.

Involving waiting times as a parameter to judge the triage upon is done to limit the risk of

experiencing crowding in one of the queues while clinicians serving the other queue are idle. At

the same time, in order to obtain realistic, consistent and comparable results, the staffing must

be set ahead of the patient inflow and be generally unaltered for different simulations of the

same triage model.

One simplification utilized in the simulation model is that the staffing of clinicians during the

24 hours of serving patient is constant. In reality, supply and demand change significantly

throughout the day and night, with clinicians relieving each other on different shifts. Although

clinicians work at different levels of efficiency when helping patients, it is believed that

introducing such randomness into the simulation would introduce increased difficulty in

assessing the impact of the triage models compared to the impact of the clinicians. As such, it

5 METHOD 35

should be noted that the choice to run the simulation for 24 hours means that the patient

inflow represents a constant demand from patients throughout the day and night,

approximately equal to the average over 24 hours.

The staffing volume is set by feeding 10’000 random patients into the triage module with

nursability scores according to the empirical discrete distributions, first assuming an endless

supply of clinicians of both professions. The infinite number of clinicians mean that the waiting

time will constantly be 0 and not a determining factor in the triage of these patients. After

running all 10’000 patients through the triage module, the expected number of meetings to

serve these patients with both professions is determined. These values are scaled to calculate

the expected meeting volume per hour based on the expected patient volume from the arrival

rate in the patient inflow module. With these values, the meeting volume is translated into an

expected staffing need based on the service productivity of the respective clinician profession.

The service productivity represents the expected number of meetings one clinician can serve

per hour based on the respective expected meeting length. These values are finally rounded up

to the nearest integer to determine the staffing of the simulation model.

Choosing 10’000 as the appropriate patient volume in the staffing module was made by gradually

increasing the patient volume before reaching satisfactory convergence. The aim is to have little

to no variance in the staffing volumes between different simulations of days with the same

triage model. The resulting staffing set-up of the module is presented along the findings of the

simulations.

One important take-away from this staffing policy in combination with the patient generator is

that the long-term result is a stable system, where supply meets demand. However, on a single

day of simulation, patient volumes might significantly differ from the expected value, causing

an excess of supply or demand. As the long-term result is a stable system, it is still deemed

appropriate to use the 24 hours of demand (i.e. total generated patient volume) with the 24

hours of supply (i.e. staffed volume) in the performance indicators.

Meetings There are three different types of meeting outcomes present in the patient flow of

the simulation model:

1. Nurse meeting, where the patient is fully helped

2. Nurse meeting, where the patient is not helped and referred to doctor

3. Doctor meeting, where the patient is fully helped

At the meeting stage of the patient flow, meeting lengths are simulated for each of the three types

of meetings. Just like the nursability scores, meeting lengths are generated from empirical mass

distribution histograms of the corresponding meeting type at the case study partner company.

The empirical mass distribution histograms have been calculated in ranges of one minute. For

historical values lower than 1 minute or exceeding 60 minutes, the data was removed as such

5 METHOD 36

meeting lengths most likely are due to technical errors, according to the analytics department at

the partner company. To generate the length of a simulated meeting, a discrete random variable

was created and the outcome simulated to find the corresponding bracket of one minute in the

histogram. The final meeting length assigned to the meeting was then generated by a uniformly

distributed random variable on the interval of one minute. The meeting length was added to

the appropriate queue dataframe.

For nurse meetings, the meeting module does also simulate whether the patient may be helped

by a nurse. This is performed by generating one sample from a Bernoulli distributed random

variable with probability for returning 1 equal to the nursability score. In other words, the

Bernoulli random variable returns the value 1, representing that the patient can be helped

by a nurse, with probability equal to the nursability score according to the definition of the

nursability score. This generated variable is attached as an attribute to the patient object.

When a simulated patient has a nurse meeting, this attribute will decide on the outcome of the

meeting and if the patient is not helped, they will be added to the back of the doctor queue with

arrival time equal to the end time of their nurse meeting. It should be noted that the Bernoulli

variable, indicating whether or not the patient would be helped by a nurse, is not considered in

the triage judgement.

Simulation of one Day This section is intended to describe how the different modules interact

and to give an understanding of the chronological order of the simulation program. The program

starts by creating the full dataframe of incoming patients based on the arrival time logic presented

above. Each patient is created as a patient class object, with nursability and the Bernoulli

generated true nursability (0 or 1) as attributes to the object. This dataframe is then iterated

through in order of arrival time, starting with the first patient arriving. For each iteration,

the expected waiting times to both professions are calculated and the triage model decides on

whether to place the patient into the nurse or the doctor queue dataframe. After deciding on

what queue dataframe to assign the patient into, the actual waiting time and actual meeting

length are calculated. If the patient is deemed to have received help at this meeting, the iteration

continues with the next patient in the inflow dataframe. However if the patient is assigned a

nurse meeting which does not satisfy the needs of the patient, the patient is added back into

the inflow dataframe with a new arrival time equal to the end of the nurse meeting and updated

nursability score set to 0. This will automatically assign the patient to the doctor queue. The

iteration then continues, with the patients in the inflow dataframe being assigned meetings in

order of arrival time. This means that a referral from nurse to doctor does not immediately

place that patient A in the doctor queue since other patients realistically might seek help during

the time that patient A has spent until their nurse meeting ended (waiting time and meeting

length). However patient A will eventually be placed in the doctor queue with arrival time equal

to the end time of the initial nurse meeting once the iterating model reaches that arrival time.

The complete patient flow of simulations and calculations with the corresponding parameters is

visualized in Figure 5.

5 METHOD 37

Figure 5: Illustration of patient flow in simulation model

5.3.3 Data Collection

To be able to model the patient flow and queuing system for a digital healthcare provider,

distributions of the simulation variables in Figure 5 had to be estimated according to the

simulation strategy. For this purpose, historical data from the case company was collected

from a 12-month-period between April 2019 to March 2020. As seasonality patterns affect the

types of symptoms, and thus assessments, made in primary care, it was deemed important to

use data from a complete year of meetings. The raw data was collected through internal

database of historical meeting data at the case study partner company and extracted using

SQL queries. To validate that the correct data was extracted, representatives from the

analytics department at the partner company verified the code. After extracting the raw data

to the online service Google Sheets, it was transformed into empirical mass distribution

histograms as presented in the simulation strategy. The estimates of the lengths of the three

types of meeting were calculated as statistical expected values of these histograms, assuming

the uniform distribution of values within each bracket. The empirical meeting length

distributions are presented in Appendix B.

Apart from the distribution of the simulation variables, additional data for salary costs were

required in the simulation. These were obtained from the business controlling unit of the

operations team within the partner company. After contacting a member of the unit through

the communications platform Slack, the total cost estimates were obtained. The estimated

hourly cost for staffing a doctor was set to 900 SEK and the nurse equivalent to 530 SEK. No

extra pay was included in the model such as incentives for efficiency or compensation for

working unsocial hours.

5.3.4 Obtaining the results

Determining number of simulations to run when simulating a stochastic model is often considered

a complex task. Increasing number of simulations has the purpose of providing more stable

predictions of performance and ensure reliable results. Ritter et al. [58] argue that in models

5 METHOD 38

where heavy computational power is needed and simulations are expensive, deciding how many

simulations to run is complex, since it is desirable to find the minimum number of simulations

that still yields reliable results. However, when the simulation is not expensive and there is

enough computational power, the minimum threshold must not necessarily be found, since the

number of simulations can easily be increased until stable results are observed. In this study,

the computational power needed for the simulation model was manageable and the number of

simulated days could be increased without overloading the simulation program. Hence, it was

decided to apply an iterative approach, where the number of simulated days was increased until

a clear convergence of the results was observed. This pattern was found around 100 days which

was chosen as the number of days to evaluate for each simulation.

Evaluating parameter group M2 - LEON Enactment was performed by use of the contingency

table presented in Section 4.3 with entries into the table based on the first consultation for each

patient. The classification was done to view patients in need of doctor consultations as the true

positive state. By doing so, the presented metrics translate according to the following:

True Positive Rate (TPR): Share of patients in need of doctor consultations that are

directly guided to and helped by doctors.

True Negative Rate (TNR): Share of patients without need of doctor consultations

that are directly guided to and helped by nurses.

Positive Predictive Value (PPV): Share of patients that are directly guided to doctors

who indeed require doctor consultations.

Negative Predictive Value (NPV): Share of patients that are directly guided to nurses

who are helped at the nurse meeting.

Accuracy (ACC): Share of Patients directly guided to the LEON level of care (i.e. nurses

if a nurse is adequate, and doctors of a doctor is needed).

Furthermore, to understand the behavior of the triage models and evaluate doctor utility, a

further two parameters were calculated and collected:

Nurse Triage Rate: TN+FNTN+FN+TP+FP Share of patients initially guided to nurses.

Doctor Resource Management Rate: TP+FNTP+FN+FP Share of doctor consultations that

serve patients with actual need of doctor consultations. Since one important element of

triage according to law is to reserve resources of higher levels of care to patients in need,

this is an important metric to evaluate.

In each of the 100 simulated days, the parameters of Table 6 were collected in accordance with

the set-out groups of performance indicators to answer RQ2, presented in Section 1.2.1:

Cost Efficiency

5 METHOD 39

M1 - Clinician salary costs

M2 - LEON enactment

Healthcare Productivity

M3 - Consultations per patient

M4 - Clinician idle time

Patient Experience

M5 - Patient waiting times

Collected Parameters Group ID Parameter ID

Number of Doctors Staffed M1 P1

Number of Nurses Staffed M1 P2

Final Doctor Meeting End Time M1 P3

Final Nurse Meeting End Time M1 P4

Clinician Salary per Patient Journey (excl. cost for idle) M1 P5

True Positive Rate M2 P6

True Negative Rate M2 P7

Positive Predictive Value M2 P8

Negative Predictive Value M2 P9

Accuracy M2 P10

Nurse Triage Rate M2 P11

Doctor Resource Management Rate M2 P12

Number of Patients M3 P13

Number of Doctor Meetings M3 P14

Number of Nurse Meetings M3 P15

Idle Share of Doctor Time M4 P16

Idle Share of Nurse Time M4 P17

50th Percentile of Waiting Time M5 P18



Table 6: Collected parameters for each simulated day

5.3.5 Triage Models

Simulations were performed according to the presented patient flow and simulation strategy. In

order to answer RQ2, these simulations were performed using different triage models to test their

relative performance compared to each other. Note, once again, that the difference between the

tested triage models lays only in the judgment stage of the triage, not the assessment stage. A

total of 8 different triage models were tested in the simulation:

5 METHOD 40

• 1 Ideal benchmark model

• 2 Traditional triage models with different underlying assumptions

• 5 MCDM models with different relative weights put on the respective criteria

These models are now presented in more detail.

Ideal Triage Model The ideal triage model as defined in this study was simulated such that

all patients were guided directly to the lowest efficient level of care. If a patient could be helped

by a nurse, the patient would be guided to a nurse. On the contrary, if care from a doctor

was required, the patient would be directly guided to a doctor meeting. In other words, this

triage model will always render a 100% value of parameters P6, P7, P8, P9, P10 and P12 in the

study. The purpose of this model was to set a benchmark for how effectively triage potentially

could be executed if it, prior to the first meetings, could be determined with a 100% probability

whether or not a nurse could help the patient. If such a certainty could be established, the most

suitable triage model wouldn’t need to take any other factors into account. Since it today is

not realistically possible for an automated triage system to perform this evaluation with 100%

accuracy, this is not a model applicable in a real setting. However, it is considered an important

benchmark for healthcare efficiency in a world of perfect information.

Model ID Model Name

IDEAL01 Ideal Triage Model

Table 7: The ideal triage models simulated

Traditional Triage Model In the digital setting of this study, the traditional triage model is

represented by always having an initial nurse meeting at first and, if needed, a following meeting

with a doctor. Hence, the traditional triage model was simulated by always placing patients

in the nurse queue and then, depending on the outcome of the Bernoulli simulation (whether

or not the patient could helped by a nurse), patients who were not helped by a nurse were

consequently placed in the doctor queue. In this study, this means that this model always will

render parameter P6 equal to 0, parameters P7, P11 and P12 equal to 100%, parameter P9

equal to parameter P10, and parameter P13 equal to parameter P15. Parameter P8 will not be

applicable for evaluation in the traditional triage model as no patient has their initial meeting

with a doctor.

For the nurse meetings where the patient was helped, the meeting length was simulated

according to the empirical mass distribution presented in Section 5.3.2 similarly to all other

models. However, since no triage assessment nor judgment has preceded the nurse meeting in

this model, the nurse will to some extent also fulfill the function of a triage officer. In the

meetings which make up the empirical mass distributions of meeting lengths, the nurses have

not had this role. Instead, in those meetings nurses have initially attempted to serve the

patients’ needs. The historical meeting lengths used in the other simulation models are based

5 METHOD 41

on meetings where a triage assessment has indicated that a nurse somewhat should be able to

serve the patient’s medical needs. As such, for the meetings with patients which end up in a

referral to subsequent doctor meetings, these historical meeting lengths are a bad

representation of reality in a triage model where nurses perform the assessment. Hence, in

cases where the patient is in need of care from a doctor, it is likely that the nurse meeting

length will be shorter than in the empirical distribution previously presented.

Medhelp is a Swedish medical counseling company, responsible for 1177’s phone triage in Region

Stockholm. Triage officers at Medhelp are nurses, whose responsibility is to give patients medical

counseling and guidance to the appropriate level of care by phone. In an SvD article from 2018,

Charlotte Bjorkman, head of operations at Medhelp, argue that phone triage meetings optimally

are in the range of 5-6 minutes [59]. Hence, to not set nurse meeting lengths unrealistically long,

it was decided to perform simulations of two different traditional triage models where patients

first see nurses. One simulation uses the empirical mass distributions obtained by the partner

company to generate meeting lengths, and one simulation uses a uniform distribution in the

range of 5-6 minutes.

Model ID Model Name Description

TRAD01 Traditional Triage Model 1 Using Empirical Data

TRAD02 Traditional Triage Model 2 Using Medhelp Statement

Table 8: The traditional triage models simulated

MCDM Triage Model The hypothesis to test in order to answer RQ2 is whether an

MCDM triage model could outperform a traditional triage model in terms of the specified

performance indicators in Section 1.2.1. Based on the findings of Phase 1 (see Section 6.1), it

was decided to base the MCDM models on three pillars: operational, financial and medical

aspects. These pillars were translated into waiting time, expected cost of treatment and the

probability of getting helped at the initial meeting, respectively. The decision matrix of the

MCDM triage model was constructed with two alternatives and three criteria. The alternatives

were to either guide the patient to a doctor or to a nurse and the three criteria were, as

previously mentioned, the metric representation of the three pillars. It was decided to use the

WPM algorithm, which is constructed to be well suited for decision problems with only two

alternatives and, as presented in Section 4.1.2, has a mathematical construction which

eliminates possibilities of contradicting results when an alternative (not the best) is replaced

by another worse alternative. The WPM is therefore likely to yield the most consistent results.

Furthermore, the WPM model is dimensionless which is suitable for the simulation which uses

minutes, probability and currency as input data. The decision matrix was then for each patient

constructed as follows:

5 METHOD 42

Criterion

Probability to help Waiting time Expected cost

w1 w2 w3

Nurse a11 a12 a13

Doctor a21 a22 a23

Regardless of maximization or minimization, the same alternative will always be advantageous.

The MCDM model in this study was constructed as a minimization case of nurses to doctors,

however the reciprocal of all ratios creates the corresponding maximization case. The

judgement of the MCDM triage model was then based on the WPM ratio presented in

Equation 10, sending the patient to the doctor queue if the nurse-to-doctor-minimization case

exceeds 1, i.e. R(Anurse/Adoctor) < 1 and otherwise sending the patient to the nurse queue.

R(Anurse/Adoctor) =

3∏j=1

(a1ja2j

)wj

(10)

5.3.6 Entering the Decision Matrix Values of the MCDM Model

To enable the WPM algorithm to return relevant judgments, there are two main factors to

calibrate the model for. First of all, there must be optimization consistency, meaning a consistent

advantageous direction to either maximize or minimize performance in all criteria. Secondly,

there must be comparable possible impact of each criterion, meaning that one criterion should

not be able to guide the outcome of the algorithm unless desired and expressed in the relative

weights of the criteria [42]. Even though it is not required that all criteria ratios map onto

the same value range, large differences in the initial model caused unreliable results. To enable

these factors, several transformations had to be performed in the MCDM triage model. This

section now present how values were calculated before entering the decision matrix and what

transformations were performed to satisfy optimization consistency and comparable possible

impact for each criterion.

1) Medical: Probability of getting helped The medical probability of a patient getting

helped at the initial meeting was represented by the nursability score for nurses, distributed on

the range (0, 1). For doctors, the corresponding value was always set to 1, since the setting of this

study assumes that all patients can be treated digitally by a doctor. Furthermore, as the lower

limit of the nursability score is 0, this criterion may come to have a significantly larger impact

than other ratios. To adhere to the principle of comparable possible impact, these probability

scores were therefore added to the upper limit of the value domain, i.e. 1. The result of this is

that the adjusted doctor value will always be equal to 2 and the adjusted nurse value equal to

the nursability score +1.

Also, compared to the other pillars representing operations and patient experience, this metric

5 METHOD 43

by definition is more advantageous at a higher value. To fulfil the requirements for applying the

WPM model in the minimization case, these probabilities were inverted in the decision matrix,

such that ai1 = 1/ai1 (for i = 1, 2), where ai1 represents the adjusted nurse and doctor values

and ai1 represents the reciprocals used in the decision matrix. One implication of this use of the

nursability score is that the ratio a11/a21 always will be equal to, or greater than one. Recalling

the definition of the WPM model in a minimization case, it is clear that this would imply that

the model, if only considering chance of getting helped, never would favor guiding the patient

to a nurse, since the probability of getting helped by a doctor always is greater than or equal to

the chance of getting helped by a nurse.

2) Operational: Waiting time As presented in Section 5.3.2, estimated meeting times were

calculated based on the expected meeting length for each profession, under the assumption that

all meetings would yield an outcome where the patient is helped. The rationale behind calculating

both actual and expected waiting time was to model a realistic setting of digital healthcare to

the largest extent possible. Hence, using actual waiting times, which in a real setting would

be impossible to know, would have yielded inaccurate results. Therefore, estimated waiting

times was calculated based on the information what would be available for the triage model in

a realistic scenario.

Waiting times without transformations do not adhere to the principle of comparable possible

impact. The value range of the ratio a12/a22 tends to infinity or nothing when one alternative has

values close to 0, causing the determining factor R(Anurse/Adoctor) to be unproportionally and

undesirably impacted. To neutralize the ratio value range of waiting times, a similar approach

as the one used for the nursability scores, was used to match the value ranges of the other

criterion-ratios as well as possible. However, a problem when doing so with expected waiting

times was that unlike the nursability score, the upper limit of the value domain for the expected

waiting time is theoretically far above a realistic value. As mentioned above, it is not required

that all criteria ratios must map onto the exact same value range. As such, it was considered

sufficient to simulate the 100 days of the ideal triage model and use a rounded value to the

closest 5-minute-mark of the largest observed expected waiting time during that simulation (one

outlier was excluded). This maximum value was therefore set to 60 minutes and the decision

matrix was subsequently transformed by letting ai2 = ai2 +60, where ai2 is the expected waiting

time for placing the patient in doctor or nurse queue and ai2 the value of the decision matrix.

3) Financial: Expected cost The expected cost of guiding patients to either nurse or doctor

consultations were calculated as the probability-weighted cost. Since this study assumes that all

patients can be helped by a doctor through digital consultations, the expected cost of the doctor

alternative will remain the same. However triage judgments that end up in initial nurse meetings

will with probability equal to the nursability score have a cost of a nurse consultation which helps

the patient. This situation will also, with probability equal to 1 minus the nursability score, have

the cost of a nurse consultation with subsequent doctor consultation, and the subsequent doctor

5 METHOD 44

consultation itself. These costs are therefore calculated as follows:

E[cnurse] = ρ ∗ ln1 ∗ sn + (1− ρ) ∗ (ld ∗ sd + ln2 ∗ sn) (11)

E[cdoctor] = ld ∗ sd (12)

Where E[cnurse] and E[cdoctor] are the expected costs of initially guiding the patient to a nurse

or doctor, respectively. The rest of the parameter descriptions and values used for calculations

are presented in Table 9.

Variable Description Value

ρ Nursability n/a

sn Salary cost nurse SEK 530/h

sd Salary cost doctor SEK 900/h

ln1 Expected meeting length nurse - patient helped 11.39 min

ln2 Expected meeting length nurse - patient not helped 9.87 min

ld Expected meeting length doctor 12.06 min

Table 9: Parameters used for expected cost calculations

Due to the nature of these equations and the used parameter values, it can be calculated that the

expected cost criterion will view the nurse alternative as the one with higher relative advantage

when the nursability score exceeds 0.52. Furthermore, the maximum relative advantage ratio

of doctors to nurses is approximately 1.48 when the nursability score goes towards 0 (i.e. the

expected cost for triage to nurses is 1.48 times higher than the expected cost for triage to doctors).

On the other end of the range, the maximum relative advantage ratio of nurses to doctors is

approximately 1.80 when the nursability score goes towards 1. This would be reflected as the

reciprocal in the minimization case where nurses are the favored alternative by smaller values,

i.e. 1/1.80 ≈ 0.56.

Given all the above transformations, the final decision matrix was constructed as follows:

Criterion

Chance of getting helped Waiting time Expected cost

w1 w2 w3

Nurse 1/(a11 + 1) a12 + 60 E[cnurse]

Doctor 1/2 a22 + 60 E[cdoctor]

5.3.7 Determining the Relative Weights of the MCDM Model

The final step of the MCDM triage model was to determine the relative weights w1, w2 and

w3 in the decision matrix. Attempts have been made to analytically derive optimal relative

weights using linear programming and optimization approaches. However, those methods quickly

reaches high complexity and requires heavy computational power. Hence, a common approach

5 METHOD 45

is to apply subjective weight estimation methods [60] [61] [62]. Given the COVID-19 situation

and heavy pressure on the entire healthcare sector during the process of this study, it was

judged inappropriate to reach out to healthcare professionals for interviews regarding the relative

rankings of criteria in the MCDM triage model. Instead, it was decided to simulate different

combinations of w1, w2 and w3 to, rather than qualitatively determine the optimal combination,

evaluate how different weight combinations affect the outcome of the model. In Table 5.3.7, the

relative weight combinations simulated are presented.

Model ID Model Name w1 w2 w3

MCDM-EQ MCDM, Equal Weights 1/3 1/3 1/3

MCDM-50WT MCDM, 50% WT 1/4 1/2 1/4

MCDM-50CO MCDM, 50% Cost 1/4 1/4 1/2

MCDM-50ME MCDM, 50% Medical 1/2 1/4 1/4

MCDM-100CO MCDM, 100%100 Cost 0 0 1

Table 10: The MCDM models simulated with their respective relative weight combinations

The rationale of choosing the above presented combinations of relative weights were to first

set a base case with equal weight on all three criteria and then see how the simulation outcome

changes as more weight is put on each individual criterion. Further, it was considered interesting

to also evaluate a model only considering expected cost. Note that a model only considering the

chance of getting helped would, as described in Section 5.3.5, triage all patients to the doctor

queue and a model only considering waiting time would not neither be a realistic solution nor

be within the boundaries of existing triage policies and laws. Hence, those models were not

considered viable as triage models and consequently not simulated as a part of this study.

5.3.8 Effect of Systematic Errors in the Digital Triage Assessment

In order to answer RQ3, the objective instead turned to investigate the impact of imperfect

information in the MCDM model. Without question, the judgment of the MCDM model rely

significantly on the nursability score as an input. However, throughout the method presented

thus far, the simulation has interpreted the nursability score assessment as 100% accurate. The

true nursability score of any patient in the study has always been either 0 or 1. A completely

accurate nursability score assessment indicates that, for all equivalent patients who display the

exact similar symptoms, the share of patients whose true nursability score is 1 is equal to the

nursability score assessment.

This might however not be attainable in a realistic setting. Every patient is unique which

undoubtedly introduces a factor of randomness into the system with non-quantifiable

magnitude. RQ3 was therefore answered by investigating scenarios in which the nursability

assessment overestimates or underestimates the appropriate nursability score.

This was performed by further simulations of the MCDM model with equal relative weights

5 METHOD 46

between the three pillars, i.e. w1 = w2 = w3 = 1/3. Four simulations were performed with

introduced factors of uncertainty at different magnitudes; two simulations of underestimating the

nursability score and two simulations of overestimating the nursability score. Whether a patient

could be helped by a nurse or nor was still generated from the empirical mass distributions. In

this way, the patient volume still represented the same symptoms as in the simulations without

systematic errors. After generating the truth about a patient’s ability to be helped by a nurse,

the nursability score attributed to the patient, and thus known to the triage system, was altered.

For a systematic underestimation of up to x percentage points in magnitude, the new nursability

score attributed to the patient was generated by a uniformly distributed random variable on the

interval (a11 − x, a11). a11 represents the nursability score generated from the empirical mass

distributions and used to generate the boolean describing if the patient may be helped by a

nurse. The overestimation case instead generated the attributed nursability score from the

interval (a11, a11 + x). The following four systematic errors were simulated:

Model ID Model Name Description

MCDM-ERR10O MCDM, 10 Over Overestimation of up to 10 percentage points

MCDM-ERR10U MCDM, 10 Under Underestimation of up to 10 percentage points

MCDM-ERR40O MCDM, 40 Over Overestimation of up to 40 percentage points

MCDM-ERR40U MCDM, 40 Under Underestimation of up to 40 percentage points

Table 11: The systematic errors simulated

If the uniformly distributed generated true assessment was set to be outside the interval (0, 1),

it was adjusted to the value of the closest endpoint of the interval.

5.4 Evaluation of Research Method Quality

Leung [63] argues that there is no consensus as to how to assess the quality of a piece of

qualitative research. He notes that a common set of pillars to use when evaluating the quality of

both quantitative and qualitative research is to partition the overall research quality into three

assessments, addressing validity, reliability and generalizability separately. In purely qualitative

research, there are two main schools of thought as to how research quality should be assessed.

Dixon-Woods et al. [64] focuses on methodology whilst the school of Lincoln et al. [65] attributes

the quality of the research to the rigor of interpretation of results. It is believed that the

evaluation of the interpretation of the results are best left to the reader. At the same time,

a growing number of authors argue that mixed method research quality should be assessed

according to its own set of criteria, as stated by Fabregues et al. [66]. In a systematic literature

review, Fabregues et al. studied 64 publications on mixed method research quality and organized

a table of the most commonly suggested criteria for evaluating research quality in mixed method

research. The table was split up in four study phases according to O’Cathain [67] out of which

the seven criteria within the undertaking of the study has been used in this evaluation of the

research quality of this study. The evaluation has been performed both through the framework

of the common set of pillars as well as through the framework of the seven criteria discovered in

5 METHOD 47

the systematic literature review of Fabregues et al.

5.4.1 Validity, Reliability and Generalizability

Validity When evaluating the validity of the research method, the discussion must address

the appropriateness of the method in order to answer the formulated research questions. In

addition, the discussion might also address the appropriateness of the research questions to

reach the desired outcome. In terms of RQ1, the study has gathered information from both

an outside and an inside perspective on the profession, in terms of reviewing policy, laws and

previous research as well as conducting interviews. This is deemed to strengthen the validity of

the study. The interview phase, however, was conducted in a setting where both interviewers

(i.e. the authors of this study) as well as interviewees were employed by the case study partner

company. This should have contributed to a lower degree of validity of the answers to RQ1. In

a stakeholder mapping, the payer role of the county councils was identified as an essential factor

to take into consideration while answering RQ1 from an objective perspective. Unfortunately,

representatives from SALAR were, as previously mentioned, not available for interviews per

mail, phone nor face-to-face.

To answer RQ2 and RQ3, the method of employing a simulation undoubtedly puts the validity

of the results in question, compared to a method of employing the triage policies in reality.

Simulations will never be able to capture the complete scenario of a real setting. However

with regards to the nature of the subject and the time constraint present of a Master Thesis,

the simulation approach is most likely the optimal solution to represent the results in a real

setting. There were simplifications applied in the simulation strategy which also may affect the

validity of the research, since it could be argued that they take the simulation setting further

from the real setting. For example, human error was completely eliminated from the simulation

setting, assuming that all patients show up to their consultations regardless of waiting times

and that all clinicians were able to treat the patients according to their profession. However

these simplifications were made such that the effect on the studied parameters was believed to

be minimal in the relevant comparisons to answer the research questions.

Reliability The reliability in terms of consistency and replicability do not present any

weaknesses in the quantitative phase. As the simulation method employs a programming

approach, full consistency and replicability are ensured. As such, the overall reliability of the

method is believed to be strong. The qualitative phase of the method is also replicable,

however due to the nature of interviews, not necessarily consistent to return exactly equal

results if repeated. However the method of data analysis ensures at least results are able to be

compared with results from other interviews.

Generalizability The generalizability of a case study is generally low. As stated by Saunders

et al., case studies within management tend to have a high internal validity as discussed above.

5 METHOD 48

At the same time is the generalizability, or external validity, not always fundamental. The

generalizability is indeed low for setting industry standards based on the results of the study

since the results of RQ2 and RQ3 in large depend on the empirical data sets obtained from a

single player. However the obtained datasets themselves are based on a significant number of

patients and consultations, indicating a somewhat good level of generalizability for at least the

objectively measurable input data. However the largest obstacle to widespread generalizability

of the results are the nursability scores. Both the patients and symptoms contributing to the

obtained scores are unique to a single provider, and most significantly, how the nursability scores

have been generated based on the patients and their symptoms.

In terms of the generalizability of the method and its enabled results to answer RQ1, the gen-

eralizability is deemed to be limited to competitors of the same market position. As the previ-

ously mentioned bias present during the interviews is suggested to have a significant impact on

the outcome, it is not believed to render generalizable results for all stakeholders in the indus-

try.

5.4.2 7 Criteria of Mixed Method Research Undertaking Quality

1) Quantitative and qualitative components are well implemented and adhere to the

quality criteria of each tradition This criterion is discussed in detail above, partitioning

the quality criteria of both the qualitative and quantitative components into validity, reliability

and generalizability. In short, the quality of the method is assessed to be good. However, due

to the nature of a case study as well as that the interviews for answering RQ1 were conducted

with and by employees of the case study partner company, the generalizability of the method

and its rendered results low.

2) Quantitative and qualitative components of the study are effectively integrated

This criterion is deemed to be fulfilled as the method effectively uses the outcome of the quali-

tative component as a guiding factor for the input parameters of the quantitative component.

3) The mixed methods design is clearly described in terms of purpose, phasing,

priority, and process of integration of the quantitative and qualitative components

It is left to the reader to judge the level of clarity of the method design. The method design of two

separate, succeeding phases was nonetheless deemed necessary to fulfil the purpose of answering

all three research questions. One of many purposes to why the study has been performed was

that the published research into digital triage was scarce, which forced an initial exploratory

research question RQ1 to be answered to appropriately guide the study when answering RQ2

and RQ3.

5 METHOD 49

4) Sampling, data collection and data analysis procedures referring to both

quantitative and qualitative components are linked to the study aims and research

questions This criterion is deemed to be fulfilled as all data collection and data analysis is

performed according to pre-determined purposes originating in the research questions. The

research questions RQ2 and RQ3 themselves stipulate that evaluation should be performed in

terms of cost efficiency, healthcare productivity and patient experience. These three evaluation

criteria are divided into five groups of performance indicator groups which, in turn, are divided

into 20 performance indicators intended to provide a complete and nuanced rationale for

evaluation.

5) Sampling, data collection and data analysis procedures referring to both quanti-

tative and qualitative components are described in sufficient detail It is once again

left to the reader to judge the level of sufficiency of the details of the method design. The de-

tails are attempted to be clarified by clearly linking the evaluation criteria of the research ques-

tions RQ2 and RQ3 to the groups of performance indicators, and in turn the actual metrics col-

lected during the simulations.

6) The mixed methods design is linked to the study aims and research questions

This criterion is deemed to be fulfilled. The research questions themselves encourage a mixed

methods design as the answer to RQ1 comes from an exploratory, qualitative investigation whilst

the answers to RQ2 and RQ3 require quantitative metrics.

7) The mixed methods design matches the rationale given for combining quantitative

and qualitative components It is once again left to the reader to judge the rationale given

of the combining of the components. It is nonetheless unequivocal that both quantitative and

qualitative components have been used in the research method to answer all three research

questions.

6 EMPIRICAL FINDINGS 50

Section 6: Empirical Findings

This chapter presents the findings of the study. It is structured in chronological order, starting

with a presentation of the interview findings and then proceeding with the simulation results.

The simulation results are structured according to the performance measurements presented in

section 1.2.1.

6.1 Phase 1 - Qualitative Interviews (RQ1)

An initial observation from the interviews is that there are no clear-cut national nor regional

laws or guidelines for how triage is supposed to be conducted in primary care. The ones that

exist are hard to precisely interpret, even for healthcare professionals with significant experience

from working in primary care. Several interviewees underlined that often in traditional primary

care triage, it is the triage officer’s responsibility to, based on personal experience and his or

her individual assessment of the patient, decide what level of care is needed. This in fact, in its

own contradicts the very definition of triage as set forth by Iserson & Moskop [6], requiring an

established system or plan to decide on where to guide the patient. As such, this complicates

developing an automated triage system, which in line with Iserson & Moskop indeed is completely

based on a formal set of rules upon which the final judgment is derived.

The common factor that all interviewees mentioned as the most important factor to consider

in an automated triage system is the medical assessment. The assessment made based on the

patient’s symptoms and medical history must be the central component of triage, regardless

whether the process is automated or performed by a triage officer. How the assessment would be

used to guide the triage judgment was focused on two variables. Most notably, every interviewee

mentioned that the assessment of the patient must be used to guide the patient to a level of

care where they are likely to be helped. Some interviewees also saw the assessment as an input

variable to estimate the expected cost of treatment for different triage alternatives. Regarding

further factors to consider, waiting time was the most occurring one. A full list of input factors

to the automated triage system mentioned throughout the interviews is presented in Table 12.

Interviewee

P1 T1 T2 T3 O1 M1 M2

Probability to help X X X X X X X

Waiting time X X X X X X

Expected cost of treatment X X X X X

The patient’s preference X

Table 12: Table of factors mentioned in interviews


6.1.1 Input Factors Included in Phase 2 Simulations

Probability to Help As mentioned, all interviewees emphasized that the single most

important factor to consider in any triage system is the medical aspect. However, the different

guiding principles does not address exactly how to interpret ”most”, ”lowest” ”best” or

”closest” efficient level of care when evaluating the medical assessment and the financial aspect

together. As an example, M2 conceptualized the potential scenario of a patient arriving and

presenting complaints which the triage officer, or an automated triage system, estimate have a

50% chance of getting helped by a nurse and almost surely would get helped by a doctor. In

this scenario, the nurse indeed is the lowest available level of care, but is a 50% chance of being

helped enough to consider the nurse as an efficient level of care, and how big chance of getting

helped by a nurse is required to be considered an efficient level of care?

P1 also mentioned this shortcoming of the existing triage policies. The medical perspective must

indeed always have the highest priority, but in the type of scenario conceptualized by M2, P1

confirmed that the official triage guidelines are insufficient to give a binary answer on what level

of care the patent should be referred to. All interviewees agreed on that in the cases where the

assessment indicates that the patient surely would get helped by a nurse, all different policies

are clear with the fact that then the patient should see a nurse. However, as soon as there is a

risk that the patient requires care from a doctor, it is hard to interpret what level of care the

patient should be referred to given the existing triage policies.

Waiting Time Even though constant efforts are made in estimating expected patient volumes

and what those translate to in terms of required staffing, there is always a margin of error in

those estimates, imposing a risk of under staffing. Further, due to the scarcity of health care

personnel in primary care, health care providers are limited in capacity and can not always

staff enough personnel to meet the expected expected volumes. Hence, for an automated triage

system not considering waiting times, there is a great risk of imbalances between waiting times

for those in need of care from a doctor and those who can be helped by a nurse.

In accordance with the primary care policies aimed at utilizing health care resources in the most

efficient way, several interviewees argue that waiting times should be considered in a triage model

to ensure that doctors are not waiting unused when they potentially could have helped a patient

waiting in the nurse queue.

Expected Cost of Treatment Several interviewees mentioned that the aim of triage in

primary care is to utilize healthcare resources in the most efficient way, which could be interpreted

as aiming at minimizing the cost per treated patient. The salary costs for doctors and nurses are

known, and from experience, or historical data, decent estimations on expected length of doctor

or nurse meetings can be made. Further, the triage officer, or an automated triage system,

can give a numerical representation of the medical assessment, i.e. the probability that a nurse

can help the patient without having to see a doctor. Interviewees from all different sample


groups mentioned that with this data, there is a potential to calculate an expected salary cost

of treatment after the patient assessment and hence, include this is a factor in the triage model.

Many interviewees agreed on that this indeed is a way of utilizing healthcare resources in the

most efficient way and that this probably is the best possible way to formalize the high level

guidelines provided by existing triage policies.

6.1.2 Input Factors Not Included in Phase 2 Simulations

The Patient’s Preference T1 mentioned that in terms of improving patient satisfaction in

primary care, taking the patient’s own preference into consideration in the triage process could

be an option. In addition to that, working in a digital setting, M1 drew attention to one of the

problems of digital assessments based on multiple-choice questions is that they do not capture

doubt. In other words, the patient’s symptoms interpreted by the digital triage system will,

without human involvement, be unable to interpret uncertainty. By allowing the patient to

choose between the available levels of care in questionable cases, the digital triage system could

then rely to a larger extent on the patient’s self-assessment to make the final judgment on the

appropriate level of care. At the same time, this introduces the risk of having patients without

need of doctor consultations using such healthcare resources in contrast to the principles of cost

efficiency.

It was elected to not include the patient’s preference into the Phase 2 simulations, mainly due

to the fact that such a policy has never been adopted before and quantitative data about the

patients’ preferences is therefore not available. There are however at the same time questions to

be asked about the appropriateness of such a policy and whether it would violate the Principle

of Need and Solidarity of the ethical platform. M2 stated that since the official triage laws and

guidelines are few in primary care, it is very much up to the healthcare provider to guide the

patient based on their own professional assessment and judgment. The patient does not have a

formal right to have his or her preferences accounted for.

Reimbursement / Gross Profit In some of the interviews, the discussion was raised

regarding reimbursement as a criteria in a digital triage model and whether it is appropriate or

not. Nurse meetings and doctor meetings may differ in terms of reimbursement, depending on

region and whether or not it is a non-resident patient or an enlisted patient. For a privately

operated healthcare provider, public reimbursement is usually the main source of income and

optimizing triage to maximize the reimbursement or gross profit might seem tempting. A

triage model which adjusts to maximize the public reimbursement or gross profit is however

extremely controversial, and could lead to intense scrutiny and negative opinion from the press

and public. As the discussion is highly sensitive and not a necessity to include for answering

the research questions, it is chosen to refrain from including reimbursement as a factor to

consider. This will be further discussed in Section 7.1


Co-Pay Just like with reimbursement, discussions were raised during some interviews

regarding the appropriateness of including co-pay as a factor to guide triage. The question

whether to include the factor or not is quite similar to the question regarding patient’s

preferences as the factor on its own only affects the patient experience. There are however

certainly no policies which encourage triage based on co-pay as a judgment factor. Also, as few

healthcare providers are present on a nation-wide scale in Sweden and the co-pay is dependant

on the regional policy, the factor is not deemed as an appropriate factor for the phase 2

simulations.

6.2 Phase 2 - Simulations Comparing Triage Models (RQ2)

In comparing the 8 different triage models, they were each simulated over a period of 100 days,

with an average of 4, 317 patients per day. This patient volume was just below the expected

value of 4, 320 patient per day, which is derived from the arrival rate of 3 patients per minute. In

total, the combined volume for these 8 triage models was 3, 453, 271 patients. In this section, the

simulation results are presented according to the performance indicators introduced in Section

5.3.4, divided into the five groups of performance indicators. Throughout this section, boxplots

will be used to visualize the results of the simulations. The boxplots are constructed such that

five horizontal values represent the median value, the upper and lower quartiles as well as the

maximum and minimum values obtained. The extreme values depicted by a horizontal line are

at most 150% of the interquartile range away from the box. Values exceeding this threshold will

be visualized as circles representing the outliers.

6.2.1 Overview of Simulation Results

In Table 13, an overview of the simulation results for the different triage models is presented.

Detailed remarks regarding each parameter will be supplied under the section for each of the

performance indicator groups.

Model IDP5 Salary

per Patient

P10

ACC

P12

DRMR

P14+P15P13

Meet. per Pat.

P20 p100

Wait. Time

IDEAL01 SEK 161 100% 100% 1.000 36.5 min

TRAD01 SEK 223 32.4% 100% 1.676 25.0 min

TRAD02 SEK 195 32.3% 100% 1.677 29.4 min

MCDM-EQ SEK 181 76.5% 76.2% 1.024 32.9 min

MCDM-50WT SEK 181 76.2% 76.0% 1.024 26.2 min

MCDM-50CO SEK 180 78.3% 79.4% 1.041 26.2 min

MCDM-50ME SEK 182 75.9% 75.3% 1.018 28.1 min

MCDM-100CO SEK 179 79.4% 82.5% 1.062 36.8 min

Table 13: RQ2: Summary of average daily results for the evaluated triage models


6.2.2 Cost Efficiency - M1 Clinician Salary Costs

P1 Doctors Staffed and P2 Nurses Staffed were distributed according to the following:

• IDEAL01: (P1=26, P2=12) in 100% of days

• TRAD01: (P1=26, P2=33) in 100% of days

• TRAD02: (P1=26, P2=23) in 100% of days

• MCDM-EQ: (P1=34, P2=5) in 100% of days

• MCDM-50WT: (P1=34, P2=5) in 100% of days

• MCDM-50CO: (P1=33, P2=7) in 84% of days, (P1=32, P2=7) in 16% of days

• MCDM-50ME: (P1=34, P2=5) in 67% of days, (P1=35, P2=4) in 20% of days, (P1=35,

P2=5) in 12% of days, (P1=34, P2=4) in 1% of days

• MCDM-100CO: (P1=31, P2=9) in 66% of days, (P1=32, P2=9) in 32% of days, (P1=32,

P2=8) in 2% of days

These distributions are illustrated in the following Figure 6, where the bubble sizes reflect the

occurrence of each pairing of P1, P2.

Figure 6: RQ2: Staffing distribution per model ID


Using the clinician salary costs from Table 9 used in calculating the MCDM expected cost values,

these staffing numbers return the following hourly staffing costs for each model in SEK.

Figure 7: RQ2: Boxplot of hourly staffing cost in SEK

P3 Final Doctor Meeting End Time and P4 Final Nurse Meeting End Time are used primarily to

verify that the staffing model does not cause the length of the simulated day to unproportionally

exceed the 24-hour-mark. As patients may enter the queue all the way up to the end of the

simulated day, their meetings may end afterwards. The logic behind this method is discussed in

Section 5.3.2.

Figure 8: RQ2: Boxplot of P3 Final Doctor Meeting End Time


Figure 9: RQ2: Boxplot of P4 Final Nurse Meeting End Time

Seen in the P3 boxplot, the three models with the highest tendency to triage patients to nurses

return the highest values for the time at which the final doctor meeting ends. This is expected

as these patients most have a high probability of coming to the doctor consultations after an

initial nurse meeting. The difference between models TRAD01 and TRAD02 is also visible

here, as the initial nurse meetings are shorter in the TRAD02 model enabling the final doctor

consultations to start, and thus end, earlier. Other than that, the distributions are quite similar

among the models and no model experiences significantly higher values which would impact the

cost assessment of the models.

For the P4 boxplot, the distributions are also similar among the models and and no model

experiences significantly higher values. Notably, the models that account for differences in

waiting times have the lowest recorded maximum values for both P3 and P4.

In the P5 parameter, the MCDM triage models clearly outperformed the traditional triage

model, with the median salary costs per patient values being up to 9% lower in the MCDM-

100CO model than in the TRAD02 model. Studying the cost per patient on a daily basis, the

non-binary nursability score distribution and varying meeting lengths clearly causes fluctuations

in daily average cost per patient. However, as illustrated in Figure 10, even the highest daily

average cost per patient for the MCDM models was lower than the corresponding lowest daily

cost for the traditional triage models.

The differences in cost per patient between the five evaluated MCDM triage models is minor,

regardless of the relative weight put on the expected cost of treatment. Further, Figure 10

shows that the value ranges seem to be be approximately equally distributed, indicating that the

randomness of the model has the same impact on cost variations for all triage models simulated.

Despite the increased cost efficiency of the MCDM triage models, there is a great future potential


of further cost reductions since the ideal model has > 10% lower cost per patient than the best

performing MCDM model.

Figure 10: RQ2: Boxplot of P5 Clinician Salary per Patient Journey (excl. cost for idle)

6.2.3 Cost Efficiency - M2 LEON Enactment

By construction, the ideal triage model outperforms all other models in terms of LEON

enactment, with 100% fulfillment of TPR, TNR, PPV, NPV, ACC and DRMR. An accuracy of

100% implies that every meeting conducted has the outcome that the patient is helped and

that every patient whose medical needs can be served by a nurse, is served by a nurse. This is

closely related to the performance of the P5 Clinician Salary per Patient parameter value,

which as presented in the previous section is more than 10% lower than for any other simulated

triage model.

The traditional triage models have a TNR of 100% and TPR of 0%, which is a consequence

of always conducting an initial nurse meeting. Further, since all patients with medical needs

that can be served by a nurse are helped in the initial nurse meeting, the DRMR is 100%. This

implies that doctor resources are only allocated to patients in absolute need of doctor care.

However, this comes at a cost since the average ACC is just above 32%, meaning that almost

68% of patients are not helped in the initial meeting and therefore require a second meeting.

This, despite optimal allocation of doctor resources, causes the increased average salary cost per

patient.

The MCDM triage models have significantly higher average ACC values than the traditional

models, ranging from 75.9% for the MCDM-50ME model to 79.4% for the MCDM-100CO model.

This is driven by a high average TPR, i.e. a large share of patients in need of doctor care being

directly guided to doctors. The average TNR differs between the different MCDM models,


ranging from the MCDM-50WT model with 33.9% to the MCDM-100CO model with 55.4%.

Compared to the TRAD models, the low TNR levels in MCDM models indicate that a substantial

part (1-TNR) of the patients not in absolute need of doctor care are guided to doctors. This

is also reflected in the average DRMR, ranging from 76.0% to 82.5% for the simulated MCDM

models.

The average values and boxplots are as follows. In the cases where values by construction are

0% or 100%, the models are not included in the boxplots. For parameters P9 NPV and P10

ACC the values for traditional models and MCDM models are presented in separate boxplots,

due to the large difference in magnitudes.

Model IDP6

TPR

P7

TNR

P8

PPV

P9

NPV

P10

ACC

P11

NTR

P12

DRMR

IDEAL01 100.0% 100.0% 100.0% 100.0% 100.0% 32.3% 100.0%

TRAD01 0.0% 100.0% n/a 32.4% 32.4% 100.0% 100.0%

TRAD02 0.0% 100.0% n/a 32.3% 32.3% 100.0% 100.0%

MCDM-EQ 96.5% 34.6% 75.6% 82.5% 76.5% 13.5% 76.2%

MCDM-50WT 96.5% 33.9% 75.4% 82.1% 76.2% 13.4% 76.0%

MCDM-50CO 94.0% 45.3% 78.3% 78.2% 78.3% 18.6% 79.4%

MCDM-50ME 97.3% 30.9% 74.7% 84.4% 75.9% 11.8% 75.3%

MCDM-100CO 90.8% 55.4% 81.0% 74.1% 79.4% 24.1% 82.5%

Table 14: RQ2: Average values of M2 LEON enactment parameters

Figure 11: RQ2: Boxplot of P6 True Positive Rate


Figure 12: RQ2: Boxplot of P7 True Negative Rate

Figure 13: RQ2: Boxplot of P8 Positive Predictive Value

Figure 14: RQ2: Boxplot of P9 Negative Predictive Value for MCDM models


Figure 15: RQ2: Boxplot of P9 Negative Predictive Value for traditional models

Figure 16: RQ2: Boxplot of P10 Accuracy for MCDM models

Figure 17: RQ2: Boxplot of P10 Accuracy for traditional models


Figure 18: RQ2: Boxplot of P11 Nurse Triage Rate

Figure 19: RQ2: Boxplot of P12 Doctor Resource Management Rate

6.2.4 Healthcare Productivity - M3 Consultations per Patient

Model IDP13

# Patients

P14 # Doc.

Meetings

P15 # Nrs.

MeetingsP14P13

P15P13

P14+P15P13

IDEAL01 4,317 2,921 1,396 0.677 0.323 1.000

TRAD01 4,317 2,919 4,317 0.676 1.000 1.676

TRAD02 4,322 2,928 4,322 0.677 1.000 1.677

MCDM-EQ 4,322 3,840 584 0.889 0.135 1.024

MCDM-50WT 4,293 3,822 574 0.890 0.134 1.024

MCDM-50CO 4,319 3,690 806 0.854 0.186 1.041

MCDM-50ME 4,325 3,895 510 0.901 0.118 1.018

MCDM-100CO 4,318 3,548 1,040 0.822 0.241 1.062

Table 15: RQ2: Average values of M3 Consultations per Patient parameters


As the number of consultations at each respective level of care only is relevant in relation to

the number of patients, boxplots will be presented for calculated values of P13, P14P13 , P15

P13 andP14+P15

P13 . As previously, models with values of 1 by construction are not included in the boxplots.

The traditional models are plotted separately in the boxplots of P14+P15P13 as the magnitudes of

the values greatly differ.

Figure 20: RQ2: Boxplot of P13 Number of Patients

Figure 21: RQ2: Boxplot of P14/P13 Doctor Meetings per Patient


Figure 22: RQ2: Boxplot of P15/P13 Nurse Meetings per Patient

Figure 23: RQ2: Boxplot of (P14+P15)/P13 Meetings per Patient for MCDM models

Figure 24: RQ2: Boxplot of (P14+P15)/P13 Meetings per Patient for traditional models


The key takeaway from these results is that all MCDM models compared to the ideal model

have a higher number of doctor meetings per patient and a lower number of nurse meetings per

patient. This could indicate that there is a by-construction-tendency in the MCDM models to

triage an unproportionally large share of patients directly to doctors.

6.2.5 Healthcare Productivity - M4 Clinician Idle Time

The idle time of clinicians is difficult to evaluate on a day-by-day basis in boxplots due to how the

simulation is performed. As the number of patients differ between days, the actual time spent

with patients differ and has a greater impact on the idle time of clinicians than the impact of

the triage models themselves. As such, only the average values of idle time should be evaluated

when comparing the triage models, with the patient volumes averaging out in the long term

perspective of 100 days.

Model IDP16 Idle Share

of Doctor Time

P17 Idle Share

of Nurse Time

IDEAL01 2.04% 4.10%

TRAD01 2.06% 1.28%

TRAD02 1.70% 1.35%

MCDM-EQ 1.40% 6.03%

MCDM-50WT 1.94% 7.44%

MCDM-50CO 2.06% 7.68%

MCDM-50ME 1.09% 13.43%

MCDM-100CO 1.30% 7.61%

Table 16: RQ2: Average values of M4 Clinician Idle Time

The idle share of doctor time is generally low throughout all models, but a pattern can be seen

in which the models with a higher tendency to send patients to nurses are among the models

with the highest idle share of doctor time. The interesting pattern however is seen in the P17

parameter where values differ greatly and the MCDM models have significantly higher idle shares

with nurses. This is most likely due to the low staffing volume of nurses. The staffing policy is

expected to staff the minimum of nurses which enable the system to remain stable with supply

meeting demand. Although if, for example, 4.01 nurses are estimated to be needed, 5 nurses will

be staffed representing an almost 25% increase in supply. This will be reflected in the idle time

as the demand does not take use of the full supply, although the waiting time ratio most likely

will dampen the magnitude of this effect where applied.


6.2.6 Patient Experience - M5 Patient Waiting Times

The final group of performance indicators concern waiting time. Figure 25 and 26 show that

the P50 and P80 waiting times for the traditional models both are lower on average and have a

lower variance than the corresponding MCDM model values. Unlike the traditional models, P50

and P80 values for the MCDM models exhibit heavy upper tails, with several values well above

the upper quantile. Among the different MCDM models, the MCDM-100CO model, which is

the only one not considering waiting times in the triage judgement, has the highest variance in

terms of P50 and P80 waiting times.

The box plots of the P100 waiting times, i.e. the longest time any patient had to wait for their

appointment per day, shows a more equal distribution between the different simulated triage

models than the corresponding P50 and P80 plots. However, similarly as for the P50 and P80

box plots, the MCDM-100CO model has the highest peaks.

Figure 25: RQ2: Boxplot of P18 50th Percentile of Waiting Time




6.3 Phase 2 - Simulations Evaluating Systematic Assessment Errors

(RQ3)

Further, to evaluate the impact of a systematic error in the triage assessment, another 4 simu-

lations were performed with an average of 4, 322 patients per day, totaling 1, 728, 757 patients.

The results of these 4 simulations compared to the MCDM-EQ model without any systematic

error are now presented in the same fashion as the section above.


6.3.1 Cost Efficiency - M1 Clinician Salary Costs

P1 Doctors Staffed and P2 Nurses Staffed were distributed according to the following:

• MCDM-50WT: (P1=34, P2=5) in 100% of days

• MCDM-ERR10O: (P1=33, P2=7) in 94% of days, (P1=33, P2=6) in 6% of days

• MCDM-ERR10U: (P1=34, P2=5) in 94% of days, (P1=35, P2=5) in 4% of days, (P1=35,

P2=4) in 2% of days

• MCDM-ERR40O: (P1=31, P2=9) in 78% of days, (P1=32, P2=9) in 14% of days, (P1=31,

P2=10) in 8% of days

• MCDM-ERR40U: (P1=37, P2=2) in 100% of days

These staffing volumes represent the following hourly staffing costs for each level of error.

Notably, the systematic errors of smaller magnitude does seem to have little or no impact on

the staffing cost whereas the errors of larger magnitude experience significant differences to the

MCDM-EQ model.

Figure 28: RQ3: Boxplot of hourly staffing cost in SEK

The P3 Final Doctor Meeting End Time and P4 Final Nurse Meeting End Time boxplots are

also once again presented to confirm that the staffing model does not cause the length of the

simulated day to unproportionally exceed the 24-hour-mark.


Figure 29: RQ3: Boxplot of P3 Final Doctor Meeting End Time

Figure 30: RQ3: Boxplot of P4 Final Nurse Meeting End Time

The P5 boxplot shows some interesting results. Most notably, only the systematic error of up to

40 percentage points underestimation lead to significantly higher salary costs per patient journey.

The MCDM-ERR10U model performs with equal outcome to that of the original MCDM-EQ

model. However interestingly, both models which overestimate the triage assessment seem to

perform lower salary costs per patient journey. Furthermore, the MCDM-ERR10O model does

seem to outperform the MCDM-ERR40O model with lower costs for both the median value as

well as the lower and upper quartiles.


Figure 31: RQ3: Boxplot of P5 Clinician Salary per Patient Journey (excl. cost for idle)

6.3.2 Cost Efficiency - M2 LEON Enactment

Model IDP6

TPR

P7

TNR

P8

PPV

P9

NPV

P10

ACC

P11

NTR

P12

DRMR

MCDM-EQ 96.5% 34.6% 75.6% 82.5% 76.5% 13.5% 76.2%

MCDM-ERR10O 94.3% 43.8% 77.9% 78.5% 78.0% 18.0% 78.9%

MCDM-ERR10U 96.9% 32.8% 75.3% 83.1% 76.3% 12.6% 75.9%

MCDM-ERR40O 88.9% 55.9% 80.9% 70.6% 78.2% 25.6% 82.6%

MCDM-ERR40U 99.0% 11.9% 70.3% 84.5% 70.9% 4.5% 70.5%

Table 17: RQ3: Average values of M2 LEON enactment parameters

The effect of overestimating or underestimating the nursability score is naturally that the rate of

which the model sends patients to nurses becomes higher or lower, respectively. This is directly

observable in parameter P11 Nurse Triage Rate. Furthermore, once again the MCDM-ERR10U

model performs similarly to the MCDM-EQ model, indicating that an underestimation of up

to 10 percentage points has little effect on the outcome. Most interesting in this group of

performance indicators that the overestimating models show the best performance in many of

the parameters from sending more patients to an initial nurse meeting.


Figure 32: RQ3: Boxplot of P6 True Positive Rate

Figure 33: RQ3: Boxplot of P7 True Negative Rate

Figure 34: RQ3: Boxplot of P8 Positive Predictive Value


Figure 35: RQ3: Boxplot of P9 Negative Predictive Value

Figure 36: RQ3: Boxplot of P10 Accuracy

Figure 37: RQ3: Boxplot of P11 Nurse Triage Rate


Figure 38: RQ3: Boxplot of P12 Doctor Resource Management Rate

6.3.3 Healthcare Productivity - M3 Consultations per Patient

Although a low number of total meetings per patient throughout all models, the effect of

overestimation and underestimation is once again reflected in the results as more patients are

in need of a second meeting when the number of nurse meetings increase. The opposite pattern

is shown for the underestimating models where fewer patients are in need of a second meeting.

Model IDP13

# Patients

P14 # Doc.

Meetings

P15 # Nrs.

MeetingsP14P13

P15P13

P14+P15P13

MCDM-EQ 4,322 3,840 584 0.889 0.135 1.024

MCDM-ERR10O 4,324 3,712 780 0.858 0.180 1.039

MCDM-ERR10U 4,324 3,870 547 0.895 0.126 1.021

MCDM-ERR40O 4,311 3,534 1,102 0.820 0.256 1.075

MCDM-ERR40U 4,329 4,164 196 0.962 0.045 1.007

Table 18: RQ3: Average values of M3 Consultations per Patient parameters


Figure 39: RQ3: Boxplot of P13 Number of Patients

Figure 40: RQ3: Boxplot of P14/P13 Doctor Meetings per Patient


Figure 41: RQ3: Boxplot of P15/P13 Nurse Meetings per Patient

Figure 42: RQ3: Boxplot of (P14+P15)/P13 Meetings per Patient


6.3.4 Healthcare Productivity - M4 Clinician Idle Time

Model IDP16 Idle Share

of Doctor Time

P17 Idle Share

of Nurse Time

MCDM-EQ 1.40% 6.03%

MCDM-ERR10O 2.01% 9.66%

MCDM-ERR10U 0.94% 3.81%

MCDM-ERR40O 1.02% 10.97%

MCDM-ERR40U 1.80% 20.77%

Table 19: RQ3: Average values of M4 Clinician Idle Time

Similar to the results in the RQ2 simulations, the idle share of doctor time is generally low

throughout all models. The large differences in clinician idle time occur in the P17 parameter

which is most likely due to the low staffing volume of nurses. The staffing policy is expected

to staff the minimum of nurses which enable the system to remain stable with supply meeting

demand. Although if, for example, 4.01 nurses are estimated to be needed, 5 nurses will be

staffed representing an almost 25% increase in supply. This will be reflected in the idle time as

the demand does not take use of the full supply.

6.3.5 Patient Experience - M5 Patient Waiting Times

No clear patterns seem to be present in the waiting times due to a systematic error in the

nursability score assessment. This is expected as the staffing module adjusts for these errors.

Interestingly, one could expect models with high average idle time with nurses to perform notably

better in terms of waiting time since the models account for differences in waiting time. However

such a pattern does not seem to be present in the boxplots, and once again, it should be noted

that the absolute value of idle time most likely is similar with all models. The low staffing volume

however makes the share of idle time larger. This could explain the lack of better performance

in models with high average of parameter P17.

7 DISCUSSION 78

Section 7: Discussion

This chapter discusses the empirical results and is structured according to the research

questions. It also includes a discussion of the sustainability aspects of this study.

7.1 RQ1

Despite a lot of medical experience and many years within the healthcare sector, the interviewees

expressed a general agreement that the laws, policies and guidelines regulating triage are hard

to interpret, especially in a digital setting. As there is neither clear targets to reach nor any

monitoring from official authorities regarding how triage is performed, the formal requirements

on a triage system are minor. Nonetheless, it can be concluded that the medical assessment is

the single most important factor to base the triage judgment on. If it however is not possible,

based on the assessment, to give a definite answer on whether a nurse can help, the regulations

are not granular enough to guide the triage judgement to a definite decision. Given that digital

healthcare and automated triage have developed rapidly during recent years, it is likely that

policies and laws have not managed to keep up with the development. Unlike traditional triage

performed by a triage officer, an automated triage model is dependent on quantitative data to

base the judgement on. Hence, triage policies and guidelines developed for triage officers do not

seem to provide the level of detail needed in an automated triage model.

Regarding the question of whether it is possible to include reimbursement as an input factor

or not, one could argue that such a policy is not in line with the Principle of Cost Efficiency

of the ethical platform. At the same time, the regional governance of primary care could be

argued to use different reimbursement policies to stimulate different behavior in the provided

care. Should, for some reason, a region wish to increase the share of patients directly guided to

a higher level of care and incentivize such a model with a higher reimbursement sum for doctor

meetings, one could argue that it would be a natural criteria to include in a digital triage model.

Regional differences in policy to support the needs of the people in that very region is one of the

reasons to why primary care indeed is organized on a regional basis. It is likely that unless more

detailed policies are in place to guide the triage in primary care, different players will interpret

this situation differently, creating potentially unfair disadvantages.

Nevertheless, there were three aspects that dominated the interviews, with an overall consensus

that those three yield the best prerequisites to make a good triage judgement in digital primary

healthcare. The first aspect to consider is the probability of helping the patient, which in the

setting of this study corresponds to the nursability score, i.e. the probability that a nurse can

fulfill the patient’s medical needs. Secondly, waiting times are to be considered to balance waiting

times for the available professions and assure that no available resources are unused. Lastly,

expected cost of treatment should be considered to maintain cost efficiency in the primary care.

7 DISCUSSION 79

7.2 RQ2

It is clear that in the chosen setting, the findings of this case study show that compared to the

traditional triage model where patients always see a nurse first, an automated MCDM triage

model increases cost efficiency in terms of clinician salary costs and productivity in terms of

fewer consultations per patient. Without significant impact on waiting times, the MCDM triage

models including chance of getting helped at first meeting, expected cost and waiting time,

decreased average salary costs per patient with up to 8.3% and reduced average number of

meetings per patient by more than 0.6.

Despite yielding an overall higher cost per patient than the MCDM models, the traditional

triage model is more efficient in terms of only utilizing doctors for patients with an absolute

need of doctor care. A doctor meeting is the highest available level of care within the primary

care sector and given the scarcity of healthcare resources, it could be argued that the purpose

of the LEON principle is to minimize the number of doctor meetings in which the patient’s

medical need could have been served by a nurse. However, this study showed that only utilizing

doctors for patients with a medical need that can not be met by a nurse is not equivalent with

minimizing the overall average cost per patient in a digital primary healthcare setting. The

reason is that a large share of the patients in this setting requires doctor care, causing a large

number of patients to require two meetings before getting their needs served if always seeing

a nurse first. Hence, despite requiring a lower number of doctor consultations, the cost of the

initial nurse meetings outweighs the decreased cost of a lower number of doctor meetings in the

traditional triage model.

The trade-off between reducing average cost per patient and minimizing unnecessary utilization

of doctor resources is clearly visualized in the findings of this study. In dividing cost efficiency

into M1 Clinician Salary Costs and M2 LEON Enactment, it is obvious that while the MCDM

models will outperform the traditional models in terms of M1, it comes at the cost of some key

performance indicators of M2. An interesting observation is that the MCDM models however

have a significantly higher share of patients directly referred to the lowest efficient level of care,

if interpreting efficient as being able to fulfill the patient’s medical needs at the first meeting.

In this study, waiting times were defined as the time from seeking care to seeing the first clinician,

which is the general definition of waiting times used within the healthcare sector. However, the

results of this study highlight a potential shortcoming of this definition. Consider an example

of an public transport commuter traveling from point A to point B. There are two potential

busses available, bus X with departure time 10.00 AM and bus Y with departure time 10.30

AM. Bus X arrive at destination B 11.10 AM and bus B at 11.00 AM. Without conducting any

research on the question at hand, it is reasonable to assume that a vast majority would prefer

bus Y, despite 30 minutes longer waiting time. We consider this reasoning to be applicable for

healthcare as well. Hence, the slightly shorter waiting times in the traditional triage model is a

bit misguiding, since approximately 67% of the patients does not get helped at the first meeting

and must therefore be placed back in the doctor queue to wait for their second meeting. A more

appropriate performance measurement might therefore be to evaluate time to getting helped,

7 DISCUSSION 80

which, given the high level of patients not getting helped at the first meeting in the traditional

triage model, would favour the MCDM model. Such an interpretation of the factor to consider

is applied when evaluating the expected cost of treatment where the total cost is calculated as a

probability-weighted sum of costs. However the factor concerning probability to help still only

concern the immediate meeting as the probability that a patient could be helped in some way

is 100% in the setting of this study.

7.3 RQ3

Simulation of systematic errors in the assessment preceding the triage judgement shows that

systematic errors with a magnitude of up to 10 percentage points has minor impact on the

overall performance of the MCDM triage model. However, for errors with a magnitude of up to

40 percentage points, the performance of the triage model is affected.

The main implication of systematic errors with large magnitude is that the share of patients

initially referred to nurses changes. For the MCMD-ERR40U model, which underestimates the

nursability with up to 40 percentage points, only 4.5% of the patients were guided to a nurse.

Despite decreasing average number of meetings per patient, it causes over utilization of doctor

resources and consequently, increases average salary cost per patient, as illustrated in Figure

31. Similarly, the MCMD-ERR40O model yields increased patient volumes guided to nurses.

However, despite increasing average number of meetings per patient with 5%, overestimation of

the nursability has a small and even slightly decreasing impact on cost per patient.

A possible explanation to the low impact of systematic errors with magnitude up to 10

percentage points is the distribution of the empirical nursability data, upon which the

nursability distribution in the simulation was constructed. As illustrated in Appendix A, the

density of the nursability distribution is concentrated around the end points of the scale.

Hence, only a minor proportion of the nursability scores are allocated in the (0.3, 0.6) range,

where the MCDM triage model may change the triage judgement from nurse to doctor or vice

versa. Therefore, a systematic error never exceeding 10 percentage points will only change the

triage judgement for a very small share of the patients.

An interesting observation is that overestimation of the nursability slightly improves the cost

efficiency of the MCDM model. Recalling the ideal triage model simulated in the RQ2 section,

32% of the patients had medical needs that could be served by nurses. Hence, overestimation

of the nursability score, which increases the share of patients referred to nurse from 13.5%

without systematic error to 25.6% with a systematic error uniformly distributed between 0 and

40 percentage points, has a nurse referral ratio closer to the ideal model than without any

systematic error. This indicates that there is room for improvement in calibrating the suggested

MCDM models. As this study has not set out to optimize the MCDM models in any way, it is

expected that such improvements could yield large benefits. One question to ask as a consequence

of this finding is whether a digital triage model should have the same rate of guiding an incoming

set of patients to nurses as the share of the same set of patients who may be helped by nurses. If

7 DISCUSSION 81

the impacts of systematic errors on such a model were to be tested, it is believed that a deeper

understanding of the negative effects from both overestimation and underestimation would be

gained.

7.4 Reflection on Sustainability Aspects

The United Nations has defined 17 sustainable development goals aimed at addressing the global

challenges our world is facing. The goals are to be reached by 2030 and aimed at achieving a

better and more sustainable future for all. This study addresses goal number 3, good health and

well being. Quoting the UN [68]:

By focusing on providing more efficient funding of health systems, improved sanitation

and hygiene, and increased access to physicians, significant progress can be made in

helping to save the lives of millions.

As previously stated, The World Health Organization estimates that by 2035, there will be a

global deficit of about 12.9 million healthcare professionals [1] and digitalization of the healthcare

services is projected to play a fundamental role in improving healthcare cost efficiency [31].

This study has shown that an automated MCDM triage model can improve cost efficiency and

productivity in digital healthcare, which is an important step in countering the rapidly increasing

deficit of healthcare professionals and resources.

8 CONCLUSION 82

Section 8: Conclusion

This chapter presents the final conclusion to the stated research questions, its contributions to

science and suggestions for further research.

8.1 Summary and Findings

RQ1 It has been determined that the medical assessment of a patient is the key component of

the basis upon which to base a triage judgment. This assessment should primarily be translated

into factors concerning the probability of each alternative outcome to help the patient, as well

as the expected cost of treatment of each alternative. Furthermore, the waiting times to each

alternative outcome was concluded to be an appropriate factor to consider, unrelated to the

medical assessment.

Discussions were held regarding the appropriateness of including reimbursement, co-pay and the

patient’s preference as input factors to the triage judgment. As the official laws and guidelines

do not present clear answers to whether those factors would be appropriate to include, no

recommendation is laid out in this study.

RQ2 This study has shown that an automated MCDM triage model can improve cost efficiency

in terms of clinician salary costs and productivity in terms of fewer consultations per patient

compared to the traditional triage model. When studying cost efficiency in terms of LEON

enactment, it is clear that the MCDM models enable over twice as many patients to be directly

guided to the appropriate level of care at their first meeting. Furthermore, the MCDM models

use nurses to a significantly effect, meaning that a larger share of nurse meetings contribute

to helping the patient. However these results come at the cost of the corresponding doctor

resource management rate. As long as there is no way of accurately classify patients such that

at least all patients who are able to get helped by a nurse indeed are helped by a nurse, doctor

resources will be used to treat patients without the need of doctor resources. The simulations

are not detailed enough to present exact values of such a trade-off but the results suggest that

on average, approximately 10% lower salary costs can be achieved if the doctor staffing volume

is increased by 20− 25% and consequently the same share of doctor meetings are used to treat

patients without need of doctor consultations.

At the same time, the study did not present any evidence supporting that the patient experience

in terms of waiting times would be significantly affected in positive nor negative fashion. This

argument is based on the definition of waiting times which calculated time until the patient has

started a consultation, rather than the time until the patient has been helped.

8 CONCLUSION 83

RQ3 The evaluation of systematic assessment errors shows that the impact is minor for

systematic errors with a magnitude of up to 10 percentage points for the studied patient group.

This is expected due to the nature of the empirical distribution of the nursability input data in

which few patients are assigned a score in proximity of 0.50. Thus, it can be concluded that

these result are significantly dependant on the input data. However, when increasing the error

to up to 40 percentage points, the performance of the triage model changes. Systematic

underestimation of the probability that a nurse can fulfill a patient’s medical need is shown to

cause an over-representation of doctor referrals, yielding increased average cost per patient and

over-utilization of doctor resources. Notably, the models that overestimate the nursability score

experience improved LEON enactment in several parameters. This is most likely due to the

fact that the baseline without systematic error has a nurse triage rate below the share of

patients that indeed can be helped by a nurse. Thus, an unexpected conclusion to be drawn

from the study of systematic errors is that the triage model will experience improved

parameter values when the systematic error contribute to a nurse triage rate closer to the one

of the ideal model where patients always are guided immediately to the LEON level of care.

8.2 Contribution

Using a simulation approach, this study has shown that an automated MCDM triage model

has potential to improve cost efficiency and increase productivity in digital primary healthcare.

Given the lack of previous research and recent public authority requests of research on

automated triage, this study is a first step in bridging the knowledge gap regarding the

potential of automatizing the triage process in Swedish digital primary care.

MCDM models have mostly been studied in a operations research context, but this study shows

that the theory of MCDM is also efficient in settings outside traditional operations research. It is

known to be an efficient tool when evaluating a set of contradicting criteria and this study show

that this strength is useful in triage judgement, where the somewhat contradicting objectives of

costs, waiting time and probability of getting helped at first meeting have to be balanced.

In line with Iserson & Moskop [6], this study seeks to be a starting point for a more fair,

accurate and efficient triage system in all parts of primary care; a system which does not rely on

the human factor but on the algorithm-based suggestion. The emergency care sector is far ahead

of the primary care section when it comes to this, although individual regions have developed

tools for primary care triage as mentioned in the literature chapter of this study.

8.3 Limitations and Further Research

The automated assessment yielding nursability scores as output is an important part of the triage

system, which has been excluded from the scope of this study as the complexity of the simulation

would have increased exponentially in order to gain reliability. This study have shown that with

accurate assessments, automated MCDM triage models have a great potential in increasing the

cost efficiency and productivity of primary care. Hence, further research on how to gather and

process medical data for high precision automated assessments is a crucial part in the digitization

of the healthcare industry. As an area of further research, we suggest studies on the initial part

of the triage process (Figure 4) aimed at increasing the understanding of the current accuracy

in automated assessments and how to collect and process medical data to increase the accuracy.

Further, this study has assumed a digital primary healthcare setting and all empirical data has

been gather from the case company, which is a leading digital primary care provider. Even

though the automated MCDM triage model showed great potential in this setting, the findings

are not necessarily applicable in a physical primary care setting, since physical care covers a

wider range of symptoms. One thing noticed in the study simulations was that the differences

between simulated days was small due to how the data was collected. However, future studies

on the potential of automated triage in both the physical primary care, a larger share of digital

primary care as well as in dealing with seasonal effects are strongly encouraged.

Lastly, the focus of this study has not been to optimize the MCDM model in any way. It is

suspected that there are improvement potential to be captured by calibrating the relative weights

and how the input data is processed to determine the relative advantage of an alternative to

another.

References

[1] World Health Organization. Global health workforce shortage to reach 12.9 million in

coming decades. url:

https : / / www . who . int / mediacentre / news / releases / 2013 / health - workforce -

shortage/en/. (accessed: 28.04.2020).

[2] Marcial Velasco Garrido, Annette Zentner, and Reinhard Busse. “The effects of

gatekeeping: a systematic review of the literature”. In: Scandinavian journal of primary

health care 29.1 (2011), pp. 28–38.

[3] Socialstyrelsen. Nationella indikatorer for God vard. url:

https : / / www . socialstyrelsen . se / globalassets / sharepoint -

dokument / artikelkatalog / nationella - riktlinjer / 2009 - 11 - 5 . pdf. (accessed:

28.04.2020).

[4] Goran Stiernstedt. “Digifysiskt vardval - Tillganglig primarvard baserad pa behov och

kontinuitet”. In: SOU:2019:42 (2019).

[5] Ylva Trolle Lagerros et al. “Effekter av inforandet av digitala vardmoten”. In: (2019).

[6] Kenneth V Iserson and John C Moskop. “Triage in medicine, part I: concept, history, and

types”. In: Annals of emergency medicine 49.3 (2007), pp. 275–281.

[7] Steven Horng et al. “Creating an automated trigger for sepsis clinical decision support at

emergency department triage using machine learning”. In: PloS one 12.4 (2017).

[8] Joany M Zachariasse et al. “Validity of the Manchester Triage System in emergency care:

A prospective observational study”. In: PloS one 12.2 (2017).

[9] Akira Kuriyama, Seigo Urushidani, and Takeo Nakayama. “Five-level emergency triage

systems: variation in assessment of validity”. In: Emerg Med J 34.11 (2017), pp. 703–710.

[10] Alexandre Chater Taleb et al. “The efficacy of telemedicine for ophthalmology triage by a

general practitioner”. In: Journal of telemedicine and telecare 11.1 suppl (2005), pp. 83–85.

[11] Jillian B Harvey et al. “The impact of telemedicine on pediatric critical care triage”. In:

Pediatric Critical Care Medicine 18.11 (2017), e555–e560.

[12] Natale Daniele Brunetti et al. “Pre-hospital electrocardiogram triage with telemedicine

near halves time to treatment in STEMI: A meta-analysis and meta-regression analysis of

non-randomized studies”. In: International journal of cardiology 232 (2017), pp. 5–11.

[13] Iain Robertson-Steel. “Evolution of triage systems”. In: Emergency Medicine Journal 23.2

(2006), pp. 154–155.

[14] Michael Christ et al. “Modern triage in the emergency department”. In: Deutsches

Arzteblatt International 107.50 (2010), p. 892.

[15] Kevin Mackway-Jones, Janet Marsden, and Jill Windle. Emergency triage: Manchester

triage group. John Wiley & Sons, 2014.

[16] Sveriges Kommuner & Regioner. Ekonomirapporten, oktober 2019 - Om kommunernas och

regionernas ekonomi. url: https://rapporter.skr.se/ekonomirapporten-oktober-

2019.html. (accessed: 18.03.2020).

https://www.who.int/mediacentre/news/releases/2013/health-workforce-shortage/en/

https://www.who.int/mediacentre/news/releases/2013/health-workforce-shortage/en/

https://www.socialstyrelsen.se/globalassets/sharepoint-dokument/artikelkatalog/nationella-riktlinjer/2009-11-5.pdf

https://www.socialstyrelsen.se/globalassets/sharepoint-dokument/artikelkatalog/nationella-riktlinjer/2009-11-5.pdf

https://rapporter.skr.se/ekonomirapporten-oktober-2019.html

https://rapporter.skr.se/ekonomirapporten-oktober-2019.html

[17] Statistiska Centralbyran. Halsorakenskaper. url:

https : / / www . scb . se / hitta - statistik / statistik - efter -

amne/nationalrakenskaper/nationalrakenskaper/halsorakenskaper/.

[18] Sveriges Kommuner & Regioner. Kop av verksamhet 2019. url: https://webbutik.skr.

se/bilder/artiklar/pdf/7585-550-9.pdf.

[19] Peter Lindgren. Ersattning i sjukvarden MODELLER, EFFEKTER,

REKOMMENDATIONER. SNS Forlag, 2014.

[20] Sveriges Kommuner & Regioner. Fast riskjusterad kapitationsersattning. url:

https : / / skr . se / demokratiledningstyrning / driftformervalfrihet /

valfrihetssystemochersattningsmodeller / halsoochsjukvard /

ersattningsmodellerhalsoochsjukvard / ersattningsmodelleriprimarvard /

fastersattning.28895.html. (accessed: 28.02.2020).

[21] Sveriges Kommuner & Regioner. Rorlig ersattning kopplat till besok eller atgarder. url:

https : / / skr . se / demokratiledningstyrning / driftformervalfrihet /

valfrihetssystemochersattningsmodeller / halsoochsjukvard /

ersattningsmodellerhalsoochsjukvard / ersattningsmodelleriprimarvard /

rorligersattning.28896.html. (accessed: 28.02.2020).

[22] Sveriges Kommuner & Regioner. Ersattningar for digitala vardtjanster i primarvarden.

url: https://skr.se/halsasjukvard/ehalsa/digitalavardtjansteriprimarvarden/

ersattningardigitalavardtjansteriprimarvarden . 28836 . html. (accessed:

18.02.2020).

[23] Sveriges Kommuner & Regioner. Meddelande fran styrelsen - Utvecklingen av digitala

vardtjanster. url: https :

/ / skr . se / download / 18 . 6c06ad7e16b45332f4e832e8 / 1560502427310 / Digitala -

vardtjanster-2019-06-14.pdf. (accessed: 25.02.2020).

[24] Statens Offentliga Utredningar. SOU 1993:93 Vardens svara val, Slutbetankande. url:

https : / / www . regeringen . se / 49b6ca / contentassets /

6c4cb9f4c3ef4296b68ea7c6cefbf1d2/del- 1- kap.- 1- t.o.m.- kap.- 8- vardens-

svara-val. (accessed: 20.02.2020).

[25] Socialstyrelsen. God vard – om ledningssystem for kvalitet och patientsakerhet i halso- och

sjukvarden. 2006.

[26] Statens Offentliga Utredningar. SOU 2016:2 Effektiv vard - Slutbetankande av En

nationell samordnare for effektivare resursutnyttjande inom halso och sjukvarden. url:

https://www.regeringen.se/contentassets/42b0aef4431c4ebf9410b8ee771830eb/

effektiv - vard --- slutbetankande - av - en - nationell - samordnare - for -

effektivare-resursutnyttjande-inom-halso--och-sjukvarden_sou-2016-2.pdf.

(accessed: 20.02.2020).

[27] Magnus Nyhlen. Principerna for varden bor uppdateras. url: https :

//www.dagenssamhalle.se/debatt/principerna-varden-bor-uppdateras-28106.

(accessed: 28.04.2020).

https://www.scb.se/hitta-statistik/statistik-efter-amne/nationalrakenskaper/nationalrakenskaper/halsorakenskaper/

https://www.scb.se/hitta-statistik/statistik-efter-amne/nationalrakenskaper/nationalrakenskaper/halsorakenskaper/

https://webbutik.skr.se/bilder/artiklar/pdf/7585-550-9.pdf

https://webbutik.skr.se/bilder/artiklar/pdf/7585-550-9.pdf

https://skr.se/demokratiledningstyrning/driftformervalfrihet/valfrihetssystemochersattningsmodeller/halsoochsjukvard/ersattningsmodellerhalsoochsjukvard/ersattningsmodelleriprimarvard/fastersattning.28895.html




https://skr.se/demokratiledningstyrning/driftformervalfrihet/valfrihetssystemochersattningsmodeller/halsoochsjukvard/ersattningsmodellerhalsoochsjukvard/ersattningsmodelleriprimarvard/rorligersattning.28896.html




https://skr.se/halsasjukvard/ehalsa/digitalavardtjansteriprimarvarden/ersattningardigitalavardtjansteriprimarvarden.28836.html

https://skr.se/halsasjukvard/ehalsa/digitalavardtjansteriprimarvarden/ersattningardigitalavardtjansteriprimarvarden.28836.html

https://skr.se/download/18.6c06ad7e16b45332f4e832e8/1560502427310/Digitala-vardtjanster-2019-06-14.pdf



https://www.regeringen.se/49b6ca/contentassets/6c4cb9f4c3ef4296b68ea7c6cefbf1d2/del-1-kap.-1-t.o.m.-kap.-8-vardens-svara-val



https://www.regeringen.se/contentassets/42b0aef4431c4ebf9410b8ee771830eb/effektiv-vard---slutbetankande-av-en-nationell-samordnare-for-effektivare-resursutnyttjande-inom-halso--och-sjukvarden_sou-2016-2.pdf



https://www.dagenssamhalle.se/debatt/principerna-varden-bor-uppdateras-28106

https://www.dagenssamhalle.se/debatt/principerna-varden-bor-uppdateras-28106

[28] Maria Ost-Backa, Marina Tuutma, and et.al. Ersatt inte sjukskoterskors beslut med AI

utan evidens. url: https://www.dagensmedicin.se/artiklar/2020/01/31/ersatt-

inte-sjukskoterskors-beslut--med-ai-utan-evidens/. (accessed: 28.04.2020).

[29] Johan Gustafsson and Ulf Osterstad. I centrum av digitaliseringen. url: https://www.

medtechmagazine.se/article/view/706618/i_centrum_av_digitaliseringen?rel=

related. (accessed: 28.04.2020).

[30] Mikael Milden. I centrum av digitaliseringen. url: https://www.medtechmagazine.se/

article/view/706618/i_centrum_av_digitaliseringen?rel=related. (accessed:

28.04.2020).

[31] Pia Hardy, Daniel Boldt-Christmas, and Magnus Tyreman. Vardet av digital teknik i den

svenska varden. url: https://www.mckinsey.com/industries/healthcare-systems-

and - services / our - insights / digitizing - healthcare - in - sweden. (accessed:

21.02.2020).

[32] Sveriges Kommuner & Regioner. Digitala vardtjanster i primarvarden. url: https://www.

skr.se/halsasjukvard/ehalsa/digitalavardtjansteriprimarvarden.28301.html.

(accessed: 18.03.2020).

[33] KRY. KRY bygger AI-triage - intervju med Andreas Mattsson. url: https://www.kry.

se/medicinsk-kvalitet/kry-bygger-ai-triage-intervju-med-andreas-mattsson/.

(accessed: 21.02.2020).

[34] Josefin Lennen Merckx. Kritik mot digitala vardgivare: ”Natlakarna dranerar varden pa

resurser”. url: https://www.svt.se/nyheter/inrikes/kritik- mot- digitala-

vardgivare-natlakarna-dranerar-varden-pa-resurser-1. (accessed: 27.04.2020).

[35] Jesper Cederberg. Digital vard hissad och dissad i debatt med strulande teknik. url: https:

//lakartidningen.se/Aktuellt/Nyheter/2018/10/Digital- vard- hissad- och-

dissad-i-strulig-debatt/. (accessed: 27.04.2020).

[36] Triantaphyllou Evangelos. “Multi-criteria decision making methods: a comparative study”.

In: Netherland: Kluwer Academic Publication 4 (2000).

[37] Hans-Jurgen Zimmermann. Fuzzy set theory—and its applications. Springer Science &

Business Media, 2011.

[38] Martin K Starr and David W Miller. Stock Image Executive Decision and Operations

Research. Prentice Hall, 1969.

[39] Roseanna W Saaty. “The analytic hierarchy process—what it is and how it is used”. In:

Mathematical modelling 9.3-5 (1987), pp. 161–176.

[40] Valerie Belton and Tony Gear. “On a short-coming of Saaty’s method of analytic hierar-

chies”. In: Omega 11.3 (1983), pp. 228–230.

[41] Thomas L Saaty. “An exposition of the AHP in reply to the paper “remarks on the analytic

hierarchy process””. In: Management science 36.3 (1990), pp. 259–268.

[42] Evangelos Triantaphyllou and Stuart H Mann. “An examination of the effectiveness of

multi-dimensional decision-making methods: a decision-making paradox”. In: Decision

Support Systems 5.3 (1989), pp. 303–312.

[43] W David Kelton. Simulation with ARENA. McGraw-hill, 2002.

https://www.dagensmedicin.se/artiklar/2020/01/31/ersatt-inte-sjukskoterskors-beslut--med-ai-utan-evidens/

https://www.dagensmedicin.se/artiklar/2020/01/31/ersatt-inte-sjukskoterskors-beslut--med-ai-utan-evidens/

https://www.medtechmagazine.se/article/view/706618/i_centrum_av_digitaliseringen?rel=related





https://www.mckinsey.com/industries/healthcare-systems-and-services/our-insights/digitizing-healthcare-in-sweden

https://www.mckinsey.com/industries/healthcare-systems-and-services/our-insights/digitizing-healthcare-in-sweden

https://www.skr.se/halsasjukvard/ehalsa/digitalavardtjansteriprimarvarden.28301.html

https://www.skr.se/halsasjukvard/ehalsa/digitalavardtjansteriprimarvarden.28301.html

https://www.kry.se/medicinsk-kvalitet/kry-bygger-ai-triage-intervju-med-andreas-mattsson/

https://www.kry.se/medicinsk-kvalitet/kry-bygger-ai-triage-intervju-med-andreas-mattsson/

https://www.svt.se/nyheter/inrikes/kritik-mot-digitala-vardgivare-natlakarna-dranerar-varden-pa-resurser-1

https://www.svt.se/nyheter/inrikes/kritik-mot-digitala-vardgivare-natlakarna-dranerar-varden-pa-resurser-1

https://lakartidningen.se/Aktuellt/Nyheter/2018/10/Digital-vard-hissad-och-dissad-i-strulig-debatt/



[44] Kevin Dooley. “Simulation research methods”. In: Companion to organizations (2002),

pp. 829–848.

[45] David L. Olson and Dursun Delen. Advanced Data Mining Techniques. Springer, 2008.

[46] Ulrika Ostlund et al. “Combining qualitative and quantitative research within mixed

method research designs: a methodological review”. In: International journal of nursing

studies 48.3 (2011), pp. 369–383.

[47] David R Thomas. “A general inductive approach for analyzing qualitative evaluation data”.

In: American journal of evaluation 27.2 (2006), pp. 237–246.

[48] Par Blomkvist and Anette Hallin. Method for engineering students: Degree projects using

the 4-phase Model. Studentlitteratur AB, 2015.

[49] Steinar Kvale and Svend Brinkmann. Den kvalitativa forskningsintervjun.

Studentlitteratur, 2014.

[50] Mark NK Saunders. Research methods for business students, 5/e. Pearson Education India,

2011.

[51] Amanda Bolderston. “Conducting a research interview”. In: Journal of Medical Imaging

and Radiation Sciences 43.1 (2012), pp. 66–76.

[52] Carter McNamara. General guidelines for conducting interviews. 1999.

[53] David Silverman. Qualitative research. Sage, 2016.

[54] Bill Gillham. Research Interviewing: The range of techniques: A practical guide. McGraw-

Hill Education (UK), 2005.

[55] Ylona Chun Tie, Melanie Birks, and Karen Francis. “Grounded theory research: A design

framework for novice researchers”. In: SAGE open medicine 7 (2019), p. 2050312118822927.

[56] Brian D Ripley. Stochastic simulation. Vol. 316. John Wiley & Sons, 2009.

[57] Arnold O Allen. Probability, statistics, and queueing theory. Academic press, 2014.

[58] Frank E Ritter et al. “Determining the number of simulation runs: Treating simulations

as theories by not sampling their behavior”. In: Human-in-the-loop simulations. Springer,

2011, pp. 97–116.

[59] Frida Forsberg, Tua Malmberg, and Kristina Sandstrom. 1177-skoterskor vittnar: ”Skulle

halla samtalen korta”. url: https://www.svd.se/1177-skoterskor-vittnar-skulle-

halla-samtalen-korta. (accessed: 06.05.2020).

[60] Ru-xin Nie, Jian-qiang Wang, and Hong-yu Zhang. “Solving solar-wind power station

location problem using an extended weighted aggregated sum product assessment

(WASPAS) technique with interval neutrosophic sets”. In: Symmetry 9.7 (2017), p. 106.

[61] Yao Ji, Guo H Huang, and Wei Sun. “Risk assessment of hydropower stations through an

integrated fuzzy entropy-weight multiple criteria decision making method: A case study of

the Xiangxi River”. In: Expert Systems with Applications 42.12 (2015), pp. 5380–5389.

[62] Song Dawei et al. “A comprehensive evaluation model of power market operation efficiency

based on extension cloud theory”. In: Energy Procedia 156 (2019), pp. 302–309.

[63] Lawrence Leung. “Validity, reliability, and generalizability in qualitative research”. In:

Journal of family medicine and primary care 4.3 (2015), p. 324.

https://www.svd.se/1177-skoterskor-vittnar-skulle-halla-samtalen-korta

https://www.svd.se/1177-skoterskor-vittnar-skulle-halla-samtalen-korta

[64] Mary Dixon-Woods et al. “The problem of appraising qualitative research”. In: BMJ

Quality & Safety 13.3 (2004), pp. 223–225.

[65] Yvonna S Lincoln, Susan A Lynham, and Egon G Guba. “Paradigmatic controversies,

contradictions, and emerging confluences, revisited”. In: The Sage handbook of qualitative

research 4 (2011), pp. 97–128.

[66] Sergi Fabregues and Jose F Molina-Azorın. “Addressing quality in mixed methods research:

A review and recommendations for a future agenda”. In: Quality & Quantity 51.6 (2017),

pp. 2847–2863.

[67] Alicia O’Cathain. “Assessing the quality of mixed methods research: Toward a

comprehensive framework”. In: Handbook of mixed methods in social and behavioral

research 531 (2010), p. 555.

[68] The United Nations. About the Sustainable Development Goals. url:

https://www.un.org/sustainabledevelopment/sustainable-development-goals/.

(accessed: 13.05.2020).

https://www.un.org/sustainabledevelopment/sustainable-development-goals/

Appendix A: Empirical Nursability Distribution

Figure 46: Empirical mass distribution of nursability scores

Appendix B: Empirical Meeting Length Distributions

Figure 47: Empirical meeting length (minutes) distribution for nurse meetings with outcome

patient helped

Figure 48: Empirical meeting length (minutes) distribution for nurse meetings with outcome

referral to doctor

Figure 49: Empirical meeting length (minutes) distribution for doctor meetings

www.kth.se

Automated Triage in Digital Primary Care

Documents

Transcript of Automated Triage in Digital Primary Care