GHM201 - Session 6: Assessing health...

document.docx 06 May 2023 of 59

Before you start working on updating this session, please turn on tracked changes.

It is also advisable that you familiarise yourself with the interactive version of the session as this editable version doesn’t have all the formatting and interactions.

http://dl.lshtm.ac.uk/DLTesting/GHM201/sessions/ghm201_s06/GHM201_s06_010_010.html

This session was written by Loveday Penn-Kekana, Ellen Nolte, Martin McKee and Dina Balabanova

GHM201 - Session 6: Assessing health systems

Session Table of Contents

1 Aims and objectives2 Planning your study3 Introduction4 Why assess health systems?5 Developing the field of health system research6 Selected approaches for health systems assessment7 Challenges in assessing health systems8 Doing health systems research9 Integrating activity10 Summary11 References

1 Aims and objectives

Page 1

Aims

The aims of this session are to introduce key concepts and practical methods used in assessing and comparing health systems in high-, middle- and low-income countries.

Objectives

By the end of this session you should be able to:




Discuss why you might want to assess and compare health systems and the challenges that you may face in doing so.

Describe some of the main approaches you might use to assess and compare health systems.

Critically appraise a range of these approaches. Explain some of the key conceptual and methodological challenges to assessing and

comparing health systems and choose approaches that are appropriate to the settings you are working in and the issues you are addressing.

2 Planning your study

Page 1

In this session you should first work through the different screens and spend time on the various activities and exercises. You will also be required to read any essential reading, as indicated. This should take you about two to three hours.

You should then complete the integrating activity, referring to the essential reading provided and the recommended reading within the session and drawing on your own personal experience. This should take you about two hours.

Finally, you should spend a further two hours on self-study covering the recommended reading and two or three texts from the further reading (optional).

3 Introduction

Page 1

In your reading for the module so far, you will have noted the wide range of different approaches and methodologies that have been used when analysing the key features of health systems. There is a growing interest in assessing how heath systems perform and sharing the lessons that can be learned. Consequently, this session will first discuss why there is now such a great interest in evaluating health systems.

If you reflect on what you have learned in the previous sessions, you will recall that assessing health systems is not a simple task.

Health systems have multiple parts (illustrated in the ‘building blocks’ framework, see Session 1).

Moreover, when you bring them together, they interact in ways that are often difficult to predict, in part because they are shaped by their context and environment, but also because they involve actions of real people, who do not always behave in the ways that you think they will, as they have their own interests, agendas and preconceptions (not all of which they may be willing to share).

Finally, health systems are often pursuing several different goals that may conflict with each other (e.g. improving health outcomes while reducing costs, or trying to respond to patients’ expectations, even when they are asking for treatment that is


ineffective), and those working in them will have their own goals, which may range from serving the poor to making a quick profit.

Before we even begin to assess or compare health systems, we have to be sure what we think constitutes a ‘health system’. Session 1 demonstrated the range of definitions of health systems that are used, and the difficulties in establishing the boundaries of a health system.

Then we need to decide what we want the system to do. Most (although not all: Remember that health care is a major sector in the economy in many countries, so some people will view it as being no different from any other sector, such as leisure services or manufacturing.) of us can agree that it should improve the health of the population.

Yet it can be very difficult to attribute health improvements to health care policies, given how many factors, both inside and outside the health system, influence health. All of these considerations are influenced by the broader context – wealth, history, values – of the country that is so important in understanding the way that health systems operate.

Page 2

This session will introduce you to the principles of health systems research and to several approaches that have been proposed to help undertake systematic health systems research.

Many people have tried to simplify this complexity. One example is that adopted by the World Health Organization (WHO), in its World Health Report (WHO 2000), which treated health systems as a sort of ‘black box’ [link – show definition on click – see box below] (although it did identify certain core functions), ranking countries on their ability to achieve a set of goals, health improvement, responsiveness to patients’ needs, and fairness of financing, but without going into great detail about how it did so.

[definition]

With a Black Box you can see what goes in and what comes out, but you will not know how the transformation takes place

[/end definition]

We will then look at another way in which we can begin to simplify the complexity of health systems, by focusing on the different levels of decision-making within them: the micro-level (the process and outcomes of patient care); the meso-level (the organisational framework within which this care occurs); and the macro-level, which includes governance, financing and design of the delivery system.

You will then work through a series of commonly used health systems research methods, discussing their advantages and disadvantages, and considering when they might be appropriate to answer particular research questions. The approaches and methods chosen will depend on the question being asked and the data available. Crucially, they will also depend on the values and objectives of those conducting the assessment. This session will not cover


in detail the methods, which are often generic to social sciences. The specific challenges facing low-income countries will be outlined, including what research capacity exists for carrying out rigorous assessment. The concepts and methods covered will then be practised in an exercise in which you will design a health system assessment.

4 Why assess health systems?

Page 1

Reflection point

Drawing on your knowledge and personal experience, why do you think it would be useful to assess the functioning or achievement of a health system?

What might be some of the likely challenges in doing so?

Let us take the example of international targets such as the Millennium Development Goals (MDGs), which have prioritised improvements in maternal and child health, and which can therefore be thought of as indicators of health system performance.

Can you think of at least three ways in which maternal and child health outcomes (MDGs 4 and 5) can be problematic as measures of health system achievement?

Please post some of your reflections on Moodle [link] and respond to other posts

Question feedback

Why study health systems? Because medical advances (new drugs, innovative technologies, increased knowledge of what works) mean that health systems are increasingly important in preventing death; because they cost money that could be spent in other ways; because they raise fundamental political questions about the obligations linking the state and the individual; and because they can attract media attention when things go wrong etc. (see Session 1).

What are the challenges? The boundaries of a health system can be difficult to define; the elements within it are complex and interact in ways that can be difficult to predict; the system is embedded in and influenced by a range of contextual factors (geographical, economic, political, historical and cultural), which make comparing and assessing health systems difficult.

The example of MDGs 4 and 5: the first problem in assessing the ability of health systems to achieve progress towards these goals, measured by maternal and child health (MCH) outcomes, is that these are strongly influenced by factors outside the conventional boundaries of a health system. Thus, access to clean water, sanitation, primary education for girls, nutrition and women’s employment may have as big an impact on MCH (particularly for under-5 mortality) as anything the health system does. Second, MCH outcomes are influenced by many factors within the health


system, making it difficult to attribute progress, or lack of it, to specific policies (thereby identifying those that might be helpful elsewhere). On the other hand, it is argued that MCH is a good global indicator of overall health system performance as it reflects so many parts of the health system – from clinics, to hospitals, to referral routes, to labs, to availability of blood, to skilled health professionals – all of which must work together to achieve improvements in maternal mortality. Disentangling the role of the different factors and their interaction will influence the way evaluation is designed and conducted. For a more detailed discussion, see Harmer, in Balabanova et al. 2011.

5 Developing the field of health system research

Page 1

Health systems performance assessment

Interest in understanding a functioning health system and its impact on health is not new. Much of the early interest of health systems research focused on assessing the performance of the various elements of health services.

Nolte and McKee (2012 ) give examples from the mid eighteenth century, with Florence Nightingale, an English nurse and, although less well known, pioneering statistician, describing variations in outcomes among different hospitals, and subsequently arguing for the systematic collection of mortality data to inform policies that might improve the quality of service delivery (quoting Spiegelhalter 1999). Others followed, most notably Emory Codman, a Boston surgeon working in the first decade of the twentieth century, who founded the ‘End Results Hospital’, one of the first attempts to monitor outcomes routinely.

However, despite the efforts of these early pioneers, there was little progress until the mid- 1980s, which saw what Relman (1988 ) described as the ‘third revolution in health care’. This stimulated the first attempts to undertake international comparisons, most notably in work by the Organisation for Economic Cooperation and Development (OECD), which in turn led to the benchmarking of countries’ performance (OECD 1985).

Interest in comparisons of health system performance made a quantum leap in 2000, with the publication of the landmark World Health Report (WHO 2000). In an extremely controversial move, the authors ranked 192 countries in terms of the performance of their health systems. This was both ambitious, because of the significant advances in methodology and the scale of data collection involved (and in many cases, estimation of data where they were lacking), and courageous, given that those countries ranked low would be unhappy, a particular challenge for the WHO as it was, and is, dependent on support (financial and political) from its member states.

The report redefined and expanded the concept of ‘performance’, setting out three goals for a health system: improving health, delivering services that are responsive to legitimate patient needs, and which generate revenue fairly. It considered both the level and distribution (and therefore equity) of health outcomes and responsiveness, making five goals in all (Nolte and


McKee 2012). As expected, the report generated widespread debate among the academic and policy communities and, although the WHO has never repeated it, it made major advances to the field of health system performance and stimulated much greater efforts to collect data that previously had been unavailable.

The rankings in the 2000 World Health Report largely sidestepped the debate about what aspects of health systems were important in achieving the outcomes being studied, although these were discussed in some detail in the accompanying text, even if not linked to outcomes. Since then, however, researchers have sought to understand what elements of health systems are most important and, in particular, the scale and nature of barriers to delivering effective care and how these can be overcome. This work has drawn attention to the importance of context – those characteristics of countries that arise from their history, culture, politics and much else, all of which impact on the delivery of health care.

In summary, much of the early work focused on examining the functioning of particular health services, in terms of structures, processes and outcomes. However, more recent work has begun to acknowledge the crucial role of context. Research has also started to demonstrate the complex pathways between the way services are organised and the outcomes and outputs, and suggests the need for a broader (or what we call ‘systems’) perspective to explain reality.

Page 2

From services to systems: towards a new understanding of health systems research

The landmark World Health Report (WHO 2000) set out a comprehensive framework within which health systems can be studied and managed. It identified certain key functions (stewardship, financing, delivery) and goals (improved health, fairness and responsiveness) (see Session 1). These were further disaggregated in subsequent work, demonstrating the complex relations among inputs that are required to achieve health systems goals. It was established that any ‘system-level’ assessment of a health system should involve examining the roles of multiple ‘building blocks’ in enabling particular outcomes, even if the question of interest referred to only one element of a health system.

Click here for an example [link – on click show hidden text]

[hidden text]

For example, understanding human resources shortages from a system perspective will require understanding of broader governance and institutional arrangements to recruit, train and retain health workers, understanding of the financial resources required, and knowledge of delivery models which affect the supply of health workers.

[/end hidden text]

Stimulated by this report and the heated debate following it, interest in assessing, comparing and evaluating health systems increased rapidly, with a proliferation of frameworks, toolkits,


studies and manuals. Some of these will be considered later in this session. However, funding for empirical work has been scarce, and many of the approaches remain untested.

The publication of the report coincided with other developments leading to increased interest in health systems research (see Session 1, topic 4). Countries at all levels of development took greater interest in how their systems were performing and whether this represented value for money. Health systems are increasingly being recognised as essential to improving population health. A massive transformation in information technology also offers the prospect, although rarely fulfilled, of collecting, storing and processing vast amounts of information that had not previously been possible. There is also interest in learning lessons from countries that are considered to have achieved success in tackling some of these issues.

The end of the 1990s marked the beginning of new global health initiatives. For example, the creation of the Global Fund to fight AIDS, Tuberculosis and Malaria (GFATM) and the Global Alliance for Vaccines and Immunisation (GAVI). Such initiatives recognised the need for what was called ‘health system strengthening’. This was seen as a departure from vertical disease-specific programmes, and the principles of ‘comprehensive and inclusive health care’, requiring ‘all-round system building’, were seen as the way forward.

As a result, the critical role of health systems is now well-recognised and there have been efforts to advance the field of health systems research both conceptually and practically. Organisations such as the European Observatory on Health Systems and Policies and the Alliance for Health Policy and Systems Research (see below) have been instrumental in promoting such research and remain important repositories of analytical work.

Reference linksEuropean Observatory on Health Systems and Policies:www.euro.who.int/en/who-we-are/partners/observatory (accessed October 2013) Alliance for Health Policy and Systems Research:www.who.int/alliance-hpsr/en/ (accessed October 2013)

Page 3

Contemporary interest: ‘systems thinking’

The early phase of health systems research (from mid 1990s to late-2000) created considerable enthusiasm and generated demand for health systems research. Many of the research questions reflected practical concerns of policy-makers and implementers.

In 2007 the WHO published a revised version of its framework and sought to operationalise key dimensions of health systems, seeking better to inform policy and research agendas. The First Symposium for Health Systems Research, held in Montreux, Switzerland, in 2010, sought to create a platform for developing shared concepts and definitions in health systems research. It, and subsequent developments, emphasised the importance of strengthening the methodological and theoretical basis of such research (Sheikh et al. 2011).

What constitutes health systems research and what research questions should it tackle? These questions have been addressed in recent papers (de Savigny and Adam 2009 ; Gilson 2012)


but they reveal a continuing debate over whether health systems research should focus on individual health system components, which can be studied in relation to other sub-systems or ‘building blocks’, or whether the focus should be on health systems as a whole (‘whole-systems’ studies – Balabanova et al. 2013).

There is also an ongoing debate about the best way to evaluate the impact of health system interventions. Bennett et al. (2011) argue that despite an increased perception of a need for it (especially in middle- and low-income countries but also growing in high-income countries), there is a lack of clarity and shared understanding regarding the scientific foundations of health policy and systems research (HPSR). The term ‘HPSR’ builds on earlier, often more technical approaches such as that used in the 2000 World Health Report to emphasise how health systems are embedded within policy and politics. However, as HPSR is a new field of research that draws on a variety of disciplines, it does not yet have a distinct body of theory and methodology.

Page 4

Systems or services?

A key distinction is between health systems research and health services research. Several authors (Bennett et al. 2011; Gilson et al. 2011 ; Gilson 2012; Sheikh et al. 2011) argue that the differences mainly relate to the questions asked. Thus health systems research is seen as particularly suited to answering why and how a certain mix of structures, policies or interventions may have shaped outcomes, but is less useful in demonstrating which intervention may have caused which outcome. The main knowledge paradigms and research questions that are used in the field of health policy and systems research are outlined in Table 6.1.

Click here to see Table 6.1 Key elements of knowledge paradigms as applied in HPSR [on click show hidden content]

[hidden content]

Knowledge paradigm Positivism Critical realism

Relativism (interpretivism/social

constructionism)

Types of questions addressed

Is the policy or intervention (cost-)effective?

What works for whom under which conditions?

How do actors experience and understand different types of interventions and policies?What are the social processes including power relations, influencing actors' understandings and


experiences?

Related disciplinary perspectives

EpidemiologyWelfare economicsPolitical science (rational choice theory)

Policy analysisOrganisational studies

AnthropologySociologyPolitical science (sociological institutionalism)

Key research

approaches and

methods

Deductive: hypothesis-drivenMeasurement through surveys, use of archival and other data recordsStatistical analysisQualitative data collected through, for example, semi-structured interviews and interviewing procedures

Deductive and inductive (theory testing and building)Multiple data collection methods including review of documents, range of interviewing methods, observation

Inductive (maybe theory building and/or testing)Multiple data collection methods including in-depth interviewing (individuals and groups), documentary review but also participant observation or life histories, for example

HPSR articles that

illustrate the

paradigm (see Part 4)

Bjökman & Svensson, 2009

Marchal, Dedzo & Kegels, 2010

Riewpalboon et al.Schiffman 2009Sheikh & Porter 2009

Table 6.1 Key elements of knowledge paradigms as applied in HPSR Source: Gilson (2012)

[/end hidden content]

The relevance of these paradigms to the field of HPSR can be illustrated by looking at the extremely contentious issue of user fees. The different questions all fall within the scope of health systems research but answering them requires insights from different disciplines (see Table 6.2).

Click here to see Table 6.2 Knowledge paradigms: an illustration [link – on click show hidden content]

[hidden content]

Positivist Critical realist and relativist Relativist

Example of What is the impact of Why were user fees How is user fee policy


study questions

user fees on service utilisation and across different groups of patients?

introduced and how was equity conceived? How do out-of- pocket payments interact with other influences on care-seeking?

experienced by those implementing it?

Illustrative policy

implications

Levels at which user fees should be setÂ Which population groups should be exempted from fees

Strengthening the voice of the poor in policy and implementation processes so as to promote more pro-poor policiesShould policy focus on addressing user fees, the key obstacle to utilisation, or would it also be necessary (or even more important) to address other barriers to care-seeking?

Strategies to empower health staff in policy development and implementation processes, so as to ensure that the framing of the policy takes account of their concerns, as a means of strengthening implementation

Table 6.2 Knowledge paradigms: an illustration


Page 5

Definition of HPSR

The Alliance for Health Policy and Systems Research defines HPSR as:

[quote]

an emerging field that seeks to understand and improve how societies organize themselves in achieving collective health goals, and how different actors interact in the policy and implementation processes to contribute to policy outcomes. By nature, it is inter-disciplinary, a blend of economics, sociology, anthropology, political science, public health and epidemiology that together draw a comprehensive picture of how health systems respond and adapt to health policies, and how health policies can shape − and be shaped by − health systems and the broader determinants of health. Alliance for Health Policy and Systems Research (2011)

[/end quote]

Gilson (2012) further sought to distinguish the characteristics of HSPR that allow it to retain a unique identity while drawing on a variety of disciplines (Table 6.3).


Click here to see Table 6.3 What is health policy and systems research (HPSR)?[link – on click show hidden content]

[hidden content]

Defining features of HSPR What HPSR is not

The types of issues it addresses: focus on real-world situations and issues

Multi-disciplinary – distinguished by issues and questions addressed not disciplinary base or set of methods

Focuses on health services but also promotion of health in general

Includes work at global and international levels, as well as national and sub-national levels

Aims to strengthen health systems so that health and society goals can be reached

Promotes work that explicitly seeks to influence policy â“� and applied research that has potential to support health system development

Issues studied include how health care is financed, organised, delivered and used; how health policies are prioritised, developed and implemented; and why some health systems achieve their goals and others don’t

Currently mainly focused on research in low- and middle-income countries â“� but increasingly in high-income countries

Basic scientific research on new pharmaceutical products or medical technologies

Assessing the clinical efficacy and effectiveness of particular treatments or technologies

Measurement of population health profiles and patterns

Research that only focuses on one particular programme or one particular aspect of service delivery without considering any other aspects of the wider health system context

Table 6.3 What is health policy and systems research (HPSR)?


However, some have argued that, in practice, these distinctions are often insufficiently clear (Adam et al. 2012). To bring about a change in health outcomes, it is necessary to intervene in many ways. We can illustrate this with the example of high blood pressure. Obviously, we need researchers in biochemistry and pharmacology to develop safe and effective drugs to treat this condition. This is crucial but it is not health systems research. Then we need health service researchers to study the best way to organise clinics and to make sure patients adhere


to their treatment. But the health systems researcher is interested in factors that can be linked back to national/regional health system-level arrangements (interventions, policies or programmes), such as how you fund the system, which governance systems work best, and how you pay providers.

The next topic will provide an overview of frequently used approaches for assessing health systems.

Page 6

Reflection point

Essential reading Smith P (2002) Measuring health system performance. European Journal of Health Economics, 3, 145–148.

Read the article by Smith and then think back to your answers to the reflection point in the previous topic [link]

Are there any issues that you didn’t think about before and that are raised in this article or the previous section?

Are there any issues that you feel should have been raised but were not?

Think about the situation in low- and middle-income countries: are there any additional issues further to the reading that you can think of – in terms of demand for health systems assessment and the factors that are likely to determine what is possible?

Please post some of your reflections on Moodle [link] and respond to other posts

Question feedback

In low and some middle-income countries the motivations for increased emphasis on assessment and evaluation are largely similar, but there are also additional factors. There is a similar financial impetus as elsewhere, but the trade-offs involved are likely to be even more significant (e.g. competing demands from other sectors such as education).

A substantial proportion of funding for health systems in many low- and middle-income countries is from external funders. For political reasons, funders and governments need to be able to show the benefit of investment in health systems. Funders are increasingly moving from horizontal disease-focused programmes to health system strengthening but need convincing that they are getting value for money. They also need outcomes they can measure progress against (see Session 1). Most projects now have monitoring and evaluation components, and there is pressure to identify what works and what doesn’t, so that successful projects can be rolled out. There is also pressure to measure progress against international targets such as the MDGs.


The increased pressure from some quarters for market-based mechanisms for delivering services, which include components such as the development of complex costing mechanisms and attempts to develop purchaser-provider splits and metrics to capture those aspects of performance that can be measured, has also been important in many countries.

Quality of information available and capacity will be lower in many such countries. This has meant that different approaches have been developed (see Nolte and McKee 2012) and Kruk and Freedman (2008) for a discussion of these issues.

6 Selected approaches for health systems assessment

Page 1

Measuring performance

There is a rapidly increasing interest in assessing how health systems perform. What we mean by ‘performance’ will vary according to the policy and operational goals we are pursuing as well as the values of the society in which we operate.

The World Health Report (WHO 2000) suggests that these definitions can change over time. Earlier work on performance focused on quality and clinical outcomes, against particular inputs and activity. However, more recently, faced with squeezes on resources, there has been greater attention paid to efficiency – i.e. the extent to which resources used by any given health system achieve the system’s objectives (Murray and Frenk 2000). A third element, responsiveness, is also now considered an essential characteristic of a well functioning health system (WHO 2000).

Click here for more information on measuring and ranking health systems goal achievement (Box 6.1) [link – on click show hidden content]

[hidden content]

Box 6.1 Measuring and ranking health systems goal achievement

The World Health Report (WHO 2000) was an important milestone in developing the concepts and methods used in assessing and comparing health systems. Health system performance was measured in terms of improving health, enhancing responsiveness to the expectation of the population (for the first time), and assuring fairness of financial contribution. Murray and Frenk (2000) argued that, by systematically comparing data on the key functions (stewardship, financing, service provision and resource generation), they could explain variations in performance and generate quantitative evidence on the outcomes of health systems, and the causes of these outcomes, going beyond previous assessments that had been seen as ‘purely ideological’.


The report ranked the achievement of countries’ health systems in terms of meeting each key goal, while trying to take into account the resources available. Each country was assessed using existing or newly-generated data; in many cases values were estimated when data were unavailable or judged as unreliable. The authors argued that, even if the basic data were incomplete, the publication of country rankings based on estimates should stimulate greater attention to data-gathering in the future. (Murray and Frenk 2001).

Adapted from WHO (2000: Chapter 2)

Performance assessment usually involves quantitative assessment, which creates many problems. First, it can only assess what can actually be measured, so that many areas where measurement is difficult or culturally and contextually embedded, such as the humanity of care, tend to be excluded. In some cases, this can lead to a failure to value certain aspects of performance which are less tangible, such as a caring and compassionate staff attitude to patients.

Page 2

Reflection point

Have a look at Table 6.4 as an example of the output generated.

Country Overall performance Performance on health level (DALE)

France 1 4

UK 18 24

Germany 25 41

Costa Rica 36 25

Sri Lanka 76 66

USA 37 72

Best performing country France Oman

Worst performing country Sierra Leone Zimbabwe

Table 6.4 Output generated by country

What are your first thoughts?

What do you learn from this – what is the most useful and the least useful fact?

In your opinion, who is the audience for this analysis?

What changes can be expected as a result of this exercise?


What impact do you think this report would have?

Post your reflections on Moodle [link] and respond to other posts

Page 3

What was measured in the World Health Report?

The World Health Report assessed the following goals of a health system.

Click on the links for details [click and reveal interaction - clicking the links reveals the hidden boxes]

Goal: improved health outcomes

The main measure used was disability adjusted life expectancy (DALE):

calculated from the proportion of people surviving to each age, calculated from birth and death rates; prevalence of each type of disability at each age; weight assigned to each type of disability, which may or may not vary with age

readily comparable across populations

used child health to measure distribution of health.

Goal: responsiveness

Measured using key informant survey of 1,791 people in 35 countries. Key informants were asked to rank the systems on seven dimensions divided into two groups:

respect for persons: dignity, autonomy, confidentiality of information

client orientation: prompt attention, access to social support networks, quality of basic amenities, choice of health care provider.

Goal: fair contribution

This was calculated as the ratio of total health contributions to total non-food spending across all households independent of their income, health status or use of the health system. In the fairest system, all households make the same relative contributions. In reality, where health systems are weak, those in poor households usually spend a much higher proportion of their disposable income on health care.

Goal: overall attainment

Countries were ranked according to a number of aggregate scores, including overall attainment. To get an overall level of attainment score, the WHO had to decide a weighting


for different aspects of a health system’s performance. A survey was conducted among 1,006 respondents, half of whom worked for the WHO, and the weighting shown in Table 6.5 was agreed. The weighing given to the different objectives was surprisingly consistent among different groups.

Health (DALE) %

Total 50

Overall or average 25

Distribution or equality 25

Responsiveness %

Total 25

Overall or average 12.5

Distribution or equality 12.5

Fair financial contribution %

Distribution or equality 25Table 6.5 Weighing of different health system goals

Additional goal: efficiency

Although efficiency was not one of the key goals in the framework, it was nevertheless measured as the relationship between the outcomes (DALE and overall performance) and the resources used to produce them. Although comparing the total attainment of Sweden and Uganda would be meaningless given the very different resources available to each, there are lessons to be learned from the experiences of countries with similar levels of health expenditure (e.g. Pakistan and Uganda) but differential outcomes (Uganda has a much higher life expectancy). Figure 6.1 shows that, as expected, higher health expenditure tends to be associated with better health outcomes, with countries spending below US$ 10 per person per year rarely achieving more than 75% of the life expectancy that should be possible. Crucially, there are large variations in performance in countries with low and middle levels of expenditure, showing the scope for health system interventions. Figure 6.1 presents the distribution of overall performance, which shows that different goals can compensate for each other. Countries performing poorly on a health goal may be achieving a higher ranking on responsiveness and fairness. The figure also shows a general concordance between the achievement of different goals (WHO 2000).

Click here to see Figure 6.1 [link – on click reveal hidden figure]


Figure 6.1 Performance on level of health (disability-adjusted life expectancy relative to health expenditure per capita, 191 Member States, 1999 (top)


Overall health system performance (all attainments) relative to health expenditure per capita, 191 Member states 1997 (bottom)Source: World Health Report, WHO 2000, reproduced with permission

Page 4

Criticism of the World Health Report

The report, and specifically the country health systems ranking, caused significant controversy and made media headlines around the world. Some authors criticised the framework and overall approach to health systems assessment. Helms (2000) wrote in the Wall Street Journal that the inclusion of fairness of financial contribution was healthcare à la Karl Marx. Others argued that the use of overall burden of disease ignored the contribution of socioeconomic determinants of health (Navarro 2000 ; Walt and Mills 2001).

Other criticisms focused on the data and the methodology used to generate the aggregate scores. Data were unavailable for 70–80% of countries so the figures used were estimated. The validity of the data on financial protection (Shaw 2002) and responsiveness (Blendon et al. 2001) has been questioned. Almeida et al. 2000 argue that:

Key informants who were used to assess responsiveness came from only 35 countries – even though responsiveness was ranked for 191 countries.

Even in countries where key informants were interviewed, they were not representative.

The measures of health inequality simply looked at the distribution within each population rather than the determinants of distribution. For example, Australia is considered a very equal system as 98% of the population have health outcomes that are quite similar. However 2%, the Aboriginal population, fare much worse. The measure used for fair financing does not reflect a conceptually sound or socially responsible view of fairness and does not differentiate between countries.

Twenty-six of the 32 methodological references cited are non-peer reviewed internal WHO documents and only two of the 32 references are by authors other than those involved in the World Health Report.

The measures of health status used at the time were criticised for undervaluing the lives of certain groups, including disabled people.

Creating indices that combine a number of components can be problematic, for example when two countries achieve the same total score but each performs well or poorly on different measures.

Important methodological limitations and controversies are not acknowledged. Nor are ideological positions which, considering the global impact of the report, could have had a profound impact on how health systems were perceived and assessed.

Finally, the conclusions and rankings produced by the report were considered counterintuitive (the US health system being most responsive), and implied value judgements. Walt and Mills (2001) point out that it is difficult to produce any ranking that is free from values and ideology – as in reality a country’s health system reflects historical and societal values and can only be understood in that context.


Page 5

Activity

Based on what you learned, think about what you think are the key reasons for measuring health performance.

Do you think you will able to measure everything you would like to measure? How? Who do you think would be interested in the results? Does health system performance assessment actually make a difference? Does public reporting of the results make a difference?

Look at the links below and make a list of pros and cons.

New York State’s publication of numbers of operations performed and surgeon- and hospital-specific mortality rates:

Essential readingNew York State Department of Health (2010). Adult Cardiac Surgery in New York State 2005-2007 [online]. Albany NY: NYSDH. Available at:www.health.ny.gov/statistics/diseases/cardiovascular/heart_disease/docs/2005-2007_adult_cardiac_surgery.pdf (accessed October 2013)

The UK Nuffield Trust has recently been asked by the Department of Health (DH) to evaluate whether it would be useful to have a national system to rate hospitals and social care institutions similar to Ofsted (the public regulator monitoring and rating education institutions in the UK).

Read the blog of the director of the Nuffield Trust:

Essential resource:Dixon J (2013). Should there be ‘Ofsted-style’ ratings for health and social care providers? [blog] 22 March. Available at:www.nuffieldtrust.org.uk/blog/should-there-be-ofsted-style-ratings-health-and-social-care-providers (accessed October 2013)

or watch this video:

Essential resource:Nuffield Trust (2013). Jennifer Dixon: ‘Ofsted-style’ ratings for care providers? [video online]. Available at:www.nuffieldtrust.org.uk/talks/videos/jennifer-dixon-ofsted-style-ratings-care-providers (accessed October 2013)

Please post some of your reflections on Moodle and respond to other posts


When you’ve finished, read the feedback

Question feedback:

When considering how you define performance it is important to first think about why you are measuring it and what you hope to achieve. Different reasons may lead to slightly definitions of what performance is and how you would measure it. Accountability is often said to be the key reason for measuring performance. This can be to funders and other stakeholders such as taxpayers, governments, regulators, health care providers, service users, the general public and donors. In this case the objective would be to inform decision-making on resource allocation and organisation. But performance could also be measured for the following interrelated reasons:

To identify areas of poor performance and centres of excellence.

To help patients – and purchasers of health care – select facilities and providers.

To encourage provider behaviour change – for example the introduction of league tables or hospital accreditation (which will be discussed later), which are meant to reward good provider practice and shame poor service providers into improving their behaviour.

To provide epidemiological and other public health data. For example, MMR immunisation rates disaggregated by geographical location and gender, can give public health officials information on who to target.

The experience of publishing surgeons’ individual performance data has been very mixed. In Pennsylvania, 63% of surgeons said they were less willing to operate on high-risk patients. There was also a sudden increase in surgeons reporting the prevalence of other conditions – such as chronic obstructive pulmonary disease – as a way of making their patients appear more seriously ill, thereby ensuring the expected mortality calculated by the risk-adjustment formula (with which their data were being compared) would be higher. Although New York State experienced declines in hospital mortality, similar declines were reported in states where there was no public reporting system in place.

The Nuffield Report argues that the publication of ratings [has] had a positive effect on improving the performance of providers (at least with respect to the indicators included in the rating) and [has] shifted the ‘quality curve’ upwards ( Nuffield Trust 2013). But ratings may also give rise to a number of important negative or perverse effects, such as manipulation of the data and distortion of priorities as attention is focused on aspects of care that are measured relative to those that are not. The more that sanctions are applied to poor ratings, the more this distortion is likely. In health care it is important that a rating system should not be used as a new system of performance management: rather it should dovetail with a more supportive, albeit challenging, programme of improvement. The Nuffield Report concludes that a new system of evaluation must be clear in its purpose, and should include:

increasing accountability to the public, users, commissioners of care, and (for publicly-funded care) to the legislature


enabling choice by users (and their relatives and carers), and by purchasers of publicly-funded care.

A systematic review of the evidence about the impact of publishing patient care performance data on quality of care concluded that evidence is scant, particularly about individual providers and practices … Evidence suggests that publicly releasing performance data stimulates quality improvement activity at the hospital level. The effect of public reporting on effectiveness, safety, and patient-centeredness remains uncertain (Fung et al. 2008).

Page 6

Measuring performance: levels of assessment

As well as considering the purpose of performance measurement, it is important to think about the level at which measurement should take place. This will determine how performance is defined and what is measured.

Performance can be assessed at different levels of decision-making within a health system (Nolte et al. 2005).

A micro-level analysis focuses on the process and outcomes of patient care – for example, assessing and comparing the performance of individual clinicians while taking account of the institutional and cultural context within which they operate, and the roles of individuals in health systems (as users, care providers and managers). In other words, how systems respectively shape and are shaped by their decisions and behaviour (Sheikh et al. 2011).

A meso-level assessment focuses on organisational and managerial interventions, such as assessing and comparing hospitals.

A macro-level assessment involves analysis of the overarching systems architecture (governance, financing, delivery models). Here, health systems studies may include assessing and comparing the performance of health systems in a particular region or assessing the impact of global health architecture.

We’ll now look in turn at each of these levels of assessment in more detail.

Page 7

Micro-level assesment

At the micro-level, health systems research involves assessing the process and outcomes of patient care. It includes studies exploring access to care and its determinants and assessing or comparing the performance of individuals, teams and facilities. This largely falls within what is termed health services research.

The earlier mentioned example of data synthesis reflecting variations in outcomes of care is the register of adult cardiac surgery in New York State.


Reference linkNew York State Department of Health (2010). Adult Cardiac Surgery in New York State 2005-2007 [online]. Albany NY: NYSDH. Available at:www.health.ny.gov/statistics/diseases/cardiovascular/heart_disease/docs/2005-2007_adult_cardiac_surgery.pdf (accessed October 2013)

The register shows the numbers of operations performed and outcomes of surgery for individual hospitals and surgeons. The data do not include information on differences in the severity of patients treated or contextual features at the systems level that may affect the outcomes (e.g. funding and delivery models).

The difficulties of identifying ‘good hospitals’ are illustrated by the seemingly surprising fact that President Clinton chose to have cardiac bypass heart surgery in 2004 in New York’s Presbyterian’s Columbia Centre, which in 2001 had been rated as one of the worst performing hospitals in New York State. The explanation given to Fox News at the time was that it was the hospital where the best doctors saw the most severe cases so its surgeons and nursing staff had the best mix of skills.

Reference linkKehnemui S, Fox News (2004). Doctors: Clinton Heart Surgery a Success. [online] Available at:www.foxnews.com/story/0,2933,131551,00.html (accessed Otober 2013)

Page 8

Meso-level assessment

At the meso level, assessing performance commonly involves a focus on organisations or subdivisions of the health system or hospitals. There are three main measures of performance at the meso-level:

[click and reveal interaction with images – clicking on images reveals hidden content]

process measuresfor example, waiting times, average length of stay

Original image by Ivan Fourie, sourced from Flickr Creative Commons


patient outcomesfor example, mortality, morbidity, responsiveness measured through patient satisfaction surveys

Source of image unknown

hospital characteristics for example, quality of food, cleanliness of toilets, etc.

Photo by Dale Leschnitzer, sourced from Flickr Creative Commons

An example is the Dr Foster project in the UK, which communicates information about the performance of public services to the public to help them make choices about which services to use. Dr Foster produces an efficiency and mortality analysis of each hospital in each area. The hospital mortality is adjusted for case mix, which it argues provides insight into preventable mortality and therefore the quality of care.

Reference linkDr Foster Health Intelligence (2013). Essential reading for smart spending. Hospital guide 2013. [online]. Available at:myhospitalguide.drfosterintelligence.co.uk (accessed October 2013)

The National Health Service (NHS) in the UK also has its own performance-related rating for hospital trusts which feeds into managerial processes.

There have been many criticisms of the use of such standardised hospital mortality rates, which attempt to adjust for differences in the severity of patients’ conditions. A systematic review published in 2007 found no conclusive evidence that hospitals with higher-risk adjusted mortality rates provided poorer quality of care (Pitches et al. 2007). A more recent article, published in 2012, also argued that case-mix adjusted hospital mortality is a poor proxy for preventable mortality (Girling et al. 2012)

Similar initiatives exist in many other European countries.

Click here for more examples [link – on click show hidden content]


[hidden content]

One such example is Sundhedskvalitet in Denmark, which since 2006 has used a system of rating a series of indicators of health care quality provided in hospitals. Reference linkwww.sundhedskvalitet.dk (accessed October 2013)

KiesBeter (‘Choose better’) in the Netherlands is a health portal providing information on hospital facilities, availability of services, specialties, waiting times and a range of quality indictors Reference linkwww.kiesBeter.nl (accessed October 2013)

In Germany the Federal Office for Quality Assurance (BQS) issues an annual national quality report, using data compiled and analysed at national level. Since 2000, all hospitals have been legally required to implement external quality check mechanisms. Hospitals identified as underperforming by the BQS have to explain why, and if necessary show what appropriate action they will undertake to improve performance. Reference linkwww.bqs-qualitaetsreport.de (accessed October 2013)


Page 9

Macro-level assessment

The macro-level of measurement involves attempts to assess and compare health systems and their performance between countries and regions. Such studies are carried out by the WHO and the OECD.

It is necessary to have information about the characteristics of a health system in order to interpret the information. The Health System in Transition (HiT) reports produced by the European Observatory on Health Systems and Health Policy do this, as do the reports of the Asia-Pacific Observatory. The HiT reviews of each country provide a comprehensive description of its health system, including its key elements, as well as major reforms and policy initiatives. The reports cover the countries of the WHO European region, some other OECD countries and some in the Asia-Pacific region, such as Mongolia, Malaysia and Fiji.

Reference linkswww.euro.who.int/en/who-we-are/partners/observatory/health-systems-in-transition-hit-series (accessed October 2013) http://www.wpro.who.int/asia_pacific_observatory/en/ (accessed October 2013)

Page 10

Snapshots, updates and rankings


Snapshots, updates and rankings are commonly used approaches to assess and monitor key health systems functions and indicators, usually over time. They usually involve both data synthesis and analysis.

Click here for some examples [link – on click show hidden content]

[hidden content]

Examples include:

MDG countdown indicators and country profiles (see Box 6.2)

HiT reports (the European Observatory on Health Systems and Policies)

WHO world health development indicators

The World Bank worldwide governance indicators (six dimensions)

Transparency International: corruption indices, ranking


Click on the tabs to see some of the advantages and disadvantages of these approaches

[tabs interaction]

[tab 1]

Advantages

Some advantages of these approaches include:

They often use routine data, which do not need to be collected using expensive and time-consuming methods.

They often involve multiple methods, including key informant interviews, policy reviews and secondary data analysis of routine data.

They often synthesise and present data that is made as comparable as possible across countries, and facilitate analytical comparisons and lesson-learning.

They help to track progress over time. Their production may build capacity and promote national ownership.

Click here for an example [link – on click show hidden content]

[hidden content]

The HiT reports offer a an example of this process whereby national experts are able to draft such reports following a template of health systems assessment with guidance on questions,


type of data required, etc. The authors are able to interpret this data and assess its reliability, and place it in the context of health system changes and the political and social environment. Reference linkwww.euro.who.int/en/who-we-are/partners/observatory/health-systems-in-transition-hit-series/hit-template-2010 (accessed October 2013)


[/end tab 1]

[tab 2]

Disadvantages

Some disadvantages of these approaches are:

The analysis and the data tend to be descriptive they can show what is happening but not why and how it is happening.

The selection of indicators is challenging. It often has to rely on what is already measured instead of what needs to be measured (e.g. essential service coverage rather than need). Existing data may not be up to date.

There are major problems with the quality and coverage of data in some countries – particularly low- and middle-income ones.

There is often insufficient scope for interpretation. There may, for example, be very good reasons why one country’s maternal or infant mortality has improved or not – related to the HIV epidemic or political shifts affecting provision of basic services – but such factors are usually not covered in depth in such studies.

[/end tab 2]

[/end tabs interaction]

Page 11

Box 6.2 Example of a snapshot approach: Countdown to 2015: Tracking Progress in Maternal, Newborn & Child Survival

The Countdown Project, tracking progress on MCH, exemplifies the snapshot approach.

Reference linkwww.countdown2015mnch.org/ (accessed October 2013)

It is a multidisciplinary collaboration of academics, activists, governments, donors, health care organisations and non-governmental organisations (NGOs) that uses country-specific data to track progress towards MDGs 4 and 5 (MCH) in the 75 countries where 95% of all maternal and child deaths occur. The Countdown takes a health systems approach measuring coverage of interventions that are proven to work, but also governance processes, regulations


and policies that facilitate MCH. The aim of these annual snapshots of country’s achievement is to provide relevant and timely data to inform effective systems management and support efforts to reach defined goals (such as the MDGs), and to assist planners, funders and implementers in directing the focus of their attention.

The project generates annual reports and country profiles on all 75 countries it tracks (e.g. www.countdown2015mnch.org/country-profiles/brazil ), as well as undertaking cross-country analysis with a focus on equity and accountability. Key data sets used are demographic health surveys (DHSs) and multiple indicator cluster surveys carried out in the various countries. Coverage of proven realistic interventions, as well as of service delivery platforms, such as antenatal and postnatal care, childbirth and family planning services, is tracked as this information provides the basic platform for delivery of multiple effective interventions to reduce maternal, newborn and child mortality.

To be tracked, an intervention or approach must be:

associated with a valid coverage indicator that is reliable comparable across countries and time nationally representative clear and comprehensible by policy-makers and programme managers available regularly in most Countdown priority countries.

As part of each Countdown reporting cycle, a broad consultative technical review process is carried out to review and update the Countdown indicator list so that it reflects the latest evidence on effective interventions for MCH.

Recognising the need for understanding the macro-level aspects of health system design and the policy environment which influences how these interventions are delivered, 13 indicators have been developed to capture information on the main health systems functions, based around the WHO ‘building blocks’: leadership and governance, health system financing, access to essential medicines, etc. For example, the status of the health information function is assessed by looking at whether there is a system of notification of maternal deaths, while the leadership and governance function is assessed by whether the government has ratified key international policies and has costed implementation plans for maternal, newborn and child health. The policy indicators are based on a biannual survey carried out by WHO’s Department of Maternal, Newborn, Child and Adolescent Health. Information is collected from the national Ministry of Health in each country, but is independently verified by WHO country offices and at least one other United Nations (UN) agency working in that country. Table 6.6 shows a sample of indicators across the health system dimension ( Countdown Working Group 2008).

Data source Internationally established benchmark

Evidence and information

Notification of maternal deaths WHO data 2007 Information systematically


recorded and reported

Leadership and governance

International Code of Marketing of Breastmilk Substitutes enacted

WHO and UNICEF data 2007 Policy systematically adopted/enacted

Maternity Protection Convention 183 ratified ILOLEX database Policy systematically

adopted/enacted

Costed implementation plan(s) for maternal, newborn and child health developed

WHO data 2007 Plan(s) systematically developed and costed

Sevice delivery

Availability of emergency obstetric care services

UNICEF and Averting Maternal Death and Disability (AMDD)

Minimum of four basic emergency obstetric care facilities and one comprehensive obstetric care facility per 500 000 population

Midwives authorised to deliver life-saving interventions WHO data 2007

Policy systematically adopted where profession is recognised/regulated

Integrated Management of Childhood Illness adapted to cover first week of life

WHO data 2007 Policy adopted and systematically implemented

Community health workers authorised to identify and manage pneumonia

WHO and UNICEF community intervention survey

Policy adopted and systematically implemented

Promotion of low osmolarity oral rehydration salts and zinc for management of diarrhoea

WHO data 2007 Policy adopted and systematically implemented

Financing

Per-head total expenditure on health World health statistics 2007 None established

General government expenditure on health World health statistics 2007

No global benchmark, but target of 15% pledged by African leaders

Out-of-pocket expenditure as a proportion of total health expenditure

World health statistics 2007 None established


Health workforce

Density of physicians, nurses and midwives

Global atlas of the health workforce

Minimum 2.5 physicians, nurses, and midwives per 1000 population to meet adequate coverage levels for primary health-care interventions

Table 6.6 Description of data sourcesSource: The Countdown Project

Page 12

Reflection point

Select one of the following Countdown country profiles.

Reference links Chad: www.countdown2015mnch.org/country-profiles/chad (accessed October 2013)

South Africa:www.countdown2015mnch.org/country-profiles/south-africa (accessed October 2013)

Myanmar:www.countdown2015mnch.org/country-profiles/myanmar (accessed October 2013)

Thinking about what you have learnt during this session, comment on what you think are the advantages and disadvantages of the Countdown approach in tracking health system performance in the area of maternal and child health.

What do these profiles tell you about the countries that are being described and what important factors do they not give you information on?

Please post some of your reflections on Moodle and respond to other posts.

Page 13

Toolkits

Toolkits represent another approach that can provide a more fine-tuned health systems assessment than that produced using the snapshot approach, combining primary and secondary data as required. Toolkits provide a comprehensive menu-based list of components of a health system that need to be assessed – as well as a way to measure their performance. Although they often include templates and a range of indicators, they can be used in a flexible manner and be adapted to context. They give a more nuanced picture of what is happening in the health system that is being studied than the snapshots and rankings.


Click here for some examples [link – on click show hidden content]

[hidden content]

The WHO’s health systems toolkit exemplifies this approach. Each function was operationalised by a working group of experts and indicators were identified.Reference linkwww.who.int/healthinfo/statistics/toolkit_hss/en/index.html (accessed October 2013) A final recommendation on what indicators to measure was made based on a trade-off their usefulness and the likely availability of data across countries (see Table 6.7).

Click here to see Table 6.7 [link – on click show hidden table]

Governance Policy index (policies in place)CPIA and CPIA score for health

Expert reviewKey informant surveys

Financing

Total health expenditure per capita % ofÂ gross domestic product (GDP) % government expenditure on healthOut of pocket as % of total health expenditure

NHA, household surveys Public expenditure reviewsExpenditure reporting systems

Workforce

Health care professionals per 10,000 population with distribution within countryHealth training graduates per 100,000

Facility assessmentsAdministrative dataProfessional society databases

Information Health information system performance index Expert review

Medical products

Tracer drug availability in health facilitiesMedian drug price ratio for tracer drugs

Facility assessmentsFacility reporting systems

Table 6.7 Selected health system indicators and sources by health system functionSource: WHO (2010)

Another example is the USAID health system assessment toolkit (Health Systems 20/20, 2012) which includes an extensive menu of questions and indicators that can be adapted to country settings. Reference linkwww.healthsystems2020.org/content/resource/detail/528/ (accessed November 2013)A health systems assessment framework is used and a means to assess each function is provided. Detailed step-by-step implementation guidance is also given.



These approaches also have a number of problems:

As with snapshots, they are insufficiently analytical – they tell you what is happening but not why. While they are focused on pragmatic solutions and operationalising complex phenomena, they do not always a make it possible to capture the interrelationships between indicators or different aspects of health systems. However, the most recent version of the Health Systems 20/20 manual seeks to identify interactions between different functions.

Given that toolkits are based on ad hoc data collection, collecting all the information required is expensive and time-consuming. For example, the Health Systems 20/20 toolkit is very complex and adapting it to a country setting may be a challenge, especially where research capacity is limited. However, the fundamental problem is that toolkits can be prescriptive, encourage box-ticking and collecting excessive amounts of information without sufficient effort to interpret the findings and provide a clear picture of what is going on in a health system and why.

Another limitation is that toolkits also rely a great deal on routine data that are not always reliable, or even available. Even establishing the number of health professionals at one point in time (a basic indicator in the WHO toolkit) may be problematic as governments and professional associations in some countries may not maintain accurate registrations, or cannot establish which health professions are working in the public sector or are still in the country.

Page 14

(Log–) framework-based assessments

The increasing use of managerial tools for organisational performance, coupled with a drive towards greater accountability has led to the proliferation of ‘log-framework’-based assessments. These usually involve measuring inputs (the resources invested), outputs (what was produced – e.g. staff trained, drugs supplied) and impact of the health system. However, they rarely explain why something happened, so there is growing demand for health systems research to help understand what happens to facilitate the transformation of the inputs into outputs. Session 4 [link] introduced a similar framework of inputs–mechanisms–outcomes, while other frameworks have introduced ‘process’ into the equation in an effort to explain what happens in the ‘black box’.

Table 6.8 demonstrates the application of the ‘Common framework for monitoring performance and evaluating progress in the scale-up for better health’ to the issue of rural retention of health workers (Huicho et al. 2010). It implies that impact can be hard to measure in the short term but defines outputs and outcomes (sometimes called process indicators) that are measurable and attributable to successful retention of health workers in rural and isolated areas, such as improved service utilisation and patient satisfaction. Long-term impact can be seen in sustained improvements in service delivery and, ultimately, improved health status.

Inputs Process Outputs Outcomes Impact


Resources:- Newly graduated students- Health workers|- Budget for HRH- Health facilities infrastructure

HRM interventions- Leadership (HRH units)- National HRH plan & policy- Regulatory framework for recruitment and retention- System for performance evaluation- Career management

AttractivityIntentions to come, stay, leave

AvailabilityEffective recruitment

RetentionDuration in serviceReduced absenteeismJob satisfaction

Workforce surveillance

ProductivityService utilisation

ResponsivenessPatient satisfaction

Accessibility

IMPROVED PERFORMANCE & SERVICE DELIVERY

towards

IMPROVED HEALTH STATUS

CONTEXT: social determinants, political situation, economic issues (fiscal space, fiscal decentralisation) individual level factors (marital status, gender)Table 6.8 The conceptual framework for measuring efforts to increase access to health workers in underserved areasSource: Huicho et al. (2010)

Page 15

Multi-method (rapid) appraisal methods using ‘tracer’ conditions

Another approach to health systems assessments at the macro-level involves the use of a ‘tracer condition’. A health problem is selected, ideally one for which those affected can easily be identified and which requires them to interact with many different aspects of the health system. This approach, sometimes called ‘rapid appraisal’, was first used in the 1970s in development studies. Since then, the approach has been adapted to assess complex health systems. The aim is to capture the reality of health care as experienced by users and front-line professionals (both in terms of objective and perceived outcomes). The approach uses a mix of research methods, blending qualitative and quantitative data (Beran et al. 2006 ; Nolte and McKee 2011). In Session 4 the use of diabetes as a tracer to assess the functioning of the health system in Georgia was considered (Balabanova et al. 2009). Tracer conditions typically share a number of characteristics.

Click here to see these (Box 6.3 Characteristics of tracers) [link – on click show hidden content]

[hidden content]

Box 6.3 Characteristics of tracers


Tracers:

have a high enough prevalence to permit collection of adequate data have a known epidemiology have a functional impact, i.e. the condition requires treatment – and without treatment

or inappropriate treatment there is clear functional impairment are relatively easy to define and diagnose require contact at different levels of the system, primary and secondary, specialist

care, etc. require support outside the system (community support) have a natural history which varies with utilisation and effectiveness of health care have available techniques of medical management which are well defined for at least

one of the following: prevention, diagnosis, treatment, rehabilitation.


The analytical process involved in rapid appraisal (when applied to comprehensive systems assessment) usually has three steps.

Click here to see these steps in Table 6.9 Analytical stages and examples of application [link – on click show hidden table]

[hidden content]

Analytical stagesExample: ongoing comparative study of diabetes in five countries in the former

Soviet Union

1. Rapid scanning using a toolkit: mapping and identifying key barriers to care for chronic diseases

Using a health systems framework, mapping problems with access to effective care Access to insulin appears to be a key problem

2. In-depth exploration of key areas, and identifying plausible pathways from inputs to outcomes

Focused study found brand instability, diverse market and lack of trust affecting adherence to prescribed medicationPoor outcomes such as coma associated with access to insulin

3. Interpretation of findings within the political and socio-economic country context

Deregulation of the pharmaceutical sector after the end of the former Soviet Union. Procurement systems do not allow for brand stability. Lack of attention to patient support within the health system

Table 6.9 Analytical stages and examples of application



Starting with a broad scan, the approach then focuses on the key problems that patients and frontline professionals experience, generating hypotheses for further investigation. It is a practical approach for settings where relevant information may be incomplete and can generate timely insights for policy. It offers an ‘overall health systems perspective’, links the micro-level (patient outcomes and experience) with the meso- and macro-levels of policy design, and so helps to identify possible causes of observed outcomes.

The challenges involved should not be underestimated. There is a danger that the analysis remains at the first level, simply producing a situation analysis with limited explanatory power. The approach is dependent on the interpretative skills of the researchers (see the comment by the founder of Dr Foster on the critical role of judgement, reference below), and is generally labour-intensive.

Reference link www.telegraph.co.uk/health/healthnews/9848414/The-frightening-truth-NHS-managers-are-incentivised-to-ignore-problems.html (accessed October 2013)

Table 6.10 demonstrates the application of this approach to diabetes, a complex chronic condition requiring contact with many aspects of the health system over a prolonged period. It is therefore a useful marker for health system performance in a country, or across several countries.

Click here to see Table 6.10 Assessing health systems performance through the lens of diabetes [link – on click show hidden table]

[hidden content]

Characteristic of the condition Effective care requires… Assessing health systems

performance

Lifelong and requires prolonged treatment Patients often have co-morbidities (i.e. other illnesses or conditions alongside diabetes such as high blood pressure) and/or complicationsInvolve a succession of contacts, access to a variety of specialist skills at different levels of the systemAt times diabetes can be managed at a primary health care (PHC) level â“� but there are also times when a patient will need specialist care â“�

Rapid diagnosticsEnsuring adherence to treatmentContinuous care and follow-upSupport for lifestyle changesReliable drug supplyPatient-focused approaches catering for multiple conditions and needs (e.g. PHC)Management at PHC level with quick access to specialist/inpatient careStrong linkages between different episodes of care and services

Outcome indicatorsMortality from diabetes and complicationsRate of acute complications â“� i.e. coma, limb amputations, blindness, that could be prevented through health system interventionsCases are easily identified and can be followed up over time but are dependent on routine systemsOutput indicatorsAccess to or use of careAppropriately trained staffGood pharmaceutical supply chain


and probably hospitalisationPotentially life-threatening consequences if not effectively controlled

(e.g. referrals, exchange of information between providers)

Uninterrupted use of insulin and blood sugar-lowering drugsSelf-control of blood sugar using a glucometerMore difficult to measure routinely, dependence on ad hoc surveysProcess IndicatorsEffective team work, linkages, communication, self managementDifficult to assess but crucial for health system responsiveness, user experience and outcomes

Table 6.10. Assessing health system performance through the lens of diabetesA more detailed discussion on the use of diabetes as a tracer can be found in Nolte and McKee (2012).


Page 16

Theory-based evaluation

In recent years there has been growing attention paid to the need to enhance the rigour of health systems evaluation, ensuring it builds on theories developed within appropriate disciplines. While the ideal is the randomised controlled trial, which is possible in health systems research (an example was the RAND Health Insurance experiment in the 1980s), the large scale of such trials, the cost, and the methodological challenges mean that they are very rare. It may also be possible to use quasi-randomised methods, such as the evaluation of the expansion of social security in Mexico, which exploited the fact that it could not be expanded in all Mexican states at once. However, even when randomised or quasi-randomised trials are conducted, the studies may not capture the complex reality of health systems embedded in political and cultural systems.

This has stimulated interest in theory-driven approaches to inquiry. Theory-based research is concerned with systematically exploring questions such as what works in which conditions for whom, why, and under what conditions, and developing ‘middle range theories’ (MRTs) (theories encompassing both what is known in social sciences and what people ‘know’ happens in reality).

Page 17

Theory-based evaluation: the realist evaluation approachÂ


The realist evaluation approach is one of the more recent applications of theory-based approaches in health systems research (Pawson 2006 ; Pawson and Tilley 1997). The approach involves asking ‘What works in which conditions for whom?’ rather than just asking ‘Does it work?’. The objectives are to help explain casual links between intervention and outcomes and increase the generalisability of findings. Key to this approach is ‘opening the black box between intervention and outcome’ and identifying ‘causal chains’ from planned intervention to actual outcome (Marchal et al. 2010). This approach is useful for assessing complex interventions where there are multiple intervening factors (both within the way the intervention is implemented and within the broader context), and when the assessment is within a cycle of learning and innovation. The choice of methods is determined according to the research question.

Theory-based evaluations often use case studies, although these may be used under other approaches for health systems research. They allow exploration of the phenomenon of interest and the interaction between the phenomenon and the context in which it occurs. Case studies can help in documenting ‘dynamic and complex situations where multiple, interacting variables may act upon intervention and outcome’. They can help to develop and unpack theories (Marchal et al. 2010).

Two examples of using realist review approaches in health system reviews are summarised in Box 6.4.

Click here to see Box 6.4 [link – on click show hidden content]

[hidden content – box 6.4]

Box 6.4 Two examples of realist review approaches

Reference link:Marchal B, Dedzo M, Kegels G (2010). A realist evaluation of the management of a well-performing regional hospital in Ghana. BMC Health Serv Res 10, 24.

This was one of the earliest examples to apply the realist evaluation approach in health systems research. It explored why a particular hospital in Ghana was performing well – and what generalisable lessons could be learnt from it. The study first sought to formulate an MRT, i.e. how the intervention leads to the desired effect and in what conditions. The MRT was developed through preliminary discussions with key informants and a literature review of human resource management and hospital management theories and literature. A case study involved qualitative and quantitative data collection and analysis. Findings were then fed back to the management team and a final analysis carried out incorporating their responses and feedback – as well as relating this back to the MRT.

The study found that it is possible to implement high commitment management practices in LMIC [low- and middle-income countries] countries and that these are perceived as being relevant by the health workers. It also found that through a well-balanced bundle of Human Resource Management practices, management teams can stimulate organisational commitment and an organisational culture of excellence.


Reference link:Ssengooba F, McPake B, Palmer N (2012). Why performance-based contracting failed in Uganda – an‘open-box’ evaluation of a complex health system intervention. Soc Sci Med, 75(2), 377–383.

The authors reiterate the core principle of realist evaluation, which is that the participants’ ‘reasoning and resources’ are what enables the programme to ‘work’ for particular people in a particular context. Programmes that are replicated cannot be guaranteed to produce the same effect, but an understanding of how mechanisms work is transferrable.

The authors conducted a theory-based case study to evaluate how and why performance-based contracting failed to achieve its objectives in Uganda. The study demonstrated that many of the problems experienced in implementation of the programme were due to poor design. When problems occurred, hasty adaptations were developed, meaning that the intervention implemented in the district was different from what had been envisaged at the design stage. Thus, not enough attention was given to the selection of service targets for the health centres, and when the ones chosen proved to be a poor choice they were already ‘locked in’. Performance auditors struggled to get to grips with what they were auditing and this meant that the results of the audits were not always valid. Importantly: financial shortfalls led to delays, short-cuts and uncertainty about the size and payments of bonuses.

A key lesson learnt was that performance-based contracting cannot be implemented hastily and cheaply. It is important that local institutional and technical capacities of implementers are strengthened before the implementation, and that the response of multiple actors needs to be carefully considered. Complex adaptive system and expectancy theory were both useful for explaining the failure of this policy.


For an example of a practical application of realist review methods in high income settings read: Clark et al. 2005.

Page 18

Historical case studies

A major challenge facing health systems researchers is to attribute outcomes to particular inputs. There are many reasons for this. First, there are complex causal chains, whereby one thing leads to another until the final outcome is achieved, is often long and takes many turns. Second, people take different lengths of time to respond to new circumstances and organizations vary in their ability to adapt. This means that they have variable lag periods between introducing a new policy and it having an effect. As health systems are open, in other words they are affected by many things going on in their environments which may themselves change during these lag periods, it can be very difficult to know what caused what. Third, health systems, like all complex systems, have feedback loops, both positive and negative. Positive feedback occurs when you start something off but people see the benefits and take it up with even greater enthusiasm, or adapt the ideas in other aspects of their work. Negative feedback is when they see think that it is not working and react against


it. Fourth, health systems exhibit path dependency. This means that the way a health system will react to a policy will depend crucially on the starting conditions, especially the distribution of power. The same policies may have very different effects in one country where the medical profession is very powerful and another where it is weak (see Session 1 [link]).

Case studies often provide only a snapshot of the system, making it impossible to assess how things became the way they are. This has given rise to the historical (longitudinal) case study approach. By tracing a course of the events, asking why and how certain decisions were made and certain policies were implemented, it makes it possible to track changes over the long term and identify plausible associations between them and the goals of the health system, while considering the influence of changing contexts.

Click here to see an example, the ‘Good Health at Low Cost 25 Years On’ project [link – on click show hidden content]

[hidden content]

This approach was used in the ‘Good Health at Low Cost 25 Years On’ project that was referred to earlier in the module (see Session 1, Conceptual frameworks for understanding health systems: defining functions and goals, Balabanova et al. 2013). It explored why some countries achieve health outcomes that are better than could be expected at their level of income. A conceptual framework was used to explore multiple interacting factors, drawing on different methodologies that combined analysis of secondary data, policy documents, semi-structured interviews and focus groups with policy-makers, implementers and providers. The study sought to:

construct rich analytical case studies tracing pathways to good health over long periods of time

develop a conceptual framework enabling analysis of the interplay between different factors (national/global; public/private; health/non-health)


Figure 6.2 Conceptual framework Source: Balabanova et al. 2013

recognise the path dependency inherent in health systems development (in other words, the consequences of every country’s unique experience)

establish plausible relationships between causes and effects

recognise recurring patterns among countries

create opportunities to generalise the findings using:

the same approaches in different settings

different approaches in similar settings.


Page 19

Monitoring and evaluation: the case of impact evaluation

Given increasing interest in understanding whether health systems deliver their intended outcomes, monitoring and evaluation (M&E) are assuming a key role. In recent years they have become incorporated into routine management systems and project cycles. One factor underlying this is the drive for greater accountability. This module does not go into detail but instead provides a short introduction to the subject area.

M&E reflects the desire to measure the effect of specific policies and interventions. However, given the considerable complexities of health systems discussed so far in this module, and in


particular the difficulty in attributing an effect to a particular cause, there is an understandable caution in claiming that there is a cause and effect relationship.

Yet, despite this, there are several methods that explicitly seek to establish impact. This section will consider briefly the use of impact evaluation in health systems research.

Page 20

The International Initiative for Impact Evaluation (3ie), an organisation seeking to promote independent and rigorous research and ensure that evidence informs policy states that:

[quote]

High-quality impact evaluations measure the net change in outcomes amongst a particular group, or groups, of people that can be attributed to a specific program using the best methodology available, feasible and appropriate to the evaluation question(s) being investigated and to the specific context. 3ie

[/end quote]

Reference linkwww.3ieimpact.org/ (accessed October 2013)

Despite recent interest in the use of mixed and qualitative research methods, this body of work draws primarily on quantitative designs. The approach prioritises experimental methods, such as randomised controlled trials, or where these do not exist, quasi-experimental methods, such as interrupted time series analysis. While these approaches are seen as useful in assessing the effectiveness of certain types of well-defined interventions with a small number of outcomes, they have been criticised for decontextualising health systems. They are often less useful in evaluation of complex interventions where there are multiple and interactive causal factors affecting the observed effects.

For example, if particular research sites are demonstrating clear impact of programme interventions, this may mean that there are a set of local conditions that can explain why and how a programme succeeds. It may not be possible to randomise these conditions.

Sheikh et al. (2011) note that:

[quote]

when there is a change in policy at the national level, there may be no obvious group against which change can be assessed, nor the opportunity to randomise units to intervention or control group. Even where it is possible to introduce variation in policy at the local level, reliance on randomised methods to rule out confounders in the measurement of impact may lead to a neglect of understanding of the specific elements of the context that are responsible for programme success or failure. For these interventions, it would seem wise to admit a


wider variety of study designs for examining and interpreting programme impact, and for generating knowledge that can be generalised to other contexts.

[/end quote]

The following two evaluations (in a high-income and in a middle income country) are examples of well-designed impact evaluations explicitly informing implementation: Sutton et al. 2012 and King et al. 2009.

Page 21

Process evaluation

A different approach is process evaluation ( Hargreaves et al. 2010), for example assessing whether, in the delivery of interventions in real life, they were delivered as planned (‘fidelity’), whether they are likely to be feasible, and whether they are acceptable and accessible to their intended clients. Process evaluation may be particularly useful where the policies or programmes elicit responses or constraints that have not been envisaged; thus unpacking unintended consequences.

Recognising context

However, both impact and process evaluation increasingly emphasise the need to take account of context. Understanding the features of context in particular may help to decide whether and how a programme may be replicated. It may also help to interpret evaluation findings and inform real-life decisions: impact evaluations offer clear policy messages based on a deep understanding of context and implementation (3ie).

7 Challenges in assessing health systems

Page 1

Conceptual and practical challenges

As noted earlier in the session, health systems research faces a range of conceptual and practical challenges.

First, it should be re-emphasised that it is conceptually challenging to define the boundaries of a health system is, and to clearly attribute health outcomes to health system activities. The paradox is that outcomes are often influenced by factors outside the health systems but are considered indicative of health system performance. A reduction in child mortality can come about through improvements in maternal education, access to sanitation, food security and a range of other factors that are outside the conventional health system boundaries. Even elimination of polio, which is clearly linked to essential health services, such as immunisation, is influenced by social values and political tensions.


Watch the video with Heidi Larson about the impact of public confidence on vaccines coverage:

Essential viewing:NewsMedicalDotNet (2013). Vaccine confidence: an interview with Dr Heidi Larson [video online]. Available at:www.youtube.com/watch?v=MMiQpi1c9a4 (accessed October 2013)

A simple conclusion ‘that poor outcomes are because health systems fail’ can be misleading. Other concerns include:

What do we (need to) measure? Can we measure impact? What is good quality evidence? How do we move from attribution to ‘contribution’ and plausible linkages? Not all outcomes that we value are measureable; often we measure what it is

feasible to measure. Indicators that are measured often, improve.

How do we manage complexity and uncertainty? Definitions vary between countries (e.g. a ‘nurse’ does a completely different

job role in different countries requiring different levels of training). Access to care may require multiple contacts and complex pathways. Outcomes are determined by different care components. Multiple factors are at play, including context. How do we account for chance events, charismatic individuals, historical

precedents (such as political change leading to health system building)? How can we undertake multi-method evaluations: combining different types of

evidence?

Page 2

Practical challenges

Apart from that there are also practical difficulties in countries at all income levels:

Scarce or poor quality data but demand for improved health system metrics (national/ international).

Lack of capacity or resources to collect and analyse data. Lack of capacity or political will to use data to inform policy and practice. Limited investment in information technologies. Pluralistic health sectors – state/non-state organisations, with different roles. Tensions between agencies commissioning and undertaking the assessment. Lack of or inappropriate time to complete an assessment.

Page 3

Reflection point


As we noted earlier in the session, government ministers, purchasers of health care and members of the public want to know different things.

If you have a sick elderly parent who suffers from dementia, what information would you want or ask for?

Question feedback:

You might be interested in issues such as the qualifications of the staff and staff-patient ratios (especially at night and over weekends). You would want to know if the staff will be kind and polite to your parent, whether they will help to feed them and attend them if the bell is rung. Will the ward and toilets be clean? Other questions might include when the visiting hours are, how easy is it to park at the hospital and how much it costs.

[show with previous question feedback]

What if you were a government minister trying to decide whether a child heart surgery unit should stay open? What questions would you ask?

Question feedback:

You would be interested in the outcomes of the unit (mortality, major complications, operative success), the cost of running it, the distance to other units, the demand for services in the area and how much support the hospital has among local politicians and the community.

Page 4

Unintended consequences

It is now recognised that health systems assessments should aim to measure not only the effect of planned health systems changes but also capture the unintended consequences (de Savigny and Adam 2009). In 1863, Florence Nightingale observed the problem of ‘creative reporting’ (quoted in Spiegelhalter 1999):

[quote]

We have known incurable cases discharged from one hospital, to which the deaths ought to have been accounted and received into another hospital, to die there in a day or two after admission, thereby lowering the mortality rate of the first at the expense of the second.

[/end quote]

A classic example of this from the UK was when hospitals had to meet a target of people not waiting for treatment on trolleys in emergency departments for longer than four hours. Wheels were removed from trolleys or they were designated as beds on wheels, while corridors and treatment rooms were called ‘pre-admission units’, all so that hospitals could


meet their targets (Watts and Donnelly 2012). A narrow focus on meeting targets may obstruct efforts to improve the quality of care as important things that are harder to measure get neglected.

Another example is how abolition of user fees in Africa had the unintended consequence of a breakdown of services, with ‘free care’ no longer available in primary health care as providers lost their often modest benefits. Thus policies have to be designed with an intention to prevent or quickly respond to unintended consequences (McPake et al. 2011).

Page 5

Activity

Read the following :

Essential readingTaylor R (2013). The frightening truth: NHS managers are incentivised to ignore problems. Telegraph, 4 February 2013 [online]. Available at:www.telegraph.co.uk/health/healthnews/9848414/The-frightening-truth-NHS-managers-are-incentivised-to-ignore-problems.html (accessed October 2013)

Thinking back on what you have learnt in this session, if you were a head of a regional health service, what would your response to this article be?

If you were asked to design a health systems assessment which would ultimately become part of a routine data collection and reporting system, what would you suggest?

Which areas would you measure, what would be the goals and indicators?

How would you avoid a narrow emphasis on a few measurable targets and promote interest in interpretation and understanding the causes?

Write down the strategy and the operational details, considering why you are measuring these aspects. Are you measuring the right things, is it feasible, who are the users? Think about potential data qualities and proxies.

What incentives could you create to avoid unintended consequences?

Please post some of your reflections on Moodle [link] and comment on other posts.

8 Doing health systems research

Page 1

Taking stock of values and power


Health system assessments require making judgements, many of which are based on values. Before initiating an assessment of a health system or some of its parts, we first have to have an idea about what a good health system should look like.

What are the societal expectations , history and belief systems that influence the shape of the health system?

Should it aim at providing the most advanced treatments for a few? Or should it be underpinned by collectivist values emphasizing fairness for all?

The answers to these questions will determine what is being measured and how.

The second question is who is doing the assessment. This is often the person who is responsible for the whole system or the particular sub-system under study (in a position of power), independent regulators, international organisations or public representatives. Each of these actors faces different constraints and is interested in particular outcomes. Additionally, some of the research approaches (e.g. realist evaluation) may influence the design and implementation of a policy or intervention and thus add to the complexity.

The third question is what is the intended use of the assessment? Is it going to be used to improve practice, empower providers, for political reasons or for a mix of all of these?

The answers to these three questions will inevitably shape the study question and design of any health system assessment. Moreover, it is important that these considerations are made explicit to research teams and intended users.

Page 2

How to make a start?

The ‘plan’ below is an example of how we can start planning a health systems assessment.

[click and reveal interaction – reveal content when each link is clicked]

Research questionAsk: What is the intellectual puzzle we would like to solve?

For example, identify what obstacles confront both people with diabetes and health professionals providing care and how they seek to overcome them.

Conceptual framework Ask: What theories or conceptual frameworks could underpin our research and help explain our findings?

An MRT: how do we get from A to B?

Realist evaluation: what works for whom and in what circumstances?


Health systems framework?

Research design Ask: How do we best go about resolving this puzzle?

Documentary review or historical analysis?

Quantitative analysis (primary or routine data)?

Qualitative study design?

Mixed methods (integrating quantitative surveys)?

Data mapping Ask: What information is already available?

Published and ‘grey’ literature?

Routine statistics (national and facility-level)?

Legal, regulatory and policy documents?

Existing surveys?

Site selection Ask: What geographical or socioeconomic setting are we most interested in?

Regions with different health outcomes/infrastructure (e.g. urban vs. rural)?

Populations with known or assumed differences in health care utilisation?

Levels of care within the same health system: primary, secondary, tertiary?

Method selection Ask: What methods are best suited to answering the research question?

Verbal autopsies?

Interviews with key stakeholders, health professionals, patients, relatives?

Focus group discussions with health professionals, patients, relatives?

Direct observation: facilities?

Stakeholder analysis?

Plan method triangulation and validation of findings.


Research team Ask: Who is best suited to carrying out the research (e.g. skills and experience)?

Health system expert: to advise the team at key study stages related to the local context?

Project coordinator: to be in charge of the study logistics and oversee the study process?

Health system analyst: to advise on study process, data collection and analysis?

Researchers: to carry out data collection, analysis and write-up?

Time frame Ask: How long do we give ourselves to collect data? What is feasible?

Trade-offs in terms of scope/depth?

Budget/team/time constraints?

User demand?

Analysis Ask: How do we make sure that we can analyse all the data we have collected?

Iterative research process and analysis till saturation is reached?

Adequate staff/time/budget allocated until the end of the project?

Dissemination Ask: How do we ensure that our findings inform practice and contribute to health systems policy and practice?

Identify audience, organise stakeholder meetings?

Identify context-appropriate outputs: papers, policy briefs, press releases?

Use social media, press conferences, film?

9 Integrating activity

Page 1

Activity

Design a country-level health systems performance assessment exercise


Choose Option 1 (suitable for countries at all income levels) or Option 2, for middle-/low- income settings.

[tabs interaction]

[tab 1]

Option 1: Countries of all income levels

Choose a country you are familiar with or you want to learn about.

You work in the Ministry of Health’s Monitoring & Evaluation Unit and you are aware of particular problems in outcomes from a particular disease or coverage. You can use the examples below or choose your own.

Click here to see the examples [link – on click show hidden content]

[hidden content]

Example of objectives of assessment:

to reduce birth complications and Caesarean rates

to increase the rate of facility delivery

to improve uptake of antenatal care

to reduce under-5 mortality

to improve MMR coverage

to effectively control hypertension at PHC/community level.


You would like to commission a health systems assessment in this area. First you develop an assessment framework that can inform the terms of reference and invite independent assessors.

In developing the assessment framework, you may consider some of the following:

Which dimensions should be considered (e.g. effectiveness, responsiveness, efficiency, equity)?

How should ‘progress’ be defined? Should the framework specify (sub-)targets and if so what should these look like (quantitative/qualitative)?

At which level should assessment take place (micro/meso/macro)?


Which indicators should be used to track progress? What are their strengths and weaknesses?

What is the likely data availability, from what sources, and what should be done to verify/ensure data quality?

What is the potential for misinterpreting performance data? Or any other data related to health systems functioning?

Should findings be made publicly available and to whom? Is there a risk of unintended negative consequences if performance data (or any other health systems data) are made publicly available and how might any effects be mitigated?

Now draw up a list of criteria against which the merits of the proposals will be judged.

Draw on the readings above and the Recommended Reading at the end of this session and look up any other alternative approaches for health systems assessment that may be applicable.

Please post some of your reflections on Moodle [link] and comment on other posts

[/end tab 1]

[tab 2]

Option 2: Middle- or low-income countries

Read the background describing a hypothetical country.

Click here to reveal the background [link – on click show hidden content]

[hidden content]

Background Mountainland is a low-income country of approximately 25 million people with a third living in urban areas. It has recently emerged from several years of conflict. Although it has achieved considerable levels of economic growth since then, averaging 7% over the past five years, poverty is still widespread and the economy remains highly dependent on foreign aid. Population health is characterised by high levels of inequalities, with the under-5 mortality rate ranging between 120 among the richest fifth of the population and 280 among the poorest fifth. Maternal mortality is also high, currently estimated at about 450 per 100,000 live births. This disease burden is further augmented by a relatively high adult HIV prevalence rate, with an estimated 15.5% of those aged 15–49 living with HIV/AIDS. Less than 50% of the population has access to basic health services, mainly because of a shortage of trained health care workers. The government has committed to improving maternal and child health and is aiming to reduce under-5 mortality to 110 per 100 live births and to halve maternal mortality by 2015. It has made substantial investments, with the help of international donors, into improving access to basic health services for all by training health care workers. It has also invested into improving the physical infrastructure and maintenance. One other key area of investment has


been targeting capacity-building in public service management so as to improve public financial management and combat corruption. However, despite the commitment and the large investments made, the government is conscious that there remain considerable weaknesses in the system that pose an obstacle towards achieving the goals in MCH set out above. The Minister of Health has been tasked with developing a comprehensive performance assessment framework to enable tracking of progress in MCH that also enables identifying potential threats to progress at an early stage.


You work in the Ministry of Health’s Monitoring & Evaluation Unit and are asked to prepare and M&E evaluation to support the new health strategy, to be enacted shortly.

You have to identify the key health priorities and indicators, and plan how these will be assessed. Then you need to design a baseline assessment (based on primary research or existing data) listing clearly the rationale, objectives, methods and capacity required to undertake the assessment.

In developing the assessment framework, you may consider some of the following:

Which dimensions should be considered (e.g. effectiveness, responsiveness, efficiency, equity)?

How should ‘progress’ be defined? Should the framework specify (sub-)targets and if so what should these look like (quantitative/qualitative)?

At which level should assessment take place (micro/meso/macro)? Which indicators should be used to track progress? What are their strengths and

weaknesses? What is the likely data availability, from what sources, and what should be done to

verify/ensure data quality? What is the potential for misinterpreting performance data? Or any other data related

to health systems functioning? Should findings be made publicly available and to whom? Is there a risk of

unintended negative consequences if performance data (or any other health systems data) are made publicly available and how might any effects be mitigated?

Draw on the readings below and the recommended reading at the end of this session, along with any other alternative approaches for health systems assessment that may be applicable.

Please post some of your reflections on Moodle [link] and comment on other posts.

Essential reading:Kruk ME & Freedman LP (2008). Assessing health system performance in developing countries: a review of the literature. Health Policy 85: 263–276.

Balabanova D, McKee M and Mills A (eds) (2011). ‘Good Health at Low Cost’ 25 Years On. What Makes a Successful Health System? [online] London: London School of Hygiene & Tropical Medicine (Chapter 2). Available from:ghlc.lshtm.ac.uk/files/2011/10/GHLC-book_Chapter-2.pdf (accessed October 2013).


Arah OA, Westert GP, Hurst J. and Klazinga NS (2006). A conceptual framework for the OECD Health Care Quality Indicators Project. Int J Qual Health Care, 18(Suppl 1), 5–13.

10 Summary

Page 1

This session has provided an overview of the emerging field of health systems research due to demand and concerns from practitioners, policy-makers and users. Demands for health system assessments have increased in recent years, partly due to a gradual shift in funding patterns – from investment in vertical disease-focused programmes, to horizontal systems aiming to strengthen health systems as a whole – and due to a need to establish the impact of this investment. There is also pressure to monitor progress related to internationally agreed developmental targets in order to inform political agendas.

Within the new field of health systems research, important questions asked include what you are trying to measure, why you are trying to measure it, and how you interpret the data you collect. The session explored a range of widely used approaches to health systems assessment, used in a variety of settings, and discussed their advantages and disadvantages. Conceptually, there is still a lot of confusion and debate about how to measure impact, therefore claims about doing so have to be carefully considered. Theory- and framework-based assessments, multi-method evaluations and historical case studies are a useful starting point in demonstrating ‘contribution’ and are able, to some extent, to show plausible associations between cause and effect.

Rather than asking ‘Does the intervention or policy work?’, health systems research asks ‘What works, under what conditions, for whom?’. Multi-method assessments are increasingly the norm, along with asking policy-relevant questions about ‘real-life’ problems, and ensuring that research findings inform practice. Placing the findings within the context is key, however there is an increasing trend to engage in comparative studies identifying lessons learned that reflect recognisable patters across countries and inform global processes but also national-level policies.

Appropriateness to context and the feasibility of each approach often have to be considered together before decisions are made. There are many technical challenges – for example, evaluating ongoing interventions/policy processes where there is no baseline and distinguishing the impact of health systems from other factors outside those systems. While the research question should inform the methods, it is important to make a judgement as to the appropriateness and feasibility of those methods early on. Identifying health systems bottlenecks and the solutions to them requires two different types of research but they are often implicitly mixed.

Health systems research in low-income countries brings its own practical challenges due to scarce or poor quality data, and a lack of capacity to collect, analyse and use data. In countries of all income levels, disclosing study findings may create its own unintended consequences.


Despite a promising start, further developing the field will depend on tackling the main conceptual and methodological challenges that underlie our understanding of, and approaches to, the measurement of how well systems work.

11 References

Page 1

Essential reading (session)

Nolte E, and McKee M. Measuring and evaluating performance. pp. 49-77. In Smith R, and Hanson K (eds). (2012) Health systems in low and middle-income countries. An economic and policy perspective. Oxford: Oxford University Press.

Adam, T., J. Hsu, et al. (2012). Evaluating health systems strengthening interventions in low-income and middle-income countries: are we asking the right questions? Health Policy Plan 27 Suppl 4: iv9-19.

Nolte E, McKee M and Wait S (2005). Research on health, health system and service evaluation. In Handbook of Health Research Methods: Investigation, Measurement and Analysis, pp. 12-34, ed. A Bowling and S Ebrahim. Maidenhead: Open University Press.

Kruk ME, Freedman LP (2008). Assessing health system performance in developing countries: A review of the literature. Health Policy; 85:263-276.

Smith P, Mossialos E, Papanicolas I (2008). Performance measurement for health system improvement: experiences, challenges and prospects. Copenhagen: World Health Organization on behalf of the European Observatory on Health Systems and Policies,. http://www.euro.who.int/__data/assets/pdf_file/0003/135975/E94887_Part_V.pdf (accessed May 2014) [read one chapter]

Essential reading (activities)

Arah OA, Westert GP, Hurst J. and Klazinga NS (2006). A conceptual framework for the OECD Health Care Quality Indicators Project. Int J Qual Health Care, 18(Suppl 1), 5â“13.�

Balabanova D, McKee M and Mills A (eds) (2011). â˜Good Health at Low Costâ� ™� 25 Years On. What Makes a Successful Health System? [online] London: London School of Hygiene & Tropical Medicine (Chapter 2, pp. 11-45) Available from: ghlc.lshtm.ac.uk/files/2011/10/GHLC-book_Chapter-2.pdf (accessed May 2014).

Kruk ME & Freedman LP (2008). Assessing health system performance in developing countries: A review of the literature. Health Policy 85: 263-276.

New York State Department of Health (2010). Adult Cardiac Surgery in New York State 2005-2007 [online]. Albany NY: NYSDH. Available at:www.health.ny.gov/statistics/diseases/cardiovascular


/heart_disease/docs/2005-2007_adult_cardiac_surgery.pdf (accessed May 2014) [have a look at a few of the results tables]

Smith P (2002) Measuring health system performance. European Journal of Health Economics, 3, 145â“148.�

Taylor R (2013). The frightening truth: NHS managers are incentivised to ignore problems. Telegraph, 4 February 2013 [online]. Available at:www.telegraph.co.uk/health/healthnews/9848414/The-frightening-truth-NHS-managers-are-incentivised-to-ignore-problems.html (accessed May 2014)

Other essential resources

Countdown to 2015 (2013). Chad Maternal and Child Health Data, [online]. Available at:http://www.countdown2015mnch.org/country-profiles/chad (accessed October 2013)

Countdown to 2015 (2013). Myanmar Maternal and Child Health Data, [online]. Available at:http://www.countdown2015mnch.org/country-profiles/myanmar (accessed October 2013)

Countdown to 2015 (2013). South Africa Maternal and Child Health Data, [online]. Available at:http://www.countdown2015mnch.org/country-profiles/south-africa (accessed October 2013)

Dixon J (2013). Should there be ‘Ofsted-style’ ratings for health and social care providers? [blog] 22 March. Available at:www.nuffieldtrust.org.uk/blog/should-there-be-ofsted-style-ratings-health-and-social-care-providers (accessed October 2013)

NewsMedicalDotNet (2013). Vaccine confidence: an interview with Dr Heidi Larson [video online]. Available at:www.youtube.com/watch?v=MMiQpi1c9a4 (accessed October 2013)

Nuffield Trust (2013). Jennifer Dixon: ‘Ofsted-style’ ratings for care providers? [video online]. Available at:www.nuffieldtrust.org.uk/talks/videos/jennifer-dixon-ofsted-style-ratings-care-providers (accessed October 2013)

Recommended reading

The Recommended reading includes materials to draw upon when undertaking your assignment or in self study, according to your interests (this may be 2-3 readings typically).

Balabanova D, McKee M and Mills A (eds) (2011). â˜Good Health at Low Costâ� ™� 25 Years On. What Makes a Successful Health System? [online] London: London School of Hygiene & Tropical Medicine (Chapter 2, pp. 11-45) Available from: ghlc.lshtm.ac.uk/files/2011/10/GHLC-book_Chapter-2.pdf (accessed May 2014).


Boerma JT & Stansfield SK (2007).Health statistics now: are we making the right investments? Lancet 369(9563):779-86.

Bennett S, Agyepong IA, Sheikh K , Hanson K, Ssengooba F & Gilson L (2011). Building the field of health policy and systems research: an agenda for action. PLoS Med 8(8):e1001081.

Commonwealth Fund Commission on a High Performance Health System (2006). Framework for a high performance health system for the United States [online]. Available athttp://www.commonwealthfund.org/Program-Areas/Archived-Programs/Commission-on-a-High-Performance-Health-System.aspx (accessed May 2014).

Gilson L, Hanson K, Sheikh K, Agyepong IA, Ssengooba F & Bennett S ( 2011). Building the field of health policy and systems research: social science matters. PLoS Med 8(8):e1001079.

Gilson L (ed.) (2012). Health policy and systems research: a methodology reader [online] Geneva: WHO. Available at www.who.int/alliance-hpsr/alliancehpsr_reader.pdf (accessed May 2014).

International Health Partnership (2008). A common framework for monitoring performance and evaluation of the scale-up for better health. Monitoring and Evaluation Working Group. Available at http://www.who.int/healthsystems/IHP_M&E_briefing_30_Jan_2008.pdf (accessed May 2014).

Lavis JN, Rottingen JA et al. (2012). Guidance for evidence-informed policies about health systems: linking guidance development to policy development. PLoS Med 9(3): e1001186.

Nolte E (2010). International benchmarking of healthcare quality. A review of the literature. Technical Report [online] Santa Monica CA: RAND Corporation. Available athttp://www.rand.org/pubs/technical_reports/TR738/ (accessed May 2014).

World Health Organization (2010). Monitoring the building blocks of health systems: a handbook of indicators and their measurement strategies [online] Geneva: WHO. Available athttp://www.who.int/healthinfo/statistics/toolkit_hss/en/index.html [have a look at a few of the health systems block and the proposed indicators.

Cited references and sources

Adam T et al. (2012). Evaluating health systems strengthening interventions in low-income and middle-income countries: are we asking the right questions? Health Policy Plan, 27(Suppl 4), 9–19.

Alliance for Health Policy and Systems Research (2013). What is HPSR? Overview [online]. Available at www.who.int/alliance-hpsr/about/hpsr/en/index.html (accessed October 2013).


Almeida C, Braveman P et al. (2000). Methodological concerns and recommendations on policy consequences of the World Health Report 2000. Lancet 357: 16982–16997.

Arah OA, Westert GP, Hurst J & Klazinga NS (2006). A conceptual framework for the OECD Health Care Quality Indicators Project. Int J Qual Health Care 18(Suppl 1), 5–13.

Asia Pacific Observatory on Health Systems and Policies (2013). Developing Asia Health Policy Fellowship, [online]. Available at:http://www.wpro.who.int/asia_pacific_observatory/en/ (accessed October 2013)

Balabanova D, McKee M, Koroleva N et al. (2009). Navigating the health system: diabetes care in Georgia. Health Policy and Planning 24, 46-54

Balabanova D, McKee M and Mills A (eds) (2011). ‘Good Health at Low Cost’ 25 Years On. What Makes a Successful Health System? London: London School of Hygiene & Tropical Medicine (Chapter 2).

Balabanova D et al. (2013). Good Health at Low Cost 25 years on: lessons for the future of health systems strengthening. Lancet 381(9883), 2118–2133.

Bennett S, Agyepong IA, Sheikh K, Hanson K, Ssengooba F & Gilson L. (2011). Building the field of health policy and systems research: an agenda for action. PLoS Med, 8, e1001081.

Beran D, Yudkin JS & de Courten M (2006). Assessing health systems for type 1 diabetes in sub-Saharan Africa: developing a 'Rapid Assessment Protocol for Insulin Access'. BMC Health Serv Res 6: 17.

Blendon RJ, Kim M & Benson JM (2001). The public versus the World Health Organization on health system performance. Health Aff 20(3), 10–20.

BQS. BQS-Qualitätsreport 2008, [online]. Available at: www.bqs-qualitaetsreport.de

Clark AM, Whelan HK, Barbour R & Macintyre PD (2005). A realist study of the mechanisms of cardiac rehabilitation . Journal of Advanced Nursing 52(4), 362 371.

Countdown to 2015 (2013), [online]. Available at:www.countdown2015mnch.org/ (accessed October 2013)

Countdown Working Group on Health Policy and Health Systems (2008). Assessment of the health system and policy environment as a critical complement to tracking intervention coverage for maternal, newborn, and child health. Lancet 371(9620), 1284–1293.

de Savigny D & Adam T (eds) (2009). Systems Thinking for Health Systems Strengthening. [online] Geneva: WHO. Available at:whqlibdoc.who.int/publications/2009/9789241563895_eng.pdf (accessed October 2013)


Dr Foster Health Intelligence (2013). Essential reading for smart spending. Hospital guide 2013. [online]. Available at:myhospitalguide.drfosterintelligence.co.uk (accessed October 2013)

European Observatory on Health Systems and Policies (2013). Health Systems in Transition (HiT) Series, [online]. Available from: www.euro.who.int/en/who-we-are/partners/observatory/health-systems-in-transition-hit-series

Fung CH, Lim YW, Mattke S, Damberg C & Shekelle PG (2008). Systematic review: the evidence that publishing patient care performance data improves quality of care. Ann Intern Med 148(2), 111–123.

Gilson L (ed.) (2012). Health Policy and Systems Research: A Methodology Reader [online] Geneva: WHO. Available at www.who.int/alliance-hpsr/resources/alliancehpsr_reader.pdf (accessed October 2013)

Gilson L, Hanson K, Sheikh K, Agyepong IA, Ssengooba F & Bennett S (2011). Building the field of health policy and systems research: social science matters. PLoS Med 8(8), e1001079.

Girling A, Hofer T, Wu J, Chilton P, Nicholl J, Mohammed M & Lilford R (2012). Case-mix adjusted hospital mortality is a poor proxy for preventable mortality: a modelling study. BMJ Qual Saf 21, 1052–1056, doi:10.1136/bmjqs-2012-001202.

Hargreaves J et al. (2010). Process evaluation of the Intervention with microfinance for AIDS and gender equity (IMAGE) in rural South Africa. Health Educ Res 25(1), 27–40, doi: 10.1093/her/cyp054.

Harmer A (2011). Improving the lives of ‘half the sky’ – how political, economic and social factors affect the health of women and their children. In Balabanova D, McKee M & Mills A (eds.) (2011). ‘Good Health at Low Cost’ 25 Years On. What Makes a Successful Health System? London: London School of Hygiene and Tropical Medicine.

Health Systems 20/20 (2012). The Health System Assessment Approach: A How-To Manual. Version 2.0 [online]. Available atwww.healthsystems2020.org/content/resource/detail/528/ (accessed October 2013).

Helms R (2000). Sick list: health care à la Karl Marx. Wall Street Journal June 29, Brussels.

Huicho L, Dieleman M, Campbell J, Codjia L, Balabanova D, Dussault G & Dolea C (2010). Increasing access to health workers in underserved areas: a conceptual framework for measuring results. Bull World Health Organ 88(5), 357–363.

International Initiative for Impact Evaluation (3ie) (2012) [online]. Available at www.3ieimpact.org/ (accessed October 2013).


Kehnemui S, Fox News (2004). Doctors: Clinton Heart Surgery a Success. [online] Available at:www.foxnews.com/story/0,2933,131551,00.html (accessed Otober 2013)

kiesBeter [online]. Available at:www.kiesBeter.nl (accessed October 2013)

King G, Gakidou E, Imai K, Lakin J, Moore RT, Nall C, Ravishankar N , Vargas M, Téllez-Rojo MM, Avila JE, Avila MH & Llamas HH (2009). Public policy for the poor? A randomised assessment of the Mexican universal health insurance programme. Lancet 373(9673):1447-54.

Kruk ME & Freedman LP (2008). Assessing health system performance in developing countries: a review of the literature. Health Policy 85, 263–276.

Maimaris W, Paty J, Perel P, Legido-Quigley H, Balabanova D, Nieuwlaat R & McKee M (2013). The influence of health systems on hypertension awareness, treatment, and control: a systematic literature review. PLoS Med 10(7), e1001490.

Marchal B, Dedzo M & Kegels G (2010). A realist evaluation of the management of a well performing regional hospital in Ghana. BMC Health Serv Res 10, 24.

McPake B, Brikci N, Cometto G, Schmidt A & Araujo E (2011). Removing user fees: learning from international experience to support the process. Health Policy Plan 26 Suppl 2:ii104-117.

Murray C & Frenk J (2000). A framework for assessing the performance of health systems. Bulletin of the World Health Organisation 78(6).

Murray C & Frenk J (2001). World Health Report 2000: a step towards evidence-based health policy. Lancet 357, 1698–1700.

Navarro V (2000). Assessment of the World Health Report 2000. Lancet 356, 1598–15601.

Nolte E & McKee M (2011). Variations in amenable mortality – trends in 16 high-incomenations. Health Policy 103(1), 47–52.

Nolte E & McKee M (2012). Measuring and evaluating performance. In Smith R & Hanson K (eds.) (2012). Health Systems in Low and Middle-income Countries. An economic and policy perspective. Oxford: Oxford University Press.

Nuffield Trust (2013). Rating providers for quality: a policy worth pursuing? A report for the Secretary of State for Health: a summary. London: Nuffield Trust.

Organization for Economic Co-operation and Development (1985). Measuring health care, 1960-1983 expenditure, costs and performance. Paris: OECD.


Pawson R & Tilly N (1997). Realistic Evaluation. London: Sage.

Pitches D, Mohammed M & Lilford R (2007). What is the empirical evidence that hospitals with higher-risk adjusted mortality rates provide poorer quality care? A systematic review of the literature. BMC Health Services Research, 7, 91

Relman AS (1988). Assessment and accountability: the third revolution in medical care. N Engl J Med, 319(18), 1220–1222.

Sheikh K, Gilson L, Agyepong IA, Hanson K, Ssengooba F & Bennett S (2011). Building the field of health policy and systems research: framing the questions. PLoS Med, 8(8), e1001073.

Shaw RP (2002). World Health Report 2000 ‘Financial fairness indicator’: useful compass or crystal ball? Int J Health Serv, 32(1), 195–203.

Smith P (2002). Measuring health system performance. European Journal of Health Economics, 3, 145–148.

Spiegelhalter D (1999). Surgical audit: statistical lessons from Nightingale and Codman. Journal of the Royal Statistical Society, 22, 45–48.

Ssengooba F, McPake B & Palmer N (2012). Why performance-based contracting failed in Uganda – an ‘open-box’ evaluation of a complex health system intervention. Soc Sci Med, 75(2), 377 383.

Sundhedskvalitet [online]. Available at:www.sundhedskvalitet.dk (accessed October 2013)

Sutton M, Nikolova S, Boaden R, Lester H, McDonald R & Roland M (2012). Reduced mortality with hospital pay for performance in England. N Engl J Med 367(19):1821-8.

Taylor R (2013). The frightening truth: NHS managers are incentivised to ignore problems. Telegraph 4 February [online]. Available at:www.telegraph.co.uk/health/healthnews/9848414/The-frightening-truth-NHS-managers-are-incentivised-to-ignore-problems.html (accessed October 2013)

Walt G & Mills A (2001). World Health Report 2000: responses to Murray and Frenk. Lancet, 357, 1702–1703

Watts R & Donnelly L (2012). Don't leave patients in ambulances to hit A&E targets, hospitals told. Telegraph 27 Oct [online]. Available at www.telegraph.co.uk/news/9637865/Dont-leave-patients-in-ambulances-to-hit-AandE-targets-hospitals-told.html?fb (accessed October 2013).

World Health Organization (2013). Health statistics and health information systems. Monitoring the building blocks of health systems: a handbook of indicators and their


measurement strategies [online]. Available at:www.who.int/healthinfo/statistics/toolkit_hss/en/index.html (accessed October 2013)

World Health Organization (2000). The World Health Report 2000. Health Systems: Improving Performance. [online] Geneva: WHO. Available at:http://www.who.int/whr/2000/en/index.html (accessed November 2013)

World Health Organization (2007). Everybody's Business: Strengthening Health Systems to Improve Health Outcomes – WHO’s Framework for Action. Geneva: WHO.

All contents © LSHTM 2015 except where otherwise indicated.

GHM201 - Session 6: Assessing health...

Documents

Transcript of GHM201 - Session 6: Assessing health...