Evaluation of Health Management Information Systems - A...

Evaluation of Health Management Information

Systems - A study of HMIS in Kerala

Dr Harikumar S

Dissertation submitted in partial fulfilment of the requirement

for the award of the degree of Master of Public Health

Achutha Menon Centre for Health Science Studies

Sree Chitra Tirunal Institute for Medical Sciences and Technology

Thiruvananthapuram, Kerala

October 2012

ACKNOWLEDGEMENT

I sincerely thank, Dr Biju Soman , my guide, for his valuable inputs, encouragement and

liberty from the concept to writing of this report.

I thank Dr K R Thankappan, Dr Sundari Ravindran, Dr V Raman Kutty, Dr Mala

Ramanathan, Dr P Sankara Sarma, Dr K Srinivasan, Dr Ravi Prasad Varma and Dr

Manju Nair for their valuable comments and suggestions.

My sincere thanks to all my batch mates who helped directly or indirectly with their

thoughts, comments and support.

I am also grateful to the Director of Health Services, Kerala for granting permission to

conduct the study in the institutions under Health Services Department.

I also thank the National Rural Health Mission for providing the financial support to

conduct this study.

I am also grateful to all the staff of Health Services Department who cooperated

wholeheartedly for the successful completion of this study.

I gratefully appreciate the support and understanding provided by my dear parents, in-

laws, wife Lekshmi and daughter Devigayatri.

Certificate

I hereby certify that the work embodied in this dissertation entitled

“ Evaluation of Health Management Information Systems - A

study of HMIS in Kerala” is a bona fide record of original

research work undertaken by Dr Harikumar S, in partial fulfilment

of the requirements for the award of degree of ‘Master of Public

Health’ under my guidance and supervision.

Dr Biju Soman MBBS, MD, DPH, MSc.

Associate Professor



Thiruvananthapuram

October 2012

DECLARATION

I hereby declare that the work embodied in this dissertation entitled

“Evaluation of Health Management Information Systems - A

study of HMIS in Kerala” is the result of original research and has not

been submitted for any degree in any other university or institution.

Dr Harikumar S



TABLE OF CONTENTSLIST OF TABLES AND FIGURES

LIST OF ABBREVIATIONS

ABSTRACT

CHAPTERS Page

1. Introduction 12. Literature Review 23. The background in Kerala and rationale for the study 84. Goals and Objectives 105. Methodology

5.1 Conceptual framework- The PRISM framework 115.2 Study design and settings 125.3 Study population and subject selection 135.4 Sources of data 145.5 Data collection tools 155.6 Ethical considerations 175.7 Data entry and Analysis 18

6. Results6.1 Overview of Health Information Systems in Kerala 206.2 Socio-Demographic characteristics of the respondents 256.3 HMIS performance 266.4 HMIS processes 326.5 Determinants of Performance and processes

6.5.1 Technical determinants 336.5.2 Behavioural determinants 346.5.3 Organisational determinants 40

7. Discussion 478. Conclusions 549. Recommendations 5510. Strengths and Limitations of the study 5611. Bibliography 57

APPENDICES

LIST OF TABLES AND FIGURES Page Table 1: Socio-demographic characteristics of the respondents 25

Table 2: Mean percentage of perceived levels of confidence for HMIS tasks,

knowledge of rationale, motivation and reward 35

Table 3: Mean percentile scores of the respondents for HMIS task competence 37

Table 4: Comparison between mean behavioural scores of respondents

categorised by HMIS performance 39

Table 5: Mean percentage levels of management functions 42

Table 6: Mean percentile scores of the respondents for perceived promotion

of culture of information 43

Table 7: Comparison between mean perceived promotion of culture of

information by the respondents categorised by HMIS performance 45

Table 8: Institutions reporting inadequate resources 47

Figure 1: Percentage of facilities within 10 percent tolerance levels for accuracy 27

Figure 2: Reported data as percentage of actual values (mean) for selected items 28

Figure 3: Reported data as percentage of actual values (mean) according to

type of facility 28

Figure 4: Reported data as percentage of actual values (mean) according to

type of sub-centre 29

Figure 5: Percentage of facilities with specific use of information in meetings 31

Figure 6: Percentage distribution of facilities showing types of display and

updated information 33

Figure 7: Technical issues at Block, District and State level 34

Figure 8: Comparison among mean perceived confidence for HMIS tasks 36

Figure 9: Comparison among mean observed competence for HMIS tasks 37

Figure 10: Comparison between mean perceived confidence and observed

competence for HMIS tasks 38

Figure 11: Mean level of management functions at PHC and sub-centre level 41

Figure 12: Mean level of management functions at higher levels – Block

District and State 41

Figure 13: Comparison between mean perception of different dimensions of

Culture of information 43

Figure 14: Comparison between mean perception of promotion of Culture of

information and observed task competence 44

LIST OF ABBREVIATIONS

CHC : Community Health Centre

DHIS : District Health Information System

HMIS : Health Management Information System

HI : Health Inspector

HISP : Health Information System Project

HS : Health Supervisor

IDSP : Integrated Disease Surveillance Project

IT : Information Technology

JHI : Junior Health Inspector

JPHN : Junior Public Health Nurse

MCTS : Mother and Child Tracking System

NAMMIS : National Anti Malaria Management Information System

NPCB : National Programme for Control of Blindness

NPCDCS : National Programme for Prevention and Control of Diabetes , Cardiovascular Diseases and Stroke

NRHM : National Rural Health Mission

NVBDCP : National Vector Borne Disease Control Programme

PHC : Primary Health Centre

PHN : Public Health Nurse

PHNS : Public Health Nurse Supervisor

PCPNDT : Pre-conception and Pre-natal Diagnostic Techniques

PRISM : Performance of Routine Information System Management

RCH : Reproductive and Child Health

RHIS : Routine Health Information System

RNTCP : Revised National Tuberculosis Control Programme

VPD : Vaccine Preventable Disease

WHO : World Health Organization

AbstractTitle: Evaluation of Health Management Information Systems : A study of HMIS in Kerala

Background: Health information is the foundation of public health and a well performing

routine health management information system is needed to improve evidence-based

decision making and health system performance. Evaluation should be an integral part of

HMIS to identify weaknesses and continuous improvement. Kerala operationalised a web

based HMIS from April 2009 to support routine reporting. This study conducts a formal

evaluation of HMIS in Kerala with the specific objective of identifying the technical,

organisational and behavioural factors affecting the processes and performance.

Methods: The Performance of Routine Information System Management (PRISM)

framework and associated tools were used for empirical assessment of the technical,

organisational and behavioural determinants, the processes and performance related to

HMIS in Kerala. The descriptive cross-sectional study involved 115 respondents from 26

sub-centres, 12 primary health centres, six blocks, two districts and the state level office.

Results: The performance measured in terms of proportion of facilities within acceptable

limits of accuracy and completeness were low at 37% and 29% respectively. Reports

based on HMIS data were available only in 5 out of 38 facilities and the level of use of

information in meetings was 35%. The functionality level of the processes of checking

accuracy, completeness and timeliness in the facilities were 79%, 79%and 88%

respectively. The overall level of data analysis was 35%. The overall confidence in HMIS

related tasks was 69.4% compared to a competence of 58%. The management functions

for governance, planning, training, supervision and quality control were 13.2%, 43.4%,

5.3%, 28.4% and 44.7% respectively at the facility level. The perceived promotion of a

culture of information was 70% and the corresponding activity level was 25 percent.

Supervision quality was 44% while feedback level was 40 percent. 32% respondents did

not have adequate access to office space while 72% reported inadequate internet

connectivity.

Conclusions: The study revealed many inadequacies in HMIS processes in the state.

Detailed analysis provide insights into the determinants of these processes and probable

avenues for improving performance. Low levels of accuracy, completeness and use of

information found in this study are consistent with low levels of competence, promotion

of culture of information,training, supervision and feedback which needs to be improved.

1. IntroductionA Health Management Information System (HMIS) is defined by the World

Health Organisation as an information system specially designed to assist in the

management and planning of health programmes, as opposed to delivery of care 1. It is an

“integrated effort to collect, process, report and use health information and knowledge to

influence policy-making, programme action and research” 2. Health information system is

different from health-care information needed for medical professionals and more general

health related awareness. It deals with the morbidity and mortality patterns of

populations, causative analysis and the scope and effectiveness of public health

interventions.

Health information systems in developing countries are highly complex and have

been shaped by political, administrative, economic and donor pressures. Improvement of

the quality and accuracy of data coming from developing countries have been promoted

since the 1990s by augmenting the routine health information system with the help of

information technology. The development and maintenance of such systems are all the

more important in the recent times of resource constraints necessitating good governance,

transparency, accountability and evidence-based decision making.

Upon the launch of computer based HMIS there should be a thorough evaluation

of its processes to ensure that it is functioning optimally in accordance with the

requirements of the country or state. This initial evaluation provides the opportunity to

fine tune the system and should be supplemented by periodic evaluations to sustain the

results achieved. This is especially important in countries like India, where there are

many policy initiatives and increasing budgetary allocations to strengthen the HMIS 3.

1

2. Literature Review

Health information is the foundation of public health and a well performing

routine health management information system is needed to improve evidence-based

decision making and health system performance. Health information is a global public

good and informed decision making based on sound health information should be

recognised as part of the human right to health care 2,4.

The major components of the managerial process for national health development

such as policy formulation, broad programming, programme budgeting, preparation of

master plan of action, detailed programming, implementation, evaluation and

reprogramming require support in the form of relevant and sensitive information at all

stages. The selectivity of information is also vital as different users require different

information in varying details to support decision making. The information needed at

different levels include policy information, types of health care, health problems,

available resources, health manpower and the costs involved 5. A well functioning health

information system is identified as one of the six building blocks of a health system by

the World Health Organisation's framework of health system strengthening 6. The core

components of the information system has been described as the development of

indicators based on management information needs, data collection, transmission,

processing and analysis, which all lead to information use 7.

The health information system allows organizational members to track their

progress routinely in meeting organizational objectives, including patient management

objectives, for which data cannot be collected otherwise 7. The health system performance

is related to the performance of the health information system 8.

Health information systems when used optimally can improve the delivery of

2

health services by encouraging rational decision making and stimulating use of

information at the lower levels of the health system. They become sustainable only if they

have the ability to provide information that help in policy making. Promoting a culture of

information use will create more demand for information which will also help to improve

and refine the information system 9.

Comprehensive socio-economic data are needed to monitor the achievement of

Millennium Development Goals and implementation of poverty reduction strategies. At

the same time the health sector is trying to improve health outcomes by addressing the

social determinants of health. Integration and linkage of health information systems with

social and economic sectors is therefore essential to minimise the duplications and

inconsistencies in the collection, reporting, analysis, and storage of socio-economic data.

There is a growing recognition for the need to strengthen and coordinate the national

information systems in developing countries and this presents an ideal opportunity for the

health sector to streamline the limited resources. The improvements in the information

systems should be sustained by capacity building efforts complemented by adequate

career prospects for information system specialists 10. The health information systems

have immense potential to strengthen human and health rights by providing an equity-

oriented empirical base for decision making in health and allied sectors 11.

Information technology has been extensively used in the delivery of primary

health care especially electronic patient registries, clinical decision support, medical

education and telemedicine. While there is consensus regarding the usefulness of

information technology in improving managerial efficiency, studies about their impact on

the general health status are rare 12.

However increasing evidence from developing countries showed that health

3

management information systems were not producing the intended results due to poor

data quality, weakness in analysis, limited use of information and poor management

practices. An evaluation of the immunisation programme reporting system in

Mozambique showed that emphasis on targets and technicalities without proper support

mechanisms leads to poor data quality and a situation wherein data is merely transmitted

upwards rather than used locally 13. Apart from a system design that discourages data use,

poor data quality has been attributed to lack of adequate supervision and feedback as well

as inadequate incentives to health workers 14. The validity of reported data consequently

comes under scrutiny especially when limited resources have to be judiciously distributed

and accounted in a transparent manner 15.

Rapid strides in information technology should be accompanied by an

organisational evolution of health systems of which it is a part and such processes will

result in improved health status only when data informs decision making rather than

being an end in itself 16. Technology though vital for the successful implementation of

HMIS, is merely a tool to facilitate access to information and data processing prior to

decision making. The sustainability of HMIS depends more on the processes affecting the

organisational information culture and ongoing evaluation of these processes 17. Due to

multiple specific health programmes undertaken in low-income countries with the help of

donor agencies, there have been a massive influx of monitoring programmes that

“threatens to flatten the unsteady pillars of local health information systems”. Information

has to flow from a solid and sustainable platform especially in a situation of accelerated

demand fostered by the global preoccupation with outcomes-based development 18.

The complex health information systems needs to be simplified in terms of data

demand, the tools available for generating data and the levels of the health system at

4

which the data is used. The demand for and supply of data vary at different levels of the

health system along a continuum. While clinical management of and local community

needs are to be addressed at the lowest levels, district level needs pertain to functioning of

health facilities and the health system as a whole. At higher levels health information is

needed for strategic policy decisions and resource allocation. Assessment of health status

in many developing countries are largely based on extrapolations and predictive models

due to the lack of relevant and robust data. To a certain extent the unmet data needs in

developing countries can be attributed to the fragmented and uncoordinated allocation of

financial resources and a lack of adequate capacity to handle health information

especially in the context of decentralised health reforms 2.

The limited availability, quality and use of data in developing countries has to be

improved by strengthening the key data sources and capacity building measures.

Strengthening of the key sources of data such as household surveys, census, vital

registration, health facility reporting, surveillance and administrative systems will enable

countries to better monitor and evaluate their own progress and performance 19. Evidence

from Kyrgyzstan suggests that strengthening the information system should begin at the

grass-root level with training and capacity building. This helped improve the quality of

data along with timely detection and reaction to health problems by the health workers.

The process of strengthening HMIS should not be driven solely by donor priorities and

external consultants. This helped to improve the hitherto neglected quality of information

as well as build a sustainable platform 20.

The initial step in strengthening health information systems should be a

comprehensive and effective assessment of the existing system to establish a baseline and

to monitor progress. The assessment is a complex process and should involve all major

5

stakeholders and address the different subsystems involved, the inputs, processes, outputs

and outcomes 21. Development of evaluation methodologies and evaluation should be an

integral part of the HMIS process and should be institutionalised as a regular activity with

appropriate allocation of resources 1. Evaluation of health information systems supports

reflective practice and is an ethical imperative though challenging and resource intense 22.

However paucity of robust evaluation methodologies are hindrances to

strengthening the existing HMIS 23. Health system assessments are useful for planning,

monitoring health system developments over time and for comparing health systems in

different areas. Decentralisation of services necessitates such assessments at lower levels.

Such assessments are manageable at regional and district levels with modest planning and

analysis support from central levels 24. The strengths and weaknesses of existing health

information systems have to be evaluated through the use of a comprehensive framework.

Efforts to develop a comprehensive set of criteria for evaluation of health

information systems in developing countries were initiated in the late 1990s. In South

Africa focal group discussions involving experts from various fields of medicine,

computer science, nursing, biostatistics and health informatics were held to identify

criteria that can be used for evaluating health information systems. Several criteria were

identified which were grouped under categories such as philosophy and objectives, policy

and procedures, functionality, facilities and equipment, management and staffing, patient

interaction, staff development and education and evaluation and quality improvement 25.

However the instrument was too extensive to be used by district health information

managers and needed refinement to identify core evaluation criteria.

Similar efforts in Kenya led to the development of separate evaluation criteria to

be used during the pre-implementation, implementation and post-implementation phases

6

of setting up a health information system. The post-implementation criteria were divided

into internal, external and ultimate criteria. The internal criterion relates primarily to data

quality, use of information and overall system design. The external criterion were meant

to assess resource availability and management issues, while the ultimate criterion

assessed the impact of health information system on the health status of the people 26.

Evaluation of the district health information in rural clinics of South Africa was designed

around the information cycle framework 27. Semi-structured key informant interviews

were conducted to assess the steps of collection, processing, presentation and use of

information 28. The Health Information Systems Project (HISP) suggested a multi-step

model for the establishment of a health information system. Monitoring and evaluation of

the implementation process in each district was suggested in terms of the levels of

achievement. The levels of achievement were related to data collection, validation,

reporting, interpretation , presentation and information use for decision making 17,29.

The National Health Systems Resource Centre, New Delhi has developed a

readiness matrix to assess the level of HMIS implementation and capability achieved to

use information for action. It is based on the dimensions of technology, information

systems processes, data quality, human capacity, institutional collaboration and use of

information for action each of which were graded from least ready to most ready 30. The

World Health Organisation has provided practical guidelines for data collection activities

for evaluation of HMIS in developing countries. The major areas to be covered include

data generation, report compilation, data utilisation, computer infrastructure, training,

monitoring and other general resources. The methods of data collection for evaluation

purposes should include key informant interviews, focus group discussions and review of

records and logbooks 1.

7

3. The background in Kerala and rationale for the study

The health management information system in Kerala can be broadly divided into 5

subsystems.

a) Epidemiological surveillance systems like Integrated Disease Surveillance

project(IDSP), Polio and Measles surveillance

b) Special programme reporting like Revised National Tuberculosis Control

Programme (RNTCP), National Vector Borne Disease Control Programme

(NVBDCP), National Anti Malaria Management Information System (NAMMIS)

ans Mother and Child Tracking System (MCTS)

c) Administrative reports like Health Services Department reports, NRHM reports,

Service Payroll and Administrative Repository of Kerala (SPARK)

d) Vital registration systems through local bodies

e) Routine reporting from sub-centres, primary health centres & community health

centres through District Health Information System-2 (DHIS2) platform.

The design, customisation and implementation of the state-wide HMIS had been

assigned to HISP-India, a non-profit organisation. The transition to a web enabled

reporting system based on the DHIS2 platform was completed in 2008 and fully

operational across the State since April 2009. The objective of the project was to set up a

HMIS to support routine reporting across the different levels of the state health system.

The project was envisaged to be flexible enough to integrate with information systems of

other programmes. Currently the DHIS2 platform is used predominantly for the

information management related to reproductive and child health.

The functioning of the system involves around 6500 reporting units and 10,000

8

personnel. Considerable investment has gone into the programme in the form of

infrastructure, manpower training and maintenance. A formal evaluation of the system has

not been done so far to understand the operational and utilisation aspects. The concurrent

evaluation by the National Rural Health Mission makes only a very limited attempt to

evaluate HMIS. Kerala was ranked first among 35 States and Union territories evaluated

for the readiness to improve HMIS using a readiness matrix developed by the National

Health Systems Research Centre, New Delhi 30.

Evaluation of the HMIS in Kerala will be a timely and worthwhile effort to

identify the strengths and weaknesses of the existing systems which will help to

overcome the shortcomings and sustain the system in an effective manner. The results

from the study will surely help to strengthen the health system and improve the

performance.

9

4. Goals and Objectives

4.1 Overall Goal

To identify the strengths and weaknesses of the existing Health Management

Information system in Kerala to provide recommendations for better monitoring of health

system performance in the State of Kerala.

4.2 Research Question

What are the technical, organisational and behavioural determinants that affect the

HMIS processes and performance in Kerala?

4.3 Major Objective

To determine the technical, organisational and behavioural determinants that affect

the Health Management Information System processes and performance in Kerala using

the PRISM framework.

4.4 Minor Objective

To do a mapping of the HMIS processes in Kerala

10

5. METHODOLOGY

5.1 The PRISM framework

PRISM (Performance of Routine Information System Management) framework

The PRISM (Performance of Routine Information System Management)

framework has been developed by the MEASURE (Monitoring & Evaluation to Assess

and Use Results) evaluation group and RHINO (Routine Health Information Network

Organisation) network. PRISM is part of the Health Metrics Network of the WHO and

has been used in developing countries like Uganda, China, Ivory Coast, Paraguay, Haiti,

South Africa and Mexico 31.

The PRISM framework hypothesises that improved performance of HMIS leads to

11

INPUTS PROCESSES OUTPUTS OUTCOMES IMPACT

RHIS Determinants

Technical Factors Complexity of the

reporting form, Procedures HIS design Computer software IT complexity

OrganisationalFactors

Governance Planning Availability of resources Training Supervision Finance Information distribution Promotion of culture of information

Behavioural Factors

Data demandData quality checking skillProblem solving for HIS tasksCompetence in HIS tasksConfidence levels for HIS tasksMotivation

RHIS Processes

Data collectionData transmissionData processingData analysisData displayData quality CheckingFeedback

Improved RHIS Performance

Data qualityInformation use

Improved Health SystemPerformance

Improved HealthStatus

better health system performance which in turn leads to better overall health status.

Improved HMIS performance is defined as improved data quality and continuous use of

information for decision making.

The framework identifies and describes the various processes that contribute to

HMIS performance such as data collection, transmission, processing, analysis, quality

checking, data display and feedback. The performance of HMIS is dependent on how

these processes are carried out. The framework describes the organisational, technical and

behavioural determinants that affect these processes. These determinants have been

identified based on their closeness to performance, their perceived importance, their

adaptability and feasibility for change, the level of control exercised by HMIS managers

and implementers and the inclination to handle them.

The framework has developed operational definitions and four different tools to

measure information system performance along with the processes and their

determinants. It provides an opportunity for the empirical assessment of the interaction

between the various determinants, the processes involved and performance of HMIS. The

PRISM framework focuses on continuous improvement of the health management

information system by identifying determinants which have a negative impact and

suggesting solutions to rectify them 8.

5.2 Study Design and settings

The study has a descriptive cross-sectional design and evaluates the district health

information systems in one district each from the North and South of Kerala. A

quantitative approach has been adopted using the PRISM (Performance of Routine

Information System Management) tools.

12

5.3 Study Population and Subject Selection

The state of Kerala has 14 districts which are distributed in a North-South

direction. Due to logistic considerations two districts were chosen purposively with

District-A representing South Kerala and District-B representing North Kerala.

Permission to undertake the study was obtained from the Director of Health

Services, Kerala as well as the District Medical Officers of the two selected districts

before commencing the study.

The state level officers who were part of the study include the Additional Director

(RCH), Additional Director(Public Health), Deputy Director(RCH), State Demographer,

State Leprosy Officer and officers in charge of the Blindness Control Programme and

National Polio Surveillance Programme.

The district level officers who were part of the study include the Deputy District

Medical Officer, Reproductive and Child Health Officer, Leprosy Officer, District

Tuberculosis Officer, Malaria Officer, Technical Assistant, District Public Health Nurse,

Mass Media Officer and statistical assistants.

4 out of 16 blocks in District-A and 2 out of 7 blocks in District-B were selected

by lottery method. At the Block PHC level the Medical Officer-in-Charge, Health

Supervisor(HS) and Public Health Nurse Supervisor(PHNS) (one each) were

administered the appropriate tools.

In each block there is a Block PHC or Community Health Centre. Under each

Block PHC there are several PHCs and under each PHC there are several sub-centres. In

each block, two PHCs were selected, including the mother PHC which caters to the

public health activities in and around the CHC. The second PHC was selected by lottery

13

method. At the PHC level the appropriate tools were administered to the Medical

Officer-in-Charge, Public Health Nurse and Health Inspector(one each).

Under each PHC, two sub-centres were selected including the main sub-centre

which caters to the public health activities in and around the PHC. The second sub-centre

was selected by lottery method. At the sub-centre level the appropriate tools were

administered to the Junior Public Health Nurse and Junior Health Inspector. The

participants were all Government officers occupying designated posts at the time of the

study and in charge of activities related to Health Management Information System.

When a designated post was vacant or the person was on long leave, efforts were made to

include the person in charge of the related activities subject to his/her consent. At the

block and PHC level if an institution head is unwilling to participate, then another

institution from the remaining lot was intended to be selected. Officers at the state level,

district level and supervisory staff at block and PHC level occupy standalone designated

posts and therefore could not be substituted. In the event of such an officer being

unwilling to participate, the quality of the study could have been affected. However all

the approached personnel were cooperative and a substitute was not necessitated.

5.4 Sources of Data

At present maternal and child health related activities are the principal

components reported through the DHIS2 based information system across different levels

of the health system hierarchy. At each level the officers responsible for maternal and

child health were the principal respondents. The Junior Public Health Nurses(JPHN),

Public Health Nurses(PHN), Public Health Nurse Supervisors(PHNS) and the respective

medical officers were the principal respondents. The study also involved a review and

observation of facility records related to maternal and child health and information

14

system infrastructure.

The Junior Health Inspectors, Health Inspectors, Health Supervisors and other

programme officers at the district and state level handle the bulk of the remaining health

related information and therefore were included in the study to assess the organisational

and behavioural determinants of information use.

5.5 Data Collection Tools

The PRISM tools used for the study include RHIS Performance Diagnostic Tool,

RHIS Overview, Facility Check-list, RHIS Management Assessment Tool and

Organisational and Behavioural Assessment Tool. These are given as appendices II-A, II-

B, II-C, II-D, III, IV, V and VI.

5.5.1 RHIS Performance Diagnostic Tool

This is the primary component of the tool set and this assesses the HMIS

performance as measured by data quality and use of information, the processes and

technical determinants. At each level the main officers in charge of HMIS related

activities was interviewed by the principal investigator. It also involved review and

observation of facility records and information system infrastructure. The four

components selected for the study purpose were data with regard to antenatal registration,

pentavalent-1 vaccine, measles vaccine and DPT-1 vaccine during the months of May and

June 2012.

The data quality was assessed only at the levels of sub-centres and PHCs as direct

entry of the selected parameters (viz. antenatal registration, pentavalent-1, measles and

DPT-1) routinely occur only at these levels and not at higher levels. Use of information

15

and the processes were assessed at all levels. Technical determinants were assessed only

at the block, district and state levels as per the PRISM tools. Therefore the tool

administered at the block, district and state levels was different from the tool administered

at the facility level.

5.5.2 RHIS Overview Tool

This examines technical determinants such as the structure and design of existing

information systems in the health sector, information flows, and interaction between

different information systems. This tool was used for information mapping and chart the

flow of information by interviewing the concerned officers.

5.5.3 Facility Check-list

This tool was used to understand the availability and status of HMIS resources and

procedures used at health facilities and higher levels

5.5.4 RHIS Management Assessment Tool

This tool was designed to rapidly take stock of the management and supportive

practices of HMIS and to aid in developing recommendations for HMIS management.

5.5.5 Organisational and Behavioural Assessment Tool

This looks at behavioural and organisational determinants that affect HMIS

performance and processes. It assesses the perceived knowledge of rationale, competence

and skills in HMIS related activities, problem-solving ability, confidence, motivation and

the perceptions about promotion of culture of information in the health system. This tool

was administered to field and management staff at all levels who are involved in the

routine HMIS processes.

16

Validity and Internal consistency of the tools

A study from Uganda provides empirical evidence for reliability and validity of

the PRISM instruments 31. The tools were separately validated by officers in charge of

Revised National Tuberculosis Control Programme and National Pulse Polio

Surveillance Programme which are two programmes with a fully functional information

system in place.

The Organisational and Behavioural Assessment Tool (OBAT) was found to have

good internal consistency with a Cronbach's alpha of 0.72, 0.89 and 0.85 respectively in

the sections of decision making, behaviour of supervisors and general staff attitude.

5.6 Ethical Considerations

The respondents shall indirectly benefit from this study along with the whole

health system of which they are a part. The respondents all belong to the Kerala State

Health Services Department and foreseeing the possibility of them being held responsible

for whatever information they are divulging, strict privacy and confidentiality with regard

to all records and data were maintained. As an additional precaution written consent from

the study participants was waived with the approval of the Institutional Ethics Committee.

The identity of the districts and facilities selected for the study is also kept confidential to

safeguard the interests of the respondents. The individual and personal details was not

recorded in any form during the verification of the immunisation and antenatal registers.

17

5.7 Data Entry and Analysis

Data entry was done in EpiData 3.1 and analysed using OpenOffice 3.3

Spreadsheet and R version 2.15.1 . Schema as in the case of PRISM DEAT (PRISM Data

Entry and Analysis Tool) software was used for analysis. PRISM tools provide the

methods to objectively measure data quality and the degree to which information is used

for evidence-based decision making. The tools themselves provide coded values which

are computed to get frequency distribution of responses and mean percentile scores with

confidence intervals. The facilities were also classified into better performing and less

performing institutions on the basis of data quality and information use to find any

significant difference in the processes and determinants.

5.7.1 Data Quality

Completeness was assessed by proportion of filled data items pertaining to

maternal health and immunisation in facility reports. A tolerance level of 90 percent was

used in grading facilities to account for systemic, random and human errors which meant

that each facility was expected to complete at least 90 percent of data elements.

Timeliness was assessed by proportion of facilities sending data before the time

deadlines. Accuracy was assessed by comparison of reports sent to higher levels with

physical registers at the facility level. A tolerance level of 10 percent was assigned which

meant that each reported data element is not expected to vary more than 10 percent from

the actual count in the registers.

5.7.2 Use of Information

Use of information was assessed by reviewing available reports based on HMIS

and records of review meetings conducted in the past three months. The reports were

18

assessed for review of strategies, review of performance and targets, mobilisation of

resources and advocacy for more resources. The meeting records were assessed for

discussion on HMIS data quality and findings, the decisions made on the discussions,

follow-up actions and referral decisions taken. Frequencies and mean percentile scores

were used to report the level of use of information.

5.7.3 HMIS Processes

This was assessed by frequencies and mean percentile scores pertaining to data

transmission, processes for checking data quality and accuracy, display of updated data,

feedback and supervisory visits.

5.7.4 Technical determinants

This was assessed by proportion of facilities using different types of analysis of

data and the proportion of respondents reporting about the manuals, forms and design of

HMIS.

5.7.5 Organisational and Behavioural determinants

Task competence was assessed by frequencies and mean percentile scores

obtained for checking data quality, calculation of indicators, plotting and interpretation of

data and use of information. Task confidence was assessed using mean percentile scores

of perceived confidence levels for different tasks. Mean percentile scores were also used

to assess other determinants such as knowledge of rationale for collecting data,

motivation and perceived promotion of culture of information.

19

6. RESULTS

The results from this study are presented under different sections in accordance

with the general standards adopted for reporting PRISM assessments. The first section

will present an overview of the health information systems in Kerala. The second

provides a description of the HMIS performance as measured by data quality and use of

information along with the various processes existing at different levels. The third section

provides a description of the technical, behavioural and organisational determinants of

HMIS performance.

6.1 Overview of Health Information Systems in Kerala

The health system in Kerala can be divided into private sector and public sector. In

the public sector health system there are different departments such as Health Services,

Medical Education, Insurance Medical services, Homoeopathy and Indian Systems of

Medicine. This study focuses on the different information systems existing within the

Health Services Department.

The Health Services Department functions through facilities and institutions at

different levels. The Directorate of Health Services (DHS) is the central institution in the

state under which all the other institutions function. There are 14 districts in the state and

each district office is headed by a District Medical Officer. In each district there are

several blocks headed by a Block Medical Officer. Under each block there will be several

primary health centres under the charge of a Medical Officer. The sub-centres are the

grass-root level institutions manned by a Junior Public Health Nurse and Junior Health

Inspector. The areas under each block, primary health centre and sub-centre is demarcated

20

for delivery of services and collection of information. In addition to these institutions

there are other institutions which do not have any designated area and provide

predominantly medical care. They include Taluk Hospitals, District Hospitals, Women &

Children Hospitals and General Hospitals. The RNTCP programme runs institutions such

as the District Tuberculosis Centres and Tuberculosis Units at the sub-district level each

having demarcated areas for tuberculosis control activities. At lower levels the RNTCP

programme is integrated with the general health system. The Kerala State AIDS Control

Society (KSACS) functions as an autonomous society and is headed by an officer deputed

from the Directorate of Health Services. The National Rural Health Mission now re-

designated as the National Health Mission provides financial and management support

through independent societies at the district level.

In the present study the officers working in sub-centres, primary health centres,

blocks, district medical offices and the state level office were interviewed to get an

overview of the information flow occurring across different levels.

Most of the data originates at the sub-centre level with either the JPHN or JHI

responsible for the data collection and transmission. These data gets aggregated at

different levels. The reporting systems associated with different programmes have a

vertical structure with little integration between them and therefore leading to

considerable load at the grass-root level.

The Junior Health Inspectors at present sends in several reports including

monthly target and achievement report, vector survey report, reports related to malaria

control programme, non-communicable disease control programmes and migrant survey

reports. At present the JPHN collects and transmits information under several sections as

listed below.

21

1. DHIS 2 which is commonly referred to as HMIS : It collects information

regarding service delivery related to maternal health and immunisation activities

on a monthly basis and the data is finally aggregated at the state level. It is based

on service given at individual institutions. Data from private institutions are also

collected by the JPHN and entered separately.

2. MCTS (Mother and Child Tracking System) : It is an initiative of the national

government and is basically a replica of the Maternal and Child Health register

used by the JPHN. All the information collected by the JPHN is entered into the

online platform and transmitted to the national level. It is supposed to be updated

on a continuous basis.

3. NRHM reports : There are at least seven reports which are sent every month to

the NRHM district office and are related to utilisation of funds, human resources

and the activities of ASHA(Accredited Social Health Activist) workers.

4. IDSP (Integrated Disease Surveillance Programme) : This is currently a paper

based weekly reporting system and collects information mainly about

communicable diseases. The data is aggregated at the district level and state level.

During epidemics and monsoon season this is augmented by a daily telephonic

reporting system.

5. Non-communicable disease control programmes: Programmes have been

initiated under the National Programme for Prevention and Control of Cancer,

Diabetes, Cardiovascular Disease and Stroke (NPCDCS) with monthly reporting

formats.

6. Area based reports: In addition to the institutional service delivery reports, the

JPHN also submits monthly reports showing various targets and achievements in

22

their respective areas. In one district it is being collected in a Form-6 while in

the other it is in a computerised format known as Community Need Assessment

(CNA) report. These reports contain data related to maternal and child health,

family welfare, communicable diseases and stock positions.

7. The JPHNs were also found to send in separate paper reports related to family

welfare, communicable diseases, palliative care, school health programme,

Vitamin-A supplementation programme, supply of iron & folic acid, vector survey

and ICDS programme (Integrated Child Development Scheme).

All the paper reports are aggregated or collected by the supervisors at the primary

health centre level and block level and transmitted to the district level. The Public Health

Nurse (PHN) at the PHC and the Public Health Nurse Supervisor (PHNS) at the Block

level prepares and sends several reports namely monthly activity report, immunisation

report, stock report, pentavalent vaccine report, iron and folic acid report, Vitamin A

supplementation report, Vaccine Preventable Disease report, stock reports and NRHM

reports.

The Health Inspector and Health Supervisors at the PHC and Block level

respectively prepare and send several reports as listed below

1. Public health activity report

2. Family welfare report

3. Death report

4. Stock position

5. Mass media report

6. Immigrant screening report

7. Palliative care report

23

8. Non-communicable disease control programme report

9. School health programme report

10. Diarrhoeal disease control programme report

11. Dangerous and offensive trade Inspection report

12. Vector survey report

13. Fish Hatchery report

14. National Vector Borne Disease Control Programme (NVBDCP) reports

15. Migratory population survey report

16. In-patient/Out-patient report

In addition each PHC and Block sends monthly reporting forms related to RNTCP

programme. Other forms which are prepared at the block level and transmitted upwards

on a monthly basis are those related to National Programme for Control of Blindness and

National Polio Surveillance Programme.

At the district level all these reports are being aggregated in the statistical division

and transmitted to the state level. The reports which originate at the district level include

those related to cancer control programme and PCPNDT Act. There is also a separate

online platform, NAMMIS (National Anti Malaria Management Information System) for

reporting anti-malaria activities.

An outline of the various reports and components of the various information

systems are given as Appendix VII and VIII.

24

6.2 Socio-demographic characteristics of the respondents

Table 1 : Socio-demographic characteristics of the respondents (N=115)1) Age in Years Mean Median Min-Max

43.8 44 24 - 552) Years of employment 16.1 16 1 – 313) Sex Frequency Percentage

i) Male 51 44.3%ii) Female 64 55.7%

4) Titlei) State Programme Officers 7 6.1%ii) District Programme Officers 11 9.6%iii)District HMIS focal person 9 7.8%iv)Medical Officer-in-Charge 12 10.4%v) Supervisory staff 30 26.1%vi) JPHN/JHI 46 40.0%

5) Educationi) Matriculation (10th) 1 0.9%ii) Intermediate (12th) 57 49.6%iii) Bachelor Degree 23 20.0%iv) Master/Post-Graduate 5 4.3%v) Professional Degree 29 25.2%

6) HMIS training in past 6 months 20 17.4%

The socio-demographic characteristics of the respondents are given in Table 1.

The study included 26 sub-centres, 12 primary health centres, 6 block level offices, 2

district level offices and 1 state level office. A total of 115 health services personnel were

interviewed of which 46 were working in sub-centres, 27 in PHCs, 15 at block level, 20 at

the district level and 7 at the state level. All of them were involved in information system

handling related to maternal and child health or other programmes.

The age of the respondents varied from 24 to 55 years with an average of 43.8

25

years and their average experience of 16.1 years in the Health Services Department.

Almost half of the respondents had an educational qualification up to intermediate level.

Overall only 17.4 percent of the respondents stated that they had received some

sort of training related to HMIS in the past six months which indicates an urgent need for

training on an ongoing basis.

6.3 HMIS performance

The PRISM framework assesses the performance of the HMIS based on data

quality and use of information.

6.3.1 Data quality

At the facility level (sub-centre and primary health centre) data quality is assessed

across the dimensions of data accuracy and completeness. Since there is no regular direct

entry of data at the block, district and state level data quality could not be assessed at

these levels. Completeness and timeliness at the district levels were assessed by the

proportion of all facilities actually sending in the reports.

6.3.1.1 Data Accuracy

Data accuracy was measured by comparing the actual monthly reports with the

registers for selected data elements during the two months of May and June 2012. The

selected data elements were antenatal registration, pentavalent-1 vaccine, measles

vaccine and DPT Ist booster vaccine. This does not measure the immunisation coverage,

but the actual process of reporting data.

Overall only 37 percent of the institutions were within acceptable limits at a

tolerance of 10 percent in all the items for the months studied. All the institutions were

26

within the set limits for antenatal registration, while only 71percent, 63 percent and 58

percent of institutions were within the set limits for pentavalent-1 vaccine, measles and

DPT booster vaccinations respectively (Figure1).

When the reported figures were expressed as a percentage of actual figures in the

registers the mean values for antenatal registration, pentavalent-1 vaccine, measles

vaccine and DPT Ist booster vaccine were 100 percent, 111.7 percent, 113.8 percent and

116.7 percent respectively, indicating over-reporting of the immunisation elements

(Figure 2). When the reported figures expressed as percentage of actual figures were

compared between sub-centres and PHCs, the sub-centres showed over-reporting for

measles and DPT vaccine while both showed over-reporting for pentavalent-1 vaccine

(Figure 3). Also only 8 out of 26 sub-centres (31%) were within the set limits while 6 out

of 12 PHCs (50%) performed well.

27

AN RegistrationPentavalent-1

MeaslesDPT-1

0

20

40

60

80

100

120

Kerala(N=38)

Perc

enta

ge

Figure 1: Percentage of facilities within 10% tolerance levels for accuracy

28

Figure 2 : Reported data as percentage of actual values (mean) for selected items

Figure 3: Reported data as percentage of actual values(mean) according to type of facility

AN Registration

Pentavalent-1

Measles

DPT-1

80 90 100 110 120Reported NumbersNumbers in RegistersPercentage

Pentavalent-1

Measles

DPT-1

80 90 100 110 120 130SubcentresPHCs

Percentage

When the sub-centres were further analysed main centres were found to contribute

more to the over-reporting(Figure 4). Only 3 out of 13 (23%) main centres were within

set limits compared to 5 out of 13 (39%) other sub-centres.

6.3.1.2 Data Completeness

Completeness was assessed by the proportion of unfilled data items pertaining to

maternal health and immunisation in facility reports for the months of May and June

2012. The average proportion of completed data elements among the facilities studied

was 76%. When a tolerance level of 90 percent is used only 11 (29%) institutions came

within the acceptable limits.

At the district level the completeness was assessed by the proportion of all

facilities in the district that send the reports. 100 percent of the facilities in each district

had send the reports.

29

Figure 4: Reported data as percentage of actual values(mean) according to type of sub-centre

Pentavalent-1

Measles

DPT-1

80 90 100 110 120 130 140 150

Main CentreOther Subcentres

Percentage

6.3.1.3 Timeliness

Timeliness of data was to be assessed at the district level and state level by the

proportion of facilities that had send the reports by the specified deadline. Even though

there were specific deadlines at the district and state level, there were no records showing

the date of receipt of reports and could not be objectively verified.

The district HMIS focal persons stated that whenever there was a delay in

receiving reports from any facility telephone messages were used to remind them.

6.3.2 Use of Information

Use of information was assessed on the basis of reports based on HMIS data and

records of review meetings conducted in the past three months.

Reports showing findings, implications and actions taken on the basis of HMIS

data were available in only five out of thirty eight facilities studied. Reports were also

available at the state level, two district level offices and two out of six blocks. A review

of the available reports in the five facilities showed an overall 60 percent use of

information. At the block, district and state level the overall level of use of information in

available reports was 44 percent.

At the facility level meeting records were available in 92 percent of facilities. The

overall level of use of information in meetings in these facilities was 35.4 percent (95%

CI 27.6,43.3). 34 percent of facilities had discussion about HMIS data quality, 74 percent

of facilities discussed HMIS findings and 37 percent of facilities made decisions based on

the discussions. Decisions were referred to higher level by 31 percent of facilities and

none of the meeting records showed any follow-up actions regarding prior decisions

(Figure 5).

30

Meeting records were available at the state level, both districts and the six blocks

and the overall level of use of information in meetings were 40 percent, 20 percent and

80 percent at the block, district and state levels respectively. The low level of use of

information at the block and district is a cause of concern even though almost one-third of

institutions showed referral of decision to higher levels. The decisions taken at the facility

level are low compared to the discussion levels which indicates either a low decision

making capacity or that the decisions are of a kind that needs approval from a higher

level.

31

Figure 5 : Percentage of facilities with specific use of information in meetings

Discuss Data quality

Discuss HMIS findings

Decisions taken

Decisions referred

Follow-up

0% 10% 20% 30% 40% 50% 60% 70% 80%

34%

74%

37%

31%

0%

Kerala(N=38)Percentage of facilities

6.4 HMIS processes

The processes that were assessed include data quality check, data transmission,

data analysis, data display, feedback and activities for promotion of use of information.

The process of checking data quality involves checking for accuracy,

completeness and timeliness. The functionality level of the processes of checking

accuracy, completeness and timely transmission of data in the facilities was 79 percent

(95% CI 68.4, 89.5) , 79 percent (95% CI 68.4, 89.5) and 88 percent (95% CI 81.1, 95.2)

respectively.

Data analysis as a process was reported by 68 percent of the institutions and the

overall level of data analysis was 34.9 percent (95%CI 30, 39.8). Regarding the types of

analyses done by the facilities, 92 percent of institutions reported calculation of indicators

while only 47 percent reported comparison of data over time. None of the institutions

reported comparison with district or state level targets or comparison among types of

service coverage.

Display of data was reported by 92 percent of the facilities. The display of data

was further analysed by selected data display and whether the displayed data was updated

or not. The display of data related to maternal health, child health, facility utilisation and

disease surveillance were observed in 84.2 percent, 84.2percent , 18.4 percent and 73.7

percent of facilities respectively. Updated data for maternal health, child health, facility

utilisation and disease surveillance was displayed in 44.7 percent, 31.6 percent, 10.5

percent and 44.7 percent facilities respectively (Figure 6).

32

Display of data was also assessed at the block, district and state levels. Selected

data for maternal health, child health, facility utilisation and disease surveillance were

present in 55 percent, 66 percent, 22 percent and 100 percent of offices respectively but

updated data was displayed only with regard to disease surveillance, probably due to the

recent increase in vector-borne communicable diseases. At the state level updated display

of information was available for child health and disease surveillance while display was

missing for maternal health and facility utilisation.

Feedback from higher levels were reported by 39.5 percent of the institutions.

6.5 Determinants of performance

6.5.1 Technical Determinants

The technical determinants were assessed at the level of block, district and state

levels. The respondents stated that the overall level of complexity of the current system

was 16.7 percent. All of them stated that the current system does not provide a

comprehensive picture of the health system even though there is considerable overlap

33

Figure 6 : Percentage distribution of facilities showing Types of display and updated information(N=38)

Maternal health

Child health

Facility Utilisation

Disease Surveillance

0 10 20 30 40 50 60 70 80 90 100

84.2

84.2

18.4

73.7

44.7

31.6

10.5

44.7

Display PresentUpdated Display

Percentage

Type

of i

nfor

mat

ion

disp

laye

d

between other information systems.

Figure 7 shows that two-third of the respondents felt the procedure manual to be

user friendly while all of them felt the software to be user friendly and forms easy to use.

The respondents also stated that the current level of technology is able to provide access

to information to programme managers to a level of 37 percent.

6.5.2 Behavioural determinants

The behavioural factors are hypothesized to be important determinants that affect

the various processes and performance of the HMIS. The various behavioural

determinants that are assessed using the PRISM framework include knowledge of

rationale for collecting various types of data, knowledge in checking data quality,

motivation and perceptions of reward. Understanding the rationale behind collecting data

will spur demand for data by the health staff and the related activities will be guided by

the meaning and values attached to them. An expectation of positive outcome will

increase the probability of performing a task and the output will depend on the confidence

and competence of the person. All these factors are empirically assessed using the PRISM

framework.

34

Manual user-friendly

Software user-friendly

Forms easy to use

IT easy to manage

Information access to programme managers

67%

100%

100%

100%

37%

Percentage

Figure 7 : Technical issues at the Block, District and State level (N=9)

6.5.2.1 Knowledge of rationale, motivation and confidence levels for HMIS tasks

The confidence levels or self-efficacy was assessed using a scale of 0 to 100 from

low to high confidence in performing a particular HMIS related task. The overall

confidence in HMIS related tasks was 69.4 percent (95%CI 65.6, 73.3). Table 2 and

Figure 8 gives the perceived levels of confidence for different HMIS tasks, knowledge of

rationale, motivation and reward.

Table 2 :Mean percentage of perceived levels of confidence for HMIS tasks, knowledge of rationale, motivation and reward

HMIS task Overall (N=115)(95% CI)

Facilities(N=73)

State-District-Block

(N=42)

1 Checking data quality 62.4 (57.9, 66.8) 62.2 62.6

2 Calculation 81.9 (77.9, 85.9) 84.1 78.13 Plotting 67.5 (62.0, 73.0) 69.7 63.64 Interpretation 67.6 (63.2, 72.0) 65.8 70.7

5 Use of information 67.7 (63.5, 72.0) 66.1 70.5

6 Knowledge of rationale 77.0 (72.8, 81.2) 75.2 80.3

7 Motivation 68.4 (66.6, 70.2) 67.1 70.68 Reward 67.2 (62.3, 72.1) 67.1 67.4

35

6.5.2.2 HMIS Task competence

The competence of the respondents were assessed using a pencil-paper test which

included problems of calculating rates and percentages, plotting and interpretation of

data. The respondents were also asked to enumerate methods of checking data quality and

also the implications of the given data at different levels. Mean percentile scores were

then calculated for their competence in checking data quality, calculation, plotting and

interpretation of data and use of information.

The overall competence for performing HMIS tasks was 58.1 percent (95%CI

53.6, 62.5). Table 3 and Figure 9 gives the mean percentile scores of the respondents for

competence in various HMIS tasks.

36

Reward

Motivation

Knowledge of HMIS rationale

Use of Information

Interpretation

Plotting

Calculation

Checking data quality

67.2%

68.4%

77.0%

67.7%

67.6%

67.5%

81.9%

62.4%

Percentage

HM

IS ta

sks

Figure 8 : Comparisons among mean perceived confidence levels for HMIS tasks (N=115)

Table 3 : Mean percentile scores of the respondents for HMIS task competence


Facilities(N=73)


(N=42)

1 Checking data quality 62.3 (57.9, 66.8) 41.1 55.6

2 Calculation 91.0 (87.9, 94.1) 91.8 89.73 Plotting 75.7 (67.7, 83.6) 71.2 83.34 Interpretation 38.4 (32.8, 44.1) 31.2 51.0

5 Use of information 38.9 (32.4, 45.4) 29.8 54.8

The competence for calculation and plotting of data was comparatively high while

37

Use of Information

Interpretation

Plotting

Calculation


38.9%

38.4%

75.7%

91.0%

62.3%

Percentage

HM

IS T

asks

Figure 9 : Comparisons among mean observed competence for HMIS tasks (N=115)

that of interpretation and use of information was low. The respondents at the block,

district and state level had better competence than those at the facilities in all the tasks

except calculation. This is probably due to greater experience in the health services

department.

Higher confidence levels are supposed to be associated with higher levels of

competence and performance. Comparison between overall confidence and overall

competence showed a positive correlation between the two with a Pearson's correlation

coefficient of 0.31 (95% CI 0.13 ,0.47, p value <0.001). Figure 10 shows a comparison

between confidence and competence for individual tasks. There is consistency between

confidence and competence for calculation, plotting data and checking data quality.

However the competence levels are much lower for interpretation of data and use of

information when compared to the corresponding confidence levels.

38

Figure 10 : Comparison between mean perceived confidence and observed competence for HMIS tasks (N=115)

Use of Information

Interpretation

Plotting

Calculation


0 10 20 30 40 50 60 70 80 90 100ConfidenceCompetence

Percentage

6.5.2.4 Behavioural determinants in facilities according to HMIS performance

The facilities were categorised into better performing and less performing

institutions based on HMIS performance across the three dimensions of data accuracy,

data completeness and use of information. The criteria used for categorising the facilities

was 90 percent tolerance for completeness and 10 percent tolerance for accuracy . The

median score of 40 was used for classifying the facilities based on use of information.

The behavioural characteristics of the respondents belonging to each of these categories

was compared across the three dimensions of HMIS performance (Table 4)

Table 4 : Comparison between mean behavioural scores of respondents categorised by HMIS performance

Accuracy Completeness Use of Information in meetings

Better performing

(N=27)

Less performing

(N=46)

Better performing

(N=20)

Less performing

(N=53)

Better performing

(N=51)

Less performing

(N=22)Knowledge of HMIS rationale

75.1% 75.2% 70.0% 77.1% 80.1% 63.6%

Overall Confidence 61.1% 62.8% 75.8% 67.2% 68.7% 71.6%

Overall Competence 51.7% 53.8% 52.0% 53.4% 57.8% 41.9%

Motivation 64.3% 68.8% 67.1% 67.1% 67.8% 65.6%

Under the use of information categories the respondents belonging to better

performing facilities had significantly better scores for knowledge of rationale and overall

competence (p value <0.05). They also had slightly better score in motivation and a

slightly lower score in confidence, but these differences were not statistically significant.

39

6.5.3 Organisational Determinants

6.5.3.1 HMIS management

Management involves effective utilisation of resources and functions to produce

better outputs. The PRISM framework defines management as “presence of mechanisms

for managing HMIS functions and resources effectively for better performance” 32. The

management functions essential for any organisation or programme include governance,

planning, training, supervision, finances and quality control.

The governance function was measured by the presence of a mission statement,

management structure, updated organizational chart, involvement of information system

managers in senior management meetings and distribution list of information reports. The

planning function was measured by availability of recent situational analysis report, long

term plans and targets. The training function was assessed by the presence of training

manuals, on-job training and schedule of planned trainings. The supervision functional

level was assessed by the presence of supervisory check-list, schedule of supervisory

visits and supervisory reports. The financial functional level was not assessed in the

present study. Quality control levels was assessed by the presence of performance

improvement tools and availability of specific standards at different levels. Mean

percentile scores were calculated for each function (Table 5).

The management functions were on the lower side with a level of 13.2 percent,

43.4 percent , 5.3 percent , 28.4 percent and 44.7 percent for governance, planning,

training, supervision and quality control respectively at the facility level (Figure 11). At

the block, district and state levels the management functional levels were 33.3 percent,

40.7 percent, 25.0 percent,70.4 percent and 37.5 percent for governance, planning,

training, supervision and quality control respectively (Figure 12). Thus there is room for

40

improvement in the management functional aspects.

41

13.2%43.4%

5.3%

28.4%

44.7%

Figure 11: Mean level of management functions at PHC and sub-centre level (N=38)

Governance

Planning

Training Supervision

Quality Assurance33.3%

40.7%

25.0%

70.4%

37.5%

Figure 12 : Mean level of management functions at higher levels Block, District and State (N=9)

Governance

Quality Assurance

SupervisionTraining

Planning

Table 5: Mean percentage levels of management functionsFunction Facilities(N=38)

(95% CI)State-District-Block(N=9)

(95% CI)1 Governance 13.2 (5.8, 20.5) 33.3 (16.7, 50.0)2 Planning 43.4 (35.6, 51.2) 40.7 (23.7, 57.8)3 Training 5.3 (1.2, 9.3) 25.0 (5.8, 44.2)4 Supervision 28.4 (18.3, 38.4) 70.4 (55.0,85.8)

5 Quality Assurance 44.7 (39.6, 49.8) 37.5 (27.6, 47.4)

6.5.3.2 Perceived promotion of a culture of information

Any successful organisation creates, promotes and sustains a set of core values

around which it functions to achieve optimal results. In the context of HMIS, these set of

values can be designated as culture of information. The health workers work and behave

in accordance with the values they believe the organisation is promoting. The PRISM

framework defines culture of information as “the capacity and control to promote values

and beliefs among members of an organization for collection, analysis and use of

information to accomplish its goals and mission” 32. The PRISM framework assesses the

culture of information by determining how strongly people believe that the health

department promotes values like emphasis on data quality, use of information, evidence

based decision making, problem solving, feedback from staff and community, sense of

responsibility and empowerment and accountability.

The overall scores (Table 6 and Figure 13) show that the respondents have a good

reason to believe that the organisation promotes a culture of information in emphasising

data quality, feedback and problem-solving. The lowest score was obtained for evidence-

based decision making which may be due to political interference or interference from

42

supervisors. Another finding is that all the scores were slightly less at higher levels

compared to facility level which may be due to exaggerated perceptions of the

respondents at the facility level.

Table 6 : Mean percentile scores of the respondents for perceived promotion of Culture of information


Facilities(PHC/SC)

(N=73)


(N=42)

1 Emphasis on Data Quality 77.6 (74.7, 80.6) 79.1 75.2

2 Use of Information 67.2 (64.4 70.0) 70.3 61.8

3 Evidence-based decision-making 64.0 (62.3 65.8) 63.4 65.2

4 Feedback from Staff & community 73.7 (70.7, 76.7) 74.8 71.9

5 Sense of Responsibility 67.8 (65.2, 70.4) 70.8 62.5

6 Empowerment & Accountability 68.0 (65.7, 70.2) 70.8 63.1

7 Problem Solving 72.1 (69.7, 74.6) 72.8 71

43

Figure 13 : Comparison between mean perception of different dimensions of Culture of information (N=115)

Emphasis on Data Quality

Use of Information

Evidence-based decision-making

Feedback from Staff & community

Sense of Responsibility

Empowerment & Accountability

Problem Solving

0 10 20 30 40 50 60 70 80 90 100

77.6

67.2

64

73.7

67.8

68

72.1

Percentage

The PRISM framework hypothesises that when there is a strong promotion of

culture of information, there will be a corresponding higher competence levels among the

health workers leading to better performance. However on comparison between the

perceived promotion of culture of information and the objectively assessed task

competence, there is a considerable gap across the dimensions of information use and

data quality checking (Figure 14). This may again be due to exaggerated perceptions by

the health workers or they are ignorant of the real situation.

A comparison between perceived promotion of culture of information between

different respondents categorised by the HMIS performance of their current institution is

given in Table 7.

44

Figure 14 : Comparison between mean perception of promotion of Culture of information and observed task competence (N=115)

Use of Information


67.2%

77.6%

38.9%

62.3%

Culture promotionCompetence

Percentage

Table 7 : Comparison between mean perceived promotion of culture of information by the respondents categorised by HMIS performance

Accuracy Completeness Use of Information in meetings

Better performing

(N=27)

Less performing

(N=46)

Better performing

(N=20)

Less performing

(N=53)

Better performing

(N=51)

Less performing

(N=22)

Emphasis on data quality 77.1% 80.2% 82.1% 77.9% 76.9% 83.9%

Use of Information 69.2% 70.9% 67.1% 71.4% 70.7% 69.2%

Evidence-based decision making 62.2% 64.1% 64.3% 63.0% 62.5% 65.3%

Feedback 73.5% 75.5% 77.9% 73.6% 72.7% 79.6%Sense of Responsibility 70.3% 71.1% 69.3% 71.3% 70.7% 70.9%

Empowerment & Accountability 70.0% 71.3% 70.4% 71.0% 69.2% 74.5%

Promote problem-solving 72.5% 72.9% 70.7% 73.5% 72.0% 74.5%

6.5.3.3 Activities for promotion of culture of information

The culture of information and use of information can be promoted in the health

organisation through certain activities. Such activities include communication about

targets, directives to use information, sharing of success stories and use of HMIS

information for advocacy. These activities are to be promoted at the facility level by the

higher levels.

In the present study the overall level of such activities at the facility level was only

25.7 percent (95% CI 19.5 , 31.8) while it was 38.9 percent at higher levels.

Communication about targets was noted in 63 percent of facilities and specific directives

for information use was observed in 29 percent of institutions. The use of examples and

advocacy using HMIS information was very low in the studied facilities.

45

6.5.3.4 Supervision

Supervision is vital to provide adequate support to the health workers and also

helps in training and continued improvement. 23 out of 38 (60.5%) institutions surveyed

had a supervisory visit at least once in the last 3 months. The quality of supervision was

assessed on the basis of whether the supervisor checked data quality, discussed

performance, helped in decision making and send feedback reports. The overall level of

supervision quality was 44.2 percent (95%CI 30.1, 58.3). In those institutions which had

supervisory visits, two-third institutions reported that the supervisor checked data quality

and discussed performance based on HMIS data. However the supervisor helped decision

making in 37.5 percent institutions and send a feedback report in only 10.5 percent of

institutions.

6.5.3.5 Availability of resources

Availability of resources is critical as it affects the confidence, motivation and

processes involved. Among the facilities surveyed 21 percent reported inadequate office

space while almost one third (32%) reported inadequate access to computers. Computers

were reported to be as under supplied by all the facilities and offices surveyed while

internet connection was reported to be slow and inadequate by 72 percent of institutions

(Table 8).

46

Table 8 : Institutions reporting inadequate resources (N=47)Resource Percentage

1 Office Space 21.0%2 Access to computers 32.0%3 Number of Computers 100.0%4 Internet connection 72.0%5 Electricity Supply 17.0%6 Forms 2.1%7 Registers 21.3%

7. Discussion

The overall objective of this study was to provide an empirical assessment of the

organisational, behavioural and technical determinants that affect the processes and

performance of HMIS in Kerala using the PRISM framework. The framework gives

adequate importance to these determinants as well as the performance and also provides

the necessary tools and methods for empirical assessment.

Overall the data quality as measured by accuracy and completeness was low

which may be due to several factors. An evaluation of the district health information

systems in Kenya found low rates for accuracy (30%) and completeness(19%) 33. An

evaluation of the HMIS in South Africa showed accuracy levels of 43 percent compared

to 37 percent in the present study 34. Over-reporting of immunisation figures have also

been reported in a study from Mozambique with higher values of 44 percent and 72

percent for BCG and measles respectively 13.

In the present study accuracy was found to be excellent for antenatal registration

while it was lower for the immunisation categories. There is a general tendency for over-

reporting immunisation achievement. Another finding is that the sub-centres were

47

contributing more towards over-reporting than PHCs and among sub-centres, the main

centres are responsible for much of the over-reporting. This could be either a deliberate

practice or due to an inherent fault in the design of the system. The antenatal registers

used in sub-centres are in a standard format with all essential columns and are supplied in

a printed format while there is no standard format for maintaining immunisation registers.

This has to be viewed also in the context of the shifting from an 'area-wise

reporting' to a 'facility-wise reporting' as part of the revision of HMIS in India. In the

changed 'facility-wise reporting' schema, only the numbers of actual service delivered

from the sub-centre (for example the number of children who are given immunisation

from the sub-centre during that month) needs to be reported whereas in the former 'area

wise reporting' schema, numbers of people in the sub-centre area who received the

services irrespective of from where they received it (for example the number of children

who are immunised in the area during that month) are to be reported. Thus earlier health

workers used to report number of people who received service from other institutes also,

if the recipients were residing in their area. Earlier area-wise reporting was used to review

the achievement of population based targets and the performance of the health workers.

The area-wise reporting was replaced in the routine HMIS in order to avoid counting the

data more than once as the same data could be reported by the sub-centre, primary health

centre, community health centre or private hospitals. However the performance of

individual health workers are still being reviewed on the basis of population targets and

achievements on a monthly basis. This requires them to prepare two types of reports and

they probably end up preparing a wrong report. The concept of 'facility-based reporting'

probably will take more time to be fully understood by the health workers especially

when area-wise reporting gives them a sense of satisfaction and also provides an

48

opportunity to highlight the work done by them.

The existence of two separate data sets for the primary health centre and the main-

centre also creates considerable confusion. Even though there is an official separation,

there is no physical demarcation between a primary health centre and a main-centre.

Beneficiaries from areas other than the main-centre area also visit the primary health

centre and avail services. At present it is required to enter these beneficiaries in the

primary health centre data set. When the JPHN enters the data at the end of the month she

will find it difficult to distinguish between beneficiaries from her own area and other

areas, especially when there is no standard format for immunisation register. It is also

possible that the register entry is made by a JHI or other staff at the time of immunisation.

Another possibility could be presence of ambiguous data elements and a lack of

understanding about data elements for health workers. Proper understanding about the

data elements is necessary for accurate data entry and can only be achieved through

ongoing training and supervision of HMIS activities. The low level of training,

supervision and quality of supervision found among the respondents and facilities could

be a possible reason for the low data quality observed in the study. Similar low levels of

training and supervision have also been reported from the evaluation study from Kenya 33.

Since there are no computers available in sub-centres JPHNs tend to convey data

over telephone to the PHC where it is then entered by another person. This can also lead

to errors in data entry.

Completeness in filling the data elements was low even though the processes for

ensuring completeness showed a high level. This can be due to ambiguous instructions or

different instructions being given at different occasions. It may also be due to lack of

knowledge about the implications of an incomplete data set or lack of time. There seems

49

to be confusion even among the district level managers about using 'zero' or leaving it

blank. The general instruction given was to leave a column blank if that particular service

is not provided at the facility and to use 'zero' when the service has not been provided

even when it is available. However the message has not been clearly passed on to the

health workers. For example since there are no deliveries occurring at the sub-centres and

PHCs the zero doses of OPV and Hepatitis B vaccine are usually not administered at

these levels. Most of the facilities have entered 'zero' for OPV-0 when they should have

kept it blank. Most of them have left the Hepatitis B column blank. Ongoing training and

supervision will help to correct such mistakes.

The large number of reports that ought to be made at the end of the month along

with an inadequate internet connection, as reported by majority of facilities could result in

a shortage of time thus forcing them to send the incomplete reports before the set

deadline. Completeness with regard to the proportion of facilities actually reporting

through HMIS is good which shows that there is a good reminder mechanism. However

the lack of direct involvement by private institutions is a cause of concern. The current

system wherein the staff collects data from the private hospitals might not be accurate,

complete or timely. Ideally the private institutions should participate by reporting directly

through the online system at least in the domains of communicable diseases, pregnancy

care, immunisation and vaccine preventable diseases.

A mechanism to monitor and evaluate timeliness of data should be designed and

incorporated into the system. It is a real inadequacy in the prevailing HMIS programs

used in developing countries as similar poor timeliness was reported from other studies 33.

Use of information was found to be low in the present study. Production of reports

showing findings, implications and action taken was very low in the facilities. Most of the

50

institutions were just compiling the data and forwarding it. Data is being collected mainly

for onward transmission rather than for locally relevant decision making. The overall

level of use of information in meetings is only 35 percent. Even though the level of

discussion on HMIS findings is high, the discussion on data quality is low. Decision

making based on the discussion also showed a low level which indicates a low capacity to

make decisions or the decisions are of a kind that needs approval from a higher level.

Evaluation of the health management information system in a province of Mexico using

the PRISM tools also found a low level of use of information even though the task

competence and level of accuracy was high 32.

The low level of use of information is consistent with the limited competence in

interpretation (38%) and use of information (39%). The competence for checking data

quality is also low (62%) while those for calculation (91%) and plotting (75%) are on the

higher side. However there is a discordance between perceived confidence levels and

competence for interpretation and use of information. A possible reason for this

discrepancy could be how well the respondents understood the questions asked and how

they defined interpretation and use of information. But the respondents were quite

objective in their self-assessment as seen by the good concordance for calculation,

plotting and data quality checking. The overall competence levels had a significant

positive correlation with overall confidence levels which also makes this explanation less

likely. A more likely explanation would be that there is limited training on data

interpretation and use of information which does not allow them to properly test their

skills on these aspects.

The perceived levels of promotion of use of information (67%) and promotion of

checking quality (78%) are comparable with the figures (72% and 70% respectively)

51

from a similar evaluation of the HMIS in Mexico 34 . The relatively lower levels of

perceived promotion of use of information(67%) and evidence based decision making

(64%) when compared to other dimensions of culture of information also correlates with

the low competency for information use (39 percent). Even though both are low, there is a

gap between competence and the perceived promotion of use of information which may

be due to exaggerated perceptions or misplaced expectations by the health workers. In

neighbouring Pakistan the low levels of the use of information generated through the

HMIS have been attributed to political motives and corruption in addition to the poor

quality of data generated 35.

The use of information is also limited by the low level of feedback process from

higher level to lower level institutions. A study in rural South Africa also found a weak

culture of information with low levels of analysis, interpretation and use of data 28.

Feedback to institutions regarding their performance along with an analysis comparing

the performance with similar institutions or regional targets will spur similar processes at

the institution level. Such a feedback process will also improve the display process. The

display of information helps in comparative analysis, monitoring progress over time and

improving transparency along with providing a visual image of the work done. In the

present study even though display of information is available in majority of institutions,

the level of display of up-to-date information is quite low. This may also be due to the

low level of supervision quality observed in the study and a lack of adequate time.

The findings are also consistent with the low level of management functions

related to supervision and training. A cross-sectional study from Brazil found weaknesses

in analysis, interpretation and use of data which was attributed to poor management

practices and a lack of regular supervision and feedback 36. HMIS supervision is probably

52

part of the general supervision and may not be oriented towards HMIS tasks such as

checking data quality and use of information. Training activities are also low and

probably limited to data collection and web-based data entry. There are no

institutionalised mechanisms for planned training on an ongoing basis. This may be due

to lack of competent trainers at the sub-district level, lack of initiative from higher levels

or a lack of finances. The lack of ongoing training and supervision quality at the facility

level restricts the available opportunities for continuous improvement.

Sustainability, self-reliance and continuous improvement also depends on the

perceived promotion of a culture of information. Promotion of a culture of information

improves the working environment which leads to more evidence-based decision making,

transparency and accountability.

The technical aspects play a vital role in modern HMIS. Computer and

information technology should be optimally utilised to improve the health status of the

population and should not hinder the work in any way. Inappropriate use of cumbersome

technology will hamper the processes and performance of HMIS. In the present study the

software used, the forms used and manual are well accepted by the respondents. But the

dream of integrating with other parallel systems have not materialised. It has also not

succeeded in reducing the work burden of the health workers. The lack of integration with

other existing systems lends it incapable of providing a comprehensive picture of the

health status. Other systems which transmit similar information are still existing and

results in considerable overlap and work duplication. Systems carrying similar

information flow even come to exist later like the MCTS necessitated by national

priorities. The lack of integration may be due to the reluctance of individual donor driven

programmes to join the general system, unwillingness to take decisions or both. Lack of

53

congruence in policy matters between the national and regional governments also plays a

role.

This study was conducted as a cross-sectional survey and provides only

descriptive comparison of the determinants, processes and performance of HMIS. Causal

inferences cannot be made from this study. The findings are internally consistent and

generally conforms to the PRISM framework which tries to provide empirical evidence to

normative thinking. The scales used for constructing the organisational and behavioural

components showed high internal consistency indicating that they can be reliably used

for assessing the components like confidence, motivation and perceived promotion of

culture of information.

7. Conclusions

The study revealed many inadequacies in HMIS processes in the state. Detailed

analysis provide insights into the determinants of these processes and probable avenues

for improving performance. Low levels of accuracy, completeness and use of information

found in this study are consistent with low levels of competence, promotion of culture of

information,training, supervision and feedback which needs to be improved.

54

8. Recommendations

Improve the skills and competency of the staff with regard to data interpretation, use

of information and evidence-based decision making through regular training programmes.

Training related to HMIS activities should be conducted regularly in a planned manner

with adequate and timely release of funds. District level master trainers should be

identified to provide guidance and training on an ongoing basis. Steps should be taken to

include HMIS training in the curriculum of in-service trainings like female and male

supervisory trainings prior to promotions.

Promote production of reports showing analysis, interpretation and actions taken

based on information generated through HMIS rather than a mere compilation of data at

the facility level. This can be done by highlighting good examples during review

meetings and through issue of newsletters.

Improve supervisory activities by training supervisory staff and developing a

supervisory check-list for data quality and information use. Supervision of HMIS

activities should be made a separate entity in monthly supervisory meetings and their

reports. Improve the feedback mechanisms at all levels by developing and disseminating

feedback guidelines to all districts.

Uniform standards should be adopted for routine reporting throughout the state and

strictly communicated to all institutions. There should be clear guidelines for the type of

reporting and management of different data elements. Incorporate a mechanism to

monitor timeliness of data in the computerised system.

The study revealed mismatch between the intention at higher levels and practice at

55

lower levels. In order to address these it is wise to deploy a core team at the state level to

engage in consultations with different programme officers to identify specific data

requirements and to create a data warehouse. This will help to identify overlapping data

elements and eliminate them thus reducing the work load of field level health workers. It

can be a first step towards integration of various information systems.

Provide adequate infrastructure in the form of computers, internet connection and

office space if necessary with the help of funds available with each institution.

Take measures to involve the private sector through regular consultations and bringing in

requisite legislations so that a comprehensive picture of the health status in the state could

be obtained.

9. Strengths and Limitations of the Study

The study uses a framework validated in other developing countries and which

provides an empirical assessment of the various determinants affecting the processes and

performance of HMIS and therefore provides a better understanding of the current

situation. The study is very timely as Kerala is going ahead with up-scaling of its

computerised routine health information system in the state. The fact that Government of

Kerala has funded this study is an indication of its relevance to the state of Kerala. The

results from the study will be useful for charting a road-map for further development and

improvement of HMIS not only for Kerala, but for country as such.

The principal investigator is working in the health system of which the HMIS is a

part and there could be an inherent possibility of bias. However sufficient caution has

been exercised to have an impartial assessment of the processes. Moreover the study tool,

56

the PRISM framework, is highly empirical and uses quantitative assessment processes.

Therefore possibility of the principal investigator's employment status influencing the

study findings is highly unlikely. The researcher states that there is no conflict of interest.

BIBLIOGRAPHY

1. World Health Organization. Developing health management information systems : a

practical guide for developing countries. Manila, Philippines: World Health

Organization, Regional Office for the Western Pacific; 2004.

2. AbouZahr C, Boerma T. Health information systems: the foundations of public

health. Bull. World Health Organ. 2005 Aug;83(8):578–83.

3. Ministry of Health and Family Welfare, Government of India. The National Health

Bill 2009,Government of India Working Draft Version 2009.

4. Hartshorne JE, Carstens IL. Role of information systems in public health services. J.

Dent. Assoc. S. Afr. 1990 Jul;45(7):313–7.

5. World Health Organization. Managerial process for national health development.

Geneva: WHO; 1981.

6. World Health Organization. Everybody’s business : strengthening health systems to

improve health outcomes : WHO’s framework for action. Geneva: World Health

Organization; 2007.

57

7. Lippeveld T, Sauerborn R, Bodart C, World Health Organization. Design and

implementation of health information systems. Geneva: World Health Organization;

2000.

8. Aqil A, Lippeveld T, Hozumi D. PRISM framework: a paradigm shift for designing,

strengthening and evaluating routine health information systems. Health Policy and

Planning. 2009 Mar 20;24(3):217–28.

9. Cibulskis RE, Hiawalyer G. Information systems for health sector monitoring in

Papua New Guinea. Bull. World Health Organ. 2002;80(9):752–8.

10. Macfarlane SB. Harmonizing health information systems with information systems

in other social and economic sectors. Bull. World Health Organ. 2005

Aug;83(8):590–6.

11. Bambas L. Integrating Equity into Health Information Systems: A Human Rights

Approach to Health and Information. PLoS Medicine. 2005;2(4):e102.

12. Tomasi E, Facchini LA, Maia M de FS. Health information technology in primary

health care in developing countries: a literature review. Bull World Health Organ.

2004 Nov;82(11):867–74.

13. Mavimbe JC, Braa J, Bjune G. Assessing immunization data quality from routine

reports in Mozambique. BMC Public Health. 2005;5:108.

14. Freund PJ, Kalumba K. Monitoring and evaluation of primary health care in rural

Zambia. A comparative study. Scand J Soc Med. 1985;13(4):137–46.

58

15. Murray CJL, Shengelia B, Gupta N, Moussavi S, Tandon A, Thieren M. Validity of

reported vaccination coverage in 45 countries. Lancet. 2003 Sep

27;362(9389):1022–7.

16. Sandiford P, Annett H, Cibulskis R. What can information systems do for primary

health care? An international perspective. Soc Sci Med. 1992 May;34(10):1077–87.

17. Williamson L, Stoops N, Heywood A. Developing a District Health Information

System in South Africa: a social process or technical solution? Stud Health Technol

Inform. 2001;84(Pt 1):773–7.

18. Evans T. Health information system in the new millennium: a gathering storm?

Bulletin of the World Health Organisation. 2003;81(12):856.

19. Chan M, Kazatchkine M, Lob-Levyt J, Obaid T, Schweizer J, Sidibe M, et al.

Meeting the Demand for Results and Accountability: A Call for Action on Health

Data from Eight Global Health Agencies. PLoS Medicine. 2010;26;7(1):e1000223.

20. Weeks RM, Svetlana F, Noorgoul S, Valentina G. Improving the monitoring of

immunization services in Kyrgyzstan. Health Policy Plan. 2000 Sep;15(3):279–86.

21. Health Metrics Network. Assessing the national health information system-An

Assessment Tool : version 4.00. Geneva: World Health Organization; 2008.

22. Ammenwerth E, Brender J, Nykänen P, Prokosch H-U, Rigby M, Talmon J. Visions

and strategies to improve evaluation of health information systems. International

Journal of Medical Informatics. 2004 Jun;73(6):479–91.

59

23. Mitchell E, Sullivan F. A descriptive feast but an evaluative famine: systematic

review of published articles on primary care computing during 1980-97. BMJ. 2001

Feb 3;322(7281):279–82.

24. Nordberg E, Oganga H, Kazibwe S, Onyango J. Rapid assessment of an African

district health system. Test of a planning tool. Int J Health Plann Manage. 1993

Sep;8(3):219–33.

25. Hanmer L. Criteria for the evaluation of district health information systems. Int J

Med Inform. 1999 Dec;56(1-3):161–8.

26. Odhiambo-Otieno GW, Odero WWO. Evaluation criteria for the district health

management information systems: lessons from the Ministry of Health, Kenya. Afr

Health Sci. 2005 Mar;5(1):59–64.

27. Heywood A, Rohde J. Using information for action. A manual for health workers at

facility level. Pretoria, South Africa: The Equity Project; 2001. Available from:

http://www.uio.no/studier/emner/matnat/ifi/INF5761/v12/undervisningsmateriale/He

ywood%20and%20Rohde%20-%20Using%20information%20for%20action%20a

%20manual%20for%20health%20.pdf (accessed 20 Oct 2012).

28. Garrib A, Stoops N, McKenzie A, Dlamini L, Govender T, Rohde J, et al. An

evaluation of the District Health Information System in rural South Africa. S. Afr.

Med. J. 2008 Jul;98(7):549–52.

29. Health Systems Trust. South African health review, 2001. Durban: Health Systems

Trust; 2002.

60

30. National Health Systems Resource Centre,New Delhi. Background note on HMIS

Assessment Frameworks, Workshop on Health Information Architecture :

Design,Implementation and Evaluation. 2011. Available from:

http://nhsrcindia.org/pdf_files/resources_thematic/Informatics_HMIS/NHSRC_Cont

ribution/HW%20Session%202%20B%20Assessment_428.pdf (accessed 10 May

2012)

31. Hotchkiss DR, Aqil A, Lippeveld T, Mukooyo E. Evaluation of the Performance of

Routine Information System Management (PRISM) framework: evidence from

Uganda. BMC Health Serv Res. 2010;10:188.

32. Aqil A, Ávila JEH, Parbul AS, Plaza B. Guanajuato SINAIS Assessment.

MEASURE Evaluation, USAID; 2010.

33. Odhiambo-Otieno GW. Evaluation of existing District Health Management

Information Systems. International Journal of Medical Informatics. 2005

Sep;74(9):733–44.

34. Aqil A. PRISM Case Studies: Strengthening and Evaluating RHIS. MEASURE

Evaluation, USAID; 2008.

35. Qazi MS, Ali M. Health Management Information System utilization in Pakistan:

challenges, pitfalls and the way forward. Biosci Trends. 2011 Dec;5(6):245–54.

36. Galvão PRS, Ferreira AT, Maciel MDGG, De Almeida RP, Hinders D, Schreuder

PAM, et al. An evaluation of the Sinan health information system as used by the

Hansen’s disease control programme, Pernambuco State, Brazil. Lepr Rev. 2008

Jun;79(2):171–82.

61

APPENDIX - I

Information sheet for study participants

Evaluation of Health Management Information Systems – A Study of HMIS in Kerala

I, (Name and designation of principal investigator) currently undergoing Master of

Public Health course at Achutha Menon Centre for Health Science Studies, Sree Chitra

Tirunal Institute for Medical Sciences and Technology, Thiruvananthapuram am

undertaking a study titled “Evaluation of Health Management Information Systems – A

Study of HMIS in Kerala” as part of the course requirement.

This study is being done under the supervision of (name and designation of

guide). Please feel free to ask any question or doubt related to this study.

Rationale & objective of the Study

Health information is the foundation of public health and a well performing

routine health management information system is needed to improve evidence based

decision making and health system performance. Kerala has been using the DHIS2

platform for routine reporting in the Health Services Department since April, 2009. This

has not been not evaluated so far to identify the merits and demerits of the system so that

it can be improved to improve health system performance.

Evaluation of the routine reporting component of HMIS in Kerala will be a timely

and worthwhile effort to identify the strengths and weaknesses of the existing systems

which will help to overcome the shortcomings and sustain the system in an effective

manner. The results from the study will surely help to strengthen the health system in

Kerala and improve the performance.

The main objective is to determine the technical, organisational and behavioural

determinants that affect the processes and performance of the routine reporting

component of Health Management Information System in Kerala

Study settings and Methods

The study is being done in 2 districts coded as District-A and District-B. The

District level HMIS team will be included in the study. At the sub-district level

institutions will be selected by lottery method. A particular set of tools called PRISM

tools will be used to undertake this evaluation and it consists of self-administered

questionnaires and interview schedules.

If you are administered a tool called Performance Diagnostic tool, it contains a section

on data accuracy, it also involves a limited review of records under your custody.

However it will not include recording of individual data. The time taken will range from

30 minutes to 2 hours depending on your designation and responsibilities.

The collected data will be used for research purpose only.

Risks:

Participating in this study will in no way affect your status, reputation or career

prospects in the department or elsewhere.

Benefits: There may not be any direct benefit for you from this study but from a public

health view point, your information may prove to be of great importance with respect to

understanding the functioning of health information system so that it can be improved for

benefit of community.

Confidentiality:Utmost priority will be given to protect the privacy and confidentiality of the information

provided by you. The collected information will not be shared with anyone not involved

in the study and reporting will be done in aggregate form only. At no stage your identity

will be revealed. All hard copies of filled interview schedules and consent forms will be

kept under the custody of principal investigator and will be destroyed properly when they

are deemed no longer needed or after one year of dissertation report submission,

whichever comes first.

Voluntary participation:

Your participation in this study is voluntary and you have the right to withdraw your

participation at any time during the interview without any explanation. Refusal to

participate will not involve any penalty or loss of benefits to which you are otherwise

entitled. If you have additional questions about this research you may contact me or the

member secretary of the Institute Ethics Committee.

Name of Principal Investigator – Mobile No:______________, Email: ______________

Name and contact details of Member Secretary, Institute ethics Committee

APPENDIX II-A

RHIS Performance Diagnostic Tool- State,District, BlockQuality of Data

Name of the state/district/block: Date of Assessment:Name of the Assessor: Title of Person Interviewed:

Data TransmissionDQ 1 Does the office keep copies of RHIS monthly reports

sent by health facilities?1.Yes 0.No

DQ 2 What is the number of facilities that are supposed to be reporting to (enrolled in) RHIS?

DQ 3 What is the number of facilities that are actually reporting to (enrolled in) RHIS?

DQ 4 Count the number of monthly reports submitted by the facilities for any two months (of the surveyor’s choosing)..

a.month b.month

DQ 5 What is the deadline for the submission of the RHIS monthly report by facility?

If no deadline is set, write no and go to Q8

DQ 6 Does the office record receipt dates of the RHIS monthly report?

1.Yes 0.No If receipt dates are not recorded, go to Q8

DQ 7

If DQ6 yes, check the dates of receipts for the two months (DQ7 the total number of Reports received before and after the deadline should be the same as in Q4).

a. Month (specify) b. Month (specify)

Item 1. Before deadline

2. After deadline

3. Before deadline

4. After deadline

Number of facilitiesDQ 8 Does the office have a record of people who receive monthly

report data by a certain deadline after receiving monthly reportsfrom the facilities?

1.Yes 0.No

DQ 9 Does the office have a record of submitting data on time to regional and/or national levels?

1.Yes 0.No

Data Accuracy

DQ 10

Manually count the number of following data items from the RHIS monthly reports for the selected two months. Compare the figures with the reports from the Computer or paper database.Item a. Month (specify) b. Month (specify)

Manual count Paper/computer Manual count Paper/ComputerDQ ADQ BDQ C

Data Processing/AnalysisDQ 11 Does a database exist to enter and process

data?0. No 1. Yes, by

paper database

2. Yes, by computerdatabase

DQ 12 Does the database produce the following?DQ 12a Calculate indicators for each facility catchment area 1.Yes 0.NoDQ 12b Data summary report for the state/district/block 1.Yes 0.NoDQ 12c Comparisons among facilities 1.Yes 0.NoDQ 12d Comparisons with state/national targets 1.Yes 0.NoDQ 12e Comparisons among types of services coverage 1.Yes 0.NoDQ 12f Comparisons of data over time (monitoring over time) 1.Yes 0.No

DQ13 Do you think that the RHIS procedure manual is user-friendly?

1.Yes 0.No

DQ 14 Do you think that the monthly report form is complex and difficult to follow?

0.Yes 1.No

DQ 15 Do you find the data software to be user-friendly? 1.Yes 0.NoDQ 16 Do you find that information technology is easy to manage? 1.Yes 0.NoDQ 17 Do you think that information system design provides a

comprehensive picture of health system performance?1.Yes 0.No

DQ 18 Do you think RHIS has information that is also included in Other information system?

1.Yes 0.No

DQ 19 Does the RHIS software integrate data from different information systems?

1.Yes 0.No

DQ 20 Does the information technology (Land Area Network –LAN or wireless network ) exist to provides access to information to all state managers and senior management

1.Yespartially

2.Yescompletely

0.No

APPENDIX II-B

RHIS Performance Diagnostic Tool-State, District,BlockUse of Information

Name of Assessor:State: Title of Respondent:

RHIS Report ProductionDU1 Does this office compile RHIS Data submitted by facilities? 1.Yes 0.NoDU2 Does the office issue any report containing RHIS information? 1.Yes 0.No If no , go to

DU4DU3 If yes, please list reports that contain data/information generated through the RHIS.

Please indicate the frequency of these reports and the number of times the reportsactually were issued during the last 12 months. Please confirm the issuance of the report by counting them and putting the number in column 3. 1. Title of the report 2.No. of

times this report is supposed to be issued per year

3. No. of times that report are actually issued for the last 12 months

DU3aDU3bDU3cDU3dDU3eDU4 Did the office send a feedback report using RHIS

information to facilities or districts during the last three months?1.Yes 0.No

Display of InformationDU5 Does the office display the following data? Please indicate the types of data

Displayed and whether the data are updated for the last reporting period.If no go to DU6

1.Indicator 2.Type of display (Please tick) 3. UpdatedDU5a Related to mother health Table 1.Yes 0.No

Graph/ChartMap

DU5b Related to child health Table 1.Yes 0.NoGraph/ChartMap

DU5c Facility Utilization Table 1.Yes 0.NoGraph/ChartMap

DU5d Disease surveillance Table 1.Yes 0.NoGraph/ChartMap

DU6 Does the office have a map of the catchment area? 1.Yes 0.NoDU7 Does the office display a summary of demographic information such

as population by target group(s)?1.Yes 0.No

DU8 Is feedback quarterly, yearly or any other report on RHIS data available, which provides guidelines/recommendations for actions?

1.Yes 0.No If no,go to DU10

DU9 If yesto DU8, what kinds of decisionsare made in reports of RHIS data/information for actions? Please check types of decision based on types of analysis present in reports.Types of decisions based on types of analysis

DU9a Appreciation and acknowledgement based on number/percentage of facilities showing performance within control limits over time

(month to month comparisons)

1.Yes 0.No

DU9b Mobilization/shifting of resources based on comparison by facilities

1.Yes 0.No

DU9c Advocacy for more resources by comparing performance by areas (districts, sub-districts, cities, villages), human resources and logistics

1.Yes 0.No

DU9d Development and revision of policies by comparing types of services

1.Yes 0.No

Discussion and decisions about use of information 1.Yes 0.NoDU10 Does the office have routine meetings for reviewing

managerial or administrative matters?1.Yes 0.No

DU11 How frequently is the meeting supposed to take place? Circle appropriate answer

4. weekly 3. After every two weeks 2. monthly 1. quarterly 0. no schedule

DU12 How many times did the meeting take place during the last three months? Circle appropriate answer

7. 12 times 6. Between 7 and 11 5. 6 times 4. either 4 or5 3. 3 times 2. 2 times 1. 1 time 0. none

DU13 Is an official record of management meetings maintained? 1.Yes 0.No If no, go to DU15

DU14 If yes, please check the meeting records for the last three months to see if the following topics were discussed:

DU14a Management of RHIS, such as data quality, reporting, or timeliness of reporting

1.Yes, observed 0. No

DU14b Discussion about RHIS findings such as patient utilization, disease data, or service coverage, or medicine stock out


DU14c Have they made any decisions based on the above discussions?


DU14d Has any follow-up action taken place on the decisions made during the previous meetings?


DU14e Are there any RHIS related issues/problems referred to regional/national level for actions?


Promotion and Use of RHIS information at state/higher level DU15 Did state annual action plan showed decisions based on HIS

information?1.Yes 0.No

DU16 Did records of state office of last three months show that state/senior management issued directives on use of information

1.Yes 0.No

DU17 Did state/national RHIS office publish newsletter/report in last three months showing examples of use of information

1.Yes 0.No

DU18 Does documentation exist showing the use information for various types of advocacy?

1.Yes 0.No

DU19 Does the official staff meeting records show attendance of persons in charge of the facilities for discussion on RHIS performance?

1.Yes 0.No

DU20: Please describe examples of how the state office uses RHIS information for health system management 0. No examples 1. Yes (details follows)

APPENDIX II-C

RHIS Performance Diagnostic ToolQuality of Data Assessment: Health Facility Form

Date of Assessment: Name of the Assessor: Name and Title of person Interviewed:

District Facility TypeData Recording

FQ1 Does this facility keep copies of the RHIS monthly reports which are sent to the district office?

1.Yes 0.No If no, go to Q5

FQ2 Count the number of RHIS monthly reports that have been kept at the facility for the last twelve months

FQ3 Does this facility keep an immunisation register? 1.Yes 0.No If no, go to Q5

Data Accuracy Check

FQ4

Find the following information in the immunisation register for the selected two months. Compare the figures with the computer-generated reports.Item a. Month (specify) b. Month (specify)

# from register

# from report

# from register # from report

4A4B4C4D

FQ5 Did you receive a directive in the last three months from the senior management or the district office to: 5A Check the accuracy of data at least once in three months? 1.Yes, Observed 0. No5B Fill the monthly report form completely 1.Yes, Observed 0. No5C Submit the report by the specified deadline 1.Yes, Observed 0. No

FQ6 During the last three months, did you receive a directive from the senior management or the district office that there will be consequences for not adhering to the following directives:6A if you do not check the accuracy of data 1.Yes, Observed 0. No6B If you do not fill in the monthly reporting form completely 1.Yes, Observed 0. No6C If you do not submit the monthly report by the specified

deadline1.Yes, Observed 0. No

Data CompletenessFQ7 How many data items does the facility need to report on in the RHIS monthly

report? This number does not include data items for services not provided by this health facility.

FQ8 Count the number of data items that are supposed to be filled in by this facility but left blank without indicating “0” in the selected month’s report.

Data Transmission/Data Processing/AnalysisFQ9 Do data processing procedures or a tally sheet exist? 1. Yes, Observed 0. NoFQ10 Does the facility produce the following?FQ A Calculate indicators facility catchment area 1. Yes, Observed 0. NoFQ B Comparisons with district or national targets 1. Yes, Observed 0. NoFQ C Comparisons among types of services coverage 1. Yes, Observed 0. NoFQ D Comparisons of data over time (monitoring over time) 1. Yes, Observed 0. NoFQ11 Does a procedure manual for data collection(with definitions)exist? 1. Yes, Observed 0. No

APPENDIX II-D

RHIS Performance Diagnostic ToolUse of Information: Facility Assessment Form

Date: Name of Assessor:Facility Name: Name of Respondent and Title:Facility Type: District:

RHIS Report ProductionFU1 Does this facility compile RHIS Data? 1.Yes 0.NoFU2 Does the facility compile any report containing RHIS information? 1.Yes 0.No If no, go to

UI4FU3 If yes, please list reports that contain data/information generated through the RHIS.

Please indicate the frequency of these reports and the number of times the reports actually were issued during the last 12 months. Please confirm the issuance of the report by counting them and putting the number in column 3. 1. Title of the report 2. No. of

times this report is supposed to be issued per year

3. No. of times this report actually has been issued duringthe last 12 months

FU3aFU3bFU3cFU3dFU4 During the last three month, did the facility receive any feedback

report from district office on their performance?1.Yes 0. No

Display of InformationFU5 Does the facility display the following data? Please indicate types of data displayed and

whether the data have been updated for the last reporting period.If no go to UI6

1. Indicator 2. Type of display (Please tick) 3. UpdatedFU5a Related to maternal health Table 1.Yes 0.No

Graph/ChartMap/other

FU5b Related to child health Table 1.Yes 0.NoGraph/ChartMap/other

FU5c Facility utilization Table 1.Yes 0.NoGraph/ChartMap/other

FU5d Disease surveillance Table 1.Yes 0.NoGraph/ChartMap/other

FU6 Does the facility have a map of the catchment area? 1.Yes 0.NoFU7 Does the office display a summary of demographic information

such as population by target group(s)?1.Yes 0.No

FU8 Is feedback, quarterly, yearly or any other report on RHIS data available, which provides guidelines/ recommendations for actions?

1.Yes 0.No If no go to UI10

FU9 If you answered yes to question DU8, what kinds of action-oriented decisions have been made in the reports (based on RHIS data)? Please check the boxes accordingly Types of decisions based on types of analyses

FU9a Review strategy by examining service performance target and actual performance from month to month

1.Yes 0.No

FU9b Review facility personnel responsibilities by comparing service targets and actual performance from month to month

1.Yes 0.No

FU9c Mobilization/shifting of resources based on comparison by services 1.Yes 0.NoFU9d Advocacy for more resources by showing gaps in ability to meet targets 1.Yes 0.No

Discussion and Decision based on RHIS informationFU10 Does the facility have routine meetings for reviewing managerial or

administrative matters?1.Yes 0.No If no,

go to UI15

FU11 How frequently is the meeting supposed to take place?

4. weekly 3. After every two weeks 2. monthly 1. quarterly 0. no schedule

FU12 How many times did the meeting actually take place during the last three months?

7. 12 times 6. Between 7 and 11 5. 6 times 4. either 4 or5 3. 3 times 2. 2 times 1. 1 time 0. none

FU13 Is an official record of management meetings maintained? 1.Yes 0.No If no, go to FUI15

FU14 If yes, please check the meeting records for the last three months to see if the following topics were discussed:

FU14a Management of RHIS, such as data quality, reporting, or timeliness of reporting


FU14b Discussion on RHIS findings such as patient utilization, disease data, or service coverage, medicine stock out


FU14c Have they made any decisions based on the above discussions?


FU14d Has any follow-up action taken place regarding the decisions made during the previous meetings?


FU14e Are there any RHIS related issues or problems that were referred to the district or regional level for actions?


Promotion and Use of RHIS information by the district/higher level FU15 Observed facility received annual/monthly planned targets based on

RHIS information 1.Yes 0.No

FU16 Do facility records for the last three months show that district/senior management issued directives concerning the use of information

1.Yes 0.No

FU17 Did the facility receive a district or national RHIS office newsletter or report in last three months giving examples of use of information

1.Yes 0.No

FU18 Does documentation exist showing the use information for advocacy purposes?

1.Yes 0.No

FU19 Did the person in charge of the facility participate in meetings at district level to discuss RHIS performance for the last three months?

1.Yes 0.No

FU20: Please give examples of how the facility uses RHIS information for health system management 0. No examples 1. Yes (details follows)

Supervision by the District Health OfficeFU21 How many times did the district supervisor visit your facility during

the last three months? (check the answer)0. 1. 23.4. >3

If zero, go to FU26

FU22 Did you observe a supervisor having a check list to assess the data quality?

1.Yes 0.No

FU23 Did the supervisor check the data quality? 1.Yes 0.NoFU24 Did the district supervisor discuss performance of health facilities

based on RHIS information when he/she visited your facility?1.Yes 0.No

FU25 Did the supervisor help you make a decision based on information from the RHIS?

1.Yes 0.No

FU26 Did the supervisor send a report/feedback/note on the last two supervisory visits?

1.Yes 0.No

APPENDIX III

Routine Health Information System OverviewOverview of Information Systems in Health Sector

(Interview HIS Manager at district and sub-national level)

Level: NationalSub-national (district, province, etc.)Name (of district, province, etc.) _____________________________

Function/Title:

Institution Code:

Department:

Mapping existing routine information systems in health sector (OPTIONAL)

Using the sheet 1: “Information system mapping,” list all routine information systems existing in the country/region/district.

This exercise will help you to understand types of health sector information that are included (or not included) by information systems. It will also help to identify duplication of information systems.

1) Write down specific names of the information systems. 2) Identify types of information covered by each system and check relevant boxes. You may

also write comments in the box. For example, an information system for EPI may handle information on drug supplies but it might be limited to vaccines. You can indicate “vaccine only” in the box. Similarly, MCH specific information systems may collect information on service utilization of MCH services only.

3) Please describe how information from different information systems are shared. For example, between TB programs and HIV/AIDS programs.

.

1: In

form

atio

n Sy

stem

Map

ping

(OPT

ION

AL)

Ty

pes o

f Inf

orm

atio

n H

andl

ed b

y E

ach

Syst

ems

Type

of i

nfor

mat

ion

syst

emSp

ecifi

c na

me

if an

ySe

rvic

e U

tiliz

atio

nO

ccur

renc

e of

se

lect

ed

dise

ase(

s)

Dise

ase

Out

brea

k (I

mm

edia

te re

port)

Fina

ncia

l In

form

atio

nD

rug,

co

ntra

cept

ive

vacc

ine,

stoc

k

Hum

an

reso

urce

sEq

uipm

ent/

Bui

ldin

gV

ital

Even

tsO

ther

sO

ther

s

Rou

tine

serv

ice

base

d re

porti

ng sy

stem

Epid

emio

logi

cal

surv

eilla

nce

for n

otifi

able

in

fect

ious

dis

ease

sSp

ecia

l pro

gram

repo

rting

sy

stem

s(EP

I)Sp

ecia

l pro

gram

repo

rting

sy

stem

s(TB

)Sp

ecia

l pro

gram

repo

rting

sy

stem

s(M

alar

ia)

Spec

ial p

rogr

am re

porti

ng

syst

ems(

HIV

/AID

S)Sp

ecia

l pro

gram

repo

rting

sy

stem

s(M

CH

)Sp

ecia

l pro

gram

repo

rting

sy

stem

s(sp

ecify

)Sp

ecia

l pro

gram

repo

rting

sy

stem

s(sp

ecify

)Sp

ecia

l pro

gram

repo

rting

sy

stem

s(sp

ecify

)C

omm

unity

Bas

e in

form

atio

n sy

stem

Adm

inis

trativ

e sy

stem

(F

inan

ce)

Adm

inis

trativ

e sy

stem

(h

uman

reso

urce

)A

dmin

istra

tive

syst

em

(Tra

inin

g)A

dmin

istra

tive

syst

em

(dru

gs, c

ontra

cept

ive,

va

ccin

e, lo

gist

ics)

Adm

inis

trativ

e sy

stem

(

Infr

astru

ctur

e,

equi

pmen

t, tra

nspo

rt)V

ital R

egis

tratio

nO

ther

syst

em

2. Data collection and transmission

Please list all data collection tools/forms that are used at the community/health facility level. If space is not enough, please add an additional sheet of paper.Facility-based datacollection tools: (such as patient registers) Comments on tools. Is the form easy to

use? Enough space to record data? Takes too much time?

•

•

•

•

•

•

Data transmission/reporting forms Comments on forms. Is the form easy to use? Enough space to record data? Takes too much time?

•

•

•

•

•

•

3. Information flowchartUsing the chart provided on the next page, illustrate the flow of information from community to health facility, health facility to district level, district levelto regional level, regional level to the central/national level. For each level, please indicate specific departments/job titles which should receive and process information received from a lower level. This exercise will help you to clarify information flows in existing information systems and identify potential problems, which affect the performance of the information systems.1) If some levels, e.g. community level and regional level,are not relevant to systems that you are examining,

please omit them from the exercise.2) Please be as specific in identifying information sources and data transmission points as possible. For

example, if different types of facilities have different reporting units at district level, you will want to indicate these differentpaths of information.

3) Add more than one information system to see interactions between information systems and how complicated or simple information flows are in your health system. You can see how basic routine health information system’s information flow interacts with special program information systems such as EPI, HIV/AIDS, and Malaria.

4) You can be creative in indicating different information flows in different colors. For example, you can indicate the data aggregation process in red and the information feedback process in blue color. Or General RHIS in green and EPI in pink, etc.

Information FlowchartInformation Flow Sheet

Levels Types of Information Systems

HM

IS

EPI

TB Mal

aria

HIV

/AID

S

MC

H

Con

trac

eptiv

e

Adm

inist

rativ

e sy

stem

(F

inan

ce)

Com

mun

ity

info

rmat

ion

sy

stem

Central/National Level

Regional Level(Province)

District Level

Facility Level

Community Level

B. Organization of the health facilityB.1. Please describe total number of persons under each category below: (Adapt according to the country situation)B.2. Title/ post Number Number

1. Medical officer 10. Health educator

2. Comprehensive nurse registered 11. Health inspector

3. Comprehensive nurse enrolled 12. Laboratory technician

4. Nursing Assistance 13. Public health dental assistant

5. Clinical officer 14. Anesthetic officer

6. Laboratory Assistant 15. Midwife

7. Health Assistant 16. Support staff

8. Dispenser 17. Other (specify)

9. Health information assistant

B.3. Who fillsin the HMIS monthly reports? Specify the codes from Q B.2.

B.4. List those staff members who received any training in the recording, processing, orreporting of health informationduring the last twoyears, the number of trainings received, and the year of the latest training.

B.4.a. Title or Post(Coding from QB.2)

B.4.b. How many trainings courses/sessions did this person received in the past three years?

B.4.c. Year of last

training?

B.4.d. Subjects of last training: 1. data collection2. data analysis3. data display/report4. 1&25. 1&36. 2&37. 1,2 & 3

8. other (specify)1.

2.

3.

4.

5.

BB1.Only for Staff at District or Higher levelStaffingBB.1 Total number of persons working in district HMIS office including sub-districts?BB.2 Total number of persons working in district HMIS office excluding sub-districts?BB.3 Total number of district and sub-districtstaff in district HMIS office trained to collect,verify and analyze information?

APPENDIX IV

Organizational and Behavioural Assessment Tool(To be filled by staff and management at all levels)

Introduction

This survey is part of the_____________________, to improve Management Information Systems in the health sector. The objective of this survey is to help develop interventions for improving information system and use of information. Please express your opinion honestly. Your responses will remain confidential and will not be shared with anyone, except for presented table forms. We appreciate your assistance and co-operation in completing this study.

Thank you.___________________________________________________

IDI. Facility Code

ID2. District Code

DD1. Title of the person filling the questionnaire (circle answer)(Make these categories appropriate to the host country)

1. State level HMIS focal person2. District Medical Officer/ Deputy District Medical Officer3.4.

District HMIS focal person

5.Medical Officer-in-Charge

6. Junior Public Health Nurse/Junior Health Inspector

DD2. Age of the person ----------------------

DD3. Sex 1. Male 2.Female

DD4. Education1. 10 years 2. Intermediate (11-12) 3. Bachelor (13-14) 4. Master5. Professional diploma/degree (specify)-----------6. Other (specify) --------------------------------------.

DD5. Years of employment -----------------------

DD6. Did you receive any training in HMIS related activities in last six months? 0. No 1.Yes

Health Supervisor/Public Health Nurse Supervisor/Health Inspector/Public Health Nurse

We would like to know your opinion about how strongly you agree with certain activities carried out by _______________. There are no right or wrong answers, but only expression of your opinion on a scale. The scale is about assessing the intensity of your belief and ranges from strongly disagree (1) to strongly agree (7). You have to determine first whether you agree or disagree with the statement. Second decide about the intensity of agreement or disagreement. If you disagree with statement then use left side of the scale and determine how much disagreement that is –strongly disagree (1), somewhat disagree (2), or disagree (3) and circle the appropriate answer. If you are not sure of the intensity of belief or think that you neither disagree nor agree then circle 4. If you agree with the statement, then use right side of the scale and determine how much agreement that is – agree (5), somewhat agree (6), or strongly agree (7) and circle the appropriate answer. Please note that you might agree or disagree with all the statements and similarly you might not have the same intensity of agreement or disagreement and thus variations are expected in expressing your agreement or disagreement. We encourage you to express those variations in your beliefs.

This information will remain confidential and would not be shared with anyone, except presented as an aggregated data report. Please be frank and choose your answer honestly.

Strongly disagree

1

Disagree

2

Disagree

3

Neither Disagree nor Agree

4

Somewhat Agree

5

Agree

6

Strongly Agree

7

To what extent, do you agree with the following on a scale of 1-7?

In health department, decisions are based on

Stro

ngly

D

isag

ree

Som

ewha

t D

isag

ree

Dis

agre

e

Nei

ther

D

isag

ree

nor

Agr

ee

Agr

ee

Som

ewha

t A

gree

Stro

ngly

A

gree

D1. Personal liking 1 2 3 4 5 6 7

D2. Superiors’ directives 1 2 3 4 5 6 7

D3. Evidence/facts 1 2 3 4 5 6 7

D4. Political interference 1 2 3 4 5 6 7

D5. Comparing data with strategic health objectives 1 2 3 4 5 6 7

D6. Health needs 1 2 3 4 5 6 7

D7. Considering costs 1 2 3 4 5 6 7

Somewhat

Stro

ngly

D

isag

ree

Som

ewha

t D

isag

ree

Dis

agre

e

Nei

ther

D

isag

ree

nor

Agr

ee

Agr

ee

Som

ewha

t A

gree

Stro

ngly

A

gree

In health department, superiors

S1. Seek feedback from concerned persons 1 2 3 4 5 6 7

S2. Emphasize data quality in monthly reports 1 2 3 4 5 6 7

S3. Discuss conflicts openly to resolve them 1 2 3 4 5 6 7

S4. Seek feedback from concerned community 1 2 3 4 5 6 7

S5. Use HMIS data for setting targets and monitoring 1 2 3 4 5 6 7

S6. Check data quality at the facility and higher level regularly 1 2 3 4 5 6 7

S7. Provide regular feedback to their staff through regular report based on evidence 1 2 3 4 5 6 7

S8. Report on data accuracy regularly 1 2 3 4 5 6 7

In health department, staff

P1. Are punctual 1 2 3 4 5 6 7

P2. Document their activities and keep records 1 2 3 4 5 6 7

P3. Feel committed in improving health status of the target population 1 2 3 4 5 6 7

P4. Set appropriate and doable target of their performance 1 2 3 4 5 6 7

P5. Feel guilty for not accomplishing the set target/performance 1 2 3 4 5 6 7

P6. Are rewarded for good work 1 2 3 4 5 6 7

Stro

ngly

D

isag

ree

Som

ewha

t D

isag

ree

Dis

agre

e

Nei

ther

D

isag

ree

nor

Agr

ee

Agr

ee

Som

ewha

t A

gree

Stro

ngly

A

gree

In health department, staff

P7. Use HMIS data for day to day management of the facility and district 1 2 3 4 5 6 7

P8. Display data for monitoring their set target 1 2 3 4 5 6 7

P9. Can gather data tofind the rootcause(s) of the problem 1 2 3 4 5 6 7

P10. Can develop appropriate criteria for selectinginterventions for a given problem 1 2 3 4 5 6 7

P11. Can develop appropriate outcomesfor a particular intervention 1 2 3 4 5 6 7

P12. Can evaluate whether the targetsor outcomes have been achieved 1 2 3 4 5 6 7

P13. Are empowered to make decisions1 2 3 4 5 6 7

P14. Able to say no to superiors and colleagues for demands/decisions not supported by evidence 1 2 3 4 5 6 7

P15. Are made accountable for poor performance 1 2 3 4 5 6 7

P16. Use HMIS data for community education and mobilization 1 2 3 4 5 6 7

P17. Admit mistakes for taking corrective actions 1 2 3 4 5 6 7

Personal

BC1. Collecting information which is not used for decision making discourages me 1 2 3 4 5 6 7

BC2. Collecting information makes me feel bored 1 2 3 4 5 6 7

Stro

ngly

D

isag

ree

Som

ewha

t D

isag

ree

Dis

agre

e

Nei

ther

D

isag

ree

nor

Agr

ee

Agr

ee

Som

ewha

t A

gree

Stro

ngly

A

gree

BC3. Collecting informationis meaningful for me 1 2 3 4 5 6 7

BC4. Collecting information gives me the feeling that data is needed for monitoring facility performance 1 2 3 4 5 6 7

BC5. Collecting information givesme the feeling that it is forced on me 1 2 3 4 5 6 7

BC6. Collecting information is appreciated by co-workers and superiors 1 2 3 4 5 6 7

U1.Describe at least three reasons for collecting data on monthly basis on the followings:

U1A. Diseases1.2.3.

U1B. Immunization1.2.3.

U1C. Why is population data of the target area needed?1.2.3.

U2. Describe at least three ways of checking data quality.

1.2.3.

SELF-EFFICACY This part of the questionnaire is about your perceived confidence in performing tasks related to health information systems. High confidence indicates that person could perform the task, while low confidence means room for improvement or training. We are interested in knowing how confident you feel in performing HMIS-related tasks. Please be frank and rate your confidence honestly.

Please rate your confidence in percentages that you can accomplish the HMIS activities.

Rate your confidence for each situation with a percentage from the following scale

0 10 20 30 40 50 60 70 80 90 100

SE1. I can check data accuracy 0 10 20 30 40 50 60 70 80 90 100SE2. I can calculate percentages/rates correctly 0 10 20 30 40 50 60 70 80 90 100SE3. I can plot data by months or years 0 10 20 30 40 50 60 70 80 90 100SE4. I can compute trend from bar charts 0 10 20 30 40 50 60 70 80 90 100SE5. I can explain findings & their implications 0 10 20 30 40 50 60 70 80 90 100SE6. I can use data for identifying gaps and setting targets 0 10 20 30 40 50 60 70 80 90 100SE7. I can use data for making various types of decisions and providing feedback 0 10 20 30 40 50 60 70 80 90 100

We would like you to solve these problems about calculating percentages, rates and plotting and interpreting information.

C1. The estimated number of pregnant mothers is 340. Antenatal clinics have registered 170 pregnant mothers. Calculate the percentage of pregnant mothers in the district attending antenatal clinics.

C2.The full immunization coverage for 12-23 month-old children were found 60%, 50%, 30%, 40%, 40% for years 1997, 1998, 1999, 2000 and 2001 respectively.

C2a. Develop a bar chart for coverage percentages by years

C2b. Explain the findings of bar chart

C2c. Did you find a trend in the data? If yes or no, explain reason for your answer

2d. Provide at least one use of above chart findings at:

UD1. Facility level

UD2. District level

UD3. Policy Level

UD4. Community level

C3. A survey in a district found 500 children under five years old that were malnourished. The total population of children less than five years old was 5,000. What is the malnutrition rate?

C4. If the malnutrition rate in children less than two years old was 20% and the number of total children less than two years old was 10,000, then calculate number of children who are malnourished.

APPENDIX V

PRISM Tools Version 3.1.

RHIS Management Assessment Tool

(Observation at facility and higher levels)Questions under grey areas are not for the facility level

MAT1. Name of the Facility MAT2. Name of the Assessor

MAT3. Name of the District MAT4: Date of Assessment

MATG1 Presence of RHIS Mission displayed at prominent position(s) 0 No 1 Yes MATG2 Presence of management structure for dealing with RHIS related

strategic and policy decisions at district and higher levels0 No 1 Yes

MATG3 Presence of an updated (last year) district health management organizational chart, showing functions related to RHIS/health information

0 No 1 Yes

MATG4 Presence of distribution list and documentation of RHIS past monthly/quarterly report distribution at district or higher level

0 No 1 Yes

MATP1 Presence of RHIS situation analysis report less than 3 year old 0 No 1 Yes MATP2 Presence of RHIS 5 year plan at district or higher level 0 No 1 Yes MATP3 Presence of RHIS targets at facility and higher level 0 No 1 Yes MATQ1 Presence of a copy of RHIS standards at district or higher levels 0 No 1 Yes MATQ2 Presence of a copy of RHIS standards at facility 0 No 1 Yes MATQ3 Presence of performance improvement tools (flow chart, control

chart etc.) at the facility0 No 1 Yes

MATT1 Does facility/district have a RHIS training manual? 0 No 1 Yes MATT2 Presence of mechanisms for on-job RHIS training 0 No 1 Yes

MATT3 Presence of schedule for planned training

0.No 1. Yes, for one year

2. Yes, 2 years or more

MATS1 Presence of RHIS supervisory check list 0 No 1 Yes MATS2 Presence of schedule for RHIS supervisory visit 0 No 1 Yes MATS3 Presence of supervisory reports 0 No 1 Yes MATF1 Presence of RHIS related expense register 0 No 1 Yes

APPENDIX VI

Facility Check List for HMIS related resources

Item Accessible(Yes/No)

Adequate(Yes/No)

Number Available

Number Functioning

Office spaceComputer Table-Chair setCupboardComputersPrintersUPSInternet ConnectionCalculatorElectricity supplyRegistersFormsUse NA where not applicable

Appendix VII: Mapping of Health Information System in Kerala

Type of information

Information System Service Delivery

Incidence of diseases

Disease outbreak

Financial Information

Drugs and vaccine stocks

Human Resources

Equipments and buildings Vital events

Routine reporting system(HMIS) X X X

Epidemiological surveillance (IDSP) X X X

VPD Surveillance System X X X

RNTCP X X X X XMCTS X XNVBDCP/NAMMIS X X X X X XNPCB X X X XNRHM X X X XNPCDCS X X XADD/ORT programme X X X X

Cancer Control programme X X X

School Health Programme X X X

NLEP X X X X XPalliative care programme X X

Administrative Reports(annual) X X X X X X

HMISMCTSNRHM NCD Clinic Form-6/CNA Iron syrup issueIron Folic AcidVitamin A ICDS reportStock

FW ReportCommunicable DiseasePalliative CareSchool HealthVector Survey(weekly)

Vector SurveyMonthly AchievementMalaria smear reportFour Plus (weekly)NCD Clinic Immigrant report

IDSPFORM-S

PHN/PHNSActivity reportImmunisation reportStock reportPentavalent vaccine reportIron Folic Acid Iron syrup Partial Immunisation statusVitamin A VPD NRHM

VPD Surveillance

Activity report HI/HSFamily Welfare Death Stock Mass Media Palliative CareNCDSchool HealthADD /ORTD&O Trade InspectionFish Hatchery reportNVBDCP-MF4,MF5,MF6NLEPIP/OP ReportMigratory population survey

Blindness Control

RNTCP

STATE

DISTRICT

BLOCK

PHC

Sub-Centre

ASHA reports

Cancer ControlPNDT/NAMMIS

JPHN JPHN JHI

IDSPFORM-P

APPENDIX VIII: Scheme showing report production and transmission at different levels

Evaluation of Health Management Information Systems - A...

Documents

Transcript of Evaluation of Health Management Information Systems - A...