FINAL Healthy Futures Evaluation Executive …...Summary 4 Overview of Healthy Futures 4 Evaluation...

www.TheCIE.com.au

E X E C U T I V E S U M M A R Y

Interim evaluation of the Victorian Life Sciences Statement: Healthy Futures

Delivering better health, research and jobs for Victorians

Prepared for

Department of Business and Innovation

14 February 2013

THE CENTRE FOR INTERNATIONAL ECONOMICS

www.TheCIE.com.au

COMMERCIAL IN CONFIDENCE

The Centre for International Economics is a private economic research agency that

provides professional, independent and timely analysis of international and domestic

events and policies.

TheCIE’s professional staff arrange, undertake and publish commissioned economic

research and analysis for industry, corporations, governments, international agencies

and individuals.

C A N B E R R A

Centre for International Economics

Ground Floor, 11 Lancaster Place

Majura Park

Canberra ACT 2609

GPO Box 2203

Canberra ACT Australia 2601

Telephone +61 2 6245 7800

Facsimile +61 2 6245 7888

Email [email protected]

Website www.TheCIE.com.au

S Y D N E Y

Centre for International Economics

Suite 1, Level 16, 1 York Street

Sydney NSW 2000

GPO Box 397

Sydney NSW Australia 2001

Telephone +61 2 9250 0800

Facsimile +61 2 9250 0888

Email [email protected]

Website www.TheCIE.com.au

DISCLAIMER

While TheCIE endeavours to provide reliable analysis and believes the material

it presents is accurate, it will not be liable for any party acting on such information.

Interim evaluation of the Victorian Life Sciences Statement: Healthy Futures 3

www.TheCIE.com.au

Contents

Fkey findingFkey findingFkey findingFkey finding 1

Summary 4

Overview of Healthy Futures 4

Evaluation approach 6

Assessment of performance 7

Interim impacts for Victoria 12

Key evaluation messages from this review 12

Summary of key findings 17

BOXES, CHARTS AND TABLES

1 Stated objectives of the Healthy Futures program 5

2 Interim assessment of Healthy Futures 2006–2012 (Horizon 1) 8

3 Enhanced Performance Monitoring Framework tool 9

4 Interim evaluation of the Victorian Life Sciences Statement: Healthy Futures

www.TheCIE.com.au

Summary

Healthy Futures is a $230.45 million Victorian Government capital investment to

enhance the health and economic wellbeing of Victorians and maintain the state’s

competitive strength in medical research.

Healthy Futures has enabled the building of a wide range of strategic capital works

that has provided physical space, workforce development and enabling technologies

for enhanced innovation. It has created opportunities to improve the quality of

medical research and contribute to the future health of Victorians.

As an interim evaluation, this assessment focuses on evaluating the outcomes and

impacts of Healthy Futures to date — or Horizon 1 (2006 – 2012). The interim

evaluation also attempts to identify early signs and indicators of impacts as well as

performance measures over the medium-term (Horizon 2) and thereafter (Horizon 3).

There is early evidence that the scale and quality of medical research has increased

as a result of Healthy Futures, and that more collaborative and translational research

behaviours are being embedded among researchers.

Overall, the projects have delivered on the outcomes anticipated to date, including

attraction of world- class researchers, expansion of training opportunities and access

to capital infrastructure and equipment, improved access to competitive funding,

increased use of platform technologies and collaboration through colocation. Tangible

signs of new outputs of research (health interventions, treatments and medicines and

associated downstream economic impacts) are expected in future Horizons.

Each of the Healthy Futures initiatives is well placed to deliver on their objectives,

understanding their contributory and facilitative role in achieving improved health

outcomes.

This interim evaluation draws together several findings in terms of what is already

working well and highlights important considerations for future assessment of the

suite of Healthy Futures initiatives.

Overview of Healthy Futures

The Victorian Life Sciences Statement: Healthy Futures (referred to hereon as Healthy Futures)

is a $230.45 million capital investment to enhance the health and economic wellbeing of

Victorians and maintain the state’s competitive strength in medical research. In essence,

Healthy Futures:

■ created new or expanded world class medical research infrastructure to expand

capacity and help attract the best and brightest people to generate commercial and

clinical opportunities from a world-class research base; and


www.TheCIE.com.au

■ enhanced access to health services and workforce, health information, and health

priorities. This access will ultimately contribute to access to new treatments for

Victorians — the best medicines, treatments and healthcare and hence to ensure they

continue to enjoy high quality health services.

The stated objectives of the Healthy Futures initiative at its time of development are

outlined in box 1.

1 Stated objectives of the Healthy Futures program

The Victorian Life Sciences Statement emphasises the ‘unique window of opportunity’ to

build upon existing strengths and take the next steps in medical research to solve

critical health problems. The following objectives of Healthy Futures are specified.

■ Capture new opportunities in areas of research where Victoria has critical mass

and a competitive advantage.

■ Attract investment and generate high quality jobs by maintaining and growing the

international reputation of Victoria’s research institutes.

■ Maximise opportunities for continued growth in national and international

collaborations and partnerships.

■ Deliver major benefits to business and industry by encouraging the

commercialisation of medical research.

■ Create a healthier future by enabling research to translate speedily into practical

health benefits for the entire community.

Healthy Futures captured an important moment in time, successfully capitalising on a

limited ‘window of opportunity’ in terms of the availability of financial capital and the

congruence of political will at the State and Commonwealth level. The Healthy Futures

investment was motivated by the need to address space constraints and overcrowding of

medical research infrastructure and to realise potential scale economies and increased

productivity of labour from increased collaborative research efforts.

It was also initiated during a period of substantial innovation investment by the Victorian

Government, reflecting a commitment to stimulate infrastructure and capability for

innovation to underpin long-term economic and social outcomes for the state. Healthy

Futures also sought to leverage additional philanthropic and Commonwealth funding

available given the favourable economic and financial landscape at the time.

While this creates challenges in attributing changes to Healthy Futures, it does mean that

the ultimate impacts associated with Healthy Futures are much larger than those that

would have been achieved in isolation.

In summary, Healthy Futures made several important contributions to the value

proposition for Victoria as being the place to do medical research in Australia, and

indeed the world. It also contributes to the broader long-term objective of boosting the

level of high quality medical research and associated clinical translations.


www.TheCIE.com.au

Evaluation approach

The evaluation methodology for Healthy Futures addresses the appropriateness,

effectiveness, and efficiency of the suite of projects within a program logic framework.

It spreads the program logic across three distinct time horizons given the long time lag

between research and workforce development, discovery, development, and translation

into clinical applications. In the case of Healthy Futures, the time lag is extended further

due to additional time required to secure the necessary resources associated with

capitalising effectively on infrastructure spending — which required additional leveraged

capital funding, leveraged operating funding, recruitment and purchasing before

additional research activities could get underway.

An innovative feature of this interim evaluation is that it tackles key challenges in

attribution for the Healthy Futures set of projects, particularly by incorporating qualitative

approaches to develop a ‘systems’ view of changes associated with the program. For

outcomes to date, inevitably many are associated and/or in some way linked to earlier or

subsequent measures. In addition, there is an extended payback period for these types of

investments and early signs of potential future impacts need to be assessed, despite

intensified challenges of attribution over the longer term.

The key methodological steps involved in this interim evaluation included the following.

■ Literature review — the information gathering phase reviewed relevant available

information regarding the ‘what’, ‘why’, and ‘how’ of the Healthy Futures initiative.

This included a review of Healthy Futures program documents and status updates,

Commonwealth and State Government policy statements, institutional annual

reports, independent evaluation reports and wider literature.

■ Stakeholder consultation — face–to–face meetings were held with each Healthy

Futures program beneficiary (for instance, various medical research institutes (MRIs),

providers of tertiary health services, CSIRO and other enabling medical research

infrastructure providers) and relevant Victorian government agencies.

■ Data collection — existing Healthy Futures evaluation surveys from 2008 to 2012 were

supplemented by open-ended stakeholder questioning. Bibliometric publications and

citations data, National Health and Medical Research Council (NHMRC) funding

data, Cooperative Research Centres (CRC) funding data, National Survey of

Research Commercialisation data (including regarding the scale of MRI patents,

licences and invention disclosures and the number of spin-off companies) and data

from the Victorian Government’s program of support for independent Medical

Research Institutes were also collected and analysed.

■ Quantitative analysis — Computable General Equilibrium modelling was undertaken

to estimate the indirect impacts of increased capital expenditure associated with

Healthy Futures to date.

■ Qualitative analysis — focus was placed on case studies of behavioural changes and

impacts, attracted expertise and associated new research outcomes, an assessment of

governance arrangements and horizons assessment of future impacts.

The methodological steps discussed above were used to undertake a horizons assessment

of Healthy Futures — meaning that each component of the program logic framework was


www.TheCIE.com.au

considered in the context of timing of incidence. As an interim evaluation, this

assessment focuses on evaluating Healthy Futures in the program logic framework over

Horizon 1 (that is, from 2006 to 2012 only).

The interim evaluation also attempts to identify early signs and indicators of impacts as

well as performance measures over the medium-term (Horizon 2) and thereafter

(Horizon 3). However, later evaluation of Healthy Futures will determine the full scope of

impacts and benefits over these future periods.

A summary of the appropriateness, efficiency and effectiveness of Healthy Futures as at

2012 is set out in chart 2.

Assessment of performance

The assessment above builds upon DBI’s existing Performance Monitoring Framework

to identify key indicators of program outcomes/impacts over various periods in order to

assess the appropriateness, effectiveness, and efficiency of the program elements. Chart 3

below summarizes the Performance Monitoring Framework developed by DBI for

Healthy Futures, and identifies additional assessment measures that were developed as

part of this interim evaluation.

The chart also shows which indicators are believed to be directly attributable to Healthy

Futures (pink) or partly attributable to Healthy Futures (black).

Assessment of Horizon 1 impacts demonstrates that Healthy Futures has made a genuine

difference to the scale and scope of medical research, workforce development and

enabling technologies in Victoria. All elements of the program are assessed to be largely

appropriate as a single event package of measures,1 although the extent to which they fill

gaps in existing capacity and enhance the productivity of research inevitably varies.

Healthy Futures has already delivered a range of economic and social returns to Victoria.

In addition, returns to date are expected to grow in future as the induced scale of

research, and changes in collaborative behaviours lead to increased research output,

clinical translations, and improved health outcomes. Indeed the outputs of Healthy

Futures are found to be critical to the probability of future success. Its value also lies in

improving the efficiency of other research and innovation spending in the State.

Healthy Futures projects have been positive enablers to achieving the overarching

objectives of the program, particularly in terms of positive workforce impacts, research

outputs and the branding value for Victoria as a destination for medical research.

1 ‘Single event’ refers to Healthy Futures as being a one-off investment in medical research

infrastructure. Hence, the positive assessment of Healthy Futures does not necessarily infer that

additional medical infrastructure investment would be warranted and/or would generate the

same benefit profile.

8


www.TheC

IE.com

.au

2 Interim assessment of Healthy Futures 2006–2012 (Horizon 1)

Ob

jecti

ves

Ob

jecti

ves

Ob

jecti

ves

Ob

jecti

ves

Horizon 1Horizon 1Horizon 1Horizon 1 2006 2006 2006 2006 ---- 2012201220122012 Financial Impact to DateFinancial Impact to DateFinancial Impact to DateFinancial Impact to Date CommentsCommentsCommentsComments

Inp

uts

Inp

uts

Inp

uts

Inp

uts

■ Total leveraged funding of

$508.1 million ($2.20 per public

$ invested)

■ Access to new equipment (e.g. bioprocessing facility) and enabling

technologies (e.g. VeRSI)

■ Access to new sources of operating expenditure from competitive funding

and philanthropic/other sources (HF seeded new funding sources). For

instance, the Australian Cancer Data Grid has been awarded independent

grants for ongoing operation (e.g. Roche, Sanofi Aventis, Novartis grants)

Assessment of appropriateness: Achieved

Ou

tpu

tsO

utp

uts

Ou

tpu

tsO

utp

uts

■ Increase Victoria GSP by about

$170 million per annum

■ Increase household

consumption by around $77

million per annum

■ 1480 employment years created

since 2006

■ Career paths for next generation researchers

■ Attraction of world-class international researchers . For instance, eight of

the twelve group leaders working at the ARMI were high profile

international recruits

■ Boost to scale, quality and functionality of research facilities (with benefits

in terms of greater collaboration and multi-disciplinary teams). For

instance, space constraints for WEHI researchers were solved and teams

were bought closer together into a common facility

■ Workforce enhancement via new training opportunities. For instance, new

research positions available at Healthy Futures institutes and more

medical university places available in rural and regional Victoria

Assessment of efficiency: Achieved

Ou

tco

me

sO

utc

om

es

Ou

tco

me

sO

utc

om

es

■ Increased clinical trials

providing increased income for

hospitals

■ Annual fee for service income of

over $1 million from industry

accruing to the Bioprocessing

facility

■ Maintenance of access to competitive Commonwealth research grants

(e.g. NHMRC, CRC, Commonwealth Education Investment Fund, ARC,

CSIRO)

■ More strategic prioritisation of research. For instance, the Victorian

Cancer Agency focuses on priority areas for improving health outcomes.

■ Providing for a faster, more efficient process to conduct clinical trials at

multiple sites

■ Upskilling the medical workforce

Imp

acts

Imp

acts

Imp

acts

Imp

acts

�� GSP impacts associated with enhanced expenditure from attracted labour from interstate/ overseas and additional capital investment

�� Small but positive increase in productivity of labour employed in medical research in Victoria

Assessment of effectiveness: Achieved

Healthy

Futures capital

funding

Attracted funding to complete

capital builds

New

equipment

Leveraged funding for operating

costs

Increased output and employment in the building and

construction phase

New platform technology

resources developed

New purpose-built facilities to support biomedical research and

capital improvements in

medical/research training

People involved in training courses on the use of Healthy

Futures funded infrastructure

Enabling facilities underpinning multiple institutions and

�access to equipment

Increased number of new medical training places

Access to ‘attracted’ world

class scientists

� capacity and uptake of PhD/Masters candidates under supervision in HF

supported activity

Increased medical and clinician

researchers, engineers and

support staff

Expatriate researchers and

clinician researchers

attracted back to Victoria

Steady/� access

to peer reviewed

(NHMRC/ARC)

research funding

Use of new

platform

technologies by

research groups

� collaboration

through

colocation

Capture new opportunities in

areas of research where

Victoria has critical mass and

a competitive advantage

Attract investment and generate high

quality jobs by maintaining and

growing the international reputation

of Victoria’s research institutes

Maximise opportunities for

continued growth in national and

international collaborations and

partnerships

Deliver major benefits to business

and industry by encouraging

the commercialisation of

medical research

Create a healthier future by

enabling research to translate

speedily into practical health

benefits for the entire community

Key

Achieved in full

Achieved as expected to date


9

www.TheC

IE.com

.au

3 Enhanced Performance Monitoring Framework tool

Indicators from DBIs Performance Monitoring Framework Additional indicators developed as part of this interim review

AP

PR

OP

RIA

TE

NE

SS

EF

FE

CTIV

EN

ES

S

EF

FIC

IEN

CY

LE

SS

ON

S

LE

AR

NE

D/

FU

TU

RE

DIR

CE

RTIO

N

Data source: The CIE

Evidence that medical research and training

capabilities were

inadequate

Evidence of potential loss of competitive

advantage

Consistency with DBI and broader Government

objectives

Impact on attraction and retention of

researchers, clinicians

and future leaders

Capacity to rapidly translate medical

research into improved

health outcomes

Maintaining/growing Victoria’s biomedical

research capabilities

Increase in quality jobs in innovative

industries

Increase in high quality

education and training

Recognition of Victoria as a world leader in biomedical research

and technology

Leveraging non-state Government funding for

facilities and

infrastructure

Attracting investment in and maximising

new research

opportunities

Enhancing the Victorian medical and research

training environment

Encouraging national/international linkages, partnerships

and collaboration

Increasing research focus on clinical / commercial

outcomes

Enhancing capacity, efficiency and

integration of existing facilities

Lessons from delivery of Healthy Futures

to date

Opportunities for future improvement in design

and delivery of

Healthy Futures

Adequacy of resources to enable Healthy Futures to achieve its

objectives

Efficiency of infrastructure

delivery

Efficiency in the use of

DBI resources

Efficiency of administrative

costs

Alignment with Commonwealth

Government objectives, policy and

investments

Alignment with the goals of

non–Government

stakeholders

Focus on additionality

(non-duplicative and

game changing)

Focus on system-wide

performance

Alignment with best practice research

trends

Primarily support for basic, pre-commercialisation

and publicly funded

research

Investment unlikely to crowd out commercial activity given market

failures

Creation of career paths for the next generation

of researchers in

Victoria

Impact on access to

new equipment

Direct economic benefits — Victorian GSP, employment, household

consumption

Strengthen Victoria’s reputation and international

presence

Attract competitive funding, industry/philanthropic support for operating

expenditure

Achieving scale and

integration

Opportunity for prioritisation of

research activity

Increase industry partnerships, spin-off

companies and

commercialisation activity

Increase clinical trial

activity in Victoria

Key

Evidence of the impact directly attributable to Healthy Futures

Evidence of the impact partly attributable to Healthy Futures


www.TheCIE.com.au

In terms of outputs:

■ Healthy Futures has already realised positive impacts on access to new equipment, both

directly and indirectly. For instance, Healthy Futures has directly provided Victorian

researchers access to a bioprocessing facility (creating a substantial private revenue

stream of $1.2 million annually by making access open to industry); equipment

required for research using bioresources (at a bioresources facility established for the

Austin Biomedical Alliance); and New Grid infrastructure. Indirectly, the impending

availability of new infrastructure and larger research capacity provided the impetus

and business case for advanced biomedical imaging equipment (Victorian Biomedical

Imaging Capability) in facilities across Melbourne, including the new Neuroscience

facilities;

■ in the majority of cases, Healthy Futures has been an effective recruitment tool and has

assisted to meet workforce challenges across the spectrum of need. It has already been

successful in assisting Victoria to attract world-class researchers. For instance, eight of

the twelve group leaders now working at the Australian Regenerative Medicine

Institute (ARMI) were high profile international recruits. In addition, the Walter and

Eliza Hall Institute (WEHI) has attracted back key research staff after various stints in

overseas research institutes following Healthy Futures investments. Healthy Futures has

also better enabled the creation of career paths for the next generation of researchers

in Victoria by enhancing the quality and quantity of research opportunities;

■ Healthy Futures has improved workforce development by facilitating improvements in

the quality of training and development, including enhancements to facilities in rural

and regional areas. For instance, the Medical University Places Capital Infrastructure

program built teaching, training and research facilities, tutorial rooms, libraries,

student accommodation and other education amenities to a) improve opportunities

for medical training in regional Victoria, and b) encourage medical graduates to work

in rural areas. In addition, increased physical capacity at Healthy Futures medical

research institutes has increased the number of research training positions available.

For instance, the WEHI currently has 80 PhD students, of which some are directly

attributable to the additional physical space that has resulted from Healthy Futures;

■ as well as bringing direct economic benefits through net capital inflow, Healthy Futures

also strengthens Victoria’s international presence and reputation as a world-class

destination for medical research. Increased reputation and leadership of Victorian

research is evidenced by, for instance, by the ARMI being selected to host the

European Molecular Biology Laboratory (EMBL) Australia. Associate Membership

with this global centre of excellence2 provides Australian researchers with access to

state-of-the-art infrastructure and technologies, expertise and networks in Europe; new

opportunities to forge collaborations with European researchers and institutions; and

access to European funding opportunities. ARMI has also recently attracted a

partnership with Japan’s Systems Biology Institute highlighting the ARMI’s success in

forging strong links with internationally recognised research organisations.

■ Healthy Futures has improved opportunities for clinical translation. Clinical practice

aligned and integrated with discovery research is a strong feature of research culture

2 EMBL is the most cited scientific institution outside of the USA in molecular biology and

genetics, with an extremely high impact of an average of 51.9 citations per paper.


www.TheCIE.com.au

and practice, which underpins Victoria’s reputation and performance in health

sciences. This was given a significant boost through the Healthy Futures investment.

– At the WEHI, new facilities have enabled much closer interaction with clinicians

at the Royal Melbourne Hospital. In addition to direct benefits resulting from more

rapid and efficient translation of research findings to clinical practice, this new

association has encouraged a more active research interest amongst clinicians,

which in turn is likely to result in improved clinical care and outcomes for patients.

– Further, a new test developed by researchers at the Burnet Institute offers a simpler

and cheaper way of monitoring HIV in sufferers and a wristwatch that

continuously monitors the health of patients with Parkinson’s disease developed by

Australian Centre for Neuroscience and Mental Health Research (ACNMHR)

scientists was announced at the international BIO2009 Conference.

At this relatively early stage, Healthy Futures has achieved several successful outcomes:

■ it has achieved scale and integration, which has helped grant recipients continue to

attract major national and international competitive funding, as well as industry and

philanthropic support;

■ it has enhanced collaborations on new ‘platforms’ underpinning future medical

science outcomes;

■ it has provided the opportunity for prioritisation of research activity following

integration of research institutions and teams. For instance, investment by Healthy

Futures in the Victorian Cancer Agency (VCA) enabled a coordinated approach on

strategy for cancer research in Victoria. Its strategy is increasingly to focus on priority

areas for improving health outcomes. The strategy includes initiating new research

areas where there is clear evidence that current research effort is misaligned relative to

priority health issues;

■ it has led to some early signals of success in indicators of clinical translation of

research efforts of MRIs including new and pre-existing international patents and

income from clinical research projects; and

■ there are also positive developments in clinical trial activity that are partly attributable

to Healthy Futures. The streamlining of ethics approval for multi-site trials has directly

addressed time and cost factors that influence trial location decisions for multinational

pharmaceutical companies and multi-site clinical trial activity in Victoria over the last

few years. This will have been at least partly facilitated by Healthy Futures. Data from

the Australian and New Zealand Clinical Trials Registry shows an increasing trend in

the number of registered trials occurring at the broader state level.

To date there are few early examples of industry partnerships induced by Healthy Futures.

An exception is Victorian technology company Global Kinetics Corporation, which

worked closely with researchers from the Florey Neuroscience Institutes to develop a

prototype micro-electro-mechanical wristband device. It is expected that spin off impacts

for biotechnology start ups are expected to increase and evolve in a way that is at least

partly attributable to various Healthy Futures projects. This is because there is a strong

supply side element to attracting downstream activity in medical research in Victoria.

Biotechnology start–ups and existing biotech and pharmaceutical companies are attracted

by a strong supply base of people, skills, and intellectual property — all of which have

been enhanced by Healthy Futures.


www.TheCIE.com.au

It is also expected that the increased reputation brand and leadership of Victoria’s

medical research sector will be evidenced in the future by additional extramural staff

attraction and in rising research income sourced internationally. Further, the quality and

quantity of Victorian research outputs are expected to continue to rise evidenced by

increasing publication rates and citations received by Victorian researchers.

Interim impacts for Victoria

Healthy Futures seed funded a total program spend of $932 million in Victoria, leveraging

over $701 million from other Government and non-Government sources that is unlikely

to have been invested without Healthy Futures.

On average, Healthy Futures funding leveraged $4 for every dollar invested by the

Victorian Government, which varied across projects from $1.20 to $18.60 per dollar

invested. Including financing from the Victorian Government in addition to Healthy

Futures, each dollar of state government funding leveraged $2.20 from Commonwealth,

philanthropic, institutional and other sources.

Based on economy-wide modelling on Victoria, by 2011-12, Healthy Futures is estimated

to have increased Victorian GSP and household consumption by approximately

$170 million and $77 million respectively. Of this, $81 million represents the return to

labour, which based on average wage rates in Victoria, is equivalent to close to 1 480

employment years created. Large future economic impacts are expected over subsequent

Horizons, which are expected to show more direct benefits in terms of research

collaboration, industry partnerships, and ultimately improved clinical treatments and

therapies that will improve health outcomes.

■ Healthy Futures has made a strategic and lasting impact on the scale and quality of

research activity and behaviours to make it more likely that successful clinical

translations will result over time, producing improved health outcomes over the

medium to longer term.

■ Given the nature of drug development and the need for MRI initiated research to be

developed by start up companies and/or downstream users, it is highly likely that

Healthy Futures has and will generate positive impacts in terms of spin off companies

over time.

■ In terms of future research outputs, Horizon 2 is expected to witness an increase in the

number of international publications in peer reviewed journals, while in Horizon 3,

the number of citations attracted by Victorian-authored publications is expected

indicating the rising quality of research outputs.

In essence, Healthy Futures is expected to have made a lasting impact to the foundation

that matters in terms of probability of successful, outcomes based research.

Key evaluation messages from this review

The interim evaluation of Healthy Futures is a positive one and there are findings that

reinforce positive elements of the program and serve to highlight areas where future effort


www.TheCIE.com.au

and resources may need to be brought to bear in order to best ensure that the potential

benefits from the investment are realised.

Governance and administration

Projects of the size of Healthy Futures take considerable resources to govern and

administer well. The following levels of governance are recognised as being important in

assessing Healthy Futures:

■ project — implementation and subsequent management for each Healthy Futures

initiative, particularly the delivery of intended outputs — this is the responsibility of

the leaders of the institutions granted Healthy Futures funding (e.g. WEHI, VCA);

■ program — oversight and review of the performance, outcomes and impacts for the

overall Healthy Futures program — this is the responsibility of the DBI; and

■ policy — oversight and review of the appropriateness and effectiveness of Healthy

Futures program in relation to higher-level, longer-term, and wider-ranging

government goals for health and medical sciences in Victoria — this is the

responsibility of the Victorian Government.

Project

Healthy Futures has a ‘project’ governance component, given that various discrete projects

each have their own strategic intent and high level governance structure in place. There

are examples where this project level of strategic governance was very sound. The WEHI

was an exemplar in this regard, complementing overall governance through its

Board/Audit and Risk Committee with a dedicated project oversight committee. The

high-level New Building Sub-Committee met monthly over the course of the project. The

Sub-Committee received full executive support from the Institute and was attended by

key project consultants (project management, architects, and quantity surveyors) and

additional consultants as required. The Sub-Committee also included opportunities for

DBI and Department of Health participation from the outset. In additional to steering the

capital development, the Sub-Committee monitored the Institute's obligations to funders

of the project and compliance with all reporting requirements. This approach ensured

strong governance for the specific project and strengthened communication with DBI and

its higher-level (program) governance role.

Project governance was arguably less strong with respect to the Australian Cancer Data

Grid project, where there could have been stronger emphasis on business planning,

strategic intent in the activities undertaken and focus on bringing the new platform to

end-users. Greater emphasis on these elements should assist with building broader

stakeholder support and more widespread use of project outputs.

In terms of enhanced project level governance, in addition to regular reporting to DBI,

leaders of the respective projects within the Healthy Futures program could be involved in

more regular and collaborative self-review, with reflections shared among program

beneficiaries. This would encourage active learning across the Healthy Futures

‘community’, through which governance of individual projects should improve over

time.


www.TheCIE.com.au

Program

Healthy Futures was a grants-based program, with much of the project management the

responsibility of medical research institutes and agencies.

DBI’s program governance role (oversight and review of performance, outcomes and

impacts of the overall Healthy Futures program) is considered appropriate. DBI had

generally sound strategic governance structures in place for interacting with each Healthy

Futures project, with interdepartmental consultation established wherever relevant,

arranged as required on an as-needed basis. These arrangements generally enabled DBI

to be close enough to projects to genuinely ensure their most appropriate, effective and

efficient implementation, with some qualification relating specifically to the

administrative resources allocated to project oversight.

With no specific allowance made in the program budget for government administration,

in-kind government resources were likely to have been spread too thin. The total

administrative costs of the Healthy Futures program were estimated by DBI to be

$1.66 million since program inception to June 2012. This represents 0.7 per cent of the

Healthy Futures investment of $230.45 million and 0.4 per cent of total program costs

when subsequent Victorian Government funding is included. We note however, the

substantial in-kind support that was mobilised by the funding recipients to deliver Healthy

Futures projects.

Australian National Audit Office (ANAO) suggests that ‘there is no apparent benchmark

for the ratio of administrative costs to program costs. Costs observed in audits of grant

programs range from as low as 1 per cent to as high as 35 per cent’.3 The appropriate

share of administrative costs to program funds will vary according to:

■ the size of the program — smaller programs will have a higher proportion of fixed

costs, which will raise the proportion of administrative costs to program funds; and

■ the risk associated with the program — higher risk individual projects or programs

will require more costly risk treatments including closer monitoring.

Given these factors, it is likely that the administrative resources allocated to Healthy

Futures were too limited. The ANAO suggests that ‘insufficient administrative resources

to manage grant programs increase the risk that the program’s objectives may not be

achieved in an efficient, effective and timely manner’. That said, administrative resources

do appear to have been managed effectively for contract management and project

oversight activities.

DBI as lead agency also undertook extensive reporting to Cabinet, in partnership with

the Department of Health. This was particularly the case in the period from inception to

late 2010, which was a critical period in the development of each initiative. No criticisms

were raised by any stakeholders on the role of DBI in managing and reviewing any of the

Healthy Futures projects.

3 Australian National Audit Office 2002, Administration of Grants: Better Practice Guide, p.14.


www.TheCIE.com.au

Policy

Healthy Futures is a good example of governance working well at the ‘policy’ level.

Healthy Futures is a highly strategic set of initiatives that are well complemented by, and

consistent with, the broader innovation framework of the Victorian and Commonwealth

Governments. Consistent with the Commonwealth’s innovation strategy Powering Ideas

and the Victorian Government’s recent Technology Plans for the Future, Healthy Futures

focuses on enhancing Victoria’s competitive strength and reputation in science and

innovation and continued priority investment in medical research. Healthy Futures had

longer-term expectations in mind, which were captured in the specific objectives defined

at the time of program inception, and have continued to be referred to by program

beneficiaries when reflecting on Healthy Futures today.

The importance of enabling ‘system’ evaluation capability in future

Longer-term impacts of a program such as Healthy Futures will be dispersed through the

medical sciences ‘system’ in Victoria. Future evaluation of Healthy Futures should capture

if, and how, the ‘system’ performance targets are being met over time. This is particularly

important given that impacts genuinely attributable to Healthy Futures are likely to be

small and understate their true value.

Improved data collection could help to achieve this, possibly facilitated by more

administrative resources being dedicated to project oversight. For instance, not all

recipients provided evaluation surveys to DBI and in most cases, responses were

incomplete. While we note that the data collection tool was less relevant to some

projects, while others were subject to separate and specific program evaluations,

incomplete survey data has created challenges for this evaluation and will continue to do

so. It is therefore important that greater compliance with reporting responsibilities be

achieved.

Further, more detailed and timely information on medical science, public health and

business innovation metrics for Victoria, to help assess changes in priority policy areas

over time. It is also important to focus on ‘drivers’ of medical sciences performance and

outcomes.

For example, bibliometric assessments of research outputs and their impact — by subject

area and institution — can be valuable for understanding the international

competitiveness of research capability in Victoria and therefore the potential to compete

beyond Victoria for research funding. Similarly, business innovation metrics for firms

involved in developing and deploying medical technologies derived from research in

Victoria’s medical research institutes can help understand constraints and opportunities

affecting business growth.

The current Healthy Futures survey tool is well designed and appropriate. However,

incomplete responses to date limit their usefulness for program evaluation and data

analysis purposes. In order to decrease the compliance burden and increase response

rates, consideration may be given to sourcing data through existing structures. For

instance, MRIs could be asked to provide responses to the National Survey of Research

Commercialisation and elaborate where deemed appropriate. Further, bibliometric data


www.TheCIE.com.au

may be sourced from independent sources, which would reduce the compliance burden

on MRIs, and also allow for comparative data analysis.

Options for future investment should be explored

Collective effort to increase access to operational funding is likely to be needed to realise

greater value from the infrastructure investments made through Healthy Futures.4 This is

likely to depend on leadership to facilitate a collaborative effort for identifying and

capitalising on funding options (including Commonwealth and international funding

agencies, commercial and philanthropic investors), thereby strengthening Victoria’s

overall competitiveness in attracting such investment.

Securing ongoing operational funding is an important issue, although it is not a direct

issue for this evaluation. All stakeholders were clear during the development of the

initiatives that operating funding was not provided for under Healthy Futures and all

stakeholders agreed that operating funding would be obtained from existing funding

sources.

Operating funding is an issue being considered in the McKeon Strategic Review of Health

and Medical Research in Australia, which advocates that MRIs receive at least 60 per cent

indirect cost loading for national competitive grants. In its response to this consultation

paper, the Victorian Government emphasised the importance of predictability and

certainty of funding for health and medical researchers over the longer-term. Consistent

with this view, it is important to articulate what the investment options are for building

on Healthy Futures achievements to date.

Consideration should also be given to encouraging and rewarding successful

implementation. As most Healthy Futures projects are still at relatively early stages

(relative to their longer-term value), there is plenty of scope to influence the overall

performance of the investments made.

Further State Government investment in medical sciences should therefore encourage

and reward project recipients to maximise the impact of the (sunk) investment already

made. This could include on-going expectations that project recipients define what is

being done, and what more is needed to maximise long-term value, as part of future

funding requests related to the infrastructure established through Healthy Futures.

Tell the Healthy Futures Story

It is important to promote the success and value of the strategic investment in Healthy

Futures. As investment priorities and mechanisms vary over time, there is merit in

ensuring a record of how different approaches work, at different stages of sector

development and maturity.

4 Operational funding may refer to a) indirect cost funding, b) direct research funding from non-

Victorian Government sources and c) equipment and infrastructure funding. Future references

to operational funding include all of the above.


www.TheCIE.com.au

Healthy Futures was implemented during a period of substantial infrastructure investment

in medical sciences and other innovation areas in Victoria. The aggregate impact of

these investments is inevitably complex and hard to track over time.

This means it may be difficult to ensure future policy makers appreciate critical features

of this investment approach. The Healthy Futures story should therefore be recorded in a

format that is easily absorbed by future policy audiences. This interim evaluation of

Healthy Futures will be important in this regard.

Summary of key findings

■ Key finding #1: Healthy Futures seed funded a total program spend of

$932.3 million, leveraging over $508 million from non-Victorian Government

sources that is unlikely to have been invested without Healthy Futures.

■ Key finding #2: On average, Healthy Futures and subsequent Victorian

Government funding leveraged $2.20 from Commonwealth, philanthropic,

institutional and other sources for every dollar invested by the State Government.

This varied across projects from $1.00 to $4.40 per dollar invested.

■ Key finding #3: Healthy Futures has had a positive impact on access to new

equipment, both directly and indirectly. In the case of the bioprocessing facility, a

private revenue stream has already been created. While this is a positive outcome

and demonstrates private value, it remains the case that Healthy Futures

investments were not intended to establish profit-making centres, but rather to

support and advance Victoria’s medical research base.

■ Key finding #4: By 2011-12, Healthy Futures is estimated to have increased

Victorian GSP and household consumption by approximately $170 million and

$77 million respectively. Of this, $81 million represents the return to labour,

which based on average wage rates in Victoria, is equivalent to close to 1 480

employment years created.

■ Key finding #5: Healthy Futures has enhanced the quality and quantity of research

training opportunities and enabled career paths for early career researchers in

Victoria, and has significantly increased the capacity for regional medical training.

■ Key finding #6: Healthy Futures has already been successful in assisting Victoria

to attract world-class researchers. As well as bringing direct economic benefits

through net capital inflow, it also strengthens Victoria’s international presence

and reputation.

■ Key finding #7: Healthy Futures has enhanced collaborative approaches to

research, including through e-Research and ICT-enabled platforms and has better

enabled clinicians to be engaged with research.

■ Key finding #8: In the few years since Healthy Futures investments became

operational, Victoria has been able to maintain its ‘first-place’ status in terms of

access to competitive Commonwealth research grants.


www.TheCIE.com.au

■ Key finding #9: Healthy Futures was an important first step in helping to improve

the viability of a clinical research environment in Victoria. The streamlining of

ethics approval for multi-site trials has directly addressed critical time and cost

factors that influence trial location decisions for multinational pharmaceutical

companies and multi-site clinical trial activity in Victoria over the last few years

will have been at least partly facilitated by Healthy Futures.

■ Key finding #10: Healthy Futures has had an important impact on supporting the

conditions for future positive translations from research to clinical practice and

treatments.

■ Key finding #11: Healthy Futures has made a strategic and lasting impact on the

scale and quality of research activity, and behaviours it has reinforced make it

more likely that successful clinical translations will result. It is the finding of this

evaluation that improved health outcomes will be a result of the program over the

medium to longer term (Horizons 2 and 3).

■ Key finding #12: Given the nature of drug development and the need for MRI

initiated research to be developed by start up companies and/or downstream

users, it is highly likely that Healthy Futures will generate positive impacts in

terms of spin off companies.

■ Key finding #13: The quantity and quality of Victorian research outputs have been

rising steadily since 2002 and indicate that, in terms of research output, Victoria

is currently ‘punching above its weight’. Healthy Futures is expected to reinforce

this trend. Horizon 2 is expected to witness an increase in the number of

international publications in peer reviewed journals, while in Horizon 3, the

number of citations attracted by Victorian-authored publications is expected,

which would indicate the rising quality of research outputs.

■ Key finding #14: Healthy Futures is expected to have an important impact on the

international profile, reputation and leadership of Victorian research. Healthy

Futures initiatives have already attracted world class researchers and peaked the

interest of global centres of excellence such as European Molecular Biology

Laboratory and Japan’s Systems Biology Institute.

■ Key finding #15: The Department of Business and Innovation should continue to

promote and facilitate the linkages between medical research institutes and

private industry in order to help institutes diversify their funding base.


www.TheCIE.com.au

THE CENTRE FOR INTERNATIONAL ECONOMICS

www.TheCIE.com.au

FINAL Healthy Futures Evaluation Executive …...Summary 4 Overview of Healthy Futures 4 Evaluation...

Documents

Transcript of FINAL Healthy Futures Evaluation Executive …...Summary 4 Overview of Healthy Futures 4 Evaluation...