27 November 2017

Final Report Pacific Women Shaping Pacific Development Support Unit

Prepared by Matthew SmithMarica Tabualevu

i

TABLE OF CONTENTS

Acronyms 1

Executive Summary 2

Background 7Support Unit 7Evaluation Purpose 8Approach 9Key Limitation 9Structure of the Report 10

Analysis and Findings 11Effectiveness 11Efficiency 22Monitoring and Evaluation 30Value for Money 31Relevance of the Modality 34

Recommendations 41

Conclusion 44

Annex 1: List of People Consulted 46

Annex 2: List of Documents Reviewed 49

Annex 3: Support Unit Organogram 51

Annex 4: Expenditure against Annual Work Plan 53

Annex 5: Comparison of Different DFAT Funded Facilities 64

Annex 6: Value for Money Analytical Matrix 70

Annex 7: Terms of Reference 73

ii

iii

Acronyms

ARF DFAT’s Adviser Remuneration FrameworkAWP Annual Work PlanDFAT Department of Foreign Affairs and TradeEQ Evaluation QuestionET Evaluation TeamEWAW End Violence Against WomenFGD Focus Group DiscussionFSM Federal States of MicronesiaFSVAC PNG Family and Sexual Violence Action

CommitteeFY Financial YearGFP Gender Focal PointKMS Knowledge Management SystemLES Locally Engaged StaffLTA Long Term AdvisorM&E Monitoring and EvaluationMC Managing ContractorMELF Monitoring, Evaluation and Learning

FrameworkPMC Program Management CommitteePNG Papua New GuineaPPA Partner Performance AssessmentPW Pacific WomenSTA Short Term AdvisorSU Support UnitTL Team LeaderToR Terms of ReferenceVfM Value for MoneyWEE Women’s Economic Empowerment

iv

Executive Summary

This evaluation has been commissioned by the Australian Department of Foreign Affairs and Trade (DFAT) and covers the period from April 2015 – June 2017. DFAT’s standards for monitoring and evaluation (DFAT, 2017) were strictly adhered to in the design and implementation of the evaluation.

A mixed methods approach was used by the Evaluation Team to provide the requested independent assessment of the Pacific Women Shaping Pacific Development Support Unit (Support Unit), and to also inform DFAT’s considerations on whether to extend the term of the Support Unit’s contract in either its current form or a revised form. Moreover, the evaluation complements the recently completed year-three evaluation of Pacific Women Shaping Pacific Development.

In response to the first evaluation question regarding the efficacy of the Support Unit, the evaluation team found that operational services provided by the Support have largely either met or exceeded the items listed in the Head Contract and Annual Work Plans, albeit the quality of the services provided have not always met expectations. Operational Support is a key strength being provided by the Support Unit, but strategic policy advice has yet to be seen by DFAT as similarly effective.

Nevertheless, the Support Unit has made several noteworthy achievements over the past two years (such as the Knowledge Management System and the Annual PNG Partner Workshops), albeit certain key deliverables (such as the Capacity Strategy and the Roadmap on Women’s Economic Empowerment, Women and Leadership and End Violence Against Women) have taken significantly longer than expected. Whilst these activities were contracted deliverables we believe that they should be recognised as achievements bearing in mind the challenges posed in providing support across a region where the country context is often complex and varies enormously between countries.

Activity management, monitoring and reporting support provided by the Support Unit are well regarded by partners, as is the logistical support provided by the Support Unit. Whilst programme level monitoring and reporting (such as the six monthly reports and other operational reporting) is a strength of the Support Unit, the production of other reporting related deliverables as specified in the Annual Work Plan (AWP) has not been as effective. Quality assurance of these deliverables, such as the Roadmaps and Counselling review, has been a constant challenge for the unit.

When the Support Unit has been given an opportunity to implement innovative ideas (such as with regards to the CSO Leadership Dialogue, and the manner in which the PNG sub-office has implemented an effective network), it has largely been successful. Nevertheless, there continues to be strong differences of opinion as to whether the Support Unit is sufficiently proactive in leveraging opportunities to be innovative, and/or whether the modality currently inhibits or promotes innovation.

With regards to whether the Support Unit is actively promoting Pacific ownership, the evaluation found that it was, but that greater ownership could be promoted through expanding the diversity of locally engaged staff. The evaluation team also found that the Support Unit has contributed to a number of positive outcomes, including the role the unit has played in convening and coordinating like-minded donors, regional organisations and programs across the Pacific. Whilst these convening and coordinating functions are what is expected of the unit as per its contract and AWP, recognition should still be made of the fact that much of the coordination and convening work done by the sub-office in PNG, for instance, is way beyond what was initially expected. There is little evidence of any negative unintended outcomes, albeit some confusion

5

persists as to whether the Support Unit is the Pacific Women program or whether the unit is simply in place to support the program.

In response to the evaluation question regarding the efficiency of the Support Unit, the evaluation team found that at inception the unit failed to have suitable quality assurance processes in place, which created inefficiencies in the delivery of its mandate. However, extensive efforts have been made to ensure greater efficiencies in the processes used by the Support Unit. For instance, the Programs team and the Operations team have found ways to maximise relationships that result in efficiencies and savings, such as leveraging relationships with the Pacific Leadership Program (PLP) and UN Women to co-facilitate financial management training in Tonga and Fiji. Moreover, the Support Unit has streamlined its own processes, to make them more efficient (as per the revised operations manual strengthening the quality assurance processes, and introducing an efficient process of collecting data through the new Knowledge Management System). Nevertheless, inefficiencies persist. Whilst there has been a gradual improvement in the accuracy and timeliness of financial reporting, DFAT reports that only one set of monthly invoices over two and a half years has been submitted without need for correction. In addition, a budget underspend in the Support Unit’s AWP for Financial Year (FY) 2016/17 of 10% meant DFAT had to reallocate at a very late stage in the FY, which entailed significant work for DFAT to do this. The underspend also signals poor financial forecasting and poor progress on regional AWP deliverables.

Whilst budget utilisation by the Support Unit has steadily increased, there does appear to be the possibility of some underspend against staffing costs at the end of the contract period (June 2018). To some extent, these savings point to certain efficiencies in the operations of the Support Unit, and have led to the unit creating a flexible pool of funding for staffing that can be used to accommodate ad hoc requests or meet additional resourcing needs. However, these underspends are also due in part to delayed AWP deliverables, delayed recruiting in certain key positions and staffing gaps that have only recently been filled.

Several restructuring processes have finally created a Support Unit that is better equipped to meet the demands being made of it. However, the restructuring process and slow recruitment processes did initially lead to several inefficiencies, especially with respect to the quality of outputs. Concerns were also strongly expressed in the interviews regarding “fit for purpose” recruitment at both Support Unit and adviser levels. Inefficiencies in recruitment and performance management have contributed to enormous levels of frustration, pronounced dissatisfaction and relatively high turnover in Support Unit staff, and substantial additional work for DFAT.

The evaluation team also found that improved and more effective management processes are in place, including regular interaction between the Support Unit and DFAT at multiple levels. Moreover, the Support Unit is an active participant in the governance structure of the Pacific Women program, which should help the unit provide high-level, strategic advice to the program. In addition, a key learning from the first two years of the Support Unit is that it has now reached a level of maturity to be given greater autonomy and thus it is no longer necessary to locate any DFAT staff within the unit.

With regards to the evaluation question pertaining to the Support Unit’s M&E, Capacity Development, Communications, Research and Learning strategies for the Pacific Women program, the evaluation team found that whilst the Support Unit has implemented a quality M&E system, the unit has yet to finalise several strategies specified in the Head Contract. In terms of continuous improvement, we have noted above the improvements introduced by the Support Unit to its processes. However, it is not clear to what extent these improvements were driven by lesson learning as the unit does not routinely report on lessons learnt.

In responding to the fourth evaluation question, the evaluation team conducted a Value for Money analysis of the Support Unit. Our analysis rates Ethics the highest, followed by Economy, Efficiency and then Effectiveness. Overall many of the criteria were rated as satisfactory. Nevertheless, there are a few areas which were not satisfactory, such as timeliness and innovation. Of concern is the inefficient use of DFAT

6

resources to assure the quality of outputs. Whilst there has been a recognition that quality is now improving, the time taken to secure the improvement means that certain key deliverables (such as several strategies) have yet to be finalised.

In answering the evaluation question as to whether the modality remains relevant, the evaluation found that whilst the modality has provided benefits to the programme and remains relevant in meeting the needs of partners, barriers and challenges could continue to hinder the efficacy and efficiency of the model. A review of similar models being implemented by DFAT in, and adjacent to, the region suggest a number of ideas to improve the model including ensuring an in-country presence when delivering across multi-countries, drawing on technical skills within the governing/advisory body to review and advise on implementation, appointing an independent facilitator/moderator to strengthen relationships between the Support Unit and DFAT, and ensuring the AWP includes specific innovative actions.

On the question of whether to extend the Managing Contractor’s contract, the Evaluation Team believe that it should be extended for at least 12 months. We suggest that DFAT and the Support Unit agree on a set of performance related targets which the unit will set out to achieve over the following 12 months. Failure to meet the agreed targets, without satisfactory mitigating factors, would give DFAT cause to reassess any further extensions of the contract.

Our overall assessment is that the unit displays a wide range of strengths and has achieved several notable achievements. Nevertheless, the failure to resolve quality assurance issues and deliver key strategies within a reasonable timeframe is unsatisfactory. Whilst the model was found to be largely effective, attention will need to be given to the barriers identified (especially the unrealistic reliance on GFPs) which we believe unduly influence effective operations, and to the underutilisation of several enablers for effective operations (such as the depth and breadth of existing relationships between the Support Unit and partners).

Based on our findings the evaluation team make the following recommendations:

Recommendation Our thinking Establish Sub-regional

offices, managed by a regional Program officer1 or at the very least, ensure that relevant SU staff visit each sub-region on a quarterly basis (having prepared an appropriate pre-departure plan that motivates and specifies activities to be completed during the visit). Indicative sub-regions for discussion between the SU and DFAT:

The breadth and spread of the countries and program activities (through the respective Country Plans) necessitate greater program coordination and adjustment.

Creating geographical clusters enables more focused yet economical support to countries, networking, reflections and learning (across countries and across thematic areas) within the same region.

Regular, programmed visits would create much needed presence and consistent, focused reach within each country in the respective sub-regions.

Visits would facilitate information flows and greater connectivity between regionally funded partners and bilaterally funded partners in countries within their region

Increase opportunities to facilitate joint forums for gender advisors; partners working in outcome areas of WEE and Women’s Leadership; and players developing new working areas in same region.

Will promote opportunities for diversifying the staffing of the Support Unit and building regional capacity.

Will allow cost-effective coordination meetings, emulating the success of

1 Existing, relevant Support Unit staff would be located within the proposed 4 sub-regions. Cost implications are relatively minimal as the 4 regional Program officers could be located with other organisations (for instance co-location is currently practised by the Pacific Leadership Program and rent is therefore kept to a minimal).

7

Recommendation Our thinking

Suva for Fiji, Solomon Islands, Vanuatu

PNG for Papua New Guinea

Samoa /Tonga for Cook Islands, Niue, Samoa, Tonga, Tuvalu

Pohnpei for FSM, Kiribati Nauru, Palau

the sub-office in PNG. Would work more consistently with GFPs to support implementing

partners, assess areas for capacity building and identify potential for innovation.

Regular visits could be used to foster greater involvement by Posts in the program

Would be a useful source for inputs to communication products

Appoint an independent Partnership broker that engages with both DFAT and the SU on a ¼ or biannual basis to help clarify and strengthen relationships and ensure the Facility remains on track.

Need for an independent facilitator/moderator between DFAT and Support Unit.

The partnership broker’s role would be to facilitate a ‘more joined up’ approach for the Program, and help strengthen relationships between the SU and DFAT.

If applicable, the broker could also facilitate relationship building between the SU and Gender Focal Points.

Establishment of an Innovation Fund, or at the very least identify innovative actions in the AWP

To date the Support Unit has had mixed results with promoting innovation.

A dedicated innovation fund would provide the incentive to the Support Unit to design and then implement innovative and experimental activities outside its core activities, but relevant to partner countries.

Alternatively, specify innovative activities to be undertaken by the SU on an annual basis in the AWP (with appropriate budget committed to these activities), to be jointly identified in discussion between DFAT and the SU.

Operational Adjustments

Adjustments to improve the effectiveness of the Support Unit:

All outstanding strategies listed in the Head Contract need to be completed within a realistic, but tight timeframe agreed between the Support Unit and DFAT.

Establish service level agreement on: categories of tasking notes and criteria to determine level of relevance

and priority2

standard time (notice period) for development of concept notes, communication products, work plans, reports

turnaround times for responses and comments to draft reports, TORs, strategies and communication products

The Support Unit should use reports and regular meetings between DFAT and the Support Unit to voice their reflective and strategic thinking, based

2 Tasking notes currently vary from AUD$5k (Support to Tonga WCC – 6 months) to AUD$1.4m (PNG Bounganville Gender Investment – 5 years)

8

Recommendation Our thinkingon their interactions with partners.

Other channels of communication should be promoted by the Support Unit, including radio.

An induction pack to be provided to each GFP, Support Unit staff and implementing partner. The pack to briefly outline the objectives of Pacific Women, roles and responsibilities of the Support Unit, GFPs and list key contacts.

Actively recruit other Pacific Islanders to work in their respective offices and/or establishing an internship programme that allows other Islanders to do a work placement in the two offices. Recruitment needs to be advertised where it is more accessible to Pacific islanders e.g. local newspapers of participating countries, email distributions in pacific networks and so on.

The swift implementation of the Capacity Building strategy and a deliberate, more specific and systematic plan for training and mentoring of Pacific islanders is essential.

As per the newly developed Capacity Strategy DFAT LES and implementing partners would benefit from spending time in the Support Unit to give them a better sense of the totality of the programme (and help promote active networking between the staff, develop a common understanding of work and so on).

A more effective recruitment and retention (performance management) system will help stabilise staffing as well as build capacity and ensure continuity.

It is an inefficient use of DFAT resources to locate any DFAT staff within the Support Unit, a weekly meeting should suffice.

A systematic process be developed to gather and then share lessons learnt, especially through reports prepared by the Support Unit

9

Background

Support Unit The Pacific Women Shaping Pacific Development Support Unit (Support Unit) has been established to support DFAT in the management and implementation of Pacific Women Shaping Pacific Development (Pacific Women) in 14 Pacific Island countries: Fiji, Cook Is, Kiribati, Marshall Islands, Federated States of Micronesia, Nauru, Niue, Palau, Papua New Guinea, Samoa, Solomon Islands, Tonga, Tuvalu and Vanuatu. The Support Unit is designed to provide responsive and flexible services to DFAT program management to ensure that Pacific Women is implemented in a manner that is consistent, coherent and meeting the objectives of the program.

Through a design and implement tendering process, Cardno Emerging Markets (Australia) Pty Ltd was contracted to design the operations and management systems of the Support Unit; and provide the technical, administrative and logistical services; outlined above. The current management of the Support Unit is contracted from 29 April 2014 to 30 June 2018 with a possible four - year extension. The value of the contract is AUD40,582,884 (Year 1 to 3) and AUD58,000,780 (Year 4 to 7) including activity costs to a maximum of AUD72,500,000.

The Support Unit has an office in Suva, Fiji and a sub-office in Port Moresby, Papua New Guinea that are expected to provide the following services at regional and country level:

10

(a) Policy and Technical Advice: actively engage with Pacific women and women’s organisations to maintain and strengthen Pacific Islander ownership of the program; provide high quality advice on gender equality approaches; contribute to policy analysis and strategic guidance of the program; and provide evidence-based input to communication products.

(b) Activity Management, Monitoring and Reporting: provide technical and administrative support to DFAT staff and manage contractual activities and stakeholder relationships to enable the implementation of country and regional plans. Support activity level monitoring, evaluation, learning and reporting.

(c) Meeting Facilitation and Logistics: organise and provide logistics and administrative support for Pacific Women Advisory Board and Program Management Committee meetings; support DFAT hosted meetings at country level; facilitate high level visits to program activities sites; and organise and provide logistics and administrative support for regional workshops and other relevant regional forums.

(d) Program Management, Monitoring, Reporting and Evaluation: implement the Pacific Women M&E Framework; oversee development of the Program’s M&E processes; and support program and activity level monitoring, evaluation and learning activities with implementing partners and stakeholders for country and regional plans. Provide work plans, progress reports, operational reports and annual reports; and facilitate reviews and evaluations for the program and Support Unit as required by DFAT. As well as maintaining a database of Pacific women consultants and Pacific Women funded activities.

(e) Communications: develop and implement a Communications Strategy; produce and manage all communication products for the program and those for use in DFAT publications and websites; and develop and maintain contacts/distribution list of stakeholder agencies and individuals for Pacific Women.

Annex 3 illustrates the extent to which the structure of the Support Unit has changed since mid – 2015 to how it is currently structured. The implications of this restructuring is discussed in the Findings section below. The Support Unit headed by a team leader (Support Unit, Suva, Fiji) who in addition to heading the SU, also has direct responsibility for the office in PNG, the Gender Advisory Unit, and the Evaluation and Learning Unit. The deputy team leader, who is also based in Suva, has direct responsibility for the Programs Unit, the Communications Unit, and the Operations Unit.

In addition to the office in Suva, Fiji there is a sub-office in PNG, under the leadership of the PNG Country Manager. The Country Manager is supported by a Program manager, M&E advisor (STA), an MEL officer, a Finance and Administration Officer, and an Administration Assistant. In addition, two new staff have been recently hired by the PNG Office (a Grants Officer and an Administration and Finance Assistant.

In order for the Support Unit to provide the services it has been contracted to undertake, the Managing Contractor has appointed a small unit within the Cardno head office to provide a range of contractual, procurement and human resources services to the Support Unit. Under the aegis of the Contractor Representative, Cardno has assigned a project manager, a project coordinator, a project support officer, a program accountant to provide the necessary services to the Support Unit.

Of the personnel listed in the current organogram 25/27 are women. A key characteristic of the Support Unit has been its evolution from an initial complement of 3 staff in the hub that was the precursor to the support unit, to the current complement comprising 13 staff in Suva (plus three ongoing short-term advisers) and five in the Papua New Guinea (PNG) sub-office (plus one ongoing long-term adviser). In addition, 3 more positions are currently being recruited, 1 in Fiji and 2 in PNG.

11

Of Pacific Women’s over 70 Partners and over 125 activities, only 20 are directly managed grants, with 14 of these in PNG and 6 managed by the SU Fiji Office. Of the other six grants, two are in Fiji, two in the Republic of Marshall Islands, one in Tonga, and one regional grant. Grant management typically includes day-to- day administrative support and technical support for activity implementation.

Evaluation PurposeThe main objective of the evaluation is to assess performance of the Support Unit, in particular its contributions to the broader objectives and outcomes of Pacific Women. The findings and recommendations will also inform DFAT’s considerations on whether to extend the term of the Support Unit contact for up to four years, in its current or a revised form.

More specifically, the purpose of the evaluation is two-fold, namely:

1 to assess the effectiveness of the Support Unit in delivering services against the DFAT Head Contract and Schedule of Requirements, in order to:

a. identify strengths and achievements of Support Unit implementation;

b. identify any barriers to the Support Unit’s effective operations and whether these are ongoing or have been resolved, and the enablers for effective operations and whether these have been fully utilised; and

c. recommend adjustments and enhancements to ensure that the Support Unit will be able to function most effectively to provide quality technical, administrative and logistical support to DFAT and implementing partners

d. support DFAT to determine whether to take the option to extend the term of the DFAT Head Contract for a period of up to 4 years.

2 to evaluate the Support Unit model, in order to:

a. assess if the current Support Unit model is the best way to support Pacific Women’s implementation;

b. identify lessons learnt from the establishment and operation of the Support Unit that DFAT could use to design and manage similar program models; and

c. provide recommendations to inform Support Unit implementation post June 2018.

ApproachA mixed methods evaluation approach was used in the evaluation (as described in detail in the Evaluation Plan, submitted 11 August 2017). The approach combined secondary data from documentary, monitoring and reporting sources (see Annex 2 for a list of documents consulted), with primary data obtained through surveys (distributed electronically), interviews, focus group discussions, and site visits to the Support Unit offices in Suva and Port Moresby. In addition, interviews were also conducted at the headquarters of the Managing Contractor in Melbourne, Australia.

84 people were either interviewed or participated in focus group discussions with the evaluation team (see Annex 1 for a fill list of those who participated). In addition, 23 directly managed grantees completed the Support Unit Partner Satisfaction Survey, 11 GFPs completed the GFP survey, and 17 members of the Support Unit completed the Support Unit Staff survey.

For the Value for Money (VfM) analysis the evaluation team developed a VfM rubric, based on DFAT's 4Es (Economy, Efficiency, Effectiveness and Ethics) and the VfM rubric proposed in the 3-Year Evaluation of Pacific Women (2017, pp. 60 – 62), albeit it has been amended to align to the ToR for this evaluation (see Annex 5 for the VfM matrix used by the evaluation team).

12

Key LimitationIn gathering evidence (and as outlined in the Evaluation Plan) the evaluation team shaped their instruments to align to the Support Unit Assessment rubric. In part this had to do with the requirement in the ToR to test the utility of the rubric, and in part it had to do with wanting to provide a benchmark against which future progress could be measured. All instruments used in the evaluation were therefore designed using the same scale as the rubric, namely:

Satisfactory Partially satisfactory UnsatisfactoryFully satisfies the criteria

Some aspects are satisfactory, but some areas do not satisfy the criteria

Does not satisfy criteria

However, by the time the fieldwork was already underway (and thus after, for instance, the surveys had been distributed electronically), the evaluation team were presented with the Support Unit’s revised rubric rating system. The revised rubric uses a 6-point rating system, which is aligned to the scale that DFAT routinely uses in its assessment of performance, in other words:

Satisfactory Less than satisfactory6 Very good; satisfies criteria in all

or almost all areas3 Less than adequate; on balance does

not satisfy criteria but does not fail in any major area

5 Good; satisfies criteria in most areas

2 Poor; does not satisfy criteria in major areas

4 Adequate; on balance satisfies criteria; does not fail in any major area

1 Very poor; does not satisfy criteria in many major areas

It was obviously not possible to readjust the scales used in the survey (one could not expect participants to re do the survey), and so the findings presented below only report against the original scale. Nevertheless, whilst the scales are different (with the revised scale being far more nuanced), it can be argued that they roughly equate to the following:

Original Scale Revised ScaleSatisfactory 4 - 6Partially satisfactory 3Unsatisfactory 1 - 2

Structure of the ReportIn the following section of the report the analysis and findings presented address the objectives of the evaluation, as set out in the ToR. As explained in the Evaluation Plan these objectives were rephrased as Key Evaluation Questions (EQs), with an accompanying series of sub-questions (outlined in detail in the Evaluation Matrix of the Evaluation Plan). What follows is therefore organised to respond to each EQ.

13

Analysis and Findings

This section reports on the analysis of data gathered from the different sources outlined in the Evaluation Matrix. Findings are presented against each of the EQs set out in the Evaluation Matrix, and a summary justification is provided in support of the ratings provided by the evaluation team. In some instances, to avoid duplication and to improve the flow of the document, the questions specified in the ToR have been re-ordered. Moreover, findings related to the modality (in answer to questions such as whether or not the original design of the Support Unit remains valid, and is the current Support Unit modality the most appropriate) are discussed as a distinct section under relevance.

EffectivenessEQ1: How effective has the Support Unit been in supporting Pacific Women achieve its purpose?

In assessing effectiveness, including examining what factors might have supported or hindered effectiveness, the evaluation team answered a series of sub – questions (as outlined in the Evaluation Matrix, and as specified in the ToR).

Sub Question 1.1: To what extent has the Support Unit provided high quality and timely operational services as outlined in the Head Contract and in Annual Work Plans?

Key Findings: The operational services provided by the Support have largely either met or exceeded the items listed in the Head Contract and Annual Work Plans, albeit the quality of the services provided have not always met expectations. Technical Support is a key strength being provided by the Support Unit, but strategic policy advice has yet to be seen by DFAT as similarly effective. Activity management, monitoring and reporting support provided by the Support Unit are well regarded by partners, as is the logistical support provided by the Support Unit. Whilst programme monitoring is an important strength of the Support Unit, the production of programme level reports has not been as effective. Quality assurance of reports has been a constant challenge for the unit, and it is only in recent months that DFAT have found programme level reporting to be satisfactory.

What follows is a review of each of the five main areas of the Head Contract (Pacific Women Support Unit Agreement/ Head Contract: Contract #70989, 29 April 2015)3.

Policy and Technical Advice Examples of policy advice can be found in a range of Support Unit reports. Two examples will suffice - see for instance the discussion regarding inclusive coalitions and advocacy groups4, and ideas for the Pacific Women Programme increasing debate on gender equality across the Pacific region5 (6 Monthly Progress Report March – August 2016, p.10).

With regards to technical advice, the Support Unit provided a number of examples which they believe testify to the extent of the technical advice they have provided. Examples shared with the evaluation team include

3 A Deed of Amendment to the Head Contract (#70989) was recently signed (1 August 2017), which addresses primarily the structural changes that have occurred since the Head Contract was signed. However, for the purposes of this evaluation the five main areas of the Head Contract remain.4 6 - Monthly Progress Report Sept 2016 – Feb 2017, p. 17. See also p.25 for additional discussion on the ‘tangible benefit in investing in the process of coalitions’.5 6 - Monthly Progress Report March – August 2016, p.10.

14

the Roadmap on Women’s Economic Empowerment, a regional review of discriminatory legislation and policies, a Family Sexual Violence policy for DFAT’s post in Kiribati, a regional review of counselling services in the Pacific, the Last Taboo research initiative, and ongoing work to develop a Disability Guidance Note to better integrate disability inclusive programming into Pacific Women activities. In addition, the Support Unit provides technical advice routinely through the reviewing of ToRs, designing of M&E plans, developing and implementation of country plans, the PNG sub-office preparing briefings and speeches and so on6.

Moreover, the Support Unit has, albeit not always in a timely and efficient manner, demonstrated its technical knowhow in the development of the KMS database, the research work and research symposium, the leadership dialogue, and the MELF for the program. Moreover, the Support Unit has provided considerable conceptualising around the Communications for the Program. This has involved a lot of consulting and thinking through how communications products (website, newsletter, Annual Progress Report) can better suit the needs of implementing partners.

Ongoing technical advice can also be seen through routine functions of the Support Unit, such as the advice provided by the M&E Unit to partners, countries and DFAT7; and the Gender Unit providing technical advice by commenting on copious concept notes, designs, TORs and so on, and Communications Officer offering technical advice with regards to how the program is viewed across the region.

Yet interviews with DFAT personnel suggest that the expectations regarding more and better quality policy and technical advice have yet to be realised. In addition, DFAT have argued that the technical advice provided by the Support Unit has largely been provided by consultants contracted by the Support Unit. DFAT believe that whilst these are examples of SU managing procurement processes and providing technical oversight, they should not be seen as the Support Unit providing technical support.

However, whilst these reports provide evidence of areas of strategic evidence these are not highlighted or expanded on in the reports. Nor does DFAT view the strategic thinking and advice provided by the Support Unit as sufficient, albeit there does seem to be differences within DFAT on this point. For instance, whilst the Support Unit in Suva until recently have not always contributed strategic advice, the sub-office in PNG is routinely invited to provide such advice.

Nevertheless, the Support Unit has started to put in place measures to ensure more systematic reporting against high level outcomes, which in turn should generate more reflective and strategic thinking. For instance, the Support Unit has started routinely mapping activity reports against outcome areas and conducting analysis on progress against outcomes and lessons learned. This process was further enhanced for the March 2017 Program Progress Report with the introduction of data analysis using the Knowledge Management System. Moreover, as noted by the Pacific Women Year 3 evaluation, the Support Unit Suva Office, now fully staffed, and should be in a better position to play a more strategic and proactive role within the programme, albeit ‘parameters of such a role are not yet fixed or clear’ (2017, p.3). In addition, the unit finally has in place a full-time gender specialist (a key role in the unit’s staff structure from the outset) and the fully operational gender panel should raise the quality of technical advice provided by the Support Unit.

Figure 1 illustrates the extent to which directly managed grantees rate the technical support provided by the Support Unit. Whilst four grantees did not express an opinion (as they had not sought this type of assistance from the Support Unit), the majority of those who did express a view (16/20) had found the support to be satisfactory. Directly managed grantees rated the M&E support the highest (68% found the nature of M&E support satisfactory).

6 For example, between March and August 2016, the Support Unit reviewed 17 technical documents including concept notes, design documents and strategic plans (March – August 2016 Support Unit Operations Report).7 See examples listed in Support Unit Operations Report March – August 2016, pp.6-7.

15

The usefulness and timeliness of technical assistance with regards to M&E that the Support Unit has provided to your organisation.

The ease in aligning your organisation’s M&E to the Pacific Women M&E requirements.

The quality and timeliness of gender technical assistance provided by the Support Unit to your organisation.

0% 20% 40% 60% 80% 100%

68.18%

63.64%

56.52%

22.73%

22.73%

26.09%

Unsatisfactory Partially Satisfactory Satisfactory No OpinionFigure 1: Rating of the Support Unit’s technical support by directly managed grantees (source: Partner Satisfaction Survey)

Activity Management, Monitoring and ReportingSince the outset, additional resources8 have been provided to strengthen the Support Unit’s approach to activity management, monitoring and reporting. As discussed below the Support Unit has undergone several restructuring processes to ensure an improvement in the management of activities, greater emphasis on monitoring, and a significant improvement in reporting. For instance, an Evaluation and Learning unit has been created to consolidate and coordinate all M&E activities within the Suva office. The unit has been effective in developing a Knowledge Management System (discussed further below) and supporting activity level M&E through the M&E panel. To date the panel has been drawn on to provide evaluations at activity and country level, producing satisfactory outputs to date.

Moreover, the Evaluation and Learning Unit have substantially revised the program’s Monitoring and Evaluation Framework (now known as the Monitoring, Evaluation and Learning Framework – MELF) in response to concerns that the initial framework was too complex and difficult to implement, that partners had difficulty in translating the MEF to country and activity level, and its indicators were not fully aligned to the objectives of Pacific Women.

The Pacific Women Year 3 evaluation (Final Report, 2017, p.54) found that ‘good groundwork has been achieved across the programme in generating technical capacity and in laying the foundations for generating more capacity, with initiatives taken by the Support Unit to enhance project M&E systems and provide technical assistance’.

Feedback received from interview and survey respondents testify to the high level of support that grantees have received with respect to the management of their activities, the monitoring of implementation, and the subsequent reporting. Figure 1 above notes that directly managed grantees were well satisfied with the M&E support received (roughly two thirds reported they were satisfied with meeting the M&E requirements of Pacific Women, and with the support they had received from the Support Unit).

Examples of the nature of M&E support includes the Support Unit working with individual or groups of implementing partners to assist them to develop project- level monitoring and evaluation frameworks such 8 The Contract amendment pending finalisation commits an extra approximately $2M to staff resources to the original contract The provision of these additional resources is largely due to the Managing Contractor providing an initial staffing budget that was insufficient for the needs of the Unit. DFAT have since had to provide additional staffing resources to ensure that in certain instances STA positions became LTA positions and/or that staffing positions were positioned correctly on the ARF to recruit suitably qualified staff. An example of the latter is in PNG where the Country Manager was initially budgeted for by the Managing Contractor as C2, but in fact will become a C4 in the 2nd Phase. Moreover, several other positions in PNG have had to be supplemented by PNG bilateral funds to ensure staff with the appropriate skills set were recruited.

16

as with Uniting World, Women in Fisheries, We Rise and WUTMI and the three national women’s centres (Fiji, Tonga and Vanuatu) funded by Pacific Women. Moreover, the respective posts in Tonga, Samoa and Vanuatu are developing monitoring and evaluation frameworks as a result of the Support Unit’s ongoing support. The nature of this specific support included in-country workshops and ongoing support to Gender Focal Points and implementing partners to review and revise frameworks ready for finalisation9.

The extent to which the reporting requirements (activity and financial) to the Support Unit are clear.

The extent to which the Support Unit provides useful and timely feedback on your reports.

0% 20% 40% 60% 80% 100%

86.96%

60.87%21.74%

Unsatisfactory Partially Satisfactory Satisfactory No OpinionFigure 2: Rating of Support Unit reporting support by directly managed grantees (source: Partner Satisfaction Survey)

In terms of activity reporting, Figure 2 illustrates the perception amongst directly managed grantees that reporting support provided by the Support Unit is effective (87% found that the reporting requirements are clear, and 61% found that the feedback on reports is both useful and timely). Feedback from interviews with grantees also found high levels of appreciation with regards to feedback, comments included

‘Unlike other donors, I get the feeling that PW reads our reports and wants to interact with us about them. Invariably they come back with insightful comments or useful questions’

‘The Support Unit provide ample time and support during reporting period, they also read the reports we submit and communicate back with comments or clarifications’.

GFP’s also expressed satisfaction with the support provided by the Support Unit with regards to reporting, as noted by one GFP: ‘The Support Unit is always ready and willing as I see it to assist and to improve on reporting from its in-country partners’.

Meeting Facilitation and LogisticsThe Support Unit provides a wide range of facilitation and logistic support, examples include support logistics for Pacific Women Advisory Board and Program Management Committee meetings, Gender Focal Point workshops, providing logistical support to the Pacific Women Parliamentary Program’s Annual Forum, facilitating all logistical arrangements for the visit of the Ambassador for Women and Girls to PNG, providing logistical support to Pacific Women Leadership Dialogue, and providing logistics for the Men’s Behaviour Change Conference in Sydney10.

The survey of directly managed grantees found that 70% of respondents reported that they were satisfied with the Support Unit’s provision of logistics to their respective organisations. Similarly, 70% of GFP’s were satisfied with the Support Unit’s provision of travel and other logistics, albeit this dropped to 60% of GFP’s

9 Information taken from the March – August 2016 Support Unit Operations Report.10 For more examples see the March – August 2016 Support Unit Operations Report, and the September 2016 – June 2017 Support Unit Operations Report.

17

reporting they were satisfied with respect to event coordination by the Support Unit. The rating contrasts sharply with Support Unit staff, of whom 94% rated event coordination, travel and logistical as satisfactory.

Program Management, Monitoring, Reporting and EvaluationAt the Programme level, the Support Unit has been responsible for establishing a Monitoring, Evaluation and Learning Framework (MELF) for Pacific Women, prepares 6-monthly progress reports on the programme, prepares the Annual Plan for the programme, and has also provided support to a range of reviews and evaluations including supporting the Year 3 evaluation of Pacific Women. Moreover, the Support Unit also provides significant programme level support in the recruitment and mobilisation of short and long-term advisors, coordinating in-country missions (including country plan reviews) and the management of 20 grants that fall under the Pacific Women programme.

Whilst the technical services provided by the Support Unit in this regard are viewed as technically sound (the new MELF for instance, meets DFAT’s M&E standards, and has been found to be very useful in guiding M&E at both country level and at activity level and the database includes retrospective data on 115 past and current Pacific Women activities), concerns have been raised with regards to the quality of deliverables at the programme level, especially at the outset.

Moreover, both performance assessments prepared by DFAT (29/04/2015 – 28/02/2016, and 1/03/2016 – 28/02/2017) found the quality of reports unsatisfactory, and both are highly critical of the quality assurance undertaken by the Support Unit. Even when the technical expertise of the Support Unit is noted, for instance with regards to the Knowledge Management system developed, the most recent performance assessment notes that ‘the KM database progress in recent months is positive, but delays could have been avoided if management recognised skills gaps earlier and responded accordingly’ (1/03/2016 – 28/02/2017).

CommunicationCommunication services provided by the Support Unit (as opposed to communication between the unit, DFAT and other key role players discussed below) have also been relatively extensive and is considered as one of the strengths of the Support Unit’s operations. Examples of communication outputs prepared by the Support Unit include the website, the distribution of e-newsletters, supporting communication related to annual Pacific Women Parliamentary Program (PWPP) Annual Forum, and compiling the Pacific Women’s Annual Progress Reports. Moreover, DFAT’s 1st performance assessment (for the period (29/04,2015 – 28/02/2016) noted that ‘the SU has demonstrated strong, consistent development of quality communication products and tools, and has effectively translated these to broad, quality print and social media coverage’.

Nevertheless, the Support Unit did initially struggle to prepare a Communication Strategy, which was delivered approximately 12 months after its due date, and only after a number of very poor drafts led to the strategy being completely re-written. Whilst the strategy emphasises the ongoing use of existing communication tools (such as the website, e-newsletters), the evaluation team believe the Support Unit should be promoting a range of other channels of communication for the program, especially radio, briefing/fact sheets and infographics.

Survey data with respect to the Support Unit’s effectiveness in communicating varied enormously between the different groups of respondents. The variable scores again illustrate the point that when the Support Unit have a direct and close relationship with a partner, the partner is typically satisfied. For instance, whilst only 20% of GFPs rate the Support Unit’s effectiveness in communicating with implementing partners as satisfactory, 77% of the actual implementing partners (directly managed grantees) rate the Support Unit’s effectiveness in communicating as satisfactory.

Recommendation: 1) All outstanding strategies listed in the Head Contract need to be completed within a realistic, but nevertheless constricted, timeframe agreed between the Support Unit and DFAT.

18

2) Establish service level agreement on: categories of tasking notes and criteria to determine level of relevance and priority11; standard time (notice period) for development of concept notes, communication products, work plans, reports, and turnaround times for responses and comments to draft reports, TORs, strategies and communication products.3) The Support Unit should use reports to voice their reflective and strategic thinking, based on their interactions with partners. 4) Other channels of communication should be promoted by the Support Unit, including radio.

Sub-question 1.2: To what extent has the Support Unit been able to be identify, support and test innovation?

Key Finding: When the Support Unit has been given an opportunity to implement innovative ideas, it has largely been successful. Nevertheless, there continues to be strong differences of opinion as to whether the Support Unit is sufficiently proactive in leveraging opportunities to be innovative, and/or whether the modality currently inhibits or promotes innovation.

Initial assessments by DFAT of the innovations of the Support Unit found the unit to be performing unsatisfactorily, with DFAT noting in the 1st Performance Assessment that the mobilisation of innovative approaches is yet to be realised’. By the time the 2nd Performance Assessment was conducted DFAT noted an improvement and recorded ‘the SU is progressively incorporating innovative practice into key tasks (e.g. MAMPU experiences in KM Database development, Pacific RISE engagement in PNG’, albeit these are both examples of the Support Unit progressing tasks through the contract and/or AWP). In addition, DFAT highlighted that they expect ‘an increase in SU engagement with identifying innovative concepts, strategic direction and contesting ideas to inform the annual work plan’. The Support Unit have countered that the existing relationship with DFAT, and their scope of work (primarily providing ‘back offices services’ to DFAT), has not created an environment within which innovation will flourish.

Nevertheless, where opportunities have arisen for innovation, the Support Unit has demonstrated its ability to support and test innovation. Two additional examples, to the ones already mentioned above, will suffice. The first is the innovative practice introduced by the PNG sub-office to host coordination meetings on regular basis (typically every 6 – 8 weeks) where a multitude of partners of the Pacific Women programme working in PNG get together to discuss their work, identify opportunities for joint work, discuss ways to avoid duplication of efforts, and deliberate on the means to leverage existing initiatives to ensure improved gender equity in PNG. A key driver in establishing the coordination forum was the realisation during the development of the PNG Country Plan that ‘uncoordinated individual activities funded under Pacific Women would fail to add up to a comprehensive approach to women’s empowerment’ (Operations Report, September 2016 – June 2017). Thus the forum helps promote greater awareness of the different activities being undertaken by partners and helps promote linkages between these activities. Moreover, as the Year 3 Evaluation Report notes (p.53-54):

PNG provides an example of where explicit efforts have been made to bring different programme actors together in learning, planning and advisory functions and where this has been widely appreciated by partners. There is ample testimony suggesting that bringing actors connected to the programme together does have a value-added effect, at the very least in generating a sense of collective goals and collective achievements, and beyond this, in generating learning, exchange, and the cross fertilisation of good practice in specialist areas. In addition, bringing stakeholders together can have operational benefits, making actors more aware of each other’s work, avoiding duplication and creating additional synergies and harmonised ways of working.

11 Tasking notes currently vary from AUD$5k (Support to Tonga WCC – 6 months) to AUD$1.4m (PNG Bougainville Gender Investment – 5 years)

19

The second example, is the initiative of the PNG sub-office in promoting SASA! training12 in collaboration with UN Women and Raising Voices. A series of training events have already been organised with representatives from PNG, Fiji, Kiribati, Solomon Islands and Vanuatu. Since the training to introduce SASA! in September 2016, the PNG sub-office has formalised arrangements with UN Women (Suva) and Raising Voices to provide technical support to SASA! in PNG. This includes negotiating arrangement with Raising Voices for approval to use SASA! adapted materials for all of PNG. To take this forward a SASA! Adaptation Task Team was established as a sub-committee of national coordination Advocacy and Coordination Technical Working Group under FSVAC leadership.

Recommendation: A joint team (DFAT and the Support Unit) must be established to determine how the Support Unit can best identify, support and test innovation. One idea might be to establish a small innovation fund which the Support Unit manages in order to facilitate innovation, or at the very least identify innovative actions during the drafting of the AWP.

Sub-question 1.3: To what extent has the Support Unit established mechanisms that will assist to build Pacific ownership, capacity and sustainability for Pacific Women?

Key Finding: The Support Unit is promoting Pacific ownership in a variety of different ways, but greater ownership could be promoted through expanding the diversity of locally engaged staff.

The Support Unit has had little opportunity to focus or invest deliberate efforts in promoting Pacific ownership at all levels within the unit despite the current Team Leader being a well-known and respected gender activist in the region. Whilst both Support Unit offices have actively recruited locally engaged staff, the staff are either from Fiji (in the case of the office in Suva) or from PNG (in the case of the sub – office in PNG), with no representation from any of the other countries that participate in Pacific Women.

However, over the past two years 72% of the vendors used by the Support Unit are local, a further 12% have been drawn from elsewhere in the Pacific, and only 17% of the vendors are international (primarily based in Australia). Furthermore, whilst 71% of STAs have been recruited internationally, the unit has managed to use 12 National STAs to date. In addition, both the Gender panel and the M&E panel include Pacific Islanders. Even though the numbers appear to be increasing, the value invested in Pacific capacity is still relatively inadequate.

The Support Unit is also well aware of the need to promote stronger ownership in the Pacific, as noted in an early Operations Report (September 2015 - February 2016):

The Support Unit needs to focus on being Pacific-run and -led in all of its activities. This can be achieved through transition steps that include: examining the roles of the Pacific Women Advisory Board and the various reference groups to ensure that they are actively contributing to and guiding strategic program direction and initiatives; prioritising Pacific Islander applications when establishing the M&E panel and gender advisory panel; and through stronger emphasis on including Pacific Islanders and building the capacity of Pacific Island expertise in program delivery and advisory support.

The notion of promoting Pacific ownership through the Support Unit was also highlighted in the Year 3 Evaluation Final Report (2017, p. 42):

This is related to the similar question of how to build local technical expertise, so that TA is not always provided by outsiders. Mechanisms to address this are being ‘tested’ in the design and

12 ‘SASA!’ which is an African word for ‘now’ seeks urgent action to change social practices that can prevent violence against women and girls. It uses a community owned and driven approach to increase awareness on gender-based violence and helps women and girls participate fully in development (http://www.pacificwomen.org/news/media-release-preventing-gender-based-violence-through-sasa-training/ )

20

construction of the Gender and M&E Specialist and Practitioner panels developed and managed by the Support Unit, along with the development of the Pacific Women Capacity Development Strategy. Methods include building in an element of capacity building in most consultancy exercises.

It is also expected that the much-delayed Capacity Development Strategy will help promote capacity and thus ultimately sustainability. In addition, capacity development of Fijians will also be a key feature in the new Fiji Women’s Fund, as noted in the Year 3 Evaluation: ‘the new mechanism due to be implemented in Fiji for the Fiji Women’s Fund – to be managed by the Support Unit – will provide targeted intensive organisational support so that potential grantees can work towards meeting the necessary criteria; similar mechanisms for other countries could be explored’.

More, however, could be done by the Support Unit to promote greater representativeness of the region in the respective offices. At present, the Support Unit is primarily staffed by locally engaged staff from either Fiji or PNG. With the program’s strong emphasis on diversity, and the need to understand the different contexts within which Pacific Women is being implemented it would be prudent to have a more diverse staff, that has a broader understanding of the region. Promoting diversity and embracing Pacific expertise should be a key aspect of the Support Unit’s approach, as explained by a respondent to the Support Unit Survey:

There needs to be a greater value placed on Pacific expertise in practice, rather than just in rhetoric. Working here can sometimes be pretty demoralising. DFAT recognises many of the challenges in implementation, and there is genuine concern that the program may not be doing its best to serve Pacific women, but in order to then take action, some changes need to take place:

greater delegated authority to SU senior management to be able to move and progress work and stakeholder relationships;

a more effective gender focal point system, with greater buy-in and connection with the senior decision-making staff at post;

Greater connection to key stakeholders, in particular women's civil society leaders, by bringing existing relationships and networks to the centre rather than periphery of the work, and SU staff will be key allies in this shift;

This means substantially improving the ways that women leaders can help guide the program (the Advisory Group on Research is a good example of this, and a good step forward), and an opportunity here would be through the follow up to the August 2017 High-Level Leadership Dialogue.

Recommendation:

1) Actively recruit other Pacific Islanders to work in their respective offices (at present the offices are staffed by nationals from either PNG or Fiji) and/or establishing an internship programme that allows other Islanders to do a work placement in the two offices. Recruitment needs to be advertised where it is more accessible to Pacific islanders e.g. local newspapers of participating countries, email distributions in pacific networks and so on.

2) The swift implementation of the Capacity Building strategy and a deliberate, more specific and systematic plan for training and mentoring of Pacific islanders is essential.

3) Rotate locally engaged staff between the two offices to give them a better sense of the totality of the programme (and help promote active networking between the staff, develop a common understanding of work and so on).

Sub question 1.4: Have Support Unit activities or operations resulted in any unintended positive or negative outcomes

21

Key Findings: The Support Unit has contributed to a number of positive outcomes, including the role the unit has played in convening and coordinating like-minded donors, regional organisations and programs across the Pacific. Whilst these convening and coordinating functions are what is expected of the unit, as per its contract and AWP, recognition should still be made of the fact that much of the coordination and convening work done by the sub-office in PNG, for instance, is way beyond what was initially expected. There is little evidence of any negative outcomes, albeit some confusion persists as to whether the Support Unit is the Pacific Women program or whether the unit is simply in place to support the program.

Several unintended positive consequences have already been noted above with regards to Support Unit operations. One of which is the increasing convening and coordination role being played by both the sub-office in PNG and the main office in Suva. As the most recent Operations Report (September 2016 – June 2017) notes:

The Gender Unit has played an increased role in convening and coordinating with other donors, regional organisations and programs. The Gender Unit has forged a strong working relationship with UN Women and is actively engaged in their work program including participation in two regional workshops on leadership and violence prevention in September 2016. The Gender Unit will participate in UN Women’s bi-monthly Gender Coordination meetings and has established informal coordination meetings with UN Women and Secretariat of the Pacific Community (SPC) on gender work in the region. In addition to supporting key partners, the Gender Unit has convened a number of events including the Roadmap consultation workshop (September 2016), the Counselling Review roundtable (April 2017) and has established a quarterly Women’s Economic Empowerment roundtable (February, May and August 2017)

Whilst these convening and coordinating functions are what is expected of the unit as per its contract and AWP, recognition should still be made of the fact that much of the coordination and convening work done by the sub-office in PNG, for instance, is way beyond what was initially expected. Another positive unintended consequence has been the role the PNG sub-office plays in providing support to the functioning of the High Commission in PNG. In the period September 2016 to June 2017, for instance, the office conducted 7 briefings to high level visitors to the High Com in PNG and helped draft at least 4 speeches for the Head of Mission.

One unintended consequence, which may be viewed by some as negative, is our finding that many of the partners we spoke to (primarily institutions not directly managed by the Support Unit, but whom do receive funding from DFAT and/or work together with Pacific Women at a regional level) do not distinguish between the Support Unit and the program has a whole. This finding supports the view expressed in in the Pacific Women Year 3 Evaluation Report (p.28), which found that ‘there was also some initial confusion over whether Pacific Women was the Support Unit, or something else’.

Sub-question 1.5: identify strengths and achievements of Support Unit implementation

Key Finding: The Support Unit has made several noteworthy achievements over the past two years, albeit certain key deliverables have taken significantly longer than expected and the quality of initial outputs was variable.

In this section of the report we highlight the achievements of the Support Unit over the past two years which illustrate the extent to which the Support Unit is becoming more effective, albeit as already previously noted there were significant delays in the delivery of many of these activities. Moreover, whilst these activities were contracted deliverables, we believe that they should be recognised as achievements bearing in mind the challenges posed in providing support across a region where the country context is often complex and varies enormously between countries. A number of these key achievements have already been referred to earlier, and include (but by no means an exhaustive list):

22

Successful delivery against the Annual Workplans13

Successful delivery of additional/ ad hoc activities against Tasking Notes Finally recruiting a full complement of staff, in addition to contracting and establishing the Gender

Advisory Panel and the M&E Panel Provision of support to the Fiji Women’s Fund including recruitment of the first two positions (Fund

Manager and Finance and Administration Officer) Development of the Knowledge Management System The Monitoring and Evaluation Framework for Pacific Women Country Plan Review Process PNG Annual Learning Workshops A substantial list of Operational related activities (such as ToR development, recruiting applicable TA,

scoping studies and reviews, conceptualising and critiquing the design of proposals and so on) A substantial list of logistic support (including Advisory Board meetings, annual GFP meetings and

other key program level meetings)

In addition, the Support Unit has been effective in delivering the key reports specified in the following table from the Head Contract, albeit the finalisation of these deliverables was not always within the due date:

Description of Report Format Qty Due Date Date Finalised

(a) Mobilisation Management Plan

Electronic copy in Microsoft Word and PDF

127 May 2015

Approved by DFAT 13 July 2015

(b) A Pacific Women Support Unit Final Design and Operations manual

Electronic copy in Microsoft Word and PDF

1

26 June 2015

Design of Support Unit approved by DFAT 31 July 2015. Operations Manual approved by DFAT 1 August 2015

(c) Annual Plan Electronic copy in Microsoft Word and PDF

1

26 June 2015 and by 1 December each

subsequent year.

First Annual Plan (2015) approved by DFAT 31 July 2015.Subsequently rescheduled, most recent submitted 1 June 2017

(d) A Final Risk Management Plan

Electronic copy in Microsoft Word and PDF

1

26 June 2015

Approved by DFAT 1 August 2015.Revised on a six-monthly basis, most recent submitted February 2017

(e) An updated regional communications strategy for Pacific Women

Electronic copy in Microsoft Word and PDF

1

28 July 2015

Revised due date 30 November 201714

(f) Bi-monthly financial Electronic copy in 1 within 2 weeks following: Met target, albeit DFAT

13 The Support Unit Operations Report (September 2016 – June 2017) notes for instance that ‘the Support Unit has progressed or completed 116 activities under the Support Unit Annual Work Plan (January 2016 – June 2017) and has progressed or completed an additional 38 additional/ad hoc activities from September 2016 – June 2017 (p.1)’.14 As specified in Amendment 1, Contract #70989, p.31. The strategy has however been through various earlier iterations, with discussions on its content stretching back to feedback being provided on the original submission in August 2015, and a subsequent revised version being submitted in April 2016, which DFAT approved in May 2016.

23

Description of Report Format Qty Due Date Date Finalised

reports Microsoft Word and PDF the end of consecutive

two month calendar periods after the Project

Start Date

reports that only one set of monthly invoices has been submitted without need for correction and that bi-monthly financial reports are submitted late.

(g) M&E Framework document

Electronic copy in Microsoft Word and PDF

1 Draft by 31 July and final by 15 September 2015.15

7 July 2017

(h) Capacity Building Strategy

Electronic copy in Microsoft Word and PDF

1 Initial due date was November 2016, new due

date 1 April 201816

Ongoing

(i) Six Monthly Pacific Women Support Unit Operations reports

Electronic copy in Microsoft Word and PDF

1

15 March and 15 September each year

Submitted as per schedule (although quality assurance issues initially meant DFAT only approved first report in December 2015) Support Unit now reports on an annual basis as agreed with DFAT

(j) Six Monthly Progress Reports on Pacific Women Program*

Electronic copy in Microsoft Word and PDF

1

15 March and 15 September each year

Submitted as per schedule, albeit DFAT raised quality assurance issues with draft versions which led to delays of the progress reports being signed off (e.g. initial report was approved by DFAT in January 2016)

(k) Annual Progress Reports

Electronic copy in Microsoft Word and PDF

1By 30 October each year

Submitted on 31 October, albeit DFAT had quality assurance issues with initial submission

(l) Annual updates to the Annual M&E Framework

Electronic copy in Microsoft Word and PDF

11 December annually

from 2016

Ongoing (DFAT agreed to revise submission date to 30 June of each year)

(m) Handover Plan Electronic copy in Microsoft Word and PDF

1

29 April 201617

Data has been changed in the amendment to the Head Contract as the original due date was not met.

Table 1: Reports to be provided by the Support Unit, as per Head Contract

However, whilst the Support Unit has been effective in producing a number key achievements and delivering the reports listed above there are other key deliverables specified in the Head Contract that have yet to be delivered (or have only very recently been completed) including:

15 Due date amended to 31 October 2017 (Amendment 1, Contract #70989, p.31)16 As specified in Amendment 1, Contract #70989, p.3117 Due date has been amended to 9 months prior to Contract end date (As specified in Amendment 1, Contract #70989).

24

the Capacity Development Strategy (due Nov 2016, and as of end June 2017 has not yet been finalised),

the Research Strategy (albeit a lengthy consultation process was necessary which whilst slowing down the delivery of the strategy has likely contributed to greater ownership across the Pacific),

The new website.

EfficiencyEQ 2: To what extent has the Support Unit provided high quality and timely management services as outlined in the Head Contract?

In assessing efficiency, including examining what factors might have supported or hindered efficiency, the evaluation team answered the sub – questions as outlined in the Evaluation Matrix, and as specified in the ToR).

Sub-question 2.1: The extent to which quality processes have been observed (i.e. clear objectives, focused on priority sectors; articulation of expected results; basic monitoring, reporting and acquittal processes established)

Key Finding: At inception the Support Unit failed to have suitable quality processes in place, which created inefficiencies in the delivery of its mandate. However, extensive efforts have been made to ensure greater efficiencies in the processes used by the Support Unit. These efforts, albeit primarily to do with compliance with the Head Contract, include the revised operations manual, improving the accuracy and timeliness of financial reporting, strengthening the quality assurance processes, and introducing an efficient process of collecting data through the new Knowledge Management System.

We noted above that that after a very unsatisfactory start with regards to basic monitoring reporting and acquittal processes, the Support Unit has made a noticeable improvement in its processes. In the 1st Performance Assessment DFAT noted that not only were key reports late (‘time extensions have been requested for all reports’), but of a poor quality and ‘all financial reports have required correction and quality assurance: including monthly invoices, bi-monthly financial reports….and the Finance section in the first progress report’. Survey data also highlights the perception that timeliness of reporting by the Support Unit has been poor. For instance, staff of the Support Unit and GFPs rated timeliness of reporting the same (only 50% rated timeliness of reporting satisfactory). Moreover, certain deliverables have taken an inordinate time to complete, which meant their relevance has been questioned once delivered. For example, the Roadmap on Women’s Economic Empowerment, Women and Leadership and End Violence Against Women took nearly a year to complete. Whilst it did employ a highly consultative process, it did also involve four different contractors before the work was finally finished. By taking so long to complete the Roadmap is seen by many as surplus to requirements. The Program has moved on and thus the original intent of the Roadmap to help shape planning has diminished.

Whilst timeliness has largely been resolved, the Support Unit continues to introduce efforts to improve the quality of the reporting (including the hiring of substantial editorial support). In addition, what also contributed to poor quality reports is the fact that the Support Unit were reliant on information of variable quality from partners. Providing remote support probably did not help this process. With few opportunities to visit delivery sites it does make the work of the Support Unit difficult, as they are not able to validate or verify the information the report on, albeit greater reliance could be made of phone and Skype calls, and using appropriate DFAT staff for verification purposes. However, with the KMS database in place, improved reporting templates, and partners fully conversant in the system this aspect of data collection and reporting should improve. Moreover, and as discussed below, the Support Unit should develop a nuanced approach to in-country monitoring by prioritising specific countries (or sectors or implementing partners) to visit and then making the case as to why the visit is necessary, what support needs to be provided, and by whom.

25

Financial reporting processes have also improved, as noted in the 2nd Performance Assessment. This improvement has come about because of several concerted interventions, including the Managing Contractor employing a finance specialist to overhaul the financial management systems being used by the Support Unit. In addition, improvements were made to the Support Unit’s approach to procurement. All of which has led to more streamlined processes within the Support Unit and contributing to greater efficiencies, and the requisite financial systems and controls are in place.

Further discussion on the efficiencies being achieved by the Support Unit can be found in the Value for Money discussion below.

Sub-question 2.2: The extent to which inputs (resources and time) have been used in the best possible way to achieve the objectives of the Support Unit

Key Finding: Whilst budget utilisation has steadily increased, there does appear to be the possibility of some underspend at the end of the contract period (June 2018). These underspends, the result of delays in activities being implemented and delays in recruitment, has had a negative impact on programming and created additional work for DFAT. Nevertheless, there are examples of efficiencies in the operations of the Support Unit.

In answering this question the evaluation team explored how resources were allocated and utilised by the Support Unit in the period under review. The Head Contract was signed on 29 April 2015 and the Support Unit commenced as of 1 May 2015, with the $40,582,884 budget allocated as shown in Table 2.

In Table 2 we can see that 72% of the budget was allocated to activity costs, 7% was allocated to the management fee of the Managing Contractor, and the remaining 21% was earmarked for costs associated with the Support Unit (staffing and operating costs).

Budget Line Contract Limits

Management Fee 3,008,380.00

Specified Personnel Fees

Long-term adviser costs 3,205,088.00

Short-term adviser costs 738,317.00

Locally engaged personnel 1,543,422.00

Subtotal - Specified Personnel Fees 5,486,827.00

Adviser Support Costs 1,563,787.00

Operational Costs 1,153,890.00

Activity Costs

Regional 7,500,000.00

Fiji 400,000.00

Papua New Guinea 10,000,000.00

Solomon Islands 5,600,000.00

Vanuatu 400,000.00

Kiribati 1,850,000.00

Tonga 1,400,000.00

Samoa 1,080,000.00

Marshall Islands 300,000.00

Palau 400,000.00

Federal States of Micronesia 90,000.00

26

Nauru 100,000.00

Tuvalu 250,000.00

Subtotal - Activity Costs 29,370,000.00

TOTAL 40,582,884.00

Table 2: Support Unit budget, May 2015 – June 2018

A review of expenditure over the first 26/36 months of the programme (i.e. May 2015 to June 2017) reveals that the budget utilisation has been roughly on track, albeit slightly lower than the 60% threshold one would have expected bearing in mind the Support Unit has less than a year left operating under the current contract.

As one would expect, bearing in mind the unsatisfactory recruitment strategy (including pegging positions to the wrong ARF level, under budgeting staffing costs, high turnover of staff during the first few months, and long delays in filling vacant positions) employed during the inception phase of the Support Unit, Figure 3 illustrates relatively slow budget utilisation through until February 2016 (cumulative expenditure at 12% after 8 months), but a steady increase since then, reaching 54% by June 2017). As discussed elsewhere in this report, poor utilisation was undoubtedly also caused by delays associated with activities taking longer than planned to commence.

May/June 2015

July/ Aug 2015

Sept/Oct

2015

Nov/Dec 2

015

Jan/Feb2016

March/A

pril 2016

May/June 2016

July/Aug 2016

Sept/Oct

2016

Nov/Dec 2

016

Jan/Feb 2017

March/A

pril 2017

May/June 2017

0102030405060708090

100

Cum

ulati

ve e

xpen

ditu

re to

dat

e (%

)

Figure 3: Support Unit budget utilisation, cumulative percentage (Source: Support Unit Bi-Monthly Financial Reports)

Annex 4 compares the expenditure of the Support Unit against the Annual Work Plan (January 2016 – June 2017)18. The table highlights the difference between budgeted amounts (Column C) and expenditure (Column F). For activities across the region, the underspend equates roughly to 10%, but there is variance per country and thus there are examples of both underspend (such as in Fiji – the largest underspend in the region - and Tonga) and a significant overspend in PNG. The substantial overspend in PNG (budget of $7.28 million was increased to $9.23 million as a result of additional activities being added to the AWP) kept the overall underspend to 10% (i.e. without the additional activities in PNG for this reporting period the underspend may have been as high as 20%). Whilst it can be argued that underspends may have been outside the ambit of the Support Unit (such as delays created by partner governments), the underspends are likely also to have been influenced by the delays in staff recruitment, the poor quality of key deliverables, and untimely procurement.

From DFAT’s perspective, whilst some underspends are expected (development work is complex and coordinating disparate partners is complicated), the lack of an effective early warning system regarding these delays can and does impact on programming. Any underspends either require considerable

18All information in Annex 4 was provided by DFAT.

27

reprogramming by DFAT (including readjusting tranche payments to different partners), or a reallocation within Posts to an alternative activity for that applicable Financial Year.

In Figure 4, 41% of the budget remains for long-term advisors, 44% for short-term advisor costs, and 54% for locally engaged personnel. Supports costs for advisers and personnel have used 48% of the budgeted amount, and thus 52% remains. The unallocated provision for Short Term advisors and locally engaged personnel have helped create the surplus referred to above. These savings will be reallocated to other areas, where and when additional resources are required, albeit as already noted the amendment to the Head Contract, pending finalisation, commits an extra approximately $2million to staff resources over and above the original contract.

Long-term advisor costs Short-term advisor costs Locally engaged personnel Adviser/ Personnel Support Costs

$0.00

$500,000.00

$1,000,000.00

$1,500,000.00

$2,000,000.00

$2,500,000.00

$3,000,000.00

$3,500,000.00

Cumulative expenditure to date Remaining

Figure 4: Expenditure on advisors and personnel, and their support costs (Source: Support Unit Bi-Monthly Financial Reports)

It was noted above that where possible the Support Unit has relied on local suppliers to reduce costs of the Support Unit. Other efficiencies can be found in, for instance, the sub-office in PNG sharing offices with another facility that was originally managed by the same Managing Contractor, and sharing the same IT provider to minimize costs. Moreover, the Programs team has found several ways to maximise relationships that result in efficiencies and savings. For example, the Unit has leveraged relationships with PLP and UN Women to co-facilitate financial management training in Tonga and Fiji. The Unit has also coordinated efforts with SPC and UN Women to ensure that STA Advisers are not duplicating work. Moreover, the Support Unit has provided advice to DFAT where work has already been carried out in areas of interest for the program. In addition, where feasible, the Support Unit has coordinated with other development partners and DFAT to support logistics for regional meetings and workshops.

Sub – question 2.3: Has the Support Unit been appropriately structured and resourced to meet the needs of DFAT and Pacific Women’s implementing partners?

Key Findings: Several restructuring processes have finally created a Support Unit that is better equipped to meet the demands being made of it. However, the restructuring process and slow recruitment processes did initially lead to several inefficiencies, especially with respect to the quality of outputs. Concerns were also strongly expressed in the interviews regarding “fit for purpose” recruitment at both Support Unit and adviser levels. Inefficiencies in recruitment and performance management have contributed to enormous levels of frustration, pronounced dissatisfaction and relatively high turnover in Support Unit staff, and substantial additional work for DFAT.

Testimony from the majority of respondents highlight the common view that the Support Unit has only recently been operating with a full complement of staff (in fact two new staff were inducted into the PNG sub-office at the conclusion of the fieldwork for this evaluation). By all accounts the Support Unit staffing resources remained sub-optimal throughout the first year of the Support Unit. The 1st Performance

28

Assessment noted that ‘both Suva and PNG permanent office establishment were delayed, with the PNG sub-office yet to be fully operational’. Moreover, under resourcing was exacerbated by several of the recruited staff being unfamiliar with DFAT’s modus operandi and the systems operated by the Managing Contractor.

In addition, under-estimations regarding staffing of the unit were made in the original tender leading to the Managing Contract having to recruit for four additional positions (as per the original restructuring proposal, January 2016). Further restructuring was conducted during 2017, which led to additional delays between what was proposed and DFAT ultimately endorsing the proposals.

From the perspective of the Support Unit, an acknowledgement that the unit was initially poorly resourced in terms of staff can be seen in the unit in conjunction with the Managing Contractor re-examining the units needs against its existing expertise, skills and resourcing challenges. Following this assessment, the Support Unit Restructure Proposal: Recruitment and Development Plan 2016–2018 was developed and submitted to DFAT on 22 January 201619. A key feature of the plan was aligning the appropriate level of skills required to the roles in the unit, with a view to recruiting personnel with appropriate expertise. However, as Table 3 illustrates there was need for further restructuring later that same year and again in 2017.

Table 3 depicts the shifting positions, roles and responsibilities of many of the LTA positions specified in the Head Contract (a signifies the position had been filled, where there is no the position was vacant). The table also highlights that only two of the original incumbents have been with the Support Unit since inception. A key feature of the table is it illustrates the extent to which the Managing Contractor and staff of the unit struggled initially to find the right structure, and appropriate alignment of roles and responsibilities to the different posts. Whilst three personnel (the outgoing Team Leader, the current M&E manager and the Communications specialist) have been with the program since the outset, all other personnel have been recruited since the inception of the Support Unit.

Moreover, as the table below also illustrates, position description changes have also fluctuated over the past two years as the unit sought to establish the most effective structure to ensure it was both responsive and flexible. It is also important to note, as indicated in the new Deed of Amendment to the Head Contract (2017, pp. 50 – 53), that several of the positions have either been introduced since the Head Contract or have been reclassified in terms of the ARF level, which has had significant cost implications for DFAT (as noted above) .

Designation Specified in Head Contract

SU Design(June 2015)

SU Restructure Proposal(March 2016)

SU Restructure Proposal(August 2017)

Part of team specified in Head Contract & still at SU (August 2017)

Team Leader Yes

Deputy Team Leader

No (formerly

Operations Manager)

(new DTL as of April 2016)

M&E Specialist Yes (now Evaluation and Learning Unit)

M&E specialist post became STA and a new

M&E Manager as of October 2016 (TBC)

(former Programs Manager now head of Evaluation and Learning Unit)

Operations Yes (see above, subsumed into DTL position)

19 Support Unit Operations Report: September20 15 – February 2016, p.2.

29

Designation Specified in Head Contract

SU Design(June 2015)

SU Restructure Proposal(March 2016)

SU Restructure Proposal(August 2017)

Part of team specified in Head Contract & still at SU (August 2017)

ManagerGender Specialist

Yes (now Gender Advisory Unit)

(temp)

Gender Specialist to transition to new Senior Program officer, with the recruitment of new Gender Specialist

Communications Specialist

Yes Engaged as needed(Communications Officer on board)

Now known as Communications

Officer

(now known as Communications Coordinator)

Research Specialist

Yes (Knowledge Management and Research Specialist post created, STA)

Programs Manager

Yes

PNG Country Manager

Yes (From March

2016)

Knowledge Management Specialist

No (P/T)

(as above)

Table 3: Shifting LTA positions within the Support Unit

This reshuffling, whilst necessary to achieve a more robust unit, and the delays in filling positions, have contributed to high levels of frustration within DFAT. Concern has been regularly expressed by DFAT that the restructuring process has led to certain inefficiencies. Not only in terms of the Support Unit being under staffed to cope with the work load, but that inadequate staffing resources were contributing to the poor performance and poor quality assurance discussed above. As the 2nd Performance Assessment notes:

‘delays continued to place pressure on staff: with the SU’s ability to provide consistent, timely gender expertise compromised due to staff turnover/delays and the management team (TL and DTL) needing to backfill other areas of work which impacted on delivery of the broader work program as well as capacity to take a more proactive strategic direction-setting role in programme delivery’.

30

Staff turnover within the Support Unit and the Managing Contractor has also contributed to high levels of frustration with DFAT. In the period under review DFAT have had to work with 3 different contractor representatives (albeit one of which was an interim arrangement), and within the unit the Deputy Team leader has been replaced, as have several other posts (including the Gender Specialist Communications Officer, and at least one Finance and Administration Assistant). In addition, the manager of the sub-office in PNG only started in March 2016, after a lengthy recruitment process. Staff turnover was further exacerbated by delays in the recruitment process initially followed during the inception phase, and delays in the establishment of both the Gender Panel and the M&E Panel.

However, DFAT did note in the same Performance Assessment that the unit (which DFAT viewed as fully staffed as of February 2017) has realised a full team, which ‘has delivered a strong mix of skills and experiences – international, regional and national, program management and development partnerships, as well as enabling DFAT to include SU staff in broader program/ partner discussions’.

Moreover, within the Managing Contractor there has been a gradual realisation since the start of the Support Unit that the support services provided by the Managing Contractor should increase substantially to better respond to the Support Unit and DFAT’s requirements. Whilst it is difficult to quantify the increased support provided by Cardno (in many instances the support has required phone calls, in-person conversations and other attempts to build relationships with DFAT), there is a sense since the start of 2017 that Cardno has taken more responsibility for addressing poor performance, improving the quality assurance of deliverables, and facilitating several sessions to help improve effective and efficient operations.

Moreover, there appears to be general consensus amongst many of those interviewed that with clearer lines of responsibility and more efficient recruitment processes20 the Support Unit is likely to be more efficient. In addition, there is a sense that those working in the Support Unit now have a better understanding of what DFAT expects, as one respondent to the Support Unit survey noted:

The Support Unit has only recently had the resources and structure to respond to DFAT's needs and expectations. Many people in the Support Unit have no prior experience working with DFAT and this has presented challenges in meeting their needs.

Recommendation: A more effective recruitment and retention (performance management) system will help stabilise staffing as well as build capacity and ensure continuity.

Sub – question 2.4: To what extent has there been effective management arrangements between DFAT and the Support Unit, including through the annual work planning process and tasking notes, management and operational level meetings, governance structures and so on?

Key Findings: Improved and more effective management processes are in place, including regular interaction between the Support Unit and DFAT at multiple levels. Moreover, the Support Unit is an active participant in the governance structure of the Pacific Women program, which should help the unit provide high-level, strategic advice to the program. In addition, a key learning from the 1st phase of the Support Unit is that it has now reached a level of maturity to be given greater autonomy and thus it is no longer necessary to locate any DFAT staff within the unit.

Whilst the Support Unit has generally been considered responsive to DFAT’s request, the performance assessment conducted by DFAT note that initially regular communication between the Support Unit and DFAT management team ‘appear to have only been partially realised’. However by the time of the 2nd performance assessment DFAT noted that improved communications can be attributed to a number of mutually agreed actions of this PPA period including DFAT, Cardno and Support Unit management teams have institutionalised monthly management meetings; ‘One-team approach’ paper and discussions led by Cardno/Support Unit; increased Support Unit/Cardno representation on the Programme Management

20 A process that is now guided by the revamped Cardno Emerging Markets’ Procurement Toolkit: Version 2, May 2017. See also the recruitment and procurement processes outlined in the revised Support Unit’s Operations Manual.

31

Committee to include the Contractor Rep and PNG Country Manager; the Support Unit TL now observer member of the Secretariat to the Advisory Board; at an operation level, DFAT and Support Unit Suva have instituted weekly workplan monitoring meets, and DFAT Suva staff participate in Support Unit six-monthly Review and Reflect meetings’. The assessment by DFAT suggest that significant steps have been taken to improve coordination between the unit and DFAT.

In addition, other elements of making management arrangements more effective and efficient include the Support Unit hosting regular interactions via email and teleconferencing between the Support Unit Senior Programs Manager, based in Suva, and Gender Focal Points at DFAT posts and refining the Annual Work Plan process as noted in the following:

The Support Unit refined the process for developing the Annual Work Plan for 2017–2018 including creating opportunities for engagement from senior-level DFAT staff at Posts through teleconference and working with Posts to develop more accurate budgets. Sixty-one per cent of respondents to the 2017 GFP survey agreed or strongly agreed that the process for developing the 2017–2018 Annual Work Plan was useful for planning their activities. The Support Unit developed the work plan using a web-based project management tool that will more efficiently track progress through live updates and produce reports directly aligned with the plan. The Support Unit provides a monthly report on the Annual Work Plan to the Pacific Women monthly management meeting and will establish monthly teleconferences from August 2017 with each Post to discuss implementation progress including expenditure and discuss any changes or new activities (Support Unit Operations Report, September 2016 – June 2017, p.4).

The 3 Year evaluation also agreed with the sentiment that management arrangements have improved, when it reported that the ‘Support unit has recently begun to be seen as an asset to the program’ and noted that there was now ‘strong communication’ between the unit and DFAT, including (as already noted), ‘weekly and monthly management meetings and initiated a formal self-assessment and joint dialogue on its performance following the Program Management Committee meeting in February 2017’.

A feature of management arrangements between DFAT and the Support Unit has seen the DFAT Pacific Women Program Manager working up to two days per week from the Support Unit Suva office and the DFAT PNG Program Manager working up to one day per week from the PNG Support Unit sub-office. In the case of the latter, this was relatively short-lived. Whilst continuous dialogue still happens between the sub-office in PNG and DFAT, the Program Manager no longer works from the Support Unit office.

However, the DFAT Pacific Women Program Manager does continue to typically spend two days a week in the Suva office. As we note in our discussion below with regards to VfM this is an inefficient use of DFAT resources. A weekly meeting with the Pacific Women Program Manager should suffice, leaving her to play a more strategic role in guiding the direction of the program, in addition to attending to her duties at the High Commission.

Recommendation: It is an inefficient use of DFAT resources to locate any DFAT staff within the Support Unit, a weekly meeting should suffice.

32

Monitoring and EvaluationEQ 4: To what extent has the Support Unit developed and implemented quality M&E, Capacity Development, Communications, Research and Learning strategies for Pacific Women and are they using the information generated for continuous program improvement

Key Findings: Whilst the Support Unit has implemented a quality M&E system, the unit has yet to finalise strategies specified in the Head Contract. In terms of continuous improvement we have noted above the improvements introduced by the Support Unit to its processes. However, it is not clear to what extent these improvements were driven by lesson learning as the unit does not routinely report on lessons learnt.

This EQ has largely been answered above. It has already been noted that substantial work has been done on developing and finalising a complex Monitoring, Evaluation and Learning system across the program (in addition to the revamping the Pacific Women MEF, and revising and restructuring the Evaluation and Learning Unit within the Support Unit). In addition, the unit has put a Knowledge Management System in place. It has also been noted that whilst several strategies, such as Capacity Development, Communications, Research are in various stages of drafting they have not been finalised. Whilst not wanting to belabour the point, these strategies should all have been developed and finalised during the inception period and are to an extent creating an unnecessary distraction in the unit’s relationship with DFAT.

In terms of continuous program improvement, the Year 3 evaluation report has already reported on the extent to which Pacific Women is improving and what needs to be done to improve it. In terms of the extent to which the Support Unit is improving, the two previous sections highlight the steps the unit are taking to ensure greater efficacy and efficiency.

Whilst the programme does not routinely report on lesson learning, a few examples of identifying lessons learnt can be seen in the Operations Reports. In addition, respondents shared several examples of the lesson learning happening as result of interactions within the Support Unit or between the Support Unit and its partners (Table 4):

Respondent Lesson learnt

DFAT Environment not enabling, instead we have created a fearful and restrictive space for the Support Unit, which has led to a loss of confidence when micro-managed. We need to step back.

GFPs Greater support needed through teleconferencing

Maintain regional funding so Support Unit does not encroach on bilateral funds

Budget allocation to be clearer at Posts

Recruitment of personnel in the Support Unit should not focus only on gender, but also on program management skills and experience

Support Unit

GFP challenges differ across countries: amongst GFPs there is enormous difference in level of understanding of Support Unit, in the level of services and support GFPs require, in their familiarity with KMS, and in their level of interaction with the Support Unit.

Trust and credibility easy to lose but hard to gain, especially when it comes to the quality of outputs

Ex DFAT staff add considerably to the effectiveness of the Support Unit, ‘as they understand DFAT’

Scheduled horizontal Team connections between Suva and PNG have improved because of regular scheduled interactions

Table 4: Examples of lesson learning (source: Interviews conducted by the evaluation team)

33

Recommendation: A systematic process be developed to gather and then share lessons learnt, especially through reports prepared by the Support Unit.

Value for Money EQ 5: Does the Support Unit Model demonstrate good value for money?

Key Findings: The VfM analysis rates Ethics the highest, followed by Economy, Efficiency and then Effectiveness. Overall many of the criteria were rated as satisfactory. Nevertheless, there are areas which were not satisfactory, such as timeliness and innovation. Of concern is the inefficient use of DFAT resources to assure the quality of outputs. Whilst there has been a recognition that quality is now improving, the time taken to secure the improvement means that certain key deliverables (such as several strategies) have yet to be finalised and/or have arrived so late that their use has diminished (such as the Roadmap).

The evaluation team used the VfM analytical matrix proposed in the Evaluation Plan. As noted in the Evaluation Plan the rubric is based primarily on a qualitative assessment of the evidence available, and uses a set of 13 questions to frame the evidence and then generate scores. It should be noted (and this point was flagged in the Evaluation Plan) that the Evaluation Team have constructed the VfM matrix based on the rubric used in the Year 3 Evaluation of Pacific Women, it part because the Year 3 Evaluation rubric represents good practice and in part because the evaluation team have not found the rubric that the Support Unit were meant to have developed (as referenced in the Performance Assessment for the period 1/03/2016 – 28/02/2017).

34

The evaluation team did however use the Support Unit evaluation rubric to shape the scale used in the VfM assessment. The scores therefore distinguish between satisfactory, partially satisfactory and unsatisfactory. The VfM analytical matrix provides a set of criteria used to guide the assignment of each score, against which we have assigned scores. By assigning a score for each question, we are then able to aggregate the scores against the 4E’s namely economy, efficiency, effectiveness, and Ethics.

As with other sections of the report the assessment is based on literature reviewed, interviews with DFAT officials and Support Unit staff, and relevant operations and financial data. Qualitative judgements have been made by the evaluation team to score each evaluation question, and a brief explanation has been provided to support the score.

4Es VfM Principles and Standards

ASatisfactory

BPartially

Satisfactory

CUnsatisfactory

Economy Efficiency in managing costs

Cost consciousnessScore = A

All decisions within the Support Unit are guided by DFAT’s procurement guidelines. The Support Unit annually reviews its service providers (for instance it recently switched IT providers after establishing that better, faster broadband could be obtained at a cheaper price elsewhere). Travel and logistic decisions are all based on cost consciousness principles. Savings have been achieved by, for example, by co-locating the PNG office with another programme that the managing contractor had been managing, and amalgamating a number of communication services into one contract. The implementation of the Fiji Women’s fund under the Support Unit, thus avoiding duplication of services is another example of cost consciousness decision making.

Encouraging CompetitionScore =A

All procurement decisions taken by the Support Unit are based on competitive tenders (typically 3 quotes), other than where standing contracts already exist (albeit these were all initially selected based on a competitive tender process)

Resource AllocationScore =B

Initially the resources allocated to operate the Support Unit and ensure quality outputs were inadequate. Considerable effort has now been take to ensure the right fit between resource allocation and quality outputs, including significant restructuring of the unit and the provision of appropriate personnel where best needed. For instance, Cardno HQ is being used far more to ensure quality outputs, and an editor has now been recruited. In addition, the programme has instituted much tighter quality assurance processes that include templates, terminology guidelines and so on. Nevertheless, challenges remain with regards to the allocation of resources, as they are primarily determined by the needs of the Annual Plan, but are not necessarily sufficient to address additional activities that become necessary through Tasking Notes.

Efficiency Efficiency in how well inputs are converted into outputs

TimelinessScore = C

In a number of instances timeliness is outside the control of the Support Unit, often relying on partners (including both service providers and Posts) to make design and implementation decisions. Where the Support Unit have had control over the delivery of specific outputs (such as the Capacity Strategy, the Communication Strategy, the Roadmap) they have had mixed success in meeting deadlines. Moreover, as noted above there were major delays in recruiting and/or replacing staff at the outset. Nevertheless, timeliness has improved significantly and all reports, for instance, are delivered on time.

Within budgetScore = B

As noted above, the SU has managed to keep expenditure within budget with regards to activities, and operating costs, and to date there has been an underspend across activities. Nevertheless, there were issues with respect to recruitment of experts and their level of compensation. Whilst this has been

35

4Es VfM Principles and Standards

ASatisfactory

BPartially

Satisfactory

CUnsatisfactory

resolved it has required a ‘top up’ from bilateral funds in PNG, and a recasting of positions in the Suva office. Initially there were also issues with the accuracy and timeliness of financial reporting, but in the most recent Performance Assessment DFAT acknowledged the improvement in end of FY forecasting and accuracy in the bi-monthly financial reports.

Efficient business processesScore = A

The managing contractor has put in place robust business processes to ensure appropriate levels of control, albeit the SU commenced (especially in PNG) with an absence of standard systems and processes for operating and managing the SU. Operational Manual has been recently revised to ensure business processes remain efficient. Moreover, as noted in the most recent Performance Assessment, DFAT notes that the unit has devolved budget management to team, added additional invoice review processes, and undertook a process audit to improve processes.

Acceptable transaction costsScore = B

The Support Unit currently provides most of its support remotely, and as things stand at present this modality is seen as the most financially viable. The down side of remote support provision is discussed elsewhere in the report, but it does mean that transaction costs are kept relatively low . The view amongst many respondents is that outputs would improve through a direct presence, and that a presence would create greater visibility to the programme. Were the Support Unit to operate offices outside of Fiji and PNG, the transaction costs would naturally increase but so would the benefits.

Effectiveness Effectiveness in how well outputs are converted to outcomes

Results focussedScore = A

Operations Reports prepared by the Support Unit illustrate that most planned activities and additional requests have been delivered in accordance with the workplan and are expected to meet the desired outcomes. Moreover, the Support Unit has become far more results focussed in its reporting. Where previously it relied heavily on partner reports, the unit now has a strong KMS in place and is able to verify reports independently through monitoring visits to ensure the validity of the data that is reported.

Experimentation and innovationScore = C

Where the Support Unit have been given opportunities to embrace innovation (for example the coordination forum conducted every 6-8 weeks by the office in PNG), the unit has been largely effective and has made a valuable contribution to promoting the programme. In many instances, however the Support Unit has little control over the design of activities and is thus not best placed to promote either experimentation or innovation.

Additional ResourcesScore = C

As both Performance Assessments prepared by DFAT illustrate, DFAT has had to provide substantial additional resources to oversee Support Unit performance. This is not sustainable, and we discuss elsewhere in the report that the time is ripe for DFAT to withdraw embedded staff from the Suva office (de facto this happened some time back in the PNG office). Providing oversight is a disproportionate use of DFAT time, and prevents DFAT officers from attending to other DFAT business. Moreover, as noted above, substantial additional resources have been provided to the Support Unit by DFAT to meet significant increases in the staffing costs of the facility. The Support Unit has only recently started operating with a full complement of staff.

Performance and Risk managementScore = A

The Managing Contractor has a performance management system in place. The system ensures that where performance has been unacceptable appropriate steps have been taken. A robust risk management system in place (the Risk Management

36

4Es VfM Principles and Standards

ASatisfactory

BPartially

Satisfactory

CUnsatisfactory

Strategy was updated and approved by the DFAT/PMC in June 2016) and the Support Unit routinely reviews risk and provides additional mitigation steps when necessary (for instance additional measures were taken to strengthen security in PNG office after most recent security review). Moreover, as part of their regular duties Support Unit staff draw DFAT’s attention to any issues that may pose a risk to progress being made by the Pacific Women Programme

Ethics Ethical Practices

Transparency and accountabilityScore = A

The Support Unit follows stringent due diligence practices, and is fully compliant with the Commonwealth Procurement Guidelines. All staff recruitment, for instance, follows best practice, is wholly transparent and recruitment reports are routinely prepared outlining how and why decisions were taken. Moreover, all DFAT policies (such as the Child Protection Policy, Gender Equality and Disability Inclusive Development Policy) are strictly adhered to, and all staff have been trained in the application of these policies.

EquityScore =A

As noted above, 72% of the vendors used by the Support Unit are local, a further 12% have been drawn from elsewhere in the Pacific, and only 17% of the vendors are international (primarily based in Australia). Furthermore, whilst 71% of STA’s have been recruited internationally, the unit has managed to use 12 National STAs to date. In addition, both the Gender panel and the M&E panel include Pacific Islanders.

37

Relevance of the ModalityEQ 3: Has the Support Unit remained appropriate to meeting the objectives it was designed to meet?

Key Findings: Whilst the modality has provided benefits to the programme and remains relevant in meeting the needs of partners, barriers and challenges could continue to hinder the efficacy and efficiency of the model. A review of similar models being implemented by DFAT in, and adjacent to, the region suggest a number of ideas to improve the model including ensuring an in-country presence when delivering across multi-countries, drawing on technical skills within the governing/advisory body to review and advise on implementation, appointing an independent facilitator/moderator to strengthen relationships between the Support Unit and DFAT, and ensuring the AWP includes specific innovative actions.

In this section of the report the evaluation team assess the relevance of the modality in answering the EQ. We structure this section slightly differently to the previous sections, albeit the sub-questions (as outlined in the Evaluation Matrix, and as specified in the ToR) remain applicable.

DFAT does not yet have an agreed definition of an aid facility. The term can be used to cover a wide range of potential arrangements, and (as Annex 5 illustrates) there is often no clear distinction between a program and a facility. Typically, the term facility covers a flexible mechanism where the high - level objectives are clearly spelt out, but the pathway to these objectives is left unspecified. Over time (usually on an annual basis) DFAT and the managing contractor of a facility would develop a joint work plan to shift resources in line with evolving priorities, context and experience. It is this responsive and flexible nature of a facility that is both attractive to DFAT (there is an expectation, for instance, that a facility leads to increased efficiency for DFAT staff through the consolidation of the administration of multiple activities, freeing them up to focus on strategic level issues) and problematic for DFAT (for instance facilities can drift if there is inadequate strategic direction, fragmented portfolios, and poor M&E when objectives are unclear)21.

The evaluation team examined five facilities operating within, or adjacent to, the Pacific to get a sense of what could be learnt from other facilities22 and thus help us to begin to answer whether the model is ‘fit for purpose’. Our review of different facilities (as outlined in Annex 5) suggests that successful facilities have several common characteristics, albeit different development contexts and needs can vary enormously. Thus whilst ‘one size does not fit all’, certain common features can be noted with regards to facilities. These common characteristics include a relatively robust design from the outset (and this includes defining both the purpose and the scope of the facility), an expectation for ongoing design activities during implementation (including the development of detailed approach to M&E), a credible approach to human resources (including ensuring those recruited have the necessary skills and competencies), and clear governance and management arrangements.

Other features of the five facilities we looked at, which may be instructive to the Support Unit, include:

Each facility is modelled on a substantial design document which outlines the purpose behind the facility, its scope of services, key personnel requirements, roles and responsibilities of both the facility and DFAT, and the governance structure that will have oversight over the facility.

21 Foreign Affairs, Defence and Trade Legislation Committee, Senate Estimates, 26 October 2017, pp.58 – 62, 100 – 103. See also de Lacy, J. “Facilities deserve a place in development’, Devpolicy Blog from the Development Policy Centre, (http://devpolicy.org/facilities-deserve-a-place-in-development-20171113/, accessed 13 November 2017).22 The five facilities are – the Investing in Women Initiative, MAMPU, Trade and Inclusive Economic Growth Facility, Market Development Facility, and the Transport Sector Support Program

38

Facilities require long term commitments (and evaluation reports highlight greater efficacy being achieved in the 2nd phase, typically because of lesson learning in the 1st phase) and are usually predicated on a sharp learning cure during initial start-up.

Reducing DFAT’s administrative function is a key, but not sole, reason for establishing a facility. There is a continuum of types of facilities, ranging from those implementing programs to those that

support implementation There is also a continuum regarding the functions of the facility. Whilst this is linked to the purpose

of the facility, DFAT have, for instance, tasked facilities with varying roles regarding influencing the strategic direction of the respective programmes.

Location of a facility is dependent on the nature of the program (single country focus typically means the facility is located within the capital of the partner country), however facilities supporting multi-country programs typically have multiple offices/country teams in place.

Core leadership teams for facilities are remarkably similar (e.g. TL, Financial Manager, M&E Specialist, Communications, Operations Manager)

Governance arrangements are influenced by the scope and nature of the facility, with varying levels of partner country participation in the governance structure of each facility. In addition, Governance bodies playing an active role in reviewing the implementation of activities, and providing technical guidance (typically based on site visits).

The facilities all play a substantial role in not only providing administrative support, but in the implementation of their respective programs

Within the Transport Sector Support Program facility there is a Programme Director (separately contracted to the Managing Contracted facility) who provides a buffer/moderator between the facility and DFAT (see description of roles and responsibilities in Annex 4) – the key feature of the PD’s role is to ‘provide strategic oversight, lead scoping of activities, oversight over engagement with PNG government, give directions to ISP on implementation, provide advice to DFAT on implementation and performance’ of the program’.

MAMPU program has established a dedicated Innovation Fund for the facility to manage, which the facility must use to identify, design and then implement innovative and experimental activities outside other activities it is supporting

Our review of facilities also identified two aspects of the Facility model where problems can arise. The first is to do with the challenge of balancing flexible and responsiveness with the need for a clear program logic to ensure activities contribute meaningfully to the realisation of objectives. Many facilities have initially struggled to develop a meaningful Theory of Change and/or MEF with which to measure progress. Typically developing such a framework or ToC is more complex than initially envisaged, and often requires an iterative approach before a robust M&E system can be implemented.

The second challenge is to successfully negotiate the trade-offs in facility relationships, especially the extent to which DFAT will be responsible for strategy and partner government relationships. Our review found facilities located across the relationship continuum. There are facilities that mange day to day activities and logistics, whilst there are other facilities where the contractor does much of the strategic thinking and planning. Our review suggests that determining the appropriate mix of responsibilities for a facility can only be determined over time, will depend on the level of trust established between DFAT and the facility, and will depend on the extent to which the facility can demonstrate its effectiveness in being both strategic and capable of building relationships with partner governments.

It is our contention that the Support Unit does satisfy the common features of a facility. After all the Support Unit has clear governance and management arrangements, has a clear purpose, is now staffed with an appropriate mix of skills and competencies, and through the AWP it is being responsive and flexible to the changing needs of the context within which it works. However, where the challenges have come in (such as

39

issues to do with quality assurance, and delays in delivery) speak more to the effectiveness of implementation rather than the modality itself.

Nevertheless, where the modality has been problematic has been to do with the managing of relationships across the program. In the original design document of the Support Unit23, the notion of the facility was premised on DFAT being able to engage at a strategic level and focus on engagements with partners on systemic blockages to gender equality, and engage in Policy dialogue and influencing work, and promoting coalitions for change. Leaving the Support Unit to assist DFAT in these initiatives, and free up DFAT staff to play these roles.

To a large extent, the Support Unit’s role has been about supporting DFAT play the strategic level role in the region (as noted above in our discussion of the extent to which the Support Unit has been effective and efficient in delivering against the Head Contract). Nevertheless, in interviews with a range of key stakeholders the evaluation team noted that there remains ongoing dissatisfaction over the role the Support Unit should be playing in promoting relationships. From the perspective of the Support Unit, there is a belief that certain assumptions about the modality may not be valid and have in turn hindered the unit from being more strategic (such as the view that support to partners can be provided remotely and that GFPs, with support from other staff in Posts and Canberra, will oversee their respective Country Plans). DFAT on the other hand has expressed frustration with the Support Unit for not seizing either the strategic or innovative opportunities they have been presented with. Moreover, DFAT staff contend that they have not prevented the Support Unit from travelling to other countries to support partners or processes, instead they requested that this be justified with specific intended outcomes. Whilst it is not the Evaluation Team’s responsibility to pass judgement on this issue, it is clear to us that this issue remains contested and greater clarity and agreement between the unit and DFAT is necessary.

Moreover, the evaluation team also observed that the successful implementation of the modality is also tested by the manner in which DFAT engages with the different offices of the Support Unit. On the one hand there is a sub-office in PNG which is directly implementing the PNG country plan activities (including the grant management of the majority of the partners in PNG), and on the other hand the office in Suva is delivering support to 13 countries across the region, manages fewer grants, but is expected to provide a significant amount of operational support to the program as a whole. The evaluation team observed that the sub-office appears to have a much closer working relationship with implementing partners than Suva has with many of its partners. However, where the Suva office has had direct relationships with partners (such as with grantees in Fiji), there is a noticeable improvement in the working relationship between the unit and the partner.

The quality assurance issue has also provided both a challenge to the Support Unit, and a barrier to developing a close working relationship with DFAT. It has already been noted above that much of the QA issue was dealt with poorly at the outset by the unit, and that the level of micro-management administered by DFAT in response has created a model that is time intensive and unlikely to be sustainable in the long term.

The Year 3 Evaluation of Pacific Women (2017, p.44/45), also observed several barriers that have had an impact on the Support Unit, namely

There have been different kinds of delays in getting started both at programme and project level. In particular, the Support Unit has been slower than anticipated in getting established for a number of reasons including recruitment challenges and a lack of clarity concerning what was expected of the

23 Pacific Women Shaping Pacific Development Design Document, April 2014. See Annex Seven (pp. 138 – 140) which clearly outlines the different roles and responsibilities for program governance and management, with specific reference to the different levels of management within DFAT.

40

Unit. Not surprisingly – given the scope of the programme – not all countries initiated activity immediately. Those that became active quickly – Cook Islands, PNG and Solomon Islands – did so without the depth of consultation that later became expected in the programme. Countries began the process of country plan development in line with their own budget cycles – for this and practical reasons related to the scale of the programme, a phased approach was taken such that some countries became operational later.

Nevertheless, despite facing a number of challenges, the modality is working well. This can be seen by the extent to which implementing partners can access the services provided by the unit and the extent to which they are satisfied with the services they have accessed.

The extent to which the Support Unit regularly communicates with your organisation.

The level of cooperation and support that the Support Unit provides your organisation.

The usefulness and timeliness of information related to the Pacific Women programme the Support Unit has provided to your organisation.

The usefulness and timeliness of information directly related to the grant the Support Unit has provided to your organisation.

The effectiveness of the Support Unit in addressing problems in relation to implementation of your activity.

0% 20% 40% 60% 80% 100%

90.91%

86.96%

86.96%

77.27%

73.91%

Unsatisfactory Partially Satisfactory Satisfactory No Opinion

Figure 5: Directly managed grantees’ rating of overall support and communication provided by the Support Unit (source: Partner Satisfaction Survey)

Figure 5 highlights that directly managed grantees are largely satisfied with the overall support they receive, and the way the Support Unit communicates with grantees (91% of grantees were satisfied with the extent to which the Support Unit communicates with them, 87% found the level of cooperation satisfactory, and 74% were satisfied with the effectiveness of the unit in addressing problems).

The extent to which payments have been on time in accordance with the agreed schedule in the contract.

The ability of the Support Unit to provide logistic support to your organisation (if required).

The ease of negotiating the contract between your organisation and the Support Unit.

The Support Unit’s responsiveness to requests for assistance with financial management and reporting.

0% 20% 40% 60% 80% 100%

82.61%

69.57%

69.57%

60.87%

26.09%

39.13%

Unsatisfactory Partially Satisfactory Satisfactory No Opinion

Figure 6: Directly managed grantees' rating of contracting, financial management and administrative support provided by Support Unit (Source: Partner Satisfaction Survey)

41

Moreover, directly managed grantees are satisfied with contracting, financial management and administrative support provided by the Support Unit (As shown in Figure 6: 83% were satisfied with payment processes, 70% were satisfied with logistics, 70% found the ease of negotiating contracts satisfactory, and 61% were satisfied with the responsiveness of the unit with respect to financial reporting and reporting). The relatively high number of directly managed grantees who chose the no opinion option did so primarily because they had not yet had an opportunity to engage with the Support Unit with these aspects of implementation.

Responses from the Gender Focal Point survey (a survey separately administered by the evaluation team, and independent of the Support Unit’s own GFP survey) paints a far more mixed level of satisfaction with Support Unit services (Figure 7 and Figure 8). Whilst the Support Unit has provided a wide range of support to GFPs (ranging from organising annual Gender Focal Point Workshops, to targeted support initiatives related to the development of Country Plans and their subsequent design, in addition to many other actions as per the AWP in order to support the implementation of the respective country plans such as budgeting, procurement, reporting, M&E and so on) the GFPs remain ambivalent in their relationship with the Support Unit as the following graphs illustrate.

Ability of Support Unit staff to engage with DFAT.

Ability of Support Unit staff to engage with implementing partners.

Ability of Support Unit staff to engage with other stakeholders.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

66.67%

44.44%

40.00%

Unsatisfactory Partially Satisfactory Satisfactory No Opinion

Figure 7: Rating by GFPs of Support Unit’s effectiveness in engagement (Source: GFP Survey)

Mixed views from the GFPs align with points made to the Evaluation team during interviews and focussed discussions with GFPs regarding the extent to which GFPs are either satisfied or dissatisfied with the Support Unit. On the one hand roughly half the GFPs commended the Support Unit (GPFs made comments such as ‘the SU invaluable in supporting work’, ‘very supportive in development of our Country Plan, and ‘great that the SU can take the admin load off us’). On the other hand roughly half the GFPs were less than satisfied with the Support Unit. Those GFPs who were dissatisfied made mention of range of concerns, including: ‘untidy contractual arrangements put in place by the SU’, ‘SU turnaround time for feedback and response was slow’, ‘ongoing confusion about the role of SU and its relationship with the partner government’, and ‘reports too text heavy – we want more visuals and illustrations’.

42

Relevant expertise and skills of staff of the Support Unit to deliver its services and support.

Quality of management and leadership of the Support Unit

Effectiveness in promoting and supporting innovation.

Appropriateness of the organisational structure of the Support Unit for delivering its services and support.

Extent to which the Support Unit delivers against its Annual Plan.

Ability to respond to additional requests that are outside the Annual Work Plan.

0% 20% 40% 60% 80% 100%

60.00%

50.00%

50.00%

40.00%

40.00%

40.00%

Unsatisfactory Partially Satisfactory Satisfactory No Opinion

Figure 8: Rating by GFPs of Support Unit’s overall effectiveness (Source: GFP Survey)

Quality of the gender technical assistance provided by the Support Unit.

Quality of Support Unit's performance with respect to travel and other logistical support.

Quality of Support Unit's performance with respect to event coordination.

0% 20% 40% 60% 80% 100%

40.00%

70.00%

60.00%

Unsatisfactory Partially Satisfactory Satisfactory No Opinion

Figure 9: Rating by GFPs of Support Unit’s effectiveness in providing logistic and technical support (Source: GFP Survey)

The graphs above highlight that whilst directly managed grantees are largely satisfied with the support they are receiving and thus the modality is providing the support they require, GFPs remain ambivalent towards the Support Unit. This is not surprising, bearing in mind that GFPs have multiple duties (not all of which are to do with Pacific Women), they have variable experience working within DFAT24, and they receive fluctuating levels of support within their Post. The Year 3 Evaluation of Pacific Women (2017, p.53), also commented on the relationship between GFPs and the Support Unit, noting that

‘there are still some areas of unaligned expectations in some relationships between DFAT Posts and the Support Unit (country case study Fiji, Kiribati) despite closely aligned overall commitment to women’s empowerment and gender equality. Some lack of clarity in the evolving division of responsibility for funded projects between the Support Unit and Posts – and between the Support Unit and the Pacific Women management team– which has not been clearly or consistently articulated, creates a situation in which expectations are not always being met, or where stakeholders perceive that responsibilities are not being appropriately allocated and managed. This includes, for example, that the Support Unit is currently accountable to DFAT (Canberra) for collating

24 Interviews with GFPs established that whilst some had been working with DFAT for many years, some had only been recruited within the past few months.

43

and analysing project reporting but does not generally have a direct relationship with projects. This has been an issue for the Support Unit when reporting has not included quality information, meaning the Support Unit needs to go back to the GFP, who then goes back to the partner, which delays reporting and seems time and resource inefficient’.

The ambivalence that GFPs have about the Support Unit also points to a need that the unit has not yet met. Whether the Support Unit can reasonably meet the needs of GFPs will need to be explored further by the program. Areas for further exploration (based on feedback provided by GFPs to the evaluation team) could include:

Ensuring GFPs have a nuanced understanding of the role and purpose of the Support Unit

Greater clarity on the roles and responsibilities of the Support Unit in country and their expected relationship with regards to GFPs (e.g. how best and how often should TA be briefing the applicable GFP on activity progress)

How best to foster better joint approaches in-country between the Support Unit and GFPs, with particular focus on ensuring a common approach to implementing the Country Plan

GFPs need to be make a conscious effort to become more familiar with the Program

More regular interaction between GFPs and Support Unit staff (such as a quarterly teleconference in order to discuss progress, challenges, and forthcoming activities.

Returning to the question of whether the modality is ‘fit for purpose’, it is our contention that whilst the modality is appropriate, certain adjustments are necessary to ensure the approach realises greater effectiveness and efficiency for the program. In the following section we provide a set recommendations regarding adjustments to the Support Unit modality which we believe will bring about greater efficacy and efficiency.

44

Recommendations

Taking our considerations into account, including our review of different facilities and the conclusions we have drawn from our findings, the evaluation team make the following recommendations to improve the Support Unit. In making these recommendations the evaluation team also considered the following:

Working relationships between DFAT, Head Contractor and the Support Unit whilst cordial could be improved to realise the stated objective of the ‘the one team approach’.

The appointments of new incumbents for DFAT Program Manager (Suva), DFAT Counsellor (PNG) and Team Leader (Support Unit) present a timely opportunity to crystalize roles and responsibilities, adjust reporting lines and improve relationships and ways of working for the Support Unit.

The Suva and PNG offices now have a full complement of staff and are better positioned to assume the responsibilities required of the Support Unit.

The operations of the Support Unit should be more effective with the updated Operations Manual, revised MELF, and recent drafts of the Communications Strategy, Capacity Building Strategy and Research Strategy.

Senior staff in the Support Unit have had sufficient experience and lessons learning in Phase 1, with the facility model and at managing relationships with both DFAT and Cardno.

Phase 2 is an opportune time to enable the Support Unit to take on a more strategic role.

Recommendation Our thinking Establish Sub-regional

offices, managed by a regional Program officer25 or at the very least, ensure that relevant SU staff visit each sub-region on a quarterly basis (having prepared an appropriate pre-departure plan that motivates and specifies activities to be completed during the visit). Indicative sub-regions for discussion between the SU and DFAT:

Suva for Fiji, Solomon Islands, Vanuatu

PNG for Papua New Guinea

Samoa/Tonga for Cook Islands, Niue, Samoa, Tonga, Tuvalu

The breadth and spread of the countries and program activities (through the respective Country Plans) necessitate greater program coordination and adjustment.

Creating geographical clusters enables more focused yet economical support to countries, networking, reflections and learning (across countries and across thematic areas) within the same region.

Regular, programmed visits would create much needed presence and consistent, focused reach within each country in the respective sub-regions.

Visits would facilitate information flows and greater connectivity between regionally funded partners and bilaterally funded partners in countries within their region

Increase opportunities to facilitate joint forums for gender advisors; partners working in outcome areas of WEE and Women’s Leadership; and players developing new working areas in same region.

Will promote opportunities for diversifying the staffing of the Support Unit and building regional capacity.

Will allow cost-effective coordination meetings, emulating the success of the sub-office in PNG.

Would work more consistently with GFPs to support implementing partners, assess areas for capacity building and identify potential for innovation.

25 Existing, relevant Support Unit staff would be located within the proposed 4 sub-regions. Cost implications are relatively minimal as the 4 regional Program officers could be located with other organisations (for instance co-location is currently practised by the Pacific Leadership Program and rent is therefore kept to a minimal).

45

Pohnpei for FSM, Kiribati Nauru, Palau

Regular visits could be used to foster greater involvement by Posts in the program

Would be a useful source for inputs to communication products

Appoint an independent Partnership broker that engages with both DFAT and the SU on a ¼ or biannual basis to help clarify and strengthen relationships and ensure the Facility remains on track.

Need for an independent facilitator/moderator between DFAT and Support Unit.

The partnership broker’s role would be to facilitate a ‘more joined up’ approach for the Program, and help strengthen relationships between the SU and DFAT.

If applicable, the broker could also facilitate relationship building between the SU and Gender Focal Points.

Establishment of an Innovation Fund, or at the very least identify innovative actions in the AWP

To date the Support Unit has had mixed results with promoting innovation.

A dedicated innovation fund would provide the incentive to the Support Unit to design and then implement innovative and experimental activities outside its core activities, but relevant to partner countries.

Alternatively, specify innovative activities to be undertaken by the SU on an annual basis in the AWP (with appropriate budget committed to these activities), to be jointly identified in discussion between DFAT and the SU.

Operational Adjustments Adjustments to improve the effectiveness of the Support Unit:

All outstanding strategies listed in the Head Contract need to be completed within a realistic, but tight timeframe agreed between the Support Unit and DFAT.

Establish service level agreement on: categories of tasking notes and criteria to determine level of

relevance and priority26

standard time (notice period) for development of concept notes, communication products, work plans, reports

turnaround times for responses and comments to draft reports, TORs, strategies and communication products

The Support Unit should use reports and regular meetings between DFAT and the Support Unit to voice their reflective and strategic thinking, based on their interactions with partners.

Other channels of communication should be promoted by the Support Unit, including radio.

An induction pack to be provided to each GFP, Support Unit staff and implementing partner. The pack to briefly outline the objectives of Pacific Women, roles and responsibilities of the Support Unit, GFPs and list key contacts.

Actively recruit other Pacific Islanders to work in their respective offices and/or establishing an internship programme that allows

26 Tasking notes currently vary from AUD$5k (Support to Tonga WCC – 6 months) to AUD$1.4m (PNG Bounganville Gender Investment – 5 years)

46

other Islanders to do a work placement in the two offices. Recruitment needs to be advertised where it is more accessible to Pacific islanders e.g. local newspapers of participating countries, email distributions in pacific networks and so on.

The swift implementation of the Capacity Building strategy and a deliberate, more specific and systematic plan for training and mentoring of Pacific islanders is essential.

As per the newly developed Capacity Strategy DFAT LES and implementing partners would benefit from spending time in the Support Unit to give them a better sense of the totality of the programme (and help promote active networking between the staff, develop a common understanding of work and so on).

A more effective recruitment and retention (performance management) system will help stabilise staffing as well as build capacity and ensure continuity.

It is an inefficient use of DFAT resources to locate any DFAT staff within the Support Unit, a weekly meeting should suffice.

A systematic process be developed to gather and then share lessons learnt, especially through reports prepared by the Support Unit

47

Conclusion

The objective of this evaluation was twofold, namely ‘to assess the effectiveness of the Support Unit in delivering services against the DFAT Head Contract and Schedule of Requirements’, and ‘to evaluate the Support Unit model’. We draw on the findings from the previous section to present our conclusive response to the two objectives.

In terms of the effectiveness of the SU in meetings its contractual obligations, the evaluation team found:

a) That whilst the issue of quality assurance has profoundly influenced perceptions of the effectiveness and efficiency of the Support Unit, the unit nevertheless displays a wide range of strengths, has employed a host of skilled people, has achieved several significant achievements, and has delivered many of the services expected of it. However, the failure to resolve quality assurance issues and deliver key strategies within a reasonable timeframe is unsatisfactory.

b) Barriers to effective operations include the assumption that support to partners can be provided remotely; that GFPs, with support from other staff in Posts and Canberra, will oversee their respective Country Plans; the underutilisation of the skills of the Support Unit (many who have been recruited for the breadth of their technical skills and knowledge of the region, yet perform relatively straightforward program management tasks); substantial delays in formalising an M&E framework for the facility which would have helped inform attempts to maintain a coherent facility; irregular contact with partner countries to build and maintain relationships; and a small pool of grantees to manage.

c) Enablers for effective operations, which have yet to be fully utilised, include breadth and depth of the Support Unit’s relationship with partners, the strategic thinking within the unit, using the recruitment process to promote a more regional workforce, and systematically capturing lesson learning to overtly influence the direction of the Support Unit.

In terms of the extent to which the modality being used is ‘fit for purpose’, the evaluation team found that:

a) Whilst the SU model is an effective means to support the implementation of the Program Women’s implementation, greater efficacy could be achieved through a series of recommendations which the evaluation team have outlined in the previous section.

b) Key lessons observed from examining both the Support Unit and other facilities in/adjacent to the Pacific region include:

a. A detailed design of the facility should be developed prior to commencement of the program.

b. An accurate and realistic resourcing plan should be developed prior to inception, and this includes the need for better forecasting of human resources.

c. The inception period should be clearly demarcated, within which all reasonable attempts need to be made to establish systems and develop pertinent strategies.

d. Recruitment and orientation of key personnel should take into account prospective candidates’ familiarity with both DFAT and Managing Contractor arrangements.

e. Staff turnover is an inevitable risk in multi-year programs, but a clear mitigation plan needs to be developed and agreed upon by all parties at the outset to minimise disruptions that will occur when key personnel depart.

f. Remote support may lead to initial cost savings, but it creates inefficiencies which can be costly to remedy.

48

g. An appropriate quality assurance system needs to be developed during the inception period, with roles and responsibilities identified and a clear description of the process that will be followed to assure quality.

On the question of whether to extend the Managing Contractor’s contract, the Evaluation Team believe that it should be extended for at least the next 12 months. Now that the Support Unit is fully staffed, and there is a recognition that performance is gradually improving, the Support Unit should be given an opportunity to demonstrate its ability to provide effective and efficient support to the Program. We suggest that DFAT and the Support Unit agree on a set of performance related targets which the Support Unit will need to meet over the following 12 months. Failure to meet the agreed targets, without satisfactory mitigating factors, would give DFAT cause to reassess any further extensions of the contract.

We believe it would be imprudent not to renew the contract for reasons already alluded to in this report. Firstly, Facilities tend to operate more effectively in the second phase than the first phase, as result of the considerable lesson learning that takes place during the initial phase. Secondly, replacing the Support Unit at this point would require considerable additional work by DFAT to re-establish relationships between a new managing contractor and partners and Posts, transfer institutional memory to a new facility, and promote effective working relationships with a new managing contractor. Thirdly, replacing the managing contractor would not necessarily equate with novating existing experienced support unit staff to the new contractor. Losing staff who now have unique insights into the needs of the Program could undermine the momentum of the Support Unit and create further delays which the Program can ill-afford.

49

Annex 1: List of People Consulted

Name Organisation DesignationTracey Newbury DFAT Director – Gender Equality and Disability Inclusiveness Section

(DFAT – Canberra)Suzanne Bent DFAT First Secretary – Gender (Support Unit Manager – Suva)Nilesh Goundar DFAT Program Manager (Support Unit – Suva)Susan Ferguson DFAT Counsellor Gender (DFAT – PNG)Gordon Burns DFAT Counsellor – Regional Development Cooperation (DFAT – Fiji)Chelsea Magini DFAT PNGPatricia Bachtiar, DFAT Senior Program Manager, Poverty and Social Development

(DFAT – Indonesia)Jason Court DFAT 1st Secretary, KiribatiErmimeta Barako DFAT GFP, KiribatiLeaine Robinson DFAT GFP, Fiji‘Ana Talanoa Baker DFAT GFP, TongaTracey Tupou DFAT GFP, TongaRonicera Fuimaono DFAT GFP, SamoaAngus Hinton DFAT GFP, NauruVeronica Halstead DFAT GFP, NauruSuzanne Gallen DFAT GFP, FSMLily – Anne Homasi DFAT GFP, TuvaluFlori Alalo DFAT GFP, Solomon IsLouise Scott DFAT GFP, Solomon IsRoselynne Kenneth DFAT GFP, PNGPatricia Fred DFAT GFP, VanuatuLinda Petersen SU (Fiji) Team LeaderSarah Boxall SU (Fiji) Deputy Team LeaderEmily Miller SU (Fiji) Monitoring & Evaluation ManagerAlina Meyer SU (Fiji) Gender SpecialistHilary Gorman SU (Fiji) Senior Programs ManagerShazia Usman SU (Fiji) Communications CoordinatorCynthia Ramirez SU (Fiji) Finance and Administration ManagerAmelia Kamanalagi SU (Fiji) Finance and Administration AssistantPranit Sami SU (Fiji) Program OfficerTara Chetty SU (Fiji) Senior Programs Officer – GenderShirly Narayan SU (Fiji) Senior Programs OfficerAmali Shaw SU (Fiji) Knowledge Management & Research OfficerVasiti Nailele SU (Fiji) Operations and Administration OfficerRichelle Tickle SU (PNG) PNG Country ManagerMercy Masta SU (PNG) Program ManagerBrenda Andrias SU (PNG) Monitoring & Evaluation OfficerTovi Amona SU (PNG) Administration & Finance OfficerChris Bray SU (PNG) Transport and Administrative Assistant Nea Harrison SU (PNG) Consultant, M&E AdvisorLois Knight Cardno Emerging

MarketsContractor Representative

Maria Hsu Cardno Emerging Markets

Program Coordinator

Colin Adams Cardno Emerging Markets

Asia Pacific Area Manager

Louise Morrison Cardno Emerging Markets

Principal

50

Name Organisation DesignationBrad Bowman Cardno Emerging

MarketsRegional Manager, PNG & Pacific

Virisila Buadromo Adaptive Leadership & CSO Facilitator

Long Term Adviser

Carol Nelson Gender Adviser, Tonga Long Term AdviserDr Helen Tavola Women’s Leadership &

Women’s Disability, Tuvalu

Long Term Adviser

Maire Dywer Gender Adviser, North Pacific

Long Term Adviser

Carol Nelson Gender Adviser, Tonga Long Term AdviserColleen Peacock Tautai

Gender Adviser, Samoa Long Term Adviser

Kim Robertson SPC Adviser – Gender StatisticsLoata Leweniqila Women in Fisheries

NetworkProgram Manager

Reema Pratap Women in Fisheries Network

Finance Manager

Milika Sobey Women in Fisheries Network

Executive Chair

Kaliti Mate South Pacific Association of Theological Schools

Program Manager

Sandra Bernklau UN Women Technical Specialist, Markets for ChangeLaisani Peterson UNICEFSarah Kernot IWDA/We Rise Senior Program ManagerShannon McVey Populations Services

International, PNGCommunications & Marketing Director

Eli Webb Centre for International Private Enterprise, PNG

Country Coordinator

Marcia Kalinoe FSVAC National CoordinatorJawad Khan Market Development

Facility, PNGMarket Development Advisor

Trish Sawford Australian Awards Professional Development and Partnership , ManagerKatherine Lepani Papua New Guinea

Governance FacilityGender Equity Specialist

Edwina Kotoisuva Justice Services and Stability for Development Program

Deputy Team Leader

Agatha Pio US Embassy Gender AdviserJosephine Yos Oil Search Foundation Program ManagerProfessor Betty Lovai University of Papua New

GuineaExecutive Dean, School of Humanities and Social Sciences

Jean Silviack Department for Community Development and Religion

Deputy Secretary

Mirriam Dogimab FHI360 Program ManagerChristine Chong UN Women Safe City ProgramDavid Ephraiam UNDP National ConsultantJustine McMahon CARE Country ManagerCharlotte Kakebeeke Oxfam Program ManagerVerena Thomas QUT Team LeaderJackie Kauli QUT Program officerAsefa Tolessa Dano UNICEF Chief Child Protection Miranda Forsyth State, Society &

Governance in Associate Professor

51

Name Organisation DesignationMelanesia Program, Australian National University

Nicole Hayley State, Society & Governance in Melanesia Program, Australian National University

Associate Professor

Robyn Cornford Centre for Excellence in Financial Inclusion

Centre for Excellence in Financial Inclusion

Michael Dalton Transport Sector Support Program PNG

Program Director and Policy Advisor

Kate McCubbin, Transport Sector Support Program PNG

Acting Team Leader

52

Annex 2: List of Documents Reviewed

1. Assessment of the Pacific Women Support Unit (May 2016 – April 2017)2. Country and Regional Annual Activities May 2014

3. Deed of Amendment 1 for Pacific Women Shaping Pacific Development Support Unit: Cardno & DFAT (11 August 2017)

4. Delivery Strategy: Pacific Women 2012-2022

5. DFAT Aid Programming Guide (Oct 2016)

6. DFAT Child Protection Policy (Sept 2016)

7. DFAT Procurement Policy (March 2017)

8. DFAT Support Unit Partnership Principles: Ways of Working

9. Gender Focal Points Survey 2017

10. Gender Focal Points Survey Results 2016

11. Pacific Women 3-year Evaluation: ITAD 7 April 2017

12. Pacific Women Annual Progress Report 2015-2016

13. Pacific Women Capacity Development Strategy 2017-2019

14. Pacific Women Communications Strategy: December 2015-December 2018

15. Pacific Women Design Document April 2014

16. Pacific Women First Progress Report 2012-2015

17. Pacific Women Monitoring & Evaluation Framework Dec 2015

18. Pacific Women Org Chart August 2017

19. Pacific Women Program Management Committee Records (Nov 2015-Feb 2017)

20. Pacific Women Program Progress Report March – Aug 2015

21. Pacific Women Program Progress Report March – Aug 2016

22. Pacific Women Program Progress Report Sept 2016 – Feb 2017

23. Pacific Women Research Strategy June 2017-June 2022 (Draft)

24. Pacific Women Roadmap May 2017

25. Pacific Women Style and Writing Guide: June 2017

26. Pacific Women Support Unit Agreement

27. Pacific Women Support Unit Consultants

28. Pacific Women Support Unit PPA: Cardno Response April 2016

29. Pacific Women Support Unit PPA: DFAT-SU Discussion (21 Feb 2016)

30. Pacific Women Support Unit: Operations Report March - August 2016

31. Pacific Women Support Unit: Operations Report Sept 2015

32. Pacific Women Support Unit: Operations Report Sept 2015-Feb 2016

33. Pacific Women Support Unit: Tasking Notes 2014-201634. Pacific Women Support Unit Vendors35. Papua New Guinea Country Plan Mid Term Review Report (draft)36. Papua New Guinea Performance Report: Pacific Women SU May 2016

53

37. Papua New Guinea Program Coordinating Meeting Minutes (25 Aug 2016 & 1 Aug 2017)

38. Partner Performance Assessment (PPA): April 2015-Feb 2016

39. Partner Performance Assessment (PPA): March 2016-Feb 2017

40. Review of Pacific Women MELF 2017

41. Support Unit Annual Work Plan Jan 2016-June 2017

42. Support Unit Annual Work Plan July 2017-June 2018

43. Support Unit Assessment Rubric (Revised Nov 2016)

44. Support Unit Design 2015

45. Support Unit Mobilisation Management Plan May-July 2015

46. Support Unit Operations Manual

47. Support Unit PNG Annual Work Plan 2014-2015 (Draft)

48. Support Unit Restructure Proposal: Recruitment and Development Plan 2016–2018

49. Support Unit Risk Management Plan 2017

50. Support Unit Work Plan 2014

54

Annex 3: Support Unit Organogram

Support Unit Organogram (June 2015):

55

Support Unit Organogram (August, 2017):

56

Annex 4: Expenditure against Annual Work Plan

Description Key Result

Area Total

Budget (Jan 2016 - June

2017) Spent: Jan - June 2016

Spent: July 2016 - June

2017

Balance as of 30 June

2017 (KRA) Programs Team        

Regional        

Uniting Church Grant          

Grant to Uniting Church KRA 6 209,000.00

102,000.00 107,000.00 -

Scoping Study on Gender Research          

Gender evidence-based research and analysis on / by Pacific islands countries Research 125,527.00

134,727.71 (3,477.94) (5,722.77)

Last taboo research Research 500,000.00

375,000.00   125,000.00

Development of the Pacific Women Research Strategy (including writeshop in Oct) Research 90,000.00   13,911.97 76,088.03

Research ** (TBD) Research 500,000.00

    500,000.00

Roadmap          

Development of Roadmap for Women and Leadership KRA 4 67,650.00 150.00 48,834.26 18,665.74

Development of Roadmap for Women’s Economic Empowerment KRA 5 121,888.00

150.00 102,167.31 19,570.69

Development of Roadmap for Ending Violence against Women KRA 6 105,849.00

10,150.00 80,122.35 15,576.65

Reference Group honorariums   45,000.00 - 18,700.00 26,300.00

Roadmap consultation costs   41,500.00   1,472.70 40,027.30

Printing, Publishing, Meetings/Workshops and Communications (Post Roadmap process)   75,380.00     75,380.00

Review of Counselling Services for Violence against Women in the Pacific          

Consultancy support to review counselling services for violence against women in the Pacific KRA 1 48,705.00 5,000.00 43,636.35 68.65

Pacific Women Parliamentary Program Annual Forum          

Logistics for Pacific Women Parliamentary Program Annual Forum KRA 4    

57

Description Key Result

Area Total

Budget (Jan 2016 - June

2017) Spent: Jan - June 2016

Spent: July 2016 - June

2017

Balance as of 30 June

2017 (KRA) 150,000.00 150,000.00

Pacific Women Program Evaluation          

Year 3 Evaluation of Pacific Women Program M&E 300,000.00

  212,795.60 87,204.40

Midterm Evaluation of SPC Progressing Equality in the Pacific M&E 49,126.08 10,000.00 38,361.20 764.88

Midterm Evaluation of the Pacific Women Parliamentary Program M&E 49,337.00 - 49,894.18 (557.18)

Regional Learning Workshops          

Leadership Learning Event (e.g. Partnership/VfM/disability inclusiveness ) MEF Workshop KRA 4 200,000.00

  45,276.40 154,723.60

SASA! Training in PNG KRA 4 45,000.00 - 11,500.00 33,500.00

Gender Focal Points Workshop KRA 4 34,466.00 - 6,557.56 27,908.44

UN Women Regional Prevention Workshop (supplemental funding) KRA 4 20,000.00   20,000.00 -

WEE/ PIPSO Women in Business Conference KRA 4 200,000.00

    200,000.00

Program Implementation          

Implementation of counselling review recommendation KRA 1 250,000.00

    250,000.00

Roadmaps Implementation          

Leadership Design KRA 4 50,000.00     50,000.00

Leadership Programming Allocation KRA 4 200,000.00

    200,000.00

WEE Design KRA 5 50,000.00     50,000.00

WEE Programming Allocation KRA 5 200,000.00

    200,000.00

Monitoring and Evaluation          

Monitoring and Evaluation Activities M&E 13,008.00 10,539.64 2,468.00 0.36

Women's Centres MEF Workshop M&E 10,000.00 - 5,105.81 4,894.19

           Regional Support to Cook Islands          

Review of Cook Islands country plan (TA and Support Unit costs) KRA 1 45,000.00   29,192.98 15,807.02

           

58

Description Key Result

Area Total

Budget (Jan 2016 - June

2017) Spent: Jan - June 2016

Spent: July 2016 - June

2017

Balance as of 30 June

2017 (KRA) Regional Support to Nauru          

Nauru Country Plan Monitoring and Review Mission (funded from regional) KRA 1 30,000.00   17,692.22 12,307.78

           Regional Support to RMI          

RMI Country Plan Midterm Review/Reflection   30,000.00     30,000.00

           Regional Support to Kiribati          

Kiribati Family Health Association Strategic Planning (ongoing activity costs) KRA 6 18,667.00 18,500.00 166.65 0.35

Kiribati Women’s Economic Feasibility Study (regional funding) KRA 1 30,000.00     30,000.00

           Regional Support to Palau          

Participation in the FWCC Training in Fiji KRA 1 18,018.00 7,400.38 10,618.01 (0.39)

           Regional Support to Solomon Islands          

Solomon Islands Country Plan Midterm Review/Reflection (funded from the Regional Program)   50,000.00   50,000.00 -

           Regional Support to Vanuatu          

Mid-Term review of Vanuatu country plan, including the development of an M&E Framework (funded regionally) KRA 1 16,935.00 12,055.81 4,879.63 (0.44)

          -

           Communications          

Website revamp, management and maintenance and Interactive Map   34,810.00 718.02 19,451.40 14,640.58

Annual Progress Report production Administration /

43,098.00 - 22,743.84 20,354.16

Travel budget for Communications Unit for story and photo collection Administration /

8,171.00 3,170.85 - 5,000.15

Printing and production costs for communication materials Administration /

8,000.00     8,000.00

Updating and Production of DVDs 10,000.00     10,000.00

59

Description Key Result

Area Total

Budget (Jan 2016 - June

2017) Spent: Jan - June 2016

Spent: July 2016 - June

2017

Balance as of 30 June

2017 (KRA) Administration /

Communications Panel Administration /

15,000.00     15,000.00

           Governance and Management Services          

Advisory Board Meeting (locations Samoa, Canberra and TBC) Administration /

228,312.92 64,052.59 88,987.74 75,272.59

Program Management Committee Administration /

8,631.00 59.59 3,815.96 4,755.45

Stakeholder Meetings Administration /

1,000.00   591.89 408.11

           Adhoc Support/Tasking Notes          

Yoslyn Sigrah participation to 66th CEDAW Conference   -   3,677.53 (3,677.53)

Pacific Women Animated Infographics Video (The Greenhouse Studios)   -   2,809.33 (2,809.33)

Yoshiko Yamaguchi - participation to Pacific Feminist Forum   -   8,294.24 (8,294.24)

Men’s Behaviour Change Conference   -   23,047.10 (23,047.10)

SASA! Pilot and Family Support Centre   -   500,000.00 (500,000.00)

Subtotal Regional   4,347,079.00

753,674.59

1,590,294.27

2,003,110.14

           Fiji        

Advisory Support          

Fiji Gender Adviser – Ministry of Women (ongoing and extension of activity costs) KRA 3 283,943.00 108,294.51 141,039.75 34,608.74

Fiji M&E Adviser KRA 7 103,165.00 - 68,264.90 34,900.10

60

Description Key Result

Area Total

Budget (Jan 2016 - June

2017) Spent: Jan - June 2016

Spent: July 2016 - June

2017

Balance as of 30 June

2017 (KRA) Fiji Women’s Fund          

Design of Fiji Women’s Fund   26,653.00 25,652.09   1,000.91

TA – drafting of Schedule of Requirements KRA 2 4,665.00 2,559.00 2,106.00 -

Recruitment Costs for Fiji Women's fund Staff KRA 2 2,000.00   4,576.94 (2,576.94)

Personnel Costs (Fund Manager, Grant and Contracts Manager, M&E Officer, Admin Officer)   164,000.00

  19,680.42 144,319.58

Office facilities equipment logistics (Suva Office)   55,000.00   9,325.63 45,674.37

Fund Governance Meetings   2,500.00   146.71 2,353.29

Design of Fiji Women’s Fund Program Logo       653.59 (653.59)

First Call of Grants   2,000.00     2,000.00

Grant A   250,000.00

    250,000.00

Grant B   350,000.00

    350,000.00

Program management for Fiji Women in Fisheries Network          

Grant to Women in Fisheries Network KRA 5 55,000.00 30,000.00 85,000.00 (60,000.00)

Program management for Pacific Conference of Churches          

Grant and Program Management Support - Pacific Conference of Churches (Grant) KRA 6 60,913.00 - 60,912.20 0.80

Partner Capacity Development          

Financial Management Training for partners KRA 1 3,500.00   5,529.99 (2,029.99)

Annual Reflection Workshop          

Printing of Fiji MEF   1,014.00 3,500.00 (2,486.23) 0.23

Logistics for first Annual Reflection Workshop KRA 1 27,485.00     27,485.00

Country plan review   47,510.00   2,492.99 45,017.01

Evaluation of Pacific Women Partner Program          

FWCC Monitoring and Evaluation   2,799.00   2,799.00 -

Evaluation of Medical Services Pacific KRA 1 3,000.00     3,000.00

61

Description Key Result

Area Total

Budget (Jan 2016 - June

2017) Spent: Jan - June 2016

Spent: July 2016 - June

2017

Balance as of 30 June

2017 (KRA) Monitoring and Evaluation          

M&E Workshop   111.00     111.00

Subtotal Fiji Bilateral   1,445,258.00

170,005.60

400,041.89

875,210.51

           Kiribati        

Kiribati Women’s Economic Feasibility Study KRA 1 50,000.00     50,000.00

Subtotal Kiribati Bilateral and Regional   50,000.00 - - 50,000.00            Nauru        

Capacity Needs          

Participate in a cross country capacity development opportunity   12,000.00     12,000.00

Subtotal Nauru Bilateral and Regional   12,000.00 - - 12,000.00            Papua New Guinea (PNG)        

Support to Family Support Centres in the Highlands          

Support for equipment procurement (for ICRC) KRA 6 37,273.00   38,714.16 (1,441.16)

Support to Tsak Valley, Enga Province          

Services contract for agricultural and business training KRA 5 16,513.00   12,246.59 4,266.41

Ginigoada expansion          

Grant to Ginigoada Foundation (if agreed) KRA 5        

Family Sexual Violence Action Committee Organisational Strengthening          

Organisational Strengthening Adviser (based on the FSVAC Review) (ongoing costs) KRA 6 218,261.00

83,642.19 195,305.85 (60,687.04)

Grant to FSVAC KRA 6 262,324.00

262,323.82   0.18

TA - Short term case management support to implement DFAT’s FSV policy KRA 6 30,000.00   4,372.17 25,627.83

Monitoring and Evaluation          

PNG M&E Adviser and Workshop Costs (ongoing costs) M&E 480,658.00

109,528.36 255,706.53 115,423.11

62

Description Key Result

Area Total

Budget (Jan 2016 - June

2017) Spent: Jan - June 2016

Spent: July 2016 - June

2017

Balance as of 30 June

2017 (KRA) Support Unit MEL Officer, includes team M&E travel costs M&E 96,462.00 25,531.56 52,743.50 18,186.94

Women in Leadership          

PNG Women in Leadership Design KRA 1 115,175.00

104,167.64 11,007.23 0.13

PNG Women in Leaders Program (if agreed) KRA 4        

Women in Mining          

PNG Women in Mining Design KRA 4 108,800.00

108,800.00   -

PNG Women in Mining Program (if agreed) KRA 4     31,188.11 (31,188.11)

Support to East Sepik          

Funding Micro-Finance training (Yangoru-Saussia) KRA 5 1,407.00 616.39 791.45 (0.84)

Voter Education in Bougainville          

Grant to IWDA KRA 4 346,917.00

203,151.15 143,765.00 0.85

Response to Gender Based and Sorcery Related Violence          

Grant to Oxfam KRA 6 1,181,135.00

- 1,181,135.00 -

Extension of Coffee Industry Support Project          

Grant to CARE Australia KRA 5 1,173,020.00

393,573.00 779,446.95 0.05

Highlands Sexual, Reproductive and Maternal Health Project          

Grant to CARE Australia KRA 7 1,000,000.00

500,000.00 475,000.00 25,000.00

Improving Response to Family and Sexual Violence ‘Safe Village Pilot and Family Support Centre Assistance Project          

Grant to Populations Services International KRA 6 620,051.00

- 120,051.00 500,000.00

Creating an Entrepreneurial Ecosystem for Women in PNG          

Grant to StartUP Cup KRA 5 162,787.00

- - 162,787.00

Grant to Centre for International Private Enterprise KRA 5 222,535.00

- 222,535.00 -

63

Description Key Result

Area Total

Budget (Jan 2016 - June

2017) Spent: Jan - June 2016

Spent: July 2016 - June

2017

Balance as of 30 June

2017 (KRA) PNG Research Training Program          

Agreement with State, Society and Governance in Melanesia Program (if agreed) KRA 1     695,458.00 (695,458.00)

Support to PNG Women’s Forum          

In form to be agreed with US Embassy KRA 3 86,867.00 36,866.89 62,298.76 (12,298.65)

Improving the impact of state and non-state interventions in overcoming sorcery related violence          

Agreement with State, Society and Governance in Melanesia Program KRA 1 242,393.00

  242,393.48 (0.48)

Sorcery Research and Communication Strategy          

Agreement with Queensland University of Technology/University of Goroka KRA 1 190,238.00

  190,237.53 0.47

Bougainville Young Women’s Leadership Project          

Grant with IWDA/BWF KRA 4 685,669.00

110,266.00 575,403.00 -

Informal Learning Pathways         -

Contract with CK Consultancy Ltd KRA 7 4,897.00   4,897.20 (0.20)

Other Adhoc Support/Tasking Notes         -

PNG Country Manager difference   - 18,917.72 44,902.00 (63,819.72)

PNG Women in Leadership Support Program (ANU)   -   1,187,592.01

(1,187,592.01)

SASA! Training   -   18,242.83 (18,242.83)

EMSPNG Ambulance Services   -   10,402.82 (10,402.82)

Second Country Plan MTR   -   68,543.36 (68,543.36)

Grant (Femili PNG) FSV Case Management   -   577,086.58 (577,086.58)

Subtotal PNG Bilateral   7,283,382.00

1,957,384.72

7,201,466.11

(1,875,468.83)

64

Description Key Result

Area Total

Budget (Jan 2016 - June

2017) Spent: Jan - June 2016

Spent: July 2016 - June

2017

Balance as of 30 June

2017 (KRA)            Republic of the Marshall Island        

Advisory Support          

Domestic Violence Prevention Technical Adviser KRA 7 196,442.00

86,986.85 197,410.47 (87,955.32)

Women United Together Marshall Islands (WUTMI)          

Grant to WUTMI KRA 2 258,092.00

58,993.58 199,098.42 -

Organisational Strengthening          

Travel costs for Support Unit KRA 2 20,215.00 5,215.88   14,999.12

Subtotal RMI Bilateral and Regional   474,749.00

151,196.31

396,508.89

(72,956.20)

           Samoa        

Advisory Support          

Samoa Gender Adviser (ongoing costs) KRA 3 168,116.00

68,231.45 84,755.72 15,128.83

Samoa Gender Adviser (CSSP Costs)       29,033.02 (29,033.02)

Technical Gender Adviser Support for DFAT Post (Panel) in relation to Gender Analysis/Gender Stocktaking of Aid Program (in main sectors) KRA 3 34,000.00     34,000.00

Subtotal Samoa Bilateral   202,116.00

68,231.45 113,788.74

20,095.81

           Solomon Islands        

Let’s Make our Families Safe (Safe Families)          

Safe Families M&E Provider (ongoing costs) M&E 262,875.00

149,787.47 116,497.16 (3,409.63)

Review of the SAFENET Program KRA 6 7,966.00 7,966.20   (0.20)

Leadership Design          

Scoping study for women in leadership program KRA 4 100,000.00

100,000.00 (14,633.80) 14,633.80

Capacity Needs          

65

Description Key Result

Area Total

Budget (Jan 2016 - June

2017) Spent: Jan - June 2016

Spent: July 2016 - June

2017

Balance as of 30 June

2017 (KRA) Participate in the Fiji country plan review. KRA 1 6,000.00     6,000.00

Country Plan Review/Reflection          

Solomon Islands Country Plan Midterm Review/Reflection (Bilateral Funding)   -   14,633.80 (14,633.80)

Subtotal Solomon Islands Bilateral and Regional   376,841.00

257,753.67

116,497.16

2,590.17

           Tonga        

Program on Violence Against Women - WCCC          

Grant to Women and Children’s Crisis Centre, Tonga KRA 6 344,537.00

- 344,357.00 180.00

Advisory Support          

Tonga Gender Equality Adviser (ongoing costs) KRA 3 133,766.00

45,937.09 52,255.56 35,573.35

Monitoring and Evaluation          

Support development of the Tonga Pacific Women M&E Framework M&E 16,935.00 2,905.32   14,029.68

M&E Short-term Adviser M&E 30,000.00     30,000.00

Support for Tonga National Centre for Women and Children          

Organizational Strengthening/Financial Management Training (SU Travel and Training Costs) KRA 2 10,000.00   1,990.79 8,009.21

Design for WEE Program for TNCWC KRA 1 65,000.00   28,790.28 36,209.72

Grant to TNCWC KRA 6 50,000.00     50,000.00

Capacity Building Support and Program Monitoring          

Support Unit Travel - Project Monitoring ( TNCWC) KRA 1 5,000.00     5,000.00

Support Unit Travel - Project Monitoring ( WCCC) KRA 1 10,000.00     10,000.00

Subtotal Tonga Bilateral   665,238.00

48,842.41 427,393.63

189,001.96

           Tuvalu        

Advisory Support          

Gender Equity and Social Inclusion Adviser (TA and Activity Costs) KRA 3 215,692.00

35,743.19 111,382.34 68,566.47

66

Description Key Result

Area Total

Budget (Jan 2016 - June

2017) Spent: Jan - June 2016

Spent: July 2016 - June

2017

Balance as of 30 June

2017 (KRA)

Psycho-social Adviser KRA 7 110,100.00

    110,100.00

National Disability Survey KRA 1 110,706.00

  72,558.28 38,147.72

Country Plan Consolidation Monitoring Trip (TA) KRA 1 5,000.00     5,000.00

Subtotal Tuvalu Bilateral   441,498.00

35,743.19 183,940.62

221,814.19

           Vanuatu        

Girl, Justice and Custom Research          

TA and Travel Costs to carry out Girls, Justice and Custom Research KRA 6 52,846.00     52,846.00

Vanuatu Roadmap Development          

TA for Vanuatu Roadmap development process KRA 1 50,000.00     50,000.00

Monitoring and Evaluation (bilateral funding)          

M&E Capacity Development Support (SU) M&E 10,000.00     10,000.00

Subtotal Vanuatu Bilateral and Regional   112,846.00

- - 112,846.00

           

Total Activity Budget (January 2016–June 2017) 15,411,007.00

3,442,831.94

10,429,931.31

1,538,243.75

67

Annex 5: Comparison of Different DFAT Funded Facilities Investing in Women

Initiative27MAMPU28 Trade and Inclusive Economic

Growth Facility29MDF30 Transport Sector Support

Programme31

Time Frame 4 + 4 years 4 + 4 years 4 + 4 years 5 + 5 years 5 + 5 yearsPurpose of facility Implement the program Facilitate and support

(but not direct) the implementation of the program

Reduce the administrative burden for national partners and DFAT

Provide quick and flexible services to access a range of international and national skills

MC seen as a neutral party

‘Trade and Inclusive Economic Growth Facility (the Facility) will be a centrepiece of Australia’s aid program to the Philippines ’(p.8).

Facility will effectively implement the program – ‘DFAT believes that a facility represents the best means of organising Australia’s trade and inclusive economic growth activities for the Philippines ’(p.9)

Consolidate a variety of agreements and activities into an efficient management structure, reducing DFAT’s administrative and process burden, and maximising DFAT staff time for policy engagement. But, of course, it does require ongoing engagement and programming, in depth.

Enables DFAT to put in place an overarching managerial framework, and develop and maintain policy coherence in which Australia can direct and adapt our economic partnership with the Philippines.

Implement the programme

Facility allows ‘’ (p.29) provides the flexibility and responsiveness necessary to mobilise and recruit a highly skilled team and implement a market systems program of the complexity and scale of MDF

Facilitate knowledge transfer between DFAT managers and contribute to DFAT’s institutional understanding of market systems

Responsive facility that respond quickly to the dynamic nature of economies

Multi-country approach chosen for reasons including (p.17/18): ‘the savings in transaction costs and senior staffing that can be realised from a multi-country facility…and the ability to more easily protect the market

Implement the program

68

Investing in Women Initiative MAMPU Trade and Inclusive Economic Growth Facility

MDF Transport Sector Support Programme

development philosophy and approach in each country whilst still enabling positive experiences to inform other aspects of Australian aid programs

Country specific programs would be seen to be more demanding on DFAT time

69

Investing in Women Initiative MAMPU Trade and Inclusive Economic Growth Facility

MDF Transport Sector Support Programme

Scope Establish business coalitions

Manage DFAT’s funding agreements

Provide access to technical skills

Operate the advocacy platform (including monitoring, evaluation and learning)

Test whether a separate private sector or not for profit entity could manage certain key functions

Initially grants to be signed with DFAT, thereafter in future rounds grants will be between national partners and the MC

MC to support implementation assisted by appropriate service providers

MC to develop financial management system, including procurement guidelines (consistent with the Commonwealth Procurement Guidelines

MC to design learning strategy

MC to manage a small Innovation Response Fund32

Implement activities through three different modalities (directly implemented by managing contractor, oversight over activities implemented by a 3rd party, advisory services provided to 3rd party who has a direct funding arrangement with DFAT)33

Identify and design activities for the facility to invest in (including annual pipeline planning)

Report against a high-level M&E Framework

Promote innovation by ‘trial, experience, and adjustment’ (p.9)

Undertake market analysis, design interventions, negotiate partnership agreements and determine the investment applicable to each intervention

responsible for all financial, procurement and administrative requirements of the Facility as well as the delivery of training, results measurement, communication and knowledge management

Administrative and management support services

Responsibility for Capacity building

ISP to deliver technical assistance, establish and manage IRG34, manage the TAC35 and provide secretariat support services

20% cap on expenditure on adviser and capacity building support

Location Full time locally engaged program coordinator at each post where there are significant activities (Indonesia, Philippines, and Vietnam)

Location proposed by Managing Contractor

Jakarta The Philippines MDF Country Representatives in each country lead the CT and are responsible for Post engagement and support as well as MDF technical quality in each country

PNG

Skills Set Strategic, technical, operational, and advocacy support to both the

AusAID/DFAT recruited the key personnel (not MC)

The managing contractor will engage an established and well-recognised leader with a

Core leadership team (Tl, DTL, Results Measurement

Program Director to provide strategic oversight, lead

32 5% of the activity budget allocated to this fund, ‘reserved for additional innovative and experimental activities, beyond the work implemented by the national partners’ (p.18).

33 The design document suggests a ratio of 60% of activities directly designed and implemented by the Contractor, 30% for 3rd party design and implemented (contract with managing contractor) and 10% of activities in which contract is DFAT managed.34 IRG – Independent Review Group to advise on strategic and program performance, strategic advice, review government planning documents.35 TAC – Technical Audit Consultancy – to sample, test and report on larger/complex maintenance activities, conduct technical audits.

70

Investing in Women Initiative MAMPU Trade and Inclusive Economic Growth Facility

MDF Transport Sector Support Programme

Australian Government and Investing in Women partners including the national business coalitions

‘influential and effective leader as CEO’ (p.36)

Operations manager responsible for the 4 different components (i.e. Business coalitions, impact investing in Women’s SMEs, Government partnerships, advocacy platform) and the MEL unit36

Gender specialist and Gender advocacy specialist

AusAID/DFAT provided start up grants to first cohort

Small core team – TL, Financial Manager, Capacity Building Coordinator, and M&E Specialist (also responsible for communication and learning)

strong reputation on trade and/or inclusive economic growth as the Facility Director. The Facility Director will be supported by an Operations Manager and three Team Leaders.

Managers, Communications Manager, Operations Manager, Country Representatives (leaders of the Country Teams)

Skills required include market analysis, brokering partnerships between market actors, assisting companies to formulate new business models, and supporting and mentoring companies taking risks to achieve market innovations

scoping of activities, oversight over engagement with PNG government, give directions to ISP on implementation, provide advice to DFAT on implementation and performance of TSSP, the ISP, advisors and other program inputs

PD provides advice to DFAT on transport infrastructure matters, access and coordinate program data and information to prepare sector briefs/analysis, participate in DFAT’s engagement with PNG Government, participates in wider dialogue on infrastructure development, provides regular advice on performance of TSSP

PD does not performance manage the ISP

PD does not get involved directly in managing the administration of activities/ operations

Core team – TL, Managers for different

36 Relevant senior positions are at either C4 level or D4 level.71

Investing in Women Initiative MAMPU Trade and Inclusive Economic Growth Facility

MDF Transport Sector Support Programme

components (e.g. Safety and security, Land Transport), M&E manager, Communications Manager, Program Finance Manager

Governance Managed by Multilateral Policy Division in Canberra

Full time DFAT Program manager

Board of Governance Clear set of roles and

responsibilities for DFAT Canberra and Posts (pp.28-30)

Host a 6-monthly roundtable with each Post

Annual review and planning workshop in Canberra

Program embedded within the Indonesian Vice- President’s Commission for the Accelerated Reduction of Poverty

Strategic Committee to provide ongoing advice to the committee

Partners Forum for purposes of program management and knowledge sharing

DFAT’s Economic Section in Manilla has direct oversight

Steering Committee Additional Technical Working

groups when necessary Annual High Level Forum (to

promote engagement with key stakeholders)

the key decision-making and coordination body for the second phase of MDF will be the Program Managers Forum (PMF)

Joint responsibility between Private Sector, Infrastructure and Agriculture Development Branch (PXB) in Canberra and five country Posts.

Advisory Group provides technical guidance, help DFAT manage the facility, and review implementation

Transport Sector Coordinating, Monitoring and Implementation Committee – primary governance and decision making body (including reviewing plans, receives reports, monitors progress, prepare budget)

72

73

Annex 6: Value for Money Analytical Matrix

4 E’s VfM Principles and Standards Satisfactory Partially Satisfactory

Unsatisfactory

Economy Efficiency in managing costs

Cost consciousness The extent to which the Support Unit demonstrates it is cost conscious in its decision making and implementation

More than 75% of decision making and implementation by the Support Unit are guided by cost consciousness principles

Between 50 – 75% of decision making and implementation by the Support Unit are guided by cost consciousness principles

Less than 50% of decision making and implementation by the Support Unit are guided by cost consciousness principles

Encouraging Competition

The extent to which competition is practiced in procurement

Competition is practiced in more than 75% of procurement by the Support Unit

Competition is practiced in between 75% and 50% of procurement by the Support Unit

Competition is practiced in less than 50% of procurement by the Support Unit

Resource allocation The extent to which the allocation of resources ensures quality outputs

In more than 75% of instances the Support Unit allocates appropriate resources to ensure quality outputs

In 50 to 75% of instances the Support Unit allocates appropriate resources to ensure quality outputs

In less than 50% of instances the Support Unit allocates appropriate resources to ensure quality outputs

Efficiency Efficiency in how well inputs are converted into outputs

Timeliness The degree to which deliverables were achieved in time

75% of activities are delivered on time

50 – 75% activities are delivered on time

Less than 50% of activities are delivered on time

Within budget The extent to which deliverables were delivered within budget

75% of activities are delivered

50 – 75% activities are

Less than 50% of activities are

74

4 E’s VfM Principles and Standards Satisfactory Partially Satisfactory

Unsatisfactory

within budget delivered within budget

delivered within budget

Efficient business processes

The extent to which the Support Unit’s business process avoid wasteful expenditure

Business processes are assessed to be satisfactory

Business processes are assessed to be partially satisfactory

Business processes are assessed to be unsatisfactory

Acceptable transaction costs

The extent to which the transaction costs of the Support Unit compare favourably to other programs

In comparison with other programs the transaction costs of the Support Unit were found to be satisfactory

In comparison with other programs the transaction costs of the Support Unit were found to be partially satisfactory

In comparison with other programs the transaction costs of the Support Unit were found to be unsatisfactory

Effectiveness Effectiveness in how well outputs are converted to outcomes

Results focussed The extent to which the Support Unit is results focussed

More than 75% of planned activities and additional requests have been delivered in accordance with the workplan, and will meet expected outcomes

50 - 75% of planned activities and additional requests have been delivered in accordance with the workplan, and will meet expected outcomes

Less than 50% of planned activities and additional requests have been delivered in accordance with the workplan, and will meet expected outcomes

Experimentation and innovation

The extent to which the Support Unit is nurturing experimental and innovative approaches to design and delivery of initiatives

Experimentation and innovation by the Support Unit is judged to be satisfactory

Experimentation and innovation by the Support Unit is judged to be partially satisfactory

Experimentation and innovation by the Support Unit is judged to be unsatisfactory

Additional Resources The extent to which additional DFAT resources are being utilised to ensure effective operation of the Support Unit

The use of The use of The use of

75

4 E’s VfM Principles and Standards Satisfactory Partially Satisfactory

Unsatisfactory

additional resources by DFAT to oversee the Support Unit is judged to be satisfactory

additional resources by DFAT to oversee the Support Unit is judged to be partially satisfactory

additional resources by DFAT to oversee the Support Unit is judged to be unsatisfactory

Performance and Risk management

The extent to which the Support Unit implements a robust approach to performance and risk management

Performance and risk management is judged to be satisfactory

Performance and risk management is judged to be partially satisfactory

Performance and risk management is judged to be unsatisfactory

Ethics Ethical practices

Transparency and accountability

The extent to which the Support Unit has adopted acceptable accountable and transparent approaches to all its actions

The level of transparency and accountability is appraised to be satisfactory

The level of transparency and accountability is appraised to be partially satisfactory

The level of transparency and accountability is appraised to be unsatisfactory

Equity The extent to which the Support Unit is promoting indigenous procurement

The level of indigenous procurement is considered to be satisfactory

The level of indigenous procurement is considered to be partially satisfactory

The level of indigenous procurement is considered to be unsatisfactory

76

Annex 7: Terms of Reference

TERMS OF REFERENCE: EVALUATION OF THE PACIFIC WOMEN SHAPING PACIFIC DEVELOPMENT SUPPORT UNIT

1. Purpose

The purpose of this evaluation is to undertake an independent assessment of the Pacific Women Support Unit, and its contributions to the broader objectives and outcomes of Pacific Women Shaping Pacific Development (Pacific Women). The findings and recommendations will also inform DFAT’s considerations on whether to extend the term of the Support Unit contact for up to four years, in its current or a revised form. It will complement the three year evaluation of Pacific Women which was undertaken from November 2016 to March 2017.2. Background

2.1 Pacific Women Shaping Pacific Development

Pacific Women Shaping Pacific Development (Pacific Women) is a 10-year program focused on enabling women and men across 14 countries in the Pacific to improve the political, social and economic opportunities for women. Pacific Women provides support to Pacific countries to meet the commitments by Pacific leaders to work for gender equality. The program commenced in August 2012.The goal of Pacific Women is: Women in the Pacific (regardless of income, location, disability, age or ethnic group) participate fully, freely and safely in political, economic and social life.The outcomes sought by Pacific Women are:

Women, and women’s interests, are increasingly and effectively represented and visible through leadership at all levels of decision-making.

Women have expanded economic opportunities to earn income and accumulate economic assets.

Violence against women is reduced and survivors of violence have access to support services and to justice.

Women in the Pacific will have a stronger sense of their own agency, supported by a changing legal and social environment and through increased access to the services they need.

Activities funded under Pacific Women are identified primarily through country plans, which are agreed between the Government of Australia’s Department of Foreign Affairs and Trade (DFAT) and counterpart governments. At regional level, a small set of regional and multi-country activities are implemented to address common issues and build on country specific activities. Management of Pacific Women is primarily through DFAT bilateral posts at Head of Mission level working with the Program Director, based in Canberra assisted by the Regional Program Manager based in Suva. Gender Focal Points at posts are responsible for day to day program management of their respective Pacific Women country plans. Pacific Women objectives are included in the respective DFAT bilateral and regional aid investment plans/agreements.

2.2 Pacific Women Support Unit

77

The Pacific Women Support Unit (the Support Unit) supports DFAT to manage and implement Pacific Women. The need for a Support Unit was identified in the Pacific Women design document. The Pacific Women model requires that DFAT fund and engage with multiple organisations, including coalitions for change, at a strategic level as well as through funding relationships. Program management can be time intensive and many Pacific organisations have a range of capacity gaps that are important to address. Pacific Women also requires increased policy dialogue and influencing work by DFAT staff and increased attention to gender mainstreaming in sector programs. The program therefore needs to support, influence and drive changes in DFAT capacity and resources to meet these challenges.

The rationale for the Support Unit is to support program management and the many facets of program level and activity level implementation. The design envisaged that the Support Unit would provide technical, administrative and logistical support to DFAT bilateral and regional programs to manage country and regional plans for Pacific Women.

This support may include:- sub-contracting and managing individual activities identified in country and regional plans as agreed

with DFAT;- technical services that could include:

o contributing to policy analysis and strategic guidance; o providing technical advice to DFAT, implementing partners government agencies, NGOs and

civil society organisations on M&E systems, plans and reporting; o developing and implementing a communications strategy to guide reporting and

communications activities; and o alongside DFAT staff, engaging with Pacific women and men and women’s organisations and

coalitions to maintain and strengthen Pacific Islander ownership of the program and facilitate inclusive, durable developmental change.

Through a design and implement tendering process, Cardno Emerging Markets (Australia) Pty Ltd was contracted to design the operations and management systems of the Support Unit; and provide the technical, administrative and logistical services; outlined above. The current management of the Support Unit is contracted from 29 April 2014 to 30 June 2018 with a possible four year extension. The value of the contract is AUD40,582,884 (Year 1 to 3) and AUD56,945,758 (Year 4 to 7) including activity costs to a maximum of AUD72,500,000.

The Support Unit was designed to provide responsive and flexible services to Pacific Women implementation and management. The Support Unit has an office in Suva, Fiji and a sub-office in Port Moresby, Papua New Guinea.

The Support Unit Fiji office is made up of five units, including:

The Management Unit – consisting of the Team Leader and Deputy Team Leader. This unit works closely with the DFAT Pacific Women management team around issues of strategic direction for Pacific Women.

The Operations Unit – under the management of the Deputy Team Leader, this unit provides logistical, administrative and financial support to Pacific Women. This includes secretariat functions, grant management and day to day financial management of the Support Unit.

The Programs Unit – under the management of the Senior Programs Manager, this unit supports the implementation of regional and country program activities. The programs unit holds discrete program management responsibilities for some Pacific Women activities, and provides support to DFAT Gender Focal Points to facilitate Pacific Women activity implementation.

The Gender Unit – under guidance from the Gender Specialist, this unit provides high-level technical expertise to DFAT and implementing partners in relation to gender equality and the intended outcomes of Pacific Women.

78

The Evaluation and Learning Unit – under direction from the Team Leader provides technical advice on monitoring and evaluation, learning, communications and research.

79

The Support Unit PNG sub-officeDue to the size of the PNG program, there is a dedicated PNG sub-office that falls under the supervision of the Support Unit Team Leader. The PNG sub-office is overseen by a Country Manager, working with a Program Manager, Monitoring, Evaluation and Learning Officer, Finance Officer and Administration Officer and Administration Assistant who all work closely with the PNG DFAT Pacific Women team to provide advice, support and assistance in the implementation of the PNG Pacific Women country plan. In the early stages of mobilisation, the Support Unit submitted to DFAT a number of criteria for use in developing a rubric to assess performance against the Support Unit’s core functions. The rubric template and assessment criteria have been continually reviewed and refined and will provide the evaluation team with clear guidance on criteria for assessing quality and value. The latest Support Unit Performance Assessment Rubric (Attachment A) has 14 assessment criteria and employs a three-point rating scale. The assessment criteria have been developed with reference to the Statement of Requirements for the Pacific Women Support Unit Head Contract and the DFAT Partner Performance Assessment template. The evaluation is an opportunity to test and review the draft rubric and, if required, make recommendations to enable its finalisation.

3. Objective of the evaluation

An independent evaluation of the Support Unit is proposed to be undertaken between July to September 2017. The objective of the evaluation is twofold:

i) to assess the effectiveness of the Support Unit in delivering services against the DFAT Head Contract and Schedule of Requirements, in order to:

a. identify strengths and achievements of Support Unit implementation;b. identify any barriers to the Support Unit’s effective operations and whether these are

ongoing or have been resolved, and the enablers for effective operations and whether these have been fully utilised; and

c. recommend adjustments and enhancements to ensure that the Support Unit will be able to function most effectively to provide quality technical, administrative and logistical support to DFAT and implementing partners

d. support DFAT to determine whether to take the option to extend the term of the DFAT Head Contract for a period of up to 4 years.

ii) to evaluate the Support Unit model, in order to:

a. assess if the current Support Unit model is the best way to support Pacific Women’s implementation;

b. identify lessons learnt from the establishment and operation of the Support Unit that DFAT could use to design and manage similar program models; and

c. provide recommendations to inform Support Unit implementation post June 2018.

The findings and recommendations from this evaluation will inform DFAT’s consideration on whether to extend the term of the Support Unit contact for up to four years, in its current, or a revised form.

The evaluation team should build on the findings and recommendations of the Pacific Women Year 3 Evaluation. All evaluation deliverables will be provided to the evaluation team.

4. Statement of Requirement

The evaluation will focus on four evaluation domains: effectiveness, efficiency, relevance, and monitoring and evaluation. The evaluation team will finalise a list of evaluation questions as part of the Evaluation Plan, guided by the primary evaluation questions proposed below. These may be supplemented with sub-questions during the desk -based review and analysis or the in-country visits. The evaluation team should

80

also consider the Support Unit Performance Assessment Rubric at Attachment A to inform development of evaluation questions and make recommendations in relation to the approach to enable its finalisation.

81

Assess the effectiveness of the Support Unit.

a) To what extent has the Support Unit provided high quality and timely operational services as outlined in the Head Contract and in Annual Work Plans, including:

a. Policy and Technical Adviceb. Activity Management, Monitoring and Reportingc. Meeting Facilitation and Logisticsd. Program Management, Monitoring, Reporting and Evaluatione. Communication

b) To what extent has the Support Unit been able to be identify, support and test innovation?c) To what extent has the Support Unit established mechanisms that will assist to build Pacific

ownership, capacity and sustainability for Pacific Women? d) Have Support Unit activities or operations resulted in any unintended positive or negative

outcomes?

Assess the efficiency of the Support Unit.

a) To what extent has the Support Unit provided high quality and timely management services as outlined in the Head Contract?

b) Has the Support Unit been appropriately structured and resourced to meet the needs of DFAT and Pacific Women’s implementing partners? Are the team work loads, planning processes and internal management supporting the effective operations of the Support Unit, including the relationship between the Fiji and PNG offices? This should include consideration of the number and skills of Support Unit staff in Fiji, PNG and in Head Office, the level and demand for Support Unit services, client satisfaction with Support Unit services.

c) To what extent has there been effective management arrangements between DFAT and the Support Unit, including through the annual work planning process and tasking notes? What are the strengths and weaknesses of the mechanisms established to support communication, management and decision making between DFAT and the Support Unit?

Assess ongoing relevance of the Support Unit.

a) Are the assumptions underpinning the design of the Support Unit still valid? b) To what extent do DFAT staff and other stakeholders understand of the model and role of the

Support Unit and how has this understanding impacted on its operations;c) How easy is it for DFAT and for implementing partners to access the services of the Support Unit?

Are there any other barriers for DFAT staff to effectively engage with the SU at an operational and strategic level?

d) Are there any remaining unmet needs that the Support Unit (or alternative) could reasonably meet?

e) Is the current Support Unit modality the most appropriate? Are there (or could there be) any viable alternative models to replace some or all of the services provided by the Support Unit in future?

Assess the Monitoring and Evaluation Systems of the Support Unit

a) To what extent has the Support Unit developed quality M&E, Communications, Research and Learning strategies for Pacific Women and are they using the information generated for continuous program improvement?

Assess the Value for Money of the Support Unit

a) Does the model demonstrate good value for money (were deliverables achieved on time and within budget and meet DFAT quality standards)? This should compare the actual costs of the delivery of this model with other comparable programs.

82

5. Duration and Phasing

The expected duration for the evaluation is from 17 July to 18 September 2017 with up to 65 working days in total, including: travel to Canberra, Melbourne, Suva, Fiji and Port Moresby, Papua New Guinea; desk review, preparation of the evaluation plan and methodology, consultation, data processing as well as preparation of reports. An indicative task list and indicative input days are outlined below.

In undertaking the evaluation, the team will:

a) complete a desk review to analyse program documents and review the Pacific Women website. The desk review may suggest a number of initial findings that in turn may point to additional or revised evaluation questions.

b) develop an Evaluation plan that responds to the TOR and information identified in the desk review. This must include an approach to value for money assessment. The plan will be finalized in consultation with DFAT and the Support Unit. The plan must include a process to consult with stakeholders inside and outside of DFAT, including covering all DFAT Posts and a representative sample of implementing partners who are not based in Fiji. An indicative list of persons to interview will be prepared by DFAT and the Support Unit.

c) Organise and pay for all logistic arrangements, including flights and accommodation.

Task Indicative days – Team Leader

Indicative days – M&E Practitioner

1 Participate in an initial briefing with DFAT (Canberra, Suva and Port Moresby) and Support Unit Management team staff (Suva and Port Moresby)

1 day 1

2 Undertake a desk review to analyse program documents. A list of documentation to be reviewed will be prepared by DFAT and the Support Unit

7 days 5

3 Develop an Evaluation Plan that responds to the TOR and includes methodology and data collection methods. The final Evaluation Plan will be finalised in consultation with DFAT and the Support Unit.

3 days 2

4 Data collection (Canberra, Melbourne, Fiji and PNG). 17 days 105 Undertake data analysis, synthesis and write Draft Evaluation

Report of no more than 30 pages, excluding annexes that meets DFAT M&E Standards (Standard 6)

10 days 5

6 Final Evaluation Report (based on comments from DFAT and Support Unit).

3 day

Total 42 days 23

6. Reporting Requirements

a) Evaluation Plan - The plan will outline the scope and methodology of the evaluation. The plan will meet DFAT M&E Standards (Standard 5) and include:

i. the methodology to be used for the evaluation, including the process for information collection and analysis, including tools such as questionnaires and/or questions to be asked during focus group discussions;

83

ii. identification of any challenges anticipated in achieving the evaluation objectives; allocation of tasks within the team;

iii. key timings and a consultation schedule identifying key stakeholders to be consulted and the focus of the consultations;

iv. roles and responsibilities of each team member andv. a draft schedule of field visits.

It is expected that the Evaluation Plan will be submitted to DFAT one week after signing the contract and at least two weeks before the in-country mission for DFAT and Support Unit feedback.

b) Draft evaluation report of no more than 30 pages, excluding annexes that meets DFAT M&E Standards (Standard 6)

c) A debrief with DFAT and the Support Unit, within 5 working days of DFAT receiving the draft report. This could include a powerpoint presentation with key findings from the evaluation.

d) A final evaluation report, within 5 working days of receiving DFAT and Support Unit comments

7. Team Composition

The evaluation team will comprise two members, an international monitoring and evaluation specialist with particular expertise in evaluating organisational systems and operations as the Team Leader (C4) and a Pacific M&E practitioner (C2-3).

The Team Leader should have the following skills and experience:

Relevant post-graduate tertiary qualifications, particularly in evaluation, social sciences international development, or a related discipline;

Demonstrated understanding of DFAT managing contractor models;

At least 15 years’ experience of successful consultancies in conducting evaluations of large, complex programs, and developing useful recommendations that meet DFAT requirements in alignment with the DFAT M&E Standards;

In depth understanding of a wide range of evaluation methodologies and approaches, including value for money assessment;

Strong written and analytical skills; and

Experiencing of working in the Pacific, regional programming experience highly valued.

The Pacific M&E Practitioner should have the following skills and experience: Relevant tertiary qualifications, particularly in evaluation, social sciences international

development, or a related discipline;

At least 5 years’ experience of successful consultancies in conducting evaluations of development programs;

Understanding of a wide range of evaluation methodologies and approaches;

Extensive experience in the Pacific; and

Good written and analytical skills.

The Team Leader will be responsible for sub-contracting the Pacific M&E Practitioner. The Team Leader will have primary responsibility for delivering all outputs to DFAT and for organising logistics. The Team Leader and Pacific M&E Practitioner should discuss whether the evaluation process provides a mentoring and capacity building opportunity for the Practitioner and how this will be undertaken. This should be included in the Evaluation Plan.

84

8. Background Documents

Pacific Women Shaping Pacific Development Design Document and Delivery StrategyPacific Women Shaping Pacific Development Monitoring and Evaluation FrameworkSupport Unit Design and RedesignProgram Operation Manual, including Risk Management StrategySupport Unit Performance Assessment rubricSupport Unit Annual WorkplansSix Monthly Operational and Six Monthly Program Progress ReportsGrants ManualWebsite and newslettersKey Outputs: Capacity Development Strategy; Communications Strategy; Legal Barriers Paper Partner Performance AssessmentsThree Year Evaluation reportDFAT M&E Standards: http://dfat.gov.au/about-us/publications/Pages/dfat-monitoring-and-evaluation-standards.aspx

85