EDF
Transcript of EDF
Review of The Enterprise Development Fund
Research, Evaluation and Monitoring Team
Industry and Regional Development Branch
MINISTRY OF ECONOMIC DEVELOPMENT
September 2005
Ministry of Economic Development Industry and Regional Development Research, Evaluation and Monitoring Level 10, 33 Bowen Street P O Box 1473 Wellington 6140 New Zealand
Contents
Acknowledgments.........................................................................................................................4
Executive Summary.......................................................................................................................5
PART ONE: Evaluation Goals and Methodology. ..................................................................12
9. Goals of the evaluation...................................................................................................12
10. The intervention logic.....................................................................................................12
11. Methodology....................................................................................................................12
12. Methodology of the survey............................................................................................13
PART TWO: Policy Rationale for the Enterprise Development Fund ................................15
1. MED Policy ..........................................................................................................................15
13. Policy objectives for the Foundation Services.............................................................15
14. Policy objectives for the Enterprise Development Fund...........................................15
2. NZTE policy.........................................................................................................................17
15. Design of the Enterprise Development Fund .............................................................17
16. Criteria for projects .........................................................................................................17
17. Scoring of applications ...................................................................................................17
18. Future changes: devolvement and criteria for 05/06..................................................18
PART THREE: Findings and Recommendations on the Implementation of Policy.........20
19. Programme implementation .........................................................................................20
20. Response by MED to the Enterprise Networks implementation.............................20
21. Operational implementation: the Business Evaluation Team ..................................21
22. Operational implementation: the Enterprise Development Team ..........................21
23. Design of the Enterprise Development Grants scheme.............................................22
24. Project activities available under EDG.........................................................................26
PART FOUR: Findings and Recommendations on Delivery of EDG .................................28
3. Uptake of the fund ..............................................................................................................28
25. Total allocations...............................................................................................................28
26. Total number of applications ........................................................................................28
27. Total number of firms.....................................................................................................29
28. Approval rate...................................................................................................................29
29. Calls to the Business Evaluation Team ........................................................................29
30. Change in application numbers over time ..................................................................30
2
4. Assessment of applications................................................................................................32
31. Change in approval rate over time...............................................................................32
32. Recommendations...........................................................................................................35
33. Time taken to approve grants .......................................................................................36
34. Average time taken to approve grants over time.......................................................36
35. Recommendation ............................................................................................................37
5. Distribution of the EDG .....................................................................................................38
36. Summary of findings on distribution of the EDG......................................................38
37. Regional spread of firms assisted .................................................................................38
38. Regional spread of allocations ......................................................................................39
39. Sectoral spread of allocations ........................................................................................40
6. Grant allocation and collection by firms..........................................................................42
40. Amounts allocated ..........................................................................................................42
41. Amount claimed by closed projects .............................................................................42
7. Analysis of applicant firms................................................................................................43
42. Summary of findings given below. ..............................................................................43
43. Overall size: turnover and FTE .....................................................................................43
44. Exporting..........................................................................................................................44
45. Age of Firms.....................................................................................................................45
46. Firm Growth ....................................................................................................................45
8. Types of assistance funded................................................................................................47
47. Advice and expertise categories .................................................................................................... 47
PART FIVE: Findings and Recommendations on Outcomes ................................................49
9. The framework for analysis of outcomes. .......................................................................49
48. Goals of the EDG.............................................................................................................49
49. Goals of the evaluation...................................................................................................49
50. Specific outcomes sought by the evaluation ...............................................................50
51. Interpretation of the policy intent: improving management or business capability?..................................................................................................................................50
52. Interpretation of the policy intent: improving management ability or knowledge? ...............................................................................................................................51
10. Quantitative findings on accepted firms. ........................................................................53
53. Guide to the findings......................................................................................................53
54. Numbers of firms responding.......................................................................................54
3
55. Numbers of firms completing projects. .......................................................................54
56. Overall satisfaction .........................................................................................................54
57. Proportions of EDG clients seeing changes in the way they manage their business......................................................................................................................................55
58. Proportions of EDG clients undertaking projects aimed at management level areas or business level areas ...................................................................................................56
59. Proportions of EDG clients seeing gains in firm activity ..........................................57
60. Proportions of EDG clients seeing changes in firm performance ............................58
61. Proportions of EDG clients gaining facts or advice ...................................................59
62. Do facts or advice projects impact on management practice?..................................60
63. Are facts or advice projects also management or business level projects?.............61
64. Do management level or business level projects impact on management practice? ............................................................................................................................................61
65. Do changes in management practice impact on firm performance?.......................62
66. Do advice or facts projects impact on firm performance? ........................................63
67. Do management or business level projects impact on firm performance?.............64
11. Quantitative findings: interpretation and conclusion ...................................................65
68. Overall ..............................................................................................................................65
69. Activity types...................................................................................................................66
70. Interpreting the results: improving management capability....................................66
71. Resolving the ambiguity between facts and advice...................................................67
72. Declined firms..................................................................................................................68
73. Recommendation ............................................................................................................68
12. Qualitative findings ............................................................................................................69
74. Guide to the findings......................................................................................................69
75. Overall ..............................................................................................................................69
76. Business/strategic plan development and feasibility studies. ..................................69
77. Intellectual property protection. ...................................................................................70
78. Strategic Design...............................................................................................................70
79. Market research and marketing plans .........................................................................71
80. Prototype ..........................................................................................................................71
81. Mentoring and training..................................................................................................72
82. Certification and systems evaluation ...........................................................................72
Appendix: Survey Questions .....................................................................................................73
4
Acknowledgments
The evaluation would like to thank Dr Richard Arnold, Senior Lecturer in Statistics with the School of Mathematics, Statistics and Computer Science of Victoria University of Wellington, for his valuable and essential advice on the evaluation and data analysis, and for reviewing many drafts of the report.
The evaluation would also like to acknowledge the openness with which New Zealand Trade and Enterprise worked with this evaluation, and the over 100 firms who gave their time to talk with the evaluation.
5
Executive Summary
1. Purpose and evaluation methodology
1. In accordance with the New Zealand Trade and Enterprise (NZTE) Foundation papers (EDC (03) 54 refers) the Ministry of Economic Development (MED) has conducted an evaluation of the Enterprise Development Fund (EDF).
2. The focus of the evaluation was to examine:
• Programme implementation
• Programme delivery
• Programme outcomes
And conclude whether the programme should continue, and continue unchanged or with changes.
3. NZTE split the EDF into ‘Enterprise Development Grants’ and ‘Enterprise Networks’. Enterprise Networks has now been removed from the EDF. This evaluation has examined implementation for the fund as a whole, but examined delivery and outcomes for the Enterprise Development Grants only.
4. The evaluation analysed NZTE data and interviewed NZTE staff responsible for assessing applicants to the EDG. In order to answer questions about outcomes the evaluation undertook a telephone survey of a representative sample of EDG recipients. The sample was a stratified simple random sample of 132 firms with a response rate of 98%.
2. Summary of the Enterprise Development Fund
5. The EDF was established in 2003 (EDC (03) 54 refers) as part of NZTE foundation services. Three programmes were amalgamated to form the EDF: the Enterprise Awards Scheme, the Export Network Programme and World Class New Zealanders. These three all delivered funding assistance to firms.
6. The EDF was appropriated $8.613M annually (for 03/04 and 04/05).
7. The EDF provides funding of 50% of total costs for businesses and entrepreneurs to:
− Engage the services of a business mentor for a finite period of time
− Undertake more advanced management and technology based training (as delivered through existing providers such as NZIM, Polytechnics etc)
− Employ specific external advice and expertise in a management area (such as feasibility studies, e‐business, market research, enhancement and uptake of new technologies, human resources, intellectual property, strategic planning, environmental management, production management)
6
− Undertake international market development activities, including new market investigation, trade fair participation, trade/business missions, business exchanges, and visiting buyers.
3. Implementation of EDF
8. MED considered the implementation of EN by NZTE sufficiently different from its policy objectives to remove Enterprise Networks from the EDF. Funding for Enterprise Networks was moved to operational expenditure (CAB Min (04) 38/4 refers).
9. EN is now part of NZTE’s sector facilitation activities. NZTE have been invited to give the policy framework for these activities, and MED is currently in discussion with NZTE on this.
4. Implementation and Delivery of EDG
10. The design of the fund was to mitigate issues present in the former schemes by improving accessibility, eliminating delays and lack of clarity in the application process, and by adhering consistently to clear criteria.
11. Some of these issues have not been fully addressed:
− Assessment involves a significant degree of subjective interpretation of criteria.
− Firms may be declined at the end of the financial year not because they fail criteria but only because the fund has been fully spent.
− NZTE does not advertise the fund widely and this reduces the chances the fund will encourage those who do not usually seek help to do so, and so limits the fund’s ability to achieve the goal of changing opinions about seeking help.
− Turnaround of applications is on average a month, from first submission to final decision.
12. By Quarterly Report figures EDG was fully subscribed for the 04/05 year, and had an under spend in 03/04.
13. The numbers of firms applying to the EDG has increased 40% over the two years since implementation, giving a total of 684 firms applying to date. 570 applications were accepted. This is a small number and the fund at its current size will probably not have significant impacts on New Zealand SME capabilities.
14. There has been a large change in the approval rate. At inception 80% of all applicants were approved but the rate has subsequently reduced. Approvals were at 70% by January 2005, 60% by April 2005 and at 50% at the end of the 04/05 year.
15. Delivery of the EDG is even across regions and sectors. There is no significant difference in the numbers being accepted or declined across regions and sectors, and the rate and size of allocations is similar across regions and sectors.
7
16. Firms applying to the EDG and firms being accepted include large numbers of exporters. New criteria will be implemented in October 2005 to eliminate the export focus in favour of ‘net benefit to NZ.’
5. Outcomes of EDG
17. 84% of firms completed the projects they had sought funding for, the remainder decided not to undertake the project or stopped part‐way.
18. 78% of firms have seen benefits from the grant, in terms of having a product ready for launch or launching it, understanding which segments of the market to target, improving the way the firm runs, gaining or clarifying a strategy for the future, or entering a new market. This means of the 84% of firms who complete a project, 93% see positive gains from it. This is a very good result for the EDG.
19. For the final outcome of improving firm performance, 39% of EDG firms have seen gains in firm performance, in terms of export or domestic sales. 37% said it is too early yet to see such gains in performance. This was usually because while the grant was closed the project was taking longer than one year.
20. For the intermediate outcome of improved management capability, 56% of firms saw changes in the way they managed the business; 30% of firms said projects had no effect on management‐ but these firms typically had undertaken projects which were not geared at impacting on management. 47% of firms gained management level assistance and 41% gained business‐level assistance.
21. The management changes included altering the internal practices of the firm, learning from the project about ways to tackle such projects, gaining direction for the firm and sometimes confirming that their own approach had been right. Firms also reported an improvement in their overall ability to manage the business.
22. The activity types of intellectual property protection and prototype development had very low rates of firms reporting changes in management practices, undertaking management level projects, or gaining advice. Intellectual property protection and prototype development are not well fitted to the goals of building management capability.
23. The evaluation undertook a simple random sample of declined firms. However it had no information on why they were declined, and so if they undertook the project themselves even while being declined it may be because they had the capability and resources to do so, and so it was appropriate that they were declined.
24. The evaluation learnt that many were unclear on why they were declined, and this left them feeling ‘brushed off’ and unwilling to reapply for assistance. In order to not defeat the goal of improving the perception of the value of external assistance, it would be appropriate to give more details of why a firm was declined.
8
Analysis of the policy goals: management and business capability
25. The evaluation noted the fund is aimed at assisting both management and business capability. At present, the fund is roughly split 50‐50 between two types of firms: those whose goals for the grant were to improve their business’ capability, and firms who wished to improve their management capability, or who were undertaking management level projects.
26. The evaluation distinguished projects by whether they were improving business or management capability, and looked to see the impacts of each project type. The evaluation has found a significant association between the proportion of firms having had management level assistance and reporting impacts on management. This means that results from the evaluation show that firms undertaking management level projects will usually also see changes in the way they manage their business. Firms undertaking business level projects are less likely to see such changes.
27. The evaluation also found that firms undertaking projects defined as management level had significantly higher rates of gains in performance. This means that results from the evaluation show that firms undertaking management level projects will usually also see gains in their performance. Firms undertaking business level projects are less likely to see such changes.
28. There is a caveat to this finding. Firms undertaking business level projects usually said they had not yet seen gains as it was too early given the project they undertook. So time is needed to see if the above finding holds. It may not if many of the business level project firms see gains in performance from the project in future months.
Analysis of the policy goals: improving capability via information or via building skills
29. The evaluation found there is uncertainty in whether the fund’s objective is to be a channel by which managers can access information, or whether it is to assist them build their capability. At present, two types of firms are accessing the fund: those who can themselves implement the project but need information to do so, and those who need assistance to gain the capability to implement their project.
30. The evaluation distinguished projects by whether they delivered advice to build capability, or delivered information to firms, and looked to see the impacts of each project type. The evaluation has found a significant association between the proportion of firms having advice or not and having management level or business level assistance; and also seeing changes in management practice or not. In particular, firms receiving advice had high rates of receiving management level assistance and seeing changes in management practice; whereas firms receiving facts had higher rates of receiving business level assistance and not seeing changes in management practice. This means that results from the evaluation show that firms having advice will usually also gain management level assistance and usually
9
see changes in management practice. Firms gaining facts are less likely to see such changes.
31. As management level projects see higher rates of gain in performance, it seems aiding management via advice is the best way to aid management to achieve gains in performance.
32. The evaluation also found a tentative association between receiving advice and seeing gains in performance. This suggests firms receiving advice will see gains in performance at a higher rate than firms not receiving advice, but this relationship may change with time.
6. Operational recommendations for EDG.
33. Scoring criteria for assessing applications need to be defined carefully to ensure they may be applied objectively and filter applicants appropriately. This is a crucial part of ensuring the goals of policy are met. NZTE are currently reviewing their criteria and scoring and it would be appropriate for MED to be involved in this.
34. Firms applying when the fund is fully subscribed or close to full subscription, and which would otherwise pass, should be held over to the beginning of the next financial year, rather than being declined.
35. If an application is felt to have insufficient information it should be returned with a request for more information. Only those applications which fail the criteria should be declined.
36. Consideration should be given to advertising the fund to extend its reach and improve its ability to improve the perception of external advice in firms.
37. The EDG assessment team needs a better feedback mechanism of the outcomes of projects to help them in assessing applicants. Currently one report is made by firms to NZTE at the close of the grant, and it has an optional provision of sales data which most firms do not complete. A better feedback mechanism would be requiring in the report the provision of information on difficulties and successes implementing the project, and what was learnt as a result. There also should be a request for a second outcome report requiring information on sales six or twelve months post end of project.
38. An examination is made of the quality of applicants over 2005/6, and the reasons firms are declined, as part of monitoring the proportions of firms accepted and declined. The proportions accepted should be around 70%. If this total is inappropriate given the overall quality of applicants, then the positioning of the fund can be reviewed.
39. It is recommended NZTE establishes a system of storing electronically data from the application forms on all applicant firms’ financial and capability needs, taken from their description in the application form of the project and their goals for the external advice or expertise. This should be able to be compared to final outcomes from the project funded by the grant, and would place both an evaluation in the
10
future and NZTE in a stronger position to assess changes in the firms post intervention. MED should be involved in establishing this system.
40. The evaluation recommends a process is instigated of detailing in a more personalized way the reasons for a firm being declined, in order to prevent these firms feeling ‘brushed off’ and deciding against seeking help again, thereby thwarting the goal of improving the perception of external advice.
7. Policy recommendations for EDG
41. To target the fund more squarely at management level projects the assessment of proposals and possibly the description of the funds goals to the market, would need to be revised. This would require a definition from MED for NZTE of at least the core of what ‘management level’ is (even if the edges of the distinction with ‘business level’ are left somewhat blurry).
42. The fund could be weighted toward firms wishing to gain advice. This can be done by adjusting the assessment to spot these sorts of firms, and possibly the description of the fund to the market.
43. The evaluation recommends a second look is taken at the sample in six to twelve months to re‐assess outcomes. If the 37% who said it was too early to see benefits from the grant see gains in performance it will raise the success rate of the fund significantly. The opportunity can be taken of assessing the current situation of the firms who had reported seeing gains in performance, to see if the gains have continued. NZTE may be in the best position to undertake this, as part of its on‐going monitoring of clients.
8. Overall conclusions
44. The evaluation concludes that the EDG is effective in delivering intermediate outcomes, and thus far has achieved for nearly half of firms the final outcomes of improved firm performance. Further evaluative work is required to see the outcomes for the remaining half.
45. Determining additionality, that is, whether the fund was the cause of the changes in management or business capability and in firm performance, is usually taken to require establishing that the fund is the sine qua non cause of the effects in question via testing the counterfactual conditional that the effect would not happen without the cause. The evaluation was unable to demonstrate this with the methodology at its disposal. A well‐designed randomized control trial is the established method of demonstrating such a causal link, but a randomized control trial takes more time than the evaluation had to implement and evaluate, and would conflict with the aims and implementation of the fund, since it would mean withholding grants from firms thought to be worthy of them. The evaluation is satisfied that the evidence gathered by survey has established a sufficient association between the fund and outcomes to merit the fund continuing.
11
46. The evaluation concludes that the Enterprise Development Grants should continue, with some adjustments as specified above. It recommends that MED review the overall size of the fund and whether it is reaching appropriate numbers of firms.
PART ONE: Evaluation Goals and Methodology.
9. Goals of the evaluation
47. The evaluation had three overarching goals.
48. The first was to understand the operational implementation of both MED and NZTE policy. This would inform the Ministry whether operation design was consistent with policy and whether the operational implementation was such the goals of the fund could be achieved.
49. The second was to understand the delivery of the fund, in terms of numbers of firms assisted and declined, regional and sectoral distribution of funding, and the types of firms who had received funding.
50. The third goal was to determine whether the outcomes policy wished to achieve were occurring for the firms undertaking the activities funded by the EDF.
10. The intervention logic
51. The intervention logic describing the EDF is given below.
52. The yellow boxes describe the problems the intervention is to overcome, the pink boxes the activities the intervention will undertake to overcome the problems, the blue boxes are the intermediate outcomes the activities should produce, and the red boxes are the final outcomes that the intermediate outcomes, and so the intervention as a whole, should achieve.
53. In other words, the activities of engaging the services of a business mentor, undertaking more advanced management and technology based training, employing specific external advice and expertise in a management area and undertaking international market development activities should result in improving management capability, improving the perception of the value of external advice, and improving collaboration between entrepreneurs and existing firms, and this should result in improved firm performance.
11. Methodology
54. The evaluation used three different methods to meet its three goals.
55. To understand implementation the evaluation considered the policy documents of MED and NZTE, and met with the NZTE teams and personnel who manage the operation of the EDF, and learnt from them their processes and systems.
56. To understand delivery the evaluation analysed the data collected by NZTE on the firms who apply and are granted or declined a grant.
57. To understand outcomes the evaluation undertook a survey of a representative sample of accepted and declined firms and questioned them on their reasons for
13
applying for a grant, the activities they undertook using the grant, what they learnt, and what outcomes had been achieved.
58. Determining additionality, that is, whether the fund is the cause of any changes in management or business capability and in firm performance, is usually taken to require establishing that the fund is the sine qua non cause of any effects in question via testing the counterfactual conditional that the effect would not happen without the cause.
59. The evaluation was unable to demonstrate this with the methodology at its disposal. A well‐designed randomized control trial is the established method of demonstrating such a causal link, but a randomized control trial takes more time than the evaluation had to implement and evaluate, and would conflict with the aims and implementation of the fund, since it would mean withholding grants from firms thought to be worthy of them.
60. The evaluation is satisfied that the evidence gathered by survey as it has designed it would establish a sufficient association between the fund and outcomes to answer questions of outcomes and the question of whether the fund merits continuing.
12. Methodology of the survey
61. The survey was by telephone interview.
62. The population was all firms who had completed a project by April 2005, giving a total of 191. The sample was a stratified simple random sample with a sample size chosen to give a margin of error of 5% for estimates of proportions. This gave a minimum sample of 129. The evaluation surveyed 132 firms.
63. The sample was stratified by the activity types available under the EDG, and the activities to be included were selected by MED policy. Policy wished to know the outcomes for firms undertaking strategic design, intellectual property protection, prototype, business or strategic plans, feasibility studies, certification, systems evaluation, training, mentoring, market research and market plans.
64. The sample was selected as a stratified simple random sample but with sampling fractions which were similar in each stratum, and also very high, the data could be analysed as a simple random sample.
Enterprise Development Fund Logic Model
Lack of early stage financing
Enterprise Development Fund provides funding to smaller firms, entrepreneurs, start-ups, exporters and groups of companies, after an initial capability assessment, to: 1. Engage the services of a
business mentor 2. Undertake more
advanced management and technology based training
3. Employ specific external advice and expertise in a management area
4. Undertake international and domestic market development activities
Improved collaboration between firms
Increased global connectedness
Improved perception in the value of developing business capability
Improved firm performance
Improved international competitiveness of products/services
Improved business, management, technology, international business, exporting and leadership capability
Lack of international linkages/contacts
Lack of business management capability (including export, international management skills). Also access issues.
Undervalue contribution of external advice
Problems
Improved perception of the value of seeking external advice
Impediments to collaboration, with resultant inefficiencies
Risk aversion of firms
Activities Intermediate outcomes Final outcomes
PART TWO: Policy Rationale for the
Enterprise Development Fund
1. MED Policy
13. Policy objectives for the Foundation Services
65. In April 2003 Cabinet agreed to establish the Enterprise Development Fund as part of the Foundation Services of NZTE. (EDC (03) 54 refers)
66. The Foundation Services are to assist small, young firms, young entrepreneurs and start‐ups, via information, training and financial assistance. The rationale was that running a business requires skills and knowledge that small young firms may not yet have acquired. Acquiring them may be made difficult by the expense of purchasing external advice and expertise and a belief that expertise is not required or is not valuable. The foundation services were established to make acquiring the skills and knowledge easier and more attractive for firms. The rationale was given in EDC (03) 54 as:
Small businesses and entrepreneurs often lack the financial ability to employ all the expertise they need to get new concepts and projects up and running. They also tend to undervalue the potential contribution of expert advice on key elements of their business ideas. Without such advice, they are often unable to satisfy the requirements of investors, banks and other financiers for information, including business planning, approaches to preserving the value of intellectual property, and how they plan to market their products and services. As a result, small businesses often do not realise their full potential.
These barriers could be overcome if access to information was made simpler and business operators were provided with better opportunities to acquire the skills and knowledge to improve their management capability.
14. Policy objectives for the Enterprise Development Fund
67. Three Industry New Zealand and Trade New Zealand programmes were amalgamated to form the EDF: the Enterprise Awards Scheme, the Export Network programme and World Class New Zealanders. These three all delivered funding assistance to firms.
68. The goal of the former Enterprise Awards Scheme (EAS) became the overall goal of the EDF. This is to assist innovative firms and entrepreneurs to build capability by enabling them to employ expertise and advice.
16
69. Funding is for 50% of total costs up to a maximum of $20,000‐ an increase from the EAS total of $10,000. This is per applicant per annum, and is for businesses and entrepreneurs to:
− Engage the services of a business mentor for a period of time
− Undertake more advanced management and technology based training (as delivered through existing providers such as NZIM, Polytechnics etc)
− Employ specific external advice and expertise in a management area (such as feasibility studies, e‐business, market research, enhancement and uptake of new technologies, human resources, intellectual property, strategic planning, environmental management, production management)
− Undertake international market development activities, including new market investigation, trade fair participation, trade/business missions, business exchanges, and visiting buyers.
70. The design of the EDF was to address issues with its predecessor schemes. The issues were:
− Criteria: the EAS was a relative, awards based scheme, where the top proportion per month received assistance, and the quality of the proposals accepted for funded varies from round to round as approval is relative. The EDF was to be an objective criteria‐based scheme, where any firm which met certain criteria gained assistance.
− Application turnaround: businesses often did not know for up to two months whether their application was successful.
− Accessibility: the limited capability of small business owners to self‐appraise their needs limits the programmes ability to effectively address critical capability issues.
− Overlaps: There were overlaps with the case managed Business Growth Fund. The limited funding of the EAS lead to a greater focus on high growth and more established businesses than was intended.
17
2. NZTE policy
15. Design of the Enterprise Development Fund
71. In a paper to the Integration Ministers (Paper to the Integration Ministers 4 July 2003 refers), NZTE gave the detailed report back on its proposed operational design and criteria for the EDF requested in EDC (03) 54.
72. The paper noted that the EDF was an amalgamation of the three predecessor programmes and that it would remain similar to them. It also noted that grants were for $20,000 and for 50% costs. The activities to be funded were also the same.
73. NZTE proposed some additional design features. These were:
− Criteria: the fund would adhere to clear criteria and would centralise all final approvals within NZTE.
− Application turnaround: the approval processes would ensure a high level of responsiveness to clients. NZTE anticipated a fast turnaround of applications, and aimed for a five day turnaround from receipt of applications.
− Overlaps: the fund would remain accessible and responsive to start‐up businesses and entrepreneurs not yet in business.
− Ease of application: the application forms for the EDG and background material would be as clear and as easy to complete as possible. Being aware that applications forms and the information required must still be reasonably demanding to ensure accountability, NZTE undertook to provide clients with assistance in completing application forms.
16. Criteria for projects
74. NZTE established criteria for the project or activity firms wished funding for. These are:
− The project must have good commercial potential and add value to an existing business or an entrepreneur’s current activities
− Applicants must demonstrate that they have the capacity and capability (including financial and planning support) to carry the project through to commercialisation
− The project or activity must be capable of generating high returns, or high levels of growth for the applicant once commercialised or undertaken.
− The project must be consistent with relevant laws and regulations
17. Scoring of applications
75. NZTE noted it expected high levels of demand for the EDG, because it had experienced high levels of demand for the Enterprise Awards Scheme, where only
18
30% of applicants were accepted. NZTE proposed to score proposals for the EDG to ration demands for funding, so only those applications which best met the goals of the programme and of NZTE received funding. Applications were proposed to be scored against criteria with weightings so certain criteria were more important to achieving the critical score. The criteria were given as:
− Robustness of the project plan
− financial and organisational stability
− ability of organisation to implement project/undertake activity
− level of innovation or additionality associated with the project or activity
− level of growth within the organisation and likely economic benefits
− level of need for government assistance
18. Future changes: devolvement and criteria for 05/06
76. FRST and NZTE have begun to align the delivery of EDG and Smartstart (EDC Min (04) 25/5 refers). The alignment involves devolving grants of $5000 and less to an agency network.
77. This, on numbers to date, is 15% of all EDG grants. A half of these have been strategic development or product development grants and a quarter have been mentoring or training. The last quarter are evenly spread across the activities.
78. In July 2005 (subsequent to the evaluation’s timeframe) Ministers agreed to revise the EDG criteria to make clearer the distinction between programme, entry and assessment criteria, and to speed up assessment processes (EDC Min (04) 25/5 and The coordinated services delivery project update of 22 July 2005 refers). NZTE aims to implement the criteria and scoring system by 1 October 2005.
79. The revised assessment criteria are now:
− Robustness of project plan;
− Ability of business/individual to undertake activity (“previously ability of organisation to implement project or activity and their commitment and drive to undertake planned work”. This criteria also picks up “financial and organisational stability”);
− Potential to improve business performance (previously “potential for growth and benefit to regional/national economies”)
− Net benefit to NZ of improved business performance (previously “level of innovation or additionally associated with the project or activity”); and
− Level of difference the funding will make (previously “level of need for govt assistance/funding assistance and a catalyst for growth”).
19
80. NZTE has removed ‘financial ability’ as a criterion, principally because the firm must expend the full amount prior to reimbursement by NZTE, and this is a sufficient step to ensure the firm is financially able to undertake the project.
20
PART THREE: Findings and Recommendations on the Implementation of Policy
19. Programme implementation
81. The EDF was to be an amalgamation of the former Enterprise Awards, World Class New Zealanders, and the Export Network Programme, delivered by Trade New Zealand and Industry New Zealand.
82. NZTE has implemented the Export Networks Programme into the EDF quite differently from its implementation of the other two, in fact it could be argued it was not implemented into the EDF at all, but kept altogether separate. It was renamed: ‘Enterprise Networks’ (EN). The other two predecessors are renamed Enterprise Development Grants (EDG).
83. NZTE split the appropriation for EDF of $8.613M between the EN and the EDG. EN received $4.48M, EDG the rest ($4.413M).
84. Export Networks aided firms in their exporting efforts by assisting groups of firms to target high quality market development opportunities offshore. It did this by assisting groups of firms to take advantage of market opportunities which individual firms could not, thereby achieving gains in forex performance unavailable to individual firms.
85. The primary issue for MED with EN was the order of practice. NZTE selected offshore opportunities and requested interest from firms in those activities; rather than establishing a process whereby firms applied for funding to attend offshore activities they had selected.
86. The selection of offshore activities by NZTE was sector‐led: events were ranked by their degree of fit to NZTE’s sector priorities, and those which fitted best were chosen for the year’s EN activity. Client managers were notified and they invited firms on their books whom they deemed appropriate to apply. Firms not selected could not apply. Offshore activities which were not selected by NZTE were not available for firms
87. The design principles of the EN contradicted CER. Targeting export education breaches the EDF criteria of no common ownership. A number of firms larger than criteria for EN have accessed the fund.
20. Response by MED to the Enterprise Networks implementation
88. MED considered the implementation of EN sufficiently different from their policy objectives to remove Enterprise Networks from the EDF. Funding for Enterprise Networks was moved to operational expenditure (CAB Min (04) 38/4 refers).
89. EN is now part of NZTE’s sector facilitation activities. NZTE have been invited to give the policy framework for these activities, and MED is currently in discussion with NZTE on this.
21
The report ends its evaluation of EN here. The rest of the report is on the evaluation of the remainder of the EDF: the Enterprise Development Grants scheme.
21. Operational implementation: the Business Evaluation Team
90. The client managers for most firms using the EDG are those of the Business Evaluation Team. The Business Evaluation Team is also the ‘frontline’ team who handle the calls to the NZTE hotline phone number. There are 14 members of the team.
91. As client managers, each member of the team has large numbers of EDF and other NZTE clients. As they have so many, they are not expected to have a proactive relationship with all their clients. Individual client managers may choose to ring firms on their books, but this is not obligatory. Hence EDF clients are called ‘light touch’ firms. They are not closely watched by NZTE.
22. Operational implementation: the Enterprise Development Team
92. The Enterprise Development Team is the central point of assessment for applications for the fund. There are four members.
93. Firms can apply for the fund directly to the team using applications from the NZTE website, or can be guided to the team by the business evaluation team. If firms apply directly to the enterprise development team and do not have a client manager, the team finds them a suitable client manager, usually one from the business evaluation team.
94. The team has two hats. They have a pre‐assessment role, where the team assists firms with the filling out of firms, reviews applications and may personally advise firms on whether the applications are sufficiently complete and what additional information is required. There is potential tension between these two roles, but careful use of advice on potential is also of benefit to firms who are clearly outside the criteria. The assessment process is discussed further in paragraphs 102‐133
95. The team’s other hat is assessment. Their process is to divide applications among team members, and then assess them. The team then meets, usually once a week, but more frequently as necessary, to discuss their assessment of applications. At this point they either confirm the assessment, or reassess them.
96. Firms are notified by letters if they are accepted or declined. From December 2004 letters to declined firms were changed to include details on why the application was declined. If firms were declined only because they lacked information but are
22
felt to have potential they are encouraged to re‐apply with the additional information.
97. The team is encouraged to become familiar with business, by allowing members to visit other NZTE client managers based across New Zealand.
23. Design of the Enterprise Development Grants scheme
98. The evaluation examined whether design issues and proposed design (given in part two) have been met and implemented.
Assessment against criteria
99. Firms are assessed against criteria. However there are some issues with assessment which means it is not as objective as it might be, and assessment does not always adhere to criteria in ways clear to firms (even if the criteria themselves are clear to firms).
100. Firms who apply late in the financial year are declined only because the funding year is finished, and not because they do not qualify. At the end of the 04/05 year the score firms had to achieve was raised in order to prevent an over spend, suggesting firms applying later in the year were disadvantaged. See section 31 for further details.
Recommendation
101. It would be more consistent with a criteria‐based scheme if firms applying very late were held over and approved against the criteria in the first weeks of the new financial year.
102. Scoring should not be raised only to prevent an over‐spend apparent towards the end of the financial year, as this unfairly disadvantages firms applying late in the financial year. Having raised the scoring NZTE intends to leave it at this level, but should not raise it again before the end of the 05/06 year.
Turnaround of applications
103. NZTE aims for a fast turnaround of applications once they are complete. NZTE stated the turnaround would be 5‐days from receipt of application, and there are some issues in meeting this. NZTE does not on average make a 5‐day turnaround from receipt of application.
104. Instead, because it assists firms fill out their applications, and sends applications back when they need more information, NZTE counts its turnaround from the date the application form is sufficiently complete. NZTE says it maintains a seven day turnaround from this point. See section 33 for further details.
23
Recommendation
105. It may be preferable to resolve the inconsistency of returning applications for more detail and declining applications but with an invitation to reapply with more detail. The evaluation recommends declining applications only when the firm actually fails the criteria, and not when there is insufficient information to tell whether it ought to pass or fail. The latter leaves firms in considerable uncertainty.
Accessibility
106. The accessibility of the fund is limited by its being advertised to economic development agencies and other consultants, because only firms already seeking assistance or advice will learn about it. This lessens the chances the fund will encourage those firms who don’t usually seek external advice to do so. It also limits the fund to those who have already decided they have a capability gap.
Recommendation
107. Advertising the fund more broadly may solve this issue, however advertising the fund broadly must be balanced against the potential resulting problem of the fund selling out very quickly in the year.
Overlaps
108. The delivery of the fund by NZTE has been successful in avoiding overlaps with the Growth Services Fund. NZTE have added to the eligibility criteria the requirement that firms have a turnover of less than NZ$5M and/or 20 FTEs or less.
109. EDG clients are typically low growth, low turnover and fairly young. However there was no definition of what ‘young’ meant, or whether it was to be interpreted in terms of age, or, for example, growth, or capability. See sections 45 and 46 for further details.
Ease of application process
110. The evaluation asked the sample of firms it surveyed about the length of application time and whether firms were satisfied with the time it took them to apply. The majority of firms said that given what resulted from the grant it was worth the time and effort.
111. Most firms did go on to say the process was long‐winded and took a great deal of time‐ for some days, others hours. On the whole however, having received funds and seen a benefit, they were felt the grant was worth the time it took to apply.
112. Many also noted that the process of writing out their business plan and plan for the project for the application form was beneficial for their own clarity of business goals and current situation.
24
113. The evaluation can not tell how many firms did give up in filling out the forms and never applied. Some firms told the evaluation they nearly gave up. If there were ways of shortening the application forms, it may be worth considering.
114. Some firms commented that they wished for more flexibility in adjusting the project once it had started. NZTE will not reimburse firms for additional tasks on the project to those in the original application. The claims must match the tasks laid out in the original application form. The only variation which is permitted is to change the external provider. Many firms find as the project develops that they need to add tasks to it, and firms have explained to the evaluation that they cannot predict the project exactly one year out, and often things eventuate which they didn’t anticipate. The firms argued that this is the nature of undertaking early‐stage projects.
115. To get around this, firms try to have applications accepted with minimal information, to give them the freedom to add tasks later, and yet still have the project fit the original broad application.
116. NZTE notes it must balance flexibility in process with the time it must spend evaluating whether a change in a project is appropriate and whether it changes the project sufficiently it as a whole no longer qualifies for funding. It feels the costs of this will outweigh the benefits to firms.
Recommendation
117. With devolvement it may be that issue of adjusting projects mid‐way can be ameliorated with the greater degree of contact between firms and the delivery agents. This issue might best be considered when the devolvement is evaluated.
Criteria and scoring of proposals
118. From implementation until October 2005 the criteria which have been used to assess firms were as initially proposed. (The changes made subsequent to the time‐frame of the evaluation were given above).
119. The categories for the criteria for the time‐frame of the evaluation were:
− robustness of proposal
− financial and organisational stability
worth it Not worth it
Worth of time taken
0
20
40
60
80
100
120
Num
ber
Sample of closed projects 1/10/2003 to 4/2005
Satisfaction with the time taken to apply
25
− ability of organisation to implement project or activity and their commitment and drive to undertake planned work
− potential for growth and benefit to regional/national economies
− level of need for government assistance/funding assistance as a catalyst for growth
− level of innovation or additionality of the product/service of the value added to existing
120. Under each category have been added descriptions and corresponding scores, ranging from poor to good.
121. The team is to match the application to the description and score accordingly. For example under: ‘ability of organisation to implement project or activity’ the descriptions and scores are:
1‐3 Lack of or limited experience in the industry
Lack of or limited experience in business
Management depth appears light in some areas
4‐6 Demonstrated level and depth of management experience
Demonstration of drive to succeed
Evidence of transferable skills
Mentors/support work engaged as needed
7‐10 Demonstrated depth of management experience
Experience in industry
Obtains external advice or mentoring
122. The higher the score the better an applicant does by that criterion. The scores are then weighted. The scores an applicant receives under ‘growth potential’ and ‘innovativeness of product’ are multiplied by 20. The others are multiplied by 15.
123. An applicant must obtain a minimum of 4 out of 10 for ‘growth potential’ and ‘innovativeness of product’ and 3 out of 10 for each of the others to be approved. Multiplied, that is 80 out of 100 and 45 out of 100 respectively.
124. An applicant must get a total of 570 overall to be approved.
125. The team meets regularly to confirm assessment scoring, however they have reported to the evaluation that there is close consistency in scoring by the members, and so they have decided to cease meeting regularly to check consistency of scoring.
Recommendation
126. The evaluation has noted that there is some ambiguity between the criterion addressing ability of the firm to implement the project they seek funding for,
26
which is measured by management experience, and the capability‐building goal of the fund.
127. Firms gain higher scores and thereby are more likely to be granted an award the more experience they have. They gain between 7 and 10 points if they have: ‘demonstrated depth of management experience’ or ‘experience in industry’ or ‘obtains external advice or mentoring.’ They will fail the criteria if they have: ‘lack of or limited experience in the industry’ or ‘lack of or limited experience in business’ or ‘management depth appears light in some areas.’
128. It seems inconsistent with building capability if a lack of or limited amount of experience fails a firm. It may help if the criterion and scoring refer specifically to the minimum kind of experience firms should have, to qualify, and not the degree of experience (measured, say, by years). The EDG can then be more easily aimed at those firms with certain basic skills who lack more sophisticated skills.
129. The tests for net benefit to NZ are improved productivity and efficiency, limited displacement (displacement would result from funding firms with significant domestic focus and significant domestic competition) and presence of spillovers. These are still high level concepts requiring variables which can identify in a project description when and to what degree productivity would be improved, or when and where a spillover would occur.
130. Translating high level concepts into concrete variables (operationalising the concepts) is a crucial part of ensuring the goals of policy are met. For example, ‘level of difference’ must be operationalised in such a way that the variables which are to indicate funding will make a level of difference, really do indicate the funding makes the required difference. If they don’t, firms are not truly being assessed against that criteria.
131. The evaluation recommends MED is involved in assisting with development of the scoring criteria for the EDG assessments.
24. Project activities available under EDG
132. The full range of project activities available to firms through the EDG is broader than first proposed. The activities currently available are:
− strategic business development (including business/strategic plan development, human resource strategic plan development, financial viability planning, strategic design advice)
− feasibility studies
− product development (including prototype design, development and testing, intellectual property protection)
− business and operational excellence (including international quality standards certification, systems evaluation and development, environmental management system)
27
− e‐commerce and e‐business strategies (including development of a e‐strategy)
− Market strategy development (including market research or new market investigation, marketing plans).
133. These activities cover a very broad range of possible activities, but are not all equally popular, and some are rarely taken up by firms. The most popular, and so most of the allocations have gone to: intellectual property protection, prototype development, strategic business development, and strategic design advice. See section 47 for further details.
28
PART FOUR: Findings and Recommendations on
Delivery of EDG
Note: NZTE has some problems with its data system and has difficulties extracting data from it. The evaluation is aware that details on at least 153 applications are missing from its data set. This is 17.4% of the total grant data for EDF. The evaluation has no details on these applicants and so has not included them in its analysis. The evaluation is confident the data it has is sufficient to give an accurate picture of the delivery of the EDG to date.
134. There are possible impacts of delivery that the evaluation is unable to determine exactly; for example, the development of a network of consultants able to help businesses and willing to encourage businesses to seek their help. Conversely, for example, it is possible that the fund stimulates consultants to adjust their fees upward to take advantage of the 50:50 funding.
3. Uptake of the fund
Note: for section 3 and 4 only the evaluation reports on data for 20 Months, that is, from October 2003 to end June 2005. This covers the time from implementation until the end of the 04/05 year.
25. Total allocations
135. NZTE allocated $4.133M to EDG.
136. The total appropriated to the EDG for 03/04 and 04/05 is $8.266M
137. The evaluation has a total allocation for 03/04 and 04/05 of $6,424,798.11.
138. By NZTE quarterly report figures there was an under spend in 03/04 but 04/05 was fully subscribed. Quarterly report figures have a total allocation for EDG of $M7.253.
26. Total number of applications
139. In 20 months NZTE has received 809 applications for the Enterprise Development Grants capability building component.
140. In the 2003/2004 year there were 320 applications.
141. In the 2004/2005 year there were 489 applications. This is a 53% increase from 03/04 to 04/05.
29
27. Total number of firms
142. Note that firms reapply, and so there have been more applications for the grant than there are firms applying
143. Over 20 months from October 2003 to 30 June 2005 684 firms have applied for the EDG. The small size of the EDG limits its impact on SMEs.
144. 285 firms applied in 03/04.
145. 399 firms applied in 04/05. This is a 40% increase from 03/04 to 04/05.
146. The graph right shows firms who reapplied. Firms claiming two grants are represented by dots in the middle of the graph. Those who did not receive two grants are the dots on $0 on both axes.
147. 136 firms (16%) have reapplied for the EDG. 68 (8%) reapplied for the same project, with 20 of those declined again, and 34 accepted. Three were declined twice then accepted. The evaluation has recommended earlier that a clear distinction be made in assessment processes between returning applications for more information and declining applications.
28. Approval rate
148. In the period October 2003 to June 2005 EDF has approved 570 applications and declined 216 applications.
149. This is 786 in total, and the difference from the earlier total given of applications (809) is that that was number of applications handled. Some of these have been returned to firms who did not reapply.
150. In the 2003/2004 year 251 applications were accepted (81%) and 57 declined (18.5%).
151. In the 2004/2005 year 319 applications were accepted (68%) and 159 were declined (33%).
29. Calls to the Business Evaluation Team
152. EDG inquiries can be made to the business evaluation team who run the ‘frontline’ NZTE hotline number, and the team may recommend firms to apply for an EDG grant. The data from the call centre does not give a picture of the interest in EDG going via the centre.
$0.00 $5,000.00 $10,000.00 $15,000.00 $20,000.00 $25,000.00
2003/2004 Amount claimed
$0.00
$5,000.00
$10,000.00
$15,000.00
$20,000.00
2004
/200
5 A
mou
nt c
laim
ed
1/10/2003 to 30/6/2005
Firms returning for a second grant
30
OC
T 20
03
NO
V 2
003
DEC
200
3
JAN
200
4
FEB
2004
MAR
200
4
APR
200
4
MAY
200
4
JUN
200
4
JUL
2004
AU
G 2
004
SEP
2004
OC
T 20
04
NO
V 2
004
DEC
200
4
JAN
200
5
FEB
2005
MAR
200
5
APR
200
5
MAY
200
5
JUN
200
5
Month
0
20
40
60
80
Num
ber o
f ass
essm
ents
1/10/2003 to 30/5/2005
Applications assessed
OC
T 20
03
NO
V 20
03
DE
C 2
003
JAN
200
4
FEB
200
4
MAR
200
4
APR
200
4
MAY
200
4
JUN
200
4
JUL
2004
AU
G 2
004
SEP
200
4
OC
T 20
04
NO
V 20
04
DE
C 2
004
JAN
200
5
FEB
200
5
MAR
200
5
Month
0
20
40
60
80
Num
ber o
f app
licat
ions
1/10/2003 to 7/4/2005
Applications lodged
153. The call centre does not tally distinctly the various NZTE grants, so all grant inquiry by telephone is under ‘grants.’ From 03/04 to March 04/05 (see graph right) the number of requests for grants handled by the hotline call centre looks steady.
154. The data on web hits does not exclude hits generated by spam, so the web data does not show whether genuine hits have increased or decreased. NZTE is working to build firewalls, and notes that this will show as a marked decrease in hits, but will not be a decrease in genuine hits.
30. Change in application numbers over time
155. There has been an increase in the numbers applying to the EDG over the two years of its operation. After a small beginning, the interest in EDG grew. Numbers applying jumped in April 2004 to roughly fifty a month and have remained around that total.
156. The graphs below show applications lodged (graph below left) and applications assessed (graph below right). Note the surge in numbers lodged in June and July 2004, with the corresponding surge in assessments in June and August 2004 and carrying over through September and October 2004.
JUL
2003
AU
G 2
003
SEP
2003
OC
T 20
03N
OV
2003
DEC
200
3JA
N 2
004
FEB
2004
MAR
200
4A
PR 2
004
MA
Y 20
04JU
N 2
004
JUL
2004
AU
G 2
004
SEP
2004
OC
T 20
04N
OV
2004
DEC
200
4JA
N 2
005
FEB
2005
MAR
200
5A
PR 2
005
Month
0
500
1000
1500
2000
2500
3000
Num
ber
Total callsWeb inquiriesGrant advice
1/17/2003 to 30/4/2005
Inquiries to the hotline team
31
157. There was a drop in decisions in July, presumably as a result of yearend 03/04 activities, and many applications lodged in July look to have been assessed in August.
32
4. Assessment of applications
31. Change in approval rate over time
158. From 03/04 to 04/05 the number of applications which have been declined has increased from 57 to 159, which is an increase of 179%. The number of applications accepted has increased from 251 to 319, which is an increase of 27%.
159. This shows that over the two years of delivery there has been a change in the approval rate. The graphs below show the numbers accepted and declined over time, with the dark line added to show the trend.
160. The graph below left shows that the numbers of accepted applications are in decline. The graph below right shows that the numbers declined are increasing.
161. As another way of viewing this change, consider the ratio of acceptances to declines.
162. If the numbers accepted and declined over the total number of applications is examined (graph below right) it can be seen that the proportion of applications accepted has steadily declined from 80% (0.80) in November 2003 to 60% in August 2004 and 50% (0.50) in May 2005.
NO
V 20
03D
EC
200
3JA
N 2
004
FEB
200
4M
AR 2
004
APR
200
4M
AY 2
004
JUN
200
4JU
L 20
04A
UG
200
4S
EP 2
004
OC
T 20
04N
OV
2004
DE
C 2
004
JAN
200
5FE
B 2
005
MAR
200
5A
PR 2
005
MAY
200
5
Month
10
20
30
40
50
60
Num
ber
Accepted applicationsMoving average of accepted applications
1/10/2003 to 30/5/2005
Accepted applications over time
NO
V 20
03D
EC
200
3JA
N 2
004
FEB
200
4M
AR 2
004
APR
200
4M
AY 2
004
JUN
200
4JU
L 20
04A
UG
200
4S
EP 2
004
OC
T 20
04N
OV
2004
DE
C 2
004
JAN
200
5FE
B 2
005
MAR
200
5A
PR 2
005
MAY
200
5
Month
0
10
20
30
40
50
60
Num
ber
Declined applicationsMoving average of declined applications
1/10/2003 to 30/5/2005
Declined applications over time
33
Date Allocation
October-03 $183,604.23November-03 $236,002.47December-03 $383,350.52
January-04 $247,223.25February-04 $115,889.17
March-04 $225,215.89April-04 $404,186.75May-04 $456,915.74
June-04 $703,380.07Total $2,955,768.09
July-04 $117,510.73August-04 $593,816.55
September-04 $468,432.43October-04 $465,113.12
November-04 $334,250.10December-04 $361,215.81
January-05 $307,229.22February-05 $391,522.81
March-05 $347,920.68Total $3,387,011.45
163. Put another way, the number of applications declined has risen from below 20% (0.20) to around 50% (0.50).
164. This is a large change. It has resulted in a very large decline rate.
165. The change in the proportions accepted and declined may have a number of explanations.
166. First, demand for the fund may be higher than the EDG can currently accommodate, and so the EDG team is forced to fail more applications than they can pass. Second, it might be that over time the applications are lowering in standard. Third, there may have been a change in assessment methods. Fourth, the there may be an issue with the criteria themselves.
167. Examining the expenditure of the fund (the data the evaluation has will be sufficiently accurate to allow this) shows it was not in serious danger of being over subscribed early in the 03/04 year when acceptance numbers first declined, nor even toward the end of the 04/05 year when declines reached over 40% (see table right). In March 2005 the fund still had ¾ of a million dollars left to spend. Decreasing numbers of accepted applications seems inconsistent with an effort to prevent oversubscription.
168. The second possible explanation was a change in the standard of applications. This may be due to a change in the types of firms applying. Perhaps a truly representative standard of applicable firms is now being seen by the team; because better than average firms applied early in the life of the fund.
169. As NZTE does not advertise the fund widely, in its early days only those firms who are proactive about seeking assistance would have known about it. These firms may have also been better than average.
170. If this is the case, the score to pass the criteria is higher than the average firm can manage, and perhaps is too high.
171. Conversely, the change in standards may have arisen because without advertising, only those firms who approach economic development agencies or other consultants discover the fund. These firms may be
NO
V 20
03D
EC
200
3JA
N 2
004
FEB
200
4M
AR
200
4AP
R 2
004
MAY
200
4JU
N 2
004
JUL
2004
AUG
200
4S
EP 2
004
OC
T 20
04N
OV
2004
DE
C 2
004
JAN
200
5FE
B 2
005
MA
R 2
005
APR
200
5M
AY 2
005
Month
0.2
0.4
0.6
0.8
Prop
ortio
n
Accepted applicationsMoving average of accepted applicationsDeclined applicationsMoving average of declined applications
1/10/2003 to 30/5/2005
Proportion of applications accepted and declined over time
34
less able than average. If this is the case, advertising may raise the standard of firms applying.
172. NZTE has recently raised the maximum score firms had to achieve to be awarded the grant, from 570 to 585. This was in March 2005 and was said to be in order to prevent an over spend in the 04/05 year. However as shown above there was a fair bit of money still available in March, and the decline in proportions accepted began earlier than March, so this cannot be the explanation for the trend as a whole. If the standards of firms applying have declined, raising the score will compound the issue. This will be able to be seen in the approval rates next year.
173. Finally, the standard of applications may explain increasing levels of declines if sufficient numbers are declined for poor quality applications, but are subsequently approved. However the numbers in this situation are too small for this. Only 8% were declined and then reapplied for the same project.
174. The third possible explanation was a change in assessment methods.
175. NZTE has told the evaluation that their assessment has become more thorough with expanded assessment instructions implemented in January 2005. The expanded assessment instructions contained detailed instructions for the existing scoring requirements to help the EDG team score applications. These might have made it easier to determine the standard of the applications, causing higher numbers to be declined, or they may have made it tougher for firms to pass assessment and be approved. Again, this change was implemented too late to be the cause of the decline.
176. The final possible cause was an issue with the criteria and their application.
177. In assessing applications by applying criteria the team makes a decision, and the evaluation has looked at the information the team has at its disposal to make its decisions, and how that is translating into decisions. Understanding this may help in understanding why there is a trend of increasing declines.
178. Considering first the reasons for declining firms, after December 2004 (which is the date from which NZTE kept detailed records on their reasons for declining firms), 75% are failed on ‘potential for growth and benefit to the economy’, 62% are failed on ‘level of innovation of the product or service, or value‐add to the firm’s activities’.
179. Less than 10% are failed on ‘level of need for government assistance’, ‘ability of firm to implement the project’ or on ‘proof of funding’. A third is failed on ‘financial stability’. There is a fairly even split on pass or fail on ‘robustness of proposal.’ This criterion is general‐ the overall feel for the application itself.
180. The main reasons for failure then are that the applications show low or poor potential for growth, from the idea, product or service proposed; or show low or no benefit to the economy. This suggests that the EDG has not had firms with lower ability, as was speculated above; else firms would be being failed on ‘ability to implement the project’ or ‘financial stability’.
35
181. Rather, the team is rejecting firms because they feel they have low or poor potential for growth and low benefit to the national economy.
182. Looking next to the information the team has available to make these decisions, the team has the information from the story woven by the firm in their application, information from other NZTE client or sector managers, and information from their own experience of assessing applications.
183. The team is limited in information from assessing previous applications because there is very little feedback for them on their previous decisions. The final reports firms file usually have no information in them on sales. There is no request in the forms for information on issues faced by the firm in implementing the project. There is no other process of follow up. As a result, the team is not learning from the outcomes of their decisions which of their decisions were right or wrong.
184. So with limited information, the team must make one of the hardest decisions to make about firms‐ the potential of a small and young business and a new product. This decision is genuinely uncertain. As the EDG team noted to the evaluation, ‘it’s a real punt’.
185. Furthermore, with the fund going to young inexperienced firms, NZTE faces genuine risk that funds go to firms who will fail anyway. NZTE may feel it needs to be risk averse in order to best fulfill its obligations to Government. This may have translated into pressure on the EDG team to be conservative in their assessments.
186. It may be that the reason firms are being declined in increasing numbers is because without good experience in spotting and acknowledging potential, and perhaps with an overly strong concern of risk, the team has become too conservative in its decisions.
32. Recommendations
187. The evaluation recommends a mechanism for more detailed feedback to the team on the results of projects and grants, so the team may learn the results of their decisions, and so learn about making these sorts of decisions.
188. It recommends a careful look at the data for the year 2005/2006, to see whether the trend in application decisions continues. It also recommends that NZTE comes to an opinion on application quality and whether standards are declining overall, and watches the quality over the next year. This will inform NZTE on whether scoring levels and overall positioning of the fund are appropriate.
189. If standards are not declining and the trend continues, the evaluation recommends further resources be provided to the EDG team in terms of at least one highly experienced case manager or former business person, perhaps temporally to assist the team adjust their scoring, or permanently. The evaluation suggests NZTE consider approving a random sample of applications to counter conservatism in the assessment.
36
190. It recommends that MED and NZTE agree that given the EDG is to help young, inexperienced firms, which are therefore a higher risk, a percentage of firms failing in spite of EDG assistance is likely and acceptable. In turn, NZTE can encourage its assessment team to be risk‐taking.
33. Time taken to approve grants
191. The EDG Team aims to assess a proposal within 7 days. The team has told the evaluation that it has been successful in keeping to this timeframe this year.
192. The graph right shows the distribution of time taken to assess applications. Durations lie mostly between 7 and 30 days, but can stretch out to 90 days and longer.
193. The team has explained to the evaluation that its turnaround of seven days is once the application is complete. An application may not be considered complete first time, and so the team must send them back. So the length of time taken to assess applications is a function of many firms not sending in applications with enough information for a proper assessment.
194. The added work for the team to reassess applications and the inconvenience to the firm in delays makes it worth considering whether the application forms are clear enough.
34. Average time taken to approve grants over time
195. The evaluation looked to see if the assessment time changed over time. A look at the average time taken for each decision per month shows that the average number of days have fallen since the large averages of March, April and May 2004 (see graph below). The average is currently a month, indicated by the horizontal line at 30 days.
196. Either the team is improving in its ability to assess applications, or firms are submitting more complete assessments. NZTE has decided not to provide data to the evaluation on time taken from initial lodgment to decision date post March 2005, so the evaluation does not know whether the team has improved further.
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 62 66 70 77 84 90 101
111
Days
0
5
10
15
20
25
30
Num
ber o
f app
licat
ions
Days taken for a grant decision
37
197. There was a very large spike in time taken in March, April and May 2004. It was a 500% shift in the average. This was due to the spike in the numbers of applications lodged causing very high workloads for the team. This shows the level of interest the team at its current size can accommodate‐ around 40 applications a month.
35. Recommendation
198. NZTE is adjusting its assessment and approval processes to shorten their duration; this may be sufficient to reduce the time taken to revue and decide on applications. Reducing the workload by 15% (the proportion being devolved) may also help. However the evaluation recommends NZTE record three dates per application‐ submission, start of assessment and decision date so time taken can be more easily determined and tracked.
DEC
200
3
JAN
200
4
FEB
2004
MAR
200
4
APR
200
4
MA
Y 20
04
JUN
200
4
JUL
2004
AU
G 2
004
SEP
2004
OC
T 20
04
NO
V 2
004
DEC
200
4
JAN
200
5
FEB
2005
MAR
200
5
Month Grant Allocated
0
30
60
90
120
150
180
Ave
rage
num
ber o
f day
s
1/12/2003 to 30/3/2005
Days taken to decide on grant, over time
38
Auck
land
Can
terb
ury
Cha
tham
Isla
nds
East
ern
Bay
of P
lent
yH
awke
's B
ayK
apiti
/Hor
owhe
nua
Kin
g C
ount
ryM
anaw
atu
Mar
lbor
ough
Nel
son/
Tasm
anN
orth
land
Ota
goR
otor
uaSo
uthl
and
Taira
whi
tiTa
rana
kiTa
raru
aTa
upo
Tham
es V
alle
yW
aika
toW
aira
rapa
anga
nui/R
uape
hu/R
angi
tikei
Wel
lingt
onW
est C
oast
Wes
tern
Bay
of P
lent
y
Region
0
50
100
150
200
250
App
licat
ions
Number of grant applications per region 1/10/2003 to 7/4/2005
5. Distribution of the EDG
Note: For the sections 5,6,7,8 the data is for 18 months: October 2003 to April 2005. This reduces the totals of applications to 723, with 517 accepted and 178 declined.
36. Summary of findings on distribution of the EDG
199. The evaluation has found the distribution is similar across regions and sectors, and there is no significant difference in acceptance rate for any region or sector.1
37. Regional spread of firms assisted
200. Auckland has had higher numbers of applications for the EDG than other regions (see graph below left). Wellington and Canterbury have the second largest numbers of applications.
201. Controlling for numbers of firm per region2 shows that the rate of grant applications per region is fairly similar, shown on the graph below right by bars of more even height.
1 For regional spread a chi-square test of independence gives a p-value of .282 and for sectoral spread a chi-square test of independence gives a p-value of .229. 2 Using Statistics New Zealand 2003 data on firms per region.
Auc
klan
dC
ante
rbur
yC
hath
am Is
land
sEa
ster
n Ba
y of
Ple
nty
Haw
ke's
Bay
Kap
iti/H
orow
henu
aK
ing
Cou
ntry
Man
awat
uM
arlb
orou
ghN
elso
n/Ta
sman
Nor
thla
ndO
tago
Rot
orua
Sout
hlan
dTa
iraw
hiti
Tara
naki
Tara
rua
Taup
oTh
ames
Val
ley
Wai
kato
Wai
rara
paan
ganu
i/Rua
pehu
/Ran
gitik
eiW
ellin
gton
Wes
t Coa
stW
este
rn B
ay o
f Ple
nty
Region
0.00
0.10
0.20
0.30
0.40
0.50
App
licat
ion
rate
%
Rate of grant applications per region 1/10/2003 to 7/4/2005
39
AucklandC
anterburyC
hatham Islands
Eastern Bay of PlentyH
awke's B
ayK
apiti/Horow
henuaK
ing Country
Manaw
atuM
arlboroughN
elson/Tasman
Northland
Otago
Rotorua
Southland
Tairawhiti
TaranakiTararuaTaupoTham
es ValleyW
aikatoW
airarapaW
anganui/Ruapehu/R
aW
ellingtonW
est Coast
Western Bay of PlentyRegion
0
20
40
60
80
100
Acc
epta
nce
rate
%
Rate of acceptances per region 1/10/2003 to 7/4/2005
Auck
land
Can
terb
ury
East
ern
Bay
of P
lent
y
Haw
ke's
Bay
Kap
iti/H
orow
henu
a
Kin
g C
ount
ry
Man
awat
u
Mar
lbor
ough
Nel
son/
Tasm
an
Nor
thla
nd
Ota
go
Rot
orua
Sout
hlan
d
Taira
whi
ti
Tara
naki
Tham
es V
alle
y
Wai
kato
Wai
rara
pa
anga
nui/R
uape
hu/R
angi
tikei
Wel
lingt
on
Wes
t Coa
st
Wes
tern
Bay
of P
lent
y
Region
0
10
20
30
40
50
60
Allo
catio
n pe
r firm
$
Allocation per firm in region 1/10/2003 to 7/4/2005
202. Controlling further for the numbers of firms applying per region, it can be seen that the proportion of applications accepted per region that the acceptance rate per region is very similar. The bars on the graph right are all of similar height.
203. From the above analysis it can be concluded that the EDF is assisting firms in all regions evenly.
38. Regional spread of allocations
204. Auckland has had the highest total allocations of grants from the EDF to date (see graph below left).
205. Controlling for the number of firms in the region3 shows the amount of allocations per region varies, but not in favour of the larger regions (see graph below right).
3 Using Statistics New Zealand 2003 data on firms per region.
Auck
land
Can
terb
ury
East
ern
Bay
of P
lent
yH
awke
's B
ayK
apiti
/Hor
owhe
nua
Kin
g C
ount
ryM
anaw
atu
Mar
lbor
ough
Nel
son/
Tasm
anN
orth
land
Ota
goR
otor
uaSo
uthl
and
Taira
whi
tiTa
rana
kiTh
ames
Val
ley
Wai
kato
Wai
rara
paan
ganu
i/Rua
pehu
/Ran
gitik
eiW
ellin
gton
Wes
t Coa
stW
este
rn B
ay o
f Ple
nty
Region
0
500000
1000000
1500000
2000000
2500000
Am
ount
allo
cate
d
Allocation per region 1/10/2003 to 7/4/2005
40
206. Controlling for the number of applicants per region shows allocations do not vary much across region (see graph right).
207. The evaluation has found there is no significant difference in the rate of acceptance for different regions.4
208. The EDF is assisting firms in all regions fairly evenly.
39. Sectoral spread of allocations
209. NZTE divides firms into seven sectors: manufacturing; information, communication and technology; creative and services; food and beverage, bio‐technology; wood processing, building and interiors and education.
210. Data received from NZTE without a firm’s sector nominated have been called ‘uncoded’.
211. The manufacturing sector has made the largest number of applications, with 206. See graph below right.
212. The next two highest are information, communication and technology and creative and services. Both had 130 applications.
213. The manufacturing has also had the highest number of grants allocated, with 160 accepted, or 31% of all grants. The information sector had 76% of applications accepted and creative and services had 60% accepted.
214. Manufacturing has received the largest total allocations: $2,095,555, but this reflects the larger number of grants applied for (see graph bottom left and table bottom left).
4 A chi-square test of independence gives a p-value of .282
Bio-
tech
nolo
gy
Cre
ativ
e &
Ser
vice
s
Educ
atio
n
Food
& B
ever
age
mun
icat
ion
& Te
chno
logy
Man
ufac
turin
g
Unc
oded
sing
Bui
ldin
g &
Inte
riors
sector
0
50
100
150
200
250
Sum
No.of firms assistedApplications
1/10/2003 to 7/4/2005
Number of firms and applications per sector
Auck
land
Can
terb
ury
East
ern
Bay
of P
lent
yH
awke
's B
ayK
apiti
/Hor
owhe
nua
Kin
g C
ount
ryM
anaw
atu
Mar
lbor
ough
Nel
son/
Tasm
anN
orth
land
Ota
goR
otor
uaSo
uthl
and
Taira
whi
tiTa
rana
kiTh
ames
Val
ley
Wai
kato
Wai
rara
paan
ganu
i/Rua
pehu
/Ran
gitik
eiW
ellin
gton
Wes
t Coa
stW
este
rn B
ay o
f Ple
nty
Region
0.00
5,000.00
10,000.00
15,000.00
20,000.00
Allo
catio
n $
Allocation per assisted firm per region 1/10/2003 to 7/4/2005
41
215. The evaluation has found there is no significant difference in the rate of acceptance for different sectors.5 The average amount allocated to a firm is also similar, no matter the sector (graph below right).
216. From this analysis the evaluation can conclude that there is no bias toward any sector in the approval or decline of applications.
5 A chi-square test of independence gives a p-value of .229
Manufacturing 2,095,557.51Information, Communication &
Technology 1,247,433.88
Creative & Services 816,522.33Food & Beverage 767,636.13
Bio-technology 657,563.62Wood Processing Building & Interiors 445,792.3
Uncoded 281,061.84Education 46,126.44
12
345678
SectorTotal
Allocation $
Bio-
tech
nolo
gy
Cre
ative
& S
ervi
ces
Educ
atio
n
Food
& B
ever
age
Info
rmat
ion,
Com
mun
icat
ion
& T
echn
olog
y
Man
ufac
turin
g
Woo
d P
roce
ssin
g B
uild
ing
& In
terio
rs
Unc
oded
Sector
0.00
5000.00
10000.00
15000.00
Am
ount
allo
cate
d
Average allocation per sector1/10/2003 to 7/4/2005
Bio-
tech
nolo
gy
Cre
ative
& S
ervi
ces
Edu
catio
n
Food
& B
ever
age
Info
rmat
ion,
Com
mun
icat
ion
& T
echn
olog
y
Man
ufac
turin
g
Unc
oded
Woo
d P
roce
ssin
g B
uild
ing
& In
terio
rs
Sector
0
500000
1000000
1500000
2000000
Allo
catio
ns ($
)
Allocation per sector
1/10/2003 to 7/4/2005
42
6. Grant allocation and collection by firms
40. Amounts allocated
217. 50% of grants are for $12000 and less. 20% are for $5000 and less.
218. The graph right shows a peak of 48 grants of $3000. This is caused by the $3000 cap on two categories of activity.
219. The most frequent single amount allocated is the maximum, $20,000. There have been three grants above $20,000, but as of April 2005 these were not claimed in full.
41. Amount claimed by closed projects
220. Firms have one year from date of acceptance of their application to claim the approved costs from the project and thereby their grant from NZTE. At one year the grant and project is closed. There have been 193 projects closed to date. They have had allocated to them a total sum of $2.165M.
221. Of this $1.79M has been claimed and paid. This is a shortfall of $366,462.43 (16.9%). The shortfall is due to not all clients claiming their full amount.
222. 15% of grants have less than 60% claimed. See graph right.
223. The evaluation spoke with many of these firms and found that for most a change in plans, either for the project or for the firm as a whole, saw their project alter or not go ahead, and so they did not claim.
224. As this sort of thing seems likely to happen to firms in the future, the EDG will probably have some amount left over each year which was allocated but not claimed.
$0.00 $5,000.00 $10,000.00 $15,000.00 $20,000.00
Amount Allocated
0
30
60
90
120
150
Freq
uenc
y
1/10/2003 to 7/4/2005
Distribution of amounts allocated
0.0000 0.2000 0.4000 0.6000 0.8000 1.0000
Proportions
0
20
40
60
80
100
Freq
uenc
y
1/10/2003 to 7/4/2005
Distribution of proportions claimed
43
7. Analysis of applicant firms
42. Summary of findings given below.
225. EDG clients are typically small, with less than 10FTEs and less than $200,000 turnover.
226. Over half of all applicants are exporters, and firms who export show significantly higher rates of acceptance into the fund.
227. It is not clear whether the policy criteria of ‘young start‐up’ refers to age, and not, for example, to growth. If it applies to age, and if ‘young’ means firms less than five years old, 40% of EDG firms are not young start‐ups; if ‘young’ means less than 10 years old, nearly a quarter of firms funded are not young start‐ups.
43. Overall size: turnover and FTE
228. Most EDG clients are small, with less than 10 FTEs and a turnover of less than $200,000.
229. One goal of the EDG as a foundation service is to move its clients through to more advanced services, such as the growth services range. Almost none of the EDG clients to date have accessed the growth services range. This may be because the gap in capability between EDF clients and GSR clients is broader than can be improved via the EDF. Certainly the GS Range is targeted at high growth potential firms and this is likely to be a small percentage of the firms eligible for the EDF.
230. Looking at the graph right, the line on the y axis is at 20 FTEs, which is the maximum allowed by the fund’s eligibility criteria, and shows that NZTE will over‐ride that restriction on occasion. The lack of dots in the upper right quadrangle shows that it is overridden only if the firms are also smaller than $5M turnover.
231. The line on the x axis is at the $5M maximum, and shows that NZTE will override that restriction on occasion also, but again, it seems only if the firms are smaller than 20FTEs.
$0 $2,000,000 $4,000,000 $6,000,000 $8,000,000 $10,000,000 $12,000,000
Last full year total turnover
0
10
20
30
40
50
Full
Tim
e St
aff
1/10/2003 to 7/4/2005
FTE and Turnover of accepted firms
44
232. The graph right shows that declined firms tend to be smaller than accepted firms, but firms have been accepted who have very low turnover.
44. Exporting
233. Based on whether they report an export turnover, over half of all EDF applicants are exporters, and almost half of all accepted firms are exporting.
234. Only about 4% of New Zealand SME firms export. If firms applying to the EDG were typical of New Zealand firms, only 4% would be exporting.
235. The table right shows the numbers exporting who apply to the EDG (40.9%). This indicates that the EDG firms are not a typical population of firms, perhaps because it is a self‐selecting sample of New Zealand firms.
236. Examining the rate of acceptance and declined for exporting and non exporting firms shows that there is a bias in favour of exporting. The evaluation has found there is a significant difference between the rates of acceptance for firms who are exporting and those who are not.6
237. More firms are being declined when they are not exporting than would be expected if there were no relationship, and fewer are being accepted. Firms who are exporting are showing somewhat higher rates of being accepted, and firms not exporting somewhat lower rates.
238. It may be that firms who are already exporting are also firms who tend to pass all the EDG criteria more easily. However the EDG has been included in NZTE’s overall focus on exporting, and the criteria ‘potential for growth’ was defined to be national or export. If firms failed this they would fail the criteria as a whole, and be declined the
6 A chi-square test of independence gives a p-value of .027
$0 $2,000,000 $4,000,000 $6,000,000 $8,000,000 $10,000,000 $12,000,000
Last full year total turnover
0
10
20
30
40
50
60
Full
Tim
e St
aff
1/10/2003 to 7/4/2005
FTE and Turnover of declined firms
All applicants
120 67.458 32.6
178 100.0291 56.3226 43.7517 100.0
Not ExportingExportingTotalNot ExportingExportingTotal
Status of ProjectDeclined
Accepted
Frequency Percent
All applicants
427 59.1296 40.9723 100.0
Not ExportingExportingTotal
Frequency Percent
45
EDG. So firms who are ‘not exporting’ had at least to be those who had the potential for export.
239. The criteria for the EDG have been revised and the focus on exporting has been shifted to a focus on improved business performance and net benefit to NZ of improved business performance. The tests for the latter are improved productivity and efficiency, limited displacement (displacement would result from funding firms with significant domestic focus and significant domestic competition) and presence of spillovers.
45. Age of Firms
240. Data was provided on the year firms accepted into the EDG were established.
241. 21% of firms were established before 1995.
242. 79% of firms were established in 1995 and after.
243. 322 firms (62%) were established in 2000 and after.
244. It is not clear whether the policy criteria of ‘young start‐up’ refers to age, and not, for example, to growth. If it applies to age, and if ‘young’ means firms less than five years old, 40% of EDG firms are not young start‐ups; if ‘young’ means less than 10 years old, nearly a quarter of firms funded are not young start‐ups.
46. Firm Growth
245. The evaluation calculated the growth of firms in the years prior to applying to the EDG from data included in applications forms.
246. Growth was calculated as the change in total full year turnover from 2 years prior to applying, to 1 year prior to applying.
247. The graph below right shows the average growth rate of accepted and declined firms for the calendar years 2003, 2004, and 2005 to April. The average growth for accepted firms is between 0 and l.5%. If, as discussed above, ‘young’ is in terms of growth, then most EDG clients are young.
1914194619531964196819701973197519781980198219841986198819901992199419961998200020022004
Year firm established
0
20
40
60
80
100
Num
ber
1/10/2003 to 30/6/2005
Year established for accepted firms
46
248. For 2003 and 2004 accepted firms had higher average growth than declined firms.
249. The 2005 data is a function of low application numbers. The data the evaluation received ended in April 2005, so the data for that year is incomplete.
250. The graphs below of applications by month show that for declined firms a few had applied with high turnover, and only a few had applied who had low turnover. So a few applications with high growth have skewed the average. With further data on turnover for declined firms for the calendar year 2005, it may be seen whether the 2005 intake is similar to previous years.
251. The graphs also show that most firms have growth of less than 5%.
Jul/03 Oct/03 Jan/04 Apr/04 Jul/04 Oct/04 Jan/05 Apr/05
Month grant allocated
-10.00
-5.00
0.00
5.00
10.00
15.00
20.00
Cha
nge
in tu
rnov
er
1/10/2003 to 7/4/2005
Change in turnover of accepted firms
Oct/03 Jan/04 Apr/04 Jul/04 Oct/04 Jan/05 Apr/05
Month grant allocated
-10.00
-5.00
0.00
5.00
10.00
15.00
20.00
Cha
nge
in tu
rnov
er
1/10/2003 to 7/4/2005
Change in turnover of declined firms
2003 2004 2005
Year grant awarded
0.00
1.00
2.00
3.00
Mea
n ch
ange
in tu
rnov
er
Status of ProjectDeclinedAccepted
1/10/2003 to 7/4/2005
Change in turnover for accepted and declined firms
8. Types of assistance funded
252. Firms apply to the EDG for assistance for activities they wish to undertake. The activities must be within specified categories and types of activities. These are described further below.
253. Firms may undertake within one grant as many categories and types of activities as they wish, so activity numbers given in this section do not match firm or application numbers.
47. Advice and expertise categories
254. The types of activities which may be funded with the EDG are divided first into advice and expertise categories. The categories are strategic business development, feasibility studies, product development, business and operational excellence, e‐commerce and e‐business strategies, market strategy development.
255. The largest categories are strategic business development, product development and market development. 35% of total allocations went to product development, 27% to strategic development, 17% to market development.
256. Within each category are areas of activity. The areas are:
− strategic business development
− business/strategic plan development
− human resource strategic plan development
− financial viability planning
− strategic design advice
− feasibility studies
− product development
− prototype design, development and testing
− intellectual property protection
− business and operational excellence
Categorye-business categoryFeasibility studiesMarket developmentMentoringOperational excellenceProduct development categStrategic developmeTraining
1/10/2003 to 7/4/2005
Distribution of funding by activity type
48
− international quality standards certification
− systems evaluation and development
− environmental management system
− e‐commerce and e‐business strategies
− development of a e‐strategy
− Market strategy development
− Market research or new market investigation
− Marketing plans
257. The largest activity areas are intellectual property protection, prototype development, strategic business development, and strategic design advice (see graph below right).
258. The graph below left shows totals allocated to date. The activities within product development, prototype development and intellectual property protection have absorbed a large proportion of funding to date, at around $3.5M total allocations. Roughly one million has gone to each per annum.
Mentoring
Training
Market investigation, research
Market planning and developme
Business/strategic plan develop
Human resource strategic plan
Financial planning
Strategic design advice
Feasibility Study
Prototype development, design
Intellectual property protection
Certification
Systems evaluation and devel
Environmental manage
e-business stra
0
30
60
90
120
150
Num
ber
1/10/2003 to 7/4/2005
Areas of activity undertaken by accepted firms
e-bu
sine
ss c
ateg
ory
Feas
ibilit
y st
udie
s
inte
llect
ual p
rope
rty
Mar
ket d
evel
opm
ent
Trai
ning
Men
torin
g
Ope
ratio
nal e
xcel
lenc
e
prot
otyp
e de
vlpm
t
stra
tegi
c pl
anni
ng
stra
tegi
c de
sign
Category
0.00
1,000,000.00
2,000,000.00
3,000,000.00
Tota
l allo
cate
d $
1/10/2003 to 7/4/2005
Funding totals per activity type
PART FIVE: Findings and Recommendations on Outcomes
9. The framework for analysis of outcomes.
48. Goals of the EDG
259. The goal of the EDG was to assist innovative firms and entrepreneurs to build capability to test early stage business concepts and projects by enabling them to employ expertise and advice.
260. Funding was to enable firms to:
− Engage the services of a business mentor for a period of time
− Undertake more advanced management and technology based training (as delivered through existing providers such as NZIM, Polytechnics etc)
− Employ specific external advice and expertise in a management area (such as feasibility studies, e‐business, market research, enhancement and uptake of new technologies, human resources, intellectual property, strategic planning, environmental management, production management)
− Undertake international market development activities, including new market investigation, trade fair participation, trade/business missions, business exchanges, and visiting buyers.
49. Goals of the evaluation
261. The evaluation sought to learn whether the outcomes policy wished to achieve were occurring for the firms undertaking the activities funded by the EDF.
262. Achieving this goal enables the evaluation to report on whether the intervention designed by policy was successful. It would be successful if the issues identified by policy were overcome by the activities firms undertook, and if the outcomes identified by policy resulted from these activities.
263. To achieve its goals the evaluation held interviews with a sample of firms. The population was all firms who had completed a project by April 2005, giving a total of 191. The sample was a stratified simple random sample with a sample size chosen to give a margin of error of 5% for estimates of proportions. This gave a minimum sample of 129. The evaluation surveyed 132 firms.
264. The sample was stratified by the activity types the EDG is broken into by NZTE, and those to be included were selected by MED policy. Further details are given in section 10.
50
50. Specific outcomes sought by the evaluation
265. The evaluation sought to understand the objectives for firms in undertaking their projects.
266. It wished to know whether the projects undertaken by firms had improved business or management capability and whether firms had experienced gains in performance as a result.
267. It wished to know the rate at which projects had gains in activity (such as launching a product or entering a market) and performance (in terms of sales) with and without a change in management practice.
268. It wished to determine any differences between the different project activities in the rate at which they improved business or management capability, and the rate at which they led to gains in performance.
269. It wished to understand the nature of the projects being undertaken in each activity type.
270. It wished to know whether firms intended pursuing further assistance, and what their opinion was on the time it took to fill out the applications.
51. Interpretation of the policy intent: improving management or business capability?
271. Attempting to determine the improvements to business and management capability by the projects undertaken with funding gave rise to a definitional issue: the difference between ‘business capability’ and ‘management capability’.
272. The statement of policy intent says the goal of the fund is to build: “basic business capability” but describes this as the capability to handle: “day‐to‐day management of the many different functions of a firm, as well as longer‐term strategic issues.”
273. Day‐to‐day business and long‐term strategic issues may well be separate tasks requiring different types of skills. Improving them may give different kinds of outcomes.
274. The evaluation categorised projects into management level and business level. If the project was on an activity of the firm considered to be a function of the firm, such as building a prototype or the design of marketing material for a product, or identification of distributors in a market, it was ‘business level.’ If it were strategic, such as improving production methods or developing a marketing plan or a branding strategy, or it was taking an overview of all firm activities and strategy, it was considered to be ‘management level’.
Business Management
Business or management level
50
55
60
65
70
75
80
Num
ber
Numbers of firms wanting business or management level assistance
51
275. The evaluation looked to see whether firms wished for help with day‐to‐day business matters, or with strategic matters, or for help with taking a broad overview of the fund and its overall strategy, and if there is a difference in the outcomes for each.
52. Interpretation of the policy intent: improving management ability or knowledge?
276. Attempting to determine how and whether external expertise gave improvements to business and management capability gave rise to a second definitional issue: the difference between projects delivering advice, and projects delivering facts. The difference is also a difference in the situations and objectives of firms for their projects.
277. The ability to manage a firm can be improved by learning ways or better ways of performing the tasks required of managers. It can also be improved by getting the information managers need to make management decisions and ensure firms move ahead not with guesswork but with strategy. So via the provision of information required to achieve a project, and via the provision of the skills required, management’s ability can be improved.
278. However, there is quite a difference between a firm who needs advice on an issue or situation that is, advice on how to tackle a situation; and a firm who needs only facts to make their own decisions on what to do. The latter has abilities the former does not; the latter knows what information it needs and knows how to act on it.
279. So the distinction between gaining knowledge and gaining ability is really about types of firms‐ one sort will have higher capability than another because it has the capability to implement a project just not the information it needs to do so.
280. The distinction can be seen in a situation where, for example, a firm already knows how to enter a market, but just does not have the facts on a particular market to know whether they should do it. This is a different type of firm from one who does not know how to enter a market, nor what information they need to make a decision on entering.
281. The distinction also shows in situations where a firm needs to purchase work from externals, such as a report, because the firm needs it to be independent, or where a firm needs external people with manufacturing machinery in order to produce a product or prototype. Neither of these are situations where the problem for the firm is a lack of ability in the sense of skills; rather it is ability in the sense that they are unable to complete a project until they get the external information or technical assistance they need.
282. The policy statement is not clear on which sort of firm it was intended that the EDG assisted (if not both) because it described assisting firms with both information and skills. The policy states that (italics added):
52
Foundation services aim to build basic business capability…through the provision of generic business information, advice and training…If entrepreneurs and managers have difficulties in accessing or developing the necessary skills, this could lower economic efficiency and national productivity. Small businesses and entrepreneurs often lack the financial ability to employ all the expertise they need to get new concepts and projects up and running…As a result, small businesses often do not realise their full potential.
283. The issue then is whether the goal of capability building is to include firms who can themselves implement the project or is to be restricted to those firms who need to learn the skills to gain capability in order to implement the project. Is the fund a channel for information to managers or is it only to build their actual capability?
284. With the possibility that the distinction does make a difference in outcomes, and in order to aid policy resolve the question of whether the fund ought to be a channel for information, the evaluation has looked to see whether facts and information or advice was sought by firms, and what the difference is in outcomes.
53
10. Quantitative findings on accepted firms.
53. Guide to the findings
285. This section lays out all the findings from the research on outcomes for those interested in the evidence itself. The evaluation’s conclusions based on the evidence are laid out in the next section (13). Those who just wish to read the summary of findings and conclusions may skip this section.
286. The survey was undertaken by telephone interview. The population was all firms who had completed a project by April 2005, giving a total of 191. The sample was a stratified simple random sample with a sample size chosen to give a margin of error of 5% for estimates of proportions. This gave a minimum sample of 129. The evaluation surveyed 132 firms.
287. The sample was stratified by the activity types the EDG is broken into by NZTE, and those to be included in the evaluation were selected by MED. The stratification was to ensure by chance the sample did not exclude firms undertaking activities the evaluation wished to sample.
288. With sampling fractions which were similar in each stratum, and also very high, the data could be analysed as a simple random sample. No corrections for the finite population size were made.
289. The high sampling fraction and response rate enables the sample to be generalized to all EDG firms undertaking the same activities as the strata. The results have only limited generalisability to the population of firms in NZ as a whole due to the fact that the population of firms applying for grants is self selecting. If the firms who apply for the fund in the future are different in character or if the numbers accepted increase to a large degree, the results on outcomes found here may not continue to apply.
290. The findings are of two sorts: proportions of firms in the EDG who saw various outcomes subsequent to completing their project, and relationships between certain activities or project types and outcomes. The graphs are the count of firms reporting the outcomes.
291. The relationship is found using the statistical test: ‘chi‐square test of independence’. The Chi‐square test is a statistical procedure to test for the existence of an association between two categorical variables. An example of such a comparison would be the variable ‘advice was given’ (possible answers are yes and no) and ‘changes in management were made (possible answers are also yes or
Stratum Population Sample size
Mentoring/Training 21 21Market research and investigation 32 21
Prototype 36 22
Systems evaluation 28 22Intellectual property protection 26 12Business and strategic development 30 19
Strategic Design 18 15
Total 191 132
54
no). The Chi‐square test provides a significance or p‐value for the comparison: small values of the p‐value mean that a significant association has been found. For the purposes of this evaluation a significance level of 5% (p=.05) is used (this corresponds to the standard 95% confidence level), and any test with a p‐value less than 0.05 as taken as evidence for an association.
292. Where there is evidence for an association caution should be exercised in the interpretation of the result: the test cannot establish whether the relationship is causal or not. Lack of evidence for an association may mean there is no association, or that the association is too weak to detect with the given sample size.
54. Numbers of firms responding.
293. For two firms the appropriate people have not been able to be reached for interview. One firm refused an interview. The firm is reasonably high‐profile, so the refusal was not due to any poor performance overall. This gives a response rate of 98%
55. Numbers of firms completing projects.
294. Of the sample 124 (94%) of firms are still in business. The 7 considered not in business were unreachable on details provided by NZTE and are unlisted with Telecom NZ.
295. Not all firms completed their projects. 111 (84%) had completed their projects. 13 firms cancelled their projects part‐way through either because of changes in their plans or because the project never quite got off the ground.
296. Both numbers completing projects and those deemed exited indicate that expecting a 100% rate of impact of the fund is unreasonable. There is likely to always be firms whose plans change and who do not undertake the project.
297. Instead, expecting an 80% completion level seems reasonable; and this entails the proportion of success in achieving outcomes will be less than 80%.
56. Overall satisfaction
298. 90% of firms were happy with the EDG and NZTE. Many wanted to make clear to the evaluation how grateful they were for the assistance. These firms also mentioned the length of the application process but felt it was justifiable given they were seeking public funds.
299. 10% of firms had some specific issue with NZTE or the EDG process which left them disgruntled. For the majority it was either the length of the applications or the inflexibility of the EDG process in its position on alterations to the project mid‐way. Some had tried to get such changes allowed and found the process exhausting.
55
300. Being disgruntled with the process often coincided with disappointment in the project they had undertaken. When the gains were small firms minded the time they had spent applying.
301. The overall satisfaction level is a good result for the EDG and NZTE.
57. Proportions of EDG clients seeing changes in the way they manage their business.
302. The evaluation looked at the proportions of firms reporting changes in the way they manage the business.
303. 56% of firms saw changes in the way they managed the business (see graph right, note it gives counts of respondents not overall percentage).
304. 30% of firms said projects had no effect on management. These firms typically had undertaken projects which were not geared at impacting on management.
305. The changes described by those who had seen changes included altering the internal practices of the firm, learning from the project about ways to tackle such projects, gaining direction for the firm and sometimes confirming that their approach had been right. Firms also reported an improvement in their overall ability to manage the business.
306. No firms reported projects having detrimental impacts on the way the firm was managed.
307. The evaluation found significant differences between the rates of effect on management for different activity types. A chi‐square test of independence gives a p‐value of p=.004. Mentoring, training, marketing and business planning had high rates of effect. Prototype had especially low rates. This means that usually firms undertaking mentoring, training, marketing and business planning projects will see impacts from these projects on their management.
Yes NoChange
0
20
40
60
80
Num
ber
Numbers of firms with changes in the way they manage
56
58. Proportions of EDG clients undertaking projects aimed at management level areas or business level areas
308. The evaluation categorised projects by whether they were focused on management level activities or on business level activities and looked at the proportions undertaking each.
309. 47% of firms were undertaking projects which were at management level and 41% were undertaking business‐level projects.
310. Of those firms who undertook management level projects, 82% said they had seen changes in management, and 17% said they had not. Of those who were undertaking business level projects, 42% said they had seen changes to management practices, and 51% said they had not.
311. The evaluation found a significant association between the proportions of firms undertaking business‐level or management level‐projects, and activity type undertaken by firms. A chi‐square test of independence gives a p‐value of p<.001. Mentoring and training, business planning, systems evaluation and strategic design had high rates of management level projects. Intellectual property and prototype had low rates.
312. This means that one can be confident that some activities‐ intellectual property and prototype‐ will usually not be undertaken by firms who want management level assistance, but instead business level. The other activities will usually be
Business Management
Business or management level
50
55
60
65
70
75
80
Num
ber
Numbers of firms wanting business or management level assistance
activity * Firm on whether it affected management Crosstabulation
12 4 1610.5 5.5 16.0
75.0% 25.0% 100.0%5 6 11
7.2 3.8 11.045.5% 54.5% 100.0%
16 2 1811.8 6.2 18.0
88.9% 11.1% 100.0%15 2 17
11.1 5.9 17.088.2% 11.8% 100.0%
7 13 2013.1 6.9 20.0
35.0% 65.0% 100.0%7 4 11
7.2 3.8 11.063.6% 36.4% 100.0%
12 8 2013.1 6.9 20.0
60.0% 40.0% 100.0%74 39 113
74.0 39.0 113.065.5% 34.5% 100.0%
CountExpected Count% within activity_cdedCountExpected Count% within activity_cdedCountExpected Count% within activity_cdedCountExpected Count% within activity_cdedCountExpected Count% within activity_cdedCountExpected Count% within activity_cdedCountExpected Count% within activity_cdedCountExpected Count% within activity_cded
Business/strategic plan
Intellectual propertyprotection
Marketing
Mentoring/Training
Prototype
Strategic Design
Systems Evaluation
activity
Total
Yes No
Firm on whether itaffected management
Total
57
undertaken by firms who want management level assistance. Whether these activities fit in the EDG depends on whether the goals of the fund are building ability in the sense of management ability, or in the sense of business ability.
59. Proportions of EDG clients seeing gains in firm activity
313. The evaluation looked at the proportions of firms reporting changes or gains in such activities as getting a product ready to launch or launching it, or understanding the firm’s market better, or gaining a strategy, or entering a new market.
314. 78% of firms achieved the results described above, so it is estimated that 78% of EDG clients have used the grant to make positive effects on their firm. This is a good result for the EDG and NZTE.
315. Furthermore, firms who report changes in management and firms who do not both see effects on firm activity.
Level of assistance * activity_cded Crosstabulation
6 11 9 1 15 4 8 54
7.9 5.1 8.8 7.9 9.3 5.1 9.8 54.0
11.1% 20.4% 16.7% 1.9% 27.8% 7.4% 14.8% 100%
11 0 10 16 5 7 13 62
9.1 5.9 10.2 9.1 10.7 5.9 11.2 62.0
17.7% .0% 16.1% 25.8% 8.1% 11.3% 21.0% 100%
17 11 19 17 20 11 21 116
17.0 11.0 19.0 17.0 20.0 11.0 21.0 116.0
14.7% 9.5% 16.4% 14.7% 17.2% 9.5% 18.1% 100%
CountExpectedCount% withinLevel ofassistanceCountExpectedCount% withinLevel ofassistanceCountExpectedCount% withinLevel ofassistance
Business
Management
Level of assistance
Total
Business/strategic
plan
Intellectualproperty
protection MarketingMentoring/Training Prototype
StrategicDesign
SystemsEvaluation
activity_cded
Total
58
Gains in activity
7 5.3
39 29.5
2 1.5
18 13.628 21.29 6.8
103 78.0132 100.0
Developing productAssists withtargeting/gainingcustomers/investorsBasic businessrequirement now in placeImproved performanceEntering MarketGave firm clear strategyTotal
Valid
Total
Frequency Percent
60. Proportions of EDG clients seeing changes in firm performance
316. The evaluation looked at the proportions of firms reporting changes in performance in terms of domestic or export sales.
317. 39% of firms saw changes in their firm’s performance, with increased domestic or foreign revenue. 37% said it was too early to see any results in revenue. 2% said the project had increased productivity.
318. As 84% of firms undertook the project, gains in performance for 39% entails nearly half of firms who make use of the grant have seen gains in performance to date.
319. The evaluation found no significant association between the proportions with effects on performance and different types of activity. A chi‐square test of independence gives a p‐value of p=.090
320. The table shows intellectual property protection had very high rates of firms reporting gains were yet to eventuate, and market research and investigation were high also. Intellectual property protection is a long and expensive process, and does not directly lead to gains in performance but for many firms is necessary for them. Market research and investigation also takes time to result in gains, and many firms received the advice or information they need but were yet to enter the market or were yet to see sales.
321. Systems evaluation and strategic design had high rates of reporting gains in terms of sales, with business planning and mentoring being fairly high. Typically these firms were undertaking projects which they were able to get to market more quickly or see gains more quickly. Prototype was evenly split between reporting gains and reporting gains were yet to eventuate, as firms were fairly evenly split in stages of prototype development.
Increased productivity
Expanded domestic/export
sales
Not yet
Change
0
20
40
60
80
Num
ber
Numbers of firms with changes in firm performance
59
61. Proportions of EDG clients gaining facts or advice
322. The evaluation looked at the proportions of firms who gained advice or assistance and at the proportions gaining information (such as facts on a market) or the use of specialist facilities.
323. 50% of firms gained advice. 38% gained facts or specialist facilities.
324. The evaluation found a significant association between the proportion of firms gaining facts or advice and the activity type undertaken by the firm. A chi‐square test of independence gives a p‐value of p<.001. Mentoring, training and strategic design had high rates for advice; intellectual property and prototype had high rates for facts or specialist skills such as manufacturing skills or legal skills (patent attorneys).
Advice Facts Legal Grant used for internal
costs
Specialist technical skills or facilities
Type of assistance sought
0
20
40
60
80
Num
ber
Numbers of firms wanting facts, advice or specialist skills
Effect on the firm performance * activity_cded Crosstabulation
0 0 0 0 1 0 2 3.4 .3 .5 .5 .6 .2 .5 3.0
.0% .0% .0% .0% 33.3% .0% 66.7% 100%
8 2 6 9 9 5 12 517.4 5.4 8.4 7.9 9.4 4.0 8.4 51.0
15.7% 3.9% 11.8% 17.6% 17.6% 9.8% 23.5% 100%
7 9 11 7 9 3 3 497.1 5.2 8.1 7.6 9.0 3.8 8.1 49.0
14.3% 18.4% 22.4% 14.3% 18.4% 6.1% 6.1% 100%
15 11 17 16 19 8 17 10315.0 11.0 17.0 16.0 19.0 8.0 17.0 103.0
14.6% 10.7% 16.5% 15.5% 18.4% 7.8% 16.5% 100%
CountExpected Count% within Effecton the firmperformanceCountExpected Count% within Effecton the firmperformanceCountExpected Count% within Effecton the firmperformanceCountExpected Count% within Effecton the firmperformance
Increasedproductivity/strengthenedposition
Expandeddomestic/export sales
Not yet
Effect on the firmperformance
Total
Business/strategic
plan
Intellectualproperty
protection MarketingMentoring/Training Prototype
StrategicDesign
SystemsEvaluation
activity_cded
Total
60
Whether firm wanted advice on what to do, or not. * Firm on whether it affectedmanagement Crosstabulation
52 14 6643.2 22.8 66.0
78.8% 21.2% 100.0%
22 25 4730.8 16.2 47.0
46.8% 53.2% 100.0%
74 39 11374.0 39.0 113.0
65.5% 34.5% 100.0%
CountExpected Count% within Whetherfirm wanted adviceon what to do, or not.CountExpected Count% within Whetherfirm wanted adviceon what to do, or not.CountExpected Count% within Whetherfirm wanted adviceon what to do, or not.
Advice
No
Whether firmwanted advice onwhat to do, or not.
Total
Yes No
Firm on whether itaffected management
Total
62. Do facts or advice projects impact on management practice?
325. The evaluation looked at whether there was any association between having facts or advice and seeing changes in the way firms managed their business, or whether these two things are independent of each other.
326. The evaluation has found a significant association between the proportion of firms with an effect on management, and having had advice. A chi‐square test of independence gives a p‐value of p<.001.
327. An association was found between advice and changes in management practice, in particular the rate of reporting the project affected management was higher for groups receiving advice. This means that one can be confident that firms having advice will usually also see changes in the way they manage their business. Firms asking for facts will usually not see any changes.
Whether firm wanted advice on what to do, or not. * activity_cded Crosstabulation
10 0 10 17 4 11 14 669.7 6.3 10.8 9.7 11.4 6.3 11.9 66.0
15.2% .0% 15.2% 25.8% 6.1% 16.7% 21.2% 100.0%
7 11 9 0 16 0 7 507.3 4.7 8.2 7.3 8.6 4.7 9.1 50.0
14.0% 22.0% 18.0% .0% 32.0% .0% 14.0% 100.0%
17 11 19 17 20 11 21 11617.0 11.0 19.0 17.0 20.0 11.0 21.0 116.0
14.7% 9.5% 16.4% 14.7% 17.2% 9.5% 18.1% 100.0%
CountExpected Count% within Whetherfirm wanted adviceon what to do, or not.CountExpected Count% within Whetherfirm wanted adviceon what to do, or not.CountExpected Count% within Whetherfirm wanted adviceon what to do, or not.
Advice
No
Whether firmwanted advice onwhat to do, or not.
Total
Business/strategic plan
Intellectualproperty
protection MarketingMentoring/Training Prototype
StrategicDesign
SystemsEvaluation
activity_cded
Total
61
328. One can be confident that firms reporting impacts on management will usually have also received advice. Firms reporting no impacts will usually have used the project to purchase information or specialist skills.
63. Are facts or advice projects also management or business level projects?
329. The evaluation looked at whether there was any association between gaining facts or advice and gaining management or business level assistance. The evaluation has found a significant association between the proportion of firms having advice or not and having management‐level or business‐level assistance. A chi‐square test of independence gives a p‐value of p<.001. In particular firms receiving advice had high rates of undertaking management level assistance; whereas firms receiving facts had higher rates of undertaking business‐level projects.
330. This means that one can be confident that firms having advice will usually also have management level assistance. Firms gaining facts will usually not.
64. Do management level or business level projects impact on management practice?
331. The evaluation categorised projects by whether they were focused on management level activities or on business level activities, and then looked at whether there was any association between the two types of projects and seeing changes in the way firms managed their business, or whether they are independent of each other.
332. The evaluation has found a significant association between the proportion of firms having had management‐level or business‐level assistance and reporting impacts on management. A chi‐square test of independence gives a p‐value of p<.001. In particular firms receiving management‐level assistance had high rates of reporting
Whether firm wanted advice on what to do, or not. * Level of assistance Crosstabulation
8 58 6630.7 35.3 66.0
12.1% 87.9% 100.0%
46 4 5023.3 26.7 50.0
92.0% 8.0% 100.0%
54 62 11654.0 62.0 116.0
46.6% 53.4% 100.0%
CountExpected Count% within Whetherfirm wanted adviceon what to do, or not.CountExpected Count% within Whetherfirm wanted adviceon what to do, or not.CountExpected Count% within Whetherfirm wanted adviceon what to do, or not.
Advice
No
Whether firmwanted advice onwhat to do, or not.
Total
Function StrategicLevel of assistance
Total
62
impacts on management; whereas business‐level had high rates of reporting no impacts on management.
333. This means that one can be confident that firms undertaking management level projects will usually also see changes in the way they manage their business. Firms undertaking business level projects usually will not.
65. Do changes in management practice impact on firm performance?
334. The evaluation has found no significant association between firms who reported management changes and those who did not and those reporting effects on performance (in terms of domestic or export performance). A chi‐square test of independence gives a p‐value of p=.170. Firms seeing changes in management and firms seeing no changes had similar rates of gains in firm performance (Note that this is rate of gain not degree of gain. The evaluation does not have data on turnover subsequent to the project’s implementation with which to calculate actual turnover change).
335. This means that gains in firm performance are seen both by firms who changed management practice and by firms who did not. Projects deliver benefits to firms in terms of gains in performance independently of whether they deliver gains in management practice.
Level of assistance * Firm on whether it affected management Crosstabulation
23 28 5133.4 17.6 51.0
45.1% 54.9% 100.0%
51 11 6240.6 21.4 62.0
82.3% 17.7% 100.0%
74 39 11374.0 39.0 113.0
65.5% 34.5% 100.0%
CountExpected Count% within Level ofassistanceCountExpected Count% within Level ofassistanceCountExpected Count% within Level ofassistance
Business
management
Level of assistance
Total
Yes No
Firm on whether itaffected management
Total
63
66. Do advice or facts projects impact on firm performance?
336. The evaluation has found an association very close to significant between projects delivering facts or advice and those reporting effects on performance (in terms of domestic or export performance). A chi‐square test of independence gives a p‐value of p=.053. This means that the association is tentative, and may differ with further evidence in time. The table shows that firms undertaking projects delivering advice have high proportions of firms also seeing either increased productivity or expanded sales. Projects delivering facts show lower proportions of firms of increased productivity or expanded sales.
Effect on the firm performance * Firm on whether it affected management Crosstabulation
3 0 32.0 1.0 3.0
100.0% .0% 100.0%
37 14 5134.2 16.8 51.0
72.5% 27.5% 100.0%
29 20 4932.8 16.2 49.0
59.2% 40.8% 100.0%
69 34 10369.0 34.0 103.0
67.0% 33.0% 100.0%
CountExpected Count% within Effect onthe firm performanceCountExpected Count% within Effect onthe firm performanceCountExpected Count% within Effect onthe firm performanceCountExpected Count% within Effect onthe firm performance
Increasedproductivity/strengthenedposition
Expandeddomestic/export sales
Not yet
Effect on the firmperformance
Total
Yes No
Firm on whether itaffected management
Total
Whether firm wanted advice on what to do, or not. * Effect on the firm's performance Crosstabulation
2 34 21 571.7 28.2 27.1 57.0
3.5% 59.6% 36.8% 100.0%
1 17 28 461.3 22.8 21.9 46.0
2.2% 37.0% 60.9% 100.0%
3 51 49 1033.0 51.0 49.0 103.0
2.9% 49.5% 47.6% 100.0%
CountExpected Count% within Whetherfirm wanted adviceon what to do, or not.CountExpected Count% within Whetherfirm wanted adviceon what to do, or not.CountExpected Count% within Whetherfirm wanted adviceon what to do, or not.
Advice
No
Whether firmwanted advice onwhat to do, or not.
Total
Increasedproductivity/strengthened position
Expandeddomestic/export sales Not yet
Effect on the firm's performance
Total
64
67. Do management or business level projects impact on firm performance?
337. The evaluation found a significant association between the proportion of firms where there was an effect on firm performance and projects assisting with management or business type projects. A chi‐square test of independence gives a p‐value of p=.023. In particular, projects assisting with management‐level activities showed a higher rate of gains in firm performance in terms of export and domestic revenue and improved productivity.
338. Business‐level projects had higher rates of revenue gains being yet to eventuate, and consistent with this had a significantly higher rate of firms subsequently launching products, and enabling firms to target market segments and attract prospective investors. A chi‐square test of independence gives a p‐value of p<.001.
339. This means that one can be confident that firms undertaking management level projects will usually also see gains in their performance. Firms undertaking business level projects will not.
340. Note, however, there is a caveat to this finding. Firms undertaking business level projects usually said they had not yet seen changes in gains as it was too early for this given the project they undertook. Time is needed to see if the result above holds. It may not if many of the business level project firms see gains in performance in, say, the next six months.
Effect on the firm performance * Level of assistance Crosstabulation
1 2 31.4 1.6 3.0
33.3% 66.7% 100.0%
17 34 5123.8 27.2 51.0
33.3% 66.7% 100.0%
30 19 4922.8 26.2 49.0
61.2% 38.8% 100.0%
48 55 10348.0 55.0 103.0
46.6% 53.4% 100.0%
CountExpected Count% within Effect onthe firm performanceCountExpected Count% within Effect onthe firm performanceCountExpected Count% within Effect onthe firm performanceCountExpected Count% within Effect onthe firm performance
Increasedproductivity/strengthenedposition
Expandeddomestic/export sales
Not yet
Effect on the firmperformance
Total
Business ManagementLevel of assistance
Total
65
11. Quantitative findings: interpretation and conclusion
341. The evaluation was interested in seeing whether the EDG had resulted in changes and gains in management practice and in activities of the firm; whether these resulted in gains in performance, and whether there was any difference in outcomes for activity type, and for firms undertaking projects delivering advice, facts, management level, or business level assistance.
68. Overall
342. 84% of firms went ahead with the project; the remainder saw changes in plans which meant they no longer wished to undertake the project.
343. 78% of firms have seen benefits from the grant, in terms of having a product ready for launch or launching it, understanding which segments of the market to target, improving the way the firm runs, gaining or clarifying a strategy for the future, or entering a new market. This means that of the 84% of firms who complete a project, 92% see positive gains from it. This is a very good result for the EDG.
344. 56% of firms saw changes in the way they managed the business; 30% of firms said projects had no effect on management‐ but these firms typically had undertaken projects which were not geared at impacting on management. 47% of firms gained management level assistance and 41% gained business‐level assistance.
345. 39% of firms have seen gains in firm performance. 37% said it is too early yet to see gains in performance. This was usually because while the grant was closed the project was taking longer than one year. The evaluation recommends a second look at the sample in six to twelve months, for if the 37% also see gains in performance it will raise the success rate of the fund significantly.
346. The difference between the roughly 50% of firms who gained in terms of management capability and the nearly 80% who have gained in terms of the activities they used the fund for is due to ambiguities in the original policy statement resulting in a broad interpretation by NZTE of building ‘management capability’.
347. Firms are undertaking projects which in a broad sense improve their management capability; but not in a narrow sense. Roughly 50% of firms had no intention when undertaking the projects for it to have any changes to or have any effects on management. So in terms of the fund’s capability objectives the success rate of the EDG is 47% for self‐reported changes and 56% for changes which have been categorised as at a management level.
348. 50% of firms use the fund to gain advice, and 38% use it to gain facts or specialist facilities.
66
69. Activity types
349. All activity types were equal in their rates of subsequent gains in firm performance in terms of domestic or export sales.
350. The types of activity differ in the rates at which they result in change in management practice. Mentoring, training, marketing projects, and business planning showed significantly higher rates of change in the way firms manage their business. This means one can be confident that these activities will usually have firms changing their management practice.
351. Activity type was also related to undertaking management level projects. All but intellectual property and prototype had high rates of being used for management level projects, with mentoring and training being especially high. This means one can be confident that these activities will usually have firms gaining management level assistance.
352. The activity types of intellectual property protection and prototype development had very low rates of firms reporting changes in management, undertaking management level projects or gaining advice. It seems these are not well fitted to the goals of the EDG.
70. Interpreting the results: improving management capability
353. The goal of the EDG is improve firm performance via an improvement in management and business capability.
354. The evaluation defined projects as business or management level, and looked to see the impacts of each project type. The evaluation has found a significant association between the proportion of firms having had management‐level or business‐level assistance and reporting impacts on management. This means that one can be confident that firms undertaking management level projects will usually also see changes in the way they manage their business. Firms undertaking business level projects usually will not.
355. The evaluation also found that firms undertaking projects defined as management level had significantly higher rates of gain in performance. This means that one can be confident that firms undertaking management level projects will usually also see gains in their performance. Firms undertaking business level projects will not.
356. Note the caveat to this finding. Firms undertaking business level projects usually said they had not yet seen changes in gains as it was too early for this given the project they undertook. Time is needed to see if the result above holds. It may not if many of the business level project firms see gains in performance in, say, the next six months.
357. Currently the proportions of firms using the EDG to undertake management level or business level projects are fairly even. To target the fund more squarely at management level projects the assessment of proposals and possibly the
67
description of the funds goals to the market, would need to be revised. This would require a definition from MED for NZTE of at least the core of what ‘management level’ is (even if the edges of the distinction with ‘business level’ are left somewhat blurry).
358. As the fund is undergoing change with its devolution, due for piloting 1 December 2005, there is time to assess the sample again, to coordinate any changes with final implementation of the devolved model in mid 2006.
71. Resolving the ambiguity between facts and advice.
359. The evaluation found there is an issue in whether the fund is a channel for information to managers or whether it is to build their capability. At present, two types of firms are accessing the fund: those who can themselves implement the project but need information to do so, and those who need to gain the capability to implement their project. It may be that the fund’s goal of capability building is achieved best by including firms who can themselves implement the project or by being restricted to those firms who need to learn the skills to gain capability in order to implement the project.
360. The evaluation defined projects as delivering advice to build capability, or delivering information to firms with the requisite capability, and looked to see the impacts of each project type. The evaluation has found a significant association between the proportion of firms having advice or not and having management‐level or business‐level assistance and seeing changes in management practice or not. In particular firms receiving advice had high rates of receiving management level assistance and seeing changes in management practice; whereas firms receiving facts had higher rates of receiving business‐level assistance and not seeing changes in management practice. This means that one can be confident that firms having advice will usually also gain management level assistance and usually see changes in management practice. Firms gaining facts will usually not see such changes.
361. As management level projects see higher rates of gain in performance, it seems aiding management via advice is the best way to aid management to achieve gains in performance.
362. The evaluation also found a tentative association between receiving advice and seeing gains in performance. Firms undertaking projects delivering advice have high proportions of firms also seeing either increased productivity or expanded sales. Projects delivering facts show lower proportions of firms gaining increased productivity or expanded sales.
363. Thus the fund could be at least weighted toward firms wishing to gain advice. This can be done by adjusting the assessment to spot these sorts of firms, and possibly the description of the fund to the market.
364. The evaluation must still make the same caveat as before, that time may eliminate these differences. Again, with the final implementation of changes to the fund due
68
mid 2006, there is time to test these possibilities by assessing the sample again, before any further changes are made to the fund.
72. Declined firms
365. The evaluation rang a simple random sample of declined firms. The sample size was 58. The evaluation contacted 70% of the sample. All who were contacted spoke to the evaluation, giving a 70% response rate. The remainder of the sample was not able to be located.
366. 38% of the declined EDG firms went on and did the project. 33% did not.
367. It may be that the firms who were declined but undertook their projects nonetheless did so because they had the resources to do so and this was the reason for them being declined. The evaluation has no information on why the firms were declined with which to confirm this supposition because many were declined prior to those records being kept.
368. The firms who had been declined and had not undertaken the project viewed that as a real loss.
369. Many who were declined felt they had been brushed off, could not understand it, and were left unhappy with NZTE and the grant process. Many said they would not reapply. This suggests that the goal of the fund to encourage firms to seek assistance must be considered a goal for those firms who are declined as well, and the process of declining firms should not discourage firms from seeking further assistance. More personalized and encouraging explanations of the reasons for being declined will help prevent the feeling of being ‘brushed off.’
73. Recommendation
370. The evaluation recommends that a process of detailing more fully the reasons for declining firms is instigated.
371. It is recommended NZTE establishes a system of storing electronically data from the application forms on all applicant firms’ financial and capability needs, taken from their description in the application form of the project and their goals for the external advice or expertise. This should be able to be compared to final outcomes from the project funded by the grant, and would place both an evaluation in the future and NZTE in a stronger position to assess changes in the firms post intervention. MED should be involved in establishing this system.
69
12. Qualitative findings
74. Guide to the findings
372. The findings in this section are qualitative. They are drawn from the sample of 132 firms. They are not statistically representative of outcomes of the EDG, or the activities occurring, that is, one cannot generalize from these statements to the EDG as a whole. The findings are to give an understanding of the range of situations, goals and experiences of the firms in the sample.
75. Overall
373. Firms said that the fund was a significant leg‐up to achieving their goals, and often gained them the results they were after in a very much shorter time frame than they felt they could have done on their own. So while it is not certain that firms would not have done the project and achieved their goals without the grant, the firms were sure they achieved them faster. Many also said they were able to do the project to a much higher standard than otherwise.
76. Business/strategic plan development and feasibility studies.
374. The situations of firms undertaking this type of activity were mainly a wish to expand into new markets, advice on the firm and its performance, or to have help commercialising a product.
375. The projects were mostly genuine efforts at tackling skill‐gaps, that is, the firms did not have the skills to the plans for the business themselves. Those not genuine included a firm using the grant to fund a report it needed written by an external, independent party, in order to secure investors. Another firm needed help in gaining FDA approval: they needed agents in the US who were proficient at organising and managing that process, as well as an independent chemical analysis of their product.
376. The latter is a grey case, because without agents who know the process a firm would be lost in the FDA bureaucracy and the firm did not have that knowledge. They said they now know how to pull together the necessary documentation, and who the right people in the US, and could help other companies in NZ do it.
377. Seven firms needed facts with which to decide how and whether to proceed with their plans. Three needed assessments of markets in order to decide whether to launch in that market, and two successfully launched as a result.
378. Firms who requested and received management capability help included a firm who wished for a long‐term strategy after a few roller‐coaster years with major international clients. They felt they were a small company who had struck gold early on but didn’t know what business planning was or how they should handle
70
down‐times between clients. As a result of their project they have developed a global plan as well as having analysed their company’s strengths and potential.
379. Other firms wished for help in improving their performance. They gained new ways of thinking about their firm as well as specific things to work on to improve performance. For one firm the outcome was a fundamental restructure‐ in direction, branding, and new relationships with external researchers, as well as by gaining a board of directors and shareholders. The former manager now operates the business and leaves its management to his new board.
77. Intellectual property protection.
380. Protecting i.p varied in its importance for firms. I.T and bio‐tech firms view it as essential. Software is easily copied once it is launched as a product, so software firms must have i.p protection. Firms in the pharmaceutical industry cannot get partners or investors without them being assured they have rights over any product they make, for development costs so much, rights at the end of development must be assured. In that regard firms say they don’t have a business without i.p protection. Other firms view i.p as only as useful as they are capable of affording to defend it. Firms said i.p was also useful for letting them know that their idea really was original and that they were right to pursue it.
381. Two firms explained that the volumes they need to produce their products at, either to meet the volume of product demanded by the market, or to meet the low costs demanded by the market, means they cannot manufacture in New Zealand. The manufacturing industry is both too small and too expensive. They said manufacturing offshore entails they must have global patents, else they will loose their product to rival offshore manufacturing firms.
382. Gaining global patents usually costs firms more than one hundred thousand dollars, and the EDG does not go far toward that, although firms were grateful for any and all assistance they had with such large costs.
383. I.p protection is not something which easily fits with the policy goals of improving management skills or abilities. Firms already knew whether they wanted or needed I.p For some i.p lay within a larger strategy of selling or manufacturing offshore, but this strategy was already developed.
78. Strategic Design
384. Projects under this type of activity included projects with a narrow focus, such as on aspects of product design, and a broader focus, on branding for the company as a whole.
385. Product design projects included such things as developing owner’s manuals and instructional DVDs, assistance with design, logos, wording and marketing material for the product. Projects also included technical advice on technology
71
projects, and in these cases the difference between strategic design and prototype development is slim.
386. Branding projects include one undertaken by a new firm still getting their product into production but who believed their product would sell on image, and so wanted the ‘feel’ of the company to be right. The firm had already spent money on this but the grant gave work a leg‐up and meant the firm moved ahead more quickly. It had a product ready for production in 12 months.
387. Another firm had been to a branding seminar and decided to re‐brand their existing firm, and said they would never start a firm again without going through a branding exercise first. They felt it was very important for marketing and success in general.
79. Market research and marketing plans
388. Some of the marketing projects were done by firms who did not know how to approach entering a new market, and needed someone who knew what information to gather and what the firm would need to do. Some of these firms felt they had learnt what to do from these projects and could now enter any market on their own.
389. Other firms knew the specific information on a market they needed, but many of these firms said they would not try to do that sort of research on their own.
390. Firms who knew what sort of information they needed said they would not feel comfortable researching a market themselves. It is a skill that some managers said they will continue to access externally while they remain too small to have an employee with such skills. One firm said for a small business it is better to contract the task out to huge experience in profiling a market, and so not be gobbled up and spat out by big players in the U.S. To not do this is false economy.
391. Firms who did not know how to develop a plan to enter a market felt they could learn such a skill. Many of these firms were also undertaking marketing work under mentoring.
80. Prototype
392. Most of the firms developing prototypes used the grant to purchase facilities for production they did not have or the technical skills needed to build the prototype their firm did not have. Two firms used the grant to fund internal costs and did not use external providers.
393. Only three of the firms used the grant to purchase advice on developing their prototype.
72
81. Mentoring and training
394. This type of activity is the one most geared to building management ability, and so has seen the greatest impact on it. Some firms have had a complete turnaround in performance and described the results as ‘mind‐blowing.’
395. Firms have also decided to keep the mentor on, or have established a board of directors for ongoing help.
396. There is some overlap with the marketing activities, some firms used the training or mentoring to develop marketing plans, but the difference is that via training and mentoring they learn both how to develop a plan and develop one for their firm.
82. Certification and systems evaluation
397. Certification was of two different types‐ compulsory and optional certification for a product, which required an assessment of the product, and compulsory and option certification of the firm, which required an analysis of procedure and documentation.
398. Some firms knew how to prepare the firm for the certification, and the grant went to their costs or the costs of the evaluation of the firm. Others did not, and the grant went to the costs of the consultants used.
399. Systems evaluation ranged from a review of the systems of the firm to development of products. One firm in fact built a prototype: the firm needed funding for the project but due to the requirement for external assistance they hired consultants to build the prototype when they had the skills in house. The results were not as good as they would have liked and they had to do a fair bit of re‐development.
400. A firm who had systems reviewed had the ‘seat‐of‐the‐pants’ systems they had built over the years reviewed and altered. The consultants documented responsibilities and removed uncertainty on roles and procedures. As a result the firm improved its performance, especially delivery time and saw revenue growth as a result.
73
Appendix: Survey Questions
401. These are the questions used for the survey, and all firms were asked these questions, but all the interviews involved asking further questions to better understand the situation of the firm, the project and the outcomes. These varied from firm to firm as required.
1. OK. to begin with, if you could think back to before the _______assistance was decided upon, when you were originally thinking to seek help, what sort of help were you after, and how did you arrive at _________?
2. So thinking about the project itself, what were your specific goals for it?
3. So you had received approval for the grant, and called in the external people you had chosen, what happened next? what did the external people do?
4. Can you describe what sort of effects the project has had on your business, if any?
5. Have you seen any impact of this on your business’ performance
6. Can you think of any ways in which the project has influenced the way you manage your business?
7. Do you think you will seek any more assistance for your business?
8. On balance, do you think what has resulted from this, then, was worth the time spent on applying?