Key Success Factors Jan 2003 - Semantic Scholar · effect of key success factors on the commercial...

Key Success Factors for R&D Project Commercialization

Thomas Åstebro Department of Management Sciences

University of Waterloo Waterloo, ON N2L 3G1, Canada [email protected]

tel (519) 888 4567, ext. 2521 fax (519) 746 7252

January, 2003

ABSTRACT

I examine the potential impact of 36 innovation, technology and market characteristics that may determine, at an early stage assessment, the probability that R&D projects will later reach the market. Analysis is based on data from 561 R&D projects developed outside of established organizations. Administrative records contained subjective evaluations of 36 project characteristics conducted at the start of the projects by project-independent outside analysts. A mail survey collected an objective measure of whether the projects were eventually commercialized or not. Four characteristics stand out as most predictive: expected profitability, technological opportunity, development risk and appropriability conditions. These key variables predict future commercial success well with an out-of-sample forward prediction accuracy of 80.9%. This model performs better than R&D managers’ predictions of their own R&D projects’ technical success and has the potential to be used as a screening tool for early stage R&D investment reviews. Key words: early stage R&D project evaluation, probability of commercialization I acknowledge financial support from the Natural Sciences and Engineering Research Council of Canada and the Social Sciences and Humanities Research Council of Canada’s joint program in Management of Technological Change, the Canadian Imperial Bank of Commerce and in-kind support from the Canadian Innovation Centre.

Key Success Factors for R&D Commercialization - 2 -

“One of the main advantages of trying to use formal selection and

estimating techniques in research and development is not that significant,

precise quantified results emerge, but rather that complex situations are

shown to be capable of logical analysis…” K. H. Binning quoted in

Twiss (1986, p. 119)

1. Introduction

Researchers disagree strongly on how an appropriate R&D project evaluation

model should be constructed and how data should be assessed (Balachandra and Friar

1997). Some models are strictly empirical and are based on statistical analysis of the

correlation between project characteristics and project success (e.g., Cooper, 1981).

Others have proposed evaluation models based on operations research principles (e.g.,

Souder, 1973). Both approaches have benefits as well as shortcomings. The former may

suffer from vaguely measured variables and from the difficulty of combining both

financial and non-financial variables on a common scale. The latter has been found to be

too complex to be of much practical use (Liberatore and Titus, 1983; Souder, 1973).

A third approach is to use discounted cash flow (DCF) analysis. In DCF, financial

success is driven by a few well-defined financial parameters. The outcome of the analysis

is a number that carries common meaning, such as the internal rate of return. The

problems with using DCF analysis have been the requirement of an understanding of

basic finance, the difficulty of finding quantitative information and the uncertainty

associated with the data. The third problem has been dealt with by assigning probabilities

to represent the uncertain nature of technological as well as commercial success (e.g.

Mansfield et al. 1977, Twiss 1986, p. 137, Graves and Ringuest 1991). A more recent

development has been to integrate option theory into R&D project evaluation (e.g.,

Herath and Park 1999; Pries et al., 2002).


Irrespective of the recent advances of the DCF method, R&D project management

research focused in the 1980’s on using statistical approaches to investigate ”key success

factors” that would explain R&D project success. In their review of more than two

decades of such research Balachandra and Friar (1997) conclude that a wide number of

studies on the topic have been unable to converge on any key success factors, mainly due

to grave methodological problems. Due to these problems research shifted gears in the

1990’s towards managing the innovation process (for a review see Brown and Eisenhardt,

1995), without ever resolving the basic question of what the key innovation and project

characteristics are that explain R&D project success.

Recent literature on managing the innovation process has therefore been criticized

for an undue focus on the process of project selection while not addressing the decision

criteria necessary to select appropriate projects (Cooper, 2000; Murphy and Kumar,

1997). Cooper (2000) summarizes recent research as addressing “how to do projects

right, without knowing which are the right projects”. Murphy and Kumar (1997) are

critical as well, stating that “improving the development process downstream while

neglecting upstream choices may be a fruitless exercise”.

Wary of these critiques I therefore revisit the fundamental question regarding the

key success factors for early stage R&D project success. My intent is to address most if

not all of the methodological problems that have been plaguing past research. With this

work I hope that researchers will be able to start building an understanding and

appreciation of appropriate statistical methodologies to be used for research in this area. I

also expect that the research will spawn the development of a set of models that map out

how various R&D project characteristics affect project success in various settings. The

research builds on past developed theory.

As Balachandra and Friar (1997) discusses, it is unlikely that there exists a single

model of R&D project success that generalizes to all types of R&D projects. Rather, their

suggestion, which I have followed in this study, is to develop models for different classes

of R&D projects and different decision-making situations to obtain better knowledge and


predictive accuracy. In this study I therefore focus on the assessment of early stage R&D

projects – the stage where minimal R&D funding has been expensed and the project can

be considered an “idea” or “concept” (Murphy and Kumar, 1997). I further limit the

scope and complexity of analysis by only examining the likelihood that the concept will

reach the market. This is an important intermediate stage for achieving financial returns

and includes both technical success as well as acceptance in the marketplace.1 But I do

not include in the measure the degree to which the project provides financial returns.

Performing an analysis of the financial returns implies a simultaneous analysis of several

outcome measures such as the duration of sales in the market conditional on commercial

acceptance, the cash flow net of operating costs in each year of sales conditional on

commercial acceptance and the probability of commercial acceptance. Further research is

necessary to examine such simultaneous and nested models.2

I examine the impact of a comprehensive set of 36 innovation and product-market

characteristics on the likelihood that an early stage R&D project will be commercialized.

These characteristics include several dimensions of technological opportunity,

competition, legal and societal conditions and twelve dimensions characterizing user

need and market demand. The examined characteristics, while comprehensive in their

coverage of technical, business and market conditions, do not include any organizational

dimensions as the projects that I study are conducted outside of established organizations

or later spawn organizations that were not in existence at the time of project evaluation.

This limitation should be kept in mind when implications are drawn. As Balachandra and

Friar (1997) suggests it is unlikely that these results generalizes to all R&D projects.

However, I will make the argument that the results are not inconsistent with past research

on evaluating early stage R&D within large corporations. I will further argue that the

1 Technical and commercial success are sometimes analyzed separately (e.g. Mansfield et al.

1977a). 2 See Åstebro (forthcoming) for an example of such analysis.


results are likely to generalize to other early stage R&D projects started outside of

established organizations.

I measure the characteristics at a very early stage of the projects’ launch before

substantial R&D investments have been made. I follow the projects’ development over

time and measure the outcomes – commercialized or not – after the projects have been

completed. I sampled a group of 561 R&D projects that were submitted by individuals

for technical and commercial evaluation to the Inventors Assistance Program (IAP) at the

Canadian Innovation Centre (CIC) in Waterloo, Ontario, Canada. I compared the ex ante

subjective evaluations of the projects by analysts at the IAP on 36 decision criteria with

the ex post probability of commercial success of these 561 projects.

To resolve previous methodological problems identified by Balachandra and Friar

(1997) I use a large sample of observations in relation to the number of predictors

investigated and therefore avoid problems associated with incorrectly accepting the null

hypothesis (of no relationship) due to low power of the test (Rosenthal and Rosnow,

1991). I collect data using standard and accepted sampling and survey techniques. The

definition of project success is objective (not subjective) and is easily replicated. Since

measures of predictors of success are taken at an early stage of the development of the

projects, whereas success is measured after the projects are completed and independent of

the collection of data on predictors, I avoid two methods bias: hindsight bias (Fischhoff,

1975) and common method variance bias (Campbell and Fiske, 1959).

The research contributions are twofold: a) this is the first statistical analysis on the

effect of key success factors on the commercial success for early stage R&D projects

generated outside of established organizations, b) to my knowledge this is the first large-

scale study of key success factors where three methods biases are avoided: hindsight bias,

common method variance bias and low statistical power.

The remainder of this paper is organized as follows. Section two surveys literature

related to R&D project evaluation and selection. Section three describes the method and


data. Section four contains results and section five concludes with both research and

management implications.

2. R&D Project Evaluation Research

There has been a great deal of statistical research on the determinants of R&D

project performance using various analytical approaches. The research falls into three

perspectives: 1) research on factors leading to success (Cooper, 1984; Goldenberg et al.,

2001; Yoon and Lilien. 1985; Voss, 1985); 2) research on factors leading to failure

(Hopkins, 1981); and 3) research on factors that separate success from failure (Abratt and

Lombard, 1993; Cooper, 1979; Cooper, 1985; Maidique and Zirger, 1984; Yoon and

Lilien, 1985). In general, these studies suggest normative strategies to enhance success or

avoid failure and have provided considerable evidence that a great number of factors can

influence the outcomes of R&D projects. The factors studied describe various

combinations of product characteristics, development processes, organizational

characteristics, strategic and market characteristics. Reviews can be found in Balachandra

and Friar (1997), Lilien and Yoon (1989) and Linton et al., (2002). Rather than re-

reviewing this extensive literature I provide a few examples and then proceed to draw

conclusions based on the excellent meta analysis by Balachandra and Friar (1997).

In the most comprehensive study to date, Cooper (1981) analyzed 195 new product

projects to compare/contrast success (102) and failure (93). A factor analysis on forty-

eight variables was conducted to generate a smaller and more manageable subset of

predictors. Thirteen factors were identified and they explained 69.3 percent of the

variance of the original forty-eight variables. A total of seven of the thirteen factors were

significantly related to perceived project success at least at the 0.10 level. These were (in

decreasing order of significance): product superiority and uniqueness, project/company

resource compatibility, market need/growth/size, economic disadvantage to consumer,

newness to firm, technological resource compatibility and finally market


competitiveness. The model had an R2 of 0.42 and an overall prediction accuracy of 84.1

percent, and performed well in a naïve split-sample test.

Goldenberg et al. (2001) argue that certain innovation-level technical solution

attributes are identifiable at project start, are generally observable and also highly

predictive of market success. They find strong empirical support for these claims. They

also find that when the technological opportunity is identified first (or concurrently with

market need) there is a higher probability that the innovation will be financially

successful. Goldenberg et al.’s study have two main constraints: a) the predictors are

assessed ex post of innovation and project completion which makes it subject to potential

methods biases, and b) they only analyze innovations that reach the market and so can

only discriminate between high and low revenue products.

While Goldenberg et al. (2001) revisit the important issue of the effect of

innovation-specific characteristics, they find little common agreement with other studies

as to the relevant “success factors” for R&D projects (Balachandra and Friar, 1997).

Indeed, performing a meta analysis of 60 papers Balachandra and Friar found

contradictory results and little stability of success factors across studies: there seems to be

no clear agreement even on the direction of influence of the specific factors analyzed, let

alone their significance, if any.3 However, aggregating up to a sufficiently generic level

of analysis, researchers seem to agree that the following classes of variables are

important: market, technology, environment and organizational (Balachandra and Friar,

1997; Lilien and Yoon, 1989; Linton et al., 2001), with some disagreement as to the

importance of additional important classes.

3 Balachandra and Friar (1997) went on to isolate only one study from an author, rather then

assessing several papers from the same author(s) based on the same data, and further deleted those studies with little empirical content or results. This did little to clear up the confusion. Balachandra and Friar found that among 72 compiled significant factors across nineteen studies (which typically used factor analysis), half of the significant factors were unique to specific studies and about 75% of the final factors were identified in just one or two studies. Even for similar factors their meaning and interpretation were often not the same because of the differences in context across the studies.


A similar lack of convergence is reached in the literature on the decision criteria

used by venture capitalists when assessing venture deals. Zopounidis (1994) summarizes

this literature with two conclusions: “the first is that the criterion of the management team

is considered predominant in all the studies concerning decisions in venture investment

and the second is the great diversity of evaluation criteria and their relative importance

(ranking of criteria) from one study to the other” (p. 63). As an example, Bacchher,

(2000) investigated the factors deemed important to judge by leading U.S. and U.K. VCs

when assessing seed and early stage technology-based ventures. A multitude of criteria

were examined (over 100), with the six most important groupings being (in order of

stated importance): management team, market demand, product offering, competitive

position, return on investment and the business plan. A problem with most of these

studies is that they do not relate the criteria given by the VC’s to an outcome measure.

There is thus no way to know whether the factors espoused by VCs to be important are

truly important.

The value of using statistical support tools for R&D decision making has been

clearly illustrated by Zacharakis and Meyer (2000) who investigated the ability of VCs to

accurately assess the future success of a venture seeking investment. The 51 interviewed

practicing U.S. VCs had on average over ten years of VC experience and over 22 years of

work experience and focused on seed and early stage deals. Zacharakis and Meyer

conducted an experiment where the VCs received several pieces of information about 25

actual investments that had subsequently achieved either success or failure.

Approximately 57% of the investments were in the seed and early stages. The VCs were

requested to evaluate the ventures as they would during the initial screening stage. Their

predictions were then compared to the actual outcomes and the percentage of correctly

classified outcomes was computed.

This study show VCs to have a low ability to correctly forecast the outcomes of

ventures -- at best the VC’s had a classification accuracy of approximately 40%.

Perplexingly, the more information about the venture that was provided to the VCs the


less able they were at correctly predicting outcomes. When information about the track

record of the team and competition was included the classification accuracy reduced to

31% and when additional information about the team and product was introduced the

classification accuracy declined to 17%. These results indicate that VCs are rather poor at

making investment decisions and that more information makes them more confused and

less accurate evaluators.

Zacharakis and Meyer then compared the ability of the VCs to the forecasting

accuracy of a simple statistical model that used the same information as the VCs to make

predictions. The classification accuracy of the statistical models ranged from 40% to

60%, always clearly surpassing the judgments made by the VCs. These preliminary

results are very encouraging. They indicate that substantial improvements are possible in

the screening stage of investment decisions by using statistical models.

To create a reliable statistical model for R&D decision support one should avoid

previous methodological mistakes. Balachandra and Friar (1997) identify four major

sources of weakness in previous research on R&D key success factors, namely quality of

data, the definition of a new product, factor selection and definition, and measurement of

factors. I will discuss these issues in some detail as my aim of this paper is to resolve

most of the empirical problems in order to arrive at a set of predictors that are stable in

test-retest situations.

Most studies on the determinants or factors influencing the success of R&D

projects have been conducted by simultaneously collecting information on independent

and dependent variables after the projects have been completed (e.g. Cooper, 1981;

Lilien and Yoon, 1989; Maidique and Zirger, 1985; Yap and Souder, 1994). These

studies therefore suffer from both common method variance bias (Campbell and Fiske,

1959) and hindsight bias (Fischhoff, 1975). Since data on independent and dependent

variables are collected at the same time by the same method from the same respondents

the measured associations are larger than the true associations. In addition, recollection of

conditions after the fact overstates causal relationships because there is a decided human


tendency to over-attend to information consistent with one's hypothesis and to disregard

contradictory information (Dawes et al., 1989; Fischhoff, 1975).4

Another serious problem in previous work on predicting the outcome of R&D

projects is the relatively sparse amount of observations in relation to the number of

predictors that have been used. Nunnally (1978) writes: "A goof rule is to have at least 10

times as many subjects as variables." (p. 421) and Cliff (1987) suggests: "With 40 or so

variables, a group of 150 persons is about the minimum, although 500 is preferable." (p.

339). Previous analyses of factors associated with the success of R&D projects all fail to

reach the preferable sample size and only Cooper (1981) reaches the absolute minimum,

as defined by Cliff (1987).5 This failure creates instability of results in test-retests and

partly explains the lack of convergence on a stable set of “success factors”.

Balachandra and Friar (1997), in addition, report confusion about the measurement

and interpretation of the dependent variable: project success.6 Cooper and Kleinschmidt

(1987) identify three measures of success – financial performance, opportunity window

and market share. Lilien and Yoon (1989) add another dimension, the length of the

project's development process. A study by Griffin and Page (1993), illustrates the

divergence of views about how to define successful projects. Summarizing their meta

analysis approach Balachandra and Friar states that “Since there is no common measure

of success, and success is a composite of a number of subjective and objective measures,

we have used success as defined by the individual authors of the studies”, (p. 277).

4 A typical example of such methods bias is the study by Maidique and Zirger (1984) where

managers were asked to select a pair of innovations, one success and one a failure, and then asked to determine, for each innovation, whether a particular variable related to the outcome, (positively or negatively) or not.

5 Maidique and Zirger (1985), for example, estimate the relationship between 60 independent variables and the outcome of R&D projects using 59 respondents, where the highest number of observations reported for any variable was 52 and the lowest was 24. It is well known that a multivariate analysis requires about 20 observations per variable to be accurate. The above study fails dismally on this point, and the authors resort to bivariate associations, which are less useful as they do not inform of the relative importance of any one variable and an overall model is not possible to construct.

6 There is also confusion in the use and meaning of success factors (Balachandra and Friar, 1997). This confusion is partly a result of using factor analysis with limited sample sizes, which produces unstable results, and possibly due to the lack of instrument validation, rather than confusion, per se, among the researchers.


An additional issue identified by Balachandra and Friar (1997) is the wide variety

of R&D projects and new product development projects studied. This is not necessarily a

methodological failure, as they suggest, but related to the inherently large variance in the

types of R&D projects conducted. The large variance reduces coefficient estimates unless

one is able to control for this variation by including control variables that distinguishes

across the various types of projects included. This is unlikely to be completely successful

as much of the variance may be undetectable. One might instead choose to study a more

well-defined subset of projects to generate more reliable but less general results.7 Quite

possibly there are no generic success factors as they might depend on the stage of the

review and other conditions. Balachandra and Friar (1997) suggest three contextual

variables that condition which variables that are important: the nature of innovation

(incremental versus radical), market (existing versus new) and technology (high versus

low).

There is an additional problem in all previous studies in that information on

predictors of success is invariable collected on so-called Likert scales. This means that

the respondent has to make a perceptual judgment as to the importance of a particular

variable, for example the importance of "market size" on a scale with tick-marks, rather

then providing objective data on the size of the market. This procedure is somewhat

defendable due to the difficulty of collecting objective data on a large set of variables

through a post-outcome survey. Nevertheless, several well-known methodological

problems arise. What does, for example, a "7" on a ten-point scale mean? When a

7 Balachandra and Friar also lament the bias towards equal representation of successful projects in

the studies they reviewed although in reality there are likely to be nine failures for each success (Griffin 1997). The approach typically used is to ask for one successful and one failed project from each firm. A matched sample is thus obtained with a mean probability of success around 0.5. This does not necessarily lead to bias in other parameter estimates, as implied by Balachandra and Friar. Indeed, Maddala (1983, pp. 90-91), shows that if one draws separate samples from two populations only the constant term changes, not the slopes. If one knows the true frequency of successes in a population it is merely a matter of rescaling the constant term to find the population-level intercept. However, I agree that researchers have rarely considered bias in parameters to be of critical concern. The generation of samples seems to have been driven more by convenience then by an attempt to provide generalisable results (e.g. Maidique and Zirger, 1984).


statistical model is estimated, what will the estimated coefficient for the variable mean?

And how can one compare data across raters and across variables when the scales are not

anchored on objective facts? Tests of inter-rater reliability can only provide comfort in

our belief that higher scores are indeed higher than lower scores. Luckily, Einhorn

(1972), among others, shows that expert judgmental scores on predictors are quite useful

in predicting outcomes, even in very difficult decision situations. In fact, statistical

models based on judgmental scores perform better then experts' judgmental predictions

based on the same underlying judgmental scores (Dawes et al., 1989). When asked to

provide information, albeit judgmental, on various characteristics one at a time, experts

indeed produce valuable information. Experts, however, face the greatest challenge when

combining together these pieces of information to form an overall judgment. Armed with

this knowledge one can claim that collecting judgmental data on various potential

predictors from well-informed individuals is useful, but asking these individual about

their overall judgment of the outcome (e.g. project success) is less useful.

I now summarize this review and draw conclusions as to the design of a study on

R&D project success. I consider one of the major deficiencies in previous studies on

predictors of success to be that relationships may be severely misrepresented by the

method of collecting data on independent and dependent variables at the same time and

after the fact (Campbell and Fiske, 1959; Fischhoff, 1975). This study therefore relies on

data about independent variables collected before the projects started and on data about

outcomes after the projects finished. I am also concerned that previous results are biased

by non-random sampling techniques and will therefore pay close attention to the

sampling methodology. In addition, I am concerned that previous analyses have been

hampered by low sample sizes leading to unreliable results. I therefore select a “large

enough” sample that satisfies statisticians (e.g. Cliff, 1987). Finally, I am concerned that

previous research has not clearly defined the type of projects studied and the dependent

variable measuring success, leading to confusion about how to interpret results. I


therefore spend some time clearly defining the sample and provide an objective and

easily interpretable measure of success.

3. Sample Selection and Data

Sampling Strategy

Four criteria governed sample selection.

1. I am interested in determining the probability of an R&D project reaching the

market when assessed at project start. To obtain unbiased coefficient estimates

this goal requires data on project characteristics to be collected at or reasonably

close to project start.

2. I want to avoid bias in judgment associated with project evaluators being

associated with the projects.

3. I would like to focus on a subset of R&D projects for which stable key success

factors can be determined.

4. The sample size should be large (at least a factor of ten) in comparison to the

number of criteria used to assess the R&D projects.

I used the Inventors’ Assessment Program (IAP) at the Canadian Innovation Centre

(CIC) as the main source for R&D project data. The CIC represents an organization

independent of the R&D projects they evaluate and therefore is not likely to suffer from

bias of judgment associated with individuals directly involved in the projects. The data

thus meets the second sampling criteria.

The IAP collects data on invention and product-market characteristics at a very

early stage of the projects’ development thus fulfilling the first selection criterion. Data

on predictors are collected on average two years before the (successful projects) reach the

market. To further illustrate the early stage of the review, the average accumulated out-

of-pocket R&D expenditures at the time of evaluation are Cdn $9,853 (1995 value). The

average accumulated out-of-pocket R&D expenditures for those reaching the market


were on the other hand Cdn $54,513 (1995 value). More data on the characteristics of

these projects are available in Åstebro (1998). Over 13,000 R&D projects have been

evaluated since the inception of the IAP program in 1976.

The IAP evaluates strictly R&D projects undertaken by individuals, often called

"independent inventors". Independent inventors conduct their efforts outside the confines

of established organizations.8 The sample therefore fulfils selection criteria three.

Incidentally, this sample is unique. There has been little analysis of the characteristics of

independent inventors’ R&D projects and to my knowledge there have not been any

analysis of these projects’ key success criteria.9

To have a project evaluated, the inventor/entrepreneur fills out a questionnaire and

pays a fee. The fee for an independent inventor in 1995 was Cdn. $262 (about U.S.

$185). The evaluation is confidential. In addition to background information about the

entrepreneur, a questionnaire by the IAP to the inventor asks for a brief description of the

idea. It asks questions regarding the idea, and asks for supplementary documentation such

as patent applications, sketches, and test reports. The questionnaire also asks about

market information, manufacturing, product costs, and the entrepreneur’s skills, plans and

professional goals. In comparison with models of R&D project’s success in established

organizations (Cooper, 1981), this review does not consider organizational factors since,

8 It is important to recognize that projects developed by independent inventors may have

characteristics that distinguish them from R&D projects undertaken in large established organizations. The major difference may be the need for independent inventors to obtain resources from the market, rather then internally as is the norm for projects developed in large organizations. This primary difference raises the issue of potential liquidity and other resource constraints that might adversely affect such projects (Åstebro, 1998). However, except for their observed lower probability of success, projects developed by independent inventors are not as different to those developed in large established organizations as one is often led to believe in the popular press. Indeed, inventions that are patented by independent inventors are technically no different in terms of their degree of novelty and their degree of detail in their specification and are as likely to have their patent fees maintained as inventions patented by established firms (Dahlin et al. 2002). Inventions patented by independent inventors do appear, however, to have a narrower scope of application than inventions patented by established firms (Dahlin et al. 2002). It has also been observed that projects developed by independent inventors have out-of-pocket costs (excluding the inventor’s own labor costs) that are about one eighth of project costs in established firms while those projects that succeed have gross margins as large as R&D projects in established firms (Åstebro, 1998).

9 For research on independent inventors see Albaum, (1975), Udell (1989), Udell et al. (1993), Parker et al. (1996) Åstebro (1998), Åstebro and Bernhardt (1999), Åstebro and Gerchak (2001), Dahlin et al. (2002) and Åstebro (forthcoming).


in an overwhelming majority of cases, there is not yet an organization to evaluate. The in-

house analyst – typically an engineer - compares the project to other similar projects and

searches various on-line databases such as patent and trademark databases. The analyst

then subjectively rates the project on 37 criteria.10 Apart from rating the 37 criteria the

analyst also subjectively determines an overall score for the project which is based in

some undisclosed way on the ratings of the 37 criteria. Analysts then have a group

meeting where the evaluating analyst presents a summary and a final overall score is

agreed upon. The evaluation process typically takes five to seven hours. A report to

inventors basically contains the overall score, ratings on each criteria (with explanations)

and a few comments on commercialization options (if applicable).

Data

For each invention submitted to review by the IAP a file number was assigned to

indicate the project's submission year and month. I selected as the sample frame all

projects reviewed from the start of the program until and including 1993. Projects review

after 1993 were excluded as I wanted to ensure that outcomes were observable in 1996

when outcome data were collected. I found the physical record of the evaluation

information for 8,797 projects conducted during 1976 to 1993 including ratings for each

of the 37 early stage characteristics. Data on the independent variables were consequently

collected before outcomes were observed and independently of this study. I therefore

avoid any potential methods bias (Rosenthal and Rosnow, 1991).

I selected a random sample of R&D projects from each year between 1976 and

1993. Using a CD-ROM of Canadian residential addresses 1,826 records were updated

with current addresses. This number represents 21% of the sample frame. I could not

reject the hypothesis that the updated records were a randomly selected subset of the

sample frame across the years of submission (χ2=0.19, d.f.=16, n.s.).

10 Thirty-three of these criteria were developed by Gerald Udell at the Oregon Innovation Center in

1974 as critical for venture success (Udell, 1989; Udell et al., 1993), and were used at Waterloo from the start in 1976. In 1989 the CIC introduced a revised list with four more criteria.


To collect data on the outcomes of the projects I followed the total survey design

method outlined by Dillman (1978). This involved pre-tests of the survey instrument on a

sample of inventors and detailed reviews of the instrument by analysts at the IAP. The

telephone survey method was chosen for its ability to generate high response rates and for

greater control of the data collection process (Lavrakas, 1993). Individuals recorded by

the IAP as responsible for the projects were first mailed a letter informing that a call

would be placed. Telephone calls were made primarily in evenings to residential

telephone numbers during an eight-week period in the spring of 1996. Among the 1,826

sampled records, 1,465 were from inventors who could be reached and asked to

participate in the survey. We obtained 1,095 responses, representing a response rate of

75%.11

The inventions in this sample are in general not “high-tech” but represent rather

modest technological improvements. A plurality of the projects are consumer oriented

(47%), most for household and general consumer use (28%), followed by sports and

leisure applications (15%). A list of successful inventions reviewed by the IAP includes a

new milk container design, an impact absorbent material sewn into the back of a T-shirt

for hockey players, a meat tenderness tester, and a toilet tissue holder. However, there is

also a reasonable fraction of “high-tech” (6%), and industrial equipment (6%) inventions

(CIC 1996). The same list of innovations includes an industrial-strength crusher of

recycled cans, a new method for repairing worn feed rolls in sawmills, a re-usable plug to

insert in wooden hydroelectric poles after testing for rot, and a computerized and

mechanically integrated tree harvester. The inventions submitted to the IAP for review

11 I was concerned that I would not be able to observe enough successes to estimate meaningful

models. I therefore included a subset of projects where analysts at the CIC had information indicating that the invention might have reached the market. Analysts had obtained this information through various sources such as newspaper clippings. There were 75 additional observations included this way..The addition of the choice-based observations did not change the distribution of observations across the overall ratings for the full 1976 – 93 sample (χ2=6.39, d.f.=4, n.s.). Neither did the addition of the choice-based observations change the distribution of observations across ratings for the 1989 – 93 sub-sample (χ2=6.34, d.f.=4, n.s.). Given that there are no changes in the underlying distribution of data with the addition of the choice-based observations, there will be no bias in regression parameter estimates, only a change in the constant (Maddala, 1983: pp. 90-91).


are at their stage of inception; none has reached the marketplace, and they range in

development efforts from brief sketches to working prototypes. Some are in the process

of patenting, very few have already patented their invention. It is also fairly obvious from

reading the descriptions of the inventions that they are not radical. Radial inventions

come very seldom in any way.

An overwhelming majority of respondents are male (89 per cent) and from the

Province of Ontario (72 per cent). A number of tests were conducted to establish that the

variation in sampling and response proportions across the years of submissions, provinces

in Canada, gender, and rating were random: that is, no selection biases were detected (for

details contact the author). The background demographic characteristics of this sample

correspond well to other samples of independent inventors (Albaum, 1975; Parker et al.,

1996). The sample contains very few multiple project submissions from the same

inventor: over a period of sixteen years, 1,044 inventors made one submission, 21 made

two submissions, and three provided three submissions. It is therefore safe to assume that

projects are independent, which simplifies the statistical analysis.

The dependent variable: project commercialization, was defined in the mail

survey.12 There can be many definitions of “project success” (Cooper and Kleinschmidt,

1987). Balachandra and Friar (1997) deplore the lack of uniformity in the use of

measures of success in the literature. The one I employ, however, is easily

operationalized, easy to replicate across studies, does not depend on a subjective

evaluation, and is certainly a necessary but not a sufficient condition for financial

success.

A change in both the scaling of variables and variable composition in July 1989

necessitated the deletion of records prior to this date.13 The resulting analysis data set for

12 I asked the inventor “Did you ever start to sell <NAME> or a later revised or improved version of

this invention?” [Yes=1, No=0]. 13 The change in period covered and the inclusion of the choice-based sample did not have a

significant effect on the distribution of the probability of success. A comparison of the distributions of both the absolute number of successes and the proportion of successes across the ratings in the 1976 - 93 sample and the 1989 - 93 sample produced no significant differences (χ2=7.78, d.f.=4, n.s. and χ2=4.13, d.f.=4, n.s.,


the period 1989-93 consisted of 561 projects containing 499 failures and 62 successes,

large enough to comply with standard requirements for multivariate analysis (Cliff,

1978).

The 37 criteria were graded on a three-point scale, i.e. A (Acceptable, which means

that the criteria appears to be favorable or satisfactory), B (Borderline. The criteria rated

as B needs to be improved or strengthened.), and C (Critical weakness, meaning it may

be necessary to discontinue the effort to commercialize the project.) For the purposes of

the statistical analysis I converted the scores on the underlying criteria into numerical

data according to the following: A = 5; B = 4; and C = 3. (It is immaterial for the analysis

which particular values that are assigned to the letters.) All variables are scaled so that a

higher value is thought to represent a higher technical and/or commercial value. Positive

coefficient estimates are therefore expected.

Because the analysts used a fixed format for entering data on the criteria (a sheet

where all criteria were listed) there were few missing observations. Nevertheless, lack of

data on approximately 43% of the observations for one criteria (“Service: Will this

innovation require less servicing or less costly servicing than alternatives?”) resulted in

that criteria being eliminated from further analysis. There were 108 missing observations

among the remaining 20,196 cells (36 variables*561 observations) representing 0.53

percent. Missing data were imputed to the mean. The 36 criteria used in this study

together with their definitions (as used by the IAP) are listed in Appendix A.

While this work is not geared towards testing a priori hypotheses it is nevertheless

useful to examine whether the 36 criteria that will be analyzed have face validity for

R&D project evaluation.

respectively). Similar comparisons between the 1989 - 93 random sample and the 1989 - 93 sample augmented with the choice-based observations produced no differences (χ2=1.18, d.f.=4, n.s. and χ2=4.07, d.f.=4, n.s., respectively.) However, the reduction in the sample to cover only 1989 – 93 resulted in a marginally significant change in the distribution of total number of observations across the overall ratings (χ2=8.94, d.f.=4, p<0.10). The change reflects the increased learning at the IAP and may indeed be reflected in slightly different parameter estimates.


V1, V2 and V4 relate to the potential technological improvement of the invention.

V23 is very similar to V2 – both a measure of product quality. V3 describes the

remaining development cost. V8 relates to (technological) resource availability. V9 and

V10 consider tooling and production costs. The above mentioned variables, together with

V33 which considers R&D uncertainty, might be grouped under the rubric “technological

opportunity” (Cohen, 1995). V32, V5 and V6 examine potential external constraints. The

variables V11 to V16 consider various measures of demand and need while V18 through

V22 are derived from the study of the diffusion of innovations concerning various market

characteristics (Rogers, 1983). V24 is an additional demand measure. Price is included

(V26) as well as two measures of competition (V27 and V28). Three additional cost

measures follow (V29-V31) while V34 examines system integration issues. V35 is a

measure of appropriability conditions (Cohen, 1995) and V37 to V40 considers various

investment criteria. The variables have face value. Except for organizational aspects, they

cover the main groups of variables identified by Balachandra and Friar (1997) and Linton

et al. (2002) and are particularly detailed on technology and demand characteristics. All

of the 36 criteria are covered in one way or another in the review by Balachandra and

Friar (1997).

4. Results To determine the key success factors of R&D project commercialization I estimate

a logistic maximum-likelihood model for a binary choice outcome,14 starting with the

exclusion of all variables and where the most significant variables are successively

included and for each inclusion the model is re-estimated (so-called forward selection).15

14 Even though I decided to use the logistic specification, a number of link functions are available

when outcomes are discrete (see Agresti, 1990). Rather than arbitrarily selecting one function, four alternative link functions were explored: logit, normit (also called probit), and gompit (also called complementary log-log). Regressions showed that all three types generated qualitatively similar results. As the results were robust to alternative specifications, I selected the logistic for further analysis as this provides log-odds ratios that are directly interpretable.

15 I experimented with other procedures and selection criteria to judge the robustness of results. The variables selected for inclusion using this procedure were identical to the ones selected for inclusion using a


A p-value of 0.05 is used to determine the inclusion of variables. That is, the final model

only contains predictors that are significant at the 5% level. This procedure allows me to

retain only those factors which are significant and drop all other unimportant variables.

The point of using this method instead of retaining the full model is that the resulting

model is easier to understand and relate to for a practicing R&D manager. The method is

also consistent with the research tradition in this field (e.g. Cooper, 1981). Finally, if the

model is to be used in practice it is important that it is robust and not overly sensitive to

the data on which it was estimated. The full model will likely fare less well in such

robustness tests as it takes into account the effect of a multitude of mainly insignificant

variables. The research method applied here is geared towards generating a practically

useful tool for R&D project evaluation rather than to test a priori hypotheses.

Following the forward selection process, the final result consisted of only four

independent variables. These are, in decreasing order of statistical significance: V40

(Expected Profitability), V33 (Development risk), V35 (Protection) and V23 (Function),

with the following parameter estimates (p-values in parenthesis):

Y = 3.105*V40 + 2.429*V33 + 1.778*V35 + 1.969*V23

(0.000) (0.000) (0.009) (0.019)

where Y = the odds-ratio of success.16 The model obtained a psuedo-R2 = 0.21. This

model correctly predicts 444 out of the 561 projects (79.1%). The model correctly

predicts 398 of the 499 failures (79.7%) and predicts 46 of 62 successes correctly

backward selection procedure with the exception of V23 (Function), which was replaced by V2 (Functional Performance). V23 and V2 are practically interchangeable predictors and have almost identical prediction accuracies. The model with V23 was selected by tossing a coin. A forward selection with a p-level of 0.10 did not cause more variables to be included. The four key success factors are therefore robust to the choice of the particular cut-off point for inclusion.

16 The “odds-ratio” is defined as follows. The odds of an event are calculated as the number (or probability) of events divided by the number (or probability) of non-events. In this particular data set the mean odds of success are 62/499= 0.124. The odds of failure are therefore 8:1. The odds ratio is calculated by dividing the odds in the treated or exposed group by the odds in the control group.


(74.2%). Interestingly, the model is about equally adequate at predicting successes as it is

predicting failures while the number of successes is an order of magnitude less.

The size of the odds-ratio for V40 “Expected Profitability” is the largest –

meaning that it has the greatest importance on the odds of success of all four variables.17 .

A one-unit increase in the variable V40 from “B” – borderline to “A” – acceptable,

means a 3.1 unit increase in the odds-ratio. The interpretation of this result is straight-

forward: “A”-rated inventions on “Expected Profitability” have odds 3.1 times higher to

reach the market than “B”-rated inventions on “Expected Profitability”. As the mean

odds of failing are 8:1 the impact is substantial.

I further test the model’s accuracy using an out-of-sample forward test. I split the

data-set with 561 observations in two, the first containing 383 projects that were

evaluated between 1989 and 1992, and the second containing 178 projects evaluated

during 1993. I estimate a model on the 1989-92 data set. I then analyze how well the

coefficient estimates from that data predict the outcomes of projects evaluated during

1993. In this test it is more difficult for a model to perform well than in than a random

split sample (as was used by Cooper, 1981), since a forward prediction period requires

the model to be consistent over time and not only across a random selection of

observation in a given time period.

As expected, the model on the subset of data from 1989 to 1992 is somewhat

different than the model for the overall data set. This model included only three

significant variables: the two previous V40 and V23 and the additional V13 (Trend of

Demand). Most of the prediction power is however generated by V40 and V23. As this

model is optimized on a subset of data it is likely to have lower classification accuracy

for the hold-out sample than what the model for the complete data set would have.

Nevertheless, the model correctly classified 80.9% of all out-of-sample projects, with

17 Odds-ratios are more straight-forward to interpret than the coefficients from a logit probability

model, but are directly comparable through the transformation: )exp( ii βϕ = , where iϕ is the odds-ratio

for the i’th coefficient iβ .


correctly classifying 60.9% of the successes and 83.7% of the failures. In this test the

ability to correctly classify successes has deteriorated, whereas the ability to correctly

classify failures has improved providing a minute increase in the classification accuracy.

Clearly the model has merit as a classifier when applied to new data.

The recommendation by the IAP is likely to have an impact on whether the

inventor continues or not (Åstebro and Chen, 2002). There are two potential effects. The

advice may shift the baseline odds of commercialization by increasing the odds that a

good project is pursued and decrease the odds that a poor project is pursued. These

effects have no impact on coefficient estimates as they affect only the intercept. Åstebro

and Chen, (2002) examine the magnitude of this “treatment” effect in detail and find that

it shifts the baseline probability of success up (for positive advise) or down (for negative

advise) on average about two percentage point. A second concern is that the evaluation

by the IAP may bias the relationship between the key success criteria and outcomes, for

example by having an inventor improve on certain key weaknesses singled out by the

IAP. This problem is difficult to address with the available data and generally requires

complicated econometric models that often rests on un-testable assumptions (see e.g.,

Heckman and Vytlacil, 2000). To the extent possible I consider this issue using reduced-

form models.

First, I examine the correlation between the underlying 36 criteria and the analysts’

rating of the project, and compare the results with those obtained from the statistical

model predicting commercial success. If analysts use the appropriate predictors of

commercial success as identified by the statistical model it is difficult to claim that they

do not use the appropriate key success factors when making their judgments.

An ordinal logistic regression model is fitted with the overall rating as the outcome

variable. The overall rating has five values that represents the advise given to the inventor

and is ordered from most promising to least promising (for details see Åstebro and

Gerchak, 2001). Using a p-value of 0.05 to determine inclusion of predictors, the

resulting model has a pseudo-R2 of 0.38 and contains 11 of the possible 36 explanatory


variables.18 With variables identical in predicting commercial success and analysts’

overall ratings underlined, the variables predicting analysts’ ratings are, “Expected

Profitability”, “Protection”, “Technical feasibility”, “Development risk”, “Potential

sales”, “Distribution”, “Size of investment”, “Function”, “Appearance”, “Legality”, and

“Duration of demand”. This test shows that experts at the IAP pay significant attention to

the variables indicated by the statistical model as most important in predicting

commercial success. However, they also take into account other information when

making their overall recommendation.

I then run a regression where the predicted value of the overall rating is inserted in

the logistic regression for predicting commercial success. The predicted value is

generated from the ordinal logit model of the overall rating. If the experts are using the

key success factors appropriately I would expect that there would be no remaining

explanatory power of the key success factors once the predicted overall rating is included.

To (statistically) identify the first-stage equation in the second stage I also include four

dummy variables, one each for the years 1990, 1991, 1992 and 1993 as well as a dummy

variable for the location of the inventor, where I assigned the value unity if the inventor is

located in the Province of Ontario and zero otherwise.19 These are likely to be

independent of the key success factors and are reasonable choices for identification

purposes.

When including the predicted overall rating and using a variable inclusion criterion

of 0.05 there are no other variables included. The predicted overall rating generates a

pseudo-R2 of 0.24 which is three percentage points greater than the model with the four

success criteria. The former and latter regressions together suggest that the overall rating

provides similar type of information as the four key success factors. While I cannot

parametrically investigate the potential degree of bias that is produced by running a

reduced-form single-stage model of the key success criteria this analysis at least shows

18 The same variables appear, albeit in different orders, if I use backward or forward selection. 19 So-called over-identification would otherwise occur in the second stage equation.


that the key success criteria are indeed important predictors of the commercial success of

early stage R&D projects. The out-of-sample forward test of the model also confirms this

conclusion.

It should be noted that previous research on key success factors has never

investigated the problem of biased coefficient estimates due to self-selection although

such selection effects are likely to be rampant. Consider for example Cooper’s (1981)

study. Project managers were asked to fill out a questionnaire for one successful and one

unsuccessful project ex post of project completion. The success and failure of these

projects are a function not only of the characteristics at the start of the projects but also of

the management behavior and resource allocations throughout the projects’ duration.

Projects that may at an early stage be considered to be most promising are likely to

receive more managerial attention and various resources leading to a potential self-

fulfilling prophecy. Any statistical analysis of project success must take such effects into

account since the model of interest is that which indicates the likelihood of success before

additional resources have been allocated.

5. Discussion and Conclusions Recent literature on managing the innovation process has been criticized for an

undue focus on the process of R&D project selection while not addressing the decision

criteria necessary to select appropriate projects (Cooper, 2000; Murphy and Kumar,

1997).

I therefore revisit the fundamental question regarding the key success factors,

specifically for early stage R&D project success. My intent was to address most if not all

of the methodological problems that have been plaguing past research in the area. With

this work I hope that researchers will be able to start building an understanding and

appreciation of appropriate statistical methodologies to be used for research in this area. I

also expect that the research will spawn the development of a set of models that map out

how various R&D project characteristics affect project success in various settings.


I examine the impact of a comprehensive set of 36 innovation and product-market

characteristics on the likelihood that an early stage R&D project will be commercialized.

These characteristics include several dimensions of technological opportunity,

competition, legal and societal conditions and nine dimensions characterizing user need

and market demand. The examined characteristics, while reasonably comprehensive in

their coverage of technical, market and commercial conditions, do not include any

organizational dimensions.

The research contributions are twofold: a) this is the first statistical analysis on the

effect of key success factors on the commercial success for early stage R&D projects

generated outside of established organizations, b) to my knowledge this is the first large-

scale study of key success factors where three methods biases are avoided: hindsight bias,

common method variance bias, and low statistical power.

I found that a statistical model based on four underlying characteristics of early

stage R&D projects as assessed subjectively by outside and project independent analysts

correctly predicts 80.9% of all project outcomes in terms of whether they will reach the

market or not in out-of-sample tests. The model is approximately equally able to correctly

classify both successful and unsuccessful ventures. This is an unexpected result as it

implies that the model is able to detect and appropriately use information in a highly

multivariate setting where there is a low signal-to-noise ratio and data is highly uncertain.

Its prediction accuracy is higher than R&D department managers’ ability to predict the

technical success of their own R&D projects, which was estimated by Mansfield (1968)

to be 66%. The model is further at least twice as accurate as seasoned VCs in the U.S.

who have a maximum prediction accuracy of 40% for seed and early stage ventures

(Zacharakis and Meyer, 2000).

The estimated model, however, perform seemingly worse than the new product

project selection model derived by Cooper (1981), which obtained a within-sample

prediction accuracy of 84.1%. But since Cooper’s study suffers from common method

variance bias where data on independent and dependent variables were collected at the


same time and after the fact it is not clear how well Cooper’s model stands up against

this, where data on predictors are obtained prior to observing outcomes. It is also not

clear how well Cooper’s model performs when used in actual decision situations as his

model is based on post-project data which may be tainted by hindsight bias. In addition,

the out-of-sample test of prediction used in this article is a more critical test than Cooper's

split-sample test and illustrates that the model is likely to perform well for other data sets

than that which it was estimated on.

One should be clear of the limitations. The statistical model applies to what it has

been calibrated on: screening for market success of seed and early stage R&D projects

conducted by independent inventors. As Balachandra and Friar (1997) discusses, it is

unlikely that there exists a model that generalizes to all types of R&D projects. Rather,

their suggestion, which I have followed in this study, is to develop models for different

classes of R&D projects and different decision-making situations to obtain better

knowledge and accuracy. In this paper I have focused on developing a model for early

stage evaluation of R&D projects with high uncertainty, low probability of success and

small development costs. Further, the projects that it applies to are primarily “low-tech”

and not radical. The model does not necessarily apply to other investments, it does not

measure return on investment and it does not substitute for due diligence investment

review once the initial screening has been undertaken. Nevertheless, it does seem to

provide a great improvement over current practice in the screening of seed and early

stage R&D investment proposals.

The four specific criteria that the data generates as most predictive: expected

profitability, technological opportunity (relative product quality), development risk and

appropriability conditions (legal protection) are no great surprises and they should not be

given that past research has already identified these characteristics (among 68 others).

What is interesting is that three variables provide additional explanatory power over and

beyond the “overall” estimate of expected returns. There might be two explanations that

are not mutually exclusive: 1. Analysts’ fail to incorporate all available information in


their estimate of expected returns through some cognitive bias. 2. Analysts consider some

variables to be predictive of technical and/or commercial success and consider some to be

determinants of financial returns conditional on technical and commercial success.

Further research is necessary to examine this issue.

The results do not fit Balachandra and Friar’s (1997) suggestion that technology

factors are less important for incremental and low-tech inventions. The degree to which

an invention represents a clear improvement over previous products and the degree to

which there is technological uncertainty associated with further development are

important technological determinants for these inventions.

Since the underlying characteristics are subjectively measured there is some

uncertainty regarding the transferability of the results to practice. The results do not

inform non-IAP analysts on how to perform the subjective assessment of the four

characteristics. However, with some training by individuals knowledgeable about the

IAPs’ process it should nevertheless be possible to use such a model by assessors at for

example VC firms focused on seed and early stage investments. This is an area of

application reasonably related to the one I have investigated. My conclusion rests on the

results by Einhorn (1972) who shows that expert judgmental scores on predictors are

useful in predicting outcomes and Dawes et al. (1989) who show that statistical models

based on judgmental scores perform better then experts' judgmental predictions based on

the same underlying judgmental scores. If the IAPs’ classification accuracy is maintained

by VCs using this model it would suggest a doubling of the rate of return on VCs’

investments on seed and early stage investments due to the improvement in screening

ability.

It is plausible that superior statistical decision-support models can be constructed

on historical data from other types of investments such as second and third round

financing, or when focusing on specific industries. It is obvious that the key criteria may

shift across the types of investments. I am currently working together with a large VC

fund to create a statistical decision-support model for a later stage fund. A statistical


problem that needs to be addressed is self-selection. Those investments that received

funding are more likely to succeed than those that did not due to the capital injection. As

discussed in this paper, there are statistical methods to control for this effect.


REFERENCES

Abratt, R. and A. A. Lombard (1993): “Determinants of Product Innovation in Specialty

Chemical Companies,” Industrial Marketing Management, Vol. 22, pp. 169-175.

Albaum, G. (1975): The Independent Inventor, Eugene, OR: University of Oregon.

Agresti, A. (1990): Categorical Data Analysis, New York: John Wiley.

Åstebro, T. (1998): “Basic Statistics on the Success Rate and Profits for Independent

Inventors,” Entrepreneurship Theory and Practice, Vol. 23, pp. 41-48.

Åstebro, T. (forthcoming): “The Return to Independent Invention: Evidence of

Unrealistic Optimism, Risk Seeking or Skewness Loving?” Economic Journal.

Åstebro T. and I. Bernhardt (1999): “The Social Rate of Return to Canada’s Inventor’s

Assistance Program,” The Engineering Economist, Vol. 44, pp. 348-61.

Åstebro T. and G. Chen (2002) “A Statistically Validated Method for Selecting Early

stage Ventures” Proceedings: What Next for Venture Capital and Private Equity?

Capital Markets Institute, University of Toronto, Toronto, June 21-22, 2002

Åstebro T. and Y. Gerchak (2001): “Profitable Advice: The Value of Information

Provided by Canada’s Inventor’s Assistance Program,” Economics of Innovation and

New Technology, Vol. 10, No. 1, pp. 45-72.

Bacchher, J.S., (2000): Venture capitalists’ investment criteria in technology-based new

ventures, Doctoral dissertation, University of Waterloo, Waterloo, Ontario, Canada.

Balachandra, R. and J. Friar (1997): “Factors for Success in R&D Projects and New

Product Innovation: A Contextual Framework,” IEEE Transactions on Engineering

Management, Vol. 44, pp. 276-87.

Brown, S. L. and K. M. Eisenhardt (1995: “Product Development – Past Research,

Present Findings and Future Directions,” Academy of Management Review, Vol. 20,

No 2, pp. 343-378.

Campbell, D. T. and D. W. Fiske (1959): “Convergent and Discriminant Validation by

the Multi-Trait-Multi-Method Matrix” Psychological Bulletin, Vol. 56, pp. 81-105.

CIC (1996): Annual Report, Canadian Industrial Innovation Centre, Waterloo, Canada.


Cliff, N. (1987): Analyzing Multivariate Data, San Diego: Harcourt Brace Jovanovich.

Cohen, W. (1995): “Empirical studies of innovative activity,” In P. Stoneman (Ed.),

Handbook of the Economics of Innovation and Technological Change, (pp. 182-264).

Oxford: Basil Blackwell.

Constandse, W. J. (1971): “Why New Product Management Fails,” Business

Management, pp. 16-19, June.

Cooper, R. G. (1979): “The Dimensions of Industrial New Product Success and Failure,”

Journal of Marketing, Vol. 43, pp. 93-103.

Cooper, R. G. (1981): “An Empirically Derived New Product Project Selection Model,”

IEEE Transactions on Engineering Management, Vol. 28, pp. 54-61.

Cooper, R. G. (1983): “A Process Model for Industrial New Product Development,”

IEEE Transactions on Engineering Management, Vol. 30, pp. 2-11.

Cooper, R. G. (1984): “How New Product Strategies Impact on Performance,” Journal of

Product Innovation Management, Vol. 1, pp. 5-18.

Cooper, R. G. (1985): “Overall Corporate Strategies for New Product Programs,”

Industrial Marketing Management, Vol. 114, pp. 179-193.

Cooper, R. G. (2000): “Product Innovation and Technology Strategy,” Research

Technology Management, Vol. 43, pp. 38-41.

Cooper, R. G. and E. J. Kleinschmidt (1987): “New Products: What Separates Winners

from Losers,” Journal of Product Innovation Management, Vol. 4, No. 3, pp.169-184.

Dahlin, K., M. Taylor and M. Fichman (2002): “Today’s Edisons: Technical Merit and

Success of Inventions by Independent Inventors,” manuscript, Carnegie Mellon

University.

Dawes, R. M., D. Faust, and P. E. Meehl (1989): “Clinical Versus Actuarial Judgment,”

Science, Vol. 243, pp. 1668-74.

Dillman, D. (1978): Mail and Telephone Surveys: The Total Design Method. New York:

Wiley.


Einhorn, H. (1972): Expert Measurement and Mechanical Combination,” Organizational

Behavior and Human Performance, Vol. 7, pp. 86-106.

Fischhoff, B. (1975): “Hindsight is not equal to foresight: The effect of outcome

knowledge on judgment under uncertainty,” Journal of Experimental Psychology:

Human Perception and Performance, 1(3) pp. 288-299.

Freeman, C. (1982): The Economics of Industrial Innovation, London: F. Pinter.

Goldenberg, J., D. R. Lehmann and D. Mazursky (2001): The Idea Itself and the

Circumstances of Its Emergence as Predictors of New Product Success,” Management

Science, Vol. 47, pp. 69-84.

Graves, S. D. & J. L. Ringuest (1991): Evaluating Competing R&D Investments.

Research & Technology Management, (July-August) pp. 32-36

Griffin, A. (1997): “PDMA Research on New Product Development Practices: Updating

Trends and Benchmarking best Practices,” Journal of Product Innovation


Griffin, A. and A. L. Page (1993): ”An Interim Report on Measuring product

Development Success and Failure,” Journal of Product Innovation Management, Vol.

10, pp. 291-308.

Heckman, J. J. and E. J. Vytlacil (2000): “Instrumental variables, selection models, and

tight bounds on the average treatment effect,” technical working paper, National

Bureau of Economic Research, MA, USA.

Herath, H. S. B. and C. S. Park (1999): “Economic Analysis of R&D Projects: An

Options Approach,” The Engineering Economist, Vol. 44, pp. 1-33.

Hopkins, D. S. (1981): “New Product Winners and Losers,” Research Management,

May, pp. 12-17.

Lavrakas, P.J. (1993): Telephone Survey Methods, 2nd ed., Newbury Park: SAGE

Publications.

Liberatore, M. J. and G. J. Titus (1983): The Practice of Management Science in R&D

Project Management, Management Science, Vol. 29, pp. 962-74.


Lilien, G. and E. Yoon (1989): Determinants of New Industrial Product Performance: A

Strategic Reexamination of the Empirical Literature,” IEEE Transactions on

Engineering Management, Vol. 36, pp. 3-10.

Linton, J. D., S. T. Walsh and J. Morabito (2002): “Analysis, Ranking and Selection of

R&D Projects in a Portfolio”, R&D Management, Vol. 32, No. 32, pp. 139-48.

Maddala, G. S. (1983): Limited Dependent and Qualitative Variables in Econometrics.

Cambridge, UK: Cambridge University Press.

Maidique, M. M. and B. J. Zirger (1984): A Study of the Success and Failure in Product

Innovation: The Case of the U.S. Electronics Industry,” IEEE Transactions on

Engineering Management, Vol. 31, pp. 192-203.

Maidique, M. A. and B. J. Zirger (1985): “The New Product Learning Cycle,” Research

Policy. Vol. 14 (6), pp. 299-313.

Mansfield, E. (1968): Industrial Research and Technological Innovation - An

Econometric Analysis. New York: W. W. Norton.

Mansfield, E., J. Rapaport, A. Romeo, E. Villani, S. Wagner & F. Husic (1977): The

Production and Application of New Industrial Technology. New York, W. W. Norton.

Murphy, S. A. and V. Kumar (1997): “The Front End of New Product Development: A

Canadian Survey,” R&D Management, Vol. 27, pp. 5-15.

Nunnally, J. (1978): Psychometric Theory, 2nd Edition, New York, McGraw-Hill.

Parker, R., G. Udell and L. Blades (1996): “The New Independent Inventor: Implications

for Corporate Policy,” Review of Business, Spring 1996, pp. 7-11, and page 34.

Pries, F., T. Åstebro and A. Obeidi (2002): “Economic Analysis of R&D Projects: Real

Option versus NPV Valuation Revisited,” Proceedings, Vol. 23, Administrative

Sciences Association of Canada, May 25-28, Winnipeg.

Rogers, E. (1983): Diffusion of Innovations, 3rd ed., New York : Free Press.

Rosenthal, R. and R. Rosnow (1991): Essentials of Behavioral Research: Methods and

Data Analysis, 2nd edition. New York: McGraw Hill.


Souder, W. E. (1973): Utility and Perceived Acceptability of R&D Project Selection

Models. Management Science, Vol. 19, pp. 1384-94.

Twiss, B. (1986): Managing Technological Innovation, 3rd edition. London: Longman.

Udell, G. (1989): “Invention Evaluation Services: A Review of the State of the Art,”

Journal of Product Innovation Management, Vol. 6, pp. 157-68.

Udell, G., R. Bottin and D. Glass (1993): “The Wal-Mart Innovation Network: An

Experiment in Stimulating American Innovation,” Journal of Product Innovation


Voss, C. A. (1985): “Determinants of Success in the Development of Application

Software,” Journal of Product Innovation Management, Vol. 2, pp 122-129.

Yap, C. M. and W. E. Souder (1994): “Factors Influencing New Product Success and

Failure in Small Entrepreneurial High-Technology Electronics Firms,” Journal of

Product Innovation Management, Vol. 11, pp. 418-432.

Yoon, E. and G. L. Lilien (1985): “New Industrial Product Performance, The Impact of

Market Characteristics and Strategy,” Journal of Product Innovation Management,

Vol. 3, pp. 134-144.

Zacharakis, A. and D. Meyer (2000): “The Potential of Actuarial Decision Models: Can

they Improve the Venture Capital Investment Decision?” Journal of Business

Venturing, Vol. 15, pp. 323-46.

Zopounidis, C., (1994): “Venture capital modelling: Evaluation criteria for the appraisal

of investment,” The Financier ACMT, Vol. 1, pp 54-64.


Appendix A Variable name, legend and explanation (as used by the IAP). Variable Name Explanation Technical Feasibility V1 Is the technical solution sound and complete? Functional Performance V2 Does this innovation work better than the alternatives? Research and Development V3 How great a burden is the remaining research and development required to bring

the innovation to a marketable stage? Technology Significance V4 How significant a contribution to technology or to its application is proposed? Safety V5 Are potential dangers or undesirable side effects expected? Environmental Impact V6 Will the innovation lead to pollution, litter, misuse of natural resources or the

like? Technology of Production V8 Are the technology and skills required to produce the invention available? Tooling Cost V9 How great a burden is the cost of production tooling required to meet the

expected demand? Cost of Production V10 Does production at a reasonable cost level appear possible? Need V11 Does the innovation solve a problem, fill a need or satisfy a want for the

customer? Potential Market V12 How large and how enduring is the total market for all products serving this

function? Trend of Demand V13 Will the demand for such an innovation be expected to rise, remain steady, or

fall in the lifetime of this idea? Duration of Demand V14 Is the demand for the innovation expected to be “long term”? Demand Predictability V15 How closely will it be possible to predict sales? Product Line Potential V16 Can the innovation lead to other profitable products or services? Societal Benefits V18 Will the innovation be of general benefit to society? Compatibility V19 Is the innovation compatible with current attitudes and ways of doing things? Learning V20 How easily can the customer learn the correct use of the innovation? Visibility V21 How evident are the advantages of the innovation to the prospective customer? Appearance V22 Does the appearance of the innovation convey a message of desirable qualities? Function V23 Does this innovation work better than the alternatives? – or fulfill a function not

now provided? Durability V24 Will this innovation endure “long usage”? Price V26 Does this innovation have a price advantage over its competitors? Existing Competition V27 Does this innovation already face competition in the marketplace that will make

its entry difficult and costly? New Competition V28 Is this innovation likely to face new competition in the marketplace from other

innovations that must be expected to threaten its market share? Marketing Research V29 How great an effort will be required to define the product and price that the final

market will find acceptable? Promotion Cost V30 Is the cost and effort of promotion to achieve market acceptance of the

innovation in line with expected earnings? Distribution V31 How difficult will it be to develop or access distribution channels for the

innovation? Legality V32 Does the invention meet the requirements of applicable laws, regulations and

product standards and avoid exposure to product liability? Development Risk V33 What degree of uncertainty is associated with complete successful development

from the present condition of the innovation to the market ready state? Dependence V34 To what degree does this innovation lose control of its market and sales due to

its dependence on other products, processes, systems or services? Protection V35 Is it likely that worthwhile commercial protection will be obtainable for this

innovation through patents, trade secrets or other means? Size of Investment V37 Is the total investment required for the project likely to be obtainable? Potential Sales V38 Is the sales volume for this particular innovation likely to be sufficient to justify

initiating the project? Payback Period V39 Will the initial investment be recovered in the early life of the innovation? Profitability V40 Will the expected revenue from the innovation provide more profits than other

investment opportunities? Note: The following variables were excluded as they were replaced by other variables by the IAP in 1989: V7 (Production Feasibility), V17 (Stage of Development) and V36 (Investment cost). V25 (Service) was dropped because of too much missing data.

Key Success Factors Jan 2003 - Semantic Scholar · effect of key success factors on the commercial...

Documents

Transcript of Key Success Factors Jan 2003 - Semantic Scholar · effect of key success factors on the commercial...