IMPACT ASSESSMENT OF AGRICULTURAL R & D PROGRAM AND PROJECTS - AN OVERVIEW

Virtually all research institutions monitor their research projects for financial and bookkeeping purposes. In addition, many attempt to evaluate the quality of their research. Unfortunately, few research institutions commit significant resources to the process of evaluating the social and environment impacts of their research.

There are several reasons for their reluctance to do so. Perhaps some of this unwillingness stems from the feeling that research resources are scarce; the benefits are obvious and we should therefore simply get on with the process. Perhaps some of the unwillingness stems from a fear that evaluation of research results would produce unfavorable benefit- cost ratios. And perhaps some of the unwillingness stems from the methodological difficulties encountered when establishing the benefits from some types of research.

It is unfortunate that agricultural institutions spend so few resources on attempting to measure the impact of their research on society because we ought to know the results of such spending. We ought to know if it pays and if so, how much it pays.

Assessing the impact of research and development attempts to quantify the costs and benefits from research and development activities. The methods used are not particularly difficult to understand. Engineers and bankers regularly do benefit cost analysis. There is not reason agriculture researchers shouldn’t have the capability to do the same thing. The impact of applied research and development is relatively easy to identify and the payoffs are usually very high.

The science of impact assessment has developed rapidly in the last few years. Despite significant advances, methods of impact assessment are required to be fine-tuned to site specific nature of agricultural research. The multiple objectives of agricultural research like food security, poverty, environmental protection, sustainability, etc. further complicate the outcome of such analysis of agricultural research program and projects.

What is impact assessment?

Impact assessment attempts to estimate the effects that research has had in the past or that it may have in the future. There are many different kinds of effects or impacts and different ways to measure them. An impact assessment may look at whether farmers accept or reject new technology, or it may focus on increases in yields and production that can be attributed to new technology. It can also estimate changes in income, employment, nutritional status, pollution, erosion, or rural-urban migration.

Impact can be examined from two perspectives: after research is completed (ex post) or during planning (ex ante). Agricultural research managers often use ex post impact assessments for positive information on results to justify requests for continued

1

funding and support. Ex ante impact assessments may be done as an aid to priority setting – to estimate the future benefits of different research projects. Most impact assessments are sponsored by development agencies and serve their own decision-making and accountability needs. NARS seldom carry out impact assessments themselves, although many agricultural research managers and policymakers realize that impact assessment is useful in setting research priorities and demonstrating results. The most common types of impact studies carried out for agricultural research are adoption studies and economic evaluations (rate-of-return studies). Relatively few social or environmental impact assessments have been done, but there are increasing demands for them.

The subject matter of impact assessment consisted of A. Economic Impact Assessment

Adoption studiesEconomic studies (returns to investment)

B. Social and Environment impact assessmentEffect on poverty, gender issue, food security, etc.Effect on pollution, sustainability and natural resources, etc.

Doing Impact Assessment

Impact assessment involves estimating the effects of agricultural research or new technologies. Many different types of effects may be examined, but assessing any one of them in any depth can take considerable time and money. For this reason, the first task in impact assessment is to focus the study by asking, Why is it being done? What information is needed? What research effort or technology should be evaluated? And what types of effects should be assessed? The focus of an impact assessment should reflect the purpose of the evaluation, who is requesting it, what their interests and information needs are, and how the results will be used. It should also take available resources into consideration (including trained and experienced personnel).

If the principal purpose of an impact assessment is to estimate the benefits of research in a way that is comparable to other public investments (such as public health or credit programs), then an economic rate-of-return study may be appropriate. If, on the other hand, there is an interest in understanding the distribution of benefits among different farming groups or different regions within a country, a more descriptive and illustrative adoption study may be called for. If there is a concern for the effects of a new technology (such as a pesticide or new tillage system) on pollution or soil erosion, then an environmental impact assessment may be needed. In each of these cases, the impact assessment would attempt to estimate the effects of research on the selected variable, be it agronomic, economic, social or environmental.

Adoption studies generally trace the results of innovations from the research station or on-farm trials through networks of adopters. These studies analyze the underlying patterns of adoption and the use of new practices. Adoption surveys use interviews with farmers to see if they are using improved technology, to look at its effects

2

on farm production, and to determine how research activities can be reoriented to make technologies more useful. They attempt to determine why a technology is or is not being used and compare the benefits of old versus new technologies. Again, no single approach is best, but the general steps are as follows:

Select the technology to be evaluated. Identify the central issues and questions to be asked. Design data collection and analysis. Field-test instruments and make adjustments if necessary. Collect the information. Analyze the information. Present the results and recommendations.

The results are normally presented in terms of percentages of adopters, changes in yield, and the reasons for the technology not being adopted or its use being discontinued. Analysis of the reasons for the technology being rejected can be used to guide or reorient research strategies.

Economic impact assessments generally estimate the economic benefits produced by research in relation to associated costs. The methods employed in economic evaluations are outlined in the following section.

Social and environmental impact assessments are concerned with the broader effects of a research project or activity on society and the natural environment. Such assessments go beyond the examination of economic returns and look at the effects on such things as cash flow, labor, or health. There has been relatively little work in this area, partly because of its complexity. However, with increasing social and political concerns for environmental issues, this is now a growing area of interest.

Assessing the effects of agricultural research is complex and costly for three reasons. First, it is quite difficult to measure changes in yield, production, nutritional status, and erosion, and it requires costly fieldwork and analysis. Agricultural research organizations often lack the personnel and operating funds needed for this, especially when several growing seasons are required for changes to be measured in most yield and production systems. Trends in dryland agriculture, for example, cannot be measured in fewer than eight cropping seasons.

Second, even where a change can be measured, it is extremely difficult to attribute it to specific research activities. For example, in an area where potato yields or milk production has increased, how can the contribution of research versus that of extension, credit programs and improvements in market conditions be estimated?

Third, research managers, policymakers, donors, and the public all tend to be impatient and to want impact estimates when research is still underway or has just recently been completed. This is neither realistic nor possible. There is often a

3

considerable time lag between the time research is started, a new technology is released, and impacts can be measured – often as long as 10 to 15 years.

More than other types of evaluation, impact assessments tend to be carried out as research studies leading to formal publications. They generally employ scientific methods drawn from economics and the social sciences and often use indirect measures or indicators of impact because the effects of technology on farm-level production, nutritional status, pollution, and the like cannot be directly measured. To cope with this problem, production-function models are often used to estimate the effects of research or technology on production, incomes and associated variables. Numerous assumptions are also often made to overcome data limitations and to simplify economic models.

Research managers and policymakers tend to be skeptical of the data and methods used in impact assessment; they may also find the reports difficult to understand, interpret, and apply. This highlights the need to plan impact studies in terms of real information needs (rather than peer interests), to pay close attention to data quality, and to make special efforts to summarize the findings. It is extremely important for results and recommendations to be presented in terms that are meaningful to policymakers, managers, and scientists.

Approaches for Impact Assessment

A number of different approaches have been used to evaluate agricultural research. Some of these are now regarded as having little to commend them and have been largely discarded. For example, personnel and programmes may be evaluated through the number of publications or reports issued, technical meetings held, committees established or seminars undertaken, although none of these criteria necessarily represents tangible activities in terms of bringing about changes in agricultural productivity.

Now there are two primary methods have been used to calculate returns to research and to estimate its impact on society:

1. The economic surplus approach (consumer- producer surplus, cost – benefit and index number (TFP) methods) estimates returns on investment by measuring the change in consumer and producer surplus from a shift to the right in the supply curve due to technological change.

2. The econometric approach (production, profit and supply function and their derivatives) treats research as a variable and allows a marginal rate of return on investment to be calculated.

Impact-evaluations are often used to try to convince policymakers that to resource allocations to research represent good investments. Progress in terms of production, income, or marketable produce is usually the yardstick of success used by policymakers, so a high rate of return to research investment can be a strong selling point for research leaders when presenting the research budget for approval by government authorities.

4

Most evaluation studies have been unable to effectively distinguish between these three variables and have implicitly assumed that the returns from agricultural research also include the benefits from education and extension. The few studies that have sought to separate these activities have had to do so subjectively. Because of the lack of sufficient theoretical instruments, these studies have generally attributed most of the benefits to research alone, and in some cases this may have resulted in erroneously high rates of return being attributed to investments in research.

Yet another problem with the measurement of outputs relates to assessing the significance of the maintenance research which is required to overcome obsolescence, particularly that due to changes in disease and pest biotypes. In order to keep pace with these changes, the output of maintenance research may be just as important as that of research on innovation, but identifying and accounting for maintenance research is not an easy task since, for example, the pathogenicity of new strains of pests and diseases does not develop following a predictable pattern.

Problems of measurement can also apply to inputs, although many of these can be defined. However, inputs of highly skilled manpower may be hard to quantify. The pricing and measurement of previous research also represents an input problem unless prior research endeavours are treated as free goods. But this ignores the fact that someone paid for the prior research, and if this cost is not taken into account it can again give a result which places an unduly high social rate of return on the research.

In spite of these problems a number of efforts have been made to use input and output data to carry out benefit / cost analysis of agricultural research. This has been done either in an ex post sense or ex ante.

Methods of Economic Evaluation

a) Ex ante evaluationScoring modelBenefit cost analysisSimulation approachMathematical programming approach

b) Ex post evaluation

Economic surplus approachProduction function approachNational income approachNutritional impact approach

Schuh and Tollini (1979) have described a number of approaches and models that have been used to make ex post evaluation. They found that there was a rather rich set of research procedures that have been developed whereby research can be evaluated and its

5

contributions and various effects analysed. Different approaches are useful for answering different questions and the particular question posed will vary a great deal depending on the individual problem situation. A major constraint to the use of ex post analysis is, however, the time period between undertaking research and being able to assess its benefits.

For ex-ante decision making there is a vast literature from industrial and military research but rather less from agricultural research. A number of models are available with methodological sophistication ranging from simpler scoring models to more complex mathematical programming models. Schuh and Tollini state that the advantages of these models are that they provide a basis for decision making with an eye to the future rather than the past. They pool information from a large number of qualified experts and they provide a means of explicitly relating the research effort to a set of goals. The disadvantages are that those methods which draw on the opinion of a large number of specialists can be quite costly and time consuming and the pooling of a large number of opinions may do little more than to pool ignorance. It is probably for these reasons that the more complicated methods have rarely been used more than once, although selected models may provide a means of feeding some rigorous analytical research into the decision-making process.

Benefit / cost analysis is widely used by governments and funding agencies for deciding on and evaluating investment in development projects. It is based on the concept of discounted cash flow. Benefit / cost analysis should include, at a minimum, the sequential estimation of eight distinct characteristics of a research programme and its impact (Bottomley 1988). They are: the annual cost of research, its duration, its initially anticipated probability of success, on-farm implementation costs, on-farm benefits, the rate of adoption, the adoption ceiling, and the life of the innovation.

A major difficulty in conducting benefit / cost analysis is in knowing how to define and measure both input and output. A World Bank publication (Schuh and Tollini 1979) has examined this subject in some depth. It stresses that the most important output of the research process is new knowledge, but this is not a quantifiable product nor is there a well-defined market for it, despite the fact that it has obvious economic value. Thus it is not possible to be precise in measuring the value of knowledge to society. Various proxies for output have been examined, including scientific publications, although publication vary widely in the quality and quantity of their content and are not a consistent measurement of value. An alternative approach is to define output in terms of some well-specified innovation, but this suffers from the problem that the level of adoption of an innovation may not necessarily relate to the quality of the research which produced the innovation.

Measuring benefits of research has been approached through different methods. The benefits are estimated using economic- surplus method. The economic surplus method is bit difficult to apply and complex to the evaluator who is not from the field of economics or statistics. There is another method to estimate benefits of technology called Gross Efficiency Index which is a close proxy to economic surplus method.

6

Yet another problem in using a formal cost / benefit analysis is a developing country research institution is that the analysis may tend to under-emphasize issues of equity. These may be of considerable importance in countries where income, wealth and power distributions are highly skewed. In many countries food security is the central goal for agricultural research. Busch (1985) has drawn attention to the fact that this has implications in terms of a number of distributive issues such as:

(a) New labour-saving technology may add to unemployment and result in a complete loss of income for certain people.

(b) Labor issues can also be important in terms of the introduction of new crops or varieties that radically alter seasonal labor needs or labor needs that interfere with other essential family activities. Such effects can reduce the demand for casual labor, thereby eliminating the traditional method of redistributing wealth and thereby contributing to reduce food security.

(c) New varieties of crops may encourage the use of marginal lands and lead to environmental degradation, undermining the food security of future generations.

(d) The role of women is significant in that they play a major role in each of the four aspects of household food use: procurement, handling, distribution and consumption. In many countries these four tasks are fully integrated and are part of the daily work activities of women. Changes in the labor patterns resulting from changes in production patterns may disturb this integration and effective agricultural research cannot ignore this intrinsic linkage, which is fundamental to food security.

(e) Broad issues of agricultural research policy may also effect food security, such as the introduction of feed grains or the encouragement of cereals as opposed to grain legumes. These can result in shifts in the protein/calorie balance, particularly of the poorer segment of the population, thereby affecting food security.

The Clients for Evaluation

The wide range of issues that can be looked at in an evaluation indicates that there can be a broad spectrum of clients for this activity. Since different clients may have different requirements from an evaluation it is important that the specific client be identified when an evaluation process is being structured. The terms of reference for an evaluation review should clearly relate to the needs of the client for the review, recognizing that many reviews will serve more than one client. Among the clients for research evaluations are:

National policymakers, who are interested in the role that research does or can play in national agricultural development. Their interest is primarily likely to be

7

in terms of strategic reviews which define overall research priorities and the resources to be allocated to research within the context of development strategy at large. Such clients are likely to be interested in the potential impact of agricultural research on production and productivity. They may also be particularly interested in the research potential when there is a major change in research capacity, technology potential, world economy or some other factor which modifies the agricultural sector and, therefore, research needs.

Donor agencies, who now play a major role in supporting agricultural research in developing countries. Evaluation reviews provide them with information on the efficiency of a research system and may help them to justify their investments in it or to identify areas needing strengthening in which their assistance programs have a potential role to play. This role may involve either technology and / or management.

Senior research managers, who are able to use the evaluation of past activities to assess the results achieved and to build the lessons of experience into the corporate memory of the institution and thereby to improve its future activities. Such manages have the task of selecting and designing new programs. Evaluation reviews can be used to determine which programs need to be strengthened, modified or deleted.

Project or program managers within the research system, who can use evaluations of ongoing activities to assess progress, and to identify bottlenecks and problems, so that they can be remedied before causing further damage.

Individual scientists, who are able to use review findings to look at their own research activities within the context of an entire program or institution, and also to place this work into the perspective of adoption and impact as conceptualized by a review team.

Obviously, there is a great deal of scope for weighting the scope of a review to cater to one or more of these client groups and the composition of a review team, the nature of its work and the focus of its activities will all need to take this into account.

The Method of Choice

There is little doubt that where appropriate data, skills and time are available, more sophisticated methods could be applied. Unfortunately many NARS lack the appropriate data, skills and time to conduct even simple cost / benefit analysis and are likely to be in this position for some time.

Another factor that needs to be taken into consideration when determining the method to be used is the fact that extent of accuracy is expected. More is the sophisticated method,

8

more is the accuracy of results. In many instances, poor management, rather than a lack of funding, appears to be the principal constraint to research impact.

Other factors responsible for selecting a method for impact assessment could be the knowledge of economics principles and statistics, data availability, computational facilities and time.

Assessing the Impact- a different View

In the discussion above, two things come out clearly. One, that impact assessment differs from general evaluation in terms of measuring the economic estimation of benefits of agricultural research. In other words, impact assessment is the measurement of benefits of research in economic terms. Another, that impact assessment could be conducted even before and after the completion of a research project. Accordingly, it is called ex ante and ex post impact assessment. It implicitly appears that impact assessment is analogs to economic evaluation of research outcomes. All the benefits generated by a research are counted. There I beg to differ from the usual definition of research evaluation and impact assessment. There is a need to differentiate between economic evaluation and impact assessment. While economic evaluation is measuring all kind of benefits of research to the society, the impact assessment measure benefits to particular section of the society and resource. For example, impact assessment answers the question, how research benefited poor or poor resources? Impact assessment analyzes the effect of research on social parameters like education, employment and health. It addresses the issues of environment, gender, sustainability, quality of natural resource, etc. Therefore, impact assessment mainly is the ex post evaluation and does not necessarily count all benefits but benefit to particular area, social group and welfare parameter. The research benefits on these could be positive or even negative. In that sense, the duration require to conduct impact assessment is more than the economic evaluation.

Usefulness of NARS

The main benefits of impact assessment for NARS are listed below:

Impact studies can motivate researchers by providing feedback from the farm community and other research clients on the use and effects of research results.

Adoption studies can help researchers refocus their research efforts by providing insights into farmers’ assessments of new technologies (vis-à-vis their current practices) and into farm-level adoption processes

Ex post studies can provide managers with evidence of the value of research to argue for continued investments.

Ex ante assessments can provide managers with a basis for allocating resources among competing research demands.

9

Lessons learned from impact assessments can be used to improve future research strategies, plans, and management.

Impact assessments show how economic policies and technology interact in determining the ultimate benefits of agricultural research. This can be useful for discussions between research leaders and policymakers.

Impact assessments can show how economic policies and technology interact in determining the ultimate benefits of agricultural research. This can be useful for discussions between research leaders and policymakers.

In agricultural research, impact assessments have been used mainly to estimate economic returns to research investments and the diffusion and adoption of new technologies. Economic studies (generally of successful cases) have produced very positive results, which have been used to justify continued support for agricultural research organizations and programs. The systematic use of impact assessments in planning or reorienting research is less common.

Research managers and policymakers need a better understanding of how impact studies can be used for decision-making. But they also need to understand what is required to do impact assessments; they tend to underestimate the time and resources needed for these kinds of studies.

Impact Assessment- Summary

1. Impact assessment attempts to determine the extent to which research contributes to higher level development goals, such as increased farm production or food self-sufficiency. One can differentiate two main types of impact assessment : one is conducted during planning (ex ante) and the other is conducted after the research results have been available for some time (ex post). Impact evaluations, which often indicate rates of return on the research investment, are primarily used to convince policymakers to allocate more resources to research. It can help in strategic planning, priority setting, and resource allocation, and can show how economic policies and technology interact.

2. Ex post impact assessment usually has a time frame of 10 or more years after research results have been released, making it less of a management tool than the other types of evaluation. As with other ex post evaluations, the baseline data, targets and assumptions from planning (ex ante evaluation) are the basis for determining progress and ultimate impact.

10

3. Research projects which may be good candidates for impact assessment, such as those with potential national results or highly innovative research, must have their needs built into the original M & E systems. For instance, if market prices need to be monitored periodically for use in a future impact evaluation, this must be identified at the planning stage and monitored during the course of the activity.

4. Evaluating the contribution of research to economic development is complex. Impact evaluations must distinguish research contributions made to national development from the contributions made by other factors, such as the existence of a good extension service, agricultural inputs, adequate infrastructure, and favorable marketing and pricing policies.

5. The two most common types of impact assessment are adoption studies and economic studies. Adoption studies emphasize issues related to the spread or diffusion of technology, the number of people affected, reasons for adoption or non-adoption, and the implications of adoption for the system. Economic studies emphasize the costs of such things as research and extension, and the benefits of a change in the production areas, or a change in yield, price, or quality.

6. The results of impact evaluations can have broad implications for future priority setting, not only for research but also for development support services.

REFERENCES

1. Management Systems International 600 Water Street, SW, NBU 7-7 Washington DC 20024, USA

2. Center for Development Information and Evaluation (CDIE) US Agency for International Development (USAID), Washington, DC 20523 - 1082, USA

3. Economic Evaluation Unit, Australian Centre for International Agricultural Research (ACIAR) GPO Box 1571, Canberra, ACT 2601, Australia.

4. Economic Advisor, Secretariat, Consultative Group on International Agricultural Research (CGIAR), The World Bank, 1818 H Street, NW, Washington DC 20433, USA

5. Anderson, J.R. (1992) Measuring the efficacy of international agricultural research. CIMMYT 1991 Annual Report. Centro International de Mejoramiento de Maiz y Trigo Mexico, DF

6. Anderson, J.R., Herdt, R.W. and Scobie, G.M. (1988) Science and food: The CGIAR and its partners. The World Bank, Washington, DC

11

7. Avila, A.F.D., Irias, L.J.M. and Paiva R.M. (1985) The socio economic impact of investments in research by EMBRAPA : Results obtained, profitability and future prospects. Empresa Brasileira de Pesquisa Agropecuaria, Brasilia.

8. Biswas A.K. and Geping Q. (1987) Environmental impact assessment for developing countries. Tycooly, London.

9. Carley, M.J. and Bustelo, E.S. (1984) Social impact assessment and monitoring : A guide to the literature. Social Impact Assessment Series 7. Westview, Boulder.

10. Echeverria R.G. (ed.) (1990) Methods for diagnosing research system constraints and assessing the impact of agricultural research. Vol. 2., Assessing the impact of agricultural research. International Service for National Agricultural Research (ISNAR), The Hague

11. Ghana Grains Development Project (1991) A study of maize technology adoption in Ghana. CIMMYT, Mexico, DF.

12. Graham - Tomasi, T. (1991) Sustainability : Concepts and implications for agricultural research policy. In : Pardey, P.G., Roseboom, J. and Anderson, J.R. (eds.), Agricultural research policy : International quantitative perspectives. Cambridge University Press, Cambridge.

13. Horton, D.E. (1986) Assessing the impact of international research and development programs. World Development 14, 453 - 468.

14. Kariuku, J.G. (1990) The economic impact of the adoption of hybrid maize in Swaziland. Farming Systems and Resource Economics in the Tropics 9. Wissenschaftsverlag Vauk Kiel, Kiel, Germany.

15. Khan, M.H. and Akbari, A.H. (1986) Impact of agricultural research and extension on crop productivity in Pakistan : A production function approach. World Development 14, 757 - 762.

16. Menz, K.M. (1991) Overview of economic assessments 1 - 12, Australian Centre for International Agricultural Research, Canberra.

17. Murphy, J., Casley D. and Curry J. (1991) Farmers' estimations as a source of production data : Methodological guidelines for cereals in Africa. Technical Paper No. 132. The World Bank, Washington, DC.

18. Sperling L and Loevinsohn, M.E. (1993) The dynamics of adoption : Distribution and mortality of bean varieties among small farmers in Rwanda, Agricultural Systems 41, 441 - 453.

12

19. Schuh, G.E. and Tollini, H (1979). Costs and Benefits of Agricultural Research: The state of art. World Bank Staff working Paper No. 360, World Bank, Washington.

LITERATURE AND PATENT CITATIONS AS A TOOL FOR EVALUATING RESEARCH PRODUCTIVITY

The evaluation of research and development activities is one of the most challenging and significant issues in R & D management today. The last few decades have seen considerable efforts in the development and testing of various measures and quantitative techniques in order to aid in project selection, project appraisal, and measurement of research outputs. These techniques are distributed across a methodological spectrum, ranging from the qualitative case study to the highly quantitative econometric analyses.

A particular concern in the development of these indicators is the question of how to develop a reliable and accurate measure of research output. Bibliometrics, or the study of publication-based data, is one of the more quantifiable methodological approaches available and has been embraced by researchers of the scientific community. There are many scholarly works on the use of bibliometrics in assessing the output of science in a wide range of disciplines. Some of these have been cited in references given in the end of notes.

Research and development activities are particularly difficult to assess. As a result of the difficulties associated with accurate measurement of the outputs of the R & D process, most measures are used as proxies for a particular aspect of the output. The research output is not always the technology. It may be ‘knowledge’ which may further, generate either advanced ‘knowledge or technology. The

13

economic evaluation in the case of research resulting into knowledge is difficult. Therefore, we look for impact assessment through alternative methods. As pressure for quantitative evidence of the “success” and impacts of scientific efforts grows, the use of quantitative measures of science, such as bibliometrics, has become more pervasive. A dilemma exists, however, in the use of bibliometric. The limitations and guidelines for the use of this important method will be taken separately in the end of the notes.

The purpose here is to provide a background and survey of bibliometric approaches to the analysis of R & D and to assess its utility for the evaluation of research and development. It provides a discussion and overview of the different utilities and techniques associated with the different forms of bibliometric analysis. It focusses on the methodological issues relevant to this form of analysis. What are the strengths and weaknesses of the approach ? What is its potential as an R & D impact evaluation tool? Guidelines for using bibliometrics in ex-post R & D evaluation are provided.

DEFINITION

Bibliometrics is the study and analysis of scientific output with the use of publication- based data. It is a clearly delineated body of research involving the measurement of ‘physical units of publications, bibliographic citation and surrogates of them.’

According of Pritchard (1969), the bibliometrics is the application of mathematics and statistical methods to books and other media of communication.

14

Publication serves to present in measurable terms the transfer of knowledge of R & D from one scientist to another. It is sometimes the only quantitative measure of research productivity.

THE TECHNICAL FORMS OF BIBLIOMETRIC ANALYSIS

Although the term “bibliometric analysis” is often used generically, it represents several different technical forms, each of which have different purposes and utilities. The common denominator in each of these forms is the concept and publications represent a flow of information primarily from basic science (Martin and Irvine, 1983). Each of these bibliometric approaches provide different information on the scientific enterprise. The main derivatives of bibliometrics are : publication, counts, citation counts, co-citation analysis, co-word analysis, scientific “mapping,” and citations in patents.

Publication Counts : This bibliometric approach involves the counting of scientific publications published by a researcher or a research group. It is the most basic of bibliometric techniques. Publication counts are sometimes considered to be a quantification of the peer review process, because each publication is likely to have been approved through a peer review system prior to acceptance for publication. Yet they are more accurately described as a simple measure of scientific productivity or output. Publication accounts are most useful for providing a measure of total research output. Their primary drawback is that they do not discern the quality of these outputs.

15

There have been many studies that have legitimized this measure as a reasonable proxy of scientific output. The findings of these studies provide support for the relative accuracy of these bibliometric measurements. Furthermore, the acceptance of this measure as indicative of scientific productivity is illustrated by its wide use. It should be noted, however, that publication counts are not useful as a single measure due to some of the methodological problems involved in their use.

Citation Counts: While publication counts measure output, citation counts are considered to go one step further and address questions of quality, influence, and the transfer of knowledge. The data being considered are the number of times that a certain publication, or author, is cited in other works. Proponents of citation counts as a quality indicator argue that the most important works will have the highest number of citations. In this case, citations are viewed as almost a quantitative peer review type of measurement. One scholar refers to citations as “frozen footprints in the landscape of scholarly achievement. More specifically, citation counts are considered to assess the impact of a particular scholarly work. The assumption is that the more important or influential a work, the more often it will be cited. This is perhaps its most useful application.

Co-Citation Analysis: Co-citation analysis, developed in the early 1970’s, is a method that identifies pairs or groups of articles that are cited together in other articles or publications. From these pairs or groups of articles a “cognitive structure” is

16

then derived. This cognitive structure provides information on the direction and flow of scientific thought.

Co-citation analysis is the most useful in monitoring how different subfields or specialities of science change and evolve over time. It can help to monitor emerging fields or changes in scientific concentrations. Co-citation analysis provides two main bodies of information. First, it defines the research problem areas that appear to be the basic units of scientific activity. Next, it measures the level of interaction between these areas. This results in the collective cognitive representation of the organization of the science. This can be done on a large (science as a whole) or small (scientific specialities or subfields) scale.

Co-Word Analysis: Often used together with co-citation analysis, co-word analysis is a relatively new development in bibliometrics. Developed in the early 1980’s, co-word analysis involves the assigning of keywords to a paper or article by professional. Papers which have the same keywords and sets of words are linked to each other via a clustering. This technique is used as part of scientific mapping. The co-word analysis uses the co-occurrence of key words to develop a distance mapping. This is expressed as a logical tree. The purpose is to map national and international cognitive networks and to position certain research institutes within these networks.

Co-word linkages have been proposed as an alternative to co-citation and citation analysis to “map” how documents are related to each other (Leydesdorff, 1989). For a set of

17

biochemistry articles, one study found that the co-word structure of citing documents was very similar to that of the cited documents (Leydesdorff, 1989). The author makes the arguments that the co-word structure reflects the intellectual structure of the document set as well as the intellectual organization of the related group of authors. This finding suggests that co-word analysis can be useful in understanding and describing intellectual organization (Leydesdorff, 1989).

Scientific “Mapping”: Strongly tied to co-citation and co-word analyses, scientific mapping involves developing a visual model or “map” of the realm of science. The “map” represents the structure of literature output of particular scientific fields. Mapping attempts to physically represent the scope of the scientific enterprise, or of a particular scientific subfield. It is developed in part through co-citation analysis and co-word analysis, but can also utilize other more subjective information, such as cognitive-psychological data that takes scientists’ perceptions of the “distances” between fields. In mapping, co-word analysis helps to illustrate the ideas and the individuals responsible for changes over time in science. Co-citation analysis helps to illustrate the base of knowledge used in particular developments in science.

Scientific mapping is most useful for R & D management and in policy formation. Mapping can help to evaluate the state of various scientific disciplines, including potential areas for investment. Some of the most important information to be gained through this approach is the identification of “invisible college.”

18

Citations in Patents: Recently developed methods of bibliometric analysis are the analysis of patents for references to

other patents and to publications. Patents are granted for advancements in the state of the art. By analyzing the patents that are

cited by later patents, the relative importance of a patent can be judged, and a network profile of individuals, organizations and

countries can be developed. Citations to publications offer a measure, in part, of the relationship between science and technology. These

developments have been possible as patent applications have become available online. Analysis of citations in patents has several potential

practical applications. One example is “technological opportunity scanning” which is based on patent linkage. Patent citation analysis

may show how a firm patent relationships may give a firm the opportunity to exploit new research areas.

Another application of patent citation analysis is to establish a link between science and technology, by noting the citations to scientific literature. Patents in highly scientific areas of technology contain as many citations to the basic research literature as the scholarly literature does. This illustrates how closely related some technologies are to science and gives a useful description of the interaction of science and technology yielding a measure of how science intensive or how “high-tech” a particular firm or country is.

LIMITATIONS OF BIBLIOMETRIC ANALYSIS

Literature citation and bibliometric analysis, in that matter has received a great deal of attention since its inception. Early on, it was viewed as a sound quantitative

19

indicator of scientific quality and output. In spite of its strengths and potential contributions to the understanding of the scientific enterprise, it has received a fair amount of criticism.

While it has distinct strengths, bibliometrics is also subject to several valid criticisms. First, there are problems of measuring quality in science using bibliometric data. While publication counts provide a measure of research output, they provide no information on quality. This is one of the most common criticisms of bibliometric data. Publication counts have also been subject to other criticisms, such as problems associated with gatuitous co-authoring of articles; different publication practices across fields; and difficulties of defining fields of research especially given strong trends towards collaborative research.

Citation analysis has also been subjected to a certain amount of criticism. A central criticism of citation analysis is that the resulting measurements do not account for quality. They favour scientists doing mainstream or main paradigm work; they overemphasize basic statistics books used in the social and behavioral sciences; and they overemphasize popular science rather than scholarly publications. Many studies found similar problems of bias that were compounded by the prevalence of negative citations and incomplete citations. The citation analysis has resulted in conclusions that are statistically unjustified. In fact, using citation counts to measure quality may be “measuring what is measurable rather than what is valid” (Lindsey, 1989). More important, not all

20

citation represent an equal contribution to science. It result into saying that bibliometric findings should not be used in science policy work until the problems with citation analysis are effectively addressed.

Several studies suggest that it is important to recognize the limitations of bibliometrics in assessing quality but also to recognize what citation analysis does reveal. Several advantages to using bibliometric analysis on a macro scale when making international comparisons are that the bibliometric data is regularly updated, and it yields comprehensive information spanning all of science and engineering. Yet it is cautioned that publication counts are valuable only as a partial indicators of scientific progress. It seems that this suggestion is the most realistic way to deal with the limitations of bibliometric data.

Although there have monumental advances in the data keeping and retrieval systems, bibliometric databases have had their share of criticism. While most of the criticisms of bibliometric analysis focus on issues of measuring the quality of science, several arguments are made about the limitations of the central bibliometric data base – the computerized Science Citation Index. Many argue that the data-base clearly is biased toward studies in the English language, that its classification of journals into fields and subfields needs updating, and that it is really only relevant for assessing basic research. As a result of the expansion of the Science Citation Index over the years, there is a great deal of potential for measurement error. Originally, the Science Citation Index covered publications in

21

approximately 800 journals. In 1981, this had expanded to more than 3000 journals. Measurement error is not limited to only a few specialities, but rather is spread over many fields. Other criticisms include problems of the Science Citation Index surrounding accounting for international co-authorship and problems associated with aggregated studies (Leydesdorff, 1989).

Among several problems with using citation analysis to evaluate scientific performance, are the problems of measurement counting citations, and issues surrounding multi-authorship. For example, the Science Citation Index only lists publications by their first author in multi-authored papers. Thus, second or other authors are not accounted for. In spite of any of these criticisms, this and other similar databases are currently the most efficient way to conduct bibliometric analysis.

GUIDELINES FOR USING BIBLIOMETRIC ANALYSIS

Bibliometrics is most used as a tool for output or impact evaluation of basic research. However, some bibliometric approaches have additional analytical potential that may be applied to other aspects of the R & D process. Analysis of patents in citations is a good example of bibliometric analysis that moves beyond the evaluation of basic research.

There are not many organizations that use bibliometric analysis as a systematized evaluative tool. Bibliometrics is a retrospective evaluation approach, useful for ex-post

22

evaluations. However, there are several points to consider when applying bibliometric analysis to R & D evaluation. The following are guidelines intended to improve the use of bibliometric analysis for R & D evaluation.

1. Use it on the appropriate unit of analysis.

While bibliometrics provide useful evaluation tools, they are not appropriate for all levels of analysis. Bibliometrics should not be used to measure the performance and productivity of individual scientists. The reasons why the performance of individual scientists, should not be judged primarily through citation analysis; the most important reason for this is the roughness of citations as a quality indicator. The potential for a large margin of error is great. Some scientists may be considered by other scientists to do high quality work. However, they may not have a high number of citations. The remedy for this problem is to 1) analyze research groups, not individuals and 2) combine citation analysis with other technologies

Similarly, it is important to conduct bibliometric analysis on matched. When comparing research groups, they should be as similar as possible. They should not only be working in the same specialty over approximately the same time period, but they should also be publishing in the same journals, be supported with similar resources, and be situated in similar organizational contexts. The argument here is that the more similar the setting, the more comparable the research output of the groups would be. Comparability would help to lessen weaknesses in the methodology as well as eliminate intervening factors that may impact on the results of the analysis.

23

2. Use it only as a partial indicator

Few would advocate the use of any single quantitative technique to capture all the qualities it attempts to measure. The same is true of bibliometric analysis. Bibliometrics capture a certain aspect of the scientific enterprise. It functions well as a partial indicator of scientific output, productivity, and in some ways, quality or impact. However, it should not be taken to completely represent any of these nuances. Quality of research, in particular, should not be accounted for solely through bibliometric analysis. For measure of quality, while citation counts remain to be the most reliable, convenient measure of quality, they must be used with care and caution. The underlying lesson is that bibliometrics can be a powerful analytical tool when used correctly and in conjunction with other measures, such as patent analysis and findings from peer review.

3. Use it for projects that have a strong science quality

Perhaps more importantly, bibliometric indicators are most useful in evaluating research that is closest to the basic research end of the spectrum. The nature of bibliometrics requires that it be used in situations with a strong science quality. It performs much more weakly with applied scientific output. Applied research is poorly represented in the bibliometric data. The activity that bibliometrics measure – scholarly publications and citations, is more familiar in the basic science realm than in the applied and development realm. Thus, the most accurate results and conclusions may be drawn

24

from analysis of basic research output, or research with a strong basic research component.

4. Do not try to account for quality with bibliometrics

One of the most frequent criticisms of bibliometric analysis is that it mistakenly attempts to account for and explain the quality of scientific pubications. This is an area that requires a great deal of caution on the part of the researcher. There are several different weaknesses of the databases utilized in bibliometric analysis that make accounting for quality very problematic. Two of the most important factors are that the Scientific Citation Index primarily covers U.S. journals and only many of the ones with very high circulation. Secondly, it is difficult to judge whether a work is being cited because of its high quality, low quality, or any other reason.

5. Provide information about what data indicate and what they do not when presenting evaluation results

When presenting bibliometric findings, it is important to provide a realistic setting for those findings. It would be inaccurate to attribute too much credence to bibliometric findings alone. Thus, bibliometric results should be presented along with the strengths and weaknesses of the data and the methodology utilized to obtain the results.

25

REFERENCES

1. Moed, H.F.; W.J.M. Burger, J.G. Frankfort and A.F.J. Van Rann, 1985. The Use of bibliometric data for the measurement of university research performance. Research Policy 14 (1985) 131-149.

2. Beckmann, M. and O. Persson, 1998. The Thirteen most cited journals in Economics. Scientometrics, Vol. 42, No. 2 (1998) 267-271.

3. Rajeswari, A.R. 1989. Forecasting of Science & Technology Expenditure of India by Simulation Method. Scientometrics, vol. 17, Nos 3-4 (1989) 227-251.

4. Singh, Prithipal and V.S.R. Krishnaiah, 1989. Analysis of work climate perceptions and performance of Research and Development Units. Scientometrics, Vol. 17, Nos. 3-4 (1989) 333-351.

5. Satyanarayana, K. and K.V. Ratnakar, 1988. Authorship paterns in life sciences, Preclinical basic and clinical research papers. Scientometrics, Vol. 17, Nos. 3-4 (1989) 363-371.

6. Leydesdorff, L. 1988. The Science Citation index and the measurement of national performance in terms of numbers of scientific publications. Scientometrics, Vol. 17, Nos. 1-2 (1989) 111-120.

7. Pouris, A. 1989. A Scientometric Assessment of Agricultural Research in South Africa. Scientometrics, Vol. 17, Nos. 5-6 (1989) 401-413.

8. Collins, Peter and Suzanne Wyatt 1988. Citations in patents to the basic research literature. Research Policy 17 (1988) 65-74.

9. Narin, Francis and Richard P. Rozek, 1987. Bibliometric analysis of U.S. pharmaceutical industry research performance. Research Policy 17 (1988) 139-154.

10. Johnes, Geraint 1987. Determinants of research output in economics departments in British universities. Research Policy 17 (1988) 171-178.

11. Patil, S.; S.R. Das; G. Ranglani; I.M.L. Ahuja; Y.P. Sehgal & A.K. Pal, 1998. Productivity of CSIR Senior Research Associates – A Review. Productivity Vol. 38, No.4, January-March, 1998.

26

COST -BENEFIT APPROACH FOR EVALUATING RETURN TO AGRICULTURAL RESEARCH

INVESTMENT

India has one of the largest and institutionally matured agricultural research system of the world. A huge amount of funds are being spent on agricultural research. Among developing countries, India's research expenditure is exceeded only by Brazil. The GOI through several agencies provides about 60% of all research funds, State Government about 20%, private companies about 12% and foreign agencies provide the rest (World Bank, 1990).

With accelerated growth of agriculture, new and multiple problems are emerging. The existing agricultural research system is under great strain as the future research is going to be more complex, more difficult and more costly. On the other hand, the funds allocated to agriculture research becoming scarce with respect to the requirement. Not only this, the allocations made in research are now subject to various economic and social evaluations for their outcomes. Under such circumstances when funds are falling short and cost to evolve new technology is increasing, there is a need for assessment and make choices among alternatives where economic principles and tools have proved to be useful. The economic evaluation helps to compare returns from investment made in alternative research project.

27

Securing and maintaining the domestic political support for the public sector component and translating it into financial support for technology development in agriculture is a fundamental issue. The policy makers who allocate funds for agricultural research want to be convinced that the scarce public resources had been judiciously and profitably deployed to earn the maximum possible returns from each rupee invested.

India is a large and diverged country with substantial differences among regions in their natural endowments as well as social and economic conditions. The new technologies responsive to the natural resources base as well as socio and economic forces play a crucial role in inducing productivity growth patterns. It influences changing resource productivity in a world of diverse resource endowments, plays a vital role in International competitiveness and trade and in International pattern of growth and development.

CONCEPT DEVELOPMENT

Agricultural research results into evolvement of new technologies, new package/process and generate correct information/ knowledge so that existing factors can produce higher output per unit of their use. In other words, technology led to improvement in productivity of all factors taken together, which is termed as total factor productivity. The change in productivity of total factor over the time period is attributed to new technologies.

The other point that is to be kept in mind is that technology at a point of time does not bear fruits immediately. There exist a time lag between the technology developed and returns to technology. The time lag may vary from commodity to commodity and from one technology package to another technology package. The time lag may to some extend be reduced by extension investments. The average time lag found by various research studies for agricultural technologies is 6-8 years. It is not like that the

28

technology evolved in time T will start yielding return only in the year T+6 but the contribution of technology in period t+1, t+2 goes on increasing and remains constant for three- four years and then starts decreasing. The contribution of new technology in agriculture is effected by the spread of information. Responsibility for bridging the gap between the research worker and the farmer rests primarily with the agricultural extension service.

The whole discussion comes to the point that in agriculture, technology development takes long time and the cost incurred is not a lump some but spread over number of years. Both these points qualify for the application of Cost- Benefit Analysis approach for technology assessment. Now, before we proceed further, let us understand the mathematics behind this approach.

Cost -Benefit Analysis:

Cost -Benefit analysis is a decision making tool for investment choice with respect of total costs and total benefits of the technology. It is regarded as a tool to measure and then compare the cost and benefits from alternative technologies. Nevertheless, all the costs and benefits are difficult to quantify in financial terms reason being a new technology may have negative and /or positive side effects on the quality of life of the society and environment. Accordingly it requires to be accounted for the assessment. How to account for the social benefits and costs [negative side effects] in this analysis will be discussed latter? But first of all let us consider only quantifiable direct benefits and costs for the simplicity of understanding.

This assumption will make our job easier to develop a flow chart of costs and benefits from technology over its viable period, which is long enough in duration. In long time investments, it is implicitly involved that (i) future payments are uncertain (ii) returns may not be in equal installments and (iii) cost may go up because of inflation. It may face

29

challenge from the alternative use of money i.e. opportunity cost. Therefore, the cost and benefit flow of a technology required to be discounted for future values and compounded for past values to calculate present value. This is an important component of Cost-Benefit Analysis.

Criteria for applying Cost-Benefit Analysis:

- Long term project- Costs & benefits are spread over number of years- Time lag between investment & benefits- Alternative projects/activities- Quantitative Benefits

Costs-benefit Analysis: Steps

1. Calculating stream of costs2. Making adjustment to costs in view of indirect costs3. Estimating stream of benefits4. Making adjustment to benefits in view of indirect benefits5. Estimating time lag & discount rates/ opportunity cost6. Calculating summary measures7. Making decision

Now, taking the steps of cost benefit analysis one by one in the following section.

A. Cost of Research: Issues

1. Types of Research2. Composition of Research Expenditure3. Segregation and Aggregation4. Database5. Estimating real value6. Social cost of research

B. Composition of research expenditure:

30

- Types of researchBasic researchApplied researchMaintenance research

- Heads of expenditure Salary Operational Capital- Heads of budget Plan expenditure Non Plan expenditure Donation & grants

C. Segregation and Aggregation: Research Expenditure

Segregation:

- Commodity-wise R. Expenditure: (Evenson, 1982) gave bibliometric analysis to separate commodity wise research expenditure in multi-commodity by counting the proportion of number of publications in each group of commodities.

- Research and Non Research Activities- Technology-wise R. Expenditure

Aggregation of R. Expenditure:

- Spatial aggregation- Aggregation of Heads of R. Expenditure (Service flow of capital, operational expenses and salaries)

D. Databases of R. Expenditure:

* Annual Accounts of ICAR* R & D Statistics of DST* CFRA of the Central and State Govt.* Annual Accounts of SAUs & Institutions

31

E. Social Cost of R. Expenditure:

- Browning (1987): 1.2 to 1.7 times the actual amount spent- Fullerton (1991): 1.07 to 1.24 times the actual amount spent- Typically, economists: Social cost is the amount spent.

Research Benefits:

Expected benefits of a Research: Technology or policy

At production level

* Improve productivity of production factors* The spread of crop in non-traditional areas Reduce harvest and post-harvest losses Improve product quality

At process level

* Value added to products* Manufacture of new products* Quantitative increase

While calculating the benefits of agricultural research, following methods are popularly recommended:

(i) Economic Surplus approach(ii) Econometric approach(iii) Total factor productivity

Economic Surplus Method:

The economic surplus method has been used to model and measure the economic effects of research-induced technical changes in market settings. It works on the economic principles of consumer and producer surplus generated because of shift in supply function as a result of increase in production due to high yielding technologies.

32Q0

P0

D

S0

S1

Q1

P1

d

a

b

cD= - PC

S0= + PP

S1= + (PP+k) orS1= ( +k)+ PP

Economic Surplus Model

The total economic surplus generated because of shift of supply curve from So to S1 is equal to

However, there are problems of estimating some of the variables e.g. it is difficult to collect information on demand and supply elasticity. It is assumed that ‘e’ (supply elasticity) is zero and ‘h’ (demand elasticity) is infinity. In that situation whole model reduce of simple model where economic benefits are equal to P0Q0ak, which could be calculated as given below.

Research Benefits: Simple model

33

1.Yield for period t-1 = Yt-12.Yield for period t = Yt3.Absolute Change in yield = Yt - Yt-14.Proportionate change in yield = (Yt - Yt-1)/ Yt-15.Impact of research on yield = K %6.Research induced change in yield = {(Yt - Yt-1)/ Yt-1}K = Jt7.Total output benefits = Jt * Qt-18.Total value of benefits = Jt * Qt-1* Pt9.Total output benefits = Jt * Qt-110.Realized output benefits = αt* Jt * Qt-111Total value of benefits = αt * Jt* Qt-1* Pt

Total benefits = P0Q0ak(1 + 1/2Zh) Where Z = ke/(e + h)Where P0 = initial price/ unit; Q0 = initial production; e = elasticity of supply; h = elasticity of demand; a = adoption rate

Calculation of summary measures:

If Bj is represents flow of benefits and Cj represents flow of costs, the present value of benefits (PVB) and present value of costs (PVC) are calculated as:

Bj CjPVB = ------------- and PVC = -------------- (1 + i)j (1 + i )j

where i= discount rate; and j= viable period of technology.

Both these values separately does not evolve any decision making information. Therefore, Net Present Value (NPV) or sometime called Net Present Worth (NPW) is calculated which is equal to :

Bj -Cj NPW = ------------- (1 + i)j

and the technology is said to be economically viable if NPW > 0 and i > market rate of interest. Another parameter which can be derived from this formula is the Internal Rate of Return (IRR). The IRR is the discount rate that give a NPW equal to zero i.e.

Bj Cj ---------- - --------- = 0 (1 + i)j (1 + i)j

This IRR is then compared with minimum acceptable rate of return and if it is either higher than or equal to the minimum acceptable rate of return, then the technology is assessed to be superior/desirable. If the IRR is 18%. This means that at a discount rate of 18%, the project just breaks even i.e it will earn back all the capital and operating costs

34

extended upon it and pay, 18% for the use of money in the meantime.

The simplest to all these parameters to understand in the analysis is Benefit - Cost Ratio (B/C). It is a ratio of PVB to PVC of a technology under assessment.

Bj/(1 + i)j

B/C Ratio = ---------------------- Cj/(1 + i)j

In other words, it is the return to one rupee invested. Therefore, B/C ratio should be more than unity.

It is to be further added that because IRR takes into account the time and amount of installment of benefits, it is recommended to be used in making decisions in ex- post evaluation studies. Whereas, NPV is preferred in ex- ante studies.

Accounting for Indirect and Side Effects: The new agricultural technologies have its direct and indirect benefits and/or positive and negative side effects. The development of high yielding variety of rice, not only increase production and income of the farmer which are the direct benefit but also increase indirectly employment, consumption and education level. On the other side, the use of insecticides increase yield by controlling the attack of insects and pests as a direct benefit but at the same time create environment and water pollution which lead to health hazards and cause society to bear additional cost on medicines. The numerous examples of agricultural technologies can be cited for their direct and indirect benefits and/or positive and negative side effects.

1. One of the method for counting negative side effects is to consider positive effects as benefits and the negative effects as costs. There we may face a problem of quantification in

35

financial terms of all the costs and benefits. Under these conditions, a close proxy of qualitative parameters is considered for quantitative analysis.

2. The other methods is the selection of discount rate. Generally, discount rate is selected either arbitrarily or equal to the cost of capital i.e. market rate of interest. More accurately discount rate can be selected on the basis of the nature of indirect benefit and side effect of a technology. If the technology has a positive side effects, the discount rate of benefits may be decreased and the vice-versa. The discount rate so calculated is called Social Discount Rate.

3. The indirect positive or negative benefits of a technology can also be taken care off if monetary benefits are multiply with a constant factor. The factor has value greater than one when it has positive side effects and less than one when it has negative side effects. However, the value of the constant factor is decided by a group of experts drawn from multiple disciplines for taking comprehensive stock of the situation.

4. The other method of accounting for indirect benefits is to use shadow prices of costs and benefits. These are prices, which are most likely to prevail at equilibrium level in the market. However, it is quite unlikely that equilibrium exist in the market due to various economic and social factors. The situation is more adverse for the occurrence of shadow prices in underdeveloped economies rather in developed economies. It is also equally difficult to estimate the shadow prices. Nevertheless, an attempt is made to estimate price nearer to equilibrium using opportunity cost or the marginal productivity.

References:

36

1. Little,I.M.D. and J.A. Mirlees. Project Appraisal and Planning for the Developing Countries. London, Heinmann Educational Books, 1974.

2. Gittinger, J.P. Economic Analysis of Agricultural Projects. Baltimore and London, The Johns Hopkins University Press. 1982.

3. Norton, George W. and Jeffrey S. Davis [1981]. Evaluating Returns to Agricultural Research: A Review. Amer. J. Agr. Econ.[Nov. 1981]: 685-699.

4. Tylor, C.R. [1980] The Nature of Benefits and Costs of Use of Pest Control Methods. Amer, J Agr. Econ. 62 [1980]: 1007- 1011.

5. Dar, Usha [1966] Benefit- cost Evaluation of Technology change in Agriculture. Indian J. agric. Econ. 21(1): 131-136.

6. Evenson, R.E 1982. Observations on Brazilian Agricultural Research and productivity. Revista de Economica Rural 20 (3) (July/ Sept.)

7. Alston, J.M.; G.W. Norton and P.G.Pardey 1995. Science under scarcity: Principle and practice for agricultural research evaluation and priority setting. Cornell University Press. Ithaca and London.

37