Evaluation of research and development

European Commission Directorate-General For The Information Society And Media Directorate-General For Research And Innovation

Description

Current practice and guidelines
Research policy and organisation

Informations

• Publié par: EUROPEAN-COMMISSION-DIRECTORATE-GENERAL-FOR-THE-INFORMATION-SOCIETY-AND-MEDIA-DI
• Nombre de lectures: 6
• Langue: English
• Poids de l'ouvrage: 1 Mo

Extrait

***
* *
*
Commission of the European Communities
Evaluation of Research & Development
Current practice and Guidelines
Synthesis Reports
Research evaluation EUR 13336 EN >
Commission of the European Communities
Evaluation of Research & Development
Current practice and Guidelines
Synthesis Reports
Authors: W. KRULL
D. SENSI
D. SOTIRIOU
- 1991 -
TJI
aro/-Al%
'vp.m ΌΠ3 Ί VJ Published by the
COMMISSION OF THE EUROPEAN COMMUNITIES
Directorate-General
Telecommunications, Information Industries and Innovation
L-2920 LUXEMBOURG
LEGAL NOTICE
Neither the Commission of the European Communities nor any person acting on behalf
of then is responsible.for the use which might be made of the following
information
Cataloguing data can be found at the end of this publication
Luxembourg: Office for Official Publications of the European Communities, 1991
ISBN 92-826-2232-0 Catalogue number: CD-NA-13336-EN-C
©ECSC—EEC—EAEC Brussels · Luxembourg, 1991
Printed in Belgium CONTENTS
Reflection on Recent Trends in the Field of INTRODUCTION
R&D Programme Evaluation and on the Perspec-
tives of this Study
CHAPTER A : Current Practice and Guidelines In a Selection
of Industrialized Countries
WILHELM KRULL
CHAPTER Β Current Practice and Guidelines In France,
Italy and Greece, and International
Organizations
19 DIMITRIS SOTIRIOU
Educational Aspects of Guidelines for the CHAPTER C
Evaluation of R&D Programmes
47 DINA SENSI
Bibliography of Guidelines and Manuals for APPENDIX I
93 the Evaluation of R&D
95 Glossary of Abbrevlations APPENDIX II
-III-INTRODUCTION - REFLECTIONS ON RECENT TRENDS IN THE FIELD OF R&D PROGRAMME
EVALUATION AND ON THE PERSPECTIVES OF THIS STUDY
In 1989-90, the Commission of the European Communities - as a part of the
Support Programme for the Evaluation Activities In the Field of Research
(SPEAR) - sponsored a study on the preparation of guidelines for the
evaluation of European Community R&D programmes.
A first step, it seemed essential to get an overview of the state-of-the-
art In the field of R&D evaluation activities. Therefore, the study team
conducted a survey, complemented by site visits and interviews, focussing
on current practice and guidelines. All in all, more than 400 experts in
the EC Member States, the Nordic Countries, Austria, Switzerland, Israel,
the United States, Canada, Australia, and in a number of International
organizations were consulted. Thanks to their willingness to cooperate with
us, we were able to obtain illuminating evidence on the state-of-the-art
and to collect a large amount of useful material on R&D evaluation. We are
most grateful indeed to these experts for all their help.
This volume contains the synthesis reports of each of the three members of
the study team. The more detailed reports on current practice and
guidelines in a selection of countries and international organizations as
well as a collection of guidelines and manuals for the evaluation of R&D
were presented to the Commission of the European Communities in separates
volumes.
The reports on current practice and guidelines show that there are a lot of
variations and different approaches to the evaluation of R&D. Evaluation
practices are considerably influenced by the national and/or organizational
context. Economic, cultural and political features as well as
administrative systems have a strong impact on the role of evaluation in
the R&D policy process and the methodological tools considered as
appropriate for being used by evaluators of R&D programmes. Furthermore, it
has, by now, been widely acknowledged that basic, strategic, and applied
research as well as industrial development have different objectives,
outputs and time-scales, which require different ways of measuring and
evaluating the results of R&D programmes. Therefore, successful evaluations heavily depend on the choice of the
methods (i.e. surveys, bibliometric or econometric studies, control group
designs, before-and-after comparisons, and peer review) that are most
appropriate for the particular programme to be evaluated, the type of
evaluators (consultant, expert panel, etc.), and the sensitivity to the
fact that an evaluation is both a scientific and a political exercise (i.e.
It has to be a state-of-the-art evaluation and, at the same time, it should
try to meet the decision-makers' needs for Information about the particular
programmes' effectiveness and efficiency as completely as possible).
Because of the Importance of the economic, political and administrative
context for the Implementation of evaluation strategies, it seems virtually
impossible to summarize the various national and international activities
In the field of R&D evaluation. However, the most recent development shows
some Interesting features which are worth mentioning
o All the international organizations visited and some national
Institutions have developed manuals or guidelines for the evaluation of
R&D. Often, these have been elaborated over a period of several years
(including discussions with all parties concerned and a field trial).
They are intended to provide a common basis for the evaluation activities
of the organization and to serve as an introduction to or a reminder of
what the relevant evaluation issues are. Subsequently, manuals and
guidelines are also designed to standardize management and evaluation
procedures, and to enhance the comparability of the findings presented in
evaluation reports. As many of these guidelines and manuals have been
developed fairly recently, it is still too early to assess how successful
they have been with regard to establishing a certain degree of
comparability and improving the overall quality of evaluations.
o In the 1980s several countries, e.g. the United Kingdom, the Netherlands,
the Nordic countries, France, Switzerland, and Canada, have emphasized
the evaluation of R&D beyond ad hoc studies. Government policy of
Increased accountability and value for money has led to a lot of
structural and institutional changes in order to efficiently distribute
available funds and to identify priority areas. Due to level budgets and
the Increasing cost of R&D, most countries are obliged to be selective in
terms of performers and of the areas of science and technology which they
want to pursue.
-VI-Although In some of the more advanced countries as well as In the
International organizations a more or less continuous move towards
management by evaluations can be observed, in many other cases there still
seems to be a lack of interaction of evaluation with the relevant decision­
making processes (I.e. evaluation results are not systematically used in
the subsequent resource allocation processes).
o Policy makers in almost all Industrialized countries are concerned with
the implementation of programmes stimulating innovation, because these
programmes are considered to be essential for securing long term
economic welfare. So far, however, only limited empirical evidence
supports the assumptions that underlies these policies. As a result, more
and more countries are incorporating evaluations Into new programmes in
order to obtain more Information on how certain policy measures
contribute to the country's innovation capacity. Most of these
evaluations employ the case-study method (I.e. they mainly deal with
particular projects or programmes and follow the development of the
relevant industrial sector for several years; sometimes complemented by
before-and-after comparisons, and control group designs). They are
expected to overcome the limitations of the usual cost-benefit analysis,
which has been used successfully in evaluating short-term development-
oriented measures that directly influence the companies' profits, but
does not seem to be an adequate approach for assessing the Impact of
programmes aiming at the production of new knowledge which eventually
may become the basis for Industrial innovation. A combination of a
qualitative approach (integrating expert-opinion) and the quantitative
methods is required If one wants to evaluate the impact of programmes in
areas like higher education, basic and strategic research, because It is
long-term and cannot easily be attributed to certain measures or
programme activities.
o Many evaluations are carried out by evaluation units within the
organizations. They are mainly concerned with the administrative,
managerial or financial aspects of the programme under review, and thus
indirectly influence the choice, design and management of future
programmes. Often, they are also used for convincing sponsors that the
programmes of the organization have sufficient quality for continued
funding.
-VII-However, meaningful assurance about the quality of a programme is best
developed through external evaluation. Except In rare cases, few
decision-makers will place credence In an Institution's or a programme'
self-proclaimed quality. While self-assessments have the advantage of
being able to enhance