It should be noted in this connection that the indicators in these models are defined at the level of the university or university of applied science. In other words, they refer to matters organized at the level of the institution as a whole, such as HR policy or relations with other organizations, for which the individual research groups within the institution are not responsible. It is not clear at the moment whether adding up all the relevant indicators for individual research projects would lead to an informative, representative picture of valorisation at the level of the university or university of applied science as a whole.
NWO stresses the importance of knowledge utilization
The Netherlands Organization for Scientific Research (NWO) has many different research financing schemes, ranging from the Applied Research (Praktijkgericht Onderzoek SIA) programme for universities of applied science and the Force of Gravity (Zwaartekracht) programme for influential innovative research by consortia of top-class researchers to investment schemes to permit the acquisition of large pieces of equipment and research grants for individual excellent researchers. A number of these NWO financing schemes focus on valorisation, such as the Responsible
Innovation (Maatschappelijk Verantwoord Innoveren) programme and the programmes relating to the Dutch government’s “top sectors” policy. There are also programmes which, while not explicitly aimed at promoting better knowledge utilization, have this as one of their criteria – in particular the Veni, Vidi and Vici grants for talented, creative researchers set up within the framework of the Innovational Research Incentive Scheme.
NWO defines knowledge utilisation as the process promoting the use of scientific knowledge outside the scientific world and by other scientific disciplines. NWO regards the use of knowledge within the scientific field, but outside one’s own discipline, as a form of knowledge utilisation too. It is the only Dutch organisation to explicitly formulate this type of knowledge utilisation.
NWO states that knowledge utilisation often demands interaction between the researcher and the intended user of the knowledge in question. This interaction may occur at any stage of the research, from the formulation of the initial research question to dissemination of the results. By pointing this out, NWO indicates its agreement with the idea that valorisation, or knowledge utilisation, is a process, and that it does not only occur towards the end of a research project. NWO asks applicants for research grants to state whether potential users are involved in the research process, and if so how and when they are involved. The applicant should also mention the expected benefits of the research for society.
Each of the sectors of NWO adds its own explanatory comments to the basic concept of knowledge utilisation, to reflect the view that the precise meaning of knowledge utilisation depends on the discipline involved.
SEP focuses on societal relevance
The Standard Evaluation Protocol (SEP) developed by VSNU, NWO and the Royal Netherlands Academy of Arts and Sciences (KNAW), lays down the criteria used for the retrospective or ex post evaluation, every six years, of the research performed in the various departments and research institutes of Dutch universities, and in the various research institutes governed by NWO and KNAW. VSNU, NWO and KNAW publish a new SEP protocol every six years. The latest is SEP 2015-2021 .
Societal relevance (valorisation) has long been included as a criterion in the SEP, but the interpretation of this criterion has been found to give rise to many questions in practice. SEP 2015- 2021 does not go into detail about relevance to society, as it is currently referred to. In connection with the quality, scale and relevance of the contributions made to specific economic, social or cultural audiences, of advisory reports for policy-makers, of contributions to public debate and the like, the protocol simply states that the contributions should be made to questions, or issues or practices that are relevant for the research unit.
SEP 2015-2021 includes a table that is clearly inspired by a number of KNAW reports (see the section on the KNAW below). This Table D1 mentions two aspects of quality: research quality and relevance to society. These two aspects correspond to two of the three criteria in the SEP. Each of these aspects is assigned three evaluation dimensions: products, use and recognition. A number of example indicators are given in the table.
The SEP does not include any clear definition of valorisation or societal relevance, but it is clear reading between the lines that the three crucial elements of valorisation apply here too. The nature of valorisation as a process is reflected by several of the example indicators, while the explanatory notes and the examples given in the SEP refer to not only the economic but also the societal and even the cultural use of research results. Furthermore, the SEP intentionally avoids including an exhaustive list of indicators in Table D1; instead, users are requested to add further indicators that are appropriate to the research in question.
The SEP also asks for a narrative description of the societal relevance or impact of the research in question. The indicators from Table D1 are intended to support this narrative. A description of the approach taken by the research group is crucial in this context. The idea underlying this question is that the impact or relevance of the research does not come about automatically. It might appear to do so (“we happened to receive a request”), but such requests are usually the result of preparatory work or contacts. The need to write a narrative description may help researchers to become aware of the often initially subconscious active role they played in ensuring that their research would have an impact.
Valorisation is a very wide field – how can you evaluate it?
The broad definition of valorisation used here has far-reaching consequences. It provides an excellent opportunity for researchers to determine what valorisation means for their discipline or in their research institute. On the other hand, it does lead to a great many questions, both among those who are evaluating the research and among the researchers and the research groups being evaluated.
Fortunately, there are answers to these questions. In the first place, few if any indicators are specified in the VSNU proposal, in the information about NWO’s Innovational Research Incentive Scheme or in the SEP. Instead, users are explicitly invited to indicate what valorisation means in their specific situation. They are free to determine which indicators are applicable to their particular case, and can include qualitative indicators as well as quantitative ones.
Furthermore, there have been various projects in the Netherlands aimed at evaluating valorisation, impact or the societal quality of research . The KNAW has produced a number of reports on research assessment, which form the basis of the current SEP. These reports also do not give exhaustive lists of indicators – not even short lists of the only right indicators. They do however provide extensive explanatory notes on social quality.
These KNAW reports are largely based on the results of the Dutch ERiC (Evaluating Research in Context) project. The context in question is a wide one, covering the discipline involved, the organisation responsible for the research and other parties taking part in the research. These details are highly relevant, because it makes a big difference whether the research is monodisciplinary or interdisciplinary, whether it is carried out in a university of technology or a mission-driven social science institute, and whether it is carried out with and for scientific peers or in a consortium that also includes societal and industrial or commercial players.
In line with this, the first questions asked in the ERiC project are: What is the objective of the study, and what is the context? The answer to these questions determines whether the evaluation of the research can be based exclusively on traditional quality indicators such as scientific publications and citations, or whether it is also necessary to include other aspects, such as the nature of the involvement of societal partners.
KNAW develops quality criteria in specific sectors
The Royal Netherlands Academy of Arts and Sciences (KNAW) is the forum, conscience and voice of Dutch science, and advises the Dutch government on issues of importance to science in the Netherlands. The KNAW has published three advisory reports on research evaluation in recent years, dealing with technical sciences (with special reference to their design and constructional aspects), the humanities and the social sciences. These reports were produced in response to the need for more specific, discipline-oriented quality criteria in these disciplines. Each report contained a brief proposal concerning research evaluation criteria.
Although each of these reports has a different structure, there are many similarities between the approaches taken. The intention in each case was definitely not to develop valorisation indicators, but to formulate appropriate quality criteria for scientific research. The Academy members always came to the conclusion that there are two relevant aspects of quality: the scientific and the societal.
The main stress in these reports – particularly in the proposals for the humanities and the social sciences – seems to be on the use to which research results are put and their impact on external target groups rather than on the cooperation between the various parties involved. However, reading between the lines it may be concluded that a reference to cooperation with social partners is hidden in the quality criteria “products for external target groups” and “use of output by society”.
These reports are also in line with the definition of valorisation adopted by the Dutch government. The publication of separate reports for different disciplines reflects the conviction that different valorisation approaches need to be taken or can be taken in different disciplines. The insight that valorisation is a process is not stated explicitly in two of the three KNAW reports, but is implicitly present. The proposed quality criteria reflect a broad view of valorisation, stressing its societal as well as its economic aspect.
Many people still recall the memorable motto “Skills and know-how mean more cash in the till” coined by Halbe Zijlstra, State Secretary in the Dutch cabinet from 2010 to 2012. Nevertheless, the Dutch government, in particular the ministry of Education, Culture and Science, have been using a much wider definition of valorisation for more than a decade now.
Societal impact is just as important as economic impact. Valorisation is an interactive process and not just a simple kind of linear leapfrog. It aims at both economic and societal impact, and can be tailored for use in each discipline.
Each of the Dutch umbrella organizations Vereniging Hogescholen, VSNU, NWO and KNAW takes its own slightly different view of valorisation – or knowledge utilisation, or relevance to society, whichever name they choose to give to this concept. But in fact the resemblances between their views are greater than the differences. What matters is not just that skills and know-how lead to economic benefits, but that research can have different kinds of impact, achieved via an appropriate process that will differ from one discipline to another. It follows that it is up to the researchers themselves to determine what kind of impact they are aiming at, and what indicators they need to evaluate their results.