The scientific peer review by referees

Suppose you want to apply for a grant from NWO: what must you do, who decides if your proposal shall be awarded funding, when do you find out if you are one of the lucky ones and what can you do if you disagree with the decision? Each year, NWO receives about 4000 funding proposals of which about 20 to 25 percent are awarded.

What does a referee do?

Referees or peers play a crucial role in NWO's evaluation system. The referee is an independent and unbiased expert who, at the request of NWO, writes a report about the funding proposal on the basis of clearly described selection criteria. This assessment, together with a rebuttal from the applicant, is submitted to the evaluation committee who prioritise the proposals on the basis of this information and issue a reasoned advice about this. Based on this advice, the Governing Board, divisional board or another NWO administrative body subsequently takes the funding decision. NWO has deliberately separated the advisory and decision-taking procedures so as to make the procedure as transparent and objective as possible.

NWO approaches referees for the vast majority of it grants and generally uses at least two per funding proposal. In the case of large grants or funding proposals that cover different scientific disciplines, NWO requests more referee reports. In the case of Vici grants, for example, four referees are often asked to issue a report. In principle, referees are not paid for their contribution.

How is a referee selected?

The policy officer of NWO responsible for supervising the call for proposals selects and approaches the referees and asks them to issue a report about a funding proposal. The selection of referees is a time-consuming and exacting task in which meticulousness, neutrality and integrity are important. Referees must be experts in the specific scientific field of the proposal. In their search for referees, NWO staff use not only their own knowledge and databases but also large international databases such as Web of Science, Medline, but also Google. An officer can also base his or her decisions on other information such as data about speakers and participants at relevant conferences within a certain scientific discipline.

The majority of referees approached by NWO, about two-thirds, come from outside of the Netherlands. There are two reasons for this. NWO wants to do everything possible to prevent conflicts of interest and prejudices of the referees. And secondly, NWO wants the Dutch funding proposals to be compared with international standards.

Before a referee is approached, the policy officer concerned investigates whether the referee has any relationship with the proposal or the applicant. Some of the criteria assessed in this respect are joint publications or other involvement in the work of the applicant. If the referee is a subject expert and can give an unbiased opinion then he or she can be approached by NWO. The referee is requested to assess the proposal against the criteria laid down for the grant.

How is a referee approached?

NWO telephones and/or writes to all potential referees and draws their attention to its Code of Conduct on Conflicts of Interest (see www.nwo.nl/gedragscode) which they should subscribe to. If at this stage there is any hint of a conflict of interest then the referee cannot act as an assessor. For example, researchers from competing institutes are also excluded. NWO promises the referee that his or her anonymity shall be guaranteed and that his or her name shall not be disclosed to the applicant. With this NWO gives the referee the room to respond to the funding proposal in complete honesty. A referee usually has three to four weeks to provide written comments on the proposal and this is usually done via e-mail. The comments are usually provided using an assessment form in which a response is requested to the various assessment criteria. For some funding rounds NWO has also recently introduced the possibility for referees to look at the funding proposal directly online via IRIS and to submit their reaction online as well. Referees are requested to treat the data with due care and confidence so that the intellectual property of the applicant is protected.

In principle, the entire procedure is completed in writing.

May the applicant also respond?

Yes, NWO offers the applicant the opportunity to respond to the content of the referees' reports. NWO considers this rebuttal to be an important part of the process. The rebuttal gives the researcher the opportunity to refute differences in insight and misinterpretations of the referee. This approach from NWO is unique within Europe, but it is a method that is increasingly being adopted elsewhere. Even if the referees issue a negative assessment, a well-argued rebuttal can, in principle, still be enough to convince the evaluation committee otherwise so that a grant can still be obtained. The referee reports received are made available to the applicant. In principle, no changes are made to the texts containing the referees' comments. However, if the referee reveals his or her identify in the comments then these passages are modified such that the identity is not revealed but the content remains unchanged.

The applicant is usually given one week in which to respond to the referees' reports. The proposal, the referees’ comments and the rebuttal from the applicant are then sent to the evaluation committee. This committee is responsible for comparing the proposals submitted.