Improving Peer Review: interventions that work. (IMPER)

The effectiveness of self-regulating mechanisms is crucial to the scientific enterprise. It was long thought that misconduct in science could hardly occur at all, as any form of misconduct would sooner or later come to light due to the peer review system, replication studies or the presence of a whistle-blower. Among these, the peer review system in particular was considered the best available gatekeeper of both quality and integrity in science. However, several recent studies suggest that peer review is under threat. They demonstrate that faulty and even fraudulent research slips through editorial peer review at alarming rates. Numerous measures have been proposed to improve the quality of the peer review system, but their effectiveness remains uncertain. This study will assess the effectiveness of measures used by scientific journals to improve peer review’s ability to detect misconduct and error. It will do so, first, by identifying which innovative peer review practices are being suggested and implemented. Second, IMPER will examine how effectively these review practices spot problematic publications with bibliometric analysis techniques. Retractions of journal publications will be used as an indicator of problematic publications. The project will hence set up a database of retractions, identifying the reason for retraction and the form of peer review that was employed to assess the retracted article. Analysis of this database will provide insight in the effectiveness of peer review practices in detecting erroneous research records. Third, IMPER will analyse the practicality and unexpected side effects of measures to improve peer review through qualitative research. This may include concerns about the costs of implemented measures as well as their potential of reducing the burden on the peer review system, which is suggested to be overloaded. The results will be of great practical value to journal editors, publishers, and funders, ultimately contributing to improved research integrity. An assessment of the peer review innovations and their effectiveness to detect misconduct will allow editors and publishers to make informed alterations to journal peer review processes in order to improve its quality and reliability. It will allow funders to consider conditions for publication of results and potentially produce suggestions that might be useful to improve review of funding proposals. Several steps will be undertaken to ensure that IMPER’s results will effectively lead to innovations in the peer review system. Most importantly, the potential consumers of our results, editors and publishers, will be directly involved in several stages of the project. By actively engaging important stakeholders throughout the project, IMPER increases both its visibility as well as it makes sure that the results meet practical desires of its consumers and fits in with their existing routines and frameworks. In addition to the policy recommendations for journal editors and publishers, IMPER will also make valuable contributions to the scholarly research on scientific integrity. It will do so by employing novel methodologies to study the peer review system, providing access to a large database of retractions and providing anchor points for further research on scientific integrity, the peer review system and reviewing mechanisms in general. The project will consist of four distinct tasks: The first task provides an inventory of peer review practices. A systematic overview will be made of practices in journal peer review and proposals that aim to improve the publication system’s quality and ability to detect misconduct. This will be done through extensive review of the scholarly literature, along with qualitative interviews with journal editors and representatives from publishers. The second phase of research involves the identification of erroneous publications and the peer review system they went through. The data collection in this phase will comprise of: (a) retracted journal publications, (b) the form of peer review in the publishing journal, (c) characteristics of the journal (e.g. rejection rates, impact factor, citation profile). In the third phase, this data set will be analysed to study how the occurrence of retractions is linked to the forms of peer review. A relatively large numbers of retractions for a given form of peer review, compared to the number of non-retracted publications, indicates a low effectiveness of detecting erroneous research. The fourth task comprises the involvement of various stakeholders through a stakeholder panel, workshops and informal feedback sessions. The stakeholders will be invited to comment on the design, goals, preliminary results and dissemination of this study. This will provide better focus for our research, increase relevance and usefulness; provide opportunities to learn from practice, as well as to generate attention for the project.