Predicting the likelihood of questionable research practices in clinical trials: a systematic and semi-automated approach

Biomedical research is in an existential crisis, and many clinical trials have systematic methodological flaws, statistical problems, and their results may be biased, exaggerated, and difficult to reproduce. This is highly problematic since clinical trials constitute the backbone of evidence-based medicine. Systematic reviews and clinical guidelines are based on the results of clinical trials, and medical doctors rely on them to determine which treatments are used in clinical practice. Questionable research practices (QRPs) play a key role in this crisis. There are major flaws in the quality assurance of modern research. Too often the methodological quality of clinical trials is insufficient, mandatory pre-registration absent or flawed, or the reported results biased. These QRPs may significantly hamper the validity and reliability of clinical research. Even though knowledge on the prevalence of these QRPs in clinical research is abundant, its predictors remain largely unknown. This is problematic as sound knowledge about these predictors can guide QRP detection and prevention strategies. This project therefore aims to identify possible predictors of QRPs in more than 100,000 clinical trials which are part of systematic reviews within meta-analyses. Factors related to the validity and reliability of clinical trials can be used to predict which existing and future trials are at risk for QRPs. This is important since it may take time before extensive quality outcomes have been carried out, and inclusion in (Cochrane) meta-analyses may take several years. With our proposal, it may be possible to predict QRPs even in absence of or prior to extensive (and often manual) quality checks have been carried out. This project aims to identify predictors of QRPs at different levels in more than 100,000 accessible clinical trials that have been included in Cochrane’s systematic reviews. In collaboration with the Dutch Cochrane Center, predictors will be semi-automatically collected from existing databases using state-of-the-art machine learning and natural language processing tools. We will characterize the following three clusters of QRPs: 1) methodological rigor, 2) complete and correct reporting, and 3) statistical rigor. Standardized predictors of QRPs in individual trials will include information at the level of 1) researcher, 2) institution of the authors, 3) clinical trial, and 4) journal. We will build a multivariable prediction models per QRP and present applicable models in an easy-to-use and open access website for individual or batch trial classifications. All data is available from existing databases but has hitherto never been combined. Models will be internally and externally validated to guarantee accurate QRP predictions. As a proof-of-concept leading to this proposal, we already automatically extracted data related to several possible QRP predictors and outcomes (e.g. statistical power, gender, and collaborations). This proposal stems from a collaborative effort across different universities and departments. Good research practices are essential to guide responsible decision making in daily practice. The results of this project are, however, not only relevant for the clinical or even the biomedical field, but are likely to apply to research practices in other fields as well, including the social sciences and humanities. We believe that this large-scale project will increase our understanding of factors that predict questionable and responsible research practices in clinical trials. The possibility to improve quality interpretation of existing and future clinical trials is without doubt of great importance.