World-first international guidelines weeds-out potentially critical scientific fraud

The number of retractions issued for scientific research articles in 2023 exceeded 10,000—smashing annual records. To date, publishers have struggled to clean up a slew of papers with serious integrity concerns.

Now a world-first framework for assessing research integrity in clinical guidelines, has been published in eClinicalMedicine.

Monash University researchers in Australia developed the Research Integrity in Guidelines and evIDence synthesis (RIGID) framework.

Clinical trials which lack integrity or present concerns over trustworthiness, can compromise patient care, both directly through unnecessary or harmful treatments, or indirectly through wasted resources and misdirected future medical research.

“It is estimated that at least 25% of clinical trials informing clinical guidelines may not be trustworthy,” according to RIGID co-lead author and research integrity expert, Professor Ben Mol.

In 2023, the RIGID framework was applied to the International Evidence-based Guidelines for Polycystic Ovary Syndrome (PCOS) in collaboration with 39 national and international societies and with input from 80 multidisciplinary experts, and consumers. The PCOS Guidelines have been downloaded over 35,000 times and the framework is now being applied in several other national and international guidelines.

The RIGID framework and accompanying checklist uses a simple six-step approach to determine integrity risk, giving guideline developers, policy-makers, clinicians and scientists a roadmap to assess research integrity, and to exclude untrustworthy research during evidence synthesis and clinical guideline development.

In 2009, the UK-based Committee on Publication Ethics (COPE) released a now widely-adopted policy for handling publication retractions. However, according to Professor Mol, “These policies relate to editorial and publishing processes for retraction and do not provide guidance for researchers and guideline developers on how to handle suspicious or untrustworthy evidence which remains in the public domain, unretracted.”

Retraction practices are also often insufficient, relying on whistleblowers to flag integrity concerns and then authors, their institutions and publishers to respond to these queries, where there is little incentive, processes or systems to do so.

“Queries to journals and authors’ institutions are frequently met with silence or defensive threats of legal action, and official conclusions are seldom reached or formulated euphemistically, with limited retractions.”

According to Professor Helena Teede, the other co-lead author, “The perpetuation of problematic research is underpinned by complex systemic shortcomings, including inadequate application of quality research reporting processes or detection systems; lack of time and resources to investigate claims; lack of incentives for journals, institutions and whistle-blowers; and barriers around reputational or legal implications.

“Most importantly, there is a lack of standardized procedures or protocols with appropriate oversight to manage integrity concerns.”

Key to the RIGID framework is transparency, where studies ranked by the integrity committee as having a moderate or high risk for integrity concerns are clearly documented, and authors are contacted to highlight the identified concerns.

This limits their inclusion in the evidence synthesis, pending clarification. Authors are provided with the opportunity to engage in processes to address these issues, generally within two weeks of contact. This is usually met with little to no response, and the research evidence in question is then not considered in formulating conclusions or guideline recommendations.

In the 2023 International Evidence-based Guideline for PCOS, the framework weeded out problematic studies to ensure only trustworthy evidence informed clinical practice. With the use of RIGID, it turned out that no less than 45% of the RCTs assessed had moderate or high risk of integrity concerns and these could not be trusted to guide practice.

The lead developer of the PCOS guideline, Professor Helena Teede, noted, “It has been approved by the National Health and Medical Research Council of Australia, including the RIGID process, and has been viewed over 130,000 times and presented at over 100 conferences globally.”

“During guideline development, patient representatives and health professionals highly prioritized the importance of only relying on trustworthy research to guide clinical practice. The RIGID framework is now being applied to other international guidelines, including the Premature Ovarian Insufficiency (POI) International Guideline and the Australian adaptation of the European Society of Human Reproduction and Embryology (ESHRE) Unexplained Infertility Guideline,” she said.

The RIGID six step approach:

1. Review: standard systematic review processes;
2. Exclude: studies which have been retracted are excluded, and those with expressions of concern are flagged for further evaluation;
3. Assess: remaining studies are assessed for integrity using an appropriate tool such as the Research Integrity Assessment (RIA) tool or Trustworthiness of Randomized Controlled Trials (TRACT) checklist and allocated an initial integrity risk rating of low, moderate or high risk for integrity concerns;
4. Discuss: integrity assessment results are discussed among integrity committee members with votes to determine final integrity risk rating allocations for each study;
5. Establish contact: low risk studies are included without author contact, whereas authors of studies ranked as moderate or high risk are contacted for clarification;
6. Reassess: studies are reassessed for inclusion using the RIGID author response algorithm (reclassified as ‘included’ where authors have provided a satisfactory response, ‘awaiting classification’ where authors have engaged but time is needed to address concerns, or ‘not included’ where authors have not responded to contact attempts).

Professor Ben Mol: Fraud detective

IN 2011 Ben Mol, a professor of obstetrics and gynecology at Monash University, in Melbourne, came across a retraction notice for a study on uterine fibroids and infertility published by a researcher. The journal which had published it was retracting it because it contained identical numbers to those in an earlier study—except that that one had been on uterine polyps. The author, it turned out, had simply copied parts of the polyp paper and changed the disease.

“From that moment I was alert,” says Dr. Mol. And his alertness was not merely as a reader of published papers. He was also, at the time, an editor of the European Journal of Obstetrics and Gynecology, and frequently also a peer reviewer for papers submitted to other journals.

Sure enough, two papers containing apparently fabricated data soon landed on his desk. He rejected them. But, a year later, he came across them again, except with the fishy data changed, published in another journal.

Since then, he has teamed up with other researchers and led initiatives to investigate groups of papers by authors shown to be data fabricators.

Wherever he saw smoke, he found fire. There were tables on patients’ characteristics that contained only even numbers. There were values that were clinically unlikely. There was an implausible 40:60 sex ratio of babies when the mothers-to-be had, purportedly, been selected at random. Eye-popping speeds of completing clinical trials were common.

Dr. Mol and his colleagues have sent their concerns about more than 900 papers to the journals that published them. But, all too often, either nothing seems to happen or investigations take years.

Only 250 of the studies they have flagged have so far been retracted, and the time to take such decisions is on average two years. Consequently, many have been included in systematic reviews—the sort of research round-ups that inform clinical practice.

The PCOS guidelines, through RIGID, are the first in the world to deal with this problem.