Pragmatic Free Trial Meta
Pragmatic Free Trial Meta is a free and non-commercial open data platform and infrastructure that supports research on pragmatic trials. It collects and distributes clean trial data, ratings and evaluations using PRECIS-2. This allows for a variety of meta-epidemiological analyses that evaluate the effects of treatment across trials of different levels of pragmatism.
Background
Pragmatic trials provide evidence from the real world that can be used to make clinical decisions. The term "pragmatic", however, is a word that is often used in contradiction and its definition and evaluation require further clarification. Pragmatic trials must be designed to inform clinical practice and policy decisions, rather than confirm a physiological or clinical hypothesis. A pragmatic study should try to be as similar to the real-world clinical environment as is possible, including its selection of participants, setting up and design, the delivery and execution of the intervention, determination and analysis of the outcomes, and primary analyses. This is a significant difference between explanatory trials as described by Schwartz & Lellouch1 which are designed to prove a hypothesis in a more thorough way.
Truly pragmatic trials should not be blind participants or the clinicians. This can result in an overestimation of the effects of treatment. The pragmatic trials also include patients from various health care settings to ensure that their results can be applied to the real world.
Finally studies that are pragmatic should focus on outcomes that are important for patients, such as quality of life or functional recovery. This is especially important for trials that involve the use of invasive procedures or could have serious adverse consequences. The CRASH trial29 compared a two-page report with an electronic monitoring system for patients in hospitals with chronic heart failure. The trial with a catheter, however was based on symptomatic catheter-related urinary tract infection as the primary outcome.
In addition to these aspects pragmatic trials should also reduce the requirements for data collection and trial procedures to cut costs and time commitments. Finally, pragmatic trials should seek to make their results as relevant to actual clinical practice as is possible by ensuring that their primary analysis is the intention-to-treat approach (as described in CONSORT extensions for pragmatic trials).
Many RCTs which do not meet the requirements for pragmatism however, they have characteristics that are contrary to pragmatism have been published in journals of varying types and incorrectly labeled as pragmatic. This can lead to false claims of pragmaticity, and the use of the term must be standardized. The development of a PRECIS-2 tool that provides an objective and standardized evaluation of the pragmatic characteristics is a good start.
Methods
In a pragmatic research study the aim is to inform policy or clinical decisions by showing how an intervention could be integrated into routine treatment in real-world settings. Explanatory trials test hypotheses concerning the cause-effect relation within idealized environments. Consequently, pragmatic trials may be less reliable than explanatory trials and might be more susceptible to bias in their design, conduct and analysis. Despite their limitations, pragmatic studies can provide valuable information for decision-making within the healthcare context.
The PRECIS-2 tool scores an RCT on 9 domains, with scores ranging between 1 and 5 (very pragmatic). In this study the areas of recruitment, organisation and flexibility in delivery, flexible adherence and follow-up received high scores. However, the main outcome and method of missing data was scored below the pragmatic limit. This suggests that it is possible to design a trial with excellent pragmatic features without damaging the quality of its outcomes.
However, it is difficult to judge how practical a particular trial really is because pragmatism is not a binary attribute; some aspects of a trial can be more pragmatic than others. Additionally, logistical or protocol modifications during the course of an experiment can alter its score in pragmatism. Additionally 36% of the 89 pragmatic trials discovered by Koppenaal et al were placebo-controlled, or conducted prior to licensing, and the majority were single-center. Therefore, they aren't very close to usual practice and can only be described as pragmatic in the event that their sponsors are supportive of the lack of blinding in such trials.
Furthermore, a common feature of pragmatic trials is that researchers attempt to make their findings more valuable by studying subgroups of the sample. However, this often leads to unbalanced comparisons with a lower statistical power, which increases the chance of not or misinterpreting differences in the primary outcome. This was a problem in the meta-analysis of pragmatic trials due to the fact that secondary outcomes were not adjusted for covariates that differed at the time of baseline.
In addition, pragmatic studies may pose challenges to collection and interpretation of safety data. This is because adverse events are typically reported by participants themselves and are susceptible to reporting delays, inaccuracies or coding deviations. It is essential to improve the quality and accuracy of outcomes in these trials.
Results
Although the definition of pragmatism does not mean that trials must be 100 percent pragmatic, there are advantages to including pragmatic components in clinical trials. These include:
Enhancing sensitivity to issues in the real world as well as reducing cost and size of the study and allowing the study results to be faster translated into actual clinical practice (by including routine patients). But pragmatic trials can be a challenge. For instance, the right type of heterogeneity could help a study to generalize its findings to a variety of settings and patients. However the wrong kind of heterogeneity can reduce assay sensitivity and therefore decrease the ability of a study to detect small treatment effects.
Several studies have attempted to classify pragmatic trials using a variety of definitions and scoring methods. Schwartz and Lellouch1 developed a framework to distinguish between explanatory studies that prove the physiological hypothesis or clinical hypothesis and pragmatic studies that help inform the selection of appropriate treatments in the real-world clinical practice. The framework was comprised of nine domains that were scored on a scale of 1-5, with 1 indicating more lucid and 5 indicating more pragmatic. The domains were recruitment and setting, delivery of intervention, flexible adherence, follow-up and primary analysis.
The original PRECIS tool3 was based on a similar scale and domains. Koppenaal and colleagues10 created an adaptation of the assessment, known as the Pragmascope, that was easier to use for systematic reviews. They discovered that pragmatic systematic reviews had higher average scores in the majority of domains but lower scores in the primary analysis domain.
This distinction in the primary analysis domain could be due to the fact that the majority of pragmatic trials analyse their data in the intention to treat manner, whereas some explanatory trials do not. The overall score for systematic reviews that were pragmatic was lower when the domains of management, flexible delivery and following-up were combined.
It is important to remember that a study that is pragmatic does not mean that a trial is of poor quality. In fact, there is an increasing number of clinical trials which use the term 'pragmatic' either in their abstract or title (as defined by MEDLINE but which is neither precise nor sensitive). The use of these words in abstracts and titles could suggest a greater awareness of the importance of pragmatism, but it isn't clear if this is evident in the contents of the articles.
Conclusions
In recent times, pragmatic trials are increasing in popularity in research because the importance of real-world evidence is becoming increasingly acknowledged. They are clinical trials that are randomized which compare real-world treatment options instead of experimental treatments under development. They have patient populations that more closely mirror those treated in routine care, they use comparisons that are commonplace in practice (e.g. existing medications), and they rely on participant self-report of outcomes. This method can help overcome the limitations of observational research, for example, the biases that come with the use of volunteers as well as the insufficient availability and coding variations in national registries.
Pragmatic trials offer other advantages, such as the ability to use existing data sources, and a greater chance of detecting significant differences than traditional trials. However, pragmatic tests may be prone to limitations that undermine their effectiveness and generalizability. The participation rates in certain trials could be lower than anticipated due to the healthy-volunteering effect, financial incentives or competition from other research studies. A lot of pragmatic trials are limited by the need to enroll participants in a timely manner. Certain pragmatic trials lack controls to ensure that any observed differences aren't due to biases during the trial.
Highly recommended Internet site of the Pragmatic Free Trial Meta identified 48 RCTs self-labeled as pragmatic and were published from 2022. They evaluated pragmatism using the PRECIS-2 tool, which includes the eligibility criteria for domains, recruitment, flexibility in intervention adherence and follow-up. They discovered that 14 of the trials scored as highly or pragmatic practical (i.e. scoring 5 or higher) in any one or more of these domains, and that the majority of these were single-center.
Trials with high pragmatism scores are likely to have more lenient criteria for eligibility than traditional RCTs. They also include populations from various hospitals. The authors claim that these characteristics could make pragmatic trials more effective and applicable to everyday practice, but they do not necessarily guarantee that a trial using a pragmatic approach is free from bias. In addition, the pragmatism that is present in a trial is not a fixed attribute; a pragmatic trial that doesn't contain all the characteristics of an explanatory trial can produce valuable and reliable results.