The European Network for Health Technology Assessment (EUnetHTA) recently published two methodological guidances: one for manufacturers and one for assessors on Direct and Indirect Comparisons.
This guidance is broadly focused on evidence of the relative efficacy or effectiveness of technologies, and randomized controlled trials (RCTs) are noted as the preferred source of comparative evidence. However, the guidance does acknowledge that non-randomized studies (e.g., observational studies or single-arm studies) may be submitted even though they have a higher risk of bias and that “the results provided by these techniques remain controversial.” Within these guidance documents, EUnetHTA offers some recommendations on planning real-world evidence (RWE) studies for health technology assessments (HTA) decision-making.
Here we outline the key takeaways from the EUnetHTA’s guidance and discuss how these recommendations fit into the larger context of European HTA decision-making and RWE best practice recommendations.
EUnetHTA and its impact on European reimbursement decision making
EUnetHTA is a network of over 80+ organizations in thirty European Countries that is focused on producing sustainable HTA. This goal is realized through connecting HTA bodies and “enabling an exchange of information to support policy decisions.” In order to reduce duplicative efforts, EUnetHTA engages in a broad range of activities, from joint EU-HTA scientific consultations and clinical assessments to building tools to help health technology assessors evaluate the quality of data and evidence (e.g., real-world data [RWD] from registries).
Recently the EU Commission passed legislation to formalize a framework for joint HTA in Europe. Select member state HTA bodies will conduct joint clinical assessments of all newly approved technologies, which will be disseminated to all other member states to assist in reimbursement decision-making. This effectively reduces the need for each member state to conduct its own clinical assessment. However, member states will retain autonomy in assessing the cost-effectiveness (where applicable) and value of the therapy before making an independent reimbursement decision. The joint clinical assessment framework will be created over the next three years with official joint clinical assessments commencing in 2025. The framework will draw extensively from EUnetHTA’s prior experiences conducting joint assessments. Additionally, EUnetHTA’s current work (including these direct/indirect comparison guidance documents) is geared toward supporting this framework, which makes these guidance documents relevant for any sponsor submitting for HTA soon.
Guidance around statistical approaches to control for confounding in RWE studies
EUnetHTA’s guidance states that naive comparisons should not be completed and that formal statistical comparisons (i.e., those based on individual patient level data) should be performed. The sponsor’s guidance outlines several statistical approaches to control for confounding (e.g., multiple regression, instrumental variables, g-computation, and propensity scores [PS]). The assessor guidance, on the other hand, outlines three additional assumptions that must be met in PS modeling (the most common method for confounding control) to have confidence in the results:
- Positivity – Patients in both groups theoretically must be eligible for both treatments of interest.
- Overlap – The distribution of propensity scores must be similar between groups.
- Balance – After adjusting for confounding, balance on confounders must be achieved between treatment groups. Acceptable absolute standard difference thresholds between groups range from 0.1 to 0.25 (depending on the context).
While this level of detail is common in epidemiology and statistics literature on PS modeling (see here and here for examples), this level of detail is rare in guidance documents. This guidance is therefore a good step toward setting minimum standards for what “good” RWE looks like.
Differing guidance for sponsors and assessors
EUnetHTA published two separate guidance documents, one geared toward sponsors and one toward assessors. While these documents are similar, there are key differences between them. For example, in the document for sponsors, EUnetHTA notes that because it is almost impossible to capture all confounders in RWE studies, the effect size must be large enough that the effect could not be due to bias. However, the guidance document to assessors of RWE does not discuss the size of the effect needed to trust the results. This discrepancy could be intentional; for example, EUnetHTA may want to guide sponsors to use RWE in the most promising situations while still giving agencies “permission” to review less favorable situations in exceptional circumstances. Thus, it is important for researchers to read both guidance documents for complete context on how RWE studies can be used to compare therapies.
While the EUnetHTA Direct and Indirect Comparisons guidance is not solely focused on providing recommendations for RWE studies, it does set forth a standard for what “good” RWE studies look like according to European HTA bodies. It is necessary to consider how each guidance fills gaps in the current body of RWE guidance, particularly as stakeholders continue to develop and publish their own recommendations on the use of RWE. The EUnetHTA guidance is unique because it offers 1) similar, but not exactly the same, guidance for both sponsors and assessors and 2) more detail on suitable statistical approaches for comparative effectiveness studies. Such guidance incentivizes sponsors and researchers to follow RWE best practices when generating evidence for decision-makers and regulatory bodies, and corresponds to similar guidance recently issued by NICE and FDA.
If you would like to learn more about RWE best practices, please view our recent webinar, Making Sense of FDA’s 2021 Real-World Evidence Guidance. To connect with one of our scientific experts at Aetion, please email: info@aetion.com.