Sensitivity analysis

Owner: Liz Bickerdike

Which theme does this roadmap item contribute to?

Adoption: The ability to conduct sensitivity analysis will definitely encourage some users to adopt RMW. Fundamental method – without it this is likely to be a blocker.

Review efficiency:

Review quality: Sensitivity analysis is an important method for review quality. Having an awareness of the robustness of the results can strengthen or weaken the conclusion of the review (MECIR C71 states that sensitivity analysis is a ‘highly desirable’ to assess the robustness of results). It is an essential feature for Network editorial staff and Methods Support Unit staff when checking review quality.

Sustainability:

What business value does this roadmap item create? / Which problem is this roadmap item trying to solve?

  • Sensitivity analysis functionality is lacking in RMW. It is very difficult to use the visual information in the forest plot to inform sensitivity analysis and impossible to see the ‘live’ changes to the effect estimate (unlike in RevMan 5). It is not easy to exclude a study from an analysis and then add that study back in.

Which assumptions are made in relation to the value?

  • The majority of reviews include some sensitivity analyses, but difficult to quantify how many.
  • It is unclear how much it will influence adoption of RMW but we think this is a potential blocker to uptake for some authors.
  • It is unclear how many Editors in CRGs utilise sensitivity analysis features when checking reviews, but many are checking robustness of analyses as described below (under ‘Authors and editors should be able to’).

What are the dependencies for fulfilling the business value?

  • EMD input (Liz B) will be needed through development
  • Need useful data sets to check accuracy

What are the risks related to this roadmap item?

  • Accuracy – starting from scratch

What is included in the scope of this solution?
Authors and Editors should be able to:

  • Use the forest plot to identify outliers and see the impact of excluding studies in the forest plot directly
  • Use the Risk of Bias graph in the forest plot to identify high risk of bias studies to be excluded in a sensitivity analysis and see the impact of excluding studies in the forest plot directly
  • Easily see and compare the impact of different analysis models on the effect estimate (random effects vs fixed effect)
  • For dichotomous outcome, easily see and compare the impact of different effect measures on the effect estimate (RR, OR, and RD)

Sensitivity analysis needs to be supported with study-centric data.