Package instalation:

Currently, both meta and metarep packages can be downloaded from GitHub, therefore make sure that the package devtools is installed. metarep also requires the latest version of meta (>= 4.11-0, available on GitHub)

Run the following commands in to install the packages:

Examples:

Here we demonstrate the approach by implementation with metarep, using examples from systematic reviews Cochrane library. These examples are detailed in the paper as well, along with a demonstration of a way to incorporate our suggestions in standard meta-analysis reporting system.

We demonstrate the with an example based on a fixed-effects meta-analysis. This example was included in Jaljuli et. al. 2020, found in review number CD002943 in the Cochrane library. This analysis explores the effect of mammogram invitation on attendance during the following 12 months.

library(metarep)
  Loading required package: meta
  Loading required package: metadat
  Loading 'meta' package (version 8.0-2).
  Type 'help(meta)' for a brief overview.
library(meta)
data(CD002943_CMP001)

m2943 <- metabin( event.e = N_EVENTS1, n.e = N_TOTAL1, 
                     event.c = N_EVENTS2, n.c = N_TOTAL2,
                     studlab = STUDY, comb.fixed = T , comb.random = F,
                     method = 'Peto', sm = CD002943_CMP001$SM[1],
                     data = CD002943_CMP001)
  Warning: Use argument 'common' instead of 'comb.fixed' (deprecated).
  Warning: Use argument 'random' instead of 'comb.random' (deprecated).

m2943
  Number of studies: k = 5
  Number of observations: o = 4166 (o.e = 2451, o.c = 1715)
  Number of events: e = 1384
  
                          OR           95%-CI    z  p-value
  Common effect model 1.6564 [1.4317; 1.9164] 6.78 < 0.0001
  
  Quantifying heterogeneity (with 95%-CIs):
   tau^2 = 0.1314 [0.0073; 1.6698]; tau = 0.3625 [0.0852; 1.2922]
   I^2 = 70.3% [24.4%; 88.3%]; H = 1.84 [1.15; 2.93]
  
  Test of heterogeneity:
       Q d.f. p-value
   13.47    4  0.0092
  
  Details of meta-analysis methods:
  - Peto method
  - Restricted maximum-likelihood estimator for tau^2
  - Q-Profile method for confidence interval of tau^2 and tau
  - Calculation of I^2 based on Q

summary(m2943)
                    OR           95%-CI %W(common)
  Sutton-1994   1.2836 [0.9933; 1.6589]       32.3
  Somkin-1997   1.8739 [1.5372; 2.2844]       54.2
  Turnbull-1991 3.5709 [1.9225; 6.6326]        5.5
  Mohler-1995   1.8764 [0.5272; 6.6788]        1.3
  Bodiya-1999   1.0758 [0.6110; 1.8941]        6.6
  
  Number of studies: k = 5
  Number of observations: o = 4166 (o.e = 2451, o.c = 1715)
  Number of events: e = 1384
  
                          OR           95%-CI    z  p-value
  Common effect model 1.6564 [1.4317; 1.9164] 6.78 < 0.0001
  
  Quantifying heterogeneity (with 95%-CIs):
   tau^2 = 0.1314 [0.0073; 1.6698]; tau = 0.3625 [0.0852; 1.2922]
   I^2 = 70.3% [24.4%; 88.3%]; H = 1.84 [1.15; 2.93]
  
  Test of heterogeneity:
       Q d.f. p-value
   13.47    4  0.0092
  
  Details of meta-analysis methods:
  - Peto method
  - Restricted maximum-likelihood estimator for tau^2
  - Q-Profile method for confidence interval of tau^2 and tau
  - Calculation of I^2 based on Q

In this meta-analysis, the effect of sending invitation letters was examined in five studies. The authors main result is that: “The odds ratio in relation to the outcome, attendance in response to the mammogram invitation during the 12 months after the invitation, was 1.66 (95% CI 1.43 to 1.92)”.

We suggest reporting the replicability-analysis results alongside: the r-value and lower confidence bounds on the number of studies. Results of complete replicability analysis can be added to the contents of a meta object or using the function metarep(...), as well as to its summary using summary( metarep(...) ).

To perform assumption-free replicability-analysis requiring replicability in at least u = 2 (default) studies, we calculate r(2) − value using truncated-Pearson’s’ test with truncation threshold t=0.05 (default):

(m2943.ra <- metarep(x = m2943 , u = 2 , common.effect = F ,t = 0.05 ,report.u.max = T))
  Number of studies: k = 5
  Number of observations: o = 4166 (o.e = 2451, o.c = 1715)
  Number of events: e = 1384
  
                          OR           95%-CI    z  p-value
  Common effect model 1.6564 [1.4317; 1.9164] 6.78 < 0.0001
  
  Quantifying heterogeneity (with 95%-CIs):
   tau^2 = 0.1314 [0.0073; 1.6698]; tau = 0.3625 [0.0852; 1.2922]
   I^2 = 70.3% [24.4%; 88.3%]; H = 1.84 [1.15; 2.93]
  
  Test of heterogeneity:
       Q d.f. p-value
   13.47    4  0.0092
  
  Details of meta-analysis methods:
  - Peto method
  - Restricted maximum-likelihood estimator for tau^2
  - Q-Profile method for confidence interval of tau^2 and tau
  - Calculation of I^2 based on Q
  - replicability analysis (r-value = 2e-04)

The bottom two lines report the r(2) − value, lower bound on the number of studies with increased effect (u_max^L) and decreased effect (u_max^R) , respectively. The evidence towards an increased effect was replicable, with r(2) − value = 0.0002. Moreover, with 95% confidence, we can conclude that at least two studies had an increased effect. For higher replicability requirement, compute r(u′) − value for u′ > 2 using metarep(u = u' , ... ).

The two-sided r(u) − value of the model can be accessed via r.value:

m2943.ra$r.value
  [1] 0.0002067774

The replicability-analysis reported was performed with an assumption free test, based on truncated-Pearson’s’ test with truncation level set at the nominal hypothesis testing level (i.e., t=0.05, default). For ordinary Pearson’s’ test, use t=1.

Although the fixed-effect model assumes that all studies are estimates of the same common effect θ, we recommend applying assumption-free replicability-analysis for protection against an (perhaps) unsupported assumption. Despite that, we extend our suggested method with the common-effect incorporation in section 7. This analysis can be performed via metarep( ... , common.effect = TRUE ).

metarep also allows adding replicability results to the conventional forest plots by meta. This can be done by simply applying forest() on a metarep object.

forest(m2943.ra, layout='revman5',digits.pval = 2 , test.overall = T )

The lower bounds u_max^L and u_max^R are calculated with 1 − α = 95% confidence level ( default), meaning that each of the null hypotheses H^u_maxL/n(L)  and  H^u_maxL/n(R) is tested at level α/2 = 2.5%, resulting in bounds in overall type error rate 5%. Type I error rate can be controlled for any desired α using the argument confidence = 1 -α.

The calculation of u_max^L and u_max^R can also be calculated directly using the function find_umax() with the option to specify one-sided alternative, confidence level, truncation threshold and common-effect assumption. For example, let’s compute u_max^L with the same confidence level as produced by m2943.ra.bounds.

find_umax(x = m2943 , common.effect = F,alternative = 'less',t = 0.05,confidence = 0.975)
  $worst.case
  [1] "Sutton-1994"   "Somkin-1997"   "Turnbull-1991" "Mohler-1995"  
  [5] "Bodiya-1999"  
  
  $side
  Direction of the stronger signal 
                            "less" 
  
  $u_max
  u^L 
    0 
  
  $r.value
  r^R 
    1 
  
  $Replicability_Analysis
  [1] "out of 5 studies, 0 with decreased effect."

Note that this function produces 2 main types of results:

1. Worst-case scenario studies: A list of n − u_max^L + 1 studies names yielding the maximum max_{∀{i1, …, in − u + 1} ⊂ {1, …, n}}{ p_{i1, …, in − u + 1}^L }

2. Replicability-analysis results, including:

   • u_max^L or u_max^R. If alternative='two-sided', then u_max = max {u_max^L , u_max^R} is also reported.

   • r(u_max^L) − value or r(u_max^R) − value if setting alternative='less' or alternative='greater', respectively. If alternative='two-sided', then rvalue = r(u_max) = 2 ⋅ min {r^R(u_max^R), r^L(u_max^L)} is also reported.

   For demonstration, see the following example.

find_umax(x = m2943 , common.effect = F,alternative = 'two-sided',t = 0.05,confidence = 0.95)
  $worst.case
  [1] "Sutton-1994"   "Turnbull-1991" "Mohler-1995"   "Bodiya-1999"  
  
  $side
  Direction of the stronger signal 
                         "greater" 
  
  $u_max
  u_max   u^L   u^R 
      2     0     2 
  
  $r.value
  r.value     r^L     r^R 
    2e-04   1e+00   1e-04 
  
  $Replicability_Analysis
  [1] "out of 5 studies: 0 with decreased effect, and 2 with increased effect."

find_umax(x = m2943 , common.effect = T,alternative = 'two-sided', confidence = 0.95)
  Performing Replicability analysis with the common-effect assumption
  $worst.case
  [1] "Sutton-1994" "Mohler-1995" "Bodiya-1999"
  
  $u_max
  [1] 3
  
  $side
  [1] "greater"
  
  $r.value
  [1] 0.0468

(m2943.raFE <- metarep(x = m2943 , u = 2 , common.effect = T ,report.u.max = T))
  Performing Replicability analysis with the common-effect assumption
  Performing Replicability analysis with the common-effect assumption
  Number of studies: k = 5
  Number of observations: o = 4166 (o.e = 2451, o.c = 1715)
  Number of events: e = 1384
  
                          OR           95%-CI    z  p-value
  Common effect model 1.6564 [1.4317; 1.9164] 6.78 < 0.0001
  
  Quantifying heterogeneity (with 95%-CIs):
   tau^2 = 0.1314 [0.0073; 1.6698]; tau = 0.3625 [0.0852; 1.2922]
   I^2 = 70.3% [24.4%; 88.3%]; H = 1.84 [1.15; 2.93]
  
  Test of heterogeneity:
       Q d.f. p-value
   13.47    4  0.0092
  
  Details of meta-analysis methods:
  - Peto method
  - Restricted maximum-likelihood estimator for tau^2
  - Q-Profile method for confidence interval of tau^2 and tau
  - Calculation of I^2 based on Q
  - replicability analysis (r-value = 0.0011)

find_umax(x = m2943 , common.effect = T,alternative = 'two-sided', confidence = 0.95)
  Performing Replicability analysis with the common-effect assumption
  $worst.case
  [1] "Sutton-1994" "Mohler-1995" "Bodiya-1999"
  
  $u_max
  [1] 3
  
  $side
  [1] "greater"
  
  $r.value
  [1] 0.0468

(m2943.raFE <- metarep(x = m2943 , u = 2 , common.effect = T ,report.u.max = T))
  Performing Replicability analysis with the common-effect assumption
  Performing Replicability analysis with the common-effect assumption
  Number of studies: k = 5
  Number of observations: o = 4166 (o.e = 2451, o.c = 1715)
  Number of events: e = 1384
  
                          OR           95%-CI    z  p-value
  Common effect model 1.6564 [1.4317; 1.9164] 6.78 < 0.0001
  
  Quantifying heterogeneity (with 95%-CIs):
   tau^2 = 0.1314 [0.0073; 1.6698]; tau = 0.3625 [0.0852; 1.2922]
   I^2 = 70.3% [24.4%; 88.3%]; H = 1.84 [1.15; 2.93]
  
  Test of heterogeneity:
       Q d.f. p-value
   13.47    4  0.0092
  
  Details of meta-analysis methods:
  - Peto method
  - Restricted maximum-likelihood estimator for tau^2
  - Q-Profile method for confidence interval of tau^2 and tau
  - Calculation of I^2 based on Q
  - replicability analysis (r-value = 0.0011)

forest(m2943.raFE, layout='revman5',digits.pval = 2 , test.overall = T )