The package metarep
is an extension to the package meta
, which allows
incorporating replicability-analysis tools to quantify consistency and
replicability of treatment effect estimates in a meta-analysis. The tool
was proposed by Jaljuli et. al. (submitted) for the fixed-effect and for
the random-effects meta-analyses, whit or without the common-effect
assumption.
Regardless of the type of meta-analysis applied, metarep
allows to perform replicability analysis with or with out the
common-effect assumption of the fixed-effects model. We recommend
applying the replicability analysis free of the common-effect assumption
to guard from a possibly faulty assumption. At this case, the
replicability analysis is performed based Fishers’ combining function
using truncated-Pearson’s’ test. If the user finds the common-effect
assumption supported and wishes to incorporate it in the replicability
analysis, they might as well do that. Whereas with this assumption, the
replicability analysis is performed using the test statistic of the
fixed-effects model, for the combining of every set of n − u + 1 studies.
Currently, both meta
and metarep
packages
can be downloaded from GitHub
, therefore make sure that the
package devtools
is installed. metarep
also
requires the latest version of meta
(>= 4.11-0,
available on GitHub
)
Run the following commands in to install the packages:
Here we demonstrate the approach by implementation with
metarep
, using examples from systematic reviews Cochrane
library. These examples are detailed in the paper as well, along with a
demonstration of a way to incorporate our suggestions in standard
meta-analysis reporting system.
We demonstrate the with an example based on a fixed-effects meta-analysis. This example was included in Jaljuli et. al. 2020, found in review number CD002943 in the Cochrane library. This analysis explores the effect of mammogram invitation on attendance during the following 12 months.
library(metarep)
Loading required package: meta
Loading required package: metadat
Loading 'meta' package (version 8.0-2).
Type 'help(meta)' for a brief overview.
library(meta)
data(CD002943_CMP001)
m2943 <- metabin( event.e = N_EVENTS1, n.e = N_TOTAL1,
event.c = N_EVENTS2, n.c = N_TOTAL2,
studlab = STUDY, comb.fixed = T , comb.random = F,
method = 'Peto', sm = CD002943_CMP001$SM[1],
data = CD002943_CMP001)
Warning: Use argument 'common' instead of 'comb.fixed' (deprecated).
Warning: Use argument 'random' instead of 'comb.random' (deprecated).
m2943
Number of studies: k = 5
Number of observations: o = 4166 (o.e = 2451, o.c = 1715)
Number of events: e = 1384
OR 95%-CI z p-value
Common effect model 1.6564 [1.4317; 1.9164] 6.78 < 0.0001
Quantifying heterogeneity (with 95%-CIs):
tau^2 = 0.1314 [0.0073; 1.6698]; tau = 0.3625 [0.0852; 1.2922]
I^2 = 70.3% [24.4%; 88.3%]; H = 1.84 [1.15; 2.93]
Test of heterogeneity:
Q d.f. p-value
13.47 4 0.0092
Details of meta-analysis methods:
- Peto method
- Restricted maximum-likelihood estimator for tau^2
- Q-Profile method for confidence interval of tau^2 and tau
- Calculation of I^2 based on Q
summary(m2943)
OR 95%-CI %W(common)
Sutton-1994 1.2836 [0.9933; 1.6589] 32.3
Somkin-1997 1.8739 [1.5372; 2.2844] 54.2
Turnbull-1991 3.5709 [1.9225; 6.6326] 5.5
Mohler-1995 1.8764 [0.5272; 6.6788] 1.3
Bodiya-1999 1.0758 [0.6110; 1.8941] 6.6
Number of studies: k = 5
Number of observations: o = 4166 (o.e = 2451, o.c = 1715)
Number of events: e = 1384
OR 95%-CI z p-value
Common effect model 1.6564 [1.4317; 1.9164] 6.78 < 0.0001
Quantifying heterogeneity (with 95%-CIs):
tau^2 = 0.1314 [0.0073; 1.6698]; tau = 0.3625 [0.0852; 1.2922]
I^2 = 70.3% [24.4%; 88.3%]; H = 1.84 [1.15; 2.93]
Test of heterogeneity:
Q d.f. p-value
13.47 4 0.0092
Details of meta-analysis methods:
- Peto method
- Restricted maximum-likelihood estimator for tau^2
- Q-Profile method for confidence interval of tau^2 and tau
- Calculation of I^2 based on Q
In this meta-analysis, the effect of sending invitation letters was examined in five studies. The authors main result is that: “The odds ratio in relation to the outcome, attendance in response to the mammogram invitation during the 12 months after the invitation, was 1.66 (95% CI 1.43 to 1.92)”.
We suggest reporting the replicability-analysis results alongside:
the r-value and lower confidence bounds on the number of studies.
Results of complete replicability analysis can be added to the contents
of a meta
object or using the function
metarep(...)
, as well as to its summary using
summary( metarep(...) )
.
To perform assumption-free replicability-analysis requiring
replicability in at least u = 2
(default) studies, we
calculate r(2) − value
using truncated-Pearson’s’ test with truncation threshold
t=0.05
(default):
(m2943.ra <- metarep(x = m2943 , u = 2 , common.effect = F ,t = 0.05 ,report.u.max = T))
Number of studies: k = 5
Number of observations: o = 4166 (o.e = 2451, o.c = 1715)
Number of events: e = 1384
OR 95%-CI z p-value
Common effect model 1.6564 [1.4317; 1.9164] 6.78 < 0.0001
Quantifying heterogeneity (with 95%-CIs):
tau^2 = 0.1314 [0.0073; 1.6698]; tau = 0.3625 [0.0852; 1.2922]
I^2 = 70.3% [24.4%; 88.3%]; H = 1.84 [1.15; 2.93]
Test of heterogeneity:
Q d.f. p-value
13.47 4 0.0092
Details of meta-analysis methods:
- Peto method
- Restricted maximum-likelihood estimator for tau^2
- Q-Profile method for confidence interval of tau^2 and tau
- Calculation of I^2 based on Q
- replicability analysis (r-value = 2e-04)
The bottom two lines report the r(2) − value,
lower bound on the number of studies with increased effect (umaxL)
and decreased effect (umaxR)
, respectively. The evidence towards an increased effect was replicable,
with r(2) − value = 0.0002.
Moreover, with 95% confidence, we can
conclude that at least two studies had an increased effect. For higher
replicability requirement, compute r(u′) − value
for u′ > 2 using
metarep(u = u' , ... )
.
The two-sided r(u) − value
of the model can be accessed via r.value
:
The replicability-analysis reported was performed with an assumption
free test, based on truncated-Pearson’s’ test with truncation level set
at the nominal hypothesis testing level (i.e., t=0.05
,
default). For ordinary Pearson’s’ test, use t=1
.
Although the fixed-effect model assumes that all studies are
estimates of the same common effect θ, we recommend applying
assumption-free replicability-analysis for protection against an
(perhaps) unsupported assumption. Despite that, we extend our suggested
method with the common-effect incorporation in section 7. This analysis
can be performed via
metarep( ... , common.effect = TRUE )
.
metarep
also allows adding replicability results to the
conventional forest plots by meta
. This can be done by
simply applying forest()
on a metarep
object.
The lower bounds umaxL and umaxR
are calculated with 1 − α = 95% confidence level (
default), meaning that each of the null hypotheses HumaxL/n(L) and HumaxL/n(R)
is tested at level α/2 = 2.5%,
resulting in bounds in overall type error rate 5%. Type I error rate can be controlled for
any desired α using the
argument confidence = 1 -
α.
The calculation of umaxL and umaxR
can also be calculated directly using the function
find_umax()
with the option to specify one-sided
alternative, confidence level, truncation threshold and common-effect
assumption. For example, let’s compute umaxL
with the same confidence level as produced by
m2943.ra.bounds
.
find_umax(x = m2943 , common.effect = F,alternative = 'less',t = 0.05,confidence = 0.975)
$worst.case
[1] "Sutton-1994" "Somkin-1997" "Turnbull-1991" "Mohler-1995"
[5] "Bodiya-1999"
$side
Direction of the stronger signal
"less"
$u_max
u^L
0
$r.value
r^R
1
$Replicability_Analysis
[1] "out of 5 studies, 0 with decreased effect."
Note that this function produces 2 main types of results:
Worst-case scenario studies: A list of n − umaxL + 1 studies names yielding the maximum max∀{i1, …, in − u + 1} ⊂ {1, …, n}{ pi1, …, in − u + 1L }
Replicability-analysis results, including:
umaxL
or umaxR.
If alternative='two-sided'
, then umax = max {umaxL , umaxR}
is also reported.
r(umaxL) − value
or r(umaxR) − value
if setting alternative='less'
or
alternative='greater'
, respectively. If
alternative='two-sided'
, then rvalue = r(umax) = 2 ⋅ min {rR(umaxR), rL(umaxL)}
is also reported.
For demonstration, see the following example.
find_umax(x = m2943 , common.effect = F,alternative = 'two-sided',t = 0.05,confidence = 0.95)
$worst.case
[1] "Sutton-1994" "Turnbull-1991" "Mohler-1995" "Bodiya-1999"
$side
Direction of the stronger signal
"greater"
$u_max
u_max u^L u^R
2 0 2
$r.value
r.value r^L r^R
2e-04 1e+00 1e-04
$Replicability_Analysis
[1] "out of 5 studies: 0 with decreased effect, and 2 with increased effect."
find_umax(x = m2943 , common.effect = T,alternative = 'two-sided', confidence = 0.95)
Performing Replicability analysis with the common-effect assumption
$worst.case
[1] "Sutton-1994" "Mohler-1995" "Bodiya-1999"
$u_max
[1] 3
$side
[1] "greater"
$r.value
[1] 0.0468
(m2943.raFE <- metarep(x = m2943 , u = 2 , common.effect = T ,report.u.max = T))
Performing Replicability analysis with the common-effect assumption
Performing Replicability analysis with the common-effect assumption
Number of studies: k = 5
Number of observations: o = 4166 (o.e = 2451, o.c = 1715)
Number of events: e = 1384
OR 95%-CI z p-value
Common effect model 1.6564 [1.4317; 1.9164] 6.78 < 0.0001
Quantifying heterogeneity (with 95%-CIs):
tau^2 = 0.1314 [0.0073; 1.6698]; tau = 0.3625 [0.0852; 1.2922]
I^2 = 70.3% [24.4%; 88.3%]; H = 1.84 [1.15; 2.93]
Test of heterogeneity:
Q d.f. p-value
13.47 4 0.0092
Details of meta-analysis methods:
- Peto method
- Restricted maximum-likelihood estimator for tau^2
- Q-Profile method for confidence interval of tau^2 and tau
- Calculation of I^2 based on Q
- replicability analysis (r-value = 0.0011)
find_umax(x = m2943 , common.effect = T,alternative = 'two-sided', confidence = 0.95)
Performing Replicability analysis with the common-effect assumption
$worst.case
[1] "Sutton-1994" "Mohler-1995" "Bodiya-1999"
$u_max
[1] 3
$side
[1] "greater"
$r.value
[1] 0.0468
(m2943.raFE <- metarep(x = m2943 , u = 2 , common.effect = T ,report.u.max = T))
Performing Replicability analysis with the common-effect assumption
Performing Replicability analysis with the common-effect assumption
Number of studies: k = 5
Number of observations: o = 4166 (o.e = 2451, o.c = 1715)
Number of events: e = 1384
OR 95%-CI z p-value
Common effect model 1.6564 [1.4317; 1.9164] 6.78 < 0.0001
Quantifying heterogeneity (with 95%-CIs):
tau^2 = 0.1314 [0.0073; 1.6698]; tau = 0.3625 [0.0852; 1.2922]
I^2 = 70.3% [24.4%; 88.3%]; H = 1.84 [1.15; 2.93]
Test of heterogeneity:
Q d.f. p-value
13.47 4 0.0092
Details of meta-analysis methods:
- Peto method
- Restricted maximum-likelihood estimator for tau^2
- Q-Profile method for confidence interval of tau^2 and tau
- Calculation of I^2 based on Q
- replicability analysis (r-value = 0.0011)
forest(m2943.raFE, layout='revman5',digits.pval = 2 , test.overall = T )