This vignette focuses on the mixed effect model for meta-analysis of diagnostic studies based on SROC curves as presented by Holling et al. (2012). The goal of this approach is to model diagnostic accuracy across studies using a parametric formulation of the ROC curve derived from the Lehmann family. In contrast to bivariate approaches, the model assumes a functional relationship between sensitivity and the false positive rate, allowing each study to be summarized by a single accuracy parameter.
Firstly, we introduce the model itself, explaining how sensitivity and the false positive rate are linked through the Lehmann formulation and how the resulting study-specific accuracy parameters are defined. We then describe how between-study heterogeneity is incorporated via a random-effects structure, yielding parameters that represent the overall diagnostic accuracy and its variability across studies. These parameters can subsequently be used to construct the SROC curve.
Afterwards, we focus on how the model can be fitted and how the SROC curve can be obtained in practice, providing a brief overview of the computational steps involved in estimating the accuracy parameter and its variance within a mixed-effects meta-analytic framework.
Model specification
The model proposed by Holling et al. (2012) is based on a parametric formulation of the ROC curve derived from the Lehmann family. For a given study \(i\), sensitivity (\(Se_i\)) and the false positive rate (\(FPR_i = 1 - Sp_i\)) are linked through the relationship
\[ Se_i = (FPR_i)^{\theta_i}, \]
where \(\theta_i > 0\) is a study-specific parameter that quantifies the diagnostic accuracy. Larger values of \(\theta_i\) correspond to better discriminatory performance, while \(\theta_i = 1\) indicates no diagnostic ability.
Taking logarithms yields a linear representation
\[ \log(Se_i) = \theta_i \log(FPR_i), \]
which allows estimation of \(\theta_i\) for each study based on the observed sensitivity and false positive rate.
To account for between-study heterogeneity, the study-specific parameters are assumed to follow a random-effects model
\[ \theta_i = \theta + u_i, \quad u_i \sim \mathcal{N}(0, \tau^2), \]
where \(\theta\) represents the overall diagnostic accuracy across studies and \(\tau^2\) denotes the between-study variance. This formulation captures variability in accuracy while treating the false positive rate as a nuisance quantity that may vary across studies due to differing thresholds.
The SROC curve is obtained by substituting the pooled estimate \(\theta\) into the Lehmann relationship
\[ Se = (FPR)^{\theta}, \]
which provides a smooth curve summarizing the trade-off between sensitivity and specificity across studies.
Profile likelihood
A key aspect of the approach is the use of a profile likelihood to simplify inference. Since either sensitivity or the false positive rate can be treated as the dependent variable in the Lehmann formulation, the choice of which quantity is considered a nuisance parameter is essentially arbitrary. The profile likelihood is invariant to this choice, ensuring that inference on the parameter of interest \(\theta\) remains unchanged regardless of whether sensitivity or the false positive rate is profiled out.
After eliminating the nuisance parameter, the resulting profile log-likelihood for \(\theta\) takes an approximately Gaussian form
\[ \ell(\theta) = -\frac{1}{2} \frac{(z - w \theta)^2}{t^2 \theta^2 + s^2}, \]
where \(z\), \(w\), \(s^2\), and \(t^2\) are quantities derived from the observed data. This expression resembles a normal log-likelihood, differing only by a term involving \(\log \sigma^2(\theta)\).
However, it is important to note that the conventional profile likelihood is not a true likelihood. In particular, its curvature may not yield valid variance estimates, as the nuisance parameters are replaced by their estimates and treated as fixed. As a consequence, the variability of \(\theta\) may be underestimated, which motivates careful interpretation and, in some cases, the use of adjusted likelihood approaches.
Model Application in MetaROC
Inside the package, the model can be fitted by either the
fit_metaROC() function or the
metaROC.metaROC() method when setting
action = "estimate". The model can then be fitted by
setting model = "holling2012lehmann":
library(metaROC)
set.seed(7)
data(hba1c)
stm_hba1c <- hba1c[hba1c$originally_published == 1,]
fit_hol_leh <- fit_metaROC(stm_hba1c, model = "holling2012lehmann")## Hello and welcome to metaROC!
## Requested model: holling2012lehmann
## This is a Lehmann family model for a single threshold per study.
## See https://doi.org/10.1177/1471082X1201200403 for more details.
#est_hol_leh <- metaROC(action ="estimate", data = stm_hba1c, model = "holling2012lehmann")As we have already filtered the HbA1c dataset using the
originally_published column, we do not need to worry about
the warning, since we have only one entry per study.
Internally, the Holling et al. (2012)
mixed effects model is fitted as described in the model specification
section. The observed counts of true positives, false negatives, false
positives, and true negatives are first used to compute study-level
sensitivities and false positive rates. These quantities are then
transformed and combined in the profile likelihood to estimate the
study-specific and pooled accuracy parameters, incorporating
between-study heterogeneity. In practice, we rely on the
phm() function from the mada package in R
(Doebler and Sousa-Pinto (2022)), which
returns an S3 object containing the fitted parameters.
The fitted model returns two key estimates: the pooled accuracy parameter \(\theta\) and the between-study variance \(\tau^2\).
fit_hol_leh$mod$coefficients## theta taus_sq
## 0.21300188 0.01300058For our model, \(\theta = 0.247\) indicates the overall diagnostic accuracy across studies on the log-transformed scale, reflecting moderate discriminatory performance of the test. The estimate \(\tau^2 = 0.014\) quantifies the variability of study-specific accuracy parameters around the pooled value, showing that there is relatively low heterogeneity among the included studies. Together, these parameters can be used to construct the summary ROC curve and assess both the typical accuracy and the consistency of the diagnostic test across studies.
summary(fit_hol_leh, ci_type = "wald")##
## *** Results of Single threshold method (STM) ***
##
## Model: Lehmann family model for single threshold
##
## Total number of studies: 38
## Total number of thresholds: 38
## Number of different thresholds: 13
##
## Results with Wald confidence intervals:
##
## Youden index (sensitivity weight = 0.5): 0.5178
## Estimated Sensitivity and Specificity [95% CI]:
## Sens: 0.6586 [0.6046; 0.7174]
## Spec: 0.8593
##
## AUC: 0.8240Firstly, the model name is returned, along with a reminder that the Holling et al. (2012) mixed effect model is a single-threshold method. Afterwards, a basic overview of the meta-analysis is provided. The output then provides the largest Youden index, although, since the threshold itself is not included in the model, this does not provide additional interpretative insights.
Following the estimation of the model parameters, the summary reports the estimated sensitivity and specificity for the diagnostic test, along with a corresponding 95% confidence interval for the sensitivity. Because the model is based on a functional relationship between sensitivity and the false positive rate, and the profile likelihood treats one parameter as dependent on the other, only a single confidence interval can be computed. Finally, the AUC is displayed, providing a summary measure of the overall diagnostic accuracy.
## specificity sensitivity youden_index
## 1 0.99900 0.2296119 0.2286119
## 2 0.98902 0.3825116 0.3715316
## 3 0.97904 0.4389885 0.4180285
## 4 0.96906 0.4769556 0.4460156
## 5 0.95908 0.5062204 0.4653004
## 6 0.94910 0.5303083 0.4794083
## 7 0.93912 0.5509229 0.4900429
## 8 0.92914 0.5690275 0.4981675
## 9 0.91916 0.5852243 0.5043843
## 10 0.90918 0.5999164 0.5090964The SROC for the Holling et al. (2012) mixed effect model is constructed as illustrated in the model specification section. Firstly, the predicted sensitivity is evaluated across a grid of false positive rates using the estimated accuracy parameters from the model. The false positive rates are converted to specificity for plotting.
Currently, the package does not support simulating data from this model, so this concludes the discussion of the Holling et al. (2012) mixed effect model.
To gain an overview of how to plot a fitted model and how to conduct simulation studies, particularly for evaluating models such as the Holling et al. (2012) mixed effect model, please refer to the other vignettes included in this package, which provide more detailed guidance on these topics.