metaROC: meta-analysis of DTA studies with (possibly) multiple thresholds per study

Function to simulate meta-analysis data from DTA studies, estimate models on either simulated or real meta-analysis of DTA studies data that can report results on one or more than one diagnostic threshold, and evaluate estimated models regarding bias and coverage for which the data-generating process is known. See details for more information on the workflow and the implemented models.

Usage

metaROC(
  action = NULL,
  data = NULL,
  data_info = NULL,
  type_sim = NULL,
  n_sims = NULL,
  control_sim = NULL,
  model_fit = NULL,
  control_fit = NULL,
  summarize = NULL
)

Arguments

action: char, default = NULL. Choose from "simulate", "estimate" or "evaluate". See Details.
data: data.frame or list of dataframes, default = NULL. Data to estimate models on. Used in actions "estimate" and "evaluate".
data_info: list, default = NULL. Optional list of data-generating parameters, needs to be the output data_info of action "simulate" if information on the data-generating parameters is to preserved for futher evaluation. Required for action "evaluate".See Details.
type_sim: char, default = NULL. Choose from "phaseII" and "phaseIII". Used in action "simulate" to distinguish the two simulation approaches. See Details.
n_sims: int, default = NULL. Number of datasets to simulate. Used in action "simulate".
control_sim: list, default = NULL. List of simulation parameters, used in action "simulate". See Details for the necessary information.
model_fit: Default = NULL. Model identifier(s) for estimation, used in action "estimate" (an be a string or a vector of strings). Alternatively, for action "evaluate", estimated models or list of estimated models of class metaROC. See Details for available models.
control_fit: list, default = NULL. List of estimation parameters, used in action "estimate". If provided, control_fit needs to be a list with as many lists inside as the number of models inside model_fit (also see example). If no estimation parameter is to be changed for a model, an empty list entry (list()) should be added for that model. See Details for the parameter that can be changed inside control_fit.
summarize: logical, default = NULL. Only used in action "evaluate". Indicates if additionally to evaluating every estimated model, summarizing information should be printed as a dataframe in results_long. See Details for more information on the returned dataframe.

Value

List of simulated data and info on the data-generating process if action = "simulate"; estimated models if action = "estimate" (see Details for output structure, depending on number of datasets and models); evaluated models if action = "evaluate" (see Details for output structure, depending on number of datasets and models).

Details

Three main features of the metaROC package are combined in this function, they can be selected using the parameter action:

simulation of datasets resembling meta-analysis of DTA studies
estimation of models for the meta-analysis of DTA studies on (simulated or real) datasets
evaluation of estimated models in terms of bias and empirical coverage in cases where information on the data-generating process is given

Depending on the chosen action, different parameters need to be supplied (see below).

Simulating data (action = `"simulate"`)

Meta-analysis of diagnostic test accuracy (DTA) studies can possess two fundamentally different structures: (1) One input row per DTA study, containing true positives (TP), false negatives (FN), false positives (FP), true negatives (TN) for some diagnostic threshold. Alternatively, multiple rows per study can be considered, where each row provides threshold-specific results, resulting in $\ge 1$ rows for each study. This function accommodates both variants (see below) where the diagnostic threshold to generate TP, FN, FP, TN is given in the column threshold and generates n_sims datasets, identified by the column replicate.

Further, two approaches to data generation are available, controlled by the parameter type_sim:

(1) Use the data-generating processes of a meta-analysis model to generate meta-analysis datasets that can be used to validate a method in a phase II simulation study (Heinze et al. 2024) .

(2) Specify data structure in eight parameter dimensions (see details) to get realistic meta-analysis datasets that can be used to validate methods in a phase III simulation study (Heinze et al. 2024) .

(1) type_sim = "phaseII":

Use the data-generating process of a meta-analysis model to generate meta-analysis datasets that can be used to validate a method in a phase II simulation study (Heinze et al. 2024) . The following models are available (controlled by the argument model).

stoye2024cloglog – discrete GLMM with cloglog-link (Stoye et al. 2024)
stoye2024logit – discrete GLMM with logit-link (Stoye et al. 2024)
hoyer2018weibullaft – Weibull AFT (Hoyer et al. 2018)
steinhauser2016logitlmm – LMM with logit-link (Steinhauser et al. 2016)
stoye2024copula – Clayton copula (copula_kind = "clayton") or asymmetric Joe copula (copula_kind = "joe").

Four main steps are conducted:

calculate true sensitivities and specificities for given true distribution parameters.
sample meta parameters for simulation: number of studies, number of individuals per study, number of diseased and non-diseased individuals per study, number of included thresholds per study, threshold values.
sample underlying test values for all individuals, this procedure depends strongly on the choice of true model. For the discrete GLMMs, the number of events in the intervals between thresholds is generated via the discrete hazard. For the logit LMM, the Weibull AFT model and the copula models, the test values are first generated using the according distributions before they are interval-censored.
create simulated dataset.

(2) type_sim = "phaseIII":

Specify data structure in eight parameter dimensions (see details) to get realistic meta-analysis datasets that can be used to validate methods in a phase III simulation study (Heinze et al. 2024) .

Both types expect the simulation parameters to be defined as a named list in the argument control_sim.

For type_sim = "phaseII", the following control parameters have to be defined in control_sim:

n_studies_min – int, minimum number of studies.
n_studies_max – int, maximum number of studies.
n_subj_min – int, minimum number of subjects in each study.
n_subj_max – int, maximum number of subjects in each study.
n_thresholds_min – int, minimum number of thresholds reported on by each study.
n_thresholds_max – int, maximum number of thresholds reported on by each study.
thresholds_lower_bound – num, minimum threshold value (rounded to one decimal).
thresholds_upper_bound – num, maximum threshold value (rounded to one decimal).
true_preval_min – num, minimum value of true prevalence in the data.
true_preval_max – num, maximum value of true prevalence in the data.
model – char, model specification for data generation, one of "stoye2024cloglog", "stoye2024logit", "steinhauser2016logitlmm", "hoyer2018weibullaft", "stoye2024copula".
model_pars List of true model parameters. Can be either list(preset = TRUE, AUC = x), where x is either "65", "75" or "85", to use a preset from metaROC:::simulation_preset() or a manually defined list that fits the model argument:
- model = stoye2024cloglog or model = stoye2024logit – Need do define parameters
  - beta11, beta12, beta13 – Intercept, linear, and quadratic parameters for true dependence of categorical parameters in linear term of GLMM for the non-diseased population
  - beta21, beta22 – Intercept, and linear parameter for true dependence of categorical parameters in linear term of GLMM for the diseased population
  - var_rand_H, var_rand_D – true variances of bivariate random effect for non-diseased (H) and diseased (D)
- model = hoyer2018weibullaft – Need to define parameters
  - beta00, lambda0 – parameters of true distribution of test values for non-diseased population (see (Hoyer et al. 2018) for details)
  - beta01, lambda1 – parameters of true distribution of test values for diseased population (see (Hoyer et al. 2018) for details)
  - var_rand_H, var_rand_D – true variances of bivariate random effect for non-diseased (H) and diseased (D)
- model = steinhauser2016logitlmm – Need to define parameters
  - beta11, beta12 – true mean and variance of normal distribution of test values for non-diseased population
  - beta21, beta22 – true mean and variance of normal distribution of test values for diseased population
  - var_rand_H, var_rand_D – true variances of bivariate random effect for non-diseased (H) and diseased (D)
- model = stoye2024copula – Need to define parameters
  - beta00, lambda0 – parameters of true distribution of test values for non-diseased population (see (Hoyer et al. 2018) for details)
  - beta01, lambda1 – parameters of true distribution of test values for diseased population (see (Hoyer et al. 2018) for details)
  - copula_kind_sim – copula family, either "clayton" or "joe"
  - cop_params – true copula parameters, either a positive float for the Clayton copula or a vector of three values for the Joe copula (first positive, second and third between 0 and 1)
  - norm_approx_sim – Bool, should a normal approximation be used instead of the exact binomial distribution in copula marginals?
  - marginal_kind_sim – distribution for the test values, either "weibull", "lognormal", "loglogistic", or "gompertz"

Optionally, the following parameter can be defined in control_sim:

sens_weight – num Weight of sensitivity (between 0 and 1) to determine true optimal sensitivity and specificity in info on the simulated data. If this parameter is not set in the control_sim list, the sensitivity weight is set to 0.5.

For type_sim = "phaseIII", the following control parameters have to be defined in control_sim:

n_studies – list(min=..., max=...) range of number of studies per data set.
n_subj – list(min=..., max=...), range of number of individuals per study.
n_thresholds – list(min=..., max=...), range of number of thresholds reported on per study.
outcome_type – char, type of test values sampled, either "continuous" or "ordinal".
true_preval – list(min=..., max=...), range of disease prevalence per study.
true_AUC – num, diagnostic test quality, measured as area under the ROC curve. Either 0.75 or 0.9.
heterogeneity – char, heterogeneity between studies per data set, either "low" or "high".
weight_standard_threshold – num, additional point weight on the inclusion of a 'standard' threshold in each study, either 0 or 0.7.
round_threshold_to_xth_decimal – int, number of decimal places to round the thresholds to in the continuous settings. Does not have to be specified for the ordinal settings.

Optionally, the following parameter can be defined in control_sim:

sens_weight – num Weight of sensitivity (between 0 and 1) to determine true optimal sensitivity and specificity in info on the simulated data. If this parameter is not set in the control_sim list, the sensitivity weight is set to 0.5.

Internally, metaROC() calls simulate_metaROC(), which can also be called directly.

Model estimation (action = "estimate")

Fits one or several models to the meta-analysis as specified by the data.

data - data.frame or list of data.frames for simulated data.
model_fit - model(s) to be fitted. Can be a single character or a list of characters. Available models:

Single threshold methods (stm) can only be fitted to real world or phaseII simulation data:
- "reitsma2005lmm" - LMM for a single threshold per study (Reitsma et al. 2005) .
- "chu2006glmm" - GLMM for a single threshold per study (Chu and Cole 2006) .
- "moses1993lm" - LM for a single threshold per study (Moses et al. 1993) .
- "holling2012lehmann" - Lehmann family model for a single threshold (Holling et al. 2012) .
Multiple threshold methods (mtm):
- "steinhauser2016logitlmm" - LMM for multiple thresholds (Steinhauser et al. 2016) .
- "hoyer2018weibullaft" - Weibull AFT model for multiple thresholds (Hoyer et al. 2018) .
- "stoye2024cloglog" - GLMM for discrete hazard (Stoye et al. 2024) .
- "stoye2024logit" - GLMM for discrete hazard (Stoye et al. 2024) .
- "stoye2024copula" - Copula model for multiple thresholds (paper forthcoming; updating citation when available).
- "froemke2022np" - Non-parametric model (Frömke et al. 2022) .
- "martinez2017np" - Non-parametric model (Martinez-Camblor 2017) .
- "hoyer2018glmm" - GLMM for multiple thresholds (Hoyer and Kuss 2018) .
- "nikoloulopoulos2015bivariate" - Copula mixed model for bivariate meta-analysis (Nikoloulopoulos 2015) .

control_fit - list of parameters required for fitting of the specified model(s). If several models are fitted and control_fit is supplied, there has to be a list for each model even if no model parameter is to be changed for that specific model. In that case simply add list() for that model:
- nAGQ - Int, default = 0. Only used in GLMMs. Controls the optimization algorithm (0: PIRLS, 1: Laplace Approximation, >1: Gauss-Hermite quadrature). PIRLS & Laplace use lme4::glmer(), Gauss-Hermite uses GLMMadaptive::mixed_model().
- copula_kind - String, default NA. Copula type if a copula model is fitted. One of c("clayton", "joe").
- fix_cop_par - Bool. Whether copula parameter(s) should be fixed during estimation or not.
- norm_approx - Bool, default FALSE. Only for copulas. Whether the binomial marginal distribution should be approximated with a normal distribution or not.
- marginal_kind - String. Only for copula models. One of c("weibull", "gompertz", "loglogistic", "lognormal").
- link - String. One of "logit","probit","cloglog","cauchit" and "log" to be used in an GLMM.
- data_info - List with information on the data-generating process, as returned by simulate_metaROC()$data_info or action="simulate". If NULL (default), no DGP information is assumed.
- verbose - Bool, default TRUE. Prints additional model information.
- variant - Only relevant for "martinez2017np". One of "random-effects" (default) or "fixed-effects".
- max_TPR - Float. Maximum false positive rate to be considered for model with id "moses1993lm" (default is max_FPR = 1)
- max_TPR - Float. Minimum true positive rate to be considered for model with id "moses1993lm" (default is min_TPR = 0)
- type - String. Character string specifying the type of meta-analytic model used in the "steinhauser2016logitlmm". One of "DICS","CICS","CS", default "DICS" "DICS": Diagnostic Indirect Comparison of Sensitivity and Specificity (default),"CICS": Conditional Indirect Comparison of Sensitivity and Specificity,"CS": Composite Summary model
- est_weibull_pars - Default NULL. List with estimated Weibull AFT parameters: list(pars=list(beta00, lambda0, beta01, lambda1), se=list(beta00, lambda0, beta01, lambda1)) Use only to convert SAS-estimated Weibull AFT parameters into a metaROC S3 object.

When a model is fitted to a single dataset, the returned object is a list containing the model object (mod), the convergence status, the estimated parameters with their standard errors (par), the step, the thresholds, the model_id, and the data the model was fitted to.

If several models are fitted to several datasets, as is common in simulation studies, the individual results are nested. On the first layer, each dataset is represented by an integer from 1 to the total number of datasets. Within each dataset entry, each model_id has a separate list element containing its specific results. For a single dataset (without list structure or without a replicate column), it is also possible to call fit_metaROC().

Model evaluation (action = "evaluate")

Evaluates a given model or list of models as returned by action="estimate". Only possible for simulated data, as information on the DGP is required.

model_fit - Fitted model or list of models as returned by action "estimate" or fitted model returned by fit_metaROC().
confidence_level - float between 0 and 1. Desired confidence level for confidence intervals. Default: 0.95.
sens_weight - float between 0 and 1. Weight of sensitivity for the Youden Index. Default: 0.5.

When setting summarize = TRUE, the returned dataframe results_long contains one row per model per replicate in each simulation set. The following performance metrics are included:

AUC bias
Sensitivity bias
Sensitivity standard error
Specificity bias
Specificity standard error
Threshold bias
Sensitivity coverage
Specificity coverage
Convergence

By default, evaluate reports these metrics separately for each model in the model_specific entry. The first layer, similar to estimate, contains the replicated datasets indexed by integers, followed by the model_id entries, within which the evaluation metrics for each replicate and model are stored.

In practice, setting summarize = TRUE is recommended, as it is easier to access and work with the simulation results in long format rather than the wide format provided in the default model_specific output.

References

Chu H, Cole SR (2006). “Bivariate meta-analysis of sensitivity and specificity with sparse data: a generalized linear mixed model approach.” Journal of clinical epidemiology, 59(12), 1331–1332. doi:10.1016/j.jclinepi.2006.06.011 .

Frömke C, Kirstein M, Zapf A (2022). “A semiparametric approach for meta-analysis of diagnostic accuracy studies with multiple cut-offs.” Research synthesis methods, 13(5), 612–621. doi:10.1002/jrsm.1579 .

Heinze G, Boulesteix A, Kammer M, Morris TP, White IR, of the STRATOS Initiative SP (2024). “Phases of methodological research in biostatistics—building the evidence base for new methods.” Biometrical Journal, 66(1), 2200222. doi:10.1002/bimj.202200222 .

Holling H, Böhning W, Böhning D (2012). “Meta-analysis of diagnostic studies based upon SROC-curves: a mixed model approach using the Lehmann family.” Statistical Modelling, 12(4), 347–375. doi:10.1177/1471082X1201200403 .

Hoyer A, Hirt S, Kuss O (2018). “Meta-analysis of full ROC curves using bivariate time-to-event models for interval-censored data.” Research synthesis methods, 9(1), 62-72. doi:10.1002/jrsm.1273 .

Hoyer A, Kuss O (2018). “Meta-analysis for the comparison of two diagnostic tests to a common gold standard: a generalized linear mixed model approach.” Statistical methods in medical research, 27(5), 1410–1421. doi:10.1177/0962280216661587 .

Martinez-Camblor P (2017). “Fully non-parametric receiver operating characteristic curve estimation for random-effects meta-analysis.” Statistical methods in medical research, 26(1), 5–20. doi:10.1177/0962280214537047 .

Moses LE, Shapiro D, Littenberg B (1993). “Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations.” Statistics in medicine, 12(14), 1293–1316. doi:10.1002/sim.4780121403 .

Nikoloulopoulos AK (2015). “A mixed effect model for bivariate meta-analysis of diagnostic test accuracy studies using a copula representation of the random effects distribution.” Statistics in medicine, 34(29), 3842–3865. doi:10.1002/sim.6595 .

Reitsma JB, Glas AS, Rutjes AW, Scholten RJ, Bossuyt PM, Zwinderman AH (2005). “Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews.” Journal of clinical epidemiology, 58(10), 982–990. doi:10.1016/j.jclinepi.2005.02.022 .

Steinhauser S, Schumacher M, Rücker G (2016). “Modelling multiple thresholds in meta-analysis of diagnostic test accuracy studies.” BMC Medical Research Methodology, 16(1), 97. doi:10.1186/S12874-016-0196-1 .

Stoye FV, Tschammler C, Kuss O, Hoyer A (2024). “A discrete time-to-event model for the meta-analysis of full ROC curves.” Research synthesis methods, 15(6), 1031–1048. doi:10.1002/jrsm.1753 .

Author

Ferdinand Stoye

Examples

# seed for reproducibility
set.seed(1905)

# simulate data using action = "simulate"

# type = "phaseII"
data_phaseII = metaROC(action="simulate", type = "phaseII", n_sims = 5,
                       control_sim = list(n_studies_min = 5, n_studies_max = 10,
                                          n_subj_min = 20, n_subj_max = 500,
                                          n_thresholds_min = 1, n_thresholds_max = 4,
                                          thresholds_lower_bound = 5, thresholds_upper_bound = 7,
                                          true_preval_min = 0.1, true_preval_max = 0.3,
                                          model = "stoye2024cloglog",
                                          model_pars = list(preset=TRUE, AUC="85"),
                                          sens_weight = 0.5))
#> Hello and welcome to metaROC!
#> Chosen action: simulate 
#> Successfully generated 5 datasets.
data_phaseII$data_info
#> $model_id
#> [1] "stoye2024cloglog"
#> 
#> $true_sens
#>  [1] 9.737029e-01 9.375570e-01 8.891743e-01 8.263907e-01 7.478580e-01
#>  [6] 6.538353e-01 5.470146e-01 4.330258e-01 3.201362e-01 2.177847e-01
#> [11] 1.341150e-01 7.345648e-02 3.513149e-02 1.440224e-02 4.971278e-03
#> [16] 1.421436e-03 3.320754e-04 6.274557e-05 9.535363e-06 1.165283e-06
#> [21] 1.151997e-07
#> 
#> $true_spec
#>  [1] 0.1524999 0.3251492 0.5047602 0.6729009 0.8110102 0.9076285 0.9632540
#>  [8] 0.9886084 0.9973761 0.9995732 0.9999535 0.9999968 0.9999999 1.0000000
#> [15] 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000
#> 
#> $true_AUC
#> [1] 0.8499734
#> 
#> $model_pars
#> $model_pars$preset
#> [1] TRUE
#> 
#> $model_pars$beta11
#> [1] -38.2
#> 
#> $model_pars$beta12
#> [1] 10.26
#> 
#> $model_pars$beta13
#> [1] -0.669
#> 
#> $model_pars$beta21
#> [1] 3.356
#> 
#> $model_pars$beta22
#> [1] -0.306
#> 
#> $model_pars$var_rand_H
#> [1] 0.6561
#> 
#> $model_pars$var_rand_D
#> [1] 0.9216
#> 
#> $model_pars$betas1
#>  [1] -3.62500 -3.27469 -2.93776 -2.61421 -2.30404 -2.00725 -1.72384 -1.45381
#>  [9] -1.19716 -0.95389 -0.72400 -0.50749 -0.30436 -0.11461  0.06176  0.22475
#> [17]  0.37436  0.51059  0.63344  0.74291  0.83900
#> 
#> $model_pars$betas2
#>  [1] 1.8260 1.7954 1.7648 1.7342 1.7036 1.6730 1.6424 1.6118 1.5812 1.5506
#> [11] 1.5200 1.4894 1.4588 1.4282 1.3976 1.3670 1.3364 1.3058 1.2752 1.2446
#> [21] 1.2140
#> 
#> $model_pars$all_available_thresholds
#>  [1] 5.0 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 6.0 6.1 6.2 6.3 6.4 6.5 6.6 6.7 6.8
#> [20] 6.9 7.0
#> 
#> 
#> $sens_weight
#> [1] 0.5
#> 
#> $true_opt_threshold
#> [1] 5.5
#> 
#> $true_opt_sensitivity
#> [1] 0.6538353
#> 
#> $true_opt_specificity
#> [1] 0.9076285
#> 
head(data_phaseII$data)
#> # A tibble: 6 × 7
#>   replicate study threshold    TP    FN    FP    TN
#>       <int> <fct> <fct>     <int> <int> <int> <int>
#> 1         1 1     6.1           7     2     2    20
#> 2         1 2     5.7          26    25    17   167
#> 3         1 2     6.9           0    51     0   184
#> 4         1 3     6.4          10    43     2   369
#> 5         1 3     7             4    49     0   371
#> 6         1 4     5.4          36     7    38    92

# type = "phaseIII"
data_phaseIII = metaROC(action="simulate", type = "phaseIII", n_sims = 5,
                       control_sim = list(n_studies = list(min=5, max=10),
                                          n_subj = list(min=20, max=500),
                                          n_thresholds = list(min=1, max=4),
                                          outcome_type = "continuous",
                                          true_preval = list(min = 0.1, max=0.3),
                                          true_AUC = 0.9, heterogeneity = "low",
                                          weight_standard_threshold = 0,
                                          round_threshold_to_xth_decimal = 1,
                                          sens_weight = 0.5))
#> Hello and welcome to metaROC!
#> Chosen action: simulate 
#> Successfully generated 5 datasets.
data_phaseIII$data_info
#> $n_replicates
#> [1] 5
#> 
#> $n_studies
#> $n_studies$min
#> [1] 5
#> 
#> $n_studies$max
#> [1] 10
#> 
#> 
#> $n_subj
#> $n_subj$min
#> [1] 20
#> 
#> $n_subj$max
#> [1] 500
#> 
#> 
#> $n_thresholds
#> $n_thresholds$min
#> [1] 1
#> 
#> $n_thresholds$max
#> [1] 4
#> 
#> 
#> $outcome_type
#> [1] "continuous"
#> 
#> $true_preval
#> $true_preval$min
#> [1] 0.1
#> 
#> $true_preval$max
#> [1] 0.3
#> 
#> 
#> $true_AUC
#> [1] 0.9002286
#> 
#> $heterogeneity
#> [1] "low"
#> 
#> $round_threshold_to_xth_decimal
#> [1] 1
#> 
#> $weight_standard_threshold
#> [1] 0
#> 
#> $sens_weight
#> [1] 0.5
#> 
#> $true_opt_threshold
#> [1] 5.844845
#> 
#> $true_opt_sensitivity
#> [1] 0.8206688
#> 
#> $true_opt_specificity
#> [1] 0.8710953
#> 
head(data_phaseIII$data)
#> # A tibble: 6 × 12
#>   replicate study threshold     N prevalence     D     H    TP    FN    FP    TN
#>       <int> <int>     <dbl> <dbl>      <dbl> <dbl> <dbl> <int> <dbl> <dbl> <int>
#> 1         1     1       5     177      0.174    31   146    27     4    41   105
#> 2         1     1       5.5   177      0.174    31   146    26     5    26   120
#> 3         1     1       7.2   177      0.174    31   146    12    19     5   141
#> 4         1     2       4.7   436      0.142    62   374    54     8   156   218
#> 5         1     3       5.5   180      0.216    39   141    29    10    18   123
#> 6         1     4       6.3   118      0.223    26    92    18     8    10    82
#> # ℹ 1 more variable: threshold_stm <dbl>

# estimate models on simulated data using action = "estimate"
# specify data_info, otherwise the models can not be evaluated afterwards

# type = "phaseII", does not need to be referernced in the function anymore
mods_phaseII = metaROC(action="estimate", data = data_phaseII$data,
                      data_info = data_phaseII$data_info,
                      model_fit = c("hoyer2018glmm","stoye2024cloglog"))
#> Hello and welcome to metaROC!
#> Chosen action: estimate 
#> Hello and welcome to metaROC!
#> Requested model: hoyer2018glmm 
#> This is a GLMM for multiple thresholds.
#>  See https://doi.org/10.1177/0962280216661587 for more details.
#> Warning: Model failed to converge with max|grad| = 0.00395732 (tol = 0.002, component 1)
#>   See ?lme4::convergence and ?lme4::troubleshooting.
#> Hello and welcome to metaROC!
#> Requested model: stoye2024cloglog 
#> This is a GLMM for the discrete Hazard for multiple thresholds.
#>  See https://doi.org/10.1002/jrsm.1753 for more details.
#> boundary (singular) fit: see help('isSingular')
#> Hello and welcome to metaROC!
#> Requested model: hoyer2018glmm 
#> This is a GLMM for multiple thresholds.
#>  See https://doi.org/10.1177/0962280216661587 for more details.
#> Hello and welcome to metaROC!
#> Requested model: stoye2024cloglog 
#> This is a GLMM for the discrete Hazard for multiple thresholds.
#>  See https://doi.org/10.1002/jrsm.1753 for more details.
#> boundary (singular) fit: see help('isSingular')
#> Hello and welcome to metaROC!
#> Requested model: hoyer2018glmm 
#> This is a GLMM for multiple thresholds.
#>  See https://doi.org/10.1177/0962280216661587 for more details.
#> Hello and welcome to metaROC!
#> Requested model: stoye2024cloglog 
#> This is a GLMM for the discrete Hazard for multiple thresholds.
#>  See https://doi.org/10.1002/jrsm.1753 for more details.
#> boundary (singular) fit: see help('isSingular')
#> boundary (singular) fit: see help('isSingular')
#> Warning: NAs introduced by coercion
#> Hello and welcome to metaROC!
#> Requested model: hoyer2018glmm 
#> This is a GLMM for multiple thresholds.
#>  See https://doi.org/10.1177/0962280216661587 for more details.
#> Hello and welcome to metaROC!
#> Requested model: stoye2024cloglog 
#> This is a GLMM for the discrete Hazard for multiple thresholds.
#>  See https://doi.org/10.1002/jrsm.1753 for more details.
#> boundary (singular) fit: see help('isSingular')
#> Hello and welcome to metaROC!
#> Requested model: hoyer2018glmm 
#> This is a GLMM for multiple thresholds.
#>  See https://doi.org/10.1177/0962280216661587 for more details.
#> Hello and welcome to metaROC!
#> Requested model: stoye2024cloglog 
#> This is a GLMM for the discrete Hazard for multiple thresholds.
#>  See https://doi.org/10.1002/jrsm.1753 for more details.
#> boundary (singular) fit: see help('isSingular')
mods_phaseII[[1]]$hoyer2018glmm
#> >> Fitted meta-analysis model of class metaROC <<
#> >> Selected model: hoyer2018glmm <<
#> >> Model estimation is converged <<
#> >> See summary() for more details <<

# type = "phaseII", does not need to be referenced in the function anymore
mods_phaseIII = metaROC(action="estimate", data = data_phaseIII$data,
                       data_info = data_phaseIII$data_info,
                       model_fit = c("hoyer2018glmm","stoye2024cloglog"))
#> Hello and welcome to metaROC!
#> Chosen action: estimate 
#> Hello and welcome to metaROC!
#> Requested model: hoyer2018glmm 
#> This is a GLMM for multiple thresholds.
#>  See https://doi.org/10.1177/0962280216661587 for more details.
#> Hello and welcome to metaROC!
#> Requested model: stoye2024cloglog 
#> This is a GLMM for the discrete Hazard for multiple thresholds.
#>  See https://doi.org/10.1002/jrsm.1753 for more details.
#> boundary (singular) fit: see help('isSingular')
#> boundary (singular) fit: see help('isSingular')
#> Warning: NAs introduced by coercion
#> Hello and welcome to metaROC!
#> Requested model: hoyer2018glmm 
#> This is a GLMM for multiple thresholds.
#>  See https://doi.org/10.1177/0962280216661587 for more details.
#> Hello and welcome to metaROC!
#> Requested model: stoye2024cloglog 
#> This is a GLMM for the discrete Hazard for multiple thresholds.
#>  See https://doi.org/10.1002/jrsm.1753 for more details.
#> boundary (singular) fit: see help('isSingular')
#> Hello and welcome to metaROC!
#> Requested model: hoyer2018glmm 
#> This is a GLMM for multiple thresholds.
#>  See https://doi.org/10.1177/0962280216661587 for more details.
#> Hello and welcome to metaROC!
#> Requested model: stoye2024cloglog 
#> This is a GLMM for the discrete Hazard for multiple thresholds.
#>  See https://doi.org/10.1002/jrsm.1753 for more details.
#> Hello and welcome to metaROC!
#> Requested model: hoyer2018glmm 
#> This is a GLMM for multiple thresholds.
#>  See https://doi.org/10.1177/0962280216661587 for more details.
#> Hello and welcome to metaROC!
#> Requested model: stoye2024cloglog 
#> This is a GLMM for the discrete Hazard for multiple thresholds.
#>  See https://doi.org/10.1002/jrsm.1753 for more details.
#> boundary (singular) fit: see help('isSingular')
#> Hello and welcome to metaROC!
#> Requested model: hoyer2018glmm 
#> This is a GLMM for multiple thresholds.
#>  See https://doi.org/10.1177/0962280216661587 for more details.
#> Warning: Model failed to converge with max|grad| = 0.00205949 (tol = 0.002, component 1)
#>   See ?lme4::convergence and ?lme4::troubleshooting.
#> Hello and welcome to metaROC!
#> Requested model: stoye2024cloglog 
#> This is a GLMM for the discrete Hazard for multiple thresholds.
#>  See https://doi.org/10.1002/jrsm.1753 for more details.
#> boundary (singular) fit: see help('isSingular')
mods_phaseIII[[1]]$hoyer2018glmm
#> >> Fitted meta-analysis model of class metaROC <<
#> >> Selected model: hoyer2018glmm <<
#> >> Model estimation is converged <<
#> >> See summary() for more details <<

# estimate models on real data
# hba1c is a meta-analysis dataset on the diagnosis of type 2 diabetes, see ?hba1c
mod_hba1c = metaROC(action = "estimate", data=hba1c, model_fit="stoye2024cloglog")
#> Hello and welcome to metaROC!
#> Chosen action: estimate 
#> Hello and welcome to metaROC!
#> Requested model: stoye2024cloglog 
#> This is a GLMM for the discrete Hazard for multiple thresholds.
#>  See https://doi.org/10.1002/jrsm.1753 for more details.
#> Warning: NAs introduced by coercion
# metaROC with action = "evaluate" can not be applied here as the data-generating
# process is unknown. Use SROC(mod_hba1c) instead to get the estimated SROC curve
SROC(mod_hba1c)$sroc_AUC
#> [1] 0.8494565

# Additional test example for Froemke model
mod_hba1c_fr = metaROC(action = "estimate", data=hba1c, model_fit="froemke2022np")
#> Hello and welcome to metaROC!
#> Chosen action: estimate 
#> Hello and welcome to metaROC!
#> Requested model: froemke2022np 
#> This is a non-parametric model for multiple thresholds.
#>  See https://doi.org/10.1002/jrsm.1579 for more details.
#> Note: for this model an unpublished R package is used.
#> See https://github.com/wbr-p/diagacc for details.
#> ℹ Loading metadata database
#> ✔ Loading metadata database ... done
#> 
#>  
#> → Will install 17 packages.
#> → Will download 17 packages with unknown size.
#> + R6           2.6.1  [dl]
#> + cli          3.6.6  [dl]
#> + diagacc      0.1.0  [bld][cmp][dl]
#> + dplyr        1.2.1  [dl]
#> + generics     0.1.4  [dl]
#> + glue         1.8.1  [dl]
#> + lifecycle    1.0.5  [dl]
#> + magrittr     2.0.5  [dl]
#> + pillar       1.11.1 [dl]
#> + pkgconfig    2.0.3  [dl]
#> + purrr        1.2.2  [dl]
#> + rlang        1.2.0  [dl]
#> + tibble       3.3.1  [dl]
#> + tidyselect   1.2.1  [dl]
#> + utf8         1.2.6  [dl]
#> + vctrs        0.7.3  [dl]
#> + withr        3.0.2  [dl]
#>   
#> ℹ Getting 17 pkgs with unknown sizes
#> ✔ Got diagacc 0.1.0 (source) (4.10 kB)
#> ✔ Got generics 0.1.4 (x86_64-pc-linux-gnu-ubuntu-24.04) (80.38 kB)
#> ✔ Got glue 1.8.1 (x86_64-pc-linux-gnu-ubuntu-24.04) (173.49 kB)
#> ✔ Got lifecycle 1.0.5 (x86_64-pc-linux-gnu-ubuntu-24.04) (131.69 kB)
#> ✔ Got magrittr 2.0.5 (x86_64-pc-linux-gnu-ubuntu-24.04) (221.62 kB)
#> ✔ Got purrr 1.2.2 (x86_64-pc-linux-gnu-ubuntu-24.04) (553.64 kB)
#> ✔ Got R6 2.6.1 (x86_64-pc-linux-gnu-ubuntu-24.04) (86.81 kB)
#> ✔ Got utf8 1.2.6 (x86_64-pc-linux-gnu-ubuntu-24.04) (151.81 kB)
#> ✔ Got tidyselect 1.2.1 (x86_64-pc-linux-gnu-ubuntu-24.04) (225.28 kB)
#> ✔ Got cli 3.6.6 (x86_64-pc-linux-gnu-ubuntu-24.04) (1.35 MB)
#> ✔ Got dplyr 1.2.1 (x86_64-pc-linux-gnu-ubuntu-24.04) (1.52 MB)
#> ✔ Got tibble 3.3.1 (x86_64-pc-linux-gnu-ubuntu-24.04) (647.68 kB)
#> ✔ Got rlang 1.2.0 (x86_64-pc-linux-gnu-ubuntu-24.04) (1.61 MB)
#> ✔ Got withr 3.0.2 (x86_64-pc-linux-gnu-ubuntu-24.04) (223.90 kB)
#> ✔ Got pillar 1.11.1 (x86_64-pc-linux-gnu-ubuntu-24.04) (660.46 kB)
#> ✔ Got pkgconfig 2.0.3 (x86_64-pc-linux-gnu-ubuntu-24.04) (18.08 kB)
#> ✔ Got vctrs 0.7.3 (x86_64-pc-linux-gnu-ubuntu-24.04) (1.86 MB)
#> ✔ Installed cli 3.6.6  (43ms)
#> ✔ Installed dplyr 1.2.1  (89ms)
#> ✔ Installed generics 0.1.4  (99ms)
#> ✔ Installed glue 1.8.1  (70ms)
#> ✔ Installed lifecycle 1.0.5  (85ms)
#> ✔ Installed magrittr 2.0.5  (123ms)
#> ✔ Installed pillar 1.11.1  (122ms)
#> ✔ Installed pkgconfig 2.0.3  (79ms)
#> ✔ Installed purrr 1.2.2  (89ms)
#> ✔ Installed R6 2.6.1  (70ms)
#> ✔ Installed rlang 1.2.0  (81ms)
#> ✔ Installed tibble 3.3.1  (91ms)
#> ✔ Installed tidyselect 1.2.1  (82ms)
#> ✔ Installed utf8 1.2.6  (82ms)
#> ✔ Installed vctrs 0.7.3  (115ms)
#> ✔ Installed withr 3.0.2  (68ms)
#> ℹ Packaging diagacc 0.1.0
#> ✔ Packaged diagacc 0.1.0 (670ms)
#> ℹ Building diagacc 0.1.0
#> ✔ Built diagacc 0.1.0 (2s)
#> ✔ Installed diagacc 0.1.0 (git::https://github.com/wbr-p/diagacc@338bf58) (53ms)
#> ✔ 1 pkg + 16 deps: added 17, dld 17 (NA B) [10.6s]
mod_hba1c_fr$auc
#>       auc_l       auc     auc_u
#> 1 0.7304824 0.7629611 0.7954397

# Example for multiple models to be fitted with control_fit
model_ids = c("hoyer2018glmm", "chu2006glmm")
control_fit <- list(
 list(),
 list(link = "probit"))
x <- metaROC(action = "estimate", model_fit = model_ids, data = hba1c,
            control_fit = control_fit)
#> Hello and welcome to metaROC!
#> Chosen action: estimate 
#> Warning: You are trying to fit an STM to a real-world dataset that may report multiple thresholds for
#>                     one or more studies. Please specify which threshold should be used by providing a column named
#>                     threshold_stm. Otherwise, each reported threshold will be treated as a separate study.
#> Hello and welcome to metaROC!
#> Requested model: hoyer2018glmm 
#> This is a GLMM for multiple thresholds.
#>  See https://doi.org/10.1177/0962280216661587 for more details.
#> Hello and welcome to metaROC!
#> Requested model: chu2006glmm 
#> This is a GLMM for a single threshold per study.
#>  See https://doi.org/10.1177/0272989X09353452 for more details.

# evaluate models estimated on simulated data using action = "evaluate"
ev_mods_phaseII = metaROC(action = "evaluate", model_fit = mods_phaseII)
#> Hello and welcome to metaROC!
#> Chosen action: evaluate 
#> Warning: Parameter "summarize" not specified. Returning model-specific results.
ev_mods_phaseII[[1]]$stoye2024cloglog
#> $AUC
#> [1] 0.7631459
#> 
#> $roc_curve
#> $roc_curve$sroc_AUC
#> [1] 0.7631459
#> 
#> $roc_curve$sroc_df
#>    threshold      sens_lo  sensitivity    sens_up     spec_lo specificity
#> 1        5.1 8.693486e-01 9.892456e-01 0.99916532 0.001054023   0.1587425
#> 2        5.4 6.882105e-01 8.831347e-01 0.94559974 0.106494595   0.4415774
#> 3        5.5 3.811130e-01 6.717076e-01 0.83306113 0.219055041   0.6791617
#> 4        5.7 2.999576e-02 3.424392e-01 0.69683271 0.441148274   0.9703573
#> 5        5.9 1.437398e-02 2.330964e-01 0.56988107 0.694643924   0.9956414
#> 6        6.0 6.862732e-03 1.734210e-01 0.50630237 0.777026141   0.9992381
#> 7        6.1 2.295691e-03 1.028578e-01 0.39463012 0.894621270   0.9999542
#> 8        6.2 3.782845e-05 4.399263e-02 0.33104509 0.905921843   0.9999964
#> 9        6.3 7.739715e-06 1.915356e-02 0.21410705 0.966301990   0.9999998
#> 10       6.4 1.722181e-08 3.613879e-03 0.13579238 0.984391727   1.0000000
#> 11       6.7 2.368951e-09 1.231226e-03 0.07569063 0.996262672   1.0000000
#> 12       6.9 0.000000e+00 1.853591e-11 0.07569063 0.996262672   1.0000000
#> 13       7.0 0.000000e+00 1.398936e-12 0.03772653 0.996262672   1.0000000
#>    spec_up youden_index
#> 1        1 1.479881e-01
#> 2        1 3.247120e-01
#> 3        1 3.508693e-01
#> 4        1 3.127965e-01
#> 5        1 2.287378e-01
#> 6        1 1.726591e-01
#> 7        1 1.028120e-01
#> 8        1 4.398904e-02
#> 9        1 1.915340e-02
#> 10       1 3.613878e-03
#> 11       1 1.231226e-03
#> 12       1 1.853584e-11
#> 13       1 1.398881e-12
#> 
#> $roc_curve$opt_threshold
#> [1] 5.5
#> 
#> 
#> $Bias_AUC
#> [1] -0.08682753
#> 
#> $Bias_sens
#> [1] 0.01787233
#> 
#> $Bias_spec
#> [1] -0.2284668
#> 
#> $Bias_threshold
#> [1] 0
#> 
#> $coverage_sens
#> [1] TRUE
#> 
#> $coverage_spec
#> [1] TRUE
#> 
#> $se_sensspec
#> $se_sensspec$sens
#> [1] 0.1482653
#> 
#> $se_sensspec$spec
#> [1] 0.2347526
#> 
#> 
#> attr(,"class")
#> [1] "evaluated"

ev_mods_phaseIII = metaROC(action = "evaluate", model_fit = mods_phaseIII)
#> Hello and welcome to metaROC!
#> Chosen action: evaluate 
#> Warning: Parameter "summarize" not specified. Returning model-specific results.
ev_mods_phaseIII[[1]]$stoye2024cloglog
#> $AUC
#> [1] 0.9178889
#> 
#> $roc_curve
#> $roc_curve$sroc_AUC
#> [1] 0.9178889
#> 
#> $roc_curve$sroc_df
#>    threshold      sens_lo sensitivity   sens_up   spec_lo specificity   spec_up
#> 1        4.6 0.8894859658  0.95890411 0.9850756 0.1021551   0.5739130 0.9988338
#> 2        4.7 0.6355011836  0.83517455 0.9307168 0.2234237   0.8222739 0.9999940
#> 3        5.0 0.3908834386  0.72741009 0.8948757 0.2972442   0.9500906 1.0000000
#> 4        5.2 0.3217843340  0.65860098 0.8505825 0.5152301   0.9881650 1.0000000
#> 5        5.5 0.2026881955  0.51747220 0.7500154 0.7052010   0.9981856 1.0000000
#> 6        5.6 0.1435459863  0.43217458 0.6826980 0.8302543   0.9997740 1.0000000
#> 7        5.9 0.0893768954  0.32769281 0.5807934 0.9247503   0.9999810 1.0000000
#> 8        6.2 0.0670135446  0.30632154 0.5716921 0.9327191   0.9999976 1.0000000
#> 9        6.3 0.0272775190  0.21206876 0.4918428 0.9525483   0.9999997 1.0000000
#> 10       7.1 0.0121754381  0.11589804 0.3128097 0.9937703   1.0000000 1.0000000
#> 11       7.2 0.0021652403  0.04486376 0.1856736 0.9975600   1.0000000 1.0000000
#> 12       7.5 0.0001893399  0.01207870 0.0915951 0.9990601   1.0000000 1.0000000
#>    youden_index
#> 1    0.53281715
#> 2    0.65744844
#> 3    0.67750070
#> 4    0.64676599
#> 5    0.51565777
#> 6    0.43194861
#> 7    0.32767378
#> 8    0.30631918
#> 9    0.21206850
#> 10   0.11589804
#> 11   0.04486376
#> 12   0.01207870
#> 
#> $roc_curve$opt_threshold
#> [1] 5
#> 
#> 
#> $Bias_AUC
#> [1] 0.01766035
#> 
#> $Bias_sens
#> [1] -0.09325875
#> 
#> $Bias_spec
#> [1] 0.0789953
#> 
#> $Bias_threshold
#> [1] -0.8448448
#> 
#> $coverage_sens
#> [1] TRUE
#> 
#> $coverage_spec
#> [1] TRUE
#> 
#> $se_sensspec
#> $se_sensspec$sens
#> [1] 0.1717004
#> 
#> $se_sensspec$spec
#> [1] 0.333091
#> 
#> 
#> attr(,"class")
#> [1] "evaluated"

# setting summarize = TRUE gives additional aggregated information on
# performance measures
ev_mods_phaseII = metaROC(action = "evaluate", model_fit = mods_phaseII,
                         summarize = TRUE)
#> Hello and welcome to metaROC!
#> Chosen action: evaluate 
ev_mods_phaseII$results_long
#>    replicate            model     AUC bias sensitivity bias
#> 1          1    hoyer2018glmm -0.046000956       0.03378029
#> 2          2    hoyer2018glmm  0.006069071       0.10050165
#> 3          3    hoyer2018glmm -0.035104972       0.10279746
#> 4          4    hoyer2018glmm -0.050599789       0.06420092
#> 5          5    hoyer2018glmm -0.051459051       0.01489479
#> 6          1 stoye2024cloglog -0.086827534       0.01787233
#> 7          2 stoye2024cloglog -0.050528602      -0.16607278
#> 8          3 stoye2024cloglog -0.091634525       0.16359393
#> 9          4 stoye2024cloglog -0.034310101       0.03406288
#> 10         5 stoye2024cloglog  0.003699266       0.18775413
#>    sensitivity standard error specificity bias specificity standard error
#> 1                   0.3433526      -0.10408617                  0.3658790
#> 2                   0.3820775      -0.08063324                  0.3707178
#> 3                   0.3860441      -0.16605056                  0.3715052
#> 4                   0.3636545      -0.15395921                  0.3116734
#> 5                   0.3262492      -0.08581949                  0.3975250
#> 6                   0.1482653      -0.22846684                  0.2347526
#> 7                   0.2458344       0.05368295                  0.3977633
#> 8                   0.1697686      -0.23069126                  0.2925567
#> 9                   0.3233882      -0.07564032                  0.3897586
#> 10                  0.1682833      -0.16845575                  0.3005837
#>    threshold bias sensitivity coverage specificity coverage convergence
#> 1      0.31034034                 TRUE                 TRUE        TRUE
#> 2      0.49840841                 TRUE                 TRUE        TRUE
#> 3      0.27526527                 TRUE                 TRUE        TRUE
#> 4      0.35726727                 TRUE                 TRUE        TRUE
#> 5      0.09495495                 TRUE                 TRUE        TRUE
#> 6      0.00000000                 TRUE                 TRUE        TRUE
#> 7      0.30000000                 TRUE                 TRUE        TRUE
#> 8      0.00000000                 TRUE                 TRUE        TRUE
#> 9      0.00000000                 TRUE                 TRUE        TRUE
#> 10     0.00000000                 TRUE                 TRUE        TRUE

ev_mods_phaseIII = metaROC(action = "evaluate", model_fit = mods_phaseIII,
                          summarize = TRUE)
#> Hello and welcome to metaROC!
#> Chosen action: evaluate 
ev_mods_phaseIII$results_long
#>    replicate            model     AUC bias sensitivity bias
#> 1          1    hoyer2018glmm  0.013919251      0.010972888
#> 2          2    hoyer2018glmm  0.018935259      0.008127183
#> 3          3    hoyer2018glmm -0.013040879     -0.029314455
#> 4          4    hoyer2018glmm  0.016830781      0.052207607
#> 5          5    hoyer2018glmm -0.032125191     -0.042480393
#> 6          1 stoye2024cloglog  0.017660350     -0.093258747
#> 7          2 stoye2024cloglog  0.040471964      0.008197969
#> 8          3 stoye2024cloglog -0.056195509     -0.237551202
#> 9          4 stoye2024cloglog  0.018089469      0.036474021
#> 10         5 stoye2024cloglog -0.009227201     -0.094018784
#>    sensitivity standard error specificity bias specificity standard error
#> 1                   0.3075754     -0.005488612                  0.1947146
#> 2                   0.3891709      0.015887704                  0.1434808
#> 3                   0.3872443     -0.019035086                  0.3057784
#> 4                   0.3696132     -0.030847818                  0.2914852
#> 5                   0.3160932     -0.047989766                  0.1945626
#> 6                   0.1717004      0.078995297                  0.3330910
#> 7                   0.2748006      0.115764761                  0.4220403
#> 8                   0.2397499      0.072209366                  0.3304310
#> 9                   0.3616741      0.059137242                  0.4662935
#> 10                  0.2217849      0.049845280                  0.3719614
#>    threshold bias sensitivity coverage specificity coverage convergence
#> 1     -0.07952953                 TRUE                 TRUE        TRUE
#> 2      0.48147147                 TRUE                 TRUE        TRUE
#> 3      0.22815816                 TRUE                 TRUE        TRUE
#> 4     -0.47841842                 TRUE                 TRUE        TRUE
#> 5      0.02117117                 TRUE                 TRUE        TRUE
#> 6     -0.84484484                 TRUE                 TRUE        TRUE
#> 7     -0.44484484                 TRUE                 TRUE        TRUE
#> 8     -0.74484484                 TRUE                 TRUE        TRUE
#> 9     -1.04484484                 TRUE                 TRUE        TRUE
#> 10    -0.84484484                 TRUE                 TRUE        TRUE