Using multiple imputation to estimate missing data in meta‐

Methods in Ecology and Evolution 2015, 6, 153–163

doi: 10.1111/2041-210X.12322

Using multiple imputation to estimate missing data in meta-regression E. Hance Ellington1*, Guillaume Bastille-Rousseau1, Cayla Austin1, Kristen N. Landolt1, Bruce A. Pond2, Erin E. Rees3, Nicholas Robar1 and Dennis L. Murray4 1

Environmental and Life Sciences, Trent University, 2140 East Bank Drive, Peterborough, ON K9J 7B8, Canada; 2Wildlife Research & Monitoring Section, Ontario Ministry of Natural Resources and Forestry, 2140 East Bank Drive, Peterborough, ON K9J 7B8, Canada; 3Department of Health Management, Atlantic Veterinary College, University of Prince Edward Island, Charlottetown, PE C1A 4P3, Canada; and 4Biology Department, Trent University, 2140 East Bank Drive, Peterborough, ON K9J 7B8, Canada

Summary 1. There is a growing need for scientific synthesis in ecology and evolution. In many cases, meta-analytic techniques can be used to complement such synthesis. However, missing data are a serious problem for any synthetic efforts and can compromise the integrity of meta-analyses in these and other disciplines. Currently, the prevalence of missing data in meta-analytic data sets in ecology and the efficacy of different remedies for this problem have not been adequately quantified. 2. We generated meta-analytic data sets based on literature reviews of experimental and observational data and found that missing data were prevalent in meta-analytic ecological data sets. We then tested the performance of complete case removal (a widely used method when data are missing) and multiple imputation (an alternative method for data recovery) and assessed model bias, precision and multimodel rankings under a variety of simulated conditions using published meta-regression data sets. 3. We found that complete case removal led to biased and imprecise coefficient estimates and yielded poorly specified models. In contrast, multiple imputation provided unbiased parameter estimates with only a small loss in precision. The performance of multiple imputation, however, was dependent on the type of data missing. It performed best when missing values were weighting variables, but performance was mixed when missing values were predictor variables. Multiple imputation performed poorly when imputing raw data which were then used to calculate effect size and the weighting variable. 4. We conclude that complete case removal should not be used in meta-regression and that multiple imputation has the potential to be an indispensable tool for meta-regression in ecology and evolution. However, we recommend that users assess the performance of multiple imputation by simulating missing data on a subset of their data before implementing it to recover actual missing data.

Key-words: multiple imputation, missing data, meta-analysis, meta-regression, complete case removal, partial data, ecological synthesis, model selection

Introduction During the last few decades, field and experimental studies have formed the backbone of ecology and evolution research, resulting in the development of large and complex data sets ranging from population trend and abundance (NERC 2010), to global data bases of mammal life history traits (Jones et al. 2009) and plant traits (Kattge et al. 2011). The existence of such data sets, combined with the need to address broad research questions that necessarily span singular studies, has led to the recognized need for robust methods of synthesizing data in ecology and evolution. Indeed, it is clear that by integrating data from a variety of sources, synthetic approaches *Correspondence author. E-mail: [email protected]

can yield not only an increased understanding of the mechanisms underlying general patterns and processes, but also help develop new avenues for future research, promote generality across studies, and assess validity of ecological theories (Carpenter et al. 2009; Pickett, Kolasa & Jones 2010). Meta-analysis is a synthetic tool that provides a statistical framework for combining results of multiple studies addressing a common research question (Arnqvist & Wooster 1995). Meta-analysis relies on inference derived from a collective body of evidence rather than individual case studies (Borenstein et al. 2011). Such techniques were first adopted by the medical and social sciences but are becoming increasingly common in ecology and evolution research, especially meta-regression which is a form of meta-analysis that allows for the inclusion of predictor variables (Chaudhary et al. 2010;

© 2014 The Authors. Methods in Ecology and Evolution © 2014 British Ecological Society

154 E. H. Ellington et al. Cadotte, Mehrkens & Menge 2012). Despite this increasing interest, several challenges constrain the wider use of metaanalysis. One major hurdle concerns the generally poor pattern of reporting ecological data, which can lead to meta-analytic data sets with studies that contain missing data (Gurevitch & Hedges 1999). In fact, meta-analyses have been abandoned in some instances due to large amounts of missing data (e.g. Riley et al. 2003). There are two kinds of missing data that challenge metaanalysts: analyses that are conducted but unpublished (commonly referred to as publication bias), and published analyses that do not provide all of the data necessary for meta-analysis. The occurrence, influence and methods for dealing with publication bias have been covered elsewhere (e.g. Jennions et al. 2013), thus we focus our efforts here on the question of missing data in published studies. The common approach to account for missing data in meta-analyses is to exclude studies that have missing data (i.e. complete case removal). However, this approach reduces the sample size and attendant statistical power of the meta-analysis, and unless the occurrence of missing values is a random event, it introduces bias (Raghunathan 2004). Indeed, Nakagawa & Freckleton (2011) showed that employing complete case removal in multivariate analyses can cause a variety of problems in addition to biased parameter estimates, including incorrect effect size estimates and model rankings, equivocal model comparisons, and misestimation of correlation structure in the data. Lajeunesse (2013) offered several suggestions for managing missing data in meta-analyses, including: (i) contacting author (s) and asking for access to raw data to fill in gaps, (ii) using ocular tools to measure data supplied only in figures (e.g. calipers to measure the height of standard error (SE) bars), (iii) using formulas to convert reported data into effect sizes and measures of variance that are useable for meta-analysis (i.e. convert p-values into mean and SE), and (iv) using multiple imputation to estimate missing values. While the first three suggestions are straightforward and easily implemented by any meta-analyst, the use of multiple imputation to estimate missing values in meta-analysis is not well explored. The aim of multiple imputation is to estimate missing values from the data set within bounds that are set by the existing data and thereby provide estimates of variance accounting for this estimation (for overview see Rubin 1996). There are three steps in multiple imputation. First, values are inserted where missing data occur, creating an imputed data set. The values are selected from a distribution model derived from the relationships among variables for cases without missing data. This process is repeated multiple times. Secondly, the statistical analysis of interest is conducted on each imputed data set, and finally, the results are pooled from each analysis of the imputed data sets (Nakagawa & Freckleton 2008). Nakagawa & Freckleton (2011) demonstrated that multiple imputation techniques recovered model weights and parameter estimates reasonably well in a multivariate analysis within an AIC framework. Yet, to date, the use of multiple imputation to estimate missing data in ecology and evolution has rarely been employed (but see Fisher, Blomberg & Owens 2003).

Although multiple imputation techniques for meta-analysis have been used for some time in the medical and social sciences (Boyle et al. 2004; Robertson, Idris & Boyle 2004), its use is still inconsistent (Wiebe et al. 2006). The few studies that examined imputation in a meta-analytic framework did so either using simple imputation processes, restricted modelling scenarios, or gauged against limited metrics (Thiessen Philbrook, Barrowman & Garg 2007; Idris & Robertson 2009; Idris & Sarudin 2011). A few manuscripts have described or used multiple imputation or similar techniques to recover missing data in ecological meta-analyses (Cleasby & Nakagawa 2012; Nakagawa & Santos 2012). However, these studies have not tested the performance of multiple imputation in the metaregression context, nor has this technique been widely adopted by ecologists. In this study, we evaluate the issue of missing data in ecological meta-regression and assess the performance of multiple imputation to resolve this problem. Our first objective was to determine the prevalence of missing data across a sample of ecological meta-analytic data sets. Our second objective was to determine the impact that missing data can have on ecological meta-regression when complete case removal is applied. Our third objective was to assess the performance of multiple imputation in estimating missing values in a meta-regression framework, using both univariate and multimodel multivariate regression. Specifically, we evaluated the performance of multiple imputation using a number of factors: percentage of missing values, type of missing values and imputation strategy. To our knowledge, this is the first study to comprehensively assess the sensitivity of meta-regression data sets to missing data and to thoroughly evaluate the performance of multiple imputation techniques in a meta-regression framework.

Methods PREVALENCE OF MISSING DATA

To investigate the prevalence and type of missing data, we examined four unpublished data sets in four thematic areas characteristic of research in ecology and evolution (Appendix S1). We categorized missing data into whether they were missing effect size, the weighting variable or methodological information. Within each data set, studies could have multiple types of missing data. We determined the sample size of each data set, excluding studies with missing data, and we determined improvements in the sample size of each data set if missing data were recovered using algebraic recalculations and multiple imputation.

ANALYSING PERFORMANCE OF MULTIPLE IMPUTATION

To assess the performance of multiple imputation on ecological data in a meta-regression framework, we used published data from Rees et al. (2008; hereafter Rees data set) and Robar, Burness & Murray (2010; hereafter Robar data set) to generate two data sets. For our purposes, we consider these data sets to be complete (i.e. they do not have missing data). We then systematically removed values to determine whether multiple imputation could estimate the values that we removed. The complete data set served as a baseline to which we compared results

© 2014 The Authors. Methods in Ecology and Evolution © 2014 British Ecological Society, Methods in Ecology and Evolution, 6, 153–163

Multiple imputation in meta-regression between incomplete and imputed versions of the same data set. All analyses were conducting using the statistical program R v3.02 (R Core Team 2013).

COMPLETE DATA SETS

We used the Rees data set to represent a straightforward meta-regression analysis (Thompson & Higgins 2002) where raccoon (Procyon lotor) population density was considered as a function of latitude. The Rees data set included three variables used in the mixed-effects metaregression: mean raccoon population density (effect size), standard error (SE) of raccoon population density (when transformed into the inverse variance became the weighting variable) and latitude (continuous predictor). The Rees data set also included two multiclass categorical predictors not involved in the meta-regression analysis but used in our imputation models. Rees et al. (2008) did not test for publication bias. We used the Robar data set to represent a complex multimodel meta-regression analysis involving ecological and environmental predictors of parasite-associated mortality. The Robar data set included eight variables used in the mixed-effects meta-regression models: (i) log odds ratio (LOR) of host mortality (effect size), (ii) SE of the LOR (when transformed into the inverse variance became the weighting variable), and (iii) latitude (continuous predictor), and several categorical predictors [predation-mediated mortality, host taxon (multiclass), parasite taxon (multiclass), load type and study type]. The Robar data set also included four binary and two multiclass categorical predictors not involved in the meta-regression models but used in our imputation models. Robar, Burness & Murray (2010) tested for publication bias using funnel plots (Egger et al. 1997) and found that studies reporting non-significant results were slightly under-represented in their sample. However, correcting for funnel asymmetry did not influence their results qualitatively, and there was no evidence of publication bias in any of the predictors except parasite taxon (Robar, Burness & Murray 2010). We used the top ten models, as reported by Robar, Burness & Murray (2010), as our model set for the Robar data set (Table S1) and calculated model-averaged coefficients, unconditional SE of the modelaveraged coefficients and importance values.

INCOMPLETE DATA SETS

For each data set, we created 27 unique situations of missing and imputed data. First, we systematically removed 5–50% of the values from particular variables at intervals of 5% (Table S2). Secondly, we varied the way in which we removed data to create our incomplete data sets (Rubin 1976). We generated data sets with data missing completely at random (MCAR) by randomly removing (uniform distribution) values within the variable of interest. We generated data sets with data missing at random (MAR) by setting the likelihood of the value being removed equal to the inverse of one other variable in the data set (for SE we used latitude, and for latitude we used SE). To generate data sets with data missing not at random (MNAR), we set the likelihood of the value being removed equal to the inverse of that value raised to the fourth. For each missing data situation and per cent removal, we generated 100 incomplete data sets. We recognize that solely removing the SE of the LOR from the Robar data set is unrealistic because the SE of the LOR is a compound variable derived from the same four components of raw data from which the LOR is derived (Robar, Burness & Murray 2010). However, incomplete data sets simulated in this manner still have explanatory merit because they can act as surrogates for other ecological data sets, which might have similar missing data patterns. To simulate realistic

155

incomplete data sets based on the Robar data set, we first expanded the data set to include the four raw data components on which the LOR and the SE of the LOR were based. Then, to simulate incomplete data sets, we followed the methods described above, removing one raw data component, randomly chosen from amongst the four components for each incomplete data set. To create raw data MCAR, MAR and MNAR, we used the rules of removal for SE of the LOR, since the raw data are used to calculate the SE of the LOR. In each incomplete data set, whenever a study had a missing raw data component we also removed the LOR and the SE of the LOR.

IMPUTED DATA SETS

For each incomplete data set, we used multiple imputation to create ten imputed data sets using the R package mi (Su et al. 2011). The package mi uses a chained equation algorithm approach in conjunction with a boot-strapped procedure. This approach is detailed in Su et al. (2011); however, we also briefly describe it in Appendix S2. For an extensive overview of the general process of multiple imputation please see van Buuren (2012). When variables were positive continuous, such as SE, we log-transformed the variable prior to imputation and then transformed the variable back prior to running the meta-regression analyses. When variables were proportions, such as latitude, we logit-transformed the variable prior to imputation and then transformed the variable back prior to running the meta-regression analyses using the R package boot (Canty & Ripley 2009). We manually specified that each variable was continuous, thus ensuring that the imputation model used by mi was not altered when large amounts of data were missing. When using multiple imputation to recover the raw data, the effect size and the weighting parameter, we actively imputed the raw data and passively imputed (i.e. calculation based on the raw data imputed) the effect size and weighting parameter. The performance of multiple imputation did not differ when we actively imputed the effect size and weighting parameter in addition to the raw data (Ellington, unpubl.). To determine how imputation strategy influenced performance, we increased the number of imputed data sets per incomplete data set from 10 to 100, and we used predictive mean-matching (PMM) to impute missing data in addition to the Bayesian GLM (Table S3). PMM is a general purpose, semiparametric imputation method, in which imputations are restricted to observed values. This is achieved by forming a small set of values from all the complete cases which are close to the predicted value based on the Bayesian GLM described in Appendix S2 and then randomly drawing from within this subset (Su et al. 2011; van Buuren 2012). We specified PMM as the imputation model using the package mi. We present the results of these analyses in the Appendix S3.

STATISTICAL ANALYSIS

We replicated the analysis of Rees et al. (2008) and Robar, Burness & Murray (2010) and conducted all subsequent meta-regressions using the R package metafor (Viechtbauer 2010). For each missing data situation (MCAR, MAR, MNAR), we had one complete data set, one hundred incomplete data sets and one hundred sets of either 10 or 100 imputed data sets. For the Robar data sets, we calculated several AICderived metrics including model weights, model-averaged coefficients and associated unconditional SE of the model-averaged coefficients. For handling multiple imputation and model averaging, we followed Nakagawa & Freckleton (2011) and conducted model averaging on the results of each imputed data set within the set of imputed data sets then generated mean model-averaged coefficient estimates and associated

© 2014 The Authors. Methods in Ecology and Evolution © 2014 British Ecological Society, Methods in Ecology and Evolution, 6, 153–163

156 E. H. Ellington et al. unconditional SE of the model-averaged coefficients. For model averaging, we used the estimator method where the coefficients were averaged only from models in which the variable of interest appeared (Burnham & Anderson 2002). Within each missing data situation at a given per cent of values removed, we averaged and generated 95% quantile confidence limits around the parameter estimates, associated SE of the parameter estimates, and other metrics from both the incomplete data sets (n = 100) and the sets of imputed data (n = 100). For the Rees data set, we compared the coefficient of latitude, SE of the latitude coefficient, and the adjusted R2 (L
opez-L
opez et al. 2013) between the complete, incomplete and imputed results. For the Robar data set, we compared the model-averaged coefficients for two important predictors (predation-mediated mortality and latitude) and the unconditional SE of these model-averaged coefficients between the complete, incomplete and imputed results. The process of multiple imputation inherently adds uncertainty around parameter estimates, and we predicted that the SE associated with a parameter estimate should increase by the standard deviation of the difference between the parameter estimates of the complete data set and the imputed data set (see Appendix S4 for derivation of this expectation). We calculated this value at each per cent of values removed for each missing data situation. We also compared the relative model weights and relative importance values of the incomplete and imputed data sets relative to the complete data set. Rees et al. (2008) and Robar, Burness & Murray (2010) used different software and equations for estimating the adjusted R2, thus, our results of the complete data sets are slightly different from their published results. We provide an annotated sample of the R code we used in Appendix S5.

INCOMPLETE DATA SETS

When data sets have missing data, the coefficient estimates of important predictors varied widely from the complete-data values and became less accurate as the amount of missing data increased (Figs 1–3). We show only figures with the estimated latitude coefficients for the Rees data set because its relationship with per cent missing data was the inverse of the relationship between per cent missing data and the adjusted R2 (Fig. S2). The SE of the coefficient estimates when data were missing was generally larger than the SE of the coefficient estimates from the complete data (Figs S2, S3 and S5). The relative model weights, model structure and importance values were highly altered when using incomplete data sets, and this pattern was exacerbated when data were MAR or MNAR (Tables 2 and 3). MULTIPLE IMPUTATION OF WEIGHTING PARAMETERS

Multiple imputation performed remarkably well at generating coefficient estimates of important predictors that were close to the complete-data estimates, especially with the Rees data set (Fig. 1a–c). The performance of multiple imputation was fairly robust to different scenarios of missing weighting parameters (MCAR, MAR and MNAR) and also fairly robust to large percentages of missing weighting parameters (Fig. 1). The coefficient estimate for the continuous predictor (latitude) when the weighting parameter was MAR and MNAR in the Robar data set was slightly underestimated with multiple imputation but it was more accurate than the incomplete data sets (Fig. 1h,i). Multiple imputation also estimated the SE of the coefficients better than the incomplete data sets, although the SE estimated by multiple imputation was generally less than what was expected due to the imputation process itself (Fig. S2). Multiple imputation performed considerably better than incomplete data sets in estimating the relative model weights of the top four models and maintaining the order of model rankings, regardless of the missing data scenario (Table 2). Multiple imputation also replicated the importance values of predation-mediated mortality, latitude and host taxon predictors well in the Robar data set regardless of the missing data scenario (Table 3).

Results PREVALENCE OF MISSING ECOLOGICAL DATA

We developed four separate meta-analytic data sets meant to reflect the diversity of such data sets currently available in ecology and evolution and found that 14–22% of the individual studies (mean = 18%) within each data set had missing data (Table 1). The weighting variable or data required to calculate the weighting variable in the meta-analytic data set was missing in up to 20% of the cases, methodological information was missing in 2–14% of the cases, and effect size was missing from 2% to 16% of the cases (Table 1). We estimated that by using algebraic recalculations and multiple imputation, we could increase the sample size by an average of 13% and up to 26% in the four data sets explored (Table 1).

Table 1. The types and quantities of missing data encountered in four unpublished ecological meta-analyses Number of studies with missing data

Meta-analytic data set

Type of study

Number of studies found

Effect size

Weighting parameters

Methodology issues

Useable studies

Recoverable studies

Behavioural response to perceived predation risk Morphological response to perceived predation risk Antipredator chemical cues Coyote home range size

Experimental

80

3

5

6

69

5

Experimental

34

3

0

3

28

3

Experimental Observational

79 60

13 1

13 12

11 1

63 47

5 12

© 2014 The Authors. Methods in Ecology and Evolution © 2014 British Ecological Society, Methods in Ecology and Evolution, 6, 153–163

Difference from complete dataset (latitude coefficient)

Difference from complete dataset (predation-mediated coefficient)

Difference from complete dataset (latitude coefficient)

Multiple imputation in meta-regression (a)

(d)

(g)

(b)

157

(c)

(e)

(f)

(h)

(i)

Percent of missing data Fig. 1. Quantiles (95%) around the differences between coefficients from the complete data set and both the incomplete (n = 100) [blue (light grey)] and imputed (n = 100) [green (dark grey)] data sets across per cent of missing data. Standard error is missing completely at random (a, d, g), missing at random (b, e, h) or missing not at random (c, f, i). Data are based on the Rees et al. (2008) data set (a–c) and the Robar, Burness & Murray (2010) data set (d–i).

MULTIPLE IMPUTATION OF PREDICTORS

The performance of multiple imputation in generating coefficient estimates of important predictors close to the completedata values was markedly different between missing weighting parameters (Fig. 1) and missing predictors (Fig. 2). For the Rees data set, multiple imputation produced coefficient estimates for latitude that were just as inaccurate and imprecise as estimates from incomplete data sets, regardless of the amount or type of missing data (Fig. 2a–c). For the Robar data set, multiple imputation produced coefficients for the continuous predictor (latitude) that were no more accurate or precise than

the incomplete data set (Fig. 2g–i). However, multiple imputation performed well when estimating the coefficient estimates of the binary predation-mediated mortality predictor when the predictor (latitude) was missing (Fig. 2d–f). The performance of multiple imputation in estimating the SE of the coefficients reflected its performance in estimating the coefficients themselves: multiple imputation performed poorly when estimating the SE of the coefficient which was missing (latitude) and performed well when estimating the SE of the non-missing coefficient (predation mediated) (Fig. S3). The additional variation due to the imputation process itself was underestimated (Fig S3). Multiple imputation was able to recover the relative

© 2014 The Authors. Methods in Ecology and Evolution © 2014 British Ecological Society, Methods in Ecology and Evolution, 6, 153–163

Difference from complete dataset (latitude coefficient)

Difference from complete dataset (predation-mediated coefficient)

Difference from complete dataset (latitude coefficient)

158 E. H. Ellington et al. (a)

(d)

(g)

(b)

(c)

(e)

(f)

(h)

(i)

Percent of missing data Fig. 2. Quantiles (95%) around the differences between coefficients from the complete data set and both the incomplete (n = 100) [blue (light grey)] and imputed (n = 100) [green (dark grey)] data sets across per cent of missing data. Latitude is missing completely at random (a, d, g), missing at random (b, e, h) or missing not at random (c, f, i). Data are based on the Rees et al. (2008) data set (a–c) and the Robar, Burness & Murray (2010) data set (d–i).

model weights, although when latitude was MNAR, the model weight of two of the top four models was overestimated (Table 2). The relative importance values were recovered remarkably well (Table 3). MULTIPLE IMPUTATION OF RAW DATA

For some effect size metrics, such as log-odds ratio, if raw data are missing then the effect size and the weighting parameter for that effect size are also missing. In this situation, coefficient estimates of important predictors following multiple imputation were not more accurate or precise than when data sets were incomplete (Fig. 3). The SE of the coefficient estimates

following multiple imputation was generally less accurate than the SE of the coefficient estimates from the incomplete data sets and the bias tended to increase sharply as the per cent of missing data increased (Fig. S4). Furthermore, multiple imputation was not able to accurately recover the relative model weights and importance values were less accurate than the incomplete data set (Tables 2 and 3).

Discussion Missing data were common in our sample of ecological metaanalytic data sets: as many as 41% of primary studies had missing data. Omitting studies with missing data from the

© 2014 The Authors. Methods in Ecology and Evolution © 2014 British Ecological Society, Methods in Ecology and Evolution, 6, 153–163

Difference from complete dataset (latitude coefficient)

Difference from complete dataset (predation-mediated coefficient)

Multiple imputation in meta-regression (a)

(d)

(b)

159

(c)

(e)

(f)

Percent of missing data Fig. 3. Quantiles (95%) around the differences between coefficients from the complete data set and both the incomplete (n = 100) [blue (light grey)] and imputed (n = 100) [green (dark grey)] data sets across per cent of missing data. Raw data are missing completely at random (a, d), missing at random (b, e) or missing not at random (c, f). All panels are based on the Robar, Burness & Murray (2010) data set.

meta-regression produced inaccurate coefficient estimates and over- or underestimated the strength of models and predictors. We found that multiple imputation could accurately reproduce coefficient estimates, relative model weights and relative predictor importance when only the weighting parameter was missing. Furthermore, these estimates were more accurate than values produced with a complete case removal technique. However, the performance of multiple imputation was mixed when independent predictors were missing and was poor when raw data used to derive both the effect size and the weighting parameter were missing. The performance of multiple imputation was largely unaffected by the missing data scenario (MCAR, MAR or MNAR), whether the model was univariate or multivariate, or whether the user employed an AIC framework. We found that alternative imputation strategies, particularly the use of PMM instead of GLM could alter performance in certain missing data situations. However, we agree with Marshall, Altman & Holder (2010) that the use of PMM could be problematic when sample sizes are small or when there are rare values within the imputed variable. As suggested by Lajeunesse & Forbes (2003), we recommend that researchers using meta-regression explicitly describe missing data in the data set by reporting the number of studies with missing data and describing the type of missing data (effect size, weighting variable, methodological or predictors). We agree with the recommendations of Lajeunesse (2013) that

the first step to recover missing data should be to contact author (s) of the original studies. This approach is often unsuccessful (Chan et al. 2004), but it is the only method that can truly recover the missing data (missing methodological information in particular). The second step is to use tools to estimate data from figures, such as calipers to measure error bars, or use formulas to convert non-effect size statistics into effect sizes and measures of variance. Care should be taken to retain the level of precision in the original reported statistic (Lajeunesse 2013). The number of studies recovered using these methods should be noted in the results. After these steps have been taken, we recommend researchers consider multiple imputation to estimate remaining missing data. However, before using multiple imputation, we recommend that researchers test the effectiveness of multiple imputation on their own data using their desired analysis framework, keeping in mind that alternative imputation strategies might perform better. This could be done by simulating missing data from the subset of the complete cases and examining the performance of multiple imputation on that subset. When using multiple imputation, be sure to report software used and any deviations from the default methods, the number of imputed data sets, the imputation model and method of pooling repeated estimates (for a comprehensive list see van Buuren 2012). The use of complete case removal should be limited to situations where these methods have been exhausted.

© 2014 The Authors. Methods in Ecology and Evolution © 2014 British Ecological Society, Methods in Ecology and Evolution, 6, 153–163

160 E. H. Ellington et al. Table 2. Model weights calculated from AICc using the Robar, Burness & Murray (2010) multimodel multivariate meta-regression analysis of the complete data set and three simulated incomplete data sets [40% of either the SE of log odd ratio (LOR; when transformed into the inverse variance became the weighting variable), latitude (continuous independent predictor) or raw data (used to derive LOR and SE of LOR) were missing under three different scenarios1)]. Model weights were calculated from AICc for the three corresponding imputed data sets in which multiple imputation was applied to the simulated incomplete data sets. Model weights greater than 0
15 are in bold font AICc Model weights Data set missing SE of LOR

Data set missing latitude

Data set missing raw data3

Missing data scenario1

Model

Complete data set2

Incomplete

Imputed

Incomplete

Imputed

Incomplete

Imputed

MCAR

Model 1 Model 2 Model 3 Model 4 Model 5 Model 6 Model 7 Model 8 Model 9 Model 10 Model 1 Model 2 Model 3 Model 4 Model 5 Model 6 Model 7 Model 8 Model 9 Model 10 Model 1 Model 2 Model 3 Model 4 Model 5 Model 6 Model 7 Model 8 Model 9 Model 10

0
26 0
25 0
19 0
19 0
08 0
02 0
01 0
00 0
00 0
00 0
26 0
25 0
19 0
19 0
08 0
02 0
01 0
00 0
00 0
00 0
26 0
25 0
19 0
19 0
08 0
02 0
01 0
00 0
00 0
00

0
16 0
18 0
07 0
10 0
02 0
07 0
17 0
13 0
10 0
01 0
07 0
10 0
03 0
04 0
00 0
02 0
28 0
25 0
15 0
06 0
17 0
31 0
11 0
25 0
03 0
05 0
03 0
03 0
02 0
00

0
26 0
21 0
22 0
19 0
09 0
03 0
00 0
00 0
00 0
00 0
36 0
12 0
26 0
09 0
13 0
03 0
00 0
00 0
00 0
00 0
22 0
21 0
21 0
22 0
12 0
02 0
00 0
00 0
00 0
00

0
36 0
00 0
15 0
00 0
04 0
11 0
13 0
11 0
09 0
01 0
40 0
00 0
29 0
00 0
09 0
09 0
05 0
04 0
03 0
01 0
15 0
00 0
08 0
00 0
01 0
07 0
31 0
19 0
12 0
07

0
28 0
22 0
20 0
17 0
11 0
01 0
00 0
00 0
00 0
00 0
26 0
25 0
19 0
19 0
10 0
01 0
00 0
00 0
00 0
00 0
37 0
11 0
25 0
09 0
14 0
02 0
00 0
00 0
00 0
00

0
16 0
34 0
07 0
19 0
01 0
04 0
07 0
06 0
05 0
00 0
23 0
16 0
07 0
06 0
01 0
04 0
13 0
16 0
11 0
02 0
18 0
31 0
13 0
28 0
03 0
05 0
01 0
00 0
00 0
00

0
18 0
09 0
21 0
12 0
40 0
01 0
00 0
00 0
00 0
00 0
28 0
03 0
25 0
03 0
38 0
02 0
00 0
00 0
00 0
00 0
04 0
02 0
05 0
02 0
87 0
01 0
00 0
00 0
00 0
00

MAR

MNAR

1

MCAR, missing completely at random; MAR, missing at random; MNAR, missing not at random. The AICc weights are different from those reported in Robar, Burness & Murray (2010) due to the use of different statistical software and method of calculating maximum likelihood. 3 The LOR, the SE of the LOR and one randomly selected component of the raw data were missing. 2

Multiple imputation can be a powerful tool to estimate missing values, however, it does have some limitations when applied in a meta-regression framework. Using multiple imputation to estimate missing predictors should be done with caution, as we were unable to recover accurate coefficient values for the predictor that was being imputed. This is especially problematic with univariate models. In multivariate models, however, the coefficient values were accurate for predictors that were not imputed. In an AIC framework, the relative model weights and importance values were also more accurate when using multiple imputation to estimate missing predictor values than when using complete-case removal. We recommend the use of multiple imputation to estimate missing predictor values only in a multivariate framework and with the caveat that bias could exist in the coefficients of the imputed

predictor. As such, the use of models should be restricted to interpretation and not projection. Care must also be taken when considering the SE of the coefficient estimates derived from multiple imputation as we found that the estimated SE was less than the SE that we expected due to the variation inherent in the imputation process. When the raw data used to derive both the effect size and weighting parameter were missing, multiple imputation failed to recover coefficient estimates, SE of the coefficient estimates, relative model weights and importance values. This is particularly troubling because many effect size metrics commonly used in meta-regression are derived from the same raw data used to calculate the weighting parameter of that effect size. Multiple imputation might perform poorly in these situations because the meta-regression model itself is more complex than

© 2014 The Authors. Methods in Ecology and Evolution © 2014 British Ecological Society, Methods in Ecology and Evolution, 6, 153–163

Multiple imputation in meta-regression

161

Table 3. Importance values (Σ) for the complete data set of Robar, Burness & Murray (2010), three simulated incomplete data sets [40% of either the SE of log odds ratio (LOR; when transformed into the inverse variance became the weighting variable), latitude (continuous independent predictor) or raw data (used to derive LOR and SE of LOR) were missing under three different scenarios1)], and three corresponding imputed data sets in which multiple imputation was applied to the simulated incomplete data sets. Importance values that have  0
15 change from the complete data set are in bold font Data set missing SE of LOR

Data set missing latitude

Data set missing raw data3

Missing data Scenario1

Predictor

Complete data set2

Incomplete

Imputed

Incomplete

Imputed

Incomplete

Imputed

MCAR

Predation mediated Latitude Load type Study type Host taxon Parasite taxon Predation mediated Latitude Load type Study type Host taxon Parasite taxon Predation mediated Latitude Load type Study type Host taxon Parasite taxon

0
91 0
98 0
21 0
20 0
80 0
25 0
91 0
98 0
21 0
20 0
80 0
25 0
91 0
98 0
21 0
20 0
80 0
25

0
82 0
80 0
15 0
26 0
62 0
18 0
72 0
73 0
11 0
32 0
47 0
10 0
94 0
93 0
17 0
27 0
70 0
31

0
90 0
97 0
25 0
19 0
81 0
21 0
87 0
97 0
29 0
10 0
90 0
12 0
88 0
98 0
23 0
22 0
78 0
21

0
83 0
78 0
27 0
13 0
76 0
00 0
86 0
87 0
39 0
05 0
91 0
00 0
68 0
74 0
21 0
31 0
50 0
00

0
89 0
98 0
21 0
17 0
83 0
22 0
90 0
99 0
20 0
19 0
81 0
25 0
85 0
98 0
27 0
09 0
91 0
11

0
92 0
90 0
12 0
26 0
69 0
34 0
86 0
81 0
13 0
19 0
67 0
16 0
97 0
94 0
19 0
28 0
71 0
31

0
60 0
99 0
22 0
12 0
88 0
09 0
62 0
98 0
27 0
04 0
96 0
03 0
13 0
99 0
05 0
02 0
98 0
02

MAR

MNAR

1

MCAR, missing completely at random; MAR, missing at random; MNAR, missing not at random. The importance values are slightly different from those reported in Robar, Burness & Murray (2010) due to the use of different statistical software and differences in the likelihood function estimation. 3 The LOR, the SE of the LOR and one randomly selected component of the raw data were missing. 2

the imputation model, and this is exacerbated by the imputation of raw data which likely does not share the same relationships with the other data as effect size and the weighting parameter do. In fact, the best predictors might not be among the independent predictors of the meta-regression model. It is imperative that methods be developed to improve the performance of multiple imputation in these complex situations, because complete-case analysis remains a poor alternative. We suggest two future avenues to improve the performance of multiple imputation: First, collect more predictors that are explicitly related to the raw data, which should lead to more accurate imputation, and second, develop an explicit imputation algorithm for each effect size/weighting parameter combination. Of course, the complexity associated with the second option may be prohibitive in many circumstances. We focused our analysis on published studies that, themselves, failed to report all the necessary information for a metaregression. A study that reports that an analysis was conducted and was non-significant, and thus the details were not reported, represents a case in which most or all of the data needed by a meta-analyst would be missing. In these situations, multiple imputation is expected to perform poorly as there is little information available to estimate the missing data in these cases. However, potential solutions might be found by combining multiple imputation and methods used to correct for publication bias, such as selection models (for an overview see Jennions et al. 2013).

Our study comprehensively examined the performance of multiple imputation, using a variety of criteria, to estimate missing data in meta-regression analyses. Previous studies have examined the performance of multiple imputation in traditional meta-analyses only (Thiessen Philbrook, Barrowman & Garg 2007; Idris & Robertson 2009; Idris & Sarudin 2011) and have not considered more complex multivariate meta-regression or in an AIC model selection framework. Furthermore, existing studies have only examined the performance of multiple imputation when estimating the weighting variable and not predictors (Thiessen Philbrook, Barrowman & Garg 2007; Idris & Robertson 2009; Idris & Sarudin 2011). In addition, these studies either did not report how they conducted multiple imputation or conducted multiple imputation using simpler imputation models. These previous studies, while informative in specific situations, were not comprehensive enough to adequately address the performance of multiple imputation in the wide variety of situations that might arise in ecological and evolutionary meta-regression. While we did not examine the performance of multiple imputation on fixed effects or random effects meta-analysis, the performance should be similar if the data sets used for multiple imputation have several covariates in addition to the effect size and weighting parameter (or the component values used to derive these values). We predict that multiple imputation will not perform well when data sets only contain effect sizes and weighting parameters (or the component values used to derive these values) because the

© 2014 The Authors. Methods in Ecology and Evolution © 2014 British Ecological Society, Methods in Ecology and Evolution, 6, 153–163

162 E. H. Ellington et al. performance of Bayesian GLM, on which multiple imputation is based, generally decreases as the number of variables available to fit the relationships decreases. While missing data can be recovered or estimated using various methods, the best way to avoid the pitfalls of missing data is to avoid such problems in the first place. Here, we echo Gurevitch et al. (1992), in that primary studies should always report some measure of variance around any reported effect sizes. We also recommend that authors of primary studies report the basic methodology and study design in such a manner that the statistical analyses can be repeated. This should ensure that sufficient methodological information is available to determine appropriateness of the study for a meta-regression or to assign methodological predictors. In the medical research field, a uniform reporting standard has been adopted (Moher, Schulz & Altman 2001) and the National Institute of Health requires protocols and results of all funded studies to be registered regardless whether the study has been published. Such measures can help ensure that future published studies are useful for synthetic efforts without the challenge of missing data.

Acknowledgements Thoughtful discussions with Catarina Ferreira, Dan Thornton, Jeff Row and other members of the Murray laboratory improved this manuscript. We thank Erin Koen, two anonymous reviewers, and an associate editor for reviewing earlier drafts of this manuscript. This study was partly funded by the Institute for Biodiversity, Ecosystem Science & Sustainability and the Sustainable Development & Strategic Science Division of the Newfoundland & Labrador Department of Environment & Conservation. DLM was supported by a Canada Research Chair in Integrative Wildlife Conservation.

Data accessibility A sample of the code we used in program R to complete our analyses is included in Appendix S5. The Robar and Rees data sets have been deposited in the Dryad repository doi:10.5061/dryad.m2v4m.

References Arnqvist, G. & Wooster, D. (1995) Meta-analysis: synthesizing research findings in ecology and evolution. Trends in Ecology & Evolution, 10, 236–240. Borenstein, M., Hedges, L.V., Higgins, J.P. & Rothstein, H.R. (2011) Introduction to Meta-Analysis. Wiley & Sons, Chichester. Boyle, P., Robertson, C., Vaughan, E.D. & Fitzpatrick, J.M. (2004) A meta-analysis of trials of transurethral needle ablation for treating symptomatic benign prostatic hyperplasia. BJU International, 94, 83–88. Burnham, K.P. & Anderson, D.R. (2002) Model Selection and Multi-Model Inference: A Practical Information-Theoretic Approach. Springer, New York, New York, USA. van Buuren, S. (2012) Flexible Imputation of Missing Data. CRC Press, Boca Raton, Florida, USA. Cadotte, M.W., Mehrkens, L.R. & Menge, D.N. (2012) Gauging the impact of meta-analysis on ecology. Evolutionary Ecology, 26, 1153–1167. Canty, A. & Ripley, B. (2009) boot: Bootstrap R (S-Plus) functions. R Package Version 1.2. Carpenter, S.R., Armbrust, E.V., Arzberger, P.W., Chapin, F.S., Elser, J.J., Hackett, E.J. et al. (2009) Accelerate synthesis in ecology and environmental sciences. BioScience, 59, 699–701. Chan, A.W., Hr
objartsson, A., Haahr, M.T., Gøtzsche, P.C. & Altman, D.G. (2004) Empirical evidence for selective reporting of outcomes in randomized trials. JAMA: Journal of the American Medical Association, 291, 2457–2465. Chaudhary, V.B., Walters, L.L., Bever, J.D., Hoeksema, J.D. & Wilson, G.W. (2010) Advancing synthetic ecology: a database system to facilitate complex

ecological meta-analyses. Bulletin of the Ecological Society of America, 91, 235–243. Cleasby, I.R. & Nakagawa, S. (2012) The influence of male age on within-pair and extra-pair paternity in passerines. Ibis, 154, 318–324. Egger, M., Smith, G.D., Schneider, M. & Minder, C. (1997) Bias in metaanalysis detected by a simple, graphical test. British Medical Journal, 315, 629–634. Fisher, D.O., Blomberg, S.P. & Owens, I.P. (2003) Extrinsic versus intrinsic factors in the decline and extinction of Australian marsupials. Proceedings of the Royal Society of London. Series B: Biological Sciences, 270, 1801–1808. Gurevitch, J. & Hedges, L.V. (1999) Statistical issues in ecological meta-analyses. Ecology, 80, 1142–1149. Gurevitch, J., Morrow, L.L., Wallace, A. & Walsh, J.S. (1992) A meta-analysis of competition in field experiments. American Naturalist, 140, 539–572. Idris, N.R.N. & Robertson, C. (2009) The effects of imputing the missing standard deviations on the standard error of meta-analysis estimates. Communications in Statistics—Simulation and Computation, 38, 513–526. Idris, N.R.N. & Sarudin, N. (2011) On selection of models for continuous meta-analysis data with incomplete variability measures. Far East Journal of Mathematical Sciences (FJMS), 59, 173–187. Jennions, M.D., Lortie, C., Rosenberg, M., Rothstein, H. (2013) Publication and related biases. Handbook of Meta-analysis in Ecology and Evolution (eds J. Koricheva, J. Gurevitch & K. Mengersen), pp. 207–236. Princeton University Press, Princeton, New Jersey, USA. Jones, K.E., Bielby, J., Cardillo, M., Fritz, S.A., O’Dell, J., Orme, C.D.L. et al. (2009) PanTHERIA: a species-level database of life history, ecology, and geography of extant and recently extinct mammals: Ecological Archives E090-184. Ecology, 90, 2648. Kattge, J., Diaz, S., Lavorel, S., Prentice, I.C., Leadley, P., B€ onisch, G., et al. (2011) TRY–a global database of plant traits. Global Change Biology, 17, 2905–2935. Lajeunesse, M.J. (2013) Recovering missing or partial data from studies: a survey of conversions and imputations for meta-analysis. Handbook of Meta-analysis in Ecology and Evolution (eds J. Koricheva, J. Gurevitch & K. Mengersen), pp. 195–206. Princeton University Press, Princeton, New Jersey, USA. Lajeunesse, M.J. & Forbes, M.R. (2003) Variable reporting and quantitative reviews: a comparison of three meta-analytical techniques. Ecology Letters, 6, 448–454. L
opez-L
opez, J.A., Mar
ın-Mart
ınez, F., S
anchez-Meca, J., Noortgate, W. & Viechtbauer, W. (2013) Estimation of the predictive power of the model in mixed-effects meta-regression: a simulation study. British Journal of Mathematical and Statistical Psychology, 67, 30–48. Marshall, A., Altman, D.G. & Holder, R.L. (2010) Comparison of imputation methods for handling missing covariate data when fitting a Cox proportional hazards model: a resampling study. BMC Medical Research Methodology, 10, 112. Moher, D., Schulz, K.F. & Altman, D.G. (2001) The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials. BMC Medical Research Methodology, 1, 2. Nakagawa, S. & Freckleton, R.P. (2008) Missing inaction: the dangers of ignoring missing data. Trends in Ecology & Evolution, 23, 592–596. Nakagawa, S. & Freckleton, R.P. (2011) Model averaging, missing data and multiple imputation: a case study for behavioural ecology. Behavioral Ecology and Sociobiology, 65, 103–116. Nakagawa, S. & Santos, E.S. (2012) Methodological issues and advances in biological meta-analysis. Evolutionary Ecology, 26, 1253–1274. NERC Centre for Population Biology, Imperial College (2010) The Global Population Dynamics Database Version 2. http://www.sw.ic.ac.uk/cpb/cpb/gpdd. html Pickett, S.T., Kolasa, J. & Jones, C.G. (2010) Ecological Understanding: The Nature of Theory and the Theory of Nature. Academic Press, San Diego, California, USA. R Core Team (2013) R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria ISBN 3-900051-07-0, http://www.R-project.org/. Raghunathan, T.E. (2004) What do we do with missing data? Some options for analysis of incomplete data. Annual Review of Public Health, 25, 99–117. Rees, E.E., Pond, B.A., Phillips, J.R. & Murray, D. (2008) Raccoon ecology database: a resource for population dynamics modelling and meta-analysis. Ecological Informatics, 3, 87–96. Riley, R.D., Abrams, K.R., Sutton, A.J., Lambert, P.C., Jones, D.R., Heney, D. & Burchill, S.A. (2003) Reporting of prognostic markers: current problems and development of guidelines for evidence-based practice in the future. British Journal of Cancer, 88, 1191–1198.

© 2014 The Authors. Methods in Ecology and Evolution © 2014 British Ecological Society, Methods in Ecology and Evolution, 6, 153–163

Multiple imputation in meta-regression Robar, N., Burness, G. & Murray, D.L. (2010) Tropics, trophics and taxonomy: the determinants of parasite-associated host mortality. Oikos, 119, 1273–1280. Robertson, C., Idris, N.R.N. & Boyle, P. (2004) Beyond classical meta-analysis: can inadequately reported studies be included? Drug Discovery Today, 9, 924–931. Rubin, D.B. (1976) Inference and missing data. Biometrika, 63, 581–592. Rubin, D.B. (1996) Multiple imputation after 18+ years. Journal of the American Statistical Association, 91, 473–489. Su, Y.S., Yajima, M., Gelman, A.E. & Hill, J. (2011) Multiple imputation with diagnostics (mi) in R: opening windows into the black box. Journal of Statistical Software, 45, 1–31. Thiessen Philbrook, H., Barrowman, N. & Garg, A.X. (2007) Imputing variance estimates do not alter the conclusions of a meta-analysis with continuous outcomes: a case study of changes in renal function after living kidney donation. Journal of Clinical Epidemiology, 60, 228–240. Thompson, S.G. & Higgins, J. (2002) How should meta-regression analyses be undertaken and interpreted? Statistics in Medicine, 21, 1559–1573. Viechtbauer, W. (2010) Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36, 1–48. Wiebe, N., Vandermeer, B., Platt, R.W., Klassen, T.P., Moher, D. & Barrowman, N.J. (2006) A systematic review identifies a lack of standardization in methods for handling missing variance data. Journal of Clinical Epidemiology, 59, 342–353. Received 19 December 2013; accepted 18 November 2014 Handling Editor: Jarrod Hadfield

163

Table S2. Description of 27 scenarios in which we tested the performance of multiple imputation in a meta-regression framework. Fig. S1. Quantiles (95%) around the mean difference between estimates from the complete dataset and estimates from both the incomplete (n = 100) (blue) and imputed (n = 100) (green) datasets across percent of missing data. Fig. S2. Quantiles (95%) around the SE of the coefficients for the incomplete (n = 100) (blue [light gray]) and imputed (n = 100) (green [dark gray]) datasets across percent of missing data. Fig. S3. Quantiles (95%) around the SE of the coefficients from the incomplete (n = 100) (blue [light gray]) and imputed (n = 100) (green [dark gray]) datasets across percent of missing data. Fig. S4. Quantiles (95%) around the SE of coefficients from the incomplete (n = 100, blue [light gray]) and imputed (n = 100, green [dark gray]) datasets across percent of missing data. Fig. S5. Confidence intervals (95%) around the mean differences between R2 values from the complete dataset and both the incomplete (n = 10) (blue) and imputed (n = 10 A & C; n = 100 B) (green) datasets across percent of missing data.

Supporting Information Additional Supporting Information may be found in the online version of this article. Appendix S1. Description of unpublished meta-analytic datasets. Appendix S2. Multiple imputation process in R package mi. Appendix S3. Results of alternative imputation strategies. Appendix S4. Derivation of expected SE due to imputation process. Appendix S5. Sample of R code and associated source function code used for simulation of missing data and testing of performance of multiple imputation for meta-regression analyses.

Fig. S6. Confidence intervals (95%) around the differences between latitude coefficients from the complete dataset and coefficients from both the incomplete (n = 10) (blue) and imputed (n = 10) (green) datasets across the percent of missing data. Fig. S7. Confidence intervals (95%) around the differences between the complete dataset and both the incomplete (n = 10) (blue) and imputed (n = 10) (green) datasets across the percent of missing data. Fig. S8. Confidence intervals (95%) around the differences between predation-mediated coefficients from the complete dataset and both the incomplete (n = 10) (blue) and imputed (n = 10) (green) datasets across the percent of missing data.

Table S1. The top models from Robar, Burness & Murray (2010). We consider these models to represent the complete dataset, to which we compared incomplete and imputed versions of the dataset.

© 2014 The Authors. Methods in Ecology and Evolution © 2014 British Ecological Society, Methods in Ecology and Evolution, 6, 153–163