Ecological Archives A025-146-A2

Takashi Yamamoto, Yutaka Watanuki, Elliott L Hazen, Bungo Nishizawa, Hiroko Sasaki, and Akinori Takahashi. 2015. Statistical integration of tracking and vessel survey data to incorporate life history differences in habitat models. Ecological Applications 25:2396–2408. http://dx.doi.org/10.1890/15-0142.1

Appendix B. Comparisons of four different modeling techniques.

The distribution of streaked shearwaters tracked in 2007 were modeled and compared with four different modeling techniques: generalized linear models (GLM), generalized additive models (GAM), Random Forests (RF), and ensemble models (EM). Ensemble models are calculated as weighted averages of single-model predictions (Araújo and New 2007, Coetzee et al. 2009, Marmion et al. 2009, Jones-Farrand et al. 2011), with weights assigned to each modeling technique based on its discriminatory power as measured by the area under the receiver-operated characteristic curve (AUC). We used a cross-validation technique by randomly splitting our datasets into 'training' and 'test', consisting of 70% and 30% of the data, respectively. Then, we used the training data to develop the models and evaluated the models using the test data. We assumed a binomial distribution with the logit-link function as density values range from 0 to 1 (Dobson 2008). Receiver operating characteristic (ROC) curves that assess the rate of false positives against true positives were used to calculate AUC to evaluate the performance of the models. AUC ranges from 0 to 1; values of 0.5 indicate a performance no better than random, values between 0.7–0.9 indicate reasonable model performance, and values greater than 0.9 indicate strong predictive accuracy (Pearce and Ferrier 2000). In addition to AUC, which assesses the predictive accuracy of distribution models, calibration measures how well the frequency of observations in the test data agrees with predicted probabilities of occurrence (Piñeiro et al. 2008, Potts and Elith 2006, Oppel et al. 2012). To measure calibration, we calculated the Pearson correlation coefficient in addition to the slope and intercept of a major axis linear regression of observed versus predicted values to evaluate the bias and consistency of model predictions. The slope and the intercept of this regression indicate the calibration and the bias of the model, respectively (Phillips and Elith 2010). Models were ranked based on multiple selection criterion, including Akaike's Information Criterion (AIC), AIC weight, AUC, correlation, calibration, and bias (Gollcher et al. 2012, Lawson et al. 2014, Warren et al. 2014), and the best performing model across all of these metrics was used to develop our predictions (Table B1).

RFs had the highest AUC and correlation of all predictions followed by EMs, but both showed larger calibration difference and bias than GAMs (Table B1). GLMs had the lowest AUC and correlation. The machine-learning methods, including RFs, show very good performance on the training data, but when used on spatially independent test data these methods likely suffer proportionally more from over fitting than parametric models (i.e. GLMs and GAMs) (Hastie et al. 2001), resulting in larger biases. In the ensemble prediction, a risk with combining multiple methods is that the stronger models may be degraded by dilution with poorly performing models (in our case GLM) (Peterson et al. 2011). Furthermore, the responses of streaked shearwater to environment variables are likely non-linear (i.e., they appear to exhibit a preference for particular ranges of SST; Yamamoto et al. 2011), meaning that GLMs would not be a good tool for use in this study. Although GAMs showed relatively lower AUC than RFs and EMs, these predictions were less biased and the performance of models was still reasonable (all AUC >0.8), including some with very good accuracy (>0.9) (Table B1).

Table B1. Model evaluation and calibration statistics of four modeling techniques. AUC = area under the receiver-operated characteristic curve; COR = point bi-serial correlation coefficient between observed and predicted values; calibration = slope of regression of observed vs. predicted values; bias = intercept of regression of observed vs. predicted values. Bolded values represent the best-performing model for each criterion.

Literature cited

Araújo, M. B., and M. New. 2007. Ensemble forecasting of species distributions. Trends in Ecology and Evolution 22:42–47.

Coetzee, B. W. T., M. P. Robertson, B. F. N. Erasmus, B. J. van Rensburg, and W. Thuiller. 2009. Ensemble models predict Important Bird Areas in southern Africa will become less effective for conserving endemic birds under climate change. Global Ecology and Biogeography 18:701–710.

Dobson, A. J. 2008. An Introduction to Generalized Linear Models Second edition. Kyoritsu Shuppan Co., Ltd, Tokyo, Japan.

Gollcher, D., A. Ford, L. Cayuela, and A. Newton. 2012. Pseudo-absences, pseudo-models and pseudo-niches: pitfalls of model selection based on the area under the curve. International Journal of Geographical Information Science 26:2049–2063.

Hastie, T., R. Tibshirani, and J. H. Friedman. 2001. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, New York, NJ.

Jones-Farrand, D. T., T. M. Fearer, W. E. Thogmartin, F. R. T. Iii, M. D. Nelson, and J. M. Tirpak. 2011. Comparison of statistical and theoretical habitat models for conservation planning: the benefit of ensemble prediction. Ecological Applications 21:2269–2282.

Lawson, C. R., J. A. Hodgson, R. J. Wilson, and S. A. Richards. 2014. Prevalence, thresholds and the performance of presence-absence models. Methods in Ecology and Evolution 5:54–64.

Marmion, M., M. Parviainen, M. Luoto, R. K. Heikkinen, and W. Thuiller. 2009. Evaluation of consensus methods in predictive species distribution modelling. Diversity and Distributions 15:59–69.

Oppel, S., A. Meirinho, I. Ramírez,, B. Gardner, A. F. O’Connell, P. I. Miller, and M. Louzao. 2012. Comparison of five modelling techniques to predict the spatial distribution and abundance of seabirds. Biological Conservation 156:94–104.

Pearce, J., and S. Ferrier. 2000. Evaluating the predictive performance of habitat models developed using logistic regression. Ecological Modelling 133:225–245.

Peterson, A. T., J. Soberón, R. G. Pearson, R. P. Anderson, E. Martínez-Meyer, M. Nakamura, and M. B. Araújo. 2011. Ecological niches and geographic distributions. Princeton University Press, Princeton, USA.

Phillips, S. J., and J. Elith. 2010. POC plots: calibrating species distribution models with presence-only data. Ecology 91:2476–2484.

Piñeiro, G., S. Perelman, J. P. Guerschman, and J. M. Paruelo. 2008. How to evaluate models: observed vs. predicted or predicted vs. observed? Ecological Modelling 216:316–322.

Potts, J. M., J. Elith. 2006. Comparing species abundance models. Ecological Modelling 199:153–163.

Warren, D. L., A. N. Wright, S. N. Seifert, and H. B. Shaffer. 2014. Incorporating model complexity and spatial sampling bias into ecological niche models of climate change risks faced by 90 California vertebrate species of concern. Diversity and Distributions 20:334–343.

Yamamoto, T., A. Takahashi, N. Oka, T. Iida, N. Katsumata, K. Sato, and P. N. Trathan. 2011. Foraging areas of streaked shearwaters in relation to seasonal changes in the marine environment of the Northwestern Pacific: inter-colony and sex-related differences. Marine Ecology Progress Series 424:191–204.

[Back to A025-146]