Ecological Archives A025146A2
Takashi Yamamoto, Yutaka Watanuki, Elliott L Hazen, Bungo Nishizawa, Hiroko Sasaki, and Akinori Takahashi. 2015. Statistical integration of tracking and vessel survey data to incorporate life history differences in habitat models. Ecological Applications 25:2396–2408. http://dx.doi.org/10.1890/150142.1
Appendix B. Comparisons of four different modeling techniques.
The distribution of streaked shearwaters tracked in 2007 were modeled and compared with four different modeling techniques: generalized linear models (GLM), generalized additive models (GAM), Random Forests (RF), and ensemble models (EM). Ensemble models are calculated as weighted averages of singlemodel predictions (Araújo and New 2007, Coetzee et al. 2009, Marmion et al. 2009, JonesFarrand et al. 2011), with weights assigned to each modeling technique based on its discriminatory power as measured by the area under the receiveroperated characteristic curve (AUC). We used a crossvalidation technique by randomly splitting our datasets into 'training' and 'test', consisting of 70% and 30% of the data, respectively. Then, we used the training data to develop the models and evaluated the models using the test data. We assumed a binomial distribution with the logitlink function as density values range from 0 to 1 (Dobson 2008). Receiver operating characteristic (ROC) curves that assess the rate of false positives against true positives were used to calculate AUC to evaluate the performance of the models. AUC ranges from 0 to 1; values of 0.5 indicate a performance no better than random, values between 0.7–0.9 indicate reasonable model performance, and values greater than 0.9 indicate strong predictive accuracy (Pearce and Ferrier 2000). In addition to AUC, which assesses the predictive accuracy of distribution models, calibration measures how well the frequency of observations in the test data agrees with predicted probabilities of occurrence (Piñeiro et al. 2008, Potts and Elith 2006, Oppel et al. 2012). To measure calibration, we calculated the Pearson correlation coefficient in addition to the slope and intercept of a major axis linear regression of observed versus predicted values to evaluate the bias and consistency of model predictions. The slope and the intercept of this regression indicate the calibration and the bias of the model, respectively (Phillips and Elith 2010). Models were ranked based on multiple selection criterion, including Akaike's Information Criterion (AIC), AIC weight, AUC, correlation, calibration, and bias (Gollcher et al. 2012, Lawson et al. 2014, Warren et al. 2014), and the best performing model across all of these metrics was used to develop our predictions (Table B1).
RFs had the highest AUC and correlation of all predictions followed by EMs, but both showed larger calibration difference and bias than GAMs (Table B1). GLMs had the lowest AUC and correlation. The machinelearning methods, including RFs, show very good performance on the training data, but when used on spatially independent test data these methods likely suffer proportionally more from over fitting than parametric models (i.e. GLMs and GAMs) (Hastie et al. 2001), resulting in larger biases. In the ensemble prediction, a risk with combining multiple methods is that the stronger models may be degraded by dilution with poorly performing models (in our case GLM) (Peterson et al. 2011). Furthermore, the responses of streaked shearwater to environment variables are likely nonlinear (i.e., they appear to exhibit a preference for particular ranges of SST; Yamamoto et al. 2011), meaning that GLMs would not be a good tool for use in this study. Although GAMs showed relatively lower AUC than RFs and EMs, these predictions were less biased and the performance of models was still reasonable (all AUC >0.8), including some with very good accuracy (>0.9) (Table B1).
Table B1. Model evaluation and calibration statistics of four modeling techniques. AUC = area under the receiveroperated characteristic curve; COR = point biserial correlation coefficient between observed and predicted values; calibration = slope of regression of observed vs. predicted values; bias = intercept of regression of observed vs. predicted values. Bolded values represent the bestperforming model for each criterion.
State 
Method 
AUC 
COR 
Calibration 
Bias 
Sangan Island 
GLM 
0.90 
0.21 
0.75336 
0.02093 

GAM 
0.96 
0.83 
1.00360 
0.00072 

RF 
0.99 
0.99 
1.02509 
0.00225 

EM 
0.96 
0.89 
1.24682 
0.02360 
Sangan male 
GLM 
0.69 
0.09 
0.76304 
0.03381 

GAM 
0.87 
0.70 
1.02229 
0.00159 

RF 
0.98 
0.98 
1.04591 
0.00531 

EM 
0.92 
0.91 
1.30309 
0.04256 
Sangan female 
GLM 
0.83 
0.20 
0.84183 
0.02012 

GAM 
0.95 
0.81 
1.01100 
0.00189 

RF 
0.99 
0.99 
1.03352 
0.00396 

EM 
0.95 
0.90 
1.26663 
0.02970 
Mikura Island 
GLM 
0.73 
0.20 
0.91643 
0.00518 

GAM 
0.82 
0.48 
0.98046 
0.00079 

RF 
0.98 
0.97 
1.06250 
0.00804 

EM 
0.91 
0.85 
1.35647 
0.04591 
Mikura male 
GLM 
0.72 
0.23 
0.98859 
0.00656 

GAM 
0.83 
0.53 
1.02062 
0.00037 

RF 
0.98 
0.97 
1.07000 
0.00805 

EM 
0.93 
0.86 
1.34494 
0.03898 
Mikura female 
GLM 
0.70 
0.14 
1.04848 
0.00763 

GAM 
0.84 
0.43 
0.99870 
0.00013 

RF 
0.98 
0.97 
1.06189 
0.01153 

EM 
0.96 
0.87 
1.43874 
0.08196 
Nonbreeder 
GLM 
0.83 
0.15 
0.70605 
0.02513 

GAM 
0.92 
0.78 
1.01550 
0.00197 

RF 
0.99 
0.99 
1.03073 
0.00265 

EM 
0.94 
0.89 
1.28249 
0.02723 
Literature cited
Araújo, M. B., and M. New. 2007. Ensemble forecasting of species distributions. Trends in Ecology and Evolution 22:42–47.
Coetzee, B. W. T., M. P. Robertson, B. F. N. Erasmus, B. J. van Rensburg, and W. Thuiller. 2009. Ensemble models predict Important Bird Areas in southern Africa will become less effective for conserving endemic birds under climate change. Global Ecology and Biogeography 18:701–710.
Dobson, A. J. 2008. An Introduction to Generalized Linear Models Second edition. Kyoritsu Shuppan Co., Ltd, Tokyo, Japan.
Gollcher, D., A. Ford, L. Cayuela, and A. Newton. 2012. Pseudoabsences, pseudomodels and pseudoniches: pitfalls of model selection based on the area under the curve. International Journal of Geographical Information Science 26:2049–2063.
Hastie, T., R. Tibshirani, and J. H. Friedman. 2001. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, New York, NJ.
JonesFarrand, D. T., T. M. Fearer, W. E. Thogmartin, F. R. T. Iii, M. D. Nelson, and J. M. Tirpak. 2011. Comparison of statistical and theoretical habitat models for conservation planning: the benefit of ensemble prediction. Ecological Applications 21:2269–2282.
Lawson, C. R., J. A. Hodgson, R. J. Wilson, and S. A. Richards. 2014. Prevalence, thresholds and the performance of presenceabsence models. Methods in Ecology and Evolution 5:54–64.
Marmion, M., M. Parviainen, M. Luoto, R. K. Heikkinen, and W. Thuiller. 2009. Evaluation of consensus methods in predictive species distribution modelling. Diversity and Distributions 15:59–69.
Oppel, S., A. Meirinho, I. Ramírez,, B. Gardner, A. F. O’Connell, P. I. Miller, and M. Louzao. 2012. Comparison of five modelling techniques to predict the spatial distribution and abundance of seabirds. Biological Conservation 156:94–104.
Pearce, J., and S. Ferrier. 2000. Evaluating the predictive performance of habitat models developed using logistic regression. Ecological Modelling 133:225–245.
Peterson, A. T., J. Soberón, R. G. Pearson, R. P. Anderson, E. MartínezMeyer, M. Nakamura, and M. B. Araújo. 2011. Ecological niches and geographic distributions. Princeton University Press, Princeton, USA.
Phillips, S. J., and J. Elith. 2010. POC plots: calibrating species distribution models with presenceonly data. Ecology 91:2476–2484.
Piñeiro, G., S. Perelman, J. P. Guerschman, and J. M. Paruelo. 2008. How to evaluate models: observed vs. predicted or predicted vs. observed? Ecological Modelling 216:316–322.
Potts, J. M., J. Elith. 2006. Comparing species abundance models. Ecological Modelling 199:153–163.
Warren, D. L., A. N. Wright, S. N. Seifert, and H. B. Shaffer. 2014. Incorporating model complexity and spatial sampling bias into ecological niche models of climate change risks faced by 90 California vertebrate species of concern. Diversity and Distributions 20:334–343.
Yamamoto, T., A. Takahashi, N. Oka, T. Iida, N. Katsumata, K. Sato, and P. N. Trathan. 2011. Foraging areas of streaked shearwaters in relation to seasonal changes in the marine environment of the Northwestern Pacific: intercolony and sexrelated differences. Marine Ecology Progress Series 424:191–204.