Model evaluation (including climate models)

A long standing interest of Fildes has been in the evaluation of various forecasting models, see Fildes (1992) and Fildes and Ord (2002). This strand of research remains both important and topical. Organisations always need to evaluate their methods and inappropriate metrics lead to false conclusions and poor decisions.

Davydenko and Fildes (2013) examine different accuracy metrics showing where they disagree, finally arguing for using the Geometric Mean of the Relative Mean Absolute Errors (RelMAE). The use of geometric mean succeeds in delivering an interpretable error measure whilst the ratio of the MAEs turns out to have a better behaved (near normal) distribution compared to other standard measures such as MAPE or MASE.

Model Evaluation is also critical in the important area of climate forecasting and global warming. Fildes working with Kourentzes (2012) has examined the validity of the global circulation models used in climate modelling which are at the heart of the IPCC argument for global warming. While not offering support to the position of climate sceptics the research argues that these global climate simulation models have not been subject to rigorous input – output testing of the forecasts they produce. In fact, comparison with various extrapolative models suggests that the large-scale climate models may be mis-specified. The research concludes by arguing that a more statistical approach to producing climate forecasts a decade or more ahead would be productive both of better models and more accurate forecasts. Limited evidence was found in the data for the importance of CO2 emissions based on a long-lag neural network. This is an important issue where the methods of forecasters and statisticians can usefully be combined with climate scientists.

A full list of publications can be found here.