Selecting the best model for improving forecasting performance
27 August 2013
27 August 2013
A major issue in forecasting is the selection of an appropriate model. Practitioners and forecasters regularly face this problem, having to choose a priori the best model from a set of alternatives. This becomes even more interesting, given that if the selection was to be done perfectly then the gains would be substantial (25-30% improvement in forecasting performance).
Moreover, given that in many modern organisations some thousands of different products are usually stocked and forecasted perhaps daily, efficient model selection requires automatic solutions.
Researchers at the Lancaster Centre for Forecasting are continuously exploring the problem of model selection for improving the forecasting performance and designing efficient automatic algorithms.
Recently, two different approaches for model selection were contrasted. The first approach examines the application of the same model across a number of series (aggregate selection). This approach has the benefit of simplicity. The second approach refers to the individual (per series) selection of the best model. Even if this implies increased complexity in the forecasting process, it is intuitively more appealing. Selection of appropriate models focused primarily on the naïve principle (what has forecast the most accurately, will forecast the most accurately on the out-of-sample data).
The study provides useful insights for the practitioner with regards to when to use each approach, with accuracy improvements being as high as 11.4% and 7.7% for unpredictable and unstable series respectively, compared to the performance of Damped Exponential Smoothing. Evidence suggests that the forecaster should include around five unrelated methods in the set to be considered. A case study on a real company data set confirmed the empirical results.
A separate stream of research explored filters (ex ante pre-modelling) and wrappers (ex post evaluation of all possible models) for model selection. Both approaches demonstrated significantly better performance compared to aggregate selection, with wrappers being more efficient. Most importantly, an add-on to SAP APO-DP has been developed, complete with superior graphical analysis and informative messages. This automatic solution consistently improves forecasting performance across various products categories, while significantly reducing time spent. This add-on allows for customisation to the organisation’s data.
If you would like to know more about the research of LCF in the area of model selection please contact Professor Robert Fildes, Dr Sven F Crone or Dr Fotios Petropoulos. The Centre is always keen to embark on new collaborations with organisations and industrial partners.