In many situations, even small improvements in forecasting accuracy can have a major influence on a company's bottom line, impacting business areas ranging from cost reduction to providing better customer service. However, technical knowledge of forecasting required for accurate planning is often lacking, resulting in material losses and significant opportunity costs. INSEAD Senior Research Fellow and Lecturer Michèle Hibon, and Assistant Professor of Information Systems Theodoros Evgeniou, utilize macroeconomic, industry and company time-series data from the M-3Competition to compare performance between combined and individual forecasting methods.
Stressing simplicity of modeling, practical benefit and minimizing risk of forecast inaccuracy, this paper aims to increase understanding of a powerful tool for planning and prediction of future events.
The method of combining forecasts was first introduced in 1969 as a potential improvement over using any one individual method, from the range of alternatives. Over the years, much research has followed, comparing methods to determine which is most effective.
In 1979, Makridakis and Hibon led an ambitious study to compare a large amount of major time series methods across multiple series. While this study offered some groundbreaking conclusions, a more important legacy may be its role as a catalyst for developing a series of competitions to serve as a forum for structuring future research.
In 1982, the M-Competition was launched (see Makridakis et al. 1982 for further information) to encourage a common ground for independent empirical research and to advance collective understanding of factors exerting influence upon forecasting accuracy. A key benefit of this competition was that it enabled conclusions from one study to be replicated using the same data, providing greater validity of results.
This proved to be a milestone in forecasting, with the paper responsible for establishing this event, <em>M-Competition: The Accuracy of Extrapolation (Time Series Methods): Results of a Forecasting Competition</em> (Makridakis, Hibon et al 1982), being recognized as so important, it was awarded the top prize as 'Favourite Paper Over the Last 25 Years' by the International Symposium on Forecasting. This success led to additional forums including the M-2 and M-3 competitions, in 1993 and 2000 respectively.
Major conclusions included the fact that statistically complex methods did not necessarily provide more accurate forecasts than simple methods, and that, on average, combinations of methods outperformed use of individual methods. However, since then other studies, notably Larrick and Soll in 2002, have challenged these conclusions, suggesting there are some conditions where it is better not to combine forecasts of experts.
To respond to this, data from the M-3 Competition was used to compare individual and combined approaches, to determine whether one method had an advantage over the other, on average, as well as whether practical considerations of risk could be significant in this study.
The findings revealed that best performance, for both individual and combinations of methods, was similar. This result also held for cases where the same methods were used for each time series studied, as well as when different methods were used. A clear advantage, however, was found in the lower range, as the worst performance of the individual methods was far more inaccurate in comparison to the worst performance of combined methods.
The most important finding may regard the practical applications of this research. While some conditions have been discovered to favor individual forecasting methods, this also introduces the very real problem of how to choose which method to use. This selection factor and the considerable difference observed in downside volatility, combine to demonstrate the presence of much greater risk when using individual methods.
Future study aimed at improving forecasting accuracy will likely focus on selection of individual methods, building on foundations from the M-3 Competition to assess factors including spread, variance and bracketing rate. While this may ultimately result in further gains in accuracy, combining methods seems to provide the best practical solution available now. These results should provide benefits for many lines of business including economic forecasting, supply chain management, drawing insight from consumer behavior and even evaluating trends in capital market prices.
International Journal of Forecasting, January-March 2005