Short answer: No.
For the past month there has been a healthy discussion about forecast accuracy metrics on the Institute of Business Forecasting & Planning discussion group on LinkedIn. This was all prompted by the question, “How does one establish an acceptable target to measure against?” posed by Catherine Waudby, Volume Planning Specialist at Volkswagen Group South Africa.
I have long argued against arbitrary forecasting performance objectives, suggesting instead that the goal should be “Do no worse than a naïve model.” We don’t know in advance how well a naïve model will forecast, so we can’t in advance set a numerical performance goal. But we can track performance over time, and determine whether we are meeting this (woefully pathetic sounding) objective.
As the online discussion wove through 20 separate comments, there was an exchange between Sam Smale and Richard Herrin that I’d like to address. Sam pointed out, quite correctly, that the goal of being “better than a naïve model” could make life too easy. It is true that just about any decent statistical forecast model should forecast better than a random walk (the standard naïve forecasting model). However, due to the biases and politics and personal agendas that plague most organizational forecasting processes, I believe that beating a random walk is still a legitimate “first test.”
(Note: If your forecasting process is doing worse than a random walk model, it is probably pretty bad!)
Richard then turns the conversation toward slightly sophisticated models, perhaps including trend or seasonality. Is it appropriate to consider these as “naïve” models against which to do comparisons?
There is a very important role for these “slightly sophisticated” models that Richard brings up. They fall between the random walk and the more sophisticated models we typically use for business forecasting. Let’s call these “default” models.
A default model is something simple to compute, that you could actually use to run your business. This is the important distinction. A default model is simple to compute (like a random walk is), but you would never use a random walk to run your business because of the instability of future forecasts.
Recall that with a random walk, whatever is the most recent observation becomes your forecast for all future periods. If you sold 100 last month, your forecast for all future months is 100. If you sell 500 this month, the forecast for all future months is changed to 500. If you sell 10 next month, the forecast for all future months changes to 10. This is problematic! You would not want to whipsaw your supply chain with such radical changes to the forecast in each future period.
To conclude, even if your forecasting process does worse than a random walk (the “first test” of process performance), you would never want to start using the random walk as your forecast. A more useful “second test” would be to compare your performance against a slightly sophisticated “default” forecast (e.g. moving average, single exponential smoothing, etc.). It would still be a reasonable (if slightly more challenging) goal to beat the default forecast. And if your process were failing to do so, you could simply ignore the process and start using the default.