There is no such thing as a crystal ball. In fact, every forecast will be wrong.
The real value in a forecast is having an accurate understanding of how wrong it will be. In other words, the real value lies in quantifying the uncertainty or risk associated with that forecast and using the understanding of the uncertainty to make more informed business decisions.
When a forecasting model lacks accuracy, it leads to uninformed business decisions. Logical decisions based on an inaccurate model can easily cause more harm than good. That is where backtesting the forecast model can have a huge impact on business outcomes.
Backtesting gives you a deeper understanding of your forecast — both its strengths and its weaknesses. Business leaders can leverage backtesting to gain a better understanding of the uncertainty of their model and leverage that understanding to make better decisions.
What is backtesting?
Backtesting is a simulation method that involves taking a current idea or strategy and applying it to data from a past situation or period of time.
This method is especially helpful for testing the validity of existing forecasting models. Businesses that want to see if a specific forecast model will lead to desirable results in the future can test that model using historical data. Essentially, businesses can use backtesting to determine whether or not certain financial decisions or business strategies should be used moving forwards by testing them on risk-free situations that have already occurred.
If the model provides inaccurate results, no harm has been done, and business knows they should try something different. If the model produces accurate results when used on data from the past, it is likely that it is a valid model that will be able to produce accurate results when used on current and future data.
How backtesting helps businesses
Backtesting allows businesses to build trust in the forecasting model they want to implement before they actually use that model to make decisions that will have real consequences for their business.
For example, having access to several models from which to make buy/sell decisions for different time horizons in the future is very useful. A tool that instantly compares the accuracy of each at the same time can make the difference between winning and losing big.
While running routine backtesting for our beef models this spring, we noticed that they were not performing as well as they had in previous years. Upon further investigation, we identified some changes in the relationships between cattle price and ratios to cut prices since 2019. As a result, our team of data scientists developed a more accurate model which takes the ratio of the graded cutout, in this example CHOICE, and takes the ratio of the cut value to the graded cutout (fig. 1).
When we backtest the new Cutout model vs the CME model, we see that in every time horizon in the future going out 25 weeks, the Mean Absolute Percentage Error (MAPE) for the Cutout model performed much better than the CME model. Note that the CME model here represents the standard ratio to cattle price model that you are likely familiar with.
Figure 1 – A backtesting graph comparing mean absolute percent error (MAPE) of several different models at different time horizons up to 25 weeks into the future.
Backtesting in commodity industries
Commodity industries, and especially beef and pork markets, have experienced increased volatility over the past several years.
Consumer demand patterns have shifted cut prices. Tight cattle herds have driven up beef prices. Droughts and other unpredictable weather events have changed outlooks. And most recently, the courts have created upheaval in pork markets (prop 12). As you can imagine, the list goes on.
But backtesting can be the secret weapon to help teams better understand their markets. By running a multitude of scenarios and applying backtesting functionality, leaders and analysts can gain a deeper understanding of the strengths and the weaknesses in their models.
It’s a powerful dynamic that relies on “human and machine” working in tandem. With backtesting in their arsenal, companies across the commodity spectrum can expect to move faster, with greater confidence, and drive better business outcomes.
