Can Retailers Trust Their Machine Learning Models?

(Vintage Tone/Shutterstock)

As we inch closer to Black Friday and the start of the holiday buying extravaganza, retailers are putting the final touches on the demand forecasts they’re using to predict the mix of goods they’ll carry this winter. There are lot of variables to juggle, including COVID, the economy, and the weather. It seems like a perfect use case for the increasingly sophisticated machine learning models that are in vogue in the industry. But can they trust their predictions?

Over the past decade, retailers and other companies in the consumer goods supply chain have started upgrading their demand forecasting systems in hopes of gaining ground in this super competitive industry.

Forward-looking retailers, in particular, are replacing the largely deterministic approaches that were favored in the past–which used simple linear regression models based on historical data with relatively static assumptions about the state of the world–with probabilistic approaches that bring more data into the equation and rely on more sophisticated machine learning algorithms, like neural nets and XGBoost, to generate more detailed forecast ranges.

The new probabilistic approaches hold the potential to provide more accurate forecasts for demand planning than the older deterministic approaches, according to supply chain consultant Stefan de Kok.

Deterministic models are being replaced by probabilstic models in retail demand forecasting (Sergey-Nivens/Shutterstock)

“The root of the value of the probabilistic approach is that it can properly distinguish between error and natural variability, and between signal and noise, which is impossible in the deterministic perspective,” de Kok writes in a 2021 story on the topic on LinkedIn. “Probabilistic approaches provide rich information to identify risks and opportunities at all levels of detail, allowing informed business decisions to be made. They also allow perfect delineation of the things you can control and improve versus the things you cannot.”

The potential for higher accuracy and a more nuanced forecast has convinced retailers to at least add probabilistic forecasts to their kit. It’s worth nothing, though, that deterministic models still have the advantage of generally greater precision, according to de Kok, since they are generally expressed as exact numbers.

While the probabilistic approach holds advantages over the deterministic approach, it’s also proving to be a bit more difficult to keep running well in the real world. That’s due to a combination of reasons, including the use of more data and the possibility for errors in the data; the black box nature of the machine learning models; and the general nature of making assumptions about the future.

While the probabilistic approach has advantages, it’s critical for retailers to frequently check the forecasts made by machine learning models to make sure they’re not going off the rails, says Liran Hason, a machine learning expert and the founder and CEO of Aporia, a provider of observability tools for machine learning models.

“There’s a lot of potential positive outcomes by using machine learning models,” says Liran Hason, a machine learning expert and the founder and CEO of Aporia. “But it’s very important to look at them very closely, I’d say on a weekly basis, especially for large retailers, when they’re dealing with large [number] of shops across different counties.”

Aporia works with clients in a number of industries, including retail, automotive, and financial services, so its machine learning observability tools aren’t made specifically for retailers, Hason says. But among the retailers that Aporia has worked with, Hason has seen a greater adoption of deep learning approaches as well as decision trees and XGBoost.

“They’re able to store much more data,” he says. “So in a sense the models we are getting are much more accurate. But part of this accuracy means that they might also be overfitted for some situations.”

Data drift can happen when underlying assumptions about the world have changed. This can happen in an instant in the retail business, such as when a hot new product enters the market and disrupts the pre-existing buying behavior. When that happens, then the accuracy of the model will suffer.

Aporia’s software provides data scientists and machine learning engineers with statistics-based insights into how their models are behaving

“While they might be performing very well during the research phase, or for a certain time period, they won’t be accurate forever,” Hason says. “And then identifying what is the right time to either retrain the model when the data is misrepresented by the model, or maybe you should just fall back to another mechanism. These kinds of questions are becoming very critical for the business.”

Companies use Aporia to alert them when their machine learning models are behaving in an abnormal fashion. The software works by continually testing the model at hand with different permutations of inputs, and analyzing the response.

“Our product is an observability platform,” Hason says. “So we allow users to visualize and see what decisions are begin made by these machine learning [models] for their business, how well they are performing and also for different populations. So the model might be performing well for a specific state, but [not so well] for another state.”

In addition to detecting data drift, it can also help to explain how a black box model actually works. That’s an important factor for today’s complex neural network-based approaches, which use a multitude of hidden layers to boost the accuracy of the predictions.

“The fact that machine learning models are black box also creates the challenge of understanding, well, can I trust this prediction or not,” Hason says. “So part of the offering we’re providing is explain ability of these decisions.”

Retailers don’t need a magical box to tell them to prepare for a turbulent holiday shopping season in 2022. With inflation flirting near 40-year-highs a recession looming, the mood among consumers is not particularly cheery. When you add COVID surges and supply chain disruptions to the mix, you have a potentially volatile situation. Some retailers have already slashed prices due to overstock conditions. The machine learners have their work cut out for them this fall.

Related Items:

Top 10 Ways AI Drives Price Optimization in Retail

Staying On Top of ML Model and Data Drift

Keeping Your Models on the Straight and Narrow