Four Common Mistakes in Machine Learning Projects
As machine learning and data analytics experts, we have considerable experience in developing and implementing machine learning projects with businesses across different industries, including manufacturing, telecoms, financial services and retail. As such, we’re all too familiar with the types of mistakes that are typically made during machine learning projects – and not only by beginners.
It is essential for businesses to learn what not to do in order to focus their efforts on solving real issues and build solutions capable of delivering ROI. Drawing on our own examples, as well as others that we have come across, we are able to highlight the costly mistakes that could have otherwise been avoided.
Here are the top four typical mistakes made by businesses during machine learning projects.
1. Trouble with Experiments
A/B testing to verify the efficacy of a machine learning model is perhaps the only real way to prove its true business value, but too many organizations conduct these tests with more variables than just the model. That’s a fundamental error.
Businesses must approach this method carefully; the only difference between ‘A’ and ‘B’ in A/B testing should be the model producing the results. All the other variables should stay the same.
For example, when experimenting with customer product recommendations via SMS, all messages must be sent on the same day and at the same time. This is the only way to understand their efficacy. You can’t adequately compare results where a variable may have been changed. For example, the effect of recommendations sent on Saturday morning via SMS will differ from those sent on a Friday night.
2. The Importance of Metrics
During any machine learning project, determining metrics is one of the most important factors impacting the success. After all, you won’t get what you don’t ask for. The metric set is the metric that gets optimized, and therefore all other factors will be ignored. Choose the wrong metrics to measure success, and the project is already off track as optimization may have led to a change for the worse.
For businesses, success should be determined by metrics that are business-focused. Many projects have problems with finding a correct technical metric that takes all business aspects into account. Because the metric is what will be optimized, if some business aspects are not incorporated into the setting of metrics, they will be ignored during the process of optimization, and the business is unlikely to reach its preferred goal.
As an example, let us look at such metric as symmetric mean absolute percentage error, or SMAPE, an accuracy measure based on percentage (or relative) errors. When it comes to slow-moving inventory in retail, it’s very efficient for the algorithm to predict zeros for all the slow-moving products if we want to optimize SMAPE. However, the retailer cannot just remove products that are sold only once or twice a week at all from the inventory, as they often influence sales of other products in a store and therefore should be in stock.
In light of this, it is important for businesses to remember that they are in control of the metric: you set the goal, and the algorithm achieves it no matter what. It doesn’t question the destination.
3. Right objective, wrong way
Are you taking the right course of action to reach your objective? Is the question you are asking the model the right one? Let’s say all interested parties have agreed a task is worth solving, and you even have the luxury of knowing how to apply machine learning to the problem. This is exactly the time when you should cross-check if the objectives noted are actually the ones you want to achieve. Even what seems like little shifts in determining the way to a solution, can eventually lead you to situation where machine learning can no better solve a task than the less advanced methods before it.
For example, imagine you operate a service company and occasionally – if not regularly – there are visiting peaks, and not enough employees working to cover them. As a business, you don’t want to ask employees to work all of the time as this isn’t cost-efficient during visiting down time. Instead, you try to forecast the top five days where visiting traffic is at its peak. However, the result of doing so leads to the prognosis that is too noisy because sometimes you have 10 peak days in a month that don’t really differ from each other. The question you might want to get an answer to would really be: “What are the days I should call in some extra help?” and the way how you answer the question should be changed accordingly.
4. Does it all make sense?
The metrics, algorithms, and testing may indicate success, but if implemented in the wrong way, the model may be blamed unfairly. Therefore, you must ensure that all efforts towards enhancing processes through a machine learning project make sense, or else you risk spending time ensuring everything is technically perfect, but to no avail.
Taking product recommendation as an example. You must ensure that the channel (email, SMS, push notification, etc.) delivering those recommendations works. While this seems like a simple suggestion, it’s something that’s often overlooked. If it doesn’t work properly, the improvements you make to the recommendations shared will add no value simply because the channel was fruitless to begin with. For example, if a customer rarely reads the emails sent by a company to begin with, but receives a product recommendation via this channel, it’s the channel at fault and not the recommendation.
Machine learning projects are complex and carry with them typical mistakes that can lead to lots of effort but no outcome. While these mistakes can create setbacks, learning from them can also lead to successful business optimisation and costing saving, in a matter of months opposed to years offered by other technologies. In the case of machine learning, the biggest mistake to make is not learning from those made by others.
About the author: Michael Levin serves as the Chief Data Scientist at Yandex Data Factory. He also teaches Algorithms and Data Structures at Yandex School of Data Analysis.