Focus Your AI on Business Value, Not Predictive Accuracy, Aible Says
If you’re building your machine learning models to optimize accuracy, then you’re probably doing it wrong. That’s the message coming from Aible, an ML company that emerged from stealth today with new software designed to optimize ML decision-making around economic value, not accuracy.
Having an accurate algorithm is good. It obviously beats the alternative. But companies that focus on accuracy above all else are likely leaving money on the table and not taking full advantage of AI, says Arijit Sengupta, founder and CEO of Aible.
“Accuracy doesn’t mean impact,” Sengupta tells Datanami. “I can make more money with less accurate models, and that is the issue. In the real world, accuracy does not tie to impact.”
The paradox of AI accuracy was demonstrated by Aible co-founder Jonathan Wray, who was a customer of Sengupta’s previous analytics startup, BeyondCore, which was acquired by SalesForce. According to Wray, the ML model used by salespeople to identify prospects was very accurate, but that accuracy didn’t correlate with better business results.
“One of the pieces of feedback I got from my team was the model is always right,” Wray says. “It’s coming back and giving me all these deals. Every deal it gives me, I win.”
But there was a big problem with the ML algorithm: It only recommended deals that it had a very high confidence in, which would likely have been identified by the salesperson anyway.
“The salesperson doesn’t care about accuracy,” he says. “The AI being accurate and only giving them five deals to go after doesn’t help them meet their quota. They still have to go and beat the bushes to do it manually for the rest.”
In fact, a less accurate model that would have provided 60 to 70 targets for salespeople to chase would have been more valuable, he says.
That is what Aible is all about: generating models that take other business factors into account, even if the predictions ostensibly are “less accurate.”
The software sits atop a customers’ existing machine learning environment, such as AWS SageMaker, and acts as a higher-level AI generator. The user inputs additional variables and constraints that correspond with their business environment, and Aible generates Python code that she can put into production.
According to Sengupta, Aible flips the script on machine learning’s status quo by prioritizing variables that have a bigger impact on the business than accuracy alone, which he says has been prioritized because of a lack of communication between the business users and the data scientists.
“The communication gap happens because we’re trying to teach humans to speak AI,” Sengupta says. “We’re trying to teach business users how to explain their problems in language that a data science would translate into something an AI understand.
“What we did is flip the equation and asked the business questions up front,” he continues. “We said, can the AI ask the business user some questions about business reality, and then the AI can create another AI that conforms to those business realities.”
Instead of just focusing on the accuracy of the predictions generated by an ML algorithm, Aible’s offering takes other factors into account. These are the three things that Sengupta says are “fundamentally…broken about AI:”
- Accounting for the cost of a bad prediction as well as the benefit of getting it right;
- Accounting for capacity constraints in the organization;
- Accounting for the state of organization’s decision-making before implementing AI.
Sengupta, who estimates he has been involved in 1,000 AI projects to date, elaborates:
“The first problem is, the benefit of getting the prediction right is never the same as the cost of getting a prediction wrong,” he says. “If I tell you a customer will buy, and the customer doesn’t buy, maybe it costs you $100 in invested phone calls. But if I tell you a customer won’t buy, when they would have bought, it could cost you a $25,000-deal.”
The second problem is AI doesn’t ask about capacity constraints. “Every business user will tell you that they have capacity constraints,” Sengupta says. “It might be how much marketing budget do I have? How many salespeople do I have? How many nurses or beds in a hospital?”
The third problem is that AI typically isn’t deployed into greenfield environments, but overlaid atop existing businesses that are based on elaborate rules and human knowledge that have been built up over years.
“Essentially, experts go in and say, hey this AI is so accurate. See how much better it is than a coin toss? It’s a good AI,” Sengupta says. “The problem is, the business user was not there flipping a coin trying to decide what they were going to do. The business user is deploying 20 years of human knowledge and it’s taking much better than coin-toss decisions. So unless you calibrate yourself to what is the current process, you can often deploy models that are less effective than current processes, even though they’re very accurate.”
Sengupta says Aible recently proved its meddle in a contest at UC Berkeley. Aible invited a bunch of high school kids and history majors to compete against data scientist majors enrolled at the university. The high school students and history majors were trained on Aible for one hour, and then asked to compete against the data science majors.
The high school students and history majors outperformed the data science students after a two-hour competition. The data science students were given five days to optimize their models, but only a few of them built models that outperformed the Aible models generated by the high schoolers and history majors, and their average scores were lower too.
In addition to announcing the availability of its software, Aible announced an integration with Tableau. Aible is currently selling access to its software for the price of $1,000 per user per year.