Charting a Course Out of the Big Data Doldrums
It’s no secret, but it bears repeating: big data is hard. According to Gartner, 60% of advanced analytics projects will fail before reaching production. The technology is getting better and better, so what’s causing the bad returns?
That’s something that caught the attention of Andrew Brust, a longtime IT media pundit and senior director of market strategy for big data software firm Datameer. As Brust explains, many companies have hit a plateau with their big data projects. Maybe they’ve installed Hadoop, loaded some data into it, and wonder what to do next.
But of course, it’s not that easy.
“Merely getting Hadoop in place… doesn’t by itself put you in a mode where you’re productively doing all kinds of analytics and getting insights and hitting the ROI that certainly has motivated a lot of people,” Brust says. “If you put technology into the customers’ hand and have them set up the Hadoop cluster, there’s no magic button you can press to get ROI.”
We’ve heard this refrain before, including from Bill Schmarzo, the EMC evangelist/writer/professor isn’t shy about expressing amazement at the overinflated expectations that people had with Hadoop to Datanami exactly one year ago. “You start with your technology, bring in some Hadoop, throw some data in there and you kind of hope magic stuff happens,” the “Dean of Big Data” told Datanami one year ago. “It’s really a process fraught with all kind of misdirection.”
Claudia Imhoff, another respected analytics expert and CEO of Intelligent Solutions, says something similar. “Part of the problem with big data is that we have become so enamored with the technology, we’ve forgotten what business problems we’re trying to solve with it,” she was quoted as saying in a 2015 analytics ROI study sponsored by Teradata.
But Hadoop, which is the poster boy for a new generation of big data technology, isn’t the main issue here. (Maybe it’s actually becoming the whipping boy.) According to Brust, who has been working in enterprise business intelligence since the turn of the century, the ROI problem is only tangentially related to the technology.
“If you dial back 15 years in the BI world, people were grappling with the same issue. Project failure rate was high,” he says. “But the issue is still the same, which is the technology by itself doesn’t really get you anywhere.”
While not everybody is listing aimlessly in the big data doldrums, there are enough of them to constitute a trend. “It’s clear that a lot of people are stuck,” Brust says.
Patterns of Adoption
To help customers get their big data projects back on track, Brust and his Datameer colleagues today unveiled a series of tools designed to help any big data shop – not just Datameer’s own customers – to find long-term success.
Datameer’s announcement has three parts including a Use Case Browser that showcases hundreds of big data successes of Datameer customers and non-customers alike; a half-day Use Case Discovery Workshop where would-be big data experts can explore potential use cases in greater depth; and new courses in Datameer University (the company’s online training initiative), where users can dive deeper into the technology, including Datameer’s Hadoop- and Spark-based products.
The main thrust of Datameer’s initiatives is emphasizing the fact that repeatable use cases exist that big data newbies can emulate, if not directly copy and paste into their enterprise.
“The crux of it is customers absolutely need to have at least one and hopefully a handful of use cases in hand before they start their data journey so they’ve got plenty of stuff to execute on,” Brust says. “The ROI on these things has already been worked out and vetted in advanced, as opposed to throwing them into a situation where we give them the capability and then they have to start looking for a problem to apply them to.”
While many in the big data industry espouse a “data first” approach–where every last piece of data is first collected, algorithms detect patterns without human assistance, and then actual business use cases are layered atop after the fact–that approach seems to be falling by the wayside as big data failures stack up.
Datameer CEO Stefan Groschupf says it’s about creating a path to success. “Companies are at a crossroads of having been convinced to purchase expensive big data hardware and software, with the promise of significant changes for their business, and having no idea what the next step is to actually get results,” he says.
Firewall from Failure
To guard against spectacular failures in big data, good advice typically revolves around a few core principals, including starting small with a simple but targeted use case, getting the right mix of personnel involved in a project (including C-suite buy-in), measuring the actual value of the benefit, and effectively explaining what it is you’ve done. Keeping Schmarzo’s SAM principle in mind is also not a bad idea to keep big data projects from crashing and burning.
It’s all about being methodical in your approach to big data, and thinking about the problems and benefits in a practical manner before you get in over your head and can’t find your way forward. It’s not necessarily a sexy approach, but it’s good sound advice, and is probably what the high-priced management consultants would tell you anyway.
“The whole idea here is to get them thinking about how technology can be applied before they’re actually in the situation of owning it and having the pressure on them to prove out the ROI,” Brust says. “I don’t think big data has an ROI problem in and of itself. But I think the way the industry by and large has sold the technology has led to those problems, because it has not necessarily focused as a first priority on ROI.”
While big data technology is advancing at a break-neck pace, it’s worth stepping back from time to time to assess what the real business value is. Adopting technology for technology’s sake—even if one is under intense pressure not to be left behind in the race to digital transformmation —rarely puts one on a good solid path to profitability.
“What we’re trying to do is help make explicit to people where big data technology can bring ROI,” Brust says. “Maybe it would be great if we could focus on the technology and not have to do that. But when the technology is gnarly new, you do have to do some of that because that’s what adoption and success comes from. It’s ridiculous to expect customer to have that sensibility and a priori knowledge in a natural and organic way, because companies aren’t focused on that. They’re focused on their business.”
If more vendors did what Datameer is doing, perhaps big data wouldn’t be dogged by the ROI question quite as much as it is.