Follow Datanami:
October 5, 2016

No Sign of Big Data Spending Slowdown, Vendors Say


Gartner turned some heads yesterday with a new study that found fewer companies had plans to invest in big data projects over the next two years. But executives with Cloudera, Teradata, Databricks, and others say they’re seeing a continued expansion in advanced analytic projects, even if they’re not called “big data.”

In June, Gartner conducted an online survey of 199 research clients, who were asked about their big data plans. According to Gartner, the percentage of organizations planning to invest in big data within the next two years fell from 31% when Gartner asked the same question in 2015, to 25% this year. “Investment in big data is up,” Gartner analyst Nick Heudecker wrote in the report, “but the survey is showing signs of slowing growth with fewer companies having a future intent to invest.”

Tom Reilly, CEO of Hadoop distributor Cloudera, was surprised to hear about Gartner’s report of decreased big data spending. “It’s very contrary to what we’re seeing. We’re seeing tremendous growth and interest in what we do,” he says. “Maybe I’m coming off a [Strata + Hadoop World] conference high, but I saw the advancement this past week. I saw very little skepticism, and tremendous optimism and excitement for the potential of this platform.”

A similar viewpoint was expressed by Bill Creekbaum, vice president of product at GoodData. “This does not mesh with what GoodData is experiencing today,” he tells Datanami. “Currently we see companies significantly increasing their investment in data.”

Big Data Spending Strong

Dan Graham, the general manager of enterprise systems at Teradata (NYSE: TDC) says it’s important to keep what Gartner said in context. “Gartner’s survey doesn’t measure actual cash spending, only the intent to spend,” he says. “Also, their survey takers fell from 437 in 2015 to 199 in 2016 so the survey sample is less accurate.”

Sales of Teradata’s big data products and services is brisk, Graham says. “Teradata continues to sell Hadoop appliances at the same pace as 2015,” he says. “We see a constant demand for Think Big consulting services in the Americas and Europe for big data projects.  At the macro level, investment in big data hasn’t slowed, but there appears to be some softening for Gartner’s premium research panel of clients.”

Kavitha Mariappan, vice president of marketing at Apache Spark backer Databricks, says customer activity contradicts Gartner’s view. “We have seen tremendous growth in the number of big data projects emerging, across a variety of market segments and industry verticals,” she tells Datanami via email “An interesting data point is that IDC yesterday reported that their research indicates that the big data analytics market is set to hit $203B in 2020. That is a double-digit increase from this year ($130.1B in 2016).”

Kavitha Mariappan, VP of marketing for Databricks, sees increased spending on Spark-related projects

Kavitha Mariappan, VP of marketing for Databricks, sees increased spending on Spark-related projects

But Mariappan agrees with Gartner’s assessment that getting value out of big data projects can be difficult. “Very few businesses have a way to integrate their increasingly massive and complex data silos, run experiments against the data, and build software application with the data,” she says. “Typically, we’ve seen organizations start out with a traditional data warehouse, then they add on a data lake, and these days there are typically a lot of data in the cloud. All these different silos of data are growing very fast….At this point, most enterprises can barely get an accurate inventory of all the data they have, let alone actually do something with the data. Many companies start by trying to centralize the data to make it easier to access, but often even the data centralization effort fails. This is the one of the major impediments to the path to ROI we are seeing in companies.”

Praveen Kankariya, CEO of Impetus Technologies, says his company will double sales of big data products and services this year, for the second straight year. “The leaders are all still investing,” he says. “In some ways, we are encountering more engagements that are far more serious than the experimental PoCs [proofs of concept] of a few years ago.”

Kankariya can see why some enterprises may be pulling back some big data investments, especially considering the lack of skills and mature offerings from ISVs. “But has the need for unified data and analytics architecture that big data based solutions can provide disappeared?” he asks. “Clearly not. I do not see any other way. Pulling back will have huge costs in competitive advantages in years to come. If I have to pick one major culprit in some of the enterprises that are slowing their investments, it is lack of leadership and internal alignment.”

Investment Shifting to Use Cases

“We don’t see big data spending slowing down,” says Priyank Patel, co-founder and CPO at Arcadia Data, which develops a BI and visualization product that runs natively on Hadoop. “We see it shifting from infrastructure into consumption and business-facing use cases within large enterprises.”

There’s a definite change occurring in the types of analytic projects companies are doing, Patel tells Datanani. “Organizations are shifting focus from big data pilots to connecting business users and analysts with big data, which means legacy BI and analytics providers must address the requirements of scale, real-time access, and performance,” he says. “Shifting from a TCO-based approach to ROI is exactly where data-savvy enterprises are headed.”

Erica Volini, a principal at Deloitte Consulting, agrees with Gartner’s view, but sees caveats in how the money is being spent. “I think that their underlying premise that big data efforts need to be more focused and targeted at specific business issues is in alignment with what we’re seeing in the market,” Volini says. “So, for me, it’s less about a slowdown in spending, but rather a more targeted focus, which may result in lower overall spend, but does not necessarily indicate a decrease in big data as a top priority for organizations.”

Zaloni CEO Ben Sharma sees

Zaloni CEO Ben Sharma sees a “recalibration” in big data spending

Like many companies, Deloitte Consulting clients are struggling with the process of turning data from insights into action. “How do they prove that the data is providing a bottom line ROI for the organization, and is driving more value than just highlighting a trend? That’s where the more targeted efforts are coming in,” Volini says. “Take for example, the issue of labor cost.  That is a huge area where we are seeing HR leverage big data to not only provide insight in terms of what the spend on labor is, but providing recommendations on how to reduce that labor cost. It’s the combination of analysis and recommendations on an issue which is front and center for the business, which is where the true value from big data comes from.”

Big data is way more complex than it needs to be, says Ashish Thusoo, the CEO of Hadoop as a service provider Qubole. “First generation deployments have been fraught with risk, including the time consuming build out of expensive infrastructure, long project timelines, and lots of complex decisions around configuration and data modeling,” he says. “The cost of mistakes is very high.”

The second generation of big data is here, and it will require customers to scale up their use cases, he says. “We believe the cloud and greater automation are going to be key to helping companies find value and success with big data.”

Ben Sharma, the CEO of Hadoop tools provider Zaloni, says he agrees with Gartner’s assessment that difficulties in getting a solid return on investment (ROI) and the challenge of moving from pilot to production are contributing to a rough patch for big data projects at the moment. “We believe that companies are taking a more measured approach to big data,” he says. “But we see it as a re-calibration rather than as dying on the vine.”

While companies are aware of some of the difficulties commonly encountered in big data project, Sharma says, they also realize there’s a “huge payoff” in terms of reduced risk, cost-savings, and new revenue sources when they finally get it right.

“Companies are realizing that their initial data lake implementations, particularly if done hastily or with a data ‘dump and run’ mindset, do not fulfill business needs,” Sharma tells Datanami via email. “So many are now embarking on their next generation data lake, and talking to companies like ours about data management and data governance best practices and solutions.”

Cloudera’s Reilly says the “dump and run” approach taken by some early Hadoop adopters is backward. “I’m very strong on this,” he tells Datanami in an interview. “If you start out by throwing all your data into a data lake, and ask that data what is it going to tell you, it’s just going to tell you you have a lot of data. You need to take a use-case approach if you want an ROI out of it. If you have a use case, you know what data you need to get.  You have a theory that if you run some algorithms, what you’re going to learn. You validate it with the data. Then you operationalize it.”

Watered-Down Semantics

Focusing on cases brings up another aspect of the supposed slowdown in big data investment that Gartner is seeing. When you focus on specific use cases for big data–such as cybersecurity, customer-360, churn reduction–it’s no longer a “big data” project, Reilly says. “Is using Apache Hoop for cybersecurity big data?

Cloudera CEO Tom Reilly says many organizations implementing advanced analytic tools and techniques do not call what they're doing "big data"

Cloudera CEO Tom Reilly says many organizations implementing advanced analytic tools and techniques do not call what they’re doing “big data”

Some might say it’s not big data. It’s cybersecurity,” he says. “Nine of the top 10 telecos on the planet are our customers. We’re approaching 100 telcos globally. They’re all using it for customer churn. I don’t think any of them would call that a big data project. It’s churn reduction. We work with 8 of top 10 banks, and they often use us for anti-money laundering or fraud detection or Basel II. I don’t know if those are big data projects. I’d call it an anti-money laundering project.”


To some extent, it’s a matter of semantics. The phrase “big data” has multiple meanings, and this must be kept in mind when assessing the results of opinion polls. Clearly, the use of machine learning and other advanced analytic techniques to glean useful information out of large amounts of data is a trend that is not going away.

That defines “big data” to many people working in this industry, but it doesn’t have to. At some point, it seems clear that this sort of activity will become just another regular part of our lives, and when that happens, it would be silly to continue calling it “big data” anymore.

Related Items:

Has Big Data Peaked?

‘What Is Big Data’ Question Finally Settled?

Companies Struggle to Find an ROI on Analytics