Survey Casts Doubt on Big Data, Hadoop Efforts

A rule of thumb in markets like consumer electronics is don’t be an early adopter; wait until the vendor works out the kinks in its product and go with a later, more mature version.

The same could be said for the growing number of big data initiatives as well as Hadoop distributions used by a growing number of enterprises: A survey of more than 100 senior executives found that about three-quarters are dissatisfied with their big data and analytics deployments.

The findings released this week indicate “a huge unmet need for better analytic tooling to enable companies to glean the full value of the data,” concluded the survey released by “Big Data 2.0” vendor Actian Corp. The findings are especially troubling since the survey also found that 61 percent of respondents consider data their most valuable company asset.

The May 2015 survey of 106 executives, IT leaders and data scientists across industries ranging from healthcare to banking highlight shortcomings in Hadoop. It found that “painful trial and error has revealed that traditional database technologies are failing to deliver on analytical workloads, so they have turned to Hadoop for help.”

While Hadoop has proven to be “a very cost-effective place to store massive amounts of data, most are finding it’s too immature to manage enterprise-grade, high-performance analytics jobs,” Ashish Gupta, Actian’s senior vice president of business development, stressed in a statement releasing the survey results.

Just over half of respondents said Hadoop could improve existing enterprise data analytics, but surprisingly only about 5 percent were asking for it. A limiting factor, the survey found, is the growing data analytics skills gap. Hence, enterprises tend to stick with what they know—in this case, primarily SQL—”to break into Hadoop.”

Along with a lack of Hadoop skills, some respondents said the framework requires “tooling to make it more enterprise-grade, secure and fast.” Forty percent said one thing they would change to would be making Hadoop data accessible via SQL and business intelligence tools.

The survey also found heavy investment by organizations in time and resources in leveraging Hadoop “because they know that their current way of managing analytical workloads won’t cut it,” said Gupta. That stakes are high: More than 25 percent of CEOs surveyed said a failed big data project is a “fireable offense” for CTOs or CIOs.

In a separate blog post, Gupta argued that “industrialized SQL analytics” are playing a greater role in plugging the current gap between analytics expectations and reality.

For example, 40 percent of data scientists and business analysts said SQL remains their primary analytics tool. In the future, the majority of those surveyed said they want to “broaden data access across their company to enable data-driven decision making [while] more than 40 percent want to improve connectivity between data stores and IT systems to enable more people to run analytics,” Gupta noted.

Recent items:

To Hadoop or Not? That is the Question

Nine Criteria to Achieve Hadoop-as-a-Service Happiness