Forrester Sees Steady Growth for Big Data, Hadoop, and NoSQL

(jijomathaidesigners/Shutterstock)

A new report from Forrester predicts the big data market growing at nearly 13% rate over the next five years, with “non-relational” platforms like Hadoop and NoSQL segments growing nearly twice as fast.

In the Forrester ForecastView, titled “New, Evolving Tools To Manage Big Data Will Grow At Double-Digit Rates,” Forrester analyst Jennifer Adams sliced and diced the current state of the data management market, and gave prognostications for how she sees it shaping up through 2021.

“Five years ago, big data was still a buzzword, but today, it’s a standard business practice,” she writes in the intro. “In 2016, almost 40% of global data and analytics decision-makers say their firms are implementing as well as expanding big data technology and solutions, and another 30% are planning to adopt big data in the next 12 months.”

Where will all this expected big data spending go? That’s the million-dollar question that big data software developers—including Hadoop distributors, NoSQL vendors, and everybody else involved in the big data racket—wants to know.

Forrester divides the overall big data management space into six segments, including enterprise data warehouse, NoSQL, Hadoop, big data integration, data virtualization, and in-memory data fabric. The predicted rate of growth for all of these, 13%, is more than twice the predicted rate of growth for IT in general, which Forrester says will be 4% to 5% over the coming years.

The enterprise data warehouse (EDW) market—which includes fast column-oriented databases like Vertica, Teradata, and the like–accounted for $15.4 billion in spending in 2015 and will grow to $25.4 billion by 2021, according to Forrester. Interestingly, Forrester sees the nascent data virtualization industry doubling from $3.3 billion in 2015 to $6.7 billion in 2021.

Forrester sees “non-relational databases,” such as NoSQL and Hadoop, exceeding the overall big data market (no matter than Hadoop is not a database, per se, but a framework built atop a distributed file system. Mere quibbles). “We forecast that NoSQL will grow by 25.0% and Hadoop will grow by 32.9% annually over the next five years,” Adams writes.

While the deltas are big, the market as measured in actual dollars isn’t. Adams sees the Hadoop market pulling about $2.3 billion in revenue by 2021, while the NoSQL business will bring about $1.6 billion. It’s likely that relational database technology will still outsell these big data big-wigs by a significant margin in five years.

Adams writes that Hadoop growth will be “robust,” thanks to its capability to run data-intensive applications that can’t run on legacy technology. She cites Arizona State University’s Hortonworks (NASDAQ: HDP) cluster, which is used to store 4 petabytes of cancer genome data, as an example of how Hadoop is breaking down barriers to research.

“Your average database of 20 billion rows is simply unapproachable with traditional, standard technology,” Forrester quotes ASU’s Dr. Kenneth Beutrow as saying. (For more about ASU’s 44-node HDP cluster, see “How Spark and Hadoop Are Advancing Cancer Research“.)

Hadoop is having a pretty good year so far in 2016, according to Adams. “Based on our survey data, 30% of respondents have implemented Hadoop in 2016 versus only 26% in 2015.”

The cloud, in particular, is turning into a major factor driving Hadoop. According to Forrester, the percentage of survey respondents using a Hadoop-based cloud service to store unstructured data has grown from 29% in 2015 to 35% in 2016. “Hadoop in the cloud allows the analysis of more data using cheaper infrastructure and enables faster advanced analytics,” Adams writes.

In fact, Hadoop could grow even faster than 30% annually. “We think there could be some upside to our estimates,” Adams writes, citing Hortonworks 46% year-over-year revenue growth.

According Forrester’s survey, 41% of respondents have already implemented a NoSQL database, while another 20% are planning to implement one in the next year. The capability to manage huge amounts of data – such as eBay’s MongoDB implementation that stores 1 billion live listings, and PayPal’s Couchbase implementation that manages 1 billion documents—account for the intense interest in NoSQL databases, writes Adams.

“Enterprises report that NoSQL products saved them more than 50% of the cost, compared with conventional DBMSes,” she writes.

Source: Forrester

Also forecast to grow quickly is the in-memory data fabric space, which Forrester sees ballooning from a $1.3-billion market last year into a $6.4 billion behemoth by 2021. That 30% compound growth rate will be driven by in-memory computing’s capability to enable “real-time computations and transactions on large-scale data sets, orders of magnitude faster than what’s possible with traditional technologies,” Adams writes.

Up until now, big data management tools have been predominantly used in the professional services, telecommunications, government, and financial service sectors. But over the next five years, Adams sees the biggest adoption of big data tech coming from companies in the pharmaceutical, transportation, and primary production industries. “These sectors are late adopters and will drive growth off relatively low bases,” Adams writes.

While Hadoop’s forecasted growth of 30% is nothing to sneeze at, it pales in comparison to some of the giant numbers analyst gropes and researchers were talking about several years ago. That includes Allied Market Research, which two years ago predicted the global Hadoop market would exceed $48 billion by 2020 and grow at a 25x (a tad more than a 25%) clip. And back in 2013, the then-independent analyst group Wikibon (which is now affiliated with the tech n3ews site Silicon Angle) forecasted that the big data market would drag $50 billion in revenues by 2017.

However, Forrester’s predictions also show there is reason to be hopeful after several reports of lackluster big data adoption, such as the 2015 TDWI report that found Hadoop used in 13% of organizations (up from 10% in 2013), or the 2013 Gartner report that found a paltry 8% of organizations had adopted Hadoop or a NoSQL database. There is clearly momentum in the big data space. Although the market isn’t shaping up to be exactly what the prognosticators said it would be five years ago, it’s growing nonetheless.

Related Items:

Forrester’s Crowded NoSQL Wave Shows Abundant Options

5 Factors Driving the Graph Database Explosion

9 Ways Retailers Are Using Big Data and Hadoop