The Big Data Market By the Numbers

Here’s a little number about big data that may surprise you: 8. As in, only 8 percent of organizations around the world have actually spent any money or time to build an actual big data application (such as Hadoop or a NoSQL cluster) in their shops, according to Gartner research findings announced last month.

That low adoption rate may seem way too low for what is arguably the IT industry’s hottest sector. Gartner is a storied analyst firm. But perhaps it got some bad data, didn’t normalize the numbers, or forgot to eliminate some crazy outliers?

Nope. A survey by the Data Warehouse Institute (TDWI) released earlier this year confirmed Gartner’s finding when it found that 10 percent of organizations were using Hadoop. Data integration vendor Talend’s August survey also found just 10 percent of its customers around the world had current big data projects underway.

That 8-10 percent range doesn’t tell the whole story, of course. In fact, big data is much bigger in North America than in other parts of the country. According to Gartner, 38 percent of organizations in the U.S. and Canada report being currently invested in big data technology (including but not limited to Hadoop and NoSQL).

The Irish research firm Markets and Markets had similar findings on North America’s big data lead. Europe has lagged North America in the adoption of big data technology, but it’s catching up. According to IDC, about one third of European organizations will have big data applications in production by the end of 1Q14. On a worldwide basis, 20 percent of organizations are piloting and experimenting with big data, while 18 percent are “working on a strategy,” according to Gartner’s survey. The remainder are either in the “knowledge gathering” phase or have no big data plans, or checked “don’t know” on the questionnaire.

How big is the big data market in terms of spending? (We’re glad you asked!) The market watchers at Wikibon, an open source provider of market analysis, recently updated its forecast for spending on Hadoop and NoSQL software and services. It now expects the global market for these technologies to grow from $542 million in 2012 to $3.48 billion in 2017, a 45 percent compound annual growth rate (CAGR). NoSQL seems to be trending a little higher than Hadoop spending at this point, the group says.

Source: Wikibon 2013

IDC issued a report 17 months ago that found Hadoop pulled about $77 million in spending in 2011–far less than the $300 million in venture capital funding that Hadoop-related companies had attracted up to that point, according to IDC. Hadoop spending was on track to grow to $812.8 million by 2016, for a CAGR of about 60 percent, according to IDC. By that math, Hadoop sales would be on a track for about $200 million in spending in 2013 (assuming that assumptions haven’t changed and that the CAGR is a smooth line, both of which are undoubtedly untrue).

The entire big data market will reach more than $18 billion in spending this year, according to Wikibon analyst Jeff Kelly, and reach $47 billion by 2017, a 31 percent CAGR. Kelly and company have put together a who’s who list of big data vendors, complete with expected 2013 revenue. IBM, by the way, is tops with $1.3 billion in big data bucks.

Markets and Markets and IDC have similar views. According to an August M&M report, the entire market for big data solutions is worth nearly $15 billion this year and is expected to grow to more than $46 billion by 2018. This represents a CAGR of about 26 percent from 2013 to 2018. Meanwhile, IDC predicts that, from 2012 to 2016, big data spending will grow at a CAGR of about 32 percent.

“The big data market is expanding rapidly as large IT companies and start-ups vie for customers and market share,” IDC’s big data guru Dan Vesset said.

While the hype of big data has definitely exceeded the substance up to this point, the dynamics are expected to change. Gartner’s study found that 64 percent of organizations were investing or had plans to invest in big data technology during 2013, compared to 58 percent who said the same in 2012. TDWI’s survey from earlier this year found that 51 percent of survey respondents planned to deploy Hadoop within three years.

Admittedly, “planning to spend” is a weak metric on which to base one’s strategy. That just goes to show that there is a lot of hype to burn off yet. That brings us to Gartner and its Hype Cycle graph.

According to Gartner, big data is currently in the “Trough of Disillusionment” stage. “When the market starts to reach 15-20 percent adoption, then big data will have reached the Plateau. That’s the end of ‘hype’ and the beginning productivity,” wrote Mark Beyer, the co-lead for Gartner big data research, in a recent blog post.

New technology adoption willl often follow some variation of Gartner’s Hype Cycle.

Where is big data used? (Again, we’re glad you asked!) According to Gartner, big data is biggest in the media and communications industries, where 39 percent of organizations have already made big data investments. That compares with 34 percent in banking and 32 percent for services. The transportation, healthcare, and insurance industries are expected to invest the most in big data solutions over the next couple of years, according to Gideon’s firm.

Looking at other areas of the big data market show that in-memory databases are selling well. According to Evans Data, the number of developers using in-memory databases increased 40 percent around the world from late 2012 into early 2013. M&M found a similar trend, and last month said it expected the in-memory database market to account for more than $13 billion in spending by 2018, on the backs of established vendors like SAP, Oracle, IBM, Altibase, and Qliktech and upstarts like GridGain, ScaleOut Software, Exasol, and Software AG.

Related Items:

Playing Stadium Tycoon with HANA and Hadoop

Venturing Into the Great Unknown with YarcData

New Cloud BI Company Numerify Emerges from Stealth Mode