Language Flags

Translation Disclaimer

HPCwire Enterprise Tech HPCwire Japan

June 27, 2013

Gartner’s Adrian Raps on Big Data’s Present and Future

In a keynote session to kick off Hadoop Summit in this week, Gartner research VP, Merv Adrian, gave a level-setting presentation on Gartner’s view of the Big Data market space, discussing the preliminary results from a recent market survey, and discussing the challenges as well as impending innovations. With numbers piping hot out of the survey oven, Adrian said that the sector has a lot to look forward to, and more work still to do.

Adrian kicked off with a numbers parade that represented some very good news for vendors looking to cash in on the big data movement. The size of the investment community for big data solutions is robust, according to Gartner, with nearly 65% of the surveyed companies saying that they either already have invested in big data, or have plans to in the next two years. Adrian said that 30% of those vendors surveyed had already made investments; 19% plan to within the next year; and 15% plan to invest within the next two years. 

This good news was not given without a caveat. “There’s something about predictions,” he quipped. “You have to remember that a CTO’s budget is a little bit like a kid’s letter to Santa Claus. Sometimes they get what they want, and sometimes they don’t.”

Looking at the negative side of the numbers, a full 31% of those surveyed have no plans for big data, and according to Adrian, given the way the questions were structured, “have no plans to have a plan.” Interestingly, in a comparison to last year, Adrian noted that this number stayed exactly the same over the last two years.

“There is an intractable third of the marketplace out there that so far says ‘big data - dismissive hand wave - hype. Let those guys go to Silicon Valley, we're not ready for any of that stuff yet.’” Adrian commented that the rising big data phenomenon still has a long way to go to demonstrate value to this sizeable dark area of the market. Adrian did note a bright spot in the negative number trends in that the number of people who didn't know what their plans were have fallen by over half from the previous year, from 11% to 5%.

In addressing challenges in the industry, Adrian noted several factors, including the complexity of Hadoop, itself. In a striking visualization, he represented the Hadoop stack with all the components for such functions as ingesting & propagating the data, describing & developing the data, monitoring, machine learning, etc. With each stack on its own plane, he streamed the potential solutions for each aspect (both open source and commercial pieces), demonstrating the complexity of Hadoop, and thus the need and opportunity for the distribution vendors in the space.

Adrian also described a culture clash that exists between the old guard he calls “the suits,” and the rising wave he represented as “the hoodies.” The suits, he said, use data warehouses that represent curated, pre-optimized collections of data that answer the set of data they know they're going to be asked, use commercial software, and are very grounded in the old world of data management. The sweatshirts are opportunistic, experimental, they play with the tools they have, use them for discovery, and are into open source software. The research analyst intimated that these cultures would need to be bridged in order for organizations to be successful in the future.

Looking at the horizon, the research analyst discussed what he saw as the next steps for Hadoop as it emerges into its 2.0 form. These included the following:

  • Search – “Let me apply the Silicon Valley standard: It’s been announced, so it’s here,” said Adrian, poking fun at the recent beta announcements. “Realistically…It is here, it is coming, it’s in products, we’re going to see it in real life, people are going to start using it.”
  • Advanced Prebuilt Analytic Functions – “We’re going to see these advanced analytic functions coming into the market, or be integrated into the stuff we’ve got.”
  • Cluster, Appliance, or Cloud – “We’re going to see a very big marketplace question answered: do we keep building clusters, node by node – or do we start buying appliances. There’s a lot of people making appliances right now – it’s not clear at all that that’s what the market wants. And of course they can go to the cloud as well, and often both.”

He also touched on virtualization and graph processing, and, of course, the belle of the Hadoop Summit ball, Apache YARN, framing the advancement as a great leap forward.

Speaking of gaps that still need to be filled, Adrian pointed to a lot of the familiar culprits, including security, governance, data warehousing tools, distributed and subproject optimization, and the ever persistent challenge of skills.

 “Gartner estimates that we will need 4.4 million people worldwide by 2015 to do the big data work,” said Adrian, explaining that this number runs across all geographies and verticals. According to Gartner, only one-third of those jobs will be filled, which he framed as good news for those who do have the proper skills. “When resources are scarce like that, bidding goes on.”

To close his talk, Adrian encouraged the developers at the event to be proactive in their organizations by finding the opportunities to uncover “dark data” that can be mapped to business opportunities, and building alliances across the organization to help facilitate the move into the new data era. He said that developers should consider cloud pilot projects that minimize capital expenditure. And of course, our favorite, he encouraged the audience to follow the news to keep up with the fast changing environment.

Related items:

IDC Talks Convergence in High Performance Data Analysis

On Algorithm Wars and Predictive Apps

HPC Analysts: Hadoop is a Tall Tree in a Broad Forest

Share Options


» Subscribe to our weekly e-newsletter


There are 0 discussion items posted.


Most Read Features

Most Read News

Most Read This Just In

Cray Supercomputer

Sponsored Whitepapers

Planning Your Dashboard Project

02/01/2014 | iDashboards

Achieve your dashboard initiative goals by paving a path for success. A strategic plan helps you focus on the right key performance indicators and ensures your dashboards are effective. Learn how your organization can excel by planning out your dashboard project with our proven step-by-step process. This informational whitepaper will outline the benefits of well-thought dashboards, simplify the dashboard planning process, help avoid implementation challenges, and assist in a establishing a post deployment strategy.

Download this Whitepaper...

Slicing the Big Data Analytics Stack

11/26/2013 | HP, Mellanox, Revolution Analytics, SAS, Teradata

This special report provides an in-depth view into a series of technical tools and capabilities that are powering the next generation of big data analytics. Used properly, these tools provide increased insight, the possibility for new discoveries, and the ability to make quantitative decisions based on actual operational intelligence.

Download this Whitepaper...

View the White Paper Library

Sponsored Multimedia

Webinar: Powering Research with Knowledge Discovery & Data Mining (KDD)

Watch this webinar and learn how to develop “future-proof” advanced computing/storage technology solutions to easily manage large, shared compute resources and very large volumes of data. Focus on the research and the application results, not system and data management.

View Multimedia

Video: Using Eureqa to Uncover Mathematical Patterns Hidden in Your Data

Eureqa is like having an army of scientists working to unravel the fundamental equations hidden deep within your data. Eureqa’s algorithms identify what’s important and what’s not, enabling you to model, predict, and optimize what you care about like never before. Watch the video and learn how Eureqa can help you discover the hidden equations in your data.

View Multimedia

More Multimedia


Job Bank

Datanami Conferences Ad

Featured Events

May 5-11, 2014
Big Data Week Atlanta
Atlanta, GA
United States

May 29-30, 2014
St. Louis, MO
United States

June 10-12, 2014
Big Data Expo
New York, NY
United States

June 18-18, 2014
Women in Advanced Computing Summit (WiAC ’14)
Philadelphia, PA
United States

June 22-26, 2014

» View/Search Events

» Post an Event