Congratulations Hadoop, You Made It–Now Disappear
Hadoop has matured at a rapid rate since it broke into the mainstream several years ago. With billions in venture funding, an eager community of developers, and a thriving ISV community, the open source big data platform seems poised to take a big step into the wider enterprise. But according to Cloudera’s chief strategy officer Mike Olson, Hadoop’s next big move may be to just disappear.
Olson predicted Hadoop’s disappearance during a keynote address at last week’s Strata + Hadoop World show in New York City. Last week’s show was Olson’s sixth Hadoop World conference, and he commenting on its incredible growth from a room at the Roosevelt Hotel to the massive Javitz Center in Manhattan, where 5,000 people attended the show last week.
In the early days of Hadoop World, the CEO and co-founder of Cloudera executive would get on stage and talk about “geeky technology” like Pig and Flume and Sqoop. Nowadays, the talk has turned to how Hadoop can enable business solutions, such as stopping fraud and encouraging customer engagement. So what happens next? “I think this year we’re going to see Hadoop disappear,” he said. “It’s a pretty ridiculous statement in a venue like this, but bear with me.”
In Olson’s reasoning, Hadoop disappearing is a good thing because it takes the focus off the enabling technology and puts it on finding ways and building applications that organizations can use to extract value from data. Instead of concentrating on getting the geeky technology to work correctly, people ought to be wondering how they can get creative with data and make something new.
Hadoop will disappear just like other underlying database technologies have disappeared, Olson says. “Historically Hadoop was this really complicated piece of technology and the IT specialists had to custom build applications that ran on it,” he says. “Now if you talked to customers of Teradata and EMC and the rest, you’ll hear business users getting enormous value from those platforms, but have no idea that they’re there. Those platforms are foundational. They’re buried beneath the applications that sit atop them and the business users have no idea that they’re there. In a really important way, Teradata, EMC, Oracle and the rest have disappeared. Hadoop is making that transition now.”
That’s not to say that there will no longer be a need for geeky technologies like Pig and Sqoop and Flume–or for Teradata, EMC, or Oracle, for that matter. In fact, Cloudera just announced partnerships with the first two. And there’s more geeky technology on the horizon, particularly with Apache Spark, which is gathering huge momentum among developers and ISVs because it enables them to build big data analytic workloads without the complicatedness of first-gen Hadoop. But even if Spark and the rest help abstract away some of the underlying complexity, the complexity is still there under the covers.
What is not in doubt is that the big data analytics business is booming. While there may be 1,000 or so paying Hadoop customers around the world, that number is now poised to grow in the coming years. Olson recited the Gartner statistic that says 73 percent of companies either have a big data analytics solution or have plans to get one. That huge potential is what’s driving the incredible valuations in this space. Cloudera, for example, is valued north of $2 billion, despite having yearly revenue on the order of $100 million (according to estimates, since the company is privately held).
In Olson’s thinking, Hadoop will have succeeded when it fades away into the background and becomes a standard component in the IT stack. Businesses don’t want to worry about building the underlying big data infrastructure. Instead, they just want applications, and it’s up to the Hadoop community to deliver them.
“All of us in this room can take advantage of that progress. We get to design applications on that platform that anyone can afford,” he says. “Enterprises buy and large don’t want to build custom homes. They don’t want to create custom tools and applications. There’s an opportunity for us to deliver value to them.”
Big data is a massive opportunity for those poised to capitalize on it. According to Olson’s math, if the market for relational databases and tools today is worth $100 billion, and the amount of data available to enterprises has grown by 1,000x, then a conservative 10x multiple puts the total big data value proposition at an eye-popping $1 trillion.
“We’re in a position how where we expand into this entire space, where we can take over the Javitz and really demonstrate the progress in the market,” he says. “Because of the progress the technology has made, the platform we’ve all built, the technology that Doug [Cutting] create on the back of some Google papers in the early part of this centrally. It’s been magnificent to work with many of you for many years, and bringing the tools and technology this far. But what’s really exciting is the opportunity before us, the chance now to drive these platform and technologies to absolutely ubiquity and to unlock huge new value.”