Follow Datanami:
January 5, 2016

Is 2016 the Beginning of the End for Big Data?


Part of my job as managing editor of Datanami is to be a hype buster. Over the past five years, the amount of hoopla and hysteria over big data has reached epic proportions at times. That’s why some in the industry are calling for a slowdown in the analytic platitudes, a return to sanity about what big data can really do. Despite my skeptic tendencies, it’s hard not to be optimistic about big data analytics, given its trajectory.

If you draw a line from where we were 10 years ago and where we are today, you will notice clear increases in the key variables at work, such as the amount of data generated and the availability of processing power to act upon it. We are creating gobs of data these days – the volume of data is said to be doubling every two years or so – and thanks to the cloud environments built by the likes of Amazon (NASDAQ: AMZN), Microsoft (NASDAQ: MSFT), and IBM (NYSE: IBM), the prospect of on-demand computing is a reality.

But it’s harder to draw that line when it comes to another key variable in the big data equation: software. Back in 2006, the biggest thing to happen to software in a generation—the rise of the iPhone and the rise of the “app” economy—didn’t even exist. Accessing the Internet from my Windows Mobile-equipped Motorola phone was a hoot, but it pales next to what the latest Android and Apple (NASDAQ: AAPL) smartphones can do today.

From the software geek’s perspective, developers were just starting to talk about writing apps using AJAX (a combination of JavaScript, Apache Web Server, and XML), atop a standard LAMP (Linux, the Apache Web Server, MySQL, and PHP) computing stack. The Web 2.0 revolution was just beginning, and the age of static websites was nearing its end.

The launch of the iPhone in 2007 had a big impact on the trajectory of big data analytics (charnsitr/Shutterstock)

The launch of the iPhone in 2007 had a big impact on the trajectory of big data analytics (charnsitr/Shutterstock)

From 2007 to 2010, the nature of the Internet and the way developers created apps for it had begun to evolve in a way that was appeared quite organic and natural at the time. But in hindsight, it’s clear that the new crop of hyper-dynamic and super-personalized applications were creating the foundation for the current big data wave, and hence the need for big data analytics. It’s no coincidence that Hadoop was created to be a bigger and better Internet search engine.

Next year, we’ll celebrate the 10th birthday of the iPhone and reflect on the enormous changes it brought to the world. But even today, so-called “smart devices” are appearing in many forms. From Toyota Corollas and Samsung TVs to Nest door locks and GE refrigerators, it’s clear we’re in the midst of a massive and unprecedented automatization trend, and “intelligent” software is the key to all that.

But here’s the thing: This grid of digitized devices, better known as the Internet of Things, is just getting started. We’re just beginning to explore and exploit the potential to squeeze more automation from our stuff, which ostensibly will give us more time to pursue more value-added activities.

I see no reason why that software line drawn from 2006 to 2016 won’t continue for the foreseeable future. When you consider the advances being made in the fields of machine learning, cognitive computing, and artificial intelligence—and how those rapidly evolving technologies will help us to build even smarter devices over the next 10 years–it’s tough not to be excited.

That’s not to say there won’t be changes and there won’t be spectacular failures of big data analytics. While the technologies and techniques behind advanced analytics have proven they can deliver a competitive advantage, it is not easy to make use of them. There is still a giant skills gap in the field, and while big data analytic software vendors are adding more automation to their wares, the most innovative applications still require the oversight of a trained and competent data scientist.

It’s no surprise that leaders in the field are equally optimistic about the future of big data. “Despite incredible advances over the past few years, we’ve barely scratched the surface on the full potential of analytics,” says Scott Zoldi, chief analytics officer at FICO (NYSE: FICO). Zoldi sees several analytic types making inroads in 2016, including streaming analytics, next-gen antivirus tools, and “lifestyle analytics” like predictive grocery ordering and remote medicine.

Artificial intelligence--backed by advances in machine learning and cognitive computing--is set to be a major contributor of big data advances in the coming years (Willyam Bradberry/

Artificial intelligence–backed by advances in machine learning and cognitive computing–is set to be a major contributor of big data advances in the coming years (Willyam Bradberry/

“We will see a turning point for data science” in 2016, says Michael Brodie, a Research Scientist at the MIT Computer Science and Artificial Intelligence Laboratory. Brodie says we’ll finally overcome the “false promises of trivialized point-and-click, self-service tools” and begin to build predictive apps that target more domain-specific problems.

Another big data optimist is Michael Benedict, the chief product officer at Progress Software Corp. (NASDAQ: PRGS). “This first wave of big data focused on the infrastructure stack–storage, scale and integration,” Benedict says. “It’s the next wave of technology that I’m most excited about, because it will make big data mainstream and consumable by everyone.”

There is bound to be some consolidation among the software and infrastructure companies enabling big data, and that isn’t necessarily a bad thing. The past five years have been marked by a rapid period of investment and innovation, and the invisible hand of the market has a way of sorting things out.

Don’t be surprised if the amount of hype surrounding big data analytics tails off, or even if naysayers proclaim the end of the big data era if we should see a large number of big data startups fold up their tents or get acquired when their business plans don’t work out. But does that signal the end of big data analytics? No, it doesn’t. Because the evidence of early adopters is clear: big data analytics is the real deal, and it’s going to be with us for a long, long time.

Related Items:

Industry Speaks: Top 33 Big Data Predictions for 2016

Three Unique Ways Machine Learning Is Being Used In the Real World

10 Tips for Beginning Your Big Data Journey