What 2015 Will Bring for Big Data
There’s no denying that 2014 was a big year in big data. The rapid maturation of technologies like Hadoop and Spark, along with the continual explosion of all types of data, led to an awakening of the potential of distributed data analytics among organizations, and also fueled a fresh eruption of venture money into startups. With that backdrop, the prospects for the new year certainly look promising, but what new analytic technologies and techniques will resonate in 2015? Datanami contacted some of the data analytic industry’s top experts and executives to find out.
Apache Hadoop and the Hadoop distributions almost certainly will have another big year in 2015. While we’ve heard predictions before about how Hadoop science projects will turn into production apps, it finally appears to be coming true this year.
“2014 was the year of Hadoop adoption, where enterprises bought Hadoop, experimented with it and learned how to best make use of it,” says Bill Jacobs, vice president of product marketing for Revolution Analytics. “The experimentation we saw in 2014 will give way to production deployment in 2015 as industries collect sufficient data and are empowered to analyze that data to inform better business decisions. Hadoop will emerge as the best option to fit into heterogeneous ecosystems in most enterprises.”
The Hadoop marketplace saw a huge amount of activity in 2014, from Intel‘s exit and investment in Cloudera to Hortonworks‘ IPO. You can expect some additional consolidation in 2015, says John Schroeder, the CEO and cofounder of Hadoop distributor MapR Technologies.
“Hadoop is in the innovation phase so vendors mistakenly adopting ‘Red Hat for Hadoop’ strategies are already exiting the market, most notably Intel and soon EMC Pivotal,” he predicts. “In 2014 the booming ecosystem around Hadoop was celebrated with a proliferation of applications, tools, and components. In 2015 the market will concentrate on the differences across platforms and the architecture required to integrate Hadoop into the data center and deliver business results.”
Data analytics and the cloud occupy a lot of common ground, in both data centers and the minds of big data architects. In 2015, we’ll see more organizations getting started with Hadoop-powered data analytics by way of the cloud, predicts Bog Muglia, the former Microsoft server boss who now heads up Snowflake Computing, a big data warehousing as a service provider.
“The requirements and complexities of Hadoop have meant that only a handful of the most sophisticated companies, with access to the right resources and talent, have been able to make Hadoop a viable option,” he says. “2015 will be a turning point in which SaaS applications for data processing and analytics will bridge the data divide.”
Ali Ghodsi, head of product management and engineering for Apache Spark-backer Databricks, sees similar trends unfolding. “Enterprises are increasingly using the cloud for Big Data analytics for a multitude of reasons: elastic infrastructure needs, faster provisioning time and time to value in the cloud, and increasing reliance on externally generated data, e.g., 3rd party data sources, Internet of Things and device generated data, and clickstream data,” he says. The fact that big enterprise vendors like SAP, Oracle, and IBM are pushing analytics in the cloud is further proof that its time has come, Ghodsi adds.
Datameer CEO Stefan Groschupf sees clouds building for big data deployments. “In 2015, we’re going to see a continuous movement out of IT and into generating ROI-orientated deployment models. Secondly, I think we’re going to see resources move away from heavy in-house big data infrastructure to big data-as-a-service in the cloud. We’re already seeing a lot of investment in this area and I expect this to steadily grow.”
CIOs who have been reluctant to put their big data in the cloud will revisit the issue in 2015, predicts John Hogan, vice president of engineering and product management at cloud storage provider Storiant. “There has been a lot of hope and a lot of hype about ‘the Cloud’ solving the ‘big data’ problem. The main issue is that the public cloud is a terrible place to put your big data,” Hogan says. “While this [move to the public cloud] has been in the works, the pain just hasn’t been severe enough. However, the dam is about to burst.”
There’s a rule of thumb that says you should use the cloud to analyze data that originates in the cloud (i.e. The Internet). With the advent of the Internet of Things (IoT) and the rapid proliferation of new network-based data sources, that rule will find new life. In fact, Puneet Pandit, the CEO and co-founder of machine data analytics company Glassbeam, goes so far as to say that the IoT will emerge to become the killer app for big data.
“In 2015, business IoT will become increasingly relevant alongside consumer IoT,” Pandit predicts. “Data from IoT will grow to zetabytes and some of the most dynamic big data apps, such as smart grids, smart cities, and connected vehicles (V2V), will come out of this IoT phenomenon and lead the big data overhaul with their IoT adoption.”
The rise of wearables in 2014 demonstrated the huge potential for new ways people can consume and generate data. With the Consumer Electronics Show set to kick off in Las Vegas tomorrow, we’re sure to hear a lot of news about the latest breakthroughs in this area. But the IoT is a lot more than just smart watches and fitness trackers, says the folks at NAS storage provider Avere.
“As we see products like Nest and smart fridges creating the ‘Smart Home,’ it’s only a matter of time before this theme expands to create ‘Smart Businesses,’ such as wristwatch time clocks for hourly employees and desk chairs that send alerts to phones when an employee has been sitting for too long,” the company says.
2015 will also bring advances in the all-important interface between big data and humans. The first generation of big data apps were difficult to use and featured interfaces that left something to be desired. But you can expect to see a continual focus on improving the end user experience with big data apps this year, according to Penny Herscher, CEO of FirstRain.
“It is time for Apps 2.0, where apps know just as much about you as you do about them,” Herscher says. “In 2015, users will demand an unprecedented level of personalization in their analytics, and the applications that can supply these personalized analytics will be the ones that businesses seek out.”
Chances are very good you’ll be hearing a lot about in-memory computing in 2015, as well as RAM’s speedy little brother, Flash. “In-memory has turned into an accepted technology, and CIOs and CTOs more and more often see an in-memory layer as crucial within their complex data management ecosystems,” says Aaron Auld, CEO of EXASOL. “Many businesses have held back from diving into the in-memory pool because there has been a lot of confusion around the terminology and offerings. However, the business pressure to stay competitive will ratchet up the adoption of in-memory technologies …”
We’ve been hearing about the importance of real-time analytics for years, but outside of a few use cases, most organizations haven’t found the need to adopt it. That freeze around real-time analytics may start to melt as technologies such as Spark, in-memory computing, and massive IoT data streams combine to present new business opportunities for those able to harness the data and technologies in the appropriate manner.
You can count Monte Zweban, CEO of Splice Machine, as one of the prognosticators firmly in the real-time analytics camp. “New big data applications will emerge that have multiple users reading and writing data concurrently, while data streams in simultaneously from connected systems,” he predicts. “Concurrent applications will overtake batch data science as the most interesting Hadoop use case.”
All types of data analytics will become more sophisticated in 2015. But there will be a special emphasis on mining certain data types, including machine-generated data from computer logs and IoT-connected devices, and unstructured text. “Unstructured data has posed many obstacles in the past, but will come into its own in 2015,” says Eldad Farkash, co-founder and CTO of on-chip BI software developer Sisense. “Text analysis will gain increasing traction, with Web data, documents, and images, with companies finally able to tackle unstructured data in meaningful ways.”
How do you see big data unfolding in 2015? We’d love to hear from the users in the big data trenches out there. Send us your big data predictions at firstname.lastname@example.org.