Follow Datanami:
July 30, 2014

Big Data Survey Finds Growing Need for CEP

A global survey of software developers working on big data and advanced analytics projects found that the large majority of respondents require real-time, complex event processing for their applications.

The survey, released July 29 by Evans Data Corp. of Santa Cruz, Calif., found that 71 percent of respondents said they require advanced processing more than half the time.

“Hadoop’s batch processing model has worked well for several years,” Evans Data CEO Janel Garvin noted in a statement releasing the survey. “But the demands and challenges of big data in our current world means real-time event processing is becoming more necessary.”

Garvin said the growing requirement explains why Apache Storm and other commercial complex events processing (CEP) products are being embraced by developers working on big data projects.

Other analysts note that CEP has evolved to better utilize memory data grids for analyzing trends, patterns and events in real-time. Assessments can now be delivered “in a matter of milliseconds,” according to a 2013 assessment by researchers at Infosys Labs.

“Event clouds, a byproduct of using CEP techniques, can be further leveraged to monitor unforeseen conditions [emerging], or even the emergence of an unknown-unknown,” the Infosys analysis noted. That in turn could provide a “potential first mover advantage” for an agile enterprise, Infosys added.

CEP products could also prove useful in other applications requiring huge amounts of data crunching, Infosys said. These include predictive analytics used in weather forecasting, health care and high frequency trading.

The Evans Data survey also highlighted the growing need for storage in pursuing advanced analytics. The researcher found that over one-third of the developers it surveyed said the overall size of their company’s data stores would grow by more than 75 percent over the next year. Nearly 15 percent said storage requirements would more than double.

Meanwhile, large- and medium sized companies handling more big data projects are increasingly turning to database sharding as a way to cope with growing data volumes. Sharding is also seen as valuable for distributed computing applications. By contrast, the Evans survey found that smaller companies with less than 100 employees have the least amount of experience with database shards. Forty percent of small companies reported they had no experience using the technique.

Drilling down, the survey revealed that 28 percent of medium-sized companies (up to 1,000 employees) surveyed currently use database shards while 20 percent of large companies with over 1,000 employees used the database technique.

Between 25 and 30 percent of medium-size and large enterprises said they expect to adopt database sharding in the next six months, the survey reported.

The big data and analytics survey is part of a series focusing on tools and methodologies for storing, managing and analyzing large datasets and databases from a variety of sources, Evans Data said.

Related items:

Capital Markets Push CEP to the Limit

Go Shard Thyself, CodeFuture Says

 

Datanami