Apache Kafka Gains Traction Among Enterprise Users
The embrace of stream processing and real-time data access is driving enterprise adoption of the Apache Kafka distributed messaging system, according to an industry survey released at a Kafka summit this week.
Confluent Inc., the Palo Alto, Calif., company founded by the creators of Apache Kafka, reported that its industry survey revealed that 88 percent of companies polled said they expect to adopt the tool for their data and application infrastructures by 2017. Moreover, nearly one-third of respondents said they work for large companies with annual sales of more than $1 billion.
That, the company claimed, illustrates how open source projects like Kafka are gaining traction in large enterprises as companies seek to leverage the stable platform’s ability to scale while speeding access to more data. Confluent also said Kafka is increasingly being used in production within real-time environments.
“We see more and more organizations embracing real-time data and stream processing, and Kafka is at the heart of that shift,” Jay Kreps, one of Kafka’s co-creators and the CEO and co-founder of Confluent, asserted in a statement announcing the survey results.
The survey also revealed that Kafka is primarily used for stream processing, with nearly three-quarters using it to stream data while 68 percent said they plan to use Kafka for stream processing of data over the next year. Of those, 74 percent say they are adding Kafka to new applications now in development while 69 percent said they are integrating it with existing applications.
Meanwhile, 60 percent said they plan to build new applications using stream processing and Kafka.
Application monitoring (60 percent), data warehousing (51 percent) and “asynchronous applications” (47 percent) were the leading Kafka use cases. Other deployments included system monitoring (39 percent), recommendation engines (35 percent), security/fraud detection (26 percent), Internet of Things applications (20 percent) and dynamic pricing applications (12 percent), according to the Confluent survey.
More than two-thirds of users polled agreed that Kafka made applications “work together in a loosely coupled manner”
while 59 percent said the used it “as [an] underlying data infrastructure for stream processing.” Other attributes cited by users included improved scaling of applications (58 percent) and higher data volumes being made available in the real time, enabling users to move beyond batch processing.
As Kafka’s creators seek to make it a “core infrastructure technology” for stream processing of data, they noted the number of use cases is expanding to include the collection of user activity data, logs, application metrics, stock ticker data and device instrumentation. Kafka is touted as making high-volume data available as a real-time stream so that processing engines can crunch data as it arrives.
The Confluent-sponsored Kafka user survey was conducted earlier this month and includes responses from more than 100 users from 20 countries and across 16 industries, including banking and financial services, telecommunications, energy and utilities, consumer products and manufacturing along with the automotive sector.