Confluent Moves Kafka to the Cloud
As adoption of the Apache Kafka accelerates, the company behind the data pipeline is rolling a cloud service designed to boost “developer velocity” while IT operations are handled in the background.
Confluent said Monday (May 8) during a Kafka confab in New York that its new cloud service that manages Kafka clusters aims to give developers the option of building and running Kafka data streaming applications in the public cloud. The company, Palo Alto, Calif., touts its cloud platform as a fleshed-out distribution of Kafka designed to run in enterprise-scale production environments.
Among the new features are access to the same open source Kafka APIs most developers have already been using to access connectors and tools supported by the open source community. At the same time, the company pitches the service as the easiest way to run Apache Kafka in the public cloud.
The introduction follows the release last week of a survey that found organizations are using Kafka for stream processing, data integration, messaging, and log aggregation. These Kafka-based systems support a variety of downstream applications, like data warehousing, application and system monitoring and recommendation engines, along with security and fraud detection, the survey found.
These and other data points underscore what the company argues is an emerging consensus among users for running Kafka as a company’s “central nervous system” as well as a basis for stream processing without worrying about IT backend.
“What we’re seeing in the world…is the emergence of this whole new category of infrastructure around streaming,” Confluent CEO Jay Kreps recently told Datanami.
“An increasing number of companies are either starting entirely in or making a shift to the public cloud and want to use Kafka as a hosted service in the cloud,” Neha Narkhede, Confluent’s co-founder and head of engineering, added in a blog post.
“We believe that all developers should be able to spin up a Kafka-based streaming platform as quickly as they spin up compute,” Narkhede added.
Narkhede along with co-founders Kreps and Jun Rao created Apache Kafka while at LinkedIn, where it ran internally as a managed service used there by several thousands developers. Confluent is betting that rapid adoption of Kafka will translate into demand for a platform that runs applications in the cloud without worrying about managing failures, tuning for performance and scaling resources up or down.
The company also stresses that its platform enables scaling without vendor lock-in, allowing developers to migrate applications to other cloud providers without changing code.
Confluent said the Kafka cloud service would initially be available in Amazon Web Services (NASDAQ: AMZN), with support for Microsoft Azure (NASDAQ: MSFT) and Google Cloud (NASDAQ: GOOGL) to be added later. It is also available via an early access program. Pricing is determined based on throughput, retention and availability, the company added.