Confluent Adds KSQL Support to Kafka Platform
The latest version of Confluent’s Kafka-based platform incorporates an open source streaming engine for Apache Kafka designed to allow developers using SQL to build real-time, streaming applications.
Confluent, the company behind open source Kafka and developer of the Confluent Platform, announced the general availability of its KSQL streaming engine on Wednesday (March 7). The engine designed to expand access to real-time data streams and is targeted at production workloads, the company added.
Confluent, Palo Alto, Calif., also unveiled additional upgrades to its flagship platform, including new Kafka APIs for verifying the accuracy of streaming applications. Also included is a new Java Message Service connector designed to ease data migration from ActiveMQ, the open source message broker, and IBM MQ messaging middleware to Apache Kafka.
According to a recent survey, Confluent reported that 81 percent of Kafka customers are using it to build data pipelines. Other common use cases include real-time monitoring, ETL, micro-services and building Internet of Things applications.
The company said this week that early KSQL deployments included real-time anomaly detection while responding to streaming data like weather and traffic updates. Another was the creation of streaming ETL platforms for delivering data to business analysts in real time.
With support on the latest version of its streaming platform, Confluent said KSQL can be used to process streaming data using SQL-like semantics in production workloads. The goal is to implement stream processing in production via KSQL, said Neha Narkhede, Confluent’s co-founder, CTO and product chief.
In review by developers since its launch last August, the SQL engine is positioned as a way to run continuous interactive SQL queries on streaming data in Kafka.
According to a company blog post, sets of KSQL processes run as a cluster. Additional processing capacity can be added by launching more instances of the KSQL server. Queries can be launched using the interactive KSQL command line client which sends commands to the cluster over a REST API. Available streams and tables can then be inspected, new queries initiated along with checking the status of running queries.
“Turning the database inside out with Kafka and KSQL has a big impact on what is now possible with all the data in a company that can naturally be represented and processed in a streaming fashion,” the company asserts.
Confluent said its version 4.1 platform will be available in early April. The new version also includes a “multi-datacenter replication” feature that allows users to replicate data stored on-premises to the cloud along with deployment and configuration tools to ease disaster recovery.
The platform with KSQL can be downloaded here in early April.