Follow Datanami:
May 18, 2020

Running Kafka on Kubernetes the Easy Way

Kubernetes is one of the most popular open source container orchestrators and management APIs. It’s one of the emerging platforms that enables companies to run and manage containerized applications globally.  Built to automate deploying, scaling and operating application containers, cloud-native support from AWS, GCP, Azure, it has a growing enterprise support ecosystem. Leveraging Kubernetes to provide tested, repeatable deployment patterns that follow best practices is a win for both developers and the operators.

Apache Kafka is a very popular choice for messaging and event streaming in modern Cloud Native applications. It’s the leading distributed streaming and queuing technology for large-scale, always-on applications. Kafka has built-in features of horizontal scalability, high-throughput and low-latency. It is highly reliable, has high-availability, and allows geographically distributed data streams and stream processing applications. Kafka is traditionally well suited to run on statically defined clusters but running it on container orchestrated clusters like Kubernetes is becoming more common. While Kubernetes is a great choice for running stateless applications, running stateful applications such as Kafka on Kubernetes can have its own unique challenges.

This talk will review different approaches for running stateful applications like Kafka on Kubernetes and then explore the popular approach of deploying Kafka on Kubernetes using operators. In the second half, we will run a demo on how to deploy a production-ready Kafka cluster in Kubernetes as well as how to deploy Prometheus in Kubernetes to monitor a cluster. By the end of the talk, you will have enough knowledge to deploy Kafka on Kubernetes.

Check out the talk here.

Speaker: Mussa Shirazi, Senior Consultant at Instaclustr

Bio: With over 13 years of IT industry experience, Mussa specializes in the field of Big Data, Low Latency Monitoring, Network and Security Technologies, Network Solutions. He’s worked with large multinational corporations around the globe helping to resolve their data and IT solutions issues.