Kubernetes Adoption Widespread for Big Data, But Monitoring and Tuning Are Issues, Survey Finds

(Image source: Pepperdata)

Kubernetes may be a complex piece of software that can be difficult to monitor and manage. But the benefits of running applications in the popular container orchestration system appear to outweigh the disadvantages, because the migration of big data apps to Kubernetes is widespread, according to a new survey from Pepperdata.

In a report on the state of big data on Kubernetes released this week, Pepperdata found that more than half of the folks it surveyed say they are “moving big data applications to Kubernetes to reduce their overall spend.”

By the end of 2021, 77% of the survey respondents say they expect to have migrated 50% or more of their big data applications to Kubernetes, with about 10% saying they will have all of their applications migrated, about 28% saying they will have three-quarters of their apps moved, and 38% saying half will be on the open source orchestration layer.

“The velocity of the shift to Kubernetes is surprising,” said Pepperdata CEO Ash Munshi in a press release. “It’s gratifying to see big data move to Kubernetes with this speed.”

Apache Spark was the most popular big data framework that companies are running atop K8s, followed by Presto, Kafka, Trino, and Flink, the survey found. “Running Spark on Kubernetes offers simpler administration, easier dependency management, and more flexible deployment,” Pepperdata says in its report.

Increasing application performance and stability were by far the number one goals for moving apps to K8s, with more than 45% of survey respondents choosing this option. The second most popular reason was to gain more flexibility and portability of workloads (cited by 27% of survey respondents), followed by saving money (17%) and avoiding cloud lock-in by leveraging a multi-cloud solution (10%).

(Image source: Pepperdata)

While migrating apps to Kubernetes can result in better uptime, scalability, and vendor agnosticism, not all applications can or will be migrated, Pepperdata says. “It’s crucial for enterprises to know which apps to migrate to Kubernetes and which apps should remain on legacy platforms,” the company says in its report.

Pepperdata also devoted a portion of the report to monitoring of Kubernetes environments. While Kubernetes simplifies the management of the containers that applications run in, it adds another layer of complexity when it comes to monitoring.

About 43% of survey respondents use a cloud vendor solution to monitor their K8s environment, followed by the combination of Prometheus and Grafana at 18%, an application performance monitoring (APM) solution for 13%, and just Prometheus (sans Grafana) for 12%. About 15% say they are looking for a monitoring solution for their K8s environment.

Pepperdata found that “allocation and reallocation of resources” was the biggest challenge that customers currently experience with containers and K8s, cited by about 30% of survey respondents. That was followed by a lack of comprehensive visibility and monitoring tools (26%), a fragmented Kubernetes marketplace (20%), complexity and scaling (15%), and the difficulty in manually tuning Kubernetes (9%).

More than half of the survey respondents “don’t know which metrics to focus on,” the company said.

“It’s difficult to optimize resource allocation and perfectly tune Kubernetes for your big data applications,” Pepperdata says in its report. “Tuning your clusters manually is not efficient and does not scale. There are a large number of applications you can use to monitor your K8s implementation. However, most look into only the most basic metrics. Combining multiple tools for a more complete view isn’t practical. A single tool to manage resource allocation and application performance provides the best solution.”

Pepperdata, which develops tools for automating the optimization of big data environments, including those running in Kubernetes, sees for itself a roll in helping customers to adapt their systems to the brave new world of big data apps in containers.

“It’s clear that this shift will create optimization challenges as customers understand the nuances of running within the Kubernetes environment,” Munshi continued in the press release. “Pepperdata is uniquely positioned to help customers moving to Kubernetes maximize resource utilization while controlling costs.”

The company’s survey was conducted in October and included 600 participants.

Related Items:

Who’s Winning In the $17B AIOps and Observability Market

AWS Launches Managed Services for Grafana, Prometheus

AIOps and the Kubernetes Revolution