NetApp Spots a Data Platform Opportunity in the Cloud

(amgun/Shutterstock)

The market for spot instances in the cloud is, well, spotty. Some days you can count on a 90% discount by buying excess capacity from the public clouds, and other times you can’t. NetApp has turned that cloud unpredictability into a business opportunity with Spot, which it acquired two years ago. And with its acquisition of DataMechanics last year and recent acquisition of Instaclustr, NetApp has a clear path to delivering a full data platform atop the spot market.

Spot built a business turning spare compute and storage capacity on AWS, Microsoft Azure, and Google Cloud into a more reliable service. By serving as the middleman between the erratic spot market and customers, Spot was able to deliver reliable compute and storage capacity to customers while meeting stringent SLAs.

Spot, which was bought by NetApp in June 2020 for $450 million, offers three main product famlies, including Ocean, Elastigroup, and Eco. Ocean is a Kubernetes offerings that runs atop spot instances on AWS and spot VMs on Azure and Google Cloud, while Elastigroup is used for deploying virtual machines (VMs) without containers on the same spot instances and spot VMs. Eco, meanwhile, provides a way to exploit reserved instances (RIs) on the public cloud.

Customers can save 80% to 90% on their cloud compute bill with Amazon EC2 Spot Instances

Ocean and Elastigroup ostensibly can save customers 80% to 90% of the cost of compute and storage, according to the company, while Eco can save customers 40% to 50%. Customers don’t quite save that much by having Spot manage and orchestrate the workloads across spot instances and RIs, according to Spot CTO Kevin McGrath, who says Spot makes money by charging a fraction of the savings they’re getting.

If there are no savings being had because spot instances or spot VMs and RIs are not available, then Spot customers don’t pay anything, McGrath says. Whichever option customers choose, they are apt to save money by utilizing the spot market, he says.

“What we’re doing is we’re taking the excess capacity that exists from kind of the natural build out of their data centers,” McGrath says. “The caveat is that they can pull this compute back at any given time. So a lot of people are wary to use it because they don’t know when the computer’s going to be pulled or how to orchestrate an application on top of that.”

The early days of Spot were focused primarily on delivering these compute and storage savings in the cloud. But today under NetApp, Spot’s focus has expanded to how it can help businesses consume a variety of enterprise compute services in a platform-centric manner, McGrath says.

“So not only cost savings, not only Spot instances through the clouds, but how do you manage reserve capacity in the clouds? How do you manage things like containers and storage and networking and all the different things that go into a cloud application?” he tells Datanami in a recent interview. “That’s what has NetApp really excited, is building a platform for cloud that makes you successful in delivering applications and using the data across those applications in the cloud.”

Spot’s Ocean service delivers Kubernetes atop spot instances on AWS, GC, and Azure

You can run arbitrary applications on Spot by NetApp (as the service is officially called), but in reality, the biggest benefits come from running stateless applications that need to scale up and down across a very wide spectrum. Machine learning training and big data analytics workloads fit this bill, transactional applications that require stateful connections less so.

The company has done something special for Apache Spark. For Spark workloads, the company has not only optimized the runtime for Spark, but it’s made tweaks to the Spark distribution to better utilize the compute that Spot exposes. It’s Ocean for Apache Spark service (see graphic) is a result of June 2021 acquisition of DataMechanics by NetApp.

“So vanilla Spark, we have customers running that today. It works great,” McGrath says. “But we can optimize it even further by optimizing at the task level of Spark and then pairing that so it understands how it’s going to run and what infrastructure it’s going to run. You automatically put all that together, you get magic.”

The company is hoping for more magic with its latest acquisition of Instaclustr, which was completed in May. Instaclustr delivers managed services based on a collection of open source data management tools, such as Apache Kafka, Apache Cassandra, PostgreSQL, Redis, Spark, OpenSearch, and Cadence.

The hope is the acquisition of Instaclustr will bring all of those managed services within the Spot family and give customers something that equals more than the sum of the parts, according to McGrath.

“When you put this together, this is the vision of where we’re headed as a platform,” he says. “So it’s not just compute and it’s not just storage. But creating a platform that application developers as they need requirements from the organization, the DevOps team can step in and say ‘Well, we use the Spot by NetApp platform, and because we do that, we can provide you with the databases that you need, we can provide you with the data processing that you need and…you don’t even have to know about it, but we’re optimizing all of the containers and the compute and the networking and the observability that make that magic happen for you to deliver your applications.’”

The plan is to deliver savings for customers with stateful applications too, thanks to the ONTAP functionality developed by NetApp. Since NetApp devices are resident in the data centers of all three major public cloud providers, the pieces are in place to deliver a more integrated hybrid storage experience for customers, McGrath says.

“We can interface with ONTAP through cloud providers,” he says. “We can interface with the cloud providers native storage. We can interface with any storage type, and pull it in as mangemnet into our platform.”

If the thought building enterprise data applications on ephemeral spot instances and RIs has you a little concerned, McGrath says not to worry. The big public cloud providers are building so many data centers that excess capacity should be plenty available for the foreseeable future.

“If they stop building data centers, then we have a problem,” he says. “I don’t think we’ll be there for the next 10 years.”

Related Items:

Open Source is Now a Big Data Service

Cloud Getting Expensive? That’s By Design, But Don’t Blame the Clouds

Cassandra, Kafka Help Scale Anomaly Detection