Follow Datanami:
July 7, 2020

Google Cloud’s Dataproc Gets a GPU-Powered Spark Boost


Google Cloud’s Dataproc – its big data platform that allows users to run Apache Hadoop and Spark jobs – is getting a boost. Apache Spark 3 and Hadoop 3 have launched general availability, enhancing users’ data analytics capabilities with a series of new features – and naturally, those features are now available on Google Cloud’s Dataproc image version 2.0.

In a blog post, Christopher Crosbie (product manager for Google Cloud) and Igor Dvorzhak (a software engineer at Google) highlighted the new features offered in the Apache Spark 3 implementation.

Adaptive queries: Spark can now optimize a query plan while execution is occuring. This will be a big gain for data lake queries that often lack proper statistics in advance of the query processing. 

Dynamic partition pruning: Avoiding unnecessary data scans are critical in queries that resemble data warehouse queries, which use a single fact table and many dimension tables. Spark 3 brings this data pruning technique to Spark.

GPU acceleration: NVIDIA has been collaborating with the open source community to bring GPUs into Spark’s native processing. This allows Spark to hand off processing to GPUs where appropriate. 

The headline of these features, the Googlers say, is simple: performance. “As data scientists shift from using traditional analytics to AI applications that better model complex market demands, CPU-based processing can’t keep up without compromising either speed or cost,”  wrote Erik Pounds, director of product marketing for Nvidia, in his own blog post. “The growing adoption of AI in analytics has created the need for a new framework to process data quickly and cost-efficiently with GPUs.”

The blog post also explains that there will be improvements to Spark on Kubernetes, as well as some deprecated features: MLLib, GraphX, DataSource API, and Python 2.7 are no longer supported in Spark 3, though replacements like SparkGraph, DataSource V2, and Python 3 are available.

Hadoop 3 is also now available in Dataproc image version 2.0, enabling features like native GPU support in the YARN schedule and YARN containerization. Finally, the new version of Dataproc includes some non-Apache upgrades, such as support for new software libraries, support for new shared libraries (and upgrades to existing shared libraries), and various optimizations.

According to Google, getting started with Spark 3 and Hadoop 3 in Dataproc image version 2.0 is as simple as entering a few lines of code. To read more about the new features and see how to work with the new capabilities yourself, click here.