Cloud Dataproc

Managed service for running Apache Spark and Hadoop clusters in the cloud. Provides a unified control plane for batch processing, interactive SQL, stream processing, and machine learning without the operational burden of self-managed clusters.

Google Dataproc appeals to teams migrating from on-premises Hadoop or self-managed Spark, or those building modern lakehouse architectures. It supports a broad ecosystem of 30+ open source components including Spark, Hadoop, Flink, Trino, and Presto, so engineers can avoid vendor lock-in while benefiting from managed scaling, initialization actions, and integration with BigQuery and Vertex AI.

Key capabilities:

• Managed Spark and Hadoop clusters with autoscaling and preemptible VMs for cost-aware scaling
• Optional components including Flink for stream processing, Trino or Presto for interactive SQL, all under a single service
• Native integration with BigQuery, Cloud Storage, and Dataplex Universal Catalog for lakehouse workflows
• GPU support and pre-configured ML runtimes for model training and batch inference, with Vertex AI for MLOps
• Customization via machine types, initialization actions, and bring-your-own container images
• Enterprise security with IAM, VPC Service Controls, and Kerberos support

Common uses include lift-and-shift migration of existing Hadoop and Spark workloads, lakehouse modernization with open formats like Apache Iceberg, and long-running ETL pipelines orchestrated via workflow templates or Airflow. Data science teams run large-scale model training and batch inference on customizable clusters. Engineers also deploy dedicated Trino or Flink clusters for interactive SQL or streaming workloads without adding operational complexity.