Apache Spark

Unified analytics engine for large-scale data engineering, data science, and machine learning. Runs on single nodes or clusters and supports Python, SQL, Scala, Java, and R through a single API, unifying batch and streaming processing in one framework.

Used by thousands of companies including most Fortune 500 firms. The project has over 2,000 contributors from industry and academia, differentiating it from alternatives like Hadoop MapReduce through in-memory processing and a single engine for ETL, streaming, SQL, and ML.

Core capabilities:

• Batch and streaming data in one engine with Structured Streaming
• Distributed ANSI SQL for analytics and ad-hoc reporting
• MLlib for scalable machine learning from laptop to cluster
• Works with Parquet, Delta Lake, Iceberg, and cloud object storage
• Adaptive Query Execution that optimizes plans at runtime

Teams use it for ETL pipelines, real-time stream processing with Kafka, exploratory data analysis at petabyte scale, and training ML models that scale from development to production. Integrates with Kubernetes and major data platforms for deployment.