Apache Hadoop

Framework for distributed storage and processing of large datasets across clusters of computers. Combines a distributed file system for high-throughput data access with a resource manager for job scheduling and a batch processing engine for parallel computation, enabling workloads that exceed single-node capacity.

Trusted by enterprises such as Uber, Walmart, Apple, and Microsoft for research and production data lakes. Serves as the foundation for much of the big data ecosystem, with projects like Hive, Spark, HBase, and Pig built on top of it.

Key capabilities:

• HDFS for fault-tolerant, scalable storage with erasure coding and federation
• YARN for cluster resource management and multi-tenant job scheduling
• MapReduce for batch processing of large datasets in parallel
• Object store and cloud compatibility via connectors
• On-premises or cloud deployment with high-availability options

Engineers use it for data lake storage, batch ETL pipelines, log aggregation, and analytics at petabyte scale. Integrates with SQL engines like Hive for ad hoc querying, streaming frameworks for real-time processing, and machine learning libraries. Often paired with Spark or Tez as alternative execution engines for interactive and low-latency workloads.