Apache Iceberg

Open table format for large-scale analytic workloads that brings SQL table reliability and simplicity to data lakes. Enables multiple compute engines to read and write the same tables concurrently while providing schema evolution, time travel, and transactional guarantees even on cloud object storage.

Apache Iceberg was developed at Netflix to address correctness and scalability issues in big-data tables. It has an open specification and reference implementation in Java, with additional libraries in Python, Go, Rust, and C++. Engine integrations include Spark, Trino, Flink, Presto, Hive and Impala, so teams can query tables from different tools without copying data.

Core capabilities:

• Expressive SQL with merge, update and targeted deletes, plus configurable compaction strategies
• Schema evolution without table rewrites: add, drop, rename or reorder columns with no side effects
• Hidden partitioning that derives partition values automatically and skips irrelevant files for faster queries
• Time travel and version rollback for reproducible queries and easy recovery from bad writes
• Serializable isolation and optimistic concurrency for multi-writer workloads on object storage

Engineers use Iceberg to modernize data lake tables, unify analytics across Spark and Trino, run CDC and streaming ingestion, and build reproducible ML pipelines. It fits well with existing Parquet or ORC data and integrates with catalog systems like Hive metastore or standalone catalogs.