Apache Airflow

Open-source platform for developing, scheduling, and monitoring batch-oriented workflows. Workflows are defined as Python code, making them maintainable, versionable, and testable. A web UI helps visualize pipelines in production, inspect logs, and troubleshoot issues when tasks fail.

Apache Airflow is trusted by enterprises like Snap, Bosch, and Coinbase for data orchestration at scale. It differentiates from cron-based or XML-driven alternatives by offering workflows as code, a rich ecosystem of providers for cloud and third-party services, and backfill support for historical data. Used by tens of thousands of organizations, it integrates with Apache Kafka, Spark, and most major cloud platforms.

Key features:

• Workflows as Python code with Jinja templating for parameterization
• Rich provider ecosystem for AWS, GCP, Azure, and hundreds of integrations
• Scalable architecture with message queue and distributed workers
• Web UI for monitoring, manual triggers, and task-level debugging
• Backfill and rerun semantics for failed tasks or historical runs

Data engineers rely on Airflow for ETL pipelines, data warehouse syncs, ML training schedules, and batch analytics. It complements streaming systems by orchestrating periodic batch jobs that process data written to storage. Developers use version control for DAGs and run unit tests on pipeline logic before deployment.