Apache Flink

Framework for stateful stream and batch processing, handling both unbounded real-time streams and bounded recorded datasets. Provides exactly-once state consistency, event-time semantics, and low-latency processing with the ability to maintain terabytes of application state at scale.

Apache Flink powers demanding stream processing workloads at companies like Uber, Zalando, and Alibaba. Unlike micro-batch systems, it processes events as they arrive for true streaming semantics. Unified SQL and DataStream APIs run the same queries on both live streams and historical data, producing identical results. Supports Kubernetes, YARN, and standalone clusters with high-availability configurations.

Key capabilities:

• Exactly-once state consistency with incremental checkpoints
• Event-time processing with watermarks and late-data handling
• Layered APIs: SQL/Table API, DataStream API, ProcessFunction for fine-grained control
• Pluggable state backends in memory or RocksDB for large state
• Savepoints for zero-downtime upgrades and A/B testing

Event-driven applications that react to incoming streams with stateful logic: fraud detection, anomaly detection, and business process monitoring. Continuous analytics on live data with ANSI SQL. Data pipelines and ETL that move and transform data between storage systems in real time. Integrates with Kafka, Kinesis, Elasticsearch, and JDBC; teams use it for real-time search index building, quality monitoring of telecom networks, and ad-hoc analysis of live consumer data.