Apache Kafka

Distributed event streaming platform that captures, stores, and processes streams of events in real time. Designed for high-throughput data pipelines, streaming analytics, and mission-critical applications, it combines publish-subscribe messaging with durable log storage and built-in stream processing in a single system.

Trusted by most Fortune 100 companies and thousands of organizations across finance, manufacturing, telecommunications, and logistics. Kafka differs from traditional message queues by treating streams as durable logs: events persist and can be replayed, enabling event sourcing, stream processing with joins and aggregations, and zero message loss with exactly-once semantics. One of the most active Apache projects, with hundreds of community connectors for databases, cloud storage, and legacy systems.

Key capabilities:

• High throughput at network-limited speeds with latencies as low as 2ms
• Scales to thousands of brokers, trillions of events per day, petabytes of data
• Durable, fault-tolerant storage with configurable retention and replication across availability zones
• Kafka Connect with hundreds of pre-built source and sink connectors for Postgres, Elasticsearch, cloud storage, and more
• Built-in stream processing with Kafka Streams: joins, aggregations, windowing, and event-time semantics

Powers event-driven architectures, real-time data pipelines, and microservices communication. Common applications include capturing and reacting to customer interactions, ingesting IoT sensor data from factories and equipment, processing payments and financial transactions at stock exchanges and banks, and unifying data across divisions into a central event log. Developers integrate via client libraries for Java, Go, Python, and many other languages, plus Kafka Connect for existing databases and systems.