Confluent's Python Client for Apache Kafka®
[!WARNING]
4.x is coming. 4.x will be a breaking change once released for production (GA). We recommend upgrading to 4.x when you can. If you aren't ready to move yet, pin to 2.x, or you will upgrade to it automatically. Use confluent-kafka~=2.16 or confluent-kafka>=2.16,<3. See the preview note below and Python packaging: versioning.
[!NOTE]
Preview: confluent-kafka 4.4
A next-generation Python client is available in preview as confluent-kafka 4.4.0b1. It is built on a native Rust core and mirrors the Apache Kafka Java client API.
- Not for production yet. The API may change before GA.
- New API. 4.x is not compatible with 2.x code.
- Opt-in.
pip install confluent-kafka still installs 2.x. To try the preview: pip install confluent-kafka==4.4.0b1.
- Why 4.4? Versions now track Apache Kafka (4.4 = Apache Kafka 4.4), so there is no 3.x. See the versioning RFC.
- Why a new repository? The Rust core and every language binding live in one monorepo, confluentinc/kafka-clients, so one change can ship across all languages. Python development for 4.x happens there. See the repository structure RFC.
Read the Python API RFC and share feedback in confluentinc/kafka-clients.
confluent-kafka-python provides a high-level Producer, Consumer and AdminClient compatible with all Apache Kafka brokers >= v0.8, Confluent Cloud and Confluent Platform.
Recommended for Production: While this client works with any Kafka deployment, it's optimized for and fully supported with Confluent Cloud (fully managed) and Confluent Platform (self-managed), which provide enterprise-grade security, monitoring, and support.
Why Choose Confluent's Python Client?
Unlike the basic Apache Kafka Python client, confluent-kafka-python provides:
- Production-Ready Performance: Built on
librdkafka (C library) for maximum throughput and minimal latency, significantly outperforming pure Python implementations.
- Enterprise Features: Schema Registry integration, transactions, exactly-once semantics, and advanced serialization support out of the box.
- AsyncIO Support: Native async/await support for modern Python applications - not available in the Apache Kafka client.
- Comprehensive Serialization: Built-in Avro, Protobuf, and JSON Schema support with automatic schema evolution handling.
- Professional Support: Backed by Confluent's engineering team with enterprise SLAs and 24/7 support options.
- Active Development: Continuously updated with the latest Kafka features and performance optimizations.
- Battle-Tested: Used by thousands of organizations in production, from startups to Fortune 500 companies.
Performance Note: The Apache Kafka Python client (kafka-python) is a pure Python implementation that, while functional, has significant performance limitations for high-throughput production use cases. confluent-kafka-python leverages the same high-performance C library (librdkafka) used by Confluent's other clients, providing enterprise-grade performance and reliability.
Key Features
- High Performance & Reliability: Built on
librdkafka, the battle-tested C client for Apache Kafka, ensuring maximum throughput, low latency, and stability. The client is supported by Confluent and is trusted in mission-critical production environments.
- Comprehensive Kafka Support: Full support for the Kafka protocol, transactions, and administration APIs.
- Queues for Kafka (Preview): A
ShareConsumer (KIP-932) for queue-like, cooperative consumption with per-record acknowledgement. See the Share Consumer guide. Preview — not recommended for production.
- AsyncIO Producer: A fully asynchronous producer (
AIOProducer) for seamless integration with modern Python applications using asyncio.
- Seamless Schema Registry Integration: Synchronous and asynchronous clients for Confluent Schema Registry to handle schema management and serialization (Avro, Protobuf, JSON Schema).
- Improved Error Handling: Detailed, context-aware error messages and exceptions to speed up debugging and troubleshooting.
- [Confluent Cloud] Automatic Zone Detection: Producers automatically connect to brokers in the same availability zone, reducing latency and data transfer costs without requiring manual configuration.
- [Confluent Cloud] Simplified Configuration Profiles: Pre-defined configuration profiles optimized for common use cases like high throughput or low latency, simplifying client setup.
- Enterprise Support: Backed by Confluent's expert support team with SLAs and 24/7 assistance for production deployments.
Usage
For a step-by-step guide on using the client, see Getting Started with Apache Kafka and Python.
This is the beginning of the README. Read the rest at the original. Read the full README ↗
