Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
You need to have Java installed.
We build and test Apache Kafka with Java versions 17 and 25. The release parameter in javac is set to 11 for the clients
and streams modules, and 17 for the rest, ensuring compatibility with their respective
minimum Java versions. Similarly, the release parameter in scalac is set to 11 for the streams modules and 17
for the rest.
Scala 2.13 is the only supported version in Apache Kafka.
Build a JAR and run it
./gradlew jar
Follow instructions in https://kafka.apache.org/quickstart
Build source JAR
./gradlew srcJar
Build aggregated javadoc
./gradlew aggregatedJavadoc --no-parallel
Build javadoc and scaladoc
./gradlew javadoc
./gradlew javadocJar # builds a javadoc jar for each module
./gradlew scaladoc
./gradlew scaladocJar # builds a scaladoc jar for each module
./gradlew docsJar # builds both (if applicable) javadoc and scaladoc jars for each module
Run unit/integration tests
./gradlew test # runs both unit and integration tests
./gradlew unitTest
./gradlew integrationTest
./gradlew test -Pkafka.test.run.flaky=true # runs tests that are marked as flaky
Force re-running tests without code change
./gradlew test --rerun-tasks
./gradlew unitTest --rerun-tasks
./gradlew integrationTest --rerun-tasks
Running a particular unit/integration test
./gradlew clients:test --tests RequestResponseTest
./gradlew streams:integration-tests:test --tests RestoreIntegrationTest
Running a particular unit/integration test N times
N=500; I=0; while [ $I -lt $N ] && ./gradlew clients:test --tests RequestResponseTest --rerun --fail-fast; do (( I=$I+1 )); echo "Completed run: $I"; sleep 1; done
Running a particular test method within a unit/integration test
./gradlew clients:test --tests org.apache.kafka.clients.MetadataTest.testTimeToNextUpdate
./gradlew clients:clients-integration-tests:test --tests org.apache.kafka.clients.producer.ProducerFailureHandlingTest.testCannotSendToInternalTopic
./gradlew streams:integration-tests:test --tests org.apache.kafka.streams.integration.RestoreIntegrationTest.shouldRestoreNullRecord
Running a particular unit/integration test with log4j output
By default, there will be only a small number of logs output while testing. You can adjust it by changing the log4j2.yaml file in the module's src/test/resources directory.
For example, if you want to see more logs for clients project tests, you can modify the line in clients/src/test/resources/log4j2.yaml
to level: INFO and then run:
./gradlew cleanTest clients:test --tests NetworkClientTest
And you should see INFO level logs in the file under the clients/build/test-results/test directory.
Specifying test retries
Retries are disabled by default, but you can set maxTestRetryFailures and maxTestRetries to enable retries.
The following example declares -PmaxTestRetries=1 and -PmaxTestRetryFailures=3 to enable a failed test to be retried once, with a total retry limit of 3.
./gradlew test -PmaxTestRetries=1 -PmaxTestRetryFailures=3
See Test Retry Gradle Plugin and build.yml for more details.
Generating test coverage reports
Generate coverage reports for the whole project:
./gradlew reportCoverage -PenableTestCoverage=true -Dorg.gradle.parallel=false
Generate coverage for a single module, i.e.:
./gradlew clients:reportCoverage -PenableTestCoverage=true -Dorg.gradle.parallel=false
Coverage reports are located within the module's build directory, categorized by module type:
Core Module (:core): core/build/reports/scoverageTest/index.html
Other Modules: <module>/build/reports/jacoco/test/html/index.html
Building a binary release gzipped tarball
./gradlew clean releaseTarGz
The release file can be found inside ./core/build/distributions/.
Building auto-generated messages
Sometimes it is only necessary to rebuild the RPC auto-generated message data when switching between branches, as they could
fail due to code changes. You can just run:
./gradlew processMessages processTestMessages
See Apache Kafka Message Definitions for details on Apache Kafka message protocol.
Running a Kafka broker
Using compiled files:
KAFKA_CLUSTER_ID="$(./bin/kafka-storage.sh random-uuid)"
./bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties
./bin/kafka-server-start.sh config/server.properties
Using docker image:
docker run -p 9092:9092 apache/kafka:latest
See docker/README.md for detailed information.
Cleaning the build
./gradlew clean
Running a task for a specific project
This is for core, examples and clients
./gradlew core:jar
./gradlew core:test
Streams has multiple sub-projects, but you can run all the tests:
./gradlew :streams:testAll
Listing all gradle tasks
./gradlew tasks
This is the beginning of the README. Read the rest at the original. Read the full README ↗
