Gearpump
Gearpump is a lightweight real-time big data streaming engine. It is inspired by recent advances in the Apache Pekko framework and a desire to improve on existing streaming frameworks.
The name Gearpump is a reference to the engineering term "gear pump", which is a super simple pump that consists of only two gears, but is very powerful at streaming water.
We model streaming within the Pekko actor hierarchy.
Per initial benchmarks we are able to process near 18 million messages/second (100 bytes per message) with a 8ms latency on a 4-node cluster.
For steps to reproduce the performance test, please check Performance benchmark.
Useful Resources
How to Build
1). Clone the Gearpump repository
git clone https://github.com/gearpump/gearpump.git
cd gearpump
2). Build package
## Please use JDK 17 or 21; the build currently uses Scala 2.13.
## The target package path is under output/target/.
sbt clean +assembly 'gearpump-pack / packArchiveZip'
After the build, there will be a package file gearpump-${version}.zip generated under output/target/ folder.
To build scala document, use
## Will generate the scala doc under target/scala_2.xx/unidoc/
sbt unidoc
NOTE:
The build requires network connection. If you are behind an enterprise proxy, make sure you have set the proxy in your env before running the build commands.
For windows:
set HTTP_PROXY=http://host:port
set HTTPS_PROXY= http://host:port
For Linux:
export HTTP_PROXY=http://host:port
export HTTPS_PROXY=http://host:port
Contributing
See the contributing guide for the contribution workflow, tests, and style checks.
How to generate the license report to generate a list of all dependencies
sbt dumpLicenseReportAggregate
Contributors (time order)
License
Gearpump itself is licensed under the Apache License (2.0).
For library it used, please see LICENSE.
Acknowledgement
The netty transport code work is based on Apache Storm. Thanks Apache Storm contributors.
The cgroup code work is based on JStorm. Thanks JStorm contributors.
Thanks to Jetbrains for providing a IntelliJ IDEA Free Open Source License.
