GCP Connectors for Apache Flink
Connectors for using Google Cloud services with Apache Flink:
BigQuery, Cloud Pub/Sub, Cloud Tasks, Bigtable, Spanner and Firestore.
Status: released. Artifacts are on
Maven Central under the
io.github.flink-gcp namespace, in two version lines per release: 1.2.0 for the supported
Flink 2.x range and 1.2.0-1.20 for the Flink 1.20 LTS. The SQL uber-jars are also attached
to the GitHub releases.
Modules
| Module | Description |
|---|
flink-connector-gcp-bigquery | BigQuery source and sink for DataStream and Table API / SQL: bounded table and query reads over the Storage Read API; a unified write API over the Storage Write API (at-least-once / exactly-once) and GCS-staged load jobs, with dynamic per-record table destinations and native protobuf serialization. Experimental CDC sink supports upserts and deletes on the at-least-once default stream |
flink-sql-connector-gcp-bigquery | The BigQuery connector as a single relocated uber-jar, for dropping into Flink's lib/ |
flink-connector-gcp-pubsub | Cloud Pub/Sub sink (dynamic topic destinations) and source, with Table API / SQL sink and source support |
flink-sql-connector-gcp-pubsub | The Pub/Sub connector as a single relocated uber-jar, for dropping into Flink's lib/ |
flink-connector-gcp-cloudtasks | Cloud Tasks sink, with a Table API / SQL sink |
flink-sql-connector-gcp-cloudtasks | The Cloud Tasks connector as a single relocated uber-jar, for dropping into Flink's lib/ |
flink-connector-gcp-bigtable | Bigtable sink, bounded scan source, and Change Streams source, with Table API / SQL sink, scan, lookup, and change-stream CDC support |
flink-sql-connector-gcp-bigtable | The Bigtable connector as a single relocated uber-jar, for dropping into Flink's lib/ |
flink-connector-gcp-spanner | Spanner sink, bounded source and Change Streams source (both dialects), with Table API / SQL sink, scan, lookup, and change-stream CDC support |
flink-sql-connector-gcp-spanner | The Spanner connector as a single relocated uber-jar, for dropping into Flink's lib/ |
flink-connector-gcp-firestore | Firestore sink and bounded source for Native-mode databases: writes through the client library's BulkWriter, and reads a collection group or one query at a single snapshot time. A second source and sink read Datastore-mode databases in key ranges at a single snapshot time and write them in non-transactional commits. A Table API / SQL source and sink scan, look up and write Native-mode collections, and a second SQL source and sink scan and write Datastore-mode kinds |
flink-sql-connector-gcp-firestore | The Firestore connector as a single relocated uber-jar, for dropping into Flink's lib/ |
Build
Requires JDK 17 or 21 and Maven (or use the included wrapper):
./mvnw verify
CI runs the same build as just verify. The toolchain and the day-to-day recipes are covered
by Development on the
documentation site.
Supported versions
| Supported |
|---|
| Apache Flink | 2.2, 2.3, and 1.20 (LTS) |
| Java | 17, 21 |
The Flink range mirrors Flink's own support policy — the current and the previous minor
release — so a new Flink minor moves both ends of the range. Moving it is a deliberate change
backed by a weekly build over every supported version, never an automatic bump. Flink 1.20 is
supported from the same source (just verify-flink 1.20.4 builds and tests for it) and its
lane is verified on Java 17 — the Java 21 row is a 2.x claim. Java 11 is not supported even
though Flink 2.x declares it.
See Supported versions
for how the range is verified and why a single artifact covers the 2.x range.
Getting started
Quickstart puts the
connectors on a job's classpath and sets up credentials, then has one complete job per connector;
Examples covers dynamic
destinations, exactly-once, auto-creation and emulator-backed local runs; and the
configuration reference lists
every option each connector takes, with its default.
Blog series
The series began with the project introduction and the five connectors available in 1.0.0.
The Firestore article covers the connector added in 1.2.0.
For current setup and supported behavior, see the documentation.
Contributing
Beyond a trivial fix, contributions start with an issue rather than a pull request — see
CONTRIBUTING.md and the
Contributing
page on the documentation site.
License
Apache License 2.0
Disclaimer
This is an independent open-source project. It is not affiliated with, endorsed by, or
supported by the Apache Software Foundation or Google. Apache Flink, Flink, and the
Flink logo are trademarks of the Apache Software Foundation.
Acknowledgements
- Development of this project is assisted by Claude under
Claude for OSS, Anthropic's support program
for open-source developers.
- Development of this project is also assisted by Codex under
Codex for Open Source, OpenAI's support program for
open-source maintainers.
