s2.dev is a serverless datastore for real-time, streaming data.
This repository contains:
- s2-cli - Command-line interface for S2
- s2-lite - Open source, self-hostable server implementation of the S2 API
- s2-sdk - Rust SDK for S2
Development
Use the nightly Cargo dependency commands so that the repository publication cooldown applies:
cargo +nightly add <crate>
cargo +nightly update
cargo +nightly update -p <crate>
cargo +nightly remove <crate>
cargo +nightly generate-lockfile
Use --locked with normal build, check, test, run, document, fetch, and metadata commands. The simulator is temporarily exempt until its separate lockfile is regenerated. The pull request dependency check verifies every proposed lock-file change before Rust build jobs start.
Install the repository Cargo tools from Homebrew bottles:
brew install cargo-deny cargo-nextest
Installation
Homebrew (macOS/Linux)
brew install s2-streamstore/s2/s2
Cargo
cargo install --locked s2-cli
Release Binaries (macOS/Linux)
curl -fsSL https://raw.githubusercontent.com/s2-streamstore/s2/main/install.sh | bash
Or specify a version with VERSION=x.y.z before the command. See all releases.
Docker
docker pull ghcr.io/s2-streamstore/s2
Authentication
Store an access token in the OS credential store:
s2 auth access-token set
Or pipe it in from a script or a secret manager:
op read 'op://S2/CLI/access-token' | s2 auth access-token set --stdin
For CI and other ephemeral environments, set S2_ACCESS_TOKEN in the environment. On a headless host with no credential store, opt into a private plaintext file instead (mode 0600 on Unix):
printf '%s' "$S2_ACCESS_TOKEN" |
s2 auth access-token set --stdin --insecure-storage
unset S2_ACCESS_TOKEN
Plaintext access_token values in config.toml are deprecated.
Migrate one with s2 auth access-token migrate, or re-store it with --insecure-storage on a
headless host that has no credential store.
s2-lite
s2-lite is embedded as the s2 lite subcommand of the CLI. It's a self-hostable server implementation of the S2 API.
It uses SlateDB as its storage engine, which relies entirely on object storage for durability.
It is easy to run s2 lite against object stores like AWS S3 and Tigris. It is a single-node binary with no other external dependencies.
You can also simply not specify a --bucket, which makes it operate entirely in-memory (or use --local-root to persist to local disk instead).
[!TIP]
When you point lite at a --bucket, data is always durable on object storage before being acknowledged or returned to readers — just like s2.dev.
The optional in-memory mode (no --bucket or --local-root specified) just makes it an effective S2 emulator for integration tests.
Quickstart
Here's how you can run in-memory without any external dependency:
# Using Docker
docker run -p 8080:80 ghcr.io/s2-streamstore/s2 lite
# Or directly with the CLI
s2 lite --port 8080
docker run -p 8080:80 \
-e AWS_PROFILE=${AWS_PROFILE} \
-v ~/.aws:/home/nonroot/.aws:ro \
ghcr.io/s2-streamstore/s2 lite \
--bucket ${S3_BUCKET} \
--path s2lite
docker run -p 8080:80 \
-e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
-e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
-e AWS_ENDPOINT_URL_S3=${AWS_ENDPOINT_URL_S3} \
ghcr.io/s2-streamstore/s2 lite \
--bucket ${S3_BUCKET} \
--path s2lite
[!NOTE]
Point the S2 CLI or SDKs at your lite instance like this:
export S2_ACCOUNT_ENDPOINT="http://localhost:8080"
export S2_BASIN_ENDPOINT="http://localhost:8080"
export S2_ACCESS_TOKEN="ignored"
Let's make sure the server is ready:
while ! curl -sf ${S2_ACCOUNT_ENDPOINT}/health -o /dev/null; do echo Waiting...; sleep 2; done && echo Up!
Install the CLI (see Installation above) or upgrade if s2 --version is older than 0.26
Let's create a basin with auto-creation of streams enabled:
s2 create-basin liteness --create-stream-on-append --create-stream-on-read
Test your performance:
s2 bench liteness --target-mibps 10 --duration 5s --catchup-delay 0s
Now let's try streaming sessions. In one or more new terminals (make sure you re-export the env vars noted above),
s2 read s2://liteness/starwars 2> /dev/null
Now back from your original terminal, let's write to the stream:
nc starwars.s2.dev 23 | s2 append s2://liteness/starwars
Kubernetes Deployment
Deploy s2-lite to Kubernetes using Helm. See the Helm chart documentation for installation instructions and configuration options.
Storage
Lite persists to an S3-compatible bucket or a local directory, or runs in-memory when neither is given.
The write-ahead log (WAL) shares the main store by default; it can be placed in a separate bucket or
directory to isolate WAL latency from flushes and compaction. --path applies to both stores.
| Setting | Main store | WAL store |
|---|
| S3 bucket | --bucket | --wal-bucket / S2LITE_WAL_BUCKET |
| Local directory | --local-root | --wal-local-root / S2LITE_WAL_LOCAL_ROOT |
| S3 endpoint | AWS_ENDPOINT_URL_S3 | S2LITE_WAL_AWS_ENDPOINT_URL_S3 |
| AWS region | AWS_REGION | S2LITE_WAL_AWS_REGION |
| Static credentials | AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY | S2LITE_WAL_AWS_ACCESS_KEY_ID + S2LITE_WAL_AWS_SECRET_ACCESS_KEY |
| Session token | AWS_SESSION_TOKEN | S2LITE_WAL_AWS_SESSION_TOKEN |
Without static credentials, the standard AWS credential chain (profile, instance role, etc.) is used.
The WAL bucket inherits the main store's S3 settings; each S2LITE_WAL_AWS_* variable overrides just
that setting, except that a WAL key pair replaces the main credentials (and session token) as a set.
A WAL store requires a persistent main store.
# LSM in S3, WAL on local disk
s2 lite --bucket my-bucket --wal-local-root /data/wal
# LSM and WAL in separate buckets
s2 lite --bucket my-bucket --wal-bucket my-wal-bucket
Monitoring
/health will return 200 on success for readiness and liveness checks
/metrics returns Prometheus text format
Internals
SlateDB settings
Settings reference
Use SL8_ prefixed environment variables, e.g.:
# Defaults to 50ms for S3, 5ms otherwise; follows the WAL store when set
SL8_FLUSH_INTERVAL=10ms
Design
Concepts
- HTTP serving is implemented using axum
- Each stream corresponds to a Tokio task called
streamer that owns the current tail position, serializes appends, and broadcasts acknowledged records to followers
- Appends are pipelined to improve performance against high-latency object storage
lite::backend::kv::Key documents the data modeling in SlateDB
Compatibility
API Coverage
Complete specs are available:
- OpenAPI for the REST-ful core
- Protobuf definitions
- S2S, which is the streaming session protocol
[!IMPORTANT]
Unlike the cloud service where the basin is implicit as a subdomain, /streams/* requests must specify the basin using the S2-Basin header. The SDKs take care of this automatically.
