Job Market Analytics

End-to-end Analytics Engineering project: typed Python ingestion from every ATS with a public keyless feed (nine today — Greenhouse, Lever, Ashby, BambooHR, Recruitee, Workable, Pinpoint, Rippling, SmartRecruiters) into a dual-target dbt project (DuckDB dev / BigQuery prod) that dedupes, lifecycle-tracks, and rule-filters postings — then emails a digest of what's new. LLM relevance scoring (in-warehouse, BigQuery AI) is the scoped V2.

flowchart LR
    A[ATS APIs<br/>9 public keyless feeds] -->|"Python adapters<br/>(typed, tested, parallel)"| B[(raw_*_jobs<br/>append-only)]
    B --> C[bronze<br/>stg views]
    C --> D[silver<br/>union · dedup · lifecycle<br/>filters + soft signals]
    D --> E[gold<br/>fct_job_postings<br/>live postings only]
    E -->|watermark| F[email digest<br/>new postings only]
    D -.->|V2| G[LLM extraction<br/>+ fit scoring]
    G -.-> E

→ Project site

Live market charts, design decisions, rendered docs and all 29 ADRs, and the dbt lineage DAG. Rebuilt in CI on every push to main, or locally with make site (output in _site/). Every number on the landing page is read from the dbt manifest, the ADRs and the test suite at build time, and the build fails on a broken link. Every PR's CI run also uploads a site-preview artifact, so a change to the site can be reviewed before it merges.

Quickstart

Prerequisite: uv (the only one — it installs Python 3.14 itself, per .python-version).

curl -LsSf https://astral.sh/uv/install.sh | sh   # macOS/Linux; or: brew install uv

make install                  # uv venv + pre-commit hooks
cp .env.example .env          # ingestion needs no secrets; fill BQ vars only for prod
cp config/companies.example.csv config/companies.csv   # your PRIVATE company list (gitignored)
# dbt/profiles.yml is committed (env-var driven, no secrets) — nothing to copy

make ingest                   # Python -> raw tables (DuckDB by default)
make dbt-dev                  # bronze -> silver -> gold on DuckDB
make test && make dbt-test

Structure

ingest/      per-ATS adapters, source registry, parallel pipeline entrypoint
shared/      config (Pydantic Settings), models, http, storage (one writer, both warehouses)
deliver/     email digest of new postings (watermark in ops.digest_runs)
config/      private company list (gitignored; .example committed)
dbt/         one dual-target project: models/{bronze,silver,gold}, seeds, macros, unit tests
scripts/     repo gates that are not lint or tests (see below), and the Pages site build
site/        landing page template + docs theme for the Pages site
tests/       pytest suite + sanitized fixtures
docs/        decisions/, v2-plan.md, research/  (ARCHITECTURE.md at repo root)
.github/     four workflows, split by permission scope — see ARCHITECTURE.md#automation

Currently 9 ingestion sources, 22 dbt models and 29 ADRs.

Quality gates

Every invariant this project relies on is enforced by something that fails, not by intent. make lint and make test run locally, in a pre-commit/pre-push hook, and again in CI, which blocks merge:

Gate	Guards against
ruff check + ruff format	style drift, unused code, import order
mypy --strict	untyped defs; no Any crossing a module boundary
sqlfluff	dbt SQL style, against the compiled models
pytest with --cov-fail-under=85	untested behavior; the gate is never lowered to pass
dbt schema + unit tests	model contracts (not_null, unique, accepted_values)
dbt source freshness	a source that silently stopped landing rows
scripts/check_docs.py	documentation that points at things which no longer exist
gitleaks	committed secrets
actionlint	a workflow that fails validation — which does not fail loudly: it stops running on its schedule and sends no alert, because no run ever starts
Dependabot	stale dependencies (uv.lock + pyproject.toml together)

The docs check is the unusual one. Prose drifts silently — a file is renamed, a make target goes away, an ADR is superseded, and the sentence referring to it still reads fine, because nothing in a test suite imports a paragraph. It verifies the references that are mechanically checkable (relative links, repo paths, line-anchored refs, ADR numbers, make targets, dbt model names, heading anchors) across both markdown and Python docstrings, and lets a doc assert a number against the code with an inline <!-- check:… --> marker, so a count cannot rot under an edit. Deliberate exceptions live in scripts/planned-refs.txt; everything else failing is drift.

Stack

Python 3.14 + Pydantic v2 · dbt-core with dbt-duckdb (dev) and dbt-bigquery (prod) · GitHub Actions (twice-daily ingest, freshness gate, email digest of new postings).

Status & roadmap

V2 (0.2.0) in production — twice-daily ingestion to BigQuery, transform, freshness gate, email digest, plus AI relevance: typed extraction and a 1–5 fit score as dbt models against BigQuery's in-SQL AI functions, score-ordered (never filtered) delivery (ADR-0020), and an embedding matcher that reports which piece of experience a posting resembles.

Deliberately unvalidated so far. Two rankings run side by side — the LLM score and embedding similarity — because choosing between them without measuring is how a project ends up carrying both forever. make evaluate scores a ranking by precision@k against human labels, and labels from an LLM cannot settle it: grading a scorer against its predecessor's judgements rewards reproducing that predecessor's mistakes. Both rankings are kept until labels exist and a month of live runs has shown the winner stays the winner; only then does the loser get deleted.

Full design: ARCHITECTURE.md. What is next: TODO.md.

License

Personal project. Not currently licensed for redistribution.