Job Market Analytics
End-to-end Analytics Engineering project: typed Python ingestion from every ATS with a
public keyless feed (nine today — Greenhouse, Lever, Ashby, BambooHR, Recruitee, Workable,
Pinpoint, Rippling, SmartRecruiters) into a dual-target dbt project (DuckDB
dev / BigQuery prod) that dedupes, lifecycle-tracks, and rule-filters postings — then emails
a digest of what's new. LLM relevance scoring (in-warehouse, BigQuery AI) is the scoped V2.
flowchart LR
A[ATS APIs<br/>9 public keyless feeds] -->|"Python adapters<br/>(typed, tested, parallel)"| B[(raw_*_jobs<br/>append-only)]
B --> C[bronze<br/>stg views]
C --> D[silver<br/>union · dedup · lifecycle<br/>filters + soft signals]
D --> E[gold<br/>fct_job_postings<br/>live postings only]
E -->|watermark| F[email digest<br/>new postings only]
D -.->|V2| G[LLM extraction<br/>+ fit scoring]
G -.-> E
Live market charts, design decisions, rendered docs and all 29 ADRs, and the dbt lineage DAG.
Rebuilt in CI on every push to main, or locally with make site (output in _site/). Every number
on the landing page is read from the dbt manifest, the ADRs and the test suite at build time, and
the build fails on a broken link. Every PR's CI run also uploads a site-preview artifact, so a
change to the site can be reviewed before it merges.
Quickstart
Prerequisite: uv (the only one — it installs Python 3.14
itself, per .python-version).
curl -LsSf https://astral.sh/uv/install.sh | sh # macOS/Linux; or: brew install uv
make install # uv venv + pre-commit hooks
cp .env.example .env # ingestion needs no secrets; fill BQ vars only for prod
cp config/companies.example.csv config/companies.csv # your PRIVATE company list (gitignored)
# dbt/profiles.yml is committed (env-var driven, no secrets) — nothing to copy
make ingest # Python -> raw tables (DuckDB by default)
make dbt-dev # bronze -> silver -> gold on DuckDB
make test && make dbt-test
Structure
ingest/ per-ATS adapters, source registry, parallel pipeline entrypoint
shared/ config (Pydantic Settings), models, http, storage (one writer, both warehouses)
deliver/ email digest of new postings (watermark in ops.digest_runs)
config/ private company list (gitignored; .example committed)
dbt/ one dual-target project: models/{bronze,silver,gold}, seeds, macros, unit tests
scripts/ repo gates that are not lint or tests (see below), and the Pages site build
site/ landing page template + docs theme for the Pages site
tests/ pytest suite + sanitized fixtures
docs/ decisions/, v2-plan.md, research/ (ARCHITECTURE.md at repo root)
.github/ four workflows, split by permission scope — see ARCHITECTURE.md#automation
Currently 9 ingestion sources, 22 dbt models and 29 ADRs.
Quality gates
Every invariant this project relies on is enforced by something that fails, not by intent.
make lint and make test run locally, in a pre-commit/pre-push hook, and again in CI, which
blocks merge:
| Gate | Guards against |
|---|
ruff check + ruff format | style drift, unused code, import order |
mypy --strict | untyped defs; no Any crossing a module boundary |
sqlfluff | dbt SQL style, against the compiled models |
pytest with --cov-fail-under=85 | untested behavior; the gate is never lowered to pass |
| dbt schema + unit tests | model contracts (not_null, unique, accepted_values) |
dbt source freshness | a source that silently stopped landing rows |
scripts/check_docs.py | documentation that points at things which no longer exist |
gitleaks | committed secrets |
actionlint | a workflow that fails validation — which does not fail loudly: it stops running on its schedule and sends no alert, because no run ever starts |
| Dependabot | stale dependencies (uv.lock + pyproject.toml together) |
The docs check is the unusual one. Prose drifts silently — a file is renamed, a make target goes
away, an ADR is superseded, and the sentence referring to it still reads fine, because nothing in
a test suite imports a paragraph. It verifies the references that are mechanically checkable
(relative links, repo paths, line-anchored refs, ADR numbers, make targets, dbt model names,
heading anchors) across both markdown and Python docstrings, and lets a doc assert a number
against the code with an inline <!-- check:… --> marker, so a count cannot rot under an edit.
Deliberate exceptions live in scripts/planned-refs.txt; everything else failing is drift.
Stack
Python 3.14 + Pydantic v2 · dbt-core with dbt-duckdb (dev) and dbt-bigquery (prod) ·
GitHub Actions (twice-daily ingest, freshness gate, email digest of new postings).
Status & roadmap
V2 (0.2.0) in production — twice-daily ingestion to BigQuery, transform, freshness gate,
email digest, plus AI relevance: typed extraction and a 1–5 fit score as dbt models against
BigQuery's in-SQL AI functions, score-ordered (never filtered) delivery (ADR-0020), and an
embedding matcher that reports which piece of experience a posting resembles.
Deliberately unvalidated so far. Two rankings run side by side — the LLM score and embedding
similarity — because choosing between them without measuring is how a project ends up carrying
both forever. make evaluate scores a ranking by precision@k against human labels, and labels
from an LLM cannot settle it: grading a scorer against its predecessor's judgements rewards
reproducing that predecessor's mistakes. Both rankings are kept until labels exist and a month
of live runs has shown the winner stays the winner; only then does the loser get deleted.
Full design: ARCHITECTURE.md. What is next: TODO.md.
License
Personal project. Not currently licensed for redistribution.
