testing-os

Operating system for testing in the AI era

Protocols, evidence stores, and learning loops for AI-assisted software.

v1.26.0 — current release. See CHANGELOG.md for what shipped.

📖 Read the handbook →

---

What This Is

testing-os records, verifies, and learns from your repo's real test evidence in an AI-native workflow. Point it at a repo, and every test run becomes a provenance-confirmed record you can trust — not a self-reported pass.

What you get:

• Provenance-confirmed records. Every submission is bound to a real CI run — keyless, via the provider's own identity — before it's accepted. The result is a tamper-evident, append-only evidence store, not an honor-system green check.
• A policy contract you control. Declare what counts as "verified" in YAML — a bounded, no-eval predicate DSL (field/op/value + all/any/not/implies) — and enforce it across your repos. Lint a policy before you ship it with dogfood-verify lint.
• A parallel-agent swarm protocol. Run multi-agent audits against a codebase, then turn raw findings into reusable patterns and doctrine.
• A live status surface. Per-repo records, indexes, and a status badge, all served from one evidence store.
• A page that says how a repository works. Atlas reads a repository's workflows and manifests, the tools they run, its imports, writes and history, and writes atlas/README.md: what comes in, what runs, where it lands, who reads it, what breaks what, what changed since the last map, where to start. No sentence on it is written by a person; atlas check fails CI when the map stops matching the code, atlas explain answers what a file, directory or part is in the system, atlas mcp lets an agent ask the map the same questions during a session, and every pull request gets the structural delta as a comment. atlas check also says, as notices, when a job pins a Node version the tool it runs does not accept, or runs npm ci from a lockfile that holds no native binding for the job's platform; and a clone behind its upstream is told which ref holds the newer map.

It's the flagship monorepo of the Dogfood Lab org — eight @dogfood-lab/* packages behind one swarm CLI and one atlas CLI.

Quick Start

npm install -g @dogfood-lab/dogfood-swarm
swarm --help

Want your own repo's test evidence recorded here? The examples/ starter kit gets you dispatching in five minutes (dogfood-report builds the submission; dogfood-init scaffolds the workflow). The operator's guide, CLI reference, schema reference, and integration recipes live in the handbook. Per-version detail is in CHANGELOG.md.

Run it for a private fleet

The public site renders every public repository that has adopted Atlas. For repositories that must not leave your machine, the same engine ships as a container with persistent memory:

mkdir -p atlas-data repos
cp docker/fleet.example.yml atlas-data/fleet.yml   # list your repositories, by mounted path or clone URL
docker compose -f docker/compose.example.yml up -d

./atlas-data is the memory: fleet.yml, every render, each repository's history and the state. The service maps once at start when the memory is empty, then on the schedule in fleet.yml, and serves the fleet list at http://127.0.0.1:8080/, each page at /?repo=owner/name, and an index for agents at /llms.txt. Nothing leaves the container except git fetches of the repositories you listed; deleting ./atlas-data is the only way to forget. The same image runs the CLI on one repository: docker run --rm -v "$PWD:/repo" ghcr.io/dogfood-lab/atlas map. Run commands and the file shapes are in docker/README.md.

Threat Model

testing-os processes dogfood submissions dispatched via repository_dispatch from trusted GitHub repos under mcp-tool-shop-org/* and dogfood-lab/*. The verifier requires CI provenance — claimed run IDs are confirmed via the provider's API, and submissions with malformed shapes, missing references, or invalid policy claims are rejected.

Provenance is the attestation. For a github submission the verifier confirms the claimed GitHub Actions run actually exists (GitHub API) and binds the submission's repo and commit_sha to that confirmed run — a live, keyless check rooted in GitHub's own OIDC identity, so a record cannot attest to a run or commit that did not happen. GitLab CI is supported opt-in (source.provider: gitlab); a GitLab submission is the one case the verifier calls a non-GitHub host (gitlab.com/api), and only for gitlab submissions.

Record integrity is tamper-EVIDENT, not tamper-proof. Every persisted record carries an integrity block (submission_digest + prev_digest) forming an append-only hash chain that node packages/ingest/run.js --verify-chain validates fully offline — detecting out-of-band tampering, disk corruption, and partial restores. Stub provenance, which confirms any claim without an API call, exists for local development only: it is refused in CI, and it writes a record only under an explicitly named INGEST_REPO_ROOT, so a stub record cannot enter this repository's own chain. The chain does not defend against the ingest credential itself, which can rewrite both a record and the chain; closing that needs an anchor outside the writer's control. An optional, off-by-default XRPL anchor (node packages/ingest/run.js --anchor-*) witnesses the chain head to the public XRP Ledger, making any truncation or rewrite below an anchored point detectable — the second disclosed non-GitHub call, and only when an operator enables it.

What testing-os touches: the submission JSON in each repository_dispatch payload; policies/, fixtures/, records/, indexes/, and dogfood/roadmap/ in this repo (the last written only by an operator-invoked swarm roadmap compile — never by the automated ingest path); outbound calls to api.github.com for provenance verification; and — for github submissions only — a read-only fetch of the submitting repo's dogfood/scenarios/<scenario_id>.yaml at the attested commit (the scenario definition that powers required-steps enforcement; size-capped and schema-validated before use, absent files simply leave that check unenforced with a visible warning).

What testing-os does NOT touch: consumer source code beyond the declared dogfood/scenarios/ definition files, secrets in consumer repos beyond the dispatch envelope, or anything outside this repo's working tree.

Finding-state transitions are evidence-bearing and append-only. The swarm control plane's closure verbs (swarm reopen, swarm close) require an explicit reason, evidence, and — for operator closures — a declared verification mode; every transition writes an immutable finding_events row recording the acting authority. No automated path can close a finding on staleness or reopen one by prediction, and no verb can rewrite the event history — a wrongly-used credential can add transitions, but each addition is itself on the record.

Network surface. By default the only egress is api.github.com (read-only: provenance confirmation + the scenario-definition fetch above). The two exceptions are both opt-in and disclosed above: a GitLab-provider submission (gitlab.com/api), and an operator-enabled XRPL anchor run. No telemetry, no analytics — this codebase never phones home; absent those two opt-in paths it exposes no network surface beyond GitHub. The receiver workflow runs with contents: write scoped to this repo only.

Packages

Package	Source	Purpose
@dogfood-lab/schemas	TypeScript	The 8 JSON schemas (record, finding, pattern, recommendation, doctrine, policy, scenario, submission).
@dogfood-lab/verify	JS	Central submission validator. Submissions pass through here before they're persisted.
@dogfood-lab/findings	JS	Finding contract + derive/review/synthesis/advise pipelines.
@dogfood-lab/ingest	JS	Pipeline glue: dispatch → verify → persist → index.
@dogfood-lab/report	JS	Submission builder for source repos.
@dogfood-lab/portfolio	JS	Cross-repo portfolio generator.
@dogfood-lab/dogfood-swarm	JS	The 10-phase parallel-agent protocol + SQLite control plane + swarm bin.
@dogfood-lab/atlas	JS	Reads a repository and writes the page that says how it works (atlas/README.md); atlas check gates the map in CI, and atlas mcp answers agents from it. No sibling dependencies; runs in any repository.

Sibling testing tools that stay independent but integrate via published APIs: shipcheck, repo-knowledge, ai-eyes-mcp, taste-engine, style-dataset-lab.

Layout

testing-os/
├── packages/                  # 8 workspace packages (@dogfood-lab/*)
├── atlas/                     # This repository's own Atlas page and map, written by `atlas map`
├── site/                      # Astro Starlight handbook → dogfood-lab.github.io/testing-os/handbook/
├── swarms/                    # Swarm-run artifacts + control-plane.db
├── indexes/                   # Generated read API: latest-by-repo.json, failing.json, stale.json, trends.json, badges/ (shields.io endpoints)
├── policies/                  # Policy YAML by repo
├── records/                   # Submission landing pad (ingest.yml writes here)
├── fixtures/                  # Test/example fixtures
├── docs/                      # Contract docs + architecture notes
├── examples/                  # Copy-paste consumer starter kit (dogfood.yml + scenario + policy)
├── scripts/                   # Repo-level utilities (sync-version, build)
└── .github/workflows/         # ci.yml, ingest.yml, pages.yml, release.yml, self-dogfood.yml, atlas-render.yml

Local Development

git clone https://github.com/dogfood-lab/testing-os.git
cd testing-os
npm install
npm run build       # tsc --build across all packages
npm test            # vitest for schemas, node --test for the rest
npm run verify      # version-sync + doc-drift + regression-pin gates + build + tests (canonical pre-commit check — NOT the same as build && test)

Requires Node ≥ 22. CI matrix runs Node 22 + 24 on ubuntu-latest; locally validated on Node 25.

Supported filesystems: APFS, HFS+, ext4 (CI baseline), NTFS — anything that implements POSIX link(2). Not supported: exFAT, FAT32. The file-lock CAS in packages/findings/lib/file-lock.js requires hardlink semantics for atomic publication; on exFAT, linkSync throws ENOTSUP (loud, not silent). Common gotcha: cross-platform external SSDs are often formatted exFAT — clone the repo to local APFS/HFS+ instead. See docs/m5-validation-2026-04-29.md for the full Session G validation matrix.

Versioning

All @dogfood-lab/* packages bump together — one number across the monorepo. Seven packages publish to npm under @dogfood-lab at v1.26.0 in lockstep (schemas, verify, report, ingest, findings, dogfood-swarm, atlas); the eighth, @dogfood-lab/portfolio, stays internal. The version line near the top of this README is auto-stamped from package.json via scripts/sync-version.mjs on every npm run build.

License

MIT © 2026 mcp-tool-shop

---

Handbook · All Repositories · Profile

Eat first. Ship second.