Tacet
A private AI assistant that lives in your terminal. No cloud, no account, no telemetry.
tacet — in musical notation: this instrument is silent in this passage. Here, the silent instrument is the network.
$ tacet
Tacet.
> how many days until 14 March?
[time] diff · 229 days
229 days.
> put that in a spreadsheet called countdown
[create_document] countdown.xlsx · written
Done — countdown.xlsx, one row, with a live =SUM() under Days left.
Why this exists
Most "local AI" tools are a thin shell around a model file. The hard part isn't running the model — it's everything around it: deciding when a tool is needed, forcing the model to produce a valid call, executing it safely, and keeping bulk data out of a context window that is measured in thousands of tokens, not millions.
Tacet is that layer, written to be read. Every non-obvious decision has a comment explaining why, and several of them record a measurement that proved the obvious approach wrong.
What makes it different
Start here if you already know this field. Constrained decoding — a schema
compiled to an automaton, a per-step logit mask, "unrepresentable rather than
validated afterwards" — is category-standard, and has been since llama.cpp
shipped GBNF in 2023. It is what
Outlines does, and
XGrammar (in vLLM, TensorRT-LLM, MAX), and
llguidance (which ships inside
OpenAI's JSON Schema mode), and lm-format-enforcer, and Apple's own on-device
guided generation. The first bold claim below is table stakes; the page used to
lead with it and say nothing about any of them, which reads as unawareness rather
than as a feature. What is actually new here, and what is
not says which is which, with citations.
The four claims below are the ones this project would defend in that room.
A call that has started cannot be finished invalidly. Once the model emits calculate(, a pushdown automaton masks the logits at every step. Malformed JSON, a field that isn't in the schema, an out-of-range number, a number the tool layer cannot read back as the type it asked for (-0 in an integer, a value past u64), an enum value or key whose JSON spelling the grammar got wrong, a missing required key — none of them can be generated. Not validated after the fact: unrepresentable. Sampling runs after masking, so no sampling strategy can escape it.
And what runs is the call the automaton passed. That was not true until the audit of 9 Oct 2026 measured it (H5): the step that strips a <think> block from the output was blind text surgery, so a </think> written inside a call's arguments could cut the validated call into a different one before it was parsed and run. It no longer edits a call. A tag inside an argument stays in the argument. A grammar call — name( or bare JSON — with both <think> and </think> in its values runs exactly as written, because it parses and so the split can prove where it ends: cargo test -p tacet-engine --lib thinking::tests::a_proven_split_has_one_reading. A recovered call — a marked <tool_call> or a name glued to its argument — carrying both is refused rather than run, because cut at its </think> it is no call and the two readings disagree; and wherever the split can only guess where the thinking ends, nothing runs unless both readings name the same call: cargo test -p tacet-tools --lib executor::tests::a_call_declined_inside_the_thinking_never_reaches_the_executor.
The sentence used to read "invalid tool calls are impossible", and measuring it showed that was wider than the truth: the grammar arms after name(, so it says nothing about a call that never starts that way. Running a real model over 115 cases, seven of the twenty-two failures were the right tool with the right arguments written in a shape nobody taught it — ```tool read_document"path=report.md" and <tool_call> read_document (path: "x") </tool_call>. Those are recovered now, by a layer that will only look behind a marker no prose contains; the underlying gap is real and is written down rather than papered over.
And a valid call has to end. That is a second property, weaker than the first and until recently not held at all: a valid prefix could wander forever. A model wrote a complete, correct calendar(…) call and then emitted whitespace for twelve minutes, because whitespace was legal at a structural position and legal again immediately. Unrepresentable-invalid and always-terminating are different claims — the grammar now bounds consecutive whitespace and every field the schema leaves open — a string at 8192 characters, a number at 400 fraction and 4 exponent digits (its integer part stops where u64, i64 or f64 do), an array at 1024 items — and the engine caps a constrained generation at 2048 tokens, measured against a largest-observed legitimate call of 1523.
A tool result cannot forge a turn. The schema makes an invalid call
unrepresentable — that is a claim about the CALL, and it says nothing about the
prompt. Measured on the prompt: a tool result went in verbatim, so a document
containing </tool_response><|im_end|> followed by <|im_start|>system rendered
as a real system turn in the ChatML prompt — a forged instruction in the role
a model is trained to obey above every other, written by whoever wrote the file.
read_document reads files the user did not write, web_fetch reads pages
nobody here controls, an MCP result is a third party's text, and a file name in
the working directory is whatever its creator chose. Every string entering the
prompt from outside — tool results, the question, memory, a user-authored skill
and its name, a bridged tool's own name and description, the directory census
in the system block, and the summary of a session imported from another agent
— now has its markers neutralised with one space
(< |im_end|> is not a token), so nothing is censored and nothing can open a
block. "Markers" means every chat-template token, not only the <|…|> ones:
Qwen3's <think> and <tool_call>, Gemma's <start_of_turn>, <eos>, <bos>
and <unusedN>, Llama-2's <s>/</s>, this prompt's own fences (<memory>,
<cwd>, <guidance>, …), and whatever special tokens the loaded tokenizer
declares. Ordinary code is left alone — F#'s x <| f and Haskell's <|> are
not markers. An audit on 9 Oct 2026 measured three gaps in the earlier version:
a file name forging a system turn, a remembered note closing its own
</memory> fence, and <think> passing as a live token. cargo test -p tacet-engine --test a_tool_result_cannot_forge_a_turn --test a_file_name_cannot_forge_a_turn --test every_marker_from_outside_is_text is
the demonstration, in all three templates; for the imported summary it is cargo test -p tacet-tools --lib sessions_tool::tests::a_transcript_cannot_forge_the_summary_or_close_the_seed.
And then the whole prompt was printed and read, which found three more. The
<memory> block was fenced twice — a nested tag with a stray closing one in
the middle, in the system block, on every turn a note matched. The <guidance>
block was fenced twice for the same reason. And the fix above had broken the
skill's own closing tag into < /guidance>, leaving that fence open — the
failure this repository has a rule about, half an order being worse than no
order, introduced by the change that closed the forged-turn hole. None of the
three failed a test and none of them would have: they are visible only by
looking at what the model is actually sent.
Memory is now shared safely between sessions. Two terminals used to be two copies of memory.json, and whichever saved last won: session A remembered X, session B remembered Y, and only Y was left (audit M20, measured). A save now takes a lock file, re-reads the file and replays only its own changes, so neither session can undo what the other did. A file that does not parse is no longer opened as an empty memory: the last good copy, memory.json.bak, is loaded instead, and the damaged file is moved aside as memory.json.damaged-*, never overwritten. To wipe memory by hand, delete memory.json and memory.json.bak together. forget deletes one note: a word that matches several notes deletes nothing and asks which one. A note can no longer carry a tag (</memory>, <|im_start|>, <think> …) into the prompt.
The network monopoly is checkable by eye. Exactly two crates may open a socket, and the HTTP dependency appears in exactly those two manifests. You do not have to trust a privacy claim you cannot audit — grep -v '^\s*#' crates/*/Cargo.toml | grep ureq is the audit, and cargo test -p tacet-cli --test network_monopoly is the same audit as a failing build. It reads every manifest as TOML (renames, dotted keys, target tables and workspace inheritance included) and walks the dependency graph cargo resolves, and asserts that only those two crates reach an HTTP client, whether declared or one crate removed. It asserts that no other crate reaches the network without one: no std::net socket, no DNS lookup, no socket-layer crate (libc, nix, rustix, the Windows bindings) declared under any name or reached with its socket feature on, no libc or Winsock socket call under any name a file gives it, and no foreign function outside a five-name allow-list (setsid, killpg, poll, signal, raise), in any file a crate compiles — its whole directory, build.rs, and whatever #[path], include! or the manifest points rustc at — and that every crate cargo builds from this repository lives under crates/, where that scan reads. It asserts that no other crate starts a program whose job is the network (docker, curl, ssh, …) without a row that says why; the only rows are tacet addon's local SearXNG setup in tacet-cli/src/addon.rs, which runs docker compose only after a y/N approval. And it asserts that no crate, those two included, starts another agent's CLI (claude, codex, agy) unless that path is either an external tool behind the approval gate (today none is) or a command you type that the model cannot reach. Today there is exactly one such command: tacet handoff note codex <id> --program <path>. It starts the Codex binary at the path you give as codex app-server, after a y/N that names the resolved file and says it runs with your Codex login and can reach the network. No model turn is started. The test fails if anything but tacet handoff calls the function that starts it, if anything but the command-line dispatch names tacet handoff itself, or if one of those files implements a model tool. (One honest asterisk: if you install the shell addon and put curl on its allow-list, you have handed a program the network. That is why shell sits behind the approval gate — see .)
Nothing leaves the device by default. Everything with outside reach is an addon you install deliberately: web search against your own SearXNG, HTTP against hosts you name, a shell against programs you list. Until you install one, its tools are not merely disabled — they are absent from the catalog the model is shown, so it cannot call them or claim it did.
Four gates on every tool call, none of which work by matching text:
| Gate | Rejects |
|---|
| Name | a tool that isn't in the catalog — the model cannot invent a callable |
| Schema | arguments that don't validate, even if the grammar was bypassed |
| Approval | outbound data in a session that has touched personal data; any shell or db_write call once the session has read text from outside the machine (a web page, an HTTP or MCP answer, an imported transcript); every call to an MCP server marked always_ask — and with nobody at a terminal to ask, the answer is no |
| Cancel | the rest of the turn, the moment you press ctrl-c — shell, run_code and write_code end their process group within about a second, and git kills the git it started (the call's 120 s deadline stops it the same way); any other tool that has not stopped after 3 s is abandoned rather than waited for, and the turn says it was left running, not that it stopped (whatever that tool was doing may still finish on its own thread). The press arrives two ways. With no turn indicator (--json, or stdout piped) it is a signal, driven at the binary by cargo test -p tacet-cli --test ctrl_c_reaches_the_running_tool — run on macOS (10 Oct 2026), not yet on Linux, and on Windows that path is not wired. At a terminal it is a key the indicator reads, and the same file types it: at_a_terminal::ctrl_c_typed_during_a_tool_stops_it_and_the_shell_lives_on runs the shell on a pty through script(1), presses ctrl-c during shell sleep 9, and holds that the chip comes back within 5 s, the sleep is gone and the next message is answered — run on macOS (11 Oct 2026; with the tool's cancel token unwired it reads exit 0 · 9554 ms, the audit's defect), not on Linux, whose script is spelled differently. The key thread alone is cargo test -p tacet-cli --bin tacet ui::tests::ctrl_c_marks_the_cancellation_and_other_keys_are_swallowed; the 3 s abandonment is only a unit test — cargo test -p tacet-tools --lib executor::guarded::a_cancelled_tool_that_will_not_stop_is_abandoned_after_the_grace |
Bulk data never enters the model. A tool that reads a 40 000-row spreadsheet puts the data in a store and hands the model a short summary plus a reference. The next tool that needs it fetches it by reference. The context window is a budget, not a bottleneck.
Almost no dependencies. OOXML (.xlsx, .docx) generation, zip, deflate and CRC32 are written by hand. So is the MCP client — JSON-RPC 2.0 over Streamable HTTP with SSE, OAuth, elicitation and task polling, in 3,503 non-test lines across twelve files (5,067 with the tests; 11 use std lines in 6 of them; re-derived 10 Oct 2026, counting each file's lines above its first #[cfg(test)]). That sentence said "~430 lines with a single use std" for a long time: true at the first commit, and never re-derived while the protocol grew. The claim it is here to support survives the correction — there is no client library underneath, and 3,503 lines of plain Rust is still a thing one person can read — but a number in a paragraph about auditability has to be the real one.
The direct dependency list is serde, serde_json, thiserror, clap, crossterm and ureq. Behind the off-by-default candle feature there are three more: candle-core, candle-transformers and tokenizers. Adding to that list is an architectural decision documented at the top of the file, not a convenience.
One honest note on what "the full list" means: those are the DIRECT dependencies, and they are the ones a reader can audit. Two of them pull a C build transitively, and both are worth naming rather than glossing:
ureq brings rustls, which brings ring, and ring compiles C. Nothing here calls it and it changes no claim about what the program does, but it is what sets the build prerequisite on Windows: not the MSVC linker it looks like, but a C compiler, however you get one. Measured, with the three routes and their sizes, in CONTRIBUTING.tokenizers brings onig → onig_sys → cc, and esaxx-rs alongside it — a second C build, on the candle feature. This paragraph disclosed the first and not the second, which made the off-by-default feature look cheaper than it is. It is off by default and the CLI runs without it; if you build with it, you are building C.
And the transitive half is checked rather than argued. cargo deny check runs on every push to main and every pull request, over the full graph with all features on — advisories, licences, wildcards, and sources. Sources, bans and licences gate the merge; advisories run as a separate job that is allowed to fail, because a new advisory against a transitive crate is news, not a defect in the pull request. The two things it found on its first run are in deny.toml with a reason and a date rather than silenced: paste is unmaintained (a proc macro under candle, no runtime, no safe upgrade, not ours to fix), and webpki-roots carries CDLA-Permissive-2.0 because it is Mozilla's root CA store — data, not code. unknown-git and unknown-registry are hard failures, because those can only be caused from inside this repository.
Install
cargo install tacet-cli --features metal # Apple GPU
cargo install tacet-cli --features candle # CPU
cargo install tacet-cli # no inference; still runs eval, tools and the addon flow
cargo install tacet-cli installs the last published release, and this page
describes main, which is ahead of it. The last publish was tacet-cli
0.1.29 on 7 Sep 2026; every fix from the audit of 9 Oct 2026 — and the
agent-sessions addon and tacet handoff below — is unreleased, and
CHANGELOG lists it under Unreleased. To run what this page
describes before the next release, install from a checkout:
cargo install --path crates/tacet-cli --features metal.
This paragraph used to say "cargo install tacet-cli is this page", which was
true on the day all eleven crates were published (6 Sep 2026) and stopped being
true with the next commit. It resolved to 0.1.25 until then — a build with no bench
subcommand, which this page names twenty times, pinned to tacet-engine ^0.1.9,
one release below the 0.1.10 floor for constrained generation on Metal. A Mac
user following the line above was installing the constraint rejected the token: 4286578688 crash. The publish order and the checks that go with it are in
CONTRIBUTING.
⚠️ cargo install does not remember --features. Upgrade without the flag
and you get a binary that cannot run a model. It refuses to answer rather
than making something up: it names the missing feature, tells you how to
reinstall, and exits non-zero. It used to fall back to a scripted engine and keep
answering — which meant the failure looked like a working product giving bad
replies, and nobody reported it. tacet --version prints the build it actually is:
tacet --version # tacet 0.1.29 (metal) ← the part in brackets is the engine
Or grab a prebuilt binary from Releases — macOS (Apple Silicon
and Intel), Linux and Windows.
Update at any time — tacet update asks GitHub for the newest release, and only when you run it:
tacet update # downloads the newest release and replaces the binary — after asking
tacet update --check # only looks: says whether a newer version exists, writes nothing
This command replaces the running binary, so it is worth saying what backs it.
It verifies the per-asset SHA-256 that the GitHub API reports, and says so
honestly when it cannot. From v0.1.28 each release asset also carries a build
provenance attestation — signed through Sigstore with a short-lived workflow
identity, binding the file to this repository, this workflow and this commit, so
there is no long-lived key to leak. Verify it without trusting the release job:
gh attestation verify tacet-aarch64-apple-darwin --repo farukciftler/tacet-cli
The SHA256SUMS file that was already there proves only that the release job
agreed with itself — the same job computed the hashes and uploaded the binaries.
That is a corruption check, not a provenance one. Releases also carry an SPDX
SBOM of the full dependency graph.
Nothing checks on its own until you say so. A few turns into your first session
the shell asks once whether it may look for a new version daily, and writes the
answer down:
tacet config set update.check on # one request a day, at the end of a session
tacet config set update.check off # never
The check is throttled to once every 24 hours, prints a single line when a newer
version exists and stays silent when it fails. It runs only at the end of an
interactive session with both stdout and stderr on a terminal — never after
tacet -m/--message, under --json, or when output is piped or captured.
Quickstart
Tacet needs a model. It never downloads one behind your back:
tacet models list # what's on disk, which roots were searched
tacet models download qwen3-4b # ~2 GB, https only, sha256 checked before it lands
Two models are in the built-in catalog — qwen3-4b (the default:
Qwen3-4B-Instruct-2507, Q4_K_M) and qwen2.5-3b (smaller, for machines with
less to spare). The name is worth reading closely: Qwen3-4B and
Qwen3-4B-Instruct-2507 are two different models, and the catalog pins the
second — the one every number on this page was measured on. Every download is over
HTTPS to an address printed on screen first, and the file is rejected unless its
sha256 matches the one compiled into the binary. Nothing is fetched until you
ask; add your own entries in ~/.tacet/packages.json.
Already have weights? Point at them and skip the download entirely:
export TACET_MODEL=/path/to/model.gguf
But the engine loads more architectures than this project has measured, and
the gap is large. qwen2, qwen3, gemma3 and llama GGUFs all load; every
number on this page comes from a Qwen. Pointing Tacet at the other two is
allowed and it will say so once, on stderr, before the first question:
(note: 'gemma3' is not one of the architectures this project measures
[qwen2, qwen3] — it will run, but no published number describes it)
That note exists because the difference is not subtle. The same binary, the
same six messages, the same minute — scripts/smoke.sh, 7 Sep 2026, M-series,
Metal, counting only answers that are unreadable rather than wrong:
| model | answers a person cannot read |
|---|
| qwen3-4b (the catalog default) | 0 of 6 |
| gemma3-4b | 4 of 6 |
Gemma3's four were walls of </tool., Do not be. eight times over, and a
digit stuck in a loop where the date belonged. Re-derive it yourself with
scripts/smoke.sh ~/models/<folder>; it needs a model and about four minutes on a built tree — it builds the binary first.
No separate tokenizer file needed. Tacet reads the tokenizer out of the
GGUF — vocabulary, merges and token types — so a file from Ollama or LM Studio
works as it is. TACET_TOKENIZER=/path/to/tokenizer.json still overrides, for a
GGUF that carries no tokenizer inside it.
Then:
tacet # interactive shell
tacet chat --message "what's 125 * 8" # one shot
tacet tools --schema # the exact schema the model sees
tacet why "summarize my changes" # which tools and which guide that message reaches
tacet eval # 78-case behavioural suite
Tools
Out of the box, with nothing installed:
calculate · time · calendar · read_document · create_document · edit_document · find_file · run_code · write_code · git · remember · archive · checksum · search_filter · message_intent
Documents are real OOXML — an .xlsx produced by Tacet contains a working =SUM(), not a pre-computed number.
git is read-only: status, diff, log. It reads a diff so the model can write a commit message; it does not commit, push, or change a branch. Nor does the repository get to run anything through it: hooks, fsmonitor, external diffs, textconv, filter drivers, gpg and promisor fetches named in a repository's own .git/config are switched off on every call. That needs git 2.31 or newer; an older git is refused with git_unsupported rather than run unprotected.
archive lists or extracts a .zip with the workspace's own inflate. It refuses the whole archive, never just one entry, in any of these cases: a name would escape the destination; an entry is a symlink; the declared sizes cross the caps; a name repeats or is both a file and another entry's folder; two entries share the same bytes; a central record names a different entry than its local header; or a deflate stream is a run of more than 1024 blocks that decode to nothing. Every gate up to the last runs on the central directory and the local headers, so both actions apply them. The last one, like the CRC and the declared-vs-actual size, is checked on extract only, because listing decodes nothing: an archive built from that flood lists as one 0-byte entry and is refused when it is extracted. That is why a listing labels its numbers "declared" and does not report them as sizes. The CRC is checked for every entry, including one whose recorded CRC is 0 (that used to be skipped). Extraction always goes into a new directory, whose name is rotated until it is free, so no argument can make it overwrite anything. Extraction is all-or-nothing: if any write fails, for example because the destination filesystem folds two names into one (APFS treats NFC and NFD spellings as the same name), the new directory is removed. It is removed only if it is still the directory that run created. A directory swapped for a link, or for anything else, is left in place and the error says so.
checksum is SHA-256 over a file: the digest, or a comparison against a published one, or against a second file. A mismatch comes back as an answer, not an error.
search_filter and message_intent are the two whose whole job is the arguments. They compute almost nothing; what they carry is a schema with closed-vocabulary slots — price is free, cheap, mid, premium or any and the automaton cannot emit a sixth — so they are where the grammar stops being a safety property and becomes a capability. Neither touches the network or the disk, which is what lets a benchmark built on them run identically on every machine.
run_code's end state is measured too: nothing the script starts is still running when the tool returns. On macOS the sandbox-exec profile refuses the spawn outright (spawn EPERM, measured 7 Sep 2026); on Linux bwrap tears the pid namespace down with --die-with-parent; and the process group is now swept on the normal exit path as well as the timeout one, which it was not — a script that exited cleanly after starting a background child used to leave it running while the tool reported success.
run_code executes behind a sandbox that blocks the network. On macOS that is sandbox-exec; on Linux, bwrap. If no sandbox is available, the tool is removed from the catalog rather than run unprotected — the model is never handed an unguarded interpreter.
Addons
Everything with reach beyond the working directory is an addon, and every addon is off until you install it. Not disabled — absent. A closed addon's tools are not in the catalog the model is shown, so it cannot call them, mention them, or fail at them.
tacet addon list # what exists, what is installed
tacet addon install shell # asks before it writes anything
tacet addon close shell # keep the config, take the tool away
| Addon | What it opens | Where the line is |
|---|
web-search | web_search, web_fetch | your own SearXNG — local under Docker, or an address you already run |
shell | shell | only the programs you list, and no shell interpretation |
workspace | (no new tool) | named directories the file tools may also reach |
http | http | only the exact hosts you list, HTTPS only, no redirects |
db | db, and db_write only if you list a file and the session is at a terminal | read-only SQLite by default, over the sqlite3 binary already on your machine |
clipboard | clipboard | reads and writes the system clipboard |
agent-sessions | sessions_list, session_read; tacet chat --from <agent>:<id> and /resume-from | read-only: each agent's own session folder for Claude Code, Codex and Antigravity — looked up when a session is read, so an agent installed later is found — or only the folders the install names, if you name any; credentials refused by name; no network, and the other agent and its model are never run. Reporting back is tacet handoff, which needs no addon — see Other agents' sessions |
Two things worth knowing before you install shell:
There is no shell. The command runs as program + a list of arguments, never through sh -c, so ; rm -rf / arrives as an argument — nothing parses it. And the allow-list is not checked after the fact: it is the argument's schema, so the constrained decoder cannot generate a program name outside it.
Allowing a program allows everything that program can do. curl is network access; git can push. So shell sits behind the same approval question as web_search and http: once a turn has touched personal data, every call that could carry it off the machine stops and asks you first. The question is only ever put to a person at a terminal. In a --message run, or a shell whose stdin is a pipe, the answer is a silent no and nothing is read from the pipe — a y on the next line is your next message, not consent. At the question only y or yes and Enter is a yes; Enter alone, Esc, ctrl-c and ctrl-d all say no, and ctrl-c there refuses the call without ending the shell.
There is a second question, and it does not depend on personal data. Once the session has read text from outside the machine (a web page, an http or MCP answer, a transcript imported from another agent), every shell and db_write call asks first, because that text may be what told the model to run it. The question says this in its own words. It does not claim that data will be sent. A "no" covers that one command only.
Writing to a database is a second tool, not a flag. db is read-only and stays read-only: its lock is sqlite3 -readonly, measured on your own binary before the tool is built. If you want a database changed, you name the file — data/app.db, relative to the project — while installing the db addon, and a separate db_write tool appears. Name nothing and it does not exist — and neither does it in a session with nobody at a terminal to approve it (a --message run, or a piped stdin), since a tool whose every call has to be refused only costs the model a turn. Not existing is the only gate that holds: the SQL is free text, so a model that has the tool can always spell DROP TABLE, and the answer is for the tool not to be there.
Every db_write call is measured before it happens. The statement runs first against a copy of the file, and what you are shown is the difference that copy actually took: objects created, dropped or redefined, row counts moved, journal mode changed. Then it asks you, at the terminal: Apply it? [y/N]. (Until 10 Oct 2026 that question could not be answered at an interactive terminal at all: the turn indicator erased it and swallowed the y, and the shell sat waiting until it was killed, so nothing was ever written — audit H1. cargo test -p tacet-cli --test consent_reaches_the_person now installs the addon with a writable file, types y and then n at this question through a pseudo-terminal on macOS, and counts the rows the file holds afterwards; on Linux and Windows that path has not been run.) Nothing is written until you say yes, the copy that was measured is left beside the database as <name>.tacet-backup, and the question is asked again for the next statement — a "no" is about one statement, not about the session.
Two things it does not promise. It is not one statement: sqlite3 re-parses the string it is given, so SELECT 1; DROP TABLE t runs both halves — measured, and the reason the effect is shown rather than the statement filtered. And allowing a file allows everything SQL can do to that file, DROP and a WHERE-less UPDATE included. What holds is the file boundary: -safe refuses ATTACH, VACUUM INTO and writefile(), all three measured at startup, so the statement cannot reach a second file.
Continuing another agent's work. With agent-sessions installed, tacet chat --from claude:<id> (or codex:, antigravity:), or /resume-from <agent>:<id> inside the shell, reads that session and sends a summary of it (goal, last request, open plan, files touched, last steps) above your next question. The summary is capped at 1,200 characters, about 480 tokens. The full transcript stays in the session's store, and the model gets only a reference to it. The session starts tainted and is treated as outside text, so outbound and host-acting tools ask first. If the other session worked in a different folder, Tacet asks before moving there, and it never moves in a --message run or a piped shell. /resume-from with no argument lists the sessions you can continue.
Third-party tools do not plug in here — they plug in through MCP.
Skills
A skill is a Markdown file with trigger phrases and a short piece of guidance. When a message matches, exactly one skill is fenced into that single turn's prompt — never into the system instruction. That distinction was measured: embedding guidance in the system instruction pushed a small model toward explaining the task instead of calling the tool.
---
name: calc
triggers: [how much, how many, times, percent]
---
Do arithmetic with the `calculate` tool. Never compute it yourself.
Drop it in ~/.tacet/skills/.
Which skill a message actually gets is checkable without running a model.
tacet why prints the guidance block alongside the tool ranking: the winning
skill, its score, the triggers that fired, and the runner-up with its margin.
That last part is not decoration — a tie is broken by store order, so a one-point
win is a decision nobody made on purpose. It is also the cheapest way to catch
the failure this repository keeps producing: a skill's trigger list and the
router's profile triggers are two separate lists, and nothing compares them.
$ tacet why "how much is a ticket to Istanbul"
the guidance block
calc (11) · how much is
That is the arithmetic guide on a price question, and no model had to run to see it.
Run over the whole suite, that check found more than a price question. Of the
166 steps the model suite scores, 42 English steps matched no skill at all and
13 matched the wrong one, and every one of the 47 Turkish steps was unguided —
the shipped skills had no Turkish triggers, in a suite whose Turkish half is the
larger one. Some were plain once seen (create-document triggered on the bare
word report, so "What does the file report.md say?" was handed the guide for
WRITING a document). One was invisible from either list: lowercase maps a
capital I to ı, which is correct Turkish, so an English sentence opening
with a capital-I word lost every trigger beginning with i — "Insert a header
line into document.md" matched nothing. The mapping did not have to change; the
comparison did.
And the sentence that makes the model reach for the web was being withheld
from web questions. The nudge — one line, the measured reason a small model
calls web_search at all — fires on the dominant intent profile, which is a
different rule from the one that ranks the tools, and nothing checked it. Three
ways it failed: search the web for the exchange rate, the most explicit
internet request there is, scored web 3 against files 6, because Files
owns bare search and Web only had the word web; one word of diary flipped
a weather question (the weather in Istanbul tomorrow → clock 8, calendar 8,
web 7); and is there a train strike going on in France lost to the learned
extraction head, which scored 12 with no written trigger behind it at all. The
tool ranking survived all three, which is precisely why eval --routing — the
measurement of the ranking — showed nothing. slot_gate's boost is documented
as raising a score and never overruling one; it now also cannot decide
dominant(), so when the written table has an opinion it decides, and when it
has none the learned half is all there is and still does.
the_web_nudge_reaches_a_web_question pins nine questions that must nudge and
eight that must not.
the_guide_and_the_expected_tool now holds both counts at zero over the whole
suite, with one exception carrying its reason in the source, and asserts that no
irrelevance case is handed a guide at all.
The same question, asked of the benchmark corpus, found seven more. A tool
that is not in the prompt cannot be called, so a benchmark step whose expected
tool falls outside the nine is not a hard case — it is an impossible one, and
every run books its failure against the model. tacet bench check has been able
to say this since it was written and nothing ran it. Run over all 947 steps
against the default catalog, it found eleven: four in the Turkish and task
sets, and seven in benchmarks/core/ — German, Russian and Chinese. The cause
was one omission with a date on it: Memory, Web, Archive and Integrity
were given triggers in the other five languages when a natively-authored
benchmark in each was gated; Files, Document and DocEdit were not. So
"再加一行" — add another line — reached no profile at all and edit_document
sat outside the nine. All eleven now route, eval --routing was unchanged at
166/166 with mean rank 1.33 (it is 177/177 at 1.32 since the eleven
agent-session cases joined it), and every_benchmark_step_can_pass runs the check
over the whole corpus on every cargo test.
MCP
Connect servers you run yourself in ~/.tacet/mcp.json. Their tools join the catalog and pass through the same four gates as built-in ones — a remote tool gets no privileges a local tool doesn't have.
A server's answers always taint the session and always count as outside text, and no setting can turn that off. The one per-connection safety setting only makes a server stricter: always_ask asks before every call to it, whether or not the session is tainted.
{
"connections": [
{ "name": "home server", "url": "https://example.com/mcp", "always_ask": true }
]
}
Other agents' sessions
If you also use Claude Code, Codex or Antigravity, the work they did is on your
disk, in three formats nobody documents as stable. With the agent-sessions
addon installed, Tacet can list those sessions, read one, and carry on from it
— tacet chat --from claude:<id> or /resume-from (see
Addons). tacet handoff reports back what Tacet did, so the other
agent sees it the next time you open the session there. The design, the threat
model and every version range are in
docs/adr/0001-agent-sessions.md; the
reporting-back protocol, with the rule text to paste into each agent's
instructions, is docs/agent-handoff.md.
What it reads. Only the folders the install names, which by default are each
agent's own: $CLAUDE_CONFIG_DIR or ~/.claude, $CODEX_HOME or ~/.codex,
and the three ~/.gemini/antigravity* roots. Inside them only .jsonl, .json
and .md files are opened, through one function that refuses credentials by
name on both the given and the resolved path — auth.json, .credentials.json,
state.vscdb, *.key, a .claude/settings* file, anything with token or
secret in its name, and copies of those — and refuses a symlink to one or (on
Unix) a hard-linked second name for one. Antigravity's conversation databases are
SQLite with protobuf inside; they are dated for the listing and never opened. A
Codex rollout compressed to .jsonl.zst is listed and refused with the advice
to run zstd -d, because reading it would need a new dependency. These roots
are handed to the two session tools and to nothing else: read_document cannot
reach ~/.codex/auth.json because you installed this addon. Reading trusts
the versions somebody measured — Claude Code 2.1.263–2.1.293, and Antigravity
IDE 2.5.5 on macOS only, where the version is read from the installed app's
product.json and trusted only for a session written after that file — and
reads any other version, every Codex version and every agy session with a
warning you see. Antigravity's own files record no version, so a read the
installed IDE cannot vouch for carries that warning, naming the version that
was measured. agy 1.1.10 was measured too, but its version cannot be read
without running the binary, so no agy read reaches that range.
What reaches the model. A summary capped at 1,200 characters (goal, last
request, open plan, a line for each of Antigravity's implementation_plan.md
and walkthrough.md, files touched, last steps), with every chat-template
marker in it neutralised, and a reference to the full transcript — those two
documents included, whole — in the session's own store. Reading a transcript taints the session and counts as outside text, so
every outbound tool, shell and db_write asks first.
No network, and no other program. The readers are file reads. The addon
declares no network, nothing behind it starts the other agent or its model, and
reading a session costs nothing. cargo test -p tacet-cli --test network_monopoly fails the build if any crate starts claude, codex or
agy where a model tool could reach it. The one command that starts another
agent's binary is one you type, tacet handoff note codex <id> --program <path>, described in What makes it different
above.
How it reports back. Continuing a session starts a handoff record,
<config>/handoff/<agent>-<id>.json with a rendered .md twin, which Tacet and
the other agents update through tacet handoff update. It starts with the
session as Tacet found it, labelled as the other agent's own work, and no
tasks: what it reports as progress is only what tacet handoff update has
written since. It holds a summary of the other agent's transcript, so tacet sessions --purge deletes it along with your conversations, the receipt log
and the transcript backups handoff note keeps. Getting it to the other
agent has two paths, and the first is the one to use:
- The agent's own hook (
tacet handoff install claude|codex|antigravity).
Claude Code runs tacet handoff hook claude when a session starts or is
resumed and puts the record in front of the model (measured below — except
a --fork-session resume, which runs no hook). Codex is documented to do the
same, and Antigravity to get it as a PreInvocation ephemeralMessage from a
hook named tacet-handoff, in the shape its own documentation gives;
neither has been seen doing it (Not measured below). Antigravity CLI
1.1.10 loaded Tacet's hook entry on 11 Oct 2026 and never ran it: it read
the workspace .agents/hooks.json as one named hook, then stopped at Google
sign-in before any model call. No session file is touched; the only thing
of the agent's ever written is its settings file, by install --path <file>, after showing the change and asking, with the old file kept beside
it.
- A note in the agent's own files, opt-in and fail-closed.
tacet handoff note claude <id> appends one user-role entry, [Note from Tacet, a local assistant, not from the user] …, chained to the session's last entry; --fork
writes it into a copy under a new id and leaves the original untouched. Each
write asks y/N (no terminal, no write), refuses a session the agent has open,
refuses a version nobody measured a write on, backs the file up first, and
rebuilds the chain afterwards — truncating the file back if the earlier
history would no longer load. The Codex rollout append and Antigravity's
task.md checkboxes exist and are refused on every version until a write is
measured on one. note codex --program <path> sends the note through Codex's
own app-server and has only been run against a stand-in server.
Measured, 9 Oct 2026, on one Mac (Apple M4 Pro, macOS 26) with Claude Code
2.1.285: four experiments on two throwaway sessions, run headless with
claude -p:
| written into the transcript | what --resume did |
|---|
| E2 | a user + assistant pair, chained to the last entry | loaded it, and the model believed it had written the assistant text itself |
| E3 | one user-role note from Tacet | loaded it and credited Tacet, not the user |
| E4 | the same note with a wrong parentUuid | dropped every earlier message from the context, silently |
| E5 | nothing; --resume --fork-session | a new session id; the original file's sha256 unchanged |
That table is why only the user role is ever written (E2), why the chain is
re-checked after every write and rolled back (E4), and why --fork exists (E5).
The experiment cost about one dollar in model turns, which is why it is run by a
person and never in CI; scripts/session-roundtrip.sh repeats it. It was
repeated on 11 Oct 2026 on the same version: E1, E3, E4 and E5 as measured (E3
once the script's judge learned to read past Markdown emphasis — re-judged from
the saved output, not a new paid run), and E2 differed: the model did not claim
the appended assistant text as its own. Only the user role is written either
way, so nothing follows from it. The Claude
Code write range is that one version, 2.1.285.
The SessionStart hook measured on 11 Oct 2026 with Claude Code 2.1.285, on
the same Mac (macOS 26.5.2): four headless claude -p turns (they reported
$1.37 between them) in a throwaway project whose .claude/settings.json held
exactly what tacet handoff install claude prints. A --resume, and a fresh
start under --session-id, each with a record written beforehand, ran the
hook: Claude Code logged that it "provided additionalContext", kept that
context in the transcript, and the model — asked without naming Tacet —
quoted a task title only the record held and said Tacet had left it. A start
with no record left nothing in the transcript. The systemMessage was stored
in the transcript as a hook system message; -p has no screen, so whether the
interactive one shows it was not seen. --resume --fork-session ran no
SessionStart hook at all, Tacet's or any other: the fork answered from the
copy of the record the earlier resume had left in its history, and missed a
task added since. Resume the original session, not a fork, to hand it the
current record. Not measured: every Codex
path, its hook included (Codex was not installed on that machine; its reader is
built from Codex's source and tested on fixtures), whether Claude Code's
interactive screen shows the hook's systemMessage, the Antigravity hook
running and its ephemeralMessage reaching the model, and any write into
Antigravity's files (the GUI was not driven). The Antigravity hook got as far
as being loaded, on the same Mac on 11 Oct 2026: three agy -p starts in a
throwaway workspace holding exactly what tacet handoff install antigravity --path writes, the CLI logging "loaded 1 named hooks" (0 with the file moved
aside), and each stopping at Google sign-in before a model call, because the
session running it reached the Mac over SSH, where the CLI could not use the
stored sign-in. docs/agent-handoff.md has the table and the one signed-in run
that is left.
Whose terms apply. Claude Code's documentation says its transcript format
"is internal to Claude Code and changes between versions", which is why reading
outside the measured range warns and writing outside it is refused.
Antigravity's terms (§6) say that using third-party software to access
Antigravity is a violation, and its FAQ says writing to its storage is not
supported. Tacet reads files you own on your own disk to show you your own
work; it does not drive Antigravity, call its model or use its credentials, and
it writes nothing of Antigravity's unless you pass
--experimental-write-task-md, which is refused on every version today and
prints that warning when you try. Whether reading your Antigravity history this
way is within the terms you accepted is your call to make, not this page's.
Architecture
tacet-cli ──────────► terminal shell; drives the turn loop
│
├── tacet-eval ──► cases, scoring, reports
│
├── tacet-grammar► ArgSchema → constrained-generation grammar (PDA + token mask)
│ depends on tacet-kernel ONLY — no inference stack
│
├── tacet-engine ► contracts: Prompt, EngineProvider, TokenCounter
│ FakeEngine (default) / CandleEngine (--features candle)
│
├── tacet-tools ─► concrete tools + ToolExecutor + Router
│ │ sessions: other agents' transcripts (read), the
│ │ handoff record, the opt-in write-back
│ ├── tacet-zip ──► hand-written zip/deflate/crc32 → OOXML
│ └── tacet-web ──► THE ONLY CRATE WITH A SOCKET (with tacet-mcp):
│ search, page fetch, the SSRF gate, the addon registry
│
├── tacet-skills ► trigger-matched guidance, fenced into one turn
├── tacet-memory ► notes on disk, injected only when relevant
│
└── tacet-kernel► the CONTRACT: Tool, ArgSchema, ToolOutcome, DataStore,
Catalog, Constrainer, defang
Arrows are dependency direction. tacet-kernel depends on nothing, so the contract never bends under pressure from an implementation. tacet-engine deliberately does not know tacet-grammar: the contract lives in the kernel, its implementation in the grammar, so a run without constraints doesn't compile grammar code it never uses.
The guarantee is not tied to this engine
The whole constraint contract is three signatures over &mut [f32] and u32:
fn session(&self) -> Box<dyn ConstraintSession>;
fn mask(&self, logits: &mut [f32]);
fn advance(&mut self, token: u32) -> Result<(), ConstraintError>;
Nothing in it names a model, a tokenizer, a device or a file — so any runtime
that can hand over a logit slice and take back a token gets the same guarantee:
llama.cpp through a binding, ONNX, a hand-written loop. cargo tree -p tacet-grammar is the kernel, serde and thiserror; there is no inference
dependency to adopt along with it.
That is a recent correction rather than an original virtue. The contract used to
live in tacet-engine, which made a runtime-independent property look like
something this engine provided, and made anyone who wanted it depend on GGUF
loading and prompt budgeting to get three method signatures.
Three things keep it honest. A test in the kernel implements a working constraint
importing only that module, so a signature that starts needing an inference
type stops compiling. cargo run -p tacet-grammar --example no_engine shows it
end to end against a pretend runtime in thirty lines. And the claim is checked
from OUTSIDE the checkout, which is the only place it can actually be false —
measured 6 Sep 2026 against the registry, not against these paths:
cargo new x && cd x
cargo add tacet-grammar tacet-kernel serde_json
cp .../examples/no_engine.rs src/main.rs && cargo run
cargo tree | grep -c tacet-engine # 0
The example prints:
after `weather({"city":"London",` the grammar allows: ["\"", " "]
`kelvin` is in the vocabulary and closed: true
kelvin is in the vocabulary and the schema does not allow it, so its logit is
-inf before the sampler runs. No temperature, top-p or beam search can
select it. That is the difference between checking output afterwards and making
the bad output unrepresentable.
What it scores — honestly
Most of this repository's green numbers come from a mock engine: they measure
Tacet's own logic, deterministically, in milliseconds. That is the right tool for
CI and the wrong one for the question people actually ask, which is whether a 4B
model on your laptop picks the right tool. So here is that number, produced on
real weights and checked into crates/tacet-eval/baselines/ with the model's
fingerprint, so the next run can be compared against it rather than against a
memory of how things used to go:
tacet eval --tool-selection --model qwen3-4b # both languages, ~48 min
tacet eval --tool-selection --model qwen3-4b --turkish # Turkish only
An interrupted run no longer starts over. --journal <dir> writes every
finished case to that directory the moment it ends, and a later run skips what
is already there — so a Ctrl-C at case 180, a closed lid or a reclaimed rented
box costs the current case rather than the 47 minutes behind it. The directory
is stamped with the model fingerprint, the engine, the catalog, the commit, a
hash of any uncommitted edits (untracked files included), a hash of the prompt
text and tool descriptions, the router options and a hash of every case. A run
against a different stamp, or holding a case rewritten since it was measured,
exits non-zero instead of quietly restarting: cases from two builds assembled into one report would be a mixture
presented as a measurement, and nothing in the JSON would say so.
tacet eval --tool-selection --model qwen3-4b --journal ~/.tacet/journal/qwen3-4b --json > after.json
Keep the journal out of /tmp. The point of it is to outlive the session that
started it, and /tmp is the one place that is emptied behind your back: a run
here that wrote to /tmp with no journal was stopped at case 150 and the
directory was cleared before it could be read — 48 minutes with nothing to
resume from. scripts/nightly-eval.sh picks the directory for you, under
~/.tacet/nightly/, keyed by what was built (see CONTRIBUTING).
| |
|---|
| tool selection | 136/160 · 85.0% |
| irrelevance gate | 24/24 · 100% |
| step chain | 166/190 · 87.4% |
| answer quality | 38/47 · 80.9% |
| could not be measured | 3 out of turns · 4 cut off |
Qwen3-4B-Instruct-2507 Q4_K_M on Metal, 184 cases in both languages, 47.7 min
— the weights tacet models download qwen3-4b fetches, pinned by digest, with
the fingerprint recorded in the baseline. Measured 6 Sep 2026 and committed in
e9787ba; the baseline in crates/tacet-eval/baselines/ is that run, with
wall_ms populated, so the wall clock is re-derivable rather than remembered. It
has not been re-run since the eval started building each pass's prompt the way
the shell does (audit M22, M23, October 2026), so today's code may score these
weights differently.
These numbers are LOWER than the ones this page carried this morning, and the
drop is the instrument getting more honest rather than the model getting worse.
An audit of the eval found that a turn which spent all four passes calling tools
and never said a word to the user scored as a HIT. Three steps did exactly that.
The comparator, run against the morning's baseline on the same weights, is as
clean a confirmation as this kind of change gets:
fixed 0
broke 3 write_code-script, tr-web-site-oku, tr-web-arama
delta -1.6 points 95% CI [-3.8, +0.0]
verdict: NOT DISTINGUISHABLE from no change at 95%.
at these rates (0 fixed, 3 broke) this instrument needs about 338
paired cases to call a 1.6-point effect; this suite has 184.
Nothing was fixed, three things broke, and they are by name the three the change
was written for.
A fourth divergence, and the largest. The shell pushes the model's own
generation into the history as an assistant turn before the tool result, with
a comment recording why: fed only the RESULT, the model saw a context-free line
with the user's question below it, took the question for unanswered, and called
the same tool again — up to the turn limit, without ever answering. That is the
OutOfTurns ending this suite reports. The eval pushed only the result. So
it measured a model with no record of its own actions, counted the repeats
against it, and published the number as the shell's — and it is also a deviation
from the template these weights were trained on, where a tool response follows
the assistant turn that asked for it. Every divergence found tonight ran the
same way: the eval stricter than the program, in the direction that looks like
the model failing.
And a third divergence was found after those numbers were taken, so the table
above still carries it. The harness built its catalog with production_catalog
— which returns three things — and kept only the first. The second is run_code's
attempt counter, and the shell resets it at every user turn; the eval never
reset it at all. A two-step case that spent both code attempts in step 1 began
step 2 with the budget already gone, and the call came back "two attempts
exhausted" without running. chain-code-write is that case by name: in the
baseline above, step 1 calls run_code twice and step 2 calls nothing and is
scored CutOff. The failure was booked against the model; nothing in the report
said the budget was the harness's. The handle is carried now, in the eval and in
bench, and the_harness_matches_the_shell asserts it survives — but the table
above was measured before the fix and has not been re-run, so it is quoted as it
was taken.
The last row is new and is the point. A step now records WHY it stopped, and
one that could not be measured is a pass on no axis: three ran out of turns, four
stopped on the token cap. Under the old rule an engine error scored as a pass on
the irrelevance axis — a dead engine calls nothing, and the rule was "called
nothing" — so a run in which generation broke read as 0/160 tool selection and
100% irrelevance, and the report called that the safety property holding.
Two more scoring corrections are in the same run. evidence is checked against
the ANSWER, where it used to be searched in a pool that also held the tool's own
output — so a correctly called tool satisfied the claim whatever the model said
(tr-hesap-ortalama asserts 20, the model answered 30, the step passed).
And forbidden, documented as "tools that must NOT be called", is compared to
the tools that were called; it was a substring search over prose, which made
1505 assertions across the benchmark corpus inert and failed correct runs whose
answer happened to write a tool's name.
This block is the Metal run; the model table further down is the same suite on
a rented RTX 3090, and that table has NOT been re-measured — it still reads
against the older code, which is why the two now differ on more than the answer
axis. The model table's 3090 wall times remain hand-recorded, because wall_ms
can only be populated by whoever runs the card.
And the GPU run is now a committed artifact, not a remembered number. The
same 184 cases on a rented RTX 3090 (vast.ai, CUDA 12.4, driver 580.159.03,
6 Sep 2026): 7m 07s against Metal's 47.7 min — 6.7×, with the report in
baselines/qwen3-4b-cuda-3090.json:
tool selection 125/160, irrelevance 24/24, step chain 153/190, answer 41/47.
The whole rental cost about twenty cents; cargo build --features cuda took
2m 47s on fourteen cores.
That file is the second 3090 run of the day. The first, which this paragraph
quoted as 7m 34s and 6.3×, was replaced in the same directory by a later run the
same evening, at 4af5484 (the batch that made CutOff zero) — seven commits after
the first run's 511f8db, and two of the six between them change what the model is
shown and how its answer is scored (384d41e, the skill guides; 1f41fc2, four
places the harness threw a right answer away). The paragraph was not
re-derived with it until 10 Oct 2026. Both figures now come from wall_ms in
the two committed files, and the_readme_quotes_the_baselines fails the build
if this page and those files disagree again.
The catalog was 12 tools there against 15 here, and that is the trap this
comparator was built for. calendar is macOS-only; run_code and
write_code need a sandbox, and inside an unprivileged container bwrap cannot
create a namespace at all (Operation not permitted) — so all three were absent
and 23 cases would have read as regressions. --compare excluded them by name.
Re-derived 10 Oct 2026 from the two committed files, no model needed:
these two runs were on DIFFERENT DEVICES (metal -> cuda). The token budget
follows the device's KV cache …
these two runs did not have the same tools, so 23 case(s) … are EXCLUDED
not in after: calendar, run_code, write_code
paired on 161 cases
delta +2.5 points 95% CI [-1.9, +6.8]
sign test p = 0.4240
verdict: NOT DISTINGUISHABLE from no change at 95%.
And the suite is deterministic, which nothing here had ever checked. Two
identical runs on the same box, same commit, same weights: 184 of 184 case
verdicts identical, every axis count identical, only the wall clock moving
(7m 35s against 8m 01s, 5.8% — machine noise). That matters more than it sounds:
it means --compare between two runs on one machine measures the CHANGE and
nothing else, so a one-case difference is a real one. It also means the gap
below is device, not variance.
Same weights, two machines, no measurable difference in what the model chose —
which is the result worth having, because it says the numbers on this page are
about the model and not about the card. The device line is new: kv_cache_budget
is 2.5 GB on Metal against 8 GB on CUDA, so the generation cap was ~14,000 tokens
on one and 52,019 on the other, and in the first comparison (the 3090 run since
replaced) ten of the cases that moved were ones that had more room to think.
That is a difference between two machines, and the comparator says so instead of
letting it read as a difference between two builds. One more thing the
comparator cannot say, so this page does: the committed Metal file predates the
batch the committed 3090 file was run after, so the pair above is two builds as
well as two machines — and still not distinguishable from no change.
Seven minutes on a GPU — and the model matters more than the card. The same
184 cases on a rented RTX 3090: 7m 07s with these weights in the committed run
above, and 53.7 minutes with Qwen/Qwen3-4B, the other model of the same name
and size, which spends a median of 237 generated tokens per turn against 19. That
second figure, and the token medians, were recorded by hand on a rented 3090 in
the first days of September 2026 (first quoted on 5 Sep), against the code of that
date, with no report kept — like the model table below, they cannot be re-derived
from this repository. That run also sent the hybrid model no soft switch — the
eval did not append one until audit M23 — so it is Qwen/Qwen3-4B in its
default thinking mode, not under the /no_think the shell puts on most turns,
and the gap a user of the shell would see between the two files has not been
measured. Two builds apart, a gap of more than seven to one is still
the point. Before reaching for a bigger card, check which file is loaded. And
when you do reach for one: running several copies of the suite side by side
wins nothing, because one stream already saturates the card (measured — two streams give ~28 tok/s each where one gives
~55). Batching several sequences into a single forward pass is the thing that
scales, and cargo run --release --example batch_decode --features candle,cuda
measures where it stops: 124 tok/s at batch 1, 504 at batch 32 on that card.
Turkish scores higher than English — 61/69 against 99/115 — which is the
opposite of what the effort spent on Turkish defects would suggest, and it is only
visible because the two now run together. The routing eval always measured both;
this one measured one language at a time and never both, so the expensive
measurement was looking at the easier half and calling it the score.
The irrelevance gate is the one to read first. Twenty-four messages that must
not reach a tool, and none of them did. A local assistant that reaches for a
tool on "thanks, that's all" is worse than no assistant, and that is the failure
this number rules out.
The failures, because a score without a diagnosis is a number to be proud of
rather than one to act on. This diagnosis is of the run before the one in the
table: the baseline as first committed in 3c2c3e4 (4 Sep 2026), which the
corrected scoring replaced two days later, and it is the run the fix below was
made against. Twenty-seven of its 184 cases failed (the table's run: 24), and
they are not scattered. Counted off that file:
| |
|---|
| called the wrong tool — 10 | 5× edit_document → read_document, 3× web_fetch → web_search, and one each of read_document → find_file and run_code → calculate |
| answered in prose, no call — 8 | including two that are ours, not the model's: one turn died on engine error: constraint rejected the token: 32, another on generation was cut off halfway |
| wrote a call in a syntax nobody taught it — 5 | three of them the same glued shape, which Turkish produced and English never did |
| declined a tool it had — 5 | web_search, git, calendar, remember |
find_filepattern: "bütçe" ← the tool name glued to its first argument
The two largest are worth naming because neither is what you would guess. Five
cases read the document and then said "no tool is available to edit or update
the content" — with edit_document sitting at rank 1 in that turn's own prompt,
confirmed with tacet why. And three handed a URL called web_search instead of
web_fetch, which is the tool description's own fault: it said "use only when a
search result summary is not enough", which is exactly the wrong instruction for
a message that already carries the address.
Both are fixed, and this is the first time the instrument has called an effect
real. The descriptions now say what those tools can do, the router puts a tool
that requires a url first when the message carries one, and a call fenced as
```json is recovered. Measured by re-running the whole suite on one rented
RTX 3090 against a same-weights, same-host run made before the change:
paired on 184 cases
before 146/184 (79.3%)
after 155/184 (84.2%)
fixed 12 broke 3
delta +4.9 points 95% CI [+1.1, +9.2]
sign test p = 0.0352
verdict: REAL at 95%.
Nine of the twelve are the cases named above. The table further up is the Metal
baseline and has not been re-measured since, so it still reads pre-fix — the two
numbers are from different machines: --compare refuses to pair two different
models, and when the catalogs differ it leaves out every case that touches a tool
one side lacked and lists them, which is why this one is quoted on its own terms
rather than folded into the table.
And the "after" report behind this verdict is not in the repository.
CONTRIBUTING asks that a claimed model improvement arrive as a PR that updates
the baseline and pastes the verdict; this pasted the verdict from a rented box
and left the other half where nobody can pair against it. baselines/ does hold
two 3090 runs — qwen3-4b-cuda-3090.json and tr-corpus-qwen3-4b-cuda.json —
but neither is the "after" of this pair: the first passes 149 of its 184 cases
whole, not the 155 above, and the second is a different benchmark. Both were
committed the next day, 6 Sep 2026, after further batches of changes. (This paragraph said baselines/ held two files and
neither was a 3090 run; it holds seven, re-counted 10 Oct 2026.) The fix is
still not to re-word this paragraph but to re-run and commit the report — with
identity.model_path reduced to a bare file name, which cargo test -p tacet-eval --test baselines already enforces.
A refusal is usually ours. An earlier run had six of them, and the cause was
the system prompt: it opened with "an assistant that runs entirely on the
device. Data never leaves the device" — a true statement about the architecture
that a small model reads as a statement about its own capabilities, and then
declines to use web_search, which tacet why confirmed was sitting at rank 2
in its own prompt. Five survive in the numbers above, and the same test applies
to each: run tacet why on the message before blaming the model.
What happened when we fixed that one is the most useful thing on this page.
This was the earlier, English-only suite — 115 cases, not the 184 above, so the
numbers below do not belong to the table. The raw score moved +3, and
eval --compare ran a sign test and refused to call it:
fixed 8 broke 5
delta +2.6 points 95% CI [-3.5, +8.7]
sign test p = 0.5811
verdict: NOT DISTINGUISHABLE from no change at 95%.
at these rates (8 fixed, 5 broke) this instrument needs about 712
paired cases to call a 2.6-point effect; this suite has 115.
The block was re-derived on 10 Oct 2026 when the sizing began counting broken
cases as well as fixed ones; it had said 230.
Two runs of anything differ by a case or two for no reason. If you take one number
away from this section, take that one — and note that the instrument volunteered
its own resolution limit rather than letting a +3 be reported as progress.
Benchmarks — measure YOUR tools
The suite above is compiled in and measures this project. A benchmark is the
other direction: a file you write, run against the tools your machine actually
has — your MCP servers, your addons, your language — answering "does this
assistant call my tools".
tacet bench check my-tools.json # no model runs; costs nothing
tacet bench run my-tools.json --model qwen3-4b
A file is JSON and the whole format fits on a screen:
{
"name": "our-github-mcp",
"language": "en",
"requires": ["gh_search_issues", "web_fetch"],
"cases": [
{ "name": "open-issues-by-label", "category": "tool",
"steps": [{ "message": "which issues are labelled regression?",
"expect": "gh_search_issues",
"evidence": ["#412"],
"forbidden": ["web_search"] }] },
{ "name": "thanks", "category": "irrelevance",
"steps": [{ "message": "great, thanks!" }] }
]
}
benchmarks/example.json is a worked one, and benchmarks/en/ holds 314
English cases across eight groups — arithmetic and time, documents, files and
archives, code and git, web, memory and calendar, 45 irrelevance cases, and 29
multi-step chains. They were drafted by a multi-agent pass and then cut down by
the gate below; what survived is what a fresh install can actually be asked.
And they have now been run against a model. qwen3-4B on the rented 3090,
6 Sep 2026, all ten files, --skip-missing because that box has no calendar
(macOS-only), no sandbox and no web addon — so 145 of the 319 cases carry a tool
expectation that could be scored there. The report is at
baselines/tr-corpus-qwen3-4b-cuda.json.
| axis | Turkish corpus | the 184-case suite, same model, same kind of card (RTX 3090) |
|---|
| tool selection | 105/145 · 72.4% | 125/160 · 78.1% — 125/140 · 89.3% skipping what the box lacked |
| irrelevance | 95/97 · 97.9% | 24/24 · 100% |
| step chain | 243/289 · 84.1% | 153/190 · 80.5% |
| answer quality | 213/289 · 73.7% | 41/47 · 87.2% |
The right-hand column is
baselines/qwen3-4b-cuda-3090.json,
the same kind of rented card on the same day (6 Sep 2026) — not provably the
same machine: the suite's report has web_search and web_fetch in its
catalog, and the corpus report says its box had no web tools. It used to be the Metal run (85.0%, 100%,
87.4%, 80.9%), which made a comparison of two corpora also a comparison of two
machines and two builds. And the two columns score a missing tool differently:
the corpus ran with --skip-missing, so a case whose tool the box lacked was
not scored, while the suite counts its 20 such cases — every one a failure —
against the model. Skipping them the same way is the bold figure.
The answer axis moved 195 → 213 in one commit, and none of it is the model.
The language gate demanded evidence FOR the language and failed an answer when it
found none — and Turkish attaches its suffixes with an apostrophe, so 480'in
was split into a token in that is on the ENGLISH list, while Turkish's own
proof letters exclude ö ü because German writes them too. Eighteen correct
Turkish answers were being failed by the instrument. On the mixed 184-case suite
the same fix is worth three; on a corpus that is entirely Turkish it is worth
eighteen, which is what a blind spot looks like when you point a set at it.
16.9 points below the curated suite on tool selection, with missing tools
skipped on both sides, and that gap is the reason the corpus was written. (5.7 if the suite's
missing-tool failures are left in. This said "thirteen points", against the
Metal column.) These are sentences a Turkish user would
actually type, including a group that holds the job constant and varies only the
register; the older suite's Turkish half was written alongside the English one
and reads like it. The corpus is harder because it is more like the field.
benchmarks/tr/ holds 319 more, in Turkish, written as Turkish — 6 Sep 2026.
Not translated: the questions are about fatura, taksit, kira, veli toplantısı,
noter, mesai, and they are phrased the way people type into a phone, including
one group (agiz-kayit.json) that keeps the job constant and varies only the
register — formal, WhatsApp-short, elderly, Anatolian, no punctuation, ALL CAPS,
emoji. That axis was never measured before, and a router built from a trigger
list is exactly the thing it should be measured against. Turkish now stands at
518 cases across the tree, against 187 before.
They cost 650 drafts. Two workflow rounds authored them; an adversarial pass in
each round threw out 279 for reading like a translation, for having two
defensible answers, or for asking something the corpus already asked — the
rejection notes name the case they duplicate, which is how a 42% rejection rate
is a sign of a working filter rather than a broken one. A mechanical pass then
dropped duplicate names and messages against the whole of benchmarks/, and
bench check --portable gates every file.
Three things about the format are deliberate:
requires is not paperwork. It is what makes the runner stop when the
machine lacks a tool, instead of scoring every case that needs it as a model
failure and publishing that as a result — the same defect eval --compare was
taught to leave out of its verdict when a Linux run paired against a macOS
baseline read nineteen absent-tool failures as a regression.
And it found the same thing again, in Turkish, at four times the scale. Of
297 verified Turkish questions, 30 expected a tool the router never shows —
so they were kept out of the files and written down instead, in
benchmarks/tr/ULASILAMAYANLAR.md, with the
sentence and the tool. The shape of the list is the finding:
| tool | sentences it never reaches |
|---|
web_search | 14 — "Ankara'da şu an kaç derece", "dün akşamki maç kaç kaç bitti", "merkez bankası faizi indirdi mi", "e-devlet çöktü mü" |
remember | 8 — "dükkan pazartesi kapalı, aklında olsun", "tercihlerimi göster", "şu bilgiyi sil" |
search_filter | 3 · message_intent 2 · checksum 2 · archive 1 |
The English half of exactly this was done and is described above: seven web
questions scored zero on every profile, the triggers went in, the questions
stayed. The Turkish half was never done, and fourteen of the most ordinary
current-information questions a Turkish user asks are the proof. The fix is not
a trigger per sentence — the file says so and means it — it is to make the class
measurable by eval --routing first and decide from the measurement. That is
named as the next step rather than done under a deadline.
bench check runs before any model does, and it asks the question nobody
writes by hand: the router shows the model nine tools, so would the expected
tool even be among them? A case whose tool never reaches the prompt measures the
router and reports the model, every time it is run, forever. Checking it is free.
It earned its keep immediately. Over the first 321 drafted questions it found
22 cases whose expected tool the router never showed — and all but seven were
the ROUTER's fault, not the question's: seven unmistakable web questions ("is
there a train strike going on in France?", "which stable Rust version is the
newest one right now", "how bad is the air quality in Delhi") scored zero on
every profile, so web_search was not among the nine and they would have been
recorded as model failures forever. Those triggers are in the router now, and the
questions stayed. Seven cases were deleted instead, because their signal is one
the router structurally cannot read — an adversarial negation ("I don't want
another file lying around"), or a follow-up whose subject is only in the previous
turn ("add coffee to that list too"), which a stateless router cannot resolve.
--portable checks against the default catalog rather than yours. It matters:
the router shows nine tools of however many exist, so a machine with 29 MCP tools
attached answers a different question, and a benchmark that only passes on its
author's laptop is not a benchmark.
There is no regex, no script and no expected answer text. evidence is a
plain substring. Scoring prose against prose needs a judge, a judge is a second
model, and a second model is a second thing to be wrong.
The score is out of 100 with the four axes printed beside it, and the weights are
in the source rather than in someone's head — irrelevance 0.40, tool 0.30, step
0.20, answer 0.10. The safety axis is heaviest on purpose: a model that fires a
tool at "thanks, that's all" pays for it dearly: at the suite's proportions
one irrelevance failure costs 1.67 points and a tool case landed with its step
earns 0.29, so it takes six of them to buy one back (91.1 -> 91.2 on the top row
of the model table). Two tests pin that exchange rate: bench.rs holds the arithmetic in both directions, and the_readme_quotes_the_baselines derives this sentence from the weights and the top row. An axis with no cases is left out and the
rest renormalised, not scored as zero — a benchmark made only of irrelevance
cases is a legitimate benchmark.
The model table
Four models, the same 184-case suite, one rented RTX 3090 each, Q4_K_M unless
noted. Score is out of 100 with the weights above — irrelevance 0.40, tool
0.30, step 0.20, answer 0.10.
| model | score | tool selection | irrelevance gate | step chain | answer | wall |
|---|
| Qwen3-4B-Instruct-2507 | 91.1 | 133/160 · 83.1% | 24/24 | 162/190 | 43/47 | 6.4 min |
| Qwen3-8B | 81.3 | 128/160 · 80.0% | 20/24 | 153/190 | 37/47 | 54.6 min |
| Qwen3-0.6B (Q8_0) | 64.0 | 52/160 · 32.5% | 24/24 | 79/190 | 28/47 | 21.3 min |
| FunctionGemma-270M (F16) | 47.4 | 0/160 | 24/24 | 24/190 | 23/47 | 28.6 min |
These four rows are hand-recorded and cannot be re-measured here. They were
taken on a rented RTX 3090 on or before 5 Sep 2026 — they were first committed in
ae0893a that morning, and the rental's own commits run from 00:07 to 01:35 the
night before — against the code of that date, and no JSON report was kept for
any of them; the box is gone, and re-running them
means downloading four sets of weights onto a rented GPU again. What was
re-checked on 10 Oct 2026 is the arithmetic — each row's score recomputed from
its own counts with the weights above, by the_readme_quotes_the_baselines —
and not the measurement. The top row is not
baselines/qwen3-4b-cuda-3090.json,
which is a later run of the same weights on the same kind of card (125/160,
153/190, 41/47, 7m 07s); the rows are kept as they were taken so that the four
models are compared on one build. Until the table was split by a stray blank
line (fixed 10 Oct 2026) GitHub rendered the last three rows as loose text.
Qwen3-8B and Qwen3-0.6B were measured in a mode the shell does not give you by
default. Both are hybrid builds: a turn with no soft switch on it runs in
their default thinking mode, and when these rows were taken the eval sent no
switch at all. The shell puts one on every ChatML turn, and under the default
thinking = auto that is /no_think on 180 of the suite's 190 turns and
/think on 10 (audit M23; the eval now appends the same switch, from
tacet_engine::thinking_switch). So those two rows are the hybrid models
thinking on every turn — the nearest a user comes to that is thinking = on —
and not what a default install gets from the same weights. How far their
scores and wall clocks would move under the shell's switch has not been
measured, for the reason above. Qwen3-4B-Instruct-2507 has no thinking mode
for the switch to turn off.
Read the score with its floor in mind, and the floor is not 40. A model that
never calls anything passes every irrelevance case, which is 0.40 of the weight —
that much was already on this page. What it left out is that the same silence
also passes those cases' STEPS and their ANSWERS. Derived from the suite rather
than asserted:
silence: tool 0/160 · irrelevance 24/24 · step 24/190 · answer 24/47 => 47.6 / 100
cargo test -p tacet-eval --test the_score_has_a_floor -- --nocapture prints
that line, and the test fails if this page and the suite disagree. FunctionGemma's
47.4 is two tenths BELOW the score of saying nothing at all — it is not a weak
tool-caller, it is a model that cannot call a single tool in this format and
loses one answer on top. The number to read beside it is the 0/160. Quoting the
floor as 40 made every row on this table look about eight points better than it
is, which is more than the gap between two of them.
The 8B is worse than the 4B, and the axis it loses on is the safety one.
Twenty of twenty-four on the irrelevance gate means four messages that must not
reach a tool did. Bigger did not mean better here, and it cost eight times the
wall clock — in the thinking mode it was measured in, not under the
/no_think the shell sends on most turns; how much of the gap is the 8B and
how much is the mode is not measured.
gemma-3-12b is absent because it does not terminate. On the same card and the
same suite it emitted a median of 2048 tokens per turn — exactly the
constrained ceiling — against Qwen3-8B's 280 (itself taken with no soft switch,
in its default thinking mode), with three of five sampled turns
stopping on Length rather than on a finished call. At 18 tok/s against the 8B's
47 that is 124 seconds per case, and a full run is six hours. The verbosity is
the finding; the score was not worth the rent.
What the grammar is worth
tacet bench gap runs the same calls twice, with the automaton on and off. Same
prompt, same sampler, both columns capped at 256 tokens so only the mask differs.
The 39 calls of benchmarks/en/arithmetic-time.json, on a rented RTX 3090,
5 Sep 2026.
This table has no committed artifact, and until recently it could not have
one — bench gap printed a table and nothing else, so it was the only figure
on this page with no machine-readable form at all, on rented hardware nobody
else has. --json now writes the run with the environment stamped beside it
(weights, quantization, device, rustc, commit, peak resident), which is what a
number on a page owes the person re-deriving it:
tacet bench gap benchmarks/en/arithmetic-time.json --model qwen3-4b --json \
> crates/tacet-eval/baselines/gap-qwen3-4b-<card>-<date>.json
The three rows below predate that and stay as they are, dated and attributed
rather than back-filled: writing a JSON file today from a run on a card this
machine does not have would be inventing an artifact, which is worse than not
having one.
| model | started a call | valid if started | correct call |
|---|
| Qwen3-4B | 76.9% → 97.4% | 86.7% → 100.0% | 66.7% → 97.4% |
| Qwen3-0.6B | 20.5% → 23.1% | 100.0% → 100.0% | 7.7% → 10.3% |
| FunctionGemma-270M | 0% → 0% | — | 0% → 0% |
The 0.6B row was asked without the soft switch. bench gap sent the bare
message until audit M23 (it now appends what the shell would under
thinking = auto), and Qwen3-0.6B is a hybrid build, so that row is the model
in its default thinking mode, not under the /no_think a default install sends
on most turns; what it does under the switch has not been measured. The 4B row
is less certain: tacet models download qwen3-4b was moved from the hybrid
build to the 2507 instruct one, which has no thinking mode to switch, during
the same rental, and which file that run loaded is not on record.
valid if started reaching exactly 100% is the front-page claim, measured.
Unconstrained, the 4B writes a malformed call roughly one time in seven; with the
automaton it never does — 38 calls of 38 here, and 100% in every earlier run of
this command as well.
The 4B also STARTS twenty points more calls with the grammar on, and that is
the mask working rather than an oddity. The automaton arms at name( — so a
generation that was drifting towards a signature echo instead of a call gets the
brace forced on it and lands as a real call. Constraint is not only rejecting the
invalid here; it is pulling a near-miss over the line. That is where the 4B's
+30.7 points of correct calls come from.
The 0.6B says almost nothing, and the honest reading is that it is too small to
be asked. It starts 8 calls of 39 unconstrained and 9 of 39
constrained. Eight is not enough to measure a rate against, and both columns being 100% valid means this suite never
caught it writing bad syntax at all — not that the guarantee is unnecessary. What
it does show is the limit: +2.6 points of correct calls. Valid is syntax,
correct is judgement, and the automaton was only ever the first.
These numbers replace an earlier table, and the correction is worth more than
the table. That version had the 0.6B starting fewer calls with the grammar on
(46% → 26%), a result printed on this page as "unexplained and left in". It was
an artefact of the measurement. bench gap decided a call had started by looking
for name( in the output, and an unconstrained Qwen3-0.6B spends about a third
of its turns parroting the tool signature back:
(time(kind: "clock", target?: "what time it is"))
calendar(kind: 'date', target?: text).
The ?: is copied straight out of the tool description. That is not a call and
never becomes one — but it contains time(, so it counted as a start, and only
ever in the unconstrained column, because the mask forbids that shape. The
measurement was reading "the grammar stopped the model parroting the schema" as
"the grammar stopped the model calling a tool". Requiring the brace that
Tacet's call format demands (name({...})) separates them, and the gap closes to
+2.6. It cost the 4B's unconstrained column twenty points too: that model echoes
signatures as well, just less often.
Two things ruled it out before the dump was read, and both are worth recording
because they are the cheap checks: the mask intervenes 19 times in 16,402
tokens and never before token 30, so it cannot be steering the opening; and
forcing both columns through one identical argmax changed the result not at
all. When the intervention is that rare and the sampler is not the difference,
the remaining suspect is what the metric counts.
Speed and memory from the same runs: time to first token 392 ms (4B) / 135 ms
(0.6B) / 142 ms (270M); decode 64 / 128 / 75 tok/s; peak resident 933 MiB for the
4B at Q4_K_M, 706 MiB for the 0.6B, and 1622 MiB for the 270M at BF16 — the one
that calls nothing is also the one that costs the most memory.
Distilling a tool-caller
A 135M model cannot call tools in this format. After three to five minutes of
training on a set Tacet generated from its own benchmarks, it can. Every column
below is measured on 189 cases the student never saw — a per-case 75/25 split
of the whole suite, teacher run over the train half only. RTX 3090, 5 Sep 2026.
| SmolLM2-135M-Instruct | base | distilled | + composed set |
|---|
| composite | 46.4 | 59.9 | 60.0 |
| irrelevance | 41/44 | 36/44 | 36/44 |
| tool selection | 6/119 | 51/119 | 52/119 |
| step chain | 51/184 | 99/184 | 99/184 |
| correct call | 0.0% | 36.4% | 27.3% |
search_filter — tool | 0/17 | 15/17 | 16/17 |
search_filter — answer | 0/17 | 1/17 | 3/17 |
message_intent — tool | 0/10 | 1/10 | 2/10 |
| decode | 130 tok/s | 120–127 tok/s | 123–126 tok/s |
| peak resident | 529 MiB | 528 MiB | 529 MiB |
| training | — | 292 s | 362 s |
These replace an earlier table that was measured on the student's own training
data. That version reported 0% → 61.5% correct calls and a search_filter
score of 44.0 → 76.4. The set had been generated from every benchmark file and
the result then reported on three of them: 179 of the 1017 pairs came from the
scored cases. Held out properly the gain is real and smaller — 0% → 36.4% — and
the recipe now splits before it generates. See training/.
The training data is the teacher's correct answers and nothing else. With
TACET_DISTIL_DIR set, every benchmark step that passes writes its rendered
prompt and the call the teacher produced. A step that called the wrong tool
contributes nothing — that prompt is exactly where the student must not copy the
teacher. "Correct" is the benchmark's own pass/fail, not a judge model. Qwen3-4B
over the 569 train cases gave 865 pairs, 851 of them usable.
THE COST IS THE ROW TO READ. Teaching a 135M model to reach for tools costs
it restraint: irrelevance falls from 41/44 to 36/44 — five messages that must not
reach a tool now do. Weighting irrelevance at 0.40 makes that loss expensive,
not impossible: a large enough gain in tool accuracy still buys it, and here it
did — with the five counted against it, 51 of 119 tools found moves the
composite from 46.4 to 59.9. The composite going up is not the verdict; the
irrelevance column beside it is the part that got worse, and why this is the row
to read.
Slot filling finally moved, and what moved it was data. search_filter goes
from calling nothing to picking the right tool 16 times out of 17 — and from 0 to
3 of 17 on the arguments. Three is not a good number. It is the first one above
zero, and it arrived when the training half went from 13 argument-extraction
rows to 58 — the task benchmarks grew from 36 cases to 131, and the router's
learned gate started carrying the teacher to those tools on 102 of 105 requests
instead of 47. Neither change alone would have done it.
The composed set has stopped being worth much, and that is the finding. On
the older, starved set, capping the answer turns and weighting the abstentions
was worth 7.5 points of composite (54.9 to 62.4). Here it is worth 0.1 — 59.9
against 60.0 — buying one tool selection and two slot answers for a loss in
correct calls. The weighting was compensating for a set with a hole in it. Fill
the hole and the compensation is noise, which is a good reason to prefer fixing
data over tuning knobs.
message_intent is where the wall still is: 1 and 2 of 17 tool selections,
0 of 10 on the arguments, from a base of 0. Classifying a quoted message and
pulling a date out of it is a harder job than filling three closed fields, and
58 slot rows spread across both tools is not enough for the second one.
The recipe, the one dependency trap, and the constraint that teacher and student
must share a chat template are in training/.
Down to 92 KiB
A choice[...] field is not a generation problem. search_filter's audience,
price and when are five values each and message_intent's intent is one of
four — an argmax over a handful of classes, where the guarantee the automaton
buys on a GPU is free because there is nothing to emit from.
So the same job, as hashed character n-grams into one int8 weight per class.
Trained on generated examples, scored on the 131 human-written cases in
benchmarks/tasks/ — 95 of them written after the model was trained. Against the
distilled 135M, on the held-out cases it was scored on:
The classifier column is the 1,894-row training set, measured 5 Sep 2026; the
generator has since grown the other tools' work as negatives, which costs
accuracy on these cases and buys a false-positive rate the cases cannot see. Both
columns, and the reason, are in esp32/README.md.
| SmolLM2-135M | classifier |
|---|
| size | 528 MiB resident | 92 KiB |
| work per message | ~200 tokens generated | 4,266 integer ops |
search_filter tool | 4/5 | 5/5 |
search_filter slots | 1/5 | 15/15 |
message_intent intent | 0/4 | 3/4 |
Which is what makes an ESP32-S3 a real target rather than a slide. A decode
step reads every weight once, so tokens/s <= bandwidth / size, and that board's
PSRAM sustains ~40 MB/s: a 135M model at Q4 is 68 MB and cannot beat 0.59 tok/s
even if it fitted, which it does not. At 92 KiB the weights are 18% of the
internal SRAM and the bandwidth wall never applies.
And the cycle assumption has now been paid for on real silicon — on the wrong
side of it. budget.py divided 4,266 operations by a documented clock and
bracketed cycles-per-operation at 1.0, 2.5 and 5.0 because nobody had run it.
The same two loops on a NodeMCU (ESP8266EX, Xtensa LX106, 80 MHz, -O2),
6 Sep 2026:
| value |
|---|
| cycles per operation | 44.50 |
| median cycles per message | 188,576 |
| per message | 2,357 µs · 424 messages/s |
agreement with slots.c | 141 of 141 messages, 0 disagreements |
Reproduced identically on three consecutive runs. The correctness column is what
licenses the timing column: all 23 accumulators are compared against slots.c
on every message, because a board that classifies differently is not measuring
this model.
44.50 does not refute the 1.0–5.0 bracket; it prices the assumption the
bracket rests on. That part has 48,952 bytes of free heap, so the weights live
in flash and every one of the ~3,300 weight reads per message goes through
pgm_read_byte and a 32 KiB cache that 92 KiB of randomly-indexed buckets
thrash. SRAM residency was load-bearing in the sentence above and had never been
costed: on a part where the weights do not fit, the identical code is an
order of magnitude slower, paid entirely for where the bytes sit. What is still
unmeasured is the other half — the same loops with the weights in DRAM, which is
the experiment esp32/device/ is built to run.
And 48 KiB of it now ships inside the router. The trigger list reaches these
two tools on 87 of the 105 requests that expect one; the tool head, added as a
signal that only ever raises a score and never overrules one, takes that to
102 — it catches 15 of the 18 requests no substring can reach, including the
ones that name no place at all. Measured against eval --routing, which is the
guard for exactly this: 166/166 reach and 166/166 in the top three, unchanged, at
pressure 0 and 20 (177/177 and 177/177 since the eleven agent-session cases
joined the suite, re-derived 10 Oct 2026 on macOS). An earlier version of the same head, trained without the other
tools' work as negatives, called 38% of the other suites' messages an extraction
request and cost fourteen of those top-three positions — which is why it is in
the repository with its false-positive rate measured rather than asserted.
What it cannot do is the honest half. city, promised_date and amount
are open text — span copying, not classification — and stay with the host. Nine
cases is a small denominator. And the ESP32-S3 row of the device table is still
arithmetic: what has been run on a board is an ESP8266 with the weights in
flash, which is the regime that row explicitly is not.
The two implementations must compute identical features or the model is being
fed n-grams it was not fitted to, so the trainer and the C are compared
accumulator by accumulator. Comparing their answers was not enough — breaking
the letter folding changed every Turkish message's features and flipped no
prediction at all. The tighter check found two real bugs in a minute: Python
lowercases İ into two codepoints, and reading every non-ASCII character as two
bytes mistakes an em dash for a Turkish letter. Details in esp32/.
Seven languages
The same Qwen3-4B against the natively-authored cores in benchmarks/core/
(~50 cases each, --skip-missing for the two web cases the rented box could not
serve):
| en | ru | es | fr | zh | de | tr |
|---|
| score | 91.1 | 94.4 | 94.1 | 91.8 | 91.0 | 88.5 | 84.5 |
| tool | 136/160 | 31/34 | 28/30 | 29/33 | 28/34 | 26/32 | 23/33 |
| irrelevance | 24/24 | 13/13 | 13/13 | 13/13 | 13/13 | 13/13 | 13/13 |
The irrelevance gate holds in all seven — 78 of 78 across the six cores,
plus 24 of 24 in the en column, 102 in total. It said "91 of 91 across the six
cores", which was never derivable from anything: the six cores hold thirteen
irrelevance cases each.
And the en column is not a core. There is no core-en.json. Those figures
are the 184-case English+Turkish baseline (crates/tacet-eval/baselines/qwen3-4b-both.json)
— a different instrument, four times larger, 66 of whose cases are Turkish. It
sits in this table because it is the only English number there is, not because it
is the seventh member of the set. Read the six cores across the row; read en
against the tables higher up the page. It is also the one column measured on
Metal rather than on the rented card. Its tool figure said 133/160 — the model
table's 3090 row — under a citation of a file that says 136/160; the column
now quotes the file it names (re-derived 10 Oct 2026, and pinned by
the_readme_quotes_the_baselines). The score is 91.1 either way.
What moves is tool selection, and Turkish is now the weakest rather than the
strongest, which is the reverse of what the English/Turkish suite shows. The two
are not the same cases: these were written natively per language, and the
comparison to make is across this row, not against the older suite.
Where this is not novel
This section exists because the alternative is worse. A reader who knows this
field and finds the standard description of constrained decoding presented as the
headline discovers the omission themselves, and then reasonably discounts every
paragraph after it. Naming the prior art costs one section and buys the rest of
the page.
The masking claim is category-standard. Compile a schema to an automaton,
mask the logits every step, make the invalid unrepresentable rather than
validated afterwards — that is the shared description of:
- llama.cpp GBNF (2023), the first version most people met;
- Outlines — Willard & Louf, Efficient Guided Generation for Large Language
Models, arXiv:2307.09702, which reframed
it as an indexed FSM walk;
- XGrammar — arXiv:2411.15100,
integrated into vLLM (Dec 2024), TensorRT-LLM, Modular MAX and OpenVINO GenAI;
- llguidance, which shipped inside OpenAI's JSON Schema
mode in May 2025 and inside
Chromium;
- guidance and lm-format-enforcer, and
- Apple's Foundation Models guided generation. Tacet's grammar exists
because moving off Apple's
DynamicGenerationSchema lost that forcing —
crates/tacet-grammar/src/lib.rs has said so since it was written.
The router-plus-distillation shape has direct prior art.
TinyAgent (arXiv:2409.00608, EMNLP 2024)
is a 1.1B/7B running on an M3 MacBook with a fine-tuned DeBERTa retriever cutting
the tool list before inference, lifted from 12.71% to 78.89% by distilling 80K
GPT-4-Turbo examples with correctness filtering. Octopus v2
(arXiv:2404.01744) and Hammer
(arXiv:2410.04587, Robust Function-Calling
for On-Device Language Models via Function Masking) occupy adjacent ground. The
router here is a hand-written trigger table plus a 48 KiB int8 classifier rather
than a fine-tuned encoder, which is a different point on the same curve — smaller
and auditable, not new.
The "irrelevance gate" is BFCL's relevance/irrelevance detection under another
name. The Berkeley Function Calling
Leaderboard
has irrelevance, live_irrelevance and live_relevance categories measuring
exactly this. Until 6 Sep 2026 no number on this page was measured against any external
benchmark — every table was this project grading itself on cases it wrote.
Measured against someone else's benchmark
is the first step out of that, and it is honestly three categories deep — BFCL's
relevance and irrelevance sets, not the leaderboard.
Measured against someone else's benchmark
Three of BFCL v4's relevance categories, run through this engine, this prompt,
this router and this grammar, with BFCL's own function definitions rather
than this project's catalog. qwen3-4B-Instruct-2507 Q4_K_M on Metal, 6 Sep 2026.
The decode is greedy, so these are stable rather than merely repeated — the first
category was run twice, case for case.
| category | n | expected | result | wall |
|---|
irrelevance | 237 | call nothing | 200 · 84.4% | 8m 42s |
live_irrelevance | 871 | call nothing | 678 · 77.8% | 58m 04s |
live_relevance | 16 | call something | 13 · 81.2% | 1m 19s |
The live_ sets are user-contributed, which shows: the same stack is six points
worse on them than on the curated set. Reports are committed under
crates/tacet-eval/baselines/, each with every
case that went the wrong way and what the model said.
curl -sLO https://raw.githubusercontent.com/ShishirPatil/gorilla/main/\
berkeley-function-call-leaderboard/bfcl_eval/data/BFCL_v4_irrelevance.json
BFCL_JSON=out.json cargo run --release -p tacet-eval --features metal \
--example bfcl -- ~/models/qwen3-4b/model.gguf BFCL_v4_irrelevance.json
This is not a leaderboard submission and the numbers are not comparable to the
published board. BFCL scores through its own harness, its own prompt and its
own parser, and each of those is part of what is measured there. What this
answers is narrower: given the same questions and the same functions, how often
does this stack invent a call — or fail to make one.
What the translation cost, stated because it makes the sets easier and
therefore the numbers better. Three things stand between BFCL's format and
this one, and the harness counts all three:
- BFCL writes
dict and float where JSON Schema writes object and number.
- 666 function names had to be rewritten across the three sets, because a dot
cannot appear in a name this call format can express —
math.sum would arm the
automaton on math.
- 126 schemas could not be expressed at all, 121 of them in
live_irrelevance
alone, and where that left a case with no functions the case is not scored
(13 of 884 there, 3 of 240 in the curated set). That number is the honest price
of an ArgSchema small enough to compile into a grammar: BFCL's live set comes
from real APIs, and real APIs use JSON Schema this project deliberately does not
implement.
It found a defect on its eighth case, which is the argument for running someone
else's cases at all. Asked to solve 3x^2 - 2x - 5, the model declined
correctly — and wrote the quadratic formula on the way. 2(3) is the denominator
2a; it is also name(args), with 3 as perfectly good JSON. Three of the
turn's four passes went to a tool named 2. Nothing ran, because gate 1 rejects
an unknown name, so it was never a safety problem — it was the turn budget, and
in any measurement it counts as a call on a case whose whole point is that there
must not be one. A name may contain digits; it may not start with one. Fixed,
with the fixture.
And the three live_relevance failures are the same known gap, not a new
one. Two of them wrote the call as search_web language="fr", query=… — the
right tool and the right arguments in a shape nobody taught it. That is the
limitation stated at the top of this page: the grammar arms after name(, so it
says nothing about a call that never starts that way.
What is actually distinctive
- Always-terminating, as a claim separate from unrepresentable-invalid.
Bounded consecutive whitespace, bounded open-schema strings, numbers and arrays, a 2048-token cap
measured against a largest-observed legitimate call of 1523. Unbounded
whitespace wander is a known live failure in this category and nobody
advertises a termination bound. The measurement that produced it — a complete,
correct
calendar(…) call followed by twelve minutes of spaces — is above.
- The coupling claim. The constraint contract is three signatures over
&mut [f32] and u32 living in tacet-kernel with no inference dependency,
guarded by a test that stops compiling if a signature starts needing an
inference type, and demonstrated by cargo run -p tacet-grammar --example no_engine — verified from outside the repository against crates.io, with
cargo tree | grep -c tacet-engine at zero. Most implementations in the list
above are coupled to a serving stack.
- Privacy that fails the build. The two-manifest network monopoly as a test
rather than a promise.
- An eval that refuses a verdict below its own resolution, and a page that
retracts its own overclaims with the measurement attached. Three of those
retractions are on this page.
- A 92 KiB int8 classifier beating a distilled 135M on closed-vocabulary
slots — 15/15 against 1/5, at 92 KiB against 528 MiB resident.
One counter-datum, scoped
The literature says grammar constraint distorts the distribution and costs task
accuracy: Grammar-Aligned Decoding
(arXiv:2405.21047) makes the distributional
argument, and the format-tax papers report accuracy costs, one of them
constraint-induced suppression of tool calling.
bench gap here runs the other way on tool selection: started 76.9% → 97.4%,
tool-name-correct 66.7% → 97.4%, with the mask overriding the model's own argmax
only 19 times in 16,402 tokens and an identical-argmax control.
The caveats belong in the same breath: n=39, one file, one card, one date; two of
three models show nothing measurable; and "correct" is call.name == want
(bench_cmd.rs), a tool-selection metric, not an answer-quality one. Framed as a
scoped counter-datum on tool selection it is publishable. Framed as a refutation
of distributional distortion it is not — the +20.5 points of started calls are
explained by the mask moving the model off its own distribution.
Platform support — honestly
| Platform | State |
|---|
| macOS (arm64) | Verified. Full suite runs here: build, clippy -D warnings, tests, eval. |
| Linux (arm64) | Published, not yet run on a board. Release binaries are built natively on an ARM runner and tacet update has always known this triple — the release simply did not carry it until v0.1.27, so a Raspberry Pi could download nothing and was told the platform was unpublished. A guard now fails the build if the updater asks for a target the release workflow does not build. What is still unmeasured is everything after the download: nobody has run the shell, the sandbox or a model on ARM64 Linux. This is the platform the work in esp32/ is heading towards, so it will be measured rather than assumed. |
| Linux | The sandbox is measured, and on a default install it is OFF. Measured on a stock Ubuntu 24.04 (4 Sep 2026), not only in CI: as an ordinary user bwrap --unshare-net fails with loopback: Failed RTM_NEWADDR: Operation not permitted, because the distro ships kernel.apparmor_restrict_unprivileged_userns=1. So run_code and write_code are absent from the catalog — the honest-refusal path, working as designed, but it is the DEFAULT on the most common Linux rather than an edge case. The shell says so on startup — the line that reports run_code as off now names the sysctl and the one-line remedy, rather than leaving "no sandbox" as the whole explanation. As root on the same machine the shield verifies and both tools appear, which is why testing with sudo hides the problem. What the shield itself does is measured: with the restriction lifted, the_network_is_really_cut, the_sandbox_cannot_be_escaped and the timeout kill all pass against a real bwrap 0.9.0 — 1069 tests, 0 failures (33 binaries, 4 Sep 2026) — the record of a sandbox measurement on that machine on that date, not the current size of the suite, which is 1993 tests in 104 test binaries (doc-tests included; 10 more ignored) on macOS as of 11 Oct 2026, re-counted from cargo test --workspace on an Apple M4 Pro (it was 1241 in 64 on 7 Sep 2026). Run under TACET_SANDBOX_MUST_RUN=1, the flag that makes the 26 tests that used to skip execute or fail the build (22 at that measurement; nothing_survives_the_call, a_loose_socket_in_tmp_is_shadowed_under_bwrap, and have joined them since, and the count is asserted against this page by ). Still unmeasured: nobody has held a conversation with the interactive shell on Linux. |
Where a guarantee holds on one platform and not another, the code says so at the point where it matters. 0600 permission stamping, for example, is deliberately not applied on Windows — there set_permissions only flips a read-only flag, which would produce the appearance of protection without the substance.
Development
cargo build --workspace
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
cargo run -p tacet-cli -- eval
The eval suite scores tool usage, not answer correctness — a case can pass with wrong arithmetic if the model called the right tool with the right arguments. That is intentional (the arithmetic is the tool's job, and the tool has its own tests), but it means eval is not a substitute for reading the output.
Contributing
Small changes welcome; the shape of the project is written down in
CONTRIBUTING.md and takes five minutes to read.
The most useful thing anyone can do right now: run the interactive shell on
Linux or Windows and say what broke. CI now exercises the Linux sandbox
against a real bwrap and it passes, but nobody has held a conversation with
tacet on either platform, and Windows has no runtime measurement at all — see
the platform table above.
Star History
License
MIT. See LICENSE.
