Nebraska State Contracts

Live: https://diepjustin.github.io/ne-contracts/

A journalism and accountability tool for scraping and publishing Nebraska state spending records from the Nebraska State Contracts Database, the public database mandated by Neb. Rev. Stat. § 84-602.04.

Two things live here: a scraper, and a static searchable site built from what it collects. index.html sits at the folder root because that root is the published URL. There is no backend and no framework.

739,605 records · 540,074 descriptions · 92 entities · one 6.85 MB page.

What's in the data	coverage, caveats, descriptions
Running it	scrape, build, preview, restore
How it works	scraper, payload, publishing
Guard rails	read before changing anything that writes
Things that bit us	expensive to rediscover
Open work	what is unfinished, and what is not worth doing

---

What's in the data

739,605 records across 92 entities — every state agency, board and commission the database lists (83), plus all nine University of Nebraska and Nebraska State College campuses. Both Active and Expired, contracts and purchase orders. The initial collection finished on 9–11 Aug 2026; since then only Active records are re-read each night, so a record the state publishes already ended can be missing (see the caveats below).

	Records
University of Nebraska Medical Center	238,763
70 state agencies with records (13 have none)	235,169
University of Nebraska Lincoln	209,526
University of Nebraska Omaha	28,167
University of Nebraska Kearney	18,696
Wayne State College	5,562
Chadron State College	871
University of Nebraska Central Administration	821
Peru State College	489
Nebraska State College System	131

Two campuses hold 61% of everything, almost all purchase orders. On the agency side three dominate: Correctional Services (61,783), Health & Human Services (60,682) and Roads (39,253) are about 70% of state agency records.

The page names any outstanding entity itself, reading the scrapers' checkpoints, so a partly-collected entity is never silently indistinguishable from an empty one.

Caveats worth knowing before you quote it

The scraper reproduces the state's records faithfully, including their errors.

• This is not a complete record of state spending, and the gaps are not random. The state's own FAQ says the database excludes contracts from Health & Human Services, the University, the State Colleges, Veterans' Affairs, Education, the Commission for the Blind and Visually Impaired, and the Nebraska Investment Finance Authority "that provide specific aid, assistance, or services to a specific individual." Those are among the largest agencies here, so their totals understate what they actually spend. Do not read an agency's total as its budget.

• A handful of records carry billion-dollar values and will dominate any total. The largest is 95601, HHS / "CREIGHTON UNIVERSITY - ALL PAYMENTS", at $38,025,000,000, then three Medicaid managed-care contracts at $6,650,000,000 each and 58-1-1451 (EBSCO, UNL) at $4,000,000,000. Some are plausibly real not-to-exceed ceilings on multi-year statewide programs rather than errors — the state's FAQ says service contracts are valued at the estimated cost of the whole contract including renewals. Treat the top of the amount column as ceilings, not money spent. The status bar's total is the sum of the recorded amounts for the rows shown, and inherits every caveat in this list.

• Vendor names are fragmented, and the site only partly repairs it. The state records one firm many ways: 26 spellings of Hausmann Construction total $1.40 B where the largest single spelling shows $903 M. About 9,000 vendor strings sit in a cluster with at least one other, holding $11.02 B between them. Four companies have been reviewed by hand and grouped; the rest have not. See Open work.

• Purchase orders are not a Higher Education thing, even though the site makes them look like one. The search form only offers a Contract/Purchase Order filter for Higher Education, so on the agency side the distinction is invisible — but it is in the records. Ten of the 32 agency document-type codes (O9, OM, OP, X7, Y6, Y7, Z8, Z9, ZO, ZP) are filed under "Purchase Orders", covering 152,210 records, about 65% of agency data. Overall the dataset is 619,601 purchase orders to 118,594 contracts — the reverse of what the interface implies.

• Document types are the source system's internal codes, not labels. The state publishes no key. maps each to the category its detail page files it under, which is what the page filters on; the raw code stays in the table (hover the Type cell) and in the CSV export. That mapping is a sampled observation, and one code contradicts it: , which the site's own Purchase Order search returns, has a detail page headed "Contracts". The map overrides it, since the search filter is the better authority.

Descriptions

The state publishes no "scope of work" field, so there is nothing to scrape for it. But most documents contain one, written by whoever filed them, and scripts/extract_scope.py lifts it out of the PDF verbatim. Nothing in a description is generated, summarised or rewritten — every description on the site is the state's own words, which is what makes it quotable. Typos, inconsistent capitalisation and abbreviations are theirs and are kept. (Some scanned documents also carry a separate, labelled, unverified AI summary beside the description; it is never mixed into one. See Generated and OCR'd content.)

540,074 of 739,605 rows have one, 94.8% of every document that could be read at all. They come from three places, in this order of preference:

source	rows	what it is
line items	533,497	what was bought, off the purchase-order table
cover sheet	3,836	a summary someone wrote by hand on a University contract
SERVICES clause	2,737	the contract stating its own scope, where there is no cover sheet

Three more are in extract_scope.py and none of them is in the published data yet — all three land at the next extraction run, and the two written on 22 Aug 2026 are still short of the check this file demands of them (see below).

source	rows	what it is
direct_purchase	7,577	Roads' one-sentence notice that a purchase produced no contract
purchasing_bureau	879	the State Purchasing Bureau's "Contract to supply and deliver…"
cover_sheet_form	548	the University's flattened DocuSign cover sheet

purchasing_bureau is ranked below line items, not above. It was tried above them, on the strength of a purchase order that reads "GSA MODEL; FRIEGHT; HULL QUOTE ITEM F" off its table and "One Time Purchase to supply and deliver AirBoats to the State of Nebraska" here. Across the corpus that ordering rewrote 671 existing descriptions and the trade went both ways: a boat whose line items gave its length, beam and transom became "to supply and deliver boat", and "SNOW GROOMER FOUR CYLINDER" became "SNOW GROOMER". The state writing a headline is not the state writing a better description, and choosing between two of its own phrasings is not this file's call. So it fills blanks and touches nothing already described.

Both parsers are unverified against source documents. All 8,456 extractions are ordered subsequences of their own document's text, and the tests pass, and that is exactly the evidence that was sufficient for the truncation bug too. The state's document service has been down since 22 Aug 2026, so the ten-real-documents check under "Guard rails" has not been run. Do not publish an extraction release built from these until it has.

cover_sheet_form is the oldest of the three. The University has a second cover sheet that is a filled PDF form rather than prose, and DocuSign flattens it on export: every field label is written out in one run, then every filled value in another, so "DESCRIPTION OF PURCHASE" ends up dozens of lines from its own answer. It recovers 548 descriptions out of the 22,086 readable documents nothing else describes — 2.5%, which is worth having and is nowhere near a solution.

How a positional parser was checked. Nothing in that form identifies the description except its position in the value run, which is how one contract's words get attributed to another. scripts/verify_form_geometry.py answers an independent question: it downloads the real PDF and reads where the ink is, taking whatever sits level with and to the right of the "DESCRIPTION OF PURCHASE" label, then compares that against what the parser claimed from flat text. The first sample of 40 agreed 37 times; the three failures were real and none of them would have been caught by a test:

• two documents whose description field was simply blank, so the term dates slid into its place and would have been published as the contract's scope
• one that extracted as /\x04EZ\x03\x11h^ where the page plainly reads "travel" — a font encoding pypdf cannot map

With guards for both, a fresh sample of 60 agreed 60 times. This is the same lesson as the truncation bug, in a new form: the tests all passed both times, and only the source document settled it. Geometry cannot be used in the parser itself — doc_text.jsonl holds no coordinates and re-reading 739,605 PDFs is a 20+ hour download — so it stays a sampling check.

Things worth knowing before quoting one:

• A description is one document's text, not the contract's official scope. It is whatever the filer typed. Read the source document before quoting; every row links to it.

• Descriptions carry administrative text alongside the substance — invoicing instructions, project numbers, contact names, change-order logs. These are kept deliberately rather than filtered, because on some documents the actual scope appears after the boilerplate, so a filter aimed at the noise removes the substance.

• A blank is not a statement about the contract. It means we could not read a description out of the document, not that the contract has no scope. The page says so where it happens rather than leaving an empty cell to be misread. It deliberately does not say why, because the reason varies and we were getting it wrong. Measured over the 118,843 blanks where the state does publish a document:

  why it is blank	documents	share
  the PDF is a scan with no text layer	89,180	75.0%
  readable text, but our parsers found no description in it	22,086	18.6%
  the state says the item was a direct purchase with no contract	7,577	6.4%

  That last group is not a failure of any kind: those documents consist of a single sentence from the state — "This item involved a direct purchase which did not result in a contract. Therefore, there is no contract available for this item." — and are every one of them Department of Roads spending. From the next extraction run they stop being blank: the sentence is the state's own answer, published verbatim as the direct_purchase source, and it beats our "No description could be read from this document" because it is true and ours is merely not false.

• Searching descriptions is opt-in. The index is a separate 9.92 MB download, fetched only when you tick the box, so it costs nothing for readers who do not use it.

• Filtering by whether a row has one is not. The page's Description control — all records / has one / has none — reads descsrc.bin, which is resident, and each description is captioned with where it came from. Both exist because the same text reads completely differently depending on which parser produced it, and because a reader clicking University contracts hits a blank five times in six and reasonably concludes the database has no descriptions at all.

94.8% is a coverage figure for readable documents, and it hides where the gap is. Purchase orders carry the vast majority of the descriptions, and a purchase order's description is a list of what was bought. Contracts — the records worth reading — are where there is almost nothing:

	rows	has a description	blank, document exists	no document
University contracts	35,681	12.8%	82.9%	4.3%
State agency records	235,410	42.0%	40.3%	17.7%
Purchase orders	468,642	93.2%	5.9%	0.8%

On a large share of University contracts the scope is not in the document at all. 1,161 of the blank ones carry the identical sentence "...to render the services and provide the deliverables identified in Section 1 of Exhibit A", and the state publishes the agreement without the exhibit. That is a limit of what is disclosed, not of the parsers: no pattern and no OCR run reaches a scope that was never filed. A heading-based parser was measured against those 7,165 documents and rejected — 56% of what it returned was template boilerplate repeated across hundreds of contracts and 32% began mid-sentence, which would have published a tax clause as a contract's scope. Guarded hard enough to be safe it reached 154 documents, 2.1%, and still got Oracle's warranty clause. This agrees with the 5.4% ceiling under "Open work"; do not spend a week here either.

Generated and OCR'd content

Everything above is the state's own text. Two things on the site are not, and both are kept apart from it.

OCR. Scanned documents have no text layer, which no parser can read (see "Open work"). scripts/ocr_pilot.py --engines vision runs them through Apple's Vision framework (via ocrmac), so it needs macOS and never runs in CI. It renders up to 200 pages and keeps up to 100,000 characters per document, and appends one record per document to data/ocr_pilot.jsonl (about 1.2 GB; the last record for a token wins). In mid-Oct 2026 it held 109,173 documents, 105,460 of them with text. OCR text never enters scope.jsonl or the descriptions: it would sit beside the state's own words, which this page promises are verbatim, so it stays local until someone decides otherwise.

AI summaries. scripts/generate_ai_summaries.py asks a local model (Ollama, llama3.1:8b-instruct-q4_K_M, temperature 0.2) for one sentence of at most 40 words about each OCR'd document, or the literal UNCLEAR when the text is too garbled. Only the first 4,000 characters of the OCR text are sent. Records append to data/ai_summaries.jsonl; --shard I/N splits the backlog across machines. Nothing checks a sentence afterwards. Neither job's dependencies are in requirements.txt: OCR's are pinned in requirements-ocr.txt (ocrmac, PyMuPDF, Pillow and PyObjC, at the versions the Mac runs), and the summaries need Ollama with ollama pull llama3.1:8b-instruct-q4_K_M. Summary records written since 23 Oct 2026 carry generated_at; older ones have no date.

Where a summary goes:

• pages.yml downloads ai_summaries.jsonl.gz from the ai-summaries-data release whenever it rebuilds (falling back to the newest extraction-data-* release while that one is missing), and warns when the asset is more than 14 days old; with neither, it builds without summaries.
• scripts/publish_ai_summaries.sh, run by hand on the Mac, publishes the file to one stable release, ai-summaries-data (asset ai_summaries.jsonl.gz), and scripts/publish_scope.sh does the same for scope.jsonl (scope-data, scope.jsonl.gz). Each refuses a file that does not parse, names a model build_site.AI_MODELS does not list, or holds less than the copy already published, and records the count and sha256 in the release notes. pages.yml reads those two releases first.
• build_site.py joins a summary to a row by the row's own view token first. Where the row's own document has no summary (or only UNCLEAR), the first summarized of the record's other documents, in the state's order, stands in — the same rule descriptions follow — and a summary keyed on a view token several rows share reaches all of them. Until Oct 2026 only the row's own document counted, and every summary of an amendment or attachment (19,639 on 22 Oct) was dropped as "unmatched". On the 22 Oct data that rule change took the rows showing a summary from 84,721 to 87,245, 2,504 of them from another document.
• The payload carries them as the ai/ blocks, aisrc.bin (which model wrote each row's summary), aidoc.bin (which of the record's documents was summarized, as a position in the state's list; 255 = not known) and meta.aiModels / meta.aiCount / meta.aiFormatVersion (2 since aidoc.bin; versioned apart from formatVersion because --ai-summaries-only never touches a resident file). The panel says "Summary of (document k of n)" above every summary and marks that document in the list; a page meeting a newer aiFormatVersion than it knows shows no summaries. A summary is not in the search index and not in the CSV export.
• The row panel shows it in its own box, labelled "AI-GENERATED SUMMARY", with the note Generated from OCR text and not verified. Meant to help you decide whether to open the source document, not to be relied on as fact — read the original before citing anything here.

What to know before trusting one (measured 13 Oct 2026):

• About one in five summarises a one-line state placeholder page — "Document Cancelled" or "PO has been deleted." — so it says nothing about the contract, and some call the document blank or incomplete: 19.1% of the generated summaries that have text (19,878 of 104,174), and about 22% of the boxes the site shows (18,933 of 84,527 match a placeholder summary exactly). Nothing filters them out.
• About 0.7% of the other summaries state a dollar amount, date or number that is not in the OCR text. Of 26 such cases checked by hand, 15 were wrong or unsupported — some amounts off by 10× to 1,000×, some years invented — 6 were true but not literal, and 5 were false alarms.
• About three quarters of all OCR'd characters (74.6%) are never sent: the model sees the first 4,000 characters of each document, and 48.7% of documents are longer.

How it runs. Both jobs run on one Mac under launchd, not in CI; their LaunchAgent plists (com.justindiep.ne-contracts.ocr-catchup and …ocr-recapture) are not in the repo. scripts/run_ocr_catchup.sh runs nightly at 04:30, from ~/Developer/ne-contracts rather than the iCloud copy, under caffeinate. launchd does not wake a sleeping Mac: a run missed while it slept fires at the next wake, which can be mid-morning. The job does more than OCR. It first downloads that day's ne-contracts-hub-data-<date> release (the CSVs and scrape_meta.json the GitHub nightly published at about 03:17) instead of scraping its own copy, which had drifted from what the site shows; --local-scrape still scrapes instead. That release carries no documents.jsonl, so the Mac's own build calls the document count of newer records unknown, which nothing it feeds depends on. Then come build_site.py, a prune of every d/ build but the current one, extract_text.py, extract_scope.py, ocr_pilot.py and generate_ai_summaries.py.

A step that fails does not stop the ones after it, but the script now exits with the first non-zero step code: 3 when the day's release is not out yet, 4 when it is unusable (nothing in data/ is replaced), 2 when OCR or summaries ran but more than half of the night's items failed, 75 when another run holds data/.ocr.lock. Each step's exit code and duration go to logs/ocr_catchup.log and to logs/ocr_status.json, which also records the start and finish times, the commit and how many OCR and summary records the night added. Nothing outside the Mac reads it yet. scripts/update_live_checkout.sh brings that checkout up to origin/main: fast-forward only, requirements reinstalled when they change, tests run, and a rollback if they fail. scripts/run_ocr_recapture.sh was a one-time backlog job. The first OCR pass kept 6 pages and 4,000 characters per document; on 17 Sep 2026 the caps went to 200 and 100,000 after a corpus-wide check found 24,344 documents (22.4%) had run past the page cap, and a recapture run redid 23,504 documents (the file as it stood before is data/ocr_pilot.jsonl.bak-before-full-page-recapture). Which documents to redo was chosen by hand and is not scripted, so running run_ocr_recapture.sh on a fresh data/ would OCR everything.

---

Running it

The workflows and the Mac's venv run Python 3.12. The venv was 3.9.6 until 26 Oct 2026, when Pillow 13 (past 58 security alerts) needed 3.10 or later; every package in requirements.txt is pinned, so all three install the same versions.

On a clean clone, restore data/ before scraping (see "Getting the data back"). data/*_scrape_progress.json is tracked in git but data/*.csv is gitignored, so a fresh clone says every combo is already complete, and scrape.py refuses to run in that state rather than write a header-only CSV and stamp it complete. To collect everything from scratch instead, pass --fresh — 20+ hours.

Once data/ is restored, bring it up to date with --daily; a full scrape of a dataset whose combos are all done has nothing to fetch and leaves its recorded time alone (and --entity X finishing X, with or without --daily, does not stamp the whole dataset either). Restore scrape_meta.json with the CSVs so that time matches the data.

cd ne-contracts
python3 -m venv venv && source venv/bin/activate && pip install -r requirements.txt

python3 scripts/scrape.py contract          # -> data/nu_contracts.csv
python3 scripts/scrape.py purchase-order    # -> data/nu_purchase_orders.csv
python3 scripts/scrape.py state             # -> data/state_agencies.csv
python3 scripts/scrape.py state --status    # progress, no network
python3 scripts/scrape.py state --hours 3   # stop cleanly after 3h
python3 scripts/scrape.py state --entity "Roads, Department of"
python3 scripts/check_entity_drift.py --strict   # has the state's entity list changed? (exit 1 if so)

python3 scripts/backfill_documents.py       # -> data/documents.jsonl (the 33h one)
python3 scripts/backfill_documents.py --dataset contract   # one dataset at a time
python3 scripts/backfill_documents.py --hours 3            # stop cleanly after 3h
python3 scripts/backfill_documents.py --status             # coverage + is a run going, no network

python3 scripts/extract_text.py             # -> data/doc_text.jsonl (the 36h one)
python3 scripts/extract_text.py --status    # progress, no network
python3 scripts/extract_text.py --retry-errors    # re-ask the 14,586 that failed
python3 scripts/extract_text.py --retry-non-pdf   # re-ask the 32,227 non-PDF answers
python3 scripts/extract_scope.py            # doc_text.jsonl -> data/scope.jsonl (~1 min)

python3 scripts/build_site.py               # data/*.csv -> d/<buildId>/  (~3 min, ~1.7 GB peak)
python3 -m http.server 8098                 # preview at http://127.0.0.1:8098/ (index.html, manifest.json and d/ sit at the repo root)
python3 -m pytest tests/ -q
python3 scripts/lint_page.py                # eslint over the script in index.html

extract_text.py refuses to start while the state is not serving documents, for the reason under "Things that bit us". --status still answers, because it makes no network calls.

Linting the page

index.html is one self-contained file with the script inline, so there is no .js for a linter to open. scripts/lint_page.py cuts the <script> body out into .lint/, blank-padded so line n of the extract is line n of index.html, and runs a pinned eslint over it through npx — no package.json, no lockfile, and nothing a reader ever downloads. eslint.config.mjs holds the rules. It runs in pages.yml beside pytest and needs Node; without Node it says so and exits 2 rather than passing quietly.

no-shadow is the rule this exists for. A local var end inside render() shadowed the module-level end column and blanked every End cell on the live site for months, while every test and the crc32 selftest passed — the payload was correct and the page was wrong (see "Things that bit us"). Seven problems were live when it was first switched on, all of them that shape or next to it:

unpackDocuments(u8, at, end)	the parameter shadowed the same end column the original bug did. Renamed until.
var start in render()	shadowed the module-level start(). Renamed startRow.
var words in showDetail()	shadowed the description search index. Renamed panel.
term in firstAtOrAfter / rowsMatching	shadowed the module-level query string. Renamed prefix.
var was twice in closeDetail()	a real double declaration. The second query is the working one — render() has replaced the row element by then, so clearing the class on the first, now detached, node does nothing. Renamed live.
th in the sort-header loop	shadowed the th holding the default-sort column. Renamed sorted.

no-undef needs the page's browser globals declared, and they are written out in the config rather than pulled from the globals package: one dependency fewer, and the list is then an explicit statement of what the page assumes a browser gives it.

Checking the page in a browser

The lint reads the code; scripts/check_page.cjs runs it. It serves a built site on 127.0.0.1, opens it in Chromium, and fails if ?selftest=1 does not end "all checks passed", if the page throws or asks for anything off that server, if a search does not narrow the count, if Tab through the rows or Enter on one drops focus on the body, or if axe-core finds a WCAG 2.1 A or AA violation with a row open — on a 1366×900 desktop and a 390×844 phone. Until Oct 2026 only a person ran the selftest, and the keyboard and screen-reader paths had never been checked at all: every scroll re-rendered the focused row away, and role=button on each row hid its document link from screen readers.

pages.yml runs it on the staged site before the upload, so a page that fails it does not deploy. ci.yml runs it on a pull request's index.html against the payload pages.yml last built (restored from the cache, never saved), and skips it with a notice when the pull request changes build_site.py, since that payload is the old builder's. Playwright and axe-core are installed outside the checkout at pinned versions, like the lint's eslint. Locally, against a build from scripts/build_site.py:

npm install --no-save --prefix /tmp/page-check [email protected] [email protected]
/tmp/page-check/node_modules/.bin/playwright install chromium
NODE_PATH=/tmp/page-check/node_modules node scripts/check_page.cjs .

Every scrape checkpoints per page and resumes; interrupting is safe. A full run is 20+ hours against a government server — don't re-scrape casually.

Output columns: Document Number, Document Type, Entity Code, Entity Name, Vendor, Amount, Begin Date, End Date, Status, Detail URL, View URL. Entity Name holds the canonical dropdown name rather than the site's rendering of it (state result grids shout it: DRY BEAN COMMISSION), because both the resume checkpoint and build_site.py's entity guard match on that exact string.

Daily updates

The full scrape is a one-time job. Day to day the state mostly adds records under Status=Active, only ~7.3% of pages, so --daily re-scans just that status and diffs it against what the CSV already knows, matched on Detail URL. "Mostly" is the catch: a record the state publishes already ended is born Expired, never appears in that list and is never collected (see the caveats and "Open work"):

python3 scripts/scrape.py contract --daily
python3 scripts/scrape.py purchase-order --daily
python3 scripts/scrape.py state --daily

An unseen record gets a detail fetch and a new row. A known record is rewritten only if its amount, vendor or dates moved (see "Amendments" below). A previously-Active record that does not appear today has its Status flipped to Expired in place — unless the disappearance is too large to be real. An entity that returns no records at all while it had five or more active ones, or that loses more than half of 20 or more, is refused and re-checked on the next run. That is not hypothetical tidiness: without it, one outage expired all 44,063 active records in a single green run (see "Things that bit us"). Below five an empty answer is believed, because a small body's last contract ending is a real ending — refusing it held the state dataset open for four nights.

That floor had a price. On a night the state answered empty for everything, every state entity with one to four active records was flipped to Expired — 17 of them, 29 records, in the 28 Sep 2026 release — while the larger ones were refused. The night did not finish, so the stamp stayed put and the log named the refusals, but the guard rail below skips entities under five, so those small flips passed it, and the rows stayed Expired until the state listed them again.

So the scrape also judges the dataset as a whole. Across every entity it finished tonight, the refused ones included, if the CSV held 50 or more active records and the state no longer lists more than half of them, it flips nothing for any entity, small ones included, prints an ::error line, and leaves the night unfinished: the stamp stays put and the last step of daily.yml turns the run red (see "A night that did not finish"). Ordinary endings are nowhere near that line: in the local 17 Sep 2026 snapshot, apart from the 2099 open-ended placeholder date, no end date is shared by more than 3% of a dataset's active records. The line also holds back the other entities' endings for a night when one entity that holds most of a dataset is refused on its own — in that snapshot the University of Nebraska Lincoln holds 52% of the active contracts and 61% of the active purchase orders — and those are flipped the next night, once the state answers.

A record that comes back is flipped back, not appended. Every record --daily flips to Expired is noted in data/<dataset>_daily_flips.json, beside the CSV in the Actions cache: a hash of its Detail URL and the date flipped. An entry is dropped once its record is no longer Expired in the CSV, and after 365 days, since most flips are real endings that never come back and the file otherwise grew by about 80,000 entries a year.

When the Active search lists one again — after a night it was missed, or after an empty night like the one above — its row is flipped back to Active in the same rewrite that applies the night's endings, and nothing is appended. The row is rewritten the way a new record's is: vendor, amount and dates from tonight's listing, and a detail fetch for its View URL and its documents.jsonl entry, made once the night has been judged (below), so a leaking search costs no fetches. Until late Oct 2026 only the Status moved, so a contract extended while it was off the list came back Active with its old end date: 12 of the 13 records restored in the 20–22 Oct releases were Active with an end date already passed. A record whose detail page cannot be read stays Expired for the night and is restored the next. Whatever moved while it was off the list goes to data/changes.jsonl, dated that night. A record the CSV holds as Expired is never appended by --daily any more, which is what keeps every Detail URL to one row.

Until Oct 2026 such a record was unknown, fetched and appended as a new Active row beside the stale one, and the site listed it twice, once falsely Expired, adding its amount into every total twice. The 28 Sep 2026 release has 302 Detail URLs carrying both statuses. Those pairs are left as they are in the CSVs, which record what was scraped, and build_site.py drops the Expired row of each: any Expired row with the same Detail URL as a published Active row. Until 26 Oct 2026 it dropped only one identical to the Active row in every other column, View URL included, and kept a pair that differed anywhere as a possible amendment (45500 below is one). That kept 209 stale rows and $487,950,817 counted twice on the 26 Oct release — 88941 at $312,259,000 beside its own current $313,292,290 — so now the Active row stands alone, and what moved belongs in data/changes.jsonl. The build prints both counts: expired twins (identical, 209 rows and $26,559,997 that day) and stale twins (an older copy).

Flipping back trusts the same Active search whose silence made the flip, so it has to know when that search is wrong the other way: when its Status filter lets Expired records through. The test is how many listed records the CSV holds as Expired and --daily never flipped — the full scrape read those from the Expired search, and they outnumber the Active rows about sixteen to one, so a leaking filter cannot hide. Five or more such strays that are also more than 5% of what an entity listed, or of what the whole dataset listed, and nothing is flipped back that night, for that entity or for the dataset, with an ::warning line naming it. Nor is anything appended: a leaking filter lists Expired records the CSV has never seen as readily as ones it has, so each entity's new rows (and their documents.jsonl entries) are held until both verdicts are in, and a leaky night's are dropped, counted as held_back in the diff report instead of newly_active. They are fetched and judged again the next night, and appended then if the filter holds. A handful below that line is a contract genuinely re-opened, and is flipped back too. How many of --daily's own flips came back is never part of the test: after an outage spread over two nights all of them come back at once, and refusing that would turn the outage into a permanent freeze. A leaky night still finishes and stamps, because what it lists can only add to what was seen, so its endings are as sound as any night's.

Strays that never go away turn the run red. Not every stray is a broken filter. A record --daily flipped before the ledger existed, or one the ledger lost (a cache rebuilt from release CSVs, a ledger set aside as corrupt, an entry past 365 days), reads as a stray when it is listed again — and it is listed every night. Six such records at an agency of a hundred is a leak, so that agency's flip-backs and new records were held back night after night with a warning and a green run. Persistence cannot tell the two apart (a filter that stays broken lists the same strays every night too), so it is not taken as proof either way. Instead data/<dataset>_daily_leaks.json counts each entity's leak nights in a row, and the dataset's own: a clean night ends a streak, and a night that judged nothing (the entity refused or unfinished, the dataset-wide breaker tripped, a same-day re-run) leaves it alone. On the third, the run prints an ::error naming the entity and a few of its strays' document numbers, and the dataset is left unstamped, so the last step of daily.yml turns the run red. Nothing else is blocked, but the site's "last updated" date for that dataset stays where it was for as long as the run is red. Look the strays up on the state's site; if they really are Active again:

gh workflow run daily.yml -f accept_strays_dataset=state -f accept_strays_entity="Military Department"

The data lives in the nightly's cache, so this runs inside daily.yml: its "Accept strays" step runs scrape.py state --daily --entity "Military Department" --accept-strays before the normal steps. (The same command works by hand on a local copy of data/.) That re-reads the named entities, takes their strays as re-opened, restores them as above, and ends their streaks (and the dataset's). If the filter is genuinely leaking, leave it: the run stays red until it stops.

Amendments are recorded rather than discarded. Amount, vendor, begin and end date are compared against what the CSV already held, and any move is appended to data/changes.jsonl — one entry per field, carrying both values and the date observed:

{"key":"…|amount","doc":"45500","field":"amount",
 "from":"$2,558,983.00","to":"$21,204,743.00","seen":"2026-08-17"}

Before this the database was a photograph: 45500 at the Medical Center reads $2,558,983 expired and $21,204,743 active, the same contract amended, and nothing in the data could show it had moved. The CSV holds only the current value — the log is the history.

The amended row is rewritten too, the way a returning record's is: tonight's listing values and a fresh detail fetch for its View URL and documents.jsonl entry, once the entity has been judged (an entity refused or unfinished keeps its rows and is read again next run). Until late Oct 2026 the row was left alone, so the site kept the amount of the last full scrape and the same move was logged again every night against a baseline that never changed. The entry is logged just before the rewrite, so a kill between the two logs it again the next night, which folding on key absorbs. At most 2,000 amended rows are rewritten a night (AMEND_LIMIT), so the first run after this change, which owed every amendment since the Aug 2026 full scrape, spreads them over a few nights. A record whose detail page cannot be read keeps its row, unlogged, until a night it can.

The same rewrite drops a row identical in every column to one above it: the same record listed twice, which the full scrape picks up when the listing shifts between two pages, and which flipping an Active row beside an identical Expired twin also makes. The hub-data CSVs carried 187 of them on 26 Oct 2026; the site already showed one of each.

Read it by folding on key and keeping the last entry, never by counting lines: a field that moves twice appears twice. It is append-only, gitignored, and cannot be regenerated from anything — a night not recorded is gone — so it belongs on the extraction Release alongside doc_text.jsonl.

Nothing on the site reads it yet. Publishing an "amended" flag and the history in the detail panel is the next step, and wants a few weeks of accumulation first.

scripts/check_daily_diff.py fails the night if any entity had more than half its previously-Active records flip to Expired in a day, skipped below 5 records where the percentage is noise. That is implausible as real attrition but exactly what a renamed or retired entity looks like — which check_entity_drift.py, run first, is meant to catch before it gets this far. It sees every entity whose flips were patched, even on a night another entity did not finish: those entries are marked "complete": false, and only the scrape time waits for every entity to finish. Its skip below five is the same line as the scrape's all-gone floor, so the small bodies an empty night expires (above) pass it; an entity the scrape refused is not patched and has nothing to report.

Getting the data back

data/ holds two things that are expensive rather than impossible to recreate, and both have a home to fetch them from. Neither is in git.

If the nightly's cache itself is gone (GitHub evicts an entry nobody has restored for 7 days), seed a new one from the newest hub-data release, which carries the CSVs, the stamps, the nightly's document log and, in daily_state.tar.gz, the amendment log, flip ledgers and leak streaks that otherwise lived only in the cache:

gh workflow run seed-cache.yml --ref main -f from_release=true

seed-cache.yml also takes artifact_run_id, a rescue-cache.yml run whose artifact to seed from; it refuses any other run's artifact, since any run here, a fork's pull request included, can upload one by that name. For a local copy:

# CSVs (363 MB, ~20h of scraping) — cached by the nightly workflow.
gh workflow run rescue-cache.yml --ref main   # then download its artifact
# ...or the newest hub-data release's copy (check the date in its tag first). It carries
# scrape_meta.json too; restore it with the CSVs so the stamps match the data.
tag=$(gh release list --json tagName --jq '[.[] | select(.tagName | startswith("ne-contracts-hub-data-"))] | sort_by(.tagName) | last | .tagName')
gh release download "$tag" -D data
(cd data && gunzip -f nu_contracts.csv.gz nu_purchase_orders.csv.gz state_agencies.csv.gz scrape_meta.json.gz)

# Document text and descriptions (1.5 GB, ~36h of downloading) — a Release, because
# nothing else keeps them. doc_text.jsonl is the one that matters; scope.jsonl
# rebuilds from it in about a minute with scripts/extract_scope.py. The release also
# carries documents.jsonl.gz (without it every row's document count is unknown) and
# ai_summaries.jsonl.gz (without it the site has no AI summaries).
gh release download extraction-data-2026-08-17 -D data
(cd data && gunzip doc_text.jsonl.gz scope.jsonl.gz documents.jsonl.gz ai_summaries.jsonl.gz)

Both were lost or nearly lost in August 2026 — the CSVs vanished from the working machine and survived only because CI happened to cache them. Re-publish the Release after any large re-extraction.

---

How it works

The scraper

The site splits entities into two categories needing different handling. Higher Education searches take a Contract/Purchase Order filter and render into a table#entitygrid; State searches have no such filter and render into a table#agencygrid. scrape.py branches on both.

Detail pages are fetched on a pool of sessions that have never run a search. This is not incidental: the site keeps search results in server-side session state and serializes every request touching it, so concurrent fetches on the searching session queue behind one another. Search-free sessions took a page of 25 detail fetches from ~26s to ~5s — about 7x over the full job. See detail_session() before changing it.

Pacing is set from measurement. A page of 25 records costs one results request plus 25 detail requests, so detail fetches are ~96% of traffic and MAX_WORKERS/DETAIL_DELAY are what actually govern load.

• More workers is not faster. An A/B against the live site had 15 and 20 workers finishing in the same ~3.98s while median detail latency rose 1.15s → 1.42s. The site is the bottleneck; extra concurrency only queues. Don't raise it.
• PAGE_DELAY was mostly dead time — consecutive results requests are already separated by the ~2.6s of detail fetching between them. It is now 0.3s.
• Pipelining the next results fetch behind the current page's detail fetches bought a real 36% (355 → 483 records/min) at no extra load.

Each combo reports its median detail-fetch time and flags a sustained climb past 2.0s. The healthy baseline is ~0.95s; a run drifting well above that is asking for more than the site wants to give.

Every request says who is asking. scrape.USER_AGENT is ne-contracts/1.0 (+https://diepjustin.github.io/ne-contracts/), sent by the scraper, the document backfill, extract_text.py, check_entity_drift.py and verify_form_geometry.py — one definition, imported rather than copied, for the same reason the canary is. It used to read Mozilla/5.0 (compatible; research scraper), which is anonymous and, in the Mozilla part, untrue, and verify_form_geometry.py sent no agent at all, so it went out as python-requests. A full run is 20+ hours against a state government server; an administrator who notices that traffic has two options, find the operator or block the client, and the first should not take detective work. The URL is a public page saying what this is and where the code is — enough to be reached through without putting a personal address in anyone's logs.

tests/test_user_agent.py fails the build on a hard-coded User-Agent anywhere in scripts/, and on any module that makes requests without reaching for the constant. That second one is the point: the failure worth guarding against is a new fetch site, which no existing test would ever call.

The document backfill

scripts/backfill_documents.py re-asks every record we already hold what it publishes. It reads Detail URL straight out of the CSVs, so it runs no searches at all — which matters more than it sounds. The site's search results live in server-side state that expires after ~2,000 pages, the failure that nearly cost 170,000 UNL records; detail URLs have no such state, and 900 of them were fetched cold, with no session or cookie, while measuring the loss. At the measured ~6-9 rows/s it is roughly 33 hours for all 741,653.

Stopping it is normal. Ctrl-C finishes the batch in flight, writes it, and exits 0; the same command resumes. data/documents.jsonl is itself the checkpoint — folded on start, never counted by line — so there is no second progress file to drift away from the data it describes. Batches are fsynced, not merely flushed, because this is a job run on a laptop that gets carried between networks.

Two brakes exist because a long run can outlive the conditions it started in:

breaker	what it catches
25 consecutive fetch failures	a closed lid, a dropped VPN, a changed network. Nothing is written for those rows, so they are simply asked again.
25 consecutive "publishes nothing"	the 17 Aug 2026 outage signature, in the one shape the start-up canary check cannot catch: a run that began healthy and went dark fourteen hours in. The canaries are re-asked before any of those rows is recorded.

A row is written only once its page has actually been read, so an unrecorded row and a row not yet reached are the same thing — which is the truth, and is what makes resuming safe.

--status also says whether a run is going, asked of ps rather than a pidfile: a pidfile outlives the run it describes, and the one that died in a laptop sleep on 28 Aug 2026 would have left one behind claiming to be alive. If ps cannot be asked it reports that, rather than "nothing is running" — the same distinction the rest of this project turns on. The run's own log is not evidence of anything. That sleep left the log with three NUL bytes and stale disk blocks from other files spliced into its tail, so grep treated it as binary and silently found nothing in a file that plainly had content. data/documents.jsonl came through it with zero NULs and every line valid, because it is fsynced every 50 rows and only ever appended to.

Why the payload is columns, not JSON

One JSON file of all 739,605 rows is ~104 MB, ~49 MB gzipped, and roughly 380 MB of heap — minutes to load and an out-of-memory crash on a phone. Packed column-wise the same values compress about 5x better, because a column of integers is far more compressible than the same integers interleaved with JSON punctuation.

So the payload is split by access pattern:

	gzipped, as Pages serves it
numeric columns, vendor names, document numbers — loaded up front	6.83 MB
which parser wrote each description (descsrc.bin) — loaded up front	0.03 MB
link tokens — fetched on click, 362 blocks	23.67 MB raw
document lists for rows publishing several — fetched on click, 363 blocks	1.51 MB raw
descriptions — fetched on click, 362 blocks	49.95 MB raw
description search index — fetched only if you tick the box	9.92 MB

docCount joined cols.u8.bin as a seventh column and moved the resident figure by 0.001 MB, because it is one byte a row and gzip eats a near-constant column. It will grow as the backfill fills in real counts; re-measure rather than trusting this line once it has finished. The document-list blocks are mostly their own u16 length header — an empty block gzips to 25 bytes, and the fullest one measured is 18 KB, paid once per click.

descsrc.bin is one byte per row and looks like it belongs in cols.u8.bin but for build_site.py --descriptions-only, which attaches descriptions to a payload that is already built and live and deliberately writes no byte of any resident column file. A column there would have gone stale on that path — descriptions present, every row reporting it has none, nothing failing. Its own file means whichever path writes the descriptions writes this beside them, and verify_descriptions() fails the build if the two ever disagree on a row.

It also exploits the structure of the state's URLs: DT is determined by document type, V maps 1:1 with vendor, and A/D/N vary together across only 98 distinct combinations, so each row carries a small index instead of three long tokens. Every URL is round-trip verified against the original before the payload is written — 1,431,967 of them — so the compression is lossless.

That verification earns its keep: A, D and N each looked entity-determined at two entities and still did across five. At 92 the rule breaks — twelve agencies carry more than one N, and three N values span agencies. The round-trip check caught it rather than shipping broken links.

scripts/ne_format.py owns the layout and is the only place that knows it. Each file holds one item size, so a section's offset is n * itemsize * index — no header, no offset table, no padding — and the reader checks every file's length against meta.count.

Measured in-browser over all rows: a keystroke filters in 4–21 ms, a column's first sort takes 56–199 ms and is cached after, and peak heap is ~90 MB (against ~154 MB for the old payload at 40% of the size). Sorting happens once per column rather than once per keystroke: the page caches an ordering and filtering walks it.

Filters in the URL

Every filter and the sort are written into the query string as they are applied, so the address bar is always a link to what is on screen, and a reporter can send an editor "Roads, contracts over $1 M since 2024" as a URL rather than as instructions:

?agency=Roads%2C+Department+of&type=Contract&from=2024&min=1000000

Values are the state's own names — an agency, a status, a type category — never this build's indices, which move on every rebuild; the same reasoning as the ?doc= permalinks, which are a separate namespace, read first and never written here. Only non-default values are written, so an unfiltered table has the bare address, and the page uses replaceState rather than pushState: a keystroke is not a navigation.

A link naming something this build does not have — an agency the state has renamed, say — is not quietly dropped, because the table underneath would then be a claim about the wrong thing. The value is left unapplied and the status bar says which one.

queryFor and stateFrom are the two directions. Both are pure and both are published under ?selftest=1.

What ?selftest=1 checks

Three things, in this order:

1. Every column against a CRC recorded at build time. Digests are per column so a failure names what broke.
2. The rendered table against the CSV export, cell for cell, over 50 rows sampled from eight scroll positions.
3. 1,000 sampled URLs rebuilt from the payload and compared against addresses taken from the source CSVs, not from the payload.

Run it against the deployed site, not a local server — only a real deployment exercises gzip in transit, and CDN behaviour produced the one design-changing surprise here.

The second one is newer than the other two and exists because they could not see the worst bug this page has had. Every End cell was blank for months while the digests passed the entire time: the payload was correct, render() was wrong, and nothing compared the two. The CSV export reads the same columns through a different function, so a disagreement between the table and the export is one of them being wrong. Each disagreement names the row and the column.

Two columns are not the same string in both places, and both are compared by undoing the display rather than by re-running the code that produced it — re-running it would agree with itself no matter what it read. The amount is read back out of the formatted cell as a number (to the cent, since the formatter is capped at two fraction digits), and the type code is taken from the type cell's hover text, which is where a reader finds it. The permalink under the document number is checked against the export's "Record Link (this site)" column, and every exported line's "Data As Of" against the build's. Three display rules are undone the same way: an open-ended End cell is compared through the recorded date its hover text leads with, a vendor name through the tidying the page applies to it, and the source-oddity mark (marked data-extra) is left out of the cell text. The export's formula guard (an apostrophe before a leading = + - @, tab or CR in a text cell) is stripped before comparing. The CRCs also cover descsrc.bin, descdoc.bin, aisrc.bin and aidoc.bin, and a build with aiFormatVersion 2 fails if aidoc.bin did not load.

Rows are sampled from eight scroll positions rather than one screenful, because past MAX_H the scroller stops being a 1:1 map of the list and scroll position is scaled onto it — a check that only ever read the top of the table would never touch the arithmetic most likely to be wrong.

It was verified the way this file asks for: by putting the bug back. A copy of the page with end re-shadowed in render(), the amount off by a dollar and the wrong type code in the hover text fails with 147 differing cells, naming each row and column. Note that ?selftest=1 now starts the page normally before reporting — it has to, since there is no rendered table to read otherwise — and prints the report below the table.

The neContracts namespace

?selftest=1 also publishes the page's pure functions on window.neContracts: fmtDate, shortType, csvCell, csvParts, docStr, docCountCell, comparator, permalink, yearFloor, yearCeil, amountBound, crc32, queryFor, stateFrom, csvUnguard, csvHead, csvName, comboFilter, isOpenEnded, endShown, oddity, oddityText, displayName, listedPositions and documentName. Every one of them is a function of its arguments and the loaded payload, and every one was unreachable from outside the IIFE, which is most of why this page has never had a unit test.

Only under ?selftest=1. A page a reader loads is unchanged and the object is not there at all. It is an outlet, not an interface: nothing in index.html may come to read it, or the test hook becomes load-bearing and the thing it was meant to observe starts depending on being observed.

yearFloor, yearCeil and amountBound are names for arithmetic that was inline in apply(). A year box becomes the smallest or largest yyyymmdd in that year, or 0 for "no bound" — which is why a row with no date is never excluded by one.

amountBound was a bare parseFloat, which stops at the first character it cannot use: 50,000 read as 50 and 50k as 50 — a filter three orders of magnitude off, showing a plausible record count with nothing on the page saying so. That never reached a reader. Min and Max are <input type="number"> and Chrome discards a comma, a $ and a k as they are typed, handing over 50000 either way; typing 50,000 into the Min box filters correctly and always has. It now strips currency signs, separators and spaces itself and requires what is left to be the whole number, for two reasons: the answer should not depend on one browser's input sanitising, and a value it cannot read in full is now "no bound" — a filter visibly not applied — rather than a truncated prefix that looks like it was.

Two things this does not cover, both unmeasured here:

• validity.badInput. Where a browser accepts unparseable text into a number input it reports value as "", so the box reads 50,000 while no bound is applied and nothing says which. Chrome never gets there; whether Firefox or Safari do has not been checked on this page.
• The year boxes. yearFloor and yearCeil use parseInt and have exactly the same truncating-prefix shape, unfixed.

Publishing

.github/workflows/pages.yml builds the payload in CI and deploys it as a Pages artifact. Pages is set to build_type: workflow; the site is not served from the branch. d/<buildId>/ and manifest.json are gitignored — building locally is how you preview a change, and the result stays on your machine.

trigger	what happens
push to main	restores the last payload from the Actions cache and deploys it. No rebuild, so a portfolio edit does not need the CSVs. (True only from 9 Sep 2026 — see "The payload cache never once hit".)
every night	daily.yml dispatches pages.yml after check_daily_diff.py passes and the data/ cache save has finished, so it rebuilds from fresh data. Its verify-deploy job then waits on that run and checks the live site (below).

The dispatch is deliberately not a workflow_run trigger, and staying that way is the point of the nightly change rather than an accident of it. workflow_run fires on any successful completion and cannot see what the run decided: it would deploy after a night the gate skipped as already done, and after a dry_run, neither of which produced anything to publish. The dispatch runs only if the guard rail step succeeded, so a scrape the guard rail rejects still never reaches the site. It also waits for the cache save: pages.yml rebuilds from the newest ne-contracts-data- cache, and until Oct 2026 the dispatch landed 11–28 s before the save did. It comes before the prunes and the release, so none of those failing can skip a deploy.

The nightly checks that the deploy happened. It used to dispatch and never look back, so a failed Pages build left the site frozen while every nightly stayed green. The verify-deploy job in daily.yml finds the pages.yml run it dispatched (the earliest workflow_dispatch run created after the dispatch time; gh workflow run returns no id), waits on it with gh run watch --exit-status, then runs scripts/check_live_site.py against the published manifest.json. That passes only if the live buildId was stamped at or after the dispatch (it is runner time, UTC), retrying for about ten minutes through the CDN. A red verify-deploy beside a green scrape job means the data is fine and the site is not. One known false red: a push to main landing while the dispatched run is still queued replaces it in the pages concurrency group, so the watch fails even though the push's run deploys the same data; the live-site check still reports what is served.

Every night is a publish night as of Sep 2026. It was Sunday only before that, which left the site up to six days behind a source the state updates daily, and left the guard rail unrun on the other six nights. What that costs, so it is on the record rather than discovered later:

• A rebuild runs nightly whether or not the scrape found anything, and every build gets a new buildId, so a returning reader re-fetches the columns they already had. Gating the dispatch on "the diff report shows something changed" would avoid both, and is not done: a silent stop is this project's characteristic failure, and a site that rebuilds unconditionally cannot go quietly stale.
• check_entity_drift.py now asks the state for its entity lists nightly rather than weekly — two requests, retried once. A drift it finds, or a failure to ask, is a ::warning annotation every night until the committed lists are updated, not a failed step: it used to exit 1, which stopped the deploy for something --daily does not even read (a new agency is simply not scraped until it is added). Run it with --strict before a full re-scrape to get the exit 1 back.
• Seven 60 MB payload cache entries a week instead of one, which is why pages.yml prunes them to the newest three.

Descriptions are built in CI from scope.jsonl.gz, downloaded from the scope-data release (or, while that does not exist or lacks the asset, the newest extraction-data-* release), laid over the previous build's. carry_descriptions_forward() is not a fallback: pages.yml restores the build being replaced from the payload cache before every nightly dispatch, so every nightly build starts from it, and only a build whose restore missed — a push that moved the key — starts from scope.jsonl alone. So a carried description keeps its parser (descsrc.bin) and the document it was read from (descdoc.bin, carried as that document's token and looked up in tonight's list), and is ranked as the record that first described it was: scope.jsonl replaces it exactly where it would have beaten the original. Until Sep 2026 only the text was carried, relabelled "unknown", placed on the record's first document and ranked as though read from there, so no later document's record could ever correct it; the first nightly after a fresh build put "description read from this one" beside the wrong document on 1,050 rows. A carried description whose document was never recorded, or cannot be placed in tonight's list, now marks no document (descdoc.bin 255) and gives way to anything scope.jsonl reads for its row.

documents.jsonl.gz rides on the same release, for the same reason: it is the output of ~23 hours of detail fetches and will never be produced in CI. It is optional — a release without it builds with every row's document count unknown, and the page then claims nothing about any row, which is what a partly-finished backfill looks like anyway. Re-upload it after any backfill sitting, or the site keeps publishing the coverage the release last carried. That is the one way this can go quietly stale: the code will be live and correct while the data behind it is a sitting old.

The build merges the release copy with the cached one rather than replacing it, and that is load-bearing. --daily writes an entry for every new record it finds, and data/ round-trips through the Actions cache, so an earlier --clobber destroyed the document list of every record scraped since the release was cut — 2,118 within a day of shipping, growing nightly, while the site reported 99.7% and looked finished. The cached file is concatenated first so the release wins any key both hold (it is the authoritative backfill) while records only the nightly has seen survive as unique keys. An unterminated last line in the cached copy is dropped first: fused to the release's first line it becomes one unparseable line in the middle of the file, which load_documents refuses to read past — and if the release were ever a single line, that fusion would instead be silently truncated, taking a real record with it.

AI summaries come the same way, from ai-summaries-data with the same fallback; both fixed tags are updated in place by the Mac job, and the build warns when either asset is more than 14 days old. All three downloads go through scripts/fetch_release_asset.py, which tells "not published" (a warning; the build goes on without the file) apart from "the download broke" (an error; the build fails). Until Oct 2026 they ran with 2>/dev/null, so a network failure looked like a missing asset and the site published without descriptions or summaries. A missing asset is looked for again after 10 s, and a failed download is retried once, because the publish script renames a new asset into place in two steps.

Only the site is deployed: index.html, manifest.json and d/ (the build the manifest names, and d/rows/). The artifact used to be the whole checkout minus an exclude list, which served scripts/, tests/ and this README on the public site.

index.html carries no build identity — it reads manifest.json with cache: 'no-store' — so a reader holding a stale page can never pair it with a different build's data.

Why it works this way: under branch publishing the payload had to be committed to be served, and nothing ever left history. .git reached 546 MB, growing ~100 MB per published build, forever. Removing 1,469 files stopped that on 17 Aug 2026. It did not reclaim the ~450 MB already in history; that needs a force-push and would invalidate every clone, so it remains a separate decision.

Nightly automation

daily.yml scrapes all three datasets with --daily every night, from two cron entries set off the hour for the late evening, Central (17 3 and 47 4 UTC) — though see the slippage below for when they actually run. Both fire every night, and the gate step makes the second a backup: a scheduled run proceeds unless every dataset already finished tonight, according to scrape_meta.json in the newest hub-data release. So the backup skips a night the first entry finished, and retries one it did not. A manual dispatch always runs, and any doubt — no release, a download error, an unreadable file — means run, because running twice is harmless and a skipped night is not. "Tonight" runs from noon to noon Central (scripts/nightly_gate.py): every run lands inside one such window whatever the delay, where "is there a release dated today" would skip an on-time 10pm run on a day whose release a late run cut at 3am. The gate this replaced let through only the entry matching the current UTC offset; at the delays below, both entries would have skipped the 14 Mar 2027 spring-forward night. One consequence: a manual run after noon Central counts as that night's scrape — a dry_run too, which publishes the hub-data release but not the site — so the scheduled entries will skip it.

The same test, a dataset at a time, runs at the top of each scrape step, against the scrape_meta.json the cache restored: a dataset already stamped tonight is skipped. A backup let through because state did not finish used to re-read contract and purchase-order from the top, since a finished dataset leaves no progress file. A manual dispatch skips them the same way, so it cannot re-read a dataset that already finished tonight.

Each scrape step has --hours 0.75, so the three stop cleanly, checkpointed, well inside the job's 180-minute limit (a normal night is about 16 minutes for all three), and PYTHONUNBUFFERED=1 so progress reaches the log as it happens. A dataset cut short keeps its old stamp, so the run goes red and the backup entry resumes it.

The scheduler is hours late, not minutes. This said "30–90 minutes" until 9 Sep 2026, which was wrong. Measured across all 28 nights from 13 Aug to 9 Sep, the live cron entry was never once on time: 33 minutes late at best, three hours at the median, and 11.8 hours on 28 Aug. It is also not random. Through 26 Aug it ran about 40 minutes late every night; from 27 Aug it moved, and every night since 2 Sep has landed between 4h20m and 4h50m late. So the scrape labelled 10pm Central has in practice been starting around 2:30am, and the publish it now triggers lands around 3am.

Nothing downstream depends on the hour — that is exactly what gating on the cron entry buys, and it is why the 4-hour regime went unnoticed for a fortnight while every run reported success. What it does cost is the reader: the site's "last updated" stamp trails the label by hours, and last night's contracts are not on the site until the small hours. The cause is not established. The obvious suspect — other scheduled workflows competing for the scheduler in the repository this ran in until the 10 Sep split — does not fit: unl-events-nightly.yml arrived 4 Sep and husker-markets-collect.yml 8 Sep (neither is in this repository), both well after the 27 Aug change. The numbers above are one month of one repo, so re-measure rather than trusting them — the dispatch time of every scheduled run is one call:

gh api "/repos/diepjustin/ne-contracts/actions/workflows/daily.yml/runs?event=schedule&per_page=100" \
  --jq '.workflow_runs[] | "\(.created_at)"' | sort

Two runs land per night, one per cron entry; normally the first scrapes and the second skips in its gate step. Before Oct 2026 the slots were 03:00 and 04:00 UTC.

Keeping the schedule alive. GitHub disables a public repository's scheduled workflows after 60 days without repository activity, and the nightly then stops without a failed run to email anyone. keepalive.yml runs on the 1st and 15th and calls the enable API for daily.yml and itself. If it has been disabled anyway, re-enable both from the Actions tab (or gh workflow enable daily.yml).

Workflow hardening. Every action is pinned to a full commit SHA with its version in a comment, and Dependabot (configured in .github/) opens weekly PRs for the actions and for requirements.txt, each tested by ci.yml. Each job declares only the token permissions it uses; in pages.yml only the one-action deploy job holds pages: write. Checkouts keep no credentials, and every run: step is bash -eo pipefail. tests/test_workflow_hardening.py fails if any workflow, including a new one, drops one of these.

Runners are ephemeral and data/*.csv is gitignored, so data/ round-trips through actions/cache: restored under a ne-contracts-data- prefix, saved under a fresh run-scoped key (cache keys are immutable, so "latest" only works via prefix-restore plus a new key per save), with the oldest pruned via gh cache delete to stay under the 10 GB/repo cap.

A fresh clone's first run has no cache and will fail — scrape.py --daily refuses to treat a missing baseline as "nothing was ever active". Bootstrap once:

1. Commit the existing data/*.csv to a short-lived branch, commenting out the data/*.csv line in .gitignore for that commit only. Use a branch, not main — these are ~346 MB and a revert would not remove the objects from history.
2. Dispatch rescue-cache.yml on that branch; it republishes data/ as an artifact.
3. Dispatch seed-cache.yml on main, passing that run's ID as artifact_run_id.
4. Delete the branch.

Step 3 must run from main. GitHub scopes each cache to the ref that saved it and shares it only with that ref or the default branch, so a cache saved on a short-lived branch is invisible to the nightly and orphaned once the branch is deleted. An earlier version of this had the whole thing on the branch, and it took the nightly down for two days with refusing --daily: data/nu_contracts.csv does not exist — the seed was intact the whole time, just permanently out of scope.

A night that did not finish

A dataset's scrape time is recorded only when every entity in it was read, so a night that does not finish — an entity refused, a deadline, a crash after the other datasets were done, or a leak streak at its third night (see "Strays that never go away") — leaves that time where it was and exits 0, and left alone the job ends green while the page's "Last updated" date stops moving (as it did from 24 Sep 2026). The last step of daily.yml, scripts/check_scrape_stamps.py, compares each dataset's recorded time with the moment tonight began (noon Central, as the gate counts it) and turns the run red, with an ::error line for each dataset that has not finished, if any is older. It compared with the moment the run began until Oct 2026; a backup that skips a dataset the first run finished could never meet that. It runs after the deploy dispatch, the cache save and the release, so it blocks none of them: a red final step beside green ones means data is stale, not that anything was lost.

Downstream consumers

ne-connect (diepjustin/ne-connect, the hub that brings this together with campaign- finance, lobbying and FEC data) reads two things from this repository, and neither can change on this side alone.

• The nightly data release. daily.yml cuts a release tagged ne-contracts-hub-data-YYYY-MM-DD, the America/Chicago date, carrying four gzipped assets: nu_contracts.csv.gz, nu_purchase_orders.csv.gz, state_agencies.csv.gz and scrape_meta.json.gz, plus documents.jsonl.gz when the nightly's data/ has a document log (the entries --daily has written since the cache was seeded, not the full backfill), and since Oct 2026 daily_state.tar.gz: the amendment log and each dataset's flip ledger and leak streaks, for seed-cache.yml (see "Getting the data back"). ne-connect downloads it with the rest and has no use for it. The CSVs keep the eleven columns listed under "Running it". A same-day re-run replaces that day's assets in place (--clobber, then removes any it did not produce), so there is no moment without a release. Only the newest three are kept, and pruning a release deletes its tag too; a nightly step clears tags earlier prunes left behind. ne-connect's nightly picks the newest by tag name, never by createdAt (the date of the commit a tag points at, which ties across nights), and downloads *.gz. Its ingest/sources.py CONTRACT_FILES names the same three CSVs, and its build/check_freshness.py reads the Central date from the tag, warning at 3 days and failing at 7.
• The vendor shards. pages.yml runs scripts/export_vendor_rows.py, which writes d/rows/<A-Z or _>.json: one file per vendor-name initial (upper-cased; any other first character goes to _), each {"VENDOR": [[document number, type, entity, amount, begin date, end date, status, detail URL], ...]}, keyed by the exact Vendor string and covering all three CSVs. ne-connect's contractShardKey() (in its build/build_site.py) computes the same key to decide which shard to fetch from this site's d/rows/. The shard to watch is A.json, because one very large vendor sits in it; export_vendor_rows.py warns only past 90 MB, within 10 MB of GitHub's 100 MB push limit, which d/ never meets because it is gitignored and only deployed.

Changing a column, an asset name, the tag prefix or the shard rule is therefore a change in two repositories. tests/test_export_vendor_rows.py pins the file tuple and the shard key for that reason, each with a comment naming where ne-connect keeps its copy; ne-connect's docs/SCHEMA.md ("Cross-fetch") describes the consumer side, though it predates state_agencies.csv.

---

Guard rails

Read this before changing anything that writes.

Descriptions are the state's words, verbatim. Nothing in them is generated or summarised: OCR text and AI summaries never enter scope.jsonl or the description blocks. The AI summaries are a separate layer with a label of their own (see Generated and OCR'd content). Typos, boilerplate and truncation that the state itself published all stay.

A row's primary document must not move. doc_text.jsonl and scope.jsonl are keyed by view token, so the row's first document is what every description hangs off. The state does not list documents in date order, so a newly filed one can appear first and shift it — build_site.py therefore keeps whatever the CSV already says and only reports the drift. Three of the first 750 rows checked had already moved. Silently adopting the new first token orphans that row's description without failing anything.

docCount 255 means "nobody has asked", never "one". The backfill runs over ~33 hours in sittings, so a build between them has hundreds of thousands of unchecked rows. Reading unknown as one would put a claim on the page about rows nothing is known about. This is meta.incomplete's failure mode, which has now happened twice, written down before it happens a third time.

Never merge vendors automatically. scripts/vendor_groups.json is the only source of truth and every entry was read by a human. A wrong merge invents spending that never happened, which is worse than the fragmentation it would fix. Three properties to keep: no row ever displays a name the state did not publish; the aggregate is labelled as ours in those words ("Grouped by us, not by the state."); and a rescrape that renames a vendor fails the build rather than silently shrinking a total.

Never record an absence you did not observe. A failed fetch is not evidence that a contract has no document, that a field is empty, or that a record is gone. A silent source is the one input that looks identical to "nothing there", and it has now caused the two worst data failures in this project: blank descriptions attributed to the state, and every active contract expired at once.

Put the refusal where the write is. A check that runs before publishing cannot protect data that the scrape has already written and cached. check_daily_diff.py was built for exactly the mass-expiry case and never fired, because it guards the publish leg rather than the daily write. Publishing nightly shortens that gap from a week to a day and does not close it: the CSV is already written and cached by the time the guard rail runs. Keep the two apart in the value itself — "" for "read it, there is nothing" and None for "could not find out" — and hold back anything unknown rather than writing it. The CSV is a record of what the state published, so a value in it must be something we actually saw.

Every script that writes needs its own refusal, not one somewhere upstream. document_service_healthy() guarded scrape.py from Aug 2026 and not extract_text.py, which fetches documents itself and files two of its four verdicts permanently. It guards both now, and the canary is imported rather than copied so there is only one to keep current.

Check extraction against the source document, not against itself. Every sanity check on the description parsers compared output to other output, and all of them passed while the University parser was truncating 93% of what it read. The bug was found by opening a PDF and reading it. When you change extract_scope.py, pull ten real documents and read them.

How to check a description is verbatim: compare its words against its own document's text as an ordered subsequence, not a contiguous string. 4,000 sampled documents pass at 100%. The contiguous check reports 82% and every failure is spurious — University descriptions are reassembled across the money columns by design, so the joined string is genuinely not contiguous in the source. Reading that 18% as a data problem sends you chasing a bug that is not there.

A parse rate is not an accuracy rate. This project reported "100%" and "97%" parses for the two purchase-order forms while both returned truncated text. Those numbers only ever meant "the pattern matched".

Count documents, not log lines. doc_text.jsonl is append-only and a later entry supersedes the earlier one, so wc -l overstates by every retry. Reading it wrong produced two false numbers in one session: 40% complete that was really 23%, and 14,551 errors that were really 63, because the retries that fixed them were counted alongside the failures they replaced. Fold on tok, keep the last entry, then count.

No tracked text file may carry a raw NUL byte; tests/test_no_nul_bytes.py enforces it. index.html carried one for five days — a sort separator written as a raw character instead of the \x00 escape. Browsers did not care, but file reported the page as data, grep matched nothing and exited silently, and git showed no diffs for the project's most important file. Then the paragraph documenting that bug contained the byte it was describing, so the handoff was itself unsearchable. If a tool suddenly stops finding text that is plainly there, check for NULs first.

meta.incomplete has now failed twice, both times in the same direction — claiming the entire state was uncollected when it was finished. First from load_progress() returning a pair bound to one name; then from moving the build into CI and leaving the scrapers' checkpoints behind on the laptop. It is the project's most consequential single value, because it speaks to whether any number on the page can be trusted. A missing checkpoint now means "unknown" and claims no gaps, and the checkpoints are tracked in git so a public claim never depends on a file riding in a cache.

Two things the publish must never do, both asserted in the staging step: ship data/ (363 MB of scraped CSVs), and deploy a manifest.json naming a build directory that is not in the artifact.

Never let an empty data/ reach the cache. Restore matches on prefix and takes the newest hit, so one cache saved from an empty directory shadows the good one on every subsequent run and the failure feeds itself.

---

Things that bit us

find where it wanted find_all, for about a year. get_view_url took the first ViewDocument link on a record's detail page and stopped. Every record with amendments, renewals or a signed copy alongside the award kept one document and dropped the rest — ~67,000 documents, ~9% of contracts and ~19% of state agency rows. Nothing failed: every row had a working link to a real PDF, the round-trip URL verification passed, and the page looked complete. It was found by a reader clicking through to the state's own page for a UNL Axon contract and counting nine documents where the site offered one.

The general shape is worth more than the fix: a scraper that takes the first match will never tell you there was a second. When a page can hold a list, count it and record the count, even when today's answer is always one.

Ranged HTTP requests are unusable on GitHub Pages. A range request advertising gzip — which browsers always do, and fetch() cannot override — is served against the compressed representation. Ask for bytes 100–115 and you get 16 bytes of a gzip stream plus a Content-Range denominator that is the compressed length. This is why deferred data is block files, never byte ranges. Pages does gzip application/octet-stream, so raw binary compresses in transit with no client-side work.

The site's search results live in server-side state that expires. After ~2,000 pages of continuous paging every further page returns "No results found", indistinguishable from the end of the data. Reading it as the end once marked UNL purchase orders complete at 144,425 of 178,573 records, with no error anywhere. The scraper re-runs the query and returns to its position; that fired twice during the Medical Center's 9,186 pages and saved ~170,000 records. Any long paginated scrape of this site needs that recovery.

A <tr> cannot be 24 million pixels tall. The virtual scroller sizes spacer rows to the full list height, which browsers cap at 2^24 px. At 739,605 rows × 33 px that is 24.4 M and the bottom third of the table silently becomes unreachable — the scrollbar just stops. The page caps the track at 15 M px and scales scroll position past that.

The sticky filter bar was taller than the phone it was pinned to. .controls had been position: sticky since the first commit, at every width. On a phone it wraps to 668px (375 wide) or 732px (320 wide), and fitScroller holds the table at MIN_TABLE_H (220px) and expects the reader to scroll the page to reach it — so the pinned bar followed that scroll down and sat on the table. Swept with document.elementFromPoint at every page-scroll offset, the most of the table ever visible was 0px at 375x667, 390x664, 320x640 and 320x568, 164px at 390x844 and 60px at 844x390, and a focused row sat underneath the bar at every one of them (focus() at each, and a real Tab press from Reset at 375x667). Desktop and tablet were fine, because the page barely scrolls there and the bar never gets the chance to stick.

The bar is now static at (max-width: 720px), (max-height: 700px). Sticky needs the viewport to hold its 12px offset, the bar and the 220px table: 900px for an upright phone 375 wide, which almost none has once the browser's own bars are counted, and ~623px above 720 wide, where the tallest bar (391px, just past the breakpoint) sits. 700 leaves a row of wrap for fonts wider than the macOS ones it was measured with. After: the full 220px of table reachable and the focused row uncovered at all six; still sticky at 768x1024, 1024x768 and 1280x800. tests/test_page_layout.py cascades the stylesheet for a given viewport and fails if the bar is sticky anywhere it cannot fit. A sticky element has to be shorter than the viewport, less whatever it must leave room for, at every width the page supports — check it at a phone size, not just the one it was drawn at.

Both purchase-order forms wrap their description column, and both parsers stopped at the first line. The single worst bug here, and it survived because the documentation asserted the opposite: that the University's 40-character column cut the text and we were faithfully reproducing the state's truncation. It does not cut, it wraps. A reader checked document 4740007268 and found the site showing "GENERAL CONSTRUCTION SERVICES FOR" where the state wrote "GENERAL CONSTRUCTION SERVICES FOR REMODEL OF BOB DEVANTEY SPORTS CENTER PER UNL INVITATION TO BID 909353-12." on a $15,027,565.88 purchase order.

	scale of the loss	cause
University	54,280 of 58,060 items wrapped; 93% truncated	read only the line carrying the money columns
State agency	892 of 1,367 tails discarded	rejected any tail containing a digit

Neither continuation can simply be appended, for different reasons. On the University form, PDF extraction does not emit the page in reading order, so the lines under a row are often the vendor address block and then the table header — FURNITURE is the stop list, built by counting the most common line following an item across 40,000 documents. On the state form the tail may be the next row; the marker is its four-decimal quantity and unit-price columns, because nobody types "1.0000" into a description and the digit test that preceded it took 1/2 PINT/CONTAINER, 1% with it.

Both caps were also doing the cutting themselves — 12 lines bound 26% of University items, and the state's 80 characters sat below the 99th percentile of genuine tails. They are now 25 and 400, chosen so they never bind. If either starts binding, the form has changed and the fix belongs in the stop lists, not the cap.

Three further row-skipping bugs came out of diffing a full re-extract rather than sampling: four-decimal unit prices (a capacitor order described itself as "SHIPPING", the one line priced in whole cents), no-charge rows printing one money column instead of two (an antibody order kept only what it was billed for), and the form being chosen by whichever pattern fired first — which made University rows start matching inside state grocery orders once the money pattern widened.

A, D and N in detail URLs are not entity-determined. True at two entities and at five, false at 92. Stored as the 98 combinations that actually occur.

load_progress() returns a pair. Binding it to one name made every entity test as uncollected. See the meta.incomplete guard rail above — this is half of why it is there.

The site renders entity names in caps on state result grids (DRY BEAN COMMISSION) but the dropdown uses title case. Rows record the canonical dropdown name.

A two-day outage marked every active contract in the database Expired. On the night of 17 Aug 2026 the state's search answered "No results found" for every entity. That is not a shape --daily was built to doubt: scrape_entity re-runs the query and resumes when a page after the first comes back empty — the guard that saved 170,000 records from expiring session state — but an empty first page is treated as a clean finish. So seen came back empty, known - seen was everything, and one run wrote:

Patched 10,358 row(s) Active -> Expired.
Patched 12,233 row(s) Active -> Expired.
Patched 21,472 row(s) Active -> Expired.

44,063, which is every active record there was. The run was green and every step succeeded. Nothing in the data could show it had happened, and nothing could undo it: --daily only ever flips Active to Expired, so there was no path back to the truth.

A guard rail for exactly this existed and did not fire. check_daily_diff.py fails when an entity loses more than half its active records in a day. It runs if: publish == 'true', which in Aug 2026 meant the Sunday-only publish leg; 17 Aug was a Monday, so it did not run for six days. And even on a Sunday it would have been too late: it gates publishing, while the damage is done by the scrape, which has already rewritten the CSV and saved it to the Actions cache. A check that runs before publishing cannot protect data that is already written.

So the refusal now lives in the scrape, where the write happens. An entity that returns nothing while it had five or more active records is refused, as is one where more than half of 20+ active records vanish at once; both re-check next run. Ordinary expiries still flip, and a small body genuinely clearing out its four contracts still flips — a guard that freezes the database would be its own bug, so both cases have tests.

It compounded quietly. Once every record read Expired, the next night's scrape had no memory of them: known_active was empty, so all 44,063 active contracts looked new and were written again as fresh rows. The only reason that never reached readers is that publishing was gated to Sundays at the time, so no build shipped in between. That margin is gone — publishing is nightly as of Sep 2026, and the same outage today would be on the site the same evening. It is the refusals inside scrape.py, added after this, that stop it now; the weekly gate was luck, not a defence. Repaired by deleting the two poisoned Actions caches and rebuilding from the pre-outage one; the restored counts came back at exactly 44,063 active and 695,542 expired, which is the local count less the 128 duplicate rows the build drops — that reconciliation is what proved the older cache was undamaged.

Then the refusal froze the database it was guarding. The first version had no floor: any entity answering empty was refused, however few records it had. On 25 Sep 2026 the state stopped listing the one active record of the Coordinating Commission for Postsecondary Education, and that one real ending was refused every night after. The scrape time and the diff report were written only on a night every entity finished, so the state dataset's stamp stuck at 24 Sep — and the page shows the oldest stamp — while the 300 flips patched on those four nights (42, 118, 120 and 20 rows) reached no report: check_daily_diff.py went on printing "Guard rail passed: 2 dataset-run(s)" with the state never among them. The test claiming a four-contract body could clear out had only ever emptied three of the four. The floor is now five, the guard rail's own MIN_BASELINE, and a night that does not finish still reports what it patched. What the floor costs on a night the state answers empty is under "Daily updates".

The End column was blank for every row, from the first build. A local var end for the virtual scroller's row window shadowed the module-level end column, so fmtDate(end[id]) read a property off a Number. Three things hid it: the payload was always correct, the crc32 selftest on end passed the entire time because it checks the payload rather than the page, and the CSV export sits outside that function — so exported files carried end dates while the table never did. Renamed to stop. Both of these were found by Justin reading the live site, not by any check in this repo.

Two things now stand where nothing did. no-shadow, via scripts/lint_page.py, refuses the declaration itself, and found six more of that shape still live — including a parameter named end in unpackDocuments. And ?selftest=1 now reads the rendered table back out of the DOM and compares it against what the CSV export builds for the same rows, so a correct payload rendered wrongly fails a check instead of shipping. See "Linting the page" and "What ?selftest=1 checks".

The page explained a blank description by guessing, and was wrong a quarter of the time. It told readers a missing description was "most likely a scanned image rather than text". True of 89,180 documents; false for the 29,663 that have a perfectly good text layer and simply defeated our parsers — 16,736 of those carry over 2,000 characters of extractable text. As a population statistic the sentence was defensible; standing under one specific contract it was a false claim about that document, and it also quietly blamed the state for our own parser's limits.

Spotted by reading the site, not the code: the documents looked like native PDFs. The page now says only "No description could be read from this document." The cause is knowable per document — doc_text.jsonl records text or scanned for every one — so this could be reported precisely rather than dropped, but a plain sentence beats a confident wrong one, and the breakdown lives here instead (see "Descriptions" above).

"No document" and "we could not find out" were the same value. For about two days from 17 Aug 2026 the state stopped serving documents: ViewDocument returned an ASP.NET error page, and detail pages kept returning HTTP 200 while quietly rendering no link at all. That is byte-for-byte what a contract with no document attached looks like, and get_view_url() returned "" for both — so a scrape during the outage would have recorded "no document" as a fact for every new contract and published a site with silently missing links. No step would have failed.

Nothing was actually damaged, but only by luck: the state's search was down at the same time, so the nightly found no records to write, and publishing was weekly then, so no build shipped either. A partial outage — search up, documents down — would have poisoned the data on any publish day, and every night is a publish night now.

get_view_url() now answers three ways: a URL, "" for a page read cleanly that offers nothing (a real fact), and None for could-not-tell. A None record is held back rather than written, so it stays unknown to the CSV and the next run asks again. Because the outage made detail pages look legitimately empty, no amount of page parsing can tell the two apart — so document_service_healthy() checks three documents known to carry files before a scrape starts, and both --daily and a full sweep refuse to run when all three have stopped offering theirs. It is deliberately biased towards "up": a canary that cannot be fetched proves nothing and is skipped, so rotted canary URLs cannot silently halt every run. One request when the state is healthy.

Not one of the 32,227 "unavailable" documents was ever a 404. extract_text.py filed a document as unavailable — the state has no file, never asked about again — on either a 404 or a 200 whose body was not a PDF. Only the second ever happened. Every one of those verdicts was reached from a 200, 17,991 of them image/tiff: real scans the state does hold and we could not parse. The corpus reported "the state has no file to serve" for 32,227 documents, and the run summary printed those words, without the state having once said so.

Found while adding the health gate below. The classifier now separates a 404 (unavailable) from a file we cannot read (unsupported, not retried but not an absence either) from anything that is not a file at all (error, retryable), and --retry-non-pdf re-asks the ones already on disk. It also keeps a hash and the first 300 characters of any non-PDF body, because the 14,185 HTML answers already recorded cannot be told apart from an outage page — nothing kept the body.

The state's document service went down again on 22 Aug 2026, and extract_text.py had no guard. Every ViewDocument request returned HTTP 200 with an HTML error page: "An internal error occured: An error occurred within the Unity API: The type initializer for 'Hyland.Core.CoreUtility' threw an exception." document_service_healthy() had guarded scrape.py since the August outage, but extract_text.py fetches documents on its own and was never taught the lesson.

A run started that morning would have recorded a permanent absence for every document it touched. Worse, --retry-errors — the pass most worth running, over 14,586 recoverable documents — would have spent that verdict on exactly those. The gate is now in front of the extraction loop, one request before the run and nothing written until it answers, and run_extraction_cycles.sh stops rather than spinning because it runs under set -e.

Nothing was damaged, again by luck: the outage happened on a day nobody was extracting. That is twice now that the thing standing between this project and poisoned data was the calendar.

A row with no document blocked every rebuild for two days. The permalink triple ?doc=&agency=&type= is disambiguated by appending &d=<view token>, and build_site.py refused outright when an ambiguous row had no token to be named by — correctly, because shipping it would mean a link that silently opens its twin. On 22 Aug 2026 the state added Peru State College's 80-3-3305 renewal beside the existing one: Tutor.com, $5,600 both times, 2025-26 expired with no document and 2026-27 active with one. A renewal, not a duplicate, so both rows need to stay addressable.

Nothing was wrong with the guard except that it had no answer. A row with no document does not need naming: within its group, not carrying &d= is what identifies it, and the page now resolves a bare permalink to the member with no document. That works for exactly one such row, so the refusal stays for two — measured across all 741,653 rows there are 302 ambiguous groups, one has a document-less member and none has two.

Worth noting how it surfaced. The build had been green since 20 Aug because CI restores the previous payload and only rebuilds when the cache key misses; the row landed in the nightly's CSVs and sat there. A push touching scripts/** missed the key, forced the first real rebuild in two days, and found it. The cache key is doing the job it was added for — but between rebuilds, a data condition that breaks the build is invisible.

write_payload() copies the meta dict it is given. The search index's wordCount and the vendor groups are added afterwards, so on a full build they never reached meta.json — only --descriptions-only rewrote the file at the end, which is the sole reason the published build ever had them. Every full build was shipping with vendor grouping silently absent. meta.json is now written last and re-read to confirm.

---

Open work

The full-text document search prototype is retired (Aug 2026). It built a SQLite index over data/doc_text.jsonl and served it in chunks. The chunked index is no longer published (GitHub Pages ranges against the compressed representation; see the note at the top of scripts/ne_format.py). scripts/build_search_index.py, scripts/chunk_search_db.py and the unreferenced scripts/run_state_scrape_cycles.sh were removed on 8 Sep 2026 and live in git history; prototype-search/ stays gitignored because the 503 MB local copy is still on disk.

~3,021 unreviewed vendor clusters, holding most of the $11.02 B above. scripts/suggest_vendor_groups.py proposes candidates ranked by spend and decides nothing; a grouping exists only once somebody writes it into scripts/vendor_groups.json. The money is concentrated, so a few dozen decisions cover most of the error. This is the highest-value unfinished work.

89,180 scanned documents — 42.9% of contracts — have no text layer, and no parser reaches them. This is the difference between 94.8% description coverage of readable documents and ~95% of all of them. OCR has been run since (Apple Vision, not Tesseract; see "Generated and OCR'd content"), but its text is deliberately kept out of the descriptions, because it would sit beside the state's own words, which are promised verbatim. What is open is whether OCR-recovered descriptions may be published under a label of their own, and the accuracy measurement that decision needs, made on descriptions specifically rather than text generally. The yield is known — the extractor finds something to return in about 16–17% of the OCR'd documents — but a yield says nothing about whether what it returns is right.

Records the state publishes already ended are not collected. The nightly searches Status=Active only, so a record that first appears Expired never enters the dataset (see the caveats): about one in four new University purchase orders is missing. The fix is an append-only Expired sweep of the four University purchase-order combos (UNMC, UNL, UNO, UNK; about 90% of the loss), ordered newest-first and stopping on Begin Date, plus a one-time catch-up for the late-Aug 2026 hole. It needs one manual check before any code: that the state's Expired search honours sort=BeginDate&sortdir=DESC as a true date sort that still holds on page 2 when the parameters are repeated on every request. Until then the caveat is the whole mitigation.

Change tracking records but does not publish. --daily now writes every amendment to data/changes.jsonl and rewrites the row, so the site shows the current values (see "Daily updates"). Nothing on the site reads the history yet: an amended flag and change count as resident columns, the history itself as deferred blocks reusing the write_desc_blocks pattern, and a filter beside the existing data-quality ones. That wants a few weeks of accumulation before it says anything worth showing.

29,663 readable documents have no description and mostly never will. This closes a line of investigation rather than opening one: 83% carry no numbered heading, no summary and no line items, because they are email threads, signature pages, notices to proceed and change-order stubs. There is no statement of the work in them for anything to find, an LLM included. Every candidate pattern measured together reached 5.4%. Three were tried and rejected: a "Project:" line (177 documents), an "engages" clause (647, all inside the SERVICES ones already parsed), an "RE:" line (21, mostly email subjects — a different thing wearing a description's clothes). Do not spend a week here.

19% of state agency documents do not exist — the state's own viewer has no file. No method reaches those rows, ever.

The payload cache never once hit. pages.yml keys the built payload on the code and checkpoints that decide what it contains, so an ordinary push can redeploy the last build instead of spending three minutes rebuilding it. It never did. From the day it was written until 9 Sep 2026 every single push logged rebuilding: no payload to fall back on.

The key was hashFiles('ne-contracts/scripts/**', 'ne-contracts/data/*_scrape_progress.json'), and it returned a different digest on every run. Seven consecutive runs, all checking out a byte-identical scripts tree (a5330ff) and identical progress blobs, produced seven different digests — so the restore looked for a key no save had ever written. Within a single run restore and save agreed, which is why nothing ever looked wrong: the failure is only visible by comparing runs, and a rebuild is silent and green.

What hashFiles was actually seeing was never identified. It is not __pycache__ under scripts/ — pytest writes none there — and it is not the data cache overwriting the progress files, because restore and save agree within a run. The fix does not depend on knowing: the key is now ne-payload-<git tree of scripts>-<git tree of data>, computed by git rev-parse HEAD:<path> in its own step. A tree id is a property of the commit, not of whatever is lying in the workspace beside it, so it cannot drift the way the digest did.

The semantics that matter are unchanged, and they are the reason this is keyed at all rather than reused blindly: the key must move when the builder does, or a push carrying a build_site.py fix redeploys the stale payload and reports success. That is the 17 Aug coverage-note bug two entries down.

Cost while it was broken: every push to this repo — including portfolio-only edits — did a full payload build and needed the CSV cache it was designed not to need, and banked another 60 MB entry. 38 of those had accumulated by 9 Sep, 2.23 GB, sharing the 10 GB budget with the ne-contracts-data- caches that stand for twenty hours of scraping.

~450 MB of superseded payloads remain in git history. Publishing no longer adds to it, but reclaiming what is there needs a force-push that invalidates every clone.

---

License

MIT for the code. The underlying records are public data from the Nebraska Department of Administrative Services.