CATS — Contextual Ambiguity & Trust Scoring
Trust intelligence for OSINT sources — not fact-checking, but source reliability over time.
What CATS measures
CATS scores how reliable a source's behaviour has been over time — narrative consistency, sentiment volatility, publishing gaps, signs of algorithmic manipulation — not whether any single piece of content is true. It does not fact-check, does not label sources as "disinformation", and does not replace human judgement: every score is an ordinal ranking with a full explanation attached, meant to prioritise where a human looks closer.
See it run
$ COHERENCE_BACKEND=sbert cats score ansa.it --source-type news
Source https://ansa.it (feed: https://ansa.it/rss.xml, 28 messages, 2026-09-20 → 2026-09-21)
Score 62.87 band: medium_high
Driver silence (share 39.8%)
Signals coherence 20.42 · volatility 7.41 · silence 0.0 · gaming 39.0 · domain_provenance 0.0
Language italian (1.0) Evidence 28 ≥ 3 ✓
Note Ordinal score, not a probability. Cross-validate key claims. See docs/architecture.md.
COHERENCE_BACKEND=sbert is not decoration: why.
Try it in 30 seconds
pip install cats-scoring
python -m spacy download it_core_news_lg # Italian NLP model (large download), needed by `coherence`
cats score <url> # any source URL — feed autodiscovery included
Without the spaCy model coherence is not measured: it enters the score as a
neutral 50, which can move the score by up to ~20 points. cats score then prints
a warning and forces human review. If feed autodiscovery fails on a site
(no RSS/Atom feed found), pass the feed URL itself, e.g.
cats score https://www.repubblica.it/rss/homepage/rss2.0.xml.
To rank several sources side by side (failures are listed, not fatal):
cats compare ansa.it ilpost.it https://www.repubblica.it/rss/homepage/rss2.0.xml --source-type news
cats compare --file sources.txt --json # one URL per line, '#' for comments
The table shows score, band, main driver, the four raw signals, message count
and review flags. Compare only sources scored together, with the same
--source-type and install: a high rank describes publishing behaviour, not
truthfulness.
The coherence backend matters
data/calibrated_weights.json was calibrated with the SBERT coherence
backend. On the default spaCy-NER backend coherence — ~0.40 of the weight for
news sources — is close to inert (mean 1.7 / sd 5.3 against SBERT's 23.3 / 11.6),
so a default install forfeits its contribution and lands nearer ≈0.62
concordance than the 0.750 the calibrated weights reach.
For the numbers quoted here:
pip install -r requirements-sbert.txt # pulls torch + transformers, ~2 GB
export COHERENCE_BACKEND=sbert
It falls back to NER silently when the model is unavailable — check the logs
for sbert_loaded to confirm it took effect.
Measurement.
Or as a library:
from cats.lite import score_feed
result = score_feed("https://example-news-outlet.it", source_type="news")
print(result["trust_score"], result["band"])
print(result["explanation"]["primary_driver"])
Or from an LLM client such as Claude, through the MCP server
(score_source, score_messages, compare_sources, explain_bands):
pip install "cats-scoring[mcp]"
claude mcp add cats -- cats-mcp
Setup and tool reference: docs/mcp.md.
Try it in the browser instead:
A real use case
An OSINT analyst vetting a list of public source URLs before citing them in a report runs cats score <url> on each one and reads the explanation — which signal drove the score, and why — rather than treating trust_score as a verdict. A medium or low band means "look closer, cross-validate", not "discard": CATS surfaces behavioural patterns (irregular cadence, narrative drift, a clone domain) for a human to weigh alongside the actual content. It never outputs a "disinformation" label.
Signals
| Signal | What it measures | Method |
|---|
| Coherence | Entity/argument consistency across messages | spaCy NER + Jaccard (or optional Sentence-BERT) similarity |
| Volatility | Abrupt narrative tone changes | TextBlob (or optional BERT) sentiment spike detection |
| Silence | Anomalous temporal gaps in publishing | Gap analysis vs. source-type thresholds |
| Gaming | Signs of algorithmic manipulation | Repetition + TTR + burst + vocab diversity |
On top of the four behavioural signals, an asymmetric domain-provenance penalty lowers the score of impersonation/clone domains (rare/cheap TLDs, free-hosting subdomains, brand typo-squats) when a source URL is supplied. It only ever lowers a score, never rewards a clean domain — see architecture.
Trust score bands
| Score | Band | Recommended action |
|---|
| 80–100 | high | Lower review priority; still cross-check key claims |
| 60–79 | medium_high | Cross-validate key claims |
| 40–59 | medium | Human review recommended |
| 20–39 | low | Human review required |
| 0–19 | very_low | Do not use without validation |
⚠️ Scores are ordinal rankings of source reliability, not absolute probabilities.
A high band means consistent publishing behaviour, not truthful content: a
source that publishes disinformation on a steady schedule can still score high.
Honest limits
- Default NLP accuracy is roughly 55–62% (spaCy NER + TextBlob); optional SBERT/BERT backends do better.
- The default install does not reproduce the calibrated numbers: the shipped weights assume
COHERENCE_BACKEND=sbert, and on the default NER backend coherence is close to inert (≈0.62 concordance instead of 0.750). See the backend note.
- Ranking still leans heavily on one signal (
silence), with coherence as the main tie-breaker.
- The default stack is Italian-optimised — other languages are detected and flagged, not blocked, but accuracy degrades.
- Calibration is validated on a 56-source train / 53-source future-holdout split — informative, not large-scale.
- Scores are ordinal, not probabilities, and are not a substitute for human review.
Full detail, numbers and the validation history: docs/architecture.md and the calibration findings.
Full deployment
Beyond the zero-infrastructure library, CATS also ships a full multi-tenant FastAPI service — PostgreSQL-backed audit logging, Redis rate limiting, batch scoring, and a GDPR Art. 13–22 contest/appeal flow — behind the same four signals and the same guarantees. See docs/api.md for the REST reference and docs/architecture.md for the deployment topology and security design.
Documentation
Roadmap
CATS has shipped through v1.9 (calibrated weights, the domain-provenance penalty, the language/evidence guardrails, then the CLI with cats compare, the MCP server and SSRF-safe feed fetching); active work is on adoption (a public demo, first users, human labels) before any further signal changes. Full plan: docs/piano_sviluppo_roadmap_2026-09.md.
Contributing
Issues and PRs are welcome — see CONTRIBUTING.md for the
dev setup, code standards, and the checklist for adding a new signal. Please
also read the Code of Conduct.
If CATS is useful to you, consider starring the repo ⭐ — it helps others
doing OSINT/disinformation work find it.
License
MIT — [email protected]
