Zmistik

Content monitoring and an editorial assistant for Telegram channels (content-arbitrary is the repository's and the deployment's name). It watches your sources — X accounts, RSS / Atom feeds, public Telegram channels, YouTube channels and websites (through their sitemaps) — and brings every new post to you for review: scored by Shadow Radar, rewritten in your channel's language, ready to edit, schedule or reject. What you approve is published to the channel.

Runs as a scheduled job on Vercel: every 15 minutes it checks every source for posts it has not seen before.

X accounts ─────────┐
RSS feeds ──────────┤
YouTube channels ───┤
Telegram channels ──┼─► sync adapters ─► processCandidate ─► Radar ─► translation ─► review ─► edit / schedule ─► your channel
Website sitemaps ───┘   (/api/cron/sync)        │
                                                └── PostgreSQL (sources, posts, decisions, cursors)

Without review switched on, posts go straight to the channel, as the bot first did.

---

Table of contents

• What it does
• Before you start
• A. Local setup
• B. Create an X developer app
• C. X API permissions and scopes
• D. Create a Telegram bot with BotFather
• E. Add the bot to your channel
• F. Get your TELEGRAM_CHAT_ID
• G. Create a PostgreSQL database
• H. Run migrations
• I. Configure .env
• J. Test the Telegram bot by hand
• K. Test with DRY_RUN
• L. Deploy to Vercel
• M. Add environment variables in Vercel
• N. How Vercel Cron works here
• O. Trigger the cron endpoint manually
• P. Debugging common failures
• Q. Telegram 429 and retry_after
• R. When X does not return the media you expect
• RSS and Atom feeds
• Telegram channels
• YouTube channels
• Website sitemap sources
• Multiple channels
• Review website
• Shadow Radar
• Publication history
• Stories
• Configuration reference
• Architecture decisions
• Project structure
• Testing
• Operating notes

---

What it does

Each run:

1. Fetches the account's recent posts (GET /2/users/:id/tweets), using a stored since_id cursor so it only pays for and processes what is new.
2. Keeps only posts that carry photos or videos — or, for a source with Posts without media switched on, text-only posts too. Reposts and replies are skipped by default, both configurable.
3. Processes posts oldest first, so a burst of posts arrives in the channel in the order they were written.
4. Picks the right Bot API method:
   • 1 photo → sendPhoto
   • 1 video → sendVideo (highest-bitrate MP4, supports_streaming)
   • 2+ items → sendMediaGroup (one album; photos and videos may mix)
   • no media (text-only posts, when enabled for the source) → sendMessage
5. Builds a clean caption: the original post text, then Source: https://x.com/….
6. Records the result so the same post is never published twice.

A published message looks like this, and nothing more:

Four photos from the shoot

Source: https://x.com/someaccount/status/1750000000000000003

---

Before you start

Plan requirements (Vercel)

The schedules in vercel.json — the sync every 15 minutes and the every-minute publisher for scheduled posts — require a Pro or Enterprise plan, which allows intervals down to once per minute and fires within the specified minute.

On Hobby, cron jobs are limited to once per day and a more frequent expression fails the deployment. Either change the schedule to a daily one (e.g. "schedule": "0 9 * * *" — Hobby crons fire somewhere within that hour, not on the minute), or keep the 15-minute schedule and trigger /api/cron/sync from an external scheduler (GitHub Actions, cron-job.org, Upstash QStash) sending Authorization: Bearer $CRON_SECRET. The same goes for /api/cron/publish-scheduled, which needs to run every minute. Nothing else differs.

The sync route asks for a maxDuration of 800s, Pro's maximum, and stops taking new posts after 560s so it never gets near it (see Operating notes). On Hobby the ceiling is 300s: lower maxDuration and SYNC_TIME_BUDGET_MS to match.

The X API is paid, per post read

X has moved to pay-per-usage pricing: you buy credits and they are drawn down per request. Reading posts costs around $0.005 per post, or about $0.001 per post when you are the authenticated owner of the account you are reading. There is a hard ceiling of ~3 million post reads per billing cycle on standard accounts, and a 24-hour de-duplication window (requesting the same resource twice in a day is billed once).

This is why the app stores a since_id cursor and why X_FETCH_LIMIT defaults to 20 rather than 100 — an idle account costs you almost nothing per run, so syncing every 15 minutes rather than hourly costs no more: the same posts are read either way, just sooner. Check current pricing at https://docs.x.com/x-api/getting-started/pricing before committing to a schedule.

---

A. Local setup

Requirements: Node.js 20.9+ and a PostgreSQL database.

git clone <your-repo-url>
cd content-arbitrary
npm install
cp .env.example .env

Fill in .env as you work through sections B–I, then:

npm run db:migrate     # create the tables
npm run telegram:check # verify the bot can post to your channel
npm run sync:local     # run one cycle (DRY_RUN=true by default)

Useful commands:

Command	What it does
npm run dev	Next dev server (the cron route is not scheduled locally)
npm run build	Production build
npm run typecheck	TypeScript, no emit
npm run lint	ESLint
npm test	Full test suite
npm run db:generate	Regenerate migrations after editing src/db/schema.ts
npm run db:studio	Drizzle Studio, to browse the tables
npm run sync:local	Run one sync cycle from the CLI
npm run telegram:check	Four-step Telegram configuration check
npm run webhook:set -- <url>	Register the approval webhook
npm run webhook:info	Show the registered webhook and last delivery error
npm run webhook:delete	Remove the webhook

---

B. Create an X developer app

1. Go to https://developer.x.com/en/portal/dashboard and sign in.
2. Create a Project, then an App inside it.
3. Open the app's Keys and tokens tab.
4. Under Authentication Tokens, generate the Bearer Token.
5. Copy it into X_BEARER_TOKEN. It is shown once — regenerate it if you lose it.

Then set the account you want to mirror:

• X_USERNAME=someaccount (without the @), and ideally
• X_USER_ID=1234567890 — the numeric id.

Set X_USER_ID if you can. Without it, every run spends an extra API call resolving the handle. To find it once:

curl -s "https://api.x.com/2/users/by/username/someaccount" \
  -H "Authorization: Bearer $X_BEARER_TOKEN"

C. X API permissions and scopes

This app only reads, so an app-only Bearer token is enough — no OAuth user flow.

• Endpoint: GET /2/users/:id/tweets
• Scopes (if you use OAuth 2.0 instead): tweet.read, users.read
• App permission: Read is sufficient

The account you mirror must be public. This project deliberately does not attempt to read protected or private accounts, bypass access controls, or strip watermarks — use it for content you own or have permission to republish.

D. Create a Telegram bot with BotFather

1. Open Telegram and message @BotFather.
2. Send /newbot.
3. Choose a display name (e.g. My Mirror Bot).
4. Choose a username ending in bot (e.g. my_mirror_bot).
5. BotFather replies with a token like 123456789:AAH.... That is TELEGRAM_BOT_TOKEN.

Treat the token like a password: anyone holding it controls the bot. If it leaks, send /revoke to BotFather.

E. Add the bot to your channel

The bot must be an administrator with permission to post.

1. Open your channel → Manage Channel → Administrators → Add Administrator.
2. Search for your bot by its @username and select it.
3. Enable Post Messages (the only permission this app needs). You may leave Edit/Delete/Pin off.
4. Save.

A bot that is merely a member of a channel cannot post; Telegram returns not enough rights to send photos to the chat.

F. Get your TELEGRAM_CHAT_ID

TELEGRAM_CHAT_ID accepts either form:

• Numeric id, e.g. -1001234567890 — works for public and private channels. Preferred.
• @channelusername — public channels only.

Easiest reliable method — post anything to the channel, then:

curl -s "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/getUpdates" | grep -o '"chat":{"id":[^,]*'

Or, if the channel is public:

curl -s "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/getChat?chat_id=@yourchannel"

The id in the response (a negative number beginning -100) is your chat id. npm run telegram:check also prints it.

G. Create a PostgreSQL database

Any PostgreSQL works. Two easy options:

Neon (https://neon.tech) — create a project and copy the pooled connection string.

Vercel Postgres — in your Vercel project: Storage → Create Database → Postgres. Vercel injects POSTGRES_URL; copy its value into DATABASE_URL (or add DATABASE_URL pointing at the same value).

Use the pooled endpoint. Serverless functions open many short-lived connections and will exhaust a direct connection limit.

H. Run migrations

npm run db:migrate

This creates three tables:

Table	Purpose
workspaces	The tenant a channel and its sources belong to. One row today, seeded from the environment.
sources	Accounts being watched. (workspace_id, platform, external_id) is UNIQUE, so the same account cannot be added twice.
processed_posts	One row per X post ever seen. (workspace_id, external_id) is UNIQUE — the guarantee against double-posting.
telegram_messages	Every Telegram message id produced, including each item of an album.
sync_state	The since_id cursor and last-run health per source.

If you change src/db/schema.ts, run npm run db:generate to create a new migration, then npm run db:migrate to apply it. Migrations are idempotent and safe to re-run.

I. Configure .env

Copy .env.example to .env and fill it in. Minimum viable configuration:

DATABASE_URL=postgresql://user:pass@host/db?sslmode=require

X_USER_ID=1234567890
X_USERNAME=someaccount
X_BEARER_TOKEN=AAAAAAAAAAAA...

TELEGRAM_BOT_TOKEN=123456789:AAH...
TELEGRAM_CHAT_ID=-1001234567890

CRON_SECRET=<at least 16 random characters>

DRY_RUN=true

Generate a secret with:

openssl rand -base64 32

Configuration is validated with Zod at startup. A missing or malformed variable fails immediately with a message naming the variable, rather than failing halfway through a publish.

.env is gitignored. Never commit real credentials.

J. Test the Telegram bot by hand

The built-in check runs four steps and tells you exactly which one fails:

npm run telegram:check

[1/4] Bot token is valid: @my_mirror_bot
[2/4] Chat resolved: My Channel (type: channel, id: -1001234567890)
[3/4] Bot status in chat: administrator
[4/4] Test message delivered. message_id=42
All checks passed.

To do the same by hand:

# 1. Is the token valid?
curl -s "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/getMe"

# 2. Can the bot see the channel?
curl -s "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/getChat?chat_id=$TELEGRAM_CHAT_ID"

# 3. Can it post text?
curl -s -X POST "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/sendMessage" \
  -d chat_id="$TELEGRAM_CHAT_ID" -d text="hello from content-arbitrary"

# 4. Can it post a photo?
curl -s -X POST "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/sendPhoto" \
  -d chat_id="$TELEGRAM_CHAT_ID" \
  -d photo="https://telegram.org/img/t_logo.png" -d caption="test"

Do this before enabling the cron. Almost every "it silently does nothing" report is a bot that is not an admin of the channel.

K. Test with DRY_RUN

DRY_RUN=true (the default) does everything except publish: it reads X, filters posts, extracts media, builds the caption and decides which Bot API method it would call.

npm run sync:local

DRY RUN:
X Post: 1750000000000000003
Media: 4 photos
Telegram method: sendMediaGroup
Caption: "Four photos from the shoot\n\nSource: https://x.com/someaccount/status/1750000000000000003"
Would publish: true

Posts previewed in a dry run are stored as pending and the cursor is not advanced, so the first real run publishes exactly what you previewed.

When the output looks right, set DRY_RUN=false and run it again for real.

L. Deploy to Vercel

npm i -g vercel
vercel            # link the project
vercel --prod     # deploy

Or push to GitHub and import the repository at https://vercel.com/new.

vercel.json already declares the schedule:

{
  "crons": [{ "path": "/api/cron/sync", "schedule": "*/15 * * * *" }]
}

After the first production deploy, confirm the job appears under Project → Settings → Cron Jobs. Cron jobs only run from production deployments.

Region. vercel.json also pins every function to fra1 (Frankfurt), next to the database (Neon, eu-central-1). Left to Vercel's default, iad1 in the US, each query crossed the Atlantic — about 90 ms, and the review queue makes eight or nine in a row — so the page took 2–3 seconds. Moving the database means moving this too.

M. Add environment variables in Vercel

Project → Settings → Environment Variables. Add every variable from your .env for the Production environment:

DATABASE_URL, X_BEARER_TOKEN, TELEGRAM_BOT_TOKEN, CRON_SECRET, plus any optional ones you use. On a fresh install, TELEGRAM_CHAT_ID (and TELEGRAM_ADMIN_CHAT_ID with approval on) set up the first channel; once that is in the database they can be removed.

Two things to get right:

• CRON_SECRET must be set on the Vercel project. Vercel reads it and automatically sends Authorization: Bearer $CRON_SECRET when it invokes your cron. It is not just your own check.
• Start with DRY_RUN=true in production too. Watch one real cron run in the logs, confirm it found what you expect, then set DRY_RUN=false and redeploy.

Changing environment variables requires a redeploy to take effect.

N. How Vercel Cron works here

• Vercel sends an HTTP GET to /api/cron/sync on your production deployment.
• The request carries Authorization: Bearer $CRON_SECRET, plus user-agent: vercel-cron/1.0 and an x-vercel-cron-schedule header.
• Expressions are standard 5-field cron (minute, hour, day-of-month, month, day-of-week), always in UTC. Names like MON or JAN are not supported, and you cannot set both day-of-month and day-of-week.
• Vercel does not retry a failed invocation. That is fine here: an unpublished post stays pending/failed in the database and the next run, 15 minutes later, picks it up.
• Delivery is best effort. A run can be missed, or occasionally delivered twice — which is exactly why the sync is idempotent (see Architecture decisions).

On Pro/Enterprise the job fires within the specified minute, so */15 * * * * means at :00, :15, :30 and :45. (Hobby spreads invocations across the hour.)

The route sets maxDuration = 800, Pro's maximum, as headroom; the run itself stops taking new posts after 560s. A run that overlaps the next one is safe: each tenant is locked while it syncs, and a tenant still busy is simply skipped by the newer run.

O. Trigger the cron endpoint manually

curl -s -H "Authorization: Bearer $CRON_SECRET" \
  "https://<your-project>.vercel.app/api/cron/sync" | jq

{
  "checked": 12,
  "newPosts": 2,
  "published": 2,
  "failed": 0,
  "skipped": 10,
  "dryRun": false,
  "durationMs": 3242,
  "runId": "282d68d5"
}

For convenience during setup, the secret may also be passed as a query parameter (?secret=…). Prefer the header where you can — URLs end up in logs.

The admin endpoint shows recent history and never returns secrets:

curl -s -H "Authorization: Bearer $ADMIN_SECRET" \
  "https://<your-project>.vercel.app/api/status" | jq

It reports the last successful sync, the last error, counts by status, and the ten most recently processed posts with their Telegram message ids. It is protected by ADMIN_SECRET, falling back to CRON_SECRET if you do not set one.

P. Debugging common failures

Logs are structured JSON, one object per line, each tagged with the runId of its sync. Filter a run in Vercel's log viewer by that id. Secrets are scrubbed before anything is written.

Symptom	Cause and fix
401 Unauthorized from the cron	CRON_SECRET missing in Vercel, or differs from what you sent. It must be ≥ 16 characters.
chat not found	Wrong TELEGRAM_CHAT_ID, or the bot was never added to the channel. Run npm run telegram:check.
not enough rights to send photos to the chat	The bot is a member but not an administrator, or lacks Post Messages.
Could not resolve @handle	Handle is misspelled, or the account is suspended/protected.
X API 401	Bad or revoked X_BEARER_TOKEN.
X API 403	Your access tier does not permit this endpoint, or the account is protected.
X API 429	Rate limited. The client honours x-rate-limit-reset and backs off; if it persists, lower X_FETCH_LIMIT or the frequency.
checked: 0 every run	The cursor has advanced past everything — normal when there is nothing new. Confirm with /api/status → lastSeenPostId.
Posts found, none published	Almost always DRY_RUN=true. Check dryRun in the response.
file is too big	Over Telegram's upload limit; the post is marked skipped with the reason. See §R.
Nothing runs at all	Cron jobs only run on production deployments — and on Hobby, only once per day.
Run seems stuck	Check /api/status for a processing row. A killed function's row is reclaimed automatically after 10 minutes.

To inspect a specific post's fate, look it up in processed_posts: status, error_message and retry_count explain what happened.

Q. Telegram 429 and retry_after

Telegram returns HTTP 429 with parameters.retry_after (seconds) when you exceed flood limits. The client obeys that value exactly rather than applying its own backoff:

{"event":"telegram.rate_limited","method":"sendVideo","retryAfterSeconds":2}
{"event":"retry.scheduled","label":"telegram:sendVideo","delayMs":2000,"honouredRetryAfter":true}

Guardrails:

• Waits at most 120 seconds for a single flood wait. Anything longer is abandoned and left for the next cron run instead of holding the function open.
• Paces sends ~1.1 s apart, matching the Bot FAQ's guidance of no more than one message per second in a single chat (and 20 messages per minute in a group).
• Other transient failures (5xx, network) use exponential backoff with jitter, up to MAX_RETRY_ATTEMPTS (default 5).

If you see sustained 429s, lower MAX_POSTS_PER_RUN. An album counts as one send, so a backlog of single posts is what actually pushes you into flood control.

R. When X does not return the media you expect

Why media is downloaded and re-uploaded, rather than handed to Telegram as a URL. Telegram can fetch a URL itself, but that path caps at 5 MB for photos and 20 MB for other files, and depends on Telegram's servers reaching the X CDN. Downloading and uploading the bytes as multipart/form-data raises the ceiling to 10 MB for photos and 50 MB for other files and removes that dependency, so it is the default (MEDIA_UPLOAD_MODE=multipart). Set MEDIA_UPLOAD_MODE=url if you prefer the cheaper path and your media is small.

Nothing is written to disk — the local filesystem does not survive between invocations. Assets are streamed with a running size check and the transfer is aborted as soon as it exceeds what Telegram would accept, so an oversized video is never pulled into memory.

Known situations and how they are handled:

Situation	Behaviour
Video has only an HLS (.m3u8) variant	skipped, reason no progressive MP4 variant. Telegram cannot ingest a playlist.
Best MP4 rendition is over the preferred budget	The best rendition that meets the budget is sent; if none does, the original is sent anyway.
Every MP4 rendition is over the hard ceiling	skipped, naming the smallest size found.
Photo larger than 10 MB	skipped with the size.
Photo where width + height > 10000, or aspect ratio > 20	skipped before uploading — X gives us the dimensions, so no bandwidth is wasted.
Post has more than 10 media items	First 10 are published as an album (Telegram's limit); a warning is logged.
Media missing from includes.media	Logged; the post is skipped if nothing usable remains.
One item of a multi-photo post fails	The whole post fails and nothing is published.

That last rule is deliberate. A partially-published album cannot be repaired by a retry — the retry would duplicate whatever did get through. So every asset is downloaded and validated before the first Bot API call: a post either appears complete or does not appear at all, with the reason recorded.

Choosing a video rendition

X encodes every video at several bitrates (commonly three: roughly 320p, 480p and 720p/1080p) and lists them all in media.variants. Rather than always taking the largest and giving up when it is over the limit, the publisher picks the highest-bitrate MP4 that actually fits:

1. keep only video/mp4 variants — HLS playlists cannot be uploaded to Telegram;
2. sort by bitrate, highest first;
3. ask the CDN for each candidate's size (HEAD → Content-Length, falling back to a one-byte range request reading Content-Range, and to a bitrate × duration estimate if the CDN reports neither);
4. send the best rendition at or under PREFERRED_VIDEO_SIZE_MB, if one exists;
5. if none qualifies, send the best rendition within the hard ceiling min(MAX_VIDEO_SIZE_MB, Telegram's 50 MB) — a large original beats a dropped post;
6. only if nothing fits the hard ceiling is the post skipped, with the smallest size named.

Setting PREFERRED_VIDEO_SIZE_MB=10 therefore means "send a version under 10 MB when X offers one, otherwise post the original anyway". Leaving it unset always sends the best quality that fits.

Probing runs highest-first and stops at the first fit, so the common case costs a single HEAD request, and an oversized rendition is never downloaded. A video with a single variant skips probing entirely.

This is why the project needs no transcoding: X has already produced the smaller encodes, so FFmpeg would only duplicate work that a size-aware choice does for free.

supports_streaming plus width/height/duration are always passed so Telegram renders a seekable player.

---

Managing sources

Accounts to watch live in the database and are managed from the bot's private chat — no environment change and no redeploy:

Command	What it does
/review	Open the review queue: posts waiting for a decision, best Radar score first — see The review queue
/sourcestats	Open the Sources page: add any source — X account, Telegram channel, RSS feed, YouTube channel, website — see how each performs, pause, resume or remove it
/addsource @karpathy	Start watching an account
/addrss <feed URL>	Start watching an RSS or Atom feed — see RSS and Atom feeds
/addtelegram @channel	Start watching a public Telegram channel — see Telegram channels
/addyoutube @handle	Start watching a YouTube channel — see YouTube channels
/addwebsite <sitemap URL>	Start watching a website's new articles — see Website sitemap sources

/addsource accepts whatever is easiest to paste — karpathy, @karpathy, x.com/karpathy, a full profile URL, or even a link to one of the account's posts.

The Sources page (the button under /sourcestats, or /sources on the review website) adds any kind of source too: ➕ Add a source has a tab per kind — X account, Telegram channel, YouTube, RSS feed, Website — and a pasted x.com, t.me, youtube.com, sitemap or other link picks the tab itself; only a bare @name needs the tab chosen. Each is checked on the server exactly as its command checks it before it is stored, and the list below refreshes with it.

Pausing, resuming and removing a source are buttons under it on the /sourcestats page, next to its numbers: posts it brought in, how many were approved, why the rest were rejected, and roughly what reading it from X cost. A resumed source picks up only what is posted from then on; a removed one keeps its past posts. The old /pausesource, /resumesource and /removesource commands, and /sources that listed them, now just point there.

The bot's command menu — the list beside the message field — is BOT_COMMANDS in src/lib/telegram/commands.ts; register it with npm run telegram:commands after changing it.

Only a workspace's reviewer may run these, for the workspaces that name them; anyone else is ignored without a reply. The commands arrive over the same webhook as the Approve buttons, so npm run webhook:set must have been run once (see below).

Per-source settings

With APP_BASE_URL set, the replies to /sourcestats and /addsource carry a ⚙️ Settings button. It opens a Mini App listing your sources, each with its own switches, saved as soon as they are flipped:

Setting	Default	What it does
Active	on	Off pauses the source, like ⏸ Pause on the /sourcestats page.
Posts without media	off	Also mirror the account's text-only posts, as text messages (sendMessage, up to 4096 characters, with the same prefix, source link and suffix as a caption).
Radar threshold	none	A score from 1 to 100: a post Radar scores under it is passed over before review, never reaching the chat or the queue — see Shadow Radar.

A new source mirrors media posts only, as every source always has. Turning Posts without media on applies to posts that arrive from then on — ones the cursor has already passed are not fetched again. A post whose media exists but cannot be sent (an HLS-only video, say) is still skipped rather than published as bare text, which would misrepresent it.

Text-only posts go through review exactly like the rest: the preview is the message itself, Edit allows up to 4096 characters, and Approve sends it to the channel. X already returns these posts in the timeline we pay for, so mirroring them costs nothing extra on the X side.

How sources are identified

The numeric X user id is the identity, not the handle. Handles get renamed and reused, so matching on one would eventually follow the wrong account; the id never changes. A rename is noticed on the next sync and the cached handle is updated in passing.

Each source keeps its own cursor (sync_state, keyed x:<userId>), so accounts never interfere with one another: adding a busy account cannot starve a quiet one, and one unreachable account does not stop the rest of the run — its error is reported per source while the others carry on. MAX_POSTS_PER_RUN applies per source for the same reason.

A new source starts from the moment it is added, and a paused one from the moment it is resumed (sources.following_since): whatever the account posted before that is read once, passed over, and the cursor moves past it — only posts published afterwards come through. Removing a source deliberately leaves its cursor behind, so re-adding the same account later reads only from where it stopped instead of re-reading — and re-paying for — the whole window; what it posted while removed is passed over the same way.

Upgrading from the single-account version

Nothing to do. On the first run with an empty source list, X_USER_ID / X_USERNAME are imported into the sources table and everything continues as before, resuming from the cursor the single-account version left behind rather than re-reading (and re-paying for) the window.

The import happens once, recorded as workspaces.legacy_source_imported_at, so a source you deliberately remove does not reappear on the next run. Keep the two variables until you have seen the source appear in /sourcestats; after that they are ignored and can be deleted. New installations should leave them blank and use /addsource.

RSS and Atom feeds

A source can be an RSS 2.0 or Atom 1.0 feed as well as an X account. Every new entry becomes an ordinary candidate: Radar scores it, the translator rewrites it, it comes to review with the same buttons, and once approved it is published like any other post.

Command	What it does
/addrss https://example.com/feed.xml	Start watching a feed
/addrss 2 https://example.com/feed.xml	The same, for a reviewer of several channels: 2 is the channel's workspace id

Or ➕ Add a source → RSS feed on the Sources page, which asks for the URL and, with several channels, which one. A reviewer of several channels who leaves the id out of /addrss is shown their channels' ids — a feed URL is too long for a channel button, and the command keeps no state between messages.

The URL must be the feed itself, not the site's home page — there is no feed discovery. It is fetched and parsed once before it is stored, so a page that is not a feed is turned away. Try one first without storing anything:

npm run rss:check -- https://www.esa.int/rssfeed/Our_Activities/Space_Science

What a candidate is. The entry's title and summary — or its content, when the summary says too little — as plain text, capped at 6000 characters. The article itself is never fetched: no scraping, no images, no enclosures. RSS candidates are text posts, whatever the source's Posts without media switch (Settings shows it only for X and Telegram channels). Under the text, the published post carries the article's link, and Telegram's preview card of it is the post's picture. The review message names the feed — Source: ESA Space Science — and links the article; an entry with no link says so.

From now on, not the backlog. The first sync after a feed is added — or resumed from a pause — records everything already in it as backlog: no review, no Radar, no message. Only entries that appear after that come to review. Each entry has a stable identity — its Atom <id>, else its RSS <guid>, else its link, else a hash of its title, date and text — so a feed that reorders itself, lists an entry twice or edits a title never offers it again, and neither do overlapping runs.

Failures. A feed that times out (12 s), answers 4xx/5xx, is larger than 5 MB or is not well-formed XML fails on its own: the error is recorded for that source and the run carries on with every other feed and account. A feed that is gone (404/410) is not removed by itself.

Safety. The server fetches what a reviewer types, so only http(s) URLs without credentials are accepted, and neither the host nor any redirect may be — or resolve to — a local, private or link-local address (no localhost, no 169.254.169.254). XML entities are not expanded.

In stats and settings. Feeds are listed by title with an RSS badge and their URL. Reading a feed costs nothing, so /sourcestats shows no X read cost for one, and leaves its backlog out of the counts.

Under the hood. processed_posts still has X's column names, and a feed entry borrows them: external_id holds rss:<feed hash>:<entry hash> (clear of X's numeric ids and of other feeds), x_post_url the article, x_author_username the feed's title, x_created_at the entry's date. TODO: when the next platform connector lands (Reddit, YouTube…), rename these to external_item_id, source_item_url, source_author and source_published_at, and move X's metrics to platform metadata. Likewise sources.username holds a feed's title and should become display_name.

Telegram channels

A source can also be a public Telegram channel — one with an @username. Its posts become ordinary candidates, exactly like an X account's: photos, videos and albums are mirrored, Radar scores them, the translator rewrites them, they come to review, and approved ones are published.

Command	What it does
/addtelegram @spacenews	Start watching a channel; t.me/spacenews or a link to one of its posts works too
/addtelegram 2 @spacenews	The same, for a reviewer of several channels: 2 is the channel's workspace id

Or ➕ Add a source → Telegram channel on the Sources page (a pasted t.me link picks that tab itself). The channel's page is read once before it is stored, so a name that is not a public channel is turned away.

How it is read. From the channel's public web page, https://t.me/s/<channel> — the page anyone gets in a browser. No Telegram account, no API key, and the bot does not need to be in the channel. That page is not an API and its markup is not promised; the parser (src/lib/telegram-channel/parser.ts) relies only on the widget's long-standing class names and drops a post it cannot read whole rather than half-read it. If t.me changes the page, sources fail with an error in their summary until the parser is updated.

What it cannot read. Private channels (invite links t.me/+… are refused), groups, people and bots have no such page and cannot be added. Polls, files, stickers, voice and video messages, and videos too large for the web player ("Media is too big") are not shown on the page: a post whose only media is one of those is skipped, never sent as text alone — the same rule as an X video that cannot be sent. Photos are the size the page shows (up to 800 px wide), not the original upload.

What comes through. Posts with photos or videos; text posts too when the source's Posts without media switch is on. Forwards from other chats are skipped, like reposts on X (INCLUDE_REPOSTS). The post's link, t.me/<channel>/<id>, is its source link. Views are kept as the post's impressions metric.

From now on, not the backlog. A channel is followed from when it was added or resumed, by date, as an X account is. Each source keeps a cursor — the last message id it got through — and each run reads the channel's newest page and then the pages before it, up to 5 (about 100 posts), until it reaches the cursor. The cursor never passes a post that failed.

Media in the review queue. The links the page gives to media carry a token, and a post can wait in the queue for days. So when a queued channel post is approved, its media are looked up again on the post's own page (t.me/<channel>/<id>?embed=1) and fetched from there. A post the channel deleted meanwhile fails to publish and stays in the queue, as a deleted X post does.

Safety. Only t.me is ever fetched for pages, with a username already checked to be one, and redirects are not followed. Media are fetched only from Telegram's own content hosts, whatever a page says.

Under the hood. sources.external_id holds the lower-cased username — the only identity the public page offers, so a renamed channel is a new source. A post's external_id is tg:<channel>/<message id>, x_post_url its t.me link, x_author_username the channel's username. In stats and settings a channel is @username with a Telegram badge and its link; reading it costs nothing.

YouTube channels

A source can also be a public YouTube channel. Its new videos become ordinary candidates: Radar scores them, they are embedded and clustered into stories with posts from every other source, they come to the review queue and the digest, and approved ones are published.

Command	What it does
/addyoutube @pbsspacetime	Start watching a channel; youtube.com/@pbsspacetime or youtube.com/channel/UC… works too
/addyoutube 2 @pbsspacetime	The same, for a reviewer of several channels: 2 is the channel's workspace id

Or ➕ Add a source → YouTube on the Sources page (a pasted youtube.com link picks that tab). A video's link — youtube.com/watch?v=…, youtu.be/…, a Short — is not a channel and is turned away: this follows channels, not single videos.

Discovery and understanding, not mirroring. A video is never downloaded. What comes to review is its title and the opening of its description — that is the post, translated and published like any other — with its link, which Telegram previews with YouTube's own player. Its transcript, when it has one, is for analysis only: Radar reads it and it shapes the video's embedding, so a clickbait title does not decide what the video is about; it is never translated, reviewed or published.

How it is read. Through the official YouTube Data API with YOUTUBE_API_KEY, server-side only. A channel is looked up once when it is added (channels.list, by handle or id — never search.list) and stored by its channel id, which never changes; its title is how it is shown, and its handle, which can change, is not relied on. Each run reads its uploads playlist (UU…, from the channel id, so no channels.list per run) newest first — one playlistItems.list page of 15 — and the new videos' details in one batched videos.list: duration, views, likes, comments, live state. A premiere not yet out is left until it is; a live stream is taken as YouTube publishes it, usually without a transcript.

From now on, not the backlog. As for a feed, a video is new by identity: published since the source was added or resumed, and not already in review or settled. A run reads back page by page (at most 3) until it meets one that is not, and hands the new ones on oldest first. The atomic claim keeps a video from being taken twice, whatever runs overlap.

Transcripts — optional, cost-bounded. YouTube's own captions API gives a video's captions only to whoever may edit it, so transcripts come from a provider behind VideoTranscriptProvider (src/lib/youtube/transcript.ts); the first is Supadata, with SUPADATA_API_KEY. It is asked only for the transcript a video already has (mode=native) — never an AI transcription, whose price grows with the video — once per video. No transcript is a settled answer, recorded as unavailable; a provider outage is failed; no key is not_configured. In every case the video goes on with its metadata, and nothing stops. The whole transcript is kept (processed_posts.analysis_text, up to 200,000 characters); Radar reads a deterministic sample of at most 6,000 characters — the whole of a short one, six excerpts spread across a long one, so an hour-long video is not judged by its intro — and the embedding 2,000. No model summarises it first.

Under the hood. A video's external_id is yt:<video id>, x_post_url its watch link, x_author_username the channel's title; what YouTube said of it beyond its text is processed_posts.source_metadata (platform-neutral: duration, counts, live state, the transcript's state and language). Radar names the source PBS Space Time (YouTube) and its media YouTube video, 15 min; the transcript is a delimited <transcript> block after the post, material like everything else, never instructions. A post with no transcript — every post from X, a feed or a Telegram channel — has a prompt and an embedding word for word as before. In stats and settings a channel is its title with a YouTube badge and its link. Each API call is logged as youtube.api_call with its quota cost (one unit), and runs as youtube.sync_started, .videos_fetched, .video_discovered, .transcript_available / _unavailable / _failed, .sync_completed and .sync_failed.

Quota. The API allows 10,000 units a day. A channel costs one unit a run (96 a day at the 15-minute cron) and one more for a run that finds new videos — about 100 a day, so a few dozen channels fit. Supadata counts one request per new video.

Website sitemap sources

A source can also be a website, followed through its sitemap. The sitemap is the cheap way to learn what is new; only a new page is fetched, and only its own HTML:

sitemap (urlset or index, .xml or .xml.gz)
   ↓  new page addresses — identity, not dates
fetch only the new pages
   ↓
Readability extraction (no scripts, no browser, no model)
   ↓  NormalizedPost: title + lead as text, the whole article as analysis text
Radar → translation → stories → review → publish — as for every other source

Command	What it does
/addwebsite https://example.com/sitemap.xml	Start watching a site; a sitemap index (sitemap_index.xml) or a gzip one (.xml.gz) works too
/addwebsite 2 https://example.com/news-sitemap.xml	The same, for a reviewer of several channels: 2 is the channel's workspace id

Or ➕ Add a source → Website on the Sources page (a pasted URL with sitemap in its file name picks that tab). The sitemap's URL is given, never searched for: there is no discovery from the home page or robots.txt, and no crawling of links. It is read once before it is stored — an index with its sitemaps — so an address that is not a <urlset> or <sitemapindex> is turned away; the reply says what was found (Sitemap contains 438 URLs, 12 sitemaps · 8,421 URLs), never that they are all articles. A sitemap of 10,000 pages or more gets a warning: a site's dedicated news or posts sitemap, when it has one, means fewer pages fetched that turn out not to be articles. The source is named by its host; the home page is not fetched for a prettier name. Try one first without storing anything:

npm run website:check -- https://blog.cloudflare.com/sitemap.xml
npm run website:check -- https://blog.cloudflare.com/sitemap.xml --extract 3

It prints the sitemap's type, host, child sitemaps, how many addresses it lists and the most recent; --extract N (at most 10) fetches the N most recent pages and shows what the article reader makes of each — the post's text and the article's length, or why it is not an article. By default it fetches no page.

From now on, not the backlog. The first sync after a site is added — or resumed — records every page its sitemaps list as backlog, in a few batched inserts: no page fetched, no Radar, no translation, no review. A sitemap of 5,000 pages costs a few sitemap downloads, not 5,000 page fetches. If a child sitemap cannot be read on that first run, nothing is recorded and the whole is tried again next run, so its pages are not later taken for new. After that, every run reads the sitemaps again, and a page is new by identity — its address — never by <lastmod>: a known page whose <lastmod> changes is not fetched again (article updates may come later). A page the sitemap dates more than a day before the source was followed, or whose article says it was published then, is backlog too.

Identity. web:<sha256 of the address>, normalised conservatively: scheme and host lower-cased and a default port dropped by the URL parser, the fragment removed; the path and the query kept as they are — some sites tell pages apart by the query alone, and no tracking parameters are stripped. The same address in two sitemaps, or under two sources of one channel, is one candidate, fetched once. Identity is fixed when the page is found, whatever canonical address it names later.

Sitemaps, bounded. Namespaces are handled; entities are not processed at all — no DTD, no external entity, nothing loaded from outside the document. An index is followed breadth first, at most MAX_SITEMAP_DEPTH = 3 levels (the sitemap given, its sitemaps, theirs), each sitemap once (a cycle A → B → A stops), newest <lastmod> first, and only on the same site as the sitemap given — as are the pages, so a sitemap cannot point the server anywhere else. A sync reads at most 25 sitemap files (WEBSITE_MAX_SITEMAPS_PER_SYNC) and 50,000 addresses (WEBSITE_MAX_URLS_PER_SYNC); a download may be at most 10 MB, and at most 20 MB once decompressed — gzip is inflated with Node's zlib, which stops at that limit itself, so a small file cannot inflate without end. Child sitemaps are re-read every run: XML is cheap next to a page, an article or a Radar call, and a cursor would be one more thing to get wrong.

Pages, bounded. Only new pages are fetched, deduplicated before any request, oldest first by the sitemap's dates (a news sitemap's <news:publication_date>, else <lastmod>; pages without one after, in sitemap order). A run fetches at most 15 pages per site (WEBSITE_MAX_PAGE_FETCHES_PER_SYNC) and hands on at most MAX_POSTS_PER_RUN articles, inside the run's time budget; what is left is not marked known and is taken on a later run. A page must be HTML, at most 3 MB, read within 12 s.

Is it an article? No model decides. The page is parsed by jsdom with scripts off and nothing loaded, and Mozilla's Readability (Firefox's Reader View) finds its article, leaving navigation, sidebars and footers behind. It counts as an article when Readability finds one with a real title and at least 400 characters of text (200 for a page whose JSON-LD or og:type declares an Article, NewsArticle or BlogPosting), the page does not declare itself a listing (CollectionPage, ItemList, ProfilePage, Product…) or read as one — several headings, most of them links — and it is not the site's home page. A page that is not an article is recorded as skipped, with why, and never fetched again; so is one that is gone (404/410) or is not HTML. A page that only a browser running its scripts would fill has too little text and is passed over the same way: JavaScript-only sites are not supported.

What a candidate is. text — what is translated, reviewed and published — is the article's title and its lead: the page's description, or failing that its opening paragraphs, at most 1,500 characters. analysisText is the whole article as plain text, at most 60,000 characters: Radar reads a sample of it (an <article_text> block, after the post) and it shapes the embedding, exactly as a video's transcript does — so stories cluster an article with the X post, the Telegram post and the video about the same event. The post links the page's canonical address when it names a safe one on the same site (else the page's own), and Telegram previews it. Its date is the article's own (JSON-LD datePublished, then article:published_time), then the news sitemap's, then <lastmod>, else none — none is guessed. It is named by the page's og:site_name, else the host. No images in this version: the article's pictures are not downloaded, sent or understood; only its og:image address is noted in source_metadata.website, with the sitemap it came from, the original address when the link differs, its author, language and length.

Failures. A site whose sitemap cannot be read fails on its own: the error is recorded for that source, and the run carries on with every other site, feed, channel and account. A page that times out, answers 5xx or 429, or refuses us (401/403 — often an anti-bot wall) is recorded as failed and tried again on a later run, up to MAX_RETRY_ATTEMPTS times, then skipped for good. Nothing is done to get past a block, a paywall or a login: only what the server returns to anyone is read.

Safety. Sitemaps and pages are untrusted input. Only http(s) addresses without credentials are fetched, and neither the host nor any redirect hop (at most 5) may be — or resolve to — a local, private or link-local address (the same checks as a feed). No cookies, no credentials, no scripts, no page resources; no HTML is kept — only plain text. Logs carry hosts, counts, ids and a short hash of a page's path, never a query string or an article's text.

Cost. HTTP bandwidth and the Radar calls every candidate gets anyway. Ingestion makes no model call of its own: no summarising, no extraction, no classification.

Under the hood. sources.external_id is the sitemap's normalised URL, username the host; sync_state is keyed website:<hash of the URL>. A page's row borrows X's columns like a feed's: x_post_url its link, x_author_username the site's name, x_created_at its date. In stats and settings a site is its host with a Web badge and its sitemap's URL; its backlog and the pages that were not articles are left out of the counts. Events: website.sitemap_fetch_started / _done, .sitemaps_found, .urls_found, .initialized, .new_urls_found, .article_fetch_started, .article_extracted, .article_not_readable, .article_failed, .source_failed.

Multiple channels

One installation can serve several channels. A workspace is one of them: a destination channel, the reviewer who approves for it, and its own list of sources. Everything else — the bot token, the X application, and the tuning in Configuration reference — is shared by all of them.

Because every workspace uses the same X application, two workspaces watching the same account each keep their own cursor and are therefore billed separately for reading it. One account mirrored into three channels costs three times one channel.

Adding a workspace

There is no command for this: creating a tenant is an operator action, not something a reviewer should be able to do from a chat. Insert the row directly.

INSERT INTO workspaces (name, telegram_chat_id, telegram_admin_chat_id)
VALUES ('second channel', '-1001234567890', '123456789');

• telegram_chat_id — the channel, found exactly as in F. Add the bot to it as an administrator with Post messages first.
• telegram_admin_chat_id — the reviewer's numeric Telegram user id: where the workspace's notifications and review messages go, and who is let in. They can run /addsource and press Approve for the workspaces that name them, and the first time they open a Mini App or sign in to the review website they are made a member of each of those workspaces. Note that a channel id is negative and begins -100, while a user id is positive.
• name — shown to a reviewer of more than one channel, so give it the channel's name.

More reviewers for one channel

A channel can have any number of reviewers. The one named by telegram_admin_chat_id is the owner; anyone added with the script is a reviewer, and each one may do everything the owner may: press the buttons, run the bot commands, and use the review website.

npm run reviewer -- list --workspace 2
npm run reviewer -- add --workspace 2 --telegram-id 123456789 --name "Olena"
npm run reviewer -- remove --workspace 2 --telegram-id 123456789

Every review goes to every reviewer's private chat: each gets its own copy, with the same buttons. A decision — approve, reject, schedule, an edited caption — applies to all copies at once, so nobody can act on a post that another reviewer has already settled. A reviewer who has never sent the bot /start gets no copy, and that does not stop the others. A digest of the review queue goes to each reviewer the same way.

To take someone off a channel, remove their membership with the script. That is all it takes for a reviewer who was added. The owner has both: telegram_admin_chat_id and a membership, so to take the owner off, change the column and remove the membership.

One reviewer for several channels

The same person may be the reviewer of any number of workspaces — use the same telegram_admin_chat_id in each row. Then:

• every button and Mini App acts on the post's own channel, whichever it is: the post decides the workspace, and the reviewer only has to be its reviewer;
• review messages start with 📢 <name>, so the two channels' posts can be told apart;
• /addsource asks which channel with a button per channel;
• /scheduled, the ⚙️ Settings page and the /sourcestats page cover every channel, grouped by name.

A reviewer of a single channel sees none of this — no labels, no questions.

The reviewer must have sent the bot a message at least once before the first post is held for them: Telegram does not allow a bot to open a conversation, so review would otherwise fail with 403 bot can't initiate conversation with a user. Their /start also reveals their user id — it is logged as the fromId of webhook.unauthorized_command.

legacy_source_imported_at needs no value. The environment account is imported for workspace 1 only, and the guard for that is the workspace id check in syncPosts, not this column.

The reviewer then adds sources from their own chat with the bot:

/addsource @someaccount

Nothing else is needed; the next cron run picks the workspace up. A workspace with no telegram_chat_id is skipped as still being set up, and so is one with no reviewer while REQUIRE_APPROVAL=true — publishing unreviewed would defeat the point — unless it publishes without review on purpose. /api/status lists every workspace with a publishable flag and the reason when it is false.

TELEGRAM_CHAT_ID and TELEGRAM_ADMIN_CHAT_ID seed workspace 1 only, and only while its columns are still empty. Once a workspace row has a destination, the row wins: repoint a channel in the database and the environment will not overwrite it on the next run.

Pausing or removing a workspace

To park a workspace without losing anything, clear its destination. The sync skips it as still being set up, and its sources, cursors and history stay exactly as they are:

UPDATE workspaces SET telegram_chat_id = NULL WHERE id = 2;

Deleting the row is destructive. ON DELETE CASCADE takes its sources, its cursors and its entire publishing history (processed_posts, and the telegram_messages hanging off them) with it:

DELETE FROM workspaces WHERE id = 2;

Losing that history also loses the duplicate protection built on it: re-create the workspace with the same sources and, because the cursors are gone too, the next run reads the window from scratch and re-publishes posts the channel has already seen. Prefer clearing the destination unless you genuinely want the tenant forgotten.

Review website

The review queue, the sources and the settings also work as a website on its own subdomain — say https://app.zmistik.com — under one header with its sections (/queue, /scheduled, /sources, /settings; the old /stats redirects to /sources) and Sign out. The queue gets room to read: each post's pictures beside its text, the text and the schedule edited in place, and the keyboard to work through the list (j / k to move, a to approve, r then 1–6 to reject, e to edit, s to schedule, o for the original, Esc to step back). The Mini Apps stay as they are: each page is one component (src/components/) in two frames, so both use the same API, the same rules and the same screens.

The post editor

✏️ Edit text opens a full editor, in the Mini App and on the website alike: headings, bold, italic, underline, strikethrough, highlight, links, lists, quotations and dividers — and the post's own photos and videos, which can be dragged anywhere between paragraphs, each with a caption, or taken out. A post saved there goes out as one Telegram rich message (Bot API sendRichMessage, from 10.1) laid out as written, with the channel's footer at the end, instead of media with a caption — so its text is no longer held to a caption's 1,024 characters; a rich message takes up to 32,768 and 50 media. Back to the plain post drops the layout. A post never opened there is published exactly as before.

Only when the layout needs one, though. A post edited just for its words — its pictures left above the text, without captions of their own, and nothing but paragraphs, bold, italic, underline, strikethrough, links and quotations — goes out as an ordinary post: the pictures as a photo, a video or an album, the text as their caption, paragraphs an empty line apart as the editor shows them (a rich message would set them flush, and copying it would carry the rich formatting along). Over a caption's 1,024 characters the pictures carry the start of the text and the whole of it follows, as for a long X post. Headings, lists, dividers, highlighting, a picture placed in the text or captioned, more than 10 pictures or more than 4,096 characters make it a rich message; the editor says which it will be beside the character count.

The editor's document is stored as it is (processed_posts.rich_doc) and turned into Telegram's rich HTML on the server when the post is published: only the nodes and marks above are accepted, every bit of text is escaped, a link must be http(s), and a picture must be one of the post's own or one of its uploads (src/lib/rich/document.ts). Media go out the same way as for any approval: Telegram's own copies for a post reviewed in the chat, fetched again from X for a queued one. The Mini App keeps its plain caption editor, and refuses to edit a post laid out on the website, where its layout is what will be published.

Uploading photos and videos. With a public Vercel Blob store connected to the project (Storage → Create → Blob → Connect to project, which sets BLOB_STORE_ID and BLOB_WEBHOOK_PUBLIC_KEY; the app reaches the store as the project, by OIDC — locally, vercel env pull brings a VERCEL_OIDC_TOKEN for a few hours — and an older store's BLOB_READ_WRITE_TOKEN works instead), the editor has ⬆ Upload photo or video: JPEG, PNG and WebP photos up to Telegram's 10 MB, and MP4 videos up to MAX_VIDEO_SIZE_MB. An image copied from another site and pasted into the text (⌘V / Ctrl+V) is uploaded the same way and placed at the cursor, and a file dragged in from the computer where it is dropped; pasted text is pasted as before. The browser sends the file straight to the store — past the 4.5 MB a Vercel function may be sent — with a presigned URL this app gives only to the post's reviewer, only while the post waits, only for a path under that post (posts/<id>/…) and for ten minutes (/api/telegram/webapp/uploads). It then registers the file (/api/telegram/webapp/uploads/register), and the app checks it in the store itself — path, type and size — before recording it in post_uploads; a file of the wrong type or size is deleted there and then. The editor places it as u<id>, beside the post's own media, and the AI assistant sees and keeps it. When the post is published its files are fetched from the store like any other media and sent with it, and then deleted from the store, placed or not; the uploads of a post turned down, or failed for good, are deleted by a sweep after each sync. The store holds a file only while its post waits. Without a store there is no upload button.

The AI assistant

✨ AI assistant opens a chat beside the post (under it on a narrow screen). Ask it to change the post — "make it shorter", "add a headline", "fix the grammar", "translate into Ukrainian" — and the change lands in the editor at once, with Undo (and Redo) under the reply; ask it to check the facts and it searches the web and answers with the pages it cites. Nothing is saved: the editor's own Save does that, as for any change.

It knows the channel's editorial profile, its voice and two of its posts (from the publication-history profile), the post's source and pictures, and the post as it reads in the editor right now — the unsaved draft, sent with each message. The post travels as a small HTML of exactly what a rich message holds (src/lib/rich/edit-html.ts); the model rewrites it through one function, edit_post, and its version is checked on the server exactly as a save is checked before the editor ever sees it. One that does not pass — a picture that is not the post's, a tag the channel cannot show — is sent back once with the reason, then reported. The conversation stays in the editor while it is open; the latest twelve messages go back with each new one.

Cost. OpenAI only for now (ASSISTANT_MODEL, GPT-6 Luna by default), through the Responses API: little reasoning, web search with little context and only when the request needs it, a bounded answer, and nothing stored on OpenAI's side. A rewrite costs a few hundredths of a cent; a fact check about twice that in tokens, plus OpenAI's own fee per web search. Every request is recorded in assistant_requests — tokens, searches, cost estimate, whether it changed the post — and a reviewer gets ASSISTANT_DAILY_LIMIT requests (100) in 24 hours; 0 turns the assistant off, and so does a missing OPENAI_API_KEY.

Where the bot's 📋 Open review queue buttons — the queue notification and /review — take you is yours to choose under Settings: the Mini App, inside Telegram (the default), or the website, in the browser (users.review_link).

Signing in

Log in with Telegram, through Telegram's OpenID Connect provider: the Authorization Code flow with PKCE, the code exchanged server side with the client secret, and the ID token's signature (from Telegram's published keys), issuer, audience, expiry and nonce checked before anything in it is believed. A sign-in is good for one callback and ten minutes, and only in the browser that started it. The person is known by the Telegram user id in the token — the same id a Mini App's signed data carries and telegram_admin_chat_id names — so the website and the Mini Apps find the same user.

The browser then holds a random session token in an __Host- cookie — httpOnly, HTTPS only, this host only, SameSite=Lax. Only its SHA-256 is stored (user_sessions), so the table signs nobody in; a session lasts 30 days, and Sign out deletes it. Anything that changes a post must also come from the website's own origin, so another site cannot make a signed-in browser act.

People, not Telegram accounts, are what the database knows:

Table	Holds
users	A person, whatever they sign in with.
user_identities	How they sign in: telegram + their Telegram user id. Another provider — Google, or Auth0 in front of anything — is another row, and another OpenID Connect provider in code.
workspace_members	Which workspaces they may review — the whole authorisation rule, for the website and the Mini Apps alike.
user_sessions, login_attempts	Website sessions, and sign-ins under way.

The migration makes every reviewer named by a workspace a user and a member, so nobody loses access. A workspace's reviewer is also made a member on their next sign-in, so setting telegram_admin_chat_id keeps granting access as it always did. Taking access away is both: change telegram_admin_chat_id, and delete the workspace_members row. For any other reviewer, deleting the membership is enough: npm run reviewer -- remove.

Setting it up

1. Vercel → Settings → Domains: add the subdomain to this project, and create the DNS record Vercel shows for it.
2. @BotFather → open its mini app → your bot → Login Widget: add the Allowed URLs https://app.zmistik.com and https://app.zmistik.com/api/auth/telegram/callback, and copy the Client ID and Client Secret it shows.
3. Vercel → Environment Variables: WEB_APP_URL=https://app.zmistik.com, TELEGRAM_OIDC_CLIENT_ID, TELEGRAM_OIDC_CLIENT_SECRET. Redeploy — WEB_APP_URL is also read at build time, to route the subdomain.

APP_BASE_URL stays on another host: the Mini Apps live at its paths, and on the website's host those same paths are the website's. In development the website answers on http://app.localhost:3000.

Approval before publishing

With REQUIRE_APPROVAL=true nothing reaches the channel unattended. New posts wait in the review queue — or, if you choose, each is delivered to your private chat with the bot, carrying ✅ Approve and 🚫 Reject buttons; the channel only sees a post once you approve it.

X ──► cron ──► your private chat  ──[Approve]──►  channel
                      │
                      └──[Reject]──► pick a reason ──►  recorded, never published

Publishing without review

Approval is switched on for the whole installation, but a channel that reposts everything its sources bring in has nothing to decide. Publish without review on the ⚙️ Settings page (workspaces.auto_publish) sends that channel's posts straight to it as they arrive, as if REQUIRE_APPROVAL were off for it alone: nothing goes to the review queue or the chat, the post keeps its Source: line (INCLUDE_SOURCE_LINK), and Radar, which scores posts on their way to review, does not run for it. Such a channel needs no reviewer. Turning it on leaves posts already waiting for review where they are.

Reject does not settle the post by itself: it swaps the buttons for a short list of reasons — Not interesting, Off-topic, Already covered, Too minor, Weak source, Other — and the post is rejected when you pick one. ↩️ Back returns to Approve / Reject if Reject was a misclick. The reason is stored in processed_posts.rejection_reason under a stable value (not_interesting, wrong_topic, already_covered, too_minor, weak_source, other); the button wording may change, those values do not.

With APP_BASE_URL set, ••• Other opens a small Mini App where you can say why in your own words; the post is rejected when you press Reject there, and the text is kept in rejection_note. Closing it without pressing Reject changes nothing. Without APP_BASE_URL, Other rejects straight away like the other reasons.

What is recorded for every post, for later analysis:

Column	Holds
source_text	The author's whole text as X gave it — the full text of a long-form post, links resolved, plain text, before any prefix, suffix, source link or truncation. Written when the post is first seen, whatever happens to it next.
original_caption / caption	The caption first sent for review, and the one published (they differ only after an edit). For a post published with no review, both hold what went out.
caption_edited_at	When the caption was last saved in the editor.
reviewed_at	When you pressed Approve or chose a reject reason; empty for posts published with no review.
rejection_reason / rejection_note	Why it was rejected, and your own words for Other.
review_media	The media you were shown — Telegram's file id per item, with X's photo URL or video still. Kept after the decision (unlike approval_payload), so Radar's backfill can show a model the same picture.
x_like_count, x_repost_count, x_reply_count, x_quote_count, x_bookmark_count, x_impression_count / x_metrics_at	The post's public engagement as X reported it when the post was first fetched, and when that was. A snapshot, never refreshed: compare posts by their age at that moment (x_metrics_at - x_created_at), since a sync sees most posts within the hour. Comes with the same read, so it costs nothing extra.

The review queue

With APP_BASE_URL set, posts are not sent to the chat one by one. Each is checked, scored by Radar and translated exactly as before, then kept in the review queue — a Mini App page with every post awaiting a decision, best Radar score first. Instead of the posts, the chat gets a digest of what came in — see The queue digest — with a 📋 Open review queue button.

X ──► cron ──► review queue ──(at most once an hour)──► "📥 7 new" ──► queue page ──[Approve]──► channel

• How often is per channel, under ⚙️ Settings: every 15 minutes, every hour (the default), every 3, 6 or 12 hours, once a day, every few days, once a week — or each post in the chat, as it arrives, which is the per-post flow described below. For a day or more you can also pick the hour it comes at — in your own time zone, as your browser gives it, and kept through the clocks changing (review_digest_hour, review_digest_timezone). The cron runs every 15 minutes, so it arrives within a quarter of an hour of it. A notification goes out only when a post has joined the queue since the last one, so a quiet stretch sends nothing. Earlier digests stay in the chat: each lists its own stories.
• The page shows each post's picture (X's own photo, or a video's still), its text as it will be published, Radar's score and one-line why. Radar only orders the list: every post is on it, whatever it scored, and the decision is yours — unless its source has a threshold, which keeps posts scored under it out.
• ✅ Approve publishes there and then. 🚫 Reject asks why, with the same reasons as the chat, and ••• Other takes your own words. ✏️ Edit text and 🕒 Schedule open their pages and come back to the queue.
• One card per story. Waiting posts of the same story — the same event from several sources, or a source repeating itself — share one card: the best-scored leads, the others are listed under it, and ⇅ Use this one puts any of them in its place. The card also says what of the story the channel has already published (📚 Already published from @esa · 12 Oct). A decision covers the whole card: Approve or Schedule publishes the post shown and rejects the rest as ♻️ Already covered; Reject turns them all down for the reason you give. Only the posts you were shown are decided — one that came in since stays.
• /review says how many posts are waiting and opens the page at any time, without waiting for the notification.

A queued post was never sent to Telegram, so there is no file_id to re-send: approving it fetches its media from X again — all of them before anything is sent, exactly as the first time — and publishes. A post whose media X no longer serves fails to publish and stays in the queue with the error. The chosen video rendition, its alternatives and X's URLs are kept in approval_payload.sourceMedia for this. The interval is workspaces.review_digest_minutes (null for the chat); processed_posts.review_queued_at says when a post joined the queue.

Posts that were sent to the chat before the switch keep their buttons there, and also appear in the queue; deciding one in either place settles it in both. Without APP_BASE_URL there is no page to open, so every channel gets its posts in the chat.

The queue digest

Each notification is an editorial digest of the period since the last one, in the channel's language — its frame in Ukrainian or English (English for a language it has no words in yet), Radar's reasons and the posts' text in the channel's own:

📡 Zmistik — за останні 6 годин

22 нові пости · 19 історій · 5 рекомендованих

1. 94% — Webb відстежив найвіддаленіший радіосплеск до його галактики
Рекордна новина від Webb — як ті, що ви публікуєте.
Джерела: @NASA, Space.com, @spacenews

2. 87% — …

[1] [2] [3] [4] [5]
[📋 Відкрити чергу]

• The counts: the posts that joined the queue in the period, the stories they make (after story clustering; a post in no story counts as its own), and the stories Radar recommends. Not the posts read at the sources: X's reposts and replies are dropped before anything is stored, so that number is not in the database, and the digest does not guess it.
• The stories: at most five, best first, each the story's best-scored post still waiting — its Radar score, the first line of its text as the channel will read it, Radar's own one-line reason, and every source that reported the story. Recommended is Radar's own "likely approve" (radar-v3, the score the review message shows); there is no threshold of the digest's own. With none recommended it says so, with the counts and the button.
• The buttons under it open the queue at each story's card (/queue?post=…), and the last one at the top.
• Never twice: when a digest goes out, the lead post of each story it showed is marked (processed_posts.digest_sent_at), and a story with a marked post is not recommended again — even when another source reports it later. The queue still shows every post.
• Nothing new is asked of a model: the digest reads Radar's stored scores and reasons, the stories and the sources, a few queries per channel. It is built in src/lib/review/digest.ts and sent by sendReviewDigest after each tenant's sync, on the interval set under ⚙️ Settings; a channel that has each post sent to the chat gets no digest, as before. Logged as review_digest.started, .sent, .skipped and .failed, with the period and the counts.

How it works

Inline buttons arrive over a webhook, which a cron-only app cannot otherwise receive, so this adds POST /api/telegram/webhook. Two independent checks guard it:

1. the X-Telegram-Bot-Api-Secret-Token header must match TELEGRAM_WEBHOOK_SECRET, proving the call came from Telegram;
2. the pressing user must be the reviewer of the post's workspace — forwarding the message to someone else does not hand them the publish button.

The review send is not wasted work. Telegram returns a file_id for every asset it stored, and re-sending by file_id needs no upload and no download. Approving therefore costs a single cheap API call and never touches the X CDN again — which also means approving hours later still works, long after the X media URLs have rotated.

A double tap is safe: the decision is taken by one conditional UPDATE, so ten simultaneous presses produce exactly one publish. If publishing fails, the post returns to the queue with the reason attached and Approve can simply be pressed again.

Albums cannot carry an inline keyboard, so for sendMediaGroup the buttons arrive on a short reply underneath the album.

Setting it up

1. Send /start to your bot in a private chat. Telegram forbids a bot from messaging a user who has never started it, so without this the review message cannot be delivered.
2. Find your numeric user id — do this before registering the webhook, since getUpdates stops working once a webhook is active:

   curl -s "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/getUpdates" | jq '.result[].message.from.id'
   

3. Generate the webhook secret (hex, not base64 — Telegram allows only A-Z a-z 0-9 _ -):

   openssl rand -hex 32
   

4. Set the variables in Vercel and redeploy: REQUIRE_APPROVAL=true and TELEGRAM_WEBHOOK_SECRET, plus TELEGRAM_ADMIN_CHAT_ID on a fresh install to seed workspace 1's reviewer. Enabling approval without the webhook secret fails validation at startup; a workspace with no reviewer is skipped by the sync with the reason, never published unreviewed.
5. Register the webhook once per deployment URL:

   npm run webhook:set -- https://your-project.vercel.app
   

   Check it any time with npm run webhook:info, remove it with npm run webhook:delete.

Without step 5 the buttons appear but nothing happens when pressed — Telegram has nowhere to deliver the callback.

Editing the post before publishing

With APP_BASE_URL set, the review message also carries ✏️ Edit text, which opens a Mini App — a small web page inside Telegram — holding the post editor, the website's own, AI assistant included. Save, then Approve as usual; Save and back to the chat closes the Mini App, and opened from the review queue it returns there.

X ──► cron ──► your private chat ──[Edit]──► Mini App ──save──► post's layout stored
                      │
                      └──[Approve]──►  channel, as the editor last saved it

What editing does and does not touch:

• what goes out is the saved layout — the words, and the post's own photos and videos placed, taken out or joined by uploads — published as an ordinary post when it can be and a rich message when it must (see The post editor);
• the preview in the chat is not rewritten to match: what Approve publishes is the editor's version, which the review queue shows;
• a post that has already been published or rejected cannot be edited. Approve and Edit race safely — whichever lands first wins, and the other is refused;
• the text first sent for review is kept in original_caption and never changes, however many times you edit, and Back to the plain post in the editor drops the layout to publish the post as it came in.

Setup is one variable:

APP_BASE_URL=https://your-app.vercel.app

It must be HTTPS, which Telegram requires for Mini Apps, and it must be the stable production domain rather than a per-deployment URL — the button is built when the post is queued and may be pressed days later. Leave it unset and review works exactly as before, without the button.

A Mini App page is openly reachable; that is normal, and it is not what guards anything. Telegram hands the page a signed initData string, every request carries it in an Authorization: tma … header, and the server does nothing until that signature verifies against the bot token, names a Telegram user who is some workspace's reviewer, and the post belongs to a workspace they review for. The post id in the URL is therefore not a credential: asking for another tenant's post returns exactly what asking for a nonexistent one returns. Signed data older than 24 hours is refused, so a captured initData string does not stay usable.

If Telegram ever refuses to open the button, set your domain under Bot Settings in BotFather and try again; the inline-keyboard button type documents only the HTTPS requirement.

Scheduling a post

With APP_BASE_URL set, a post in review also carries 🕒 Schedule. It opens a Mini App with a date and time picker — in your phone's time zone — and a few shortcuts (in an hour, tonight, tomorrow morning). Pick a time and the post is approved for then: the review message becomes "🕒 Scheduled for Mon 5 Oct, 18:00" with these buttons:

Button	What it does
⚡ Publish now	Publishes it immediately.
↩️ Unschedule	Back to review, undecided, with Approve / Reject / Schedule again.
🕒 Change time	The same picker, to move it.
✏️ Edit text	Still editable until it goes out; the latest text is what is published.

/scheduled lists what is waiting, soonest first, at the times you picked.

On the website, Scheduled (/scheduled) shows the same list by day — Today, Tomorrow, Fri 16 Oct — with each post's pictures, text and how long until it goes out. Each post has ✏️ Edit (the full editor, which comes back to this page), 🕒 Change time (a picker in your browser's zone; the list re-sorts), ↩️ Back to review and Original. The scheduler reads a post's text and time only when the time comes, so whatever was saved up to that minute is what goes out.

Telegram does not let bots use a channel's own scheduled messages, so these do not appear in the channel's Scheduled list. The queue lives in the database instead, and GET /api/cron/publish-scheduled — run every minute by Vercel Cron — publishes whatever is due, so a post goes out within a minute of its time. Most runs find nothing and cost one indexed query.

Publishing is the same path as Approve: the stored file_ids, the current caption, the full text follow-up of a long post. Each post is claimed with a conditional UPDATE before it is sent, so overlapping runs or a Publish now pressed in the same minute publish it exactly once. A failed publish stays scheduled and is tried again the next minute; after MAX_RETRY_ATTEMPTS failures it goes back to review with the error, and the review message says so and offers the buttons again — rather than retrying forever.

Times are stored as exact moments (scheduled_for, UTC) with the zone they were picked in (scheduled_timezone), used only to show them back to you. A post can be scheduled up to a year ahead. scheduled_for is kept after publishing, so the planned time can be compared with when it actually went out.

New states

Status	Meaning
awaiting_approval	In the review queue or sent to your chat, waiting for a decision. Not in the channel.
rejected	You declined it. Never published, never retried, never re-synced.
scheduled	Approved for a later time; published by the scheduler when it comes.

/api/cron/sync reports an awaitingApproval count alongside published, and /api/status shows both statuses in its counts and recent posts.

Translation

A channel that publishes in another language than its sources can have each post rewritten in it before review:

UPDATE workspaces SET language = 'uk' WHERE id = 2;   -- NULL turns it off

language is a BCP 47 code (uk, de, pt-BR). Each post's text is then rewritten by RADAR_PROVIDER's model (GPT-6 Luna by default) and becomes its caption: what the reviewer sees, can edit, and what approval publishes. Prefix, suffix and the source link are added as before, and the review message keeps its link to the original post, so the original is one tap away.

It is not word for word: the model writes it as the channel would — in the voice and format of the newest publication-history profile and of three of the channel's own representative posts, when there are — keeping every fact, number and name of the original and adding nothing; URLs stay as they are, the source is not credited in the text (the link does that), and a post already in the channel's language is left as it is.

Only the caption changes. processed_posts.source_text keeps the source's own text, which Shadow Radar, embeddings and the history searches go on reading — so Radar scores the original while the translation runs alongside it, and neither waits for the other.

Never in the way: a translation that fails, times out (25 s) or comes back empty leaves the post in its own language, and it goes on as it would have. A post that is skipped, and every post of a dry run, is not translated. Each translation is logged (translation.done) with its tokens; at Luna's list price a post of a few hundred characters costs about $0.0002–0.0003.

Post footer

A line the channel adds under every post — its name and a link, say — set once per workspace instead of typed on each post:

UPDATE workspaces
SET post_footer = '[ВЕКТОР | космос · футуризм · sci-fi](https://t.me/vector_space2035)'
WHERE id = 2;   -- NULL removes it

It is plain text with Markdown-style links, [text](https://…): http, https and tg links become links, everything else is shown as typed. At most 200 visible characters.

It goes last, under the text (and the source line, if any), on the review message and on the published post — and under a long post's full-text follow-up too. It is never cut: a long post is shortened to leave room for it. In the editor it is shown under the text but is not part of it, so editing cannot break its link, and the character counter allows for it.

A post already waiting for review keeps the footer it was sent with; a footer set or changed later applies to posts that arrive after.

Shadow Radar

An experiment: can a model predict which posts the editor will publish? Radar scores each post on its way to the reviewer and records the prediction. Nothing is filtered, reordered or delayed — except where the editor set a source a threshold — and a Radar failure leaves the post exactly as it would have been. Whether the scores are any good is measured against the decisions the editor makes.

On the review message. Under the link, the message with the buttons shows the score, the predicted decision, Radar's one-line reason, and a ♻️ flag when it thinks the channel may have covered this already — e.g. 📡 Radar v5: 62/100 · likely approve. It is the live score from radar-v5-similar-rejected on the text alone when there is one (the best match so far), falling back to radar-v2-history-retrieval and then radar-v1; a post Radar could not score shows no note. The review queue shows it as a badge named v5, green from 60 and amber from 30, and orders by it; the review notification's recommendations follow it too. radar-v3 is still scored live, so the report can go on comparing the two, but it is not shown and nothing is ordered by it: its scale runs higher, and a post scored by it alone would jump the queue. A hint only: every post still comes to review. Seeing it can sway a decision, so decisions made since it was shown (8 October 2026) are a little less independent a test of Radar than those before.

A source's threshold. In the source settings, a source may be given a Radar threshold from 1 to 100 (sources.radar_threshold; none by default). A post of that source scored under it — the same score the queue shows — is passed over: it is never sent to the chat or put in the queue, and is recorded as skipped, with error_message saying radar threshold: scored 23, under the source's 40. Not as rejected: no one turned it down, so it is never one of the decisions Radar is shown or measured against, and the source's approval rate leaves it out. The Sources page counts these posts as 📡 under threshold. A post Radar could not score — it failed, timed out, ran out of the run's budget — goes to review as always, and a threshold does nothing for a channel Radar does not score (no editorial profile, or published without review). It applies to posts that arrive after it is set.

What it sees. The tenant's editorial_profile (a few lines, in the editor's words, on what the channel publishes and what it turns down), the publication-history profile when one has been generated, the editor's 10 most recent approvals and 10 most recent rejections with their reasons, the share of posts they approve, and the post: its source, text and kind of media. A post's picture is not sent with every version: its first photo is understood once, and the media prompt reads that description. (The earlier experiment — scoring the text with its raw first photo too, the text_image variant — runs live only with RADAR_LIVE_IMAGE_VARIANT=true; backfills still score it.)

Prompts, side by side. Seven versions build on each other, so each addition is measured on the same posts, not on two different sets: radar-v1 (the baseline above); radar-v2-history-retrieval, which adds the channel's five most similar past publications and asks what they show; radar-v3-retrieval-approved, which also adds the five most similar posts the editor already approved; and radar-v4-media-understanding, which also reads what images show — the new post's, and the past publications', approved posts' and examples' — from their stored descriptions, never the image itself; and radar-v5-similar-rejected, which is radar-v3 (not radar-v4: image descriptions did not help) plus the five most similar posts the editor turned down; and radar-v6-nearest-decisions, which is radar-v3 plus the editor's ten nearest decisions, approved and rejected together; and radar-v7-story-sources, which is radar-v3 plus the post's story so far: the other sources that reported the same event before it, and what the editor had done with their posts. Live, every post is scored with radar-v1, radar-v3 and radar-v5 (since 13 October 2026, to test on fresh posts what a backfill suggested); the others run on backfills only — radar-v4 scored below radar-v3 live, radar-v6 did no better than radar-v5, and radar-v7 ranked below — and backfills score all seven by default. The review message still shows 's score. in is the live list.

What it records, in radar_evaluations: a 0–100 score (the probability the editor publishes it), the predicted decision and rejection reason, three sub-scores, a one-sentence reason — in the channel's language, so the review message and the digest read in it (Ukrainian for a channel with none set; a Ukrainian channel's prompt is word for word what it was before) — which past decisions, which editorial profile (as editorial_profile_hash, the first 16 hex characters of its SHA-256) and which publication-history profile it was shown, the model and prompt version, and the tokens it cost. The retrieval prompt's rows also record which past publications it was shown, with their similarity (history_retrieval), and what the model made of them (historical_assessment: relevant, possibly already covered, why). Failed and skipped attempts are rows too, so coverage is visible.

Limits. It runs inside the sync, so it has a 20-second timeout per call and two minutes per run; after three failures in a row it stops for that run. Posts it did not get to go to review unscored.

Which model. RADAR_PROVIDER picks it: openai (the default) asks GPT-6 Luna through the Responses API, with low reasoning effort, images at low detail, and nothing stored on OpenAI's side; anthropic asks Claude Haiku 4.5. Both get the same prompt and the same schema, every score records the model that gave it, and the report keeps models apart — so switching mid-way compares them rather than mixing them. Luna's list price is a tenth of Haiku's.

Turning it on

1. Add the chosen provider's key (OPENAI_API_KEY, or ANTHROPIC_API_KEY with RADAR_PROVIDER=anthropic) to your local .env for the scripts below — and to Vercel only if you want posts scored live.

2. Run npm run db:migrate.

3. Write the channel's profile on the settings page — 📡 Radar: what this channel publishes, in the Mini App and on the website alike, one box per channel. It is the one part of Radar's prompt that is the editor's to write: the instructions around it stay fixed per prompt version, so scores stay comparable. Up to 2000 characters; Save stores it, Cancel puts back what is saved, and Clear (after a confirmation) empties it, which turns Radar off for that channel. The post editor's AI assistant reads the same text.

   A new text applies from the next sync: posts already scored keep their scores, and each score records which text it was made with (editorial_profile_hash), so scores made before and after a rewrite can be told apart.

Scoring past decisions

npm run radar:backfill -- --workspace 2 [--prompt radar-v1,radar-v3-retrieval-approved]

Scores posts the editor has already decided, each with only the decisions made before it arrived — what a live Radar would have seen — so the result measures prediction, not hindsight. Since live scores change nothing during the experiment, this measures the same thing for less: requests go through the provider's batch API at half price, and nothing runs while nobody is looking.

It submits the requests, waits for the batch (usually minutes, at most 24 hours) and reads the results in. If it is interrupted while waiting, pick the batch up with --resume <batch id> (the id is printed) rather than running it again, or the same posts are paid for twice; --no-wait submits and stops. Re-running later scores only what is new and retries what failed; a score already recorded is never requested or changed again.

Posts that arrived before there were five approvals and five rejections behind them are left out (--min-per-class N). The picture comes from review_media: a photo from Telegram, falling back to X's URL, a video's still from X. Posts decided before that column existed have none and are scored on their text; --text-only skips that, --limit N stops after scoring N posts (the oldest first).

Every prompt version goes into the same batch by default; --prompt picks some. The retrieval prompts need OPENAI_API_KEY for embeddings and the history embedded (npm run history:embed) first. A post whose similarity search fails has its retrieval request held back and retried by the next run, so a retrieval score is never one made without retrieval by accident. Do not re-embed history while a batch is pending: what a result is recorded as having been shown is read again at ingest. The same goes for the editorial profile: its hash is recorded from the text as it is at ingest.

A resumed batch is read with the provider it was submitted to, whatever RADAR_PROVIDER says now.

To keep Radar to backfills only, leave the API keys out of Vercel and set them just in your local .env: without the chosen provider's key the sync never calls Radar.

Reading the results

npm run radar:report -- --workspace 2
npm run radar:report -- --workspace 2 --compare radar-v1,radar-v2-history-retrieval

Per mode (live, backfill) and variant: how often the editor approved, the mean score of approved and rejected posts, how well the scores separate them (AUC: 0.5 is chance, 1.0 perfect), the daily hit rate, precision at 80, recall at 50, approval rate by score band, and — for a few thresholds — how much review work hiding the posts below it would save and how many approved posts it would have hidden. A live score made after the editor had already decided is not counted. Text against text-and-image is compared on the same posts only.

For the retrieval prompt it adds how the search went (matches found, no history, no text, failed) and how often its "possibly already covered" flag agreed with the editor's already_covered rejections. --compare puts two prompt versions side by side on exactly the posts both scored, per mode, variant and model: AUC, the daily hit rate, mean scores, precision at 80, recall and work saved at each threshold, approvals scored under 50 — and which of those one version rescued and the other missed — and the cost.

The core measure: the daily hit rate

The editor reads the review queue about once a day, best score first. So what decides whether a change to Radar is kept is how well it orders each day's posts: for every day, take Radar's top N posts, N being the number the editor approved that day, and count how many of them are the ones they approved — "had you published Radar's top N, how many would have been yours". Shown as 13/41 32%, beside what a random order would get on the same days (chance), and next to it the share of approvals within the top quarter of each day: what reading only a quarter of the queue would have found. Days are arrival days in UTC; tied scores at the cut count at their expected share. With a few weeks of decisions the counts are small, which the 13/41 is there to show: a difference of two or three approvals is noise.

What the history profile is worth. Scores with and without the publication-history profile are never of the same posts: a backfilled post gets the profile only if it was made before the post arrived. So radar-v3-without-history-profile is radar-v3 with the profile left out and nothing else changed, for backfills only; it skips the posts that had no profile to begin with, where it would only repeat radar-v3. Score both on the same posts and compare:

npm run radar:backfill -- --workspace 2 --prompt radar-v3-retrieval-approved,radar-v3-without-history-profile --text-only
npm run radar:report -- --workspace 2 --compare radar-v3-retrieval-approved,radar-v3-without-history-profile

On VECTOR's first 351 decided posts after its profile was made (41 approved) the profile paid its way: with it AUC 0.71 against 0.67 without, 18 of the 41 approvals in each day's top quarter against 12, precision at 80 26% against 15%, and 9 approvals scored under 50 against 13 — for about a fifth more input tokens.

A prompt change gets a new version in src/lib/radar/prompt.ts: results of different prompts are reported separately and never mixed — and so are scores made with different history profiles, or none. radar-v0 is the prompt before publication history; radar-v1 adds it and puts the editorial profile explicitly first; radar-v2-history-retrieval is radar-v1 plus similar past publications; radar-v3-retrieval-approved is radar-v2 plus similar approved posts.

To see exactly what Radar would be shown for a post — Radar's model is not called; the retrieval prompt's search embeds the post once:

npm run radar:context -- --workspace 2 [--post <processed post id>] [--prompt radar-v1]

Publication history

Three kinds of evidence describe a channel, and Radar keeps them apart:

• Editorial profile (workspaces.editorial_profile): what the editor says the channel should publish now. The current policy, and the first thing Radar weighs.
• Approval history (processed_posts): what the editor approves and rejects when offered candidates, with reasons. The strongest evidence of current taste.
• Publication history (publication_history_items): what the channel has published, imported from its own export. Background: who the channel has been. It is positive-only — a subject missing from it says nothing about whether the editor would take it.

npm run history:import -- --workspace 2 --adapter telegram-json --file ./result.json
npm run history:profile -- --workspace 2
npm run history:embed -- --workspace 2
npm run radar:report -- --workspace 2

Importing history

A workspace's own back catalogue is imported from an export file into publication_history_items.

npm run history:import -- --workspace 2 --adapter telegram-json --file ./result.json
npm run history:import -- --workspace 2 --adapter telegram-json --file ./result.json --dry-run

Telegram. In Telegram Desktop open the channel → ⋮ → Export chat history, choose the Machine-readable JSON format, and pass the result.json it writes. Media files need not be exported: a post whose photo or video was left out is still imported, its media recorded as available: false. Posts without a caption but with media are kept; service events (pins, title changes) and messages with neither text nor media are skipped and counted by reason. The export carries no channel username and no view counts, so no post URL is stored and the metrics hold reactions only. Each photo of an album is its own message in the export and is stored as one. Dates come from date_unixtime; the export's date is the exporting computer's local time.

Re-imports. An item is identified by workspace, platform, publication (for Telegram, the channel's Bot API id, e.g. -1001234567890) and its id there. Importing the same export again changes nothing; importing a later export adds the new posts and updates stored ones whose text, media or reactions differ — the latest import wins. Identity is never changed.

Records. Every run is logged in publication_history_imports with its file's SHA-256, its counts and its outcome. A failed or interrupted (Ctrl+C) import is marked failed and writes no items: they are stored in one transaction. --dry-run reads and checks the file, reports how many posts are new, and writes nothing.

Size. The file is read whole into memory, up to 256 MB. A chat export's JSON holds only text (media files sit beside it), so this is tens of thousands of posts.

Other platforms. History is platform-neutral: the tables, the import service and the report know nothing of Telegram. Telegram Desktop JSON is only the first adapter; another format is a new file in src/lib/history/adapters/ that turns its export into HistoricalPublicationItems, registered in adapters/index.ts.

Profiling history

npm run history:profile -- --workspace 2 [--force] [--dry-run]

Distils the imported history into a compact, structured profile in publication_history_profiles: a summary, core topics with their strength, recurring angles, content patterns, tone, format, hooks, recurring names, 5–10 representative post ids, observations and caveats. The history is never sent whole: posts (each cut to 1,200 characters) are packed into batches of about 24,000 characters, a model writes notes on each batch, notes are merged while they are too many for one request, and the profile is written from the final notes and from facts measured in code (median length, share with media, links, emoji). The prompts tell the model that every post is one the channel chose to publish — evidence of what it does, never of what it avoids — and nothing is inferred from reactions. Media is described by type only; no images are looked at.

It uses RADAR_PROVIDER's model and key. A profile is identified by a fingerprint of the history (each item's identity, date, text and media types), PUBLICATION_PROFILE_PROMPT_VERSION and the model: when all three are unchanged the command says the profile is up to date and spends nothing, unless --force. When history has changed since the newest profile, it says so. --dry-run prints the profile without storing it (the model calls are still made). Profiles are never edited; each run that generates one adds a row. Nothing regenerates them automatically: import, then profile.

In Radar. Every live score from then on carries the newest profile, as background below the editorial profile, and records its id in radar_evaluations.publication_history_profile_id. With no profile, or one that cannot be read, Radar scores exactly as without it; it never delays or changes the post.

No hindsight in backfills. A profile records history_cutoff_at, the newest publication it was made from. A score — live or backfill — may only use a profile whose cutoff is at or before the moment the post arrived. Live, that is always the newest profile; in a backfill, a profile built today from history that includes posts published after an old candidate arrived is left out, and the candidate is scored without one. So most past posts are backfilled without a profile until there are profiles made at earlier points in time; that is deliberate.

Similar past publications

npm run history:embed -- --workspace 2 [--dry-run]

(It also embeds the workspace's processed posts, for similar approved posts.)

The profile is a summary; it loses the concrete posts. So each history item with text is also embedded — OpenAI's HISTORY_EMBEDDING_MODEL, text-embedding-3-small by default (1,536 dimensions, $0.02 per million tokens; text-embedding-3-large is 3,072 and $0.13) — into publication_history_embeddings, and the retrieval prompt shows Radar the five past publications nearest to the new post, with their dates and similarity.

What is embedded is the title and the text, whitespace evened out, cut at 6,000 characters — not the platform, type, dates, ids or metrics: those are the same across a channel's history, or not meaning, and would only pull every vector the same way. A new post is embedded the same way from its text. Items without text (photo or video alone) get no vector and stay in history as they are; media is not embedded yet.

Incremental. Each vector stores a SHA-256 fingerprint of the exact text embedded. A run embeds only items with no vector for the current model, or whose title or text changed since — run it after every import; an unchanged history costs nothing. An item that lost its text loses its vector.

Models never mix. Vectors are stored per item per model, with their length, and the search only compares vectors of the same model and length. The column has no fixed dimension, so trying another model needs no migration — set HISTORY_EMBEDDING_MODEL, run history:embed again, and the old vectors stay for the old model. The search is an exact scan; a workspace's history is small enough not to need an approximate index.

The search is one query, with every boundary in it: only the workspace's own history, only publications from strictly before the post arrived — live and in a backfill alike, so a backfill never sees what the channel published later — and the five most similar by cosine similarity, with no threshold. The prompt tells the model that the five are the nearest of whatever exists and may be unrelated, that similarity ranks them without settling anything, and to tell a recurring topic (a sign of fit) from the same story (a likely repeat) and from a new development of a known story (not a repeat). Each post's vector is stored in radar_candidate_embeddings, so it is embedded once, and a backfill can read back exactly what its requests were shown.

Never in the way. A post without text, a workspace without embedded history, a missing OPENAI_API_KEY, an API or database error: the retrieval prompt then runs without matches, its row says why (history_retrieval.status), and the baseline is unaffected. Live, the search gets five seconds.

Similar approved posts

Most repeats an editor turns down are not of what the channel published long ago but of what they approved last week from the same stream of sources — which, while posts go to a draft channel, is not in the channel's exported history at all. So radar-v3-retrieval-approved also shows Radar the five posts the editor already approved (published or scheduled) that are nearest to the new one, with when they were approved, and tells it that one reporting the same story is the most direct sign of a repeat — the more so the more recent.

The vectors are the posts' own, in radar_candidate_embeddings: every post Radar searches with is embedded on the way, so a post approved after that is already searchable; npm run history:embed embeds the posts that never were (all of a workspace's posts with text, once). The search has the same boundaries, in the query: this workspace, the same model, approved — and approved strictly before the new post arrived, so a backfill never sees an approval the editor had not yet made — and never the post itself. It shares the post's vector with the history search, so it costs no extra call, and runs even for a workspace with no imported history. What it found is recorded in history_retrieval.approved, with its own status.

Similar rejected posts

Ten recent rejections say little about a kind of post the editor keeps turning down, and the approved search alone can mislead: for a striking galaxy photo it finds the one or two of its kind the editor did take, and nothing of the dozen they declined. So radar-v5-similar-rejected is radar-v3 plus the five posts the editor rejected that are nearest to the new one, each with its rejection reason and note, and tells Radar to read them beside the approved ones: the same kind of material turned down again and again is the editor's answer for this post too, however appealing it is. It is built on radar-v3 rather than radar-v4, which scored worse than radar-v3 live, so radar-v3 against radar-v5 measures this one addition. A galaxy posted as "wow" is still found among galaxies: the post's image description is embedded with its text, whichever prompt reads it. Same vectors, same boundaries (rejected strictly before the new post arrived), no extra embedding call; recorded in history_retrieval.rejected.

On the first backfill, 199 posts, it ranked no better than radar-v3 (same AUC and daily hit rate) and turned pessimistic about everything: with nine posts in ten rejected, rejected neighbours turn up for almost any post, and it read their mere presence as a verdict — halving the galaxy photos scored high, and missing approvals with a bare caption too. On the 219 newer posts after them, though, it ordered each day far better: 10 of 24 approvals in each day's top against radar-v3's 4 (chance: 3.5), AUC 0.74 against 0.69 — over all 418, a daily hit rate of 41% against 32%. The likely reason is that later posts have more rejections behind them to be found. Three days and 24 approvals are not proof, so it ran live beside radar-v3. Its scores sit lower (27 approvals under 50 against 15), so read its threshold lower too.

Live, on the same 332 posts over 11 days, it did better still: 19 of 33 approvals in each day's top against radar-v3's 8 (chance: 17%), AUC 0.79 against 0.68. So since 22 October 2026 the review queue, its colours and the digest's recommendations follow radar-v5, and radar-v3 is scored live but no longer shown.

Nearest decisions

So radar-v6-nearest-decisions shows both kinds at once: radar-v3 plus the ten posts the editor had decided nearest to the new one, approved and rejected together, most similar first, counted ("3 approved, 7 rejected"), each rejection with its reason and note. Radar reads them as a vote weighed against the editor's usual approval rate: a share approved well above it says they take this kind of post, well below it that they turn it down, near it nothing either way. They are the nearest of the approved and rejected searches merged — ten of each fetched, the first five approved still shown as radar-v3 shows them — so no extra query or embedding; recorded in history_retrieval.nearest.

It did no better: on the same backfill its daily hit rate matched radar-v3's, and like radar-v5 it mostly lowered every score — cleaner at the very top, more approvals under 50. Ranking posts by the share approved among their ten nearest decisions alone, with no model at all, gets about 30–34% against radar-v3's 41%: the model earns its keep, but more neighbours in its prompt add nothing. It runs on backfills only; over all 418 posts its daily hit rate (38%) fell between radar-v3's and radar-v5's.

Cost. One embedding call per new post (a few hundred tokens), and the history once: VECTOR's 83 posts with text, about 24,000 characters, cost a fraction of a cent. history:embed prints the tokens and the cost; the per-post calls are logged (radar.history_retrieved) and their tokens stored with the post's vector.

Image understanding

A post that says only "wow" under a photo of an aurora is about the aurora — but text search, and a text-only Radar prompt, cannot know it. So a post's first photo is looked at once by a vision model, and what it shows is stored as a few short, factual fields: a one- or two-sentence summary, the content_type (photo, illustration, screenshot, meme, chart, diagram, other), up to six topics and entities, any meaningful visible_text, and an information_value (decorative, supporting, essential: how much the post depends on the image). Those words are then reused everywhere, with no second look:

• Radar — radar-v4-media-understanding reads them in place of the image;
• similarity search — a post's embedded text is its title and text plus what its image shows (buildSemanticContentRepresentation), for new posts and past publications alike, so an image-only post can be found, and find others.

Once, and only once. An understanding is stored in media_understandings under the SHA-256 of the image's bytes and the model, prompt version and detail level — never a URL, and never a workspace: the model is shown the image and the caption, nothing of any channel, so the same picture means the same thing to every tenant and is looked at once for all of them. Every Radar version, the retrieval and later backfills read the stored row. A new post's photo is the one already downloaded for the send; it is fetched only when the send does not download it (MEDIA_UPLOAD_MODE=url). A post records which image it is in processed_posts.image_fingerprint; each radar_evaluations row of the media prompt records which understanding it read (media_understanding_id, whose row names the model and prompt version).

The call. MEDIA_UNDERSTANDING_MODEL (GPT-6 Luna by default), OpenAI's Responses API, no reasoning, the image at low detail, a short prompt that treats everything in the image and the caption as data, never instructions, and never names a person from their appearance. Only the first photo of a post is looked at — an album's others, videos, GIFs and anything that is not a JPEG, PNG or WebP are not, yet. It is enrichment, never in the way: an image that cannot be fetched, a timeout or a refusal is recorded (a failed row, retried by the next request for the same image) and the post is scored and reviewed as it would have been. The usage the API reports and its list-price cost are stored on each row.

The history's images are read from the export's files, once:

npm run history:understand-images -- --workspace 2 --media-root ./ChatExport_2026-10-06 --dry-run
npm run history:understand-images -- --workspace 2 --media-root ./ChatExport_2026-10-06 --limit 50
npm run history:understand-images -- --workspace 2 --media-root ./ChatExport_2026-10-06
npm run history:embed -- --workspace 2

--media-root is the folder with result.json and photos/; each item's stored relativePath is read under it, and never outside it. An item whose photo the export left out, or whose file is missing, is skipped and counted (Skipped unavailable), never fetched from Telegram. It is resumable: what is already understood is passed over, so a run that stopped, or one with --limit, is simply run again. history:embed then embeds again only the items whose image was newly understood, and gives image-only posts their first vector; posts without an image keep their vectors untouched. It uses the standard API, not the batch one: a channel's history is a few hundred images.

Cost, measured: about 700 input and 85 output tokens an image, ≈ $0.00011 an image at GPT-6 Luna's list price, ≈ $0.011 per 100, and ≈ $0.016 for VECTOR's 143 history posts with a photo. Live, one such call per new post with a photo, then the media prompt is one more text call — and since the raw image is no longer sent to every version, an image post costs less to score than before.

Comparing. npm run radar:report -- --workspace 2 --compare radar-v3-retrieval-approved,radar-v4-media-understanding puts the two side by side on the same posts — and again on the posts with images only, where an effect would otherwise drown among text posts — with what the images cost to look at.

Stories

The same event comes in from several sources: a launch from the company, the agency, two news accounts and a feed, within hours. Story clustering groups them — one story per event — as they arrive. The review queue shows one card per story, and marks repeats (below); Radar's radar-v7, on backfills, reads the story so far. stories:cluster reports what the clustering made of a workspace.

How a post is placed. By the vector Radar's retrieval already made of it (text and what its image shows), so no extra embedding call. It is compared with the posts of its workspace that arrived in the 48 hours before it; when the nearest is at least 0.75 similar, the post joins that post's story, otherwise it starts its own. One link is enough, so a story can run for days while each new post is close to one in the window. What was found is kept either way — in story_posts, the nearest post, its similarity and the threshold used — so the threshold can be judged and changed from what the posts actually looked like. A post without a vector (no text and no understood image, or a workspace without Radar) is not placed.

The threshold. On VECTOR's first week, a nearest post at 0.75 or more was almost always the same event from another account — a crew's return, a reentry video, an obituary, the same record fast radio burst; between 0.70 and 0.75 about half were; below that it was mostly the same subject, another galaxy or nebula. STORY_SIMILARITY_THRESHOLD in src/lib/stories/cluster.ts.

Placing past posts, and trying another threshold:

npm run stories:cluster -- --workspace 2
npm run stories:cluster -- --workspace 2 --rebuild --threshold 0.72 --dry-run
npm run stories:cluster -- --workspace 2 --report

Repeats. Some sources post the same text and picture again every day or two. A post whose nearest earlier post is from the same source and at least 0.97 similar (REPEAT_SIMILARITY in src/lib/stories/repeats.ts) is marked as a repeat, with what became of the earlier copy: 🔁 Repeat of 12 Oct · rejected beside the source in the review queue, and 🔁 Repeat of a post from 12 Oct · rejected under Radar's note on the message with the buttons. A mark only: it still comes to review, and its Radar score is unchanged. On VECTOR's first week 50 posts were repeats — 47 rejected, 3 published the second time round — all from the same source; no two sources came that close.

It places, in the order they arrived, the posts that have a vector and no story yet — exactly as live — and prints the report: how many stories hold more than one post and from how many sources, the nearest similarities by band, the stories the editor published twice, and how many of their "already covered" rejections sit in a story they had already published. --rebuild forgets the workspace's stories first; --dry-run rolls everything back.

The story so far, in Radar

radar-v7-story-sources shows Radar the post's story as it stood when the post arrived: how many earlier posts reported the same event and from how many other sources, and the latest eight of them — source, how many hours before, and what the editor had done with each by then (published, rejected and why, or undecided). It is asked to read the number of independent sources as a sign of how big the story is, a post of it already published as a likely repeat, and a story already turned down as likely turned down again. Only what came before is shown: posts that arrived later and decisions made later are not, so a backfill measures prediction. Live, the post is placed in its story first, from the vector the retrieval just made; a backfill reads the placement stories:cluster made. A post with no vector is shown as "not known". What the prompt was shown is recorded in radar_evaluations.history_retrieval.story.

It did worse than radar-v3. On the same 418 posts (58 approved): AUC 0.68 against 0.71, a daily hit rate of 24% against 32%, the top quarter of each day 40% against 48%, and 21 approvals under 50 against 15. Most of what it newly missed were repeats the editor published the second time round after turning the first copy down — the rule that a story turned down will be turned down again read them the wrong way — and with only 15 stories from more than one source in the week, the number of sources had little to tell. It runs on backfills only.

Configuration reference

Variable	Default	Description
DATABASE_URL	—	Required. PostgreSQL connection string (use the pooled endpoint).
X_USER_ID	—	Deprecated. Legacy single-source config, imported once then ignored.
X_USERNAME	—	Deprecated. Legacy single-source config, imported once then ignored.
X_BEARER_TOKEN	—	Required. App-only Bearer token.
X_API_BASE_URL	https://api.x.com	Override only if proxying.
TELEGRAM_BOT_TOKEN	—	Required. From BotFather.
TELEGRAM_CHAT_ID	—	Seeds workspace 1's channel (-1001234567890 or @channelusername), once, while its column is empty. Unused after that; safe to remove.
TELEGRAM_API_BASE_URL	https://api.telegram.org	Override for a local Bot API server.
TELEGRAM_DISABLE_NOTIFICATION	false	Post silently.
CRON_SECRET	—	Required, ≥ 16 chars. Protects /api/cron/sync.
REQUIRE_APPROVAL	false	Hold every post for review instead of publishing directly.
TELEGRAM_ADMIN_CHAT_ID	—	Seeds workspace 1's reviewer (your numeric Telegram user id), once, while its column is empty. Unused after that; safe to remove.
TELEGRAM_WEBHOOK_SECRET	—	≥ 16 chars, A-Z a-z 0-9 _ - only. Required when REQUIRE_APPROVAL is on.
APP_BASE_URL	—	Public HTTPS origin, e.g. https://your-app.vercel.app. Enables the Edit button; without it review works unchanged.
WEB_APP_URL

Booleans accept true/false, 1/0, yes/no, on/off.

Verified API limits

These are read from the official documentation and live in src/lib/telegram/limits.ts as the single source of truth:

Limit	Value
Caption length	1024 characters
Message text length	4096 characters
Album size	2–10 items (photos and videos may be mixed)
Upload via multipart	10 MB photo / 50 MB other
Upload via URL	5 MB photo / 20 MB other
Photo dimensions	width + height ≤ 10000, ratio ≤ 20

Long posts. The text comes from X's note_tweet field where there is one, so a long-form post (over 280 characters) is mirrored whole rather than as X's 280-character cut. A text that does not fit the 1024-character caption goes out as media with a shortened caption, followed by the whole text as its own message. A text longer than even that message may be (4096) is shortened there too, with an ellipsis; in both, only the author's text is cut — CAPTION_PREFIX, the source link and CAPTION_SUFFIX are always kept. With approval on, the reviewer's preview shows that follow-up message too, exactly as it will be sent, between the media and the buttons. A post changed in the editor goes out as laid out there, with a follow-up of its own when its text needs one.

---

Architecture decisions

One adapter per platform. The sync run knows nothing of X, feeds or channels: it hands each source to its platform's SourceAdapter (src/lib/sync/source-adapters.ts), which reads what is new in its own way — a cursor on X and Telegram, identity on a feed or a sitemap — and passes each post, normalised, to the path every platform shares from claim to review (processCandidate). A platform's identity — the prefix of its posts' ids in processed_posts.external_id (none for X, rss:, tg:, yt:, web:) and how a source is named — is in src/lib/sources/display.ts. Both are typed by source_platform, so a new platform (a newsletter, say) does not compile until it has both. The column was x_post_id until every platform used it (migration 0031_external_id).

Drizzle ORM, not Prisma. Prisma ships a query-engine binary that inflates the serverless bundle and cold-start time. Drizzle is plain TypeScript with generated SQL migrations.

postgres.js, not the Neon HTTP driver. The HTTP driver cannot do interactive transactions or session-level advisory locks, and this design needs both. postgres.js works unchanged against Neon, Vercel Postgres, Supabase or self-hosted PostgreSQL. The pool is capped at 2 connections: the advisory lock reserves one for the run's duration, and the second serves queries. (With a pool of 1, the lock starves every subsequent query and the run deadlocks — there is a regression test for exactly this.)

HTML parse mode, not MarkdownV2. MarkdownV2 requires escaping 18 characters wherever they appear, and one missed character makes Telegram reject the entire message with can't parse entities. HTML needs three (&, <, >), which is far safer on arbitrary author-written text.

Workspace scoping, ahead of multi-tenancy. sources and processed_posts carry a workspace_id, and duplicate protection is UNIQUE (workspace_id, external_id) rather than global. Only one workspace exists and the runtime still reads the channel and reviewer from the environment — but the columns are in place now because they are cheap to add to small tables and expensive later: once a second tenant exists, the same X post legitimately belongs to two channels, and swapping a global unique index under live traffic means dropping duplicate protection while it rebuilds. processed_posts.source_id records which account a post came from, and survives that source being deleted, so removing a source cannot cause its published posts to be offered for review again.

Idempotency in three layers. Vercel documents that cron delivery can duplicate an invocation and that a long run may overlap the next one, so duplicate protection cannot rely on scheduling:

1. A PostgreSQL advisory lock means only one sync runs at a time; an overlapping invocation exits immediately with lockBusy rather than queueing.
2. An atomic claim — a single INSERT … ON CONFLICT DO UPDATE … WHERE against the UNIQUE external_id — decides ownership of each post in one statement. Ten simultaneous runners produce exactly one winner. This is the guarantee that actually prevents double-posting; the lock is only an optimisation.
3. A processing lease: a row claimed by a function that was then killed is reclaimed after 10 minutes, so a crash cannot strand a post forever.

The cursor only advances when nothing failed. Moving since_id past a failed post would hide it from every future run.

Unicode-safe truncation. Captions are cut on grapheme boundaries via Intl.Segmenter, so an emoji, flag or ZWJ sequence is never split into replacement characters. The X API's entity offsets are in code points while JavaScript strings are UTF-16, so t.co links are removed by matching the literal URL rather than slicing by offset — slicing corrupts any post containing emoji.

Link handling. t.co links are replaced with the author's real expanded_url. The trailing pic.x.com/… link X appends for a post's own media is removed entirely, since the media is attached to the Telegram message. Links the author genuinely wrote are preserved.

---

Project structure

src/
  app/
    api/cron/sync/route.ts     GET /api/cron/sync   — CRON_SECRET protected
    api/status/route.ts        GET /api/status      — ADMIN_SECRET protected
  db/
    schema.ts                  Drizzle tables
    migrations/                Generated SQL
  lib/
    env.ts                     Zod-validated configuration + redacted summary
    db.ts                      Pooled connection
    auth.ts                    Constant-time secret comparison
    errors.ts                  Transient vs permanent taxonomy
    logger.ts                  Structured JSON logs with secret scrubbing
    x/
      client.ts                X API v2 transport
      schemas.ts               Zod models of X responses
      get-new-posts.ts         Fetch, filter, order oldest-first
      normalize-post.ts        Text cleanup, t.co handling, video variant choice
      download-media.ts        Size-guarded streaming download
    telegram/
      client.ts                Bot API transport, 429 / retry_after
      limits.ts                Documented limits, single source of truth
      format-caption.ts        Caption assembly, HTML escaping, safe truncation
      send-media.ts            sendPhoto / sendVideo / sendMediaGroup / sendMessage
    rss/
      url.ts                   Feed URL checks, private-address (SSRF) guard
      client.ts                Size- and time-bounded fetch, redirects checked hop by hop
      parser.ts                RSS 2.0 / Atom 1.0 → ParsedFeed, entities never expanded
      text.ts                  HTML and entities to plain text
      identity.ts              Stable entry keys, rss:<feed>:<entry> item ids
      normalize.ts             Feed entry → candidate post
    telegram-channel/
      username.ts              @name / t.me links → a channel's username; invite links refused
      client.ts                t.me/s/<channel> and one post's embed, no redirects, size-bounded
      parser.ts                Channel page → posts, media and what the page cannot show
      normalize.ts             Channel post → candidate post, tg:<channel>/<id> ids
      refresh-media.ts         Fresh media links for a queued post, before it is published
    youtube/
      channel.ts               @handle / channel links → a channel; video links refused; UC → UU playlist
      client.ts                The Data API: channels, playlistItems, videos — never search; quota logged
      normalize.ts             Video → candidate post (title + description), yt:<id> ids
      transcript.ts            VideoTranscriptProvider, Supadata (native transcripts only)
    website/
      url.ts                   Sitemap and page addresses: normalised, same-site, web:<hash> ids
      http.ts                  One bounded public GET: private addresses refused hop by hop, size and time caps
      sitemap-parser.ts        <urlset> / <sitemapindex> in any namespace prefix, entities never expanded
      sitemap.ts               The sitemap tree: gzip, depth, cycles, same site, per-sync limits
      article-client.ts        A new page's HTML, and nothing else
      article-extractor.ts     jsdom (no scripts) + Readability: the article, or why it is none
      normalize.ts             Article → candidate post (title + lead; the article as analysis text)
    analysis-text.ts           A post's analysis-only text, sampled across it for Radar and the embedding
    sources/
      display.ts               Each platform's identity: its post ids' prefix, and how a source is named
      add-rss.ts               Check-then-store for /addrss and Settings
      add-telegram.ts          Check-then-store for /addtelegram and Settings
      add-youtube.ts           Look-up-then-store for /addyoutube and Settings
      add-website.ts           Read-the-sitemap-then-store for /addwebsite and the Sources page
    sync/
      sync-posts.ts            Orchestration, X, RSS and Telegram sources alike
      source-adapters.ts       One adapter per platform: the orchestrator hands each source to its own
      sync-x-source.ts         One X account: cursor, candidates
      sync-rss-source.ts       One feed: backlog on follow, known-id dedup, candidates
      sync-telegram-source.ts  One public channel: message-id cursor, pages back to it
      sync-youtube-source.ts   One YouTube channel: uploads newest first, back to what is known
      sync-website-source.ts   One website: sitemap backlog on follow, new pages by identity, bounded fetches
      process-candidate.ts     Claim → Radar → translate → review/publish → record, shared
      process-post.ts          Per-post publish, all-or-nothing media policy
      acquire-media.ts         Fetch and check every asset before a send, for both paths
      repository.ts            Atomic claim and state transitions
      locks.ts                 Advisory lock
      retry.ts                 Backoff with jitter
    history/
      types.ts                 Canonical history item, adapter contract
      import-history.ts        Import service: adapter → repository → report
      repository.ts            Upserts, import records
      adapters/telegram-json.ts  Telegram Desktop JSON export
      profile/                 History → editorial profile: sources, prompts, map/reduce, store
      embeddings/              Embedded text and fingerprint, OpenAI embeddings, backfill, search
    stories/
      cluster.ts               A post into the story of the nearest close post before it, or a new one
      report.ts                What the clustering made of a workspace, beside the editor's decisions
      repeats.ts               A source sending the same post again: found, and said in review
      context.ts               A post's story as it stood when it arrived, for Radar's v7
    rich/
      document.ts              A rich post: the editor's document, checked, to Telegram rich HTML
      edit-html.ts             The document as the small HTML the AI assistant edits, and back
    assistant/
      prompt.ts                What the post editor's assistant is told: channel, source, draft
      openai.ts                One streamed turn on OpenAI: web search, edit_post
      repository.ts            Whether it is set up; assistant_requests and the daily limit
      events.ts                The stream's events, shared with the editor's panel
    uploads/
      uploads.ts               The editor's photos and videos in Vercel Blob: checked, recorded, sent, deleted
    review/
      queue.ts                 The review queue: its page's list, the notification and when it is due
      same-story.ts            A story's card in the queue: the rest of it decided with its post
      digest.ts                The queue notification: the period's stories, Radar's picks, in the channel's language
      text.ts                  A post's text as it will be published, for the queue and the digest
    accounts/
      users.ts                 Users, identities, memberships; a reviewer let in on first sign-in
      sessions.ts              Website sessions: random token in a __Host- cookie, its hash stored
      oidc.ts                  OpenID Connect sign-in, any provider: PKCE, code exchange, ID token checks
      telegram.ts              Telegram as an OIDC provider; its token's Telegram user id
      viewer.ts                Who is asking — Mini App initData or website session — for every API
  components/
    review-queue.tsx           The review queue, shared by the Mini App and the website
    settings-panel.tsx         Settings, likewise
    source-stats-panel.tsx     The Sources page — adding and stats — likewise
    add-source-form.tsx        ➕ Add a source: X, Telegram, YouTube, RSS or a website, on the Sources page
    mini-app-frame.tsx         The Mini App frame: Telegram's script, initData-signed requests
    rich-editor/               The website's post editor (Tiptap), its picture block and AI assistant panel
    media/
      understanding.ts         ImageUnderstanding, its prompt, schema and config
      provider.ts              The vision call: one low-detail image, no reasoning
      understand.ts            Understand an image at most once, never throwing
      repository.ts            media_understandings, by fingerprint and config
      history.ts               The history's images, from the export's files
scripts/                       migrate, run-sync, telegram-check, telegram-commands, rss-check, website-check,
                               import-history, history-profile, history-embed,
                               history-understand-images, radar-backfill,
                               radar-report, radar-context, stories-cluster
tests/                         Unit + integration suites

---

Testing

npm test

208 tests. The unit suite runs anywhere. The integration suites need a real PostgreSQL, because the guarantees under test — UNIQUE races, ON CONFLICT semantics, advisory locks — only exist in the database; they are skipped rather than failed when no database is configured. The database needs the pgvector extension, hence the pgvector/pgvector image.

docker run -d --name ca-test-pg \
  -e POSTGRES_PASSWORD=test -e POSTGRES_DB=content_arbitrary_test \
  -p 55432:5432 pgvector/pgvector:pg16

DATABASE_URL=postgresql://postgres:test@localhost:55432/content_arbitrary_test \
  npm run db:migrate

TEST_DATABASE_URL=postgresql://postgres:test@localhost:55432/content_arbitrary_test \
  npm test

Coverage includes caption building and Unicode-safe truncation; t.co media-link removal; source links; reply/repost/quote filtering; oldest-first ordering; method selection (1 photo → sendPhoto, 1 video → sendVideo, many → sendMediaGroup); album caption placement; retry and backoff; Telegram 429 retry_after; X and Telegram response parsing; media size guards; duplicate prevention under ten concurrent claims; two concurrent syncs publishing each post exactly once; and the pool-starvation deadlock regression.

---

Operating notes

• Watch the first real run. Set DRY_RUN=false, trigger /api/cron/sync by hand, and check the channel before trusting the schedule.
• There is no backfill. A source publishes only what its account posts after the source was added or last resumed; the posts already on its timeline are skipped.
• Who goes first rotates. Each run starts with the tenant, and within it the source, that has waited longest since its last sync — never-synced first — so when a run does hit its time budget, it is not always the same accounts that wait.
• A backlog is read back to the cursor. X returns the newest posts first, a page at a time. If more arrived since the last run than one page (X_FETCH_LIMIT) holds, the sync reads on, page by page, down to its cursor, so the oldest — the next to publish — are not skipped. It reads at most 5 pages per source per run; beyond that the oldest posts are not seen, and the run logs sync.window_overflow and reports windowOverflow for that source. X bills a post read again within 24 hours only once, so reading a backlog over several runs costs no more.
• A run has a time budget. /api/cron/sync may run for 800 s (the Pro plan's maximum), but stops taking new posts after 560 s (SYNC_TIME_BUDGET_MS) and leaves the rest to the next run, with each source's cursor stopping at the last post it finished — nothing it did not reach is skipped. The response then carries timeBudgetReached: true. Requests have their own limits: 30 s for X, 120 s for a media download and for a Telegram call. A Telegram call that times out is not resent in the same run, since it may have gone through; the post is retried by a later run.
• The channel is append-only. Deleting a Telegram message does not change the database; the post stays published and will not be re-sent. Delete its processed_posts row to republish.
• npm audit reports moderate advisories from drizzle-kit's bundled esbuild. These are dev-only tooling and are not part of the deployed function.

Licence

Provided as-is. You are responsible for complying with the X Developer Agreement, Telegram's Terms of Service, and the copyright of anything you republish.