PortalJS Ask (search.portaljs.com)
Ask a question about an organisation's open data in plain English and get a short data story
with charts. Every number comes from a SQL query run on the real data, and every chart shows
its query.
One engine, one config folder per client. Live:
search.portaljs.com (DataHub, the default) ·
search.portaljs.com/demo/toronto
How it works in depth (pipeline, costs, hosting, decisions): ARCHITECTURE.md.
1. Which solution should I use?
Datopian has two ways to build a client demo. They are different products, in different repos.
| PortalJS Ask (this repo) | Portal demo |
|---|
| What the client sees | One page with a question box. A question gets a short data story with charts | Their own portal, modernised: catalogue, search, dataset pages, plus an optional data story and "Ask AI" |
| What it sells | The AI question-answering layer on its own | A portal revamp or migration (with AI as a bonus) |
| Best for | Organisations with open data but no portal they care about, or an AI add-on for an existing client | Anyone who already runs a portal: CKAN, ArcGIS Hub, EntryScape, Opendatasoft, Socrata |
| Effort | Hours: pick 5–20 datasets, write data notes and example questions | Days: connect the backend, brand it, write one story |
| Repo | datopian/portaljs-ask, one folder per client in portals/ | datopian/data-portal-examples (internal), see its CLIENT-DEMO-GUIDE.md |
| URL | search.portaljs.com/demo/<slug> or the client's own domain | <slug>.arc.portaljs.com or <slug>.portaljs.com |
They combine well: a portal demo's story page can link to the client's PortalJS Ask page
for follow-up questions.
2. What it is, and what it is not
It is:
- A Next.js app on Vercel. A visitor's question goes to Claude, which writes SQL. The SQL runs
in the visitor's browser (DuckDB) over the client's data as parquet files, and Claude then
writes the story from the result rows.
- Fact-checked automatically: a second AI call compares each claim in a new story with the
rows and fixes wrong ones before the story is shown or cached.
- Cheap to show: example questions are pre-computed (free and instant), and repeated
questions are cached for 30 days. Only new typed questions cost money, about 5–8 US cents each.
- Configuration, not code, per client: a folder in
portals/<slug>/. Never a fork.
It is not:
- Not a document search or chatbot. It answers from tables only. The document demos
(
/security, /governance) were removed on 2026-10-12.
- Not a data portal. There's no catalogue, dataset pages or publishing. For that, build a
portal demo (§1).
- Not for private or big data. Data files are public, and it works up to tens of MB per table.
Non-public or multi-GB data needs a server-side setup, which is a separate offer.
- Not a live feed. Each portal is a snapshot we download, clean and describe. Refreshing it
is a deliberate step (
portal-refresh).
- Not logins or permissions. Everyone sees the same data.
3. Where everything lives
| What | Where |
|---|
| Code | This repo. main is production |
| Hosting | Vercel, Datopian team, project portaljs-ask. Every pull request gets a preview URL |
| Client configs | portals/<slug>/: portal.json, sources.json, notes.md, profile.md, instant.json |
| Which portal is the homepage, client domains | portals/index.json |
| Data files | Vercel Blob store portaljs-ask-data, one folder per portal and snapshot. Not in git |
| Team skills for Claude Code | .claude/skills/: portal-new, portal-verify, portal-refresh, ask-cost-report |
Existing portals: datahub (17 DataHub core datasets, the homepage) and toronto (20 City of
Toronto datasets).
4. Run it locally
You need Node 20+ and an Anthropic API key (ask for the team key; never commit it).
gh repo clone datopian/portaljs-ask
cd portaljs-ask
npm install
cp .env.example .env.local # set ANTHROPIC_API_KEY at minimum
npm run dev # builds the portal configs, then starts Next.js
- http://localhost:3000 shows the default portal; http://localhost:3000/demo/toronto shows Toronto.
- Example questions load from
instant.json and cost nothing. Typed questions call the API and cost money.
- Without the Upstash Redis variables, limits and the answer cache are per process. That's fine locally.
- After editing anything in
portals/, re-run npm run portals (dev and build do it for you).
5. Add a client portal
Easiest: in Claude Code, from this repo, run the portal-new skill and give it the
organisation, its catalogue URL and (optionally) the datasets. It follows the steps below and
stops at a preview link.
By hand (full detail in ARCHITECTURE.md §6 and .claude/skills/portal-new/SKILL.md):
- Branch:
git checkout -b portal/<slug>
- Config: create
portals/<slug>/portal.json by copying portals/toronto/portal.json
and changing every field (name, owner, headline, theme colours, logo, source, catalogue,
data.base = /data/<slug>/, data.snapshot = today). Add sources.json with 5–20 tables.
Pick tabular datasets with dates and categories people ask about; skip map-only, PDF-only
or very large datasets.
- Data:
node scripts/portal-data.mjs <slug> downloads, cleans and converts the data and
writes profile.md. Resolve every Check item it lists.
- Notes: write
portals/<slug>/notes.md: for every table, what one row is, every column
with unit and range, how to count correctly, and traps. This is where answer quality comes
from. Check every number you write with DuckDB.
- Questions: add
datasets and 15–20 examples to portal.json. The first 3 are the
headline ones.
- Upload data:
BLOB_READ_WRITE_TOKEN=... node scripts/upload-data.mjs <slug>, then put the
printed URL in data.remote in portal.json. Deploys fail without it.
- Preview:
npm run portals && npx tsc --noEmit -p ., commit, push, open a draft PR and
wait for the Vercel preview.
- Pre-compute and verify:
ASK_ADMIN_TOKEN=... node scripts/precompute.mjs <slug> --api <preview URL>, then run the
portal-verify skill. Read every saved story against its rows (counts, ratios,
"highest", periods), not only the headline numbers.
- Merge once the portal owner agrees. It's live at
search.portaljs.com/demo/<slug>.
Zero-risk mode: set "live": false in portal.json. The question box disappears, only
the pre-computed examples show, and the AI cost is zero. Use it for demos sent by email.
Client's own domain (e.g. ask.example.org): add "<host>": "<slug>" to domains in
portals/index.json, add the domain to the Vercel project, and have the client point a CNAME
at cname.vercel-dns.com. See ARCHITECTURE.md §6.
6. Costs and limits
| Limit | Default | Env var |
|---|
| Monthly AI spend for live questions | $40 (Anthropic account hard cap: $50) | ASK_MONTHLY_BUDGET_USD |
| Live questions per day, all portals | 30 | ASK_DAILY_TOTAL |
| Live questions per visitor per day | 5 | ASK_DAILY_PER_VISITOR |
When a limit is hit, live questions pause and the examples keep working. Check spend with the
ask-cost-report skill, or:
curl -H "x-ask-admin: $ASK_ADMIN_TOKEN" https://search.portaljs.com/api/data/status
7. Before showing a portal to a client
8. Troubleshooting
| Problem | Fix |
|---|
| "Live questions are paused" | A daily or monthly limit was hit. Examples still work. See §6 |
| Deployment fails | A portal is missing data.remote in portal.json (step 6), or npx tsc --noEmit -p . fails |
| Wrong or odd numbers in answers | Fix notes.md (units, how to count, traps), then re-run precompute.mjs (--restory rewrites stories from existing rows) |
| Question outside the data | Expected: the page offers a free catalogue search linking to matching datasets |
