The point of this project is not "let a model give a score". It is to decompose grading into a set of
atomic, checkable questions, decide each of them, and combine the probabilities in code. The decision
engine is swappable: a general-LLM judge by default, TypeSafe Jev (System One) when a
TYPESAFE_API_KEY is configured, and a lexical baseline for offline demos. Every point in the final
score can be audited in the report.
Four things differ from the usual approach: point-by-point grading (not one opaque 0–100 score),
uncertainty surfaced instead of hidden (a weak judgment becomes "needs review" with a score range),
FSRS review scheduling after the exam, and one-page explainers for concepts that need a picture.
Model access is a choice: bring your own key (DeepSeek, GLM, Qwen, Kimi, OpenAI, Claude) or use platform
credits.
Screenshots come from the offline demo mode (the fallback engine used when no API key is configured), so
question style is mechanical and grading uses lexical overlap instead of Jev. With TYPESAFE_API_KEY the
UI is identical and the grading quality is not.
To see the output without installing anything, open the
hosted sample report (single file, fully offline),
or the copy in the repository at docs/demo/report.html.
Three guarantees you can check
| Guarantee | How | Where |
|---|
| Material facts are locatable | source_anchor and evidence_span must appear verbatim in the material; failures are listed as violations | src/lib/provenance.ts |
| Coverage is not faked | The material is split into units, and units covered by no question are listed as uncovered (9/14) | src/lib/coverage.ts |
| Uncertainty is surfaced | A weakly judged key point turns the question into "needs review" with a score range, excluded from mastery | src/lib/grading/subjective.ts |
Every field in the report is tagged material (locatable verbatim) or model-added (generated).
Q&A follows the same contract: every factual sentence carries a [n] citation pointing at a verbatim span
(with a page number for PDFs). Out-of-range markers are deleted and reported by code, sentences with no
source are listed separately, and anything outside the material must be tagged "model-added". When retrieval
finds nothing relevant the app answers "the material does not say" without calling a model at all.
How it works
| Step | Owner | Note |
|---|
| Reading material, writing questions and rubric points | a generative LLM / your agent | the decision engine generates no text, so it cannot do this |
| Objective grading | deterministic code | normalised comparison; only cloze triggers one semantic-equivalence question |
| Written-answer grading | the decision engine (one noul per rubric point) | the probability is the point score; code weights and combines them |
A decision model such as TypeSafe Jev takes text or state in and gives typed probabilities out (noul
0–1, a choice distribution, a score on an ordered rubric). It returns no text, no rationale and does
no arithmetic. Since noul carries no confidence field, this project measures judgment strength as
strength = |p − 0.5| × 2.
Swapping engines requires implementing one interface, DecisionEngine (src/lib/types.ts).
Usage
1. Web app (full loop)
npm install
cp .env.example .env # optional: it runs without any key in offline demo mode
npm run dev # http://localhost:3000
Invite codes, material library, topic confirmation, answering, a point-level report, a mistake log,
PDF/Word upload with automatic extraction, material Q&A with sources, spaced review (FSRS-5),
daily quotas and BYOK keys are all included.
Upload a file instead of pasting: PDFs are parsed page by page and the page boundaries are stored, so a
citation can point at "page 3"; Word documents give up their .docx text. Extraction only fills the form —
nothing is saved until you review the text, so a bad extraction never lands in your material silently.
Your data stays yours: settings export Markdown / Anki CSV / a full JSON backup, and the account can be
deleted in one place. When a judgment is wrong there is a "graded this wrong?" button on every question; the
report freezes the question, your answer and the per-point probabilities at that moment for human review.
Mistakes are not just collected: after you submit, every question you lost points on enters the review
queue (due the same day), and each review maps the judged score onto an FSRS rating that pushes the next
due date out. Handwritten answers that cannot be auto-judged can be self-rated. The review page shows due
cards with overdue days and learning state, and the dashboard shows how many cards are due today.
2. Agent Skill / CLI (no server)
git clone https://github.com/Simon-zj1/eyedot.git ~/.agents/skills/eyedot # Codex / Copilot CLI
git clone https://github.com/Simon-zj1/eyedot.git ~/.claude/skills/eyedot # Claude Code
cd ~/.agents/skills/eyedot && npm install
Then say "quiz me on this material": the agent writes exam.json following SKILL.md, and the
scripts handle verification, grading and rendering.
npx tsx scripts/study.ts verify --material examples/agent-interview-notes.md --exam exam.json --strict
npx tsx scripts/study.ts answer-template --exam exam.json --out answers.json
npx tsx scripts/study.ts grade --material examples/agent-interview-notes.md --exam exam.json \
--answers answers.json --out ./learning_work --engine auto
The output is a self-contained offline report (report.html) with no external requests or CDN assets.
Before the package is on npm, three ways to run it — most reliable first:
# 1. install once from the GitHub release (no npm account needed)
npm i -g https://github.com/Simon-zj1/eyedot/releases/download/v0.15.1/eyedot-0.15.1.tgz
eyedot --help
# 2. clone and run (if you want the source too)
git clone https://github.com/Simon-zj1/eyedot && cd eyedot && npm install
npx tsx scripts/study.ts verify --material ... --exam ...
# 3. no git: download the source tarball
curl -L https://github.com/Simon-zj1/eyedot/archive/refs/heads/main.tar.gz | tar xz
cd eyedot-main && npm install && npx tsx scripts/study.ts verify --material ... --exam ...
Change the version in the URL to install another release. npx github:Simon-zj1/eyedot ... goes over
SSH by default and fails without a working GitHub SSH key.
Requirements
Supported model providers (bring your own key)
The platform adapts the endpoint and default model; you pick a provider and paste a key.
Providers reachable from mainland China come first in the settings UI.
| Provider | Base URL (auto-filled) | Default model |
|---|
| DeepSeek | https://api.deepseek.com/v1 | deepseek-flash |
| 智谱 GLM | https://open.bigmodel.cn/api/paas/v4 | glm-4.6 |
| 通义千问 (Aliyun Bailian) | https://dashscope.aliyuncs.com/compatible-mode/v1 | qwen-plus |
| Kimi (Moonshot) | https://api.moonshot.cn/v1 | kimi-k2-turbo-preview |
| MiniMax | https://api.minimax.chat/v1 | MiniMax-Text-01 |
| SiliconFlow | https://api.siliconflow.cn/v1 | deepseek-ai/DeepSeek-V3 |
| OpenAI | https://api.openai.com/v1 | gpt-5-mini |
| Anthropic | https://api.anthropic.com/v1 | claude-haiku-4-5 |
| Google Gemini | .../v1beta/openai | gemini-2.5-flash |
| OpenRouter / Groq / xAI / Vercel Gateway | provider-specific | see settings |
| Local (Ollama / vLLM / LM Studio) | http://localhost:11434/v1 | qwen3:8b |
Settings has a Test connection button that verifies key + endpoint + model and explains common
failures (401 / 404 / 429 / network) in plain language.
Publishing to npm (maintainers)
# 1. Create a granular token with Packages: Read and write — https://www.npmjs.com/settings/~/tokens
# 2. Save it to ~/.config/eyedot/npm-token (chmod 600), then:
npm run publish:npm
The script writes the token into a temporary .npmrc and deletes it afterwards, so it never
lands in ~/.npmrc and never shows up in ps. Use npm publish --dry-run --access public first
to inspect the tarball without publishing (currently 272 files, 1.2 MB).
Deploy the web app
docs/deploy.md walks through Vercel + a hosted Postgres, environment variables, a custom subdomain,
and a pre-launch checklist. One-click button:
Compatibility
| Item | Requirement | If missing |
|---|
| Node.js | ≥ 20 | cannot run |
| Database | Postgres in production | falls back to in-memory storage (cleared on restart) |
| Sign-in email | AUTH_EMAIL_WEBHOOK_URL or RESEND_API_KEY + AUTH_EMAIL_FROM | production refuses to issue login codes |
| Grading | TYPESAFE_API_KEY (Jev) | falls back to the LLM judge, then to a lexical demo engine (clearly labelled) |
| Question generation | PLATFORM_LLM_API_KEY (OpenAI-compatible) | falls back to an offline heuristic generator |
| Material format | pasted text / Markdown, or PDF / Word (.docx) upload | scanned or image-only PDFs are not OCR'd |
Never paste secrets into chat. Put TYPESAFE_API_KEY into the gitignored .env.local (or
~/.config/eyedot/typesafe-api-key) and run npm run set:typesafe-key; the script writes it to Vercel via
stdin without printing it.
Verification
npm test # unit, integration and HTTP-level tests
npm run typecheck
npm run build
npm run eval:judge -- --enforce --consistency 5
The evaluation script reports per-point accuracy, Brier score, calibration buckets, review rate and
self-consistency, gated at ≥ 90% accuracy with monotonic calibration. The offline demo engine scores
71.4% per-point accuracy with a Brier score of 0.216 and monotonic calibration, but flags 9 of the 12
questions as needing review because its judgment strength is too low — that is the argument for using a
calibrated decision model instead of word overlap.
The benchmark's data format, reproduction commands and metric gates live in
docs/benchmark.md; npm run benchmark writes a machine-readable
docs/benchmark-results.json for leaderboards or regression diffs.
Security
Uploaded material is treated as untrusted data: instructions found inside it are never executed, and a
scanner flags prompt injection, role hijacking and script injection. The provenance contract keeps generated
content from masquerading as a quoted source. See SECURITY.md.
Known limits: scanned or image-only PDFs are not OCR'd (the app says which pages had no text), legacy .doc
files must be re-saved as .docx, uploads are capped at 4 MB because of the serverless request-body limit,
and retrieval inside a material is lexical (TF × IDF: words for Latin, bigrams for Chinese) — a paraphrase
may not be found, and the app then says so instead of guessing.
Costs shown in settings are estimates derived from public price lists, useful for spotting trends and
setting caps, not a provider invoice. The platform spend cap (default $0.50 per user per day, platform keys
only) blocks the next model call once today's estimate is used up, so it can overshoot by one request.
Registration is open: an email code is enough, and an invite code is an optional credit bonus.
Daily count quotas (material / question / judgment / ask / explain) are reserved atomically in a database transaction,
and excess question-generation quota is refunded if the generator returns fewer questions than requested.
Model access is an explicit choice: bring your own key (never capped, never charged), or spend platform
credits at 1.5× the official price — 1 credit = 1,000 tokens of value, with every debit written to an
append-only credit_ledger you can audit line by line.
There is no native app yet: the delivery surfaces are web, PWA, CLI, Agent Skill and MCP. The PWA caches
static assets only — pages are server-rendered per logged-in user, and caching them in Cache Storage
would leak data across accounts on a shared device. Offline you get an explanation page; genuinely offline
material reading would need a local database and sync. The CLI and Agent Skill run fully offline. iPad
handwriting (PencilKit → recognition → the same grading pipeline) is not implemented.
License
MIT © 2026 Simon · More at www.simon-zj.top
