Viteval

[!WARNING] V1 is a release candidate. Install viteval@rc. Untagged installs resolve to v0. V0 support ends when v1 launches. The v0 branch contains the stable release.

Features

Viteval is an independent open-source project. It is not affiliated with, endorsed by, or supported by VoidZero.

• Viteval-owned configuration and CLI, with Vitest-powered execution, watch mode, retries, and CI exit codes.
• Typed tasks, reusable datasets, built-in scorers, and custom scoring functions.
• JSON storage by default, with atomic writes. Inspect saved results locally with viteval ui.
• Optional SQLite, Braintrust, Langfuse, and hosted VoltOps providers for datasets and evaluation results, plus S3/GCS dataset storage. Backend SDKs and database drivers are opt-in.
• CLI commands for project setup, dataset generation, and the dashboard.

Installation

pnpm add -D viteval@rc
pnpm exec viteval init --no-env-file
pnpm exec viteval run
pnpm exec viteval ui

Requires Node >=26. Initialization creates viteval.config.ts (.mts for CommonJS) and a passing src/viteval.eval.ts example. JSON results are saved under .viteval/results. Existing Viteval configuration is preserved; no runner config, setup module, or reporter is required.

Usage

Define an evaluation in src/uppercase.eval.ts. This example runs without a model or API key:

import { evaluate, scorers } from 'viteval';

evaluate('Uppercase', {
  data: [
    { input: 'hello', expected: 'HELLO' },
    { input: 'world', expected: 'WORLD' },
  ],
  task: ({ input }) => input.toUpperCase(),
  scorers: [scorers.exactMatch()],
});

Run it locally or in CI:

pnpm exec viteval run

Replace task with your model, agent, or workflow. Each case passes when its configured mean, median, or sum meets the threshold. Failed execution and scorer errors fail the case and remain inspectable in the JSON report.

Development

pnpm validate
pnpm validate:consumer
pnpm --filter @viteval/cli test:binary
pnpm docs:types
pnpm docs:build

The consumer check installs packed packages outside the workspace and verifies initialization, evaluations, JSON datasets, SQLite, server-rendered dashboard deep links, JavaScript/CSS delivery, and API/404 behavior. Run pnpm dev:ui examples/basic to develop the dashboard with Vite on port 6274. See v1 completion for the release contract and evidence.