frontend-visualqa

Gives coding agents eyes for frontend work — visual QA and verification powered by Yutori Navigator.

What it does

• Verifies explicit visual claims against a running localhost frontend
• Captures screenshots for quick visual inspection
• Reuses browser sessions across MCP tool calls for multi-step debugging
• Works as a CLI (frontend-visualqa verify), MCP server (frontend-visualqa serve), or agent skill (/frontend-visualqa)

Does not start your dev server. If the URL is unreachable, claims return not_testable.

Why visualqa?

Playwright MCP can click, type, and assert against the DOM — but it cannot see the page. It can run cleanly on the wrong page, assert modal.isVisible() on a modal rendered off-screen, or miss a layout that broke on mobile.

Navigator is an AI agent powered by a computer-use model (n1.5) that perceives a webpage via a screenshot and predicts actions to take in a browser. Two capabilities matter here:

• Self-correcting navigation — Point the agent at the product catalog instead of a specific product page and Navigator recognizes the wrong page, clicks through to the right one, and reports trace.wrong_page_recovered: true. Playwright MCP would run assertions on the wrong page and silently pass — garbage in, garbage out.

• Rich visual evaluation — On the cart page, both items show sale prices ($149.99 and $79.99) but Navigator caught that the subtotal of $279.98 uses the original prices — the discount was never applied. On the API dashboard, the quota label reads "100%" but the progress bar is visibly only two-thirds full. Playwright MCP would pass both — the DOM text is consistent and the progress bar width is just a CSS value.

• Native <select> dropdowns — Native HTML <select> dropdown options can still be unreliable because they render as OS-level widgets outside the browser viewport. If your page uses native selects, prefer custom in-browser dropdown components for visual testing, or pre-fill the selection via URL parameters.

Install

Prerequisites

Install uv if you don't already have it:

curl -LsSf https://astral.sh/uv/install.sh | sh

Quick install (recommended)

1. Install the CLI and browser dependency:

   uv tool install frontend-visualqa
   uv tool run --with frontend-visualqa playwright install chromium
   

   This installs the frontend-visualqa CLI and downloads Chromium with Playwright resolved alongside frontend-visualqa.

   If you use uv >= 0.8.5 and want yutori and playwright exposed as top-level shims too, you can use the convenience form instead:

   uv tool install frontend-visualqa \
     --with-executables-from yutori \
     --with-executables-from playwright
   playwright install chromium
   

   The --with-executables-from flag was added in uv 0.8.5; older uv versions fail with unexpected argument '--with-executables-from'.

2. Log into Yutori API:

   uv tool run --with frontend-visualqa yutori auth login
   

   This opens your browser to save your Yutori API key to ~/.yutori/config.json. If you installed the optional top-level yutori shim above, yutori auth login works too.

   Go to platform.yutori.com and add your key to the config file:

   mkdir -p ~/.yutori
   cat > ~/.yutori/config.json << 'EOF'
   {"api_key": "yt-your-api-key"}
   EOF
   

3. Register the MCP server using add-mcp (works with all clients):

   npx add-mcp -g -n frontend-visualqa "frontend-visualqa serve"
   

   Pick the clients you want to configure.

4. Install skills using skills.sh:

   npx skills add yutori-ai/frontend-visualqa -g
   

   -g installs at user scope. Omit -g for project-local install.

   Invoke the skill with /frontend-visualqa in Claude Code and $frontend-visualqa in Codex CLI. Agents can also auto-invoke.

5. Restart your agent (Codex, Claude Code, etc) so the installs are picked up.

Manual per-client setup

Plugin (recommended) — installs MCP tools + skill together:

/plugin marketplace add yutori-ai/frontend-visualqa
/plugin install frontend-visualqa@frontend-visualqa-plugins

MCP only (if you prefer not to use the plugin):

claude mcp add --scope user frontend-visualqa -- frontend-visualqa serve

codex mcp add frontend-visualqa -- frontend-visualqa serve

Skills can be installed via npx skills add above, or with $skill-installer inside Codex:

$skill-installer install https://github.com/yutori-ai/frontend-visualqa/tree/main/.agents/skills/frontend-visualqa

Use the checked-in .mcp.json, or point your client at frontend-visualqa serve.

For development, or to run unreleased changes from a local checkout:

git clone https://github.com/yutori-ai/frontend-visualqa.git
cd frontend-visualqa
uv sync   # dev environment for tests and lint (see CONTRIBUTING.md)
uv tool install --editable .   # so the checked-in .mcp.json runs your local source
uv tool run --with-editable . playwright install chromium
uv tool run --with-editable . yutori auth login

Install Chromium with Playwright resolved through the editable tool install above, using uv tool run --with-editable . or the optional uv >= 0.8.5 playwright shim. Do not use uv run playwright install. uv tool install resolves its own dependency versions, which can differ from the pinned Playwright in .venv, and the frontend-visualqa command runs from the tool environment, so its browser must come from there too. Using uv run installs Chromium into .venv instead, leaving the CLI without a matching browser and producing a not_testable "Executable doesn't exist" error at run time.

See CONTRIBUTING.md for tests and lint.

Uninstall

/plugin uninstall frontend-visualqa@frontend-visualqa-plugins -s user

Remove the MCP server entry from ~/.codex/config.toml, then delete the skill directory:

rm -rf ~/.agents/skills/frontend-visualqa

Restart Codex after removing.

Examples

The repo includes demo pages you can use immediately — no dev server required:

# From the repo root, serve the included demo pages
cd /path/to/frontend-visualqa
lsof -ti:8000 | xargs kill 2>/dev/null; python3 -m http.server 8000 -d examples &

Self-correcting navigation — start on the wrong page and watch Navigator find its way:

# Navigator lands on the product catalog, clicks through to find the product detail page
# After each evidence screenshot, the Yutori overlay replays the last action
frontend-visualqa verify http://localhost:8000/ecommerce_store.html \
  --headed \
  --claims 'The product detail page shows Wireless Headphones Pro priced at $149.99'

Catching regressions — mix passing and failing claims:

frontend-visualqa verify http://localhost:8000/analytics_dashboard.html \
  --headed \
  --claims \
  'The API status indicator shows Active' \
  'The monthly quota progress bar is completely filled'
# → first claim passes, second fails (label says 100% but bar is ~65% full)

Catching pricing bugs — verify that discounts are actually applied:

frontend-visualqa verify 'http://localhost:8000/ecommerce_store.html#/cart' \
  --headed \
  --claims 'The displayed cart subtotal equals the sum of the visible sale prices'
# → fails: Navigator sees the sale prices sum to $229.98 while the displayed subtotal is $279.98

Autonomous form filling — Navigator fills a multi-step form, picks a date, and catches a timezone bug:

frontend-visualqa verify 'http://localhost:8000/booking_form.html' \
  --headed \
  --max-steps-per-claim 25 \
  --claims 'The date on the confirmation page matches the date selected on the calendar' \
  --navigation-hint "Fill out the form with example data (grayed text is showing example format, not filled out values)"
# → fails: Navigator fills the form, picks a date, books the slot, and catches the off-by-one on the confirmation page

--navigation-hint gives Navigator context it can't infer from pixels alone. Here, the booking form shows placeholder text like "John Doe" and "555-0123" — the model can mistake these for already-filled values and skip the form. The hint tells it that grayed text is placeholder format, not real data, so it fills every field correctly.

Login flow with visual bug detection — when only the first claim needs setup, put the hint next to that claim in a claims file (examples/login_flow_claims.md). The runner reuses the logged-in session for later claims, and only the claim that carries the hint gets the login guidance:

frontend-visualqa verify http://localhost:8000/yutori_login.html \
  --headed \
  --no-reset-between-claims \
  --max-steps-per-claim 20 \
  --claims-file examples/login_flow_claims.md
# → first two claims pass, third fails: label says "100% used" but the progress bar is ~40% filled

Use against your own frontend the same way — just swap the URL:

frontend-visualqa screenshot http://localhost:3000
frontend-visualqa verify http://localhost:3000/dashboard \
  --claims 'The revenue chart is visible without scrolling'

Navigation hint for claims that require interaction:

frontend-visualqa verify http://localhost:8000/ecommerce_store.html \
  --headed \
  --claims 'The cart badge shows 3 items' \
  --navigation-hint "Click 'Add to Cart' on the Mechanical Keyboard K7 product card."

Scrolling to find off-screen content:

frontend-visualqa verify http://localhost:8000/analytics_dashboard.html \
  --headed \
  --claims 'The /api/v1/webhooks endpoint returned a 200 OK status' \
  --navigation-hint 'Scroll down to the Recent Requests table and inspect the row for /api/v1/webhooks.'
# → fails: Navigator scrolls to the request table and finds a 500 Error

Form validation — triggering and verifying an error message:

frontend-visualqa verify http://localhost:8000/yutori_login.html \
  --headed \
  --claims 'The email field shows "Please enter a valid email address" after submitting the empty form' \
  --navigation-hint 'The grayed text in the fields is placeholder, not real input. Click the Continue button immediately without typing anything.'

Persistent session across claims — if different claims need different setup, use --claims-file with per-claim metadata or split warm-session MCP calls instead of relying on one global hint to apply selectively inside the batch.

See examples/CLAIMS.md for the full list of pages, claims, and expected results.

MCP tools

Tool	Description
verify_visual_claims	Structured pass/fail visual checks with screenshot evidence
take_screenshot	Capture current page state
manage_browser	Inspect, restart, close, resize, or open a persistent login session in the shared browser

Recommended agent workflow

1. Ensure the local frontend is running
2. If the page is auth-gated, call manage_browser with action="login" and the login page URL to open a persistent headed browser the user can sign into
3. take_screenshot to confirm page state
4. Write 1–5 concrete visual claims
5. verify_visual_claims
6. Fix code, rerun claims until they pass

CLI reference

frontend-visualqa <command> [options]

Command	Description
verify	Verify visual claims against a URL
screenshot	Capture a screenshot
login	Open a headed browser to log in and save the session
serve	Start the MCP stdio server
status	Show browser status as JSON

frontend-visualqa verify <url> --claims 'claim1' 'claim2' [options]
# or
frontend-visualqa verify <url> --claims-file claims.md [options]

Flag	Default	Description
--claims		One or more visual claims. Mutually exclusive with --claims-file; one of the two is required.
--claims-file		Read root-level Markdown bullets from a file. Annotated reports stay rerunnable by stripping task-list markers on parse.
--navigation-hint		Interaction guidance before judging
--width / --height	1280 / 800	Viewport size
--device-scale-factor	1.0	DPR
--headed	off	Show the browser (implies --visualize)
--visualize / --no-visualize	on when headed	Show in-browser action overlay (cursor, click pulses, scroll arrows)
--browser-mode	ephemeral	ephemeral or persistent
--user-data-dir		Custom profile directory
--session-key	default	Named browser session. Persistent mode supports one named session at a time.
--run-name		Optional label included in JSON output and reports
--reuse-session / --no-reuse-session	on	Reuse the named browser session if it already exists.
--max-steps-per-claim	12	Max actions per claim
--claim-timeout-seconds	120	Per-claim timeout
--run-timeout-seconds	300	Whole-run timeout
--verbose, -v	off	Stream service-responsiveness logs to stderr. -v for INFO, -vv for DEBUG.
--reset-between-claims / --no-reset-between-claims	on	Return to the base URL between claims.

Progress is written to stderr as each claim starts and completes. Final JSON still goes to stdout.

Markdown claims file example:

# Dashboard checks

- The page title reads "Analytics Dashboard"
- The API status indicator shows Active

Per-claim metadata can live under an individual claim bullet:

- After logging in, the dashboard shows "Welcome back, Developer"
  - navigation_hint: Type "[email protected]" in the email field, type "password123" in the password field, then click Continue.
- The API Calls Today stat card shows the value 1,247

frontend-visualqa verify http://localhost:8000/yutori_login.html \
  --no-reset-between-claims \
  --max-steps-per-claim 20 \
  --claims-file examples/login_flow_claims.md \
  --reporter native \
  --reporter markdown

This file-based input path is CLI-only; the MCP tool still takes an explicit list[str] of claims. For MCP and skill-driven workflows, make separate warm-session verify_visual_claims calls when the setup differs between claims instead of expecting one navigation hint to apply selectively within a batch.

Browser modes and visualization

Mode	Flag	Cookies persist?	Use case
Ephemeral (default)	—	No	Public pages, CI
Persistent	--browser-mode persistent	Yes	Auth-gated local dev

Persistent mode uses one shared Playwright profile-backed context and supports one named session at a time. Use --run-name if you only want to tag CI output, and use ephemeral mode if you need multiple simultaneous named sessions.

Log in once, reuse for all future runs:

# 1. One-time login — opens a headed browser, log in, press Enter to save
frontend-visualqa login http://localhost:3000/login

# 2. Subsequent runs reuse the saved session
frontend-visualqa verify http://localhost:3000/dashboard \
  --browser-mode persistent \
  --run-name dashboard-auth \
  --claims 'The user avatar is visible in the header'

For MCP-driven agent workflows, the equivalent bootstrap step is manage_browser(action="login", url=...), which opens the persistent headed browser and returns immediately so a human can finish authentication. When the authenticated session is no longer needed, call manage_browser(action="close", session_key=...) to close it; if login had to switch the shared browser away from its original mode, that close call restores the original configuration (provided no other sessions remain open).

Profile stored at ~/.cache/frontend-visualqa/browser-profile/ by default. Override with --user-data-dir:

frontend-visualqa login http://localhost:3000/login \
  --user-data-dir /tmp/my-project-profile

frontend-visualqa verify http://localhost:3000/dashboard \
  --browser-mode persistent \
  --user-data-dir /tmp/my-project-profile \
  --claims 'The dashboard loads without a login redirect'

When running in headed mode (--headed), the browser shows visual effects illustrating what Navigator is doing:

• post-capture cursor replays for click, scroll, drag, and typing actions
• a compact thought card when a tool-using model turn includes reasoning text

To disable it, use --no-visualize:

frontend-visualqa verify http://localhost:3000 \
  --headed --no-visualize \
  --claims 'The API status indicator shows Active'

The MCP tool verify_visual_claims accepts a per-call visualize parameter to control this independently of the server's default.

In headed mode with visualization enabled (the default; disable with --no-visualize), overlay elements fade out before every evidence screenshot (waiting for any in-progress thought card to settle, then a short opacity transition) rather than hiding instantly, and action replays are injected only after capture so no visualization appears in the screenshot sent to the model or saved to artifacts. Headless runs never render these overlays, so this fade/replay behavior does not apply to them.

Writing good claims

Claims should be observable, scoped, and provable from pixels. Prefer direct, falsifiable wording over broad interpretations like "is correct."

Good	Weak
The cart total is $261.37	The cart works correctly
The displayed subtotal equals the sum of the visible sale prices	The cart subtotal is correct
The product price shows $149.99 in monospace font	The page looks polished
At 375px width, the stat cards stack in a single column	The dashboard is responsive

If a claim requires interaction first, use --navigation-hint instead of encoding steps in the claim text.

Result statuses

Status	Meaning
passed	Claim matched the visual evidence
failed	Claim was visually false
inconclusive	Runner explored but couldn't determine confidently
not_testable	Environment blocked verification (server down, auth wall)

For the CLI, frontend-visualqa verify exits 0 only when every claim passes. It exits 1 if any claim is failed, inconclusive, or not_testable. Usage errors still exit with argparse's standard 2.

After the JSON output, verify prints a colored summary line to stderr with a checkmark or cross glyph. Colors respect NO_COLOR and FORCE_COLOR environment variables.

Reporters

Output format for persisted artifacts. Does not affect CLI stdout or MCP tool responses (always native JSON).

Reporter	File	Description
native (default)	run_result.json	Full domain-specific schema with all fields
ctrf	ctrf-report.json	CTRF standard JSON for CI/CD integration
markdown	report.md	Annotated Markdown report, either from the original claims file or synthesized from the run result

When a Markdown claims file is provided, the markdown reporter preserves the original non-claim lines and annotates only the claim bullets so the report can be fed back in as input.

Each claim result contains:

• finding — the verdict explanation (what was observed)
• proof — the decisive artifact paths, step number, and a compact extracted-text preview
• page — URL and viewport where the claim was evaluated
• trace — the execution trace: actions taken, rich events, screenshot paths, and the saved trace path

{
  "claim": "The monthly quota progress bar is completely filled",
  "status": "failed",
  "finding": "The quota label reads '100%' and '12,500 / 12,500 requests used', but the progress bar is visually only about 65% filled — the bar and the label disagree.",
  "proof": {
    "screenshot_path": "artifacts/run-.../claim-02/step-04.webp",
    "step": 4,
    "after_action": "scroll([640, 720], direction=down, amount=1)",
    "text": null,
    "text_path": null
  },
  "page": {
    "url": "http://localhost:8000/analytics_dashboard.html",
    "viewport": { "width": 1280, "height": 800, "device_scale_factor": 1.0 }
  },
  "trace": {
    "steps_taken": 4,
    "wrong_page_recovered": false,
    "screenshot_paths": ["..."],
    "actions": ["..."],
    "trace_path": "artifacts/run-.../claim-02/trace.json"
  }
}

proof.screenshot_path points to the screenshot the model was examining when it rendered the verdict. proof.text and proof.text_path are optional and are usually null unless a tool returns saved text output. trace.trace_path points to trace.json, which contains the full machine-readable event trace with reasoning and verdict metadata. Events are excluded from the JSON output by default to keep it compact; access them programmatically via result.trace.events or read trace.json directly.

frontend-visualqa verify http://localhost:3000 \
  --claims 'The checkout total matches the sum of line items' \
  --reporter native --reporter ctrf

CI / GitHub Actions

The repo includes a GitHub Actions workflow (.github/workflows/visualqa.yml) that runs visual QA checks on pull requests targeting main (and supports manual dispatch via workflow_dispatch). Use it as a template for adding frontend-visualqa to your own CI pipeline.

What the workflow does

1. Installs frontend-visualqa via uv tool install and downloads Playwright's Chromium via playwright install chromium --with-deps
2. Serves the example pages with Python's built-in HTTP server
3. Runs visual claims against the login page — element checks, form validation, post-login dashboard
4. Verifies that known visual bugs are caught (progress bar mismatch)
5. Uploads screenshot artifacts and CTRF reports for inspection

Setting up in your own repo

1. Add your Yutori API key as a secret. Go to your repo's Settings > Secrets and variables > Actions and add YUTORI_TESTING_API_KEY with your Yutori API key.

2. Copy the workflow. Adapt .github/workflows/visualqa.yml to your project — replace python3 -m http.server with your dev server start command and wait-on or a sleep until it's ready:

   - name: Start dev server
     run: |
       npm start &
       npx wait-on http://localhost:3000 --timeout 60000
   

3. Write claims for your pages. Each frontend-visualqa verify step tests a set of visual claims against a URL:

   - name: Visual QA — Dashboard
     run: |
       set -o pipefail
       frontend-visualqa verify http://localhost:3000/dashboard \
         --claims \
         'The revenue chart is visible without scrolling' \
         'The sidebar shows 5 navigation items' \
         --reporter native --reporter ctrf | tee visualqa-dashboard.json
   

4. Upload artifacts so screenshots and reports are available even when tests fail:

   - name: Upload visual QA artifacts
     if: always()
     uses: actions/upload-artifact@v7
     with:
       name: visualqa-results
       path: |
         artifacts/
         visualqa-*.json
   

Testing claims that should fail

To verify that frontend-visualqa catches known bugs, capture the exit code and validate the output contains a real failure (not a crash):

- name: Visual QA — Catch known bug
  run: |
    set -o pipefail
    exit_code=0
    frontend-visualqa verify http://localhost:3000 \
      --claims 'The progress bar matches the displayed percentage' \
      --reporter native --reporter ctrf | tee visualqa-bug.json \
      || exit_code=$?

    if [ "$exit_code" -eq 0 ]; then
      echo "UNEXPECTED: claim passed" && exit 1
    fi

    # Verify tool produced a real failed claim, not just a crash
    if [ ! -s visualqa-bug.json ]; then
      echo "ERROR: no output — tool may have crashed" && exit 1
    fi

    if ! python3 -c "
    import json, sys
    data = json.load(open('visualqa-bug.json'))
    results = data.get('results', [])
    if not results or not any(r.get('status') == 'failed' for r in results):
        print('ERROR: output has no failed claim'); sys.exit(1)
    "; then exit 1; fi

    echo "Expected failure: visual bug detected"