Inspect AI for VS Code

VS Code extension for the Inspect framework for large language model evaluations. This extension provides support for developing evaluations using Inspect, including:

• Integrated viewer for evaluation log files
• Panel to browse, run, and debug tasks in the workspace
• Panel for editing Inspect .env file
• Panel for configuring task CLI options and args
• Commands and key-bindings for running and debugging tasks

Installation

Install from the VS Code Marketplace or OpenVSX, or search for "Inspect AI" in the Extensions view (Ctrl+Shift+X). A .vsix package is also attached to each GitHub release.

Prerequisites

• Inspect AI >= 0.3.8 installed in your active Python environment
• Scout features (see below) only activate if Inspect Scout is also installed

Log Viewer

The inspect view command is used to automatically display the log for tasks executed within the workspace (this behavior can be controlled with an option).

Task Navigation

The Tasks panel displays a listing of all the Inspect tasks within your workspace. Selecting the source file or task within the listing will open the task source code in the source editor (or Notebook viewer). You can display a tree of tasks including folders and hierarchy or a flat list of tasks sorted alphabetically.

Configuration Panel

Use the Configuration (.env) panel to edit common settings in your .env file including the model provider and name, and the log directory and level.

Task Panel

Use the Task panel to edit CLI options for a task, set task args, and run or debug a task. Values will be saved for each task and used whenever the task is run or debugged from within the Inspect VS Code extension.

Running and Debugging

The Inspect VS Code extension includes commands and keyboard shortcuts for running or debugging tasks. After the task has been completed, inspect view is used behind the scenes to provide a results pane within VS Code alongside your source code.

Use the run or debug commands to execute the current task. You can alternatively use the Ctrl+Shift+U keyboard shortcut to run a task, or the Ctrl+Shift+T keyboard shortcut to debug a task.

Note that on the Mac you should use Cmd rather than Ctrl as the prefix for all Inspect keyboard shortcuts.

Scout

When Inspect Scout is installed, the extension provides a dedicated Scout activity bar with additional features:

• Scan Viewer — Custom editor for browsing and inspecting scan results
• Scan Listing — Tree view of all available scans with refresh, delete, and reveal-in-explorer actions
• Run & Debug Scans — Execute or debug Scout scans directly from VS Code
• Scout View — Tabbed interface for navigating project configuration, transcripts, scans, and validations
• Project Configuration — Detects and validates scout.yml/scout.yaml files in your workspace
• Scan Notifications — Configurable notifications when scans complete, with a quick link to view results

Settings

Setting	Default	Description
inspect_ai.notifyEvalComplete	true	Show a notification when an evaluation completes.
inspect_ai.notifyScanComplete	true	Show a notification when a scan completes.
inspect_ai.taskListView	tree	Display task outline as a tree or list.
inspect_ai.debugSingleSample	true	Limit evaluation to one sample when debugging.
inspect_ai.debugSingleTranscript	true	Limit scanning to one transcript when debugging.
inspect_ai.useSubdirectoryEnvironments	true	Run and debug commands using subdirectory environments when present.

Contributing

See CONTRIBUTING.md for development setup and guidelines. To report a suspected security issue, please see SECURITY.md.

License

MIT © Meridian Labs