Benchmark

VFS-Bench measures agents that work through AgentVFS, a virtual filesystem with VFS.md: the agent reaches every service with shell commands over files, and with CLIs for actions. Each task keeps its original instruction, data and grader.

VFS-Bench 1.0 compared AgentVFS with MCP on 200 tasks from MCP-Atlas and Toolathlon; its tasks, results and trajectories are at agentvfs.org/benchmark/vfs-bench-1. VFS-Bench 2.0, in progress, runs recent, hard agent benchmarks through AgentVFS alone.

Here is what the agent is given in the VFS arm of a Toolathlon task over Google Drive, abridged. It replaces the MCP server's tool schemas:

---
name: workspace-filesystem
description: The virtual filesystem layout of this workspace.
---

# Virtual Filesystem

Google Drive is virtualized at `/gd`. Read it with shell commands; spreadsheet
operations are listed in SKILL.md.

| Path | Description |
|---|---|
| `/gd` | Google Drive, read-only. Directories are Drive folders; change spreadsheets with the `google-sheet` CLI. |

and the agent works on the data with the tools it already knows. These are commands from recorded runs of the task:

find /gd -name '*.gsheet.json'
f="/gd/NHL-B2B-Analysis/NHL 2425 Schedule.gsheet.json"
jq -r '.sheets[0].data[0].rowData[1:][] | [.values[].formattedValue] | @tsv' "$f" > s.tsv
cut -f4 s.tsv | sort | uniq -c
google-sheet create-spreadsheet '{"title":"NHL-B2B-Analysis"}'
google-sheet update-cells "$(jq -c '{spreadsheet_id:"f0003",sheet:"Sheet1",range:"A1:F33",data:.}' out.json)"

About

• Unified interface. One virtual filesystem presents every service as files and commands, and agents read them live without copying anything locally.
• Performance. Agents filter and chain data with Unix-like tools and scripts, saving more than half of MCP's tokens on average with better accuracy.
• Policy and security. The filesystem is purely virtual, so a policy checks every command and decides what agents may read, change or run, keeping secrets and tests out of reach.
• Same task, same data, same grader. Both arms get the byte-identical task text, the same seeded fixtures and the original, unmodified evaluator; services run as local fakes that reproduce their APIs, reset and reseeded before every run, so the benchmark reruns without live accounts.
• Same operations. The VFS arm's virtual CLIs offer the MCP servers' operations with the same arguments, so both interfaces reach the same actions. Every divergence from upstream is declared in one file.

AgentVFS has four parts, and VFS-Bench measures them together against MCP:

1. Virtual Filesystem. Each service's data appears as files in the shape of its API resources, read with ls, grep, cat and jq.
2. VFS.md. A short Markdown file describes the layout and the file formats, so the agent finds the data without trial and error.
3. Virtual CLIs. Actions with side effects run as CLIs that SKILL.md describes, so every change to a service is an explicit command.
4. Policy. Every read, write and command passes through a policy before it runs.

Services

The agent reaches every service through the virtual filesystem and its CLIs:

以上为 README 的开头部分。其余内容请阅读原文。 查看完整 README ↗