BC-Bench

A benchmark for evaluating coding agents on real-world Business Central (AL) development tasks, inspired by SWE-Bench.

Purpose

BC-Bench provides a reproducible evaluation framework for coding agents working on real-world Business Central development tasks:

• Measure performance of different models on authentic AL issues
• Quantify impact of tooling changes (MCP servers, custom instructions, custom agents, etc)
• Track progress with transparent, comparable metrics over time
• Rapidly iterate on agent configurations and setups

Dataset

We follow the SWE-Bench schema with BC-specific adjustments:

• environment_setup_commit and version are combined into environment_setup_version
• project_paths to enumerate AL project roots touched by the fix
• problem_statement and hints_text are not included in the jsonl file but stored under problemstatement for screenshots in repro steps

Agents Under Evaluation

GitHub Copilot CLI

The GitHub Copilot CLI supports MCP servers, tools, and agent mode. It closely simulates real developers' workflow (both VS Code and Coding Agent), making it an ideal candidate for evaluating automated workflows.

Claude Code

Claude Code is Anthropic's agentic coding tool. It supports MCP servers, custom system prompts, and agent mode. BC-Bench integrates with Claude Code using the same shared configuration as Copilot.

BC PR Review

BC PR Review is the production-fidelity BC-ALAgents + BCQuality runner for the code-review category. It remains separate from the category contract so its results can be compared with GitHub Copilot CLI and Claude Code on the same dataset and scorer.

Getting Started

BC-Bench is open source, and you're welcome to fork and adapt it for your own use. We are not accepting external contributions in this repository at this time. You can run evaluations locally and replace the dataset under dataset/ with tasks from your own codebase.

Documentation map

• CONTRIBUTING.md — fork setup, repo layout, versioning, day-to-day maintainer ops
• EXPERIMENT.md — run an experiment (toggle instructions / skills / agents / MCP / model) against an existing category
• CATEGORIES.md — add a new evaluation category alongside the existing bug-fix / test-generation / code-review / nl2al

Citation

The paper and its LaTeX source describe BC-Bench's design and evaluation. If you use BC-Bench in your research, please cite:

@misc{sun2026bcbench,
	title={{BC-Bench}: Evaluating Agentic Engineering in a Domain-Specific Language for ERP},
	author={Sun, Haoran and Hansen, Klaus Marius},
	year={2026},
	eprint={2608.20851},
	archivePrefix={arXiv},
	primaryClass={cs.SE},
	doi={10.48550/arXiv.2608.20851},
	url={https://arxiv.org/abs/2608.20851}
}