Protos Benchmarks
Single GraalVM benchmarking harness for
Protos, GraalJS, and GraalPy.
The host entry point is benchmark.py; the root Makefile forwards to
it. The workload catalog is cases/catalog.json (currently 17
cases, with one source file per implementation and case). The current adapters
are Protos canonical and GraalJS/GraalPy executable-value, configured in
harness/policy/cases.json.
The old Docker, runner, PERF-specific and A/B harnesses are not part of this
working tree. Their history remains recoverable in Git.
Running on the Linux host
Podman is the container runtime. The harness never implicitly pulls or builds
an image during measurement. Image preparation is an explicit operation.
Podman must be installed on the host; the measurement JVM and toolchain run
inside the pinned runner image.
Inspect the available commands and validate the catalog without a container:
make help
make validate
For the complete command-line options:
python3 benchmark.py --help
The harness supports correctness, smoke/reference timing, optional JFR
profiling, and graph capture/analysis through the explicit commands shown by
make help. Select one case with --case, an implementation with
--language protos|js|python|all, and the Protos checkout with --product.
Outputs should be stored beneath the ignored results/local/ directory.
The harness accepts independent --protos-version, --js-version, and
--python-version selectors. The Protos selector must match the actual
checkout; GraalJS/GraalPy dependencies must be prepared explicitly with
peer-prepare before offline measurement. Correctness must pass before
timing is accepted.
Use --temperature cold|hot|both independently of --stage smoke|reference.
Cold is the first guest invocation after context initialization, not JVM
startup; hot selects the minimum observed per-call time among the steady
samples, without a convergence gate. Both phases and all raw samples are
captured together. Existing valid results are reused when the exact
implementation version, source/producer, immutable container image, Java,
host/CPU, case and measurement policy identities match, unless
remeasurement is explicitly requested.
For timing results, the harness automatically derives a readable
machine_id from the CPU model and a short hash of the real Linux host
hostname. The label is exposed in timing metadata and summaries.
The existing host/CPU identity prevents incompatible timing reuse,
while Podman container hostnames do not create artificial differences.
Graph structural measurements are not partitioned by machine_id.
See BENCHMARKING.md for the current methodology and
AGENTS.work/ for the binding measurement and reproducibility rules.
Retained evidence
results/peer-baselines/ preserves the current GraalJS and GraalPy baseline
archive and its manifest. Historical PERF and graph experiment outputs have
been removed from the working tree; their committed history remains available
in Git. Local and newly generated results are not affected by that cleanup.
License
Protos Benchmarks is licensed under the Adaptive Public License 1.0 (APL-1.0).
See LICENSE.TXT.
