Benchmarks (local only)
pnpm bench runs micro-benchmarks under bench/*.bench.ts with Vitest’s bench (tinybench). They are a
local signal, not a gate: they are not part of pnpm check and there is no CI on main (private repo).
Shared-runner timings are too noisy to threshold anyway — the projects that gate on perf (Prisma, TanStack
Router) do it with CPU-simulation services; everyone else (Node, Deno, Hono, Effect) treats benchmarks as
informational. We do the same.
pnpm bench # run everything, print the tables
pnpm bench bench/kernel # one file
pnpm bench:baseline # write bench/.baseline.json (git-ignored: numbers are machine-specific)
pnpm bench:compare # re-run and print the delta against the saved baseline
Workflow for a change that might affect performance: pnpm bench:baseline on the base commit, make the
change, pnpm bench:compare. Read the mean/p99 columns and the rme (relative margin of error): a delta
inside the rme of either run is noise.
What is measured
| File | Surface | Workloads |
|---|---|---|
kernel.bench.ts | Kernel reconciler, in-memory backends | install N-plugin dependency chains in order and reversed (every install re-settles); disable/enable a root provider under 99 dependents; 200 contributors with 0 and 5 observePoint observers; contributions() reads; dispose() undo of 200 fibers |
executors.bench.ts | NodeExecutor (worker thread) vs QuickJSExecutor (WASM) | mount + dispose; 100 tool-call round-trips agent → host → agent |
stores.bench.ts | store-sqlite, store-postgres (PGlite), store-do (fake SqlStorage) | open/close; 100 log appends + all(); 50 blob puts (dedup) + gets; collections boot + CRUD through the plugin (bootCollections from the contract test) |
judge.bench.ts | createJudgePlugin gate pipeline | boot cost (reference); a submission rejected by a static gate; a full submission through the load gate (throwaway sandbox) to mount |
host-node.bench.ts | createNodeHost over loopback | GET; POST with a 1 KiB body |
Benchmarks live in a top-level bench/ directory, outside the package layering on purpose: they import
every layer at once, which no package may do. scripts/quality.ts still applies file-length and parameter
limits to them; tests-present and layering do not apply (no package.json, app layer).
Adding one
Copy the shape of the nearest file: describe per surface, bench per workload, do all setup inside the
bench callback unless it is genuinely shared (then a top-level await plus afterAll teardown, as
host-node.bench.ts does with its server). Prefer a workload with a stated N (x100) over a single call, so the number is above the
timer’s resolution. Reuse test helpers by relative import (bootCollections, fakeSqlStorage) rather than
duplicating boot code.
Known numbers worth watching
From the first run (Apple Silicon, Node 22): installing a 100-plugin chain costs ~40× a 10-plugin chain
(settle is superlinear); a full judge submission is dominated by the load gate’s sandbox spin-up (tens
of ms) versus microseconds for the static gates; QuickJS mounts an order of magnitude faster than a worker
thread but its tool-call round-trip is slower per call. These are observations, not targets.
Vitest 4’s --outputJson/--compare are removed in Vitest 5 (replaced by an in-file bench.compare API);
bench:baseline/bench:compare will need rewriting on that upgrade.