BABEL Benchmark
Two tools can each pass their own tests and still disagree about dates, units, identifiers, or other shared data. BABEL measures whether a method can predict and locate those composition failures.
BABEL measures whether local validation is enough when agent tools are connected into a workflow. It is method-neutral: the included structural diagnostic is the reference baseline to beat, not the only permitted approach.
Key results
Public reference numbers use the frozen dev split (514 instances). There, the structural reference diagnostic achieves macro-averaged Spearman ρ typically ≥ 0.91 across families (Track A), with family-level R² ≥ 0.86. Frontier LLMs evaluated via live API calls score negative R² under standard prompts (ρ near noise). Oracle-style prompts recover ranking for some models but not magnitude — see the checked-in leaderboard.
| Method | Type | Spearman ρ | R² | Notes |
|---|---|---|---|---|
| Structural reference (sheaf / holonomy) | Reference | ≥ 0.91 | ≥ 0.86 | Macro-averaged across families, dev split |
| Best conventional baseline (family-dependent) | Ablations | varies | ≤ ~0.97 | cycle_plain / bounded_depth_8 — see leaderboard |
| Frontier LLM (live API, standard prompts) | LLM | ≈ 0.05–0.12 | < 0 | e.g. GPT-4o / Claude Sonnet — negative R² in table |
Full tables (Track A/B/C, real-MCP Bronze+/Silver, oracle and CoT variants) live in the repo: benchmark/coherence-gym/results_canonical/LEADERBOARD.md. Official ranking uses the hidden holdout; the dev split is the public reproducibility surface.
What this check cannot tell you
Public scores are frozen dev-split reference results; official ranking uses a hidden holdout and controlled benchmark performance is not production safety.
Benchmark design
| Property | Value |
|---|---|
| Total instances | 932 (514 dev + 209 test + 209 hidden holdout) |
| Workflow families | 7 (synthetic scaling, invoice, calendar, policy, 2 real-MCP, external APIs) |
| Ground truth | Deterministic symbolic execution (mean holonomy) |
| Scale range | 5 to 50 agents per composition |
| Convention dimensions | 6 per family (date format, amount unit, ID offset, etc.) |
| Registered baselines | 10 (structural, LLM, standard eval, ablations) |
Three tracks
- Track A: Failure prediction — Can a method rank compositions by failure severity? (Spearman ρ, R²)
- Track B: Failure localization — Can a method identify which edges will fail? (Precision, recall, F1)
- Track C: Budgeted repair — Given a budget of K repairs, can a method reduce failure most? (Holonomy reduction)
Ground truth
No human annotation. No model-as-judge. Ground truth is computed by a deterministic symbolic executor that propagates random test vectors around fundamental cycles of the composition graph. The primary metric is mean holonomy: average relative error when data traverses a cycle. Zero means perfectly coherent. Nonzero means at least one convention mismatch corrupted the round-trip.
Real MCP families
Two of the seven families use actual MCP servers communicating via stdio transport — not synthetic simulations:
- real_mcp_calendar — 3 custom FastMCP servers + official Memory reference server
- real_mcp_invoice — 2 custom FastMCP servers + MarkItDown Docker MCP server + official Memory reference server
Evaluate
$ cd benchmark/coherence-gym
$ pip install -e .
$ python -m coherence_gym demo
$ python -m coherence_gym evaluate --split devTry python -m coherence_gym demo and python -m coherence_gym show <instance> for the composition-challenge on-ramp; see challenge/README.md. Official benchmark evaluation (hidden split) is maintainer-run.