Multi-agent research synthesis — open PoC for swarm vs supervisor

Every team building multi-agent systems faces the same orchestration question and answers it from intuition, not measurement. "Supervisor is cleaner" vs "swarm is faster" gets stated as fact in a hundred conference talks without a single side-by-side benchmark anyone can reproduce. This PoC builds and measures both, on a task with a defensible ground truth.

• 1No public side-by-side benchmark of supervisor vs swarm orchestration
• 2Research synthesis tasks are usually evaluated qualitatively, not with reproducible scoring
• 3Most multi-agent libraries do not let you swap orchestration patterns without rewriting
• 4Eval datasets for synthesis tasks are scarce; this PoC ships one

A reproducible benchmark: same task (synthesise a literature review across 12 papers on a given topic), same model, same MCP tool registry, same eval rubric. Three runners — single-agent (baseline), supervisor pattern, swarm pattern — each scored on factuality, citation accuracy, coverage, and cost. Code + eval data + raw runs all open-sourced.

Every benchmark run is fully traced. Anybody can re-run the same task and see the same patterns. The raw traces from the published runs are committed alongside the code.

• Pinned model version (locked at publish time)
• Seeded sampling for reproducibility within the model's nondeterminism band
• Raw Langfuse exports committed for each runner's 30 runs
• CI script that reruns the benchmark on any contributor PR

The supervisor pattern won on every metric for this task shape — task with knowable structure, finite corpus, defensible ground truth. Swarm narrowed the gap on wall time only when the corpus exceeded what the supervisor could plan against in one call. Single-agent lost everywhere.

• 01On structured tasks with knowable scope, supervisor wins on every axis once you measure. The "swarm is always faster" claim is too coarse — it depends on the corpus shape and the planner's scope.
• 02Citation accuracy is the metric that exposes weak multi-agent systems. Coverage can be gamed; citation accuracy cannot.
• 03Caching swung cost. Three runners hitting the same MCP tool registry shared the cache on the system prefix. The supervisor pattern called the model more times but cheaper per call.
• 04A reproducible benchmark, even imperfect, is more useful to the field than a dozen unreproducible claims. Eval > intuition.

The next versions extend the benchmark to (a) code-generation tasks (different scoring rubric), (b) longer corpora that exceed supervisor planning capacity (forcing the swarm interesting), and (c) model-comparison runs across Opus / Sonnet / Haiku at each orchestration.

• Add code-generation task variant with executable test rubric
• Larger corpus runs where supervisor saturates planning
• Model-comparison runs (Opus / Sonnet / Haiku) per orchestration
• External contributors welcome — open eval data + open benchmark scripts

Working on multi-agent orchestration?

The benchmark is open-source. Use it. Run your own task variants. Drop a PR with results. If you want help applying these patterns inside your codebase, the multi-agent workflows solution is the conversion path.