Skip to content

Run independent arena games in parallel - #25

Merged
JuneQQQ merged 1 commit into
mainfrom
perf/parallel-arena
May 20, 2026
Merged

Run independent arena games in parallel#25
JuneQQQ merged 1 commit into
mainfrom
perf/parallel-arena

Conversation

@JuneQQQ

@JuneQQQ JuneQQQ commented May 20, 2026

Copy link
Copy Markdown
Owner

Summary

Code-optimisation half of this maintenance cycle.

Independent arena games can now run in parallel via a thread pool. Real-LLM benchmarks spend almost all their wall-clock waiting on the provider (HTTP IO releases the GIL), so this scales close to linearly with worker count for the cases that actually matter. Aggregation is commutative — Arena._record under a single lock — so the resulting ArenaReport is bit-identical to the sequential run regardless of completion order.

Changes

  • Arena.run gains max_workers: int = 1 (keyword-only). With >1, games are dispatched onto a ThreadPoolExecutor; recording is serialised under a lock; progress hook fires per game in completion order. Default behaviour unchanged.
  • Leaderboard.run threads max_workers into the inner Arena.run calls.
  • CLI: --workers N on deepwolf arena and deepwolf leaderboard.
  • Docstring documents the thread-safety expectation: agents that share mutable state — notably MockProvider's RNG — should be instantiated per game by the factory in parallel mode. Factories that return fresh agents (the usual pattern) are safe automatically.

Why threads, not processes

For real-LLM use the bottleneck is network IO, which releases the GIL → threads scale linearly. Processes would require picklable factories (lambdas don't pickle) and would add startup overhead. The library's existing API is preserved exactly.

Checklist

  • ruff / mypy / pytest (92 tests, 3 new) all pass
  • Parallel run produces a bit-identical report to sequential (test_parallel_arena_matches_sequential_bit_for_bit)
  • Default (max_workers=1) behaviour unchanged
  • CHANGELOG updated

Real-LLM benchmarks spend almost all their wall-clock waiting on the provider:
games are independent, so dispatching them onto a thread pool scales close to
linearly with worker count. Aggregation is commutative, so the report is
bit-identical to the sequential run regardless of completion order.

- Arena.run gains `max_workers: int = 1` (keyword-only). >1 runs games through
  a ThreadPoolExecutor with a lock around _record. Default is unchanged.
- Leaderboard.run threads max_workers into the inner Arena runs.
- CLI: --workers N on `deepwolf arena` and `deepwolf leaderboard`.
- Documented the thread-safety expectation (agents that share mutable state —
  notably MockProvider's RNG — should be instantiated per game by the factory).
- 3 new arena tests: bit-for-bit match vs sequential, progress hook coverage,
  default-is-sequential.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JuneQQQ

JuneQQQ commented May 20, 2026

Copy link
Copy Markdown
Owner Author

Code review (post-merge record).

  • Determinism preserved: test_parallel_arena_matches_sequential_bit_for_bit verifies the parallel report equals the sequential one in every aggregate field (village_wins, werewolf_wins, total_days, role_stats, agent_stats). Aggregation is commutative and protected by a single lock, so completion order doesn't matter. ✅
  • Backward compatible: max_workers=1 is the default and runs the original sequential path; test_max_workers_one_uses_the_sequential_path confirms parity. ✅
  • Threads, not processes: correct call for the real-LLM use case where >99% of wall time is network IO (GIL released). Avoids the pickling pitfalls of ProcessPoolExecutor for closure factories. ✅
  • Thread-safety expectation is documented in the Arena.run docstring — factories returning fresh agents per call are safe; shared mutable state (notably MockProvider's RNG) needs per-game instantiation. ✅
  • Progress hook still fires once per game; the count is monotonic under the lock. ✅
  • Leaderboard composes cleanly via max_workers pass-through; CLI exposes --workers N on both commands.

Solid optimisation. Merged via squash.

@JuneQQQ
JuneQQQ merged commit cc7e8f4 into main May 20, 2026
4 checks passed
@JuneQQQ
JuneQQQ deleted the perf/parallel-arena branch May 20, 2026 05:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant