A classical-vs-learned benchmark for mobile autonomy. Same tasks, same metrics, same seeds — measure which paradigm wins each layer of the autonomy stack, and what it costs (compute, data, robustness).
Same map, same moving obstacles, same seed: A*+pure-pursuit (left) collides; A*+MPPI (right) weaves through.
Most robotics projects commit to one approach in isolation, and production stacks ship one paradigm. I rarely saw anyone put classical autonomy (A*, NMPC, MPPI, SLAM) and learned autonomy (RL, imitation, diffusion) on the same scenarios, with the same metrics, and measure who actually wins and what it costs. So I built that benchmark: every layer of the stack — mapping/SLAM, planning, control, and high-level decision — has at least one classical and one learned implementation behind a shared interface, so you can swap them, race them, and read the trade-offs off a leaderboard.
The scenarios scale in difficulty so the comparison stays interesting: point-to-point navigation → moving obstacles → semantic object search → and, as the capstone that exercises every layer at once, a robot-soccer arena (1 robot → 1v1 → NvN teams). Soccer is the stress test, not the point — the point is the measurement.
| Stack layer | Best classical | Best learned | What the benchmark shows |
|---|---|---|---|
| Local control (dynamic) | MPPI 0.95 @ 3.4 ms | PPO 0.65 @ 0.2 ms | classical wins reliability; learned wins latency — and the hybrid cascade matches MPPI at ~40% of its cost ⭐ |
| Safety gating | forward-sim 0.90 @ 1.4 ms | learned/uncertainty gates | an explicit model beats both learned gates |
| Mapping / SLAM | online map ≈ given map (1.00) | — | mapping is ~free; localization costs ~20% success |
| Semantic search | frontier 0.38 | semantic prior 0.62 | a prior lifts success +63%, SPL 2.2× |
| 1v1 soccer | scripted expert | self-play beats it 22-1-1 | imitation→RL→self-play surpasses the expert |
| Team soccer (2v2) | scripted roles (best) | imitation ties, MARL doesn't | beating a scripted team with RL stays open |
Every cell is reproducible (python -m ballpark bench|sweep|abtest …, 20 seeds, Wilson CIs); the full
tables and analysis are below and in docs/BENCHMARKS.md. Negative results are
reported with the same rigor as the wins.
# from the repo root
python -m pip install -e . # core + learned controllers (numpy, matplotlib, pyyaml, imageio, torch); casadi optional for NMPC
# benchmark one stack on one scenario
python -m ballpark bench --stack mppi --scenario dynamic --seeds 20
# the headline study: every controller × {static, dynamic} → a ranked leaderboard
python -m ballpark sweep --sweep controllers
# render one episode (MPPI shows its sample fan), or two stacks side-by-side
python -m ballpark demo --stack mppi --scenario dynamic --seed 0 --out out/demo.gif
python -m ballpark compare --stacks classical mppi --scenario dynamic --seed 0 --out out/compare.gif| Layer | Implementations |
|---|---|
| Simulator | Lite2D — fast, deterministic 2D diff-drive world with raycast lidar + moving obstacles; ball-physics and 1v1/NvN soccer arenas behind the same interface |
| Mapping / SLAM | A* on a given grid and online log-odds occupancy mapping + correlative-scan-matching localization (full SLAM, no privileged pose) |
| Global planner | A* on an occupancy grid (given or self-built) |
| Local control | pure-pursuit · DWA · MPPI · obstacle-aware NMPC (classical) · PPO · SAC · diffusion policy (learned) · safety-filter · shielded hybrid · portfolio cascade (hybrid) — all behind one interface |
| High-level decision | classical role agents, imitation (BC + DAgger), league self-play, semantic frontier exploration, parameter-shared / centralized-critic multi-agent RL |
| Benchmark | Seeded, optionally-parallel runner + multi-stack sweep → success / SPL / collisions / path-efficiency / latency, with Wilson confidence intervals and significance tests |
| Visualization | Episode renderer (MPPI sample fan, tier-coloured cascade) + side-by-side comparison GIFs + publication-style figures |
The full roadmap and design rationale live in docs/PROJECT_PLAN.md, and the
consolidated results in docs/BENCHMARKS.md.
Every controller, same interface, same 20 seeds:
| controller | static success | dynamic success | latency | character |
|---|---|---|---|---|
| A* + pure-pursuit | 1.00 | 0.55 | 0.05 ms | cheapest; blind to moving obstacles |
| A* + DWA | 0.95 | 0.85 | ~4–12 ms | reactive sampling |
| A* + MPPI | 1.00 | 0.95 | ~5–14 ms | best reactive, modest cost |
| A* + NMPC (CasADi) | 1.00 | 0.85 | ~28–43 ms | constraint-aware, costly |
| A* + learned PPO | 0.70 | 0.65 | 0.27 ms | very cheap, moderate reliability |
Takeaway: no controller wins everywhere; MPPI is Pareto-best across regimes. After a safety-margin hardening pass, the from-scratch PPO policy now edges pure-pursuit on dynamics (0.65 vs 0.55) at near-equal latency — but it stays well behind MPPI, capped by its single-frame observation (can't perceive obstacle motion). Temporal observation is the clear next lever. (20 seeds; nothing is special-cased — see docs/FAIRNESS.md.) Full analysis: EXP-0002 · EXP-0003 · leaderboards. RL training curve:
The same trade-off as a Pareto plot (cheaper-left, better-up):
And with uncertainty quantified — Wilson 95% intervals on the dynamic success rates. Overlapping intervals mean the difference is within noise; only large gaps are significant at 20 seeds (EXP-0016):
A hybrid recovers safety for free: wrapping the RL controller in a classical safety filter
(rl_safe) eliminates static collisions (0.20 → 0.00) without retraining — a learned policy given a
hard classical guarantee. (EXP-0006)
A new Pareto point — the shielded hybrid. Run the cheap learned controller by default, but
forward-simulate its command and substitute MPPI for that step when a collision is imminent
(model-predictive shielding). MPPI is invoked only 34% of steps
(dynamic), yet the hybrid reaches 0.90 success at 1.84 ms — most of MPPI's reliability (0.95) at
~40% of its cost. Composing classical + learned beats either alone.
(EXP-0012)
Generalize that 2-way switch into an anytime, cost-aware portfolio: order controllers cheapest → most-reliable, forward-simulate each candidate, and commit to the cheapest collision-safe one, escalating to the optimizer only when needed. The result Pareto-dominates MPPI — same success at a fraction of the cost:
| scenario | cascade | MPPI (best single) |
|---|---|---|
| static | 1.00 @ 0.99 ms | 1.00 @ 3.38 ms |
| dynamic | 0.95 @ 1.34 ms | 0.95 @ 3.41 ms |
Cheap pure-pursuit handles ~75% of steps; MPPI is escalated to only where a collision is predicted (23% → 27% as obstacles start moving). A single controller can't be both cheap and reliable — a cost-ordered portfolio behind a safety gate can. (EXP-0024)
The robot is coloured by the active tier — green = pure-pursuit, red = escalated to MPPI near hazards:
Which gate is best? The shield/cascade decide when to escalate by forward-simulating the command. I also tried a learned classifier (EXP-0017) and an uncertainty / ensemble-disagreement gate (EXP-0025). The result: explicit forward-simulation wins — it's both precise and cheap; the learned gate ties it and the uncertainty gate reaches MPPI safety but over-escalates (93–100% of steps), making it slower than MPPI itself. A rigorous, self-skeptical comparison.
The headline scenario: a robot that perceives, pursues, and dribbles a ball — reusing the entire
stack behind the same interfaces (pursuit needed zero new controllers; obs.goal is just set to the
ball). Two dribblers compete on identical seeds: a classical get-behind-and-push strategy and a
learned PPO policy.
ballpark demo --stack mppi --scenario ball_pursuit --seed 0 --out out/pursuit.gif
ballpark demo --stack dribble_classical --scenario dribble --seed 3 --out out/dribble.gif| task | classical | learned |
|---|---|---|
| pursuit (reach the ball) | 1.00 (all controllers) | 1.00 |
| dribble (ball → target zone) | 1.00, 0 collisions | 1.00 (imitation: BC + DAgger) |
The learned dribbler is distilled from the classical expert via behavior cloning + DAgger (rollout success 0.50 → 1.00 as DAgger fixes BC's distribution shift) — minutes of CPU training, matching the expert. (PPO from scratch couldn't crack dribbling in hours — when a competent expert exists, imitation is the right tool; see EXP-0004.)
Two robots, one ball, two goals — each robot reuses a dribbler to attack the opponent's goal (the opponent shows up in its lidar). Zero new agent code.
A sharp result falls out: the classical and imitation-learned dribblers tie on the solo dribble task, but in 1v1 the classical wins 38–4 (both sides, 60 matches). The learned policy was cloned from an expert that never saw an opponent, so the adversary is out-of-distribution — imitation matches the expert on-distribution but doesn't inherit its robustness off-distribution. (EXP-0005)
Decentralized multi-robot navigation/swarm (ORCA, flow-fields, CBF, GNN+PPO) is deliberately out of scope here — it's covered by the complementary project FlowSwarm.
2v2 team soccer scales it up: 4 robots with dynamic role assignment (nearest-to-ball = striker, teammate defends). The learned striker transfers to teams — the RL-striker team beats the classical team 6–2 (the same policy that won 1v1 23–4). Zero new learning — components compose via the shared interface. (EXP-0011)
Closing the loop with RL — the headline result. No expert exists for adversarial play, so we go imitation → RL fine-tuning → league self-play (train vs the classical expert and frozen snapshots of itself). The learned striker goes from losing badly to beating the same expert ~5:1:
| stage | vs the same classical expert |
|---|---|
| imitation (cloned the expert) | loses 4–38 (out of distribution — never saw an opponent) |
| RL fine-tune, fixed opponent | even, 8–8–8 (recovers) |
| league self-play | wins 23–4 (surpasses the expert) |
From 4–38 to 23–4 against the same opponent — by choosing the right method at each step: imitation to bootstrap, RL fine-tuning to adapt to the adversary, self-play to exceed it where no expert exists. That sequence is the result. (EXP-0007 · EXP-0008)
"Go to the target object" in an unknown map — the robot knows only what its sensors have revealed
(a tri-state ExploredMap) and which objects it has detected (no ground-truth peeking). The target sits
near a distinctive anchor landmark visible from afar. Two decision strategies, same agent:
| strategy | success | SPL | steps | coverage |
|---|---|---|---|---|
| nearest-frontier (blind) | 0.38 | 0.236 | 1229 | 0.58 |
| semantic prior (head to the landmark) | 0.62 | 0.523 | 764 | 0.50 |
The semantic prior gets +63% success, 2.2× SPL, −38% steps — and explores less (it beelines to the landmark instead of mapping the whole arena). The "prior" is a deliberate, clearly-labelled stand-in for an LLM/VLM ranker (swappable). (EXP-0009)
ballpark objectnav --stacks objnav_nearest objnav_semantic --seeds 24Role-assigned teams (striker + support + spread defenders, all dynamic) scale from 2v2 to 4v4. Scaling up surfaces an emergent defensive saturation: more robots crowd the goal, so goals drop and draws rise.
| size | learned-team record vs classical | goals / 8 matches |
|---|---|---|
| 2v2 | 2–2–4 | 4 |
| 3v3 | 2–1–5 | 3 |
| 4v4 | 2–0–6 | 2 |
python -m ballpark team --a dribble_classical --b soccer_rl_snap --team-size 4 --seed 1 --out out/4v4.gif
Learned cooperation (multi-agent RL capstone). A full MARL harness (parameter-shared PPO, multi-agent vec-env, learned-team controller) pits a learned team against the hand-coded roles. RL-from-scratch loses 0–10 (it chases the ball but never learns to score on a competent defense); imitation of the role-based team recovers competitive play and scales to NvN from one shared policy (2v2 0–3–9, 3v3 0–5–7, 4v4 0–1–11 — draw-dominated, it reproduces the team it cloned). The project's thesis (naive RL < structure, imitation bridges it) in the hardest multi-agent setting. RL fine-tuning degrades the learned team (EXP-0021); centralized-critic MAPPO + self-play (CTDE) only ties it (EXP-0022); a KL-to-BC anchor + opponent pool stops the self-play divergence in the training metric, but on rigorous 16-seed eval the selected checkpoint is worse (net −7) — the 8-seed in-training metric was too noisy to trust (EXP-0023). Across the arc (16-seed, net vs classical): RL-scratch −10 · BC −4 (best) · IPPO-finetune −7 · MAPPO −4 · MAPPO+KL −7. Beating a competent scripted team with multi-agent RL stays an open, compute-bound problem — a limitation I state plainly rather than paper over. (EXP-0020 · EXP-0021 · EXP-0022 · EXP-0023)
Three learned controllers share one interface alongside the classical ones — measured apples-to-apples:
| controller | point_goal | dynamic | latency | note |
|---|---|---|---|---|
| MPPI (classical) | 1.00 | 0.95 | ~3.6 ms | Pareto-best on success |
| PPO (on-policy) | 0.70 | 0.65 | 0.19 ms | ~1,000k steps |
| SAC (off-policy) | 0.65 | 0.50 | 0.20 ms | ~20× more sample-efficient |
| diffusion (distilled) | 0.75 | 0.45 | 2.3 ms | DDPM head, ex-MPPI |
SAC reaches PPO-level performance in ~1/20 the environment steps (off-policy replay); diffusion brings a modern generative policy class into the arena. Neither beats MPPI on success — they win on latency and sample efficiency. (EXP-0014 · EXP-0015)
python -m ballpark train --task nav --algo sac --scenario rl_train --steps 150000 --out models/sac_nav.pt
python -m ballpark distill --scenario rl_train --out models/diffusion_nav.pt # diffusion ex-MPPIThe robot can navigate without being handed the map — it builds a log-odds occupancy grid online from lidar and, in full SLAM, estimates its own pose (noisy odometry + correlative scan matching, never reading the true pose). This lets me quantify the cost of the one modelling shortcut I rely on elsewhere:
| navigation (point_goal) | success | SPL |
|---|---|---|
| GT-map (planner given the map) | 1.00 | 0.96 |
| built-map (maps online) | 1.00 | 0.96 |
| full SLAM (also localizes) | 0.80 | 0.76 |
Knowing the map ahead was nearly worthless (the robot discovers + replans), but localization is the hard part (drift costs ~20%). Green = true pose, blue = belief. (EXP-0013)
Interface-first: every layer is a contract with interchangeable classical and learned implementations. The benchmark, match runner, and renderer depend only on the interfaces — never on a concrete algorithm.
flowchart TB
subgraph SIM["Simulation backends (sim-agnostic core)"]
L2["Lite2D — nav"]:::s --- BALL["BallEnv — pursuit/dribble"]:::s --- SOC["SoccerEnv — 1v1"]:::s
end
SIM -->|"Observation (lidar, pose, extras)"| STACK
subgraph STACK["Autonomy stack (pluggable behind shared interfaces)"]
MAP["Mapping / SLAM given grid · online occupancy · scan-match localization"]:::c --> PLAN["Global planner A*"]:::c
PLAN --> CTRL["Local control classical: pure-pursuit · DWA · MPPI · NMPC learned: PPO · SAC · diffusion hybrid: safety-filter · shielded · cascade"]:::c
DEC["High-level decision role agents · imitation (BC+DAgger) · league self-play semantic exploration · multi-agent RL"]:::c
end
STACK -->|"Twist"| SIM
STACK --> BENCH["Benchmark · sweep · match · tournament"]:::b
BENCH --> OUT["Leaderboards · figures · GIFs"]:::o
classDef s fill:#1f6feb,stroke:#fff,color:#fff;
classDef c fill:#238636,stroke:#fff,color:#fff;
classDef b fill:#8957e5,stroke:#fff,color:#fff;
classDef o fill:#bb8009,stroke:#fff,color:#fff;
Learning is crash-proof: both PPO and imitation persist their best policy to disk during training, so an interrupted run never loses progress.
Everything is deterministic under a seed, and every figure and GIF is regenerated from code:
python -m pip install -e ".[dev]" # install (CasADi optional, for NMPC)
pytest && ruff check src tests # 104 tests, lint
bash scripts/reproduce.sh # headline leaderboards + figures end-to-end
python scripts/portfolio_figures.py # publication figures from measured results
python scripts/regenerate_media.py # every demo GIF, from current code/policiesCommitted policy weights live in models/ so the demos and benchmarks run without retraining; the training
commands that produced them are in each experiment card and in scripts/reproduce.sh.
src/ballpark/ core interfaces + registry, sims, planners, controllers, learning, agents, benchmark, viz
configs/ scenarios, stacks, and sweeps (YAML)
models/ committed policy weights (so demos/benchmarks run without retraining)
docs/BENCHMARKS.md consolidated benchmark report + evaluation methodology
docs/results/ leaderboards, publication figures, and demo GIFs
benchmarks/ versioned, append-only results history
scripts/ reproduce + figure/media generators
tests/ pytest suite
- Benchmark report — all results + methodology in one place · Project plan & roadmap
- Evaluation fairness & methodology · Contributing & development guide
- Background: state of the art · capability analysis
Designed, built, and maintained by Manas Arumalla. Released under the MIT License.
If you find it useful, a citation (see CITATION.cff) is appreciated.