🌐 Website · 📄 Paper · 🏆 Leaderboard
RoboLab is a task-based evaluation benchmark for robot manipulation policies built on NVIDIA Isaac Lab. It provides 100+ manipulation tasks with automated success detection, a server-client policy architecture, and multi-environment parallel evaluation — designed for reproducible, large-scale benchmarking of generalist robot policies in simulation.
- RoboLab-120: An initial set of 120 brand new benchmark tasks spanning pick-and-place, stacking, rearrangement, tool use, and more — each with language instructions and automated success/failure detection via composable predicates.
- Bring your own robot: Tasks are not tied to a specific robot embodiment, so you can plug in any robot compatible with IsaacLab!
- Rich Asset Libraries: See a list of objects, scenes, and curated backgrounds — everything you need to create new scenes and new tasks for your own evaluation needs.
- AI-Enabled Workflows: Generate new scenes and tasks in minutes using natural language with the /robolab-scenegen and /robolab-taskgen Claude Code skills.
- Multi-Environment Parallel Evaluation: Run multiple episodes in parallel across environments with vectorized conditionals and per-environment termination.
- Results Dashboard with Episode Videos and Cross-Experiment Analysis: A self-contained web dashboard for browsing scenes/tasks, replaying episode videos, and comparing results across experiments.
Requires uv and a system ffmpeg (used for video recording). The IsaacSim/IsaacLab stack is selected at install time via a mutually-exclusive extra — isaac50 (IsaacSim 5.0 / IsaacLab 2.2.0, default) or isaac51 (IsaacSim 5.1 / IsaacLab 2.3.2.post1). See Requirements for hardware.
sudo apt install ffmpeg
git clone <repo_url>
cd robolab
uv venv --python 3.11
source .venv/bin/activate
uv sync --extra isaac50 # IsaacSim 5.0 / IsaacLab 2.2.0 (default)
# uv sync --extra isaac51 # IsaacSim 5.1 / IsaacLab 2.3.2.post1The two stacks cannot coexist in one environment. To keep both available, install each into its own venv via UV_PROJECT_ENVIRONMENT:
UV_PROJECT_ENVIRONMENT=.venv uv sync --extra isaac50
UV_PROJECT_ENVIRONMENT=.venv-51 uv sync --extra isaac51Verify installation:
uv run pytest tests/This runs the install-verification suite end-to-end: isaaclab importable, all task definitions valid, env factory populated, one full episode runs. The suite auto-accepts the NVIDIA Omniverse EULA so the run is fully headless with no prompts. More details at Debugging → Diagnostic Scripts.
Running without activating the venv: if you don't
source .venv/bin/activate, prefix everypythoncommand withuv run(e.g.uv run pytest tests/).
EULA outside the test suite: when running other entry points (e.g.
policies/pi0_family/run.py) for the first time, setexport OMNI_KIT_ACCEPT_EULA=Yonce. Cached after first acceptance.
# Run an empty episode with random actions
python examples/run_empty.py --headless
# Playback recorded demonstration data
python examples/run_recorded.py --headless
# Toggle the gripper open/closed while holding the arm fixed (sanity-check
# the gripper action path; saves sensor + viewport video to
# output/run_gripper_toggle/<task>/)
python examples/run_gripper_toggle.py --task BananaInBowlTask --headlessReplay:
run_recorded.pyrestores the recorded initial state, replays the recorded actions open-loop, and by default replays with the exact env configuration saved next to the recording (env_cfg.json). Note that the recorded outcome is not invariant across simulator versions — contact dynamics evolve between IsaacSim/IsaacLab releases (see Requirements) — and faithful reproduction requires recording and replaying with a single env. See Replaying Recorded Episodes for the full guide, including replaying your own recordings,--env-config, and--validate-states.
RoboLab uses a server-client architecture: your model runs as a standalone server, and RoboLab connects to it via a lightweight inference client. To quickly test RoboLab, try Pi0.5 via OpenPI.
Quick run after install in the RoboLab terminal, to see it working:
cd robolab
uv run python policies/pi0_family/run.py --policy pi05 --task BananaInBowlTask --num-envs 10Use the dashboard to view the output written to your local folder.
# Run headlessly
python policies/pi0_family/run.py --policy pi05 --headless
# Run on specific tasks (these two are good for sanity checking)
python policies/pi0_family/run.py --policy pi05 --task BananaInBowlTask RubiksCubeAndBananaTask
# Run on a tag of tasks
python policies/pi0_family/run.py --policy pi05 --tag semantics
# Run 12 parallel episodes per task
python policies/pi0_family/run.py --policy pi05 --headless --num-envs 12
# Disable subtask progress tracking (on by default; drops score/reason from results)
python policies/pi0_family/run.py --policy pi05 --disable-subtask
# Resume a previous run (skips completed episodes)
python policies/pi0_family/run.py --policy pi05 --output-folder-name my_previous_runSee the full Benchmark Task Library for all 120 tasks.
"Make sure all the white mugs are upright so that the opening is facing upwards."
"Put all plastic bottles away in the bin."
"Put the orange measuring cup and the blue measuring cup outside of the plate."
A self-contained web dashboard for browsing the benchmark (scenes and tasks) and analyzing your experiment results.
uv run robolab-dashboard
# open http://localhost:8080robolab_dashboard.mp4
See docs/dashboard.md for the full feature tour, CLI flags, and the API endpoints under the hood.
Full documentation is at docs/README.md, covering:
- Objects, Scenes, Tasks — Creating and managing assets and benchmark tasks
- Robots, Cameras, Lighting, Backgrounds — Configuring simulation parameters
- Environment Registration — Combining tasks with robot/observation/action configs
- Inference Clients — A list of supported open-source models and clients
- Replaying Recorded Episodes — Playing back recorded HDF5 episodes faithfully
- Analysis and Results — Summarizing, comparing, and auditing results
- Dashboard — Interactive web viewer for benchmark, tasks, scenes, and eval results
- Subtask Checking, Conditionals, Event Tracking
| Dependency | Version |
|---|---|
| Isaac Sim | 5.0 (default) or 5.1 |
| Isaac Lab | 2.2.0 (default) or 2.3.2.post1 |
| Python | 3.11 |
| Linux | Ubuntu 22.04+ |
Note on simulator versions: IsaacSim 5.0 and 5.1 ship different PhysX builds, so contact-rich dynamics (grasping, object settling) are not invariant across the two stacks. Benchmark results may be subject to differences in simulator dynamics between versions, and are best compared against runs on the same stack. Recorded demonstrations replay most faithfully on the stack they were recorded with.
- Disk space: ~8 GB (assets account for ~7 GB)
- GPU: NVIDIA RTX GPU required. Recommend 48GB+ VRAM. See Isaac Lab's hardware requirements for recommended GPUs and VRAM.
- Speed: 30 GPU hours / 100 tasks, 1.4 it/s (assuming ~200ms inference step)
The RoboLab framework is released under the Apache License 2.0.
Third-party dependency licenses are listed in THIRD_PARTY_NOTICES.md.
@inproceedings{yang2026robolab,
author = {Xuning Yang and Rishit Dagli and Alex Zook and Hugo Hadfield and Ankit Goyal and Stan Birchfield and Fabio Ramos and Jonathan Tremblay},
title = {{RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies}},
booktitle = {Proceedings of Robotics: Science and Systems},
year = {2026},
address = {Sydney, Australia},
month = {July},
url = {https://arxiv.org/abs/2604.09860}
}See CONTRIBUTING.md for acknowledgements, issues, and how to contribute.