A System Towards Generalisable Grasping

An ego-video-guided robotic grasping research system that separates perception, planning, and action. Instead of training a monolithic vision-language-action policy, the system extracts useful affordance signals from human demonstrations and uses them to guide robot grasp selection.

The system combines saved object analysis, human-prior extraction, candidate re-ranking, visualization, and robot execution. Vector OS Nano provides the robot-agent and skill-orchestration foundation.

Research codebase: model weights, datasets, robot-specific checkpoints, grasp-backend source, and selected third-party simulation meshes are distributed separately because of licensing, size, and hardware-specific setup requirements.

System Architecture

The system uses three explicit layers so that perception errors, prior-transfer errors, and execution failures can be inspected independently.

Ego video + RGB-D observation
  -> hand and object reconstruction
  -> contact event and descriptor extraction
  -> object-centric human grasp prior
  -> candidate scoring and re-ranking
  -> camera-to-robot alignment
  -> motion planning and execution

System in Action

The project was evaluated through both software visualization and physical robot experiments. The GIF previews below play directly on the repository page.

Full perception-to-action stack	Plate grasp	Earphone grasp

Ego-Video Affordance Pipeline

The key design decision is to use ego video as a signal, not as the complete control policy. Human demonstrations provide the contacted region, approach direction, and object-relative grasp geometry.

Hand and object reconstruction

Contact frames combine reconstructed hand motion with an object mask, monocular depth, point cloud, and object-oriented bounding box.

Raw object view	Object geometry and contact analysis

Human-prior re-ranking

Candidates are scored using contact proximity, approach alignment, object axes, geometric face proximity, and the underlying candidate confidence. The human prior changes the ordering while leaving candidate generation behind a replaceable interface.

Candidate ranking	Human-prior-guided ranking

Engineering Highlights

• Uses the Vector OS Nano robot-agent and skill architecture to connect perception, planning, and execution.
• Built a reusable adapter from Object Analysis results.npy to a compact human grasp prior.
• Added object-centric OBB alignment and named-region handling for scissors and an eye-drop bottle.
• Added raw-versus-prior candidate visualization, point-cloud diagnostics, and dry-run command generation.
• Structured third-party integrations as explicit dependency boundaries or overlays instead of committing weights and vendor code.
• Added repository validation, environment-independent scripts, and license attribution.

Repository Map

object_analysis/                 object segmentation, depth, point cloud, OBB, contact cues
object_pose/human_prior/         prior representation, extraction, and candidate scoring
object_pose/main.py              robot-frame alignment and optional backend integration
grasping_pipeline/pipeline.py    Object Analysis -> prior -> grasp-stage orchestration
grasping_pipeline/vector_os_nano Vector OS Nano framework and project-specific skill
overlays/vitra/                  project modifications applied to a local VITRA checkout
scripts/                         reproducible entry points and repository validation
docs/media/                      selected project images and animated demos

Inspect the Pipeline Without Models

The dry-run path shows the exact analysis-to-grasp command chain without loading CUDA models or moving a robot:

powershell -NoProfile -ExecutionPolicy Bypass `
  -File scripts\run_grasping_pipeline.ps1 `
  -Results data\results\santen\results.npy `
  -PriorOutput object_pose\priors\bottle_pipeline `
  -Object bottle `
  -Region body `
  -Mode preview `
  -DryRun

Core command construction tests use only the Python standard library:

python -m unittest discover -s tests -p test_grasping_pipeline.py -v

Current Limitations

• Monocular depth and object geometry remain sensitive to camera viewpoint.
• Transfer to unseen objects depends on segmentation and object-frame stability.
• A human approach pose may be kinematically unreachable for the robot.
• This repository does not include a grasp-candidate backend or its model assets.

Attribution and License

The included Vector OS Nano code comes from VectorRobotics/vector-os-nano and is licensed under Apache-2.0. Upstream copyright and SPDX headers are retained. Repository-specific changes are listed in MODIFICATIONS.md, and required attribution is preserved in NOTICE.

This repository does not redistribute VITRA, SAM3, grasp-backend source, model weights, datasets, or checkpoints. See THIRD_PARTY.md for integration boundaries.