VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation

CVPR 2026

Official implementation of VIRST, a video-instructed reasoning framework for spatiotemporal segmentation.

TODO

• release model code
• release checkpoint
• release data code
• release utility scripts
• release eval script
• release training scripts
• demo script

Overview

This repository contains the core training and evaluation code for VIRST, including:

• model definition in model/
• training entrypoints in train.py and train_stage3.py
• RVOS evaluation in eval.py
• dataset handling in data/
• utility code in utils/

Installation

git clone https://github.com/AIDASLab/VIRST
cd VIRST
conda create -n virst python=3.10 -y 
conda activate virst
pip install -r requirements.txt

The requirements use PyTorch's CUDA 12.8 wheels so the environment works on both Blackwell GPUs and older supported NVIDIA architectures. VIRST defaults to PyTorch SDPA, so FlashAttention is optional. If you have a matching CUDA toolkit with nvcc, install it only after PyTorch:

pip install flash-attn==2.7.4.post1 --no-build-isolation

If Conda's default package cache is not writable, point it at a writable directory before creating the environment:

export CONDA_PKGS_DIRS="$PWD/.conda-pkgs"

Model assets

Download the VideoChat-Flash source, the exact OpenGVLab/VideoChat-Flash-Qwen2-7B_res448 base checkpoint, the SAM2.1 Hiera Large checkpoint, and the VIRST checkpoint with:

bash scripts/setup_assets.sh

This creates the following ignored local files:

third_party/VideoChat-Flash/
checkpoints/videochat/
checkpoints/sam2.1_hiera_large.pt
checkpoints/virst_checkpoint.pt

The two VideoChat-Flash paths serve different purposes:

• third_party/VideoChat-Flash/ is a clone of the upstream VideoChat-Flash source repository.
• checkpoints/videochat/ is only the local destination name for the exact OpenGVLab/VideoChat-Flash-Qwen2-7B_res448 Hugging Face model. VIRST loads its base weights from this directory.

The setup script performs the model download equivalent to:

huggingface-cli download OpenGVLab/VideoChat-Flash-Qwen2-7B_res448 \
  --local-dir checkpoints/videochat

Checkpoint

Pretrained checkpoint: Google Drive

Dataset

• Download Ref-DAVIS, Ref-YouTube-VOS, MeViS, ReVOS
• By default, data/dataset_config.py resolves dataset paths to absolute paths under <repo>/dataset/.
• You can override the defaults with VIRST_LISA_ROOT, VIRST_RVOS_ROOT, VIRST_CHATUNIVI_ROOT, and VIRST_VQA_VIDEO_ROOT.
• Store them in the following directory

For the documented mevis_valid evaluation, download the official MeViS valid_u split from FudanCVL/MeViSv2 after installing the Python requirements:

mkdir -p dataset/RVOS_ROOT/mevis/valid_u

huggingface-cli download FudanCVL/MeViSv2 \
  --repo-type dataset \
  --include "valid_u/*" \
  --local-dir dataset/.mevis-download

tar -xf dataset/.mevis-download/valid_u/JPEGImages.tar \
  -C dataset/RVOS_ROOT/mevis/valid_u
cp dataset/.mevis-download/valid_u/mask_dict.json \
  dataset/RVOS_ROOT/mevis/valid_u/mask_dict.json
cp dataset/.mevis-download/valid_u/meta_expressions_v2.json \
  dataset/RVOS_ROOT/mevis/valid_u/meta_expressions.json

The filename mapping in the final command is required because VIRST expects the annotation file to be named meta_expressions.json.

RVOS_ROOT
├── ReVOS
│   ├── JPEGImages 
│   ├── mask_dict.json             
│   ├── mask_dict_foreground.json   
│   ├── meta_expressions_train_.json 
│   └── meta_expressions_valid_.json 
├── lvvis
│   └── train
|       ├── JPEGImages
|       ├── mask_dict.json
|       └── meta_expressions.json
├── Ref-Youtube-VOS
│   ├── meta_expressions
|   |   ├── train/meta_expressions.json
|   |   └── valid/meta_expressions.json
│   ├── train
|   |   ├── JPEGImages
|   |   └── mask_dict.pkl
│   └── valid
|       └── JPEGImages
├── davis17
│   ├── meta_expressions
|   |   ├── train/meta_expressions.json
|   |   └── valid/meta_expressions.json
│   ├── train
|   |   ├── JPEGImages
|   |   └── mask_dict.pkl
│   └── valid
|       ├── JPEGImages
|       └── mask_dict.pkl
└── mevis

Evaluation

MeViS v1

Run MeViS evaluation with:

MODEL_CHECKPOINT=checkpoints/virst_checkpoint.pt \
bash scripts/eval_mevis.sh mevis_valid

If your dataset is not stored under the default <repo>/dataset/RVOS_ROOT, set RVOS_ROOT explicitly:

MODEL_CHECKPOINT=checkpoints/virst_checkpoint.pt \
RVOS_ROOT=/path/to/RVOS_ROOT \
bash scripts/eval_mevis.sh mevis_valid

To use non-default model locations, set VIDEOCHAT_CHECKPOINT and SAM2_CHECKPOINT when invoking the evaluation script.

Supported dataset names for the script are:

• mevis_valid
• mevis_test

Note:

• Predictions are saved under ./eval_results/mevis_valid/ by default.

To compute the MeViS metric after inference:

python -m utils.evaluation.eval_rvos ./eval_results/mevis_valid/<run_name> --dataset mevis_valid

Notes

• The project page will be updated as the release is polished further.

Acknowledgements

This project builds upon prior work, including VISA, LISA, VideoChat-Flash, and SAM2.

We thank the authors for releasing their code and models.