A partial open-source replication of D4RT: Efficiently Reconstructing Dynamic Scenes One D4RT at a Time (Google DeepMind, CVPR 2026), built using the pretrained VGGT-1B encoder and publicly available datasets.
The original paper introduces a unified transformer architecture that jointly infers depth, 3D point tracks, and camera parameters from a single video using a novel independent query decoding mechanism. This replication reproduces the decoder and query system on top of a frozen VGGT encoder trained on public data.
- Paper: https://arxiv.org/abs/2512.08924
- Original project page: https://d4rt-paper.github.io
- Author contact: d4rt@msajjadi.com
This project implements:
- The 5-parameter query encoding system
(u, v, t_src, t_tgt, t_cam)with Fourier features, learned timestep embeddings, and local patch embeddings - The lightweight cross-attention decoder that maps each query independently into a 3D point prediction
- All auxiliary prediction heads: 2D reprojection, surface normals, visibility, motion vectors, and confidence
- The unified task interface for depth maps, point tracks, point clouds, and camera extrinsics/intrinsics recovery
- The full multi-task loss from Section 2.3 of the paper including confidence-weighted 3D point loss
- Dense tracking via Algorithm 1 from the paper
- Evaluation on TAPVid-3D and Sintel depth benchmarks
What is not replicated:
- The original D4RT encoder (proprietary, trained on internal Google DeepMind synthetic data)
- Internal training datasets not publicly available
- Training at the scale used in the paper (hundreds of A100 GPU-hours)
The encoder used here is VGGT-1B from Meta (CVPR 2025 Best Paper), frozen during training. The performance gap between this replication and the original paper reflects the contribution of the encoder and data scale, not the decoder architecture itself.
Input video (T frames)
|
v
+------------------+
| VGGT-1B | <-- frozen pretrained encoder (facebook/VGGT-1B)
| (ViT backbone) |
+------------------+
|
v
Global Scene Representation F [B, N, 768]
|
+------------------------------------------+
| |
v v
+------------------+ +---------------------+
| Query Encoder | | Query Encoder |
| (u, v, t_src, | ...Q queries... | (independent) |
| t_tgt, t_cam) | +---------------------+
+------------------+
|
v (B, Q, 768)
+----------------------------+
| Lightweight Cross- |
| Attention Decoder | <-- trained from scratch
| (4 layers, no self-attn) |
+----------------------------+
|
v
Per-query predictions:
points_3d (B, Q, 3)
points_2d (B, Q, 2)
visibility (B, Q, 1)
surface_normal(B, Q, 3)
motion_vector (B, Q, 3)
confidence (B, Q, 1)
The key design choice inherited from the paper: each query is decoded independently. There is no self-attention between queries. This enables parallel decoding of arbitrary sets of queries at inference time, which is what makes dense tracking (Algorithm 1) efficient.
d4rt-replication/
|
|-- camera_geometry.py # Pinhole projection, unprojection, Umeyama alignment
|-- attention.py # Multi-head cross-attention, transformer decoder layer
|-- query_encoder.py # Fourier features, patch embedder, timestep embeddings
|-- encoder.py # VGGT-1B wrapper, frozen forward pass
|-- decoder.py # D4RTDecoder and D4RTModel
|-- tasks.py # Unified task interface (depth, tracks, point cloud, camera)
|-- losses.py # All loss functions from Section 2.3
|-- dataset.py # Kubric dataloader (HuggingFace)
|-- augmentations.py # Temporal subsampling, random crop, color jitter
|-- train.py # Training loop, checkpointing
|-- train_config.py # Default training configuration
|-- evaluate.py # TAPVid-3D and Sintel benchmark evaluation
|-- inference.py # Standalone CLI for running inference on a video
|-- visualize.py # Point track and depth map visualizations
|-- colab_setup.py # Google Colab setup and resume cell
|-- tests/ # Pytest unit tests for all modules
|-- requirements.txt
|-- LICENSE
|-- README.md
git clone https://github.com/<your-username>/d4rt-replication.git
cd d4rt-replication
pip install -r requirements.txtpython inference.py --video_path my_video.mp4 --task depth
python inference.py --video_path my_video.mp4 --task point_track
python inference.py --video_path my_video.mp4 --task point_cloud
python inference.py --video_path my_video.mp4 --task extrinsicsOutputs are saved to ./output/ as .npy arrays and visualization images.
The first run downloads VGGT-1B weights automatically from HuggingFace (~4GB). A trained decoder checkpoint is required for meaningful results. See the Pretrained Weights section below.
Requirements:
- Python 3.10 or higher
- PyTorch 2.x
- CUDA 11.8+ recommended (CPU fallback supported for debugging)
Install all dependencies:
pip install -r requirements.txtInstall VGGT separately:
pip install git+https://github.com/facebookresearch/vggt.gitVerify setup:
python -c "from encoder import VideoEncoder; print('Encoder OK')"
python -c "from decoder import D4RTModel; print('Model OK')"The primary training dataset. Synthetic videos with ground-truth 3D point trajectories, depth, and camera parameters.
HuggingFace (recommended):
# Install datasets library
pip install datasets
# The dataset is loaded automatically by dataset.py
# To pre-download to a local directory:
python -c "
from datasets import load_dataset
ds = load_dataset('zbww/tapip3d-kubric', cache_dir='./data/kubric')
print('Downloaded:', len(ds['train']), 'training videos')
"Dataset page: https://huggingface.co/datasets/zbww/tapip3d-kubric
Alternatively, using tensorflow-datasets (original Kubric):
pip install tensorflow tensorflow-datasets
python -c "
import tensorflow_datasets as tfds
ds = tfds.load('movi_e', data_dir='gs://kubric-public/tfds')
"Kubric GitHub: https://github.com/google-research/kubric
Used for the main 3D point tracking evaluation. 4,000+ real-world videos with metric 3D point trajectories.
# Clone the tapnet repo which contains the TAPVid-3D download scripts
git clone https://github.com/google-deepmind/tapnet.git
cd tapnet
# Follow the instructions in tapnet/tapvid3d/README.md
# to download the minival split (recommended, 150 videos total)TAPVid-3D project page: https://tapvid3d.github.io
Full documentation: https://github.com/google-deepmind/tapnet/tree/main/tapvid3d
Used for static scene depth estimation evaluation.
# Download from the official MPI Sintel page
wget http://files.is.tue.mpg.de/sintel/MPI-Sintel-depth-training-20150305.zip
unzip MPI-Sintel-depth-training-20150305.zip -d ./data/sintelDataset page: http://sintel.is.tue.mpg.de/downloads
VGGT-1B encoder (required, loads automatically):
The encoder loads on first run via HuggingFace. Manual download if needed:
from huggingface_hub import hf_hub_download
hf_hub_download(repo_id="facebook/VGGT-1B", filename="model.pt", local_dir="./checkpoints/vggt")HuggingFace page: https://huggingface.co/facebook/VGGT-1B
D4RT decoder checkpoint (this replication):
A trained decoder checkpoint will be released here once training runs are complete. Check the Releases tab of this repository.
python train.py --config train_config.py --device cpu --batch_size 1 --num_epochs 2This runs a smoke test on your machine without requiring a GPU. Use this to verify the pipeline before moving to Colab.
Open colab_setup.py and copy the cell contents into a Colab notebook. The cell:
- Installs all dependencies
- Mounts Google Drive for checkpoint persistence
- Resumes from the latest checkpoint if one exists
Recommended Colab runtime: A100 (Colab Pro+) or T4 (free tier for small experiments)
Edit train_config.py to change hyperparameters:
config = {
"batch_size": 4,
"T": 8, # frames per video clip
"H": 256,
"W": 256,
"num_queries_per_video": 64,
"learning_rate": 1e-4,
"weight_decay": 1e-4,
"num_epochs": 50,
"dataset": "zbww/tapip3d-kubric",
"freeze_encoder": True, # keep VGGT frozen
"checkpoint_dir": "./checkpoints",
"val_every_n_epochs": 5,
}python train.py --resume ./checkpoints/epoch_10.ptpython evaluate.py \
--checkpoint ./checkpoints/best.pt \
--benchmark tapvid3d \
--data_path ./data/tapvid3d \
--split minival \
--results_dir ./resultspython evaluate.py \
--checkpoint ./checkpoints/best.pt \
--benchmark sintel \
--data_path ./data/sintel \
--results_dir ./resultspython evaluate.py \
--checkpoint ./checkpoints/best.pt \
--benchmark all \
--data_path ./data \
--results_dir ./resultsResults are saved to ./results/results.json and printed as a comparison table.
Results will be updated here as training runs complete. The gap between this replication and the original D4RT numbers is expected and reflects encoder and data scale differences, not the decoder architecture.
| Benchmark | Metric | D4RT (paper) | This replication |
|---|---|---|---|
| TAPVid-3D minival | delta_avg | reported in paper | TBD |
| TAPVid-3D minival | OA | reported in paper | TBD |
| Sintel depth | AbsRel | reported in paper | TBD |
| Sintel depth | delta_1.25 | reported in paper | TBD |
pip install pytest
pytest tests/ -vEach test file corresponds to a module and runs the same verification checks built into each module's __main__ block.
usage: inference.py [-h] --video_path VIDEO_PATH --task TASK
[--checkpoint CHECKPOINT] [--output_dir OUTPUT_DIR]
[--num_frames NUM_FRAMES] [--device DEVICE]
arguments:
--video_path Path to input video file (.mp4, .avi, or directory of frames)
--task One of: depth, point_track, point_cloud, extrinsics
--checkpoint Path to decoder checkpoint (default: auto-download if available)
--output_dir Directory to save outputs (default: ./output)
--num_frames Number of frames to sample from video (default: 8)
--device cuda or cpu (default: auto-detect)
Independent query decoding. The decoder uses no self-attention between queries. This is the central architectural choice in D4RT that enables the unified task interface. Any set of queries can be batched and decoded in parallel, regardless of what task they represent.
Frozen encoder. VGGT-1B is used as a frozen backbone. Fine-tuning the encoder's final layers is supported via the freeze_encoder: False config flag but is computationally expensive and not recommended without significant GPU resources.
Query sampling during training. Rather than decoding full depth maps or point clouds during training (which would be millions of queries per video), the training loop randomly samples 64 queries per video per step. This is consistent with the paper's approach and makes training tractable.
Umeyama alignment for camera recovery. Camera extrinsics between frames are recovered by decoding 3D point positions in two different coordinate frames and finding the rigid transformation between them using Umeyama's algorithm. This avoids a separate camera estimation head.
Median focal length for intrinsics. Intrinsics are recovered by the median focal length trick described in Section 2.1 of the paper: decode a grid of 3D points and compute the implied focal length from each pixel's (u, v, pz, px, py) values, then take the median.
This project replicates work by Mehdi S. M. Sajjadi and colleagues at Google DeepMind. The original paper is:
@inproceedings{sajjadi2025d4rt,
title={Efficiently Reconstructing Dynamic Scenes One D4RT at a Time},
author={Sajjadi, Mehdi S. M. and others},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2026}
}
The encoder used in this replication is VGGT by Meta AI Research:
@inproceedings{wang2025vggt,
title={VGGT: Visual Geometry Grounded Transformer},
author={Wang, Jianyuan and Chen, Minghao and Karaev, Nikita and Vedaldi, Andrea and Rupprecht, Christian and Novotny, David},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2025}
}
Training data from the Kubric pipeline:
@inproceedings{greff2022kubric,
title={Kubric: A scalable dataset generator},
author={Greff, Klaus and others},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2022}
}
Evaluation benchmark TAPVid-3D:
@article{koppula2024tapvid3d,
title={TAPVid-3D: A Benchmark for Tracking Any Point in 3D},
author={Koppula, Skanda and others},
journal={arXiv preprint arXiv:2407.05921},
year={2024}
}
This project is released under the MIT License. See LICENSE for details.
Note that VGGT-1B has its own license from Meta. Review the VGGT license at https://github.com/facebookresearch/vggt before using this replication in any commercial context.
The Kubric and TAPVid-3D datasets each carry their own licenses described in their respective repositories. This replication does not redistribute any dataset files.
Grai Rudolf
GitHub: https://github.com/grairudolf