Recovering 6-DoF camera pose from a single RGB photograph against a prebuilt 3D point cloud, without keypoint detection, with a focus on indoor-to-outdoor generalization.
Undergraduate research project carried out under the SURGE program at IIT Kanpur (2026), supervised by Prof. Tushar Sandhan.
Given one photograph and a prebuilt point cloud of the same scene, the goal is to recover where the camera was: its position and orientation, six degrees of freedom. The difficulty is that pixels and 3D points are different modalities, so classical keypoint detectors do not transfer across them. Recent methods instead learn cross-modal correspondences directly, an approach known as detection-free registration.
This project has two parts:
-
It tests whether a state-of-the-art detection-free model (UniCorrn, CVPR 2026), trained on indoor scenes, transfers to a large outdoor scene. It does not. On an outdoor scan of about 79 million points, its learned cross-modal matching produces only 3.5 to 4% inliers and physically impossible camera poses on a fixed camera. The data and projection pipeline themselves are sound, confirmed independently by manual-correspondence PnP; the failure is specifically the learned cross-modal matching going out of distribution.
-
It provides a working alternative, called render-and-match, that sidesteps that failure and recovers correct, consistent poses on the same outdoor data.
This repository contains the render-and-match pipeline and its supporting tooling. As context, the broader project also reproduced 2D3D-MATR (ICCV 2023) and UniCorrn (CVPR 2026) on their published indoor benchmarks before extending to this outdoor setting.
The key idea is that the hard step, matching a 2D photograph to 3D points, can be replaced by an easy step: matching two images of the same kind.
- Render the textured 3D mesh from an approximate seed pose, producing a synthetic photograph together with its depth buffer.
- Match the real photograph against the rendered one using RoMa, a robust dense 2D-2D matcher. Same-modality matching is a far easier problem than cross-modal matching.
- For each matched pixel in the render, read its exact 3D location from the render's depth buffer. This turns the 2D-2D matches into thousands of dense, sub-pixel, depth-exact 2D-3D correspondences spread across the full frame.
- Solve the camera pose with classical PnP-RANSAC, where RANSAC and the matcher's confidence scores reject foliage and a person moving through the scene.
The entire improvement over a manual baseline lives in the correspondences: thousands of dense automatic matches rather than a handful of hand-picked, near-coplanar points. PnP and RANSAC are classical geometry, not learned components.
Implementation note: the portrait photograph (4284 x 5712) is isotropically center-cropped and resized to 1280 for matching, with the mesh rendered at the same resolution and a correspondingly scaled intrinsic matrix. The recovered pose is resolution-independent, and the final overlays use the full-resolution intrinsics.
Two fixed-camera sequences were captured outdoors, Seq A and Seq B, each from a single tripod position. Because the camera does not move within a sequence, one pose is recovered per sequence and reused across all of its frames.
| Seq A | Seq B | |
|---|---|---|
| Reprojection on inliers | 2.02 px | 1.79 px |
| Inlier correspondences | 904 | 536 |
| Recovered camera centre (m) | (-8.278, 1.858, 1.084) | (-8.473, -1.752, 1.029) |
| Convergence | clean | noisier (more foliage) |
The camera was positioned by hand using a walking metric-reference protocol, so there is no ground-truth pose. Verification is therefore by consistency rather than by comparison to a known pose:
- Independent overlay. Projecting an independent point cloud (not the one used for matching) with the recovered pose lands crisply on the real building, veranda, and tree. A pose that were self-consistent but wrong would not do this.
- Physically sane geometry. Both cameras sit about 1.0 to 1.1 m above the ground and roughly 30 m from the building facade.
- Cross-sequence consistency. The two recovered cameras share the same distance from the building and the same height, differing only in lateral position, with about a 3.6 m baseline between them, exactly as two real side-by-side setups should.
- High inlier rate. Render-and-match yields an abundant set of usable correspondences (about 71% inliers on Seq A), in contrast to the indoor-trained cross-modal model, which collapsed to single-digit inlier rates on the same scene.
Honest framing of the numbers: the roughly 2 px figure is reprojection self-consistency on the photo-to-render correspondences, not registration accuracy against ground truth, of which there is none by design. The real evidence is qualitative, namely the projection landing on independent geometry and the agreement between the two sequences. Seq B is noisier than Seq A because its target line passes more foliage, leaving fewer matches. Far-field sharpness is bounded by the resolution of the mesh texture. This work does not claim to have repaired or improved UniCorrn; the negative out-of-distribution result stands, and render-and-match is a separate route around it.
render_and_match.py— the main pipeline. Renders the mesh, matches with RoMa, lifts depth into 3D, solves PnP-RANSAC, and writes the recovered pose. Usage:python render_and_match.py seqA.rmatch_overlay.py— generates before-and-after projection overlays for a sequence.gate_render.py— preflight check that the mesh textures load and a test frame renders with real texture. This matters because the pipeline degrades silently to noise if textures are missing rather than throwing an error.stage0_seed_view.py— renders a full-resolution view from the seed pose with its depth buffer and pixel-picking grids.crop_zoom.py— zoom and grid helper for manually picking pixels.stage0_solve.py— manual-correspondence PnP solver and overlay. This is the fallback baseline that also seeds the main pipeline.stage0_overlay.py— dense before-and-after overlay for the manual baseline.
Not included: the point cloud, the textured mesh, the field photographs, and model weights. See Data and Setup below.
The pipeline expects:
- Python 3.11 with PyTorch (CUDA build)
- RoMa (used here as RoMaV2) for dense 2D-2D matching, available on the Python path
- trimesh and pyrender, with an EGL backend for headless rendering
- OpenCV (
cv2), NumPy, and Pillow (withpillow-heiffor HEIC photographs)
Example environment activation and variables (adjust the paths to your machine):
conda activate rmatch
export PYTHONPATH=/path/to/RoMaV2/src
export PYOPENGL_PLATFORM=egl
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:TrueThe UniCorrn weights, used for the out-of-distribution comparison rather than by render-and-match itself, are available on Hugging Face at prajnan/unicorrn.
# verify the mesh textures load and a test frame renders correctly
python gate_render.py
# recover the camera pose for each sequence (about 30 s after model load)
python render_and_match.py seqA
python render_and_match.py seqB
# generate before-and-after projection overlays
python rmatch_overlay.py seqA
python rmatch_overlay.py seqBThe 3D data is a Matterport MatterPak scan of the outdoor scene (a point cloud of about 79 million points plus a textured mesh), and the imagery is two 31-frame fixed-camera sequences captured with a walking metric reference. This data belongs to the host laboratory and is not redistributed here. It can be made available on request for legitimate reproduction.
This project builds directly on the work below.
- RoMa: Robust Dense Feature Matching. Johan Edstedt, Qiyu Sun, Georg Bökman, Mårten Wadenbäck, Michael Felsberg. CVPR 2024, pp. 19790-19800. arXiv:2305.15404. Code: https://github.com/Parskatt/RoMa The robust same-modality dense matcher (used here as RoMaV2) at the heart of render-and-match.
- UniCorrn: Unified Correspondence Transformer Across 2D and 3D. Prajnan Goswami, Tianye Ding, Feng Liu, Huaizu Jiang. CVPR 2026. arXiv:2605.04044. Project: https://neu-vi.github.io/UniCorrn/ The detection-free unified correspondence model evaluated for indoor-to-outdoor transfer. Pretrained weights: huggingface.co/prajnan/unicorrn
- 2D3D-MATR: 2D-3D Matching Transformer for Detection-free Registration between Images and Point Clouds. Minhao Li, Zheng Qin, Zhirui Gao, Renjiao Yi, Chenyang Zhu, Yulan Guo, Kai Xu. ICCV 2023, pp. 14082-14092. arXiv:2308.05667. Code: https://github.com/minhaolee/2D3DMATR
Carried out as part of the Summer Undergraduate Research Grant Experience (SURGE) at the Indian Institute of Technology Kanpur, under the supervision of Prof. Tushar Sandhan.
Released under the MIT License. See the LICENSE file.