MoCA3D: Monocular 3D Bounding Box Prediction in the Image Plane

Abstract

Monocular 3D object understanding has largely been cast as a 2D RoI-to-3D box lifting problem. However, emerging downstream applications require image-plane geometry, such as projected 3D box corners, which cannot be easily obtained without known camera intrinsics. This is a fundamental limitation for object understanding in the wild, where camera metadata is often unavailable or unreliable.

We introduce MoCA3D, a monocular, class-agnostic 3D model that predicts projected 3D bounding box corners and per-corner depths without requiring camera intrinsics at inference time. MoCA3D formulates pixel-space localization and depth assignment as dense prediction through corner heatmaps and depth maps.

To evaluate image-plane geometric fidelity, we propose Pixel-Aligned Geometry (PAG), which directly measures projected-corner and depth consistency. Extensive experiments show that MoCA3D improves image-plane corner PAG by 22.8% while remaining competitive on 3D IoU and using up to 57× fewer trainable parameters than prior lifting approaches.

Qualitative Results

Qualitative comparison between input boxes, MoCA3D predictions, and DetAny3D predictions. — MoCA3D predictions (red) remain aligned with the image across KITTI, Omni3D, and Hypersim.

Motivation

No camera intrinsics

Predict projected 3D geometry from only an RGB image and a 2D box.

Dense supervision

Learn corner heatmaps and depth maps instead of a sparse regression vector.

Directly usable geometry

Output pixel-aligned corner-depth pairs for downstream image-space tasks.

Conventional monocular 3D methods predict a cuboid in camera coordinates and then project it back to the image. This requires camera intrinsics, which are often missing or unreliable in real-world images. Direct corner regression avoids calibration, but provides only sparse supervision.

MoCA3D resolves both limitations by predicting eight projected corners and their depths as dense heatmaps and depth maps. The resulting representation stays aligned with image pixels and can be used directly for controllable generation, image editing, robotics, and spatial control.

Method

A box-conditioned transformer turns frozen image features into eight corner heatmaps and corresponding depth maps.

MoCA3D architecture with a DINOv3 backbone, box-conditioned 3D Geometry Transformer, and dense output heads. — MoCA3D architecture. A frozen DINOv3 backbone provides geometry-aware features, the 2D box conditions dense image tokens, and lightweight heads predict projected corners and depths.

1. Image features

A frozen DINOv3 backbone extracts geometry-aware features from the complete image rather than an isolated object crop.

2. Box conditioning

A relative-coordinate box prior conditions the encoder, while box embeddings provide keys and values to the decoder so every image token can reason about the target instance.

3. Dense prediction

Lightweight upsampling layers feed corner and depth heads that recover eight pixel-aligned corner-depth pairs.

Comparison of corner heatmaps with peak-weighted and uniform supervision. — Peak-weighted supervision concentrates gradients near the target corners and produces sharper, more reliable heatmaps.

Why dense prediction?

Each of the eight heatmaps is converted into a continuous corner coordinate using differentiable soft-argmax. Peak-weighted supervision emphasizes the region around the ground-truth mode, improving sub-pixel localization and stabilizing coordinate refinement.

The predicted heatmaps also act as confidence maps for per-corner depth supervision. Depth is sampled at the recovered corner location, keeping localization and depth assignment aligned by construction.

Experiments

We evaluate image-plane and 3D-space geometry under a shared oracle-2D-box protocol across six domains.

22.8%

better corner PAG_uv

19.0M

trainable parameters

57×

fewer trainable parameters

Pixel-Aligned Geometry

PAG_uv measures projected-corner error in pixels, while PAG_d measures relative depth error at the eight corners. These metrics expose reprojection failures that may be obscured by conventional 3D box metrics.

MoCA3D achieves the best PAG_uv across all six evaluated domains and ranks second overall on PAG_d. It remains competitive on NHD and IoU_3D despite not explicitly regressing a camera-space cuboid.

The model uses 19.0M trainable parameters, compared with 1.08B for DetAny3D, while offering a favorable inference-time and image-plane accuracy trade-off.

Trainable parameter count and efficiency versus performance for MoCA3D and competing methods. — Trainable parameter count and efficiency-performance comparison. Lower PAG_uv is better.

Downstream Application

Pixel-aligned projected geometry enables controllable generation without test-time camera calibration.

MoCA3D outputs projected corners in the coordinate system directly consumed by layout- and geometry-conditioned generation models. This avoids reconstructing camera calibration or repeatedly converting between image and camera coordinates.

In driving-scene generation, the recovered boxes preserve perspective and depth ordering while allowing appearance, lighting, and object class to be changed independently.

Driving-scene generation guided by projected corners predicted by MoCA3D. — Driving-scene variations guided by the projected box corners recovered by MoCA3D.

BibTeX

@article{jeon2026moca3d,
  title={MoCA3D: Monocular 3D Bounding Box Prediction in the Image Plane},
  author={Jeon, Changwoo and Upadhyay, Rishi and Kadambi, Achuta},
  journal={arXiv preprint arXiv:2603.19538},
  year={2026}
}