Lifting Unlabeled Internet-level Data for 3D Scene Understanding
CVPR 2026

Annotated 3D scene data is scarce. We build an automated data engine that lifts web videos into structured 3D supervision — instance-level point clouds, object layouts, spatial VQA, and vision-language navigation — and show through experiments that this generated data has strong potential to supplement the broad 3D scene understanding.

This is the public release of the training code and data pipeline from the paper.

huggingface-cli download bigai/SceneVersepp --repo-type dataset --local-dir ./svpp

The scripts in data_processing/ (video download, frame extraction, pose visualization) use a light-weight environment defined by requirements.txt:

conda create -n svpp python=3.10 -y
conda activate svpp
pip install -r requirements.txt

The training stacks under PQ3D/ and SpatialLM/ each have their own heavier environments. See their respective READMEs.

# Download YouTube videos referenced by each scene's data_info.json
python data_processing/download_videos.py ./svpp

# Extract raw and cropped frames into images/ and crop_images/
python data_processing/extract_images.py ./svpp

# (Optional) Visualize camera poses for one scene with Open3D
python data_processing/view_camera_poses.py ./svpp --scene-name bedroom_100_3o5KSzfdOSE

Each training stack is independent and ships with its own README.md:

• PQ3D/README.md — segmentation data generation and two-stage training
• SpatialLM/README.md — layout generation, pretraining, fine-tuning, inference, and evaluation

@inproceedings{chen2026lifting,
  title     = {Lifting Unlabeled Internet-level Data for 3D Scene Understanding},
  author    = {Chen, Yixin and Zhang, Yaowei and Yu, Huangyue and He, Junchao and Wang, Yan and Huang, Jiangyong and Shen, Hongyu and Ni, Junfeng and Wang, Shaofei and Jia, Baoxiong and Zhu, Song-Chun and Huang, Siyuan},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2026}
}

This repository builds on:

• PQ3D
• SpatialLM