This is the unorganized version of the code for RoboTron-Mani: All-in-One Multimodal Large Model for Robotic Manipulation.

RoboTron-Mani: All-in-One Multimodal Large Model for Robotic Manipulation
Fen Yan*, Fanfan Liu*, Liming Zheng, Yufeng Zhong, Yiyang Huang, Zechao Guan, Chenjian Feng, Lin Ma†
Email: bphengyan@163.com *Equal Contribution †Corresponding Authors

In recent years, robotics has advanced significantly through the integration of larger models and large-scale datasets. However, challenges remain in applying these models to 3D spatial interactions and managing data collection costs. To address these issues, we propose the multimodal robotic manipulation model, RoboMM, along with the comprehensive dataset, RoboData. RoboMM enhances 3D perception through camera parameters and occupancy supervision. Building on OpenFlamingo, it incorporates Modality-Isolation-Mask and multimodal decoder blocks, improving modality fusion and fine-grained perception. % , thus boosting performance in robotic manipulation tasks. RoboData offers the complete evaluation system by integrating several well-known datasets, achieving the first fusion of multi-view images, camera parameters, depth maps, and actions, and the space alignment facilitates comprehensive learning from diverse robotic datasets. Equipped with RoboData and the unified physical space, RoboMM is the first generalist policy that enables simultaneous evaluation across all tasks within multiple datasets, rather than focusing on limited selection of data or tasks. Its design significantly enhances robotic manipulation performance, increasing the average sequence length on the CALVIN from 1.7 to 3.3 and ensuring cross-embodiment capabilities, achieving state-of-the-art results across multiple datasets. The code will be released following acceptance.

• 2025.10: Excited to share that our paper has been accepted to ICCV 2025!
• 2024.12: We release RoboMM paper on arxiv！We release the training and inference code!

git clone https://github.com/RoboUniview/RoboMM.git
cd RoboMM
bash install.sh

See train.sh for the system environment configuration of different simulators.

Download the data from Data and extract it. Modify the corresponding paths in the config file and use the following files for training. If you just want to download the Calvin data, you can use the following method with the huggingface-cli download command:

huggingface-cli download --repo-type dataset --resume-download liufanfanlff/RoboData  --include "calvin*"  --local-dir RoboData
cat ./calvin_task_D_D_part_* > calvin_task_D_D.tar.gz

bash tools/train.sh 8 ${config}

bash tools/test.sh 8 ${ckpt}

• RoboMM traing code
• RoboMM inference code
• RoboMM evaluation code
• RoboMM five training data
• RoboMM nine training data
• RoboMM model

Original: https://github.com/mees/calvin License: MIT

Original: https://github.com/Farama-Foundation/Metaworld License: MIT

Original: https://github.com/Lifelong-Robot-Learning/LIBERO License: MIT

Original: https://github.com/robocasa/robocasa License: MIT

Original: https://github.com/ARISE-Initiative/robomimic License: MIT

Original: https://github.com/notFoundThisPerson/RoboCAS-v0 License: MIT

Original: https://github.com/stepjam/RLBench License: MIT

Original: https://github.com/robot-colosseum/robot-colosseum License: MIT

Original: https://github.com/haosulab/ManiSkill/tree/v0.5.3 License: Apache

Original: https://github.com/openai/CLIP License: MIT

Original: https://github.com/mlfoundations/open_flamingo License: MIT

Original: https://github.com/RoboFlamingo/RoboFlamingo License: MIT

Original: https://github.com/RoboUniview/RoboUniview License: MIT

@InProceedings{Yan_2025_ICCV,
    author    = {Yan, Feng and Liu, Fanfan and Huang, Yiyang and Guan, Zechao and Zheng, Liming and Zhong, Yufeng and Feng, Chengjian and Ma, Lin},
    title     = {RoboTron-Mani: All-in-One Multimodal Large Model for Robotic Manipulation},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2025},
    pages     = {13707-13718}
}