This is the official implementation of HaWoR, a hand reconstruction model in the world coordinates:

git clone --recursive https://github.com/ThunderVVV/HaWoR.git
cd HaWoR

The code has been tested with PyTorch 1.13 and CUDA 11.7. Higher torch and cuda versions should be also compatible. It is suggested to use an anaconda environment to install the the required dependencies:

conda create --name hawor python=3.10
conda activate hawor

pip install torch==1.13.0+cu117 torchvision==0.14.0+cu117 --extra-index-url https://download.pytorch.org/whl/cu117
# Install requirements
pip install -r requirements.txt
pip install pytorch-lightning==2.2.4 --no-deps
pip install lightning-utilities torchmetrics==1.4.0

cd thirdparty/DROID-SLAM
python setup.py install

Download DROID-SLAM official weights droid.pth, put it under ./weights/external/.

Download Metric3D official weights metric_depth_vit_large_800k.pth, put it under thirdparty/Metric3D/weights.

wget https://huggingface.co/spaces/rolpotamias/WiLoR/resolve/main/pretrained_models/detector.pt -P ./weights/external/
wget https://huggingface.co/ThunderVVV/HaWoR/resolve/main/hawor/checkpoints/hawor.ckpt -P ./weights/hawor/checkpoints/
wget https://huggingface.co/ThunderVVV/HaWoR/resolve/main/hawor/checkpoints/infiller.pt -P ./weights/hawor/checkpoints/
wget https://huggingface.co/ThunderVVV/HaWoR/resolve/main/hawor/model_config.yaml -P ./weights/hawor/

It is also required to download MANO model from MANO website. Create an account by clicking Sign Up and download the models (mano_v*_*.zip). Unzip and put the hand model to the _DATA/data/mano/MANO_RIGHT.pkl and _DATA/data_left/mano_left/MANO_LEFT.pkl.

Note that MANO model falls under the MANO license.

For visualizaiton in world view, run with:

python demo.py --video_path ./example/video_0.mp4  --vis_mode world

For visualizaiton in camera view, run with:

python demo.py --video_path ./example/video_0.mp4 --vis_mode cam

The training code will be released soon.

Get Hot3DAria_download_urls.json and Hot3DAssets_download_urls.json from hot3d website and put them under hot3d/data_downloader/.

Download a copy of MANO offical website model(mano_v1_2.zip) and put them to hot3d/mano_v1_2

cd hot3d/data_downloader
python3 dataset_downloader_base_main.py -c Hot3DAssets_download_urls.json -o ../dataset --sequence_name all
python3 dataset_downloader_base_main.py -c Hot3DAria_download_urls.json -o ../dataset --data_types all --sequence_name P0001_a68492d5 P0001_9b6feab7 P0014_8254f925 P0011_76ea6d47 P0014_84ea2dcc P0001_8d136980 P0012_476bae57 P0012_130a66e1 P0014_24cb3bf0 P0010_1c9fe708 P0002_2ea9af5b P0011_11475e24 P0010_0ecbf39f P0010_160e551c P0015_42b8b389 P0012_915e71c6 P0002_65085bfc P0011_47878e48 P0011_cee8fe4f P0002_016222d1 P0012_d85e10f6 P0012_119de519 P0010_41c4c626 P0012_f7e3880b P0009_02511c2f P0011_72efb935 P0010_924e574e 

*: Downloading and processing code under hot3d/ is adapted from Official HOT3D Toolkit.

mkdir datasets
cd hot3d
python export_gt.py
mv hot3d_dataset_export ../datasets/hot3d_valset_export

python lib/datasets/hot3d_dataset_preprocess.py --video_root datasets/hot3d_valset_export --set_file val.json --for_eval

Run hand motion estimation:

python scripts/scripts_eval/eval_hawor_hot3d.py --inference_stage --gen_hand_mask

Then run SLAM stage:

python scripts/scripts_eval/test_mdslam_hot3d.py

Evaluation:

python scripts/scripts_eval/eval_hawor_hot3d.py --eval_stage

DexYCB evaluation code (not cleaned) is available at https://github.com/ThunderVVV/dex-ycb-toolkit .

Parts of the code are taken or adapted from the following repos:

• HaMeR
• WiLoR
• SLAHMR
• TRAM
• CMIB

HaWoR models fall under the CC-BY-NC--ND License. This repository depends also on MANO Model, which are fall under their own licenses. By using this repository, you must also comply with the terms of these external licenses.

If you find HaWoR useful for your research, please consider citing our paper:

@article{zhang2025hawor,
      title={HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos},
      author={Zhang, Jinglei and Deng, Jiankang and Ma, Chao and Potamias, Rolandos Alexandros},
      journal={arXiv preprint arXiv:2501.02973},
      year={2025}
    }