People Prefer, We generate! A brand new aesthetic-oriented music-dance dataset: PopDanceSet, combined with POPDG, let’s generate the most popular dances of today!

Abstract

Generating dances that are both lifelike and well-aligned with music continues to be a challenging task in the cross-modal domain. This paper introduces PopDanceSet, the first dataset tailored to the preferences of young audiences, enabling the generation of aesthetically oriented dances. And it surpasses the AIST++ dataset in music genre diversity and the intricacy and depth of dance movements. Moreover, the proposed POPDG model within the iDDPM framework enhances dance diversity and, through the Space Augmentation Algorithm, strengthens spatial physical connections between human body joints, ensuring that increased diversity does not compromise generation quality. A streamlined Alignment Module is also designed to improve the temporal alignment between dance and music. Extensive experiments show that POPDG achieves SOTA results on two datasets. Furthermore, the paper also expands on current evaluation metrics. The dataset and code are available at https://github.com/Luke-Luo1/POPDG.

PopDanceSet

We constructed a popularity function to filter popular dance videos on the BiliBili website (The most popular video website among young people in China), thereby creating the PopDanceSet, which features dances favored by the public. Below are some rendered examples of various dance genres from PopDanceSet:

Pipeline

pipeline

The core innovative points of POPDG are threefold: First, the Space Augmentation Algorithm enhances the spatial connections between the upper body joints and the root joint in the SMPL format. Second, iDDPM maintains a balance between the quality and diversity of dance generation. Third, a streamlined alignment module is designed to improve the matching between music and dance.

Video Demo(Dedicated to Akira Toriyama, the author of Dragon Ball)

At the beginning of March this year, Akira Toriyama (鳥山 明), the author of my favorite manga 'Dragon Ball', passed away. Therefore, I used the OP and ED songs from the Dragon Ball anime as background music and generated dances as demo displays to commemorate Akira Toriyama!
background music: DAN DAN 心魅かれてく
background music: ロマンティックあげるよ
background music: 魔訶不思議アドベンチャーよ

Citation

@InProceedings{Luo_2024_CVPR,
    author    = {Luo, Zhenye and Ren, Min and Hu, Xuecai and Huang, Yongzhen and Yao, Li},
    title     = {POPDG: Popular 3D Dance Generation with PopDanceSet},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {26984-26993}
}