Model-based 3D pose and shape estimation methods reconstruct a full 3D mesh for the human body by estimating several parameters. However, learning the abstract parameters is a highly non-linear process and suffers from image-model misalignment, leading to mediocre model performance. In contrast, 3D keypoint estimation methods combine deep CNN network with the volumetric representation to achieve pixel-level localization accuracy but may predict unrealistic body structure.
Recently, a group led by Dr Cewu Lu try to address the above issues by bridging the gap between body mesh estimation and 3D keypoint estimation. They propose a novel hybrid inverse kinematics solution (HybrIK). HybrIK directly transforms accurate 3D joints to relative body-part rotations for 3D body mesh reconstruction, via the twistand-swing decomposition. The swing rotation is analytically solved with 3D joints, and the twist rotation is derived from the visual cues through the neural network.
The group show that HybrIK preserves both the accuracy of 3D pose and the realistic body structure of the parametric human model, leading to a pixel-aligned 3D body mesh and a more accurate 3D pose than the pure 3D keypoint estimation methods. Without bells and whistles, the proposed method surpasses the state-of-the-art methods by a large margin on various 3D human pose and shape benchmarks. As an illustrative example, HybrIK outperforms all the previous methods by 13.2 mm MPJPE and 21.9 mm PVE on 3DPW dataset.
The work was accepted by CVPR 2021 and could be accessed at [2011.14672] HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation (arxiv.org). The code is available at GitHub - Jeff-sjtu/HybrIK: Official code of HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation, CVPR 2021.
About Professor Lu
Dr Cewu Lu is a Professor at Shanghai Jiao Tong University (SJTU). Before he joined SJTU, he was a research fellow at Stanford University working under Prof. Fei-Fei Li and Prof. Leonidas J. Guibas. He was a Research Assistant Professor at Hong Kong University of Science and Technology with Prof. Chi Keung Tang. He got his PhD degree from The Chinese Univeristy of Hong Kong, supervised by Prof. Jiaya Jia.
He was selected as one of the Overseas High-Level Young Introduced Talents in 2016, selected as one of China's Top 35 Under 35 Science and Technology Elite (MIT TR35) by MIT Review in 2018, awarded the Quyi Outstanding Young Scholar Award in 2019, honored with the Shanghai Science and Technology Progress Special Award ( ranked third ) in 2020, and published more than 100 articles in Nature, Nature Machine Intelligence, TPAMI, CVPR and other high-ranking journals and conferences with correspondent or first authorship. He was the program chair of CVM 2018, division chair of CVPR 2020, ICCV 2021, and IROS 2021.
Dr Lu is mainly engaged in computer vision and robotics research, and has achieved several breakthrough research results. He has published open source AI frameworks and datasets with top international level, such as Alphapose (GitHub Star 5000+), HAKE (Human Behavior Engine), and GraspNet (High Performance Robot Grasping System), a real-time human posture estimation system.
About SIAS
Shanghai Institute for Advanced Study of Zhejiang University (SIAS) is a jointly launched new institution of research and development by Shanghai Municipal Government and Zhejiang University in June, 2020. The platform represents an intersection of technology and economic development, serving as a market leading trail blazer to cultivate a novel community for innovation amongst enterprises.
SIAS is seeking top talents working on the frontiers of computational sciences who can envision and actualize a research program that will bring out new solutions to areas include, but not limited to, Artificial Intelligence, Computational Biology, Computational Engineering and Fintech.