MiShape: Accurate Human Silhouettes and Body Joints from Commodity Millimeter-Wave Devices
Aakriti Adhikari, Hem Regmi, Srihari Nelakuditi, Sanjib Sur
Computer Science and Engineering
University of South Carolina
MiShape is a millimeter-wave wireless signal based imaging system that generates high-resolution human silhouettes and predicts 3D locations of body joints.
Overview
We propose MiShape, a millimeter-wave (mmWave) wireless signal based imaging system that generates high-resolution human silhouettes and predicts 3D locations of body joints. The system can capture human motions in real-time under low light and low-visibility conditions. Unlike existing vision-based motion capture systems, MiShape is privacy non invasive and can generalize to a wide range of motion tracking applications at-home. To overcome the challenges with low-resolution, specularity, and aliasing in images from Commercial-Off-The-Shelf (COTS) mmWave systems, MiShape designs deep learning models based on conditional Generative Adversarial Networks and incorporates the rules of human biomechanics. We have customized MiShape for gait monitoring, but the model is well adaptive to any tracking applications with limited fine-tuning samples. We experimentally evaluate MiShape with real data collected from a COTS mmWave system for 10 volunteers, with diverse ages, gender, height, and somatotype, performing different poses. Our experimental results demonstrate that MiShape delivers high-resolution silhouettes and accurate body poses on par with an existing vision-based system, and unlocks the potential of mmWave systems, such as 5G home wireless routers, for privacy-noninvasive healthcare applications.
Publications
- MiShape: Accurate Human Silhouettes and Body Joints from Commodity Millimeter-Wave Devices
Aakriti Adhikari, Hem Regmi, Sanjib Sur, Srihari Nelakuditi
IMWUT’22 Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Atlanta, USA, September 2022. [Paper] [Slides] [Talk] - MilliFit: A Millimeter-Wave Wireless Sensing Based At-Home Exercise Classification
Edward Sitar, Sanjib Sur
MSN’22 IEEE International Conference on Mobility, Sensing and Networking, Guangzhou, China, December 2022. [Paper] [Slide] [Talk] - A Millimeter-Wave Wireless Sensing Approach for At-Home Exercise Recognition
Edward Sitar, Moh Sabbir Saadat, Sanjib Sur
MobiSys’22 Proceedings of the ACM International Conference on Mobile Systems, Applications, and Services, Portland, Oregon, June 2022. [Paper] [Poster] - MilliPose: Facilitating Full Body Silhouette Imaging from Millimeter-Wave Device
Aakriti Adhikari, Sanjib Sur
UbiComp-ISWC’21 ACM International Joint Conference on Pervasive and Ubiquitous Computing and ACM International Symposium on Wearable Computers, Virtual, September 2021. [Paper] [Poster]