MiShape: Accurate Human Silhouettes and Body Joints from Commodity Millimeter-Wave Devices

Aakriti Adhikari, Hem Regmi, Sanjib Sur
Computer Science and Engineering
University of South Carolina

MiShape is a millimeter-wave wireless signal based imaging system that generates high-resolution human silhouettes and predicts 3D locations of body joints.

Overview

We propose MiShape, a millimeter-wave (mmWave) wireless signal based imaging system that generates high-resolution human silhouettes and predicts 3D locations of body joints. The system can capture human motions in real-time under low light and low-visibility conditions. Unlike existing vision-based motion capture systems, MiShape is privacy non invasive and can generalize to a wide range of motion tracking applications at-home. To overcome the challenges with low-resolution, specularity, and aliasing in images from Commercial-Off-The-Shelf (COTS) mmWave systems, MiShape designs deep learning models based on conditional Generative Adversarial Networks and incorporates the rules of human biomechanics. We have customized MiShape for gait monitoring, but the model is well adaptive to any tracking applications with limited fine-tuning samples. We experimentally evaluate MiShape with real data collected from a COTS mmWave system for 10 volunteers, with diverse ages, gender, height, and somatotype, performing different poses. Our experimental results demonstrate that MiShape delivers high-resolution silhouettes and accurate body poses on par with an existing vision-based system, and unlocks the potential of mmWave systems, such as 5G home wireless routers, for privacy-noninvasive healthcare applications.

Publications

  • MiShape: Accurate Human Silhouettes and Body Joints from Commodity Millimeter-Wave Devices
    Aakriti Adhikari, Hem Regmi, Sanjib Sur, Srihari Nelakuditi
    IMWUT’22 Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Atlanta, USA, September 2022.  [Paper] [Slides] [Talk]
  • A Millimeter-Wave Wireless Sensing Approach for At-Home Exercise Recognition
    Edward Sitar, Moh Sabbir Saadat, Sanjib Sur
    MobiSys’22 Proceedings of the ACM International Conference on Mobile Systems, Applications, and Services, Portland, Oregon, June 2022. [Paper] [Poster]
  • MilliPose: Facilitating Full Body Silhouette Imaging from Millimeter-Wave Device
    Aakriti Adhikari, Sanjib Sur
    UbiComp-ISWC’21 ACM International Joint Conference on Pervasive and Ubiquitous Computing and ACM International Symposium on Wearable Computers, Virtual, September 2021. [Paper] [Poster]

Presentations

Code and Dataset

Also Check Out