Abstract
Within the context of the 2021 OpenCV AI Competition, the corpusLabe prototype was developed exploring the use of the OpenCV AI Kit for clinical assessment of posture, as well as, a gamified exercise platform. Two evaluation strategies were pursued: the first, characterizing the proposed system while poking at the problem of human body pose estimation on an edge device; the second, investigating on the use of computed pose coordinates for measuring shoulder joint angles in comparison to a gold standard goniometer. The prototype integrating the BlazePose model running on DepthAI hardware was not only able to provide measurements of shoulder angle in an accessible and intuitive way, but also to demonstrate a concept of a gamified goniometer. Despite not being conclusive, the measurement discrepancies between the corpusLabe prototype and the standard instrument for the evaluated shoulder angle were found to be clinically significant in the group of recruited participants.
Motivation
As breast cancer survivors (BCS) are living longer, the adverse effects resulting from the cancer treatment are more frequent. Upper body morbidity (UBM) (e.g. decreased range of motion, muscle strength, pain and lymphedema) are among the most prevalent side effects [1]. Arm/shoulder mobility, usually assessed by goniometer-based measurements of flexion or abduction, is an objective measure of UBM that has been used in breast cancer rehabilitation, although its well established use covers much broader application scenarios [2]–[4] (Figure1).
Despite its recognized relevancy [1], [5], [6], and although some methods for monitoring and assessing eventual side effects from breast cancer treatments do exist [7]–[9], an approach able to achieve early detection, promote risk-reduction and self-management, while engaging the user in an appropriate follow-up strategy, seems to be still missing for a myriad of reasons [10]–[12].
corpusLabe prototype baseline
Taking the breast cancer treatment follow-up scenario as a study case, and two distinct use case settings (virtual goniometer and gamified exercise platform) the set up schematized in Figure 2 was outlined and developed, considering three main fundamental tasks: 3D pose estimation, analysis and visualization.
3D pose estimation from images
Even though being outside the scope of this report a review of the topic, it can be mentioned that a significant amount of recent work exists appertaining to the long lasting problem of 3D human pose recovery from a single image [13]–[15]. For its convenience and availability within the context of the OpenCV AI Competition, the BlazePose made available by Geax was used as a baseline human pose estimation method for the prototype to which this report refers. BlazePose [13] is a regression based single person body pose estimation method suitable to run on mobile devices that can compute (x,y,z) coordinates of 33 skeleton keypoints. Its inference pipeline comprises a body pose detector followed by a pose tracker network. Its pose detector approach extends previous work on the stacked hourglass architecture using an encoder-decoder followed by another encoder network to predict joint’s heatmaps and regress its coordinates. The aforementioned heatmap step is discarded during inference in order to make it particularly lightweight and thus, contributing to the overall attractiveness of the method to real-time use cases.
Experiments
For exploratory quantitatively evaluation on the 3D joint estimation task, the database Human3.6M [16] was used. The latter database is a standard benchmark captured in a lab environment including millions of 3D human poses from distinct subjects performing 15 actions, such as eating, sitting and walking, acquired from a MoCap system with corresponding images from 4 points of view. The protocol described in previous works [17]–[19] was transcribed, such that, results for mean per-joint position error (MPJPE) in millimeters (mm) that measures the mean Euclidean distance between the predicted and ground-truth joint positions without any transformation, are reported on data corresponding to subjects S9 and S11 on a 17-joint skeleton. The aforementioned baseline DepthAI version of the BlazePose model (tested baseline) was compared with reference and state-of-the-art 3D human pose recovery from single-image methods results, found in the literature, as a rough benchmarking of performance. The results are summarized in Table 1.
Discussion
The tested baseline model has high average error although alongside the method that accompanied the original publication of the considered database. Among several points of discussion the comparison suffers from some deviations as the originally proposed protocol implied a given training methodology that is not uniform among the reported results. Moreover, the tested baseline can be recognized as a suboptimal version of the model with several tuning opportunities available that were not explored (namely: competitive landmark models available, or number of SHAVES associated with the MyriadX specific architecture).
Pose analysis and visualization
This work focused on a single pose to explore the use of the outlined prototype: the maximum angle of shoulder abduction, as illustrated in Figure 3.
Similarly as in [7], and using the skeletal data provided by the BlazePose model (Figure 4 a)), the positions of shoulder and elbow joints relative to the vertical projection of the shoulder were used to measure the angles of shoulder abduction in degrees, as illustrated in Figure 4 b). Considering indications from past work [22] that a more anthropomorphised representation in a virtual world, seems to not be the preferred representation of the self among the breast cancer survivors population, an abstract representation was adopted (Figure 4 c) and d)).
corpusLabe as a virtual goniometer
The hereafter evaluated virtual goniometer was accomplished by modifying the corpusLabe prototype baseline (outlined in Figure 4) to store the maximum registered reference shoulder joint angles and adding the possibility of an operator to reset the registered maximum angles or saving the values to file (Figure 5).
Methods
Shoulder joint angle of maximum attempted abduction for each side was assessed using the corpusLabe virtual goniometer prototype and a standard goniometer.
Participants
A convenience sample of four adult females participated in the study recruited via personal invitation from surgeon-led follow-up consultations of breast cancer survivors at the the Breast Cancer Care unit at the University Hospital Center of São João. All participants were fluent in Portuguese and did not get paid for their participation.
Data capture and processing
Goniometer
Goniometric measurements of shoulder maximum abduction were performed using standardised methods [3], [7]. The goniometer axis was aligned to the posterior aspect of the shoulder joint by the examiner, who would also read and record the measurement (in degrees), a single time for each side of each recruited participant.
corpusLabe prototype
Respecting the aforementioned reference protocol, the considered pose was performed facing the corpusLabe prototype (Figure 2). The examiner would be able to save at a press of a button the maximum values of abduction registered during a session.
Statistical analysis
The 95% limits of agreement (LOA) between the corpusLabe prototype and the measurement reference goniometer for the shoulder joint angle were computed for the considered pose to approximate a validity study, similarly as in the work by Huber et al. [7]. To obtain the 95% LOA, the mean of the two shoulder angle measurements from each method was calculated. Next, the mean and standard deviation (SD) of differences between the corpusLabe prototype and the goniometer measurements were computed. The 95% LOA were defined as the mean difference ±1.96 SD of the difference, such that 95% of differences lay within these limits. If the 95% LOA were greater than ±5°, the discrepancies between measurement systems were considered to be clinically significant.
Results
The 95% LOA between the corpusLabe prototype and the goniometer are shown in Fig. 6. Abduction to maximum was the only pose in which the corpusLabe prototype measures of shoulder angle were compared with the reference goniometer. However, the 95% LOA for the discrepancy between systems exceeded ±5°, which was defined as clinically significant.
Discussion
Using the skeletal data from the aforementioned version of the BlazePose model for DepthAI hardware and the proposed pose analysis approach, the corpusLabe prototype was found to be able to provide measurements of shoulder angle. However, the measurement discrepancies between the corpusLabe prototype and the measurement standard were clinically significant. Notwithstanding the above, the small recruited sample, the variability of the movement considered, particularly in the studied population, and the width of the confidence intervals in the Bland-Altman analysis, suggests that further experimentation should be explored, including, and not exhaustively, more poses to be evaluated or adding more repetitions for each pose to the protocol of data acquisition.
corpusLabe as a gamified exercise platform
Considering the aforementioned approach to pose analysis and visualization and inspired by previous work on breast cancer survivors’ expressed preferences in a physical activity promotion intervention [22], a gamified goniometer was developed as schematized in Figure 7 and summarily illustrated in Figure 8. Considering the selected movement (Figure 3), a straightforward approach using restrictions to the detected arm pose and average reference angles, enabled an exercise counter and the implementation of a guided execution of alternating arms abduction movement game with predefined target goals for each new user to attempt to reach.
Closing Remarks
A prototype exploring the use of the OpenCV AI Kit for a particular task of clinical assessment of posture and a related approach including gamifying elements aimed at promoting physical activity was developed. Even though a suboptimal version of the BlazePose model was used to perform inference on the DepthAI hardware, the reported average mean per-joint position error in the Human3.6M database standard testing subset is in line with the results of the article that accompanied the publication of said database. Notwithstanding, and considering, not just the tuning opportunities of the evaluated method that were not investigated in this reported work, but also, the possibility to explore from the vast array of methods and databases related to the task of 3D human pose estimation recently made available, the reported results of relatively high average error in Table 1 can, in the humble opinion of the authors of the report, be regarded as somewhat pessimistic with respect to what the system can achieve, although this can, and should, be further explored in future work. Somewhat similarly to what is stated above, the very limited sample of recruited users and evaluated poses, as well as, the specificity of the considered pose of maximum shoulder adbuction and its natural variability, doesn’t allow the validity study presented to be conclusive, even though, the results reported show clinically significant differences between the angle measurements obtained using the corpusLabe prototype and the gold standard goniometer. Although a playable version of the proposed gamified goniometer was achieved, this application was not properly evaluated. Its installation is, notwithstanding, planned for two distinct contexts (namely, a waiting room in a breast cancer treatment unit, and a common room in a technological institute). It is intended with this experiment to collect usage data (e.g. number of people detected, and number of exercises completed) as a rough assessment of the possibility of such aimplementation to promote some type of autonomously executed physical activity, and to investigate how different types of population would respond to the considered prototype.
Bibliography
[1] Elizabeth A. Chrischilles, Danielle Riley, Elena Letuchy, Linda Koehler, Joan Neuner, Cheryl Jernigan, Brian Gryzlak, Neil Segal, Bradley McDowell, Brian Smith, Sonia L. Sugg, Jane M. Armer, and Ingrid M. Lizarraga. “Upper extremity disability and quality of life after breast cancer treatment in the Greater Plains Collaborative clinical research network”. In: Breast Cancer Research and Treatment 175.3 (2019), pp. 675–689. doi: 10.1007/s10549-019-05184-1.
[2] A Williams Andrews and Richard W Bohannon. “Decreased Shoulder Range of Motion on Paretic Side After Stroke”. In: Physical Therapy 69.9 (1989), pp. 768–772. doi: 10.1093/ptj/69.9.768.
[3] Morey J. Kolber, Cydne Fuller, Jessica Marshall, Amanda Wright, and William J. Hanney. “The reliability and concurrent validity of scapular plane shoulder elevation measurements using a digital inclinometer and goniometer”. In: Physiotherapy Theory and Practice 28.2 (2011), pp. 161–168. doi: 10.3109/09593985.2011.574203.
[4] Burakhan Çubukçu, Uğur Yüzgeç, Raif Zileli, and Ahu Zileli. “Reliability and validity analyzes of Kinect V2 based measurement system for shoulder motions”. In: Medical Engineering & Physics 76 (2020), pp. 20–31. doi: 10.1016/j.medengphy.2019.10.017.
[5] Nicole L. Stout, Jill M. Binkley, Kathryn H. Schmitz, Kimberly Andrews, Sandra C. Hayes, Kristin L. Campbell, Margaret L. McNeely, Peter W. Soballe, Ann M. Berger, Andrea L. Cheville, Carol Fabian, Lynn H. Gerber, Susan R. Harris, Karin Johansson, Andrea L. Pusic, Robert G. Prosnitz, and Robert A. Smith. “A prospective surveillance model for rehabilitation for women with breast cancer”. In: Cancer 118.S8 (2012), pp. 2191–2200. doi: 10.1002/cncr.27476.
[6] Catherine Duggan, Allison Dvaladze, Anne F. Rositch, Ophira Ginsburg, Cheng-Har Yip, Susan Horton, Rolando Camacho Rodriguez, Alexandru Eniu, Miriam Mutebi, Jean-Marc Bourque, Shahla Masood, Karla Unger-Saldaña, Anna Cabanes, Robert W. Carlson, Julie R. Gralow, and Benjamin O. Anderson. “The Breast Health Global Initiative 2018 Global Summit on Improving Breast Healthcare Through Resource-Stratified Phased Implementation: Methods and overview”. In: Cancer 126.S10 (2020), pp. 2339–2352. doi: 10.1002/cncr.32891.
[7] M.E. Huber, A.L. Seitz, M. Leeser, and D. Sternad. “Validity and reliability of Kinect skeleton for measuring shoulder joint angles: a feasibility study”. In: Physiotherapy 101.4 (2015), pp. 389–393. doi: 10.1016/j.physio.2015.02.002.
[8] Justin W. L. Keogh, Alistair Cox, Sarah Anderson, Bernard Liew, Alicia Olsen, Ben Schram, and James Furness. “Reliability and validity of clinically accessible smartphone applications to measure joint range of motion: A systematic review”. In: PLOS ONE 14.5 (2019). Ed. by Juliane Müller, e0215806. doi: 10.1371/journal.pone.0215806.
[9] Bojan Milosevic, Alberto Leardini, and Elisabetta Farella. “Kinect and wearable inertial sensors for motor rehabilitation programs at home: state of the art and an experimental comparison”. In: BioMedical Engineering OnLine 19.1 (2020). doi: 10.1186/s12938-020-00762-7.
[10] Danton S. Char, Michael D. Abràmoff, and Chris Feudtner. “Identifying Ethical Considerations for Machine Learning Health- care Applications”. In: The American Journal of Bioethics 20.11 (2020), pp. 7–17. doi: 10.1080/15265161.2020.1819469.
[11] Georgios A. Kaissis, Marcus R. Makowski, Daniel Rückert, and Rickmer F. Braren. “Secure, privacy-preserving and federated machine learning in medical imaging”. In: Nature Machine Intelligence 2.6 (2020), pp. 305–311. doi: 10.1038/s42256- 020- 0186-1.
[12] Sebastian Vollmer, Bilal A Mateen, Gergo Bohner, Franz J Király, Rayid Ghani, Pall Jonsson, Sarah Cumbers, Adrian Jonas, Katherine S L McAllister, Puja Myles, David Grainger, Mark Birse, Richard Branson, Karel G M Moons, Gary S Collins, John P A Ioannidis, Chris Holmes, and Harry Hemingway. “Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness”. In: BMJ (2020), p. l6927. doi: 10.1136/bmj.l6927.
[13] Valentin Bazarevsky, Ivan Grishchenko, Karthik Raveendran, Tyler Zhu, Fan Zhang, and Matthias Grundmann. “BlazePose: On-device Real-time Body Pose tracking”. In: CoRR abs/2006.10204 (2020). arXiv: 2006.10204.
[14] Yu Sun, Qian Bao, Wu Liu, Yili Fu, and Tao Mei. “CenterHMR: a Bottom-up Single-shot Method for Multi-person 3D Mesh Recovery from a Single Image”. In: CoRR abs/2008.12272 (2020). arXiv: 2008.12272.
[15] Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black. “VIBE: Video Inference for Human Body Pose and Shape Estimation”. In: CoRR abs/1912.05656 (2019). arXiv: 1912.05656.
[16] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. “Human3.6M: Large Scale Datasets and Predic- tive Methods for 3D Human Sensing in Natural Environments”. In: IEEE Transactions on Pattern Analysis and Machine Intelligence 36.7 (2014), pp. 1325–1339. doi: 10.1109/tpami.2013.248.
[17] Xiaowei Zhou, Menglong Zhu, Spyridon Leonardos, Konstantinos G. Derpanis, and Kostas Daniilidis. “Sparseness Meets Deep- ness: 3D Human Pose Estimation From Monocular Video”. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016.
[18] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik. “End-to-End Recovery of Human Shape and Pose”. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018.
[19] Tianlang Chen, Chen Fang, Xiaohui Shen, Yiheng Zhu, Zhili Chen, and Jiebo Luo. “Anatomy-aware 3D Human Pose Estimation with Bone-based Pose Decomposition”. In: IEEE Transactions on Circuits and Systems for Video Technology (2021), pp. 1–1. doi: 10.1109/tcsvt.2021.3057267.
[20] Kehong Gong, Jianfeng Zhang, and Jiashi Feng. “PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021, pp. 8575–8584.
[21] Georgios Pavlakos, Xiaowei Zhou, and Kostas Daniilidis. “Ordinal Depth Supervision for 3D Human Pose Estimation”. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018.
[22] João Pedro Monteiro, Carolina Teixeira Lopes, Nuno Correia Duarte, André Torres Magalhães, and Hélder Pinto de Oliveira. “Investigations on the Impact of Anthropomorphism and Gamification on Breast Cancer Survivors’ Expressed Preferences in a Physical Activity Promotion Intervention”. In: 2019, pp. 139–143. isbn: 9781612086880.