Video-Based Lifting Action Recognition Using Rank-Altered Kinematic Feature Pairs

Abstract

Objective

To identify lifting actions and count the number of lifts performed in videos based on robust class prediction and a streamlined process for reliable real-time monitoring of lifting tasks.

Background

Traditional methods for recognizing lifting actions often rely on deep learning classifiers applied to human motion data collected from wearable sensors. Despite their high performance, these methods can be difficult to implement on systems with limited hardware resources.

Method

The proposed method follows a five-stage process: (1) BlazePose, a real-time pose estimation model, detects key joints of the human body. (2) These joints are preprocessed by smoothing, centering, and scaling techniques. (3) Kinematic features are extracted from the preprocessed joints. (4) Video frames are classified as lifting or nonlifting using rank-altered kinematic feature pairs. (5) A lifting counting algorithm counts the number of lifts based on the class predictions.

Results

Nine rank-altered kinematic feature pairs are identified as key pairs. These pairs were used to construct an ensemble classifier, which achieved 0.89 or above in classification metrics, including accuracy, precision, recall, and F1 score. This classifier showed an accuracy of 0.90 in lifting counting and a latency of 0.06 ms, which is at least 12.5 times faster than baseline classifiers.

Conclusion

This study demonstrates that computer vision-based kinematic features could be adopted to effectively and efficiently recognize lifting actions.

Application

The proposed method could be deployed on various platforms, including mobile devices and embedded systems, to monitor lifting tasks in real-time for the proactive prevention of work-related low-back injuries.

Keywords

computer vision lifting counting musculoskeletal disorder top scoring pair workplace safety

Get full access to this article

View all access options for this article.

References

Anagnostis

Benos

Tsaopoulos

Tagarakis

Tsolakis

Bochtis

(2021). Human activity recognition through recurrent neural networks for human–robot interaction in agriculture. Applied Sciences, 11(5), 2188. https://doi.org/10.3390/APP11052188

Anderson

C. K.

Chaffin

D. B.

(1986). A biomechanical evaluation of five lifting techniques. Applied Ergonomics, 17(1), 2–8. https://doi.org/10.1016/0003-6870(86)90186-9

Bangaru

S. S.

Wang

Busam

S. A.

Aghazadeh

(2021). ANN-based automated scaffold builder activity recognition through wearable EMG and IMU sensors. Automation in Construction, 126(1), Article 103653. https://doi.org/10.1016/J.AUTCON.2021.103653

Bazarevsky

Grishchenko

Raveendran

Zhu

Zhang

Grundmann

(2020). BlazePose: On-device real-time body pose tracking. https://arxiv.org/abs/2006.10204v1

Bernard

B. P.

Putz-Anderson

Susan Burt Libby L Cole

M. E.

Fairfield-Estill Lawrence Fine

C. J.

Katharyn Grant

D. A.

Gjessing Lynn Jenkins Joseph Hurrell Jr

C. J.

Nelson

Pfirman Robert Roberts Diana Stetson

Haring-Sweeney

Tanaka

(1997). Musculoskeletal disorders and workplace factors: A critical review of epidemiologic evidence for work-related musculoskeletal disorders of the neck, upper extremity, and low back. https://stacks.cdc.gov/view/cdc/21745

Callaghan

J. P.

McGill

S. M.

(2001). Intervertebral disc herniation: Studies on a porcine model exposed to highly repetitive flexion/extension motion with compressive force. Clinical Biomechanics, 16(1), 28–37. https://doi.org/10.1016/S0268-0033(00)00063-2

Carreira

Zisserman

Com

Deepmind (2017). Quo Vadis, action recognition? A new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 6299–6308).

Cob-Parro

A. C.

Losada-Gutiérrez

Marrón-Romera

Gardel-Vicente

Bravo-Muñoz

(2024). A new framework for deep learning video based Human Action Recognition on the edge. Expert Systems with Applications, 238, Article 122220. https://doi.org/10.1016/J.ESWA.2023.122220

Desmoulin

G. T.

Pradhan

Milner

T. E.

(2020). Mechanical aspects of intervertebral disc injury and implications on biomechanics. Spine, 45(8), E457–E464. https://doi.org/10.1097/BRS.0000000000003291

10.

Dolan

Adams

M. A.

(1998). Repetitive lifting tasks fatigue the back muscles and increase the bending moment acting on the lumbar spine. Journal of Biomechanics, 31(8), 713–721. https://doi.org/10.1016/S0021-9290(98)00086-4

11.

Dollar

A. M.

(2014). Classifying human hand use and the activities of daily living. Springer Tracts in Advanced Robotics, 95(1), 201–216. https://doi.org/10.1007/978-3-319-03017-3_10

12.

Donisi

Cesarelli

Capodaglio

Panigazzi

D’Addio

Cesarelli

Amato

(2022). A logistic regression model for biomechanical risk classification in lifting tasks. Diagnostics, 12(11), 2624. https://doi.org/10.3390/DIAGNOSTICS12112624

13.

Elhayek

De Aguiar

Jain

Thompson

Pishchulin

Andriluka

Bregler

Schiele

Theobalt

(2017). MARCOnI - ConvNet-Based MARker-less motion capture in outdoor and indoor scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(3), 501–514. https://doi.org/10.1109/TPAMI.2016.2557779

14.

Feng

Zhao

S. Z.

Pan

Chang

Chen

Wang

Yang

A. Y.

(2023). Digital twin tracking dataset (DTTD): A new RGB+Depth 3D dataset for longer-range object tracking applications (pp. 3289–3298). https://optitrack.com/

15.

Geman

D’Avignon

Naiman

D. Q.

Winslow

R. L.

(2004). Classifying gene expression profiles from pairwise mRNA comparisons. Statistical Applications in Genetics and Molecular Biology, 3(1), 19. https://doi.org/10.2202/1544-6115.1071

16.

Golestani

Moghaddam

(2020). Human activity recognition using magnetic induction-based motion signals and deep recurrent neural networks. Nature Communications, 11(1), 1–11. https://doi.org/10.1038/s41467-020-15086-2

17.

Greene

R. L.

M. L.

Barim

M. S.

Wang

Hayden

Y. H.

Radwin

R. G.

(2022). Estimating trunk angle kinematics during lifting using a computationally efficient computer vision method. Human Factors, 64(3), 482–498. https://doi.org/10.1177/0018720820958840

18.

Han

Mao

Dally

W. J.

(2015). Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. In 4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings. https://arxiv.org/abs/1510.00149v5

19.

Hendry

Chai

Campbell

Hopper

O’Sullivan

Straker

(2020). Development of a human activity recognition system for ballet tasks. Sports Medicine - Open, 6(1), 10. https://doi.org/10.1186/s40798-020-0237-5

20.

Howard

A. G.

Zhu

Chen

Kalenichenko

Wang

Weyand

Andreetto

Adam

(2017). MobileNets: Efficient convolutional neural networks for mobile Vision Applications. https://arxiv.org/abs/1704.04861v1

21.

Huang

Zeng

Liu

Lai

(2020). DeepFuse: An IMU-aware network for real-time 3D human pose estimation from multi-view image (pp. 429–438).

22.

Insafutdinov

Pishchulin

Andres

Andriluka

Schiele

(2016). Deepercut: A deeper, stronger, and faster multi-person pose estimation model. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 9910 LNCS (pp. 34–50). Springer. https://doi.org/10.1007/978-3-319-46466-4_3

23.

Kim

S. H.

Lin

C. W.

Tseng

G. C.

(2016). MetaKTSP: A meta-analytic top scoring pair method for robust cross-study validation of omics prediction analysis. Bioinformatics, 32(13), 1966–1973. https://doi.org/10.1093/BIOINFORMATICS/BTW115

24.

Kluwak

Klempous

Chaczko

Rozenblit

J. W.

Kulbacki

(2021). People lifting patterns—A reference dataset for practitioners. Sensors, 21(9), 3142. https://doi.org/10.3390/S21093142

25.

Kumari

B. A.

Bhargavi

Aswini

Yamini

Vedavathi

(2024). Human action recognition from video frames using recurrent neural network. In 2024 2nd International Conference on Disruptive Technologies, ICDT 2024 (pp. 731–737). IEEE. https://doi.org/10.1109/ICDT61202.2024.10489658

26.

Prabhu

Xie

Wang

(2021). Lifting posture prediction with generative models for improving occupational safety. IEEE Transactions on Human-Machine Systems, 51(5), 494–503. https://doi.org/10.1109/THMS.2021.3102511

27.

Xie

(2020). MOPED25: A multimodal dataset of full-body pose and motion in occupational tasks. Journal of Biomechanics, 113, Article https://doi.org/10.1016/J.JBIOMECH.2020.110086

28.

Lin

T. Y.

Maire

Belongie

Hays

Perona

Ramanan

Dollár

Zitnick

C. L.

(2014). Microsoft COCO: Common objects in context. In Lecture Notes in Computer Science (pp. 740–755). Springer. https://doi.org/10.1007/978-3-319-10602-1_48

29.

Luo

Wang

Weng

Hsu

L. T.

Chen

(2021). Integration of GNSS and BLE technology with inertial sensors for real-time positioning in urban environments. IEEE Access, 9, 15744–15763. https://doi.org/10.1109/ACCESS.2021.3052733

30.

Maher

Underwood

Buchbinder

(2017). Non-specific low back pain. Lancet (London, England), 389(10070), 736–747. https://doi.org/10.1016/S0140-6736(16)30970-9

31.

Marras

W. S.

Parakkat

Chany

A. M.

Yang

Burr

Lavender

S. A.

(2006). Spine loading as a function of lift frequency, exposure duration, and work experience. Clinical Biomechanics, 21(4), 345–352. https://doi.org/10.1016/J.CLINBIOMECH.2005.10.004

32.

Meng

Yang

Ying

(2022). A structure-aware adversarial framework with the keypoint biorientation field for multiperson pose estimation. Wireless Communications and Mobile Computing, 2022(1), 1–17. https://doi.org/10.1155/2022/3447827

33.

Minhas

R. A.

Javed

Irtaza

Mahmood

M. T.

Joo

Y. B.

(2019). Shot classification of field sports videos using AlexNet convolutional neural network. Applied Sciences, 9(3), 483. https://doi.org/10.3390/APP9030483

34.

Ordóñez

F. J.

Roggen

Liu

Xiao

Chao

H.-C.

Chu

(2016). Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors, 16(1), 115. https://doi.org/10.3390/S16010115

35.

Qin

Guo

Zhou

Sun

Chen

Qiu

(2024). Gait recognition based on two-stream CNNs with multisensor progressive feature fusion. IEEE Sensors Journal, 24(8), 13676–13685. https://doi.org/10.1109/JSEN.2024.3373100

36.

Sharma

Gupta

Kumar

Mishra

(2021). Video processing using deep learning techniques: A systematic literature review. IEEE Access, 9, 139489–139507. https://doi.org/10.1109/ACCESS.2021.3118541

37.

Shi

Ray

Zhu

Kon

M. A.

(2011). Top scoring pairs for feature selection in machine learning and applications to cancer outcome prediction. BMC Bioinformatics, 12(1), 1–15. https://doi.org/10.1186/1471-2105-12-375

38.

Tan

A. C.

Naiman

D. Q.

Winslow

R. L.

Geman

(2005). Simple decision rules for classifying human cancers from gene expression profiles. Bioinformatics, 21(20), 3896–3904. https://doi.org/10.1093/BIOINFORMATICS/BTI631

39.

Thomas

M. L.

Jha

Bertrand

(2022). Machine learning for detection and risk assessment of lifting action. IEEE Transactions on Human-Machine Systems, 52(6), 1196–1204. https://doi.org/10.1109/THMS.2022.3212666

40.

U.S. Bureau of Labor Statistics . (2023). Employer-Reported Workplace Injuries and Illnesses (Annual) News Release. Retrieved May 9, 2024, from: https://www.bls.gov/news.release/osh.nr0.htm

41.

Varrecchia

De Marchis

Draicchio

Schmid

Conforto

Ranavolo

(2020). Lifting activity assessment using kinematic features and neural networks. Applied Sciences, 10(6), 1989. https://doi.org/10.3390/APP10061989

42.

Wang

Xie

(2021). A computer-vision method to estimate joint angles and L5/S1 moments during lifting tasks through a single camera. Journal of Biomechanics, 129(1), Article 110860. https://doi.org/10.1016/J.JBIOMECH.2021.110860

43.

Waters

T. R.

Putz-Anderson

Garg

Fine

L. J.

(1993). Revised NIOSH equation for the design and evaluation of manual lifting tasks. Ergonomics, 36(7), 749–776. https://doi.org/10.1080/00140139308967940

44.

Zhang

Chen

(2023). Spatial and contextual aware network based on multi-resolution for human pose estimation. The Visual Computer, 39(2), 651–662. https://doi.org/10.1007/s00371-021-02364-3

45.

Zhou

Aggarwal

Yin

(2022). A computer vision approach for estimating lifting load contributors to injury risk. IEEE Transactions on Human-Machine Systems, 52(2), 207–219. https://doi.org/10.1109/THMS.2022.3148339