The generous application of robot-assisted minimally invasive surgery (RAMIS) promotes human-machine interaction (HMI). Identifying various behaviors of doctors can enhance the RAMIS procedure for the redundant robot. It bridges intelligent robot control and activity recognition strategies in the operating room, including hand gestures and human activities. In this paper, to enhance identification in a dynamic situation, we propose a multimodal data fusion framework to provide multiple information for accuracy enhancement. Firstly, a multi-sensors based hardware structure is designed to capture varied data from various devices, including depth camera and smartphone. Furthermore, in different surgical tasks, the robot control mechanism can shift automatically. The experimental results evaluate the efficiency of developing the multimodal framework for RAMIS by comparing it with a single sensor system. Implementing the KUKA LWR4+ in a surgical robot environment indicates that the surgical robot systems can work with medical staff in the future.
AlharthiASYunasSUOzanyanKB (2019) Deep learning for monitoring of human gait: A review. IEEE Sensors Journal19(21): 9575–9591.
2.
AtawnihAPapageorgiouDDoulgeriZ (2014) Reaching for redundant arms with human-like motion and compliance properties. Robotics and Autonomous Systems62(12): 1731–1741.
3.
CaccianigaGMarianiADe MomiE, et al. (2020) An evaluation of inanimate and virtual reality training for psychomotor skill development in robot-assisted minimally invasive surgery. IEEE Transactions on Medical Robotics and Bionics2(2): 118–129.
4.
ChenYGodageISuH, et al. (2019) Stereotactic systems for mri-guided neurosurgeries: A state-of-the-art review. Annals Of Biomedical Engineering47(2): 335–353.
5.
ChenYWangLGallowayK, et al. (2020) Modal-based kinematics and contact detection of soft robots. Soft Robotics. DOI: 10.1089/soro.2019.0095
6.
Ehatisham-Ul-HaqMJavedAAzamMA, et al. (2019) Robust human activity recognition using multimodal feature-level fusion. IEEE Access7: 60736–60751. DOI:10.1109/ACCESS.2019.2913393
7.
EnayatiNDe MomiEFerrignoG (2016) Haptics in robot-assisted surgery: Challenges and benefits. IEEE Reviews in Biomedical Engineering9: 49–65. DOI:10.1109/RBME.2016.2538080
8.
GaoHZhuJLiX, et al. (2021) Automatic parking control of unmanned vehicle based on switching control algorithm and backstepping. IEEE/ASME Transactions on Mechatronics. DOI:10.1109/TMECH.2020.3037215.
9.
GravinaRAliniaPGhasemzadehHFortinoG (2017) Multi-sensor fusion in body sensor networks: State-of-the-art and research challenges. Information Fusion35: 68–80. DOI:10.1016/j. inffus.2016.09.005
10.
HassanMMUddinMZMohamedAAlmogrenA (2018) A robust human activity recognition system using smartphone sensors and deep learning. Future Generation Computer Systems81: 307–313. DOI:10.1016/j.future.2017.11.029
Huynh-TheTHuaCHTuNAKimDS (2020) Physical activity recognition with statistical-deep fusion model using multiple sensory data for smart health. IEEE Internet of Things Journal. DOI: 10.1109/JIOT.2020.3013272
13.
KonstantinovaJJiangAAlthoeferK, et al. (2014) Implementation of tactile sensing for palpation in robot-assisted minimally invasive surgery: A review. IEEE Sensors Journal14(8): 2490–2501.
14.
LiJWangJWangS, et al. (2020) Parallel structure of six wheel-legged robot trajectory tracking control with heavy payload under uncertain physical interaction. Assembly Automation. DOI: 10.1108/aa-08-2019-0148
15.
LiZHuangBAjoudaniA, et al. (2017) Asymmetric bimanual control of dual-arm exoskeletons for human-cooperative manipulations. IEEE Transactions on Robotics34(1): 264–271.
16.
LiZRenZZhaoK, et al. (2019) Human-cooperative control design of a walking exoskeleton for body weight support. IEEE Transactions on Industrial Informatics16(5): 2985–2996.
17.
LiuZKarimiHRYuJ (2020) Passivity-based robust sliding mode synthesis for uncertain delayed stochastic systems via state observer. Automatica. DOI: 10.1016/j.automatica.2019.108596
18.
MarioMO (2018) Human activity recognition based on single sensor square HV acceleration images and convolutional neural networks. IEEE Sensors Journal19(4): 1487–1498.
19.
MathurAFoodyGM (2008) Multiclass and binary svm classification: Implications for training and classification users. IEEE Geoscience and Remote Sensing Letters5(2): 241–245.
20.
MilsteinAGanelTBermanSNiskyI (2018) Human-centered transparency of grasping via a robot-assisted minimally invasive surgery system. IEEE Transactions on Human-Machine Systems48(4): 349–358.
21.
Mondéjar-GuerraVNovoJRoucoJ, et al. (2019) Heartbeat classification fusing temporal and morphological information of ECGS via ensemble of classifiers. Biomedical Signal Processing and Control47: 41–48. DOI:10.1016/j.bspc.2018.08.007
22.
NaikGRSelvanSEArjunanSP, et al. (2018) An ICA-EBM-based semg classifier for recognizing lower limb movements in individuals with and without knee pathology. IEEE Transactions on Neural Systems and Rehabilitation Engineering26(3): 675–686.
23.
NorgaardSSaeediRGebremedhinAH (2019) Multi-sensor time-series classification for activity tracking under variable length. IEEE Sensors Journal20(5): 2701–2709.
24.
QiWAlivertiA (2019) A multimodal wearable system for continuous and real-time breathing pattern monitoring during daily activity. IEEE Journal of Biomedical and Health Informatics. DOI: 10.1109/JBHI.2019.2963048
25.
QiWSuHAlivertiA (2020) A smartphone-based adaptive recognition and real-time monitoring system for human activities. IEEE Transactions on Human-Machine Systems50(5): 414–423.
26.
QiWZongGKarimiHR (2019) Finite-time observer-based sliding mode control for quantized semi-markov switching systems with application. IEEE Transactions on Industrial Informatics16(2): 1259–1271.
27.
Reyes-OrtizJLOnetoLSamàA, et al. (2016) Transition-aware human activity recognition using smartphones. Neurocomputing171: 754–767. DOI:10.1016/j.neucom.2015.07.085
28.
SandovalJSuHVieyresP, et al. (2018) Collaborative framework for robot-assisted minimally invasive surgery using a 7-dof anthropomorphic robot. Robotics and Autonomous Systems106: 95–106. DOI:10.1016/j.robot.2018.04.001
29.
SuHHuYKarimiHR, et al. (2020a) Improved recurrent neural network-based manipulator control with remote center of motion constraints: Experimental results. Neural Networks131: 291–299. DOI:10.1016/j.neunet.2020.07.033
30.
SuHMarianiAOvur SalihE, et al. (2021a) Towards teaching by demonstration for robot-assisted minimally invasive surgery. IEEE Transactions on Automation Science and Engineering. DOI: 10.1109/TASE.2020.3045655
31.
SuHQiWHuY, et al. (2021b) An incremental learning framework for human-like redundancy optimization of anthropomorphic manipulators. IEEE Transactions on Industrial Informatics. DOI:10.1109/TII.2020.3036693.
32.
SuHQiWYangC, et al. (2020b) Deep neural network approach in robot tool dynamics identification for bilateral teleoperation. IEEE Robotics and Automation Letters5(2): 2943–2949.
33.
TatinatiSVeluvoluKCAngWT (2014) Multistep prediction of physiological tremor based on machine learning for robotics assisted microsurgery. IEEE Transactions on Cybernetics45(2): 328–339.
34.
TufekNYalcinMAltintasM, et al. (2019) Human action recognition using deep learning methods on limited sensory data. IEEE Sensors Journal20(6): 3101–3112.
35.
WangYCangSYuH (2018) A data fusion-based hybrid sensory system for older people’s daily activity and daily routine recognition. IEEE Sensors Journal18(16): 6874–6888.
36.
WangYKarimiHRLamHKYanH (2019) Fuzzy output tracking control and filtering for nonlinear discrete-time descriptor systems under unreliable communication links. IEEE Transactions on Cybernetics50(6): 2369–2379.
37.
YangCChenCWangN, et al. (2018) Biologically inspired motion modeling and neural control for robot learning from demonstrations. IEEE Transactions on Cognitive and Developmental Systems11(2): 281–291.
