Gesture control is one of the effective and flexible communication method between humans and robots. However, it always depends on complex hardware and configurations in human-robot collaboration systems. Simplifying the design of gesture-interaction systems and avoiding miscommunication are challenging problems. In this paper, we proposed a method that utilizes an RGB sensor to realize spatial human-robot collaboration. A random forest based depth estimator is presented to supplement the additional spatial information for hand gesture recognition. Additionally, we demonstrate the construction of secure human-robot collaboration scenarios in Unity and validate our approach in real-world settings, based on which a digital twin system oriented to human-machine collaboration is constructed to realize rapid human-machine task simulation, safety specification testing, and real-scene applications development.
LiSWangRZhengP, et al. Towards proactive human–robot collaboration: A foreseeable cognitive manufacturing paradigm. J Manuf Syst2021; 60: 547–552.
2.
MezianeROtisMJDEzzaidiH.Human-robot collaboration while sharing production activities in dynamic environment: Spader system. Robot Comput Integr Manuf2017; 48: 243–253.
3.
RozoLSilverioJCalinonS, et al. Learning controllers for reactive and proactive behaviors in human–robot collaboration. Front Robot AI2016; 3: 30.
4.
NayyarAKumarA. A Roadmap to Industry 4.0: Smart Production, Sharp Business and Sustainable Development. Berlin: Springer, 2020.
5.
El ZaatariSMareiMLiW, et al. Cobot programming for collaborative industrial tasks: An overview. Rob Auton Syst2019; 116: 162–180.
6.
SluÿtersALambotSVanderdoncktJ. Hand gesture recognition for an off-the-shelf radar by electromagnetic modeling and inversion. In: 27th International conference on intelligent user interfaces, Helsinki, Finland, 2022, pp. 506–522. New York: Association for Computing Machinery.
7.
NishiyamaMWatanabeK.Wearable sensing glove with embedded hetero-core fiber-optic nerves for unconstrained hand motion capture. IEEE Trans Instrum Meas2009; 58(12): 3995–4000.
8.
GuXZhangYSunW, et al. Dexmo: An inexpensive and lightweight mechanical exoskeleton for motion capture and force feedback in vr. In: Proceedings of the 2016 CHI conference on human factors in computing systems, California, San Jose, 2016, pp. 1991–1995. New York: Association for Computing Machinery.
9.
OudahMAl-NajiAChahlJ.Hand gesture recognition based on computer vision: a review of techniques. J Imaging2020; 6(8): 73.
10.
MainpriceJBerensonD. Human-robot collaborative manipulation planning using early prediction of human motion. In: 2013 IEEE/RSJ international conference on intelligent robots and systems. Tokyo, Japan, 2013, IEEE, pp. 299–306. New York: IEEE.
11.
ShinSHLucyszynS.Benchmarking a commercial (sub- ) thz focal plane array against a custom-built millimeter-wave single-pixel camera. IEEE Access2020; 8: 191174–191190.
12.
MillerJL. Principles of infrared technology. New York, NY: Springer, 1994.
13.
MathesonEMintoRZampieriEG, et al. Human–robot collaboration in manufacturing applications: a review. Robotics2019; 8(4): 100.
14.
VysockyANovakP.Human-robot collaboration in industry. MM Sci J2016; 9(2): 903–906.
15.
AyvazSAlpayK.Predictive maintenance system for production lines in manufacturing: a machine learning approach using iot data in real-time. Expert Syst Appl2021; 173: 114598.
16.
EvjemoLDGjerstadTGrøtliEI, et al. Trends in smart manufacturing: role of humans and industrial robots in smart factories. Curr Robot Rep2020; 1: 35–41.
17.
SuHOvurSEZhouX, et al. Depth vision guided hand gesture recognition using electromyographic signals. Adv Robot2020; 34(15): 985–997.
18.
DingIJSuJL.Designs of human–robot interaction using depth sensor-based hand gesture communication for smart material-handling robot operations. J Eng Manuf2023; 237(3): 392–413.
19.
CheYQiY.Embedding gesture prior to joint shape optimization based real-time 3d hand tracking. IEEE Access2020; 8: 34204–34214.
20.
LiQYangP. Keep up with me: A gesture guided moving robot with microsoft kinect. In: 2013 IEEE 10th international conference on mobile ad-hoc and sensor systems. Hangzhou, China, 2013, pp. 435–436. New York: IEEE.
21.
RautiainenSPantanoMTraganosK, et al. Multimodal interface for human–robot collaboration. Machines2022; 10(10): 957.
22.
KianoushSSavazziSBeschiM, et al. A multisensory edge-cloud platform for opportunistic radio sensing in cobot environments. IEEE Internet Things J2020; 8(2): 1154–1168.
23.
LiuBFuWWangW, et al. Cobot motion planning algorithm for ensuring human safety based on behavioral dynamics. Sensors2022; 22(12): 4376.
24.
LiuWLiangXZhengM.Task-constrained motion planning considering uncertainty-informed human motion prediction for human–robot collaborative disassembly. IEEE/ASME Trans Mechatron2023; 28(4): 2056–2063.
25.
TrinhMKötterDChuA, et al. Safe and flexible planning of collaborative assembly processes using behavior trees and computer vision. Intell Hum Syst Integr (IHSI)2023; 69: 869–879.
26.
JantosTGHamdadMAGranigWet al. Poet: Pose estimation transformer for single-view, multi-object 6d pose estimation. In: Conference on robot learning. Auckland, New Zealand, 2023PMLR, pp. 1060–1070.
27.
AgnusdeiGPEliaVGnoniMG.A classification proposal of digital twin applications in the safety domain. Comput Ind Eng2021; 154: 107137.
28.
DouthwaiteJALesageBGleirscherM, et al. A modular digital twinning framework for safety assurance of collaborative robotics. Front Robot AI2021; 8: 758099.
29.
YeZJingyuLHongweiY.A digital twin-based human-robot collaborative system for the assembly of complex-shaped architectures. J Eng Manuf2022: 09544054 221110960.
30.
KutsVMarvelJAAksuM, et al. Digital twin as industrial robots manipulation validation tool. Robotics2022; 11(5): 113.
31.
OyekanJOHutabaratWTiwariA, et al. The effectiveness of virtual environments in developing collaborative strategies between industrial robots and humans. Robot Comput Integr Manuf2019; 55: 41–54.
32.
DuGHanRYaoG, et al. A gesture-and speech-guided robot teleoperation method based on mobile interaction with unrestricted force feedback. IEEE/ASME Trans Mechatron2021; 27(1): 360–371.
33.
CheYQiY. Fast hand-object interaction using gesture guide optimization. In 2020 IEEE conference on virtual reality and 3D user interfaces abstracts and workshops (VRW). Atlanta, GA, USA, 2020, pp. 792–793. New York: IEEE.
34.
AntakliASpieldennerTRubinsteinD, et al. Agent-based web supported simulation of human-robot collaboration. In: WEBIST, Vienna, Austria, pp. 88–99. Lda, Setubal, Portugal: SCITEPRESS - Science and Technology Publications.
35.
MalikAAMasoodTBilbergA.Virtual reality in manufacturing: immersive and collaborative artificial-reality in design of human-robot workspace. Int J Comput Integr Manuf2020; 33(1): 22–37.
WinkelhausSZhangMGrosseEH, et al. Hybrid order picking: a simulation model of a joint manual and autonomous order picking system. Comput Ind Eng2022; 167: 107981.
38.
ChenXGuoHWangG, et al. Motion feature augmented recurrent neural network for skeleton-based dynamic hand gesture recognition. In 2017 IEEE international conference on image processing (ICIP). Beijing, China, 2017, IEEE, pp. 2881–2885. New York: IEEE.
39.
ZhangXLiQMoHet al. End-to-end hand mesh recovery from a monocular rgb image. In: Proceedings of the IEEE/CVF international conference on computer vision, Seoul, Korea (South), 2019, pp. 2354–2364. New York: IEEE.
40.
PrakashKBEluriRKNaiduNBet al. Accurate hand gesture recognition using cnn and rnn approaches. Int J Adv Trends Comp Sci Eng2020; 9(3): 3216–3222.
41.
ZimmermannCBroxT. Learning to estimate 3d hand pose from single rgb images. In: Proceedings of the IEEE international conference on computer vision.Venice, Italy, 2017, pp. 4903–4911. New York: IEEE.
