Design,analysis and performance evaluation of a hand gesture platform for navigation

Abstract

Prevailing technological solutions that address the problems that are experienced by the infirm and elderly people in terms of locomotion needs, offer limited options when it comes to control mechanism and customization. For more than a decade, joysticks have served the purpose of steering and navigation of autonomous wheelchairs. An alternative gesture-based method for navigation of wheelchairs by the physically impaired could very well replace the conventional joystick controls. A prototype system, ‘Mudra’ (Sanskrit word for gesture), incorporates a gesture capture module, developed for recognition and identification of hand gestures. Mudra is a no-nonsense user-friendly system that enables control of the navigational platform, merely by four gestures of the human hand. This paper presents a comprehensive report on the implementation of the Mudra system’s hardware and software, performance analysis and safety evaluation. Verification tests of the conceptual design show promising results, where 97.8% of the hand gestures were recognized accurately. Furthermore, the response timings of wheelchairs with Mudra controls were noticeably faster than the joystick-based wheelchairs, as affirmed by live testing with wheelchair-users. Pursuant to the positive feedback from the wheelchair-user experience, we conclude that Mudra’s gesture controlled wheelchairs would be a preferable alternative to joystick-controlled wheelchairs.

Keywords

Wheelchair navigation gesture image processing elderly

1. Introduction

The commonly available mechanism for navigation of a powered wheelchair is the joystick control method [29, 30, 31, 32]. A survey conducted (Section 5) shows that even though users are uncomfortable with the joystick control of their wheelchair, they feel compelled to use joysticks as it is the only available option to steer their wheelchair. The survey revealed users who discontinued usage of the joystick-controlled wheelchair due to the difficulty in exertion of manual force to steer the wheelchair. The authors in [1, 3, 4, 6, 28] discuss hand-gesture-based control methods, although not specifically for navigation of wheelchairs; they differ in the techniques adopted for identification of gestures. Similar studies reported in [16, 17] stop at evaluation of gesture recognition approaches, in a stand-alone setting, rather than being integrated into a viable system [37] reports of a research on physical features of wheelchair like seat adjustment [36], and ergonomically reconfigurable Powered Wheelchairs (PW). Gesture-based navigation systems are perfect for the differently-abled users who have problems with their motor control, i.e. limb movements [5]. Head gesture was used to control wheelchairs, as noted in [2]. Various assistive technologies, for different categories of users, include assistive robots [33]. Authors in paper [8] discussed a novel method to control a non-holonomic wheeled mobile robot, by introduction of a robust adaptive-controller, backed by powerful neural networks. Papers [7, 11, 12, 13, 38] reviewed brain-controlled motorized wheelchairs, by employment of electro-encephalogram (EEG) signals of the user, for navigation of a wheelchair. The authors in paper [14] explored a control method, employing the human tongue, while a control system providing visual stimuli, involving eye movements, was proposed in paper [15]. Paper [16] reported an investigation of a vision-based hand gesture recognition framework, for development of desktop applications. In papers [19, 20], the authors discussed vision-driven wheelchair and automotive interfaces, for people with mobility issues. Hidden Markov Model- (HMM-) based, and Finite State Machine based hand gesture recognitions were used in [21, 22, 2, 23, 24, 25, 26, 27]. Other navigation methods reported in [9] employed Kalman-based active observer controller for wheeled mobile robots. As reported in [10], motion control was based on quarry of environmental information. Our earlier work on this is presented in [40, 41, 42].

Figure 1.

Mudra system architecture.

In the Mudra system, we defined four hand gestures: forward, reverse, right, and left, as is shown in Fig. 1. The system can be tailor-made, conforming to the degree of disability in either of the wheelchair-user’s hands. The user can choose the most comfortable hand gestures for control of the wheelchair. User-adaptability was a key performance objective of the Mudra concept, so that the human foot can be substituted for the hand, for users who find their feet more pliable. The remaining part of this paper is organized as follows: the Mudra system architecture is explained in Section 2, followed by the modelling of the human arm in Section 3; Section 4 covers design verification of the Mudra system, and in Section 5 we discuss the inference from the experimental results.

2. System architecture

The proposed Mudra system contains a small camera that can track slight movements of the user’s fingers, and decipher the direction in which the user wishes to travel. The vision-based Mudra system is a robust, gesture recognition engine for real-time control of a wheelchair robot. Figure 1 shows a diagram of the Mudra system architecture, which is comprised of four (4) key subsystems: Gesture Capture Module (GCM), Gesture Recognition Module (GRM), Hand Interface Module (HIM) and Motor Control Module (MCM). Collaborative interplay of these components facilitates reliable navigation of the wheelchair robot. The GCM module includes an infra-red (IR) camera, an IR LED array, and a diffuser mask, enclosed in a box setup to capture the hand gestures. The GRM is comprised of a microprocessor unit that recognizes the hand gestures, and a Microcontroller unit is the HIM to interface to the MCM. The MCM module drives the motors. The IR camera, within the GCM, captures images of gestures, by the wheelchair-user, while MATLAB and Statistics Toolbox are marshalled for image processing of the gestures. The four Mudra hand-gestures are shown in Fig. 2. The safety module uses ultrasonic sensors to detect obstacles.

Figure 2.

Hand gestures.

3. Modeling human arm effort

A kinematic model of manipulation of finger gaits was proposed in [34], while skill evaluation of human operators, for mobile working machines, has been proffered in [35]. We have developed simpler models of the human arm, in action, emulating the usage of the gesture-based control as well as a conventional joystick control, as shown in Fig. 3a–j. The different parts of an arm are represented as follows: ‘H’ for hand, ‘F’ for the forearm, ‘A’ for the arm. The joints are labelled as ‘W’ for wrist, ‘E’ for elbow and ‘S’ for shoulder joint. Figure 3a–e represent various positions of the arm, when the user makes simple gestures for steering the wheelchair, while Fig. 3f–j show the arm positions, when the user is using a joystick for navigation of a PW. In Fig. 3a–e, $\varphi$ represents the forces acting on the hand when the user makes various gestures, and $\theta$ represents the angles at various joints. Then the effort $e_{g}$ is simply defined as,

$\displaystyle e_{g}=f(\varphi,\theta)$ (1)

$\varphi$ and $\theta$ vary as per the gestures made by a wheelchair-user. Similarly when the user uses a joystick to steer a PW, $\varepsilon$ represents the force acting on the hand and $\theta$ represents the angles at various joints. Then the effort $e_{j}$ is defined as,

$\displaystyle e_{j}=f(\varepsilon,\theta)$ (2)

$\varepsilon$ and $\theta$ vary as per movement of the joystick.

Figure 3.

Human arm model, in action, emulating gesture control and joystick control. Figure 3a through 3e show the forces acting on the hand, the angle at joints while the user is performing gestures: (a) when arm is at rest, (b) when user performs forward gesture, (c) for reverse gesture, (d) left gesture and (e) right gesture. Similarly Fig. 3f through 3j show the forces and angles when user uses joystick: (f) when holding Joystick, (g) moving joystick forward, (h) moving reverse, (i) moving left and (j) moving right.

In addition, the different forces and joint angles acting on the arm, when the user places the arm over the smooth platform, for any gesture, are shown in Fig. 3a–e.When the user places the arm over the platform, and is yet to make any gesture, there is only the downward force $\varphi_{1}$ of the hand acting downwards. To keep the model as simple as possible, forces on other parts of the arm were ignored. $\theta_{1}$ is the elbow angle and $\theta_{2}$ is the shoulder angle. When the user makes forward gestures, the respective forces and joint angles are shown in Fig. 3b. $\varphi_{11}$ is the downward force and $\varphi_{12}$ is the force in the forward direction of the movement of the hand. In case of a conventional joystick, multiple forces come into play, when a user has to merely grasp the joystick by the hand, even without moving it. This may be seen in Fig. 3f, with a hand H shown as a curved line, to indicate the user holding the joystick, where $\varepsilon_{1}$ is the force acting downwards; $\varepsilon_{2}$ and $\varepsilon_{3}$ are the forces acting when the hand is clasping the joystick. The different forces acting on the hand and angles of the joints are summarized in Table 1, for both gesture controlled, and joystick-controlled PW. We can see that the effort required for joystick control is greater than effort required for gesture control i.e. $e_{j}>e_{g}$ .

Table 1

Summary – gesture control and joystick control human effort

Hand position	Gesture control		Joystick control
	Force	Angle	Force	Angle
Rest	$f(\varphi_{1})$	$\theta_{1},\theta_{2}$	$f(\varepsilon_{1},\varepsilon_{2},\varepsilon_{3})$	$\theta_{1},\theta_{2}$
Forward	f( $\varphi_{11},\varphi_{12})$	$\theta_{1}+\alpha,\theta_{2}+\beta$	$f(\varepsilon_{11},\varepsilon_{12},\varepsilon_{13},\varepsilon_{1})$	$\theta_{1}+\yen,\theta_{2}+\yen,\theta_{3}$
Reverse	f( $\varphi_{21},\varphi_{22})$	$\theta_{1}-\alpha,\theta_{2}-\beta$	$f(\varepsilon_{21},\varepsilon_{22},\varepsilon_{23},\varepsilon_{2})$	$\theta_{1}-\yen,\theta_{2}-\yen,\theta_{3}$
Right	$f(\varphi_{31},\varphi_{3})$	$\theta_{1},\theta_{2}$	$f(\varepsilon_{31},\varepsilon_{32})$	$\theta_{1},\theta_{2},\theta_{5}$
Left	$f(\varphi_{41},\varphi_{4})$	$\theta_{1},\theta_{2}$	$f(\varepsilon_{41},\varepsilon_{42})$	$\theta_{1},\theta_{2},\theta_{6}$

Figure 4.

Hand movements with respect to the wrist.

4. Experimental setup and evaluation

4.1 System customization

Prior to deployment of the Mudra system for navigation of wheelchairs, the different hand gestures shown in Fig. 1, have to be pre-defined and stored for each user, as the sizes and shapes of the hands, and fingers, are unique to each user. This process is called customization. A customized Mudra system enables a user to have a personalized set of image templates, according to the user’s comfort. The user can use a single finger or multiple fingers for customization. Even the foot can be used for recognitions of gestures. A healthy person can move the hand over a flat surface, either left or right, with normal ulnar deviations (right side) ranging from 30 ${}^{\circ}$ to 50 ${}^{\circ}$ , and normal radial (left side) deviation of 20 ${}^{\circ}$ [18], as depicted in Fig. 4.

Figure 5.

Gesture image correlation value vs. mudra wheelchair response time.

4.2 Fixing correlation value

Determining the best fit for the hand gesture images correlation value of the image comparison algorithm used in the GRM, is the next step. For six different values of correlation, the wheelchair response times for forward, reverse, right and left gestures were calculated and plotted, as may be seen in Fig. 5. The best response time for any one of the four gestures always corresponds to a correlation value of 0.65, and this value was chosen as the threshold for correlation.

4.3 Hand gesture recognition

The tests were carried out in several phases. In the first phase, the GCM/GRM modules were tested with 9 users, for appraisal of accurate recognition of the hand gesture, with respect to the user’s intended movement of the wheelchair. In this phase, the users were asked to perform the same hand gesture 25 times, the Mudra system’s response was observed to assess whether the gestures were decoded consistently. The users performed various combinations of gesture changes (reverse to forward, forward to reverse, brake to forward etc.), and each combination was repeated 25 times,; the success was plotted as shown in Fig. 6. The average measure of success of each of the gesture change was assessed as follows: Reverse to Forward – 97.3%; Forward to Reverse – 97.7%; Forward to Right – 97.3%; Right to Forward – 97.3%; Forward to Left – 96.8%; Left to Forward – 97.8%; Left to Brake – 97.3%; Right to Brake – 99.1%; Forward to Brake – 98.2% and Reverse to Brake – 99.1%. The successful outcomes of the assessment tests bear testimony to the Mudra System’s effectiveness.

Figure 6.

Accuracy evaluation of hand gestures.

Figure 7.

Gesture inputs vs. Mudra system response time.

4.4 Testing GCM/GRM

In the second phase, the GCM/GRM modules were tested, with 25 users, for determination of the response time of the GRM, when the user changes from one hand gesture to another, as shown in Fig. 7. In this phase, the response time is defined as MATLAB processing delay. The observed standard deviations (in seconds) for the gesture changes were as follows: from brake to reverse – 006 $\times$ 10 E-03, reverse to forward – 007 $\times$ 10 E-03, brake to right – 004 $\times$ 10 E-03, and brake to left: 006 $\times$ 10 E-03. The standard deviation values indicate that response timings are not dispersed far apart from their mean value.

4.5 Mudra system evaluation

A survey was conducted, at the Department of Physical Medicine and Rehabilitation, Amrita Institute of Medical Sciences (AIMS), Kerala, India, to assess the preferences of wheelchair patients in order to derive their key requirements. The rehabilitation department, at AIMS, treats patients with neck and spinal-cord injuries, and stoke patients. The survey questionnaire was comprised of the following questions:

1.
How many patients, to whom a wheelchair is suggested, visit this clinic/hospital each day?
2.
Which factors are considered when prescribing a wheelchair?
3.
Do people request a specific type of wheelchair or does the Rehab staff recommend particular models based on the patients’ physical condition?
4.
Do conventional wheelchairs no longer meet the patients’ needs efficiently?
5.
What are the challenges reported by patients during their use of wheelchairs?
6.
What are the expected desired improvements in the design features of wheelchairs for patients’ utmost satisfaction?
7.
In what current situations do joystick-based wheelchairs fail?
8.
Has usage of wheelchairs been a factor in excessive wear or breakdown of the patients’ skin?
9.
What fraction of patients prefers using manual wheelchairs, even when they could use autonomous wheelchairs?
10.
What criteria is used to recommended a wheelchair to a patient?
11.
Do patients buy or rent a wheelchair?
12.
How often are the patients recommended to change the wheelchair?
13.
What type of wheelchair (manual or powered) is commonly offered to the patients?

The survey questions were answered by Dr. Ravi Sankaran, Asst. Professor, at the department. Analysis of the survey-responses led to wheelchairs being prescribed for three patients, on average, every day. Prior to recommendation of wheelchairs, the doctors considered the following factors pertinent to each patient: primary diagnosis, co-morbidities, functional status of hands or legs, socio-economic status, and potential of patient’s return to work. Some patients needed hand therapy before they were able to use the Mudra wheelchair. The existing (joystick-based) wheelchairs are not suitable for stroke patients with severe disabilities, like C1 and C2 category of quadriplegics [39], who cannot move their hands. When asked about the side effects of using wheelchairs, the doctors at AIMS reported that heat may be produced because of proximity to the battery, placed underneath the seat, which may damage the skin of wheelchair users.

With the Mudra system, the response time is a combination of three delays: frame delay, computational delay and hardware delay. Frame delay is the delay for the image capture and is specific to webcam used. Computational delay is that of the algorithm employed and the hardware delay is a constant value, which is independent of the algorithm used. In order to assess Mudra’s performance of response time, a commercial joystick-controlled wheelchair, Ostrich Mobility Verve Lx Electric Wheelchair (WheelchairX) was procured. Seven users’ feedback were taken into consideration. As shown in Table 2, each user was asked to repeat four gestures, 25 times, using the Mudra system. These seven users were asked to repeat the same test procedure with WheelchairX. The average response timings of each user (A1 to A7) are given in Table 2 (for the Mudra System), and Table 3 (commercial joystick-controlled wheelchair). Table 4 shows the comparison of average timings of Tables 2 and 3. During brake to forward and brake to reverse gesture changes, Mudra’s response time was 0.2 seconds, and 0.3 seconds lesser than that of WheelchairX, respectively.

Table 2
Mudra response time (in seconds)

Gesture change A1 A2 A3 A4 A5 A6 A7

Brake to forward 0.423 0.467 0.521 0.430 0.493 0.581 0.511

Brake to reverse 0.543 0.487 0.474 0.625 0.653 0.503 0.530

Brake to right 0.781 0.790 0.823 0.749 0.951 0.576 0.953

Brake to left 0.612 0.672 0.862 0.703 0.701 0.826 0.748

Table 3
WheelchairX response time (in seconds)

Gesture change A1 A2 A3 A4 A5 A6 A7

Brake to forward 0.676 0.571 0.504 0.672 0.702 0.683 0.724

Brake to reverse 0.975 0.774 0.874 0.774 0.976 0.802 0.702

Brake to right 0.774 0.841 0.874 0.505 0.568 0.702 0.762

Brake to left 0.841 0.774 0.874 0.712 0.668 0.718 0.688

Table 4
Mudra vs. WheelchairX response times (in seconds)

Gesture change WheelchairX Mudra

Brake to forward 0.647 0.489

Brake to reverse 0.839 0.545

Brake to right 0.718 0.803

Brake to left 0.753 0.732

Table 5
Group 1 user responses

Joystick use in years Joystick rating Would prefer different control Mudra rating Prefers gesture control

2 4 Yes 4 Yes

$<$ 1 1 Yes 5 Yes

2.5 3 Don’t know 2 Can’t tell

2 5 No 4 No

1 2 Yes 4 Yes

1.5 4 Don’t know 4 Can’t tell

$<$ 1 0 Yes 5 Yes

$<$ 1 1 Yes 5 Yes

3 4 Yes 4 Yes

1 2 Yes 3 Yes

Two groups, Groups 1 and 2 of 10 volunteers in each group were selected for the final phase of evaluation after getting their consent to participate in this evaluation. The participants in this phase of evaluation volunteered themselves to take part as test subjects. Group 1 participants were experienced (ranging from less than a year to three years) in using joystick wheelchairs, while Group 2 members never used any kind of wheelchairs.

On using the Mudra system, the Group 1 users were asked to fill out a survey, whose results are tabulated in Table 5. It was quite a surprise that one of the Group 1 member allocated a rating of 0 for the joystick wheelchair. This user didn’t have any other option and was compelled to use the joystick-based powered wheelchair. Only one of the Group 1 users gave the wheelchair a rating of 5. In Group 2, 8 of the 10 users assigned ratings of at least 4 for the Mudra system, which imply that people are looking out for better wheelchair-options. In addition, seven of them preferred gesture based wheelchair control, like a Mudra, to a joystick-based wheelchair control; 2 of them had no preferences. The second group of participants who have never used any kind of powered wheelchair were given both the joystick-based control WheelchairX, and a Mudra PW and were asked to rate both PWs on a scale of (0–5), where 5 is the highest score. All of the Group 2 members preferred Mudra to the joystick-based controls. The results are tabulated in Table 6.

Table 6
Group 2 user response

User Joystick control rating Mudra rating Preference joystick/gesture

1 4 5 Gesture

2 4 5 Gesture

3 3 4 Gesture

4 4 4 Can’t tell

5 2 4 Gesture

6 4 5 Gesture

7 3 3 Can’t tell

8 3 5 Gesture

9 3 5 Gesture

10 2 4 Gesture

Figure 8.
Mudra model.

At the end of the final phase of the evaluation, the questionnaire given to group 1 users is shown below.

1.
How long have you been using a joystick-based wheelchair?
2.
Are you comfortable with the current joystick controlling mechanism?
3.
Are you comfortable with the current speed of the wheelchair?
4.
How often do you charge the wheelchair battery?
5.
Do you prefer an addition of any of the following gadgets to your wheelchair? (Bottle holder, mobile charger, horn, head light, tail lights)
6.
Do you have any complaints on the following sub-systems of the wheelchairs? (Battery, joystick, motor, frame)
7.
Do you like to add a multi-purpose pad for eating food or reading books?
8.
Are you comfortable with the present placement of the control module?
9.
Which orientation of control module is more comfortable?
10.
Does your current wheelchair have any breaking mechanisms?
11.
What kind of control do you prefer for speed control? (On/off, knob, push button)
12.
What rating (0–5) can you give for the joystick-based wheelchair? (___/5)
13.
What is the overall satisfaction with the gesture-based wheelchair?

(Great, good, moderate, unsatisfactory)
14.
Do you like to replace your current joystick-controlled wheelchair with a gesture-based wheelchair?

The Mudra model is shown in Fig. 8. Figure 9 shows the Mudra PW as implemented and tested by the authors of this paper. Figure 10a shows one of the paths traversed by Mudra on this corridor. One of the test spaces, used to test Mudra, is shown in Fig. 10b. This is the corridor in one of the floors of our university. Figure 10e shows a user getting inside the lift. Figure 10c and d show Mudra with different users.

Figure 9.
Mudra prototype.

Figure 10.
(a) A typical path taken by Mudra in the university corridor. (b) The corridor was used for one of the tests. (c) A user testing Mudra in the corridor. (d) A user testing Mudra inside the research lab. (e) A user getting into the lift.

5. Conclusion

Gesture change	A1	A2	A3	A4	A5	A6	A7
Brake to forward	0.423	0.467	0.521	0.430	0.493	0.581	0.511
Brake to reverse	0.543	0.487	0.474	0.625	0.653	0.503	0.530
Brake to right	0.781	0.790	0.823	0.749	0.951	0.576	0.953
Brake to left	0.612	0.672	0.862	0.703	0.701	0.826	0.748

Gesture change	A1	A2	A3	A4	A5	A6	A7
Brake to forward	0.676	0.571	0.504	0.672	0.702	0.683	0.724
Brake to reverse	0.975	0.774	0.874	0.774	0.976	0.802	0.702
Brake to right	0.774	0.841	0.874	0.505	0.568	0.702	0.762
Brake to left	0.841	0.774	0.874	0.712	0.668	0.718	0.688

Gesture change	WheelchairX	Mudra
Brake to forward	0.647	0.489
Brake to reverse	0.839	0.545
Brake to right	0.718	0.803
Brake to left	0.753	0.732

We have reported on systematic evaluation of the joystick control method of powered wheelchairs available in the market, including their features and prices. Successful in gesture recognition, response time calculations based on Intel I5 processor board, and comparisons with a joystick-controlled wheelchair were carried out. Results of evaluation of these units by 20 users, ten of whom were experienced in using joystick-controlled wheelchairs, and ten of them without any prior experience in using wheelchairs, validate the robustness of the proposed Mudra system. We named our model – ‘Mudra’, a Sanskrit word which means Gesture in English. A gesture based Mudra system demonstrated the feasibility of simple hand gesture control for navigation of powered wheelchairs. Even though the option of gestures by foot were not captured and tested, the success rate of hand gesture controls indicates that the GCM box, can be placed at the foot plate position in the wheelchair, for certain category of users who can only make use of their legs, for navigation of the wheelchair. Our survey also points that wheelchair users are looking for a change in their control options. We believe that Mudra systems will be preferred by the intended users in their daily lives, for convenience and better user experience.

Footnotes

Acknowledgments

We thank the Humanitarian Labs and the Department of Electronics and Communication of Amrita Vishwa Vidyapeetham, Amritapuri, Kollam for providing all the necessary lab facilities, and a highly encouraging work environment, which were key factors towards completion of this research project.

Conflict of interest

None to report.

References

Chen

Zhang

Zhou

. Hand Gesture Recognition Framework and Wearable Gesture-Based Interaction Prototype for Mobile Devices. IEEE Transactions On Human-Machine Systems April 2014; 44(2).

Yoshinori

Murashima

Shimada

Shirai

. Understanding and learning of gestures through human-robot interaction. Intelligent Robots and Systems, 2000 (IROS 2000). Proceedings. 2000 IEEE/RSJ International Conference on. vol. 3, pp. 2133-2138.

Manresa

Varona

Mas

Perales

. Real –Time Hand Tracking and Gesture Recognition for Human-Computer Interaction. Electronic Letters on Computer Vision and Image Analysis 2000; 1-7, ELCVIA ISSN: 1577-5097.

. A Fuzzy Rule-Based Approach to Spatio-Temporal Hand Gesture Recognition. IEEE Transactions on Systems, Man, and Cybernetics 30(2).

Jia

Yuan

TLK

. Head gesture recognition for hands-free control of an intelligent wheelchair. Industrial Robot: An International Journal; 34(1): 60-68.

Metcalf

Robinson

Malpass

Bogle

Dell

Harris

Demain

. Markerless Motion Capture and Measurement of Hand Kinematics: Validation and Application to Home-Based Upper Limb Rehabilitation. IEEE Transactions On Biomedical Engineering August 2013; 60(8).

Ituratte

Antellis

Kubler

Minguez

. A Non-invasive brain actuated wheelchair based on a P300 Neuropsychological protocol and Automated Navigation. IEEE Transactions on Robotics June 2009; 25(3).

Dong

Kuhnert

. Robust adaptive control of non-holonomic mobile robot with parameter and non-parameter uncertainties. IEEE Transactions on Robotics 2005; 21(2): 261-266.

Katsura

Ohnishi

. Semiautonomous wheelchair based on quarry of environmental information. IEEE Transactions on Industrial Electronics 2006; 53(4): 1373-1382.

10.

Tashiro

Murokami

. Step passage control of a power assisted wheelchair for a caregiver. IEEE Transactions on Industrial Electronics April 2008; 25(4).

11.

Montesano

Díaz

Bhaskar

Minguez

. Towards an Intelligent Wheelchair system for users with cerebral palsy. IEEE Transactions on Neural Systems and Rehabilitation Engineering April 2010; 18(2).

12.

Luca

Leeb

Tavella

Perdikis

Millán

JDR

. The role of shared-control in BCI-based telepresence. In Systems Man and Cybernetics (SMC), 2010 IEEE International Conference on, pp. 1462-1466.

13.

Johan

Millán

JDR

Vanacker

Lew

Galán

Ferrez

Brussel

Nuttin

. Adaptive shared control of a brain-actuated simulated wheelchair. In Rehabilitation Robotics, 2007. ICORR 2007. IEEE 10th International Conference on, pp. 408-414.

14.

Jeonghee

Huo

Minocha

Holbrook

Laumann

Ghovanloo

. Evaluation of a smartphone platform as a wireless interface between tongue drive system and electric-powered wheelchairs. IEEE Transactions on Biomedical Engineering 2012; 59(6): 1787-1796.

15.

Omer

Sherkat

Lotf

. TeleGaze: Teleoperation through eye gaze. In Cybernetic Intelligent Systems, 2008. CIS 2008. 7th IEEE International Conference on, pp. 1-6.

16.

Yuan

. Contour model-based hand-gesture recognition using the Kinect sensor. IEEE Transactions on Circuits and Systems for Video Technology 2014; 24(11): 1935-1944.

17.

Theodoridis

Maier

. Kinect Enabled Monte Carlo Localisation for a Robotic Wheelchair. Intelligent Autonomous Systems 12, AISC 193, pp. 153-163, Springer-Verlag Berlin Heidelberg 2013.

18.

Kitsoulis

Paraskevas

Iliou

Kanavaros

Marini

. Clinical study of the factors affecting radioulnar deviation of the wrist joint. Journal of Negative Results in Medicine, DOI: 10.1186/1471-2474-11-9.

19.

Ohn-Bar

Trivedi

. Hand Gesture Recognition in Real-Time for Automotive Interfaces: A Multimodal Vision-based Approach and Evaluations. IEEE Transactions on Intelligent Transportation Systems 2014.

20.

Manresa

Varona

Mas

Perales

. Real –Time Hand Tracking and Gesture Recognition for Human-Computer Interaction. Electronic Letters on Computer Vision and Image Analysis 2000; 1-7, ELCVIA ISSN: 1577-5097.

21.

Yamato

Ohya

Ishii

. Recognizing human action in time sequential images using hidden Markov model. Proc. IEEE Int. Conf. Comput. Vis. Pattern Recogn., Champaign, IL, 1992; pp. 379-385.

22.

Starner

Pentland

. Real-time American Sign Language recognition from video using hidden Markov models. MIT Media Lab., MIT, Cambridge, MA, Tech. Rep. TR-375, 1995.

23.

Weaver

Starner

Pentland

. Real-time American Sign Language recognition using desk and wearable computer based video, IEEE Trans. Pattern Anal. Mach. Intell 1998; 33(12): 1371-1378.

24.

Yoon

Soh

Bae

Yang

. Hand gesture recognition using combined features of location, angle and velocity. Pattern Recogn 2001; 34: 1491-1501.

25.

Ramamoorthy

Vaswani

Chaudhury

Banerjee

. Recognition of dynamic hand gestures. Pattern Recogn 2003; 36: 2069-2081.

26.

Yeasin

Chaudhuri

. Visual understanding of dynamic hand gestures. Pattern Recogn 2000; 33: 1805-1817.

27.

Hong

Turk

Huang

. Gesture modelling and recognition using finite state machines. Proc. 4th IEEE Int. Conf. Autom. Face Gesture Recogn. Grenoble, France, Mar. 2000, pp. 410-415.

28.

Megalingam

Nair

Prakhya

Mohan

. Gest-BOT’ – A Highly Convenient Locomotive Solution for the Elderly and Physically Challenged. Global Humanitarian Technology Conference (GHTC), 21–24 October in Seattle, Washington, USA, IEEE Xplorer. pp. 350-355; DOI: 10.1109/GHTC.2012.83.

29.

Meyer

Just

DeCarlo

Zefran

Oishi

. Notch filter and MPC for powered wheelchair operation under Parkinson’s tremor. American Control Conference (ACC), 2014; pp. 4114-4120; DOI: 10.1109/ACC.2014.6859442.

30.

Wada

Kameda

. A joystick type car drive interface for wheelchair users. The 18th IEEE International Symposium on Robot and Human Interactive Communication. 2009. RO-MAN 2009. DOI: 10.1109/ROMAN.2009.5326085.

31.

Tsai

Wen

. Hsueh Synchronized motion control for 2D joystick-based electric wheelchair driven by two wheel motors. 2012 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM) pp. 702-707; DOI: 10.1109/AIM.2012.6266027.

32.

Rabhi

Mrabet

Fnaiech

Gorce

. Intelligent joystick for controlling power wheelchair navigation. 2013 3rd International Conference on Systems and Control (ICSC), pp. 1020-1025, DOI: 10.1109/ICoSC.2013.6750981.

33.

Park

Lee

Kim

Bien

. A Steward robot for Human-Friendly Human-Machine Interaction in a smart house environment. IEEE Transactions On Automation Science and Engineering January 2008; 5(1): DOI: 10.1109/TASE.2007.911674.

34.

. A kinematic model of finger gaits by multifingered hand as hybrid automation. IEEE Transactions On Automation Science and Engineering July 2008; 5(3): DOI: 10.1109/TASE.2007.911686.

35.

Tervo

Palmroth

Koivo

. Skill evaluation of human operators in partly automated mobile working machines. IEEE Transactions On Automation Science and Engineering January 2010; 7(1): DOI: 10.1109/TASE.2009.2025505.

36.

Lee

Yao

Kuo

. Development of a self-adjustable Power Wheelchair. Proceedings of 2011 International Conference on System Science and Engineering, 2011, DOI: 10.1109/ICSSE.5961953.

37.

Zou

Yuan

Ren

. A Reconfigurable Wheelchair/Bed system for the elderly and handicapped. Proceedings of 2012 IEEE International Conference on Mechatronics and Automation DOI: 10.1109/ICMA.2012.6284380.

38.

. EEG-based hybrid BCIs and Their Applications. 2015 3rd International Winter Conference on Brain Computer Interface (BCI), pp. 1-4, DOI: 10.1109/IWWBCI.2015.7073035.

39.

https://www.sci-info-pages.com/levels.html.

40.

Megalingam,

Chacko

Kumar

Jacob

Gautham

. Gesture controlled wheel chair using IR-LED TSOP pairs along with collision avoidance. International Conference on Robotics and Automation for Humanitarian Applications, RAHA 2016 – Conference Proceedings, ISBN: 978-150905203-5, DOI: 10.1109/RAHA.2016.7931872.

41.

Megalingam

Sreekanth

Govardhan

Teja

Raj

. Wireless gesture controlled wheelchair. 2017 4th International Conference on Advanced Computing and Communication Systems, ICACCS 2017, ISBN: 978-150904559-4, DOI: 10.1109/ICACCS.2017.8014621.

42.

Megalingam

Rangan

Krishnan

Edichery Alinkeezhil

. IR Sensor-Based Gesture Control Wheelchair for Stroke and SCI Patients, ISSN: 1530437X, DOI: 10.1109/JSEN2016.2585582.

Joystick use in years	Joystick rating	Would prefer different control	Mudra rating	Prefers gesture control
2	4	Yes	4	Yes
$<$ 1	1	Yes	5	Yes
2.5	3	Don’t know	2	Can’t tell
2	5	No	4	No
1	2	Yes	4	Yes
1.5	4	Don’t know	4	Can’t tell
$<$ 1	0	Yes	5	Yes
$<$ 1	1	Yes	5	Yes
3	4	Yes	4	Yes
1	2	Yes	3	Yes

User	Joystick control rating	Mudra rating	Preference joystick/gesture
1	4	5	Gesture
2	4	5	Gesture
3	3	4	Gesture
4	4	4	Can’t tell
5	2	4	Gesture
6	4	5	Gesture
7	3	3	Can’t tell
8	3	5	Gesture
9	3	5	Gesture
10	2	4	Gesture