Abstract
A tele-operated robot stereo vision system is used for stretching out the operator’s eye-hand motion and its distance based co-ordination with experts. The major challenge is the reduction of communication delay by using effective decisions to avoid tele-operation instability. This problem can be handled effectively by using the principles of Augmented Reality which provides facilities for superimposing virtual objects onto the real video images of the workspace to create a simulation plan in the client system. In this paper, we propose a new feature selection algorithm called Fuzzy Rules and Information Gain Ratio based Feature Selection Algorithm for selecting the optimal number of features from the full set of available features. Also, a new Fuzzy Rule based Neuro-Genetic Classification Algorithm is proposed in this paper for classifying the augmented images more accurately. The main advantages of the proposed model are reduction in classification and communication time and increase in decision accuracy.
Introduction
Augmented Reality (AR) is a real time live environment which works by adding the digital information on the images or video to form the combination of real and virtual objects in the real environment [8]. It is the extended version of virtual reality and it creates the simulated data environment using computer based systems. The concept of augmented reality enables the interaction between the real and computer generated interfaces for providing effective services. The real and virtual objects can be seen either through a head mounted display or as an image captured by a camera. The main functions of the AR system are (i) it combines the real and virtual objects in a real environment. (ii) It runs interactively in real time. (iii) It aligns virtual objects against the real objects. Computer vision provides techniques for automatically analyzing the camera captured images and it has the potential to solve different challenging problems in image processing and Augmented Reality. In indoor AR applications, marker based applications control the real time tracking with fast and accurately by artificial markers placed in the real environment [6].
The applications of Augmented Reality include health care systems, gaming, manufacturing systems, sports and marketing. AR is categorized into marker based AR, marker-less AR, Protection based AR and super imposition based AR [3]. Wherever the high operational accuracy is needed in the industries, autonomous robotics systems are required. In this work, an Augmented Reality system is proposed for handling robotic arm motions through the analysis of images. For this purpose, images are captured in real time, features are selected from the corresponding images and classification algorithms are applied to predict the next optimal move by the robot. Moreover, we are proposing new techniques which are useful to control the stereo vision tele-operated robots in order to reduce the self-motion misalignment and to get accurate object size and shape for performing the control more accurately. The marker based AR uses visual markers such as Quick Response (QR) code or two dimensional bar codes. The camera captured images obtained from that images are recognized using the QR code and they produce the result patterns by using sensor readers. On the other hand, the marker-less AR is mainly used for performing location based and position based applications by using the Global positioning system, digital compass and accelerometer. In the smart phones, marker-less AR systems are used for detecting the current location.
In projection based AR, an artificial light is projected into a real world surface. It allows the humans to interact with the real world surface by projecting the lights. Finally, the super imposition based AR replaces the real view of an object with augmented view of that object. In this work, the super imposition AR model is used for making the object view. Moreover, fuzzy neural networks are used in this work and hence a new algorithm for feature extraction is proposed in this paper. With the advances of tele-robotics, exploration and access to the hazardous environment with high radiation are minimized since the robot supports the human operator to handle the situation easily by allowing the controlling of it from a remote location. In this scenario, the augmented reality compensates the confusion caused by time delay by allowing the operators to observe the real working and also the virtual simulation results simultaneously for rectifying the measures effectively. To handle the uncertainty in the classification process, fuzzy neural networks are employed for performing the feature selection process and classification tasks are carried out using fuzzy rule based neuro-genetic classification algorithm [12, 35]. In addition, the proposed work has been compared with decision trees, K-Nearest Neighbor (KNN) based classification and Support Vector Machines (SVMs) for effectively selecting the features. In addition, the features are represented using the K-Dimensional tree data structure in order to search the data very fast from the feature database. Therefore in this paper, we propose a new feature selection algorithm called Fuzzy Rules and Information Gain Ratio based Feature Selection Algorithm for selecting the optimal number of features from the full set of available features. In addition, a new classification algorithm called Fuzzy Rule based Neuro-Genetic Classification Algorithm is proposed in this paper for classifying the augmented images more accurately.
The novelty of the paper is the proposed feature selection algorithm is that it uses information gain and fuzzy rules for effective decision making. By the use of fuzzy rules, the proposed model is able to select optimal number of features from the set of full features available in the dataset. This feature reduction has been tested with classification algorithms and found that the optimal number of features selected using the proposed feature selection algorithm reduces the classification time and at the same time which increases the classification accuracy by handling the over fitting problem faced by the classifiers. The performance of the proposed algorithm has been compared with other existing features selection algorithms and it is proves through the experiments that the proposed feature selection algorithm contributes more effectively in the decision making process. Moreover, the new classification algorithm proposed in this paper uses fuzzy rules for weight adjustments along with genetic algorithms in the neural classifier to make a new fuzzy rule based neuro-genetic classifier that enhances the classification accuracy. This classifier also uses fuzzy rules with genetic based optimization for enhancing the classification accuracy further. The combination of feature selection with this novel classifier provides an efficient decision making approach for augmented reality which is a new contribution in the area of augmented reality. In this proposed work, augmented reality applications with robots are taken and the classification problem is solved using fuzzy inference method.
The remainder of this paper is organized as follows: Section 2 provides a survey of related works. Section 3 explains the feature selection algorithm proposed in this work. Section 4 provides the results and discussion obtained from the selected features and full features with different classifiers. Section 5 gives conclusions on this proposed work and shows some possible future works.
Literature review
There are many works in the literature which discuss about the techniques for image analysis, feature selection and classification used in AR systems and other areas in which feature selection is performed before classification [4, 26]. Portela Sotelo et al. [21] developed new techniques to create an augmented reality environment with the help of a generic model and descriptor files. The authors proposed a 3D model based multiple object video tracking system to monitor a patient and the equipment present in a treatment room. Their model is based on the selection of features from filmed information present on the scene element by applying fuzzy logic. Moreover, they used the fuzzy rules for effectively merging the heterogeneous data. Therefore, the authors defined two fuzzy sets which are corresponding to visibility and displacement in the 3D space.
Ishan Rabbi and Sehat Ullah [14] presented an extended the marker tracking distance from layered marker. Hasan Ali and Baris [11] proposed a hyper-spectral image classification model using spatial-spectral filtering and feature extraction method. They combined the functional data feature extraction method with 3D Gaussian filter and 3D adaptive non-local means filter. Roland and Aural [24] developed a robot arm control using color recognition algorithms to move any physical reachable 3D space with the help of stereo vision. Similarly, many researchers worked in the areas of robotics, AR and computational intelligence [1, 34].
Salvatore Livatino et al. [25] proposed an augmented reality 3D graphics and stereoscopic view by using laser and visual sensors for enhancing the information. Their model is useful to perform 3D analysis on image data. Thierry Viville et al. [31] proposed a new robotic system which not only forecasts the 3D location and motion of a point target but also identifies the class of motion and model ruptures. Therefore, their work is useful for finding the locations of objects more accurately. Syed Islam et al. [28] presented a detailed model for developing an AR stereo vision system. They designed a 3D graphical arm and super imposed the video images using camera calibration and various registration modules. Also, they developed a motion activation algorithm and discussed about network data transmission and data security challenges.
Gao Xin et al. [10] proposed a new speed and precise registration method for handling dynamic stereo video and they achieved effective integration of virtual robot and real stereo scene. Rastogi et al. [23] developed a tele robotic AR system that implemented 3D view in the monitors. It uses position tracking through mechanical and optical sensors for camera calibration. Therefore, communication is possible through the use of sensors. Matthias Harders et al. [17] proposed new calibration methods for visuo-haptic integration and also developed a hybrid tracking technique for stable alignment of the augmentation. Their model is provided with a dispensed framework for ensuring low latency and component synchronization. Akash Bapat et al. [2] proposed a new marker-less approach for outward visual tracking of AR and Virtual Reality (VR) devices that attained high accuracy and low latency without the need for any off-device tracking elements. Takashi Okuma et al. [30] defined a new video see-through natural feature based 3D object tracking method for managing the wearable augmented reality systems. In their model, they combined database handling techniques with object tracking for making effective analysis of video data.
In the past, Omar Abu Arqub [18] proposed a novel algorithm for solving fuzzy Fredholm–Volterra integro differential equations using fuzzy rules. The author provided a significant contribution in the area of mathematical problem solving using fuzzy rules. However, similar ideas are necessary to be incorporated in the design of medical robots for making them intelligence with respect to decision making. Moreover, Omar Abu Arqub et al. [19] proposed a numerical method for solving fuzzy differential equation using Hilbert spaces. Their contribution is more significant in the areas of set topology and functional analysis in mathematics. Such an approach can be applied for solving robotic applications so that it is possible to perform accurate inferences through fuzzy reasoning. In addition, Omar Abu Arqub et al. [20] approached the solution of second order two point boundary value problems based on the kernel theory in which the authors solved the fuzzy equations more efficiently to provide fast convergence of the numerical problems through approximations. These types of mathematical solutions are useful to enhance the decision making process more efficiently with respect to time and accuracy of solutions.
Pulkit Khandelwal et al. [22] proposed a new algorithm for developing a marker-less AR. In their paper, they explained about the natural features based approach which can distinguish objects clearly and it is followed in the recognition of real life objects such as buildings, cars, trees, humans etc. Taihú Pire et al. [29] explained a real-time feature-based stereo system that is forceful and perfect in a variety of conditions for video processing. They presented a mature stereo system for robot localization which incrementally builds a point based sparse map representation of the workspace, using a stereo camera, and tracks the camera pose within it. Archana Kale et al. [3] proposed a new genetic algorithm based approach for feature selection and it was checked using a classification algorithm and proved that their model improved the classification accuracy. Similarly, many other researchers also developed decision making systems using soft computing techniques [5, 33].
Jianbin Qiu et al. [15] developed a neuro fuzzy control system for nonlinear systems with full state constraints and actuator hysteresis. They applied radial basis functions for solving the problems of errors and convergence in closed loop control systems. Their model is more suitable for enhancing the performance of robotic systems through effective mathematical modeling.
In spite of the presence of all these works in the literature, most of the works focused on AR based images for different applications rather than robotic applications. Moreover, for controlling the robot from a remote location, fast data communication is necessary [24]. Therefore, it is necessary to propose an efficient feature selection algorithm which improves the classification accuracy and thus aiding in providing fast decisions in order to reduce the communication delay. Finally, the robotic applications not only need AR support but also intelligent decision making algorithms for providing efficiency. Hence, a new feature selection algorithm is proposed in this paper which reduces the communication delay by supporting in the decision making through fast classification of images used in the robotic arm control.
System architecture
The architecture of the proposed feature extraction, selection and classification system is depicted in Fig. 1.
System Architecture.
It consists of nine main modules namely AR Image Data set, Camera Captured Image Data set, User Interface(UI) module, Preprocessing module, Classification module, the Decision manager, Fuzzy Rule manager, Fuzzy rule base and Feature Database.
In this work, a new feature selection algorithm called Fuzzy Rules and Information Gain Ratio based Feature Selection Algorithm (FIGRFSA) is proposed. It considers the information gain ratio for selecting optimal number of features from the set of 25 features consisting of shape, statistical, temporal, color, texture and spatial features. Moreover, the proposed model uses neuro-fuzzy classifier in which fuzzy logic is used for weight adjustment and genetic algorithms are used for finding the optimal number of features. This algorithm has been proposed using intelligent fuzzy rules and Information Gain Ratio for performing effective feature selection. In order to achieve this, each image from the data set DS is divided into n number of sub components namely C1,C2, ... Cn. The features Fi having the maximum number of non-zero values with high significance are chosen by the intelligent rules through weight adjustment in the neuro classifier where the Information Gain Ratio (IGR) is computed using Equations 1, 2 and 3 [27].
The steps of the proposed feature selection algorithm are as follows:
Fuzzy rules are used in both feature selection and classification phases. In feature selection, optimal number of features can be obtained by performing qualitative reasoning based on fuzzy rules. Therefore, the main motivation for using fuzzy rules is that it is used to handle uncertainty occurring in the dataset in an efficient way. Moreover, the machine learning algorithms are able to provide required classification accuracy only through the adjustments of its weights in the training phase. This motivates the use of fuzzy rule based approach in combination with the genetic algorithms for effective optimization of in the features.
Classification is an important process in machine learning and data mining, which is used to categorize every instance in the input dataset into various classes. In this work, classification is performed using the proposed Fuzzy Rule based Neuro-Genetic Classification Algorithm (FRNGCA), decision tree algorithm, K-NN classifier and Support Vector Machines (SVMs) classification algorithms. The neural network is designed in such a way that it has one input layer with ten nodes, two hidden layers and one output layer. Now, the fuzzy rules are applied for weight adjustment and also Bias value is introduced to enhance the classification accuracy. Finally, the optimal classification is checked using genetic algorithms. The steps of the proposed classification algorithm are as follows:
Figure 2 shows the Kinova Jaco 6 DOF robot, stereovision camera and Lap top controller system. Camera is positioned in front of the robot to capture the entire area of robot and the object. The captured images from the camera are stored in a database and are used by the proposed system for analysis and robot control. This system uses the FIGRFSA and FRNGA algorithms for efficient feature selection and classification of the image data.
Kinova jaco robot with stereo vision setup.
The experiments were conducted in an AR environment in which the camera, sensor and the robot were located in the same room. A human controller was monitoring the data collection and sending operations to the control stations for making effective decisions. The robot was allowed to perform its operations on its own most of the times by applying the proposed feature selection and classification algorithms for forming rules and to make decisions. In rare situations, the human controller was allowed to intervene in the operations of the robot.
Experimental setup
The experiments were conducted for 10 days in which each day the experiments were repeated 5 times each consisting of one hour duration. The experiments conducted in this work with the existing classification algorithms without feature selection, proposed classification algorithm without feature selection and then finally the classification using existing and proposed classification algorithms with feature selection. The list of features considered for classification which is known as the full set of features is shown in Table 1.
Full list of features
Full list of features
Table 2 shows the list of features selected from the images after applying the feature selection algorithm proposed in this paper. The experiments were repeated with different types of images pertaining to robotic arm movement for data collection and communication. The average values of information gain ratio were considered for selecting the optimal number of features from the full set of features. All other robot controlling processes are followed by these filtered features with efficiently.
List of selected features
These fifteen features are used for classifying the images captured by the stereo vision camera. It is useful to identify the robot current position and estimate the image data set for next position and actuates the robot towards the object by the feedback value of this system.
Figure 3 shows the classification accuracy without feature selection for the proposed classification algorithm and the existing classification algorithms such as K-NN, DT and SVM. Here, we have conducted five different experiments for evaluating the performance of the proposed classifier without feature selection.
Classification Accuracy without Feature Selection.
From Fig. 3, it can be seen that the performance of the proposed classifier is performing well when it is compared with the existing classifiers such as K-NN, DT and SVM.
Figure 4 show the classification accuracy with feature selection for the proposed classification algorithm and the existing classification algorithms such as K-NN, DT and SVM. Here, we have conducted five different experiments for evaluating the performance of the proposed classifier with feature selection.
Classification accuracy with Existing Feature selection algorithm.
From Fig. 4, it can be seen that the performance of the proposed model perform well when it is compared with the existing classifiers such as K-NN, DT and SVM with the proposed feature selection algorithm.
This is due to the fact that the use of effective and intelligent fuzzy rules, neural network and genetic algorithm. These are very helpful for making effective decisions on segmented images in training and testing process in the proposed model.
Figure 5 shows that classification accuracy analysis and comparison between the existing classifiers namely KNN, Decision Tree and Supported Vector Machine with the proposed fuzzy genetic neuro classification algorithm were the classification is made using the features selected by the proposed Fuzzy Rules and Information Gain Ratio based Feature Selection Algorithm. From Fig. 5, it can be observed that the classification accuracy is increased when the classification is carried out with the features selected using the proposed feature selection algorithm. This improvement is achieved due to the effectiveness of the proposed feature selection FIGRFSA algorithm.
Classification accuracy with proposed Feature selection algorithm.
Figure 6 shows the communication delay between the proposed model and the existing classification algorithms such as K-NN, DT and SVM with the proposed feature selection algorithm. Here, we have conducted five different experiments with various distances such as 20, 40, 60, 80 and 100 meter per second for evaluating the performance of the proposed model.
Communication Delay Analysis.
In this paper, an AR model is proposed for controlling the robot arm and to provide a robot arm planning system by proposing a fuzzy rule based approach for feature selection and classification. This model is useful to find the most important movement through image analysis and reduces the communication delay by increasing the classification accuracy. Therefore, a new feature selection algorithm called Fuzzy Rules and Information Gain Ratio based Feature Selection Algorithm (FIGRFSA) is proposed in this paper for selecting the optimal number of features from the full set of available features. In addition, a new classification algorithm called Fuzzy Rule based Neuro-Genetic Classification Algorithm (FRNGCA) is proposed in this paper for classifying the augmented images more accurately. From the experiments conducted in this work, the combination of feature selection and classification using the proposed model reduced the delay in communication and increase the decision accuracy through efficient classification. Future works in this direction can be the use of rough sets for handling uncertainty further and to reduce the communication delay. Moreover, the efficiency of communication can be improved by introducing intelligent agents and performing agent based communication for providing effective monitoring and control facilities.