Control a 2-Axis Servomechanism by Gesture Recognition Using a Generic WebCam

1. Introduction

The main objective of this work was the developing of a control system for a servomechanism, equipped with two linear motion axes, based on gesture recognition. With that purpose, we decided to use a generic webcam for the image acquisition process, and the following implementation philosophy: In the first phase, the learning phase, the control system should make the association of each hand gesture image to the desired control order. In the next phase, the working one, by comparing the correspondent order gesture image with the ones previously considered in the learning phase, the control system process the desired order. In the first considered approach for the servomechanism's vision control system, we identify the control orders by calculating the moments associated with the control object, (Awcock, G. & Thomas, R., 1995), (Jain, R.; Kasturi, R.; Schunk, B. & Brian, G., 1995). This approach, although simple, did not allow the distinction of a satisfactory orders number. To solve that problem, we then implemented a different approach based on orientation histograms, (Freeman, W. & Roth, M., 1995), (Freeman, W.; Tanaka, K.; Ohta, J. & Kyuma, K., 1996), (Freeman, W.; Anderson, D.; Beardsley, P.; Dodge, C.; Roth, M.; Weissman, C.; Yerazunis, W; Kage, H.; Kyuma, K.; Miyake, Y. & Tanaka, K., 1998), (Freeman, W.; Beardsley, P.; Kage, H.; Tanaka, K.; Kyuma, K. & Weissman, C., 1999). This last one, although more complex than the previous approach, is also very easy to implement, little demanding in computational resources and already allows a reasonable orders number. These two approaches will be described in this paper. We have divided this paper in the following way: In the next section, is briefly presented the servomechanism used. In the third, we describe the interface developed to control and monitor the used servomechanism through a personal computer. After that, we describe both approaches implemented for the servomechanism's vision control system: the one based on objects' moments and the one based on orientation histograms; and, by the end of this section, we present two examples of satisfactorily images control sets. In the forth, and last section of this paper, are indicated some conclusions and possible future work perspectives.

2. Servomechanism Description

A structural static base and a dynamic one, the working table, compose the used 2-axis servomechanism, Fig. 1. Two hydraulic cylinders, one vertical and one horizontal, of 600 and 350 mm respectively, manipulate the referred table.

OPEN IN VIEWER

Fig. 1.

Servomechanism used in this work

The servomechanism is also equipped with four sensors whose function is the checking of each cylinder's limits. To enable the interface between the control personal computer's board and the servomechanism's command system, was used an AX757 board which has eight digital signal inputs and eight relays. In this work, were used the digital signal inputs to monitor the four sensors referred above, and two of the eight relays in the servomechanism command. The AX757 board suggests the sending and receiving of electrical signals through a computer to the hydraulically pump's motor and the loading valve. To send the command signals from the computer, we had used the AX5411, from which we only used the digital capabilities, because there was no speed variations involved. As we can see in Fig. 1, the used servomechanism was already equipped with a user interface element, previous developed, which allows its manual control. The main objective of this work was to add to this servomechanism another fully automatic control system based on gesture recognition.

3. Servomechanism Control

The first step of this work was the development of a friendly user interface that allows, through a personal computer, the total servomechanism control and monitor.

This interface was developed for Microsoft Windows platforms, (Richter, J., 1998), using the Microsoft Visual C++ programming environment, (Young, M., 1998), and integrated in an already existing generic image processing software package, (Tavares, J., 2000).

In Fig. 2 we present the developed interface for the servomechanism computer control. Through this interface, we can check the sensors state, turn on/off the hydraulic central motor, control the loading valve, move each cylinder, change the speed levels (fast/slow), and an emergency stop.

OPEN IN VIEWER

Fig. 2.

Developed interface for the servomechanism's control by a personal computer

The next step of this work was the developing of the control system based on gesture recognition. The two implemented approaches will be presented in the following subsections.

3.1. Object's Moments

In the image analysis domain, the use of objects' moments to characterize them in position, size and orientation, and therefore recognize them, is very common, (Awcock, G. & Thomas, R., 1995), (Jain, R.; Kasturi, R.; Schunk, B. & Brian, G., 1995). As our problem is indeed a recognition problem, and due to this just referred common use, we initiated the developing of the vision control system using this approach. Thus to compute the object's shape area A, the zero order moment's value, the object's centroid coordinates (x̄, ȳ) the first order moments, and the object's elongation axis orientation θ, the second order moment's axis, we use:

A = ∑ i = 1 n ∑ j = 1 m B [ i , j ] ,

(1)

x ̄ = ∑ i = 1 n ∑ j = 1 m j B [ i , j ] A ,

(2)

y ̄ = ∑ i = 1 n ∑ j = 1 m i B [ i , j ] A ,

(3)

and

tan ( 2 θ ) = ± b a - c ,

(4)

where: (n, m) are the image's dimensions and a b, and c, the second object's order moments, are defined as:

a = ∑ i = 1 n ∑ j = 1 m ( x i j ′ ) 2 B [ i , j ] ,

(5)

b = ∑ i = 1 n ∑ j = 1 m ( x i j ′ ) ( y i j ′ ) B [ i , j ] ,

(6)

and

c = ∑ i = 1 n ∑ j = 1 m ( y i j ′ ) 2 B [ i , j ] ,

(7)

with x′ = x – x̄, y′ = y – ȳ and B [i,j] set to 1 (one), if the pixel belongs to the object, or to 0 (zero) if not.

In Fig. 3 we present two examples of getting the geometric properties of objects using the above formulated moments' method. The results are, as can been seen, satisfactory.

OPEN IN VIEWER

Fig. 3.

Determination of two object's geometric properties using the moment's method

3.2. Orientation Histograms

However, although simple, this method presents some disadvantages: the control object must be represented by a single region, with shape preferably rectangular, the image binary and, because the obtained orientation is always between 0 and 180°, the number of possible control orders must be reduced. To overcome these problems, we implemented a second methodology based on orientation histograms which will be described in the next section.

To overcome the problems associated with the moment's method, previously described, we implemented for the vision control system a second methodology based on orientation histograms, (Freeman, W. & Roth, M., 1995), (Freeman, W.; Tanaka, K.; Ohta, J. & Kyuma, K., 1996), (Freeman, W.; Anderson, D.; Beardsley, P.; Dodge, C.; Roth, M.; Weissman, C.; Yerazunis, W; Kage, H.; Kyuma, K.; Miyake, Y. & Tanaka, K., 1998), (Freeman, W.; Beardsley, P.; Kage, H.; Tanaka, K.; Kyuma, K. & Weissman, C., 1999). This methodology consists on calculating the orientation histogram of each acquired control image and compares them with each histogram considered in the learning phase (determined from the images corresponding to the desired orders). In Fig. 4 and 5 are represented the working diagrams for the vision servomechanism's control, based on image orientation histograms method, in the learning and working phase respectively.

OPEN IN VIEWER

Fig. 4.

Learning phase of the vision servomechanism's control based on orientation histograms' method

Therefore, for each 256 grey levels control image we calculated the orientation his-togram and stored in a 16-component vector. Then, in order to even its components, this vector is smooth. The comparison of the current order's image, with the ones considered in the learning stage, is obtained computing the differences between histogram's vectors. The orientation of the pixel with coordinates (i, j) is:

θ [ i , j ] = arctan ( di , dj )

(8)

with

d i = B [ i , j ] - B [ i - l , j ] ,

(9)

d i = B [ i , j ] - B [ i , j - l ] ,

(10)

and B[i,j] equals to the pixel's grey level (generally, between [0, 255]).

In the orientation histograms' calculations, we only consider the pixels, which have intensity level above a predetermined value, thus neglecting the noise pixels, and, to neglecting pixels from areas with reduced meaning, a contrast value higher than a certain predefined value. In Fig. 6 and 7 we can see the implemented interfaces for the vision control system in its learning and working phases.

OPEN IN VIEWER OPEN IN VIEWER

Fig. 6.

Developed interface for the servomechanism's vision control system in the learning phase. (To the visible image will be associated a command order by choosing the corresponding button)

OPEN IN VIEWER

Fig. 7.

Developed interface for the servomechanism's vision control system in the working phase for manual processing mode. (In the left window is visible the actual captured image, and in the right one the last image processed)

The considered control orders were the following: stop movement, move to the left, to the right, up, down and movement speed change (fast/slow). In Fig. 8 and 9 are presented two examples of successfully orders images sets

OPEN IN VIEWER

Fig. 8.

First example of a satisfactory working orders image set for the vision control system based on image orientation histograms' method

OPEN IN VIEWER

Fig. 9.

Second example of a satisfactory working orders image set for the vision control system based on image orientation histograms' method

In the implementation done, the vision control system when in automatic working mode is constantly acquiring images and cyclically, in predetermined time intervals, does the active image interpretation and processes the associated working order, Fig. 10. When there is a considerable difference between the processing image's orientation histogram and the ones stored in the learning phase, the system rejects the associated order. The four sensors are also monitored cyclically, also in predetermined time intervals, which inhibit, or not, the respective cylinder's movement.

OPEN IN VIEWER

Fig. 10.

Servomechanism's vision control system based on orientation histograms' method in the automatic working mode

4. Conclusions and Future Work

In this paper, we presented a servomechanism's control system based on gesture recognition. We have described the 2-axis servomechanism used, presented the developed interface to control and monitor it by a personal computer, and both approaches implemented for the vision control system were described. The first approach considered, based in object's moments, presents simplicity and low computational cost; however presents some disadvantages, mainly the reduced number of possible control orders. The second approach considered, based on orientation histograms, overcomes this problem successfully. In Table 1, we present some of the advantages and disadvantages of each approach that we have considered.

OPEN IN VIEWER

Table 1.

Advantages and disadvantages of each approach used for the vision control system

Moments:	Orientation Histograms:
√ Common in computational vision to recognize objects; √ Simplicity; √ Low computational cost; × Single region with shape preferentially rectangular; × Binary images; × Reduced number of possible control orders.	√ Simplicity (although more complex than the moment's method); √ Low computational cost (although heavier than the moment's method); √ All image is used; √ Grey levels images; √ Higher number of possible control orders; √ More adjustable to the working environment (several control parameters available).

During the several experimental tests done, we have concluded that the methodology based on orientation histograms presents two advantages: implementation simplicity and execution quickness. The referred approach works in satisfying manner controlling the used servomechanism and could be considered in other kinds of friendly interfaces: games, computerized applications, home systems, robotic systems, remote controls, etc.. However, we also found that this approach presents some limitations: As the used webcam does not compensate light changes, the control system does not react the same way if those variations are considerably. Another problem, with the actual control system version, is related with the control object size and how it domain the image. This last problem is augmented by the fact that the used webcam does not have an auto-focus system.

For future works, to become the vision control system based on orientation histograms, more robust to the problems previously referred, we can suggest: The tracking of the control object through images sequence using, for example, Kalman filters, (Maybeck, P., 1979), (Tavares, J., 1995), (Tavares, J. & Padilha, A., 1995a), with active contours, (Bascle, B. & Deriche, R., 1992), (Blake, A. & Isard, M., 1998), (Kass, M.; Witkin. A. & Terzopoulos. D., 1987), (Tavares, J., 2000), as suggested in (Blake, A.; Curwen, R. & Zisserman, A., 1993), (Delagnes, P.; Benois, J. & Barba, D., 1995). The usage of a more sophisticated camera and the comparison of the new control system's behavior with the one obtained with the actual webcam. This camera upgrade, by itself, should turn more robust the implemented vision control system.