Robotic Platform for Automated Search and Rescue Missions of Humans

2.1 Robot Kinematics

Figure 2 presents the robot wheel configuration. Next we will discuss the kinematics and the speed control mechanism of the robotic platform.

OPEN IN VIEWER

Figure 2.

Robot wheel configuration.

The common robot kinematics can be written as:

ζ r = ψ ⋅ ζ d e s i r e d

(1)

where,

ζ r = [ φ ⋅ 1 φ ⋅ 2 φ ⋅ 3 φ ⋅ 4 ] T ζ d e s i r e d = [ x ⋅ y ⋅ θ ⋅ ] T ψ = [ cos θ ⋅ sin α 1 + cos α 1 ⋅ sin θ sin θ ⋅ sin α 1 − cos α 1 ⋅ cos θ − d cos θ ⋅ sin α 2 + cos α 2 ⋅ sin θ sin θ ⋅ sin α 2 − cos α 2 ⋅ cos θ − d cos θ ⋅ sin α 3 + cos α 3 ⋅ sin θ sin θ ⋅ sin α 3 − cos α 3 ⋅ cos θ − d cos θ ⋅ sin α 4 + cos α 4 ⋅ sin θ sin θ ⋅ sin α 4 − cos α 4 ⋅ cos θ − d ]

where α_i is the angle between wheel i and the robot x-axis, φ_i is the angular velocity of wheel i, θ is the robot angular velocity about the z-axis of the global reference frame and d is the distance between wheels and the robot centre. Desired robot velocity (ζ_desired) is used to generate the robot's wheel angular velocity vector (ζ_r). This wheel angular vector is the control signal, which is sent from the FPGA processing board. The output angular velocity (ζ_sensed) of each wheel is sensed by each motor encoder.

The output velocity contains information about disturbances. The disturbance error can be found by comparing the desired velocity and the sensed velocity. The output velocity can be defined as in Eq. (2), where it is assumed that the disturbance is constant for a specific surface. Two kinds of disturbance are modelled in Eq. (2):

ζ s e n s e d = ( ψ † + ε ) ⋅ ζ r + Δ

(2)

where φ^† is the pseudo inverse of the kinematic equation, ε is the disturbance gain matrix due to the robot coupling velocity friction and δ is the disturbance vector due to the surface friction.

The correction of the speed of each motor is done by a PID algorithm for each motor speed.

The four motor control adds additional PID controllers for the robot translation and rotational vectors.

The implemented control system is shown in Fig. 3. It uses wheel encoders as feedback sensors to close the control loop. Each wheel PID controller is responsible for making it rotate at the commanded speed. Interleaved with them are two PID controllers responsible for the robot motion. The robot drives faster when no object is sensed by the distance sensors or when the distance to the front wall or object is above a predefined threshold.

OPEN IN VIEWER

Figure 3.

Control loop using four motor wheel encoders, desired robot speed and distance to be travelled.

We tested the validity of pulse width modulation (PWM) pulses for motor speed with the test bench presented in Fig. 4.

OPEN IN VIEWER

Figure 4.

Test bench for 70% duty cycle PWM signal.

2.2 Localization and mapping using particle filter

In particle filters, the samples of a posterior distribution are called particles [23] and are denoted as:

X t : = x t [ 1 ] , x t [ 2 ] , ... , x t [ M ]

(3)

Each particle x_t^[M] (with 1 ≤ m ≤ M) is a concrete instantiation of the state at time t, i.e., a hypothesis as to what the true world state may be at time t. Here M denotes the number of particles in the particle set X_t. In practice, the number of particles M is often a large number. In some implementations M is a function of t or of other quantities related to the belief bel(x_t).

The particle filter [23], which is nonparametric filter, relies on Bayes filter posterior:

1: Algorithm Particle_filter (χ_t-1,u_t,z_t): 2: X̄_t=χ_t=0 3: form = 1 to M do 4: Sample x_t^|m| ∼ p(x_t | u_t, x_t-1^|m|) 5: w_t^|m| = p(z_t |x_t^|m|) 6: X̄_t = X̄_t + 〈x^|m|_t,〈w^|m|_t〉 7: endfor 8: for m = 1 to M do 9: draw i with probability α w_t^|i| 10: add x_t^|i| to χ _t 11: endfor 12: return χ _t

where particle x_t^|m| (with 1 ≤ m ≤ M) is a concrete instantiation of the state at time t, i.e., a hypothesis as to what the true world state may be at time t. χ_t is the particle set or the samples of a posterior distribution, M is the number of particles and w_t^|m| is the weight of a particle.

Here the robot localization is carried out using particle filtering, or the sequential Monte Carlo method as just described. The inputs we use are objects detected by the distance sensors, motors tracking and the compass data. By using the filter, we were able to make an educated guess of the robot's location and direction, relative to the fixed objects in the field (corners, doorways or obstacles).

The algorithm goes as follows: M particles represent the robot's position estimation. The initial global uncertainty is achieved by randomly generating M particles (Fig. 5(a)). Given the sensors' data, particles are graded by the angle the object is seen at and by distance. Noise is added to each input (Fig. 5(b)). After a few iterations, the particles converge to the robot's real location (Fig. 5(c)). After some more iterations, the particles are very close to the robot's real location. Note that some randomly scattered particles (10%) are added to the particle sets in each iteration (Fig. 5(d)), in order to be able to handle failures such as robot kidnapping or global localization failures.

OPEN IN VIEWER

Figure 5.

Localization using particle filter.

2.3 Navigation

We will interpret an apartment as a maze and use maze-solving methods for navigating through the apartment. The maze is built with a main course, with sideways (rooms) breaching from the main course.

In order to decide upon the navigation procedure we used QFD (quality function deployment) [24], the Pugh concept selection method [25] and FMEA (Failure Mode and Effects Analysis) [26] design tools. The layout of the QFD diagram is shown in Fig. 6. It is often called the house of quality because of this configuration. Each room in the house has a name that implies its function. Table 1 describes what is found in each of the “rooms” of this house.

OPEN IN VIEWER

Table 1.

Description of what is found in each of the “rooms” of the house of quality.

Room No.	What is found in each of the “rooms” of the house of quality
	Customer requirements (whats). Can be a problem statement etc.
	Competitive assessment shows how the top two or three competitive products, ideas or algorithms etc. rank with respect to customer requirements. This section starts with ranking each customer requirement on a scale of 1 to 5 and then by considering the planned improvement and any requirements that are planned for special attention and then builds up to an importance rating.
	Importance rating: Here the importance of each requirement is evaluated based on customer importance rate, the existing product and the planned product.
	Engineering characteristics (hows) that enable satisfying the customer requirements are listed in columns.
	The correlation matrix shows the degree of interdependence among the engineering characteristics in the “roof of the house”. Often this “roof” is not used because either the EC correlation is irrelevant or the supply information is useless. In cases where trade-offs should be made, the “roof” data is important.
	The relationship matrix determines the correlation between the engineering characteristics (EC) and the customer requirements (CR). To do this, each CR is taken and for each EC, in turn, it is asked whether it will significantly, moderately, slightly, or not at all impact the customer need. Generally a nonlinear 9/3/1/0 scale (sometimes five is added) is used to weight disproportionately those ECs that strongly affect CR.
	To determine the absolute importance, first multiply the numerical value in each of the cells of the relationship matrix (6) by the importance rating (3). Then, sum the numbers in the cells of each column. These totals show the absolute importance of each EC in meeting the CR.
	Relative importance is the absolute importance, normalized on a scale from 1 to 100, by taking each absolute importance value, dividing it by the sum of absolute importance values and multiplying it by 100. Those ECs with the highest rating should be given special attention, for these are the ones that have the greatest effect upon customer satisfaction.
	The technical competitive assessment benchmarks your company performance against two or three top competitors for each of the engineering characteristics. Generally a scale of 1–5 (five being the best) is used. Often this information is obtained by acquiring examples of the competitor's product and testing them.
	Technical difficulty indicates the ease with which each of the engineering characteristics can be achieved.
	Setting target values. By knowing which the most important ECs are, understanding the technical competition and having a feel for the technical difficulty, the team is in a good position to set the targets for each engineering characteristic.

OPEN IN VIEWER

Figure 6.

House of quality rooms' function.

Another popular conceptual design tool is the Pugh concept selection method. This tool compares each concept relative to a reference or datum concept and for each criterion determines whether the concept in question is better than (+), poorer than (-), or about the same as the reference concept (S). Thus, it is a relative comparison technique. Studies show that a small group is better than an individual at selecting ideas.

One of the design tools for preventing failures is Failure Mode and Effects Analysis (FMEA). This is a proactive design tool technique and quality method that enables the identification and prevention of process or product errors before they occur. It assists in identifying the different failure modes of a system and might be used for reliability evaluation.

The FMEA procedure is composed of 15 steps: 1. Describe the product/process and its function. 2. Use a diagram to begin listing items or functions. 3. Identify failure modes. 4. Each failure should be listed in technical terms. Failure modes should be listed for functions of each component or process step. 5. Describe the effects of those failure modes. 6. Establish a numerical ranking for the severity of the effect. 7. Identify the causes for each failure mode. 8. Enter the probability factor (probability of the cause occurring). 9. Identify current controls (design or process). 10. Determine the likelihood of detection. 11. Review Risk Priority Numbers (RPN). The Risk Priority Number is a mathematical product of the numerical severity, probability and detection ratings: RPN = (severity) x (probability) x (detection). 12. Determine the recommended action(s) to address potential failures that have a high RPN. 13. Assign responsibility and a target completion date for these actions. 14. Indicate actions taken. 15. Update the FMEA as the design or process changes, the assessment changes, or new information becomes available.

Using the FMEA, Pugh and QFD analyses we decided upon the following left wall navigation, presented in Fig. 7.

OPEN IN VIEWER

Figure 7.

Left side wall navigation.

The algorithm allows for complete area cover with a complexity of order of O(n). In each junction the robot checks (with ultrasonic distance sensors) for a left turn option or a right turn in each 90 degree vertex following wall on the left side of the robot. Each robot's turn and driving direction is recorded based on compass data. Locked doors are identified as such by taking into account a standard lintel length of 10cm-20cm and a door length of 75cm-85cm. The positions of locked doors are recorded and saved in the memory and reported to the control station.

The robot uses a special memory location for keeping track of its movements. Each forward movement adds a positive distance, according to the wheel encoder counting number, to a longitudinal memory variable. Each backward movement adds a negative distance to the longitudinal memory variable. Each left movement adds a negative distance to a width wise memory variable. Each right movement adds a positive distance to the width wise memory variable. The scan is complete when both the longitudinal and the width wise memory variables contain zero with a small tolerance.

2.4 Scanning for life signals

The remote photonic life signal sensing device is installed on top of the robot and is capable of rotating ±90° relative to the robot driving direction and rotates faster than the robot speed. The scanning system will be continuously active. When a life signal is detected the location of the robot, the robot's angle and the angle between the robot and the device are recorded. Then the scan device returns to be aligned with the robot's forward direction. Then the robot rotates to the position of the scan device where life signals were just detected and the system verifies that indeed the life signal exists, in order to double check. The location is then transmitted to the control station.

Each angle is recorded in the memory using First In First Out (FIFO) data structure. In the case of an additional need for the robot to guide the fire fighters in the apartment, the robot uses the following method. With no loss of generality we assume that all turns are right angles. When the robot starts navigating from the entrance, its direction is regarded as a reference to all angles. Four directions relative to the reference direction are defined:

These are different from the common angle notation presented in Fig. 8.

OPEN IN VIEWER

Figure 8.

Definition of angles relative to reference direction.

The robot can navigate in the apartment corridor, stopping at locations of victims or by doorways with victims. Sequences of DABCB or BCDAD can be omitted from the course as they imply a room search. Then, multiple consecutive occurrences of the same direction can be diminished to one.

In the case of the example of an apartment presented in Fig. 7, the sequence of angles recorded is: ADABCBADABCBABCBCDADCBCDADCD, where underlined sequences indicate in-room searches that can be omitted. The navigation with fire fighters will be AAABCCCD, decreased to ABCD. During performing this course the robot stops near each room with casualties in it and near each casualty in the corridor. When a victim is found in a room the robot returns using Last In First Out (LIFO) data that is extracted from the stack. The robot navigates the shortest path by deleting routes that are redundant, using the logic described above.

However, in real rooms furniture and other obstacles might be attached to walls. For this reason, we change the method somewhat. When the robot encounters a doorway it assigns the letter “O” before navigating in the room. When it goes out of the room it assigns another “O”. Then for the shortened navigation it deletes all sequences between the two “O”s (“O”s inclusive) and resumes as before.

2.5 Control Station

The control station is used to build a map of the apartment with locations of casualties. The user interface allows for on-line and off-line presentations of the mapping process and location display (see Fig. 9).

OPEN IN VIEWER

Figure 9.

Control station display.

The control station “listens” constantly to the robot. When a message is received from the robot, it goes to a specific memory address. Then the message is decoded and a map component is built. If the message reports a casualty location, a special sign is placed over the map.

3.1 Operation principle of the photonic device

The photonic device is based on tracking the temporal change in the secondary speckle pattern reflected from the inspected subject. The reflected secondary speckle pattern is actually localized self interfering patterns (and therefore the detection module does not require an external interferometer). In the case of movement of the reflecting surface, those self-interference random patterns are randomly changed. We have previously demonstrated [1, 2] that properly defocusing the imaging lens of the camera yields the movement of the reflecting surface and its tilting is mainly expressed not in a random change of those patterns, but rather in their movement. Thus, the change of the tilting of the reflecting surface in time causes the movement of the random pattern in time. Capturing a set of images versus time and correlating each sequential image with another can yield the relative movement of the patterns. This relative movement is obtained simply by allocating the time varying position of the correlation peak for each one of the correlation images (simply by allocating the position of the maximum in the correlation image). This temporal movement of the position of the correlation peak is plotted versus time after proper high pass filtering to remove possible noises. The obtained curve is related to the temporal movement of the surface of the skin and as this movement is connected to the mechanics of the beating of the heart, the desired sensing is extracted.

As previously stated in our setup we measured a relative value of the spatial displacement of correlation peaks between two adjacent frames taken by a fast digital camera. Relative displacement under proper defocusing is equal to:

β ​ = 4 π tan α ​ λ ≈ 4 π α λ

(4)

where β is a relative spatial shift of the speckle pattern due to object displacement, α is the tilting angle of the object and λ is an optical wavelength. Assuming that the change in the angle is small enough, we obtain a linear proportion between the relative shift and the actual relative radial movement of the inspected surface:

t a n ​ α ~ α ~ d

(5)

where d is the axial displacement of the vibrating object. The measurement of this displacement is done with nanometre accuracy because the shift of the speckle pattern is a function of a fraction of the optical wavelength (which in our experiment was 532nm).

3.2 Smoke and fire experiments with the photonic device

The goal of the cold smoke experiment was to obtain a good pulse reading in thick smoke conditions using the developed photonic sensing device. In the experiment we wanted to check the capability of the photonic system to find heartbeats of a subject who was located in a shed wrapped in smoke. As previously mentioned, the final aim is rescuing people during a fire involving difficulties such as bad vision conditions, mainly because of the smoke.

The experimental setup included a computer (the processing was done with Matlab software), a laser that projected light at wavelength of 532nm (SUWTECH, DPGL −2100, CW @ 532nm), high-resolution camera (Pixelink BL-771) and a power source. The light beam was projected by the laser source onto the fireman's chest, positioned within a room with cold and dense smoke. With the camera we can observe the modulations and the software finds out if there is a heartbeat. The schematic sketch of the experimental setup is seen in Fig. 10(a), where the square resembles the shed, which was filled with the dense, cold smoke.

OPEN IN VIEWER

Figure 10.

(a). Top view schematic sketch of the experimental setup. (b). The obtained experimental results.

The fireman who was positioned in the smoke shed was equipped with an oxygen balloon and a mask. He was positioned about 8 metres away from the laser-based system. We took a reference and measured his pulse. Afterwards we filled the room with smoke, until the visibility was zero. We measured the man's pulse, while we kept filling the room with smoke as the smoke began to scatter. It is important to mention that when the shed was full of smoke one could not see anything but the reflections coming back from the laser beam.

The experiment was run about 20 times. An example of the obtained results is seen in Fig. 10(b), where one may see the remote sensing of the heartbeat without and with the presence of the heavy, cold smoke. The pulse shapes are easily seen in the graphs. Each black arrow points at a pulse. Each pulse represents one heartbeat. The measured patterns presented in Fig. 10(b) were very repeatable. The deviations in the heart rate were less than 5%. Thus, in the experiment we proved that the remote life detecting system is capable of measuring heartbeats at a distance of several metres, even in heavy smoke conditions. Note that although the provided results were applied on a single fireman due to security and safety reasons, they were collected over a long period of recording, including 20 separate experiments in different smoke conditions. Another important point to mention is related to the fact that in bad visual conditions, as in a smoky environment, one may have difficulty distinguishing between humans and pets, for example. However, as the heartbeat signature of humans is very different from animals in the rate of the beating, as well as by the shape of each beat, such discrimination is actually quite a simple task.

In the second experiment we validated the capability of the optical remote sensing platform to sense the heart beating through fire. In this experiment we sent a subject to a distance of 100 metres away from the sensing optics (the white rectangle mark in Fig. 11(a) designates the subject as it was seen with our optical infra-red sensor) and placed a strong fire (we used a barbecue device) in front of him (Fig. 11(b)). This time we used remote sensing optics that work in infrared, rather than at 532nm. In Fig. 11(c) we show the experimentally obtained heartbeat of the subject while the sensing was done and while illuminating through the fire with our laser. One can see the periodic beating repeating itself every second (approximately) and a repeated beating pattern very well. This experiment was run 25 times. The measured patterns presented in Fig. 11 were very repeatable. The deviations in the heart rate were less than 3%.

OPEN IN VIEWER

Figure 11.

(a). The target as being seen in the infrared sensor of our optical remote sensing platform, from a distance of 100m. (b). The subject of the experiment was sitting in a chair while a large fire was set in front of him. (c). The sensing of the heartbeat of the subject while the sensing was performed by illuminating with the laser through the fire.

For a better understanding of the principles described in this paper, in Fig. 12 we show one of the prototypes we worked with for the photonic remote sensing device (left) as well as the robotic platform (on the right).

OPEN IN VIEWER

Figure 12.

Image of two prototypes for the photonic remote sensing device (left) and the robotic platform (right).

Note that the developed sensor can measure, with nanometric accuracy, the movement of any inspected surfaces. As the precision is so high (nanometric) various biomedical parameters can be extracted. Due to blood pulsation and the heartbeat, the surface of our skin moves. This is a very small movement but as said before it can be detected on a nanometric scale. Obviously closer to the blood vessel the movement is larger and if the measurement is done with the surface being remote from the vessel then the movement is proportionally smaller. Nevertheless, one does not need to be located exactly on the blood vessel. Moreover the subject in our experiment was wearing a full set of fireman's equipment that isolated his skin from the illuminating beam. Nevertheless due to the high accuracy of the proposed sensing approach the beating of his heart could be sensed. Thus, not only is where the measurement is done not critical (which is good as in heavy smoke conditions the visibility is very low), but also the subject is not required to expose the relevant skin for performing the measurement.