Abstract
This paper presents a fast approach to organizing arm grasping and placing manipulations for mobile robot transportation systems in life science laboratories. The approach builds a blind framework to realize the robot arm operations without integrating any other sensors or recognizing computation, but only adopting the robot's existing on-board ultrasonic sensors originally installed for collision avoidance. To achieve high-precision indoor positioning performance for the proposed blind arm strategy, a hybrid method is proposed, including a StarGazer system for all laboratory environments and an ultrasonic sensor-based component for the local areas where the arm operations are expected. At the same time, two error-correcting algorithms are presented for the improvement of the high-precision localization and the selection of the robot arm operations. In addition, the architecture of all the robotic controlling centres and their key APIs are also explained. Finally, an experiment proves that the proposed blind strategy is effective and economically viable for the laboratory automation.
Keywords
1. Introduction
In recent years, with the maturing of robotic technologies, mobile robots have been proposed for transportation in indoor laboratory environments. N. Matshuira et al. presented a mobile robot-based shopping support system for supermarkets [1]. In the system, the mobile robots track the customers to carry heavy goods; M. Takahashi et al. proposed a mobile robot for hospital transportation using a human detection algorithm [2]. In their application, a new autonomous mobile robot named MKR was developed, which was equipped with a wagon truck to transfer luggage, specimens and medical materials. B. Horan et al. proposed a transportation system using OzTug mobile robots for manufacturing environments [3]. In the presented system, a computer-vision-based controller was provided for multiple OzTug robots for the transportation paths; a strategy to organize the OzTug robots was also considered. M. Wojtczyk et al. studied a vision-based human robot interface for robotic walkthroughs in a biotechnology laboratory [4]. They employed a mobile robot to transfer biotechnology facilities. From all these studies, it can be seen that to develop a mobile robot-based indoor transportation system many technical issues need to be solved, including robot indoor localization, transportation organization and path planning, door access control, communication network, etc. Besides those technical aspects, for a big automated laboratory there are other, more specific considerations, such as the convenience of integrating the robotic systems into the laboratory automation process, the cost and expandability of the systems, the system real-time performance, etc.
This paper focuses on robot-arm blind manipulation in the laboratory transportation process. It is well known that robot arm manipulation is one of the most important technical contents in robotics. K. T. Song et al. presented a vision-based grasping strategy for a humanoid robot arm [7]. In the strategy, a Kinect depth sensor was adopted to recognize and find the target object from the real-time video combining a new Speed Up Robust Feature (SURF) computational algorithm. As the authors mention in the paper, the real-time requirement was the biggest challenge. M. Trabelsi et al. developed a robot arm manipulator for a mobile robot [8]. In the manipulator, a wireless camera was utilized to capture the colour image of the targets and an ultrasonic sensor was used to recognize the shapes of the targets. An Artificial Neural Networks (ANN)-based classifier was also proposed to improve the accuracy of the arm operations. From the technical viewpoint, those two applications ([7, 8]) belong to the same type (sensing strategy), which always combines sensors (e.g., camera, ultrasonic sensors, etc.) and the kinematics of arms to realize different kinds of operations. In some cases, intelligent algorithms (e.g., Genetic Algorithm, Artificial Neural Networks) can improve the accuracy. Generally, this sensing type works effectively. However, in this study we present some different ideas: firstly, the mobile robots definitely need some sensors to recognize the laboratory environments. For instance, the ultrasonic sensors are always used for indoor collision avoidance. So, is it possible to use these existing avoidance ultrasonic sensors also for the arm manipulations? If so, the robot arms do not need additional sensors. Secondly, in a transportation process the arm operations directly affect the efficiency of the whole system. If the errors of the arms can be compensated in advance by the robot positioning, the procedures for arm kinematic computation or arm sensing measurements can be omitted. This could save considerable arm computation time and simplify the architecture of the transportation system. Based on those thoughts, a fast blind trial is provided in this study, which not only realizes the robot arm blind grasping/placing operations but also presents a reference to combine the arm manipulation and the transportation motion. This strategy will be included in the whole transportation organization, cooperating with the robot's indoor high-precision localization and the robot path planning.
2. Architecture of Blind Approach
2.1 Mobile Robot Transportation System
As mentioned in [6], [9] and [10], a new Laboratory Mobile Robot Transportation System (LMRTS) has been developed by our research group at the Centre for Life Science Automation (CELISCA), University of Rostock, Germany (see Figure 1).
Robot-based indoor transportation
As shown in Figure 2, the LMRTS includes four sub control centres. (a) The PMS (Process Management System) is in charge of presenting a required transportation task by scheduling the whole automated process of laboratories. This is the highest level, scheduling all the laboratory's automated systems/facilities, including the mobile robot systems to realize laboratory automation. (b) The RRC (Robot Remote Centre) is a middle managing level between the higher PMS systems and the lower typical mobile robotic systems. The RRC translates the PMS transportation commands to the executable robotic parameters, which can be understood by the mobile robots. It will also do path-planning computations for the dynamic transportation tasks. (c) The RBC (Robot Boarding Centre/Robot On-board Centre) is the lowest transportation executing centre in the LMRTS, which runs in the robot's on-board laptops. It is developed to control all the hardware modules (e.g., motion, arm, indoor navigation, power) inside a mobile robot. (d) The RAC (Robot Arm Centre) is typically proposed for controlling the dual arm joints to generate different kinds of grasping and placing operations. In this study, one kind of mobile robot, named H20, from the Canadian DrRobot Company, is utilized to demonstrate the transportation framework and its relevant blind arm manipulations. The details of the LMRTS can be found in [6].
The architecture of the LMRTS at Celisca, Germany
2.2 Blind Strategy
In the LMRTS, a new fast blind strategy is presented for the robot arms in the distributed transportation. The blind strategy in the study is composed of three aspects:
To improve the robot indoor localization/positioning performance of the existent StarGazer System (SGS) adopted by the H20 robots, a Motion Correcting Algorithm (MCA) is presented. The MCA can be regarded as a local localization process compared to the SGS approach. The reason why the MCA has been proposed can be explained thus: from the reference [6], it can be seen that the SGS-based method has an impressive advantage, which can be extended to suit any size of laboratory environment. However, at the same time we find the SGS is easily affected by the referential laboratory conditions, such as strong ceiling lights. The accuracy level of the SGS is sufficient for robot movement control but insufficient for robot blind arm manipulation. The steps of MCA correction can be seen in Section 3. An Error Compensation Algorithm (ECA) is proposed for the arm manipulation. Two ultrasonic sensors installed in the H20 robot bases, originally for collision avoidance, will be used to measure the real-time distances between the robot bases and the automated tables where the arm grasping and placing operations will be executed. The measured distances will be adopted to select the best arm-controlling file to store all the H20 robot's arm joint values by evaluating the robot's final posture. These two channels of ultrasonic distance are also needed for the MCA process. The details of the ECA can be seen in Section 4. The proposed arm blind manipulation is a part of the whole LMRTS system and it should be highly compatible with the other system components (e.g., the door automated access, the motion planning) to finish a transportation process. To realize the arm blind activities automatically, lots of APIs between the RBC and the RAC have been established. For instance, how and when should the arm manipulation be activated by the RBC when a robot reaches the desired position in a transportation process? What kind of communication protocol should exist between the RBC and the RAC? Detailed explanations of these APIs are demonstrated in Section 5.
3. Transportation Organization
Indoor localization is the basis for the mobile robot transportation. The StarGazer System (SGS) from Korea's Hagisonic Company is adopted for the robot's indoor positioning. The SGS is composed of an Infrared Radio (IR) camera and a series of ceiling passive landmarks. Every H20 mobile robot's on-board SGS IR camera reads the shared ceiling landmarks and provides the indoor coordinates of the robots (i.e., X Position, Y Position and Orientation) in laboratory environments to the LMRTS, as demonstrated in Figure 3. The detailed parameters of the SGS module can be found in reference [11].
The StarGazer localization
Besides the indoor positioning measurement, the issue of the transportation organization is also important. A graph theory-based strategy is proposed to organize the transportation activities.
A map with a number of waypoints is established to cover the whole laboratory environment. Those points are classified into five types based on their different functions. As displayed in Figure 4, the red, green, blue and grey points represent in-between positions, door opening positions, door closing positions and starting/destination positions, respectively. All of those positions/points will be defined by the robot's on-board RBCs. The definition of a point can be done conveniently by using the developed definition GUI. To define a new transportation graph point, the user only need move the corresponding mobile robots to stand at those positions where the robots are expected to pass through or execute an arm operation (i.e., object grasping or placing); then, the robot's on-board SGS modules will measure the X/Y/Direction coordinates automatically. Besides the coordinates, every point will also include the parameters of robot moving mode (forward or backward to the point), robot running velocity, position stop time, etc. (see Figure 5). In the transportation organizing process, at the beginning all of the positions are inactivated and shown in grey. This means they have not been selected by an enable transportation activity. In the LMRTS, when an RBC has been connected by a remote RRC, all defined points in the RBCs will be sent to the RRC for the path-planning computation. In this study, a hybrid approach has been developed for the RRC path planning, as given in references [9] and [12]. The RRC will calculate all the shortest paths for any pair of points in an RBC-defined graph map. The path-planning results will be stored in the RRC data class. When the RRC receives a task from a PMS, it will execute the following steps: firstly, parse the PMS commands to understand the transportation request; secondly, select the best robot among the available connected ones by considering either their robotic power status or distances to the grasping/starting position; thirdly, when a robot has been chosen, search for a best transportation path (always the shortest) by searching for the starting and destination points from the pre-calculated path planning results; fourthly, send the selected path (a sequence of way point numbers) to the selected mobile robot's on-board RBC; fifthly, when the RBC of a mobile robot receives a given path sequence from a connected RRC, extract the robot hardware controlling parameters from the prepared RBC points by referring to the number sequence. After understanding all the hardware controlling parameters, the RBC can control the corresponding mobile robot to arrive at the expected arm grasping and placing positions/points where the arm blind controlling process will be carried out. During the process of the RBC parameter extraction, all the related points will be activated one by one. For example, several points will be enabled to open or close the access doors during the robot movements, and a number of points will be adopted for the MCA correction as mentioned in Section 2.2 (a). As Figure 4 shows, a path is generated by the RRC for a PMS transportation request. This transportation will transfer an object from the Work Bench #1 in Room #1 to the Work Bench #2 in Room #4. Suppose a mobile robot standing at Point 10 has been selected by the RRC for this task. Based on the strategy proposed in this study, this robot will complete the following steps to finish the transportation: firstly, it starts to move to the grasping Point 1 using a path of 10->7->6->5->4->3->2->1 through the Automated Door #1; secondly, after grasping the object at Point 1, it will go back through the Automated Doors #1, #2 and #3 using a path of 1->5->6->7->11->12->13->14->15->16->17->18 to reach the placing Point 18 in the Room #4. In those two sections of paths, the Points 2, 3 and 4 are defined to carry out the MCA for the arm grasping, and Points 15, 16 and 17 are selected to carry out the MCA for the arm placing, and a number of points such as Points 6 and 7 are enabled for door access controlling. To guide the MCA process, two channels of ultrasonic sensors installed in the robot bases are adopted, which will be explained in Section 4.
The schema of the transportation organization
The GUI of map definition and execution monitoring in the RBC of the LMRTS
4. MCA and ECA at Grasping and Placing Positions
In this study two correcting algorithms (the MCA and the ECA) are proposed to realize the robot arm blind manipulations at the transportation grasping and placing positions. Both of the two corrections are based on two channels of ultrasonic sensors, as shown in Figure 6.
The concept of the robot MCA and the arm ECA
As displayed in Figures 5 and 6, the MCA can be explained as follows: when a mobile robot starts to move to an expected arm grasping/placing position, at the beginning it will adopt the global SGS localization in the whole laboratory to reach the required areas. After going inside the areas, the robot will not only use the SGS mode but also adopt the MCA mode to improve its positioning accuracy. The MCA includes three points, the first for the robot posture correction, the second to reduce the robot moving velocity, and the third for the final correction. In the path-planning process of the RRC, the MCA path will be considered automatically. After passing through this series of three correcting points, if the final results of the two ultrasonic sensors show that the robot positioning is still unsatisfactory, the robot will be controlled to move backwards to the posture correcting point to make another MCA attempt. During the MCA process, the motion motors of the robots execute a standard feedback-based PID procedure. The performance of the MCA is proved in Section 6.
Since the arms of the H20 robots do not have the third-part sensors, an ECA strategy combining with the ultrasonic measurement is proposed for the arm blind manipulation in this study. The ECA consists of several steps, as follows. (a) When the TCP/IP communication between the RBC and the RAC is available; the RAC will send all of the pre-prepared arm file names to the RBC. Those arm files are defined to grasp or place the transportation objects at different robot parking positions. The numbers of arm-controlling files is decided by the error range of the existent localization and the accuracy of the expected arm operations. Ten correcting files are provided with 1 cm error solution in this application. (b) When a robot completes its MCA process to be ready for the arm actions at the final position, the related RBC will measure the final distances between the bases of the moving robots and the front automated tables using the same side ultrasonic sensors which have been utilized in the MCA before, then use the measured distances to choose the best arm file among the received list of arm files from the RAC. During the file choosing, the RBC will use the average value of the two distances to search for the target. (c) Once the RBC finds a suitable arm file, it will transmit the file name to the connected RAC through the TCP/IP RBC-RAC API. Once the RAC receives the file name, it will match it to the file list, extract all of the controlling parameters of the arm joints and load them to the arm hardware servo modules. (d) When the RAC finishes an arm operation, it will notify the RBC to leave the current transportation point and move to the next one. The switch from the arm operation to the next motion action is managed by the RBC. The details of the RBC-RAC can be seen in Section 5.
5. Control APIs related to RRC, RBC and RAC
Figure 7 shows the main APIs for the RRC, the RBC and the RAC. As demonstrated in Figure 7, there are four APIs, as follows. (a) One API is for the robot indoor localization, which connects to the SGS module to measure the robot indoor coordinates. As the blue frame shows in Figure 7, a group of indoor coordinates are measured, including the robot Position X: −8.02, Position Y: 1.03 and Direction: 134.30. In addition, the ID number 2722 of the related ceiling landmark is also encoded by the API. (b) A second API establishes the TCP/IP communication sockets between the RRC and the RBC. It provides two TCP/IP channels for the robot hardware measurement and the path-planning computation, respectively. When this API is activated, the robot key data (including the robot's indoor positioning coordinates, the robot's power voltages and the coordinates of the defined graphs/maps) will be sent from the RBCs to the RRC. Once the RRC receives those data, it will use the coordinates of the transportation maps to do the path-planning computation and evaluate the robot current positions and power status to determine the best candidate for a coming PMS task. When the RRC finishes the path planning and the robot selection process, the API will be applied to transmit the chosen transportation path from the RRC to the RBC. The red frame shown in Figure 7 shows a transportation path (Distance: 9.66 cm, Sequence number: 1->2->3->4->5->6->7->8) distributed to the RBC. (c) A third API is for robot-door integration. As explained in Section 3, for fully automated transportation, the mobile robots inevitably need to open and close the laboratory doors by themselves. In this application, all of the doors in the laboratory are remotely controlled and monitored by this API, which has been embedded in every RBC. Every door is given a unique I/O identification number, so the mobile robots can recognize and control them separately. As the black frame displays in Figure 7, all the automated doors at Celisca laboratories are monitored by the API now. (d) A further API is for the selection of the arm files. This API is typically designed for the arm blind strategy discussed in Section 4. As the yellow frame shows in Figure 7, two channels of ultrasonic sensors are activated to measure the final distances between the moving robot and the expected grasping table. Based on the results (Left sensor: 0.23 m; Right sensor: 0.24), a robot-controlling file named ‘arm 23’ is selected for the coming manipulation. The chosen file is sent to the relevant RAC, which can also be found in the RAC GUI, as shown in Figure 8.
The APIs for the RRC, the RBC and the RAC
Figure 8 illustrates the working process of the developed RAC GUI, which includes five steps, as follows. (a) When the GUI starts, it will connect to the arm hardware module automatically. As shown in the green frame in Figure 8, an arm servo module (IP address: 192.168.7.181, Port: 10001) is connected by the RAC successfully. (b) After connecting to the arm hardware module, the RBC will connect to the related RBC to obtain the arm operations commands and the name of the arm-controlling file. As shown in the blue frame in Figure 8, the RAC connects to an RBC and receives a command type (MOVEUP) and an arm file (arm23.xml). As mentioned in Section 4, the standard for the arm file selection is based on the average of the two ultrasonic channels. Obviously, the RAC GUI in Figure 8 communicates with the RBC GUI in Figure 7. (c) Once the GUI of the RAC receives the selected arm file name, it will match the file name to the list of pre-defined arm files to extract the specific arm joint controlling parameters. As displayed in the yellow frame in Figure 8, ‘arm23.xml’ is found in the list of arm files. (d) The GUI loads the arm joint parameters described in the ‘arm23.xml’ file to the arm hardware module through the built TCP/IP socket. In this study, an H20 mobile robot has two arms of 16 joint parameters, which can be defined in the arm XML files. Besides the joint moving values, the joint moving velocity can also be calculated with this kind of XML controlling files.
The GUI of RAC
6. Experiments
An experiment is provided to verify the effectiveness of the presented blind approach in mobile robot-based laboratory transportation.
Step 1: Environment Initialization
A number of landmarks are defined in a laboratory at Celisca, Germany, for the experiment, as shown in Figure 9. In this case, an H20 mobile robot will be controlled to grasp a laboratory object from an automated workbench, to bring it to a transportation patrol in the laboratory and then place it at the same grasping position on the same automated workbench. In the experiment, the performance both of the robot's high-precision motion positioning and the arm manipulation can be estimated.
Experiment environment: (a) the ceiling landmarks; and (b) the related automated workbench
Step 2: Transportation Map Definition
As introduced in Section 3, a graph map needs to be established to organize the robot transportation, which is composed of an arm grasping point, an arm placing point, several MCA points and a number of in-between points. In this experiment, a map has been built for the expected life science laboratory (see Figure 9), as shown in Figure 10. From Figure 10, that the following can be seen. (a) There are seven points selected. (b) Point 6 is defined as both the arm grasping position and the placing position. A mobile robot will be controlled to arrive at the point to grasp an expected object then return the object back to the same position accurately after executing an outside transportation patrol. (c) Points 3, 4 and 5 are MCA positions where the mobile robot will adopt the on-board ultrasonic sensors to carry out local high-precision positioning correction. Every time any mobile robot wants to approach the automated table, they have to combine those four correcting positions in their paths to attain high-precision positioning performance for the later blind arm manipulations. Once a mobile robot reaches Position 3, the ultrasonic distance between the aim table and the robot base will be measured and used to guide the robot's following movements besides the SGS. (d) Points 1, 2 and 7 are in-between positions, which are determined by the laboratory environments and the transportation types. In this experiment, the robot will be asked to patrol Point 7 purposely after grasping the object at Point 6. This map can be completed in several minutes by using the developed GUI, as demonstrated in Figure 5.
Sketch map of the experimental transportation
After defining the map and parameters in the RBC, the related mobile robot is ready. In the LMRTS, all of mobile robots and their RBCs are distributed a unique IP address, which can be recognized by an authorized RRC. As displayed in Figure 11, a mobile robot named H20 4D owing the upper built map is being connected by a RRC. The GUI of PMS command communication and parsing in the RRC is also provided in Figure 12. By using those GUIs in Figures 11 and 12, the communication for the procedure from the highest PMS and the lowest robot hardware can be set up.
The GUI of robot connection in the RRC of the LMRTS
The GUI of PMS command communication and parsing in the RRC of the LMRTS
Step 3: Transportation Execution
From Figure 6, it can be seen that: (a) to complete the experimental transportation, the selected robot needs to execute two paths of movements (i.e., 1->2->3->4->5->6 and 6->5->2->7->2->3->4->5->6) for the arm grasping and the arm placing, respectively; and (b) the mobile robot will do the MCA local positioning at Points 3, 4 and 5 twice, one time for the grasping and the other time for placing.
Figure 13 shows the robot moving to grasping Point 6 using the path sequence 1->2->3->4->5->6. Figure 14 displays the robot doing the real-time grasping operations at Point 6. When the robot reaches Point 6, the best file will be selected by referring to the distance between the robot base and the aim front table. As shown by the red frame in Figure 15, the results of the two on-board ultrasonic sensors are 0.19 m and 0.18 m, respectively. Based on those two ultrasonic values, we can find that: (a) the performance of the robot's final positioning at Point 6 is satisfactory, because the difference of the two results is only 1 cm; and (b) the best arm grasping action can be selected accurately (see Figure 14). In addition, from the recorded path numbers given in Figure 15, we also can see that the robot completes the grasping movements as we expect. Figure 16 displays the robot leaving Point 6 after the grasping operation to execute a transportation patrol, and then going back to Point 6 to place the grasped object, which adopts the path sequence 6->5->2->7->2->3->4->5->6. Figure 17 shows the robot executing the real-time placing operations at Point 6.
The Robot 4D is moving to the grasping position 6
The Robot 4D executing the grasping operation
Results of the ultrasonic measurements at the grasping position
The Robot 4D leaving the grasping position for a patrol: (a) the robot leaves grasping position 6; (b) the robot moves to position 2; (c) the robot reaches position 2; (d) (e) and (f) the robot patrols at position 7; (g) the robot leaves position 7 and moves to position 2; (h) the robot leaves position 2 and moves to position 3; (i) the robot reaches position 3; (j) the robot rotates at position 4; (k) the robot corrects its posture at position 5; (l) the robot finally reaches position 6 and is ready for the arm placing
The Robot 4D is executing the placing operation
In this experiment, the transportation is repeated 50 times to check the stability of the method. The results show that the successful rate is 92%, which means the proposed method is correct.
7. Conclusions
In this paper a blind strategy has been proposed for robot arm grasping and placing operations in the laboratory transportation. Differently to some other standard sensing-based arm manipulations, this strategy does not require any other sensors or any object-recognizing computation. By adopting two simple ultrasonic sensors in the robot bases originally installed for collision avoidance, a fast, high-precision arm operation has been realized by the proposed strategy. The experiment presented proves that the proposed solution is effective and can be integrated in the organization of a complex laboratory transportation process. Due to its real-time performance and convenient expandability, it is especially suitable for laboratory robotics and automation.
The main contribution of the method proposed in this paper is the simplicity of the integration. The method does not require any additional image processing or sensing computation and programming in the robot arm sides. Only by combining the robot's existing ultrasonic measurement and the proposed fast blind error correcting strategies can the method achieve satisfactory robotic arm manipulations.
8. Acknowledgements
This work was funded by the Federal Ministry of Education and Research (03Z1KN11). The authors would like to thank the Canadian DrRobot Company for technical support provided for the H20 mobile robots used in this study. The authors would also like to thank the reviewers for their valuable comments, which have improved the quality of this submission considerably.