Abstract
As the Internet of Things (IoT) penetrates different domains and application areas, it has recently entered also the world of robotics. Robotics constitutes a modern and fast-evolving technology, increasingly being used in industrial, commercial and domestic settings. IoT, together with the Web of Things (WoT) could provide many benefits to robotic systems. Some of the benefits of IoT in robotics have been discussed in related work. This paper moves one step further, studying the actual current use of IoT in robotics, through various real-world examples encountered through a bibliographic research. The paper also examines the potential of WoT, together with robotic systems, investigating which concepts, characteristics, architectures, hardware, software and communication methods of IoT are used in existing robotic systems, which sensors and actions are incorporated in IoT-based robots, as well as in which application areas. Finally, the current application of WoT in robotics is examined and discussed.
Introduction
Robots are increasingly being used in industrial, commercial and domestic applications, as well as for rescue operations where there are safety risks for humans [126]. Robot comes from the Czech word robota which means forced work or labour. The word robot today means any man-made machine that can perform work or other actions normally performed by humans, either automatically or by remote control. Robots are employed because it is often cheaper to use them over humans, easier for robots to do some job and sometimes the only possible way to accomplish some tasks. An example of a robot with a human-like shape is provided in Fig. 1. Most robots are composed of the following components:
A controller, i.e. the brain of the robot.
Mechanics, i.e. motors, pistons, grippers, wheels and gears that make the robot move, grab, turn, and lift.
Sensors, i.e. sensing equipment that helps the robot perceive its surroundings.
Robotics constitutes a modern and fast-evolving technology [106]. Robotics can be defined as the branch of engineering that involves the conception, design, manufacture and operation of robots [33]. Although robots have been mostly used in industrial applications till date, recent technological progress in the emerging domains of cognition, manipulation and interactions is moving the robotics industry toward service robots and human-centric design [136].
An example robot pouring some liquid from one cup to another (source: [106]).
The Internet of things (IoT) [133] is the extension of Internet connectivity into physical devices and everyday objects. In this context, physical things become uniquely addressable and interconnected through the Internet, providing support for the IPv4 or (more recently) the IPv6 protocol [79]. The Internet Protocol (IP) is the principal communications protocol in the Internet protocol suite. IPv6 is the most recent version of the IP, intended to replace IPv4 by offering support for trillions of devices connected to the Internet.
Similar to the IoT, the Web of Things (WoT) [39,134], is about approaches, software architectural styles and programming patterns that allow real-world objects to be part of the World Wide Web. Via the WoT, physical things have access to the Web, interacting through services and application programming interfaces (APIs) based on the Hypertext Transfer Protocol (HTTP). HTTP is the foundation of data communication for the World Wide Web, where hypertext documents include hyperlinks to other resources that the user can easily access. Enhanced with existing Web services, physical devices offer advanced features and intelligence, such as smart umbrellas that warn the user when it is about to rain or kettles that heat water when the electricity tariff is low [124]. This new possibility of blending physical sensor-based capabilities and functionality, together with real-time Web services and APIs has been defined as physical mashups [40].
Numerous research areas and application domains have been influenced in the past by the introduction of IoT and WoT. These areas include home automation [61,113], smart buildings [18], smart grid [23], remote healthcare [137], agriculture [56], as well as a limited number of smart city applications [58,59].
Examples of this influence by IoT/WoT involve reduced risks of vendor lock-in, adopting machinery (i.e. for agriculture) and sensing/automation systems from different companies, as these could become easily interoperable in the overall smart systems, easier data exchange among different, heterogeneous components, increased automation with less effort by means of Internet and Web standards etc. Use of open standards brings seamless connectivity and advanced interoperability, while the publishing of data produced by IoT sensors as open data on the Web promotes knowledge sharing and is important for the advancement of research in these fields.
As the IoT and the WoT penetrate different domains, application areas and scientific disciplines [80], it is worth the effort to examine their impact, interaction and application in the emerging research area of robotics, as they transition from industrial applications into the everyday lives of people in home settings, buildings, commercial centres, communities, airports, supermarkets, etc.
The interconnection and relationship between the IoT and robotics has been defined as Internet of Robotic Things (IoRT) [96,108,125]. IoRT has been defined as a a global infrastructure enabling advanced robotic services by interconnecting robotic things based on existing and evolving interoperable information and communication technologies such as cloud computing, cloud storage and other existing Internet technologies. IoT allows robots to communicate by means of the IP protocol, especially its IPv6 version, which is designed for billions of Internet-connected objects [49]. Internet connection permits updating information (and possibly the robot’s firmware) in real-time [122], storing/processing data on the cloud and taking advantage of Internet protocols for security, authentication, data integrity, message routing, etc. [110,114].
The main motivation for preparing this paper stems from the fact that related work in this field [1,21,38,96,108,125] has discussed only generally some of the opportunities of IoT in robotics. There is a gap in the literature in relation to the actual degree of penetration of IoT in robotic systems and services, as well as how IoT has been used in robot systems till date.
One might argue that robots constitute just some extra “things” in the IoT ecosystem. However, robots constitute complex systems with a wide range of technologies, behaviours and components, which makes their research domain quite unique. Nevertheless, the two research fields of robotics/IoT have started interacting in the last decade, linking the research communities together in many aspects, creating a new interdisciplinary field. According to [1], IoRT is at a level more advanced than IoT, due to the need to integrate numerous modern technologies together, in order to satisfy the needs and requirements of high-tech robotic systems that operate beyond the industrial level.
Differences between this survey and related ones
Moreover, we argue that IoRT should also embrace the WoT, which has not yet been considered in the survey papers mentioned above. WoT could offer additional benefits to the IoT-enabling of robotics-based systems, especially in terms of higher interoperability among the robot’s sensory components, but also robot-to-robot (R2R) and robot-to-human (R2H) communication at the application layer. Additionally, WoT could facilitate the combination of Web services and robotic services, towards physical robotic mashups, realizing the notion of a Web of Robotic Things (WoRT). These potential benefits are discussed in Section 6, in more detail, together with a small historical journey on the first robots that had presence and allowed remote control via the Web, back in 1995.
Therefore, the contribution of this paper is to study the current use of IoT in robotics, through various real-world examples encountered through bibliography-based research and to examine the potential of the WoT in robotic systems. To the authors’ knowledge, it is the most complete survey paper to date, aiming to present the actual research taking place at the intersection of these research domains.
The rest of this paper is organized as follows: Section 3 describes the methodology adopted and Section 4 introduces the concept of robotics, clarifying its relationship with IoT. Section 5 presents the existing penetration of IoT in robotics, while Section 6 studies the connection between WoT and robotic systems. Finally, Section 7 discusses the overall findings and Section 8 concludes the paper.
As stated before, related work in this field [21,38,96,108,125] has not discussed elaborately the use, characteristics, developments, technologies and opportunities of IoT in robotics.
In particular, the work in [108] focuses on the lower and higher-level abilities of IoT-enabled robots, while Vermesan et al. [125] discuss generally some technologies of IoT that can support robotic systems. Ray [96] lists some examples of existing robots envisaged for an IoRT architecture, but the linking between the examples mentioned in the paper with the IoT is unclear and not convincing. The differences between this survey and the relevant ones are summarized in Table 1.
Further, the current application and future potential of WoT in robotics have not been discussed in any relevant survey paper, except from [38], where the aspect of semantic consensus has been generally discussed, suggesting that this issue could be approached via Semantic Web technologies.
Methodology
This paper aims to fill the aforementioned gaps in the literature, by addressing the following research questions:
Which concepts, characteristics, architectures, platforms, software, hardware and communication standards of IoT have been used by existing robotic systems and services?
Which sensors and actuators have been incorporated in IoT-based robots?
In which application areas has IoRT been applied?
Which technologies of IoT have been used in robotics till today?
Has WoT been used in robotics? If yes, by means of which technologies?
Which is the overall potential of WoT in combination with robotics, towards a WoRT?
As the IoT has been defined in different ways in literature, it must be clarified that IoT – in the context of this paper – is perceived as a system that involves real-world things, which can communicate/interact over the Internet and they can be remotely monitored and controlled via Internet protocols. These devices involve robots and robotic systems in the context of this work.
Search for related work was performed through the Web scientific indexing services Web of Science and Google Scholar. The following query was used:
Robotics AND [“Internet of Things” OR “Web of Things”]
Thirty seven (37) papers were found via this approach. To increase the range of our bibliography, a search of the related work as appeared in these 37 papers was also performed. This effort allowed to increase the number of papers discovered to 61. From these 61 papers, 12 papers (20%) were discarded for the following reasons:
They did not involve a real-world implementation of a robotic system but only mentioned theoretically how a robotic system could benefit from the connection to the Internet/Web, or from the concepts of IoT.
They did involve a real-world implementation of some robot or robotic system, however, this robot did not somehow relate to the concepts of IoT and/or WoT.
Forty nine (49) papers were finally selected in order to be analyzed in more detail. Each of these papers was studied in detail, aiming to address the aforementioned six research questions. The results of our research are presented in the next sections.
We note that this paper focuses on the connection between robotics and the concepts/principles of IoT and WoT, aiming to close the existing gap in the literature, as described in the introductory section. This paper does not intend to compare specific hardware and software platforms used for sensors/robots development and/or development of IoT-based systems. For such comparisons, the reader should consider relevant studies [20,34,55,78,98,125,136], which cover quite well the spectrum of embedded hardware/software development, plus low-power communications. The research areas targeted by this paper are displayed as a Venn diagram in Fig. 2. WoT is considered a subset of IoT and they both interact with robotics via the emerging research areas of IoRT and WoRT. Additional circles in the figure could be sensor technologies and communication protocols, as they are extensively used both by IoT and robotics.
A Venn diagram showing research areas involved.
We note that IoRT is mostly about technical aspects of robotic systems, as well as technologies for communication and message exchange or services and data understanding. It is not about artificial intelligence, robot perception and empathetic behaviour, where robotics touches upon other research disciplines.
Robots are becoming a fundamental part of our society and they will become even more important in the future. The past decades were characterized by the massive automation in the industry, as for example in the case of automatic machines and industrial manipulators. In this context, the robots work in a perfectly-known and modeled environment and safety layers are built around them to prevent harming people and other machinery. Furthermore, the actions of these robots are completely programmed in order to avoid any unpredictable behaviours. The robots work fast and accurately in order to improve the efficiency of industrial processes.
Layers of the OSI stack where IoT and WoT are located.
However, to further progress in the integration with everyday life, robots, usually referred to as service robots in this context, need to be able to perceive and understand unknown and complex environments and to be capable of planning and acting in unforeseen situations. In robotics, these concepts are usually gathered together under the word cognition. Cognition is the key element in order to deal with the variety of environmental aspects, parameters and tasks in which the service robots operate, i.e. houses, warehouses and offices. The second major challenge is the manipulation of a high variety of objects, deformable or not, in the operational environment. The final hurdle for robots is the interaction. Robots and humans are expected to end up in the same environments, working in parallel and collaborating together, without any safety layer to keep them apart. Thus, the robots’ behaviour must be predictable and safe for humans, for the surroundings and eventually for other co-operating robots. The concept of interaction, however, cannot be limited only to physical and safe interaction between humans and robots, but it must be extended to more abstract ways of interacting. Interpreting verbal and non-verbal communication such as facial expression, body movements and gestures, as well as an understanding of social interactions are clearly necessary steps to be taken in the near future.
Since robots are entering the everyday life of humans, supporting their tasks and interacting with them, the sensory equipment, hardware platforms, software and communication patterns of the robot machines become more complicated and demanding. As mentioned before, robots need to interact with their environment, i.e. other robots, devices and machines (R2R) and also with humans (R2H). To achieve this in a highly interoperable way towards true machine-to-machine (M2M) interaction involving context understanding, considering the wide variety of protocols and hardware/software/communication architectures and solutions available, the IoT and WoT come into play. The particular benefits from a WoT integration are listed in Section 4.2 below. The main difference between IoT and WoT is that IoT operates in the lower layers of the OSI stack, while WoT mainly at the application layer. Open Systems Interconnection (OSI) protocols are a family of information exchange standards developed jointly by the ISO and the ITU-T standardization bodies. OSI is a conceptual model that characterises and standardises the communication functions of a telecommunication or computing system without regard to its underlying internal structure and technology. In this context, some overlap between IoT and WoT may exist at the presentation and session layers of the OSI stack. This difference is illustrated in Fig. 3. The right part of the figure lists many of the technologies and acronyms mentioned through this survey, under the relevant part of the ISO stack where they belong, as well as whether they constitute mainly IoT or WoT technologies.
IoT could improve robotics with higher productivity (i.e. by re-using well-accepted and understood software and protocols), lower costs, better customer experiences due to the easier integration with existing components of the nearby environment and with cloud computing, high-quality data in terms of semantics awareness, context understanding and many other possibilities listed in related work [80,133,134].
Moreover, robotics can benefit from the plethora of research and development in IoT, in terms of resource-constraint hardware and software, low-power communication algorithms and protocols, as well as optimal solutions for wireless sensor networks (WSN), such as networking, mobility, data propagation, topology building and maintenance etc. [97]. Finally, the unique naming and addressing capabilities provided by the IPv6 protocol, allow robots and robotic systems to become uniquely addressable citizens of the Internet, exploiting the TCP/IP protocols for device discovery, message exchange, security etc. The Transmission Control Protocol/Internet Protocol (TCP/IP) is a suite of communication protocols used to interconnect network devices on the internet. TCP operates at the transport layer of the OSI stack, while IP at the network layer (see Fig. 3).
Web of Things and robotics
One of the first robots having a basic presence on the Web was MERCURY [36], which started its operation in 1994. It was one of the first teleoperated manipulators on the Web. It enabled Web users to excavate artefacts buried in a sand-filled terrarium. The TELE-GARDEN robot in 1995 [37], successor of MERCURY, allowed people to control the planting of flowers via a Web interface, being able to coordinate requests by multiple users. XAVIER followed in 2002 [107]. Although it had only a basic Web interface allowing only basic interaction possibilities, it became quite successful due to the possibilities of remote control of a robot via any Web browser around the world.
As mentioned in the introduction, WoT could offer more benefits to robotic systems than IoT alone [39,134], especially in terms of interoperability at the application layer among robotic components and services, but also between robots (R2R) and humans (R2H), as well as among other machines (M2M). Some of these benefits are listed below, in more detail, in relation to robotic systems:
Data from the sensors of the robots may be easily exported into Web applications in popular, well-understood standard formats, for easier reuse. Representation formats may be negotiated in real-time, depending on the formats supported by the machines involved.
Exposing the services provided by the robots (and their individual sensing and actuating components) as interoperable APIs, would provide the primitives to users with little programming experience to perform advanced tasks. Users may select any programming language that supports the HTTP protocol, such as Python, Java, Ruby, C, PHP, JavaScript, etc.
Exposing robotic services as APIs would facilitate application-layer interoperability between robots (R2R) and humans (R2H). When these APIs become standardized, enhanced with semantic technologies, interaction between robots and humans can become automatic and in real-time, using the Web as the common platform and language for communication.
Combining robotic services with Web services and resources would allow the creation of Web-based robotic mashups, where the robots exploit seamlessly knowledge, information and context already available at the Web, towards more informed choices and aware behaviour.
Uniform access to heterogeneous embedded devices installed on the robot, where the robot itself becomes a homogeneous environment (i.e. at the application layer), where any sensor/actuation can be individually accessed in a standardised way, facilitating coordination, action-taking and control. This homogeneous access would allow easier connection between robotic systems and cloud computing, for more advanced real-time computing processing and for permanent storage of information such as sensory measurements.
Harnessing well-defined protocols used for years on the Web for device and service discovery, service and data description, semantics understanding, security and privacy, orchestration and routing.
Particularly related to the semantics of services and information, the Semantic Web (see Section 6.1 below) involves numerous technologies and implementations for uniformly describing devices, services and data, allowing common understanding and advanced reasoning between different entities. The Semantic Web is an extension of the World Wide Web through standards set by the World Wide Web Consortium (W3C). The goal of the Semantic Web is to make Internet data machine-readable, enabling the encoding of semantics with the data.
Applications of IoT in robotics
Applications of IoT in robotics
We note that APIs, in the WoT research area, are expected to be resource-oriented and to follow the REpresentational State Transfer (REST) [31]. REST is an architectural style for developing Web Services and it has been adopted by WoT due to its simplicity and the fact that it builds upon existing systems and features of the HTTP protocol. A RESTful API follows the principles of REST for providing Web Services to the users, based on the HTTP request/response protocol. Thus, basic HTTP-based interfaces that do not follow REST can not be considered part of the WoT. However, they can still be considered part of the IoT, as long as they employ TCP/IP.
This section addresses the first three research questions as defined in Section 3. First, Section 5.1 lists the existing applications of IoT in robotics, presenting how sensors and robot actions have been used in different application domains. Then, Section 5.2 shows some of the hardware elements and sensory equipment used in the work under study. Afterwards, Section 5.3 presents the software platforms and communication protocols used during programming and control of IoT-based robotic systems. Finally, Section 5.4 summarizes in which ways the IoT-enabled robots of related work have made actual use of Internet technologies.
Applications
Table 2 lists the different application domains and their specific application areas, where IoT-based robots and robotic services have been used. The most popular categories are entertainment (8 papers), health (7 papers), education (6 papers), surveillance (6 papers) and culture (5 papers). Some categories are overlapping, such as health with domestic support, surveillance with military, as well as emergency/disaster response with rescue operations. Autonomous cars are used as moving robots in the area of transportation, while a warfare robot car has been developed for military purposes in [48]. Unmanned aerial vehicles (UAV) (i.e. drones) are considered as flying robots in surveillance, emergency/disaster response and rescue operations. UAV/drones are small aircrafts without a human pilot onboard.
Example robots as they appear in different application areas of the work under study are shown in Fig. 4.
Example robots as presented in related work under study.
Figure 5 maps the sensing/actuation capabilities of the IoT robots, together with their actions, in relationship to the different application areas as encountered in this study by analyzing the related bibliography. The most popular sensors, actions and application areas are highlighted in blue color. As the figure shows, the most commonly used sensors are the ones that measure proximity (for the actions of touching, picking and placing, reminding), vision (for the actions of picking and placing, moving, observing, flying, spraying and pruning, reminding), voice (for the actions of speaking, listening and voice recognition) and motion (for the actions of picking and placing, speaking, observing, climbing, swimming and displaying text).
The connection between IoT robot sensors, actions and IoT application areas, as they appear in related work under study.
The most popular actions involve moving, observing, flying and navigating. Some actions are appropriate only for specific application areas, such as spray and prune for agriculture, reminders for health and domestic support, pick-and-place for customer support and industrial applications, etc.
In relation to the possible actions performed by the robot, based on its sensory equipment, the application areas where most actions have been recorded are agriculture (7 actions), domestic support (6 actions), surveillance (6 actions), military (6 actions), rescue operations (5 actions) and entertainment (5 actions).
Hardware
Besides the mechanical parts which vary significantly per related work under study, there is some common hardware used among the surveyed papers, such as Raspberry Pi [12,17,27,44,48,66,105,123,129] and Arduino [9,105,130]. Both Raspberry Pi and Arduino constitute open-source hardware and electronic prototyping platforms, enabling users to create interactive electronic objects. In the context of this survey, researchers in related work have used them as mini computers, connecting external sensors, actuators and mechanical parts, in order to give intelligence to their robots. TelosB was the sensor platform selected in [102].
Some efforts tried to develop humanoid robots [16,27,109,118], while others employed autonomous vehicles [35,129,130], a space exploration rocket [11], a warfare car robot [48] and UAV/drones [3,30,66,68,82,83,102,130,138]. An UAV enhanced with IoT sensors has been introduced in [102], where the UAV interacts better with its environment towards more effective surveillance, gathering data coming from temperature, humidity and light sensors.
The remaining papers used robots with application-specific mechanics and characteristics. The interesting concept of biobots was introduced in [43], which was about real animals (in this case, dogs) equipped with sensors such as cameras or gas detectors. These sensors allowed the animals to sense additional environmental aspects (i.e. search rubble for casualties or detect dangers such as a gas leak). In this case, the mechanical parts of a robot are replaced by the physical capabilities of the animals involved.
Sensors
Robots were equipped with a wide variety of different sensors, such as radio-frequency identification (RFID) [15,45,128–130], video cameras [11,35,36,51,81,93,100,116,131], infrared and light sensors [81], smoke sensors [12], temperature and gas [9,123], temperature, humidity and light sensors [102], medical sensors [6,7,45], accelerometers and gyroscopes [42,64], occupancy [25] and infrared sensors [44,81,100]. Global Positioning System (GPS) receivers were also installed in many robots for localization and navigation [30,35,42,48,66,68,130].
RFID uses electromagnetic fields to automatically identify and track tags attached to objects. GPS receivers exploit a satellite-based radio-navigation system to deduce the geo-location of objects they are attached to [57].
The relationship between the aforementioned sensors and the robots’ sensing/actuation capabilities, as they appear in related work, is presented in Table 3. The relationship between the intended use of these sensors, the desired actions and targeted application areas is illustrated in Fig. 5.
Sensors and intended use
Sensors and intended use
Software
In regards to software, some papers [3,30,66,102] employed the popular Linux-based Robot Operating System (ROS) [95], which provides the communications infrastructure to program, operate, debug and control the robot as a system of systems.
ROS is an open-source framework for writing software for robotic systems. It consists of a number of libraries, tools and sets of conventions to simplify the task of writing software for complex mechatronic systems [95]. ROS includes a large variety of algorithms and functions for creating new software components and drivers. ROS supports multiple sensor technologies as well as programming languages, of which C++ and Python are the most important. Nowadays, ROS is one of the most exploited tools for developing algorithms in the context of robotics due to its flexibility and simplicity. However, it is worth mentioning that ROS has no real-time capabilities. This means that ROS does not provide guarantees about the timing of operations, hence it is not intended for operations that have strict timing requirements.
In the case of safety critical systems in which hard real-time constraints exist, real-time operating systems must be adopted. The real-time operating system (RTOS) is one such system, used in [42], which handles the execution of tasks in order to meet their time deadline, but also facilitates memory management and accessing resources. The two main design philosophies are: event-driven and time sharing. An event-driven scheduler switches between tasks when an event of higher priority requires to be accommodated. On the other hand, a time sharing scheduler switches among tasks based on a periodic clock signal.
Other papers used GOLEX [16,118], Embedded C [9], the Multi-target Robot Language (MRL) [81], Node.js [105] as well as OpenWSN [102]. OpenWSN is an open-source implementation of the IEEE/IETF 6TiSCH protocol stack [132]. 6TiSCH is a promising Working Group that aims to achieve industrial-grade performance in terms of jitter, latency, scalability, reliability and low-power operation for IPv6 over IEEE802.15.4e TSCH. Timeslotted Channel Hopping (TSCH) is a Medium Access Control (MAC) protocol based on Time Division Multiple Access (TDMA) schedule.
Besides the aforementioned software systems and platforms, Ray [96] has listed numerous emerging cloud-based robotics platforms that could be used together with IoRT architecture. As an example, Simoens et al. [109] employed the DYAMAND middleware [85] for abstracting the different protocols and interfaces of the installed sensors of the robot.
Communication
In terms of communication protocols, the majority of related work used Wi-Fi (i.e. 16 papers), while IEEE802.15.4 and ZigBee [45,48,102,128,129], third or fourth generation of broadband cellular network technology (3G/4G) [6,7,66] and Bluetooth [6,7,45,48,64,66,109] were also used. Regarding Bluetooth in particular, the Bluetooth Low Energy (BLE) technology was used for communication among robot components.
Wi-Fi has been used where wide coverage was required (i.e. up to 100 meters) and/or it was not possible to propagate messages via intermediate nodes [5]. Since Wi-Fi consumes much energy, it has been used where autonomy had not been an important issue (e.g. education, health, tour guiding in museums). 3G/4G has also been used in scenarios where connectivity was difficult (e.g. rescue operations) but it consumes more energy in comparison to Wi-Fi. On the other hand, ZigBee and BLE have been used for short-range and low-energy communication scenarios, where there was a specific (indoor) topology/infrastructure and need for autonomy (e.g. industry, domestic support, entertainment). Zigbee is a worldwide standard for low power, self-healing, mesh networks offering a complete and interoperable IoT solution for home and building automation.
An aspect not explicitly addressed in the surveyed papers is the security of software, communication messages and the actual data involved. Two papers mentioned the use of the HTTPS protocol for secure communication [6,7] while NASA employed the WITS Encrypted Data Delivery System (WEDDS) and its public key infrastructure during the Mars polar lander mission [11]. HTTPS is an extension of the HTTP protocol for secure communication.
Summary
It is worth investigating in which ways the IoT-enabled robots of the related work under study make actual use of Internet technologies, according to the definition of IoRT [96,108,125], as mentioned in the introduction. Table 4 presents the classification of the surveyed papers in different classes, based on how they use Internet technologies.
Use of Internet technologies in IoRT
Use of Internet technologies in IoRT
The majority of the surveyed papers use TCP/IP communications for interaction between the robot and the outside world [98]. Some of these papers incorporate principles of the WoT and they would be analyzed in the Section 6. Java ORB is used in [51,74] for managing distributed program objects, while Brizzi et al. [15] employ an IPv6-based 6LoWPAN architecture for communication between robots and wireless sensor networks. 6LoWPAN is a working group and standard for the application of IPv6 over low-power sensors and wireless sensor networks [84]. An implementation of 6LoWPAN via the 6TiSCH technology [132] was performed in [102], combining 6LoWPAN with relevant software protocols and implementations such as the Constrained Application Protocol (COAP) [104] and the RPL IPv6 routing protocol [135].
Finally, the last category of Table 4 is about papers using cloud services for storage, processing, updates and management/control. These papers touch upon the research area of cloud robotics [46], which deals with infrastructures and protocols for machine-to-cloud (M2C) communications. Some interesting relevant concepts mentioned in related work [35,129] are Information Centric Networking (ICN) [2] and Software-Defined Networking (SDN) [77]. ICN is an approach to evolve the Internet infrastructure away from a host-centric paradigm based on the end-to-end principle, to a network architecture in which information is the focal point. SDN is an architecture that aims to make networks agile and flexible, improving network control. In the context of IoRT, ICN and SDN are approaches/architectures for facilitating robot control, as well as communication and networking between robots and the outside world through the Internet.
Table 5 lists which Web technologies have been used in the surveyed papers. Papers that do not use any Web technologies have been omitted from the table. Most of the papers use a basic HTTP interface for interacting with the robot, while the underlying communication is realized using different communication protocols. In Section 4.2, it was mentioned that WoT-based developments should employ REST and not only basic HTTP interfaces. However, we still include in Table 5 papers with only basic HTTP interfaces for interaction with robots, because they constitute the majority of related work.
Use of web technologies in WoRT
Use of web technologies in WoRT
Actual Web servers on the robot are installed in [42] (i.e. commercial Keil Tools C/C++ cloud-based compiler) and [105] (i.e. NodeJS server platform), while some papers move one step further, creating RESTful APIs for interacting with the robot’s features and operations [27,66,123]. NodeJS is a platform built on the browser Chrome’s JavaScript runtime for easily building fast and scalable network applications.
To enable Web-based interaction, some papers used platforms such as the WebIOPi IoT framework [123], the Web interface for telescience (WITS) [11], the WAMP server [27], the MASSIF platform [14,109] and NodeJS [105]. These platforms provide Web servers and support for exposing robot services as API calls, control and debugging of the software used for programming the robots (e.g. WebIOPi for Raspberry Pi), control dashboards and easy installation setups (e.g. WAMP), remote access to the robot (e.g. WITS for the planetary rover mission to Mars [11]), management of asynchronous messaging and handling of thousands of concurrent connections (e.g. NodeJS), semantic annotation, reasoning and integration of IoT data (e.g. MASSIF [14]) etc.
Shin et al. [105] used the Web Service Description Language (WSDL) to describe the services provided by the REST API of their UAV [22]. WSDL [22], similar to Web Application Description Language (WADL) which is generally more suitable for Web-based applications [41], is used to describe service and their semantics in order for humans and machines to be able automatically to use these services by creating the appropriate requests via TCP/IP calls.
Finally, Web-based message brokers, such as RabbitMQ and Crossbar.io, are used in [27,109]. Message brokers are intermediary computer program modules that translate a message from the formal messaging protocol of the sender to the formal messaging protocol of the receiver. These messaging brokers are suitable for high-performance, scalable distributed messaging where multiple publishers and subscribers of information are involved (e.g. health monitoring scenarios with care-providers involved [27,109]). In this case, they are called publish/subscribe architectures or infrastructures.
An advanced aspect of WoT towards seamless M2M communication and understanding is the use of Semantic Web technologies to describe services and data, towards a semantic WoT [92]. The semantic WoT involves technologies for uniformly describing WoT data streams, devices and services, allowing easy and fast integration with other sources of information, platforms and applications towards advanced knowledge and reasoning [59].
Semantics are particularly important in robotics, for the understanding of space, ambient environment and surroundings of the robot and for better reasoning. Semantics can be described and analyzed by means of ontologies and vocabularies (e.g. OWL, SSN [24], IoT [103]). Ontologies include concepts and categories in a subject area or domain (i.e. where the robot is operating) that show, describe and explain their properties and the relations between them. Ontologies are not enough though and need to be accompanied by description languages (e.g. RDF, OWL), as well as query languages (e.g. SPARQL, CQEL). Description languages facilitate consistent encoding, exchange and processing of semantically-annotated content. The Resource Description Framework (RDF) is a general-purpose language for representing information on the Web, while the Web Ontology Language (OWL) is a formal language for representing ontologies in the Semantic Web. Semantic query languages are necessary for retrieving and manipulating data stored in description languages such as RDF/OWL, being able to answer complex queries and produce advanced knowledge by combining different sources of information together.
From the surveyed papers, Simoens et al. [109] employed semantic technologies to describe the robots’ produced data by means of the WSN and SSN ontologies while Brizzi et al. [15] described robotic services by means of semantic web services. Semantic Web services [76] are similar to Web Services, but they additionally employ standards for the interchange of semantic data.
Discussion
This section discusses the general findings of the survey. Specifically, Section 7.1 refers to the technical aspects of the surveyed work and Section 7.2 examines the actual relationship between related work under study and the general principles of IoT/WoT. Then, Section 7.3 captures the big picture in relation to research and development in the area of robotic IoT, while Section 7.5 provides future research opportunities in the IoRT domain. Finally, Section 7.6 summarizes the take-home messages of this survey paper.
Technical aspects
A large percentage of the research work under study (36%) employed open-source hardware and prototyping platforms, connecting them to a wide variety of sensors (i.e. 14 different sensor types). The mechanical parts of the robots involved 16 different actions, with movement, observation and navigation being the most popular ones. Sixteen application areas have been recorded, with health being the most popular application domain for IoT-based robotics. This makes sense, considering that health applications, especially at domestic level, require easy interconnection with other devices of a smart home or fast notification/alerting and communication with care providers. Thus, the IoT/WoT protocols are appropriate for this interconnection with low effort.
In regards to communications, Wi-Fi was the most popular technology (36% of the papers), followed by Bluetooth (16%) and ZigBee (10%). The characteristics of these communication protocols are described in [5] and researchers should consider the most appropriate technology for their implementations, taking into account application requirements such as range and coverage, energy consumption and autonomy of the robot, security, mobility aspects etc.
Unfortunately, only 2 papers (4%) explicitly mentioned security measures during message communication. Some papers might have used the security features provided by the underlying platforms and operating systems used (e.g. ROS, RTOS, MRL), but this has not been specified by the authors explicitly. We argue that nowadays IoT offers a wide range of secure communication protocols [86] and they should be harnessed by researchers to increase the security of their robot implementations. IoT could also be used to enhance humans’ access control via smart authentication, i.e. by means of various biometrics, face or voice recognition etc.
Connection to the Internet/Web of Things
Our research shows that some papers claimed to be IoT-enabled without giving any details of the connection between IoT and robotics [3,64,82,83]. Also, the majority of the surveyed papers (29 papers, 59%) use merely TCP/IP communication, which is only a small part of the concepts of IoT and IoTR [96,108,125]. In regards to the WoT, many of the surveyed papers (20 papers, 40%) perform HTTP-based communication or provide only some basic HTTP interface for interacting with the robots.
The most complete papers in IoRT are those employing cloud robotics (see Table 4), while the most complete ones in WoRT are those using REST APIs for robot control and/or semantic web technologies (see Table 5). The observations that a limited number of papers employ IPv6 architectures or cloud robotics (12 papers, 24%) and/or use REST APIs or semantics (5 papers, 10%) are indications that the penetration of IoT/WoT in robotics is still low and that the IoT/WoT are still not largely and properly used in robotics. This phenomenon has been observed also in WoT frameworks in the past [55], where the authors claimed to have WoT-ready frameworks, however their developments missed some important elements of the WoT principles.
As mentioned before, cloud networked robotics is a modern, promising research area [46]. New cloud-based software systems make the integration between robotics and IoT much easier [42,120]. The ROS could be used for connecting robots to the cloud [95] (i.e. together with the FIROS tool [67]). Such possibilities are also provided by RoboEarth [127], a system for sharing knowledge between robots. Rapyuta, as the RoboEarth cloud engine, helps robots to offload heavy computation by providing secured customizable computing environments in the cloud [47]. Towards WoRT, the work in [63] allows to build REST APIs for ROS, while development of Web-based services for robotic devices is possible via [75].
The big picture
To complete the discussion on the IoRT/WoRT topics, we try to capture here the big picture of this research area. Figure 6 depicts the general suggested architecture for IoRT/WoRT systems, considering the principles of IoT and WoT applied in robotics. Communication between sensors and electronic platforms is via the IPv6 protocol, while sensory platforms can embed Web servers themselves [61]. Figure 6 lists some of the operating systems, as identified in related work under study for sensors’ and robots’ programming, or for deploying Web servers on the robots.
The general architecture for IoRT/WoRT systems.
Some promising platforms for robots’ programming not employed in the surveyed papers but still worth mentioning are FIROS [67], a tool for connecting mobile robots to the cloud (i.e. by using ROS), BrainOS,1
BrainCorp, BrainOS, https://www.braincorp.com/brainos-autonomous-navigation-platform.
Towards full Web integration, robots may expose their services as a REST API, for easier reuse of their capabilities and features by authorized third parties. Robots may also incorporate semantic engines for annotating semantically their services and data, towards seamless M2M interaction. As mentioned in Section 6.1, researchers employed various ontologies to describe robotic data (i.e. WSN, SSN) and robotic services (i.e. Brizzi et al. [15]). A set of interesting ontologies that could be used in future IoRT systems is the IEEE Ontologies for Robotics and Automation (ORA) [94]. These ontologies could also describe the relationship between a robot agent and its physical environment. As an example, Jorge et al. [52] used the ORA ontology for spatial reasoning between two robots that must coordinate for providing a missing tool to a human. Another interesting set of semantics is proposed in [117], for projecting the effects of actions and processes performed by the robots and their sequence. Besides ontologies, query languages and engines (e.g. SPARQL, CQEL) are important for semantic reasoning.
For completing IoT integration, especially in cases where robot swarms are involved or high scalability/performance are needed, cloud services could be the solution. The cloud could be used for more advanced processing and for storage of information, but also for efficient messaging via publish/subscribe infrastructures. Concepts such as ICN [2] and SDN [77] could also be employed for better overall management and control of the robots in a more abstract and generic manner.
As mentioned in Section 3, comparison of IoT/WoT existing software and hardware platforms is out of the scope of this paper. It is worth commenting however that the inclusion of IoT/WoT protocols gives important benefits to (not only) robotic systems, such as interoperability, seamless M2M communication, easier integration to existing systems and infrastructures, as well as use of well-known and popular technologies for programming, management and control [39,134]. Some of these benefits have been highlighted in Sections 4.1 and 4.2. As mentioned before however, more studies are needed in order to further validate the impact of the IoT/WoT protocols and technologies on the real-time performance of robots.
It is important to define and accept some standard performance metrics that can be used to assess the performance and/or compare various robotic systems [112]. In traditional robotics, many performance metrics exist in order to assess the quality of a robot or a network/swarm of robots, while the ones most commonly adopted are power/energy consumption, bandwidth, latency, throughput, resilience to errors, packet loss and locality of program execution. Some metrics are more suitable for UAV, such as setup time, flight time, inaccuracy of land, haptic control effort and coverage ratio [53].
Unfortunately, our analysis cannot get in depth on this aspect because none of the cited authors discussed, mentioned or evaluated their proposed robotic systems using any assessment metrics and/or comparing their systems with existing similar robotic implementations. The only exception was the work in [102], where the authors assessed the quality of service of their robotic system (i.e. UAV with Telosb sensors), expressed in terms of network joining time, data retrieval delay and packet loss ratio. Their conclusion was that the performance of the UAV based on the aforementioned metrics satisfied the mission requirements.
Generally, authors of work under study preferred to focus their research on the feasibility and demonstration of connecting robots to the Internet/Web, as well as to the new possibilities (i.e. sensing, actions, application domains) that arose by their robot models/implementations (see Fig. 5). We acknowledge the fact that in many cases there had not been any similar robotic systems to compare with.
We will try here to predict the metrics which should be used for the assessment of IoRT systems as this trend is emerging around the world. The fast growth in autonomy and new functionality/capabilities of robots (including their Internet/Web-enabling) is followed by the fast growth in complexity of algorithms as well. This translates in higher demands for data, computational and storage resources (that is common, for example, for machine learning-based algorithms). However, due to the limited amount of computing and storage capabilities of (mobile) robots, the energy consumption has great importance in IoRT, more than in general IoT applications. Furthermore, in the near future, IoRT is expected to involve more operations that require real-time and safety-critical robot functionalities (e.g. R2H interaction). In such scenarios, the consistent reduction of the latency plays a crucial role [1]. It is still important to quantify and assess the impact of the TCP/IP stack and/or Web technologies to the real-time operations of robotic systems. The papers under study have not discussed this issue. Although other research works demonstrated that the impact of IoT/WoT on embedded devices is low [28,49,61,98,101], this might not be the case for time-critical robots.
Robots are non-homogeneous and complex compositions of sensors, actuators, control systems, mechanical parts and artificial intelligence algorithms, which are meant to solve different and continuously-changing tasks. The flexibility and adaptability of robotic systems must be reflected in the IoRT/WoRT systems as well. Thus, an important aspect for the further development of IoRT systems is the ability to quickly adjust to changes of context, without application-specific adaptations (e.g. different number and kind of robots, objectives, tasks, environments) [125]. These contextual changes can occur very fast in an Internet/Web-enabled system, because the robot has real-time access to information and content, provided by Web services, APIs, other robots of the swarm or other citizens of the IoT ecosystem.
Challenges and future opportunities
Robotics, together with IoT, constitute dynamic and active research fields and there is much ongoing research in these areas. This section focuses on the existing challenges and barriers, as well as future research opportunities that arise from the combination of these technologies together in the future. Some of these challenges and/or opportunities are the following:
Although the impact of IoT/WoT technologies and protocols on embedded devices may be low [28,49,61,98,101], this impact might still be considerable in time-critical robotic systems, where decisions need to be taken in fragments of a second. We welcome research efforts dealing with this aspect, studying in detail how different technologies and protocols of IoT and WoT affect performance. This could be well related to the security of the messages exchanged between robots and infrastructures [114]. As mentioned in Section 7.3, it is unfortunate that no comparisons of the robots with existing literature in terms of performance metrics was performed in the related work under study. We expect that future research will focus more on performance, studying the possibilities and constraints of the robots considering low-level metrics such as energy consumption and autonomy, bandwidth, latency, throughput, resilience and fault tolerance, as well as high-level metrics such as ease of use and control of the robot, overall engagement and social behaviour, safety etc. It would be also important to study the penalty and trade-offs in energy consumption and autonomy when Web servers and the IPv6 stack are added to the robots [49,61]. In relation to the above, an important challenge(and desired characteristic) of robots is autonomy in operation. IoT has active research in breakthrough high-performance architectures, algorithms and hardware that will allow wireless networks to be highly efficient [73], powered by tiny batteries, energy-harvesting [54], or over-the-air power transfer. Moreover, new communication systems based on biology and chemistry are expected to evolve in IoT research, enabling a wide range of new micro- and macro-scale applications [4]. Another important research challenge for robots is the collective behaviour of robot swarms, or the coordination and control of multi-robot systems towards an optimized outcome. Such swarms have been proposed for collective industrial construction [91], transportation and box pushing [89], as well as for smart homes [99] and search and rescue [26]. More generally, the authors in [70] propose a plan-based approach, based on PEIS Ecology [99], to automatically generate a preferred configuration of a robot ecology given a task, environment and set of resources. In this robot swarm ecosystem, IoT can offer effective solutions for networking, communication among the robots and mobility [26]. Just like there exist complete operating systems (e.g. TinyOS, Contiki) and application-layer protocols (e.g. COAP, ZigBee) for IoT sensors and devices, we expect similar operating systems to appear for robots [136]. As mentioned before, early efforts in this direction are BrainOS and MIRA. A culture of developers and researchers still needs to be built around these platforms/frameworks, sharing code, experiences and solutions to the community. Worth mentioning is the Brains For Bots SDK by Neurala,2 Neurala, Brains For Bots,
Combining online social networking with IoT/WoT-enabled robots [121]. In this case, online social networks could be used for storing and sharing links to resources of interest for the R2R and R2H interactions, facilitating sharing of robotic services among online trustful contacts [60]. Robots could recognize or authenticate users, giving them access to some of their controls, based on the peoples’ online profiles and endorsements they have from other authorized people.
In industrial applications and logistics, technologies that blend the physical and digital context are important. Broader and more expansive information capture and processing via IoT/WoT, combined with smarter manipulation and movement of physical materials via robots, can deliver new benefits such as higher efficiency of operations, more insights and visibility, as well as better interaction between components, systems and actors. Blockchain could be relevant, to keep an immutable distributed ledger among IoT sensors and robotic systems of untrusted partners involved in some supply chain ecosystem [119,125]. A blockchain is a growing list of records, called blocks, that are linked using cryptography [62].
The trend towards human-centric design of robotic systems is expected to continue [136] and robots will become more integrated in our everyday lives, either for assistance in common tasks or for actual support of people in need, e.g. movement of people with paralysis. Thus, they need to capture humans’ emotions and social behaviour to understand how to react in different situations [32]. This information might come from IoT sensors, either installed on the robots or in the nearby environment (i.e. smart homes/buildings) or from the online social networking presence and activity of humans. Ethics is an important dimension in this direction [65] and needs to be considered a-priory.
Towards human-centric robotics, the teaching of robots to take actions by means of natural language instructions is a key aspect. This challenge could derive knowledge and ideas from Web Semantic technologies, as well as from ontology-based natural language interfaces for controlling IoT devices, such as im4Things [87] and home automation [10]. These language interfaces allow people to interact with machines/robots using a human language.
We expect service robots to enter new real-world environments in which IoT has already penetrated [58]. These environments could be sports (i.e. assistance and safety in sports having risks such as climbing and parachuting), health (i.e. not only monitoring patients but also taking first-aid actions if needed), ecology and environmental monitoring, surveillance and security in social spaces (i.e. schools, airports, urban hotspots etc.), gaming (i.e. robots become comrades or opponents in real-world gaming scenarios), customer service (i.e. restaurants, hotels, gyms, tourism), the movie industry and many others. The autonomous learning capabilities of robots, together with their potential actions due to their mechanical parts, could offer more opportunities in these new environments.
In recent year, research has mostly progressed in the direction of developing novel applications and combinations of IoT/WoT and robotics. While few theoretical frameworks have been proposed for IoT and WoT [88], there has not yet been particular focus and wide interest on studying a general theoretical framework for IoRT and WoRT. This task, although not yet studied enough, can play an important role for further strengthening and broadening the applicability of IoRT/WoRT to new robotics challenges.
Summarizing the study performed in this paper, some take-home messages can be the following:
As the IoT penetrates different domains, application areas and scientific disciplines, it started to penetrate also the research area of robotic systems. There is a wide range of sensors used, intended actions of the robots and application areas where the IoRT/WoRT-based robots operate. The most popular actions involve moving, observing via computer vision, flying and navigating. The most popular application areas are entertainment, health, education and surveillance. A large percentage of the surveyed papers employed open-source hardware and prototyping platforms, making the physical connection to a wide variety of different sensors possible. Regarding communications, various protocols are being used, with Wi-Fi being the most popular, followed by IEEE802.15.4 and Bluetooth. The decision on which communication protocol to use was mainly influenced by the coverage range required, as well as the need for autonomy and longer lifetime of operation (i.e. energy consumption). The IoT is still not used at its full potential in robotics. Cloud robotics and the IPv6 protocol are not fully utilized. The same holds for the principles of WoT, which has not yet been fully exploited in robotic systems. Services of the robots are not generally exposed as a REST API, nor the services and data are described via Web semantic technologies. There is an increasing number of operating systems, software tools and platforms for developing IoRT/WoRT-based robotic systems. Most of this software enables access to cloud computing and publish-subscribe infrastructures for scaling computation and storage capabilities and for supporting robot swarms more easily. Related work mostly focuses on feasibility studies and demonstrations, which showcase the benefits of IoT/WoT in robotics, mainly in terms of ubiquitous access via Internet/Web and easy interoperability with other systems. Aspects of performance, especially considering the overhead produced by the TCP/IP protocol, have not been well studied. There are still numerous open issues and gaps for future research in this emerging intersection of research areas (see Section 7.5).
Conclusion
This paper studied the current use of the Internet of Things (IoT) in robotics, through various real-world examples encountered through a research based on a bibliographic-based method. The concepts, characteristics and architectures of IoT, as they are being used in existing robotic systems, have been recorded and listed, together with popular software, hardware and communication methods. Moreover, the application areas, sensors and robot services/actions incorporated in IoT-based robots are presented. Further, the current application of the Web of Things (WoT) in robotics has been investigated and the overall potential of the Web of Robotic Things has been discussed in the paper. A general observation is that some of the advanced concepts of IoT/WoT are not yet being used by researchers in robotics. Finally, future research directions and opportunities are proposed.
Footnotes
Acknowledgements
Andreas Kamilaris has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 739578 complemented by the Government of the Republic of Cyprus through the Directorate General for European Programmes, Coordination and Development.
Nicolò Botteghi has received funding from Smart Tooling. Smart Tooling is an Interreg Flanders-Netherlands project sponsored by the European Union focused on automation in the process industry: making maintenance safer, cheaper, cleaner, and more efficient by developing new robot prototypes and tools.