CoBuilt 4.0: Investigating the potential of collaborative robotics for subject matter experts

Construction sector: challenges

Globally, the construction sector labour-productivity growth averages one percent a year over the past two decades, compared with 2.8 percent for the total world economy and 3.6 percent for manufacturing. A recent global McKinsey Construction Report identifies this as a global challenge where less than 25% of construction firms matched the productivity growth achieved in the overall economies over the past decade. ² Growth and development in the construction sector are complex as this non-homogeneous field consists of both large scale, heavy construction works (i.e. civil and industrial work, or large scale housing) and fragmented specialized trades (i.e. carpentry, masonry, and stonemasonry) as subcontracted work on small scale projects. Whereas the first usually shows higher productivity due to volume, the second is labour and time-sensitive. Furthermore, many practices and technologies adopted in the construction industry overall have remained largely unchanged for decades. This has resulted in a sector characterized by low productivity, an alarming safety record, mounting costs, and high wastage of materials and labour shortages. Significantly, the McKinsey report also identified seven key moves that simultaneously hold potential to increase productivity, advising namely to: reshape regulation; rewire the contractual framework to reshape industry dynamics; rethink design and engineering processes; improve procurement and supply-chain management; improve on-site execution; infuse digital technology, new materials and advanced automation; and reskill the workforce. ³ Amongst these, the last two could be solved by introduction of robotic technologies to the construction sector, building up on advancements of Industry 4.0.

Industry 4.0, interoperable systems

Industrial development and growth follow systems innovation that shape production processes and methods, and in this line Industry 4.0 is a result of the Fourth Industrial Revolution and equally a game-changer as were the introduction of mechanization and steam/power, of mass production and electricity, and of computer and automation. This fourth Industrial Revolution commonly refers to cyber-physical systems and manufacturing technologies that fuse the physical and digital worlds, including advancements in artificial intelligence (AI), the Internet of Things (IoT), 3D printing, autonomous vehicles and robotics. Driven by a digitisation of the manufacturing sector, Industry 4.0 is characterized by a rise in data volumes, computational power, and connectivity in wide-area networks; the emergence of analytics and business intelligence capabilities; new forms of human-machine interaction such as touch interfaces and augmented-reality systems; and improvements in transferring digital instructions to the physical world, such as 3-D printing and advanced robotics. ⁴ Within these dimensions and of particular interest for robotic technologies are: (a) the concept of ‘interoperability’, referring to the ability of machines, devices, sensors and people to connect and communicate with each other via the Internet of Things (IoT) or the Internet of People (IoP); and (b) the available technical assistance of cyber-physical systems to support humans by conducting a range of physical tasks. In this context, robotic technologies potentially could play a vital role. The adoption of robotics technologies on sites could minimize hazards to workers, speed up processes, reduce waste, shorten construction timescales and reduce costs. ⁵ The 2016 US National Robotics Roadmap stresses the importance that productivity gains benefit all of society, and not just the owners of robots. ⁶ Robotics technologies have the potential to make work safer and more satisfying while reducing the dirty, dull and dangerous aspects of work in many occupations, ⁷ including the construction sector. The use of robots is rapidly gathering pace as demonstrated by the interest of major industry players and increasing number of start-up companies. Furthermore, and significant in the context of this research, the 2017 McKinsey report suggests that investigation into several areas simultaneously holds potential to double productivity, including amongst others to: rethink design and engineering processes; improve onsite execution; infuse digital technology, new materials, and advanced automation; and reskill the workforce. This creates a compelling argument for the implementation of robots as co-workers, collaborators in and potentially also demonstrators of construction processes.

Robotics and cobots

Recent investigations into construction standards and methods with industrial robotic arms have resulted in novel methods for bricklaying, fluid deposition, timber sheet cutting and assembly, bespoke welding of steel elements, or tile cutting. There exists an extensively body of research by NCCR (Gramazio&Kohler Research, ETH Zurich), the Institute of Computational Design (ICD Stuttgart), Hyperbody Research Group (TU Delft), the Institute of Advanced Architecture Catalunya (IAAC, Barcelona), DRG Robotics Group (Harvard GSD) amongst others. However, whereas these investigations typically demonstrate highly complex geometries, material performance, or bespoke fabrication of elements, the integration towards standard industry processes, or a focus on expanding human labour through robot co-working and support is not at the forefront of these investigations. Moreover, several challenges potentially impact on expanding towards onsite and direct industry/trade implementation, including the overall size of robot arms (i.e. large scale, weight, power supply) that pose movability issues. Industrial robot setups robotic arms are generally confined to a caged area and operated from a distance to ensure the safety of the operator, as industrial robots can weigh upwards of one metric ton and can generate a deadly level of torque and momentum. Collaborative work or even operating large industrial robots remains high-risk, safety issues and closed work envelopes restrict direct engagement in predefined robot protocols. The dynamic process of interaction between human-robot co-work requires sensor-based work protocols and timed choreographies. Fast adapting to unexpected changes is difficult to process due to closed robot programming, and workflows developed through laboratory’s experimentations cannot be directly transferred to onsite applications. Robot programming is generally based on outcome and product orientation rather than movement, with robots lacking in capacity of learning from the surrounding. The bespoke nature of tasks often requires robots to be controlled by tailored pieces of software for performing actions. The construction industry requires case-by-case scenarios and thus high adaptability of the robotic code and, therefore, a deep understanding of the programming language by the technician/computational designer in charge.

In contrast to standard robotic practices, CoBots are categorised as robotic devices that manipulate objects in collaboration with a human operator,^8,9 or otherwise assist humans in activities through direct interactive communication and action. The principle of CoBots refers here both to robots with integrated sensor specifications such as ABBs Yumi or KUKAs iiwa, or other robots custom designed for human interaction, but more importantly to the larger concept of collaboration between humans and machines. In that sense, CoBots can provide a viable alternative and extension to standard processes of manufacturing and construction, which provides an initial framework for strategizing collaboration:

Current research into collaborative robots investigates: autonomous mobile service robots that perform tasks in a multi-level office environment (Coral at Carnegie Melon University), directed towards ‘Indoor Mobile Robot Localization’; ‘Symbiotic Human-Robot Interaction’ and ‘User-centred Task Planning and Scheduling’. Other investigations focus on the situation-dependent active task contribution for collaborative processes involving creativity (Interactive Robotics Group (IRG) at Massachusetts Institute of Technology (MIT) to address creative solutions onsite and adhoc. To this extent, new industrial CoBots are manufactured for commercial enterprise: Baxter (Rethink Robotics), iiwa (KUKA) or YuMi (ABB). Baxter is designed to be taught through motion tracking whereby workers perform tasks by simply guiding the robotic arm, indicating clutch and release for object placement. ¹⁰ Similarly based on integrated sensor features, the iiwa provides direct interfaces for motion learning, including speed, force and directionality through external motion input. YuMi offers an added second work arm to expand to a seven-axis system, thus offering one extra degree of freedom, including kinematic redundancy that allows YuMi to rotate its arm to avoid collision and twisting while keeping its end effector in the exact same position.

Industry survey: complexity of tasks, safety, risk awareness

Initially, the research focused on a survey of industry and construction practice to understand challenges and concerns in adopting robotic technologies and CoBots for Australian construction standards. To this extent, discussions with industry and construction companies served to explore potential opportunities and challenges identify drivers of adoption and survey how technologies might be introduced in different organizations to different stakeholders, from investors, business owners/contractors, clients to end-users such as tradesperson, sub-contractors, apprentices. Two focus groups were created. As illustrated in Figure 1, a thematic data analysis investigated potential opportunities, challenges or barriers, drivers for adoption, introducing new ideas and education materials (i.e. formats and needs). A subset of secondary themes identifies recurring patterns mentioned by participants such as activities, job types, benefits of CoBots, or need for humans in tasks or other machines.

OPEN IN VIEWER

Figure 1.

Comparison of two focus groups and information derived (FG|1 and FG|2). Primary and secondary themes were identified responses for potentials, challenges, ideas introduction, education across discussion groups.

With varying participants across the two focus groups and different perspectives based on the specific nuances of their work environments, crosscutting themes were arising, which yielded two different quadrant analysis: one based on safety and complexity of tasks, and a second based on the approaches towards innovation and costs of adoption.

Figure 2 illustrates the discussion for evaluating characteristics of single or multi-task robots by intersecting safety with the complexity of task. Quadrant 1 (bottom left) highlights the benefit of single task CoBots for relatively safe, repetitive, simple tasks in controlled surroundings. Quadrant 2 (bottom-right) shows single task CoBots for dangerous but simple to execute work. Quadrant 3 (top left) illustrates benefits for multi-tasking CoBots working to a degree of complexity while providing safety. Quadrant 3 (top-right) features the true potential and extent of multi-tasking CoBots for dangerous and complex tasks.

OPEN IN VIEWER

Figure 2.

Differentiation of task groups in relation to complexity and safety.

Pilot studies: investigating task sequences to identify potential capture of computable actions

Based on the results derived from these discussions and survey, the consecutive pilot studies focused on carpentry in small-to-medium size enterprises in the Australian construction industry, seeking to understand and identify opportunities in the current workflows of carpenters for the role of collaborative robots. We found that several repetitive functions relevant for CoBot prototype development include tasks that are also relevant to other trades, such as cutting, sawing, passing, standard cuts, lifting, moving, holding/securing.

The research developed a novel methodology for the capture and analysis of the body movements of carpenters, resulting in a suite of visual resources to aid us in thinking through where, what, and how a collaborative robot could participate in the carpentry task. ¹¹ Based on interaction analysis, this method focuses on the physical actions, movements, material and tool use, taking place in space and time, and a specific orientation to collaborative physical actions. The research contributes here through a novel set of visual resources generated from the interaction analysis, enabling identification and evaluation of potential insertion points for CoBots in the already established workflows of construction workers (Figure 3).

OPEN IN VIEWER

Figure 3.

Methodology Phase 2. From observation of task performance to identified sequences in workflow.

We investigated a pilot study in a workshop situation with in-house carpenters as foundation for a real-time, onsite study at a construction site (Figure 4). This allowed us to empirically observe, capture and analyse the work practices of carpenters building a timber frame structure (such as for a residential house).

OPEN IN VIEWER

Figure 4.

Simulation of work sequence in pilot study (DMaF, University of Sydney, in-house carpenters).

Importantly, we developed and tested the observe-and-record step of the proposed methodology. In presenting and explaining the results of the analysis, the structured sequence of actions and single visual resource (Figures 5–7, Table 1) enables us to understand, access and close investigation to the complex, fluid reality of an active building site.

OPEN IN VIEWER

Figure 5.

Sequenced action across ten-stage construction for building the timber structure of a house from prefabricated frames on a live building site.

OPEN IN VIEWER

Figure 6.

Detailed framing actions highlighting how the carpenters use their hands and feet to help prop and stabilize the frames.

OPEN IN VIEWER

Figure 7.

Person-to-Material, Person-to-Tool and Person-to-Tool-&-Material.

OPEN IN VIEWER

Table 1.

Evaluation of likelihood of CoBots helping a human carpenter in a range of collaborative actions related to framing. Gradient code indicates functionality adaption (white: not possible/adequate, midgrey: partial application, dark grey: good functionality).

Collaborative action	Type of CoBot (arms/mobility)
	Single- arm CoBot, static	Dual-arm CoBot, static	Single-arm CoBot, mobile	Dual-arm CoBot, mobile
1. Sorting frame (lifting and moving to pile)	No use, due to lack of mobility.	No use, due to lack of mobility.	No use, as two arms are required to hold and stabilize the frame.	Possible, depending on relative size of cobot to frame.
2. Carrying frames (2 carpenters)	No use, due to lack of mobility.	No use, due to lack of mobility.	Can hold one end/side of timber/structure with a grip end effector, and locomote.	Can hold one end/side of timber/structure with two grip end effectors, and locomote.
3. Sliding frames	No use, due to lack of mobility.	No use, due to lack of mobility.	Can hold/push one end/side of timber/structure with a grip end effector	Can hold/push one end/side of timber/structure with two grip end effectors
4. Rotating frames	No use, due to lack of mobility.	No use, due to lack of mobility.	Can hold/lever one end/side of timber/structure with a grip end effector. May be limited by relative size.	Can hold/lever one end/side of timber/structure with two grip end effectors. May be limited by relative size.
5. Foot tapping to location	Can a robot arm replace the kicking action of a human foot?	Ditto	Ditto	Ditto
6. Hammer tapping into position	No use, due to lack of mobility and one arm.	Limited. Only if the hammer tapping point is where cobot is stationed. Assuming one arm can stabilise the frame,whilst the other arm hammer taps.	Cobot can move to where hammer tapping needs to be done, but has only one arm.	CoBot can move to where hammer tapping needs to be done. One arm can stabilize the frame, whilst the other arm hammer taps.
8. Nailing frame to slab	No use, due to lack of mobility and one arm.	Limited. Only if the nailing point is where cobot is stationed. Assuming one arm can stabilize the frame, while the other arm uses nail gun.	CoBot can move to where nailing needs to be done, but has only one arm.	CoBot can move to where tapping nailing needs to be done. One arm can stabilize the frame, whilst the other arm uses the nail gun.

In order to identify opportunities for CoBot insertion into timber framing workflow, as illustrated in Table 1, the research juxtaposes sets of collaborative actions currently performed by human construction workers carrying out a series of timber framing tasks against a CoBot with two variables of a number of articulated arms (one/two) and mobility (stationary/mobile). The degree of locomotion mobility and the number of arms has implications for the type of work the CoBot can do, and where in the workflow the CoBot can be inserted. In principle, the human action repertoire observed on the building site is amenable to performance by a CoBot. Consequently, methods of performing the action are critical, both in terms of shared process or sequenced task. For example, the collaborative task of carrying a structure implies the negotiation of physical forces between the two workers (human and robot) in an ergonomic manner. In contrast, the task of bracing a structure to enable the other worker to secure the structure with a nail gun is a different type of physical collaboration involving two discrete actions.

In summary, the analysis revealed how the team of carpenters employed a range of agreed-upon processes and routines that were intimately linked with the materials and tools they used. Verbal talk was important for checking and confirming that tasks were performed correctly and safely, as well as for convivial social interaction to enrich work. Given the amount of verbal and non-verbal communication that takes place on a building site, any CoBot to be integrated into construction workflows will need a degree of fluency in understanding human speech and gesture. This shows potential for the adoption of CoBots into construction work, which will inevitably change current workflows and traditional practices of building. Introducing a CoBot to established working processes, to onsite conditions and adding ad-hoc improvisations would require a period of mutual learning and adaptation by both the human carpenters and CoBots.

We also found that in a relatively simple task such as a building a timber frame structure for a residential house a very large number of movements exist that can be replaced by a CoBot (Figures 8 and 9, exert). However, programming each of these movements would be a near impossible task. Consequently, we argue that CoBots should adopt new movements via machine learning so that industrial robots can be trained to learn how to perform a task, instead of simply following a pre-programmed script. As next steps, the research investigated the potential for machine learning.

OPEN IN VIEWER

Figure 8.

CoBot- Carpenter Collaborative workspace: CoBot reach in static position (top left), CoBot supporting frame whilst carpenter levels (top right), CoBot assisting carpenter sliding frame (bottom left), CoBot assisting carpenter lift frame (bottom right).

OPEN IN VIEWER

Figure 9.

CoBot nailing frame to slab with custom end effector whilst carpenter supports frame.

Capturing motions and machine learning

Further strategies needed to be considered to address a complexity of purposeful movements in a seamless sequence so that a human-robot team is jointly able to deliver an action catalogue. Strategies can be based on coupling a preprogramed movement trajectory with path correction via sensor based input, ¹² or introducing a semantic robot language. ¹³ Other options are available through sensor tracking a human motion and transferring data to an adaptive (or open/interactive) robotic protocol, ¹⁴ or haptic learning to control robot movements. ¹⁵ However, machine learning process for training paths/movements is a viable alternative since knowledge in this domain has advanced to industrial robots already being trained to learn how to perform a task, such as current automated, servicing humanoid applications, ¹⁶ or collaborative repositioning in robotic-human assembly. ¹⁷

Thus, as direct extension of the pilot studies and structured investigation into action protocols, the research explored methods for data capturing processes, including movement tracking vs motion programming or machine learning. This included the establishment of a methodology for workflow capture and analysis of carpentry tasks towards human-robot collaboration in the case study investigations; and a framework that outlines two pathways of training robots through machine learning – supervised and reinforcement learning (building on previous research including Brynolfsson et al., ¹⁸ Shalev-Shwartz and Ben-David, ¹⁹ Michalski et al. ²⁰ ).

Significant is here a concept of machine learning that addresses the objective of training a CoBot arm to complete movements. However, large quantities of data are required to train a machine learning model. This is particularly complex for human movement coupled with motion capture as a source of training data. Motion capturing is a process of recording the movements of objects and people (body and limbs), whereas performance capture refers to the capturing of subtle expressions such as fingers or face. For motion, capturing movements of one or more actors are sampled many times per second using often-passive markers made out of retroreflective material positioned on the body, a technique we used in the following stages of research (see Figure 10). Other techniques include active markers where an active optical system triangulates positions by illuminating one LED at a time. Alternatively, multiple LEDs with identification software for relative positions akin to celestial navigation or Time Modulated Active Markers can be adopted as systems in which active markers can further be refined by strobing one marker at a time. Moreover, the motion capturing of arm, hand, and finger movements while completing tasks can be used to provide a data sets for machine learning so that consequently CoBots can be trained to collaborate in a movement or task.

OPEN IN VIEWER

Figure 10.

Scenes from motion capturing workshop to gain data for supervised machine learning.

An approach of motion capture plays a significant role in developing robot movement, particularly where a robots learns from a human demonstrator. ²¹ This has been successfully demonstrated with related research projects using Kinect sensing for tracking brick laying. ²² There exists a range of techniques for motion capture, including relatively limited technologies such as leap motion and Kinect sensors, or full body suits with incorporated data gloves that produce highly accurate results. The resulting raw data can be exported and processed for supervised machine learning, and to complete a desired movement or task within a range of movements. We investigated capture techniques for motion capturing of arm/hand/finger movements towards developing complex robotic systems. Once a pre-determined motion has been captured, this provides a training data set for a neural network, which enables a machine to learn the arm/hand/finger movements, once machine learning has been established all movements can be replicated via a CoBot arm. This can be of significance particularly as a technique for robotic systems that carry out a human task deemed demanding or strenuous. ²³ Furthermore, differentiations between supervised and unsupervised machine become significant since supervised learning is time-consuming and resource intensive, where certain movements require multiple action learning and also multiple participant tracking to provide a suitable data set that can be used to train. Unsupervised learning, such as reinforcement learning does not require an input to process output but allow for machine learning algorithms to use trial and error to develop a tooling path suitable to complete the task it was assigned for which skips the step of collecting motion capture data for a training set. Moreover, reinforcement learning is comparative to learning through trial and error, and particularly equipped for industrial robot arms with inbuilt sensor-based facilities. This opens a unique space for tasking robots with achieving a given goal where no training data is required. Instead, a reinforcement learning approach adopted for robotic processes can potentially uncover new approaches towards manufacturing and fabrication, where data collected can be used for objective measures for delivering similarity.

In a first iteration, the research team developed a machine learning model that simulated ‘learned’ movements (Figure 11), using Google’s Tensorflow library as reinforcement learning algorithm to task a Kuka iiwa robotic arm to learn how to search and find an object.

OPEN IN VIEWER

Figure 11.

Screenshot of a simulation environment for the Kuka iiwa robot arm that understands the mechanisms and data flow to control the arm, and weave a reinforcement learning algorithm to interact with the simulation. The ‘to-be-found’ is indicated as green bar underneath the robot arm.

Based on the research in Phase 2 towards capturing computable actions, we observed that several interaction sequences between humans start with one human reaching for a tool. Consequently we used the movement ‘autonomous searching for tool/identifying tool/reaching for tool’ as the first step for a reinforcement learning trajectory. In the set-up of the machine-learning program, the robot arm is equipped with a virtual camera (see three small images in Figure 11 on the left top). Further, the machine-learning program knows what degree of freedom the robot arm has and that the ‘to-be-found’ object (Figure 11) is within the physical reach of the Kuka iwaa arm. When executing the machine-learning program, the robot arm starts searching, is rewarded similar to any other reinforcement learning models, and improves the finding strategy over time. With completing this first movement, the research went on to simulate the movement ‘autonomous searching for tool/identifying tool/reaching for tool’ in a physical set up.