CoBuilt 4.0: Investigating the potential of collaborative robotics for subject matter experts

Abstract

Human-robot interactions can offer alternatives and new pathways for construction industries, industrial growth and skilled labour, particularly in a context of industry 4.0. This research investigates the potential of collaborative robots (CoBots) for the construction industry and subject matter experts; by surveying industry requirements and assessments of CoBot acceptance; by investing processes and sequences of work protocols for standard architecture robots; and by exploring motion capture and tracking systems for a collaborative framework between human and robot co-workers. The research investigates CoBots as a labour and collaborative resource for construction processes that require precision, adaptability and variability.

Thus, this paper reports on a joint industry, government and academic research investigation in an Australian construction context. In section 1, we introduce background data to architecture robotics in the context of construction industries and reports on three sections. Section 2 reports on current industry applications and survey results from industry and trade feedback for the adoption of robots specifically to task complexity, perceived safety, and risk awareness. Section 3, as a result of research conducted in Section 2, introduces a pilot study for carpentry task sequences with capture of computable actions. Section 4 provides a discussion of results and preliminary findings. Section 5 concludes with an outlook on how the capture of computable actions provide the foundation to future research for capturing motion and machine learning.

Keywords

Industry 4.0 collaborative robotics on-site robotic fabrication industry research machine learning

Introduction

The physical adaptability and versatility of industrial six-axis robots towards construction and manufacturing processes in design, architecture and the built environment have progressed successfully through academic and industry research in recent years. However, applications in architecture remain limited due to lacking computational, sensory and interface capabilities, which is of significance for work processes that require multiple steps, variable toolsets, or sequenced work protocols. A robot’s six degrees of freedom affords extensive movement and thus unprecedented spatial reachability in comparison to other mechanical tools introduced on the market. Lowered investment costs and multifunctional applications provide advantages for manufacturing, fabrication and construction industries. Instead of developing highly specialized, single-use machines, a standard industrial robot arm can be equipped with a broad spectrum of end effectors (similar to a human hand using different tools) and thus becomes a certified, reliable and increasingly affordable machine – an agent for Industry 4.0.

In this paper, we argue that in a context of industrial growth and shortage of skilled labour in the construction industries, the increased demands for production can be answered by a focus on human-robot collaborations, specifically investing in processes and sequences of work protocols instead of product and outcomes of such processes. This requires investigation into the potential for close interactions between co-workers with different ‘skillsets’, whereby a human provides extended movement range, fine-motor skills, variable ‘gripper system’ for toolsets through hands, and a robot provides precision, non-fatigue, or repetitive skills. To this extent, developments in the fast advancing fields of Collaborative Robots (CoBots), Cyber-Physical System (CPS), Social Robotics, and Human-Computer Interaction (HCI) can also support architecture robotics for development of frameworks, and modes of production and fabrication. Working closely with humans in shared work environments, CoBots could be adopted to perform repetitive, dangerous or specified task sequences with optimized precision. Moreover, we argue that the movement, task sequences, construction techniques and tooling processes that evolved through centuries of craftsmanship and trade practices represent a vast potential for joint human-robot collaborations and work interactions if reconsidered as a ‘learning’ framework for material and technique focused demonstrations for construction purposes.

The premise for this research is the expected escalation of housing costs and construction times as a result of severe shortage in skilled construction labour of building tradespersons (bricklayers, carpenters and stonemasons), in a context of unprecedented growth of accommodating 1.6 million new Australian residents over the next 20 years, outlined by the 2014 ‘Plan for Growing Sydney’, NSW Government, Australia.¹ Consequently, in the specific context of the Australian construction industry, this research collaboration between government agencies (Urban Growth and Infrastructure NSW), industry and trade, and academic partners (University of NSW, University of Sydney and Western Sydney University) aims to:

a) survey a spectrum of trades (i.e. carpentry, stonemasonry, masonry, plastering, glazing and ironwork) to determine the health and safety risks behind the repetition of a specific movement over a long period; coupled with a survey of tradespeople for potential adoption, including reservations;

b) develop an understanding for technical, economic and social challenges regarding the adoption of robotic and collaborative robotic technologies and develop a framework for potential applications;

c) identify a spectrum of possible approaches and limitations in using collaborative robotics for fabrication, and investigate the development of a scientifically validated workflow prototype of on-site collaboration between robots and humans while performing a set of repetitive tasks related to the field of architectural fabrication

d) develop potentials for a prototypical preliminary workflow in collaborative construction by in-situ (site-based) surveys and ex-situ (laboratory-based) trials, challenging the human to machine interaction, hardware and software limitations, and operative constraints with testing of a selected range of materials’ performance;

e) evaluate data collation to overcome closed pre-programmed protocols, and expand towards other potential transfer for movement-based interactions (optitrack) to machine learning with continuous data absorption and interpretation.

In the following, the paper will discuss a context of architecture robotics for construction, a feedback study into potential adoption for architecture robotics by construction industries, and a pilot case study. In section 1, we introduce background data to architecture robotics in the context of construction industries, Industry 4.0, and state of the art in robotics and CoBots. Here, we concentrate exclusively on robot arms as we later study and research the translation of human jobs/tasks/movement into robotic movements. Section 2 reports on current industry applications and survey results from industry and trade feedback for the adoption of robots specifically to task complexity, perceived safety, and risk awareness. Section 3 introduces a pilot study for carpentry task sequences with the capture of computable actions. Section 4 provides a discussion of results and preliminary findings. Section 5 concludes with an outlook to future research for capturing motion and machine learning.

Background

In this section, we introduce a context for the construction sector, the application of robots in architecture, and the potential of collaborative robots.

Construction sector: challenges

Globally, the construction sector labour-productivity growth averages one percent a year over the past two decades, compared with 2.8 percent for the total world economy and 3.6 percent for manufacturing. A recent global McKinsey Construction Report identifies this as a global challenge where less than 25% of construction firms matched the productivity growth achieved in the overall economies over the past decade.² Growth and development in the construction sector are complex as this non-homogeneous field consists of both large scale, heavy construction works (i.e. civil and industrial work, or large scale housing) and fragmented specialized trades (i.e. carpentry, masonry, and stonemasonry) as subcontracted work on small scale projects. Whereas the first usually shows higher productivity due to volume, the second is labour and time-sensitive. Furthermore, many practices and technologies adopted in the construction industry overall have remained largely unchanged for decades. This has resulted in a sector characterized by low productivity, an alarming safety record, mounting costs, and high wastage of materials and labour shortages. Significantly, the McKinsey report also identified seven key moves that simultaneously hold potential to increase productivity, advising namely to: reshape regulation; rewire the contractual framework to reshape industry dynamics; rethink design and engineering processes; improve procurement and supply-chain management; improve on-site execution; infuse digital technology, new materials and advanced automation; and reskill the workforce.³ Amongst these, the last two could be solved by introduction of robotic technologies to the construction sector, building up on advancements of Industry 4.0.

Industry 4.0, interoperable systems

Industrial development and growth follow systems innovation that shape production processes and methods, and in this line Industry 4.0 is a result of the Fourth Industrial Revolution and equally a game-changer as were the introduction of mechanization and steam/power, of mass production and electricity, and of computer and automation. This fourth Industrial Revolution commonly refers to cyber-physical systems and manufacturing technologies that fuse the physical and digital worlds, including advancements in artificial intelligence (AI), the Internet of Things (IoT), 3D printing, autonomous vehicles and robotics. Driven by a digitisation of the manufacturing sector, Industry 4.0 is characterized by a rise in data volumes, computational power, and connectivity in wide-area networks; the emergence of analytics and business intelligence capabilities; new forms of human-machine interaction such as touch interfaces and augmented-reality systems; and improvements in transferring digital instructions to the physical world, such as 3-D printing and advanced robotics.⁴ Within these dimensions and of particular interest for robotic technologies are: (a) the concept of ‘interoperability’, referring to the ability of machines, devices, sensors and people to connect and communicate with each other via the Internet of Things (IoT) or the Internet of People (IoP); and (b) the available technical assistance of cyber-physical systems to support humans by conducting a range of physical tasks. In this context, robotic technologies potentially could play a vital role. The adoption of robotics technologies on sites could minimize hazards to workers, speed up processes, reduce waste, shorten construction timescales and reduce costs.⁵ The 2016 US National Robotics Roadmap stresses the importance that productivity gains benefit all of society, and not just the owners of robots.⁶ Robotics technologies have the potential to make work safer and more satisfying while reducing the dirty, dull and dangerous aspects of work in many occupations,⁷ including the construction sector. The use of robots is rapidly gathering pace as demonstrated by the interest of major industry players and increasing number of start-up companies. Furthermore, and significant in the context of this research, the 2017 McKinsey report suggests that investigation into several areas simultaneously holds potential to double productivity, including amongst others to: rethink design and engineering processes; improve onsite execution; infuse digital technology, new materials, and advanced automation; and reskill the workforce. This creates a compelling argument for the implementation of robots as co-workers, collaborators in and potentially also demonstrators of construction processes.

Robotics and cobots

Recent investigations into construction standards and methods with industrial robotic arms have resulted in novel methods for bricklaying, fluid deposition, timber sheet cutting and assembly, bespoke welding of steel elements, or tile cutting. There exists an extensively body of research by NCCR (Gramazio&Kohler Research, ETH Zurich), the Institute of Computational Design (ICD Stuttgart), Hyperbody Research Group (TU Delft), the Institute of Advanced Architecture Catalunya (IAAC, Barcelona), DRG Robotics Group (Harvard GSD) amongst others. However, whereas these investigations typically demonstrate highly complex geometries, material performance, or bespoke fabrication of elements, the integration towards standard industry processes, or a focus on expanding human labour through robot co-working and support is not at the forefront of these investigations. Moreover, several challenges potentially impact on expanding towards onsite and direct industry/trade implementation, including the overall size of robot arms (i.e. large scale, weight, power supply) that pose movability issues. Industrial robot setups robotic arms are generally confined to a caged area and operated from a distance to ensure the safety of the operator, as industrial robots can weigh upwards of one metric ton and can generate a deadly level of torque and momentum. Collaborative work or even operating large industrial robots remains high-risk, safety issues and closed work envelopes restrict direct engagement in predefined robot protocols. The dynamic process of interaction between human-robot co-work requires sensor-based work protocols and timed choreographies. Fast adapting to unexpected changes is difficult to process due to closed robot programming, and workflows developed through laboratory’s experimentations cannot be directly transferred to onsite applications. Robot programming is generally based on outcome and product orientation rather than movement, with robots lacking in capacity of learning from the surrounding. The bespoke nature of tasks often requires robots to be controlled by tailored pieces of software for performing actions. The construction industry requires case-by-case scenarios and thus high adaptability of the robotic code and, therefore, a deep understanding of the programming language by the technician/computational designer in charge.

In contrast to standard robotic practices, CoBots are categorised as robotic devices that manipulate objects in collaboration with a human operator,^8,9 or otherwise assist humans in activities through direct interactive communication and action. The principle of CoBots refers here both to robots with integrated sensor specifications such as ABBs Yumi or KUKAs iiwa, or other robots custom designed for human interaction, but more importantly to the larger concept of collaboration between humans and machines. In that sense, CoBots can provide a viable alternative and extension to standard processes of manufacturing and construction, which provides an initial framework for strategizing collaboration:

CoBots employ sensors and underlying intelligent systems that support awareness of and how they themselves are situated in their environs;

interactive, allowing for variation in their tasks depending on the actions of their human collaborators and their surroundings;

cognizant, agile and mobile;

designed to collaborate with human colleagues, and further are made to work in close-quarters with human beings, thus not raising health and safety concerns;

able to learn new processes from their human counterparts in an intuitive manner

minimize programming and software procedures available;

CoBots have rules that govern their behaviour in order to avoid collisions with humans and other obstacles in their span of reach.

Current research into collaborative robots investigates: autonomous mobile service robots that perform tasks in a multi-level office environment (Coral at Carnegie Melon University), directed towards ‘Indoor Mobile Robot Localization’; ‘Symbiotic Human-Robot Interaction’ and ‘User-centred Task Planning and Scheduling’. Other investigations focus on the situation-dependent active task contribution for collaborative processes involving creativity (Interactive Robotics Group (IRG) at Massachusetts Institute of Technology (MIT) to address creative solutions onsite and adhoc. To this extent, new industrial CoBots are manufactured for commercial enterprise: Baxter (Rethink Robotics), iiwa (KUKA) or YuMi (ABB). Baxter is designed to be taught through motion tracking whereby workers perform tasks by simply guiding the robotic arm, indicating clutch and release for object placement.¹⁰ Similarly based on integrated sensor features, the iiwa provides direct interfaces for motion learning, including speed, force and directionality through external motion input. YuMi offers an added second work arm to expand to a seven-axis system, thus offering one extra degree of freedom, including kinematic redundancy that allows YuMi to rotate its arm to avoid collision and twisting while keeping its end effector in the exact same position.

Research project: collaborative robotics for subject matter experts

Importantly, human-robot collaborations can offer alternatives and new pathways for construction industries, particularly for support of industrial growth and skilled labour. Consequently, this research focuses on the potential of collaborative robotics for subject matter experts, by investigating industry requirements and assessments of CoBot acceptance; investing in processes and sequences of work protocols for standard architecture robot; investigation of expanded tracking systems and framework, towards human-robot collaboration and machine learning.

Industry survey: complexity of tasks, safety, risk awareness

Initially, the research focused on a survey of industry and construction practice to understand challenges and concerns in adopting robotic technologies and CoBots for Australian construction standards. To this extent, discussions with industry and construction companies served to explore potential opportunities and challenges identify drivers of adoption and survey how technologies might be introduced in different organizations to different stakeholders, from investors, business owners/contractors, clients to end-users such as tradesperson, sub-contractors, apprentices. Two focus groups were created. As illustrated in Figure 1, a thematic data analysis investigated potential opportunities, challenges or barriers, drivers for adoption, introducing new ideas and education materials (i.e. formats and needs). A subset of secondary themes identifies recurring patterns mentioned by participants such as activities, job types, benefits of CoBots, or need for humans in tasks or other machines.

Figure 1.

Comparison of two focus groups and information derived (FG|1 and FG|2). Primary and secondary themes were identified responses for potentials, challenges, ideas introduction, education across discussion groups.

With varying participants across the two focus groups and different perspectives based on the specific nuances of their work environments, crosscutting themes were arising, which yielded two different quadrant analysis: one based on safety and complexity of tasks, and a second based on the approaches towards innovation and costs of adoption.

Figure 2 illustrates the discussion for evaluating characteristics of single or multi-task robots by intersecting safety with the complexity of task. Quadrant 1 (bottom left) highlights the benefit of single task CoBots for relatively safe, repetitive, simple tasks in controlled surroundings. Quadrant 2 (bottom-right) shows single task CoBots for dangerous but simple to execute work. Quadrant 3 (top left) illustrates benefits for multi-tasking CoBots working to a degree of complexity while providing safety. Quadrant 3 (top-right) features the true potential and extent of multi-tasking CoBots for dangerous and complex tasks.

Figure 2.

Differentiation of task groups in relation to complexity and safety.

Pilot studies: investigating task sequences to identify potential capture of computable actions

Based on the results derived from these discussions and survey, the consecutive pilot studies focused on carpentry in small-to-medium size enterprises in the Australian construction industry, seeking to understand and identify opportunities in the current workflows of carpenters for the role of collaborative robots. We found that several repetitive functions relevant for CoBot prototype development include tasks that are also relevant to other trades, such as cutting, sawing, passing, standard cuts, lifting, moving, holding/securing.

The research developed a novel methodology for the capture and analysis of the body movements of carpenters, resulting in a suite of visual resources to aid us in thinking through where, what, and how a collaborative robot could participate in the carpentry task.¹¹ Based on interaction analysis, this method focuses on the physical actions, movements, material and tool use, taking place in space and time, and a specific orientation to collaborative physical actions. The research contributes here through a novel set of visual resources generated from the interaction analysis, enabling identification and evaluation of potential insertion points for CoBots in the already established workflows of construction workers (Figure 3).

Figure 3.

Methodology Phase 2. From observation of task performance to identified sequences in workflow.

We investigated a pilot study in a workshop situation with in-house carpenters as foundation for a real-time, onsite study at a construction site (Figure 4). This allowed us to empirically observe, capture and analyse the work practices of carpenters building a timber frame structure (such as for a residential house).

Figure 4.

Simulation of work sequence in pilot study (DMaF, University of Sydney, in-house carpenters).

Importantly, we developed and tested the observe-and-record step of the proposed methodology. In presenting and explaining the results of the analysis, the structured sequence of actions and single visual resource (Figures 5 –7, Table 1) enables us to understand, access and close investigation to the complex, fluid reality of an active building site.

Figure 5.

Sequenced action across ten-stage construction for building the timber structure of a house from prefabricated frames on a live building site.

Figure 6.

Detailed framing actions highlighting how the carpenters use their hands and feet to help prop and stabilize the frames.

Figure 7.

Person-to-Material, Person-to-Tool and Person-to-Tool-&-Material.

Table 1.

Evaluation of likelihood of CoBots helping a human carpenter in a range of collaborative actions related to framing. Gradient code indicates functionality adaption (white: not possible/adequate, midgrey: partial application, dark grey: good functionality).

Collaborative action	Type of CoBot (arms/mobility)
Collaborative action	Single- arm CoBot, static	Dual-arm CoBot, static	Single-arm CoBot, mobile	Dual-arm CoBot, mobile
1. Sorting frame (lifting and moving to pile)	No use, due to lack of mobility.	No use, due to lack of mobility.	No use, as two arms are required to hold and stabilize the frame.	Possible, depending on relative size of cobot to frame.
2. Carrying frames (2 carpenters)	No use, due to lack of mobility.	No use, due to lack of mobility.	Can hold one end/side of timber/structure with a grip end effector, and locomote.	Can hold one end/side of timber/structure with two grip end effectors, and locomote.
3. Sliding frames	No use, due to lack of mobility.	No use, due to lack of mobility.	Can hold/push one end/side of timber/structure with a grip end effector	Can hold/push one end/side of timber/structure with two grip end effectors
4. Rotating frames	No use, due to lack of mobility.	No use, due to lack of mobility.	Can hold/lever one end/side of timber/structure with a grip end effector. May be limited by relative size.	Can hold/lever one end/side of timber/structure with two grip end effectors. May be limited by relative size.
5. Foot tapping to location	Can a robot arm replace the kicking action of a human foot?	Ditto	Ditto	Ditto
6. Hammer tapping into position	No use, due to lack of mobility and one arm.	Limited. Only if the hammer tapping point is where cobot is stationed. Assuming one arm can stabilise the frame,whilst the other arm hammer taps.	Cobot can move to where hammer tapping needs to be done, but has only one arm.	CoBot can move to where hammer tapping needs to be done. One arm can stabilize the frame, whilst the other arm hammer taps.
8. Nailing frame to slab	No use, due to lack of mobility and one arm.	Limited. Only if the nailing point is where cobot is stationed. Assuming one arm can stabilize the frame, while the other arm uses nail gun.	CoBot can move to where nailing needs to be done, but has only one arm.	CoBot can move to where tapping nailing needs to be done. One arm can stabilize the frame, whilst the other arm uses the nail gun.

In order to identify opportunities for CoBot insertion into timber framing workflow, as illustrated in Table 1, the research juxtaposes sets of collaborative actions currently performed by human construction workers carrying out a series of timber framing tasks against a CoBot with two variables of a number of articulated arms (one/two) and mobility (stationary/mobile). The degree of locomotion mobility and the number of arms has implications for the type of work the CoBot can do, and where in the workflow the CoBot can be inserted. In principle, the human action repertoire observed on the building site is amenable to performance by a CoBot. Consequently, methods of performing the action are critical, both in terms of shared process or sequenced task. For example, the collaborative task of carrying a structure implies the negotiation of physical forces between the two workers (human and robot) in an ergonomic manner. In contrast, the task of bracing a structure to enable the other worker to secure the structure with a nail gun is a different type of physical collaboration involving two discrete actions.

In summary, the analysis revealed how the team of carpenters employed a range of agreed-upon processes and routines that were intimately linked with the materials and tools they used. Verbal talk was important for checking and confirming that tasks were performed correctly and safely, as well as for convivial social interaction to enrich work. Given the amount of verbal and non-verbal communication that takes place on a building site, any CoBot to be integrated into construction workflows will need a degree of fluency in understanding human speech and gesture. This shows potential for the adoption of CoBots into construction work, which will inevitably change current workflows and traditional practices of building. Introducing a CoBot to established working processes, to onsite conditions and adding ad-hoc improvisations would require a period of mutual learning and adaptation by both the human carpenters and CoBots.

We also found that in a relatively simple task such as a building a timber frame structure for a residential house a very large number of movements exist that can be replaced by a CoBot (Figures 8 and 9, exert). However, programming each of these movements would be a near impossible task. Consequently, we argue that CoBots should adopt new movements via machine learning so that industrial robots can be trained to learn how to perform a task, instead of simply following a pre-programmed script. As next steps, the research investigated the potential for machine learning.

Figure 8.

CoBot- Carpenter Collaborative workspace: CoBot reach in static position (top left), CoBot supporting frame whilst carpenter levels (top right), CoBot assisting carpenter sliding frame (bottom left), CoBot assisting carpenter lift frame (bottom right).

Figure 9.

CoBot nailing frame to slab with custom end effector whilst carpenter supports frame.

Capturing motions and machine learning

Further strategies needed to be considered to address a complexity of purposeful movements in a seamless sequence so that a human-robot team is jointly able to deliver an action catalogue. Strategies can be based on coupling a preprogramed movement trajectory with path correction via sensor based input,¹² or introducing a semantic robot language.¹³ Other options are available through sensor tracking a human motion and transferring data to an adaptive (or open/interactive) robotic protocol,¹⁴ or haptic learning to control robot movements.¹⁵ However, machine learning process for training paths/movements is a viable alternative since knowledge in this domain has advanced to industrial robots already being trained to learn how to perform a task, such as current automated, servicing humanoid applications,¹⁶ or collaborative repositioning in robotic-human assembly.¹⁷

Thus, as direct extension of the pilot studies and structured investigation into action protocols, the research explored methods for data capturing processes, including movement tracking vs motion programming or machine learning. This included the establishment of a methodology for workflow capture and analysis of carpentry tasks towards human-robot collaboration in the case study investigations; and a framework that outlines two pathways of training robots through machine learning – supervised and reinforcement learning (building on previous research including Brynolfsson et al.,¹⁸ Shalev-Shwartz and Ben-David,¹⁹ Michalski et al.²⁰).

Significant is here a concept of machine learning that addresses the objective of training a CoBot arm to complete movements. However, large quantities of data are required to train a machine learning model. This is particularly complex for human movement coupled with motion capture as a source of training data. Motion capturing is a process of recording the movements of objects and people (body and limbs), whereas performance capture refers to the capturing of subtle expressions such as fingers or face. For motion, capturing movements of one or more actors are sampled many times per second using often-passive markers made out of retroreflective material positioned on the body, a technique we used in the following stages of research (see Figure 10). Other techniques include active markers where an active optical system triangulates positions by illuminating one LED at a time. Alternatively, multiple LEDs with identification software for relative positions akin to celestial navigation or Time Modulated Active Markers can be adopted as systems in which active markers can further be refined by strobing one marker at a time. Moreover, the motion capturing of arm, hand, and finger movements while completing tasks can be used to provide a data sets for machine learning so that consequently CoBots can be trained to collaborate in a movement or task.

Figure 10.

Scenes from motion capturing workshop to gain data for supervised machine learning.

An approach of motion capture plays a significant role in developing robot movement, particularly where a robots learns from a human demonstrator.²¹ This has been successfully demonstrated with related research projects using Kinect sensing for tracking brick laying.²² There exists a range of techniques for motion capture, including relatively limited technologies such as leap motion and Kinect sensors, or full body suits with incorporated data gloves that produce highly accurate results. The resulting raw data can be exported and processed for supervised machine learning, and to complete a desired movement or task within a range of movements. We investigated capture techniques for motion capturing of arm/hand/finger movements towards developing complex robotic systems. Once a pre-determined motion has been captured, this provides a training data set for a neural network, which enables a machine to learn the arm/hand/finger movements, once machine learning has been established all movements can be replicated via a CoBot arm. This can be of significance particularly as a technique for robotic systems that carry out a human task deemed demanding or strenuous.²³ Furthermore, differentiations between supervised and unsupervised machine become significant since supervised learning is time-consuming and resource intensive, where certain movements require multiple action learning and also multiple participant tracking to provide a suitable data set that can be used to train. Unsupervised learning, such as reinforcement learning does not require an input to process output but allow for machine learning algorithms to use trial and error to develop a tooling path suitable to complete the task it was assigned for which skips the step of collecting motion capture data for a training set. Moreover, reinforcement learning is comparative to learning through trial and error, and particularly equipped for industrial robot arms with inbuilt sensor-based facilities. This opens a unique space for tasking robots with achieving a given goal where no training data is required. Instead, a reinforcement learning approach adopted for robotic processes can potentially uncover new approaches towards manufacturing and fabrication, where data collected can be used for objective measures for delivering similarity.

In a first iteration, the research team developed a machine learning model that simulated ‘learned’ movements (Figure 11), using Google’s Tensorflow library as reinforcement learning algorithm to task a Kuka iiwa robotic arm to learn how to search and find an object.

Figure 11.

Screenshot of a simulation environment for the Kuka iiwa robot arm that understands the mechanisms and data flow to control the arm, and weave a reinforcement learning algorithm to interact with the simulation. The ‘to-be-found’ is indicated as green bar underneath the robot arm.

Based on the research in Phase 2 towards capturing computable actions, we observed that several interaction sequences between humans start with one human reaching for a tool. Consequently we used the movement ‘autonomous searching for tool/identifying tool/reaching for tool’ as the first step for a reinforcement learning trajectory. In the set-up of the machine-learning program, the robot arm is equipped with a virtual camera (see three small images in Figure 11 on the left top). Further, the machine-learning program knows what degree of freedom the robot arm has and that the ‘to-be-found’ object (Figure 11) is within the physical reach of the Kuka iwaa arm. When executing the machine-learning program, the robot arm starts searching, is rewarded similar to any other reinforcement learning models, and improves the finding strategy over time. With completing this first movement, the research went on to simulate the movement ‘autonomous searching for tool/identifying tool/reaching for tool’ in a physical set up.

Discussion

Human-robot collaborations for the construction industries, as has been argued, can potentially offer alternatives and new pathways and help reconsider the approach to skilled labour. To explore this, the research adopted multiple modes of investigation, ranging from industry and trade assessments of CoBot to investing processes and sequences of work protocols and exploring motion capture and tracking systems for a collaborative framework between human and robot co-workers.

As a contribution to the construction industry and specific to an Australian context, the research has in collaboration with industry and stakeholders identified and explored the potential opportunities for robotics for a bandwidth of trades, including painters, plasterers, bricklayers, carpenters and stonemasons. It has further outlined these fields of enquiry for professions that hold potential for investigation and adaptation. Furthermore, a methodology has been established for the empirical capture and analysis of human physical activity on a construction site, as simulated in a lab environment and verified onsite. Grounded in interaction analysis and with a specific focus on the physical actions, movements, material and tool use, taking place in space and time, this methodology holds potential for specific focus to collaborative physical actions for a range of physical actions identifiable for construction methods. Explicit visual resources were generated from the interaction analysis, enabling identification and evaluation of potential insertion points for CoBots in the already established workflows of construction workers. The research illustrated the methodology in action through the case study of the framing process of carpenters on a building construction site. Furthermore, this methodology holds potential to be applied across differentiated actions that could each be implemented through an industrial robot as a six-axis multi-tool and multi-process co-worker for human construction.

However, these are initial steps and some limitations should be discussed here. A direct translation from simulation in a virtual environment to physical movement of the robot arm either as recorded or real-time is undoubtedly challenging, due to interoperability issues between software used for machine learning (Python) and the operating system of the Kuka iiwa robot (Java). Moreover, the programming language Java is only supported by a limited number of existing open source machine learning libraries, and so connections would need to be established between machine learning environments and the operating systems for a robot to transfer pre-learned movements onto the operating system. In addition, there exist no detailed comparison studies for implementing human-to-robot motion between standard industrial arms and recent sensor-based robots such as Kuka’s Iiwa or ABB’s Yumi. Yet with ongoing advancements in machine learning and software developments in computer science to connect machine learning tools to CoBots, robot programming, sensor feedback and machine learning environment can be resolved in the near future.

Conclusion and future work

This research has discussed foundational data scoping and pilot studies for implementation of industrial robotic arms as a labour and collaborative resource for construction processes to address growth and economic demands in a context of Industry 4.0. A methodology and framework for human-robot collaboration in the context of masonry have been presented, which can be further deployed for an investigation into movement protocols for other trades towards CoBots.

Importantly, programming (i.e. robot programming, machine learning, software interoperability) is also linked to robot and hardware. Two further considerations should be noted here.

Firstly, this paper has discussed industrially available and standard CoBots that to date have significant size, reach and payload limitations, yet can be implemented to body-related work modules as a customary on building sites (i.e. bricks, tiles, timber elements). Moreover, while this research investigates human workflows, movements and gestures to support repetitive actions that lead to health issues in the course of a human work-life with 30 plus years, more research is required into CoBot systems that are safe and could extend current heavy-duty restrictions.

Secondly, as the pilot studies showed, carpenters use a series of tools when constructing a timber frame structure for a residential house (see Table 2: Node Table noting repetitive actions and use of tools over 30 min). We argue that it is, therefore, challenging to change the end effector of a robot each time a different tool is required, moreover, it would add additional costs to the construction company to purchase a large number of end effectors that are able to perform the work required. Thus, one can further conclude and argue for the need of an anthropomorphic end effector attached to the CoBot, capable of picking up and operating any tool present on the construction site. Parts of this research identified that in a context of architecture no currently existing anthropomorphic end effector makes use of the gripping strength of the palm, but only uses fingers.²⁴ Using the palm to hold and to apply strength to an object such as hammer, drill, brush, etc. is crucial, and with current end effectors only making use of the fingers it will remain a challenge to deploy CoBots on construction sites.

Table 2.

Nodes table noting repetitive actions and use of tools over 30 min.

Physical movements	Files	References	Use of tools	Files	References
Bracing	1	NSW Planning¹	Circular saw	3	US National Robot Road Map⁶
Carrying (1× Carpenter)	1	NSW Planning¹	Concrete nail gun	3	Takayama et al.⁷
Carrying (2 × Carpenter)	3	16	Hammer	3	Reinhardt¹¹
Changing nail gun cartridge	1	NSW Planning¹	Measuring tape	1	NSW Planning¹
Cutting	3	Bogue⁵	Nail gun (1× Carpenter)	2	Bogue⁵
Hammering into position	3	Bogue⁵
Lifting / Sorting	2	Bogue⁵
Loosening straps	3	Bogue⁵
Measuring	1	NSW Planning¹
Passing tools	1	NSW Planning¹
Position with feet	3	Rossi and Nicholas¹⁵
Short vertical lift	1	NSW Planning¹
Sliding frame	2	McKinsey²
Supporting in position	2	Rosenberg et al.³
Vertical rotation	3	Braumann and Brell-Cokcan¹⁴

Future investigations for human-robot collaborations could be to: further investigate human work movements; develop anthropomorphic end effectors; and the proposed methodology extended to encompass the transition from the current output of the identification of actions suitable for CoBots; to laboratory experimentation; and testing of training the physical actions of CoBots through machine learning and end effector development, for new robotic applications in a context of Industry 4.0.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by a 2018-2019 community of practice Landcom Grant (Sydney, Australia), titled “CoBuilt 4.0-Investigating the Potential of Collaborative Robotics for Subject Matter Experts”.

ORCID iDs

Matthias Hank Haeusler

Alessandra Fabbri

References

NSW Planning. Plan for growing Sydney. Australia: NSW Government, https://www.planning.nsw.gov.au/Plans-for-your-area/A-Metropolis-of-Three-Cities/A-Metropolis-of-Three-Cities (2014, accessed 14 December 2018).

McKinsey. Digital Australia: seizing the opportunity from the fourth industrial revolution. McKinsey, https://www.mckinsey.com/~/media/McKinsey/Featured%20Insights/Asia%20Pacific/Digital%20Australia%20Seizing%20the%20opportunity%20from%20the%20Fourth%20Industrial%20Revolution/Digital-Australia-Seizing-the-opportunity-from-the-fourth-industrial-revolution-vf.ashx (2017, accessed 14 February 2020).

Rosenberg

Haeusler

Araullo

, et al. Smart architecture-bots and industry 4.0 principles for architecture. In: 33rd eCAADe. Vienna: eCAADe and Faculty of Architecture and Urban Planning, TU Wien, 2015, pp. 251–259.

Østergaard

. White paper – the role of cobots in industry 4.0. Universal Robots, 2015.

Bogue

. Rehabilitation robots. Ind Rob 2018; 45(3): 301–306.

US National Robot Road Map. A roadmap for US robotics from internet to robotics 2016 edition, https://cra.org/ccc/wp-content/uploads/sites/2/2016/11/roadmap3-final-rs-1.pdf (2016, accessed 12 March 2020).

Takayama

Nass

. Beyond dirty, dangerous and dull: what everyday people think robots should do. In: Proceedings of the 3rd ACM/IEEE international conference on human robot interaction, HRI 2008, Amsterdam, The Netherlands, 12–15 March 2008.

Colgate

JEC

. Cobots: robots for collaboration with human operators. In: Proceedings of the international mechanical engineering congress and exhibition, Atlanta, GA, 1996, pp. 433–439.

Peshkin

Colgate E.

Cobots

. Ind Rob 1999; 26(5): 335–341.

10.

Powell

. Future of robotics: manufacturing gets a makeover with Baxter. Forbes, https://www.forbes.com/sites/jenniferhicks/2012/09/18/part-1-future-ofrobotics-manufacturing-gets-a-makeover-with-baxter/#2e6de5774c8f; (2017, accessed 21 January 2019).

11.

Reinhardt

Haeusler

Loke

, et al. Cobuilt: towards a novel methodology for workflow capture and analysis of carpentry tasks for human-robot collaboration. In: Proceedings of the 37th eCAADe and 23rd SIGraDi Conference - Volume 2, University of Porto, Porto, Portugal, 11–13 September 2019, pp. 207–216.

12.

Dubor

Camprodom

Bello Diaz

, et al. Sensors and workflow evolutions: developing a framework for instant, robotic toolpath revision. In: Reinhardt

Burry

Saunders

(eds) Robotic fabrication in architecture, art and design 2016. Switzerland: Springer International Publishing, 2016, pp. 411–425.

13.

Andraos

. DMR: a semantic robotic control language. In: Martens

Wurzer

Grasl

, et al. (eds) Real time – Proceedings of the 33rd eCAADe conference – Volume 2, Vienna University of Technology, Vienna, Austria, 16–18 September 2015, pp. 261–268.

14.

Braumann

Brell-Cokcan

. Adaptive robot control – new parametric workflows Directly from design to KUKA robots. In: Martens

Wurzer

Grasl

, et al. (eds) Real time – proceedings of the 33rd eCAADe conference – Volume 2, Vienna University of Technology, Vienna, Austria, 16–18 September 2015, pp. 243–250

15.

Rossi

Nicholas

. Haptic learning – towards neural-network-based adaptive Cobot path-planning for unstructured spaces. In: Sousa

Xavier

Castro Henriques

(eds), Architecture in the age of the 4th industrial revolution. In: Proceedings of the 37th eCAADe and 23rd SIGraDi conference – Volume 2, University of Porto, Porto, Portugal, 11–13 September 2019, pp. 201–210.

16.

Schwartz

. Use of a low-cost humanoid for tiling as a study in on-site fabrication: techniques and methods. In: ACADIA // 2016: POSTHUMAN FRONTIERS: Data, Designers, and Cognitive Machines [Proceedings of the 36th Annual Conference of the Association for Computer Aided Design in Architecture (ACADIA) ISBN 978-0-692-77095-5], Ann Arbor, 27–29 October 2016, pp. 214–223.

17.

Wibranek

Belousov

Sadybakasov

, et al. Interactive structure – robotic repositioning of vertical elements in man-machine collaborative assembly through vision-based tactile sensing. In: Sousa

Xavier

Castro Henriques

(eds) Architecture in the age of the 4th industrial revolution. Proceedings of the 37th eCAADe and 23rd SIGraDi conference – Volume 2, 11–13 September 2019, pp. 705–713, University of Porto, Porto, Portugal.

18.

Brynjolfsson

McAfee

. The second machine age: work, progress, and prosperity in a time of brilliant technologies. New York: W. W. Norton & Company, 2014.

19.

Shalev-Shwartz

Ben-David

. Understanding machine learning: from theory to algorithms. Cambridge: Cambridge University Press, 2014.

20.

Michalski

Carbonell

Mitchell

. Machine learning: an artificial intelligence approach. New York Vienna: Springer, 1983.

21.

Stanton

Bogdanovych

Ratanasena

. Teleoperation of a humanoid robot using full-body motion capture, example movements, and machine learning. In: Proceedings of Australasian Conference on Robotics and Automation (ACRA), 3–5 December 2012, pp. 260-269, Victoria University Of Wellington, New Zealand.

22.

Das

Worre Foged

Jensen

, et al. Collaborative robotic masonry and early stage fatigue prediction. In: Sousa

Xavier

Castro Henriques

(eds) Architecture in the age of the 4th industrial revolution. In: Proceedings of the 37th eCAADe and 23rd SIGraDi conference – Volume 3, 11–13 September 2019, pp. 171–178, University of Porto, Porto, Portugal.

23.

Firth

Dunn

Haeusler

. Review of motion capture techniques and role in developing complex robotic systems. In: Real / Material / Etheral - The 2nd annual design research conference, 3–4 October 2019, Monash University Melbourne.

24.

Firth

Dunn

King

, et al. Development of an anthropomorphic end effector for collaborative use on construction sites. In: Proceedings of the 25th CAADRIA conference, 5–8 August 2020, Chulalongkorn University Bangkok, Bangkok, Thailand.