Collaborative timber joint assembly: Augmented reality for multi-level human-robot interaction

Background

The timber industry is undergoing a profound digital transformation, redefining traditional practices and reshaping the way the Architecture, Engineering, and Construction (AEC) sector designs and builds with wood. This metamorphosis encompasses a broad spectrum of technological advancements including automated timber grading, timber prefabrication with Computer Numerical Control (CNC) machines, robotic integration to perform complex assembly tasks, and computational modeling and simulation-based digital tools. All these advancements share the overarching goal of enhancing workflow efficiency, precision, and collaboration throughout the construction lifecycle. Furthermore, as the demand for sustainable building practices increases, there is a rising need for advanced digital frameworks to fully exploit the potential of timber construction.

As technology permeates the timber construction landscape, questions arise regarding the role of human workers in this evolving ecosystem. Automation has the potential to transform construction processes, leading to increased efficiency and reduced manual labor requirements. However, this shift introduces both opportunities and challenges as traditional timber construction skills intersect with cutting-edge technologies. Major concerns include potential job losses, the decline of small businesses, and the replacement of skilled craft with less engaging machine supervision.^1–3 Consequently, an essential aspect of this digitization journey is to carefully consider its impact on the workforce. This means ensuring that technological progress enhances and empowers timber construction workers, rather than displacing their expertise. This involves not only preserving traditional skills but also providing workers with opportunities to adapt and thrive in a technology-driven environment.

At the beginning of the 20th century, the Second Industrial Revolution conceptualized factory work as a linear chain of tasks to be performed by isolated agents (humans or machines). ⁴ This approach led to the establishment of assembly lines, where humans and machines were positioned sequentially according to the actions to be executed (i.e., screwing, inserting, or welding parts). While efficiency gains were undeniable, this separation and specialization of labor has led to more repetition, ultimately resulting in significant work alienation. ⁵ With the advent of information technology and the respective progress of Industry 3.0, 4.0, and 5.0, the 21st century holds the potential to challenge this conception. ⁶ In particular, by enabling human and robot agents to be more versatile and by offering a working environment in which they collaboratively complement each other.

AR, in particular, holds the promise of seamlessly blending the tactile craft of woodworkers with the power of robotic systems. It empowers workers by overlaying digital instructions, annotations, and simulations onto the physical environment. Using AR devices, construction workers are equipped with a dynamic digital toolkit that enhances their decision-making, reduces errors, and streamlines complex assembly processes by avoiding the need for 2D plans.^7,8 By enhancing direct communication under the umbrella of a digital realm and boosting collaboration at scale, AR can serve as an ecological lever between robotic and human agents in the ecosystem of automated timber construction. ⁹

State of the art

Recent research on robotic timber construction has demonstrated the possibility of using industrial robotic arms to build timber structures connected by wood joints. The joinery in such timber systems is established solely through the interlocking of the structural elements using 5-axis CNC routers or robotic arms to create integral connections. Despite the recent breakthroughs in the design and construction of mass timber systems with fastener-free integral joints, the research indicates that there are recurring challenges associated with tolerances during the assembly process and robotic insertion.^10–13 Wood is a fibrous natural material that presents a potential for shrinkage and expansion. It thus requires more local adaptations than concrete or steel during the fabrication and assembly phases. This makes wood, and even engineered timber products, more challenging to process in highly automated workflows. While robots can carry higher payloads than human workers and assemble pieces with unmatched accuracy, humans are more capable when it comes to adaptation and decision-making. For instance, knots, fiber orientations, and hygrometric variations are factors that influence friction coefficients and assembly tolerances, ultimately preventing the robot from performing the insertion. ¹¹ In contrast, in human-based assembly, the position of the pieces and the direction of the insertion force can be adjusted intuitively, and on the spot, to ensure tightly fitting joints.

Considering the respective strengths of robotic and human agents, recent research efforts have explored Human-Robot Collaboration (HRC) and its beneficial contribution to timber construction. One of the early large-scale flagship structures was the design and construction of the DFAB house where each of the timber beams was accurately positioned by gantry-mounted robot arms and connected to the rest of the assembly by humans using hand-held screwdrivers. ¹⁴ Recently, the research focus has shifted. Instead of delegating tasks that robots cannot perform to human agents, researchers are exploring the benefits of having human and robot agents work together. Xi Han and Parascho ¹⁵ investigated the integration of improvisation principles in robotic assemblies by realizing a structure composed of 500 bamboo rods using two robots and multiple human agents. Skevaki et al. ¹⁶ also introduced a setup for human-robot interactive design and assembly. Timber elements were positioned by robots while 3D-printed connections were adjusted by humans to mitigate construction tolerance. Mitterberger et al. ¹⁷ developed a framework where two mobile robots and two human agents can place timber rods that are then manually tied with rope knots.

Researchers have developed approaches to combining robotic construction with AR for designing and manufacturing timber structures. Hughes et al. ¹⁸ used an AR headset to fabricate and assemble a reciprocal timber frame, specifically a Zollinger structure. They demonstrated the benefits of integrating AR in digital timber construction workflows for logistical support and for checking robot trajectories to avoid collisions. Besides, Morse et al. ¹⁹ used an AR overlay to assemble a robotically milled demonstrator. Kyaw et al. ²⁰ combined gesture recognition and mixed reality to assemble robotically kerfed timber round woods. In the three latter studies, components were primarily fabricated by means of robots before being manually assembled with AR guidance. Kyjanek et al. ²¹ proposed an HRC framework where AR is used to manipulate robotic path planning in real-time. In this installation, pieces were positioned by the robot and screwed by the human agent.

Industrial applications of AR in design and construction are also gaining traction, with companies like Fologram ²² developing AR tools to bridge the gap between digital models and physical fabrication. Additionally, design firms like Foster and Partners ²³ are exploring on-site AR to enhance precision and efficiency in construction processes, further integrating these technologies into mainstream practice.

Beyond the timber construction sector, other studies have shown how AR can foster HRC, especially in improving safety protocols and user experience. Hietanen et al. ²⁴ demonstrated that dynamically displaying the working zones of human and robot agents can enhance the performance and safety of assembly tasks. Furthermore, Carierro et al. ²⁵ showed that AR interfaces can facilitate bi-directional communication between human and robot agents. In addition to visualizing the workspace, their application made it easy for the human worker to keep track of the robot’s state and manipulate the end-effector position. Moreover, Tsamis et al. ²⁶ demonstrated that AR can display the robot’s intended motion to human agents, further contributing to safer and smoother interactions.

In broader research on Human-Robot Interactions (HRI), three levels of interactions between humans and robot agents are defined and investigated^7,27,28:

• Coexistence: Robots and humans operate independently. Robots are designed to detect human presence and cease movement if human agents are within proximity.

The usage of the terms collaboration, cooperation, and interaction is often inconsistent within the architecture and construction field, especially when differentiating between collaboration and cooperation. Moreover, recent studies have primarily focused on only one type of interaction between human and robot agents,^14,16,21 i.e., robot agents have been responsible for carrying components, and human agents have been responsible for attaching them. In such configurations, the role of each agent is clearly defined and separated. It is within this context that further research is required to explore a more refined task distribution, which would enable a wider range of interactions between robotic and human agents and potentially transform assembly lines into more engaging work environments.

Aims and objectives

This research investigates assembly scenarios involving cooperation and collaboration between human and robot agents. Following the literature review, cooperation is defined in this study as the agents acting sequentially in the workspace, with each agent inserting a timber component independently, without the assistance of the other agent. In contrast, collaboration is defined as both agents working together simultaneously, requiring concurrent contact with the timber components. This study aims to establish the conditions under which cooperation or collaboration between agents is more favorable and subsequently develop an algorithm to automate task allocation for the insertion of timber elements connected by wood joints.

Furthermore, the research examines the potential benefits of AR in HRI. The latter aligns with the state-of-the-art research efforts that highlighted the importance of developing intuitive user interfaces to improve interactions between human and robot agents.^16,21,24 As such, this paper aims to automate the generation of holographic instructions for timber structures and study how AR can support multiple levels of interactions by augmenting human agents during manual and robotic assemblies.

This research further contributes to the development of a fully automated workflow from design to construction for timber structures with interlocked assemblies. The work builds on previously developed tools including Manis ²⁹ – an open-source computational framework integrating fabrication, assembly, and structural engineering constraints for the design of timber structures connected by wood joints. The research thus broadens the functionality of Manis with a new task allocation algorithm and develops a streamlined data flow from the design interface to the AR display.

The main contributions of the paper are organized into four sections: Section 2 introduces the computational framework. The developed algorithms are presented independently of any target geometry. Section 3 delves into the specific case study of a timber frame structure featuring three kinds of timber joints. The prototype is assembled through HRC and different types of interactions between human and robot agents are highlighted. Section 4 discusses observations about HRI, AR, and assembly performance. Finally, Section 5 concludes with a report on research achievements and future outlooks.

Developing a streamlined design-to-construction process

The methodology involves automating communication processes between various off-the-shelf tools. This development relies on the Rhinoceros/Grasshopper ecosystem and a number of subsequent community-developed plugins to create a streamlined workflow from computational design to AR-guided construction (Figure 1). The open-source Manis plugin ²⁹ is utilized to automatically generate timber joints, export fabrication files for CNC machining, and plot trajectories for robotic assembly. The solver accepts beams, columns, and panels modeled as B-Reps (boundary representation) as inputs. Each element must have at least two parallel faces, and all faces must be planar. With Manis, users can manipulate the joint geometry using a set of customizable parameters in Grasshopper. Parameters such as the number, width, length, and spacing of Through Tenon joints can be altered. Furthermore, the framework ensures the compatibility of the connections’ geometry with the user-defined assembly sequence by computing the insertion vector of each element prior to joint generation.OPEN IN VIEWER

Determining the suitability of a piece for manual assembly

Using the data structure generated by Manis, a custom Python algorithm is developed to check if timber pieces can be assembled manually and/or robotically. For the manual assembly, the verification relies on two criteria:

• The weight of each assembly component

For both parameters, the threshold varies in function of the strength and body height of the operator. The British Regulator for Workplace Health and Safety (HSE) provides guidelines on the weight that can be lifted by a worker depending on their posture. ³⁰ The prescriptions also vary according to the worker’s gender. For example, with arms outstretched, it is not advisable to carry more than 5 kg for men, 3 kg for women, while with arms by side, it is not advisable to carry more than 25 kg for men, 20 kg for women. Based on those considerations, the HSE guidelines can be generalized to establish a linear relationship between the weight to be lifted and the working height (Figure 2). The relationship in terms of relative height is expressed in percentage (%) rather than absolute height (m). Furthermore, workers are allowed to specify their maximum weight. Accordingly, the gender factor is eliminated to make the model inclusive and gender-free. Ultimately, the suitability of a part for manual assembly is verified if both of the following expressions are true:

h > 0.625 h t o t w max w − 0.1875

(1)

h < 0.625 h t o t w max w − h t o t

(2)

OPEN IN VIEWER

Figure 2.

Custom model to determine the suitability for manual assembly based on the worker’s height and the payload’s weight. The model extrapolates manual handling guidance from Britain’s national regulator for workplace health and safety.

Determining the suitability of a piece for robotic assembly

The algorithm follows the decision tree shown in Figure 3 to determine if a piece is suitable for robotic assembly. Similar to checking the suitability for manual assembly, the algorithm first verifies that the piece is not too heavy for the robot and that the target position is within the robotic arm’s reach. While some industrial robotic arms can manipulate up to one ton of payload, most standard-grade robots do not exceed a hundred kilograms. Furthermore, the end-effector is usually the limiting factor. Standard vacuum grippers used in the industry to manipulate timber panels can typically lift between 10 and 50 kg. Robot reachability also varies according to the specifications of the setup. The algorithm computes an estimated height threshold for the assembly based on the distance between the target location and the base of the robotic arm.OPEN IN VIEWER

In addition to the weight and height of the piece, the developed algorithm includes considerations about the difficulty of assembly. As highlighted through prior experiments,^10,11 it can be difficult for a robot to insert multiple timber joints at the same time, especially if the piece needs to be inserted into two or more pieces. Apolinarska et al. ¹³ demonstrated that combining force sensors and machine learning algorithms can help to overcome the latter issue. However, existing solutions focus on specific types of connections, making it challenging to generalize the method for all kinds of timber joints and configurations.

Moreover, assembly tolerances can potentially lead to significant discrepancies between virtual and physical models, possibly requiring manual adjustments for proper joint alignment. By analyzing the structure’s connectivity from Manis, it becomes possible to identify when a part must be assembled into more than one other part. In such cases, even if a robot can perform the assembly, human intervention may be necessary. To ensure operator safety, the robot will pause just above the assembly and wait for confirmation that all joints are correctly aligned. This approach allows humans to make necessary adjustments, leveraging their observational skills and decision-making abilities based on the situation.

Some insertion motions can be difficult or impossible to perform with a robot while human-based assembly remains feasible. This is typically the case if an element needs to pass through another element. Human agents can change their grip, relying on their two arms to move the element through the hole, whereas robot agents will not have the same flexibility. In addition, the robotic workflow developed in Manis can only compute trajectories from standard translation motion. Consequently, more complex motions are discarded by the algorithm and require to be manually assembled. Otherwise, a potential robot trajectory is computed using the open-source framework COMPAS RRC. ³¹ The validity of the trajectory is tested using a custom Python algorithm, which checks for potential collisions between the robot, the payload, the structure, and the environment through mesh intersection operations.

The stability of the piece after insertion is assessed by computing the associated overturning moments. The selected method has the advantage of remaining independent of the material and connection type. Simulation-based approaches, whether using the continuum-based Finite Element Method or reduced-order macro models,^32,33 can provide relevant structural design feedback. However, regardless of the method of analysis, it is crucial to assess the rigidity of the connections beforehand. In this context, the state of equilibrium of a piece is evaluated in the current research by examining the sign of the sum of the overturning moments, even without knowing the material density.

An object is considered stable if it returns to its original position after a slight displacement. Hence, for each piece, the impact of slight rotations in the XZ and YZ planes on the equilibrium state is initially examined. A specific threshold angle θ is set to indicate the stability status, and the overturning moment around a pivot point P, as illustrated in Figure 4, is computed. The pivot point’s location is determined by the contact areas from the neighboring supporting pieces.OPEN IN VIEWER

In the XZ plane, the points with the lowest and highest X coordinates are used to validate anticlockwise and clockwise rotations, respectively. Similarly, in the YZ plane, the points with the lowest and highest Y coordinates as pivot points are employed to verify the piece’s stability. In light of these computations, four checks are performed to confirm the stability of a given piece.

If a piece is deemed unstable, the algorithm evaluates whether it will stabilize after the next piece is inserted. If it does stabilize, human-robot collaboration is required. The robot must hold the first piece while the second one is manually inserted. If the piece remains unstable even after inserting the next one, the original design must be revised. Alternatively, the assembly sequence can be adjusted to ensure a stable assembly.

Exporting data for fabrication and assembly

Once task allocation is solved, the algorithm generates the corresponding assembly instructions as animated 3D holograms. For robotic assemblies, the animation shows the different steps of the robot’s trajectory, which helps the operator predict the next robot moves. For manual assemblies, the animation shows where and how to insert the piece. These animations are displayed through an AR headset (Hololens 2) using Fologram’s Grasshopper plugin. ²²

Regarding the fabrication of the pieces, the CNC toolpath generated by Manis is converted to the associated G-code using the Raccoon plugin. ³⁴ The latter plugin allows the simulation of the machining trajectory directly inside the Rhino interface before sending the code to a 5-axis CNC. On the robotics side, a custom Python algorithm, which relies on COMPAS RRC, ³¹ is used to execute the robot trajectories directly from Grasshopper.

Prototype design and fabrication

The developed workflow is tested through the design and assembly of a custom timber frame structure. The frame has 2.4 m length, 0.3 m width, and 2.13 m height (Figure 5). It is composed of 11 pieces of hardwood beech Laminated Veneer Lumber (39 mm of thickness and 730 kg/m3 of unit weight). The design includes three types of timber joints generated with Manis: half-lap joints, sunrise dovetail joints, and through tenon joints. The shape, weight, and dimensions of the parts, along with the type and layout of the connections, are varied more than in a typical timber frame structure. This is done to experiment with different assembly scenarios.OPEN IN VIEWER

Previous research has shown the difficulty of inserting several joints at the same time with a robot – especially when one piece is inserted into two or more pieces.^10,11 This applies to pieces 2, 4, 5, and 7 (illustrated in Figure 5), each of which needs to be inserted into two other pieces, as well as pieces 8 and 9, both of which must be inserted into three pieces. With the intention of testing a maximum number of configurations in a single prototype, the number of connections per assembly also varies from 1 (between pieces 0 and 1) to 6 (between pieces 9 and 10). Additionally, to facilitate the insertion of the pieces, half-lap joints are chamfered, and all joints are designed with a tolerance of 0.5 mm.

Task allocation

The worker’s height is set to 1.83 m and the maximum weight to 15 kg. With those parameters, pieces 1, 3, 6, 9, and 10 are considered too heavy to be lifted. Equations (1) and (2) are only respected for pieces 2, 4, 5, 7, and 8 as shown in Figure 6. The base of the structure, piece 0, is not included in the algorithm. It is assumed to be in place before the assembly starts.OPEN IN VIEWER

The results provided by the task allocation algorithm for each piece are summarized in Figure 7. It is observed that piece 8 cannot be assembled by the robot as the insertion motion is not a pure translation. This piece needs to be first inserted horizontally through pieces 1, 3, and 6. In addition, pieces 2, 4, 5, 7, 8, and 9 need to be inserted into multiple pieces at the same time. Consequently, manual assembly is prioritized for those pieces due to potential discrepancies. Since piece 9 is too heavy for manual assembly, it needs to be robotically assembled. However, the algorithm requires human supervision to ensure the alignment of the connections.OPEN IN VIEWER

According to the algorithm introduced in Section 2.3, pieces 1, 3, and 6 are not in a stable equilibrium right after being inserted (Figure 7). However, the equilibrium becomes stable after inserting the next pieces (i.e., pieces 2, 4, 5, and 7). Therefore, human-robot collaboration is required to accomplish the task. While pieces 2, 4, 5, and 7 can be robotically assembled, they need to be inserted manually to stabilize the previous pieces. Meanwhile, the robot needs to hold pieces 1, 3, and 6 in place until the next one is inserted by the human agent. As a result, the assembly sequence alternates between cooperation and collaboration phases (Figure 8).OPEN IN VIEWER

AR-guided assembly

Once it is confirmed that all pieces can be assembled manually or robotically, a fabrication toolpath is generated with Manis and Racoon. The panels are cut using a 5-axis CNC router. Next, the structure is assembled with the help of an AR headset (Hololens 2). As determined during task allocation (i.e., Section 3.2), half of the structure is assembled by hand, and the other half with a robotic arm capable of traveling on track. In the current study, the robot agent is an ABB 6400 robotic arm mounted on a 6 m-long track (Figure 9). A QR code is used to align piece 0 with the holographic model (Figure 9(a)). The robotic system is calibrated by manually referencing the position of piece 0 (Figure 9(a)) and the loading square (Figure 9(b)).OPEN IN VIEWER

Robot trajectories are computed by Manis from the insertion vectors associated with the pieces. Each pick-and-place trajectory generated by the plugin consists of nine steps, as comprehensively elaborated by Rogeau et al. ²⁹ Holographic animations are automatically generated as introduced in Section 2.4 and displayed through the Hololens headset with the Fologram application. This allows the simulation of the entire assembly sequence at full scale while checking for potential collisions between the robotic arm and the work environment. Once verified, the trajectory is executed. Depending on how each piece needs to be inserted, holographic animations automatically change to provide adequate guidance. For robotic assemblies, the animation shows the next robot’s move to the human agent (Figure 10(a)). For manual assemblies, the animation instructs the human agent about what piece to select and how to insert it (Figure 10(b)).OPEN IN VIEWER

To ensure the safety of the human operator, the robot’s speed is capped at 0.25 m/s when HRC is required. This follows ISO standards ³⁵ and previous safety perception studies. ³⁶ During piece insertion, the speed is further reduced to 0.01 m/s, allowing joints to slide into each other gradually without snapping. A warning appears on the AR interface when the robot is about to move at full speed, advising the worker to keep a safe distance. It is important to note that for this research, a complete lockdown of the assembly area is not possible. As a result, the robot’s maximum speed is always limited to 0.5 m/s. In an industrial setting, higher working speeds could be implemented if it is guaranteed that no human worker is near the robot, by using sensors and a closed cell.

Different levels of human-robot interactions

The developed algorithmic framework enables four scenarios of assembly which are exemplified through the realization of the prototype:

• Scenario I: Assembly by human agent only (piece 8)

As a result, different levels of interactions between robotic and human agents are supported. Each scenario can be associated with Cooperation or Collaboration following the definitions given in Section 1.3. Scenarios I and II correspond to Cooperation as robotic and human agents perform their actions individually and sequentially while scenarios III and IV involve Collaboration as both agents work simultaneously on the structure.

The assembly of the demonstrator highlights the benefits of considering construction as a combination of cooperative and collaborative tasks. As reported in the above scenarios, the assembly of some components requires the collaboration of human and robot agents. However, certain parts can only be inserted by the human agent due to the complexity of the insertion path, while others must be assembled by the robot agent due to weight and height constraints. Two forms of collaboration are also distinguished. In Scenario III, the assembly is led by the human agent with the support of the robot, and conversely, in Scenario IV, the robot inserts the piece with support from the human agent. As a result, the developed framework recognizes both the value of collaboration and the respective skills of each agent.

Impact of Augmented Reality on collaborative construction

This study demonstrates three use cases of AR applied to the prefabrication of timber structures with robots:

• AR as a support tool to guide manual assembly

As a support tool, AR proves to be particularly useful for assembling timber joints with non-orthogonal insertion vectors. When the vector is perpendicular to the contact zone, the assembly is fairly intuitive and can be done without guidance. However, with non-orthogonal vectors or when several joints are to be assembled simultaneously, visualizing the correct trajectory greatly assists in alignment and time-saving. Furthermore, seeing a full-scale representation of the piece in 3D lets the worker easily confirm they have the right piece, reducing errors commonly associated with paper plans. As a result, the skills of the human agents are augmented by the AR headset, facilitating cooperation in Scenario I and collaboration in Scenario III.

Using AR as a simulation tool allows for each robot trajectory to be verified directly within the workspace, rather than in a replicated environment on the laptop screen. The main benefit is ensuring that the computed trajectory does not lead to any collisions. This feature is particularly useful during the assembly of the structure. Despite the algorithm indicating a collision-free trajectory on the screen, some discrepancies between the robotic setup and its digital twin mean that the piece carried by the robot could potentially hit the structure or any other obstacles in the workspace. By checking the trajectory with the AR headset, this error is successfully prevented. Consequently, AR also benefits agents’ cooperation in Scenario II.

As a prediction tool, AR helps anticipate the movements of the robot. Although all trajectories are computed beforehand and can be previsualized before construction, AR provides reassurance to the worker to know the next move of the robotic arm in real-time. Therefore, AR holograms can serve as a communication medium between robots and humans. This is particularly useful during the assembly of the final piece of the prototype (piece 10), which requires the presence of human agents in the robot workspace to supervise the assembly. The ability to see the final position of the piece before it is inserted helps the worker anticipate whether all connections are aligned or if an intervention will be necessary. Ultimately, AR also supports the agent’s collaboration in Scenario IV.

Design integrating assembly constraints

The developed workflow integrates assembly constraints into the design process, providing instant feedback on the feasibility of manual or robotic assembly. For instance, the size of piece 8 is based on feedback from the task allocation algorithm. Earlier iterations of the prototype deemed the piece too heavy for manual insertion, and robot insertion was ruled out as the piece needed to be inserted through other pieces (i.e., pieces 1, 3, and 6). A slight size reduction of piece 8 kept the element in the yellow area of Figure 6 without necessitating drastic modifications to the design of the overall structure. In addition, results from the stability algorithm reported in Figure 7 also highlight the importance of pieces 2, 4, 5, and 7. These parts were not present in the first version of the prototype. However, these smaller triangular elements proved to be necessary to stabilize parts 1, 3, and 6. Although this study focuses on the construction of the prototype rather than its design, the integration of the task allocation algorithm into Manis also provides new insights for assessing the assembly feasibility of structural components during the design phase.

Performance assessment

The main challenge of digital construction workflows is managing fabrication and assembly tolerances. There is a need to address technological limitations and environmental factors causing discrepancies between the virtual model and the prototype, even though the structure was eventually assembled. Although optimizing the rigidity of the connections is not the primary focus of this research, observations about the accuracy of the different processes are included.

During the fabrication stage, CNC machining provides accuracy within a tenth of a millimeter. However, the thickness of the panel from which the elements are cut varies between 38 and 40 mm. Therefore, it is necessary to surface the joints to maintain a consistent thickness of 38 mm and an insertion tolerance of 0.5 mm in the joints. Concerning the material, beech LVL has relative stability against hygrometric variations. However, the slenderness of some pieces can lead to minor bending deformations upon assembly. Therefore, linking pieces 1, 3, and 6 through piece 8 is critical to avoid large deviations at the top of the structure.

The accuracy of the robotic assembly relied on millimetric manual calibration. By referencing the loading and assembly zones separately, it is possible to dismiss slight variations as the robot moves along the track. However, the size and weight of the components are at the limits of the pneumatic gripper’s capacity, making it necessary to reduce the travel speed of the robot to prevent the pieces from tilting. Anti-slip patches are also added to the gripper to avoid rotation. The rounded chamfer applied on all half-lap joints also effectively counterbalances the potential accumulation of tolerances in the structure, in particular for pieces 8 and 9.

Regarding the holograms, their positions are accurate only within 5 cm, and this varies depending on the viewing angle and distance from the referenced QR code. Despite this effect, the accuracy is sufficient for the application studied. The use of integral timber joints, as opposed to nailed joints, is advantageous because it guarantees that each piece ends up in the correct place despite the AR headset’s relatively low accuracy. More precision is needed to guarantee the validity of the feedback provided, especially concerning the display of robotic simulation. Nonetheless, holographic animations provide useful supplementary information for a better understanding of the assembly.

Achievements

In conclusion, a new task allocation algorithm is introduced and verified during the assembly of a demonstrator. The new algorithm considers the physical characteristics of human and robot agents to determine whether components can be assembled. For each component, four different assembly scenarios involving either the cooperation or the collaboration of both agents are evaluated. Therefore, different levels of interaction between human and robot agents are enabled throughout the assembly. The study highlights that these different levels of interaction are complementary. While certain assemblies can only be achieved through collaboration and cannot be realized by one agent alone, the opposite is also true, and asynchronous cooperation sometimes proves to be necessary.

Moreover, this research leads to the establishment of a streamlined workflow for automatically generating holographic animations from 3D models of timber structures. This contributes towards a holistic and fully integrated design interface that considers all design aspects related to the construction of timber systems, including constraints related to digital fabrication, assembly, structural integrity, and stability. Furthermore, the use of an AR headset facilitates human-robot interactions in the four identified assembly scenarios. It supports cooperation and collaboration between agents by allowing the worker to visualize robot trajectories before and during execution and by supporting manual assembly through holographic guidance. Besides, as the number of pieces in a structure increases, the importance of AR-guided assemblies is expected to grow due to the ability to easily track pieces and simplify data transfer.

Outlook

The current task allocation algorithm determines the suitability of a piece for manual assembly based on the strength and reach of the human agent whereas for robotic assembly, motion complexity and stability are also considered. In the future, it would be interesting to integrate other criteria such as the expertise or the fatigue of the operator. In addition, further developments are needed to expand the methodology to assembly operations involving several human and robot agents with different characteristics. This would enable the emergence of new scenarios of interactions involving the collaboration or cooperation of three or more agents. Future work could also focus on generalizing the methodology beyond the context of timber construction or interlocked assemblies. In the absence of timber joints to guide the assembly, the precise positioning provided by robot agents would be an advantage over human agents. Therefore, collaboration and cooperation scenarios would be modified.

The robustness of holographic projection is still a hurdle for a wider application of AR in construction, yet Kyaw et al. have managed to reach millimetric precision in the AR-aided creation of Glued Laminated Timber beams using markers placed every 40 cm and drift correction software. ³⁷ Additionally, AR is leveraging the latest breakthroughs in AI and simultaneous localization and mapping (SLAM), potentially eliminating the requirement for markers and minimizing drift issues. Haptic tools also offer the potential for more instinctive and multi-sensory interactions. ³⁸ Therefore, these technological improvements could be explored to achieve more precise holographic instructions and an enhanced user experience.

In addition, the impact of the integration of assembly constraints on the design process should also be further investigated. The current research demonstrates that design choices can be influenced by the task allocation algorithm (as reported in Section 4.3). Furthermore, constraining the design space to components that can be only carried by robotic or human agents implies working with relatively short-span elements and, therefore, an increased number of connections. As a result, future research could explore the architectural production associated with the developed framework.

Lastly, more research is needed on the social impact of the developed workflow, particularly on the integration and acceptance of AR and robotics under real conditions. Wearing an AR headset for an entire workday currently seems impractical due to visual discomfort. Therefore, it could be beneficial to explore other AR options, such as those using smartphones, tablets, or projectors. As sometimes joints can be easily inserted without AR guidance, it would also be interesting to identify when the AR headset is effectively augmenting the worker and when it might be considered a hindrance. Finally, the digital transformation of the timber construction sector extends beyond AR and robotics. It may encompass various other technological advancements and new policies. The primary challenge is to interconnect these different elements to build a coherent ecosystem where people can work confidently and securely.