Using deep learning to generate design spaces for architecture

Motivation

In this paper, we describe the development of an AI-based 3D design tool. We present an early prototype of a design system that uses Deep Learning methodology—Conditional Variational Autoencoders (CVAE). We believe our approach presents tentative solutions to two of the main obstacles that deep learning based 3D design tools currently face: the lack of training data and the lack of labels describing those data semantically. We demonstrate that by creating our own training datasets and by using a comprehensive taxonomy (we used the one from the booklet on Operative Design by Di Mari and Yoo ³ ) we arrive at custom design spaces that can be interactively and meaningfully explored.

As we explain in some detail in the following sections, our approach to solving the lack of training data is to use parametric models from which we generate thousands of 3D model variations. These models are then voxelized and reduced to the same 24 by 24 by 24 unit grid. The parametric models each have a semantic label, using the Operative Design ³ taxonomy. Note that the way we use the term “semantic label” in this paper is different from the way it is commonly used in deep learning projects to do with image segmentation, where each pixel is semantically labeled as being part of an object. In this paper, semantic labels describe a geometric property of the whole composition rather than individual voxels.

These labels (we used six in total in the tests described here) are encoded along with the voxel models and create a latent space that can also be navigated using and mixing expressions such as bend, taper, and twist. Needless to say, given the low resolution and the limited number of labels, the samples of our current results are a far cry from the astonishing results available in the AI text-to-image world. We are still only laying the groundwork. To do our prototype system justice, one needs to keep the bigger picture in mind. In this introduction, we therefore want to spend some time discussing this bigger picture. What can 3D generative deep learning tools, such as the one we are proposing, eventually do for architectural design?

On parametric design and design spaces

Parametric design is becoming increasingly popular and important in architectural practice. Arguably this is not only because parametric design tools have become more powerful and more accessible to the user but also because there is a growing need for their benefits. The main benefit is that parametric models define what is referred to as a “design space” or “solution space”: the sum of all possible design variations that result from changing its parameter values. A good design or solution space is one, which a user can then easily navigate to find fitting solutions.

A good parametric model will, for example, allow designers to quickly compare variations of a building’s proportions or to tweak a façade until the rhythm of the openings looks just right. Given the immense time-pressure architects tend to work under, this speed and flexibility is a strong advantage. But it is not the only way design spaces can be used. Parametric design also opens the door for optimization in design. A well-defined design space can not only be explored by human designers but also by algorithms that can compare and find the best solutions according to certain criteria. In other words: design spaces enable optimization.

While the idea of optimization in architecture traditionally enjoyed little popularity with architects as it appeared to go against their artistic aspirations, this is currently changing. With the imperative demands climate change imposes on the building sector, just guessing at resource- and energy-efficiency in design will no longer do. Algorithmic methods, not as a way to replace designers, but as a way to allow them to make more informed design decisions, are already used in many high-profile projects. As demand for environmentally conscious construction can be expected to continue to rise, optimization in design likewise will continue to become more important and more commonly used. ⁴

The problem with this premise, however, is that creating good parametric models is not trivial. In fact, it must be considered a design challenge in its own right. ⁵ In this paper, we argue that deep learning methods can be used as alternative ways to create design spaces—design spaces with unusual properties and novel ways to explore them.

Using deep learning to create design spaces

As mentioned earlier, our training sets are not real-world data, like in most deep learning developments, but generated from parametric models. In an earlier paper, we have shown that the design solution space of a parametric design model can under certain circumstances be successfully translated into the mathematical latent space of a neural network—not perfectly, but reasonably well. ⁶ Creating an imperfect copy of a parametric model with a neural network may seem pointless, but conceptually, this translation was an important step in developing our system. It provided the basis for the subsequent developments we report on in this paper, which go beyond this equivalency. We were able to combine training data from a variety of different parametric models in such a way that the resulting latent space contained not only the models used in training but also combinations of those models.

As we demonstrate in this paper, the deep learning method we used (CVAE) resulted in a design space of considerable complexity. Creating an equivalent design space with a parametric model, would be a difficult design challenge. On top of that, our AI-generated design space also contains semantic labels, opening up novel, and intuitive ways to navigate it.

Use of neural networks in architectural design

As et al. ⁷ used Graph Neural Networks to explore an implementation of deep learning for conceptual design. Their aim was to evaluate existing design blocks, extract significant building blocks from them and merge them into new components. Through implementation of Generative Adversarial Networks (GANs) they were able to generate new designs. All their designs were represented through mathematical graphs. Newton ⁸ explored the implementation of GANs with regard to architectural design tasks. Most of the work was focused on 2D image generation, while also looking into the possibility of 3D voxel-based outputs. De Miguel et al. ⁹ implemented a Variational Autoencoder (VAE) for form finding of structural typologies. Although their implementation was successful the authors pointed out that they were unable to obtain a large diverse dataset, using instead variations of just two different 3D models. Koh ¹⁰ describes two 3D GAN based projects. The project 3D-GAN-Ar-chair-tecture implemented three trained 3D GAN Models. One was trained with 10,000 chair models, one with 4000 high-rise building models, and the third with a combination of both. The author was able to generate new variations of both chairs and buildings as well as interpolations between the two. The project 3D-GAN-Housing was trained on 5000 3D models from the Singapore Housing Development Board’s apartments, while encoding semantics and configurations of architectural programs directly into the dataset. After training a smooth interpolation between different buildings was possible. Asmaar and Sareen ¹¹ developed a new workflow where through the implementation of GANs multiple 2D outputs were vertically stacked to create 3D voxel structures. Ren and Zheng ¹² implemented a Neural Style Transfer for 3D voxelized shapes. This was achieved through the implementation of multiple 2D style transfers combining those outputs into a 3D structure afterwards. A well-known example is ArchiGAN by Chaillou. ¹³ Unlike some of the other examples, this GAN is completely image based, so its results are purely 2D. It can produce pixel based floorplans in different scales as well as different styles, depending on the training data: building footprints, program layout, and furniture layout.

Our approach differs from the ones cited here in that we use a different type of neural network: a Conditional Variational Autoencoder (CVAE). Furthermore, we solely work with 3D data derived from parametric models.

Creating custom datasets for AI based architectural design applications

The question of large enough datasets for neural network training is a key issue in deep learning. In particular 3D object datasets (especially models relevant to architectural design) are difficult to come by. A possible solution is the generation of one’s own datasets. Alymani et al. ¹⁴ presented a novel workflow to train a neural network to suggest appropriate building to ground relationships. The 3D architectural models were represented using topological graphs. They created the training datasets themselves. Newton ¹⁵ explored multi-objective qualitative optimization through the implementation of 3D-Convolutional neural networks. The necessary voxel based data for training was generated through the implementation of generative parametric design tools.

We also created our own datasets using parametric models. In fact, we used a set of very simple parametric massing models, inspired by the booklet on Operative Design by Di Mari and Yoo. ³

Generative neural networks for 3D voxel based design

Using 3D voxel data instead of pixel data is a common approach for 3D AI applications and we are by no means the first to use it. Andries et al. ¹⁶ implemented a Variational Autoencoder (VAE) to combine voxel geometries based on their affordances. Their work was focused on the creation of furniture and fixtures using 3D models taken from the ModelNet dataset. ¹ Wu et al. ¹⁷ implemented a 3D GAN to generate novel voxel based objects, mostly furniture. Their approach explored a probabilistic latent space to generate these new outputs. Spick et al. ¹⁸ propose a 3D GAN which includes color information within the generated voxel outputs.

Our 3D voxel datasets were generated from simple parametric models and also include semantic descriptions.

Autoencoders—Network architecture and principles

An autoencoder is a neural network which consists of three main parts: The encoder, the decoder, and the latent space or representation vector. ²² When the encoder compresses the input geometry, it decreases the number of digits needed for representing that input. This is achieved using so-called 3D-Convolutional layers whose function is to summarize and thus reduce certain parts of the input data into smaller chunks. Through the chaining of multiple 3D-Convolutional layers the encoder gradually reduces the original geometry representation, in our case 24³ (13,824) digits, down to the size of the representation vector, which for this project were 256 digits. This compressed 256-long representation vector is also referred to as a “latent space” vector. The latent space can be understood as a high dimensional coordinate system, where each individual digit of the representation vector stands for a single axis inside this space. Choosing any point (multidimensional coordinate) inside this latent space and passing it through the decoder generates novel output. Thus, each latent space point represents one unique geometry.

Advantages of variational autoencoders

As Foster ²² explains, regular autoencoders are not well suited as generative tools since they perform poorly when synthesizing novel outputs. This is because there are gaps in the latent space between the points and the decoder does not have information about how to reconstruct output from those gaps. Variational Autoencoders (VAE) were developed to address these problems. ²³ Using a multivariate normal distribution during encoding VAE can be trained so as to generate meaningful outputs from any place inside the latent space. This makes VAE much better suited as generative tools than regular autoencoders. However, navigating a 256-dimensional space is still very confusing. Rahman ²⁴ points out that VAEs do not offer great control over the generation of new outputs. In other words, in order to be able to use the latent space like a design space comparable to that of a parametric model, we need more control over how to navigate it.

Advantages of conditional variational autoencoders

A Conditional Variational Autoencoder (CVAE) ²⁵ aims to solve this navigation problem. The CVAE functions just like a VAE, however, it requires two inputs for the encoder: The geometry (just like a VAE) but also a semantic label describing that geometry. Consequently, the decoder also requires two inputs: The latent space point (just as the VAE) and a semantic label. Receiving those, the decoder generates an output representing the provided semantic label. To a human user, these labels provide much needed orientation for navigating the latent space. Figure 1 shows a diagram of a CVAE with all its essential parts.OPEN IN VIEWER

Training dataset creation

The size, quality, and diversity of the training dataset are key to producing a well working generative deep learning tool. We wanted to create a dataset that fulfilled the following requirements:

1. The 3D geometries should have different semantic labels.

The booklet Operative Design: A catalog of Special Verbs ³ by Di Mari and Yoo presents a taxonomy of simple geometrical operations that fulfills the above requirements. The authors explain: “These spatial operations are not ends unto themselves, but instead a set of illustrated beginnings to activate architectural inquiry, assembled to ignite the design process.” [3, p8]

In the logic laid out in the book, these operations can be used and combined in design exploration to create 3D shapes suitable for conceptual design. The authors demonstrate how certain combined operations have led to the real world realization of architectural projects. What makes Di Mari and Yoo’s work interesting for our experiment is not only the fact that their catalog of operations can easily be implemented in parametric design but that it is indeed a taxonomy, where every single operation has been semantically labeled by the authors. The combination of these operations is not only what is intriguing about their work, it is also precisely where our interest in our deep learning approach lies.

For this research six different operations were chosen from the catalog. Those were parametrically implemented using the Houdini FX 3D software. ² For each operation, a parametric script was created and its geometric output converted into voxel form. This data-generation workflow was automated. The following six geometrical operators were chosen for the project.

1. Bend: A bending deformation of the base volume with the bend axis at its middle section.

For each operation, 10,000 random variations were created and exported totaling a dataset of 60,000 voxel shapes and their corresponding labels. Figure 2 shows samples of each of the six operations before voxelization was applied. Figure 3 shows multiple voxelized geometries from the dataset.OPEN IN VIEWER

OPEN IN VIEWER

Figure 3.

A subset of the dataset used to train the CVAE. All geometries are represented in voxelized form.

Training the neural network

The CVAE was created using the Python 3 programming language utilizing the Tensorflow 2 ³ deep learning library. The basic network layer structure was adapted from the work of Brock et al. ²⁶ implementing changes necessary to convert a VAE into a CVAE.

The encoder consisted of six 3D convolutional layers, transforming the input shape of 24 × 24 × 24 to the size of the latent space (1 × 256). The decoder’s input consisted of two layers: one of the latent space shape, and one of 1 × 6 for the labels input. Both layers were concatenated and upscaled with a series of seven 3D transposed convolution layers shaping to the final voxel output of 24 × 24 × 24. A detailed diagram of both the encoder and decoder with all their layers can be seen on Figures 4 and 5.OPEN IN VIEWER

OPEN IN VIEWER

Figure 5.

A detailed diagram of the decoder, showing concatenation layer for input, followed by two fully connected dense layers, followed by seven Convolutional Transpose layers with alternating padding settings. Batch normalization layers were used throughout the network.

The reconstruction loss was calculated using binary cross entropy, while the KL loss was calculated using the Kullback–Leibler (KL) divergence. The overall loss was the addition of both reconstruction loss and KL loss. We weight the reconstruction with the hyper parameter R-Loss Factor to ensure that it is well balanced with KL divergence loss. Table 1 shows the chosen hyper parameters for the final model. Figure 6 shows the graphs for the reconstruction loss and KL loss for both the training set and the validation set. The validation set was implemented during training with a validation split of 10%.

OPEN IN VIEWER

Table 1.

The chosen hyper parameters for training the final CVAE model.

Hyperparamter	Value
Latent dimension	256
Batch size	32
Epochs	300
Starting learning rate	0.00025
Learning rate decrease	0.9 every 4 epoch after epoch # 50
R-loss factor	1000
Warm up of KL divergence loss	60 epoch
Optimizer	ADAM (default settings)

OPEN IN VIEWER

Figure 6.

The training graphs for the reconstruction loss (left) and the KL-loss (right) showing both the curves for training data and validation data. Note: A Warm up phase was used for the KL-Loss in order to slowly ramp it up for the first 50 epochs during training.

Multiple training runs were performed with different settings using the Google Colab Pro ⁴ online computational service. Each trained model was saved for performance comparison. Comparing the inputs to the outputs voxel-elementwise was an objective quantitative method for evaluating reconstruction performance.

The generative performance of the model was evaluated in a more subjective manner by visually inspecting the generated outputs. However, two quantitative metrics could be established for that task: Firstly, inspecting the distribution of data points for each individual axis in the latent space. Secondly, projecting the 256 dimensional latent space into 2D using a method called Principal Component Analysis (PCA) and thus evaluating if the distribution of latent space points appeared to be continuous. The model we chose for further exploration had a reconstruction accuracy of around 75%. Figure 7 on the left side shows the distribution of the individual latent space axes for the first 50 axes. We expected those to be normally distributed, which was the case. Figure 7 on the right side shows 5000 latent space points projected from 256 dimensions into a 2D plane. We see a mostly continuous and even distribution of the points around the center (inside the black rectangle) with minor gaps. Figure 8 shows original geometries and their reconstructed counterparts. We note that certain geometrical operations are harder to reconstruct than others (like Twist), and that larger objects are also easier to reconstruct than smaller ones.OPEN IN VIEWER

OPEN IN VIEWER

Figure 8.

Evaluating reconstruction performance of the CVAE on five examples. On top the original geometry from the dataset. On the bottom the reconstructed geometry.

Latent space navigation

When the latent space and the decoder are detached from the neural network, the CVAE becomes a generative tool. The latent space of the CVAE is now the equivalent of a design space. It contains all the parametric models’ individual design spaces used for training the CVAE, but not only that. While the individual design spaces of each of the parametric models is quite narrow, the CVAE’s latent space is wide as it contains objects that share traits of the various training models. Navigating the latent space is a bit like witnessing morphing operations between the original parametric objects.

Similarly, the encoding of semantic labels into the final design space now allows those original labels to be mixed. Although the neural network was trained with binary labels—meaning every geometry had only one of the six label flags (Bend, Expand, Inscribe, Pinch, Taper, Twist) set to 100% and the other five to 0%—the decoder can be provided with different label configurations, and still synthesize geometry. For example, a label combination of 10% Bend, 20% Expand, 0% Inscribe, 0% Pinch, 80% Taper, and 20% Twist will also produce a result (note that the percentages do not have to add up to 100%). Since the neural network was not trained with objects of such label combinations, the CVAE’s design space is the mathematical interpolation based on the network’s feature extraction abilities. This interpolation is not linear and leads to novel geometric assemblies. Despite the low resolution of the voxel space, the combination of the geometric operations is usually well legible. This is very much in the spirit of Operative Design, ³ which suggests that these operations be combined.

Random latent space points with label switching

The setup shown in Figure 9 features a simple graphical user interface (GUI) with six sliders (each for one of the semantic labels), which give the user the ability to control the influence of each attribute (sliders ranging from 0–1, representing 0%–100%) for any point in the latent space. An image preview shows the output. The user then has the opportunity to modify the slider composition and create a new object, or create a new geometry with a different random latent space variable, keeping the previously specified label configuration. Pressing the Export Geometry button saves the produced output as an .stl file. Upon pressing the Analyze button, a 2D image projection is created showing the user where in the latent space every previously produced object is situated, as well as the average position of all objects from one single category, that is, the Functional Form (more information regarding Functional Form in Section Object addition through importance vector extraction).OPEN IN VIEWER

One advantage of this setup is that it allows for label switching: With the same latent space point coordinates, objects can be created with different labels set at 100%, This can be seen as a simple “style-transfer” from one object to the other. Figure 10 shows the results of this style-transfer for four different latent space points. The generative tool successfully learned the basic geometric properties of the base volume residing at that latent space coordinate. This includes the base volumes’ general shape, size, and orientation.OPEN IN VIEWER

The same GUI can also create shapes based on multiple labels. Through this method, geometrical operations can be mixed. The results can be seen in Figure 11: Five different geometrical shapes were created with different amounts and degrees of label mixtures. For geometries created out of two labels, the individual operations can be easily read. Mixing more labels the operations get more intertwined, with their individual label readability decreasing. The ability to do label switching is possible because the neural network has a conditional decoder. This conditional generative structure enforces the disentanglement across the six operative terms via the conditional vector.OPEN IN VIEWER

Morphing between two objects

A different method for creating new geometries is the linear interpolation, that is, morphing, between two geometries. For any two geometries from the dataset the coordinates in the latent space can be recorded. Then multiple equally spaced apart latent space coordinates are calculated between the two. The number of new coordinates can be specified by the user. In case the two objects have different semantic labels, those labels are also recorded and gradually interpolated with the same number of steps. The pairs of interpolated coordinates and labels are then fed through the decoder to produce a morphing sequence from one object to another. Besides producing new geometries, this method is also useful to evaluate how evenly the latent space is distributed (KL-Loss performance). Figure 12 shows the morphing between two geometries of different labels depicted in 12 steps. We can observe a smooth transition between each step. Around the midpoint interpolation we achieve an even mixture of both objects.OPEN IN VIEWER

Object addition through importance vector extraction

Andries et al. ¹⁶ developed a method for adding voxel geometries, based on the concept of functional form. They used a VAE-generated latent space and inferred label categories directly from the dataset. The functional form is calculated by finding the mean coordinate for each object category with the same label. We modified their approach for use with a CVAE. We found that this method increases the level of control over the design space compared to using just the label categories as shown in Section Random latent space points with label switching. Figure 13 shows the functional forms of the six operations. We can see how the functional forms capture the geometric essence of each individual operation.OPEN IN VIEWER

Using a so-called importance vector, the functional form method allows geometries to be combined arithmetically, preserving the main features of both geometries in the combined output. When combining two geometries, a base-object has to be declared as well as a top-object. The latent space point for the combined object is then calculated using the importance vector. For this experiment, a GUI was created that allows the user to control this geometry arithmetic, using sliders and buttons similar to the GUI shown in Figure 6. We performed a variety of additions of two geometries, the results can be seen in Figure 14.OPEN IN VIEWER

The setup allows multiple additions to be performed on top of each other, and to experiment with various label slider combinations. The results are geometries such as the ones shown in Figure 15.OPEN IN VIEWER

Latent space as design space

Our approach is based on the idea that the latent space of a neural network can be seen as the equivalent of a design space in parametric modeling. While setting up a parametric model of some complexity is in itself a considerable design challenge, the training of our neural network is in fact rather straightforward provided there is a sufficient dataset. The complexity of its latent space results from the combination of very simple parametric models.

By generating our own training datasets, we achieve two objectives: Firstly, we solve the problem of a lack of 3D architectural data applicable for deep learning training. Secondly, we demonstrate in Sections Random latent space points with label switching through Object addition through importance vector extraction that there are interesting ways available for users to navigate the resulting latent space as a design space. Operations such as style-transfer, morphing, or semantic mixing are usually difficult to achieve in parametric models. For our deep learning based generative tool, they are straightforward built-in features. So, not only is the design space richer, more complex, and more varied than that of the individual parametric models that were used for training, there are also ways to explore it that are intuitive as well as surprising.

Regarding the creation of the training data: We do not suggest that a sufficient dataset can be built from a single parametric design script. This would defeat the purpose of the whole deep learning based generative network. However, we demonstrate that our approach allows for many different parametric models to be combined to generate a design space that can be navigated in novel and meaningful ways. In this way, multiple parametric design scripts, potentially from multiple designers, are combined to form a valid dataset. We propose that such custom datasets be created to tackle specific design challenges and that architects using this approach would not only navigate the latent space but also curate the dataset it is based on.

Design spaces and the optimization paradox

For algorithmic applications such as optimization, the latent space is indeed equivalent to a design space. As briefly mentioned in the introduction, optimization is becoming increasingly important in design. Design spaces are a necessary prerequisite for any optimization. As long as options are infinite, no optimization is possible. Of course the flip side of this is that any design space will, by its very nature, rule out many more potential schemes than it contains. So, the paradoxical fact is that design spaces at the same time enable as well as severely limit the impact and potential benefit of optimization methods. Our system of course does not change this fact. But it makes it easier to create design spaces that contain a larger variety of different schemes as well as any mixed type between them than typical parametric design approaches would.

“Operative design” as test case

We applied some of the taxonomy described by Di Mari and Yoo ³ in what they call “Operative Design” as a test case for our CVAE. Their basic operations not only directed the simple parametric models we used in our training, they also provided the semantic labels we used to control object generation. That the CVAE allows us to combine and mix their operations and play with them just as they demonstrate in their booklet, validates the idea that our system could become the basis of a conceptual design tool. At the same time, we want to point out that our system could also be trained with very different kinds of parametric models and labels and then also be used for very different design space explorations at other stages of the architectural design process.

Designing via the mixing console

Emmerer ²⁷ envisions digital tools for architects in analogy to an audio-mixing console where designers can interactively control different semantic design features and enhance or reduce their influence as a way of testing out design alternatives. We find this vision rather compelling. While sliders as a way to control parametric models are now common, Emmerer’s vision goes much further in that he doesn’t see his mixing console as controlling individual geometric parameters, but rather spatial qualities. As he puts it: “A machine-based counterpart needs to be posited vis-à-vis the mysterious ‘associative bond’ that allows us to recognize the qualities of a building design.” [27, p. 65]. Emmerer proposes what he calls “Architecture Routines” to create this associative bond. The semantic labels we use to control the generation of geometries are very different from Emmerer’s routines, yet, arguably they similarly do not control any individual dimension, but rather act globally on the whole model.