Hyve-3D and the 3D Cursor: Architectural co-design with freedom in Virtual Reality

Abstract

Hybrid Virtual Environment 3D (Hyve-3D) is a system that allows architectural co-design inside Virtual Reality by a new model of interaction through a 3D cursor. It augments the concept of the cursor to better interact with three-dimensional virtual spaces, rethinking it as a drawing/control plane and viewpoints inside the virtual world. Handheld tablets intuitively manipulate 3D cursors. Users can simultaneously access their individual complementary views on the tablets as personal windows into the shared immersive display. They can concurrently sketch in three dimensions, transform, and manipulate three-dimensional objects using the tablets as tangible props and collectively navigate the scene using the tablet as a 3D trackpad. The system implementation and co-design assessments of different settings are presented.

Keywords

Co-design virtual reality human-computer interaction 3D cursor and 3D sketching

Introduction

The mouse, based on the two-dimensional (2D) x–y coordinate system, was adopted as the standard input model for graphical interaction. Planar and pen/tablet-based interactions and multi-touch interfaces, which are similarly 2D, are commonly used by three-dimensional (3D) design software. These models of interaction increase the complexity of the Graphical User Interface (GUI) and make it difficult to design and interact within the 3D space. Also, they only provide a window of the 3D content, making impossible for the user to master spatial configuration and proportions. Although Virtual Reality (VR) seems to be a solution to this problem, due to the absence of appropriate interfaces and interactions for architectural design, most VR systems only act as a visualization tools for content created by conventional 3D software. Often conceptual design is not carried out on the computer but using intuitive methods like sketching. Moreover, the “personal computer” which is designed for individual use, is hindering the collaborative nature of the architectural design process by not allowing collective work.

This article presents Hyve-3D and the 3D cursor as a natural and interaction model to provide design freedom during the first steps of architectural co-design. This new interaction is implemented in a non-intrusive immersive anamorphic projection system, based on spherical panoramas. In Hyve-3D, the notion of the 3D cursor is proposed as 2D construction and control planes that build 3D sketches and transform 3D models. Each user can move their 3D cursor intuitively by manipulating tablets. Once the cursor is in place, the user can create freehand sketches, which are then transformed onto a virtual 3D plane. The tablets display complementary orthogonal views of the 3D scene, to the perspective immersive view surrounding the users. Users can see orthogonal views from any direction by moving and rotating the tablet freely in the air. Multiple users controlling individual 3D cursors can generate 3D sketches and manipulate/transform imported 3D models collaboratively.

Sketching in architectural design process

GUI complexity arises in 3D modeling due to the fact that 3D data need to be supplied via abstract 2D interfaces. This complexity distances the architect from creative thinking since he or she is not focusing on the design task but distracted by the system requirements. Structured interaction of the mouse with menus forces the architect to make premature decisions, demanding more accuracy compared to pen-on-paper techniques.¹

Architects still use traditional media to produce freehand sketches as external representations for ideation before transitioning to the computer to develop or present the idea. Sketches function as conduits for visual thinking and problem solving, as well as externalization and communication of design ideas during several iterative phases of design development, from ideation to presentation.² The importance of the sketch has been shown by several studies suggesting that its characteristic ambiguity (alternative meanings), abstraction (simplification), and inaccuracy (flexibility) help the conceptual design.^1,3–5 Sketches provide a medium of freedom with a flexible degree of abstraction, allowing multiple readings and interpretations, thus supporting creative leaps.⁶ Examination of sketches can provide visual cues that suggest ways to fine-tune design ideas.⁷ They are time-effective, flexible, tolerant, and transformable and ultimately supportive of feedback loops.⁸

Compared to accurate plans and finished shapes, sketches are a better way to represent architects’ thinking and deliberations during the first steps of the process.⁵ Sketches allow the discovery of new ideas in unexpected ways.^9,10 Unlike conceptually rigid computer-aided design (CAD) models, sketches provide affordances for design and redesign because they are much easier to manipulate.¹¹

In co-design settings, sketching enhances design problem solving by enhancing collaboration, supporting short- and long-term memory, lateral thinking, and enabling cycles of generation and re-interpretation.¹² In other instances, sketching was shown as not strongly influential for reciprocal re-interpretations of each other’s ideas in a collaborative setting, however contributing more to individual creative process, and accessing and enhancing earlier ideas.¹³

Sketching in VR

The lack of immersion, the difficulty to represent and understand complex 3D shapes, unintended proportion errors, disregard of the human scale, and the observer’s fixed angle of vision have been described by Landsdown¹⁴ as the drawbacks of freehand sketches if we compare them to the advantages offered by VR. Creating sketches directly in 3D in VR opens up a new dimension in the application of sketching in architectural co-design.

Immersive 3D sketching is proposed as an evolution of the traditional 2D sketch.^15,16 Totally different from traditional sketching, this approach creates new problems in addition to those of the traditional sketch: control of proportions in perspective view, accuracy in the 3D space, scale, and problems of configuration. Moreover, other solutions were proposed to sketch in VR in a non-immersive approach.^17–19 Nevertheless, it is questionable whether these successfully replicate the freedom of the traditional sketch but rather use it mainly as triggers.

Although VR is the product of technological evolution, we need to analyze it from the point of view of an experience and not just hardware.²⁰ The key to define VR in terms of user experience and not as a technology is the concept of presence, defined as the sensation of being in a 3D environment. It is obtained mainly through the user viewpoint control and direct interaction in real-time.²¹ The goal of immersive VR is to place the user inside a 3D environment that can be manipulated directly. The idea is that the users believe that they are interacting intuitively with the 3D environment and not with the computer. Developers employ sophisticated and intrusive equipment such as helmets, projection environments such as the CAVE (Cave Automatic Virtual Environment),²² and cockpits to obtain a total visual immersion allowing movement and direct manipulation.

The challenge is to design inside VR without all the problems of current interfaces, as it is the case with 3D modeling. Systems like the CAVE are passive with regard to the ideation process. Navigating and visualizing make us interact with VR in a passive way from a design point of view. The 3D models are still being made through 3D software outside VR.

Related work

Real-time concave projection

The anamorphic technique produces distorted projections that look normal when viewed from a particular position, projected on a spherical surface, or viewed through a curved mirror or lens. Spherical recording and projection of video content have also been explored recently, in which case necessary anamorphic deformations were achieved either by employing optical methods or by digital post-production of video content.²³ Real-time projection of 3D graphics on spherically curved displays presents a relatively novel challenge, as the virtual content needs to be transformed onto the projection surface using methods that are appropriate to the physical projection configuration that is used.

Several solutions have been proposed in dome-planetaria that display dynamic 3D content. Ott and Davis²⁴ describe the process of simulating a fish-eye lens for the production of full-dome animations, involving the use of 3D rendering software to produce square faces of a cubic panorama. Following this, an image-processing technique is described that carries out the anamorphic transformations from the cubic panorama to a flattened semi-spherical image, which is to be projected on the curved screen. Real-time implementations of the same method²⁵ exploit OpenGL features in order to carry out the anamorphic transformations required to match the perspective distortions implicated by the shape of the projection screen, which often is a spherical section.

3D sketching and working planes

Several past examples explore sketching in 3D space. Sketchpad+²⁶ proposes the concept of drawing planes as a method to create 3D sketch-models on a pen-based tabletop display. The users can manipulate the drawing planes using simple pen-gestures. SketchBoX²⁷ allows 2D sketching planes to be placed in Virtual Reality Modeling Language (VRML) scenes. The use of transparent drawing planes facilitates the creation of 3D sketches and annotations on top of existing VRML geometry. However, the placement and manipulation of these planes in 3D have not been described in detail. Moderato²⁸ is a similar plane-based 3D sketching environment where working planes can be positioned using a point-and-click method on existing 3D geometry. 3D Tractus²⁹ is a 3D sketching interface, which is based on the concept of a traditional adjustable drawing board. A tablet-sketching device is placed on a horizontal tracked surface that the users can move up or down. This allows direct mapping between virtual and physical spaces allowing intuitive interaction with 3D content. However, only vertical movement of the drawing plane is allowed and the proposed sketching software is designed for axis-aligned orthogonal drawing canvases, which limits the geometric possibilities. Mental Canvas³⁰ uses a set of pre-configured planar canvasses as a method of creating sketches in 3D with standard 3D transform tools to manipulate and position them in space. Sketches are made in an orthographic window and then transformed into perspective projection based on the actual position of the active canvas. Users cycle between previously used planes using the keyboard. Napkin Sketch³¹ pushes the 3D tracing paper metaphor further through an application built using a handheld tablet-based mixed-reality interface. The placement of the canvases is pointed out as a major bottleneck. The users are allowed to define new drawing planes by making a simple stroke gesture followed by a device tilting gesture. While the interaction is physically augmented, there is a limitation of scale because of the system’s design. Insitu³² is a contextually based design system with a pen–tablet interface. The system allows sketching within an environmental context created using 3D “pop-ups” derived from geographic data and 2D images collected on site. Drawing planes can be moved in reference to each other by dragging their intersection lines. Sachs et al.’s³³ 3-Draw uses a handheld palette and pen-based input for 3D surface modeling in a desktop application. The users sketch in the air, using the palette, to hold the virtual object. A stereoscopic version was developed by Fiorentino et al.³⁴ but still non-immersive, using scaled 3D models. This is also the case for Gravity Sketch.³⁵ Virtual planes are manipulated also using physical props. Hinckley et al.³⁶ use a cutting-plane prop and a doll’s head for a brain visualization task. The cutting-plane prop oriented relative to the doll’s head produces a clipped view of a brain scan in the display.

The techniques quoted above employ a version of the 2D-canvas metaphor to create sketches in 3D. Most of these studies rely on conventional input methods, with the only significant haptic improvement being the use of the tablet and stylus. Most of the realizations utilize a discrete set of drawing areas that stay within the drawing space once they are instanced. These then can be reused and moved in the scene. We find this aspect problematic because the user has to have a continuous awareness of the previously created drawing planes and their placement in 3D, and this would become burdensome for complex scenes.

Interaction

Concerning interaction, the CAT (Control Action Table)³⁷ allows the user to rotate 3D objects in 6 degrees of freedom (6DOF) and navigate in 3D space by interacting with a stationary tabletop interface. In T(ether),³⁸ 3D-tracked tablets are used to view and interact with volumetric data in an augmented reality context. It is a direct, one-to-one overlap between the physical creation surface and the 3D virtual space at the time of sketching not allowing large- or small-scale environments. Also, the visual experience is limited to the tablet screen. Katzakis et al.³⁹ presented a 3D cursor with a plan casting allowing 2D movements of the object with finger gestures on the touch screen of a smartphone.

Hyve-3D

Hyve-3D setup

Users are situated in a room, facing a display showing a 3D virtual environment. Each user holds a tablet that controls a personal 3D cursor (Figure 1). A satellite application running on the tablet couples the device’s three-axis orientation along with the multi-touch interactions on the screen and sends them to the computer (laptop) that manages the 3D scene. The 3D positions of the tablets could also be tracked in real-time using the magnetic 6DOF tracker. While the figure shows the spherical immersive configuration, a non-immersive display can be used (using a flat screen).

Figure 1.

Hyve-3D setup.

The 3D cursor

The 3D cursor is represented in the virtual world as a rectangular frame with the same ratio as the tablet display. The physical position and orientation of the tablet are used to manipulate the 3D cursor in virtual space (Figure 2). The 3D cursor is moved only when one of the four axial freedoms and restraints buttons (FREE, PLANAR, NORMAL, HINGE) is pressed and held. The four modes were implemented to facilitate the creation of 3D sketches and geometrically precise control of the 3D cursor.

Figure 2.

The 3D cursor.

The 3D cursor freely translates the physical intentionality of the user into VR through on-screen and physical/spatial tablet gestures. It becomes an extension of the user’s hands. The 3D cursor provides room for new affordances: the plane of the 3D cursor can function as communication device allowing the users to illustrate their verbal intensions via natural gestures (placing the plane can define space, or the tablet movement can easily explain the movement direction of a door). The position of the 3D cursor can be modified by the following two methods.

Using a 3D tracker

In this configuration, which employs a magnetic tracking system, the 3D position of the device in the real world is tracked relative to a receiver located in a static base station. To match the coordinate systems of the position of the device in the real world with the desired coordinate system of the 3D cursor, we apply the necessary conversion coordinate system transformations in the computer. Orientation data can be calculated using a combination of data produced by the accelerometer and gyroscope sensors embedded in the tablet.

Using a multi-touch gesture

The 3D cursor can also be moved in the virtual world by multi-touch gestures on the tablet. The rotation of tablet is used to change the orientation of the 3D cursor, and then every increment in the position of the finger on the screen is used to move the plane in the virtual world in the direction of the calculated rotated increment vector. Longer movements can be achieved by a series of accumulative panning motions.

Both 3D-tracked and multi-touch methods can be combined with the freedom and restraint movements we implemented (Free, Planar, Depth, Hinge) to better control the 3D cursor (see Figure 3).

Figure 3.

Axial freedoms and restraints: (a) Free, (b) Planar, (c) Normal/Depth and (d) Hinge.

Navigating VR: the 3D trackpad

Users can move and rotate the displayed 3D scene with multi-touch gestures (Figure 4). Combining the NAVIGATION button with a single-finger dragging gesture moves the view in the desired 3D direction, taking into account the current 3D orientation of the tablet (3D-Panning). For example, sliding up on the tablet screen while the device is held horizontally results in a forward movement. The same gesture results in an upward movement when the tablet is held vertically or a climbing movement when the device is held diagonally. This way, the tablet becomes a walking plane that translates planar movements into 3D.

Figure 4.

3D Trackpad, Panning, and Orbiting in different 3D orientations.

A two-finger twisting gesture rotates the point of view around the Z-axis located at the 3D cursor coordinates (Orbiting). A rotation around the vertical axis is preferred in order to maintain the verticalness of the users’ view, avoiding forward–backward or left–right tilting during rotation, thus maintaining the natural head orientation. The TILT button allows users to temporarily tilt the point of view in left–right and front–back directions by tilting the tablet. This helps to visualize a high-rise building or a ceiling. Releasing the TILT button restores the upright orientation of the view.

3D sketching

When none of the constraint or navigation buttons are pressed, the 3D cursor is fixed in space and it acts as a Drawing Area. Users can create freehand sketches on the tablet screen by either finger or pressure-sensitive stylus (Figure 5). The sketches made on the tablet are replicated on the Drawing Area. During this mode, the position and orientation inputs of the tablet are ignored, allowing the tablet to be held at any desired angle while sketching.

Figure 5.

3D sketching in Hyve-3D.

The size of the drawing area can be adjusted using a pinch-zoom gesture. This allows the users to work in multiple scales. While zooming, the position and orientation of the drawing area are also maintained. The methods described allow the users to create complex 3D forms as a series of sketched 2D profiles. Non-planar sketches (free 3D sketches) can also be generated, simply by sketching while using one of the four constraint modes allowing the 3D cursor to move during sketching.

A complementary display

The 3D cursor also functions as a window or a virtual camera that can be positioned anywhere in the virtual world. The complementary view from this camera is displayed on the tablet screen (Figure 6). Since each user controls a different tablet, it acts as a personal secondary display allowing the users to have different work areas and focal points in the design while the shared perspective is maintained in the immersive display. The tablet presents an orthogonal view, and depending on the placement and orientation of the tablet, the user can generate plans, elevations, and axonometric views of the 3D scene at any scale. The 3D cursor can also be used as a cutting plane in the virtual world. Placing the 3D cursor in a position that intersects an existing 3D object (i.e. a building), produces a sectional view on the tablet.

Figure 6.

Complementary orthographic view on the tablet and navigation mode (orange).

Selection and edition

Imported 3D geometry (Wavefront .OBJ) and 3D sketches can be selected using the 3D cursor via Butterfly-Net metaphor. This selection method is built upon the 3D cursor behavior. The user can select or deselect pressing the select button to activate the Butterfly-Net mode. The 3D objects intersected by the 3D cursor are selected/deselected. After the objects are selected, affine transformations, such as moving, rotating, scaling, and duplicating, can be done.

The selected objects become connected to the physical movements of the tablet; thus, the tablet becomes a prop or the tangible handle controlling the selected objects. This way a piece of furniture can be placed in a given space with user freedom. Scaling can be achieved using a pinch gesture. Selected objects can be duplicated while affine transformation actions are in progress. Instead of copying, and then pasting and transforming with separate actions, objects can be copied as part of the affine transformation action. Pressing the COPY button on the tablet screen while the objects are selected places a copy at the 3D cursor location. This way the objects can be previewed in location, orientation, and scale before being placed. Also, the navigation mode is available when objects are selected, making it possible to move objects beyond the immediate reach of the 3D cursor.

Co-design

Hyve-3D is specially conceived to support co-design by either using several 3D cursors connected to the same system or interconnecting to remote systems. Using an Internet protocol (IP)-based network, several systems running the same virtual world can be connected. The 3D cursor movements, selections, edition, and so on are synchronized between the systems in real-time. Furthermore, an icon of the point of view of the remote system is displayed to maintain the sense co-presence. A third party VoIP system is used for the verbal communication.

Immersion

The interaction schema is designed independently from the display configuration. In the non-immersive setting, the viewport to the virtual world is a conventional perspective camera. This allows presenting the virtual world on a conventional display. In immersive setting, a complete immersive (real-time) 3D scene is displayed via the panoramic projection. For the immersive setup, we implemented a spherical projection technique based on cubic environment texture mapping inspired by methods proposed by Ott and Davis²⁴ and Bourke.²¹ Our implementation is built using a real-time rendering engine and makes use of OpenGL vertex and pixel shading capabilities to render the 3D scene onto a spherical projection in real-time. Using a single projector, the spherically distorted image is reflected off a dome mirror and displayed on a 2.5-m-high, 5-m-diameter concave spherical screen made of an opaque fabric material. The screen allows the projection of the distorted image around an audience (see Figure 1).

Assessment

Immersive and non-immersive

We carried out preliminary evaluations of the two settings using three methodologies: the System Usability Scale (SUS),⁴⁰ log-file analysis, and an open exit interview. The protocol studied three configurations of Hyve-3D used by two teams (of two designers) in a co-design project: both teams using the immersive, both using the non-immersive, and asymmetric (only one immersive). The protocol was repeated for co-located and remote configurations, resulting in 12 tests, with 24 individual configurations (Figures 7 –9). For each test, teams were given 30 minutes to complete each different design project with equivalent complexity (bus shelter, newspaper kiosk, public bench, small marketplace, a dual phone booth, and bicycle rack).

Figure 7.

Protocol configurations.

Figure 8.

Immersive co-located configuration.

Figure 9.

Non-immersive co-located configuration.

Interface utilization

The duration where the interface is used actively corresponded to 28% of the total time to realize the project. About 72% of the time, the software interface was idle. We observed that this period was mainly utilized for design conversations (Figure 10; see Dorta et al.⁴¹). This finding is also in line with our expectations since collaborative design requires a lot of verbal and gestural communication between the teammates.

Figure 10.

Interface usage by time.

Using log-files from Hyve-3D application, we were able find out that all of the navigation, 3D cursor placement, and sketching functions were effectively utilized (Figure 10). In total, 38% of the active time was used for navigation (pan + orbit), which was followed by 34% for 3D cursor placement (tracked) (free, planar, and normal movements) and 28% for sketching. At the moment of the test, the hinge functionality was not implemented. The measured durations only include pen-down states; hence, we think that this is a well-balanced distribution. We did not find a remarkable difference in utilization patterns between the multiple configurations.

System Usability Scale

The immersive configurations received the highest average SUS score of 83 over 100. Asymmetric and non-immersive configurations received relatively lower average scores of 73.1 and 68.8, respectively. Immersion seems to have a strong positive effect on the usability of the system. While the system is usable in non-immersive mode, there seems to be a big usability penalty associated with this configuration.

Participation ratio

We also analyzed the interface usage data in order to find out whether different configurations influence the way the teammates contributed to the design activity while using the system. We calculated this by comparing the duration of utilization by the most active user and the less active one for each test and then calculating the average for a specific setting. We found that in both co-located and remote settings, one of the two teammates dominated the activity. In co-located settings, 61% of all the actions logged belonged to the most active user and 39% to the other. In remote collaboration, the distribution was 58% versus 42%, indicating a relatively more balanced participation. The imbalance was most pronounced in the navigation action where the same user in co-located collaboration settings did 78% of the navigation. In remote collaboration settings, this ratio dropped to 63%. The immersive-remote configuration resulted in the most balanced activity ratio, at 53% versus 47%.

Open exit interview

The users reported that the immersive-co-localized configuration produces freedom in gestures, fluidity, autonomy, and better and more communication between participants. One user has to give way to the other’s actions because of difficulties to manipulate the tool at the same time (navigating and placing the 3D cursor) or working at different scales on the project. This is consistent with participation ratio analysis. However, they appreciated the presence of the other person in order to know what and how to do.

In the immersive-remote configuration, the users have a feeling of design freedom because they feel unrestricted to move and sketch without affecting the other. However, compared to co-localized, they experienced a lower sense of co-presence of the other user influencing their actions and attentions (what and how to do), producing more silent moments requiring more effort to communicate. Compared to the immersive setting, the non-immersive-co-localized configuration produces difficulties in navigation and 3D cursor placement (having two translucent 3D cursors on the same screen creates conflict). The participants reported that the 27-inch flat display was too narrow to view the virtual environment. However, communication was easier allowing more discussions. As immersive-co-localized configuration, concessions were given between participants. Some ergonomic difficulties regarding tablet manipulation were mentioned due to the seated position. This particular configuration received the lowest SUS score of 63.75, which is explained by this feedback. Although the users report that the non-immersive-remote configuration provided the same freedom as the remote immersive, they also noted that it was hard to visualize the overview of the project and locate the other user’s 3D cursor.

Finally, in the asymmetric configuration, aside from the problems already described for non-immersion and remote collaboration, the teammate placed in immersion did not report any difference compared to immersive-remote configuration.

3D cursor tracking method

We performed a user assessment of the tracked and non-tracked (multi-touch cursor placement) version of the 3D cursor with four interior designers. Mental workload and usability were evaluated through the Raw Task Load Index (Raw TLX)⁴² and the SUS,⁴⁰ respectively. From Raw TLX we obtained a global score of mental workload and six sub-scales: mental, physical, and temporal demands, performance, effort, and frustration. In addition, we added five items aiming at measuring some qualities about the interface on a 5-point Likert scale: drawing precision, easiness in 3D cursor placement, sense of orientation within the system, the ability to perform intended tasks, and the quality of experience. Both versions of the 3D cursor required low-to-medium mental workload (Figure 11). A Wilcoxon signed-rank test determined that, in the non-tracked version, there was a statistically significant median decrease in global mental workload (–18%; z = −1.82; p < 0.10). Both versions resulted highly usable without significant differences. Globally, user experience resulted to be positive (Figure 12). Wilcoxon signed-rank test determined that there was a significant median increase in drawing precision (+25%) in the non-tracked version.

Figure 11.

Raw TLX global score.

Figure 12.

Qualities about the interface.

Overall, both tracked and non-tracked versions were easy to use. Designers were able to stay oriented within the system, they were able to do the intended tasks, and they found the interface enjoyable. However, non-tracked version required lower mental workload (34% vs 52%) possibly because it allowed more precision in drawings (88% vs. 63%).

Conclusion

This article presented a new intuitive interface for architectural co-design inside VR by 3D sketching using handheld tablets. Hyve-3D has adapted the ideation process where ideas are still abstract without using complex GUI and helped users to focus on the design task rather than on the tool. It is a natural interface since the body is instinctively used to position the 3D cursor in VR, using skills that we already possess to master the real world. An innovative user interaction is proposed with the 3D cursor using the tablet to create, view, and manipulate 3D content not only in immersion but also for conventional displays making the personal computer collective.

Compared to other explorations of 3D sketching which solely use the tablet surface as an augmented window to the virtual space, our approach uses the tablet as a 3D cursor linked to a separate control plane scaled and positioned relatively to the user’s position. For architectural projects, this allows freedom of scale, freedom of drawing area placement independent of physical constrains, and freedom of multiple views while maintaining a shared sense of immersion.

Despite appearing more intuitive to physically move the tablet to move the 3D cursor, the non-tracked version is more precise in placing the 3D cursor and for 3D sketching. The tracked version seems to be suitable for larger/general movements, while the non-tracked version provides better accuracy.

Based on the preliminary assessment, our system performs best when utilized in immersive setting. This is due to the fact that the proposed interface engages the user in an embodied manner making immersion an integral part of the interaction. We observed that remote settings seem to be preferred due to the provided flexibility: the users were able to adapt their collaboration pattern to suit the specificities of the co-presence by taking specialized roles (e.g. one navigating while other sketching). While usability scores were higher for immersive settings, we foresee that the non-immersive setting can be used as a complementary system for remote collaboration and small office or studio use.

Footnotes

Acknowledgements

The authors would like to thank Davide Pierini and Sana Boudhraâ for their valuable contribution, and all the participants.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

This work was supported by the SSHRC-Insight research grant from the Social Sciences and Humanities Research Council (SSHRC) and the CFI-Leaders Opportunity Fund research grant from the Canada Foundation for Innovation (CFI).

References

Gross

EY-L

. Ambiguous intentions: a paper-like interface for creative design. In: Proceedings of the ACM symposium on user interface software and technology, Seattle, WA, 6–8 November 1996, pp. 183–192. New York: ACM.

Goldschmidt

. Serial sketching: visual problem solving in designing. Cybernet Syst 1992; 23(2): 191–219.

Daru

. Sketch as sketch can: design sketching with imperfect aids and sketchpads of the future. In: Proceedings of the eCAADe conference: experiences with CAAD in Education and Practice (ed Pittioni

), Munich, Germany, 17–19 October 1991, pp. 162–172. Munich, Germany: eCAADe.

Oxman

Shaphir

Yukla

. Beyond sketching: visual reasoning through re-presentation in cognitive design media. In: Proceedings of the CAADRIA’98 conference, Osaka, Japan, 22–24 April 1998, pp. 337–346. Japan: CAADRIA.

Goel

. Sketches of thought. Cambridge, MA: MIT Press, 1994.

Ullman

Wood

Craig

. The importance of drawing in the mechanical design process. Comput Graph 1990; 14(2): 263–274.

Suwa

Tversky

. How do designers shift their focus of attention in their own sketches? In: Anderson

Meyer

Olivier

(eds) Diagrammatic representation and reasoning. London: Springer, 2002, pp. 241–254.

Goldschmidt

. Modeling the role of sketching in design idea generation. In: Chakrabarti

Blessing

LTM

(eds) An anthology of theories and models of design. London: Springer, 2014, pp. 433–450.

Suwa

Gero

Purcell

. Unexpected discoveries: how designers discover hidden features in sketches. In: Gero

Tversky

(eds) Visual and spatial reasoning in design, key center of design, computing and cognition. Sydney, NSW, Australia: The University of Sydney, 1999, pp. 145–162.

10.

Shimizu

Terhorst

McConville

. OmniMap: Projective Perspective Mapping API for Non-planar Immersive Display Surfaces (eds Babis

.). Berlin, Heidelberg: Springer-Verlag, 2008, pp. 975–986.

11.

Tversky

. Affording design, affording redesign. In: Toshiharu

(ed.) Principia Designae: Pre-Design, Design, and Post-Design. Kobe, Japan: Springer, 2015, pp. 91–102.

12.

Craft

Cairns

. Sketching sketching: outlines of a collaborative design method. In: Proceedings of the 23rd HCI group annual conference on people and computers: celebrating people and technology (BCS-HCI’09), Cambridge, 1–5 September 2009, pp. 65–72. New York: ACM.

13.

Van Der Lugt

. Functions of sketching in design idea generation meetings. In: Proceedings of the 4th conference on creativity & cognition (C&C’02), Loughborough, 14–16 October 2002, pp. 72–79. New York: ACM.

14.

Landsdown

. Visualizing design ideas. In: MacDonald

Vince

(eds) Interacting with virtual environments. Toronto, ON, Canada: Wiley, 1994, pp. 61–77.

15.

Donath

Regenbrecht

. Vrad (virtual reality aided design) in the early phases of the architectural design process. In: Proceedings of the sixth international conference on computer-aided architectural design futures: the global design studio (ed Tan

Teh

), Singapore, 24–26 September 1995, pp. 313–322. Singapore: CAAD Futures.

16.

Donath

Regenbrecht

. Using virtual reality aided design techniques for three-dimensional architectural sketching. In: Proceedings of the design computation: collaboration, reasoning, pedagogy (ACADIA’96) (ed McIntosh

Ozel

), Tucson, AZ, 31 October–3 November 1996, pp. 199–212.

17.

Achten

De Vries

Jessurun

. DDDoolz: a virtual reality sketch tool for early design. In: Proceedings of the fifth conference on computer Aided architectural design research in Asia (CAADRIA 2000) (ed Tan

Tan

Wong

Y-C

), Singapore, 18–19 May 2000, pp. 451–460.

18.

EY-L

. Sketch that scene for me: creating virtual worlds by freehand drawing. In: Proceedings of the promise and reality: state of the art versus state of practice in computing for the design and planning (eCAADe, 2000) (ed Donath

), Weimar, 22–24 June 2000, pp. 265–268.

19.

EY-L

sketchpad

. Computer Aided Architectural Design Futures 2001. In: Proceedings of the ninth international conference (ed De Vries

van Leeuwen

Achten

), Eindhoven, 8–11 July 2011, pp. 161–172. Dordrecht: Kluwer Academic Publishers.

20.

Steurer

, Defining virtual reality: dimensions determining telepresence. In: Biocca

Levy

(eds) The communication in the age of virtual reality. Hillsdale, NJ: Lawrence Erlbaum Associates, 1995, pp. 33–55.

21.

Zeltzer

. Autonomy, interaction and presence. Presence: Teleop Virt 1992; 1(1): 127–132.

22.

Cruz-Neira

Sandin

Defanti

. Surround-screen projection-based virtual reality: the design and implementation of the cave. In: Proceedings of the SIGGRAPH’93, Anaheim, CA, 2–6 August 19931993, pp. 135–142. New York: ACM.

23.

Bourke

. An alternative approach, http://paulbourke.net/oldstuff/ozviz2004/ozviz2004.pdf (2004, March 2016).

24.

Ott

Davis

. Simulating a virtual fisheye lens for the production of full-dome animations. In: Proceedings of the ACMSE 2007, Winston-Salem, WA, 23–24 March 2007, pp. 294–299. New York: ACM.

25.

Bourke

. Low cost projection environment for immersive gaming. J Multimed 2008; 3(1): 41–46.

26.

Piccolotto

. Sketchpad+ architectural modeling through perspective sketching on a pen-based display. Master’s Thesis, Cornell University, New York, 1998.

27.

Stellingwerff

. Sketchbox. In: Proceedings of the architectural computing from Turing to 2000: 17th eCAADe conference proceedings (ed Brown

Knight

Berridge

), Liverpool, 1999, pp. 491–497.

28.

Mase

. Moderato: 3D sketch cad with quick positioned working plane and texture modelling. In: Proceedings of the promise and reality: state of the art versus state of practice in computing for the design and planning process (18th eCAADe) (ed Donath

), Weimar, 22–24 June 2000, pp. 269–272.

29.

Lapides

Sharlin

Sousa

. The 3D Tractus: a three dimensional drawing board. In: Proceedings of the TableTop2006: first IEEE international workshop on horizontal interactive human-computer systems, Adelaide, SA, Australia, 5–7 January 2006, pp. 169–176. New York: IEEE.

30.

Dorsey

Smedresman

. The Mental Canvas: a tool for conceptual architectural design and analysis. In: Proceedings of the 15th Pacific conference on computer graphics and applications, Maui, HI, 29 October–2 November 2007, pp. 201–210. New York: IEEE.

31.

Xin

Sharlin

Sousa

. Napkin Sketch—handheld mixed reality 3D sketching. In: Proceedings of the VRST’08, Bordeaux, 27–29 October 2008, pp. 223–226. New York: ACM.

32.

Paczkowski

Kim

Morvan

. Insitu: sketching architectural designs in context. ACM T Graphic 2011; 30(6): 182.

33.

Sachs

Roberts

Stoops

. 3-Draw: a tool for designing shapes. IEEE Comput Graph 1991; 11(6): 18–26.

34.

Fiorentino

Amicis

Monno

. Spacedesign: a mixed reality workspace for aesthetic industrial design. In: Proceedings of the 1st international symposium on mixed and augmented reality (ISMAR’02), Darmstadt, Germany, 30 September 2002, pp. 86–94. New York: ACM.

35.

http://www.gravitysketch.com (accessed 28 August 2015).

36.

Hinckley

Pausch

Proffitt

. Two handed virtual manipulation. ACM T Comput-Hum Int 1998; 5(3): 260–302.

37.

Hachet

Guitton

Reuter

. The CAT for efficient 2D and 3D interaction as an alternative to mouse adaptations. In: ACM SIGGRAPH 2004, Los Angeles, CA, 8–12 August 2004, pp. 731–731. New York: ACM.

38.

Lakatos

Blackshaw

Olwal

. T(ether): spatially-aware handhelds, gestures and proprioception for multi-user 3D modeling and animation. In: Proceedings of the 2nd ACM symposium on spatial user interaction (SUI’14), Honolulu, HI, 4–5 October 2014, pp. 90–93. New York: ACM.

39.

Katzakis

Kiyokawa

Takemura

. Plane-casting: 3D cursor control with a SmartPhone. In: Proceedings of APCHI’13, Bangalore, India, 24–27 September 2013, pp. 24–27. New York: ACM.

40.

Brooke

. SUS: a quick and dirty usability scale. In: Jordan

Thomas

Weerdmeester

. (eds) Usability evaluation in industry. London: Taylor and Francis, 1996, pp. 189–194.

41.

Dorta

Kalay

Lesage

. Elements of design conversation in the interconnected HIS. Int J Des Sci Technol 2011; 18(2): 65–80.

42.

Hart

. NASA-Task Load Index (NASA-TLX); 20 years later. Proc Hum Factor Ergon Soc Annu Meet 2006; 50(9): 904–908.