A shared realities workflow for interactive design using virtual reality and three-dimensional depth sensing

Abstract

This article presents the ongoing development and testing of a “shared realities” computational workflow to support iterative user-centered design with an interactive system. The broader aim is to address the challenges associated with observing and recording user interactions within the context of use for improving the performance of an interactive system. A museum installation is used as an initial test bed to validate the following hypothesis: by integrating three-dimensional depth sensing and virtual reality for interaction design and user behavior observations, the shared realities workflow provides an iterative feedback loop that allows for remote observations and recordings for faster and effective decision-making. The methods presented focus on the software development for gestural interaction and user point cloud observations, as well as the integration of virtual reality tools for iterative design of the interface and system performance assessment. Experimental testing demonstrates viability of the shared realities workflow for observing and recording user interaction behaviors and evaluating system performance. Contributions to computational design, technical challenges, and ethical considerations are discussed, as well as directions for future work.

Keywords

Interactive architecture user-centered design virtual reality three-dimensional depth sensing user interactions

Introduction

Since the mid-20th century, desires for citizen participation in design and planning decision-making coupled with developments in computer-aided design and information technologies have cultivated “technologically mediated” approaches in which design agency is distributed between users, technologies, and their designers.¹ As a critical player in this interdisciplinary approach, the field of architecture continues to engage design processes that blur the lines between designer and user toward what 1970s interactive design thinkers envisioned for a more resilient and democratic design process,² that is, one which directly engages users in the shaping of their built environments. This disciplinary expansion is inspired by decades of research in cybernetic theory,³ explorations in architectural adaptivity,⁴ and computing research in ambient intelligence.⁵ Pask, a pioneer in interactive design thinking, targeted the dynamically evolving “dance of agency” between the human and non-human as new design territory.⁶ Pask’s collaboration with Cedric Price on the Fun Palace was one of the earliest building-scale architectural examples exploring the possibilities of human–space interaction that established a system for its inhabitants to continuously construct an endless variety of spatial and programmatic options. Although never built, the interchangeable play of program in the Fun Palace sought to address the diverse and changing desires of inhabitants while employing cybernetics, game theory, and a unique display interface. Called the “Pillar of Information,” this interface was used to track and provoke users’ queries, expose them to others’ choices, and suggest multiple knowledge pathways within this interactive system.⁷

Recent work in interactive architecture explores various modes of interactivity for new space-defining systems and user experiences. Interactions range from passive to active, from ephemeral to embodied, and from low to high tech.⁸ Interactive systems at the building and interior scale have included transformable structures that engage users physically,⁹ immersive bioresponsive facade simulations that utilize gestural interactions,¹⁰ and biofeedback interfaces that respond to inhabitants’ physiological data through environmental actuations.¹¹ Interactive environments can support multiple users simultaneously and can be driven by sensor-augmented systems that respond to different types of human, material, and environmental stimuli. For example, Ada senses occupants using vision, audio, and touch detection,¹² and the Hylozoic Series employs adaptive distributed computing and mechanics with embedded machine intelligence to promote “living chemical exchanges” between the human and environment.¹³ Interactive systems for urban-scale design and analysis include platforms such as MIT’s CityScope project which uses rapid prototyping via computer vision and projection mapping coupled with a tangible urban model. CityScope’s user interface and real-time feedback have enabled iterative, evidence-based decision-making between traditionally siloed stakeholders,¹⁴ demonstrating the power of interactive data visualization and intuitive interfaces.

The human-centric, physical nature of these multiscalar interactive prototypes, environments, and platforms has the potential to engage and empower multiple users simultaneously, exposing one’s interactive experiences to others and thus inspiring new knowledge pathways much like the Fun Palace’s Pillar of Information. In order to sustain user engagement with such systems, the design of the user experience requires iterative testing and evaluation, ideally, with actual users in the environment in which they would use the system.¹⁵

Challenges to integrating user feedback in the design process

Methods to collect or map human behavior information with architectural spaces and systems range from computational models to observation of human subjects in physical facilities. In computational simulations, multiple data acquisition techniques and modeling methods have been developed to simulate occupant behavior for the design of intelligent environmental controls, for enhancing building system services to users, and for helping designers and engineers integrate such systems.¹⁶ Methods have involved artificial intelligence (AI) for mapping human behavior,¹⁷ ubiquitous computing for measuring degrees of user engagement,¹⁸ and agent-based modeling to simulate occupancy and building energy performance.¹⁹ Even these advanced computational methodologies face uncertainties associated with human behavior model inputs that can lead to simulation results that vary widely from actual outcomes. The logic behind simulated behavior requires assumptions that may not accurately reflect conditions and social dynamics particularly in relation to negotiations and group behavior.

Data collection on human behaviors in physical facilities can be categorized as in situ, laboratory, and surveys, which have potential to explore the social and psychological factors affecting behavior.²⁰ Laboratory facilities can be used to gather feedback from multiple participants simultaneously in a dedicated testing space through embedded sensors and controls in various settings and orientations.^21,22 While state-of-the-art laboratory facilities offer a more realistic physical setting, there are limitations to the number and profile of participants, and experiments are typically for a short and intense period of time outside of normal working or living contexts.

Despite these advanced methodologies, there is still a need for design tools that support modeling of the interaction between user and system, with the ability to visualize and synthesize interactive performance behaviors in order to test the robustness of the design.¹³ Constructing the physical or computational infrastructure to experiment with interactive behaviors during the design process is critical to understanding how interactive systems will perform.

Research objectives: toward a shared realities workflow

This research proposes a novel user-centered computational design workflow called “shared realities” that combines depth sensing and virtual reality (VR) visualization for improving the performance of interactive systems. The broader aim is to address the challenges associated with observing and recording user interactions with a system during the design process within the context of use. The specific objectives are to develop and test the shared realities workflow as a method for iterative design, observation of user interactions, and assessment of system performance. A museum installation is used as an initial case to validate the following hypothesis: by integrating three-dimensional (3D) depth sensing and VR for interaction design and user behavior observations, the shared realities workflow provides an iterative feedback loop that allows for remote observations and recordings for faster and effective decision-making. Like Negroponte’s²³ “surrogate self” model, in which the designer/machine models and converses with the user like a sentient being, the introduction of surrogate tools, sensors, and virtual spaces into the design process blurs the definition of designer and end-user while moving toward the decades-long vision of resilient and democratic design thinking.

Methodology

To address the research aims and objectives, the shared realities workflow combines a 3D physical model, 3D digital modeling tools, geographic information systems (GIS) and energy simulation software, multiple projectors, 3D depth sensors, VR devices, and the vvvv design environment (Figure 1). vvvv is a general purpose toolkit for programming large media environments with physical interfaces using a data-flow approach and a visual programming interface for rapid prototyping and developing.²⁴ Two types of interactive feedback are possible through this workflow. First is feedback through the display to users as they gesture to the installation and receive visual signals in the form of highlighted selections and graphic elements that follow their hand gestures. The second type of interactive feedback is in the form of real-time or recorded user point clouds, which can be viewed in the virtual design environment and through a VR headset. Through this hybrid digital–physical approach, the designer is able to virtually position themselves among the museum installation visitors in order to remotely observe their actual behaviors in context and make modifications in a virtual design environment. In essence, the designer and user share the same prototypical design space in different realities. This provides the designer with access to a multiperspectival, real-time view of visitors’ behaviors and gestural movements, creating new relationships between designer, user, and the mediating technologies. The following sections further describe the methods for (1) developing the interactive museum installation, (2) acquiring 3D user point cloud data for interaction and observation, (3) integrating VR for design feedback, and (4) measuring system performance.

Figure 1.

Diagram illustrating the shared realities workflow combining the following: a 3D physical model, 3D digital modeling tools, GIS and energy simulation software, projectors, 3D depth sensors, and VR devices for an interactive design feedback loop.

Test bed: development of an interactive urban energy platform

The interactive museum installation used as a test bed is part of an exhibition located at the Milton J. Rubenstein Museum of Science and Technology in Syracuse, New York, USA. The goal of the installation is to enable visitors to explore solar energy data sets through gestural interaction with projected media onto a physical model of the city of Syracuse.²⁵ Multiple projectors display texture maps and simulated data onto a vertical screen and physical model. To achieve gestural interaction with the projected media, two 3D depth sensors are located above the vertical screen for tracking visitors’ head and hand positions through point cloud recognition. The screen displays graphic information and directions for users to gesturally select data sets by pointing their hand and dragging selections to the model. Multiple users can interact simultaneously, selecting different data sets to view, overlay, and compare across the model (Figure 2).

Figure 2.

Schematic diagram of the physical installation setup, including the vertical screen, sensors, projectors, and 3D physical model.

The construction of the installation involves the development of a 3D digital model of the city in Rhinoceros 3D, which serves as a template for building the physical model as well as the geometry for generating projected media—in this case simulations of monthly solar radiation represented in kWh/m². Simulations are generated using the DIVA-for-Rhino plug-in grid-based analysis to obtain radiation map files containing X, Y, and Z locations, normal directions, and R, G, and B irradiances for each node. The R, G, and B irradiances are averaged to obtain a single kWh/m² value, and the node coordinates and their associated kWh/m² values are imported as a CSV file to vvvv. The CSV files are pre-processed into smoothed heat map image files in vvvv, which are then directly sampled at runtime. This keeps the runtime data and speed constant, as the heat map files are of much higher resolution than any potential sampling data. Once the heat map files are integrated into vvvv, the digital model and projected media can be calibrated to the physical model using vvvv and programmed for various types of user interactions. In our approach, the solar radiation heat maps are masked as “spotlights” that follow a user’s hand gestures (Figure 3).

Figure 3.

Photograph of a user with the museum installation using hand gestures to move the solar radiation map “spotlight” across the physical model (left), and screenshot of the vvvv design environment illustrating the user in the form of a point cloud interacting with the virtual installation model (right).

Acquisition of 3D depth data on user interactions

Methods for multiuser gestural interaction utilize Kinect for Windows V2 depth sensors combined with our highly optimized point cloud user recognition code developed by NOIRFLUX. The availability of color and depth (RGB-D) sensing systems creates opportunities for tracking user patterns while providing anonymous point cloud data at a high frame rate. Various applications of depth sensors have included occupancy detection and profiling for building energy management,²⁶ scanning of large environments to generate virtual representations in interactive 3D environments,²⁷ and capturing physical objects for representation in a real-time user-interactive, immersive VR environment.²⁸

The traditional methods of 3D user tracking with depth sensors, via Open Natural Interaction (OpenNI) and Kinect Software Development Kit (SDK) skeleton tracking, proved to be too limited for our application due to restrictions on sensor placement and the number of users that could be continuously tracked. Most gestural interaction only requires tracking each user’s head and hands which can be fairly easily deduced directly from a user point cloud, which itself is derived from the overall point cloud via a simple grouping algorithm. From the head and hands 3D positions, it can be deduced where the user is pointing as well as when they make gestures such as swiping and tapping. By operating on the raw point cloud, multiple cameras can be used in arbitrary positions and their point clouds combined to give good spatial and angular coverage, including being able to track gestures right up to a surface such as a table or screen directly in front of the user (Figure 4). This also allows tracking an arbitrary number of users, unconstrained by SDK skeleton limitations.

Figure 4.

Screenshot in vvvv of the virtual installation illustrating multiple users’ 3D point clouds interacting with the model simultaneously.

Recordings of complete point clouds versus just skeleton joint positions provide much greater fidelity for live interaction and later analysis and development use. A key feature of our approach is the ability to maintain user tracking continuity even with a partial point cloud so that the interaction context is preserved; this allows each user to make a selection which then stays associated with them. For example, each user can select what data type they wish to see and view their data displayed where they point at the model independently of what data other users are viewing. This has the added benefit of making the interaction process a social one, as users can see the result of not only their interaction but also of other users as well.

One of the major challenges with initially designing an interactive installation deals with on-site exchange of digital information and the continuous collection and analysis of anonymous user feedback, which is difficult to coordinate in a fully functioning space—in our case a science museum. This feedback is essential for future research on how to program an interactive system that people of different ages and backgrounds will find meaningful and comprehensible. The shared realities workflow aims to address these challenges by developing a method for integrating the 3D depth sensing system with VR visualization to streamline the interaction design process and user behavior observations, allowing for remote and real-time design feedback for fast and effective decision-making.

Integration of VR for seamless design feedback

The use of VR in architectural design has gained significant momentum over the past few decades, with digital tools, models, and games opening up new perspectives and the diverse involvement of stakeholders.²⁹ VR has been used in early building design evaluation,³⁰ human behavioral studies,³¹ and remote codesign in urban planning,³² holding potential for use in “pre-occupancy” evaluation stages of design.³³ In our approach, we leverage VR as both an observational tool and as an interactive design space for a seamless design and programming experience.

The vvvv environment is a live visual and textual programming system, which means that there is no edit/compile/run cycle, and changes made to the running system are immediately visible. As a result, developing an interaction experience is a directly interactive process in itself. The open architecture of vvvv has led to a large number of user-contributed plugins, notably in this case one for the OpenVR library,³⁴ which allows simple integration of the HTC Vive VR system as an input/output device. However, the need to constantly take the head mounted display (HMD) off to make a change in vvvv at the keyboard and then put the HMD back on to test it breaks the direct and continuous processes inherent in vvvv. To address this, a method is developed to allow working directly and continuously in VR.²⁵

The first step is to get the programming environment into the VR experience; to do this, the open-source program ScreenCapturer Recorder³⁵ is used, which captures the PC desktop and presents it as a video input device. Most if not all 3D programming systems allow capturing a video stream and displaying it on a surface in the 3D model, and in this way the PC desktop can be placed in the VR design environment. The next step is to put a view of the real-world keyboard and mouse in the VR design environment, so that the developer can fully interact with the virtual desktop just like the real desktop (Figure 5). This can be done by placing a video camera over the work area and displaying that in VR, but our method utilizes the Vive headset which has a built-in slightly downward-facing video camera on its front. As this camera appears to software as a standard video device, its live video is also easily incorporated into the VR scene.

Figure 5.

Screenshot in vvvv of the museum installation in VR and the integration of the designer’s desktop workstation screen into the VR design environment. The workstation uses the HTC Vive VR HMD camera for a split view of the virtual desktop and the physical keyboard and mouse, all accessible at full scale in the VR design environment.

By properly placing the virtual desktop and inserted video display in the VR model, the user can see a larger-than-life programming environment and directly interact with it while seeing their hands, keyboard, and mouse. This offers several additional advantages: first, it preserves the full interaction and development process without the limitations of systems that use VR controllers for click-and-drag types of interactions and does so without any specific programming. Second, the desktop and keyboard can be made to disappear except when the user is in proximity and actively looking at them. Third, to address situations where the developer is working with other people in the same physical space and needs to communicate with or view them directly, the complete overlay of the Vive video camera can be activated by a keyboard key or controller button, so that the developer can see the full real-world environment through the HMD. Using this system, while “in” VR the developer can watch a user point cloud interacting with the installation, make changes to the interaction process, and immediately replay the user interaction or see the live user data and view the effect of the change.

Methods for measuring system performance

The shared reality workflow enables evaluation of the performance of interactive systems utilizing feedback provided through the vvvv design environment, activity logs, and data observed or derived from user point clouds. One key indicator of performance is the frame rate of the display program, which is a measure of responsiveness and can be viewed in real-time in vvvv either through the PC monitor or in VR. A low frame rate affects the user experience in two ways: if too slow, the visuals will cease to be fluid, and the resulting stuttering intrudes into the conscious experience. A lower frame rate also means greater latency between the user’s action and the observed response. As vvvv is a data-flow programming environment, it provides a full graph evaluation for each frame.

Methods to evaluate system performance also utilize raw point cloud data and data derived from the vvvv program. The point cloud data excel for being able to view overall user pose and actions in VR and for playback to improve discrimination and tracking algorithms. For interaction metrics such as user dwell time and the time spent actually interacting, a more compact form of the data greatly simplifies analysis. In addition to the raw point cloud data, derived user data that are used by the interaction program are also recorded. These data consist of a timestamp, and for each identified user their assigned user number (for maintaining user interaction history and state), total point cloud size (a measure of quality), the center and size of the point cloud bounding box (used to help distinguish merged users), and the derived points for body center, head center, and any extended hand. The head and hand data are then used to figure where in the exhibit the user is pointing/interacting. From these data, interaction metrics and what specific parts of the exhibit were interacted with can be easily determined. Since individual users are labeled, more complex behaviors can be measured, such as the amount of time it takes for the users to interact with the core features, or how long it takes for the users to try interacting before leaving. Analysis of individual user data can examine if users explore most features or only a certain subset, and whether they explore things in the sequence we expect. These data are summarized on a daily basis and used to assess the effectiveness of any changes to the content and interaction structure.

Experimental testing

Design experiments examined the effectiveness of the shared reality workflow in improving system performance by testing two different interactive displays of media on the installation. The displays differed in their representation of solar data, the font style and size, the icons instructing users, and the translation of solar data from kWh/m² to a visual representing the number of homes potentially powered by the sun (Figure 6). User point cloud data were analyzed through observation in VR and through derived data for a period of 1 month during regular museum hours (Wednesday through Sunday 10:00 a.m. to 05:00 p.m.). For each iteration, experiments examined (1) observations in VR of user point clouds, including gestural tendencies and impeding factors; (2) the number of visitors who interacted with the installation and their interaction time; and (3) factors affecting frame rate and the minimum frames per second (fps) acceptable for interaction.

Figure 6.

Screenshots in vvvv of two different iterations of displayed media content on the interactive installation.

Observations in VR of user point clouds

For both iterations, observations of recorded user point clouds revealed interferences with the gestural interactions of multiple users and limitations to the physical setup and sensors. User discrimination is a balance between ignoring gaps in the point cloud so that things such as arms are included with a body and to being able to discriminate separate bodies that are close to each other. After extensive field use, we determined that roughly a quarter-meter is a good inclusion distance for our point cloud densities and typical sensor occlusion gaps. Observations also demonstrated that the placement of sensors in our installation has stable point cloud generation at roughly 4 m out. We adapted the setup so that users are typically 3 m from the sensors for best results, yet they can point at areas ranging from a half-meter to 4 m. Other limiting factors of the sensors dealt with the color of user clothing. Since the sensors use infrared (IR) for tracking, and some dark clothing includes dyes that absorb IR, then those parts of a person covered in dark clothing would produce no point cloud points, which interferes with user discrimination. Another insight observed from the point cloud data on the gestural interactions was the benefits of one-handed interaction versus two-handed. Many people were physically carrying things when they interacted with the installation (i.e. children, books, bags, cups, etc.), and one-handed interaction allowed them to experience it more fluidly. Another observation dealt with the way people were treating the interface like a smart phone while oblivious to on-screen instructions (Figure 7). By reviewing the activity logs and point cloud data for the first iteration and by adjusting the design interface, the use (per-user engagement time) of the installation tripled as a result.

Figure 7.

Snapshots of recorded user point clouds illustrating various gestural movements with the interactive platform. User number 28 represented by pink point clouds (top row) is attempting a one-handed gesture. User number 31 represented by purple point clouds (bottom row) is using both one- and two-handed interactions.

Analysis of user interactions

Preliminary experiments tested the interaction patterns, which included quantifying the number of visitors who interacted with the installation each day for 1 month, as well as their maximum and average interaction times. Over the course of 1 month, 1516 visitors who passed by the installation actually interacted with it. An interaction was indicated by someone who pointed at something on the installation for at least 15 s. Activity logs show that 30%–40% of the people walking by engaged for at least 15 s, with averages consistently near a minute (Figure 8). The data also reflect the museum schedule and patterns of occupation, for example, when it was closed on Mondays and Tuesdays or had its peak day on Saturdays.

Figure 8.

Activity log analysis of user point cloud data for 1 month showing number of visitors who interacted with the installation and their maximum and average interaction times.

Observation of the typical interaction sequencing was for users to approach the installation, look at the city model, look at the screen, and then start to point as directed. As the act of raising their hand caused a reaction on the model before they have fully pointed up at the screen to make a data selection, many users just kept pointing around the model without making a selection first. Some users interacted with the installation in the sequence that was intended. The entire discovery process typically took users less than 20–30 s. Other factors impacting the ease of interaction were related to graphics such as icons versus textual directions, the hierarchy and style of the text on the vertical media screen, and the graphic representation of solar energy. Users rarely read textual instructions and simple pictograms worked better than detailed illustrations to be best understood at a glance. We modified the media content to include both short text and action pictograms to guide users through the interaction.

Evaluation of system performance

Factors affecting frame rate were also examined to evaluate system performance. Our target frame rate is 60 fps, which is the display rate of the projectors in the installation. In practice, our frame rate varied from 60 fps with a single user down to 35 fps with five to six users, which is the maximum that can be discriminated as individuals located in the given space. With both iterations, 35 fps was still acceptable for interaction. Responsiveness suffered below 30 fps, as the increased overall system latency, which is typically three to four times the frame rate, created a noticeable action-response disconnect. Even though vvvv provides a full graph evaluation for each frame, the actual frame rate was dependent on the amount of processing that needs to occur within each frame. In our interactive setting, that changed with the actions and number of the users and the complexity of the resulting data that were presented. Analysis also showed that even though user point cloud processing is handled in its own process to decouple it from the display process frame rate, an increase in points per-user point cloud did require more time to process. However, through our optimized point cloud recognition method, the processing time scaled more closely to linearly rather than geometrically with increasing number of points.

Discussion

Advantages of the shared realities workflow

The actual user point cloud data observed and recorded in the context of use through this shared reality workflow produced many useful insights that cannot easily be obtained by conventional means. Experimental testing demonstrated that the workflow supported the evaluation and iterative design for an interactive media installation in several ways. First, it enabled remote observation of recorded and real-time user interaction behaviors through the VR method. This was important since users were not influenced by our presence and thus interacted candidly in the actual context of use. Second, working remotely in VR reduced time and resources by making the observation schedule flexible and ongoing. This allowed us to take note of particular times when user engagement was high and potentially straining the user interface. Third, the recordings of anonymous point cloud data protected users’ privacy. Finally, working in VR allowed our entire design team to observe and playback the identical user interactions from either the same or different vantage points, creating opportunities for new insights through different perspectives of user observations (Figure 9).

Figure 9.

Screenshots of a video recording the designer/developer wearing the VR headset and viewing recorded user point clouds from multiple vantage points in the VR design space.

Ultimately, within the shared realities workflow, independent but associated physical, digital, and virtual environments build upon one another to generate an intertwined condition that is occupiable by both users and designers across multiple realities and scales (Figure 10). In this way, it reconceives Nigel Cross’ concept of design participation by redefining the threshold between “designable” and “occupiable” space.

Figure 10.

Diagram of the interconnected digital–physical condition that is occupiable by the designer and user in different realities at multiple scales.

Limitations to the methodology

By utilizing the shared realities workflow, challenges with cost, physical constraints, and ethical issues related to surveillance should all be considered. Upfront costs for sensing equipment remain high,³⁶ and the installation of the 3D depth sensing cameras and computing equipment for acquiring point cloud data in situ requires physical infrastructural support, intermittent access, and continued maintenance. Furthermore, within our particular installation, the designer has control over continuously modifying the digital content but not the physical components of the model. Changes to the physical model or prototype would require the system to be temporarily paused and inaccessible for the addition of new physical content or model repairs. However, once the system is back up and running, data collection can easily resume for feedback on the modified system. These physical constraints do not necessarily apply to all interactive systems, for example, where a physical prototype contains mechanically moving components actuated through sensors and controls. The algorithms driving the physical system behaviors could still be modified in VR, which would enable remote virtual access to modify the system in its context of use and observe user responses via point cloud as intended by the shared realities workflow.

There are also ethical and security implications to this methodology that require careful consideration, especially when implementing such systems in public space. Although an individual user’s identity remains anonymous, this detection and response system raises questions about surveillance with interactive systems relative to point cloud resolution and recorded behaviors. Ensuring that interactive architecture and its monitoring systems develop in an ethically and legally responsible manner requires greater reflection on the values embedded within system design.⁸ This is where the convergence of architecture, user-centered design, and interaction design becomes critical.

Future work

Directions for future work focus on continued monthly reporting and analysis of system performance as new media content and interaction sequences are introduced to the installation. Monthly reports of interactivity moving forward will indicate the effect of specific changes, as well as records of how many people select certain data sets, which data sets users selected, how long it took users to make a selection, how it changed throughout their time interacting, and how the presence of other users affected the time they spent interacting. Continuous automated reporting using the shared realities will be beneficial for comparing exhibit metrics against the museum’s house attendance levels, the types of groups visiting at different times, and the effect of changes to nearby exhibits or traffic flow. The introduction of new media content can include data ranging from wind flow, building type, green infrastructure, traffic patterns, and other mapping overlays that can leverage GIS tools and simulations (Figure 11). Next steps also involve increasing the number of sensors in order to broaden the interaction area and involve more than five to six users at a time. This will require further testing of the workflow to examine its ability to incorporate point cloud recognition for larger groups and more complex actions at larger scales. In expanding the workflow to other applications, future areas for testing the potential and limits of the workflow could include responsive facade systems, interactive interior environments, and environmental control systems that typically require a human factor testing phase. For interactive systems—especially those that are already leveraging occupant sensing technologies—the ability of the designer/programmer to view user interaction behaviors through the shared realities VR experience could lead to insights that might otherwise be overlooked.

Figure 11.

Examples of spatial and temporal data sets being projected onto a 3D physical model prototype of the interactive urban design platform.

Conclusion

This article described the development and testing of a computational design workflow called “shared realities” for incorporating real-time observations and recordings of user interactions with an interactive museum installation. It described methods for integrating 3D depth sensor data of user point clouds and VR devices into a vvvv programming environment for gestural interaction and iterative design and testing of the installation in the context of use. A process was developed to allow the designer/developer to work directly and continuously in VR with the ability to make changes to the interaction process and observe and record user responses in the form of 3D point cloud data. Design experiments tested the shared realities workflow for iterative design of the interface, user point cloud observations, and system performance assessment. Experimental testing demonstrates the effectiveness as well as the challenges of the workflow in providing an iterative design environment for improving the system and its user experience. Contributions include new processes to integrate 3D depth sensing with VR, and new methods for data collection on user interaction behaviors that address current issues in the field of interactive architecture. The work leverages computational, sensing, and visualization technologies to generate new workflows between designers, programmers, and the users of interactive systems.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported in part by the New York State Department of Economic Development through the Syracuse Center of Excellence in Environmental and Energy Systems (SyracuseCoE), the Syracuse University School of Architecture, and the Milton J. Rubenstein Museum of Science and Technology (MoST).

ORCID iD

Bess Krietemeyer

References

Vardouli

. Who designs?—Technological mediation in design participation. In: Bihanic

(ed.) User empowerment: interdisciplinary studies and combined approaches for technological products and services. London: Springer-Verlag, 2015, pp. 13–41.

Cross

. Preface. In: Cross

(ed.) Design participation: proceedings of the design research society’s conference, Manchester, September 1971. London: Academy Editions, 1972, p. 6.

Wiener

. The human use of human beings: cybernetics and society. New York: Avon Books, 1967.

Pask

. The architectural relevance of cybernetics. Archit Design 1969; 39: 494–496.

Weiser

. The computer for the 21st century. Sci Am 1991; 1(1): 19–25.

Pickering

. Ontological theatre Gordon Pask, cybernetics, and the arts. Cybern Hum Knowing 2007; 14(4): 43–57.

Matthews

. The fun palace at fifty. Art Am 2016; 104(9): 114–121.

Urquhart

Schnädelbach

Jäger

. Adaptive architecture: regulating human building interaction. Int Rev Law Comput Tech 2019; 33(1): 3–33.

Sparrman

Matthews

Kernizan

et al . Large-scale lightweight transformable structures. In: Proceedings of the 37th annual conference of the association for computer aided design in architecture (ACADIA) ISBN 978-0-692-96506-1, Cambridge, MA, USA, 2–4 November 2017, pp. 572–581.

10.

Krietemeyer

Andow

Dyson

. A computational design framework supporting human interaction with environmentally-responsive building envelopes. Int J Architect Comput 2015; 13(1): 1–24.

11.

Schnädelbach

Irune

Kirk

et al . ExoBuilding: physiologically driven adaptive architecture. ACM T Comput: Hum Int 2012; 19(4): 1–22.

12.

Eng

Babler

Bernardet

et al . Ada—intelligent space: an artificial creature for the SwissExpo.02. In: Proceedings of the 2003 IEEE international conference on robotics and automation (ICRA’2003), Taipei, Taiwan, 14–19 September 2003.

13.

Beesley

. Near-living architecture. Toronto, ON, Canada: Riverside Architectural Press, 2014.

14.

Alonso

Zhang

Grignard

et al . CityScope: a data-driven interactive simulation tool for urban design. Use case Volpe. In: Morales

Gershenson

Braha

et al . (eds) Unifying themes in complex systems IX. Cham: Springer, 2018, pp. 253–261.

15.

Abras

Maloney-Krichmar

Preece

. User-centered design. In: Bainbridge

(ed.) Encyclopedia of human-computer interaction. Thousand Oaks, CA: SAGE, 2004, pp. 763–768.

16.

Jia

Srinivasan

Raheem

. From occupancy to occupant behavior: an analytical survey of data acquisition technologies, modeling methodologies and simulation coupling mechanisms for building energy efficiency. Renew Sust Energ Rev 2017; 68: 525–540.

17.

Peters

. SuperSpace: computing human-centric architecture. In: Peters

Peters

(eds) Computing the environment: digital design tools for simulation and visualisation of sustainable architecture. Chichester: John Wiley & Sons, 2018, pp. 192–200.

18.

Hansen

. Designing responsive environments through User Experience research. Int J Architect Comput 2016; 14(4): 372–385.

19.

Hong

Taylor-Lange

D’Oca

et al . Advances in research and applications of energy-related occupant behavior in buildings. Energ Buildings 2016; 116: 694–702.

20.

Hong

Langevin

Sun

. Building simulation: ten challenges. Build Simul 2018; 11(5): 871–898.

21.

FLEXLAB, https://flexlab.lbl.gov (accessed 1 April 2019).

22.

Jamrozik

Ramos

Zhao

et al . A novel methodology to realistically monitor office occupant reactions and environmental conditions using a living lab. Build Environ 2018; 130: 190–199.

23.

Negroponte

. Aspects of living in an architecture machine. In: Cross

(ed.) Design participation: proceedings of the design research society’s conference, Manchester, September 1971. London: Academy Editions, 1972, pp. 63–67.

24.

vvvv—a multipurpose toolkit, https://vvvv.org (accessed 1 April 2019).

25.

Krietemeyer

Bartosh

Covington

. Shared realities: a method for adaptive design incorporating real-time user feedback using virtual reality and 3D depth-sensing systems. In: Proceedings of the 37th annual conference of the association for computer aided design in architecture (ACADIA), Cambridge, MA, 2–4 November 2017.

26.

Diraco

Leone

Siciliano

. People occupancy detection and profiling with 3D depth sensors for building energy management. Energ Buildings 2015; 92: 246–266.

27.

Whelan

Kaess

Fallon

et al . Kintinuous: spatially extended kinectfusion. Technical report, Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, July 2012.

28.

Lesniak

Terpenny

Tucker

et al . Immersive distributed design through real-time capture, translation, and rendering of three-dimensional mesh data. J Comput Inf Sci Eng 2017; 17(3): 031010.

29.

Yamu

Poplin

Devisch

et al . The virtual and the real in planning and urban design: perspectives, practices and applications. New York: Routledge, 2017.

30.

Schneider

Kuliga

Holscher

et al . Educating architecture students to design buildings from the inside out: experiences from a research-based design studio. In: Proceedings of the 9th international space syntax symposium, Seoul, South Korea, 31 October–3 November 2013.

31.

Heydarian

Becerik-Gerber

. Use of immersive virtual environments for occupant behaviour monitoring and data collection. J Build Perform Simu 2017; 10(5–6): 484–498.

32.

Drettakis

Roussou

Reche

et al . Design and evaluation of a real-world virtual environment for architecture and urban planning. Presence: Teleop Virt 2007; 16(3): 318–332.

33.

Kuliga

Thrash

Dalton

et al . Virtual reality as an empirical research tool—exploring user experience in a real building and a corresponding virtual model. Comput Environ Urban 2015; 54: 363–375.

34.

vvvv.OpenVR, https://vvvv.org/contribution/vvvv.openvr (accessed 1 April 2019).

35.

Screen-capture-recorder-to-video-windows-free, https://github.com/rdp/screen-capture-recorder-to-video-windows-free (accessed 1 April 2019).

36.

US Department of Energy. Wireless occupancy sensors for lighting controls: an applications guide for federal facility managers. Washington, DC: US Department of Energy, 2015.