Abstract
Time with users is a precious commodity that has become increasingly difficult to come by, making it difficult to always employ traditional user study methods. Such methods also run into challenges in sensitive industries, where it is not always possible to openly engage with end-users. These barriers jeopardize good design, and limited user access seeds temptation to bypass user engagement altogether. The primary goal of this work is to share techniques and lessons learned for successful user engagement in the face of limited access to end-users. Three methods are presented here, usage of subject matter experts (SMEs), analogous use cases, and surrogate users, along with key successes and challenges for each. The lessons, limitations, and suggestions presented here reflect work performed across multiple projects over the past several years, where we have utilized combinations of these techniques for conducting user-centered design in the absence of regular direct user engagement.
Introduction
As human factors practitioners, time with users is our most precious commodity. In recent years, this invaluable resource has become increasingly difficult to come by as industries continue to adapt to changes like increasingly distributed workforces, remote work, and flex time scheduling. Research has shown that distributed work is here to stay. While remote work has decreased slightly from its peak in 2020, 76% of workers with jobs that can be done remotely are working from home at least some of the time with nearly half of that being full-time remote (Parker, 2023). Another survey found that 74% of workers would prefer to work from home at least 2 days a week, and only 17% would prefer to work full-time in office (Punjwani & Campbell, 2024). As this new work paradigm continues to evolve, securing consistent and adequate time with end-users to carry out traditional methods of user engagement will continue to be a challenge.
Traditional user studies also face elevated challenges in industries like healthcare, where the confidential aspects of work become an additional hurdle to overcome before any understanding of user needs can be obtained. In other industries like defense, decision makers have historically not always made end-users available to developers, though there are recent pushes attempting to address this with calls for close coordination with users (U.S. Army Public Affairs, 2024). Barriers like these are just some of the many that increasingly prevalent challenges to accessing end-users. This lack of access jeopardizes good design practices, where limited access to user communities can seed the temptation to bypass user engagement altogether.
Objective
The primary goal of the work presented here is to share techniques and lessons learned for successful user engagement as work paradigms continue to shift. User engagement goes far beyond user studies, and valuable insights can be found even in the most casual of user interactions. The methods and lessons presented here reflect work performed across multiple projects over the past several years, where we have utilized combinations of these techniques for conducting user-centered design in the absence of traditional direct user engagement.
Despite the many benefits, these techniques are not a panacea. They present challenges of their own, and they are not a replacement for genuine user engagement. If outcomes are not contextualized, it can be easy to garner a false sense of user understanding that inaccurately guides decision-making and leads a project astray. Through the case study presented here, these pitfalls can be better understood and avoided while successfully employing these techniques. It is our hope that this work may serve as a practical guide for human factors practitioners navigating the unfortunately increasingly common occurrence of limited user availability that comes from the realities of modern-day work.
Specific aims of this industry case study include: (a) enumerate methodologies and techniques for maintaining user-centered design despite various forms of restricted user access, (b) key successes and challenges of employing these methods through the lens of multiple research and development projects, and (c) practical guidance and recommendation for how to best utilize these methods while avoiding common risks.
Methods
A case study of projects with limited user access in the defense industry are utilized to guide usage of methodologies and techniques for navigating limited access to users. Techniques analyzed over the course of this study include utilizing integrating subject matter experts (SMEs) into project teams, usage of analogous user groups, and surrogate user evaluation. Many of these techniques hold true value on their own, but their real strength lies in employing combinations of them to provide a team with a critical mass of accurate, albeit bite sized, user information. The work presented here was performed by teams of 5 to 6 interdisciplinary members including for example, human factors engineers, software engineers, data scientists, and user experience designers. Projects also frequently had multiple stakeholders, where the end-users were not the final decision makers, further complicating the design process.
Subject Matter Experts
Subject Matter Experts (SMEs) are often the obvious choice for replacement for end-users themselves. They once were end-users, completely understand the use case, and have an abundance of time to work alongside the project team. However, while they may provide a good proxy for end-users, researchers must carefully consider both the past and present experiences of the individual. Some may still professionally exist within the world of the end-user, while others may have previously performed that work and now consult full time. In the case of the latter, it is vital to consider whether tasks and work environments have undergone changes since a SME was a truly representative user.
Analogous User Groups
Human factors practitioners are not the only project team members that are affected by limited end-user engagements. Often project teams do not have access to relevant data sets to develop algorithms and novel systems, which makes system development challenging for all involved in the project, not just UX/UI engineers. Multiple project teams at Aptima have solved this problem by exploring analogous use cases and using those to develop the prototype system. These use cases have different user groups that can be compared to the end-users and used as substitutes. This can be a benefit, as these analogous user groups are more readily accessible to the project team than the eventual true end-users.
Across disparate domains, many users suffer from the same sorts of challenges. Solutions developed in one domain are often applicable across multiple domains. Often the users in these different domains have similar challenges as one another and are more alike than they are different. Recognizing adjacent disciplines that face similar challenges can open the door for much more readily available user groups and data.
Surrogate Users
A final method we have found valuable is tasking certain team members to act as surrogate users throughout a project in the intervals between opportunities for user engagement. While on the surface this may seem no different than simply developing for stretches without access to end-users, our experience has found that it does offer tangible benefits to the project team. Moreover, we have found that human factors practitioners are ideal to step into this role.
This is best employed by first capitalizing on whatever limited time actual users can offer early in a design project. The surrogate practitioner should be heavily involved in carrying out typical knowledge elicitation and cognitive engineering exercises to best understand the user and their work role, environment, competencies, needs, and requirements. Equipped with this experience and knowledge, a human factors practitioner can then act in the stead of users during feedback sessions with other members of the project team. As additional opportunities arise to engage with end-users, the surrogate practitioner should maximize their time with users to further develop their knowledge and ensure that their representation remains accurate.
Results
Retrospective analysis of projects that each utilized one of these three techniques for addressing limited user engagement found that each method presented unique combinations of key successes and key challenges.
Subject Matter Experts
Key Successes
End-users may have different perspectives toward project teams. Some users may feel indifferent or positive, while others may have a strong negative or distrustful attitude. These attitudes may come from previous experiences when their needs were not properly considered with new technology they were forced to work on. Their perception is that engineers and product designers don’t understand them and don’t care to. In cases where users are indifferent, job-specific language can cause barriers and misunderstanding between researchers and users. SMEs can bridge these gaps by building trust with end-users by having “one of their own” involved including advocating for them with the project team, and serving as a translator between the end-users and project team to ensure true intent is captured. SMEs make end-user engagements more successful and more efficient through interpretation of end-user action and language while making end-users more comfortable and willing to share.
SMEs are also most useful when the project team is in the early stages of building understanding of a new user including their work activities and environment. Since end-users are often so focused on their day-to-day, they can have a hard time explaining their work to someone that doesn’t have any background. Meanwhile, SMEs should be practiced in explaining their work and can provide the base knowledge of work activities, environment, and user mindsets so the project team can focus on more advanced and specific topics with the end-users.
Key Challenges
As SMEs become more involved with the project team, the line between serving as a resource and as a designer can become very blurred. It is human nature to become partial to something that a lot of personal effort was given. In these situations, it is difficult for the SME to give impartial feedback as to how an end-user would likely react to the software. For example, throughout a project designing cognitive assistants for drone pilots, SMEs were involved in the design of features for explaining how complex models work through node graphs. They insisted that end-users must be able to visually judge how well the model was able to represent the situation. However, during user testing it was found that current users were much more comfortable with the idea of utilizing models to aid decision-making and would rather understand what information the model had access to, to better understand if it was missing any essential information, for example from a modified document.
Analogous User Groups
Key Successes
Early recognition of an analogous user group can quickly open doors to accelerate a project. In our own experience, this method can be particularly beneficial when a project’s use case is connected to sensitive or classified data.
In some cases, we have even found that our own staff are an appropriate analogous user group. This naturally provides very consistent access to a user group and immediate access to relevant data. Coworkers can be interviewed at ease to develop user-centered artifacts, including personas and journey maps. These comparable users and the artefacts generated by them have granted software engineers a user-focused lens to develop their prototypes. Similarly, UI/UX designers can draw heavily from the personas and journey maps that were created through interactions with the stand-in user group.
Key Challenges
Even if a system can be developed using an analogous use case with a parallel dataset, the project teams should still engage in user-centered design practices. As the project matures, it is important for the team to analyze how their analogous use case may differ from the intended end-user group. Following this, adjustments to the system’s design must be carried out accordingly.
Surrogate Users
Key Successes
The active mindset brought on by this method is where it really succeeds over designing without an answer for limited user engagement. It inherently forces the human factors practitioner, and the rest of a project team through their surrogacy, to consciously consider things from the perspective of the end-user. This deliberate shift to an active mindset opens the door for more candid responses toward the work of other project team members, resulting in tighter feedback loops more in tune with the needs of the user. The greater the familiarity the researcher has with the intended end-user, the greater success this method can offer. That is inclusive of both utilizing successive engagements to increase knowledge as well as gradual domain expertise accrued over the course of multiple projects or even a career. At the highest levels, this begins to resemble the benefits afforded to a project from SMEs as described previously.
In our own experience, this method was used successfully over the course of a 2-year research and development project where user engagement was limited to a few times per year. Users were physically far from project team, and the urgent nature of their daily jobs made scheduling calls near impossible. For the time between engagements, the team’s human factors engineer performed the role of being a surrogate user to great effect. During design sprints their feedback was provided not from their own background but the perspective of the end-users, built on their own understanding of their needs through those limited engagements. This proved to be especially beneficial in this instance because the development team had minimal experience with the project’s use case.
Key Challenges
This method places a heavy burden on the practitioner who takes on the role of the surrogate user. Firstly, in a limited amount of time they must adequately familiarize themselves with mindset of the end-users and their use case. Additionally, they must be comfortable providing feedback. They must not only have a willingness to give strong feedback to teammates, but also an ability to do so through the lens of the user and not from their own human factors background.
Discussion
The benefits and potential hurdles associated with each of these techniques warrants careful consideration when deciding which to employ in the face of limited user access. Figure 1 showcases a decision tree that human factors practitioners can utilize to decide which methods are best suited for their project. It’s important to start with considering how much user access is available. If there is likely no direct interaction, surrogate users and analogous use cases help project teams to get started while SMEs can be brought on later to bridge gaps to the real use case. With more access, SMEs are a great choice to include the voice of the user during design where choices can be validated with user studies. Finally, when users are more available, SMEs can provide the project team with the basics of the user’s environment while serving as an interpreter during engagements.
Method selection decision tree for limited user access.
Subject Matter Experts
SME support requires access to either users or secondary users close to the task/environment to mitigate assumptions. Another strategy for mitigating this includes utilizing multiple SMEs, but separating project duties so one can be engaged during development while another provides an impartial source of testing feedback. Also focusing on SME input for gathering a general background and understanding of the user is a best practice to avoid over-indexing on perspectives that may have become dated. Finding a SME that still interacts with and has buy-in with the target users helps to provide the most value.
Analogous User Groups
This method was found to provide a strong level of confidence across the team in successful system development, particularly when considering the end results that would come from a project with no user-centered artefacts. Further this method is invaluable for developing a demonstration so that stakeholders may better understand how the system is intended to work. Bridging the gap to the intended environment from the analogous setting is often much easier than from a theoretical perspective. However, design decisions made in another context may not transfer to the intended environment. Building hardware and software as flexible as possible provides mitigation to pivot in these situations.
Surrogate Users
Even more so than with the other methods discussed in this paper, it is critical to consider what elements of feedback are not representative of actual end-users. It is a necessity to ensure that any assumptions which have been made on behalf of the user are confirmed at the earliest opportunity to avoid extended effort being put toward a design which will ultimately fail. If there is absolutely no opportunity to engage with end-users, this could be a risky method to employ. Some mitigation could also come from utilization of subject matter experts for intermittent validation, however the same risks of over-indexing on opinion outside of the end-users still apply.
Conclusion
While no combination of modified techniques can replace genuine and substantial time with end-users, the techniques discussed here were found to hold real value. While they cannot always provide the richness and granularity of full user studies, their outcomes vastly outperform the alternatives of nonexistent or aimless user engagement. They do come with limitations though which must be kept in mind for successful results.
Like any research, over-indexing a specific sample by obtaining perspectives from the same SMEs or over-specified secondary user research presents a real risk to human factors practitioners. Utilizing the user-centered techniques and methodologies discussed here can help super-charge the value of limited user engagement when properly staged so that any user sampling is representative of the broader user population.
Implementing these adapted user engagement practices has greatly benefited our project teams, and we have found that using an ala carte approach offers both flexibility and value. Based on our findings, we will continue to employ them. Finally, we hope that concretely sharing these techniques through a case study will aid human factors practitioners to design better solutions in the face of limited user access.
Footnotes
Acknowledgements
The authors would like to acknowledge their Aptima teammates and project partners for their contributions to the larger research and development efforts discussed in this case study.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This material is based upon works supported by the U.S. Army C5ISR Center under Contract No W56KGY-21-C-0014, U.S. Air Force Research Lab (AFRL) under Contract No FA864921P1676, and the Missile Defense Agency (MDA) under Contract No. HQ0860-23-C-7506. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the U.S. Army C5ISR Center, U.S. AFRL, or MDA.