Evaluating behavioral intervention technologies: Integrating human-centered design and implementation science outcomes

Abstract

Background

Behavioral intervention technologies (BITs) offer scalable, cost-effective healthcare solutions but often show reduced impacts in community settings and are rarely sustained. Human-centered design (HCD) enhances usability by tailoring BITs to stakeholder needs, while implementation science (IS) identifies contextual barriers and strategies to promote uptake and sustainability. Both HCD and IS aim to improve BIT usability and implementation through iterative, user-focused processes but are rarely integrated.

Objective

We introduced the user-centered and sustainable implementation science (USIS) model, a novel and systematic framework that combines HCD and IS principles to enhance BIT effectiveness and sustainability. We aimed to (1) synthesize the HCD and IS outcomes and integrate them into a USIS framework; (2) apply USIS to a case study: the sleep shared-management intervention for children with juvenile idiopathic arthritis and their parents (SLEEPSMART).

Methods

We conducted a narrative literature review on HCD and IS outcomes for BITs from the PubMed, CINAHL, and Web of Science databases. Articles were selected based on their focus on usability, implementation outcomes, and evidence-based healthcare practices. This synthesis informed the development of the USIS model.

Results

The USIS model incorporates five domains: (1) user-centeredness (empathy, engagement, and equity), (2) efficiency (cost, timeliness, and rapidity), (3) feasibility (learnability, memorability, error reduction, and low cognitive load), (4) satisfaction (acceptability and appropriateness), and (5) fidelity (adoption, penetration, and sustainability). We applied the USIS model to evaluate the SLEEPSMART project to demonstrate the application of the USIS model in guiding BIT design and refinement. Insights from the review informed the design principles applied during the development and testing stages of SLEEPSMART.

Conclusion

The USIS model serves as a practical and theoretical guide to improve BIT design and evaluation. By emphasizing implementation considerations early and centering user needs, USIS provides a pathway to bridge HCD and IS approaches to enhance the real-world impact and sustainability of digital health innovations.

Keywords

Behavioral intervention technologies human-centered design implementation science usability digital health

Introduction

Behavioral intervention technologies (BITs) refer to evidence-based behavioral and psychological interventions delivered via various technology media, including smartphone applications, computer programs, virtual reality, wearable technology, and electronic messaging.^1,2 BITs use behavioral principles like reinforcement, self-regulation, and reward to improve accessibility, engagement, and adherence to health interventions.^2–8 Despite the growing use of BITs across health disciplines, many interventions fail to achieve sustained use or impact in real-world contexts. Usability issues, such as excessive cognitive demands, lack of cultural or contextual alignment, and low adaptability, often limit BITs from being widely adopted and maintained in the long term.^4,9,10 These challenges contribute to a persistent research-practice gap in the digital health field.

Human-centered design (HCD)

HCD is an increasingly recognized approach that can address these implementation challenges early in the development stage by incorporating the stakeholders’ needs and learning styles and optimizing the design features. HCD is an iterative and non-linear design process that makes the services highly usable by centering users’ needs.¹⁰ The HCD process includes five stages: (1) Empathize, also referred to as user research, where designers and researchers conduct comprehensive observation, communication, and investigation with users on their needs, behaviors, and habits. (2) Define, based on the information gathering in the first stage, this is, where designers synthesize findings, define users’ problems, and formulate the problem statement. (3) Ideate, with a solid understanding of the users and a clear problem statement. This phase generates creative solutions by thinking outside the box. (4) Prototype is the transition from ideas into tangible actions tested on real users. A prototype can be low fidelity (e.g., a sketch) or high fidelity (e.g., a model). (5) Test, real users will try the prototype and report their perceptions of what works and what needs to change/improve to function better. The testing results often inform the designers and researchers whether they need to return to the first stage and undergo several rounds of ideation before testing again. The “trial-and-error” application is unique to HCD, with each iteration resulting in a more usable, contextually appropriate, and innovative solutions.^11,12 One ultimate goal of HCD is to achieve the usability of the solution. Usability was initially defined only as “ease of use”¹³ and was modified to “usefulness, ease of learning, and satisfaction.”¹⁴ Usability considers the extent to which the users can adopt a product to reach their goals with effectiveness, efficiency, and satisfaction.^15,16 Emerging studies have paid more attention to BITs’ design characteristics, which strongly influence interventions’ adoption (i.e., the extent to which an intervention is used by end users), implementation, and sustainability in healthcare settings.^17–22 For example, the design problems that affect implementation success could be too complicated to use (i.e., low learnability and memorability), too challenging to understand and navigate (i.e., high cognitive burden and low flexibility), and not fit for delivery context (i.e., not considering context factors).^2,4,23

Implementation science (IS)

Even though HCD can enhance content and functionality, there is still a challenge in implementing BIT in real-world situations in terms of adoption and sustainability. Thus, IS can be a powerful tool to fill the gap. IS helps researchers understand barriers (what hinders the implementation) and facilitators (what supports the implementation) across multiple levels of context and determinants, develop strategies that overcome obstacles, and enhance the uptake and sustainability of innovations in real-world settings.²⁴

How HCD and IS work together

HCD and IS contribute to enhancing BITs' effectiveness differently and complement each other. The models share common goals to improve intervention usability and implementation by applying iterative and contextual design processes, engaging stakeholders as design partner, involving a multidisciplinary design team, and conducting iterative testing and evaluation as early and frequently as possible. Although both HCD and IS aim to improve the effectiveness of interventions, they have traditionally operated as distinct disciplines. HCD primarily considers usability by ensuring BITs are tailored to user needs, with the rationale that a well-designed and compelling innovation is much more likely to be adopted and sustained.^23,25,26 In contrast, IS focuses mainly on identifying barriers and facilitators to promote BIT uptake and sustainability.^24,26,27 Integrating HCD and IS in designing, developing, and evaluating BITs offers complementary approaches to improving BIT effectiveness by incorporating iterative, user-focused design with evidence-based implementation strategies. However, few models offer a concrete, unified approach to guide such integration in the development and evaluation of BITs.

Overview of research

To address this gap, we introduce the USIS (user-centered and sustainable implementation science) model, a comprehensive framework that brings together HCD and IS principles to guide the full lifecycle of BITs. By synthesizing the core outcomes of both domains, the USIS model provides a comprehensive structure for improving the effectiveness, adoption, and sustainability of BITs. To develop the model, we will (1) synthesize the existing HCD and IS outcomes and integrate them into the USIS framework for guiding BIT design, development, and testing. (2) Then, to demonstrate the application of the USIS model we will utilize a case study—SLEEPSMART, a sleep self-management intervention developed for children with juvenile idiopathic arthritis (JIA) and their parents. By applying the USIS model to the SLEEPSMART project, we will show how it systematically evaluates the intervention across its design, development, and testing phases. It provides both a conceptual foundation and a practical roadmap for researchers and developers creating BITs that are not only user-centered but also sustainable and impactful in real-world health systems.

Methods

Search strategy

We conducted a narrative literature review based on the narrative review methodology.^28–30 The narrative review approach was chosen for its flexibility, which allows for synthesizing diverse theoretical frameworks and empirical outcomes across interdisciplinary fields. Unlike systematic review, which focuses on narrower defined questions, narrative review is particularly suited for addressing broad and exploratory research questions. In collaboration with a social science librarian, we searched and reviewed the existing literature from the PubMed, CINAHL, and Web of Science databases centered on usability constructs and implementation outcomes on BITs with date limits to peer-reviewed English-language articles published between January 2000 and January 2023. Article inclusion criteria are as follows: (1) articles focused on behavioral intervention technologies in healthcare settings; (2) studies that reported usability and/or implementation outcomes; and (3) interventions aligned with evidence-based health practices. We defined evidence-based practices as interventions, services, and policies informed by the best available evidence and designed to improve patient outcomes, care quality, and healthcare delivery efficiency. This rationale was reflected by the inclusion of terms such as “Evidence-Based Practice” [MeSH], “digital health,” and “intervention.” These terms ensured that our review captured studies focused on designing and implementing interventions within healthcare settings that adhered to evidence-based methodologies. Articles were excluded if they did not include conceptualization of usability or if they described implementation strategies to improve specific outcomes on products outside the healthcare field. The search strategy included combinations of terms such as “usability,” “human-centered design,” “implementation science,” and “evidence-based practice” to capture studies focusing on BITs in clinical contexts. The full search strategy with filters and limits for each database can be found in the Supplemental Material.

Study selection

Two authors (SZ and YW) utilized two steps of data extraction. The first step was title and abstract review, and the second step was full-text article review and selection. In each step, two reviewers compared and discussed results until a consensus was reached. The initial search resulted in 128 articles after removing duplicates; 89 articles were removed because they reported on behavioral interventions in the context of health, and 39 articles were included for full-text review. Of the 39 full-text articles, 15 articles were removed because they lacked integration usability or implementation concepts, and 24 full-text articles were included for review. From the included studies, we extracted key HCD and IS outcomes, documented shared principles, and synthesized them into a unified conceptual model—the USIS model. Since this narrative review only summarizes existing research, it did not require an Institutional Review Board (IRB), as no new data is being collected from human subjects.

Case study application

To demonstrate the model's practical application, we evaluated a digital sleep intervention—SLEEPSMART—using the USIS framework. The SLEEPSMART project followed a full HCD process (discovery, prototyping, usability testing, and pilot), and data were mapped to USIS domains to assess usability, feasibility, and implementation potential. We used deductive coding to analyze feedback and design decisions from each development stage. The coding schema aligned with USIS model constructs and allowed us to track how each outcome domain was addressed in real-world intervention development. Figure 1 shows the development and application workflow of the USIS model.

Figure 1.

The user-centered and sustainable implementation science (USIS) model development and application workflow.

Results

HCD outcomes

HCD serves not only primary users (e.g., patients) but also people who interact with the primary users (e.g. healthcare providers and caregivers).²³ Improving usability maximizes the BIT's design features and optimizes patient outcomes. We found three dominant approaches to conceptualizing usability in past research. Usability was initially conceptualized as learnability, efficiency, memorability, error frequency/severity, and satisfaction.³¹ Later, more factors were identified and added, including functional minimalism, low cognitive load, requiring minimal training, exploitation of the natural constraints, and clear feedback to users.^32,33 In 2016, Lyon and his team redefined and broadened the usability constructs specific to behavioral and psychological science and offered a comprehensive guideline on designing and evaluating the BITs.¹⁶ The seven principles include (1) learnability: users should be able to understand and utilize an intervention easily and quickly; (2) efficiency: users should spend minimal time, cost, and effort in using the intervention to solve their problems; (3) memorability: refers how well the target users can remember the intervention; (4) error reduction: users can be prevented or allowed quick recovery from incorrect clicks to reduce errors in a system; (5) satisfaction: the level of positive feeling/experience after receiving an intervention; (6) low cognitive load: the design only requires users’ minimal mental efforts to process the intervention; and (7) exploit natural constraints: a successful intervention should address the destination context's static properties that limit intervention use. The primary goal of applying HCD in BIT is to improve BIT's usability, which is theorized to be one determinant of implementation outcomes and influence BITs’ adoption and penetration.^34–36

Implementation outcomes

Frameworks in IS can be categorized into three types: process, determinant, and evaluation. The process framework focuses on the steps of implementing an intervention. This framework is most useful for planning and execution of implementation.^32,33 Determinant framework focuses on identifying barriers and facilitators that affect implementation outcomes. This framework is most useful for understanding what factors contribute to the success or failure of the implementation.^37–41 Evaluation framework focuses on assessing the effectiveness and impact of implementation. This type of framework is most useful for measuring implementation outcomes.^39,40,42

We briefly summarized the past work on the framework of IS outcomes. In 2011, Proctor's team was among the first to develop a conceptual model of implementation research that encompasses three distinct but interrelated domains: implementation outcomes (acceptability, adoption, appropriateness, costs, feasibility, fidelity, penetration, and sustainability), service outcomes (efficiency, safety, effectiveness, equity, patient-centeredness, and timeliness) and client outcomes (satisfaction, function, and symptomatology).⁴⁴ Later, Proctor et al.⁴³ further explored and defined eight outcomes for IS that have been broadly adopted. In 2019, Hermes et al.⁴ recharacterized the implementation outcomes specific to BIT study use. Eight BIT implementation outcomes were recharacterized to better meet the unique aspects of BITs and include (1) acceptability: how well the end users will receive it; (2) adoption: the extent to which target users use an intervention; (3) appropriateness: how well the intervention addresses problems; (4) feasibility: the extent to which BITs can be successfully implemented within a context; (5) fidelity: the extent of BITs being delivered as intended; (6) implementation cost: cost associated with implementing BITs; (7) penetration: the degree to which an intervention is integrated and used in a setting; and (8) sustainability: the ability of an intervention to maintain its effects over a long time. The recharacterized outcomes expanded our understanding of factors related to successful implementation and enabled us to identify the areas that require further development.

A conceptual evaluation model in BITs: USIS model

Building on the evolved frameworks of usability and IS outcomes, we introduce the USIS model—a comprehensive framework that integrates HCD and IS principles to guide the development and evaluation of BITs. The USIS model was created through a systematic analysis of existing HCD and IS outcomes, highlighting their overlapping concepts and shared goals. We incorporated additional components to address gaps and enhance their relevance in healthcare settings. By integrating HCD and IS outcomes into a unified framework, the USIS model seeks to improve the effectiveness, adoption, and sustainability of BITs throughout the design, development, and testing processes. The synthesis and categorization process is illustrated in Table 1.

Table 1.

HCD and IS outcomes integration.

HCD and IS outcomes	Key Concept
	We believe user-centeredness is the foundation of HCD and the premise of IS success. We propose empathy, engagement, and equity as three core components after synthesizing literature.
	Time, efforts, and costs associated with the development and implementation of the BIT.
	These factors contribute to how well the intervention can be carried out as designed.
	These factors lead to users’ perception of acceptable, useful, valuable, and helpful for their needs.
	These factors are indicators of the extent to which the protocols deliver core components of interventions as intended.

HCD: human-centered design; IS: implementation science.

We expanded the USIS model to include practical applications in clinical settings. For example, user-centeredness was applied in a behavioral intervention targeting medication adherence by integrating patient interviews during the prototype phase to ensure relevance to diverse populations. Efficiency (e.g., rapidity) was demonstrated through iterative usability testing, which reduced the time required to navigate the intervention. Similarly, fidelity (e.g., adoption and sustainability) was monitored by tracking long-term user engagement through digital logs and follow-up surveys. The USIS model incorporates behavioral principles to enhance intervention effectiveness. For example, habit formation connects to the usability construct of learnability by helping users incorporate the intervention into daily routines. Reinforcement supports engagement by providing real-time feedback and rewards, while self-regulation ensures the low cognitive load required for effective implementation. USIS conceptualizes constructs to reflect key outcomes of BIT's usability and implementation, addresses the missing piece of user-focused outcomes like equity, accessibility, empathy, and engagement, and implementation outcomes such as timeliness and rapidity. Table 2 shows the proposed USIS constructs and definitions.

Table 2.

USIS constructs and definition.

Constructs	Concepts	Definition	Example application
User centeredness: Focusing on the users’ needs throughout the design process is the foundation of HCD and the premise of the success of IS.	Empathy Engagement Equity	Equity: the practice of purposefully involving minoritized or disadvantaged communities throughout a design; it acknowledges that equity happens with intent and focus. It includes teams (users and researchers), content, and implementation. - Team and content: (1) diverse and inclusive users, multidisciplinary researcher team; (2) contextually appropriate and culturally relevant to fit users and their context of the use - Implementation: (1) accessibility for those with disabilities and low digital health literacy; (2) uptake: users who are not computer savvy, do not trust technology, or who are exposed through informal social networks; (3) adherence: intervention usability, literacy burden, access to money, time and coping skills; (4) effectiveness: different groups of people, focus on groups of people, and access. Empathy: the efforts to understand the users without judgment, including their behavior and why, physical and emotional needs, and what is meaningful to them. Engagement: users’ involvement in BIT development to equip, enable, and empower user-centered design decisions	Conducting interviews with minority groups to adapt intervention
Efficiency: Users spend minimal time, cost, and effort in designing/using/implementing the intervention to solve problems.	Cost Timeliness Rapidity	Cost: the cost related to user spending and BIT implementation. Rapidity: Rapid iteration and prototyping are used to test and make improvements quickly. Timeliness: accounts for ongoing iteration, evolving, and fast updates over time.	Iterative testing to reduce navigation errors
Feasibility: The extent to which the intervention can be done or carried out as designed.	Learnability Memorability Error reduction Low cognitive load	Learnability: users can rapidly understand and use BITs. Memorability: users can remember and successfully operate. Error reduction: prevent or rapidly recovery from errors. Low cognitive load: users only need minimal efforts to complete learning and tasks with simplified functions.	Simplified user interface for older adults with low digital literacy
Satisfaction: The level of the intervention fulfills the users’ needs. Users perceive it as highly acceptable, helpful, and valuable for solving problems, including appropriateness and acceptability.	Acceptability Appropriateness	Acceptability: how well the intervention is being received by target populations, and the extent to which the intervention meets the needs of the target populations. Appropriateness: fit and relevance to users’ issues.	Post-session surveys or exit interviews to measure user satisfaction
Fidelity: The extent to which the protocols deliver core components of interventions as intended.	Adoption Penetration Sustainability	Adoption: actual use (e.g., log-in timestamps) of the BIT by users Penetration: the extent of integration of BITs within a context. Sustainability: The extent of BIT's practice and effect is maintained and long-lasting.	Tracking user engagement via app analytics

USIS: user-centered and sustainable implementation science; HCD: human-centered design; IS: implementation science.

Domain 1. User-centeredness: emphasizes prioritizing user needs throughout the design process, which is key to HCD and IS success. We propose that empathy, user engagement, and equity fall under this domain. (1) Empathy is the most critical element in HCD as we understand users by seeing the world through their eyes, listening to their experiences, and setting aside preconceptions, which ensures designs resonate with user needs and preferences.^12,45 (2) User engagement actively involves users in the design process to align with their interests, motivations, and goals. Engaged users contribute to better products, enhanced communication, and sustained use.^46,47 (3) Equity allows the voices of minoritized or disadvantaged communities to directly impact how the solution will address the inequity by purposefully involving them throughout the design process.⁴⁸ The equitable design acknowledges that equity doesn’t happen by chance, but with intent and focus⁴⁸ and it includes diverse and multidisciplinary teams.^46,49 Users need to be diverse and representative of the target user. Carefully identifying representative user needs can correct this bias and enhance product quality.⁵⁰ The content of the intervention needs to be contextually appropriate and culturally relevant.^50–52 Accessibility is for those with disabilities and low digital health literacy. Design with accessibility enables addressing barriers like digital literacy and resource limitations to ensure uptake and adherence.⁵²

Domain 2. Efficiency: reflects that users should spend minimal time, cost, and effort using the intervention to solve their problems. Rapid iteration and prototyping help refine the intervention quickly based on feedback. Timeliness ensures BITs are continually updated and adapted to remain relevant and effective in evolving context.^50,51,53

Domain 3. Feasibility is constructed with learnability, memorability, error reduction, and low cognitive load. Feasibility focuses on how easily users can learn, remember, and effectively use an intervention. User-friendly designs with low complexity enhance learnability, reduce errors, and minimize cognitive effort. These features make BITs more practical and increase the likelihood of adoption among diverse user groups.²⁵

Domain 4. Satisfaction refers to users perceiving it as highly acceptable, helpful, and valuable for solving problems, including appropriateness (how well the intervention fits users’ problems)^4,43 and acceptability (how positively the target population receives the intervention).^4,40

Domain 5. Fidelity refers to the extent to which the BITs deliver core components of the interventions as intended.⁴⁰ It includes adoption (actual use of the intervention by users), penetration (the extent of integration of intervention within a context), and sustainability (maintaining long-term practice and impact). These can be advanced via adherence and enhancing intervention fidelity, which can be achieved by developing intervention manuals or protocols and training interventionists.^54–56

USIS model in the HCD process

When developing and evaluating the BITs guided by the HCD approach, having a clear roadmap of the particular outcomes that occur in each HCD stage is beneficial for researchers and designers. We propose a USIS model by locating specific USIS outcomes to each HCD stage to maximize the potential of success in BITs’ implementation. We believe the USIS model offers a pathway for improving the design process by considering implementation earlier while prioritizing users’ needs throughout the process. Table 3 shows the USIS stage model in the context of BITs design, development, and evaluation. (1) Throughout the HCD process, as the foundation and core element, user-centeredness composed of empathy, engagement, and equity should be prioritized and embeded in each operation. (2) In stages of ideating and prototype, usability testing, and pilot testing, designers and researchers’ teams should focus primarily on efficiency, including cost, timeliness, and rapidity. The mantra, “fail early and often,” conveys the spirit of rapid prototyping and iteration, the inexpensive exploration of novel ideas, and making ideas tangible to test and improve quickly. Additionally, feasibility should be taken into account during these stages. When generating new ideas, it is helpful to create a “low-fidelity” version of a product to model the innovation. Low-fidelity prototyping is a simple and quick way to translate design ideas into testable artifacts, such as paper sketches. (3) Users’ satisfaction is a vital indicator during the usability and pilot testing stages. The designer and researcher team take the opportunity to refine and adjust the BIT in response to users’ perceptions of whether the products are acceptable, useful, and enjoyable to use in terms of their content, complexity level, delivery method, and credibility. (4) Fidelity must be considered during the pilot testing and future testing in community/clinical settings. Based on users’ feedback, further improvement and refinement of the intervention may be needed.

Table 3.

USIS HCD stage model.

USIS constructs	Concepts	HCD process
USIS constructs	Concepts	Discover and define	Ideate and prototype	Test and refine	Pilot testing
User centeredness	Empathy Engagement Equity	X	X	X	X
Efficiency	Cost Timeliness Rapidity		X	X	X
Feasibility	LearnabilityMemorabilityerror reductionlow cognitive load		X	X	X
Satisfaction	AcceptabilityAppropriateness			X	X
Fidelity	AdoptionPenetration Sustainability				X

USIS: user-centered and sustainable implementation science; HCD: human-centered design.

Case study: The sleep intervention for children and their parents

This case study is based on an internet-delivered SLEEPSMART for children with JIA and their parents.^57,58

The SLEEPSMART project demonstrates the application of the USIS model in designing and evaluating a BIT. Insights from the narrative review informed the design principles applied during the development and testing stages of SLEEPSMART.

About SLEEPSMART. Sleep deficiency, which includes insufficient sleep duration and poor sleep quality, is highly comorbid in children with JIA. A multidisciplinary research team developed and pilot-tested SLEEPSMART for children with JIA and their parents.⁵⁷ SLEEPSMART was designed to improve a child's sleep by modifying the child/parent's negative beliefs, increasing outcome expectations and self-efficacy, facilitating changes in the social and physical environments, and encouraging child/parent activation. In the SLEEPSMART program, children partnered with their parents. They learned through six interactive educational modules (voice-over slideshow), completed quizzes and activities after each module, and were supported by a sleep coach throughout the program.

SLEEPSMART development with HCD. The evaluation was based on the data collected during the SLEEPSMART development process at each HCD stage.

Stage I. Discover and define. The research team conducted in-depth interviews with six parent-child dyads. Designers and researchers partnered together and developed an interview protocol. Data were collected from children with JIA and their parents on their perceptions, needs, preferences, and daily behaviors regarding sleep habits, routines and perceptions, JIA management, mobile/web habits, and information-seeking behaviors. All interview questions were user-centered and aimed to identify the users’ true needs and preferences. Based on users’ feedback, designers identified and defined the users’ needs. For content, users requested age-appropriate language (fourth-grade reading level or above) with simple and sufficient information and without overwhelming amounts of text. For design features, users needed an interface with engaging themes (users brought up space, animal, or sports themes), easy to navigate, and interactive; users (especially children) wanted to learn with fun interaction and rewards, incorporating accessibility and functionality. Parents proposed fonts should be easy to read for individuals with dyslexia.

Stage II. Ideate and prototype. In this stage, among many approaches, participatory design (PD) is one of the methods used to design collaboratively with users via a think-aloud protocol. Participants verbally described what they are doing while working through the product flow.⁵⁵ In the SLEEPSMART project, four parents and child dyads were invited to join the PD session. Thedesigners aimedto apply the generative and evaluative techniques to better understand: (1) what content types were most engaging for and appealing for 9 to 11-year-old children (as well as to their parents), (2) what theme/design was most engaging for and appealing to 9 to 11-year-old children, (3) what would help motivate 9 to 11-year-old children (with arthritis and sleep issues) to go through a sequence of learning modules. The users and designers worked with each other to brainstorm ideas about the intervention design and had group discussions to generate and evaluate ideas.⁵⁹ In addition, low-cost and rapid iterative user personas and wireframes were used to model users’ behaviors before usability testing. A persona is an approach to model and synthesis by observing many people in a targeting market.⁶⁰ Wireframe illustrates an interface's space location and content layout.⁴⁷ Based on users’ input, the design and contents featured in SLEEPSMART include numerous aspects. For designs, SLEEPSMART used many visuals, separated text with images, used bright colors, enabled audio playing to improve accessibility, used an empathetic tone, and ensured multiple ways submit assignments. For content, a balanced number of images, texts, and videos are incorporated; and information on arthritis and sleep is clear and well-structured.

Stage III. Usability testing and refinement. Two usability testing sessions were conducted with two parent-child dyads between June and August 2019. The goals of the usability testing focused on (1) assessing the overall usability of each of the three platforms that the user must navigate in the learning module process (email, SLEEPSMART website, and REDCap Site). REDCap is a secure, web-based software platform designed to support data capture and management was used for the data collection; (2) assessing the overall usability of navigating between the three platforms in the learning module process (email → SLEEPSMART website → REDCap site); (3) assess task comprehension of the first learning module; (4) determine the level of ease in completing the core tasks required by each platform in the learning module; and (5) identify obstacles to completing essential tasks. The research and design teams identified what worked well and identified opportunities for improvement. They repeated the iterative process, discussed potential solutions to the problems, and revised the design for the intervention website, the content of the email, and REDCap. Based on users’ feedback in this stage, designers kept the brief text in the SLEEPSMART website and included each step at the beginning of the website. For example, in step 1, review the overview of the weekly lesson; in step 2, listen to the slide show; in step 3, take a quiz; and in step 4, complete your weekly activities and upload them to REDCap. For the design, designers used bold/underlining to highlight the essential information in the email, created sub-sections within each step, resized the slideshow (or video) to fit the screen size, and added a play button or arrow for the slideshow.

Stage IV. Pilot testing. From 2019 to 2021, to test the feasibility, acceptability, and efficacy of SLEEPSMART, children with JIA and their parents were recruited from a large urban pediatric hospital. A list of names and addresses of children who met the study eligibility criteria was obtained from an electronic medical review by the clinical research assistant (CRA). Potential subjects were mailed a letter with the study website and the CRA's contact information. Interested participants were directed to the SLEEPSMART study website, which described the study. There were 63 dyads screened for this study, and 13 were excluded (due to ineligibility), resulting in a final sample of 50 dyads. Of the 50 dyads enrolled in the study, six dropped out, 24 received access to the SLEEPSMART program, and 20 received usual standard care. They were randomized to either the SLEEPSMART group or the control group. The SLEEPSMART intervention group received interactive modules on (1) learning a wind-down and wake-up routine, (2) reducing time in bed, (3) reducing sleep-related worry, (4) negotiating sleep in the environment, (5) correcting unhelpful sleep-related beliefs, and (6) sleep maintenance and relapse prevention. They also interacted with a sleep coach to answer their questions and adjust their goals. Participants in the control group received standard rheumatology care, including medications, regular follow-ups with pediatric rheumatologists, and, for some, physical therapy to maintain joint mobility and strength. Data were collected on the feasibility and acceptability of SLEEPSMART and outcomes in sleep (e.g. sleep quality, total sleep time, and sleep efficiency) and shared management (e.g. sleep attitudes and beliefs and self-efficacy) via REDCap. After intervention completion, researchers conducted exit interviews with 12 parent-child dyads to understand their perceptions of what they liked and found useful/acceptable/helpful in the SLEEPSMART intervention and identify the areas/gaps they did not like and needed to change for refinement. In summary, based on users’ feedback, parent-child dyads were most satisfied with five aspects: (1) learning modality, the flexibility of self-paced online access learning; (2) intervention content and structure, the content was informative and reliable; (3) research team, research team was highly responsive, informative and accessible; (4) empowered children, children could assume responsibility and take control to improve their sleep; and (5) parent-child shared management, the intervention partnered parent and child together toward the same goal. Dyads also reported their suggestions for SLEEPSMART, which is reported in the evaluation section.

Data analysis. All sessions were audiotaped after the families provided consent and assent. To review and evaluate SLEEPSMART, we analyzed and coded the SLEEPSMART development process data in deductive coding methods using the USIS model. According to each USIS outcome, we coded the SLEEPSMART design data under each HCD process as the research questions, user feedback, and design decisions. For example, for the user-centeredness outcome, under the discover and define stage, we summarized the design/research questions on parent-child dyads’ sleep, JIA, mobile/web behaviors, dyads’ feedback to these questions, and the design decisions around dyads’ input including content and design features. The Supplemental Material shows the data collected in each HCD stage during the SLEEPSMART development process.

Evaluation of SLEEPSMART by USIS

Guided by the USIS model, we identified the strengths and areas for improvement for the SLEEPSMART project. The strengths include (1) structured intervention design: a clear and purposeful structure consisting of essential milestones milestones with the right direction throughout the design and development life cycle, ensuring the final high-quality delivery (design and product), and solving the right problem. (2) Prioritization of user-centeredness: research questions were comprehensively exploratoryand closely tied to identifying users’ needs, users’ feedback was fully transcribed and documented, and each design decision was based on users’ input and needs. (3) Application of various HCD methods: the team used in-depth interviews to empathize with users, applied participatory design methods (stickies and big picture activity) to empower “co-design” and users’ engagement, and adopted user persona and wireframe to model users’ behaviors in a low-cost and rapid iterative way. They applied two usability testing sessions as opportunities to improve the intervention's learnability, memorability, error reduction, and cognitive burden and examine users’ satisfaction with the intervention.

The research team also pilot-tested the intervention to assess the feasibility, acceptability, and preliminary efficacy and identified areas for refinement. (4) Consistency across design stages: design decisions were maintained and consolidated across the HCD process. For example, design decisions made in the discover and define stage were well preserved and expanded in the ideate and prototype stage, and the decision became more concrete in the test and refine stage. We also found areas for further improvement. (1) The participants were not diversely representative. All participants were White and had middle to high socioeconomic status. Future studies should include participants from diverse populations with various socioeconomic, cultural, ethnic, and geographic backgrounds to reflect the equity principle. (2) The current long video presentation module could be restructured to improve learnability. The interventional module video could be divided into smaller sections. To account for the children's physical symptoms and age-appropriate short-time attention span, each learning module needs to be shortened (e.g. < 7 minutes) and gamified with more interactive features (e.g. learning through play-based activity). The knowledge needs to be delivered in a more storytelling way and develop an actual reward system (e.g. children will receive physical awards after completing each milestone). (3) The current approach requires participants to navigate multiple platforms (e.g. intervention website, REDCap for surveys, and a separate link for the online sleep diary), increasing the likelihood of operational errors. Consolidating these platforms into a single and easy dashboard would simplifynavigation. Participants would only need to go to this dashboard, check out daily tasks, and track their progress with tools. Participants will stay on track quickly and self-monitor their completion status, potentially motivating them to follow through with the intervention program. (4) Improving fidelity (adoption, penetration, and sustainability) and the organizational aspects: future BIT development should integrate fidelity considerations during the early design stages to enhance long-term implementation success. Several strategies can support this goal^55,56: (a) develop standard operating protocols (SOP) for the intervention; (b) provide training sessions to research teams; (c) establish an implementation team to support the research team; (d) ensure implementational problems are discussed regularly and solved in a timely manner; and (5) build a system for monitoring implementation.

Discussion

To date, little work has been done to directly integrate HCD and IS in designing, developing, and evaluating BITs to maximize their real-world success. To our knowledge, this work is one of the first to propose a new conceptual framework, the USIS model - that unifies HCD and IS outcomes to support more usable, acceptable, and sustainable BITs. Our review of this initial exploratory model development study found that the HCD framework excels at guiding user-centered design with limited mechanisms to address long-term sustainability. IS framework identifies adoption barriers and contextual facilitators but may overlook critical elements like usability or user engagement in early-stage design. Our proposed USIS model addresses the disconnection by linking and integrating the key outcomes of HCD and IS, allowing researchers to consider implementation research earlier in the BIT development process and making implementation planning a part of the design rather than a separate activity.

The five domains of the USIS model—user-centeredness, efficiency, feasibility, satisfaction, and fidelity—form a platform that links iterative design practices with implementation outcomes. For example, user-centeredness and feasibility in the early discovery and prototyping phases influence satisfaction and fidelity during the pilot stage. This structured alignment allows researchers to design for long-term use, reducing the need for costly redesign later. It also provides a shared language for multidisciplinary teams, from designers and clinicians to implementation scientists.

Applying the USIS model to the SLEEPSMART case study demonstrated the utility of this framework in practice. The model helped identify where user feedback aligned with key outcomes (e.g. low cognitive load, equity, and learnability) and where additional iteration was needed to improve feasibility and long-term adoption. These findings underscore the model's value in academic research and community-based or clinical innovation settings, where intervention fidelity and usability directly impact outcomes.

Beyond the SLEEPSMART example, the USIS model can serve as a planning and evaluation tool for a wide range of BITs, particularly in health equity and digital inclusion initiatives. For instance, by embedding equity within the user-centeredness domain, the model encourages intervention teams to engage diverse users, reducing design biases that often limit reach and adherence in underserved populations.

Future research is needed to empirically test the USIS model in other health contexts. Quantitative studies could examine the relationships between specific USIS domains and intervention success metrics (e.g. adherence rates, long-term use, and health outcomes). At the same time, qualitative work may explore how teams operationalize these domains during intervention development. In addition, evaluating the model's utility across different organizational and cultural settings can further strengthen its generalizability.

Limitation

As a preliminary study, this work had limitations and left unanswered questions for future exploration. First, we did not systematically review all empirical research studies that focused on HCD and implementation research, and the sources we used were limited to those written in English. Second, when analyzing the core elements of usability and implementation research outcomes, we only relied on the published work, which can be heterogeneous. We conducted several peer discussions and debriefing rounds to ensure the best interpretation. Third, this model remains largely conceptual, and we need more empirical evidence to test it. Finally, applying the USIS model can be challenging for HCD and IS scientists to shoulder the learning efforts of incorporating two disciplines into one new research approach.

Conclusion

The USIS model offers an easy-to-follow roadmap for designing, developing, and evaluating BITs. By embedding implementation considerations early into the HCD process, the USIS model creates a platform for multidisciplinary teams to work together to address common barriers to adoption, scale, and sustainability in BITs. In addition, beyond healthcare, USIS can also be applied in academic, clinical, organizational, and community settingsto improve the efficiency and efficacy of developing new solutions. From a policy perspective, the USIS model provides a lens for policymakers to consider inclusive, equitable digital health innovation and promote accessibility, usability standards, and meaningful user involvement.

Future research is needed to test the USIS model empirically across diverse populations, validate how USIS impacts BIT outcomes, explore its performance in different cultural and organizational contexts, and assess its effectiveness in guiding cross-sector collaboration among designers, researchers, and implementers.

In summary, the USIS model bridges two historically distinct disciplines—HCD and IS—offering a unified language and framework for identifying multilevel barriers to implementation, guiding and evaluating BITs, and contributing to the evolving field of digital behavioral health.

Supplemental Material

sj-xlsx-1-dhj-10.1177_20552076251348579 - Supplemental material for Evaluating behavioral intervention technologies: Integrating human-centered design and implementation science outcomes

Supplemental material, sj-xlsx-1-dhj-10.1177_20552076251348579 for Evaluating behavioral intervention technologies: Integrating human-centered design and implementation science outcomes by Shumenghui Zhai, Weichao Yuwen, Tiffany L Zyniewicz, Jennifer Sonney, Jonika Hash, Maida Chen and Teresa M Ward in DIGITAL HEALTH

Footnotes

ORCID iDs

Shumenghui Zhai

Jonika Hash

Ethical considerations

The authors confirm no Ethical Committee approval is needed for this project.

Author contributions

SZ and WY conceived the study. SZ wrote the first draft. All authors reviewed, provided critical several rounds of feedback, edited the article, and approved the final version of the article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by NIH/NINR Grant NR016585. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the US Government.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Guarantor

Dr Weichao Yuwen.

Patient consent statements

The authors confirm that the Patient Consent Statement is not required since this study did not involve any patients or subjects.

Supplemental material

Supplemental material for this article is available online.

References

Michie

Yardley

West

, et al. Developing and evaluating digital interventions to promote behavior change in health and health care: recommendations resulting from an international workshop. J Med Internet Res 2017; 19: e7126.

Wray

Kahler

Simpanen

, et al. User-centered, interaction design research approaches to inform the development of health risk behavior intervention technologies. Internet Interv 2019; 15: 1–9.

Bausback

Bunge

. Meta-analysis of parent training programs utilizing behavior intervention technologies. Soc Sci 2021; 10: 367.

Hermes

Lyon

Schueller

, et al. Measuring the implementation of behavioral intervention technologies: recharacterization of established outcomes. J Med Internet Res 2019; 21: e11752.

Lee

Choi

Lee

, et al. Effective behavioral intervention strategies using mobile health applications for chronic disease management: a systematic review. BMC Med Inform Decis Mak 2018; 18: 12.

Mohr

Burns

Schueller

, et al. Behavioral intervention technologies: evidence review and recommendations for future research in mental health. Gen Hosp Psychiatry 2013; 35(4): 332–38.

Mohr

Schueller

Montague

, et al. The behavioral intervention technology model: an integrated conceptual and technological framework for eHealth and mHealth interventions. J Med Internet Res 2014; 16: e3077.

Stiles-Shields

Crowe

Driscoll

CFB

, et al. A systematic review of behavioral intervention technologies for youth with chronic health conditions and physical and intellectual disabilities: implications for adolescents and young adults with spina Bifida. J Pediatr Psychol 2019; 44: 349–362.

Onken

. Implementation science at the National Institute on Aging: the principles of it. Public Policy Aging Rep 2022; 32: 39–41.

10.

ISO. ISO. ISO. Accessed January 23, 2021. https://www.iso.org/cms/render/live/en/sites/isoorg/contents/data/standard/05/20/52075.html. 1999.

11.

Melles

Albayrak

Goossens

. Innovating health care: key characteristics of human-centered design. Int J Qual Health Care 2021; 33: 37–44.

12.

Norman

Hamm

Schenker

, et al. Assessing the application of human-centered design to translational research. J Clin Transl Sci 2021; 5: e130.

13.

Hartson

. Human-computer interaction: interdisciplinary roots and trends. J Syst Softw. Published online 1998; 43(2): 103–118.

14.

Lund

. Measuring usability with the use questionnaire12. Usability User Exp Newsl STC Usability SIG 2001; 8(2): 3–6.

15.

ISO. ISO. Accessed January 23, 2021. https://www.iso.org/cms/render/live/en/sites/isoorg/contents/data/standard/01/68/16883.html. 1998.

16.

Lyon

Koerner

. User-centered design for psychosocial intervention development and implementation. Clin Psychol Publ Div Clin Psychol Am Psychol Assoc 2016; 23: 180–200.

17.

Hooley

Amano

Markovitz

, et al. Assessing implementation strategy reporting in the mental health literature: a narrative review. Adm Policy Ment Health Ment Health Serv Res 2020; 47: 19–35.

18.

Klaic

Kapp

Hudson

, et al. Implementability of healthcare interventions: an overview of reviews and development of a conceptual framework. Implement Sci IS 2022; 17: 10.

19.

Powell

McMillen

Proctor

, et al. A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev 2012; 69: 123–157.

20.

Powell

Beidas

Lewis

, et al. Methods to improve the selection and tailoring of implementation strategies. J Behav Health Serv Res 2017; 44: 177–194.

21.

Waltz

Powell

Matthieu

, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the expert recommendations for implementing change (ERIC) study. Implement Sci 2015; 10: 109.

22.

Waltz

Powell

Fernández

, et al. Choosing implementation strategies to address contextual barriers: diversity in recommendations and future directions. Implement Sci 2019; 14: 42.

23.

Dopp

Parisi

Munson

, et al. A glossary of user-centered design strategies for implementation experts. Transl Behav Med 2019; 9: 1057–1064.

24.

Bauer

Kirchner

. Implementation science: what is it and why should I care? Psychiatry Res 2020; 283: 112376.

25.

Munson

Friedman

Osterhage

, et al. Usability issues in evidence-based psychosocial interventions and implementation strategies: cross-project analysis. J Med Internet Res 2022; 24: e37585.

26.

Lyon

Brewer

Areán

. Leveraging human-centered design to implement modern psychological science: return on an early investment. Am Psychol 2020; 75: 1067–1079.

27.

Beres

Simbeza

Holmes

, et al. Human-centered design lessons for implementation science: improving the implementation of a patient-centered care intervention. J Acquir Immune Defic Syndr 1999 2019; 82: S230–S243.

28.

Grant

Booth

. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Inf Libr J 2009; 26: 91–108.

29.

Green

Johnson

Adams

. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. J Chiropr Med 2006; 5: 101–117.

30.

Ferrari

. Writing narrative style literature reviews. Med Writ 2015; 24: 230–235.

31.

Nielsen. Usability Inspection Methods: Book by Jakob Nielsen. Nielsen Norman Group, 1994, New York.

32.

Maguire

. Methods to support human-centred design. Int J Hum-Comput Stud 2001; 55: 587–634.

33.

Tognazzini

. First Principles of Interaction Design (Revised & Expanded). askTog. March 6, 2014. Accessed March 22, 2023. https://asktog.com/atc/principles-of-interaction-design/.

34.

Lyon

Bruns

. User-centered redesign of evidence-based psychosocial interventions to enhance implementation—hospitable soil or better seeds? JAMA Psychiatry 2019; 76: 3–4.

35.

Chinman

Hunter

Ebener

, et al. The getting to outcomes demonstration and evaluation: an illustration of the prevention support system. Am J Community Psychol 2008; 41: 206–224.

36.

Graham

Logan

Harrison

, et al.

Lost in knowledge translation: time for a map?

J Contin Educ Health Prof 2006; 26: 13.

37.

Aarons

Hurlburt

Horwitz

. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health Ment Health Serv Res 2011; 38: 4–23.

38.

Damschroder

Aron

Keith

, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009; 4: 50.

39.

Harvey

Kitson

. Implementing Evidence-Based Practice in Healthcare: A Facilitation Guide. London, 2015. doi:10.4324/9780203557334.

40.

Stirman

Miller

Toder

, et al. Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implement Sci 2013; 8: 65.

41.

Wiltsey Stirman

Baumann

Miller

. The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implement Sci 2019; 14: 58.

42.

Glasgow

Vogt

Boles

. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999; 89: 1322–1327.

43.

Proctor

Silmere

Raghavan

, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011; 38: 65–76.

44.

Proctor

Landsverk

Aarons

, et al. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health 2009; 36: 10.1007/s10488-008-0197-4.

45.

Surma-aho

Hölttä-Otto

. Conceptualization and operationalization of empathy in design research. Des Stud 2022; 78: 101075.

46.

Birnbaum

Lewis

Rosen

, et al. Patient engagement and the design of digital health. Acad Emerg Med Off J Soc Acad Emerg Med 2015; 22: 754–756.

47.

Göttgens

Oertelt-Prigione

. The application of human-centered design approaches in health research and innovation: a narrative review of current practices. JMIR MHealth UHealth 2021; 9: e28102.

48.

Kwak

. How Equity-Centered Design Supports Anti-Racism in the Classroom. Every Learner Everywhere. November 30, 2020. Accessed April 18, 2023. https://www.everylearnereverywhere.org/blog/how-equity-centered-design-supports-anti-racism-in-the-classroom/.

49.

Liu

Cruz

Rockhill

, et al. Mind the gap: considering disparities in implementing measurement-based care. J Am Acad Child Adolesc Psychiatry 2019; 58: 459–461.

50.

Lyon

Dopp

Brewer

, et al. Designing the future of children’s mental health services. Adm Policy Ment Health Ment Health Serv Res 2020; 47: 735–751.

51.

Richardson

Lawrence

Schoenthaler

, et al. A framework for digital health equity. Npj Digit Med 2022; 5: 1–6.

52.

Veinot

Mitchell

Ancker

. Good intentions are not enough: how informatics interventions can worsen inequality. J Am Med Inform Assoc 2018; 25: 1080–1088.

53.

Graham

Wildes

Reddy

, et al. User-centered design for technology-enabled services for eating disorders. Int J Eat Disord 2019; 52: 1095–1107.

54.

Mohr

Riper

Schueller

. A solution-focused research approach to achieve an implementable revolution in digital mental health. JAMA Psychiatry 2018; 75: 113–114.

55.

Toomey

Hardeman

Hankonen

, et al. Focusing on fidelity: narrative review and recommendations for improving intervention fidelity within trials of health behaviour change interventions. Health Psychol Behav Med 2020; 8: 132–151.

56.

Walton

Spector

Williamson

, et al. Developing quality fidelity and engagement measures for complex health interventions. Br J Health Psychol 2020; 25: 39–60.

57.

Ward

Skubic

Rantz

, et al. Human-centered approaches that integrate sensor technology across the lifespan: opportunities and challenges. Nurs Outlook 2020; 68: 734–744.

58.

Zhai

Palermo

Shenoi

, et al. A shared-management web-based intervention for sleep deficiency in school-age children with juvenile idiopathic arthritis and their parents: feasibility and acceptability study. J Clin Sleep Med. Published online March 7, 2025. doi:https://doi.org/10.5664/jcsm.11610.

59.

Kang

Choo

Watters

. Design for experiencing: participatory design approach with multidisciplinary perspectives. Procedia - Soc Behav Sci 2015; 174: 830–833.

60.

Bhattacharyya

Mossman

Gustafsson

, et al. Using human-centered design to build a digital health advisor for patients with complex needs: persona and prototype development. J Med Internet Res 2019; 21: e10318.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB