Mixed-Method Formative Evaluation and Cultural Adaptation of Digital Health Communication Content and Delivery Promoting Lung Cancer Screening Shared Decision-Making

Abstract

Introduction

Lung cancer remains a leading cause of cancer death, though lung cancer screening (LCS) uptake remains low, particularly among low-resource patient populations. LungSMART Utah uses a multi-level approach to promote LCS to patients served by Community Health Centers (CHCs). This paper reports on the formative evaluation and cultural adaptation of the project’s patient-facing digital health communication content and delivery, in service of the larger population health effort to promote shared decision-making for LCS.

Methods

We used a mixed-method approach (cross-sectional surveys and usability observations) to evaluate developed content and ensure cultural adaptation to serve all patients within the CHCs. First, a messaging study (N=79) used an online survey to evaluate patient perceptions of content quality and to collect feedback on the quality of content in both English and Spanish. Second, a usability assessment (N=17) employed think-aloud protocols to evaluate user experience and technical delivery.

Results

Patients in the messaging study generally rated content favorably. Content evaluations were similar in English and Spanish, though some participants did suggest phrasing changes. The usability assessment identified critical technical and delivery hurdles, including feeling overloaded by choices for some project components and message sequencing concerns. Cultural adaptation insights from Spanish-speaking participants highlighted the need for conversational alternatives to some formal medical terms, explicit transparency about cost, and addressing access anxiety. All participants indicated noting message content came from the CHCs helped foster trust.

Conclusion

Mixed-method formative evaluation and cultural adaptation provide important insights for developing digital population health tools. Addressing technical delivery gaps and specific community concerns will improve the reach and efficacy of LCS promotion in low-resource settings.

Keywords

chatbot community health centers cultural adaptation digital health formative evaluation health communication lung cancer screening text messaging

Lung cancer continues to be the leading cause of cancer death in the United States for both men and women.¹ Currently, the United States Preventive Services Task Force (USPSTF) recommends annual lung cancer screening (LCS) in the form of low-dose chest computed tomography for adults who are 50 to 80 years of age, have a 20 pack-year smoking history, and currently smoke or have quit in the past 15 years.^2,3 LCS has been demonstrated to reduce lung cancer mortality,⁴ but the screening is not without risks including false positive results leading to invasive diagnostic follow-ups that can result in patient harm.⁵ As such, the USPSTF recommends shared decision-making (SDM) to weigh the benefits and harms of screening for eligible individuals.²

Compared to cancer screening for breast and colorectal cancer, the recommendation for LCS is relatively new, requires a shared decision-making (SDM) session prior to referral, and LCS uptake is low throughout the US.⁶ While there is a clear benefit for early detection to widespread implementation of LCS processes for population health, current system-level (e.g., incomplete smoking history in the electronic health record [EHR]),⁷ provider-level (e.g., time burden and complexity of SDM),⁸ and patient-level (e.g., lack of awareness)⁹ barriers help explain the low uptake rates and make implementation more challenging. This is particularly true for healthcare providers like community health centers, who may have limited EHR capabilities and already overburdened providers, resulting in differences in access to cancer screening in general.^10,11 Digital health promotion is uniquely positioned to address some of these barriers by involving the patient outside of clinical appointments.

Previous research on communicating with patients about LCS has focused primarily on improving SDM interactions.^12,13 A recent review¹³ of digital health approaches to support and promote lung cancer screening found most studies did not test patient-facing messaging to promote the SDM session, and in turn LCS, with one exception.¹⁴ Related research on communicating about LCS has found that videos with LCS information can be informative and motivating,^15,16 content used in LCS promotion should avoid stigmatizing eligible patients,^17,18 social marketing approaches can raise awareness about LCS,¹⁹ and both patients and providers want complete information on LCS (i.e., eligibility, cost, risks/benefits, etc.).¹⁸ Recent research on LCS messaging suggests many eligible patients have high threat perceptions related to past smoking behavior and LCS, so focusing on mitigating barriers to screening and emphasizing benefits may be most beneficial to LCS promotion.²⁰ Taken together, there are gaps in evidence related to best practices for promoting LCS to patients directly, how to integrate appeals about LCS into digital health programs, and identifying challenges about the bilingual promotion of LCS. The current study is focused on providing information about LCS to promote patients’ initiation and completion of an SDM session.

The LungSMART Utah pragmatic trial promoting LCS is a multi-level, two-phase sequential multiple assignment (SMART) randomized controlled trial of digital health tools designed to address LCS uptake challenges and ensure broad implementation in low-resource care settings.²¹ The overall goal of the trial is to improve LCS uptake among patients served by safety net community health centers (CHC) throughout the state of Utah, a state with one of the lowest LCS rates (45^th) in the US.²² The project leverages a centralized hub infrastructure to alleviate the burden of SDM and referrals to LCS on under-resourced CHCs, identifying potentially eligible patients via population-level EHR algorithms and engaging those patients through low-barrier technologies (e.g., bidirectional text messaging [TM] content, video delivered via TM, a rules-based chatbot, and telehealth-delivered SDM). By integrating automated screening assessments with low-tech digital health solutions and telehealth-delivered SDM, as well as patient navigation after SDM for eligible and interested patients, the project components systematically address potential barriers to LCS like cost, patient-preferred language, information deficits, and transportation. Ultimately, the project aims to identify scalable and sustainable implementation approaches to broaden the reach of LCS among low-resource populations, including low-income, rural, and Spanish-speaking communities, to reduce the disproportionate burden of lung cancer mortality. The research reported here focuses on the development, adaptation, and evaluation of project components focused on informing patients about LCS and connecting them to an SDM encounter, with the goal of increasing the number of eligible patients completing LCS if it is right for them.

Frameworks Guiding Project Design and Digital Health Content/Delivery

Two frameworks directly guided digital health content and delivery for the current project: (1) the Behavior Change Wheel (BCW),²³ and (2) the Integrated Framework for Intervention Adaptation (IFIA).²⁴ An additional framework, the Consolidated Framework for Implementation Research (CFIR),²⁵ was important to overall project development and logistics, but did not directly influence content development. CFIR is still important to mention as all content developed needed to complement other project components for the larger trial activities.

Behavior Change Wheel (BCW)

The BCW is a comprehensive model that integrates insights and tenets of several behavior change and policy theories and frameworks.²³ The goal of the BCW is to provide researchers and practitioners with an actionable behavior change framework for planning intervention activities. Individual mechanisms are central to the BCW, suggesting that people engaging in a behavior must have the capability, opportunity, and motivation to do so. This component is referred to as COM-B. For example, in the case of LCS, people would need to be able to physically and mentally engage in the behavior, while also having sufficient knowledge and skills (capability). Additionally, those people would need to have factors outside of their control (e.g., social drivers of health) be conducive to the behavior taking place (opportunity). Finally, people must be energized and willing to complete processes (e.g., learn about LCS, engage in SDM about LCS, schedule LCS, and attend the scheduled LCS appointment) that result in the desired behavioral outcome (motivation).

Content for the LungSMART project, particularly the initial phase where the goal is to connect patients to an SDM encounter with a CHC nurse, uses COM-B principles through educating individuals about the importance of LCS and enabling patients to engage in SDM. For the present study, content being successful means patients will be more educated about LCS and be interested in talking to a nurse to engage in SDM about LCS.

Integrated Framework for Intervention Adaptation (IFIA)

Escoffery and colleagues synthesized cultural adaptation strategies into what we refer to as the IFIA.²⁴ The IFIA provides steps to adapt project components, including content, involving community members and project stakeholders. In total, the IFIA identifies 11 steps common to adaptation processes, enabling researchers to engage in as many steps as makes sense for specific project contexts. Steps relevant to the current project include community assessment, expert consultations, stakeholder consultations, identification of elements for adaptation, adapt elements as needed, and evaluate adapted materials prior to implementation and overall, comprehensive evaluation.²⁴ We use the IFIA to engage in meaningful cultural adaptation, which requires moving from simple “surface structure” changes (e.g., simple translation) to “deep structure” modification²⁶ (e.g., accounting for cultural nuances and real-world situations of the intended audience). For the current study, the need for cultural adaptation was to ensure content was adapted for Latino/a and Hispanic patients beyond simple translation that may not capture the cultural and systemic factors unique to those communities.

LungSMART Utah Content Development Process

Those frameworks influenced our approach to designing and adapting content and delivery for LungSMART Utah. Prior to developing content, we met with patient and other CHC stakeholders in an advisory committee meeting to gather initial thoughts and ideas about LCS. This community assessment allowed the project team to learn about community knowledge about LCS, current CHC practices related to LCS, and perceptions of benefits and barriers to LCS. Next, the project team—made up of subject matter experts in health communication, implementation science, biomedical informatics, and included team members fluent in Spanish—created the overall communication strategy for the project, identified specific content needed (and the goals of that content), and then drafted all needed content. Once the initial draft of the content was created, additional expert review occurred by project leadership. Initial revisions were made to the content at that point, and an initial translation of content from English into Spanish took place. Following translation, we presented content (in English and Spanish) to stakeholders in a follow-up advisory committee meeting. Further revisions were made to the content in both languages, at which point we decided the process was ready to undergo further adaptation and evaluation activities as described below.

In the current paper, we report the development, evaluation, and adaptation of digital health content delivery for the initial phases of the LungSMART Utah trial (i.e., content focused on encouraging eligible patients to engage in SDM). More specifically, we provide results from a bilingual messaging study among eligible CHC patients that evaluated the text and video content developed, as well as a bilingual usability assessment that further evaluated content and provided insights into user experience with delivery. The key research questions underlying these studies were as follows: (1) How does content need to be adapted to be viewed as high quality in both English and Spanish? And (2) What aspects of project content delivery need to be modified based on English- and Spanish-language user experiences?

Methods

Bilingual Messaging Study

Participants and Recruitment

We recruited patients from a CHC who were potentially eligible for LCS and would be eligible to participate in the LungSMART Utah trial, to participate in an online survey study. Our intent was to collect data from as many patients as our resources allowed.²⁷ The limited resource for sampling in the present study was patient willingness to participate in the bilingual messaging study. Patients needed to be between the ages of 50 and 80, be a current or former tobacco user per their EHR data, have a phone that could receive TM, and have internet access. Patients were recruited via email based on information from the CHC EHR (e.g., aged 50 to 80 years, visit to CHC within 1 year, active/former smoker, and an available email address).

Procedures

Study participants received an email with a link inviting them to complete a survey. The first page of the linked survey was a consent cover letter informing participants of their rights and general study procedures. If participants agreed to complete the survey, they responded to questions about their smoking history²⁸ and then completed the main survey. One of the screening questions asked participants their language preference and language proficiency for English and Spanish. The main survey had all participants complete three content evaluation tasks in their preferred language. Bilingual participants received a few additional questions asking them to compare English and Spanish versions of content (i.e., rated content as all participants did in their preferred language, but then also reviewed juxtaposed messages in English and Spanish).

The first content evaluation task was the most complex and related to the initial, introductory TM patients would receive. This task was important because it provided feedback about how and if people would respond to the initial messages from the study team (see Supplemental Material for all presented information). Within the survey, we showed a mocked-up TM interface that included the introductory message. After reading the introductory message, patients responded to items about their perceptions of the trustworthiness, appropriateness of the amount, helpfulness, clarity, and tone of information. One item was used for each perception from an often-used battery of items on information evaluation.²⁹ For those items, and all other quantitative items, participants responded on five-point Likert-type scales where higher values from 1 to 5 indicated higher positive perceptions of those variables. We then presented the draft introductory message to patients again, placing it alongside more in-depth information about LCS. With the TM and additional LCS information juxtaposed next to each other, we asked participants to evaluate if the presented information introduced LCS well and was consistent with the newly provided, more detailed LCS information. Finally, participants were given space to provide open-ended feedback to improve the information as presented. Bilingual participants were then shown the Spanish and English versions of the introductory messages next to each other and asked for feedback about the translation.

The second content evaluation task had all patients examine the chatbot content providing information about the LCS process. For this task, we created two distinct versions of potential content to show participants. Participants initially viewed the team’s initial draft of the content and responded to items like those in evaluation task one (i.e., five message perceptions and if the information made them want to learn more about LCS). We then showed participants the original message they reviewed about the LCS process juxtaposed with different wording about the LCS process that conveyed the same information (see Supplemental Material for both messages). We asked participants if the information presented was similar overall and in terms of the amount of information conveyed. Participants were also given the opportunity to provide open-ended feedback about the information they evaluated.

For the third and final content evaluation task, participants viewed one of three remaining content sections that would appear in the chatbot response options, i.e. the basics of LCS, benefits of LCS, or risks of LCS (see Supplemental Material for all content). Like the other content evaluation tasks, participants responded to the five items about their perceptions of the information and if the information made them want to learn more about LCS. Participants also could provide open-ended comments about the information; bilingual participants also could provide open-ended feedback about the translations. At the end of the survey, participants were thanked for their time and provided a $25 incentive.

Data Analysis

All quantitative survey data were analyzed in SPSS. We report primarily descriptive information about the content evaluated in both English and Spanish, as well as the comparisons by bilingual evaluators. The team aimed to achieve content quality (e.g., trustworthiness, clarity, tone) rated at an aggregate level of 4 or greater, determined by simple descriptive statistics for each item, to indicate agreement from participants that the content was above average quality. We also conducted a series of independent-samples Mann-Whitney U tests, using listwise deletion of missing cases, for content evaluation dimensions related to information perceptions to determine if there were differences between English and Spanish language evaluators, excluding bilingual evaluators for those analyses. We used a nonparametric approach due to uneven sample sizes of English and Spanish evaluators. These analyses allowed us to evaluate if content in English or Spanish was perceived as lower quality and would require additional adaptation prior to launching the pilot trial.

Bilingual Usability Assessment

Participants and Recruitment

We again recruited patients from the same CHC who were potentially eligible for LCS and would be eligible for LungSMART Utah to participate in either an in-person or virtual usability assessment. Like the messaging study, the determination of sample size for this study was related to resource constraints.²⁷ For the usability assessment, resource constraints included patient willingness and the ability for the team to conduct long user experience sessions.

Procedures

Patients from the CHC likely to meet study inclusion criteria received an email inviting them to participate in the usability assessment study. The email contained a link to a brief screener survey, and consent information about the study, to ensure patients met the inclusion criteria (identical to the messaging study). Eligible patients were contacted by members of the study team, and a usability session was scheduled to take place virtually over Zoom or in person at a local public library. Patients could participate whether they spoke English or Spanish, as the study team had facilitators to conduct sessions in either language.

Usability assessment sessions were scheduled for 60 minutes. Informed consent was obtained at the beginning of the session. Each assessment session followed a basic think-aloud protocol, where participants were given tasks to complete on their mobile devices. The project’s technical team created an interface that could be controlled by the usability session facilitators to ensure delivery of different study components to participants to test out variable user experiences (e.g., if the video content played properly, responses to chatbot content and timing of message delivery, etc.). Each participant was randomly assigned to complete up to three different tasks that would expose them to all content elements from the study. The goal was to ensure sufficient feedback was acquired across participants for all content, as well as all potential user experiences across the different conditions. Participants followed prescribed steps through the study components and, as they completed those tasks, explained to the session facilitators what they were thinking about content, technical aspects of delivery, or anything else that came to mind. Additionally, we aimed to gather further information that would assist in cultural adaptation and delivery of the content. Some cultural adaptation questions were asked to all participants (e.g., asking if terms like SDM made sense), with a few additional questions only for Spanish-speaking participants (e.g., “How can we ensure that people like you continue responding to the messages?”). All usability sessions were recorded and then transcribed for analysis. Session facilitators also took notes during sessions. Participants received a $75 gift card for completing a session.

Data Analysis

Session facilitators led data analysis of the usability sessions. Session transcripts and notes were reviewed to identify and catalog technical and delivery issues, content insights, and cultural adaptation insights from the Spanish-language sessions. No formal qualitative analyses were performed on the data, as the goal was to catalog and address all technical errors and content suggestions comprehensively.

The research activities reported here were approved by the University of Utah Institutional Review Board (IRB_00169869, approved May 2024; IRB_00185700, approved January 2025). Participants provided consent prior to completing any research procedures. The reporting of this study conforms to the STROBE guidelines.³⁰

Results

Data collection for the cross-sectional messaging study took place between December 2024 and January 2025. We completed data collection for the cross-sectional usability assessment sessions in March and April 2025.

Messaging Study

Recruitment

In total, recruitment emails were sent to 1,400 patients. From those emails, 132 (9.4%) responded in some way. We removed responses due to ineligibility determined by a screener questionnaire (n=15), incomplete screener or consent information (n=16), failure to complete at least half of the survey (n=19), not consenting (n=1), or providing incoherent/impossible responses (e.g., incoherent open-ended responses or reporting only speaking Spanish but took the survey in English; n=2). For all reported analyses, we used responses with at least 75% of items completed (study N=79). See Table 1 for demographic information.

Table 1.

Demographic Information for the Messaging Study and Usability Assessment

	Messaging study	Usability assessment
N	79 (100%)	17 (100%)
Sex
Male	35 (44%)	8 (47%)
Female	44 (56%)	9 (53%)
Age	M=60.2 years	M=59.5 years
	SD=7.4 years	SD=7.9 years
	Range=50 to 79	Range=50 to 79
Ethnicity
Hispanic/Latino(a)	30 (38%)	4 (24%)
Not Hispanic/Latino(a)	44 (56%)	13 (76%)
Did Not Respond/Unknown	5 (6%)	0 (0%)
Race
White	61 (77%)	14 (82%)
Black/African American	1 (1%)	0 (0%)
Asian/Pacific Islander	1 (1%)	0 (0%)
Multiple Options Selected	4 (5%)	0 (0%)
Did Not Respond/Unknown	12 (15%)	3 (18%)
Language Spoken
Spanish	23 (29%)	3 (18%)
English	48 (61%)	14 (82%)
Bilingual	8 (10%)	0 (0%)
Education
High school diploma or less	30 (38%)	Not Collected
Some college, no diploma	24 (30%)	Not Collected
Some college degree	21 (27%)	Not Collected
Prefer not to answer/no answer	4 (5%)	Not Collected
Income
Less than $30,000	42 (53%)	10 (59%)
More than $30,000	24 (30%)	7 (41%)
Don’t know	3 (4%)	0 (0%)
Prefer not to answer/no answer	10 (13%)	0 (0%)
Insurance
Yes	58 (73%)	16 (94%)
No	19 (24%)	1 (6%)
Prefer not to answer	2 (3%)	0 (0%)
Smoking Status
Yes to eligibility question #1	62 (78%)	-
Yes to eligibility question #2	57 (72%)	-
Reformed (former) smoker	-	12 (71%)
Current smoker	-	5 (29%)

Note. Some % values may not add to 100 within demographic categories due to rounding. Eligibility question #1 = “If you add up all the years when you regularly smoked cigarettes, have you smoked for 20 years or more of your life?” Eligibility question #2 = “At any time in your life, did you regularly smoke 1 or more packs of cigarettes per day?” Almost all participants in the messaging study would have qualified for participation in the main trial, as 94% (n=74) replied yes to at least one of the eligibility screener questions.

Content Evaluation Task 1 (Introductory Information)

Table 2 includes mean and standard deviation for each of the five message perceptions and the item that asked if the content made patients want to learn more about LCS. There were no statistically significant differences when comparing English and Spanish patient respondents on message perceptions across individual Mann-Whitney U tests (all p-values ≥ .497). While the lack of difference does not indicate equivalence, the concern of this formative evaluation was if content in one language indeed performed differently not equivalently. All perceptions of content were rated at 4 or higher across individual groups and across all participants. Participants did not always report as high of a rating related to their desire to learn more about LCS. Bilingual participants thought the English and Spanish content were consistent (M=3.71, SD=1.38), with the mean score being brought down by one participant responding 1 but providing no feedback or information about what was inconsistent, suggesting the response may have been in error.

Table 2.

Means and Standard Deviations of Content Evaluations

Content evaluation task	Survey language	Trust	Amount	Helpful	Clarity	Tone	Learn more
1 (Initial Intro Messages & Basic Info)	English	4.19 (.77)	4.11 (.91)	4.00 (.96)	4.21 (.88)	4.13 (.95)	3.79 (1.08)
	Spanish	4.04 (.88)	4.14 (.71)	4.09 (1.04)	4.30 (.88)	4.09 (1.13)	3.91 (1.00)
	Bilingual	4.50 (.76)	4.25 (1.04)	4.50 (.54)	4.50 (.76)	4.25 (1.17)	4.13 (.99)
	All	4.18 (.80)	4.13 (.86)	4.08 (.95)	4.27 (.86)	4.13 (.95)	3.86 (1.04)
2 (LCS Process Info)	English	4.29 (.74)	4.21 (.80)	4.21 (.77)	4.23 (.76)	4.17 (.78)	3.88 (.84)
	Spanish	4.14 (.89)	4.04 (.98)	4.00 (.90)	4.09 (.85)	4.04 (.88)	3.96 (.98)
	Bilingual	4.25 (.89)	4.25 (1.04)	4.50 (.54)	4.50 (.54)	4.50 (.54)	4.50 (.54)
	All	4.24 (.79)	4.16 (.87)	4.18 (.80)	4.22 (.77)	4.16 (.79)	3.96 (.87)
3a (LCS Basics Info)	English	4.00 (1.00)	3.93 (.92)	4.00 (.96)	4.14 (1.03)	4.07 (.997)	3.86 (1.03)
	Spanish	4.00 (.00)	4.13 (.35)	4.13 (.35)	4.13 (.35)	4.13 (.35)	4.13 (.35)
	Bilingual	4.50 (.71)	4.50 (.71)	4.00 (1.41)	5.00 (.00)	4.50 (.71)	5.00 (.00)
	All	4.04 (.77)	4.04 (.75)	4.04 (.81)	4.21 (.83)	4.13 (.80)	4.04 (.86)
3b (LCS Benefits Info)	English	4.38 (.62)	4.25 (.86)	4.38 (.72)	4.50 (.73)	4.25 (1.04)	4.13 (.89)
	Spanish	4.25 (.71)	4.13 (.99)	4.38 (.74)	4.25 (1.04)	4.25 (1.04)	4.13 (.84)
	Bilingual	4.40 (.55)	4.40 (.55)	4.40 (.55)	4.40 (.55)	4.40 (.55)	4.40 (.55)
	All	4.34 (.61)	4.24 (.83)	4.38 (.68)	4.41 (.78)	4.28 (1.00)	4.17 (.81)
3c (LCS Risks Info)	English	3.71 (.77)	3.47 (.94)	3.41 (1.18)	3.88 (.99)	3.25 (1.39)	3.47 (1.33)
	Spanish	4.29 (.49)	4.29 (.49)	4.29 (.49)	4.29 (.49)	4.29 (.49)	4.29 (.49)
	Bilingual	5.00 (n/a)	5.00 (n/a)	5.00 (n/a)	5.00 (n/a)	5.00 (n/a)	5.00 (n/a)
	All	3.92 (.76)	3.76 (.93)	3.72 (1.10)	4.04 (.89)	3.63 (1.28)	3.76 (1.20)

Notes. Overall, there were 48 participants who completed the survey in English, 23 in Spanish, and 8 bilingually. The sample size for any specific comparison may vary based on data completeness rates but never dropped below 95%. For content evaluation tasks 3a, 3b, and 3c, participants completed only one randomized task, resulting in sample sizes of 23, 28, and 25, respectively, and samples for some cells (e.g., bilingual 3c - n=1). Bold text indicates the mean across all participants for each task.

Forty-two (53%) participants wrote something in the feedback box, though most respondents simply said nothing could be improved about the content or that the content was fine as is. Comments provided by participants encouraged language to be clearer and more direct, asked for information to be changed that cannot be changed (e.g., offer screening for people who smoked less than 20 pack years), asked to offer information beyond the content’s scope (e.g., provide Utah Quit Services information, which were being offered through a separate project), or sought clarification about interactive response options (e.g., what if someone wanted more information but also wanted to answer the eligibility question).

Content Evaluation Task 2 (Comparing LCS Process Content)

Table 2 contains mean values and standard deviations for all variables reported for the first content evaluation task. There were again no statistically significant differences when comparing English and Spanish patient respondents on message perceptions across nonparametric comparisons (p-values ≥ .326).

Open-ended responses were about comparing the two sets of information and general comments about the content. Thirty-six participants provided comparative feedback on the juxtaposed message content versions. Many responses again suggested general satisfaction and no changes required. Participants also noted differences in the messages and varied in their preferences (e.g., some patients said the message on the left was more professional in tone, while the message on the right was thorough but may have contained too much information). Some specific language differences were highlighted that would benefit from being held consistent across informational messages (e.g., one message saying patients hold their breath and the other says breathe normally), though many suggested revisions were not consistent across respondents. For general comments about the content, there were no consistent suggestions across 24 responses, where most again simply suggested the informational messages were of reasonable quality (e.g., “good”) in their current form. Some patients did mention it would be beneficial for the credibility of messages to more frequently mention from which CHC the messages were coming, but no other revisions were consistently suggested.

Content Evaluation Task 3 (Other Conversational Agent Content)

Smaller groups of participants evaluated content for task 3 (see Table 2). Average ratings were generally favorable for information about the basics (3a) and benefits (3b). Ratings were under 4 for several perceptions related to risks (3c) among English participants, though all Spanish responses were above 4. There were no statistically significant differences when comparing English and Spanish patient respondents for 3a (p-values ≥ .525), 3b (p-values ≥ .653), or 3c (p-values ≥ .055). For 3c, the univariate comparison for appropriateness of the amount of information (p=.055), indicated near-significant (but ultimately nonsignificant) difference, with English-language respondents reporting lower perceptions than Spanish-language respondents (see Table 2).

Because fewer patients viewed each of the three content areas in the third evaluation task, few meaningful insights were gathered from the open-ended responses. For the messages about risks (3c), participants in English noted being a bit scared, uncertain, or stressed out by the information. The risk information presented to patients was accurate, though their feedback suggests discomfort in learning about LCS risks.

Usability Assessment

Recruitment

In total, we contacted 1,720 patients about participating in the usability assessment. Of those, 84 patients (4.9%) responded to the interest form attached to the email. Four of those patients were not eligible due to age or smoking status and 80 were eligible. All 80 were contacted to participate. Ultimately, sixteen usability sessions were conducted individually, though one Spanish-language session included two participants, one of whom had experience in professional translation services. Thus, the usability assessment included 17 participants (N=17) who completed procedures in English (n=13) or Spanish (n=4), which is consistent with the sample size for usability assessments of older individuals.³¹ Eleven participants in the usability assessment also participated in the messaging study (9 in English and 2 in Spanish). See Table 1 for demographic information.

Overall, feedback from patients during usability assessments provided critical insights into the real-world delivery of LungSMART Utah. We organized the feedback into three categories: (1) technical and delivery issues, (2) content quality and improvement insights, and (3) cultural adaptation insights.

Technical and Delivery Issues

A primary concern identified was the timing and sequencing of content delivery. Consecutive TMs (e.g., messages arriving back-to-back) sometimes arrived too rapidly, shifting content off screen and forcing users to scroll back up to maintain context. Some messages were also reported to arrive out of order. Participants noted that delays following a “no” response to eligibility questions were particularly frustrating and could lead to disengagement in a real-world setting. While participants found repeated eligibility questions frustrating, facilitator notes suggested this may have been exacerbated by the repetitive nature of the usability tasks themselves.

Regarding the rules-based chatbot, users expressed frustration over the inability to select multiple topics simultaneously. For example, some participants attempted to enter a sequence of numbers (e.g., “123”) to access multiple information categories, but the system provided no response or error message to correct the invalid entry. Additionally, technical performance of the video (one of the study components) was inconsistent. Some participants found that embedded videos failed to load, buffered excessively, or otherwise failed to play on their devices. Some participants also said they would have benefitted from a “press play” prompt to play the video.

Content Quality and Improvement Insights

Content was generally well-received, with participants specifically praising the animated video content for being more approachable and less threatening than videos featuring actors and real people. However, some participants felt the automated TM content was robotic in tone and identified specific terms, such as radiation, as potential triggers for anxiety. The term bot was viewed unfavorably, with participants suggesting virtual assistant as an alternative.

Presentation of choices was a significant hurdle in more complex display of project components. When presenting options for text, video, and the chatbot, participants felt overwhelmed when eligibility questions were presented alongside the study component options. This choice overload led to confusion about how to prioritize responses, prompting a recommendation for a more linear, less complicated message structure. Finally, the inclusion of the CHC name was cited as a major strength. Participants indicated that frequent reminders of their clinic’s involvement were essential for establishing trust and distinguishing the study TM from potential scams.

Cultural Adaptation Insights

The Spanish-language sessions provided essential adaptation insights. Participants identified several terms that were unfamiliar in a medical context, such as cribado (screening) and decisiones compartidas (shared decision-making). They recommended using more conversational equivalents and replacing formal phrasing like sin costo (without cost) with more direct language like gratuito (free).

A critical finding was participant desire to have content address fears regarding immigration and legal status. As the usability assessments were conducted in Spring 2025, a period of increased national news coverage regarding immigration enforcement, participants strongly recommended including an explicit message stating that immigration status does not matter. Further, participants called for greater transparency regarding financial costs. The term bajo costo (low cost) was perceived as too vague, and specific cost information was requested to alleviate financial anxiety. Finally, participants commented on messaging that signaled the next phase of the study (e.g., that a nurse would reach out to discuss LCS, which would focus on SDM). Within this context, participants emphasized a strong preference for Spanish-speaking staff who prioritize empathy and cultural understanding over clerical and clinical efficiency.

Discussion

Overall, the feedback received across the two studies was encouraging insofar as patients generally seemed to be favorable toward the content, interested in information about LCS, and able to navigate the project components via their mobile phones. Still, there were several areas across the two studies that guided improvement on content and delivery strategies based on the feedback. Below, we present the insights gained from the studies for delivering digital health tools and what modifications we made specific to LungSMART Utah prior to the deployment of a full project pilot study.

Insights into Improving Content Quality and Adaptation

The formative evaluation studies highlighted the critical role of trust and language clarity in digital health communication content. Participants valued having CHC clinics as an information source, allowing for a heuristic that the messages were not an attempt to scam them. This suggests for the current project that clinical integration, and identifying the CHCs as sources of content,³² is an important path to evoking trust in delivered content, though patients often have complicated feelings of trust toward providers and clinics.³³ Given the importance of patient trust in completing cancer screening, knowing that the CHCs are trusted sources of information in general will facilitate improved communication with patients for the larger trial. Because all participants seemed to view mentions of CHC clinics positively, we integrated additional mentions of the CHC in the TM and chatbot content to use this positive connection to encourage continued engagement with digital health project components.

The identification of anxiety-inducing language (e.g., related to LCS risks) demonstrates a challenge and tension for program planners to weigh the importance of language specificity (e.g., clinical accuracy) against patient fears (e.g., avoiding non-essential information about LCS) and full disclosure in the spirit of SDM. Consistent with past research²⁰ emphasizing benefits more than risks could be beneficial, but patients responded well to having comprehensive information consistent with other past work,¹⁸ even though some English-language participants seemed uncomfortable with information about LCS risks. Participants suggested adding information about quit resources to the provided content, but the project team decided such information would duplicate other programs running at the CHCs promoting such resources. Future work determining how to integrate existing, complementary programs at CHCs could reduce costs, improve the sustainability of programs, and increase positive health outcomes.

Easier to address content changes included dislikes with technological nomenclature (e.g., bots), which can be avoided quite easily. Further, we gained actionable insights into improving project delivery in Spanish including and beyond changes to translated terms. In response to Spanish-language participants suggesting more transparency about cost, we ensured the video content continued to cover cost and added a topic for the rules-based chatbot that specifically addressed cost in English and Spanish. Given other research with Hispanic and Latino populations for other cancer screening types also found cost to be an important decision-making factor,³⁴ future screening-based programs should explicitly address cost regardless of population health or clinical contexts.

Other Spanish-specific changes that needed to be made related to important terms for LCS including “screening” and “shared decision-making.” Had we not identified these content issues during the usability assessments, our pilot (and main) trials may have taken place with a large percentage of our intended audience not being certain about essential terms relevant to LCS decision-making. This finding makes it clear that cultural adaptation processes are essential to include within digital health trial planning and development phases.

While we considered adding an explicit statement in English and Spanish about immigration status, the team ultimately decided not to do so because of the complex, sensitive, and uncertain nature of the topic and the current healthcare policy environment.

Overall, we found the usability assessments provided more insights into content suggestions than the messaging study, even with fewer participants. This was especially true for cultural adaptation concerns (i.e., identifying terms like screening and shared decision-making that were not clear to participants). There were no comments from the messaging study that did not come up in the usability assessments, suggesting that more thorough and diverse suggestions might be elicited using the user experience interviews. One explanation for this could be that exposure to study components in a more comprehensive way, as done in the usability assessment, motivates more insights than just presenting the message contents on a survey and asking participants to evaluate the content separate from delivery context. Given limited resources related to patient time and interest, similar projects might consider focusing only on expanded usability assessments than traditional messaging studies if the goal is to catalog and act upon user-centered suggestions. The usability assessments also proved to be essential to our cultural adaptation process as well, with considerably more feedback elicited from a smaller number of Spanish speakers.

Insights into Improving Digital Health Content Delivery

The technical challenges we observed during the usability assessments indicated the importance of information flow for digital health delivery. LungSMART Utah is delivered via TM, but TM at scale can result in some unexpected challenges. Participants perceived some of the messages being delivered too rapidly (i.e., back-to-back messages), and that those consecutive messages sometimes appeared out of order, and such sequencing errors were a source of frustration or confusion for participants. Appearing out of sequence undoubtedly muddles the potential positive influence of even high-quality content, so making sure the technical infrastructure is set up to space out messages becomes essential. In general, results suggest a need to focus more on message delivery timing in digital health projects via TM. While there have been some innovations in research on the timing of content delivery in digital health interventions,³⁵ more research is needed that engages with nuanced, testable operationalizations in specific study contexts (e.g., LCS). We added a slight delay between consecutive messages in hopes of reducing patient frustration with content delivery.

The most important identified issue was participants feeling overwhelmed by the choices presented to them, which unintentionally resulted in confusion and disinterest among participants. Health information overload can happen among health consumers for various reasons,³⁶ though we did not expect that the limited options presented here would result in such outcomes. We shifted language and choice options to avoid participants feeling overloaded by choices, ensuring that choices were unique and informing patients more clearly how to proceed based on their response.

Related, the inability of the rules-based chatbot to handle multi-selection inputs identified an area where the chatbot needed to handle errors more effectively to avoid user frustration and disengagement. We added system error messages to account for participants entering more than one number to avoid lags in system responses, dead ends, and content presentation lapses.

We found few culturally specific adaptations required for improving delivery, as when issues were brought up about delivery there were few unique concerns raised (i.e., all participants shared similar technical concerns). As such, cultural adaptation for digital health interventions seems to be much more of a content concern than a technical delivery consideration.

Limitations and Future Directions

The studies had important limitations. First, recruitment was done primarily via email. As such, patients with lower digital access and literacy may not be represented in the current study. Second, the small sample size for the Spanish-language usability assessments may not have captured as comprehensive feedback as from sessions in English. Third, response rates for both studies were low and that could bias the current findings and limit generalizability to some other populations and contexts. Finally, the study was conducted in a relatively small geographic area with patients from one CHC. Findings regarding trust and clinic-specific branding may vary in different healthcare systems. Future research, including upcoming phases of LungSMART Utah, should continue to monitor perceptions of trust.

Conclusion

Our mixed-method formative evaluation demonstrates the value of coupling quantitative content evaluation with usability assessment observations when developing and delivering digital health content for linguistically and culturally diverse populations. A central finding is that meaningful cultural adaptation for Spanish-speaking patients requires moving beyond translation. Usability sessions identified that Spanish medical terminology (e.g., “cribado” for screening, “decisiones compartidas” for SDM) was unfamiliar to participants in a healthcare context, that vague cost framing (e.g., “bajo costo”) needed to be more specific and transparent, and that broader sociopolitical concerns (e.g., immigration status) created access-related anxiety that posed a likely barrier to engagement with digital health content. These insights would have been challenging to identify within traditional translation contexts, suggesting the need for thoughtful cultural adaptation in projects aimed at reducing health disparities and improving equitable implementation of screening programs. For research teams developing digital health tools in similar multilingual contexts, these findings demonstrate that multi-language usability assessment, not translation review alone, is essential to identifying terminology, framing, and contextual issues that could otherwise undermine engagement. Other key findings highlight the role of clinical partnerships and CHC branding to ensure patients view digital health content as trustworthy, as well as the importance of carefully timed message delivery to prevent patients from feeling overwhelmed or overloaded by information. Together, these insights provide guidance for developing patient-centered, population-level digital health strategies that meaningfully serve the needs of patients across demographic and linguistic groups.

Supplemental Material

Supplemental Material - Mixed-Method Formative Evaluation and Cultural Adaptation of Digital Health Communication Content and Delivery Promoting Lung Cancer Screening Shared Decision-Making

Supplemental Material for Mixed-Method Formative Evaluation and Cultural Adaptation of Digital Health Communication Content and Delivery Promoting Lung Cancer Screening Shared Decision-Making by Andy J. King, Indy Li, Jennyffer P. Morales, Anthony Banks, Ellen Wight, Kimberly A. Kaphingst, Cho Y. Lam, Peter A. Taber, Emerson P. Borsato, Chelsey R. Schlechter, Guilherme Del Fiol, Kensaku Kawamoto, and David W. Wetter in Cancer Control.

Footnotes

Acknowledgments

Thanks to the Association for Utah Community Health and our project advisory boards for their assistance with project efforts. The work reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under award number UG3CA287109. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

ORCID iDs

Andy J. King

Anthony Banks

Ellen Wight

Kimberly A. Kaphingst

Cho Y. Lam

Peter A. Taber

Emerson P. Borsato

Chelsey R. Schlechter

Guilherme Del Fiol

Kensaku Kawamoto

David W. Wetter

Ethical Considerations

The research activities reported here were approved by the University of Utah Institutional Review Board (IRB_00169869, approved May 2024; IRB_00185700, approved January 2025).

Consent to Participate

Participants provided consent prior to completing any research procedures.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Cancer Institute under award number UG3CA287109.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Supplemental Material

Supplemental material for this article is available online.

References

Siegel

Kratzer

Wagle

Sung

Jemal

. Cancer statistics, 2026. CA Cancer J Clin. 2026;76(1):e70043. doi: 10.3322/caac.70043.

US Preventive Services Task Force Krist

Davidson

, et al. Screening for lung cancer: US Preventive Services Task Force recommendation statement. JAMA. 2021;325(10):962-970. doi: 10.1001/jama.2021.1117.

de Koning

van der Aalst

de Jong

, et al. Reduced lung-cancer mortality with volume CT screening in a randomized trial. N Engl J Med. 2020;382(6):503-513. doi: 10.1056/NEJMoa1911793.

Field

Vulkan

Davies

MPA

, et al. Lung cancer mortality reduction by LDCT screening: UKLS randomised trial results and international meta-analysis. Lancet Reg Health Eur. 2021;10:100179. doi: 10.1016/j.lanepe.2021.100179.

Bradley

Shinkins

Kennedy

MPT

. What is the balance of benefits and harms for lung cancer screening with low-dose computed tomography? J R Soc Med. 2021;114(4):164-170. doi: 10.1177/0141076821991108.

Band

Landy

Star

Kratzer

Smith

Jemal

. Lung cancer deaths prevented and life-years gained from lung cancer screening. JAMA. 2025;334(24):2225-2227. doi: 10.1001/jama.2025.19798.

Kukhareva

Caverly

, et al. Inaccuracies in electronic health records smoking data and a potential approach to address resulting underestimation in determining lung cancer screening eligibility. J Am Med Inform Assoc. 2022;29(5):779-788. doi: 10.1093/jamia/ocac020.

Carter-Bawa

Lafata

Slaven

Jr. , et al. Navigating shared decision-making in lung cancer screening: insights into barriers, training, and referral propensity among clinicians. Patient Educ Couns. 2025;140:109303. doi: 10.1016/j.pec.2025.109303.

Rehman

Lim

Sidhu

Ramis

Rohren

. Barriers to lung cancer screening. Cancer Epidemiol. 2025;94:102722. doi: 10.1016/j.canep.2024.102722.

10.

Pradhan

Iyer

Rebbeck

. Geographic and temporal patterns of screening for breast, cervical, and colorectal cancer in the US, 1997-2019. JAMA Netw Open. 2025;8(10):e2537905. doi: 10.1001/jamanetworkopen.2025.37905.

11.

Carter-Bawa

. Shifting the lens on lung cancer screening inequities. JAMA Netw Open. 2024;7(5):e2412782. doi: 10.1001/jamanetworkopen.2024.12782.

12.

Landsteiner

Zerzan

Ullman

, et al. Shared decision-making for lung cancer screening: a systematic review. CHEST Pulmonary. 2025;4(1):100222. doi: 10.1016/j.chpulm.2025.100222.

13.

Odole

Andersen

Richman

. Digital interventions to support lung cancer screening: a systematic review. Am J Prev Med. 2024;66(5):899-908. doi: 10.1016/j.amepre.2024.01.007.

14.

Begnaud

Joseph

Lindgren

. Randomized electronic promotion of lung cancer screening: a pilot. JCO Clin Cancer Inform. 2017;1:1-6. doi: 10.1200/CCI.17.00033.

15.

Volk

Lowenstein

Leal

, et al. Effect of a patient decision aid on lung cancer screening decision-making by persons who smoke: a randomized clinical trial. JAMA Netw Open. 2020;3(1):e1920362. doi: 10.1001/jamanetworkopen.2019.20362.

16.

Lindsay

Odole

Belliveau

Abujarad

Gross

Richman

. Developing a digital video to support lung cancer screening in diverse populations. PEC Innov. 2025;6:100388. doi: 10.1016/j.pecinn.2025.100388.

17.

Hirsch

Hoover

Studts

. “They need to feel non-judgmental”: results of participatory photovoice research to inform lung cancer screening imagery. Cancer Control. 2024;31:10732748241292567. doi: 10.1177/10732748241292567.

18.

Hudson

Quinn

Wilson

Simmons

. Evaluation of promotional materials to promote low-dose computed tomography (LDCT) screening to high-risk consumers and health care providers. J Cancer Educ. 2018;33(5):1043-1051. doi: 10.1007/s13187-017-1204-9.

19.

Athey

Suckling

Tod

Walters

Rogers

. Early diagnosis of lung cancer: evaluation of a community-based social marketing intervention. Thorax. 2012;67(5):412-417. doi: 10.1136/thoraxjnl-2011-200714.

20.

Boatman

McCauley-Hixenbaugh

Starkey

Allen

Kennedy-Rea

. A new communication approach to encourage lung cancer screening action in rural eligible populations. PEC Innov. 2024;4:100298. doi: 10.1016/j.pecinn.2024.100298.

21.

NIH Pragmatic Trials Collaboratory . UG3 project: Population health management approaches to increase lung cancer screening in community health centers (LungSMART). https://rethinkingclinicaltrials.org/demonstration-projects/lungsmart/

22.

American Lung Association . State of Lung Cancer 2025: Utah. 2026. https://www.lung.org/research/state-of-lung-cancer/states/utah

23.

Michie

van Stralen

West

. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011;6:42. doi: 10.1186/1748-5908-6-42.

24.

Escoffery

Lebow-Skelley

Udelson

, et al. A scoping study of frameworks for adapting public health evidence-based interventions. Transl Behav Med. 2018;9(1):1-10. doi: 10.1093/tbm/ibx067.

25.

Damschroder

Aron

Keith

Kirsh

Alexander

Lowery

. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. doi: 10.1186/1748-5908-4-50.

26.

González Castro

Barrera

Jr. Martinez

Jr. . The cultural adaptation of prevention interventions: resolving tensions between fidelity and fit. Prev Sci. 2004;5(1):41-45. doi: 10.1023/b:prev.0000013980.12412.cd.

27.

Lakens

. Sample size justification. Collabra: Psych. 2022;8(1):33267. doi: 10.1525/collabra.33267.

28.

Rendle

Steltz

Cohen

, et al. Estimating pack-year eligibility for lung cancer screening using 2 yes or no questions. JAMA Netw Open. 2023;6(8):e2327363. doi: 10.1001/jamanetworkopen.2023.27363.

29.

Lee

Strong

Kahn

, et al. AIMQ: a methodology for information quality assessment. Inf Manag. 2002;40(2):133-146. doi: 10.1016/s0378-7206(02)00043-5.

30.

von Elm

Altman

Egger

, et al. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008;61(4):344-349. doi: 10.1016/j.jclinepi.2007.11.008.

31.

Wang

Liu

Zhou

, et al. Usability evaluation of mHealth apps for elderly individuals: a scoping review. BMC Med Inform Decis Mak. 2022;22(1):317. doi: 10.1186/s12911-022-02064-5.

32.

Gupta

Brenner

Ratanawongsa

Inadomi

. Patient trust in physician influences colorectal cancer screening in low-income patients. Am J Prev Med. 2014;47(4):417-423. doi: 10.1016/j.amepre.2014.04.020.

33.

Greene

. Patients’ perspectives on trust and trustworthiness of health care organizations. Milbank Q. 2022;100(2):365-369. doi: 10.1111/1468-0009.12556.

34.

Martens

Crutchfield

Laping

, et al. Why wait until our community gets cancer?: exploring CRC screening barriers and facilitators in the Spanish-speaking community in North Carolina. J Cancer Educ. 2016;31(4):652-659. doi: 10.1007/s13187-015-0890-4.

35.

Morrison

Hargood

Pejovic

, et al. The effect of timing and frequency of push notifications on usage of a smartphone-based stress management intervention: an exploratory trial. PLoS One. 2017;12(1):e0169162. doi: 10.1371/journal.pone.0169162.

36.

Khaleel

Wimmer

Peterson

, et al. Health information overload among health consumers: a scoping review. Patient Educ Couns. 2020;103(1):15-32. doi: 10.1016/j.pec.2019.08.008.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

1.04 MB

0.00 MB