Development of Proposed Heuristics for Evaluating Multilingual Website Usability

Introduction

Websites are increasingly available in more languages as the world becomes more globalized. In the United States, the need for websites in languages other than English has grown (Sun, 2001; Mahdi et al., 2014). In 2019, 67.8 million people living in the U.S. (almost 1 in 5 of the population) who were aged 5 years and older reported speaking a language other than English at home. Of those, 19.1% rated their own English-speaking proficiency as “not well” or “not at all” (Dietrich & Hernandez, 2022). It is important for individuals living in the U.S. with limited English proficiency (LEP) to have equitable access to critical safety and benefits information online, such as natural disaster emergency warnings in their areas or financial benefits they may qualify for. In large part, this means having access to information they can understand and respond to. Lack of high-quality translated content and poor usability of translated websites blocks equitable access. Critical websites must be usable as well as multilingual.

A multilingual website is a website which was initially developed in one language (“A”) and then translated or otherwise modified to be used by speakers of another language (“B”). Multilingual usability implies that the translated or otherwise modified site is not only usable, but is also culturally competent; essentially, that the content and interface design of the site in Language B can function effectively within the user's cultural contexts and needs (National Prevention Information Network, 2021; Sun, 2001).

Methods: Initial Heuristics Creation

We first reviewed the literature for suitable multilingual usability heuristics. The following search keywords were used: multilingual; multi languages; bilingual; language; translation; standards; criteria; heuristics; principles; best practices; examples; approach; framework; website; system; and, graphical user interface accessibility. We did not discover validated multilingual heuristics in a peer-reviewed literature search. When we were unable to find a heuristic framework accounting for both system multilingualism and usability, we then examined peer- and non-peer-reviewed articles related to the keywords and selected the closest approximations to multilingual usability heuristics: Nielsen’s 10 Heuristics (Nielsen, 1994), chosen for its wide usage among usability practitioners (Gonzalez-Holland et al., 2017) and for its applicability to websites in general; and Digital.gov’s Top 10 Best Practices for Multilingual Websites (Godfrey et al., n.d.), guidance for managing multilingual government websites. We selected the latter because the principles seemed easily comparable to a heuristic format; because the principles are popularly used, evidenced by how frequently they are referenced; and because of Digital.gov’s reputation as a reliable source on civic technology development.

Although we did not find published testing or validation of Digital.gov’s 10 Best Practices, we believe the principles seem suitable to the goal of conducting comprehensive heuristic evaluations on multilingual websites. Digital.gov’s Best Practices are frequently referenced by both government (Limited English Proficiency Committee, 2021; Child Care State Capacity Building Center, 2019) and industry (Migration Policy Institute, n.d.; Taoti Creative, 2014) websites as recommendations for multilingual website development. Digital.gov, made up of a cross-functional team of writers, editors, strategists, technologists, and designers who work for the U.S. Federal Government (Digital.gov, n.d.)., provides trusted guidance on building effective and accessible digital services. In sum, Digital.gov’s Top 10 Best Practices for Multilingual Websites provides a robust overview of considerations for managing multilingual websites, is frequently replicated and referenced by others, and was developed by a reliable source.

Best practices are useful as generally accepted techniques for producing desired outcomes, but they lack the reliability of validated usability heuristics, a proven engineering method to assess a system's usability thoroughly and efficiently. Additionally, best practices are intended to be suggestions for a development process, while heuristics are evaluative statements. Therefore, we adapted Digital.gov’s Best Practices into concise, descriptive statements to serve as actionable heuristics supporting usage to identify issues, evaluate websites, and inform design. For example, Digital.Gov's Best Practice #3 is named Access to Multilingual Information and reads, “Enable users to find your multilingual website via prominent access on the English site. Access to multilingual websites should be made available on the global navigation on the top right of every English page. The Federal Trade Commission provides the “En español” link as part of their global navigation on their English site.” We adapted this to the stand-alone, action-focused heuristic M_i1: Make access to multilingual options discoverable. We edited the description to be more succinct, informative, and focused on usability concepts, as seen in the heuristic’s Item Detail (Table 1).

OPEN IN VIEWER

Table 1.

Initial Proposed Multilingual Usability Heuristics.

Item	Item Name	Item Detail
Mi1	Make access to multilingual options discoverable	Enable users to easily find multilingual state via prominent access
Mi2	Communicate in users’ preferred language	Correctly translate information, considering cultural relevancy and appropriateness
Mi3	Provide a culturally relevant experience	Ensure the experience is culturally relevant to achieve an emotional connection with the audience
Mi4	Ensure comparable experiences across languages	Ensure that the multilingual states provide a comparable user experience to the English state
Mi5	Allow users to toggle between languages	Enable users to toggle between English and multilingual states
Mi6	Manage language expectations	Warn users when they are going to navigate to an English-only state

Based on relevance to multilingual web usability, the 10 converted heuristics were down-selected to 6 Proposed Multilingual Usability Heuristics, shown in Table 1. We use M_i1–M_i6 shorthand for the Initial Multilingual Heuristics, where M_i1 refers to the first, M_i2 to the second, etc. The i subscript indicates “initial.” URL Strategy was removed because it describes Search Engine Optimization as part of marketing strategy, not usability for multilingual audiences. This principle suggests using a unique URL in users’ preferred language so that users who speak other languages can more easily find the site when performing general searches. Presentation of links and findability from a user-centered perspective are covered by other principles concerned with discoverable access to multilingual options and communicating in the user’s preferred language.

Integrated Operations and Marketing and Outreach are, likewise, not directed at interface usability. These were eliminated because they address channels of translated content offerings (phone, chat, email, social media, etc.); and the use of translated content in marketing and outreach materials; both of which are outside the scope of website usability.

We removed Digital Features and Functionality because of its overlap with Comparability and Maintenance. Where a multilingual website provides a comparable user experience to the English site, it will, by definition, “provide similar digital features and functionality” (Digital Features and Functionality principle) on any translated webpages of the English site. Experience accounts not just for content, look, and feel, but for interactions vis-à-vis features and functionality.

Results: Initial Heuristics Creation

We proposed a new set of heuristics for multilingual website usability assessment, shown in Table 1, based on Digital.gov’s Best Practices. Item Details provide guidance on interpreting and deploying the heuristics.

Methods: IRS Case Study

Next, we selected multilingual websites on which to perform heuristic evaluations using both Nielsen’s 10 Usability Heuristics and the initial proposed Multilingual Usability Heuristics. A bilingual, native Korean-speaking usability expert in the team carried out two multilingual usability evaluations on unpublished IRS applications in their Korean and English versions: Sales Tax Calculator and Earned Income Tax Credit (EITC) Calculator. IRS web tools were chosen because IRS services offer critical information for taxpayers to be able to accurately fulfill tax obligations. If an IRS service has limited multilingual support, LEP users may have difficulty accessing information that is available to English speakers. These specific applications were chosen because they were readily available to the team in Korean and English. The outcomes were then reviewed by several other usability experts in English.

Results: IRS Case Study

The multilingual heuristic evaluation was successful in evaluating the IRS web applications because it discovered usability issues and opportunities to improve usability in the websites, fulfilling the goals of the heuristic evaluation method. The usability experts identified several usability issues in detail and with actionable recommendations using the Multilingual Usability Heuristics and Nielsen’s 10 Heuristics together. The most violated Multilingual Usability Heuristic across the two apps were M_i2, which identified three issues; M_i1, which found two issues; and M_i4, which also found two issues. Overall, the Multilingual Usability Heuristics (specifically, M_i1, M_i2, and M_i4) identified three issues that Nielsen’s 10 Heuristics did not identify.

Four issues violated both the Multilingual Usability Heuristics and Nielsen’s 10 Heuristics: once for M_i1 and N4 (Consistency and Standards) together, twice for M_i2 and N4 together, and once for M_i6 and N4 together. Overall, N4 was the only Nielsen’s Heuristic that co-occurred with Multilingual Usability Heuristics. Note that N1–N10 refers to Nielsen’s 10 Heuristics: N1 to the first, N2 to the second, etc.

Violations of Nielsen’s 10 Heuristics without the Multilingual Usability Heuristics occurred seven times. Three issues violated more than one of Nielsen’s Heuristics at a time: one violated both N1 (Visibility of System Status) and N5 (Error Prevention); one, N4 and N5; and another, N5 and N6 (Recognition Rather than Recall). Four separate issues violated a single Nielsen’s Heuristic each: N2 (Match Between System and Real World); N4; N5; and N10 (Help and Documentation).

Methods: Final Heuristics Creation

After evaluating the results of the case study, our team of three usability experts reflected on where the initial proposed heuristics succeeded and where they raised further questions or were difficult to apply and edited the set accordingly. We considered how often the heuristics identified issues and consulted other bilingual usability experts to update and confirm the final Multilingual Usability Heuristics.

Results: Final Heuristics Creation

Based on the case study, we updated M_i2 to address accuracy and consistency of language. Table 2 reports the Final Proposed Multilingual Usability Heuristics. M1–M6 (no subscript i) are used to refer to the Final Multilingual Heuristics.

OPEN IN VIEWER

Table 2.

Final Proposed Multilingual Usability Heuristics; changes from initial heuristics are shown in italics.

Item	Item Name	Item Detail
M1	Make access to multilingual options discoverable	Enable users to easily find and manage options to change site language
M2	Communicate in users’ preferred language accurately and with consistency	Correctly translate information, considering cultural relevancy and appropriateness. Ensure translated content is accurate and consistent, internally and externally, especially with regard to wording and tone. Only localize translated webpages if it is in line with the organization’s existing style or if you have evidence to support localization; otherwise, keep translated pages globalized.
M3	Provide a culturally relevant experience	Ensure the experience is culturally relevant to achieve an emotional connection with the audience
M4	Ensure comparable experiences across languages	Ensure that the multilingual states provide a comparable user experience to the English state
M5	Allow users to toggle between languages	Enable users to toggle between English and multilingual states
M6	Manage language expectations	Warn users when they are going to navigate to an English-only state

Accuracy came up as a concern when the heuristic evaluation found content that was translated incorrectly; for example, a Korean-language page used the transliteration of the English word “edit” (phonetic spelling in Korean characters of the English word “edit”) rather than the translation to the Korean word for “edit.”

Consistency has several facets, from the local, internal to a product or grouping of products, to the global, consistent to established norms external to the product. Internal consistency includes uniform translation of words and phrases; content style (tone, level of formality, etc.); visual elements and style; page, element, and content layout; and interaction patterns and element behaviors across a page or family of pages. It also includes consistency across a user’s journey through the service, such as consistency of experience at every point in a multi-step check-out process on an e-commerce website. External consistency can include content style; page, element, and content layout; and interaction patterns and element behaviors; but in this case, their appearances and behaviors fit conventions of related and unrelated products. For example, matching a common multi-step check-out process used by multiple popular e-commerce websites can support external consistency.

The case studies discovered several internal consistency issues. On one webpage, translation practice was inconsistent when two instances of the word “edit” were treated differently: one instance as a translation of “edit” to the Korean “편집,” and another to the phonetic transliteration of “edit” as “에디트.” While in this case, the method consistent with the rest of the site was to translate “edit,” there are circumstances in which transliteration could be appropriate – but the choice to translate or transliterate must be carried out consistently. On several occasions, links that brought the user to English-only pages did not warn they would redirect to an English-only page, also violating M_i6. There were a few instances of untranslated words: 1. The word “residence” was not translated in a form question; 2. Error messages were often untranslated across pages. Besides violating M_i4, failing to translate all content also violates external consistency because it breaks the wide-spread norm of content on a Language X website being fully available to the user in Language X.

Several accuracy and consistency aspects arose through discussion when creating the final heuristics, though they had not emerged from the case study. One was internal consistency to the organization's writing style, specifically, level of formality as it relates to tone. Korean and many other languages modify words depending on the audience and the relationship between the author and audience (Winter & Grawunder, 2012). Level of formality of any specified content should correspond to the overall formality of the organization’s website to ensure internal consistency. Content formality style should also be externally consistent with expectations for the organization. For example, official government websites typically use a more formal tone to match user expectations, while social media platforms tend to adopt an informal tone.

Localization also emerged in Final Heuristic Creation as a pertinent topic for multilingual websites. Localization is the adaptation of translated content to the linguistic and cultural needs of a different target population (Sandrini, 2005), including changing surface level features (content language, currency, measurements, etc.) and cultural level features (logic, functionality, images, appeals, etc.). For example, the English word “pants” refers to an outer garment extending from the waist down the leg in the United States, whereas in the United Kingdom, it means underwear (Editors of Merriam-Webster, 2016). Not localizing the concept can result in confusion and even offense. Localization can apply to language; cultural assumptions; cultural references; humor; currency; date; time; address; phone number; color; and graphics. Organizations may also choose not to localize their multilingual products or services, and to instead globalize them – make them understandable broadly across multiple regional markets. Globalization is its own process and does not simply mean the absence of localization (Sandrini, 2005).

Heuristic M2 suggests deciding to localize or globalize the site based on the organization’s existing style (ensuring internal consistency) or on compelling evidence, if available, to support one choice over the other. It is likely best to conduct user research to discover which approach is most appropriate for the system, and then verify that the translated design and content accurately communicates intended information and functions to the user (Schoua-Glusberg et al., 2018). The fields of translation, localization, and globalization offer tools to create and iterate localized or globalized content (Sandrini, 2005).

Lastly, heuristic M_i1’s Item Detail was updated for readability and clarity. No other heuristics were changed.

Discussion

As websites increasingly becoming available in diverse languages and considering the tens of millions of LEP people living in the U.S., the need for multilingual websites in the U.S. is apparent and growing. Access to multilingual websites is critical to improving access to key resources and information.

Our Multilingual Usability Heuristics discovered three usability-related issues that Nielsen's 10 Heuristics did not. Conversely, Nielsen’s surfaced seven issues that the Multilingual Usability Heuristics did not. These results indicate that both Multilingual Usability Heuristics as well as general system usability heuristics like Nielsen’s are needed to effectively evaluate multilingual websites. The two sets filled each other’s respective gaps in identifying issues. Using both sets in a complementary method caught more usability issues than either alone, improving the usefulness and impact of the heuristic evaluation overall.

We have proposed a novel set of heuristics for multilingual web usability in response to a gap in the usability field. The Multilingual Usability Heuristics work alongside current usability heuristic sets and fit the usability heuristic review method. Each heuristic’s Item Detail gives additional guidance but is not comprehensive of possible interpretations, so we invite colleagues to contribute additional interpretations to the multilingual usability field. The novel Multilingual Usability Heuristics should be used in specific contexts:

We explored, iterated, and reframed existing Digital.gov best practices into heuristics. This work formalizes the best practices as heuristics and improves upon them so they can be easily used by UX experts. We have designed these new heuristics to be understandable, actionable, and geared toward identifying issues and proposing solutions. When used in tandem with established usability heuristics, the Multilingual Usability Heuristics strengthen the usability evaluation, enabling it to assess and improve web usability for users across multiple languages. As the need for multilingual usability gains visibility, we invite researchers to use and test these Multilingual Usability Heuristics and propose updates from their own experiences.

Limitations and Future Directions

There were several limitations to the Multilingual Usability Heuristics. First, they were tested through a single case study and expert discussion, not through a rigorous validation procedure. While this brings up questions about the heuristics’ validity, our goal is to propose a potential set of multilingual heuristics rather than to validate them. Second, the case study was conducted only in Korean and English. Our findings do not necessarily generalize to other languages. Further research is needed to ensure the Multilingual Usability Heuristics cover the cultural considerations unique to other languages. Third, we relied on Nielsen’s 10 Heuristics as the complementary set and did not explore other usability heuristics that could work with our Multilingual Usability Heuristics. However, we stand by the selection of Nielsen’s 10 Heuristics due to it being an industry standard, and we present them as broad rules of thumb rather than specific usability guidelines. Fourth, the Multilingual Usability Heuristics apply only to digital GUIs and do not cover other interfaces such as hardware and voice interfaces. Finally, the case study did not have a second Korean-speaking evaluator checking for any missed issues and errors.

Despite the listed limitations, these proposed Multilingual Usability Heuristics still serve as a sound starting point for usability experts to evaluate multilingual website usability.

There are several future directions we suggest for further investigation. The Multilingual Usability Heuristics should be expanded to apply to other interaction modalities and services, such as Voice User Interfaces and VR/AR interfaces. We believe the heuristics are applicable to industry as well as government systems because usability and multilingualism are relevant across both, but future work could evaluate how generalizable they are for industry systems. We also offer them to other researchers to test and validate. The Multilingual Usability Heuristics can serve as a starting point to fill the gap in multilingual usability literature and practice.