eHealth policy in Spain: A comparative study between general population and groups at risk of social exclusion in Spain

Abstract

Background

In recent years, the healthcare sector has experienced accelerated progress in terms of the inclusion of Information and Communication Technologies (ICTs) in its procedures and formalities. However, public administrations have been unable to adapt to the peculiarities of the most vulnerable groups. This leads to marginalization of at-risk groups within the healthcare system and contributes to the widening of the so-called digital divide.

Methods

Based on the analysis of the literature, three dimensions have been constructed to identify the level of inclusion of web portals: the perception of quality, the perception of usability, and the importance attributed to their content. In order to obtain data on each of these dimensions, a study was designed based on the user test methodology as a central element (identifying and evaluating 11 vulnerable groups) and, in parallel, a survey of the general population and heuristic tests.

Results

A high percentage of vulnerable people at risk of social exclusion are not receiving adequate healthcare due to the digitalization of the system and the implementation of scarcely inclusive health web portals, which pose significant barriers in service provision for the 11 selected collectives. Meanwhile, the general population is being introduced to eHealth benefits, although the latter did not have high levels of digitization or a broad portfolio of services.

Conclusions

The general population has a relatively positive perspective of eHealth services, but only a minority of them make intensive use of them, and they are not widespread in Spain as a whole. Meanwhile, the population at risk of exclusion lacks the skills and resources to make real use of eHealth, needing “digital intermediaries” from the social sphere to achieve results. eHealth policies are not taking into account people with higher levels of marginalization, aggravating their exclusion and the digital divide.

Keywords

eHealth telemedicine digital divide equity social exclusion health policies

Background

In recent years, especially in the last decade, the healthcare sector has experienced accelerated progress in the inclusion of Information and Communication Technologies (ICTs) in its procedures and processes^1–5 via eHealth, which consists of a series of electronic tools used in a variety of tasks related to health and lifestyle management.⁶

The appearance of COVID-19 shows that new technologies are also important allies in the management of this pandemic, offering a wide range of opportunities related to new services, greater productivity, easier communication, etc.⁷

In this context, despite the existing consensus on the relevant role of ICT in health, the starting hypothesis of this research is that most public administrations have been unable to adapt to the peculiarities and demands of those groups that have been left out of the health system, thus contributing to the widening of the digital divide, this is understood as the differences that occur between individuals, households, companies, and geographic areas of different socioeconomic levels in terms not only of their opportunities for access to ICTs but also in terms of the usability and interaction capabilities of certain social groups.^7–22

Scientific contributions have repeatedly related the digital divide to various factors that explain its origin and impact on society. Among them, age,²³ lack of educational background,²⁴ gender,²⁵ geographic setting and the rural-urban dichotomy,^8,26,27 income level,^28–31 employment status,³² migrant status,³³ and finally, functional or psychomotor issues that may condition individual personal situations.^12,34 Consequently, ICTs contribute to more democratic social interaction but also to creating new patterns of inequality and new forms of division.³⁵

In this regard, taking the specialized literature as a reference, the following 11 groups were identified (Table 1) as revealing serious difficulties in accessing digital health services due to the inequalities generated by both social exclusion and the digital divide.

Table 1.

List of groups identified and main characteristics.

Socially excluded groups	Features
People with limited resources	Having a low income presents a wide range of problems for families, health care access is the most complex and prevalent.³⁶
Older adults	The literature observed the existence of a very pronounced digital divide between young and old adults³⁷ that results in the difficulty, and even exclusion, of the latter to have equal access to health services.
Economic immigrants	The immigrant population has been affected by social determinants of health such as stress, poverty, food and housing insecurity, lack of educational attainment and problems with access to health care. Regarding this last determinant, immigrants point out linguistic and cultural barriers, those related to the interaction with health personnel and financial difficulties as the main barriers to this access.³³
People with mental pathologies	This population finds significant barriers in accessing the health system, especially those related to the complexity of the process, but also those related to experiences of stigmatization in health centers or hospitals³⁸.
Population deprived of liberty or in semi-open regime	This population often faces problems related to a lack of digital literacy due to their isolation in prisons. However, it is a group that presents marked health care needs due to the prevalence of infectious and psychiatric diseases, as well as dental diseases caused by substance dependence, which also results in a higher mortality rate than the rest of the population.³⁹
Substance use disorders	People with drug dependence are more likely than other patients to be in need of medical care. However, they often face a number of problems and barriers that make it difficult, such as a lack of rehabilitation programs⁴⁰ or stigma among healthcare personnel.⁴¹
People providing paid sex (prostitution)	Despite suffering from major problems related to their health status, such as depression, tuberculosis, or numerous sexual aggressions,⁴² some studies report the existence of barriers to their health care, such as lack of information on where to go for treatment or how to obtain a health card.⁴³
Gypsy population	Gypsy people present social inequalities in health as evidenced, for example, by mortality and morbidity in transmissible, chronic and environmental diseases, something that, in general, has been associated to socioeconomic and cultural factors, as well as to poor access to health services and low utilization of these services.^44,45

Source: Own elaboration.

The blind population was also added to these groups, as it is understood that they are at a disadvantage in their interaction with the different Internet connection platforms and, more specifically, with web pages; as well as, for the same reasons, people affected by functional diversity.

In this respect, it should be noted that Spain has a decentralized territorial system in which the different Autonomous Communities (AC) assume the health matter and its provision. This article presents the results of a research work developed between 2018 and 2021, which aims to verify how autonomous health systems address the limitations that different groups of users or collectives, in exclusion or at risk of social exclusion, experience in their interaction with electronic health services, determining how inclusive these web portals are (or are not).

Methods

A comparative analysis has been carried out between the health web portals of the 17 Autonomous Communities to check how the health systems address the limitations that different groups of users have in accessing and interacting with eHealth services. A research method was applied that combines quantitative and qualitative techniques based on two tools widely used in the literature on website usability: the heuristic test and the user test.^46–56

In order to detect differences in the knowledge and use of eHealth tools between the general population and vulnerable groups, a third technique was applied as a survey to elucidate, from a representative sample, the general perception that citizens have of these portals.

This combination of techniques is used with the aim of observing the contrast between the general perception and the particular perception of the selected groups, with heuristic tests as a preliminary framework for the detection of services and characteristics. This diversity of tools and approaches also responds to the extreme difficulty of having a representative sample of the people who are part of the user test target, which requires the use of qualitative techniques to obtain information.

To organize the results obtained from this triple approach, three dimensions were constructed to identify the level of inclusion of web portals: the perception of the quality of the portal, the perception of usability, and the importance attributed to its content in relation to the information and the services available on it.

In addition, to obtain data on each of these dimensions and to verify the way in which groups of people are excluded from eHealth services, as well as to determine the possible barriers that websites can generate in the provision of the service, a qualitative study based on the methodology of the user test was chosen as the central element of the research.

Digital inclusion requires inclusive websites. This factor, the reverse of the digital divide, goes beyond mere digital literacy, insofar as it not only consists, according to Olarte Encabo,⁵⁷ in the acquisition of digital and informational, educational and knowledge skills. It also entails a real social impact, measured in the improvement of the living conditions of individuals and groups. The European Commission⁵⁸ points out that digital inclusion implies that all people can contribute to and benefit from the digital economy and society through a set of measures such as ensuring access to ICT, the development of assistive technologies aimed at people with disabilities, the digital literacy of the population, as well as social inclusion, focusing on the most disadvantaged sectors of society with specific programs to help them join the digital world.

Digital inclusion in the field of web pages is reflected in two main attributes: accessibility and usability. While accessibility is the possibility for the greatest possible number of people to access and use the functionalities of a web page, regardless of the individual's limitations or those derived from the context of use,⁵⁹ usability is a quality attribute that evaluates the ease of use of web pages.⁶⁰

The three dimensions of analysis outlined above derive directly from studies on website quality. This approach is based on all the scientific literature reviewed and commented on this initial section, attempting to produce a synthesis of the services from the user's perspective. Although there is no unanimity among the scientific and professional community on what web quality is, and in the absence of a unified formal definition, there are proposals⁶¹ to define it as the ability of a website to meet the expectations of its users and owners, determined by a set of measurable attributes.

The first dimension focuses on the analysis of the perception that users have of the general quality of the regional health portals; the second focuses on the study of the usability of these sites in order to know; the third dimension is the importance that users give to the two main resources of a health website: information and online services.⁵³

According to Codina and Pedraza-Jiménez,⁶² there are different methods for evaluating the quality of websites that can be grouped into two major subsets: user studies and expert analysis. Among the first ones are the works on web usability. In the second group, the main model took the form of heuristic testing, that is, an analysis carried out by experts. This technique consists of evaluating the elements of an interface on the basis of a checklist.

This technique makes it possible to observe how end users interact with a website by subjecting a group of volunteers—in this case, a representative sample of excluded groups—to various interaction tests with the site to record the problems they encounter while browsing.⁶³ The test consisted of a first part in which a pretest questionnaire was carried out to assess the health status profile of the participant, as well as the use that the person makes or has made of the web pages of the regional public health services. In the second part, the person is subjected to two preliminary tests (access to the web page and first evaluation of its initial content) and three main tests (identify a label, carry out a procedure or service, and search for information on a topic present on the page). Finally, in the third part, a questionnaire is carried out after the main tasks and a post-browsing test is performed.

Sample user test

As Travieso and Planella Ribera³⁵ points out, it is important to remember that when dealing with problems related to groups in vulnerable situations, stigmatization should be avoided. However, it is useful for research to define social groups whose members share norms, codes, forms of communication, behavior, experiences, etc., to establish categories delimited by different types of personal and social needs or deficiencies. This facilitates monitoring, evaluation or even the reformulation of intervention programs.

In accordance with the user profiles outlined in Table 1, fieldwork was designed to include all of them, as shown in Table 2. In order to put the data collected into context and understand the functioning of the health web portals, two other techniques were applied in parallel to cover each of the above needs.

Table 2.

List of groups, collaborating entities, and interview locations.

Group	Collaborating entity	Interview location
People with mental pathologies	Asociación de Familiares e amigos dos enfermos mentais Fonte da Virxe (Fonte da Virxe Association of Relatives and Friends of the Mentally Ill)	Office of the Asociación de Familiares e amigos dos enfermos mentais Fonte da Virxe, in Santiago de Compostela
Economic immigrants	Cáritas Diocesana de Santiago de Compostela (Diocesan Caritas of Santiago de Compostela)	Office of the Cáritas Interparroquial de Santiago, in Santiago de Compostela
People with limited resources	Cociña Económica de Santiago (Economic Cociña of Santiago)	Cociña Económica de Santiago, in Santiago de Compostela
Homeless people		Cociña Económica de Santiago, in Santiago de Compostela
People affected by functional diversity (physical, psychic or sensory)	Confederación Galega de Persoas con Discapacidade (COGAMI) (Galician Confederation of People with Disabilities)	Central office of the COGAMI Galicia, in Santiago de Compostela.
	Fundación Down Galicia (Down Galicia Foundation)	Office of the Down Galicia (Down Compostela Building), in Santiago de Compostela
Older adults	Cruz Vermella Santiago (Red Cross Santiago)	Office of the Cruz Roja Santiago, in Santiago de Compostela
People deprived of liberty or in semi-open regime	Emaus Fundación Social (Emaus Social Foundation)	Office of the Emaus Fundación Social, in Vigo
Gypsy population	Fundación Secretariado Xitano (Xitano Secretariat Foundation)	Office of the Fundación Secretariado Xitano, in Santiago de Compostela
Blind population	ONCE- Santiago de Compostela	Office of the ONCE- Santiago de Compostela, in Santiago de Compostela.
Blind population	ONCE- Santiago de Compostela	Facultad de Ciencias Políticas y Sociales de la Universidad de Santiago de Compostela, in Santiago de Compostela
People providing remunerated sex (prostitution)	Programa Vagalume (Cáritas) (Vagalume Program—Caritas)	Office of the Programa Vagalume, in Santiago de Compostela
Substance use disorders	Unidade Municipal de Atención á Drogodependencia-UMAD Santiago de Compostela (Municipal Unit of Attention to Drug Dependency–UMAD Santiago de Compostela)	Office of the municipal de la UMAD-Santiago, in Santiago de Compostela

Source: Own elaboration.

The sample survey

To get a better understanding of the context, a questionnaire was carried out with questions regarding citizens” knowledge, use, and valuation of eHealth services. The interviews, lasting between 5.7 ± 12.3 min, were carried out in Spanish territory (excluding Ceuta and Melilla) to a population of legal age with legal residence in Spain between May 24 and June 21, 2018. Data collection was performed with the assistance of a computer with script and interface (CATI; Computer Assisted Telephone Interviews).

The sample survey consisted of 1695 interviews and was based on proportional criteria, using quotas to capture relevant differences and ensure adequate representativeness. The telephone directory used (Infobel) territorially located the landlines and cell phones that were called for the interview. The selection of cases considered, in addition to the size of the municipalities of residence, other sociodemographic profile characteristics, such as gender, age, and housing, with a confidence level of 95% and a margin of error of ± 2.45 for the sample as a whole.

Heuristic tests

In order to know the functioning of the regional health web portals, heuristic tests were carried out in the initial stage of the project, so that, before the application of the user test, relevant information on the configuration and usability of these sites was available.^59,64–69

The heuristic test is a technique belonging to the “inspection methods”^51,70 in which a group of expert researchers analyze the web space and describe potential problems by applying a list of criteria. For this study, it was decided to adapt the following dimensions of analysis from the work of Bouzas and Mahou⁵³ and of Mahou and Varela⁷¹: usability, previously defined, information offer, and online services.

Thus, the electronic service can be defined as a set of services implemented by an entity through applications stored on their respective websites (the web pages that have been analyzed for the research can be consulted in the annex). On the other hand, the information offer includes a range of data and contents related to a specific area of competence that public administrations make available to users in a unilateral and unidirectional way.⁵³ The following figure allows us to identify the interconnection between the data collection instruments used in the research and the objective sought (Figure 1).

Figure 1.

Techniques used and organization of the project. Source: Own elaboration.

Results

Perception of web portal quality

Most of the people who participated in the user test consider the existence of the web portals positively, with 58.82% having a good or very good perception of their quality. At the same time, they point out that, from their personal experience, these platforms are not useful, either because they do not have the necessary knowledge to take advantage of the full potential they offer or because they do not have the appropriate means of connection.

Similarly, the results of the survey indicate that 47.55% of the people surveyed consider the web portal of the health service of their autonomous community to be of high quality. In general terms, 66.9% rate the web page as good, compared to 18.4% who rate it as regular.

As for the “experts” analysis through the heuristic test, the data reveal that the overall comprehensibility of the websites is good (52.9%), although in 7 Autonomous Communities the design and style are confusing, with Catalonia being the Community with the worst score in this dimension.

Regarding the quality of the contents, 70.6% of the health portals adequately highlight the most important information; and 82.4% of them, except for the websites of Catalonia, Extremadura, and Murcia, organize it in a coherent and categorized manner.

Although most websites do not clearly distinguish between information and services, 53% of Autonomous Communities have a catalog of online services. However, Aragon, Murcia, and the Basque Country, although they do have a catalog, it is confusing both in terms of content and organization. Regarding the presence of personalized profiles, the vast majority (94.1%) show the contents according to the user profile and only Aragon lacks this option.

In terms of style and color, all the web portals show consistency and coherence, as well as the contrast between the texts and the background. In three out of four Autonomous Communities, the texts are legible and the font size and spacing allow for comfortable reading and interaction, but not in the websites of the Canary Islands, Castile and Leon, and Castile-La Mancha and Extremadura.

Finally, the data on multimedia materials and the availability of windows for displaying content indicate that their presence does not represent a problem in site navigation for 82.4% of the Autonomous Communities.

Perception of web portal usability

The representatives of the different groups—user test—showed in the tests serious difficulties in the use of the regional web portals. In the main tests as well as in the post-questionnaire and in the post-navigation test, a majority of people are unable to complete the management in an optimal way. In the information access test and the services test, the participants attribute this inability to various causes, from the inadequate design of the pages and the location of these resources in a peripheral place on the pages, to the numerous actions they have to take to complete the management. This causes them to abandon the task on numerous occasions, try to access through other unofficial websites or need help to return to the page and carry out the tests. At this point, migrants are more vulnerable as they are not in possession of a health card and are therefore excluded from the system.

Regarding the labels, their identification generated confusion in most cases and the interviewees stated that they did not know what their statement was alluding to or referred to inappropriate and inaccurate content. In this respect, some of the comments point to a language not very close and very technical.

The following table shows the usability problems detected by each group (Table 3).

Table 3.

Main usability problems by group.

Group	Main problems	Potential solutions
People with limited resources	Problems finding the requested information (opening pop-up windows generates confusion).	Make adjustments to the design of health web portals so that all page content appears on one main screen, with no new browser windows opening and no advertising elements appearing.
	Excessive advertising elements reduce page reliability.
	The data required to access some services (e.g., MCH, prior appointment) generates confusion.	Simplify processes. Often, to make an appointment or to consult the MCH, health card data are requested. These data respond to concepts with which users are unfamiliar, such as the social security number. Therefore, one way to simplify the process would be to request more common user data, such as the telephone number or even the ID card number.
Older adults	The vocabulary used by the site is unfamiliar and acts as a barrier to accessing some options.	Rethinking and using a more colloquial and user-friendly language. Sometimes too much technical vocabulary is used, which can be confusing for certain groups.
	The font size is very small.	Adapt the font to a larger size, or, in any case, offer different text size options, so that people with vision problems do not encounter problems in their navigation and can choose the size that best suits their circumstances. Extend the use of text-to-speech reader tools.
	Advanced computer skills are required to take full advantage of the site.	Governments should offer digital training courses adapted to different groups. This training should precede the implementation of eHealth policies.
	The data required for some services (e.g., MCH, prior appointment) generates confusion.	Simplify processes. Often, to make an appointment or to consult the MCH, health card data are requested. These data respond to concepts with which users are unfamiliar, such as the social security number. Therefore, one way to simplify the process would be to request more common user data, such as the telephone number or even the ID card number.
Economic migrants	The site has no other language options (language conversion).	It would be convenient to adapt the web portals to other major languages, such as English, French or Arab, in addition to the official languages of the State.
	The site does not have its own search browser.	Incorporate a search browser.
	Important services, such as prior appointments or information on diseases, are poorly positioned on the site.	Make design and usability readjustments, so that the main services or those most used by users, such as requesting an appointment in primary care, appear in the central part of the screen and in a shortcut.
	The data required for some services (e.g., MCH, prior appointment) is confusing.	Simplify processes. Often, to make an appointment or to consult the MCH, health card data are requested. These data respond to concepts with which users are unfamiliar, such as the social security number. Therefore, one way to simplify the process would be to request more common user data, such as the telephone number or even the ID card number.
People with mental pathologies	The font size is too small.	Adapt the font to a larger size, or, in any case, offer different text size options, so that people with vision problems do not encounter problems in their navigation and can choose the size that best suits their circumstances. Extend the use of text-to-speech reader tools.
	Inability to interact with some of the options offered by the pages.	Governments should offer digital training courses adapted to different groups. This training should precede the implementation of eHealth policies.
	The data requested for some services (e.g., MCH, prior appointment) generate confusion.	Simplify processes. Often, to make an appointment or to consult the MCH, health card data are requested. These data respond to concepts with which users are unfamiliar, such as the social security number. Therefore, one way to simplify the process would be to request more common user data, such as the telephone number or even the ID card number.
People deprived of liberty or in semi-open regime	Important services, such as prior appointments or information on diseases, are poorly positioned on the site.	Make design and usability readjustments, so that the main services or those most used by users, such as requesting an appointment in primary care, appear in the central part of the screen and in a shortcut.
People deprived of liberty or in semi-open regime	The vocabulary used by the site is unfamiliar and acts as a barrier to accessing some options.	Rethinking and using a more colloquial and user-friendly language. Sometimes too much technical vocabulary is used, which can be confusing for certain groups.
Substance use disorders	Important services, such as prior appointments or information on diseases, are poorly positioned on the site.	Make design and usability readjustments, so that the main services or those most used by users, such as requesting an appointment in primary care, appear in the central part of the screen and in a shortcut.
Substance use disorders	The vocabulary used by the site is unfamiliar and acts as a barrier to accessing some options.	Rethinking and using a more colloquial and user-friendly language. Sometimes too much technical vocabulary is used, which can be confusing for certain groups.
Persons providing paid sex (prostitution)	The site does not have its own search browser.	Incorporate a search browser.
	The design of the site is unattractive.	Make adjustments to the design
	The font size is very small.	Adapt the font to a larger size, or, in any case, offer different text size options, so that people with vision problems do not encounter problems in their navigation and can choose the size that best suits their circumstances. Extend the use of text-to-speech reader tools.
	Important services, such as prior appointments or information on diseases, are poorly positioned on the site.	Make design and usability readjustments, so that the main services or those most used by users, such as requesting an appointment in primary care, appear in the central part of the screen and in a shortcut, requesting an appointment in primary care, appear in the central part of the screen and in a shortcut.
Homeless people	The design of the site is unattractive	Make adjustments to the design
	Excessive advertising elements reduce page reliability.	Make design and usability readjustments, so that the main services or those most used by users, such as requesting an appointment in primary care, appear in the central part of the screen and in a shortcut.
	Advanced computer skills are required to take full advantage of the site.	Governments should offer digital training courses adapted to different groups. This training should precede the implementation of eHealth policies.
Blind people	Accumulation of content on the main page interferes with the use of speech synthesizer.	Optimally synthesizing the content of the main pages
	Pages do not have an integrated speech synthesizer.	Make design adjustments to allow the integration of a voice synthesizer or include this synthesizer in the web portal itself.
	The site does not have its own search browser.	Incorporate a search browser.
	The page contains moving images (flash images) that interfere with the use of speech synthesizers.	Reduce the use of images and restrict it to very specific occasions when it is necessary to explain, for example, the appearance of certain parts of the body in the presence of a disease.
Gypsy population	Advanced computer skills are required to take full advantage of the site.	Governments should offer digital training courses adapted to different groups. This training should precede the implementation of eHealth policies.
	Important services, such as prior appointments or information on diseases, are poorly positioned on the site.	Make design and usability readjustments, so that the main services or those most used by users, such as requesting an appointment in primary care, appear in the central part of the screen and in a shortcut.
	The data required to access some services (e.g., MCH, prior appointment) generates confusion.	Simplify processes. Often, to make an appointment or to consult the MCH, health card data are requested. These data respond to concepts with which users are unfamiliar, such as the social security number. Therefore, one way to simplify the process would be to request more common user data, such as the telephone number or even the ID card number.
	Excessive advertising elements reduce page reliability.	Make design and usability readjustments, so that the main services or those most used by users, such as requesting an appointment in primary care, appear in the central part of the screen and in a shortcut.
Persons affected by functional diversity (physical, mental, or sensory)	The design of the site is unattractive	Make adjustments to the design
	Important services, such as appointments or information on diseases, are poorly positioned on the site.	Make design and usability readjustments, so that the main services or those most used by users, such as requesting an appointment in primary care, appear in the central part of the screen and in a shortcut.
	The font size is very small.	Adapt the font to a larger size, or, in any case, offer different text size options, so that people with vision problems do not encounter problems in their navigation and can choose the size that best suits their circumstances. Extend the use of text-to-speech reader tools.

Source: Own elaboration.

These data contrast very significantly with the results of both the survey and the heuristic test. In relation to the survey, 88.1% of the people interviewed believe that the web portals are easy to use and 90.83% affirm that they were able to do what they were looking for when visiting the site. In contrast to the representatives of marginalized groups, 51.2% of the people interviewed who were unable to access a service or source of information attributed this to the fact that they were unable to find it, and only 17.6% considered the site to be difficult to use.

As for the heuristic test, the experts analyzed the usability dimension of the web portals on the basis of the following parameters:

“Design and style” or ease of use, complexity and clarity of content (value 39%).

“Web Browser usability” or use of home pages/landing pages without excessive loading of videos or images, correct adaptation to mobile screens and formats, with several languages and with the search browser function within the page enabled, together with the updating and functioning of links to specific third party websites (value 37%).

“Identity” or existence of logos and signs identifying the organization (value 8%).

“Security” or information on data protection and use of certificates (value 8%).

“Web browser for searches” or ease of detection of the website browser according to its position and visibility (value 5%).

From the data collected, experts highlight the high degree of usability of the sites in 10 Autonomous Communities. These portals achieve scores above 70 points (of a maximum of 100), while the rest reach average usability of between 57 and 75 points. The complete results can be seen in Figure 2:

Figure 2.

Average score of each AA. CC. obtained in the heuristic usability test (0–100). Source: Own elaboration.

Importance attributed to the content of the web portal: information and services

In this last section, the expert test made it possible to determine the existence and degree of information available on health websites. Specifically, the quality of information on the main eHealth services was analyzed: health card, choice of doctor, choice of medical center, basic appointment, appointment for tests and specialist, waiting lists, second medical opinion, prior instructions, complaints and suggestions, and eHistory and ePrescription.

The services with the best results were those related to the health card, prior instructions, eHistory, complaints and suggestions, all of them scoring more than 3.74 points (out of a possible 4). The lowest scoring services are choice or change of physician and second medical opinion, above 2 points, and choice of center, which drops to 1.65 points.

In general, information on these services is “good” or “very good,” although in most cases they are processed in person. To address this last question, the levels of digitization of services were analyzed. The results of the heuristic test show a reduced number of services offered in a fully online format. On the other hand, the low average obtained by some of the Administrations examined is because not all of them offer a full range of basic services, except for the application for the health card.

Examining the average scores according to the service categories analyzed, a mixed picture is detected. On the one hand, “health care facilitation services” (processing of the health card; request for a duplicate health card; modification and consultation of data, etc.) show an average of 19.71 out of 36 points. On the other hand, “documentation services” (reimbursement of healthcare expenses; registration of latest wishes; access to anticipated wills, etc.) obtain an average of 6.29 points out of 16, showing that this type of resource is scarcely implemented in the regional health systems.

Analyzing the data from the survey, data reveal that telematic appointment is the most used service. In terms of satisfaction with this aspect, users are generally satisfied with the information and in more than eight out of ten cases they found what they expected on the healthcare portals.

When comparing this information with that coming from the groups—user test—, some differences were found. Firstly, with regard to the results of the preliminary tests, it was verified that access to the page was difficult and in most cases required redirection to the reference URL, either because phrases incompatible for accessing the content were inserted in the browser, or because, even when entering the elements that could facilitate access, users were unable to select the correct results among those obtained.

The second part of the user test examines citizens” knowledge, access, skills, and uses of ICTs in relation to their health and well-being. In this respect, participants highlight the services offered on health websites as very useful, especially the service for requesting appointments in primary care. However, despite identifying this service as something positive, we have been able to identify a significant inability to reach it in an optimal way and to complete the management.

This tends to occur either because it is a service that does not occupy a central place on the page or because the data necessary for access are difficult and require many previous steps before concluding the management. On this point, migrants are more vulnerable, as evidenced throughout the user test, as they do not have a health card and are therefore excluded from the system in general and are unable to carry out many of the procedures allowed by the public health system websites.

The same happens with the test that refers to the search for information. Although it is interpreted as something positive, in many cases it is not known how to get to it due to an inadequate location within the page or to very complex processes involving several steps.

In the third phase of the test, based on the post-task questionnaire and the post-browsing test, issues related to the inadequate design of the pages were especially highlighted, including poor placement of relevant services (information content, request for appointment, consultation of medical record). Likewise, it is especially relevant that a high percentage of participants stated that they would need help if they had to use the page again and just as many others commented that they could not do the test if it were not guided. Finally, a lack of familiarity of the participants with the computer equipment used (laptop) was detected, as well as a lack of confidence and spontaneity when facing the tasks assigned in the test.

Discussion

The results obtained in this research corroborate the idea present in numerous studies that a high percentage of vulnerable people affected by risk factors are not receiving adequate health care due, in part, to the digitalization of the system and the implementation of scarcely inclusive health web portals.^53,72–74

Considering the three study dimensions applied in this research (quality, usability, content), the health web portals show significant barriers to service provision for the 11 selected groups:

− From the point of view of quality, although the majority of the population participating in the study has a good perception of the web portals, a high percentage do not find access to them useful for two main reasons: on the one hand, they do not have sufficient knowledge to make profitable navigation, in line with what has been expressed by some official organisms in recent reports⁷⁵; and, on the other hand, due to the influence of some factors such as age, education, level of employment or income which, added to a lack of adequate means of connection, influence the use of the different technological devices.⁷⁶

− From the point of view of the perceived usability of the websites, the 11 groups studied encounter significant limitations in the use of the portals. Most of the groups point out that the existence of these platforms is a positive development, but at the same time, it highlights their limited usefulness. The lack of material and immaterial resources leads these people to waste the resources available on the health portals.^75–78 They do not manage to obtain the information requested and/or complete the process of a service, making navigation a confusing and frustrating experience and causing the person to abandon the search or have to ask for help, depriving them of all autonomy. Focusing on each group:

The data obtained for older people are in line with the most recent reports on digital skills, which show that older people, pensioners, and people with lower levels of education and income have lower digital skills.⁷⁷ Thus, both people with limited resources and the older adults have difficulty in gaining equal access to health services.^36,37

For their part, economic migrants identify as one of the main barriers the absence of more linguistic options in the web portals, as revealed by some research.³³ In addition, immigrants consider that the information required to access some services is confusing or that they cannot attest them due to their irregular situation or because they do not know how the Spanish health care system works.^79,80

The group of people who provide sexual services, closely linked to immigration, presents access barriers very similar to those seen for the previous group, in addition to other obstacles related to the health care received, such as the lack of information on certain treatments and on obtaining a health care card.^43,81

In relation to people with mental pathologies, despite the approval in 2001 of the document International Classification of Functioning, Disability and Health⁸² by the World Health Organization and the development of policies and/or programs focused on eHealth,^4,83 these people find it difficult to browse websites due to the lack of facilities to complete health processes,³⁸ as revealed by the data obtained in this study.

In line with the literature,⁸⁴ the design of the page is not very attractive for people affected by functional diversity, highlighting the small font size and the poor location of the most important services or the lack of them. In the particular case of the blind population, the main difficulties derive from the absence of speech synthesizers or their internal malfunction.

With respect to the population deprived of liberty or under a semi-open regime, a study on the use of ICT by incarcerated people shows the significant digital training deficiencies of this group.⁸⁵ According to the results, these limitations are a major obstacle to access to eHealth information and services.³⁹ These people consider that they are not well located on the web page and that the vocabulary used is barely understandable. These same access problems are shared by the group of substance use disorders, whose situation is aggravated by their lack of participation in social activities⁸⁶ and by the stigma among health care personnel.⁴¹

Homeless people present important problems of access to health services, despite reporting worse levels of health than the general population.⁸⁷ This study, in line with some recent research, shows that this group encounters significant obstacles in terms of access through ICTs, due to the unattractive design of web portals and the need for advanced computer skills to take full advantage of the site.⁸⁸

Finally, the gypsy population is one of the groups most affected by social exclusion in Spain, especially in the field of health, which causes the health status of these people to be more deficient and to have a lower life expectancy.⁸⁹ Despite these data, present in the literature and contrasted with the results of this study, they are explained by the limited opportunities given to these people to have optimal digital training.^90,91

− From the point of view of the importance attributed to the content of the web portal (information and services), the data collected throughout this research are similar to other studies.^76,77,92,93 Thus, the perception among the general population about services and the fulfillment of expectations is relatively good, despite recurring demands such as increasing the reliability and trust of health information, improving the ability to access medical records and waiting lists, as well as the implementation of services such as telecare, telerehabilitation, or taking and sending medical data.

Conclusions

Despite the benefits of ICT application in healthcare management and care, especially observable during the COVID-19 pandemic, many vulnerable groups have remained digitally excluded from the opportunities and benefits of eHealth access and use. The digital divide, therefore, remains a defining element of the relationship between certain population groups and eHealth, contributing to the intensification of the medical poverty trap.

Throughout this work, the degree of usability of the resources hosted on the web portals of the health departments has been evaluated, both from the perspective of the expert in the operation of an institutional website and from the users” browsing experience, focusing on the study of three dimensions that measure the degree of inclusion of the web portals: the perception of the quality of the web portal, the usability of the web portals, and the importance attributed to the contents of the web portal (information and services).

The main conclusion drawn from the data collected from the three research techniques is that users belonging to the main groups in exclusion or at risk of social exclusion have a different perception of the regional health web portals from that held by experts and citizens.

First, although the perception of quality of websites by citizens and vulnerable users is high, there is a wide difference, with citizens generally attributing a higher quality to websites than vulnerable users.

Regarding the use of the sites, users report a low percentage of access to them. However, if health status is correlated with access, there are differences in use. Citizens with chronic illnesses, or who state that they are in fair or poor health condition, access web portals more frequently. On the other hand, most of the people who took the user test stated that they had never accessed them before.

Despite the low level of access to the web pages, the importance attributed by both user profiles to the regional health portals is high, especially to the online services. This is very notable because, as can be seen when addressing usability, the portals are fundamentally informative and efforts are aimed at providing relevant information on health.

Regarding usability, the perception is different according to the point of view of the general population, the vulnerable population, and experts. If, in general terms, the first one mostly states that the web portals are easy to use, the second one considers that the websites are not sufficiently adapted for people with little computer knowledge and scarce connection resources; the guiding of eHealth services through web portals and the lack of formats adapted to smartphones and mHealth are important elements in this aspect. For their part, the experts maintain an intermediate position, stating that the sites are well designed and have an attractive range of information, but with limited online service provision and practically non-existent mechanisms for participation in health management and policy.

From the point of view of technical usability, the main problem detected by citizens is the difficulty in locating information and services. Experts and users coincide in highlighting the difficulty of understanding some labels, the technical language and the absence of linguistic options. Users add to this list the inability to reach the information due to its location and the numerous steps that must be followed, the poor location of certain services, an unattractive design that is not adapted to viewing on smartphones—a tool that vulnerable groups point out as more widespread than computers—the small size of the font or the lack of speech synthesizer, among others.

With regard to the supply of information, while the public does not report any problems due to their low access to these resources, the experts highlight the general lack of information on patient and professional organizations and associations, as well as health agencies, and users admit their lack of knowledge about this type of information.

With respect to the dimension of online services, the data derived from the heuristic test indicate that the request for a health card, one of the central elements of health policies, shows discrete numbers, as does the request for a duplicate. Likewise, only one out of every four Autonomous Communities provides access to health care services. Similarly, only one in four Autonomous Regions provides access to test results. Services such as access to the latest wishes and requests for certificates or reports are not provided on most portals. Other services, such as access to waiting lists, changing/choosing a physician, or requesting a second medical opinion, are provided electronically and in person almost equally.

Despite the offer of various services in the web portals, the frequency of use is very low by citizens and users express their inability to reach most of them due to their location, the demand for a series of access requirements (data), and the numerous steps that must be followed to complete the service. In this regard, one of the limitations of this study is that the field conducted in 2018 could not consider the use of mobile applications given that few of the Autonomous Communities had them.

Therefore, in line with the starting hypothesis of this research, eHealth policies have not yet succeeded in creating inclusive health web portals, adapted to the needs and particularities of specific groups. This contributes to amplify the digital divide and to deepen the exclusion of those people who have already started from a previous situation of economic and social inequality. Consequently, the Spanish public authorities should design specific eHealth plans that include specific measures aimed at containing the growing digital divide, among others, adapting the content and health services hosted on health web portals to the demands, peculiarities, and real needs of these groups based on standardized access and usability criteria.

In this line, and despite the findings of this research, it is necessary to continue advancing in the study on the inclusivity of web portals with respect to mobile applications. Although there is some development in the literature on the use of mHealth by the general population, there is a significant lack of knowledge about access and use by the most vulnerable groups.

Footnotes

Acknowledgments

The authors thank the informants who participated in the study for their time and collaboration.

Contributorship

The authors of the manuscript have been equally involved in all phases of development of this study. All three undersigned authors agree with the publication.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

When this article was written, the University of Santiago de Compostela had a bioethics committee for research of this kind, but not for the Social Sciences. Consequently, though we could not request a preliminary ethics report, we have attempted to follow the recommendations of other studies that had favorable reports from relevant ethics committees (Chuah, Tan, Yeo et al., 2018; Khazaee-Pool, Moeeni, Ponnet et al., 2018). Similarly, having the participants sign an informed consent was not considered necessary.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Ministry of Economy and Competitiveness (Programa Estatal de Investigación, Desarrollo e Innovación Orientada a los Retos de la Sociedad, Convocatoria 2014). This publication is part of the activities of the research project. “Digital divide and inhibitors in the implementation of e-Government. Special impact in the field of health” (CSO2014-53014-R) (2015-2018).

Patient consent

In the absence of elements of ethical consent of personal data, in the case of this research, no data were used that would allow the identification of the individuals who participated in the research. The survey respected the current legislation on data protection: regulation EU/2016/679 of 27 April 2016, General Data Protection (GDPR). No personal data was collected from users without their knowledge, nor was it disclosed to third parties. The opinions and data offered by respondents were only used to conduct the research; part of those results are set out in this article. Once the processing has been completed, the data are only kept for the time necessary to fulfill the purpose for which they were collected and the publication of the data does not include any personal data.

Moreover, the user test requires a sampling model that guarantees the anonymity of the participants. The organizations we contacted are the ones that gave visibility and brought the project closer to the participants and verified on the other hand that they met the selection criteria given by the research team. This guaranteed anonymity by not collecting or publishing any data that would allow the recognition of these people. The organizations contacted people likely to meet the profile, but these people would not necessarily have to go to the centers where the test was carried out. On the other hand, they were only asked their first name to address these people during the interview (with the possibility of refusal) and all the people gave their explicit consent and were informed of the use of these data, which are opinions and evaluations of the services without any ethical conflict or possibility of recognition and publicity of private data since, as we insist, they are evaluations of the portals made in spaces guaranteed by the non-profit organizations that collaborated in the project, as shown in .

Guarantor

N/A

ORCID iDs

Ángela Fernández Da Silva

Bran Barral Buceta

Annex 1.

CC. AA.	Website URL (when analyzed)
Andalusia	http://www.juntadeandalucia.es/servicioandaluzdesalud/principal/default.asp
Aragon	https://www.saludinforma.es/portalsi/web/salud
Principality of Asturias	http://www.asturias.es/portal/site/astursalud
Balearic Islands	http://www.ibsalut.es/ibsalut/es
Canary Islands	http://www3.gobiernodecanarias.org/sanidad/scs/
Cantabria	http://www.scsalud.es/
Castile and León	https://www.saludcastillayleon.es/ciudadanos/es
Castile-La Mancha	http://sescam.castillalamancha.es/
Catalonia	http://web.gencat.cat/es/temes/salut/
Valencian Community	http://www.san.gva.es/inicio
Extremadura	http://saludextremadura.gobex.es/
Galicia	https://www.sergas.es/
La Rioja	http://www.riojasalud.es/
Madrid	http://www.madrid.org/cs/Satellite?pagename=PortalSalud/Page/PTSA_home
Region of Murcia	http://www.murciasalud.es/principal.php
Navarre	http://www.navarra.es/home_es/Temas/Portal + de + la + Salud/Ciudadania/
Basque Country	http://www.osakidetza.euskadi.eus/r85-ghhome00/es/

References

JASEHN. Report on EU state of play on telemedicine services and uptake recommendations, https://ec.europa.eu/health/sites/health/files/ehealth/docs/ev_20171128_co09_en.pdf (2017).

World Health Organization. mHealth: New horizons for health through mobile technologies., https://www.who.int/goe/publications/goe_mhealth_web.pdf (2011).

World Health Organization (WHO Global Observatory for eHealth). Telemedicine—Opportunities and developments in Member States, http://www.who.int/goe/publications/ehealth_series_vol2/en (2011).

Cernadas

Barral

Fernández Da Silva

, et al. Ehealth in Spain: Evolution, current status and future prospects. Saude e Soc 29, Epub ahead of print 2020. DOI: 10.1590/S0104-12902020190886.

Empirica

. eHealth in Action Good Practice in European Countries Good eHealth Report [Internet]. Brussels; 2009. Report No.: 9789279111389.

European Commission. Smarter, Faster, Better eGovernment. 8th eGovermment benchmark measurement, http://uploadi.www.ris.org/editor/1278348425capgemini_8_egov_benchmark_2009.pdf (2009).

Colom

. As brechas digitales que deben preocuparnos y ocuparnos. Ekon Rev vasca Econ 2020; 98: 350–353.

Norris

. Digital divide: Civic engagement, information poverty, and the Internet worldwide. Cambridge: Cambridge University Press, 2001, Epub ahead of print 2001. DOI: https://doi.org/10.1017/CBO9781139164887.

van Dijk

. The Deepening Divide: Inequality in the Information Society. Epub ahead of print 2005. DOI: 10.4135/9781452229812.

10.

Economist Intelligence Unit. Redefining the digital divide. London: EIU, 2013, https://eiuperspectives.economistcom/technology-innovation/redefining-digital-divide .

11.

Andreasson

. Digital divides: The new challenges and opportunities of e-inclusion. Florida: CRC Press, 2015.

12.

Varela

. La brecha digital en España. Estudio sobre la desigualdad postergada. Madrid, http://portal.ugt.org/Brecha_Digital/BRECHADIGITAL_WEB.pdf (2015).

13.

Menou

. La alfabetización informacional dentro de las políticas nacionales sobre tecnologías de la información y comunicación (TICS): la cultura de la información, una dimensión ausente. An Doc 2004; 7: 241–261.

14.

Wilches-Flórez

ÁM

. Posibilidades y limitaciones en el desarrollo humano desde la influencia de las tecnologías de la información y la comunicación en la salud: el caso latinoamericano. Persona y Bioética 2017; 21: 114–133.

15.

OECD. Understanding the digital divide. Paris: Epub ahead of print 2001. DOI: https://doi.org/10.1787/236405667766.

16.

González

Bouzas-Lorenzo

Mahou

. El impacto de la producción normativa en el desarrollo y la capacidad de servicio de la e-Salud en España. Rev Estud Políticos 2013; 162: 199–222.

17.

Borrell

Artazcoz

. Las políticas para disminuir las desigualdades en salud. Gac Sanit 2008; 22: 465–473.

18.

Barral

Bouzas-Lorenzo

Cernadas

, et al. Incorporando la equidad a la salud. Un análisis de los planes autonómicos sanitarios. Rev Esp Salud Publica 2021; 95: 1–15.

19.

Pérez MD

Rodríguez JF

Batista N

. Una década de reflexión sobre los planes de salud en españa. Informe SESPAS 2010. Gac Sanit 2010; 24: 37–41.

20.

Sen

Health equity: perspectives, measurability, and criteria. In: Challenging inequities in health. Oxford: Oxford University Press, 2001, pp. 68–75.

21.

Whitehead

Dahlgren

. Levelling up (part 1) : a discussion paper on concepts and principles for tackling social inequities in health / by Margaret Whitehead and Göran Dahlgren. 2006; 30.

22.

Sen

. Capability and well-being. In: Nussbaum

Sen

(eds) The quality of life. Oxford: Clarendon Press, 1993, pp. 62–66.

23.

Autry

Berge

. Digital natives and digital immigrants: getting to know each other. Ind Commer Train 2011; 43: 460–466.

24.

Almenara J

Ruiz-Palmero

. Las Tecnologías de la Información y Comunicación para la inclusión: reformulando la brecha digital. IJERI Int J Educ Res Innov 2017; 0: 16–30.

25.

Arcaya

Subramanian S

. Inequalities in health: definitions, concepts, and theories. Glob Health Action 2015; 8: 27106.

26.

Gorski

. Education equity and the digital divide. AACE Rev (Formerly AACE Journal) 2005; 13: 3–45.

27.

Wei

Hindman

. Does the digital divide matter more? Comparing the effects of new media and old media use on the education-based knowledge gap. Mass Commun Soc 2011; 14: 216–235.

28.

Marmot

Allen

Bell

, et al. WHO European review of social determinants of health and the health divide. Lancet 2012; 380: 1011–1029.

29.

Singh

Siahpush

. Widening rural-urban disparities in life expectancy, U.S., 1969-2009. Am J Prev Med 2014; 46: e19–e29.

30.

Tolosana

. Crisis, austeridad y áreas rurales: estudio cualitativo de las percepciones sobre los impactos en el sistema sanitario y la salud en Navarra, España. Saúde e Sociedade 2018; 27: 898–908.

31.

Wessels

. The reproduction and reconfiguration of inequality: differentiation and class, status and power in the dynamics of digital divides. In: Ragnedda

Muschert

(eds) The digital divide: the internet and social inequality in international perspective. London: Routledge (Taylor & Francis Group), 2013, pp.17–28.

32.

Toboso

. Rethinking disability in Amartya Sen’s approach: ICT and equality of opportunity. Ethics Inf Technol 2011; 13: 107–118.

33.

Salami

Mason

Salma

, et al. Access to Healthcare for Immigrant Children in Canada. International Journal of Environmental Research and Public Health; 17. Epub ahead of print 2020. DOI: 10.3390/ijerph17093320.

34.

Vázquez

ÁD

Vázquez-Cano

Montoro

MRB

, et al. Análisis bibliométrico del impacto de la investigación educativa en diversidad funcional y competencia digital: web of Science y Scopus. Aula Abierta 2019; 48: 147–155.

35.

Travieso

Planella Ribera

. La alfabetización digital como factor de inclusión social: una mirada crítica. UOC Pap Rev sobre la Soc del Conoc 2008; 7: 1–9.

36.

Lazar

Davenport

. Barriers to health care access for low income families: a review of literature. J Community Health Nurs 2018; 35: 28–37.

37.

Hall

Bernhardt

Dodd

, et al. The digital health divide: evaluating online health information access and use among older adults. Health Educ Behav 2015; 42: 202–209.

38.

Corscadden

Callander

Topp

. Who experiences unmet need for mental health services and what other barriers to accessing health care do they face? Findings from Australia and Canada. Int J Health Plan Manage 2019; 34: 761–772.

39.

Muela

Aliri

Presa

, et al. Randomised controlled trial of a treatment adherence programme for prisoners with mental health problems in Spain. Crim Behav Ment Heal 2020; 30: 6–15.

40.

Shevchuk

Rzhevska

Korop

, et al. Implementation of drug addicts right to health protection (separate aspects). Georgian Med News 2018; 276: 166–172.

41.

van Boekel

Brouwers

EPM

van Weeghel

, et al. Stigma among health professionals towards patients with substance use disorders and its consequences for healthcare delivery: systematic review. Drug Alcohol Depend 2013; 131: 23–35.

42.

Beattie

Smilenova

Krishnaratne

, et al. Mental health problems among female sex workers in low- and middle-income countries: a systematic review and meta-analysis. PLoS Med 2020; 17: e1003297.

43.

Baker

Case

Policicchio

. General health problems of inner-city sex workers: a pilot study. J Med Libr Assoc 2003; 91: 67–71.

44.

Ferrer

. El estado de salud del pueblo gitano en España: una revisión de la bibliografía. Gac Sanit 2003; 17: 2–8.

45.

Martínez-Frías

Bermejo

. Prevalence of congenital anomaly syndromes in a Spanish gypsy population. J Med Genet 1992; 29: 483–486.

46.

Serrano Mascaraque

. Accesibilidad vs usabilidad web: evaluación y correlación. Investig Bibl 2009; 23: 61–103.

47.

Perurena Cancio

Moráguez Bergues

. Usabilidad de los sitios Web, los métodos y las técnicas para la evaluación. Rev Cuba Inf en Ciencias la Salud 2013; 24: 176–194.

48.

Boulos

MNK

Wheeler

Tavares

, et al. How smartphones are changing the face of mobile and participatory healthcare: An overview, with example from eCAALYX. Biomed Eng Online; 10. Epub ahead of print 2011. DOI: 10.1186/1475-925X-10-24.

49.

Carrión

. Usabilidad WEB: Pensando en el bienestar del usuario. Rev Tecnológica-ESPOL 2014; 27: 67–78, http://200.10.150.204/index.php/tecnologica/article/view/302.

50.

Newman

Zimmerman

Crall

, et al. User-friendly web mapping: lessons from a citizen science website. Int J Geogr Inf Sci 2010; 24: 1851–1869.

51.

Nielsen

. Usability Engineering. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1994.

52.

Maguire

. Methods to support human-centred design. Int J Hum Comput Stud 2001; 55: 587–634.

53.

Bouzas-Lorenzo

Mahou Lago

. The study of the functionality of government portals a methodological proposal from the E-health | El estudio de la funcionalidad de los portales gubernamentales: una propuesta metodológica desde la e-Salud. Gest y Polit Publica 2013; 22: 261–308.

54.

Karkin

Janssen

. Evaluating websites from a public value perspective: a review of Turkish local government websites. Int J Inf Manage 2014; 34: 351–363.

55.

García-Lacalle

Pina

Royo

. The unpromising quality and evolution of Spanish public hospital web sites. Online Inf Rev 2011; 35: 86–112.

56.

López-Gil

J-M

Navarro-Molina

García

, et al. Análisis de la arquitectura de webs mediante tests de estrés de navegación, de usabilidad y eye tracking. Prof la Inf 2010; 19: 359–367.

57.

Encabo

. Brecha digital, pobreza y exclusión social. Temas laborales Rev andaluza Trab y bienestar Soc 2017; 138: 285–313.

58.

European Comission. Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions 2030 Digital Compass: the European way for the Digital Decade, https://eur-lex.europa.eu/legal-content/en/TXT/?uri=CELEX:52021DC0118 (2021).

59.

Hassan

Martín

. Qué es la accesibilidad web. No sólo usabilidad 2003; 2: 1–18.

60.

Hassan Montero

. Introducción a la Usabilidad. No sólo usabilidad.

61.

Morales Vargas

Pedraza Jiménez

Codina

. Website quality: An analysis of scientific production.

62.

Codina Bonilla

Pedraza-Jiménez

. Características y componentes de un sistema de análisis de medios digitales: el SAAMD. In: Calidad en sitios web: método de análisis general, e-comerce, imágenes, hemerotecas y turismo. Editorial UOC, 2016, pp. 15–39.

63.

Marcos

Rovira

. Evaluación de la usabilidad en sistemas de información web municipales: metodología de análisis y desarrollo. In: La dimensió humana de l’organització del coneixement. Facultat de Biblioteconomia i Documentació, pp. 415–432.

64.

Fernandez

Abrahão

Insfran

. A Web Usability Evaluation Process for Model-Driven Web Development BT - Advanced Information Systems Engineering. In: Mouratidis H, Rolland C (eds). Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 108–122.

65.

Welle Donker-Kuijer

de Jong

Lentz

. Usable guidelines for usable websites? An analysis of five e-government heuristics. Gov Inf Q 2010; 27: 254–263.

66.

Bangor

Kortum

Miller

. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009; 4: 114–123.

67.

Dumas

Redish

. A practical guide to usability testing. Exeter, England; Portland, OR: Intellect Books, 1999.

68.

Goodman Kuniavsky

Andrea

. Observing the user experience a practitioner’s guide to user research. Amsterdam: Elsevier, 2019.

69.

Gant

. Web portal functionality and state government e-service. In: Proceedings of the 35th Annual Hawaii International Conference on System Sciences. 2002, pp. 1627–1636.

70.

Nielsen

. Designing web usability. Berkeley, Calif.: New Riders, 2006.

71.

Mahou-Lago

Varela-Álvarez

. Innovation and opportunities for citizen participation in Spanish smart cities. Public Admin Inf Technol 2016; 11: 367–392.

72.

Luchenski

Maguire

Aldridge

, et al. What works in inclusion health: overview of effective interventions for marginalised and excluded populations. Lancet 2018; 391: 266–280.

73.

Lupiáñez-Villanueva

. Salud y sociedad red: Análisis de los usos de Internet relacionados con la salud. Barcelona: Grupo Planeta (GBS), 2010.

74.

Fitzpatrick

Bramley

Johnsen

. Pathways into multiple exclusion homelessness in seven UK cities. Urban Stud 2013; 50: 148–168.

75.

Fundación

COTEC

. Competencias digitales y colectivos en riesgo de exclusión en España. Madrid, https://cotec.es/proyecto/competencias-digitales/51a02688-a11f-4fee-b047-41288ea0e0ac (2021).

76.

ONTSI. Tendencias en el uso de dispositivos tecnológicos 2021, https://www.ontsi.es/es/publicaciones/Tendencias-en-el-uso-de-dispositivos-tecnologicos-2021 (2021).

77.

ONTSI. Perfil sociodemográfico de los internautas. Análisis de datos INE 2020, https://www.ontsi.es/es/publicaciones/Perfil-sociodemografico-de-los-internautas-14 (2020).

78.

INE. Encuesta sobre Equipamiento y Uso de Tecnologías de Información y Comunicación en los Hogares. Año 2020, https://www.ine.es/prensa/tich_2020.pdf (2020).

79.

Bas-Sarmiento

Fernández-Gutiérrez

Albar-Marín

, et al. Percepción y experiencias en el acceso y el uso de los servicios sanitarios en población inmigrante. Gac Sanit 2015; 29: 244–251.

80.

Llop-Gironés

Vargas Lorenzo

Garcia-Subirats

, et al. Acceso a los servicios de salud de la población inmigrante en España. Revista Española de Salud Pública 2014; 88: 715–734.

81.

Mayorga

. Artlculaciones De La Exclusión: la Política De Atención A Prostitutas Inmigrantes En Madrid. Si Somos Americanos 2012; 12: 49–74.

82.

Pan American Health Organization, Secretaría General de Asuntos Sociales, World Health Organization. Clasificación Internacional del Funcionamiento, de la Discapacidad y de la Salud. Ginebra: World Health Organization, 2001, https://www.imserso.es/InterPresent2/groups/imserso/documents/binario/435cif.pdf .

83.

Barral

Bouzas-Lorenzo

Cernadas

, et al. Equity policies in health plans: accessibility and something more? Rev Saude Publica; 55. Epub ahead of print 28 May 2021. DOI: 10.11606/S1518-8787.2021055002560.

84.

Perona

Yera

Arbelaitz

, et al. Towards Automatic Problem Detection in Web Navigation Based on Client-Side Interaction Data. In: Proceedings of the XX International Conference on Human Computer Interaction. New York, NY, USA: Association for Computing Machinery. Epub ahead of print 2019. DOI: 10.1145/3335595.3335642.

85.

del Campo

SdA

Matilla

Bermejo

JMM

. Educomunicación, tics y prisión: testimonios de personas privadas de libertad en torno a la exclusión digital. Fonseca J Commun 2021; 23: 275–290.

86.

Morgan

Burns

Fitzpatrick

, et al. Social exclusion and mental health: conceptual and methodological review. Br J Psychiatry 2007; 191: 477–483.

87.

Rodríguez-Pellejero

Núñez

Hernández

. Personality profiles and clinical syndromes in homeless. Rev Psicopatol y Psicol Clin 2017; 22: 197–206.

88.

Calvo García

. Exclusión residencial y tecnologías de la información y la comunicación: Oportunidades de las redes sociales en línea para mejorar la salud mental de personas en situación de sinhogarismo. Barcelona: Universitat Ramon Llull, 2019, http://www.tdx.cat/handle/10803/667595 .

89.

Fundación

FOESSA

. VII Informe sobre exclusión y desarrollo social en España. Madrid, 2014. https://caritas365-my.sharepoint.com/personal/cominc_caritas_es/_layouts/15/onedrive.aspx?id=%2Fpersonal%2Fcominc_caritas_es%2FDocuments%2FSensibilización%2FFOESSA%2FWEB%2FDocumentos enlazados%2FVII_INFORME FOESSA.pdf&parent=%2Fpersonal%2Fcominc_caritas_e.

90.

de la Rica

Gorjón

Miller

, et al. Estudio comparado sobre la situación de la población gitana en España en relación al empleo y la pobreza 2018. Madrid: Fundación Secretariado Giano, 2019, https://iseak.eu/wp-content/uploads/2019/09/informe_FSG_ISEAK.pdf .

91.

Fundación

FOESSA

. VIII Informe sobre exclusión y desarrollo social en España. Madrid, https://www.foessa.es/main-files/uploads/sites/16/2019/06/Informe-FOESSA-2019_web-completo.pdf (2019).

92.

Ministerio de Sanidad. e-Salud. Informe Anual del Sistema Nacional de Salud 2018. Madrid, 2018, https://www.sanidad.gob.es/estadEstudios/estadisticas/sisInfSanSNS/tablasEstadisticas/InfAnualSNS2018/Cap.8_e_Salud.pdf .

93.

ONTSI. Los ciudadanos ante la e-sanidad. Opiniones y expectativas de los ciudadanos sobre el uso y aplicación de las TIC en el ámbito sanitario, https://www.ontsi.red.es/ontsi/sites/ontsi/files/los_ciudadanos_ante_la_e-sanidad.pdf (2016, accessed 28 November 2018).