U.S. state health agencies and organizational learning: An exploratory analysis of website accessibility during COVID-19

Abstract

The COVID-19 pandemic has exposed the inaccessibility of government information and communication technologies (ICTs) for members of the disability community. Organizational learning around ICT accessibility can be impacted by factors influencing strategies and assumptions or values and norms. Using data collected over two time periods in 2021, we study how the accessibility of US state health agencies COVID-19 information and vaccine websites improve over time. We examine how time, state policies, and partisanship influence organizational learning around website accessibility. Our analysis determines that the longer a COVID-19 related website exists on the Internet, the less accessible the website. We also find that more extensive internal state accessibility policies are more correlated with websites that meet fundamental accessibility requirements. Additionally, we find that partisanship plays an unexpected role in meeting fundamental accessibility demands, although both state policies and politics do not influence if an ICT meets the best practices standards of accessibility. Our paper initiates a discussion around the factors that influence organizational learning about government website accessibility and points to future research questions where the primary ICT function is not influenced by a rapidly evolving pandemic.

Keywords

Website accessibility organizational learning state health agencies COVID-19Key points for practitioners:•Accessible websites are a moral obligation not merely a compliance burden;•Organizations go through a learning process to improve the accessibility of their websites;•Factors like timing internal state policies and partisanship impact the learning process by influencing organization’s strategies and assumptions (single-loop learning process) and values and norms (double-loop learning process).

1. Introduction

E-government is the use of information and communication technologies (ICTs) by government agencies. ICTs allow citizens to access information and resources and create opportunities to participate in the political process (MacLean & Titah, 2022). ICTs are a useful tool to improve transparency, increase communication, and meet the needs of diverse populations (Yu, 2021). However, the impacts of e-government differ greatly for individuals with disabilities (Cook & Harrison, 2015).

One in four Americans are disabled and that number is predicted to rise due to COVID-19 (Hereth et al., 2022). Previous definitions of disability were formed in the medical community and focused on the limitations a person’s disability placed on their social life and ability to complete basic life tasks (Lollar & Crews, 2003). This definition permeated into aspects of governance, leading to systemic discrimination in the public health sector (Rimmer, 2011; Sabatello et al., 2020). Currently, definitions of disability focus on how an external environment does not provide a functional experience for all people (Lollar & Crews, 2003). This definition encompasses a wide range of disabilities, including hearing, mobility, vision, cognitive, self-care, and independent living disabilities (Centers for Disease Control & Prevention, 2019).

The disability community faces inclusion issues in public life (Rimmer, 2011), encountering accessibility issues in most physical and electronic infrastructures (Rimmer, 2011; Sabatello et al., 2020). As an underserved community, individuals with disabilities face challenges to access basic public services, including Internet access (Sabatello et al., 2020). The digital divide, which identifies people with access to the Internet, is greater for individuals with disabilities, especially for those who also identify as a person of color or as low-income (Dobransky and Hargittai, 2016). In November 2021, 70.6% of individuals with disabilities used the internet at home, compared to 83.8% of individuals without a disability (National Telecommunications and Information Administration, 2022).

Even after overcoming the digital divide, the disability community faces ICT accessibility barriers. Legally, federal and state government services are required to provide reasonable accommodations, both online and in-person, but most accommodations focus on legal compliance (Sabatello et al., 2020). Exclusion from public and online spaces leads to feelings of isolation and decreased physical and psychological health (McDonald et al., 2015). The traditional focus on legal compliance rather than functional accessibility is especially concerning considering the communication needed between a government and its population during a crisis (National Council on Disability, 2014).

With a wide range of individual circumstances at play, defining ICT inaccessibility is difficult, but is often based on the Web Content Accessibility Guidelines (WCAG). State and federal websites tend to be more accessible when compared to randomly selected private websites, but many government websites fail to meet basic accessibility standards, including 42% of popular federal websites (McQuinn & Castro, 2017). Accessibility barriers include: interacting with screen readers, but not using strong, contrasting colors, using very small text sizes, having broken links etc. (Yu & Parmanto, 2011). The lack of awareness around ICT accessibility is especially critical during disasters, like the COVID-19 pandemic. Ableism was exacerbated during the COVID-19 pandemic when individuals with disabilities reported a major barrier to state-sponsored vaccination efforts was the limited ability to find online information about vaccines or schedule an appointment online (Ryerson et al., 2021).

As government services increasingly move towards online functions, research should focus on how ICTs work to enhance social equity (Blessett et al., 2019), a pillar of public administration research. Understanding how government-provided services are created with inclusion in mind should reduce participation barriers for the disability community (Turnbull & Stowe, 2001). Government ICTs must provide equal access and engagement in specific information policies to overcome the systematic accessibility gap between the disability and non-disability community (Dobransky & Hargaittai, 2016).

In delivering public services, state agencies engage in an organizational learning process to improve outputs and provide better services. While governments increasingly use the Internet as a tool for information sharing and feedback (Gilson et al., 2009), ICT accessibility differs widely across states. We ask, “What factors influence a U.S. state health agency’s organizational learning process leading to increasing accessibility of COVID-19 related websites?” We focus on four factors that could influence organizational learning: length of website existence, website complexity for users, internal state accessibility policies, and partisanship. We use assessments from two points in 2021 to capture the influence of these factors on the organizational learning process leading to changes in website accessibility. Our research shows learning mostly occurs when factors influence an organization’s values, instead of strategy. We first review ICT accessibility and organizational learning. We present hypotheses based on the single and double loop learning model. We then report our data, methods, and discuss our findings.

2. Online communication, inclusion, and the disability community

The digital divide extends beyond Internet availability to website content accessibility (Baker, Hanson, & William, 2009). Accessibility on the Internet focuses on the capacity to equally meet an individual’s need to obtain and meaningfully interact with web-based content (Jaeger & Xie, 2009). Accessibility in e-government requires governments to ensure equal access to information provided through ICTs, in addition to compatibility with any Internet-connected devices (Sonnenberg, 2020). Current web accessibility standards are formed from a mixture of government regulations and private sector guidelines (Jaeger & Xie, 2009). From a state government standpoint, accessibility focuses on compliance with dated federal statutes like the American with Disabilities Act (ADA) and Sections 504 and 508 of the Rehabilitation Act (Yang & Chen, 2015). In lieu of specific, up-to-date federal compliance mandates, the U.S. Justice Department suggests accessibility guidelines based on WCAG 2.0’s minimal (level A) and acceptable (level AA) standards (Barrett & Greene, 2021; U.S. Access Board, n.d.). Government employees tasked with creating accessible websites often turn to outside resources, like the Web Accessibility Initiative (Jaeger & Xie, 2009).

ICT accessibility varies based on social and political factors. Successful ICT delivery depends on effective organizational learning through meaningful and systematic evaluation (quickly detecting and correcting errors) and adjusting outputs based on lessons learned (Dunlop, 2017). Research on e-government is less clear about the relationship between organizational learning and success in generating accessible ICTs, particularly what factors influence this type of organizational learning. Most research focuses on compliance, viewing the work of accessibility not as a social equity imperative and ethical obligation (Palmer & Palmer, 2018), but as an organizational burden. Viewing ICT accessibility as a compliance burden ignores organizational knowledge deficits and what factors influence an organization’s learning process when creating and maintaining accessible websites (O’Neill, 2021), a gap this paper tries to fill.

2.1 Organizational learning

Organizational learning is the process of gathering and employing knowledge to improve an organization’s performance. Organizational learning is relational in nature and occurs as a social and political process (Vince & Saleem, 2004). The learning process is critical to successful organizational performance, especially when learning is passed from the individual to the team and organization (Jerez-Gómez et al., 2005). Governments engage in organizational learning to modify the effectiveness, efficiency, and equity of public goods/services delivery (Choi & Chandler, 2020). When errors occur in public service, governments are expected to learn and respond appropriately (Dekker Hansén, 2004). Government agencies engaging in organizational learning face unique contexts (political, social, and otherwise), making understanding the factors influencing learning an important point of emphasis in public administration research (Rashman, Withers, & Hartley, 2009).

Traditionally, organizational learning is evaluated either using experience curves, which focus on growing stated outcomes or outputs (Jerez-Gómez et al., 2005), or the organization’s capacity to learn (Gilson et al., 2009). We focus on the changes in an organization’s outputs (Jerez-Gómez et al., 2005). We assume that organizational learning takes place over time within the government agencies that provide COVID-19 information websites (Gkeredakis et al., 2021; Gilson et al., 2009). Evidence of learning lies in the changes within the organization’s output – in this case, increased accessibility for COVID-19 related websites. We are interested in what factors influence these agencies’ learning process and examine whether certain factors affect an agency’s strategies and assumptions (single-loop organizational learning) or values and norms (double-loop learning) to produce an updated, more accessible COVID-19 related website.

Figure 1.

Single-loop and double-loop learning.

Argyris & Schön (1996) describe organizational learning using a single and double-loop analogy (see Fig. 1). In single-loop learning, an organization generates a set of strategies and assumptions around producing an output. These strategies and assumptions lead to consequences that alter the output (Basten & Haamann, 2018). When a mismatch occurs between the expected and executed output, organizations adapt their strategies and assumptions to improve. Argyris & Schön (1996) compare this type of learning to quality control inspectors working to correct product defects.

In double-loop learning, an organization’s values and norms dictate the strategies and assumptions about producing an output. When a mismatch between the intended and realized output is identified, organizations respond to ensure that the output falls within the organization’s values (Patky, 2020). The focus is not on the quality of the outcome (although errors are corrected), but on the connection between the outcome and the organization’s values (Argyris & Schön, 1996). Government bodies also modify actions either through systematic problem solving or with subsequent adjustments to agency norms and values (Gilson et al., 2009; Basten & Haamann, 2018).

Within public service organizations, different factors can influence the use of single-loop or double-loop learning. For example, single-loop learning may be implemented when complaints about the public service arise (Tagg, 2010). Limitations in the customer experience (Younis, 1997) and changing expectations about the output (Diefenbach, 2009) may also influence single-loop learning. On the other hand, pressure from political authorities may promote double-loop learning (Barbieri & Salvatore, 2010; Balasubramanian, Al-Ahbabi, & Sreejith, 2020). Changes in organizational resources, capability (Pee and Kankanhalli, 2016), and government policy traditions (Voorberg et al., 2017) may also influence double-loop learning.

We examine various factors impacting U.S. state health agencies’ learning around the accessibility of COVID-19 related websites. We focus solely on website accessibility, which is an important predictor of usability (Bai, 2019). As agencies had to adapt quickly due to the accelerated nature of information gathering during the COVID pandemic, we evaluate websites at two different time periods: March 2021 and December 2021. We analyze websites by examining the accessibility of COVID-19 information and COVID-19 vaccine location websites. We define accessibility as equivalent access to and the ability for users with disabilities to engage with a website’s content. We examine accessibility using two tiers: (1) fundamental accessibility, which refers to the basic requirements for users to engage with website content (i.e., errors) and (2) best practices accessibility, which focuses on maximizing website structure to ensure equivalent engagement is possible (i.e., alerts).

We argue that the single and double loop learning perspective explains why some states are more successful in providing accessible websites (see Fig. 2). Different factors influence the single and double loop learning process. We examine four factors: website complexity from the user side, timing of website launch, state policies, and partisanship. From a single loop learning perspective, state health agencies identify a mismatch between the expected and realized website. Potential influencing factors, like the timing of the website launch or the complexity of the website from the user perspective, may lead agencies to adjust their strategies and assumptions in creating and maintaining accessible websites. As the state health agency engages in the organizational learning process, the strategies and assumptions about website creation change and the website becomes more accessible. No agency values or norms are altered. Rather, assumptions around website functionality and strategies to generate accessible websites are influenced.

Figure 2.

Factors influencing single-loop and double-loop learning for state health agencies’ COVID-19 related websites.

From a double loop learning perspective, state health agencies respond to the mismatch between an expected and realized output. Factors, like internal state policies and partisanship, influence the agency’s examination of values and norms. The agency adjusts their values and norms, which subsequently alters their strategies and assumptions around accessible websites, leading to an improved, more accessible website.

2.2 Single loop: Website complexity from the user side

Most websites are not created with accessibility as the default (Wentz, Jaeger, & Lazar, 2011); website accessibility standards face escalating hurdles as new Internet-capable devices are created (Sonnenberg, 2020). Information-based, or static, websites (i.e., descriptions of public health crises) are comparably more inclusive for individuals with disabilities than interactive, or dynamic, websites (i.e., maps showing COVID-19 vaccination sites) (Jaeger & Xie, 2009; Sonnenberg, 2020). State health agency COVID-19 information websites list details about the virus, preventative measures, and other pandemic-related information. State health agency COVID-19 vaccine information websites are more complex. Users interacting with COVID-19 vaccine information websites used maps to locate a vaccine clinic, filled out forms to identify inoculation eligibility, and navigated through PDFs, pop-up windows, or online videos. Dynamic websites require more individual specifications options for different users to experience the same level of access (Jaeger & Xie, 2009).

Website complexity on the user’s end can cause increased accessibility challenges, especially in accelerated time frames (Gkeredakis et al., 2021). As state health agencies learn about the mismatch between the intended ICT output and the actual, inaccessible website, the complexity of the user-end experience might pressure organizations to revisit strategies and assumptions around website creation and maintenance leading to an altered, more accessible ICT. For COVID-19 information websites, accessibility challenges are easy to identify and address meaning the state health agencies might be more likely in the single-loop learning process. For the more dynamic, COVID-19 vaccine locator websites, the complexity of the website presents additional accessibility challenges requiring more in-depth learning to subsequently alter strategies and assumptions. This burden of increased learning suggests state health agencies will be less likely to engage in the process and produce more accessible dynamic websites, especially with the high demands on agency resources during the COVID-19 pandemic. We hypothesize:

Hypothesis 1a: The COVID-19 information websites will be negatively related to errors over time compared to COVID-19 vaccine locator websites. Hypothesis 1b: The COVID-19 information websites will be negatively related to alerts over time compared to COVID-19 vaccine locator websites.

2.3 Single loop: Timing of website launch

Pressure-filled and changing time frames can frustrate organizational learning processes and lead to performance failures. When timing issues lead to website accessibility failures, barriers of access for groups already experiencing unequal accessibility levels increase (Gkeredakis et al., 2021). If rapid iterations of a product are required, timing demands can interfere with organizational strategies and assumptions limiting the spread of needed information (Bowser et al., 2021).

Government agencies responded to the COVID-19 pandemic by quickly creating ICTs (Meijer et al., 2020). Due to the evolving nature of the COVID-19 pandemic, the speed of state health agency website development increased. We believe the timing of a website’s deployment will be negatively associated with the accessibility of a COVID-19 related website (for both the main website and vaccine locator page). We believe the earlier a COVID-19 related website is deployed, the less accessible the website is over time, due to the disruption in a state health agency’s learning process. We hypothesize:

Hypothesis 2: Early launching COVID-19 related websites will be positively related to website errors and alerts.

2.4 Double loop: State accessibility policies

The values and norms of government agencies are shaped by environmental, political, and social factors. A state health agency’s organizational learning process may depend on power-based institutional mechanisms, such as control strategies and formal policies, that can regulate values and norms (Gilson et al., 2009; Lawrence et al., 2005). Technology in government settings is mostly initiated from institutional policies and often requires top-level support to be feasible and effective (Gilson et al., 2009). Leeuw et al. (1994) argue that governments have increased learning when information comes from internal, rather than external, sources. Internal policies about website accessibility may be viewed as more credible and provide knowledge and support resources. Internal policies may carry more weight when influencing organizational values, due to the hierarchical nature of government.

We believe that state governments with more extensive internal website accessibility policies are more likely to have accessible COVID-19 related websites. The most extensive policies require state websites to meet all federally-required standards and support WCAG’s list of best-standard practices. When the mismatch between the preference for accessibility and the reality of inaccessible websites is identified, state health agencies engage in the learning process. After confronting and adjusting any norms and values in opposition to internal accessibility policies, changes in the strategies and assumptions around website accessibility practices should occur, ultimately creating a more accessible website. We hypothesize:

Hypothesis 3: The extensiveness of a state’s accessibility policy will be negatively related to COVID-19 related website errors and alerts

2.5 Double loop: Partisanship

Research in accessibility shows the importance of measuring partisanship when considering website accessibility (Rubaii-Barrett & Wise, 2008). Partisanship is also considered a strong predictor of what, and how, public health policies are adopted (Adolph et al., 2021). The executive branch’s political party (Holmgren, 2018) and a location’s general partisan atmosphere (Dekker & Hansén, 2004) helps establish norms around accessibility and ICT use in general. Elected officials, like governors, seek to control the actions of public agency personnel (Doherty, Lewis, & Limbocker, 2019). Partisan politics influence a state health agency’s commitment to establishing and maintaining access for certain disadvantaged populations (Rubaii-Barrett & Wise, 2008). In the e-government literature, partisanship has been found, in general, to impede e-government and influence the likelihood of adopting new digital innovations (Norris & Moon, 2005).

Conservative states with a high number of Republican legislators are more likely to disapprove of e-government initiatives as a way to reduce the size of government and cut costs (West, 2000). During the pandemic, governors responded to partisan opinions when making public health decisions (Choi, Allgood, & Swindell, 2022). We believe that more conservative states are less likely to have accessible COVID-19 related websites. The partisan leaning of a state should indicate if state health agencies adjusted norms and values around accessible websites during the learning process, due to the politically charged atmosphere around the COVID-19 pandemic response. We hypothesize:

Hypothesis 4: States that voted conservatively in the 2020 elections will have COVID-19 related websites that are positively related to errors and alerts.

3. Data and methods

Over a period of two weeks spaced nine months apart (Time 1: 3/15/2021–3/21/2021 and Time 2: 12/13/2021–12/20/2021), we reviewed all 50 U.S. state health agency websites. The time periods were chosen to capture the effects of COVID-19 on organizational learning after the start, and normalizing of, pandemic-based protocols. We assessed website errors and alerts using the WAVE Google Chrome extension provided by Utah State University’s WebAIM project (WebAIM, n.d.). The WAVE tool has been used to evaluate governments (Youngblood & Mackiewicz, 2011) and institutions of higher education (Massengale & Vasquez, 2016). For example, Yu (2021) compares the violations of fundamental WCAG guidelines (i.e., errors) for local Australian governmental and non-governmental COVID-19 information websites over time, identifying that both non-governmental and governmental websites sharing COVID-19 information were presented in an inaccessible manner. The focus of our assessment is on state health agency’s websites for (a) COVID-19 related information and (b) COVID-19 vaccine location information. The approach for locating vaccines varied from state to state, so only the landing page for the vaccine location information is evaluated. Each researcher tackled 25 state websites, focusing on a different set for the second review, and uploaded the results for review by her colleague.

The assessment focused on the accessibility of publicly available information retrieved from each state’s COVID-19 related websites. COVID-19 information websites covered a myriad of topics including how to obtain a COVID-19 test, how to quarantine, and press releases. COVID-19 vaccine locator websites presented variations on the following: a location search for vaccine sites, an inoculation eligibility form, and/or a text or map-based list of vaccine sites. Two states provided only one website with general COVID-19 information and vaccine location on the same webpage.1

¹
We include clustered standard errors in the analysis to account for any duplication of information.

We measure accessibility using errors and alerts as reported by the WAVE tool. Users download a browser extension to assess website accessibility using the most up to date WCAG guidelines (WebAIM, n.d.). Evaluations provide feedback on multiple elements including two specific accessibility categories: errors and alerts (WebAIM, n.d.). We consider errors to be a measure of fundamental accessibility and alerts to be a stand-in for best practices accessibility. Errors refer to web elements that are missing or broken. Alerts refer to websites with web elements that are insufficient in addressing accessibility needs or have the potential to create additional accessibility barriers as the element is incomplete or incorrectly used. Websites with no errors and a low number of alerts indicate a more accessible website. A full list of the errors and alerts can be found in the Appendix.

We measure timing by identifying the difference between the first available website cache and a pandemic-related date of interest. The earliest cached dates for each website are identified using the Wayback Machine, an Internet archive. The Wayback Machine website can be accessed using this link: https://archive.org/web/. For COVID-19 information websites, we compare the timing of the U.S. state health agency website launch as first cached on the Internet archive to the day the WHO declared a global pandemic (March 11, 2020) (World Health Organization, 2022). For COVID-19 vaccine locator websites, we compare the first cached date of website launch to the day the U.S. FDA approved a general COVID-19 vaccine for individuals over age 16 (December 11, 2020) (U.S. Food and Drug Administration, 2020). We calculate how many weeks before or after the website was published, compared to the pandemic date of interest, to determine how early each type of COVID-19 related website was first archived on the Internet.

We measure the extensiveness of each state’s website accessibility policies by rating the policy on a three-point scale. States are given a zero (0) if the website accessibility policy does not go beyond federally mandated practices (from the ADA and Sections 504 and 508) and focuses only on minimal WCAG compliance standards. States are given a one (1) if the website accessibility policy does not go beyond federally mandated practices and focuses on acceptable WCAG compliance. States are given a two (2) if the website accessibility policy goes beyond federally mandated practices and starts to consider optimal WCAG compliance.

We use a dummy variable (0 for Republican and 1 for Democrat) to measure state partisanship. To capture partisanship, we rely on the political party whose presidential candidate won a state’s 2020 electoral college votes. While the 2020 election results may not reflect a state’s normative partisanship, these results reflect the overall partisan leanings of each state during the COVID-19 crisis. We create the partisanship dummy variable using data collected from the United States National Archives and Records Administration (National Archives and Records Administration, 2021). For states that split electoral votes, we coded the variable based on the presidential candidate who won the majority of that state’s electoral votes.

Table 1

Descriptive statistics

	$N$	Mean		SE		Min		Max
COVID-19 information websites
Total Errors (Time 1)	50	23.	56	37.	05	1		199
Total Errors (Time 2)	50	11.	70	17.	78	0		99
Total Alerts (Time 1)	50	50.	74	95.	18	2		648
Total Alerts (Time 2)	50	40.	68	55.	90	2		282
Timing ${}^{1}$ (Time 1)	50	$-$ 2.	65	10.	31	$-$ 40.	14	6.	14
Timing ${}^{1}$ (Time 2)	50	$-$ 43.	14	12.	47	$-$ 92.	29	$-$ 1.	14
COVID-19 vaccine locator websites
Total Errors (Time 1)	50	21.	40	31.	46	0		144
Total Errors (Time 2)	50	11.	94	20.	59	0		110
Total Alerts (Time 1)	50	55.	98	105.	77	1		648
Total Alerts (Time 2)	50	40.	60	58.	93	1		272
Timing ${}^{1}$ (Time 1)	50	36.	64	10.	31	$-$ 0.	86	45.	43
Timing ${}^{1}$ (Time 2)	50	$-$ 3.	85	12.	47	$-$ 53.	00	38.	14
State Characteristics
Partisanship ${}^{2}$	50	0.	58	0.	64	0.	00	1.	00
Accessibility policy only meets federal mandates (0)	50	0.	64	0.	48	0.	00	1.	00
Accessibility policy meets federal mandates and focuses on WCAG acceptable compliance (1)	50	0.	3	0.	47	0.	00	1.	00
Accessibility policy meets federal mandates and focuses on WCAG optimal compliance (2)	50	0.	06	0.	23	0.	00	1.	00

Notes: ${}^{1}$ Refers to the difference between the date of the first cached version of the webpage and referenced pandemic event (3/11/2020) or (12/11/2020). ${}^{2}$ Dummy variable reflecting 2020 election results (0 $=$ Republican and 1 $=$ Democrat).

The descriptive statistics for all variables of interest are displayed in Table 1. We focus on the organization as the unit of analysis. 58% of states voted for the Democratic candidate in 2020. 40% of states require government website accessibility to only meet the federal standards of accessibility. Most COVID-19 information websites went live about two and half weeks before March 11, 2020. Most COVID-19 vaccine locator websites went live a little over a month before December 11, 2020.

We capture state health agencies’ organizational learning as evidenced by changes in COVID-19 related website accessibility. There is evidence some organizational learning occurs as the average and total number of errors and alerts decrease over time. On average, a state’s COVID-19 information homepage was observed to have about 24 total errors in Time 1 and about 12 total errors in Time 2; a state’s COVID-19 vaccine locator webpage was observed to have about 21 total errors in Time 1 and about 12 total errors in Time 2, indicating, at face value increased accessibility (Appendix Table 1 reports the full breakdown). We assess the differences in total errors and alerts for Time 1 (March 2021) and Time 2 (December 2021) using two sample T-tests. As Table 2 shows, we find a statistically significant difference in overall total errors between Time 1 and Time 2 ( $p<$ 0.10). We find significant differences in website errors between the two time periods for both website types ( $p<$ 0.10).

Table 2

Two sample $t$ -test results

	Time 1 (March 2021)		Time 2 (December 2021)
	$N$	Mean	$N$	Mean	$t$ -test
Both websites
Total Errors	50	44.96 (59.00)	50	23.64 (35.32)	0.03**
Total Alerts	50	106.72 (191.42)	50	81.28 (108.27)	NS
COVID-19 information website
Total Errors	50	23.56 (37.05)	50	11.70 (17.78)	0.04**
Total Alerts	50	50.74 (95.18)	50	40.68 (55.90)	NS
COVID-19 vaccine locator website
Total Errors	50	21.40 (31.46)	50	11.94 (20.59)	0.08*
Total Alerts	50	55.98 (105.77)	50	40.60 (58.93)	NS

Note: Standard errors reported in parentheses. ${}^{***}p<$ 0.01, ${}^{**}p<$ 0.05, ${}^{*}p<$ 0.10.

We further our exploration using a regression analysis. The dependent variables (i.e., website errors and alerts) are count measures indicating a Poisson or negative binomial maximum likelihood model is most appropriate due to the skewed distributions (see the Appendix) (Cameron & Trivedi, 1998). Poisson regressions assume that the dependent variables have no over-dispersion. Negative binomial regressions adjust for any inflated variance and account for over dispersed dependent variables (Hilbe, 2007). To determine the most appropriate method, we test whether the dependent variable is over-dispersed using the Poisson goodness-of-fit test in STATA (version 15.1). We reject the null hypothesis that the mean equals the variance ( $p=$ 0.001 for total errors and $p=$ 0.02 for total alerts), making the negative binomial regression the preferred modeling approach2

Zero inflated negative binomial estimation is not needed due to the low number of zeros.

(Greene, 2003). The first-difference estimation with our complete panel dataset and clustered analysis on the state level allow us to control for time-invariant (observed and unobserved) variables.

4. Results

We present the results for each hypothesis with results displayed in Table 3.

Table 3
Negative binomial regression results

	Dependent Variable: (count of) Errors			Dependent Variable: (count of) Alerts
	Model 1: Websites Combined
	Model 2: COVID-19 Information Website
	Model 3: COVID-19 Vaccine Locator
Website	Model 4: Websites Combined
	Model 5: COVID-19 Information Website
	Model 6: COVID-19 Vaccine Locator Website
Timing ${}^{1}$	0.02 ${}^{**}$ (0.01)	0.02 ${}^{***}$ (0.01)	0.01 ${}^{*}$ (0.01)	0.01 ${}^{**}$ (0.01)	0.01 ${}^{***}$ (0.01)	0.01 ${}^{**}$ (0.01)
Partisanship ${}^{2}$	$-$ 0.24 (0.23)	$-$ 0.53 ${}^{**}$ (0.25)	$-$ 0.17 (0.27)	$-$ 0.04 (0.20)	$-$ 0.35 ${}^{*}$ (0.21)	0.07 (0.25)
Accessibility Policy ${}^{3}$	0.21 (0.22)	$-$ 0.41 ${}^{**}$ (0.20)	0.43 ${}^{*}$ (0.26)	0.17 (0.25)	$-$ 0.22 (0.22)	0.25 (0.28)
Constant	4.40 ${}^{***}$ (0.23)	3.60 ${}^{***}$ (0.29)	3.88 ${}^{***}$ (0.29)	3.89 ${}^{***}$ (0.23)	2.79 ${}^{***}$ (0.31)	3.50 ${}^{***}$ (0.28)
N	100	100	100	100	100	100
Pseudo R ${}^{2}$	0.01	0.03	0.01	0.01	0.02	0.01
Log likelihood (/lnalpha)	$-$ 0.25 (0.17)	0.09 (0.15)	$-$ 0.07 (0.20)	$-$ 0.09 (0.14)	0.09 (0.18)	0.15 (0.15)

Notes: Clustered standard errors reported in parentheses. ${}^{***}p<$ 0.01, ${}^{**}p<$ 0.05, ${}^{*}p<$ 0.10. ${}^{1}$ Refers to the difference between the date of the first cached version of the webpage and referenced pandemic event (3/11/2020) or (12/11/2020). ${}^{2}$ Dummy variable reflecting 2020 Election Results (0 $=$ Republican and 1 $=$ Democrat). ${}^{3}$ Extensiveness of accessibility policies (0 $=$ Only mirrors federal policy; 1 $=$ Mirrors federal policy and focuses on acceptable compliance; 2 $=$ Mirrors federal policy and focuses on optimal compliance).

Hypotheses 1a and 1b state that COVID-19 information websites will have a negative relationship to errors (1a) and alerts (1b) over time when compared to COVID-19 vaccine locator websites. Using a seemingly unrelated regression analysis, we compare total errors for the COVID-19 information and vaccine locator pages (Models 2 and 3). We fail to reject that the coefficient equations are proportional ( $\chi^{2}=$ 1.08, $p=$ 0.58). We compare total alerts for the COVID-19 information and the COVID-19 vaccine locator websites (Models 5 and 6) and fail to reject ( $\chi^{2}=$ 2.30, $p=$ 0.32). Thus, Hypothesis 1a and 1b are not supported.

Hypothesis 2 states that early launching COVID-19 websites will be positively related to increased website alerts and errors. Early launching of both COVID-19 information or vaccination websites are significantly related to website errors and alerts across all models. Models 1 & 4 (combined websites) show that early launching increases errors, by a factor3

These are the Incidence Rate Ratios (IRR) based on the coefficients found in Table 3.

of 1.01, (

B=

0.02,

p=

0.02) and alerts, by a factor of 1.02 (

B=

0.01,

p=

0.01). We find strong support for Hypothesis 2.

Hypothesis 3 states that the extensiveness of a state’s accessibility policy will be related to a decrease in COVID-19 website errors and alerts. Model 2 shows that the extensiveness of a state’s accessibility policy reduces the website errors for a COVID-19 information website by a factor of 0.66 ( $B=$ 0.41, $p=$ 0.01) We find partial support for Hypothesis 3.

Hypothesis 4 claims that conservative states will have COVID-19 websites that increase in website errors and alerts. Model 2 finds liberal states are likely to have lower errors by a factor of 0.59 ( $B=$ 0.53, $p=$ 0.04) for COVID-19 information websites. Model 3 shows that liberal states are likely to have lower alerts by a factor of 0.70 ( $B=$ 0.35, $p=$ 0.09) for COVID-19 vaccine locator websites. We find partial support for Hypothesis 4.

5. Discussion and conclusion

During the COVID-19 pandemic, state governments relied on COVID-19 related websites to communicate important information about the virus and vaccination efforts. These websites were published quickly and updated frequently to keep up with the ever-changing pandemic. This rapid pace produced inaccessible websites raising concerns that state health agencies were reinforcing ICT inaccessibility barriers (Moreno et al. 2018). As website accessibility issues were identified, state health agencies engaged in single and double loop organizational learning to increase ICT accessibility.

We examine two potential influencing factors for the single-loop learning: user-end complexity and website launch timing. We find no relationship between the user-end complexity of a website and changes in accessibility. We do find the earlier a COVID-19 related website went live, in relation to important pandemic dates, the less likely it is that website accessibility improves. While state health agencies may consider accessibility at the initial stage of website creation, that consideration may lag as content is edited (Barrett & Greene, 2021). Information changed rapidly and government services faced overwhelming demands throughout the course of the pandemic (Carter & May, 2020); the practical issue of updating complex websites frequently may explain why organizations did not engage in the learning process to improve accessibility.

With double-loop learning, we examine the influence of internal state level accessibility policies and partisanship on a state health agency’s values and norms. We find that internal policies influence learning leading to an improvement in fundamental accessibility (i.e., a reduction in errors), both COVID-19 related websites over time. The more extensive the state’s accessibility policy, the more likely the state is to have a fundamentally accessible website. A similar pattern exists for states that elected the 2020 Democratic presidential candidate. The connection between a state’s accessibility policy and a state health agency’s learning process point to the potential importance of detailed compliance expectations established in more extensive accessibility policies. The partisan vantage point of health policy (Pollack & Bagenstos, 2020) may explain the influence of partisanship on an agency’s learning process. As Dekker and Hansén (2004) point out, organizational learning can be positively influenced by political attention. In 2020, leading Democratic politicians focused on presenting comprehensive disability-focused policies (Mosely, 2020), which may have influenced the state health agency to adjust internal values and norms to produce a more accessible website.

State accessibility policy and partisan leaning have no influence on decreasing alerts (i.e., improving accessibility on a best practices level). The lack of improvement in accessibility on the best practices standard is unfortunate, but not surprising. The incentive to move beyond fundamental accessibility and required legal compliance to a standard of best practices is low, due to the cost, time, and required resources (Bai, 2019). However, when websites follow the best practices of accessibility, the experience is improved for all users (Bai, 2019). As we find support that organizational learning is occurring, we believe that an additional commitment of resources could catalyze increased accessibility at the best practices level.

Organizational learning can help researchers understand why organizational outputs change over time (Basten & Haamann, 2018). With our focus on factors influencing agency learning around accessible websites, we did not study the learning process or capacity that shifted (Wang & Huang, 2013). Future research may address this limitation by studying the learning capability of state health agencies (Jerez-Gomez, Céspedes-Lorente, & Valle-Cabrera, 2005) or identifying the learning process itself (Huber, 1991). We feel our research captures overall learning around an organization’s output as a response to website accessibility issues, rather than a microcosm of individual changes in behaviors or attitudes.

We chose to examine the accessibility of COVID-19 related websites a year into the pandemic to capture more regular, albeit frequent, changes to COVID-19 related websites while avoiding the frenetic chaos that encapsulated the first year of the pandemic. This research serves as an exploratory study that can be applied across different agencies in state or local government that engage in regular website maintenance. The amount of data stored in the Internet Archive allows for more data points to be collected over longer periods of time. We encourage other researchers to examine the changes in accessibility local governments websites, and other ICT forms, both within and outside the U.S.

This paper contributes to the literature and theory in several ways. First, we provide insight into how different factors influence a state health agencies’ learning processes as evidenced by improved website accessibility. Second, public administration researchers rarely focus on the inequities experienced by the disability community (Sabharwal, Levin, & d’Agostino, 2018), which are perpetuated by an ever-growing reliance on technology. Members of the disability community face untenable barriers when accessing government ICTs. State health agencies, and other government units, must learn from current website accessibility failures to improve their service for all constituents. More research is needed to understand what other factors may influence organizational learning about website accessibility, including organizational characteristics. State governments should consider doing more than the bare minimum to increase ICT accessibility for constituents, especially the disability community.

Footnotes

Supplementary data

The supplementary files are available to download from https://dx-doi-org.web.bisu.edu.cn/10.3233/IP-220045.

Author biography

Michelle Allgood is an assistant professor in the University of New Mexico’s School of Public Affairs. Her research focuses on public management, workplace coping and employee well-being, and equity and access issues, especially for members of the disability community.

Ashlee Frandell is an assistant professor at University of Nevada at Las Vegas. Her research interests are on technology use and equity in the public sector, which includes public-private partnerships, STEM workforce mobility and policy communication and the inclusion of disadvantaged groups.

References

Adolph

Amano

Bang-Jensen

Fullman

, & Wilkerson

(2021). Pandemic politics: Timing state-level social distancing responses to COVID-19. Journal of Health Politics, Policy and Law, 46(2), 211-233.

Argyris

, & Schön

D. A.

(1996). Organizational learning II: Theory, method, and practice (2nd ed.). Reading, MA: Addison-Wesley.

Bai

(2019). The Relationship between Website Accessibility and Usability: An Examination of U.S County Government Online Portals. Electronic Journal of e-Government, 17(1), 47-62.

Baker

P. M.A.

Hanson

, & Myhill

W. N.

(2009). The Promise of Municipal WiFi and Failed Policies of Inclusion: The Disability Divide. Information Polity, 14(1-2), 47-59.

Balasubramanian

Al-Ahbabi

, & Sreejith

(2020). Knowledge management processes and performance: The impact of ownership of public sector organizations. International Journal of Public Sector Management, 33(1), 1-21.

Barbieri

, & Salvatore

(2010). Incentive power and authority types: Towards a model of public service delivery. International Review of Administrative Sciences, 76(2), 347-365.

Barrett

, & Greene

(2021). The Growing Challenge of Producing ADA-Compliant Accessible Websites. Government Finance Review, 37(5), 74-77.

Basten

, & Haamann

(2018). Approaches for Organizational Learning: A Literature Review. SAGE Open, 8(3), 2158244018794224.

Blessett

Dodge

Edmond

Goerdel

Gooden

Headley

Riccucci

, & Williams

(2019). Social equity in public administration: A call to action. Perspectives on Public Management and Governance, 2(4), 283-299.

10.

Bowser

Long

Novak

Parker

, & Weinberg

(2021). Stitching together a solution: Lessons from the open source hardwareresponse to COVID-19. Engelberg Center on Innovation Law & Policy Report.

11.

Carter

D. P.

, & May

P. J.

(2020). Making sense of the US COVID-19 pandemic response: A policy regime perspective. Administrative Theory & Praxis, 42(2), 265-277.

12.

Centers for Disease Control and Prevention. (2019). Disability and Health Data System (DHDS) Data [online]. [accessed Feb 18, 2022]. https://dhds.cdc.gov.

13.

Choi

Allgood

, & Swindell

(2022). Policymaking During COVID-19: Preemptive State Interventions and the Factors Influencing Policy Implementation Success. Public Performance & Management Review.

14.

Choi

, & Chandler

S.M.

(2020). Knowledge vacuum: An organizational learning dynamic of how e-government innovations fail. Government Information Quarterly, 37(1), 101416.

15.

Cook

, & Harrison

T. M.

(2015). Using public value thinking for government IT planning and decision making: A case study. Information Polity, 20(2, 3), 183-197.

16.

Dekker

& Hansén

(2004). Learning under Pressure: The Effects of Politicization on Organizational Learning in Public Bureaucracies. Journal of Public Administration Research and Theory, 14(2), 211-230.

17.

Diefenbach

(2009). New Public Management In Public Sector Organizations: The Dark Sides of Managerialistic “Enlightenment.” Public Administration, 87(4), 892-909. doi: 10.1111/j.1467-9299.2009.01766.x

18.

Dobransky

, & Hargittai

(2016). Unrealized potential: Exploring the digital disability divide. Poetics, 58, 18-28.

19.

Doherty

K. M.

Lewis

D. E.

, & Limbocker

(2019). Executive Controland Turnover in the Senior Executive Service. Journal of Public Administration Research and Theory, 29(2), 159-174.

20.

Dunlop

C. A.

(2017). Pathologies of policy learning: what are they and how do they contribute to policy failure? Policy & Politics, 45(1), 19-37.

21.

Gkeredakis

Lifshitz-Assaf

, & Barrett

(2021). Crisis as opportunity, disruption and exposure: Exploring emergent responses to crisis through digital technology. Information and Organization, 100344.

22.

Gilson

Dunleavy

, & Tinkler

(2009). Organizational learning in government sector organizations: Literature review. LSE Public Policy Group, London School of Economics and Political Science, London, UK.

23.

Hereth

Tubig

Sorrels

Muldoon

Hills

, & Evans

N. G.

(2022). Long covid and disability: a brave new world. Bmj, 378.

24.

Holmgren

(2018). Partisan Politics and Institutional Choice in Public Bureaucracies: Evidence from Sweden. Journal of Public Administration Research and Theory, 28(3), 355-370.

25.

Huber

G. P.

(1991). Organizational learning: The contributing processes and the literatures. Organization Science, 2, 88-115.

26.

Jaeger

P. T.

, & Xie

(2009). Developing Online Community Accessibility Guidelines for Persons With Disabilities and Older Adults. Journal of Disability Policy Studies, 20(1), 55-63.

27.

Jerez-Gómez

Céspedes-Lorente

, & Valle-Cabrera

(2005). Organizational learning capability: A proposal of measurement. Journal of Business Research, 58(6), 715-725.

28.

Krahn

& Campbell

(2011). Evolving views of disability and public health: The roles of advocacy and public health. Disability and Health Journal, 4, 12-18.

29.

Leeuw

F. L.

Rist

R. C.

, & Sonnichsen

R. C.

(eds) (1994). Can governments learn? comparative perspectives on evaluation and organizational learning. New Brunswick, NJ: Transaction Publishers.

30.

Lollar

D. J.

, & Crews

J. E.

(2003). Redefining the role of public health in disability. Annual review of public health, 24(1), 195-208.

31.

MacLean

, & Titah

(2022). A Systematic Literature Review of Empirical Research on the Impacts of E-Government: A Public Value Perspective. Public Administration Review.

32.

Massengale

L. R.

, & Vasquez III

(2016). Assessing Accessibility: How Accessible Are Online Courses for Students with Disabilities? Journal of the Scholarship of Teaching and Learning, 16(1), 69-79.

33.

McDonald

K. E.

Williamson

Weiss

Adya

, & Blanck

(2015). The march goes on: community access for people with disabilities. Journal of Community Psychology, 43(3), 348-363.

34.

McQuinn

, & Castro

(2017, March 8). Benchmarking U.S. Government websites. https://itif.org/publications/2017/03/08/benchmarking-us-government-websites.

35.

Meijer

Webster

William

, & Contributing, Authors. (2020). The COVID-19-crisis and the Information Polity: An Overview of Responses and Discussions in Twenty-one Countries from Six Continents. Information Polity, 25(3), 243-274.

36.

Moreno

Martínez

Muguerza

, & Abascal

(2018). Support resource based on standards for accessible e-Government transactional services. Computer Standards & Interfaces, 58, 146-157.

37.

National Archives and Records Administration. (2021). 2020 electoral college results. National Archives and Records Administration. Retrieved June 2, 2021, from https://www.archives.gov/electoral-college/2020.

38.

National Council on Disability. (2014). Effective communications for people with disabilities: before, during, and after emergencies evaluated. National Council on Disability. Available at: https://ncd.gov/newsroom/2014/05272014. Accessed April 27,2020.

39.

National Telecommunications and Information Administration. (2022, October 5). Digital Nation Data Explorer. Digital Nation Data Explorer. Retrieved February 16, 2023, from https://ntia.gov/other-publication/2022/digital-nation-data-explorer#sel=internetAtHome&demo=disability&pc=count&disp=chart.

40.

Norris

D. F.

, & Moon

M. J.

(2005). Advancing e-government at the grassroots: Tortoise or hare? Public Administration Review, 65, 64-75.

41.

O’ Neill

J.L.

(2021). Accessibility for all abilities: How universal design, universal design for learning, and inclusive design combat inaccessibility and ableism. Journal of Open Access to Law, 9(1), 1-15.

42.

Palmer

Z. B.

, & Palmer

R. H.

(2018). Legal and Ethical Implications of Website Accessibility. Business and Professional Communication Quarterly, 81(4), 399-420.

43.

Patky

(2020). The influence of organizational learning on performance and innovation: a literature review. Journal of Workplace Learning.

44.

Pee

L. G.

, & Kankanhalli

(2016). Interactions among factors influencing knowledge management in public-sector organizations: A resource-based view. Government Information Quarterly, 33(1), 188-199.

45.

Pollack

H.A.

, & Bagenstos

S.R.

(2020). The Hidden Disability Consensus in the 2020 Campaign. JAMA Health Forum, 1(2): e200152.

46.

Rashman

Withers

& Hartley

(2009). Organizational learning and knowledge in public service organizations: A systematic review of the literature. International Journal of Management Reviews, 11, 463-494.

47.

Rimmer

(2011). Building a future in disability and public health. Disability and Health Journal, 4, 6-11.

48.

Rubaii-Barrett

, & Wise

L. R.

(2008). Disability access and e-government: An empirical analysis of state practices. Journal of Disability Policy Studies, 19(1), 52-64.

49.

Ryerson

A.B.

Rice

C.E.

Hung

Patel

S.A.

Weeks

J.D.

Kriss

J.L.

Peacock

Lu,

P-J.

Asif

A.F.

Jackson

H.L.

, & Singleton

J.A.

(2021). National Immunization Survey Adult COVID Module, United States, May 30–June 26, 2021. MMWR Morb Mortal Wkly Rep 2021, 70, 1365-1371.

50.

Sabatello

Burke

T. B.

McDonald

K. E.

, & Appelbaum

P. S.

(2020). Disability, ethics, and health care in the COVID-19 pandemic. American Journal of Public Health, 110(10), 1523-1527.

51.

Sabatello

Landes

S. D.

, & McDonald

K. E.

(2020). People with disabilities in COVID-19: Fixing our priorities. The American Journal of Bioethics, 20(7), 187-190.

52.

Sabharwal

Levine

, & D’Agostino

(2018). A Conceptual Content Analysis of 75 Years of Diversity Research in Public Administration. Review of Public Personnel Administration, 38(2), 248-267.

53.

Sonnenberg

(2020). E-Government and Social Media: The Impact on Accessibility. Journal of Disability Policy Studies, 31(3), 181-191.

54.

Tagg

(2010). The learning-paradigm campus: From single-to double-loop learning. New Directions for Teaching and Learning, 2010(123), 51-61.

55.

Turnbull,

, III., & Stowe

(2001). Five Models for Thinking About Disability: Implications for Policy Responses. Journal of Disability Policy Studies, 12(3), 198-205.

56.

U.S. Access Board. (n.d.). 508 Chapter 2: Scoping Requirements. Revised 508 Standards and 255 Guidelines. Retrieved January 20, 2022, from https://www.access-board.gov/ict/.

57.

U.S. Food and Drug Administration. (11 Dec 2020). FDA takes key action in fight against COVID-19 by issuing emergency use authorization for first COVID-19 vaccine. U.S. Food and Drug Administration. Retrieved January 17, 2022, from https://www.fda.gov/news-events/press-announcements/fda-takes-key-action-fight-against-covid-19-issuing-emergency-use-authorization-first-covid-19.

58.

Voorberg

Bekkers

Timeus

Tonurist

, & Tummers

(2017). Changing public service delivery: Learning in co-creation. Policy and Society, 36(2), 178-194.

59.

Vince

, & Saleem

(2004). The impact of caution and blame on organizational learning. Management Learning, 35(2), 133-154.

60.

Wang

Y. L.

, & Huang

(2013). Organizational learning and human resource: A review of the theory and literatures. International Proceedings of Economics Development and Research, 61, 56-56.

61.

WebAIM. (n.d.). WAVE Web Accessibility Evaluation Tool. https://wave.webaim.org/about.

62.

Wentz

Jaeger

P. T.

, & Lazar

(2011). Retrofitting accessibility: The legal inequality of after-the-fact online access for persons with disabilities in the United States. First Monday, 16(11-11).

63.

West

D. M.

(2000). Assessing e-government. Providence, RI: Brown University, Taubman Center for Public Policy.

64.

World Health Organization. (2022). WHO Coronavirus (COVID-19) Dashboard. https://covid19.who.int.

65.

Yang

Y. T.

, & Chen

(2015). Web Accessibility for Older Adults: A Comparative Analysis of Disability Laws. The Gerontologist, 55(5), 854-864.

66.

Youngblood

N. E.

, & Mackiewicz

(2012). A usability analysis of municipal government website home pages in Alabama. Government Information Quarterly, 29(4), 582-588.

67.

Younis

(1997). Customers’ expectations of public sector services: Does quality have its limits?. Total Quality Management, 8(4), 115-129.

68.

D. X.

, & Parmanto

(2011). U.S. state government websites demonstrate better in terms of accessibility compared to federal government and commercial websites. Government Information Quarterly, 28(4), 484-490.

69.

S. Y.

(2021). A review of the accessibility of ACT COVID-19 information portals. Technology in Society, 64, 101467-101467.

U.S. state health agencies and organizational learning: An exploratory analysis of website accessibility during COVID-19

Abstract

Keywords

1. Introduction

2. Online communication, inclusion, and the disability community

2.1 Organizational learning

2.3 Single loop: Timing of website launch

2.4 Double loop: State accessibility policies

2.5 Double loop: Partisanship

3. Data and methods

1 We include clustered standard errors in the analysis to account for any duplication of information.

Table 3 Negative binomial regression results

Footnotes

Supplementary data

Author biography

References

¹
We include clustered standard errors in the analysis to account for any duplication of information.

Table 3
Negative binomial regression results