Webportal vs google for finding government information on the web: From a website-centric approach to a web ecology perspective

Abstract

Webportals – websites that operate as front doors or guides into government on the web – are central to government web strategy and presence. However, little is known about their success in enabling people to quickly and accurately access public sector information and services. In these days of Google and generic web search engines, government webportals are not the only way to find government on the web. This paper argues that an effective evaluation of government webportals requires shifting from a website perspective to a whole-of-web (or web ecology) perspective. This perspective is illuminated by an online quasi-experiment of the effectiveness of the British government’s webportal, www.gov.uk. Participants’ performance in using the webportal to find information about public services were compared with those using commercial web search tools (such as Google). There was mixed evidence that the portal provided greater accuracy in finding public service information, but no evidence for greater speeds. The findings suggest that government web strategy focus less on creating large webportals and more on small functionally-defined web units that offer enhanced opportunities for commercial search engine discoverability and flexibility for change.

Keywords

E-government webportal web ecology navigation UK online experiment IT evaluation

1. Introduction

Over the 20 years since the popularization of the internet, governments throughout the world have developed an extensive web presence as a new communication channel to variously enhance administrative efficiency, service accessibility and information provision, while providing new types of consumer-citizen engagement and services. With the wholesale shift to online government, key concerns arise about overall government web strategy and how to design websites to enable people to find information about public services, and to interact and transact with public service providers. Webportals are a common web design approach used by governments to achieve these objectives.

A webportal is a website that operates as an entranceway onto, or guide into, the web (Van Deursen, & Van Dijk, 2009). Within the world wide web, a vast unordered space of over one billion websites, webportals help users navigate within a localized ‘web ecology’ of interconnected webpages and websites. Governments have readily created webportals to provide such a ‘gateway’ to government websites, but also to achieve long-desired goals for more holistic, joined-up government (6, 1997; 6, Leat et al., 2002; Christensen & Lægreid, 2007). Advocates argue that portals mean that people do not need to understand the structure of government or the names of government agencies in order to find and engage with government online. Indeed, the creation of webportals “can help to make even uncoordinated government look coherent from a citizen’s perspective” (Eynon & Dutton, 2007, p. 239).

Government webportals are now a mainstay, and often the centerpiece, of a government’s web presence and web strategy. Governments worldwide spend sizeable sums on building their portals as their formal public online face and ‘single point of access’, to both define their online image and enhance citizens’ access to government. As Wahed and El Gohary suggest: “To the greatest extent possible, citizens should be able to do everything they have to do or want to do with their government through one egovernment [sic] portal” (Wahed & El Gohary, 2013, p. 295). Nevertheless, how successful are government webportals in accurately and efficiently helping people find information about and access to public services? This is the question that this paper seeks to address.

In addressing this question, we argue that the dominant approaches to evaluating webportals (and websites more broadly) are fundamentally limited due to their focus on the portal (site). Rather such evaluations must begin from a perspective of how users experience the internet. As there are proverbially multiple ways to skin a cat, there are multiple ways of finding government-related (information and services) online, more often from commercial web search engines which do not treat the web as a series of websites, but webpages (and increasingly units of information within webpages). Moreover, users can often find government-related information and services on websites not operated by government. These realities mean that to understand how useful government webportals are, it is necessary to understand the multiple and counterfactual ways government-related information and services may be found, beginning from outside of webportals. To advance this important conceptual reorientation, we propose the web ecology as an innovative approach to conceptualizing the web, instead of the dominant perspective of websites connected to websites.1

¹
This limited perspective underpins virtually all web analytical approaches to evaluation of web presence, is used in international rankings of government web presence, and web network analyses.

Thus in answering the question – how successful are government webportals in accurately and efficiently helping people find information about and access to public service? – the next section examines the stated rationales of government webportals, which are typically construed as fast and trustworthy places to locate government information and services. We then outline how government webportals are traditionally evaluated by focusing on the website, and argue that if governments (and private sector organizations) are to effectively evaluate their webportals they need to compare the performance of locating government on the web via webportals compared to commercial search engines, such as Google. This in turn requires a web ecology perspective, a novel conceptual framework developed in this paper. Based on this framework, original empirical research was undertaken to assess the relative performance of the UK government’s webportal, Gov.uk, a portal specifically designed to enhance user experience of British government on the web. We did this by posing a change of address scenario that asks study participants to find information about public services and comparing the accuracy and timeliness of research participants by the web strategy they used (commercial search or government portal). The concluding discussion examines what the findings mean for the conceptualization of the world wide web in general, the approach to government web search in particular, and the practical design of governments’ web presence. In summary, this paper is significant for not only for providing a new approach to evaluating websites, but outlines a novel conceptual approach – that of a web ecology – that is coupled with an innovative digital research tool.

To be sure, this paper is not directly about how people find things on the internet, but about assessing the effectiveness of a website by comparing routes to finding information within a website compared to other routes. There exists an enormous body of information science literature about information seeking, including the various search terms used, how search efficacy and approach varies by people’s characteristics, and the different online information strategies used (Chen & Macredie, 2010; Markey, 2007; Pang et al., 2016; Van Deursen & Van Dijk, 2009; White & Drucker, 2007; Zhou & Society, 2013). This literature provides generalised insights about information seeking that can inform the design of government webportals, but it does not provide any insight into the performance of particular webportals, or comparative performance of the online presence of different public service domains, as is examined here.

2. Rationales and designs of government webportals

Government webportals arise from diverse agenda, designed for varying purposes and have different design logics, architectures and functionalities. Some portals are designed as a navigational tool to content on other government websites (e.g., the USA portal, www.usa.gov). In contrast, some portals are designed as a single enormous website to consolidate and replace separate government websites (e.g. the UK portal, www.gov.uk) by holding consistent, consolidated information.2

²
One view is that the Gov.uk is not strictly a portal as it does not link users to external websites, but is a platform. We argue that as it is designed to operate as an entranceway or guide to government on the web it works as a portal, albeit the linkages are largely within the website, not external.

Henman and Graham (under review-a) have recently characterized these design logics as respectively information referral and information repositories.

Government webportals also have different types of functionality. Some focus on access to information. Some provide access to integrated, whole-of-government electronic transactions (such as Australia’s, www.my.gov.au). Others are aimed at open government and citizen e-participation through whole-of-government access to open data, e-petitions, e-consultations and so on. How these various architectures perform in meeting their objectives, particularly in relation to the experience of citizen-consumers, is not well studied.

The rationale for government webportals is evidenced by what governments themselves say about their websites. In short, they are the online location in which to start finding government online. To illustrate, the UK government’s webportal Gov.uk was created in 2012 to absorb a myriad of proliferating government websites (as was its predecessor Direct.gov.uk in 2004). This is evident on the home page where it states that “The websites of all [35] government departments and [385] many other agencies and public bodies have been merged into GOV.UK”. The UK portal was thus created as a super-size government website. In addition, its front-page header (Fig. 1) clearly states its other rationale: “The best place to find government services and information. Simpler, clearer, faster” (emphasis in original). This header further highlights that the home page is the place to start, with a clear welcome address. As is ubiquitous among webportals, there is the prominent presence of a search tool.

Figure 1.

Homepage banner on www.gov.uk. Source: Excerpt from https://www.gov.uk/. Copyright of the British Crown, published under Open Government Licence.

As a second example, consider the Australian government’s 2015 initiative to reform its portal Australia.gov.au. Paul Shelter, the inaugural CEO of the Digital Transformation Office explained the objective of the new webportal by observing “that everyone who needs to use government services should be able to find what they need, quickly and easily” (2015). In a similar view, the Australian portal says of itself:

There are lots of government websites and we know it can be hard to find the information you need. Australia.gov.au puts you on the right path by linking to information and services on around 900 Australian government websites as well as selected state and territory resources (Australia, 2018a).

This webpage also highlights a further rationale of the webportal, as it provides a “trusted source of government information” (Australia, 2018a). In contrast to the UK’s rationale, it is clear that the Australian webportal is about directing the user to other “government websites” for the desired information or service.

The USA webportal, Usa.gov, explains its rationale in similar terms with its home page stating it is “your online guide to government information and services”. Its primary purpose to find government online is reinforced by its ‘How to Use’ page (www.usa.gov/how-to-use) that explains that people can browse by topic, use its search box to return only government websites, and to provide contact information of all Federal government agencies. It reinforces that the webportal, unlike commercial web search tools, returns only trusted government sites without advertising.

These three examples of current government webportals clearly articulate their primary rationale and purpose to be the premier gateway to government information and services. It is implicit that the way to find such information is through the webportal’s homepage. Their inclusion of a prominent search box reinforces this, regardless of whether the located webpage is internal or external to the webportal site. The trustworthiness of a government webportal is contrasted to apparent less-trustworthy commercial whole-of-web search engines as an alternative form to find government-related information and services, as the former only directs users to government-endorsed webpages that are implied to be more accurate.

3. Evaluating government webportals

As with public policies and services, evaluation of webportals is a necessary activity for both public accountability and improvement. The evaluation of public sector information technology projects is a fraught area, often done after costly failures, rather than necessarily built within their original project brief (Gauld & Goldfinch, 2006). Given the justification for creating government webportals is to find trustworthy public sector related information and services quickly, how well do webportals perform in achieving these aims?

3.1 The dominant website centric approach

As done in the commercial sector, regular monitoring of government websites typically occurs using ‘web analytics’. Web analytics involve the collection and analysis of web data to understand the usage and effectiveness of a website. Its practice has been in existence since shortly after websites were created, but over the last decade web analytics has become an enormous science and industry, with many companies and tools on offer, such as Google Analytics (Analytics.google.com) and Clicky (Clicky.com).

As outlined by Booth and Jensen (2010) (see also Clifton, 2012; Dhyani et al., 2002; Singal et al., 2014), standard areas of website evaluation are: website usage information (including visitor type, visit length, internal search information); referrers (that is, which website a user came from immediately before arriving at the examined website); content analysis (including the path of webpages users visit within the website, the most visited webpages); and quality assurance (such as broken hyperlinks). For example, Google Analytics (Analytics.google.com) provides real-time graphs and reports about user traffic and dynamics. In this way, website evaluation using Google Analytics is operationalized in terms of maximizing not only traffic volume, but also return rates, visit length, and quality of traffic (Moral et al., 2014, p. 577).

Web evaluation is also undertaken at the webpage or interface design level. Such analysis focuses on the location of items on the screen, where users look, how long it takes users to find relevant information on a webpage, and the usability of a webpage layout and functions (Wang & Senecal, 2007). Accessibility of webpages, such as for people with disabilities or on different devices, is a long-standing approach to evaluating web design (Nielson, 2000). Customer satisfaction with a website using surveys is a further form of assessment.

In each of these methods, the focus on the website and the webpage as a component of that website. These metrics analyze a government’s web presence at the website level. Such metrics potentially provide a means for assessing government web portals, in terms of whether or not people found what they were looking for in a timely fashion, by looking at clickstreams within the webportal, but they can only do so once a user has entered the webportal. These metrics provide little insight into the process by which users come to the government webportal. Our approach addresses this important limitation.

Significantly, given the rationale for a government webportal is that it is “The best place to find government services and information. Simpler, clearer, faster”, then it necessarily requires a comparison with other approaches to finding government services and information, such as through commercial search engines or other websites.3

³
The Amazon-owned Alexa company (Alexa.com) provides comparative web statistics for ‘competing’ websites. However, it is unclear what a competing website is for a government portal or department, and Alexa does not appear to provide a service for government sites.

Indeed, users seeking information about public services may in fact never encounter the government portal. Consequently, evaluations of government webportals need to focus less on the website itself, and more from how users experience the web, recognizing that users will often not start at the portal. This, we argue, requires governments to evaluate their webportals at the level of the entire government web footprint, rather than a single site, within the entire World Wide Web. For that, a web ecology, not a website, perspective of government on the web is required. To be sure, evaluations at the website level offer important, but partial, insights that need complementary knowledge.

3.2 A web ecology perspective

The web is a vast unordered space of webpages grouped into websites and interconnected by hyperlinks. Webpages (and websites) are hyperlinked to other webpages (and websites) by relevance. Those hyperlinks are an equally important design feature of the web as webpages. A web ecology refers to the hyperlink network of webpages, a network or system of diverse, inter-connected or inter-related webpages (Henman & Graham, under review-b). A web ecology perspective alerts us to the flows through hyperlinks, just as much to the content and functionality of webpages that is dominant in many ranking exercises.4

⁴
A web ecology perspective has similarities with the information governance perspective advocated by Mayer-Schönberger and Lazer (2007).

Government webportals are situated in this large web ecology, inter-connected to webpages in other government websites (within and beyond that jurisdiction and within and beyond that country) and webpages in the full gamut of other websites, including social media, mass media and organizations delivering government services.

A web ecology perspective is more akin to how web users typically experience the web, not at the level of websites, but rather as a navigation via hyperlinks through inter-related webpages that seamlessly cross the boundaries of websites. A web ecology perspective is agnostic of websites, as a defined or formalized set of webpages with shared URLs owned and managed by an organization or agency.

The web ecology perspective is compatible with the clickstream analysis of users within a website, which as explained above can be used to evaluate the effectiveness of a government webportal. The implicit assumption is that users begin at the portal homepage and navigate from there. However, web users increasingly use whole-of-web search engines (such as Google), which enable users to arrive at (internal) webpages of a government portal without going through a website’s homepage or portal. Alternatively, users can arrive at a government webportal from other sites. For example, of all the users arriving at the Australian government’s log-in page of the service transaction portal, My.gov.au, in January 2017, only 24% came directly to the site by typing in the URL, whereas 37% came directly from Google, while the remainder came from a variety of websites or search engines (Australia, 2018b).

When understood within a web ecology perspective, evaluating government webportals means analyzing the web experience of people who seek to engage with government online at the web space level, and not solely at the website level. Evaluations need to understand how and where people start the process of finding government related information and services, and where they find it, and if they even make use of the portal.

Evaluation of government webportals within a web ecology, and not at the website level, is crucial for several reasons. One, it enables evaluators to compare the relative search performance of government webportals against other search engines. Two, government webportals may not contain the government information or services that the user is actually seeking to find, but could be hosted on other government (or non-government) websites. This means that an analysis at the website level does not allow satisfaction with the portal to be determined without reference to where the desired data is ultimately found. Three, government information or services that citizens seek to encounter online are now located in a wide variety of locations including outside of government websites. Increasingly the ‘nodality’ or centrality of government (Hood & Margetts, 2007) is challenged by other online providers of public sector relevant information and services. A focus on the webpages that government webportals link a user to does not recognize the alternative places such information may be available. Thus, not only does a web ecology perspective allow an evaluation of a webportal’s comparative search performance, but it also enables an evaluation of the different landing points that people may end up at, thereby enabling a comparison of the accuracy of the information discovered. In summary, evaluators need to consider how citizens experience and approach the web when finding government information, engaging with public services or transacting with government, because the routes to the ‘solutions’ they find are diverse and the solutions themselves may not be on government websites.

There has been very limited research examining the effectiveness of online portals by examining the web users’ experiences of finding online information or services relating to government, Dunleavy et al. (2007) and Margetts and Hale (Margetts & Hale, n.d.) being exceptions. Both studies undertook an experimental design, whereby some research participants were given a clear ‘open search’ screen as their starting point and an intervention group were given one or more portals as their starting point for their task. In a study of 69 subjects, Dunleavy et al found that those using open search were faster in finding answers to questions posed (2007, pp. 55–56) than those mandated to use the UK government webportal (Direct.gov.uk), but no different in their accuracy.5

⁵

In a similar manner, members of the Democracy.org.uk collective created the website DirectionlessGov.com to race the speed of the government’s Direct.gov.uk search engine against Google and finding that DirectionlessGov.co-Google typically won (Cross, 2007).

Margetts and Hale (n.d.) investigated access to information relating to cross-EU services and life events, such as students undertaking overseas study and loss of a passport. Participants were randomly given an ‘open search’ screen or a suite of portals that might be useful. They found that those who used an open search only-strategy and those who used a portal-only strategy found their answers in shorter average times than swapping strategies. Notably, no tests for statistical significance were undertaken in these studies (only headline descriptive statistics were presented), so no robust conclusions can be made about these results.

4. Conducting the research

4.1 Research design

The effectiveness of the UK government’s webportal, Gov.uk,6

⁶
Although the portal was established in 2012, at the time of our research in mid-2014, some residual webpages from the previous portal Direct.gov.uk were linked from the new portal for functions yet to be fully integrated.

was assessed by measuring the time taken for participants to find public (and private) service information and transaction points. This was done by comparing the performance of users using the government webportal to find the information with that using a commercial search engine. Accordingly. we initially constructed the evaluation as a quasi-experiment. The control (‘open search’) group was given a clear webpage as the starting point for finding the requested information, and the intervention (‘UK portal’) group was given the government webportal homepage as their starting point. Participants were randomly allocated to either the control or the intervention group. Participants in the latter group were advised of the portal’s existence and purpose, and were encouraged, but not required, to use it. We thus adopted a similar approach to that of Dunleavy et al. (2007) and Margetts and Hale (n.d.). Online experiments for social research is relatively uncommon (Margetts, 2011), partly because experiments are not readily possible in social research, although experiments in psychology are well developed, and partly due to the emerging availability of technology.

In order to assess the time and accuracy of finding government service related information and services on the web, hypothetical relocation scenario was posed where participants were asked to imagine moving residential locations: “You are moving to Manchester/London for work and you want to find as much as you can about the city and prepare for your move as possible before you leave by using online sources”. Participants were randomly allocated a Manchester a London scenario (and given a specific address for their new residence), in order to test any differences in results resulting from different local authorities. Participants were then given a range of multi-choice questions relating to seven service domains: public transport; land tax; schooling; GP health services; local planning; private telephone services; and political representation; and open government (see Appendix).7

⁷

Questions 5 (town planning) and 7 (voting) have two choices to select from, whereas the others have three.

The questions were designed to capture a mix of public (and private) services. The questions had been carefully designed and pre-tested by the authors to pre-confirm the correct answers using a range of methods including the government portal and Google. A range of demographic data, such as familiarity with London or Manchester, highest educational level, age and level of internet use, was also collected to identify any possible effects these may have on results.

4.2 Data collection, sample and analysis

To collect data, customized a browser plugin8

⁸
A Firefox plugin that utilises Javascript, PHP, and a MySQL database was built based on previous software developed by Margetts and Hale (n.d.).

was developed to prompt participants with multi-choice questions presented in a left-hand sidebar with the main standard browser format in the large right hand window. Participants would then navigate the web using this browser to locate an answer to each question posed. The software also recorded the web address (URL) for each webpage they visited (e.g., http://www.manchester.gov.uk/counciltax), the time of the click (using the Unix time standard), and their selected answer for each question (from the multi-choice options provided). This data respectively enabled an analysis of the navigation process and techniques used by participants, the time they took and the accuracy of their responses to be recorded.

Participants were recruited in June 2014 via an email invitation to people listed in an Oxford Internet Institute database of past research participants who had indicated willingness to be contacted about future internet related research. The invitations indicated that the research would be conducted in one of four two-hour sessions across two weekdays in an Oxford University owned computer lab. The invitation also indicated that a small honorarium would be paid for participation. In total, 80 adult volunteers were recruited. Just over three-quarters of participants (77.5%) were UK residents. Gender was fairly evenly distributed: 50% male; 44% female; and 6% did not specify a gender. The majority of participants were aged either 20–29 (26%) or 30–39 (29%), with a reasonable representation in other age groups. The majority of participants were employed (40%) or students (29%). A further 15% of participants were unemployed, with 7% retired, 4% housewife/husband, and 5% specifying ‘other/NA’.9

⁹

We undertook regression analyses of the average time each participant took to obtain correct answers to the questions they were given against all the collected demographics. Only self-assessed level of internet competence showed significance ( $p=$ 0.00994).

Table 1 provides a breakdown of the number of participants by intervention group and control group (respectively UK portal or Open search), and by city (London or Manchester). As the difference in performance of participants in the Manchester and London groups – as measured by time and accuracy – were not statistically significant they are not further discussed in this paper. Due to data reliability, one participant’s data was removed, leaving 79 valid respondents.

Table 1

Number of participants per group

	Manchester	London
Open search	20	23
UK portal	18	18

Our quasi-experimental research design involves comparing the performance of the two groups – control and intervention/treatment – based on the assumption that the activities of the intervention group who were given the government webportal homepage as the starting point to answer their questions is different to that used by the control group. Implicit is that the former would use the webportal and the latter would use whatever means they determined. This assumption was not supported by the research participants’ behavior. Rather, the group a participant was placed did not guarantee a particular mode of navigation to answer the given question. Participants in the open search group could readily begin by typing www.gov.uk into their browser and proceeding from there, which effectively makes them part of the portal group. Similarly, participants in the portal group could type www.google.co.uk or a search phrase into their browser, in which case they are effectively in the open search group. Moreover, participants could begin searching in one way (portal or open search), become frustrated and give up, and try an alternative approach. In short, in a web ecology the starting point does not determine the approach one takes to finding data. The ease with which a person can change modes of navigating the web undermined the quasi-experimental design.

In recognizing these realities, the evaluation of the webportal needed to be based on the nature of the search people undertook, not on the group they were allocated. An analysis of people’s information seeking behaviors in answering the multi-choice questions was conducted by referring to the URLs people visited. Six different modes for answering questions were identified:

•

Search-Only is the strategy of using a generic search engine (such as Google, Yahoo or Bing) to obtain an answer. We regard the situation when people move from Google search results to a webpage and then navigate from there to other webpages as ‘search only’ as the search was the primary way they obtained their result, and used localized navigation to pinpoint the answer.

•

Gov-Search is when a participant starts with the government portal, but then goes to a generic search engine and uses the results from that to proceed to an answer as in Search-Only.

•

Gov-Only involves starting on the government webportal and navigating from that portal (including using its internal search tool or navigating hyperlinks from page to page) to find their answer.

•

Search-Gov is the reverse strategy to Gov-Search, namely they start with a generic search, but then go to the government portal (or in some rare circumstances do a search within a separate government website) to complete their search and answer. However, people who use a generic search to go to a government webpage (other than the portal homepage) to navigate to an answer are regarded as Search-Only and not Search-Gov.

•

Navigate is when a participant uses webpage hyperlinks to navigate their way from an initial webpage that is neither an open search nor government portal to the result. In the cases analyzed in this paper this occurs when a person types in a known website or webpage (e.g., www.nationalrail.co.uk) and proceeds from there.

•

Nil is used for people who answer the question from their current webpage, which typically means that they answer the question based on their own knowledge or guess.

Importantly, the search mode used does not reflect the webpage location on which respondents finally use to answer the question. Furthermore, people allocated to the portal group can only be allocated the Gov-Search, Gov-Only, Navigate or Nil categories, whereas those in the open search group can only be categorized as using the Search-Only, Search-Gov, Navigate or Nil categories. Accordingly, Gov-Search includes both people who started searching through the portal then gave up and used a generic search, and those that ignore the portal and immediately proceed to a generic search.

Figure 2.

Proportion of search types used, by question.

Figure 2 illustrates the proportion of six navigation types people used for each of the seven questions. It is notable that there is no consistent pattern between different domain questions. Significantly, navigation (or responding based on one’s own knowledge or guess) was rarely used, except for the town planning question (where 22 percent adopted this mode). Search-Only is the most common, used by half of the participants in three of the seven questions (Transport, GP & Telephone). Gov-Search is also widely used, especially in those same three questions. This suggests that these three questions were very difficult to find an answer through searching within government websites, most commonly the portal.

The ‘switchers’, those that start in the portal or open search, but end up swapping to a generic search or government web mode respectively. The switcher group ranges from a quarter to 43% of respondents, thereby demonstrating that group allocation in the experiment did not strongly shape online search behavior.

Understanding the processes of switching from one information seeking approach to another can be of interest (Aula et al., 2010; Jiang et al., 2014), as is comparing the relative time and accuracy of the six different approaches. For the purposes of this paper, we simplify the evaluation of the effectiveness of the UK webportal by focusing on the technique that a user ultimately used to obtain their answer. Consequently, we grouped Search-Only and Gov-Search together as both groups ultimately achieved their answer through a generic search (we denote Generic Search). Similarly, groups Search-Gov and Gov-Only are grouped as they ultimately used the government portal (or occasionally a search on a different government website) to answer the question (this group we denote Government Web).

5. Results

Figure 3.

Proportion of major search types used, by question.

Figure 3 provides the breakdown of the web navigation technique participants ultimately used to obtain their answer: by the assistance of a generic search; by use a government portal/search (Government Web); by navigating from a website that is not the government portal or a commercial search engine (Navigate); or by simply answering the question from their personal knowledge or guess (Nil). The figure demonstrates that only in the school and voting questions did the majority use the Government Web search strategy (56% and 61% respectively), whereas a Generic Search strategy was used by the majority of participants in all other questions, sometimes this was close to 100% (Transport, GP and Telephone), in others it was closer to half of the participants. It is tempting to assume that the minimal success of Government Web in these three services is because they are privately provided. However, in the UK primary health care is largely delivered through GPs employed under the National Health Service, not through a private market.

The nature of a search strategy itself used to obtain an answer does not indicate the effectiveness of the strategy. How successful were these strategies in terms of accuracy and timing of responses?

Figure 4.

Percentage correct answer by major search type, by question.

When considering the accuracy and timing of search strategies, we removed from consideration strategies that were used by less than five per cent of respondents. Figure 4 provides the percentage of correct answers for each search strategy used: Generic Search; Government Web; and Navigate. Accuracy ranges from about 40% for people using the Government Web approach for the town planning question, to 98% for the Government Web approach in the voting question. Apart from the town planning question, people using the Government Web strategy obtained high accuracy (87% and above). The Generic Search strategy does not appear to exceed the accuracy of the Government Web approach (except in town planning). At times it is similar (GP, voting) and in others underperforms (council tax and schooling) the Government Web approach. Importantly, in both transport and telephone the Government Web strategy did not appear to be a viable approach to find an answer at all (which could be attributed to the fact that both these domains are delivered by private companies, albeit with considerable government regulation and oversight). In summary, search techniques through government websites are sometimes more accurate than generic search techniques,10

¹⁰

T-tests demonstrated statistical significance that the Government Web strategy was more accurate than Generic Search in the case of council tax ( $p=$ 0.013), and schooling ( $p=$ 0.008), and Generic Search was more accurate than Government Web for town planning ( $p=$ 0.007).

but only when they can produce an answer. Note that people who used no search method (Nil) – they answered based on their knowledge – were very accurate with voting, but just over half were in the town planning approach.

However, accuracy may not equate to efficiency. Figure 5 provides the mean time respondents took to produce accurate responses by search strategy type.11

¹¹

Times of two standard deviations or more above the mean were removed to ensure that circumstances such as toilet stops or system malfunctions were not included.

There appears to be some questions in which the Generic Search strategy is faster and some in which Government Web is faster. However, when the differences were tested for statistical significance using a T-test, only the town planning question (Generic Search performed better, 138 versus 207 seconds,

p=

0.030) and the voting question (Government Web performed better, 51 versus 69 minutes,

p=

0.048). As expected, using one’s own knowledge (i.e., Nil) was quick, with a mean time of about half a minute. It is also important to observe that there is a great variety in mean response times between questions, with information about local schooling, GPs and public transport taking from three to five minutes, while information about mobile telephone coverage and voting took around one minute.

Figure 5.

Mean time (seconds) to find correct answer by major search type, by question.

Our final consideration relates to the webpage on which participants obtained their answers. One might assume that the search strategies above are different routes to the same eventual finishing point. However, this is not the case. Indeed, Fig. 6 shows that four of the seven questions were answered using a mix of government and non-government webpages.12

¹²

For this analysis, government webpages as those with a gov.uk domain name, while other public sector sites include the British National Health Service, which has a separate domain name (nhs.uk), and schools (.sch.uk). The non-government sites are primarily commercial (.co.uk), but also include a few net.uk and org.uk sites (combined in Fig. 6 as .co*).

This finding reinforces that when analyzing the performance of government portal a whole-of-web perspective is more insightful than one focused on a website. Portal performance in finding accurate and timely information necessarily requires a consideration of other websites that offer similar information to users.

Figure 6.

Proportion website domains used to find correct answers, by question.

To illustrate the diversity of the complex web ecology for finding information consider the case of finding the best local GP. Figures 4 and 5 demonstrate that both Generic Search and Government Web search strategies are reasonably effective in finding the correct result, yet Fig. 6 demonstrates that about a third of the correct results was obtained from non-government webpages. A closer look at where respondents found their answers shows a diversity of webpages – from those located on the British National Health Service – nhs.uk – domain, to a variety of commercial websites (mysurgery.co.uk, postcodes.findthebest.co.uk and gp-practices.findthebest.cok.uk) to one response from the UK government webportal gov.uk.

6. Discussion and conclusion

6.1 Evaluating the findings

This paper started by observing considerable effort in building government webportals as a key plank in government online strategy, yet government webportals are often not evaluated in terms of their objectives to assist people to find government-related information, services and democratic processes quickly and accurately. To progress such evaluations, we argued that rather than a focus on people’s experience of navigation within a government webportal, a perspective that locates government’s webportal within the wider web is required as this better reflects how individuals encounter government on the web. This perspective recognizes that people do not experience websites qua websites, but as a series of web pages (within a website) within a wide network of webpages joined by hyperlinks, often navigated through generic search engines, such as Google.

As an analogy, websites can be thought of as houses, where each website-house is carefully designed for people to enter through the front door, and be directed from the front hall to specific rooms in the house in order to find what they are after, before rummaging through the room to find it. Governments have built variously sized houses as webportals to help people find things in government’s real estate portfolio. In the case of the UK, the strategy has been to build a massive mega-building to replace most of their previously owned houses.

Unfortunately, generic search engines disrupt the house building model of websites. Rather than entering in through the front door, search engines provide hyperlink entry to houses through windows, water mains and air conditioning vents, and often teleport or ‘apparate’ (Rowling, 2000, p. 7) people directly to the specific item within a room they seek. Moreover, the web ecology perspective highlights that what people wish to find on the web, can often be found in many different website-houses (see Fig. 6). With the dominance of search engines, governments cannot even guarantee that people will turn up in their webportal-house at all, or even in any of their website-housing portfolio.

Given these circumstances, public administration and government web strategists must importantly consider whether webportals are effective in terms of cost to government and efficiencies for users. To answer this question, we applied new digital research methods that involve tracking the way people find information about English government policies and services on the web. We initially used a quasi-experimental design whereby half of the participants were given a clear screen to go and find the information using their own resources, whereas the other half were given the UK government’s webportal. However, participants’ starting point did not determine the way they ultimately sought to address their task. By looking at how they eventually found their answer – using a generic search or navigating or searching on the government web(portal) – we observed big variations in navigation techniques used by policy domain (Fig. 3), whereby three of the seven related questions (public transport, GP and mobile telephone) could not be answered by the government portal. Admittedly, the mobile space is in the private sector as is the now privatized British rail system, but the GP task falls firmly in the UK government’s National Health Service (c.f. Fig. 6). Participants using either search strategy successfully answered the other public sector domains (schooling, council tax, town planning and voting).

Significantly, in most cases where Government Web search strategy was viable, people typically were either as accurate as or more accurate than the Generic Search technique (Fig. 4). The town planning was a very notable exception, with about 40% correct using Government Web strategy compared to over 90% using Generic Search. Apart from the planning case, this finding is potentially reassuring for government and the argument that portals are more trustworthy. In terms of timeliness in correctly finding information, only two of the five questions where both types of search modes were used was there any statistically significant difference (Fig. 5), but the results give divergent evaluations of the government portal. For town planning, people using the Government Web strategy were both less likely to find correct answers, and when they did, it took them longer than the Generic Search strategy (3.5 vs 2.3 minutes). In the case of voting, the Government Web strategy performed the same in accuracy, but faster in obtaining correct answers, but the time differences were relatively small (53 vs 69 seconds). In summary, these findings do not provide a particularly strong justification for large investments in building government webportals.

This paper has significant implications for both e-government practitioners and internet researchers, which are now separately canvassed.

6.2 Implications for e-government practitioners

One notable observation from this study has been the quite big differences between times taken to correctly find answer similar types of questions in different policy/service domains (Fig. 5). Hard to find information was in relation to schooling (about 4.5 minutes), GP services (3 minutes) and town planning (3 minutes), whereas council tax (2.5 minutes) and voting (1 minute) were relatively quick. In comparison, the related private sector services of public transport (4 minutes) and mobile phone (1.5 minutes) varied considerably. These findings suggest that government could focus on the improving the online presence of public service areas that are relatively difficult to find accurate information.

To be sure, speed and accuracy in public sector information finding is not just a function of a website design, but of a range of factors that governments need to consider. Differences between policy/service domains and between jurisdictions can generate variations in performance. Thus, the capacity to find quickly and accurately information relates to: the readability and usability of websites where required information is located; the discoverability, visibility and navigability to that website; the complexity or simplicity of the underlying policy and service delivery setting (as complex policy settings are likely to make it harder for administrators to present and users to comprehend); and the level of commitment by a government in trying to make such information or services readily available and accessible online. Indeed, comparative performance differences provide important information about website design, policy and service delivery settings, and political priorities, all of which can be adjusted.

Three further lessons arise from this study. First, when imagining, building and evaluating a web strategy, governments would benefit from adopting a web ecology perspective. Rather than thinking in terms of large webportal-houses or even more modest website-houses as the unit of analysis, government strategy could conceptualize users as operating in a web ecology. This in turn means a focus, less on front doors, and more on landing points to which generic search engines are likely to teleport people. Consequently, government should identify different types of functions that people are likely to want to do – such as change their address or enroll in a local school or GP clinic – and then design a functionally based hut-like web presence that deals with each function.13

¹³
Designing web presence by smaller functional units is also more responsive and flexible to changes in government policy.

Such functional web-huts could be clustered into websites or portals, but these should be the unit of analysis, design and construction. Consequent to this design approach is that each hut is readily discoverable, navigable and strategies deployed to ensure that generic search engines find these web-huts and their internal contents readily. This means that designers think in terms of how search engines apprehend and sort the web, which in turn requires making information machine-readable and standardized. It further implies designers construct webpages in a way that maximizes their importance in the web ecology vis-à-vis google PageRank algorithm and related signals, for example by building hyperlink relationships with other important sites.

Secondly, the web ecology perspective recognizes that what people want to find can be located on a range of webpages, many of which government agencies do not operate. This is a direct outcome of the ease of circulating digital data, but also of political and public management strategies for open data. For example, a decade ago Mayo and Steinberg’s The Power of Information (2007) argued for the need to release public sector data for social, cultural and economic development. It is only natural that such information then appears across the web, as Fig. 6 illustrates. This means that public administrators also have a role to ensure that the public sector information that people find on the web is accurate and not out of date, a hard task given the promiscuous nature of digital data. Ensuring that government web-huts achieve a high level of visibility to generic search engines goes a considerable way to addressing this problem especially since our research suggests greater accuracy on such sites.

Thirdly, our study highlighted important characteristics of the UK government that enhance web navigability. The UK is largely a unitary state with reasonably consistent policy across its expanse, though with some major differences with Scotland, and a major role of local authorities in delivering public services (e.g., schooling). In contrast, federal states such as the USA, Germany and Australia may face greater challenges in web strategy because of jurisdictional differences. Postcodes in UK are particularly beneficial for searching for local public services, as they refer to a relatively small geographical area or population. As such, they operate as a fine-grained geospatial information infrastructure for searching for places in space. Accordingly, postcodes are the centerpiece of search tools for British schooling, GP, town planning and council tax. Many other countries do not have such a fine-tuned geospatial information infrastructure and they could be expected to perform worse in the tasks examined in our study. Moreover, UK government sites made good use of maps of public services, which other government jurisdictions still fail to do. A sensitivity to navigating geospatial aspects of public services is a further task governments could progress.

6.3 Implications for internet researchers

This paper has outlined significant conceptual and methodological innovations for internet researchers, particularly in the social sciences. The idea of a web ecology provides an important conceptual shift in how to apprehend, investigate and analyze the web. As shown earlier in the paper, dominant approaches to internet studies focus on particular websites or platforms. By adopting a web ecology perspective attention is also given to the ways in which people experience the web through websites and platforms, such as jumping from social media platform to a website and then using an app. For this a fundamental change in perspective is required, and we have offered the conceptual beginnings of such a venture.

Secondly, we have reported on a highly innovative digital research tool that allows tracking of web navigation through time. Such information is not readily available, and as such many questions and types of research are not readily able to be examined. By advancing a web tracking tool, the research opportunities of internet researchers have been greatly enhanced. In particular, while the research reported in this paper was obtained through a traditional computer laboratory setting, recent updates to the tool have enabled it to be deployed anywhere a person is located, thereby greatly expanding the opportunities for data collection.

In conclusion, we point to an area of future research. We have proposed a web ecology perspective to stimulate a different approach to thinking about government on the web. This web ecology is part of the entire web universe of a billion-plus websites that generic search engines like Google crawl. This web universe actually has parallel digital universes in social media platforms – such as Facebook and Twitter – that are not readily accessible to Google. These parallel digital universes are increasingly important because it is through these that people increasingly discover information about the world, including government. For example, a government official may Tweet a link to a government service that users see in Twitter. Similarly with Facebook. By clicking on these links, users are delivered through a ‘wormhole’ hyperlink to the web universe of traditional websites. Thinking about digital government strategy beyond the web universe requires new conceptual and methodological approaches to which we must boldly go.

Footnotes

Acknowledgments

We acknowledge the intellectual and technical contributions of Dr Scott Hale and Professor Helen Margetts of the Oxford Internet Institute in helping to design and executing the study, as well as the facilities of the University of Oxford’s IT lab. This work was supported by the Australian Research Council Discovery Program [DP110100446].

Appendix

Appendix – Questions used in online experiment

What is the cost for an anytime dayreturn adult fare from your new local rail station of [Bredbury/Thornton Heath] to [Manchester/London] city centre?

As a local resident, you will need to pay local council tax. Your new property is classified in which local tax band?

You need to organise a primary school for your child. From all primary schools within 1 mile of your residence, you choose the one with the highest OFSTED result. Which one is that?

You need to register with a local GP (General Practitioner) for your family in your new home. You find three in the area and want to select the one with the best patient ratings. Which one is it?

Before moving you wish to make some modifications to your terrace house to make it suitable for you. You decide the best option is to include a single-level, small-sized conservatory. Would you normally need to apply for permission?

You want to discover the level of mobile coverage in your new location. What level of 3G mobile coverage does your provide Virgin provide?

You are a British citizen registered to vote. However, in moving to your new city, you need to register to vote at your new address. Can you find a web page where you can do this?

References

6, P. (1997). Holistic Government. London: Demos.

6, P. Leat,

Setzler,

, & Stoker,

(2002). Towards holistic governance: The new reform agenda. Basingstoke: Palgrave.

Aula,

Khan,

R. M.

, & Guan,

(2010). How does search behavior change as search becomes more difficult? Paper presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.

Australia. (2018a). About australia.gov.au. Retrieved from http://www.australia.gov.au/about.

Australia. (2018b). GOV.AU Performance Dashboard. Retrieved from https://dashboard.gov.au/dashboards/1-mygov.

Booth,

, & Jansen,

B. J.

(2010). A review of methodologies for analyzing websites. In Web technologies: Concepts, methodologies, tools, and applications (pp. 145-166): IGI Global.

Chen,

S. Y.

, & Macredie,

(2010). Web-based interaction: A review of three important human factors. International Journal of Information Management, 30(5), 379-387.

Christensen,

, & Lægreid,

(2007). The whole-of-government approach to public sector reform. Public Administration Review, 67(6), 1059-1066.

Clifton,

(2012). Advanced web metrics with Google Analytics. New York: John Wiley & Sons.

10.

Cross,

(2007, 22 August). Portal Contact. The Guardian. Retrieved from https://www.theguardian.com/society/2007/aug/22/guardiansocietysupplement.egovernment.

11.

Dhyani,

Ng,

W. K.

, & Bhowmick,

(2002). A survey of web metrics. ACM Computing Surveys, 34(4), 469-503.

12.

Dunleavy,

Margetts,

Bartholomeou,

Bastow,

Escher,

Pearce,

, et al., (2007). Government on the Internet: Progress in delivering information and services online: Research Report. Retrieved from http://www.governmentontheweb.org/sites/governmentontheweb.org/files/Government_On_The_Internet_Research-Report.pdf.

13.

Eynon,

, & Dutton,

W. H.

(2007). Barriers to networked governments: Evidence from Europe. Prometheus, 25(3), 225-242.

14.

Gauld,

, & Goldfinch,

(2006). Dangerous enthusiasms: e-Government, computer failure and information systems development . Otago: Univ of Otago Pr.

15.

Henman,

, & Graham,

(under review-a). Towards a taxonomy of government webportals: an international comparison of portal hyperlink networks.

16.

Henman,

, & Graham,

(under review-b). The web ecology of the online state: the case of Australia.

17.

Hood,

C. C.

, & Margetts,

H. Z.

(2007). The tools of government in the digital age. Basingstoke: Palgrave Macmillan.

18.

Jiang,

He,

, & Allan,

(2014). Searching, browsing, and clicking in a search session: changes in user behavior by task and over time. Paper presented at the Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval.

19.

Margetts,

, & Hale,

(n.d.). Report on a user experiment: study of user expectations of a life events approach for designing e-government services. unpublished.

20.

Margetts,

H. Z.

(2011). Experiments for public management research. Public Management Review, 13(2), 189-208.

21.

Markey,

(2007). Twenty-five years of end-user searching, Part 1: Research findings. Journal of the American Society for Information Science Technology, 58(8), 1071-1081.

22.

Mayer-Schönberger,

, & Lazer,

(2007). Governance and information technology: From electronic government to information government. Boston: MIT Press.

23.

Mayo,

, & Steinberg,

(2007). The power of information. London: Office of Public Sector Information.

24.

Moral,

Gonzalez,

, & Plaza,

(2014). Methodologies for monitoring website performance: Assessing the effectiveness of AdWords campaigns on a tourist SME website. Online Information Review, 38(4), 575-588.

25.

Nielson,

(2000). Designing Web Usability. Indianapolis, IN: New Rider.

26.

Pang,

P. C.-I.

Chang,

Verspoor,

, & Pearce,

(2016). Designing Health Websites Based on Users’ Web-Based Information-Seeking Behaviors: A Mixed-Method Observational Study. Journal of Medical Internet Research, 18(6).

27.

Rowling,

J. K.

(2000). Potter, Harry and the Goblet of Fire. London: Bloomsbury.

28.

Shelter,

(2015, 14 October). Announcing our work programme. Retrieved from https://www.dta.gov.au/blog/announcing-our-work-programme.

29.

Singal,

Kohli,

, & Sharma,

A. K.

(2014). Web analytics: State-of-art & literature assessment. Paper presented at the Confluence The Next Generation Information Technology Summit (Confluence), 2014 5th International Conference-.

30.

Van Deursen,

, & Van Dijk,

(2009). Using the Internet: Skill related problems in users’ online behavior. Interacting with Computers, 21(5-6), 393-402.

31.

Wahed,

M. E.-S.

, & El Gohary,

E. M.

(2013). The Future Vision for the Design of E-government in Egypt. International Journal of Computer Science Issues, 10(3), 292.

32.

Wang,

, & Senecal,

(2007). Measuring perceived website usability. Journal of Internet Commerce, 6(4), 97-112.

33.

White,

R. W.

, & Drucker,

S. M.

(2007). Investigating behavioral variability in web search. Paper presented at the Proceedings of the 16th international conference on World Wide Web.

34.

Zhou,

, & Society. (2013). A systematic understanding of successful web searches in information-based tasks. Journal of Educational Technology, 16(1), 321.

Webportal vs google for finding government information on the web: From a website-centric approach to a web ecology perspective

Abstract

Keywords

1. Introduction

1 This limited perspective underpins virtually all web analytical approaches to evaluation of web presence, is used in international rankings of government web presence, and web network analyses.

3.1 The dominant website centric approach

3 The Amazon-owned Alexa company (Alexa.com) provides comparative web statistics for ‘competing’ websites. However, it is unclear what a competing website is for a government portal or department, and Alexa does not appear to provide a service for government sites.

4 A web ecology perspective has similarities with the information governance perspective advocated by Mayer-Schönberger and Lazer (2007).

4.1 Research design

6 Although the portal was established in 2012, at the time of our research in mid-2014, some residual webpages from the previous portal Direct.gov.uk were linked from the new portal for functions yet to be fully integrated.

8 A Firefox plugin that utilises Javascript, PHP, and a MySQL database was built based on previous software developed by Margetts and Hale (n.d.).

6.1 Evaluating the findings

6.2 Implications for e-government practitioners

13 Designing web presence by smaller functional units is also more responsive and flexible to changes in government policy.

Footnotes

Acknowledgments

Appendix

Appendix – Questions used in online experiment

References

¹
This limited perspective underpins virtually all web analytical approaches to evaluation of web presence, is used in international rankings of government web presence, and web network analyses.

³
The Amazon-owned Alexa company (Alexa.com) provides comparative web statistics for ‘competing’ websites. However, it is unclear what a competing website is for a government portal or department, and Alexa does not appear to provide a service for government sites.

⁴
A web ecology perspective has similarities with the information governance perspective advocated by Mayer-Schönberger and Lazer (2007).

⁶
Although the portal was established in 2012, at the time of our research in mid-2014, some residual webpages from the previous portal Direct.gov.uk were linked from the new portal for functions yet to be fully integrated.

⁸
A Firefox plugin that utilises Javascript, PHP, and a MySQL database was built based on previous software developed by Margetts and Hale (n.d.).

¹³
Designing web presence by smaller functional units is also more responsive and flexible to changes in government policy.