Webportal vs google for finding government information on the web: From a website-centric approach to a web ecology perspective

3.1 The dominant website centric approach

As done in the commercial sector, regular monitoring of government websites typically occurs using ‘web analytics’. Web analytics involve the collection and analysis of web data to understand the usage and effectiveness of a website. Its practice has been in existence since shortly after websites were created, but over the last decade web analytics has become an enormous science and industry, with many companies and tools on offer, such as Google Analytics (Analytics.google.com) and Clicky (Clicky.com).

As outlined by Booth and Jensen (2010) (see also Clifton, 2012; Dhyani et al., 2002; Singal et al., 2014), standard areas of website evaluation are: website usage information (including visitor type, visit length, internal search information); referrers (that is, which website a user came from immediately before arriving at the examined website); content analysis (including the path of webpages users visit within the website, the most visited webpages); and quality assurance (such as broken hyperlinks). For example, Google Analytics (Analytics.google.com) provides real-time graphs and reports about user traffic and dynamics. In this way, website evaluation using Google Analytics is operationalized in terms of maximizing not only traffic volume, but also return rates, visit length, and quality of traffic (Moral et al., 2014, p. 577).

Web evaluation is also undertaken at the webpage or interface design level. Such analysis focuses on the location of items on the screen, where users look, how long it takes users to find relevant information on a webpage, and the usability of a webpage layout and functions (Wang & Senecal, 2007). Accessibility of webpages, such as for people with disabilities or on different devices, is a long-standing approach to evaluating web design (Nielson, 2000). Customer satisfaction with a website using surveys is a further form of assessment.

In each of these methods, the focus on the website and the webpage as a component of that website. These metrics analyze a government’s web presence at the website level. Such metrics potentially provide a means for assessing government web portals, in terms of whether or not people found what they were looking for in a timely fashion, by looking at clickstreams within the webportal, but they can only do so once a user has entered the webportal. These metrics provide little insight into the process by which users come to the government webportal. Our approach addresses this important limitation.

Significantly, given the rationale for a government webportal is that it is “The best place to find government services and information. Simpler, clearer, faster”, then it necessarily requires a comparison with other approaches to finding government services and information, such as through commercial search engines or other websites.3

The web is a vast unordered space of webpages grouped into websites and interconnected by hyperlinks. Webpages (and websites) are hyperlinked to other webpages (and websites) by relevance. Those hyperlinks are an equally important design feature of the web as webpages. A web ecology refers to the hyperlink network of webpages, a network or system of diverse, inter-connected or inter-related webpages (Henman & Graham, under review-b). A web ecology perspective alerts us to the flows through hyperlinks, just as much to the content and functionality of webpages that is dominant in many ranking exercises.4

A web ecology perspective is more akin to how web users typically experience the web, not at the level of websites, but rather as a navigation via hyperlinks through inter-related webpages that seamlessly cross the boundaries of websites. A web ecology perspective is agnostic of websites, as a defined or formalized set of webpages with shared URLs owned and managed by an organization or agency.

The web ecology perspective is compatible with the clickstream analysis of users within a website, which as explained above can be used to evaluate the effectiveness of a government webportal. The implicit assumption is that users begin at the portal homepage and navigate from there. However, web users increasingly use whole-of-web search engines (such as Google), which enable users to arrive at (internal) webpages of a government portal without going through a website’s homepage or portal. Alternatively, users can arrive at a government webportal from other sites. For example, of all the users arriving at the Australian government’s log-in page of the service transaction portal, My.gov.au, in January 2017, only 24% came directly to the site by typing in the URL, whereas 37% came directly from Google, while the remainder came from a variety of websites or search engines (Australia, 2018b).

When understood within a web ecology perspective, evaluating government webportals means analyzing the web experience of people who seek to engage with government online at the web space level, and not solely at the website level. Evaluations need to understand how and where people start the process of finding government related information and services, and where they find it, and if they even make use of the portal.

Evaluation of government webportals within a web ecology, and not at the website level, is crucial for several reasons. One, it enables evaluators to compare the relative search performance of government webportals against other search engines. Two, government webportals may not contain the government information or services that the user is actually seeking to find, but could be hosted on other government (or non-government) websites. This means that an analysis at the website level does not allow satisfaction with the portal to be determined without reference to where the desired data is ultimately found. Three, government information or services that citizens seek to encounter online are now located in a wide variety of locations including outside of government websites. Increasingly the ‘nodality’ or centrality of government (Hood & Margetts, 2007) is challenged by other online providers of public sector relevant information and services. A focus on the webpages that government webportals link a user to does not recognize the alternative places such information may be available. Thus, not only does a web ecology perspective allow an evaluation of a webportal’s comparative search performance, but it also enables an evaluation of the different landing points that people may end up at, thereby enabling a comparison of the accuracy of the information discovered. In summary, evaluators need to consider how citizens experience and approach the web when finding government information, engaging with public services or transacting with government, because the routes to the ‘solutions’ they find are diverse and the solutions themselves may not be on government websites.

There has been very limited research examining the effectiveness of online portals by examining the web users’ experiences of finding online information or services relating to government, Dunleavy et al. (2007) and Margetts and Hale (Margetts & Hale, n.d.) being exceptions. Both studies undertook an experimental design, whereby some research participants were given a clear ‘open search’ screen as their starting point and an intervention group were given one or more portals as their starting point for their task. In a study of 69 subjects, Dunleavy et al found that those using open search were faster in finding answers to questions posed (2007, pp. 55–56) than those mandated to use the UK government webportal (Direct.gov.uk), but no different in their accuracy.5

4.1 Research design

The effectiveness of the UK government’s webportal, Gov.uk,6

In order to assess the time and accuracy of finding government service related information and services on the web, hypothetical relocation scenario was posed where participants were asked to imagine moving residential locations: “You are moving to Manchester/London for work and you want to find as much as you can about the city and prepare for your move as possible before you leave by using online sources”. Participants were randomly allocated a Manchester a London scenario (and given a specific address for their new residence), in order to test any differences in results resulting from different local authorities. Participants were then given a range of multi-choice questions relating to seven service domains: public transport; land tax; schooling; GP health services; local planning; private telephone services; and political representation; and open government (see Appendix).7

To collect data, customized a browser plugin8

Participants were recruited in June 2014 via an email invitation to people listed in an Oxford Internet Institute database of past research participants who had indicated willingness to be contacted about future internet related research. The invitations indicated that the research would be conducted in one of four two-hour sessions across two weekdays in an Oxford University owned computer lab. The invitation also indicated that a small honorarium would be paid for participation. In total, 80 adult volunteers were recruited. Just over three-quarters of participants (77.5%) were UK residents. Gender was fairly evenly distributed: 50% male; 44% female; and 6% did not specify a gender. The majority of participants were aged either 20–29 (26%) or 30–39 (29%), with a reasonable representation in other age groups. The majority of participants were employed (40%) or students (29%). A further 15% of participants were unemployed, with 7% retired, 4% housewife/husband, and 5% specifying ‘other/NA’.9

Our quasi-experimental research design involves comparing the performance of the two groups – control and intervention/treatment – based on the assumption that the activities of the intervention group who were given the government webportal homepage as the starting point to answer their questions is different to that used by the control group. Implicit is that the former would use the webportal and the latter would use whatever means they determined. This assumption was not supported by the research participants’ behavior. Rather, the group a participant was placed did not guarantee a particular mode of navigation to answer the given question. Participants in the open search group could readily begin by typing www.gov.uk into their browser and proceeding from there, which effectively makes them part of the portal group. Similarly, participants in the portal group could type www.google.co.uk or a search phrase into their browser, in which case they are effectively in the open search group. Moreover, participants could begin searching in one way (portal or open search), become frustrated and give up, and try an alternative approach. In short, in a web ecology the starting point does not determine the approach one takes to finding data. The ease with which a person can change modes of navigating the web undermined the quasi-experimental design.

In recognizing these realities, the evaluation of the webportal needed to be based on the nature of the search people undertook, not on the group they were allocated. An analysis of people’s information seeking behaviors in answering the multi-choice questions was conducted by referring to the URLs people visited. Six different modes for answering questions were identified:

•

Search-Only is the strategy of using a generic search engine (such as Google, Yahoo or Bing) to obtain an answer. We regard the situation when people move from Google search results to a webpage and then navigate from there to other webpages as ‘search only’ as the search was the primary way they obtained their result, and used localized navigation to pinpoint the answer.

Importantly, the search mode used does not reflect the webpage location on which respondents finally use to answer the question. Furthermore, people allocated to the portal group can only be allocated the Gov-Search, Gov-Only, Navigate or Nil categories, whereas those in the open search group can only be categorized as using the Search-Only, Search-Gov, Navigate or Nil categories. Accordingly, Gov-Search includes both people who started searching through the portal then gave up and used a generic search, and those that ignore the portal and immediately proceed to a generic search.

OPEN IN VIEWER

Figure 2 illustrates the proportion of six navigation types people used for each of the seven questions. It is notable that there is no consistent pattern between different domain questions. Significantly, navigation (or responding based on one’s own knowledge or guess) was rarely used, except for the town planning question (where 22 percent adopted this mode). Search-Only is the most common, used by half of the participants in three of the seven questions (Transport, GP & Telephone). Gov-Search is also widely used, especially in those same three questions. This suggests that these three questions were very difficult to find an answer through searching within government websites, most commonly the portal.

The ‘switchers’, those that start in the portal or open search, but end up swapping to a generic search or government web mode respectively. The switcher group ranges from a quarter to 43% of respondents, thereby demonstrating that group allocation in the experiment did not strongly shape online search behavior.

Understanding the processes of switching from one information seeking approach to another can be of interest (Aula et al., 2010; Jiang et al., 2014), as is comparing the relative time and accuracy of the six different approaches. For the purposes of this paper, we simplify the evaluation of the effectiveness of the UK webportal by focusing on the technique that a user ultimately used to obtain their answer. Consequently, we grouped Search-Only and Gov-Search together as both groups ultimately achieved their answer through a generic search (we denote Generic Search). Similarly, groups Search-Gov and Gov-Only are grouped as they ultimately used the government portal (or occasionally a search on a different government website) to answer the question (this group we denote Government Web).

6.1 Evaluating the findings

This paper started by observing considerable effort in building government webportals as a key plank in government online strategy, yet government webportals are often not evaluated in terms of their objectives to assist people to find government-related information, services and democratic processes quickly and accurately. To progress such evaluations, we argued that rather than a focus on people’s experience of navigation within a government webportal, a perspective that locates government’s webportal within the wider web is required as this better reflects how individuals encounter government on the web. This perspective recognizes that people do not experience websites qua websites, but as a series of web pages (within a website) within a wide network of webpages joined by hyperlinks, often navigated through generic search engines, such as Google.

As an analogy, websites can be thought of as houses, where each website-house is carefully designed for people to enter through the front door, and be directed from the front hall to specific rooms in the house in order to find what they are after, before rummaging through the room to find it. Governments have built variously sized houses as webportals to help people find things in government’s real estate portfolio. In the case of the UK, the strategy has been to build a massive mega-building to replace most of their previously owned houses.

Unfortunately, generic search engines disrupt the house building model of websites. Rather than entering in through the front door, search engines provide hyperlink entry to houses through windows, water mains and air conditioning vents, and often teleport or ‘apparate’ (Rowling, 2000, p. 7) people directly to the specific item within a room they seek. Moreover, the web ecology perspective highlights that what people wish to find on the web, can often be found in many different website-houses (see Fig. 6). With the dominance of search engines, governments cannot even guarantee that people will turn up in their webportal-house at all, or even in any of their website-housing portfolio.

Given these circumstances, public administration and government web strategists must importantly consider whether webportals are effective in terms of cost to government and efficiencies for users. To answer this question, we applied new digital research methods that involve tracking the way people find information about English government policies and services on the web. We initially used a quasi-experimental design whereby half of the participants were given a clear screen to go and find the information using their own resources, whereas the other half were given the UK government’s webportal. However, participants’ starting point did not determine the way they ultimately sought to address their task. By looking at how they eventually found their answer – using a generic search or navigating or searching on the government web(portal) – we observed big variations in navigation techniques used by policy domain (Fig. 3), whereby three of the seven related questions (public transport, GP and mobile telephone) could not be answered by the government portal. Admittedly, the mobile space is in the private sector as is the now privatized British rail system, but the GP task falls firmly in the UK government’s National Health Service (c.f. Fig. 6). Participants using either search strategy successfully answered the other public sector domains (schooling, council tax, town planning and voting).

Significantly, in most cases where Government Web search strategy was viable, people typically were either as accurate as or more accurate than the Generic Search technique (Fig. 4). The town planning was a very notable exception, with about 40% correct using Government Web strategy compared to over 90% using Generic Search. Apart from the planning case, this finding is potentially reassuring for government and the argument that portals are more trustworthy. In terms of timeliness in correctly finding information, only two of the five questions where both types of search modes were used was there any statistically significant difference (Fig. 5), but the results give divergent evaluations of the government portal. For town planning, people using the Government Web strategy were both less likely to find correct answers, and when they did, it took them longer than the Generic Search strategy (3.5 vs 2.3 minutes). In the case of voting, the Government Web strategy performed the same in accuracy, but faster in obtaining correct answers, but the time differences were relatively small (53 vs 69 seconds). In summary, these findings do not provide a particularly strong justification for large investments in building government webportals.

This paper has significant implications for both e-government practitioners and internet researchers, which are now separately canvassed.

6.2 Implications for e-government practitioners

One notable observation from this study has been the quite big differences between times taken to correctly find answer similar types of questions in different policy/service domains (Fig. 5). Hard to find information was in relation to schooling (about 4.5 minutes), GP services (3 minutes) and town planning (3 minutes), whereas council tax (2.5 minutes) and voting (1 minute) were relatively quick. In comparison, the related private sector services of public transport (4 minutes) and mobile phone (1.5 minutes) varied considerably. These findings suggest that government could focus on the improving the online presence of public service areas that are relatively difficult to find accurate information.

To be sure, speed and accuracy in public sector information finding is not just a function of a website design, but of a range of factors that governments need to consider. Differences between policy/service domains and between jurisdictions can generate variations in performance. Thus, the capacity to find quickly and accurately information relates to: the readability and usability of websites where required information is located; the discoverability, visibility and navigability to that website; the complexity or simplicity of the underlying policy and service delivery setting (as complex policy settings are likely to make it harder for administrators to present and users to comprehend); and the level of commitment by a government in trying to make such information or services readily available and accessible online. Indeed, comparative performance differences provide important information about website design, policy and service delivery settings, and political priorities, all of which can be adjusted.

Three further lessons arise from this study. First, when imagining, building and evaluating a web strategy, governments would benefit from adopting a web ecology perspective. Rather than thinking in terms of large webportal-houses or even more modest website-houses as the unit of analysis, government strategy could conceptualize users as operating in a web ecology. This in turn means a focus, less on front doors, and more on landing points to which generic search engines are likely to teleport people. Consequently, government should identify different types of functions that people are likely to want to do – such as change their address or enroll in a local school or GP clinic – and then design a functionally based hut-like web presence that deals with each function.13

Secondly, the web ecology perspective recognizes that what people want to find can be located on a range of webpages, many of which government agencies do not operate. This is a direct outcome of the ease of circulating digital data, but also of political and public management strategies for open data. For example, a decade ago Mayo and Steinberg’s The Power of Information (2007) argued for the need to release public sector data for social, cultural and economic development. It is only natural that such information then appears across the web, as Fig. 6 illustrates. This means that public administrators also have a role to ensure that the public sector information that people find on the web is accurate and not out of date, a hard task given the promiscuous nature of digital data. Ensuring that government web-huts achieve a high level of visibility to generic search engines goes a considerable way to addressing this problem especially since our research suggests greater accuracy on such sites.

Thirdly, our study highlighted important characteristics of the UK government that enhance web navigability. The UK is largely a unitary state with reasonably consistent policy across its expanse, though with some major differences with Scotland, and a major role of local authorities in delivering public services (e.g., schooling). In contrast, federal states such as the USA, Germany and Australia may face greater challenges in web strategy because of jurisdictional differences. Postcodes in UK are particularly beneficial for searching for local public services, as they refer to a relatively small geographical area or population. As such, they operate as a fine-grained geospatial information infrastructure for searching for places in space. Accordingly, postcodes are the centerpiece of search tools for British schooling, GP, town planning and council tax. Many other countries do not have such a fine-tuned geospatial information infrastructure and they could be expected to perform worse in the tasks examined in our study. Moreover, UK government sites made good use of maps of public services, which other government jurisdictions still fail to do. A sensitivity to navigating geospatial aspects of public services is a further task governments could progress.

This paper has outlined significant conceptual and methodological innovations for internet researchers, particularly in the social sciences. The idea of a web ecology provides an important conceptual shift in how to apprehend, investigate and analyze the web. As shown earlier in the paper, dominant approaches to internet studies focus on particular websites or platforms. By adopting a web ecology perspective attention is also given to the ways in which people experience the web through websites and platforms, such as jumping from social media platform to a website and then using an app. For this a fundamental change in perspective is required, and we have offered the conceptual beginnings of such a venture.

Secondly, we have reported on a highly innovative digital research tool that allows tracking of web navigation through time. Such information is not readily available, and as such many questions and types of research are not readily able to be examined. By advancing a web tracking tool, the research opportunities of internet researchers have been greatly enhanced. In particular, while the research reported in this paper was obtained through a traditional computer laboratory setting, recent updates to the tool have enabled it to be deployed anywhere a person is located, thereby greatly expanding the opportunities for data collection.

In conclusion, we point to an area of future research. We have proposed a web ecology perspective to stimulate a different approach to thinking about government on the web. This web ecology is part of the entire web universe of a billion-plus websites that generic search engines like Google crawl. This web universe actually has parallel digital universes in social media platforms – such as Facebook and Twitter – that are not readily accessible to Google. These parallel digital universes are increasingly important because it is through these that people increasingly discover information about the world, including government. For example, a government official may Tweet a link to a government service that users see in Twitter. Similarly with Facebook. By clicking on these links, users are delivered through a ‘wormhole’ hyperlink to the web universe of traditional websites. Thinking about digital government strategy beyond the web universe requires new conceptual and methodological approaches to which we must boldly go.