Runners,repeaters,strangers and aliens: Operationalising efficient output disclosure control

1. Introduction

In the last two decades, one of the key growth areas in official statistics has been the availability of confidential data for research user by academics, private sector analysts and government departments. On the demand side, users want increasing granularity in the data to address more specific policy issues. On the supply side, government data holders are under pressure to leverage their investment in data collection by maximising data use across a range of stakeholders.

Much of this data is confidential and personal, such as health or tax data. Traditionally, the privacy of respondents was managed by reducing the detail in the data, either to a level at which the data could be distributed without restriction (public use files, or PUFs), or with more detail left in the data but access limited to licensed users (scientific use files, or SUFs).

As data use has grown, so have concerns about whether the confidentiality protection is adequate. The new risks include [1] the re-identification possibilities of social media, the third-party holding of confidential data implied by the growth in administrative data as a source, and massive computing power with the ability to re-identify source data through brute force methods. Anonymization methods which were widespread and deemed adequate some years ago (for example, simple rounding of totals or cell counts of three) no longer meet acceptable standards.

Observed practice suggests five solutions: (1) increased reduction in detail; this risks making the data valueless (2) for SUFs, tighter contracts or licences; this assumes that there is a linear relationship between strict licensing conditions and user behaviour for which there is no strong evidence (3) replacing genuine data with synthetic data; this does not eliminate risk, and users are often uncomfortable about basing analysis on imputed data (4) ‘query servers’ allowing queries on the data without seeing it, and (5) research data centres (RDCs) which allow full access to ‘secure use files’ (SecUFs) in an environment physically and/or digitally overseen by the NSI.

Table query servers, producing simple cross- tabulations and counts, are becoming widespread and effective at meeting many users’ needs for dynamic tabulations. More complex query servers offering a much wider range of analysis are now being developed, such as Statistics Norway’s elegant system at www.microdata.no. Query servers apply confidentiality checks automatically to outputs.

However, for detailed analysis researchers need access to the full microdata via an RDC. The great success story of this century for official statistics has been the use of virtual RDCs (vRDCs), where thin client technology has allowed data holders to provide the security of a physically restricted environment whilst allowing users to access the environment from more convenient locations. Many high-income countries have at least one facility operated by the National Statistics Institute (NSI) or a data archive; in the UK alone there are six general-purpose vRDCs offering the microdata underlying official statistics to a variety of users in government and academia.

RDCs manage confidentiality at the point of access but create a new risk of confidentiality breach through publication [2]. Both SUFs and SecUFs have some identification risk, so it is possible that a published output might reveal some confidential information. This risk is higher for SecUFs as the data is much more detailed. All RDCs therefore operate a system of output-checking before publication (output statistical disclosure control, or OSDC) to manage this risk.

There are two approaches to managing output-checking for conformance to regulation [3]: ‘rules-based’ and ‘principles-based’. The former sets strict rules for releasing output and applies simple yes/no criteria; the latter uses flexible rules-of-thumb and creates an environment for negotiation between researcher and output-checker. Because rules-based is very limiting in research environments, our experience is that most organisations claiming to be rules-based operate a ‘rules-based but sometimes …’ system allowing for ad hoc relaxation of rules. We refer to this as ‘ad hoc’ output checking.

Data holders choosing between OSDC regimes consider two issues: risk, and operational efficiency. Superficially, risk should be irrelevant; all regimes operate the same statistical models for determining whether a specific output poses any risk. In practice, as is demonstrated below, rules-based regimes have a higher statistical risk but are often perceived to have lower risk.

However, our experience is that data holders are usually more focused on operational efficiency: which approach uses resources most effectively? A rules-based system can, in theory, be automated or run by humans with little statistical training. The principles-based solution requires input by humans able to discuss technical matters with researchers. Prima facie, the need for flexibility in principles-based models implies a more costly and laborious solution; this has long been established in the management literature on bespoke production [4]. In reality, the principles-based solution was designed specifically to reduce resource cost while also reducing risk [5], and the available evidence tends to support this.

There are two reasons for the misperception of the principles-based model as high-cost. First, data holders are often unfamiliar with the activities of research users of data, and so view them through the lens of their own outputs; these are typically tabulations which have strict rules applied for comparability across time and alternative breakdowns. Second, data holders’ experience of OSDC is usually limited to the statistical literature, which focuses on arbitrary ‘intruders’ applying mechanical procedures to breach confidentiality (see e.g. [6]). Together, these factors encourage an over-simplistic view of the research environment which drives data-holders’ perceptions of risk and benefits.

The extensive management literature on process efficiency is as robust and canonical as the statistical literature on SDC, but has played little role to date in OSDC thinking. This paper seeks to change this by introducing a model familiar to operations management literature: that of ‘runners-repeaters-strangers-aliens’ (RRSA) [7, 8]. This model can be used to differentiate inputs of demand from customers (in this case, the requests from researchers for data cleared for publication) and uses the different characteristics of those inputs to develop optimal operational responses to increase organisational readiness. Using this framework, we contrast how the rules-based and principles-based approaches address the different challenges posed by real research environments. It is then straightforward to demonstrate how the ‘one-size-fits-all’ rules-based model achieves neither operational efficiency nor effective risk reduction in the RDC environment. Similarly, we can also analyse why the ‘rules-based-but …’ ad hoc approach fails to achieve the operational advantages of the full principles-based approach.

The next section summarises the literature on the topic on output regimes and on process management. In section three we develop the output-checking problem, and in section four we show how the RRSA model can be applied to this procedure. Section five discusses empirical cost assessments. Section six concludes.

While acknowledging that many government departments produce data for re-use by researchers in academia and government, for clarity in this article we assume that the data has been collected and made available by a national statistical institute (NSI).

2. Literature review

Figure 1 shows a typical output-checking process from a secure environment managed by a National Statistical Institute (NSI).

The researcher works in an environment where he or she cannot directly take away statistical results (note: some facilities allow more ‘trusted’ users to check and release their own outputs). The researcher places the outputs to be released in some predefined location in the secure environment and asks the support team to check and release the output. The support team can extract results from the secure environment. If the support team decides the output is non-disclosive, it sends the results out to the researcher’s (open) home environment.

For expository purposes, we will assume that the researcher has asked for a frequency table to be released, and that the support team operates a simple threshold rule of three; that is, the table must have at least three observations underlying each cell in the table. So, Table 1, part (a) passes the SDC rule but part (b) does not.

Under a rules-based approach (RBOSDC), this is a hard limit; no exceptions are allowed. Setting the rule is problematic as the rule is trying to balance both confidentiality and utility. Too high a value prevents the publication of useful but non-disclosive findings; too low a threshold allows many useful results to be published but increases the risk that disclosive results leak out.

Under the principles-based approach (PBOSDC), the rule becomes a ‘rule-of-thumb’: it guides decisions but is not always followed, and can be adjusted up and down as necessary. The researcher can argue [5] that the rule can be ignored if, and only if:

the output is non-disclosive, and

The first condition is obvious. The second condition ensures that the output-checker and researcher only spend time negotiating over an output when the result matters to the researcher. This is appealing to researchers as it puts them in charge of deciding when something is ‘important’, rather than the output checker. Thus, Table (b) above could be released if the researcher demonstrated that the small value was non-disclosive and essential for publication. The third condition ensures that researchers do not abuse the system. Note that the terms “important” and “rare” are not specified – this is an area for the researcher and output-checker to negotiate [5]. As a result, training the researcher to understand the concept is necessary; effectively, this is the model of low-cost accommodation [29].

PBOSDC is two-way: the output-checker can also argue that the rule-of-thumb is inappropriate in a specific case because it does not protect confidentiality. For example, in the above case the output-checker may argue that a higher threshold is needed because the data are particularly sensitive and the patients are easily identified. Some organisations (for example, National Records for Scotland) operate a two-tier system with a lower ‘regular’ threshold and a higher threshold for outputs based on more sensitive data.

PBOSDC systems usually use higher thresholds (10 is common) than RBOSDC. Use of an overly restrictive rule-of-thumb does not limit research as the researchers always have the opportunity to argue for an exception. Hence, the threshold can be set high as it only has to address the confidentiality problem; the utility problem is dealt with by the exception mechanism. The rule-of-thumb threshold is set at a level which provides a high degree of confidence in the non-disclosiveness of results while simultaneously not leading to many requests for exceptions. In practice, genuine research environments can tolerate much higher thresholds under principles-based models.

It is the combination of stricter rules-of-thumb and the ability to use discretion in applying those stringent rules that gives the principles-based approach its superior risk management. Under RBOSDC, a single rule must do two jobs: protect confidentiality (by having a higher threshold, for example), and allow useful, non-disclosive output to be published (which is limited by having a high threshold). Security and efficiency must be traded off. In contrast, under PBOSDC, the rule has one job (protect confidentiality in most cases); efficiency comes through negotiation when it matters.

Rules-based models also fail to provide the imagined guarantees over security. Consider the following tables.

In (c), all cells have at least 5 underlying observations. However, it is clear that an implicit table is being generated: the complement to the proportion with the genetic marker is the proportion without it. Part (c) shows that there are 2 males (10% of the 20 in total), diagnosed with diabetes in the dataset who have the genetic marker.

Part (d) shows the problem of class disclosure. All males in this dataset diagnosed with diabetes have a BMI greater than 25 i.e. they are overweight or obese. It doesn’t matter that there are twenty individuals in this group, well above the threshold; something is now known about all males with diabetes diagnosed.

These pedagogical examples illustrate some limitations in a simple threshold rule; other examples can be developed. It can be argued that the rules-based organisation simply needs to develop rules to cover these cases. However, as [15] demonstrates, developing complex rules to cover special cases quickly becomes unwieldy compared to the principles-based approach of simple rules-of-thumb and flexibility in interpretation. When combining the higher thresholds used in PBOSDC with the need to explicitly define what happens in many states of the world in a rules-based model, it is clear that RBOSDC is the higher-risk option.

If this is the case, why do risk-averse organisations use RBOSDC? Two reasons are invariably given.

First, rules are said to be simpler for everyone to use (researchers and output-checkers) and easier to explain to data-holders who want to be reassured when depositing their data. The latter is a valid point: data holders, if aware of SDC at all, are likely to be familiar only with the traditional model of SDC for tabular data in a hostile environment. Simple rules reflecting that knowledge have immediate appeal, even though the sense of security in the familiar is not warranted.

The second, and more common, reason given is that RBOSDC uses fewer resources: applying simple rules should be easier and require lower-skilled operators than a system which leaves open the possibility of negotiation over any statistical artefact. Principles-based systems cannot be less resource-intensive than rules-based models in the absence of queries and must require more resources if the checking staff must deal with queries. Moreover, those resources involve output-checkers with statistical skills, which are not necessary for the rules-based system.

There is an alternative to RBOSDC and PBOSDC which is widely implemented. Almost no rules-based organisations operate in the simplistic way described above. All have some informal arrangement allowing researchers to argue that outputs which break the rules can be released in certain circumstances. We will refer to this as ‘ad hoc’ output SDC (AHOSDC). This method can provide some of the flexibility/efficiency gains of the full principles-based approach without the potential free-for-all.

At first glance, this approach seems to offer the best of both worlds. In practice, it suffers the key problems of both. First, it does not address the risky nature of rules-based by making no allowance for output checkers ignoring rules to block disclosive outputs. More importantly, not formally acknowledging that rules are flexible can create a lack of clarity, causing uncertainty and inefficiency. It also makes resource allocation harder: should output checkers have statistical skills when the formal policy of the organisation says that they do not need them?

Some organisations argue that the simple threshold rule presented above is a straw man: more complicated rules can achieve both the security and the flexibility of PBOSDC. Unfortunately, this is extremely difficult to do in genuine research environments. Ritchie [15] provides a counter example where a simple, specific, unambiguous, 17-word threshold rule rapidly becomes a woolly 47-word mouthful which requires specialist interpretation, and which is easily challenged by a researcher wanting to make a point.

Fundamentally, the reason why RBOSDC (and AHOSDC) fails to meet the twin targets of efficiency and security is because it is grounded in the SDC literature which sees this as a statistical problem generated by an arbitrary ‘user’ type of individual. In contrast, PBOSDC sees output checking as a process problem, caused by multiple types of client and client needs. To see why this makes such a difference in implementation, we now turn to the management literature.

The process perspective on output checking starts from the recognition that different types of researchers, and types of output, produce different demands on the NSI. As noted in the literature review, the concept of ‘requisite variety’ was established as far back as 1956, and there is a well-established management literature which uses segmentation of customer demand variety as a way of efficiently allocating resources to better achieve operational readiness. Simpler demands are automated as far as possible, leaving specialist resources to be concentrated on the more specialist and typically more complex high-value cases.

We employ the ‘runners, repeaters strangers and aliens’ (RRSA) model. Using the RRSA terminology, we can divide output requests into four separate types, with some indication of how often these occur (based on personal experience).

Like the example made earlier of checking account balances in a bank, the runners are the bread-and-butter of microdata research. They include simple descriptive statistics such as mean of the observations, or frequency counts in categories, which are usually presented with high numbers of observations. The runners also include ‘safe’ (or ‘low review’) statistics, such as regression coefficients, where there is no meaningful disclosure risk [36]. In theory, these outputs could be reviewed automatically using the programs tau-Argus or sdcMicro, for example. In practice, they are manually reviewed as this is faster.

The repeaters are the outputs which require the reviewer to make a judgement based on context. This aligns to the example provided earlier described the case where bank staff evaluates the official documentation for opening an account. At the NSI, a scatter plot of regression residuals might be submitted; the checker would want to evaluate the risk in any outliers. Alternatively, this could be a simple table with counts below the threshold (as in a PBOSDC ‘exception’ request); the checker is then being asked to make a judgment on whether this is non-disclosive, infrequent, and important.

These two cover almost all outputs from research centres. Note that in RBOSDC, only the runners exist: an output cannot be cleared unless a known unambiguous rule exists. For PBOSDC, allowing for repeaters is essential: this flexibility to review outputs in context allows much more restrictive (that is, protective) rules to be placed on runners.

The strangers are where the researcher produces something that the output checking team hasn’t seen before. This aligns to the earlier example where a bank customer produces unrecognised but official documentation to support opening an account. It could be a novel output (for example, being asked to make a decision on a Herfindahl index for the first time), or familiar outputs presented in an unfamiliar way (in one case, a project which required a very large number of intersecting tables). These require multiple skills: a reasonable degree of statistical knowledge, an ability to judge evidence effectively, and the social skills to hold productive discussions with the researcher.

OPEN IN VIEWER

This highly skilled resource is expensive; analytical work is the most efficient utilisation of that resource, and so limiting the time spent by that resource on checking outputs is important for the organisation. Ideally, a stranger only appears once: the role of the reviewer is to decide whether this specific output is to be released, and how future outputs of the same type should be classified. For example, on first encountering a heat map or box-and-whisker plot, the former would be classified as a runner, the latter as a repeater. In the bank analogy, once the unrecognised documentation has been reviewed, this would be expected to generate a rule as to whether similar documents are valid in future.

Finally, the aliens are those outputs for which the facility was not designed; for example, the release of a linked dataset constructed by a researcher. This does not require statistical knowledge at all, but rather understanding of the purpose of the facility. It may lead to a redesign of the facility (say, an isolated section to allow linking to take place).

An amended model of the process is presented in Fig. 2, which reflects the various process flows associated with a triage activity sorting runners, repeaters, and strangers.

As well as managing resources, this structure also allows the NSI to make the researcher an active part of the clearance process; a co-production approach which utilises ‘low cost accommodation’ [29]. PBOSDC training for researchers [5] emphasises three points:

Runners are done quicker than repeaters

The aim is to make the researcher see that his or her behaviour has a direct impact on clearance times, and to provide the researcher with incentives to improve them. By making the researcher a co-producer in the clearance chain in carrying out the preparatory work, the output-checker finds his or her workload reduced. The awareness of researchers that they can directly affect response times builds a feeling of control and hence engagement in the process (see [38] for a detailed review of community-based training, and [5] for delivery). Consider Table 4 below.

Table 4 reveals the impact the RRSA classification has on the need for staff resources to review outputs. Of the eight examples provided, five of the output types classified as runners can be managed with little-to-no staff resource consumption. This has the obvious efficiency impacts for the organisation, while also providing the user a simpler, faster ability to generate the data reports they require. The approach of co-production permits the user to shape their input in such a way as to make them ‘runners’ produced with little organisational friction. However, to fully exploit this, the user also needs to be aware of their role, and hence this sort of discussion is central to PBSDOC training such as [5].

The RRSA model also provides a clear structure for staff resources. Consider the skills needed by an output-checker for the different types of output, and that person’s discretion to ignore the rules of thumb.

This creates a hierarchy of technical skills allowing different staff to be allocated to different roles. It also simplifies skills acquisition and staff training, by providing a clear path to personal development based upon experience and knowledge of the data.

This differentiation of skills is the second reason why apparently lower-cost models of output clearance fail to achieve the operational gains of PBOSDC. For RBOSDC, only runners and strangers should exist: there are fixed rules, which may be added to as new statistical products occur. This implies that the bulk of the work can be carried out by checkers with minimal training in statistics or data.

However, the systems run by most NSIs are ad hoc; that is, notionally the rules are hard but in practice researchers ask for, and get, some flexibility. The flexibility may depend on the data, the statistic, or sometimes whether the researcher is ‘trusted’ or not. The flexibility is important: without it, the NSI is likely to lose the goodwill of the researcher. The difficulty is that, because the flexibility is not officially sanctioned, it is less clear whether a clearance is going to be simple or complex, allowed or blocked. Clearance times become less certain; and because any clearance might be an exception, all clearance staff need to have the ability to handle exceptions. The efficiency gains from having clearly delineated production processes have been lost.

6. Conclusion

Secure research access to the most sensitive microdata has been one of the great success stories for NSIs this century. It came from realising that simply reducing data detail was a dead-end. Instead, novel ways of working with researchers opened a range of options. For all of these new ways of working, output-checking is a key part of the operational system.

Perceptions of output-checking have been dominated by the statistical literature, which is designed to address the safe production of statistical aggregates. The rules-based approach is well-suited to statistical aggregates, but not research outputs. Hence OSDC was born as a field, with PBOSDC its standard-bearer. But to those brought up on the traditional statistics, PBOSDC seemed an operational nightmare: how can an explicitly ‘flexible’ (i.e. uncertain) world, requiring greater statistical understanding and more training for everyone, be both safe and scalable?

When viewed from an operations management perspective, the answer is that the efficiency gains come precisely from the elements that worry traditionalists. Tough, simple rules with flexibility at the margin for high-value outputs means that the 80%–90% of runners can be handled quickly by automated processes or staff with minimal training. Allowing researchers to choose when their runners become repeaters saves the output-checker carrying out this function. This is an example of both Frei’s low-cost accommodation [29] and customer co-production [34]: the NSI has effectively turned the customer into part of the workforce. More importantly, it gives the researcher some control over the process, and so builds engagement. There are upfront training costs, but these should be seen as investment expenditure.

From the management literature, there are no surprises that a one-size-fits-all model allocates resources less efficiently than a segmented-markets model; nor that the latter is better at exploiting customer self-service. This ‘requisite variety’ has been a core of management thinking for over half a century. This operational lens can also shed light on how some very efficient rules-based or ad hoc systems manage their resources so well: substantial upfront investments are supported by instant feedback mechanisms. These ensure that expectations are managed effectively so that customers can self-select into the ‘runners’ category.

What is perhaps less obvious is that this also produces better statistical outcomes: PBOSDC is inherently lower-risk than RBOSDC. In theory, the disclosure risk in an output is independent of the checking process: the criteria for assessing disclosiveness are statistical, not operational. This is only true if resources for checking are unlimited. In practice, the checking process does affect statistical outcomes.

First, rules-based systems have to trade off confidentiality protection and utility in a single rule, but in PBOSDC the rules-of-thumb are there just to manage confidentiality; utility is managed separately by the exception mechanism. Ceteris paribus, a PBOSDC system operates stricter statistical criteria then its rules-based equivalent while providing greater utility.

Second, PBOSDC concentrates output checker resources on important and rare cases, rather than spreading those resources across all outputs. As a side-effect, PBOSDC also reduces dissatisfaction amongst users, a known risk factor for restricted-access systems [39].

This illustrates a wider issue. The traditional focus on statistical measures of risk, without considering the implications of operational choices, has been strongly criticised (e.g. [39]) as risky and inefficient. [40] also suggested that this is doomed to failure in a big data/machine learning world. In contrast, research into operations has much to say about effective risk management, particularly in relation to digital services (such as the ‘data supply chain’ model [41]. A change in emphasis from statistical to operational models of risk drawing on the extensive management literature (as in ONS’ 2019 course for output checkers which uses the SSRA framing) should help NSIs to improve delivery on the joint objectives of security, efficiency, and customer service.