Statistical Journal of the IAOS

This paper analyses the future prospects of statistics as a profession and how data science will change it. Indeed, according to Hadley Wickham, Chief Scientist at Rstudio, “a data scientist is a useful statistician”, establishing a strong connection between data science and applied statistics.

In this direction, the aim is to look to the future by proposing a structural approach to future scenarios. Some possible definitions of data science are then discussed, considering the relationship with statistics as a scientific discipline. The focus then turns to an assessment of the skills required by the labor market for data scientists and the specific characteristics of this profession. Finally, the phases of a data science project are considered, outlining how these can be exploited by a statistician.

To improve the analysis of respondent comments from the Canadian Census of Population, data scientists at Statistics Canada compared and evaluated traditional machine learning, deep learning and transformer-based techniques. Cross-lingual Language Model-Robustly Optimized Bidirectional Encoder Representations from Transformers (XLM-R), a cross-lingual language model, fine-tuned on census respondent comments yield the best result of 89.91% F1 score overall despite language and class imbalances. Following the evaluation, the fine-tuned model was implemented successfully to objectively categorize comments from the 2021 Census of Population, with high accuracy. As a result, feedback from respondents was directed to the appropriate subject matter analysts, for them to analyze post-collection.

There is an ever-present demand for statistical agencies to improve the timeliness, granularity and cost-efficiency of their official statistics. Our methodology for small area estimation using time-to-event data addresses these demands, as it utilises existing data sources to produce timely estimates at finer levels of geography. We illustrate this methodology with our application to the Australian Building Activity Survey, which has been successfully repurposed to obtain small area estimates of newly completed dwellings with associated uncertainty estimates. The methodology is widely applicable, and we discuss further subject areas where it can be introduced to improve value for users of official statistics.

An ever-increasing deluge of big data is becoming available to national statistical offices globally, but it is well documented that statistics produced by big data alone often suffer from selection bias and are not usually representative of the population at large. In this paper, we construct a new design-based estimator of the median by integrating big data and survey data. Our estimator is asymptotically unbiased and has a smaller variance than a median estimator produced using survey data alone.

Identity and ownership are two conceptual pillars used to define Indigenous enterprise. Approaches that use administrative data offer the opportunity to identify Indigenous-owned enterprises without the burden of a survey. It remains unclear, however, if Indigenous-owned enterprises are also likely to self-identify as Indigenous. Thus, in this paper we examine if self-identification as an Indigenous business in Aotearoa New Zealand is driven by Māori ownership. We link information from businesses that had the opportunity to self-identify as Māori in an annual survey with administrative data from Stats NZ’s Integrated Data Infrastructure to calculate their proportion of Māori ownership. Then, we fit models of varying complexity using a Bayesian multilevel approach to predict the probability of self-identification as a Māori business as a function of businesses’ demographic variables and proportion of Indigenous ownership. Using model comparison and out-of-sample predictions we show that Māori ownership is a weak predictor of self-identification as a Māori business. We also show how the probability of self-identification as an Indigenous enterprise changes between regions, sectors, and industries to illustrate the benefits of a quantitative approach to target businesses likely to self-identify as Māori. Predicting the extent to which enterprise owners might choose to self-identify as a Māori business is critical to identifying a robust population of Indigenous businesses and to have better estimates of the Indigenous economy.

In Hong Kong, merchandise trade statistics are compiled based on the commodity information given on the trade declarations submitted by traders. Due to the complexity of the standardised commodity classification system (i.e. Hong Kong Harmonized System, or HKHS in short), there are often reporting errors, especially in the commodity codes and quantities. With around 20 million declarations received annually, the availability of this big data source motivates us to adopt deep learning techniques to detect the reporting errors. This paper proposes a mechanism consisting of three deep learning models for checking the commodity code, quantity and value, which offers an end-to-end solution to data quality assurance for declarations. The results show that the proposed mechanism could enhance the accuracy of error detection, which is conducive to improving the quality of trade statistics. With the use of text analytics techniques, the mechanism could fully utilise free-text commodity descriptions declared by traders to check the accuracy of the declared information comprehensively. It also overcomes some limitations of the traditional rule-based models. The whole study demonstrates the potential of using deep learning approach in quality assurance of existing statistical systems for official statistics.

Nigeria was the first to announce the discovery of COVID-19 cases in Sub-Saharan Africa on the 27th of February 2020 and ever since then, the rate of spread has been rapid. The effects of Socio-Economic Indicators such as the percentage of the population below and above 65 years, Unemployment rate, HIV/AIDS prevalence rate, Population density, Literacy rate and Mortality rate on the incidence rate of COVID-19 in Nigeria cannot be overemphasized. The research thus used the Spatial Modelling techniques to examine the variations in COVID-19 incidence rate in the affected states in Nigeria occasioned by the above-listed socioeconomic factors, by applying four (4) different models vis-à-vis Ordinary Least Squared (OLS), Spatial Lag Model (SLM), Spatial Error Model (SEM) and Geographically weighted regression (GWR) to cover its scope and also performed spatial diagnostic analysis to ascertain the model goodness of fit. Based on the findings, GWR outperformed other models as it explained about 97.6% of the variability among the variables in the model, has the highest Log likelihood (16.873) which shows the goodness of fit and the lowest AIC ( - 4.192). More so, the results were able to estimate a consistent prediction and spatial variability of the incidence rate of COVID-19 in Nigerian states which could help drive policymaking.

The paper deals with the concept and the definitions of hard-to-reach groups and the ways of capturing them in administrative sources, providing a detailed discussion of the meaning of hard-to-reach in the context of administrative sources and in relation to the traditional hard-to-count groups in censuses and surveys. The review of country practices shows that hard-to-reach populations in administrative data can be interpreted in different ways and that their definition is dependent on countries’ circumstances, though there are two main reasons for identifying a group as hard-to-reach in administrative sources. One of the interpretations is selecting some groups, typically considered difficult to reach with traditional survey methods (such as homeless, illegal immigrants or indigenous people) and then trying to capture them in registers to overcome the challenges of traditional field collection or to get more complete information. At first glance, administrative data might offer the potential to improve frame coverage for some target populations, but may also lead to other hard-to-reach or “hidden” populations for different population groups. Indeed, another interpretation refers to the incompleteness of registers or linked administrative databases, which makes some groups, such as children or elders, hard-to-reach and hence describe with data, due to time lag in reporting of some events or to other accuracy problems with the source itself. The paper summarizes the experience of national statistical offices in accessing hard-to-reach groups and describes problems and challenges in capturing them. It also proposes further possible work to improve access to hard-to-reach groups using administrative data.

In this paper, we focus on respondent-driven sampling (RDS), which is a valuable survey methodology to estimate the size and the characteristics of hidden or hard-to-measure population groups. The RDS methodology makes it possible to gather information on these populations by exploiting the relationships between their components. However, RDS suffers from the lack of an estimation methodology that is sufficiently robust to accommodate the varying conditions under which it is applied. In this paper, we address the estimation problem of the RDS methodology and, by approaching it as a particular indirect sampling technique, we propose three unbiased estimation methods as possible solutions.

Like many countries, Ireland has been researching new systems of population estimates compiled using administrative data. Ireland does not have a Central Population Register from which the estimates can be compiled.

The primary step in compiling population estimates from administrative data is to first build a Statistical Population Dataset (SPD). Ideally an SPD will have one record for each person in the population containing the relevant attributes. The ideal SPD then allows compilation of statistics by simply counting over records.

In practice, the compilation of SPDs is prone to error. These errors can be classified into 4 types of error; overcoverage, undercoverage, domain misclassification and linkage error.

Ireland, to date, has investigated 2 different approaches to the compilation of population estimates from administrative data. The first, labeled in this paper as the simple count method, is based on building an SPD which minimises the overall number of individual record errors such that simple counts from the SPD will provide population estimates. The second, labeled in this paper as the estimation method, is based on building an SPD which aims to eliminate all error types bar that of undercoverage and then adjusts counts for undercoverage using Dual System Estimation (DSE) methods to obtain population estimates.

This paper explores the advantages and disadvantages of both methods before considering how they could be integrated to eliminate the disadvantages.

Many NSIs will be considering similar challenges when compiling annual Census like population estimates and this paper aims to contribute to that discussion.

The use of registers has been increasingly popular in the field of population census because of its advantages over the traditional census. While the traditional census requires a large amount of fieldwork and data collection, the registered-based census can rely on pre-existing administrative data. As a result, the register-based census can save both time and budget. Thailand explored a use of the register-based census in 2020. In this paper, the authors layout the methodology in an aspect of data preparation and integration as well as analyze data quality of the register-based census compared with the traditional census in Chachoengsao province, Thailand. In addition, we compared conceptual frameworks that are commonly used for a register-based census in several countries and the number of databases (a recent single database VS multiple databases) used to construct the register-based census. We found that using a conceptual framework that counts the number of populations based on the main census variables on a single recent database is better than using a framework that counts population who appears on many registers in term of overcoverage and data distribution regarding to sex. This provides the evidence of using one recent and complete database is sufficient for conducting the register-based census. The authors end up with recommendations for conducing the register-based census.

Household financial distress is a complicated problem. Several social problems have been identified as potential risk factors. Conversely, financial distress has also been identified as a risk factor for some of those social problems. Graphical models can be used to better understand the co-dependencies between these problems. In this approach, problem variables are network nodes and the relations between them are represented by weighted edges. Linked administrative data on social service usage by 6 , 848 households from neighbourhoods with a high proportion of social housing were used to estimate a pairwise Markov random field with binary variables. The main challenges in graph estimation from data are ( a ) determining which nodes are directly connected by edges and ( b ) assigning weights to those edges. The eLasso method used in psychological networks addresses both these challenges. In the resulting graph financial distress occupies a central position that connects to both youth related problems as well as adult social problems. The graph approach contributes to a better theoretical understanding of financial distress and it offers valuable insights to social policy makers.

German official statistics publish statistics on personal insolvency. These statistics have been recently enhanced using web scraping to extract additional information from a public website on which the insolvency announcements are published. The currently scraped data is used for quality assurance and to derive an early indicator of personal insolvency. This paper provides novel methodological analyses for the same administrative database and presents further opportunities to improve the current official statistics regarding detail and timeliness using web scraping and text mining. These newly derived statistics inform on several aspects regarding personal insolvency’s demographic and spatial distribution.

Official price statistics in the Philippines are mainly sourced from the conduct of regular surveys and censuses which entail high costs. As businesses move into digital platforms, alternatives to these traditional data sources have become more available; one of which is web scraping, a process of collecting information from the web. As digital and online platforms become increasingly utilized for commerce, web scraping offers a way to increase the frequency of data collection while reducing its cost compared to price surveys. This paper provides a survey of experiences of various government statistical agencies in their conduct of web scraping for the Consumer Price Index (CPI). Moreover, it details the Philippines’ experience using web scraped data to estimate the food and alcoholic beverages CPI of the National Capital Region in the Philippines, and that is compared to the official CPI estimate of the Philippine Statistics Authority. Finally, this paper discusses the challenges encountered and the recommendations for enhancing the approach.

In this paper, we formulate the problem of estimating the resident population, i.e. correcting for over-counts in administrative register data, as a binary classification problem. We propose a solution based on machine learning algorithms. The selection and the optimisation of the best algorithm is shown to depend on the goal of prediction. We illustrate this method for two important cases of official statistics, Census resident population and survey design with minimum non-response. The performance of the algorithms, the uncertainty of estimates and of the evaluation metrics are described in detail and implemented in shared, open source code. We exemplify with the results obtained by applying this method to Icelandic register and survey data.

Recent years have seen increased interest in the use of alternative data sources in the definition and production of official statistics and indicators for the UN Sustainable Development Goals. In this paper, we consider the application of data science to the production of official statistics, illustrating our perspective through the use of poverty targeting as an application. We show that machine learning can play a central role in the generation of official statistics, combining a variety of types of data (survey, administrative and alternative). We focus on the problem of poverty targeting using the Proxy Means Test in Indonesia, comparing a number of existing statistical and machine learning methods, then introducing new approaches in the spirit of small area estimation that utilize area-level features and data augmentation at the subdistrict level to develop more refined models at the district level, evaluating the methods on three districts in Indonesia on the problem of estimating 2020 per capita household expenditure using data from 2016–2019. The best performing method, XGBoost, is able to reduce inclusion/exclusion errors on the problem of identifying the poorest 40% of the population in comparison to the commonly used Ridge Regression method by between 4.5% and 13.9% in the districts studied.

Seasonal adjustment (SA) is a crucial factor in the process of producing official macroeconomic statistics. The most important SA methods, X-13Arima-Seats and Tramo-Seats, are currently included into JDemetra+, a universal open-source environment, which is available on several platforms and operating systems, as a result of adoption of Java programming language for source codes, and Xml metalanguage for the definition of input specifications. This paper focuses on the potentials of RJDemetra, the R library developed for JDemetra+ suite. Its structure and functionalities will be illustrated with several examples, reporting the associated R scripts. In addition, a new operational practices will be suggested, exposing an alternative procedure to enhance interactive time-series updating in SA revision policies step, and also to ensure consistency checking in input system, in order to improve and to speed up the SA estimation process, providing greater security and efficiency. Finally, the interaction between two very different environments such as SAS-IML, and R will be displayed through a new SAS-R procedure available for estimating Quarterly Accounts SA series.

Many statistical classifications exist in a statistical ecosystem, where they are interlinked with other classifications. When statistics on the same topic are compiled using different classifications, they need to be transformed in order to become comparable by means of a correspondence table – but sometimes, no correspondence table between the two classifications involved exists. This paper presents the newly developed ‘correspondenceTables’ R package, available on CRAN, which automates much of the ‘mechanical’ work required for developing a correspondence table (thus allowing statistical classification experts to focus on tasks with higher value added). Moreover, the paper presents lessons learned along the way, including unforeseen quality issues with the input data (that required considerable efforts to be successfully tackled), and outlines areas for future improvement.