Predicting macroeconomic indicators from online activity data: A review

The current Google Search public repository for obtaining historical data based on Internet searches is Google Trends,2

GT provides updated reports on the relative demand for keywords, search terms or categories via a default normalised index, the Google Index (GI) [15]. The GI corresponds to the inquiries on Google Search given a geographical area and period, represented on a scale from 0 to 100, where 0 stands for the lack of interest in the period and 100 denotes maximum popularity [16, 17, 18]. In addition, the GI may assume the label “ ” for periods representing minimum popularity [19].

GI displays varied sampling frequencies according to the length of days of the requested series. The hourly sampling frequency is exhibited for searches concerning the latest data, i.e., inquiries covering up to seven days of recent data [20]. Irrespective of the data recency, the GI is daily for search periods comprehending up to 270 days, weekly for searches ranging from 270 to 1825 days, and monthly for more than 1825 days [21].

GT does not detail the GI construction [22, 20]. However, the idea of GI in weekly sampling frequency is as follows [15, 23]. Let a keyword V in a geographical area r and a week w i in the period [ 0 , t ] . The weekly average of the number of daily queries, S ω i ⁢ r , is given by Eq. (1), where V d , r represents the volume of daily searches for a keyword in a given week for a geographical area, and T d , r , the total volume of daily searches of the location for that week. The GI results from the normalisation of S ω i ⁢ r over the weeks in the whole period of interest [ 0 , t ] , as in Eq. (2). This formalisation hints that distinct time lengths produce different GIs for a keyword in a geographical area.

S ω i ⁢ r = 1 7 ⁢ ∑ d ∈ ω i V d , r T d , r(1) 𝐺𝐼 = 100 max ω i ∈ [ 0 , t ] ⁢ S ω i ⁢ r ⁢ S ω i ⁢ r(2)

Gathering GT data requires the selection of keyword(s), search term(s) or categories as proxies for the phenomenon under study, as well as local and cultural aspects to mimic the rational thought of a search engine user [12].

GT provides data directly from its website,5

While Facebook, Pinterest, and Twitter are widely used social media by Internet users (Appendix A), the selected literature primarily utilises Twitter data to predict macroeconomic indicators. No research among the selection discusses explicitly the use of Facebook or Pinterest data in the intended context.

Twitter enables users to express themselves through various forms of content, including text-based tweets[24]. These tweets can consist of text within the 280-character limit, as well as images or videos, associated or not with hashtags.8

Querying Twitter as a data source requires searching for specific terms, hashtags, or public profiles to result in a collection of tweets and retweets. The search process allows for filtering based on time restrictions, language, geographical location or specific accounts. However, data availability depends on individual users’ privacy settings and willingness to share geographical positioning and hashtags. It is important to note that Twitter data has a high sampling frequency design [25], as tweets are posted continuously, with new content generated every second.

The primary data from Twitter are unstructured and, thus, not ready to use [25], requiring a certain degree of preprocessing before its use [26]. The data preparation aims to transform the data into a proper structure to explore patterns, speeches, symbols and pictures, thus building relevant indicators related to the phenomenon under investigation.

Collecting public data from Twitter requires developer credentials to interact with APIs,9

Mobile devices have become ubiquitous in daily life. Smartphones, wearable devices and GPS sensors have played a significant role in the widespread adoption of mobile technology. However, there is scarce literature to scrutinise the potential of this data source to predict macroeconomic indicators.

The literature reviewed encompasses two distinct types of mobile-type-generated data. One strand uses telecommunications companies’ proprietary data [5], which, due to regulatory limitations and the General Data Protection Law, is a bureaucratic source that requires non-disclosure, confidentiality and procedures agreements. The other mobile-type-generated data strand considers the combination of spatial and temporal comprehensiveness of massive location logs from GPS smartphone applications [27] not publicly available.

Gathering data from online activity sources relies on targeting data mainly related to the expectation of the macroeconomic indicator. The keyword-driven strategy is the most widely used approach to collecting data, which requires researchers to carefully select keywords that reflect relevant aspects of the predicted macroeconomic indicator. Other methods include geo-located image tweets [26] and Twitter accounts of significant influence and relevance in economics [25].

Data from social media are primarily textual, which demands researchers transform the data into quantitative information for forecasting purposes either by volume aggregation of messages [26] or by textual analysis. A crucial step is cleaning and filtering data to eliminate irrelevant messages e.g. [4, 26], uncommon abbreviations, misspellings and slang terms [25]. Text mining techniques, such as part-of-speech tagging [24] and sentiment analysis [24, 25], are essential to extract practical information. Contrastingly, data from search engines are generally ready to use without requiring preprocessing.

Based on a keyword-driven strategy, publications draw their analysis based on single e.g. [28] or multiple keywords e.g. [29]. However, utilising a single keyword to forecast can be precarious, mainly due to the relationship breakdown with the target prediction, which could result in a divergence from the desired outcome [16]. Accordingly, there are challenges associated with selecting appropriate search terms [30], and the choice of target keyword(s) varies in the literature according to the forecasted macroeconomic indicator, as follows.

Under labour market research, the commonplace is to look for terms such as job(s) e.g. [31, 32], job offer(s) e.g. [33, 23] and unemployment e.g. [4, 47]. When choosing search terms, it is essential to consider both relevance and regionality. Therefore, conducting a practical analysis of keywords in the appropriate context is reasonable, as shown in the following examples. In investigating British unemployment, [16] looks for redundancy and its derivatives because it refers to the dismissal of a job when employers need to reduce their workforce in the UK. If a worker is made redundant, the individual might be eligible for some rights, including redundancy payments. Within the German market, [35] distinguish groups of search terms to proxy (i) the unemployment flows (using the keywords unemployment office, unemployment agency and names of the most popular online job search engines in Germany), (ii) the highly skilled workers’ reaction to possible layoffs and company restructuring (querying personnel consultant) and (iii) the natural term associated with the labour market (unemployment rate).

Selecting search terms to forecast the GDP is not trivial. As this indicator is a grouped measure representing the value of final goods and services generated within a country, electing keywords is an extensive exercise. Consequently, the use of GT categories to forecast figures related to the GDP is beneficial e.g. [36, 37, 38]. For example, [30] explore GT data to forecast the German GDP by counting on the demand of seventeen pre-chosen Google categories (e.g., Autos & Vehicles, Real Estate, Business & Industrial) and their subcategories, resulting in 200 GT series. To deal with numerous such variables, the authors resort to variable selection methods and dimensionality reduction, such as principal component analysis (PCA) and least absolute shrinkage and selection operator (LASSO). Alternatively to the use of categories, the keywords recession e.g. [28], foreclosure help and layoff e.g. [39] are also employed.

When investigating the consumer price index, typically considered an official measure of inflation, the keyword inflation is commonly used e.g. [40, 41, 42]. However, [43] explore alternative keywords such as price and cpi, and to capture perspectives on price dynamics, search terms like increase price or price dropping. Another study examines the dynamics of the consumer price index across 13 economies employing the keywords cpi and inflation in English and translated into country languages [44]. Furthermore, [42] not only incorporates traditional keywords but also complements the analysis with additional terms, for example, referring to the labour market (unemployment), the exchange rate (exchange rate), public expenditure (government expenditure) and interest rates (bonds yield).

The selected publications investigating private consumption rely primarily on numerous Google consump- tion-related categories, such as Food and Drink, Computers and Electronics, Entertainment and Vehicle Shopping, narrowed down into a few variables using the unweighted least-squares method [45] or PCA [46]. Additionally, data are considered for the keywords recession and layoff targeted in the news GT category [46].

3.2.4 Limitations

The leading general concern common to data sources is the sample bias that may arise when the data is not a random representative sample of the population e.g. [5, 23, 26]. In fact, estimates report that 33.7% of the world’s population does not have access to the Internet [1] and that the proportion of online data produced worldwide is unbalanced towards more developed economies. The amount of online activity data diverges according to the world region and is also affected by exogenous factors such as age and income [47]. While some global areas retain vast proportions of Internet users, others are below the world average [1], as seen in Appendix C. These observations further corroborate the digital divide, highlighting disparities in Internet access and usage patterns across social-demographic profiles e.g. [48, 49].

Other general concerns include the non-guaranteed public data continuity, which relates to the data dissemination suspension without previous notice [7], and the noise in data, which refers to the difference between data collected and factual purposes e.g. [47, 34], since the gathered data may not represent the true researcher’s intentions.

Specific concerns surrounding GT arise from the fact that the data source only provides data starting in 2004 e.g. [50, 51], making it challenging to conduct historical analyses before this period and the need for more comprehensive information about the data. For instance, the data provider omits absolute search volumes e.g. [15, 52] and fails to report details on classification and aggregation methods into categories and the cut-off point for not displaying unpopular searches [37]. Additionally, GT data rely on samples e.g. [48, 6], leading to sampling noise that causes slight variations in indexes collected on different dates due to the sampling process when collecting data [12]. To mitigate this issue, researchers average the indexes from various collecting days to reduce variations e.g. [47, 53, 54, 20]. Furthermore, GT data have limitations regarding series extension in high-frequency sampling [55], so most studies focus on low-frequency series to cover longer time spans [22]. [22] and [21] propose different methodologies based on multi-data gathering and overlapping windows to address this limitation.

Limitations regarding Twitter data encompass the following aspects: Firstly, the data’s unstructured nature demands analysis preprocessing [26]. Secondly, there is evidence that Twitter APIs do not provide uniform activity samples, as usually assumed [4]. Moreover, the data quality enhancement depends on filtering and selecting user accounts [4]. In addition, data are subjected to biases related to opinion [25], language and demographics [26].

Concerns about mobile-generated data stem from the difficulties of their widespread availability. The inherent passivity of data generation presents researchers with constraints in defining the data to be collected, determining the timing of collection and ensuring proper preprocessing by the data collector [5]. This complexity poses challenges in maintaining data quality and integrity [5].

3.3.1 Assessing online activity data as predictors

Most selected publications focus on establishing online activities as resourceful predictors of macroeconomic trends and indicate usefulness in macroeconomic forecasting. To this end, publications compare predictions generated from benchmarks exempted from online activity data to counterparts incorporating web-based data, aiming to demonstrate the added predictive power of online activity data.

The benchmark predictions derive mainly from the univariate ARMA family of models e.g. [37, 56], augmented or not by either other macroeconomic indicators e.g. [57, 38] or survey-based data e.g. [45, 46], or univariate non-linear time series models [15]. Choosing keywords to build predictors is essential in constructing counterpart predictions relying on online activity data. The logical-related keyword(s) to the macroeconomic indicator presented in Subsubsection 3.2.3 are frequent in counterpart predictions, which results in general forecast accuracy gains compared over benchmarks e.g. [58, 59, 60].

Publications seldom report unsuccessful results regarding the predictive relationship between the macroeconomic indicator and web-based data. A contrasting example assumes the Spanish unemployment prediction, including the research of [53] and [61], in which both consider the same keyword (unemployment) and prediction methods (ARMA family of models) but different periods of analysis. While the prediction accuracy of [53] using the online activity data does not overcome the benchmark, [61] find that web-based data improved forecast performance compared to the benchmark. These results indicate that different periods under analysis lead to different outcomes, as remarked by [32]. Another example concerns the CPI prediction for five countries (the USA, the UK, Canada, Germany and Japan), where [51] consider structural time series predictions as a benchmark and forecasts from an extension of the Bayesian structural time series model as a counterpart, using online activity data from 60 Google categories. The counterpart predictions fail to outperform the benchmark. This publication suggests that the possible causes of the failure might be the weak connections between search activity and the CPI, the non-optimal choice of Google categories and other factors not captured by Google search data that exert a more substantial influence on the CPI.

Nowcasting and forecasting are both essential techniques in macroeconomic indicator predictions using online activity data. The real-time information from web-based activities favours nowcasting applications [59] with productive outputs e.g. [62, 16, 27], which can timely benefit policymakers in terms of decision-making e.g. [58, 18] once time lags in publishing official macroeconomic statistics are frequent e.g. [53, 58]. Regarding forecasting applications, analyses of the prediction behaviour on distinct horizons corroborate the forecasting ability of online activity data, as shown in the following examples, which forecast US unemployment up to 12 steps ahead. While [34] concludes that predictions built are functional only for short-term predictions, [15] document the useful predictive power of web-based data in both short-term and long-term forecasting. Additionally, [15] note that the online activity-based prediction performance improves with the forecast horizon compared to the benchmark.

Regardless of the framework adopted, macroeconomic indicator or keyword strategy embraced, researchers conclude that web-based activity data generally provide relevant relationships with economic measures for forecasting intentions. Such a conclusion is valid for a range of research approaches, such as carrying out horseraces among modelling toolboxes [63, 40, 44], investigating the macroeconomic indicator association with search-terms usage in preceding periods [35, 39, 42] or duelling for models considering the inclusion or not of online activity data [45, 53, 15].

3.3.2 Targeted predictions for fragments of sociodemographic profiles

Although selected papers focus primarily on exploring the prediction ability at national levels, researchers also seek to demonstrate the predictive power of online activity data for regional levels, age, race and sex as follows.

Studies on regional forecasting capability point to favourable results and demonstrate the potential of using online activity data to improve territorial economic forecasts’ accuracy and provide guidance for policy-making e.g. [64, 25, 18]. However, the publications recognise the need for further research to fully understand the applicability of online data across different regions and contexts. Performing regional forecasting, [57] target forecasting youth (15-to-24 years old) unemployment. This publication confirms the improved prediction accuracy for the 17 Spanish regions using web activity data against benchmark predictions using the inflation rate as a predictor; however, against random walk predictions, one of the regions fails to be more accurate.

Youth unemployment is a critical indicator in some countries, such as Spain [57], Italy [23], and France [31], and research forecasting such an indicator based on online activities reports superior prediction performance over benchmarks. Youth unemployment prediction may reduce the sample bias limitation presented in Subsubsection 3.2.4 since young individuals tend to use the Internet more than adults [53] and seek jobs online [23]. In addition to the youth unemployment emphasis, [31] perform predictions for two more age groups (25-to-49 and 50+ years old), both with enhanced prediction power over benchmarks, evidencing the French population ageing as influential to the forecast performance and reporting that increases in age tend to decrease the accuracy’s improvement.

The sample bias issue is present in [49] and [48], both exploring unequal access to the Internet, characterised as the digital divide. In focusing on American youth unemployment, [48] delves into the racial digital divide and demonstrates prominent variations in the predictive power of Internet activity data among the races. Data predict youth unemployment more accurately for White Americans than for Hispanic Americans, while predictions for African Americans are unsatisfactory. Comparing the Spanish unemployment forecast accuracy of web-activity data for different age and gender groups, [49] conclude that accurate predictive power is biased towards groups with better Internet access, i.e., digitally included. The publication concludes that the digital divide effect relates to the youngest unemployed males and produces enhanced prediction performance for females compared to male unemployment, which is robust across different age groups.

Exploring the trade-off of labour-leisure, [17] assesses the Canadian unemployment of individuals aged between 25 and 44. The study favours the predictive performance of online activity data incorporating free online leisure activities keywords against predictions relying only on job search-related keywords.

3.3.3 Predictions under disruptive events

A fraction of selected papers focus on the degree of effectiveness of forecasts based on online activity data to predict economic indicators under disruptive events, such as mass layoffs [5], the 2008 financial crisis e.g. [12, 39] and the COVID-19 pandemic e.g. [55, 6]. These disturbances profoundly impact national and regional economies, resulting in widespread job losses, business closures, economic shocks and short- and long-term effects on individuals, families and communities.

Under a mass layoff in undisclosed European locations, [5] report significant regional forecast accuracy improvement in unemployment changes using online activity data compared to not using such data. Additionally, the research identifies employment disturbances at individual and community levels and reveals that job loss consistently attenuates workers’ behaviour regarding mobility and social interactions.

In 2008, the Great Recession impacted economies worldwide. Publications report that incorporating web-activity data into forecasting models leads to proper macroeconomic predictions in such an unstable period e.g. [12, 55], including the sharp increase of the recession [20, 39]. In addition, this troubled period is known for official statistics needing to have anticipated the magnitude of the economic challenge from the beginning [64]; thus, reinforcing the argument of using online databases for prediction purposes has important implications for policymakers [20].

The impact of the 2020 COVID-19 outbreak on macroeconomic indicators using online activity data also emerges in publications. On the one hand, researchers demonstrate that predictions based on online activity data are more timely and accurate than benchmark forecasts, particularly during the recent pandemic [55, 20]. On the other hand, there is evidence that online activity-based predictions perform poorly compared to forecasts that account for indicators of the timing of state-of-emergency declarations [6].

3.3.4 An insight into error prediction metrics

Two widely used error prediction metrics to qualify predictions are the mean absolute percentage error (MAPE) and the root mean squared error (RMSE). The MAPE measures the average percentage difference between actual and predicted values e.g. [31, 33]. On the other hand, the RMSE calculates the square root of the average squared differences between the actual and predicted values e.g. [43, 40]. The lower the value of these metrics, the better the prediction performance.

The comparison of error metrics measures the predictive accuracy of forecast sets, over which researchers frequently employ statistical tests that account for uncertainty to evaluate and compare the predictive performance of different forecast sets. These tests determine whether the differences in performance between the predictions are statistically significant or due to chance. Examples of such tests include those proposed by [65] e.g. [42, 30], [66] e.g. [45, 62] and [67] e.g. [53].