Exploring sensor data for agricultural statistics: The fruit is not hanging as low as we thought

1.1 A very brief history of business data collection

Before the 20th century, data for statistics were mainly gathered via a census approach: every individual within a population was counted. In the second half of the 20th century, sample surveys became the main method to collect data for official statistics. Sampling has proven to result in both unbiased and precise statistics, provided that they are designed and conducted carefully, and error sources as defined in the Total Survey Error framework are minimized. This makes surveys costly both for National Statistical Institute (NSIs) and businesses, who may experience high response burden, especially when they are multi-surveyed [4, 5, 24, 28]. In Europe, since the 1970s secondary data collection was increasingly relied upon for official statistics: registers and administrative data were being used to replace or supplement surveys or for data validation. These registers are maintained by other organisations such as the Tax Office, Chambers of Commerce, or municipalities in support of their own administrative goals. CBS e.g. by law [18] has access to more than 200 registers coming from governmental organisations, serving as input for social and economic statistics. In Finland e.g. agricultural statistics are mainly based on register data, although surveys are still needed to collect additional data [30].

Since a few years now, the potential of a third data source, big data, is being explored, characterised by the four Vs: volume, velocity, variety and veracity [33]. Some examples of this are satellite data, ship tracking data, and traffic loop data [7, 12]. Examples of big data that are relevant in the agricultural sector are remote sensing data like satellite images, GPS data, climate and weather data, real-time data generated by sensors [31, 50, 54]. Statistics Poland [32] and Estonia [53] e.g., studied satellite images and radar data (Sentinel 1 and 2) for identifying arable fields and crop detection in order to estimate crop yields. Unlike surveys, and as with registers, statistical institutes have no control over the data generating process of these secondary type of data [43].

Many countries nowadays have a two-step data collection policy [3, 21, 22]. Only in case secondary sources such as register data and big data do not provide the necessary information, surveys are conducted. The main driver for this policy is cost reduction, both at the NSI side and the business side, i.e. reduction of response burden.

1.2 Smart industry: Sensor data as a data source for official business statistics

Above we mentioned real-time data generated by sensors as an example of big data. In this paper we will focus on these data and discuss its usability as input for business statistics. Businesses are evolving and innovating rapidly [47, 48]. Nowadays, more and more business processes generate a huge amount of data: these (sensor) data are being generated through digitalisation and robotics, and integrated through the Internet of Things. This revolution is referred to as the development of Smart industry, also called Industry 4.0 [26].

In the agricultural industry we see a similar development, referred to as smart farming or precision agriculture (PA) [35]. In contrast with traditional ways of producing crops in arable farming, where actions are taken on field-level, in PA actions are specifically tailored for a certain section within a field. This level of detail requires good insight into the fields (see e.g. [17, 50, 52]), and requires the availability and analysis of many data provided by sensors, occasionally supplemented by manual measurements. Charania and Li [11], Pham and Stack [35], and CEMA [9] provide an overview of agricultural technologies applied in PA, including:

Machines, drones and robots with sensors. These sensors collect a large number data on crops, growing conditions, and farming activities, like soil and crop health (soil moisture and temperature, nutrients, crop diseases), spraying of crop protection chemicals.

An overview of PA data sources that could serve as input for official statistics is given by Arnold et al. [1]: satellite data, sensor-generated data, use of robots, use of drones, FMIS, and apps to be used on the farmer’s smart phone. FMISs are the core registration system and database used by farmers. Based on FMISs farm data on crops, planting, fertilization, and harvesting can be produced. Apart from being a database, an FMIS also enables communication with e.g. sensors on the one side (data input), and data deliveries and reports to end-users like the farmer himself, but also wholesale buyers of the products, banks, and government agencies like the Ministry of Agriculture and statistical institutes (data output). For a discussion on FMIS functionalities in PA we refer to Kaloxylos et al. [29]. FMIS might be a good source for official statistics.

In the discussion on using sensor data for agricultural statistics we also need to take into account the farmers perspective. In the Netherlands sample surveys are still an important collection method for agricultural statistics. However, collecting data via questionnaires is becoming more and more cumbersome which was reflected in low response rates as low as 45% for voluntary agricultural surveys. In 2017 most of these surveys became mandatory by law [20], which increased response rates to about 75% with voluntary surveys staying at the same low response level [44]. At the same time, the attitude of farmers towards completing questionnaires did not change: it is time consuming, and requires a lot of internal data retrieval from their FMIS where data are electronically available [46]. Farmers considered this a waste of time; their typical reactions was: “What is in it for me?”.

In addition to PA data sources offering potential to create more real-time and detailed statistics, and lowering the response burden on farmers [43], these data can be combined with existing administrative and survey data to address relevant current policy questions regarding new production systems and indicators such as sustainable farming and natural capital. This research question was however not part of our study.

2.1 Research questions

In order to study the potentials of PA sensor data for official statistics, a small-scale data use case has been carried out whereby CBS joined forces with an innovative arable farmer and the Eindhoven University of Technology (TU/e). The farmer applies precision farming technologies, and made a selection of his (sensor) data available to CBS.

Two research questions lie at the heart of this case study:

Is this data source fit for purpose with regard to replacing (parts of) agricultural survey questionnaires by these data? And if so, how can these data be collected: can PA data be collected without using questionnaires, or can the data be used to automatically complete (parts of) a questionnaire?

This section summarises the main findings of this case study [36, 37, 38], and will focus on the first research question by investigating the overlap between the data generated by the farmer and the data needs of NSIs. In this sense, the study has a qualitative nature and looks at the level of the different variables in the data of one farmer. It has neither the possibility nor the ambition to make extrapolations to other farmer data and will only make some assumptions on what could be possible for official statistics when more data became available. In Section 2.2, we discuss the data on the supply side, including the data generating process at the farm, the data infrastructure (i.e. the data flow and the systems in place used for data storage), and the quality of these data. In Section 2.3, we discuss the demand side: data that are asked for in agricultural surveys targeted at arable farmers. In Section 2.4, the data selection made available by the arable farmer is analysed with respect to the data asked for in these surveys, i.e. the overlap with the surveys where we will aim to address the first research question.

OPEN IN VIEWER

Figure 1.

The farmer’s data infrastructure.

Thus far we have discussed the farmer’s data (the data supply side), now let us turn to the data demand side: the data users. The two platforms taken together (CRS and DIPPA) contain the data that are needed for reports to be sent to other parties, like the Ministry of Agriculture, wholesale customers, the Tax office, commercials banks, but also to the Netherlands Enterprise Agency (RVO) and CBS. The reports to RVO and CBS include the following four surveys, specific for arable farming:

The Combined Survey (conducted by RVO annually, and sent out in May; data are shared with CBS).

The surveys conducted by CBS are part of the mandate that CBS has to collect data for European agricultural statistics. This mandate is based on European regulations [20] and falls under the European Statistical System (ESS). These statistics are mandatory since 2017 (with exception of the Grassland Usage Survey). The data are collected by means of conducting a sample survey: an electronic questionnaire is sent to the sampled farmers, which they have to complete and submit.

The Combined Survey deals with all kinds of topics such as number of employees, farm area, manure, farm animals, horticulture, floriculture, mushrooms, and subsidies. The Crop Yield Survey deals with the (expected) yields of a number of crops (like wheat, barley, rye, corn, and potatoes), and the sowing of winter crops (e.g. winter wheat, barley, and rye). The Crop Protection Survey asks the farmer of their use of chemical, mechanical and biological measures taken in order to protect their crops. Finally, the Grassland Usage Survey asks about the harvesting of grass on the farm: how much, how it is stored and for what purpose?

2.4 Data overlap between the farmer’s data and surveys

We have seen that our farmer uses two different systems. Therefore, in the analysis with respect to the overlap with the surveys, it was decided to include both. First, with respect to the PA-Specific System (DIPPA [36]): The analysis of the sensor data showed that a lot of the questions asked in the surveys can either be directly answered or indirectly deduced from the data in the DIPPA database. One of the most interesting findings in our study was that in the case of the Crop Yield Survey, the production part of the survey could be retrieved directly from the database. We make therefore the assumption that when sensor data of other crops than potatoes are available, likewise, crop yield surveys do not need to be filled in by the farmer anymore. The more complex Combined Survey, can be partly filled out: surface area and fertilization. The section that deals with employees can potentially be filled out using the database, however key details about the relationships to the farm owner and contract specification are still missing. Figure 2 shows the subsections of all surveys and how well they overlap with the PA-Specific System (DIPPA), where green indicates a strong overlap, yellow meaning partial overlap and red indicating no overlap at all.

For the CRS/FMIS used by our farmer, a similar analysis has been conducted: Here also all variables related to pesticide and fertilizer usage are available, as well as the sowing of (winter) crops. The combined results for both data systems are shown in Fig. 3. Neither of the two systems are useful to gather information about subsidies. The CRS/FMIS houses the more traditional information about a farm and has therefore, unsurprisingly, more in common with the surveys sent out by CBS, whereas, the PA-Specific System has very detailed information about the farm (collected with PA technology).

OPEN IN VIEWER

Figure 3.

Overlap between agricultural surveys and data in the two farmer’s data systems.

3.1 Sensor data: Criteria for their utility

So far, we have discussed the data of one farmer in relation to the data needs of an NSI. Sensorized harvesters collect e.g. geolocated data about crop yields like wet mass and average moisture percentage. These sensor data can be used to complete parts of the Crop Yield Survey Questionnaire (see Fig. 2). We have concluded that there is some overlap between his PA data and data asked for in surveys. This could make these data a good source to replace (parts of) surveys. We also observe that the data quality needs to be improved, both metadata and the data quality. However, going back to our first research question (Is this data source fit for purpose with regard to replacing (parts of) survey questionnaires by these data?), more issues need to be considered in order to assess its fitness for purpose, i.e. the utility of these data. In this section we will discuss PA data and its fitness for purpose.

The use of sensor data also requires a separate and new IT architecture and infrastructure, presenting investments by NSIs. Furthermore, businesses (like farmers) may not be willing to share the data. We could state that sensor data are candidates to enrich or replace survey data when a survey is relatively costly and/or inaccurate, when sensors are relatively cheap and/or accurate, produce timely and accurate statistics, and when businesses are willing to share sensor data. Fitness for purpose criteria are considered from three perspectives: the survey quality-costs, the sensor data collection quality-costs, and the respondent (the business) perspective [15]. Here we have adapted these criteria for social surveys to business data collection processes.

From the survey perspective existing survey topics may be candidates for enrichment or replacement with sensor data, when they satisfy at least one of the following criteria:

Burden: The survey topic(s) are burdensome for a business, in terms of time, cognitive effort, or data retrieval;

Examples that satisfy the first criterion are topics that require a lot of data retrieval from business records, and additional data processing to calculate the requested answers to the questions. Business records often are event based (harvesting or fertilizing fields) whereas questionnaires ask for aggregated numbers on the business level. The second criterion is applicable to many business surveys. Often businesses ask: “why do you need all these details?” And: “What is in it for me?” [46]. The third criterion could apply to complex (economic) concepts, e.g. sustainable agriculture. However, even for seemingly simple concepts (like the number of jobs in a business, or what fields need to be included in the counting) developing a valid measuring instrument can be quite complicated.

From the sensor data collection point of view, the main criteria are:

Omnipresence: The sensor(s) are applied within a business covering the entire business process and by most businesses in the population, i.e. that many farmers would use PA technology and collect these data;

These four criteria all link to the utility of the resulting sensor data. The omnipresence criterion refers to the coverage. This relates to the question how many farmers do actually apply PA technology, and use specific sensors. The data access criterion means that sensor data can be accessed, collected, manipulated, processed, evaluated and interpreted. This does not necessarily mean that the sensor data themselves have to be collected and stored by an NSI, it may as well be that e.g. farmers grant access to their sensor data, and that the requested data are calculated and submitted. The quality criterion originates from the statistical objective to produce accurate (i.e. unbiased and precise) statistics [5], and to be able to compare statistics between persons or companies and in time. Sensor accuracy is the equivalent of survey accuracy, and sensor data should be evaluated on equivalent error sources, as defined in the Total Survey Error Framework [24, 25]. For instance, location data can be used to establish the position of fields but are subject to missing data, measurement errors, sensor data drift and potentially also device effects. The final criterion refers to costs associated with the development and implementation of new data collection processes using sensor data.

From the respondent (farmer) point of view, sensors may vary in their intrusiveness. Four criteria follow:

Willingness: Businesses are willing to consent to provide access to or share the sensor data;

In order to employ sensors, businesses need to be asked for consent to grant access to sensor data, or activate sensors and to store and send data. Even without the technical necessity to ask for consent, there are legal and ethical reasons why consent is imperative. Willingness to consent varies per type of sensor and depends on the context and purpose of the measurements. Sensor data can be collected passively or actively. Sensor data are collected passively when there is no respondent intervention or feedback, apart from consent. This is the case when sensor data are collected directly from a central data hub. In active sensor data collection, respondents are asked to check, revise, accept and/or supplement sensor data, i.e. the respondent (the farmer in our case) is involved in the data collection. Motives for active sensor data collection are increased willingness to provide the data, increased data quality and hybrid forms of survey and sensor data: the data may also be shown to respondents for quality checks and for enrichment of sensor data. Finally, the sensor data may be fed back to businesses in an aggregated form and may provide them with valuable benchmarking information.

Combining our experiences and the results from the data use case and these above criteria, we can now define a number of methodological challenges with regard to the use of sensor data as a source for official statistics. In addition, relevant papers on precision farming have been checked to validate and supplement these challenges.

3.2 Methodological challenges

3.3 Data collection systems, IT infrastructure and costs

With the growing availability of PA sensor data, integrated in FMISs, and having overcome the challenges as discussed above, more modern data collection methods become feasible, like S2S data collection, or Electronic Data Interchange (EDI) (see Buiten et al. [8] for a discussion on EDI). This can be implemented in various ways:

Starting with surveys, we can link electronic questionnaires to the farmer’s database, extract the required data, calculate the requested data and fill in those numbers in the questionnaire. Alternative ways would include the uploading of e.g. csv files in the questionnaire, which have been extracted from the farmers FMIS [46]. The farmer would still have control over his data, as the filled-in/uploaded answers can be checked and finally submitted. This would still be a sample-based primary data collection method.

We will return to these options in the discussion.

Depending on the developments in the agricultural sector, investments in a chosen data collection solution need to be worthwhile. Once all of the above has been dealt with, we also need to include the time dimension: statistics are especially useful when published over time. Therefore, it is of the utmost importance that the data delivery is constant and consistent over time, so time series can be produced.

Assuring the interconnectivity and interoperability between all stakeholders through all steps in an integrated business data chain [8], and depending on the applied data collection method(s), technical standards that are easily and widely accepted, have to be put in place. In the role of stakeholder, for NSIs it’s important to see the potential and establish the statistical output using agricultural sensor data. NSIs can then set the requirements and develop (or stimulate the development of) systems that make it possible to process and filter data on the fly during the data collection process. Data can be treated at farm or field level, reducing drastically both the IT costs associated to the storage and possible big data capacity issues. Both businesses as well as an NSI need to invest in these intelligent systems.

4.1 Conclusions and discussion

The aim of our research was to address two research questions with regards to the use of existing PA data in farm businesses for official statistical purposes:

Are these data source fit for purpose with regard to replacing (parts of) agricultural survey questionnaires by these data? And if so, how can these data be collected: can PA data be collected without using questionnaires, or can the data be used to automatically complete (parts of) a questionnaire?

In this paper, we discussed the first research question by investigating whether sensor-generated PA data could be used to (partly) substitute surveys, with a focus on data collection and data quality issues. This question was examined in the context of precision agriculture and sensor data of one innovative arable farmer was used. The aim of this data use case was broadened as we looked into the data infrastructure at this farm and general FMISs used by farmers in the Netherlands.

It was found that the generated PA data from our farmer are stored in two different platforms: a PA-specific system (called DIPPA) and a CRS (or FMIS). The first system is tailored to the needs of the farmer and houses most of the PA data. An initial exploratory analysis of this platform showed that parts of questionnaires can be completed using these data, while other parts still have to be completed manually. Questions about the area of the farm and the harvest estimate can be filled in to a large extent, but the sections on planting, fertilization and the number of employees can only be partially answered. Financial information requires a completely different data source. It was concluded that parts of the sensor data are matching with some of the concepts and data asked for in the agricultural survey questionnaires.

In addition to the PA-specific data system, we also investigated the farmer’s CRS. This is the FMIS that houses the more traditional information about farm activities. FMISs are used by many farms in the Netherlands. In the system it is possible to import, save, interpret and share data about the fields. These systems can be entered manually as well digitally via farm machines. We found that these systems showed the most overlap with the agricultural survey questionnaires. These include sections about field and crop registration, pesticide use, fertilization and harvesting. Due to the limited number of used systems in the Netherlands there is some standardisation; data harmonisation by EDI-Crop [23] is supported by these systems. At the moment these FMISs may be a better source to investigate than farmer specific data systems. A pre-test study for the redesigned 2020 Crop Protection Survey questionnaire confirmed this conclusion [46].

With regard to the second research question we looked into utility criteria for the use of sensor data and other challenges that need to be taken into account when assessing the fitness for purpose of this data source in the context of official statistics. These included getting access to the data, data ownership, omnipresence and market penetration, IT infrastructure, standardisation, costs, data harmonisation and interoperability. From this small-scale data use case study we concluded that still much has to be improved to use these sensor data. Precision Agriculture data offers potential as input for official agricultural statistics but the fruit is not hanging so low as we thought! The question arises if NSIs should take a more active stance as stakeholders by communicating closely with farmers and other stakeholders (like FMIS developers, PA machine producers offering cloud solutions).

CBS has defined an innovation strategy with a clear focus on making policy-relevant statistics based on new methods and new data sources. Sensor data offer potential. For an NSI the challenges lie in data access to privately owned data, concept validation, data quality, data harmonisation and interoperability. In order for sensor data to be of any use, these issues still need to be addressed in order to use sensor data as a data source for official business statistics. As long as PA technology is not widespread in many parts of the farming sector (except for horticulture), there is still a chance to influence these new data systems and processes. The role of an NSI in these emerging data infrastructure systems could be, through and in close collaboration with software and IT developers, to set the requirements for standardisation and harmonisation, consider new technical options such as edge computing (pushing computation out), assist in the analysis and processing of the data, enrich the data with relevant statistical information based on other data sources, and develop new statistics that address policy and users’ needs.

4.2 Next steps

In the short-term the most promising data sources are sensorized agricultural machinery data and FMIS data. Based on the data use case and the literature review we have identified a number of next steps. The future work consists of three pillars with regard to innovating data collection for agricultural statistics:

Scouting and following of data infrastructure projects within the agricultural sector.

Above we concluded that CBS should not stand aside with regard to new data solutions, but should be actively involved as a stakeholder. We have learned that many data infrastructure projects in the agricultural sector have been initiated. We also mentioned open data sources (like big data) as a source for agricultural statistics [1, 50] but these were out of scope of our data use case study.

Innovating agricultural data collection also entails collection of PA data from individual farmers using APIs. Some of the main manufacturers of precision arable farming machines in the Netherlands (John Deere, CNH Industrial, and Claas) provide access to their clouds through software interfaces (Application Programming Interfaces, APIs [40]). These clouds are promising data sources because they store detailed PA data about field operations, machine data, agronomic service providers activity, soil conditions and environmental conditions, as we have seen. For a proof of concept of this S2S architecture applied to agricultural surveys, we refer to Snijkers, Gómez Pérez and De Broe [45].

Above we have concluded that FMISs are a good source for data that are asked for in official surveys. We will further investigate the applicability of these FMISs for statistical purposes. This includes a more detailed exploration of the FMIS data contents and IT systems, aimed at the development of S2S data communication tools.

The demand for information is obviously wider than the information that surveys are designed for and aim to collect. In order to fully assess the potential of PA data, one would need to investigate whether the current surveys sufficiently address current policy issues and questions with regards to sustainable farming and natural capital. For these phenomena new indicators are required where PA data offer possibilities.