Implementing FAIR data for people and machines: Impacts and implications - results of a research data community workshop

4.1. National Institutes of Health

Huerta opened his remarks by clarifying the distinction between data science and open science. Data science focuses on using tools, whereas open science is about paradigms. Data science tools are typically developed for specific purposes within certain fields. For open science to take hold in each of these fields, scientists must adopt practices that will not necessarily help them achieve their specific, immediate tasks, but that will be helpful in the long term. This has led many agencies to put in place open data initiatives. Unfortunately, these tend to remain disconnected from initivatives in other agencies. A more promising approach is to encourage all agencies and all scientific groups to adhere to the FAIR principles. This is the primary goal of the NIH’s and NLM’s strategic plans for open science.

NIH is a large research organization with over 5,000,000 users that distributes 115 Tb of health data each year. NLM, the largest biomedical research library in the world, is responsible for housing and distributing these data. The key objective of the NIH Strategic Plan for Data Science is to promote FAIR data sharing for all NIH research [28]. This entails providing FAIR-enabled, open access to all datasets supporting publications funded by NIH. NIH encourages all researchers to place data in domain-specific repositories as a first choice. Other options include uploading up to 2 GB of data along with a published paper, uploading up to 20 GB in an NIH repository, or uploading petabytes of data to the cloud using STRIDES (Science and Technology Research Infrastructure for Discovery, Experimentation, and Sustainability Initiative). STRIDES was developed in partnership with Google Cloud and Amazon Web Services [30]. As a subset of the NIH, the three goals of the NLM Strategic Plan are (1) connect and enhance resources through computational curation and critical valuation of resources, (2) optimize the user experience by enhancing outreach and engagement, and (3) offer data science and open science training to employees, collaborating organizations, and users [38]. The pillars of NIH’s strategic plan focus on data stewardship, workforce development, data management tools, modernizing data platforms, and improving infrastructure by storing data in the cloud.

4.2. National Science Foundation

NSF has been changing its approach to scientific support and funding. Beth Plale serves as an advisor at NSF and works to provide public access to NSF-funded research, to encourage good data management practices, and to ensure that research data is shared in machine-readable formats by working closely with GO FAIR. One of NSF’s boldest initiatives has been the NSF Public Access Repository (NSF-PAR), which makes all published work and data that were developed using NSF funding available for free [40]. NSF-PAR is a repository centralized at NSF that is constructed using the DOE PAGES template [8]. All award recipients must deposit the author manuscript of a publication as part of the reporting process [39]. This has recently been extended to include reports of workshops funded by NSF. In 2019, NSF produced a letter outlining its support for open science principles as well as guidelines on using globally unique persistent identifiers for research data and machine-readable data management plans (DMPs) [41]. NSF is working in coordination with other agencies based on a recent effort to encourage open science for all research funded with public money [25]. The Open Science by Design consensus study outlines the current state of open science, the vision for the future of open science, and how to transition from one to the other quickly. NSF is also coordinating with GO FAIR to improve the metadata within its Big Data Innovation Hubs [43,44].

4.3. National Institute of Standards and Technology

Robert Hanisch (NIST) has been working within the Office of Data and Informatics (ODI) to bring about cultural changes related to data sharing [33]. Much like NSF and NIH, NIST requires that papers derived from NIST-funded research be accompanied by data that is compliant with FAIR principles [32]. NIST offers four areas of active support to researchers through ODI. These include (1) curating and making available standard reference data (SRD) for exploratory research (49 free SRD databases) [37], (2) providing research data support through training and tool development, (3) providing data science infrastructure that includes cloud services and research domain specific expertise, and (4) engaging with other data related communities and organizations like the Research Data Alliance (RDA), CODATA, and the National Data Service (NDS). Along with these organizations, NIST is supporting the institution of the FAIR principles to maximize the return on federal research investment and address reproducibility issues within science. ODI has been a strong leader in the effort to improve research reproducibility and it continues to be one of the primary drivers of their work [17].

Hanisch spoke about some specific tools that NIST supports, such as the NIST Materials Data Repository, which is designed to help communities work across organizational boundaries on data that is not yet ready for publication [34]. The NIST Data Discovery Platform is a searchable interface for all data and resources provided through NIST [35]. One goal of ODI is to encourage the use of Laboratory Information Management Systems (LIMS) so that FAIR data and metadata are captured at the moment of observation or, ideally, directly from the instruments themselves. NIST is also developing a Research Data Framework (RDaF) that will be used as an overall guide to the actors and stakeholders involved in creating or using research data [36].

4.4. Department of Energy

DOE is at the forefront of the movement to use machine-learning to increase the rate of advancement of scientific research [46]. DOE has recently built the Summit supercomputer at Oak Ridge National Lab (ORNL), currently the most powerful supercomputer in the world [12,50]. Laura Biven emphasized the important part that DOE plays in encouraging FAIR data practices. DOE is integrated within the scientific community because of its high-performance computing resources and 17 national labs, and it is one of the primary funders of research in the United States, which means that DOE can greatly influence the data sharing practices of scientists. This bodes well for the scientific community because DOE has been a leader in providing public access to research through its DOE PAGES portal, available since 2014 [8,9]. Currently, the DOE Office of Science funds approximately 40% of the scientific research done in the United States [10]. The DOE Office of Science has recently been organizing discussions on how to advance and support machine-learning within science. The Office of Science wants to develop a cartography (i.e., a map of related data) based on relationships among data within their repositories, to see what gaps remain for the implementation of machine-learning processes. Biven stated that DOE is uniquely positioned to lead the use of machine-learning in science as its mission demands paying attention to and adapting new technologies for multiple purposes [11,13].

4.5. General discussion on federal perspectives

The many efforts of NIH, NSF, NIST, and DOE inspired an interesting discussion among the speakers and members of the audience. One of the first issues mentioned by Huerta was that agency efforts often progress on their own without outreach to help align efforts with other institutions. There were some concerns expressed about this issue by the audience, but it was clear that each of the panel members was committed to being open about its practices and to using FAIR data practices, which should make their research data efforts trasparent and usable across institutions. Another important discussion related to updating old observational data within these institutional repositories according to the FAIR principles. The general consensus was that this might be done for some important datasets, but that most funding is directed towards new data rather than old. Plale mentioned that even if older scientific data could not be made FAIR, the metadata records themselves could be greatly improved.

The idea of rewarding scientists for sharing research openly was also raised for discussion among the panelists. Hanisch pointed out that the return on investment is already there in fields such as astronomy, in which archivists look at and annotate highly detailed images of stars and planets to generate research findings. Plale said that NSF funds many small grants and creates community resources that encourage FAIR data practices. One point mentioned by an audience member was that librarians at research institutions have been promoting FAIR data practices and trying to change publishing incentives at universities for many years. It was suggested that some of their efforts and ideas should be taken into account by federal agencies. This connected to the next discussion on how funding is distributed to encourage data stewardship. Each of the panelists stated that their agencies were beginning to hire and distribute funds to information specialists such as librarians. Hanisch stated that NIST already works very closely with librarians, but Huerta, Plale, and Biven felt that the cohort of information specialists at national laboratories is still small and more investment is needed.

5.1. An architectural approach to FAIR digital objects

Manepalli’s presentation focused on network architecture as an engine for implementing FAIR DOs. Manepalli summarized the end goal of FAIR as allowing computers to produce results from DOs at a rate faster than humans without much human input. This issue has been solved in some areas. For example, mobile weather apps use automated radar to predict weather, send the data to a server, and then automatically send the data to app users’ smartphones. There is very little interaction with humans in this process. Areas in which machine-learning is still not a tractable solution include email servers. Emails contain ambiguous headings, expressions, different languages, and inside jokes, which makes them almost impossible to control automatically. How can ambiguous environments such as this be automated? The approach at CNRI is to switch from a system-centric platform to an information-centric network based on DOs. This would allow humans to interact with DOs as needed while still having most processes automated.

The three pillars of the information-centric approach are (1) assigning unique resolvable identifiers to DOs that are free of semantic ambiguities and are long-lasting, (2) using DOs with identifiers that allow easy consumer interaction and interpretation, and (3) providing an interface protocol that helps DO operations remain consistent even upon manipulation by consumers. Manepalli used the analogy of the success of the internet protocol suite for distributing DOs across computers, which has allowed for both automation and human interaction to coexist on the same network successfully. This approach would allow for machine-readable DOs to be distributed as TCP/IP packets across networks and interpreted in personalized human readable formats at various gateways, minimizing the need for complex metadata within the packets themselves.

5.2. Data sharing perceptions of scientists

The Data Observation Network for Earth (DataONE) began in 2009 at the University of New Mexico, and was one of the first data nets, federated networks of repositories, funded by NSF [23]. DataONE focused on earth, ecology, life, and environmental data. Koskela was the Executive Director of DataONE from its inception and encouraged research that sheds light on the attitudes and perceptions of scientists towards data sharing. There have been three large surveys of scientists carried out by researchers at DataONE [47–49]. The results of these surveys indicate that most scientists (∼75%) are satisfied with their short-term data storage practices, while only ∼50% are satisfied with the long-term practices in data storage. However, the majority of researchers (68%) do not follow any established community practice when storing their data. Most scientists neither receive sufficient training in data management, metadata assistance, or citation practices, nor seek out the help of librarians or data management experts to help with their data. Researchers tend to contact only their immediate colleagues on data matters. Scientists do not tend to be concerned about sharing data unless they can be guaranteed credit for their work, and many do not feel that they can trust data gathered by others to conduct their own research. On a positive note, funding for data storage has improved and more scientists have become more aware of locations where they can store their research data.

Technical barriers and issues of trust have led Koskela to endorse the Core Trust Seal repository certification, which seeks to improve trust in shared research data by guaranteeing a high standard of data quality and openness [6]. Koskela emphasized that joining the Core Trust Seal cohort of certified repositories would build stakeholder confidence, raise awareness about digital preservation, ensure transparency, and provide recognition of trustworthiness for all those involved. Koskela also serves as the Executive Director of EarthCube, an NSF-funded geoscience repository, which was the first repository to be certified by Core Trust Seal [14]. Koskela’s goals for the future are to gather more information on perceptions of data sharing, support training initiatives for research data management, and provide opportunities for quality data repositories to become certified.

5.3. Research infrastructure for natural science collections

The Distributed System of Scientific Collections (DiSSCo) repository is a collection of information about existing biological and geological specimens that are widely distributed across museums, universities, botanical gardens, and other institutions throughout Europe [22]. This repository unifies scientific assets under common curation and makes the data more FAIR. Currently, the repository contains data on more than 1.5 billion specimens from 119 collaborating institutions, 5,000 scientists, and 21 countries. DiSSCo was funded in 2019 for an additional three years and 31 new collaborators. Lannom works with DiSSCo to make scientific data in Europe align with the FAIR principles and views the move to FAIR data practices as being vital to the future of science. Currently, there are too many societal problems that need scientific solutions and too much quality data going to waste due to lack of access. Lannom noted that, although FAIR is widely accepted in Europe, there are still many practical issues remaining. Researchers need to have more tools available to easily make their data FAIR. Making data FAIR will broaden research possibilities for future generations. Problems that now seem intractable could easily be addressed using machine-learning later if data openness is given a higher priority now.

5.4. General discussion on case studies

An audience member led with a provocative assertion by stating that attempts to automate scientific processes with machine learning have not had much success in the past and that more success has been gained from making the jobs of researchers easier (e.g., by developing simple data management tools that enable point-and-click metadata creation and data organization) and by creating federal initiatives that require data management and sharing to obtain research funding. Given this history, the questioner asked if technology has changed to the point where automation is more feasible or whether scientists should expect the same difficulties that have plagued previous machine-learning research approaches. Manepalli responded that it is difficult to tell. Human beings have certain flaws with respect to data management and consistency that computers do not, so we should be prepared to switch to automation when it is possible to do so. The way that humans can continue to contribute in meaningful ways is by adding context to data that is compatible with multiple environments. In the short term, this will be more important than constructing DOs that are usable by machines. Lannom stated that advances in computing have been dramatic and that although we may not be ready now, the switch to automated research will come quickly and we should be prepared.

6.1. Core criteria of FAIR principles

Both Bonino and Russell emphasized similar concepts in their discussion on how to apply the FAIR principles to research data in a practical way. Both focused on how the FAIR principles need to be flexible so that different communities can use them in ways that suit their specific needs. Some key components, such as improving metadata and creating unique identifiers for research data, should be applicable in all fields, but each scientific community must be able to decide for itself which standards they require for their published data and which standards work best for their research. Russell emphasized that the FAIR principles cannot be too strict and that assessment should reflect the various needs of different scientific fields. Compelling all scientists to use a single set of FAIR metrics would be a disservice to scientists that work in fields where some data security or restrictions are needed, or where data is much more difficult to organize. Russell proposes that the core stakeholders within many research communities be brought together to discuss assessment methods for the FAIR principles. This will help determine which communities have the infrastructure to support FAIRness and which principles are easiest to implement. For example, interoperability is one of the more difficult principles to implement, as it requires designing metadata and code that can work on many different platforms. In some fields there are too many platforms in use for this to be possible in the near future. Both Bonino and Russell emphasized that the key to addressing these issues is to develop metrics that do not harshly compare researchers or research communities to each other, and that are based on input from members of each research community.

6.2. The state of FAIR in neuroscience

Martone is a prominent member of the neuroscience data community, which has many unique data needs that cannot be easily captured by a broad-stroke FAIR data initiative. Neuroscientific research deals with diverse datasets that include highly detailed, annotated images from instruments, as well as Brain Atlases, which are complex 3D models of brains. At this stage, none of these Brain Atlases have been designed with the FAIR principles in mind [20]. Martone stated that the FAIR principles make sense as concepts that could apply to a wide range of scientific pursuits because they are ideas to be strived for rather than items that can be achieved in full. The Brain Research through Advancing Innovative Neurotechnologies (BRAIN) initiative is funded by NIH and is meant to increase the rate of understanding of the human brain through technological innovation [29]. The BRAIN initiative seeks to incorporate the FAIR principles into its research methodology, but the standards for doing so are not yet well defined [31]. Martone mentioned the Neuroinformatics Coordinating Facility (INCF), which is a community platform and standards organization based in Stockholm, has been supporting the development of neuroscientific standards with some success. The successes of INCF stem from the guidance/training materials and tools that it provides on the platform, a set of consistent criteria that support FAIR principles, a focus on incorporating community input, and its ability to interface with the broader health community [19].

To improve the extent to which neuroscience embraces the FAIR principles, Martone suggested that data experts spend some time training people that are not directly involved in research but still produce important data (e.g., technicians and nurses) and find funding mechanisms to support this training. Martone also mentioned that neuroscientists often make use of distributed data repositories like dkNET, a search portal funded by NIH’s National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK), which should allow for the easy implementation of FAIR practices and advertisement of best data practices and training opportunities [27]. In closing, Martone stated that assessment and metrics can backfire if put into force before they are vetted. For now, FAIR-minded people should focus on understanding how the FAIR principles can be applied within the resource constraints of the various fields.

6.3. General discussion on FAIR metrics

The audience and speakers discussed which organizations or communities should be responsible for defining good data practices, issues with interoperability in science, and the dangers of using metrics. In Martone’s view, decisions regarding FAIR data practices need to be made at multiple levels. Broad suggestions can be made by institutions, but these decisions cannot be so inflexible as to limit domain-specific needs. Each scientific community must be in charge of the day-to-day data standards that it implements. This is why incentives and easy-to-use tools are so important to FAIR practices. Martone suggested targeting neuroscience communities that already have incentives to share data and developing infrastructure for their specific needs. The need to put communities in charge of their own standards is what makes interoperability such a difficult FAIR principle to implement. Bonino suggested that the solution to this issue may be to focus on making the metadata interoperable rather than the data itself. Data formats and types vary too widely among fields to be completely defined at this point. Issues often arise when trying to find semantic relationships between different fields because they use different terms and emphasize different meanings for similar terms. Eliminating the use of implicit semantics in metadata is one important way to improve interoperability.

7.1. Quantifying FAIR in the DataONE repository network

The 42 repositories and over 150,000 researchers that use the DataONE repository network share a common data interface. This makes tracking users’ data practices quite simple. Jones and the team at DataONE broke the FAIR principles down into a series of simple checks to see which scientists using the repository network were complying with FAIR principles [21]. Some of the checks for FAIR compliance are provided in Table 1. MetaDIG is a metadata-checking tool created by the National Center for Ecological Analysis and Synthesis (NCEAS) that is available on Github. It provides an outline that shows which metadata concepts should be included in the analysis of the data in the repository network [26]. The results of the quantification of compliance with FAIR principles should be seen as comparative rather than punitive. Each scientific community must decide on its own requirements. Furthermore, FAIR compliance should not be seen as binary. Compliance exists on a continuum that is heavily dependent on the scientific field in question.

The checks in Table 1 can be used to determine whether individual researchers or institutions comply with the FAIR principles for datasets on different projects or to distinguish between the needs of different fields. The results so far indicate that the Reusable checks tend to have the lowest rates of compliance. Individuals can change their levels of compliance with each new dataset they submit from month to month. The FAIR scores varied widely for each of the repositories within the DataONE network. Each of the FAIR principles were applied with differing levels of compliance to each field and each field presented unique challenges for FAIR compliance. One of the reasons that the FAIR principles were written broadly and somewhat flexibly was so they could be adapted to the needs of any scientific community. These results indicate that community input is needed to determine the necessary levels of FAIR compliance and to interpret the FAIR principles for each unique case.

Lowenberg focused on how credit might be given to those who comply with the FAIR data principles. Currently, sharing data is not really rewarded in academia. This led Lowenberg and her colleagues at UC3 to develop six steps that should lead to reasonable data metrics.

Value data rather than only value publications.

7.3. General discussion on quantification of the FAIR principles

There were some concerns among audience members that metrics of any kind might lead certain researchers to try to take advantage of the system. For example, some researchers might split a dataset produced from their work into smaller pieces or simply recombine and publish other datasets to inflate their citation rates without doing a significant amount of additional work. However, this is often done with research articles and is generally not done purely to increase citations. There are often good reasons to split up both articles and datasets, especially if doing so makes the publication easier to follow or eliminates data that researchers citing the work may find superfluous. Furthermore, combining, cleaning, and publishing data combined from multiple sources is a good practice that can make that data useful to a wider range of researchers. Since it is difficult to distinguish between legitimate and fraudulent publication behavior, the researchers at DataONE felt that the benefits of a user-controlled, open archive model outweighed the potential negative incentives. For that reason, DataONE decided to use a profile-centered design, allowing researchers to decide what goes in each dataset and associating each dataset with the user who published it.