Applying artificial intelligence in the science & technology cycle

This section examines problems in the S&T cycle of varied nature. They are all problems that one could recognize as societal demands. These may not be the demands that directly impact all society members, but indirectly these problems impact us all because they make the cycle inefficient and ineffective, which leads to wastage of funds that were meant to be used for societal benefit.

In the first step of the S&T cycle, between society and funding agencies, investments are transferred for funding agencies to address societal problems. The main problem in this step is that no one knows how much science advancements cost. This means that neither Congress nor funding agencies can predict or at least produce an informed estimation of how much would be needed to fund a desired impact in societal demands. Every other industry invests fortunes to track investments and create financial analysis to evaluate its effectiveness and efficacy. Entire fields such as microeconomics and financial management are dedicated to study investments and return. This is not the case for investments in science and technology. It is consensual that better methods and tools are urgently needed [5]. Statistics indicators and economic models use expenditures to make inferences, but do not help with decision-making [6]. The Organization for Economic Co-operation and Development (OECD) [7] studies and tracks how science and technology compares against foreign countries and the only metric ever used is that of expenditures - outcomes are not considered.

The second problem we propose refers to the structure of funding agencies. Selecting which researchers to fund in order to conduct scientific endeavors is no small task. It is a highly-specialized challenge that requires highly specialized experts. It is not a task that can be fully standardized, which leads to a process that is nearly impossible to evaluate. The main problem is how many researchers this task borrows from actual research projects that will actually better society. It is not only the experts hired by the funding agencies on a permanent basis, but it also includes a large number of researchers who are asked on an ad hoc basis to participate in review panels. The result is that none of these researchers are addressing societal problems. The process is inconsistent and is hardly ever evaluated - making it difficult to improve.

The problems for researchers start with the fact they lack proper supporting tools to do their job competently. The levels of reproducible scientific research are minimal [8]. The rush into publishing to meet requirements affects even the most successful researchers. In 2010, a Nobel laureate retracted two articles published in Science and Nature from earlier years claiming inability to repeat the published findings [9]. There are no means to produce an exhaustive literature review that can guarantee that an author is not committing unintentional plagiarism. It is important to point out that any form of omission of existing related work does not only impact the omitted author, it denies readers the chance to learn about existing work, which leads to duplication [10]. No one can tell how many research projects are redundant. A clinical trial misinformed by an incomplete review has caused the death of a human subject [11]. Nobody knows how much of the produced findings are promoted by the funds distributed by funding agencies.

These same researchers are asked to volunteer to help review proposals for funding and are the ones who actually delivered peer reviews. A similar lack of support is for researchers dealing with data and processing. This is studied as e-science [12], cyberinfrastructure [13], or e-Infrastructure [14]. The main efforts to ease this problem have been in the development of scientific workflows [15]. There is still a lot to be done until the cyberinfrastructure is such that it has made researchers’ work efficient and effective.

The problems with education can (and have) been the topic of multiple books (e.g., [16]). We mention here problems that could be neutralized with proper use of data and that impact the productivity of researchers in the S&T cycle. Curriculum analysis is one of the main problems. In academia, because educators are researchers who keep up-to-date with cutting edge advancements in their fields, they are tasked with analyzing and updating curricula. The problem is one of coverage given that educators can only keep up with the advancements of a narrow sub-field that their research projects focus on. The burden is placed on educators to adjust and adapt curricula and predict the contents necessary to equip students for successful careers.

Another problem in education that directly impacts educators is one that also impacts scientific publishers – it is that of producing educational materials. Scientific publications are used by educators to create course contents, supporting materials, and exams. Unsurprisingly, these same publications end up being published in textbooks. There is no synergy in this process, at least not in general. The educators, who are also researchers, are also the ones who prepare those materials and compose the textbooks. There are no processes to make these tasks synergistic. The copyrights of textbooks are also transferred to the scientific publishers who have to fund the infrastructure to offer and distribute textbooks. In fields that advance quickly, textbooks are often obsolete when published, and yet they still carry a high price tag for students. It is not hard to recognize how inefficient the production and dissemination of scientific knowledge is today.

The scientific publishers hold the copyrights of scientific publications and textbooks so they can be remunerated for their efforts with commercial sales. As stated earlier, the world has changed. This change has modified how decisions are made and how the economy works. This is impacting businesses of all kinds. Today, any company has to be a data company, or it may simply cease to exist. Scientific publishers are at risk of losing their niche market to the Big Tech giants (e.g., Microsoft, Amazon, etc.). Scientific publishers are a regulated industry because of their role in the S&T cycle. Their current situation is a serious problem for everyone.

The last step of the cycle is where scientific publications are to be converted into wealth to meet societal needs. This is yet another problem in the cycle. Studies in individual segments such as pharmaceuticals show how scientific advancements lead to wealth. These studies are isolated and cover several years. But because of the lack of clarity in the rest of the cycle, this last step is even more obscure.

2.2. The opportunities in the science & technology cycle

Referring back to Fig. 1, the main players in the S&T cycle are the people in government, in funding agencies, in education, in research and development (R&D) institutions, in the scientific publishing industry, which includes the industry regulators, and in the organizations that manufacture and distribute new products and services. These organizations are all comprised of people. The changes in our world and economy have made people opportunities (Fig. 3) because they are users of electronic software and services. As argued by Holmen [17], the reason big technology giants are big is because they own the users, not services or products. So a cycle that has so many users, is a cycle of opportunities, as these users are under-served considering the problems discussed above. The opportunity is to serve these users and own them.

OPEN IN VIEWER

Fig. 3.

A cycle of people is a cycle of users and opportunities.

There are other ways to view the S&T cycle. The way we described the cycle starting from the government identifying demands, and then allocating funds to meet said demands, it starts as a cycle of money. When visualizing this as a cycle of money (Fig. 4), it has a new component that comes from the economic leveraging resultant from the new scientific knowledge that affords new products, services, and modifications that open new niches and produce new jobs. Even if the evidence for this wealth may be scarce [5], this is the general gist of the money flow. Because it is a cycle of money, it can be better understood through the volume of money that flows in its different stages. The opportunity is that it can be budgeted.

OPEN IN VIEWER

Fig. 4.

The S&T cycle is a cycle of money.

There is still one other valuable asset populating the entire S&T cycle: it is data. The OPEN Government Data Act refers to the Open Data Policy (M13-13) [18] that mandates federal agencies to publish all their information as open data. Notwithstanding politics and other obstacles, in the United States, the people earned this right. Fortunately, the majority of funding agencies are federal, so there are open data for the two first players. Not all universities are public, but information about all funded research is also open. Their products, however, are not all freely-accessible. When scientific publishers are the players, then data is mostly proprietary. As we move into patents, data is also available. The S&T cycle is indeed a cycle of data (Fig. 5), even if not all of the data is publicly-accessible. This is the greatest opportunity anyone can ever expect now that the world has changed. The data is there and the methods to convert them into user services through automated decision agents are described in the next section.

OPEN IN VIEWER

Fig. 5.

The vision is a Third Wave AI approach to support for users engaged in conducting literature reviews through interpretable citation recommender that leads to trust.

4.1. Third wave of AI for literature reviews

Efforts towards automating literature reviews have been proposed since 1999 [35], and continue to be explored (e.g., [36,37]). Despite having been around for so many years, we do not see these implementations yet available in the mainstream digital libraries that are currently offered to researchers and students. We may conjecture about multiple reasons for this, but the last section suggests the reason is lack of transparency, which prevents users from the opportunity to learn about these systems enough to trust them [30,31]. Consequently, we propose an approach for improving support to users conducting literature reviews based on the Third Wave of AI [38,39].

First, we propose a personalized citation recommender as the central model to search for relevant related research similar to that proposed in Bhagavatula et al. [40] that uses different deep-learning steps to separately address recall and precision. Second, we recommend that this tool relies on state-of-the-art (SOA) contextual embeddings given their exceptional quality (e.g., [4,41]). The personalization in citation recommenders is realized through multiple sentences that users enter as a query that describe the fields of interest better than a few keywords.

For interpretability, we propose that citations are augmented by the class of citation type that would characterize the citation if the user chooses to take action and accept the decision of the recommender and cite it in their work. Given previous work investigating mental models of users of information retrieval systems (e.g., [42,43]), we recommend aligning the explanation with those studies and also present sentences from the query text entered by the user and the publication being recommended demonstrating how they match.

When deployed, these systems will be partnering with humans to increase the aggregated understanding of scientific literature. These systems, in combination with methods for extraction of claims [44] and other crucial elements of scientific discourse [22], will support the proposed vision. Some of the services that the resultant technologies would be able to provide with high degrees of accuracy include answers to questions such as, “What are the open questions in this field? Which scientific contributions are redundant? Which scientific contributions are contradictory?”

4.2. Third wave of AI for funding agencies

Placing a program officer from a funding agency as supervisor of multiple automated decision agents that can deal with big data and a high level of understanding of the scientific literature can decrease the volume of highly-qualified experts allocated to the agency. This allows more highly qualified researchers to do their job, which is to produce novel scientific findings for the betterment of society.

A widely-studied problem related to allocating federal funds is the evaluation of researcher quality [45]. The limitations of the Second Wave to automate some of its decisions has contributed to the creation of the Leiden Manifesto [46], which dictates ten principles to be followed when assessing research quality. Not surprisingly, one of them makes it a requirement that methods are transparent, presenting adequate explanations that justify assessments. One AI methodology conceived in the First Wave, case-based reasoning (CBR) [47], affords explanations with high levels of transparency, and was hence the methodology proposed in [45] to assess researcher quality and meet Leiden Manifesto’s principles. The initial version [45] does not yet include explanations, but already meets other principles [ibid].

Still considering the work of funding agencies with respect to researchers, one application that has been vastly studied is that of expert locator systems. Researchers are located today and assessed based on their profiles in applications such as Google Scholar. These profiles are semi-automatically populated and are incomplete. Furthermore, they are partially self-populated, which has its own sets of limitations. The better understanding of the scientific literature can also help with this task. One idea that has been proposed [48] is to characterize researchers based upon their most recent contributions to the scientific literature [ibid]. This would eliminate the limitations of both automatic- and self-populated expert locators. Advancements such as those from [44] to extract crucial passages from scientific discourse and the development of a representation structure for scientific contributions such as the one in [22] can be used for an interpretable expert locator system.

There is a plethora of applications that can be developed in support of processes in funding agencies that include the summarization and the drafting of documents. These all can benefit from the better understanding of the scientific literature such as identifying gaps and making correlations to societal demands, which become possibly achievable goals.

Another set of applications in support of funding agencies includes those based on data analytics using open governmental data. One such application is the budgeting of scientific advances, which can help answer questions about the cost of science. The investigations needed for this budgeting are beyond the scope of this article.

4.3. Third wave of AI for education

Third Wave AI applications for education target partnerships with pedagogists and educators. These applications require both extensive data analytics and the advanced representation of scientific knowledge discussed above. They encompass services for curriculum creation and maintenance, course contents, and course materials.

Curriculum creation requires a combination of data analytics to help us predict the placement of students in the workforce. Barbosky [49] stated how the demand for data analysts is larger than the market can fill. Note, however, that the big data revolution has been observed in the scientific literature [50] since 2000. This means that if educators who are tasked to create, maintain, and adapt curricula were supported by the proper knowledge of scientific facts, they would have realized that the curricula had to account for this labor demand. We propose applications where curricula experts would be supervising multiple automated decision agents that would constantly and continuously adapt curricula to markets’ needs. This reality would render obsolete course contents at ever academic year or semester.

To meet the new demands of agile curricula management, Third Wave AI applications would need to use a representation framework [22] to prepare course contents, textbook materials, presentations, videos, and assignments [34]. An initial strategy for such applications is given in [ibid].

4.4. Third wave of AI for scientific content creation

In this section, we discuss Third Wave AI applications for scientific content creation, as opposed to the previous section where we discussed educational content creation. These applications are needed in support of researchers as they deliver scientific projects to learn new scientific findings.

Scientific workflows [15] make experimental repeatability possible through implementing steps to deliver scientific discovery. When representing experiments, they enable reproducible contributions, decreasing the chance of error in manual handling and increasing the quality of experiments. There are various systems for managing workflows such as taverna.org.uk and pegasus.isi.edu. These systems provide a window into the future of the automation of the production of knowledge. To select an existing workflow, researchers do not have to know how to write code. They merely need to understand if that operation is what they need. Once researchers begin using a workflow system, they benefit from features such as statistical analyses, visualizations, etc.

The vision we propose for Third Wave AI support for scientific content creation connects scientific workflows with a representation framework [22], working as metadata for scientific workflows. Consider how much easier it would be to manage scientific conferences with semi-standardized conference submissions. Consider, for example, a visualization procedure such as the ROC chart method for comparison of discrete classifiers [51]: (see also: Receiver Operating Characteristic at https://www.semanticscholar.org/paper/ROC-Graphs%3A-Notes-and-Practical-Considerations-for-Fawcett/44bb1605c4ab8a8ce2764fa20424f6a148101ca4). Now consider that every author of a conference paper that uses the code that creates a ROC chart has to include a couple of sentences describing it with a citation. What we are proposing is that the draft of the conference paper be automatically prepared including the sentences explaining what a ROC chart is and why it is advantageous over other methods, its required references, and the citations populated in the reference list. This kind of application would be an ideal set of requirements for a Third Wave AI conference management system.

This same approach for semi-automation of conference papers can be used for preparation of journal manuscripts. Besides the methodology section as mentioned above, other parts of a manuscript can be drafted with the literature review and motivation being the lowest hanging opportunities. Based on recent efforts in automating literature reviews to create summaries of these reviews, summaries of publications with specific foci may be ideal for this application. Once accepting a citation recommended to incorporate into a manuscript, the citation recommender personalization may be able to offer capabilities beyond those proposed above and produce the drafts of the description of how the recommended article is relevant to the user-entered query draft.

The development of automated drafting is a consequence of the better understanding of a representation framework for scientific knowledge and the automated analysis of scientific literature. Sections like results can utilize metadata from scientific workflows and a draft of a discussion section can explore meta discussions learned from other publications.

5.1. Benefits

Applying AI to automated decision making to provide users of the S&T cycle with answers to many unanswered questions has multiple benefits. The most immediate is greater understanding of the cycle itself, which shall directly lead to better decisions. This responds to the challenges posed by Lane and Bertuzzi [52] to create a system to capture, analyze, distribute, and represent scientific data in support of adequate measurement, budgeting, and economic analyses. These needed tasks will make the S&T cycle transparent to its players, but furthermore they are business opportunities that can be undertaken by the publishing industry, which has plenty of room to grow.

The most significant result from making the S&T cycle transparent would likely be the automated analysis of scientific literature. Being able to automatically, and with high degrees of accuracy, identify research gaps and redundant works would increase efficiency of an industry that, in 2018, was estimated at $1.3 trillion dollars [53]. Another potential realization from the automated analysis of scientific literature is to enable easy access to scientific facts. Consider that easier access to scientific facts can be a crucial weapon against ignorance and the use of fallacies by advertisers and politicians, enhancing the education of society. Better access to scientific facts is also a weapon against crimes such as those categorized as confidence crimes and fake news. Another result would be the understanding of which disciplines generate wealth and which do not, at least in the short term. This would encourage private investments and allow governments to invest in the fields that would not be supported by for-profit segments.

Third Wave AI methods are deployed in partnerships with human users. This new paradigm will afford further discoveries about computer support for automated decision making in the cycle. For example, Third Wave AI support for literature reviews will lead to further developments in automated analysis of scientific literature because users will supervise the results and provide the needed understanding for automated systems that may be lacking today.

In addition to direct decisions stemming from AI applications, the benefits of proposed AI applications are expected to impact organizations from the cycle. Consider how funding agencies rely on so many highly-specialized experts that could be directly working on addressing societal improvements. Organizations that prioritize their own processes rather than their final goal are infected by a form of bureaucratic disease. Relieving scientists from the process and freeing them to work in research projects will attenuate this unwanted phenomenon.

The further development of Third Wave AI applications for the S&T cycle by federally-funded researchers promotes advances in the fields of XAI, applied AI, and domain areas such as Science of Science. These advances will lead to the development of applications that follow standards for information services thus guaranteeing access to knowledge to all readers.

Agile creation of curriculum and course contents will enable the reduction of time from the moment that a new scientific advance is learned to the time that it reaches educators and students and the marketplace. This may also make the connection between the scientific publishers and creation of wealth faster because today there is a gap from the moment that a patent is filed to the point where organizations can have an educated and trained workforce to deliver the new knowledge. Agile curricula and course contents would benefit this step of the S&T cycle.

Benefits of automating content creation will facilitate the work of researchers and scientific publishers. Today, the publishing industry is investing in tools to streamline the infrastructure needed in support of peer-reviewed publications [54]. One example is the use of automated proof editing and plagiarism detection. Publishers claim a lot of time is wasted using volunteer reviewers who often dismiss manuscripts due to their low presentation quality. Consider how this problem would be attenuated with semi-automated drafting of manuscripts that would eliminate ambiguity in the composition of a manuscript.

5.2. Challenges

There are many challenges to the realization of so many benefits. The challenges that pose the most risk to the realization of the vision in this article are, not surprisingly, of economic nature. The social challenges are mostly in earning user trust and in humans learning to adapt to a new paradigm interacting with automated agents. There are also some technical challenges.

Researchers do not have access to the full text of all scientific publications. It is quite unusual to find a scientific article that manages to get authorization from scientific publishers to use full texts. One example used 15 million full-text articles [55]. In the same year this academic work was published, a news article disclosed that Microsoft had 176 million articles in their academic graph [56]. Google Scholar had 160 million articles in 2014 [57]. This challenge is multifold. First, scientific publishers do not want to lose what may be the last source of revenue they have as technology giants like Microsoft, Facebook, Apple and Amazon own the users and are taking over this market [17]. Second, scientific publishers are only preventing themselves and researchers from producing alternatives that can actually stand a chance in competing against technology giants. These two challenges have multiple consequences and need to be better explained.

Technology giants are powerful, and they are already offering services to users from the S&T cycle, namely, Google Scholar. They do not yet solve the problem, but the challenge is that they will propose solutions that will shrink our knowledge base because they are not regulated and have no obligation to follow the standards that guarantee readers access to scientific knowledge. They are only committed to remunerating their shareholders. Furthermore, their processes are not transparent, and it is easy to manipulate citation counting [58].

Scientific publishers are utilizing and offering services with limited data, and their efforts may be described as timid in comparison to technology giants because they either do not use full text or only use their own copyrighted publications, or just have a small number of alliances that allow them to use copyrighted materials from their competitors. Consider, for example, a tool to detect plagiarism [54] that uses full-text data from just a handful of publishers. This tool will miss out on detecting plagiarism unless it uses all scientific publications. The same problem occurs with recommender systems. TrendMD [59] provides recommendation of articles based on articles a user reads or retrieves. TrendMD reported having about 300 partnerships [ibid], but they still only use abstracts and not full text.

Earning the trust of users is the main goal of XAI [26,29,30]. This body of research is moving in this direction. Although this may still be considered a challenge, it is only a matter of time until automated agents built with high technical standards are distinguishable by users and they are the ones that will be trusted.

Another social challenge refers to the shift in the nature of work. Humans need to adapt to a new paradigm to become supervisors of automated agents. This is also a challenge that XAI will contribute to overcome.